2024-03-24T00:15:45.622 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T00:15:45.623 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/exports} --name mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --verbose --owner mchangir --archive /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439 -- /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439/orig.config.yaml 2024-03-24T00:15:45.664 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T00:15:45.741 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439 branch: wip-mchangir-testing-main-20240318.032620 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/exports} email: mchangir@redhat.com first_in_suite: false job_id: '7616439' kernel: flavor: default kdb: true sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - Replacing daemon mds sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 3550 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 613/1024 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOIROjU8alZ2MN5QoW9J73ZR2EA4xzHHJqXzdUUZhxpZn2xukBbmH5MmEP4f2xX3XR+PCznb+rsWN6qVGjT02Zg= smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKNKWqVs1XCeXnKRCFKfBlzYJzTo46EZn7pnrIMmZ42DE0CEaUJrjKTjobTmS/4GXn53Bvm8aYKZEGA5g+eICNI= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_exports teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_09:49:57 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T00:15:45.741 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-24T00:15:45.742 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-24T00:15:45.742 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T00:15:45.744 INFO:teuthology.task.internal:Checking packages... 2024-03-24T00:15:45.767 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-24T00:15:45.768 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T00:15:45.768 INFO:teuthology.packaging:ref: None 2024-03-24T00:15:45.768 INFO:teuthology.packaging:tag: None 2024-03-24T00:15:45.768 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-24T00:15:45.768 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:15:45.768 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-24T00:15:45.992 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145-g3be9a5c9-1jammy 2024-03-24T00:15:45.993 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T00:15:46.000 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T00:15:46.000 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T00:15:46.033 INFO:teuthology.task.internal:Saving configuration 2024-03-24T00:15:46.049 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T00:15:46.067 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T00:15:46.091 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:10:01.929763', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIN0Fmvas8nTc0v9JtH7xnZOo2IRj3XTPqJRBT6AsQo4'} 2024-03-24T00:15:46.113 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:10:01.927952', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIl7BNz2llhjTU+FArjO2fiJyQQb82bGcvnIAIJOa2+Q'} 2024-03-24T00:15:46.113 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T00:15:46.119 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-24T00:15:46.119 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-24T00:15:46.119 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T00:15:46.219 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f948e67a3a0>, signals=[15]) 2024-03-24T00:15:46.219 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T00:15:46.224 INFO:teuthology.task.internal:Opening connections... 2024-03-24T00:15:46.224 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:15:46.226 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:15:46.308 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:15:46.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:15:46.387 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T00:15:46.393 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-03-24T00:15:46.399 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-03-24T00:15:46.399 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-03-24T00:15:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-24T00:15:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="Ubuntu" 2024-03-24T00:15:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="22.04" 2024-03-24T00:15:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-24T00:15:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_CODENAME=jammy 2024-03-24T00:15:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:ID=ubuntu 2024-03-24T00:15:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE=debian 2024-03-24T00:15:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-24T00:15:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-24T00:15:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-24T00:15:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-24T00:15:46.445 INFO:teuthology.orchestra.run.smithi040.stdout:UBUNTU_CODENAME=jammy 2024-03-24T00:15:46.446 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-03-24T00:15:46.473 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-03-24T00:15:46.478 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-03-24T00:15:46.478 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-03-24T00:15:46.525 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-03-24T00:15:46.525 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="Ubuntu" 2024-03-24T00:15:46.525 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="22.04" 2024-03-24T00:15:46.525 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-03-24T00:15:46.525 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_CODENAME=jammy 2024-03-24T00:15:46.525 INFO:teuthology.orchestra.run.smithi053.stdout:ID=ubuntu 2024-03-24T00:15:46.525 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE=debian 2024-03-24T00:15:46.526 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://www.ubuntu.com/" 2024-03-24T00:15:46.526 INFO:teuthology.orchestra.run.smithi053.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-03-24T00:15:46.526 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-03-24T00:15:46.526 INFO:teuthology.orchestra.run.smithi053.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-03-24T00:15:46.526 INFO:teuthology.orchestra.run.smithi053.stdout:UBUNTU_CODENAME=jammy 2024-03-24T00:15:46.526 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-03-24T00:15:46.547 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T00:15:46.556 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T00:15:46.584 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T00:15:46.584 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-03-24T00:15:46.587 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-03-24T00:15:46.591 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T00:15:46.617 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T00:15:46.617 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-03-24T00:15:46.634 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-03-24T00:15:46.650 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T00:15:46.704 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T00:15:46.718 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'} 2024-03-24T00:15:46.718 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}}, timeout 300 2024-03-24T00:15:46.719 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T00:15:46.719 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T00:15:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:15:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:15:46.725 DEBUG:teuthology.task.kernel:sha1 for mon.a is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T00:15:46.725 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-24T00:15:46.725 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-03-24T00:15:46.726 DEBUG:teuthology.task.kernel:sha1 for mon.b is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T00:15:46.726 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-24T00:15:46.726 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-03-24T00:15:46.770 INFO:teuthology.orchestra.run.smithi053.stdout:5.15.0-56-generic 2024-03-24T00:15:46.770 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T00:15:46.771 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-24T00:15:46.771 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T00:15:46.775 INFO:teuthology.orchestra.run.smithi040.stdout:5.15.0-56-generic 2024-03-24T00:15:46.775 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T00:15:46.775 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-24T00:15:46.775 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T00:15:46.996 INFO:teuthology.task.kernel:Installing linux-firmware on mon.b... 2024-03-24T00:15:46.997 DEBUG:teuthology.orchestra.run.smithi053:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-03-24T00:15:47.005 INFO:teuthology.task.kernel:Installing linux-firmware on mon.a... 2024-03-24T00:15:47.005 DEBUG:teuthology.orchestra.run.smithi040:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-03-24T00:15:47.022 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-03-24T00:15:47.027 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-03-24T00:15:47.152 INFO:teuthology.orchestra.run.smithi040.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-03-24T00:15:47.152 INFO:teuthology.orchestra.run.smithi040.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-03-24T00:15:47.152 INFO:teuthology.orchestra.run.smithi040.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-03-24T00:15:47.152 INFO:teuthology.orchestra.run.smithi040.stderr:hint: 2024-03-24T00:15:47.152 INFO:teuthology.orchestra.run.smithi040.stderr:hint: git config --global init.defaultBranch 2024-03-24T00:15:47.152 INFO:teuthology.orchestra.run.smithi040.stderr:hint: 2024-03-24T00:15:47.152 INFO:teuthology.orchestra.run.smithi040.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-03-24T00:15:47.152 INFO:teuthology.orchestra.run.smithi040.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-03-24T00:15:47.152 INFO:teuthology.orchestra.run.smithi040.stderr:hint: 2024-03-24T00:15:47.153 INFO:teuthology.orchestra.run.smithi040.stderr:hint: git branch -m 2024-03-24T00:15:47.153 INFO:teuthology.orchestra.run.smithi040.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-03-24T00:15:47.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-03-24T00:15:47.158 INFO:teuthology.orchestra.run.smithi053.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-03-24T00:15:47.159 INFO:teuthology.orchestra.run.smithi053.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-03-24T00:15:47.159 INFO:teuthology.orchestra.run.smithi053.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-03-24T00:15:47.159 INFO:teuthology.orchestra.run.smithi053.stderr:hint: 2024-03-24T00:15:47.159 INFO:teuthology.orchestra.run.smithi053.stderr:hint: git config --global init.defaultBranch 2024-03-24T00:15:47.159 INFO:teuthology.orchestra.run.smithi053.stderr:hint: 2024-03-24T00:15:47.159 INFO:teuthology.orchestra.run.smithi053.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-03-24T00:15:47.159 INFO:teuthology.orchestra.run.smithi053.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-03-24T00:15:47.159 INFO:teuthology.orchestra.run.smithi053.stderr:hint: 2024-03-24T00:15:47.159 INFO:teuthology.orchestra.run.smithi053.stderr:hint: git branch -m 2024-03-24T00:15:47.160 INFO:teuthology.orchestra.run.smithi053.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-03-24T00:15:47.162 DEBUG:teuthology.orchestra.run.smithi053:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-03-24T00:15:47.220 DEBUG:teuthology.orchestra.run.smithi040:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-03-24T00:15:47.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-03-24T00:15:47.276 DEBUG:teuthology.orchestra.run.smithi040:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-03-24T00:15:47.288 DEBUG:teuthology.orchestra.run.smithi053:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-03-24T00:16:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-03-24T00:16:45.216 INFO:teuthology.orchestra.run.smithi040.stderr: * [new branch] main -> origin/main 2024-03-24T00:16:45.250 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20190312 -> 20190312 2024-03-24T00:16:45.250 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20190416 -> 20190416 2024-03-24T00:16:45.250 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20190514 -> 20190514 2024-03-24T00:16:45.250 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20190618 -> 20190618 2024-03-24T00:16:45.251 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20190717 -> 20190717 2024-03-24T00:16:45.251 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20190815 -> 20190815 2024-03-24T00:16:45.251 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20190923 -> 20190923 2024-03-24T00:16:45.251 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20191022 -> 20191022 2024-03-24T00:16:45.251 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20191215 -> 20191215 2024-03-24T00:16:45.251 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20200122 -> 20200122 2024-03-24T00:16:45.252 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20200316 -> 20200316 2024-03-24T00:16:45.252 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20200421 -> 20200421 2024-03-24T00:16:45.252 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20200519 -> 20200519 2024-03-24T00:16:45.252 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20200619 -> 20200619 2024-03-24T00:16:45.252 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20200721 -> 20200721 2024-03-24T00:16:45.252 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20200817 -> 20200817 2024-03-24T00:16:45.253 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20200918 -> 20200918 2024-03-24T00:16:45.253 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20201022 -> 20201022 2024-03-24T00:16:45.253 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20201118 -> 20201118 2024-03-24T00:16:45.253 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20201218 -> 20201218 2024-03-24T00:16:45.254 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20210208 -> 20210208 2024-03-24T00:16:45.254 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20210315 -> 20210315 2024-03-24T00:16:45.254 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20210511 -> 20210511 2024-03-24T00:16:45.254 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20210716 -> 20210716 2024-03-24T00:16:45.255 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20210818 -> 20210818 2024-03-24T00:16:45.255 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20210919 -> 20210919 2024-03-24T00:16:45.255 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20211027 -> 20211027 2024-03-24T00:16:45.255 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20211216 -> 20211216 2024-03-24T00:16:45.256 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20220209 -> 20220209 2024-03-24T00:16:45.256 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20220310 -> 20220310 2024-03-24T00:16:45.256 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20220411 -> 20220411 2024-03-24T00:16:45.256 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20220509 -> 20220509 2024-03-24T00:16:45.256 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20220610 -> 20220610 2024-03-24T00:16:45.257 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20220708 -> 20220708 2024-03-24T00:16:45.257 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20220815 -> 20220815 2024-03-24T00:16:45.257 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20220913 -> 20220913 2024-03-24T00:16:45.257 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20221012 -> 20221012 2024-03-24T00:16:45.258 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20221109 -> 20221109 2024-03-24T00:16:45.258 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20221214 -> 20221214 2024-03-24T00:16:45.258 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20230117 -> 20230117 2024-03-24T00:16:45.258 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20230210 -> 20230210 2024-03-24T00:16:45.258 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20230310 -> 20230310 2024-03-24T00:16:45.259 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20230404 -> 20230404 2024-03-24T00:16:45.259 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20230515 -> 20230515 2024-03-24T00:16:45.259 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20230625 -> 20230625 2024-03-24T00:16:45.259 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20230804 -> 20230804 2024-03-24T00:16:45.259 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20230919 -> 20230919 2024-03-24T00:16:45.259 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20231030 -> 20231030 2024-03-24T00:16:45.260 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20231111 -> 20231111 2024-03-24T00:16:45.260 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20231211 -> 20231211 2024-03-24T00:16:45.260 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20240115 -> 20240115 2024-03-24T00:16:45.260 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20240220 -> 20240220 2024-03-24T00:16:45.260 INFO:teuthology.orchestra.run.smithi040.stderr: * [new tag] 20240312 -> 20240312 2024-03-24T00:16:46.307 INFO:teuthology.orchestra.run.smithi053.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-03-24T00:16:46.307 INFO:teuthology.orchestra.run.smithi053.stderr: * [new branch] main -> origin/main 2024-03-24T00:16:46.341 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20190312 -> 20190312 2024-03-24T00:16:46.342 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20190416 -> 20190416 2024-03-24T00:16:46.342 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20190514 -> 20190514 2024-03-24T00:16:46.342 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20190618 -> 20190618 2024-03-24T00:16:46.342 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20190717 -> 20190717 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20190815 -> 20190815 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20190923 -> 20190923 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20191022 -> 20191022 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20191215 -> 20191215 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20200122 -> 20200122 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20200316 -> 20200316 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20200421 -> 20200421 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20200519 -> 20200519 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20200619 -> 20200619 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20200721 -> 20200721 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20200817 -> 20200817 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20200918 -> 20200918 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20201022 -> 20201022 2024-03-24T00:16:46.344 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20201118 -> 20201118 2024-03-24T00:16:46.345 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20201218 -> 20201218 2024-03-24T00:16:46.347 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20210208 -> 20210208 2024-03-24T00:16:46.347 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20210315 -> 20210315 2024-03-24T00:16:46.347 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20210511 -> 20210511 2024-03-24T00:16:46.347 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20210716 -> 20210716 2024-03-24T00:16:46.347 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20210818 -> 20210818 2024-03-24T00:16:46.347 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20210919 -> 20210919 2024-03-24T00:16:46.347 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20211027 -> 20211027 2024-03-24T00:16:46.347 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20211216 -> 20211216 2024-03-24T00:16:46.347 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20220209 -> 20220209 2024-03-24T00:16:46.348 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20220310 -> 20220310 2024-03-24T00:16:46.348 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20220411 -> 20220411 2024-03-24T00:16:46.348 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20220509 -> 20220509 2024-03-24T00:16:46.348 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20220610 -> 20220610 2024-03-24T00:16:46.348 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20220708 -> 20220708 2024-03-24T00:16:46.348 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20220815 -> 20220815 2024-03-24T00:16:46.348 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20220913 -> 20220913 2024-03-24T00:16:46.348 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20221012 -> 20221012 2024-03-24T00:16:46.348 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20221109 -> 20221109 2024-03-24T00:16:46.348 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20221214 -> 20221214 2024-03-24T00:16:46.348 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20230117 -> 20230117 2024-03-24T00:16:46.348 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20230210 -> 20230210 2024-03-24T00:16:46.350 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20230310 -> 20230310 2024-03-24T00:16:46.350 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20230404 -> 20230404 2024-03-24T00:16:46.350 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20230515 -> 20230515 2024-03-24T00:16:46.350 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20230625 -> 20230625 2024-03-24T00:16:46.350 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20230804 -> 20230804 2024-03-24T00:16:46.350 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20230919 -> 20230919 2024-03-24T00:16:46.350 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20231030 -> 20231030 2024-03-24T00:16:46.350 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20231111 -> 20231111 2024-03-24T00:16:46.350 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20231211 -> 20231211 2024-03-24T00:16:46.350 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20240115 -> 20240115 2024-03-24T00:16:46.350 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20240220 -> 20240220 2024-03-24T00:16:46.350 INFO:teuthology.orchestra.run.smithi053.stderr: * [new tag] 20240312 -> 20240312 2024-03-24T00:16:49.674 INFO:teuthology.orchestra.run.smithi040.stderr:Updating files: 48% (1758/3598) Updating files: 49% (1764/3598) Updating files: 50% (1799/3598) Updating files: 51% (1835/3598) Updating files: 52% (1871/3598) Updating files: 53% (1907/3598) Updating files: 54% (1943/3598) Updating files: 55% (1979/3598) Updating files: 56% (2015/3598) Updating files: 57% (2051/3598) Updating files: 58% (2087/3598) Updating files: 59% (2123/3598) Updating files: 60% (2159/3598) Updating files: 61% (2195/3598) Updating files: 61% (2225/3598) Updating files: 62% (2231/3598) Updating files: 63% (2267/3598) Updating files: 64% (2303/3598) Updating files: 65% (2339/3598) Updating files: 66% (2375/3598) Updating files: 67% (2411/3598) Updating files: 68% (2447/3598) Updating files: 69% (2483/3598) Updating files: 70% (2519/3598) Updating files: 71% (2555/3598) Updating files: 71% (2580/3598) Updating files: 72% (2591/3598) Updating files: 73% (2627/3598) Updating files: 74% (2663/3598) Updating files: 75% (2699/3598) Updating files: 76% (2735/3598) Updating files: 77% (2771/3598) Updating files: 78% (2807/3598) Updating files: 79% (2843/3598) Updating files: 80% (2879/3598) Updating files: 81% (2915/3598) Updating files: 82% (2951/3598) Updating files: 83% (2987/3598) Updating files: 83% (3008/3598) Updating files: 84% (3023/3598) Updating files: 85% (3059/3598) Updating files: 86% (3095/3598) Updating files: 87% (3131/3598) Updating files: 88% (3167/3598) Updating files: 89% (3203/3598) Updating files: 90% (3239/3598) Updating files: 91% (3275/3598) Updating files: 92% (3311/3598) Updating files: 93% (3347/3598) Updating files: 94% (3383/3598) Updating files: 95% (3419/3598) Updating files: 96% (3455/3598) Updating files: 97% (3491/3598) Updating files: 98% (3527/3598) Updating files: 99% (3563/3598) Updating files: 100% (3598/3598) Updating files: 100% (3598/3598), done. 2024-03-24T00:16:49.680 INFO:teuthology.orchestra.run.smithi040.stdout:HEAD is now at e068ccf0 Merge branch 'robot/pr-0-1711100280' into 'main' 2024-03-24T00:16:49.702 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.a... 2024-03-24T00:16:49.702 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T00:16:49.851 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g5ec3e6e88b2e/ 2024-03-24T00:16:49.851 DEBUG:teuthology.orchestra.run.smithi040:> rm -f /tmp/linux-image.deb && echo linux-image-6.8.0-g5ec3e6e88b2e_6.8.0-g5ec3e6e88b2e-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g5ec3e6e88b2e/ --input-file=- 2024-03-24T00:16:49.854 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi040.front.sepia.ceph.com to complete... 2024-03-24T00:16:50.786 INFO:teuthology.orchestra.run.smithi053.stderr:Updating files: 48% (1755/3598) Updating files: 49% (1764/3598) Updating files: 50% (1799/3598) Updating files: 51% (1835/3598) Updating files: 52% (1871/3598) Updating files: 53% (1907/3598) Updating files: 54% (1943/3598) Updating files: 55% (1979/3598) Updating files: 56% (2015/3598) Updating files: 57% (2051/3598) Updating files: 58% (2087/3598) Updating files: 59% (2123/3598) Updating files: 60% (2159/3598) Updating files: 61% (2195/3598) Updating files: 61% (2224/3598) Updating files: 62% (2231/3598) Updating files: 63% (2267/3598) Updating files: 64% (2303/3598) Updating files: 65% (2339/3598) Updating files: 66% (2375/3598) Updating files: 67% (2411/3598) Updating files: 68% (2447/3598) Updating files: 69% (2483/3598) Updating files: 70% (2519/3598) Updating files: 71% (2555/3598) Updating files: 71% (2580/3598) Updating files: 72% (2591/3598) Updating files: 73% (2627/3598) Updating files: 74% (2663/3598) Updating files: 75% (2699/3598) Updating files: 76% (2735/3598) Updating files: 77% (2771/3598) Updating files: 78% (2807/3598) Updating files: 79% (2843/3598) Updating files: 80% (2879/3598) Updating files: 81% (2915/3598) Updating files: 82% (2951/3598) Updating files: 83% (2987/3598) Updating files: 83% (3008/3598) Updating files: 84% (3023/3598) Updating files: 85% (3059/3598) Updating files: 86% (3095/3598) Updating files: 87% (3131/3598) Updating files: 88% (3167/3598) Updating files: 89% (3203/3598) Updating files: 90% (3239/3598) Updating files: 91% (3275/3598) Updating files: 92% (3311/3598) Updating files: 93% (3347/3598) Updating files: 94% (3383/3598) Updating files: 95% (3419/3598) Updating files: 96% (3455/3598) Updating files: 97% (3491/3598) Updating files: 98% (3527/3598) Updating files: 99% (3563/3598) Updating files: 100% (3598/3598) Updating files: 100% (3598/3598), done. 2024-03-24T00:16:50.791 INFO:teuthology.orchestra.run.smithi053.stdout:HEAD is now at e068ccf0 Merge branch 'robot/pr-0-1711100280' into 'main' 2024-03-24T00:16:50.813 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.b... 2024-03-24T00:16:50.813 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-24T00:16:50.957 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g5ec3e6e88b2e/ 2024-03-24T00:16:50.957 DEBUG:teuthology.orchestra.run.smithi053:> rm -f /tmp/linux-image.deb && echo linux-image-6.8.0-g5ec3e6e88b2e_6.8.0-g5ec3e6e88b2e-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://3.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g5ec3e6e88b2e/ --input-file=- 2024-03-24T00:16:50.960 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi053.front.sepia.ceph.com to complete... 2024-03-24T00:16:51.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24 00:16:52 URL:https://3.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g5ec3e6e88b2e/linux-image-6.8.0-g5ec3e6e88b2e_6.8.0-g5ec3e6e88b2e-1_amd64.deb [39270258/39270258] -> "/tmp/linux-image.deb" [1] 2024-03-24T00:16:51.374 INFO:teuthology.orchestra.run.smithi040.stderr:FINISHED --2024-03-24 00:16:52-- 2024-03-24T00:16:51.374 INFO:teuthology.orchestra.run.smithi040.stderr:Total wall clock time: 1.5s 2024-03-24T00:16:51.374 INFO:teuthology.orchestra.run.smithi040.stderr:Downloaded: 1 files, 37M in 1.4s (27.1 MB/s) 2024-03-24T00:16:51.378 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.a... 2024-03-24T00:16:51.378 DEBUG:teuthology.orchestra.run.smithi040:> sudo dpkg -i /tmp/linux-image.deb 2024-03-24T00:16:51.621 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-image-6.8.0-g5ec3e6e88b2e. 2024-03-24T00:16:52.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24 00:16:53 URL:https://3.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/ubuntu/jammy/flavors/default/pool/main/l/linux-6.8.0-g5ec3e6e88b2e/linux-image-6.8.0-g5ec3e6e88b2e_6.8.0-g5ec3e6e88b2e-1_amd64.deb [39270258/39270258] -> "/tmp/linux-image.deb" [1] 2024-03-24T00:16:52.705 INFO:teuthology.orchestra.run.smithi053.stderr:FINISHED --2024-03-24 00:16:53-- 2024-03-24T00:16:52.705 INFO:teuthology.orchestra.run.smithi053.stderr:Total wall clock time: 1.7s 2024-03-24T00:16:52.705 INFO:teuthology.orchestra.run.smithi053.stderr:Downloaded: 1 files, 37M in 1.4s (26.7 MB/s) 2024-03-24T00:16:52.709 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on mon.b... 2024-03-24T00:16:52.709 DEBUG:teuthology.orchestra.run.smithi053:> sudo dpkg -i /tmp/linux-image.deb 2024-03-24T00:16:52.916 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package linux-image-6.8.0-g5ec3e6e88b2e. 2024-03-24T00:16:53.632 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-03-24T00:16:53.636 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-03-24T00:16:53.883 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-image-6.8.0-g5ec3e6e88b2e (6.8.0-g5ec3e6e88b2e-1) ... 2024-03-24T00:16:54.933 INFO:teuthology.orchestra.run.smithi053.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-03-24T00:16:54.937 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-03-24T00:16:55.039 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking linux-image-6.8.0-g5ec3e6e88b2e (6.8.0-g5ec3e6e88b2e-1) ... 2024-03-24T00:16:57.773 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-image-6.8.0-g5ec3e6e88b2e (6.8.0-g5ec3e6e88b2e-1) ... 2024-03-24T00:16:57.874 INFO:teuthology.orchestra.run.smithi040.stderr:update-initramfs: Generating /boot/initrd.img-6.8.0-g5ec3e6e88b2e 2024-03-24T00:16:58.594 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up linux-image-6.8.0-g5ec3e6e88b2e (6.8.0-g5ec3e6e88b2e-1) ... 2024-03-24T00:16:58.715 INFO:teuthology.orchestra.run.smithi053.stderr:update-initramfs: Generating /boot/initrd.img-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:19.520 INFO:teuthology.orchestra.run.smithi053.stderr:Sourcing file `/etc/default/grub' 2024-03-24T00:17:19.534 INFO:teuthology.orchestra.run.smithi053.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-24T00:17:19.558 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2024-03-24T00:17:19.870 INFO:teuthology.orchestra.run.smithi053.stderr:Found linux image: /boot/vmlinuz-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:19.883 INFO:teuthology.orchestra.run.smithi053.stderr:Found initrd image: /boot/initrd.img-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:19.978 INFO:teuthology.orchestra.run.smithi040.stderr:Sourcing file `/etc/default/grub' 2024-03-24T00:17:19.993 INFO:teuthology.orchestra.run.smithi040.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-24T00:17:20.045 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2024-03-24T00:17:20.165 INFO:teuthology.orchestra.run.smithi053.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-24T00:17:20.172 INFO:teuthology.orchestra.run.smithi053.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-24T00:17:20.233 INFO:teuthology.orchestra.run.smithi053.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-24T00:17:20.238 INFO:teuthology.orchestra.run.smithi053.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-24T00:17:20.397 INFO:teuthology.orchestra.run.smithi053.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-24T00:17:20.397 INFO:teuthology.orchestra.run.smithi053.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-24T00:17:20.397 INFO:teuthology.orchestra.run.smithi053.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-24T00:17:20.429 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:20.443 INFO:teuthology.orchestra.run.smithi053.stderr:done 2024-03-24T00:17:20.448 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:20.639 DEBUG:teuthology.orchestra.run.smithi053:> dpkg-deb -c /tmp/linux-image.deb 2024-03-24T00:17:20.647 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./ 2024-03-24T00:17:20.647 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./boot/ 2024-03-24T00:17:20.647 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 4474370 2024-03-19 20:13 ./boot/System.map-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:20.660 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 170904 2024-03-19 20:13 ./boot/config-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:20.661 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11555840 2024-03-19 20:13 ./boot/vmlinuz-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:20.681 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/ 2024-03-24T00:17:20.681 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/ 2024-03-24T00:17:20.681 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/postinst.d/ 2024-03-24T00:17:20.681 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/postrm.d/ 2024-03-24T00:17:20.681 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/preinst.d/ 2024-03-24T00:17:20.681 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/prerm.d/ 2024-03-24T00:17:20.681 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/ 2024-03-24T00:17:20.681 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/ 2024-03-24T00:17:20.681 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/ 2024-03-24T00:17:20.681 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/ 2024-03-24T00:17:20.682 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/ 2024-03-24T00:17:20.682 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/ 2024-03-24T00:17:20.682 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ 2024-03-24T00:17:20.682 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 490449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/aesni-intel.ko 2024-03-24T00:17:20.684 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-24T00:17:20.685 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-24T00:17:20.685 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-24T00:17:20.685 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 93081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-24T00:17:20.685 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 106697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-24T00:17:20.685 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 152145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-24T00:17:20.685 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-24T00:17:20.685 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-24T00:17:20.685 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 83857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-24T00:17:20.685 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-24T00:17:20.685 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-24T00:17:20.685 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-24T00:17:20.685 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-24T00:17:20.686 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-24T00:17:20.686 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-24T00:17:20.686 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 58489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-24T00:17:20.686 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 50393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-24T00:17:20.686 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-24T00:17:20.686 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-24T00:17:20.687 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/ 2024-03-24T00:17:20.687 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/ 2024-03-24T00:17:20.687 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/ 2024-03-24T00:17:20.687 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 41433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-24T00:17:20.687 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpuid.ko 2024-03-24T00:17:20.687 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/msr.ko 2024-03-24T00:17:20.688 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/ 2024-03-24T00:17:20.688 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 354265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-24T00:17:20.688 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 886177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-24T00:17:20.688 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 2294937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-24T00:17:20.703 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ 2024-03-24T00:17:20.703 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 53841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/af_alg.ko 2024-03-24T00:17:20.703 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/algif_hash.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/algif_skcipher.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/asymmetric_keys/ 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/ 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-24T00:17:20.704 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cryptd.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_simd.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-24T00:17:20.705 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lrw.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-24T00:17:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 143033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xts.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ 2024-03-24T00:17:20.707 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_extlog.ko 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/apei/ 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/apei/einj.ko 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 104433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 89937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci.ko 2024-03-24T00:17:20.708 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 88313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-24T00:17:20.709 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 52177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_legacy.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-24T00:17:20.710 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 37697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_radisys.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rz1000.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-24T00:17:20.711 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 83257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 70305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 37281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 44705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ 2024-03-24T00:17:20.712 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/ 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 124457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/ 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/ 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 113833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/ 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 964817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 169705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/ 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 94217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 116337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/ 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 110329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 98193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-24T00:17:20.713 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 263641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-24T00:17:20.758 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/ 2024-03-24T00:17:20.758 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 98121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-24T00:17:20.758 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/ 2024-03-24T00:17:20.758 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 58777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-24T00:17:20.758 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/applicom.ko 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/ 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 148521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 139361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-24T00:17:20.759 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 70873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 53417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/ 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 84025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/nvram.ko 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/ 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-24T00:17:20.760 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/ 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/xillybus_class.ko 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 50849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/xillybus_core.ko 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/ 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-palmas.ko 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-si5351.ko 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-twl6040.ko 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-wm831x.ko 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/ 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-24T00:17:20.761 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 70473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 246841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/ 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/ 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/ 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 570497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/ 2024-03-24T00:17:20.762 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 70433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/ 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/ 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 122817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 80305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 41641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-24T00:17:20.763 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 56737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/ 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-gpio.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-max14577.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-max77693.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-max8997.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-palmas.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-rt8973a.ko 2024-03-24T00:17:20.764 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-sm5502.ko 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/ 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/dmi-sysfs.ko 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/efi/ 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/efi/efi-pstore.ko 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/ 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-adp5520.ko 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-amd8111.ko 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-arizona.ko 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-da9052.ko 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-da9055.ko 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-dln2.ko 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 77753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-f7188x.ko 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-generic.ko 2024-03-24T00:17:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-kempld.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-lp3943.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-max7300.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-max730x.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-max732x.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-pca953x.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-pcf857x.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-rdc321x.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-sch.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-sch311x.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-tps65912.ko 2024-03-24T00:17:20.766 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-twl4030.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-twl6040.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-vx855.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-wm831x.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-wm8350.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-wm8994.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/ 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-a4tech.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-apple.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-belkin.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-cherry.ko 2024-03-24T00:17:20.767 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-chicony.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 60665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-cp2112.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-cypress.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ezkey.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-generic.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-24T00:17:20.768 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kensington.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lg-g15.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 57977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 101545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 80921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-magicmouse.ko 2024-03-24T00:17:20.769 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-microsoft.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-monterey.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 50281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ntrig.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 72865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-24T00:17:20.770 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-24T00:17:20.771 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 70993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 143393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-24T00:17:20.772 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 269233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid.ko 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/ 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 125641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/usbhid.ko 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/usbkbd.ko 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/usbmouse.ko 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 218881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/ 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 64305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 90113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 304953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 66801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-24T00:17:20.773 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 61817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 97889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 45465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 74321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-24T00:17:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 69313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 55113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 58529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 40729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/da9052-hwmon.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/da9055-hwmon.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 99513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-24T00:17:20.775 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc2103.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 66897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 94601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 41313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 55361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 46713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 45441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hih6130.ko 2024-03-24T00:17:20.776 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 140137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/jc42.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 48513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-24T00:17:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 72745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 46065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 37825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 108753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-24T00:17:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mc13783-adc.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-03-24T00:17:20.779 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 119849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 61441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 59649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 82721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 54345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-24T00:17:20.780 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 82481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 40489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 40361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 37289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-24T00:17:20.781 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 43449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 52369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 53209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 58265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 79841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 71257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 63137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 71233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-24T00:17:20.782 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 101385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 119417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/ 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/ 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/ 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-03-24T00:17:20.783 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 6833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 50305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 67689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-24T00:17:20.784 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 50977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-24T00:17:20.785 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/ 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 273529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 883945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-24T00:17:20.786 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 69185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 324337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 97841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 282345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 66057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/ 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/ 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 493185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/ 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 420593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/ 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 741737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 296881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-24T00:17:20.787 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/ocrdma/ 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 153729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/usnic/ 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 169145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 266577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 100281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 111337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 128665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ 2024-03-24T00:17:20.788 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 129801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/ 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/evbug.ko 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/ff-memless.ko 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/ 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 41457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 41713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/ 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 37281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-24T00:17:20.789 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/lkkbd.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/lm8323.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/lm8333.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/newtonkbd.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/qt1070.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/qt2160.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/stowaway.ko 2024-03-24T00:17:20.790 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/sunkbd.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/xtkbd.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/matrix-keymap.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/ 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 123201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 89393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/gpio_mouse.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 335921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/psmouse.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-24T00:17:20.791 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/ 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 109601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/ 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/ct82c710.ko 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/parkbd.ko 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/pcips2.ko 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/ps2mult.ko 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serport.ko 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-24T00:17:20.792 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/ 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-88pm860x.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-adp5520.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 59657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-bd2802.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-da903x.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-da9052.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-gpio.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3533.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm355x.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3642.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp8788.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-max8997.ko 2024-03-24T00:17:20.793 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-mc13783.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-menf21bmc.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-pca9532.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-pca955x.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-pca963x.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-tca6507.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-wm831x-status.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-wm8350.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-24T00:17:20.794 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/ 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/ 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 647121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 40489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 93321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bufio.ko 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 155825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 107937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 53769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-era.ko 2024-03-24T00:17:20.795 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 41441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-mirror.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 77881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 70985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-region-hash.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 109457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-snapshot.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 181777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 53737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-zero.ko 2024-03-24T00:17:20.796 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/ 2024-03-24T00:17:20.797 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 199881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-24T00:17:20.797 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 37321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-24T00:17:20.797 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 100841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-24T00:17:20.797 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 126369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-24T00:17:20.797 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 368497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-24T00:17:20.797 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/ 2024-03-24T00:17:20.797 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/ 2024-03-24T00:17:20.797 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 178049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-24T00:17:20.840 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 63241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-24T00:17:20.840 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 46249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-24T00:17:20.840 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 37121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-24T00:17:20.840 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 123233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-24T00:17:20.840 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 80921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-24T00:17:20.840 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 52865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-24T00:17:20.840 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/ 2024-03-24T00:17:20.840 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/88pm800.ko 2024-03-24T00:17:20.840 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/88pm805.ko 2024-03-24T00:17:20.840 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/88pm80x.ko 2024-03-24T00:17:20.840 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/arizona-i2c.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 123649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/arizona.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/bcm590xx.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/cros_ec_dev.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/dln2.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/janz-cmodio.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 53489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/kempld-core.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lm3533-core.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lp3943.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 56969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/max8907.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mc13xxx-core.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-03-24T00:17:20.841 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/menf21bmc.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/pcf50633-adc.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/pcf50633-gpio.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/pcf50633.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/retu-mfd.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 80345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/si476x-core.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 48369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/tps65010.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/tps6507x.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 44265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wm8994.ko 2024-03-24T00:17:20.842 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 48393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ad525x_dpot.ko 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/ 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 46513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 45497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/c2port/ 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/c2port/core.ko 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/ 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ds1682.ko 2024-03-24T00:17:20.843 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/dummy-irq.ko 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/echo/ 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/echo/echo.ko 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/ 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/genwqe/ 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 152073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hmc6352.ko 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ibmasm/ 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 72561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ics932s401.ko 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-24T00:17:20.844 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/ 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/ 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 126153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 56377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 313601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/phantom.ko 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ti-st/ 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 74305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ti-st/st_drv.ko 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-24T00:17:20.845 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 48345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/ 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 183569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/ 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 491609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/ 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/ 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 105937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 40225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/ 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 459001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-24T00:17:20.846 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/ 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 45961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 130665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 68785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 64617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 101617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/ 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 54625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/ 2024-03-24T00:17:20.847 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 69169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/ 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 59993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/ 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 41409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/ 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 54617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 87457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/ 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/ 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 97201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 102497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-03-24T00:17:20.848 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 90209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/ 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 81321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 69545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/ 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 78385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 176697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1434505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 128369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/ 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 131281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 314329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/ 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/ 2024-03-24T00:17:20.849 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 323737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/ 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 193593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 347217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1045449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 166145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/ 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/ 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 204249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-24T00:17:20.850 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/ 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/ 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 69433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 61425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 141201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 62833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/ 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 60673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 62833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 45489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ec_bhf.ko 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/ 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/ 2024-03-24T00:17:20.851 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 281217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fujitsu/ 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 45553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 90305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/ 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 295697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/ 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 584353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/ 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 246041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/ 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 944545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-24T00:17:20.852 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/ 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 288529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/ 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 513673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/ 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 93489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 735169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 151745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 86993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/ 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 102441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 122609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-24T00:17:20.853 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/ 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 727481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 278881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1122297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 52985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 40881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 95577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/ 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 108409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-24T00:17:20.854 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 73617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/ 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 151097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/ 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 136801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/ 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 50433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/ 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 227865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 87017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-03-24T00:17:20.855 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 672057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/ 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/ 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 67281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 83809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 194073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/samsung/ 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 104049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/ 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1067001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/ 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-24T00:17:20.856 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/ 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 54729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 63201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/ 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 61425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 67297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 73257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 68049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/ 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 495777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/ 2024-03-24T00:17:20.857 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 103185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 166049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 66273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 78721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/ 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 50017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/ 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 92345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/ 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 79937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 83721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/ 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 46313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-24T00:17:20.858 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/ 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 59921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/ 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 41185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/defxx.ko 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/skfp/ 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 217113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/skfp/skfp.ko 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 74993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/ 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 237265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/ 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 85857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 62897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-24T00:17:20.859 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/ 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-gpio.ko 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 45057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/ 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-24T00:17:20.860 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 44633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 44825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 81961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 76569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 99009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-24T00:17:20.861 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/plip/ 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 43769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/plip/plip.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sb1000.ko 2024-03-24T00:17:20.862 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/ 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 64561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/ 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 107233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 86825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/ 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 144361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-24T00:17:20.863 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/ 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 307193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/ 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 144017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/ 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/ 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 267745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 103361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/ 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 131329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 85217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-24T00:17:20.864 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/ 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/ 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pci-stub.ko 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 46369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/ 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 146153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pcmcia.ko 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 62081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pcmcia_core.ko 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 112793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/ 2024-03-24T00:17:20.865 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/broadcom/ 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/samsung/ 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/ 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/ 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 69033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/ 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/ 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec.ko 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-03-24T00:17:20.866 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/ 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 73601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 59569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 131945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 40169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-24T00:17:20.867 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/ibm_rtl.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/ 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 44537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-24T00:17:20.868 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 48481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/ 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/88pm860x_battery.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/88pm860x_charger.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 54961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/bq2415x_charger.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 59505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/bq24190_charger.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/bq24735-charger.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/da9030_battery.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/da9052-battery.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2760_battery.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2780_battery.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2781_battery.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2782_battery.ko 2024-03-24T00:17:20.869 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/gpio-charger.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/isp1704_charger.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/lp8727_charger.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max14577_charger.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max17042_battery.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max8903_charger.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max8925_power.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/pcf50633-charger.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 37841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/sbs-battery.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/test_power.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/tps65090-charger.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/wm831x_backup.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/wm831x_power.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/wm8350_power.ko 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/ 2024-03-24T00:17:20.870 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 58105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/ 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/ 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 41801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 70873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/ 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-88pm80x.ko 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-88pm860x.ko 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-24T00:17:20.871 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-da9052.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-da9055.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-da9063.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 68481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-24T00:17:20.872 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-lp8788.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t86.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8907.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8925.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8997.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8998.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-palmas.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 46537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-24T00:17:20.873 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf50633.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rc5t583.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-s35390a.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-tps6586x.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-tps65910.ko 2024-03-24T00:17:20.874 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-wm831x.ko 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-wm8350.ko 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 77529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 57721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 55017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 135345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/ 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 271441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 100473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/ 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 356865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 295929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-24T00:17:20.875 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic94xx/ 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 189145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/ 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 108209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/ 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 246937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/ 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1171953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/ 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 238401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/ 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 115329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 40785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/ 2024-03-24T00:17:20.876 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 308961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/ 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 76041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 101425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 146753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 75977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/ 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 212385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 60505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/ 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 78225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 157729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-24T00:17:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fdomain.ko 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/ 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 294953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 201729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 48945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/imm.ko 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 43977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 265945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 101881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/ 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 317193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 52865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-24T00:17:20.878 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/ 2024-03-24T00:17:20.879 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 315321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-24T00:17:20.879 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 145497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-24T00:17:20.879 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 53809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-24T00:17:20.879 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/ 2024-03-24T00:17:20.879 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 216401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-24T00:17:20.879 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/ 2024-03-24T00:17:20.879 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 2194089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-24T00:17:20.900 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-24T00:17:20.912 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-24T00:17:20.926 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/ 2024-03-24T00:17:20.927 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 73473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-24T00:17:20.927 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-24T00:17:20.927 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 310689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-24T00:17:20.927 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 85521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-24T00:17:20.927 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/ 2024-03-24T00:17:20.927 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 659633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-24T00:17:20.927 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/ 2024-03-24T00:17:20.927 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 119489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-24T00:17:20.927 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 78041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-24T00:17:20.927 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/ 2024-03-24T00:17:20.927 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 81561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-03-24T00:17:20.927 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-03-24T00:17:20.928 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-03-24T00:17:20.928 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-03-24T00:17:20.928 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/ 2024-03-24T00:17:20.928 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 366337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-24T00:17:20.928 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 124601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-24T00:17:20.928 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ppa.ko 2024-03-24T00:17:20.928 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 55337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-24T00:17:20.928 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/ 2024-03-24T00:17:20.928 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1876753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-24T00:17:20.928 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 85193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-24T00:17:20.928 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/ 2024-03-24T00:17:20.928 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 558953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qlogicfas408.ko 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 261081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 173929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 280113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 97625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 73865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 44889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 115649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 46737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/ 2024-03-24T00:17:20.929 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 164889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 44857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 50233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/xen-scsifront.ko 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 129393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/ 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/ 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 736761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/ 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-24T00:17:20.930 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 931521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 44153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 122489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/ 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 75521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/ 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/ 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/ 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-24T00:17:20.931 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-24T00:17:20.932 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-24T00:17:20.932 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-24T00:17:20.932 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-24T00:17:20.932 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-24T00:17:20.932 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-24T00:17:20.932 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-24T00:17:20.932 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-24T00:17:20.932 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-24T00:17:20.932 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-24T00:17:20.932 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-24T00:17:20.932 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 78353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 57777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/moxa.ko 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 50865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/mxser.ko 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 108489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 67841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/ 2024-03-24T00:17:20.933 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/ 2024-03-24T00:17:20.934 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-03-24T00:17:20.934 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 59401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-24T00:17:20.934 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-03-24T00:17:20.934 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/altera_uart.ko 2024-03-24T00:17:20.934 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-24T00:17:20.934 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 72553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-03-24T00:17:20.934 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/ 2024-03-24T00:17:20.934 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 101993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-24T00:17:20.934 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/rp2.ko 2024-03-24T00:17:20.934 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 53177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/sc16is7xx.ko 2024-03-24T00:17:20.934 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/ 2024-03-24T00:17:20.934 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ 2024-03-24T00:17:20.935 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 545401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-24T00:17:20.935 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ 2024-03-24T00:17:20.935 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-24T00:17:20.935 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-03-24T00:17:20.935 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/ 2024-03-24T00:17:20.935 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 44473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-24T00:17:20.935 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-24T00:17:20.935 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-24T00:17:20.935 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_dmem_genirq.ko 2024-03-24T00:17:20.935 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_mf624.ko 2024-03-24T00:17:20.935 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_netx.ko 2024-03-24T00:17:20.935 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/ 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/c67x00/ 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 73225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/c67x00/c67x00.ko 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 145793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/ 2024-03-24T00:17:20.936 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 87009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 89769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/common/ 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/common/ulpi.ko 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc2/ 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/ 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 452825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/dwc3.ko 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/ 2024-03-24T00:17:20.937 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/ 2024-03-24T00:17:20.938 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 60329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/u_ether.ko 2024-03-24T00:17:20.938 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/u_serial.ko 2024-03-24T00:17:20.938 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-03-24T00:17:20.938 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-03-24T00:17:20.938 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-03-24T00:17:20.938 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-03-24T00:17:20.938 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 92409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-03-24T00:17:20.938 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 50945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-03-24T00:17:20.938 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 122257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-03-24T00:17:20.938 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 59865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-03-24T00:17:20.938 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-03-24T00:17:20.938 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-03-24T00:17:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 57633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-03-24T00:17:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 85521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-03-24T00:17:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-03-24T00:17:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 58025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-03-24T00:17:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 67353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-03-24T00:17:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/ 2024-03-24T00:17:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-03-24T00:17:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-03-24T00:17:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-03-24T00:17:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-03-24T00:17:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-03-24T00:17:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-03-24T00:17:20.940 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-03-24T00:17:20.940 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-03-24T00:17:20.940 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-03-24T00:17:20.940 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-03-24T00:17:20.940 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-03-24T00:17:20.940 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-03-24T00:17:20.940 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 62369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-03-24T00:17:20.940 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-03-24T00:17:20.940 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 161233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/libcomposite.ko 2024-03-24T00:17:20.940 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/ 2024-03-24T00:17:20.940 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-03-24T00:17:20.940 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/bdc/ 2024-03-24T00:17:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 157617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-03-24T00:17:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-03-24T00:17:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 52929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-03-24T00:17:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 69617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-03-24T00:17:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 55953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-03-24T00:17:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 75969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/net2272.ko 2024-03-24T00:17:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 74985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/net2280.ko 2024-03-24T00:17:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 73017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-03-24T00:17:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 65881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-03-24T00:17:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 66001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-03-24T00:17:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 81409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-03-24T00:17:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 156337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/ 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/bcma-hcd.ko 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 80017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/isp116x-hcd.ko 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 87193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 78777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811_cs.ko 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/ssb-hcd.ko 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/ 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-24T00:17:20.942 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 41633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/cytherm.ko 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ehset.ko 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-24T00:17:20.943 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/lvstest.ko 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/ 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 64745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 70721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbtest.ko 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 41817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/mon/ 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 72825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/mon/usbmon.ko 2024-03-24T00:17:20.944 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/musb/ 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 301633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/musb/musb_hdrc.ko 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/ 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-isp1301.ko 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-tahvo.ko 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 70281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-24T00:17:20.945 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-24T00:17:20.946 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 58121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-24T00:17:20.946 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-24T00:17:20.946 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-24T00:17:20.946 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 44185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/f81232.ko 2024-03-24T00:17:20.946 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 175105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-24T00:17:20.946 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 53017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-24T00:17:20.946 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 136985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-24T00:17:20.946 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 124745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-24T00:17:20.946 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 67681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-24T00:17:20.946 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-24T00:17:20.946 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-24T00:17:20.946 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 58137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-24T00:17:20.947 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 107433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-24T00:17:20.947 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-24T00:17:20.947 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-24T00:17:20.947 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-24T00:17:20.947 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-24T00:17:20.947 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/metro-usb.ko 2024-03-24T00:17:20.947 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 66361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-24T00:17:20.947 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 78129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-24T00:17:20.947 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 56249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mxuport.ko 2024-03-24T00:17:20.947 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-24T00:17:20.947 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-24T00:17:20.947 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 191825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 54265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 61393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-24T00:17:20.948 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 125529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usbserial.ko 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 37313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/wishbone-serial.ko 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 52761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-24T00:17:20.949 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-24T00:17:20.950 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-24T00:17:20.950 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-24T00:17:20.950 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-24T00:17:20.950 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-24T00:17:20.950 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-24T00:17:20.950 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-24T00:17:20.950 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-24T00:17:20.950 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-24T00:17:20.950 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-24T00:17:20.950 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 180329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-24T00:17:20.950 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/ 2024-03-24T00:17:20.950 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 67401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 71025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 104065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/ 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/ 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 162865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 103673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 76441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/ 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 113665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 58001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-24T00:17:20.951 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 89057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/ 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/ 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/aat2870_bl.ko 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/adp5520_bl.ko 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/adp8860_bl.ko 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/adp8870_bl.ko 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/as3711_bl.ko 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/bd6107.ko 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/da903x_bl.ko 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/da9052_bl.ko 2024-03-24T00:17:20.952 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/gpio_backlight.ko 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/kb3886_bl.ko 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lm3533_bl.ko 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lm3639_bl.ko 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lv5207lp.ko 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/max8925_bl.ko 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/pandora_bl.ko 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/wm831x_bl.ko 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/ 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ 2024-03-24T00:17:20.953 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-24T00:17:20.954 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-24T00:17:20.954 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/matrox_w1.ko 2024-03-24T00:17:20.954 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/w1-gpio.ko 2024-03-24T00:17:20.954 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/ 2024-03-24T00:17:20.954 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-24T00:17:20.954 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-24T00:17:20.954 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-24T00:17:20.954 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-24T00:17:20.954 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-24T00:17:20.954 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-24T00:17:20.954 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-24T00:17:20.954 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-24T00:17:20.955 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-24T00:17:20.955 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-24T00:17:20.955 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 56337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-24T00:17:20.955 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 103049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-24T00:17:20.955 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ 2024-03-24T00:17:20.955 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/acquirewdt.ko 2024-03-24T00:17:20.955 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/advantechwdt.ko 2024-03-24T00:17:20.955 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-24T00:17:20.955 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-24T00:17:20.955 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/cpu5wdt.ko 2024-03-24T00:17:20.955 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/da9052_wdt.ko 2024-03-24T00:17:20.955 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/da9055_wdt.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/da9063_wdt.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/dw_wdt.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/eurotechwdt.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-24T00:17:20.956 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/kempld_wdt.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/mena21_wdt.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pc87413_wdt.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/retu_wdt.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-03-24T00:17:20.957 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sc1200wdt.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/twl4030_wdt.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wafer5823wdt.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-24T00:17:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wm831x_wdt.ko 2024-03-24T00:17:20.959 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wm8350_wdt.ko 2024-03-24T00:17:20.959 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-24T00:17:20.959 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/ 2024-03-24T00:17:20.959 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-24T00:17:20.959 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-24T00:17:20.959 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-24T00:17:20.959 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/ 2024-03-24T00:17:20.959 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 176385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-24T00:17:20.959 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 40097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-24T00:17:20.959 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 71153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-scsiback.ko 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/ 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/ 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 158281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/adfs/ 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 61721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/adfs/adfs.ko 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/ 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 163417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/afs/ 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1019225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/afs/kafs.ko 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/autofs/ 2024-03-24T00:17:20.960 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 103225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/autofs/autofs4.ko 2024-03-24T00:17:20.961 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/ 2024-03-24T00:17:20.961 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 64625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-24T00:17:20.961 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/bfs/ 2024-03-24T00:17:20.961 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 41017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/bfs/bfs.ko 2024-03-24T00:17:20.961 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-24T00:17:20.961 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/ 2024-03-24T00:17:20.961 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 3487553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-24T00:17:20.961 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/ 2024-03-24T00:17:20.961 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 305897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-24T00:17:20.961 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ 2024-03-24T00:17:20.961 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1335721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-24T00:17:20.961 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/ 2024-03-24T00:17:20.962 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 97769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-24T00:17:20.962 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/configfs/ 2024-03-24T00:17:20.962 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 108577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/configfs/configfs.ko 2024-03-24T00:17:20.962 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/ 2024-03-24T00:17:20.962 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-24T00:17:20.962 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/ 2024-03-24T00:17:20.962 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 644249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-24T00:17:20.962 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/efs/ 2024-03-24T00:17:20.962 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/efs/efs.ko 2024-03-24T00:17:20.962 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/ 2024-03-24T00:17:20.962 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1527665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-24T00:17:20.962 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/ 2024-03-24T00:17:20.963 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-24T00:17:20.963 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/freevxfs/ 2024-03-24T00:17:20.963 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/freevxfs/freevxfs.ko 2024-03-24T00:17:20.963 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/ 2024-03-24T00:17:20.963 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-24T00:17:20.963 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/ 2024-03-24T00:17:20.963 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1231361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-24T00:17:20.963 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/ 2024-03-24T00:17:20.963 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 141713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-24T00:17:20.963 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/ 2024-03-24T00:17:20.963 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 223281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-24T00:17:20.963 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hpfs/ 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 193545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hpfs/hpfs.ko 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/ 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 74737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/ 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 420449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/ 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 281705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/ 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 88937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/ 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 490505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/ 2024-03-24T00:17:20.964 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/ 2024-03-24T00:17:20.965 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 81457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-24T00:17:20.965 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/ 2024-03-24T00:17:20.965 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 59353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-24T00:17:20.965 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/ 2024-03-24T00:17:20.965 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 117377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-24T00:17:20.965 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1058377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-24T00:17:20.965 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 66361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv2.ko 2024-03-24T00:17:20.965 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 120193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-24T00:17:20.965 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 2041825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-24T00:17:20.965 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/ 2024-03-24T00:17:20.965 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-24T00:17:20.965 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-24T00:17:20.966 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/ 2024-03-24T00:17:20.966 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1511665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-24T00:17:20.966 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/ 2024-03-24T00:17:20.966 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 582529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-24T00:17:20.966 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/ 2024-03-24T00:17:20.966 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-24T00:17:20.966 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-24T00:17:20.966 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-24T00:17:20.966 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-24T00:17:20.966 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-24T00:17:20.966 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-24T00:17:20.966 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-24T00:17:20.967 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-24T00:17:20.967 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-24T00:17:20.967 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-24T00:17:20.967 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-24T00:17:20.967 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ascii.ko 2024-03-24T00:17:20.967 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-24T00:17:20.967 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-24T00:17:20.967 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-24T00:17:20.967 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-24T00:17:20.967 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-24T00:17:20.967 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-24T00:17:20.967 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 94041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 142577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-24T00:17:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 165649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 118025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-24T00:17:20.969 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ntfs/ 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 220545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ntfs/ntfs.ko 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 244673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 440681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ 2024-03-24T00:17:20.970 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 54177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-24T00:17:20.971 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 2590849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-24T00:17:20.971 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-24T00:17:20.971 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-24T00:17:20.971 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-24T00:17:20.971 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/omfs/ 2024-03-24T00:17:20.971 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 53705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/omfs/omfs.ko 2024-03-24T00:17:20.971 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/ 2024-03-24T00:17:20.971 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 360953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-24T00:17:20.971 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ 2024-03-24T00:17:20.971 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 56945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-24T00:17:20.971 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx4/ 2024-03-24T00:17:20.971 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx4/qnx4.ko 2024-03-24T00:17:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx6/ 2024-03-24T00:17:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 46625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx6/qnx6.ko 2024-03-24T00:17:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/ 2024-03-24T00:17:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/quota_tree.ko 2024-03-24T00:17:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/quota_v1.ko 2024-03-24T00:17:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/quota_v2.ko 2024-03-24T00:17:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/ 2024-03-24T00:17:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 532289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-24T00:17:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/ 2024-03-24T00:17:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-24T00:17:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/ 2024-03-24T00:17:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/ 2024-03-24T00:17:20.973 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 2712601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-24T00:17:20.986 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-24T00:17:20.986 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-24T00:17:21.014 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/ 2024-03-24T00:17:21.015 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-24T00:17:21.015 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-24T00:17:21.015 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/ 2024-03-24T00:17:21.015 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 134809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-24T00:17:21.015 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/ 2024-03-24T00:17:21.015 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 91329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-24T00:17:21.015 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/ 2024-03-24T00:17:21.015 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 248409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-24T00:17:21.015 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ 2024-03-24T00:17:21.015 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 162785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-24T00:17:21.015 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/ 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 4036193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/ 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/locking/ 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 80489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/locking/locktorture.ko 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/time/ 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/time/test_udelay.ko 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 60265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 6737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 6441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc7.ko 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 6937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/ 2024-03-24T00:17:21.016 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-24T00:17:21.017 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-24T00:17:21.017 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/interval_tree_test.ko 2024-03-24T00:17:21.017 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-24T00:17:21.017 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-24T00:17:21.017 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/ 2024-03-24T00:17:21.017 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-24T00:17:21.017 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-24T00:17:21.017 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/ 2024-03-24T00:17:21.017 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 6529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-24T00:17:21.017 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/memory-notifier-error-inject.ko 2024-03-24T00:17:21.017 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/notifier-error-inject.ko 2024-03-24T00:17:21.017 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/percpu_test.ko 2024-03-24T00:17:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/pm-notifier-error-inject.ko 2024-03-24T00:17:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/ 2024-03-24T00:17:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 143337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-24T00:17:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/rbtree_test.ko 2024-03-24T00:17:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/ 2024-03-24T00:17:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-24T00:17:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test-kstrtox.ko 2024-03-24T00:17:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test-string_helpers.ko 2024-03-24T00:17:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 4863225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_bpf.ko 2024-03-24T00:17:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 48993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_firmware.ko 2024-03-24T00:17:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 6281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_module.ko 2024-03-24T00:17:21.018 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_user_copy.ko 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/xz/ 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/xz/xz_dec_test.ko 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/ 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/ 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 55777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 6961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 6969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-24T00:17:21.019 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 6961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-24T00:17:21.020 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 6897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-24T00:17:21.020 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 6969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-24T00:17:21.020 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 6905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-24T00:17:21.020 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-24T00:17:21.020 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/ 2024-03-24T00:17:21.020 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-24T00:17:21.020 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-24T00:17:21.020 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-24T00:17:21.020 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-24T00:17:21.020 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-24T00:17:21.020 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/ 2024-03-24T00:17:21.020 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 82897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/ 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 198833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/ 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 667545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/ 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 67769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 790673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-24T00:17:21.021 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-24T00:17:21.022 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-24T00:17:21.022 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-24T00:17:21.022 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-24T00:17:21.022 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-24T00:17:21.022 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-24T00:17:21.022 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-24T00:17:21.022 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-24T00:17:21.022 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-24T00:17:21.022 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-24T00:17:21.022 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-24T00:17:21.022 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-24T00:17:21.022 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-24T00:17:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-24T00:17:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-24T00:17:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-24T00:17:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-24T00:17:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-24T00:17:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-24T00:17:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 55945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-24T00:17:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-24T00:17:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-24T00:17:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/ 2024-03-24T00:17:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1046825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-24T00:17:21.023 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/ 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 125705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/ 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 194505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 58745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 61401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/ 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 390089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-24T00:17:21.024 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/hsr/ 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 102073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/hsr/hsr.ko 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 45505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 284721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 47881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 48249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 51217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-24T00:17:21.025 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 61137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-24T00:17:21.026 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 61513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-24T00:17:21.026 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-24T00:17:21.026 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-24T00:17:21.026 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 29081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-24T00:17:21.026 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ 2024-03-24T00:17:21.026 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-24T00:17:21.026 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-24T00:17:21.026 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-24T00:17:21.026 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ip_tables.ko 2024-03-24T00:17:21.026 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-24T00:17:21.026 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-24T00:17:21.027 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-24T00:17:21.028 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-03-24T00:17:21.028 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-24T00:17:21.028 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-24T00:17:21.028 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-24T00:17:21.028 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-24T00:17:21.028 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-24T00:17:21.028 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-24T00:17:21.028 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-24T00:17:21.028 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-24T00:17:21.028 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-24T00:17:21.028 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-24T00:17:21.028 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 45057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-24T00:17:21.029 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 79929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_gre.ko 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 94169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 54273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/mip6.ko 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-24T00:17:21.030 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-24T00:17:21.031 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 74881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/ 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 84401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-24T00:17:21.032 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/ 2024-03-24T00:17:21.033 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 125209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-24T00:17:21.033 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-24T00:17:21.033 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 25817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-24T00:17:21.033 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-24T00:17:21.033 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 37377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-24T00:17:21.033 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-24T00:17:21.033 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 53233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-24T00:17:21.033 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/lapb/ 2024-03-24T00:17:21.033 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 48121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/lapb/lapb.ko 2024-03-24T00:17:21.033 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/ 2024-03-24T00:17:21.033 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 268521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc2.ko 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/ 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 309417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/ 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 119705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-24T00:17:21.034 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 99993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 100281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 102553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 102345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 111673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 59193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 108345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 112145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 112409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 110337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 113289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ 2024-03-24T00:17:21.035 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 412513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-24T00:17:21.036 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 411769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 118161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 108313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-24T00:17:21.037 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 66537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 92233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 96521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-24T00:17:21.038 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 44137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 623473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 42513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 22937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 40809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-24T00:17:21.039 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_osf.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-24T00:17:21.040 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_synproxy.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tproxy.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 89233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/x_tables.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 18081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 38777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-24T00:17:21.041 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-24T00:17:21.042 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-24T00:17:21.042 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-24T00:17:21.042 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-24T00:17:21.042 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-24T00:17:21.042 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-24T00:17:21.042 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-24T00:17:21.042 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-24T00:17:21.042 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-24T00:17:21.042 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-24T00:17:21.042 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-24T00:17:21.042 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-24T00:17:21.042 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-24T00:17:21.043 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-24T00:17:21.043 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-24T00:17:21.043 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-24T00:17:21.043 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-24T00:17:21.043 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-24T00:17:21.043 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-24T00:17:21.043 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-24T00:17:21.043 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-24T00:17:21.043 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-24T00:17:21.043 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-24T00:17:21.043 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-24T00:17:21.043 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-24T00:17:21.044 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-24T00:17:21.044 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-24T00:17:21.044 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-24T00:17:21.044 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-24T00:17:21.044 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 43201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-24T00:17:21.044 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-24T00:17:21.044 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-24T00:17:21.044 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-24T00:17:21.044 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-24T00:17:21.044 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-24T00:17:21.044 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-24T00:17:21.044 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-24T00:17:21.045 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-24T00:17:21.045 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-24T00:17:21.045 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-24T00:17:21.045 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-24T00:17:21.045 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-24T00:17:21.045 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-24T00:17:21.045 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-24T00:17:21.045 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-24T00:17:21.045 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-24T00:17:21.045 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-24T00:17:21.045 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-24T00:17:21.046 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-24T00:17:21.046 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-24T00:17:21.046 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-24T00:17:21.046 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-24T00:17:21.046 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-24T00:17:21.046 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-24T00:17:21.046 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 16817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-24T00:17:21.046 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-24T00:17:21.046 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-24T00:17:21.046 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-24T00:17:21.046 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 8457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-24T00:17:21.046 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpudp.ko 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/ 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 14441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/ 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 12337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/ 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 384201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/ 2024-03-24T00:17:21.047 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/phonet/ 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 112553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/phonet/phonet.ko 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 55057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/phonet/pn_pep.ko 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/ 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 253961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 165105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 68097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rxrpc/ 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 828297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rxrpc/rxrpc.ko 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/ 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-24T00:17:21.048 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 37089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 43353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 19185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_cgroup.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 39521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-24T00:17:21.049 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 62657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-24T00:17:21.050 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-24T00:17:21.050 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 13321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-24T00:17:21.050 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 33425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-24T00:17:21.050 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-24T00:17:21.050 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-24T00:17:21.050 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-24T00:17:21.050 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-24T00:17:21.050 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-24T00:17:21.050 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 34593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-24T00:17:21.050 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-24T00:17:21.050 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq_codel.ko 2024-03-24T00:17:21.050 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 35457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-24T00:17:21.051 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 49849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-24T00:17:21.051 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 21977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-24T00:17:21.051 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 66449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-24T00:17:21.051 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-24T00:17:21.051 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 30225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-24T00:17:21.051 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-24T00:17:21.051 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-24T00:17:21.051 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 36153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-24T00:17:21.051 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 27409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-24T00:17:21.051 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-24T00:17:21.051 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 24777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-24T00:17:21.051 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 48537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 32817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 26793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 31321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 28273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/ 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 932945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/ 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/ 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 337009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 55041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-24T00:17:21.052 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 1513905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-24T00:17:21.053 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/ 2024-03-24T00:17:21.053 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 704761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-24T00:17:21.053 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/ 2024-03-24T00:17:21.053 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 10401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-24T00:17:21.053 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 867249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-24T00:17:21.053 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/ 2024-03-24T00:17:21.053 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-24T00:17:21.053 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/ 2024-03-24T00:17:21.053 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 87745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-24T00:17:21.053 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 67953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-24T00:17:21.053 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 98113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-24T00:17:21.054 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 11081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-24T00:17:21.054 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 17817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-24T00:17:21.054 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/x25/ 2024-03-24T00:17:21.054 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 131273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/x25/x25.ko 2024-03-24T00:17:21.054 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/ 2024-03-24T00:17:21.054 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_algo.ko 2024-03-24T00:17:21.054 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 20553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-24T00:17:21.054 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 105217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_user.ko 2024-03-24T00:17:21.054 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/ 2024-03-24T00:17:21.054 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/ 2024-03-24T00:17:21.054 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 15441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-24T00:17:21.054 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 642383 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 625850 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 7677 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 23498 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 9916 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 70732 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 147214 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 233213 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 210 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 72614 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 879 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 257955 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 312782 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-24T00:17:21.055 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/ 2024-03-24T00:17:21.056 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/share/ 2024-03-24T00:17:21.056 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/share/doc/ 2024-03-24T00:17:21.056 INFO:teuthology.orchestra.run.smithi053.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/share/doc/linux-image-6.8.0-g5ec3e6e88b2e/ 2024-03-24T00:17:21.056 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 171 2024-03-19 20:13 ./usr/share/doc/linux-image-6.8.0-g5ec3e6e88b2e/changelog.Debian.gz 2024-03-24T00:17:21.056 INFO:teuthology.orchestra.run.smithi053.stdout:-rw-r--r-- root/root 691 2024-03-19 20:13 ./usr/share/doc/linux-image-6.8.0-g5ec3e6e88b2e/copyright 2024-03-24T00:17:21.061 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:21.061 INFO:teuthology.task.kernel:searching for kernel 6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:21.061 DEBUG:teuthology.orchestra.run.smithi053:> egrep '(submenu|menuentry.*6.8.0-g5ec3e6e88b2e).*{' /boot/grub/grub.cfg 2024-03-24T00:17:21.068 INFO:teuthology.orchestra.run.smithi053.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-24T00:17:21.068 INFO:teuthology.orchestra.run.smithi053.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g5ec3e6e88b2e' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g5ec3e6e88b2e-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-24T00:17:21.068 INFO:teuthology.orchestra.run.smithi053.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g5ec3e6e88b2e (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g5ec3e6e88b2e-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-24T00:17:21.069 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-03-24T00:17:21.069 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:21.069 DEBUG:teuthology.orchestra.run.smithi053:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-03-24T00:17:21.114 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi053.front.sepia.ceph.com to complete... 2024-03-24T00:17:21.176 INFO:teuthology.orchestra.run.smithi040.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-24T00:17:21.177 INFO:teuthology.orchestra.run.smithi040.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-24T00:17:21.177 INFO:teuthology.orchestra.run.smithi040.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-24T00:17:21.220 INFO:teuthology.orchestra.run.smithi040.stderr:done 2024-03-24T00:17:21.406 DEBUG:teuthology.orchestra.run.smithi040:> dpkg-deb -c /tmp/linux-image.deb 2024-03-24T00:17:21.413 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./ 2024-03-24T00:17:21.414 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./boot/ 2024-03-24T00:17:21.414 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 4474370 2024-03-19 20:13 ./boot/System.map-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:21.424 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 170904 2024-03-19 20:13 ./boot/config-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:21.424 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11555840 2024-03-19 20:13 ./boot/vmlinuz-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/ 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/ 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/postinst.d/ 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/postrm.d/ 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/preinst.d/ 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./etc/kernel/prerm.d/ 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/ 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/ 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/ 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/ 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/ 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/ 2024-03-24T00:17:21.443 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ 2024-03-24T00:17:21.444 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 490449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/aesni-intel.ko 2024-03-24T00:17:21.446 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-24T00:17:21.447 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-24T00:17:21.447 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-24T00:17:21.447 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 93081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-24T00:17:21.447 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 106697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-24T00:17:21.447 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 152145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-24T00:17:21.447 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-24T00:17:21.447 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-24T00:17:21.447 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 83857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-24T00:17:21.447 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-24T00:17:21.447 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 58489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 50393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/ 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/ 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/ 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 41433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpuid.ko 2024-03-24T00:17:21.448 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/msr.ko 2024-03-24T00:17:21.449 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/ 2024-03-24T00:17:21.449 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 354265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-24T00:17:21.449 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 886177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-24T00:17:21.449 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 2294937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-24T00:17:21.522 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ 2024-03-24T00:17:21.522 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 53841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/af_alg.ko 2024-03-24T00:17:21.522 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/algif_hash.ko 2024-03-24T00:17:21.522 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/algif_skcipher.ko 2024-03-24T00:17:21.522 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/asymmetric_keys/ 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/ 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-24T00:17:21.523 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cryptd.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_simd.ko 2024-03-24T00:17:21.524 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lrw.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-24T00:17:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-24T00:17:21.526 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-24T00:17:21.526 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-24T00:17:21.526 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 143033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-24T00:17:21.526 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-24T00:17:21.526 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-24T00:17:21.526 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-24T00:17:21.526 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-24T00:17:21.526 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-24T00:17:21.526 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xts.ko 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_extlog.ko 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/apei/ 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/apei/einj.ko 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-24T00:17:21.527 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-24T00:17:21.528 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 104433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-24T00:17:21.528 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ 2024-03-24T00:17:21.528 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-24T00:17:21.528 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 89937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci.ko 2024-03-24T00:17:21.528 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-24T00:17:21.528 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 88313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci.ko 2024-03-24T00:17:21.528 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-24T00:17:21.528 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-24T00:17:21.528 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-24T00:17:21.528 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-24T00:17:21.528 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-24T00:17:21.528 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-24T00:17:21.529 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-24T00:17:21.529 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-24T00:17:21.529 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-24T00:17:21.529 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-24T00:17:21.529 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-24T00:17:21.529 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-24T00:17:21.529 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-24T00:17:21.529 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-24T00:17:21.529 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-24T00:17:21.529 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-24T00:17:21.529 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-24T00:17:21.529 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 52177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_legacy.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 37697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-24T00:17:21.530 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_radisys.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rz1000.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 83257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 70305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-24T00:17:21.531 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 37281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 44705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/ 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 124457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/ 2024-03-24T00:17:21.532 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/ 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 113833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/ 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 964817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 169705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/ 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 94217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 116337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/ 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 110329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 98193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 263641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/ 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 98121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/ 2024-03-24T00:17:21.533 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 58777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/applicom.ko 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/ 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 148521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-24T00:17:21.534 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 139361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 70873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 53417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/ 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 84025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/nvram.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/ 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-03-24T00:17:21.535 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/ 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/xillybus_class.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 50849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/xillybus_core.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/ 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-palmas.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-si5351.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-twl6040.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/clk/clk-wm831x.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/ 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-24T00:17:21.536 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 70473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 246841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/ 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/ 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/ 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 570497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/ 2024-03-24T00:17:21.537 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 70433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/ 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/ 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 122817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 80305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 41641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-24T00:17:21.538 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 56737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/ 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-gpio.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-max14577.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-max77693.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-max8997.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-palmas.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-rt8973a.ko 2024-03-24T00:17:21.539 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/extcon/extcon-sm5502.ko 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/ 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/dmi-sysfs.ko 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/efi/ 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/efi/efi-pstore.ko 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/ 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-adp5520.ko 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-amd8111.ko 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-arizona.ko 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-da9052.ko 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-da9055.ko 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-dln2.ko 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 77753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-f7188x.ko 2024-03-24T00:17:21.540 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-generic.ko 2024-03-24T00:17:21.541 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-24T00:17:21.541 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-03-24T00:17:21.541 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-kempld.ko 2024-03-24T00:17:21.541 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-lp3943.ko 2024-03-24T00:17:21.541 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-max7300.ko 2024-03-24T00:17:21.541 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-max730x.ko 2024-03-24T00:17:21.541 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-max732x.ko 2024-03-24T00:17:21.541 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-03-24T00:17:21.541 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-pca953x.ko 2024-03-24T00:17:21.541 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-pcf857x.ko 2024-03-24T00:17:21.541 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-rdc321x.ko 2024-03-24T00:17:21.542 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-sch.ko 2024-03-24T00:17:21.542 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-sch311x.ko 2024-03-24T00:17:21.542 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-tps65912.ko 2024-03-24T00:17:21.542 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-twl4030.ko 2024-03-24T00:17:21.542 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-twl6040.ko 2024-03-24T00:17:21.542 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-24T00:17:21.542 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-vx855.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-wm831x.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-wm8350.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-wm8994.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/ 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-a4tech.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-apple.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-belkin.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-cherry.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-chicony.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 60665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-cp2112.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-cypress.ko 2024-03-24T00:17:21.543 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ezkey.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-generic.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kensington.ko 2024-03-24T00:17:21.544 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lg-g15.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 57977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 101545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 80921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-magicmouse.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-microsoft.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-monterey.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 50281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ntrig.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-24T00:17:21.545 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 72865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-24T00:17:21.546 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-24T00:17:21.547 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-24T00:17:21.547 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-24T00:17:21.547 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-24T00:17:21.547 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-24T00:17:21.547 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-24T00:17:21.547 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-24T00:17:21.547 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-24T00:17:21.547 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-24T00:17:21.547 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-24T00:17:21.547 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-24T00:17:21.547 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-24T00:17:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-24T00:17:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-24T00:17:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-24T00:17:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-24T00:17:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-24T00:17:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-24T00:17:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-24T00:17:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 70993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-24T00:17:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-24T00:17:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 143393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-24T00:17:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-24T00:17:21.548 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-24T00:17:21.549 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-24T00:17:21.549 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 269233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid.ko 2024-03-24T00:17:21.549 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-24T00:17:21.549 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/ 2024-03-24T00:17:21.549 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 125641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/usbhid.ko 2024-03-24T00:17:21.549 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/usbkbd.ko 2024-03-24T00:17:21.549 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/usbhid/usbmouse.ko 2024-03-24T00:17:21.549 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 218881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-24T00:17:21.549 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/ 2024-03-24T00:17:21.549 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 64305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-24T00:17:21.549 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 90113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-24T00:17:21.549 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 304953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 66801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 61817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 97889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 45465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-24T00:17:21.550 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 74321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 69313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 55113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 58529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 40729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/da9052-hwmon.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/da9055-hwmon.ko 2024-03-24T00:17:21.551 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 99513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc2103.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 66897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 94601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 41313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 55361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-24T00:17:21.552 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 46713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 45441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hih6130.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 140137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/jc42.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-24T00:17:21.553 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 48513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 72745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 46065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 37825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 108753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-24T00:17:21.554 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-24T00:17:21.555 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-24T00:17:21.556 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-24T00:17:21.556 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mc13783-adc.ko 2024-03-24T00:17:21.556 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-24T00:17:21.556 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-03-24T00:17:21.556 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-24T00:17:21.556 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 119849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-24T00:17:21.556 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 61441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-24T00:17:21.556 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 59649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-24T00:17:21.556 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 82721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-24T00:17:21.556 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 54345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-24T00:17:21.556 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-24T00:17:21.556 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ 2024-03-24T00:17:21.557 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-24T00:17:21.557 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-24T00:17:21.557 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-24T00:17:21.557 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-24T00:17:21.557 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-24T00:17:21.557 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-24T00:17:21.557 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-24T00:17:21.557 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 82481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-24T00:17:21.557 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-24T00:17:21.557 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-24T00:17:21.557 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-24T00:17:21.557 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 40489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 40361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 37289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-24T00:17:21.558 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 43449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 52369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 53209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 58265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 79841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 71257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 63137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 71233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 101385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 119417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-24T00:17:21.559 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/ 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/ 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/ 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-24T00:17:21.560 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 6833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 50305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 67689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-24T00:17:21.561 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 50977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-24T00:17:21.562 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/ 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 273529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 883945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-24T00:17:21.563 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 69185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 324337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 97841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 282345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 66057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/ 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/ 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 493185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/ 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 420593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/ 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 741737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 296881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/ocrdma/ 2024-03-24T00:17:21.564 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 153729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/usnic/ 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 169145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 266577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 100281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 111337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 128665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 129801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-24T00:17:21.565 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/ 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/evbug.ko 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/ff-memless.ko 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/ 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 41457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 41713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/ 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-03-24T00:17:21.566 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 37281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/lkkbd.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/lm8323.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/lm8333.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/newtonkbd.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/qt1070.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/qt2160.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-03-24T00:17:21.567 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/stowaway.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/sunkbd.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/xtkbd.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/matrix-keymap.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/ 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 123201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 89393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/gpio_mouse.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 335921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/psmouse.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-24T00:17:21.568 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/ 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 109601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/ 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/ct82c710.ko 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/parkbd.ko 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/pcips2.ko 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/ps2mult.ko 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-24T00:17:21.569 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serport.ko 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/ 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-88pm860x.ko 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-adp5520.ko 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 59657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-bd2802.ko 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-da903x.ko 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-da9052.ko 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-gpio.ko 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3533.ko 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm355x.ko 2024-03-24T00:17:21.570 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3642.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp8788.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-max8997.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-mc13783.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-menf21bmc.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-pca9532.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-pca955x.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-pca963x.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-tca6507.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-wm831x-status.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-wm8350.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-24T00:17:21.571 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/ 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/ 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 647121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 40489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 93321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bufio.ko 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-24T00:17:21.572 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 155825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 107937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 53769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-era.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 41441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-mirror.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 77881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 70985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-region-hash.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 109457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-snapshot.ko 2024-03-24T00:17:21.573 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 181777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 53737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-zero.ko 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/ 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 199881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 37321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 100841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 126369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 368497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/ 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/ 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 178049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-24T00:17:21.574 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 63241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 46249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 37121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 123233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 80921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 52865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/ 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/88pm800.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/88pm805.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/88pm80x.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/arizona-i2c.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 123649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/arizona.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/bcm590xx.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/cros_ec_dev.ko 2024-03-24T00:17:21.575 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/dln2.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/janz-cmodio.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 53489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/kempld-core.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lm3533-core.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lp3943.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 56969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/max8907.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mc13xxx-core.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/menf21bmc.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/pcf50633-adc.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/pcf50633-gpio.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/pcf50633.ko 2024-03-24T00:17:21.576 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/retu-mfd.ko 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 80345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/si476x-core.ko 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 48369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/tps65010.ko 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/tps6507x.ko 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 44265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wm8994.ko 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 48393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ad525x_dpot.ko 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/ 2024-03-24T00:17:21.577 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 46513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 45497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/c2port/ 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/c2port/core.ko 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/ 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ds1682.ko 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/dummy-irq.ko 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/echo/ 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/echo/echo.ko 2024-03-24T00:17:21.578 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/ 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/genwqe/ 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 152073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hmc6352.ko 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ibmasm/ 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 72561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ics932s401.ko 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-24T00:17:21.579 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/ 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/ 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 126153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 56377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 313601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/phantom.ko 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ti-st/ 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 74305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/ti-st/st_drv.ko 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 48345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-24T00:17:21.580 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/ 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 183569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/ 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 491609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/ 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/ 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 105937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 40225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/ 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 459001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-24T00:17:21.581 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/ 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 45961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 130665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 68785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 64617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 101617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/ 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 54625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/ 2024-03-24T00:17:21.582 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 69169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/ 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 59993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/ 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 41409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/ 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 54617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 87457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/ 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/ 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 97201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 102497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-03-24T00:17:21.583 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 90209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/ 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 81321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 69545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/ 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 78385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 176697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1434505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 128369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/ 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 131281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 314329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-24T00:17:21.584 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/ 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/ 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 323737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/ 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 193593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 347217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1045449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 166145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/ 2024-03-24T00:17:21.585 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/ 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 204249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/ 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/ 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 69433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 61425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 141201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 62833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/ 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 60673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 62833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-24T00:17:21.586 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 45489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ec_bhf.ko 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/ 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/ 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 281217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fujitsu/ 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 45553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 90305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/ 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 295697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/ 2024-03-24T00:17:21.587 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 584353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-24T00:17:21.588 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/ 2024-03-24T00:17:21.588 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 246041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-24T00:17:21.588 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/ 2024-03-24T00:17:21.588 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 944545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-24T00:17:21.589 INFO:teuthology.orchestra.run.smithi053.stderr:Sourcing file `/etc/default/grub' 2024-03-24T00:17:21.589 INFO:teuthology.orchestra.run.smithi053.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-24T00:17:21.589 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2024-03-24T00:17:21.664 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/ 2024-03-24T00:17:21.664 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 288529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/ 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 513673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/ 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 93489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 735169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 151745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 86993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/ 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 102441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 122609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-24T00:17:21.665 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/ 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 727481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 278881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1122297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 52985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 40881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 95577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/ 2024-03-24T00:17:21.666 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 108409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 73617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/ 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 151097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/ 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 136801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/ 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 50433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/ 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-03-24T00:17:21.667 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 227865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 87017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 672057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/ 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/ 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 67281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 83809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 194073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/samsung/ 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-03-24T00:17:21.668 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 104049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/ 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1067001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/ 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/ 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 54729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 63201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/ 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 61425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 67297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 73257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 68049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/ 2024-03-24T00:17:21.669 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 495777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/ 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 103185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 166049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 66273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 78721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/ 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 50017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/ 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 92345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-24T00:17:21.670 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/ 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 79937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 83721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/ 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 46313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/ 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 59921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/ 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 41185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/defxx.ko 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/skfp/ 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 217113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/fddi/skfp/skfp.ko 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 74993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/ 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 237265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-24T00:17:21.671 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/ 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 85857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 62897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/ 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-gpio.ko 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-24T00:17:21.672 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 45057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/ 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 44633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 44825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-24T00:17:21.673 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 81961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 76569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 99009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/plip/ 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 43769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/plip/plip.ko 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ 2024-03-24T00:17:21.674 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sb1000.ko 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/ 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 64561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/ 2024-03-24T00:17:21.675 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 107233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-24T00:17:21.676 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-24T00:17:21.676 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-24T00:17:21.676 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-24T00:17:21.676 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-24T00:17:21.676 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-24T00:17:21.676 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 86825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-24T00:17:21.676 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/ 2024-03-24T00:17:21.676 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 144361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-24T00:17:21.676 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/ 2024-03-24T00:17:21.676 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 307193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-24T00:17:21.676 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/ 2024-03-24T00:17:21.676 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 144017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/ 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/ 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 267745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 103361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/ 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 131329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 85217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/ 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/ 2024-03-24T00:17:21.677 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pci-stub.ko 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 46369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/ 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 146153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pcmcia.ko 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 62081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pcmcia_core.ko 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 112793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/ 2024-03-24T00:17:21.678 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/broadcom/ 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/samsung/ 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/ 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/ 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 69033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/ 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/ 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec.ko 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-03-24T00:17:21.679 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/ 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 73601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 59569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 131945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 40169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-24T00:17:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/ibm_rtl.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/ 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 44537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-24T00:17:21.681 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 48481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/ 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/88pm860x_battery.ko 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/88pm860x_charger.ko 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 54961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/bq2415x_charger.ko 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 59505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/bq24190_charger.ko 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/bq24735-charger.ko 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/da9030_battery.ko 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/da9052-battery.ko 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2760_battery.ko 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2780_battery.ko 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2781_battery.ko 2024-03-24T00:17:21.682 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/ds2782_battery.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/gpio-charger.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/isp1704_charger.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/lp8727_charger.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max14577_charger.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max17042_battery.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max8903_charger.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/max8925_power.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/pcf50633-charger.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 37841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/sbs-battery.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/test_power.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/tps65090-charger.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/wm831x_backup.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/wm831x_power.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/power/supply/wm8350_power.ko 2024-03-24T00:17:21.683 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/ 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 58105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/ 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/ 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 41801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 70873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/ 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-88pm80x.ko 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-88pm860x.ko 2024-03-24T00:17:21.684 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-da9052.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-da9055.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-da9063.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 68481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-24T00:17:21.685 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-lp8788.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t86.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8907.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8925.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8997.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max8998.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-24T00:17:21.686 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-palmas.ko 2024-03-24T00:17:21.687 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 46537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-24T00:17:21.687 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf50633.ko 2024-03-24T00:17:21.687 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-24T00:17:21.687 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-24T00:17:21.687 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-24T00:17:21.687 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-24T00:17:21.687 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rc5t583.ko 2024-03-24T00:17:21.687 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-24T00:17:21.687 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-24T00:17:21.687 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-24T00:17:21.687 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-24T00:17:21.688 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-24T00:17:21.688 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-s35390a.ko 2024-03-24T00:17:21.688 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-24T00:17:21.688 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-tps6586x.ko 2024-03-24T00:17:21.688 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-tps65910.ko 2024-03-24T00:17:21.688 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-wm831x.ko 2024-03-24T00:17:21.688 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-wm8350.ko 2024-03-24T00:17:21.688 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-24T00:17:21.688 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 77529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 57721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 55017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 135345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/ 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 271441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 100473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/ 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 356865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 295929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic94xx/ 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 189145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-03-24T00:17:21.689 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/ 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 108209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/ 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 246937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/ 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1171953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/ 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 238401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/ 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 115329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 40785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-24T00:17:21.690 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/ 2024-03-24T00:17:21.691 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 308961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-24T00:17:21.691 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/ 2024-03-24T00:17:21.691 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-03-24T00:17:21.691 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 76041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-24T00:17:21.691 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-03-24T00:17:21.691 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 101425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-24T00:17:21.691 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 146753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-24T00:17:21.691 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 75977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-24T00:17:21.691 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-24T00:17:21.691 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/ 2024-03-24T00:17:21.691 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 212385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 60505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/ 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 78225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 157729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fdomain.ko 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/ 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 294953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 201729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 48945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/imm.ko 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 43977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 265945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-24T00:17:21.692 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 101881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/ 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 317193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 52865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/ 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 315321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 145497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 53809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/ 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 216401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/ 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 2194089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/ 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 73473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-24T00:17:21.693 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 310689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 85521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/ 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 659633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/ 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 119489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 78041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/ 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 81561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/ 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 366337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-24T00:17:21.694 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 124601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ppa.ko 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 55337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/ 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1876753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 85193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/ 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 558953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qlogicfas408.ko 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 261081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 173929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 280113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 97625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-24T00:17:21.695 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 73865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-24T00:17:21.696 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 44889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-24T00:17:21.696 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-24T00:17:21.696 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 115649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-24T00:17:21.696 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 46737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-24T00:17:21.696 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/ 2024-03-24T00:17:21.696 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 164889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-24T00:17:21.696 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 44857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-24T00:17:21.696 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 50233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-24T00:17:21.696 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-24T00:17:21.696 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/xen-scsifront.ko 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 129393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/ 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/ 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 736761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/ 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 931521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 44153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 122489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-24T00:17:21.697 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/ 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 75521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/ 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/ 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/ 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-24T00:17:21.698 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 78353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 57777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/moxa.ko 2024-03-24T00:17:21.699 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 50865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/mxser.ko 2024-03-24T00:17:21.700 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 108489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-24T00:17:21.700 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-24T00:17:21.700 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 67841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-24T00:17:21.700 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/ 2024-03-24T00:17:21.700 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/ 2024-03-24T00:17:21.700 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-03-24T00:17:21.700 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 59401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-24T00:17:21.700 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-03-24T00:17:21.700 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/altera_uart.ko 2024-03-24T00:17:21.700 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-24T00:17:21.700 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 72553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-03-24T00:17:21.700 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/ 2024-03-24T00:17:21.701 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 101993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-24T00:17:21.701 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/rp2.ko 2024-03-24T00:17:21.701 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 53177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/sc16is7xx.ko 2024-03-24T00:17:21.701 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/ 2024-03-24T00:17:21.701 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ 2024-03-24T00:17:21.701 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 545401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-24T00:17:21.701 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ 2024-03-24T00:17:21.701 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-24T00:17:21.701 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-03-24T00:17:21.701 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/ 2024-03-24T00:17:21.701 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 44473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-24T00:17:21.701 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-24T00:17:21.702 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-24T00:17:21.702 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_dmem_genirq.ko 2024-03-24T00:17:21.702 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_mf624.ko 2024-03-24T00:17:21.702 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_netx.ko 2024-03-24T00:17:21.702 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-24T00:17:21.702 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-03-24T00:17:21.702 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-24T00:17:21.702 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/ 2024-03-24T00:17:21.702 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/c67x00/ 2024-03-24T00:17:21.702 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 73225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/c67x00/c67x00.ko 2024-03-24T00:17:21.702 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ 2024-03-24T00:17:21.702 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 145793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/ 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 87009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 89769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/common/ 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/common/ulpi.ko 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc2/ 2024-03-24T00:17:21.703 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-03-24T00:17:21.704 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/ 2024-03-24T00:17:21.704 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-03-24T00:17:21.704 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-03-24T00:17:21.704 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 452825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/dwc3/dwc3.ko 2024-03-24T00:17:21.704 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/ 2024-03-24T00:17:21.704 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/ 2024-03-24T00:17:21.704 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 60329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/u_ether.ko 2024-03-24T00:17:21.704 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/u_serial.ko 2024-03-24T00:17:21.704 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-03-24T00:17:21.704 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 92409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 50945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 122257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 59865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 57633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 85521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 58025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 67353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-03-24T00:17:21.705 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/ 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 62369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-03-24T00:17:21.706 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 161233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/libcomposite.ko 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/ 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/bdc/ 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 157617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 52929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 69617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 55953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 75969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/net2272.ko 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 74985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/net2280.ko 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 73017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-03-24T00:17:21.707 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 65881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 66001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 81409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 156337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/ 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/bcma-hcd.ko 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 80017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/isp116x-hcd.ko 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 87193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 78777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811_cs.ko 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/ssb-hcd.ko 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-03-24T00:17:21.708 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/ 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 41633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/cytherm.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ehset.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-24T00:17:21.709 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/lvstest.ko 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/ 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 64745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 70721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbtest.ko 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 41817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/mon/ 2024-03-24T00:17:21.710 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 72825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/mon/usbmon.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/musb/ 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 301633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/musb/musb_hdrc.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/ 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-isp1301.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-tahvo.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 70281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 58121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-24T00:17:21.711 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-24T00:17:21.712 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-24T00:17:21.712 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 44185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/f81232.ko 2024-03-24T00:17:21.712 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 175105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-24T00:17:21.712 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 53017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-24T00:17:21.712 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 136985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-24T00:17:21.712 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 124745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-24T00:17:21.712 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 67681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-24T00:17:21.712 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-24T00:17:21.712 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-24T00:17:21.712 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 58137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-24T00:17:21.712 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 107433 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-24T00:17:21.712 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/metro-usb.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 66361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 78129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 56249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mxuport.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 191825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 54265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-24T00:17:21.713 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 61393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 125529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usbserial.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-24T00:17:21.714 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 37313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/wishbone-serial.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 52761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-24T00:17:21.715 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 180329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/ 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 67401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 71025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 104065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/ 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/ 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 162865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 103673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 76441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-24T00:17:21.716 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/ 2024-03-24T00:17:21.717 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 113665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-24T00:17:21.717 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-24T00:17:21.717 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 58001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-24T00:17:21.717 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 89057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-24T00:17:21.717 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/ 2024-03-24T00:17:21.717 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/ 2024-03-24T00:17:21.717 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-03-24T00:17:21.717 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/aat2870_bl.ko 2024-03-24T00:17:21.717 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/adp5520_bl.ko 2024-03-24T00:17:21.717 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/adp8860_bl.ko 2024-03-24T00:17:21.717 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/adp8870_bl.ko 2024-03-24T00:17:21.717 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/as3711_bl.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/bd6107.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/da903x_bl.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/da9052_bl.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/gpio_backlight.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/kb3886_bl.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lm3533_bl.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lm3639_bl.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lv5207lp.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/max8925_bl.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/pandora_bl.ko 2024-03-24T00:17:21.718 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-03-24T00:17:21.719 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-24T00:17:21.719 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/wm831x_bl.ko 2024-03-24T00:17:21.719 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/ 2024-03-24T00:17:21.719 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ 2024-03-24T00:17:21.719 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-24T00:17:21.719 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-24T00:17:21.719 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/matrox_w1.ko 2024-03-24T00:17:21.719 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/w1-gpio.ko 2024-03-24T00:17:21.719 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/ 2024-03-24T00:17:21.719 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-24T00:17:21.719 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-24T00:17:21.719 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 56337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 103049 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/acquirewdt.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/advantechwdt.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-24T00:17:21.720 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/cpu5wdt.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/da9052_wdt.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/da9055_wdt.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/da9063_wdt.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/dw_wdt.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/eurotechwdt.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-24T00:17:21.721 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21393 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/kempld_wdt.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/mena21_wdt.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pc87413_wdt.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/retu_wdt.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-24T00:17:21.722 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sc1200wdt.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21481 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/twl4030_wdt.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wafer5823wdt.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wm831x_wdt.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13689 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wm8350_wdt.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15609 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-24T00:17:21.723 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/ 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/ 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 176385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 40097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 71153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-scsiback.ko 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/ 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/ 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 158281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/adfs/ 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 61721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/adfs/adfs.ko 2024-03-24T00:17:21.724 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/ 2024-03-24T00:17:21.725 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 163417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-24T00:17:21.725 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/afs/ 2024-03-24T00:17:21.725 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1019225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/afs/kafs.ko 2024-03-24T00:17:21.725 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/autofs/ 2024-03-24T00:17:21.725 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 103225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/autofs/autofs4.ko 2024-03-24T00:17:21.725 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/ 2024-03-24T00:17:21.725 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 64625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-24T00:17:21.725 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/bfs/ 2024-03-24T00:17:21.725 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 41017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/bfs/bfs.ko 2024-03-24T00:17:21.725 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-24T00:17:21.725 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/ 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 3487553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/ 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 305897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1335721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/ 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 97769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/configfs/ 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 108577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/configfs/configfs.ko 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/ 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/ 2024-03-24T00:17:21.726 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 644249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-24T00:17:21.727 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/efs/ 2024-03-24T00:17:21.727 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/efs/efs.ko 2024-03-24T00:17:21.727 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/ 2024-03-24T00:17:21.727 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1527665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-24T00:17:21.727 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/ 2024-03-24T00:17:21.727 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-24T00:17:21.727 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/freevxfs/ 2024-03-24T00:17:21.727 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/freevxfs/freevxfs.ko 2024-03-24T00:17:21.727 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/ 2024-03-24T00:17:21.727 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-24T00:17:21.727 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/ 2024-03-24T00:17:21.727 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1231361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/ 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 141713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/ 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 223281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hpfs/ 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 193545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hpfs/hpfs.ko 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/ 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 74737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/ 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 420449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/ 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 281705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-24T00:17:21.728 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/ 2024-03-24T00:17:21.729 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 88937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-24T00:17:21.729 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/ 2024-03-24T00:17:21.729 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 490505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-24T00:17:21.729 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/ 2024-03-24T00:17:21.729 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/ 2024-03-24T00:17:21.729 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 81457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-24T00:17:21.729 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/ 2024-03-24T00:17:21.729 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 59353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-24T00:17:21.729 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/ 2024-03-24T00:17:21.729 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 117377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-24T00:17:21.729 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1058377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 66361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv2.ko 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 120193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 2041825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/ 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/ 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1511665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/ 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 582529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/ 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-24T00:17:21.730 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ascii.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-24T00:17:21.731 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 94041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 142577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-24T00:17:21.732 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 165649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 118025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-24T00:17:21.733 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ntfs/ 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 220545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ntfs/ntfs.ko 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 244673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 440681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ 2024-03-24T00:17:21.734 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 54177 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 2590849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/omfs/ 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 53705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/omfs/omfs.ko 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/ 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 360953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 56945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx4/ 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx4/qnx4.ko 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx6/ 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 46625 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/qnx6/qnx6.ko 2024-03-24T00:17:21.735 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/ 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/quota_tree.ko 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/quota_v1.ko 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/quota/quota_v2.ko 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/ 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 532289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/ 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/ 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/ 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 2712601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/ 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-24T00:17:21.736 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-24T00:17:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/ 2024-03-24T00:17:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 134809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-24T00:17:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/ 2024-03-24T00:17:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 91329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-24T00:17:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/ 2024-03-24T00:17:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 248409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-24T00:17:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ 2024-03-24T00:17:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 162785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-24T00:17:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/ 2024-03-24T00:17:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 4036193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-24T00:17:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/ 2024-03-24T00:17:21.737 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/locking/ 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 80489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/locking/locktorture.ko 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/time/ 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/time/test_udelay.ko 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 60265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 6737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 6441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc7.ko 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 6937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/ 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/interval_tree_test.ko 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-24T00:17:21.738 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/ 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/ 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 6529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/memory-notifier-error-inject.ko 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/notifier-error-inject.ko 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/percpu_test.ko 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/pm-notifier-error-inject.ko 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/ 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 143337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/rbtree_test.ko 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/ 2024-03-24T00:17:21.739 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-24T00:17:21.740 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test-kstrtox.ko 2024-03-24T00:17:21.740 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test-string_helpers.ko 2024-03-24T00:17:21.740 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 4863225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_bpf.ko 2024-03-24T00:17:21.774 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 48993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_firmware.ko 2024-03-24T00:17:21.774 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 6281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_module.ko 2024-03-24T00:17:21.774 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/test_user_copy.ko 2024-03-24T00:17:21.774 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-24T00:17:21.774 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/xz/ 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/xz/xz_dec_test.ko 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/ 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18129 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/ 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 55777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 6961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 6969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 6961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 6897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 6969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 6905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-24T00:17:21.775 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/ 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/ 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 82897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/ 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 198833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-24T00:17:21.776 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/ 2024-03-24T00:17:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 667545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-24T00:17:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/ 2024-03-24T00:17:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 67769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-24T00:17:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 790673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-24T00:17:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ 2024-03-24T00:17:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-24T00:17:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-24T00:17:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-24T00:17:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-24T00:17:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-24T00:17:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-24T00:17:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-24T00:17:21.778 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 55945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/ 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1046825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/ 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 125705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/ 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 194505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 58745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 61401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/ 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 390089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-24T00:17:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12601 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/hsr/ 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 102073 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/hsr/hsr.ko 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 45505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 284721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 47881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 48249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-24T00:17:21.780 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 51217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 61137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 61513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 29081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ip_tables.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9585 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-24T00:17:21.781 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14785 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-24T00:17:21.782 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15969 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-24T00:17:21.783 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10857 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 45057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 79929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_gre.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 94169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 54273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-24T00:17:21.784 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/mip6.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-24T00:17:21.785 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15025 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8913 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 74881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-24T00:17:21.786 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/ 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 84401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/ 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 125209 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 25817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36497 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 37377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 53233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/lapb/ 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 48121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/lapb/lapb.ko 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/ 2024-03-24T00:17:21.787 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 268521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc2.ko 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/ 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 309417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/ 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 119705 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28617 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 99993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 100281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 102553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-24T00:17:21.788 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 102345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-24T00:17:21.789 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 111673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-24T00:17:21.789 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 59193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-24T00:17:21.789 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 108345 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-24T00:17:21.789 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 112145 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-24T00:17:21.789 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 112409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-24T00:17:21.789 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 110337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-24T00:17:21.789 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 113289 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-24T00:17:21.789 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-24T00:17:21.789 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ 2024-03-24T00:17:21.789 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 412513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-24T00:17:21.789 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11993 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10641 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9265 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13545 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7721 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-24T00:17:21.790 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 411769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15881 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 118161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 108313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 66537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 92233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-24T00:17:21.791 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-24T00:17:21.792 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-24T00:17:21.792 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 96521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-24T00:17:21.792 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-24T00:17:21.792 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12529 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-24T00:17:21.792 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-24T00:17:21.792 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-24T00:17:21.792 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-24T00:17:21.792 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 44137 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-24T00:17:21.792 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 623473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-24T00:17:21.792 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 42513 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-24T00:17:21.792 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30161 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 22937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49649 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 40809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9505 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20033 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-24T00:17:21.793 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14329 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11753 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_osf.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9985 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12841 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_synproxy.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tproxy.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 89233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/x_tables.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-24T00:17:21.794 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7633 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 18081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13281 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 38777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13801 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11561 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-24T00:17:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-24T00:17:21.796 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-24T00:17:21.796 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-24T00:17:21.796 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-24T00:17:21.796 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-24T00:17:21.796 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8233 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-24T00:17:21.796 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-24T00:17:21.796 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-24T00:17:21.796 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-24T00:17:21.796 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-24T00:17:21.796 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-24T00:17:21.796 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-24T00:17:21.796 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11257 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12305 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10889 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7769 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10569 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 43201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11825 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-24T00:17:21.797 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11697 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10121 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11729 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13737 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10017 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11897 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-24T00:17:21.798 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9385 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8169 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36361 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13241 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30001 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 16817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9417 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8713 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 8457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15681 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpudp.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9313 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/ 2024-03-24T00:17:21.799 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 14441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/ 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 12337 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/ 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 384201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10369 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11833 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/ 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13473 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/phonet/ 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 112553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/phonet/phonet.ko 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 55057 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/phonet/pn_pep.ko 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/ 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 253961 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-24T00:17:21.800 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 165105 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-24T00:17:21.801 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 68097 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-24T00:17:21.801 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rxrpc/ 2024-03-24T00:17:21.801 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 828297 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rxrpc/rxrpc.ko 2024-03-24T00:17:21.801 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/ 2024-03-24T00:17:21.801 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-24T00:17:21.801 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23489 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-24T00:17:21.801 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 37089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-24T00:17:21.801 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23377 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-24T00:17:21.801 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 43353 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20193 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35201 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36577 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33873 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 19185 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_cgroup.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 39521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 62657 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7465 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-24T00:17:21.802 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 13321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 33425 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7921 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10929 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21673 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20865 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 34593 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31521 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq_codel.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 35457 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 49849 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 21977 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 66449 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-24T00:17:21.803 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 30225 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-24T00:17:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10809 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-24T00:17:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26089 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-24T00:17:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 36153 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-24T00:17:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 27409 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-24T00:17:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10665 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-24T00:17:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 24777 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-24T00:17:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 48537 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-24T00:17:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 32817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-24T00:17:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 26793 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-24T00:17:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 31321 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-24T00:17:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-24T00:17:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 28273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-24T00:17:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/ 2024-03-24T00:17:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 932945 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-24T00:17:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-24T00:17:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/ 2024-03-24T00:17:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/ 2024-03-24T00:17:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 337009 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-24T00:17:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 55041 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-24T00:17:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 1513905 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-24T00:17:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/ 2024-03-24T00:17:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 704761 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-24T00:17:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/ 2024-03-24T00:17:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 10401 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 867249 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/ 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15937 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/ 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 87745 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 67953 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 98113 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 11081 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 17817 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/x25/ 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 131273 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/x25/x25.ko 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/ 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23065 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_algo.ko 2024-03-24T00:17:21.806 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 20553 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 105217 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_user.ko 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/ 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/ 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 15441 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 642383 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 625850 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 7677 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 23498 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 9916 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 70732 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 147214 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 233213 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 210 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-24T00:17:21.807 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 72614 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-24T00:17:21.808 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 879 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-24T00:17:21.808 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 257955 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-24T00:17:21.808 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 312782 2024-03-19 20:13 ./lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-24T00:17:21.808 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/ 2024-03-24T00:17:21.808 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/share/ 2024-03-24T00:17:21.808 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/share/doc/ 2024-03-24T00:17:21.808 INFO:teuthology.orchestra.run.smithi040.stdout:drwxr-xr-x root/root 0 2024-03-19 20:13 ./usr/share/doc/linux-image-6.8.0-g5ec3e6e88b2e/ 2024-03-24T00:17:21.808 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 171 2024-03-19 20:13 ./usr/share/doc/linux-image-6.8.0-g5ec3e6e88b2e/changelog.Debian.gz 2024-03-24T00:17:21.808 INFO:teuthology.orchestra.run.smithi040.stdout:-rw-r--r-- root/root 691 2024-03-19 20:13 ./usr/share/doc/linux-image-6.8.0-g5ec3e6e88b2e/copyright 2024-03-24T00:17:21.813 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:21.813 INFO:teuthology.task.kernel:searching for kernel 6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:21.813 DEBUG:teuthology.orchestra.run.smithi040:> egrep '(submenu|menuentry.*6.8.0-g5ec3e6e88b2e).*{' /boot/grub/grub.cfg 2024-03-24T00:17:21.814 INFO:teuthology.orchestra.run.smithi053.stderr:Found linux image: /boot/vmlinuz-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:21.814 INFO:teuthology.orchestra.run.smithi053.stderr:Found initrd image: /boot/initrd.img-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:21.820 INFO:teuthology.orchestra.run.smithi040.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-24T00:17:21.820 INFO:teuthology.orchestra.run.smithi040.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g5ec3e6e88b2e' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g5ec3e6e88b2e-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-24T00:17:21.820 INFO:teuthology.orchestra.run.smithi040.stdout: menuentry 'Ubuntu, with Linux 6.8.0-g5ec3e6e88b2e (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.8.0-g5ec3e6e88b2e-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-03-24T00:17:21.824 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-03-24T00:17:21.824 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:21.824 DEBUG:teuthology.orchestra.run.smithi040:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-03-24T00:17:21.862 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi040.front.sepia.ceph.com to complete... 2024-03-24T00:17:22.060 INFO:teuthology.orchestra.run.smithi053.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-24T00:17:22.066 INFO:teuthology.orchestra.run.smithi053.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-24T00:17:22.106 INFO:teuthology.orchestra.run.smithi053.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-24T00:17:22.111 INFO:teuthology.orchestra.run.smithi053.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-24T00:17:22.329 INFO:teuthology.orchestra.run.smithi053.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-24T00:17:22.329 INFO:teuthology.orchestra.run.smithi053.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-24T00:17:22.329 INFO:teuthology.orchestra.run.smithi053.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-24T00:17:22.344 INFO:teuthology.orchestra.run.smithi040.stderr:Sourcing file `/etc/default/grub' 2024-03-24T00:17:22.346 INFO:teuthology.orchestra.run.smithi040.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-03-24T00:17:22.349 INFO:teuthology.orchestra.run.smithi053.stderr:done 2024-03-24T00:17:22.356 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2024-03-24T00:17:22.614 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:22.621 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-6.8.0-g5ec3e6e88b2e 2024-03-24T00:17:22.919 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-03-24T00:17:22.924 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-03-24T00:17:22.965 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-03-24T00:17:22.970 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-03-24T00:17:23.129 INFO:teuthology.orchestra.run.smithi040.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-03-24T00:17:23.129 INFO:teuthology.orchestra.run.smithi040.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-03-24T00:17:23.129 INFO:teuthology.orchestra.run.smithi040.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-03-24T00:17:23.139 INFO:teuthology.orchestra.run.smithi040.stderr:done 2024-03-24T00:17:53.506 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T00:17:53.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-24T00:17:53.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:17:54.274 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T00:17:54.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-24T00:17:54.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:18:53.567 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T00:18:54.277 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T00:19:02.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-24T00:19:02.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:19:02.891 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-03-24T00:19:03.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-24T00:19:03.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:19:03.640 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-24T00:19:03.640 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.8.0-g5ec3e6e88b2e"... 2024-03-24T00:19:03.641 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-03-24T00:19:03.691 INFO:teuthology.orchestra.run.smithi053.stdout:6.8.0-g5ec3e6e88b2e 2024-03-24T00:19:03.691 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-24T00:19:03.691 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T00:19:03.691 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-24T00:19:04.692 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T00:19:04.693 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T00:19:04.782 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2024-03-24T00:19:04.840 DEBUG:teuthology.parallel:result is None 2024-03-24T00:19:10.495 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-03-24T00:19:22.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-24T00:19:22.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:19:22.794 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-03-24T00:19:23.497 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-24T00:19:23.498 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.8.0-g5ec3e6e88b2e"... 2024-03-24T00:19:23.498 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-03-24T00:19:23.546 INFO:teuthology.orchestra.run.smithi040.stdout:6.8.0-g5ec3e6e88b2e 2024-03-24T00:19:23.547 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-24T00:19:23.547 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T00:19:23.547 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-03-24T00:19:24.548 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T00:19:24.548 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T00:19:24.644 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2024-03-24T00:19:24.682 DEBUG:teuthology.parallel:result is None 2024-03-24T00:19:24.682 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T00:19:24.689 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T00:19:24.689 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T00:19:24.693 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T00:19:24.702 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T00:19:24.722 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T00:19:24.758 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T00:19:24.758 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T00:19:24.762 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T00:19:24.785 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T00:19:24.806 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T00:19:24.807 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T00:19:24.825 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T00:19:24.852 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:19:24.856 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:19:24.866 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:19:24.871 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T00:19:24.873 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T00:19:24.880 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T00:19:24.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T00:19:24.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T00:19:24.935 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T00:19:24.945 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T00:19:24.945 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T00:19:24.978 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T00:19:24.987 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:19:25.035 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:19:25.087 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:19:25.088 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T00:19:25.161 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T00:19:25.170 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T00:19:25.219 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:19:25.219 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T00:19:25.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-03-24T00:19:25.299 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-03-24T00:19:25.441 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T00:19:25.448 INFO:teuthology.task.internal:Starting timer... 2024-03-24T00:19:25.448 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T00:19:25.495 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T00:19:25.527 DEBUG:teuthology.task.selinux:Excluding smithi040: OS 'ubuntu' does not support SELinux 2024-03-24T00:19:25.527 DEBUG:teuthology.task.selinux:Excluding smithi053: OS 'ubuntu' does not support SELinux 2024-03-24T00:19:25.527 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T00:19:25.527 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-03-24T00:19:25.528 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T00:19:25.528 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T00:19:25.559 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T00:19:25.577 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T00:19:25.578 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com,smithi053.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T00:27:11.986 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi053.front.sepia.ceph.com')] 2024-03-24T00:27:11.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-24T00:27:11.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:27:12.071 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-03-24T00:27:12.201 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-03-24T00:27:12.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-24T00:27:12.202 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:27:12.283 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-03-24T00:27:12.393 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-03-24T00:27:12.393 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T00:27:12.403 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T00:27:12.404 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:27:12.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:27:12.407 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:27:12.407 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:27:12.445 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-24T00:27:12.445 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: Command line: ntpd -gq 2024-03-24T00:27:12.445 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: ---------------------------------------------------- 2024-03-24T00:27:12.445 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: ntp-4 is maintained by Network Time Foundation, 2024-03-24T00:27:12.445 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-24T00:27:12.445 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: corporation. Support and training for ntp-4 are 2024-03-24T00:27:12.445 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: available at https://www.nwtime.org/support 2024-03-24T00:27:12.445 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: ---------------------------------------------------- 2024-03-24T00:27:12.446 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: proto: precision = 0.042 usec (-24) 2024-03-24T00:27:12.446 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: basedate set to 2022-02-04 2024-03-24T00:27:12.446 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: gps base set to 2022-02-06 (week 2196) 2024-03-24T00:27:12.446 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-24T00:27:12.446 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: restrict ::: KOD does nothing without LIMITED. 2024-03-24T00:27:12.446 INFO:teuthology.orchestra.run.smithi040.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-24T00:27:12.447 INFO:teuthology.orchestra.run.smithi040.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-24T00:27:12.448 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: Listen and drop on 0 v6wildcard [::]:123 2024-03-24T00:27:12.448 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-24T00:27:12.448 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: Listen normally on 2 lo 127.0.0.1:123 2024-03-24T00:27:12.448 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: Listen normally on 3 ens1f0 172.21.15.40:123 2024-03-24T00:27:12.448 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: Listen normally on 4 lo [::1]:123 2024-03-24T00:27:12.448 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:12cc%4]:123 2024-03-24T00:27:12.448 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:12 ntpd[17083]: Listening on routing socket on fd #22 for interface updates 2024-03-24T00:27:12.475 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-03-24T00:27:12.475 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: Command line: ntpd -gq 2024-03-24T00:27:12.475 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: ---------------------------------------------------- 2024-03-24T00:27:12.476 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: ntp-4 is maintained by Network Time Foundation, 2024-03-24T00:27:12.476 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-03-24T00:27:12.476 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: corporation. Support and training for ntp-4 are 2024-03-24T00:27:12.476 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: available at https://www.nwtime.org/support 2024-03-24T00:27:12.476 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: ---------------------------------------------------- 2024-03-24T00:27:12.476 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: proto: precision = 0.044 usec (-24) 2024-03-24T00:27:12.477 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: basedate set to 2022-02-04 2024-03-24T00:27:12.477 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: gps base set to 2022-02-06 (week 2196) 2024-03-24T00:27:12.477 INFO:teuthology.orchestra.run.smithi053.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-24T00:27:12.477 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-03-24T00:27:12.477 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: restrict ::: KOD does nothing without LIMITED. 2024-03-24T00:27:12.478 INFO:teuthology.orchestra.run.smithi053.stderr:restrict ::: KOD does nothing without LIMITED. 2024-03-24T00:27:12.479 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: Listen and drop on 0 v6wildcard [::]:123 2024-03-24T00:27:12.479 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-03-24T00:27:12.479 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: Listen normally on 2 lo 127.0.0.1:123 2024-03-24T00:27:12.479 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: Listen normally on 3 ens1f0 172.21.15.53:123 2024-03-24T00:27:12.479 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: Listen normally on 4 lo [::1]:123 2024-03-24T00:27:12.479 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:3718%3]:123 2024-03-24T00:27:12.479 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:12 ntpd[17063]: Listening on routing socket on fd #22 for interface updates 2024-03-24T00:27:13.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:13 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:13.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:13 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:13.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:13 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:13.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:13 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:13.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:13 ntpd[17083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:13.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:13 ntpd[17083]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:13.476 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:13 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:13.476 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:13 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:13.476 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:13 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:13.476 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:13 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:13.476 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:13 ntpd[17063]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:13.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:13 ntpd[17063]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:14.445 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:14 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:14.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:14 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:14.476 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:14 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:14.476 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:14 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:15.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:15 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:15.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:15 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:15.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:15 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:15.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:15 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:15.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:15 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:15.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:15 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:15.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:15 ntpd[17083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:15.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:15 ntpd[17083]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:15.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:15 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:15.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:15 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:15.448 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:15 ntpd[17083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:15.448 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:15 ntpd[17083]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:15.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:15 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:15.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:15 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:15.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:15 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:15.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:15 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:15.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:15 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:15.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:15 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:15.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:15 ntpd[17063]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:15.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:15 ntpd[17063]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:15.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:15 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:15.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:15 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:15.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:15 ntpd[17063]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:15.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:15 ntpd[17063]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:16.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:16 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:16.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:16 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:16.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:16 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:16.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:16 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:16.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:16 ntpd[17083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:16.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:16 ntpd[17083]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:17.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:17 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:17.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:17 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:17.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:17 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:17.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:17 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:17.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:17 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:17.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:17 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:17.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:17 ntpd[17083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:17.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:17 ntpd[17083]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:17.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:17 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:17.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:17 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:17.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:17 ntpd[17083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:17.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:17 ntpd[17083]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:17.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:17.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:17.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:17.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:17.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:17.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:17.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:17.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:17.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:17.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:17.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:17.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:17.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:17.479 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:17.479 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:17.479 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:17.479 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:17.479 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:17 ntpd[17063]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:18 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:18 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:18 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:18 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:18 ntpd[17083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:18 ntpd[17083]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:19 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:19 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:19 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:19 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:19 ntpd[17083]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:19 ntpd[17083]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:19 ntpd[17083]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:19.446 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:19 ntpd[17083]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:19.447 INFO:teuthology.orchestra.run.smithi040.stdout:24 Mar 00:27:19 ntpd[17083]: ntpd: time slew +0.001132 s 2024-03-24T00:27:19.447 INFO:teuthology.orchestra.run.smithi040.stdout:ntpd: time slew +0.001132s 2024-03-24T00:27:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:19 ntpd[17083]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-24T00:27:19.447 INFO:teuthology.orchestra.run.smithi040.stderr:24 Mar 00:27:19 ntpd[17083]: can't open /var/log/ntpstats/loopstats.20240324: Permission denied 2024-03-24T00:27:19.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:19.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:19.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:19.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:19.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:19.477 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:19.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:19.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:19.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:19.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:19.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:19.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:19.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:19.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:19.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-03-24T00:27:19.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: can't open /var/log/ntpstats/rawstats.20240324: Permission denied 2024-03-24T00:27:19.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-03-24T00:27:19.478 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: can't open /var/log/ntpstats/peerstats.20240324: Permission denied 2024-03-24T00:27:19.479 INFO:teuthology.orchestra.run.smithi053.stdout:24 Mar 00:27:19 ntpd[17063]: ntpd: time slew -0.000557 s 2024-03-24T00:27:19.479 INFO:teuthology.orchestra.run.smithi053.stdout:ntpd: time slew -0.000557s 2024-03-24T00:27:19.479 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-03-24T00:27:19.479 INFO:teuthology.orchestra.run.smithi053.stderr:24 Mar 00:27:19 ntpd[17063]: can't open /var/log/ntpstats/loopstats.20240324: Permission denied 2024-03-24T00:27:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: remote refid st t when poll reach delay offset jitter 2024-03-24T00:27:19.568 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================== 2024-03-24T00:27:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-24T00:27:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-24T00:27:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-24T00:27:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-24T00:27:19.583 INFO:teuthology.orchestra.run.smithi053.stdout: remote refid st t when poll reach delay offset jitter 2024-03-24T00:27:19.583 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================== 2024-03-24T00:27:19.584 INFO:teuthology.orchestra.run.smithi053.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-24T00:27:19.584 INFO:teuthology.orchestra.run.smithi053.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-24T00:27:19.584 INFO:teuthology.orchestra.run.smithi053.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-24T00:27:19.584 INFO:teuthology.orchestra.run.smithi053.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-03-24T00:27:19.585 INFO:teuthology.run_tasks:Running task install... 2024-03-24T00:27:19.593 DEBUG:teuthology.task.install:project ceph 2024-03-24T00:27:19.593 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-24T00:27:19.593 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-24T00:27:19.594 INFO:teuthology.task.install:Using flavor: default 2024-03-24T00:27:19.599 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T00:27:19.599 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-24T00:27:19.600 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-key list | grep Ceph 2024-03-24T00:27:19.600 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-key list | grep Ceph 2024-03-24T00:27:19.716 INFO:teuthology.orchestra.run.smithi040.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-24T00:27:19.738 INFO:teuthology.orchestra.run.smithi053.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-03-24T00:27:19.768 INFO:teuthology.orchestra.run.smithi040.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-24T00:27:19.768 INFO:teuthology.orchestra.run.smithi040.stdout:uid [ unknown] Ceph.com (release key) 2024-03-24T00:27:19.769 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-24T00:27:19.769 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-24T00:27:19.770 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:27:19.790 INFO:teuthology.orchestra.run.smithi053.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-03-24T00:27:19.790 INFO:teuthology.orchestra.run.smithi053.stdout:uid [ unknown] Ceph.com (release key) 2024-03-24T00:27:19.791 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-03-24T00:27:19.791 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-03-24T00:27:19.791 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:27:19.989 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default/ 2024-03-24T00:27:19.990 INFO:teuthology.task.install.deb:Package version is 19.0.0-2145-g3be9a5c9-1jammy 2024-03-24T00:27:20.008 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default/ 2024-03-24T00:27:20.008 INFO:teuthology.task.install.deb:Package version is 19.0.0-2145-g3be9a5c9-1jammy 2024-03-24T00:27:20.123 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:27:20.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-24T00:27:20.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo apt-get update 2024-03-24T00:27:20.161 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:27:20.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-03-24T00:27:20.181 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get update 2024-03-24T00:27:20.334 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-24T00:27:20.374 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-24T00:27:20.384 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-24T00:27:20.405 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-24T00:27:20.437 INFO:teuthology.orchestra.run.smithi053.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy InRelease 2024-03-24T00:27:20.444 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-03-24T00:27:20.474 INFO:teuthology.orchestra.run.smithi053.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-24T00:27:20.476 INFO:teuthology.orchestra.run.smithi040.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy InRelease 2024-03-24T00:27:20.485 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-03-24T00:27:20.511 INFO:teuthology.orchestra.run.smithi053.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-24T00:27:20.513 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-03-24T00:27:20.526 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-03-24T00:27:20.548 INFO:teuthology.orchestra.run.smithi053.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-24T00:27:20.550 INFO:teuthology.orchestra.run.smithi040.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy Release.gpg 2024-03-24T00:27:20.586 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,404 B] 2024-03-24T00:27:20.608 INFO:teuthology.orchestra.run.smithi053.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-03-24T00:27:20.634 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-03-24T00:27:20.634 INFO:teuthology.orchestra.run.smithi053.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,404 B] 2024-03-24T00:27:20.939 INFO:teuthology.orchestra.run.smithi053.stdout:Fetched 28.9 kB in 1s (50.6 kB/s) 2024-03-24T00:27:20.981 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 28.9 kB in 1s (50.5 kB/s) 2024-03-24T00:27:21.931 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2024-03-24T00:27:21.945 DEBUG:teuthology.orchestra.run.smithi053:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2145-g3be9a5c9-1jammy cephadm=19.0.0-2145-g3be9a5c9-1jammy ceph-mds=19.0.0-2145-g3be9a5c9-1jammy ceph-mgr=19.0.0-2145-g3be9a5c9-1jammy ceph-common=19.0.0-2145-g3be9a5c9-1jammy ceph-fuse=19.0.0-2145-g3be9a5c9-1jammy ceph-test=19.0.0-2145-g3be9a5c9-1jammy ceph-volume=19.0.0-2145-g3be9a5c9-1jammy radosgw=19.0.0-2145-g3be9a5c9-1jammy python3-rados=19.0.0-2145-g3be9a5c9-1jammy python3-rgw=19.0.0-2145-g3be9a5c9-1jammy python3-cephfs=19.0.0-2145-g3be9a5c9-1jammy python3-rbd=19.0.0-2145-g3be9a5c9-1jammy libcephfs2=19.0.0-2145-g3be9a5c9-1jammy libcephfs-dev=19.0.0-2145-g3be9a5c9-1jammy librados2=19.0.0-2145-g3be9a5c9-1jammy librbd1=19.0.0-2145-g3be9a5c9-1jammy rbd-fuse=19.0.0-2145-g3be9a5c9-1jammy python3-cephfs=19.0.0-2145-g3be9a5c9-1jammy cephfs-shell=19.0.0-2145-g3be9a5c9-1jammy cephfs-top=19.0.0-2145-g3be9a5c9-1jammy cephfs-mirror=19.0.0-2145-g3be9a5c9-1jammy 2024-03-24T00:27:21.983 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2024-03-24T00:27:21.998 DEBUG:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2145-g3be9a5c9-1jammy cephadm=19.0.0-2145-g3be9a5c9-1jammy ceph-mds=19.0.0-2145-g3be9a5c9-1jammy ceph-mgr=19.0.0-2145-g3be9a5c9-1jammy ceph-common=19.0.0-2145-g3be9a5c9-1jammy ceph-fuse=19.0.0-2145-g3be9a5c9-1jammy ceph-test=19.0.0-2145-g3be9a5c9-1jammy ceph-volume=19.0.0-2145-g3be9a5c9-1jammy radosgw=19.0.0-2145-g3be9a5c9-1jammy python3-rados=19.0.0-2145-g3be9a5c9-1jammy python3-rgw=19.0.0-2145-g3be9a5c9-1jammy python3-cephfs=19.0.0-2145-g3be9a5c9-1jammy python3-rbd=19.0.0-2145-g3be9a5c9-1jammy libcephfs2=19.0.0-2145-g3be9a5c9-1jammy libcephfs-dev=19.0.0-2145-g3be9a5c9-1jammy librados2=19.0.0-2145-g3be9a5c9-1jammy librbd1=19.0.0-2145-g3be9a5c9-1jammy rbd-fuse=19.0.0-2145-g3be9a5c9-1jammy python3-cephfs=19.0.0-2145-g3be9a5c9-1jammy cephfs-shell=19.0.0-2145-g3be9a5c9-1jammy cephfs-top=19.0.0-2145-g3be9a5c9-1jammy cephfs-mirror=19.0.0-2145-g3be9a5c9-1jammy 2024-03-24T00:27:22.022 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2024-03-24T00:27:22.074 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2024-03-24T00:27:22.205 INFO:teuthology.orchestra.run.smithi053.stdout:Building dependency tree... 2024-03-24T00:27:22.205 INFO:teuthology.orchestra.run.smithi053.stdout:Reading state information... 2024-03-24T00:27:22.248 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2024-03-24T00:27:22.249 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2024-03-24T00:27:22.375 INFO:teuthology.orchestra.run.smithi053.stdout:The following packages were automatically installed and are no longer required: 2024-03-24T00:27:22.375 INFO:teuthology.orchestra.run.smithi053.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-24T00:27:22.375 INFO:teuthology.orchestra.run.smithi053.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-24T00:27:22.375 INFO:teuthology.orchestra.run.smithi053.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-24T00:27:22.376 INFO:teuthology.orchestra.run.smithi053.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-24T00:27:22.376 INFO:teuthology.orchestra.run.smithi053.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-24T00:27:22.376 INFO:teuthology.orchestra.run.smithi053.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-24T00:27:22.377 INFO:teuthology.orchestra.run.smithi053.stdout:The following additional packages will be installed: 2024-03-24T00:27:22.377 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-24T00:27:22.377 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-24T00:27:22.377 INFO:teuthology.orchestra.run.smithi053.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-24T00:27:22.377 INFO:teuthology.orchestra.run.smithi053.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-24T00:27:22.379 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout:Suggested packages: 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout:Recommended packages: 2024-03-24T00:27:22.380 INFO:teuthology.orchestra.run.smithi053.stdout: btrfs-tools 2024-03-24T00:27:22.414 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2024-03-24T00:27:22.414 INFO:teuthology.orchestra.run.smithi040.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-24T00:27:22.415 INFO:teuthology.orchestra.run.smithi040.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-24T00:27:22.415 INFO:teuthology.orchestra.run.smithi040.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-24T00:27:22.415 INFO:teuthology.orchestra.run.smithi040.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-24T00:27:22.415 INFO:teuthology.orchestra.run.smithi040.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-24T00:27:22.415 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-24T00:27:22.416 INFO:teuthology.orchestra.run.smithi040.stdout:The following additional packages will be installed: 2024-03-24T00:27:22.416 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-03-24T00:27:22.416 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-03-24T00:27:22.418 INFO:teuthology.orchestra.run.smithi040.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-03-24T00:27:22.419 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-03-24T00:27:22.419 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-03-24T00:27:22.419 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-03-24T00:27:22.419 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-03-24T00:27:22.419 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-03-24T00:27:22.419 INFO:teuthology.orchestra.run.smithi040.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-03-24T00:27:22.419 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-03-24T00:27:22.419 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-03-24T00:27:22.419 INFO:teuthology.orchestra.run.smithi040.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-03-24T00:27:22.420 INFO:teuthology.orchestra.run.smithi040.stdout:Suggested packages: 2024-03-24T00:27:22.420 INFO:teuthology.orchestra.run.smithi040.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-03-24T00:27:22.420 INFO:teuthology.orchestra.run.smithi040.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-03-24T00:27:22.420 INFO:teuthology.orchestra.run.smithi040.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-03-24T00:27:22.420 INFO:teuthology.orchestra.run.smithi040.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-03-24T00:27:22.420 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-03-24T00:27:22.420 INFO:teuthology.orchestra.run.smithi040.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-03-24T00:27:22.420 INFO:teuthology.orchestra.run.smithi040.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-03-24T00:27:22.420 INFO:teuthology.orchestra.run.smithi040.stdout:Recommended packages: 2024-03-24T00:27:22.420 INFO:teuthology.orchestra.run.smithi040.stdout: btrfs-tools 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout:The following NEW packages will be installed: 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-24T00:27:22.540 INFO:teuthology.orchestra.run.smithi053.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-24T00:27:22.541 INFO:teuthology.orchestra.run.smithi053.stdout: socat unzip xmlstarlet zip 2024-03-24T00:27:22.574 INFO:teuthology.orchestra.run.smithi040.stdout:The following NEW packages will be installed: 2024-03-24T00:27:22.575 INFO:teuthology.orchestra.run.smithi040.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-03-24T00:27:22.575 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-03-24T00:27:22.575 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-03-24T00:27:22.576 INFO:teuthology.orchestra.run.smithi040.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-03-24T00:27:22.576 INFO:teuthology.orchestra.run.smithi040.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-03-24T00:27:22.577 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-03-24T00:27:22.578 INFO:teuthology.orchestra.run.smithi040.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-03-24T00:27:22.578 INFO:teuthology.orchestra.run.smithi040.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-03-24T00:27:22.578 INFO:teuthology.orchestra.run.smithi040.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-03-24T00:27:22.578 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-03-24T00:27:22.578 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-03-24T00:27:22.578 INFO:teuthology.orchestra.run.smithi040.stdout: socat unzip xmlstarlet zip 2024-03-24T00:27:22.613 INFO:teuthology.orchestra.run.smithi053.stdout:0 upgraded, 124 newly installed, 0 to remove and 288 not upgraded. 2024-03-24T00:27:22.613 INFO:teuthology.orchestra.run.smithi053.stdout:Need to get 178 MB of archives. 2024-03-24T00:27:22.613 INFO:teuthology.orchestra.run.smithi053.stdout:After this operation, 738 MB of additional disk space will be used. 2024-03-24T00:27:22.614 INFO:teuthology.orchestra.run.smithi053.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-24T00:27:22.634 INFO:teuthology.orchestra.run.smithi053.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-24T00:27:22.644 INFO:teuthology.orchestra.run.smithi053.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-24T00:27:22.690 INFO:teuthology.orchestra.run.smithi053.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2145-g3be9a5c9-1jammy [3,553 kB] 2024-03-24T00:27:22.695 INFO:teuthology.orchestra.run.smithi053.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-24T00:27:22.719 INFO:teuthology.orchestra.run.smithi053.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-24T00:27:22.722 INFO:teuthology.orchestra.run.smithi053.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-24T00:27:22.725 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 124 newly installed, 0 to remove and 288 not upgraded. 2024-03-24T00:27:22.725 INFO:teuthology.orchestra.run.smithi040.stdout:Need to get 178 MB of archives. 2024-03-24T00:27:22.725 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 738 MB of additional disk space will be used. 2024-03-24T00:27:22.725 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2145-g3be9a5c9-1jammy [3,553 kB] 2024-03-24T00:27:22.735 INFO:teuthology.orchestra.run.smithi053.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-24T00:27:22.774 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-03-24T00:27:22.782 INFO:teuthology.orchestra.run.smithi053.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-24T00:27:22.785 INFO:teuthology.orchestra.run.smithi053.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-24T00:27:22.795 INFO:teuthology.orchestra.run.smithi053.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-24T00:27:22.799 INFO:teuthology.orchestra.run.smithi053.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-24T00:27:22.799 INFO:teuthology.orchestra.run.smithi053.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-24T00:27:22.807 INFO:teuthology.orchestra.run.smithi053.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-24T00:27:22.809 INFO:teuthology.orchestra.run.smithi053.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-24T00:27:22.811 INFO:teuthology.orchestra.run.smithi053.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-24T00:27:22.812 INFO:teuthology.orchestra.run.smithi053.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-24T00:27:22.813 INFO:teuthology.orchestra.run.smithi053.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-24T00:27:22.832 INFO:teuthology.orchestra.run.smithi053.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-24T00:27:22.834 INFO:teuthology.orchestra.run.smithi053.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-24T00:27:22.835 INFO:teuthology.orchestra.run.smithi053.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-24T00:27:22.836 INFO:teuthology.orchestra.run.smithi053.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-24T00:27:22.836 INFO:teuthology.orchestra.run.smithi053.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-24T00:27:22.837 INFO:teuthology.orchestra.run.smithi053.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-24T00:27:22.838 INFO:teuthology.orchestra.run.smithi053.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-24T00:27:22.838 INFO:teuthology.orchestra.run.smithi053.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-24T00:27:22.839 INFO:teuthology.orchestra.run.smithi053.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-24T00:27:22.854 INFO:teuthology.orchestra.run.smithi053.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-24T00:27:22.858 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-03-24T00:27:22.859 INFO:teuthology.orchestra.run.smithi053.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-24T00:27:22.859 INFO:teuthology.orchestra.run.smithi053.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-24T00:27:22.862 INFO:teuthology.orchestra.run.smithi053.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-24T00:27:22.862 INFO:teuthology.orchestra.run.smithi053.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-24T00:27:22.863 INFO:teuthology.orchestra.run.smithi053.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-24T00:27:22.863 INFO:teuthology.orchestra.run.smithi053.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-24T00:27:22.864 INFO:teuthology.orchestra.run.smithi053.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-24T00:27:22.864 INFO:teuthology.orchestra.run.smithi053.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-24T00:27:22.874 INFO:teuthology.orchestra.run.smithi053.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-24T00:27:22.885 INFO:teuthology.orchestra.run.smithi053.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-24T00:27:22.886 INFO:teuthology.orchestra.run.smithi053.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-24T00:27:22.886 INFO:teuthology.orchestra.run.smithi053.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-24T00:27:22.891 INFO:teuthology.orchestra.run.smithi053.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-24T00:27:22.891 INFO:teuthology.orchestra.run.smithi053.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-24T00:27:22.892 INFO:teuthology.orchestra.run.smithi053.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-24T00:27:22.892 INFO:teuthology.orchestra.run.smithi053.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-24T00:27:22.893 INFO:teuthology.orchestra.run.smithi053.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-24T00:27:22.897 INFO:teuthology.orchestra.run.smithi053.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-24T00:27:22.901 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-03-24T00:27:22.915 INFO:teuthology.orchestra.run.smithi053.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-24T00:27:22.916 INFO:teuthology.orchestra.run.smithi053.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-24T00:27:22.917 INFO:teuthology.orchestra.run.smithi053.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-24T00:27:22.919 INFO:teuthology.orchestra.run.smithi053.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-24T00:27:22.919 INFO:teuthology.orchestra.run.smithi053.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-24T00:27:22.920 INFO:teuthology.orchestra.run.smithi053.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-24T00:27:22.922 INFO:teuthology.orchestra.run.smithi053.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-24T00:27:22.987 INFO:teuthology.orchestra.run.smithi053.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-24T00:27:22.988 INFO:teuthology.orchestra.run.smithi053.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-24T00:27:22.989 INFO:teuthology.orchestra.run.smithi053.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-24T00:27:22.990 INFO:teuthology.orchestra.run.smithi053.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-24T00:27:22.991 INFO:teuthology.orchestra.run.smithi053.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-24T00:27:22.996 INFO:teuthology.orchestra.run.smithi053.stdout:Get:59 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2145-g3be9a5c9-1jammy [3,223 kB] 2024-03-24T00:27:23.017 INFO:teuthology.orchestra.run.smithi053.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-24T00:27:23.019 INFO:teuthology.orchestra.run.smithi053.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-24T00:27:23.019 INFO:teuthology.orchestra.run.smithi053.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-24T00:27:23.028 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2145-g3be9a5c9-1jammy [3,223 kB] 2024-03-24T00:27:23.049 INFO:teuthology.orchestra.run.smithi053.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-24T00:27:23.049 INFO:teuthology.orchestra.run.smithi053.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-24T00:27:23.050 INFO:teuthology.orchestra.run.smithi053.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-24T00:27:23.051 INFO:teuthology.orchestra.run.smithi053.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-24T00:27:23.051 INFO:teuthology.orchestra.run.smithi053.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-24T00:27:23.052 INFO:teuthology.orchestra.run.smithi053.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-24T00:27:23.067 INFO:teuthology.orchestra.run.smithi053.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-24T00:27:23.067 INFO:teuthology.orchestra.run.smithi053.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-24T00:27:23.071 INFO:teuthology.orchestra.run.smithi053.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-24T00:27:23.086 INFO:teuthology.orchestra.run.smithi053.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-24T00:27:23.088 INFO:teuthology.orchestra.run.smithi053.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-24T00:27:23.089 INFO:teuthology.orchestra.run.smithi053.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-24T00:27:23.092 INFO:teuthology.orchestra.run.smithi053.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-24T00:27:23.095 INFO:teuthology.orchestra.run.smithi053.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-24T00:27:23.095 INFO:teuthology.orchestra.run.smithi053.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-24T00:27:23.097 INFO:teuthology.orchestra.run.smithi053.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-24T00:27:23.106 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-03-24T00:27:23.106 INFO:teuthology.orchestra.run.smithi053.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-24T00:27:23.111 INFO:teuthology.orchestra.run.smithi053.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-24T00:27:23.112 INFO:teuthology.orchestra.run.smithi053.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-24T00:27:23.126 INFO:teuthology.orchestra.run.smithi053.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-24T00:27:23.126 INFO:teuthology.orchestra.run.smithi053.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-24T00:27:23.127 INFO:teuthology.orchestra.run.smithi053.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-24T00:27:23.128 INFO:teuthology.orchestra.run.smithi053.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-24T00:27:23.129 INFO:teuthology.orchestra.run.smithi053.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-24T00:27:23.130 INFO:teuthology.orchestra.run.smithi053.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-24T00:27:23.139 INFO:teuthology.orchestra.run.smithi053.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-24T00:27:23.146 INFO:teuthology.orchestra.run.smithi053.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-24T00:27:23.146 INFO:teuthology.orchestra.run.smithi053.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-24T00:27:23.149 INFO:teuthology.orchestra.run.smithi053.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-24T00:27:23.166 INFO:teuthology.orchestra.run.smithi053.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-24T00:27:23.200 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-03-24T00:27:23.201 INFO:teuthology.orchestra.run.smithi053.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2145-g3be9a5c9-1jammy [712 kB] 2024-03-24T00:27:23.213 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-03-24T00:27:23.223 INFO:teuthology.orchestra.run.smithi053.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2145-g3be9a5c9-1jammy [356 kB] 2024-03-24T00:27:23.234 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2145-g3be9a5c9-1jammy [712 kB] 2024-03-24T00:27:23.258 INFO:teuthology.orchestra.run.smithi040.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2145-g3be9a5c9-1jammy [356 kB] 2024-03-24T00:27:23.265 INFO:teuthology.orchestra.run.smithi040.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-03-24T00:27:23.271 INFO:teuthology.orchestra.run.smithi053.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2145-g3be9a5c9-1jammy [32.8 kB] 2024-03-24T00:27:23.272 INFO:teuthology.orchestra.run.smithi053.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2145-g3be9a5c9-1jammy [180 kB] 2024-03-24T00:27:23.278 INFO:teuthology.orchestra.run.smithi053.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2145-g3be9a5c9-1jammy [64.6 kB] 2024-03-24T00:27:23.280 INFO:teuthology.orchestra.run.smithi053.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2145-g3be9a5c9-1jammy [330 kB] 2024-03-24T00:27:23.291 INFO:teuthology.orchestra.run.smithi053.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2145-g3be9a5c9-1jammy [6,659 kB] 2024-03-24T00:27:23.305 INFO:teuthology.orchestra.run.smithi040.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2145-g3be9a5c9-1jammy [32.8 kB] 2024-03-24T00:27:23.306 INFO:teuthology.orchestra.run.smithi040.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2145-g3be9a5c9-1jammy [180 kB] 2024-03-24T00:27:23.312 INFO:teuthology.orchestra.run.smithi040.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2145-g3be9a5c9-1jammy [64.6 kB] 2024-03-24T00:27:23.314 INFO:teuthology.orchestra.run.smithi040.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2145-g3be9a5c9-1jammy [330 kB] 2024-03-24T00:27:23.326 INFO:teuthology.orchestra.run.smithi040.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2145-g3be9a5c9-1jammy [6,659 kB] 2024-03-24T00:27:23.439 INFO:teuthology.orchestra.run.smithi040.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-03-24T00:27:23.448 INFO:teuthology.orchestra.run.smithi040.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-03-24T00:27:23.477 INFO:teuthology.orchestra.run.smithi040.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-03-24T00:27:23.485 INFO:teuthology.orchestra.run.smithi040.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-03-24T00:27:23.649 INFO:teuthology.orchestra.run.smithi040.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-03-24T00:27:23.751 INFO:teuthology.orchestra.run.smithi053.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2145-g3be9a5c9-1jammy [112 kB] 2024-03-24T00:27:23.752 INFO:teuthology.orchestra.run.smithi053.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2145-g3be9a5c9-1jammy [467 kB] 2024-03-24T00:27:23.767 INFO:teuthology.orchestra.run.smithi053.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2145-g3be9a5c9-1jammy [25.0 MB] 2024-03-24T00:27:23.783 INFO:teuthology.orchestra.run.smithi040.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2145-g3be9a5c9-1jammy [112 kB] 2024-03-24T00:27:23.784 INFO:teuthology.orchestra.run.smithi040.stdout:Get:23 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2145-g3be9a5c9-1jammy [467 kB] 2024-03-24T00:27:23.801 INFO:teuthology.orchestra.run.smithi040.stdout:Get:24 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2145-g3be9a5c9-1jammy [25.0 MB] 2024-03-24T00:27:24.231 INFO:teuthology.orchestra.run.smithi040.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-03-24T00:27:24.246 INFO:teuthology.orchestra.run.smithi040.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-03-24T00:27:24.262 INFO:teuthology.orchestra.run.smithi040.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-03-24T00:27:24.270 INFO:teuthology.orchestra.run.smithi040.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-03-24T00:27:24.271 INFO:teuthology.orchestra.run.smithi040.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-03-24T00:27:24.287 INFO:teuthology.orchestra.run.smithi040.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-03-24T00:27:24.325 INFO:teuthology.orchestra.run.smithi040.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-03-24T00:27:24.338 INFO:teuthology.orchestra.run.smithi040.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-03-24T00:27:24.342 INFO:teuthology.orchestra.run.smithi040.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-03-24T00:27:24.343 INFO:teuthology.orchestra.run.smithi040.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-03-24T00:27:24.512 INFO:teuthology.orchestra.run.smithi040.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-03-24T00:27:24.595 INFO:teuthology.orchestra.run.smithi040.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-03-24T00:27:24.601 INFO:teuthology.orchestra.run.smithi040.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-03-24T00:27:24.676 INFO:teuthology.orchestra.run.smithi040.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-03-24T00:27:24.704 INFO:teuthology.orchestra.run.smithi040.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-03-24T00:27:24.715 INFO:teuthology.orchestra.run.smithi040.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-03-24T00:27:24.757 INFO:teuthology.orchestra.run.smithi040.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-03-24T00:27:24.934 INFO:teuthology.orchestra.run.smithi040.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-03-24T00:27:24.948 INFO:teuthology.orchestra.run.smithi040.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-03-24T00:27:24.955 INFO:teuthology.orchestra.run.smithi040.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-03-24T00:27:24.960 INFO:teuthology.orchestra.run.smithi040.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-03-24T00:27:25.144 INFO:teuthology.orchestra.run.smithi040.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-03-24T00:27:25.227 INFO:teuthology.orchestra.run.smithi040.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-03-24T00:27:25.242 INFO:teuthology.orchestra.run.smithi040.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-03-24T00:27:25.422 INFO:teuthology.orchestra.run.smithi053.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2145-g3be9a5c9-1jammy [5,083 kB] 2024-03-24T00:27:25.427 INFO:teuthology.orchestra.run.smithi040.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-03-24T00:27:25.461 INFO:teuthology.orchestra.run.smithi040.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2145-g3be9a5c9-1jammy [5,083 kB] 2024-03-24T00:27:25.478 INFO:teuthology.orchestra.run.smithi040.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-03-24T00:27:25.491 INFO:teuthology.orchestra.run.smithi040.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-03-24T00:27:25.648 INFO:teuthology.orchestra.run.smithi040.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-03-24T00:27:25.649 INFO:teuthology.orchestra.run.smithi040.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-03-24T00:27:25.655 INFO:teuthology.orchestra.run.smithi040.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-03-24T00:27:25.659 INFO:teuthology.orchestra.run.smithi040.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-03-24T00:27:25.673 INFO:teuthology.orchestra.run.smithi040.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-03-24T00:27:25.758 INFO:teuthology.orchestra.run.smithi053.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2145-g3be9a5c9-1jammy [244 kB] 2024-03-24T00:27:25.769 INFO:teuthology.orchestra.run.smithi053.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2145-g3be9a5c9-1jammy [124 kB] 2024-03-24T00:27:25.773 INFO:teuthology.orchestra.run.smithi053.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2145-g3be9a5c9-1jammy [1,525 kB] 2024-03-24T00:27:25.798 INFO:teuthology.orchestra.run.smithi040.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2145-g3be9a5c9-1jammy [244 kB] 2024-03-24T00:27:25.808 INFO:teuthology.orchestra.run.smithi040.stdout:Get:59 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2145-g3be9a5c9-1jammy [124 kB] 2024-03-24T00:27:25.812 INFO:teuthology.orchestra.run.smithi040.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2145-g3be9a5c9-1jammy [1,525 kB] 2024-03-24T00:27:25.858 INFO:teuthology.orchestra.run.smithi040.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-03-24T00:27:25.860 INFO:teuthology.orchestra.run.smithi053.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2145-g3be9a5c9-1jammy [6,254 kB] 2024-03-24T00:27:25.908 INFO:teuthology.orchestra.run.smithi040.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2145-g3be9a5c9-1jammy [6,254 kB] 2024-03-24T00:27:26.112 INFO:teuthology.orchestra.run.smithi040.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-03-24T00:27:26.245 INFO:teuthology.orchestra.run.smithi040.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-03-24T00:27:26.289 INFO:teuthology.orchestra.run.smithi040.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-03-24T00:27:26.291 INFO:teuthology.orchestra.run.smithi053.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2145-g3be9a5c9-1jammy [22.8 MB] 2024-03-24T00:27:26.327 INFO:teuthology.orchestra.run.smithi040.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2145-g3be9a5c9-1jammy [22.8 MB] 2024-03-24T00:27:26.380 INFO:teuthology.orchestra.run.smithi040.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-03-24T00:27:26.391 INFO:teuthology.orchestra.run.smithi040.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-03-24T00:27:26.402 INFO:teuthology.orchestra.run.smithi040.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-03-24T00:27:26.445 INFO:teuthology.orchestra.run.smithi040.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-03-24T00:27:26.708 INFO:teuthology.orchestra.run.smithi040.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-03-24T00:27:26.709 INFO:teuthology.orchestra.run.smithi040.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-03-24T00:27:26.711 INFO:teuthology.orchestra.run.smithi040.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-03-24T00:27:26.875 INFO:teuthology.orchestra.run.smithi040.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-03-24T00:27:27.217 INFO:teuthology.orchestra.run.smithi040.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-03-24T00:27:27.617 INFO:teuthology.orchestra.run.smithi040.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-03-24T00:27:27.626 INFO:teuthology.orchestra.run.smithi040.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-03-24T00:27:27.627 INFO:teuthology.orchestra.run.smithi040.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-03-24T00:27:27.695 INFO:teuthology.orchestra.run.smithi040.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-03-24T00:27:27.696 INFO:teuthology.orchestra.run.smithi040.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-03-24T00:27:27.696 INFO:teuthology.orchestra.run.smithi040.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-03-24T00:27:27.698 INFO:teuthology.orchestra.run.smithi040.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-03-24T00:27:27.699 INFO:teuthology.orchestra.run.smithi040.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-03-24T00:27:27.700 INFO:teuthology.orchestra.run.smithi040.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-03-24T00:27:27.757 INFO:teuthology.orchestra.run.smithi053.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2145-g3be9a5c9-1jammy [14.0 kB] 2024-03-24T00:27:27.757 INFO:teuthology.orchestra.run.smithi053.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2145-g3be9a5c9-1jammy [870 kB] 2024-03-24T00:27:27.803 INFO:teuthology.orchestra.run.smithi053.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2145-g3be9a5c9-1jammy [2,385 kB] 2024-03-24T00:27:27.864 INFO:teuthology.orchestra.run.smithi040.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-03-24T00:27:27.871 INFO:teuthology.orchestra.run.smithi040.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2145-g3be9a5c9-1jammy [14.0 kB] 2024-03-24T00:27:27.871 INFO:teuthology.orchestra.run.smithi040.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2145-g3be9a5c9-1jammy [870 kB] 2024-03-24T00:27:27.935 INFO:teuthology.orchestra.run.smithi053.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2145-g3be9a5c9-1jammy [777 kB] 2024-03-24T00:27:27.937 INFO:teuthology.orchestra.run.smithi040.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2145-g3be9a5c9-1jammy [2,385 kB] 2024-03-24T00:27:27.967 INFO:teuthology.orchestra.run.smithi053.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2145-g3be9a5c9-1jammy [144 kB] 2024-03-24T00:27:27.972 INFO:teuthology.orchestra.run.smithi053.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2145-g3be9a5c9-1jammy [1,936 kB] 2024-03-24T00:27:28.067 INFO:teuthology.orchestra.run.smithi040.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-03-24T00:27:28.094 INFO:teuthology.orchestra.run.smithi053.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2145-g3be9a5c9-1jammy [8,625 kB] 2024-03-24T00:27:28.132 INFO:teuthology.orchestra.run.smithi040.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2145-g3be9a5c9-1jammy [777 kB] 2024-03-24T00:27:28.193 INFO:teuthology.orchestra.run.smithi040.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2145-g3be9a5c9-1jammy [144 kB] 2024-03-24T00:27:28.202 INFO:teuthology.orchestra.run.smithi040.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2145-g3be9a5c9-1jammy [1,936 kB] 2024-03-24T00:27:28.348 INFO:teuthology.orchestra.run.smithi040.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2145-g3be9a5c9-1jammy [8,625 kB] 2024-03-24T00:27:28.391 INFO:teuthology.orchestra.run.smithi040.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-03-24T00:27:28.435 INFO:teuthology.orchestra.run.smithi040.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-03-24T00:27:28.462 INFO:teuthology.orchestra.run.smithi040.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-03-24T00:27:28.471 INFO:teuthology.orchestra.run.smithi040.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-03-24T00:27:28.517 INFO:teuthology.orchestra.run.smithi040.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-03-24T00:27:28.542 INFO:teuthology.orchestra.run.smithi040.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-03-24T00:27:28.544 INFO:teuthology.orchestra.run.smithi040.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-03-24T00:27:28.550 INFO:teuthology.orchestra.run.smithi040.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-03-24T00:27:28.554 INFO:teuthology.orchestra.run.smithi040.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-03-24T00:27:28.586 INFO:teuthology.orchestra.run.smithi053.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2145-g3be9a5c9-1jammy [14.2 kB] 2024-03-24T00:27:28.586 INFO:teuthology.orchestra.run.smithi053.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2145-g3be9a5c9-1jammy [50.2 MB] 2024-03-24T00:27:28.750 INFO:teuthology.orchestra.run.smithi040.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-03-24T00:27:29.009 INFO:teuthology.orchestra.run.smithi040.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-03-24T00:27:29.012 INFO:teuthology.orchestra.run.smithi040.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2145-g3be9a5c9-1jammy [14.2 kB] 2024-03-24T00:27:29.038 INFO:teuthology.orchestra.run.smithi040.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2145-g3be9a5c9-1jammy [50.2 MB] 2024-03-24T00:27:29.273 INFO:teuthology.orchestra.run.smithi040.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-03-24T00:27:29.274 INFO:teuthology.orchestra.run.smithi040.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-03-24T00:27:29.291 INFO:teuthology.orchestra.run.smithi040.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-03-24T00:27:29.296 INFO:teuthology.orchestra.run.smithi040.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-03-24T00:27:29.333 INFO:teuthology.orchestra.run.smithi040.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-03-24T00:27:29.348 INFO:teuthology.orchestra.run.smithi040.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-03-24T00:27:29.439 INFO:teuthology.orchestra.run.smithi040.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-03-24T00:27:29.440 INFO:teuthology.orchestra.run.smithi040.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-03-24T00:27:29.442 INFO:teuthology.orchestra.run.smithi040.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-03-24T00:27:29.622 INFO:teuthology.orchestra.run.smithi040.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-03-24T00:27:29.885 INFO:teuthology.orchestra.run.smithi040.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-03-24T00:27:31.500 INFO:teuthology.orchestra.run.smithi053.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2145-g3be9a5c9-1jammy [121 kB] 2024-03-24T00:27:31.501 INFO:teuthology.orchestra.run.smithi053.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2145-g3be9a5c9-1jammy [228 kB] 2024-03-24T00:27:31.503 INFO:teuthology.orchestra.run.smithi053.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2145-g3be9a5c9-1jammy [34.0 kB] 2024-03-24T00:27:31.503 INFO:teuthology.orchestra.run.smithi053.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2145-g3be9a5c9-1jammy [23.4 kB] 2024-03-24T00:27:31.506 INFO:teuthology.orchestra.run.smithi053.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2145-g3be9a5c9-1jammy [38.5 kB] 2024-03-24T00:27:31.511 INFO:teuthology.orchestra.run.smithi053.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2145-g3be9a5c9-1jammy [13.2 MB] 2024-03-24T00:27:32.300 INFO:teuthology.orchestra.run.smithi053.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2145-g3be9a5c9-1jammy [91.7 kB] 2024-03-24T00:27:32.527 INFO:teuthology.orchestra.run.smithi040.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2145-g3be9a5c9-1jammy [121 kB] 2024-03-24T00:27:32.528 INFO:teuthology.orchestra.run.smithi040.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2145-g3be9a5c9-1jammy [228 kB] 2024-03-24T00:27:32.530 INFO:teuthology.orchestra.run.smithi040.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2145-g3be9a5c9-1jammy [34.0 kB] 2024-03-24T00:27:32.530 INFO:teuthology.orchestra.run.smithi040.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2145-g3be9a5c9-1jammy [23.4 kB] 2024-03-24T00:27:32.531 INFO:teuthology.orchestra.run.smithi040.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2145-g3be9a5c9-1jammy [38.5 kB] 2024-03-24T00:27:32.531 INFO:teuthology.orchestra.run.smithi040.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2145-g3be9a5c9-1jammy [13.2 MB] 2024-03-24T00:27:32.960 INFO:teuthology.orchestra.run.smithi040.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2145-g3be9a5c9-1jammy [91.7 kB] 2024-03-24T00:27:32.961 INFO:teuthology.orchestra.run.smithi053.stdout:Fetched 178 MB in 10s (18.3 MB/s) 2024-03-24T00:27:33.214 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-24T00:27:33.281 INFO:teuthology.orchestra.run.smithi053.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132923 files and directories currently installed.) 2024-03-24T00:27:33.284 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-24T00:27:33.336 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-24T00:27:33.621 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-24T00:27:33.623 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 178 MB in 10s (17.2 MB/s) 2024-03-24T00:27:33.635 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-24T00:27:33.680 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-24T00:27:33.801 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyperclip. 2024-03-24T00:27:33.872 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132923 files and directories currently installed.) 2024-03-24T00:27:33.876 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-03-24T00:27:33.916 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-03-24T00:27:34.008 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cmd2. 2024-03-24T00:27:34.022 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-24T00:27:34.091 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-24T00:27:34.177 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-wcwidth. 2024-03-24T00:27:34.191 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-03-24T00:27:34.235 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-24T00:27:34.588 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cmd2. 2024-03-24T00:27:34.604 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-03-24T00:27:34.628 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-24T00:27:34.646 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-24T00:27:34.646 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-03-24T00:27:34.728 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-24T00:27:35.098 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-24T00:27:35.113 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-24T00:27:35.116 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-03-24T00:27:35.132 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-03-24T00:27:35.148 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-24T00:27:35.216 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-24T00:27:35.409 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-24T00:27:35.425 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-24T00:27:35.462 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-24T00:27:35.787 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-03-24T00:27:35.803 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-03-24T00:27:35.837 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-24T00:27:35.946 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-24T00:27:35.961 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-24T00:27:36.013 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T00:27:36.148 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-03-24T00:27:36.163 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-03-24T00:27:36.206 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-24T00:27:36.458 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-24T00:27:36.473 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-24T00:27:36.516 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T00:27:36.685 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-03-24T00:27:36.701 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-24T00:27:36.752 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T00:27:36.886 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-24T00:27:36.902 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-24T00:27:36.944 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T00:27:37.122 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-03-24T00:27:37.137 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-24T00:27:37.172 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T00:27:37.289 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-24T00:27:37.304 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-24T00:27:37.339 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-24T00:27:37.542 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-03-24T00:27:37.557 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-03-24T00:27:37.600 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T00:27:37.659 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package librados2. 2024-03-24T00:27:37.674 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../010-librados2_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:37.717 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking librados2 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:37.961 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-03-24T00:27:37.976 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-03-24T00:27:38.020 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-24T00:27:38.195 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package librbd1. 2024-03-24T00:27:38.210 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../011-librbd1_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:38.253 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking librbd1 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:38.381 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librados2. 2024-03-24T00:27:38.396 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../010-librados2_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:38.431 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librados2 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:38.665 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libcephfs2. 2024-03-24T00:27:38.679 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:38.715 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libcephfs2 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:38.859 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librbd1. 2024-03-24T00:27:38.874 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../011-librbd1_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:38.917 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librbd1 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:38.967 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-rados. 2024-03-24T00:27:38.982 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:39.025 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-rados (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:39.361 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libcephfs2. 2024-03-24T00:27:39.376 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:39.428 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libcephfs2 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:39.429 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-24T00:27:39.444 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:27:39.487 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-ceph-argparse (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:39.697 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rados. 2024-03-24T00:27:39.713 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:39.740 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cephfs. 2024-03-24T00:27:39.756 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:39.756 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rados (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:39.798 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cephfs (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:40.067 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-03-24T00:27:40.083 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:27:40.093 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-24T00:27:40.108 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:27:40.116 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-ceph-argparse (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:40.142 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-ceph-common (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:40.352 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cephfs. 2024-03-24T00:27:40.368 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:40.410 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cephfs (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:40.479 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-prettytable. 2024-03-24T00:27:40.495 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-24T00:27:40.537 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-24T00:27:40.781 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-ceph-common. 2024-03-24T00:27:40.796 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:27:40.824 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-rbd. 2024-03-24T00:27:40.838 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-ceph-common (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:40.839 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:40.873 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-rbd (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:41.178 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-prettytable. 2024-03-24T00:27:41.190 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-03-24T00:27:41.218 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-24T00:27:41.233 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-03-24T00:27:41.234 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-24T00:27:41.276 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-24T00:27:41.461 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rbd. 2024-03-24T00:27:41.476 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:41.511 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rbd (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:41.588 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-24T00:27:41.603 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-24T00:27:41.646 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-24T00:27:41.872 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-03-24T00:27:41.888 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-03-24T00:27:41.930 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-24T00:27:41.932 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-24T00:27:41.947 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-24T00:27:41.982 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-24T00:27:42.208 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-03-24T00:27:42.223 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-03-24T00:27:42.266 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-03-24T00:27:42.275 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package lua5.1. 2024-03-24T00:27:42.291 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-24T00:27:42.344 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-24T00:27:42.586 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-03-24T00:27:42.601 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-03-24T00:27:42.644 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-24T00:27:42.655 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package lua-any. 2024-03-24T00:27:42.670 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-24T00:27:42.713 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-24T00:27:42.923 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package lua5.1. 2024-03-24T00:27:42.938 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-03-24T00:27:42.966 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package zip. 2024-03-24T00:27:42.972 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-03-24T00:27:42.981 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-24T00:27:43.052 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking zip (3.0-12build2) ... 2024-03-24T00:27:43.309 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package lua-any. 2024-03-24T00:27:43.324 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-03-24T00:27:43.344 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package unzip. 2024-03-24T00:27:43.360 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-24T00:27:43.392 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking lua-any (27ubuntu1) ... 2024-03-24T00:27:43.394 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-24T00:27:43.645 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package zip. 2024-03-24T00:27:43.660 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-03-24T00:27:43.703 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking zip (3.0-12build2) ... 2024-03-24T00:27:43.815 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package luarocks. 2024-03-24T00:27:43.830 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-24T00:27:43.873 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-24T00:27:44.006 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package unzip. 2024-03-24T00:27:44.022 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-03-24T00:27:44.056 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-03-24T00:27:44.265 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package librgw2. 2024-03-24T00:27:44.279 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../027-librgw2_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:44.323 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking librgw2 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:44.469 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package luarocks. 2024-03-24T00:27:44.484 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-03-24T00:27:44.526 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-03-24T00:27:44.968 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librgw2. 2024-03-24T00:27:44.983 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../027-librgw2_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:44.984 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-rgw. 2024-03-24T00:27:44.999 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:45.034 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-rgw (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:45.034 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librgw2 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:45.387 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-24T00:27:45.402 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-24T00:27:45.445 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-24T00:27:45.765 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rgw. 2024-03-24T00:27:45.765 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libradosstriper1. 2024-03-24T00:27:45.781 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:45.781 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:45.823 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libradosstriper1 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:45.829 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rgw (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:46.210 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-common. 2024-03-24T00:27:46.226 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:46.260 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-common (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:46.265 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liboath0:amd64. 2024-03-24T00:27:46.281 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-03-24T00:27:46.332 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-03-24T00:27:46.652 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libradosstriper1. 2024-03-24T00:27:46.667 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:46.718 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libradosstriper1 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:47.263 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-common. 2024-03-24T00:27:47.279 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:47.330 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-common (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:47.374 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-base. 2024-03-24T00:27:47.390 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:47.442 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-base (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:48.014 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-24T00:27:48.029 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-24T00:27:48.072 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-24T00:27:48.333 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cheroot. 2024-03-24T00:27:48.348 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-24T00:27:48.383 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-24T00:27:48.419 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-base. 2024-03-24T00:27:48.434 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:48.486 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-base (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:48.728 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-24T00:27:48.743 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-24T00:27:48.786 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-24T00:27:49.097 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-24T00:27:49.113 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-24T00:27:49.115 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-03-24T00:27:49.130 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-03-24T00:27:49.155 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-24T00:27:49.181 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-03-24T00:27:49.492 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-24T00:27:49.507 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-24T00:27:49.550 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-24T00:27:49.576 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cheroot. 2024-03-24T00:27:49.592 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-03-24T00:27:49.642 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-24T00:27:49.803 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-tempora. 2024-03-24T00:27:49.818 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-24T00:27:49.861 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-24T00:27:50.029 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-03-24T00:27:50.053 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-03-24T00:27:50.104 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-03-24T00:27:50.197 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-portend. 2024-03-24T00:27:50.212 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-24T00:27:50.256 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-24T00:27:50.440 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jaraco.text. 2024-03-24T00:27:50.456 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-03-24T00:27:50.498 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-03-24T00:27:50.525 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-24T00:27:50.540 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-24T00:27:50.575 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-24T00:27:50.895 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-24T00:27:50.910 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-24T00:27:50.952 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-24T00:27:50.993 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-03-24T00:27:51.009 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-03-24T00:27:51.060 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-03-24T00:27:51.323 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-natsort. 2024-03-24T00:27:51.339 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-24T00:27:51.381 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-24T00:27:51.455 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-tempora. 2024-03-24T00:27:51.470 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-03-24T00:27:51.521 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-03-24T00:27:51.676 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-logutils. 2024-03-24T00:27:51.691 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-24T00:27:51.726 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-24T00:27:51.891 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-portend. 2024-03-24T00:27:51.906 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-03-24T00:27:51.957 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-03-24T00:27:51.962 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-24T00:27:51.977 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-24T00:27:52.020 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-24T00:27:52.285 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-03-24T00:27:52.301 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-03-24T00:27:52.340 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-mako. 2024-03-24T00:27:52.352 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-03-24T00:27:52.356 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-24T00:27:52.398 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-24T00:27:52.714 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cherrypy3. 2024-03-24T00:27:52.729 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-03-24T00:27:52.734 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-24T00:27:52.750 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-24T00:27:52.780 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-03-24T00:27:52.792 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-24T00:27:53.087 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-24T00:27:53.103 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-24T00:27:53.137 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-24T00:27:53.159 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-natsort. 2024-03-24T00:27:53.174 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-03-24T00:27:53.217 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-03-24T00:27:53.382 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-webob. 2024-03-24T00:27:53.397 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-24T00:27:53.436 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-24T00:27:53.470 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-logutils. 2024-03-24T00:27:53.485 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-03-24T00:27:53.537 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-03-24T00:27:53.793 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-waitress. 2024-03-24T00:27:53.808 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-24T00:27:53.831 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-markupsafe. 2024-03-24T00:27:53.847 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-03-24T00:27:53.854 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-24T00:27:53.889 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-03-24T00:27:54.162 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-tempita. 2024-03-24T00:27:54.175 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-mako. 2024-03-24T00:27:54.178 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-24T00:27:54.191 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-03-24T00:27:54.220 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-24T00:27:54.234 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-24T00:27:54.531 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-paste. 2024-03-24T00:27:54.547 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-24T00:27:54.589 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-24T00:27:54.594 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-simplegeneric. 2024-03-24T00:27:54.609 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-03-24T00:27:54.654 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-03-24T00:27:54.899 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-singledispatch. 2024-03-24T00:27:54.901 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-24T00:27:54.914 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-03-24T00:27:54.916 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-24T00:27:54.948 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-03-24T00:27:54.951 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-24T00:27:55.243 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-webob. 2024-03-24T00:27:55.258 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-03-24T00:27:55.426 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-03-24T00:27:55.462 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-24T00:27:55.478 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-24T00:27:55.520 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-24T00:27:56.404 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-waitress. 2024-03-24T00:27:56.420 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-03-24T00:27:56.474 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-24T00:27:56.582 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-webtest. 2024-03-24T00:27:56.598 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-24T00:27:56.640 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-24T00:27:56.782 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-tempita. 2024-03-24T00:27:56.798 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-03-24T00:27:56.866 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-03-24T00:27:56.935 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pecan. 2024-03-24T00:27:56.951 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-24T00:27:56.993 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-24T00:27:57.152 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-paste. 2024-03-24T00:27:57.168 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-03-24T00:27:57.210 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-03-24T00:27:57.390 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-certifi. 2024-03-24T00:27:57.405 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-24T00:27:57.439 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-24T00:27:57.505 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-03-24T00:27:57.521 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-03-24T00:27:57.555 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-03-24T00:27:57.701 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-idna. 2024-03-24T00:27:57.716 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-24T00:27:57.759 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-24T00:27:57.883 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pastedeploy. 2024-03-24T00:27:57.898 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-03-24T00:27:57.941 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-03-24T00:27:58.130 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-urllib3. 2024-03-24T00:27:58.144 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-24T00:27:58.186 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-24T00:27:58.228 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-webtest. 2024-03-24T00:27:58.243 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-03-24T00:27:58.286 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-03-24T00:27:58.532 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-requests. 2024-03-24T00:27:58.547 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-24T00:27:58.590 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-24T00:27:58.639 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pecan. 2024-03-24T00:27:58.655 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-03-24T00:27:58.689 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-03-24T00:27:58.901 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-24T00:27:58.917 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-24T00:27:58.959 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-24T00:27:59.035 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-certifi. 2024-03-24T00:27:59.051 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-03-24T00:27:59.085 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-03-24T00:27:59.287 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-24T00:27:59.303 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:27:59.345 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:59.388 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-idna. 2024-03-24T00:27:59.404 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-03-24T00:27:59.522 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-idna (3.3-1) ... 2024-03-24T00:27:59.767 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-24T00:27:59.783 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:27:59.825 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:27:59.841 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-urllib3. 2024-03-24T00:27:59.857 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-03-24T00:27:59.899 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-24T00:28:00.119 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-24T00:28:00.135 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-24T00:28:00.169 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-24T00:28:00.236 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-requests. 2024-03-24T00:28:00.252 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-03-24T00:28:00.294 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-24T00:28:00.464 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-openssl. 2024-03-24T00:28:00.480 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-24T00:28:00.522 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-24T00:28:00.647 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-werkzeug. 2024-03-24T00:28:00.664 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-03-24T00:28:00.705 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-24T00:28:00.892 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr. 2024-03-24T00:28:00.908 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:00.950 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:01.001 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-03-24T00:28:01.016 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:01.058 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:01.270 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mon. 2024-03-24T00:28:01.286 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:01.320 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mon (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:01.647 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-03-24T00:28:01.663 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:01.705 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:01.923 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-osd. 2024-03-24T00:28:01.939 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:01.982 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-osd (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:02.008 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-bcrypt. 2024-03-24T00:28:02.024 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-03-24T00:28:02.091 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-03-24T00:28:02.394 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-openssl. 2024-03-24T00:28:02.410 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-03-24T00:28:02.452 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-03-24T00:28:02.766 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr. 2024-03-24T00:28:02.782 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:02.810 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph. 2024-03-24T00:28:02.824 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:02.825 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../068-ceph_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:02.868 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:03.137 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-fuse. 2024-03-24T00:28:03.153 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:03.195 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-fuse (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:03.261 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mon. 2024-03-24T00:28:03.276 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:03.319 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mon (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:03.574 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mds. 2024-03-24T00:28:03.589 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:03.632 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mds (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:03.914 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-osd. 2024-03-24T00:28:03.928 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:03.972 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-osd (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:04.043 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package cephadm. 2024-03-24T00:28:04.059 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../071-cephadm_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:04.101 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking cephadm (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:04.395 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-24T00:28:04.411 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-24T00:28:04.445 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-24T00:28:04.759 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph. 2024-03-24T00:28:04.776 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../068-ceph_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:04.790 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-24T00:28:04.805 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-24T00:28:04.817 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:04.848 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-24T00:28:05.136 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-fuse. 2024-03-24T00:28:05.152 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:05.194 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-fuse (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:05.531 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mds. 2024-03-24T00:28:05.546 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:05.589 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mds (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:05.961 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-babel. 2024-03-24T00:28:05.967 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package cephadm. 2024-03-24T00:28:05.975 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-24T00:28:05.983 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../071-cephadm_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:06.019 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-24T00:28:06.025 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking cephadm (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:06.337 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-asyncssh. 2024-03-24T00:28:06.352 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-03-24T00:28:06.380 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-jinja2. 2024-03-24T00:28:06.394 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-03-24T00:28:06.395 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-24T00:28:06.438 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-24T00:28:06.689 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-babel-localedata. 2024-03-24T00:28:06.704 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-03-24T00:28:06.738 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-24T00:28:06.808 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-24T00:28:06.823 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:06.867 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:07.245 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-24T00:28:07.259 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-24T00:28:07.303 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-24T00:28:07.606 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-routes. 2024-03-24T00:28:07.620 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-24T00:28:07.664 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-24T00:28:07.760 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-babel. 2024-03-24T00:28:07.775 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-03-24T00:28:07.818 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-03-24T00:28:08.026 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-24T00:28:08.041 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:08.084 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:08.138 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jinja2. 2024-03-24T00:28:08.153 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-03-24T00:28:08.229 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-24T00:28:08.504 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-24T00:28:08.519 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-24T00:28:08.579 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-24T00:28:08.608 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-03-24T00:28:08.624 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:08.666 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:09.019 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-repoze.lru. 2024-03-24T00:28:09.035 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-03-24T00:28:09.091 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-joblib. 2024-03-24T00:28:09.094 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-03-24T00:28:09.105 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-24T00:28:09.149 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-24T00:28:09.380 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-routes. 2024-03-24T00:28:09.396 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-03-24T00:28:09.438 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-03-24T00:28:09.527 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-24T00:28:09.542 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-24T00:28:09.671 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-24T00:28:09.783 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-03-24T00:28:09.799 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:09.841 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:09.974 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-sklearn. 2024-03-24T00:28:09.988 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-24T00:28:10.057 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-24T00:28:10.278 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-03-24T00:28:10.294 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-03-24T00:28:10.336 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-24T00:28:10.733 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-24T00:28:10.747 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:10.807 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:10.873 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-joblib. 2024-03-24T00:28:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-03-24T00:28:10.931 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-03-24T00:28:11.277 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-03-24T00:28:11.292 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-03-24T00:28:11.428 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-03-24T00:28:11.878 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-cachetools. 2024-03-24T00:28:11.894 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-24T00:28:11.923 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-sklearn. 2024-03-24T00:28:11.927 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-24T00:28:11.939 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-03-24T00:28:11.981 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-24T00:28:12.172 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-24T00:28:12.187 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-24T00:28:12.230 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-24T00:28:12.592 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-24T00:28:12.608 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-24T00:28:12.650 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-24T00:28:12.657 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-03-24T00:28:12.672 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:12.715 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:12.962 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-rsa. 2024-03-24T00:28:12.977 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-24T00:28:13.020 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-24T00:28:13.365 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-google-auth. 2024-03-24T00:28:13.381 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-24T00:28:13.432 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-24T00:28:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cachetools. 2024-03-24T00:28:13.735 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-03-24T00:28:13.735 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-24T00:28:13.751 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-24T00:28:13.768 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-03-24T00:28:13.793 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-24T00:28:14.046 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyasn1. 2024-03-24T00:28:14.057 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-03-24T00:28:14.096 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-03-24T00:28:14.129 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-websocket. 2024-03-24T00:28:14.145 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-24T00:28:14.187 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-24T00:28:14.466 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-03-24T00:28:14.475 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-03-24T00:28:14.507 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-24T00:28:14.516 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-03-24T00:28:14.523 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-24T00:28:14.588 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-24T00:28:14.828 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rsa. 2024-03-24T00:28:14.837 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-03-24T00:28:14.877 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rsa (4.8-1) ... 2024-03-24T00:28:15.215 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-google-auth. 2024-03-24T00:28:15.224 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-03-24T00:28:15.264 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-03-24T00:28:15.341 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-24T00:28:15.355 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:15.404 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:15.568 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-03-24T00:28:15.580 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-03-24T00:28:15.609 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-24T00:28:15.723 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-24T00:28:15.738 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-24T00:28:15.879 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-websocket. 2024-03-24T00:28:15.889 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-03-24T00:28:15.928 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-03-24T00:28:15.982 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-24T00:28:16.282 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-kubernetes. 2024-03-24T00:28:16.292 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-03-24T00:28:16.310 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-24T00:28:16.325 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-24T00:28:16.331 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-24T00:28:16.360 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-24T00:28:16.604 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package jq. 2024-03-24T00:28:16.620 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-24T00:28:16.662 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-24T00:28:16.940 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package socat. 2024-03-24T00:28:16.956 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-24T00:28:16.998 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-24T00:28:17.162 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-03-24T00:28:17.172 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:17.212 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:17.327 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package xmlstarlet. 2024-03-24T00:28:17.348 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-24T00:28:17.385 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-24T00:28:17.515 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libonig5:amd64. 2024-03-24T00:28:17.524 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-03-24T00:28:17.564 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-24T00:28:17.781 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-test. 2024-03-24T00:28:17.796 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:17.839 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-test (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:17.910 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libjq1:amd64. 2024-03-24T00:28:17.920 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-03-24T00:28:17.951 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-24T00:28:18.179 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package jq. 2024-03-24T00:28:18.189 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-03-24T00:28:18.228 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-03-24T00:28:18.506 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package socat. 2024-03-24T00:28:18.516 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-03-24T00:28:18.556 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-03-24T00:28:18.893 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package xmlstarlet. 2024-03-24T00:28:18.903 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-03-24T00:28:18.942 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-03-24T00:28:19.305 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-test. 2024-03-24T00:28:19.315 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:19.355 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-test (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:19.835 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ceph-volume. 2024-03-24T00:28:19.851 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:19.893 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ceph-volume (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:20.213 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-24T00:28:20.229 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:20.263 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking cephfs-mirror (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:20.600 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-colorama. 2024-03-24T00:28:20.616 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-24T00:28:20.658 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-24T00:28:20.919 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package cephfs-shell. 2024-03-24T00:28:20.932 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:20.968 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking cephfs-shell (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:21.271 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package cephfs-top. 2024-03-24T00:28:21.287 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:21.309 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-volume. 2024-03-24T00:28:21.320 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:21.329 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking cephfs-top (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:21.367 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-volume (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:21.565 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-24T00:28:21.579 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:21.615 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libcephfs-dev (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:21.755 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package cephfs-mirror. 2024-03-24T00:28:21.765 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:21.805 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking cephfs-mirror (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:22.068 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-24T00:28:22.082 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-24T00:28:22.126 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-24T00:28:22.425 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-colorama. 2024-03-24T00:28:22.438 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-03-24T00:28:22.464 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-24T00:28:22.478 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-24T00:28:22.491 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-03-24T00:28:22.522 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-24T00:28:22.802 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package cephfs-shell. 2024-03-24T00:28:22.814 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:22.860 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking cephfs-shell (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:22.934 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package nvme-cli. 2024-03-24T00:28:22.949 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-24T00:28:22.992 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-24T00:28:23.189 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package cephfs-top. 2024-03-24T00:28:23.199 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2145-g3be9a5c9-1jammy_all.deb ... 2024-03-24T00:28:23.246 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking cephfs-top (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:23.439 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package pkg-config. 2024-03-24T00:28:23.454 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-24T00:28:23.490 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libcephfs-dev. 2024-03-24T00:28:23.497 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-24T00:28:23.502 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:23.540 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libcephfs-dev (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:23.750 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-24T00:28:23.765 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-24T00:28:23.800 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-24T00:28:23.943 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package lua-socket:amd64. 2024-03-24T00:28:23.957 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-03-24T00:28:24.009 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-24T00:28:24.287 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-24T00:28:24.301 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-24T00:28:24.345 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-24T00:28:24.347 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package lua-sec:amd64. 2024-03-24T00:28:24.360 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-03-24T00:28:24.397 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-03-24T00:28:24.589 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pastescript. 2024-03-24T00:28:24.603 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-24T00:28:24.639 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-24T00:28:24.810 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package nvme-cli. 2024-03-24T00:28:24.820 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-03-24T00:28:24.860 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-03-24T00:28:24.978 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pluggy. 2024-03-24T00:28:24.992 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-24T00:28:25.036 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-24T00:28:25.256 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package pkg-config. 2024-03-24T00:28:25.267 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-03-24T00:28:25.306 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-03-24T00:28:25.364 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-psutil. 2024-03-24T00:28:25.378 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-24T00:28:25.422 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-24T00:28:25.517 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-03-24T00:28:25.528 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-03-24T00:28:25.559 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-03-24T00:28:25.791 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-py. 2024-03-24T00:28:25.806 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-24T00:28:25.849 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-24T00:28:25.970 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-iniconfig. 2024-03-24T00:28:25.981 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-03-24T00:28:26.012 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-03-24T00:28:26.186 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pygments. 2024-03-24T00:28:26.201 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-24T00:28:26.244 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-24T00:28:26.298 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pastescript. 2024-03-24T00:28:26.308 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-03-24T00:28:26.347 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-03-24T00:28:26.685 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pluggy. 2024-03-24T00:28:26.695 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-03-24T00:28:26.698 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-24T00:28:26.712 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-24T00:28:26.734 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-03-24T00:28:26.756 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-24T00:28:27.026 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-toml. 2024-03-24T00:28:27.029 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-psutil. 2024-03-24T00:28:27.039 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-03-24T00:28:27.042 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-24T00:28:27.078 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-03-24T00:28:27.101 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-24T00:28:27.379 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-pytest. 2024-03-24T00:28:27.394 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-24T00:28:27.437 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-24T00:28:27.456 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-py. 2024-03-24T00:28:27.467 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-03-24T00:28:27.506 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-py (1.10.0-1) ... 2024-03-24T00:28:27.749 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package python3-simplejson. 2024-03-24T00:28:27.764 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-24T00:28:27.823 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-24T00:28:27.843 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pygments. 2024-03-24T00:28:27.854 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-03-24T00:28:27.893 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-03-24T00:28:28.059 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-24T00:28:28.075 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-24T00:28:28.109 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-24T00:28:28.380 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyinotify. 2024-03-24T00:28:28.390 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-03-24T00:28:28.430 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-03-24T00:28:28.683 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-toml. 2024-03-24T00:28:28.694 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-03-24T00:28:28.724 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-03-24T00:28:28.922 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package radosgw. 2024-03-24T00:28:28.936 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../122-radosgw_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:28.980 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking radosgw (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:29.027 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pytest. 2024-03-24T00:28:29.038 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-03-24T00:28:29.077 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-03-24T00:28:29.423 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-simplejson. 2024-03-24T00:28:29.433 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-03-24T00:28:29.464 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-03-24T00:28:29.599 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package rbd-fuse. 2024-03-24T00:28:29.614 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:29.649 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking rbd-fuse (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:29.692 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-03-24T00:28:29.702 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-03-24T00:28:29.733 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-03-24T00:28:29.943 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-24T00:28:30.213 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-24T00:28:30.339 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-24T00:28:30.496 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package radosgw. 2024-03-24T00:28:30.505 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../122-radosgw_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:30.545 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking radosgw (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:30.606 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-24T00:28:31.018 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-24T00:28:31.215 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package rbd-fuse. 2024-03-24T00:28:31.225 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2145-g3be9a5c9-1jammy_amd64.deb ... 2024-03-24T00:28:31.265 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking rbd-fuse (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:31.583 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-03-24T00:28:31.611 INFO:teuthology.orchestra.run.smithi053.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-24T00:28:31.667 INFO:teuthology.orchestra.run.smithi053.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-03-24T00:28:31.707 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up cephadm (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:31.895 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-03-24T00:28:32.021 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-03-24T00:28:32.226 INFO:teuthology.orchestra.run.smithi053.stdout:Adding system user cephadm....done 2024-03-24T00:28:32.294 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-03-24T00:28:32.388 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-24T00:28:32.689 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-24T00:28:32.747 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-03-24T00:28:32.957 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-24T00:28:33.108 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-24T00:28:33.368 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-24T00:28:33.396 INFO:teuthology.orchestra.run.smithi040.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-03-24T00:28:33.441 INFO:teuthology.orchestra.run.smithi040.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-03-24T00:28:33.644 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-24T00:28:33.770 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-24T00:28:33.949 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up cephadm (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:34.097 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-24T00:28:34.457 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-24T00:28:34.585 INFO:teuthology.orchestra.run.smithi040.stdout:Adding system user cephadm....done 2024-03-24T00:28:34.741 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-24T00:28:34.746 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-03-24T00:28:35.009 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-24T00:28:35.064 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-03-24T00:28:35.335 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-24T00:28:35.357 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-03-24T00:28:35.508 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-03-24T00:28:35.562 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-24T00:28:35.793 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-03-24T00:28:35.871 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-24T00:28:36.061 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-03-24T00:28:36.131 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-24T00:28:36.229 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-py (1.10.0-1) ... 2024-03-24T00:28:36.398 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-ceph-argparse (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:36.572 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-03-24T00:28:36.675 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-24T00:28:36.817 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-24T00:28:36.952 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-24T00:28:36.974 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-03-24T00:28:37.219 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-24T00:28:37.300 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-03-24T00:28:37.320 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-24T00:28:37.596 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-24T00:28:37.651 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-03-24T00:28:37.939 INFO:teuthology.orchestra.run.smithi053.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-24T00:28:37.952 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-03-24T00:28:37.969 INFO:teuthology.orchestra.run.smithi053.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-24T00:28:38.025 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-24T00:28:38.151 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-24T00:28:38.237 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-03-24T00:28:38.522 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-03-24T00:28:38.535 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-24T00:28:38.661 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-24T00:28:38.823 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-03-24T00:28:38.938 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-24T00:28:39.107 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-ceph-argparse (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:39.247 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-24T00:28:39.367 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-03-24T00:28:39.501 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-03-24T00:28:39.548 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up zip (3.0-12build2) ... 2024-03-24T00:28:39.661 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-03-24T00:28:39.674 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-24T00:28:39.945 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-03-24T00:28:40.072 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-03-24T00:28:40.347 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-03-24T00:28:40.418 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-24T00:28:40.711 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-24T00:28:40.753 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-03-24T00:28:40.803 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-03-24T00:28:40.837 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-24T00:28:40.868 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-03-24T00:28:40.938 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-24T00:28:40.993 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-03-24T00:28:41.264 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-24T00:28:41.403 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-03-24T00:28:41.529 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-03-24T00:28:41.673 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-24T00:28:41.805 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-03-24T00:28:41.933 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-24T00:28:42.090 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-03-24T00:28:42.368 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-24T00:28:42.441 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up zip (3.0-12build2) ... 2024-03-24T00:28:42.542 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-03-24T00:28:42.728 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-24T00:28:43.045 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-24T00:28:43.286 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-03-24T00:28:43.388 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-24T00:28:43.587 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-03-24T00:28:43.664 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-24T00:28:43.738 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-03-24T00:28:43.791 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-ceph-common (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:43.889 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-03-24T00:28:44.108 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-idna (3.3-1) ... 2024-03-24T00:28:44.265 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-03-24T00:28:44.493 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-24T00:28:44.674 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-03-24T00:28:44.951 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-03-24T00:28:45.353 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-03-24T00:28:45.708 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-03-24T00:28:45.774 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-24T00:28:46.017 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T00:28:46.055 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-03-24T00:28:46.176 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-24T00:28:46.440 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-03-24T00:28:46.461 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-24T00:28:46.570 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-24T00:28:46.671 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-24T00:28:46.699 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-03-24T00:28:46.842 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-ceph-common (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:46.964 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-24T00:28:47.169 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-idna (3.3-1) ... 2024-03-24T00:28:47.549 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-24T00:28:47.553 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-03-24T00:28:47.817 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T00:28:47.944 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-24T00:28:48.286 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-24T00:28:48.546 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-24T00:28:48.805 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-24T00:28:48.853 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-03-24T00:28:49.056 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-24T00:28:49.171 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T00:28:49.315 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-24T00:28:49.330 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-toml (0.10.2-1) ... 2024-03-24T00:28:49.600 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-24T00:28:49.623 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-03-24T00:28:49.724 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-03-24T00:28:49.725 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-24T00:28:49.825 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-03-24T00:28:50.027 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-24T00:28:50.128 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-24T00:28:50.135 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-03-24T00:28:50.404 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-24T00:28:50.428 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-03-24T00:28:50.721 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T00:28:50.722 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-24T00:28:50.888 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-03-24T00:28:51.014 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-24T00:28:51.140 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-24T00:28:51.215 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-03-24T00:28:51.266 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-24T00:28:51.355 INFO:teuthology.orchestra.run.smithi053.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-24T00:28:51.493 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-03-24T00:28:51.602 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-24T00:28:51.769 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-24T00:28:51.784 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-03-24T00:28:52.060 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-03-24T00:28:52.171 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-24T00:28:52.328 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-03-24T00:28:52.455 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-24T00:28:52.564 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-24T00:28:52.637 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-03-24T00:28:52.797 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-03-24T00:28:52.833 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T00:28:52.958 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-24T00:28:53.123 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-03-24T00:28:53.274 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-03-24T00:28:53.292 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-24T00:28:53.609 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-03-24T00:28:53.619 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-24T00:28:53.878 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-24T00:28:53.943 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-03-24T00:28:54.115 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-24T00:28:54.211 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-03-24T00:28:54.387 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-03-24T00:28:54.399 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-24T00:28:54.538 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-03-24T00:28:54.643 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-03-24T00:28:54.701 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-24T00:28:54.898 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-03-24T00:28:54.986 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-24T00:28:55.049 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-03-24T00:28:55.371 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-24T00:28:55.484 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-03-24T00:28:55.680 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-24T00:28:55.760 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up lua-any (27ubuntu1) ... 2024-03-24T00:28:55.886 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-portend (3.0.0-1) ... 2024-03-24T00:28:56.033 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-24T00:28:56.159 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up librados2 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:56.162 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-03-24T00:28:56.288 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-03-24T00:28:56.318 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:56.428 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-24T00:28:56.623 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-03-24T00:28:56.722 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libcephfs2 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:56.848 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libradosstriper1 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:56.957 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-03-24T00:28:57.001 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-24T00:28:57.233 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-03-24T00:28:57.301 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up librbd1 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:57.427 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-modules-core (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:57.470 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rsa (4.8-1) ... 2024-03-24T00:28:57.570 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-fuse (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:57.780 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-24T00:28:57.780 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-24T00:28:57.805 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-03-24T00:28:58.098 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-03-24T00:28:58.365 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-03-24T00:28:58.423 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libcephfs-dev (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:58.589 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-rados (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:58.706 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up librgw2 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:58.751 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-03-24T00:28:58.832 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up cephfs-top (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:58.959 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-24T00:28:59.052 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-03-24T00:28:59.430 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-03-24T00:28:59.614 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librados2 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:59.715 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:28:59.816 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-03-24T00:29:00.152 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libcephfs2 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:00.278 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libradosstriper1 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:00.296 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-rbd (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:00.397 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up rbd-fuse (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:00.437 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-03-24T00:29:00.498 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-rgw (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:00.657 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up python3-cephfs (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:00.747 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librbd1 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:00.784 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up cephfs-shell (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:00.898 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-modules-core (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:00.925 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-common (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:01.049 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-fuse (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:01.288 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-24T00:29:01.288 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-03-24T00:29:01.471 INFO:teuthology.orchestra.run.smithi053.stdout:Adding group ceph....done 2024-03-24T00:29:01.847 INFO:teuthology.orchestra.run.smithi053.stdout:Adding system user ceph....done 2024-03-24T00:29:01.934 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libcephfs-dev (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:02.016 INFO:teuthology.orchestra.run.smithi053.stdout:Setting system user ceph properties....done 2024-03-24T00:29:02.033 INFO:teuthology.orchestra.run.smithi053.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-24T00:29:02.060 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rados (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:02.185 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-24T00:29:02.220 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librgw2 (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:02.345 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up cephfs-top (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:02.472 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-03-24T00:29:02.583 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-24T00:29:03.269 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up cephfs-mirror (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:03.469 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:29:03.469 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:29:03.875 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rbd (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:04.018 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up rbd-fuse (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:04.071 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-test (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:04.154 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rgw (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:04.172 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up radosgw (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:04.279 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cephfs (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:04.405 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up cephfs-shell (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:04.530 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-common (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:04.679 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:29:04.679 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:29:05.067 INFO:teuthology.orchestra.run.smithi040.stdout:Adding group ceph....done 2024-03-24T00:29:05.324 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-base (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:05.452 INFO:teuthology.orchestra.run.smithi040.stdout:Adding system user ceph....done 2024-03-24T00:29:05.688 INFO:teuthology.orchestra.run.smithi040.stdout:Setting system user ceph properties....done 2024-03-24T00:29:05.691 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-24T00:29:05.707 INFO:teuthology.orchestra.run.smithi040.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-03-24T00:29:05.865 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-03-24T00:29:06.271 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-03-24T00:29:06.350 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mds (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:06.580 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-24T00:29:06.580 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-24T00:29:06.924 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up cephfs-mirror (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:07.140 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:29:07.140 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:29:07.221 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:07.448 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-24T00:29:07.448 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-24T00:29:07.800 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-test (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:07.909 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up radosgw (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:08.088 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-osd (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:08.384 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-24T00:29:08.384 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-24T00:29:08.702 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:29:08.702 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:29:09.031 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:09.157 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:09.300 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mon (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:09.331 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-base (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:09.516 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-24T00:29:09.516 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-24T00:29:09.695 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-03-24T00:29:10.159 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-cephadm (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:10.331 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mds (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:10.343 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:10.444 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-mgr-dashboard (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:10.544 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-24T00:29:10.544 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-03-24T00:29:10.570 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ceph-volume (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:10.940 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-24T00:29:11.074 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-24T00:29:11.166 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:11.226 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-24T00:29:11.403 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-24T00:29:11.403 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-03-24T00:29:12.067 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-osd (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:12.352 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-24T00:29:12.367 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-24T00:29:12.367 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-03-24T00:29:13.010 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:13.089 INFO:teuthology.orchestra.run.smithi053.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-24T00:29:13.094 DEBUG:teuthology.orchestra.run.smithi053:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-24T00:29:13.128 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:13.170 INFO:teuthology.orchestra.run.smithi053.stdout:Reading package lists... 2024-03-24T00:29:13.271 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mon (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:13.346 INFO:teuthology.orchestra.run.smithi053.stdout:Building dependency tree... 2024-03-24T00:29:13.346 INFO:teuthology.orchestra.run.smithi053.stdout:Reading state information... 2024-03-24T00:29:13.488 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-24T00:29:13.488 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-03-24T00:29:13.510 INFO:teuthology.orchestra.run.smithi053.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-24T00:29:13.511 INFO:teuthology.orchestra.run.smithi053.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-24T00:29:13.511 INFO:teuthology.orchestra.run.smithi053.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-03-24T00:29:13.511 INFO:teuthology.orchestra.run.smithi053.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-03-24T00:29:13.511 INFO:teuthology.orchestra.run.smithi053.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-24T00:29:13.511 INFO:teuthology.orchestra.run.smithi053.stdout:The following packages were automatically installed and are no longer required: 2024-03-24T00:29:13.511 INFO:teuthology.orchestra.run.smithi053.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-24T00:29:13.511 INFO:teuthology.orchestra.run.smithi053.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-24T00:29:13.511 INFO:teuthology.orchestra.run.smithi053.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-24T00:29:13.512 INFO:teuthology.orchestra.run.smithi053.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-24T00:29:13.512 INFO:teuthology.orchestra.run.smithi053.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-24T00:29:13.512 INFO:teuthology.orchestra.run.smithi053.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-24T00:29:13.513 INFO:teuthology.orchestra.run.smithi053.stdout:The following additional packages will be installed: 2024-03-24T00:29:13.513 INFO:teuthology.orchestra.run.smithi053.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-24T00:29:13.513 INFO:teuthology.orchestra.run.smithi053.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-24T00:29:13.513 INFO:teuthology.orchestra.run.smithi053.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-03-24T00:29:13.513 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-24T00:29:13.513 INFO:teuthology.orchestra.run.smithi053.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-24T00:29:13.514 INFO:teuthology.orchestra.run.smithi053.stdout:Suggested packages: 2024-03-24T00:29:13.514 INFO:teuthology.orchestra.run.smithi053.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-24T00:29:13.514 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-03-24T00:29:13.514 INFO:teuthology.orchestra.run.smithi053.stdout: libengine-pkcs11-openssl 2024-03-24T00:29:13.566 INFO:teuthology.orchestra.run.smithi053.stdout:The following NEW packages will be installed: 2024-03-24T00:29:13.570 INFO:teuthology.orchestra.run.smithi053.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-24T00:29:13.570 INFO:teuthology.orchestra.run.smithi053.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-24T00:29:13.570 INFO:teuthology.orchestra.run.smithi053.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-24T00:29:13.570 INFO:teuthology.orchestra.run.smithi053.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-24T00:29:13.570 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-24T00:29:13.570 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-24T00:29:13.570 INFO:teuthology.orchestra.run.smithi053.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-24T00:29:13.571 INFO:teuthology.orchestra.run.smithi053.stdout:The following packages will be upgraded: 2024-03-24T00:29:13.571 INFO:teuthology.orchestra.run.smithi053.stdout: libssl3 2024-03-24T00:29:13.629 INFO:teuthology.orchestra.run.smithi053.stdout:1 upgraded, 39 newly installed, 0 to remove and 287 not upgraded. 2024-03-24T00:29:13.629 INFO:teuthology.orchestra.run.smithi053.stdout:Need to get 54.6 MB of archives. 2024-03-24T00:29:13.629 INFO:teuthology.orchestra.run.smithi053.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-24T00:29:13.630 INFO:teuthology.orchestra.run.smithi053.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-24T00:29:13.749 INFO:teuthology.orchestra.run.smithi053.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-24T00:29:13.816 INFO:teuthology.orchestra.run.smithi053.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-24T00:29:13.824 INFO:teuthology.orchestra.run.smithi053.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-24T00:29:13.825 INFO:teuthology.orchestra.run.smithi053.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-24T00:29:13.826 INFO:teuthology.orchestra.run.smithi053.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-24T00:29:13.828 INFO:teuthology.orchestra.run.smithi053.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-24T00:29:13.828 INFO:teuthology.orchestra.run.smithi053.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-24T00:29:13.829 INFO:teuthology.orchestra.run.smithi053.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-24T00:29:13.830 INFO:teuthology.orchestra.run.smithi053.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-24T00:29:13.830 INFO:teuthology.orchestra.run.smithi053.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-24T00:29:13.837 INFO:teuthology.orchestra.run.smithi053.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-24T00:29:13.850 INFO:teuthology.orchestra.run.smithi053.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-24T00:29:14.081 INFO:teuthology.orchestra.run.smithi053.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-24T00:29:14.083 INFO:teuthology.orchestra.run.smithi053.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-24T00:29:14.083 INFO:teuthology.orchestra.run.smithi053.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-24T00:29:14.086 INFO:teuthology.orchestra.run.smithi053.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-24T00:29:14.086 INFO:teuthology.orchestra.run.smithi053.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-24T00:29:14.091 INFO:teuthology.orchestra.run.smithi053.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-24T00:29:14.092 INFO:teuthology.orchestra.run.smithi053.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-24T00:29:14.100 INFO:teuthology.orchestra.run.smithi053.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-24T00:29:14.100 INFO:teuthology.orchestra.run.smithi053.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-24T00:29:14.122 INFO:teuthology.orchestra.run.smithi053.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-24T00:29:14.122 INFO:teuthology.orchestra.run.smithi053.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-24T00:29:14.130 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-cephadm (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:14.133 INFO:teuthology.orchestra.run.smithi053.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-24T00:29:14.153 INFO:teuthology.orchestra.run.smithi053.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-24T00:29:14.156 INFO:teuthology.orchestra.run.smithi053.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-24T00:29:14.157 INFO:teuthology.orchestra.run.smithi053.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-24T00:29:14.157 INFO:teuthology.orchestra.run.smithi053.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-24T00:29:14.158 INFO:teuthology.orchestra.run.smithi053.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-24T00:29:14.176 INFO:teuthology.orchestra.run.smithi053.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-24T00:29:14.178 INFO:teuthology.orchestra.run.smithi053.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-24T00:29:14.326 INFO:teuthology.orchestra.run.smithi053.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-24T00:29:14.326 INFO:teuthology.orchestra.run.smithi053.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-24T00:29:14.326 INFO:teuthology.orchestra.run.smithi053.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-24T00:29:14.326 INFO:teuthology.orchestra.run.smithi053.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-24T00:29:14.327 INFO:teuthology.orchestra.run.smithi053.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-24T00:29:14.327 INFO:teuthology.orchestra.run.smithi053.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-24T00:29:14.332 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:14.340 INFO:teuthology.orchestra.run.smithi053.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-24T00:29:14.340 INFO:teuthology.orchestra.run.smithi053.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-24T00:29:14.491 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-dashboard (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:14.625 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-volume (19.0.0-2145-g3be9a5c9-1jammy) ... 2024-03-24T00:29:14.733 INFO:teuthology.orchestra.run.smithi053.stdout:Preconfiguring packages ... 2024-03-24T00:29:14.893 INFO:teuthology.orchestra.run.smithi053.stdout:Fetched 54.6 MB in 1s (71.5 MB/s) 2024-03-24T00:29:14.979 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-03-24T00:29:15.034 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package flex. 2024-03-24T00:29:15.095 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-24T00:29:15.106 INFO:teuthology.orchestra.run.smithi053.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140586 files and directories currently installed.) 2024-03-24T00:29:15.110 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-24T00:29:15.149 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-24T00:29:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-24T00:29:15.746 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-24T00:29:15.889 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-24T00:29:16.288 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-24T00:29:16.482 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-24T00:29:16.873 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package bison. 2024-03-24T00:29:16.950 INFO:teuthology.orchestra.run.smithi053.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140659 files and directories currently installed.) 2024-03-24T00:29:16.953 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-24T00:29:16.992 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-24T00:29:17.305 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-24T00:29:17.309 DEBUG:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-03-24T00:29:17.376 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-24T00:29:17.385 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2024-03-24T00:29:17.391 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-24T00:29:17.434 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-24T00:29:17.560 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2024-03-24T00:29:17.561 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2024-03-24T00:29:17.726 INFO:teuthology.orchestra.run.smithi040.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-03-24T00:29:17.726 INFO:teuthology.orchestra.run.smithi040.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-03-24T00:29:17.726 INFO:teuthology.orchestra.run.smithi040.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2024-03-24T00:29:17.726 INFO:teuthology.orchestra.run.smithi040.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2024-03-24T00:29:17.726 INFO:teuthology.orchestra.run.smithi040.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-03-24T00:29:17.726 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2024-03-24T00:29:17.726 INFO:teuthology.orchestra.run.smithi040.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-03-24T00:29:17.727 INFO:teuthology.orchestra.run.smithi040.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-03-24T00:29:17.727 INFO:teuthology.orchestra.run.smithi040.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-03-24T00:29:17.727 INFO:teuthology.orchestra.run.smithi040.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-03-24T00:29:17.727 INFO:teuthology.orchestra.run.smithi040.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-03-24T00:29:17.727 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2024-03-24T00:29:17.728 INFO:teuthology.orchestra.run.smithi040.stdout:The following additional packages will be installed: 2024-03-24T00:29:17.728 INFO:teuthology.orchestra.run.smithi040.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-03-24T00:29:17.728 INFO:teuthology.orchestra.run.smithi040.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-03-24T00:29:17.728 INFO:teuthology.orchestra.run.smithi040.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2024-03-24T00:29:17.728 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-03-24T00:29:17.728 INFO:teuthology.orchestra.run.smithi040.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-24T00:29:17.729 INFO:teuthology.orchestra.run.smithi040.stdout:Suggested packages: 2024-03-24T00:29:17.729 INFO:teuthology.orchestra.run.smithi040.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-03-24T00:29:17.729 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2024-03-24T00:29:17.729 INFO:teuthology.orchestra.run.smithi040.stdout: libengine-pkcs11-openssl 2024-03-24T00:29:17.779 INFO:teuthology.orchestra.run.smithi040.stdout:The following NEW packages will be installed: 2024-03-24T00:29:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-03-24T00:29:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-03-24T00:29:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-03-24T00:29:17.780 INFO:teuthology.orchestra.run.smithi040.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-03-24T00:29:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-03-24T00:29:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-03-24T00:29:17.781 INFO:teuthology.orchestra.run.smithi040.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-03-24T00:29:17.781 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be upgraded: 2024-03-24T00:29:17.782 INFO:teuthology.orchestra.run.smithi040.stdout: libssl3 2024-03-24T00:29:17.804 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package dump. 2024-03-24T00:29:17.820 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-24T00:29:17.862 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking dump (0.4b47-3) ... 2024-03-24T00:29:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:1 upgraded, 39 newly installed, 0 to remove and 287 not upgraded. 2024-03-24T00:29:17.966 INFO:teuthology.orchestra.run.smithi040.stdout:Need to get 54.6 MB of archives. 2024-03-24T00:29:17.967 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 202 MB of additional disk space will be used. 2024-03-24T00:29:17.967 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-03-24T00:29:18.149 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package indent. 2024-03-24T00:29:18.164 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-24T00:29:18.199 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-24T00:29:18.461 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-03-24T00:29:18.622 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-24T00:29:18.637 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-24T00:29:18.680 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-24T00:29:18.720 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-03-24T00:29:18.754 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-03-24T00:29:18.756 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-03-24T00:29:18.761 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-03-24T00:29:18.765 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-03-24T00:29:18.765 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-03-24T00:29:18.768 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-03-24T00:29:18.770 INFO:teuthology.orchestra.run.smithi040.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-03-24T00:29:18.771 INFO:teuthology.orchestra.run.smithi040.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-03-24T00:29:18.935 INFO:teuthology.orchestra.run.smithi040.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-03-24T00:29:18.967 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-24T00:29:18.984 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-24T00:29:19.017 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-24T00:29:19.268 INFO:teuthology.orchestra.run.smithi040.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-03-24T00:29:19.410 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-24T00:29:19.410 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-24T00:29:19.410 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-24T00:29:19.690 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-24T00:29:19.710 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-24T00:29:19.748 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-24T00:29:20.026 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-24T00:29:20.043 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-24T00:29:20.084 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-24T00:29:20.312 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libjson-perl. 2024-03-24T00:29:20.328 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-24T00:29:20.329 INFO:teuthology.orchestra.run.smithi040.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-03-24T00:29:20.331 INFO:teuthology.orchestra.run.smithi040.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-03-24T00:29:20.331 INFO:teuthology.orchestra.run.smithi040.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-03-24T00:29:20.334 INFO:teuthology.orchestra.run.smithi040.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-03-24T00:29:20.335 INFO:teuthology.orchestra.run.smithi040.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-03-24T00:29:20.353 INFO:teuthology.orchestra.run.smithi040.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-03-24T00:29:20.354 INFO:teuthology.orchestra.run.smithi040.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-03-24T00:29:20.362 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-24T00:29:20.364 INFO:teuthology.orchestra.run.smithi040.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-03-24T00:29:20.364 INFO:teuthology.orchestra.run.smithi040.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-03-24T00:29:20.604 INFO:teuthology.orchestra.run.smithi040.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-03-24T00:29:20.686 INFO:teuthology.orchestra.run.smithi040.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-03-24T00:29:20.706 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-24T00:29:20.723 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-24T00:29:20.765 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-24T00:29:21.160 INFO:teuthology.orchestra.run.smithi040.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-03-24T00:29:21.290 INFO:teuthology.orchestra.run.smithi040.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-03-24T00:29:21.300 INFO:teuthology.orchestra.run.smithi040.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-03-24T00:29:21.301 INFO:teuthology.orchestra.run.smithi040.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-03-24T00:29:21.302 INFO:teuthology.orchestra.run.smithi040.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-03-24T00:29:21.304 INFO:teuthology.orchestra.run.smithi040.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-03-24T00:29:21.337 INFO:teuthology.orchestra.run.smithi040.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-03-24T00:29:21.341 INFO:teuthology.orchestra.run.smithi040.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-03-24T00:29:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-03-24T00:29:21.943 INFO:teuthology.orchestra.run.smithi040.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-03-24T00:29:22.025 INFO:teuthology.orchestra.run.smithi040.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-03-24T00:29:22.025 INFO:teuthology.orchestra.run.smithi040.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-03-24T00:29:22.110 INFO:teuthology.orchestra.run.smithi040.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-03-24T00:29:22.117 INFO:teuthology.orchestra.run.smithi040.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-03-24T00:29:22.126 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-24T00:29:22.142 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-24T00:29:22.184 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-24T00:29:22.454 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-24T00:29:22.470 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-24T00:29:22.512 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-24T00:29:22.584 INFO:teuthology.orchestra.run.smithi040.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-03-24T00:29:22.586 INFO:teuthology.orchestra.run.smithi040.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-03-24T00:29:22.823 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-24T00:29:22.839 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-24T00:29:22.881 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-24T00:29:22.993 INFO:teuthology.orchestra.run.smithi040.stdout:Preconfiguring packages ... 2024-03-24T00:29:23.166 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 54.6 MB in 5s (11.4 MB/s) 2024-03-24T00:29:23.234 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-24T00:29:23.250 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-24T00:29:23.292 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-24T00:29:23.341 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package flex. 2024-03-24T00:29:23.414 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140586 files and directories currently installed.) 2024-03-24T00:29:23.417 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-03-24T00:29:23.464 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking flex (2.6.4-8build2) ... 2024-03-24T00:29:23.696 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-24T00:29:23.712 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-24T00:29:23.754 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-24T00:29:24.090 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-24T00:29:24.107 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-24T00:29:24.144 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-24T00:29:24.148 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-24T00:29:24.354 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-03-24T00:29:24.501 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-24T00:29:24.518 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-24T00:29:24.559 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-24T00:29:24.853 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-24T00:29:24.879 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-24T00:29:24.894 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-24T00:29:24.937 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-24T00:29:25.173 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-24T00:29:25.190 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-24T00:29:25.223 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-24T00:29:25.422 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package bison. 2024-03-24T00:29:25.495 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140659 files and directories currently installed.) 2024-03-24T00:29:25.499 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-03-24T00:29:25.549 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-03-24T00:29:25.693 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-24T00:29:25.709 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-24T00:29:25.743 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-24T00:29:26.000 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-03-24T00:29:26.014 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-03-24T00:29:26.058 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-03-24T00:29:26.088 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package modemmanager. 2024-03-24T00:29:26.104 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-24T00:29:26.146 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-24T00:29:26.353 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package dump. 2024-03-24T00:29:26.368 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-03-24T00:29:26.402 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking dump (0.4b47-3) ... 2024-03-24T00:29:26.706 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package indent. 2024-03-24T00:29:26.722 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-03-24T00:29:26.764 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-24T00:29:26.786 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package network-manager. 2024-03-24T00:29:26.802 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-24T00:29:26.852 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-24T00:29:27.034 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-03-24T00:29:27.049 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-03-24T00:29:27.092 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-24T00:29:27.462 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-03-24T00:29:27.479 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-03-24T00:29:27.493 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package ppp. 2024-03-24T00:29:27.509 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-24T00:29:27.520 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-24T00:29:27.765 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-03-24T00:29:27.781 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-03-24T00:29:27.814 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-03-24T00:29:27.905 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-24T00:29:28.168 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libfl2:amd64. 2024-03-24T00:29:28.184 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-03-24T00:29:28.223 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package pptp-linux. 2024-03-24T00:29:28.229 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-03-24T00:29:28.238 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-24T00:29:28.281 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-24T00:29:28.520 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-03-24T00:29:28.537 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-03-24T00:29:28.578 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-24T00:29:28.619 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-24T00:29:28.634 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-24T00:29:28.677 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-24T00:29:28.831 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libjson-perl. 2024-03-24T00:29:28.847 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-03-24T00:29:28.881 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-03-24T00:29:29.022 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-24T00:29:29.038 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-24T00:29:29.088 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking postgresql-client-common (238) ... 2024-03-24T00:29:29.167 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libllvm14:amd64. 2024-03-24T00:29:29.183 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-03-24T00:29:29.217 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-24T00:29:29.459 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-24T00:29:29.476 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-24T00:29:29.642 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-24T00:29:30.264 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package postgresql-common. 2024-03-24T00:29:30.281 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-24T00:29:30.342 INFO:teuthology.orchestra.run.smithi053.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-24T00:29:30.371 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking postgresql-common (238) ... 2024-03-24T00:29:30.646 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-03-24T00:29:30.662 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-24T00:29:30.704 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-24T00:29:30.772 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package postgresql-14. 2024-03-24T00:29:30.788 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-24T00:29:30.858 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-24T00:29:30.973 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libmbim-proxy. 2024-03-24T00:29:30.990 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-03-24T00:29:31.031 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-24T00:29:31.343 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-03-24T00:29:31.359 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-24T00:29:31.401 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-24T00:29:31.812 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libndp0:amd64. 2024-03-24T00:29:31.828 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-03-24T00:29:31.879 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-24T00:29:31.967 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package postgresql. 2024-03-24T00:29:31.982 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-24T00:29:32.025 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking postgresql (14+238) ... 2024-03-24T00:29:32.244 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package postgresql-client. 2024-03-24T00:29:32.259 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-24T00:29:32.298 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libnm0:amd64. 2024-03-24T00:29:32.314 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-03-24T00:29:32.326 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking postgresql-client (14+238) ... 2024-03-24T00:29:32.365 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-24T00:29:32.530 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-24T00:29:32.545 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-24T00:29:32.596 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-24T00:29:32.784 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libpq5:amd64. 2024-03-24T00:29:32.800 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-24T00:29:32.876 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-24T00:29:32.991 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package tcl8.6. 2024-03-24T00:29:33.007 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-24T00:29:33.082 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-24T00:29:33.246 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-03-24T00:29:33.262 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-24T00:29:33.312 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-24T00:29:33.427 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package tcl. 2024-03-24T00:29:33.444 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-24T00:29:33.498 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-24T00:29:33.623 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqmi-proxy. 2024-03-24T00:29:33.639 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-03-24T00:29:33.681 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-24T00:29:33.942 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-03-24T00:29:33.958 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-03-24T00:29:34.000 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-24T00:29:34.432 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package wpasupplicant. 2024-03-24T00:29:34.448 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-24T00:29:34.498 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-24T00:29:34.529 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-03-24T00:29:34.545 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-03-24T00:29:34.612 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-03-24T00:29:34.961 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-24T00:29:34.964 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package modemmanager. 2024-03-24T00:29:34.977 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-24T00:29:34.981 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-03-24T00:29:35.023 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-24T00:29:35.027 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-24T00:29:35.456 INFO:teuthology.orchestra.run.smithi053.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-24T00:29:35.472 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-24T00:29:35.523 INFO:teuthology.orchestra.run.smithi053.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-24T00:29:35.805 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package network-manager. 2024-03-24T00:29:35.822 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-03-24T00:29:35.851 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up postgresql-client-common (238) ... 2024-03-24T00:29:35.872 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-03-24T00:29:36.062 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-24T00:29:36.163 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-24T00:29:36.298 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-24T00:29:36.424 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-24T00:29:36.537 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ppp. 2024-03-24T00:29:36.550 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-24T00:29:36.554 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-03-24T00:29:36.676 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-24T00:29:36.913 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-03-24T00:29:37.250 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package pptp-linux. 2024-03-24T00:29:37.265 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-03-24T00:29:37.300 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-03-24T00:29:37.486 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package network-manager-pptp. 2024-03-24T00:29:37.501 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-03-24T00:29:37.535 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-03-24T00:29:37.646 INFO:teuthology.orchestra.run.smithi053.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-24T00:29:37.782 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-24T00:29:37.906 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package postgresql-client-common. 2024-03-24T00:29:37.922 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-03-24T00:29:37.964 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking postgresql-client-common (238) ... 2024-03-24T00:29:38.154 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-24T00:29:38.154 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-24T00:29:38.276 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package postgresql-client-14. 2024-03-24T00:29:38.292 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-24T00:29:38.334 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-24T00:29:38.830 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package postgresql-common. 2024-03-24T00:29:38.846 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-03-24T00:29:38.885 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-24T00:29:38.899 INFO:teuthology.orchestra.run.smithi040.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-03-24T00:29:38.937 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking postgresql-common (238) ... 2024-03-24T00:29:39.020 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-24T00:29:39.154 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-24T00:29:39.339 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-24T00:29:39.421 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package postgresql-14. 2024-03-24T00:29:39.437 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-03-24T00:29:39.473 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-24T00:29:39.482 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-24T00:29:39.591 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-24T00:29:39.700 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-24T00:29:40.643 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-24T00:29:40.870 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up dump (0.4b47-3) ... 2024-03-24T00:29:41.049 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package postgresql. 2024-03-24T00:29:41.065 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-03-24T00:29:41.082 INFO:teuthology.orchestra.run.smithi053.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-24T00:29:41.107 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking postgresql (14+238) ... 2024-03-24T00:29:41.190 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-24T00:29:41.334 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package postgresql-client. 2024-03-24T00:29:41.350 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-03-24T00:29:41.392 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking postgresql-client (14+238) ... 2024-03-24T00:29:41.466 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-24T00:29:41.670 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package postgresql-contrib. 2024-03-24T00:29:41.686 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-03-24T00:29:41.728 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking postgresql-contrib (14+238) ... 2024-03-24T00:29:41.736 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-24T00:29:41.947 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-24T00:29:42.006 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package tcl8.6. 2024-03-24T00:29:42.022 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-03-24T00:29:42.055 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-24T00:29:42.081 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-24T00:29:42.172 INFO:teuthology.orchestra.run.smithi053.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-24T00:29:42.258 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-24T00:29:42.308 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package tcl. 2024-03-24T00:29:42.325 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-03-24T00:29:42.367 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-24T00:29:42.405 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-03-24T00:29:42.527 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-24T00:29:42.653 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-24T00:29:42.753 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package wpasupplicant. 2024-03-24T00:29:42.769 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-03-24T00:29:42.779 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-24T00:29:42.811 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-24T00:29:42.972 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up postgresql-client (14+238) ... 2024-03-24T00:29:43.098 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-24T00:29:43.151 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-03-24T00:29:43.169 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-03-24T00:29:43.232 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-03-24T00:29:43.612 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package usb-modeswitch. 2024-03-24T00:29:43.629 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-03-24T00:29:43.657 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:29:43.657 INFO:teuthology.orchestra.run.smithi053.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-24T00:29:43.658 INFO:teuthology.orchestra.run.smithi053.stdout:which means they are currently configured by ifupdown: 2024-03-24T00:29:43.658 INFO:teuthology.orchestra.run.smithi053.stdout:- ens1f0 2024-03-24T00:29:43.658 INFO:teuthology.orchestra.run.smithi053.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-24T00:29:43.658 INFO:teuthology.orchestra.run.smithi053.stdout:remove their configuration from /etc/network/interfaces. 2024-03-24T00:29:43.658 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:29:43.678 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-24T00:29:43.797 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-24T00:29:43.974 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up postgresql-client-common (238) ... 2024-03-24T00:29:44.193 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up flex (2.6.4-8build2) ... 2024-03-24T00:29:44.223 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-24T00:29:44.336 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-03-24T00:29:44.462 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-03-24T00:29:44.588 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-03-24T00:29:44.643 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-24T00:29:44.714 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-03-24T00:29:44.840 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-24T00:29:45.343 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-24T00:29:45.486 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-24T00:29:45.587 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-24T00:29:45.688 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up postgresql-common (238) ... 2024-03-24T00:29:45.801 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-03-24T00:29:45.930 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-03-24T00:29:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-24T00:29:46.316 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-03-24T00:29:46.568 INFO:teuthology.orchestra.run.smithi053.stdout:Adding user postgres to group ssl-cert 2024-03-24T00:29:46.862 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:29:46.863 INFO:teuthology.orchestra.run.smithi053.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-24T00:29:47.033 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-03-24T00:29:47.117 INFO:teuthology.orchestra.run.smithi053.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-24T00:29:47.117 INFO:teuthology.orchestra.run.smithi053.stdout:Removing obsolete dictionary files: 2024-03-24T00:29:47.134 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-03-24T00:29:47.243 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-03-24T00:29:47.369 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-03-24T00:29:47.495 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-03-24T00:29:47.654 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-03-24T00:29:47.789 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-03-24T00:29:47.834 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-24T00:29:48.315 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-24T00:29:48.444 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-03-24T00:29:48.570 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up dump (0.4b47-3) ... 2024-03-24T00:29:48.665 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-03-24T00:29:48.676 INFO:teuthology.orchestra.run.smithi053.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-24T00:29:48.694 INFO:teuthology.orchestra.run.smithi053.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-24T00:29:48.730 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-03-24T00:29:48.743 INFO:teuthology.orchestra.run.smithi053.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-24T00:29:48.743 INFO:teuthology.orchestra.run.smithi053.stdout:This user must also own the server process. 2024-03-24T00:29:48.744 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:29:48.744 INFO:teuthology.orchestra.run.smithi053.stdout:The database cluster will be initialized with locale "en_US". 2024-03-24T00:29:48.744 INFO:teuthology.orchestra.run.smithi053.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-24T00:29:48.744 INFO:teuthology.orchestra.run.smithi053.stdout:The default text search configuration will be set to "english". 2024-03-24T00:29:48.744 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:29:48.744 INFO:teuthology.orchestra.run.smithi053.stdout:Data page checksums are disabled. 2024-03-24T00:29:48.744 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:29:48.744 INFO:teuthology.orchestra.run.smithi053.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-24T00:29:48.745 INFO:teuthology.orchestra.run.smithi053.stdout:creating subdirectories ... ok 2024-03-24T00:29:48.745 INFO:teuthology.orchestra.run.smithi053.stdout:selecting dynamic shared memory implementation ... posix 2024-03-24T00:29:48.768 INFO:teuthology.orchestra.run.smithi053.stdout:selecting default max_connections ... 100 2024-03-24T00:29:48.804 INFO:teuthology.orchestra.run.smithi053.stdout:selecting default shared_buffers ... 128MB 2024-03-24T00:29:48.818 INFO:teuthology.orchestra.run.smithi053.stdout:selecting default time zone ... Etc/UTC 2024-03-24T00:29:48.819 INFO:teuthology.orchestra.run.smithi053.stdout:creating configuration files ... ok 2024-03-24T00:29:48.856 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-03-24T00:29:48.983 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up tcl (8.6.11+1build2) ... 2024-03-24T00:29:48.992 INFO:teuthology.orchestra.run.smithi053.stdout:running bootstrap script ... ok 2024-03-24T00:29:49.108 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libjson-perl (4.04000-1) ... 2024-03-24T00:29:49.235 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-03-24T00:29:49.349 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-03-24T00:29:49.419 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-03-24T00:29:49.528 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-03-24T00:29:49.622 INFO:teuthology.orchestra.run.smithi053.stdout:performing post-bootstrap initialization ... ok 2024-03-24T00:29:49.680 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-03-24T00:29:49.815 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-03-24T00:29:49.941 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-03-24T00:29:50.109 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up postgresql-client (14+238) ... 2024-03-24T00:29:50.156 INFO:teuthology.orchestra.run.smithi053.stdout:syncing data to disk ... ok 2024-03-24T00:29:50.260 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-03-24T00:29:50.674 INFO:teuthology.orchestra.run.smithi053.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-24T00:29:50.837 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:29:50.838 INFO:teuthology.orchestra.run.smithi040.stdout:The following network interfaces were found in /etc/network/interfaces 2024-03-24T00:29:50.838 INFO:teuthology.orchestra.run.smithi040.stdout:which means they are currently configured by ifupdown: 2024-03-24T00:29:50.838 INFO:teuthology.orchestra.run.smithi040.stdout:- ens1f0 2024-03-24T00:29:50.838 INFO:teuthology.orchestra.run.smithi040.stdout:If you want to manage those interfaces with NetworkManager instead 2024-03-24T00:29:50.838 INFO:teuthology.orchestra.run.smithi040.stdout:remove their configuration from /etc/network/interfaces. 2024-03-24T00:29:50.838 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:29:50.977 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-03-24T00:29:51.402 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-03-24T00:29:51.865 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-03-24T00:29:52.570 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-03-24T00:29:52.680 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-03-24T00:29:52.840 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-03-24T00:29:52.966 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up postgresql-common (238) ... 2024-03-24T00:29:53.623 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-24T00:29:53.749 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-24T00:29:53.876 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up postgresql (14+238) ... 2024-03-24T00:29:53.917 INFO:teuthology.orchestra.run.smithi040.stdout:Adding user postgres to group ssl-cert 2024-03-24T00:29:54.001 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-24T00:29:54.147 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:29:54.147 INFO:teuthology.orchestra.run.smithi040.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-03-24T00:29:54.160 INFO:teuthology.orchestra.run.smithi053.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-24T00:29:54.418 INFO:teuthology.orchestra.run.smithi040.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-03-24T00:29:54.418 INFO:teuthology.orchestra.run.smithi040.stdout:Removing obsolete dictionary files: 2024-03-24T00:29:54.420 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-24T00:29:54.421 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-24T00:29:55.138 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-24T00:29:55.164 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-03-24T00:29:55.432 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-24T00:29:55.583 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-24T00:29:55.643 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-03-24T00:29:55.998 INFO:teuthology.orchestra.run.smithi040.stdout:Creating new PostgreSQL cluster 14/main ... 2024-03-24T00:29:56.013 INFO:teuthology.orchestra.run.smithi040.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-03-24T00:29:56.071 INFO:teuthology.orchestra.run.smithi040.stdout:The files belonging to this database system will be owned by user "postgres". 2024-03-24T00:29:56.072 INFO:teuthology.orchestra.run.smithi040.stdout:This user must also own the server process. 2024-03-24T00:29:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:29:56.072 INFO:teuthology.orchestra.run.smithi040.stdout:The database cluster will be initialized with locale "en_US". 2024-03-24T00:29:56.072 INFO:teuthology.orchestra.run.smithi040.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-03-24T00:29:56.072 INFO:teuthology.orchestra.run.smithi040.stdout:The default text search configuration will be set to "english". 2024-03-24T00:29:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:29:56.072 INFO:teuthology.orchestra.run.smithi040.stdout:Data page checksums are disabled. 2024-03-24T00:29:56.072 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:29:56.072 INFO:teuthology.orchestra.run.smithi040.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-03-24T00:29:56.073 INFO:teuthology.orchestra.run.smithi040.stdout:creating subdirectories ... ok 2024-03-24T00:29:56.073 INFO:teuthology.orchestra.run.smithi040.stdout:selecting dynamic shared memory implementation ... posix 2024-03-24T00:29:56.098 INFO:teuthology.orchestra.run.smithi040.stdout:selecting default max_connections ... 100 2024-03-24T00:29:56.133 INFO:teuthology.orchestra.run.smithi040.stdout:selecting default shared_buffers ... 128MB 2024-03-24T00:29:56.147 INFO:teuthology.orchestra.run.smithi040.stdout:selecting default time zone ... Etc/UTC 2024-03-24T00:29:56.148 INFO:teuthology.orchestra.run.smithi040.stdout:creating configuration files ... ok 2024-03-24T00:29:56.320 INFO:teuthology.orchestra.run.smithi040.stdout:running bootstrap script ... ok 2024-03-24T00:29:56.802 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-24T00:29:56.894 INFO:teuthology.orchestra.run.smithi053.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-24T00:29:56.947 INFO:teuthology.orchestra.run.smithi040.stdout:performing post-bootstrap initialization ... ok 2024-03-24T00:29:57.456 INFO:teuthology.orchestra.run.smithi053.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-24T00:29:57.463 DEBUG:teuthology.parallel:result is None 2024-03-24T00:29:57.484 INFO:teuthology.orchestra.run.smithi040.stdout:syncing data to disk ... ok 2024-03-24T00:29:58.018 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-03-24T00:30:01.076 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-03-24T00:30:01.193 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up postgresql-contrib (14+238) ... 2024-03-24T00:30:01.319 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up postgresql (14+238) ... 2024-03-24T00:30:01.479 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-03-24T00:30:01.605 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-03-24T00:30:01.875 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-03-24T00:30:01.875 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-03-24T00:30:02.641 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-03-24T00:30:02.934 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-03-24T00:30:03.078 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-03-24T00:30:04.681 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-03-24T00:30:04.773 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-03-24T00:30:05.319 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-03-24T00:30:05.326 DEBUG:teuthology.parallel:result is None 2024-03-24T00:30:05.327 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:30:05.476 DEBUG:teuthology.orchestra.run.smithi040:> dpkg-query -W -f '${Version}' ceph 2024-03-24T00:30:05.498 INFO:teuthology.orchestra.run.smithi040.stdout:19.0.0-2145-g3be9a5c9-1jammy 2024-03-24T00:30:05.499 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145-g3be9a5c9-1jammy 2024-03-24T00:30:05.499 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145-g3be9a5c9-1jammy is installed. 2024-03-24T00:30:05.501 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:30:05.640 DEBUG:teuthology.orchestra.run.smithi053:> dpkg-query -W -f '${Version}' ceph 2024-03-24T00:30:05.665 INFO:teuthology.orchestra.run.smithi053.stdout:19.0.0-2145-g3be9a5c9-1jammy 2024-03-24T00:30:05.665 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145-g3be9a5c9-1jammy 2024-03-24T00:30:05.665 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145-g3be9a5c9-1jammy is installed. 2024-03-24T00:30:05.667 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T00:30:05.667 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:05.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:30:05.687 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:05.687 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:30:05.728 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T00:30:05.728 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:05.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:30:05.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:30:05.809 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:05.810 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:30:05.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:30:05.896 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T00:30:05.896 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:05.896 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:30:05.916 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:30:05.976 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:05.976 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:30:05.997 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:30:06.063 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T00:30:06.064 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:06.064 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:30:06.083 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:30:06.144 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:06.144 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:30:06.166 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:30:06.228 INFO:teuthology.run_tasks:Running task ceph... 2024-03-24T00:30:06.317 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-24T00:30:06.317 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /var/log/ceph 2024-03-24T00:30:06.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /var/log/ceph 2024-03-24T00:30:06.341 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-24T00:30:06.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-24T00:30:06.384 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-24T00:30:06.405 INFO:tasks.ceph:Creating extra log directories... 2024-03-24T00:30:06.405 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-24T00:30:06.448 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-24T00:30:06.477 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-24T00:30:06.477 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-24T00:30:06.477 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-24T00:30:06.477 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:06.477 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-24T00:30:06.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-24T00:30:06.609 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:06.609 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-24T00:30:06.618 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-24T00:30:06.715 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-24T00:30:06.715 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-24T00:30:06.715 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439', 'branch': 'wip-mchangir-testing-main-20240318.032620', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/exports}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7616439', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-testing-main-20240318.032620'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'Replacing daemon mds'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-mchangir-testing-main-20240318.032620', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, 'owner': 'mchangir', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 3550, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'sleep_before_teardown': 0, 'subset': '613/1024', 'suite': 'fs', 'suite_branch': 'wip-mchangir-testing-main-20240318.032620', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi040.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOIROjU8alZ2MN5QoW9J73ZR2EA4xzHHJqXzdUUZhxpZn2xukBbmH5MmEP4f2xX3XR+PCznb+rsWN6qVGjT02Zg=', 'smithi053.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKNKWqVs1XCeXnKRCFKfBlzYJzTo46EZn7pnrIMmZ42DE0CEaUJrjKTjobTmS/4GXn53Bvm8aYKZEGA5g+eICNI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'Replacing daemon mds'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_exports']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-22_09:49:57', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-24T00:30:06.715 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-24T00:30:06.718 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-24T00:30:06.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/run/ceph 2024-03-24T00:30:06.768 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/run/ceph 2024-03-24T00:30:06.791 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:06.791 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:30:06.837 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:30:06.837 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-03-24T00:30:06.890 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:30:06.890 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:30:06.890 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 893 Links: 1 2024-03-24T00:30:06.890 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:30:06.890 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-24 00:23:37.973302897 +0000 2024-03-24T00:30:06.890 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-24 00:23:37.709305256 +0000 2024-03-24T00:30:06.890 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-24 00:23:37.709305256 +0000 2024-03-24T00:30:06.890 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-03-24 00:23:37.709305256 +0000 2024-03-24T00:30:06.891 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:30:06.951 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-03-24T00:30:06.952 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-03-24T00:30:06.952 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000396491 s, 1.3 MB/s 2024-03-24T00:30:06.954 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:30:07.008 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-03-24T00:30:07.058 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:30:07.058 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:30:07.058 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 913 Links: 1 2024-03-24T00:30:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:30:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-24 00:23:38.469298467 +0000 2024-03-24T00:30:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-24 00:23:38.201300862 +0000 2024-03-24T00:30:07.059 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-24 00:23:38.201300862 +0000 2024-03-24T00:30:07.059 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-03-24 00:23:38.201300862 +0000 2024-03-24T00:30:07.059 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:30:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-03-24T00:30:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-03-24T00:30:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000350179 s, 1.5 MB/s 2024-03-24T00:30:07.127 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:30:07.184 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-03-24T00:30:07.234 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:30:07.234 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:30:07.234 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 930 Links: 1 2024-03-24T00:30:07.234 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:30:07.234 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-24 00:23:39.009293646 +0000 2024-03-24T00:30:07.234 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-24 00:23:38.725296182 +0000 2024-03-24T00:30:07.234 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-24 00:23:38.725296182 +0000 2024-03-24T00:30:07.234 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-03-24 00:23:38.725296182 +0000 2024-03-24T00:30:07.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:30:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-03-24T00:30:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-03-24T00:30:07.297 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000372582 s, 1.4 MB/s 2024-03-24T00:30:07.299 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:30:07.352 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-03-24T00:30:07.402 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:30:07.402 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:30:07.403 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 944 Links: 1 2024-03-24T00:30:07.403 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:30:07.403 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-03-24 00:23:39.521289072 +0000 2024-03-24T00:30:07.403 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-03-24 00:23:39.269291322 +0000 2024-03-24T00:30:07.403 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-03-24 00:23:39.269291322 +0000 2024-03-24T00:30:07.403 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-03-24 00:23:39.269291322 +0000 2024-03-24T00:30:07.403 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:30:07.464 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-03-24T00:30:07.464 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-03-24T00:30:07.464 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000378304 s, 1.4 MB/s 2024-03-24T00:30:07.466 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:30:07.520 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T00:30:07.520 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:07.520 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:30:07.529 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:30:07.530 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-03-24T00:30:07.580 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:30:07.580 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:30:07.580 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 899 Links: 1 2024-03-24T00:30:07.580 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:30:07.580 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-24 00:23:38.255310956 +0000 2024-03-24T00:30:07.580 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-24 00:23:37.991313316 +0000 2024-03-24T00:30:07.580 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-24 00:23:37.991313316 +0000 2024-03-24T00:30:07.580 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-24 00:23:37.991313316 +0000 2024-03-24T00:30:07.581 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:30:07.642 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-24T00:30:07.642 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-24T00:30:07.643 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000263255 s, 1.9 MB/s 2024-03-24T00:30:07.644 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:30:07.697 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-03-24T00:30:07.748 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:30:07.748 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:30:07.748 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 917 Links: 1 2024-03-24T00:30:07.748 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:30:07.748 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-24 00:23:38.755306488 +0000 2024-03-24T00:30:07.748 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-24 00:23:38.471309026 +0000 2024-03-24T00:30:07.749 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-24 00:23:38.471309026 +0000 2024-03-24T00:30:07.749 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-24 00:23:38.471309026 +0000 2024-03-24T00:30:07.749 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:30:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-24T00:30:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-24T00:30:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.0002596 s, 2.0 MB/s 2024-03-24T00:30:07.812 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:30:07.865 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-03-24T00:30:07.919 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:30:07.920 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:30:07.920 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 933 Links: 1 2024-03-24T00:30:07.920 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:30:07.920 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-24 00:23:39.263301947 +0000 2024-03-24T00:30:07.920 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-24 00:23:38.999304307 +0000 2024-03-24T00:30:07.920 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-24 00:23:38.999304307 +0000 2024-03-24T00:30:07.920 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-24 00:23:38.999304307 +0000 2024-03-24T00:30:07.920 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:30:07.982 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-24T00:30:07.982 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-24T00:30:07.982 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000263813 s, 1.9 MB/s 2024-03-24T00:30:07.984 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:30:08.041 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-03-24T00:30:08.095 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:30:08.095 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:30:08.096 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 951 Links: 1 2024-03-24T00:30:08.096 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:30:08.096 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-03-24 00:23:39.787297263 +0000 2024-03-24T00:30:08.096 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-03-24 00:23:39.527299589 +0000 2024-03-24T00:30:08.096 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-03-24 00:23:39.527299589 +0000 2024-03-24T00:30:08.096 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-03-24 00:23:39.527299589 +0000 2024-03-24T00:30:08.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:30:08.162 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-03-24T00:30:08.162 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-03-24T00:30:08.162 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000256999 s, 2.0 MB/s 2024-03-24T00:30:08.164 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:30:08.221 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T00:30:08.222 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-24T00:30:08.222 INFO:tasks.ceph:Generating config... 2024-03-24T00:30:08.223 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-24T00:30:08.223 INFO:tasks.ceph:[client] debug client = 20 2024-03-24T00:30:08.223 INFO:tasks.ceph:[client] debug ms = 1 2024-03-24T00:30:08.223 INFO:tasks.ceph:[client] pid file = 2024-03-24T00:30:08.223 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-24T00:30:08.223 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-24T00:30:08.223 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-24T00:30:08.223 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-24T00:30:08.223 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-24T00:30:08.223 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-24T00:30:08.223 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-24T00:30:08.224 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-24T00:30:08.224 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-24T00:30:08.224 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-24T00:30:08.224 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-24T00:30:08.224 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-24T00:30:08.224 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-24T00:30:08.224 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-24T00:30:08.224 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-24T00:30:08.224 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-24T00:30:08.224 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-24T00:30:08.224 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-24T00:30:08.224 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-24T00:30:08.224 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-24T00:30:08.224 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-24T00:30:08.224 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-24T00:30:08.224 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-24T00:30:08.224 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-24T00:30:08.225 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-24T00:30:08.225 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-24T00:30:08.225 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-24T00:30:08.225 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-24T00:30:08.225 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-24T00:30:08.225 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-24T00:30:08.225 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-24T00:30:08.225 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-24T00:30:08.225 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-24T00:30:08.225 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-24T00:30:08.225 INFO:tasks.ceph:Setting up mon.a... 2024-03-24T00:30:08.225 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-24T00:30:08.272 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.keyring 2024-03-24T00:30:08.287 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-24T00:30:08.337 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T00:30:08.357 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.40'), ('mon.b', '172.21.15.53'), ('mon.c', '[v2:172.21.15.53:3301,v1:172.21.15.53:6790]')] 2024-03-24T00:30:08.358 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.40,172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-24T00:30:08.359 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:08.359 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-24T00:30:08.409 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.40 --add b 172.21.15.53 --addv c '[v2:172.21.15.53:3301,v1:172.21.15.53:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:08.505 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:08.505 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: generated fsid 185b869e-2a51-452f-88b9-d083720a50b9 2024-03-24T00:30:08.505 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = quincy 2024-03-24T00:30:08.505 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 0 2024-03-24T00:30:08.505 INFO:teuthology.orchestra.run.smithi040.stdout:fsid 185b869e-2a51-452f-88b9-d083720a50b9 2024-03-24T00:30:08.505 INFO:teuthology.orchestra.run.smithi040.stdout:last_changed 2024-03-24T00:30:08.507837+0000 2024-03-24T00:30:08.505 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-03-24T00:30:08.507837+0000 2024-03-24T00:30:08.505 INFO:teuthology.orchestra.run.smithi040.stdout:min_mon_release 17 (quincy) 2024-03-24T00:30:08.506 INFO:teuthology.orchestra.run.smithi040.stdout:election_strategy: 1 2024-03-24T00:30:08.506 INFO:teuthology.orchestra.run.smithi040.stdout:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-03-24T00:30:08.506 INFO:teuthology.orchestra.run.smithi040.stdout:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-03-24T00:30:08.506 INFO:teuthology.orchestra.run.smithi040.stdout:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-03-24T00:30:08.506 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-24T00:30:08.508 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-24T00:30:08.516 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 185b869e-2a51-452f-88b9-d083720a50b9... 2024-03-24T00:30:08.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T00:30:08.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-24T00:30:08.604 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2024-03-24T00:30:08.604 INFO:teuthology.orchestra.run.smithi040.stdout: chdir = "" 2024-03-24T00:30:08.604 INFO:teuthology.orchestra.run.smithi040.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T00:30:08.604 INFO:teuthology.orchestra.run.smithi040.stdout: auth supported = cephx 2024-03-24T00:30:08.604 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: filestore xattr use omap = true 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: mon clock drift allowed = 1.000 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: osd crush chooseleaf type = 0 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: auth debug = true 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on old message = true 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on bug = true 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: mon pg warn max object skew = 0 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.605 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default size = 2 2024-03-24T00:30:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow primary affinity = true 2024-03-24T00:30:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow pg remap = true 2024-03-24T00:30:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on legacy crush tunables = false 2024-03-24T00:30:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on crush straw calc version zero = false 2024-03-24T00:30:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on no sortbitwise = false 2024-03-24T00:30:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on osd down out interval zero = false 2024-03-24T00:30:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on too few osds = false 2024-03-24T00:30:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T00:30:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T00:30:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: mon_allow_pool_size_one = true 2024-03-24T00:30:08.606 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: osd default data pool replay window = 5 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: mon allow pool delete = true 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: mon cluster log file level = debug 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: debug asserts on shutdown = true 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: mon health detail to clog = false 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: mon host = "172.21.15.40,172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790]" 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = 185b869e-2a51-452f-88b9-d083720a50b9 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout:[osd] 2024-03-24T00:30:08.607 INFO:teuthology.orchestra.run.smithi040.stdout: osd journal size = 100 2024-03-24T00:30:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub load threshold = 5.0 2024-03-24T00:30:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub max interval = 600 2024-03-24T00:30:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock profile = high_recovery_ops 2024-03-24T00:30:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: osd recover clone overlap = true 2024-03-24T00:30:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: osd recovery max chunk = 1048576 2024-03-24T00:30:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug shutdown = true 2024-03-24T00:30:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug op order = true 2024-03-24T00:30:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify stray on activate = true 2024-03-24T00:30:08.608 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug trim objects = true 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: osd open classes on start = true 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug pg log writeout = true 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: osd deep scrub update digest min age = 30 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: osd map max advance = 10 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: journal zero on create = true 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: filestore ondisk finisher threads = 3 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: filestore apply finisher threads = 3 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.609 INFO:teuthology.orchestra.run.smithi040.stdout: bdev debug aio = true 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug misdirected ops = true 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: bdev async discard = True 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: bdev enable discard = True 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore allocator = bitmap 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore block size = 96636764160 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore fsck on mount = True 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluefs = 1/20 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluestore = 1/20 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: debug osd = 20 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: debug rocksdb = 4/10 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd full ratio = 0.9 2024-03-24T00:30:08.610 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd nearfull ratio = 0.8 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: osd failsafe full ratio = 0.95 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: osd objectstore = bluestore 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: osd op complaint time = 180 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr] 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: debug mgr = 20 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: mgr/telemetry/nag = false 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.611 INFO:teuthology.orchestra.run.smithi040.stdout:[mon] 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: debug paxos = 20 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: mon data avail warn = 5 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: mon mgr mkfs grace = 240 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd reporter subtree level = osd 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd prime pg temp = true 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T00:30:08.612 INFO:teuthology.orchestra.run.smithi040.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: # 1m isn't quite enough 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: mon_down_mkfs_grace = 2m 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_filestore_osds = false 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: mon op complaint time = 120 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout:[client] 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: rgw cache enabled = true 2024-03-24T00:30:08.613 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable ops log = true 2024-03-24T00:30:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable usage log = true 2024-03-24T00:30:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T00:30:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T00:30:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: client mount timeout = 600 2024-03-24T00:30:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: debug client = 20 2024-03-24T00:30:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-03-24T00:30:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: pid file = "" 2024-03-24T00:30:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: rados mon op timeout = 900 2024-03-24T00:30:08.614 INFO:teuthology.orchestra.run.smithi040.stdout: rados osd op timeout = 900 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.a] 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.b] 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.c] 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.a] 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.c] 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.b] 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.d] 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout:[mds] 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: debug mds = 20 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: debug mds balancer = 20 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: mds debug frag = True 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: mds debug scatterstat = True 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: mds op complaint time = 180 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: mds verify scatter = True 2024-03-24T00:30:08.615 INFO:teuthology.orchestra.run.smithi040.stdout: osd op complaint time = 180 2024-03-24T00:30:08.616 INFO:teuthology.orchestra.run.smithi040.stdout: rados mon op timeout = 900 2024-03-24T00:30:08.616 INFO:teuthology.orchestra.run.smithi040.stdout: rados osd op timeout = 900 2024-03-24T00:30:08.616 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-03-24T00:30:08.616 INFO:teuthology.orchestra.run.smithi053.stdout: chdir = "" 2024-03-24T00:30:08.616 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-24T00:30:08.616 INFO:teuthology.orchestra.run.smithi053.stdout: auth supported = cephx 2024-03-24T00:30:08.616 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.616 INFO:teuthology.orchestra.run.smithi053.stdout: filestore xattr use omap = true 2024-03-24T00:30:08.616 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: mon clock drift allowed = 1.000 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: osd crush chooseleaf type = 0 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: auth debug = true 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on old message = true 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on bug = true 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: mon pg warn max object skew = 0 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: # disable pg_autoscaler by default for new pools 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default size = 2 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.617 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow primary affinity = true 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow pg remap = true 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on legacy crush tunables = false 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on crush straw calc version zero = false 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on no sortbitwise = false 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on osd down out interval zero = false 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on too few osds = false 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: mon_allow_pool_size_one = true 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.618 INFO:teuthology.orchestra.run.smithi053.stdout: osd default data pool replay window = 5 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon allow pool delete = true 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon cluster log file level = debug 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: debug asserts on shutdown = true 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon health detail to clog = false 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: mon host = "172.21.15.40,172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790]" 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = 185b869e-2a51-452f-88b9-d083720a50b9 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout:[osd] 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: osd journal size = 100 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub load threshold = 5.0 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub max interval = 600 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock profile = high_recovery_ops 2024-03-24T00:30:08.619 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: osd recover clone overlap = true 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: osd recovery max chunk = 1048576 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug shutdown = true 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug op order = true 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify stray on activate = true 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug trim objects = true 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: osd open classes on start = true 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug pg log writeout = true 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: osd deep scrub update digest min age = 30 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: osd map max advance = 10 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.620 INFO:teuthology.orchestra.run.smithi053.stdout: journal zero on create = true 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: filestore ondisk finisher threads = 3 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: filestore apply finisher threads = 3 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: bdev debug aio = true 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug misdirected ops = true 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: bdev async discard = True 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: bdev enable discard = True 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore allocator = bitmap 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore block size = 96636764160 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore fsck on mount = True 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluefs = 1/20 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluestore = 1/20 2024-03-24T00:30:08.621 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: debug osd = 20 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: debug rocksdb = 4/10 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd backfillfull_ratio = 0.85 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd full ratio = 0.9 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd nearfull ratio = 0.8 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: osd failsafe full ratio = 0.95 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: osd objectstore = bluestore 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: osd op complaint time = 180 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr] 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: debug mgr = 20 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:30:08.622 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: mgr/telemetry/nag = false 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout:[mon] 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: debug paxos = 20 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: mon data avail warn = 5 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: mon mgr mkfs grace = 240 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd reporter subtree level = osd 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd prime pg temp = true 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.623 INFO:teuthology.orchestra.run.smithi053.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: auth mon ticket ttl = 660 # 11m 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: auth service ticket ttl = 240 # 4m 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: # don't complain about insecure global_id in the test suite 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: # 1m isn't quite enough 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: mon_down_mkfs_grace = 2m 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_filestore_osds = false 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: mon op complaint time = 120 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout:[client] 2024-03-24T00:30:08.624 INFO:teuthology.orchestra.run.smithi053.stdout: rgw cache enabled = true 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable ops log = true 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable usage log = true 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout: client mount timeout = 600 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout: debug client = 20 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = "" 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout: rados mon op timeout = 900 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout: rados osd op timeout = 900 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.a] 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.b] 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.c] 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.a] 2024-03-24T00:30:08.625 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.c] 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.b] 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.d] 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout:[mds] 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout: debug mds = 20 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout: debug mds balancer = 20 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout: mds debug frag = True 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout: mds debug scatterstat = True 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout: mds op complaint time = 180 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout: mds verify scatter = True 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout: osd op complaint time = 180 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout: rados mon op timeout = 900 2024-03-24T00:30:08.626 INFO:teuthology.orchestra.run.smithi053.stdout: rados osd op timeout = 900 2024-03-24T00:30:08.628 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-24T00:30:08.628 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-24T00:30:08.731 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-24T00:30:08.731 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:08.731 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-24T00:30:08.740 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:08.740 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-24T00:30:08.789 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:30:08.789 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:08.789 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-24T00:30:08.789 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T00:30:08.861 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:08.861 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:08.913 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:30:08.914 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:08.914 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-24T00:30:08.914 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-24T00:30:08.950 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:08.950 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:08.999 INFO:tasks.ceph:Setting up mon nodes... 2024-03-24T00:30:08.999 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-24T00:30:08.999 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-24T00:30:09.055 INFO:teuthology.orchestra.run.smithi040.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-24T00:30:09.060 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-24T00:30:09.122 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-24T00:30:09.245 INFO:tasks.ceph:Setting up mds nodes... 2024-03-24T00:30:09.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-24T00:30:09.303 INFO:teuthology.orchestra.run.smithi040.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-24T00:30:09.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-24T00:30:09.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-24T00:30:09.428 INFO:teuthology.orchestra.run.smithi040.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-24T00:30:09.432 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-24T00:30:09.452 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-24T00:30:09.514 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-24T00:30:09.518 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-24T00:30:09.540 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-24T00:30:09.643 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-24T00:30:09.648 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-24T00:30:09.668 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-24T00:30:09.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T00:30:09.712 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-24T00:30:09.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-24T00:30:09.815 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-24T00:30:09.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T00:30:09.877 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-24T00:30:09.894 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-24T00:30:09.983 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-24T00:30:10.000 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-24T00:30:10.000 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-24T00:30:10.000 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-24T00:30:10.020 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T00:30:10.020 INFO:tasks.ceph:role: osd.0 2024-03-24T00:30:10.020 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:30:10.020 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-24T00:30:10.088 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:30:10.088 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:30:10.088 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:30:10.089 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-24T00:30:10.089 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:30:10.089 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:30:10.089 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:30:10.089 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-24T00:30:10.089 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:30:10.089 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:30:10.553 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-03-24T00:30:10.556 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-03-24T00:30:10.556 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-24T00:30:10.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-24T00:30:10.738 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2024-03-24T00:30:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:30:10.739 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-24T00:30:10.802 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T00:30:10.802 INFO:tasks.ceph:role: osd.1 2024-03-24T00:30:10.802 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:30:10.802 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-24T00:30:10.874 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:30:10.875 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:30:10.875 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:30:10.875 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-24T00:30:10.875 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:30:10.875 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:30:10.875 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:30:10.875 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-24T00:30:10.875 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:30:10.875 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:30:11.304 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-03-24T00:30:11.307 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-03-24T00:30:11.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-24T00:30:11.409 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-24T00:30:11.425 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2024-03-24T00:30:11.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:30:11.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-24T00:30:11.490 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T00:30:11.490 INFO:tasks.ceph:role: osd.2 2024-03-24T00:30:11.490 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:30:11.490 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-24T00:30:11.565 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:30:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:30:11.565 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:30:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-24T00:30:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:30:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:30:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:30:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-24T00:30:11.566 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:30:11.566 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:30:12.026 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-03-24T00:30:12.029 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-03-24T00:30:12.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-24T00:30:12.169 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-24T00:30:12.187 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2024-03-24T00:30:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:30:12.188 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-24T00:30:12.255 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-24T00:30:12.255 INFO:tasks.ceph:role: osd.3 2024-03-24T00:30:12.255 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:30:12.255 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-24T00:30:12.331 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:30:12.331 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:30:12.331 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:30:12.331 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-24T00:30:12.331 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:30:12.331 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:30:12.331 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:30:12.331 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-24T00:30:12.331 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:30:12.331 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:30:12.821 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-03-24T00:30:12.825 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-03-24T00:30:12.825 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-24T00:30:12.976 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-24T00:30:12.994 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2024-03-24T00:30:12.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:30:12.996 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:13.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:13.132+0000 7ff080a2f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-24T00:30:13.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:13.132+0000 7ff080a2f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-24T00:30:13.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:13.132+0000 7ff080a2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T00:30:13.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:13.132+0000 7ff080a2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T00:30:13.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:13.132+0000 7ff080a2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-24T00:30:13.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:13.136+0000 7ff080a2f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-24T00:30:16.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T00:30:16.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:16.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:16.236+0000 7f1ac8eb47c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-24T00:30:16.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:16.236+0000 7f1ac8eb47c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-24T00:30:16.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:16.236+0000 7f1ac8eb47c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T00:30:16.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:16.236+0000 7f1ac8eb47c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T00:30:16.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:16.236+0000 7f1ac8eb47c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-24T00:30:16.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:16.240+0000 7f1ac8eb47c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-24T00:30:19.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-24T00:30:19.243 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:19.344+0000 7fbb83e5e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-24T00:30:19.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:19.344+0000 7fbb83e5e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-24T00:30:19.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:19.344+0000 7fbb83e5e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T00:30:19.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:19.344+0000 7fbb83e5e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T00:30:19.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:19.344+0000 7fbb83e5e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-24T00:30:19.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:19.348+0000 7fbb83e5e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-24T00:30:22.311 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-24T00:30:22.332 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:22.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:22.432+0000 7f659bfa27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-24T00:30:22.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:22.432+0000 7f659bfa27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-24T00:30:22.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:22.432+0000 7f659bfa27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T00:30:22.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:22.432+0000 7f659bfa27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T00:30:22.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:22.432+0000 7f659bfa27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-24T00:30:22.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:22.432+0000 7f659bfa27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-24T00:30:25.400 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-24T00:30:25.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-24T00:30:25.446 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T00:30:25.446 INFO:tasks.ceph:role: osd.4 2024-03-24T00:30:25.446 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:30:25.446 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-24T00:30:25.508 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:30:25.508 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:30:25.508 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:30:25.508 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-24T00:30:25.508 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:30:25.508 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:30:25.509 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:30:25.509 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-24T00:30:25.509 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:30:25.509 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:30:25.532 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-03-24T00:30:25.535 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-03-24T00:30:25.535 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-24T00:30:25.609 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-24T00:30:25.667 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: /sbin/restorecon: command not found 2024-03-24T00:30:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:30:25.668 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-24T00:30:25.731 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T00:30:25.731 INFO:tasks.ceph:role: osd.5 2024-03-24T00:30:25.731 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:30:25.731 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-24T00:30:25.794 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:30:25.795 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:30:25.795 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:30:25.795 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-24T00:30:25.795 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:30:25.795 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:30:25.795 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:30:25.795 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-24T00:30:25.795 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:30:25.795 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:30:25.819 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-03-24T00:30:25.821 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-03-24T00:30:25.821 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-24T00:30:25.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-24T00:30:25.950 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: /sbin/restorecon: command not found 2024-03-24T00:30:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:30:25.951 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-24T00:30:26.017 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T00:30:26.017 INFO:tasks.ceph:role: osd.6 2024-03-24T00:30:26.018 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:30:26.018 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-24T00:30:26.077 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:30:26.077 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:30:26.077 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:30:26.077 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-24T00:30:26.077 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:30:26.077 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:30:26.078 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:30:26.078 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-24T00:30:26.078 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:30:26.078 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:30:26.101 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-03-24T00:30:26.104 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-03-24T00:30:26.104 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-24T00:30:26.175 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-24T00:30:26.238 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: /sbin/restorecon: command not found 2024-03-24T00:30:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:30:26.239 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-24T00:30:26.302 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-24T00:30:26.302 INFO:tasks.ceph:role: osd.7 2024-03-24T00:30:26.302 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:30:26.302 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-24T00:30:26.361 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-24T00:30:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-24T00:30:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-24T00:30:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-03-24T00:30:26.361 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-24T00:30:26.361 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-03-24T00:30:26.361 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-24T00:30:26.362 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-03-24T00:30:26.362 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-24T00:30:26.362 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-24T00:30:26.386 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-03-24T00:30:26.390 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-03-24T00:30:26.390 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-24T00:30:26.455 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-24T00:30:26.518 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: /sbin/restorecon: command not found 2024-03-24T00:30:26.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:30:26.520 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:26.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:26.636+0000 7fd8244af7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-24T00:30:26.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:26.636+0000 7fd8244af7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-24T00:30:26.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:26.636+0000 7fd8244af7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T00:30:26.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:26.636+0000 7fd8244af7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T00:30:26.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:26.636+0000 7fd8244af7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-24T00:30:26.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:26.640+0000 7fd8244af7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-24T00:30:29.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-24T00:30:29.642 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:29.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:29.740+0000 7f21301197c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-24T00:30:29.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:29.740+0000 7f21301197c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-24T00:30:29.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:29.740+0000 7f21301197c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T00:30:29.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:29.740+0000 7f21301197c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T00:30:29.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:29.740+0000 7f21301197c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-24T00:30:29.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:29.744+0000 7f21301197c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-24T00:30:32.723 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T00:30:32.745 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:32.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:32.840+0000 7fc8ebf207c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-24T00:30:32.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:32.840+0000 7fc8ebf207c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-24T00:30:32.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:32.840+0000 7fc8ebf207c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T00:30:32.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:32.840+0000 7fc8ebf207c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T00:30:32.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:32.840+0000 7fc8ebf207c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-24T00:30:32.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:32.844+0000 7fc8ebf207c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-24T00:30:35.848 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-24T00:30:35.869 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:35.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:35.968+0000 7fc6819057c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-24T00:30:35.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:35.968+0000 7fc6819057c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-24T00:30:35.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:35.968+0000 7fc6819057c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T00:30:35.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:35.968+0000 7fc6819057c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T00:30:35.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:35.968+0000 7fc6819057c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-24T00:30:35.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:35.968+0000 7fc6819057c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-24T00:30:36.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:30:36.720 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:30:38.958 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-24T00:30:38.980 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-24T00:30:38.980 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:38.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-24T00:30:39.003 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:39.003 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-24T00:30:39.066 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:39.066 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-24T00:30:39.130 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:39.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-24T00:30:39.194 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:39.194 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-24T00:30:39.262 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:39.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-24T00:30:39.330 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:39.331 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-24T00:30:39.399 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:39.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-24T00:30:39.421 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:39.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-24T00:30:39.484 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:39.484 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-24T00:30:39.548 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:39.548 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-24T00:30:39.615 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:39.615 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-24T00:30:39.680 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:39.680 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-24T00:30:39.743 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:39.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-24T00:30:39.807 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:39.808 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-24T00:30:39.817 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:39.817 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-24T00:30:39.866 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:39.866 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-24T00:30:39.875 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:39.875 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-24T00:30:39.927 INFO:tasks.ceph:Adding keys to all mons... 2024-03-24T00:30:39.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-24T00:30:39.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-24T00:30:39.974 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.y] 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lEnZyAxAAW5Lrm7LZPTVfLCOY6As1PQ== 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.a] 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lCkU6EhAA0q21F90epsGPK5x17s9NLw== 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.c] 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lKMmwGRAAAKaOSnhTRbHIpPKXXkAe+Q== 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.0] 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAVdP9lVk3uBxAA9TC1OpeB5GIFlfJl5+ILBg== 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.1] 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAYdP9lUc1IDhAAPwCV9eqL7bFibkfMFt9wPw== 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.2] 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAbdP9lwjyuFBAAJ1JAIAnLXAn03+lQPwwY9Q== 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.3] 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAedP9lZ1bgGRAAuLAay48N7jXjj8/yaYwSqg== 2024-03-24T00:30:39.975 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.x] 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lebNrBxAAvbcCAJhDQn/5Uk1dJ01oHw== 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.b] 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9llkfOHhAAqKKFyEJzG0UxbLJMFPET8A== 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout:[mds.d] 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9l7quKJhAA8QsquiFFmj5H1G5yck0FpA== 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.4] 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAidP9lvyk4JhAA8w4/XBYHDLoqJ1zop9sUiw== 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.5] 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAldP9lcb9qLBAAXts6VU/4kVqUWS/WORV7zQ== 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.6] 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAodP9l8ipTMhAAjpagojh+woHyGvlQlIkHDw== 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.7] 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQArdP9lmGfpORAAPvow5nF/O7RLmuXEqNWENg== 2024-03-24T00:30:39.976 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-03-24T00:30:39.977 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw== 2024-03-24T00:30:39.977 INFO:teuthology.orchestra.run.smithi040.stdout:[client.2] 2024-03-24T00:30:39.977 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9l6gXMMBAAd6Oe6S73hj/WfLXTsYiEZw== 2024-03-24T00:30:39.977 INFO:teuthology.orchestra.run.smithi040.stdout:[client.1] 2024-03-24T00:30:39.977 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lpIZrNBAAzpGWZuQ6ndjNMAOmAsebvQ== 2024-03-24T00:30:39.977 INFO:teuthology.orchestra.run.smithi040.stdout:[client.3] 2024-03-24T00:30:39.977 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9llFHCOhAATtv8gcL6BHPnxB1dHrg6vw== 2024-03-24T00:30:39.989 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.y] 2024-03-24T00:30:39.989 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQARdP9lEnZyAxAAW5Lrm7LZPTVfLCOY6As1PQ== 2024-03-24T00:30:39.989 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.a] 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQARdP9lCkU6EhAA0q21F90epsGPK5x17s9NLw== 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.c] 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQARdP9lKMmwGRAAAKaOSnhTRbHIpPKXXkAe+Q== 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.0] 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAVdP9lVk3uBxAA9TC1OpeB5GIFlfJl5+ILBg== 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.1] 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAYdP9lUc1IDhAAPwCV9eqL7bFibkfMFt9wPw== 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.2] 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAbdP9lwjyuFBAAJ1JAIAnLXAn03+lQPwwY9Q== 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.3] 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAedP9lZ1bgGRAAuLAay48N7jXjj8/yaYwSqg== 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.x] 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQARdP9lebNrBxAAvbcCAJhDQn/5Uk1dJ01oHw== 2024-03-24T00:30:39.990 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.b] 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQARdP9llkfOHhAAqKKFyEJzG0UxbLJMFPET8A== 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.d] 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQARdP9l7quKJhAA8QsquiFFmj5H1G5yck0FpA== 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.4] 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAidP9lvyk4JhAA8w4/XBYHDLoqJ1zop9sUiw== 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.5] 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAldP9lcb9qLBAAXts6VU/4kVqUWS/WORV7zQ== 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.6] 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQAodP9l8ipTMhAAjpagojh+woHyGvlQlIkHDw== 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.7] 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQArdP9lmGfpORAAPvow5nF/O7RLmuXEqNWENg== 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw== 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout:[client.2] 2024-03-24T00:30:39.991 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQARdP9l6gXMMBAAd6Oe6S73hj/WfLXTsYiEZw== 2024-03-24T00:30:39.992 INFO:teuthology.orchestra.run.smithi053.stdout:[client.1] 2024-03-24T00:30:39.992 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQARdP9lpIZrNBAAzpGWZuQ6ndjNMAOmAsebvQ== 2024-03-24T00:30:39.992 INFO:teuthology.orchestra.run.smithi053.stdout:[client.3] 2024-03-24T00:30:39.992 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQARdP9llFHCOhAATtv8gcL6BHPnxB1dHrg6vw== 2024-03-24T00:30:39.993 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T00:30:40.024 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T00:30:40.098 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:30:40.101 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:30:40.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:30:40.164 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:30:40.223 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.226 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.283 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.337 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.395 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.454 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T00:30:40.457 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-24T00:30:40.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:30:40.515 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:30:40.573 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:30:40.576 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-24T00:30:40.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.694 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.697 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.756 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.811 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-24T00:30:40.868 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:30:40.872 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:30:40.929 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:30:40.932 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:30:40.989 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:30:40.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:30:41.050 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:30:41.053 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-24T00:30:41.111 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-24T00:30:41.111 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-24T00:30:41.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T00:30:41.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-24T00:30:41.260 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-24T00:30:41.280 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T00:30:41.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-24T00:30:41.413 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-24T00:30:41.474 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-24T00:30:41.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-24T00:30:41.600 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:41.603 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-24T00:30:41.651 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-24T00:30:41.652 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-24T00:30:41.652 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-24T00:30:41.655 INFO:tasks.ceph.mon.a:Started 2024-03-24T00:30:41.655 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-24T00:30:41.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-24T00:30:41.697 INFO:tasks.ceph.mon.b:Started 2024-03-24T00:30:41.698 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-24T00:30:41.698 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-24T00:30:41.701 INFO:tasks.ceph.mon.c:Started 2024-03-24T00:30:41.701 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-24T00:30:41.701 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-24T00:30:41.701 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-24T00:30:41.704 INFO:tasks.ceph.mgr.y:Started 2024-03-24T00:30:41.704 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-24T00:30:41.704 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-24T00:30:41.707 INFO:tasks.ceph.mgr.x:Started 2024-03-24T00:30:41.707 INFO:tasks.ceph:Setting crush tunables to default 2024-03-24T00:30:41.707 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd crush tunables default 2024-03-24T00:30:41.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.848+0000 7f3cd4400640 1 Processor -- start 2024-03-24T00:30:41.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.848+0000 7f3cd4400640 1 -- start start 2024-03-24T00:30:41.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.852+0000 7f3cd4400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc15e170 0x7f3ccc06f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:41.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.852+0000 7f3cd4400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ccc0cfcb0 con 0x7f3ccc160080 2024-03-24T00:30:41.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.852+0000 7f3cd4400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ccc06fe20 con 0x7f3ccc0d0a10 2024-03-24T00:30:41.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.852+0000 7f3cd4400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc070100 con 0x7f3ccc15e170 2024-03-24T00:30:41.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.852+0000 7f3cd1a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc15e170 0x7f3ccc06f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:41.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.852+0000 7f3cd1a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc15e170 0x7f3ccc06f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:39872/0 (socket says 172.21.15.40:39872) 2024-03-24T00:30:41.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.852+0000 7f3cd1a00640 1 -- 172.21.15.40:0/3161112858 learned_addr learned my addr 172.21.15.40:0/3161112858 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:41.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.852+0000 7f3cd1a00640 1 -- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc15e170 msgr2=0x7f3ccc06f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:30:41.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.852+0000 7f3cd1a00640 1 -- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc15e170 msgr2=0x7f3ccc06f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:30:41.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.852+0000 7f3cd1a00640 1 --2- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc15e170 0x7f3ccc06f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:30:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.852+0000 7f3cd1a00640 1 --2- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc15e170 0x7f3ccc06f7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:30:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.996+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260114204 0 0) 0x7f3cc40014a0 con 0x7f3ccc160080 2024-03-24T00:30:41.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.996+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cb4003530 con 0x7f3ccc160080 2024-03-24T00:30:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.996+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689029116 0 0) 0x7f3ccc06fe20 con 0x7f3ccc0d0a10 2024-03-24T00:30:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.996+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cc40014a0 con 0x7f3ccc0d0a10 2024-03-24T00:30:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.996+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1626803410 0 0) 0x7f3cb4003530 con 0x7f3ccc160080 2024-03-24T00:30:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.996+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc15e170 msgr2=0x7f3ccc06f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:30:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.996+0000 7f3ccbe00640 1 --2- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc15e170 0x7f3ccc06f7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:41.996+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 >> v1:172.21.15.53:6789/0 conn(0x7f3ccc0d0a10 legacy=0x7f3ccc15f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc0cff30 con 0x7f3ccc160080 2024-03-24T00:30:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f3cc4001bd0 con 0x7f3ccc160080 2024-03-24T00:30:41.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 >> v1:172.21.15.40:6789/0 conn(0x7f3ccc160080 legacy=0x7f3ccc15e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3ccbe00640 1 --2- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cb4003b80 0x7f3cb4003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3ccbe00640 1 --2- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc15e170 0x7f3cb4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3ccbe00640 1 --2- 172.21.15.40:0/3161112858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cb4004d50 0x7f3cb40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc070100 con 0x7f3cb4004d50 2024-03-24T00:30:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb400bb60 con 0x7f3cb4003b80 2024-03-24T00:30:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb400bd60 con 0x7f3ccc15e170 2024-03-24T00:30:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3cd1a00640 1 --2- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cb4003b80 0x7f3cb4003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3cd2400640 1 --2- 172.21.15.40:0/3161112858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cb4004d50 0x7f3cb40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3cd2400640 1 -- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc15e170 msgr2=0x7f3cb4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3cd2400640 1 --2- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc15e170 0x7f3cb4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3cd2400640 1 -- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cb4003b80 msgr2=0x7f3cb4003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3cd2400640 1 --2- 172.21.15.40:0/3161112858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cb4003b80 0x7f3cb4003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.000+0000 7f3cd2400640 1 -- 172.21.15.40:0/3161112858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb4003530 con 0x7f3cb4004d50 2024-03-24T00:30:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.004+0000 7f3cd2400640 1 --2- 172.21.15.40:0/3161112858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cb4004d50 0x7f3cb40051d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3cc400c530 tx=0x7f3cc401f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548254ec1f22ad80 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.004+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc40b4070 con 0x7f3cb4004d50 2024-03-24T00:30:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.004+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc40a8e50 con 0x7f3cb4004d50 2024-03-24T00:30:42.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.004+0000 7f3ccbe00640 1 -- 172.21.15.40:0/3161112858 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc400c180 con 0x7f3cb4004d50 2024-03-24T00:30:42.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.004+0000 7f3cd4400640 1 -- 172.21.15.40:0/3161112858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cb4004d50 msgr2=0x7f3cb40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.004+0000 7f3cd4400640 1 --2- 172.21.15.40:0/3161112858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cb4004d50 0x7f3cb40051d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3cc400c530 tx=0x7f3cc401f540 comp rx=0 tx=0).stop 2024-03-24T00:30:42.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.004+0000 7f3cd2400640 1 -- 172.21.15.40:0/3161112858 reap_dead start 2024-03-24T00:30:42.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.004+0000 7f3cd4400640 1 -- 172.21.15.40:0/3161112858 shutdown_connections 2024-03-24T00:30:42.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.004+0000 7f3cd4400640 1 -- 172.21.15.40:0/3161112858 >> 172.21.15.40:0/3161112858 conn(0x7f3ccc0b36a0 msgr2=0x7f3ccc0b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:42.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.004+0000 7f3cd4400640 1 -- 172.21.15.40:0/3161112858 shutdown_connections 2024-03-24T00:30:42.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.004+0000 7f3cd4400640 1 -- 172.21.15.40:0/3161112858 wait complete. 2024-03-24T00:30:42.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.008+0000 7f3cd4400640 1 Processor -- start 2024-03-24T00:30:42.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.008+0000 7f3cd4400640 1 -- start start 2024-03-24T00:30:42.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.008+0000 7f3cd4400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cb4004d50 0x7f3ccc0b5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.008+0000 7f3cd4400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc1f5040 0x7f3ccc15cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.008+0000 7f3cd4400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc15d190 0x7f3ccc1f7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.008+0000 7f3cd4400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0b43f0 con 0x7f3cb4004d50 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.008+0000 7f3cd4400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0d04f0 con 0x7f3ccc15d190 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.008+0000 7f3cd4400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0d0e30 con 0x7f3ccc1f5040 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cb4004d50 0x7f3ccc0b5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cb4004d50 0x7f3ccc0b5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47318/0 (socket says 172.21.15.40:47318) 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd1a00640 1 -- 172.21.15.40:0/2737429758 learned_addr learned my addr 172.21.15.40:0/2737429758 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd2400640 1 --2- 172.21.15.40:0/2737429758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc15d190 0x7f3ccc1f7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd1a00640 1 -- 172.21.15.40:0/2737429758 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc1f5040 msgr2=0x7f3ccc15cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd1000640 1 --2- 172.21.15.40:0/2737429758 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc1f5040 0x7f3ccc15cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd1a00640 1 --2- 172.21.15.40:0/2737429758 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc1f5040 0x7f3ccc15cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd1a00640 1 -- 172.21.15.40:0/2737429758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc15d190 msgr2=0x7f3ccc1f7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd1a00640 1 --2- 172.21.15.40:0/2737429758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc15d190 0x7f3ccc1f7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd1a00640 1 -- 172.21.15.40:0/2737429758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc0c6a90 con 0x7f3cb4004d50 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd2400640 1 --2- 172.21.15.40:0/2737429758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc15d190 0x7f3ccc1f7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd1000640 1 --2- 172.21.15.40:0/2737429758 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc1f5040 0x7f3ccc15cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:30:42.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd1a00640 1 --2- 172.21.15.40:0/2737429758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cb4004d50 0x7f3ccc0b5fb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3cc00108f0 tx=0x7f3cc0010db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cc9600640 1 -- 172.21.15.40:0/2737429758 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc001a020 con 0x7f3cb4004d50 2024-03-24T00:30:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cc9600640 1 -- 172.21.15.40:0/2737429758 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc000eb60 con 0x7f3cb4004d50 2024-03-24T00:30:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cc9600640 1 -- 172.21.15.40:0/2737429758 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc000f040 con 0x7f3cb4004d50 2024-03-24T00:30:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd4400640 1 -- 172.21.15.40:0/2737429758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ccc0c6d70 con 0x7f3cb4004d50 2024-03-24T00:30:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd4400640 1 -- 172.21.15.40:0/2737429758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ccc0c7330 con 0x7f3cb4004d50 2024-03-24T00:30:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cc9600640 1 -- 172.21.15.40:0/2737429758 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3cc001d020 con 0x7f3cb4004d50 2024-03-24T00:30:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cc9600640 1 -- 172.21.15.40:0/2737429758 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3cc00168b0 con 0x7f3cb4004d50 2024-03-24T00:30:42.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.012+0000 7f3cd4400640 1 -- 172.21.15.40:0/2737429758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ccc068170 con 0x7f3cb4004d50 2024-03-24T00:30:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.028+0000 7f3cc9600640 1 -- 172.21.15.40:0/2737429758 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3cc0024030 con 0x7f3cb4004d50 2024-03-24T00:30:42.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.096+0000 7f3cd4400640 1 -- 172.21.15.40:0/2737429758 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f3ccc15d570 con 0x7f3cb4004d50 2024-03-24T00:30:42.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.100+0000 7f3cc9600640 1 -- 172.21.15.40:0/2737429758 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3cc0016ab0 con 0x7f3cb4004d50 2024-03-24T00:30:42.098 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2024-03-24T00:30:42.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.100+0000 7f3cd4400640 1 -- 172.21.15.40:0/2737429758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cb4004d50 msgr2=0x7f3ccc0b5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.100+0000 7f3cd4400640 1 --2- 172.21.15.40:0/2737429758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cb4004d50 0x7f3ccc0b5fb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3cc00108f0 tx=0x7f3cc0010db0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.100+0000 7f3cd4400640 1 -- 172.21.15.40:0/2737429758 shutdown_connections 2024-03-24T00:30:42.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.100+0000 7f3cd4400640 1 --2- 172.21.15.40:0/2737429758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc15d190 0x7f3ccc1f7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.100+0000 7f3cd4400640 1 --2- 172.21.15.40:0/2737429758 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ccc1f5040 0x7f3ccc15cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.100+0000 7f3cd4400640 1 --2- 172.21.15.40:0/2737429758 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cb4004d50 0x7f3ccc0b5fb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.100+0000 7f3cd4400640 1 -- 172.21.15.40:0/2737429758 >> 172.21.15.40:0/2737429758 conn(0x7f3ccc0b36a0 msgr2=0x7f3ccc15afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:42.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.100+0000 7f3cd4400640 1 -- 172.21.15.40:0/2737429758 shutdown_connections 2024-03-24T00:30:42.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:42.100+0000 7f3cd4400640 1 -- 172.21.15.40:0/2737429758 wait complete. 2024-03-24T00:30:42.118 INFO:tasks.ceph:check_enable_crimson: False 2024-03-24T00:30:42.118 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-24T00:30:42.118 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:42.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-24T00:30:42.138 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:42.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-24T00:30:42.201 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:42.201 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-24T00:30:42.237 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:42.236+0000 7fb50d43a180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:30:42.238 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:42.235+0000 7f4e6efc2180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:30:42.256 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:30:42.257 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-24T00:30:42.317 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:42.317 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-24T00:30:42.338 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:42.338 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-24T00:30:42.399 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:42.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-24T00:30:42.458 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:30:42.458 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-24T00:30:42.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 449203c7-93f4-418d-beef-133d2e26dc9f 0 2024-03-24T00:30:42.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.707+0000 7fae6f200640 1 Processor -- start 2024-03-24T00:30:42.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.707+0000 7fae6f200640 1 -- start start 2024-03-24T00:30:42.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6f200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015e570 0x7fae7015e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6f200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fae700cecd0 0x7fae700cf110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6f200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae700d00a0 con 0x7fae7015af90 2024-03-24T00:30:42.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6f200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae700b3c60 con 0x7fae700cecd0 2024-03-24T00:30:42.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6f200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7015ee90 con 0x7fae7015e570 2024-03-24T00:30:42.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6de00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015e570 0x7fae7015e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6de00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015e570 0x7fae7015e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:45640/0 (socket says 172.21.15.53:45640) 2024-03-24T00:30:42.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6e800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fae7015af90 0x7fae7015b420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:45366/0 (socket says 172.21.15.53:45366) 2024-03-24T00:30:42.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6de00640 1 -- 172.21.15.53:0/3327717539 learned_addr learned my addr 172.21.15.53:0/3327717539 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-24T00:30:42.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae67e00640 1 --2- 172.21.15.53:0/3327717539 >> v2:172.21.15.40:3300/0 conn(0x7fae700cecd0 0x7fae700cf110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6de00640 1 -- 172.21.15.53:0/3327717539 >> v2:172.21.15.40:3300/0 conn(0x7fae700cecd0 msgr2=0x7fae700cf110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:42.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6de00640 1 --2- 172.21.15.53:0/3327717539 >> v2:172.21.15.40:3300/0 conn(0x7fae700cecd0 0x7fae700cf110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6de00640 1 -- 172.21.15.53:0/3327717539 >> v1:172.21.15.53:6789/0 conn(0x7fae7015af90 legacy=0x7fae7015b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae6de00640 1 -- 172.21.15.53:0/3327717539 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae700cf800 con 0x7fae7015e570 2024-03-24T00:30:42.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.711+0000 7fae67e00640 1 --2- 172.21.15.53:0/3327717539 >> v2:172.21.15.40:3300/0 conn(0x7fae700cecd0 0x7fae700cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:30:42.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6de00640 1 --2- 172.21.15.53:0/3327717539 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015e570 0x7fae7015e950 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fae60009870 tx=0x7fae6001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea2376d654139e29 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:42.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6d400640 1 -- 172.21.15.53:0/3327717539 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae600b1070 con 0x7fae7015e570 2024-03-24T00:30:42.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6d400640 1 -- 172.21.15.53:0/3327717539 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae600ac620 con 0x7fae7015e570 2024-03-24T00:30:42.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6d400640 1 -- 172.21.15.53:0/3327717539 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae600ac9f0 con 0x7fae7015e570 2024-03-24T00:30:42.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6f200640 1 -- 172.21.15.53:0/3327717539 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015e570 msgr2=0x7fae7015e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6f200640 1 --2- 172.21.15.53:0/3327717539 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015e570 0x7fae7015e950 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fae60009870 tx=0x7fae6001c4d0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6f200640 1 -- 172.21.15.53:0/3327717539 shutdown_connections 2024-03-24T00:30:42.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6f200640 1 --2- 172.21.15.53:0/3327717539 >> v2:172.21.15.40:3300/0 conn(0x7fae700cecd0 0x7fae700cf110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6f200640 1 --2- 172.21.15.53:0/3327717539 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015e570 0x7fae7015e950 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6f200640 1 -- 172.21.15.53:0/3327717539 >> 172.21.15.53:0/3327717539 conn(0x7fae7006b750 msgr2=0x7fae700b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:42.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6f200640 1 -- 172.21.15.53:0/3327717539 shutdown_connections 2024-03-24T00:30:42.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6f200640 1 -- 172.21.15.53:0/3327717539 wait complete. 2024-03-24T00:30:42.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6f200640 1 Processor -- start 2024-03-24T00:30:42.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.715+0000 7fae6f200640 1 -- start start 2024-03-24T00:30:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae700cecd0 0x7fae7013a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6f200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015af90 0x7fae7015bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae7015e570 0x7fae7013a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae700d07f0 con 0x7fae700cecd0 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae700d0670 con 0x7fae7015e570 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6f200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae70166110 con 0x7fae7015af90 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae67e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015af90 0x7fae7015bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae7015e570 0x7fae7013a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae67e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015af90 0x7fae7015bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:45646/0 (socket says 172.21.15.53:45646) 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae67e00640 1 -- 172.21.15.53:0/4119946329 learned_addr learned my addr 172.21.15.53:0/4119946329 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6de00640 1 --2- 172.21.15.53:0/4119946329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae700cecd0 0x7fae7013a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6de00640 1 -- 172.21.15.53:0/4119946329 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015af90 msgr2=0x7fae7015bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6de00640 1 --2- 172.21.15.53:0/4119946329 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015af90 0x7fae7015bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6de00640 1 -- 172.21.15.53:0/4119946329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae7015e570 msgr2=0x7fae7013a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6de00640 1 --2- 172.21.15.53:0/4119946329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae7015e570 0x7fae7013a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6de00640 1 -- 172.21.15.53:0/4119946329 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae70167910 con 0x7fae700cecd0 2024-03-24T00:30:42.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae67e00640 1 --2- 172.21.15.53:0/4119946329 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015af90 0x7fae7015bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:42.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6de00640 1 --2- 172.21.15.53:0/4119946329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae700cecd0 0x7fae7013a800 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fae700d0270 tx=0x7fae6001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:42.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae66000640 1 -- 172.21.15.53:0/4119946329 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae600bb020 con 0x7fae700cecd0 2024-03-24T00:30:42.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae66000640 1 -- 172.21.15.53:0/4119946329 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae600b7640 con 0x7fae700cecd0 2024-03-24T00:30:42.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6f200640 1 -- 172.21.15.53:0/4119946329 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae70156650 con 0x7fae700cecd0 2024-03-24T00:30:42.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae66000640 1 -- 172.21.15.53:0/4119946329 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae600b1040 con 0x7fae700cecd0 2024-03-24T00:30:42.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6f200640 1 -- 172.21.15.53:0/4119946329 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae701556f0 con 0x7fae700cecd0 2024-03-24T00:30:42.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae66000640 1 -- 172.21.15.53:0/4119946329 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fae600be020 con 0x7fae700cecd0 2024-03-24T00:30:42.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae66000640 1 -- 172.21.15.53:0/4119946329 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fae600c0980 con 0x7fae700cecd0 2024-03-24T00:30:42.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.719+0000 7fae6f200640 1 -- 172.21.15.53:0/4119946329 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7004f410 con 0x7fae700cecd0 2024-03-24T00:30:42.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.723+0000 7fae66000640 1 -- 172.21.15.53:0/4119946329 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fae600c5030 con 0x7fae700cecd0 2024-03-24T00:30:42.757 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:42.756+0000 7fb50d43a180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:30:42.773 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:42.771+0000 7f4e6efc2180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:30:42.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.795+0000 7fae6f200640 1 -- 172.21.15.53:0/4119946329 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd new", "uuid": "449203c7-93f4-418d-beef-133d2e26dc9f", "id": 0} v 0) v1 -- 0x7fae700b5cc0 con 0x7fae700cecd0 2024-03-24T00:30:42.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.795+0000 7fae66000640 1 -- 172.21.15.53:0/4119946329 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "449203c7-93f4-418d-beef-133d2e26dc9f", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fae60016150 con 0x7fae700cecd0 2024-03-24T00:30:42.798 INFO:teuthology.orchestra.run.smithi053.stdout:0 2024-03-24T00:30:42.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.795+0000 7fae6f200640 1 -- 172.21.15.53:0/4119946329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae700cecd0 msgr2=0x7fae7013a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.795+0000 7fae6f200640 1 --2- 172.21.15.53:0/4119946329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae700cecd0 0x7fae7013a800 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fae700d0270 tx=0x7fae6001ad10 comp rx=0 tx=0).stop 2024-03-24T00:30:42.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.795+0000 7fae6f200640 1 -- 172.21.15.53:0/4119946329 shutdown_connections 2024-03-24T00:30:42.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.795+0000 7fae6f200640 1 --2- 172.21.15.53:0/4119946329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae7015e570 0x7fae7013a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.795+0000 7fae6f200640 1 --2- 172.21.15.53:0/4119946329 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae7015af90 0x7fae7015bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.795+0000 7fae6f200640 1 --2- 172.21.15.53:0/4119946329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae700cecd0 0x7fae7013a800 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.795+0000 7fae6f200640 1 -- 172.21.15.53:0/4119946329 >> 172.21.15.53:0/4119946329 conn(0x7fae7006b750 msgr2=0x7fae70158b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:42.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.799+0000 7fae6f200640 1 -- 172.21.15.53:0/4119946329 shutdown_connections 2024-03-24T00:30:42.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.799+0000 7fae6f200640 1 -- 172.21.15.53:0/4119946329 wait complete. 2024-03-24T00:30:42.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 8f8cbfbc-88e3-423a-a729-0586b4f90926 1 2024-03-24T00:30:42.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.955+0000 7f2383c00640 1 Processor -- start 2024-03-24T00:30:42.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.955+0000 7f2383c00640 1 -- start start 2024-03-24T00:30:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.959+0000 7f2383c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f237c15df80 0x7f237c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.959+0000 7f2383c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c06d0f0 con 0x7f237c0d2270 2024-03-24T00:30:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.959+0000 7f2383c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c06b280 con 0x7f237c06c950 2024-03-24T00:30:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.959+0000 7f2383c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c06b890 con 0x7f237c15df80 2024-03-24T00:30:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.959+0000 7f237be00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f237c06c950 0x7f237c0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:45378/0 (socket says 172.21.15.53:45378) 2024-03-24T00:30:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.959+0000 7f2381200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f237c15df80 0x7f237c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.959+0000 7f237be00640 1 -- 172.21.15.53:0/1661200423 learned_addr learned my addr 172.21.15.53:0/1661200423 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-24T00:30:42.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.959+0000 7f2381200640 1 -- 172.21.15.53:0/1661200423 >> v1:172.21.15.53:6789/0 conn(0x7f237c06c950 legacy=0x7f237c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.959+0000 7f2381200640 1 -- 172.21.15.53:0/1661200423 >> v1:172.21.15.40:6789/0 conn(0x7f237c0d2270 legacy=0x7f237c161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.959+0000 7f2381200640 1 -- 172.21.15.53:0/1661200423 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c06ce10 con 0x7f237c15df80 2024-03-24T00:30:42.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.959+0000 7f237b400640 1 -- 172.21.15.53:0/1661200423 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909135575 0 0) 0x7f237c06b280 con 0x7f237c06c950 2024-03-24T00:30:42.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f2381200640 1 --2- 172.21.15.53:0/1661200423 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f237c15df80 0x7f237c070ff0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f23700097f0 tx=0x7f237001c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fbc8867264cc6a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:42.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f237b400640 1 -- 172.21.15.53:0/1661200423 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23700a7070 con 0x7f237c15df80 2024-03-24T00:30:42.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f237b400640 1 -- 172.21.15.53:0/1661200423 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23700b0070 con 0x7f237c15df80 2024-03-24T00:30:42.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f237b400640 1 -- 172.21.15.53:0/1661200423 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23700ab890 con 0x7f237c15df80 2024-03-24T00:30:42.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f2383c00640 1 -- 172.21.15.53:0/1661200423 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f237c15df80 msgr2=0x7f237c070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f2383c00640 1 --2- 172.21.15.53:0/1661200423 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f237c15df80 0x7f237c070ff0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f23700097f0 tx=0x7f237001c4b0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f2383c00640 1 -- 172.21.15.53:0/1661200423 shutdown_connections 2024-03-24T00:30:42.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f2383c00640 1 --2- 172.21.15.53:0/1661200423 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f237c15df80 0x7f237c070ff0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f2383c00640 1 -- 172.21.15.53:0/1661200423 >> 172.21.15.53:0/1661200423 conn(0x7f237c06ecf0 msgr2=0x7f237c06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:42.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f2383c00640 1 -- 172.21.15.53:0/1661200423 shutdown_connections 2024-03-24T00:30:42.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f2383c00640 1 -- 172.21.15.53:0/1661200423 wait complete. 2024-03-24T00:30:42.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f2383c00640 1 Processor -- start 2024-03-24T00:30:42.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f2383c00640 1 -- start start 2024-03-24T00:30:42.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f2383c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f237c06c950 0x7f237c14f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.963+0000 7f2383c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f237c0d2270 0x7f237c15fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2383c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f237c15df80 0x7f237c1602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2383c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c16bfa0 con 0x7f237c15df80 2024-03-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2383c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c16be20 con 0x7f237c06c950 2024-03-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2383c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c16c120 con 0x7f237c0d2270 2024-03-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2381200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f237c06c950 0x7f237c14f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f237be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f237c0d2270 0x7f237c15fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2381200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f237c06c950 0x7f237c14f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34412/0 (socket says 172.21.15.53:34412) 2024-03-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2381200640 1 -- 172.21.15.53:0/586189982 learned_addr learned my addr 172.21.15.53:0/586189982 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2381c00640 1 --2- 172.21.15.53:0/586189982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f237c15df80 0x7f237c1602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2381200640 1 -- 172.21.15.53:0/586189982 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f237c0d2270 msgr2=0x7f237c15fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2381200640 1 --2- 172.21.15.53:0/586189982 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f237c0d2270 0x7f237c15fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2381200640 1 -- 172.21.15.53:0/586189982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f237c15df80 msgr2=0x7f237c1602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2381200640 1 --2- 172.21.15.53:0/586189982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f237c15df80 0x7f237c1602d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2381200640 1 -- 172.21.15.53:0/586189982 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c15ccf0 con 0x7f237c06c950 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f237be00640 1 --2- 172.21.15.53:0/586189982 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f237c0d2270 0x7f237c15fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2381200640 1 --2- 172.21.15.53:0/586189982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f237c06c950 0x7f237c14f760 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f237001a0e0 tx=0x7f2370019250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2378c00640 1 -- 172.21.15.53:0/586189982 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23700abd60 con 0x7f237c06c950 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2378c00640 1 -- 172.21.15.53:0/586189982 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23700bf450 con 0x7f237c06c950 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2378c00640 1 -- 172.21.15.53:0/586189982 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23700bf840 con 0x7f237c06c950 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2383c00640 1 -- 172.21.15.53:0/586189982 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c15cfd0 con 0x7f237c06c950 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2383c00640 1 -- 172.21.15.53:0/586189982 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c15d590 con 0x7f237c06c950 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2378c00640 1 -- 172.21.15.53:0/586189982 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f23700b43f0 con 0x7f237c06c950 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2378c00640 1 -- 172.21.15.53:0/586189982 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f23700b3730 con 0x7f237c06c950 2024-03-24T00:30:42.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.967+0000 7f2383c00640 1 -- 172.21.15.53:0/586189982 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237c068170 con 0x7f237c06c950 2024-03-24T00:30:42.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:42.971+0000 7f2378c00640 1 -- 172.21.15.53:0/586189982 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f23700c6030 con 0x7f237c06c950 2024-03-24T00:30:42.974 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:42.972+0000 7fb50d43a180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:30:43.004 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:43.003+0000 7f4e6efc2180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:30:43.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.039+0000 7f2383c00640 1 -- 172.21.15.53:0/586189982 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8f8cbfbc-88e3-423a-a729-0586b4f90926", "id": 1} v 0) v1 -- 0x7f237c0cf9d0 con 0x7f237c06c950 2024-03-24T00:30:43.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.047+0000 7f2378c00640 1 -- 172.21.15.53:0/586189982 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8f8cbfbc-88e3-423a-a729-0586b4f90926", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f23700160d0 con 0x7f237c06c950 2024-03-24T00:30:43.048 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-03-24T00:30:43.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.047+0000 7f2383c00640 1 -- 172.21.15.53:0/586189982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f237c06c950 msgr2=0x7f237c14f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.047+0000 7f2383c00640 1 --2- 172.21.15.53:0/586189982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f237c06c950 0x7f237c14f760 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f237001a0e0 tx=0x7f2370019250 comp rx=0 tx=0).stop 2024-03-24T00:30:43.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.047+0000 7f2383c00640 1 -- 172.21.15.53:0/586189982 shutdown_connections 2024-03-24T00:30:43.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.047+0000 7f2383c00640 1 --2- 172.21.15.53:0/586189982 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f237c15df80 0x7f237c1602d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.047+0000 7f2383c00640 1 --2- 172.21.15.53:0/586189982 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f237c0d2270 0x7f237c15fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.047+0000 7f2383c00640 1 --2- 172.21.15.53:0/586189982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f237c06c950 0x7f237c14f760 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.047+0000 7f2383c00640 1 -- 172.21.15.53:0/586189982 >> 172.21.15.53:0/586189982 conn(0x7f237c06ecf0 msgr2=0x7f237c160bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:43.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.047+0000 7f2383c00640 1 -- 172.21.15.53:0/586189982 shutdown_connections 2024-03-24T00:30:43.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.047+0000 7f2383c00640 1 -- 172.21.15.53:0/586189982 wait complete. 2024-03-24T00:30:43.068 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 66b79393-78ff-48c2-9fa2-e06f21854c29 2 2024-03-24T00:30:43.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.203+0000 7fc86d600640 1 Processor -- start 2024-03-24T00:30:43.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.203+0000 7fc86d600640 1 -- start start 2024-03-24T00:30:43.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc86d600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc86815bb10 0x7fc86806f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc86d600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc86806f570 0x7fc868066ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc86d600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc868067c60 con 0x7fc868067430 2024-03-24T00:30:43.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc86d600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc86806f9d0 con 0x7fc86815bb10 2024-03-24T00:30:43.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc86d600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86815ef60 con 0x7fc86806f570 2024-03-24T00:30:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc866a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc86815bb10 0x7fc86806f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc85fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc86806f570 0x7fc868066ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc867400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc868067430 0x7fc868071b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:45390/0 (socket says 172.21.15.53:45390) 2024-03-24T00:30:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc867400640 1 -- 172.21.15.53:0/1402242033 learned_addr learned my addr 172.21.15.53:0/1402242033 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-24T00:30:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc866a00640 1 -- 172.21.15.53:0/1402242033 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc86806f570 msgr2=0x7fc868066ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc866a00640 1 --2- 172.21.15.53:0/1402242033 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc86806f570 0x7fc868066ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc866a00640 1 -- 172.21.15.53:0/1402242033 >> v1:172.21.15.53:6789/0 conn(0x7fc868067430 legacy=0x7fc868071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc866a00640 1 -- 172.21.15.53:0/1402242033 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86815e910 con 0x7fc86815bb10 2024-03-24T00:30:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc85fe00640 1 --2- 172.21.15.53:0/1402242033 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc86806f570 0x7fc868066ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc866000640 1 -- 172.21.15.53:0/1402242033 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121088995 0 0) 0x7fc868067c60 con 0x7fc868067430 2024-03-24T00:30:43.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.207+0000 7fc866a00640 1 --2- 172.21.15.53:0/1402242033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc86815bb10 0x7fc86806f030 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc8580097c0 tx=0x7fc85801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47cc149daf8bde45 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:43.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc866000640 1 -- 172.21.15.53:0/1402242033 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8580b0070 con 0x7fc86815bb10 2024-03-24T00:30:43.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc866000640 1 -- 172.21.15.53:0/1402242033 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8580ab600 con 0x7fc86815bb10 2024-03-24T00:30:43.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc866000640 1 -- 172.21.15.53:0/1402242033 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8580ab9d0 con 0x7fc86815bb10 2024-03-24T00:30:43.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 -- 172.21.15.53:0/1402242033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc86815bb10 msgr2=0x7fc86806f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 --2- 172.21.15.53:0/1402242033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc86815bb10 0x7fc86806f030 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc8580097c0 tx=0x7fc85801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 -- 172.21.15.53:0/1402242033 shutdown_connections 2024-03-24T00:30:43.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 --2- 172.21.15.53:0/1402242033 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc86806f570 0x7fc868066ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 --2- 172.21.15.53:0/1402242033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc86815bb10 0x7fc86806f030 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 -- 172.21.15.53:0/1402242033 >> 172.21.15.53:0/1402242033 conn(0x7fc8680b27c0 msgr2=0x7fc8680b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:43.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 -- 172.21.15.53:0/1402242033 shutdown_connections 2024-03-24T00:30:43.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 -- 172.21.15.53:0/1402242033 wait complete. 2024-03-24T00:30:43.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 Processor -- start 2024-03-24T00:30:43.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 -- start start 2024-03-24T00:30:43.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc868067430 0x7fc8681f7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc86806f570 0x7fc8681521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc86815bb10 0x7fc868160d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8680679e0 con 0x7fc86806f570 2024-03-24T00:30:43.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc868067860 con 0x7fc86815bb10 2024-03-24T00:30:43.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc86d600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86815f3f0 con 0x7fc868067430 2024-03-24T00:30:43.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc867400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc86815bb10 0x7fc868160d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc866a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc868067430 0x7fc8681f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc867400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc86815bb10 0x7fc868160d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34430/0 (socket says 172.21.15.53:34430) 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc867400640 1 -- 172.21.15.53:0/3534527242 learned_addr learned my addr 172.21.15.53:0/3534527242 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.211+0000 7fc85fe00640 1 --2- 172.21.15.53:0/3534527242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc86806f570 0x7fc8681521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc867400640 1 -- 172.21.15.53:0/3534527242 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc868067430 msgr2=0x7fc8681f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc867400640 1 --2- 172.21.15.53:0/3534527242 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc868067430 0x7fc8681f7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc867400640 1 -- 172.21.15.53:0/3534527242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc86806f570 msgr2=0x7fc8681521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc867400640 1 --2- 172.21.15.53:0/3534527242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc86806f570 0x7fc8681521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc867400640 1 -- 172.21.15.53:0/3534527242 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8680ada60 con 0x7fc86815bb10 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc867400640 1 --2- 172.21.15.53:0/3534527242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc86815bb10 0x7fc868160d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc86000c7d0 tx=0x7fc86000cc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc85ea00640 1 -- 172.21.15.53:0/3534527242 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc860018070 con 0x7fc86815bb10 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc85ea00640 1 -- 172.21.15.53:0/3534527242 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86000a640 con 0x7fc86815bb10 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc85ea00640 1 -- 172.21.15.53:0/3534527242 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86000aa70 con 0x7fc86815bb10 2024-03-24T00:30:43.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc86d600640 1 -- 172.21.15.53:0/3534527242 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8680add40 con 0x7fc86815bb10 2024-03-24T00:30:43.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc86d600640 1 -- 172.21.15.53:0/3534527242 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8680ae300 con 0x7fc86815bb10 2024-03-24T00:30:43.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc85ea00640 1 -- 172.21.15.53:0/3534527242 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc86000ac10 con 0x7fc86815bb10 2024-03-24T00:30:43.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc85ea00640 1 -- 172.21.15.53:0/3534527242 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fc860012870 con 0x7fc86815bb10 2024-03-24T00:30:43.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.215+0000 7fc86d600640 1 -- 172.21.15.53:0/3534527242 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc868068170 con 0x7fc86815bb10 2024-03-24T00:30:43.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.219+0000 7fc85ea00640 1 -- 172.21.15.53:0/3534527242 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc860021070 con 0x7fc86815bb10 2024-03-24T00:30:43.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.287+0000 7fc86d600640 1 -- 172.21.15.53:0/3534527242 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd new", "uuid": "66b79393-78ff-48c2-9fa2-e06f21854c29", "id": 2} v 0) v1 -- 0x7fc8680d0450 con 0x7fc86815bb10 2024-03-24T00:30:43.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.291+0000 7fc85ea00640 1 -- 172.21.15.53:0/3534527242 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "66b79393-78ff-48c2-9fa2-e06f21854c29", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc860012450 con 0x7fc86815bb10 2024-03-24T00:30:43.294 INFO:teuthology.orchestra.run.smithi053.stdout:2 2024-03-24T00:30:43.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.295+0000 7fc86d600640 1 -- 172.21.15.53:0/3534527242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc86815bb10 msgr2=0x7fc868160d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.295+0000 7fc86d600640 1 --2- 172.21.15.53:0/3534527242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc86815bb10 0x7fc868160d30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc86000c7d0 tx=0x7fc86000cc90 comp rx=0 tx=0).stop 2024-03-24T00:30:43.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.295+0000 7fc86d600640 1 -- 172.21.15.53:0/3534527242 shutdown_connections 2024-03-24T00:30:43.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.295+0000 7fc86d600640 1 --2- 172.21.15.53:0/3534527242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc86815bb10 0x7fc868160d30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.295+0000 7fc86d600640 1 --2- 172.21.15.53:0/3534527242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc86806f570 0x7fc8681521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.295+0000 7fc86d600640 1 --2- 172.21.15.53:0/3534527242 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc868067430 0x7fc8681f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.295+0000 7fc86d600640 1 -- 172.21.15.53:0/3534527242 >> 172.21.15.53:0/3534527242 conn(0x7fc8680b27c0 msgr2=0x7fc8680b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:43.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.295+0000 7fc86d600640 1 -- 172.21.15.53:0/3534527242 shutdown_connections 2024-03-24T00:30:43.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.295+0000 7fc86d600640 1 -- 172.21.15.53:0/3534527242 wait complete. 2024-03-24T00:30:43.321 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 29eacc84-6841-4519-ab42-3fed72243aa8 3 2024-03-24T00:30:43.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.459+0000 7f05d6400640 1 Processor -- start 2024-03-24T00:30:43.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.459+0000 7f05d6400640 1 -- start start 2024-03-24T00:30:43.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.459+0000 7f05d6400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d015bd40 0x7f05d00d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.459+0000 7f05d6400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d015f970 con 0x7f05d00d1520 2024-03-24T00:30:43.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.459+0000 7f05d6400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d015fc50 con 0x7f05d015b960 2024-03-24T00:30:43.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.459+0000 7f05d6400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d0160580 con 0x7f05d015bd40 2024-03-24T00:30:43.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.463+0000 7f05c7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d015bd40 0x7f05d00d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.463+0000 7f05cf400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f05d015b960 0x7f05d0070d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50590/0 (socket says 172.21.15.53:50590) 2024-03-24T00:30:43.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.463+0000 7f05cf400640 1 -- 172.21.15.53:0/1267390723 learned_addr learned my addr 172.21.15.53:0/1267390723 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-24T00:30:43.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.463+0000 7f05c7e00640 1 -- 172.21.15.53:0/1267390723 >> v1:172.21.15.53:6789/0 conn(0x7f05d015b960 legacy=0x7f05d0070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.463+0000 7f05c7e00640 1 -- 172.21.15.53:0/1267390723 >> v1:172.21.15.40:6789/0 conn(0x7f05d00d1520 legacy=0x7f05d00cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.463+0000 7f05cea00640 1 -- 172.21.15.53:0/1267390723 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558056747 0 0) 0x7f05d015fc50 con 0x7f05d015b960 2024-03-24T00:30:43.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.463+0000 7f05c7e00640 1 -- 172.21.15.53:0/1267390723 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d015ff30 con 0x7f05d015bd40 2024-03-24T00:30:43.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.463+0000 7f05c7e00640 1 --2- 172.21.15.53:0/1267390723 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d015bd40 0x7f05d00d0fe0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f05b80097f0 tx=0x7f05b801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd27b2371daf0bd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:43.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.463+0000 7f05cea00640 1 -- 172.21.15.53:0/1267390723 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b80b0070 con 0x7f05d015bd40 2024-03-24T00:30:43.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.463+0000 7f05cea00640 1 -- 172.21.15.53:0/1267390723 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b80ab600 con 0x7f05d015bd40 2024-03-24T00:30:43.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.463+0000 7f05cea00640 1 -- 172.21.15.53:0/1267390723 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b80ab9d0 con 0x7f05d015bd40 2024-03-24T00:30:43.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 -- 172.21.15.53:0/1267390723 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d015bd40 msgr2=0x7f05d00d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 --2- 172.21.15.53:0/1267390723 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d015bd40 0x7f05d00d0fe0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f05b80097f0 tx=0x7f05b801c4e0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 -- 172.21.15.53:0/1267390723 shutdown_connections 2024-03-24T00:30:43.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 --2- 172.21.15.53:0/1267390723 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d015bd40 0x7f05d00d0fe0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 -- 172.21.15.53:0/1267390723 >> 172.21.15.53:0/1267390723 conn(0x7f05d006ecf0 msgr2=0x7f05d006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:43.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 -- 172.21.15.53:0/1267390723 shutdown_connections 2024-03-24T00:30:43.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 -- 172.21.15.53:0/1267390723 wait complete. 2024-03-24T00:30:43.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 Processor -- start 2024-03-24T00:30:43.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 -- start start 2024-03-24T00:30:43.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d00d1520 0x7f05d014c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05d015b960 0x7f05d00c5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d015bd40 0x7f05d0159200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d00cecb0 con 0x7f05d015b960 2024-03-24T00:30:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d00ceb30 con 0x7f05d015bd40 2024-03-24T00:30:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05d6400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d0169b80 con 0x7f05d00d1520 2024-03-24T00:30:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05cf400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d00d1520 0x7f05d014c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05cf400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d00d1520 0x7f05d014c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:51750/0 (socket says 172.21.15.53:51750) 2024-03-24T00:30:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05cfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d015bd40 0x7f05d0159200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05cf400640 1 -- 172.21.15.53:0/1879192940 learned_addr learned my addr 172.21.15.53:0/1879192940 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-24T00:30:43.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05c7e00640 1 --2- 172.21.15.53:0/1879192940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05d015b960 0x7f05d00c5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05cf400640 1 -- 172.21.15.53:0/1879192940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d015bd40 msgr2=0x7f05d0159200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05cf400640 1 --2- 172.21.15.53:0/1879192940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d015bd40 0x7f05d0159200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05cf400640 1 -- 172.21.15.53:0/1879192940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05d015b960 msgr2=0x7f05d00c5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05cf400640 1 --2- 172.21.15.53:0/1879192940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05d015b960 0x7f05d00c5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05cf400640 1 -- 172.21.15.53:0/1879192940 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d00c65e0 con 0x7f05d00d1520 2024-03-24T00:30:43.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05cfe00640 1 --2- 172.21.15.53:0/1879192940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d015bd40 0x7f05d0159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:43.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.467+0000 7f05c7e00640 1 --2- 172.21.15.53:0/1879192940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05d015b960 0x7f05d00c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:43.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.471+0000 7f05cf400640 1 --2- 172.21.15.53:0/1879192940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d00d1520 0x7f05d014c6e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f05c000c4a0 tx=0x7f05c000c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:43.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.471+0000 7f05c7400640 1 -- 172.21.15.53:0/1879192940 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0018070 con 0x7f05d00d1520 2024-03-24T00:30:43.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.471+0000 7f05c7400640 1 -- 172.21.15.53:0/1879192940 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c00074e0 con 0x7f05d00d1520 2024-03-24T00:30:43.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.471+0000 7f05c7400640 1 -- 172.21.15.53:0/1879192940 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0004a30 con 0x7f05d00d1520 2024-03-24T00:30:43.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.471+0000 7f05d6400640 1 -- 172.21.15.53:0/1879192940 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d00c68c0 con 0x7f05d00d1520 2024-03-24T00:30:43.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.471+0000 7f05d6400640 1 -- 172.21.15.53:0/1879192940 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d00c6e80 con 0x7f05d00d1520 2024-03-24T00:30:43.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.471+0000 7f05c7400640 1 -- 172.21.15.53:0/1879192940 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f05c0004bd0 con 0x7f05d00d1520 2024-03-24T00:30:43.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.471+0000 7f05c7400640 1 -- 172.21.15.53:0/1879192940 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f05c001b7f0 con 0x7f05d00d1520 2024-03-24T00:30:43.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.471+0000 7f05d6400640 1 -- 172.21.15.53:0/1879192940 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d0068170 con 0x7f05d00d1520 2024-03-24T00:30:43.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.475+0000 7f05c7400640 1 -- 172.21.15.53:0/1879192940 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f05c0021070 con 0x7f05d00d1520 2024-03-24T00:30:43.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.543+0000 7f05d6400640 1 -- 172.21.15.53:0/1879192940 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd new", "uuid": "29eacc84-6841-4519-ab42-3fed72243aa8", "id": 3} v 0) v1 -- 0x7f05d00b26e0 con 0x7f05d00d1520 2024-03-24T00:30:43.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.547+0000 7f05c7400640 1 -- 172.21.15.53:0/1879192940 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "29eacc84-6841-4519-ab42-3fed72243aa8", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f05c001b360 con 0x7f05d00d1520 2024-03-24T00:30:43.550 INFO:teuthology.orchestra.run.smithi053.stdout:3 2024-03-24T00:30:43.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.547+0000 7f05d6400640 1 -- 172.21.15.53:0/1879192940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d00d1520 msgr2=0x7f05d014c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.547+0000 7f05d6400640 1 --2- 172.21.15.53:0/1879192940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d00d1520 0x7f05d014c6e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f05c000c4a0 tx=0x7f05c000c960 comp rx=0 tx=0).stop 2024-03-24T00:30:43.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.547+0000 7f05d6400640 1 -- 172.21.15.53:0/1879192940 shutdown_connections 2024-03-24T00:30:43.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.547+0000 7f05d6400640 1 --2- 172.21.15.53:0/1879192940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d015bd40 0x7f05d0159200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.547+0000 7f05d6400640 1 --2- 172.21.15.53:0/1879192940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05d015b960 0x7f05d00c5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.547+0000 7f05d6400640 1 --2- 172.21.15.53:0/1879192940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d00d1520 0x7f05d014c6e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.547+0000 7f05d6400640 1 -- 172.21.15.53:0/1879192940 >> 172.21.15.53:0/1879192940 conn(0x7f05d006ecf0 msgr2=0x7f05d00b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:43.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.547+0000 7f05d6400640 1 -- 172.21.15.53:0/1879192940 shutdown_connections 2024-03-24T00:30:43.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.547+0000 7f05d6400640 1 -- 172.21.15.53:0/1879192940 wait complete. 2024-03-24T00:30:43.570 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new e9144226-cc4c-4d16-9c5f-8940de541a44 4 2024-03-24T00:30:43.620 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:43.620+0000 7fb50d43a180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:30:43.673 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:43.671+0000 7f4e6efc2180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:30:43.685 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:43.684+0000 7fb50d43a180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:30:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.707+0000 7f20b2a00640 1 Processor -- start 2024-03-24T00:30:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.707+0000 7f20b2a00640 1 -- start start 2024-03-24T00:30:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.707+0000 7f20b2a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20ac15bd40 0x7f20ac0d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.707+0000 7f20b2a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f20ac0d1520 0x7f20ac0cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.707+0000 7f20b2a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ac15f970 con 0x7f20ac15b960 2024-03-24T00:30:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.707+0000 7f20b2a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ac160580 con 0x7f20ac0d1520 2024-03-24T00:30:43.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.707+0000 7f20b2a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac15fc50 con 0x7f20ac15bd40 2024-03-24T00:30:43.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20abe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f20ac15b960 0x7f20ac070d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50596/0 (socket says 172.21.15.53:50596) 2024-03-24T00:30:43.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20abe00640 1 -- 172.21.15.53:0/1691536519 learned_addr learned my addr 172.21.15.53:0/1691536519 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-24T00:30:43.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20a3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20ac15bd40 0x7f20ac0d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20b0a00640 1 --2- 172.21.15.53:0/1691536519 >> v2:172.21.15.40:3300/0 conn(0x7f20ac0d1520 0x7f20ac0cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20a3e00640 1 -- 172.21.15.53:0/1691536519 >> v2:172.21.15.40:3300/0 conn(0x7f20ac0d1520 msgr2=0x7f20ac0cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20a3e00640 1 --2- 172.21.15.53:0/1691536519 >> v2:172.21.15.40:3300/0 conn(0x7f20ac0d1520 0x7f20ac0cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20a3e00640 1 -- 172.21.15.53:0/1691536519 >> v1:172.21.15.53:6789/0 conn(0x7f20ac15b960 legacy=0x7f20ac070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20a3e00640 1 -- 172.21.15.53:0/1691536519 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac15ff30 con 0x7f20ac15bd40 2024-03-24T00:30:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20ab400640 1 -- 172.21.15.53:0/1691536519 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91701069 0 0) 0x7f20ac15f970 con 0x7f20ac15b960 2024-03-24T00:30:43.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20a3e00640 1 --2- 172.21.15.53:0/1691536519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20ac15bd40 0x7f20ac0d0fe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f20940097c0 tx=0x7f209401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=510aa4815f432ba9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:43.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20ab400640 1 -- 172.21.15.53:0/1691536519 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20940b0070 con 0x7f20ac15bd40 2024-03-24T00:30:43.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20ab400640 1 -- 172.21.15.53:0/1691536519 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20940ab600 con 0x7f20ac15bd40 2024-03-24T00:30:43.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.711+0000 7f20ab400640 1 -- 172.21.15.53:0/1691536519 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20940ab9d0 con 0x7f20ac15bd40 2024-03-24T00:30:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.715+0000 7f20b2a00640 1 -- 172.21.15.53:0/1691536519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20ac15bd40 msgr2=0x7f20ac0d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.715+0000 7f20b2a00640 1 --2- 172.21.15.53:0/1691536519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20ac15bd40 0x7f20ac0d0fe0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f20940097c0 tx=0x7f209401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.715+0000 7f20b2a00640 1 -- 172.21.15.53:0/1691536519 shutdown_connections 2024-03-24T00:30:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.715+0000 7f20b2a00640 1 --2- 172.21.15.53:0/1691536519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20ac15bd40 0x7f20ac0d0fe0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.715+0000 7f20b2a00640 1 --2- 172.21.15.53:0/1691536519 >> v2:172.21.15.40:3300/0 conn(0x7f20ac0d1520 0x7f20ac0cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.715+0000 7f20b2a00640 1 -- 172.21.15.53:0/1691536519 >> 172.21.15.53:0/1691536519 conn(0x7f20ac06ecf0 msgr2=0x7f20ac06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:43.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.715+0000 7f20b2a00640 1 -- 172.21.15.53:0/1691536519 shutdown_connections 2024-03-24T00:30:43.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.715+0000 7f20b2a00640 1 -- 172.21.15.53:0/1691536519 wait complete. 2024-03-24T00:30:43.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.715+0000 7f20b2a00640 1 Processor -- start 2024-03-24T00:30:43.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.715+0000 7f20b2a00640 1 -- start start 2024-03-24T00:30:43.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20b2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20ac0d1520 0x7f20ac0ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20b2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20ac15b960 0x7f20ac14fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20b2a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20ac15bd40 0x7f20ac14ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20b2a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac0ceae0 con 0x7f20ac0d1520 2024-03-24T00:30:43.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20b2a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac0ce960 con 0x7f20ac15b960 2024-03-24T00:30:43.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20b2a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac0cec60 con 0x7f20ac15bd40 2024-03-24T00:30:43.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20ac15b960 0x7f20ac14fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20b0a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20ac15bd40 0x7f20ac14ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20ac15b960 0x7f20ac14fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45532/0 (socket says 172.21.15.53:45532) 2024-03-24T00:30:43.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a3e00640 1 -- 172.21.15.53:0/1954270274 learned_addr learned my addr 172.21.15.53:0/1954270274 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-24T00:30:43.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20abe00640 1 --2- 172.21.15.53:0/1954270274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20ac0d1520 0x7f20ac0ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a3e00640 1 -- 172.21.15.53:0/1954270274 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20ac15bd40 msgr2=0x7f20ac14ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a3e00640 1 --2- 172.21.15.53:0/1954270274 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20ac15bd40 0x7f20ac14ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a3e00640 1 -- 172.21.15.53:0/1954270274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20ac0d1520 msgr2=0x7f20ac0ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a3e00640 1 --2- 172.21.15.53:0/1954270274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20ac0d1520 0x7f20ac0ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a3e00640 1 -- 172.21.15.53:0/1954270274 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac0caae0 con 0x7f20ac15b960 2024-03-24T00:30:43.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20b0a00640 1 --2- 172.21.15.53:0/1954270274 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20ac15bd40 0x7f20ac14ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:43.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20abe00640 1 --2- 172.21.15.53:0/1954270274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20ac0d1520 0x7f20ac0ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:43.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a3e00640 1 --2- 172.21.15.53:0/1954270274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20ac15b960 0x7f20ac14fa80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f209401ab50 tx=0x7f2094019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:43.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a8c00640 1 -- 172.21.15.53:0/1954270274 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20940b0040 con 0x7f20ac15b960 2024-03-24T00:30:43.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a8c00640 1 -- 172.21.15.53:0/1954270274 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20940be070 con 0x7f20ac15b960 2024-03-24T00:30:43.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a8c00640 1 -- 172.21.15.53:0/1954270274 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20940b76c0 con 0x7f20ac15b960 2024-03-24T00:30:43.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20b2a00640 1 -- 172.21.15.53:0/1954270274 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac0cadc0 con 0x7f20ac15b960 2024-03-24T00:30:43.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20b2a00640 1 -- 172.21.15.53:0/1954270274 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac0cb380 con 0x7f20ac15b960 2024-03-24T00:30:43.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a8c00640 1 -- 172.21.15.53:0/1954270274 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f20940b3040 con 0x7f20ac15b960 2024-03-24T00:30:43.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20a8c00640 1 -- 172.21.15.53:0/1954270274 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f20940b5430 con 0x7f20ac15b960 2024-03-24T00:30:43.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.719+0000 7f20b2a00640 1 -- 172.21.15.53:0/1954270274 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ac068170 con 0x7f20ac15b960 2024-03-24T00:30:43.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.723+0000 7f20a8c00640 1 -- 172.21.15.53:0/1954270274 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f2094016090 con 0x7f20ac15b960 2024-03-24T00:30:43.743 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:43.739+0000 7f4e6efc2180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:30:43.749 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:43.748+0000 7fb50d43a180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:30:43.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.791+0000 7f20b2a00640 1 -- 172.21.15.53:0/1954270274 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e9144226-cc4c-4d16-9c5f-8940de541a44", "id": 4} v 0) v1 -- 0x7f20ac1551e0 con 0x7f20ac15b960 2024-03-24T00:30:43.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.799+0000 7f20a8c00640 1 -- 172.21.15.53:0/1954270274 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e9144226-cc4c-4d16-9c5f-8940de541a44", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f20940b7860 con 0x7f20ac15b960 2024-03-24T00:30:43.800 INFO:teuthology.orchestra.run.smithi053.stdout:4 2024-03-24T00:30:43.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.799+0000 7f20b2a00640 1 -- 172.21.15.53:0/1954270274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20ac15b960 msgr2=0x7f20ac14fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.799+0000 7f20b2a00640 1 --2- 172.21.15.53:0/1954270274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20ac15b960 0x7f20ac14fa80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f209401ab50 tx=0x7f2094019240 comp rx=0 tx=0).stop 2024-03-24T00:30:43.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.799+0000 7f20b2a00640 1 -- 172.21.15.53:0/1954270274 shutdown_connections 2024-03-24T00:30:43.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.799+0000 7f20b2a00640 1 --2- 172.21.15.53:0/1954270274 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20ac15bd40 0x7f20ac14ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.799+0000 7f20b2a00640 1 --2- 172.21.15.53:0/1954270274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20ac15b960 0x7f20ac14fa80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.799+0000 7f20b2a00640 1 --2- 172.21.15.53:0/1954270274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20ac0d1520 0x7f20ac0ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.799+0000 7f20b2a00640 1 -- 172.21.15.53:0/1954270274 >> 172.21.15.53:0/1954270274 conn(0x7f20ac06ecf0 msgr2=0x7f20ac0cf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:43.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.799+0000 7f20b2a00640 1 -- 172.21.15.53:0/1954270274 shutdown_connections 2024-03-24T00:30:43.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.799+0000 7f20b2a00640 1 -- 172.21.15.53:0/1954270274 wait complete. 2024-03-24T00:30:43.808 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:43.807+0000 7f4e6efc2180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:30:43.820 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new c5a3943d-da15-4f7b-be59-3fa00b734007 5 2024-03-24T00:30:43.843 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:43.844+0000 7fb50d43a180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:30:43.905 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:43.903+0000 7f4e6efc2180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:30:43.907 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:43.908+0000 7fb50d43a180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:30:43.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb122e00640 1 Processor -- start 2024-03-24T00:30:43.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb122e00640 1 -- start start 2024-03-24T00:30:43.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb122e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb11c15bd40 0x7fb11c0d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb122e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c0d1520 0x7fb11c0cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb122e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c15f970 con 0x7fb11c15b960 2024-03-24T00:30:43.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb122e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb11c160580 con 0x7fb11c15bd40 2024-03-24T00:30:43.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb122e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c15fc50 con 0x7fb11c0d1520 2024-03-24T00:30:43.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb113400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb11c15bd40 0x7fb11c0d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb120e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c0d1520 0x7fb11c0cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb120e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c0d1520 0x7fb11c0cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:51776/0 (socket says 172.21.15.53:51776) 2024-03-24T00:30:43.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb11be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb11c15b960 0x7fb11c070d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.53:57920/0 (socket says 172.21.15.53:57920) 2024-03-24T00:30:43.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb113400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb11c15bd40 0x7fb11c0d0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45538/0 (socket says 172.21.15.53:45538) 2024-03-24T00:30:43.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb120e00640 1 -- 172.21.15.53:0/3434137811 learned_addr learned my addr 172.21.15.53:0/3434137811 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-24T00:30:43.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb113400640 1 -- 172.21.15.53:0/3434137811 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c0d1520 msgr2=0x7fb11c0cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:43.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb113400640 1 --2- 172.21.15.53:0/3434137811 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c0d1520 0x7fb11c0cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb113400640 1 -- 172.21.15.53:0/3434137811 >> v1:172.21.15.40:6789/0 conn(0x7fb11c15b960 legacy=0x7fb11c070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb113400640 1 -- 172.21.15.53:0/3434137811 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c15ff30 con 0x7fb11c15bd40 2024-03-24T00:30:43.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.959+0000 7fb120e00640 1 --2- 172.21.15.53:0/3434137811 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c0d1520 0x7fb11c0cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:43.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.963+0000 7fb113400640 1 --2- 172.21.15.53:0/3434137811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15bd40 0x7fb11c0d0fe0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb104009d90 tx=0x7fb10401c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8669f68913cbb405 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:43.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.963+0000 7fb11b400640 1 -- 172.21.15.53:0/3434137811 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1040b2070 con 0x7fb11c15bd40 2024-03-24T00:30:43.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.963+0000 7fb11b400640 1 -- 172.21.15.53:0/3434137811 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1040a5830 con 0x7fb11c15bd40 2024-03-24T00:30:43.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.963+0000 7fb11b400640 1 -- 172.21.15.53:0/3434137811 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1040a5c00 con 0x7fb11c15bd40 2024-03-24T00:30:43.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.963+0000 7fb122e00640 1 -- 172.21.15.53:0/3434137811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15bd40 msgr2=0x7fb11c0d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.963+0000 7fb122e00640 1 --2- 172.21.15.53:0/3434137811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15bd40 0x7fb11c0d0fe0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb104009d90 tx=0x7fb10401c970 comp rx=0 tx=0).stop 2024-03-24T00:30:43.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.963+0000 7fb122e00640 1 -- 172.21.15.53:0/3434137811 shutdown_connections 2024-03-24T00:30:43.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.963+0000 7fb122e00640 1 --2- 172.21.15.53:0/3434137811 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c0d1520 0x7fb11c0cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.963+0000 7fb122e00640 1 --2- 172.21.15.53:0/3434137811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15bd40 0x7fb11c0d0fe0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.963+0000 7fb122e00640 1 -- 172.21.15.53:0/3434137811 >> 172.21.15.53:0/3434137811 conn(0x7fb11c06ecf0 msgr2=0x7fb11c06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:43.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.963+0000 7fb122e00640 1 -- 172.21.15.53:0/3434137811 shutdown_connections 2024-03-24T00:30:43.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.963+0000 7fb122e00640 1 -- 172.21.15.53:0/3434137811 wait complete. 2024-03-24T00:30:43.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb122e00640 1 Processor -- start 2024-03-24T00:30:43.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb122e00640 1 -- start start 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb122e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c0d1520 0x7fb11c1f7d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb122e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15b960 0x7fb11c13da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb122e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb11c15bd40 0x7fb11c0c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb122e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c0ceb60 con 0x7fb11c15bd40 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb122e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c0ce9e0 con 0x7fb11c0d1520 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb122e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c0cece0 con 0x7fb11c15b960 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb11be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c0d1520 0x7fb11c1f7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb113400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15b960 0x7fb11c13da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb113400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15b960 0x7fb11c13da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:51778/0 (socket says 172.21.15.53:51778) 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb113400640 1 -- 172.21.15.53:0/1848644990 learned_addr learned my addr 172.21.15.53:0/1848644990 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb120e00640 1 --2- 172.21.15.53:0/1848644990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb11c15bd40 0x7fb11c0c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb113400640 1 -- 172.21.15.53:0/1848644990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c0d1520 msgr2=0x7fb11c1f7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb113400640 1 --2- 172.21.15.53:0/1848644990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c0d1520 0x7fb11c1f7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb113400640 1 -- 172.21.15.53:0/1848644990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb11c15bd40 msgr2=0x7fb11c0c7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb113400640 1 --2- 172.21.15.53:0/1848644990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb11c15bd40 0x7fb11c0c7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb113400640 1 -- 172.21.15.53:0/1848644990 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c0c8470 con 0x7fb11c15b960 2024-03-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb113400640 1 --2- 172.21.15.53:0/1848644990 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15b960 0x7fb11c13da70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb1040b0040 tx=0x7fb104019710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:43.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb118c00640 1 -- 172.21.15.53:0/1848644990 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1040b2050 con 0x7fb11c15b960 2024-03-24T00:30:43.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb118c00640 1 -- 172.21.15.53:0/1848644990 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1040be070 con 0x7fb11c15b960 2024-03-24T00:30:43.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb118c00640 1 -- 172.21.15.53:0/1848644990 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1040ad850 con 0x7fb11c15b960 2024-03-24T00:30:43.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb122e00640 1 -- 172.21.15.53:0/1848644990 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11c15c960 con 0x7fb11c15b960 2024-03-24T00:30:43.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb122e00640 1 -- 172.21.15.53:0/1848644990 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11c15cf20 con 0x7fb11c15b960 2024-03-24T00:30:43.972 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:43.967+0000 7f4e6efc2180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:30:43.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb118c00640 1 -- 172.21.15.53:0/1848644990 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb1040ad9f0 con 0x7fb11c15b960 2024-03-24T00:30:43.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.967+0000 7fb118c00640 1 -- 172.21.15.53:0/1848644990 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fb1040adce0 con 0x7fb11c15b960 2024-03-24T00:30:43.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.971+0000 7fb122e00640 1 -- 172.21.15.53:0/1848644990 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11c068170 con 0x7fb11c15b960 2024-03-24T00:30:43.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:43.971+0000 7fb118c00640 1 -- 172.21.15.53:0/1848644990 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb1040c7020 con 0x7fb11c15b960 2024-03-24T00:30:44.001 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:44.000+0000 7fb50d43a180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:30:44.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.039+0000 7fb122e00640 1 -- 172.21.15.53:0/1848644990 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c5a3943d-da15-4f7b-be59-3fa00b734007", "id": 5} v 0) v1 -- 0x7fb11c070910 con 0x7fb11c15b960 2024-03-24T00:30:44.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.047+0000 7fb118c00640 1 -- 172.21.15.53:0/1848644990 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c5a3943d-da15-4f7b-be59-3fa00b734007", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb104016560 con 0x7fb11c15b960 2024-03-24T00:30:44.049 INFO:teuthology.orchestra.run.smithi053.stdout:5 2024-03-24T00:30:44.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.047+0000 7fb122e00640 1 -- 172.21.15.53:0/1848644990 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15b960 msgr2=0x7fb11c13da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.047+0000 7fb122e00640 1 --2- 172.21.15.53:0/1848644990 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15b960 0x7fb11c13da70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb1040b0040 tx=0x7fb104019710 comp rx=0 tx=0).stop 2024-03-24T00:30:44.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.047+0000 7fb122e00640 1 -- 172.21.15.53:0/1848644990 shutdown_connections 2024-03-24T00:30:44.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.047+0000 7fb122e00640 1 --2- 172.21.15.53:0/1848644990 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb11c15bd40 0x7fb11c0c7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.047+0000 7fb122e00640 1 --2- 172.21.15.53:0/1848644990 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15b960 0x7fb11c13da70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.047+0000 7fb122e00640 1 --2- 172.21.15.53:0/1848644990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c0d1520 0x7fb11c1f7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.047+0000 7fb122e00640 1 -- 172.21.15.53:0/1848644990 >> 172.21.15.53:0/1848644990 conn(0x7fb11c06ecf0 msgr2=0x7fb11c0b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:44.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.047+0000 7fb122e00640 1 -- 172.21.15.53:0/1848644990 shutdown_connections 2024-03-24T00:30:44.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.047+0000 7fb122e00640 1 -- 172.21.15.53:0/1848644990 wait complete. 2024-03-24T00:30:44.069 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:44.067+0000 7f4e6efc2180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:30:44.069 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:44.068+0000 7fb50d43a180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:30:44.070 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4 6 2024-03-24T00:30:44.137 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:44.135+0000 7f4e6efc2180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:30:44.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.207+0000 7faed1400640 1 Processor -- start 2024-03-24T00:30:44.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.207+0000 7faed1400640 1 -- start start 2024-03-24T00:30:44.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.207+0000 7faed1400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faecc067590 0x7faecc0cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.207+0000 7faed1400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0cf070 con 0x7faecc06dcf0 2024-03-24T00:30:44.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.207+0000 7faed1400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc06e120 con 0x7faecc145690 2024-03-24T00:30:44.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.207+0000 7faed1400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faecc0727b0 con 0x7faecc067590 2024-03-24T00:30:44.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.207+0000 7faec1e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faecc067590 0x7faecc0cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.207+0000 7faeca800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7faecc145690 0x7faecc066e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50598/0 (socket says 172.21.15.53:50598) 2024-03-24T00:30:44.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.207+0000 7faeca800640 1 -- 172.21.15.53:0/2805226448 learned_addr learned my addr 172.21.15.53:0/2805226448 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-03-24T00:30:44.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.207+0000 7faec1e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faecc067590 0x7faecc0cd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45568/0 (socket says 172.21.15.53:45568) 2024-03-24T00:30:44.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faec1e00640 1 -- 172.21.15.53:0/2805226448 >> v1:172.21.15.53:6789/0 conn(0x7faecc145690 legacy=0x7faecc066e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faec1e00640 1 -- 172.21.15.53:0/2805226448 >> v1:172.21.15.40:6789/0 conn(0x7faecc06dcf0 legacy=0x7faecc0cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faec1e00640 1 -- 172.21.15.53:0/2805226448 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc072a90 con 0x7faecc067590 2024-03-24T00:30:44.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faec1e00640 1 --2- 172.21.15.53:0/2805226448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faecc067590 0x7faecc0cd5a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faeb40097c0 tx=0x7faeb401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c84ab95a229c774 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:44.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faec9e00640 1 -- 172.21.15.53:0/2805226448 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb40b0070 con 0x7faecc067590 2024-03-24T00:30:44.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faec9e00640 1 -- 172.21.15.53:0/2805226448 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb4004230 con 0x7faecc067590 2024-03-24T00:30:44.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faec9e00640 1 -- 172.21.15.53:0/2805226448 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb4003620 con 0x7faecc067590 2024-03-24T00:30:44.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faed1400640 1 -- 172.21.15.53:0/2805226448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faecc067590 msgr2=0x7faecc0cd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faed1400640 1 --2- 172.21.15.53:0/2805226448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faecc067590 0x7faecc0cd5a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faeb40097c0 tx=0x7faeb401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faed1400640 1 -- 172.21.15.53:0/2805226448 shutdown_connections 2024-03-24T00:30:44.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faed1400640 1 --2- 172.21.15.53:0/2805226448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faecc067590 0x7faecc0cd5a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faed1400640 1 -- 172.21.15.53:0/2805226448 >> 172.21.15.53:0/2805226448 conn(0x7faecc0d0b90 msgr2=0x7faecc0d0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:44.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faed1400640 1 -- 172.21.15.53:0/2805226448 shutdown_connections 2024-03-24T00:30:44.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.211+0000 7faed1400640 1 -- 172.21.15.53:0/2805226448 wait complete. 2024-03-24T00:30:44.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faed1400640 1 Processor -- start 2024-03-24T00:30:44.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faed1400640 1 -- start start 2024-03-24T00:30:44.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faed1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faecc067590 0x7faecc13c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faed1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faecc06dcf0 0x7faecc13ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faed1400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faecc145690 0x7faecc0cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faed1400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc0d0330 con 0x7faecc067590 2024-03-24T00:30:44.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faed1400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faecc06fa90 con 0x7faecc06dcf0 2024-03-24T00:30:44.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faed1400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc13bfc0 con 0x7faecc145690 2024-03-24T00:30:44.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faeca800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faecc067590 0x7faecc13c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faec1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faecc06dcf0 0x7faecc13ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faecb200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faecc145690 0x7faecc0cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faeca800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faecc067590 0x7faecc13c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.53:56174/0 (socket says 172.21.15.53:56174) 2024-03-24T00:30:44.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faeca800640 1 -- 172.21.15.53:0/2107504964 learned_addr learned my addr 172.21.15.53:0/2107504964 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-24T00:30:44.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faec1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faecc06dcf0 0x7faecc13ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45582/0 (socket says 172.21.15.53:45582) 2024-03-24T00:30:44.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faeca800640 1 -- 172.21.15.53:0/2107504964 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faecc145690 msgr2=0x7faecc0cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faeca800640 1 --2- 172.21.15.53:0/2107504964 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faecc145690 0x7faecc0cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faeca800640 1 -- 172.21.15.53:0/2107504964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faecc06dcf0 msgr2=0x7faecc13ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faeca800640 1 --2- 172.21.15.53:0/2107504964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faecc06dcf0 0x7faecc13ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faeca800640 1 -- 172.21.15.53:0/2107504964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc0cca40 con 0x7faecc067590 2024-03-24T00:30:44.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faecb200640 1 --2- 172.21.15.53:0/2107504964 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faecc145690 0x7faecc0cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:44.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faec1e00640 1 --2- 172.21.15.53:0/2107504964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faecc06dcf0 0x7faecc13ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:44.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faeca800640 1 --2- 172.21.15.53:0/2107504964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faecc067590 0x7faecc13c190 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faebc00c530 tx=0x7faebc00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:44.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.215+0000 7faec3400640 1 -- 172.21.15.53:0/2107504964 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc017070 con 0x7faecc067590 2024-03-24T00:30:44.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.219+0000 7faec3400640 1 -- 172.21.15.53:0/2107504964 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faebc005cf0 con 0x7faecc067590 2024-03-24T00:30:44.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.219+0000 7faec3400640 1 -- 172.21.15.53:0/2107504964 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faebc007270 con 0x7faecc067590 2024-03-24T00:30:44.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.219+0000 7faed1400640 1 -- 172.21.15.53:0/2107504964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc0b3df0 con 0x7faecc067590 2024-03-24T00:30:44.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.219+0000 7faed1400640 1 -- 172.21.15.53:0/2107504964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc0b43b0 con 0x7faecc067590 2024-03-24T00:30:44.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.219+0000 7faec3400640 1 -- 172.21.15.53:0/2107504964 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7faebc01a070 con 0x7faecc067590 2024-03-24T00:30:44.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.219+0000 7faec3400640 1 -- 172.21.15.53:0/2107504964 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7faebc01cc10 con 0x7faecc067590 2024-03-24T00:30:44.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.219+0000 7faed1400640 1 -- 172.21.15.53:0/2107504964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faecc068880 con 0x7faecc067590 2024-03-24T00:30:44.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.223+0000 7faec3400640 1 -- 172.21.15.53:0/2107504964 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7faebc007410 con 0x7faecc067590 2024-03-24T00:30:44.246 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:44.248+0000 7fb50d43a180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:30:44.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.291+0000 7faed1400640 1 -- 172.21.15.53:0/2107504964 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4", "id": 6} v 0) v1 -- 0x7faecc0d0970 con 0x7faecc067590 2024-03-24T00:30:44.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.295+0000 7faec3400640 1 -- 172.21.15.53:0/2107504964 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7faebc0075f0 con 0x7faecc067590 2024-03-24T00:30:44.296 INFO:teuthology.orchestra.run.smithi053.stdout:6 2024-03-24T00:30:44.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.295+0000 7faed1400640 1 -- 172.21.15.53:0/2107504964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faecc067590 msgr2=0x7faecc13c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.295+0000 7faed1400640 1 --2- 172.21.15.53:0/2107504964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faecc067590 0x7faecc13c190 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7faebc00c530 tx=0x7faebc00c9f0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.295+0000 7faed1400640 1 -- 172.21.15.53:0/2107504964 shutdown_connections 2024-03-24T00:30:44.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.295+0000 7faed1400640 1 --2- 172.21.15.53:0/2107504964 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faecc145690 0x7faecc0cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.295+0000 7faed1400640 1 --2- 172.21.15.53:0/2107504964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faecc06dcf0 0x7faecc13ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.295+0000 7faed1400640 1 --2- 172.21.15.53:0/2107504964 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faecc067590 0x7faecc13c190 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.295+0000 7faed1400640 1 -- 172.21.15.53:0/2107504964 >> 172.21.15.53:0/2107504964 conn(0x7faecc0d0b90 msgr2=0x7faecc1446c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:44.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.295+0000 7faed1400640 1 -- 172.21.15.53:0/2107504964 shutdown_connections 2024-03-24T00:30:44.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.295+0000 7faed1400640 1 -- 172.21.15.53:0/2107504964 wait complete. 2024-03-24T00:30:44.315 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new ada23732-17bc-4ce0-b7e0-8974631732b0 7 2024-03-24T00:30:44.316 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:44.315+0000 7f4e6efc2180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:30:44.321 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:44.320+0000 7fb50d43a180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:30:44.393 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:44.391+0000 7f4e6efc2180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:30:44.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.451+0000 7f07d1c00640 1 Processor -- start 2024-03-24T00:30:44.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.451+0000 7f07d1c00640 1 -- start start 2024-03-24T00:30:44.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.455+0000 7f07d1c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f07cc0d0d50 0x7f07cc0cf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.455+0000 7f07d1c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc0b0b90 con 0x7f07cc15e100 2024-03-24T00:30:44.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.455+0000 7f07d1c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc06cd90 con 0x7f07cc072f10 2024-03-24T00:30:44.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.455+0000 7f07d1c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07cc074570 con 0x7f07cc0d0d50 2024-03-24T00:30:44.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.455+0000 7f07cb400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f07cc0d0d50 0x7f07cc0cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.455+0000 7f07cb400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f07cc0d0d50 0x7f07cc0cf550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45596/0 (socket says 172.21.15.53:45596) 2024-03-24T00:30:44.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.455+0000 7f07c3e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f07cc072f10 0x7f07cc0cee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50610/0 (socket says 172.21.15.53:50610) 2024-03-24T00:30:44.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.455+0000 7f07cb400640 1 -- 172.21.15.53:0/3420424341 learned_addr learned my addr 172.21.15.53:0/3420424341 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-24T00:30:44.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.455+0000 7f07cb400640 1 -- 172.21.15.53:0/3420424341 >> v1:172.21.15.53:6789/0 conn(0x7f07cc072f10 legacy=0x7f07cc0cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.455+0000 7f07cb400640 1 -- 172.21.15.53:0/3420424341 >> v1:172.21.15.40:6789/0 conn(0x7f07cc15e100 legacy=0x7f07cc072800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.455+0000 7f07cb400640 1 -- 172.21.15.53:0/3420424341 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc0b0540 con 0x7f07cc0d0d50 2024-03-24T00:30:44.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07cb400640 1 --2- 172.21.15.53:0/3420424341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07cc0d0d50 0x7f07cc0cf550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009e50 tx=0x7f07c401ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82d3a2fe9d97aace server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:44.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07ca000640 1 -- 172.21.15.53:0/3420424341 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c40a5780 con 0x7f07cc0d0d50 2024-03-24T00:30:44.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07ca000640 1 -- 172.21.15.53:0/3420424341 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c40a5920 con 0x7f07cc0d0d50 2024-03-24T00:30:44.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07ca000640 1 -- 172.21.15.53:0/3420424341 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c40a5cf0 con 0x7f07cc0d0d50 2024-03-24T00:30:44.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07d1c00640 1 -- 172.21.15.53:0/3420424341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07cc0d0d50 msgr2=0x7f07cc0cf550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07d1c00640 1 --2- 172.21.15.53:0/3420424341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07cc0d0d50 0x7f07cc0cf550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009e50 tx=0x7f07c401ca30 comp rx=0 tx=0).stop 2024-03-24T00:30:44.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07d1c00640 1 -- 172.21.15.53:0/3420424341 shutdown_connections 2024-03-24T00:30:44.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07d1c00640 1 --2- 172.21.15.53:0/3420424341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07cc0d0d50 0x7f07cc0cf550 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07d1c00640 1 -- 172.21.15.53:0/3420424341 >> 172.21.15.53:0/3420424341 conn(0x7f07cc0d0230 msgr2=0x7f07cc0d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:44.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07d1c00640 1 -- 172.21.15.53:0/3420424341 shutdown_connections 2024-03-24T00:30:44.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07d1c00640 1 -- 172.21.15.53:0/3420424341 wait complete. 2024-03-24T00:30:44.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07d1c00640 1 Processor -- start 2024-03-24T00:30:44.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.459+0000 7f07d1c00640 1 -- start start 2024-03-24T00:30:44.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07d1c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07cc072f10 0x7f07cc1499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07d1c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07cc0d0d50 0x7f07cc153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07d1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc15e100 0x7f07cc1538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07d1c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc074350 con 0x7f07cc15e100 2024-03-24T00:30:44.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07d1c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc0741d0 con 0x7f07cc072f10 2024-03-24T00:30:44.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07d1c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc0cfea0 con 0x7f07cc0d0d50 2024-03-24T00:30:44.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07caa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07cc072f10 0x7f07cc1499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07c3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07cc0d0d50 0x7f07cc153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07caa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07cc072f10 0x7f07cc1499e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45600/0 (socket says 172.21.15.53:45600) 2024-03-24T00:30:44.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07caa00640 1 -- 172.21.15.53:0/2438154258 learned_addr learned my addr 172.21.15.53:0/2438154258 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-03-24T00:30:44.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07cb400640 1 --2- 172.21.15.53:0/2438154258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc15e100 0x7f07cc1538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07c3e00640 1 -- 172.21.15.53:0/2438154258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07cc072f10 msgr2=0x7f07cc1499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07c3e00640 1 --2- 172.21.15.53:0/2438154258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07cc072f10 0x7f07cc1499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07c3e00640 1 -- 172.21.15.53:0/2438154258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc15e100 msgr2=0x7f07cc1538b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07c3e00640 1 --2- 172.21.15.53:0/2438154258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc15e100 0x7f07cc1538b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07c3e00640 1 -- 172.21.15.53:0/2438154258 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc149f20 con 0x7f07cc0d0d50 2024-03-24T00:30:44.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07caa00640 1 --2- 172.21.15.53:0/2438154258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07cc072f10 0x7f07cc1499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:44.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07c3e00640 1 --2- 172.21.15.53:0/2438154258 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07cc0d0d50 0x7f07cc153370 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f07b400c430 tx=0x7f07b400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:44.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07c2a00640 1 -- 172.21.15.53:0/2438154258 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b40074e0 con 0x7f07cc0d0d50 2024-03-24T00:30:44.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07c2a00640 1 -- 172.21.15.53:0/2438154258 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b4007680 con 0x7f07cc0d0d50 2024-03-24T00:30:44.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07c2a00640 1 -- 172.21.15.53:0/2438154258 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b4004a10 con 0x7f07cc0d0d50 2024-03-24T00:30:44.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07d1c00640 1 -- 172.21.15.53:0/2438154258 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07cc0ad280 con 0x7f07cc0d0d50 2024-03-24T00:30:44.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07d1c00640 1 -- 172.21.15.53:0/2438154258 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07cc0ad840 con 0x7f07cc0d0d50 2024-03-24T00:30:44.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07c2a00640 1 -- 172.21.15.53:0/2438154258 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f07b401a020 con 0x7f07cc0d0d50 2024-03-24T00:30:44.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07c2a00640 1 -- 172.21.15.53:0/2438154258 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f07b4013a60 con 0x7f07cc0d0d50 2024-03-24T00:30:44.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.463+0000 7f07d1c00640 1 -- 172.21.15.53:0/2438154258 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07cc068880 con 0x7f07cc0d0d50 2024-03-24T00:30:44.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.467+0000 7f07c2a00640 1 -- 172.21.15.53:0/2438154258 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f07b4021070 con 0x7f07cc0d0d50 2024-03-24T00:30:44.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.535+0000 7f07d1c00640 1 -- 172.21.15.53:0/2438154258 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ada23732-17bc-4ce0-b7e0-8974631732b0", "id": 7} v 0) v1 -- 0x7f07cc06eb90 con 0x7f07cc0d0d50 2024-03-24T00:30:44.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.539+0000 7f07c2a00640 1 -- 172.21.15.53:0/2438154258 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ada23732-17bc-4ce0-b7e0-8974631732b0", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f07b4013400 con 0x7f07cc0d0d50 2024-03-24T00:30:44.541 INFO:teuthology.orchestra.run.smithi053.stdout:7 2024-03-24T00:30:44.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.539+0000 7f07d1c00640 1 -- 172.21.15.53:0/2438154258 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07cc0d0d50 msgr2=0x7f07cc153370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.539+0000 7f07d1c00640 1 --2- 172.21.15.53:0/2438154258 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07cc0d0d50 0x7f07cc153370 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f07b400c430 tx=0x7f07b400c8f0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.539+0000 7f07d1c00640 1 -- 172.21.15.53:0/2438154258 shutdown_connections 2024-03-24T00:30:44.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.539+0000 7f07d1c00640 1 --2- 172.21.15.53:0/2438154258 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07cc15e100 0x7f07cc1538b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.539+0000 7f07d1c00640 1 --2- 172.21.15.53:0/2438154258 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07cc0d0d50 0x7f07cc153370 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.539+0000 7f07d1c00640 1 --2- 172.21.15.53:0/2438154258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07cc072f10 0x7f07cc1499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.539+0000 7f07d1c00640 1 -- 172.21.15.53:0/2438154258 >> 172.21.15.53:0/2438154258 conn(0x7f07cc0d0230 msgr2=0x7f07cc06d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:44.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.539+0000 7f07d1c00640 1 -- 172.21.15.53:0/2438154258 shutdown_connections 2024-03-24T00:30:44.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:30:44.539+0000 7f07d1c00640 1 -- 172.21.15.53:0/2438154258 wait complete. 2024-03-24T00:30:44.560 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-24T00:30:44.560 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-24T00:30:44.563 INFO:tasks.ceph.osd.0:Started 2024-03-24T00:30:44.563 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-24T00:30:44.563 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-24T00:30:44.566 INFO:tasks.ceph.osd.1:Started 2024-03-24T00:30:44.567 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-24T00:30:44.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-24T00:30:44.569 INFO:tasks.ceph.osd.2:Started 2024-03-24T00:30:44.570 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-24T00:30:44.570 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-24T00:30:44.573 INFO:tasks.ceph.osd.3:Started 2024-03-24T00:30:44.573 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-24T00:30:44.573 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-24T00:30:44.576 INFO:tasks.ceph.osd.4:Started 2024-03-24T00:30:44.576 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-24T00:30:44.577 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-24T00:30:44.579 INFO:tasks.ceph.osd.5:Started 2024-03-24T00:30:44.579 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-24T00:30:44.579 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-24T00:30:44.582 INFO:tasks.ceph.osd.6:Started 2024-03-24T00:30:44.582 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-24T00:30:44.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-24T00:30:44.590 INFO:tasks.ceph.osd.7:Started 2024-03-24T00:30:44.590 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:30:44.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.819+0000 7fb526a00640 1 Processor -- start 2024-03-24T00:30:44.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.819+0000 7fb526a00640 1 -- start start 2024-03-24T00:30:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb526a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb520141bc0 0x7fb5200ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb526a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5200ce630 0x7fb520071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb526a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb520070e00 con 0x7fb52013ec80 2024-03-24T00:30:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb526a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5200cee20 con 0x7fb5200ce630 2024-03-24T00:30:44.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb526a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5200cf100 con 0x7fb520141bc0 2024-03-24T00:30:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb524a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5200ce630 0x7fb520071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb524a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5200ce630 0x7fb520071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47350/0 (socket says 172.21.15.40:47350) 2024-03-24T00:30:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb524a00640 1 -- 172.21.15.40:0/2437260168 learned_addr learned my addr 172.21.15.40:0/2437260168 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb51f400640 1 --2- 172.21.15.40:0/2437260168 >> v2:172.21.15.53:3300/0 conn(0x7fb520141bc0 0x7fb5200ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb524a00640 1 -- 172.21.15.40:0/2437260168 >> v2:172.21.15.53:3300/0 conn(0x7fb520141bc0 msgr2=0x7fb5200ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb524a00640 1 --2- 172.21.15.40:0/2437260168 >> v2:172.21.15.53:3300/0 conn(0x7fb520141bc0 0x7fb5200ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb524a00640 1 -- 172.21.15.40:0/2437260168 >> v1:172.21.15.53:6789/0 conn(0x7fb52013ec80 legacy=0x7fb5201414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb524a00640 1 -- 172.21.15.40:0/2437260168 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5200cf3e0 con 0x7fb5200ce630 2024-03-24T00:30:44.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.823+0000 7fb51f400640 1 --2- 172.21.15.40:0/2437260168 >> v2:172.21.15.53:3300/0 conn(0x7fb520141bc0 0x7fb5200ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:30:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb524a00640 1 --2- 172.21.15.40:0/2437260168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5200ce630 0x7fb520071300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb52004fa60 tx=0x7fb51801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a813e24ec56f86 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb51ea00640 1 -- 172.21.15.40:0/2437260168 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5180a58e0 con 0x7fb5200ce630 2024-03-24T00:30:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb51ea00640 1 -- 172.21.15.40:0/2437260168 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5180a5a80 con 0x7fb5200ce630 2024-03-24T00:30:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb51ea00640 1 -- 172.21.15.40:0/2437260168 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5180a5e50 con 0x7fb5200ce630 2024-03-24T00:30:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 -- 172.21.15.40:0/2437260168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5200ce630 msgr2=0x7fb520071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 --2- 172.21.15.40:0/2437260168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5200ce630 0x7fb520071300 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb52004fa60 tx=0x7fb51801c9f0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 -- 172.21.15.40:0/2437260168 shutdown_connections 2024-03-24T00:30:44.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 --2- 172.21.15.40:0/2437260168 >> v2:172.21.15.53:3300/0 conn(0x7fb520141bc0 0x7fb5200ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 --2- 172.21.15.40:0/2437260168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5200ce630 0x7fb520071300 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 -- 172.21.15.40:0/2437260168 >> 172.21.15.40:0/2437260168 conn(0x7fb5200b17b0 msgr2=0x7fb5200b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:44.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 -- 172.21.15.40:0/2437260168 shutdown_connections 2024-03-24T00:30:44.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 -- 172.21.15.40:0/2437260168 wait complete. 2024-03-24T00:30:44.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 Processor -- start 2024-03-24T00:30:44.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 -- start start 2024-03-24T00:30:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5200ce630 0x7fb520156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52013ec80 0x7fb5200adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb520141bc0 0x7fb520159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5200723e0 con 0x7fb52013ec80 2024-03-24T00:30:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb520072260 con 0x7fb520141bc0 2024-03-24T00:30:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb526a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb520072560 con 0x7fb5200ce630 2024-03-24T00:30:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb51f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52013ec80 0x7fb5200adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb51f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52013ec80 0x7fb5200adb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47362/0 (socket says 172.21.15.40:47362) 2024-03-24T00:30:44.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb51f400640 1 -- 172.21.15.40:0/2058808887 learned_addr learned my addr 172.21.15.40:0/2058808887 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb51fe00640 1 --2- 172.21.15.40:0/2058808887 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5200ce630 0x7fb520156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.827+0000 7fb524a00640 1 --2- 172.21.15.40:0/2058808887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb520141bc0 0x7fb520159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb51f400640 1 -- 172.21.15.40:0/2058808887 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5200ce630 msgr2=0x7fb520156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb51f400640 1 --2- 172.21.15.40:0/2058808887 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5200ce630 0x7fb520156a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb51f400640 1 -- 172.21.15.40:0/2058808887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb520141bc0 msgr2=0x7fb520159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb51f400640 1 --2- 172.21.15.40:0/2058808887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb520141bc0 0x7fb520159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb51f400640 1 -- 172.21.15.40:0/2058808887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52006d500 con 0x7fb52013ec80 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb51f400640 1 --2- 172.21.15.40:0/2058808887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52013ec80 0x7fb5200adb40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00e810 tx=0x7fb50c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb517e00640 1 -- 172.21.15.40:0/2058808887 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c017070 con 0x7fb52013ec80 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb517e00640 1 -- 172.21.15.40:0/2058808887 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50c00c8d0 con 0x7fb52013ec80 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb517e00640 1 -- 172.21.15.40:0/2058808887 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c00f040 con 0x7fb52013ec80 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb526a00640 1 -- 172.21.15.40:0/2058808887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb52006d7e0 con 0x7fb52013ec80 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb526a00640 1 -- 172.21.15.40:0/2058808887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb52006dda0 con 0x7fb52013ec80 2024-03-24T00:30:44.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb517e00640 1 -- 172.21.15.40:0/2058808887 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb50c01a070 con 0x7fb52013ec80 2024-03-24T00:30:44.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb517e00640 1 -- 172.21.15.40:0/2058808887 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb50c01cd10 con 0x7fb52013ec80 2024-03-24T00:30:44.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.831+0000 7fb526a00640 1 -- 172.21.15.40:0/2058808887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb520068170 con 0x7fb52013ec80 2024-03-24T00:30:44.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.835+0000 7fb517e00640 1 -- 172.21.15.40:0/2058808887 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fb50c00ccc0 con 0x7fb52013ec80 2024-03-24T00:30:44.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.891+0000 7fb526a00640 1 -- 172.21.15.40:0/2058808887 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb52015ecb0 con 0x7fb52013ec80 2024-03-24T00:30:44.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.891+0000 7fb517e00640 1 -- 172.21.15.40:0/2058808887 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fb50c01c450 con 0x7fb52013ec80 2024-03-24T00:30:44.893 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:44.893 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":10,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:30:44.540889+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:30:44.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.891+0000 7fb526a00640 1 -- 172.21.15.40:0/2058808887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52013ec80 msgr2=0x7fb5200adb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:44.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.891+0000 7fb526a00640 1 --2- 172.21.15.40:0/2058808887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52013ec80 0x7fb5200adb40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00e810 tx=0x7fb50c00ecd0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.891+0000 7fb526a00640 1 -- 172.21.15.40:0/2058808887 shutdown_connections 2024-03-24T00:30:44.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.891+0000 7fb526a00640 1 --2- 172.21.15.40:0/2058808887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb520141bc0 0x7fb520159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.891+0000 7fb526a00640 1 --2- 172.21.15.40:0/2058808887 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb52013ec80 0x7fb5200adb40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.891+0000 7fb526a00640 1 --2- 172.21.15.40:0/2058808887 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5200ce630 0x7fb520156a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:44.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.891+0000 7fb526a00640 1 -- 172.21.15.40:0/2058808887 >> 172.21.15.40:0/2058808887 conn(0x7fb5200b17b0 msgr2=0x7fb5200b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:44.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.891+0000 7fb526a00640 1 -- 172.21.15.40:0/2058808887 shutdown_connections 2024-03-24T00:30:44.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:44.891+0000 7fb526a00640 1 -- 172.21.15.40:0/2058808887 wait complete. 2024-03-24T00:30:44.914 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-24T00:30:44.914 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-24T00:30:45.021 INFO:tasks.ceph.mgr.x.smithi053.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-24T00:30:45.021 INFO:tasks.ceph.mgr.x.smithi053.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-24T00:30:45.022 INFO:tasks.ceph.mgr.x.smithi053.stderr: from numpy import show_config as show_numpy_config 2024-03-24T00:30:45.076 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:45.075+0000 7f4e6efc2180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:30:45.076 INFO:tasks.ceph.mgr.y.smithi040.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-03-24T00:30:45.076 INFO:tasks.ceph.mgr.y.smithi040.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-03-24T00:30:45.076 INFO:tasks.ceph.mgr.y.smithi040.stderr: from numpy import show_config as show_numpy_config 2024-03-24T00:30:45.126 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:45.127+0000 7fb50d43a180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:30:45.223 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-03-24T00:30:45.223+0000 7f3b6c02c7c0 -1 Falling back to public interface 2024-03-24T00:30:45.231 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:45.227+0000 7f4e6efc2180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:30:45.240 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-24T00:30:45.239+0000 7f7d20c807c0 -1 Falling back to public interface 2024-03-24T00:30:45.241 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-24T00:30:45.239+0000 7f964a6c17c0 -1 Falling back to public interface 2024-03-24T00:30:45.241 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-24T00:30:45.239+0000 7f7d1f800640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6803/0: (98) Address already in use 2024-03-24T00:30:45.251 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-03-24T00:30:45.251+0000 7fbd632657c0 -1 Falling back to public interface 2024-03-24T00:30:45.252 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-03-24T00:30:45.251+0000 7fa1392957c0 -1 Falling back to public interface 2024-03-24T00:30:45.252 INFO:tasks.ceph.osd.3.smithi040.stderr:2024-03-24T00:30:45.251+0000 7f379755a7c0 -1 Falling back to public interface 2024-03-24T00:30:45.252 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-24T00:30:45.251+0000 7fa2b0d387c0 -1 Falling back to public interface 2024-03-24T00:30:45.256 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-24T00:30:45.255+0000 7f3d8d19b7c0 -1 Falling back to public interface 2024-03-24T00:30:45.288 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:45.287+0000 7fb50d43a180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:30:45.417 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:45.415+0000 7f4e6efc2180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:30:45.484 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:45.483+0000 7fb50d43a180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:30:45.493 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:45.491+0000 7f4e6efc2180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:30:45.561 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:45.559+0000 7fb50d43a180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:30:45.651 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:45.647+0000 7f4e6efc2180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:30:45.719 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T00:30:45.720 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:45.719+0000 7fb50d43a180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:30:45.851 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad4ee00640 1 Processor -- start 2024-03-24T00:30:45.851 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad4ee00640 1 -- start start 2024-03-24T00:30:45.852 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad4ee00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fad4806df20 0x7fad48166c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:45.852 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad4ee00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad48072b80 con 0x7fad480726f0 2024-03-24T00:30:45.852 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad4ee00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad48071680 con 0x7fad48145550 2024-03-24T00:30:45.852 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad4ee00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad48071fb0 con 0x7fad4806df20 2024-03-24T00:30:45.852 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad3fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fad480726f0 0x7fad480ce5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52852/0 (socket says 172.21.15.40:52852) 2024-03-24T00:30:45.852 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad3fe00640 1 -- 172.21.15.40:0/2485592667 learned_addr learned my addr 172.21.15.40:0/2485592667 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:30:45.852 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad4ce00640 1 --2- 172.21.15.40:0/2485592667 >> v2:172.21.15.53:3300/0 conn(0x7fad4806df20 0x7fad48166c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:45.854 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331195758 0 0) 0x7fad48072b80 con 0x7fad480726f0 2024-03-24T00:30:45.854 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad2c003530 con 0x7fad480726f0 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431971451 0 0) 0x7fad48071680 con 0x7fad48145550 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad48072b80 con 0x7fad48145550 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 948030867 0 0) 0x7fad2c003530 con 0x7fad480726f0 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 >> v2:172.21.15.53:3300/0 conn(0x7fad4806df20 msgr2=0x7fad48166c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 --2- 172.21.15.40:0/2485592667 >> v2:172.21.15.53:3300/0 conn(0x7fad4806df20 0x7fad48166c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 >> v1:172.21.15.53:6789/0 conn(0x7fad48145550 legacy=0x7fad480cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad48071960 con 0x7fad480726f0 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fad30001680 con 0x7fad480726f0 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad4ce00640 1 --2- 172.21.15.40:0/2485592667 >> v2:172.21.15.53:3300/0 conn(0x7fad4806df20 0x7fad48166c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 >> v1:172.21.15.40:6789/0 conn(0x7fad480726f0 legacy=0x7fad480ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 --2- 172.21.15.40:0/2485592667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad2c003b80 0x7fad2c003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 --2- 172.21.15.40:0/2485592667 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad2c004810 0x7fad2c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 --2- 172.21.15.40:0/2485592667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad2c005270 0x7fad2c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad48071fb0 con 0x7fad2c003b80 2024-03-24T00:30:45.855 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c00c1c0 con 0x7fad2c005270 2024-03-24T00:30:45.856 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad2c0107d0 con 0x7fad2c004810 2024-03-24T00:30:45.856 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47e00640 1 --2- 172.21.15.40:0/2485592667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad2c003b80 0x7fad2c003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:45.856 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad4ce00640 1 --2- 172.21.15.40:0/2485592667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad2c005270 0x7fad2c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:45.856 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad3fe00640 1 --2- 172.21.15.40:0/2485592667 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad2c004810 0x7fad2c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:45.856 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47e00640 1 -- 172.21.15.40:0/2485592667 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad2c004810 msgr2=0x7fad2c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:45.856 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47e00640 1 --2- 172.21.15.40:0/2485592667 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad2c004810 0x7fad2c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:45.856 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47e00640 1 -- 172.21.15.40:0/2485592667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad2c005270 msgr2=0x7fad2c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:45.856 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47e00640 1 --2- 172.21.15.40:0/2485592667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad2c005270 0x7fad2c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:45.856 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad47e00640 1 -- 172.21.15.40:0/2485592667 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2c003530 con 0x7fad2c003b80 2024-03-24T00:30:45.856 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.851+0000 7fad4ce00640 1 -- 172.21.15.40:0/2485592667 reap_dead start 2024-03-24T00:30:45.857 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.855+0000 7fad47e00640 1 --2- 172.21.15.40:0/2485592667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad2c003b80 0x7fad2c003f80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fad380018d0 tx=0x7fad3801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c05b24023f23fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:45.857 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.855+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38001520 con 0x7fad2c003b80 2024-03-24T00:30:45.857 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.855+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad380052f0 con 0x7fad2c003b80 2024-03-24T00:30:45.857 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.855+0000 7fad47400640 1 -- 172.21.15.40:0/2485592667 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad38003840 con 0x7fad2c003b80 2024-03-24T00:30:45.858 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.855+0000 7fad4ee00640 1 -- 172.21.15.40:0/2485592667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad2c003b80 msgr2=0x7fad2c003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:45.858 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.855+0000 7fad4ee00640 1 --2- 172.21.15.40:0/2485592667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad2c003b80 0x7fad2c003f80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fad380018d0 tx=0x7fad3801c500 comp rx=0 tx=0).stop 2024-03-24T00:30:45.858 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.855+0000 7fad4ee00640 1 -- 172.21.15.40:0/2485592667 shutdown_connections 2024-03-24T00:30:45.858 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.855+0000 7fad4ee00640 1 --2- 172.21.15.40:0/2485592667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad2c003b80 0x7fad2c003f80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:45.858 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.855+0000 7fad4ee00640 1 -- 172.21.15.40:0/2485592667 >> 172.21.15.40:0/2485592667 conn(0x7fad480b1770 msgr2=0x7fad480b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:45.858 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.855+0000 7fad4ee00640 1 -- 172.21.15.40:0/2485592667 shutdown_connections 2024-03-24T00:30:45.858 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.855+0000 7fad4ee00640 1 -- 172.21.15.40:0/2485592667 wait complete. 2024-03-24T00:30:45.859 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad4ee00640 1 Processor -- start 2024-03-24T00:30:45.859 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad4ee00640 1 -- start start 2024-03-24T00:30:45.861 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad4ee00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad2c003b80 0x7fad48159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:45.861 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad4ee00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad48143e30 0x7fad48144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:45.861 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad4ee00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad480ce3f0 0x7fad480ce870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:45.861 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad4ee00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4806da90 con 0x7fad48143e30 2024-03-24T00:30:45.861 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad4ee00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4806d910 con 0x7fad480ce3f0 2024-03-24T00:30:45.861 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad4ee00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4806dc10 con 0x7fad2c003b80 2024-03-24T00:30:45.861 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad3fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad48143e30 0x7fad48144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:45.862 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad3fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad48143e30 0x7fad48144290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47378/0 (socket says 172.21.15.40:47378) 2024-03-24T00:30:45.862 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad3fe00640 1 -- 172.21.15.40:0/3613116310 learned_addr learned my addr 172.21.15.40:0/3613116310 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:45.862 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad47e00640 1 --2- 172.21.15.40:0/3613116310 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad2c003b80 0x7fad48159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:45.862 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad4ce00640 1 --2- 172.21.15.40:0/3613116310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad480ce3f0 0x7fad480ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:45.862 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad3fe00640 1 -- 172.21.15.40:0/3613116310 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad2c003b80 msgr2=0x7fad48159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:45.862 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad3fe00640 1 --2- 172.21.15.40:0/3613116310 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad2c003b80 0x7fad48159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:45.862 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad3fe00640 1 -- 172.21.15.40:0/3613116310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad480ce3f0 msgr2=0x7fad480ce870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:45.862 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad3fe00640 1 --2- 172.21.15.40:0/3613116310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad480ce3f0 0x7fad480ce870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:45.862 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad3fe00640 1 -- 172.21.15.40:0/3613116310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad48140170 con 0x7fad48143e30 2024-03-24T00:30:45.862 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad3fe00640 1 --2- 172.21.15.40:0/3613116310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad48143e30 0x7fad48144290 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fad30004ed0 tx=0x7fad30007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:45.863 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad44c00640 1 -- 172.21.15.40:0/3613116310 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3000f750 con 0x7fad48143e30 2024-03-24T00:30:45.863 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad44c00640 1 -- 172.21.15.40:0/3613116310 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad30018070 con 0x7fad48143e30 2024-03-24T00:30:45.863 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad4ee00640 1 -- 172.21.15.40:0/3613116310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad481f6e40 con 0x7fad48143e30 2024-03-24T00:30:45.863 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad44c00640 1 -- 172.21.15.40:0/3613116310 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3000faf0 con 0x7fad48143e30 2024-03-24T00:30:45.864 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.859+0000 7fad4ee00640 1 -- 172.21.15.40:0/3613116310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4813f3b0 con 0x7fad48143e30 2024-03-24T00:30:45.864 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.863+0000 7fad44c00640 1 -- 172.21.15.40:0/3613116310 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fad3000fc90 con 0x7fad48143e30 2024-03-24T00:30:45.864 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.863+0000 7fad44c00640 1 -- 172.21.15.40:0/3613116310 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fad300133f0 con 0x7fad48143e30 2024-03-24T00:30:45.864 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.863+0000 7fad4ee00640 1 -- 172.21.15.40:0/3613116310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4804f410 con 0x7fad48143e30 2024-03-24T00:30:45.865 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.863+0000 7fad44c00640 1 -- 172.21.15.40:0/3613116310 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fad3001d460 con 0x7fad48143e30 2024-03-24T00:30:45.924 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.923+0000 7fad4ee00640 1 -- 172.21.15.40:0/3613116310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad48071270 con 0x7fad48143e30 2024-03-24T00:30:45.924 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.923+0000 7fad44c00640 1 -- 172.21.15.40:0/3613116310 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fad3001d640 con 0x7fad48143e30 2024-03-24T00:30:45.924 INFO:teuthology.misc.health.smithi040.stdout: 2024-03-24T00:30:45.925 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":10,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:30:44.540889+0000","last_up_change":"0.000000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:30:45.926 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.923+0000 7fad4ee00640 1 -- 172.21.15.40:0/3613116310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad48143e30 msgr2=0x7fad48144290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:45.926 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.923+0000 7fad4ee00640 1 --2- 172.21.15.40:0/3613116310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad48143e30 0x7fad48144290 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fad30004ed0 tx=0x7fad30007430 comp rx=0 tx=0).stop 2024-03-24T00:30:45.926 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.923+0000 7fad4ee00640 1 -- 172.21.15.40:0/3613116310 shutdown_connections 2024-03-24T00:30:45.926 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.923+0000 7fad4ee00640 1 --2- 172.21.15.40:0/3613116310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad480ce3f0 0x7fad480ce870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:45.926 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.923+0000 7fad4ee00640 1 --2- 172.21.15.40:0/3613116310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad48143e30 0x7fad48144290 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:45.926 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.923+0000 7fad4ee00640 1 --2- 172.21.15.40:0/3613116310 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad2c003b80 0x7fad48159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:45.926 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.923+0000 7fad4ee00640 1 -- 172.21.15.40:0/3613116310 >> 172.21.15.40:0/3613116310 conn(0x7fad480b1770 msgr2=0x7fad4815b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:45.926 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.923+0000 7fad4ee00640 1 -- 172.21.15.40:0/3613116310 shutdown_connections 2024-03-24T00:30:45.926 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:45.923+0000 7fad4ee00640 1 -- 172.21.15.40:0/3613116310 wait complete. 2024-03-24T00:30:45.942 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-24T00:30:46.360 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:46.359+0000 7f4e6efc2180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:30:46.443 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:46.443+0000 7fb50d43a180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:30:46.451 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:46.451+0000 7f4e6efc2180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:30:46.535 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:46.535+0000 7fb50d43a180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:30:46.664 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:46.663+0000 7f4e6efc2180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:30:46.747 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:46.747+0000 7fb50d43a180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:30:46.753 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:46.751+0000 7f4e6efc2180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:30:46.842 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:46.839+0000 7fb50d43a180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:30:47.106 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:47.103+0000 7f4e6efc2180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:30:47.180 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:47.179+0000 7f4e6efc2180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:30:47.189 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:47.187+0000 7fb50d43a180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:30:47.266 INFO:tasks.ceph.mgr.y.smithi040.stderr:2024-03-24T00:30:47.263+0000 7fb50d43a180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:30:48.202 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:48.199+0000 7f4e15000640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-24T00:30:48.204 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:30:48.203+0000 7f4e15000640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-24T00:30:48.977 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-03-24T00:30:48.975+0000 7f3b6c02c7c0 -1 osd.2 0 log_to_monitors true 2024-03-24T00:30:48.999 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-24T00:30:48.995+0000 7f964a6c17c0 -1 osd.4 0 log_to_monitors true 2024-03-24T00:30:49.108 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-24T00:30:49.107+0000 7f7d20c807c0 -1 osd.5 0 log_to_monitors true 2024-03-24T00:30:49.121 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-24T00:30:49.119+0000 7f3d8d19b7c0 -1 osd.7 0 log_to_monitors true 2024-03-24T00:30:49.131 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-03-24T00:30:49.131+0000 7fa1392957c0 -1 osd.0 0 log_to_monitors true 2024-03-24T00:30:49.147 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-24T00:30:49.143+0000 7fa2b0d387c0 -1 osd.6 0 log_to_monitors true 2024-03-24T00:30:49.151 INFO:tasks.ceph.osd.3.smithi040.stderr:2024-03-24T00:30:49.151+0000 7f379755a7c0 -1 osd.3 0 log_to_monitors true 2024-03-24T00:30:49.172 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-03-24T00:30:49.171+0000 7fbd632657c0 -1 osd.1 0 log_to_monitors true 2024-03-24T00:30:51.960 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-24T00:30:51.959+0000 7f7d1bc00640 -1 osd.5 0 waiting for initial osdmap 2024-03-24T00:30:51.965 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-24T00:30:51.963+0000 7fa2ac600640 -1 osd.6 0 waiting for initial osdmap 2024-03-24T00:30:51.971 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-24T00:30:51.967+0000 7f9646000640 -1 osd.4 0 waiting for initial osdmap 2024-03-24T00:30:51.989 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-24T00:30:51.987+0000 7f7d11c00640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:30:51.995 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-24T00:30:51.995+0000 7f3d88000640 -1 osd.7 0 waiting for initial osdmap 2024-03-24T00:30:52.004 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-24T00:30:52.003+0000 7fa2a1c00640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:30:52.020 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-24T00:30:52.019+0000 7f963b600640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:30:52.029 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-24T00:30:52.027+0000 7f3d7e000640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:30:52.514 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-03-24T00:30:52.511+0000 7fbd5e200640 -1 osd.1 0 waiting for initial osdmap 2024-03-24T00:30:52.517 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-03-24T00:30:52.515+0000 7fa134200640 -1 osd.0 0 waiting for initial osdmap 2024-03-24T00:30:52.534 INFO:tasks.ceph.osd.3.smithi040.stderr:2024-03-24T00:30:52.535+0000 7f3792400640 -1 osd.3 0 waiting for initial osdmap 2024-03-24T00:30:52.550 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-03-24T00:30:52.547+0000 7f3b66400640 -1 osd.2 0 waiting for initial osdmap 2024-03-24T00:30:52.559 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-03-24T00:30:52.559+0000 7fbd54200640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:30:52.559 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-03-24T00:30:52.559+0000 7fa12a200640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:30:52.570 INFO:tasks.ceph.osd.3.smithi040.stderr:2024-03-24T00:30:52.571+0000 7f3788400640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:30:52.599 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-03-24T00:30:52.599+0000 7f3b5c400640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:30:52.748 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-24T00:30:52.878 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7002c00640 1 Processor -- start 2024-03-24T00:30:52.879 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7002c00640 1 -- start start 2024-03-24T00:30:52.880 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7002c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6ffc0cfd30 0x7f6ffc15eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:52.880 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7002c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc0b2c60 0x7f6ffc15f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:52.880 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7002c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ffc0b2a50 con 0x7f6ffc0d0110 2024-03-24T00:30:52.880 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7002c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc0b1cd0 con 0x7f6ffc0cfd30 2024-03-24T00:30:52.880 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7002c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc0b3200 con 0x7f6ffc0b2c60 2024-03-24T00:30:52.881 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7002200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc0b2c60 0x7f6ffc15f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:52.881 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7001800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6ffc0cfd30 0x7f6ffc15eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:52.881 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7001800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6ffc0cfd30 0x7f6ffc15eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:38822/0 (socket says 172.21.15.40:38822) 2024-03-24T00:30:52.881 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7001800640 1 -- 172.21.15.40:0/3934556161 learned_addr learned my addr 172.21.15.40:0/3934556161 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:52.881 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7000e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6ffc0d0110 0x7f6ffc15fb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:43932/0 (socket says 172.21.15.40:43932) 2024-03-24T00:30:52.881 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7002200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc0b2c60 0x7f6ffc15f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:40676/0 (socket says 172.21.15.40:40676) 2024-03-24T00:30:52.881 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7001800640 1 -- 172.21.15.40:0/3934556161 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc0b2c60 msgr2=0x7f6ffc15f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:52.881 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7001800640 1 --2- 172.21.15.40:0/3934556161 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc0b2c60 0x7f6ffc15f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:52.881 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7001800640 1 -- 172.21.15.40:0/3934556161 >> v1:172.21.15.53:6789/0 conn(0x7f6ffc0d0110 legacy=0x7f6ffc15fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:52.881 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.879+0000 7f7001800640 1 -- 172.21.15.40:0/3934556161 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc0b1f70 con 0x7f6ffc0cfd30 2024-03-24T00:30:52.883 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f7001800640 1 --2- 172.21.15.40:0/3934556161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0cfd30 0x7f6ffc15eef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009870 tx=0x7f6ff001c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f938df9424e6ab9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:52.883 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f6ffbe00640 1 -- 172.21.15.40:0/3934556161 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff00b1070 con 0x7f6ffc0cfd30 2024-03-24T00:30:52.883 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f6ffbe00640 1 -- 172.21.15.40:0/3934556161 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff00a8050 con 0x7f6ffc0cfd30 2024-03-24T00:30:52.884 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f6ffbe00640 1 -- 172.21.15.40:0/3934556161 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff00ac860 con 0x7f6ffc0cfd30 2024-03-24T00:30:52.884 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f7002c00640 1 -- 172.21.15.40:0/3934556161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0cfd30 msgr2=0x7f6ffc15eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:52.885 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f7002c00640 1 --2- 172.21.15.40:0/3934556161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0cfd30 0x7f6ffc15eef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0009870 tx=0x7f6ff001c4d0 comp rx=0 tx=0).stop 2024-03-24T00:30:52.885 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f7002c00640 1 -- 172.21.15.40:0/3934556161 shutdown_connections 2024-03-24T00:30:52.885 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f7002c00640 1 --2- 172.21.15.40:0/3934556161 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc0b2c60 0x7f6ffc15f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:52.885 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f7002c00640 1 --2- 172.21.15.40:0/3934556161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0cfd30 0x7f6ffc15eef0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:52.885 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f7002c00640 1 -- 172.21.15.40:0/3934556161 >> 172.21.15.40:0/3934556161 conn(0x7f6ffc0b3570 msgr2=0x7f6ffc0b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:52.885 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f7002c00640 1 -- 172.21.15.40:0/3934556161 shutdown_connections 2024-03-24T00:30:52.885 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f7002c00640 1 -- 172.21.15.40:0/3934556161 wait complete. 2024-03-24T00:30:52.886 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f7002c00640 1 Processor -- start 2024-03-24T00:30:52.886 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.883+0000 7f7002c00640 1 -- start start 2024-03-24T00:30:52.888 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7002c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc0b2c60 0x7f6ffc156c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:52.888 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7002c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc0cfd30 0x7f6ffc1faa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7002c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0d0110 0x7f6ffc1fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7002c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc168e90 con 0x7f6ffc0b2c60 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7002c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc0b4270 con 0x7f6ffc0d0110 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7002c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc169010 con 0x7f6ffc0cfd30 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7001800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc0b2c60 0x7f6ffc156c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7001800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc0b2c60 0x7f6ffc156c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39908/0 (socket says 172.21.15.40:39908) 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7001800640 1 -- 172.21.15.40:0/2358239250 learned_addr learned my addr 172.21.15.40:0/2358239250 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7000e00640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc0cfd30 0x7f6ffc1faa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7002200640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0d0110 0x7f6ffc1fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7001800640 1 -- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc0cfd30 msgr2=0x7f6ffc1faa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7001800640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc0cfd30 0x7f6ffc1faa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7001800640 1 -- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0d0110 msgr2=0x7f6ffc1fafb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7001800640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0d0110 0x7f6ffc1fafb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:52.889 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7001800640 1 -- 172.21.15.40:0/2358239250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc165d40 con 0x7f6ffc0b2c60 2024-03-24T00:30:52.890 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7002200640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0d0110 0x7f6ffc1fafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:30:52.890 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7001800640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc0b2c60 0x7f6ffc156c10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6ff001ab80 tx=0x7f6ff001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:52.890 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f6ff9600640 1 -- 172.21.15.40:0/2358239250 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff00bb020 con 0x7f6ffc0b2c60 2024-03-24T00:30:52.890 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f6ff9600640 1 -- 172.21.15.40:0/2358239250 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff00ace10 con 0x7f6ffc0b2c60 2024-03-24T00:30:52.890 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7002c00640 1 -- 172.21.15.40:0/2358239250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc166020 con 0x7f6ffc0b2c60 2024-03-24T00:30:52.890 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f6ff9600640 1 -- 172.21.15.40:0/2358239250 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff00b1040 con 0x7f6ffc0b2c60 2024-03-24T00:30:52.890 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7002c00640 1 -- 172.21.15.40:0/2358239250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc1665e0 con 0x7f6ffc0b2c60 2024-03-24T00:30:52.890 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.887+0000 7f7002c00640 1 -- 172.21.15.40:0/2358239250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ffc04f410 con 0x7f6ffc0b2c60 2024-03-24T00:30:52.891 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.891+0000 7f6ff9600640 1 -- 172.21.15.40:0/2358239250 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f6ff00be020 con 0x7f6ffc0b2c60 2024-03-24T00:30:52.891 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.891+0000 7f6ff9600640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fdc0675d0 0x7f6fdc069a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:52.891 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.891+0000 7f6ff9600640 1 -- 172.21.15.40:0/2358239250 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f6ff011f080 con 0x7f6ffc0b2c60 2024-03-24T00:30:52.892 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.891+0000 7f7000e00640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fdc0675d0 0x7f6fdc069a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:52.892 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.891+0000 7f7000e00640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fdc0675d0 0x7f6fdc069a90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc1584f0 tx=0x7f6fe8007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:52.895 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:52.895+0000 7f6ff9600640 1 -- 172.21.15.40:0/2358239250 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6ff00c5070 con 0x7f6ffc0b2c60 2024-03-24T00:30:53.071 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 -- 172.21.15.40:0/2358239250 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ffc0cc040 con 0x7f6ffc0b2c60 2024-03-24T00:30:53.072 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f6ff9600640 1 -- 172.21.15.40:0/2358239250 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10849 (secure 0 0 0) 0x7f6ff0016150 con 0x7f6ffc0b2c60 2024-03-24T00:30:53.072 INFO:teuthology.misc.health.smithi040.stdout: 2024-03-24T00:30:53.072 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":13,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:30:52.990515+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:30:53.074 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 -- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fdc0675d0 msgr2=0x7f6fdc069a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.074 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fdc0675d0 0x7f6fdc069a90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc1584f0 tx=0x7f6fe8007e90 comp rx=0 tx=0).stop 2024-03-24T00:30:53.074 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 -- 172.21.15.40:0/2358239250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc0b2c60 msgr2=0x7f6ffc156c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.074 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc0b2c60 0x7f6ffc156c10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6ff001ab80 tx=0x7f6ff001ad10 comp rx=0 tx=0).stop 2024-03-24T00:30:53.074 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 -- 172.21.15.40:0/2358239250 shutdown_connections 2024-03-24T00:30:53.074 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fdc0675d0 0x7f6fdc069a90 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc1584f0 tx=0x7f6fe8007e90 comp rx=0 tx=0).stop 2024-03-24T00:30:53.074 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0d0110 0x7f6ffc1fafb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.075 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc0cfd30 0x7f6ffc1faa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.075 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 --2- 172.21.15.40:0/2358239250 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc0b2c60 0x7f6ffc156c10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.075 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 -- 172.21.15.40:0/2358239250 >> 172.21.15.40:0/2358239250 conn(0x7f6ffc0b3570 msgr2=0x7f6ffc06f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:53.075 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 -- 172.21.15.40:0/2358239250 shutdown_connections 2024-03-24T00:30:53.075 INFO:teuthology.misc.health.smithi040.stderr:2024-03-24T00:30:53.071+0000 7f7002c00640 1 -- 172.21.15.40:0/2358239250 wait complete. 2024-03-24T00:30:53.094 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-24T00:30:53.094 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-24T00:30:53.095 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:30:53.095 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:30:53.098 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:30:53.098 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:30:53.098 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:30:53.101 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:30:53.101 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:30:53.101 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:30:53.104 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:30:53.104 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:30:53.104 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:30:53.107 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:30:53.108 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-24T00:30:53.108 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:30:53.195 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:30:53.197 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:30:53.249 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:30:53.249 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:30:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.259+0000 7f7dd4200640 1 Processor -- start 2024-03-24T00:30:53.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.259+0000 7f7dd4200640 1 -- start start 2024-03-24T00:30:53.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.259+0000 7f7dd4200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7dcc0b2aa0 0x7f7dcc0d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:53.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.259+0000 7f7dd4200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7dcc0d0a80 0x7f7dcc0cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:53.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.259+0000 7f7dd4200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dcc0b2870 con 0x7f7dcc0b2490 2024-03-24T00:30:53.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.259+0000 7f7dd4200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dcc06b6d0 con 0x7f7dcc0d0a80 2024-03-24T00:30:53.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.259+0000 7f7dd4200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dcc0b3190 con 0x7f7dcc0b2aa0 2024-03-24T00:30:53.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.259+0000 7f7dd1800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7dcc0b2490 0x7f7dcc0cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:43958/0 (socket says 172.21.15.40:43958) 2024-03-24T00:30:53.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.259+0000 7f7dd1800640 1 -- 172.21.15.40:0/2036334882 learned_addr learned my addr 172.21.15.40:0/2036334882 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:30:53.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.259+0000 7f7dd2200640 1 --2- 172.21.15.40:0/2036334882 >> v2:172.21.15.40:3300/0 conn(0x7f7dcc0d0a80 0x7f7dcc0cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:53.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.259+0000 7f7dd0e00640 1 --2- 172.21.15.40:0/2036334882 >> v2:172.21.15.53:3300/0 conn(0x7f7dcc0b2aa0 0x7f7dcc0d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:53.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.263+0000 7f7dd0e00640 1 -- 172.21.15.40:0/2036334882 >> v2:172.21.15.40:3300/0 conn(0x7f7dcc0d0a80 msgr2=0x7f7dcc0cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:53.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.263+0000 7f7dd0e00640 1 --2- 172.21.15.40:0/2036334882 >> v2:172.21.15.40:3300/0 conn(0x7f7dcc0d0a80 0x7f7dcc0cc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.263+0000 7f7dd0e00640 1 -- 172.21.15.40:0/2036334882 >> v1:172.21.15.53:6789/0 conn(0x7f7dcc0b2490 legacy=0x7f7dcc0cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.263+0000 7f7dd0e00640 1 -- 172.21.15.40:0/2036334882 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc0b3470 con 0x7f7dcc0b2aa0 2024-03-24T00:30:53.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.263+0000 7f7dcbe00640 1 -- 172.21.15.40:0/2036334882 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120659337 0 0) 0x7f7dcc0b2870 con 0x7f7dcc0b2490 2024-03-24T00:30:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.263+0000 7f7dd0e00640 1 --2- 172.21.15.40:0/2036334882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0b2aa0 0x7f7dcc0d0540 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc072db0 tx=0x7f7db801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd15a1d23a124d7e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.263+0000 7f7dcbe00640 1 -- 172.21.15.40:0/2036334882 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db80b1070 con 0x7f7dcc0b2aa0 2024-03-24T00:30:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.263+0000 7f7dcbe00640 1 -- 172.21.15.40:0/2036334882 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db80ac5f0 con 0x7f7dcc0b2aa0 2024-03-24T00:30:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.263+0000 7f7dcbe00640 1 -- 172.21.15.40:0/2036334882 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db80ac9c0 con 0x7f7dcc0b2aa0 2024-03-24T00:30:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 -- 172.21.15.40:0/2036334882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0b2aa0 msgr2=0x7f7dcc0d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 --2- 172.21.15.40:0/2036334882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0b2aa0 0x7f7dcc0d0540 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc072db0 tx=0x7f7db801c480 comp rx=0 tx=0).stop 2024-03-24T00:30:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 -- 172.21.15.40:0/2036334882 shutdown_connections 2024-03-24T00:30:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 --2- 172.21.15.40:0/2036334882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0b2aa0 0x7f7dcc0d0540 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 --2- 172.21.15.40:0/2036334882 >> v2:172.21.15.40:3300/0 conn(0x7f7dcc0d0a80 0x7f7dcc0cc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 -- 172.21.15.40:0/2036334882 >> 172.21.15.40:0/2036334882 conn(0x7f7dcc0d1d10 msgr2=0x7f7dcc0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 -- 172.21.15.40:0/2036334882 shutdown_connections 2024-03-24T00:30:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 -- 172.21.15.40:0/2036334882 wait complete. 2024-03-24T00:30:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 Processor -- start 2024-03-24T00:30:53.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 -- start start 2024-03-24T00:30:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dcc0b2490 0x7f7dcc14f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0b2aa0 0x7f7dcc0aea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0d0a80 0x7f7dcc15f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc0cd580 con 0x7f7dcc0b2490 2024-03-24T00:30:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc0cd400 con 0x7f7dcc0b2aa0 2024-03-24T00:30:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd4200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc0cd700 con 0x7f7dcc0d0a80 2024-03-24T00:30:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd1800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dcc0b2490 0x7f7dcc14f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd1800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dcc0b2490 0x7f7dcc14f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39966/0 (socket says 172.21.15.40:39966) 2024-03-24T00:30:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd1800640 1 -- 172.21.15.40:0/593811744 learned_addr learned my addr 172.21.15.40:0/593811744 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd2200640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0d0a80 0x7f7dcc15f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:53.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd0e00640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0b2aa0 0x7f7dcc0aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:53.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd1800640 1 -- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0d0a80 msgr2=0x7f7dcc15f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd1800640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0d0a80 0x7f7dcc15f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.267+0000 7f7dd1800640 1 -- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0b2aa0 msgr2=0x7f7dcc0aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.271+0000 7f7dd1800640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0b2aa0 0x7f7dcc0aea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.271+0000 7f7dd1800640 1 -- 172.21.15.40:0/593811744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc15a740 con 0x7f7dcc0b2490 2024-03-24T00:30:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.271+0000 7f7dd1800640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dcc0b2490 0x7f7dcc14f200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000c410 tx=0x7f7dc000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.271+0000 7f7dc9600640 1 -- 172.21.15.40:0/593811744 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc00074e0 con 0x7f7dcc0b2490 2024-03-24T00:30:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.271+0000 7f7dc9600640 1 -- 172.21.15.40:0/593811744 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc00047f0 con 0x7f7dcc0b2490 2024-03-24T00:30:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.271+0000 7f7dc9600640 1 -- 172.21.15.40:0/593811744 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0007680 con 0x7f7dcc0b2490 2024-03-24T00:30:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.271+0000 7f7dd4200640 1 -- 172.21.15.40:0/593811744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dcc151190 con 0x7f7dcc0b2490 2024-03-24T00:30:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.271+0000 7f7dd4200640 1 -- 172.21.15.40:0/593811744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dcc15a180 con 0x7f7dcc0b2490 2024-03-24T00:30:53.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.271+0000 7f7dd4200640 1 -- 172.21.15.40:0/593811744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dcc15f580 con 0x7f7dcc0b2490 2024-03-24T00:30:53.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.271+0000 7f7dc9600640 1 -- 172.21.15.40:0/593811744 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f7dc0011070 con 0x7f7dcc0b2490 2024-03-24T00:30:53.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.271+0000 7f7dc9600640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7dac0675d0 0x7f7dac069a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:53.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.271+0000 7f7dc9600640 1 -- 172.21.15.40:0/593811744 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f7dc0084cf0 con 0x7f7dcc0b2490 2024-03-24T00:30:53.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.275+0000 7f7dd0e00640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7dac0675d0 0x7f7dac069a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:53.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.275+0000 7f7dd0e00640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7dac0675d0 0x7f7dac069a90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7db80ae000 tx=0x7f7db80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:53.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.279+0000 7f7dc9600640 1 -- 172.21.15.40:0/593811744 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7dc0055960 con 0x7f7dcc0b2490 2024-03-24T00:30:53.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.439+0000 7f7dd4200640 1 -- 172.21.15.40:0/593811744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dcc068170 con 0x7f7dcc0b2490 2024-03-24T00:30:53.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.439+0000 7f7dc9600640 1 -- 172.21.15.40:0/593811744 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10849 (secure 0 0 0) 0x7f7dc0059a00 con 0x7f7dcc0b2490 2024-03-24T00:30:53.442 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:53.442 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":13,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:30:52.990515+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:30:53.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.443+0000 7f7dd4200640 1 -- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7dac0675d0 msgr2=0x7f7dac069a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.443+0000 7f7dd4200640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7dac0675d0 0x7f7dac069a90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7db80ae000 tx=0x7f7db80a7000 comp rx=0 tx=0).stop 2024-03-24T00:30:53.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.443+0000 7f7dd4200640 1 -- 172.21.15.40:0/593811744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dcc0b2490 msgr2=0x7f7dcc14f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.443+0000 7f7dd4200640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dcc0b2490 0x7f7dcc14f200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000c410 tx=0x7f7dc000c8d0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.443+0000 7f7dd4200640 1 -- 172.21.15.40:0/593811744 shutdown_connections 2024-03-24T00:30:53.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.443+0000 7f7dd4200640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7dac0675d0 0x7f7dac069a90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.443+0000 7f7dd4200640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0d0a80 0x7f7dcc15f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.443+0000 7f7dd4200640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0b2aa0 0x7f7dcc0aea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.443+0000 7f7dd4200640 1 --2- 172.21.15.40:0/593811744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7dcc0b2490 0x7f7dcc14f200 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.443+0000 7f7dd4200640 1 -- 172.21.15.40:0/593811744 >> 172.21.15.40:0/593811744 conn(0x7f7dcc0d1d10 msgr2=0x7f7dcc0c97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:53.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.443+0000 7f7dd4200640 1 -- 172.21.15.40:0/593811744 shutdown_connections 2024-03-24T00:30:53.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.443+0000 7f7dd4200640 1 -- 172.21.15.40:0/593811744 wait complete. 2024-03-24T00:30:53.464 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-24T00:30:53.464 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:30:53.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.603+0000 7fe980c00640 1 Processor -- start 2024-03-24T00:30:53.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.603+0000 7fe980c00640 1 -- start start 2024-03-24T00:30:53.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.607+0000 7fe980c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe97c0b5be0 0x7fe97c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:53.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.607+0000 7fe980c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe97c072c10 con 0x7fe97c1477a0 2024-03-24T00:30:53.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.607+0000 7fe980c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe97c072ef0 con 0x7fe97c0725a0 2024-03-24T00:30:53.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.607+0000 7fe980c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe97c15ebc0 con 0x7fe97c0b5be0 2024-03-24T00:30:53.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.607+0000 7fe97a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe97c0b5be0 0x7fe97c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:53.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.607+0000 7fe979e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe97c1477a0 0x7fe97c06f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35070/0 (socket says 172.21.15.40:35070) 2024-03-24T00:30:53.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.607+0000 7fe979e00640 1 -- 172.21.15.40:0/2007251079 learned_addr learned my addr 172.21.15.40:0/2007251079 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:30:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.607+0000 7fe97a800640 1 -- 172.21.15.40:0/2007251079 >> v1:172.21.15.53:6789/0 conn(0x7fe97c0725a0 legacy=0x7fe97c0b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.607+0000 7fe97a800640 1 -- 172.21.15.40:0/2007251079 >> v1:172.21.15.40:6789/0 conn(0x7fe97c1477a0 legacy=0x7fe97c06f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.607+0000 7fe97a800640 1 -- 172.21.15.40:0/2007251079 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c15e850 con 0x7fe97c0b5be0 2024-03-24T00:30:53.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe97a800640 1 --2- 172.21.15.40:0/2007251079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe97c0b5be0 0x7fe97c15f050 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe9740097c0 tx=0x7fe97401c2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f186043f64ed82bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:53.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe979400640 1 -- 172.21.15.40:0/2007251079 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9740b0070 con 0x7fe97c0b5be0 2024-03-24T00:30:53.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe979400640 1 -- 172.21.15.40:0/2007251079 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9740ab600 con 0x7fe97c0b5be0 2024-03-24T00:30:53.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe979400640 1 -- 172.21.15.40:0/2007251079 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9740ab9d0 con 0x7fe97c0b5be0 2024-03-24T00:30:53.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe980c00640 1 -- 172.21.15.40:0/2007251079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe97c0b5be0 msgr2=0x7fe97c15f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe980c00640 1 --2- 172.21.15.40:0/2007251079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe97c0b5be0 0x7fe97c15f050 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe9740097c0 tx=0x7fe97401c2a0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe980c00640 1 -- 172.21.15.40:0/2007251079 shutdown_connections 2024-03-24T00:30:53.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe980c00640 1 --2- 172.21.15.40:0/2007251079 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe97c0b5be0 0x7fe97c15f050 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe980c00640 1 -- 172.21.15.40:0/2007251079 >> 172.21.15.40:0/2007251079 conn(0x7fe97c06b150 msgr2=0x7fe97c06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:53.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe980c00640 1 -- 172.21.15.40:0/2007251079 shutdown_connections 2024-03-24T00:30:53.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe980c00640 1 -- 172.21.15.40:0/2007251079 wait complete. 2024-03-24T00:30:53.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe980c00640 1 Processor -- start 2024-03-24T00:30:53.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.611+0000 7fe980c00640 1 -- start start 2024-03-24T00:30:53.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe980c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c0725a0 0x7fe97c15d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:53.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe980c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe97c0b5be0 0x7fe97c144b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:53.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe980c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe97c1477a0 0x7fe97c0cf060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:53.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe980c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c0d1490 con 0x7fe97c1477a0 2024-03-24T00:30:53.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe980c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c0d1310 con 0x7fe97c0725a0 2024-03-24T00:30:53.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe980c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c1446d0 con 0x7fe97c0b5be0 2024-03-24T00:30:53.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe97a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe97c1477a0 0x7fe97c0cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:53.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe971400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe97c0b5be0 0x7fe97c144b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:53.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe971400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe97c0b5be0 0x7fe97c144b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:40748/0 (socket says 172.21.15.40:40748) 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe97a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe97c1477a0 0x7fe97c0cf060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39988/0 (socket says 172.21.15.40:39988) 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe971400640 1 -- 172.21.15.40:0/1590192668 learned_addr learned my addr 172.21.15.40:0/1590192668 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe979e00640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c0725a0 0x7fe97c15d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe971400640 1 -- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c0725a0 msgr2=0x7fe97c15d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe971400640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c0725a0 0x7fe97c15d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe971400640 1 -- 172.21.15.40:0/1590192668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe97c1477a0 msgr2=0x7fe97c0cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe971400640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe97c1477a0 0x7fe97c0cf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe971400640 1 -- 172.21.15.40:0/1590192668 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c141930 con 0x7fe97c0b5be0 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe971400640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe97c0b5be0 0x7fe97c144b60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe96400c3a0 tx=0x7fe96400c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe972a00640 1 -- 172.21.15.40:0/1590192668 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9640052d0 con 0x7fe97c0b5be0 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe972a00640 1 -- 172.21.15.40:0/1590192668 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe964005470 con 0x7fe97c0b5be0 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe972a00640 1 -- 172.21.15.40:0/1590192668 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe964007260 con 0x7fe97c0b5be0 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe980c00640 1 -- 172.21.15.40:0/1590192668 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe97c143ac0 con 0x7fe97c0b5be0 2024-03-24T00:30:53.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe980c00640 1 -- 172.21.15.40:0/1590192668 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe97c144080 con 0x7fe97c0b5be0 2024-03-24T00:30:53.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe980c00640 1 -- 172.21.15.40:0/1590192668 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe97c068170 con 0x7fe97c0b5be0 2024-03-24T00:30:53.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.615+0000 7fe972a00640 1 -- 172.21.15.40:0/1590192668 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 4) v1 ==== 87027+0+0 (secure 0 0 0) 0x7fe964007400 con 0x7fe97c0b5be0 2024-03-24T00:30:53.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.619+0000 7fe972a00640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe9580675d0 0x7fe958069a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:53.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.619+0000 7fe972a00640 1 -- 172.21.15.40:0/1590192668 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fe964084750 con 0x7fe97c0b5be0 2024-03-24T00:30:53.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.619+0000 7fe979e00640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe9580675d0 0x7fe958069a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:53.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.619+0000 7fe979e00640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe9580675d0 0x7fe958069a90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0012f0 tx=0x7fe96c0018c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:53.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.623+0000 7fe972a00640 1 -- 172.21.15.40:0/1590192668 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe97c141930 con 0x7fe97c0b5be0 2024-03-24T00:30:53.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 -- 172.21.15.40:0/1590192668 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe97c144850 con 0x7fe97c0b5be0 2024-03-24T00:30:53.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe972a00640 1 -- 172.21.15.40:0/1590192668 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7fe964055290 con 0x7fe97c0b5be0 2024-03-24T00:30:53.820 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:53.820 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/501744669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":501744669},{"type":"v1","addr":"172.21.15.40:6833","nonce":501744669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-24T00:30:53.821 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 2 2024-03-24T00:30:53.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 -- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe9580675d0 msgr2=0x7fe958069a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe9580675d0 0x7fe958069a90 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe96c0012f0 tx=0x7fe96c0018c0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 -- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe97c0b5be0 msgr2=0x7fe97c144b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe97c0b5be0 0x7fe97c144b60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe96400c3a0 tx=0x7fe96400c860 comp rx=0 tx=0).stop 2024-03-24T00:30:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 -- 172.21.15.40:0/1590192668 shutdown_connections 2024-03-24T00:30:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe9580675d0 0x7fe958069a90 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe97c1477a0 0x7fe97c0cf060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe97c0b5be0 0x7fe97c144b60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 --2- 172.21.15.40:0/1590192668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe97c0725a0 0x7fe97c15d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 -- 172.21.15.40:0/1590192668 >> 172.21.15.40:0/1590192668 conn(0x7fe97c06b150 msgr2=0x7fe97c142780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 -- 172.21.15.40:0/1590192668 shutdown_connections 2024-03-24T00:30:53.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:53.819+0000 7fe980c00640 1 -- 172.21.15.40:0/1590192668 wait complete. 2024-03-24T00:30:55.843 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:30:55.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.983+0000 7f23fc400640 1 Processor -- start 2024-03-24T00:30:55.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.983+0000 7f23fc400640 1 -- start start 2024-03-24T00:30:55.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23fc400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23f40cfdf0 0x7f23f40d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:55.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23fc400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f23f40b43a0 0x7f23f40d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:55.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23fc400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f40af710 con 0x7f23f415f090 2024-03-24T00:30:55.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23fc400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f40b2bb0 con 0x7f23f40b43a0 2024-03-24T00:30:55.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23fc400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f40b26a0 con 0x7f23f40cfdf0 2024-03-24T00:30:55.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23fa400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f23f415f090 0x7f23f415f4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35080/0 (socket says 172.21.15.40:35080) 2024-03-24T00:30:55.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23fa400640 1 -- 172.21.15.40:0/2260947919 learned_addr learned my addr 172.21.15.40:0/2260947919 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:30:55.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23f9000640 1 --2- 172.21.15.40:0/2260947919 >> v2:172.21.15.53:3300/0 conn(0x7f23f40b43a0 0x7f23f40d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:55.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23f9a00640 1 --2- 172.21.15.40:0/2260947919 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23f40cfdf0 0x7f23f40d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:55.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23f3e00640 1 -- 172.21.15.40:0/2260947919 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532551155 0 0) 0x7f23f40af710 con 0x7f23f415f090 2024-03-24T00:30:55.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23f3e00640 1 -- 172.21.15.40:0/2260947919 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23dc003530 con 0x7f23f415f090 2024-03-24T00:30:55.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23f9000640 1 -- 172.21.15.40:0/2260947919 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23f40cfdf0 msgr2=0x7f23f40d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:55.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23f9000640 1 --2- 172.21.15.40:0/2260947919 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23f40cfdf0 0x7f23f40d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:55.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23f9000640 1 -- 172.21.15.40:0/2260947919 >> v1:172.21.15.40:6789/0 conn(0x7f23f415f090 legacy=0x7f23f415f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:55.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23f9000640 1 -- 172.21.15.40:0/2260947919 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f40b34f0 con 0x7f23f40b43a0 2024-03-24T00:30:55.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.987+0000 7f23f9a00640 1 --2- 172.21.15.40:0/2260947919 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23f40cfdf0 0x7f23f40d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:55.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23f9000640 1 --2- 172.21.15.40:0/2260947919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23f40b43a0 0x7f23f40d0710 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f23e00097c0 tx=0x7f23e001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=746d0ccef5156081 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:55.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23f3e00640 1 -- 172.21.15.40:0/2260947919 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e00b0070 con 0x7f23f40b43a0 2024-03-24T00:30:55.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23f3e00640 1 -- 172.21.15.40:0/2260947919 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e00ab600 con 0x7f23f40b43a0 2024-03-24T00:30:55.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23f3e00640 1 -- 172.21.15.40:0/2260947919 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e00ab9d0 con 0x7f23f40b43a0 2024-03-24T00:30:55.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 -- 172.21.15.40:0/2260947919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23f40b43a0 msgr2=0x7f23f40d0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:55.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 --2- 172.21.15.40:0/2260947919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23f40b43a0 0x7f23f40d0710 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f23e00097c0 tx=0x7f23e001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:30:55.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 -- 172.21.15.40:0/2260947919 shutdown_connections 2024-03-24T00:30:55.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 --2- 172.21.15.40:0/2260947919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23f40b43a0 0x7f23f40d0710 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:55.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 --2- 172.21.15.40:0/2260947919 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23f40cfdf0 0x7f23f40d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:55.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 -- 172.21.15.40:0/2260947919 >> 172.21.15.40:0/2260947919 conn(0x7f23f40b3880 msgr2=0x7f23f40b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:55.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 -- 172.21.15.40:0/2260947919 shutdown_connections 2024-03-24T00:30:55.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 -- 172.21.15.40:0/2260947919 wait complete. 2024-03-24T00:30:55.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 Processor -- start 2024-03-24T00:30:55.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 -- start start 2024-03-24T00:30:55.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23f40b43a0 0x7f23f40c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:55.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23f40cfdf0 0x7f23f40c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:55.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23f415f090 0x7f23f414d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:55.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f40b07d0 con 0x7f23f40cfdf0 2024-03-24T00:30:55.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f40b0650 con 0x7f23f415f090 2024-03-24T00:30:55.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.991+0000 7f23fc400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f40b0950 con 0x7f23f40b43a0 2024-03-24T00:30:55.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f9000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23f40cfdf0 0x7f23f40c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:55.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f9a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23f40b43a0 0x7f23f40c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:55.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f9000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23f40cfdf0 0x7f23f40c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39994/0 (socket says 172.21.15.40:39994) 2024-03-24T00:30:55.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f9000640 1 -- 172.21.15.40:0/3249030304 learned_addr learned my addr 172.21.15.40:0/3249030304 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:55.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23fa400640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23f415f090 0x7f23f414d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:55.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f9000640 1 -- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23f40b43a0 msgr2=0x7f23f40c6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:55.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f9000640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23f40b43a0 0x7f23f40c6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:55.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f9000640 1 -- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23f415f090 msgr2=0x7f23f414d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:55.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f9000640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23f415f090 0x7f23f414d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:55.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f9000640 1 -- 172.21.15.40:0/3249030304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f40c63c0 con 0x7f23f40cfdf0 2024-03-24T00:30:55.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23fa400640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23f415f090 0x7f23f414d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:30:55.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f9000640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23f40cfdf0 0x7f23f40c5e80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f23e00ae040 tx=0x7f23e0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:55.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f1600640 1 -- 172.21.15.40:0/3249030304 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e00b0040 con 0x7f23f40cfdf0 2024-03-24T00:30:55.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f1600640 1 -- 172.21.15.40:0/3249030304 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e00ba070 con 0x7f23f40cfdf0 2024-03-24T00:30:55.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23fc400640 1 -- 172.21.15.40:0/3249030304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f40c7440 con 0x7f23f40cfdf0 2024-03-24T00:30:55.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23f1600640 1 -- 172.21.15.40:0/3249030304 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e00b4850 con 0x7f23f40cfdf0 2024-03-24T00:30:55.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23fc400640 1 -- 172.21.15.40:0/3249030304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f40c7a00 con 0x7f23f40cfdf0 2024-03-24T00:30:55.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.995+0000 7f23fc400640 1 -- 172.21.15.40:0/3249030304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f4068170 con 0x7f23f40cfdf0 2024-03-24T00:30:55.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.999+0000 7f23f1600640 1 -- 172.21.15.40:0/3249030304 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f23e00ab480 con 0x7f23f40cfdf0 2024-03-24T00:30:56.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.999+0000 7f23f1600640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f23d40675d0 0x7f23d4069a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.999+0000 7f23f1600640 1 -- 172.21.15.40:0/3249030304 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f23e011fd50 con 0x7f23f40cfdf0 2024-03-24T00:30:56.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.999+0000 7f23f9a00640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f23d40675d0 0x7f23d4069a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:55.999+0000 7f23f9a00640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f23d40675d0 0x7f23d4069a90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009580 tx=0x7f23e8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:56.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.003+0000 7f23f1600640 1 -- 172.21.15.40:0/3249030304 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f23e0016090 con 0x7f23f40cfdf0 2024-03-24T00:30:56.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 -- 172.21.15.40:0/3249030304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23f4067c70 con 0x7f23f40cfdf0 2024-03-24T00:30:56.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23f1600640 1 -- 172.21.15.40:0/3249030304 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3688 (secure 0 0 0) 0x7f23e00f0d50 con 0x7f23f40cfdf0 2024-03-24T00:30:56.208 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:56.208 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/501744669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":501744669},{"type":"v1","addr":"172.21.15.40:6833","nonce":501744669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/281354690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":281354690},{"type":"v1","addr":"172.21.15.53:6836","nonce":281354690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3092758821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3092758821},{"type":"v1","addr":"172.21.15.53:6837","nonce":3092758821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4264,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/4139303536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":4139303536},{"type":"v1","addr":"172.21.15.40:6835","nonce":4139303536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-24T00:30:56.208 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 3 2024-03-24T00:30:56.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 -- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f23d40675d0 msgr2=0x7f23d4069a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f23d40675d0 0x7f23d4069a90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009580 tx=0x7f23e8009190 comp rx=0 tx=0).stop 2024-03-24T00:30:56.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 -- 172.21.15.40:0/3249030304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23f40cfdf0 msgr2=0x7f23f40c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23f40cfdf0 0x7f23f40c5e80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f23e00ae040 tx=0x7f23e0019240 comp rx=0 tx=0).stop 2024-03-24T00:30:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 -- 172.21.15.40:0/3249030304 shutdown_connections 2024-03-24T00:30:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f23d40675d0 0x7f23d4069a90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23f415f090 0x7f23f414d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f23f40cfdf0 0x7f23f40c5e80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 --2- 172.21.15.40:0/3249030304 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23f40b43a0 0x7f23f40c6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 -- 172.21.15.40:0/3249030304 >> 172.21.15.40:0/3249030304 conn(0x7f23f40b3880 msgr2=0x7f23f40b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 -- 172.21.15.40:0/3249030304 shutdown_connections 2024-03-24T00:30:56.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.207+0000 7f23fc400640 1 -- 172.21.15.40:0/3249030304 wait complete. 2024-03-24T00:30:56.230 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:30:56.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.367+0000 7fdfcb800640 1 Processor -- start 2024-03-24T00:30:56.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.367+0000 7fdfcb800640 1 -- start start 2024-03-24T00:30:56.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfcb800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfc406e620 0x7fdfc40b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfcb800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdfc406ea00 0x7fdfc40b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfcb800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc4070f60 con 0x7fdfc40b2ff0 2024-03-24T00:30:56.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfcb800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc40b4c00 con 0x7fdfc406ea00 2024-03-24T00:30:56.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfcb800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc40b3620 con 0x7fdfc406e620 2024-03-24T00:30:56.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfc8e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfc406e620 0x7fdfc40b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfbbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdfc406ea00 0x7fdfc40b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfc9800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fdfc40b2ff0 0x7fdfc40ccfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:43984/0 (socket says 172.21.15.40:43984) 2024-03-24T00:30:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfc9800640 1 -- 172.21.15.40:0/3377413570 learned_addr learned my addr 172.21.15.40:0/3377413570 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:30:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfbbe00640 1 -- 172.21.15.40:0/3377413570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfc406e620 msgr2=0x7fdfc40b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfbbe00640 1 --2- 172.21.15.40:0/3377413570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfc406e620 0x7fdfc40b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfbbe00640 1 -- 172.21.15.40:0/3377413570 >> v1:172.21.15.53:6789/0 conn(0x7fdfc40b2ff0 legacy=0x7fdfc40ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfbbe00640 1 -- 172.21.15.40:0/3377413570 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc40b3900 con 0x7fdfc406ea00 2024-03-24T00:30:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfc8e00640 1 --2- 172.21.15.40:0/3377413570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfc406e620 0x7fdfc40b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:56.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.371+0000 7fdfc3e00640 1 -- 172.21.15.40:0/3377413570 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296488200 0 0) 0x7fdfc4070f60 con 0x7fdfc40b2ff0 2024-03-24T00:30:56.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfbbe00640 1 --2- 172.21.15.40:0/3377413570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfc406ea00 0x7fdfc40b2ab0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0097c0 tx=0x7fdfac01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88e6e8fa3bf99e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:56.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfc3e00640 1 -- 172.21.15.40:0/3377413570 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac0b0070 con 0x7fdfc406ea00 2024-03-24T00:30:56.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfc3e00640 1 -- 172.21.15.40:0/3377413570 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac0ab600 con 0x7fdfc406ea00 2024-03-24T00:30:56.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfc3e00640 1 -- 172.21.15.40:0/3377413570 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac0ab9d0 con 0x7fdfc406ea00 2024-03-24T00:30:56.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfcb800640 1 -- 172.21.15.40:0/3377413570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfc406ea00 msgr2=0x7fdfc40b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfcb800640 1 --2- 172.21.15.40:0/3377413570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfc406ea00 0x7fdfc40b2ab0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0097c0 tx=0x7fdfac01c410 comp rx=0 tx=0).stop 2024-03-24T00:30:56.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfcb800640 1 -- 172.21.15.40:0/3377413570 shutdown_connections 2024-03-24T00:30:56.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfcb800640 1 --2- 172.21.15.40:0/3377413570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfc406ea00 0x7fdfc40b2ab0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfcb800640 1 --2- 172.21.15.40:0/3377413570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfc406e620 0x7fdfc40b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfcb800640 1 -- 172.21.15.40:0/3377413570 >> 172.21.15.40:0/3377413570 conn(0x7fdfc40b17b0 msgr2=0x7fdfc40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfcb800640 1 -- 172.21.15.40:0/3377413570 shutdown_connections 2024-03-24T00:30:56.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfcb800640 1 -- 172.21.15.40:0/3377413570 wait complete. 2024-03-24T00:30:56.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfcb800640 1 Processor -- start 2024-03-24T00:30:56.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.375+0000 7fdfcb800640 1 -- start start 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfcb800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfc406e620 0x7fdfc4141100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfcb800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfc406ea00 0x7fdfc413c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfcb800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfc40b2ff0 0x7fdfc4140450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfcb800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc4144920 con 0x7fdfc406ea00 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfcb800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc406ee60 con 0x7fdfc406e620 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfcb800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc413c6a0 con 0x7fdfc40b2ff0 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfc8e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfc406e620 0x7fdfc4141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfbbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfc406ea00 0x7fdfc413c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfc8e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfc406e620 0x7fdfc4141100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:38928/0 (socket says 172.21.15.40:38928) 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfc8e00640 1 -- 172.21.15.40:0/1408840021 learned_addr learned my addr 172.21.15.40:0/1408840021 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfc9800640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfc40b2ff0 0x7fdfc4140450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfbbe00640 1 -- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfc40b2ff0 msgr2=0x7fdfc4140450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfbbe00640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfc40b2ff0 0x7fdfc4140450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfbbe00640 1 -- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfc406e620 msgr2=0x7fdfc4141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfbbe00640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfc406e620 0x7fdfc4141100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfbbe00640 1 -- 172.21.15.40:0/1408840021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc413c940 con 0x7fdfc406ea00 2024-03-24T00:30:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfc8e00640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfc406e620 0x7fdfc4141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:56.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfbbe00640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfc406ea00 0x7fdfc413c160 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0ab000 tx=0x7fdfac0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfc1600640 1 -- 172.21.15.40:0/1408840021 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac0b8020 con 0x7fdfc406ea00 2024-03-24T00:30:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfc1600640 1 -- 172.21.15.40:0/1408840021 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfac0b4410 con 0x7fdfc406ea00 2024-03-24T00:30:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfc1600640 1 -- 172.21.15.40:0/1408840021 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfac0b0040 con 0x7fdfc406ea00 2024-03-24T00:30:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfcb800640 1 -- 172.21.15.40:0/1408840021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc413cc20 con 0x7fdfc406ea00 2024-03-24T00:30:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfcb800640 1 -- 172.21.15.40:0/1408840021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc413d1e0 con 0x7fdfc406ea00 2024-03-24T00:30:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfcb800640 1 -- 172.21.15.40:0/1408840021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc4068170 con 0x7fdfc406ea00 2024-03-24T00:30:56.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.379+0000 7fdfc1600640 1 -- 172.21.15.40:0/1408840021 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 87027+0+0 (secure 0 0 0) 0x7fdfac0bb020 con 0x7fdfc406ea00 2024-03-24T00:30:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.383+0000 7fdfc1600640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdfa0067600 0x7fdfa0069ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.383+0000 7fdfc1600640 1 -- 172.21.15.40:0/1408840021 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdfac11ee70 con 0x7fdfc406ea00 2024-03-24T00:30:56.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.383+0000 7fdfc8e00640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdfa0067600 0x7fdfa0069ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.383+0000 7fdfc8e00640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdfa0067600 0x7fdfa0069ac0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009580 tx=0x7fdfb4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:56.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.387+0000 7fdfc1600640 1 -- 172.21.15.40:0/1408840021 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdfac0c2030 con 0x7fdfc406ea00 2024-03-24T00:30:56.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.551+0000 7fdfcb800640 1 -- 172.21.15.40:0/1408840021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfc4072f50 con 0x7fdfc406ea00 2024-03-24T00:30:56.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfc1600640 1 -- 172.21.15.40:0/1408840021 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12824 (secure 0 0 0) 0x7fdfac016090 con 0x7fdfc406ea00 2024-03-24T00:30:56.636 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:56.636 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":16,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:30:55.997510+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:30:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfcb800640 1 -- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdfa0067600 msgr2=0x7fdfa0069ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfcb800640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdfa0067600 0x7fdfa0069ac0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009580 tx=0x7fdfb4009190 comp rx=0 tx=0).stop 2024-03-24T00:30:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfcb800640 1 -- 172.21.15.40:0/1408840021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfc406ea00 msgr2=0x7fdfc413c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfcb800640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfc406ea00 0x7fdfc413c160 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0ab000 tx=0x7fdfac0191b0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfcb800640 1 -- 172.21.15.40:0/1408840021 shutdown_connections 2024-03-24T00:30:56.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfcb800640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdfa0067600 0x7fdfa0069ac0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfcb800640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfc40b2ff0 0x7fdfc4140450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfcb800640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfc406ea00 0x7fdfc413c160 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfcb800640 1 --2- 172.21.15.40:0/1408840021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfc406e620 0x7fdfc4141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfcb800640 1 -- 172.21.15.40:0/1408840021 >> 172.21.15.40:0/1408840021 conn(0x7fdfc40b17b0 msgr2=0x7fdfc4068fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfcb800640 1 -- 172.21.15.40:0/1408840021 shutdown_connections 2024-03-24T00:30:56.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.635+0000 7fdfcb800640 1 -- 172.21.15.40:0/1408840021 wait complete. 2024-03-24T00:30:56.657 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:30:56.657 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:30:56.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.799+0000 7f89f4400640 1 Processor -- start 2024-03-24T00:30:56.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.799+0000 7f89f4400640 1 -- start start 2024-03-24T00:30:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.799+0000 7f89f4400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f89ec15bd00 0x7f89ec0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.799+0000 7f89f4400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f89ec0728d0 0x7f89ec0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.799+0000 7f89f4400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ec072d60 con 0x7f89ec15f8c0 2024-03-24T00:30:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.799+0000 7f89f4400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ec0cf790 con 0x7f89ec0728d0 2024-03-24T00:30:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.799+0000 7f89f4400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ec071de0 con 0x7f89ec15bd00 2024-03-24T00:30:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.799+0000 7f89f1000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f89ec0728d0 0x7f89ec0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.799+0000 7f89f2400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f89ec15f8c0 0x7f89ec15fcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35088/0 (socket says 172.21.15.40:35088) 2024-03-24T00:30:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.799+0000 7f89f2400640 1 -- 172.21.15.40:0/42208124 learned_addr learned my addr 172.21.15.40:0/42208124 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:30:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.799+0000 7f89f1a00640 1 --2- 172.21.15.40:0/42208124 >> v2:172.21.15.53:3300/0 conn(0x7f89ec15bd00 0x7f89ec0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.803+0000 7f89ebe00640 1 -- 172.21.15.40:0/42208124 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708629437 0 0) 0x7f89ec072d60 con 0x7f89ec15f8c0 2024-03-24T00:30:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.803+0000 7f89ebe00640 1 -- 172.21.15.40:0/42208124 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89d4003530 con 0x7f89ec15f8c0 2024-03-24T00:30:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.803+0000 7f89f1a00640 1 -- 172.21.15.40:0/42208124 >> v2:172.21.15.40:3300/0 conn(0x7f89ec0728d0 msgr2=0x7f89ec0cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.803+0000 7f89f1a00640 1 --2- 172.21.15.40:0/42208124 >> v2:172.21.15.40:3300/0 conn(0x7f89ec0728d0 0x7f89ec0cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.803+0000 7f89f1a00640 1 -- 172.21.15.40:0/42208124 >> v1:172.21.15.40:6789/0 conn(0x7f89ec15f8c0 legacy=0x7f89ec15fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.803+0000 7f89f1a00640 1 -- 172.21.15.40:0/42208124 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec0cea50 con 0x7f89ec15bd00 2024-03-24T00:30:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.803+0000 7f89f1a00640 1 --2- 172.21.15.40:0/42208124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ec15bd00 0x7f89ec0ae8e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0733c0 tx=0x7f89e001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6d377908f5249fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.803+0000 7f89ebe00640 1 -- 172.21.15.40:0/42208124 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e00ab650 con 0x7f89ec15bd00 2024-03-24T00:30:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.803+0000 7f89ebe00640 1 -- 172.21.15.40:0/42208124 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e00b0070 con 0x7f89ec15bd00 2024-03-24T00:30:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.803+0000 7f89ebe00640 1 -- 172.21.15.40:0/42208124 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e00aba10 con 0x7f89ec15bd00 2024-03-24T00:30:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.803+0000 7f89f4400640 1 -- 172.21.15.40:0/42208124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ec15bd00 msgr2=0x7f89ec0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.803+0000 7f89f4400640 1 --2- 172.21.15.40:0/42208124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ec15bd00 0x7f89ec0ae8e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f89ec0733c0 tx=0x7f89e001c450 comp rx=0 tx=0).stop 2024-03-24T00:30:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 -- 172.21.15.40:0/42208124 shutdown_connections 2024-03-24T00:30:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 --2- 172.21.15.40:0/42208124 >> v2:172.21.15.40:3300/0 conn(0x7f89ec0728d0 0x7f89ec0cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 --2- 172.21.15.40:0/42208124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ec15bd00 0x7f89ec0ae8e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 -- 172.21.15.40:0/42208124 >> 172.21.15.40:0/42208124 conn(0x7f89ec0b1660 msgr2=0x7f89ec0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 -- 172.21.15.40:0/42208124 shutdown_connections 2024-03-24T00:30:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 -- 172.21.15.40:0/42208124 wait complete. 2024-03-24T00:30:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 Processor -- start 2024-03-24T00:30:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 -- start start 2024-03-24T00:30:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ec0728d0 0x7f89ec14e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f89ec15bd00 0x7f89ec0a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ec15f8c0 0x7f89ec0aa360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec06d320 con 0x7f89ec15f8c0 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec15c0e0 con 0x7f89ec0728d0 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f4400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ec06d4a0 con 0x7f89ec15bd00 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ec15f8c0 0x7f89ec0aa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ec15f8c0 0x7f89ec0aa360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40028/0 (socket says 172.21.15.40:40028) 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f2400640 1 -- 172.21.15.40:0/3056356403 learned_addr learned my addr 172.21.15.40:0/3056356403 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f1a00640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ec0728d0 0x7f89ec14e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f1000640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f89ec15bd00 0x7f89ec0a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f2400640 1 -- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f89ec15bd00 msgr2=0x7f89ec0a9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f2400640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f89ec15bd00 0x7f89ec0a9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f2400640 1 -- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ec0728d0 msgr2=0x7f89ec14e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f2400640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ec0728d0 0x7f89ec14e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f2400640 1 -- 172.21.15.40:0/3056356403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ec0aa8a0 con 0x7f89ec15f8c0 2024-03-24T00:30:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89f2400640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ec15f8c0 0x7f89ec0aa360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f89e400caa0 tx=0x7f89e400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.807+0000 7f89e9600640 1 -- 172.21.15.40:0/3056356403 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4017070 con 0x7f89ec15f8c0 2024-03-24T00:30:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.811+0000 7f89e9600640 1 -- 172.21.15.40:0/3056356403 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89e400ac00 con 0x7f89ec15f8c0 2024-03-24T00:30:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.811+0000 7f89f4400640 1 -- 172.21.15.40:0/3056356403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ec14d4d0 con 0x7f89ec15f8c0 2024-03-24T00:30:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.811+0000 7f89e9600640 1 -- 172.21.15.40:0/3056356403 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89e4012400 con 0x7f89ec15f8c0 2024-03-24T00:30:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.811+0000 7f89f4400640 1 -- 172.21.15.40:0/3056356403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ec14dd70 con 0x7f89ec15f8c0 2024-03-24T00:30:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.811+0000 7f89f4400640 1 -- 172.21.15.40:0/3056356403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ec14d8c0 con 0x7f89ec15f8c0 2024-03-24T00:30:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.811+0000 7f89e9600640 1 -- 172.21.15.40:0/3056356403 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f89e401a070 con 0x7f89ec15f8c0 2024-03-24T00:30:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.811+0000 7f89e9600640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f89cc0675d0 0x7f89cc069a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.811+0000 7f89e9600640 1 -- 172.21.15.40:0/3056356403 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f89e4085610 con 0x7f89ec15f8c0 2024-03-24T00:30:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.811+0000 7f89f1a00640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f89cc0675d0 0x7f89cc069a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.815+0000 7f89f1a00640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f89cc0675d0 0x7f89cc069a90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f89e00af000 tx=0x7f89e00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.815+0000 7f89e9600640 1 -- 172.21.15.40:0/3056356403 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f89e4021020 con 0x7f89ec15f8c0 2024-03-24T00:30:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 -- 172.21.15.40:0/3056356403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f89ec068170 con 0x7f89ec15f8c0 2024-03-24T00:30:56.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89e9600640 1 -- 172.21.15.40:0/3056356403 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f89ec0aa8a0 con 0x7f89ec15f8c0 2024-03-24T00:30:56.984 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:30:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 -- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f89cc0675d0 msgr2=0x7f89cc069a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f89cc0675d0 0x7f89cc069a90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f89e00af000 tx=0x7f89e00a7000 comp rx=0 tx=0).stop 2024-03-24T00:30:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 -- 172.21.15.40:0/3056356403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ec15f8c0 msgr2=0x7f89ec0aa360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ec15f8c0 0x7f89ec0aa360 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f89e400caa0 tx=0x7f89e400cf60 comp rx=0 tx=0).stop 2024-03-24T00:30:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 -- 172.21.15.40:0/3056356403 shutdown_connections 2024-03-24T00:30:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f89cc0675d0 0x7f89cc069a90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ec15f8c0 0x7f89ec0aa360 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f89ec15bd00 0x7f89ec0a9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 --2- 172.21.15.40:0/3056356403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ec0728d0 0x7f89ec14e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 -- 172.21.15.40:0/3056356403 >> 172.21.15.40:0/3056356403 conn(0x7f89ec0b1660 msgr2=0x7f89ec0b04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 -- 172.21.15.40:0/3056356403 shutdown_connections 2024-03-24T00:30:56.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:56.983+0000 7f89f4400640 1 -- 172.21.15.40:0/3056356403 wait complete. 2024-03-24T00:30:57.006 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:30:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.143+0000 7f7d28200640 1 Processor -- start 2024-03-24T00:30:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.143+0000 7f7d28200640 1 -- start start 2024-03-24T00:30:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d28200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7d2013ec80 0x7f7d201414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d28200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7d201419f0 0x7f7d200ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d28200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d20070e00 con 0x7f7d200ce630 2024-03-24T00:30:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d28200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d20141e50 con 0x7f7d201419f0 2024-03-24T00:30:57.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d28200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d200cee20 con 0x7f7d2013ec80 2024-03-24T00:30:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d24e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7d201419f0 0x7f7d200ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d26200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7d200ce630 0x7f7d2015f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55042/0 (socket says 172.21.15.40:55042) 2024-03-24T00:30:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d24e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7d201419f0 0x7f7d200ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56938/0 (socket says 172.21.15.40:56938) 2024-03-24T00:30:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d26200640 1 -- 172.21.15.40:0/2047538787 learned_addr learned my addr 172.21.15.40:0/2047538787 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:30:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d24e00640 1 -- 172.21.15.40:0/2047538787 >> v2:172.21.15.53:3300/0 conn(0x7f7d2013ec80 msgr2=0x7f7d201414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d25800640 1 --2- 172.21.15.40:0/2047538787 >> v2:172.21.15.53:3300/0 conn(0x7f7d2013ec80 0x7f7d201414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d24e00640 1 --2- 172.21.15.40:0/2047538787 >> v2:172.21.15.53:3300/0 conn(0x7f7d2013ec80 0x7f7d201414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d24e00640 1 -- 172.21.15.40:0/2047538787 >> v1:172.21.15.40:6789/0 conn(0x7f7d200ce630 legacy=0x7f7d2015f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d24e00640 1 -- 172.21.15.40:0/2047538787 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d200cf100 con 0x7f7d201419f0 2024-03-24T00:30:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d25800640 1 --2- 172.21.15.40:0/2047538787 >> v2:172.21.15.53:3300/0 conn(0x7f7d2013ec80 0x7f7d201414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:30:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d1fe00640 1 -- 172.21.15.40:0/2047538787 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931633873 0 0) 0x7f7d20070e00 con 0x7f7d200ce630 2024-03-24T00:30:57.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.147+0000 7f7d24e00640 1 --2- 172.21.15.40:0/2047538787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d201419f0 0x7f7d200ce0f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7d2006b170 tx=0x7f7d0c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47cfebbe27f40cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:57.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d1fe00640 1 -- 172.21.15.40:0/2047538787 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c0aa600 con 0x7f7d201419f0 2024-03-24T00:30:57.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d1fe00640 1 -- 172.21.15.40:0/2047538787 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0c0af070 con 0x7f7d201419f0 2024-03-24T00:30:57.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d1fe00640 1 -- 172.21.15.40:0/2047538787 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0c0aa9c0 con 0x7f7d201419f0 2024-03-24T00:30:57.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 -- 172.21.15.40:0/2047538787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d201419f0 msgr2=0x7f7d200ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:57.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 --2- 172.21.15.40:0/2047538787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d201419f0 0x7f7d200ce0f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7d2006b170 tx=0x7f7d0c01c480 comp rx=0 tx=0).stop 2024-03-24T00:30:57.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 -- 172.21.15.40:0/2047538787 shutdown_connections 2024-03-24T00:30:57.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 --2- 172.21.15.40:0/2047538787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d201419f0 0x7f7d200ce0f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 --2- 172.21.15.40:0/2047538787 >> v2:172.21.15.53:3300/0 conn(0x7f7d2013ec80 0x7f7d201414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 -- 172.21.15.40:0/2047538787 >> 172.21.15.40:0/2047538787 conn(0x7f7d200b17b0 msgr2=0x7f7d200b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:57.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 -- 172.21.15.40:0/2047538787 shutdown_connections 2024-03-24T00:30:57.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 -- 172.21.15.40:0/2047538787 wait complete. 2024-03-24T00:30:57.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 Processor -- start 2024-03-24T00:30:57.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 -- start start 2024-03-24T00:30:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d200ce630 0x7f7d2015baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d2013ec80 0x7f7d201561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d201419f0 0x7f7d20159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d20168ec0 con 0x7f7d201419f0 2024-03-24T00:30:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d20168d40 con 0x7f7d2013ec80 2024-03-24T00:30:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d28200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d20169040 con 0x7f7d200ce630 2024-03-24T00:30:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d26200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d201419f0 0x7f7d20159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d26200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d201419f0 0x7f7d20159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56946/0 (socket says 172.21.15.40:56946) 2024-03-24T00:30:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d26200640 1 -- 172.21.15.40:0/3416961199 learned_addr learned my addr 172.21.15.40:0/3416961199 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d25800640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d200ce630 0x7f7d2015baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.151+0000 7f7d24e00640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d2013ec80 0x7f7d201561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d26200640 1 -- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d200ce630 msgr2=0x7f7d2015baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d26200640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d200ce630 0x7f7d2015baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d26200640 1 -- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d2013ec80 msgr2=0x7f7d201561d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:57.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d26200640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d2013ec80 0x7f7d201561d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d26200640 1 -- 172.21.15.40:0/3416961199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2015aa50 con 0x7f7d201419f0 2024-03-24T00:30:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d24e00640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d2013ec80 0x7f7d201561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:30:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d26200640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d201419f0 0x7f7d20159fd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800c7a0 tx=0x7f7d1800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d1d600640 1 -- 172.21.15.40:0/3416961199 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d18018070 con 0x7f7d201419f0 2024-03-24T00:30:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d1d600640 1 -- 172.21.15.40:0/3416961199 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1800a640 con 0x7f7d201419f0 2024-03-24T00:30:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d28200640 1 -- 172.21.15.40:0/3416961199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2015ad30 con 0x7f7d201419f0 2024-03-24T00:30:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d1d600640 1 -- 172.21.15.40:0/3416961199 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1800aab0 con 0x7f7d201419f0 2024-03-24T00:30:57.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d28200640 1 -- 172.21.15.40:0/3416961199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2015b2f0 con 0x7f7d201419f0 2024-03-24T00:30:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d28200640 1 -- 172.21.15.40:0/3416961199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d20068170 con 0x7f7d201419f0 2024-03-24T00:30:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d1d600640 1 -- 172.21.15.40:0/3416961199 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f7d1801b070 con 0x7f7d201419f0 2024-03-24T00:30:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d1d600640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7d000675d0 0x7f7d00069a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d1d600640 1 -- 172.21.15.40:0/3416961199 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7d180849e0 con 0x7f7d201419f0 2024-03-24T00:30:57.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.155+0000 7f7d25800640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7d000675d0 0x7f7d00069a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.159+0000 7f7d25800640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7d000675d0 0x7f7d00069a90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d20165070 tx=0x7f7d14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.163+0000 7f7d1d600640 1 -- 172.21.15.40:0/3416961199 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7d18055430 con 0x7f7d201419f0 2024-03-24T00:30:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 -- 172.21.15.40:0/3416961199 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7d20072b90 con 0x7f7d201419f0 2024-03-24T00:30:57.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d1d600640 1 -- 172.21.15.40:0/3416961199 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7d180594d0 con 0x7f7d201419f0 2024-03-24T00:30:57.331 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:30:57.331 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-03-24T00:30:57.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:30:57.332 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-03-24T00:30:57.332 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-03-24T00:30:57.332 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:30:57.332 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-03-24T00:30:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 -- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7d000675d0 msgr2=0x7f7d00069a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7d000675d0 0x7f7d00069a90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d20165070 tx=0x7f7d14008040 comp rx=0 tx=0).stop 2024-03-24T00:30:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 -- 172.21.15.40:0/3416961199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d201419f0 msgr2=0x7f7d20159fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d201419f0 0x7f7d20159fd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7d1800c7a0 tx=0x7f7d1800cc60 comp rx=0 tx=0).stop 2024-03-24T00:30:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 -- 172.21.15.40:0/3416961199 shutdown_connections 2024-03-24T00:30:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7d000675d0 0x7f7d00069a90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7d201419f0 0x7f7d20159fd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d2013ec80 0x7f7d201561d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 --2- 172.21.15.40:0/3416961199 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d200ce630 0x7f7d2015baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 -- 172.21.15.40:0/3416961199 >> 172.21.15.40:0/3416961199 conn(0x7f7d200b17b0 msgr2=0x7f7d20071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:57.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 -- 172.21.15.40:0/3416961199 shutdown_connections 2024-03-24T00:30:57.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.331+0000 7f7d28200640 1 -- 172.21.15.40:0/3416961199 wait complete. 2024-03-24T00:30:57.352 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:30:57.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:30:57.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1663400640 1 Processor -- start 2024-03-24T00:30:57.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1663400640 1 -- start start 2024-03-24T00:30:57.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1663400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f165c15f5c0 0x7f165c0b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:57.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1663400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f165c0cf1e0 0x7f165c0d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:57.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1663400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165c0b0f10 con 0x7f165c0cfbb0 2024-03-24T00:30:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1663400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f165c15f9a0 con 0x7f165c15f5c0 2024-03-24T00:30:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1663400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165c0cf630 con 0x7f165c0cf1e0 2024-03-24T00:30:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1660a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f165c15f5c0 0x7f165c0b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1653e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f165c0cfbb0 0x7f165c0b1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55050/0 (socket says 172.21.15.40:55050) 2024-03-24T00:30:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1653e00640 1 -- 172.21.15.40:0/1600239477 learned_addr learned my addr 172.21.15.40:0/1600239477 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:30:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1661400640 1 --2- 172.21.15.40:0/1600239477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f165c0cf1e0 0x7f165c0d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1660a00640 1 -- 172.21.15.40:0/1600239477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f165c0cf1e0 msgr2=0x7f165c0d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1660a00640 1 --2- 172.21.15.40:0/1600239477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f165c0cf1e0 0x7f165c0d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1660a00640 1 -- 172.21.15.40:0/1600239477 >> v1:172.21.15.40:6789/0 conn(0x7f165c0cfbb0 legacy=0x7f165c0b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1660a00640 1 -- 172.21.15.40:0/1600239477 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165c0cd130 con 0x7f165c15f5c0 2024-03-24T00:30:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f1661400640 1 --2- 172.21.15.40:0/1600239477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f165c0cf1e0 0x7f165c0d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:30:57.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.495+0000 7f165be00640 1 -- 172.21.15.40:0/1600239477 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81047555 0 0) 0x7f1644001120 con 0x7f165c0cfbb0 2024-03-24T00:30:57.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.499+0000 7f1660a00640 1 --2- 172.21.15.40:0/1600239477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f165c15f5c0 0x7f165c0b3060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f164c0097c0 tx=0x7f164c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8450899177b0f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:57.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.499+0000 7f165be00640 1 -- 172.21.15.40:0/1600239477 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c0b2070 con 0x7f165c15f5c0 2024-03-24T00:30:57.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.499+0000 7f165be00640 1 -- 172.21.15.40:0/1600239477 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164c0ad640 con 0x7f165c15f5c0 2024-03-24T00:30:57.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.499+0000 7f165be00640 1 -- 172.21.15.40:0/1600239477 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c0ada10 con 0x7f165c15f5c0 2024-03-24T00:30:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.499+0000 7f1663400640 1 -- 172.21.15.40:0/1600239477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f165c15f5c0 msgr2=0x7f165c0b3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.499+0000 7f1663400640 1 --2- 172.21.15.40:0/1600239477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f165c15f5c0 0x7f165c0b3060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f164c0097c0 tx=0x7f164c01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.499+0000 7f1663400640 1 -- 172.21.15.40:0/1600239477 shutdown_connections 2024-03-24T00:30:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.499+0000 7f1663400640 1 --2- 172.21.15.40:0/1600239477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f165c0cf1e0 0x7f165c0d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.499+0000 7f1663400640 1 --2- 172.21.15.40:0/1600239477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f165c15f5c0 0x7f165c0b3060 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.499+0000 7f1663400640 1 -- 172.21.15.40:0/1600239477 >> 172.21.15.40:0/1600239477 conn(0x7f165c0b3810 msgr2=0x7f165c0b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.499+0000 7f1663400640 1 -- 172.21.15.40:0/1600239477 shutdown_connections 2024-03-24T00:30:57.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.499+0000 7f1663400640 1 -- 172.21.15.40:0/1600239477 wait complete. 2024-03-24T00:30:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1663400640 1 Processor -- start 2024-03-24T00:30:57.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1663400640 1 -- start start 2024-03-24T00:30:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1663400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f165c0cf1e0 0x7f165c156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1663400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f165c0cfbb0 0x7f165c1578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1663400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f165c15f5c0 0x7f165c157e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1663400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c0b2030 con 0x7f165c0cf1e0 2024-03-24T00:30:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1663400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c0cdcb0 con 0x7f165c0cfbb0 2024-03-24T00:30:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1663400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165c0b21b0 con 0x7f165c15f5c0 2024-03-24T00:30:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1660a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f165c0cf1e0 0x7f165c156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1660a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f165c0cf1e0 0x7f165c156410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56954/0 (socket says 172.21.15.40:56954) 2024-03-24T00:30:57.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1660a00640 1 -- 172.21.15.40:0/2735285638 learned_addr learned my addr 172.21.15.40:0/2735285638 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1660a00640 1 -- 172.21.15.40:0/2735285638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f165c15f5c0 msgr2=0x7f165c157e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1660a00640 1 --2- 172.21.15.40:0/2735285638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f165c15f5c0 0x7f165c157e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1660a00640 1 -- 172.21.15.40:0/2735285638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f165c0cfbb0 msgr2=0x7f165c1578e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1660a00640 1 --2- 172.21.15.40:0/2735285638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f165c0cfbb0 0x7f165c1578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1660a00640 1 -- 172.21.15.40:0/2735285638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165c0b4af0 con 0x7f165c0cf1e0 2024-03-24T00:30:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1660a00640 1 --2- 172.21.15.40:0/2735285638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f165c0cf1e0 0x7f165c156410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f164c0b0040 tx=0x7f164c019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1659600640 1 -- 172.21.15.40:0/2735285638 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c0b2040 con 0x7f165c0cf1e0 2024-03-24T00:30:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1659600640 1 -- 172.21.15.40:0/2735285638 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164c0bc070 con 0x7f165c0cf1e0 2024-03-24T00:30:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1659600640 1 -- 172.21.15.40:0/2735285638 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c0b6670 con 0x7f165c0cf1e0 2024-03-24T00:30:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1663400640 1 -- 172.21.15.40:0/2735285638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165c0b4dd0 con 0x7f165c0cf1e0 2024-03-24T00:30:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1663400640 1 -- 172.21.15.40:0/2735285638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165c0b5390 con 0x7f165c0cf1e0 2024-03-24T00:30:57.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.503+0000 7f1663400640 1 -- 172.21.15.40:0/2735285638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f165c068170 con 0x7f165c0cf1e0 2024-03-24T00:30:57.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.507+0000 7f1659600640 1 -- 172.21.15.40:0/2735285638 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 4) v1 ==== 87027+0+0 (secure 0 0 0) 0x7f164c0b6810 con 0x7f165c0cf1e0 2024-03-24T00:30:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.507+0000 7f1659600640 1 --2- 172.21.15.40:0/2735285638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1638067600 0x7f1638069ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:57.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.507+0000 7f1659600640 1 -- 172.21.15.40:0/2735285638 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f164c11f490 con 0x7f165c0cf1e0 2024-03-24T00:30:57.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.507+0000 7f1653e00640 1 --2- 172.21.15.40:0/2735285638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1638067600 0x7f1638069ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:57.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.507+0000 7f1653e00640 1 --2- 172.21.15.40:0/2735285638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1638067600 0x7f1638069ac0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f165c0cc360 tx=0x7f1644001320 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:57.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.511+0000 7f1659600640 1 -- 172.21.15.40:0/2735285638 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f164c016090 con 0x7f165c0cf1e0 2024-03-24T00:30:57.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:57.675+0000 7f1663400640 1 -- 172.21.15.40:0/2735285638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f165c06f290 con 0x7f165c0cf1e0 2024-03-24T00:30:58.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.011+0000 7f1659600640 1 -- 172.21.15.40:0/2735285638 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f164c0f4e20 con 0x7f165c0cf1e0 2024-03-24T00:30:58.013 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-03-24T00:30:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.011+0000 7f1663400640 1 -- 172.21.15.40:0/2735285638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1638067600 msgr2=0x7f1638069ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.011+0000 7f1663400640 1 --2- 172.21.15.40:0/2735285638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1638067600 0x7f1638069ac0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f165c0cc360 tx=0x7f1644001320 comp rx=0 tx=0).stop 2024-03-24T00:30:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.011+0000 7f1663400640 1 -- 172.21.15.40:0/2735285638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f165c0cf1e0 msgr2=0x7f165c156410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.011+0000 7f1663400640 1 --2- 172.21.15.40:0/2735285638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f165c0cf1e0 0x7f165c156410 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f164c0b0040 tx=0x7f164c019240 comp rx=0 tx=0).stop 2024-03-24T00:30:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.011+0000 7f1663400640 1 -- 172.21.15.40:0/2735285638 shutdown_connections 2024-03-24T00:30:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.011+0000 7f1663400640 1 --2- 172.21.15.40:0/2735285638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1638067600 0x7f1638069ac0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.011+0000 7f1663400640 1 --2- 172.21.15.40:0/2735285638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f165c15f5c0 0x7f165c157e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.011+0000 7f1663400640 1 --2- 172.21.15.40:0/2735285638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f165c0cfbb0 0x7f165c1578e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.011+0000 7f1663400640 1 --2- 172.21.15.40:0/2735285638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f165c0cf1e0 0x7f165c156410 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.011+0000 7f1663400640 1 -- 172.21.15.40:0/2735285638 >> 172.21.15.40:0/2735285638 conn(0x7f165c0b3810 msgr2=0x7f165c0cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.015+0000 7f1663400640 1 -- 172.21.15.40:0/2735285638 shutdown_connections 2024-03-24T00:30:58.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.015+0000 7f1663400640 1 -- 172.21.15.40:0/2735285638 wait complete. 2024-03-24T00:30:58.045 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:30:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe28400640 1 Processor -- start 2024-03-24T00:30:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe28400640 1 -- start start 2024-03-24T00:30:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe28400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efe2015c0e0 0x7efe20074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe28400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe20073360 0x7efe200aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe28400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe200b2df0 con 0x7efe200afa60 2024-03-24T00:30:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe28400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe200cdb80 con 0x7efe20073360 2024-03-24T00:30:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe28400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe200cd8a0 con 0x7efe2015c0e0 2024-03-24T00:30:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe26400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe20073360 0x7efe200aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe26400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe20073360 0x7efe200aec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56964/0 (socket says 172.21.15.40:56964) 2024-03-24T00:30:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe26400640 1 -- 172.21.15.40:0/4278962925 learned_addr learned my addr 172.21.15.40:0/4278962925 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe25a00640 1 --2- 172.21.15.40:0/4278962925 >> v2:172.21.15.53:3300/0 conn(0x7efe2015c0e0 0x7efe20074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:58.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe26400640 1 -- 172.21.15.40:0/4278962925 >> v2:172.21.15.53:3300/0 conn(0x7efe2015c0e0 msgr2=0x7efe20074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:58.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe26400640 1 --2- 172.21.15.40:0/4278962925 >> v2:172.21.15.53:3300/0 conn(0x7efe2015c0e0 0x7efe20074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:58.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe26400640 1 -- 172.21.15.40:0/4278962925 >> v1:172.21.15.53:6789/0 conn(0x7efe200afa60 legacy=0x7efe200ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:58.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe26400640 1 -- 172.21.15.40:0/4278962925 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe200add20 con 0x7efe20073360 2024-03-24T00:30:58.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.191+0000 7efe25a00640 1 --2- 172.21.15.40:0/4278962925 >> v2:172.21.15.53:3300/0 conn(0x7efe2015c0e0 0x7efe20074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:30:58.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.195+0000 7efe26400640 1 --2- 172.21.15.40:0/4278962925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe20073360 0x7efe200aec30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efe200b6370 tx=0x7efe1801c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f3dc47ee38960f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:58.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.195+0000 7efe1fe00640 1 -- 172.21.15.40:0/4278962925 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe180a5740 con 0x7efe20073360 2024-03-24T00:30:58.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.195+0000 7efe1fe00640 1 -- 172.21.15.40:0/4278962925 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe200af380 con 0x7efe20073360 2024-03-24T00:30:58.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.195+0000 7efe1fe00640 1 -- 172.21.15.40:0/4278962925 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe180a5b00 con 0x7efe20073360 2024-03-24T00:30:58.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.195+0000 7efe28400640 1 -- 172.21.15.40:0/4278962925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe20073360 msgr2=0x7efe200aec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:58.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.195+0000 7efe28400640 1 --2- 172.21.15.40:0/4278962925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe20073360 0x7efe200aec30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efe200b6370 tx=0x7efe1801c860 comp rx=0 tx=0).stop 2024-03-24T00:30:58.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.195+0000 7efe28400640 1 -- 172.21.15.40:0/4278962925 shutdown_connections 2024-03-24T00:30:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.195+0000 7efe28400640 1 --2- 172.21.15.40:0/4278962925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe20073360 0x7efe200aec30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.195+0000 7efe28400640 1 --2- 172.21.15.40:0/4278962925 >> v2:172.21.15.53:3300/0 conn(0x7efe2015c0e0 0x7efe20074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.195+0000 7efe28400640 1 -- 172.21.15.40:0/4278962925 >> 172.21.15.40:0/4278962925 conn(0x7efe20072870 msgr2=0x7efe20072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.195+0000 7efe28400640 1 -- 172.21.15.40:0/4278962925 shutdown_connections 2024-03-24T00:30:58.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.195+0000 7efe28400640 1 -- 172.21.15.40:0/4278962925 wait complete. 2024-03-24T00:30:58.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe28400640 1 Processor -- start 2024-03-24T00:30:58.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe28400640 1 -- start start 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe28400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe20073360 0x7efe200cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe28400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe200afa60 0x7efe20147f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe28400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe2015c0e0 0x7efe200cf350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe28400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe200cd550 con 0x7efe200afa60 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe28400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe200cd3d0 con 0x7efe2015c0e0 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe28400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe20168aa0 con 0x7efe20073360 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe25000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe200afa60 0x7efe20147f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe25a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe20073360 0x7efe200cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe25000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe200afa60 0x7efe20147f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56976/0 (socket says 172.21.15.40:56976) 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe25000640 1 -- 172.21.15.40:0/3350899608 learned_addr learned my addr 172.21.15.40:0/3350899608 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe26400640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe2015c0e0 0x7efe200cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe25000640 1 -- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe20073360 msgr2=0x7efe200cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe25000640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe20073360 0x7efe200cebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe25000640 1 -- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe2015c0e0 msgr2=0x7efe200cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe25000640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe2015c0e0 0x7efe200cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:58.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe25000640 1 -- 172.21.15.40:0/3350899608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2014fd60 con 0x7efe200afa60 2024-03-24T00:30:58.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe25000640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe200afa60 0x7efe20147f90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00c410 tx=0x7efe0c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:58.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe1d600640 1 -- 172.21.15.40:0/3350899608 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c011790 con 0x7efe200afa60 2024-03-24T00:30:58.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe1d600640 1 -- 172.21.15.40:0/3350899608 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0c017070 con 0x7efe200afa60 2024-03-24T00:30:58.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe28400640 1 -- 172.21.15.40:0/3350899608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe20148b40 con 0x7efe200afa60 2024-03-24T00:30:58.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe1d600640 1 -- 172.21.15.40:0/3350899608 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0c011bb0 con 0x7efe200afa60 2024-03-24T00:30:58.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe28400640 1 -- 172.21.15.40:0/3350899608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe200b54a0 con 0x7efe200afa60 2024-03-24T00:30:58.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.199+0000 7efe28400640 1 -- 172.21.15.40:0/3350899608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe20068170 con 0x7efe200afa60 2024-03-24T00:30:58.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.203+0000 7efe1d600640 1 -- 172.21.15.40:0/3350899608 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7efe0c01c460 con 0x7efe200afa60 2024-03-24T00:30:58.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.203+0000 7efe1d600640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe00067690 0x7efe00069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:58.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.203+0000 7efe1d600640 1 -- 172.21.15.40:0/3350899608 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7efe0c085110 con 0x7efe200afa60 2024-03-24T00:30:58.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.203+0000 7efe25a00640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe00067690 0x7efe00069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:58.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.203+0000 7efe25a00640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe00067690 0x7efe00069b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efe14006fd0 tx=0x7efe14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:58.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.207+0000 7efe1d600640 1 -- 172.21.15.40:0/3350899608 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7efe0c021020 con 0x7efe200afa60 2024-03-24T00:30:58.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:58.371+0000 7efe28400640 1 -- 172.21.15.40:0/3350899608 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7efe20156cb0 con 0x7efe200afa60 2024-03-24T00:30:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.019+0000 7efe1d600640 1 -- 172.21.15.40:0/3350899608 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7efe0c003810 con 0x7efe200afa60 2024-03-24T00:30:59.022 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-03-24T00:30:59.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.023+0000 7efe28400640 1 -- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe00067690 msgr2=0x7efe00069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:59.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.023+0000 7efe28400640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe00067690 0x7efe00069b50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efe14006fd0 tx=0x7efe14008040 comp rx=0 tx=0).stop 2024-03-24T00:30:59.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.023+0000 7efe28400640 1 -- 172.21.15.40:0/3350899608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe200afa60 msgr2=0x7efe20147f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:59.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.023+0000 7efe28400640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe200afa60 0x7efe20147f90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efe0c00c410 tx=0x7efe0c00c8d0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.023+0000 7efe28400640 1 -- 172.21.15.40:0/3350899608 shutdown_connections 2024-03-24T00:30:59.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.023+0000 7efe28400640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe00067690 0x7efe00069b50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.023+0000 7efe28400640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe2015c0e0 0x7efe200cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.023+0000 7efe28400640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe200afa60 0x7efe20147f90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.023+0000 7efe28400640 1 --2- 172.21.15.40:0/3350899608 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe20073360 0x7efe200cebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.023+0000 7efe28400640 1 -- 172.21.15.40:0/3350899608 >> 172.21.15.40:0/3350899608 conn(0x7efe20072870 msgr2=0x7efe200ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:59.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.023+0000 7efe28400640 1 -- 172.21.15.40:0/3350899608 shutdown_connections 2024-03-24T00:30:59.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.027+0000 7efe28400640 1 -- 172.21.15.40:0/3350899608 wait complete. 2024-03-24T00:30:59.062 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:30:59.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07ab200640 1 Processor -- start 2024-03-24T00:30:59.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07ab200640 1 -- start start 2024-03-24T00:30:59.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07ab200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f07ac15e0e0 0x7f07ac0cd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:59.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07ab200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f07ac0b0bd0 0x7f07ac0ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:59.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07ab200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ac0cf2e0 con 0x7f07ac0d0070 2024-03-24T00:30:59.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07ab200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ac0cd0e0 con 0x7f07ac15e0e0 2024-03-24T00:30:59.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07ab200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ac0cc7b0 con 0x7f07ac0b0bd0 2024-03-24T00:30:59.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07a9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f07ac15e0e0 0x7f07ac0cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:59.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07a9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f07ac15e0e0 0x7f07ac0cd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56990/0 (socket says 172.21.15.40:56990) 2024-03-24T00:30:59.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07a9e00640 1 -- 172.21.15.40:0/3682461073 learned_addr learned my addr 172.21.15.40:0/3682461073 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:59.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07a9400640 1 --2- 172.21.15.40:0/3682461073 >> v2:172.21.15.53:3300/0 conn(0x7f07ac0b0bd0 0x7f07ac0ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:59.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07a9e00640 1 -- 172.21.15.40:0/3682461073 >> v2:172.21.15.53:3300/0 conn(0x7f07ac0b0bd0 msgr2=0x7f07ac0ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:30:59.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07a9e00640 1 --2- 172.21.15.40:0/3682461073 >> v2:172.21.15.53:3300/0 conn(0x7f07ac0b0bd0 0x7f07ac0ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07a9e00640 1 -- 172.21.15.40:0/3682461073 >> v1:172.21.15.53:6789/0 conn(0x7f07ac0d0070 legacy=0x7f07ac06b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:59.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.207+0000 7f07a9e00640 1 -- 172.21.15.40:0/3682461073 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac0cca90 con 0x7f07ac15e0e0 2024-03-24T00:30:59.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07a9e00640 1 --2- 172.21.15.40:0/3682461073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ac15e0e0 0x7f07ac0cd570 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f07ac161a40 tx=0x7f079c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7224f961ed2b1c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:59.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07a8a00640 1 -- 172.21.15.40:0/3682461073 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c0ab430 con 0x7f07ac15e0e0 2024-03-24T00:30:59.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07a8a00640 1 -- 172.21.15.40:0/3682461073 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079c0b0070 con 0x7f07ac15e0e0 2024-03-24T00:30:59.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07a8a00640 1 -- 172.21.15.40:0/3682461073 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c0ab7f0 con 0x7f07ac15e0e0 2024-03-24T00:30:59.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07ab200640 1 -- 172.21.15.40:0/3682461073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ac15e0e0 msgr2=0x7f07ac0cd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:59.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07ab200640 1 --2- 172.21.15.40:0/3682461073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ac15e0e0 0x7f07ac0cd570 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f07ac161a40 tx=0x7f079c01c520 comp rx=0 tx=0).stop 2024-03-24T00:30:59.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07ab200640 1 -- 172.21.15.40:0/3682461073 shutdown_connections 2024-03-24T00:30:59.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07ab200640 1 --2- 172.21.15.40:0/3682461073 >> v2:172.21.15.53:3300/0 conn(0x7f07ac0b0bd0 0x7f07ac0ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07ab200640 1 --2- 172.21.15.40:0/3682461073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ac15e0e0 0x7f07ac0cd570 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07ab200640 1 -- 172.21.15.40:0/3682461073 >> 172.21.15.40:0/3682461073 conn(0x7f07ac0b3620 msgr2=0x7f07ac0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:30:59.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07ab200640 1 -- 172.21.15.40:0/3682461073 shutdown_connections 2024-03-24T00:30:59.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07ab200640 1 -- 172.21.15.40:0/3682461073 wait complete. 2024-03-24T00:30:59.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.211+0000 7f07ab200640 1 Processor -- start 2024-03-24T00:30:59.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07ab200640 1 -- start start 2024-03-24T00:30:59.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07ab200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07ac0b0bd0 0x7f07ac1fb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07ab200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ac0d0070 0x7f07ac15c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07ab200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07ac15e0e0 0x7f07ac155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07ab200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac06bdf0 con 0x7f07ac0d0070 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07ab200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac06bc70 con 0x7f07ac0b0bd0 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07ab200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac16b630 con 0x7f07ac15e0e0 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ac0d0070 0x7f07ac15c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07ac0b0bd0 0x7f07ac1fb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ac0d0070 0x7f07ac15c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57000/0 (socket says 172.21.15.40:57000) 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a9400640 1 -- 172.21.15.40:0/1638392434 learned_addr learned my addr 172.21.15.40:0/1638392434 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07aa800640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07ac15e0e0 0x7f07ac155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a9400640 1 -- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07ac15e0e0 msgr2=0x7f07ac155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a9400640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07ac15e0e0 0x7f07ac155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a9400640 1 -- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07ac0b0bd0 msgr2=0x7f07ac1fb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a9400640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07ac0b0bd0 0x7f07ac1fb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:30:59.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a9400640 1 -- 172.21.15.40:0/1638392434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac0c5b10 con 0x7f07ac0d0070 2024-03-24T00:30:59.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07aa800640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07ac15e0e0 0x7f07ac155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:30:59.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a9e00640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07ac0b0bd0 0x7f07ac1fb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:30:59.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a9400640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ac0d0070 0x7f07ac15c2b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f079800d5f0 tx=0x7f079800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:59.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a2000640 1 -- 172.21.15.40:0/1638392434 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07980125f0 con 0x7f07ac0d0070 2024-03-24T00:30:59.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a2000640 1 -- 172.21.15.40:0/1638392434 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0798019070 con 0x7f07ac0d0070 2024-03-24T00:30:59.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07ab200640 1 -- 172.21.15.40:0/1638392434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ac0c5df0 con 0x7f07ac0d0070 2024-03-24T00:30:59.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07a2000640 1 -- 172.21.15.40:0/1638392434 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07980129b0 con 0x7f07ac0d0070 2024-03-24T00:30:59.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07ab200640 1 -- 172.21.15.40:0/1638392434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ac0c63b0 con 0x7f07ac0d0070 2024-03-24T00:30:59.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.215+0000 7f07ab200640 1 -- 172.21.15.40:0/1638392434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ac04f410 con 0x7f07ac0d0070 2024-03-24T00:30:59.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.219+0000 7f07a2000640 1 -- 172.21.15.40:0/1638392434 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f0798012b50 con 0x7f07ac0d0070 2024-03-24T00:30:59.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.219+0000 7f07a2000640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f078c067660 0x7f078c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:30:59.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.219+0000 7f07a2000640 1 -- 172.21.15.40:0/1638392434 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f0798086690 con 0x7f07ac0d0070 2024-03-24T00:30:59.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.219+0000 7f07a9e00640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f078c067660 0x7f078c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:30:59.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.219+0000 7f07a9e00640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f078c067660 0x7f078c069b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f079c0af000 tx=0x7f079c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:30:59.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.223+0000 7f07a2000640 1 -- 172.21.15.40:0/1638392434 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f07980573e0 con 0x7f07ac0d0070 2024-03-24T00:30:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:30:59.423+0000 7f07ab200640 1 -- 172.21.15.40:0/1638392434 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f07ac0c8480 con 0x7f07ac0d0070 2024-03-24T00:31:00.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.031+0000 7f07a2000640 1 -- 172.21.15.40:0/1638392434 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0798056ce0 con 0x7f07ac0d0070 2024-03-24T00:31:00.039 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 2 and data pool 3 2024-03-24T00:31:00.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.043+0000 7f07ab200640 1 -- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f078c067660 msgr2=0x7f078c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.043+0000 7f07ab200640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f078c067660 0x7f078c069b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f079c0af000 tx=0x7f079c0a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:00.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.043+0000 7f07ab200640 1 -- 172.21.15.40:0/1638392434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ac0d0070 msgr2=0x7f07ac15c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.043+0000 7f07ab200640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ac0d0070 0x7f07ac15c2b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f079800d5f0 tx=0x7f079800dab0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.043+0000 7f07ab200640 1 -- 172.21.15.40:0/1638392434 shutdown_connections 2024-03-24T00:31:00.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.043+0000 7f07ab200640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f078c067660 0x7f078c069b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.043+0000 7f07ab200640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07ac15e0e0 0x7f07ac155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.043+0000 7f07ab200640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07ac0d0070 0x7f07ac15c2b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.043+0000 7f07ab200640 1 --2- 172.21.15.40:0/1638392434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07ac0b0bd0 0x7f07ac1fb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.043+0000 7f07ab200640 1 -- 172.21.15.40:0/1638392434 >> 172.21.15.40:0/1638392434 conn(0x7f07ac0b3620 msgr2=0x7f07ac070350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:00.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.043+0000 7f07ab200640 1 -- 172.21.15.40:0/1638392434 shutdown_connections 2024-03-24T00:31:00.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.043+0000 7f07ab200640 1 -- 172.21.15.40:0/1638392434 wait complete. 2024-03-24T00:31:00.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:00.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa386200640 1 Processor -- start 2024-03-24T00:31:00.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa386200640 1 -- start start 2024-03-24T00:31:00.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa386200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa38015dfc0 0x7fa3800cd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa386200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa3800d0b40 0x7fa3800cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa386200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3800cfaa0 con 0x7fa38015e3a0 2024-03-24T00:31:00.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa386200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3800cf870 con 0x7fa3800d0b40 2024-03-24T00:31:00.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa386200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3800cfd80 con 0x7fa38015dfc0 2024-03-24T00:31:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa37fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa3800d0b40 0x7fa3800cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa37fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa3800d0b40 0x7fa3800cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57014/0 (socket says 172.21.15.40:57014) 2024-03-24T00:31:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa37fe00640 1 -- 172.21.15.40:0/3558986127 learned_addr learned my addr 172.21.15.40:0/3558986127 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa37f400640 1 --2- 172.21.15.40:0/3558986127 >> v2:172.21.15.53:3300/0 conn(0x7fa38015dfc0 0x7fa3800cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa37fe00640 1 -- 172.21.15.40:0/3558986127 >> v2:172.21.15.53:3300/0 conn(0x7fa38015dfc0 msgr2=0x7fa3800cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa37fe00640 1 --2- 172.21.15.40:0/3558986127 >> v2:172.21.15.53:3300/0 conn(0x7fa38015dfc0 0x7fa3800cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa37fe00640 1 -- 172.21.15.40:0/3558986127 >> v1:172.21.15.53:6789/0 conn(0x7fa38015e3a0 legacy=0x7fa3800ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.219+0000 7fa37fe00640 1 -- 172.21.15.40:0/3558986127 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3800d0060 con 0x7fa3800d0b40 2024-03-24T00:31:00.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.223+0000 7fa37fe00640 1 --2- 172.21.15.40:0/3558986127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3800d0b40 0x7fa3800cd8d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa3800720f0 tx=0x7fa37801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f43027373d9f4f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:00.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.223+0000 7fa37ea00640 1 -- 172.21.15.40:0/3558986127 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3780a56a0 con 0x7fa3800d0b40 2024-03-24T00:31:00.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.223+0000 7fa37ea00640 1 -- 172.21.15.40:0/3558986127 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3780a5840 con 0x7fa3800d0b40 2024-03-24T00:31:00.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.223+0000 7fa37ea00640 1 -- 172.21.15.40:0/3558986127 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3780a5c10 con 0x7fa3800d0b40 2024-03-24T00:31:00.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.223+0000 7fa386200640 1 -- 172.21.15.40:0/3558986127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3800d0b40 msgr2=0x7fa3800cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.223+0000 7fa386200640 1 --2- 172.21.15.40:0/3558986127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3800d0b40 0x7fa3800cd8d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa3800720f0 tx=0x7fa37801c8d0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.223+0000 7fa386200640 1 -- 172.21.15.40:0/3558986127 shutdown_connections 2024-03-24T00:31:00.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.223+0000 7fa386200640 1 --2- 172.21.15.40:0/3558986127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3800d0b40 0x7fa3800cd8d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.223+0000 7fa386200640 1 --2- 172.21.15.40:0/3558986127 >> v2:172.21.15.53:3300/0 conn(0x7fa38015dfc0 0x7fa3800cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.223+0000 7fa386200640 1 -- 172.21.15.40:0/3558986127 >> 172.21.15.40:0/3558986127 conn(0x7fa3800b37f0 msgr2=0x7fa3800b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:00.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.223+0000 7fa386200640 1 -- 172.21.15.40:0/3558986127 shutdown_connections 2024-03-24T00:31:00.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.223+0000 7fa386200640 1 -- 172.21.15.40:0/3558986127 wait complete. 2024-03-24T00:31:00.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa386200640 1 Processor -- start 2024-03-24T00:31:00.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa386200640 1 -- start start 2024-03-24T00:31:00.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa386200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa3800d0b40 0x7fa38015b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa386200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa38015dfc0 0x7fa3800b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa386200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa38015e3a0 0x7fa380156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa386200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38016ab60 con 0x7fa38015e3a0 2024-03-24T00:31:00.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa386200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38016a9e0 con 0x7fa38015dfc0 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa386200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38016ace0 con 0x7fa3800d0b40 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa37fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa38015e3a0 0x7fa380156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa37fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa38015e3a0 0x7fa380156210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57026/0 (socket says 172.21.15.40:57026) 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa37fe00640 1 -- 172.21.15.40:0/4059283527 learned_addr learned my addr 172.21.15.40:0/4059283527 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa376a00640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa38015dfc0 0x7fa3800b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa37f400640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa3800d0b40 0x7fa38015b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa37fe00640 1 -- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa3800d0b40 msgr2=0x7fa38015b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa37fe00640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa3800d0b40 0x7fa38015b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa37fe00640 1 -- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa38015dfc0 msgr2=0x7fa3800b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa37fe00640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa38015dfc0 0x7fa3800b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa37fe00640 1 -- 172.21.15.40:0/4059283527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa380152310 con 0x7fa38015e3a0 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa37f400640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa3800d0b40 0x7fa38015b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa376a00640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa38015dfc0 0x7fa3800b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa37fe00640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa38015e3a0 0x7fa380156210 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa3780af040 tx=0x7fa378019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:00.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa377e00640 1 -- 172.21.15.40:0/4059283527 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3780bc020 con 0x7fa38015e3a0 2024-03-24T00:31:00.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa377e00640 1 -- 172.21.15.40:0/4059283527 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3780b8470 con 0x7fa38015e3a0 2024-03-24T00:31:00.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa377e00640 1 -- 172.21.15.40:0/4059283527 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa378009510 con 0x7fa38015e3a0 2024-03-24T00:31:00.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa386200640 1 -- 172.21.15.40:0/4059283527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3801525f0 con 0x7fa38015e3a0 2024-03-24T00:31:00.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.227+0000 7fa386200640 1 -- 172.21.15.40:0/4059283527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa380152bb0 con 0x7fa38015e3a0 2024-03-24T00:31:00.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.231+0000 7fa386200640 1 -- 172.21.15.40:0/4059283527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa380068170 con 0x7fa38015e3a0 2024-03-24T00:31:00.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.231+0000 7fa377e00640 1 -- 172.21.15.40:0/4059283527 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa3780bf020 con 0x7fa38015e3a0 2024-03-24T00:31:00.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.231+0000 7fa377e00640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa35c0676e0 0x7fa35c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.231+0000 7fa377e00640 1 -- 172.21.15.40:0/4059283527 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa378120500 con 0x7fa38015e3a0 2024-03-24T00:31:00.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.231+0000 7fa37f400640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa35c0676e0 0x7fa35c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.231+0000 7fa37f400640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa35c0676e0 0x7fa35c069ba0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa3800b35d0 tx=0x7fa370008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:00.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.235+0000 7fa377e00640 1 -- 172.21.15.40:0/4059283527 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa3780c6030 con 0x7fa38015e3a0 2024-03-24T00:31:00.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 -- 172.21.15.40:0/4059283527 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3800b7c20 con 0x7fa38015e3a0 2024-03-24T00:31:00.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa377e00640 1 -- 172.21.15.40:0/4059283527 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16558 (secure 0 0 0) 0x7fa3780165e0 con 0x7fa38015e3a0 2024-03-24T00:31:00.400 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:00.401 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":19,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:30:59.427346+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:00.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 -- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa35c0676e0 msgr2=0x7fa35c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa35c0676e0 0x7fa35c069ba0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa3800b35d0 tx=0x7fa370008040 comp rx=0 tx=0).stop 2024-03-24T00:31:00.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 -- 172.21.15.40:0/4059283527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa38015e3a0 msgr2=0x7fa380156210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa38015e3a0 0x7fa380156210 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa3780af040 tx=0x7fa378019670 comp rx=0 tx=0).stop 2024-03-24T00:31:00.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 -- 172.21.15.40:0/4059283527 shutdown_connections 2024-03-24T00:31:00.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa35c0676e0 0x7fa35c069ba0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa38015e3a0 0x7fa380156210 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa38015dfc0 0x7fa3800b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 --2- 172.21.15.40:0/4059283527 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa3800d0b40 0x7fa38015b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 -- 172.21.15.40:0/4059283527 >> 172.21.15.40:0/4059283527 conn(0x7fa3800b37f0 msgr2=0x7fa3800ce3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:00.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 -- 172.21.15.40:0/4059283527 shutdown_connections 2024-03-24T00:31:00.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.399+0000 7fa386200640 1 -- 172.21.15.40:0/4059283527 wait complete. 2024-03-24T00:31:00.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:00.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.559+0000 7f00d1400640 1 Processor -- start 2024-03-24T00:31:00.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.559+0000 7f00d1400640 1 -- start start 2024-03-24T00:31:00.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00d1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f00cc13ec80 0x7f00cc1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00d1400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00cc0ce800 0x7f00cc071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00d1400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00cc070e00 con 0x7f00cc1419f0 2024-03-24T00:31:00.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00d1400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00cc141e50 con 0x7f00cc13ec80 2024-03-24T00:31:00.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00d1400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00cc0cee20 con 0x7f00cc0ce800 2024-03-24T00:31:00.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00ca800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f00cc13ec80 0x7f00cc1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00ca800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f00cc13ec80 0x7f00cc1414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57038/0 (socket says 172.21.15.40:57038) 2024-03-24T00:31:00.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00ca800640 1 -- 172.21.15.40:0/311225719 learned_addr learned my addr 172.21.15.40:0/311225719 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:00.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00c1e00640 1 --1- 172.21.15.40:0/311225719 >> v1:172.21.15.40:6789/0 conn(0x7f00cc1419f0 0x7f00cc0ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55052/0 (socket says 172.21.15.40:55052) 2024-03-24T00:31:00.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00cb200640 1 --2- 172.21.15.40:0/311225719 >> v2:172.21.15.53:3300/0 conn(0x7f00cc0ce800 0x7f00cc071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00ca800640 1 -- 172.21.15.40:0/311225719 >> v2:172.21.15.53:3300/0 conn(0x7f00cc0ce800 msgr2=0x7f00cc071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00ca800640 1 --2- 172.21.15.40:0/311225719 >> v2:172.21.15.53:3300/0 conn(0x7f00cc0ce800 0x7f00cc071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00ca800640 1 -- 172.21.15.40:0/311225719 >> v1:172.21.15.40:6789/0 conn(0x7f00cc1419f0 legacy=0x7f00cc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00ca800640 1 -- 172.21.15.40:0/311225719 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00cc0cf100 con 0x7f00cc13ec80 2024-03-24T00:31:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.563+0000 7f00cb200640 1 --2- 172.21.15.40:0/311225719 >> v2:172.21.15.53:3300/0 conn(0x7f00cc0ce800 0x7f00cc071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00ca800640 1 --2- 172.21.15.40:0/311225719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00cc13ec80 0x7f00cc1414b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00cc067e60 tx=0x7f00bc01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e72512844934a563 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00c9e00640 1 -- 172.21.15.40:0/311225719 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc0a58e0 con 0x7f00cc13ec80 2024-03-24T00:31:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00c9e00640 1 -- 172.21.15.40:0/311225719 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00bc0a5a80 con 0x7f00cc13ec80 2024-03-24T00:31:00.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00c9e00640 1 -- 172.21.15.40:0/311225719 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc0a5e50 con 0x7f00cc13ec80 2024-03-24T00:31:00.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 -- 172.21.15.40:0/311225719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00cc13ec80 msgr2=0x7f00cc1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 --2- 172.21.15.40:0/311225719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00cc13ec80 0x7f00cc1414b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f00cc067e60 tx=0x7f00bc01c9f0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 -- 172.21.15.40:0/311225719 shutdown_connections 2024-03-24T00:31:00.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 --2- 172.21.15.40:0/311225719 >> v2:172.21.15.53:3300/0 conn(0x7f00cc0ce800 0x7f00cc071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 --2- 172.21.15.40:0/311225719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00cc13ec80 0x7f00cc1414b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 -- 172.21.15.40:0/311225719 >> 172.21.15.40:0/311225719 conn(0x7f00cc0b17b0 msgr2=0x7f00cc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:00.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 -- 172.21.15.40:0/311225719 shutdown_connections 2024-03-24T00:31:00.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 -- 172.21.15.40:0/311225719 wait complete. 2024-03-24T00:31:00.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 Processor -- start 2024-03-24T00:31:00.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 -- start start 2024-03-24T00:31:00.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00cc0ce800 0x7f00cc140690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00cc13ec80 0x7f00cc1541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00cc1419f0 0x7f00cc154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00cc0723e0 con 0x7f00cc13ec80 2024-03-24T00:31:00.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00cc072260 con 0x7f00cc1419f0 2024-03-24T00:31:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00d1400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00cc072560 con 0x7f00cc0ce800 2024-03-24T00:31:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00c1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00cc13ec80 0x7f00cc1541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00c1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00cc13ec80 0x7f00cc1541e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57050/0 (socket says 172.21.15.40:57050) 2024-03-24T00:31:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.567+0000 7f00c1e00640 1 -- 172.21.15.40:0/4085263942 learned_addr learned my addr 172.21.15.40:0/4085263942 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00cb200640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00cc1419f0 0x7f00cc154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00ca800640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00cc0ce800 0x7f00cc140690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00c1e00640 1 -- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00cc0ce800 msgr2=0x7f00cc140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00c1e00640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00cc0ce800 0x7f00cc140690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00c1e00640 1 -- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00cc1419f0 msgr2=0x7f00cc154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00c1e00640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00cc1419f0 0x7f00cc154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00c1e00640 1 -- 172.21.15.40:0/4085263942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00cc13cca0 con 0x7f00cc13ec80 2024-03-24T00:31:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00cb200640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00cc1419f0 0x7f00cc154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00c1e00640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00cc13ec80 0x7f00cc1541e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f00b400c420 tx=0x7f00b400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00c3400640 1 -- 172.21.15.40:0/4085263942 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40074e0 con 0x7f00cc13ec80 2024-03-24T00:31:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00c3400640 1 -- 172.21.15.40:0/4085263942 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b4007680 con 0x7f00cc13ec80 2024-03-24T00:31:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00c3400640 1 -- 172.21.15.40:0/4085263942 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4004a30 con 0x7f00cc13ec80 2024-03-24T00:31:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00d1400640 1 -- 172.21.15.40:0/4085263942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00cc13cf80 con 0x7f00cc13ec80 2024-03-24T00:31:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00d1400640 1 -- 172.21.15.40:0/4085263942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00cc13d540 con 0x7f00cc13ec80 2024-03-24T00:31:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00d1400640 1 -- 172.21.15.40:0/4085263942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00cc068170 con 0x7f00cc13ec80 2024-03-24T00:31:00.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00c3400640 1 -- 172.21.15.40:0/4085263942 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f00b401a020 con 0x7f00cc13ec80 2024-03-24T00:31:00.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00c3400640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f00a8067660 0x7f00a8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.571+0000 7f00c3400640 1 -- 172.21.15.40:0/4085263942 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00b40851b0 con 0x7f00cc13ec80 2024-03-24T00:31:00.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.575+0000 7f00ca800640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f00a8067660 0x7f00a8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.575+0000 7f00ca800640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f00a8067660 0x7f00a8069b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0ae000 tx=0x7f00bc0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:00.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.575+0000 7f00c3400640 1 -- 172.21.15.40:0/4085263942 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f00b4055d90 con 0x7f00cc13ec80 2024-03-24T00:31:00.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.739+0000 7f00d1400640 1 -- 172.21.15.40:0/4085263942 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00cc072b90 con 0x7f00cc13ec80 2024-03-24T00:31:00.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00c3400640 1 -- 172.21.15.40:0/4085263942 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16558 (secure 0 0 0) 0x7f00b4055690 con 0x7f00cc13ec80 2024-03-24T00:31:00.743 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:00.744 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":19,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:30:59.427346+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:00.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00d1400640 1 -- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f00a8067660 msgr2=0x7f00a8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00d1400640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f00a8067660 0x7f00a8069b20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0ae000 tx=0x7f00bc0a7070 comp rx=0 tx=0).stop 2024-03-24T00:31:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00d1400640 1 -- 172.21.15.40:0/4085263942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00cc13ec80 msgr2=0x7f00cc1541e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00d1400640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00cc13ec80 0x7f00cc1541e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f00b400c420 tx=0x7f00b400c8e0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00d1400640 1 -- 172.21.15.40:0/4085263942 shutdown_connections 2024-03-24T00:31:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00d1400640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f00a8067660 0x7f00a8069b20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00d1400640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00cc1419f0 0x7f00cc154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00d1400640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00cc13ec80 0x7f00cc1541e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00d1400640 1 --2- 172.21.15.40:0/4085263942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00cc0ce800 0x7f00cc140690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00d1400640 1 -- 172.21.15.40:0/4085263942 >> 172.21.15.40:0/4085263942 conn(0x7f00cc0b17b0 msgr2=0x7f00cc13b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00d1400640 1 -- 172.21.15.40:0/4085263942 shutdown_connections 2024-03-24T00:31:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.743+0000 7f00d1400640 1 -- 172.21.15.40:0/4085263942 wait complete. 2024-03-24T00:31:00.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:31:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.907+0000 7f0809000640 1 Processor -- start 2024-03-24T00:31:00.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.907+0000 7f0809000640 1 -- start start 2024-03-24T00:31:00.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.907+0000 7f0809000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08040b2360 0x7f08040b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.907+0000 7f0809000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08040cf090 con 0x7f08040cebd0 2024-03-24T00:31:00.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.907+0000 7f0809000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08040cf330 con 0x7f08040b4330 2024-03-24T00:31:00.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.907+0000 7f0809000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08040d0f60 con 0x7f08040b2360 2024-03-24T00:31:00.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.907+0000 7f0801e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08040b2360 0x7f08040b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.907+0000 7f0801e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f08040b2360 0x7f08040b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57054/0 (socket says 172.21.15.40:57054) 2024-03-24T00:31:00.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.907+0000 7f0801e00640 1 -- 172.21.15.40:0/1833169412 learned_addr learned my addr 172.21.15.40:0/1833169412 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.907+0000 7f0801e00640 1 -- 172.21.15.40:0/1833169412 >> v1:172.21.15.53:6789/0 conn(0x7f08040b4330 legacy=0x7f08040ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.907+0000 7f0801e00640 1 -- 172.21.15.40:0/1833169412 >> v1:172.21.15.40:6789/0 conn(0x7f08040cebd0 legacy=0x7f08040cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.907+0000 7f0801e00640 1 -- 172.21.15.40:0/1833169412 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08040d0910 con 0x7f08040b2360 2024-03-24T00:31:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.911+0000 7f0801e00640 1 --2- 172.21.15.40:0/1833169412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08040b2360 0x7f08040b2740 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009870 tx=0x7f07f401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb5f6afcdd5827b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.911+0000 7f0801400640 1 -- 172.21.15.40:0/1833169412 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f40b1070 con 0x7f08040b2360 2024-03-24T00:31:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.911+0000 7f0801400640 1 -- 172.21.15.40:0/1833169412 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f40ac630 con 0x7f08040b2360 2024-03-24T00:31:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.911+0000 7f0801400640 1 -- 172.21.15.40:0/1833169412 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f40a8050 con 0x7f08040b2360 2024-03-24T00:31:00.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.911+0000 7f0809000640 1 -- 172.21.15.40:0/1833169412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08040b2360 msgr2=0x7f08040b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.911+0000 7f0809000640 1 --2- 172.21.15.40:0/1833169412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08040b2360 0x7f08040b2740 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009870 tx=0x7f07f401c3d0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.911+0000 7f0809000640 1 -- 172.21.15.40:0/1833169412 shutdown_connections 2024-03-24T00:31:00.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.911+0000 7f0809000640 1 --2- 172.21.15.40:0/1833169412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08040b2360 0x7f08040b2740 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.911+0000 7f0809000640 1 -- 172.21.15.40:0/1833169412 >> 172.21.15.40:0/1833169412 conn(0x7f08040b3810 msgr2=0x7f08040b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:00.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.911+0000 7f0809000640 1 -- 172.21.15.40:0/1833169412 shutdown_connections 2024-03-24T00:31:00.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.911+0000 7f0809000640 1 -- 172.21.15.40:0/1833169412 wait complete. 2024-03-24T00:31:00.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0809000640 1 Processor -- start 2024-03-24T00:31:00.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0809000640 1 -- start start 2024-03-24T00:31:00.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0809000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08040b2360 0x7f0804158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0809000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f08040b4330 0x7f08040b71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0809000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08040cebd0 0x7f0804156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0809000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08040cd4e0 con 0x7f08040b2360 2024-03-24T00:31:00.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0809000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08040cd360 con 0x7f08040cebd0 2024-03-24T00:31:00.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0809000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08040cd660 con 0x7f08040b4330 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0801e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08040b2360 0x7f0804158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0801e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08040b2360 0x7f0804158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57056/0 (socket says 172.21.15.40:57056) 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0801e00640 1 -- 172.21.15.40:0/1414563979 learned_addr learned my addr 172.21.15.40:0/1414563979 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f07fbe00640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f08040b4330 0x7f08040b71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0802800640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08040cebd0 0x7f0804156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0801e00640 1 -- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f08040b4330 msgr2=0x7f08040b71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0801e00640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f08040b4330 0x7f08040b71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0801e00640 1 -- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08040cebd0 msgr2=0x7f0804156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0801e00640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08040cebd0 0x7f0804156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0801e00640 1 -- 172.21.15.40:0/1414563979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08041577c0 con 0x7f08040b2360 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0801e00640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08040b2360 0x7f0804158860 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f07f40af040 tx=0x7f07f401ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f07fa000640 1 -- 172.21.15.40:0/1414563979 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f40bb020 con 0x7f08040b2360 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f07fa000640 1 -- 172.21.15.40:0/1414563979 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f40b7640 con 0x7f08040b2360 2024-03-24T00:31:00.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0809000640 1 -- 172.21.15.40:0/1414563979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0804157aa0 con 0x7f08040b2360 2024-03-24T00:31:00.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f07fa000640 1 -- 172.21.15.40:0/1414563979 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f40b1040 con 0x7f08040b2360 2024-03-24T00:31:00.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0809000640 1 -- 172.21.15.40:0/1414563979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0804158060 con 0x7f08040b2360 2024-03-24T00:31:00.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.915+0000 7f0809000640 1 -- 172.21.15.40:0/1414563979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0804068170 con 0x7f08040b2360 2024-03-24T00:31:00.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.919+0000 7f07fa000640 1 -- 172.21.15.40:0/1414563979 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f07f40be020 con 0x7f08040b2360 2024-03-24T00:31:00.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.919+0000 7f07fa000640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07e0067660 0x7f07e0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:00.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.919+0000 7f07fa000640 1 -- 172.21.15.40:0/1414563979 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07f411fcb0 con 0x7f08040b2360 2024-03-24T00:31:00.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.919+0000 7f07fbe00640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07e0067660 0x7f07e0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:00.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.919+0000 7f07fbe00640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07e0067660 0x7f07e0069b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08040b2b60 tx=0x7f07ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:00.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:00.923+0000 7f07fa000640 1 -- 172.21.15.40:0/1414563979 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f07f4016110 con 0x7f08040b2360 2024-03-24T00:31:01.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:01.119+0000 7f0809000640 1 -- 172.21.15.40:0/1414563979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0804068170 con 0x7f08040b2360 2024-03-24T00:31:02.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.043+0000 7f07fa000640 1 -- 172.21.15.40:0/1414563979 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f07f40f0890 con 0x7f08040b2360 2024-03-24T00:31:02.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.047+0000 7f0809000640 1 -- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07e0067660 msgr2=0x7f07e0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.047+0000 7f0809000640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07e0067660 0x7f07e0069b20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08040b2b60 tx=0x7f07ec008040 comp rx=0 tx=0).stop 2024-03-24T00:31:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.047+0000 7f0809000640 1 -- 172.21.15.40:0/1414563979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08040b2360 msgr2=0x7f0804158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.047+0000 7f0809000640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08040b2360 0x7f0804158860 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f07f40af040 tx=0x7f07f401ac10 comp rx=0 tx=0).stop 2024-03-24T00:31:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.047+0000 7f0809000640 1 -- 172.21.15.40:0/1414563979 shutdown_connections 2024-03-24T00:31:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.047+0000 7f0809000640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07e0067660 0x7f07e0069b20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.047+0000 7f0809000640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08040cebd0 0x7f0804156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.047+0000 7f0809000640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f08040b4330 0x7f08040b71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.047+0000 7f0809000640 1 --2- 172.21.15.40:0/1414563979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08040b2360 0x7f0804158860 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.047+0000 7f0809000640 1 -- 172.21.15.40:0/1414563979 >> 172.21.15.40:0/1414563979 conn(0x7f08040b3810 msgr2=0x7f08040b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.047+0000 7f0809000640 1 -- 172.21.15.40:0/1414563979 shutdown_connections 2024-03-24T00:31:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.047+0000 7f0809000640 1 -- 172.21.15.40:0/1414563979 wait complete. 2024-03-24T00:31:02.071 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:31:02.071 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:31:02.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.223+0000 7f4e46800640 1 Processor -- start 2024-03-24T00:31:02.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.223+0000 7f4e46800640 1 -- start start 2024-03-24T00:31:02.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e46800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e40161360 0x7f4e401617f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:02.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e46800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e400cff70 con 0x7f4e400d0d30 2024-03-24T00:31:02.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e46800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e400d0250 con 0x7f4e400ce700 2024-03-24T00:31:02.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e46800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e400d1190 con 0x7f4e40161360 2024-03-24T00:31:02.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e37e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4e400d0d30 0x7f4e400ceae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55072/0 (socket says 172.21.15.40:55072) 2024-03-24T00:31:02.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e37e00640 1 -- 172.21.15.40:0/199547950 learned_addr learned my addr 172.21.15.40:0/199547950 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:02.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3fe00640 1 --2- 172.21.15.40:0/199547950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e40161360 0x7f4e401617f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:02.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528031986 0 0) 0x7f4e400cff70 con 0x7f4e400d0d30 2024-03-24T00:31:02.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e24003530 con 0x7f4e400d0d30 2024-03-24T00:31:02.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2071696812 0 0) 0x7f4e24003530 con 0x7f4e400d0d30 2024-03-24T00:31:02.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e40161360 msgr2=0x7f4e401617f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:02.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 --2- 172.21.15.40:0/199547950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e40161360 0x7f4e401617f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:02.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 >> v1:172.21.15.53:6789/0 conn(0x7f4e400ce700 legacy=0x7f4e4006bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e400d0530 con 0x7f4e400d0d30 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f4e28001680 con 0x7f4e400d0d30 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 >> v1:172.21.15.40:6789/0 conn(0x7f4e400d0d30 legacy=0x7f4e400ceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 --2- 172.21.15.40:0/199547950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e240039f0 0x7f4e24003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 --2- 172.21.15.40:0/199547950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e40161360 0x7f4e24004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 --2- 172.21.15.40:0/199547950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e24004d50 0x7f4e240051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e400d1190 con 0x7f4e24004d50 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2400bab0 con 0x7f4e240039f0 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2400bcb0 con 0x7f4e40161360 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3fe00640 1 --2- 172.21.15.40:0/199547950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e24004d50 0x7f4e240051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3f400640 1 --2- 172.21.15.40:0/199547950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e240039f0 0x7f4e24003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e37e00640 1 --2- 172.21.15.40:0/199547950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e40161360 0x7f4e24004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3fe00640 1 -- 172.21.15.40:0/199547950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e40161360 msgr2=0x7f4e24004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3fe00640 1 --2- 172.21.15.40:0/199547950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e40161360 0x7f4e24004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3fe00640 1 -- 172.21.15.40:0/199547950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e240039f0 msgr2=0x7f4e24003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3fe00640 1 --2- 172.21.15.40:0/199547950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e240039f0 0x7f4e24003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.227+0000 7f4e3fe00640 1 -- 172.21.15.40:0/199547950 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24003530 con 0x7f4e24004d50 2024-03-24T00:31:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e3fe00640 1 --2- 172.21.15.40:0/199547950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e24004d50 0x7f4e240051d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e38009f90 tx=0x7f4e3801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cc7c378aac6e228 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:02.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e380b2070 con 0x7f4e24004d50 2024-03-24T00:31:02.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e380a5c50 con 0x7f4e24004d50 2024-03-24T00:31:02.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e3ea00640 1 -- 172.21.15.40:0/199547950 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e38009c10 con 0x7f4e24004d50 2024-03-24T00:31:02.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e46800640 1 -- 172.21.15.40:0/199547950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e24004d50 msgr2=0x7f4e240051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:02.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e46800640 1 --2- 172.21.15.40:0/199547950 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e24004d50 0x7f4e240051d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e38009f90 tx=0x7f4e3801cc40 comp rx=0 tx=0).stop 2024-03-24T00:31:02.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e3fe00640 1 -- 172.21.15.40:0/199547950 reap_dead start 2024-03-24T00:31:02.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e46800640 1 -- 172.21.15.40:0/199547950 shutdown_connections 2024-03-24T00:31:02.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e46800640 1 -- 172.21.15.40:0/199547950 >> 172.21.15.40:0/199547950 conn(0x7f4e400b3730 msgr2=0x7f4e400b3b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:02.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e46800640 1 -- 172.21.15.40:0/199547950 shutdown_connections 2024-03-24T00:31:02.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e46800640 1 -- 172.21.15.40:0/199547950 wait complete. 2024-03-24T00:31:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e46800640 1 Processor -- start 2024-03-24T00:31:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.231+0000 7f4e46800640 1 -- start start 2024-03-24T00:31:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e46800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e24004d50 0x7f4e40158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e46800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e401f7330 0x7f4e40159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e46800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e4015a740 0x7f4e4015abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e46800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e400cf1f0 con 0x7f4e401f7330 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e46800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e400d08a0 con 0x7f4e4015a740 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e46800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e401617f0 con 0x7f4e24004d50 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e401f7330 0x7f4e40159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e401f7330 0x7f4e40159f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57084/0 (socket says 172.21.15.40:57084) 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37e00640 1 -- 172.21.15.40:0/2786366614 learned_addr learned my addr 172.21.15.40:0/2786366614 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e3f400640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e24004d50 0x7f4e40158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e3fe00640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e4015a740 0x7f4e4015abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37e00640 1 -- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e24004d50 msgr2=0x7f4e40158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37e00640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e24004d50 0x7f4e40158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37e00640 1 -- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e4015a740 msgr2=0x7f4e4015abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37e00640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e4015a740 0x7f4e4015abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:02.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37e00640 1 -- 172.21.15.40:0/2786366614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e40154f60 con 0x7f4e401f7330 2024-03-24T00:31:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e3f400640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e24004d50 0x7f4e40158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e3fe00640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e4015a740 0x7f4e4015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37e00640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e401f7330 0x7f4e40159f60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4e28007290 tx=0x7f4e28007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37400640 1 -- 172.21.15.40:0/2786366614 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2800f8e0 con 0x7f4e401f7330 2024-03-24T00:31:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37400640 1 -- 172.21.15.40:0/2786366614 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e28018070 con 0x7f4e401f7330 2024-03-24T00:31:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e46800640 1 -- 172.21.15.40:0/2786366614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e401f7b80 con 0x7f4e401f7330 2024-03-24T00:31:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37400640 1 -- 172.21.15.40:0/2786366614 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2800fce0 con 0x7f4e401f7330 2024-03-24T00:31:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e46800640 1 -- 172.21.15.40:0/2786366614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e401f8140 con 0x7f4e401f7330 2024-03-24T00:31:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e46800640 1 -- 172.21.15.40:0/2786366614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e40068170 con 0x7f4e401f7330 2024-03-24T00:31:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.235+0000 7f4e37400640 1 -- 172.21.15.40:0/2786366614 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4e2801c410 con 0x7f4e401f7330 2024-03-24T00:31:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.239+0000 7f4e37400640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4e1c067690 0x7f4e1c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.239+0000 7f4e37400640 1 -- 172.21.15.40:0/2786366614 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e28085700 con 0x7f4e401f7330 2024-03-24T00:31:02.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.239+0000 7f4e3f400640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4e1c067690 0x7f4e1c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:02.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.239+0000 7f4e3f400640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4e1c067690 0x7f4e1c069b50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4e30007fd0 tx=0x7f4e30009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.243+0000 7f4e37400640 1 -- 172.21.15.40:0/2786366614 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4e280562e0 con 0x7f4e401f7330 2024-03-24T00:31:02.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:02.439+0000 7f4e46800640 1 -- 172.21.15.40:0/2786366614 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4e400b7ba0 con 0x7f4e401f7330 2024-03-24T00:31:03.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e37400640 1 -- 172.21.15.40:0/2786366614 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4e28055be0 con 0x7f4e401f7330 2024-03-24T00:31:03.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e46800640 1 -- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4e1c067690 msgr2=0x7f4e1c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e46800640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4e1c067690 0x7f4e1c069b50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4e30007fd0 tx=0x7f4e30009040 comp rx=0 tx=0).stop 2024-03-24T00:31:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e46800640 1 -- 172.21.15.40:0/2786366614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e401f7330 msgr2=0x7f4e40159f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e46800640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e401f7330 0x7f4e40159f60 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4e28007290 tx=0x7f4e28007750 comp rx=0 tx=0).stop 2024-03-24T00:31:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e46800640 1 -- 172.21.15.40:0/2786366614 shutdown_connections 2024-03-24T00:31:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e46800640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4e1c067690 0x7f4e1c069b50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e46800640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e4015a740 0x7f4e4015abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e46800640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4e401f7330 0x7f4e40159f60 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e46800640 1 --2- 172.21.15.40:0/2786366614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e24004d50 0x7f4e40158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e46800640 1 -- 172.21.15.40:0/2786366614 >> 172.21.15.40:0/2786366614 conn(0x7f4e400b3730 msgr2=0x7f4e400b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e46800640 1 -- 172.21.15.40:0/2786366614 shutdown_connections 2024-03-24T00:31:03.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.054+0000 7f4e46800640 1 -- 172.21.15.40:0/2786366614 wait complete. 2024-03-24T00:31:03.077 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:31:03.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:31:03.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.214+0000 7fd3e1a00640 1 Processor -- start 2024-03-24T00:31:03.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.214+0000 7fd3e1a00640 1 -- start start 2024-03-24T00:31:03.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3e1a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd3dc0cd470 0x7fd3dc168fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3e1a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3dc0b2870 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3e1a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3dc0b3190 con 0x7fd3dc0b2490 2024-03-24T00:31:03.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3e1a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3dc06b6d0 con 0x7fd3dc0cd470 2024-03-24T00:31:03.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3d3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd3dc0b2aa0 0x7fd3dc0d0540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55086/0 (socket says 172.21.15.40:55086) 2024-03-24T00:31:03.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3d3e00640 1 -- 172.21.15.40:0/2391625272 learned_addr learned my addr 172.21.15.40:0/2391625272 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:03.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3db400640 1 --2- 172.21.15.40:0/2391625272 >> v2:172.21.15.53:3300/0 conn(0x7fd3dc0cd470 0x7fd3dc168fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3da000640 1 -- 172.21.15.40:0/2391625272 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144180103 0 0) 0x7fd3dc0b2870 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3da000640 1 -- 172.21.15.40:0/2391625272 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3c0003530 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3da000640 1 -- 172.21.15.40:0/2391625272 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 788786159 0 0) 0x7fd3c0003530 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3db400640 1 -- 172.21.15.40:0/2391625272 >> v1:172.21.15.53:6789/0 conn(0x7fd3dc0b2490 legacy=0x7fd3dc0cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3db400640 1 -- 172.21.15.40:0/2391625272 >> v1:172.21.15.40:6789/0 conn(0x7fd3dc0b2aa0 legacy=0x7fd3dc0d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.218+0000 7fd3db400640 1 -- 172.21.15.40:0/2391625272 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc0b3470 con 0x7fd3dc0cd470 2024-03-24T00:31:03.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3db400640 1 --2- 172.21.15.40:0/2391625272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3dc0cd470 0x7fd3dc168fb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009d90 tx=0x7fd3d401c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24b06448de4ebf63 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:03.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3da000640 1 -- 172.21.15.40:0/2391625272 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d40b2070 con 0x7fd3dc0cd470 2024-03-24T00:31:03.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3da000640 1 -- 172.21.15.40:0/2391625272 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d40a55e0 con 0x7fd3dc0cd470 2024-03-24T00:31:03.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3da000640 1 -- 172.21.15.40:0/2391625272 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4001680 con 0x7fd3dc0cd470 2024-03-24T00:31:03.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 -- 172.21.15.40:0/2391625272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3dc0cd470 msgr2=0x7fd3dc168fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 --2- 172.21.15.40:0/2391625272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3dc0cd470 0x7fd3dc168fb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009d90 tx=0x7fd3d401c890 comp rx=0 tx=0).stop 2024-03-24T00:31:03.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 -- 172.21.15.40:0/2391625272 shutdown_connections 2024-03-24T00:31:03.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 --2- 172.21.15.40:0/2391625272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3dc0cd470 0x7fd3dc168fb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 -- 172.21.15.40:0/2391625272 >> 172.21.15.40:0/2391625272 conn(0x7fd3dc0d1d10 msgr2=0x7fd3dc0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:03.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 -- 172.21.15.40:0/2391625272 shutdown_connections 2024-03-24T00:31:03.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 -- 172.21.15.40:0/2391625272 wait complete. 2024-03-24T00:31:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 Processor -- start 2024-03-24T00:31:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 -- start start 2024-03-24T00:31:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3dc0b2490 0x7fd3dc1560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0b2aa0 0x7fd3dc15ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3dc0cd470 0x7fd3dc15efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc16c0a0 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc16bf20 con 0x7fd3dc0cd470 2024-03-24T00:31:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.222+0000 7fd3e1a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3dc16c220 con 0x7fd3dc0b2490 2024-03-24T00:31:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0b2aa0 0x7fd3dc15ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0b2aa0 0x7fd3dc15ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57098/0 (socket says 172.21.15.40:57098) 2024-03-24T00:31:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d3e00640 1 -- 172.21.15.40:0/3523740361 learned_addr learned my addr 172.21.15.40:0/3523740361 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3daa00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3dc0b2490 0x7fd3dc1560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3db400640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3dc0cd470 0x7fd3dc15efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d3e00640 1 -- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3dc0b2490 msgr2=0x7fd3dc1560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d3e00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3dc0b2490 0x7fd3dc1560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d3e00640 1 -- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3dc0cd470 msgr2=0x7fd3dc15efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d3e00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3dc0cd470 0x7fd3dc15efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d3e00640 1 -- 172.21.15.40:0/3523740361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc15f8c0 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d3e00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0b2aa0 0x7fd3dc15ea70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400c700 tx=0x7fd3c400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d2a00640 1 -- 172.21.15.40:0/3523740361 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4005ae0 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d2a00640 1 -- 172.21.15.40:0/3523740361 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c4005c80 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3e1a00640 1 -- 172.21.15.40:0/3523740361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3dc15a370 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d2a00640 1 -- 172.21.15.40:0/3523740361 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c4007290 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3e1a00640 1 -- 172.21.15.40:0/3523740361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3dc0aec00 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3e1a00640 1 -- 172.21.15.40:0/3523740361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3dc068170 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.226+0000 7fd3d2a00640 1 -- 172.21.15.40:0/3523740361 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd3c4007430 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.230+0000 7fd3d2a00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd3b8067660 0x7fd3b8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.230+0000 7fd3d2a00640 1 -- 172.21.15.40:0/3523740361 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3c40856b0 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.230+0000 7fd3daa00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd3b8067660 0x7fd3b8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.230+0000 7fd3daa00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd3b8067660 0x7fd3b8069b20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0c8d00 tx=0x7fd3cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:03.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.234+0000 7fd3d2a00640 1 -- 172.21.15.40:0/3523740361 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd3dc15f8c0 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.442+0000 7fd3e1a00640 1 -- 172.21.15.40:0/3523740361 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3dc156600 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.442+0000 7fd3d2a00640 1 -- 172.21.15.40:0/3523740361 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4965 (secure 0 0 0) 0x7fd3c4019300 con 0x7fd3dc0b2aa0 2024-03-24T00:31:03.447 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:03.447 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/501744669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":501744669},{"type":"v1","addr":"172.21.15.40:6833","nonce":501744669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/281354690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":281354690},{"type":"v1","addr":"172.21.15.53:6836","nonce":281354690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3092758821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3092758821},{"type":"v1","addr":"172.21.15.53:6837","nonce":3092758821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:30:59.427652+0000","modified":"2024-03-24T00:31:02.444298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4139303536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":4139303536},{"type":"v1","addr":"172.21.15.40:6835","nonce":4139303536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-24T00:31:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 8 2024-03-24T00:31:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.446+0000 7fd3e1a00640 1 -- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd3b8067660 msgr2=0x7fd3b8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.446+0000 7fd3e1a00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd3b8067660 0x7fd3b8069b20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc0c8d00 tx=0x7fd3cc009040 comp rx=0 tx=0).stop 2024-03-24T00:31:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.446+0000 7fd3e1a00640 1 -- 172.21.15.40:0/3523740361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0b2aa0 msgr2=0x7fd3dc15ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.446+0000 7fd3e1a00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0b2aa0 0x7fd3dc15ea70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400c700 tx=0x7fd3c400cbc0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.446+0000 7fd3e1a00640 1 -- 172.21.15.40:0/3523740361 shutdown_connections 2024-03-24T00:31:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.446+0000 7fd3e1a00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd3b8067660 0x7fd3b8069b20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.446+0000 7fd3e1a00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3dc0cd470 0x7fd3dc15efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.446+0000 7fd3e1a00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3dc0b2aa0 0x7fd3dc15ea70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.446+0000 7fd3e1a00640 1 --2- 172.21.15.40:0/3523740361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3dc0b2490 0x7fd3dc1560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.446+0000 7fd3e1a00640 1 -- 172.21.15.40:0/3523740361 >> 172.21.15.40:0/3523740361 conn(0x7fd3dc0d1d10 msgr2=0x7fd3dc0cb5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.446+0000 7fd3e1a00640 1 -- 172.21.15.40:0/3523740361 shutdown_connections 2024-03-24T00:31:03.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.446+0000 7fd3e1a00640 1 -- 172.21.15.40:0/3523740361 wait complete. 2024-03-24T00:31:03.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:03.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.606+0000 7f770ea00640 1 Processor -- start 2024-03-24T00:31:03.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.606+0000 7f770ea00640 1 -- start start 2024-03-24T00:31:03.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.606+0000 7f770ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f77080b2490 0x7f77080cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.606+0000 7f770ea00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080b2870 con 0x7f77080b2aa0 2024-03-24T00:31:03.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.606+0000 7f770ea00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77080b3190 con 0x7f77080d0a80 2024-03-24T00:31:03.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.606+0000 7f770ea00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f770806b6d0 con 0x7f77080b2490 2024-03-24T00:31:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.606+0000 7f76ffe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f77080b2aa0 0x7f77080d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55096/0 (socket says 172.21.15.40:55096) 2024-03-24T00:31:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.606+0000 7f76ffe00640 1 -- 172.21.15.40:0/592425468 learned_addr learned my addr 172.21.15.40:0/592425468 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.606+0000 7f7707e00640 1 --2- 172.21.15.40:0/592425468 >> v2:172.21.15.53:3300/0 conn(0x7f77080b2490 0x7f77080cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.606+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520604570 0 0) 0x7f77080b2870 con 0x7f77080b2aa0 2024-03-24T00:31:03.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.606+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76ec003530 con 0x7f77080b2aa0 2024-03-24T00:31:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 654641503 0 0) 0x7f76ec003530 con 0x7f77080b2aa0 2024-03-24T00:31:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 >> v2:172.21.15.53:3300/0 conn(0x7f77080b2490 msgr2=0x7f77080cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 --2- 172.21.15.40:0/592425468 >> v2:172.21.15.53:3300/0 conn(0x7f77080b2490 0x7f77080cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 >> v1:172.21.15.53:6789/0 conn(0x7f77080d0a80 legacy=0x7f77080cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080b3470 con 0x7f77080b2aa0 2024-03-24T00:31:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f76f0001680 con 0x7f77080b2aa0 2024-03-24T00:31:03.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707e00640 1 --2- 172.21.15.40:0/592425468 >> v2:172.21.15.53:3300/0 conn(0x7f77080b2490 0x7f77080cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 >> v1:172.21.15.40:6789/0 conn(0x7f77080b2aa0 legacy=0x7f77080d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 --2- 172.21.15.40:0/592425468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76ec0039c0 0x7f76ec003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 --2- 172.21.15.40:0/592425468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76ec004810 0x7f76ec004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 --2- 172.21.15.40:0/592425468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76ec0051d0 0x7f76ec00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770806b6d0 con 0x7f76ec0039c0 2024-03-24T00:31:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec00c070 con 0x7f76ec0051d0 2024-03-24T00:31:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec00c270 con 0x7f76ec004810 2024-03-24T00:31:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707e00640 1 --2- 172.21.15.40:0/592425468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76ec0039c0 0x7f76ec003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f76ffe00640 1 --2- 172.21.15.40:0/592425468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76ec004810 0x7f76ec004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f770ca00640 1 --2- 172.21.15.40:0/592425468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76ec0051d0 0x7f76ec00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707e00640 1 -- 172.21.15.40:0/592425468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76ec004810 msgr2=0x7f76ec004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707e00640 1 --2- 172.21.15.40:0/592425468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76ec004810 0x7f76ec004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707e00640 1 -- 172.21.15.40:0/592425468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76ec0051d0 msgr2=0x7f76ec00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707e00640 1 --2- 172.21.15.40:0/592425468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76ec0051d0 0x7f76ec00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707e00640 1 -- 172.21.15.40:0/592425468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec003530 con 0x7f76ec0039c0 2024-03-24T00:31:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f770ca00640 1 -- 172.21.15.40:0/592425468 reap_dead start 2024-03-24T00:31:03.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f76ffe00640 1 --2- 172.21.15.40:0/592425468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76ec004810 0x7f76ec004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:03.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707e00640 1 --2- 172.21.15.40:0/592425468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76ec0039c0 0x7f76ec003de0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f76f8009cb0 tx=0x7f76f801cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca19ff1bb93e0a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:03.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f80b2070 con 0x7f76ec0039c0 2024-03-24T00:31:03.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f80a5850 con 0x7f76ec0039c0 2024-03-24T00:31:03.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.610+0000 7f7707400640 1 -- 172.21.15.40:0/592425468 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f8009880 con 0x7f76ec0039c0 2024-03-24T00:31:03.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.614+0000 7f770ea00640 1 -- 172.21.15.40:0/592425468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76ec0039c0 msgr2=0x7f76ec003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.614+0000 7f770ea00640 1 --2- 172.21.15.40:0/592425468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76ec0039c0 0x7f76ec003de0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f76f8009cb0 tx=0x7f76f801cc10 comp rx=0 tx=0).stop 2024-03-24T00:31:03.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.614+0000 7f770ea00640 1 -- 172.21.15.40:0/592425468 shutdown_connections 2024-03-24T00:31:03.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.614+0000 7f770ea00640 1 --2- 172.21.15.40:0/592425468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76ec0039c0 0x7f76ec003de0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.614+0000 7f770ea00640 1 -- 172.21.15.40:0/592425468 >> 172.21.15.40:0/592425468 conn(0x7f77080d1d10 msgr2=0x7f77080d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:03.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.614+0000 7f770ea00640 1 -- 172.21.15.40:0/592425468 shutdown_connections 2024-03-24T00:31:03.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.614+0000 7f770ea00640 1 -- 172.21.15.40:0/592425468 wait complete. 2024-03-24T00:31:03.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.614+0000 7f770ea00640 1 Processor -- start 2024-03-24T00:31:03.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.614+0000 7f770ea00640 1 -- start start 2024-03-24T00:31:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76ec0039c0 0x7f770815f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f770814bb20 0x7f770814bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f770815b1d0 0x7f770815b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ea00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77080ce590 con 0x7f770815b1d0 2024-03-24T00:31:03.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ea00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77080d0800 con 0x7f76ec0039c0 2024-03-24T00:31:03.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ea00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77080ce710 con 0x7f770814bb20 2024-03-24T00:31:03.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f770815b1d0 0x7f770815b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f770815b1d0 0x7f770815b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57114/0 (socket says 172.21.15.40:57114) 2024-03-24T00:31:03.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ca00640 1 -- 172.21.15.40:0/243680787 learned_addr learned my addr 172.21.15.40:0/243680787 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:03.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f7707e00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76ec0039c0 0x7f770815f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f76ffe00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f770814bb20 0x7f770814bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ca00640 1 -- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f770814bb20 msgr2=0x7f770814bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ca00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f770814bb20 0x7f770814bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ca00640 1 -- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76ec0039c0 msgr2=0x7f770815f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ca00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76ec0039c0 0x7f770815f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ca00640 1 -- 172.21.15.40:0/243680787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770815bf40 con 0x7f770815b1d0 2024-03-24T00:31:03.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f7707e00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76ec0039c0 0x7f770815f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:03.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ca00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f770815b1d0 0x7f770815b610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f77080d16b0 tx=0x7f770000e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:03.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f7704c00640 1 -- 172.21.15.40:0/243680787 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700019070 con 0x7f770815b1d0 2024-03-24T00:31:03.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f7704c00640 1 -- 172.21.15.40:0/243680787 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7700006d70 con 0x7f770815b1d0 2024-03-24T00:31:03.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ea00640 1 -- 172.21.15.40:0/243680787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f770815c220 con 0x7f770815b1d0 2024-03-24T00:31:03.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f7704c00640 1 -- 172.21.15.40:0/243680787 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7700003ba0 con 0x7f770815b1d0 2024-03-24T00:31:03.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ea00640 1 -- 172.21.15.40:0/243680787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f770815c7e0 con 0x7f770815b1d0 2024-03-24T00:31:03.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.618+0000 7f770ea00640 1 -- 172.21.15.40:0/243680787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7708068170 con 0x7f770815b1d0 2024-03-24T00:31:03.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.622+0000 7f7704c00640 1 -- 172.21.15.40:0/243680787 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f770001c070 con 0x7f770815b1d0 2024-03-24T00:31:03.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.622+0000 7f7704c00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f76e4067690 0x7f76e4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.622+0000 7f7704c00640 1 -- 172.21.15.40:0/243680787 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7700084d90 con 0x7f770815b1d0 2024-03-24T00:31:03.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.622+0000 7f7707e00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f76e4067690 0x7f76e4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.622+0000 7f7707e00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f76e4067690 0x7f76e4069b50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f76f80af000 tx=0x7f76f80a7040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:03.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.630+0000 7f7704c00640 1 -- 172.21.15.40:0/243680787 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f77000553b0 con 0x7f770815b1d0 2024-03-24T00:31:03.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.798+0000 7f770ea00640 1 -- 172.21.15.40:0/243680787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f770806c090 con 0x7f770815b1d0 2024-03-24T00:31:03.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f7704c00640 1 -- 172.21.15.40:0/243680787 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7f7700059450 con 0x7f770815b1d0 2024-03-24T00:31:03.803 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:03.803 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f770ea00640 1 -- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f76e4067690 msgr2=0x7f76e4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f770ea00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f76e4067690 0x7f76e4069b50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f76f80af000 tx=0x7f76f80a7040 comp rx=0 tx=0).stop 2024-03-24T00:31:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f770ea00640 1 -- 172.21.15.40:0/243680787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f770815b1d0 msgr2=0x7f770815b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f770ea00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f770815b1d0 0x7f770815b610 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f77080d16b0 tx=0x7f770000e9e0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f770ea00640 1 -- 172.21.15.40:0/243680787 shutdown_connections 2024-03-24T00:31:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f770ea00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f76e4067690 0x7f76e4069b50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f770ea00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f770815b1d0 0x7f770815b610 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f770ea00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f770814bb20 0x7f770814bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f770ea00640 1 --2- 172.21.15.40:0/243680787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76ec0039c0 0x7f770815f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f770ea00640 1 -- 172.21.15.40:0/243680787 >> 172.21.15.40:0/243680787 conn(0x7f77080d1d10 msgr2=0x7f770806fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:03.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f770ea00640 1 -- 172.21.15.40:0/243680787 shutdown_connections 2024-03-24T00:31:03.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.802+0000 7f770ea00640 1 -- 172.21.15.40:0/243680787 wait complete. 2024-03-24T00:31:03.829 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:31:03.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:31:03.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.966+0000 7fb10da00640 1 Processor -- start 2024-03-24T00:31:03.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.966+0000 7fb10da00640 1 -- start start 2024-03-24T00:31:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.966+0000 7fb10da00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb1080cd470 0x7fb108168fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.966+0000 7fb10da00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1080b2870 con 0x7fb1080b2490 2024-03-24T00:31:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.966+0000 7fb10da00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1080b3190 con 0x7fb1080b2aa0 2024-03-24T00:31:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.966+0000 7fb10da00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10806b6d0 con 0x7fb1080cd470 2024-03-24T00:31:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.970+0000 7fb106a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb1080b2490 0x7fb1080cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55104/0 (socket says 172.21.15.40:55104) 2024-03-24T00:31:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.970+0000 7fb106a00640 1 -- 172.21.15.40:0/3982000007 learned_addr learned my addr 172.21.15.40:0/3982000007 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:03.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.970+0000 7fb107400640 1 --2- 172.21.15.40:0/3982000007 >> v2:172.21.15.53:3300/0 conn(0x7fb1080cd470 0x7fb108168fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.970+0000 7fb107400640 1 -- 172.21.15.40:0/3982000007 >> v1:172.21.15.53:6789/0 conn(0x7fb1080b2aa0 legacy=0x7fb1080d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.970+0000 7fb107400640 1 -- 172.21.15.40:0/3982000007 >> v1:172.21.15.40:6789/0 conn(0x7fb1080b2490 legacy=0x7fb1080cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.970+0000 7fb107400640 1 -- 172.21.15.40:0/3982000007 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1080b3470 con 0x7fb1080cd470 2024-03-24T00:31:03.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.970+0000 7fb106000640 1 -- 172.21.15.40:0/3982000007 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706280902 0 0) 0x7fb1080b2870 con 0x7fb1080b2490 2024-03-24T00:31:03.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.970+0000 7fb107400640 1 --2- 172.21.15.40:0/3982000007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1080cd470 0x7fb108168fb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb100009d90 tx=0x7fb10001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aafeb008c1b5f88b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:03.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb106000640 1 -- 172.21.15.40:0/3982000007 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1000b2070 con 0x7fb1080cd470 2024-03-24T00:31:03.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb106000640 1 -- 172.21.15.40:0/3982000007 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1000a5a50 con 0x7fb1080cd470 2024-03-24T00:31:03.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb106000640 1 -- 172.21.15.40:0/3982000007 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1000a5e20 con 0x7fb1080cd470 2024-03-24T00:31:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 -- 172.21.15.40:0/3982000007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1080cd470 msgr2=0x7fb108168fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 --2- 172.21.15.40:0/3982000007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1080cd470 0x7fb108168fb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb100009d90 tx=0x7fb10001c970 comp rx=0 tx=0).stop 2024-03-24T00:31:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 -- 172.21.15.40:0/3982000007 shutdown_connections 2024-03-24T00:31:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 --2- 172.21.15.40:0/3982000007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1080cd470 0x7fb108168fb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 -- 172.21.15.40:0/3982000007 >> 172.21.15.40:0/3982000007 conn(0x7fb1080d1d10 msgr2=0x7fb1080d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 -- 172.21.15.40:0/3982000007 shutdown_connections 2024-03-24T00:31:03.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 -- 172.21.15.40:0/3982000007 wait complete. 2024-03-24T00:31:03.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 Processor -- start 2024-03-24T00:31:03.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 -- start start 2024-03-24T00:31:03.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1080b2490 0x7fb1081546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1080b2aa0 0x7fb108153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1080cd470 0x7fb10815e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10816c0a0 con 0x7fb1080b2490 2024-03-24T00:31:03.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10816bf20 con 0x7fb1080b2aa0 2024-03-24T00:31:03.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb10da00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10816c220 con 0x7fb1080cd470 2024-03-24T00:31:03.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb106a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1080b2490 0x7fb1081546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb106a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1080b2490 0x7fb1081546d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57130/0 (socket says 172.21.15.40:57130) 2024-03-24T00:31:03.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.974+0000 7fb106a00640 1 -- 172.21.15.40:0/3776669811 learned_addr learned my addr 172.21.15.40:0/3776669811 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:03.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb107400640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1080cd470 0x7fb10815e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb0ffe00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1080b2aa0 0x7fb108153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb106a00640 1 -- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1080cd470 msgr2=0x7fb10815e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb106a00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1080cd470 0x7fb10815e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb106a00640 1 -- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1080b2aa0 msgr2=0x7fb108153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:03.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb106a00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1080b2aa0 0x7fb108153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:03.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb106a00640 1 -- 172.21.15.40:0/3776669811 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10815ec10 con 0x7fb1080b2490 2024-03-24T00:31:03.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb106a00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1080b2490 0x7fb1081546d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800c440 tx=0x7fb0f800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:03.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb0fea00640 1 -- 172.21.15.40:0/3776669811 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8018070 con 0x7fb1080b2490 2024-03-24T00:31:03.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb0fea00640 1 -- 172.21.15.40:0/3776669811 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f80074e0 con 0x7fb1080b2490 2024-03-24T00:31:03.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb10da00640 1 -- 172.21.15.40:0/3776669811 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb10815eef0 con 0x7fb1080b2490 2024-03-24T00:31:03.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb0fea00640 1 -- 172.21.15.40:0/3776669811 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8004a10 con 0x7fb1080b2490 2024-03-24T00:31:03.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb10da00640 1 -- 172.21.15.40:0/3776669811 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10815f4b0 con 0x7fb1080b2490 2024-03-24T00:31:03.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb10da00640 1 -- 172.21.15.40:0/3776669811 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb108068170 con 0x7fb1080b2490 2024-03-24T00:31:03.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb0fea00640 1 -- 172.21.15.40:0/3776669811 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb0f801b070 con 0x7fb1080b2490 2024-03-24T00:31:03.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb0fea00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0e4067660 0x7fb0e4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:03.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.978+0000 7fb0fea00640 1 -- 172.21.15.40:0/3776669811 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0f8084b50 con 0x7fb1080b2490 2024-03-24T00:31:03.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.982+0000 7fb0ffe00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0e4067660 0x7fb0e4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:03.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.982+0000 7fb0ffe00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0e4067660 0x7fb0e4069b20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb1080cdd30 tx=0x7fb0f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:03.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:03.986+0000 7fb0fea00640 1 -- 172.21.15.40:0/3776669811 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb0f80557c0 con 0x7fb1080b2490 2024-03-24T00:31:04.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.146+0000 7fb10da00640 1 -- 172.21.15.40:0/3776669811 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb108069140 con 0x7fb0e4067660 2024-03-24T00:31:04.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.166+0000 7fb0fea00640 1 -- 172.21.15.40:0/3776669811 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240055 (secure 0 0 0) 0x7fb108069140 con 0x7fb0e4067660 2024-03-24T00:31:04.171 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:04.173 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:31:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.170+0000 7fb10da00640 1 -- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0e4067660 msgr2=0x7fb0e4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.170+0000 7fb10da00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0e4067660 0x7fb0e4069b20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb1080cdd30 tx=0x7fb0f0009040 comp rx=0 tx=0).stop 2024-03-24T00:31:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.170+0000 7fb10da00640 1 -- 172.21.15.40:0/3776669811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1080b2490 msgr2=0x7fb1081546d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.170+0000 7fb10da00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1080b2490 0x7fb1081546d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb0f800c440 tx=0x7fb0f800c900 comp rx=0 tx=0).stop 2024-03-24T00:31:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.170+0000 7fb10da00640 1 -- 172.21.15.40:0/3776669811 shutdown_connections 2024-03-24T00:31:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.170+0000 7fb10da00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0e4067660 0x7fb0e4069b20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.170+0000 7fb10da00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1080cd470 0x7fb10815e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.170+0000 7fb10da00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1080b2aa0 0x7fb108153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.170+0000 7fb10da00640 1 --2- 172.21.15.40:0/3776669811 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb1080b2490 0x7fb1081546d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.170+0000 7fb10da00640 1 -- 172.21.15.40:0/3776669811 >> 172.21.15.40:0/3776669811 conn(0x7fb1080d1d10 msgr2=0x7fb1080cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.170+0000 7fb10da00640 1 -- 172.21.15.40:0/3776669811 shutdown_connections 2024-03-24T00:31:04.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.170+0000 7fb10da00640 1 -- 172.21.15.40:0/3776669811 wait complete. 2024-03-24T00:31:04.191 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-24T00:31:03.197833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235884,"kb_used_data":2448,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738836,"statfs":{"total":773094113280,"available":772852568064,"internally_reserved":0,"allocated":2506752,"data_stored":1581774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.173272"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028717+0000","last_change":"2024-03-24T00:30:58.028717+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028717+0000","last_undegraded":"2024-03-24T00:30:58.028717+0000","last_fullsized":"2024-03-24T00:30:58.028717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.032033+0000","last_change":"2024-03-24T00:30:58.032033+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.032033+0000","last_undegraded":"2024-03-24T00:30:58.032033+0000","last_fullsized":"2024-03-24T00:30:58.032033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029157+0000","last_change":"2024-03-24T00:30:58.029157+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029157+0000","last_undegraded":"2024-03-24T00:30:58.029157+0000","last_fullsized":"2024-03-24T00:30:58.029157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029986+0000","last_change":"2024-03-24T00:30:58.029986+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029986+0000","last_undegraded":"2024-03-24T00:30:58.029986+0000","last_fullsized":"2024-03-24T00:30:58.029986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.034511+0000","last_change":"2024-03-24T00:30:58.034511+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.034511+0000","last_undegraded":"2024-03-24T00:30:58.034511+0000","last_fullsized":"2024-03-24T00:30:58.034511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029515+0000","last_change":"2024-03-24T00:30:58.029515+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029515+0000","last_undegraded":"2024-03-24T00:30:58.029515+0000","last_fullsized":"2024-03-24T00:30:58.029515+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028833+0000","last_change":"2024-03-24T00:30:58.028833+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028833+0000","last_undegraded":"2024-03-24T00:30:58.028833+0000","last_fullsized":"2024-03-24T00:30:58.028833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033549+0000","last_change":"2024-03-24T00:30:58.033549+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033549+0000","last_undegraded":"2024-03-24T00:30:58.033549+0000","last_fullsized":"2024-03-24T00:30:58.033549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.022064+0000","last_change":"2024-03-24T00:30:58.022064+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.022064+0000","last_undegraded":"2024-03-24T00:30:58.022064+0000","last_fullsized":"2024-03-24T00:30:58.022064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.030365+0000","last_change":"2024-03-24T00:30:58.030365+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.030365+0000","last_undegraded":"2024-03-24T00:30:58.030365+0000","last_fullsized":"2024-03-24T00:30:58.030365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.032033+0000","last_change":"2024-03-24T00:30:58.032033+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.032033+0000","last_undegraded":"2024-03-24T00:30:58.032033+0000","last_fullsized":"2024-03-24T00:30:58.032033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.032696+0000","last_change":"2024-03-24T00:30:58.032696+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.032696+0000","last_undegraded":"2024-03-24T00:30:58.032696+0000","last_fullsized":"2024-03-24T00:30:58.032696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.020662+0000","last_change":"2024-03-24T00:30:58.020662+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.020662+0000","last_undegraded":"2024-03-24T00:30:58.020662+0000","last_fullsized":"2024-03-24T00:30:58.020662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033693+0000","last_change":"2024-03-24T00:30:58.033693+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033693+0000","last_undegraded":"2024-03-24T00:30:58.033693+0000","last_fullsized":"2024-03-24T00:30:58.033693+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028209+0000","last_change":"2024-03-24T00:30:58.028209+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028209+0000","last_undegraded":"2024-03-24T00:30:58.028209+0000","last_fullsized":"2024-03-24T00:30:58.028209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033002+0000","last_change":"2024-03-24T00:30:58.033002+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033002+0000","last_undegraded":"2024-03-24T00:30:58.033002+0000","last_fullsized":"2024-03-24T00:30:58.033002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-24T00:30:58.023125+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:30:58.023125+0000","last_peered":"2024-03-24T00:30:58.023125+0000","last_clean":"2024-03-24T00:30:58.023125+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:30:58.023125+0000","last_undegraded":"2024-03-24T00:30:58.023125+0000","last_fullsized":"2024-03-24T00:30:58.023125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029224+0000","last_change":"2024-03-24T00:30:58.029224+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029224+0000","last_undegraded":"2024-03-24T00:30:58.029224+0000","last_fullsized":"2024-03-24T00:30:58.029224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.019558+0000","last_change":"2024-03-24T00:30:58.019558+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.019558+0000","last_undegraded":"2024-03-24T00:30:58.019558+0000","last_fullsized":"2024-03-24T00:30:58.019558+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033479+0000","last_change":"2024-03-24T00:30:58.033479+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033479+0000","last_undegraded":"2024-03-24T00:30:58.033479+0000","last_fullsized":"2024-03-24T00:30:58.033479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029244+0000","last_change":"2024-03-24T00:30:58.029244+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029244+0000","last_undegraded":"2024-03-24T00:30:58.029244+0000","last_fullsized":"2024-03-24T00:30:58.029244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028482+0000","last_change":"2024-03-24T00:30:58.028482+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028482+0000","last_undegraded":"2024-03-24T00:30:58.028482+0000","last_fullsized":"2024-03-24T00:30:58.028482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.019738+0000","last_change":"2024-03-24T00:30:58.019738+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.019738+0000","last_undegraded":"2024-03-24T00:30:58.019738+0000","last_fullsized":"2024-03-24T00:30:58.019738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028257+0000","last_change":"2024-03-24T00:30:58.028257+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028257+0000","last_undegraded":"2024-03-24T00:30:58.028257+0000","last_fullsized":"2024-03-24T00:30:58.028257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033671+0000","last_change":"2024-03-24T00:30:58.033671+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033671+0000","last_undegraded":"2024-03-24T00:30:58.033671+0000","last_fullsized":"2024-03-24T00:30:58.033671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.020377+0000","last_change":"2024-03-24T00:30:58.020377+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.020377+0000","last_undegraded":"2024-03-24T00:30:58.020377+0000","last_fullsized":"2024-03-24T00:30:58.020377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.032623+0000","last_change":"2024-03-24T00:30:58.032623+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.032623+0000","last_undegraded":"2024-03-24T00:30:58.032623+0000","last_fullsized":"2024-03-24T00:30:58.032623+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029744+0000","last_change":"2024-03-24T00:30:58.029744+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029744+0000","last_undegraded":"2024-03-24T00:30:58.029744+0000","last_fullsized":"2024-03-24T00:30:58.029744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.019863+0000","last_change":"2024-03-24T00:30:58.019863+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.019863+0000","last_undegraded":"2024-03-24T00:30:58.019863+0000","last_fullsized":"2024-03-24T00:30:58.019863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.019696+0000","last_change":"2024-03-24T00:30:58.019696+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.019696+0000","last_undegraded":"2024-03-24T00:30:58.019696+0000","last_fullsized":"2024-03-24T00:30:58.019696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.021452+0000","last_change":"2024-03-24T00:30:58.021452+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.021452+0000","last_undegraded":"2024-03-24T00:30:58.021452+0000","last_fullsized":"2024-03-24T00:30:58.021452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028166+0000","last_change":"2024-03-24T00:30:58.028166+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028166+0000","last_undegraded":"2024-03-24T00:30:58.028166+0000","last_fullsized":"2024-03-24T00:30:58.028166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.022059+0000","last_change":"2024-03-24T00:30:58.022059+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.022059+0000","last_undegraded":"2024-03-24T00:30:58.022059+0000","last_fullsized":"2024-03-24T00:30:58.022059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31924,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339916,"statfs":{"total":96636764160,"available":96604073984,"internally_reserved":0,"allocated":761856,"data_stored":641952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28388,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343452,"statfs":{"total":96636764160,"available":96607694848,"internally_reserved":0,"allocated":761856,"data_stored":641952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:31:04.195 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:31:04.214 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-03-24T00:31:04.210+0000 7f6bcb800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:31:04.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.350+0000 7f36c5c00640 1 Processor -- start 2024-03-24T00:31:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.350+0000 7f36c5c00640 1 -- start start 2024-03-24T00:31:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.350+0000 7f36c5c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f36c015f600 0x7f36c00b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.350+0000 7f36c5c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36c00af170 0x7f36c0165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.350+0000 7f36c5c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c00b1e50 con 0x7f36c00d2830 2024-03-24T00:31:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.350+0000 7f36c5c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c00b3630 con 0x7f36c015f600 2024-03-24T00:31:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.350+0000 7f36c5c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c00d2ca0 con 0x7f36c00af170 2024-03-24T00:31:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.354+0000 7f36bf400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36c00af170 0x7f36c0165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.354+0000 7f36bea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f36c015f600 0x7f36c00b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.354+0000 7f36bf400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36c00af170 0x7f36c0165e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:47156/0 (socket says 172.21.15.40:47156) 2024-03-24T00:31:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.354+0000 7f36bf400640 1 -- 172.21.15.40:0/1575296418 learned_addr learned my addr 172.21.15.40:0/1575296418 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.354+0000 7f36bf400640 1 -- 172.21.15.40:0/1575296418 >> v2:172.21.15.53:3300/0 conn(0x7f36c015f600 msgr2=0x7f36c00b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.354+0000 7f36bf400640 1 --2- 172.21.15.40:0/1575296418 >> v2:172.21.15.53:3300/0 conn(0x7f36c015f600 0x7f36c00b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.354+0000 7f36bf400640 1 -- 172.21.15.40:0/1575296418 >> v1:172.21.15.53:6789/0 conn(0x7f36c00d2830 legacy=0x7f36c00aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.354+0000 7f36bf400640 1 -- 172.21.15.40:0/1575296418 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c00b20f0 con 0x7f36c00af170 2024-03-24T00:31:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.354+0000 7f36bf400640 1 --2- 172.21.15.40:0/1575296418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36c00af170 0x7f36c0165e40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f36b8009e50 tx=0x7f36b801f210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c8ad5816b64b384 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.354+0000 7f36be000640 1 -- 172.21.15.40:0/1575296418 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b80b4070 con 0x7f36c00af170 2024-03-24T00:31:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.354+0000 7f36be000640 1 -- 172.21.15.40:0/1575296418 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b80af400 con 0x7f36c00af170 2024-03-24T00:31:04.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.354+0000 7f36be000640 1 -- 172.21.15.40:0/1575296418 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b80af7d0 con 0x7f36c00af170 2024-03-24T00:31:04.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 -- 172.21.15.40:0/1575296418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36c00af170 msgr2=0x7f36c0165e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 --2- 172.21.15.40:0/1575296418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36c00af170 0x7f36c0165e40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f36b8009e50 tx=0x7f36b801f210 comp rx=0 tx=0).stop 2024-03-24T00:31:04.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 -- 172.21.15.40:0/1575296418 shutdown_connections 2024-03-24T00:31:04.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 --2- 172.21.15.40:0/1575296418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36c00af170 0x7f36c0165e40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 --2- 172.21.15.40:0/1575296418 >> v2:172.21.15.53:3300/0 conn(0x7f36c015f600 0x7f36c00b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 -- 172.21.15.40:0/1575296418 >> 172.21.15.40:0/1575296418 conn(0x7f36c00d1d10 msgr2=0x7f36c00d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:04.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 -- 172.21.15.40:0/1575296418 shutdown_connections 2024-03-24T00:31:04.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 -- 172.21.15.40:0/1575296418 wait complete. 2024-03-24T00:31:04.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 Processor -- start 2024-03-24T00:31:04.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 -- start start 2024-03-24T00:31:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36c00af170 0x7f36c015aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c00d2830 0x7f36c006bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c015f600 0x7f36c0150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c00b2fa0 con 0x7f36c015f600 2024-03-24T00:31:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c00b2740 con 0x7f36c00d2830 2024-03-24T00:31:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36c5c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c015f9e0 con 0x7f36c00af170 2024-03-24T00:31:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36bf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c015f600 0x7f36c0150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36bf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c015f600 0x7f36c0150ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57134/0 (socket says 172.21.15.40:57134) 2024-03-24T00:31:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36bf400640 1 -- 172.21.15.40:0/4122134644 learned_addr learned my addr 172.21.15.40:0/4122134644 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:04.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36bea00640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36c00af170 0x7f36c015aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36b7e00640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c00d2830 0x7f36c006bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36bf400640 1 -- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36c00af170 msgr2=0x7f36c015aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36bf400640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36c00af170 0x7f36c015aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36bf400640 1 -- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c00d2830 msgr2=0x7f36c006bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36bf400640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c00d2830 0x7f36c006bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36bf400640 1 -- 172.21.15.40:0/4122134644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c0158e60 con 0x7f36c015f600 2024-03-24T00:31:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.358+0000 7f36b7e00640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c00d2830 0x7f36c006bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:04.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.362+0000 7f36bf400640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c015f600 0x7f36c0150ea0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f36b801ce40 tx=0x7f36b801bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.362+0000 7f36b6a00640 1 -- 172.21.15.40:0/4122134644 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b80b4040 con 0x7f36c015f600 2024-03-24T00:31:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.362+0000 7f36b6a00640 1 -- 172.21.15.40:0/4122134644 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b80c0070 con 0x7f36c015f600 2024-03-24T00:31:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.362+0000 7f36b6a00640 1 -- 172.21.15.40:0/4122134644 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b8009510 con 0x7f36c015f600 2024-03-24T00:31:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.362+0000 7f36c5c00640 1 -- 172.21.15.40:0/4122134644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c0159140 con 0x7f36c015f600 2024-03-24T00:31:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.362+0000 7f36c5c00640 1 -- 172.21.15.40:0/4122134644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c0159700 con 0x7f36c015f600 2024-03-24T00:31:04.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.362+0000 7f36c5c00640 1 -- 172.21.15.40:0/4122134644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c0068170 con 0x7f36c015f600 2024-03-24T00:31:04.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.362+0000 7f36b6a00640 1 -- 172.21.15.40:0/4122134644 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f36b80a8d10 con 0x7f36c015f600 2024-03-24T00:31:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.366+0000 7f36b6a00640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f369c067690 0x7f369c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.366+0000 7f36b6a00640 1 -- 172.21.15.40:0/4122134644 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f36b81238a0 con 0x7f36c015f600 2024-03-24T00:31:04.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.366+0000 7f36bea00640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f369c067690 0x7f369c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.366+0000 7f36bea00640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f369c067690 0x7f369c069b50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009b60 tx=0x7f36b0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:04.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.370+0000 7f36b6a00640 1 -- 172.21.15.40:0/4122134644 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f36b8018d70 con 0x7f36c015f600 2024-03-24T00:31:04.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.526+0000 7f36c5c00640 1 -- 172.21.15.40:0/4122134644 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36c00d20f0 con 0x7f369c067690 2024-03-24T00:31:04.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.534+0000 7f36b6a00640 1 -- 172.21.15.40:0/4122134644 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240055 (secure 0 0 0) 0x7f36c00d20f0 con 0x7f369c067690 2024-03-24T00:31:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:04.541 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:31:04.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.538+0000 7f36c5c00640 1 -- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f369c067690 msgr2=0x7f369c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.538+0000 7f36c5c00640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f369c067690 0x7f369c069b50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009b60 tx=0x7f36b0009190 comp rx=0 tx=0).stop 2024-03-24T00:31:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.538+0000 7f36c5c00640 1 -- 172.21.15.40:0/4122134644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c015f600 msgr2=0x7f36c0150ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.538+0000 7f36c5c00640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c015f600 0x7f36c0150ea0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f36b801ce40 tx=0x7f36b801bfb0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.538+0000 7f36c5c00640 1 -- 172.21.15.40:0/4122134644 shutdown_connections 2024-03-24T00:31:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.538+0000 7f36c5c00640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f369c067690 0x7f369c069b50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.538+0000 7f36c5c00640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36c015f600 0x7f36c0150ea0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.538+0000 7f36c5c00640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c00d2830 0x7f36c006bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.538+0000 7f36c5c00640 1 --2- 172.21.15.40:0/4122134644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36c00af170 0x7f36c015aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.538+0000 7f36c5c00640 1 -- 172.21.15.40:0/4122134644 >> 172.21.15.40:0/4122134644 conn(0x7f36c00d1d10 msgr2=0x7f36c00ceae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.538+0000 7f36c5c00640 1 -- 172.21.15.40:0/4122134644 shutdown_connections 2024-03-24T00:31:04.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.538+0000 7f36c5c00640 1 -- 172.21.15.40:0/4122134644 wait complete. 2024-03-24T00:31:04.559 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-24T00:31:03.197833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235884,"kb_used_data":2448,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738836,"statfs":{"total":773094113280,"available":772852568064,"internally_reserved":0,"allocated":2506752,"data_stored":1581774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.173272"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028717+0000","last_change":"2024-03-24T00:30:58.028717+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028717+0000","last_undegraded":"2024-03-24T00:30:58.028717+0000","last_fullsized":"2024-03-24T00:30:58.028717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.032033+0000","last_change":"2024-03-24T00:30:58.032033+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.032033+0000","last_undegraded":"2024-03-24T00:30:58.032033+0000","last_fullsized":"2024-03-24T00:30:58.032033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029157+0000","last_change":"2024-03-24T00:30:58.029157+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029157+0000","last_undegraded":"2024-03-24T00:30:58.029157+0000","last_fullsized":"2024-03-24T00:30:58.029157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029986+0000","last_change":"2024-03-24T00:30:58.029986+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029986+0000","last_undegraded":"2024-03-24T00:30:58.029986+0000","last_fullsized":"2024-03-24T00:30:58.029986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.034511+0000","last_change":"2024-03-24T00:30:58.034511+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.034511+0000","last_undegraded":"2024-03-24T00:30:58.034511+0000","last_fullsized":"2024-03-24T00:30:58.034511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029515+0000","last_change":"2024-03-24T00:30:58.029515+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029515+0000","last_undegraded":"2024-03-24T00:30:58.029515+0000","last_fullsized":"2024-03-24T00:30:58.029515+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028833+0000","last_change":"2024-03-24T00:30:58.028833+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028833+0000","last_undegraded":"2024-03-24T00:30:58.028833+0000","last_fullsized":"2024-03-24T00:30:58.028833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033549+0000","last_change":"2024-03-24T00:30:58.033549+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033549+0000","last_undegraded":"2024-03-24T00:30:58.033549+0000","last_fullsized":"2024-03-24T00:30:58.033549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.022064+0000","last_change":"2024-03-24T00:30:58.022064+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.022064+0000","last_undegraded":"2024-03-24T00:30:58.022064+0000","last_fullsized":"2024-03-24T00:30:58.022064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.030365+0000","last_change":"2024-03-24T00:30:58.030365+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.030365+0000","last_undegraded":"2024-03-24T00:30:58.030365+0000","last_fullsized":"2024-03-24T00:30:58.030365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.032033+0000","last_change":"2024-03-24T00:30:58.032033+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.032033+0000","last_undegraded":"2024-03-24T00:30:58.032033+0000","last_fullsized":"2024-03-24T00:30:58.032033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.032696+0000","last_change":"2024-03-24T00:30:58.032696+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.032696+0000","last_undegraded":"2024-03-24T00:30:58.032696+0000","last_fullsized":"2024-03-24T00:30:58.032696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.020662+0000","last_change":"2024-03-24T00:30:58.020662+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.020662+0000","last_undegraded":"2024-03-24T00:30:58.020662+0000","last_fullsized":"2024-03-24T00:30:58.020662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033693+0000","last_change":"2024-03-24T00:30:58.033693+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033693+0000","last_undegraded":"2024-03-24T00:30:58.033693+0000","last_fullsized":"2024-03-24T00:30:58.033693+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028209+0000","last_change":"2024-03-24T00:30:58.028209+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028209+0000","last_undegraded":"2024-03-24T00:30:58.028209+0000","last_fullsized":"2024-03-24T00:30:58.028209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033002+0000","last_change":"2024-03-24T00:30:58.033002+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033002+0000","last_undegraded":"2024-03-24T00:30:58.033002+0000","last_fullsized":"2024-03-24T00:30:58.033002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-24T00:30:58.023125+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:30:58.023125+0000","last_peered":"2024-03-24T00:30:58.023125+0000","last_clean":"2024-03-24T00:30:58.023125+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:30:58.023125+0000","last_undegraded":"2024-03-24T00:30:58.023125+0000","last_fullsized":"2024-03-24T00:30:58.023125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029224+0000","last_change":"2024-03-24T00:30:58.029224+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029224+0000","last_undegraded":"2024-03-24T00:30:58.029224+0000","last_fullsized":"2024-03-24T00:30:58.029224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.019558+0000","last_change":"2024-03-24T00:30:58.019558+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.019558+0000","last_undegraded":"2024-03-24T00:30:58.019558+0000","last_fullsized":"2024-03-24T00:30:58.019558+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033479+0000","last_change":"2024-03-24T00:30:58.033479+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033479+0000","last_undegraded":"2024-03-24T00:30:58.033479+0000","last_fullsized":"2024-03-24T00:30:58.033479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029244+0000","last_change":"2024-03-24T00:30:58.029244+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029244+0000","last_undegraded":"2024-03-24T00:30:58.029244+0000","last_fullsized":"2024-03-24T00:30:58.029244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028482+0000","last_change":"2024-03-24T00:30:58.028482+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028482+0000","last_undegraded":"2024-03-24T00:30:58.028482+0000","last_fullsized":"2024-03-24T00:30:58.028482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.019738+0000","last_change":"2024-03-24T00:30:58.019738+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.019738+0000","last_undegraded":"2024-03-24T00:30:58.019738+0000","last_fullsized":"2024-03-24T00:30:58.019738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028257+0000","last_change":"2024-03-24T00:30:58.028257+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028257+0000","last_undegraded":"2024-03-24T00:30:58.028257+0000","last_fullsized":"2024-03-24T00:30:58.028257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033671+0000","last_change":"2024-03-24T00:30:58.033671+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033671+0000","last_undegraded":"2024-03-24T00:30:58.033671+0000","last_fullsized":"2024-03-24T00:30:58.033671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.020377+0000","last_change":"2024-03-24T00:30:58.020377+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.020377+0000","last_undegraded":"2024-03-24T00:30:58.020377+0000","last_fullsized":"2024-03-24T00:30:58.020377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.032623+0000","last_change":"2024-03-24T00:30:58.032623+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.032623+0000","last_undegraded":"2024-03-24T00:30:58.032623+0000","last_fullsized":"2024-03-24T00:30:58.032623+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029744+0000","last_change":"2024-03-24T00:30:58.029744+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029744+0000","last_undegraded":"2024-03-24T00:30:58.029744+0000","last_fullsized":"2024-03-24T00:30:58.029744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.019863+0000","last_change":"2024-03-24T00:30:58.019863+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.019863+0000","last_undegraded":"2024-03-24T00:30:58.019863+0000","last_fullsized":"2024-03-24T00:30:58.019863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.019696+0000","last_change":"2024-03-24T00:30:58.019696+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.019696+0000","last_undegraded":"2024-03-24T00:30:58.019696+0000","last_fullsized":"2024-03-24T00:30:58.019696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.021452+0000","last_change":"2024-03-24T00:30:58.021452+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.021452+0000","last_undegraded":"2024-03-24T00:30:58.021452+0000","last_fullsized":"2024-03-24T00:30:58.021452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028166+0000","last_change":"2024-03-24T00:30:58.028166+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028166+0000","last_undegraded":"2024-03-24T00:30:58.028166+0000","last_fullsized":"2024-03-24T00:30:58.028166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.022059+0000","last_change":"2024-03-24T00:30:58.022059+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.022059+0000","last_undegraded":"2024-03-24T00:30:58.022059+0000","last_fullsized":"2024-03-24T00:30:58.022059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31924,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339916,"statfs":{"total":96636764160,"available":96604073984,"internally_reserved":0,"allocated":761856,"data_stored":641952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28388,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343452,"statfs":{"total":96636764160,"available":96607694848,"internally_reserved":0,"allocated":761856,"data_stored":641952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:31:04.562 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:31:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.738+0000 7fa1f3200640 1 Processor -- start 2024-03-24T00:31:04.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.738+0000 7fa1f3200640 1 -- start start 2024-03-24T00:31:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1f3200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa1ec0b4b80 0x7fa1ec0b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1f3200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec0cd3d0 con 0x7fa1ec15e0b0 2024-03-24T00:31:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1f3200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ec0cd6b0 con 0x7fa1ec06bd10 2024-03-24T00:31:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1f3200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ec15e490 con 0x7fa1ec0b4b80 2024-03-24T00:31:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1ebe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa1ec15e0b0 0x7fa1ec06b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55120/0 (socket says 172.21.15.40:55120) 2024-03-24T00:31:04.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1ebe00640 1 -- 172.21.15.40:0/1312106042 learned_addr learned my addr 172.21.15.40:0/1312106042 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864965676 0 0) 0x7fa1ec0cd3d0 con 0x7fa1ec15e0b0 2024-03-24T00:31:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1f1200640 1 --2- 172.21.15.40:0/1312106042 >> v2:172.21.15.53:3300/0 conn(0x7fa1ec0b4b80 0x7fa1ec0b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d0003530 con 0x7fa1ec15e0b0 2024-03-24T00:31:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2415553053 0 0) 0x7fa1d0003530 con 0x7fa1ec15e0b0 2024-03-24T00:31:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 >> v2:172.21.15.53:3300/0 conn(0x7fa1ec0b4b80 msgr2=0x7fa1ec0b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 --2- 172.21.15.40:0/1312106042 >> v2:172.21.15.53:3300/0 conn(0x7fa1ec0b4b80 0x7fa1ec0b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 >> v1:172.21.15.53:6789/0 conn(0x7fa1ec06bd10 legacy=0x7fa1ec0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec0cd990 con 0x7fa1ec15e0b0 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fa1dc001680 con 0x7fa1ec15e0b0 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 >> v1:172.21.15.40:6789/0 conn(0x7fa1ec15e0b0 legacy=0x7fa1ec06b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 --2- 172.21.15.40:0/1312106042 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1d00039c0 0x7fa1d0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 --2- 172.21.15.40:0/1312106042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1d0004810 0x7fa1d0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 --2- 172.21.15.40:0/1312106042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1d00051d0 0x7fa1d000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d000c050 con 0x7fa1d00051d0 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d000c200 con 0x7fa1d0004810 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d000c400 con 0x7fa1d00039c0 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1f1200640 1 --2- 172.21.15.40:0/1312106042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1d00051d0 0x7fa1d000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1ebe00640 1 --2- 172.21.15.40:0/1312106042 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1d00039c0 0x7fa1d0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1e3e00640 1 --2- 172.21.15.40:0/1312106042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1d0004810 0x7fa1d0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1f1200640 1 -- 172.21.15.40:0/1312106042 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1d00039c0 msgr2=0x7fa1d0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1f1200640 1 --2- 172.21.15.40:0/1312106042 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1d00039c0 0x7fa1d0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1f1200640 1 -- 172.21.15.40:0/1312106042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1d0004810 msgr2=0x7fa1d0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:04.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1f1200640 1 --2- 172.21.15.40:0/1312106042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1d0004810 0x7fa1d0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1f1200640 1 -- 172.21.15.40:0/1312106042 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d0003530 con 0x7fa1d00051d0 2024-03-24T00:31:04.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.742+0000 7fa1e3e00640 1 --2- 172.21.15.40:0/1312106042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1d0004810 0x7fa1d0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:04.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.746+0000 7fa1f1200640 1 -- 172.21.15.40:0/1312106042 reap_dead start 2024-03-24T00:31:04.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.746+0000 7fa1f1200640 1 --2- 172.21.15.40:0/1312106042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1d00051d0 0x7fa1d000b9d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009f90 tx=0x7fa1e401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e92602d5d87c8b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:04.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.746+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40b2070 con 0x7fa1d00051d0 2024-03-24T00:31:04.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.746+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e40a5c90 con 0x7fa1d00051d0 2024-03-24T00:31:04.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.746+0000 7fa1eb400640 1 -- 172.21.15.40:0/1312106042 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e4009c10 con 0x7fa1d00051d0 2024-03-24T00:31:04.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.746+0000 7fa1f3200640 1 -- 172.21.15.40:0/1312106042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1d00051d0 msgr2=0x7fa1d000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.746+0000 7fa1f3200640 1 --2- 172.21.15.40:0/1312106042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1d00051d0 0x7fa1d000b9d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009f90 tx=0x7fa1e401cc80 comp rx=0 tx=0).stop 2024-03-24T00:31:04.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.746+0000 7fa1f3200640 1 -- 172.21.15.40:0/1312106042 shutdown_connections 2024-03-24T00:31:04.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.746+0000 7fa1f3200640 1 --2- 172.21.15.40:0/1312106042 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1d00051d0 0x7fa1d000b9d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.746+0000 7fa1f3200640 1 -- 172.21.15.40:0/1312106042 >> 172.21.15.40:0/1312106042 conn(0x7fa1ec0b3880 msgr2=0x7fa1ec0b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:04.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.746+0000 7fa1f3200640 1 -- 172.21.15.40:0/1312106042 shutdown_connections 2024-03-24T00:31:04.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.746+0000 7fa1f3200640 1 -- 172.21.15.40:0/1312106042 wait complete. 2024-03-24T00:31:04.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f3200640 1 Processor -- start 2024-03-24T00:31:04.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f3200640 1 -- start start 2024-03-24T00:31:04.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f3200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1d00051d0 0x7fa1ec1f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec1602d0 0x7fa1ec0ced00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1ec15fd80 0x7fa1ec0cf240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec06c480 con 0x7fa1ec15fd80 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f3200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0cdfe0 con 0x7fa1ec1602d0 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f3200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec0b43a0 con 0x7fa1d00051d0 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1ec15fd80 0x7fa1ec0cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1ec15fd80 0x7fa1ec0cf240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57164/0 (socket says 172.21.15.40:57164) 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f1200640 1 -- 172.21.15.40:0/2283648246 learned_addr learned my addr 172.21.15.40:0/2283648246 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1ebe00640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1d00051d0 0x7fa1ec1f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1e3e00640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec1602d0 0x7fa1ec0ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f1200640 1 -- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1d00051d0 msgr2=0x7fa1ec1f31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f1200640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1d00051d0 0x7fa1ec1f31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f1200640 1 -- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec1602d0 msgr2=0x7fa1ec0ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f1200640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec1602d0 0x7fa1ec0ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f1200640 1 -- 172.21.15.40:0/2283648246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec0b6150 con 0x7fa1ec15fd80 2024-03-24T00:31:04.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f1200640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1ec15fd80 0x7fa1ec0cf240 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa1e401cab0 tx=0x7fa1e4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:04.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1e8c00640 1 -- 172.21.15.40:0/2283648246 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40bd020 con 0x7fa1ec15fd80 2024-03-24T00:31:04.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1e8c00640 1 -- 172.21.15.40:0/2283648246 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e40a6e50 con 0x7fa1ec15fd80 2024-03-24T00:31:04.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f3200640 1 -- 172.21.15.40:0/2283648246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ec0b6430 con 0x7fa1ec15fd80 2024-03-24T00:31:04.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1e8c00640 1 -- 172.21.15.40:0/2283648246 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e40b2040 con 0x7fa1ec15fd80 2024-03-24T00:31:04.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.750+0000 7fa1f3200640 1 -- 172.21.15.40:0/2283648246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ec0b69f0 con 0x7fa1ec15fd80 2024-03-24T00:31:04.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.754+0000 7fa1f3200640 1 -- 172.21.15.40:0/2283648246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ec068170 con 0x7fa1ec15fd80 2024-03-24T00:31:04.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.754+0000 7fa1e8c00640 1 -- 172.21.15.40:0/2283648246 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa1e40b8420 con 0x7fa1ec15fd80 2024-03-24T00:31:04.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.754+0000 7fa1e8c00640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1c8067660 0x7fa1c8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:04.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.754+0000 7fa1e8c00640 1 -- 172.21.15.40:0/2283648246 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1e41205c0 con 0x7fa1ec15fd80 2024-03-24T00:31:04.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.754+0000 7fa1ebe00640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1c8067660 0x7fa1c8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:04.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.754+0000 7fa1ebe00640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1c8067660 0x7fa1c8069b20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc000c10 tx=0x7fa1dc007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:04.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.762+0000 7fa1e8c00640 1 -- 172.21.15.40:0/2283648246 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa1e40c7060 con 0x7fa1ec15fd80 2024-03-24T00:31:04.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.922+0000 7fa1f3200640 1 -- 172.21.15.40:0/2283648246 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1ec1614f0 con 0x7fa1c8067660 2024-03-24T00:31:04.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.930+0000 7fa1e8c00640 1 -- 172.21.15.40:0/2283648246 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240055 (secure 0 0 0) 0x7fa1ec1614f0 con 0x7fa1c8067660 2024-03-24T00:31:04.935 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:04.938 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:31:04.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.934+0000 7fa1f3200640 1 -- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1c8067660 msgr2=0x7fa1c8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.934+0000 7fa1f3200640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1c8067660 0x7fa1c8069b20 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc000c10 tx=0x7fa1dc007040 comp rx=0 tx=0).stop 2024-03-24T00:31:04.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.934+0000 7fa1f3200640 1 -- 172.21.15.40:0/2283648246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1ec15fd80 msgr2=0x7fa1ec0cf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:04.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.934+0000 7fa1f3200640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1ec15fd80 0x7fa1ec0cf240 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa1e401cab0 tx=0x7fa1e4009750 comp rx=0 tx=0).stop 2024-03-24T00:31:04.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.934+0000 7fa1f3200640 1 -- 172.21.15.40:0/2283648246 shutdown_connections 2024-03-24T00:31:04.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.934+0000 7fa1f3200640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1c8067660 0x7fa1c8069b20 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.934+0000 7fa1f3200640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1ec15fd80 0x7fa1ec0cf240 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.934+0000 7fa1f3200640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec1602d0 0x7fa1ec0ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.934+0000 7fa1f3200640 1 --2- 172.21.15.40:0/2283648246 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1d00051d0 0x7fa1ec1f31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:04.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.934+0000 7fa1f3200640 1 -- 172.21.15.40:0/2283648246 >> 172.21.15.40:0/2283648246 conn(0x7fa1ec0b3880 msgr2=0x7fa1ec15d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:04.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.934+0000 7fa1f3200640 1 -- 172.21.15.40:0/2283648246 shutdown_connections 2024-03-24T00:31:04.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:04.934+0000 7fa1f3200640 1 -- 172.21.15.40:0/2283648246 wait complete. 2024-03-24T00:31:04.963 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-03-24T00:31:03.197833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235884,"kb_used_data":2448,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738836,"statfs":{"total":773094113280,"available":772852568064,"internally_reserved":0,"allocated":2506752,"data_stored":1581774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.173272"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028717+0000","last_change":"2024-03-24T00:30:58.028717+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028717+0000","last_undegraded":"2024-03-24T00:30:58.028717+0000","last_fullsized":"2024-03-24T00:30:58.028717+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.032033+0000","last_change":"2024-03-24T00:30:58.032033+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.032033+0000","last_undegraded":"2024-03-24T00:30:58.032033+0000","last_fullsized":"2024-03-24T00:30:58.032033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029157+0000","last_change":"2024-03-24T00:30:58.029157+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029157+0000","last_undegraded":"2024-03-24T00:30:58.029157+0000","last_fullsized":"2024-03-24T00:30:58.029157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029986+0000","last_change":"2024-03-24T00:30:58.029986+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029986+0000","last_undegraded":"2024-03-24T00:30:58.029986+0000","last_fullsized":"2024-03-24T00:30:58.029986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.034511+0000","last_change":"2024-03-24T00:30:58.034511+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.034511+0000","last_undegraded":"2024-03-24T00:30:58.034511+0000","last_fullsized":"2024-03-24T00:30:58.034511+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029515+0000","last_change":"2024-03-24T00:30:58.029515+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029515+0000","last_undegraded":"2024-03-24T00:30:58.029515+0000","last_fullsized":"2024-03-24T00:30:58.029515+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028833+0000","last_change":"2024-03-24T00:30:58.028833+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028833+0000","last_undegraded":"2024-03-24T00:30:58.028833+0000","last_fullsized":"2024-03-24T00:30:58.028833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033549+0000","last_change":"2024-03-24T00:30:58.033549+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033549+0000","last_undegraded":"2024-03-24T00:30:58.033549+0000","last_fullsized":"2024-03-24T00:30:58.033549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.022064+0000","last_change":"2024-03-24T00:30:58.022064+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.022064+0000","last_undegraded":"2024-03-24T00:30:58.022064+0000","last_fullsized":"2024-03-24T00:30:58.022064+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.030365+0000","last_change":"2024-03-24T00:30:58.030365+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.030365+0000","last_undegraded":"2024-03-24T00:30:58.030365+0000","last_fullsized":"2024-03-24T00:30:58.030365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.032033+0000","last_change":"2024-03-24T00:30:58.032033+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.032033+0000","last_undegraded":"2024-03-24T00:30:58.032033+0000","last_fullsized":"2024-03-24T00:30:58.032033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.032696+0000","last_change":"2024-03-24T00:30:58.032696+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.032696+0000","last_undegraded":"2024-03-24T00:30:58.032696+0000","last_fullsized":"2024-03-24T00:30:58.032696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.020662+0000","last_change":"2024-03-24T00:30:58.020662+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.020662+0000","last_undegraded":"2024-03-24T00:30:58.020662+0000","last_fullsized":"2024-03-24T00:30:58.020662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033693+0000","last_change":"2024-03-24T00:30:58.033693+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033693+0000","last_undegraded":"2024-03-24T00:30:58.033693+0000","last_fullsized":"2024-03-24T00:30:58.033693+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028209+0000","last_change":"2024-03-24T00:30:58.028209+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028209+0000","last_undegraded":"2024-03-24T00:30:58.028209+0000","last_fullsized":"2024-03-24T00:30:58.028209+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033002+0000","last_change":"2024-03-24T00:30:58.033002+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033002+0000","last_undegraded":"2024-03-24T00:30:58.033002+0000","last_fullsized":"2024-03-24T00:30:58.033002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-03-24T00:30:58.023125+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:30:58.023125+0000","last_peered":"2024-03-24T00:30:58.023125+0000","last_clean":"2024-03-24T00:30:58.023125+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:30:58.023125+0000","last_undegraded":"2024-03-24T00:30:58.023125+0000","last_fullsized":"2024-03-24T00:30:58.023125+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029224+0000","last_change":"2024-03-24T00:30:58.029224+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029224+0000","last_undegraded":"2024-03-24T00:30:58.029224+0000","last_fullsized":"2024-03-24T00:30:58.029224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.019558+0000","last_change":"2024-03-24T00:30:58.019558+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.019558+0000","last_undegraded":"2024-03-24T00:30:58.019558+0000","last_fullsized":"2024-03-24T00:30:58.019558+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033479+0000","last_change":"2024-03-24T00:30:58.033479+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033479+0000","last_undegraded":"2024-03-24T00:30:58.033479+0000","last_fullsized":"2024-03-24T00:30:58.033479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029244+0000","last_change":"2024-03-24T00:30:58.029244+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029244+0000","last_undegraded":"2024-03-24T00:30:58.029244+0000","last_fullsized":"2024-03-24T00:30:58.029244+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028482+0000","last_change":"2024-03-24T00:30:58.028482+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028482+0000","last_undegraded":"2024-03-24T00:30:58.028482+0000","last_fullsized":"2024-03-24T00:30:58.028482+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.019738+0000","last_change":"2024-03-24T00:30:58.019738+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.019738+0000","last_undegraded":"2024-03-24T00:30:58.019738+0000","last_fullsized":"2024-03-24T00:30:58.019738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028257+0000","last_change":"2024-03-24T00:30:58.028257+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028257+0000","last_undegraded":"2024-03-24T00:30:58.028257+0000","last_fullsized":"2024-03-24T00:30:58.028257+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.033671+0000","last_change":"2024-03-24T00:30:58.033671+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.033671+0000","last_undegraded":"2024-03-24T00:30:58.033671+0000","last_fullsized":"2024-03-24T00:30:58.033671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.020377+0000","last_change":"2024-03-24T00:30:58.020377+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.020377+0000","last_undegraded":"2024-03-24T00:30:58.020377+0000","last_fullsized":"2024-03-24T00:30:58.020377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.032623+0000","last_change":"2024-03-24T00:30:58.032623+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.032623+0000","last_undegraded":"2024-03-24T00:30:58.032623+0000","last_fullsized":"2024-03-24T00:30:58.032623+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.029744+0000","last_change":"2024-03-24T00:30:58.029744+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.029744+0000","last_undegraded":"2024-03-24T00:30:58.029744+0000","last_fullsized":"2024-03-24T00:30:58.029744+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.019863+0000","last_change":"2024-03-24T00:30:58.019863+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.019863+0000","last_undegraded":"2024-03-24T00:30:58.019863+0000","last_fullsized":"2024-03-24T00:30:58.019863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.019696+0000","last_change":"2024-03-24T00:30:58.019696+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.019696+0000","last_undegraded":"2024-03-24T00:30:58.019696+0000","last_fullsized":"2024-03-24T00:30:58.019696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.021452+0000","last_change":"2024-03-24T00:30:58.021452+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.021452+0000","last_undegraded":"2024-03-24T00:30:58.021452+0000","last_fullsized":"2024-03-24T00:30:58.021452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.028166+0000","last_change":"2024-03-24T00:30:58.028166+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.028166+0000","last_undegraded":"2024-03-24T00:30:58.028166+0000","last_fullsized":"2024-03-24T00:30:58.028166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T00:30:59.014979+0000","last_change":"2024-03-24T00:30:59.014979+0000","last_active":"2024-03-24T00:30:59.014979+0000","last_peered":"2024-03-24T00:30:59.014979+0000","last_clean":"2024-03-24T00:30:59.014979+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:59.014979+0000","last_undegraded":"2024-03-24T00:30:59.014979+0000","last_fullsized":"2024-03-24T00:30:59.014979+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1,"reported_epoch":17,"state":"creating+peering","last_fresh":"2024-03-24T00:30:58.022059+0000","last_change":"2024-03-24T00:30:58.022059+0000","last_active":"2024-03-24T00:30:58.008493+0000","last_peered":"2024-03-24T00:30:58.008493+0000","last_clean":"2024-03-24T00:30:58.008493+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T00:30:58.022059+0000","last_undegraded":"2024-03-24T00:30:58.022059+0000","last_fullsized":"2024-03-24T00:30:58.022059+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31924,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339916,"statfs":{"total":96636764160,"available":96604073984,"internally_reserved":0,"allocated":761856,"data_stored":641952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31344,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340496,"statfs":{"total":96636764160,"available":96604667904,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574850,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27180,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344660,"statfs":{"total":96636764160,"available":96608931840,"internally_reserved":0,"allocated":155648,"data_stored":45767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28388,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343452,"statfs":{"total":96636764160,"available":96607694848,"internally_reserved":0,"allocated":761856,"data_stored":641952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:31:04.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3f is not active+clean 2024-03-24T00:31:04.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3e is not active+clean 2024-03-24T00:31:04.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3d is not active+clean 2024-03-24T00:31:04.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3c is not active+clean 2024-03-24T00:31:04.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3b is not active+clean 2024-03-24T00:31:04.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3a is not active+clean 2024-03-24T00:31:04.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.39 is not active+clean 2024-03-24T00:31:04.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.38 is not active+clean 2024-03-24T00:31:04.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.37 is not active+clean 2024-03-24T00:31:04.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.36 is not active+clean 2024-03-24T00:31:04.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.35 is not active+clean 2024-03-24T00:31:04.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.34 is not active+clean 2024-03-24T00:31:04.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.33 is not active+clean 2024-03-24T00:31:04.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.32 is not active+clean 2024-03-24T00:31:04.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.31 is not active+clean 2024-03-24T00:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.30 is not active+clean 2024-03-24T00:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2f is not active+clean 2024-03-24T00:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2e is not active+clean 2024-03-24T00:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2d is not active+clean 2024-03-24T00:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2c is not active+clean 2024-03-24T00:31:04.970 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2b is not active+clean 2024-03-24T00:31:04.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2a is not active+clean 2024-03-24T00:31:04.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.29 is not active+clean 2024-03-24T00:31:04.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.28 is not active+clean 2024-03-24T00:31:04.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.27 is not active+clean 2024-03-24T00:31:04.971 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.971 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.26 is not active+clean 2024-03-24T00:31:04.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.25 is not active+clean 2024-03-24T00:31:04.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.24 is not active+clean 2024-03-24T00:31:04.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.23 is not active+clean 2024-03-24T00:31:04.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.972 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.22 is not active+clean 2024-03-24T00:31:04.972 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.21 is not active+clean 2024-03-24T00:31:04.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.20 is not active+clean 2024-03-24T00:31:04.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1a is not active+clean 2024-03-24T00:31:04.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1b is not active+clean 2024-03-24T00:31:04.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.028717+0000', 'last_change': '2024-03-24T00:30:58.028717+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.028717+0000', 'last_undegraded': '2024-03-24T00:30:58.028717+0000', 'last_fullsized': '2024-03-24T00:30:58.028717+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:31:04.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1b is not active+clean 2024-03-24T00:31:04.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.973 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1a is not active+clean 2024-03-24T00:31:04.973 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.032033+0000', 'last_change': '2024-03-24T00:30:58.032033+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.032033+0000', 'last_undegraded': '2024-03-24T00:30:58.032033+0000', 'last_fullsized': '2024-03-24T00:30:58.032033+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:31:04.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.19 is not active+clean 2024-03-24T00:31:04.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.029157+0000', 'last_change': '2024-03-24T00:30:58.029157+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.029157+0000', 'last_undegraded': '2024-03-24T00:30:58.029157+0000', 'last_fullsized': '2024-03-24T00:30:58.029157+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:31:04.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.18 is not active+clean 2024-03-24T00:31:04.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.18 is not active+clean 2024-03-24T00:31:04.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.029986+0000', 'last_change': '2024-03-24T00:30:58.029986+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.029986+0000', 'last_undegraded': '2024-03-24T00:30:58.029986+0000', 'last_fullsized': '2024-03-24T00:30:58.029986+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:31:04.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.19 is not active+clean 2024-03-24T00:31:04.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.974 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.17 is not active+clean 2024-03-24T00:31:04.974 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.034511+0000', 'last_change': '2024-03-24T00:30:58.034511+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.034511+0000', 'last_undegraded': '2024-03-24T00:30:58.034511+0000', 'last_fullsized': '2024-03-24T00:30:58.034511+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:31:04.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.16 is not active+clean 2024-03-24T00:31:04.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.16 is not active+clean 2024-03-24T00:31:04.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.029515+0000', 'last_change': '2024-03-24T00:30:58.029515+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.029515+0000', 'last_undegraded': '2024-03-24T00:30:58.029515+0000', 'last_fullsized': '2024-03-24T00:30:58.029515+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:31:04.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.17 is not active+clean 2024-03-24T00:31:04.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.15 is not active+clean 2024-03-24T00:31:04.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.028833+0000', 'last_change': '2024-03-24T00:30:58.028833+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.028833+0000', 'last_undegraded': '2024-03-24T00:30:58.028833+0000', 'last_fullsized': '2024-03-24T00:30:58.028833+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:31:04.975 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.14 is not active+clean 2024-03-24T00:31:04.975 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.14 is not active+clean 2024-03-24T00:31:04.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.033549+0000', 'last_change': '2024-03-24T00:30:58.033549+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.033549+0000', 'last_undegraded': '2024-03-24T00:30:58.033549+0000', 'last_fullsized': '2024-03-24T00:30:58.033549+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:31:04.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.15 is not active+clean 2024-03-24T00:31:04.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.13 is not active+clean 2024-03-24T00:31:04.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.022064+0000', 'last_change': '2024-03-24T00:30:58.022064+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.022064+0000', 'last_undegraded': '2024-03-24T00:30:58.022064+0000', 'last_fullsized': '2024-03-24T00:30:58.022064+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:31:04.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.12 is not active+clean 2024-03-24T00:31:04.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.12 is not active+clean 2024-03-24T00:31:04.976 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.030365+0000', 'last_change': '2024-03-24T00:30:58.030365+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.030365+0000', 'last_undegraded': '2024-03-24T00:30:58.030365+0000', 'last_fullsized': '2024-03-24T00:30:58.030365+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:31:04.976 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.13 is not active+clean 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.11 is not active+clean 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.032033+0000', 'last_change': '2024-03-24T00:30:58.032033+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.032033+0000', 'last_undegraded': '2024-03-24T00:30:58.032033+0000', 'last_fullsized': '2024-03-24T00:30:58.032033+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.10 is not active+clean 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.10 is not active+clean 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.032696+0000', 'last_change': '2024-03-24T00:30:58.032696+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.032696+0000', 'last_undegraded': '2024-03-24T00:30:58.032696+0000', 'last_fullsized': '2024-03-24T00:30:58.032696+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.11 is not active+clean 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.f is not active+clean 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.020662+0000', 'last_change': '2024-03-24T00:30:58.020662+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.020662+0000', 'last_undegraded': '2024-03-24T00:30:58.020662+0000', 'last_fullsized': '2024-03-24T00:30:58.020662+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.e is not active+clean 2024-03-24T00:31:04.977 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.e is not active+clean 2024-03-24T00:31:04.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.033693+0000', 'last_change': '2024-03-24T00:30:58.033693+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.033693+0000', 'last_undegraded': '2024-03-24T00:30:58.033693+0000', 'last_fullsized': '2024-03-24T00:30:58.033693+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:31:04.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.f is not active+clean 2024-03-24T00:31:04.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.d is not active+clean 2024-03-24T00:31:04.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.028209+0000', 'last_change': '2024-03-24T00:30:58.028209+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.028209+0000', 'last_undegraded': '2024-03-24T00:30:58.028209+0000', 'last_fullsized': '2024-03-24T00:30:58.028209+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:31:04.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.c is not active+clean 2024-03-24T00:31:04.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.c is not active+clean 2024-03-24T00:31:04.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.033002+0000', 'last_change': '2024-03-24T00:30:58.033002+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.033002+0000', 'last_undegraded': '2024-03-24T00:30:58.033002+0000', 'last_fullsized': '2024-03-24T00:30:58.033002+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:31:04.978 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.d is not active+clean 2024-03-24T00:31:04.978 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.3 is not active+clean 2024-03-24T00:31:04.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.029224+0000', 'last_change': '2024-03-24T00:30:58.029224+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.029224+0000', 'last_undegraded': '2024-03-24T00:30:58.029224+0000', 'last_fullsized': '2024-03-24T00:30:58.029224+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:31:04.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2 is not active+clean 2024-03-24T00:31:04.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.0 is not active+clean 2024-03-24T00:31:04.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.019558+0000', 'last_change': '2024-03-24T00:30:58.019558+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.019558+0000', 'last_undegraded': '2024-03-24T00:30:58.019558+0000', 'last_fullsized': '2024-03-24T00:30:58.019558+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:31:04.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1 is not active+clean 2024-03-24T00:31:04.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1 is not active+clean 2024-03-24T00:31:04.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.033479+0000', 'last_change': '2024-03-24T00:30:58.033479+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.033479+0000', 'last_undegraded': '2024-03-24T00:30:58.033479+0000', 'last_fullsized': '2024-03-24T00:30:58.033479+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-24T00:31:04.979 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.0 is not active+clean 2024-03-24T00:31:04.979 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.2 is not active+clean 2024-03-24T00:31:04.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.029244+0000', 'last_change': '2024-03-24T00:30:58.029244+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.029244+0000', 'last_undegraded': '2024-03-24T00:30:58.029244+0000', 'last_fullsized': '2024-03-24T00:30:58.029244+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-24T00:31:04.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.3 is not active+clean 2024-03-24T00:31:04.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.4 is not active+clean 2024-03-24T00:31:04.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.028482+0000', 'last_change': '2024-03-24T00:30:58.028482+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.028482+0000', 'last_undegraded': '2024-03-24T00:30:58.028482+0000', 'last_fullsized': '2024-03-24T00:30:58.028482+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:31:04.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.5 is not active+clean 2024-03-24T00:31:04.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.980 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.5 is not active+clean 2024-03-24T00:31:04.980 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.019738+0000', 'last_change': '2024-03-24T00:30:58.019738+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.019738+0000', 'last_undegraded': '2024-03-24T00:30:58.019738+0000', 'last_fullsized': '2024-03-24T00:30:58.019738+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:31:04.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.4 is not active+clean 2024-03-24T00:31:04.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.6 is not active+clean 2024-03-24T00:31:04.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.028257+0000', 'last_change': '2024-03-24T00:30:58.028257+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.028257+0000', 'last_undegraded': '2024-03-24T00:30:58.028257+0000', 'last_fullsized': '2024-03-24T00:30:58.028257+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:31:04.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.7 is not active+clean 2024-03-24T00:31:04.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.7 is not active+clean 2024-03-24T00:31:04.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.033671+0000', 'last_change': '2024-03-24T00:30:58.033671+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.033671+0000', 'last_undegraded': '2024-03-24T00:30:58.033671+0000', 'last_fullsized': '2024-03-24T00:30:58.033671+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-24T00:31:04.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.6 is not active+clean 2024-03-24T00:31:04.981 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.981 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.8 is not active+clean 2024-03-24T00:31:04.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.020377+0000', 'last_change': '2024-03-24T00:30:58.020377+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.020377+0000', 'last_undegraded': '2024-03-24T00:30:58.020377+0000', 'last_fullsized': '2024-03-24T00:30:58.020377+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:31:04.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.9 is not active+clean 2024-03-24T00:31:04.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.9 is not active+clean 2024-03-24T00:31:04.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.032623+0000', 'last_change': '2024-03-24T00:30:58.032623+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.032623+0000', 'last_undegraded': '2024-03-24T00:30:58.032623+0000', 'last_fullsized': '2024-03-24T00:30:58.032623+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:31:04.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.8 is not active+clean 2024-03-24T00:31:04.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.a is not active+clean 2024-03-24T00:31:04.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.029744+0000', 'last_change': '2024-03-24T00:30:58.029744+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.029744+0000', 'last_undegraded': '2024-03-24T00:30:58.029744+0000', 'last_fullsized': '2024-03-24T00:30:58.029744+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-24T00:31:04.982 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.b is not active+clean 2024-03-24T00:31:04.982 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.b is not active+clean 2024-03-24T00:31:04.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.019863+0000', 'last_change': '2024-03-24T00:30:58.019863+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.019863+0000', 'last_undegraded': '2024-03-24T00:30:58.019863+0000', 'last_fullsized': '2024-03-24T00:30:58.019863+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:31:04.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.a is not active+clean 2024-03-24T00:31:04.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1d is not active+clean 2024-03-24T00:31:04.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1c is not active+clean 2024-03-24T00:31:04.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.019696+0000', 'last_change': '2024-03-24T00:30:58.019696+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.019696+0000', 'last_undegraded': '2024-03-24T00:30:58.019696+0000', 'last_fullsized': '2024-03-24T00:30:58.019696+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:31:04.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1c is not active+clean 2024-03-24T00:31:04.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.983 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1d is not active+clean 2024-03-24T00:31:04.983 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.021452+0000', 'last_change': '2024-03-24T00:30:58.021452+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.021452+0000', 'last_undegraded': '2024-03-24T00:30:58.021452+0000', 'last_fullsized': '2024-03-24T00:30:58.021452+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-24T00:31:04.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1f is not active+clean 2024-03-24T00:31:04.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1e is not active+clean 2024-03-24T00:31:04.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.028166+0000', 'last_change': '2024-03-24T00:30:58.028166+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.028166+0000', 'last_undegraded': '2024-03-24T00:30:58.028166+0000', 'last_fullsized': '2024-03-24T00:30:58.028166+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-24T00:31:04.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1e is not active+clean 2024-03-24T00:31:04.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-24T00:30:59.014979+0000', 'last_change': '2024-03-24T00:30:59.014979+0000', 'last_active': '2024-03-24T00:30:59.014979+0000', 'last_peered': '2024-03-24T00:30:59.014979+0000', 'last_clean': '2024-03-24T00:30:59.014979+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:59.014979+0000', 'last_undegraded': '2024-03-24T00:30:59.014979+0000', 'last_fullsized': '2024-03-24T00:30:59.014979+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:59.014979+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-24T00:31:04.984 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.1f is not active+clean 2024-03-24T00:31:04.984 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 17, 'state': 'creating+peering', 'last_fresh': '2024-03-24T00:30:58.022059+0000', 'last_change': '2024-03-24T00:30:58.022059+0000', 'last_active': '2024-03-24T00:30:58.008493+0000', 'last_peered': '2024-03-24T00:30:58.008493+0000', 'last_clean': '2024-03-24T00:30:58.008493+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-24T00:30:58.022059+0000', 'last_undegraded': '2024-03-24T00:30:58.022059+0000', 'last_fullsized': '2024-03-24T00:30:58.022059+0000', 'mapping_epoch': 17, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 17, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'last_clean_scrub_stamp': '2024-03-24T00:30:58.008493+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-24T00:31:04.985 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-24T00:31:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.142+0000 7ff5aec00640 1 Processor -- start 2024-03-24T00:31:05.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.142+0000 7ff5aec00640 1 -- start start 2024-03-24T00:31:05.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.142+0000 7ff5aec00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5a815df80 0x7ff5a8070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.142+0000 7ff5aec00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff5a806c950 0x7ff5a80d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.142+0000 7ff5aec00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5a806bd20 0x7ff5a80d20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.142+0000 7ff5aec00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a806b890 con 0x7ff5a806bd20 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.142+0000 7ff5aec00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a806d0f0 con 0x7ff5a806c950 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.142+0000 7ff5aec00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a806b280 con 0x7ff5a815df80 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.146+0000 7ff5acc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5a806bd20 0x7ff5a80d20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.146+0000 7ff5acc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5a806bd20 0x7ff5a80d20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57180/0 (socket says 172.21.15.40:57180) 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.146+0000 7ff5acc00640 1 -- 172.21.15.40:0/1256139091 learned_addr learned my addr 172.21.15.40:0/1256139091 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.146+0000 7ff5a7e00640 1 --2- 172.21.15.40:0/1256139091 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5a815df80 0x7ff5a8070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.146+0000 7ff59fe00640 1 --2- 172.21.15.40:0/1256139091 >> v2:172.21.15.53:3300/0 conn(0x7ff5a806c950 0x7ff5a80d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.146+0000 7ff5acc00640 1 -- 172.21.15.40:0/1256139091 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5a815df80 msgr2=0x7ff5a8070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.146+0000 7ff5acc00640 1 --2- 172.21.15.40:0/1256139091 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5a815df80 0x7ff5a8070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.146+0000 7ff5acc00640 1 -- 172.21.15.40:0/1256139091 >> v2:172.21.15.53:3300/0 conn(0x7ff5a806c950 msgr2=0x7ff5a80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.146+0000 7ff5acc00640 1 --2- 172.21.15.40:0/1256139091 >> v2:172.21.15.53:3300/0 conn(0x7ff5a806c950 0x7ff5a80d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.146+0000 7ff5acc00640 1 -- 172.21.15.40:0/1256139091 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a806ce10 con 0x7ff5a806bd20 2024-03-24T00:31:05.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.146+0000 7ff59fe00640 1 --2- 172.21.15.40:0/1256139091 >> v2:172.21.15.53:3300/0 conn(0x7ff5a806c950 0x7ff5a80d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:05.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.146+0000 7ff5a7e00640 1 --2- 172.21.15.40:0/1256139091 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5a815df80 0x7ff5a8070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:05.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5acc00640 1 --2- 172.21.15.40:0/1256139091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a806bd20 0x7ff5a80d20a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff5a806e090 tx=0x7ff5a001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1801dc65e45241c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5a7400640 1 -- 172.21.15.40:0/1256139091 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a00a5800 con 0x7ff5a806bd20 2024-03-24T00:31:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5a7400640 1 -- 172.21.15.40:0/1256139091 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a00a59a0 con 0x7ff5a806bd20 2024-03-24T00:31:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5a7400640 1 -- 172.21.15.40:0/1256139091 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a00a5d70 con 0x7ff5a806bd20 2024-03-24T00:31:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5aec00640 1 -- 172.21.15.40:0/1256139091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a806bd20 msgr2=0x7ff5a80d20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:05.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5aec00640 1 --2- 172.21.15.40:0/1256139091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a806bd20 0x7ff5a80d20a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff5a806e090 tx=0x7ff5a001c9c0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5aec00640 1 -- 172.21.15.40:0/1256139091 shutdown_connections 2024-03-24T00:31:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5aec00640 1 --2- 172.21.15.40:0/1256139091 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a806bd20 0x7ff5a80d20a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5aec00640 1 --2- 172.21.15.40:0/1256139091 >> v2:172.21.15.53:3300/0 conn(0x7ff5a806c950 0x7ff5a80d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5aec00640 1 --2- 172.21.15.40:0/1256139091 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5a815df80 0x7ff5a8070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5aec00640 1 -- 172.21.15.40:0/1256139091 >> 172.21.15.40:0/1256139091 conn(0x7ff5a806ecf0 msgr2=0x7ff5a806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5aec00640 1 -- 172.21.15.40:0/1256139091 shutdown_connections 2024-03-24T00:31:05.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.150+0000 7ff5aec00640 1 -- 172.21.15.40:0/1256139091 wait complete. 2024-03-24T00:31:05.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5aec00640 1 Processor -- start 2024-03-24T00:31:05.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5aec00640 1 -- start start 2024-03-24T00:31:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5aec00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5a806bd20 0x7ff5a8155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5aec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a806c950 0x7ff5a8151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5aec00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a815df80 0x7ff5a8154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:05.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5aec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a8162460 con 0x7ff5a806c950 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5aec00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a81622e0 con 0x7ff5a815df80 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5aec00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a81625e0 con 0x7ff5a806bd20 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff59fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a806c950 0x7ff5a8151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff59fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a806c950 0x7ff5a8151450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57184/0 (socket says 172.21.15.40:57184) 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff59fe00640 1 -- 172.21.15.40:0/2688582880 learned_addr learned my addr 172.21.15.40:0/2688582880 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5a7e00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5a806bd20 0x7ff5a8155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5acc00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a815df80 0x7ff5a8154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff59fe00640 1 -- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5a806bd20 msgr2=0x7ff5a8155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff59fe00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5a806bd20 0x7ff5a8155e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff59fe00640 1 -- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a815df80 msgr2=0x7ff5a8154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff59fe00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a815df80 0x7ff5a8154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff59fe00640 1 -- 172.21.15.40:0/2688582880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a8154de0 con 0x7ff5a806c950 2024-03-24T00:31:05.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff59fe00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a806c950 0x7ff5a8151450 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff59000e810 tx=0x7ff59000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5a4c00640 1 -- 172.21.15.40:0/2688582880 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590017070 con 0x7ff5a806c950 2024-03-24T00:31:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5a4c00640 1 -- 172.21.15.40:0/2688582880 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59000c8d0 con 0x7ff5a806c950 2024-03-24T00:31:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5aec00640 1 -- 172.21.15.40:0/2688582880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a81550c0 con 0x7ff5a806c950 2024-03-24T00:31:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5a4c00640 1 -- 172.21.15.40:0/2688582880 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59000f040 con 0x7ff5a806c950 2024-03-24T00:31:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5aec00640 1 -- 172.21.15.40:0/2688582880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a8155680 con 0x7ff5a806c950 2024-03-24T00:31:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.154+0000 7ff5aec00640 1 -- 172.21.15.40:0/2688582880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a8068170 con 0x7ff5a806c950 2024-03-24T00:31:05.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.158+0000 7ff5a4c00640 1 -- 172.21.15.40:0/2688582880 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff59001a070 con 0x7ff5a806c950 2024-03-24T00:31:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.158+0000 7ff5a4c00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff584067660 0x7ff584069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.158+0000 7ff5a4c00640 1 -- 172.21.15.40:0/2688582880 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff590085400 con 0x7ff5a806c950 2024-03-24T00:31:05.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.158+0000 7ff5a7e00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff584067660 0x7ff584069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:05.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.158+0000 7ff5a7e00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff584067660 0x7ff584069b20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8154220 tx=0x7ff598008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:05.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.162+0000 7ff5a4c00640 1 -- 172.21.15.40:0/2688582880 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff590055a20 con 0x7ff5a806c950 2024-03-24T00:31:05.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.382+0000 7ff5aec00640 1 -- 172.21.15.40:0/2688582880 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7ff5a80cf9d0 con 0x7ff5a806c950 2024-03-24T00:31:05.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.382+0000 7ff5a4c00640 1 -- 172.21.15.40:0/2688582880 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1107 (secure 0 0 0) 0x7ff590059ac0 con 0x7ff5a806c950 2024-03-24T00:31:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:05.385 INFO:teuthology.orchestra.run.smithi040.stdout:{"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":23,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":20,"num_osds":8,"num_up_osds":8,"osd_up_since":1711240252,"num_in_osds":8,"osd_in_since":1711240244,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":241545216,"bytes_avail":772852568064,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":8,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"c","status":"up:active","gid":4264}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-03-24T00:30:41.989897+0000","services":{}},"progress_events":{}} 2024-03-24T00:31:05.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.386+0000 7ff5aec00640 1 -- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff584067660 msgr2=0x7ff584069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:05.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.386+0000 7ff5aec00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff584067660 0x7ff584069b20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff5a8154220 tx=0x7ff598008040 comp rx=0 tx=0).stop 2024-03-24T00:31:05.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.386+0000 7ff5aec00640 1 -- 172.21.15.40:0/2688582880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a806c950 msgr2=0x7ff5a8151450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:05.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.386+0000 7ff5aec00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a806c950 0x7ff5a8151450 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff59000e810 tx=0x7ff59000ecd0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.386+0000 7ff5aec00640 1 -- 172.21.15.40:0/2688582880 shutdown_connections 2024-03-24T00:31:05.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.386+0000 7ff5aec00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff584067660 0x7ff584069b20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.386+0000 7ff5aec00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff5a815df80 0x7ff5a8154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.386+0000 7ff5aec00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5a806c950 0x7ff5a8151450 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.386+0000 7ff5aec00640 1 --2- 172.21.15.40:0/2688582880 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5a806bd20 0x7ff5a8155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.386+0000 7ff5aec00640 1 -- 172.21.15.40:0/2688582880 >> 172.21.15.40:0/2688582880 conn(0x7ff5a806ecf0 msgr2=0x7ff5a815efc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:05.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.386+0000 7ff5aec00640 1 -- 172.21.15.40:0/2688582880 shutdown_connections 2024-03-24T00:31:05.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.386+0000 7ff5aec00640 1 -- 172.21.15.40:0/2688582880 wait complete. 2024-03-24T00:31:05.408 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-24T00:31:05.408 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:31:05.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c42800640 1 Processor -- start 2024-03-24T00:31:05.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c42800640 1 -- start start 2024-03-24T00:31:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c42800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c3c15f600 0x7f0c3c0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c42800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c3c0d2c70 con 0x7f0c3c0d2830 2024-03-24T00:31:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c42800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c3c0b1e50 con 0x7f0c3c0af170 2024-03-24T00:31:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c42800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c0b3630 con 0x7f0c3c15f600 2024-03-24T00:31:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c33e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0c3c0d2830 0x7f0c3c0aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55136/0 (socket says 172.21.15.40:55136) 2024-03-24T00:31:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c33e00640 1 -- 172.21.15.40:0/3740060742 learned_addr learned my addr 172.21.15.40:0/3740060742 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c3b400640 1 --2- 172.21.15.40:0/3740060742 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c3c15f600 0x7f0c3c0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c3aa00640 1 -- 172.21.15.40:0/3740060742 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928433356 0 0) 0x7f0c3c0d2c70 con 0x7f0c3c0d2830 2024-03-24T00:31:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c3aa00640 1 -- 172.21.15.40:0/3740060742 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c20003530 con 0x7f0c3c0d2830 2024-03-24T00:31:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c3aa00640 1 -- 172.21.15.40:0/3740060742 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601523548 0 0) 0x7f0c3c0b1e50 con 0x7f0c3c0af170 2024-03-24T00:31:05.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c3aa00640 1 -- 172.21.15.40:0/3740060742 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c3c0d2c70 con 0x7f0c3c0af170 2024-03-24T00:31:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c3b400640 1 -- 172.21.15.40:0/3740060742 >> v1:172.21.15.53:6789/0 conn(0x7f0c3c0af170 legacy=0x7f0c3c165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c3b400640 1 -- 172.21.15.40:0/3740060742 >> v1:172.21.15.40:6789/0 conn(0x7f0c3c0d2830 legacy=0x7f0c3c0aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.546+0000 7f0c3b400640 1 -- 172.21.15.40:0/3740060742 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3c0b20f0 con 0x7f0c3c15f600 2024-03-24T00:31:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.550+0000 7f0c3b400640 1 --2- 172.21.15.40:0/3740060742 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c3c15f600 0x7f0c3c0b2910 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0097f0 tx=0x7f0c2c01ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=560c16510eedd5d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.550+0000 7f0c3aa00640 1 -- 172.21.15.40:0/3740060742 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c0b5070 con 0x7f0c3c15f600 2024-03-24T00:31:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.550+0000 7f0c3aa00640 1 -- 172.21.15.40:0/3740060742 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2c0a89d0 con 0x7f0c3c15f600 2024-03-24T00:31:05.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.550+0000 7f0c3aa00640 1 -- 172.21.15.40:0/3740060742 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c0a8da0 con 0x7f0c3c15f600 2024-03-24T00:31:05.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.550+0000 7f0c42800640 1 -- 172.21.15.40:0/3740060742 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c3c15f600 msgr2=0x7f0c3c0b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:05.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.550+0000 7f0c42800640 1 --2- 172.21.15.40:0/3740060742 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c3c15f600 0x7f0c3c0b2910 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0097f0 tx=0x7f0c2c01ecd0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.550+0000 7f0c42800640 1 -- 172.21.15.40:0/3740060742 shutdown_connections 2024-03-24T00:31:05.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.550+0000 7f0c42800640 1 --2- 172.21.15.40:0/3740060742 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c3c15f600 0x7f0c3c0b2910 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.550+0000 7f0c42800640 1 -- 172.21.15.40:0/3740060742 >> 172.21.15.40:0/3740060742 conn(0x7f0c3c0d1d10 msgr2=0x7f0c3c0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:05.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.550+0000 7f0c42800640 1 -- 172.21.15.40:0/3740060742 shutdown_connections 2024-03-24T00:31:05.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.550+0000 7f0c42800640 1 -- 172.21.15.40:0/3740060742 wait complete. 2024-03-24T00:31:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c42800640 1 Processor -- start 2024-03-24T00:31:05.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c42800640 1 -- start start 2024-03-24T00:31:05.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c42800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c3c0af170 0x7f0c3c1523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:05.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c42800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c3c0d2830 0x7f0c3c06feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:05.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c42800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3c15f600 0x7f0c3c0703f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:05.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c42800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c0b2fa0 con 0x7f0c3c15f600 2024-03-24T00:31:05.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c42800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c0b2740 con 0x7f0c3c0d2830 2024-03-24T00:31:05.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c42800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c15f9e0 con 0x7f0c3c0af170 2024-03-24T00:31:05.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c3be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3c15f600 0x7f0c3c0703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:05.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c3be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3c15f600 0x7f0c3c0703f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57194/0 (socket says 172.21.15.40:57194) 2024-03-24T00:31:05.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c3be00640 1 -- 172.21.15.40:0/3551562599 learned_addr learned my addr 172.21.15.40:0/3551562599 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:05.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c3b400640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c3c0af170 0x7f0c3c1523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:05.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c33e00640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c3c0d2830 0x7f0c3c06feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c3be00640 1 -- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c3c0af170 msgr2=0x7f0c3c1523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c3be00640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c3c0af170 0x7f0c3c1523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c3be00640 1 -- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c3c0d2830 msgr2=0x7f0c3c06feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c3be00640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c3c0d2830 0x7f0c3c06feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c3be00640 1 -- 172.21.15.40:0/3551562599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3c1528e0 con 0x7f0c3c15f600 2024-03-24T00:31:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c3b400640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c3c0af170 0x7f0c3c1523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:05.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c3be00640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3c15f600 0x7f0c3c0703f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400c760 tx=0x7f0c3400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:05.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c33400640 1 -- 172.21.15.40:0/3551562599 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34003b80 con 0x7f0c3c15f600 2024-03-24T00:31:05.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c33400640 1 -- 172.21.15.40:0/3551562599 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c34003d20 con 0x7f0c3c15f600 2024-03-24T00:31:05.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c33400640 1 -- 172.21.15.40:0/3551562599 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3400aa90 con 0x7f0c3c15f600 2024-03-24T00:31:05.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c42800640 1 -- 172.21.15.40:0/3551562599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3c152b90 con 0x7f0c3c15f600 2024-03-24T00:31:05.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.554+0000 7f0c42800640 1 -- 172.21.15.40:0/3551562599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3c153150 con 0x7f0c3c15f600 2024-03-24T00:31:05.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.558+0000 7f0c42800640 1 -- 172.21.15.40:0/3551562599 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c3c068170 con 0x7f0c3c15f600 2024-03-24T00:31:05.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.558+0000 7f0c33400640 1 -- 172.21.15.40:0/3551562599 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f0c3400ac30 con 0x7f0c3c15f600 2024-03-24T00:31:05.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.558+0000 7f0c33400640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c18067660 0x7f0c18069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:05.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.558+0000 7f0c33400640 1 -- 172.21.15.40:0/3551562599 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c34085870 con 0x7f0c3c15f600 2024-03-24T00:31:05.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.558+0000 7f0c3b400640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c18067660 0x7f0c18069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:05.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.562+0000 7f0c3b400640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c18067660 0x7f0c18069b20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0b2000 tx=0x7f0c2c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:05.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.562+0000 7f0c33400640 1 -- 172.21.15.40:0/3551562599 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0c34056450 con 0x7f0c3c15f600 2024-03-24T00:31:05.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.722+0000 7f0c42800640 1 -- 172.21.15.40:0/3551562599 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0c3c0d2220 con 0x7f0c18067660 2024-03-24T00:31:05.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.726+0000 7f0c33400640 1 -- 172.21.15.40:0/3551562599 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251947 (secure 0 0 0) 0x7f0c3c0d2220 con 0x7f0c18067660 2024-03-24T00:31:05.733 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:05.734 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:31:05.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.730+0000 7f0c42800640 1 -- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c18067660 msgr2=0x7f0c18069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:05.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.730+0000 7f0c42800640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c18067660 0x7f0c18069b20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0b2000 tx=0x7f0c2c008040 comp rx=0 tx=0).stop 2024-03-24T00:31:05.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.730+0000 7f0c42800640 1 -- 172.21.15.40:0/3551562599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3c15f600 msgr2=0x7f0c3c0703f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:05.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.730+0000 7f0c42800640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3c15f600 0x7f0c3c0703f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400c760 tx=0x7f0c3400cc20 comp rx=0 tx=0).stop 2024-03-24T00:31:05.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.730+0000 7f0c42800640 1 -- 172.21.15.40:0/3551562599 shutdown_connections 2024-03-24T00:31:05.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.730+0000 7f0c42800640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c18067660 0x7f0c18069b20 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.730+0000 7f0c42800640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c3c15f600 0x7f0c3c0703f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.730+0000 7f0c42800640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c3c0d2830 0x7f0c3c06feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.730+0000 7f0c42800640 1 --2- 172.21.15.40:0/3551562599 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c3c0af170 0x7f0c3c1523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:05.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.730+0000 7f0c42800640 1 -- 172.21.15.40:0/3551562599 >> 172.21.15.40:0/3551562599 conn(0x7f0c3c0d1d10 msgr2=0x7f0c3c15d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:05.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.730+0000 7f0c42800640 1 -- 172.21.15.40:0/3551562599 shutdown_connections 2024-03-24T00:31:05.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:05.730+0000 7f0c42800640 1 -- 172.21.15.40:0/3551562599 wait complete. 2024-03-24T00:31:05.754 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-03-24T00:31:05.198149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235632,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739088,"statfs":{"total":773094113280,"available":772852826112,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.173588"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046898+0000","last_change":"2024-03-24T00:31:00.052380+0000","last_active":"2024-03-24T00:31:01.046898+0000","last_peered":"2024-03-24T00:31:01.046898+0000","last_clean":"2024-03-24T00:31:01.046898+0000","last_became_active":"2024-03-24T00:31:00.049815+0000","last_became_peered":"2024-03-24T00:31:00.049815+0000","last_unstale":"2024-03-24T00:31:01.046898+0000","last_undegraded":"2024-03-24T00:31:01.046898+0000","last_fullsized":"2024-03-24T00:31:01.046898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:09.487064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047941+0000","last_change":"2024-03-24T00:31:00.048401+0000","last_active":"2024-03-24T00:31:01.047941+0000","last_peered":"2024-03-24T00:31:01.047941+0000","last_clean":"2024-03-24T00:31:01.047941+0000","last_became_active":"2024-03-24T00:31:00.048157+0000","last_became_peered":"2024-03-24T00:31:00.048157+0000","last_unstale":"2024-03-24T00:31:01.047941+0000","last_undegraded":"2024-03-24T00:31:01.047941+0000","last_fullsized":"2024-03-24T00:31:01.047941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909458+0000","last_change":"2024-03-24T00:31:00.052594+0000","last_active":"2024-03-24T00:31:01.909458+0000","last_peered":"2024-03-24T00:31:01.909458+0000","last_clean":"2024-03-24T00:31:01.909458+0000","last_became_active":"2024-03-24T00:31:00.052289+0000","last_became_peered":"2024-03-24T00:31:00.052289+0000","last_unstale":"2024-03-24T00:31:01.909458+0000","last_undegraded":"2024-03-24T00:31:01.909458+0000","last_fullsized":"2024-03-24T00:31:01.909458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:34.269764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909931+0000","last_change":"2024-03-24T00:31:00.072520+0000","last_active":"2024-03-24T00:31:01.909931+0000","last_peered":"2024-03-24T00:31:01.909931+0000","last_clean":"2024-03-24T00:31:01.909931+0000","last_became_active":"2024-03-24T00:31:00.071355+0000","last_became_peered":"2024-03-24T00:31:00.071355+0000","last_unstale":"2024-03-24T00:31:01.909931+0000","last_undegraded":"2024-03-24T00:31:01.909931+0000","last_fullsized":"2024-03-24T00:31:01.909931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:17.380183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046806+0000","last_change":"2024-03-24T00:31:00.045641+0000","last_active":"2024-03-24T00:31:01.046806+0000","last_peered":"2024-03-24T00:31:01.046806+0000","last_clean":"2024-03-24T00:31:01.046806+0000","last_became_active":"2024-03-24T00:31:00.045230+0000","last_became_peered":"2024-03-24T00:31:00.045230+0000","last_unstale":"2024-03-24T00:31:01.046806+0000","last_undegraded":"2024-03-24T00:31:01.046806+0000","last_fullsized":"2024-03-24T00:31:01.046806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:49.592126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909658+0000","last_change":"2024-03-24T00:31:00.057373+0000","last_active":"2024-03-24T00:31:01.909658+0000","last_peered":"2024-03-24T00:31:01.909658+0000","last_clean":"2024-03-24T00:31:01.909658+0000","last_became_active":"2024-03-24T00:31:00.056322+0000","last_became_peered":"2024-03-24T00:31:00.056322+0000","last_unstale":"2024-03-24T00:31:01.909658+0000","last_undegraded":"2024-03-24T00:31:01.909658+0000","last_fullsized":"2024-03-24T00:31:01.909658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045369+0000","last_change":"2024-03-24T00:31:00.048380+0000","last_active":"2024-03-24T00:31:01.045369+0000","last_peered":"2024-03-24T00:31:01.045369+0000","last_clean":"2024-03-24T00:31:01.045369+0000","last_became_active":"2024-03-24T00:31:00.046936+0000","last_became_peered":"2024-03-24T00:31:00.046936+0000","last_unstale":"2024-03-24T00:31:01.045369+0000","last_undegraded":"2024-03-24T00:31:01.045369+0000","last_fullsized":"2024-03-24T00:31:01.045369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:32.847580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909134+0000","last_change":"2024-03-24T00:31:00.057172+0000","last_active":"2024-03-24T00:31:01.909134+0000","last_peered":"2024-03-24T00:31:01.909134+0000","last_clean":"2024-03-24T00:31:01.909134+0000","last_became_active":"2024-03-24T00:31:00.056982+0000","last_became_peered":"2024-03-24T00:31:00.056982+0000","last_unstale":"2024-03-24T00:31:01.909134+0000","last_undegraded":"2024-03-24T00:31:01.909134+0000","last_fullsized":"2024-03-24T00:31:01.909134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:38.408736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047010+0000","last_change":"2024-03-24T00:31:00.053066+0000","last_active":"2024-03-24T00:31:01.047010+0000","last_peered":"2024-03-24T00:31:01.047010+0000","last_clean":"2024-03-24T00:31:01.047010+0000","last_became_active":"2024-03-24T00:31:00.052749+0000","last_became_peered":"2024-03-24T00:31:00.052749+0000","last_unstale":"2024-03-24T00:31:01.047010+0000","last_undegraded":"2024-03-24T00:31:01.047010+0000","last_fullsized":"2024-03-24T00:31:01.047010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:17.791692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060254+0000","last_change":"2024-03-24T00:31:00.060700+0000","last_active":"2024-03-24T00:31:01.060254+0000","last_peered":"2024-03-24T00:31:01.060254+0000","last_clean":"2024-03-24T00:31:01.060254+0000","last_became_active":"2024-03-24T00:31:00.060373+0000","last_became_peered":"2024-03-24T00:31:00.060373+0000","last_unstale":"2024-03-24T00:31:01.060254+0000","last_undegraded":"2024-03-24T00:31:01.060254+0000","last_fullsized":"2024-03-24T00:31:01.060254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060101+0000","last_change":"2024-03-24T00:31:00.061316+0000","last_active":"2024-03-24T00:31:01.060101+0000","last_peered":"2024-03-24T00:31:01.060101+0000","last_clean":"2024-03-24T00:31:01.060101+0000","last_became_active":"2024-03-24T00:31:00.060980+0000","last_became_peered":"2024-03-24T00:31:00.060980+0000","last_unstale":"2024-03-24T00:31:01.060101+0000","last_undegraded":"2024-03-24T00:31:01.060101+0000","last_fullsized":"2024-03-24T00:31:01.060101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045818+0000","last_change":"2024-03-24T00:31:00.046162+0000","last_active":"2024-03-24T00:31:01.045818+0000","last_peered":"2024-03-24T00:31:01.045818+0000","last_clean":"2024-03-24T00:31:01.045818+0000","last_became_active":"2024-03-24T00:31:00.045715+0000","last_became_peered":"2024-03-24T00:31:00.045715+0000","last_unstale":"2024-03-24T00:31:01.045818+0000","last_undegraded":"2024-03-24T00:31:01.045818+0000","last_fullsized":"2024-03-24T00:31:01.045818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:38.106034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909352+0000","last_change":"2024-03-24T00:31:00.045645+0000","last_active":"2024-03-24T00:31:01.909352+0000","last_peered":"2024-03-24T00:31:01.909352+0000","last_clean":"2024-03-24T00:31:01.909352+0000","last_became_active":"2024-03-24T00:31:00.045372+0000","last_became_peered":"2024-03-24T00:31:00.045372+0000","last_unstale":"2024-03-24T00:31:01.909352+0000","last_undegraded":"2024-03-24T00:31:01.909352+0000","last_fullsized":"2024-03-24T00:31:01.909352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909717+0000","last_change":"2024-03-24T00:31:00.042718+0000","last_active":"2024-03-24T00:31:01.909717+0000","last_peered":"2024-03-24T00:31:01.909717+0000","last_clean":"2024-03-24T00:31:01.909717+0000","last_became_active":"2024-03-24T00:31:00.042424+0000","last_became_peered":"2024-03-24T00:31:00.042424+0000","last_unstale":"2024-03-24T00:31:01.909717+0000","last_undegraded":"2024-03-24T00:31:01.909717+0000","last_fullsized":"2024-03-24T00:31:01.909717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047428+0000","last_change":"2024-03-24T00:31:00.038791+0000","last_active":"2024-03-24T00:31:01.047428+0000","last_peered":"2024-03-24T00:31:01.047428+0000","last_clean":"2024-03-24T00:31:01.047428+0000","last_became_active":"2024-03-24T00:31:00.038543+0000","last_became_peered":"2024-03-24T00:31:00.038543+0000","last_unstale":"2024-03-24T00:31:01.047428+0000","last_undegraded":"2024-03-24T00:31:01.047428+0000","last_fullsized":"2024-03-24T00:31:01.047428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909570+0000","last_change":"2024-03-24T00:31:00.055749+0000","last_active":"2024-03-24T00:31:01.909570+0000","last_peered":"2024-03-24T00:31:01.909570+0000","last_clean":"2024-03-24T00:31:01.909570+0000","last_became_active":"2024-03-24T00:31:00.055296+0000","last_became_peered":"2024-03-24T00:31:00.055296+0000","last_unstale":"2024-03-24T00:31:01.909570+0000","last_undegraded":"2024-03-24T00:31:01.909570+0000","last_fullsized":"2024-03-24T00:31:01.909570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910277+0000","last_change":"2024-03-24T00:31:00.055782+0000","last_active":"2024-03-24T00:31:01.910277+0000","last_peered":"2024-03-24T00:31:01.910277+0000","last_clean":"2024-03-24T00:31:01.910277+0000","last_became_active":"2024-03-24T00:31:00.055369+0000","last_became_peered":"2024-03-24T00:31:00.055369+0000","last_unstale":"2024-03-24T00:31:01.910277+0000","last_undegraded":"2024-03-24T00:31:01.910277+0000","last_fullsized":"2024-03-24T00:31:01.910277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:10.867603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186278+0000","last_change":"2024-03-24T00:31:00.044665+0000","last_active":"2024-03-24T00:31:01.186278+0000","last_peered":"2024-03-24T00:31:01.186278+0000","last_clean":"2024-03-24T00:31:01.186278+0000","last_became_active":"2024-03-24T00:31:00.044399+0000","last_became_peered":"2024-03-24T00:31:00.044399+0000","last_unstale":"2024-03-24T00:31:01.186278+0000","last_undegraded":"2024-03-24T00:31:01.186278+0000","last_fullsized":"2024-03-24T00:31:01.186278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:14.595788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047109+0000","last_change":"2024-03-24T00:31:00.045352+0000","last_active":"2024-03-24T00:31:01.047109+0000","last_peered":"2024-03-24T00:31:01.047109+0000","last_clean":"2024-03-24T00:31:01.047109+0000","last_became_active":"2024-03-24T00:31:00.045023+0000","last_became_peered":"2024-03-24T00:31:00.045023+0000","last_unstale":"2024-03-24T00:31:01.047109+0000","last_undegraded":"2024-03-24T00:31:01.047109+0000","last_fullsized":"2024-03-24T00:31:01.047109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:31.480032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060057+0000","last_change":"2024-03-24T00:31:00.061257+0000","last_active":"2024-03-24T00:31:01.060057+0000","last_peered":"2024-03-24T00:31:01.060057+0000","last_clean":"2024-03-24T00:31:01.060057+0000","last_became_active":"2024-03-24T00:31:00.060983+0000","last_became_peered":"2024-03-24T00:31:00.060983+0000","last_unstale":"2024-03-24T00:31:01.060057+0000","last_undegraded":"2024-03-24T00:31:01.060057+0000","last_fullsized":"2024-03-24T00:31:01.060057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909460+0000","last_change":"2024-03-24T00:31:00.054714+0000","last_active":"2024-03-24T00:31:01.909460+0000","last_peered":"2024-03-24T00:31:01.909460+0000","last_clean":"2024-03-24T00:31:01.909460+0000","last_became_active":"2024-03-24T00:31:00.054275+0000","last_became_peered":"2024-03-24T00:31:00.054275+0000","last_unstale":"2024-03-24T00:31:01.909460+0000","last_undegraded":"2024-03-24T00:31:01.909460+0000","last_fullsized":"2024-03-24T00:31:01.909460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:38.408736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909730+0000","last_change":"2024-03-24T00:31:00.063073+0000","last_active":"2024-03-24T00:31:01.909730+0000","last_peered":"2024-03-24T00:31:01.909730+0000","last_clean":"2024-03-24T00:31:01.909730+0000","last_became_active":"2024-03-24T00:31:00.062746+0000","last_became_peered":"2024-03-24T00:31:00.062746+0000","last_unstale":"2024-03-24T00:31:01.909730+0000","last_undegraded":"2024-03-24T00:31:01.909730+0000","last_fullsized":"2024-03-24T00:31:01.909730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:32.847580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909247+0000","last_change":"2024-03-24T00:31:00.059874+0000","last_active":"2024-03-24T00:31:01.909247+0000","last_peered":"2024-03-24T00:31:01.909247+0000","last_clean":"2024-03-24T00:31:01.909247+0000","last_became_active":"2024-03-24T00:31:00.059565+0000","last_became_peered":"2024-03-24T00:31:00.059565+0000","last_unstale":"2024-03-24T00:31:01.909247+0000","last_undegraded":"2024-03-24T00:31:01.909247+0000","last_fullsized":"2024-03-24T00:31:01.909247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:22.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909721+0000","last_change":"2024-03-24T00:31:00.056550+0000","last_active":"2024-03-24T00:31:01.909721+0000","last_peered":"2024-03-24T00:31:01.909721+0000","last_clean":"2024-03-24T00:31:01.909721+0000","last_became_active":"2024-03-24T00:31:00.055816+0000","last_became_peered":"2024-03-24T00:31:00.055816+0000","last_unstale":"2024-03-24T00:31:01.909721+0000","last_undegraded":"2024-03-24T00:31:01.909721+0000","last_fullsized":"2024-03-24T00:31:01.909721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:57:10.621965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909977+0000","last_change":"2024-03-24T00:31:00.059028+0000","last_active":"2024-03-24T00:31:01.909977+0000","last_peered":"2024-03-24T00:31:01.909977+0000","last_clean":"2024-03-24T00:31:01.909977+0000","last_became_active":"2024-03-24T00:31:00.058762+0000","last_became_peered":"2024-03-24T00:31:00.058762+0000","last_unstale":"2024-03-24T00:31:01.909977+0000","last_undegraded":"2024-03-24T00:31:01.909977+0000","last_fullsized":"2024-03-24T00:31:01.909977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060333+0000","last_change":"2024-03-24T00:31:00.054838+0000","last_active":"2024-03-24T00:31:01.060333+0000","last_peered":"2024-03-24T00:31:01.060333+0000","last_clean":"2024-03-24T00:31:01.060333+0000","last_became_active":"2024-03-24T00:31:00.054249+0000","last_became_peered":"2024-03-24T00:31:00.054249+0000","last_unstale":"2024-03-24T00:31:01.060333+0000","last_undegraded":"2024-03-24T00:31:01.060333+0000","last_fullsized":"2024-03-24T00:31:01.060333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047846+0000","last_change":"2024-03-24T00:31:00.037600+0000","last_active":"2024-03-24T00:31:01.047846+0000","last_peered":"2024-03-24T00:31:01.047846+0000","last_clean":"2024-03-24T00:31:01.047846+0000","last_became_active":"2024-03-24T00:31:00.037327+0000","last_became_peered":"2024-03-24T00:31:00.037327+0000","last_unstale":"2024-03-24T00:31:01.047846+0000","last_undegraded":"2024-03-24T00:31:01.047846+0000","last_fullsized":"2024-03-24T00:31:01.047846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909766+0000","last_change":"2024-03-24T00:31:00.061415+0000","last_active":"2024-03-24T00:31:01.909766+0000","last_peered":"2024-03-24T00:31:01.909766+0000","last_clean":"2024-03-24T00:31:01.909766+0000","last_became_active":"2024-03-24T00:31:00.061147+0000","last_became_peered":"2024-03-24T00:31:00.061147+0000","last_unstale":"2024-03-24T00:31:01.909766+0000","last_undegraded":"2024-03-24T00:31:01.909766+0000","last_fullsized":"2024-03-24T00:31:01.909766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909601+0000","last_change":"2024-03-24T00:31:00.049400+0000","last_active":"2024-03-24T00:31:01.909601+0000","last_peered":"2024-03-24T00:31:01.909601+0000","last_clean":"2024-03-24T00:31:01.909601+0000","last_became_active":"2024-03-24T00:31:00.048885+0000","last_became_peered":"2024-03-24T00:31:00.048885+0000","last_unstale":"2024-03-24T00:31:01.909601+0000","last_undegraded":"2024-03-24T00:31:01.909601+0000","last_fullsized":"2024-03-24T00:31:01.909601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:22.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909905+0000","last_change":"2024-03-24T00:31:00.056183+0000","last_active":"2024-03-24T00:31:01.909905+0000","last_peered":"2024-03-24T00:31:01.909905+0000","last_clean":"2024-03-24T00:31:01.909905+0000","last_became_active":"2024-03-24T00:31:00.055346+0000","last_became_peered":"2024-03-24T00:31:00.055346+0000","last_unstale":"2024-03-24T00:31:01.909905+0000","last_undegraded":"2024-03-24T00:31:01.909905+0000","last_fullsized":"2024-03-24T00:31:01.909905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:08.370031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186114+0000","last_change":"2024-03-24T00:31:00.066718+0000","last_active":"2024-03-24T00:31:01.186114+0000","last_peered":"2024-03-24T00:31:01.186114+0000","last_clean":"2024-03-24T00:31:01.186114+0000","last_became_active":"2024-03-24T00:31:00.066330+0000","last_became_peered":"2024-03-24T00:31:00.066330+0000","last_unstale":"2024-03-24T00:31:01.186114+0000","last_undegraded":"2024-03-24T00:31:01.186114+0000","last_fullsized":"2024-03-24T00:31:01.186114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:24.063587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059863+0000","last_change":"2024-03-24T00:31:00.045401+0000","last_active":"2024-03-24T00:31:01.059863+0000","last_peered":"2024-03-24T00:31:01.059863+0000","last_clean":"2024-03-24T00:31:01.059863+0000","last_became_active":"2024-03-24T00:31:00.044163+0000","last_became_peered":"2024-03-24T00:31:00.044163+0000","last_unstale":"2024-03-24T00:31:01.059863+0000","last_undegraded":"2024-03-24T00:31:01.059863+0000","last_fullsized":"2024-03-24T00:31:01.059863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:22.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045365+0000","last_change":"2024-03-24T00:31:00.039890+0000","last_active":"2024-03-24T00:31:01.045365+0000","last_peered":"2024-03-24T00:31:01.045365+0000","last_clean":"2024-03-24T00:31:01.045365+0000","last_became_active":"2024-03-24T00:31:00.038341+0000","last_became_peered":"2024-03-24T00:31:00.038341+0000","last_unstale":"2024-03-24T00:31:01.045365+0000","last_undegraded":"2024-03-24T00:31:01.045365+0000","last_fullsized":"2024-03-24T00:31:01.045365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:26.001997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910135+0000","last_change":"2024-03-24T00:30:59.052547+0000","last_active":"2024-03-24T00:31:01.910135+0000","last_peered":"2024-03-24T00:31:01.910135+0000","last_clean":"2024-03-24T00:31:01.910135+0000","last_became_active":"2024-03-24T00:30:59.052277+0000","last_became_peered":"2024-03-24T00:30:59.052277+0000","last_unstale":"2024-03-24T00:31:01.910135+0000","last_undegraded":"2024-03-24T00:31:01.910135+0000","last_fullsized":"2024-03-24T00:31:01.910135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:28:37.149520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909635+0000","last_change":"2024-03-24T00:31:00.064484+0000","last_active":"2024-03-24T00:31:01.909635+0000","last_peered":"2024-03-24T00:31:01.909635+0000","last_clean":"2024-03-24T00:31:01.909635+0000","last_became_active":"2024-03-24T00:31:00.064211+0000","last_became_peered":"2024-03-24T00:31:00.064211+0000","last_unstale":"2024-03-24T00:31:01.909635+0000","last_undegraded":"2024-03-24T00:31:01.909635+0000","last_fullsized":"2024-03-24T00:31:01.909635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059739+0000","last_change":"2024-03-24T00:30:59.053767+0000","last_active":"2024-03-24T00:31:01.059739+0000","last_peered":"2024-03-24T00:31:01.059739+0000","last_clean":"2024-03-24T00:31:01.059739+0000","last_became_active":"2024-03-24T00:30:59.050782+0000","last_became_peered":"2024-03-24T00:30:59.050782+0000","last_unstale":"2024-03-24T00:31:01.059739+0000","last_undegraded":"2024-03-24T00:31:01.059739+0000","last_fullsized":"2024-03-24T00:31:01.059739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909297+0000","last_change":"2024-03-24T00:30:59.047982+0000","last_active":"2024-03-24T00:31:01.909297+0000","last_peered":"2024-03-24T00:31:01.909297+0000","last_clean":"2024-03-24T00:31:01.909297+0000","last_became_active":"2024-03-24T00:30:59.047680+0000","last_became_peered":"2024-03-24T00:30:59.047680+0000","last_unstale":"2024-03-24T00:31:01.909297+0000","last_undegraded":"2024-03-24T00:31:01.909297+0000","last_fullsized":"2024-03-24T00:31:01.909297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:28.293820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910046+0000","last_change":"2024-03-24T00:31:00.044921+0000","last_active":"2024-03-24T00:31:01.910046+0000","last_peered":"2024-03-24T00:31:01.910046+0000","last_clean":"2024-03-24T00:31:01.910046+0000","last_became_active":"2024-03-24T00:31:00.044583+0000","last_became_peered":"2024-03-24T00:31:00.044583+0000","last_unstale":"2024-03-24T00:31:01.910046+0000","last_undegraded":"2024-03-24T00:31:01.910046+0000","last_fullsized":"2024-03-24T00:31:01.910046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:57:10.621965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909657+0000","last_change":"2024-03-24T00:30:59.058420+0000","last_active":"2024-03-24T00:31:01.909657+0000","last_peered":"2024-03-24T00:31:01.909657+0000","last_clean":"2024-03-24T00:31:01.909657+0000","last_became_active":"2024-03-24T00:30:59.058004+0000","last_became_peered":"2024-03-24T00:30:59.058004+0000","last_unstale":"2024-03-24T00:31:01.909657+0000","last_undegraded":"2024-03-24T00:31:01.909657+0000","last_fullsized":"2024-03-24T00:31:01.909657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047409+0000","last_change":"2024-03-24T00:31:00.043778+0000","last_active":"2024-03-24T00:31:01.047409+0000","last_peered":"2024-03-24T00:31:01.047409+0000","last_clean":"2024-03-24T00:31:01.047409+0000","last_became_active":"2024-03-24T00:31:00.043486+0000","last_became_peered":"2024-03-24T00:31:00.043486+0000","last_unstale":"2024-03-24T00:31:01.047409+0000","last_undegraded":"2024-03-24T00:31:01.047409+0000","last_fullsized":"2024-03-24T00:31:01.047409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060075+0000","last_change":"2024-03-24T00:30:59.057537+0000","last_active":"2024-03-24T00:31:01.060075+0000","last_peered":"2024-03-24T00:31:01.060075+0000","last_clean":"2024-03-24T00:31:01.060075+0000","last_became_active":"2024-03-24T00:30:59.057215+0000","last_became_peered":"2024-03-24T00:30:59.057215+0000","last_unstale":"2024-03-24T00:31:01.060075+0000","last_undegraded":"2024-03-24T00:31:01.060075+0000","last_fullsized":"2024-03-24T00:31:01.060075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:40.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910145+0000","last_change":"2024-03-24T00:31:00.057658+0000","last_active":"2024-03-24T00:31:01.910145+0000","last_peered":"2024-03-24T00:31:01.910145+0000","last_clean":"2024-03-24T00:31:01.910145+0000","last_became_active":"2024-03-24T00:31:00.055543+0000","last_became_peered":"2024-03-24T00:31:00.055543+0000","last_unstale":"2024-03-24T00:31:01.910145+0000","last_undegraded":"2024-03-24T00:31:01.910145+0000","last_fullsized":"2024-03-24T00:31:01.910145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:49.592126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910049+0000","last_change":"2024-03-24T00:30:59.058992+0000","last_active":"2024-03-24T00:31:01.910049+0000","last_peered":"2024-03-24T00:31:01.910049+0000","last_clean":"2024-03-24T00:31:01.910049+0000","last_became_active":"2024-03-24T00:30:59.058546+0000","last_became_peered":"2024-03-24T00:30:59.058546+0000","last_unstale":"2024-03-24T00:31:01.910049+0000","last_undegraded":"2024-03-24T00:31:01.910049+0000","last_fullsized":"2024-03-24T00:31:01.910049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:20.001460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909878+0000","last_change":"2024-03-24T00:31:00.057510+0000","last_active":"2024-03-24T00:31:01.909878+0000","last_peered":"2024-03-24T00:31:01.909878+0000","last_clean":"2024-03-24T00:31:01.909878+0000","last_became_active":"2024-03-24T00:31:00.057227+0000","last_became_peered":"2024-03-24T00:31:00.057227+0000","last_unstale":"2024-03-24T00:31:01.909878+0000","last_undegraded":"2024-03-24T00:31:01.909878+0000","last_fullsized":"2024-03-24T00:31:01.909878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047997+0000","last_change":"2024-03-24T00:30:59.055938+0000","last_active":"2024-03-24T00:31:01.047997+0000","last_peered":"2024-03-24T00:31:01.047997+0000","last_clean":"2024-03-24T00:31:01.047997+0000","last_became_active":"2024-03-24T00:30:59.055734+0000","last_became_peered":"2024-03-24T00:30:59.055734+0000","last_unstale":"2024-03-24T00:31:01.047997+0000","last_undegraded":"2024-03-24T00:31:01.047997+0000","last_fullsized":"2024-03-24T00:31:01.047997+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:39.280192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045606+0000","last_change":"2024-03-24T00:31:00.053717+0000","last_active":"2024-03-24T00:31:01.045606+0000","last_peered":"2024-03-24T00:31:01.045606+0000","last_clean":"2024-03-24T00:31:01.045606+0000","last_became_active":"2024-03-24T00:31:00.053312+0000","last_became_peered":"2024-03-24T00:31:00.053312+0000","last_unstale":"2024-03-24T00:31:01.045606+0000","last_undegraded":"2024-03-24T00:31:01.045606+0000","last_fullsized":"2024-03-24T00:31:01.045606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:40.286678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060097+0000","last_change":"2024-03-24T00:30:59.050368+0000","last_active":"2024-03-24T00:31:01.060097+0000","last_peered":"2024-03-24T00:31:01.060097+0000","last_clean":"2024-03-24T00:31:01.060097+0000","last_became_active":"2024-03-24T00:30:59.050062+0000","last_became_peered":"2024-03-24T00:30:59.050062+0000","last_unstale":"2024-03-24T00:31:01.060097+0000","last_undegraded":"2024-03-24T00:31:01.060097+0000","last_fullsized":"2024-03-24T00:31:01.060097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046667+0000","last_change":"2024-03-24T00:31:00.041643+0000","last_active":"2024-03-24T00:31:01.046667+0000","last_peered":"2024-03-24T00:31:01.046667+0000","last_clean":"2024-03-24T00:31:01.046667+0000","last_became_active":"2024-03-24T00:31:00.041387+0000","last_became_peered":"2024-03-24T00:31:00.041387+0000","last_unstale":"2024-03-24T00:31:01.046667+0000","last_undegraded":"2024-03-24T00:31:01.046667+0000","last_fullsized":"2024-03-24T00:31:01.046667+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:34:52.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909659+0000","last_change":"2024-03-24T00:30:59.054246+0000","last_active":"2024-03-24T00:31:01.909659+0000","last_peered":"2024-03-24T00:31:01.909659+0000","last_clean":"2024-03-24T00:31:01.909659+0000","last_became_active":"2024-03-24T00:30:59.053818+0000","last_became_peered":"2024-03-24T00:30:59.053818+0000","last_unstale":"2024-03-24T00:31:01.909659+0000","last_undegraded":"2024-03-24T00:31:01.909659+0000","last_fullsized":"2024-03-24T00:31:01.909659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:28.293820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909855+0000","last_change":"2024-03-24T00:31:00.054481+0000","last_active":"2024-03-24T00:31:01.909855+0000","last_peered":"2024-03-24T00:31:01.909855+0000","last_clean":"2024-03-24T00:31:01.909855+0000","last_became_active":"2024-03-24T00:31:00.054255+0000","last_became_peered":"2024-03-24T00:31:00.054255+0000","last_unstale":"2024-03-24T00:31:01.909855+0000","last_undegraded":"2024-03-24T00:31:01.909855+0000","last_fullsized":"2024-03-24T00:31:01.909855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910033+0000","last_change":"2024-03-24T00:30:59.059529+0000","last_active":"2024-03-24T00:31:01.910033+0000","last_peered":"2024-03-24T00:31:01.910033+0000","last_clean":"2024-03-24T00:31:01.910033+0000","last_became_active":"2024-03-24T00:30:59.058947+0000","last_became_peered":"2024-03-24T00:30:59.058947+0000","last_unstale":"2024-03-24T00:31:01.910033+0000","last_undegraded":"2024-03-24T00:31:01.910033+0000","last_fullsized":"2024-03-24T00:31:01.910033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:39.280192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046876+0000","last_change":"2024-03-24T00:31:00.061372+0000","last_active":"2024-03-24T00:31:01.046876+0000","last_peered":"2024-03-24T00:31:01.046876+0000","last_clean":"2024-03-24T00:31:01.046876+0000","last_became_active":"2024-03-24T00:31:00.061094+0000","last_became_peered":"2024-03-24T00:31:00.061094+0000","last_unstale":"2024-03-24T00:31:01.046876+0000","last_undegraded":"2024-03-24T00:31:01.046876+0000","last_fullsized":"2024-03-24T00:31:01.046876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:38.650664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059711+0000","last_change":"2024-03-24T00:30:59.046862+0000","last_active":"2024-03-24T00:31:01.059711+0000","last_peered":"2024-03-24T00:31:01.059711+0000","last_clean":"2024-03-24T00:31:01.059711+0000","last_became_active":"2024-03-24T00:30:59.046575+0000","last_became_peered":"2024-03-24T00:30:59.046575+0000","last_unstale":"2024-03-24T00:31:01.059711+0000","last_undegraded":"2024-03-24T00:31:01.059711+0000","last_fullsized":"2024-03-24T00:31:01.059711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:39.280192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059910+0000","last_change":"2024-03-24T00:31:00.044599+0000","last_active":"2024-03-24T00:31:01.059910+0000","last_peered":"2024-03-24T00:31:01.059910+0000","last_clean":"2024-03-24T00:31:01.059910+0000","last_became_active":"2024-03-24T00:31:00.043498+0000","last_became_peered":"2024-03-24T00:31:00.043498+0000","last_unstale":"2024-03-24T00:31:01.059910+0000","last_undegraded":"2024-03-24T00:31:01.059910+0000","last_fullsized":"2024-03-24T00:31:01.059910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.185968+0000","last_change":"2024-03-24T00:30:59.039042+0000","last_active":"2024-03-24T00:31:01.185968+0000","last_peered":"2024-03-24T00:31:01.185968+0000","last_clean":"2024-03-24T00:31:01.185968+0000","last_became_active":"2024-03-24T00:30:59.038744+0000","last_became_peered":"2024-03-24T00:30:59.038744+0000","last_unstale":"2024-03-24T00:31:01.185968+0000","last_undegraded":"2024-03-24T00:31:01.185968+0000","last_fullsized":"2024-03-24T00:31:01.185968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:20.001460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045531+0000","last_change":"2024-03-24T00:31:00.047666+0000","last_active":"2024-03-24T00:31:01.045531+0000","last_peered":"2024-03-24T00:31:01.045531+0000","last_clean":"2024-03-24T00:31:01.045531+0000","last_became_active":"2024-03-24T00:31:00.047379+0000","last_became_peered":"2024-03-24T00:31:00.047379+0000","last_unstale":"2024-03-24T00:31:01.045531+0000","last_undegraded":"2024-03-24T00:31:01.045531+0000","last_fullsized":"2024-03-24T00:31:01.045531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:08.370031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045647+0000","last_change":"2024-03-24T00:30:59.043700+0000","last_active":"2024-03-24T00:31:01.045647+0000","last_peered":"2024-03-24T00:31:01.045647+0000","last_clean":"2024-03-24T00:31:01.045647+0000","last_became_active":"2024-03-24T00:30:59.043305+0000","last_became_peered":"2024-03-24T00:30:59.043305+0000","last_unstale":"2024-03-24T00:31:01.045647+0000","last_undegraded":"2024-03-24T00:31:01.045647+0000","last_fullsized":"2024-03-24T00:31:01.045647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:16.392363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046650+0000","last_change":"2024-03-24T00:31:00.048570+0000","last_active":"2024-03-24T00:31:01.046650+0000","last_peered":"2024-03-24T00:31:01.046650+0000","last_clean":"2024-03-24T00:31:01.046650+0000","last_became_active":"2024-03-24T00:31:00.048162+0000","last_became_peered":"2024-03-24T00:31:00.048162+0000","last_unstale":"2024-03-24T00:31:01.046650+0000","last_undegraded":"2024-03-24T00:31:01.046650+0000","last_fullsized":"2024-03-24T00:31:01.046650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:17.380183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186240+0000","last_change":"2024-03-24T00:30:59.041080+0000","last_active":"2024-03-24T00:31:01.186240+0000","last_peered":"2024-03-24T00:31:01.186240+0000","last_clean":"2024-03-24T00:31:01.186240+0000","last_became_active":"2024-03-24T00:30:59.040599+0000","last_became_peered":"2024-03-24T00:30:59.040599+0000","last_unstale":"2024-03-24T00:31:01.186240+0000","last_undegraded":"2024-03-24T00:31:01.186240+0000","last_fullsized":"2024-03-24T00:31:01.186240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047022+0000","last_change":"2024-03-24T00:31:00.060654+0000","last_active":"2024-03-24T00:31:01.047022+0000","last_peered":"2024-03-24T00:31:01.047022+0000","last_clean":"2024-03-24T00:31:01.047022+0000","last_became_active":"2024-03-24T00:31:00.060301+0000","last_became_peered":"2024-03-24T00:31:00.060301+0000","last_unstale":"2024-03-24T00:31:01.047022+0000","last_undegraded":"2024-03-24T00:31:01.047022+0000","last_fullsized":"2024-03-24T00:31:01.047022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:41.152930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047876+0000","last_change":"2024-03-24T00:30:59.045891+0000","last_active":"2024-03-24T00:31:01.047876+0000","last_peered":"2024-03-24T00:31:01.047876+0000","last_clean":"2024-03-24T00:31:01.047876+0000","last_became_active":"2024-03-24T00:30:59.044107+0000","last_became_peered":"2024-03-24T00:30:59.044107+0000","last_unstale":"2024-03-24T00:31:01.047876+0000","last_undegraded":"2024-03-24T00:31:01.047876+0000","last_fullsized":"2024-03-24T00:31:01.047876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:16.613800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910353+0000","last_change":"2024-03-24T00:31:00.056640+0000","last_active":"2024-03-24T00:31:01.910353+0000","last_peered":"2024-03-24T00:31:01.910353+0000","last_clean":"2024-03-24T00:31:01.910353+0000","last_became_active":"2024-03-24T00:31:00.055891+0000","last_became_peered":"2024-03-24T00:31:00.055891+0000","last_unstale":"2024-03-24T00:31:01.910353+0000","last_undegraded":"2024-03-24T00:31:01.910353+0000","last_fullsized":"2024-03-24T00:31:01.910353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:17.791692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186119+0000","last_change":"2024-03-24T00:30:59.058994+0000","last_active":"2024-03-24T00:31:01.186119+0000","last_peered":"2024-03-24T00:31:01.186119+0000","last_clean":"2024-03-24T00:31:01.186119+0000","last_became_active":"2024-03-24T00:30:59.058639+0000","last_became_peered":"2024-03-24T00:30:59.058639+0000","last_unstale":"2024-03-24T00:31:01.186119+0000","last_undegraded":"2024-03-24T00:31:01.186119+0000","last_fullsized":"2024-03-24T00:31:01.186119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046886+0000","last_change":"2024-03-24T00:31:00.059336+0000","last_active":"2024-03-24T00:31:01.046886+0000","last_peered":"2024-03-24T00:31:01.046886+0000","last_clean":"2024-03-24T00:31:01.046886+0000","last_became_active":"2024-03-24T00:31:00.059071+0000","last_became_peered":"2024-03-24T00:31:00.059071+0000","last_unstale":"2024-03-24T00:31:01.046886+0000","last_undegraded":"2024-03-24T00:31:01.046886+0000","last_fullsized":"2024-03-24T00:31:01.046886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:10.867603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046267+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:31:01.046267+0000","last_peered":"2024-03-24T00:31:01.046267+0000","last_clean":"2024-03-24T00:31:01.046267+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:31:01.046267+0000","last_undegraded":"2024-03-24T00:31:01.046267+0000","last_fullsized":"2024-03-24T00:31:01.046267+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909596+0000","last_change":"2024-03-24T00:30:59.059377+0000","last_active":"2024-03-24T00:31:01.909596+0000","last_peered":"2024-03-24T00:31:01.909596+0000","last_clean":"2024-03-24T00:31:01.909596+0000","last_became_active":"2024-03-24T00:30:59.058424+0000","last_became_peered":"2024-03-24T00:30:59.058424+0000","last_unstale":"2024-03-24T00:31:01.909596+0000","last_undegraded":"2024-03-24T00:31:01.909596+0000","last_fullsized":"2024-03-24T00:31:01.909596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909334+0000","last_change":"2024-03-24T00:31:00.058762+0000","last_active":"2024-03-24T00:31:01.909334+0000","last_peered":"2024-03-24T00:31:01.909334+0000","last_clean":"2024-03-24T00:31:01.909334+0000","last_became_active":"2024-03-24T00:31:00.058509+0000","last_became_peered":"2024-03-24T00:31:00.058509+0000","last_unstale":"2024-03-24T00:31:01.909334+0000","last_undegraded":"2024-03-24T00:31:01.909334+0000","last_fullsized":"2024-03-24T00:31:01.909334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046137+0000","last_change":"2024-03-24T00:30:59.050656+0000","last_active":"2024-03-24T00:31:01.046137+0000","last_peered":"2024-03-24T00:31:01.046137+0000","last_clean":"2024-03-24T00:31:01.046137+0000","last_became_active":"2024-03-24T00:30:59.050364+0000","last_became_peered":"2024-03-24T00:30:59.050364+0000","last_unstale":"2024-03-24T00:31:01.046137+0000","last_undegraded":"2024-03-24T00:31:01.046137+0000","last_fullsized":"2024-03-24T00:31:01.046137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:23.150413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909215+0000","last_change":"2024-03-24T00:31:00.059875+0000","last_active":"2024-03-24T00:31:01.909215+0000","last_peered":"2024-03-24T00:31:01.909215+0000","last_clean":"2024-03-24T00:31:01.909215+0000","last_became_active":"2024-03-24T00:31:00.059618+0000","last_became_peered":"2024-03-24T00:31:00.059618+0000","last_unstale":"2024-03-24T00:31:01.909215+0000","last_undegraded":"2024-03-24T00:31:01.909215+0000","last_fullsized":"2024-03-24T00:31:01.909215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:34:42.663552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186071+0000","last_change":"2024-03-24T00:30:59.040845+0000","last_active":"2024-03-24T00:31:01.186071+0000","last_peered":"2024-03-24T00:31:01.186071+0000","last_clean":"2024-03-24T00:31:01.186071+0000","last_became_active":"2024-03-24T00:30:59.040519+0000","last_became_peered":"2024-03-24T00:30:59.040519+0000","last_unstale":"2024-03-24T00:31:01.186071+0000","last_undegraded":"2024-03-24T00:31:01.186071+0000","last_fullsized":"2024-03-24T00:31:01.186071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:28.293820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047374+0000","last_change":"2024-03-24T00:31:00.041015+0000","last_active":"2024-03-24T00:31:01.047374+0000","last_peered":"2024-03-24T00:31:01.047374+0000","last_clean":"2024-03-24T00:31:01.047374+0000","last_became_active":"2024-03-24T00:31:00.040568+0000","last_became_peered":"2024-03-24T00:31:00.040568+0000","last_unstale":"2024-03-24T00:31:01.047374+0000","last_undegraded":"2024-03-24T00:31:01.047374+0000","last_fullsized":"2024-03-24T00:31:01.047374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910220+0000","last_change":"2024-03-24T00:30:59.061747+0000","last_active":"2024-03-24T00:31:01.910220+0000","last_peered":"2024-03-24T00:31:01.910220+0000","last_clean":"2024-03-24T00:31:01.910220+0000","last_became_active":"2024-03-24T00:30:59.059297+0000","last_became_peered":"2024-03-24T00:30:59.059297+0000","last_unstale":"2024-03-24T00:31:01.910220+0000","last_undegraded":"2024-03-24T00:31:01.910220+0000","last_fullsized":"2024-03-24T00:31:01.910220+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:37.099548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045947+0000","last_change":"2024-03-24T00:31:00.050644+0000","last_active":"2024-03-24T00:31:01.045947+0000","last_peered":"2024-03-24T00:31:01.045947+0000","last_clean":"2024-03-24T00:31:01.045947+0000","last_became_active":"2024-03-24T00:31:00.050376+0000","last_became_peered":"2024-03-24T00:31:00.050376+0000","last_unstale":"2024-03-24T00:31:01.045947+0000","last_undegraded":"2024-03-24T00:31:01.045947+0000","last_fullsized":"2024-03-24T00:31:01.045947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:17.620286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047738+0000","last_change":"2024-03-24T00:30:59.045279+0000","last_active":"2024-03-24T00:31:01.047738+0000","last_peered":"2024-03-24T00:31:01.047738+0000","last_clean":"2024-03-24T00:31:01.047738+0000","last_became_active":"2024-03-24T00:30:59.044864+0000","last_became_peered":"2024-03-24T00:30:59.044864+0000","last_unstale":"2024-03-24T00:31:01.047738+0000","last_undegraded":"2024-03-24T00:31:01.047738+0000","last_fullsized":"2024-03-24T00:31:01.047738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:37.099548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910242+0000","last_change":"2024-03-24T00:31:00.056075+0000","last_active":"2024-03-24T00:31:01.910242+0000","last_peered":"2024-03-24T00:31:01.910242+0000","last_clean":"2024-03-24T00:31:01.910242+0000","last_became_active":"2024-03-24T00:31:00.055694+0000","last_became_peered":"2024-03-24T00:31:00.055694+0000","last_unstale":"2024-03-24T00:31:01.910242+0000","last_undegraded":"2024-03-24T00:31:01.910242+0000","last_fullsized":"2024-03-24T00:31:01.910242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:38.650664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047000+0000","last_change":"2024-03-24T00:30:59.048260+0000","last_active":"2024-03-24T00:31:01.047000+0000","last_peered":"2024-03-24T00:31:01.047000+0000","last_clean":"2024-03-24T00:31:01.047000+0000","last_became_active":"2024-03-24T00:30:59.047943+0000","last_became_peered":"2024-03-24T00:30:59.047943+0000","last_unstale":"2024-03-24T00:31:01.047000+0000","last_undegraded":"2024-03-24T00:31:01.047000+0000","last_fullsized":"2024-03-24T00:31:01.047000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:13.589302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047659+0000","last_change":"2024-03-24T00:31:00.041416+0000","last_active":"2024-03-24T00:31:01.047659+0000","last_peered":"2024-03-24T00:31:01.047659+0000","last_clean":"2024-03-24T00:31:01.047659+0000","last_became_active":"2024-03-24T00:31:00.041096+0000","last_became_peered":"2024-03-24T00:31:00.041096+0000","last_unstale":"2024-03-24T00:31:01.047659+0000","last_undegraded":"2024-03-24T00:31:01.047659+0000","last_fullsized":"2024-03-24T00:31:01.047659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047950+0000","last_change":"2024-03-24T00:30:59.046901+0000","last_active":"2024-03-24T00:31:01.047950+0000","last_peered":"2024-03-24T00:31:01.047950+0000","last_clean":"2024-03-24T00:31:01.047950+0000","last_became_active":"2024-03-24T00:30:59.046619+0000","last_became_peered":"2024-03-24T00:30:59.046619+0000","last_unstale":"2024-03-24T00:31:01.047950+0000","last_undegraded":"2024-03-24T00:31:01.047950+0000","last_fullsized":"2024-03-24T00:31:01.047950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:40.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909976+0000","last_change":"2024-03-24T00:31:00.047744+0000","last_active":"2024-03-24T00:31:01.909976+0000","last_peered":"2024-03-24T00:31:01.909976+0000","last_clean":"2024-03-24T00:31:01.909976+0000","last_became_active":"2024-03-24T00:31:00.047453+0000","last_became_peered":"2024-03-24T00:31:00.047453+0000","last_unstale":"2024-03-24T00:31:01.909976+0000","last_undegraded":"2024-03-24T00:31:01.909976+0000","last_fullsized":"2024-03-24T00:31:01.909976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060070+0000","last_change":"2024-03-24T00:30:59.047379+0000","last_active":"2024-03-24T00:31:01.060070+0000","last_peered":"2024-03-24T00:31:01.060070+0000","last_clean":"2024-03-24T00:31:01.060070+0000","last_became_active":"2024-03-24T00:30:59.045528+0000","last_became_peered":"2024-03-24T00:30:59.045528+0000","last_unstale":"2024-03-24T00:31:01.060070+0000","last_undegraded":"2024-03-24T00:31:01.060070+0000","last_fullsized":"2024-03-24T00:31:01.060070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:23.057101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909624+0000","last_change":"2024-03-24T00:31:00.050676+0000","last_active":"2024-03-24T00:31:01.909624+0000","last_peered":"2024-03-24T00:31:01.909624+0000","last_clean":"2024-03-24T00:31:01.909624+0000","last_became_active":"2024-03-24T00:31:00.050421+0000","last_became_peered":"2024-03-24T00:31:00.050421+0000","last_unstale":"2024-03-24T00:31:01.909624+0000","last_undegraded":"2024-03-24T00:31:01.909624+0000","last_fullsized":"2024-03-24T00:31:01.909624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045080+0000","last_change":"2024-03-24T00:30:59.047814+0000","last_active":"2024-03-24T00:31:01.045080+0000","last_peered":"2024-03-24T00:31:01.045080+0000","last_clean":"2024-03-24T00:31:01.045080+0000","last_became_active":"2024-03-24T00:30:59.047497+0000","last_became_peered":"2024-03-24T00:30:59.047497+0000","last_unstale":"2024-03-24T00:31:01.045080+0000","last_undegraded":"2024-03-24T00:31:01.045080+0000","last_fullsized":"2024-03-24T00:31:01.045080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:37.099548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046279+0000","last_change":"2024-03-24T00:31:00.043642+0000","last_active":"2024-03-24T00:31:01.046279+0000","last_peered":"2024-03-24T00:31:01.046279+0000","last_clean":"2024-03-24T00:31:01.046279+0000","last_became_active":"2024-03-24T00:31:00.043334+0000","last_became_peered":"2024-03-24T00:31:00.043334+0000","last_unstale":"2024-03-24T00:31:01.046279+0000","last_undegraded":"2024-03-24T00:31:01.046279+0000","last_fullsized":"2024-03-24T00:31:01.046279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:41.254416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047463+0000","last_change":"2024-03-24T00:30:59.044547+0000","last_active":"2024-03-24T00:31:01.047463+0000","last_peered":"2024-03-24T00:31:01.047463+0000","last_clean":"2024-03-24T00:31:01.047463+0000","last_became_active":"2024-03-24T00:30:59.044240+0000","last_became_peered":"2024-03-24T00:30:59.044240+0000","last_unstale":"2024-03-24T00:31:01.047463+0000","last_undegraded":"2024-03-24T00:31:01.047463+0000","last_fullsized":"2024-03-24T00:31:01.047463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910154+0000","last_change":"2024-03-24T00:31:00.043313+0000","last_active":"2024-03-24T00:31:01.910154+0000","last_peered":"2024-03-24T00:31:01.910154+0000","last_clean":"2024-03-24T00:31:01.910154+0000","last_became_active":"2024-03-24T00:31:00.043030+0000","last_became_peered":"2024-03-24T00:31:00.043030+0000","last_unstale":"2024-03-24T00:31:01.910154+0000","last_undegraded":"2024-03-24T00:31:01.910154+0000","last_fullsized":"2024-03-24T00:31:01.910154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:32.847580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047581+0000","last_change":"2024-03-24T00:30:59.045395+0000","last_active":"2024-03-24T00:31:01.047581+0000","last_peered":"2024-03-24T00:31:01.047581+0000","last_clean":"2024-03-24T00:31:01.047581+0000","last_became_active":"2024-03-24T00:30:59.044888+0000","last_became_peered":"2024-03-24T00:30:59.044888+0000","last_unstale":"2024-03-24T00:31:01.047581+0000","last_undegraded":"2024-03-24T00:31:01.047581+0000","last_fullsized":"2024-03-24T00:31:01.047581+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910336+0000","last_change":"2024-03-24T00:31:00.052157+0000","last_active":"2024-03-24T00:31:01.910336+0000","last_peered":"2024-03-24T00:31:01.910336+0000","last_clean":"2024-03-24T00:31:01.910336+0000","last_became_active":"2024-03-24T00:31:00.051147+0000","last_became_peered":"2024-03-24T00:31:00.051147+0000","last_unstale":"2024-03-24T00:31:01.910336+0000","last_undegraded":"2024-03-24T00:31:01.910336+0000","last_fullsized":"2024-03-24T00:31:01.910336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:08.370031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046873+0000","last_change":"2024-03-24T00:30:59.053443+0000","last_active":"2024-03-24T00:31:01.046873+0000","last_peered":"2024-03-24T00:31:01.046873+0000","last_clean":"2024-03-24T00:31:01.046873+0000","last_became_active":"2024-03-24T00:30:59.053111+0000","last_became_peered":"2024-03-24T00:30:59.053111+0000","last_unstale":"2024-03-24T00:31:01.046873+0000","last_undegraded":"2024-03-24T00:31:01.046873+0000","last_fullsized":"2024-03-24T00:31:01.046873+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:40.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059908+0000","last_change":"2024-03-24T00:31:00.061892+0000","last_active":"2024-03-24T00:31:01.059908+0000","last_peered":"2024-03-24T00:31:01.059908+0000","last_clean":"2024-03-24T00:31:01.059908+0000","last_became_active":"2024-03-24T00:31:00.061567+0000","last_became_peered":"2024-03-24T00:31:00.061567+0000","last_unstale":"2024-03-24T00:31:01.059908+0000","last_undegraded":"2024-03-24T00:31:01.059908+0000","last_fullsized":"2024-03-24T00:31:01.059908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910099+0000","last_change":"2024-03-24T00:31:00.069008+0000","last_active":"2024-03-24T00:31:01.910099+0000","last_peered":"2024-03-24T00:31:01.910099+0000","last_clean":"2024-03-24T00:31:01.910099+0000","last_became_active":"2024-03-24T00:31:00.065217+0000","last_became_peered":"2024-03-24T00:31:00.065217+0000","last_unstale":"2024-03-24T00:31:01.910099+0000","last_undegraded":"2024-03-24T00:31:01.910099+0000","last_fullsized":"2024-03-24T00:31:01.910099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:34:52.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046539+0000","last_change":"2024-03-24T00:30:59.060493+0000","last_active":"2024-03-24T00:31:01.046539+0000","last_peered":"2024-03-24T00:31:01.046539+0000","last_clean":"2024-03-24T00:31:01.046539+0000","last_became_active":"2024-03-24T00:30:59.060208+0000","last_became_peered":"2024-03-24T00:30:59.060208+0000","last_unstale":"2024-03-24T00:31:01.046539+0000","last_undegraded":"2024-03-24T00:31:01.046539+0000","last_fullsized":"2024-03-24T00:31:01.046539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:16.613800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910297+0000","last_change":"2024-03-24T00:31:00.071223+0000","last_active":"2024-03-24T00:31:01.910297+0000","last_peered":"2024-03-24T00:31:01.910297+0000","last_clean":"2024-03-24T00:31:01.910297+0000","last_became_active":"2024-03-24T00:31:00.069882+0000","last_became_peered":"2024-03-24T00:31:00.069882+0000","last_unstale":"2024-03-24T00:31:01.910297+0000","last_undegraded":"2024-03-24T00:31:01.910297+0000","last_fullsized":"2024-03-24T00:31:01.910297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:09.487064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046833+0000","last_change":"2024-03-24T00:30:59.052020+0000","last_active":"2024-03-24T00:31:01.046833+0000","last_peered":"2024-03-24T00:31:01.046833+0000","last_clean":"2024-03-24T00:31:01.046833+0000","last_became_active":"2024-03-24T00:30:59.049277+0000","last_became_peered":"2024-03-24T00:30:59.049277+0000","last_unstale":"2024-03-24T00:31:01.046833+0000","last_undegraded":"2024-03-24T00:31:01.046833+0000","last_fullsized":"2024-03-24T00:31:01.046833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:23.057101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909752+0000","last_change":"2024-03-24T00:31:00.072313+0000","last_active":"2024-03-24T00:31:01.909752+0000","last_peered":"2024-03-24T00:31:01.909752+0000","last_clean":"2024-03-24T00:31:01.909752+0000","last_became_active":"2024-03-24T00:31:00.071692+0000","last_became_peered":"2024-03-24T00:31:00.071692+0000","last_unstale":"2024-03-24T00:31:01.909752+0000","last_undegraded":"2024-03-24T00:31:01.909752+0000","last_fullsized":"2024-03-24T00:31:01.909752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910030+0000","last_change":"2024-03-24T00:30:59.045778+0000","last_active":"2024-03-24T00:31:01.910030+0000","last_peered":"2024-03-24T00:31:01.910030+0000","last_clean":"2024-03-24T00:31:01.910030+0000","last_became_active":"2024-03-24T00:30:59.045515+0000","last_became_peered":"2024-03-24T00:30:59.045515+0000","last_unstale":"2024-03-24T00:31:01.910030+0000","last_undegraded":"2024-03-24T00:31:01.910030+0000","last_fullsized":"2024-03-24T00:31:01.910030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:20.001460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909974+0000","last_change":"2024-03-24T00:31:00.047507+0000","last_active":"2024-03-24T00:31:01.909974+0000","last_peered":"2024-03-24T00:31:01.909974+0000","last_clean":"2024-03-24T00:31:01.909974+0000","last_became_active":"2024-03-24T00:31:00.047245+0000","last_became_peered":"2024-03-24T00:31:00.047245+0000","last_unstale":"2024-03-24T00:31:01.909974+0000","last_undegraded":"2024-03-24T00:31:01.909974+0000","last_fullsized":"2024-03-24T00:31:01.909974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909924+0000","last_change":"2024-03-24T00:30:59.053442+0000","last_active":"2024-03-24T00:31:01.909924+0000","last_peered":"2024-03-24T00:31:01.909924+0000","last_clean":"2024-03-24T00:31:01.909924+0000","last_became_active":"2024-03-24T00:30:59.053159+0000","last_became_peered":"2024-03-24T00:30:59.053159+0000","last_unstale":"2024-03-24T00:31:01.909924+0000","last_undegraded":"2024-03-24T00:31:01.909924+0000","last_fullsized":"2024-03-24T00:31:01.909924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:28:37.149520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:31:06.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:31:06.809 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:31:08.759 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:31:08.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8da0000640 1 Processor -- start 2024-03-24T00:31:08.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8da0000640 1 -- start start 2024-03-24T00:31:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8da0000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8d980ccfa0 0x7f8d980cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8da0000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d98070f60 con 0x7f8d9806e620 2024-03-24T00:31:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8da0000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d980b3620 con 0x7f8d980b46c0 2024-03-24T00:31:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8da0000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d980b3900 con 0x7f8d980ccfa0 2024-03-24T00:31:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8d9d600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8d9806e620 0x7f8d9806ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57840/0 (socket says 172.21.15.40:57840) 2024-03-24T00:31:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8d9d600640 1 -- 172.21.15.40:0/153738961 learned_addr learned my addr 172.21.15.40:0/153738961 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:08.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8d9e000640 1 --2- 172.21.15.40:0/153738961 >> v2:172.21.15.40:3300/0 conn(0x7f8d980ccfa0 0x7f8d980cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8d97e00640 1 -- 172.21.15.40:0/153738961 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460145190 0 0) 0x7f8d98070f60 con 0x7f8d9806e620 2024-03-24T00:31:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8d97e00640 1 -- 172.21.15.40:0/153738961 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d80005740 con 0x7f8d9806e620 2024-03-24T00:31:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8d9e000640 1 -- 172.21.15.40:0/153738961 >> v1:172.21.15.53:6789/0 conn(0x7f8d980b46c0 legacy=0x7f8d980b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8d9e000640 1 -- 172.21.15.40:0/153738961 >> v1:172.21.15.40:6789/0 conn(0x7f8d9806e620 legacy=0x7f8d9806ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:08.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.898+0000 7f8d9e000640 1 -- 172.21.15.40:0/153738961 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d980b3be0 con 0x7f8d980ccfa0 2024-03-24T00:31:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.902+0000 7f8d9e000640 1 --2- 172.21.15.40:0/153738961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d980ccfa0 0x7f8d980cd3b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009d90 tx=0x7f8d9001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ab2d2b0f98bb6b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.902+0000 7f8d97e00640 1 -- 172.21.15.40:0/153738961 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d900b2070 con 0x7f8d980ccfa0 2024-03-24T00:31:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.902+0000 7f8d97e00640 1 -- 172.21.15.40:0/153738961 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d900a5780 con 0x7f8d980ccfa0 2024-03-24T00:31:08.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.902+0000 7f8d97e00640 1 -- 172.21.15.40:0/153738961 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d900a5b50 con 0x7f8d980ccfa0 2024-03-24T00:31:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 -- 172.21.15.40:0/153738961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d980ccfa0 msgr2=0x7f8d980cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 --2- 172.21.15.40:0/153738961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d980ccfa0 0x7f8d980cd3b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8d90009d90 tx=0x7f8d9001c8b0 comp rx=0 tx=0).stop 2024-03-24T00:31:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 -- 172.21.15.40:0/153738961 shutdown_connections 2024-03-24T00:31:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 --2- 172.21.15.40:0/153738961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d980ccfa0 0x7f8d980cd3b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 -- 172.21.15.40:0/153738961 >> 172.21.15.40:0/153738961 conn(0x7f8d980b17b0 msgr2=0x7f8d980b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 -- 172.21.15.40:0/153738961 shutdown_connections 2024-03-24T00:31:08.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 -- 172.21.15.40:0/153738961 wait complete. 2024-03-24T00:31:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 Processor -- start 2024-03-24T00:31:08.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 -- start start 2024-03-24T00:31:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9806e620 0x7f8d981434b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:08.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d980b46c0 0x7f8d9815b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d980ccfa0 0x7f8d98159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d98144920 con 0x7f8d980ccfa0 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d980b31c0 con 0x7f8d980b46c0 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8da0000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d98142e40 con 0x7f8d9806e620 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8d9e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d980ccfa0 0x7f8d98159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8d9e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d980ccfa0 0x7f8d98159f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46888/0 (socket says 172.21.15.40:46888) 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8d9e000640 1 -- 172.21.15.40:0/772850916 learned_addr learned my addr 172.21.15.40:0/772850916 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8d9cc00640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d980b46c0 0x7f8d9815b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8d9d600640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9806e620 0x7f8d981434b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8d9e000640 1 -- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9806e620 msgr2=0x7f8d981434b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8d9e000640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9806e620 0x7f8d981434b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8d9e000640 1 -- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d980b46c0 msgr2=0x7f8d9815b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8d9e000640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d980b46c0 0x7f8d9815b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:08.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.906+0000 7f8d9e000640 1 -- 172.21.15.40:0/772850916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d981420c0 con 0x7f8d980ccfa0 2024-03-24T00:31:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.910+0000 7f8d9cc00640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d980b46c0 0x7f8d9815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.910+0000 7f8d9e000640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d980ccfa0 0x7f8d98159f50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8d9001a4e0 tx=0x7f8d90019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.910+0000 7f8d95600640 1 -- 172.21.15.40:0/772850916 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d900b2050 con 0x7f8d980ccfa0 2024-03-24T00:31:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.910+0000 7f8d95600640 1 -- 172.21.15.40:0/772850916 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d900be070 con 0x7f8d980ccfa0 2024-03-24T00:31:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.910+0000 7f8d95600640 1 -- 172.21.15.40:0/772850916 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d900ad850 con 0x7f8d980ccfa0 2024-03-24T00:31:08.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.910+0000 7f8da0000640 1 -- 172.21.15.40:0/772850916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9815a490 con 0x7f8d980ccfa0 2024-03-24T00:31:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.910+0000 7f8da0000640 1 -- 172.21.15.40:0/772850916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d98142640 con 0x7f8d980ccfa0 2024-03-24T00:31:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.910+0000 7f8da0000640 1 -- 172.21.15.40:0/772850916 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d98068170 con 0x7f8d980ccfa0 2024-03-24T00:31:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.910+0000 7f8d95600640 1 -- 172.21.15.40:0/772850916 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8d900ad9f0 con 0x7f8d980ccfa0 2024-03-24T00:31:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.910+0000 7f8d95600640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8d78067690 0x7f8d78069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:08.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.910+0000 7f8d95600640 1 -- 172.21.15.40:0/772850916 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d90120270 con 0x7f8d980ccfa0 2024-03-24T00:31:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.914+0000 7f8d9d600640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8d78067690 0x7f8d78069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:08.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.914+0000 7f8d9d600640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8d78067690 0x7f8d78069b50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c005890 tx=0x7f8d8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:08.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:08.918+0000 7f8d95600640 1 -- 172.21.15.40:0/772850916 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8d900c7020 con 0x7f8d980ccfa0 2024-03-24T00:31:09.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.126+0000 7f8da0000640 1 -- 172.21.15.40:0/772850916 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d98072f50 con 0x7f8d78067690 2024-03-24T00:31:09.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.130+0000 7f8d95600640 1 -- 172.21.15.40:0/772850916 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251947 (secure 0 0 0) 0x7f8d98072f50 con 0x7f8d78067690 2024-03-24T00:31:09.140 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:09.141 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:31:09.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.134+0000 7f8da0000640 1 -- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8d78067690 msgr2=0x7f8d78069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.134+0000 7f8da0000640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8d78067690 0x7f8d78069b50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c005890 tx=0x7f8d8c009040 comp rx=0 tx=0).stop 2024-03-24T00:31:09.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.134+0000 7f8da0000640 1 -- 172.21.15.40:0/772850916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d980ccfa0 msgr2=0x7f8d98159f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.134+0000 7f8da0000640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d980ccfa0 0x7f8d98159f50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8d9001a4e0 tx=0x7f8d90019650 comp rx=0 tx=0).stop 2024-03-24T00:31:09.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.138+0000 7f8da0000640 1 -- 172.21.15.40:0/772850916 shutdown_connections 2024-03-24T00:31:09.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.138+0000 7f8da0000640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8d78067690 0x7f8d78069b50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.138+0000 7f8da0000640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8d980ccfa0 0x7f8d98159f50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.138+0000 7f8da0000640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d980b46c0 0x7f8d9815b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.138+0000 7f8da0000640 1 --2- 172.21.15.40:0/772850916 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9806e620 0x7f8d981434b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.138+0000 7f8da0000640 1 -- 172.21.15.40:0/772850916 >> 172.21.15.40:0/772850916 conn(0x7f8d980b17b0 msgr2=0x7f8d980cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.138+0000 7f8da0000640 1 -- 172.21.15.40:0/772850916 shutdown_connections 2024-03-24T00:31:09.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.138+0000 7f8da0000640 1 -- 172.21.15.40:0/772850916 wait complete. 2024-03-24T00:31:09.158 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-03-24T00:31:07.198669+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235632,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739088,"statfs":{"total":773094113280,"available":772852826112,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001756"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046898+0000","last_change":"2024-03-24T00:31:00.052380+0000","last_active":"2024-03-24T00:31:01.046898+0000","last_peered":"2024-03-24T00:31:01.046898+0000","last_clean":"2024-03-24T00:31:01.046898+0000","last_became_active":"2024-03-24T00:31:00.049815+0000","last_became_peered":"2024-03-24T00:31:00.049815+0000","last_unstale":"2024-03-24T00:31:01.046898+0000","last_undegraded":"2024-03-24T00:31:01.046898+0000","last_fullsized":"2024-03-24T00:31:01.046898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:09.487064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047941+0000","last_change":"2024-03-24T00:31:00.048401+0000","last_active":"2024-03-24T00:31:01.047941+0000","last_peered":"2024-03-24T00:31:01.047941+0000","last_clean":"2024-03-24T00:31:01.047941+0000","last_became_active":"2024-03-24T00:31:00.048157+0000","last_became_peered":"2024-03-24T00:31:00.048157+0000","last_unstale":"2024-03-24T00:31:01.047941+0000","last_undegraded":"2024-03-24T00:31:01.047941+0000","last_fullsized":"2024-03-24T00:31:01.047941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909458+0000","last_change":"2024-03-24T00:31:00.052594+0000","last_active":"2024-03-24T00:31:01.909458+0000","last_peered":"2024-03-24T00:31:01.909458+0000","last_clean":"2024-03-24T00:31:01.909458+0000","last_became_active":"2024-03-24T00:31:00.052289+0000","last_became_peered":"2024-03-24T00:31:00.052289+0000","last_unstale":"2024-03-24T00:31:01.909458+0000","last_undegraded":"2024-03-24T00:31:01.909458+0000","last_fullsized":"2024-03-24T00:31:01.909458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:34.269764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909931+0000","last_change":"2024-03-24T00:31:00.072520+0000","last_active":"2024-03-24T00:31:01.909931+0000","last_peered":"2024-03-24T00:31:01.909931+0000","last_clean":"2024-03-24T00:31:01.909931+0000","last_became_active":"2024-03-24T00:31:00.071355+0000","last_became_peered":"2024-03-24T00:31:00.071355+0000","last_unstale":"2024-03-24T00:31:01.909931+0000","last_undegraded":"2024-03-24T00:31:01.909931+0000","last_fullsized":"2024-03-24T00:31:01.909931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:17.380183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046806+0000","last_change":"2024-03-24T00:31:00.045641+0000","last_active":"2024-03-24T00:31:01.046806+0000","last_peered":"2024-03-24T00:31:01.046806+0000","last_clean":"2024-03-24T00:31:01.046806+0000","last_became_active":"2024-03-24T00:31:00.045230+0000","last_became_peered":"2024-03-24T00:31:00.045230+0000","last_unstale":"2024-03-24T00:31:01.046806+0000","last_undegraded":"2024-03-24T00:31:01.046806+0000","last_fullsized":"2024-03-24T00:31:01.046806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:49.592126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909658+0000","last_change":"2024-03-24T00:31:00.057373+0000","last_active":"2024-03-24T00:31:01.909658+0000","last_peered":"2024-03-24T00:31:01.909658+0000","last_clean":"2024-03-24T00:31:01.909658+0000","last_became_active":"2024-03-24T00:31:00.056322+0000","last_became_peered":"2024-03-24T00:31:00.056322+0000","last_unstale":"2024-03-24T00:31:01.909658+0000","last_undegraded":"2024-03-24T00:31:01.909658+0000","last_fullsized":"2024-03-24T00:31:01.909658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045369+0000","last_change":"2024-03-24T00:31:00.048380+0000","last_active":"2024-03-24T00:31:01.045369+0000","last_peered":"2024-03-24T00:31:01.045369+0000","last_clean":"2024-03-24T00:31:01.045369+0000","last_became_active":"2024-03-24T00:31:00.046936+0000","last_became_peered":"2024-03-24T00:31:00.046936+0000","last_unstale":"2024-03-24T00:31:01.045369+0000","last_undegraded":"2024-03-24T00:31:01.045369+0000","last_fullsized":"2024-03-24T00:31:01.045369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:32.847580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909134+0000","last_change":"2024-03-24T00:31:00.057172+0000","last_active":"2024-03-24T00:31:01.909134+0000","last_peered":"2024-03-24T00:31:01.909134+0000","last_clean":"2024-03-24T00:31:01.909134+0000","last_became_active":"2024-03-24T00:31:00.056982+0000","last_became_peered":"2024-03-24T00:31:00.056982+0000","last_unstale":"2024-03-24T00:31:01.909134+0000","last_undegraded":"2024-03-24T00:31:01.909134+0000","last_fullsized":"2024-03-24T00:31:01.909134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:38.408736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047010+0000","last_change":"2024-03-24T00:31:00.053066+0000","last_active":"2024-03-24T00:31:01.047010+0000","last_peered":"2024-03-24T00:31:01.047010+0000","last_clean":"2024-03-24T00:31:01.047010+0000","last_became_active":"2024-03-24T00:31:00.052749+0000","last_became_peered":"2024-03-24T00:31:00.052749+0000","last_unstale":"2024-03-24T00:31:01.047010+0000","last_undegraded":"2024-03-24T00:31:01.047010+0000","last_fullsized":"2024-03-24T00:31:01.047010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:17.791692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060254+0000","last_change":"2024-03-24T00:31:00.060700+0000","last_active":"2024-03-24T00:31:01.060254+0000","last_peered":"2024-03-24T00:31:01.060254+0000","last_clean":"2024-03-24T00:31:01.060254+0000","last_became_active":"2024-03-24T00:31:00.060373+0000","last_became_peered":"2024-03-24T00:31:00.060373+0000","last_unstale":"2024-03-24T00:31:01.060254+0000","last_undegraded":"2024-03-24T00:31:01.060254+0000","last_fullsized":"2024-03-24T00:31:01.060254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060101+0000","last_change":"2024-03-24T00:31:00.061316+0000","last_active":"2024-03-24T00:31:01.060101+0000","last_peered":"2024-03-24T00:31:01.060101+0000","last_clean":"2024-03-24T00:31:01.060101+0000","last_became_active":"2024-03-24T00:31:00.060980+0000","last_became_peered":"2024-03-24T00:31:00.060980+0000","last_unstale":"2024-03-24T00:31:01.060101+0000","last_undegraded":"2024-03-24T00:31:01.060101+0000","last_fullsized":"2024-03-24T00:31:01.060101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045818+0000","last_change":"2024-03-24T00:31:00.046162+0000","last_active":"2024-03-24T00:31:01.045818+0000","last_peered":"2024-03-24T00:31:01.045818+0000","last_clean":"2024-03-24T00:31:01.045818+0000","last_became_active":"2024-03-24T00:31:00.045715+0000","last_became_peered":"2024-03-24T00:31:00.045715+0000","last_unstale":"2024-03-24T00:31:01.045818+0000","last_undegraded":"2024-03-24T00:31:01.045818+0000","last_fullsized":"2024-03-24T00:31:01.045818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:38.106034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909352+0000","last_change":"2024-03-24T00:31:00.045645+0000","last_active":"2024-03-24T00:31:01.909352+0000","last_peered":"2024-03-24T00:31:01.909352+0000","last_clean":"2024-03-24T00:31:01.909352+0000","last_became_active":"2024-03-24T00:31:00.045372+0000","last_became_peered":"2024-03-24T00:31:00.045372+0000","last_unstale":"2024-03-24T00:31:01.909352+0000","last_undegraded":"2024-03-24T00:31:01.909352+0000","last_fullsized":"2024-03-24T00:31:01.909352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909717+0000","last_change":"2024-03-24T00:31:00.042718+0000","last_active":"2024-03-24T00:31:01.909717+0000","last_peered":"2024-03-24T00:31:01.909717+0000","last_clean":"2024-03-24T00:31:01.909717+0000","last_became_active":"2024-03-24T00:31:00.042424+0000","last_became_peered":"2024-03-24T00:31:00.042424+0000","last_unstale":"2024-03-24T00:31:01.909717+0000","last_undegraded":"2024-03-24T00:31:01.909717+0000","last_fullsized":"2024-03-24T00:31:01.909717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047428+0000","last_change":"2024-03-24T00:31:00.038791+0000","last_active":"2024-03-24T00:31:01.047428+0000","last_peered":"2024-03-24T00:31:01.047428+0000","last_clean":"2024-03-24T00:31:01.047428+0000","last_became_active":"2024-03-24T00:31:00.038543+0000","last_became_peered":"2024-03-24T00:31:00.038543+0000","last_unstale":"2024-03-24T00:31:01.047428+0000","last_undegraded":"2024-03-24T00:31:01.047428+0000","last_fullsized":"2024-03-24T00:31:01.047428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909570+0000","last_change":"2024-03-24T00:31:00.055749+0000","last_active":"2024-03-24T00:31:01.909570+0000","last_peered":"2024-03-24T00:31:01.909570+0000","last_clean":"2024-03-24T00:31:01.909570+0000","last_became_active":"2024-03-24T00:31:00.055296+0000","last_became_peered":"2024-03-24T00:31:00.055296+0000","last_unstale":"2024-03-24T00:31:01.909570+0000","last_undegraded":"2024-03-24T00:31:01.909570+0000","last_fullsized":"2024-03-24T00:31:01.909570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910277+0000","last_change":"2024-03-24T00:31:00.055782+0000","last_active":"2024-03-24T00:31:01.910277+0000","last_peered":"2024-03-24T00:31:01.910277+0000","last_clean":"2024-03-24T00:31:01.910277+0000","last_became_active":"2024-03-24T00:31:00.055369+0000","last_became_peered":"2024-03-24T00:31:00.055369+0000","last_unstale":"2024-03-24T00:31:01.910277+0000","last_undegraded":"2024-03-24T00:31:01.910277+0000","last_fullsized":"2024-03-24T00:31:01.910277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:10.867603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186278+0000","last_change":"2024-03-24T00:31:00.044665+0000","last_active":"2024-03-24T00:31:01.186278+0000","last_peered":"2024-03-24T00:31:01.186278+0000","last_clean":"2024-03-24T00:31:01.186278+0000","last_became_active":"2024-03-24T00:31:00.044399+0000","last_became_peered":"2024-03-24T00:31:00.044399+0000","last_unstale":"2024-03-24T00:31:01.186278+0000","last_undegraded":"2024-03-24T00:31:01.186278+0000","last_fullsized":"2024-03-24T00:31:01.186278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:14.595788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047109+0000","last_change":"2024-03-24T00:31:00.045352+0000","last_active":"2024-03-24T00:31:01.047109+0000","last_peered":"2024-03-24T00:31:01.047109+0000","last_clean":"2024-03-24T00:31:01.047109+0000","last_became_active":"2024-03-24T00:31:00.045023+0000","last_became_peered":"2024-03-24T00:31:00.045023+0000","last_unstale":"2024-03-24T00:31:01.047109+0000","last_undegraded":"2024-03-24T00:31:01.047109+0000","last_fullsized":"2024-03-24T00:31:01.047109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:31.480032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060057+0000","last_change":"2024-03-24T00:31:00.061257+0000","last_active":"2024-03-24T00:31:01.060057+0000","last_peered":"2024-03-24T00:31:01.060057+0000","last_clean":"2024-03-24T00:31:01.060057+0000","last_became_active":"2024-03-24T00:31:00.060983+0000","last_became_peered":"2024-03-24T00:31:00.060983+0000","last_unstale":"2024-03-24T00:31:01.060057+0000","last_undegraded":"2024-03-24T00:31:01.060057+0000","last_fullsized":"2024-03-24T00:31:01.060057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909460+0000","last_change":"2024-03-24T00:31:00.054714+0000","last_active":"2024-03-24T00:31:01.909460+0000","last_peered":"2024-03-24T00:31:01.909460+0000","last_clean":"2024-03-24T00:31:01.909460+0000","last_became_active":"2024-03-24T00:31:00.054275+0000","last_became_peered":"2024-03-24T00:31:00.054275+0000","last_unstale":"2024-03-24T00:31:01.909460+0000","last_undegraded":"2024-03-24T00:31:01.909460+0000","last_fullsized":"2024-03-24T00:31:01.909460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:38.408736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909730+0000","last_change":"2024-03-24T00:31:00.063073+0000","last_active":"2024-03-24T00:31:01.909730+0000","last_peered":"2024-03-24T00:31:01.909730+0000","last_clean":"2024-03-24T00:31:01.909730+0000","last_became_active":"2024-03-24T00:31:00.062746+0000","last_became_peered":"2024-03-24T00:31:00.062746+0000","last_unstale":"2024-03-24T00:31:01.909730+0000","last_undegraded":"2024-03-24T00:31:01.909730+0000","last_fullsized":"2024-03-24T00:31:01.909730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:32.847580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909247+0000","last_change":"2024-03-24T00:31:00.059874+0000","last_active":"2024-03-24T00:31:01.909247+0000","last_peered":"2024-03-24T00:31:01.909247+0000","last_clean":"2024-03-24T00:31:01.909247+0000","last_became_active":"2024-03-24T00:31:00.059565+0000","last_became_peered":"2024-03-24T00:31:00.059565+0000","last_unstale":"2024-03-24T00:31:01.909247+0000","last_undegraded":"2024-03-24T00:31:01.909247+0000","last_fullsized":"2024-03-24T00:31:01.909247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:22.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909721+0000","last_change":"2024-03-24T00:31:00.056550+0000","last_active":"2024-03-24T00:31:01.909721+0000","last_peered":"2024-03-24T00:31:01.909721+0000","last_clean":"2024-03-24T00:31:01.909721+0000","last_became_active":"2024-03-24T00:31:00.055816+0000","last_became_peered":"2024-03-24T00:31:00.055816+0000","last_unstale":"2024-03-24T00:31:01.909721+0000","last_undegraded":"2024-03-24T00:31:01.909721+0000","last_fullsized":"2024-03-24T00:31:01.909721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:57:10.621965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909977+0000","last_change":"2024-03-24T00:31:00.059028+0000","last_active":"2024-03-24T00:31:01.909977+0000","last_peered":"2024-03-24T00:31:01.909977+0000","last_clean":"2024-03-24T00:31:01.909977+0000","last_became_active":"2024-03-24T00:31:00.058762+0000","last_became_peered":"2024-03-24T00:31:00.058762+0000","last_unstale":"2024-03-24T00:31:01.909977+0000","last_undegraded":"2024-03-24T00:31:01.909977+0000","last_fullsized":"2024-03-24T00:31:01.909977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060333+0000","last_change":"2024-03-24T00:31:00.054838+0000","last_active":"2024-03-24T00:31:01.060333+0000","last_peered":"2024-03-24T00:31:01.060333+0000","last_clean":"2024-03-24T00:31:01.060333+0000","last_became_active":"2024-03-24T00:31:00.054249+0000","last_became_peered":"2024-03-24T00:31:00.054249+0000","last_unstale":"2024-03-24T00:31:01.060333+0000","last_undegraded":"2024-03-24T00:31:01.060333+0000","last_fullsized":"2024-03-24T00:31:01.060333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047846+0000","last_change":"2024-03-24T00:31:00.037600+0000","last_active":"2024-03-24T00:31:01.047846+0000","last_peered":"2024-03-24T00:31:01.047846+0000","last_clean":"2024-03-24T00:31:01.047846+0000","last_became_active":"2024-03-24T00:31:00.037327+0000","last_became_peered":"2024-03-24T00:31:00.037327+0000","last_unstale":"2024-03-24T00:31:01.047846+0000","last_undegraded":"2024-03-24T00:31:01.047846+0000","last_fullsized":"2024-03-24T00:31:01.047846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909766+0000","last_change":"2024-03-24T00:31:00.061415+0000","last_active":"2024-03-24T00:31:01.909766+0000","last_peered":"2024-03-24T00:31:01.909766+0000","last_clean":"2024-03-24T00:31:01.909766+0000","last_became_active":"2024-03-24T00:31:00.061147+0000","last_became_peered":"2024-03-24T00:31:00.061147+0000","last_unstale":"2024-03-24T00:31:01.909766+0000","last_undegraded":"2024-03-24T00:31:01.909766+0000","last_fullsized":"2024-03-24T00:31:01.909766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909601+0000","last_change":"2024-03-24T00:31:00.049400+0000","last_active":"2024-03-24T00:31:01.909601+0000","last_peered":"2024-03-24T00:31:01.909601+0000","last_clean":"2024-03-24T00:31:01.909601+0000","last_became_active":"2024-03-24T00:31:00.048885+0000","last_became_peered":"2024-03-24T00:31:00.048885+0000","last_unstale":"2024-03-24T00:31:01.909601+0000","last_undegraded":"2024-03-24T00:31:01.909601+0000","last_fullsized":"2024-03-24T00:31:01.909601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:22.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909905+0000","last_change":"2024-03-24T00:31:00.056183+0000","last_active":"2024-03-24T00:31:01.909905+0000","last_peered":"2024-03-24T00:31:01.909905+0000","last_clean":"2024-03-24T00:31:01.909905+0000","last_became_active":"2024-03-24T00:31:00.055346+0000","last_became_peered":"2024-03-24T00:31:00.055346+0000","last_unstale":"2024-03-24T00:31:01.909905+0000","last_undegraded":"2024-03-24T00:31:01.909905+0000","last_fullsized":"2024-03-24T00:31:01.909905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:08.370031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186114+0000","last_change":"2024-03-24T00:31:00.066718+0000","last_active":"2024-03-24T00:31:01.186114+0000","last_peered":"2024-03-24T00:31:01.186114+0000","last_clean":"2024-03-24T00:31:01.186114+0000","last_became_active":"2024-03-24T00:31:00.066330+0000","last_became_peered":"2024-03-24T00:31:00.066330+0000","last_unstale":"2024-03-24T00:31:01.186114+0000","last_undegraded":"2024-03-24T00:31:01.186114+0000","last_fullsized":"2024-03-24T00:31:01.186114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:24.063587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059863+0000","last_change":"2024-03-24T00:31:00.045401+0000","last_active":"2024-03-24T00:31:01.059863+0000","last_peered":"2024-03-24T00:31:01.059863+0000","last_clean":"2024-03-24T00:31:01.059863+0000","last_became_active":"2024-03-24T00:31:00.044163+0000","last_became_peered":"2024-03-24T00:31:00.044163+0000","last_unstale":"2024-03-24T00:31:01.059863+0000","last_undegraded":"2024-03-24T00:31:01.059863+0000","last_fullsized":"2024-03-24T00:31:01.059863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:22.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045365+0000","last_change":"2024-03-24T00:31:00.039890+0000","last_active":"2024-03-24T00:31:01.045365+0000","last_peered":"2024-03-24T00:31:01.045365+0000","last_clean":"2024-03-24T00:31:01.045365+0000","last_became_active":"2024-03-24T00:31:00.038341+0000","last_became_peered":"2024-03-24T00:31:00.038341+0000","last_unstale":"2024-03-24T00:31:01.045365+0000","last_undegraded":"2024-03-24T00:31:01.045365+0000","last_fullsized":"2024-03-24T00:31:01.045365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:26.001997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910135+0000","last_change":"2024-03-24T00:30:59.052547+0000","last_active":"2024-03-24T00:31:01.910135+0000","last_peered":"2024-03-24T00:31:01.910135+0000","last_clean":"2024-03-24T00:31:01.910135+0000","last_became_active":"2024-03-24T00:30:59.052277+0000","last_became_peered":"2024-03-24T00:30:59.052277+0000","last_unstale":"2024-03-24T00:31:01.910135+0000","last_undegraded":"2024-03-24T00:31:01.910135+0000","last_fullsized":"2024-03-24T00:31:01.910135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:28:37.149520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909635+0000","last_change":"2024-03-24T00:31:00.064484+0000","last_active":"2024-03-24T00:31:01.909635+0000","last_peered":"2024-03-24T00:31:01.909635+0000","last_clean":"2024-03-24T00:31:01.909635+0000","last_became_active":"2024-03-24T00:31:00.064211+0000","last_became_peered":"2024-03-24T00:31:00.064211+0000","last_unstale":"2024-03-24T00:31:01.909635+0000","last_undegraded":"2024-03-24T00:31:01.909635+0000","last_fullsized":"2024-03-24T00:31:01.909635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059739+0000","last_change":"2024-03-24T00:30:59.053767+0000","last_active":"2024-03-24T00:31:01.059739+0000","last_peered":"2024-03-24T00:31:01.059739+0000","last_clean":"2024-03-24T00:31:01.059739+0000","last_became_active":"2024-03-24T00:30:59.050782+0000","last_became_peered":"2024-03-24T00:30:59.050782+0000","last_unstale":"2024-03-24T00:31:01.059739+0000","last_undegraded":"2024-03-24T00:31:01.059739+0000","last_fullsized":"2024-03-24T00:31:01.059739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909297+0000","last_change":"2024-03-24T00:30:59.047982+0000","last_active":"2024-03-24T00:31:01.909297+0000","last_peered":"2024-03-24T00:31:01.909297+0000","last_clean":"2024-03-24T00:31:01.909297+0000","last_became_active":"2024-03-24T00:30:59.047680+0000","last_became_peered":"2024-03-24T00:30:59.047680+0000","last_unstale":"2024-03-24T00:31:01.909297+0000","last_undegraded":"2024-03-24T00:31:01.909297+0000","last_fullsized":"2024-03-24T00:31:01.909297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:28.293820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910046+0000","last_change":"2024-03-24T00:31:00.044921+0000","last_active":"2024-03-24T00:31:01.910046+0000","last_peered":"2024-03-24T00:31:01.910046+0000","last_clean":"2024-03-24T00:31:01.910046+0000","last_became_active":"2024-03-24T00:31:00.044583+0000","last_became_peered":"2024-03-24T00:31:00.044583+0000","last_unstale":"2024-03-24T00:31:01.910046+0000","last_undegraded":"2024-03-24T00:31:01.910046+0000","last_fullsized":"2024-03-24T00:31:01.910046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:57:10.621965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909657+0000","last_change":"2024-03-24T00:30:59.058420+0000","last_active":"2024-03-24T00:31:01.909657+0000","last_peered":"2024-03-24T00:31:01.909657+0000","last_clean":"2024-03-24T00:31:01.909657+0000","last_became_active":"2024-03-24T00:30:59.058004+0000","last_became_peered":"2024-03-24T00:30:59.058004+0000","last_unstale":"2024-03-24T00:31:01.909657+0000","last_undegraded":"2024-03-24T00:31:01.909657+0000","last_fullsized":"2024-03-24T00:31:01.909657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047409+0000","last_change":"2024-03-24T00:31:00.043778+0000","last_active":"2024-03-24T00:31:01.047409+0000","last_peered":"2024-03-24T00:31:01.047409+0000","last_clean":"2024-03-24T00:31:01.047409+0000","last_became_active":"2024-03-24T00:31:00.043486+0000","last_became_peered":"2024-03-24T00:31:00.043486+0000","last_unstale":"2024-03-24T00:31:01.047409+0000","last_undegraded":"2024-03-24T00:31:01.047409+0000","last_fullsized":"2024-03-24T00:31:01.047409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060075+0000","last_change":"2024-03-24T00:30:59.057537+0000","last_active":"2024-03-24T00:31:01.060075+0000","last_peered":"2024-03-24T00:31:01.060075+0000","last_clean":"2024-03-24T00:31:01.060075+0000","last_became_active":"2024-03-24T00:30:59.057215+0000","last_became_peered":"2024-03-24T00:30:59.057215+0000","last_unstale":"2024-03-24T00:31:01.060075+0000","last_undegraded":"2024-03-24T00:31:01.060075+0000","last_fullsized":"2024-03-24T00:31:01.060075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:40.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910145+0000","last_change":"2024-03-24T00:31:00.057658+0000","last_active":"2024-03-24T00:31:01.910145+0000","last_peered":"2024-03-24T00:31:01.910145+0000","last_clean":"2024-03-24T00:31:01.910145+0000","last_became_active":"2024-03-24T00:31:00.055543+0000","last_became_peered":"2024-03-24T00:31:00.055543+0000","last_unstale":"2024-03-24T00:31:01.910145+0000","last_undegraded":"2024-03-24T00:31:01.910145+0000","last_fullsized":"2024-03-24T00:31:01.910145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:49.592126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910049+0000","last_change":"2024-03-24T00:30:59.058992+0000","last_active":"2024-03-24T00:31:01.910049+0000","last_peered":"2024-03-24T00:31:01.910049+0000","last_clean":"2024-03-24T00:31:01.910049+0000","last_became_active":"2024-03-24T00:30:59.058546+0000","last_became_peered":"2024-03-24T00:30:59.058546+0000","last_unstale":"2024-03-24T00:31:01.910049+0000","last_undegraded":"2024-03-24T00:31:01.910049+0000","last_fullsized":"2024-03-24T00:31:01.910049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:20.001460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909878+0000","last_change":"2024-03-24T00:31:00.057510+0000","last_active":"2024-03-24T00:31:01.909878+0000","last_peered":"2024-03-24T00:31:01.909878+0000","last_clean":"2024-03-24T00:31:01.909878+0000","last_became_active":"2024-03-24T00:31:00.057227+0000","last_became_peered":"2024-03-24T00:31:00.057227+0000","last_unstale":"2024-03-24T00:31:01.909878+0000","last_undegraded":"2024-03-24T00:31:01.909878+0000","last_fullsized":"2024-03-24T00:31:01.909878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047997+0000","last_change":"2024-03-24T00:30:59.055938+0000","last_active":"2024-03-24T00:31:01.047997+0000","last_peered":"2024-03-24T00:31:01.047997+0000","last_clean":"2024-03-24T00:31:01.047997+0000","last_became_active":"2024-03-24T00:30:59.055734+0000","last_became_peered":"2024-03-24T00:30:59.055734+0000","last_unstale":"2024-03-24T00:31:01.047997+0000","last_undegraded":"2024-03-24T00:31:01.047997+0000","last_fullsized":"2024-03-24T00:31:01.047997+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:39.280192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045606+0000","last_change":"2024-03-24T00:31:00.053717+0000","last_active":"2024-03-24T00:31:01.045606+0000","last_peered":"2024-03-24T00:31:01.045606+0000","last_clean":"2024-03-24T00:31:01.045606+0000","last_became_active":"2024-03-24T00:31:00.053312+0000","last_became_peered":"2024-03-24T00:31:00.053312+0000","last_unstale":"2024-03-24T00:31:01.045606+0000","last_undegraded":"2024-03-24T00:31:01.045606+0000","last_fullsized":"2024-03-24T00:31:01.045606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:40.286678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060097+0000","last_change":"2024-03-24T00:30:59.050368+0000","last_active":"2024-03-24T00:31:01.060097+0000","last_peered":"2024-03-24T00:31:01.060097+0000","last_clean":"2024-03-24T00:31:01.060097+0000","last_became_active":"2024-03-24T00:30:59.050062+0000","last_became_peered":"2024-03-24T00:30:59.050062+0000","last_unstale":"2024-03-24T00:31:01.060097+0000","last_undegraded":"2024-03-24T00:31:01.060097+0000","last_fullsized":"2024-03-24T00:31:01.060097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046667+0000","last_change":"2024-03-24T00:31:00.041643+0000","last_active":"2024-03-24T00:31:01.046667+0000","last_peered":"2024-03-24T00:31:01.046667+0000","last_clean":"2024-03-24T00:31:01.046667+0000","last_became_active":"2024-03-24T00:31:00.041387+0000","last_became_peered":"2024-03-24T00:31:00.041387+0000","last_unstale":"2024-03-24T00:31:01.046667+0000","last_undegraded":"2024-03-24T00:31:01.046667+0000","last_fullsized":"2024-03-24T00:31:01.046667+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:34:52.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909659+0000","last_change":"2024-03-24T00:30:59.054246+0000","last_active":"2024-03-24T00:31:01.909659+0000","last_peered":"2024-03-24T00:31:01.909659+0000","last_clean":"2024-03-24T00:31:01.909659+0000","last_became_active":"2024-03-24T00:30:59.053818+0000","last_became_peered":"2024-03-24T00:30:59.053818+0000","last_unstale":"2024-03-24T00:31:01.909659+0000","last_undegraded":"2024-03-24T00:31:01.909659+0000","last_fullsized":"2024-03-24T00:31:01.909659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:28.293820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909855+0000","last_change":"2024-03-24T00:31:00.054481+0000","last_active":"2024-03-24T00:31:01.909855+0000","last_peered":"2024-03-24T00:31:01.909855+0000","last_clean":"2024-03-24T00:31:01.909855+0000","last_became_active":"2024-03-24T00:31:00.054255+0000","last_became_peered":"2024-03-24T00:31:00.054255+0000","last_unstale":"2024-03-24T00:31:01.909855+0000","last_undegraded":"2024-03-24T00:31:01.909855+0000","last_fullsized":"2024-03-24T00:31:01.909855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910033+0000","last_change":"2024-03-24T00:30:59.059529+0000","last_active":"2024-03-24T00:31:01.910033+0000","last_peered":"2024-03-24T00:31:01.910033+0000","last_clean":"2024-03-24T00:31:01.910033+0000","last_became_active":"2024-03-24T00:30:59.058947+0000","last_became_peered":"2024-03-24T00:30:59.058947+0000","last_unstale":"2024-03-24T00:31:01.910033+0000","last_undegraded":"2024-03-24T00:31:01.910033+0000","last_fullsized":"2024-03-24T00:31:01.910033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:39.280192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046876+0000","last_change":"2024-03-24T00:31:00.061372+0000","last_active":"2024-03-24T00:31:01.046876+0000","last_peered":"2024-03-24T00:31:01.046876+0000","last_clean":"2024-03-24T00:31:01.046876+0000","last_became_active":"2024-03-24T00:31:00.061094+0000","last_became_peered":"2024-03-24T00:31:00.061094+0000","last_unstale":"2024-03-24T00:31:01.046876+0000","last_undegraded":"2024-03-24T00:31:01.046876+0000","last_fullsized":"2024-03-24T00:31:01.046876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:38.650664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059711+0000","last_change":"2024-03-24T00:30:59.046862+0000","last_active":"2024-03-24T00:31:01.059711+0000","last_peered":"2024-03-24T00:31:01.059711+0000","last_clean":"2024-03-24T00:31:01.059711+0000","last_became_active":"2024-03-24T00:30:59.046575+0000","last_became_peered":"2024-03-24T00:30:59.046575+0000","last_unstale":"2024-03-24T00:31:01.059711+0000","last_undegraded":"2024-03-24T00:31:01.059711+0000","last_fullsized":"2024-03-24T00:31:01.059711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:39.280192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059910+0000","last_change":"2024-03-24T00:31:00.044599+0000","last_active":"2024-03-24T00:31:01.059910+0000","last_peered":"2024-03-24T00:31:01.059910+0000","last_clean":"2024-03-24T00:31:01.059910+0000","last_became_active":"2024-03-24T00:31:00.043498+0000","last_became_peered":"2024-03-24T00:31:00.043498+0000","last_unstale":"2024-03-24T00:31:01.059910+0000","last_undegraded":"2024-03-24T00:31:01.059910+0000","last_fullsized":"2024-03-24T00:31:01.059910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.185968+0000","last_change":"2024-03-24T00:30:59.039042+0000","last_active":"2024-03-24T00:31:01.185968+0000","last_peered":"2024-03-24T00:31:01.185968+0000","last_clean":"2024-03-24T00:31:01.185968+0000","last_became_active":"2024-03-24T00:30:59.038744+0000","last_became_peered":"2024-03-24T00:30:59.038744+0000","last_unstale":"2024-03-24T00:31:01.185968+0000","last_undegraded":"2024-03-24T00:31:01.185968+0000","last_fullsized":"2024-03-24T00:31:01.185968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:20.001460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045531+0000","last_change":"2024-03-24T00:31:00.047666+0000","last_active":"2024-03-24T00:31:01.045531+0000","last_peered":"2024-03-24T00:31:01.045531+0000","last_clean":"2024-03-24T00:31:01.045531+0000","last_became_active":"2024-03-24T00:31:00.047379+0000","last_became_peered":"2024-03-24T00:31:00.047379+0000","last_unstale":"2024-03-24T00:31:01.045531+0000","last_undegraded":"2024-03-24T00:31:01.045531+0000","last_fullsized":"2024-03-24T00:31:01.045531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:08.370031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045647+0000","last_change":"2024-03-24T00:30:59.043700+0000","last_active":"2024-03-24T00:31:01.045647+0000","last_peered":"2024-03-24T00:31:01.045647+0000","last_clean":"2024-03-24T00:31:01.045647+0000","last_became_active":"2024-03-24T00:30:59.043305+0000","last_became_peered":"2024-03-24T00:30:59.043305+0000","last_unstale":"2024-03-24T00:31:01.045647+0000","last_undegraded":"2024-03-24T00:31:01.045647+0000","last_fullsized":"2024-03-24T00:31:01.045647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:16.392363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046650+0000","last_change":"2024-03-24T00:31:00.048570+0000","last_active":"2024-03-24T00:31:01.046650+0000","last_peered":"2024-03-24T00:31:01.046650+0000","last_clean":"2024-03-24T00:31:01.046650+0000","last_became_active":"2024-03-24T00:31:00.048162+0000","last_became_peered":"2024-03-24T00:31:00.048162+0000","last_unstale":"2024-03-24T00:31:01.046650+0000","last_undegraded":"2024-03-24T00:31:01.046650+0000","last_fullsized":"2024-03-24T00:31:01.046650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:17.380183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186240+0000","last_change":"2024-03-24T00:30:59.041080+0000","last_active":"2024-03-24T00:31:01.186240+0000","last_peered":"2024-03-24T00:31:01.186240+0000","last_clean":"2024-03-24T00:31:01.186240+0000","last_became_active":"2024-03-24T00:30:59.040599+0000","last_became_peered":"2024-03-24T00:30:59.040599+0000","last_unstale":"2024-03-24T00:31:01.186240+0000","last_undegraded":"2024-03-24T00:31:01.186240+0000","last_fullsized":"2024-03-24T00:31:01.186240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047022+0000","last_change":"2024-03-24T00:31:00.060654+0000","last_active":"2024-03-24T00:31:01.047022+0000","last_peered":"2024-03-24T00:31:01.047022+0000","last_clean":"2024-03-24T00:31:01.047022+0000","last_became_active":"2024-03-24T00:31:00.060301+0000","last_became_peered":"2024-03-24T00:31:00.060301+0000","last_unstale":"2024-03-24T00:31:01.047022+0000","last_undegraded":"2024-03-24T00:31:01.047022+0000","last_fullsized":"2024-03-24T00:31:01.047022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:41.152930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047876+0000","last_change":"2024-03-24T00:30:59.045891+0000","last_active":"2024-03-24T00:31:01.047876+0000","last_peered":"2024-03-24T00:31:01.047876+0000","last_clean":"2024-03-24T00:31:01.047876+0000","last_became_active":"2024-03-24T00:30:59.044107+0000","last_became_peered":"2024-03-24T00:30:59.044107+0000","last_unstale":"2024-03-24T00:31:01.047876+0000","last_undegraded":"2024-03-24T00:31:01.047876+0000","last_fullsized":"2024-03-24T00:31:01.047876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:16.613800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910353+0000","last_change":"2024-03-24T00:31:00.056640+0000","last_active":"2024-03-24T00:31:01.910353+0000","last_peered":"2024-03-24T00:31:01.910353+0000","last_clean":"2024-03-24T00:31:01.910353+0000","last_became_active":"2024-03-24T00:31:00.055891+0000","last_became_peered":"2024-03-24T00:31:00.055891+0000","last_unstale":"2024-03-24T00:31:01.910353+0000","last_undegraded":"2024-03-24T00:31:01.910353+0000","last_fullsized":"2024-03-24T00:31:01.910353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:17.791692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186119+0000","last_change":"2024-03-24T00:30:59.058994+0000","last_active":"2024-03-24T00:31:01.186119+0000","last_peered":"2024-03-24T00:31:01.186119+0000","last_clean":"2024-03-24T00:31:01.186119+0000","last_became_active":"2024-03-24T00:30:59.058639+0000","last_became_peered":"2024-03-24T00:30:59.058639+0000","last_unstale":"2024-03-24T00:31:01.186119+0000","last_undegraded":"2024-03-24T00:31:01.186119+0000","last_fullsized":"2024-03-24T00:31:01.186119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046886+0000","last_change":"2024-03-24T00:31:00.059336+0000","last_active":"2024-03-24T00:31:01.046886+0000","last_peered":"2024-03-24T00:31:01.046886+0000","last_clean":"2024-03-24T00:31:01.046886+0000","last_became_active":"2024-03-24T00:31:00.059071+0000","last_became_peered":"2024-03-24T00:31:00.059071+0000","last_unstale":"2024-03-24T00:31:01.046886+0000","last_undegraded":"2024-03-24T00:31:01.046886+0000","last_fullsized":"2024-03-24T00:31:01.046886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:10.867603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046267+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:31:01.046267+0000","last_peered":"2024-03-24T00:31:01.046267+0000","last_clean":"2024-03-24T00:31:01.046267+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:31:01.046267+0000","last_undegraded":"2024-03-24T00:31:01.046267+0000","last_fullsized":"2024-03-24T00:31:01.046267+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909596+0000","last_change":"2024-03-24T00:30:59.059377+0000","last_active":"2024-03-24T00:31:01.909596+0000","last_peered":"2024-03-24T00:31:01.909596+0000","last_clean":"2024-03-24T00:31:01.909596+0000","last_became_active":"2024-03-24T00:30:59.058424+0000","last_became_peered":"2024-03-24T00:30:59.058424+0000","last_unstale":"2024-03-24T00:31:01.909596+0000","last_undegraded":"2024-03-24T00:31:01.909596+0000","last_fullsized":"2024-03-24T00:31:01.909596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909334+0000","last_change":"2024-03-24T00:31:00.058762+0000","last_active":"2024-03-24T00:31:01.909334+0000","last_peered":"2024-03-24T00:31:01.909334+0000","last_clean":"2024-03-24T00:31:01.909334+0000","last_became_active":"2024-03-24T00:31:00.058509+0000","last_became_peered":"2024-03-24T00:31:00.058509+0000","last_unstale":"2024-03-24T00:31:01.909334+0000","last_undegraded":"2024-03-24T00:31:01.909334+0000","last_fullsized":"2024-03-24T00:31:01.909334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046137+0000","last_change":"2024-03-24T00:30:59.050656+0000","last_active":"2024-03-24T00:31:01.046137+0000","last_peered":"2024-03-24T00:31:01.046137+0000","last_clean":"2024-03-24T00:31:01.046137+0000","last_became_active":"2024-03-24T00:30:59.050364+0000","last_became_peered":"2024-03-24T00:30:59.050364+0000","last_unstale":"2024-03-24T00:31:01.046137+0000","last_undegraded":"2024-03-24T00:31:01.046137+0000","last_fullsized":"2024-03-24T00:31:01.046137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:23.150413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909215+0000","last_change":"2024-03-24T00:31:00.059875+0000","last_active":"2024-03-24T00:31:01.909215+0000","last_peered":"2024-03-24T00:31:01.909215+0000","last_clean":"2024-03-24T00:31:01.909215+0000","last_became_active":"2024-03-24T00:31:00.059618+0000","last_became_peered":"2024-03-24T00:31:00.059618+0000","last_unstale":"2024-03-24T00:31:01.909215+0000","last_undegraded":"2024-03-24T00:31:01.909215+0000","last_fullsized":"2024-03-24T00:31:01.909215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:34:42.663552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186071+0000","last_change":"2024-03-24T00:30:59.040845+0000","last_active":"2024-03-24T00:31:01.186071+0000","last_peered":"2024-03-24T00:31:01.186071+0000","last_clean":"2024-03-24T00:31:01.186071+0000","last_became_active":"2024-03-24T00:30:59.040519+0000","last_became_peered":"2024-03-24T00:30:59.040519+0000","last_unstale":"2024-03-24T00:31:01.186071+0000","last_undegraded":"2024-03-24T00:31:01.186071+0000","last_fullsized":"2024-03-24T00:31:01.186071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:28.293820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047374+0000","last_change":"2024-03-24T00:31:00.041015+0000","last_active":"2024-03-24T00:31:01.047374+0000","last_peered":"2024-03-24T00:31:01.047374+0000","last_clean":"2024-03-24T00:31:01.047374+0000","last_became_active":"2024-03-24T00:31:00.040568+0000","last_became_peered":"2024-03-24T00:31:00.040568+0000","last_unstale":"2024-03-24T00:31:01.047374+0000","last_undegraded":"2024-03-24T00:31:01.047374+0000","last_fullsized":"2024-03-24T00:31:01.047374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910220+0000","last_change":"2024-03-24T00:30:59.061747+0000","last_active":"2024-03-24T00:31:01.910220+0000","last_peered":"2024-03-24T00:31:01.910220+0000","last_clean":"2024-03-24T00:31:01.910220+0000","last_became_active":"2024-03-24T00:30:59.059297+0000","last_became_peered":"2024-03-24T00:30:59.059297+0000","last_unstale":"2024-03-24T00:31:01.910220+0000","last_undegraded":"2024-03-24T00:31:01.910220+0000","last_fullsized":"2024-03-24T00:31:01.910220+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:37.099548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045947+0000","last_change":"2024-03-24T00:31:00.050644+0000","last_active":"2024-03-24T00:31:01.045947+0000","last_peered":"2024-03-24T00:31:01.045947+0000","last_clean":"2024-03-24T00:31:01.045947+0000","last_became_active":"2024-03-24T00:31:00.050376+0000","last_became_peered":"2024-03-24T00:31:00.050376+0000","last_unstale":"2024-03-24T00:31:01.045947+0000","last_undegraded":"2024-03-24T00:31:01.045947+0000","last_fullsized":"2024-03-24T00:31:01.045947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:17.620286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047738+0000","last_change":"2024-03-24T00:30:59.045279+0000","last_active":"2024-03-24T00:31:01.047738+0000","last_peered":"2024-03-24T00:31:01.047738+0000","last_clean":"2024-03-24T00:31:01.047738+0000","last_became_active":"2024-03-24T00:30:59.044864+0000","last_became_peered":"2024-03-24T00:30:59.044864+0000","last_unstale":"2024-03-24T00:31:01.047738+0000","last_undegraded":"2024-03-24T00:31:01.047738+0000","last_fullsized":"2024-03-24T00:31:01.047738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:37.099548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910242+0000","last_change":"2024-03-24T00:31:00.056075+0000","last_active":"2024-03-24T00:31:01.910242+0000","last_peered":"2024-03-24T00:31:01.910242+0000","last_clean":"2024-03-24T00:31:01.910242+0000","last_became_active":"2024-03-24T00:31:00.055694+0000","last_became_peered":"2024-03-24T00:31:00.055694+0000","last_unstale":"2024-03-24T00:31:01.910242+0000","last_undegraded":"2024-03-24T00:31:01.910242+0000","last_fullsized":"2024-03-24T00:31:01.910242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:38.650664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047000+0000","last_change":"2024-03-24T00:30:59.048260+0000","last_active":"2024-03-24T00:31:01.047000+0000","last_peered":"2024-03-24T00:31:01.047000+0000","last_clean":"2024-03-24T00:31:01.047000+0000","last_became_active":"2024-03-24T00:30:59.047943+0000","last_became_peered":"2024-03-24T00:30:59.047943+0000","last_unstale":"2024-03-24T00:31:01.047000+0000","last_undegraded":"2024-03-24T00:31:01.047000+0000","last_fullsized":"2024-03-24T00:31:01.047000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:13.589302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047659+0000","last_change":"2024-03-24T00:31:00.041416+0000","last_active":"2024-03-24T00:31:01.047659+0000","last_peered":"2024-03-24T00:31:01.047659+0000","last_clean":"2024-03-24T00:31:01.047659+0000","last_became_active":"2024-03-24T00:31:00.041096+0000","last_became_peered":"2024-03-24T00:31:00.041096+0000","last_unstale":"2024-03-24T00:31:01.047659+0000","last_undegraded":"2024-03-24T00:31:01.047659+0000","last_fullsized":"2024-03-24T00:31:01.047659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047950+0000","last_change":"2024-03-24T00:30:59.046901+0000","last_active":"2024-03-24T00:31:01.047950+0000","last_peered":"2024-03-24T00:31:01.047950+0000","last_clean":"2024-03-24T00:31:01.047950+0000","last_became_active":"2024-03-24T00:30:59.046619+0000","last_became_peered":"2024-03-24T00:30:59.046619+0000","last_unstale":"2024-03-24T00:31:01.047950+0000","last_undegraded":"2024-03-24T00:31:01.047950+0000","last_fullsized":"2024-03-24T00:31:01.047950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:40.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909976+0000","last_change":"2024-03-24T00:31:00.047744+0000","last_active":"2024-03-24T00:31:01.909976+0000","last_peered":"2024-03-24T00:31:01.909976+0000","last_clean":"2024-03-24T00:31:01.909976+0000","last_became_active":"2024-03-24T00:31:00.047453+0000","last_became_peered":"2024-03-24T00:31:00.047453+0000","last_unstale":"2024-03-24T00:31:01.909976+0000","last_undegraded":"2024-03-24T00:31:01.909976+0000","last_fullsized":"2024-03-24T00:31:01.909976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060070+0000","last_change":"2024-03-24T00:30:59.047379+0000","last_active":"2024-03-24T00:31:01.060070+0000","last_peered":"2024-03-24T00:31:01.060070+0000","last_clean":"2024-03-24T00:31:01.060070+0000","last_became_active":"2024-03-24T00:30:59.045528+0000","last_became_peered":"2024-03-24T00:30:59.045528+0000","last_unstale":"2024-03-24T00:31:01.060070+0000","last_undegraded":"2024-03-24T00:31:01.060070+0000","last_fullsized":"2024-03-24T00:31:01.060070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:23.057101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909624+0000","last_change":"2024-03-24T00:31:00.050676+0000","last_active":"2024-03-24T00:31:01.909624+0000","last_peered":"2024-03-24T00:31:01.909624+0000","last_clean":"2024-03-24T00:31:01.909624+0000","last_became_active":"2024-03-24T00:31:00.050421+0000","last_became_peered":"2024-03-24T00:31:00.050421+0000","last_unstale":"2024-03-24T00:31:01.909624+0000","last_undegraded":"2024-03-24T00:31:01.909624+0000","last_fullsized":"2024-03-24T00:31:01.909624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045080+0000","last_change":"2024-03-24T00:30:59.047814+0000","last_active":"2024-03-24T00:31:01.045080+0000","last_peered":"2024-03-24T00:31:01.045080+0000","last_clean":"2024-03-24T00:31:01.045080+0000","last_became_active":"2024-03-24T00:30:59.047497+0000","last_became_peered":"2024-03-24T00:30:59.047497+0000","last_unstale":"2024-03-24T00:31:01.045080+0000","last_undegraded":"2024-03-24T00:31:01.045080+0000","last_fullsized":"2024-03-24T00:31:01.045080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:37.099548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046279+0000","last_change":"2024-03-24T00:31:00.043642+0000","last_active":"2024-03-24T00:31:01.046279+0000","last_peered":"2024-03-24T00:31:01.046279+0000","last_clean":"2024-03-24T00:31:01.046279+0000","last_became_active":"2024-03-24T00:31:00.043334+0000","last_became_peered":"2024-03-24T00:31:00.043334+0000","last_unstale":"2024-03-24T00:31:01.046279+0000","last_undegraded":"2024-03-24T00:31:01.046279+0000","last_fullsized":"2024-03-24T00:31:01.046279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:41.254416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047463+0000","last_change":"2024-03-24T00:30:59.044547+0000","last_active":"2024-03-24T00:31:01.047463+0000","last_peered":"2024-03-24T00:31:01.047463+0000","last_clean":"2024-03-24T00:31:01.047463+0000","last_became_active":"2024-03-24T00:30:59.044240+0000","last_became_peered":"2024-03-24T00:30:59.044240+0000","last_unstale":"2024-03-24T00:31:01.047463+0000","last_undegraded":"2024-03-24T00:31:01.047463+0000","last_fullsized":"2024-03-24T00:31:01.047463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910154+0000","last_change":"2024-03-24T00:31:00.043313+0000","last_active":"2024-03-24T00:31:01.910154+0000","last_peered":"2024-03-24T00:31:01.910154+0000","last_clean":"2024-03-24T00:31:01.910154+0000","last_became_active":"2024-03-24T00:31:00.043030+0000","last_became_peered":"2024-03-24T00:31:00.043030+0000","last_unstale":"2024-03-24T00:31:01.910154+0000","last_undegraded":"2024-03-24T00:31:01.910154+0000","last_fullsized":"2024-03-24T00:31:01.910154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:32.847580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047581+0000","last_change":"2024-03-24T00:30:59.045395+0000","last_active":"2024-03-24T00:31:01.047581+0000","last_peered":"2024-03-24T00:31:01.047581+0000","last_clean":"2024-03-24T00:31:01.047581+0000","last_became_active":"2024-03-24T00:30:59.044888+0000","last_became_peered":"2024-03-24T00:30:59.044888+0000","last_unstale":"2024-03-24T00:31:01.047581+0000","last_undegraded":"2024-03-24T00:31:01.047581+0000","last_fullsized":"2024-03-24T00:31:01.047581+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910336+0000","last_change":"2024-03-24T00:31:00.052157+0000","last_active":"2024-03-24T00:31:01.910336+0000","last_peered":"2024-03-24T00:31:01.910336+0000","last_clean":"2024-03-24T00:31:01.910336+0000","last_became_active":"2024-03-24T00:31:00.051147+0000","last_became_peered":"2024-03-24T00:31:00.051147+0000","last_unstale":"2024-03-24T00:31:01.910336+0000","last_undegraded":"2024-03-24T00:31:01.910336+0000","last_fullsized":"2024-03-24T00:31:01.910336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:08.370031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046873+0000","last_change":"2024-03-24T00:30:59.053443+0000","last_active":"2024-03-24T00:31:01.046873+0000","last_peered":"2024-03-24T00:31:01.046873+0000","last_clean":"2024-03-24T00:31:01.046873+0000","last_became_active":"2024-03-24T00:30:59.053111+0000","last_became_peered":"2024-03-24T00:30:59.053111+0000","last_unstale":"2024-03-24T00:31:01.046873+0000","last_undegraded":"2024-03-24T00:31:01.046873+0000","last_fullsized":"2024-03-24T00:31:01.046873+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:40.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059908+0000","last_change":"2024-03-24T00:31:00.061892+0000","last_active":"2024-03-24T00:31:01.059908+0000","last_peered":"2024-03-24T00:31:01.059908+0000","last_clean":"2024-03-24T00:31:01.059908+0000","last_became_active":"2024-03-24T00:31:00.061567+0000","last_became_peered":"2024-03-24T00:31:00.061567+0000","last_unstale":"2024-03-24T00:31:01.059908+0000","last_undegraded":"2024-03-24T00:31:01.059908+0000","last_fullsized":"2024-03-24T00:31:01.059908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910099+0000","last_change":"2024-03-24T00:31:00.069008+0000","last_active":"2024-03-24T00:31:01.910099+0000","last_peered":"2024-03-24T00:31:01.910099+0000","last_clean":"2024-03-24T00:31:01.910099+0000","last_became_active":"2024-03-24T00:31:00.065217+0000","last_became_peered":"2024-03-24T00:31:00.065217+0000","last_unstale":"2024-03-24T00:31:01.910099+0000","last_undegraded":"2024-03-24T00:31:01.910099+0000","last_fullsized":"2024-03-24T00:31:01.910099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:34:52.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046539+0000","last_change":"2024-03-24T00:30:59.060493+0000","last_active":"2024-03-24T00:31:01.046539+0000","last_peered":"2024-03-24T00:31:01.046539+0000","last_clean":"2024-03-24T00:31:01.046539+0000","last_became_active":"2024-03-24T00:30:59.060208+0000","last_became_peered":"2024-03-24T00:30:59.060208+0000","last_unstale":"2024-03-24T00:31:01.046539+0000","last_undegraded":"2024-03-24T00:31:01.046539+0000","last_fullsized":"2024-03-24T00:31:01.046539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:16.613800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910297+0000","last_change":"2024-03-24T00:31:00.071223+0000","last_active":"2024-03-24T00:31:01.910297+0000","last_peered":"2024-03-24T00:31:01.910297+0000","last_clean":"2024-03-24T00:31:01.910297+0000","last_became_active":"2024-03-24T00:31:00.069882+0000","last_became_peered":"2024-03-24T00:31:00.069882+0000","last_unstale":"2024-03-24T00:31:01.910297+0000","last_undegraded":"2024-03-24T00:31:01.910297+0000","last_fullsized":"2024-03-24T00:31:01.910297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:09.487064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046833+0000","last_change":"2024-03-24T00:30:59.052020+0000","last_active":"2024-03-24T00:31:01.046833+0000","last_peered":"2024-03-24T00:31:01.046833+0000","last_clean":"2024-03-24T00:31:01.046833+0000","last_became_active":"2024-03-24T00:30:59.049277+0000","last_became_peered":"2024-03-24T00:30:59.049277+0000","last_unstale":"2024-03-24T00:31:01.046833+0000","last_undegraded":"2024-03-24T00:31:01.046833+0000","last_fullsized":"2024-03-24T00:31:01.046833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:23.057101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909752+0000","last_change":"2024-03-24T00:31:00.072313+0000","last_active":"2024-03-24T00:31:01.909752+0000","last_peered":"2024-03-24T00:31:01.909752+0000","last_clean":"2024-03-24T00:31:01.909752+0000","last_became_active":"2024-03-24T00:31:00.071692+0000","last_became_peered":"2024-03-24T00:31:00.071692+0000","last_unstale":"2024-03-24T00:31:01.909752+0000","last_undegraded":"2024-03-24T00:31:01.909752+0000","last_fullsized":"2024-03-24T00:31:01.909752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910030+0000","last_change":"2024-03-24T00:30:59.045778+0000","last_active":"2024-03-24T00:31:01.910030+0000","last_peered":"2024-03-24T00:31:01.910030+0000","last_clean":"2024-03-24T00:31:01.910030+0000","last_became_active":"2024-03-24T00:30:59.045515+0000","last_became_peered":"2024-03-24T00:30:59.045515+0000","last_unstale":"2024-03-24T00:31:01.910030+0000","last_undegraded":"2024-03-24T00:31:01.910030+0000","last_fullsized":"2024-03-24T00:31:01.910030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:20.001460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909974+0000","last_change":"2024-03-24T00:31:00.047507+0000","last_active":"2024-03-24T00:31:01.909974+0000","last_peered":"2024-03-24T00:31:01.909974+0000","last_clean":"2024-03-24T00:31:01.909974+0000","last_became_active":"2024-03-24T00:31:00.047245+0000","last_became_peered":"2024-03-24T00:31:00.047245+0000","last_unstale":"2024-03-24T00:31:01.909974+0000","last_undegraded":"2024-03-24T00:31:01.909974+0000","last_fullsized":"2024-03-24T00:31:01.909974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909924+0000","last_change":"2024-03-24T00:30:59.053442+0000","last_active":"2024-03-24T00:31:01.909924+0000","last_peered":"2024-03-24T00:31:01.909924+0000","last_clean":"2024-03-24T00:31:01.909924+0000","last_became_active":"2024-03-24T00:30:59.053159+0000","last_became_peered":"2024-03-24T00:30:59.053159+0000","last_unstale":"2024-03-24T00:31:01.909924+0000","last_undegraded":"2024-03-24T00:31:01.909924+0000","last_fullsized":"2024-03-24T00:31:01.909924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:28:37.149520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574851,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574851,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:31:09.163 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:31:09.163 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T00:31:09.163 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-24T00:31:09.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-24T00:31:09.164 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-24T00:31:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.342+0000 7f603d000640 1 Processor -- start 2024-03-24T00:31:09.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.342+0000 7f603d000640 1 -- start start 2024-03-24T00:31:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.342+0000 7f603d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f60380b2aa0 0x7f60380d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.342+0000 7f603d000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f60380d0a80 0x7f60380cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.342+0000 7f603d000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60380b2870 con 0x7f60380b2490 2024-03-24T00:31:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.342+0000 7f603d000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603806b6d0 con 0x7f60380d0a80 2024-03-24T00:31:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.342+0000 7f603d000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60380b3190 con 0x7f60380b2aa0 2024-03-24T00:31:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.342+0000 7f6036800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f60380d0a80 0x7f60380cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.342+0000 7f6036800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f60380d0a80 0x7f60380cc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46894/0 (socket says 172.21.15.40:46894) 2024-03-24T00:31:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.342+0000 7f6036800640 1 -- 172.21.15.40:0/766987477 learned_addr learned my addr 172.21.15.40:0/766987477 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.346+0000 7f602fe00640 1 --2- 172.21.15.40:0/766987477 >> v2:172.21.15.53:3300/0 conn(0x7f60380b2aa0 0x7f60380d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.346+0000 7f6036800640 1 -- 172.21.15.40:0/766987477 >> v2:172.21.15.53:3300/0 conn(0x7f60380b2aa0 msgr2=0x7f60380d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.346+0000 7f6036800640 1 --2- 172.21.15.40:0/766987477 >> v2:172.21.15.53:3300/0 conn(0x7f60380b2aa0 0x7f60380d0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.346+0000 7f6036800640 1 -- 172.21.15.40:0/766987477 >> v1:172.21.15.53:6789/0 conn(0x7f60380b2490 legacy=0x7f60380cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.346+0000 7f6036800640 1 -- 172.21.15.40:0/766987477 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60380b3470 con 0x7f60380d0a80 2024-03-24T00:31:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.346+0000 7f6036800640 1 --2- 172.21.15.40:0/766987477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380d0a80 0x7f60380cc460 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6038072db0 tx=0x7f603001c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ebc7d58b6a2e309 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.346+0000 7f6035400640 1 -- 172.21.15.40:0/766987477 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60300a5890 con 0x7f60380d0a80 2024-03-24T00:31:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.346+0000 7f6035400640 1 -- 172.21.15.40:0/766987477 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60300a5a30 con 0x7f60380d0a80 2024-03-24T00:31:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.346+0000 7f6035400640 1 -- 172.21.15.40:0/766987477 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60300a5e00 con 0x7f60380d0a80 2024-03-24T00:31:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 -- 172.21.15.40:0/766987477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380d0a80 msgr2=0x7f60380cc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 --2- 172.21.15.40:0/766987477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380d0a80 0x7f60380cc460 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6038072db0 tx=0x7f603001c9b0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 -- 172.21.15.40:0/766987477 shutdown_connections 2024-03-24T00:31:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 --2- 172.21.15.40:0/766987477 >> v2:172.21.15.53:3300/0 conn(0x7f60380b2aa0 0x7f60380d0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 --2- 172.21.15.40:0/766987477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380d0a80 0x7f60380cc460 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 -- 172.21.15.40:0/766987477 >> 172.21.15.40:0/766987477 conn(0x7f60380d1d10 msgr2=0x7f60380d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 -- 172.21.15.40:0/766987477 shutdown_connections 2024-03-24T00:31:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 -- 172.21.15.40:0/766987477 wait complete. 2024-03-24T00:31:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 Processor -- start 2024-03-24T00:31:09.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 -- start start 2024-03-24T00:31:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60380b2490 0x7f603814f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380b2aa0 0x7f60381f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60380d0a80 0x7f6038155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60380cd580 con 0x7f60380b2aa0 2024-03-24T00:31:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60380cd400 con 0x7f60380b2490 2024-03-24T00:31:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f603d000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60380cd700 con 0x7f60380d0a80 2024-03-24T00:31:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f602fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380b2aa0 0x7f60381f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f602fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380b2aa0 0x7f60381f79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46908/0 (socket says 172.21.15.40:46908) 2024-03-24T00:31:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f602fe00640 1 -- 172.21.15.40:0/262180736 learned_addr learned my addr 172.21.15.40:0/262180736 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f6035e00640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60380b2490 0x7f603814f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f6036800640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60380d0a80 0x7f6038155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f602fe00640 1 -- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60380d0a80 msgr2=0x7f6038155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f602fe00640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60380d0a80 0x7f6038155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f602fe00640 1 -- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60380b2490 msgr2=0x7f603814f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f602fe00640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60380b2490 0x7f603814f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.350+0000 7f602fe00640 1 -- 172.21.15.40:0/262180736 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60381501c0 con 0x7f60380b2aa0 2024-03-24T00:31:09.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.354+0000 7f602fe00640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380b2aa0 0x7f60381f79d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f602000d830 tx=0x7f602000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.354+0000 7f602e000640 1 -- 172.21.15.40:0/262180736 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602000b6c0 con 0x7f60380b2aa0 2024-03-24T00:31:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.354+0000 7f602e000640 1 -- 172.21.15.40:0/262180736 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6020019070 con 0x7f60380b2aa0 2024-03-24T00:31:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.354+0000 7f602e000640 1 -- 172.21.15.40:0/262180736 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602000ba60 con 0x7f60380b2aa0 2024-03-24T00:31:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.354+0000 7f603d000640 1 -- 172.21.15.40:0/262180736 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60381504a0 con 0x7f60380b2aa0 2024-03-24T00:31:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.354+0000 7f603d000640 1 -- 172.21.15.40:0/262180736 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6038150a60 con 0x7f60380b2aa0 2024-03-24T00:31:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.354+0000 7f603d000640 1 -- 172.21.15.40:0/262180736 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6038068170 con 0x7f60380b2aa0 2024-03-24T00:31:09.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.354+0000 7f602e000640 1 -- 172.21.15.40:0/262180736 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f602001e460 con 0x7f60380b2aa0 2024-03-24T00:31:09.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.354+0000 7f602e000640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6014067660 0x7f6014069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:09.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.354+0000 7f602e000640 1 -- 172.21.15.40:0/262180736 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6020085e90 con 0x7f60380b2aa0 2024-03-24T00:31:09.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.354+0000 7f6035e00640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6014067660 0x7f6014069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:09.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.358+0000 7f6035e00640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6014067660 0x7f6014069b20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f60280098e0 tx=0x7f6028009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:09.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.358+0000 7f602e000640 1 -- 172.21.15.40:0/262180736 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6020056420 con 0x7f60380b2aa0 2024-03-24T00:31:09.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.558+0000 7f603d000640 1 -- 172.21.15.40:0/262180736 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f60380ca680 con 0x7f60380b2aa0 2024-03-24T00:31:09.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.562+0000 7f602e000640 1 -- 172.21.15.40:0/262180736 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+166524 (secure 0 0 0) 0x7f6020087090 con 0x7f60380b2aa0 2024-03-24T00:31:09.567 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:09.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.566+0000 7f603d000640 1 -- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6014067660 msgr2=0x7f6014069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.566+0000 7f603d000640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6014067660 0x7f6014069b20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f60280098e0 tx=0x7f6028009060 comp rx=0 tx=0).stop 2024-03-24T00:31:09.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.566+0000 7f603d000640 1 -- 172.21.15.40:0/262180736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380b2aa0 msgr2=0x7f60381f79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.566+0000 7f603d000640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380b2aa0 0x7f60381f79d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f602000d830 tx=0x7f602000dcf0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.566+0000 7f603d000640 1 -- 172.21.15.40:0/262180736 shutdown_connections 2024-03-24T00:31:09.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.566+0000 7f603d000640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6014067660 0x7f6014069b20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.566+0000 7f603d000640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60380d0a80 0x7f6038155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.566+0000 7f603d000640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60380b2aa0 0x7f60381f79d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.566+0000 7f603d000640 1 --2- 172.21.15.40:0/262180736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60380b2490 0x7f603814f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.566+0000 7f603d000640 1 -- 172.21.15.40:0/262180736 >> 172.21.15.40:0/262180736 conn(0x7f60380d1d10 msgr2=0x7f60380cfdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:09.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.566+0000 7f603d000640 1 -- 172.21.15.40:0/262180736 shutdown_connections 2024-03-24T00:31:09.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.566+0000 7f603d000640 1 -- 172.21.15.40:0/262180736 wait complete. 2024-03-24T00:31:09.587 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":4217283018},{"type":"v1","addr":"172.21.15.53:6833","nonce":4217283018}]},"active_addr":"172.21.15.53:6833/4217283018","active_change":"2024-03-24T00:30:47.185316+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":138148973}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":3833621039}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":226131153}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":364034933}]}]} 2024-03-24T00:31:09.590 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-24T00:31:09.590 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-24T00:31:09.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:09.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.726+0000 7ffb17e00640 1 Processor -- start 2024-03-24T00:31:09.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb17e00640 1 -- start start 2024-03-24T00:31:09.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb17e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb18145520 0x7ffb1815f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:09.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb17e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1815fe60 0x7ffb1806cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:09.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb17e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb180cec10 con 0x7ffb1806d330 2024-03-24T00:31:09.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb17e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb181602a0 con 0x7ffb18145520 2024-03-24T00:31:09.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb17e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb180ceeb0 con 0x7ffb1815fe60 2024-03-24T00:31:09.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb16a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb18145520 0x7ffb1815f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:09.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb16a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffb18145520 0x7ffb1815f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46916/0 (socket says 172.21.15.40:46916) 2024-03-24T00:31:09.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb16a00640 1 -- 172.21.15.40:0/1534644579 learned_addr learned my addr 172.21.15.40:0/1534644579 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:09.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb0fe00640 1 --2- 172.21.15.40:0/1534644579 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1815fe60 0x7ffb1806cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:09.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb16a00640 1 -- 172.21.15.40:0/1534644579 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1815fe60 msgr2=0x7ffb1806cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:09.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb16a00640 1 --2- 172.21.15.40:0/1534644579 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1815fe60 0x7ffb1806cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb16a00640 1 -- 172.21.15.40:0/1534644579 >> v1:172.21.15.40:6789/0 conn(0x7ffb1806d330 legacy=0x7ffb1806d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb16a00640 1 -- 172.21.15.40:0/1534644579 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180cf190 con 0x7ffb18145520 2024-03-24T00:31:09.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.730+0000 7ffb16000640 1 -- 172.21.15.40:0/1534644579 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002633005 0 0) 0x7ffb180cec10 con 0x7ffb1806d330 2024-03-24T00:31:09.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb16a00640 1 --2- 172.21.15.40:0/1534644579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb18145520 0x7ffb1815f920 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009870 tx=0x7ffb0801cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5ff2c940fc6d47 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:09.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb16000640 1 -- 172.21.15.40:0/1534644579 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb080aa620 con 0x7ffb18145520 2024-03-24T00:31:09.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb16000640 1 -- 172.21.15.40:0/1534644579 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb080af070 con 0x7ffb18145520 2024-03-24T00:31:09.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb16000640 1 -- 172.21.15.40:0/1534644579 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb080aa9e0 con 0x7ffb18145520 2024-03-24T00:31:09.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb17e00640 1 -- 172.21.15.40:0/1534644579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb18145520 msgr2=0x7ffb1815f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb17e00640 1 --2- 172.21.15.40:0/1534644579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb18145520 0x7ffb1815f920 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ffb08009870 tx=0x7ffb0801cf00 comp rx=0 tx=0).stop 2024-03-24T00:31:09.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb17e00640 1 -- 172.21.15.40:0/1534644579 shutdown_connections 2024-03-24T00:31:09.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb17e00640 1 --2- 172.21.15.40:0/1534644579 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1815fe60 0x7ffb1806cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb17e00640 1 --2- 172.21.15.40:0/1534644579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb18145520 0x7ffb1815f920 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb17e00640 1 -- 172.21.15.40:0/1534644579 >> 172.21.15.40:0/1534644579 conn(0x7ffb180b1740 msgr2=0x7ffb180b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:09.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb17e00640 1 -- 172.21.15.40:0/1534644579 shutdown_connections 2024-03-24T00:31:09.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb17e00640 1 -- 172.21.15.40:0/1534644579 wait complete. 2024-03-24T00:31:09.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb17e00640 1 Processor -- start 2024-03-24T00:31:09.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.734+0000 7ffb17e00640 1 -- start start 2024-03-24T00:31:09.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb17e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1806d330 0x7ffb18144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:09.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb17e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb18145520 0x7ffb18072570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:09.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb17e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1815fe60 0x7ffb1813a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:09.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb17e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb181605a0 con 0x7ffb1806d330 2024-03-24T00:31:09.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb17e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18145b10 con 0x7ffb18145520 2024-03-24T00:31:09.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb17e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb18160720 con 0x7ffb1815fe60 2024-03-24T00:31:09.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb16a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1806d330 0x7ffb18144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:09.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb16a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1806d330 0x7ffb18144770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46922/0 (socket says 172.21.15.40:46922) 2024-03-24T00:31:09.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb16a00640 1 -- 172.21.15.40:0/3364651749 learned_addr learned my addr 172.21.15.40:0/3364651749 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:09.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb0fe00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb18145520 0x7ffb18072570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:09.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb17400640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1815fe60 0x7ffb1813a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:09.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb16a00640 1 -- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1815fe60 msgr2=0x7ffb1813a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb16a00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1815fe60 0x7ffb1813a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb16a00640 1 -- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb18145520 msgr2=0x7ffb18072570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb16a00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb18145520 0x7ffb18072570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb16a00640 1 -- 172.21.15.40:0/3364651749 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180b2c70 con 0x7ffb1806d330 2024-03-24T00:31:09.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb0fe00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb18145520 0x7ffb18072570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:09.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb16a00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1806d330 0x7ffb18144770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffb0801a0d0 tx=0x7ffb0801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:09.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb0ea00640 1 -- 172.21.15.40:0/3364651749 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb080aadb0 con 0x7ffb1806d330 2024-03-24T00:31:09.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb0ea00640 1 -- 172.21.15.40:0/3364651749 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb080bd3f0 con 0x7ffb1806d330 2024-03-24T00:31:09.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb17e00640 1 -- 172.21.15.40:0/3364651749 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb180b4b30 con 0x7ffb1806d330 2024-03-24T00:31:09.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb0ea00640 1 -- 172.21.15.40:0/3364651749 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb080bd7e0 con 0x7ffb1806d330 2024-03-24T00:31:09.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb17e00640 1 -- 172.21.15.40:0/3364651749 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb181608f0 con 0x7ffb1806d330 2024-03-24T00:31:09.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb17e00640 1 -- 172.21.15.40:0/3364651749 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb18073540 con 0x7ffb1806d330 2024-03-24T00:31:09.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.738+0000 7ffb0ea00640 1 -- 172.21.15.40:0/3364651749 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ffb080c6440 con 0x7ffb1806d330 2024-03-24T00:31:09.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.742+0000 7ffb0ea00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffaf4067660 0x7ffaf4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:09.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.742+0000 7ffb0ea00640 1 -- 172.21.15.40:0/3364651749 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb0811fdf0 con 0x7ffb1806d330 2024-03-24T00:31:09.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.742+0000 7ffb0fe00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffaf4067660 0x7ffaf4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:09.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.742+0000 7ffb0fe00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffaf4067660 0x7ffaf4069b20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb180732d0 tx=0x7ffb00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:09.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.746+0000 7ffb0ea00640 1 -- 172.21.15.40:0/3364651749 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ffb08016150 con 0x7ffb1806d330 2024-03-24T00:31:09.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.910+0000 7ffb17e00640 1 -- 172.21.15.40:0/3364651749 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb1804f4f0 con 0x7ffb1806d330 2024-03-24T00:31:09.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb0ea00640 1 -- 172.21.15.40:0/3364651749 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7ffb080f0a60 con 0x7ffb1806d330 2024-03-24T00:31:09.916 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:09.916 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:09.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb17e00640 1 -- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffaf4067660 msgr2=0x7ffaf4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb17e00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffaf4067660 0x7ffaf4069b20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffb180732d0 tx=0x7ffb00008040 comp rx=0 tx=0).stop 2024-03-24T00:31:09.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb17e00640 1 -- 172.21.15.40:0/3364651749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1806d330 msgr2=0x7ffb18144770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:09.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb17e00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1806d330 0x7ffb18144770 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffb0801a0d0 tx=0x7ffb0801ace0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb17e00640 1 -- 172.21.15.40:0/3364651749 shutdown_connections 2024-03-24T00:31:09.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb17e00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffaf4067660 0x7ffaf4069b20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb17e00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1815fe60 0x7ffb1813a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb17e00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb18145520 0x7ffb18072570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb17e00640 1 --2- 172.21.15.40:0/3364651749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1806d330 0x7ffb18144770 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:09.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb17e00640 1 -- 172.21.15.40:0/3364651749 >> 172.21.15.40:0/3364651749 conn(0x7ffb180b1740 msgr2=0x7ffb18155150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:09.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb17e00640 1 -- 172.21.15.40:0/3364651749 shutdown_connections 2024-03-24T00:31:09.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:09.914+0000 7ffb17e00640 1 -- 172.21.15.40:0/3364651749 wait complete. 2024-03-24T00:31:09.937 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-24T00:31:09.937 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:10.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.074+0000 7fcc86000640 1 Processor -- start 2024-03-24T00:31:10.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.074+0000 7fcc86000640 1 -- start start 2024-03-24T00:31:10.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.074+0000 7fcc86000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcc80072510 0x7fcc80072910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.074+0000 7fcc86000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8006d900 con 0x7fcc8015d890 2024-03-24T00:31:10.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.074+0000 7fcc86000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8006db70 con 0x7fcc800cd1a0 2024-03-24T00:31:10.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.074+0000 7fcc86000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc800cd580 con 0x7fcc80072510 2024-03-24T00:31:10.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.078+0000 7fcc85600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcc8015d890 0x7fcc8015dd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57862/0 (socket says 172.21.15.40:57862) 2024-03-24T00:31:10.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.078+0000 7fcc85600640 1 -- 172.21.15.40:0/1127536293 learned_addr learned my addr 172.21.15.40:0/1127536293 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:10.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.078+0000 7fcc7fe00640 1 --2- 172.21.15.40:0/1127536293 >> v2:172.21.15.53:3300/0 conn(0x7fcc80072510 0x7fcc80072910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.078+0000 7fcc7f400640 1 -- 172.21.15.40:0/1127536293 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920313564 0 0) 0x7fcc8006d900 con 0x7fcc8015d890 2024-03-24T00:31:10.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.078+0000 7fcc7f400640 1 -- 172.21.15.40:0/1127536293 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc68003530 con 0x7fcc8015d890 2024-03-24T00:31:10.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.078+0000 7fcc7fe00640 1 -- 172.21.15.40:0/1127536293 >> v1:172.21.15.53:6789/0 conn(0x7fcc800cd1a0 legacy=0x7fcc8006d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.078+0000 7fcc7fe00640 1 -- 172.21.15.40:0/1127536293 >> v1:172.21.15.40:6789/0 conn(0x7fcc8015d890 legacy=0x7fcc8015dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.078+0000 7fcc7fe00640 1 -- 172.21.15.40:0/1127536293 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8006de30 con 0x7fcc80072510 2024-03-24T00:31:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.078+0000 7fcc7fe00640 1 --2- 172.21.15.40:0/1127536293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc80072510 0x7fcc80072910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0097c0 tx=0x7fcc6c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4772161308476a60 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc7f400640 1 -- 172.21.15.40:0/1127536293 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c0b1070 con 0x7fcc80072510 2024-03-24T00:31:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc7f400640 1 -- 172.21.15.40:0/1127536293 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c0ac5f0 con 0x7fcc80072510 2024-03-24T00:31:10.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc7f400640 1 -- 172.21.15.40:0/1127536293 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c0ac9c0 con 0x7fcc80072510 2024-03-24T00:31:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 -- 172.21.15.40:0/1127536293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc80072510 msgr2=0x7fcc80072910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 --2- 172.21.15.40:0/1127536293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc80072510 0x7fcc80072910 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0097c0 tx=0x7fcc6c01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 -- 172.21.15.40:0/1127536293 shutdown_connections 2024-03-24T00:31:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 --2- 172.21.15.40:0/1127536293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc80072510 0x7fcc80072910 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 -- 172.21.15.40:0/1127536293 >> 172.21.15.40:0/1127536293 conn(0x7fcc800b1c80 msgr2=0x7fcc800b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 -- 172.21.15.40:0/1127536293 shutdown_connections 2024-03-24T00:31:10.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 -- 172.21.15.40:0/1127536293 wait complete. 2024-03-24T00:31:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 Processor -- start 2024-03-24T00:31:10.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 -- start start 2024-03-24T00:31:10.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80072510 0x7fcc800c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc800cd1a0 0x7fcc8015bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8015d890 0x7fcc8015c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8009e0a0 con 0x7fcc80072510 2024-03-24T00:31:10.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8009df20 con 0x7fcc8015d890 2024-03-24T00:31:10.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc86000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc80169890 con 0x7fcc800cd1a0 2024-03-24T00:31:10.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc84c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80072510 0x7fcc800c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc84c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80072510 0x7fcc800c7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46928/0 (socket says 172.21.15.40:46928) 2024-03-24T00:31:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.082+0000 7fcc84c00640 1 -- 172.21.15.40:0/2583966288 learned_addr learned my addr 172.21.15.40:0/2583966288 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc85600640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8015d890 0x7fcc8015c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc7fe00640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc800cd1a0 0x7fcc8015bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc84c00640 1 -- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc800cd1a0 msgr2=0x7fcc8015bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc84c00640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc800cd1a0 0x7fcc8015bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc84c00640 1 -- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8015d890 msgr2=0x7fcc8015c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc84c00640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8015d890 0x7fcc8015c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc84c00640 1 -- 172.21.15.40:0/2583966288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc800ca010 con 0x7fcc80072510 2024-03-24T00:31:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc84c00640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80072510 0x7fcc800c7960 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcc7400c370 tx=0x7fcc7400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc7cc00640 1 -- 172.21.15.40:0/2583966288 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc740052d0 con 0x7fcc80072510 2024-03-24T00:31:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc7cc00640 1 -- 172.21.15.40:0/2583966288 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc74005470 con 0x7fcc80072510 2024-03-24T00:31:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc7cc00640 1 -- 172.21.15.40:0/2583966288 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74007260 con 0x7fcc80072510 2024-03-24T00:31:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc86000640 1 -- 172.21.15.40:0/2583966288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc800cddc0 con 0x7fcc80072510 2024-03-24T00:31:10.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc86000640 1 -- 172.21.15.40:0/2583966288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc800ce380 con 0x7fcc80072510 2024-03-24T00:31:10.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.086+0000 7fcc86000640 1 -- 172.21.15.40:0/2583966288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8009e2e0 con 0x7fcc80072510 2024-03-24T00:31:10.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.090+0000 7fcc7cc00640 1 -- 172.21.15.40:0/2583966288 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcc74007400 con 0x7fcc80072510 2024-03-24T00:31:10.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.090+0000 7fcc7cc00640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc60067660 0x7fcc60069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.090+0000 7fcc7cc00640 1 -- 172.21.15.40:0/2583966288 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc74084bc0 con 0x7fcc80072510 2024-03-24T00:31:10.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.090+0000 7fcc7fe00640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc60067660 0x7fcc60069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.090+0000 7fcc7fe00640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc60067660 0x7fcc60069b20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0ae000 tx=0x7fcc6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.094+0000 7fcc7cc00640 1 -- 172.21.15.40:0/2583966288 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcc74055280 con 0x7fcc80072510 2024-03-24T00:31:10.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.258+0000 7fcc86000640 1 -- 172.21.15.40:0/2583966288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc800c3890 con 0x7fcc80072510 2024-03-24T00:31:10.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.258+0000 7fcc7cc00640 1 -- 172.21.15.40:0/2583966288 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7fcc74059320 con 0x7fcc80072510 2024-03-24T00:31:10.261 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:10.261 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:10.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.262+0000 7fcc86000640 1 -- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc60067660 msgr2=0x7fcc60069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.262+0000 7fcc86000640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc60067660 0x7fcc60069b20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0ae000 tx=0x7fcc6c008040 comp rx=0 tx=0).stop 2024-03-24T00:31:10.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.262+0000 7fcc86000640 1 -- 172.21.15.40:0/2583966288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80072510 msgr2=0x7fcc800c7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.262+0000 7fcc86000640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80072510 0x7fcc800c7960 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcc7400c370 tx=0x7fcc7400c830 comp rx=0 tx=0).stop 2024-03-24T00:31:10.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.262+0000 7fcc86000640 1 -- 172.21.15.40:0/2583966288 shutdown_connections 2024-03-24T00:31:10.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.262+0000 7fcc86000640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc60067660 0x7fcc60069b20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.262+0000 7fcc86000640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8015d890 0x7fcc8015c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.262+0000 7fcc86000640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc800cd1a0 0x7fcc8015bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.262+0000 7fcc86000640 1 --2- 172.21.15.40:0/2583966288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc80072510 0x7fcc800c7960 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.262+0000 7fcc86000640 1 -- 172.21.15.40:0/2583966288 >> 172.21.15.40:0/2583966288 conn(0x7fcc800b1c80 msgr2=0x7fcc800ca4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.262+0000 7fcc86000640 1 -- 172.21.15.40:0/2583966288 shutdown_connections 2024-03-24T00:31:10.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.262+0000 7fcc86000640 1 -- 172.21.15.40:0/2583966288 wait complete. 2024-03-24T00:31:10.293 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-24T00:31:10.293 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-24T00:31:10.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-24T00:31:10.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-24T00:31:10.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-24T00:31:10.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-24T00:31:10.295 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-24T00:31:10.296 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-24T00:31:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.546+0000 7fd024e00640 1 Processor -- start 2024-03-24T00:31:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.546+0000 7fd024e00640 1 -- start start 2024-03-24T00:31:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd024e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd02015e0a0 0x7fd0200ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd024e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0200cf240 0x7fd0200d0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd024e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd0200d0aa0 0x7fd0200b5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd024e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd02015e480 con 0x7fd02015e0a0 2024-03-24T00:31:10.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd024e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0200d2c10 con 0x7fd0200d0aa0 2024-03-24T00:31:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd024e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200d0f10 con 0x7fd0200cf240 2024-03-24T00:31:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd01ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0200cf240 0x7fd0200d0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd01ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0200cf240 0x7fd0200d0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:53898/0 (socket says 172.21.15.40:53898) 2024-03-24T00:31:10.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd01ea00640 1 -- 172.21.15.40:0/1617181808 learned_addr learned my addr 172.21.15.40:0/1617181808 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd01fe00640 1 --2- 172.21.15.40:0/1617181808 >> v2:172.21.15.53:3300/0 conn(0x7fd0200d0aa0 0x7fd0200b5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd01ea00640 1 -- 172.21.15.40:0/1617181808 >> v2:172.21.15.53:3300/0 conn(0x7fd0200d0aa0 msgr2=0x7fd0200b5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd01ea00640 1 --2- 172.21.15.40:0/1617181808 >> v2:172.21.15.53:3300/0 conn(0x7fd0200d0aa0 0x7fd0200b5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd01ea00640 1 -- 172.21.15.40:0/1617181808 >> v2:172.21.15.40:3300/0 conn(0x7fd02015e0a0 msgr2=0x7fd0200ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd01ea00640 1 --2- 172.21.15.40:0/1617181808 >> v2:172.21.15.40:3300/0 conn(0x7fd02015e0a0 0x7fd0200ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.550+0000 7fd01ea00640 1 -- 172.21.15.40:0/1617181808 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0200d2ed0 con 0x7fd0200cf240 2024-03-24T00:31:10.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7ff236c00640 1 Processor -- start 2024-03-24T00:31:10.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7f22a3e00640 1 Processor -- start 2024-03-24T00:31:10.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7ff236c00640 1 -- start start 2024-03-24T00:31:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7fcadd600640 1 Processor -- start 2024-03-24T00:31:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7f22a3e00640 1 -- start start 2024-03-24T00:31:10.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7fcadd600640 1 -- start start 2024-03-24T00:31:10.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7ff236c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff2300d2d20 0x7ff2300d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7ff236c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff2300d0e30 0x7ff2300b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7ff236c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2300cf5b0 con 0x7ff23015e180 2024-03-24T00:31:10.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7ff236c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2300d1260 con 0x7ff2300d0e30 2024-03-24T00:31:10.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7ff236c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff23015e560 con 0x7ff2300d2d20 2024-03-24T00:31:10.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7fcadd600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcad80d05c0 0x7fcad815db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7fcadd600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcad806cc80 0x7fcad8140510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7f22a3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22a4162250 0x7f22a4162700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7f22a3e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a4070c70 con 0x7f22a406e770 2024-03-24T00:31:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7f22a3e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a4070f50 con 0x7f22a406e390 2024-03-24T00:31:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7f22a3e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a4071880 con 0x7f22a4162250 2024-03-24T00:31:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7ff22f400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff2300d2d20 0x7ff2300d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7ff22fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff23015e180 0x7ff2300d34e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57874/0 (socket says 172.21.15.40:57874) 2024-03-24T00:31:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7ff22fe00640 1 -- 172.21.15.40:0/3038658639 learned_addr learned my addr 172.21.15.40:0/3038658639 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7fcad6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcad80d05c0 0x7fcad815db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7fcad6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcad80d05c0 0x7fcad815db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46950/0 (socket says 172.21.15.40:46950) 2024-03-24T00:31:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a2a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f22a406e390 0x7f22a40b3450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58336/0 (socket says 172.21.15.40:58336) 2024-03-24T00:31:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a2a00640 1 -- 172.21.15.40:0/2098494962 learned_addr learned my addr 172.21.15.40:0/2098494962 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:10.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a3400640 1 --2- 172.21.15.40:0/2098494962 >> v2:172.21.15.53:3300/0 conn(0x7f22a4162250 0x7f22a4162700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.554+0000 7ff234c00640 1 --2- 172.21.15.40:0/3038658639 >> v2:172.21.15.40:3300/0 conn(0x7ff2300d0e30 0x7ff2300b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7ff22f400640 1 -- 172.21.15.40:0/3038658639 >> v2:172.21.15.40:3300/0 conn(0x7ff2300d0e30 msgr2=0x7ff2300b6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fcad6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcad806cc80 0x7fcad8140510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fcadd600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad815e440 con 0x7fcad806d7a0 2024-03-24T00:31:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fcadd600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad8140a50 con 0x7fcad80d05c0 2024-03-24T00:31:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fcadd600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad8070d40 con 0x7fcad806cc80 2024-03-24T00:31:10.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317748932 0 0) 0x7f22a4070f50 con 0x7f22a406e390 2024-03-24T00:31:10.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2288003530 con 0x7f22a406e390 2024-03-24T00:31:10.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7ff22ea00640 1 -- 172.21.15.40:0/3038658639 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708307472 0 0) 0x7ff2300cf5b0 con 0x7ff23015e180 2024-03-24T00:31:10.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fcad6a00640 1 -- 172.21.15.40:0/1929970058 learned_addr learned my addr 172.21.15.40:0/1929970058 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fd01ea00640 1 --2- 172.21.15.40:0/1617181808 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0200cf240 0x7fd0200d0560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd020067690 tx=0x7fd01801c730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91c75286a2db0f74 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2615691133 0 0) 0x7f2288003530 con 0x7f22a406e390 2024-03-24T00:31:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 >> v2:172.21.15.53:3300/0 conn(0x7f22a4162250 msgr2=0x7f22a4162700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 --2- 172.21.15.40:0/2098494962 >> v2:172.21.15.53:3300/0 conn(0x7f22a4162250 0x7f22a4162700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 >> v1:172.21.15.40:6789/0 conn(0x7f22a406e770 legacy=0x7f22a4161b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a4071230 con 0x7f22a406e390 2024-03-24T00:31:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f2290001680 con 0x7f22a406e390 2024-03-24T00:31:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 >> v1:172.21.15.53:6789/0 conn(0x7f22a406e390 legacy=0x7f22a40b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 --2- 172.21.15.40:0/2098494962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22880039c0 0x7f2288003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 --2- 172.21.15.40:0/2098494962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2288004810 0x7f2288004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 --2- 172.21.15.40:0/2098494962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2288005240 0x7f228800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4071880 con 0x7f2288004810 2024-03-24T00:31:10.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228800c1a0 con 0x7f22880039c0 2024-03-24T00:31:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fcad6000640 1 -- 172.21.15.40:0/1929970058 >> v2:172.21.15.40:3300/0 conn(0x7fcad80d05c0 msgr2=0x7fcad815db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fcad6000640 1 --2- 172.21.15.40:0/1929970058 >> v2:172.21.15.40:3300/0 conn(0x7fcad80d05c0 0x7fcad815db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228800c3a0 con 0x7f2288005240 2024-03-24T00:31:10.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fd01e000640 1 -- 172.21.15.40:0/1617181808 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0180a64b0 con 0x7fd0200cf240 2024-03-24T00:31:10.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fd01e000640 1 -- 172.21.15.40:0/1617181808 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0180a6650 con 0x7fd0200cf240 2024-03-24T00:31:10.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fd01e000640 1 -- 172.21.15.40:0/1617181808 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0180a6a20 con 0x7fd0200cf240 2024-03-24T00:31:10.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a3400640 1 --2- 172.21.15.40:0/2098494962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2288005240 0x7f228800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7f22a2a00640 1 --2- 172.21.15.40:0/2098494962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22880039c0 0x7f2288003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fcad6000640 1 -- 172.21.15.40:0/1929970058 >> v1:172.21.15.40:6789/0 conn(0x7fcad806d7a0 legacy=0x7fcad815a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7fcad6000640 1 -- 172.21.15.40:0/1929970058 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad815e720 con 0x7fcad806cc80 2024-03-24T00:31:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7f22a2a00640 1 -- 172.21.15.40:0/2098494962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2288005240 msgr2=0x7f228800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7f22a2a00640 1 --2- 172.21.15.40:0/2098494962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2288005240 0x7f228800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7f22a2a00640 1 -- 172.21.15.40:0/2098494962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2288004810 msgr2=0x7f2288004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7f22a2a00640 1 --2- 172.21.15.40:0/2098494962 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2288004810 0x7f2288004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7ff22f400640 1 --2- 172.21.15.40:0/3038658639 >> v2:172.21.15.40:3300/0 conn(0x7ff2300d0e30 0x7ff2300b6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7ff22f400640 1 -- 172.21.15.40:0/3038658639 >> v1:172.21.15.40:6789/0 conn(0x7ff23015e180 legacy=0x7ff2300d34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7ff22f400640 1 -- 172.21.15.40:0/3038658639 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2300d1c80 con 0x7ff2300d2d20 2024-03-24T00:31:10.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.558+0000 7ff234c00640 1 --2- 172.21.15.40:0/3038658639 >> v2:172.21.15.40:3300/0 conn(0x7ff2300d0e30 0x7ff2300b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:10.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7fcad6a00640 1 --2- 172.21.15.40:0/1929970058 >> v2:172.21.15.40:3300/0 conn(0x7fcad80d05c0 0x7fcad815db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7f22a2a00640 1 -- 172.21.15.40:0/2098494962 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2288003530 con 0x7f22880039c0 2024-03-24T00:31:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7f22a3400640 1 -- 172.21.15.40:0/2098494962 reap_dead start 2024-03-24T00:31:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7fd024e00640 1 -- 172.21.15.40:0/1617181808 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0200cf240 msgr2=0x7fd0200d0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7fd024e00640 1 --2- 172.21.15.40:0/1617181808 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0200cf240 0x7fd0200d0560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd020067690 tx=0x7fd01801c730 comp rx=0 tx=0).stop 2024-03-24T00:31:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7fd024e00640 1 -- 172.21.15.40:0/1617181808 shutdown_connections 2024-03-24T00:31:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7fd024e00640 1 --2- 172.21.15.40:0/1617181808 >> v2:172.21.15.53:3300/0 conn(0x7fd0200d0aa0 0x7fd0200b5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7fd024e00640 1 --2- 172.21.15.40:0/1617181808 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0200cf240 0x7fd0200d0560 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7fd024e00640 1 --2- 172.21.15.40:0/1617181808 >> v2:172.21.15.40:3300/0 conn(0x7fd02015e0a0 0x7fd0200ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7fd024e00640 1 -- 172.21.15.40:0/1617181808 >> 172.21.15.40:0/1617181808 conn(0x7fd020071d70 msgr2=0x7fd020066e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7fd024e00640 1 -- 172.21.15.40:0/1617181808 shutdown_connections 2024-03-24T00:31:10.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.562+0000 7fd024e00640 1 -- 172.21.15.40:0/1617181808 wait complete. 2024-03-24T00:31:10.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fcad6000640 1 --2- 172.21.15.40:0/1929970058 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad806cc80 0x7fcad8140510 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcad806eba0 tx=0x7fcac401ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70ed264dff28b8a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fd024e00640 1 Processor -- start 2024-03-24T00:31:10.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fcad5600640 1 -- 172.21.15.40:0/1929970058 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac40b5070 con 0x7fcad806cc80 2024-03-24T00:31:10.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7f22a2a00640 1 --2- 172.21.15.40:0/2098494962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22880039c0 0x7f2288003dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f229000c190 tx=0x7f229001cc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad0d956cf966b735 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fcad5600640 1 -- 172.21.15.40:0/1929970058 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac40a8990 con 0x7fcad806cc80 2024-03-24T00:31:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7ff22f400640 1 --2- 172.21.15.40:0/3038658639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2300d2d20 0x7ff2300d08f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff2300b6b20 tx=0x7ff22801c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f4f040a7be8ec4d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7ff22ea00640 1 -- 172.21.15.40:0/3038658639 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2280b2070 con 0x7ff2300d2d20 2024-03-24T00:31:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fcad5600640 1 -- 172.21.15.40:0/1929970058 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac40a8da0 con 0x7fcad806cc80 2024-03-24T00:31:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290007c70 con 0x7f22880039c0 2024-03-24T00:31:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22900a9070 con 0x7f22880039c0 2024-03-24T00:31:10.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7f22a1600640 1 -- 172.21.15.40:0/2098494962 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2290005d60 con 0x7f22880039c0 2024-03-24T00:31:10.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7ff22ea00640 1 -- 172.21.15.40:0/3038658639 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2280a58e0 con 0x7ff2300d2d20 2024-03-24T00:31:10.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7ff22ea00640 1 -- 172.21.15.40:0/3038658639 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2280a5cb0 con 0x7ff2300d2d20 2024-03-24T00:31:10.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7f22a3e00640 1 -- 172.21.15.40:0/2098494962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22880039c0 msgr2=0x7f2288003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7f22a3e00640 1 --2- 172.21.15.40:0/2098494962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22880039c0 0x7f2288003dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f229000c190 tx=0x7f229001cc00 comp rx=0 tx=0).stop 2024-03-24T00:31:10.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fd024e00640 1 -- start start 2024-03-24T00:31:10.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7f22a3e00640 1 -- 172.21.15.40:0/2098494962 shutdown_connections 2024-03-24T00:31:10.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7f22a3e00640 1 --2- 172.21.15.40:0/2098494962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22880039c0 0x7f2288003dc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7f22a3e00640 1 -- 172.21.15.40:0/2098494962 >> 172.21.15.40:0/2098494962 conn(0x7f22a40b1740 msgr2=0x7f22a40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fcadd600640 1 -- 172.21.15.40:0/1929970058 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad806cc80 msgr2=0x7fcad8140510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fcadd600640 1 --2- 172.21.15.40:0/1929970058 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad806cc80 0x7fcad8140510 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcad806eba0 tx=0x7fcac401ec90 comp rx=0 tx=0).stop 2024-03-24T00:31:10.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7ff236c00640 1 -- 172.21.15.40:0/3038658639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2300d2d20 msgr2=0x7ff2300d08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7ff236c00640 1 --2- 172.21.15.40:0/3038658639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2300d2d20 0x7ff2300d08f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff2300b6b20 tx=0x7ff22801c930 comp rx=0 tx=0).stop 2024-03-24T00:31:10.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fcadd600640 1 -- 172.21.15.40:0/1929970058 shutdown_connections 2024-03-24T00:31:10.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7f22a3e00640 1 -- 172.21.15.40:0/2098494962 shutdown_connections 2024-03-24T00:31:10.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7f22a3e00640 1 -- 172.21.15.40:0/2098494962 wait complete. 2024-03-24T00:31:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fcadd600640 1 --2- 172.21.15.40:0/1929970058 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad806cc80 0x7fcad8140510 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7ff236c00640 1 -- 172.21.15.40:0/3038658639 shutdown_connections 2024-03-24T00:31:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7ff236c00640 1 --2- 172.21.15.40:0/3038658639 >> v2:172.21.15.40:3300/0 conn(0x7ff2300d0e30 0x7ff2300b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7ff236c00640 1 --2- 172.21.15.40:0/3038658639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2300d2d20 0x7ff2300d08f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7ff236c00640 1 -- 172.21.15.40:0/3038658639 >> 172.21.15.40:0/3038658639 conn(0x7ff230066e80 msgr2=0x7ff230067290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7f2e20e00640 1 Processor -- start 2024-03-24T00:31:10.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fcadd600640 1 --2- 172.21.15.40:0/1929970058 >> v2:172.21.15.40:3300/0 conn(0x7fcad80d05c0 0x7fcad815db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7f2e20e00640 1 -- start start 2024-03-24T00:31:10.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd024e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0200d0aa0 0x7fd02015b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd024e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd02015e0a0 0x7fd0201f0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd024e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0200cb9f0 0x7fd0201f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd024e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200b72a0 con 0x7fd0200cb9f0 2024-03-24T00:31:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd024e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200b7120 con 0x7fd0200d0aa0 2024-03-24T00:31:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd024e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200b7420 con 0x7fd02015e0a0 2024-03-24T00:31:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd01f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0200d0aa0 0x7fd02015b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd01fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0200cb9f0 0x7fd0201f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd01f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0200d0aa0 0x7fd02015b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39560/0 (socket says 172.21.15.40:39560) 2024-03-24T00:31:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd01f400640 1 -- 172.21.15.40:0/481339343 learned_addr learned my addr 172.21.15.40:0/481339343 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd01ea00640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd02015e0a0 0x7fd0201f0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd01ea00640 1 -- 172.21.15.40:0/481339343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0200d0aa0 msgr2=0x7fd02015b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd01ea00640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0200d0aa0 0x7fd02015b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd01ea00640 1 -- 172.21.15.40:0/481339343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0200cb9f0 msgr2=0x7fd0201f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd01ea00640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd0200cb9f0 0x7fd0201f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd01ea00640 1 -- 172.21.15.40:0/481339343 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0200710e0 con 0x7fd02015e0a0 2024-03-24T00:31:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd01ea00640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd02015e0a0 0x7fd0201f0970 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd0180ab000 tx=0x7fd0180194d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7f22a3e00640 1 Processor -- start 2024-03-24T00:31:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7ff236c00640 1 -- 172.21.15.40:0/3038658639 shutdown_connections 2024-03-24T00:31:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7ff236c00640 1 -- 172.21.15.40:0/3038658639 wait complete. 2024-03-24T00:31:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7f2e20e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2e1c1443b0 0x7f2e1c0b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7f2e20e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2e1c162160 0x7f2e1c06dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7f2e20e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1c0b4f40 con 0x7f2e1c06db40 2024-03-24T00:31:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7f2e20e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1c06e4e0 con 0x7f2e1c162160 2024-03-24T00:31:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7f2e20e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e1c0b4570 con 0x7f2e1c1443b0 2024-03-24T00:31:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7f2e19e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2e1c1443b0 0x7f2e1c0b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7f2e19e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2e1c1443b0 0x7f2e1c0b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39570/0 (socket says 172.21.15.40:39570) 2024-03-24T00:31:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7f2e19e00640 1 -- 172.21.15.40:0/154173206 learned_addr learned my addr 172.21.15.40:0/154173206 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd017400640 1 -- 172.21.15.40:0/481339343 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0180bf070 con 0x7fd02015e0a0 2024-03-24T00:31:10.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7f22a3e00640 1 -- start start 2024-03-24T00:31:10.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd017400640 1 -- 172.21.15.40:0/481339343 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd018009150 con 0x7fd02015e0a0 2024-03-24T00:31:10.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd017400640 1 -- 172.21.15.40:0/481339343 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0180a6a60 con 0x7fd02015e0a0 2024-03-24T00:31:10.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7ff236c00640 1 Processor -- start 2024-03-24T00:31:10.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fcadd600640 1 -- 172.21.15.40:0/1929970058 >> 172.21.15.40:0/1929970058 conn(0x7fcad80cfaa0 msgr2=0x7fcad80cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fcadd600640 1 -- 172.21.15.40:0/1929970058 shutdown_connections 2024-03-24T00:31:10.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.566+0000 7fcadd600640 1 -- 172.21.15.40:0/1929970058 wait complete. 2024-03-24T00:31:10.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.570+0000 7fd024e00640 1 -- 172.21.15.40:0/481339343 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0200caf10 con 0x7fd02015e0a0 2024-03-24T00:31:10.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f2e1a800640 1 --2- 172.21.15.40:0/154173206 >> v2:172.21.15.40:3300/0 conn(0x7f2e1c162160 0x7f2e1c06dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7fd024e00640 1 -- 172.21.15.40:0/481339343 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0200cb4d0 con 0x7fd02015e0a0 2024-03-24T00:31:10.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22880039c0 0x7f22a4154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f22a4146ef0 0x7f22a41549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22a40b0730 0x7f22a40b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4071e90 con 0x7f22880039c0 2024-03-24T00:31:10.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4071d10 con 0x7f22a40b0730 2024-03-24T00:31:10.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a3e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a4162700 con 0x7f22a4146ef0 2024-03-24T00:31:10.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f2e18a00640 1 -- 172.21.15.40:0/154173206 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191646424 0 0) 0x7f2e1c0b4f40 con 0x7f2e1c06db40 2024-03-24T00:31:10.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f2e18a00640 1 -- 172.21.15.40:0/154173206 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e00003530 con 0x7f2e1c06db40 2024-03-24T00:31:10.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22a40b0730 0x7f22a40b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22a40b0730 0x7f22a40b0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39572/0 (socket says 172.21.15.40:39572) 2024-03-24T00:31:10.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a3400640 1 -- 172.21.15.40:0/806279970 learned_addr learned my addr 172.21.15.40:0/806279970 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f2e1a800640 1 -- 172.21.15.40:0/154173206 >> v2:172.21.15.53:3300/0 conn(0x7f2e1c1443b0 msgr2=0x7f2e1c0b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a2000640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f22a4146ef0 0x7f22a41549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f2e1a800640 1 --2- 172.21.15.40:0/154173206 >> v2:172.21.15.53:3300/0 conn(0x7f2e1c1443b0 0x7f2e1c0b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a2a00640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22880039c0 0x7f22a4154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f2e1a800640 1 -- 172.21.15.40:0/154173206 >> v1:172.21.15.53:6789/0 conn(0x7f2e1c06db40 legacy=0x7f2e1c161a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f2e18a00640 1 -- 172.21.15.40:0/154173206 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3416138309 0 0) 0x7f2e00003530 con 0x7f2e1c06db40 2024-03-24T00:31:10.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7fb817600640 1 Processor -- start 2024-03-24T00:31:10.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7fb817600640 1 -- start start 2024-03-24T00:31:10.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fb817600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb81015e240 0x7fb8100d35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fb817600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8100d2140 con 0x7fb8100d3b30 2024-03-24T00:31:10.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fb817600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8100d2420 con 0x7fb8100d10f0 2024-03-24T00:31:10.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fb817600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8100d31e0 con 0x7fb81015e240 2024-03-24T00:31:10.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7fd014c00640 1 -- 172.21.15.40:0/481339343 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd02004f480 con 0x7fd02015e0a0 2024-03-24T00:31:10.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7fd017400640 1 -- 172.21.15.40:0/481339343 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd0180a6c00 con 0x7fd02015e0a0 2024-03-24T00:31:10.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fd017400640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd000067760 0x7fd000069c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fd017400640 1 -- 172.21.15.40:0/481339343 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd01811f450 con 0x7fd02015e0a0 2024-03-24T00:31:10.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fd017400640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.40:6808/4268319376,v1:172.21.15.40:6812/4268319376] conn(0x7fd00006f980 0x7fd000071da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fd017400640 1 -- 172.21.15.40:0/481339343 --> [v2:172.21.15.40:6808/4268319376,v1:172.21.15.40:6812/4268319376] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0180a6650 con 0x7fd00006f980 2024-03-24T00:31:10.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fd017400640 1 -- 172.21.15.40:0/481339343 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd018126050 con 0x7fd02015e0a0 2024-03-24T00:31:10.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a2000640 1 -- 172.21.15.40:0/806279970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22a40b0730 msgr2=0x7f22a40b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a2000640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22a40b0730 0x7f22a40b0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a2000640 1 -- 172.21.15.40:0/806279970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22880039c0 msgr2=0x7f22a4154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a2000640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f22880039c0 0x7f22a4154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a2000640 1 -- 172.21.15.40:0/806279970 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a414bc40 con 0x7f22a4146ef0 2024-03-24T00:31:10.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a2000640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f22a4146ef0 0x7f22a41549b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2294009cc0 tx=0x7f229400c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f229aa00640 1 -- 172.21.15.40:0/806279970 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2294018030 con 0x7f22a4146ef0 2024-03-24T00:31:10.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f229aa00640 1 -- 172.21.15.40:0/806279970 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229400cd50 con 0x7f22a4146ef0 2024-03-24T00:31:10.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f229aa00640 1 -- 172.21.15.40:0/806279970 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2294004260 con 0x7f22a4146ef0 2024-03-24T00:31:10.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a3e00640 1 -- 172.21.15.40:0/806279970 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a414aba0 con 0x7f22a4146ef0 2024-03-24T00:31:10.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f22a3e00640 1 -- 172.21.15.40:0/806279970 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a414b160 con 0x7f22a4146ef0 2024-03-24T00:31:10.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7ff236c00640 1 -- start start 2024-03-24T00:31:10.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7fcadd600640 1 Processor -- start 2024-03-24T00:31:10.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7fcadd600640 1 -- start start 2024-03-24T00:31:10.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f2e1a800640 1 -- 172.21.15.40:0/154173206 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c0b3f60 con 0x7f2e1c162160 2024-03-24T00:31:10.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.574+0000 7f2e19e00640 1 --2- 172.21.15.40:0/154173206 >> v2:172.21.15.53:3300/0 conn(0x7f2e1c1443b0 0x7f2e1c0b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:10.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fb815600640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb8100d10f0 0x7fb8100b7060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58360/0 (socket says 172.21.15.40:58360) 2024-03-24T00:31:10.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fb815600640 1 -- 172.21.15.40:0/2475761861 learned_addr learned my addr 172.21.15.40:0/2475761861 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:10.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753929169 0 0) 0x7fb8100d2420 con 0x7fb8100d10f0 2024-03-24T00:31:10.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcadd600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcad806cc80 0x7fcad8142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcadd600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcad806d7a0 0x7fcad81f14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcadd600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80d05c0 0x7fcad80b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcadd600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad80721e0 con 0x7fcad806d7a0 2024-03-24T00:31:10.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcadd600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad815ea00 con 0x7fcad80d05c0 2024-03-24T00:31:10.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcadd600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad8072360 con 0x7fcad806cc80 2024-03-24T00:31:10.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7f229aa00640 1 -- 172.21.15.40:0/806279970 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f229401b070 con 0x7f22a4146ef0 2024-03-24T00:31:10.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f4003530 con 0x7fb8100d10f0 2024-03-24T00:31:10.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7ff236c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2300d0e30 0x7ff23015ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7ff236c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2300d2d20 0x7ff230206080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7ff236c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff23015e180 0x7ff230160500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcad6a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcad806cc80 0x7fcad8142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7ff22fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2300d0e30 0x7ff23015ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7ff22fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2300d0e30 0x7ff23015ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39598/0 (socket says 172.21.15.40:39598) 2024-03-24T00:31:10.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcad6a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcad806cc80 0x7fcad8142a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:53938/0 (socket says 172.21.15.40:53938) 2024-03-24T00:31:10.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcad6a00640 1 -- 172.21.15.40:0/504746701 learned_addr learned my addr 172.21.15.40:0/504746701 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7ff22fe00640 1 -- 172.21.15.40:0/3116225376 learned_addr learned my addr 172.21.15.40:0/3116225376 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2300d3bf0 con 0x7ff2300d2d20 2024-03-24T00:31:10.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2300d2800 con 0x7ff2300d0e30 2024-03-24T00:31:10.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7ff234c00640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff23015e180 0x7ff230160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2300d3d70 con 0x7ff23015e180 2024-03-24T00:31:10.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcad6a00640 1 -- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80d05c0 msgr2=0x7fcad80b1020 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T00:31:10.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcad6a00640 1 --2- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcad80d05c0 0x7fcad80b1020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcad6a00640 1 -- 172.21.15.40:0/504746701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcad806d7a0 msgr2=0x7fcad81f14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.578+0000 7fcad6a00640 1 --2- 172.21.15.40:0/504746701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcad806d7a0 0x7fcad81f14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fcad6a00640 1 -- 172.21.15.40:0/504746701 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad8143100 con 0x7fcad806cc80 2024-03-24T00:31:10.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7f22a3e00640 1 -- 172.21.15.40:0/806279970 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f227c001040 con 0x7f22a4146ef0 2024-03-24T00:31:10.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7ff234c00640 1 -- 172.21.15.40:0/3116225376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2300d0e30 msgr2=0x7ff23015ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fcad6a00640 1 --2- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcad806cc80 0x7fcad8142a40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcad000efc0 tx=0x7fcad000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7ff234c00640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2300d0e30 0x7ff23015ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7ff234c00640 1 -- 172.21.15.40:0/3116225376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2300d2d20 msgr2=0x7ff230206080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7ff234c00640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff2300d2d20 0x7ff230206080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7ff234c00640 1 -- 172.21.15.40:0/3116225376 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2301616b0 con 0x7ff23015e180 2024-03-24T00:31:10.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7ff22fe00640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2300d0e30 0x7ff23015ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:10.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fd01fe00640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.40:6808/4268319376,v1:172.21.15.40:6812/4268319376] conn(0x7fd00006f980 0x7fd000071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fd01f400640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd000067760 0x7fd000069c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7f229aa00640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f22840676b0 0x7f2284069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7f229aa00640 1 -- 172.21.15.40:0/806279970 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2294087a90 con 0x7f22a4146ef0 2024-03-24T00:31:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7f22a2a00640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f22840676b0 0x7f2284069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7f229aa00640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.40:6800/2411062682,v1:172.21.15.40:6801/2411062682] conn(0x7f228406f8b0 0x7f2284071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7f229aa00640 1 -- 172.21.15.40:0/806279970 --> [v2:172.21.15.40:6800/2411062682,v1:172.21.15.40:6801/2411062682] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2294012e50 con 0x7f228406f8b0 2024-03-24T00:31:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fcacea00640 1 -- 172.21.15.40:0/504746701 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad0014480 con 0x7fcad806cc80 2024-03-24T00:31:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fcacea00640 1 -- 172.21.15.40:0/504746701 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad0019070 con 0x7fcad806cc80 2024-03-24T00:31:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fcadd600640 1 -- 172.21.15.40:0/504746701 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad81433e0 con 0x7fcad806cc80 2024-03-24T00:31:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7f22a3400640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.40:6800/2411062682,v1:172.21.15.40:6801/2411062682] conn(0x7f228406f8b0 0x7f2284071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7f229aa00640 1 -- 172.21.15.40:0/806279970 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2294057f50 con 0x7f22a4146ef0 2024-03-24T00:31:10.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7ff234c00640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff23015e180 0x7ff230160500 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff22000e630 tx=0x7ff22000eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7ff227e00640 1 -- 172.21.15.40:0/3116225376 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220017070 con 0x7ff23015e180 2024-03-24T00:31:10.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7ff227e00640 1 -- 172.21.15.40:0/3116225376 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22000c620 con 0x7ff23015e180 2024-03-24T00:31:10.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff230160e10 con 0x7ff23015e180 2024-03-24T00:31:10.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fcadd600640 1 -- 172.21.15.40:0/504746701 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad81439a0 con 0x7fcad806cc80 2024-03-24T00:31:10.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fcacea00640 1 -- 172.21.15.40:0/504746701 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad0014480 con 0x7fcad806cc80 2024-03-24T00:31:10.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb814c00640 1 --2- 172.21.15.40:0/2475761861 >> v2:172.21.15.40:3300/0 conn(0x7fb81015e240 0x7fb8100d35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567746395 0 0) 0x7fb8100d2140 con 0x7fb8100d3b30 2024-03-24T00:31:10.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8100d2420 con 0x7fb8100d3b30 2024-03-24T00:31:10.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1311700734 0 0) 0x7fb7f4003530 con 0x7fb8100d10f0 2024-03-24T00:31:10.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 >> v2:172.21.15.40:3300/0 conn(0x7fb81015e240 msgr2=0x7fb8100d35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 --2- 172.21.15.40:0/2475761861 >> v2:172.21.15.40:3300/0 conn(0x7fb81015e240 0x7fb8100d35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 >> v1:172.21.15.40:6789/0 conn(0x7fb8100d3b30 legacy=0x7fb8100d09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8100d2700 con 0x7fb8100d10f0 2024-03-24T00:31:10.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fb808001980 con 0x7fb8100d10f0 2024-03-24T00:31:10.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2301613d0 con 0x7ff23015e180 2024-03-24T00:31:10.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7ff227e00640 1 -- 172.21.15.40:0/3116225376 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220017070 con 0x7ff23015e180 2024-03-24T00:31:10.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fd01f400640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd000067760 0x7fd000069c20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0099b0 tx=0x7fd00c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fcadd600640 1 -- 172.21.15.40:0/504746701 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcad8068170 con 0x7fcad806cc80 2024-03-24T00:31:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7f2e1a800640 1 --2- 172.21.15.40:0/154173206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1c162160 0x7f2e1c06dfa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c067df0 tx=0x7f2e1401c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83520cccf26da57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 >> v1:172.21.15.53:6789/0 conn(0x7fb8100d10f0 legacy=0x7fb8100b7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 --2- 172.21.15.40:0/2475761861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7f4003b50 0x7fb7f4003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 --2- 172.21.15.40:0/2475761861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f4004810 0x7fb7f4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 --2- 172.21.15.40:0/2475761861 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7f40051d0 0x7fb7f400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100d31e0 con 0x7fb7f4004810 2024-03-24T00:31:10.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f400bff0 con 0x7fb7f4003b50 2024-03-24T00:31:10.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f400c1f0 con 0x7fb7f40051d0 2024-03-24T00:31:10.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.582+0000 7fb807e00640 1 --2- 172.21.15.40:0/2475761861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f4004810 0x7fb7f4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7fb815600640 1 --2- 172.21.15.40:0/2475761861 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7f40051d0 0x7fb7f400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7fd01fe00640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.40:6808/4268319376,v1:172.21.15.40:6812/4268319376] conn(0x7fd00006f980 0x7fd000071da0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7f2e18a00640 1 -- 172.21.15.40:0/154173206 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c001680 con 0x7f2e1c162160 2024-03-24T00:31:10.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7f2e18a00640 1 -- 172.21.15.40:0/154173206 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e140a6500 con 0x7f2e1c162160 2024-03-24T00:31:10.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7f2e18a00640 1 -- 172.21.15.40:0/154173206 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e140a68d0 con 0x7f2e1c162160 2024-03-24T00:31:10.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7fb807e00640 1 -- 172.21.15.40:0/2475761861 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7f40051d0 msgr2=0x7fb7f400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7fb807e00640 1 --2- 172.21.15.40:0/2475761861 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7f40051d0 0x7fb7f400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7fb807e00640 1 -- 172.21.15.40:0/2475761861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7f4003b50 msgr2=0x7fb7f4003f30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:31:10.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7fb807e00640 1 --2- 172.21.15.40:0/2475761861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7f4003b50 0x7fb7f4003f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7fb807e00640 1 -- 172.21.15.40:0/2475761861 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8100d2700 con 0x7fb7f4004810 2024-03-24T00:31:10.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7fd017400640 1 -- 172.21.15.40:0/481339343 <== osd.1 v2:172.21.15.40:6808/4268319376 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd0180a6650 con 0x7fd00006f980 2024-03-24T00:31:10.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff208001040 con 0x7ff23015e180 2024-03-24T00:31:10.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7fcacea00640 1 -- 172.21.15.40:0/504746701 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcad0012050 con 0x7fcad806cc80 2024-03-24T00:31:10.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7fb815600640 1 -- 172.21.15.40:0/2475761861 reap_dead start 2024-03-24T00:31:10.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7f2e20e00640 1 -- 172.21.15.40:0/154173206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1c162160 msgr2=0x7f2e1c06dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7ff227e00640 1 -- 172.21.15.40:0/3116225376 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff22001a070 con 0x7ff23015e180 2024-03-24T00:31:10.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7f2e20e00640 1 --2- 172.21.15.40:0/154173206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1c162160 0x7f2e1c06dfa0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c067df0 tx=0x7f2e1401c7c0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7f2e20e00640 1 -- 172.21.15.40:0/154173206 shutdown_connections 2024-03-24T00:31:10.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7f2e20e00640 1 --2- 172.21.15.40:0/154173206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1c162160 0x7f2e1c06dfa0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7f2e20e00640 1 --2- 172.21.15.40:0/154173206 >> v2:172.21.15.53:3300/0 conn(0x7f2e1c1443b0 0x7f2e1c0b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7f2e20e00640 1 -- 172.21.15.40:0/154173206 >> 172.21.15.40:0/154173206 conn(0x7f2e1c0b1740 msgr2=0x7f2e1c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7f2e20e00640 1 -- 172.21.15.40:0/154173206 shutdown_connections 2024-03-24T00:31:10.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7fcacea00640 1 --2- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcab8067860 0x7fcab8069d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7fcacea00640 1 -- 172.21.15.40:0/504746701 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcad0086540 con 0x7fcad806cc80 2024-03-24T00:31:10.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7fcacea00640 1 --2- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:6802/3270909220,v1:172.21.15.53:6803/3270909220] conn(0x7fcab806fa00 0x7fcab8071e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7fcacea00640 1 -- 172.21.15.40:0/504746701 --> [v2:172.21.15.53:6802/3270909220,v1:172.21.15.53:6803/3270909220] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcad001ee50 con 0x7fcab806fa00 2024-03-24T00:31:10.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.586+0000 7f2e20e00640 1 -- 172.21.15.40:0/154173206 wait complete. 2024-03-24T00:31:10.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7fcacea00640 1 -- 172.21.15.40:0/504746701 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcad0086930 con 0x7fcad806cc80 2024-03-24T00:31:10.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7f22a2a00640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f22840676b0 0x7f2284069b70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f22900ab000 tx=0x7f22900b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7f22a3400640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.40:6800/2411062682,v1:172.21.15.40:6801/2411062682] conn(0x7f228406f8b0 0x7f2284071cd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7fcad6000640 1 --2- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcab8067860 0x7fcab8069d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7fcad6000640 1 --2- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcab8067860 0x7fcab8069d20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcac40b2000 tx=0x7fcac4004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7ff227e00640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff210067860 0x7ff210069d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7ff227e00640 1 -- 172.21.15.40:0/3116225376 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff220089060 con 0x7ff23015e180 2024-03-24T00:31:10.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7ff227e00640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.40:6809/2475081617,v1:172.21.15.40:6811/2475081617] conn(0x7ff21006fa60 0x7ff210071e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7ff227e00640 1 -- 172.21.15.40:0/3116225376 --> [v2:172.21.15.40:6809/2475081617,v1:172.21.15.40:6811/2475081617] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff210072530 con 0x7ff21006fa60 2024-03-24T00:31:10.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7ff227e00640 1 -- 172.21.15.40:0/3116225376 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff220084570 con 0x7ff23015e180 2024-03-24T00:31:10.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7ff22fe00640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff210067860 0x7ff210069d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7ff22f400640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.40:6809/2475081617,v1:172.21.15.40:6811/2475081617] conn(0x7ff21006fa60 0x7ff210071e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7f2e20e00640 1 Processor -- start 2024-03-24T00:31:10.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7fb807e00640 1 --2- 172.21.15.40:0/2475761861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f4004810 0x7fb7f4004c90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8001920 tx=0x7fb7f801c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f58dd950913ef86 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7ff22f400640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.40:6809/2475081617,v1:172.21.15.40:6811/2475081617] conn(0x7ff21006fa60 0x7ff210071e80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7ff22fe00640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff210067860 0x7ff210069d20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff21c000f70 tx=0x7ff21c007ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7fcad7400640 1 --2- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:6802/3270909220,v1:172.21.15.53:6803/3270909220] conn(0x7fcab806fa00 0x7fcab8071e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f80014f0 con 0x7fb7f4004810 2024-03-24T00:31:10.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.590+0000 7f2e20e00640 1 -- start start 2024-03-24T00:31:10.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e20e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1c06db40 0x7f2e1c0ce550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7fcad7400640 1 --2- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:6802/3270909220,v1:172.21.15.53:6803/3270909220] conn(0x7fcab806fa00 0x7fcab8071e20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e20e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e1c1443b0 0x7f2e1c13a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e20e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e1c162160 0x7f2e1c145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e20e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c0712e0 con 0x7f2e1c06db40 2024-03-24T00:31:10.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e20e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c162940 con 0x7f2e1c1443b0 2024-03-24T00:31:10.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e20e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1c071460 con 0x7f2e1c162160 2024-03-24T00:31:10.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f8007020 con 0x7fb7f4004810 2024-03-24T00:31:10.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7fb80fe00640 1 -- 172.21.15.40:0/2475761861 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f8007270 con 0x7fb7f4004810 2024-03-24T00:31:10.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f229aa00640 1 -- 172.21.15.40:0/806279970 <== osd.2 v2:172.21.15.40:6800/2411062682 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2294012e50 con 0x7f228406f8b0 2024-03-24T00:31:10.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e19400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e1c1443b0 0x7f2e1c13a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e1a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e1c162160 0x7f2e1c145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e19400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e1c1443b0 0x7f2e1c13a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39622/0 (socket says 172.21.15.40:39622) 2024-03-24T00:31:10.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e19400640 1 -- 172.21.15.40:0/3611665255 learned_addr learned my addr 172.21.15.40:0/3611665255 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7ff227e00640 1 -- 172.21.15.40:0/3116225376 <== osd.3 v2:172.21.15.40:6809/2475081617 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff210072530 con 0x7ff21006fa60 2024-03-24T00:31:10.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e19e00640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1c06db40 0x7f2e1c0ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7fcacea00640 1 -- 172.21.15.40:0/504746701 <== osd.4 v2:172.21.15.53:6802/3270909220 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcad001ee50 con 0x7fcab806fa00 2024-03-24T00:31:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e1a800640 1 -- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e1c1443b0 msgr2=0x7f2e1c13a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e1a800640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e1c1443b0 0x7f2e1c13a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7fb817600640 1 -- 172.21.15.40:0/2475761861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f4004810 msgr2=0x7fb7f4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7fb817600640 1 --2- 172.21.15.40:0/2475761861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f4004810 0x7fb7f4004c90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8001920 tx=0x7fb7f801c550 comp rx=0 tx=0).stop 2024-03-24T00:31:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7fb817600640 1 -- 172.21.15.40:0/2475761861 shutdown_connections 2024-03-24T00:31:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7fb817600640 1 --2- 172.21.15.40:0/2475761861 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f4004810 0x7fb7f4004c90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7fb817600640 1 -- 172.21.15.40:0/2475761861 >> 172.21.15.40:0/2475761861 conn(0x7fb810066e80 msgr2=0x7fb810067260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e1a800640 1 -- 172.21.15.40:0/3611665255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1c06db40 msgr2=0x7f2e1c0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e1a800640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1c06db40 0x7f2e1c0ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e1a800640 1 -- 172.21.15.40:0/3611665255 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1c15b4f0 con 0x7f2e1c162160 2024-03-24T00:31:10.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7f2e19e00640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e1c06db40 0x7f2e1c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:10.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7fb817600640 1 -- 172.21.15.40:0/2475761861 shutdown_connections 2024-03-24T00:31:10.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.594+0000 7fb817600640 1 -- 172.21.15.40:0/2475761861 wait complete. 2024-03-24T00:31:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7f2e1a800640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e1c162160 0x7f2e1c145db0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2e140ac000 tx=0x7f2e14019560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7f2e12000640 1 -- 172.21.15.40:0/3611665255 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e140be020 con 0x7f2e1c162160 2024-03-24T00:31:10.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7f2e12000640 1 -- 172.21.15.40:0/3611665255 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e140ba670 con 0x7f2e1c162160 2024-03-24T00:31:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7f2e12000640 1 -- 172.21.15.40:0/3611665255 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e14009450 con 0x7f2e1c162160 2024-03-24T00:31:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7f2e20e00640 1 -- 172.21.15.40:0/3611665255 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1c0ab610 con 0x7f2e1c162160 2024-03-24T00:31:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7f2e20e00640 1 -- 172.21.15.40:0/3611665255 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1c0abbd0 con 0x7f2e1c162160 2024-03-24T00:31:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7fb817600640 1 Processor -- start 2024-03-24T00:31:10.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7fb817600640 1 -- start start 2024-03-24T00:31:10.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7f2e20e00640 1 -- 172.21.15.40:0/3611665255 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2e1c04f480 con 0x7f2e1c162160 2024-03-24T00:31:10.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7fb817600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f4004810 0x7fb8101f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7fb817600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb8101f69d0 0x7fb8100b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7fb817600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8100b5330 0x7fb8100b8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7fb817600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81015e620 con 0x7fb7f4004810 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7fb817600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100d2cc0 con 0x7fb8100b5330 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7fb817600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8100b33c0 con 0x7fb8101f69d0 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7fb814c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f4004810 0x7fb8101f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.598+0000 7fb815600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8100b5330 0x7fb8100b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb807e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb8101f69d0 0x7fb8100b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb815600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8100b5330 0x7fb8100b8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39638/0 (socket says 172.21.15.40:39638) 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb815600640 1 -- 172.21.15.40:0/794231400 learned_addr learned my addr 172.21.15.40:0/794231400 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb814c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f4004810 0x7fb8101f3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47046/0 (socket says 172.21.15.40:47046) 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb815600640 1 -- 172.21.15.40:0/794231400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb8101f69d0 msgr2=0x7fb8100b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb815600640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb8101f69d0 0x7fb8100b4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb815600640 1 -- 172.21.15.40:0/794231400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f4004810 msgr2=0x7fb8101f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb815600640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f4004810 0x7fb8101f3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb815600640 1 -- 172.21.15.40:0/794231400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8101f6e50 con 0x7fb8100b5330 2024-03-24T00:31:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb807e00640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb8101f69d0 0x7fb8100b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb814c00640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7f4004810 0x7fb8101f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb815600640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8100b5330 0x7fb8100b8100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb808001360 tx=0x7fb808011d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb80d600640 1 -- 172.21.15.40:0/794231400 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb808010040 con 0x7fb8100b5330 2024-03-24T00:31:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb80d600640 1 -- 172.21.15.40:0/794231400 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80800fa70 con 0x7fb8100b5330 2024-03-24T00:31:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb80d600640 1 -- 172.21.15.40:0/794231400 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80800fe60 con 0x7fb8100b5330 2024-03-24T00:31:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb817600640 1 -- 172.21.15.40:0/794231400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8101f7510 con 0x7fb8100b5330 2024-03-24T00:31:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7fb817600640 1 -- 172.21.15.40:0/794231400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8101f7ad0 con 0x7fb8100b5330 2024-03-24T00:31:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7f2e12000640 1 -- 172.21.15.40:0/3611665255 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2e140c1020 con 0x7f2e1c162160 2024-03-24T00:31:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.602+0000 7f2e12000640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2dfc0676b0 0x7f2dfc069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.606+0000 7fb817600640 1 -- 172.21.15.40:0/794231400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb8100b5760 con 0x7fb8100b5330 2024-03-24T00:31:10.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.606+0000 7f2e12000640 1 -- 172.21.15.40:0/3611665255 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e140ee690 con 0x7f2e1c162160 2024-03-24T00:31:10.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.606+0000 7f2e19e00640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2dfc0676b0 0x7f2dfc069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.606+0000 7f2e12000640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:6821/2592328924,v1:172.21.15.53:6824/2592328924] conn(0x7f2dfc06f8d0 0x7f2dfc071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.610+0000 7f2e12000640 1 -- 172.21.15.40:0/3611665255 --> [v2:172.21.15.53:6821/2592328924,v1:172.21.15.53:6824/2592328924] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e140a6500 con 0x7f2dfc06f8d0 2024-03-24T00:31:10.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.606+0000 7fb80d600640 1 -- 172.21.15.40:0/794231400 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb808020470 con 0x7fb8100b5330 2024-03-24T00:31:10.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.606+0000 7fb80d600640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb7ec067660 0x7fb7ec069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.606+0000 7fb80d600640 1 -- 172.21.15.40:0/794231400 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb808088ac0 con 0x7fb8100b5330 2024-03-24T00:31:10.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.610+0000 7fb80d600640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.40:6810/201988360,v1:172.21.15.40:6813/201988360] conn(0x7fb7ec06f860 0x7fb7ec071c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.610+0000 7fb80d600640 1 -- 172.21.15.40:0/794231400 --> [v2:172.21.15.40:6810/201988360,v1:172.21.15.40:6813/201988360] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb808016e50 con 0x7fb7ec06f860 2024-03-24T00:31:10.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.610+0000 7fb80d600640 1 -- 172.21.15.40:0/794231400 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb80808ae60 con 0x7fb8100b5330 2024-03-24T00:31:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.610+0000 7f2e12000640 1 -- 172.21.15.40:0/3611665255 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e14125060 con 0x7f2e1c162160 2024-03-24T00:31:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.610+0000 7f2e19400640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:6821/2592328924,v1:172.21.15.53:6824/2592328924] conn(0x7f2dfc06f8d0 0x7f2dfc071cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.610+0000 7f2e19e00640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2dfc0676b0 0x7f2dfc069b70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c146970 tx=0x7f2e08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.610+0000 7fb807e00640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.40:6810/201988360,v1:172.21.15.40:6813/201988360] conn(0x7fb7ec06f860 0x7fb7ec071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.610+0000 7fb814c00640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb7ec067660 0x7fb7ec069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.610+0000 7fb807e00640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.40:6810/201988360,v1:172.21.15.40:6813/201988360] conn(0x7fb7ec06f860 0x7fb7ec071c80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.610+0000 7fb80d600640 1 -- 172.21.15.40:0/794231400 <== osd.0 v2:172.21.15.40:6810/201988360 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb808016e50 con 0x7fb7ec06f860 2024-03-24T00:31:10.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.614+0000 7f2e19400640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:6821/2592328924,v1:172.21.15.53:6824/2592328924] conn(0x7f2dfc06f8d0 0x7f2dfc071cf0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.614+0000 7fb814c00640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb7ec067660 0x7fb7ec069b20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb800006aa0 tx=0x7fb800008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.614+0000 7f2e12000640 1 -- 172.21.15.40:0/3611665255 <== osd.7 v2:172.21.15.53:6821/2592328924 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2e140a6500 con 0x7f2dfc06f8d0 2024-03-24T00:31:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f22a3e00640 1 -- 172.21.15.40:0/806279970 --> [v2:172.21.15.40:6800/2411062682,v1:172.21.15.40:6801/2411062682] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f227c002de0 con 0x7f228406f8b0 2024-03-24T00:31:10.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f229aa00640 1 -- 172.21.15.40:0/806279970 <== osd.2 v2:172.21.15.40:6800/2411062682 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f227c002de0 con 0x7f228406f8b0 2024-03-24T00:31:10.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f22a3e00640 1 -- 172.21.15.40:0/806279970 >> [v2:172.21.15.40:6800/2411062682,v1:172.21.15.40:6801/2411062682] conn(0x7f228406f8b0 msgr2=0x7f2284071cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f22a3e00640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.40:6800/2411062682,v1:172.21.15.40:6801/2411062682] conn(0x7f228406f8b0 0x7f2284071cd0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f22a3e00640 1 -- 172.21.15.40:0/806279970 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f22840676b0 msgr2=0x7f2284069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f22a3e00640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f22840676b0 0x7f2284069b70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f22900ab000 tx=0x7f22900b0040 comp rx=0 tx=0).stop 2024-03-24T00:31:10.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f22a3e00640 1 -- 172.21.15.40:0/806279970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f22a4146ef0 msgr2=0x7f22a41549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f22a3e00640 1 --2- 172.21.15.40:0/806279970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f22a4146ef0 0x7f22a41549b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2294009cc0 tx=0x7f229400c430 comp rx=0 tx=0).stop 2024-03-24T00:31:10.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f22a3400640 1 -- 172.21.15.40:0/806279970 reap_dead start 2024-03-24T00:31:10.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f22a3e00640 1 -- 172.21.15.40:0/806279970 shutdown_connections 2024-03-24T00:31:10.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f22a3e00640 1 -- 172.21.15.40:0/806279970 >> 172.21.15.40:0/806279970 conn(0x7f22a40b1740 msgr2=0x7f22a415da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f22a3e00640 1 -- 172.21.15.40:0/806279970 shutdown_connections 2024-03-24T00:31:10.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7fd014c00640 1 -- 172.21.15.40:0/481339343 --> [v2:172.21.15.40:6808/4268319376,v1:172.21.15.40:6812/4268319376] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd0200cad50 con 0x7fd00006f980 2024-03-24T00:31:10.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.622+0000 7f22a3e00640 1 -- 172.21.15.40:0/806279970 wait complete. 2024-03-24T00:31:10.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.626+0000 7fd017400640 1 -- 172.21.15.40:0/481339343 <== osd.1 v2:172.21.15.40:6808/4268319376 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd010009820 con 0x7fd00006f980 2024-03-24T00:31:10.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.626+0000 7fcadd600640 1 -- 172.21.15.40:0/504746701 --> [v2:172.21.15.53:6802/3270909220,v1:172.21.15.53:6803/3270909220] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcad815ecd0 con 0x7fcab806fa00 2024-03-24T00:31:10.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.626+0000 7fd024e00640 1 -- 172.21.15.40:0/481339343 >> [v2:172.21.15.40:6808/4268319376,v1:172.21.15.40:6812/4268319376] conn(0x7fd00006f980 msgr2=0x7fd000071da0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.626+0000 7fd024e00640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.40:6808/4268319376,v1:172.21.15.40:6812/4268319376] conn(0x7fd00006f980 0x7fd000071da0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.626+0000 7fd024e00640 1 -- 172.21.15.40:0/481339343 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd000067760 msgr2=0x7fd000069c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.626+0000 7fd024e00640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd000067760 0x7fd000069c20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0099b0 tx=0x7fd00c009190 comp rx=0 tx=0).stop 2024-03-24T00:31:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.626+0000 7fd024e00640 1 -- 172.21.15.40:0/481339343 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd02015e0a0 msgr2=0x7fd0201f0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.626+0000 7fd024e00640 1 --2- 172.21.15.40:0/481339343 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd02015e0a0 0x7fd0201f0970 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd0180ab000 tx=0x7fd0180194d0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.626+0000 7fd01fe00640 1 -- 172.21.15.40:0/481339343 reap_dead start 2024-03-24T00:31:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.626+0000 7fd024e00640 1 -- 172.21.15.40:0/481339343 shutdown_connections 2024-03-24T00:31:10.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.626+0000 7fd024e00640 1 -- 172.21.15.40:0/481339343 >> 172.21.15.40:0/481339343 conn(0x7fd020071d70 msgr2=0x7fd0200b8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.626+0000 7fcacea00640 1 -- 172.21.15.40:0/504746701 <== osd.4 v2:172.21.15.53:6802/3270909220 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcad815ecd0 con 0x7fcab806fa00 2024-03-24T00:31:10.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.630+0000 7fcadd600640 1 -- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:6802/3270909220,v1:172.21.15.53:6803/3270909220] conn(0x7fcab806fa00 msgr2=0x7fcab8071e20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.630+0000 7fcadd600640 1 --2- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:6802/3270909220,v1:172.21.15.53:6803/3270909220] conn(0x7fcab806fa00 0x7fcab8071e20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.630+0000 7fd024e00640 1 -- 172.21.15.40:0/481339343 shutdown_connections 2024-03-24T00:31:10.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.630+0000 7fd024e00640 1 -- 172.21.15.40:0/481339343 wait complete. 2024-03-24T00:31:10.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.630+0000 7fcadd600640 1 -- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcab8067860 msgr2=0x7fcab8069d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.630+0000 7fcadd600640 1 --2- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcab8067860 0x7fcab8069d20 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcac40b2000 tx=0x7fcac4004740 comp rx=0 tx=0).stop 2024-03-24T00:31:10.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.630+0000 7fcadd600640 1 -- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcad806cc80 msgr2=0x7fcad8142a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.630+0000 7fcadd600640 1 --2- 172.21.15.40:0/504746701 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcad806cc80 0x7fcad8142a40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcad000efc0 tx=0x7fcad000c490 comp rx=0 tx=0).stop 2024-03-24T00:31:10.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.630+0000 7fcad7400640 1 -- 172.21.15.40:0/504746701 reap_dead start 2024-03-24T00:31:10.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.630+0000 7fcadd600640 1 -- 172.21.15.40:0/504746701 shutdown_connections 2024-03-24T00:31:10.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.630+0000 7fcadd600640 1 -- 172.21.15.40:0/504746701 >> 172.21.15.40:0/504746701 conn(0x7fcad80cfaa0 msgr2=0x7fcad80b2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.634+0000 7fcadd600640 1 -- 172.21.15.40:0/504746701 shutdown_connections 2024-03-24T00:31:10.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.634+0000 7fcadd600640 1 -- 172.21.15.40:0/504746701 wait complete. 2024-03-24T00:31:10.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.634+0000 7fb817600640 1 -- 172.21.15.40:0/794231400 --> [v2:172.21.15.40:6810/201988360,v1:172.21.15.40:6813/201988360] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb810068880 con 0x7fb7ec06f860 2024-03-24T00:31:10.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7fb80d600640 1 -- 172.21.15.40:0/794231400 <== osd.0 v2:172.21.15.40:6810/201988360 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb810068880 con 0x7fb7ec06f860 2024-03-24T00:31:10.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 --> [v2:172.21.15.40:6809/2475081617,v1:172.21.15.40:6811/2475081617] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff208002de0 con 0x7ff21006fa60 2024-03-24T00:31:10.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7fb817600640 1 -- 172.21.15.40:0/794231400 >> [v2:172.21.15.40:6810/201988360,v1:172.21.15.40:6813/201988360] conn(0x7fb7ec06f860 msgr2=0x7fb7ec071c80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7fb817600640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.40:6810/201988360,v1:172.21.15.40:6813/201988360] conn(0x7fb7ec06f860 0x7fb7ec071c80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7ff227e00640 1 -- 172.21.15.40:0/3116225376 <== osd.3 v2:172.21.15.40:6809/2475081617 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff208002de0 con 0x7ff21006fa60 2024-03-24T00:31:10.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 >> [v2:172.21.15.40:6809/2475081617,v1:172.21.15.40:6811/2475081617] conn(0x7ff21006fa60 msgr2=0x7ff210071e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7ff236c00640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.40:6809/2475081617,v1:172.21.15.40:6811/2475081617] conn(0x7ff21006fa60 0x7ff210071e80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7fb817600640 1 -- 172.21.15.40:0/794231400 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb7ec067660 msgr2=0x7fb7ec069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7fb817600640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb7ec067660 0x7fb7ec069b20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb800006aa0 tx=0x7fb800008040 comp rx=0 tx=0).stop 2024-03-24T00:31:10.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7fb817600640 1 -- 172.21.15.40:0/794231400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8100b5330 msgr2=0x7fb8100b8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7fb817600640 1 --2- 172.21.15.40:0/794231400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8100b5330 0x7fb8100b8100 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb808001360 tx=0x7fb808011d60 comp rx=0 tx=0).stop 2024-03-24T00:31:10.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7fb815600640 1 -- 172.21.15.40:0/794231400 reap_dead start 2024-03-24T00:31:10.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.638+0000 7fb817600640 1 -- 172.21.15.40:0/794231400 shutdown_connections 2024-03-24T00:31:10.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.642+0000 7fb817600640 1 -- 172.21.15.40:0/794231400 >> 172.21.15.40:0/794231400 conn(0x7fb810066e80 msgr2=0x7fb81015f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.642+0000 7fb817600640 1 -- 172.21.15.40:0/794231400 shutdown_connections 2024-03-24T00:31:10.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.642+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff210067860 msgr2=0x7ff210069d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.642+0000 7ff236c00640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff210067860 0x7ff210069d20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff21c000f70 tx=0x7ff21c007ee0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.642+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff23015e180 msgr2=0x7ff230160500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.642+0000 7ff236c00640 1 --2- 172.21.15.40:0/3116225376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff23015e180 0x7ff230160500 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff22000e630 tx=0x7ff22000eaf0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.642+0000 7ff234c00640 1 -- 172.21.15.40:0/3116225376 reap_dead start 2024-03-24T00:31:10.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.642+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 shutdown_connections 2024-03-24T00:31:10.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.642+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 >> 172.21.15.40:0/3116225376 conn(0x7ff230066e80 msgr2=0x7ff2300cedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.642+0000 7fb817600640 1 -- 172.21.15.40:0/794231400 wait complete. 2024-03-24T00:31:10.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.646+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 shutdown_connections 2024-03-24T00:31:10.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.646+0000 7ff236c00640 1 -- 172.21.15.40:0/3116225376 wait complete. 2024-03-24T00:31:10.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.650+0000 7f2e20e00640 1 -- 172.21.15.40:0/3611665255 --> [v2:172.21.15.53:6821/2592328924,v1:172.21.15.53:6824/2592328924] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2e1c0cf680 con 0x7f2dfc06f8d0 2024-03-24T00:31:10.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.650+0000 7f2e12000640 1 -- 172.21.15.40:0/3611665255 <== osd.7 v2:172.21.15.53:6821/2592328924 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2e1c0cf680 con 0x7f2dfc06f8d0 2024-03-24T00:31:10.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.654+0000 7f1d18e00640 1 Processor -- start 2024-03-24T00:31:10.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.654+0000 7f1d18e00640 1 -- start start 2024-03-24T00:31:10.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.654+0000 7f1d18e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d1406deb0 0x7f1d1415db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f2e20e00640 1 -- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:6821/2592328924,v1:172.21.15.53:6824/2592328924] conn(0x7f2dfc06f8d0 msgr2=0x7f2dfc071cf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f2e20e00640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:6821/2592328924,v1:172.21.15.53:6824/2592328924] conn(0x7f2dfc06f8d0 0x7f2dfc071cf0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d18e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1415e160 con 0x7f1d14140770 2024-03-24T00:31:10.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d11e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1d14140770 0x7f1d1406d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57886/0 (socket says 172.21.15.40:57886) 2024-03-24T00:31:10.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d11e00640 1 -- 172.21.15.40:0/2252230116 learned_addr learned my addr 172.21.15.40:0/2252230116 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:10.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d18e00640 1 -- 172.21.15.40:0/2252230116 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1415e440 con 0x7f1d1406cc80 2024-03-24T00:31:10.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d18e00640 1 -- 172.21.15.40:0/2252230116 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d14070d40 con 0x7f1d1406deb0 2024-03-24T00:31:10.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f2e20e00640 1 -- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2dfc0676b0 msgr2=0x7f2dfc069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f2e20e00640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2dfc0676b0 0x7f2dfc069b70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c146970 tx=0x7f2e08008040 comp rx=0 tx=0).stop 2024-03-24T00:31:10.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f2e20e00640 1 -- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e1c162160 msgr2=0x7f2e1c145db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f2e20e00640 1 --2- 172.21.15.40:0/3611665255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e1c162160 0x7f2e1c145db0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2e140ac000 tx=0x7f2e14019560 comp rx=0 tx=0).stop 2024-03-24T00:31:10.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f2e1a800640 1 -- 172.21.15.40:0/3611665255 reap_dead start 2024-03-24T00:31:10.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f2e20e00640 1 -- 172.21.15.40:0/3611665255 shutdown_connections 2024-03-24T00:31:10.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f2e20e00640 1 -- 172.21.15.40:0/3611665255 >> 172.21.15.40:0/3611665255 conn(0x7f2e1c0b1740 msgr2=0x7f2e1c13cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d11400640 1 --2- 172.21.15.40:0/2252230116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d1406deb0 0x7f1d1415db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327677268 0 0) 0x7f1d1415e160 con 0x7f1d14140770 2024-03-24T00:31:10.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf8003530 con 0x7f1d14140770 2024-03-24T00:31:10.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2788464097 0 0) 0x7f1cf8003530 con 0x7f1d14140770 2024-03-24T00:31:10.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d1406deb0 msgr2=0x7f1d1415db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 --2- 172.21.15.40:0/2252230116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d1406deb0 0x7f1d1415db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 >> v1:172.21.15.53:6789/0 conn(0x7f1d1406cc80 legacy=0x7f1d140711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1415e720 con 0x7f1d14140770 2024-03-24T00:31:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f1d00004e70 con 0x7f1d14140770 2024-03-24T00:31:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 >> v1:172.21.15.40:6789/0 conn(0x7f1d14140770 legacy=0x7f1d1406d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 --2- 172.21.15.40:0/2252230116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1cf80039f0 0x7f1cf8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 --2- 172.21.15.40:0/2252230116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf80043e0 0x7f1cf80048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 --2- 172.21.15.40:0/2252230116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf80050b0 0x7f1cf800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d14070d40 con 0x7f1cf80043e0 2024-03-24T00:31:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf800bfc0 con 0x7f1cf80050b0 2024-03-24T00:31:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.658+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf800c1c0 con 0x7f1cf80039f0 2024-03-24T00:31:10.666 INFO:teuthology.orchestra.run.smithi040.stdout:55834574854 2024-03-24T00:31:10.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.662+0000 7f2e20e00640 1 -- 172.21.15.40:0/3611665255 shutdown_connections 2024-03-24T00:31:10.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-24T00:31:10.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.662+0000 7f1d11400640 1 --2- 172.21.15.40:0/2252230116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf80043e0 0x7f1cf80048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.662+0000 7f2e20e00640 1 -- 172.21.15.40:0/3611665255 wait complete. 2024-03-24T00:31:10.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.662+0000 7f1d12800640 1 --2- 172.21.15.40:0/2252230116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf80050b0 0x7f1cf800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.662+0000 7f1d11e00640 1 --2- 172.21.15.40:0/2252230116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1cf80039f0 0x7f1cf8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.662+0000 7f1d12800640 1 -- 172.21.15.40:0/2252230116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1cf80039f0 msgr2=0x7f1cf8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.662+0000 7f1d12800640 1 --2- 172.21.15.40:0/2252230116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1cf80039f0 0x7f1cf8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.662+0000 7f1d12800640 1 -- 172.21.15.40:0/2252230116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf80043e0 msgr2=0x7f1cf80048b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.662+0000 7f1d12800640 1 --2- 172.21.15.40:0/2252230116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1cf80043e0 0x7f1cf80048b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.662+0000 7f1d12800640 1 -- 172.21.15.40:0/2252230116 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1415e720 con 0x7f1cf80050b0 2024-03-24T00:31:10.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.666+0000 7f34d8600640 1 Processor -- start 2024-03-24T00:31:10.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.666+0000 7f1d12800640 1 --2- 172.21.15.40:0/2252230116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf80050b0 0x7f1cf800b9d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1d1406d130 tx=0x7f1d0c01c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af043ac1c14cc125 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.666+0000 7f34d8600640 1 -- start start 2024-03-24T00:31:10.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.666+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c0040b0 con 0x7f1cf80050b0 2024-03-24T00:31:10.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34d8600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f34d006cc80 0x7f34d015db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34d8600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d015e160 con 0x7f34d006deb0 2024-03-24T00:31:10.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34d8600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d015e440 con 0x7f34d0140770 2024-03-24T00:31:10.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34d8600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d0070d40 con 0x7f34d006cc80 2024-03-24T00:31:10.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34d5c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f34d0140770 0x7f34d006d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58376/0 (socket says 172.21.15.40:58376) 2024-03-24T00:31:10.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34d5c00640 1 -- 172.21.15.40:0/3284632085 learned_addr learned my addr 172.21.15.40:0/3284632085 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:10.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34d5200640 1 --1- 172.21.15.40:0/3284632085 >> v1:172.21.15.40:6789/0 conn(0x7f34d006deb0 0x7f34d00711d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57900/0 (socket says 172.21.15.40:57900) 2024-03-24T00:31:10.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34d6600640 1 --2- 172.21.15.40:0/3284632085 >> v2:172.21.15.53:3300/0 conn(0x7f34d006cc80 0x7f34d015db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298266427 0 0) 0x7f34d015e160 con 0x7f34d006deb0 2024-03-24T00:31:10.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b8003530 con 0x7f34d006deb0 2024-03-24T00:31:10.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.666+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0c0a96a0 con 0x7f1cf80050b0 2024-03-24T00:31:10.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.666+0000 7f1d10a00640 1 -- 172.21.15.40:0/2252230116 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0c0a9ad0 con 0x7f1cf80050b0 2024-03-24T00:31:10.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f1d18e00640 1 -- 172.21.15.40:0/2252230116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf80050b0 msgr2=0x7f1cf800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f1d18e00640 1 --2- 172.21.15.40:0/2252230116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cf80050b0 0x7f1cf800b9d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1d1406d130 tx=0x7f1d0c01c680 comp rx=0 tx=0).stop 2024-03-24T00:31:10.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2993247396 0 0) 0x7f34b8003530 con 0x7f34d006deb0 2024-03-24T00:31:10.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 >> v2:172.21.15.53:3300/0 conn(0x7f34d006cc80 msgr2=0x7f34d015db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 --2- 172.21.15.40:0/3284632085 >> v2:172.21.15.53:3300/0 conn(0x7f34d006cc80 0x7f34d015db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 >> v1:172.21.15.53:6789/0 conn(0x7f34d0140770 legacy=0x7f34d006d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d015e720 con 0x7f34d006deb0 2024-03-24T00:31:10.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f34c4001680 con 0x7f34d006deb0 2024-03-24T00:31:10.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 >> v1:172.21.15.40:6789/0 conn(0x7f34d006deb0 legacy=0x7f34d00711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 --2- 172.21.15.40:0/3284632085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b80039c0 0x7f34b8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 --2- 172.21.15.40:0/3284632085 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b8004810 0x7f34b8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 --2- 172.21.15.40:0/3284632085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b80051d0 0x7f34b800ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d0070d40 con 0x7f34b80051d0 2024-03-24T00:31:10.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b8009bc0 con 0x7f34b80039c0 2024-03-24T00:31:10.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b8009dc0 con 0x7f34b8004810 2024-03-24T00:31:10.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f1d12800640 1 -- 172.21.15.40:0/2252230116 reap_dead start 2024-03-24T00:31:10.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f1d18e00640 1 -- 172.21.15.40:0/2252230116 shutdown_connections 2024-03-24T00:31:10.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f1d18e00640 1 -- 172.21.15.40:0/2252230116 >> 172.21.15.40:0/2252230116 conn(0x7f1d140cfaa0 msgr2=0x7f1d140cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f1d18e00640 1 -- 172.21.15.40:0/2252230116 shutdown_connections 2024-03-24T00:31:10.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.670+0000 7f1d18e00640 1 -- 172.21.15.40:0/2252230116 wait complete. 2024-03-24T00:31:10.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.674+0000 7f34d5200640 1 --2- 172.21.15.40:0/3284632085 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b8004810 0x7f34b8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.674+0000 7f34d6600640 1 --2- 172.21.15.40:0/3284632085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b80051d0 0x7f34b800ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.680 INFO:teuthology.orchestra.run.smithi040.stdout:55834574854 2024-03-24T00:31:10.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.674+0000 7f34d5200640 1 -- 172.21.15.40:0/3284632085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b80039c0 msgr2=0x7f34b8003de0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:31:10.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.674+0000 7f34d5200640 1 --2- 172.21.15.40:0/3284632085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b80039c0 0x7f34b8003de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-24T00:31:10.682 INFO:teuthology.orchestra.run.smithi040.stdout:55834574854 2024-03-24T00:31:10.683 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-24T00:31:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.674+0000 7f34d5200640 1 -- 172.21.15.40:0/3284632085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b80051d0 msgr2=0x7f34b800ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.674+0000 7f34d5200640 1 --2- 172.21.15.40:0/3284632085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b80051d0 0x7f34b800ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.674+0000 7f34d5200640 1 -- 172.21.15.40:0/3284632085 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d015e440 con 0x7f34b8004810 2024-03-24T00:31:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.678+0000 7f34d6600640 1 --2- 172.21.15.40:0/3284632085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34b80051d0 0x7f34b800ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.678+0000 7f34d6600640 1 -- 172.21.15.40:0/3284632085 reap_dead start 2024-03-24T00:31:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.678+0000 7f34d5200640 1 --2- 172.21.15.40:0/3284632085 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b8004810 0x7f34b8004c90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f34c400c110 tx=0x7f34c401f310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1b5f2a9f9c7e69e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.678+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4007c40 con 0x7f34b8004810 2024-03-24T00:31:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.678+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c4007170 con 0x7f34b8004810 2024-03-24T00:31:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.678+0000 7f34cfe00640 1 -- 172.21.15.40:0/3284632085 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c40075a0 con 0x7f34b8004810 2024-03-24T00:31:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.678+0000 7f34d8600640 1 -- 172.21.15.40:0/3284632085 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b8004810 msgr2=0x7f34b8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.678+0000 7f34d8600640 1 --2- 172.21.15.40:0/3284632085 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b8004810 0x7f34b8004c90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f34c400c110 tx=0x7f34c401f310 comp rx=0 tx=0).stop 2024-03-24T00:31:10.685 INFO:teuthology.orchestra.run.smithi040.stdout:55834574853 2024-03-24T00:31:10.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-24T00:31:10.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f34d8600640 1 -- 172.21.15.40:0/3284632085 shutdown_connections 2024-03-24T00:31:10.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f1d18e00640 1 Processor -- start 2024-03-24T00:31:10.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f34d8600640 1 --2- 172.21.15.40:0/3284632085 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b8004810 0x7f34b8004c90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f34d8600640 1 -- 172.21.15.40:0/3284632085 >> 172.21.15.40:0/3284632085 conn(0x7f34d00cfaa0 msgr2=0x7f34d00cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f1d18e00640 1 -- start start 2024-03-24T00:31:10.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f34d8600640 1 -- 172.21.15.40:0/3284632085 shutdown_connections 2024-03-24T00:31:10.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f34d8600640 1 -- 172.21.15.40:0/3284632085 wait complete. 2024-03-24T00:31:10.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f1d18e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1cf80050b0 0x7f1d141eefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f1d18e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d14154f00 0x7f1d14152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f1d18e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d1413b630 0x7f1d1413bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f1d18e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d140d05c0 con 0x7f1d1413b630 2024-03-24T00:31:10.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f1d18e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1415ea00 con 0x7f1d14154f00 2024-03-24T00:31:10.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f1d18e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d140d0740 con 0x7f1cf80050b0 2024-03-24T00:31:10.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f1d11e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1cf80050b0 0x7f1d141eefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.682+0000 7f1d12800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d1413b630 0x7f1d1413bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d11e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1cf80050b0 0x7f1d141eefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:54030/0 (socket says 172.21.15.40:54030) 2024-03-24T00:31:10.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d11e00640 1 -- 172.21.15.40:0/985841247 learned_addr learned my addr 172.21.15.40:0/985841247 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d11400640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d14154f00 0x7f1d14152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f34d8600640 1 Processor -- start 2024-03-24T00:31:10.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d11400640 1 -- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1cf80050b0 msgr2=0x7f1d141eefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d11400640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1cf80050b0 0x7f1d141eefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d11400640 1 -- 172.21.15.40:0/985841247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d1413b630 msgr2=0x7f1d1413bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d11400640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d1413b630 0x7f1d1413bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d11400640 1 -- 172.21.15.40:0/985841247 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1413d7a0 con 0x7f1d14154f00 2024-03-24T00:31:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d11400640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d14154f00 0x7f1d14152d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1d04003f30 tx=0x7f1d04009a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.695 INFO:teuthology.orchestra.run.smithi040.stdout:55834574853 2024-03-24T00:31:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f34d8600640 1 -- start start 2024-03-24T00:31:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d8600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b8004810 0x7f34d01fac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d8600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d01431e0 0x7f34d013e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d8600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34d0143660 0x7f34d013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d8600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00d05c0 con 0x7f34d0143660 2024-03-24T00:31:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d8600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d015ea00 con 0x7f34d01431e0 2024-03-24T00:31:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d8600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00d0740 con 0x7f34b8004810 2024-03-24T00:31:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d01431e0 0x7f34d013e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d01431e0 0x7f34d013e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39672/0 (socket says 172.21.15.40:39672) 2024-03-24T00:31:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d5200640 1 -- 172.21.15.40:0/3461212782 learned_addr learned my addr 172.21.15.40:0/3461212782 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d5c00640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b8004810 0x7f34d01fac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d0a000640 1 -- 172.21.15.40:0/985841247 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04017d20 con 0x7f1d14154f00 2024-03-24T00:31:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d0a000640 1 -- 172.21.15.40:0/985841247 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0401b070 con 0x7f1d14154f00 2024-03-24T00:31:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d0a000640 1 -- 172.21.15.40:0/985841247 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04012710 con 0x7f1d14154f00 2024-03-24T00:31:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.686+0000 7f1d18e00640 1 -- 172.21.15.40:0/985841247 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d14151840 con 0x7f1d14154f00 2024-03-24T00:31:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f1d18e00640 1 -- 172.21.15.40:0/985841247 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1413dd20 con 0x7f1d14154f00 2024-03-24T00:31:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f1d18e00640 1 -- 172.21.15.40:0/985841247 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1d141ef510 con 0x7f1d14154f00 2024-03-24T00:31:10.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-24T00:31:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d5200640 1 -- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b8004810 msgr2=0x7f34d01fac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d5200640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b8004810 0x7f34d01fac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d5200640 1 -- 172.21.15.40:0/3461212782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34d0143660 msgr2=0x7f34d013ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d5200640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34d0143660 0x7f34d013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.690+0000 7f34d5200640 1 -- 172.21.15.40:0/3461212782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d0201da0 con 0x7f34d01431e0 2024-03-24T00:31:10.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f1d0a000640 1 -- 172.21.15.40:0/985841247 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f1d040128b0 con 0x7f1d14154f00 2024-03-24T00:31:10.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f34d5200640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d01431e0 0x7f34d013e800 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f34c401f140 tx=0x7f34b8005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f34cd600640 1 -- 172.21.15.40:0/3461212782 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4001680 con 0x7f34d01431e0 2024-03-24T00:31:10.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f34cd600640 1 -- 172.21.15.40:0/3461212782 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c40b6030 con 0x7f34d01431e0 2024-03-24T00:31:10.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f34cd600640 1 -- 172.21.15.40:0/3461212782 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4007540 con 0x7f34d01431e0 2024-03-24T00:31:10.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f34d8600640 1 -- 172.21.15.40:0/3461212782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d0202060 con 0x7f34d01431e0 2024-03-24T00:31:10.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f34d8600640 1 -- 172.21.15.40:0/3461212782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d0202620 con 0x7f34d01431e0 2024-03-24T00:31:10.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f1d0a000640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1cf40676e0 0x7f1cf4069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f1d0a000640 1 -- 172.21.15.40:0/985841247 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d0408abd0 con 0x7f1d14154f00 2024-03-24T00:31:10.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f34d8600640 1 -- 172.21.15.40:0/3461212782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f34d0202a00 con 0x7f34d01431e0 2024-03-24T00:31:10.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f1d0a000640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:6816/3127578810,v1:172.21.15.53:6817/3127578810] conn(0x7f1cf406f900 0x7f1cf4071d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f1d0a000640 1 -- 172.21.15.40:0/985841247 --> [v2:172.21.15.53:6816/3127578810,v1:172.21.15.53:6817/3127578810] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d040121c0 con 0x7f1cf406f900 2024-03-24T00:31:10.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f1d0a000640 1 -- 172.21.15.40:0/985841247 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d0408af80 con 0x7f1d14154f00 2024-03-24T00:31:10.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.694+0000 7f1d11e00640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1cf40676e0 0x7f1cf4069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f1d12800640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:6816/3127578810,v1:172.21.15.53:6817/3127578810] conn(0x7f1cf406f900 0x7f1cf4071d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f34cd600640 1 -- 172.21.15.40:0/3461212782 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f34c40047f0 con 0x7f34d01431e0 2024-03-24T00:31:10.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f34cd600640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f34b4067690 0x7f34b4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f34cd600640 1 -- 172.21.15.40:0/3461212782 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34c40f0cc0 con 0x7f34d01431e0 2024-03-24T00:31:10.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f1d11e00640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1cf40676e0 0x7f1cf4069ba0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000bb10 tx=0x7f1d0000baa0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f34d5c00640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f34b4067690 0x7f34b4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f34cd600640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:6800/200349945,v1:172.21.15.53:6801/200349945] conn(0x7f34b4070440 0x7f34b40728e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f34cd600640 1 -- 172.21.15.40:0/3461212782 --> [v2:172.21.15.53:6800/200349945,v1:172.21.15.53:6801/200349945] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f34b4072fb0 con 0x7f34b4070440 2024-03-24T00:31:10.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f34cd600640 1 -- 172.21.15.40:0/3461212782 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34c40ef650 con 0x7f34d01431e0 2024-03-24T00:31:10.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f34d6600640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:6800/200349945,v1:172.21.15.53:6801/200349945] conn(0x7f34b4070440 0x7f34b40728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.707 INFO:teuthology.orchestra.run.smithi040.stdout:55834574853 2024-03-24T00:31:10.708 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-24T00:31:10.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f34d5c00640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f34b4067690 0x7f34b4069b50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f34c0007fd0 tx=0x7f34c0007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f1d12800640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:6816/3127578810,v1:172.21.15.53:6817/3127578810] conn(0x7f1cf406f900 0x7f1cf4071d20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.698+0000 7f34d6600640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:6800/200349945,v1:172.21.15.53:6801/200349945] conn(0x7f34b4070440 0x7f34b40728e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.702+0000 7f34cd600640 1 -- 172.21.15.40:0/3461212782 <== osd.5 v2:172.21.15.53:6800/200349945 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f34b4072fb0 con 0x7f34b4070440 2024-03-24T00:31:10.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.702+0000 7f1d0a000640 1 -- 172.21.15.40:0/985841247 <== osd.6 v2:172.21.15.53:6816/3127578810 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1d040121c0 con 0x7f1cf406f900 2024-03-24T00:31:10.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.730+0000 7f34d8600640 1 -- 172.21.15.40:0/3461212782 --> [v2:172.21.15.53:6800/200349945,v1:172.21.15.53:6801/200349945] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f34d0068170 con 0x7f34b4070440 2024-03-24T00:31:10.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.730+0000 7f34cd600640 1 -- 172.21.15.40:0/3461212782 <== osd.5 v2:172.21.15.53:6800/200349945 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f34d0068170 con 0x7f34b4070440 2024-03-24T00:31:10.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.730+0000 7f34d8600640 1 -- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:6800/200349945,v1:172.21.15.53:6801/200349945] conn(0x7f34b4070440 msgr2=0x7f34b40728e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.730+0000 7f34d8600640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:6800/200349945,v1:172.21.15.53:6801/200349945] conn(0x7f34b4070440 0x7f34b40728e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.734+0000 7f34d8600640 1 -- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f34b4067690 msgr2=0x7f34b4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.734+0000 7f34d8600640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f34b4067690 0x7f34b4069b50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f34c0007fd0 tx=0x7f34c0007dc0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.734+0000 7f34d8600640 1 -- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d01431e0 msgr2=0x7f34d013e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.734+0000 7f34d8600640 1 --2- 172.21.15.40:0/3461212782 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d01431e0 0x7f34d013e800 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f34c401f140 tx=0x7f34b8005840 comp rx=0 tx=0).stop 2024-03-24T00:31:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.734+0000 7f34d6600640 1 -- 172.21.15.40:0/3461212782 reap_dead start 2024-03-24T00:31:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.734+0000 7f34d8600640 1 -- 172.21.15.40:0/3461212782 shutdown_connections 2024-03-24T00:31:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.734+0000 7f34d8600640 1 -- 172.21.15.40:0/3461212782 >> 172.21.15.40:0/3461212782 conn(0x7f34d00cfaa0 msgr2=0x7f34d00af5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.734+0000 7f34d8600640 1 -- 172.21.15.40:0/3461212782 shutdown_connections 2024-03-24T00:31:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.734+0000 7f34d8600640 1 -- 172.21.15.40:0/3461212782 wait complete. 2024-03-24T00:31:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.730+0000 7f1d18e00640 1 -- 172.21.15.40:0/985841247 --> [v2:172.21.15.53:6816/3127578810,v1:172.21.15.53:6817/3127578810] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1d141ef510 con 0x7f1cf406f900 2024-03-24T00:31:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.734+0000 7f1d0a000640 1 -- 172.21.15.40:0/985841247 <== osd.6 v2:172.21.15.53:6816/3127578810 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1d0c0a9480 con 0x7f1cf406f900 2024-03-24T00:31:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.742+0000 7f1d18e00640 1 -- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:6816/3127578810,v1:172.21.15.53:6817/3127578810] conn(0x7f1cf406f900 msgr2=0x7f1cf4071d20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.742+0000 7f1d18e00640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:6816/3127578810,v1:172.21.15.53:6817/3127578810] conn(0x7f1cf406f900 0x7f1cf4071d20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.742+0000 7f1d18e00640 1 -- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1cf40676e0 msgr2=0x7f1cf4069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.742+0000 7f1d18e00640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1cf40676e0 0x7f1cf4069ba0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1d0000bb10 tx=0x7f1d0000baa0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.742+0000 7f1d18e00640 1 -- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d14154f00 msgr2=0x7f1d14152d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.742+0000 7f1d18e00640 1 --2- 172.21.15.40:0/985841247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d14154f00 0x7f1d14152d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1d04003f30 tx=0x7f1d04009a80 comp rx=0 tx=0).stop 2024-03-24T00:31:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.742+0000 7f1d12800640 1 -- 172.21.15.40:0/985841247 reap_dead start 2024-03-24T00:31:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.742+0000 7f1d18e00640 1 -- 172.21.15.40:0/985841247 shutdown_connections 2024-03-24T00:31:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.742+0000 7f1d18e00640 1 -- 172.21.15.40:0/985841247 >> 172.21.15.40:0/985841247 conn(0x7f1d140cfaa0 msgr2=0x7f1d140cf010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.742+0000 7f1d18e00640 1 -- 172.21.15.40:0/985841247 shutdown_connections 2024-03-24T00:31:10.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.742+0000 7f1d18e00640 1 -- 172.21.15.40:0/985841247 wait complete. 2024-03-24T00:31:10.775 INFO:teuthology.orchestra.run.smithi040.stdout:55834574854 2024-03-24T00:31:10.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-24T00:31:10.787 INFO:teuthology.orchestra.run.smithi040.stdout:55834574854 2024-03-24T00:31:10.787 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-24T00:31:10.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.914+0000 7fad9e800640 1 Processor -- start 2024-03-24T00:31:10.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.914+0000 7fad9e800640 1 -- start start 2024-03-24T00:31:10.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.914+0000 7fad9e800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad98072bc0 0x7fad98071540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.914+0000 7fad9e800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fad980b39f0 0x7fad9815ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.914+0000 7fad9e800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad98071b20 con 0x7fad98158210 2024-03-24T00:31:10.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.914+0000 7fad9e800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad98072730 con 0x7fad980b39f0 2024-03-24T00:31:10.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.914+0000 7fad9e800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad98071e00 con 0x7fad98072bc0 2024-03-24T00:31:10.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.918+0000 7fad9ca00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad98072bc0 0x7fad98071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.918+0000 7fad9ca00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad98072bc0 0x7fad98071540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:54050/0 (socket says 172.21.15.40:54050) 2024-03-24T00:31:10.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.918+0000 7fad9ca00640 1 -- 172.21.15.40:0/1734835269 learned_addr learned my addr 172.21.15.40:0/1734835269 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.918+0000 7fad9de00640 1 --2- 172.21.15.40:0/1734835269 >> v2:172.21.15.53:3300/0 conn(0x7fad980b39f0 0x7fad9815ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.918+0000 7fad9ca00640 1 -- 172.21.15.40:0/1734835269 >> v2:172.21.15.53:3300/0 conn(0x7fad980b39f0 msgr2=0x7fad9815ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.918+0000 7fad9ca00640 1 --2- 172.21.15.40:0/1734835269 >> v2:172.21.15.53:3300/0 conn(0x7fad980b39f0 0x7fad9815ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.922+0000 7fad9ca00640 1 -- 172.21.15.40:0/1734835269 >> v1:172.21.15.40:6789/0 conn(0x7fad98158210 legacy=0x7fad980b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.922+0000 7fad9ca00640 1 -- 172.21.15.40:0/1734835269 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad980720e0 con 0x7fad98072bc0 2024-03-24T00:31:10.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.926+0000 7fca6fe00640 1 Processor -- start 2024-03-24T00:31:10.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.926+0000 7fad9ca00640 1 --2- 172.21.15.40:0/1734835269 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad98072bc0 0x7fad98071540 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad84009870 tx=0x7fad8401c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5772adbe761014dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.926+0000 7fca6fe00640 1 -- start start 2024-03-24T00:31:10.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.926+0000 7fca6fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca700b2260 0x7fca70070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.926+0000 7fca6fe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca700b3c30 con 0x7fca70071030 2024-03-24T00:31:10.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.926+0000 7fca6fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca700b3f10 con 0x7fca701405f0 2024-03-24T00:31:10.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.926+0000 7fca6fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca700b41f0 con 0x7fca700b2260 2024-03-24T00:31:10.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fad97e00640 1 -- 172.21.15.40:0/1734835269 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad840ab5f0 con 0x7fad98072bc0 2024-03-24T00:31:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fad97e00640 1 -- 172.21.15.40:0/1734835269 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad840b0070 con 0x7fad98072bc0 2024-03-24T00:31:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fad97e00640 1 -- 172.21.15.40:0/1734835269 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad840ab9b0 con 0x7fad98072bc0 2024-03-24T00:31:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca700b2260 0x7fca70070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca700b2260 0x7fca70070af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:54064/0 (socket says 172.21.15.40:54064) 2024-03-24T00:31:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6ea00640 1 -- 172.21.15.40:0/817833327 learned_addr learned my addr 172.21.15.40:0/817833327 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083890881 0 0) 0x7fca700b3c30 con 0x7fca70071030 2024-03-24T00:31:10.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fad9e800640 1 -- 172.21.15.40:0/1734835269 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad98072bc0 msgr2=0x7fad98071540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fad9e800640 1 --2- 172.21.15.40:0/1734835269 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad98072bc0 0x7fad98071540 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad84009870 tx=0x7fad8401c520 comp rx=0 tx=0).stop 2024-03-24T00:31:10.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca54003530 con 0x7fca70071030 2024-03-24T00:31:10.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697039371 0 0) 0x7fca700b3f10 con 0x7fca701405f0 2024-03-24T00:31:10.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca700b3c30 con 0x7fca701405f0 2024-03-24T00:31:10.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fad9e800640 1 -- 172.21.15.40:0/1734835269 shutdown_connections 2024-03-24T00:31:10.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fad9e800640 1 --2- 172.21.15.40:0/1734835269 >> v2:172.21.15.53:3300/0 conn(0x7fad980b39f0 0x7fad9815ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fad9e800640 1 --2- 172.21.15.40:0/1734835269 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad98072bc0 0x7fad98071540 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fad9e800640 1 -- 172.21.15.40:0/1734835269 >> 172.21.15.40:0/1734835269 conn(0x7fad980b1660 msgr2=0x7fad980b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 507693632 0 0) 0x7fca54003530 con 0x7fca70071030 2024-03-24T00:31:10.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca700b2260 msgr2=0x7fca70070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 --2- 172.21.15.40:0/817833327 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca700b2260 0x7fca70070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 >> v1:172.21.15.53:6789/0 conn(0x7fca701405f0 legacy=0x7fca700b2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fad9e800640 1 -- 172.21.15.40:0/1734835269 shutdown_connections 2024-03-24T00:31:10.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fad9e800640 1 -- 172.21.15.40:0/1734835269 wait complete. 2024-03-24T00:31:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca700b44d0 con 0x7fca70071030 2024-03-24T00:31:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fca60001680 con 0x7fca70071030 2024-03-24T00:31:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6ea00640 1 --2- 172.21.15.40:0/817833327 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca700b2260 0x7fca70070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 >> v1:172.21.15.40:6789/0 conn(0x7fca70071030 legacy=0x7fca70164e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 --2- 172.21.15.40:0/817833327 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca700b2260 0x7fca54004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 --2- 172.21.15.40:0/817833327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca54005150 0x7fca540037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 --2- 172.21.15.40:0/817833327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca54003cf0 0x7fca54004170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca700b41f0 con 0x7fca54003cf0 2024-03-24T00:31:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5400ba00 con 0x7fca54005150 2024-03-24T00:31:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.930+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca5400bc00 con 0x7fca700b2260 2024-03-24T00:31:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.934+0000 7fca6ea00640 1 --2- 172.21.15.40:0/817833327 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca700b2260 0x7fca54004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.934+0000 7fca6ea00640 1 -- 172.21.15.40:0/817833327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca54005150 msgr2=0x7fca540037b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:31:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.934+0000 7fca6ea00640 1 --2- 172.21.15.40:0/817833327 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca54005150 0x7fca540037b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.934+0000 7fca6ea00640 1 -- 172.21.15.40:0/817833327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca54003cf0 msgr2=0x7fca54004170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.934+0000 7fca6ea00640 1 --2- 172.21.15.40:0/817833327 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca54003cf0 0x7fca54004170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.934+0000 7fca6ea00640 1 -- 172.21.15.40:0/817833327 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca700b44d0 con 0x7fca700b2260 2024-03-24T00:31:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9e800640 1 Processor -- start 2024-03-24T00:31:10.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7f9170e00640 1 Processor -- start 2024-03-24T00:31:10.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9e800640 1 -- start start 2024-03-24T00:31:10.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7f9170e00640 1 -- start start 2024-03-24T00:31:10.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad980b39f0 0x7fad98154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad980b39f0 0x7fad98154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad980b39f0 0x7fad98154c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47126/0 (socket says 172.21.15.40:47126) 2024-03-24T00:31:10.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fca6ea00640 1 --2- 172.21.15.40:0/817833327 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca700b2260 0x7fca54004c10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009d20 tx=0x7fca5c01cd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8496fa8f95e0c01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9d400640 1 -- 172.21.15.40:0/2359687004 learned_addr learned my addr 172.21.15.40:0/2359687004 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9e800640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad98158210 0x7fad980cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9e800640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad98161650 0x7fad980cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7f9170e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f916c13cdf0 0x7f916c0b34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c0b2070 con 0x7fca700b2260 2024-03-24T00:31:10.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fca6d600640 1 -- 172.21.15.40:0/817833327 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca5c0a58f0 con 0x7fca700b2260 2024-03-24T00:31:10.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7f9170e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916c06e250 0x7f916c15a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7f9170e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916c0b2a40 con 0x7f916c15a410 2024-03-24T00:31:10.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7f9170e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f916c0b49b0 con 0x7f916c13cdf0 2024-03-24T00:31:10.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7f9170e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0b39f0 con 0x7f916c06e250 2024-03-24T00:31:10.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7f9169e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f916c13cdf0 0x7f916c0b34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7f9169e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f916c13cdf0 0x7f916c0b34b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47134/0 (socket says 172.21.15.40:47134) 2024-03-24T00:31:10.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7f9169e00640 1 -- 172.21.15.40:0/3537251901 learned_addr learned my addr 172.21.15.40:0/3537251901 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fca6fe00640 1 -- 172.21.15.40:0/817833327 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca700b2260 msgr2=0x7fca54004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fca6fe00640 1 --2- 172.21.15.40:0/817833327 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca700b2260 0x7fca54004c10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009d20 tx=0x7fca5c01cd10 comp rx=0 tx=0).stop 2024-03-24T00:31:10.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fca6f400640 1 -- 172.21.15.40:0/817833327 reap_dead start 2024-03-24T00:31:10.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad980ae480 con 0x7fad980b39f0 2024-03-24T00:31:10.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fca6fe00640 1 -- 172.21.15.40:0/817833327 shutdown_connections 2024-03-24T00:31:10.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fca6fe00640 1 -- 172.21.15.40:0/817833327 >> 172.21.15.40:0/817833327 conn(0x7fca700b1740 msgr2=0x7fca700b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fca6fe00640 1 -- 172.21.15.40:0/817833327 shutdown_connections 2024-03-24T00:31:10.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fb09c800640 1 Processor -- start 2024-03-24T00:31:10.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad980ae300 con 0x7fad98161650 2024-03-24T00:31:10.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad980ae600 con 0x7fad98158210 2024-03-24T00:31:10.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9d400640 1 -- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad98158210 msgr2=0x7fad980cbb50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T00:31:10.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9d400640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad98158210 0x7fad980cbb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9d400640 1 -- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad98161650 msgr2=0x7fad980cd4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T00:31:10.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9d400640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad98161650 0x7fad980cd4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.938+0000 7fad9d400640 1 -- 172.21.15.40:0/2359687004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad980cc410 con 0x7fad980b39f0 2024-03-24T00:31:10.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fad9d400640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad980b39f0 0x7fad98154c20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00c490 tx=0x7fad8c001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fad95600640 1 -- 172.21.15.40:0/2359687004 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c017070 con 0x7fad980b39f0 2024-03-24T00:31:10.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fad95600640 1 -- 172.21.15.40:0/2359687004 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8c003600 con 0x7fad980b39f0 2024-03-24T00:31:10.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fad95600640 1 -- 172.21.15.40:0/2359687004 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8c003a30 con 0x7fad980b39f0 2024-03-24T00:31:10.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad980cc6f0 con 0x7fad980b39f0 2024-03-24T00:31:10.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad980cccb0 con 0x7fad980b39f0 2024-03-24T00:31:10.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9804f480 con 0x7fad980b39f0 2024-03-24T00:31:10.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fb09c800640 1 -- start start 2024-03-24T00:31:10.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7fca6fe00640 1 -- 172.21.15.40:0/817833327 wait complete. 2024-03-24T00:31:10.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fca6fe00640 1 Processor -- start 2024-03-24T00:31:10.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7f9169e00640 1 -- 172.21.15.40:0/3537251901 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916c06e250 msgr2=0x7f916c15a870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:31:10.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7f9169e00640 1 --2- 172.21.15.40:0/3537251901 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916c06e250 0x7f916c15a870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7f9169e00640 1 -- 172.21.15.40:0/3537251901 >> v1:172.21.15.53:6789/0 conn(0x7f916c15a410 legacy=0x7f916c06db40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:31:10.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.942+0000 7f9169e00640 1 -- 172.21.15.40:0/3537251901 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c13d1d0 con 0x7f916c13cdf0 2024-03-24T00:31:10.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fca6fe00640 1 -- start start 2024-03-24T00:31:10.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fb09c800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb09406deb0 0x7fb09415db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fb099e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb094140770 0x7fb09406d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58412/0 (socket says 172.21.15.40:58412) 2024-03-24T00:31:10.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fb099e00640 1 -- 172.21.15.40:0/2388699593 learned_addr learned my addr 172.21.15.40:0/2388699593 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:10.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fb09c800640 1 --2- 172.21.15.40:0/2388699593 >> v2:172.21.15.53:3300/0 conn(0x7fb09406cc80 0x7fb0940711d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fb099400640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb09406deb0 0x7fb09415db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fad95600640 1 -- 172.21.15.40:0/2359687004 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fad8c01a070 con 0x7fad980b39f0 2024-03-24T00:31:10.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fad95600640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fad780676b0 0x7fad78069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fb09a800640 1 --2- 172.21.15.40:0/2388699593 >> v2:172.21.15.53:3300/0 conn(0x7fb09406cc80 0x7fb0940711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fb09c800640 1 -- 172.21.15.40:0/2388699593 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb09415e160 con 0x7fb094140770 2024-03-24T00:31:10.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fad9ca00640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fad780676b0 0x7fad78069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fca6fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca700b2260 0x7fca70152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fb09c800640 1 -- 172.21.15.40:0/2388699593 --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb094070d40 con 0x7fb09406cc80 2024-03-24T00:31:10.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fca6fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca7013a240 0x7fca70152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fca6fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca7015bc80 0x7fca7014c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fca6fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca70070c70 con 0x7fca700b2260 2024-03-24T00:31:10.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fca6fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca70070af0 con 0x7fca7013a240 2024-03-24T00:31:10.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.946+0000 7fca6fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca70070df0 con 0x7fca7015bc80 2024-03-24T00:31:10.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca6e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca7013a240 0x7fca70152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb09c800640 1 -- 172.21.15.40:0/2388699593 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09415e440 con 0x7fb09406deb0 2024-03-24T00:31:10.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054169279 0 0) 0x7fb09415e160 con 0x7fb094140770 2024-03-24T00:31:10.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca6e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca7013a240 0x7fca70152cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39716/0 (socket says 172.21.15.40:39716) 2024-03-24T00:31:10.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca6e000640 1 -- 172.21.15.40:0/2628216665 learned_addr learned my addr 172.21.15.40:0/2628216665 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb07c005740 con 0x7fb094140770 2024-03-24T00:31:10.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca6f400640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca7015bc80 0x7fca7014c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca6e000640 1 -- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca7015bc80 msgr2=0x7fca7014c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3431503233 0 0) 0x7fb07c005740 con 0x7fb094140770 2024-03-24T00:31:10.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb09406deb0 msgr2=0x7fb09415db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb09406deb0 0x7fb09415db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca6e000640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca7015bc80 0x7fca7014c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca6e000640 1 -- 172.21.15.40:0/2628216665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca700b2260 msgr2=0x7fca70152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca6e000640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca700b2260 0x7fca70152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca6e000640 1 -- 172.21.15.40:0/2628216665 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca70151110 con 0x7fca7013a240 2024-03-24T00:31:10.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 >> v2:172.21.15.53:3300/0 conn(0x7fb09406cc80 msgr2=0x7fb0940711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fad95600640 1 -- 172.21.15.40:0/2359687004 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad8c0532c0 con 0x7fad980b39f0 2024-03-24T00:31:10.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 --2- 172.21.15.40:0/2388699593 >> v2:172.21.15.53:3300/0 conn(0x7fb09406cc80 0x7fb0940711d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09415e720 con 0x7fb094140770 2024-03-24T00:31:10.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fb084001680 con 0x7fb094140770 2024-03-24T00:31:10.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 >> v1:172.21.15.53:6789/0 conn(0x7fb094140770 legacy=0x7fb09406d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb07c005bd0 0x7fb07c005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb07c006530 0x7fb07c006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb07c006f60 0x7fb07c0073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094070d40 con 0x7fb07c006530 2024-03-24T00:31:10.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09415e440 con 0x7fb07c006f60 2024-03-24T00:31:10.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c0147d0 con 0x7fb07c005bd0 2024-03-24T00:31:10.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb099e00640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb07c005bd0 0x7fb07c005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb09a800640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb07c006f60 0x7fb07c0073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb099e00640 1 -- 172.21.15.40:0/2388699593 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb07c006f60 msgr2=0x7fb07c0073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:10.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb099e00640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb07c006f60 0x7fb07c0073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb099e00640 1 -- 172.21.15.40:0/2388699593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb07c006530 msgr2=0x7fb07c006a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:31:10.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fad9ca00640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fad780676b0 0x7fad78069b70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fad840af000 tx=0x7fad840a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca6e000640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca7013a240 0x7fca70152cb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fca6800cb20 tx=0x7fca68007e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca66a00640 1 -- 172.21.15.40:0/2628216665 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68017070 con 0x7fca7013a240 2024-03-24T00:31:10.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca66a00640 1 -- 172.21.15.40:0/2628216665 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca68003d90 con 0x7fca7013a240 2024-03-24T00:31:10.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca66a00640 1 -- 172.21.15.40:0/2628216665 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6800a9a0 con 0x7fca7013a240 2024-03-24T00:31:10.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca6fe00640 1 -- 172.21.15.40:0/2628216665 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7015b910 con 0x7fca7013a240 2024-03-24T00:31:10.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fca6fe00640 1 -- 172.21.15.40:0/2628216665 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7014e670 con 0x7fca7013a240 2024-03-24T00:31:10.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb099400640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb07c006530 0x7fb07c006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb099e00640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb07c006530 0x7fb07c006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.950+0000 7fb099e00640 1 -- 172.21.15.40:0/2388699593 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09415e720 con 0x7fb07c005bd0 2024-03-24T00:31:10.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.954+0000 7fca6fe00640 1 -- 172.21.15.40:0/2628216665 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca70073540 con 0x7fca7013a240 2024-03-24T00:31:10.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.954+0000 7f9169e00640 1 --2- 172.21.15.40:0/3537251901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f916c13cdf0 0x7f916c0b34b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f915c009870 tx=0x7f915c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d876556c9feee709 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.954+0000 7fca66a00640 1 -- 172.21.15.40:0/2628216665 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fca6801a070 con 0x7fca7013a240 2024-03-24T00:31:10.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.954+0000 7fca66a00640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca500676e0 0x7fca50069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fb099400640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb07c006530 0x7fb07c006a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:10.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.954+0000 7fca66a00640 1 -- 172.21.15.40:0/2628216665 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca680850c0 con 0x7fca7013a240 2024-03-24T00:31:10.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fca6ea00640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca500676e0 0x7fca50069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fb099e00640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb07c005bd0 0x7fb07c005ff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb08400c290 tx=0x7fb08401f450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6d0fbbd293d33ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084007d60 con 0x7fb07c005bd0 2024-03-24T00:31:10.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb084005b00 con 0x7fb07c005bd0 2024-03-24T00:31:10.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fb098a00640 1 -- 172.21.15.40:0/2388699593 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084007070 con 0x7fb07c005bd0 2024-03-24T00:31:10.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fad95600640 1 -- 172.21.15.40:0/2359687004 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fad8c055920 con 0x7fad980b39f0 2024-03-24T00:31:10.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fb09c800640 1 -- 172.21.15.40:0/2388699593 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb07c005bd0 msgr2=0x7fb07c005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fb09c800640 1 --2- 172.21.15.40:0/2388699593 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb07c005bd0 0x7fb07c005ff0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb08400c290 tx=0x7fb08401f450 comp rx=0 tx=0).stop 2024-03-24T00:31:10.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fb09a800640 1 -- 172.21.15.40:0/2388699593 reap_dead start 2024-03-24T00:31:10.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7f9168a00640 1 -- 172.21.15.40:0/3537251901 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c0b2070 con 0x7f916c13cdf0 2024-03-24T00:31:10.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fb09c800640 1 -- 172.21.15.40:0/2388699593 shutdown_connections 2024-03-24T00:31:10.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fb09c800640 1 -- 172.21.15.40:0/2388699593 >> 172.21.15.40:0/2388699593 conn(0x7fb0940cfaa0 msgr2=0x7fb0940cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7f9168a00640 1 -- 172.21.15.40:0/3537251901 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c0a9070 con 0x7f916c13cdf0 2024-03-24T00:31:10.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7f9168a00640 1 -- 172.21.15.40:0/3537251901 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c0b2070 con 0x7f916c13cdf0 2024-03-24T00:31:10.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.958+0000 7fb09c800640 1 -- 172.21.15.40:0/2388699593 shutdown_connections 2024-03-24T00:31:10.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7fb09c800640 1 -- 172.21.15.40:0/2388699593 wait complete. 2024-03-24T00:31:10.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7fca6ea00640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca500676e0 0x7fca50069ba0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fca5c0af000 tx=0x7fca5c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7fdd67e00640 1 Processor -- start 2024-03-24T00:31:10.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7f9170e00640 1 -- 172.21.15.40:0/3537251901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f916c13cdf0 msgr2=0x7f916c0b34b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7f9170e00640 1 --2- 172.21.15.40:0/3537251901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f916c13cdf0 0x7f916c0b34b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f915c009870 tx=0x7f915c01c4b0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7fca66a00640 1 -- 172.21.15.40:0/2628216665 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fca68055db0 con 0x7fca7013a240 2024-03-24T00:31:10.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7f9170e00640 1 -- 172.21.15.40:0/3537251901 shutdown_connections 2024-03-24T00:31:10.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7f9170e00640 1 --2- 172.21.15.40:0/3537251901 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916c06e250 0x7f916c15a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7f9170e00640 1 --2- 172.21.15.40:0/3537251901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f916c13cdf0 0x7f916c0b34b0 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f915c009870 tx=0x7f915c01c4b0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7f9170e00640 1 -- 172.21.15.40:0/3537251901 >> 172.21.15.40:0/3537251901 conn(0x7f916c0b1740 msgr2=0x7f916c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7fdd67e00640 1 -- start start 2024-03-24T00:31:10.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7fdd67e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600d2830 0x7fdd60166e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7fdd67e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd600af880 con 0x7fdd600b2910 2024-03-24T00:31:10.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7fdd67e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd600b1e50 con 0x7fdd6015f600 2024-03-24T00:31:10.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7fdd67e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd600b3630 con 0x7fdd600d2830 2024-03-24T00:31:10.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7f9170e00640 1 -- 172.21.15.40:0/3537251901 shutdown_connections 2024-03-24T00:31:10.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7f9170e00640 1 -- 172.21.15.40:0/3537251901 wait complete. 2024-03-24T00:31:10.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7fb09c800640 1 Processor -- start 2024-03-24T00:31:10.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.962+0000 7fdd65400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fdd6015f600 0x7fdd600aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58428/0 (socket says 172.21.15.40:58428) 2024-03-24T00:31:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fb09c800640 1 -- start start 2024-03-24T00:31:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fdd64a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdd600b2910 0x7fdd600af170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57936/0 (socket says 172.21.15.40:57936) 2024-03-24T00:31:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fdd64a00640 1 -- 172.21.15.40:0/1250820562 learned_addr learned my addr 172.21.15.40:0/1250820562 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fdd65e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600d2830 0x7fdd60166e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7f9170e00640 1 Processor -- start 2024-03-24T00:31:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fb09c800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb07c005bd0 0x7fb0941586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fb09c800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0940aa820 0x7fb0940aaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fb099400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0940aa820 0x7fb0940aaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fb099400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0940aa820 0x7fb0940aaca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47146/0 (socket says 172.21.15.40:47146) 2024-03-24T00:31:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fb099400640 1 -- 172.21.15.40:0/2812652496 learned_addr learned my addr 172.21.15.40:0/2812652496 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fb09c800640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0940ac450 0x7fb0940ac900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0940d05c0 con 0x7fb0940aa820 2024-03-24T00:31:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09415ea00 con 0x7fb0940ac450 2024-03-24T00:31:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0940d0740 con 0x7fb07c005bd0 2024-03-24T00:31:10.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fdd5fe00640 1 -- 172.21.15.40:0/1250820562 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896449831 0 0) 0x7fdd600af880 con 0x7fdd600b2910 2024-03-24T00:31:10.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7fdd5fe00640 1 -- 172.21.15.40:0/1250820562 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd44003530 con 0x7fdd600b2910 2024-03-24T00:31:10.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7fdd65e00640 1 -- 172.21.15.40:0/1250820562 >> v1:172.21.15.53:6789/0 conn(0x7fdd6015f600 legacy=0x7fdd600aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7fdd65e00640 1 -- 172.21.15.40:0/1250820562 >> v1:172.21.15.40:6789/0 conn(0x7fdd600b2910 legacy=0x7fdd600af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7fdd65e00640 1 -- 172.21.15.40:0/1250820562 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd600d2c40 con 0x7fdd600d2830 2024-03-24T00:31:10.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7fb099e00640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb07c005bd0 0x7fb0941586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7fb09a800640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0940ac450 0x7fb0940ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7fb099e00640 1 -- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0940ac450 msgr2=0x7fb0940ac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7fb099e00640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0940ac450 0x7fb0940ac900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7fb099e00640 1 -- 172.21.15.40:0/2812652496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0940aa820 msgr2=0x7fb0940aaca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7fb099e00640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0940aa820 0x7fb0940aaca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7fb099e00640 1 -- 172.21.15.40:0/2812652496 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0940ab300 con 0x7fb07c005bd0 2024-03-24T00:31:10.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7fb09a800640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0940ac450 0x7fb0940ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:10.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7fb099e00640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb07c005bd0 0x7fb0941586b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb07c00bee0 tx=0x7fb07c007a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.966+0000 7f9170e00640 1 -- start start 2024-03-24T00:31:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7f9170e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916c06e250 0x7f916c13ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7f9170e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916c15a410 0x7f916c1391e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7f9170e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f916c13b300 0x7f916c13b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7f9169e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916c06e250 0x7f916c13ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7f9169e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916c06e250 0x7f916c13ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39740/0 (socket says 172.21.15.40:39740) 2024-03-24T00:31:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7f9169400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916c15a410 0x7f916c1391e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7f9169400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916c15a410 0x7f916c1391e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:54144/0 (socket says 172.21.15.40:54144) 2024-03-24T00:31:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7f916a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f916c13b300 0x7f916c13b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7f916a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f916c13b300 0x7f916c13b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47152/0 (socket says 172.21.15.40:47152) 2024-03-24T00:31:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7f9169e00640 1 -- 172.21.15.40:0/2567627973 learned_addr learned my addr 172.21.15.40:0/2567627973 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.970+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0cbec0 con 0x7f916c13b300 2024-03-24T00:31:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c0b2f90 con 0x7f916c06e250 2024-03-24T00:31:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c13dcf0 con 0x7f916c15a410 2024-03-24T00:31:10.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7fb08e000640 1 -- 172.21.15.40:0/2812652496 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084005cb0 con 0x7fb07c005bd0 2024-03-24T00:31:10.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7fb08e000640 1 -- 172.21.15.40:0/2812652496 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb084005e50 con 0x7fb07c005bd0 2024-03-24T00:31:10.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7fb08e000640 1 -- 172.21.15.40:0/2812652496 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb084007270 con 0x7fb07c005bd0 2024-03-24T00:31:10.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0940ab5e0 con 0x7fb07c005bd0 2024-03-24T00:31:10.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb094141f20 con 0x7fb07c005bd0 2024-03-24T00:31:10.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7f9169400640 1 -- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916c06e250 msgr2=0x7f916c13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7f9169400640 1 --2- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916c06e250 0x7f916c13ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7f9169400640 1 -- 172.21.15.40:0/2567627973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f916c13b300 msgr2=0x7f916c13b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7f9169400640 1 --2- 172.21.15.40:0/2567627973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f916c13b300 0x7f916c13b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7f9169400640 1 -- 172.21.15.40:0/2567627973 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c139a60 con 0x7f916c15a410 2024-03-24T00:31:10.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094068170 con 0x7fb07c005bd0 2024-03-24T00:31:10.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7f9169400640 1 --2- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916c15a410 0x7f916c1391e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f915800b410 tx=0x7f915800b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7fdd65e00640 1 --2- 172.21.15.40:0/1250820562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600d2830 0x7fdd60166e50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdd500097f0 tx=0x7fdd5001ed10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d41fad8f06fbc269 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7fdd5fe00640 1 -- 172.21.15.40:0/1250820562 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd500b5070 con 0x7fdd600d2830 2024-03-24T00:31:10.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7fdd5fe00640 1 -- 172.21.15.40:0/1250820562 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd500a89d0 con 0x7fdd600d2830 2024-03-24T00:31:10.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7fdd5fe00640 1 -- 172.21.15.40:0/1250820562 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd500a8da0 con 0x7fdd600d2830 2024-03-24T00:31:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.974+0000 7fb08e000640 1 -- 172.21.15.40:0/2812652496 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb084007410 con 0x7fb07c005bd0 2024-03-24T00:31:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7f9162000640 1 -- 172.21.15.40:0/2567627973 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158015020 con 0x7f916c15a410 2024-03-24T00:31:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7f9162000640 1 -- 172.21.15.40:0/2567627973 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9158011830 con 0x7f916c15a410 2024-03-24T00:31:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7f9162000640 1 -- 172.21.15.40:0/2567627973 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158011c20 con 0x7f916c15a410 2024-03-24T00:31:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f916c139d40 con 0x7f916c15a410 2024-03-24T00:31:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f916c13a300 con 0x7f916c15a410 2024-03-24T00:31:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7fb08e000640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0780676e0 0x7fb078069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7fb08e000640 1 -- 172.21.15.40:0/2812652496 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb084122e00 con 0x7fb07c005bd0 2024-03-24T00:31:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7fb099400640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0780676e0 0x7fb078069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916c04f480 con 0x7f916c15a410 2024-03-24T00:31:10.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7fdd67e00640 1 -- 172.21.15.40:0/1250820562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600d2830 msgr2=0x7fdd60166e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7fdd67e00640 1 --2- 172.21.15.40:0/1250820562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600d2830 0x7fdd60166e50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fdd500097f0 tx=0x7fdd5001ed10 comp rx=0 tx=0).stop 2024-03-24T00:31:10.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7fdd67e00640 1 -- 172.21.15.40:0/1250820562 shutdown_connections 2024-03-24T00:31:10.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7fdd67e00640 1 --2- 172.21.15.40:0/1250820562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600d2830 0x7fdd60166e50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7fdd67e00640 1 -- 172.21.15.40:0/1250820562 >> 172.21.15.40:0/1250820562 conn(0x7fdd600d1d10 msgr2=0x7fdd600d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:10.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7fdd67e00640 1 -- 172.21.15.40:0/1250820562 shutdown_connections 2024-03-24T00:31:10.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7fdd67e00640 1 -- 172.21.15.40:0/1250820562 wait complete. 2024-03-24T00:31:10.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.978+0000 7fb099400640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0780676e0 0x7fb078069ba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb0940cf410 tx=0x7fb088008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.982+0000 7fdd67e00640 1 Processor -- start 2024-03-24T00:31:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.982+0000 7fdd67e00640 1 -- start start 2024-03-24T00:31:10.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd67e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600b2910 0x7fdd600c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd67e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd600d2830 0x7fdd601f82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd65400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600b2910 0x7fdd600c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd65400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600b2910 0x7fdd600c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:54146/0 (socket says 172.21.15.40:54146) 2024-03-24T00:31:10.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd65400640 1 -- 172.21.15.40:0/2842341435 learned_addr learned my addr 172.21.15.40:0/2842341435 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:10.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fb08e000640 1 -- 172.21.15.40:0/2812652496 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb084019160 con 0x7fb07c005bd0 2024-03-24T00:31:10.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd67e00640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd6015f600 0x7fdd601f8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd600b2740 con 0x7fdd6015f600 2024-03-24T00:31:10.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd64a00640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd600d2830 0x7fdd601f82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd600b2d80 con 0x7fdd600d2830 2024-03-24T00:31:10.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd600b23d0 con 0x7fdd600b2910 2024-03-24T00:31:10.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd65e00640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd6015f600 0x7fdd601f8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd65400640 1 -- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd600d2830 msgr2=0x7fdd601f82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd65400640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd600d2830 0x7fdd601f82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd65400640 1 -- 172.21.15.40:0/2842341435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd6015f600 msgr2=0x7fdd601f8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:10.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd65400640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd6015f600 0x7fdd601f8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:10.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd65400640 1 -- 172.21.15.40:0/2842341435 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd600c96c0 con 0x7fdd600b2910 2024-03-24T00:31:10.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd65400640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600b2910 0x7fdd600c8de0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800c780 tx=0x7fdd5800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:10.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7f9162000640 1 -- 172.21.15.40:0/2567627973 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f9158018020 con 0x7f916c15a410 2024-03-24T00:31:10.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd5d600640 1 -- 172.21.15.40:0/2842341435 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd58017070 con 0x7fdd600b2910 2024-03-24T00:31:10.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.986+0000 7fdd5d600640 1 -- 172.21.15.40:0/2842341435 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5800a640 con 0x7fdd600b2910 2024-03-24T00:31:10.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.990+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd600c99a0 con 0x7fdd600b2910 2024-03-24T00:31:10.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.990+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd600c9f60 con 0x7fdd600b2910 2024-03-24T00:31:10.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.990+0000 7f9162000640 1 --2- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f914c067660 0x7f914c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.990+0000 7f9162000640 1 -- 172.21.15.40:0/2567627973 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9158085cb0 con 0x7f916c15a410 2024-03-24T00:31:10.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.990+0000 7f9162000640 1 -- 172.21.15.40:0/2567627973 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f915801f020 con 0x7f916c15a410 2024-03-24T00:31:10.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.990+0000 7f9169e00640 1 --2- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f914c067660 0x7f914c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:10.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.994+0000 7fdd5d600640 1 -- 172.21.15.40:0/2842341435 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5800a9f0 con 0x7fdd600b2910 2024-03-24T00:31:10.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.994+0000 7fdd5d600640 1 -- 172.21.15.40:0/2842341435 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fdd5801a070 con 0x7fdd600b2910 2024-03-24T00:31:10.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.994+0000 7fdd5d600640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd40067790 0x7fdd40069c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:10.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.994+0000 7f8e60a00640 1 Processor -- start 2024-03-24T00:31:10.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.998+0000 7f8e60a00640 1 -- start start 2024-03-24T00:31:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.998+0000 7f9169e00640 1 --2- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f914c067660 0x7f914c069b20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f915c009870 tx=0x7f915c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.998+0000 7fdd64a00640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd40067790 0x7fdd40069c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.998+0000 7fdd5d600640 1 -- 172.21.15.40:0/2842341435 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd58084f20 con 0x7fdd600b2910 2024-03-24T00:31:11.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.998+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd380052a0 con 0x7fdd600b2910 2024-03-24T00:31:11.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.998+0000 7f8e60a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5c06deb0 0x7f8e5c15db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.998+0000 7fdd64a00640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd40067790 0x7fdd40069c50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdd600d0f00 tx=0x7fdd4c007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:10.998+0000 7f8e60a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c15e160 con 0x7f8e5c06cc80 2024-03-24T00:31:11.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.002+0000 7f8e5a800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8e5c06cc80 0x7f8e5c0711d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57938/0 (socket says 172.21.15.40:57938) 2024-03-24T00:31:11.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.002+0000 7f8e5a800640 1 -- 172.21.15.40:0/3555483254 learned_addr learned my addr 172.21.15.40:0/3555483254 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.002+0000 7f8e60a00640 1 -- 172.21.15.40:0/3555483254 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c15e440 con 0x7f8e5c140770 2024-03-24T00:31:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.002+0000 7f8e60a00640 1 -- 172.21.15.40:0/3555483254 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c070d40 con 0x7f8e5c06deb0 2024-03-24T00:31:11.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.002+0000 7f8e58a00640 1 -- 172.21.15.40:0/3555483254 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788372932 0 0) 0x7f8e5c15e440 con 0x7f8e5c140770 2024-03-24T00:31:11.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.002+0000 7f8e59400640 1 --2- 172.21.15.40:0/3555483254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5c06deb0 0x7f8e5c15db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.002+0000 7f8e58a00640 1 -- 172.21.15.40:0/3555483254 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e40003530 con 0x7f8e5c140770 2024-03-24T00:31:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.002+0000 7f8e58a00640 1 -- 172.21.15.40:0/3555483254 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033459565 0 0) 0x7f8e5c15e160 con 0x7f8e5c06cc80 2024-03-24T00:31:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.002+0000 7f8e58a00640 1 -- 172.21.15.40:0/3555483254 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5c15e440 con 0x7f8e5c06cc80 2024-03-24T00:31:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.002+0000 7f8e59400640 1 -- 172.21.15.40:0/3555483254 >> v1:172.21.15.53:6789/0 conn(0x7f8e5c140770 legacy=0x7f8e5c06d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.002+0000 7f8e59400640 1 -- 172.21.15.40:0/3555483254 >> v1:172.21.15.40:6789/0 conn(0x7f8e5c06cc80 legacy=0x7f8e5c0711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.002+0000 7f8e59400640 1 -- 172.21.15.40:0/3555483254 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c15e720 con 0x7f8e5c06deb0 2024-03-24T00:31:11.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.006+0000 7fdd5d600640 1 -- 172.21.15.40:0/2842341435 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdd580555c0 con 0x7fdd600b2910 2024-03-24T00:31:11.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.018+0000 7f8e59400640 1 --2- 172.21.15.40:0/3555483254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5c06deb0 0x7f8e5c15db80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c070ec0 tx=0x7f8e4c01cb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f54e9fa0248ff3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.018+0000 7f8e58a00640 1 -- 172.21.15.40:0/3555483254 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0ab070 con 0x7f8e5c06deb0 2024-03-24T00:31:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.018+0000 7f8e58a00640 1 -- 172.21.15.40:0/3555483254 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c0a5c10 con 0x7f8e5c06deb0 2024-03-24T00:31:11.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.018+0000 7f8e58a00640 1 -- 172.21.15.40:0/3555483254 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0af400 con 0x7f8e5c06deb0 2024-03-24T00:31:11.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.022+0000 7f8e60a00640 1 -- 172.21.15.40:0/3555483254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5c06deb0 msgr2=0x7f8e5c15db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.022+0000 7f8e60a00640 1 --2- 172.21.15.40:0/3555483254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5c06deb0 0x7f8e5c15db80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c070ec0 tx=0x7f8e4c01cb20 comp rx=0 tx=0).stop 2024-03-24T00:31:11.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.022+0000 7f8e60a00640 1 -- 172.21.15.40:0/3555483254 shutdown_connections 2024-03-24T00:31:11.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.022+0000 7f8e60a00640 1 --2- 172.21.15.40:0/3555483254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5c06deb0 0x7f8e5c15db80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.022+0000 7f8e60a00640 1 -- 172.21.15.40:0/3555483254 >> 172.21.15.40:0/3555483254 conn(0x7f8e5c0cfaa0 msgr2=0x7f8e5c0cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.034+0000 7f8e60a00640 1 -- 172.21.15.40:0/3555483254 shutdown_connections 2024-03-24T00:31:11.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.034+0000 7f8e60a00640 1 -- 172.21.15.40:0/3555483254 wait complete. 2024-03-24T00:31:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.034+0000 7f8e60a00640 1 Processor -- start 2024-03-24T00:31:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e60a00640 1 -- start start 2024-03-24T00:31:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e60a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e5c06cc80 0x7f8e5c153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e60a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5c06deb0 0x7f8e5c157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e60a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5c140770 0x7f8e5c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e60a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c072480 con 0x7f8e5c140770 2024-03-24T00:31:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e60a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c072300 con 0x7f8e5c06cc80 2024-03-24T00:31:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e60a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c167d20 con 0x7f8e5c06deb0 2024-03-24T00:31:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e59e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e5c06cc80 0x7f8e5c153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e59400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5c06deb0 0x7f8e5c157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e59e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e5c06cc80 0x7f8e5c153b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39762/0 (socket says 172.21.15.40:39762) 2024-03-24T00:31:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e59e00640 1 -- 172.21.15.40:0/2928384583 learned_addr learned my addr 172.21.15.40:0/2928384583 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e5a800640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5c140770 0x7f8e5c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e59e00640 1 -- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5c06deb0 msgr2=0x7f8e5c157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e59e00640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5c06deb0 0x7f8e5c157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e59e00640 1 -- 172.21.15.40:0/2928384583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5c140770 msgr2=0x7f8e5c154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e59e00640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5c140770 0x7f8e5c154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e59e00640 1 -- 172.21.15.40:0/2928384583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c154080 con 0x7f8e5c06cc80 2024-03-24T00:31:11.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e5a800640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5c140770 0x7f8e5c154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:11.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e59400640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5c06deb0 0x7f8e5c157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:11.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.042+0000 7f8e59e00640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e5c06cc80 0x7f8e5c153b40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800c3c0 tx=0x7f8e4800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.046+0000 7f8e52000640 1 -- 172.21.15.40:0/2928384583 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48017070 con 0x7f8e5c06cc80 2024-03-24T00:31:11.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.046+0000 7f8e52000640 1 -- 172.21.15.40:0/2928384583 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e480052d0 con 0x7f8e5c06cc80 2024-03-24T00:31:11.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.046+0000 7f8e52000640 1 -- 172.21.15.40:0/2928384583 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e480054c0 con 0x7f8e5c06cc80 2024-03-24T00:31:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.050+0000 7f8e60a00640 1 -- 172.21.15.40:0/2928384583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5c154a40 con 0x7f8e5c06cc80 2024-03-24T00:31:11.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.050+0000 7f8e60a00640 1 -- 172.21.15.40:0/2928384583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5c155f30 con 0x7f8e5c06cc80 2024-03-24T00:31:11.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.050+0000 7f8e52000640 1 -- 172.21.15.40:0/2928384583 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8e4801a070 con 0x7f8e5c06cc80 2024-03-24T00:31:11.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.054+0000 7f8e52000640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8e3c0676b0 0x7f8e3c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.054+0000 7f8e52000640 1 -- 172.21.15.40:0/2928384583 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e48084c00 con 0x7f8e5c06cc80 2024-03-24T00:31:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.054+0000 7f8e59400640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8e3c0676b0 0x7f8e3c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.054+0000 7f8e60a00640 1 -- 172.21.15.40:0/2928384583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5c068170 con 0x7f8e5c06cc80 2024-03-24T00:31:11.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.054+0000 7f8e59400640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8e3c0676b0 0x7f8e3c069b70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0b0000 tx=0x7f8e4c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.058+0000 7f8e52000640 1 -- 172.21.15.40:0/2928384583 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8e48055210 con 0x7f8e5c06cc80 2024-03-24T00:31:11.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.114+0000 7f33e7e00640 1 Processor -- start 2024-03-24T00:31:11.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.114+0000 7f33e7e00640 1 -- start start 2024-03-24T00:31:11.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.118+0000 7f33e7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f33e80b8f60 0x7f33e80d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.118+0000 7f33e7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33e80d1170 0x7f33e8167940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.118+0000 7f33e7e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e80d1d50 con 0x7f33e815dd80 2024-03-24T00:31:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.118+0000 7f33e7e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33e80d15c0 con 0x7f33e80b8f60 2024-03-24T00:31:11.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.118+0000 7f33e7e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e80d2c20 con 0x7f33e80d1170 2024-03-24T00:31:11.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.118+0000 7f33e6a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f33e815dd80 0x7f33e80d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58450/0 (socket says 172.21.15.40:58450) 2024-03-24T00:31:11.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.118+0000 7f33e6a00640 1 -- 172.21.15.40:0/599039451 learned_addr learned my addr 172.21.15.40:0/599039451 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:11.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.118+0000 7f33e6000640 1 --2- 172.21.15.40:0/599039451 >> v2:172.21.15.40:3300/0 conn(0x7f33e80b8f60 0x7f33e80d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.118+0000 7f33e5600640 1 -- 172.21.15.40:0/599039451 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067845025 0 0) 0x7f33e80d1d50 con 0x7f33e815dd80 2024-03-24T00:31:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.118+0000 7f33e5600640 1 -- 172.21.15.40:0/599039451 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33cc003530 con 0x7f33e815dd80 2024-03-24T00:31:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.122+0000 7f33e7400640 1 --2- 172.21.15.40:0/599039451 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33e80d1170 0x7f33e8167940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.122+0000 7f33e7400640 1 -- 172.21.15.40:0/599039451 >> v2:172.21.15.40:3300/0 conn(0x7f33e80b8f60 msgr2=0x7f33e80d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.122+0000 7f33e7400640 1 --2- 172.21.15.40:0/599039451 >> v2:172.21.15.40:3300/0 conn(0x7f33e80b8f60 0x7f33e80d0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.122+0000 7f33e7400640 1 -- 172.21.15.40:0/599039451 >> v1:172.21.15.53:6789/0 conn(0x7f33e815dd80 legacy=0x7f33e80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.122+0000 7f33e7400640 1 -- 172.21.15.40:0/599039451 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e80d2010 con 0x7f33e80d1170 2024-03-24T00:31:11.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.126+0000 7fa1a8a00640 1 Processor -- start 2024-03-24T00:31:11.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.126+0000 7fa1a8a00640 1 -- start start 2024-03-24T00:31:11.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a8a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1a40670e0 0x7fa1a406db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7f33e7400640 1 --2- 172.21.15.40:0/599039451 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33e80d1170 0x7f33e8167940 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009d90 tx=0x7f33e001f190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f5fb3ecaabdc259 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7f33e5600640 1 -- 172.21.15.40:0/599039451 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e00b4070 con 0x7f33e80d1170 2024-03-24T00:31:11.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7f33e5600640 1 -- 172.21.15.40:0/599039451 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e00af600 con 0x7f33e80d1170 2024-03-24T00:31:11.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7f33e5600640 1 -- 172.21.15.40:0/599039451 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e00af9b0 con 0x7f33e80d1170 2024-03-24T00:31:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a8a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a40b6920 con 0x7fa1a414bef0 2024-03-24T00:31:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a8a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a406e130 con 0x7fa1a40674c0 2024-03-24T00:31:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a8a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a406ed40 con 0x7fa1a40670e0 2024-03-24T00:31:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa1a40674c0 0x7fa1a414b7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58460/0 (socket says 172.21.15.40:58460) 2024-03-24T00:31:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1e00640 1 -- 172.21.15.40:0/119408650 learned_addr learned my addr 172.21.15.40:0/119408650 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:11.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a2800640 1 --2- 172.21.15.40:0/119408650 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1a40670e0 0x7fa1a406db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a3200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa1a414bef0 0x7fa1a406f1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57954/0 (socket says 172.21.15.40:57954) 2024-03-24T00:31:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752799861 0 0) 0x7fa1a406e130 con 0x7fa1a40674c0 2024-03-24T00:31:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa188003530 con 0x7fa1a40674c0 2024-03-24T00:31:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 25417930 0 0) 0x7fa188003530 con 0x7fa1a40674c0 2024-03-24T00:31:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1a40670e0 msgr2=0x7fa1a406db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 --2- 172.21.15.40:0/119408650 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1a40670e0 0x7fa1a406db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 >> v1:172.21.15.40:6789/0 conn(0x7fa1a414bef0 legacy=0x7fa1a406f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a406e410 con 0x7fa1a40674c0 2024-03-24T00:31:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fa190001680 con 0x7fa1a40674c0 2024-03-24T00:31:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 >> v1:172.21.15.53:6789/0 conn(0x7fa1a40674c0 legacy=0x7fa1a414b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 --2- 172.21.15.40:0/119408650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1880039c0 0x7fa188003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 --2- 172.21.15.40:0/119408650 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa188004810 0x7fa188004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 --2- 172.21.15.40:0/119408650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1880051d0 0x7fa18800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a406ed40 con 0x7fa1880039c0 2024-03-24T00:31:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18800c100 con 0x7fa1880051d0 2024-03-24T00:31:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.130+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18800c300 con 0x7fa188004810 2024-03-24T00:31:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.134+0000 7fa1a3200640 1 --2- 172.21.15.40:0/119408650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1880051d0 0x7fa18800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.134+0000 7fa1a1e00640 1 --2- 172.21.15.40:0/119408650 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa188004810 0x7fa188004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.134+0000 7fa1a3200640 1 -- 172.21.15.40:0/119408650 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa188004810 msgr2=0x7fa188004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.134+0000 7fa1a3200640 1 --2- 172.21.15.40:0/119408650 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa188004810 0x7fa188004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.134+0000 7fa1a3200640 1 -- 172.21.15.40:0/119408650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1880039c0 msgr2=0x7fa188003de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:31:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.134+0000 7fa1a3200640 1 --2- 172.21.15.40:0/119408650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1880039c0 0x7fa188003de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.134+0000 7fa1a3200640 1 -- 172.21.15.40:0/119408650 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a406e410 con 0x7fa1880051d0 2024-03-24T00:31:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.138+0000 7f33e7e00640 1 -- 172.21.15.40:0/599039451 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33e80d1170 msgr2=0x7f33e8167940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.138+0000 7f33e7e00640 1 --2- 172.21.15.40:0/599039451 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33e80d1170 0x7f33e8167940 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009d90 tx=0x7f33e001f190 comp rx=0 tx=0).stop 2024-03-24T00:31:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.138+0000 7f33e7e00640 1 -- 172.21.15.40:0/599039451 shutdown_connections 2024-03-24T00:31:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.138+0000 7f33e7e00640 1 --2- 172.21.15.40:0/599039451 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33e80d1170 0x7f33e8167940 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.138+0000 7f33e7e00640 1 --2- 172.21.15.40:0/599039451 >> v2:172.21.15.40:3300/0 conn(0x7f33e80b8f60 0x7f33e80d0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.138+0000 7f33e7e00640 1 -- 172.21.15.40:0/599039451 >> 172.21.15.40:0/599039451 conn(0x7f33e8071b70 msgr2=0x7f33e8071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.138+0000 7f33e7e00640 1 -- 172.21.15.40:0/599039451 shutdown_connections 2024-03-24T00:31:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.138+0000 7f33e7e00640 1 -- 172.21.15.40:0/599039451 wait complete. 2024-03-24T00:31:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.142+0000 7f33e7e00640 1 Processor -- start 2024-03-24T00:31:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.142+0000 7f33e7e00640 1 -- start start 2024-03-24T00:31:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33e80b8f60 0x7f33e80b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e80d1170 0x7f33e8155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815dd80 0x7f33e814add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e80d1170 0x7f33e8155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815dd80 0x7f33e814add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e80d1170 0x7f33e8155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39784/0 (socket says 172.21.15.40:39784) 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815dd80 0x7f33e814add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47196/0 (socket says 172.21.15.40:47196) 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e80d28b0 con 0x7f33e815dd80 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e6000640 1 -- 172.21.15.40:0/195750802 learned_addr learned my addr 172.21.15.40:0/195750802 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7e00640 1 -- 172.21.15.40:0/195750802 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e80d25d0 con 0x7f33e80d1170 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7e00640 1 -- 172.21.15.40:0/195750802 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e815e160 con 0x7f33e80b8f60 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7400640 1 -- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33e80b8f60 msgr2=0x7f33e80b71b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7400640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33e80b8f60 0x7f33e80b71b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7400640 1 -- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e80d1170 msgr2=0x7f33e8155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7400640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e80d1170 0x7f33e8155010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e7400640 1 -- 172.21.15.40:0/195750802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e81529d0 con 0x7f33e815dd80 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.146+0000 7f33e6000640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e80d1170 0x7f33e8155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-24T00:31:11.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.150+0000 7f33e7400640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815dd80 0x7f33e814add0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f33e00af000 tx=0x7f33e001bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.150+0000 7fa1a3200640 1 --2- 172.21.15.40:0/119408650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1880051d0 0x7fa18800b9d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa19c001e90 tx=0x7fa19c01c770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bafcfd8e02e3c20a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.150+0000 7f33dea00640 1 -- 172.21.15.40:0/195750802 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e00b4040 con 0x7f33e815dd80 2024-03-24T00:31:11.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.150+0000 7f33dea00640 1 -- 172.21.15.40:0/195750802 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e00af400 con 0x7f33e815dd80 2024-03-24T00:31:11.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.150+0000 7f33e7e00640 1 -- 172.21.15.40:0/195750802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e8152c70 con 0x7f33e815dd80 2024-03-24T00:31:11.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.150+0000 7f33e7e00640 1 -- 172.21.15.40:0/195750802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e8153230 con 0x7f33e815dd80 2024-03-24T00:31:11.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.150+0000 7f33dea00640 1 -- 172.21.15.40:0/195750802 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e0009450 con 0x7f33e815dd80 2024-03-24T00:31:11.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.154+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c001ab0 con 0x7fa1880051d0 2024-03-24T00:31:11.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.154+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c004d70 con 0x7fa1880051d0 2024-03-24T00:31:11.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.154+0000 7fa1a1400640 1 -- 172.21.15.40:0/119408650 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c007290 con 0x7fa1880051d0 2024-03-24T00:31:11.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.154+0000 7fa1a8a00640 1 -- 172.21.15.40:0/119408650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1880051d0 msgr2=0x7fa18800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.154+0000 7fa1a8a00640 1 --2- 172.21.15.40:0/119408650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1880051d0 0x7fa18800b9d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa19c001e90 tx=0x7fa19c01c770 comp rx=0 tx=0).stop 2024-03-24T00:31:11.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.154+0000 7fa1a3200640 1 -- 172.21.15.40:0/119408650 reap_dead start 2024-03-24T00:31:11.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.154+0000 7fa1a8a00640 1 -- 172.21.15.40:0/119408650 shutdown_connections 2024-03-24T00:31:11.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.154+0000 7fa1a8a00640 1 -- 172.21.15.40:0/119408650 >> 172.21.15.40:0/119408650 conn(0x7fa1a40b2810 msgr2=0x7fa1a40b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.154+0000 7fa1a8a00640 1 -- 172.21.15.40:0/119408650 shutdown_connections 2024-03-24T00:31:11.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.154+0000 7fa1a8a00640 1 -- 172.21.15.40:0/119408650 wait complete. 2024-03-24T00:31:11.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.154+0000 7f33dea00640 1 -- 172.21.15.40:0/195750802 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f33e00c3020 con 0x7f33e815dd80 2024-03-24T00:31:11.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.158+0000 7fa1a8a00640 1 Processor -- start 2024-03-24T00:31:11.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.158+0000 7f33e7e00640 1 -- 172.21.15.40:0/195750802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e8072c40 con 0x7f33e815dd80 2024-03-24T00:31:11.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.158+0000 7f33dea00640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f33c80676b0 0x7f33c8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.158+0000 7f33dea00640 1 -- 172.21.15.40:0/195750802 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33e0123a40 con 0x7f33e815dd80 2024-03-24T00:31:11.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.158+0000 7f33e6a00640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f33c80676b0 0x7f33c8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.158+0000 7fa1a8a00640 1 -- start start 2024-03-24T00:31:11.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.158+0000 7fa1a8a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1880051d0 0x7fa1a4161c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.158+0000 7fa1a8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1a40b3d40 0x7fa1a40b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.158+0000 7fa1a8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1a40cd180 0x7fa1a40cd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.158+0000 7f33e6a00640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f33c80676b0 0x7f33c8069b70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33d8007fd0 tx=0x7f33d8007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.162+0000 7fa1a2800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1880051d0 0x7fa1a4161c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.162+0000 7fa1a1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1a40b3d40 0x7fa1a40b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.162+0000 7fa1a2800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1880051d0 0x7fa1a4161c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:54202/0 (socket says 172.21.15.40:54202) 2024-03-24T00:31:11.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.162+0000 7fa1a1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1a40b3d40 0x7fa1a40b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39796/0 (socket says 172.21.15.40:39796) 2024-03-24T00:31:11.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.162+0000 7fa1a2800640 1 -- 172.21.15.40:0/1504352318 learned_addr learned my addr 172.21.15.40:0/1504352318 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:11.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.162+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a406dcd0 con 0x7fa1a40cd180 2024-03-24T00:31:11.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.162+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a406db50 con 0x7fa1a40b3d40 2024-03-24T00:31:11.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.162+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a406de50 con 0x7fa1880051d0 2024-03-24T00:31:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7fa1a3200640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1a40cd180 0x7fa1a40cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7fa1a2800640 1 -- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1a40b3d40 msgr2=0x7fa1a40b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7fa1a2800640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1a40b3d40 0x7fa1a40b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7fa1a2800640 1 -- 172.21.15.40:0/1504352318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1a40cd180 msgr2=0x7fa1a40cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7fa1a2800640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1a40cd180 0x7fa1a40cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7fa1a2800640 1 -- 172.21.15.40:0/1504352318 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a4162140 con 0x7fa1880051d0 2024-03-24T00:31:11.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7fa1a2800640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1880051d0 0x7fa1a4161c00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa19400d480 tx=0x7fa19400d940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7fa19aa00640 1 -- 172.21.15.40:0/1504352318 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1940125f0 con 0x7fa1880051d0 2024-03-24T00:31:11.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7fa19aa00640 1 -- 172.21.15.40:0/1504352318 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa194019070 con 0x7fa1880051d0 2024-03-24T00:31:11.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a40cdeb0 con 0x7fa1880051d0 2024-03-24T00:31:11.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a4147890 con 0x7fa1880051d0 2024-03-24T00:31:11.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7f33dea00640 1 -- 172.21.15.40:0/195750802 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f33e0018cf0 con 0x7f33e815dd80 2024-03-24T00:31:11.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.166+0000 7fa19aa00640 1 -- 172.21.15.40:0/1504352318 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1940129d0 con 0x7fa1880051d0 2024-03-24T00:31:11.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.170+0000 7fa19aa00640 1 -- 172.21.15.40:0/1504352318 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa194012b70 con 0x7fa1880051d0 2024-03-24T00:31:11.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.170+0000 7fa19aa00640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa184067760 0x7fa184069c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.170+0000 7fa19aa00640 1 -- 172.21.15.40:0/1504352318 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa194085910 con 0x7fa1880051d0 2024-03-24T00:31:11.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.170+0000 7fa1a1e00640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa184067760 0x7fa184069c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.174+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17c0052a0 con 0x7fa1880051d0 2024-03-24T00:31:11.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.174+0000 7fa1a1e00640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa184067760 0x7fa184069c20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4163840 tx=0x7fa190007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.178+0000 7fa19aa00640 1 -- 172.21.15.40:0/1504352318 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa194055fb0 con 0x7fa1880051d0 2024-03-24T00:31:11.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.322+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fad981520a0 con 0x7fad980b39f0 2024-03-24T00:31:11.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.322+0000 7fad95600640 1 -- 172.21.15.40:0/2359687004 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fad8c055920 con 0x7fad980b39f0 2024-03-24T00:31:11.327 INFO:teuthology.orchestra.run.smithi040.stdout:55834574853 2024-03-24T00:31:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.326+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fad780676b0 msgr2=0x7fad78069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.326+0000 7fad9e800640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fad780676b0 0x7fad78069b70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fad840af000 tx=0x7fad840a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.326+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad980b39f0 msgr2=0x7fad98154c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.326+0000 7fad9e800640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad980b39f0 0x7fad98154c20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00c490 tx=0x7fad8c001e80 comp rx=0 tx=0).stop 2024-03-24T00:31:11.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.334+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 shutdown_connections 2024-03-24T00:31:11.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.334+0000 7fad9e800640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fad780676b0 0x7fad78069b70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.334+0000 7fad9e800640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad98161650 0x7fad980cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.334+0000 7fad9e800640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad98158210 0x7fad980cbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.334+0000 7fad9e800640 1 --2- 172.21.15.40:0/2359687004 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad980b39f0 0x7fad98154c20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.334+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 >> 172.21.15.40:0/2359687004 conn(0x7fad980b1660 msgr2=0x7fad9815f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.334+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 shutdown_connections 2024-03-24T00:31:11.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.334+0000 7fad9e800640 1 -- 172.21.15.40:0/2359687004 wait complete. 2024-03-24T00:31:11.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.334+0000 7fca6fe00640 1 -- 172.21.15.40:0/2628216665 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fca7004f4f0 con 0x7fca7013a240 2024-03-24T00:31:11.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.342+0000 7fca66a00640 1 -- 172.21.15.40:0/2628216665 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fca680556b0 con 0x7fca7013a240 2024-03-24T00:31:11.343 INFO:teuthology.orchestra.run.smithi040.stdout:55834574854 2024-03-24T00:31:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.342+0000 7fca6fe00640 1 -- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca500676e0 msgr2=0x7fca50069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.342+0000 7fca6fe00640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca500676e0 0x7fca50069ba0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fca5c0af000 tx=0x7fca5c004740 comp rx=0 tx=0).stop 2024-03-24T00:31:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.342+0000 7fca6fe00640 1 -- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca7013a240 msgr2=0x7fca70152cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.342+0000 7fca6fe00640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca7013a240 0x7fca70152cb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fca6800cb20 tx=0x7fca68007e90 comp rx=0 tx=0).stop 2024-03-24T00:31:11.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.354+0000 7fca6fe00640 1 -- 172.21.15.40:0/2628216665 shutdown_connections 2024-03-24T00:31:11.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.354+0000 7fca6fe00640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca500676e0 0x7fca50069ba0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.354+0000 7fca6fe00640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca7015bc80 0x7fca7014c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.354+0000 7fca6fe00640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca7013a240 0x7fca70152cb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.354+0000 7fca6fe00640 1 --2- 172.21.15.40:0/2628216665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca700b2260 0x7fca70152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.354+0000 7fca6fe00640 1 -- 172.21.15.40:0/2628216665 >> 172.21.15.40:0/2628216665 conn(0x7fca700b1740 msgr2=0x7fca7006cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.354+0000 7fca6fe00640 1 -- 172.21.15.40:0/2628216665 shutdown_connections 2024-03-24T00:31:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.354+0000 7fca6fe00640 1 -- 172.21.15.40:0/2628216665 wait complete. 2024-03-24T00:31:11.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.362+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb0940cf160 con 0x7fb07c005bd0 2024-03-24T00:31:11.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.362+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fdd38005590 con 0x7fdd600b2910 2024-03-24T00:31:11.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.362+0000 7fb08e000640 1 -- 172.21.15.40:0/2812652496 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb0840f3bb0 con 0x7fb07c005bd0 2024-03-24T00:31:11.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.362+0000 7fdd5d600640 1 -- 172.21.15.40:0/2842341435 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdd58059660 con 0x7fdd600b2910 2024-03-24T00:31:11.365 INFO:teuthology.orchestra.run.smithi040.stdout:55834574854 2024-03-24T00:31:11.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.362+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f916c0b0a00 con 0x7f916c15a410 2024-03-24T00:31:11.368 INFO:teuthology.orchestra.run.smithi040.stdout:55834574853 2024-03-24T00:31:11.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7f9162000640 1 -- 172.21.15.40:0/2567627973 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f915800f2e0 con 0x7f916c15a410 2024-03-24T00:31:11.369 INFO:teuthology.orchestra.run.smithi040.stdout:55834574854 2024-03-24T00:31:11.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0780676e0 msgr2=0x7fb078069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fb09c800640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0780676e0 0x7fb078069ba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb0940cf410 tx=0x7fb088008040 comp rx=0 tx=0).stop 2024-03-24T00:31:11.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb07c005bd0 msgr2=0x7fb0941586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fb09c800640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb07c005bd0 0x7fb0941586b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb07c00bee0 tx=0x7fb07c007a50 comp rx=0 tx=0).stop 2024-03-24T00:31:11.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd40067790 msgr2=0x7fdd40069c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 shutdown_connections 2024-03-24T00:31:11.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fb09c800640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0780676e0 0x7fb078069ba0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb0940cf410 tx=0x7fb088008040 comp rx=0 tx=0).stop 2024-03-24T00:31:11.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fb09c800640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0940ac450 0x7fb0940ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fb09c800640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0940aa820 0x7fb0940aaca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fb09c800640 1 --2- 172.21.15.40:0/2812652496 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb07c005bd0 0x7fb0941586b0 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb07c00bee0 tx=0x7fb07c007a50 comp rx=0 tx=0).stop 2024-03-24T00:31:11.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 >> 172.21.15.40:0/2812652496 conn(0x7fb0940cfaa0 msgr2=0x7fb09413f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fdd67e00640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd40067790 0x7fdd40069c50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdd600d0f00 tx=0x7fdd4c007d40 comp rx=0 tx=0).stop 2024-03-24T00:31:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600b2910 msgr2=0x7fdd600c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fdd67e00640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600b2910 0x7fdd600c8de0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800c780 tx=0x7fdd5800cc40 comp rx=0 tx=0).stop 2024-03-24T00:31:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 shutdown_connections 2024-03-24T00:31:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fdd67e00640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd40067790 0x7fdd40069c50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fdd67e00640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd6015f600 0x7fdd601f8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fdd67e00640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd600d2830 0x7fdd601f82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fdd67e00640 1 --2- 172.21.15.40:0/2842341435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd600b2910 0x7fdd600c8de0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 >> 172.21.15.40:0/2842341435 conn(0x7fdd600d1d10 msgr2=0x7fdd600b1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.366+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 shutdown_connections 2024-03-24T00:31:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.370+0000 7fdd67e00640 1 -- 172.21.15.40:0/2842341435 wait complete. 2024-03-24T00:31:11.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.370+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 shutdown_connections 2024-03-24T00:31:11.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.370+0000 7fb09c800640 1 -- 172.21.15.40:0/2812652496 wait complete. 2024-03-24T00:31:11.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.374+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f914c067660 msgr2=0x7f914c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.374+0000 7f9170e00640 1 --2- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f914c067660 0x7f914c069b20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f915c009870 tx=0x7f915c009040 comp rx=0 tx=0).stop 2024-03-24T00:31:11.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.374+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916c15a410 msgr2=0x7f916c1391e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.374+0000 7f9170e00640 1 --2- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916c15a410 0x7f916c1391e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f915800b410 tx=0x7f915800b8d0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.374+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 shutdown_connections 2024-03-24T00:31:11.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.374+0000 7f9170e00640 1 --2- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f914c067660 0x7f914c069b20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.374+0000 7f9170e00640 1 --2- 172.21.15.40:0/2567627973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f916c13b300 0x7f916c13b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.374+0000 7f9170e00640 1 --2- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916c15a410 0x7f916c1391e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.374+0000 7f9170e00640 1 --2- 172.21.15.40:0/2567627973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916c06e250 0x7f916c13ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.374+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 >> 172.21.15.40:0/2567627973 conn(0x7f916c0b1740 msgr2=0x7f916c0afa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.374+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 shutdown_connections 2024-03-24T00:31:11.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.374+0000 7f9170e00640 1 -- 172.21.15.40:0/2567627973 wait complete. 2024-03-24T00:31:11.386 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2024-03-24T00:31:11.386 DEBUG:teuthology.parallel:result is None 2024-03-24T00:31:11.400 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2024-03-24T00:31:11.400 DEBUG:teuthology.parallel:result is None 2024-03-24T00:31:11.409 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2024-03-24T00:31:11.409 DEBUG:teuthology.parallel:result is None 2024-03-24T00:31:11.414 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2024-03-24T00:31:11.414 DEBUG:teuthology.parallel:result is None 2024-03-24T00:31:11.417 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2024-03-24T00:31:11.417 DEBUG:teuthology.parallel:result is None 2024-03-24T00:31:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.414+0000 7f8e60a00640 1 -- 172.21.15.40:0/2928384583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f8e5c157550 con 0x7f8e5c06cc80 2024-03-24T00:31:11.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.414+0000 7f8e52000640 1 -- 172.21.15.40:0/2928384583 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8e480592b0 con 0x7f8e5c06cc80 2024-03-24T00:31:11.418 INFO:teuthology.orchestra.run.smithi040.stdout:55834574853 2024-03-24T00:31:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.418+0000 7f8e60a00640 1 -- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8e3c0676b0 msgr2=0x7f8e3c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.418+0000 7f8e60a00640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8e3c0676b0 0x7f8e3c069b70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0b0000 tx=0x7f8e4c0a7070 comp rx=0 tx=0).stop 2024-03-24T00:31:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.418+0000 7f8e60a00640 1 -- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e5c06cc80 msgr2=0x7f8e5c153b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.418+0000 7f8e60a00640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e5c06cc80 0x7f8e5c153b40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800c3c0 tx=0x7f8e4800c880 comp rx=0 tx=0).stop 2024-03-24T00:31:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.418+0000 7f8e60a00640 1 -- 172.21.15.40:0/2928384583 shutdown_connections 2024-03-24T00:31:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.418+0000 7f8e60a00640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8e3c0676b0 0x7f8e3c069b70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.418+0000 7f8e60a00640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e5c140770 0x7f8e5c154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.418+0000 7f8e60a00640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5c06deb0 0x7f8e5c157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.418+0000 7f8e60a00640 1 --2- 172.21.15.40:0/2928384583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e5c06cc80 0x7f8e5c153b40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.418+0000 7f8e60a00640 1 -- 172.21.15.40:0/2928384583 >> 172.21.15.40:0/2928384583 conn(0x7f8e5c0cfaa0 msgr2=0x7f8e5c142740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.418+0000 7f8e60a00640 1 -- 172.21.15.40:0/2928384583 shutdown_connections 2024-03-24T00:31:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.418+0000 7f8e60a00640 1 -- 172.21.15.40:0/2928384583 wait complete. 2024-03-24T00:31:11.450 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.7 2024-03-24T00:31:11.451 DEBUG:teuthology.parallel:result is None 2024-03-24T00:31:11.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa17c005590 con 0x7fa1880051d0 2024-03-24T00:31:11.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa19aa00640 1 -- 172.21.15.40:0/1504352318 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa19405a050 con 0x7fa1880051d0 2024-03-24T00:31:11.492 INFO:teuthology.orchestra.run.smithi040.stdout:55834574854 2024-03-24T00:31:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa184067760 msgr2=0x7fa184069c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa184067760 0x7fa184069c20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4163840 tx=0x7fa190007040 comp rx=0 tx=0).stop 2024-03-24T00:31:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1880051d0 msgr2=0x7fa1a4161c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1880051d0 0x7fa1a4161c00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa19400d480 tx=0x7fa19400d940 comp rx=0 tx=0).stop 2024-03-24T00:31:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 shutdown_connections 2024-03-24T00:31:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa184067760 0x7fa184069c20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1a40cd180 0x7fa1a40cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1a40b3d40 0x7fa1a40b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 --2- 172.21.15.40:0/1504352318 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1880051d0 0x7fa1a4161c00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 >> 172.21.15.40:0/1504352318 conn(0x7fa1a40b2810 msgr2=0x7fa1a40d0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 shutdown_connections 2024-03-24T00:31:11.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.490+0000 7fa1a8a00640 1 -- 172.21.15.40:0/1504352318 wait complete. 2024-03-24T00:31:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.502+0000 7f33e7e00640 1 -- 172.21.15.40:0/195750802 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f33e80b81a0 con 0x7f33e815dd80 2024-03-24T00:31:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.502+0000 7f33dea00640 1 -- 172.21.15.40:0/195750802 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f33e0123e20 con 0x7f33e815dd80 2024-03-24T00:31:11.504 INFO:teuthology.orchestra.run.smithi040.stdout:55834574854 2024-03-24T00:31:11.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.506+0000 7f33e7e00640 1 -- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f33c80676b0 msgr2=0x7f33c8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.506+0000 7f33e7e00640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f33c80676b0 0x7f33c8069b70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33d8007fd0 tx=0x7f33d8007900 comp rx=0 tx=0).stop 2024-03-24T00:31:11.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.506+0000 7f33e7e00640 1 -- 172.21.15.40:0/195750802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815dd80 msgr2=0x7f33e814add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.506+0000 7f33e7e00640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815dd80 0x7f33e814add0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f33e00af000 tx=0x7f33e001bf30 comp rx=0 tx=0).stop 2024-03-24T00:31:11.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.506+0000 7f33e7e00640 1 -- 172.21.15.40:0/195750802 shutdown_connections 2024-03-24T00:31:11.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.506+0000 7f33e7e00640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f33c80676b0 0x7f33c8069b70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.506+0000 7f33e7e00640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f33e815dd80 0x7f33e814add0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.506+0000 7f33e7e00640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e80d1170 0x7f33e8155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.506+0000 7f33e7e00640 1 --2- 172.21.15.40:0/195750802 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33e80b8f60 0x7f33e80b71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.506+0000 7f33e7e00640 1 -- 172.21.15.40:0/195750802 >> 172.21.15.40:0/195750802 conn(0x7f33e8071b70 msgr2=0x7f33e80ccd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.506+0000 7f33e7e00640 1 -- 172.21.15.40:0/195750802 shutdown_connections 2024-03-24T00:31:11.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.506+0000 7f33e7e00640 1 -- 172.21.15.40:0/195750802 wait complete. 2024-03-24T00:31:11.513 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-03-24T00:31:11.514 DEBUG:teuthology.parallel:result is None 2024-03-24T00:31:11.526 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.6 2024-03-24T00:31:11.526 DEBUG:teuthology.parallel:result is None 2024-03-24T00:31:11.526 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-24T00:31:11.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:31:11.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.706+0000 7f6582a00640 1 Processor -- start 2024-03-24T00:31:11.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.706+0000 7f6582a00640 1 -- start start 2024-03-24T00:31:11.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6582a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f657c072510 0x7f657c072910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6582a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f657c15d890 0x7f657c15dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6582a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657c06d900 con 0x7f657c0cd1a0 2024-03-24T00:31:11.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6582a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657c0cd580 con 0x7f657c072510 2024-03-24T00:31:11.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6582a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657c15d400 con 0x7f657c15d890 2024-03-24T00:31:11.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6581600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f657c0cd1a0 0x7f657c06d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57962/0 (socket says 172.21.15.40:57962) 2024-03-24T00:31:11.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6581600640 1 -- 172.21.15.40:0/595020142 learned_addr learned my addr 172.21.15.40:0/595020142 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:11.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6580c00640 1 --2- 172.21.15.40:0/595020142 >> v2:172.21.15.40:3300/0 conn(0x7f657c072510 0x7f657c072910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6582000640 1 --2- 172.21.15.40:0/595020142 >> v2:172.21.15.53:3300/0 conn(0x7f657c15d890 0x7f657c15dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6580c00640 1 -- 172.21.15.40:0/595020142 >> v2:172.21.15.53:3300/0 conn(0x7f657c15d890 msgr2=0x7f657c15dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:11.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6580c00640 1 --2- 172.21.15.40:0/595020142 >> v2:172.21.15.53:3300/0 conn(0x7f657c15d890 0x7f657c15dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6580c00640 1 -- 172.21.15.40:0/595020142 >> v1:172.21.15.40:6789/0 conn(0x7f657c0cd1a0 legacy=0x7f657c06d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f657be00640 1 -- 172.21.15.40:0/595020142 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416898027 0 0) 0x7f657c06d900 con 0x7f657c0cd1a0 2024-03-24T00:31:11.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6580c00640 1 -- 172.21.15.40:0/595020142 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c06db70 con 0x7f657c072510 2024-03-24T00:31:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f6580c00640 1 --2- 172.21.15.40:0/595020142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f657c072510 0x7f657c072910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f657c0cdcb0 tx=0x7f656801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d7d28aa32386d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.710+0000 7f657be00640 1 -- 172.21.15.40:0/595020142 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65680ab650 con 0x7f657c072510 2024-03-24T00:31:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f657be00640 1 -- 172.21.15.40:0/595020142 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65680b0070 con 0x7f657c072510 2024-03-24T00:31:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f657be00640 1 -- 172.21.15.40:0/595020142 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65680aba10 con 0x7f657c072510 2024-03-24T00:31:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 -- 172.21.15.40:0/595020142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f657c072510 msgr2=0x7f657c072910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 --2- 172.21.15.40:0/595020142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f657c072510 0x7f657c072910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f657c0cdcb0 tx=0x7f656801c450 comp rx=0 tx=0).stop 2024-03-24T00:31:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 -- 172.21.15.40:0/595020142 shutdown_connections 2024-03-24T00:31:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 --2- 172.21.15.40:0/595020142 >> v2:172.21.15.53:3300/0 conn(0x7f657c15d890 0x7f657c15dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 --2- 172.21.15.40:0/595020142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f657c072510 0x7f657c072910 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 -- 172.21.15.40:0/595020142 >> 172.21.15.40:0/595020142 conn(0x7f657c0b1c80 msgr2=0x7f657c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 -- 172.21.15.40:0/595020142 shutdown_connections 2024-03-24T00:31:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 -- 172.21.15.40:0/595020142 wait complete. 2024-03-24T00:31:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 Processor -- start 2024-03-24T00:31:11.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 -- start start 2024-03-24T00:31:11.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f657c072510 0x7f657c0b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f657c0cd1a0 0x7f657c0ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f657c15d890 0x7f657c0aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c09df20 con 0x7f657c15d890 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c072e50 con 0x7f657c072510 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c09e0a0 con 0x7f657c0cd1a0 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f657c15d890 0x7f657c0aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f657c15d890 0x7f657c0aeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47242/0 (socket says 172.21.15.40:47242) 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582000640 1 -- 172.21.15.40:0/1997727305 learned_addr learned my addr 172.21.15.40:0/1997727305 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6581600640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f657c072510 0x7f657c0b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6580c00640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f657c0cd1a0 0x7f657c0ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582000640 1 -- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f657c0cd1a0 msgr2=0x7f657c0ae5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582000640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f657c0cd1a0 0x7f657c0ae5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582000640 1 -- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f657c072510 msgr2=0x7f657c0b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582000640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f657c072510 0x7f657c0b06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6582000640 1 -- 172.21.15.40:0/1997727305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c0af030 con 0x7f657c15d890 2024-03-24T00:31:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6581600640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f657c072510 0x7f657c0b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.714+0000 7f6580c00640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f657c0cd1a0 0x7f657c0ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.718+0000 7f6582000640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f657c15d890 0x7f657c0aeaf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6574007e60 tx=0x7f657400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.718+0000 7f6579600640 1 -- 172.21.15.40:0/1997727305 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574014420 con 0x7f657c15d890 2024-03-24T00:31:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.718+0000 7f6579600640 1 -- 172.21.15.40:0/1997727305 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6574019070 con 0x7f657c15d890 2024-03-24T00:31:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.718+0000 7f6579600640 1 -- 172.21.15.40:0/1997727305 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574014820 con 0x7f657c15d890 2024-03-24T00:31:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.718+0000 7f6582a00640 1 -- 172.21.15.40:0/1997727305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657c0ad000 con 0x7f657c15d890 2024-03-24T00:31:11.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.718+0000 7f6582a00640 1 -- 172.21.15.40:0/1997727305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657c0ad5c0 con 0x7f657c15d890 2024-03-24T00:31:11.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.718+0000 7f6582a00640 1 -- 172.21.15.40:0/1997727305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657c09e2e0 con 0x7f657c15d890 2024-03-24T00:31:11.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.718+0000 7f6579600640 1 -- 172.21.15.40:0/1997727305 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f65740149c0 con 0x7f657c15d890 2024-03-24T00:31:11.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.718+0000 7f6579600640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f655c067660 0x7f655c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:11.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.718+0000 7f6579600640 1 -- 172.21.15.40:0/1997727305 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65740861c0 con 0x7f657c15d890 2024-03-24T00:31:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.718+0000 7f6581600640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f655c067660 0x7f655c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.722+0000 7f6581600640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f655c067660 0x7f655c069b20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6570000f70 tx=0x7f6570009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:11.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.722+0000 7f6579600640 1 -- 172.21.15.40:0/1997727305 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6574056da0 con 0x7f657c15d890 2024-03-24T00:31:11.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.882+0000 7f6582a00640 1 -- 172.21.15.40:0/1997727305 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f657c0c3890 con 0x7f655c067660 2024-03-24T00:31:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.890+0000 7f6579600640 1 -- 172.21.15.40:0/1997727305 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7f657c0c3890 con 0x7f655c067660 2024-03-24T00:31:11.895 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:11.897 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:31:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.894+0000 7f6582a00640 1 -- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f655c067660 msgr2=0x7f655c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.894+0000 7f6582a00640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f655c067660 0x7f655c069b20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6570000f70 tx=0x7f6570009040 comp rx=0 tx=0).stop 2024-03-24T00:31:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.894+0000 7f6582a00640 1 -- 172.21.15.40:0/1997727305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f657c15d890 msgr2=0x7f657c0aeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.894+0000 7f6582a00640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f657c15d890 0x7f657c0aeaf0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6574007e60 tx=0x7f657400e4b0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.894+0000 7f6582a00640 1 -- 172.21.15.40:0/1997727305 shutdown_connections 2024-03-24T00:31:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.894+0000 7f6582a00640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f655c067660 0x7f655c069b20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.894+0000 7f6582a00640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f657c15d890 0x7f657c0aeaf0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.894+0000 7f6582a00640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f657c0cd1a0 0x7f657c0ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.894+0000 7f6582a00640 1 --2- 172.21.15.40:0/1997727305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f657c072510 0x7f657c0b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.894+0000 7f6582a00640 1 -- 172.21.15.40:0/1997727305 >> 172.21.15.40:0/1997727305 conn(0x7f657c0b1c80 msgr2=0x7f657c0afbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.894+0000 7f6582a00640 1 -- 172.21.15.40:0/1997727305 shutdown_connections 2024-03-24T00:31:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:11.894+0000 7f6582a00640 1 -- 172.21.15.40:0/1997727305 wait complete. 2024-03-24T00:31:11.916 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-24T00:31:11.199799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235632,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739088,"statfs":{"total":773094113280,"available":772852826112,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.157152"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046898+0000","last_change":"2024-03-24T00:31:00.052380+0000","last_active":"2024-03-24T00:31:01.046898+0000","last_peered":"2024-03-24T00:31:01.046898+0000","last_clean":"2024-03-24T00:31:01.046898+0000","last_became_active":"2024-03-24T00:31:00.049815+0000","last_became_peered":"2024-03-24T00:31:00.049815+0000","last_unstale":"2024-03-24T00:31:01.046898+0000","last_undegraded":"2024-03-24T00:31:01.046898+0000","last_fullsized":"2024-03-24T00:31:01.046898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:09.487064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047941+0000","last_change":"2024-03-24T00:31:00.048401+0000","last_active":"2024-03-24T00:31:01.047941+0000","last_peered":"2024-03-24T00:31:01.047941+0000","last_clean":"2024-03-24T00:31:01.047941+0000","last_became_active":"2024-03-24T00:31:00.048157+0000","last_became_peered":"2024-03-24T00:31:00.048157+0000","last_unstale":"2024-03-24T00:31:01.047941+0000","last_undegraded":"2024-03-24T00:31:01.047941+0000","last_fullsized":"2024-03-24T00:31:01.047941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909458+0000","last_change":"2024-03-24T00:31:00.052594+0000","last_active":"2024-03-24T00:31:01.909458+0000","last_peered":"2024-03-24T00:31:01.909458+0000","last_clean":"2024-03-24T00:31:01.909458+0000","last_became_active":"2024-03-24T00:31:00.052289+0000","last_became_peered":"2024-03-24T00:31:00.052289+0000","last_unstale":"2024-03-24T00:31:01.909458+0000","last_undegraded":"2024-03-24T00:31:01.909458+0000","last_fullsized":"2024-03-24T00:31:01.909458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:34.269764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909931+0000","last_change":"2024-03-24T00:31:00.072520+0000","last_active":"2024-03-24T00:31:01.909931+0000","last_peered":"2024-03-24T00:31:01.909931+0000","last_clean":"2024-03-24T00:31:01.909931+0000","last_became_active":"2024-03-24T00:31:00.071355+0000","last_became_peered":"2024-03-24T00:31:00.071355+0000","last_unstale":"2024-03-24T00:31:01.909931+0000","last_undegraded":"2024-03-24T00:31:01.909931+0000","last_fullsized":"2024-03-24T00:31:01.909931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:17.380183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046806+0000","last_change":"2024-03-24T00:31:00.045641+0000","last_active":"2024-03-24T00:31:01.046806+0000","last_peered":"2024-03-24T00:31:01.046806+0000","last_clean":"2024-03-24T00:31:01.046806+0000","last_became_active":"2024-03-24T00:31:00.045230+0000","last_became_peered":"2024-03-24T00:31:00.045230+0000","last_unstale":"2024-03-24T00:31:01.046806+0000","last_undegraded":"2024-03-24T00:31:01.046806+0000","last_fullsized":"2024-03-24T00:31:01.046806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:49.592126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909658+0000","last_change":"2024-03-24T00:31:00.057373+0000","last_active":"2024-03-24T00:31:01.909658+0000","last_peered":"2024-03-24T00:31:01.909658+0000","last_clean":"2024-03-24T00:31:01.909658+0000","last_became_active":"2024-03-24T00:31:00.056322+0000","last_became_peered":"2024-03-24T00:31:00.056322+0000","last_unstale":"2024-03-24T00:31:01.909658+0000","last_undegraded":"2024-03-24T00:31:01.909658+0000","last_fullsized":"2024-03-24T00:31:01.909658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045369+0000","last_change":"2024-03-24T00:31:00.048380+0000","last_active":"2024-03-24T00:31:01.045369+0000","last_peered":"2024-03-24T00:31:01.045369+0000","last_clean":"2024-03-24T00:31:01.045369+0000","last_became_active":"2024-03-24T00:31:00.046936+0000","last_became_peered":"2024-03-24T00:31:00.046936+0000","last_unstale":"2024-03-24T00:31:01.045369+0000","last_undegraded":"2024-03-24T00:31:01.045369+0000","last_fullsized":"2024-03-24T00:31:01.045369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:32.847580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909134+0000","last_change":"2024-03-24T00:31:00.057172+0000","last_active":"2024-03-24T00:31:01.909134+0000","last_peered":"2024-03-24T00:31:01.909134+0000","last_clean":"2024-03-24T00:31:01.909134+0000","last_became_active":"2024-03-24T00:31:00.056982+0000","last_became_peered":"2024-03-24T00:31:00.056982+0000","last_unstale":"2024-03-24T00:31:01.909134+0000","last_undegraded":"2024-03-24T00:31:01.909134+0000","last_fullsized":"2024-03-24T00:31:01.909134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:38.408736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047010+0000","last_change":"2024-03-24T00:31:00.053066+0000","last_active":"2024-03-24T00:31:01.047010+0000","last_peered":"2024-03-24T00:31:01.047010+0000","last_clean":"2024-03-24T00:31:01.047010+0000","last_became_active":"2024-03-24T00:31:00.052749+0000","last_became_peered":"2024-03-24T00:31:00.052749+0000","last_unstale":"2024-03-24T00:31:01.047010+0000","last_undegraded":"2024-03-24T00:31:01.047010+0000","last_fullsized":"2024-03-24T00:31:01.047010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:17.791692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060254+0000","last_change":"2024-03-24T00:31:00.060700+0000","last_active":"2024-03-24T00:31:01.060254+0000","last_peered":"2024-03-24T00:31:01.060254+0000","last_clean":"2024-03-24T00:31:01.060254+0000","last_became_active":"2024-03-24T00:31:00.060373+0000","last_became_peered":"2024-03-24T00:31:00.060373+0000","last_unstale":"2024-03-24T00:31:01.060254+0000","last_undegraded":"2024-03-24T00:31:01.060254+0000","last_fullsized":"2024-03-24T00:31:01.060254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060101+0000","last_change":"2024-03-24T00:31:00.061316+0000","last_active":"2024-03-24T00:31:01.060101+0000","last_peered":"2024-03-24T00:31:01.060101+0000","last_clean":"2024-03-24T00:31:01.060101+0000","last_became_active":"2024-03-24T00:31:00.060980+0000","last_became_peered":"2024-03-24T00:31:00.060980+0000","last_unstale":"2024-03-24T00:31:01.060101+0000","last_undegraded":"2024-03-24T00:31:01.060101+0000","last_fullsized":"2024-03-24T00:31:01.060101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045818+0000","last_change":"2024-03-24T00:31:00.046162+0000","last_active":"2024-03-24T00:31:01.045818+0000","last_peered":"2024-03-24T00:31:01.045818+0000","last_clean":"2024-03-24T00:31:01.045818+0000","last_became_active":"2024-03-24T00:31:00.045715+0000","last_became_peered":"2024-03-24T00:31:00.045715+0000","last_unstale":"2024-03-24T00:31:01.045818+0000","last_undegraded":"2024-03-24T00:31:01.045818+0000","last_fullsized":"2024-03-24T00:31:01.045818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:38.106034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909352+0000","last_change":"2024-03-24T00:31:00.045645+0000","last_active":"2024-03-24T00:31:01.909352+0000","last_peered":"2024-03-24T00:31:01.909352+0000","last_clean":"2024-03-24T00:31:01.909352+0000","last_became_active":"2024-03-24T00:31:00.045372+0000","last_became_peered":"2024-03-24T00:31:00.045372+0000","last_unstale":"2024-03-24T00:31:01.909352+0000","last_undegraded":"2024-03-24T00:31:01.909352+0000","last_fullsized":"2024-03-24T00:31:01.909352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909717+0000","last_change":"2024-03-24T00:31:00.042718+0000","last_active":"2024-03-24T00:31:01.909717+0000","last_peered":"2024-03-24T00:31:01.909717+0000","last_clean":"2024-03-24T00:31:01.909717+0000","last_became_active":"2024-03-24T00:31:00.042424+0000","last_became_peered":"2024-03-24T00:31:00.042424+0000","last_unstale":"2024-03-24T00:31:01.909717+0000","last_undegraded":"2024-03-24T00:31:01.909717+0000","last_fullsized":"2024-03-24T00:31:01.909717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047428+0000","last_change":"2024-03-24T00:31:00.038791+0000","last_active":"2024-03-24T00:31:01.047428+0000","last_peered":"2024-03-24T00:31:01.047428+0000","last_clean":"2024-03-24T00:31:01.047428+0000","last_became_active":"2024-03-24T00:31:00.038543+0000","last_became_peered":"2024-03-24T00:31:00.038543+0000","last_unstale":"2024-03-24T00:31:01.047428+0000","last_undegraded":"2024-03-24T00:31:01.047428+0000","last_fullsized":"2024-03-24T00:31:01.047428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909570+0000","last_change":"2024-03-24T00:31:00.055749+0000","last_active":"2024-03-24T00:31:01.909570+0000","last_peered":"2024-03-24T00:31:01.909570+0000","last_clean":"2024-03-24T00:31:01.909570+0000","last_became_active":"2024-03-24T00:31:00.055296+0000","last_became_peered":"2024-03-24T00:31:00.055296+0000","last_unstale":"2024-03-24T00:31:01.909570+0000","last_undegraded":"2024-03-24T00:31:01.909570+0000","last_fullsized":"2024-03-24T00:31:01.909570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910277+0000","last_change":"2024-03-24T00:31:00.055782+0000","last_active":"2024-03-24T00:31:01.910277+0000","last_peered":"2024-03-24T00:31:01.910277+0000","last_clean":"2024-03-24T00:31:01.910277+0000","last_became_active":"2024-03-24T00:31:00.055369+0000","last_became_peered":"2024-03-24T00:31:00.055369+0000","last_unstale":"2024-03-24T00:31:01.910277+0000","last_undegraded":"2024-03-24T00:31:01.910277+0000","last_fullsized":"2024-03-24T00:31:01.910277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:10.867603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186278+0000","last_change":"2024-03-24T00:31:00.044665+0000","last_active":"2024-03-24T00:31:01.186278+0000","last_peered":"2024-03-24T00:31:01.186278+0000","last_clean":"2024-03-24T00:31:01.186278+0000","last_became_active":"2024-03-24T00:31:00.044399+0000","last_became_peered":"2024-03-24T00:31:00.044399+0000","last_unstale":"2024-03-24T00:31:01.186278+0000","last_undegraded":"2024-03-24T00:31:01.186278+0000","last_fullsized":"2024-03-24T00:31:01.186278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:14.595788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047109+0000","last_change":"2024-03-24T00:31:00.045352+0000","last_active":"2024-03-24T00:31:01.047109+0000","last_peered":"2024-03-24T00:31:01.047109+0000","last_clean":"2024-03-24T00:31:01.047109+0000","last_became_active":"2024-03-24T00:31:00.045023+0000","last_became_peered":"2024-03-24T00:31:00.045023+0000","last_unstale":"2024-03-24T00:31:01.047109+0000","last_undegraded":"2024-03-24T00:31:01.047109+0000","last_fullsized":"2024-03-24T00:31:01.047109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:31.480032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060057+0000","last_change":"2024-03-24T00:31:00.061257+0000","last_active":"2024-03-24T00:31:01.060057+0000","last_peered":"2024-03-24T00:31:01.060057+0000","last_clean":"2024-03-24T00:31:01.060057+0000","last_became_active":"2024-03-24T00:31:00.060983+0000","last_became_peered":"2024-03-24T00:31:00.060983+0000","last_unstale":"2024-03-24T00:31:01.060057+0000","last_undegraded":"2024-03-24T00:31:01.060057+0000","last_fullsized":"2024-03-24T00:31:01.060057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909460+0000","last_change":"2024-03-24T00:31:00.054714+0000","last_active":"2024-03-24T00:31:01.909460+0000","last_peered":"2024-03-24T00:31:01.909460+0000","last_clean":"2024-03-24T00:31:01.909460+0000","last_became_active":"2024-03-24T00:31:00.054275+0000","last_became_peered":"2024-03-24T00:31:00.054275+0000","last_unstale":"2024-03-24T00:31:01.909460+0000","last_undegraded":"2024-03-24T00:31:01.909460+0000","last_fullsized":"2024-03-24T00:31:01.909460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:38.408736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909730+0000","last_change":"2024-03-24T00:31:00.063073+0000","last_active":"2024-03-24T00:31:01.909730+0000","last_peered":"2024-03-24T00:31:01.909730+0000","last_clean":"2024-03-24T00:31:01.909730+0000","last_became_active":"2024-03-24T00:31:00.062746+0000","last_became_peered":"2024-03-24T00:31:00.062746+0000","last_unstale":"2024-03-24T00:31:01.909730+0000","last_undegraded":"2024-03-24T00:31:01.909730+0000","last_fullsized":"2024-03-24T00:31:01.909730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:32.847580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909247+0000","last_change":"2024-03-24T00:31:00.059874+0000","last_active":"2024-03-24T00:31:01.909247+0000","last_peered":"2024-03-24T00:31:01.909247+0000","last_clean":"2024-03-24T00:31:01.909247+0000","last_became_active":"2024-03-24T00:31:00.059565+0000","last_became_peered":"2024-03-24T00:31:00.059565+0000","last_unstale":"2024-03-24T00:31:01.909247+0000","last_undegraded":"2024-03-24T00:31:01.909247+0000","last_fullsized":"2024-03-24T00:31:01.909247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:22.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909721+0000","last_change":"2024-03-24T00:31:00.056550+0000","last_active":"2024-03-24T00:31:01.909721+0000","last_peered":"2024-03-24T00:31:01.909721+0000","last_clean":"2024-03-24T00:31:01.909721+0000","last_became_active":"2024-03-24T00:31:00.055816+0000","last_became_peered":"2024-03-24T00:31:00.055816+0000","last_unstale":"2024-03-24T00:31:01.909721+0000","last_undegraded":"2024-03-24T00:31:01.909721+0000","last_fullsized":"2024-03-24T00:31:01.909721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:57:10.621965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909977+0000","last_change":"2024-03-24T00:31:00.059028+0000","last_active":"2024-03-24T00:31:01.909977+0000","last_peered":"2024-03-24T00:31:01.909977+0000","last_clean":"2024-03-24T00:31:01.909977+0000","last_became_active":"2024-03-24T00:31:00.058762+0000","last_became_peered":"2024-03-24T00:31:00.058762+0000","last_unstale":"2024-03-24T00:31:01.909977+0000","last_undegraded":"2024-03-24T00:31:01.909977+0000","last_fullsized":"2024-03-24T00:31:01.909977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060333+0000","last_change":"2024-03-24T00:31:00.054838+0000","last_active":"2024-03-24T00:31:01.060333+0000","last_peered":"2024-03-24T00:31:01.060333+0000","last_clean":"2024-03-24T00:31:01.060333+0000","last_became_active":"2024-03-24T00:31:00.054249+0000","last_became_peered":"2024-03-24T00:31:00.054249+0000","last_unstale":"2024-03-24T00:31:01.060333+0000","last_undegraded":"2024-03-24T00:31:01.060333+0000","last_fullsized":"2024-03-24T00:31:01.060333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047846+0000","last_change":"2024-03-24T00:31:00.037600+0000","last_active":"2024-03-24T00:31:01.047846+0000","last_peered":"2024-03-24T00:31:01.047846+0000","last_clean":"2024-03-24T00:31:01.047846+0000","last_became_active":"2024-03-24T00:31:00.037327+0000","last_became_peered":"2024-03-24T00:31:00.037327+0000","last_unstale":"2024-03-24T00:31:01.047846+0000","last_undegraded":"2024-03-24T00:31:01.047846+0000","last_fullsized":"2024-03-24T00:31:01.047846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909766+0000","last_change":"2024-03-24T00:31:00.061415+0000","last_active":"2024-03-24T00:31:01.909766+0000","last_peered":"2024-03-24T00:31:01.909766+0000","last_clean":"2024-03-24T00:31:01.909766+0000","last_became_active":"2024-03-24T00:31:00.061147+0000","last_became_peered":"2024-03-24T00:31:00.061147+0000","last_unstale":"2024-03-24T00:31:01.909766+0000","last_undegraded":"2024-03-24T00:31:01.909766+0000","last_fullsized":"2024-03-24T00:31:01.909766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909601+0000","last_change":"2024-03-24T00:31:00.049400+0000","last_active":"2024-03-24T00:31:01.909601+0000","last_peered":"2024-03-24T00:31:01.909601+0000","last_clean":"2024-03-24T00:31:01.909601+0000","last_became_active":"2024-03-24T00:31:00.048885+0000","last_became_peered":"2024-03-24T00:31:00.048885+0000","last_unstale":"2024-03-24T00:31:01.909601+0000","last_undegraded":"2024-03-24T00:31:01.909601+0000","last_fullsized":"2024-03-24T00:31:01.909601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:22.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909905+0000","last_change":"2024-03-24T00:31:00.056183+0000","last_active":"2024-03-24T00:31:01.909905+0000","last_peered":"2024-03-24T00:31:01.909905+0000","last_clean":"2024-03-24T00:31:01.909905+0000","last_became_active":"2024-03-24T00:31:00.055346+0000","last_became_peered":"2024-03-24T00:31:00.055346+0000","last_unstale":"2024-03-24T00:31:01.909905+0000","last_undegraded":"2024-03-24T00:31:01.909905+0000","last_fullsized":"2024-03-24T00:31:01.909905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:08.370031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186114+0000","last_change":"2024-03-24T00:31:00.066718+0000","last_active":"2024-03-24T00:31:01.186114+0000","last_peered":"2024-03-24T00:31:01.186114+0000","last_clean":"2024-03-24T00:31:01.186114+0000","last_became_active":"2024-03-24T00:31:00.066330+0000","last_became_peered":"2024-03-24T00:31:00.066330+0000","last_unstale":"2024-03-24T00:31:01.186114+0000","last_undegraded":"2024-03-24T00:31:01.186114+0000","last_fullsized":"2024-03-24T00:31:01.186114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:24.063587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059863+0000","last_change":"2024-03-24T00:31:00.045401+0000","last_active":"2024-03-24T00:31:01.059863+0000","last_peered":"2024-03-24T00:31:01.059863+0000","last_clean":"2024-03-24T00:31:01.059863+0000","last_became_active":"2024-03-24T00:31:00.044163+0000","last_became_peered":"2024-03-24T00:31:00.044163+0000","last_unstale":"2024-03-24T00:31:01.059863+0000","last_undegraded":"2024-03-24T00:31:01.059863+0000","last_fullsized":"2024-03-24T00:31:01.059863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:22.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045365+0000","last_change":"2024-03-24T00:31:00.039890+0000","last_active":"2024-03-24T00:31:01.045365+0000","last_peered":"2024-03-24T00:31:01.045365+0000","last_clean":"2024-03-24T00:31:01.045365+0000","last_became_active":"2024-03-24T00:31:00.038341+0000","last_became_peered":"2024-03-24T00:31:00.038341+0000","last_unstale":"2024-03-24T00:31:01.045365+0000","last_undegraded":"2024-03-24T00:31:01.045365+0000","last_fullsized":"2024-03-24T00:31:01.045365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:26.001997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910135+0000","last_change":"2024-03-24T00:30:59.052547+0000","last_active":"2024-03-24T00:31:01.910135+0000","last_peered":"2024-03-24T00:31:01.910135+0000","last_clean":"2024-03-24T00:31:01.910135+0000","last_became_active":"2024-03-24T00:30:59.052277+0000","last_became_peered":"2024-03-24T00:30:59.052277+0000","last_unstale":"2024-03-24T00:31:01.910135+0000","last_undegraded":"2024-03-24T00:31:01.910135+0000","last_fullsized":"2024-03-24T00:31:01.910135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:28:37.149520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909635+0000","last_change":"2024-03-24T00:31:00.064484+0000","last_active":"2024-03-24T00:31:01.909635+0000","last_peered":"2024-03-24T00:31:01.909635+0000","last_clean":"2024-03-24T00:31:01.909635+0000","last_became_active":"2024-03-24T00:31:00.064211+0000","last_became_peered":"2024-03-24T00:31:00.064211+0000","last_unstale":"2024-03-24T00:31:01.909635+0000","last_undegraded":"2024-03-24T00:31:01.909635+0000","last_fullsized":"2024-03-24T00:31:01.909635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059739+0000","last_change":"2024-03-24T00:30:59.053767+0000","last_active":"2024-03-24T00:31:01.059739+0000","last_peered":"2024-03-24T00:31:01.059739+0000","last_clean":"2024-03-24T00:31:01.059739+0000","last_became_active":"2024-03-24T00:30:59.050782+0000","last_became_peered":"2024-03-24T00:30:59.050782+0000","last_unstale":"2024-03-24T00:31:01.059739+0000","last_undegraded":"2024-03-24T00:31:01.059739+0000","last_fullsized":"2024-03-24T00:31:01.059739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909297+0000","last_change":"2024-03-24T00:30:59.047982+0000","last_active":"2024-03-24T00:31:01.909297+0000","last_peered":"2024-03-24T00:31:01.909297+0000","last_clean":"2024-03-24T00:31:01.909297+0000","last_became_active":"2024-03-24T00:30:59.047680+0000","last_became_peered":"2024-03-24T00:30:59.047680+0000","last_unstale":"2024-03-24T00:31:01.909297+0000","last_undegraded":"2024-03-24T00:31:01.909297+0000","last_fullsized":"2024-03-24T00:31:01.909297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:28.293820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910046+0000","last_change":"2024-03-24T00:31:00.044921+0000","last_active":"2024-03-24T00:31:01.910046+0000","last_peered":"2024-03-24T00:31:01.910046+0000","last_clean":"2024-03-24T00:31:01.910046+0000","last_became_active":"2024-03-24T00:31:00.044583+0000","last_became_peered":"2024-03-24T00:31:00.044583+0000","last_unstale":"2024-03-24T00:31:01.910046+0000","last_undegraded":"2024-03-24T00:31:01.910046+0000","last_fullsized":"2024-03-24T00:31:01.910046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:57:10.621965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909657+0000","last_change":"2024-03-24T00:30:59.058420+0000","last_active":"2024-03-24T00:31:01.909657+0000","last_peered":"2024-03-24T00:31:01.909657+0000","last_clean":"2024-03-24T00:31:01.909657+0000","last_became_active":"2024-03-24T00:30:59.058004+0000","last_became_peered":"2024-03-24T00:30:59.058004+0000","last_unstale":"2024-03-24T00:31:01.909657+0000","last_undegraded":"2024-03-24T00:31:01.909657+0000","last_fullsized":"2024-03-24T00:31:01.909657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047409+0000","last_change":"2024-03-24T00:31:00.043778+0000","last_active":"2024-03-24T00:31:01.047409+0000","last_peered":"2024-03-24T00:31:01.047409+0000","last_clean":"2024-03-24T00:31:01.047409+0000","last_became_active":"2024-03-24T00:31:00.043486+0000","last_became_peered":"2024-03-24T00:31:00.043486+0000","last_unstale":"2024-03-24T00:31:01.047409+0000","last_undegraded":"2024-03-24T00:31:01.047409+0000","last_fullsized":"2024-03-24T00:31:01.047409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060075+0000","last_change":"2024-03-24T00:30:59.057537+0000","last_active":"2024-03-24T00:31:01.060075+0000","last_peered":"2024-03-24T00:31:01.060075+0000","last_clean":"2024-03-24T00:31:01.060075+0000","last_became_active":"2024-03-24T00:30:59.057215+0000","last_became_peered":"2024-03-24T00:30:59.057215+0000","last_unstale":"2024-03-24T00:31:01.060075+0000","last_undegraded":"2024-03-24T00:31:01.060075+0000","last_fullsized":"2024-03-24T00:31:01.060075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:40.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910145+0000","last_change":"2024-03-24T00:31:00.057658+0000","last_active":"2024-03-24T00:31:01.910145+0000","last_peered":"2024-03-24T00:31:01.910145+0000","last_clean":"2024-03-24T00:31:01.910145+0000","last_became_active":"2024-03-24T00:31:00.055543+0000","last_became_peered":"2024-03-24T00:31:00.055543+0000","last_unstale":"2024-03-24T00:31:01.910145+0000","last_undegraded":"2024-03-24T00:31:01.910145+0000","last_fullsized":"2024-03-24T00:31:01.910145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:49.592126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910049+0000","last_change":"2024-03-24T00:30:59.058992+0000","last_active":"2024-03-24T00:31:01.910049+0000","last_peered":"2024-03-24T00:31:01.910049+0000","last_clean":"2024-03-24T00:31:01.910049+0000","last_became_active":"2024-03-24T00:30:59.058546+0000","last_became_peered":"2024-03-24T00:30:59.058546+0000","last_unstale":"2024-03-24T00:31:01.910049+0000","last_undegraded":"2024-03-24T00:31:01.910049+0000","last_fullsized":"2024-03-24T00:31:01.910049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:20.001460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909878+0000","last_change":"2024-03-24T00:31:00.057510+0000","last_active":"2024-03-24T00:31:01.909878+0000","last_peered":"2024-03-24T00:31:01.909878+0000","last_clean":"2024-03-24T00:31:01.909878+0000","last_became_active":"2024-03-24T00:31:00.057227+0000","last_became_peered":"2024-03-24T00:31:00.057227+0000","last_unstale":"2024-03-24T00:31:01.909878+0000","last_undegraded":"2024-03-24T00:31:01.909878+0000","last_fullsized":"2024-03-24T00:31:01.909878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047997+0000","last_change":"2024-03-24T00:30:59.055938+0000","last_active":"2024-03-24T00:31:01.047997+0000","last_peered":"2024-03-24T00:31:01.047997+0000","last_clean":"2024-03-24T00:31:01.047997+0000","last_became_active":"2024-03-24T00:30:59.055734+0000","last_became_peered":"2024-03-24T00:30:59.055734+0000","last_unstale":"2024-03-24T00:31:01.047997+0000","last_undegraded":"2024-03-24T00:31:01.047997+0000","last_fullsized":"2024-03-24T00:31:01.047997+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:39.280192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045606+0000","last_change":"2024-03-24T00:31:00.053717+0000","last_active":"2024-03-24T00:31:01.045606+0000","last_peered":"2024-03-24T00:31:01.045606+0000","last_clean":"2024-03-24T00:31:01.045606+0000","last_became_active":"2024-03-24T00:31:00.053312+0000","last_became_peered":"2024-03-24T00:31:00.053312+0000","last_unstale":"2024-03-24T00:31:01.045606+0000","last_undegraded":"2024-03-24T00:31:01.045606+0000","last_fullsized":"2024-03-24T00:31:01.045606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:40.286678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060097+0000","last_change":"2024-03-24T00:30:59.050368+0000","last_active":"2024-03-24T00:31:01.060097+0000","last_peered":"2024-03-24T00:31:01.060097+0000","last_clean":"2024-03-24T00:31:01.060097+0000","last_became_active":"2024-03-24T00:30:59.050062+0000","last_became_peered":"2024-03-24T00:30:59.050062+0000","last_unstale":"2024-03-24T00:31:01.060097+0000","last_undegraded":"2024-03-24T00:31:01.060097+0000","last_fullsized":"2024-03-24T00:31:01.060097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046667+0000","last_change":"2024-03-24T00:31:00.041643+0000","last_active":"2024-03-24T00:31:01.046667+0000","last_peered":"2024-03-24T00:31:01.046667+0000","last_clean":"2024-03-24T00:31:01.046667+0000","last_became_active":"2024-03-24T00:31:00.041387+0000","last_became_peered":"2024-03-24T00:31:00.041387+0000","last_unstale":"2024-03-24T00:31:01.046667+0000","last_undegraded":"2024-03-24T00:31:01.046667+0000","last_fullsized":"2024-03-24T00:31:01.046667+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:34:52.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909659+0000","last_change":"2024-03-24T00:30:59.054246+0000","last_active":"2024-03-24T00:31:01.909659+0000","last_peered":"2024-03-24T00:31:01.909659+0000","last_clean":"2024-03-24T00:31:01.909659+0000","last_became_active":"2024-03-24T00:30:59.053818+0000","last_became_peered":"2024-03-24T00:30:59.053818+0000","last_unstale":"2024-03-24T00:31:01.909659+0000","last_undegraded":"2024-03-24T00:31:01.909659+0000","last_fullsized":"2024-03-24T00:31:01.909659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:28.293820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909855+0000","last_change":"2024-03-24T00:31:00.054481+0000","last_active":"2024-03-24T00:31:01.909855+0000","last_peered":"2024-03-24T00:31:01.909855+0000","last_clean":"2024-03-24T00:31:01.909855+0000","last_became_active":"2024-03-24T00:31:00.054255+0000","last_became_peered":"2024-03-24T00:31:00.054255+0000","last_unstale":"2024-03-24T00:31:01.909855+0000","last_undegraded":"2024-03-24T00:31:01.909855+0000","last_fullsized":"2024-03-24T00:31:01.909855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910033+0000","last_change":"2024-03-24T00:30:59.059529+0000","last_active":"2024-03-24T00:31:01.910033+0000","last_peered":"2024-03-24T00:31:01.910033+0000","last_clean":"2024-03-24T00:31:01.910033+0000","last_became_active":"2024-03-24T00:30:59.058947+0000","last_became_peered":"2024-03-24T00:30:59.058947+0000","last_unstale":"2024-03-24T00:31:01.910033+0000","last_undegraded":"2024-03-24T00:31:01.910033+0000","last_fullsized":"2024-03-24T00:31:01.910033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:39.280192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046876+0000","last_change":"2024-03-24T00:31:00.061372+0000","last_active":"2024-03-24T00:31:01.046876+0000","last_peered":"2024-03-24T00:31:01.046876+0000","last_clean":"2024-03-24T00:31:01.046876+0000","last_became_active":"2024-03-24T00:31:00.061094+0000","last_became_peered":"2024-03-24T00:31:00.061094+0000","last_unstale":"2024-03-24T00:31:01.046876+0000","last_undegraded":"2024-03-24T00:31:01.046876+0000","last_fullsized":"2024-03-24T00:31:01.046876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:38.650664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059711+0000","last_change":"2024-03-24T00:30:59.046862+0000","last_active":"2024-03-24T00:31:01.059711+0000","last_peered":"2024-03-24T00:31:01.059711+0000","last_clean":"2024-03-24T00:31:01.059711+0000","last_became_active":"2024-03-24T00:30:59.046575+0000","last_became_peered":"2024-03-24T00:30:59.046575+0000","last_unstale":"2024-03-24T00:31:01.059711+0000","last_undegraded":"2024-03-24T00:31:01.059711+0000","last_fullsized":"2024-03-24T00:31:01.059711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:39.280192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059910+0000","last_change":"2024-03-24T00:31:00.044599+0000","last_active":"2024-03-24T00:31:01.059910+0000","last_peered":"2024-03-24T00:31:01.059910+0000","last_clean":"2024-03-24T00:31:01.059910+0000","last_became_active":"2024-03-24T00:31:00.043498+0000","last_became_peered":"2024-03-24T00:31:00.043498+0000","last_unstale":"2024-03-24T00:31:01.059910+0000","last_undegraded":"2024-03-24T00:31:01.059910+0000","last_fullsized":"2024-03-24T00:31:01.059910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.185968+0000","last_change":"2024-03-24T00:30:59.039042+0000","last_active":"2024-03-24T00:31:01.185968+0000","last_peered":"2024-03-24T00:31:01.185968+0000","last_clean":"2024-03-24T00:31:01.185968+0000","last_became_active":"2024-03-24T00:30:59.038744+0000","last_became_peered":"2024-03-24T00:30:59.038744+0000","last_unstale":"2024-03-24T00:31:01.185968+0000","last_undegraded":"2024-03-24T00:31:01.185968+0000","last_fullsized":"2024-03-24T00:31:01.185968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:20.001460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045531+0000","last_change":"2024-03-24T00:31:00.047666+0000","last_active":"2024-03-24T00:31:01.045531+0000","last_peered":"2024-03-24T00:31:01.045531+0000","last_clean":"2024-03-24T00:31:01.045531+0000","last_became_active":"2024-03-24T00:31:00.047379+0000","last_became_peered":"2024-03-24T00:31:00.047379+0000","last_unstale":"2024-03-24T00:31:01.045531+0000","last_undegraded":"2024-03-24T00:31:01.045531+0000","last_fullsized":"2024-03-24T00:31:01.045531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:08.370031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045647+0000","last_change":"2024-03-24T00:30:59.043700+0000","last_active":"2024-03-24T00:31:01.045647+0000","last_peered":"2024-03-24T00:31:01.045647+0000","last_clean":"2024-03-24T00:31:01.045647+0000","last_became_active":"2024-03-24T00:30:59.043305+0000","last_became_peered":"2024-03-24T00:30:59.043305+0000","last_unstale":"2024-03-24T00:31:01.045647+0000","last_undegraded":"2024-03-24T00:31:01.045647+0000","last_fullsized":"2024-03-24T00:31:01.045647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:16.392363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046650+0000","last_change":"2024-03-24T00:31:00.048570+0000","last_active":"2024-03-24T00:31:01.046650+0000","last_peered":"2024-03-24T00:31:01.046650+0000","last_clean":"2024-03-24T00:31:01.046650+0000","last_became_active":"2024-03-24T00:31:00.048162+0000","last_became_peered":"2024-03-24T00:31:00.048162+0000","last_unstale":"2024-03-24T00:31:01.046650+0000","last_undegraded":"2024-03-24T00:31:01.046650+0000","last_fullsized":"2024-03-24T00:31:01.046650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:17.380183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186240+0000","last_change":"2024-03-24T00:30:59.041080+0000","last_active":"2024-03-24T00:31:01.186240+0000","last_peered":"2024-03-24T00:31:01.186240+0000","last_clean":"2024-03-24T00:31:01.186240+0000","last_became_active":"2024-03-24T00:30:59.040599+0000","last_became_peered":"2024-03-24T00:30:59.040599+0000","last_unstale":"2024-03-24T00:31:01.186240+0000","last_undegraded":"2024-03-24T00:31:01.186240+0000","last_fullsized":"2024-03-24T00:31:01.186240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047022+0000","last_change":"2024-03-24T00:31:00.060654+0000","last_active":"2024-03-24T00:31:01.047022+0000","last_peered":"2024-03-24T00:31:01.047022+0000","last_clean":"2024-03-24T00:31:01.047022+0000","last_became_active":"2024-03-24T00:31:00.060301+0000","last_became_peered":"2024-03-24T00:31:00.060301+0000","last_unstale":"2024-03-24T00:31:01.047022+0000","last_undegraded":"2024-03-24T00:31:01.047022+0000","last_fullsized":"2024-03-24T00:31:01.047022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:41.152930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047876+0000","last_change":"2024-03-24T00:30:59.045891+0000","last_active":"2024-03-24T00:31:01.047876+0000","last_peered":"2024-03-24T00:31:01.047876+0000","last_clean":"2024-03-24T00:31:01.047876+0000","last_became_active":"2024-03-24T00:30:59.044107+0000","last_became_peered":"2024-03-24T00:30:59.044107+0000","last_unstale":"2024-03-24T00:31:01.047876+0000","last_undegraded":"2024-03-24T00:31:01.047876+0000","last_fullsized":"2024-03-24T00:31:01.047876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:16.613800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910353+0000","last_change":"2024-03-24T00:31:00.056640+0000","last_active":"2024-03-24T00:31:01.910353+0000","last_peered":"2024-03-24T00:31:01.910353+0000","last_clean":"2024-03-24T00:31:01.910353+0000","last_became_active":"2024-03-24T00:31:00.055891+0000","last_became_peered":"2024-03-24T00:31:00.055891+0000","last_unstale":"2024-03-24T00:31:01.910353+0000","last_undegraded":"2024-03-24T00:31:01.910353+0000","last_fullsized":"2024-03-24T00:31:01.910353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:17.791692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186119+0000","last_change":"2024-03-24T00:30:59.058994+0000","last_active":"2024-03-24T00:31:01.186119+0000","last_peered":"2024-03-24T00:31:01.186119+0000","last_clean":"2024-03-24T00:31:01.186119+0000","last_became_active":"2024-03-24T00:30:59.058639+0000","last_became_peered":"2024-03-24T00:30:59.058639+0000","last_unstale":"2024-03-24T00:31:01.186119+0000","last_undegraded":"2024-03-24T00:31:01.186119+0000","last_fullsized":"2024-03-24T00:31:01.186119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046886+0000","last_change":"2024-03-24T00:31:00.059336+0000","last_active":"2024-03-24T00:31:01.046886+0000","last_peered":"2024-03-24T00:31:01.046886+0000","last_clean":"2024-03-24T00:31:01.046886+0000","last_became_active":"2024-03-24T00:31:00.059071+0000","last_became_peered":"2024-03-24T00:31:00.059071+0000","last_unstale":"2024-03-24T00:31:01.046886+0000","last_undegraded":"2024-03-24T00:31:01.046886+0000","last_fullsized":"2024-03-24T00:31:01.046886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:10.867603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046267+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:31:01.046267+0000","last_peered":"2024-03-24T00:31:01.046267+0000","last_clean":"2024-03-24T00:31:01.046267+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:31:01.046267+0000","last_undegraded":"2024-03-24T00:31:01.046267+0000","last_fullsized":"2024-03-24T00:31:01.046267+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909596+0000","last_change":"2024-03-24T00:30:59.059377+0000","last_active":"2024-03-24T00:31:01.909596+0000","last_peered":"2024-03-24T00:31:01.909596+0000","last_clean":"2024-03-24T00:31:01.909596+0000","last_became_active":"2024-03-24T00:30:59.058424+0000","last_became_peered":"2024-03-24T00:30:59.058424+0000","last_unstale":"2024-03-24T00:31:01.909596+0000","last_undegraded":"2024-03-24T00:31:01.909596+0000","last_fullsized":"2024-03-24T00:31:01.909596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909334+0000","last_change":"2024-03-24T00:31:00.058762+0000","last_active":"2024-03-24T00:31:01.909334+0000","last_peered":"2024-03-24T00:31:01.909334+0000","last_clean":"2024-03-24T00:31:01.909334+0000","last_became_active":"2024-03-24T00:31:00.058509+0000","last_became_peered":"2024-03-24T00:31:00.058509+0000","last_unstale":"2024-03-24T00:31:01.909334+0000","last_undegraded":"2024-03-24T00:31:01.909334+0000","last_fullsized":"2024-03-24T00:31:01.909334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046137+0000","last_change":"2024-03-24T00:30:59.050656+0000","last_active":"2024-03-24T00:31:01.046137+0000","last_peered":"2024-03-24T00:31:01.046137+0000","last_clean":"2024-03-24T00:31:01.046137+0000","last_became_active":"2024-03-24T00:30:59.050364+0000","last_became_peered":"2024-03-24T00:30:59.050364+0000","last_unstale":"2024-03-24T00:31:01.046137+0000","last_undegraded":"2024-03-24T00:31:01.046137+0000","last_fullsized":"2024-03-24T00:31:01.046137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:23.150413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909215+0000","last_change":"2024-03-24T00:31:00.059875+0000","last_active":"2024-03-24T00:31:01.909215+0000","last_peered":"2024-03-24T00:31:01.909215+0000","last_clean":"2024-03-24T00:31:01.909215+0000","last_became_active":"2024-03-24T00:31:00.059618+0000","last_became_peered":"2024-03-24T00:31:00.059618+0000","last_unstale":"2024-03-24T00:31:01.909215+0000","last_undegraded":"2024-03-24T00:31:01.909215+0000","last_fullsized":"2024-03-24T00:31:01.909215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:34:42.663552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186071+0000","last_change":"2024-03-24T00:30:59.040845+0000","last_active":"2024-03-24T00:31:01.186071+0000","last_peered":"2024-03-24T00:31:01.186071+0000","last_clean":"2024-03-24T00:31:01.186071+0000","last_became_active":"2024-03-24T00:30:59.040519+0000","last_became_peered":"2024-03-24T00:30:59.040519+0000","last_unstale":"2024-03-24T00:31:01.186071+0000","last_undegraded":"2024-03-24T00:31:01.186071+0000","last_fullsized":"2024-03-24T00:31:01.186071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:28.293820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047374+0000","last_change":"2024-03-24T00:31:00.041015+0000","last_active":"2024-03-24T00:31:01.047374+0000","last_peered":"2024-03-24T00:31:01.047374+0000","last_clean":"2024-03-24T00:31:01.047374+0000","last_became_active":"2024-03-24T00:31:00.040568+0000","last_became_peered":"2024-03-24T00:31:00.040568+0000","last_unstale":"2024-03-24T00:31:01.047374+0000","last_undegraded":"2024-03-24T00:31:01.047374+0000","last_fullsized":"2024-03-24T00:31:01.047374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910220+0000","last_change":"2024-03-24T00:30:59.061747+0000","last_active":"2024-03-24T00:31:01.910220+0000","last_peered":"2024-03-24T00:31:01.910220+0000","last_clean":"2024-03-24T00:31:01.910220+0000","last_became_active":"2024-03-24T00:30:59.059297+0000","last_became_peered":"2024-03-24T00:30:59.059297+0000","last_unstale":"2024-03-24T00:31:01.910220+0000","last_undegraded":"2024-03-24T00:31:01.910220+0000","last_fullsized":"2024-03-24T00:31:01.910220+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:37.099548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045947+0000","last_change":"2024-03-24T00:31:00.050644+0000","last_active":"2024-03-24T00:31:01.045947+0000","last_peered":"2024-03-24T00:31:01.045947+0000","last_clean":"2024-03-24T00:31:01.045947+0000","last_became_active":"2024-03-24T00:31:00.050376+0000","last_became_peered":"2024-03-24T00:31:00.050376+0000","last_unstale":"2024-03-24T00:31:01.045947+0000","last_undegraded":"2024-03-24T00:31:01.045947+0000","last_fullsized":"2024-03-24T00:31:01.045947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:17.620286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047738+0000","last_change":"2024-03-24T00:30:59.045279+0000","last_active":"2024-03-24T00:31:01.047738+0000","last_peered":"2024-03-24T00:31:01.047738+0000","last_clean":"2024-03-24T00:31:01.047738+0000","last_became_active":"2024-03-24T00:30:59.044864+0000","last_became_peered":"2024-03-24T00:30:59.044864+0000","last_unstale":"2024-03-24T00:31:01.047738+0000","last_undegraded":"2024-03-24T00:31:01.047738+0000","last_fullsized":"2024-03-24T00:31:01.047738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:37.099548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910242+0000","last_change":"2024-03-24T00:31:00.056075+0000","last_active":"2024-03-24T00:31:01.910242+0000","last_peered":"2024-03-24T00:31:01.910242+0000","last_clean":"2024-03-24T00:31:01.910242+0000","last_became_active":"2024-03-24T00:31:00.055694+0000","last_became_peered":"2024-03-24T00:31:00.055694+0000","last_unstale":"2024-03-24T00:31:01.910242+0000","last_undegraded":"2024-03-24T00:31:01.910242+0000","last_fullsized":"2024-03-24T00:31:01.910242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:38.650664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047000+0000","last_change":"2024-03-24T00:30:59.048260+0000","last_active":"2024-03-24T00:31:01.047000+0000","last_peered":"2024-03-24T00:31:01.047000+0000","last_clean":"2024-03-24T00:31:01.047000+0000","last_became_active":"2024-03-24T00:30:59.047943+0000","last_became_peered":"2024-03-24T00:30:59.047943+0000","last_unstale":"2024-03-24T00:31:01.047000+0000","last_undegraded":"2024-03-24T00:31:01.047000+0000","last_fullsized":"2024-03-24T00:31:01.047000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:13.589302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047659+0000","last_change":"2024-03-24T00:31:00.041416+0000","last_active":"2024-03-24T00:31:01.047659+0000","last_peered":"2024-03-24T00:31:01.047659+0000","last_clean":"2024-03-24T00:31:01.047659+0000","last_became_active":"2024-03-24T00:31:00.041096+0000","last_became_peered":"2024-03-24T00:31:00.041096+0000","last_unstale":"2024-03-24T00:31:01.047659+0000","last_undegraded":"2024-03-24T00:31:01.047659+0000","last_fullsized":"2024-03-24T00:31:01.047659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047950+0000","last_change":"2024-03-24T00:30:59.046901+0000","last_active":"2024-03-24T00:31:01.047950+0000","last_peered":"2024-03-24T00:31:01.047950+0000","last_clean":"2024-03-24T00:31:01.047950+0000","last_became_active":"2024-03-24T00:30:59.046619+0000","last_became_peered":"2024-03-24T00:30:59.046619+0000","last_unstale":"2024-03-24T00:31:01.047950+0000","last_undegraded":"2024-03-24T00:31:01.047950+0000","last_fullsized":"2024-03-24T00:31:01.047950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:40.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909976+0000","last_change":"2024-03-24T00:31:00.047744+0000","last_active":"2024-03-24T00:31:01.909976+0000","last_peered":"2024-03-24T00:31:01.909976+0000","last_clean":"2024-03-24T00:31:01.909976+0000","last_became_active":"2024-03-24T00:31:00.047453+0000","last_became_peered":"2024-03-24T00:31:00.047453+0000","last_unstale":"2024-03-24T00:31:01.909976+0000","last_undegraded":"2024-03-24T00:31:01.909976+0000","last_fullsized":"2024-03-24T00:31:01.909976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060070+0000","last_change":"2024-03-24T00:30:59.047379+0000","last_active":"2024-03-24T00:31:01.060070+0000","last_peered":"2024-03-24T00:31:01.060070+0000","last_clean":"2024-03-24T00:31:01.060070+0000","last_became_active":"2024-03-24T00:30:59.045528+0000","last_became_peered":"2024-03-24T00:30:59.045528+0000","last_unstale":"2024-03-24T00:31:01.060070+0000","last_undegraded":"2024-03-24T00:31:01.060070+0000","last_fullsized":"2024-03-24T00:31:01.060070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:23.057101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909624+0000","last_change":"2024-03-24T00:31:00.050676+0000","last_active":"2024-03-24T00:31:01.909624+0000","last_peered":"2024-03-24T00:31:01.909624+0000","last_clean":"2024-03-24T00:31:01.909624+0000","last_became_active":"2024-03-24T00:31:00.050421+0000","last_became_peered":"2024-03-24T00:31:00.050421+0000","last_unstale":"2024-03-24T00:31:01.909624+0000","last_undegraded":"2024-03-24T00:31:01.909624+0000","last_fullsized":"2024-03-24T00:31:01.909624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045080+0000","last_change":"2024-03-24T00:30:59.047814+0000","last_active":"2024-03-24T00:31:01.045080+0000","last_peered":"2024-03-24T00:31:01.045080+0000","last_clean":"2024-03-24T00:31:01.045080+0000","last_became_active":"2024-03-24T00:30:59.047497+0000","last_became_peered":"2024-03-24T00:30:59.047497+0000","last_unstale":"2024-03-24T00:31:01.045080+0000","last_undegraded":"2024-03-24T00:31:01.045080+0000","last_fullsized":"2024-03-24T00:31:01.045080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:37.099548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046279+0000","last_change":"2024-03-24T00:31:00.043642+0000","last_active":"2024-03-24T00:31:01.046279+0000","last_peered":"2024-03-24T00:31:01.046279+0000","last_clean":"2024-03-24T00:31:01.046279+0000","last_became_active":"2024-03-24T00:31:00.043334+0000","last_became_peered":"2024-03-24T00:31:00.043334+0000","last_unstale":"2024-03-24T00:31:01.046279+0000","last_undegraded":"2024-03-24T00:31:01.046279+0000","last_fullsized":"2024-03-24T00:31:01.046279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:41.254416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047463+0000","last_change":"2024-03-24T00:30:59.044547+0000","last_active":"2024-03-24T00:31:01.047463+0000","last_peered":"2024-03-24T00:31:01.047463+0000","last_clean":"2024-03-24T00:31:01.047463+0000","last_became_active":"2024-03-24T00:30:59.044240+0000","last_became_peered":"2024-03-24T00:30:59.044240+0000","last_unstale":"2024-03-24T00:31:01.047463+0000","last_undegraded":"2024-03-24T00:31:01.047463+0000","last_fullsized":"2024-03-24T00:31:01.047463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910154+0000","last_change":"2024-03-24T00:31:00.043313+0000","last_active":"2024-03-24T00:31:01.910154+0000","last_peered":"2024-03-24T00:31:01.910154+0000","last_clean":"2024-03-24T00:31:01.910154+0000","last_became_active":"2024-03-24T00:31:00.043030+0000","last_became_peered":"2024-03-24T00:31:00.043030+0000","last_unstale":"2024-03-24T00:31:01.910154+0000","last_undegraded":"2024-03-24T00:31:01.910154+0000","last_fullsized":"2024-03-24T00:31:01.910154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:32.847580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047581+0000","last_change":"2024-03-24T00:30:59.045395+0000","last_active":"2024-03-24T00:31:01.047581+0000","last_peered":"2024-03-24T00:31:01.047581+0000","last_clean":"2024-03-24T00:31:01.047581+0000","last_became_active":"2024-03-24T00:30:59.044888+0000","last_became_peered":"2024-03-24T00:30:59.044888+0000","last_unstale":"2024-03-24T00:31:01.047581+0000","last_undegraded":"2024-03-24T00:31:01.047581+0000","last_fullsized":"2024-03-24T00:31:01.047581+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910336+0000","last_change":"2024-03-24T00:31:00.052157+0000","last_active":"2024-03-24T00:31:01.910336+0000","last_peered":"2024-03-24T00:31:01.910336+0000","last_clean":"2024-03-24T00:31:01.910336+0000","last_became_active":"2024-03-24T00:31:00.051147+0000","last_became_peered":"2024-03-24T00:31:00.051147+0000","last_unstale":"2024-03-24T00:31:01.910336+0000","last_undegraded":"2024-03-24T00:31:01.910336+0000","last_fullsized":"2024-03-24T00:31:01.910336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:08.370031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046873+0000","last_change":"2024-03-24T00:30:59.053443+0000","last_active":"2024-03-24T00:31:01.046873+0000","last_peered":"2024-03-24T00:31:01.046873+0000","last_clean":"2024-03-24T00:31:01.046873+0000","last_became_active":"2024-03-24T00:30:59.053111+0000","last_became_peered":"2024-03-24T00:30:59.053111+0000","last_unstale":"2024-03-24T00:31:01.046873+0000","last_undegraded":"2024-03-24T00:31:01.046873+0000","last_fullsized":"2024-03-24T00:31:01.046873+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:40.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059908+0000","last_change":"2024-03-24T00:31:00.061892+0000","last_active":"2024-03-24T00:31:01.059908+0000","last_peered":"2024-03-24T00:31:01.059908+0000","last_clean":"2024-03-24T00:31:01.059908+0000","last_became_active":"2024-03-24T00:31:00.061567+0000","last_became_peered":"2024-03-24T00:31:00.061567+0000","last_unstale":"2024-03-24T00:31:01.059908+0000","last_undegraded":"2024-03-24T00:31:01.059908+0000","last_fullsized":"2024-03-24T00:31:01.059908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910099+0000","last_change":"2024-03-24T00:31:00.069008+0000","last_active":"2024-03-24T00:31:01.910099+0000","last_peered":"2024-03-24T00:31:01.910099+0000","last_clean":"2024-03-24T00:31:01.910099+0000","last_became_active":"2024-03-24T00:31:00.065217+0000","last_became_peered":"2024-03-24T00:31:00.065217+0000","last_unstale":"2024-03-24T00:31:01.910099+0000","last_undegraded":"2024-03-24T00:31:01.910099+0000","last_fullsized":"2024-03-24T00:31:01.910099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:34:52.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046539+0000","last_change":"2024-03-24T00:30:59.060493+0000","last_active":"2024-03-24T00:31:01.046539+0000","last_peered":"2024-03-24T00:31:01.046539+0000","last_clean":"2024-03-24T00:31:01.046539+0000","last_became_active":"2024-03-24T00:30:59.060208+0000","last_became_peered":"2024-03-24T00:30:59.060208+0000","last_unstale":"2024-03-24T00:31:01.046539+0000","last_undegraded":"2024-03-24T00:31:01.046539+0000","last_fullsized":"2024-03-24T00:31:01.046539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:16.613800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910297+0000","last_change":"2024-03-24T00:31:00.071223+0000","last_active":"2024-03-24T00:31:01.910297+0000","last_peered":"2024-03-24T00:31:01.910297+0000","last_clean":"2024-03-24T00:31:01.910297+0000","last_became_active":"2024-03-24T00:31:00.069882+0000","last_became_peered":"2024-03-24T00:31:00.069882+0000","last_unstale":"2024-03-24T00:31:01.910297+0000","last_undegraded":"2024-03-24T00:31:01.910297+0000","last_fullsized":"2024-03-24T00:31:01.910297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:09.487064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046833+0000","last_change":"2024-03-24T00:30:59.052020+0000","last_active":"2024-03-24T00:31:01.046833+0000","last_peered":"2024-03-24T00:31:01.046833+0000","last_clean":"2024-03-24T00:31:01.046833+0000","last_became_active":"2024-03-24T00:30:59.049277+0000","last_became_peered":"2024-03-24T00:30:59.049277+0000","last_unstale":"2024-03-24T00:31:01.046833+0000","last_undegraded":"2024-03-24T00:31:01.046833+0000","last_fullsized":"2024-03-24T00:31:01.046833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:23.057101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909752+0000","last_change":"2024-03-24T00:31:00.072313+0000","last_active":"2024-03-24T00:31:01.909752+0000","last_peered":"2024-03-24T00:31:01.909752+0000","last_clean":"2024-03-24T00:31:01.909752+0000","last_became_active":"2024-03-24T00:31:00.071692+0000","last_became_peered":"2024-03-24T00:31:00.071692+0000","last_unstale":"2024-03-24T00:31:01.909752+0000","last_undegraded":"2024-03-24T00:31:01.909752+0000","last_fullsized":"2024-03-24T00:31:01.909752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910030+0000","last_change":"2024-03-24T00:30:59.045778+0000","last_active":"2024-03-24T00:31:01.910030+0000","last_peered":"2024-03-24T00:31:01.910030+0000","last_clean":"2024-03-24T00:31:01.910030+0000","last_became_active":"2024-03-24T00:30:59.045515+0000","last_became_peered":"2024-03-24T00:30:59.045515+0000","last_unstale":"2024-03-24T00:31:01.910030+0000","last_undegraded":"2024-03-24T00:31:01.910030+0000","last_fullsized":"2024-03-24T00:31:01.910030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:20.001460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909974+0000","last_change":"2024-03-24T00:31:00.047507+0000","last_active":"2024-03-24T00:31:01.909974+0000","last_peered":"2024-03-24T00:31:01.909974+0000","last_clean":"2024-03-24T00:31:01.909974+0000","last_became_active":"2024-03-24T00:31:00.047245+0000","last_became_peered":"2024-03-24T00:31:00.047245+0000","last_unstale":"2024-03-24T00:31:01.909974+0000","last_undegraded":"2024-03-24T00:31:01.909974+0000","last_fullsized":"2024-03-24T00:31:01.909974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909924+0000","last_change":"2024-03-24T00:30:59.053442+0000","last_active":"2024-03-24T00:31:01.909924+0000","last_peered":"2024-03-24T00:31:01.909924+0000","last_clean":"2024-03-24T00:31:01.909924+0000","last_became_active":"2024-03-24T00:30:59.053159+0000","last_became_peered":"2024-03-24T00:30:59.053159+0000","last_unstale":"2024-03-24T00:31:01.909924+0000","last_undegraded":"2024-03-24T00:31:01.909924+0000","last_fullsized":"2024-03-24T00:31:01.909924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:28:37.149520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:31:11.920 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:31:12.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc72fa00640 1 Processor -- start 2024-03-24T00:31:12.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc72fa00640 1 -- start start 2024-03-24T00:31:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc72fa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc7280d2830 0x7fc728166e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc72fa00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7280af880 con 0x7fc72815f600 2024-03-24T00:31:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc72fa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7280b1e50 con 0x7fc7280b2910 2024-03-24T00:31:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc72fa00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7280b3630 con 0x7fc7280d2830 2024-03-24T00:31:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc72d000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc72815f600 0x7fc7280aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57970/0 (socket says 172.21.15.40:57970) 2024-03-24T00:31:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc72d000640 1 -- 172.21.15.40:0/764632855 learned_addr learned my addr 172.21.15.40:0/764632855 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc72da00640 1 --2- 172.21.15.40:0/764632855 >> v2:172.21.15.53:3300/0 conn(0x7fc7280d2830 0x7fc728166e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530524705 0 0) 0x7fc7280af880 con 0x7fc72815f600 2024-03-24T00:31:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc710003530 con 0x7fc72815f600 2024-03-24T00:31:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1427596478 0 0) 0x7fc710003530 con 0x7fc72815f600 2024-03-24T00:31:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 >> v2:172.21.15.53:3300/0 conn(0x7fc7280d2830 msgr2=0x7fc728166e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 --2- 172.21.15.40:0/764632855 >> v2:172.21.15.53:3300/0 conn(0x7fc7280d2830 0x7fc728166e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 >> v1:172.21.15.53:6789/0 conn(0x7fc7280b2910 legacy=0x7fc7280af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7280d2c40 con 0x7fc72815f600 2024-03-24T00:31:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fc71c001680 con 0x7fc72815f600 2024-03-24T00:31:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 >> v1:172.21.15.40:6789/0 conn(0x7fc72815f600 legacy=0x7fc7280aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 --2- 172.21.15.40:0/764632855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7100039c0 0x7fc710003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 --2- 172.21.15.40:0/764632855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc710004810 0x7fc710004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 --2- 172.21.15.40:0/764632855 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7100051d0 0x7fc71000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7280b3630 con 0x7fc710004810 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71000c080 con 0x7fc7100039c0 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71000c280 con 0x7fc7100051d0 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727e00640 1 --2- 172.21.15.40:0/764632855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc710004810 0x7fc710004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc72d000640 1 --2- 172.21.15.40:0/764632855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7100039c0 0x7fc710003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc72da00640 1 --2- 172.21.15.40:0/764632855 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7100051d0 0x7fc71000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727e00640 1 -- 172.21.15.40:0/764632855 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7100051d0 msgr2=0x7fc71000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727e00640 1 --2- 172.21.15.40:0/764632855 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7100051d0 0x7fc71000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727e00640 1 -- 172.21.15.40:0/764632855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7100039c0 msgr2=0x7fc710003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727e00640 1 --2- 172.21.15.40:0/764632855 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7100039c0 0x7fc710003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc727e00640 1 -- 172.21.15.40:0/764632855 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc710003530 con 0x7fc710004810 2024-03-24T00:31:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.098+0000 7fc72da00640 1 -- 172.21.15.40:0/764632855 reap_dead start 2024-03-24T00:31:12.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.102+0000 7fc727e00640 1 --2- 172.21.15.40:0/764632855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc710004810 0x7fc710004c90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc714001780 tx=0x7fc71401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc4eff9c1bd6e2d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:12.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.102+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7140013a0 con 0x7fc710004810 2024-03-24T00:31:12.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.102+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7140052f0 con 0x7fc710004810 2024-03-24T00:31:12.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.102+0000 7fc727400640 1 -- 172.21.15.40:0/764632855 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc714003840 con 0x7fc710004810 2024-03-24T00:31:12.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.102+0000 7fc72fa00640 1 -- 172.21.15.40:0/764632855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc710004810 msgr2=0x7fc710004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.102+0000 7fc72fa00640 1 --2- 172.21.15.40:0/764632855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc710004810 0x7fc710004c90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc714001780 tx=0x7fc71401c400 comp rx=0 tx=0).stop 2024-03-24T00:31:12.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.102+0000 7fc72fa00640 1 -- 172.21.15.40:0/764632855 shutdown_connections 2024-03-24T00:31:12.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.102+0000 7fc72fa00640 1 --2- 172.21.15.40:0/764632855 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc710004810 0x7fc710004c90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.102+0000 7fc72fa00640 1 -- 172.21.15.40:0/764632855 >> 172.21.15.40:0/764632855 conn(0x7fc7280d1d10 msgr2=0x7fc7280d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:12.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.102+0000 7fc72fa00640 1 -- 172.21.15.40:0/764632855 shutdown_connections 2024-03-24T00:31:12.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.102+0000 7fc72fa00640 1 -- 172.21.15.40:0/764632855 wait complete. 2024-03-24T00:31:12.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72fa00640 1 Processor -- start 2024-03-24T00:31:12.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72fa00640 1 -- start start 2024-03-24T00:31:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72fa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc710004810 0x7fc7280c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72fa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc728154710 0x7fc7280c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72fa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7280c7a40 0x7fc7280cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72fa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7280b2740 con 0x7fc7280c7a40 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72fa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7280b2d80 con 0x7fc710004810 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72fa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7280b23d0 con 0x7fc728154710 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7280c7a40 0x7fc7280cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7280c7a40 0x7fc7280cb0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47260/0 (socket says 172.21.15.40:47260) 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72da00640 1 -- 172.21.15.40:0/4263091162 learned_addr learned my addr 172.21.15.40:0/4263091162 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72d000640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc710004810 0x7fc7280c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc727e00640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc728154710 0x7fc7280c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72da00640 1 -- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc728154710 msgr2=0x7fc7280c7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72da00640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc728154710 0x7fc7280c7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72da00640 1 -- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc710004810 msgr2=0x7fc7280c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72da00640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc710004810 0x7fc7280c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72da00640 1 -- 172.21.15.40:0/4263091162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7280c5df0 con 0x7fc7280c7a40 2024-03-24T00:31:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72d000640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc710004810 0x7fc7280c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72da00640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7280c7a40 0x7fc7280cb0a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc7280cc600 tx=0x7fc72000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc724c00640 1 -- 172.21.15.40:0/4263091162 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72000fd80 con 0x7fc7280c7a40 2024-03-24T00:31:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc724c00640 1 -- 172.21.15.40:0/4263091162 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72001d070 con 0x7fc7280c7a40 2024-03-24T00:31:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc724c00640 1 -- 172.21.15.40:0/4263091162 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720015670 con 0x7fc7280c7a40 2024-03-24T00:31:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72fa00640 1 -- 172.21.15.40:0/4263091162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7280c60d0 con 0x7fc7280c7a40 2024-03-24T00:31:12.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.106+0000 7fc72fa00640 1 -- 172.21.15.40:0/4263091162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7280c6690 con 0x7fc7280c7a40 2024-03-24T00:31:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.110+0000 7fc72fa00640 1 -- 172.21.15.40:0/4263091162 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc728068170 con 0x7fc7280c7a40 2024-03-24T00:31:12.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.110+0000 7fc724c00640 1 -- 172.21.15.40:0/4263091162 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc720015810 con 0x7fc7280c7a40 2024-03-24T00:31:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.110+0000 7fc724c00640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc708067660 0x7fc708069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.110+0000 7fc724c00640 1 -- 172.21.15.40:0/4263091162 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc720086490 con 0x7fc7280c7a40 2024-03-24T00:31:12.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.110+0000 7fc72d000640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc708067660 0x7fc708069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.114+0000 7fc72d000640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc708067660 0x7fc708069b20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc728050d50 tx=0x7fc71c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:12.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.114+0000 7fc724c00640 1 -- 172.21.15.40:0/4263091162 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc720027020 con 0x7fc7280c7a40 2024-03-24T00:31:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.274+0000 7fc72fa00640 1 -- 172.21.15.40:0/4263091162 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc7280b3280 con 0x7fc708067660 2024-03-24T00:31:12.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.282+0000 7fc724c00640 1 -- 172.21.15.40:0/4263091162 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251837 (secure 0 0 0) 0x7fc7280b3280 con 0x7fc708067660 2024-03-24T00:31:12.288 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:12.289 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:31:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.286+0000 7fc72fa00640 1 -- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc708067660 msgr2=0x7fc708069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.286+0000 7fc72fa00640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc708067660 0x7fc708069b20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc728050d50 tx=0x7fc71c009040 comp rx=0 tx=0).stop 2024-03-24T00:31:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.286+0000 7fc72fa00640 1 -- 172.21.15.40:0/4263091162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7280c7a40 msgr2=0x7fc7280cb0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.286+0000 7fc72fa00640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7280c7a40 0x7fc7280cb0a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc7280cc600 tx=0x7fc72000f580 comp rx=0 tx=0).stop 2024-03-24T00:31:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.286+0000 7fc72fa00640 1 -- 172.21.15.40:0/4263091162 shutdown_connections 2024-03-24T00:31:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.286+0000 7fc72fa00640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc708067660 0x7fc708069b20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.286+0000 7fc72fa00640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7280c7a40 0x7fc7280cb0a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.286+0000 7fc72fa00640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc728154710 0x7fc7280c7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.286+0000 7fc72fa00640 1 --2- 172.21.15.40:0/4263091162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc710004810 0x7fc7280c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.286+0000 7fc72fa00640 1 -- 172.21.15.40:0/4263091162 >> 172.21.15.40:0/4263091162 conn(0x7fc7280d1d10 msgr2=0x7fc72815ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.286+0000 7fc72fa00640 1 -- 172.21.15.40:0/4263091162 shutdown_connections 2024-03-24T00:31:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.286+0000 7fc72fa00640 1 -- 172.21.15.40:0/4263091162 wait complete. 2024-03-24T00:31:12.309 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-24T00:31:11.199799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235632,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739088,"statfs":{"total":773094113280,"available":772852826112,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.157152"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046898+0000","last_change":"2024-03-24T00:31:00.052380+0000","last_active":"2024-03-24T00:31:01.046898+0000","last_peered":"2024-03-24T00:31:01.046898+0000","last_clean":"2024-03-24T00:31:01.046898+0000","last_became_active":"2024-03-24T00:31:00.049815+0000","last_became_peered":"2024-03-24T00:31:00.049815+0000","last_unstale":"2024-03-24T00:31:01.046898+0000","last_undegraded":"2024-03-24T00:31:01.046898+0000","last_fullsized":"2024-03-24T00:31:01.046898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:09.487064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047941+0000","last_change":"2024-03-24T00:31:00.048401+0000","last_active":"2024-03-24T00:31:01.047941+0000","last_peered":"2024-03-24T00:31:01.047941+0000","last_clean":"2024-03-24T00:31:01.047941+0000","last_became_active":"2024-03-24T00:31:00.048157+0000","last_became_peered":"2024-03-24T00:31:00.048157+0000","last_unstale":"2024-03-24T00:31:01.047941+0000","last_undegraded":"2024-03-24T00:31:01.047941+0000","last_fullsized":"2024-03-24T00:31:01.047941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909458+0000","last_change":"2024-03-24T00:31:00.052594+0000","last_active":"2024-03-24T00:31:01.909458+0000","last_peered":"2024-03-24T00:31:01.909458+0000","last_clean":"2024-03-24T00:31:01.909458+0000","last_became_active":"2024-03-24T00:31:00.052289+0000","last_became_peered":"2024-03-24T00:31:00.052289+0000","last_unstale":"2024-03-24T00:31:01.909458+0000","last_undegraded":"2024-03-24T00:31:01.909458+0000","last_fullsized":"2024-03-24T00:31:01.909458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:34.269764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909931+0000","last_change":"2024-03-24T00:31:00.072520+0000","last_active":"2024-03-24T00:31:01.909931+0000","last_peered":"2024-03-24T00:31:01.909931+0000","last_clean":"2024-03-24T00:31:01.909931+0000","last_became_active":"2024-03-24T00:31:00.071355+0000","last_became_peered":"2024-03-24T00:31:00.071355+0000","last_unstale":"2024-03-24T00:31:01.909931+0000","last_undegraded":"2024-03-24T00:31:01.909931+0000","last_fullsized":"2024-03-24T00:31:01.909931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:17.380183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046806+0000","last_change":"2024-03-24T00:31:00.045641+0000","last_active":"2024-03-24T00:31:01.046806+0000","last_peered":"2024-03-24T00:31:01.046806+0000","last_clean":"2024-03-24T00:31:01.046806+0000","last_became_active":"2024-03-24T00:31:00.045230+0000","last_became_peered":"2024-03-24T00:31:00.045230+0000","last_unstale":"2024-03-24T00:31:01.046806+0000","last_undegraded":"2024-03-24T00:31:01.046806+0000","last_fullsized":"2024-03-24T00:31:01.046806+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:49.592126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909658+0000","last_change":"2024-03-24T00:31:00.057373+0000","last_active":"2024-03-24T00:31:01.909658+0000","last_peered":"2024-03-24T00:31:01.909658+0000","last_clean":"2024-03-24T00:31:01.909658+0000","last_became_active":"2024-03-24T00:31:00.056322+0000","last_became_peered":"2024-03-24T00:31:00.056322+0000","last_unstale":"2024-03-24T00:31:01.909658+0000","last_undegraded":"2024-03-24T00:31:01.909658+0000","last_fullsized":"2024-03-24T00:31:01.909658+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045369+0000","last_change":"2024-03-24T00:31:00.048380+0000","last_active":"2024-03-24T00:31:01.045369+0000","last_peered":"2024-03-24T00:31:01.045369+0000","last_clean":"2024-03-24T00:31:01.045369+0000","last_became_active":"2024-03-24T00:31:00.046936+0000","last_became_peered":"2024-03-24T00:31:00.046936+0000","last_unstale":"2024-03-24T00:31:01.045369+0000","last_undegraded":"2024-03-24T00:31:01.045369+0000","last_fullsized":"2024-03-24T00:31:01.045369+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:32.847580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909134+0000","last_change":"2024-03-24T00:31:00.057172+0000","last_active":"2024-03-24T00:31:01.909134+0000","last_peered":"2024-03-24T00:31:01.909134+0000","last_clean":"2024-03-24T00:31:01.909134+0000","last_became_active":"2024-03-24T00:31:00.056982+0000","last_became_peered":"2024-03-24T00:31:00.056982+0000","last_unstale":"2024-03-24T00:31:01.909134+0000","last_undegraded":"2024-03-24T00:31:01.909134+0000","last_fullsized":"2024-03-24T00:31:01.909134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:38.408736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047010+0000","last_change":"2024-03-24T00:31:00.053066+0000","last_active":"2024-03-24T00:31:01.047010+0000","last_peered":"2024-03-24T00:31:01.047010+0000","last_clean":"2024-03-24T00:31:01.047010+0000","last_became_active":"2024-03-24T00:31:00.052749+0000","last_became_peered":"2024-03-24T00:31:00.052749+0000","last_unstale":"2024-03-24T00:31:01.047010+0000","last_undegraded":"2024-03-24T00:31:01.047010+0000","last_fullsized":"2024-03-24T00:31:01.047010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:17.791692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060254+0000","last_change":"2024-03-24T00:31:00.060700+0000","last_active":"2024-03-24T00:31:01.060254+0000","last_peered":"2024-03-24T00:31:01.060254+0000","last_clean":"2024-03-24T00:31:01.060254+0000","last_became_active":"2024-03-24T00:31:00.060373+0000","last_became_peered":"2024-03-24T00:31:00.060373+0000","last_unstale":"2024-03-24T00:31:01.060254+0000","last_undegraded":"2024-03-24T00:31:01.060254+0000","last_fullsized":"2024-03-24T00:31:01.060254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060101+0000","last_change":"2024-03-24T00:31:00.061316+0000","last_active":"2024-03-24T00:31:01.060101+0000","last_peered":"2024-03-24T00:31:01.060101+0000","last_clean":"2024-03-24T00:31:01.060101+0000","last_became_active":"2024-03-24T00:31:00.060980+0000","last_became_peered":"2024-03-24T00:31:00.060980+0000","last_unstale":"2024-03-24T00:31:01.060101+0000","last_undegraded":"2024-03-24T00:31:01.060101+0000","last_fullsized":"2024-03-24T00:31:01.060101+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045818+0000","last_change":"2024-03-24T00:31:00.046162+0000","last_active":"2024-03-24T00:31:01.045818+0000","last_peered":"2024-03-24T00:31:01.045818+0000","last_clean":"2024-03-24T00:31:01.045818+0000","last_became_active":"2024-03-24T00:31:00.045715+0000","last_became_peered":"2024-03-24T00:31:00.045715+0000","last_unstale":"2024-03-24T00:31:01.045818+0000","last_undegraded":"2024-03-24T00:31:01.045818+0000","last_fullsized":"2024-03-24T00:31:01.045818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:38.106034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909352+0000","last_change":"2024-03-24T00:31:00.045645+0000","last_active":"2024-03-24T00:31:01.909352+0000","last_peered":"2024-03-24T00:31:01.909352+0000","last_clean":"2024-03-24T00:31:01.909352+0000","last_became_active":"2024-03-24T00:31:00.045372+0000","last_became_peered":"2024-03-24T00:31:00.045372+0000","last_unstale":"2024-03-24T00:31:01.909352+0000","last_undegraded":"2024-03-24T00:31:01.909352+0000","last_fullsized":"2024-03-24T00:31:01.909352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909717+0000","last_change":"2024-03-24T00:31:00.042718+0000","last_active":"2024-03-24T00:31:01.909717+0000","last_peered":"2024-03-24T00:31:01.909717+0000","last_clean":"2024-03-24T00:31:01.909717+0000","last_became_active":"2024-03-24T00:31:00.042424+0000","last_became_peered":"2024-03-24T00:31:00.042424+0000","last_unstale":"2024-03-24T00:31:01.909717+0000","last_undegraded":"2024-03-24T00:31:01.909717+0000","last_fullsized":"2024-03-24T00:31:01.909717+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047428+0000","last_change":"2024-03-24T00:31:00.038791+0000","last_active":"2024-03-24T00:31:01.047428+0000","last_peered":"2024-03-24T00:31:01.047428+0000","last_clean":"2024-03-24T00:31:01.047428+0000","last_became_active":"2024-03-24T00:31:00.038543+0000","last_became_peered":"2024-03-24T00:31:00.038543+0000","last_unstale":"2024-03-24T00:31:01.047428+0000","last_undegraded":"2024-03-24T00:31:01.047428+0000","last_fullsized":"2024-03-24T00:31:01.047428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909570+0000","last_change":"2024-03-24T00:31:00.055749+0000","last_active":"2024-03-24T00:31:01.909570+0000","last_peered":"2024-03-24T00:31:01.909570+0000","last_clean":"2024-03-24T00:31:01.909570+0000","last_became_active":"2024-03-24T00:31:00.055296+0000","last_became_peered":"2024-03-24T00:31:00.055296+0000","last_unstale":"2024-03-24T00:31:01.909570+0000","last_undegraded":"2024-03-24T00:31:01.909570+0000","last_fullsized":"2024-03-24T00:31:01.909570+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910277+0000","last_change":"2024-03-24T00:31:00.055782+0000","last_active":"2024-03-24T00:31:01.910277+0000","last_peered":"2024-03-24T00:31:01.910277+0000","last_clean":"2024-03-24T00:31:01.910277+0000","last_became_active":"2024-03-24T00:31:00.055369+0000","last_became_peered":"2024-03-24T00:31:00.055369+0000","last_unstale":"2024-03-24T00:31:01.910277+0000","last_undegraded":"2024-03-24T00:31:01.910277+0000","last_fullsized":"2024-03-24T00:31:01.910277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:10.867603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186278+0000","last_change":"2024-03-24T00:31:00.044665+0000","last_active":"2024-03-24T00:31:01.186278+0000","last_peered":"2024-03-24T00:31:01.186278+0000","last_clean":"2024-03-24T00:31:01.186278+0000","last_became_active":"2024-03-24T00:31:00.044399+0000","last_became_peered":"2024-03-24T00:31:00.044399+0000","last_unstale":"2024-03-24T00:31:01.186278+0000","last_undegraded":"2024-03-24T00:31:01.186278+0000","last_fullsized":"2024-03-24T00:31:01.186278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:14.595788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047109+0000","last_change":"2024-03-24T00:31:00.045352+0000","last_active":"2024-03-24T00:31:01.047109+0000","last_peered":"2024-03-24T00:31:01.047109+0000","last_clean":"2024-03-24T00:31:01.047109+0000","last_became_active":"2024-03-24T00:31:00.045023+0000","last_became_peered":"2024-03-24T00:31:00.045023+0000","last_unstale":"2024-03-24T00:31:01.047109+0000","last_undegraded":"2024-03-24T00:31:01.047109+0000","last_fullsized":"2024-03-24T00:31:01.047109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:31.480032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060057+0000","last_change":"2024-03-24T00:31:00.061257+0000","last_active":"2024-03-24T00:31:01.060057+0000","last_peered":"2024-03-24T00:31:01.060057+0000","last_clean":"2024-03-24T00:31:01.060057+0000","last_became_active":"2024-03-24T00:31:00.060983+0000","last_became_peered":"2024-03-24T00:31:00.060983+0000","last_unstale":"2024-03-24T00:31:01.060057+0000","last_undegraded":"2024-03-24T00:31:01.060057+0000","last_fullsized":"2024-03-24T00:31:01.060057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909460+0000","last_change":"2024-03-24T00:31:00.054714+0000","last_active":"2024-03-24T00:31:01.909460+0000","last_peered":"2024-03-24T00:31:01.909460+0000","last_clean":"2024-03-24T00:31:01.909460+0000","last_became_active":"2024-03-24T00:31:00.054275+0000","last_became_peered":"2024-03-24T00:31:00.054275+0000","last_unstale":"2024-03-24T00:31:01.909460+0000","last_undegraded":"2024-03-24T00:31:01.909460+0000","last_fullsized":"2024-03-24T00:31:01.909460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:44:38.408736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909730+0000","last_change":"2024-03-24T00:31:00.063073+0000","last_active":"2024-03-24T00:31:01.909730+0000","last_peered":"2024-03-24T00:31:01.909730+0000","last_clean":"2024-03-24T00:31:01.909730+0000","last_became_active":"2024-03-24T00:31:00.062746+0000","last_became_peered":"2024-03-24T00:31:00.062746+0000","last_unstale":"2024-03-24T00:31:01.909730+0000","last_undegraded":"2024-03-24T00:31:01.909730+0000","last_fullsized":"2024-03-24T00:31:01.909730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:32.847580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909247+0000","last_change":"2024-03-24T00:31:00.059874+0000","last_active":"2024-03-24T00:31:01.909247+0000","last_peered":"2024-03-24T00:31:01.909247+0000","last_clean":"2024-03-24T00:31:01.909247+0000","last_became_active":"2024-03-24T00:31:00.059565+0000","last_became_peered":"2024-03-24T00:31:00.059565+0000","last_unstale":"2024-03-24T00:31:01.909247+0000","last_undegraded":"2024-03-24T00:31:01.909247+0000","last_fullsized":"2024-03-24T00:31:01.909247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:22.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909721+0000","last_change":"2024-03-24T00:31:00.056550+0000","last_active":"2024-03-24T00:31:01.909721+0000","last_peered":"2024-03-24T00:31:01.909721+0000","last_clean":"2024-03-24T00:31:01.909721+0000","last_became_active":"2024-03-24T00:31:00.055816+0000","last_became_peered":"2024-03-24T00:31:00.055816+0000","last_unstale":"2024-03-24T00:31:01.909721+0000","last_undegraded":"2024-03-24T00:31:01.909721+0000","last_fullsized":"2024-03-24T00:31:01.909721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:57:10.621965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909977+0000","last_change":"2024-03-24T00:31:00.059028+0000","last_active":"2024-03-24T00:31:01.909977+0000","last_peered":"2024-03-24T00:31:01.909977+0000","last_clean":"2024-03-24T00:31:01.909977+0000","last_became_active":"2024-03-24T00:31:00.058762+0000","last_became_peered":"2024-03-24T00:31:00.058762+0000","last_unstale":"2024-03-24T00:31:01.909977+0000","last_undegraded":"2024-03-24T00:31:01.909977+0000","last_fullsized":"2024-03-24T00:31:01.909977+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060333+0000","last_change":"2024-03-24T00:31:00.054838+0000","last_active":"2024-03-24T00:31:01.060333+0000","last_peered":"2024-03-24T00:31:01.060333+0000","last_clean":"2024-03-24T00:31:01.060333+0000","last_became_active":"2024-03-24T00:31:00.054249+0000","last_became_peered":"2024-03-24T00:31:00.054249+0000","last_unstale":"2024-03-24T00:31:01.060333+0000","last_undegraded":"2024-03-24T00:31:01.060333+0000","last_fullsized":"2024-03-24T00:31:01.060333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047846+0000","last_change":"2024-03-24T00:31:00.037600+0000","last_active":"2024-03-24T00:31:01.047846+0000","last_peered":"2024-03-24T00:31:01.047846+0000","last_clean":"2024-03-24T00:31:01.047846+0000","last_became_active":"2024-03-24T00:31:00.037327+0000","last_became_peered":"2024-03-24T00:31:00.037327+0000","last_unstale":"2024-03-24T00:31:01.047846+0000","last_undegraded":"2024-03-24T00:31:01.047846+0000","last_fullsized":"2024-03-24T00:31:01.047846+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909766+0000","last_change":"2024-03-24T00:31:00.061415+0000","last_active":"2024-03-24T00:31:01.909766+0000","last_peered":"2024-03-24T00:31:01.909766+0000","last_clean":"2024-03-24T00:31:01.909766+0000","last_became_active":"2024-03-24T00:31:00.061147+0000","last_became_peered":"2024-03-24T00:31:00.061147+0000","last_unstale":"2024-03-24T00:31:01.909766+0000","last_undegraded":"2024-03-24T00:31:01.909766+0000","last_fullsized":"2024-03-24T00:31:01.909766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:27.902630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909601+0000","last_change":"2024-03-24T00:31:00.049400+0000","last_active":"2024-03-24T00:31:01.909601+0000","last_peered":"2024-03-24T00:31:01.909601+0000","last_clean":"2024-03-24T00:31:01.909601+0000","last_became_active":"2024-03-24T00:31:00.048885+0000","last_became_peered":"2024-03-24T00:31:00.048885+0000","last_unstale":"2024-03-24T00:31:01.909601+0000","last_undegraded":"2024-03-24T00:31:01.909601+0000","last_fullsized":"2024-03-24T00:31:01.909601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:22.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909905+0000","last_change":"2024-03-24T00:31:00.056183+0000","last_active":"2024-03-24T00:31:01.909905+0000","last_peered":"2024-03-24T00:31:01.909905+0000","last_clean":"2024-03-24T00:31:01.909905+0000","last_became_active":"2024-03-24T00:31:00.055346+0000","last_became_peered":"2024-03-24T00:31:00.055346+0000","last_unstale":"2024-03-24T00:31:01.909905+0000","last_undegraded":"2024-03-24T00:31:01.909905+0000","last_fullsized":"2024-03-24T00:31:01.909905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:08.370031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186114+0000","last_change":"2024-03-24T00:31:00.066718+0000","last_active":"2024-03-24T00:31:01.186114+0000","last_peered":"2024-03-24T00:31:01.186114+0000","last_clean":"2024-03-24T00:31:01.186114+0000","last_became_active":"2024-03-24T00:31:00.066330+0000","last_became_peered":"2024-03-24T00:31:00.066330+0000","last_unstale":"2024-03-24T00:31:01.186114+0000","last_undegraded":"2024-03-24T00:31:01.186114+0000","last_fullsized":"2024-03-24T00:31:01.186114+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:24.063587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059863+0000","last_change":"2024-03-24T00:31:00.045401+0000","last_active":"2024-03-24T00:31:01.059863+0000","last_peered":"2024-03-24T00:31:01.059863+0000","last_clean":"2024-03-24T00:31:01.059863+0000","last_became_active":"2024-03-24T00:31:00.044163+0000","last_became_peered":"2024-03-24T00:31:00.044163+0000","last_unstale":"2024-03-24T00:31:01.059863+0000","last_undegraded":"2024-03-24T00:31:01.059863+0000","last_fullsized":"2024-03-24T00:31:01.059863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:42:22.300219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045365+0000","last_change":"2024-03-24T00:31:00.039890+0000","last_active":"2024-03-24T00:31:01.045365+0000","last_peered":"2024-03-24T00:31:01.045365+0000","last_clean":"2024-03-24T00:31:01.045365+0000","last_became_active":"2024-03-24T00:31:00.038341+0000","last_became_peered":"2024-03-24T00:31:00.038341+0000","last_unstale":"2024-03-24T00:31:01.045365+0000","last_undegraded":"2024-03-24T00:31:01.045365+0000","last_fullsized":"2024-03-24T00:31:01.045365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:26.001997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910135+0000","last_change":"2024-03-24T00:30:59.052547+0000","last_active":"2024-03-24T00:31:01.910135+0000","last_peered":"2024-03-24T00:31:01.910135+0000","last_clean":"2024-03-24T00:31:01.910135+0000","last_became_active":"2024-03-24T00:30:59.052277+0000","last_became_peered":"2024-03-24T00:30:59.052277+0000","last_unstale":"2024-03-24T00:31:01.910135+0000","last_undegraded":"2024-03-24T00:31:01.910135+0000","last_fullsized":"2024-03-24T00:31:01.910135+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:28:37.149520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909635+0000","last_change":"2024-03-24T00:31:00.064484+0000","last_active":"2024-03-24T00:31:01.909635+0000","last_peered":"2024-03-24T00:31:01.909635+0000","last_clean":"2024-03-24T00:31:01.909635+0000","last_became_active":"2024-03-24T00:31:00.064211+0000","last_became_peered":"2024-03-24T00:31:00.064211+0000","last_unstale":"2024-03-24T00:31:01.909635+0000","last_undegraded":"2024-03-24T00:31:01.909635+0000","last_fullsized":"2024-03-24T00:31:01.909635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059739+0000","last_change":"2024-03-24T00:30:59.053767+0000","last_active":"2024-03-24T00:31:01.059739+0000","last_peered":"2024-03-24T00:31:01.059739+0000","last_clean":"2024-03-24T00:31:01.059739+0000","last_became_active":"2024-03-24T00:30:59.050782+0000","last_became_peered":"2024-03-24T00:30:59.050782+0000","last_unstale":"2024-03-24T00:31:01.059739+0000","last_undegraded":"2024-03-24T00:31:01.059739+0000","last_fullsized":"2024-03-24T00:31:01.059739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909297+0000","last_change":"2024-03-24T00:30:59.047982+0000","last_active":"2024-03-24T00:31:01.909297+0000","last_peered":"2024-03-24T00:31:01.909297+0000","last_clean":"2024-03-24T00:31:01.909297+0000","last_became_active":"2024-03-24T00:30:59.047680+0000","last_became_peered":"2024-03-24T00:30:59.047680+0000","last_unstale":"2024-03-24T00:31:01.909297+0000","last_undegraded":"2024-03-24T00:31:01.909297+0000","last_fullsized":"2024-03-24T00:31:01.909297+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:28.293820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910046+0000","last_change":"2024-03-24T00:31:00.044921+0000","last_active":"2024-03-24T00:31:01.910046+0000","last_peered":"2024-03-24T00:31:01.910046+0000","last_clean":"2024-03-24T00:31:01.910046+0000","last_became_active":"2024-03-24T00:31:00.044583+0000","last_became_peered":"2024-03-24T00:31:00.044583+0000","last_unstale":"2024-03-24T00:31:01.910046+0000","last_undegraded":"2024-03-24T00:31:01.910046+0000","last_fullsized":"2024-03-24T00:31:01.910046+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:57:10.621965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909657+0000","last_change":"2024-03-24T00:30:59.058420+0000","last_active":"2024-03-24T00:31:01.909657+0000","last_peered":"2024-03-24T00:31:01.909657+0000","last_clean":"2024-03-24T00:31:01.909657+0000","last_became_active":"2024-03-24T00:30:59.058004+0000","last_became_peered":"2024-03-24T00:30:59.058004+0000","last_unstale":"2024-03-24T00:31:01.909657+0000","last_undegraded":"2024-03-24T00:31:01.909657+0000","last_fullsized":"2024-03-24T00:31:01.909657+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047409+0000","last_change":"2024-03-24T00:31:00.043778+0000","last_active":"2024-03-24T00:31:01.047409+0000","last_peered":"2024-03-24T00:31:01.047409+0000","last_clean":"2024-03-24T00:31:01.047409+0000","last_became_active":"2024-03-24T00:31:00.043486+0000","last_became_peered":"2024-03-24T00:31:00.043486+0000","last_unstale":"2024-03-24T00:31:01.047409+0000","last_undegraded":"2024-03-24T00:31:01.047409+0000","last_fullsized":"2024-03-24T00:31:01.047409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060075+0000","last_change":"2024-03-24T00:30:59.057537+0000","last_active":"2024-03-24T00:31:01.060075+0000","last_peered":"2024-03-24T00:31:01.060075+0000","last_clean":"2024-03-24T00:31:01.060075+0000","last_became_active":"2024-03-24T00:30:59.057215+0000","last_became_peered":"2024-03-24T00:30:59.057215+0000","last_unstale":"2024-03-24T00:31:01.060075+0000","last_undegraded":"2024-03-24T00:31:01.060075+0000","last_fullsized":"2024-03-24T00:31:01.060075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:40.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910145+0000","last_change":"2024-03-24T00:31:00.057658+0000","last_active":"2024-03-24T00:31:01.910145+0000","last_peered":"2024-03-24T00:31:01.910145+0000","last_clean":"2024-03-24T00:31:01.910145+0000","last_became_active":"2024-03-24T00:31:00.055543+0000","last_became_peered":"2024-03-24T00:31:00.055543+0000","last_unstale":"2024-03-24T00:31:01.910145+0000","last_undegraded":"2024-03-24T00:31:01.910145+0000","last_fullsized":"2024-03-24T00:31:01.910145+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:49.592126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910049+0000","last_change":"2024-03-24T00:30:59.058992+0000","last_active":"2024-03-24T00:31:01.910049+0000","last_peered":"2024-03-24T00:31:01.910049+0000","last_clean":"2024-03-24T00:31:01.910049+0000","last_became_active":"2024-03-24T00:30:59.058546+0000","last_became_peered":"2024-03-24T00:30:59.058546+0000","last_unstale":"2024-03-24T00:31:01.910049+0000","last_undegraded":"2024-03-24T00:31:01.910049+0000","last_fullsized":"2024-03-24T00:31:01.910049+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:20.001460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909878+0000","last_change":"2024-03-24T00:31:00.057510+0000","last_active":"2024-03-24T00:31:01.909878+0000","last_peered":"2024-03-24T00:31:01.909878+0000","last_clean":"2024-03-24T00:31:01.909878+0000","last_became_active":"2024-03-24T00:31:00.057227+0000","last_became_peered":"2024-03-24T00:31:00.057227+0000","last_unstale":"2024-03-24T00:31:01.909878+0000","last_undegraded":"2024-03-24T00:31:01.909878+0000","last_fullsized":"2024-03-24T00:31:01.909878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047997+0000","last_change":"2024-03-24T00:30:59.055938+0000","last_active":"2024-03-24T00:31:01.047997+0000","last_peered":"2024-03-24T00:31:01.047997+0000","last_clean":"2024-03-24T00:31:01.047997+0000","last_became_active":"2024-03-24T00:30:59.055734+0000","last_became_peered":"2024-03-24T00:30:59.055734+0000","last_unstale":"2024-03-24T00:31:01.047997+0000","last_undegraded":"2024-03-24T00:31:01.047997+0000","last_fullsized":"2024-03-24T00:31:01.047997+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:39.280192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045606+0000","last_change":"2024-03-24T00:31:00.053717+0000","last_active":"2024-03-24T00:31:01.045606+0000","last_peered":"2024-03-24T00:31:01.045606+0000","last_clean":"2024-03-24T00:31:01.045606+0000","last_became_active":"2024-03-24T00:31:00.053312+0000","last_became_peered":"2024-03-24T00:31:00.053312+0000","last_unstale":"2024-03-24T00:31:01.045606+0000","last_undegraded":"2024-03-24T00:31:01.045606+0000","last_fullsized":"2024-03-24T00:31:01.045606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:40.286678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060097+0000","last_change":"2024-03-24T00:30:59.050368+0000","last_active":"2024-03-24T00:31:01.060097+0000","last_peered":"2024-03-24T00:31:01.060097+0000","last_clean":"2024-03-24T00:31:01.060097+0000","last_became_active":"2024-03-24T00:30:59.050062+0000","last_became_peered":"2024-03-24T00:30:59.050062+0000","last_unstale":"2024-03-24T00:31:01.060097+0000","last_undegraded":"2024-03-24T00:31:01.060097+0000","last_fullsized":"2024-03-24T00:31:01.060097+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046667+0000","last_change":"2024-03-24T00:31:00.041643+0000","last_active":"2024-03-24T00:31:01.046667+0000","last_peered":"2024-03-24T00:31:01.046667+0000","last_clean":"2024-03-24T00:31:01.046667+0000","last_became_active":"2024-03-24T00:31:00.041387+0000","last_became_peered":"2024-03-24T00:31:00.041387+0000","last_unstale":"2024-03-24T00:31:01.046667+0000","last_undegraded":"2024-03-24T00:31:01.046667+0000","last_fullsized":"2024-03-24T00:31:01.046667+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:34:52.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909659+0000","last_change":"2024-03-24T00:30:59.054246+0000","last_active":"2024-03-24T00:31:01.909659+0000","last_peered":"2024-03-24T00:31:01.909659+0000","last_clean":"2024-03-24T00:31:01.909659+0000","last_became_active":"2024-03-24T00:30:59.053818+0000","last_became_peered":"2024-03-24T00:30:59.053818+0000","last_unstale":"2024-03-24T00:31:01.909659+0000","last_undegraded":"2024-03-24T00:31:01.909659+0000","last_fullsized":"2024-03-24T00:31:01.909659+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:28.293820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909855+0000","last_change":"2024-03-24T00:31:00.054481+0000","last_active":"2024-03-24T00:31:01.909855+0000","last_peered":"2024-03-24T00:31:01.909855+0000","last_clean":"2024-03-24T00:31:01.909855+0000","last_became_active":"2024-03-24T00:31:00.054255+0000","last_became_peered":"2024-03-24T00:31:00.054255+0000","last_unstale":"2024-03-24T00:31:01.909855+0000","last_undegraded":"2024-03-24T00:31:01.909855+0000","last_fullsized":"2024-03-24T00:31:01.909855+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910033+0000","last_change":"2024-03-24T00:30:59.059529+0000","last_active":"2024-03-24T00:31:01.910033+0000","last_peered":"2024-03-24T00:31:01.910033+0000","last_clean":"2024-03-24T00:31:01.910033+0000","last_became_active":"2024-03-24T00:30:59.058947+0000","last_became_peered":"2024-03-24T00:30:59.058947+0000","last_unstale":"2024-03-24T00:31:01.910033+0000","last_undegraded":"2024-03-24T00:31:01.910033+0000","last_fullsized":"2024-03-24T00:31:01.910033+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:39.280192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046876+0000","last_change":"2024-03-24T00:31:00.061372+0000","last_active":"2024-03-24T00:31:01.046876+0000","last_peered":"2024-03-24T00:31:01.046876+0000","last_clean":"2024-03-24T00:31:01.046876+0000","last_became_active":"2024-03-24T00:31:00.061094+0000","last_became_peered":"2024-03-24T00:31:00.061094+0000","last_unstale":"2024-03-24T00:31:01.046876+0000","last_undegraded":"2024-03-24T00:31:01.046876+0000","last_fullsized":"2024-03-24T00:31:01.046876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:38.650664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059711+0000","last_change":"2024-03-24T00:30:59.046862+0000","last_active":"2024-03-24T00:31:01.059711+0000","last_peered":"2024-03-24T00:31:01.059711+0000","last_clean":"2024-03-24T00:31:01.059711+0000","last_became_active":"2024-03-24T00:30:59.046575+0000","last_became_peered":"2024-03-24T00:30:59.046575+0000","last_unstale":"2024-03-24T00:31:01.059711+0000","last_undegraded":"2024-03-24T00:31:01.059711+0000","last_fullsized":"2024-03-24T00:31:01.059711+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:52:39.280192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059910+0000","last_change":"2024-03-24T00:31:00.044599+0000","last_active":"2024-03-24T00:31:01.059910+0000","last_peered":"2024-03-24T00:31:01.059910+0000","last_clean":"2024-03-24T00:31:01.059910+0000","last_became_active":"2024-03-24T00:31:00.043498+0000","last_became_peered":"2024-03-24T00:31:00.043498+0000","last_unstale":"2024-03-24T00:31:01.059910+0000","last_undegraded":"2024-03-24T00:31:01.059910+0000","last_fullsized":"2024-03-24T00:31:01.059910+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:52:08.184546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.185968+0000","last_change":"2024-03-24T00:30:59.039042+0000","last_active":"2024-03-24T00:31:01.185968+0000","last_peered":"2024-03-24T00:31:01.185968+0000","last_clean":"2024-03-24T00:31:01.185968+0000","last_became_active":"2024-03-24T00:30:59.038744+0000","last_became_peered":"2024-03-24T00:30:59.038744+0000","last_unstale":"2024-03-24T00:31:01.185968+0000","last_undegraded":"2024-03-24T00:31:01.185968+0000","last_fullsized":"2024-03-24T00:31:01.185968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:20.001460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045531+0000","last_change":"2024-03-24T00:31:00.047666+0000","last_active":"2024-03-24T00:31:01.045531+0000","last_peered":"2024-03-24T00:31:01.045531+0000","last_clean":"2024-03-24T00:31:01.045531+0000","last_became_active":"2024-03-24T00:31:00.047379+0000","last_became_peered":"2024-03-24T00:31:00.047379+0000","last_unstale":"2024-03-24T00:31:01.045531+0000","last_undegraded":"2024-03-24T00:31:01.045531+0000","last_fullsized":"2024-03-24T00:31:01.045531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:08.370031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045647+0000","last_change":"2024-03-24T00:30:59.043700+0000","last_active":"2024-03-24T00:31:01.045647+0000","last_peered":"2024-03-24T00:31:01.045647+0000","last_clean":"2024-03-24T00:31:01.045647+0000","last_became_active":"2024-03-24T00:30:59.043305+0000","last_became_peered":"2024-03-24T00:30:59.043305+0000","last_unstale":"2024-03-24T00:31:01.045647+0000","last_undegraded":"2024-03-24T00:31:01.045647+0000","last_fullsized":"2024-03-24T00:31:01.045647+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:16.392363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046650+0000","last_change":"2024-03-24T00:31:00.048570+0000","last_active":"2024-03-24T00:31:01.046650+0000","last_peered":"2024-03-24T00:31:01.046650+0000","last_clean":"2024-03-24T00:31:01.046650+0000","last_became_active":"2024-03-24T00:31:00.048162+0000","last_became_peered":"2024-03-24T00:31:00.048162+0000","last_unstale":"2024-03-24T00:31:01.046650+0000","last_undegraded":"2024-03-24T00:31:01.046650+0000","last_fullsized":"2024-03-24T00:31:01.046650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:17.380183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186240+0000","last_change":"2024-03-24T00:30:59.041080+0000","last_active":"2024-03-24T00:31:01.186240+0000","last_peered":"2024-03-24T00:31:01.186240+0000","last_clean":"2024-03-24T00:31:01.186240+0000","last_became_active":"2024-03-24T00:30:59.040599+0000","last_became_peered":"2024-03-24T00:30:59.040599+0000","last_unstale":"2024-03-24T00:31:01.186240+0000","last_undegraded":"2024-03-24T00:31:01.186240+0000","last_fullsized":"2024-03-24T00:31:01.186240+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047022+0000","last_change":"2024-03-24T00:31:00.060654+0000","last_active":"2024-03-24T00:31:01.047022+0000","last_peered":"2024-03-24T00:31:01.047022+0000","last_clean":"2024-03-24T00:31:01.047022+0000","last_became_active":"2024-03-24T00:31:00.060301+0000","last_became_peered":"2024-03-24T00:31:00.060301+0000","last_unstale":"2024-03-24T00:31:01.047022+0000","last_undegraded":"2024-03-24T00:31:01.047022+0000","last_fullsized":"2024-03-24T00:31:01.047022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:41.152930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047876+0000","last_change":"2024-03-24T00:30:59.045891+0000","last_active":"2024-03-24T00:31:01.047876+0000","last_peered":"2024-03-24T00:31:01.047876+0000","last_clean":"2024-03-24T00:31:01.047876+0000","last_became_active":"2024-03-24T00:30:59.044107+0000","last_became_peered":"2024-03-24T00:30:59.044107+0000","last_unstale":"2024-03-24T00:31:01.047876+0000","last_undegraded":"2024-03-24T00:31:01.047876+0000","last_fullsized":"2024-03-24T00:31:01.047876+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:16.613800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910353+0000","last_change":"2024-03-24T00:31:00.056640+0000","last_active":"2024-03-24T00:31:01.910353+0000","last_peered":"2024-03-24T00:31:01.910353+0000","last_clean":"2024-03-24T00:31:01.910353+0000","last_became_active":"2024-03-24T00:31:00.055891+0000","last_became_peered":"2024-03-24T00:31:00.055891+0000","last_unstale":"2024-03-24T00:31:01.910353+0000","last_undegraded":"2024-03-24T00:31:01.910353+0000","last_fullsized":"2024-03-24T00:31:01.910353+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:27:17.791692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186119+0000","last_change":"2024-03-24T00:30:59.058994+0000","last_active":"2024-03-24T00:31:01.186119+0000","last_peered":"2024-03-24T00:31:01.186119+0000","last_clean":"2024-03-24T00:31:01.186119+0000","last_became_active":"2024-03-24T00:30:59.058639+0000","last_became_peered":"2024-03-24T00:30:59.058639+0000","last_unstale":"2024-03-24T00:31:01.186119+0000","last_undegraded":"2024-03-24T00:31:01.186119+0000","last_fullsized":"2024-03-24T00:31:01.186119+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046886+0000","last_change":"2024-03-24T00:31:00.059336+0000","last_active":"2024-03-24T00:31:01.046886+0000","last_peered":"2024-03-24T00:31:01.046886+0000","last_clean":"2024-03-24T00:31:01.046886+0000","last_became_active":"2024-03-24T00:31:00.059071+0000","last_became_peered":"2024-03-24T00:31:00.059071+0000","last_unstale":"2024-03-24T00:31:01.046886+0000","last_undegraded":"2024-03-24T00:31:01.046886+0000","last_fullsized":"2024-03-24T00:31:01.046886+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:10.867603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046267+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:31:01.046267+0000","last_peered":"2024-03-24T00:31:01.046267+0000","last_clean":"2024-03-24T00:31:01.046267+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:31:01.046267+0000","last_undegraded":"2024-03-24T00:31:01.046267+0000","last_fullsized":"2024-03-24T00:31:01.046267+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909596+0000","last_change":"2024-03-24T00:30:59.059377+0000","last_active":"2024-03-24T00:31:01.909596+0000","last_peered":"2024-03-24T00:31:01.909596+0000","last_clean":"2024-03-24T00:31:01.909596+0000","last_became_active":"2024-03-24T00:30:59.058424+0000","last_became_peered":"2024-03-24T00:30:59.058424+0000","last_unstale":"2024-03-24T00:31:01.909596+0000","last_undegraded":"2024-03-24T00:31:01.909596+0000","last_fullsized":"2024-03-24T00:31:01.909596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909334+0000","last_change":"2024-03-24T00:31:00.058762+0000","last_active":"2024-03-24T00:31:01.909334+0000","last_peered":"2024-03-24T00:31:01.909334+0000","last_clean":"2024-03-24T00:31:01.909334+0000","last_became_active":"2024-03-24T00:31:00.058509+0000","last_became_peered":"2024-03-24T00:31:00.058509+0000","last_unstale":"2024-03-24T00:31:01.909334+0000","last_undegraded":"2024-03-24T00:31:01.909334+0000","last_fullsized":"2024-03-24T00:31:01.909334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046137+0000","last_change":"2024-03-24T00:30:59.050656+0000","last_active":"2024-03-24T00:31:01.046137+0000","last_peered":"2024-03-24T00:31:01.046137+0000","last_clean":"2024-03-24T00:31:01.046137+0000","last_became_active":"2024-03-24T00:30:59.050364+0000","last_became_peered":"2024-03-24T00:30:59.050364+0000","last_unstale":"2024-03-24T00:31:01.046137+0000","last_undegraded":"2024-03-24T00:31:01.046137+0000","last_fullsized":"2024-03-24T00:31:01.046137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:23.150413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909215+0000","last_change":"2024-03-24T00:31:00.059875+0000","last_active":"2024-03-24T00:31:01.909215+0000","last_peered":"2024-03-24T00:31:01.909215+0000","last_clean":"2024-03-24T00:31:01.909215+0000","last_became_active":"2024-03-24T00:31:00.059618+0000","last_became_peered":"2024-03-24T00:31:00.059618+0000","last_unstale":"2024-03-24T00:31:01.909215+0000","last_undegraded":"2024-03-24T00:31:01.909215+0000","last_fullsized":"2024-03-24T00:31:01.909215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:34:42.663552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.186071+0000","last_change":"2024-03-24T00:30:59.040845+0000","last_active":"2024-03-24T00:31:01.186071+0000","last_peered":"2024-03-24T00:31:01.186071+0000","last_clean":"2024-03-24T00:31:01.186071+0000","last_became_active":"2024-03-24T00:30:59.040519+0000","last_became_peered":"2024-03-24T00:30:59.040519+0000","last_unstale":"2024-03-24T00:31:01.186071+0000","last_undegraded":"2024-03-24T00:31:01.186071+0000","last_fullsized":"2024-03-24T00:31:01.186071+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:28.293820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047374+0000","last_change":"2024-03-24T00:31:00.041015+0000","last_active":"2024-03-24T00:31:01.047374+0000","last_peered":"2024-03-24T00:31:01.047374+0000","last_clean":"2024-03-24T00:31:01.047374+0000","last_became_active":"2024-03-24T00:31:00.040568+0000","last_became_peered":"2024-03-24T00:31:00.040568+0000","last_unstale":"2024-03-24T00:31:01.047374+0000","last_undegraded":"2024-03-24T00:31:01.047374+0000","last_fullsized":"2024-03-24T00:31:01.047374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910220+0000","last_change":"2024-03-24T00:30:59.061747+0000","last_active":"2024-03-24T00:31:01.910220+0000","last_peered":"2024-03-24T00:31:01.910220+0000","last_clean":"2024-03-24T00:31:01.910220+0000","last_became_active":"2024-03-24T00:30:59.059297+0000","last_became_peered":"2024-03-24T00:30:59.059297+0000","last_unstale":"2024-03-24T00:31:01.910220+0000","last_undegraded":"2024-03-24T00:31:01.910220+0000","last_fullsized":"2024-03-24T00:31:01.910220+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:37.099548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045947+0000","last_change":"2024-03-24T00:31:00.050644+0000","last_active":"2024-03-24T00:31:01.045947+0000","last_peered":"2024-03-24T00:31:01.045947+0000","last_clean":"2024-03-24T00:31:01.045947+0000","last_became_active":"2024-03-24T00:31:00.050376+0000","last_became_peered":"2024-03-24T00:31:00.050376+0000","last_unstale":"2024-03-24T00:31:01.045947+0000","last_undegraded":"2024-03-24T00:31:01.045947+0000","last_fullsized":"2024-03-24T00:31:01.045947+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:17.620286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047738+0000","last_change":"2024-03-24T00:30:59.045279+0000","last_active":"2024-03-24T00:31:01.047738+0000","last_peered":"2024-03-24T00:31:01.047738+0000","last_clean":"2024-03-24T00:31:01.047738+0000","last_became_active":"2024-03-24T00:30:59.044864+0000","last_became_peered":"2024-03-24T00:30:59.044864+0000","last_unstale":"2024-03-24T00:31:01.047738+0000","last_undegraded":"2024-03-24T00:31:01.047738+0000","last_fullsized":"2024-03-24T00:31:01.047738+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:37.099548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910242+0000","last_change":"2024-03-24T00:31:00.056075+0000","last_active":"2024-03-24T00:31:01.910242+0000","last_peered":"2024-03-24T00:31:01.910242+0000","last_clean":"2024-03-24T00:31:01.910242+0000","last_became_active":"2024-03-24T00:31:00.055694+0000","last_became_peered":"2024-03-24T00:31:00.055694+0000","last_unstale":"2024-03-24T00:31:01.910242+0000","last_undegraded":"2024-03-24T00:31:01.910242+0000","last_fullsized":"2024-03-24T00:31:01.910242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:38.650664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047000+0000","last_change":"2024-03-24T00:30:59.048260+0000","last_active":"2024-03-24T00:31:01.047000+0000","last_peered":"2024-03-24T00:31:01.047000+0000","last_clean":"2024-03-24T00:31:01.047000+0000","last_became_active":"2024-03-24T00:30:59.047943+0000","last_became_peered":"2024-03-24T00:30:59.047943+0000","last_unstale":"2024-03-24T00:31:01.047000+0000","last_undegraded":"2024-03-24T00:31:01.047000+0000","last_fullsized":"2024-03-24T00:31:01.047000+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:13.589302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047659+0000","last_change":"2024-03-24T00:31:00.041416+0000","last_active":"2024-03-24T00:31:01.047659+0000","last_peered":"2024-03-24T00:31:01.047659+0000","last_clean":"2024-03-24T00:31:01.047659+0000","last_became_active":"2024-03-24T00:31:00.041096+0000","last_became_peered":"2024-03-24T00:31:00.041096+0000","last_unstale":"2024-03-24T00:31:01.047659+0000","last_undegraded":"2024-03-24T00:31:01.047659+0000","last_fullsized":"2024-03-24T00:31:01.047659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047950+0000","last_change":"2024-03-24T00:30:59.046901+0000","last_active":"2024-03-24T00:31:01.047950+0000","last_peered":"2024-03-24T00:31:01.047950+0000","last_clean":"2024-03-24T00:31:01.047950+0000","last_became_active":"2024-03-24T00:30:59.046619+0000","last_became_peered":"2024-03-24T00:30:59.046619+0000","last_unstale":"2024-03-24T00:31:01.047950+0000","last_undegraded":"2024-03-24T00:31:01.047950+0000","last_fullsized":"2024-03-24T00:31:01.047950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:40.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909976+0000","last_change":"2024-03-24T00:31:00.047744+0000","last_active":"2024-03-24T00:31:01.909976+0000","last_peered":"2024-03-24T00:31:01.909976+0000","last_clean":"2024-03-24T00:31:01.909976+0000","last_became_active":"2024-03-24T00:31:00.047453+0000","last_became_peered":"2024-03-24T00:31:00.047453+0000","last_unstale":"2024-03-24T00:31:01.909976+0000","last_undegraded":"2024-03-24T00:31:01.909976+0000","last_fullsized":"2024-03-24T00:31:01.909976+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:14.537604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.060070+0000","last_change":"2024-03-24T00:30:59.047379+0000","last_active":"2024-03-24T00:31:01.060070+0000","last_peered":"2024-03-24T00:31:01.060070+0000","last_clean":"2024-03-24T00:31:01.060070+0000","last_became_active":"2024-03-24T00:30:59.045528+0000","last_became_peered":"2024-03-24T00:30:59.045528+0000","last_unstale":"2024-03-24T00:31:01.060070+0000","last_undegraded":"2024-03-24T00:31:01.060070+0000","last_fullsized":"2024-03-24T00:31:01.060070+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:23.057101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909624+0000","last_change":"2024-03-24T00:31:00.050676+0000","last_active":"2024-03-24T00:31:01.909624+0000","last_peered":"2024-03-24T00:31:01.909624+0000","last_clean":"2024-03-24T00:31:01.909624+0000","last_became_active":"2024-03-24T00:31:00.050421+0000","last_became_peered":"2024-03-24T00:31:00.050421+0000","last_unstale":"2024-03-24T00:31:01.909624+0000","last_undegraded":"2024-03-24T00:31:01.909624+0000","last_fullsized":"2024-03-24T00:31:01.909624+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.045080+0000","last_change":"2024-03-24T00:30:59.047814+0000","last_active":"2024-03-24T00:31:01.045080+0000","last_peered":"2024-03-24T00:31:01.045080+0000","last_clean":"2024-03-24T00:31:01.045080+0000","last_became_active":"2024-03-24T00:30:59.047497+0000","last_became_peered":"2024-03-24T00:30:59.047497+0000","last_unstale":"2024-03-24T00:31:01.045080+0000","last_undegraded":"2024-03-24T00:31:01.045080+0000","last_fullsized":"2024-03-24T00:31:01.045080+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:18:37.099548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046279+0000","last_change":"2024-03-24T00:31:00.043642+0000","last_active":"2024-03-24T00:31:01.046279+0000","last_peered":"2024-03-24T00:31:01.046279+0000","last_clean":"2024-03-24T00:31:01.046279+0000","last_became_active":"2024-03-24T00:31:00.043334+0000","last_became_peered":"2024-03-24T00:31:00.043334+0000","last_unstale":"2024-03-24T00:31:01.046279+0000","last_undegraded":"2024-03-24T00:31:01.046279+0000","last_fullsized":"2024-03-24T00:31:01.046279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:41.254416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047463+0000","last_change":"2024-03-24T00:30:59.044547+0000","last_active":"2024-03-24T00:31:01.047463+0000","last_peered":"2024-03-24T00:31:01.047463+0000","last_clean":"2024-03-24T00:31:01.047463+0000","last_became_active":"2024-03-24T00:30:59.044240+0000","last_became_peered":"2024-03-24T00:30:59.044240+0000","last_unstale":"2024-03-24T00:31:01.047463+0000","last_undegraded":"2024-03-24T00:31:01.047463+0000","last_fullsized":"2024-03-24T00:31:01.047463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:21:24.995510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910154+0000","last_change":"2024-03-24T00:31:00.043313+0000","last_active":"2024-03-24T00:31:01.910154+0000","last_peered":"2024-03-24T00:31:01.910154+0000","last_clean":"2024-03-24T00:31:01.910154+0000","last_became_active":"2024-03-24T00:31:00.043030+0000","last_became_peered":"2024-03-24T00:31:00.043030+0000","last_unstale":"2024-03-24T00:31:01.910154+0000","last_undegraded":"2024-03-24T00:31:01.910154+0000","last_fullsized":"2024-03-24T00:31:01.910154+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:32.847580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.047581+0000","last_change":"2024-03-24T00:30:59.045395+0000","last_active":"2024-03-24T00:31:01.047581+0000","last_peered":"2024-03-24T00:31:01.047581+0000","last_clean":"2024-03-24T00:31:01.047581+0000","last_became_active":"2024-03-24T00:30:59.044888+0000","last_became_peered":"2024-03-24T00:30:59.044888+0000","last_unstale":"2024-03-24T00:31:01.047581+0000","last_undegraded":"2024-03-24T00:31:01.047581+0000","last_fullsized":"2024-03-24T00:31:01.047581+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:36.080572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910336+0000","last_change":"2024-03-24T00:31:00.052157+0000","last_active":"2024-03-24T00:31:01.910336+0000","last_peered":"2024-03-24T00:31:01.910336+0000","last_clean":"2024-03-24T00:31:01.910336+0000","last_became_active":"2024-03-24T00:31:00.051147+0000","last_became_peered":"2024-03-24T00:31:00.051147+0000","last_unstale":"2024-03-24T00:31:01.910336+0000","last_undegraded":"2024-03-24T00:31:01.910336+0000","last_fullsized":"2024-03-24T00:31:01.910336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:05:08.370031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046873+0000","last_change":"2024-03-24T00:30:59.053443+0000","last_active":"2024-03-24T00:31:01.046873+0000","last_peered":"2024-03-24T00:31:01.046873+0000","last_clean":"2024-03-24T00:31:01.046873+0000","last_became_active":"2024-03-24T00:30:59.053111+0000","last_became_peered":"2024-03-24T00:30:59.053111+0000","last_unstale":"2024-03-24T00:31:01.046873+0000","last_undegraded":"2024-03-24T00:31:01.046873+0000","last_fullsized":"2024-03-24T00:31:01.046873+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:07:40.247930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.059908+0000","last_change":"2024-03-24T00:31:00.061892+0000","last_active":"2024-03-24T00:31:01.059908+0000","last_peered":"2024-03-24T00:31:01.059908+0000","last_clean":"2024-03-24T00:31:01.059908+0000","last_became_active":"2024-03-24T00:31:00.061567+0000","last_became_peered":"2024-03-24T00:31:00.061567+0000","last_unstale":"2024-03-24T00:31:01.059908+0000","last_undegraded":"2024-03-24T00:31:01.059908+0000","last_fullsized":"2024-03-24T00:31:01.059908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:32:32.915331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910099+0000","last_change":"2024-03-24T00:31:00.069008+0000","last_active":"2024-03-24T00:31:01.910099+0000","last_peered":"2024-03-24T00:31:01.910099+0000","last_clean":"2024-03-24T00:31:01.910099+0000","last_became_active":"2024-03-24T00:31:00.065217+0000","last_became_peered":"2024-03-24T00:31:00.065217+0000","last_unstale":"2024-03-24T00:31:01.910099+0000","last_undegraded":"2024-03-24T00:31:01.910099+0000","last_fullsized":"2024-03-24T00:31:01.910099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:34:52.483712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046539+0000","last_change":"2024-03-24T00:30:59.060493+0000","last_active":"2024-03-24T00:31:01.046539+0000","last_peered":"2024-03-24T00:31:01.046539+0000","last_clean":"2024-03-24T00:31:01.046539+0000","last_became_active":"2024-03-24T00:30:59.060208+0000","last_became_peered":"2024-03-24T00:30:59.060208+0000","last_unstale":"2024-03-24T00:31:01.046539+0000","last_undegraded":"2024-03-24T00:31:01.046539+0000","last_fullsized":"2024-03-24T00:31:01.046539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:16.613800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910297+0000","last_change":"2024-03-24T00:31:00.071223+0000","last_active":"2024-03-24T00:31:01.910297+0000","last_peered":"2024-03-24T00:31:01.910297+0000","last_clean":"2024-03-24T00:31:01.910297+0000","last_became_active":"2024-03-24T00:31:00.069882+0000","last_became_peered":"2024-03-24T00:31:00.069882+0000","last_unstale":"2024-03-24T00:31:01.910297+0000","last_undegraded":"2024-03-24T00:31:01.910297+0000","last_fullsized":"2024-03-24T00:31:01.910297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:57:09.487064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.046833+0000","last_change":"2024-03-24T00:30:59.052020+0000","last_active":"2024-03-24T00:31:01.046833+0000","last_peered":"2024-03-24T00:31:01.046833+0000","last_clean":"2024-03-24T00:31:01.046833+0000","last_became_active":"2024-03-24T00:30:59.049277+0000","last_became_peered":"2024-03-24T00:30:59.049277+0000","last_unstale":"2024-03-24T00:31:01.046833+0000","last_undegraded":"2024-03-24T00:31:01.046833+0000","last_fullsized":"2024-03-24T00:31:01.046833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:23.057101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909752+0000","last_change":"2024-03-24T00:31:00.072313+0000","last_active":"2024-03-24T00:31:01.909752+0000","last_peered":"2024-03-24T00:31:01.909752+0000","last_clean":"2024-03-24T00:31:01.909752+0000","last_became_active":"2024-03-24T00:31:00.071692+0000","last_became_peered":"2024-03-24T00:31:00.071692+0000","last_unstale":"2024-03-24T00:31:01.909752+0000","last_undegraded":"2024-03-24T00:31:01.909752+0000","last_fullsized":"2024-03-24T00:31:01.909752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:23:53.304428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.910030+0000","last_change":"2024-03-24T00:30:59.045778+0000","last_active":"2024-03-24T00:31:01.910030+0000","last_peered":"2024-03-24T00:31:01.910030+0000","last_clean":"2024-03-24T00:31:01.910030+0000","last_became_active":"2024-03-24T00:30:59.045515+0000","last_became_peered":"2024-03-24T00:30:59.045515+0000","last_unstale":"2024-03-24T00:31:01.910030+0000","last_undegraded":"2024-03-24T00:31:01.910030+0000","last_fullsized":"2024-03-24T00:31:01.910030+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:20.001460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909974+0000","last_change":"2024-03-24T00:31:00.047507+0000","last_active":"2024-03-24T00:31:01.909974+0000","last_peered":"2024-03-24T00:31:01.909974+0000","last_clean":"2024-03-24T00:31:01.909974+0000","last_became_active":"2024-03-24T00:31:00.047245+0000","last_became_peered":"2024-03-24T00:31:00.047245+0000","last_unstale":"2024-03-24T00:31:01.909974+0000","last_undegraded":"2024-03-24T00:31:01.909974+0000","last_fullsized":"2024-03-24T00:31:01.909974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:59.014979+0000","last_clean_scrub_stamp":"2024-03-24T00:30:59.014979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:47.419891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-24T00:31:01.909924+0000","last_change":"2024-03-24T00:30:59.053442+0000","last_active":"2024-03-24T00:31:01.909924+0000","last_peered":"2024-03-24T00:31:01.909924+0000","last_clean":"2024-03-24T00:31:01.909924+0000","last_became_active":"2024-03-24T00:30:59.053159+0000","last_became_peered":"2024-03-24T00:30:59.053159+0000","last_unstale":"2024-03-24T00:31:01.909924+0000","last_undegraded":"2024-03-24T00:31:01.909924+0000","last_fullsized":"2024-03-24T00:31:01.909924+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:58.008493+0000","last_clean_scrub_stamp":"2024-03-24T00:30:58.008493+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:28:37.149520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31964,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339876,"statfs":{"total":96636764160,"available":96604033024,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:31:12.313 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-24T00:31:12.313 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T00:31:12.313 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-24T00:31:12.313 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-24T00:31:12.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4746800640 1 Processor -- start 2024-03-24T00:31:12.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4746800640 1 -- start start 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4746800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f474015bd30 0x7f47400d03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4746800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f474015f8a0 0x7f474015fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4746800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47400cd830 con 0x7f47400d25c0 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4746800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4740160250 con 0x7f474015f8a0 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4746800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47400cf630 con 0x7f474015bd30 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4744a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f47400d25c0 0x7f47400ccfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57980/0 (socket says 172.21.15.40:57980) 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4744a00640 1 -- 172.21.15.40:0/461242741 learned_addr learned my addr 172.21.15.40:0/461242741 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4745e00640 1 --2- 172.21.15.40:0/461242741 >> v2:172.21.15.40:3300/0 conn(0x7f474015f8a0 0x7f474015fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4745400640 1 --2- 172.21.15.40:0/461242741 >> v2:172.21.15.53:3300/0 conn(0x7f474015bd30 0x7f47400d03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f473fe00640 1 -- 172.21.15.40:0/461242741 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783652175 0 0) 0x7f47400cd830 con 0x7f47400d25c0 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f473fe00640 1 -- 172.21.15.40:0/461242741 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4728003530 con 0x7f47400d25c0 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4745e00640 1 -- 172.21.15.40:0/461242741 >> v2:172.21.15.53:3300/0 conn(0x7f474015bd30 msgr2=0x7f47400d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4745e00640 1 --2- 172.21.15.40:0/461242741 >> v2:172.21.15.53:3300/0 conn(0x7f474015bd30 0x7f47400d03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4745e00640 1 -- 172.21.15.40:0/461242741 >> v1:172.21.15.40:6789/0 conn(0x7f47400d25c0 legacy=0x7f47400ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.490+0000 7f4745e00640 1 -- 172.21.15.40:0/461242741 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47400cf910 con 0x7f474015f8a0 2024-03-24T00:31:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.494+0000 7f4745e00640 1 --2- 172.21.15.40:0/461242741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f474015f8a0 0x7f474015fd10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4740072130 tx=0x7f473801c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbef677c552c740c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.494+0000 7f473fe00640 1 -- 172.21.15.40:0/461242741 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47380b2070 con 0x7f474015f8a0 2024-03-24T00:31:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.494+0000 7f473fe00640 1 -- 172.21.15.40:0/461242741 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47380a5950 con 0x7f474015f8a0 2024-03-24T00:31:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.494+0000 7f473fe00640 1 -- 172.21.15.40:0/461242741 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47380a5d20 con 0x7f474015f8a0 2024-03-24T00:31:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.494+0000 7f4746800640 1 -- 172.21.15.40:0/461242741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f474015f8a0 msgr2=0x7f474015fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.494+0000 7f4746800640 1 --2- 172.21.15.40:0/461242741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f474015f8a0 0x7f474015fd10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4740072130 tx=0x7f473801c870 comp rx=0 tx=0).stop 2024-03-24T00:31:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.494+0000 7f4746800640 1 -- 172.21.15.40:0/461242741 shutdown_connections 2024-03-24T00:31:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.494+0000 7f4746800640 1 --2- 172.21.15.40:0/461242741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f474015f8a0 0x7f474015fd10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.494+0000 7f4746800640 1 --2- 172.21.15.40:0/461242741 >> v2:172.21.15.53:3300/0 conn(0x7f474015bd30 0x7f47400d03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.494+0000 7f4746800640 1 -- 172.21.15.40:0/461242741 >> 172.21.15.40:0/461242741 conn(0x7f47400d19c0 msgr2=0x7f47400d1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:12.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4746800640 1 -- 172.21.15.40:0/461242741 shutdown_connections 2024-03-24T00:31:12.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4746800640 1 -- 172.21.15.40:0/461242741 wait complete. 2024-03-24T00:31:12.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4746800640 1 Processor -- start 2024-03-24T00:31:12.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4746800640 1 -- start start 2024-03-24T00:31:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4746800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47400d25c0 0x7f4740157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4746800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f474015bd30 0x7f47401f6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4746800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f474015f8a0 0x7f47401f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4746800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474006f3f0 con 0x7f47400d25c0 2024-03-24T00:31:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4746800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474015c110 con 0x7f474015bd30 2024-03-24T00:31:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4746800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474006f570 con 0x7f474015f8a0 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4745400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47400d25c0 0x7f4740157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4745400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47400d25c0 0x7f4740157610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47278/0 (socket says 172.21.15.40:47278) 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4745400640 1 -- 172.21.15.40:0/1265609678 learned_addr learned my addr 172.21.15.40:0/1265609678 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4744a00640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f474015bd30 0x7f47401f6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4745e00640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f474015f8a0 0x7f47401f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4745400640 1 -- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f474015f8a0 msgr2=0x7f47401f6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4745400640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f474015f8a0 0x7f47401f6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4745400640 1 -- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f474015bd30 msgr2=0x7f47401f6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4745400640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f474015bd30 0x7f47401f6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4745400640 1 -- 172.21.15.40:0/1265609678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47401f7c00 con 0x7f47400d25c0 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4745400640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47400d25c0 0x7f4740157610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f473400e930 tx=0x7f473400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f473d600640 1 -- 172.21.15.40:0/1265609678 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734017070 con 0x7f47400d25c0 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f473d600640 1 -- 172.21.15.40:0/1265609678 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473400ca00 con 0x7f47400d25c0 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.498+0000 7f4746800640 1 -- 172.21.15.40:0/1265609678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47400cb140 con 0x7f47400d25c0 2024-03-24T00:31:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.502+0000 7f473d600640 1 -- 172.21.15.40:0/1265609678 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473400cdb0 con 0x7f47400d25c0 2024-03-24T00:31:12.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.502+0000 7f4746800640 1 -- 172.21.15.40:0/1265609678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47400cb9a0 con 0x7f47400d25c0 2024-03-24T00:31:12.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.502+0000 7f4746800640 1 -- 172.21.15.40:0/1265609678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474004f410 con 0x7f47400d25c0 2024-03-24T00:31:12.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.502+0000 7f473d600640 1 -- 172.21.15.40:0/1265609678 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f473401a070 con 0x7f47400d25c0 2024-03-24T00:31:12.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.502+0000 7f473d600640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4720067660 0x7f4720069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.502+0000 7f473d600640 1 -- 172.21.15.40:0/1265609678 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4734085860 con 0x7f47400d25c0 2024-03-24T00:31:12.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.502+0000 7f4744a00640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4720067660 0x7f4720069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.506+0000 7f4744a00640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4720067660 0x7f4720069b20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f472c007fd0 tx=0x7f472c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:12.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.506+0000 7f473d600640 1 -- 172.21.15.40:0/1265609678 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4734055f40 con 0x7f47400d25c0 2024-03-24T00:31:12.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 -- 172.21.15.40:0/1265609678 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f47400b7940 con 0x7f47400d25c0 2024-03-24T00:31:12.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f473d600640 1 -- 172.21.15.40:0/1265609678 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f4734059fe0 con 0x7f47400d25c0 2024-03-24T00:31:12.708 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:12.708 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T00:31:12.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 -- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4720067660 msgr2=0x7f4720069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4720067660 0x7f4720069b20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f472c007fd0 tx=0x7f472c009040 comp rx=0 tx=0).stop 2024-03-24T00:31:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 -- 172.21.15.40:0/1265609678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47400d25c0 msgr2=0x7f4740157610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47400d25c0 0x7f4740157610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f473400e930 tx=0x7f473400edf0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 -- 172.21.15.40:0/1265609678 shutdown_connections 2024-03-24T00:31:12.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4720067660 0x7f4720069b20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f474015f8a0 0x7f47401f6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f474015bd30 0x7f47401f6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 --2- 172.21.15.40:0/1265609678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47400d25c0 0x7f4740157610 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 -- 172.21.15.40:0/1265609678 >> 172.21.15.40:0/1265609678 conn(0x7f47400d19c0 msgr2=0x7f47400ae790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 -- 172.21.15.40:0/1265609678 shutdown_connections 2024-03-24T00:31:12.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.706+0000 7f4746800640 1 -- 172.21.15.40:0/1265609678 wait complete. 2024-03-24T00:31:12.729 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-24T00:31:12.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:12.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.866+0000 7fd54f000640 1 Processor -- start 2024-03-24T00:31:12.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.866+0000 7fd54f000640 1 -- start start 2024-03-24T00:31:12.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.866+0000 7fd54f000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5480cd470 0x7fd548168fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.866+0000 7fd54f000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5480b2870 con 0x7fd5480b2aa0 2024-03-24T00:31:12.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.866+0000 7fd54f000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5480b3190 con 0x7fd5480b2490 2024-03-24T00:31:12.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.866+0000 7fd54f000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54806b6d0 con 0x7fd5480cd470 2024-03-24T00:31:12.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.866+0000 7fd53f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd5480b2aa0 0x7fd5480d0540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57996/0 (socket says 172.21.15.40:57996) 2024-03-24T00:31:12.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.866+0000 7fd53f400640 1 -- 172.21.15.40:0/3194626264 learned_addr learned my addr 172.21.15.40:0/3194626264 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:12.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.870+0000 7fd54d000640 1 --2- 172.21.15.40:0/3194626264 >> v2:172.21.15.53:3300/0 conn(0x7fd5480cd470 0x7fd548168fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.870+0000 7fd547400640 1 -- 172.21.15.40:0/3194626264 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369864296 0 0) 0x7fd5480b2870 con 0x7fd5480b2aa0 2024-03-24T00:31:12.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.870+0000 7fd547400640 1 -- 172.21.15.40:0/3194626264 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd52c003530 con 0x7fd5480b2aa0 2024-03-24T00:31:12.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.870+0000 7fd547400640 1 -- 172.21.15.40:0/3194626264 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709412218 0 0) 0x7fd5480b3190 con 0x7fd5480b2490 2024-03-24T00:31:12.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.870+0000 7fd547400640 1 -- 172.21.15.40:0/3194626264 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5480b2870 con 0x7fd5480b2490 2024-03-24T00:31:12.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.870+0000 7fd547400640 1 -- 172.21.15.40:0/3194626264 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 98418326 0 0) 0x7fd52c003530 con 0x7fd5480b2aa0 2024-03-24T00:31:12.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.870+0000 7fd54d000640 1 -- 172.21.15.40:0/3194626264 >> v1:172.21.15.53:6789/0 conn(0x7fd5480b2490 legacy=0x7fd5480cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.870+0000 7fd54d000640 1 -- 172.21.15.40:0/3194626264 >> v1:172.21.15.40:6789/0 conn(0x7fd5480b2aa0 legacy=0x7fd5480d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.870+0000 7fd54d000640 1 -- 172.21.15.40:0/3194626264 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5480b3470 con 0x7fd5480cd470 2024-03-24T00:31:12.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.870+0000 7fd54d000640 1 --2- 172.21.15.40:0/3194626264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5480cd470 0x7fd548168fb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd540009d90 tx=0x7fd54001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66943eed030f28f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:12.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.870+0000 7fd547400640 1 -- 172.21.15.40:0/3194626264 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5400b2070 con 0x7fd5480cd470 2024-03-24T00:31:12.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd547400640 1 -- 172.21.15.40:0/3194626264 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5400a55e0 con 0x7fd5480cd470 2024-03-24T00:31:12.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd547400640 1 -- 172.21.15.40:0/3194626264 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530001680 con 0x7fd5480cd470 2024-03-24T00:31:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 -- 172.21.15.40:0/3194626264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5480cd470 msgr2=0x7fd548168fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 --2- 172.21.15.40:0/3194626264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5480cd470 0x7fd548168fb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd540009d90 tx=0x7fd54001c890 comp rx=0 tx=0).stop 2024-03-24T00:31:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 -- 172.21.15.40:0/3194626264 shutdown_connections 2024-03-24T00:31:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 --2- 172.21.15.40:0/3194626264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5480cd470 0x7fd548168fb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 -- 172.21.15.40:0/3194626264 >> 172.21.15.40:0/3194626264 conn(0x7fd5480d1d10 msgr2=0x7fd5480d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 -- 172.21.15.40:0/3194626264 shutdown_connections 2024-03-24T00:31:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 -- 172.21.15.40:0/3194626264 wait complete. 2024-03-24T00:31:12.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 Processor -- start 2024-03-24T00:31:12.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 -- start start 2024-03-24T00:31:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5480b2490 0x7fd54815c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5480b2aa0 0x7fd54815a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5480cd470 0x7fd54815fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54816c0a0 con 0x7fd5480b2aa0 2024-03-24T00:31:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54816bf20 con 0x7fd5480cd470 2024-03-24T00:31:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54f000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54816c220 con 0x7fd5480b2490 2024-03-24T00:31:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd53f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5480b2aa0 0x7fd54815a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd53f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5480b2aa0 0x7fd54815a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47296/0 (socket says 172.21.15.40:47296) 2024-03-24T00:31:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd53f400640 1 -- 172.21.15.40:0/2557173578 learned_addr learned my addr 172.21.15.40:0/2557173578 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd54d000640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5480cd470 0x7fd54815fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.874+0000 7fd547e00640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5480b2490 0x7fd54815c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd53f400640 1 -- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5480b2490 msgr2=0x7fd54815c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd53f400640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5480b2490 0x7fd54815c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd53f400640 1 -- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5480cd470 msgr2=0x7fd54815fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:12.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd53f400640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5480cd470 0x7fd54815fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:12.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd53f400640 1 -- 172.21.15.40:0/2557173578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd548160370 con 0x7fd5480b2aa0 2024-03-24T00:31:12.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd547e00640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5480b2490 0x7fd54815c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:12.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd53f400640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5480b2aa0 0x7fd54815a3b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd53000c760 tx=0x7fd53000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:12.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd544c00640 1 -- 172.21.15.40:0/2557173578 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530017070 con 0x7fd5480b2aa0 2024-03-24T00:31:12.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd544c00640 1 -- 172.21.15.40:0/2557173578 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd530005ae0 con 0x7fd5480b2aa0 2024-03-24T00:31:12.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd544c00640 1 -- 172.21.15.40:0/2557173578 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd530007070 con 0x7fd5480b2aa0 2024-03-24T00:31:12.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd54f000640 1 -- 172.21.15.40:0/2557173578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd548160650 con 0x7fd5480b2aa0 2024-03-24T00:31:12.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd54f000640 1 -- 172.21.15.40:0/2557173578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54815eef0 con 0x7fd5480b2aa0 2024-03-24T00:31:12.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd54f000640 1 -- 172.21.15.40:0/2557173578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd548068170 con 0x7fd5480b2aa0 2024-03-24T00:31:12.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.878+0000 7fd544c00640 1 -- 172.21.15.40:0/2557173578 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd53001a070 con 0x7fd5480b2aa0 2024-03-24T00:31:12.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.882+0000 7fd544c00640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd524067660 0x7fd524069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:12.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.882+0000 7fd544c00640 1 -- 172.21.15.40:0/2557173578 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd530084eb0 con 0x7fd5480b2aa0 2024-03-24T00:31:12.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.882+0000 7fd547e00640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd524067660 0x7fd524069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.882+0000 7fd547e00640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd524067660 0x7fd524069b20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd5380011f0 tx=0x7fd538009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:12.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:12.886+0000 7fd544c00640 1 -- 172.21.15.40:0/2557173578 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd530055a90 con 0x7fd5480b2aa0 2024-03-24T00:31:13.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 -- 172.21.15.40:0/2557173578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5480cdf70 con 0x7fd5480b2aa0 2024-03-24T00:31:13.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd544c00640 1 -- 172.21.15.40:0/2557173578 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7fd530055390 con 0x7fd5480b2aa0 2024-03-24T00:31:13.052 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:13.053 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 -- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd524067660 msgr2=0x7fd524069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd524067660 0x7fd524069b20 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd5380011f0 tx=0x7fd538009040 comp rx=0 tx=0).stop 2024-03-24T00:31:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 -- 172.21.15.40:0/2557173578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5480b2aa0 msgr2=0x7fd54815a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5480b2aa0 0x7fd54815a3b0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd53000c760 tx=0x7fd53000cc20 comp rx=0 tx=0).stop 2024-03-24T00:31:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 -- 172.21.15.40:0/2557173578 shutdown_connections 2024-03-24T00:31:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd524067660 0x7fd524069b20 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5480cd470 0x7fd54815fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5480b2aa0 0x7fd54815a3b0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 --2- 172.21.15.40:0/2557173578 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5480b2490 0x7fd54815c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 -- 172.21.15.40:0/2557173578 >> 172.21.15.40:0/2557173578 conn(0x7fd5480d1d10 msgr2=0x7fd54815e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 -- 172.21.15.40:0/2557173578 shutdown_connections 2024-03-24T00:31:13.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.050+0000 7fd54f000640 1 -- 172.21.15.40:0/2557173578 wait complete. 2024-03-24T00:31:13.073 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:13.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:13.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1407000640 1 Processor -- start 2024-03-24T00:31:13.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1407000640 1 -- start start 2024-03-24T00:31:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1407000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f140006cd70 0x7f140006b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1407000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14000cf640 0x7f14000d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1407000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14000ce360 0x7f14000ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1407000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140006ed90 con 0x7f14000cf640 2024-03-24T00:31:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1407000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f140006f070 con 0x7f140006cd70 2024-03-24T00:31:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1407000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140006f350 con 0x7f14000ce360 2024-03-24T00:31:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1405200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14000cf640 0x7f14000d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1405200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14000cf640 0x7f14000d04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47308/0 (socket says 172.21.15.40:47308) 2024-03-24T00:31:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1405200640 1 -- 172.21.15.40:0/3755253703 learned_addr learned my addr 172.21.15.40:0/3755253703 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1405c00640 1 --2- 172.21.15.40:0/3755253703 >> v2:172.21.15.53:3300/0 conn(0x7f140006cd70 0x7f140006b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1406600640 1 --2- 172.21.15.40:0/3755253703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14000ce360 0x7f14000ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1405200640 1 -- 172.21.15.40:0/3755253703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14000ce360 msgr2=0x7f14000ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1405200640 1 --2- 172.21.15.40:0/3755253703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14000ce360 0x7f14000ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1405200640 1 -- 172.21.15.40:0/3755253703 >> v2:172.21.15.53:3300/0 conn(0x7f140006cd70 msgr2=0x7f140006b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1405200640 1 --2- 172.21.15.40:0/3755253703 >> v2:172.21.15.53:3300/0 conn(0x7f140006cd70 0x7f140006b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1405200640 1 -- 172.21.15.40:0/3755253703 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140006f630 con 0x7f14000cf640 2024-03-24T00:31:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1405c00640 1 --2- 172.21.15.40:0/3755253703 >> v2:172.21.15.53:3300/0 conn(0x7f140006cd70 0x7f140006b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:13.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.214+0000 7f1406600640 1 --2- 172.21.15.40:0/3755253703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14000ce360 0x7f14000ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:13.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f1405200640 1 --2- 172.21.15.40:0/3755253703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14000cf640 0x7f14000d04a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f14000b4550 tx=0x7f13ec01cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44467576ca7ae462 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:13.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f13ffe00640 1 -- 172.21.15.40:0/3755253703 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec0ab5f0 con 0x7f14000cf640 2024-03-24T00:31:13.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f13ffe00640 1 -- 172.21.15.40:0/3755253703 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13ec0b0070 con 0x7f14000cf640 2024-03-24T00:31:13.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f13ffe00640 1 -- 172.21.15.40:0/3755253703 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13ec0ab9b0 con 0x7f14000cf640 2024-03-24T00:31:13.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f1407000640 1 -- 172.21.15.40:0/3755253703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14000cf640 msgr2=0x7f14000d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f1407000640 1 --2- 172.21.15.40:0/3755253703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14000cf640 0x7f14000d04a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f14000b4550 tx=0x7f13ec01cf80 comp rx=0 tx=0).stop 2024-03-24T00:31:13.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f1407000640 1 -- 172.21.15.40:0/3755253703 shutdown_connections 2024-03-24T00:31:13.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f1407000640 1 --2- 172.21.15.40:0/3755253703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14000ce360 0x7f14000ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f1407000640 1 --2- 172.21.15.40:0/3755253703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14000cf640 0x7f14000d04a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f1407000640 1 --2- 172.21.15.40:0/3755253703 >> v2:172.21.15.53:3300/0 conn(0x7f140006cd70 0x7f140006b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f1407000640 1 -- 172.21.15.40:0/3755253703 >> 172.21.15.40:0/3755253703 conn(0x7f14000d19e0 msgr2=0x7f14000d1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:13.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f1407000640 1 -- 172.21.15.40:0/3755253703 shutdown_connections 2024-03-24T00:31:13.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.218+0000 7f1407000640 1 -- 172.21.15.40:0/3755253703 wait complete. 2024-03-24T00:31:13.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1407000640 1 Processor -- start 2024-03-24T00:31:13.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1407000640 1 -- start start 2024-03-24T00:31:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1407000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f140006cd70 0x7f140015d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1407000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14000ce360 0x7f14001fa940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1407000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14000cf640 0x7f14001576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1407000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140015de60 con 0x7f14000cf640 2024-03-24T00:31:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1407000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f140006d3a0 con 0x7f14000ce360 2024-03-24T00:31:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1407000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140015dfe0 con 0x7f140006cd70 2024-03-24T00:31:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1406600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14000cf640 0x7f14001576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1406600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14000cf640 0x7f14001576e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47324/0 (socket says 172.21.15.40:47324) 2024-03-24T00:31:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1406600640 1 -- 172.21.15.40:0/577005229 learned_addr learned my addr 172.21.15.40:0/577005229 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1405200640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14000ce360 0x7f14001fa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1405c00640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f140006cd70 0x7f140015d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1406600640 1 -- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f140006cd70 msgr2=0x7f140015d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1406600640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f140006cd70 0x7f140015d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1406600640 1 -- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14000ce360 msgr2=0x7f14001fa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1406600640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14000ce360 0x7f14001fa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1406600640 1 -- 172.21.15.40:0/577005229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14001fae80 con 0x7f14000cf640 2024-03-24T00:31:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1406600640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14000cf640 0x7f14001576e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f13f800cef0 tx=0x7f13f800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f13fd600640 1 -- 172.21.15.40:0/577005229 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f801a070 con 0x7f14000cf640 2024-03-24T00:31:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f13fd600640 1 -- 172.21.15.40:0/577005229 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f800d860 con 0x7f14000cf640 2024-03-24T00:31:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1407000640 1 -- 172.21.15.40:0/577005229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14000b69c0 con 0x7f14000cf640 2024-03-24T00:31:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f13fd600640 1 -- 172.21.15.40:0/577005229 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f800dc50 con 0x7f14000cf640 2024-03-24T00:31:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1407000640 1 -- 172.21.15.40:0/577005229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14000b1930 con 0x7f14000cf640 2024-03-24T00:31:13.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.222+0000 7f1407000640 1 -- 172.21.15.40:0/577005229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14000b1440 con 0x7f14000cf640 2024-03-24T00:31:13.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.226+0000 7f13fd600640 1 -- 172.21.15.40:0/577005229 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f13f801d070 con 0x7f14000cf640 2024-03-24T00:31:13.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.226+0000 7f13fd600640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13e0067660 0x7f13e0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.226+0000 7f13fd600640 1 -- 172.21.15.40:0/577005229 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13f80860e0 con 0x7f14000cf640 2024-03-24T00:31:13.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.226+0000 7f1405c00640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13e0067660 0x7f13e0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.226+0000 7f1405c00640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13e0067660 0x7f13e0069b20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f14000b78a0 tx=0x7f13f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:13.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.230+0000 7f13fd600640 1 -- 172.21.15.40:0/577005229 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f13f8056730 con 0x7f14000cf640 2024-03-24T00:31:13.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.542+0000 7f1407000640 1 -- 172.21.15.40:0/577005229 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f140004f410 con 0x7f14000cf640 2024-03-24T00:31:13.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f13fd600640 1 -- 172.21.15.40:0/577005229 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f13f805a7d0 con 0x7f14000cf640 2024-03-24T00:31:13.549 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:13.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f1407000640 1 -- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13e0067660 msgr2=0x7f13e0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f1407000640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13e0067660 0x7f13e0069b20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f14000b78a0 tx=0x7f13f4008040 comp rx=0 tx=0).stop 2024-03-24T00:31:13.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f1407000640 1 -- 172.21.15.40:0/577005229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14000cf640 msgr2=0x7f14001576e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f1407000640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14000cf640 0x7f14001576e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f13f800cef0 tx=0x7f13f800a710 comp rx=0 tx=0).stop 2024-03-24T00:31:13.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f1407000640 1 -- 172.21.15.40:0/577005229 shutdown_connections 2024-03-24T00:31:13.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f1407000640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13e0067660 0x7f13e0069b20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f1407000640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14000cf640 0x7f14001576e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f1407000640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14000ce360 0x7f14001fa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f1407000640 1 --2- 172.21.15.40:0/577005229 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f140006cd70 0x7f140015d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f1407000640 1 -- 172.21.15.40:0/577005229 >> 172.21.15.40:0/577005229 conn(0x7f14000d19e0 msgr2=0x7f14000d0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:13.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f1407000640 1 -- 172.21.15.40:0/577005229 shutdown_connections 2024-03-24T00:31:13.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.546+0000 7f1407000640 1 -- 172.21.15.40:0/577005229 wait complete. 2024-03-24T00:31:13.574 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.714+0000 7f53fda00640 1 Processor -- start 2024-03-24T00:31:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.714+0000 7f53fda00640 1 -- start start 2024-03-24T00:31:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.714+0000 7f53fda00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f815b960 0x7f53f8070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53fda00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f815f970 con 0x7f53f80d16f0 2024-03-24T00:31:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53fda00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f815fc50 con 0x7f53f815bd40 2024-03-24T00:31:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53fda00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f8160580 con 0x7f53f815b960 2024-03-24T00:31:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f7400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f53f80d16f0 0x7f53f80cda00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58006/0 (socket says 172.21.15.40:58006) 2024-03-24T00:31:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f7400640 1 -- 172.21.15.40:0/2492181959 learned_addr learned my addr 172.21.15.40:0/2492181959 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6a00640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f815b960 0x7f53f8070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711668961 0 0) 0x7f53f815f970 con 0x7f53f80d16f0 2024-03-24T00:31:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53dc003530 con 0x7f53f80d16f0 2024-03-24T00:31:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451929773 0 0) 0x7f53f815fc50 con 0x7f53f815bd40 2024-03-24T00:31:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f815f970 con 0x7f53f815bd40 2024-03-24T00:31:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3816120844 0 0) 0x7f53dc003530 con 0x7f53f80d16f0 2024-03-24T00:31:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f815b960 msgr2=0x7f53f8070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f815b960 0x7f53f8070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 >> v1:172.21.15.53:6789/0 conn(0x7f53f815bd40 legacy=0x7f53f80d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f815ff30 con 0x7f53f80d16f0 2024-03-24T00:31:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f53f0001a00 con 0x7f53f80d16f0 2024-03-24T00:31:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6a00640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f815b960 0x7f53f8070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 >> v1:172.21.15.40:6789/0 conn(0x7f53f80d16f0 legacy=0x7f53f80cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53dc003b80 0x7f53dc003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc004810 0x7f53dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f815b960 0x7f53dc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f8160580 con 0x7f53dc003b80 2024-03-24T00:31:13.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc00ba00 con 0x7f53dc004810 2024-03-24T00:31:13.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc00bc00 con 0x7f53f815b960 2024-03-24T00:31:13.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6a00640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53dc003b80 0x7f53dc003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f7400640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f815b960 0x7f53dc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53ee000640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc004810 0x7f53dc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6a00640 1 -- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f815b960 msgr2=0x7f53dc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:13.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6a00640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f815b960 0x7f53dc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6a00640 1 -- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc004810 msgr2=0x7f53dc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:13.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6a00640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc004810 0x7f53dc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53f6a00640 1 -- 172.21.15.40:0/2492181959 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc003530 con 0x7f53dc003b80 2024-03-24T00:31:13.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.718+0000 7f53ee000640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc004810 0x7f53dc004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:13.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.722+0000 7f53f6a00640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53dc003b80 0x7f53dc003f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009c80 tx=0x7f53e801cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4901c53d061880 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:13.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.722+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e80b2070 con 0x7f53dc003b80 2024-03-24T00:31:13.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.722+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e80a5650 con 0x7f53dc003b80 2024-03-24T00:31:13.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.722+0000 7f53f6000640 1 -- 172.21.15.40:0/2492181959 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e8009880 con 0x7f53dc003b80 2024-03-24T00:31:13.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.722+0000 7f53fda00640 1 -- 172.21.15.40:0/2492181959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53dc003b80 msgr2=0x7f53dc003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.722+0000 7f53fda00640 1 --2- 172.21.15.40:0/2492181959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53dc003b80 0x7f53dc003f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009c80 tx=0x7f53e801cbb0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.722+0000 7f53f7400640 1 -- 172.21.15.40:0/2492181959 reap_dead start 2024-03-24T00:31:13.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.722+0000 7f53fda00640 1 -- 172.21.15.40:0/2492181959 shutdown_connections 2024-03-24T00:31:13.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.722+0000 7f53fda00640 1 -- 172.21.15.40:0/2492181959 >> 172.21.15.40:0/2492181959 conn(0x7f53f806ecf0 msgr2=0x7f53f806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:13.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.722+0000 7f53fda00640 1 -- 172.21.15.40:0/2492181959 shutdown_connections 2024-03-24T00:31:13.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.722+0000 7f53fda00640 1 -- 172.21.15.40:0/2492181959 wait complete. 2024-03-24T00:31:13.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.722+0000 7f53fda00640 1 Processor -- start 2024-03-24T00:31:13.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53fda00640 1 -- start start 2024-03-24T00:31:13.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53fda00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc003b80 0x7f53f80cb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53fda00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f806cc60 0x7f53f80c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53fda00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f8150460 0x7f53f81527e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53fda00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f80cecb0 con 0x7f53f806cc60 2024-03-24T00:31:13.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53fda00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f80ceb30 con 0x7f53dc003b80 2024-03-24T00:31:13.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53fda00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f8167b70 con 0x7f53f8150460 2024-03-24T00:31:13.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53ee000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f806cc60 0x7f53f80c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53ee000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f806cc60 0x7f53f80c9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47328/0 (socket says 172.21.15.40:47328) 2024-03-24T00:31:13.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53f6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc003b80 0x7f53f80cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53ee000640 1 -- 172.21.15.40:0/1213783871 learned_addr learned my addr 172.21.15.40:0/1213783871 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:13.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53f7400640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f8150460 0x7f53f81527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53ee000640 1 -- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f8150460 msgr2=0x7f53f81527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53ee000640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f8150460 0x7f53f81527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53ee000640 1 -- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc003b80 msgr2=0x7f53f80cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53ee000640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc003b80 0x7f53f80cb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53ee000640 1 -- 172.21.15.40:0/1213783871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f80cbac0 con 0x7f53f806cc60 2024-03-24T00:31:13.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53ee000640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f806cc60 0x7f53f80c9be0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f53e000e580 tx=0x7f53e000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:13.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53ef400640 1 -- 172.21.15.40:0/1213783871 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e00052e0 con 0x7f53f806cc60 2024-03-24T00:31:13.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53ef400640 1 -- 172.21.15.40:0/1213783871 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e0007020 con 0x7f53f806cc60 2024-03-24T00:31:13.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53fda00640 1 -- 172.21.15.40:0/1213783871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f80c8e20 con 0x7f53f806cc60 2024-03-24T00:31:13.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53ef400640 1 -- 172.21.15.40:0/1213783871 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e0005480 con 0x7f53f806cc60 2024-03-24T00:31:13.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.726+0000 7f53fda00640 1 -- 172.21.15.40:0/1213783871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f80c93e0 con 0x7f53f806cc60 2024-03-24T00:31:13.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.730+0000 7f53fda00640 1 -- 172.21.15.40:0/1213783871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f8068170 con 0x7f53f806cc60 2024-03-24T00:31:13.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.730+0000 7f53ef400640 1 -- 172.21.15.40:0/1213783871 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f53e001a020 con 0x7f53f806cc60 2024-03-24T00:31:13.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.730+0000 7f53ef400640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53d4067660 0x7f53d4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:13.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.730+0000 7f53ef400640 1 -- 172.21.15.40:0/1213783871 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53e0084fb0 con 0x7f53f806cc60 2024-03-24T00:31:13.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.730+0000 7f53f6a00640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53d4067660 0x7f53d4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:13.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.730+0000 7f53f6a00640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53d4067660 0x7f53d4069b20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f53e80af000 tx=0x7f53e80046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:13.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.734+0000 7f53ef400640 1 -- 172.21.15.40:0/1213783871 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f53e0055550 con 0x7f53f806cc60 2024-03-24T00:31:13.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 -- 172.21.15.40:0/1213783871 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f53f815acb0 con 0x7f53f806cc60 2024-03-24T00:31:13.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53ef400640 1 -- 172.21.15.40:0/1213783871 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f53e00595f0 con 0x7f53f806cc60 2024-03-24T00:31:13.900 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:13.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 -- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53d4067660 msgr2=0x7f53d4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53d4067660 0x7f53d4069b20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f53e80af000 tx=0x7f53e80046d0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 -- 172.21.15.40:0/1213783871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f806cc60 msgr2=0x7f53f80c9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:13.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f806cc60 0x7f53f80c9be0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f53e000e580 tx=0x7f53e000ea40 comp rx=0 tx=0).stop 2024-03-24T00:31:13.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 -- 172.21.15.40:0/1213783871 shutdown_connections 2024-03-24T00:31:13.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53d4067660 0x7f53d4069b20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53f8150460 0x7f53f81527e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53f806cc60 0x7f53f80c9be0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 --2- 172.21.15.40:0/1213783871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc003b80 0x7f53f80cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:13.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 -- 172.21.15.40:0/1213783871 >> 172.21.15.40:0/1213783871 conn(0x7f53f806ecf0 msgr2=0x7f53f815d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:13.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 -- 172.21.15.40:0/1213783871 shutdown_connections 2024-03-24T00:31:13.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:13.898+0000 7f53fda00640 1 -- 172.21.15.40:0/1213783871 wait complete. 2024-03-24T00:31:13.920 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:14.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff226a00640 1 Processor -- start 2024-03-24T00:31:14.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff226a00640 1 -- start start 2024-03-24T00:31:14.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff226a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff22015df80 0x7ff220070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff226a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff2200d2270 0x7ff220161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff226a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22006d0f0 con 0x7ff22006c950 2024-03-24T00:31:14.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff226a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22006b890 con 0x7ff22015df80 2024-03-24T00:31:14.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff226a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff22006b280 con 0x7ff2200d2270 2024-03-24T00:31:14.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff21fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff22015df80 0x7ff220070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff21fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff22015df80 0x7ff220070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47336/0 (socket says 172.21.15.40:47336) 2024-03-24T00:31:14.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff21fe00640 1 -- 172.21.15.40:0/2293404544 learned_addr learned my addr 172.21.15.40:0/2293404544 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:14.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff224a00640 1 --2- 172.21.15.40:0/2293404544 >> v2:172.21.15.53:3300/0 conn(0x7ff2200d2270 0x7ff220161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff21fe00640 1 -- 172.21.15.40:0/2293404544 >> v2:172.21.15.53:3300/0 conn(0x7ff2200d2270 msgr2=0x7ff220161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:14.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff21fe00640 1 --2- 172.21.15.40:0/2293404544 >> v2:172.21.15.53:3300/0 conn(0x7ff2200d2270 0x7ff220161ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff21fe00640 1 -- 172.21.15.40:0/2293404544 >> v1:172.21.15.40:6789/0 conn(0x7ff22006c950 legacy=0x7ff2200d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.062+0000 7ff21fe00640 1 -- 172.21.15.40:0/2293404544 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22006ce10 con 0x7ff22015df80 2024-03-24T00:31:14.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff21fe00640 1 --2- 172.21.15.40:0/2293404544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff22015df80 0x7ff220070ff0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff2200ce480 tx=0x7ff21001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ed9593e0380e3fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:14.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff21ea00640 1 -- 172.21.15.40:0/2293404544 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2100aa610 con 0x7ff22015df80 2024-03-24T00:31:14.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff21ea00640 1 -- 172.21.15.40:0/2293404544 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2100af070 con 0x7ff22015df80 2024-03-24T00:31:14.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff21ea00640 1 -- 172.21.15.40:0/2293404544 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2100aa9d0 con 0x7ff22015df80 2024-03-24T00:31:14.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff226a00640 1 -- 172.21.15.40:0/2293404544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff22015df80 msgr2=0x7ff220070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff226a00640 1 --2- 172.21.15.40:0/2293404544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff22015df80 0x7ff220070ff0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff2200ce480 tx=0x7ff21001c4f0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff226a00640 1 -- 172.21.15.40:0/2293404544 shutdown_connections 2024-03-24T00:31:14.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff226a00640 1 --2- 172.21.15.40:0/2293404544 >> v2:172.21.15.53:3300/0 conn(0x7ff2200d2270 0x7ff220161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff226a00640 1 --2- 172.21.15.40:0/2293404544 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff22015df80 0x7ff220070ff0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff226a00640 1 -- 172.21.15.40:0/2293404544 >> 172.21.15.40:0/2293404544 conn(0x7ff22006ecf0 msgr2=0x7ff22006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:14.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff226a00640 1 -- 172.21.15.40:0/2293404544 shutdown_connections 2024-03-24T00:31:14.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff226a00640 1 -- 172.21.15.40:0/2293404544 wait complete. 2024-03-24T00:31:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff226a00640 1 Processor -- start 2024-03-24T00:31:14.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.066+0000 7ff226a00640 1 -- start start 2024-03-24T00:31:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff226a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff22006c950 0x7ff2200b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff226a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff2200d2270 0x7ff220151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff226a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff22015df80 0x7ff220158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff226a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22016af90 con 0x7ff22006c950 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff226a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff22016ae10 con 0x7ff22015df80 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff226a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff22016b110 con 0x7ff2200d2270 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff21fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff22006c950 0x7ff2200b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff21fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff22006c950 0x7ff2200b01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47338/0 (socket says 172.21.15.40:47338) 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff21fe00640 1 -- 172.21.15.40:0/173977774 learned_addr learned my addr 172.21.15.40:0/173977774 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff21f400640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff2200d2270 0x7ff220151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff224a00640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff22015df80 0x7ff220158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff21fe00640 1 -- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff2200d2270 msgr2=0x7ff220151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff21fe00640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff2200d2270 0x7ff220151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff21fe00640 1 -- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff22015df80 msgr2=0x7ff220158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff21fe00640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff22015df80 0x7ff220158830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff21fe00640 1 -- 172.21.15.40:0/173977774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2200add20 con 0x7ff22006c950 2024-03-24T00:31:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff21fe00640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff22006c950 0x7ff2200b01e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff210019100 tx=0x7ff21001ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff217e00640 1 -- 172.21.15.40:0/173977774 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2100bd5f0 con 0x7ff22006c950 2024-03-24T00:31:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff217e00640 1 -- 172.21.15.40:0/173977774 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2100bd790 con 0x7ff22006c950 2024-03-24T00:31:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff217e00640 1 -- 172.21.15.40:0/173977774 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2100bdb80 con 0x7ff22006c950 2024-03-24T00:31:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff226a00640 1 -- 172.21.15.40:0/173977774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2200ae000 con 0x7ff22006c950 2024-03-24T00:31:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff226a00640 1 -- 172.21.15.40:0/173977774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2200ae5c0 con 0x7ff22006c950 2024-03-24T00:31:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.070+0000 7ff226a00640 1 -- 172.21.15.40:0/173977774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff220068170 con 0x7ff22006c950 2024-03-24T00:31:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.074+0000 7ff217e00640 1 -- 172.21.15.40:0/173977774 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff2100b33f0 con 0x7ff22006c950 2024-03-24T00:31:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.074+0000 7ff217e00640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff200067660 0x7ff200069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.074+0000 7ff217e00640 1 -- 172.21.15.40:0/173977774 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff210120050 con 0x7ff22006c950 2024-03-24T00:31:14.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.074+0000 7ff21f400640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff200067660 0x7ff200069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.074+0000 7ff21f400640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff200067660 0x7ff200069b20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff220157020 tx=0x7ff20c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:14.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.082+0000 7ff217e00640 1 -- 172.21.15.40:0/173977774 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff2100a7070 con 0x7ff22006c950 2024-03-24T00:31:14.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.246+0000 7ff226a00640 1 -- 172.21.15.40:0/173977774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff2200cf9d0 con 0x7ff22006c950 2024-03-24T00:31:14.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff217e00640 1 -- 172.21.15.40:0/173977774 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff210016110 con 0x7ff22006c950 2024-03-24T00:31:14.252 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff226a00640 1 -- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff200067660 msgr2=0x7ff200069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff226a00640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff200067660 0x7ff200069b20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff220157020 tx=0x7ff20c009040 comp rx=0 tx=0).stop 2024-03-24T00:31:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff226a00640 1 -- 172.21.15.40:0/173977774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff22006c950 msgr2=0x7ff2200b01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff226a00640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff22006c950 0x7ff2200b01e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff210019100 tx=0x7ff21001ad30 comp rx=0 tx=0).stop 2024-03-24T00:31:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff226a00640 1 -- 172.21.15.40:0/173977774 shutdown_connections 2024-03-24T00:31:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff226a00640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff200067660 0x7ff200069b20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff226a00640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff22015df80 0x7ff220158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff226a00640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff2200d2270 0x7ff220151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff226a00640 1 --2- 172.21.15.40:0/173977774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff22006c950 0x7ff2200b01e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff226a00640 1 -- 172.21.15.40:0/173977774 >> 172.21.15.40:0/173977774 conn(0x7ff22006ecf0 msgr2=0x7ff2200b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:14.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff226a00640 1 -- 172.21.15.40:0/173977774 shutdown_connections 2024-03-24T00:31:14.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.250+0000 7ff226a00640 1 -- 172.21.15.40:0/173977774 wait complete. 2024-03-24T00:31:14.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:31:14.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.410+0000 7fd6d4e00640 1 Processor -- start 2024-03-24T00:31:14.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.410+0000 7fd6d4e00640 1 -- start start 2024-03-24T00:31:14.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6d4e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd6d00cd1c0 0x7fd6d009dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6d4e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6d006e8a0 0x7fd6d0072510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6d4e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6d0072a50 0x7fd6d015f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6d4e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d006e370 con 0x7fd6d0072a50 2024-03-24T00:31:14.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6d4e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d006dc00 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6d4e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00ad430 con 0x7fd6d006e8a0 2024-03-24T00:31:14.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6cfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6d0072a50 0x7fd6d015f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6cfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6d0072a50 0x7fd6d015f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47352/0 (socket says 172.21.15.40:47352) 2024-03-24T00:31:14.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6cfe00640 1 -- 172.21.15.40:0/302894772 learned_addr learned my addr 172.21.15.40:0/302894772 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:14.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6c6a00640 1 --2- 172.21.15.40:0/302894772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6d006e8a0 0x7fd6d0072510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6cf400640 1 --2- 172.21.15.40:0/302894772 >> v2:172.21.15.53:3300/0 conn(0x7fd6d00cd1c0 0x7fd6d009dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6cfe00640 1 -- 172.21.15.40:0/302894772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6d006e8a0 msgr2=0x7fd6d0072510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:14.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6cfe00640 1 --2- 172.21.15.40:0/302894772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6d006e8a0 0x7fd6d0072510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6cfe00640 1 -- 172.21.15.40:0/302894772 >> v2:172.21.15.53:3300/0 conn(0x7fd6d00cd1c0 msgr2=0x7fd6d009dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:14.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6cfe00640 1 --2- 172.21.15.40:0/302894772 >> v2:172.21.15.53:3300/0 conn(0x7fd6d00cd1c0 0x7fd6d009dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.414+0000 7fd6cfe00640 1 -- 172.21.15.40:0/302894772 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d006de30 con 0x7fd6d0072a50 2024-03-24T00:31:14.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6cfe00640 1 --2- 172.21.15.40:0/302894772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d0072a50 0x7fd6d015f5e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00b2c50 tx=0x7fd6c801c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb87b1812538df6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:14.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6cea00640 1 -- 172.21.15.40:0/302894772 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80a64d0 con 0x7fd6d0072a50 2024-03-24T00:31:14.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6cea00640 1 -- 172.21.15.40:0/302894772 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c80a6670 con 0x7fd6d0072a50 2024-03-24T00:31:14.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6cea00640 1 -- 172.21.15.40:0/302894772 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80a6a40 con 0x7fd6d0072a50 2024-03-24T00:31:14.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 -- 172.21.15.40:0/302894772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d0072a50 msgr2=0x7fd6d015f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 --2- 172.21.15.40:0/302894772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d0072a50 0x7fd6d015f5e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00b2c50 tx=0x7fd6c801c730 comp rx=0 tx=0).stop 2024-03-24T00:31:14.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 -- 172.21.15.40:0/302894772 shutdown_connections 2024-03-24T00:31:14.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 --2- 172.21.15.40:0/302894772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d0072a50 0x7fd6d015f5e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 --2- 172.21.15.40:0/302894772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6d006e8a0 0x7fd6d0072510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 --2- 172.21.15.40:0/302894772 >> v2:172.21.15.53:3300/0 conn(0x7fd6d00cd1c0 0x7fd6d009dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 -- 172.21.15.40:0/302894772 >> 172.21.15.40:0/302894772 conn(0x7fd6d00b1c80 msgr2=0x7fd6d00b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:14.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 -- 172.21.15.40:0/302894772 shutdown_connections 2024-03-24T00:31:14.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 -- 172.21.15.40:0/302894772 wait complete. 2024-03-24T00:31:14.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 Processor -- start 2024-03-24T00:31:14.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 -- start start 2024-03-24T00:31:14.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6d006e8a0 0x7fd6d01f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6d0072a50 0x7fd6d0154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d00cd1c0 0x7fd6d0154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00ae440 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00ae2c0 con 0x7fd6d0072a50 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.418+0000 7fd6d4e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d00ae5c0 con 0x7fd6d006e8a0 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6cfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d00cd1c0 0x7fd6d0154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6cfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d00cd1c0 0x7fd6d0154b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47360/0 (socket says 172.21.15.40:47360) 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6cfe00640 1 -- 172.21.15.40:0/2646124865 learned_addr learned my addr 172.21.15.40:0/2646124865 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6cf400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6d006e8a0 0x7fd6d01f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6c6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6d0072a50 0x7fd6d0154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6cfe00640 1 -- 172.21.15.40:0/2646124865 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6d006e8a0 msgr2=0x7fd6d01f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6cfe00640 1 --2- 172.21.15.40:0/2646124865 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6d006e8a0 0x7fd6d01f6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6cfe00640 1 -- 172.21.15.40:0/2646124865 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6d0072a50 msgr2=0x7fd6d0154610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6cfe00640 1 --2- 172.21.15.40:0/2646124865 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6d0072a50 0x7fd6d0154610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6cfe00640 1 -- 172.21.15.40:0/2646124865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d00cb540 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6cfe00640 1 --2- 172.21.15.40:0/2646124865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d00cd1c0 0x7fd6d0154b50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd6c801ade0 tx=0x7fd6c80194d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:14.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6c7e00640 1 -- 172.21.15.40:0/2646124865 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80bf070 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6c7e00640 1 -- 172.21.15.40:0/2646124865 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c80a6820 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6d4e00640 1 -- 172.21.15.40:0/2646124865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d00cb820 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6c7e00640 1 -- 172.21.15.40:0/2646124865 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80a6c10 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6d4e00640 1 -- 172.21.15.40:0/2646124865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d00cbde0 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6d4e00640 1 -- 172.21.15.40:0/2646124865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d009e2e0 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.422+0000 7fd6c7e00640 1 -- 172.21.15.40:0/2646124865 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd6c80c2070 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.426+0000 7fd6c7e00640 1 --2- 172.21.15.40:0/2646124865 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd6ac067660 0x7fd6ac069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.426+0000 7fd6c7e00640 1 -- 172.21.15.40:0/2646124865 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6c811f420 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.426+0000 7fd6cf400640 1 --2- 172.21.15.40:0/2646124865 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd6ac067660 0x7fd6ac069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.426+0000 7fd6cf400640 1 --2- 172.21.15.40:0/2646124865 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd6ac067660 0x7fd6ac069b20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009ac0 tx=0x7fd6c0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:14.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.430+0000 7fd6c7e00640 1 -- 172.21.15.40:0/2646124865 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd6c80165e0 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.662+0000 7fd6d4e00640 1 -- 172.21.15.40:0/2646124865 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6d00732d0 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.662+0000 7fd6c7e00640 1 -- 172.21.15.40:0/2646124865 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4965 (secure 0 0 0) 0x7fd6c80efb60 con 0x7fd6d00cd1c0 2024-03-24T00:31:14.666 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:14.667 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/501744669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":501744669},{"type":"v1","addr":"172.21.15.40:6833","nonce":501744669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/281354690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":281354690},{"type":"v1","addr":"172.21.15.53:6836","nonce":281354690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3092758821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3092758821},{"type":"v1","addr":"172.21.15.53:6837","nonce":3092758821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:30:59.427652+0000","modified":"2024-03-24T00:31:02.444298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4139303536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":4139303536},{"type":"v1","addr":"172.21.15.40:6835","nonce":4139303536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-24T00:31:14.667 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 8 2024-03-24T00:31:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.666+0000 7fd6d4e00640 1 -- 172.21.15.40:0/2646124865 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd6ac067660 msgr2=0x7fd6ac069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.666+0000 7fd6d4e00640 1 --2- 172.21.15.40:0/2646124865 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd6ac067660 0x7fd6ac069b20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009ac0 tx=0x7fd6c0009190 comp rx=0 tx=0).stop 2024-03-24T00:31:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.666+0000 7fd6d4e00640 1 -- 172.21.15.40:0/2646124865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d00cd1c0 msgr2=0x7fd6d0154b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.666+0000 7fd6d4e00640 1 --2- 172.21.15.40:0/2646124865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d00cd1c0 0x7fd6d0154b50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd6c801ade0 tx=0x7fd6c80194d0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.666+0000 7fd6d4e00640 1 -- 172.21.15.40:0/2646124865 shutdown_connections 2024-03-24T00:31:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.666+0000 7fd6d4e00640 1 --2- 172.21.15.40:0/2646124865 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd6ac067660 0x7fd6ac069b20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.666+0000 7fd6d4e00640 1 --2- 172.21.15.40:0/2646124865 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6d00cd1c0 0x7fd6d0154b50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.666+0000 7fd6d4e00640 1 --2- 172.21.15.40:0/2646124865 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6d0072a50 0x7fd6d0154610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.666+0000 7fd6d4e00640 1 --2- 172.21.15.40:0/2646124865 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6d006e8a0 0x7fd6d01f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.666+0000 7fd6d4e00640 1 -- 172.21.15.40:0/2646124865 >> 172.21.15.40:0/2646124865 conn(0x7fd6d00b1c80 msgr2=0x7fd6d00b0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.666+0000 7fd6d4e00640 1 -- 172.21.15.40:0/2646124865 shutdown_connections 2024-03-24T00:31:14.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.666+0000 7fd6d4e00640 1 -- 172.21.15.40:0/2646124865 wait complete. 2024-03-24T00:31:14.688 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc35000640 1 Processor -- start 2024-03-24T00:31:14.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc35000640 1 -- start start 2024-03-24T00:31:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc35000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbc3013ec80 0x7fbc301414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc35000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc301419f0 0x7fbc300ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc35000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc30070e00 con 0x7fbc300ce630 2024-03-24T00:31:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc35000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc30141e50 con 0x7fbc3013ec80 2024-03-24T00:31:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc35000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc300cee20 con 0x7fbc301419f0 2024-03-24T00:31:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc2e800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbc300ce630 0x7fbc3015f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58024/0 (socket says 172.21.15.40:58024) 2024-03-24T00:31:14.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc2e800640 1 -- 172.21.15.40:0/3680378726 learned_addr learned my addr 172.21.15.40:0/3680378726 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc2de00640 1 --2- 172.21.15.40:0/3680378726 >> v2:172.21.15.53:3300/0 conn(0x7fbc3013ec80 0x7fbc301414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc25400640 1 --2- 172.21.15.40:0/3680378726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc301419f0 0x7fbc300ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc2d400640 1 -- 172.21.15.40:0/3680378726 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23695208 0 0) 0x7fbc30070e00 con 0x7fbc300ce630 2024-03-24T00:31:14.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc2d400640 1 -- 172.21.15.40:0/3680378726 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc14003530 con 0x7fbc300ce630 2024-03-24T00:31:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc25400640 1 -- 172.21.15.40:0/3680378726 >> v2:172.21.15.53:3300/0 conn(0x7fbc3013ec80 msgr2=0x7fbc301414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc25400640 1 --2- 172.21.15.40:0/3680378726 >> v2:172.21.15.53:3300/0 conn(0x7fbc3013ec80 0x7fbc301414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc25400640 1 -- 172.21.15.40:0/3680378726 >> v1:172.21.15.40:6789/0 conn(0x7fbc300ce630 legacy=0x7fbc3015f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc25400640 1 -- 172.21.15.40:0/3680378726 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc300cf100 con 0x7fbc301419f0 2024-03-24T00:31:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.846+0000 7fbc2de00640 1 --2- 172.21.15.40:0/3680378726 >> v2:172.21.15.53:3300/0 conn(0x7fbc3013ec80 0x7fbc301414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:14.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc25400640 1 --2- 172.21.15.40:0/3680378726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc301419f0 0x7fbc300ce0f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbc18009d90 tx=0x7fbc1801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76694e909da668f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:14.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc2d400640 1 -- 172.21.15.40:0/3680378726 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc180b2070 con 0x7fbc301419f0 2024-03-24T00:31:14.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc2d400640 1 -- 172.21.15.40:0/3680378726 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc180a5a50 con 0x7fbc301419f0 2024-03-24T00:31:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc2d400640 1 -- 172.21.15.40:0/3680378726 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc180a5e00 con 0x7fbc301419f0 2024-03-24T00:31:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 -- 172.21.15.40:0/3680378726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc301419f0 msgr2=0x7fbc300ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 --2- 172.21.15.40:0/3680378726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc301419f0 0x7fbc300ce0f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbc18009d90 tx=0x7fbc1801c970 comp rx=0 tx=0).stop 2024-03-24T00:31:14.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 -- 172.21.15.40:0/3680378726 shutdown_connections 2024-03-24T00:31:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 --2- 172.21.15.40:0/3680378726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc301419f0 0x7fbc300ce0f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 --2- 172.21.15.40:0/3680378726 >> v2:172.21.15.53:3300/0 conn(0x7fbc3013ec80 0x7fbc301414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 -- 172.21.15.40:0/3680378726 >> 172.21.15.40:0/3680378726 conn(0x7fbc300b17b0 msgr2=0x7fbc300b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 -- 172.21.15.40:0/3680378726 shutdown_connections 2024-03-24T00:31:14.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 -- 172.21.15.40:0/3680378726 wait complete. 2024-03-24T00:31:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 Processor -- start 2024-03-24T00:31:14.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 -- start start 2024-03-24T00:31:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc300ce630 0x7fbc30155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc3013ec80 0x7fbc300b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc301419f0 0x7fbc3014ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc30168ec0 con 0x7fbc301419f0 2024-03-24T00:31:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc30168d40 con 0x7fbc300ce630 2024-03-24T00:31:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.854+0000 7fbc35000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc30169040 con 0x7fbc3013ec80 2024-03-24T00:31:14.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc2e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc301419f0 0x7fbc3014ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc2e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc301419f0 0x7fbc3014ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47366/0 (socket says 172.21.15.40:47366) 2024-03-24T00:31:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc2e800640 1 -- 172.21.15.40:0/1615268987 learned_addr learned my addr 172.21.15.40:0/1615268987 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc25400640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc3013ec80 0x7fbc300b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc2de00640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc300ce630 0x7fbc30155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc2e800640 1 -- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc3013ec80 msgr2=0x7fbc300b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc2e800640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc3013ec80 0x7fbc300b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc2e800640 1 -- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc300ce630 msgr2=0x7fbc30155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc2e800640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc300ce630 0x7fbc30155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc2e800640 1 -- 172.21.15.40:0/1615268987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc30155250 con 0x7fbc301419f0 2024-03-24T00:31:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc2e800640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc301419f0 0x7fbc3014ebd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c3f0 tx=0x7fbc2800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc26a00640 1 -- 172.21.15.40:0/1615268987 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28017070 con 0x7fbc301419f0 2024-03-24T00:31:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc26a00640 1 -- 172.21.15.40:0/1615268987 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc28005cf0 con 0x7fbc301419f0 2024-03-24T00:31:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc26a00640 1 -- 172.21.15.40:0/1615268987 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28007310 con 0x7fbc301419f0 2024-03-24T00:31:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc35000640 1 -- 172.21.15.40:0/1615268987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc30156fe0 con 0x7fbc301419f0 2024-03-24T00:31:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc35000640 1 -- 172.21.15.40:0/1615268987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc301575a0 con 0x7fbc301419f0 2024-03-24T00:31:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc35000640 1 -- 172.21.15.40:0/1615268987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc30068170 con 0x7fbc301419f0 2024-03-24T00:31:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.858+0000 7fbc26a00640 1 -- 172.21.15.40:0/1615268987 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbc280074b0 con 0x7fbc301419f0 2024-03-24T00:31:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.862+0000 7fbc26a00640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc0c067660 0x7fbc0c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.862+0000 7fbc26a00640 1 -- 172.21.15.40:0/1615268987 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc28084ba0 con 0x7fbc301419f0 2024-03-24T00:31:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.862+0000 7fbc2de00640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc0c067660 0x7fbc0c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.862+0000 7fbc2de00640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc0c067660 0x7fbc0c069b20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbc30158110 tx=0x7fbc20009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:14.866+0000 7fbc26a00640 1 -- 172.21.15.40:0/1615268987 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbc30155250 con 0x7fbc301419f0 2024-03-24T00:31:15.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.138+0000 7fbc35000640 1 -- 172.21.15.40:0/1615268987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc3006e0d0 con 0x7fbc301419f0 2024-03-24T00:31:15.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.138+0000 7fbc26a00640 1 -- 172.21.15.40:0/1615268987 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7fbc28055130 con 0x7fbc301419f0 2024-03-24T00:31:15.143 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:15.143 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:15.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.142+0000 7fbc35000640 1 -- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc0c067660 msgr2=0x7fbc0c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.142+0000 7fbc35000640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc0c067660 0x7fbc0c069b20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbc30158110 tx=0x7fbc20009190 comp rx=0 tx=0).stop 2024-03-24T00:31:15.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.142+0000 7fbc35000640 1 -- 172.21.15.40:0/1615268987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc301419f0 msgr2=0x7fbc3014ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.142+0000 7fbc35000640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc301419f0 0x7fbc3014ebd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c3f0 tx=0x7fbc2800c8b0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.142+0000 7fbc35000640 1 -- 172.21.15.40:0/1615268987 shutdown_connections 2024-03-24T00:31:15.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.142+0000 7fbc35000640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc0c067660 0x7fbc0c069b20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.142+0000 7fbc35000640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc301419f0 0x7fbc3014ebd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.142+0000 7fbc35000640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc3013ec80 0x7fbc300b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.142+0000 7fbc35000640 1 --2- 172.21.15.40:0/1615268987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc300ce630 0x7fbc30155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.142+0000 7fbc35000640 1 -- 172.21.15.40:0/1615268987 >> 172.21.15.40:0/1615268987 conn(0x7fbc300b17b0 msgr2=0x7fbc3006e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:15.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.142+0000 7fbc35000640 1 -- 172.21.15.40:0/1615268987 shutdown_connections 2024-03-24T00:31:15.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.142+0000 7fbc35000640 1 -- 172.21.15.40:0/1615268987 wait complete. 2024-03-24T00:31:15.164 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:15.165 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:15.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.306+0000 7f81a0400640 1 Processor -- start 2024-03-24T00:31:15.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.306+0000 7f81a0400640 1 -- start start 2024-03-24T00:31:15.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.306+0000 7f81a0400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f819815a560 0x7f81980ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:15.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.306+0000 7f81a0400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8198070ed0 con 0x7f81980aec70 2024-03-24T00:31:15.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.306+0000 7f81a0400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81980cf240 con 0x7f819806d3a0 2024-03-24T00:31:15.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.306+0000 7f81a0400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81980cf520 con 0x7f819815a560 2024-03-24T00:31:15.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.306+0000 7f819d000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f81980aec70 0x7f81980cc4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58038/0 (socket says 172.21.15.40:58038) 2024-03-24T00:31:15.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.306+0000 7f819d000640 1 -- 172.21.15.40:0/4290377986 learned_addr learned my addr 172.21.15.40:0/4290377986 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:15.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.306+0000 7f819da00640 1 --2- 172.21.15.40:0/4290377986 >> v2:172.21.15.53:3300/0 conn(0x7f819815a560 0x7f81980ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.306+0000 7f8197e00640 1 -- 172.21.15.40:0/4290377986 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309203557 0 0) 0x7f8198070ed0 con 0x7f81980aec70 2024-03-24T00:31:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f8197e00640 1 -- 172.21.15.40:0/4290377986 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8180003530 con 0x7f81980aec70 2024-03-24T00:31:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f8197e00640 1 -- 172.21.15.40:0/4290377986 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211264917 0 0) 0x7f81980cf240 con 0x7f819806d3a0 2024-03-24T00:31:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f8197e00640 1 -- 172.21.15.40:0/4290377986 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8198070ed0 con 0x7f819806d3a0 2024-03-24T00:31:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f819da00640 1 -- 172.21.15.40:0/4290377986 >> v1:172.21.15.53:6789/0 conn(0x7f819806d3a0 legacy=0x7f81980b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f819da00640 1 -- 172.21.15.40:0/4290377986 >> v1:172.21.15.40:6789/0 conn(0x7f81980aec70 legacy=0x7f81980cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f819da00640 1 -- 172.21.15.40:0/4290377986 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81980b3aa0 con 0x7f819815a560 2024-03-24T00:31:15.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f819da00640 1 --2- 172.21.15.40:0/4290377986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819815a560 0x7f81980ae6b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f818c0097c0 tx=0x7f818c01c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bcf5729e38b7762 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:15.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f8197e00640 1 -- 172.21.15.40:0/4290377986 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c0b1070 con 0x7f819815a560 2024-03-24T00:31:15.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f8197e00640 1 -- 172.21.15.40:0/4290377986 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818c0ac600 con 0x7f819815a560 2024-03-24T00:31:15.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f8197e00640 1 -- 172.21.15.40:0/4290377986 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c0ac9d0 con 0x7f819815a560 2024-03-24T00:31:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f81a0400640 1 -- 172.21.15.40:0/4290377986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819815a560 msgr2=0x7f81980ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f81a0400640 1 --2- 172.21.15.40:0/4290377986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819815a560 0x7f81980ae6b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f818c0097c0 tx=0x7f818c01c380 comp rx=0 tx=0).stop 2024-03-24T00:31:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f81a0400640 1 -- 172.21.15.40:0/4290377986 shutdown_connections 2024-03-24T00:31:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.310+0000 7f81a0400640 1 --2- 172.21.15.40:0/4290377986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819815a560 0x7f81980ae6b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f81a0400640 1 -- 172.21.15.40:0/4290377986 >> 172.21.15.40:0/4290377986 conn(0x7f81980b1660 msgr2=0x7f81980b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:15.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f81a0400640 1 -- 172.21.15.40:0/4290377986 shutdown_connections 2024-03-24T00:31:15.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f81a0400640 1 -- 172.21.15.40:0/4290377986 wait complete. 2024-03-24T00:31:15.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f81a0400640 1 Processor -- start 2024-03-24T00:31:15.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f81a0400640 1 -- start start 2024-03-24T00:31:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f81a0400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f819806d3a0 0x7f8198157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f81a0400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81980aec70 0x7f819813e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f81a0400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819815a560 0x7f8198155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:15.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f81a0400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819815aac0 con 0x7f81980aec70 2024-03-24T00:31:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f81a0400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819815a940 con 0x7f819815a560 2024-03-24T00:31:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f81a0400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8198169380 con 0x7f819806d3a0 2024-03-24T00:31:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f819d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81980aec70 0x7f819813e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f819da00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f819806d3a0 0x7f8198157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f819d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81980aec70 0x7f819813e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47374/0 (socket says 172.21.15.40:47374) 2024-03-24T00:31:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f819d000640 1 -- 172.21.15.40:0/2444586320 learned_addr learned my addr 172.21.15.40:0/2444586320 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f819e400640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819815a560 0x7f8198155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f819d000640 1 -- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f819806d3a0 msgr2=0x7f8198157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f819d000640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f819806d3a0 0x7f8198157a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f819d000640 1 -- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819815a560 msgr2=0x7f8198155590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f819d000640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819815a560 0x7f8198155590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f819d000640 1 -- 172.21.15.40:0/2444586320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81981f7a80 con 0x7f81980aec70 2024-03-24T00:31:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f819d000640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81980aec70 0x7f819813e200 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f818400c430 tx=0x7f818400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f8195600640 1 -- 172.21.15.40:0/2444586320 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184017070 con 0x7f81980aec70 2024-03-24T00:31:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.314+0000 7f8195600640 1 -- 172.21.15.40:0/2444586320 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8184005cf0 con 0x7f81980aec70 2024-03-24T00:31:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.318+0000 7f8195600640 1 -- 172.21.15.40:0/2444586320 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184007310 con 0x7f81980aec70 2024-03-24T00:31:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.318+0000 7f81a0400640 1 -- 172.21.15.40:0/2444586320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81981f7d60 con 0x7f81980aec70 2024-03-24T00:31:15.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.318+0000 7f81a0400640 1 -- 172.21.15.40:0/2444586320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81981f7740 con 0x7f81980aec70 2024-03-24T00:31:15.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.318+0000 7f81a0400640 1 -- 172.21.15.40:0/2444586320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8198068170 con 0x7f81980aec70 2024-03-24T00:31:15.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.318+0000 7f8195600640 1 -- 172.21.15.40:0/2444586320 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f818401a070 con 0x7f81980aec70 2024-03-24T00:31:15.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.318+0000 7f8195600640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8178067690 0x7f8178069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:15.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.318+0000 7f8195600640 1 -- 172.21.15.40:0/2444586320 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8184084cb0 con 0x7f81980aec70 2024-03-24T00:31:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.318+0000 7f819da00640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8178067690 0x7f8178069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:15.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.318+0000 7f819da00640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8178067690 0x7f8178069b50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f818c0ae000 tx=0x7f818c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:15.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.322+0000 7f8195600640 1 -- 172.21.15.40:0/2444586320 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8184055240 con 0x7f81980aec70 2024-03-24T00:31:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.490+0000 7f81a0400640 1 -- 172.21.15.40:0/2444586320 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8198071ba0 con 0x7f81980aec70 2024-03-24T00:31:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.490+0000 7f8195600640 1 -- 172.21.15.40:0/2444586320 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f81840592e0 con 0x7f81980aec70 2024-03-24T00:31:15.493 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.490+0000 7f81a0400640 1 -- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8178067690 msgr2=0x7f8178069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.490+0000 7f81a0400640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8178067690 0x7f8178069b50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f818c0ae000 tx=0x7f818c0a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.490+0000 7f81a0400640 1 -- 172.21.15.40:0/2444586320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81980aec70 msgr2=0x7f819813e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.490+0000 7f81a0400640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81980aec70 0x7f819813e200 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f818400c430 tx=0x7f818400c8f0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.490+0000 7f81a0400640 1 -- 172.21.15.40:0/2444586320 shutdown_connections 2024-03-24T00:31:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.490+0000 7f81a0400640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8178067690 0x7f8178069b50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.494+0000 7f81a0400640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f819815a560 0x7f8198155590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.494+0000 7f81a0400640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81980aec70 0x7f819813e200 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.494+0000 7f81a0400640 1 --2- 172.21.15.40:0/2444586320 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f819806d3a0 0x7f8198157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.494+0000 7f81a0400640 1 -- 172.21.15.40:0/2444586320 >> 172.21.15.40:0/2444586320 conn(0x7f81980b1660 msgr2=0x7f81980ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.494+0000 7f81a0400640 1 -- 172.21.15.40:0/2444586320 shutdown_connections 2024-03-24T00:31:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.494+0000 7f81a0400640 1 -- 172.21.15.40:0/2444586320 wait complete. 2024-03-24T00:31:15.514 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:15.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.654+0000 7fd554800640 1 Processor -- start 2024-03-24T00:31:15.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.654+0000 7fd554800640 1 -- start start 2024-03-24T00:31:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd554800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd54c15df80 0x7fd54c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd554800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd54c0d2270 0x7fd54c161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd554800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd54c06d0f0 con 0x7fd54c06c950 2024-03-24T00:31:15.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd554800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54c06b890 con 0x7fd54c0d2270 2024-03-24T00:31:15.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd554800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54c06b280 con 0x7fd54c15df80 2024-03-24T00:31:15.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd552800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd54c0d2270 0x7fd54c161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:15.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd552800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd54c0d2270 0x7fd54c161ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47380/0 (socket says 172.21.15.40:47380) 2024-03-24T00:31:15.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd552800640 1 -- 172.21.15.40:0/2657372015 learned_addr learned my addr 172.21.15.40:0/2657372015 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:15.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd551e00640 1 --2- 172.21.15.40:0/2657372015 >> v2:172.21.15.53:3300/0 conn(0x7fd54c15df80 0x7fd54c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:15.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd552800640 1 -- 172.21.15.40:0/2657372015 >> v2:172.21.15.53:3300/0 conn(0x7fd54c15df80 msgr2=0x7fd54c070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:15.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd552800640 1 --2- 172.21.15.40:0/2657372015 >> v2:172.21.15.53:3300/0 conn(0x7fd54c15df80 0x7fd54c070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd552800640 1 -- 172.21.15.40:0/2657372015 >> v1:172.21.15.53:6789/0 conn(0x7fd54c06c950 legacy=0x7fd54c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.658+0000 7fd552800640 1 -- 172.21.15.40:0/2657372015 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c06ce10 con 0x7fd54c0d2270 2024-03-24T00:31:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd552800640 1 --2- 172.21.15.40:0/2657372015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd54c0d2270 0x7fd54c161ad0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd54c0d2c90 tx=0x7fd54801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba9ac3c322fd18d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd550a00640 1 -- 172.21.15.40:0/2657372015 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5480a5840 con 0x7fd54c0d2270 2024-03-24T00:31:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd550a00640 1 -- 172.21.15.40:0/2657372015 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5480a59e0 con 0x7fd54c0d2270 2024-03-24T00:31:15.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd550a00640 1 -- 172.21.15.40:0/2657372015 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5480a5db0 con 0x7fd54c0d2270 2024-03-24T00:31:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd554800640 1 -- 172.21.15.40:0/2657372015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd54c0d2270 msgr2=0x7fd54c161ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd554800640 1 --2- 172.21.15.40:0/2657372015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd54c0d2270 0x7fd54c161ad0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd54c0d2c90 tx=0x7fd54801c9c0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd554800640 1 -- 172.21.15.40:0/2657372015 shutdown_connections 2024-03-24T00:31:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd554800640 1 --2- 172.21.15.40:0/2657372015 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd54c0d2270 0x7fd54c161ad0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd554800640 1 --2- 172.21.15.40:0/2657372015 >> v2:172.21.15.53:3300/0 conn(0x7fd54c15df80 0x7fd54c070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd554800640 1 -- 172.21.15.40:0/2657372015 >> 172.21.15.40:0/2657372015 conn(0x7fd54c06ecf0 msgr2=0x7fd54c06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd554800640 1 -- 172.21.15.40:0/2657372015 shutdown_connections 2024-03-24T00:31:15.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd554800640 1 -- 172.21.15.40:0/2657372015 wait complete. 2024-03-24T00:31:15.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd554800640 1 Processor -- start 2024-03-24T00:31:15.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.662+0000 7fd554800640 1 -- start start 2024-03-24T00:31:15.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd554800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd54c06c950 0x7fd54c151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:15.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd554800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd54c0d2270 0x7fd54c0b1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:15.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd554800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd54c15df80 0x7fd54c0b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:15.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd554800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c16af90 con 0x7fd54c0d2270 2024-03-24T00:31:15.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd554800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c16ae10 con 0x7fd54c15df80 2024-03-24T00:31:15.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd554800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd54c16b110 con 0x7fd54c06c950 2024-03-24T00:31:15.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd551400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd54c0d2270 0x7fd54c0b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd551e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd54c06c950 0x7fd54c151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd551400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd54c0d2270 0x7fd54c0b1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47382/0 (socket says 172.21.15.40:47382) 2024-03-24T00:31:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd551400640 1 -- 172.21.15.40:0/1066127961 learned_addr learned my addr 172.21.15.40:0/1066127961 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd552800640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd54c15df80 0x7fd54c0b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd551400640 1 -- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd54c06c950 msgr2=0x7fd54c151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd551400640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd54c06c950 0x7fd54c151960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd551400640 1 -- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd54c15df80 msgr2=0x7fd54c0b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd551400640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd54c15df80 0x7fd54c0b1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd551400640 1 -- 172.21.15.40:0/1066127961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54c0ada70 con 0x7fd54c0d2270 2024-03-24T00:31:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd551400640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd54c0d2270 0x7fd54c0b1730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00c450 tx=0x7fd53c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd546000640 1 -- 172.21.15.40:0/1066127961 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c016020 con 0x7fd54c0d2270 2024-03-24T00:31:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd546000640 1 -- 172.21.15.40:0/1066127961 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c0052d0 con 0x7fd54c0d2270 2024-03-24T00:31:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd546000640 1 -- 172.21.15.40:0/1066127961 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c007260 con 0x7fd54c0d2270 2024-03-24T00:31:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd554800640 1 -- 172.21.15.40:0/1066127961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd54c13e3c0 con 0x7fd54c0d2270 2024-03-24T00:31:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd554800640 1 -- 172.21.15.40:0/1066127961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd54c151ea0 con 0x7fd54c0d2270 2024-03-24T00:31:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.666+0000 7fd554800640 1 -- 172.21.15.40:0/1066127961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd54c068170 con 0x7fd54c0d2270 2024-03-24T00:31:15.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.670+0000 7fd546000640 1 -- 172.21.15.40:0/1066127961 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd53c007400 con 0x7fd54c0d2270 2024-03-24T00:31:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.670+0000 7fd546000640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd530067660 0x7fd530069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.670+0000 7fd546000640 1 -- 172.21.15.40:0/1066127961 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd53c085120 con 0x7fd54c0d2270 2024-03-24T00:31:15.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.670+0000 7fd551e00640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd530067660 0x7fd530069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:15.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.670+0000 7fd551e00640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd530067660 0x7fd530069b20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd54c1fa470 tx=0x7fd540009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:15.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.674+0000 7fd546000640 1 -- 172.21.15.40:0/1066127961 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd53c020020 con 0x7fd54c0d2270 2024-03-24T00:31:15.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.834+0000 7fd554800640 1 -- 172.21.15.40:0/1066127961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd54c0cf9d0 con 0x7fd54c0d2270 2024-03-24T00:31:15.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd546000640 1 -- 172.21.15.40:0/1066127961 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd53c005480 con 0x7fd54c0d2270 2024-03-24T00:31:15.840 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:15.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd554800640 1 -- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd530067660 msgr2=0x7fd530069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd554800640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd530067660 0x7fd530069b20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd54c1fa470 tx=0x7fd540009190 comp rx=0 tx=0).stop 2024-03-24T00:31:15.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd554800640 1 -- 172.21.15.40:0/1066127961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd54c0d2270 msgr2=0x7fd54c0b1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:15.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd554800640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd54c0d2270 0x7fd54c0b1730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00c450 tx=0x7fd53c00c910 comp rx=0 tx=0).stop 2024-03-24T00:31:15.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd554800640 1 -- 172.21.15.40:0/1066127961 shutdown_connections 2024-03-24T00:31:15.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd554800640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd530067660 0x7fd530069b20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd554800640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd54c15df80 0x7fd54c0b1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd554800640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd54c0d2270 0x7fd54c0b1730 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd554800640 1 --2- 172.21.15.40:0/1066127961 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd54c06c950 0x7fd54c151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:15.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd554800640 1 -- 172.21.15.40:0/1066127961 >> 172.21.15.40:0/1066127961 conn(0x7fd54c06ecf0 msgr2=0x7fd54c0d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:15.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd554800640 1 -- 172.21.15.40:0/1066127961 shutdown_connections 2024-03-24T00:31:15.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:15.838+0000 7fd554800640 1 -- 172.21.15.40:0/1066127961 wait complete. 2024-03-24T00:31:15.861 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.002+0000 7f27d2600640 1 Processor -- start 2024-03-24T00:31:16.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.002+0000 7f27d2600640 1 -- start start 2024-03-24T00:31:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.002+0000 7f27d2600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f27cc15bd00 0x7f27cc0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.002+0000 7f27d2600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc072d60 con 0x7f27cc0728d0 2024-03-24T00:31:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.002+0000 7f27d2600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc071de0 con 0x7f27cc15ffd0 2024-03-24T00:31:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.002+0000 7f27d2600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27cc0cea50 con 0x7f27cc15bd00 2024-03-24T00:31:16.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.002+0000 7f27cb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f27cc15bd00 0x7f27cc0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.002+0000 7f27c3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f27cc0728d0 0x7f27cc15f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58042/0 (socket says 172.21.15.40:58042) 2024-03-24T00:31:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.002+0000 7f27c3e00640 1 -- 172.21.15.40:0/1715072212 learned_addr learned my addr 172.21.15.40:0/1715072212 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.002+0000 7f27cb400640 1 -- 172.21.15.40:0/1715072212 >> v1:172.21.15.53:6789/0 conn(0x7f27cc15ffd0 legacy=0x7f27cc06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.002+0000 7f27cb400640 1 -- 172.21.15.40:0/1715072212 >> v1:172.21.15.40:6789/0 conn(0x7f27cc0728d0 legacy=0x7f27cc15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.002+0000 7f27cb400640 1 -- 172.21.15.40:0/1715072212 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc0ced30 con 0x7f27cc15bd00 2024-03-24T00:31:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.006+0000 7f27cb400640 1 --2- 172.21.15.40:0/1715072212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27cc15bd00 0x7f27cc0ae8e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0097c0 tx=0x7f27bc01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5eee2a0552b4bf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.006+0000 7f27caa00640 1 -- 172.21.15.40:0/1715072212 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc0ab610 con 0x7f27cc15bd00 2024-03-24T00:31:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.006+0000 7f27caa00640 1 -- 172.21.15.40:0/1715072212 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc0b0070 con 0x7f27cc15bd00 2024-03-24T00:31:16.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.006+0000 7f27caa00640 1 -- 172.21.15.40:0/1715072212 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc0ab9d0 con 0x7f27cc15bd00 2024-03-24T00:31:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.006+0000 7f27d2600640 1 -- 172.21.15.40:0/1715072212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27cc15bd00 msgr2=0x7f27cc0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.006+0000 7f27d2600640 1 --2- 172.21.15.40:0/1715072212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27cc15bd00 0x7f27cc0ae8e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0097c0 tx=0x7f27bc01c430 comp rx=0 tx=0).stop 2024-03-24T00:31:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.006+0000 7f27d2600640 1 -- 172.21.15.40:0/1715072212 shutdown_connections 2024-03-24T00:31:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.006+0000 7f27d2600640 1 --2- 172.21.15.40:0/1715072212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27cc15bd00 0x7f27cc0ae8e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.006+0000 7f27d2600640 1 -- 172.21.15.40:0/1715072212 >> 172.21.15.40:0/1715072212 conn(0x7f27cc0b1660 msgr2=0x7f27cc0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.006+0000 7f27d2600640 1 -- 172.21.15.40:0/1715072212 shutdown_connections 2024-03-24T00:31:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.006+0000 7f27d2600640 1 -- 172.21.15.40:0/1715072212 wait complete. 2024-03-24T00:31:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27d2600640 1 Processor -- start 2024-03-24T00:31:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27d2600640 1 -- start start 2024-03-24T00:31:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27d2600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27cc0728d0 0x7f27cc144420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27d2600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27cc15bd00 0x7f27cc15c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27d2600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27cc15ffd0 0x7f27cc150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27d2600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc15c0e0 con 0x7f27cc15ffd0 2024-03-24T00:31:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27d2600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc0cf660 con 0x7f27cc0728d0 2024-03-24T00:31:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27d2600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc16a420 con 0x7f27cc15bd00 2024-03-24T00:31:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27cc15ffd0 0x7f27cc150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27cc15ffd0 0x7f27cc150a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47408/0 (socket says 172.21.15.40:47408) 2024-03-24T00:31:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27cbe00640 1 -- 172.21.15.40:0/2916518288 learned_addr learned my addr 172.21.15.40:0/2916518288 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27cb400640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27cc0728d0 0x7f27cc144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27c3e00640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27cc15bd00 0x7f27cc15c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27cbe00640 1 -- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27cc15bd00 msgr2=0x7f27cc15c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27cbe00640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27cc15bd00 0x7f27cc15c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27cbe00640 1 -- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27cc0728d0 msgr2=0x7f27cc144420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27cbe00640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27cc0728d0 0x7f27cc144420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27cbe00640 1 -- 172.21.15.40:0/2916518288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc15cc90 con 0x7f27cc15ffd0 2024-03-24T00:31:16.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27cbe00640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27cc15ffd0 0x7f27cc150a70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f27c400b970 tx=0x7f27c400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:16.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27c3400640 1 -- 172.21.15.40:0/2916518288 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4019070 con 0x7f27cc15ffd0 2024-03-24T00:31:16.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27c3400640 1 -- 172.21.15.40:0/2916518288 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c400cb40 con 0x7f27cc15ffd0 2024-03-24T00:31:16.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27c3400640 1 -- 172.21.15.40:0/2916518288 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4014440 con 0x7f27cc15ffd0 2024-03-24T00:31:16.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27d2600640 1 -- 172.21.15.40:0/2916518288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc15cf70 con 0x7f27cc15ffd0 2024-03-24T00:31:16.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27d2600640 1 -- 172.21.15.40:0/2916518288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc15d530 con 0x7f27cc15ffd0 2024-03-24T00:31:16.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.010+0000 7f27d2600640 1 -- 172.21.15.40:0/2916518288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27cc068170 con 0x7f27cc15ffd0 2024-03-24T00:31:16.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.014+0000 7f27c3400640 1 -- 172.21.15.40:0/2916518288 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f27c401c070 con 0x7f27cc15ffd0 2024-03-24T00:31:16.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.014+0000 7f27c3400640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f27a8067660 0x7f27a8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.014+0000 7f27c3400640 1 -- 172.21.15.40:0/2916518288 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27c40853f0 con 0x7f27cc15ffd0 2024-03-24T00:31:16.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.014+0000 7f27cb400640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f27a8067660 0x7f27a8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.014+0000 7f27cb400640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f27a8067660 0x7f27a8069b20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0af000 tx=0x7f27bc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:16.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.018+0000 7f27c3400640 1 -- 172.21.15.40:0/2916518288 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f27c4055a10 con 0x7f27cc15ffd0 2024-03-24T00:31:16.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 -- 172.21.15.40:0/2916518288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f27cc152ad0 con 0x7f27cc15ffd0 2024-03-24T00:31:16.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27c3400640 1 -- 172.21.15.40:0/2916518288 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f27c4059ab0 con 0x7f27cc15ffd0 2024-03-24T00:31:16.248 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:16.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 -- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f27a8067660 msgr2=0x7f27a8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f27a8067660 0x7f27a8069b20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0af000 tx=0x7f27bc0a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:16.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 -- 172.21.15.40:0/2916518288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27cc15ffd0 msgr2=0x7f27cc150a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27cc15ffd0 0x7f27cc150a70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f27c400b970 tx=0x7f27c400be30 comp rx=0 tx=0).stop 2024-03-24T00:31:16.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 -- 172.21.15.40:0/2916518288 shutdown_connections 2024-03-24T00:31:16.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f27a8067660 0x7f27a8069b20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f27cc15ffd0 0x7f27cc150a70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27cc15bd00 0x7f27cc15c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 --2- 172.21.15.40:0/2916518288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27cc0728d0 0x7f27cc144420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 -- 172.21.15.40:0/2916518288 >> 172.21.15.40:0/2916518288 conn(0x7f27cc0b1660 msgr2=0x7f27cc148b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:16.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 -- 172.21.15.40:0/2916518288 shutdown_connections 2024-03-24T00:31:16.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.246+0000 7f27d2600640 1 -- 172.21.15.40:0/2916518288 wait complete. 2024-03-24T00:31:16.270 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:31:16.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ada200640 1 Processor -- start 2024-03-24T00:31:16.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ada200640 1 -- start start 2024-03-24T00:31:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ada200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ad415bca0 0x7f4ad40d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ada200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ad415c080 0x7f4ad40d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ada200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad40d0780 con 0x7f4ad40ce490 2024-03-24T00:31:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ada200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad40b4a90 con 0x7f4ad415bca0 2024-03-24T00:31:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ada200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40b4d70 con 0x7f4ad415c080 2024-03-24T00:31:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ad8e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ad415bca0 0x7f4ad40d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ad8e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4ad415bca0 0x7f4ad40d2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47424/0 (socket says 172.21.15.40:47424) 2024-03-24T00:31:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ad9800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4ad40ce490 0x7f4ad40cc5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:58052/0 (socket says 172.21.15.40:58052) 2024-03-24T00:31:16.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ad8e00640 1 -- 172.21.15.40:0/1674052895 learned_addr learned my addr 172.21.15.40:0/1674052895 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:16.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ad3e00640 1 --2- 172.21.15.40:0/1674052895 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ad415c080 0x7f4ad40d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ad8e00640 1 -- 172.21.15.40:0/1674052895 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ad415c080 msgr2=0x7f4ad40d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:16.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ad8e00640 1 --2- 172.21.15.40:0/1674052895 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ad415c080 0x7f4ad40d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ad8e00640 1 -- 172.21.15.40:0/1674052895 >> v1:172.21.15.40:6789/0 conn(0x7f4ad40ce490 legacy=0x7f4ad40cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.410+0000 7f4ad8e00640 1 -- 172.21.15.40:0/1674052895 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad40b5050 con 0x7f4ad415bca0 2024-03-24T00:31:16.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.414+0000 7f4ad8e00640 1 --2- 172.21.15.40:0/1674052895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad415bca0 0x7f4ad40d2590 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009870 tx=0x7f4ac801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f0c2088ce4c1d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:16.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.414+0000 7f4ad3400640 1 -- 172.21.15.40:0/1674052895 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac80b2070 con 0x7f4ad415bca0 2024-03-24T00:31:16.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.414+0000 7f4ad3400640 1 -- 172.21.15.40:0/1674052895 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac80a9070 con 0x7f4ad415bca0 2024-03-24T00:31:16.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.414+0000 7f4ad3400640 1 -- 172.21.15.40:0/1674052895 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac80ad880 con 0x7f4ad415bca0 2024-03-24T00:31:16.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 -- 172.21.15.40:0/1674052895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad415bca0 msgr2=0x7f4ad40d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 --2- 172.21.15.40:0/1674052895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad415bca0 0x7f4ad40d2590 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4ac8009870 tx=0x7f4ac801c560 comp rx=0 tx=0).stop 2024-03-24T00:31:16.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 -- 172.21.15.40:0/1674052895 shutdown_connections 2024-03-24T00:31:16.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 --2- 172.21.15.40:0/1674052895 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ad415c080 0x7f4ad40d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 --2- 172.21.15.40:0/1674052895 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad415bca0 0x7f4ad40d2590 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 -- 172.21.15.40:0/1674052895 >> 172.21.15.40:0/1674052895 conn(0x7f4ad40d1aa0 msgr2=0x7f4ad40d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:16.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 -- 172.21.15.40:0/1674052895 shutdown_connections 2024-03-24T00:31:16.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 -- 172.21.15.40:0/1674052895 wait complete. 2024-03-24T00:31:16.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 Processor -- start 2024-03-24T00:31:16.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 -- start start 2024-03-24T00:31:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad40ce490 0x7f4ad415dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad415bca0 0x7f4ad40c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ad415c080 0x7f4ad415c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad4168c50 con 0x7f4ad40ce490 2024-03-24T00:31:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40cd4d0 con 0x7f4ad415bca0 2024-03-24T00:31:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ada200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad4168dd0 con 0x7f4ad415c080 2024-03-24T00:31:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ad8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad40ce490 0x7f4ad415dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ad8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad40ce490 0x7f4ad415dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47430/0 (socket says 172.21.15.40:47430) 2024-03-24T00:31:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ad8e00640 1 -- 172.21.15.40:0/1640512796 learned_addr learned my addr 172.21.15.40:0/1640512796 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ad9800640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ad415c080 0x7f4ad415c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ad3e00640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad415bca0 0x7f4ad40c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ad8e00640 1 -- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ad415c080 msgr2=0x7f4ad415c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ad8e00640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ad415c080 0x7f4ad415c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ad8e00640 1 -- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad415bca0 msgr2=0x7f4ad40c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ad8e00640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad415bca0 0x7f4ad40c7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ad8e00640 1 -- 172.21.15.40:0/1640512796 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad415cd40 con 0x7f4ad40ce490 2024-03-24T00:31:16.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ad3e00640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad415bca0 0x7f4ad40c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:16.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.418+0000 7f4ad8e00640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad40ce490 0x7f4ad415dde0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80b0040 tx=0x7f4ac801ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:16.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.422+0000 7f4ad0c00640 1 -- 172.21.15.40:0/1640512796 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac80b2040 con 0x7f4ad40ce490 2024-03-24T00:31:16.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.422+0000 7f4ad0c00640 1 -- 172.21.15.40:0/1640512796 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac80b8420 con 0x7f4ad40ce490 2024-03-24T00:31:16.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.422+0000 7f4ad0c00640 1 -- 172.21.15.40:0/1640512796 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac80b8830 con 0x7f4ad40ce490 2024-03-24T00:31:16.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.422+0000 7f4ada200640 1 -- 172.21.15.40:0/1640512796 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad415d020 con 0x7f4ad40ce490 2024-03-24T00:31:16.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.422+0000 7f4ada200640 1 -- 172.21.15.40:0/1640512796 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad415d5e0 con 0x7f4ad40ce490 2024-03-24T00:31:16.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.422+0000 7f4ac6000640 1 -- 172.21.15.40:0/1640512796 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad409fb10 con 0x7f4ad40ce490 2024-03-24T00:31:16.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.422+0000 7f4ad0c00640 1 -- 172.21.15.40:0/1640512796 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4ac80c0030 con 0x7f4ad40ce490 2024-03-24T00:31:16.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.426+0000 7f4ad0c00640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ab4067690 0x7f4ab4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.426+0000 7f4ad0c00640 1 -- 172.21.15.40:0/1640512796 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ac811f600 con 0x7f4ad40ce490 2024-03-24T00:31:16.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.426+0000 7f4ad3e00640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ab4067690 0x7f4ab4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.426+0000 7f4ad3e00640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ab4067690 0x7f4ab4069b50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4ad409f8b0 tx=0x7f4ac0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:16.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.430+0000 7f4ad0c00640 1 -- 172.21.15.40:0/1640512796 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4ac80c7060 con 0x7f4ad40ce490 2024-03-24T00:31:16.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.626+0000 7f4ac6000640 1 -- 172.21.15.40:0/1640512796 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ad404f4f0 con 0x7f4ad40ce490 2024-03-24T00:31:16.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.626+0000 7f4ad0c00640 1 -- 172.21.15.40:0/1640512796 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4965 (secure 0 0 0) 0x7f4ac8016150 con 0x7f4ad40ce490 2024-03-24T00:31:16.630 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:16.630 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/501744669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":501744669},{"type":"v1","addr":"172.21.15.40:6833","nonce":501744669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/281354690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":281354690},{"type":"v1","addr":"172.21.15.53:6836","nonce":281354690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3092758821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3092758821},{"type":"v1","addr":"172.21.15.53:6837","nonce":3092758821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:30:59.427652+0000","modified":"2024-03-24T00:31:02.444298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4139303536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":4139303536},{"type":"v1","addr":"172.21.15.40:6835","nonce":4139303536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-24T00:31:16.631 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 8 2024-03-24T00:31:16.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.630+0000 7f4ac6000640 1 -- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ab4067690 msgr2=0x7f4ab4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.630+0000 7f4ac6000640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ab4067690 0x7f4ab4069b50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4ad409f8b0 tx=0x7f4ac0008040 comp rx=0 tx=0).stop 2024-03-24T00:31:16.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.630+0000 7f4ac6000640 1 -- 172.21.15.40:0/1640512796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad40ce490 msgr2=0x7f4ad415dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.630+0000 7f4ac6000640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad40ce490 0x7f4ad415dde0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4ac80b0040 tx=0x7f4ac801ada0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.630+0000 7f4ac6000640 1 -- 172.21.15.40:0/1640512796 shutdown_connections 2024-03-24T00:31:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.630+0000 7f4ac6000640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ab4067690 0x7f4ab4069b50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.630+0000 7f4ac6000640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ad415c080 0x7f4ad415c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.630+0000 7f4ac6000640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad415bca0 0x7f4ad40c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.630+0000 7f4ac6000640 1 --2- 172.21.15.40:0/1640512796 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ad40ce490 0x7f4ad415dde0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.630+0000 7f4ac6000640 1 -- 172.21.15.40:0/1640512796 >> 172.21.15.40:0/1640512796 conn(0x7f4ad40d1aa0 msgr2=0x7f4ad406f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.630+0000 7f4ac6000640 1 -- 172.21.15.40:0/1640512796 shutdown_connections 2024-03-24T00:31:16.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.630+0000 7f4ac6000640 1 -- 172.21.15.40:0/1640512796 wait complete. 2024-03-24T00:31:16.652 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:16.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc8000640 1 Processor -- start 2024-03-24T00:31:16.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc8000640 1 -- start start 2024-03-24T00:31:16.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc8000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7bc015de80 0x7f7bc0066e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc8000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7bc00b6450 0x7f7bc00673c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc8000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bc0162a80 con 0x7f7bc00b6b30 2024-03-24T00:31:16.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc8000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bc006f870 con 0x7f7bc00b6450 2024-03-24T00:31:16.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc8000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc0161cc0 con 0x7f7bc015de80 2024-03-24T00:31:16.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc4c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7bc00b6450 0x7f7bc00673c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc4c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7bc00b6450 0x7f7bc00673c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47440/0 (socket says 172.21.15.40:47440) 2024-03-24T00:31:16.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc4c00640 1 -- 172.21.15.40:0/3431054141 learned_addr learned my addr 172.21.15.40:0/3431054141 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:16.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc5600640 1 --2- 172.21.15.40:0/3431054141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7bc015de80 0x7f7bc0066e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc4c00640 1 -- 172.21.15.40:0/3431054141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7bc015de80 msgr2=0x7f7bc0066e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:16.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc4c00640 1 --2- 172.21.15.40:0/3431054141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7bc015de80 0x7f7bc0066e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc4c00640 1 -- 172.21.15.40:0/3431054141 >> v1:172.21.15.53:6789/0 conn(0x7f7bc00b6b30 legacy=0x7f7bc006ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:16.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.790+0000 7f7bc4c00640 1 -- 172.21.15.40:0/3431054141 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc0161fa0 con 0x7f7bc00b6450 2024-03-24T00:31:16.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.794+0000 7f7bc4c00640 1 --2- 172.21.15.40:0/3431054141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bc00b6450 0x7f7bc00673c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009870 tx=0x7f7bac01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99da267b1afaa2f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:16.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.794+0000 7f7bbfe00640 1 -- 172.21.15.40:0/3431054141 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0b1070 con 0x7f7bc00b6450 2024-03-24T00:31:16.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.794+0000 7f7bbfe00640 1 -- 172.21.15.40:0/3431054141 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bac0ac620 con 0x7f7bc00b6450 2024-03-24T00:31:16.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.794+0000 7f7bbfe00640 1 -- 172.21.15.40:0/3431054141 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bac0ac9f0 con 0x7f7bc00b6450 2024-03-24T00:31:16.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.794+0000 7f7bc8000640 1 -- 172.21.15.40:0/3431054141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bc00b6450 msgr2=0x7f7bc00673c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.794+0000 7f7bc8000640 1 --2- 172.21.15.40:0/3431054141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bc00b6450 0x7f7bc00673c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7bac009870 tx=0x7f7bac01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.794+0000 7f7bc8000640 1 -- 172.21.15.40:0/3431054141 shutdown_connections 2024-03-24T00:31:16.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.794+0000 7f7bc8000640 1 --2- 172.21.15.40:0/3431054141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bc00b6450 0x7f7bc00673c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.794+0000 7f7bc8000640 1 --2- 172.21.15.40:0/3431054141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7bc015de80 0x7f7bc0066e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.794+0000 7f7bc8000640 1 -- 172.21.15.40:0/3431054141 >> 172.21.15.40:0/3431054141 conn(0x7f7bc00b2a20 msgr2=0x7f7bc00b2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:16.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.794+0000 7f7bc8000640 1 -- 172.21.15.40:0/3431054141 shutdown_connections 2024-03-24T00:31:16.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.794+0000 7f7bc8000640 1 -- 172.21.15.40:0/3431054141 wait complete. 2024-03-24T00:31:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc8000640 1 Processor -- start 2024-03-24T00:31:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc8000640 1 -- start start 2024-03-24T00:31:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc8000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bc00b6450 0x7f7bc01f6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc8000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bc00b6b30 0x7f7bc0071f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc8000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7bc015de80 0x7f7bc0160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc8000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc015e260 con 0x7f7bc00b6450 2024-03-24T00:31:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc8000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc006f520 con 0x7f7bc00b6b30 2024-03-24T00:31:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc8000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc015e3e0 con 0x7f7bc015de80 2024-03-24T00:31:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bc00b6450 0x7f7bc01f6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bc00b6450 0x7f7bc01f6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47448/0 (socket says 172.21.15.40:47448) 2024-03-24T00:31:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc5600640 1 -- 172.21.15.40:0/3557278049 learned_addr learned my addr 172.21.15.40:0/3557278049 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:16.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc4c00640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bc00b6b30 0x7f7bc0071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc6000640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7bc015de80 0x7f7bc0160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc5600640 1 -- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7bc015de80 msgr2=0x7f7bc0160600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc5600640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7bc015de80 0x7f7bc0160600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc5600640 1 -- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bc00b6b30 msgr2=0x7f7bc0071f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:16.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc5600640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bc00b6b30 0x7f7bc0071f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc5600640 1 -- 172.21.15.40:0/3557278049 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc0160fc0 con 0x7f7bc00b6450 2024-03-24T00:31:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc6000640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7bc015de80 0x7f7bc0160600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc5600640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bc00b6450 0x7f7bc01f6d20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400d800 tx=0x7f7bb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bbd600640 1 -- 172.21.15.40:0/3557278049 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb400b6c0 con 0x7f7bc00b6450 2024-03-24T00:31:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bbd600640 1 -- 172.21.15.40:0/3557278049 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bb4019070 con 0x7f7bc00b6450 2024-03-24T00:31:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc8000640 1 -- 172.21.15.40:0/3557278049 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bc01612a0 con 0x7f7bc00b6450 2024-03-24T00:31:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bbd600640 1 -- 172.21.15.40:0/3557278049 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bb400ba60 con 0x7f7bc00b6450 2024-03-24T00:31:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc8000640 1 -- 172.21.15.40:0/3557278049 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bc0157c60 con 0x7f7bc00b6450 2024-03-24T00:31:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.798+0000 7f7bc8000640 1 -- 172.21.15.40:0/3557278049 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bc0068170 con 0x7f7bc00b6450 2024-03-24T00:31:16.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.802+0000 7f7bbd600640 1 -- 172.21.15.40:0/3557278049 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7bb400bc00 con 0x7f7bc00b6450 2024-03-24T00:31:16.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.802+0000 7f7bbd600640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7ba0067660 0x7f7ba0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:16.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.802+0000 7f7bbd600640 1 -- 172.21.15.40:0/3557278049 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bb4085d70 con 0x7f7bc00b6450 2024-03-24T00:31:16.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.802+0000 7f7bc4c00640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7ba0067660 0x7f7ba0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:16.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.802+0000 7f7bc4c00640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7ba0067660 0x7f7ba0069b20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0ae000 tx=0x7f7bac0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:16.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:16.806+0000 7f7bbd600640 1 -- 172.21.15.40:0/3557278049 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7bb4056950 con 0x7f7bc00b6450 2024-03-24T00:31:17.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 -- 172.21.15.40:0/3557278049 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bc01f7260 con 0x7f7bc00b6450 2024-03-24T00:31:17.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bbd600640 1 -- 172.21.15.40:0/3557278049 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7f7bb4056250 con 0x7f7bc00b6450 2024-03-24T00:31:17.113 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:17.113 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 -- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7ba0067660 msgr2=0x7f7ba0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7ba0067660 0x7f7ba0069b20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0ae000 tx=0x7f7bac0a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 -- 172.21.15.40:0/3557278049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bc00b6450 msgr2=0x7f7bc01f6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bc00b6450 0x7f7bc01f6d20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7bb400d800 tx=0x7f7bb400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 -- 172.21.15.40:0/3557278049 shutdown_connections 2024-03-24T00:31:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7ba0067660 0x7f7ba0069b20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7bc015de80 0x7f7bc0160600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bc00b6b30 0x7f7bc0071f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 --2- 172.21.15.40:0/3557278049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7bc00b6450 0x7f7bc01f6d20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 -- 172.21.15.40:0/3557278049 >> 172.21.15.40:0/3557278049 conn(0x7f7bc00b2a20 msgr2=0x7f7bc015ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 -- 172.21.15.40:0/3557278049 shutdown_connections 2024-03-24T00:31:17.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.110+0000 7f7bc8000640 1 -- 172.21.15.40:0/3557278049 wait complete. 2024-03-24T00:31:17.135 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:31:17.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb5cc00640 1 Processor -- start 2024-03-24T00:31:17.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb5cc00640 1 -- start start 2024-03-24T00:31:17.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb5cc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feb58160030 0x7feb58070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:17.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb5cc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb580cec10 con 0x7feb5806de00 2024-03-24T00:31:17.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb5cc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb580ceeb0 con 0x7feb58145520 2024-03-24T00:31:17.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb5cc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb580cf190 con 0x7feb58160030 2024-03-24T00:31:17.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb4fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feb58160030 0x7feb58070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:17.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb57e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7feb5806de00 0x7feb58071030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43120/0 (socket says 172.21.15.40:43120) 2024-03-24T00:31:17.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb4fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feb58160030 0x7feb58070af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35056/0 (socket says 172.21.15.40:35056) 2024-03-24T00:31:17.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb57e00640 1 -- 172.21.15.40:0/2458721173 learned_addr learned my addr 172.21.15.40:0/2458721173 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:17.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb4fe00640 1 -- 172.21.15.40:0/2458721173 >> v1:172.21.15.53:6789/0 conn(0x7feb58145520 legacy=0x7feb5815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:17.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb4fe00640 1 -- 172.21.15.40:0/2458721173 >> v1:172.21.15.40:6789/0 conn(0x7feb5806de00 legacy=0x7feb58071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:17.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.274+0000 7feb4fe00640 1 -- 172.21.15.40:0/2458721173 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb580cf470 con 0x7feb58160030 2024-03-24T00:31:17.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.278+0000 7feb4fe00640 1 --2- 172.21.15.40:0/2458721173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58160030 0x7feb58070af0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feb400097a0 tx=0x7feb4001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46be9e097df9da69 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:17.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.278+0000 7feb56a00640 1 -- 172.21.15.40:0/2458721173 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb400ab610 con 0x7feb58160030 2024-03-24T00:31:17.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.278+0000 7feb56a00640 1 -- 172.21.15.40:0/2458721173 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb400b0070 con 0x7feb58160030 2024-03-24T00:31:17.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.278+0000 7feb56a00640 1 -- 172.21.15.40:0/2458721173 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb400ab9d0 con 0x7feb58160030 2024-03-24T00:31:17.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.278+0000 7feb5cc00640 1 -- 172.21.15.40:0/2458721173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58160030 msgr2=0x7feb58070af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:17.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.278+0000 7feb5cc00640 1 --2- 172.21.15.40:0/2458721173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58160030 0x7feb58070af0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7feb400097a0 tx=0x7feb4001c3f0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.278+0000 7feb5cc00640 1 -- 172.21.15.40:0/2458721173 shutdown_connections 2024-03-24T00:31:17.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.278+0000 7feb5cc00640 1 --2- 172.21.15.40:0/2458721173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58160030 0x7feb58070af0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.278+0000 7feb5cc00640 1 -- 172.21.15.40:0/2458721173 >> 172.21.15.40:0/2458721173 conn(0x7feb580b1740 msgr2=0x7feb580b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:17.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.278+0000 7feb5cc00640 1 -- 172.21.15.40:0/2458721173 shutdown_connections 2024-03-24T00:31:17.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.278+0000 7feb5cc00640 1 -- 172.21.15.40:0/2458721173 wait complete. 2024-03-24T00:31:17.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb5cc00640 1 Processor -- start 2024-03-24T00:31:17.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb5cc00640 1 -- start start 2024-03-24T00:31:17.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb5cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb5806de00 0x7feb581f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:17.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb5cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb58145520 0x7feb5813ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:17.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb5cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58160030 0x7feb5814a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:17.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb5cc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb58169c70 con 0x7feb58160030 2024-03-24T00:31:17.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb5cc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb58145b10 con 0x7feb5806de00 2024-03-24T00:31:17.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb5cc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb58169df0 con 0x7feb58145520 2024-03-24T00:31:17.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb57e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58160030 0x7feb5814a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:17.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb57e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58160030 0x7feb5814a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35068/0 (socket says 172.21.15.40:35068) 2024-03-24T00:31:17.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb57e00640 1 -- 172.21.15.40:0/3438560242 learned_addr learned my addr 172.21.15.40:0/3438560242 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:17.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb57400640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb5806de00 0x7feb581f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:17.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb4fe00640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb58145520 0x7feb5813ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:17.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb57e00640 1 -- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb58145520 msgr2=0x7feb5813ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:17.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb57e00640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb58145520 0x7feb5813ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb57e00640 1 -- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb5806de00 msgr2=0x7feb581f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:17.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb57e00640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb5806de00 0x7feb581f70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb57e00640 1 -- 172.21.15.40:0/3438560242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb581f7630 con 0x7feb58160030 2024-03-24T00:31:17.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb57e00640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58160030 0x7feb5814a0d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feb5000cbc0 tx=0x7feb5000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:17.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb4f400640 1 -- 172.21.15.40:0/3438560242 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb50017070 con 0x7feb58160030 2024-03-24T00:31:17.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb4f400640 1 -- 172.21.15.40:0/3438560242 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5000ad40 con 0x7feb58160030 2024-03-24T00:31:17.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb5cc00640 1 -- 172.21.15.40:0/3438560242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb581f78e0 con 0x7feb58160030 2024-03-24T00:31:17.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb4f400640 1 -- 172.21.15.40:0/3438560242 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb50012600 con 0x7feb58160030 2024-03-24T00:31:17.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.282+0000 7feb5cc00640 1 -- 172.21.15.40:0/3438560242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb581f7ea0 con 0x7feb58160030 2024-03-24T00:31:17.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.286+0000 7feb5cc00640 1 -- 172.21.15.40:0/3438560242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb58073540 con 0x7feb58160030 2024-03-24T00:31:17.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.286+0000 7feb4f400640 1 -- 172.21.15.40:0/3438560242 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7feb5001a070 con 0x7feb58160030 2024-03-24T00:31:17.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.286+0000 7feb4f400640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb34067660 0x7feb34069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:17.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.286+0000 7feb4f400640 1 -- 172.21.15.40:0/3438560242 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb50085590 con 0x7feb58160030 2024-03-24T00:31:17.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.286+0000 7feb57400640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb34067660 0x7feb34069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:17.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.286+0000 7feb57400640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb34067660 0x7feb34069b20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feb48001650 tx=0x7feb48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:17.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.290+0000 7feb4f400640 1 -- 172.21.15.40:0/3438560242 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7feb50055b20 con 0x7feb58160030 2024-03-24T00:31:17.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 -- 172.21.15.40:0/3438560242 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb581f8240 con 0x7feb58160030 2024-03-24T00:31:17.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb4f400640 1 -- 172.21.15.40:0/3438560242 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4965 (secure 0 0 0) 0x7feb50059bc0 con 0x7feb58160030 2024-03-24T00:31:17.493 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:17.493 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/501744669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":501744669},{"type":"v1","addr":"172.21.15.40:6833","nonce":501744669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/281354690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":281354690},{"type":"v1","addr":"172.21.15.53:6836","nonce":281354690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3092758821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3092758821},{"type":"v1","addr":"172.21.15.53:6837","nonce":3092758821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:30:59.427652+0000","modified":"2024-03-24T00:31:02.444298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4139303536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":4139303536},{"type":"v1","addr":"172.21.15.40:6835","nonce":4139303536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-24T00:31:17.493 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 8 2024-03-24T00:31:17.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 -- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb34067660 msgr2=0x7feb34069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:17.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb34067660 0x7feb34069b20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7feb48001650 tx=0x7feb48009040 comp rx=0 tx=0).stop 2024-03-24T00:31:17.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 -- 172.21.15.40:0/3438560242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58160030 msgr2=0x7feb5814a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:17.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58160030 0x7feb5814a0d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7feb5000cbc0 tx=0x7feb5000a430 comp rx=0 tx=0).stop 2024-03-24T00:31:17.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 -- 172.21.15.40:0/3438560242 shutdown_connections 2024-03-24T00:31:17.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb34067660 0x7feb34069b20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb58160030 0x7feb5814a0d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb58145520 0x7feb5813ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 --2- 172.21.15.40:0/3438560242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb5806de00 0x7feb581f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:17.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 -- 172.21.15.40:0/3438560242 >> 172.21.15.40:0/3438560242 conn(0x7feb580b1740 msgr2=0x7feb58154850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:17.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 -- 172.21.15.40:0/3438560242 shutdown_connections 2024-03-24T00:31:17.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:17.490+0000 7feb5cc00640 1 -- 172.21.15.40:0/3438560242 wait complete. 2024-03-24T00:31:17.916 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:30:59.427652+0000', 'modified': '2024-03-24T00:31:02.444298+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4264}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4264': {'gid': 4264, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/4139303536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 4139303536}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 4139303536}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4264, 'qdb_cluster': [4264]} 2024-03-24T00:31:17.916 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:31:17.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:31:18.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae9800640 1 Processor -- start 2024-03-24T00:31:18.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae9800640 1 -- start start 2024-03-24T00:31:18.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae9800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ae40ae6d0 0x7f2ae40aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae9800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae40ad630 con 0x7f2ae415f3b0 2024-03-24T00:31:18.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae9800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae40ad910 con 0x7f2ae415efd0 2024-03-24T00:31:18.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae9800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae40adbf0 con 0x7f2ae40ae6d0 2024-03-24T00:31:18.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2adbe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2ae415f3b0 0x7f2ae406d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43130/0 (socket says 172.21.15.40:43130) 2024-03-24T00:31:18.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2adbe00640 1 -- 172.21.15.40:0/3462545767 learned_addr learned my addr 172.21.15.40:0/3462545767 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:18.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae3400640 1 --2- 172.21.15.40:0/3462545767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ae40ae6d0 0x7f2ae40aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae2000640 1 -- 172.21.15.40:0/3462545767 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158132169 0 0) 0x7f2ae40ad630 con 0x7f2ae415f3b0 2024-03-24T00:31:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae2000640 1 -- 172.21.15.40:0/3462545767 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac8003530 con 0x7f2ae415f3b0 2024-03-24T00:31:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae2000640 1 -- 172.21.15.40:0/3462545767 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208947046 0 0) 0x7f2ae40ad910 con 0x7f2ae415efd0 2024-03-24T00:31:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae2000640 1 -- 172.21.15.40:0/3462545767 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae40ad630 con 0x7f2ae415efd0 2024-03-24T00:31:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae3400640 1 -- 172.21.15.40:0/3462545767 >> v1:172.21.15.53:6789/0 conn(0x7f2ae415efd0 legacy=0x7f2ae40af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae3400640 1 -- 172.21.15.40:0/3462545767 >> v1:172.21.15.40:6789/0 conn(0x7f2ae415f3b0 legacy=0x7f2ae406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.058+0000 7f2ae3400640 1 -- 172.21.15.40:0/3462545767 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae40aded0 con 0x7f2ae40ae6d0 2024-03-24T00:31:18.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.062+0000 7f2ae3400640 1 --2- 172.21.15.40:0/3462545767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ae40ae6d0 0x7f2ae40aeaf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2adc009dc0 tx=0x7f2adc01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f77bb919597f5ddd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:18.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.062+0000 7f2ae2000640 1 -- 172.21.15.40:0/3462545767 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc0b2070 con 0x7f2ae40ae6d0 2024-03-24T00:31:18.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.062+0000 7f2ae2000640 1 -- 172.21.15.40:0/3462545767 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2adc0a5960 con 0x7f2ae40ae6d0 2024-03-24T00:31:18.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.062+0000 7f2ae2000640 1 -- 172.21.15.40:0/3462545767 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc0a5d30 con 0x7f2ae40ae6d0 2024-03-24T00:31:18.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.062+0000 7f2ae9800640 1 -- 172.21.15.40:0/3462545767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ae40ae6d0 msgr2=0x7f2ae40aeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.062+0000 7f2ae9800640 1 --2- 172.21.15.40:0/3462545767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ae40ae6d0 0x7f2ae40aeaf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2adc009dc0 tx=0x7f2adc01c990 comp rx=0 tx=0).stop 2024-03-24T00:31:18.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.062+0000 7f2ae9800640 1 -- 172.21.15.40:0/3462545767 shutdown_connections 2024-03-24T00:31:18.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.062+0000 7f2ae9800640 1 --2- 172.21.15.40:0/3462545767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ae40ae6d0 0x7f2ae40aeaf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.062+0000 7f2ae9800640 1 -- 172.21.15.40:0/3462545767 >> 172.21.15.40:0/3462545767 conn(0x7f2ae40b1f00 msgr2=0x7f2ae40b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:18.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.062+0000 7f2ae9800640 1 -- 172.21.15.40:0/3462545767 shutdown_connections 2024-03-24T00:31:18.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.062+0000 7f2ae9800640 1 -- 172.21.15.40:0/3462545767 wait complete. 2024-03-24T00:31:18.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2ae9800640 1 Processor -- start 2024-03-24T00:31:18.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2ae9800640 1 -- start start 2024-03-24T00:31:18.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2ae9800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ae40ae6d0 0x7f2ae40c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2ae9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ae415efd0 0x7f2ae4156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2ae9800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ae415f3b0 0x7f2ae4156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2ae9800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae40ad360 con 0x7f2ae415efd0 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2ae9800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae40ad1e0 con 0x7f2ae415f3b0 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2ae9800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae40d0990 con 0x7f2ae40ae6d0 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2adbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ae415efd0 0x7f2ae4156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2adbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ae415efd0 0x7f2ae4156800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35084/0 (socket says 172.21.15.40:35084) 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2adbe00640 1 -- 172.21.15.40:0/3912327772 learned_addr learned my addr 172.21.15.40:0/3912327772 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2ae3400640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ae415f3b0 0x7f2ae4156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2ae2a00640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ae40ae6d0 0x7f2ae40c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2adbe00640 1 -- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ae40ae6d0 msgr2=0x7f2ae40c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2adbe00640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ae40ae6d0 0x7f2ae40c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2adbe00640 1 -- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ae415f3b0 msgr2=0x7f2ae4156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2adbe00640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ae415f3b0 0x7f2ae4156d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2adbe00640 1 -- 172.21.15.40:0/3912327772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae4150d10 con 0x7f2ae415efd0 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2adbe00640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ae415efd0 0x7f2ae4156800 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00c450 tx=0x7f2acc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:18.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.066+0000 7f2adaa00640 1 -- 172.21.15.40:0/3912327772 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc017070 con 0x7f2ae415efd0 2024-03-24T00:31:18.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.070+0000 7f2adaa00640 1 -- 172.21.15.40:0/3912327772 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2acc005cf0 con 0x7f2ae415efd0 2024-03-24T00:31:18.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.070+0000 7f2adaa00640 1 -- 172.21.15.40:0/3912327772 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc007290 con 0x7f2ae415efd0 2024-03-24T00:31:18.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.070+0000 7f2ae9800640 1 -- 172.21.15.40:0/3912327772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae4150ff0 con 0x7f2ae415efd0 2024-03-24T00:31:18.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.070+0000 7f2ae9800640 1 -- 172.21.15.40:0/3912327772 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae40974e0 con 0x7f2ae415efd0 2024-03-24T00:31:18.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.070+0000 7f2adaa00640 1 -- 172.21.15.40:0/3912327772 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2acc01a070 con 0x7f2ae415efd0 2024-03-24T00:31:18.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.070+0000 7f2aecd9c000 1 Processor -- start 2024-03-24T00:31:18.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.070+0000 7f2aecd9c000 1 -- start start 2024-03-24T00:31:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2adaa00640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2ac0067660 0x7f2ac0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2adaa00640 1 -- 172.21.15.40:0/3912327772 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2acc084c10 con 0x7f2ae415efd0 2024-03-24T00:31:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27dd920 0x557ef27ddd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27de240 0x557ef2793ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef2793ff0 0x557ef27943d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef25e1680 con 0x557ef27de240 2024-03-24T00:31:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef2568b00 con 0x557ef2793ff0 2024-03-24T00:31:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ef25dd720 con 0x557ef27dd920 2024-03-24T00:31:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2adbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27de240 0x557ef2793ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2ae3400640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2ac0067660 0x7f2ac0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2adbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27de240 0x557ef2793ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35100/0 (socket says 172.21.15.40:35100) 2024-03-24T00:31:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2adbe00640 1 -- 172.21.15.40:0/3283850271 learned_addr learned my addr 172.21.15.40:0/3283850271 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2ae2a00640 1 --2- 172.21.15.40:0/3283850271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27dd920 0x557ef27ddd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2ae3400640 1 --2- 172.21.15.40:0/3283850271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef2793ff0 0x557ef27943d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2adbe00640 1 -- 172.21.15.40:0/3283850271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27dd920 msgr2=0x557ef27ddd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2adbe00640 1 --2- 172.21.15.40:0/3283850271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27dd920 0x557ef27ddd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2adbe00640 1 -- 172.21.15.40:0/3283850271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef2793ff0 msgr2=0x557ef27943d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2adbe00640 1 --2- 172.21.15.40:0/3283850271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef2793ff0 0x557ef27943d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2adbe00640 1 -- 172.21.15.40:0/3283850271 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ef25b9940 con 0x557ef27de240 2024-03-24T00:31:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2adbe00640 1 --2- 172.21.15.40:0/3283850271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27de240 0x557ef2793ab0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2acc054e80 tx=0x7f2acc055240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1847cdc0c30b9759 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2ad3400640 1 -- 172.21.15.40:0/3283850271 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc053480 con 0x557ef27de240 2024-03-24T00:31:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2ad3400640 1 -- 172.21.15.40:0/3283850271 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2acc0517f0 con 0x557ef27de240 2024-03-24T00:31:18.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2ad3400640 1 -- 172.21.15.40:0/3283850271 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc05d640 con 0x557ef27de240 2024-03-24T00:31:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 -- 172.21.15.40:0/3283850271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27de240 msgr2=0x557ef2793ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/3283850271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27de240 0x557ef2793ab0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2acc054e80 tx=0x7f2acc055240 comp rx=0 tx=0).stop 2024-03-24T00:31:18.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2ae3400640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2ac0067660 0x7f2ac0069b20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2adc0af000 tx=0x7f2adc004ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 -- 172.21.15.40:0/3283850271 shutdown_connections 2024-03-24T00:31:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/3283850271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef2793ff0 0x557ef27943d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/3283850271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27de240 0x557ef2793ab0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/3283850271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27dd920 0x557ef27ddd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 -- 172.21.15.40:0/3283850271 >> 172.21.15.40:0/3283850271 conn(0x557ef278ffa0 msgr2=0x557ef2792390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 -- 172.21.15.40:0/3283850271 shutdown_connections 2024-03-24T00:31:18.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 -- 172.21.15.40:0/3283850271 wait complete. 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 -- start start 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.074+0000 7f2aecd9c000 1 Processor -- start 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 0x557ef27c2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27dd920 0x557ef27c3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef27de240 0x557ef27c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef25e0990 con 0x557ef2793ff0 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef259bae0 con 0x557ef27de240 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ef2579750 con 0x557ef27dd920 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 10 client.0 _ll_register_callbacks cb 0x557ef277eff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ae2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 0x557ef27c2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ae2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 0x557ef27c2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35106/0 (socket says 172.21.15.40:35106) 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ae2a00640 1 -- 172.21.15.40:0/1653487082 learned_addr learned my addr 172.21.15.40:0/1653487082 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:18.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ae3400640 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef27de240 0x557ef27c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2adbe00640 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27dd920 0x557ef27c3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ae2a00640 1 -- 172.21.15.40:0/1653487082 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27dd920 msgr2=0x557ef27c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:18.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ae2a00640 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27dd920 0x557ef27c3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ae2a00640 1 -- 172.21.15.40:0/1653487082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef27de240 msgr2=0x557ef27c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:18.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ae2a00640 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef27de240 0x557ef27c37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ae2a00640 1 -- 172.21.15.40:0/1653487082 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ef2626d80 con 0x557ef2793ff0 2024-03-24T00:31:18.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ae2a00640 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 0x557ef27c2d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2ad40114b0 tx=0x7f2ad4011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b826ed9b298ca3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:18.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ac7e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:31:18.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ac7e00640 1 -- 172.21.15.40:0/1653487082 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad40052e0 con 0x557ef2793ff0 2024-03-24T00:31:18.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ac7e00640 1 -- 172.21.15.40:0/1653487082 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ad4003600 con 0x557ef2793ff0 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 1 -- 172.21.15.40:0/1653487082 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2acc0517f0 con 0x557ef2793ff0 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ac7e00640 1 -- 172.21.15.40:0/1653487082 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ad4022050 con 0x557ef2793ff0 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 10 client.4608 fetch_fsmap learned FSMap version 8 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 1 -- 172.21.15.40:0/1653487082 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x557ef2568d10 con 0x557ef2793ff0 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ac7e00640 1 -- 172.21.15.40:0/1653487082 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2acc0517f0 con 0x557ef2793ff0 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ac7e00640 1 -- 172.21.15.40:0/1653487082 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 8) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2ad4025020 con 0x557ef2793ff0 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 10 client.4608 fetch_fsmap finished waiting for FSMap version 8 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 10 client.4608 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 20 client.4608 populate_metadata read hostname 'smithi040' 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.40:6834/4139303536,v1:172.21.15.40:6835/4139303536] conn(0x557ef27c4060 0x557ef27efea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 4 client.4608 mds_command: new command op to 4264 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2aecd9c000 1 -- 172.21.15.40:0/1653487082 --> [v2:172.21.15.40:6834/4139303536,v1:172.21.15.40:6835/4139303536] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557ef257cb60 con 0x557ef27c4060 2024-03-24T00:31:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2adbe00640 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.40:6834/4139303536,v1:172.21.15.40:6835/4139303536] conn(0x557ef27c4060 0x557ef27efea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2ac7e00640 10 client.4608 ms_handle_connect on v2:172.21.15.40:6834/4139303536 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.078+0000 7f2adbe00640 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.40:6834/4139303536,v1:172.21.15.40:6835/4139303536] conn(0x557ef27c4060 0x557ef27efea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5f7c747849c8cb9 server_cookie=a30040ebbf4dfaf6 in_seq=0 out_seq=0 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2ac7e00640 1 -- 172.21.15.40:0/1653487082 <== mds.0 v2:172.21.15.40:6834/4139303536 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557ef257cb60 con 0x557ef27c4060 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2ac7e00640 10 client.4608 handle_command_reply: tid=0 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2aecd9c000 1 client.4608 shutdown 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2aecd9c000 1 -- 172.21.15.40:0/1653487082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 msgr2=0x557ef27c2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 0x557ef27c2d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2ad40114b0 tx=0x7f2ad4011970 comp rx=0 tx=0).stop 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2aecd9c000 1 -- 172.21.15.40:0/1653487082 shutdown_connections 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.40:6834/4139303536,v1:172.21.15.40:6835/4139303536] conn(0x557ef27c4060 0x557ef27efea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef27de240 0x557ef27c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27dd920 0x557ef27c3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/1653487082 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 0x557ef27c2d40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2ac7e00640 0 client.4608 ms_handle_reset on v2:172.21.15.40:6834/4139303536 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2aecd9c000 1 -- 172.21.15.40:0/1653487082 >> 172.21.15.40:0/1653487082 conn(0x557ef278ffa0 msgr2=0x557ef2792c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:18.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2aecd9c000 1 -- 172.21.15.40:0/1653487082 shutdown_connections 2024-03-24T00:31:18.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2aecd9c000 1 -- 172.21.15.40:0/1653487082 wait complete. 2024-03-24T00:31:18.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.082+0000 7f2aecd9c000 20 client.4608 trim_cache size 0 max 16384 2024-03-24T00:31:18.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 Processor -- start 2024-03-24T00:31:18.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 -- start start 2024-03-24T00:31:18.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef2793ff0 0x557ef27dc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27dd920 0x557ef27dca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27de240 0x557ef27c05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae4202eb0 con 0x557ef27dd920 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae4202070 con 0x557ef2793ff0 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae4155e30 con 0x557ef27de240 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2adbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27dd920 0x557ef27dca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2adbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27dd920 0x557ef27dca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35108/0 (socket says 172.21.15.40:35108) 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2adbe00640 1 -- 172.21.15.40:0/3668577342 learned_addr learned my addr 172.21.15.40:0/3668577342 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2ae2a00640 1 --2- 172.21.15.40:0/3668577342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef2793ff0 0x557ef27dc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2adbe00640 1 -- 172.21.15.40:0/3668577342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27de240 msgr2=0x557ef27c05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2adbe00640 1 --2- 172.21.15.40:0/3668577342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27de240 0x557ef27c05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2adbe00640 1 -- 172.21.15.40:0/3668577342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef2793ff0 msgr2=0x557ef27dc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2adbe00640 1 --2- 172.21.15.40:0/3668577342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef2793ff0 0x557ef27dc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2adbe00640 1 -- 172.21.15.40:0/3668577342 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ef25ae540 con 0x557ef27dd920 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2ae2a00640 1 --2- 172.21.15.40:0/3668577342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef2793ff0 0x557ef27dc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2adbe00640 1 --2- 172.21.15.40:0/3668577342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27dd920 0x557ef27dca40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2acc054e80 tx=0x7f2acc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2aca5075e5e03ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:18.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2ad2a00640 1 -- 172.21.15.40:0/3668577342 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc053480 con 0x557ef27dd920 2024-03-24T00:31:18.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2ad2a00640 1 -- 172.21.15.40:0/3668577342 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2acc0517f0 con 0x557ef27dd920 2024-03-24T00:31:18.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2ad2a00640 1 -- 172.21.15.40:0/3668577342 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2acc05d600 con 0x557ef27dd920 2024-03-24T00:31:18.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 -- 172.21.15.40:0/3668577342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27dd920 msgr2=0x557ef27dca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/3668577342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27dd920 0x557ef27dca40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2acc054e80 tx=0x7f2acc014000 comp rx=0 tx=0).stop 2024-03-24T00:31:18.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 -- 172.21.15.40:0/3668577342 shutdown_connections 2024-03-24T00:31:18.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/3668577342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27de240 0x557ef27c05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/3668577342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef27dd920 0x557ef27dca40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/3668577342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef2793ff0 0x557ef27dc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 -- 172.21.15.40:0/3668577342 >> 172.21.15.40:0/3668577342 conn(0x557ef278ffa0 msgr2=0x557ef27eaf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:18.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 -- 172.21.15.40:0/3668577342 shutdown_connections 2024-03-24T00:31:18.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.098+0000 7f2aecd9c000 1 -- 172.21.15.40:0/3668577342 wait complete. 2024-03-24T00:31:18.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2aecd9c000 1 -- start start 2024-03-24T00:31:18.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2aecd9c000 1 Processor -- start 2024-03-24T00:31:18.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2aecd9c000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 0x557ef27f0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2aecd9c000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef27de240 0x557ef27f1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2aecd9c000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27f1970 0x557ef27f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2aecd9c000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ef2589a80 con 0x557ef2793ff0 2024-03-24T00:31:18.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2aecd9c000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae4201aa0 con 0x557ef27de240 2024-03-24T00:31:18.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2aecd9c000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ef269e700 con 0x557ef27f1970 2024-03-24T00:31:18.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2aecd9c000 10 client.0 _ll_register_callbacks cb 0x7f2ad40079f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:31:18.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2ae2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 0x557ef27f0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2ae2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 0x557ef27f0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35122/0 (socket says 172.21.15.40:35122) 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2ae2a00640 1 -- 172.21.15.40:0/879659655 learned_addr learned my addr 172.21.15.40:0/879659655 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2adbe00640 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef27de240 0x557ef27f1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2ae3400640 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27f1970 0x557ef27f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2ae2a00640 1 -- 172.21.15.40:0/879659655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27f1970 msgr2=0x557ef27f1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2ae2a00640 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27f1970 0x557ef27f1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2ae2a00640 1 -- 172.21.15.40:0/879659655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef27de240 msgr2=0x557ef27f1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2ae2a00640 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef27de240 0x557ef27f1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2ae2a00640 1 -- 172.21.15.40:0/879659655 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557ef25754b0 con 0x557ef2793ff0 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2ae2a00640 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 0x557ef27f0ef0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4000eb0 tx=0x7f2ad4001160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39ae0910bb1d1ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.102+0000 7f2ac7e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ac7e00640 1 -- 172.21.15.40:0/879659655 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4001bf0 con 0x557ef2793ff0 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ac7e00640 1 -- 172.21.15.40:0/879659655 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ad4003c20 con 0x557ef2793ff0 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 -- 172.21.15.40:0/879659655 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2acc0517f0 con 0x557ef2793ff0 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ac7e00640 1 -- 172.21.15.40:0/879659655 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ad4003e40 con 0x557ef2793ff0 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 10 client.4614 fetch_fsmap learned FSMap version 8 2024-03-24T00:31:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 -- 172.21.15.40:0/879659655 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x557ef25882f0 con 0x557ef2793ff0 2024-03-24T00:31:18.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ac7e00640 1 -- 172.21.15.40:0/879659655 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2acc0517f0 con 0x557ef2793ff0 2024-03-24T00:31:18.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ac7e00640 1 -- 172.21.15.40:0/879659655 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 8) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2ad4025070 con 0x557ef2793ff0 2024-03-24T00:31:18.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 10 client.4614 fetch_fsmap finished waiting for FSMap version 8 2024-03-24T00:31:18.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 10 client.4614 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:31:18.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 20 client.4614 populate_metadata read hostname 'smithi040' 2024-03-24T00:31:18.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.40:6834/4139303536,v1:172.21.15.40:6835/4139303536] conn(0x557ef27f2290 0x557ef284b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 4 client.4614 mds_command: new command op to 4264 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:31:18.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 -- 172.21.15.40:0/879659655 --> [v2:172.21.15.40:6834/4139303536,v1:172.21.15.40:6835/4139303536] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557ef25eebc0 con 0x557ef27f2290 2024-03-24T00:31:18.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2adbe00640 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.40:6834/4139303536,v1:172.21.15.40:6835/4139303536] conn(0x557ef27f2290 0x557ef284b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2adbe00640 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.40:6834/4139303536,v1:172.21.15.40:6835/4139303536] conn(0x557ef27f2290 0x557ef284b1f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c95a7e6082d7486c server_cookie=4d368c2f3b81e862 in_seq=0 out_seq=0 2024-03-24T00:31:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ac7e00640 10 client.4614 ms_handle_connect on v2:172.21.15.40:6834/4139303536 2024-03-24T00:31:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ac7e00640 1 -- 172.21.15.40:0/879659655 <== mds.0 v2:172.21.15.40:6834/4139303536 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x557ef25eebc0 con 0x557ef27f2290 2024-03-24T00:31:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ac7e00640 10 client.4614 handle_command_reply: tid=0 2024-03-24T00:31:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 client.4614 shutdown 2024-03-24T00:31:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 -- 172.21.15.40:0/879659655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 msgr2=0x557ef27f0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 0x557ef27f0ef0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4000eb0 tx=0x7f2ad4001160 comp rx=0 tx=0).stop 2024-03-24T00:31:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 -- 172.21.15.40:0/879659655 shutdown_connections 2024-03-24T00:31:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.40:6834/4139303536,v1:172.21.15.40:6835/4139303536] conn(0x557ef27f2290 0x557ef284b1f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557ef27f1970 0x557ef27f1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557ef27de240 0x557ef27f1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ac7e00640 0 client.4614 ms_handle_reset on v2:172.21.15.40:6834/4139303536 2024-03-24T00:31:18.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 --2- 172.21.15.40:0/879659655 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557ef2793ff0 0x557ef27f0ef0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 -- 172.21.15.40:0/879659655 >> 172.21.15.40:0/879659655 conn(0x557ef278ffa0 msgr2=0x557ef27ea680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:18.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 -- 172.21.15.40:0/879659655 shutdown_connections 2024-03-24T00:31:18.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 1 -- 172.21.15.40:0/879659655 wait complete. 2024-03-24T00:31:18.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2aecd9c000 20 client.4614 trim_cache size 0 max 16384 2024-03-24T00:31:18.113 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:31:18.113 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:31:18.113 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:31:18.113 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4264, 2024-03-24T00:31:18.113 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:31:18.114 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:31:18.114 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:31:18.114 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 18.060421929, 2024-03-24T00:31:18.114 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 8, 2024-03-24T00:31:18.114 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 20, 2024-03-24T00:31:18.114 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 20, 2024-03-24T00:31:18.114 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 24.905922106999999 2024-03-24T00:31:18.114 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:31:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ae9800640 1 -- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2ac0067660 msgr2=0x7f2ac0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ae9800640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2ac0067660 0x7f2ac0069b20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2adc0af000 tx=0x7f2adc004ac0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ae9800640 1 -- 172.21.15.40:0/3912327772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ae415efd0 msgr2=0x7f2ae4156800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ae9800640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ae415efd0 0x7f2ae4156800 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00c450 tx=0x7f2acc00c910 comp rx=0 tx=0).stop 2024-03-24T00:31:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ae9800640 1 -- 172.21.15.40:0/3912327772 shutdown_connections 2024-03-24T00:31:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ae9800640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2ac0067660 0x7f2ac0069b20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ae9800640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ae415f3b0 0x7f2ae4156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ae9800640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ae415efd0 0x7f2ae4156800 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ae9800640 1 --2- 172.21.15.40:0/3912327772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ae40ae6d0 0x7f2ae40c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ae9800640 1 -- 172.21.15.40:0/3912327772 >> 172.21.15.40:0/3912327772 conn(0x7f2ae40b1f00 msgr2=0x7f2ae40b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:18.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ae9800640 1 -- 172.21.15.40:0/3912327772 shutdown_connections 2024-03-24T00:31:18.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.106+0000 7f2ae9800640 1 -- 172.21.15.40:0/3912327772 wait complete. 2024-03-24T00:31:18.137 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-24T00:31:18.169 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-24T00:31:18.169 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-24T00:31:18.170 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-24T00:31:18.170 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:31:18.170 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:31:18.170 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-24T00:31:18.171 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:31:18.171 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-24T00:31:18.171 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-24T00:31:18.171 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:31:18.171 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T00:31:18.172 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-24T00:31:18.172 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:31:18.172 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-24T00:31:18.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:18.172 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-03-24T00:31:18.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:18.183 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link delete ceph-brx 2024-03-24T00:31:18.249 INFO:teuthology.orchestra.run.smithi053.stderr:Cannot find device "ceph-brx" 2024-03-24T00:31:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:31:18.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:18.252 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-24T00:31:18.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:18.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-03-24T00:31:18.325 INFO:teuthology.orchestra.run.smithi040.stderr:Cannot find device "ceph-brx" 2024-03-24T00:31:18.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:31:18.327 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-24T00:31:18.328 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:31:18.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.506+0000 7fc4af000640 1 Processor -- start 2024-03-24T00:31:18.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.506+0000 7fc4af000640 1 -- start start 2024-03-24T00:31:18.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.510+0000 7fc4af000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc4a8161890 0x7fc4a8162430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.510+0000 7fc4af000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a8162b10 con 0x7fc4a81614b0 2024-03-24T00:31:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.510+0000 7fc4af000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a80cf1d0 con 0x7fc4a80cfd10 2024-03-24T00:31:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.510+0000 7fc4af000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a80cf540 con 0x7fc4a8161890 2024-03-24T00:31:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.510+0000 7fc49fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc4a8161890 0x7fc4a8162430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.510+0000 7fc49fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc4a8161890 0x7fc4a8162430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35136/0 (socket says 172.21.15.40:35136) 2024-03-24T00:31:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.510+0000 7fc49fe00640 1 -- 172.21.15.40:0/2599259566 learned_addr learned my addr 172.21.15.40:0/2599259566 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.510+0000 7fc4a7e00640 1 --1- 172.21.15.40:0/2599259566 >> v1:172.21.15.40:6789/0 conn(0x7fc4a81614b0 0x7fc4a8161d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43140/0 (socket says 172.21.15.40:43140) 2024-03-24T00:31:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.510+0000 7fc49fe00640 1 -- 172.21.15.40:0/2599259566 >> v1:172.21.15.53:6789/0 conn(0x7fc4a80cfd10 legacy=0x7fc4a806f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.510+0000 7fc49fe00640 1 -- 172.21.15.40:0/2599259566 >> v1:172.21.15.40:6789/0 conn(0x7fc4a81614b0 legacy=0x7fc4a8161d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.510+0000 7fc4a7400640 1 -- 172.21.15.40:0/2599259566 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234225774 0 0) 0x7fc4a8162b10 con 0x7fc4a81614b0 2024-03-24T00:31:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.510+0000 7fc49fe00640 1 -- 172.21.15.40:0/2599259566 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a80ceee0 con 0x7fc4a8161890 2024-03-24T00:31:18.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc49fe00640 1 --2- 172.21.15.40:0/2599259566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4a8161890 0x7fc4a8162430 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc490009870 tx=0x7fc49001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5a4f8fbed32457 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:18.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4a7400640 1 -- 172.21.15.40:0/2599259566 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4900a9640 con 0x7fc4a8161890 2024-03-24T00:31:18.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4a7400640 1 -- 172.21.15.40:0/2599259566 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4900a97e0 con 0x7fc4a8161890 2024-03-24T00:31:18.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4a7400640 1 -- 172.21.15.40:0/2599259566 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4900a9bb0 con 0x7fc4a8161890 2024-03-24T00:31:18.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 -- 172.21.15.40:0/2599259566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4a8161890 msgr2=0x7fc4a8162430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 --2- 172.21.15.40:0/2599259566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4a8161890 0x7fc4a8162430 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc490009870 tx=0x7fc49001c560 comp rx=0 tx=0).stop 2024-03-24T00:31:18.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 -- 172.21.15.40:0/2599259566 shutdown_connections 2024-03-24T00:31:18.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 --2- 172.21.15.40:0/2599259566 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4a8161890 0x7fc4a8162430 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 -- 172.21.15.40:0/2599259566 >> 172.21.15.40:0/2599259566 conn(0x7fc4a80d1c80 msgr2=0x7fc4a80d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:18.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 -- 172.21.15.40:0/2599259566 shutdown_connections 2024-03-24T00:31:18.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 -- 172.21.15.40:0/2599259566 wait complete. 2024-03-24T00:31:18.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 Processor -- start 2024-03-24T00:31:18.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 -- start start 2024-03-24T00:31:18.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4a80cfd10 0x7fc4a815f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc4a81614b0 0x7fc4a815dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4a8161890 0x7fc4a81602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a816c400 con 0x7fc4a8161890 2024-03-24T00:31:18.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a816c280 con 0x7fc4a80cfd10 2024-03-24T00:31:18.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.514+0000 7fc4af000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a816c580 con 0x7fc4a81614b0 2024-03-24T00:31:18.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4ad000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4a8161890 0x7fc4a81602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4a7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4a80cfd10 0x7fc4a815f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4ad000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4a8161890 0x7fc4a81602d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35138/0 (socket says 172.21.15.40:35138) 2024-03-24T00:31:18.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4ad000640 1 -- 172.21.15.40:0/651686926 learned_addr learned my addr 172.21.15.40:0/651686926 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:18.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc49fe00640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc4a81614b0 0x7fc4a815dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4ad000640 1 -- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc4a81614b0 msgr2=0x7fc4a815dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4ad000640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc4a81614b0 0x7fc4a815dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4ad000640 1 -- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4a80cfd10 msgr2=0x7fc4a815f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4ad000640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4a80cfd10 0x7fc4a815f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4ad000640 1 -- 172.21.15.40:0/651686926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a8160810 con 0x7fc4a8161890 2024-03-24T00:31:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4ad000640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4a8161890 0x7fc4a81602d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000c730 tx=0x7fc4a000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4a4c00640 1 -- 172.21.15.40:0/651686926 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0003b80 con 0x7fc4a8161890 2024-03-24T00:31:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4a4c00640 1 -- 172.21.15.40:0/651686926 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4a0003d20 con 0x7fc4a8161890 2024-03-24T00:31:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4af000640 1 -- 172.21.15.40:0/651686926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a815e230 con 0x7fc4a8161890 2024-03-24T00:31:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4a4c00640 1 -- 172.21.15.40:0/651686926 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a0005310 con 0x7fc4a8161890 2024-03-24T00:31:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4af000640 1 -- 172.21.15.40:0/651686926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a815e7f0 con 0x7fc4a8161890 2024-03-24T00:31:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4af000640 1 -- 172.21.15.40:0/651686926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a8068170 con 0x7fc4a8161890 2024-03-24T00:31:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.518+0000 7fc4a4c00640 1 -- 172.21.15.40:0/651686926 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc4a00056c0 con 0x7fc4a8161890 2024-03-24T00:31:18.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.522+0000 7fc4a4c00640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc484067690 0x7fc484069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:18.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.522+0000 7fc4a4c00640 1 -- 172.21.15.40:0/651686926 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc4a0085800 con 0x7fc4a8161890 2024-03-24T00:31:18.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.522+0000 7fc4a7e00640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc484067690 0x7fc484069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:18.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.522+0000 7fc4a7e00640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc484067690 0x7fc484069b50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc498000f70 tx=0x7fc498007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:18.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.526+0000 7fc4a4c00640 1 -- 172.21.15.40:0/651686926 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc4a0055e20 con 0x7fc4a8161890 2024-03-24T00:31:18.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.722+0000 7fc4af000640 1 -- 172.21.15.40:0/651686926 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc4a80d0ed0 con 0x7fc4a8161890 2024-03-24T00:31:18.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.722+0000 7fc4a4c00640 1 -- 172.21.15.40:0/651686926 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc4a0059ec0 con 0x7fc4a8161890 2024-03-24T00:31:18.726 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:31:18.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.726+0000 7fc4af000640 1 -- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc484067690 msgr2=0x7fc484069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.726+0000 7fc4af000640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc484067690 0x7fc484069b50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc498000f70 tx=0x7fc498007240 comp rx=0 tx=0).stop 2024-03-24T00:31:18.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.726+0000 7fc4af000640 1 -- 172.21.15.40:0/651686926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4a8161890 msgr2=0x7fc4a81602d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:18.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.726+0000 7fc4af000640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4a8161890 0x7fc4a81602d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc4a000c730 tx=0x7fc4a000cbf0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.726+0000 7fc4af000640 1 -- 172.21.15.40:0/651686926 shutdown_connections 2024-03-24T00:31:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.726+0000 7fc4af000640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc484067690 0x7fc484069b50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.726+0000 7fc4af000640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc4a8161890 0x7fc4a81602d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.726+0000 7fc4af000640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc4a81614b0 0x7fc4a815dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.726+0000 7fc4af000640 1 --2- 172.21.15.40:0/651686926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4a80cfd10 0x7fc4a815f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.726+0000 7fc4af000640 1 -- 172.21.15.40:0/651686926 >> 172.21.15.40:0/651686926 conn(0x7fc4a80d1c80 msgr2=0x7fc4a81559e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.726+0000 7fc4af000640 1 -- 172.21.15.40:0/651686926 shutdown_connections 2024-03-24T00:31:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:18.726+0000 7fc4af000640 1 -- 172.21.15.40:0/651686926 wait complete. 2024-03-24T00:31:18.747 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:31:18.747 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:31:18.747 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-03-24T00:31:18.747 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:31:18.747 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:31:18.748 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:31:18.748 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:31:18.748 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:31:18.748 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:31:18.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:18.748 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-03-24T00:31:18.757 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:31:18.757 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:31:18.757 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:31:18.757 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:18.757 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-24T00:31:18.757 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:18.757 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:31:18.757 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:18.757 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-03-24T00:31:18.757 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:31:18.757 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:18.757 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42471sec preferred_lft 42471sec 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 02:42:9e:ab:82:f9 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-24T00:31:18.758 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:18.760 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T00:31:18.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:18.760 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:31:18.760 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:31:18.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add name ceph-brx type bridge 2024-03-24T00:31:18.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr flush dev ceph-brx 2024-03-24T00:31:18.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set ceph-brx up 2024-03-24T00:31:18.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T00:31:18.760 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:31:18.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:18.872 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:31:18.890 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-03-24T00:31:18.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:18.892 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-03-24T00:31:18.962 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-03-24T00:31:18.962 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:31:18.962 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-24T00:31:18.963 INFO:teuthology.orchestra.run.smithi040.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-24T00:31:18.963 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-24T00:31:18.963 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:31:18.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:18.963 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:31:18.963 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:31:18.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T00:31:18.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T00:31:18.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T00:31:18.964 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:31:19.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:19.052 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-24T00:31:19.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:19.061 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list-id 2024-03-24T00:31:19.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:19.115 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:31:19.115 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:31:19.116 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:31:19.116 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T00:31:19.116 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:31:19.199 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T00:31:19.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:19.199 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:31:19.199 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:31:19.199 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T00:31:19.199 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T00:31:19.199 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T00:31:19.199 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T00:31:19.199 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T00:31:19.199 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:31:19.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:19.382 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:31:19.382 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:31:19.382 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-03-24T00:31:19.382 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T00:31:19.382 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:31:19.424 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T00:31:19.424 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:31:19.424 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:31:19.475 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:31:19.475 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:31:19.475 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:31:19.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:31:19.587 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:31:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:31:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:31:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:31:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:31:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:31:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:31:19.641 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:31:19.642 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:31:19.642 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:31:19.642 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:31:19.642 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:31:19.642 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:31:19.642 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:31:19.642 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:31:19.642 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:31:19.642 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:31:19.642 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:31:19.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:19.643 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:31:19.695 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T00:31:19.695 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T00:31:19.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:31:19.763 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:31:19.816 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:31:19.816 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:31:19.816 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:31:19.816 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:31:19.817 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:31:19.818 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:31:19.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:19.819 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:31:19.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:2024-03-24T00:31:19.846+0000 7f950a520580 -1 init, newargv = 0x5624dd8b1610 newargc=15 2024-03-24T00:31:19.848 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[34310]: starting ceph client 2024-03-24T00:31:19.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[34310]: starting fuse 2024-03-24T00:31:20.958 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:31:20.979 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:31:21.032 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:31:21.032 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:31:21.032 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:31:21.032 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:31:21.032 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:31:21.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:31:21.034 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:31:21.034 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:31:21.034 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:31:21.034 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:31:21.034 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:31:21.034 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:31:21.034 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:31:21.034 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:31:21.034 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:31:21.034 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:31:21.034 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:31:21.034 INFO:teuthology.orchestra.run.smithi040.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-24T00:31:21.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:21.035 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:31:21.086 INFO:teuthology.orchestra.run.smithi040.stdout:43 2024-03-24T00:31:21.087 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-24T00:31:21.087 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:31:21.087 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-03-24T00:31:21.087 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-03-24T00:31:21.087 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-03-24T00:31:21.087 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-03-24T00:31:21.087 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:31:21.087 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-03-24T00:31:21.087 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T00:31:21.087 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-03-24T00:31:21.087 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-03-24T00:31:21.088 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:31:21.089 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:31:21.089 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.0")) 2024-03-24T00:31:21.089 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-03-24T00:31:21.221 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.0.34310.asok 2024-03-24T00:31:21.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:31:21.234 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.34310.asok 2024-03-24T00:31:21.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:21.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.34310.asok status 2024-03-24T00:31:21.407 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:31:21.407 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-03-24T00:31:21.407 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:31:21.407 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "0", 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "34310", 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4632, 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4632 2024-03-24T00:31:21.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:31:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:31:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:31:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:31:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 102167829 2024-03-24T00:31:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:31:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:31:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:31:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:31:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:31:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 102167829 2024-03-24T00:31:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:31:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.4632 192.168.0.1:0/102167829", 2024-03-24T00:31:21.410 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "192.168.0.1:0/102167829", 2024-03-24T00:31:21.410 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-03-24T00:31:21.410 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 8, 2024-03-24T00:31:21.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 20, 2024-03-24T00:31:21.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:31:21.410 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-03-24T00:31:21.410 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-03-24T00:31:21.410 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:31:21.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:31:21.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.558+0000 7f9a05000640 1 Processor -- start 2024-03-24T00:31:21.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.558+0000 7f9a05000640 1 -- start start 2024-03-24T00:31:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f9a05000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a0015f600 0x7f9a000b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f9a05000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a000d2c70 con 0x7f9a000d2830 2024-03-24T00:31:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f9a05000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a000b1e50 con 0x7f9a000af170 2024-03-24T00:31:21.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f9a05000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a000b3630 con 0x7f9a0015f600 2024-03-24T00:31:21.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f99f7e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9a000d2830 0x7f9a000aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43142/0 (socket says 172.21.15.40:43142) 2024-03-24T00:31:21.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f99f7e00640 1 -- 172.21.15.40:0/2597706432 learned_addr learned my addr 172.21.15.40:0/2597706432 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:21.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f99fde00640 1 --2- 172.21.15.40:0/2597706432 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a0015f600 0x7f9a000b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f99fd400640 1 -- 172.21.15.40:0/2597706432 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031912895 0 0) 0x7f9a000d2c70 con 0x7f9a000d2830 2024-03-24T00:31:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f99fd400640 1 -- 172.21.15.40:0/2597706432 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e4003530 con 0x7f9a000d2830 2024-03-24T00:31:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f99fd400640 1 -- 172.21.15.40:0/2597706432 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842069380 0 0) 0x7f9a000b1e50 con 0x7f9a000af170 2024-03-24T00:31:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f99fd400640 1 -- 172.21.15.40:0/2597706432 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a000d2c70 con 0x7f9a000af170 2024-03-24T00:31:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f99fd400640 1 -- 172.21.15.40:0/2597706432 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1778909802 0 0) 0x7f99e4003530 con 0x7f9a000d2830 2024-03-24T00:31:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f99fde00640 1 -- 172.21.15.40:0/2597706432 >> v1:172.21.15.53:6789/0 conn(0x7f9a000af170 legacy=0x7f9a00165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f99fde00640 1 -- 172.21.15.40:0/2597706432 >> v1:172.21.15.40:6789/0 conn(0x7f9a000d2830 legacy=0x7f9a000aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.562+0000 7f99fde00640 1 -- 172.21.15.40:0/2597706432 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a000b20f0 con 0x7f9a0015f600 2024-03-24T00:31:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f99fde00640 1 --2- 172.21.15.40:0/2597706432 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a0015f600 0x7f9a000b2910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f99f00097f0 tx=0x7f99f001ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41bfbcd71f0fe530 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f99fd400640 1 -- 172.21.15.40:0/2597706432 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f00b5070 con 0x7f9a0015f600 2024-03-24T00:31:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f99fd400640 1 -- 172.21.15.40:0/2597706432 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f00a8670 con 0x7f9a0015f600 2024-03-24T00:31:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f99fd400640 1 -- 172.21.15.40:0/2597706432 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8001680 con 0x7f9a0015f600 2024-03-24T00:31:21.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f9a05000640 1 -- 172.21.15.40:0/2597706432 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a0015f600 msgr2=0x7f9a000b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:21.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f9a05000640 1 --2- 172.21.15.40:0/2597706432 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a0015f600 0x7f9a000b2910 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f99f00097f0 tx=0x7f99f001ebf0 comp rx=0 tx=0).stop 2024-03-24T00:31:21.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f9a05000640 1 -- 172.21.15.40:0/2597706432 shutdown_connections 2024-03-24T00:31:21.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f9a05000640 1 --2- 172.21.15.40:0/2597706432 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a0015f600 0x7f9a000b2910 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:21.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f9a05000640 1 -- 172.21.15.40:0/2597706432 >> 172.21.15.40:0/2597706432 conn(0x7f9a000d1d10 msgr2=0x7f9a000d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f9a05000640 1 -- 172.21.15.40:0/2597706432 shutdown_connections 2024-03-24T00:31:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f9a05000640 1 -- 172.21.15.40:0/2597706432 wait complete. 2024-03-24T00:31:21.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f9a05000640 1 Processor -- start 2024-03-24T00:31:21.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.566+0000 7f9a05000640 1 -- start start 2024-03-24T00:31:21.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f9a05000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a000af170 0x7f9a00157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:21.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f9a05000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a000d2830 0x7f9a001f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:21.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f9a05000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0015f600 0x7f9a000c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:21.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f9a05000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a000b2fa0 con 0x7f9a0015f600 2024-03-24T00:31:21.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f9a05000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a000b2740 con 0x7f9a000af170 2024-03-24T00:31:21.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f9a05000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0015f9e0 con 0x7f9a000d2830 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99fe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0015f600 0x7f9a000c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99fe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0015f600 0x7f9a000c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35142/0 (socket says 172.21.15.40:35142) 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99fe800640 1 -- 172.21.15.40:0/2269440639 learned_addr learned my addr 172.21.15.40:0/2269440639 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99f7e00640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a000d2830 0x7f9a001f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99fde00640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a000af170 0x7f9a00157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99fe800640 1 -- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a000d2830 msgr2=0x7f9a001f1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99fe800640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a000d2830 0x7f9a001f1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99fe800640 1 -- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a000af170 msgr2=0x7f9a00157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99fe800640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a000af170 0x7f9a00157450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99fe800640 1 -- 172.21.15.40:0/2269440639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a001e4b40 con 0x7f9a0015f600 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99fde00640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a000af170 0x7f9a00157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99fe800640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0015f600 0x7f9a000c97f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f99f800c730 tx=0x7f99f800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99f6000640 1 -- 172.21.15.40:0/2269440639 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f8003b80 con 0x7f9a0015f600 2024-03-24T00:31:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99f6000640 1 -- 172.21.15.40:0/2269440639 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f8003d20 con 0x7f9a0015f600 2024-03-24T00:31:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99f6000640 1 -- 172.21.15.40:0/2269440639 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f800aa90 con 0x7f9a0015f600 2024-03-24T00:31:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f9a05000640 1 -- 172.21.15.40:0/2269440639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a001e4e20 con 0x7f9a0015f600 2024-03-24T00:31:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f9a05000640 1 -- 172.21.15.40:0/2269440639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a001e53e0 con 0x7f9a0015f600 2024-03-24T00:31:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f9a05000640 1 -- 172.21.15.40:0/2269440639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a00068170 con 0x7f9a0015f600 2024-03-24T00:31:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.570+0000 7f99f6000640 1 -- 172.21.15.40:0/2269440639 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f99f800ac30 con 0x7f9a0015f600 2024-03-24T00:31:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.574+0000 7f99f6000640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f99dc067660 0x7f99dc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.574+0000 7f99f6000640 1 -- 172.21.15.40:0/2269440639 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99f8085750 con 0x7f9a0015f600 2024-03-24T00:31:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.574+0000 7f99fde00640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f99dc067660 0x7f99dc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:21.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.574+0000 7f99fde00640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f99dc067660 0x7f99dc069b20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f99f00b2000 tx=0x7f99f00aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:21.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.578+0000 7f99f6000640 1 -- 172.21.15.40:0/2269440639 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f99f8056460 con 0x7f9a0015f600 2024-03-24T00:31:21.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.774+0000 7f9a05000640 1 -- 172.21.15.40:0/2269440639 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9a000caff0 con 0x7f9a0015f600 2024-03-24T00:31:21.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.774+0000 7f99f6000640 1 -- 172.21.15.40:0/2269440639 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f99f8055d60 con 0x7f9a0015f600 2024-03-24T00:31:21.779 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:31:21.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.778+0000 7f9a05000640 1 -- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f99dc067660 msgr2=0x7f99dc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:21.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.778+0000 7f9a05000640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f99dc067660 0x7f99dc069b20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f99f00b2000 tx=0x7f99f00aa070 comp rx=0 tx=0).stop 2024-03-24T00:31:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.778+0000 7f9a05000640 1 -- 172.21.15.40:0/2269440639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0015f600 msgr2=0x7f9a000c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.778+0000 7f9a05000640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0015f600 0x7f9a000c97f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f99f800c730 tx=0x7f99f800cbf0 comp rx=0 tx=0).stop 2024-03-24T00:31:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.778+0000 7f9a05000640 1 -- 172.21.15.40:0/2269440639 shutdown_connections 2024-03-24T00:31:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.778+0000 7f9a05000640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f99dc067660 0x7f99dc069b20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.778+0000 7f9a05000640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a0015f600 0x7f9a000c97f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.778+0000 7f9a05000640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a000d2830 0x7f9a001f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.778+0000 7f9a05000640 1 --2- 172.21.15.40:0/2269440639 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a000af170 0x7f9a00157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.778+0000 7f9a05000640 1 -- 172.21.15.40:0/2269440639 >> 172.21.15.40:0/2269440639 conn(0x7f9a000d1d10 msgr2=0x7f9a0006eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.778+0000 7f9a05000640 1 -- 172.21.15.40:0/2269440639 shutdown_connections 2024-03-24T00:31:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:21.778+0000 7f9a05000640 1 -- 172.21.15.40:0/2269440639 wait complete. 2024-03-24T00:31:21.800 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:31:21.800 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:31:21.801 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-03-24T00:31:21.801 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-24T00:31:21.801 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-24T00:31:21.801 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:31:21.801 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:31:21.801 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:31:21.801 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-24T00:31:21.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:21.801 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-03-24T00:31:21.810 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:31:21.810 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:31:21.810 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:31:21.810 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:21.810 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-24T00:31:21.810 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:21.810 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:31:21.810 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42468sec preferred_lft 42468sec 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:21.811 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 02:42:9e:ab:82:f9 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:31:21.812 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-24T00:31:21.813 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:21.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:21.814 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:31:21.872 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-03-24T00:31:21.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:21.875 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-03-24T00:31:21.925 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-03-24T00:31:21.969 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:31:21.969 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-24T00:31:21.969 INFO:teuthology.orchestra.run.smithi040.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-24T00:31:21.969 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-24T00:31:21.969 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:31:21.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:21.970 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:31:21.970 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:31:21.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T00:31:21.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T00:31:21.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T00:31:21.970 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:31:22.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:22.021 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-24T00:31:22.029 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:31:22.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:22.030 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list-id 2024-03-24T00:31:22.078 INFO:teuthology.orchestra.run.smithi040.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-24T00:31:22.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:22.080 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:31:22.080 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:31:22.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:31:22.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-24T00:31:22.080 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:31:22.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:22.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-24T00:31:22.224 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:31:22.224 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:31:22.224 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:31:22.225 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:22.225 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-24T00:31:22.225 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:22.225 INFO:teuthology.orchestra.run.smithi040.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:31:22.225 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 22:37:53:31:f9:69 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T00:31:22.225 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T00:31:22.225 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:22.225 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::2037:53ff:fe31:f969/64 scope link 2024-03-24T00:31:22.225 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:22.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:22.227 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-24T00:31:22.286 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:31:22.286 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:31:22.287 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:31:22.287 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:22.287 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-24T00:31:22.287 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:22.287 INFO:teuthology.orchestra.run.smithi040.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:31:22.287 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 22:37:53:31:f9:69 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T00:31:22.287 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T00:31:22.287 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:22.287 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::2037:53ff:fe31:f969/64 scope link 2024-03-24T00:31:22.287 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:22.288 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-24T00:31:22.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:22.289 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:31:22.289 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:31:22.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-24T00:31:22.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-24T00:31:22.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-24T00:31:22.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-24T00:31:22.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-24T00:31:22.289 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:31:22.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:22.425 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:31:22.426 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:31:22.426 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.1 up 2024-03-24T00:31:22.426 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set dev brx.1 master ceph-brx 2024-03-24T00:31:22.426 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:31:22.464 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-24T00:31:22.464 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:31:22.465 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-24T00:31:22.515 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-24T00:31:22.515 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:31:22.515 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-24T00:31:22.570 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:31:22.631 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:31:22.684 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:31:22.684 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:31:22.684 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:31:22.684 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:31:22.684 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:31:22.684 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:31:22.684 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:31:22.684 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:31:22.685 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:31:22.686 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:31:22.686 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:31:22.686 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:31:22.686 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:31:22.686 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:31:22.686 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:31:22.686 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:31:22.686 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:31:22.686 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:31:22.686 INFO:teuthology.orchestra.run.smithi040.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-24T00:31:22.686 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:31:22.686 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:31:22.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:22.687 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:31:22.738 INFO:teuthology.orchestra.run.smithi040.stdout:43 2024-03-24T00:31:22.739 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-03-24T00:31:22.739 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-24T00:31:22.785 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:31:22.806 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:31:22.860 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:31:22.860 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:31:22.860 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:31:22.861 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:31:22.862 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:31:22.862 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:31:22.862 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:31:22.862 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:31:22.862 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:31:22.862 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:31:22.862 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:31:22.862 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:31:22.862 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:31:22.862 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:31:22.863 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:31:22.863 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:31:22.863 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:31:22.863 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:31:22.863 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:31:22.863 INFO:teuthology.orchestra.run.smithi040.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-24T00:31:22.863 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:31:22.863 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:31:22.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:22.865 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:31:22.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi040.stderr:2024-03-24T00:31:22.886+0000 7f69e9c4d580 -1 init, newargv = 0x561921e1f040 newargc=15 2024-03-24T00:31:22.891 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi040.stderr:ceph-fuse[34450]: starting ceph client 2024-03-24T00:31:22.898 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi040.stderr:ceph-fuse[34450]: starting fuse 2024-03-24T00:31:22.900 INFO:teuthology.orchestra.run.smithi040.stdout:43 2024-03-24T00:31:22.900 INFO:teuthology.orchestra.run.smithi040.stdout:44 2024-03-24T00:31:22.900 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-03-24T00:31:22.900 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:31:22.900 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-03-24T00:31:22.900 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-03-24T00:31:22.900 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-03-24T00:31:22.900 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-03-24T00:31:22.900 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-03-24T00:31:22.901 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-03-24T00:31:22.902 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-03-24T00:31:22.902 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:31:22.902 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:31:22.902 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.2")) 2024-03-24T00:31:22.902 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-03-24T00:31:23.029 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.2.34450.asok 2024-03-24T00:31:23.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:31:23.041 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.34450.asok 2024-03-24T00:31:23.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:23.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.34450.asok status 2024-03-24T00:31:23.215 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:31:23.215 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-03-24T00:31:23.215 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "2", 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "34450", 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4644, 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-03-24T00:31:23.216 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 4644 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.2:0", 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 821665935 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.2:0", 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 821665935 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.4644 192.168.0.2:0/821665935", 2024-03-24T00:31:23.217 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "192.168.0.2:0/821665935", 2024-03-24T00:31:23.218 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-03-24T00:31:23.218 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 8, 2024-03-24T00:31:23.218 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 20, 2024-03-24T00:31:23.218 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:31:23.218 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-03-24T00:31:23.218 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-03-24T00:31:23.218 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:31:23.231 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:31:23.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f198c600640 1 Processor -- start 2024-03-24T00:31:23.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f198c600640 1 -- start start 2024-03-24T00:31:23.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f198c600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b2aa0 0x7f19840d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:23.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f198c600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19840b2870 con 0x7f19840b2490 2024-03-24T00:31:23.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f198c600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19840b3190 con 0x7f19840d0a80 2024-03-24T00:31:23.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f198c600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f198406b6d0 con 0x7f19840b2aa0 2024-03-24T00:31:23.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1989c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f19840b2490 0x7f19840cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43148/0 (socket says 172.21.15.40:43148) 2024-03-24T00:31:23.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1989c00640 1 -- 172.21.15.40:0/391797254 learned_addr learned my addr 172.21.15.40:0/391797254 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:23.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1989200640 1 --2- 172.21.15.40:0/391797254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b2aa0 0x7f19840d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:23.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222372285 0 0) 0x7f19840b2870 con 0x7f19840b2490 2024-03-24T00:31:23.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f196c003530 con 0x7f19840b2490 2024-03-24T00:31:23.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185170482 0 0) 0x7f19840b3190 con 0x7f19840d0a80 2024-03-24T00:31:23.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19840b2870 con 0x7f19840d0a80 2024-03-24T00:31:23.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1685756616 0 0) 0x7f196c003530 con 0x7f19840b2490 2024-03-24T00:31:23.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b2aa0 msgr2=0x7f19840d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:23.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 --2- 172.21.15.40:0/391797254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b2aa0 0x7f19840d0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 >> v1:172.21.15.53:6789/0 conn(0x7f19840d0a80 legacy=0x7f19840cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:23.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19840b3470 con 0x7f19840b2490 2024-03-24T00:31:23.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f1978001680 con 0x7f19840b2490 2024-03-24T00:31:23.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 >> v1:172.21.15.40:6789/0 conn(0x7f19840b2490 legacy=0x7f19840cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:23.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 --2- 172.21.15.40:0/391797254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f196c003b80 0x7f196c003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 --2- 172.21.15.40:0/391797254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c004810 0x7f196c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 --2- 172.21.15.40:0/391797254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b2aa0 0x7f196c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f198406b6d0 con 0x7f196c004810 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196c00ba80 con 0x7f196c003b80 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196c00bc50 con 0x7f19840b2aa0 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1989200640 1 --2- 172.21.15.40:0/391797254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c004810 0x7f196c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f1989c00640 1 --2- 172.21.15.40:0/391797254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f196c003b80 0x7f196c003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.370+0000 7f198a600640 1 --2- 172.21.15.40:0/391797254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b2aa0 0x7f196c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f1989200640 1 -- 172.21.15.40:0/391797254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b2aa0 msgr2=0x7f196c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f1989200640 1 --2- 172.21.15.40:0/391797254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b2aa0 0x7f196c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f1989200640 1 -- 172.21.15.40:0/391797254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f196c003b80 msgr2=0x7f196c003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f1989200640 1 --2- 172.21.15.40:0/391797254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f196c003b80 0x7f196c003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f1989200640 1 -- 172.21.15.40:0/391797254 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f196c003530 con 0x7f196c004810 2024-03-24T00:31:23.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f1989200640 1 --2- 172.21.15.40:0/391797254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c004810 0x7f196c004c90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1970009ae0 tx=0x7f197001ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271de3ea1a454b10 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:23.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19700b2070 con 0x7f196c004810 2024-03-24T00:31:23.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19700a5600 con 0x7f196c004810 2024-03-24T00:31:23.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f1983e00640 1 -- 172.21.15.40:0/391797254 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19700096b0 con 0x7f196c004810 2024-03-24T00:31:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f198c600640 1 -- 172.21.15.40:0/391797254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c004810 msgr2=0x7f196c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f198c600640 1 --2- 172.21.15.40:0/391797254 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c004810 0x7f196c004c90 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1970009ae0 tx=0x7f197001ca10 comp rx=0 tx=0).stop 2024-03-24T00:31:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f198a600640 1 -- 172.21.15.40:0/391797254 reap_dead start 2024-03-24T00:31:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f198c600640 1 -- 172.21.15.40:0/391797254 shutdown_connections 2024-03-24T00:31:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f198c600640 1 -- 172.21.15.40:0/391797254 >> 172.21.15.40:0/391797254 conn(0x7f19840d1d10 msgr2=0x7f19840d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:23.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f198c600640 1 -- 172.21.15.40:0/391797254 shutdown_connections 2024-03-24T00:31:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.374+0000 7f198c600640 1 -- 172.21.15.40:0/391797254 wait complete. 2024-03-24T00:31:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f198c600640 1 Processor -- start 2024-03-24T00:31:23.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f198c600640 1 -- start start 2024-03-24T00:31:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f198c600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c004810 0x7f19841f4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f198c600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f198409e8c0 0x7f19841f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f198c600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f198409c6a0 0x7f198409cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f198c600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19840d06c0 con 0x7f196c004810 2024-03-24T00:31:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f198c600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19840d0540 con 0x7f198409e8c0 2024-03-24T00:31:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f198c600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19840d0840 con 0x7f198409c6a0 2024-03-24T00:31:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1989c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c004810 0x7f19841f4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1989c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c004810 0x7f19841f4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:35164/0 (socket says 172.21.15.40:35164) 2024-03-24T00:31:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1989c00640 1 -- 172.21.15.40:0/3496105418 learned_addr learned my addr 172.21.15.40:0/3496105418 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1989200640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f198409e8c0 0x7f19841f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:23.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f198a600640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f198409c6a0 0x7f198409cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1989c00640 1 -- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f198409c6a0 msgr2=0x7f198409cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1989c00640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f198409c6a0 0x7f198409cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1989c00640 1 -- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f198409e8c0 msgr2=0x7f19841f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1989c00640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f198409e8c0 0x7f19841f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1989c00640 1 -- 172.21.15.40:0/3496105418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198409ed60 con 0x7f196c004810 2024-03-24T00:31:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f198a600640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f198409c6a0 0x7f198409cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1989c00640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c004810 0x7f19841f4400 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f19780072c0 tx=0x7f1978007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1981600640 1 -- 172.21.15.40:0/3496105418 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197800f9c0 con 0x7f196c004810 2024-03-24T00:31:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1981600640 1 -- 172.21.15.40:0/3496105418 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1978018070 con 0x7f196c004810 2024-03-24T00:31:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f198c600640 1 -- 172.21.15.40:0/3496105418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f198409f040 con 0x7f196c004810 2024-03-24T00:31:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f1981600640 1 -- 172.21.15.40:0/3496105418 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f197800fde0 con 0x7f196c004810 2024-03-24T00:31:23.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.378+0000 7f198c600640 1 -- 172.21.15.40:0/3496105418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f198409f600 con 0x7f196c004810 2024-03-24T00:31:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.382+0000 7f198c600640 1 -- 172.21.15.40:0/3496105418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1984068170 con 0x7f196c004810 2024-03-24T00:31:23.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.382+0000 7f1981600640 1 -- 172.21.15.40:0/3496105418 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f197801d460 con 0x7f196c004810 2024-03-24T00:31:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.382+0000 7f1981600640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1964067660 0x7f1964069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:23.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.382+0000 7f1981600640 1 -- 172.21.15.40:0/3496105418 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1978085c10 con 0x7f196c004810 2024-03-24T00:31:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.382+0000 7f1989200640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1964067660 0x7f1964069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:23.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.386+0000 7f1989200640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1964067660 0x7f1964069b20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f19700af000 tx=0x7f19700046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:23.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.386+0000 7f1981600640 1 -- 172.21.15.40:0/3496105418 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f19780567f0 con 0x7f196c004810 2024-03-24T00:31:23.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.586+0000 7f198c600640 1 -- 172.21.15.40:0/3496105418 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f198406bc00 con 0x7f196c004810 2024-03-24T00:31:23.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f1981600640 1 -- 172.21.15.40:0/3496105418 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f19780560f0 con 0x7f196c004810 2024-03-24T00:31:23.591 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:31:23.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f198c600640 1 -- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1964067660 msgr2=0x7f1964069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:23.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f198c600640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1964067660 0x7f1964069b20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f19700af000 tx=0x7f19700046d0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f198c600640 1 -- 172.21.15.40:0/3496105418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c004810 msgr2=0x7f19841f4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:23.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f198c600640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c004810 0x7f19841f4400 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f19780072c0 tx=0x7f1978007780 comp rx=0 tx=0).stop 2024-03-24T00:31:23.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f198c600640 1 -- 172.21.15.40:0/3496105418 shutdown_connections 2024-03-24T00:31:23.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f198c600640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1964067660 0x7f1964069b20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f198c600640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f198409c6a0 0x7f198409cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f198c600640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f198409e8c0 0x7f19841f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f198c600640 1 --2- 172.21.15.40:0/3496105418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f196c004810 0x7f19841f4400 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:23.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f198c600640 1 -- 172.21.15.40:0/3496105418 >> 172.21.15.40:0/3496105418 conn(0x7f19840d1d10 msgr2=0x7f19840b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:23.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f198c600640 1 -- 172.21.15.40:0/3496105418 shutdown_connections 2024-03-24T00:31:23.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:23.590+0000 7f198c600640 1 -- 172.21.15.40:0/3496105418 wait complete. 2024-03-24T00:31:23.613 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:31:23.613 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:31:23.613 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-03-24T00:31:23.613 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-24T00:31:23.614 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T00:31:23.614 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:31:23.614 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:31:23.614 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:31:23.614 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-24T00:31:23.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:23.614 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-03-24T00:31:23.625 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:31:23.625 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:31:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:31:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-03-24T00:31:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:23.626 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:31:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-03-24T00:31:23.626 INFO:teuthology.orchestra.run.smithi053.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:31:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-03-24T00:31:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-24T00:31:23.626 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 42447sec preferred_lft 42447sec 2024-03-24T00:31:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link 2024-03-24T00:31:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:23.627 INFO:teuthology.orchestra.run.smithi053.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:31:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-03-24T00:31:23.627 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:31:23.627 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:23.628 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-03-24T00:31:23.628 INFO:teuthology.orchestra.run.smithi053.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-24T00:31:23.628 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 02:42:ea:7c:60:87 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:23.628 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-24T00:31:23.628 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:23.628 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T00:31:23.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:23.628 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-03-24T00:31:23.628 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-03-24T00:31:23.628 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add name ceph-brx type bridge 2024-03-24T00:31:23.629 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip addr flush dev ceph-brx 2024-03-24T00:31:23.629 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set ceph-brx up 2024-03-24T00:31:23.629 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T00:31:23.629 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-03-24T00:31:23.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:23.742 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:31:23.761 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-03-24T00:31:23.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:23.764 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-03-24T00:31:23.833 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-03-24T00:31:23.834 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:31:23.834 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-24T00:31:23.834 INFO:teuthology.orchestra.run.smithi053.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-24T00:31:23.834 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-24T00:31:23.834 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:31:23.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:23.834 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-03-24T00:31:23.835 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-03-24T00:31:23.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T00:31:23.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T00:31:23.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T00:31:23.835 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-03-24T00:31:23.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:23.932 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-03-24T00:31:23.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:23.942 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list-id 2024-03-24T00:31:23.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:23.997 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-03-24T00:31:23.997 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-03-24T00:31:23.997 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:31:23.997 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-24T00:31:23.997 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-03-24T00:31:24.083 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-24T00:31:24.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:24.083 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-03-24T00:31:24.083 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-03-24T00:31:24.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-24T00:31:24.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T00:31:24.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-24T00:31:24.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-24T00:31:24.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-24T00:31:24.084 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-03-24T00:31:24.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:24.239 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-03-24T00:31:24.239 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-03-24T00:31:24.239 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.0 up 2024-03-24T00:31:24.239 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T00:31:24.239 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-03-24T00:31:24.280 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-24T00:31:24.280 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:31:24.280 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-24T00:31:24.336 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-24T00:31:24.336 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:31:24.336 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-24T00:31:24.387 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-03-24T00:31:24.448 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:31:24.501 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-03-24T00:31:24.501 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-03-24T00:31:24.501 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-03-24T00:31:24.501 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-03-24T00:31:24.501 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/run/lock 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/bpf 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/fs/nfsd 2024-03-24T00:31:24.502 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-03-24T00:31:24.503 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-03-24T00:31:24.503 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:31:24.503 INFO:teuthology.orchestra.run.smithi053.stdout:/run/rpc_pipefs 2024-03-24T00:31:24.503 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-03-24T00:31:24.503 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-03-24T00:31:24.503 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-4 2024-03-24T00:31:24.503 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-5 2024-03-24T00:31:24.503 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-6 2024-03-24T00:31:24.503 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-7 2024-03-24T00:31:24.503 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-03-24T00:31:24.503 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:31:24.503 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:31:24.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:24.504 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-03-24T00:31:24.555 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T00:31:24.556 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-24T00:31:24.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-03-24T00:31:24.619 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:31:24.669 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-03-24T00:31:24.669 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-03-24T00:31:24.669 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-03-24T00:31:24.669 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/run/lock 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/bpf 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-03-24T00:31:24.670 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/fs/nfsd 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/run/rpc_pipefs 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-4 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-5 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-6 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-7 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:31:24.671 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:31:24.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:24.672 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-03-24T00:31:24.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi053.stderr:2024-03-24T00:31:24.697+0000 7fcfde6ba580 -1 init, newargv = 0x55ce66b1b710 newargc=15 2024-03-24T00:31:24.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi053.stderr:ceph-fuse[31936]: starting ceph client 2024-03-24T00:31:24.708 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi053.stderr:ceph-fuse[31936]: starting fuse 2024-03-24T00:31:24.710 INFO:teuthology.orchestra.run.smithi053.stdout:43 2024-03-24T00:31:24.710 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-24T00:31:24.710 DEBUG:teuthology.orchestra.run.smithi053:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:31:24.710 DEBUG:teuthology.orchestra.run.smithi053:> import glob 2024-03-24T00:31:24.710 DEBUG:teuthology.orchestra.run.smithi053:> import re 2024-03-24T00:31:24.710 DEBUG:teuthology.orchestra.run.smithi053:> import os 2024-03-24T00:31:24.710 DEBUG:teuthology.orchestra.run.smithi053:> import subprocess 2024-03-24T00:31:24.710 DEBUG:teuthology.orchestra.run.smithi053:> 2024-03-24T00:31:24.710 DEBUG:teuthology.orchestra.run.smithi053:> def _find_admin_socket(client_name): 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> files = glob.glob(asok_path) 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> # Given a non-glob path, it better be there 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> if "*" not in asok_path: 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> assert(len(files) == 1) 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> return files[0] 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> for f in files: 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> contents = proc_f.read() 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> if mountpoint in contents: 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> return f 2024-03-24T00:31:24.711 DEBUG:teuthology.orchestra.run.smithi053:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:31:24.712 DEBUG:teuthology.orchestra.run.smithi053:> 2024-03-24T00:31:24.712 DEBUG:teuthology.orchestra.run.smithi053:> print(_find_admin_socket("client.1")) 2024-03-24T00:31:24.712 DEBUG:teuthology.orchestra.run.smithi053:> ' 2024-03-24T00:31:24.834 INFO:teuthology.orchestra.run.smithi053.stdout:/var/run/ceph/ceph-client.1.31936.asok 2024-03-24T00:31:24.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:31:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:31:24.845 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.31936.asok 2024-03-24T00:31:24.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:24.845 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.31936.asok status 2024-03-24T00:31:25.020 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-03-24T00:31:25.020 INFO:teuthology.orchestra.run.smithi053.stdout: "metadata": { 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "entity_id": "1", 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "hostname": "smithi053", 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "pid": "31936", 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "root": "/" 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_count": 0, 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_pinned_count": 0, 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4499, 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "inst": { 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "name": { 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "client", 2024-03-24T00:31:25.021 INFO:teuthology.orchestra.run.smithi053.stdout: "num": 4499 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 3455473421 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 3455473421 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: "inst_str": "client.4499 192.168.0.1:0/3455473421", 2024-03-24T00:31:25.022 INFO:teuthology.orchestra.run.smithi053.stdout: "addr_str": "192.168.0.1:0/3455473421", 2024-03-24T00:31:25.023 INFO:teuthology.orchestra.run.smithi053.stdout: "inode_count": 1, 2024-03-24T00:31:25.023 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_epoch": 8, 2024-03-24T00:31:25.023 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch": 20, 2024-03-24T00:31:25.023 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:31:25.023 INFO:teuthology.orchestra.run.smithi053.stdout: "blocklisted": false, 2024-03-24T00:31:25.023 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs" 2024-03-24T00:31:25.023 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-03-24T00:31:25.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:31:25.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fc6600640 1 Processor -- start 2024-03-24T00:31:25.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fc6600640 1 -- start start 2024-03-24T00:31:25.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fc6600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0fc00b2aa0 0x7f0fc00d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fc6600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0fc00d0a80 0x7f0fc00cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fc6600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc00b2870 con 0x7f0fc00b2490 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fc6600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc006b6d0 con 0x7f0fc00d0a80 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fc6600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc00b3190 con 0x7f0fc00b2aa0 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fbf400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0fc00b2490 0x7f0fc00cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43152/0 (socket says 172.21.15.40:43152) 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fbf400640 1 -- 172.21.15.40:0/2228318192 learned_addr learned my addr 172.21.15.40:0/2228318192 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fbfe00640 1 --2- 172.21.15.40:0/2228318192 >> v2:172.21.15.40:3300/0 conn(0x7f0fc00d0a80 0x7f0fc00cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fb7e00640 1 --2- 172.21.15.40:0/2228318192 >> v2:172.21.15.53:3300/0 conn(0x7f0fc00b2aa0 0x7f0fc00d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fbea00640 1 -- 172.21.15.40:0/2228318192 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331114658 0 0) 0x7f0fc00b2870 con 0x7f0fc00b2490 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fbea00640 1 -- 172.21.15.40:0/2228318192 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa4003530 con 0x7f0fc00b2490 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fb7e00640 1 -- 172.21.15.40:0/2228318192 >> v2:172.21.15.40:3300/0 conn(0x7f0fc00d0a80 msgr2=0x7f0fc00cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fb7e00640 1 --2- 172.21.15.40:0/2228318192 >> v2:172.21.15.40:3300/0 conn(0x7f0fc00d0a80 0x7f0fc00cc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fb7e00640 1 -- 172.21.15.40:0/2228318192 >> v1:172.21.15.40:6789/0 conn(0x7f0fc00b2490 legacy=0x7f0fc00cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:25.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.174+0000 7f0fb7e00640 1 -- 172.21.15.40:0/2228318192 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc00b3470 con 0x7f0fc00b2aa0 2024-03-24T00:31:25.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.178+0000 7f0fb7e00640 1 --2- 172.21.15.40:0/2228318192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00b2aa0 0x7f0fc00d0540 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0072db0 tx=0x7f0fa801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=511fa03622214b6b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:25.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.178+0000 7f0fbea00640 1 -- 172.21.15.40:0/2228318192 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa80b1070 con 0x7f0fc00b2aa0 2024-03-24T00:31:25.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.178+0000 7f0fbea00640 1 -- 172.21.15.40:0/2228318192 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa80ac5f0 con 0x7f0fc00b2aa0 2024-03-24T00:31:25.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.178+0000 7f0fbea00640 1 -- 172.21.15.40:0/2228318192 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa80ac9a0 con 0x7f0fc00b2aa0 2024-03-24T00:31:25.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.178+0000 7f0fc6600640 1 -- 172.21.15.40:0/2228318192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00b2aa0 msgr2=0x7f0fc00d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:25.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.178+0000 7f0fc6600640 1 --2- 172.21.15.40:0/2228318192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00b2aa0 0x7f0fc00d0540 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0072db0 tx=0x7f0fa801c480 comp rx=0 tx=0).stop 2024-03-24T00:31:25.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.178+0000 7f0fc6600640 1 -- 172.21.15.40:0/2228318192 shutdown_connections 2024-03-24T00:31:25.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.178+0000 7f0fc6600640 1 --2- 172.21.15.40:0/2228318192 >> v2:172.21.15.40:3300/0 conn(0x7f0fc00d0a80 0x7f0fc00cc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:25.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.178+0000 7f0fc6600640 1 --2- 172.21.15.40:0/2228318192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00b2aa0 0x7f0fc00d0540 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:25.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.178+0000 7f0fc6600640 1 -- 172.21.15.40:0/2228318192 >> 172.21.15.40:0/2228318192 conn(0x7f0fc00d1d10 msgr2=0x7f0fc00d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:25.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.178+0000 7f0fc6600640 1 -- 172.21.15.40:0/2228318192 shutdown_connections 2024-03-24T00:31:25.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.178+0000 7f0fc6600640 1 -- 172.21.15.40:0/2228318192 wait complete. 2024-03-24T00:31:25.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.182+0000 7f0fc6600640 1 Processor -- start 2024-03-24T00:31:25.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.182+0000 7f0fc6600640 1 -- start start 2024-03-24T00:31:25.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.182+0000 7f0fc6600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00b2490 0x7f0fc00aee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:25.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.182+0000 7f0fc6600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fc00b2aa0 0x7f0fc01585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:25.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.182+0000 7f0fc6600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fc00d0a80 0x7f0fc0150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:25.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.182+0000 7f0fc6600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00cd580 con 0x7f0fc00b2aa0 2024-03-24T00:31:25.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.182+0000 7f0fc6600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00cd400 con 0x7f0fc00b2490 2024-03-24T00:31:25.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.182+0000 7f0fc6600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc00cd700 con 0x7f0fc00d0a80 2024-03-24T00:31:25.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fbf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00b2490 0x7f0fc00aee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:25.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fbf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00b2490 0x7f0fc00aee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39346/0 (socket says 172.21.15.40:39346) 2024-03-24T00:31:25.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fbf400640 1 -- 172.21.15.40:0/2474755754 learned_addr learned my addr 172.21.15.40:0/2474755754 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:25.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fbfe00640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fc00d0a80 0x7f0fc0150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:25.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fb7e00640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fc00b2aa0 0x7f0fc01585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:25.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fbf400640 1 -- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fc00d0a80 msgr2=0x7f0fc0150970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:25.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fbf400640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fc00d0a80 0x7f0fc0150970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:25.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fbf400640 1 -- 172.21.15.40:0/2474755754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fc00b2aa0 msgr2=0x7f0fc01585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:25.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fbf400640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fc00b2aa0 0x7f0fc01585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:25.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fbf400640 1 -- 172.21.15.40:0/2474755754 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc0158e10 con 0x7f0fc00b2490 2024-03-24T00:31:25.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fb7e00640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fc00b2aa0 0x7f0fc01585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:25.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fbf400640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00b2490 0x7f0fc00aee10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000c3c0 tx=0x7f0fb000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:25.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fb7400640 1 -- 172.21.15.40:0/2474755754 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb0017070 con 0x7f0fc00b2490 2024-03-24T00:31:25.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fb7400640 1 -- 172.21.15.40:0/2474755754 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb0005cf0 con 0x7f0fc00b2490 2024-03-24T00:31:25.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fb7400640 1 -- 172.21.15.40:0/2474755754 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb00072f0 con 0x7f0fc00b2490 2024-03-24T00:31:25.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fc6600640 1 -- 172.21.15.40:0/2474755754 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc01590f0 con 0x7f0fc00b2490 2024-03-24T00:31:25.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fc6600640 1 -- 172.21.15.40:0/2474755754 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc0158b20 con 0x7f0fc00b2490 2024-03-24T00:31:25.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fc6600640 1 -- 172.21.15.40:0/2474755754 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc00b0730 con 0x7f0fc00b2490 2024-03-24T00:31:25.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.186+0000 7f0fb7400640 1 -- 172.21.15.40:0/2474755754 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f0fb001a070 con 0x7f0fc00b2490 2024-03-24T00:31:25.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.190+0000 7f0fb7400640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0f9c067660 0x7f0f9c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:25.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.190+0000 7f0fb7400640 1 -- 172.21.15.40:0/2474755754 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fb0084c90 con 0x7f0fc00b2490 2024-03-24T00:31:25.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.190+0000 7f0fb7e00640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0f9c067660 0x7f0f9c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:25.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.190+0000 7f0fb7e00640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0f9c067660 0x7f0f9c069b20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80ae000 tx=0x7f0fa80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:25.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.194+0000 7f0fb7400640 1 -- 172.21.15.40:0/2474755754 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0fb0055220 con 0x7f0fc00b2490 2024-03-24T00:31:25.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 -- 172.21.15.40:0/2474755754 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0fc00ca200 con 0x7f0fc00b2490 2024-03-24T00:31:25.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fb7400640 1 -- 172.21.15.40:0/2474755754 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0fb00592c0 con 0x7f0fc00b2490 2024-03-24T00:31:25.416 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:31:25.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 -- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0f9c067660 msgr2=0x7f0f9c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:25.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0f9c067660 0x7f0f9c069b20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0fa80ae000 tx=0x7f0fa80a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:25.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 -- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00b2490 msgr2=0x7f0fc00aee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:25.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00b2490 0x7f0fc00aee10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0fb000c3c0 tx=0x7f0fb000c880 comp rx=0 tx=0).stop 2024-03-24T00:31:25.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 -- 172.21.15.40:0/2474755754 shutdown_connections 2024-03-24T00:31:25.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0f9c067660 0x7f0f9c069b20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:25.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fc00d0a80 0x7f0fc0150970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:25.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0fc00b2aa0 0x7f0fc01585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:25.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 --2- 172.21.15.40:0/2474755754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc00b2490 0x7f0fc00aee10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:25.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 -- 172.21.15.40:0/2474755754 >> 172.21.15.40:0/2474755754 conn(0x7f0fc00d1d10 msgr2=0x7f0fc00cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:25.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 -- 172.21.15.40:0/2474755754 shutdown_connections 2024-03-24T00:31:25.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:25.414+0000 7f0fc6600640 1 -- 172.21.15.40:0/2474755754 wait complete. 2024-03-24T00:31:25.446 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:31:25.446 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:31:25.446 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-03-24T00:31:25.446 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-24T00:31:25.446 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-24T00:31:25.446 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:31:25.446 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:31:25.446 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:31:25.446 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-24T00:31:25.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:25.447 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-03-24T00:31:25.456 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 42445sec preferred_lft 42445sec 2024-03-24T00:31:25.457 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 02:42:ea:7c:60:87 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:31:25.458 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-24T00:31:25.459 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:31:25.459 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.459 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link tentative 2024-03-24T00:31:25.459 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.459 INFO:teuthology.orchestra.run.smithi053.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:31:25.459 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:31:25.459 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link tentative 2024-03-24T00:31:25.459 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:25.460 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:31:25.522 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-03-24T00:31:25.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:25.525 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-03-24T00:31:25.578 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-03-24T00:31:25.578 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:31:25.578 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-24T00:31:25.578 INFO:teuthology.orchestra.run.smithi053.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-24T00:31:25.578 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-24T00:31:25.579 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:31:25.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:25.579 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-03-24T00:31:25.579 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-03-24T00:31:25.579 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T00:31:25.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T00:31:25.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T00:31:25.580 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-03-24T00:31:25.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:25.673 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-03-24T00:31:25.682 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-24T00:31:25.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:25.683 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list-id 2024-03-24T00:31:25.732 INFO:teuthology.orchestra.run.smithi053.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-24T00:31:25.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:25.732 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-03-24T00:31:25.732 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-03-24T00:31:25.732 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T00:31:25.732 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-24T00:31:25.733 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-03-24T00:31:25.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:25.815 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-24T00:31:25.876 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:31:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:31:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:31:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-03-24T00:31:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.877 INFO:teuthology.orchestra.run.smithi053.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:31:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 46:d5:8e:75:39:7c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T00:31:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T00:31:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::44d5:8eff:fe75:397c/64 scope link tentative 2024-03-24T00:31:25.877 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:25.879 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-24T00:31:25.940 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:31:25.940 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:31:25.940 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:31:25.940 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.940 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-03-24T00:31:25.940 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.940 INFO:teuthology.orchestra.run.smithi053.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:31:25.941 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 46:d5:8e:75:39:7c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-24T00:31:25.941 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-24T00:31:25.941 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.941 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::44d5:8eff:fe75:397c/64 scope link tentative 2024-03-24T00:31:25.941 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:31:25.942 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-24T00:31:25.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:25.942 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-03-24T00:31:25.943 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-03-24T00:31:25.943 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-24T00:31:25.943 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-24T00:31:25.943 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-24T00:31:25.943 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-24T00:31:25.943 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-24T00:31:25.943 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-03-24T00:31:26.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:26.078 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-03-24T00:31:26.078 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-03-24T00:31:26.078 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.1 up 2024-03-24T00:31:26.078 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set dev brx.1 master ceph-brx 2024-03-24T00:31:26.078 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-03-24T00:31:26.118 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-24T00:31:26.118 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:31:26.118 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-24T00:31:26.172 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-24T00:31:26.172 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:31:26.172 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-24T00:31:26.223 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-03-24T00:31:26.284 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:31:26.337 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-03-24T00:31:26.337 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-03-24T00:31:26.337 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-03-24T00:31:26.337 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-03-24T00:31:26.337 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-03-24T00:31:26.337 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/run/lock 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/bpf 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/fs/nfsd 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-03-24T00:31:26.338 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/run/rpc_pipefs 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-4 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-5 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-6 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-7 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T00:31:26.339 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T00:31:26.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:26.340 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-03-24T00:31:26.391 INFO:teuthology.orchestra.run.smithi053.stdout:43 2024-03-24T00:31:26.391 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-03-24T00:31:26.392 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-24T00:31:26.435 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-03-24T00:31:26.455 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:31:26.505 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-03-24T00:31:26.505 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-03-24T00:31:26.505 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-03-24T00:31:26.505 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/run/lock 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/bpf 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-03-24T00:31:26.506 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/fs/nfsd 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/run/rpc_pipefs 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-4 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-5 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-6 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-7 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:31:26.507 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:31:26.508 INFO:teuthology.orchestra.run.smithi053.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-24T00:31:26.508 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T00:31:26.508 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T00:31:26.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:26.509 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-03-24T00:31:26.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi053.stderr:2024-03-24T00:31:26.529+0000 7f22fc613580 -1 init, newargv = 0x55b406b9f650 newargc=15 2024-03-24T00:31:26.530 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi053.stderr:ceph-fuse[32042]: starting ceph client 2024-03-24T00:31:26.537 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi053.stderr:ceph-fuse[32042]: starting fuse 2024-03-24T00:31:26.538 INFO:teuthology.orchestra.run.smithi053.stdout:43 2024-03-24T00:31:26.539 INFO:teuthology.orchestra.run.smithi053.stdout:44 2024-03-24T00:31:26.539 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-03-24T00:31:26.539 DEBUG:teuthology.orchestra.run.smithi053:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:31:26.539 DEBUG:teuthology.orchestra.run.smithi053:> import glob 2024-03-24T00:31:26.539 DEBUG:teuthology.orchestra.run.smithi053:> import re 2024-03-24T00:31:26.539 DEBUG:teuthology.orchestra.run.smithi053:> import os 2024-03-24T00:31:26.539 DEBUG:teuthology.orchestra.run.smithi053:> import subprocess 2024-03-24T00:31:26.539 DEBUG:teuthology.orchestra.run.smithi053:> 2024-03-24T00:31:26.539 DEBUG:teuthology.orchestra.run.smithi053:> def _find_admin_socket(client_name): 2024-03-24T00:31:26.539 DEBUG:teuthology.orchestra.run.smithi053:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> files = glob.glob(asok_path) 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> # Given a non-glob path, it better be there 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> if "*" not in asok_path: 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> assert(len(files) == 1) 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> return files[0] 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> for f in files: 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> contents = proc_f.read() 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> if mountpoint in contents: 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> return f 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:31:26.540 DEBUG:teuthology.orchestra.run.smithi053:> 2024-03-24T00:31:26.541 DEBUG:teuthology.orchestra.run.smithi053:> print(_find_admin_socket("client.3")) 2024-03-24T00:31:26.541 DEBUG:teuthology.orchestra.run.smithi053:> ' 2024-03-24T00:31:26.663 INFO:teuthology.orchestra.run.smithi053.stdout:/var/run/ceph/ceph-client.3.32042.asok 2024-03-24T00:31:26.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-03-24T00:31:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:31:26.675 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.32042.asok 2024-03-24T00:31:26.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:26.675 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.32042.asok status 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout: "metadata": { 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout: "entity_id": "3", 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout: "hostname": "smithi053", 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout: "pid": "32042", 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout: "root": "/" 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_count": 0, 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_pinned_count": 0, 2024-03-24T00:31:26.852 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4505, 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: "inst": { 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: "name": { 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "client", 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: "num": 4505 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.2:0", 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 1709703859 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-03-24T00:31:26.853 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.2:0", 2024-03-24T00:31:26.854 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 1709703859 2024-03-24T00:31:26.854 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-03-24T00:31:26.854 INFO:teuthology.orchestra.run.smithi053.stdout: "inst_str": "client.4505 192.168.0.2:0/1709703859", 2024-03-24T00:31:26.854 INFO:teuthology.orchestra.run.smithi053.stdout: "addr_str": "192.168.0.2:0/1709703859", 2024-03-24T00:31:26.854 INFO:teuthology.orchestra.run.smithi053.stdout: "inode_count": 1, 2024-03-24T00:31:26.854 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_epoch": 8, 2024-03-24T00:31:26.854 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch": 20, 2024-03-24T00:31:26.854 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:31:26.854 INFO:teuthology.orchestra.run.smithi053.stdout: "blocklisted": false, 2024-03-24T00:31:26.854 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs" 2024-03-24T00:31:26.854 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-03-24T00:31:26.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:26.866 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-03-24T00:31:26.866 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T00:31:26.875 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-03-24T00:31:26.876 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T00:31:26.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:26.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:31:26.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:26.946 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-03-24T00:31:26.946 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-24T00:31:26.997 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-03-24T00:31:26.997 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-24T00:31:26.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:26.997 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-24T00:31:27.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:27.061 DEBUG:teuthology.orchestra.run.smithi053:> stat --file-system '--printf=%T 2024-03-24T00:31:27.061 DEBUG:teuthology.orchestra.run.smithi053:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-24T00:31:27.070 INFO:teuthology.orchestra.run.smithi053.stdout:fuseblk 2024-03-24T00:31:27.070 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-24T00:31:27.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:27.070 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-24T00:31:27.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:27.132 DEBUG:teuthology.orchestra.run.smithi053:> stat --file-system '--printf=%T 2024-03-24T00:31:27.132 DEBUG:teuthology.orchestra.run.smithi053:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-24T00:31:27.184 INFO:teuthology.orchestra.run.smithi053.stdout:fuseblk 2024-03-24T00:31:27.185 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-24T00:31:27.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:27.185 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-24T00:31:27.248 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-24T00:31:27.266 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:27.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa52ce00640 1 Processor -- start 2024-03-24T00:31:27.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa52ce00640 1 -- start start 2024-03-24T00:31:27.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa52ce00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa528159850 0x7fa5280ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:27.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa52ce00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa528159c30 0x7fa5280d0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:27.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa52ce00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52806e860 con 0x7fa52815fa90 2024-03-24T00:31:27.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa52ce00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa52806eb40 con 0x7fa528159c30 2024-03-24T00:31:27.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa52ce00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa52806ee20 con 0x7fa528159850 2024-03-24T00:31:27.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa526800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa52815fa90 0x7fa5280b6030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41100/0 (socket says 172.21.15.40:41100) 2024-03-24T00:31:27.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa526800640 1 -- 172.21.15.40:0/3856272477 learned_addr learned my addr 172.21.15.40:0/3856272477 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:27.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa51fe00640 1 --2- 172.21.15.40:0/3856272477 >> v2:172.21.15.40:3300/0 conn(0x7fa528159c30 0x7fa5280d0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:27.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa525e00640 1 --2- 172.21.15.40:0/3856272477 >> v2:172.21.15.53:3300/0 conn(0x7fa528159850 0x7fa5280ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:27.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa51fe00640 1 -- 172.21.15.40:0/3856272477 >> v2:172.21.15.53:3300/0 conn(0x7fa528159850 msgr2=0x7fa5280ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:27.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa51fe00640 1 --2- 172.21.15.40:0/3856272477 >> v2:172.21.15.53:3300/0 conn(0x7fa528159850 0x7fa5280ceaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa525400640 1 -- 172.21.15.40:0/3856272477 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730748598 0 0) 0x7fa52806e860 con 0x7fa52815fa90 2024-03-24T00:31:27.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa51fe00640 1 -- 172.21.15.40:0/3856272477 >> v1:172.21.15.40:6789/0 conn(0x7fa52815fa90 legacy=0x7fa5280b6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:27.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.406+0000 7fa51fe00640 1 -- 172.21.15.40:0/3856272477 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52806f100 con 0x7fa528159c30 2024-03-24T00:31:27.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.410+0000 7fa51fe00640 1 --2- 172.21.15.40:0/3856272477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa528159c30 0x7fa5280d0400 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa528067f10 tx=0x7fa51001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373b7b456746ee8b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:27.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.410+0000 7fa525400640 1 -- 172.21.15.40:0/3856272477 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5100b1070 con 0x7fa528159c30 2024-03-24T00:31:27.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.410+0000 7fa525400640 1 -- 172.21.15.40:0/3856272477 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5100a8050 con 0x7fa528159c30 2024-03-24T00:31:27.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.410+0000 7fa525400640 1 -- 172.21.15.40:0/3856272477 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5100ac860 con 0x7fa528159c30 2024-03-24T00:31:27.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.410+0000 7fa52ce00640 1 -- 172.21.15.40:0/3856272477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa528159c30 msgr2=0x7fa5280d0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:27.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.410+0000 7fa52ce00640 1 --2- 172.21.15.40:0/3856272477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa528159c30 0x7fa5280d0400 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa528067f10 tx=0x7fa51001c3c0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.410+0000 7fa52ce00640 1 -- 172.21.15.40:0/3856272477 shutdown_connections 2024-03-24T00:31:27.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.410+0000 7fa52ce00640 1 --2- 172.21.15.40:0/3856272477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa528159c30 0x7fa5280d0400 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.410+0000 7fa52ce00640 1 --2- 172.21.15.40:0/3856272477 >> v2:172.21.15.53:3300/0 conn(0x7fa528159850 0x7fa5280ceaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.410+0000 7fa52ce00640 1 -- 172.21.15.40:0/3856272477 >> 172.21.15.40:0/3856272477 conn(0x7fa5280b2a20 msgr2=0x7fa5280b2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:27.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.410+0000 7fa52ce00640 1 -- 172.21.15.40:0/3856272477 shutdown_connections 2024-03-24T00:31:27.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.410+0000 7fa52ce00640 1 -- 172.21.15.40:0/3856272477 wait complete. 2024-03-24T00:31:27.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa52ce00640 1 Processor -- start 2024-03-24T00:31:27.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa52ce00640 1 -- start start 2024-03-24T00:31:27.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa52ce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528159850 0x7fa528150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:27.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa52ce00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa528159c30 0x7fa52814f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:27.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa52ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa52815fa90 0x7fa528156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:27.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa52ce00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52815d050 con 0x7fa52815fa90 2024-03-24T00:31:27.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa52ce00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5280b6950 con 0x7fa528159850 2024-03-24T00:31:27.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa52ce00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52815d1d0 con 0x7fa528159c30 2024-03-24T00:31:27.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa526800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa52815fa90 0x7fa528156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:27.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa525e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528159850 0x7fa528150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:27.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa51fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa528159c30 0x7fa52814f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:27.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa525e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528159850 0x7fa528150590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:38288/0 (socket says 172.21.15.40:38288) 2024-03-24T00:31:27.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa525e00640 1 -- 172.21.15.40:0/2504366670 learned_addr learned my addr 172.21.15.40:0/2504366670 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:27.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa526800640 1 -- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa528159c30 msgr2=0x7fa52814f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:27.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa526800640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa528159c30 0x7fa52814f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa526800640 1 -- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528159850 msgr2=0x7fa528150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:27.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa526800640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528159850 0x7fa528150590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa526800640 1 -- 172.21.15.40:0/2504366670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa528165aa0 con 0x7fa52815fa90 2024-03-24T00:31:27.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa525e00640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528159850 0x7fa528150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:27.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa526800640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa52815fa90 0x7fa528156f00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa52000c750 tx=0x7fa52000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:27.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa51e000640 1 -- 172.21.15.40:0/2504366670 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520018070 con 0x7fa52815fa90 2024-03-24T00:31:27.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa51e000640 1 -- 172.21.15.40:0/2504366670 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa520004d80 con 0x7fa52815fa90 2024-03-24T00:31:27.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa52ce00640 1 -- 172.21.15.40:0/2504366670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5281fa040 con 0x7fa52815fa90 2024-03-24T00:31:27.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa51e000640 1 -- 172.21.15.40:0/2504366670 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52000a880 con 0x7fa52815fa90 2024-03-24T00:31:27.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.414+0000 7fa52ce00640 1 -- 172.21.15.40:0/2504366670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52814fe00 con 0x7fa52815fa90 2024-03-24T00:31:27.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.418+0000 7fa52ce00640 1 -- 172.21.15.40:0/2504366670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa528068170 con 0x7fa52815fa90 2024-03-24T00:31:27.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.418+0000 7fa51e000640 1 -- 172.21.15.40:0/2504366670 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa52001b070 con 0x7fa52815fa90 2024-03-24T00:31:27.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.418+0000 7fa51e000640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa504067660 0x7fa504069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:27.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.418+0000 7fa51e000640 1 -- 172.21.15.40:0/2504366670 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa520084d10 con 0x7fa52815fa90 2024-03-24T00:31:27.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.418+0000 7fa525e00640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa504067660 0x7fa504069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:27.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.418+0000 7fa525e00640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa504067660 0x7fa504069b20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa5281f9e00 tx=0x7fa518009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:27.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.422+0000 7fa51e000640 1 -- 172.21.15.40:0/2504366670 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa52008a020 con 0x7fa52815fa90 2024-03-24T00:31:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.586+0000 7fa52ce00640 1 -- 172.21.15.40:0/2504366670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5280b6030 con 0x7fa52815fa90 2024-03-24T00:31:27.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa51e000640 1 -- 172.21.15.40:0/2504366670 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7fa520055330 con 0x7fa52815fa90 2024-03-24T00:31:27.592 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:27.592 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa52ce00640 1 -- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa504067660 msgr2=0x7fa504069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa52ce00640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa504067660 0x7fa504069b20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa5281f9e00 tx=0x7fa518009190 comp rx=0 tx=0).stop 2024-03-24T00:31:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa52ce00640 1 -- 172.21.15.40:0/2504366670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa52815fa90 msgr2=0x7fa528156f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa52ce00640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa52815fa90 0x7fa528156f00 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa52000c750 tx=0x7fa52000cc10 comp rx=0 tx=0).stop 2024-03-24T00:31:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa52ce00640 1 -- 172.21.15.40:0/2504366670 shutdown_connections 2024-03-24T00:31:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa52ce00640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa504067660 0x7fa504069b20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa52ce00640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa52815fa90 0x7fa528156f00 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa52ce00640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa528159c30 0x7fa52814f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa52ce00640 1 --2- 172.21.15.40:0/2504366670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa528159850 0x7fa528150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa52ce00640 1 -- 172.21.15.40:0/2504366670 >> 172.21.15.40:0/2504366670 conn(0x7fa5280b2a20 msgr2=0x7fa5280b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa52ce00640 1 -- 172.21.15.40:0/2504366670 shutdown_connections 2024-03-24T00:31:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.590+0000 7fa52ce00640 1 -- 172.21.15.40:0/2504366670 wait complete. 2024-03-24T00:31:27.614 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:27.614 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:27.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.754+0000 7fc6f7800640 1 Processor -- start 2024-03-24T00:31:27.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.754+0000 7fc6f7800640 1 -- start start 2024-03-24T00:31:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.754+0000 7fc6f7800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6f015df80 0x7fc6f0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.754+0000 7fc6f7800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc6f006c950 0x7fc6f00d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.754+0000 7fc6f7800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f006d0f0 con 0x7fc6f006bd20 2024-03-24T00:31:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.754+0000 7fc6f7800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f006b890 con 0x7fc6f006c950 2024-03-24T00:31:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.754+0000 7fc6f7800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f006b280 con 0x7fc6f015df80 2024-03-24T00:31:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.758+0000 7fc6e7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc6f006c950 0x7fc6f00d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.758+0000 7fc6e7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc6f006c950 0x7fc6f00d1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:38304/0 (socket says 172.21.15.40:38304) 2024-03-24T00:31:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.758+0000 7fc6e7e00640 1 -- 172.21.15.40:0/1929559697 learned_addr learned my addr 172.21.15.40:0/1929559697 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.758+0000 7fc6f4e00640 1 --2- 172.21.15.40:0/1929559697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6f015df80 0x7fc6f0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.758+0000 7fc6f5800640 1 --1- 172.21.15.40:0/1929559697 >> v1:172.21.15.53:6789/0 conn(0x7fc6f006bd20 0x7fc6f0161ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:36964/0 (socket says 172.21.15.40:36964) 2024-03-24T00:31:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.758+0000 7fc6e7e00640 1 -- 172.21.15.40:0/1929559697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6f015df80 msgr2=0x7fc6f0070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.758+0000 7fc6e7e00640 1 --2- 172.21.15.40:0/1929559697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6f015df80 0x7fc6f0070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.758+0000 7fc6e7e00640 1 -- 172.21.15.40:0/1929559697 >> v1:172.21.15.53:6789/0 conn(0x7fc6f006bd20 legacy=0x7fc6f0161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.758+0000 7fc6e7e00640 1 -- 172.21.15.40:0/1929559697 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f006ce10 con 0x7fc6f006c950 2024-03-24T00:31:27.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.758+0000 7fc6e7e00640 1 --2- 172.21.15.40:0/1929559697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f006c950 0x7fc6f00d1b60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009870 tx=0x7fc6d801c530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6096e79a24246bd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6efe00640 1 -- 172.21.15.40:0/1929559697 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d80ab640 con 0x7fc6f006c950 2024-03-24T00:31:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6efe00640 1 -- 172.21.15.40:0/1929559697 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d80b0070 con 0x7fc6f006c950 2024-03-24T00:31:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6efe00640 1 -- 172.21.15.40:0/1929559697 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d80aba00 con 0x7fc6f006c950 2024-03-24T00:31:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 -- 172.21.15.40:0/1929559697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f006c950 msgr2=0x7fc6f00d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 --2- 172.21.15.40:0/1929559697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f006c950 0x7fc6f00d1b60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009870 tx=0x7fc6d801c530 comp rx=0 tx=0).stop 2024-03-24T00:31:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 -- 172.21.15.40:0/1929559697 shutdown_connections 2024-03-24T00:31:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 --2- 172.21.15.40:0/1929559697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f006c950 0x7fc6f00d1b60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 --2- 172.21.15.40:0/1929559697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6f015df80 0x7fc6f0070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 -- 172.21.15.40:0/1929559697 >> 172.21.15.40:0/1929559697 conn(0x7fc6f006ecf0 msgr2=0x7fc6f006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 -- 172.21.15.40:0/1929559697 shutdown_connections 2024-03-24T00:31:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 -- 172.21.15.40:0/1929559697 wait complete. 2024-03-24T00:31:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 Processor -- start 2024-03-24T00:31:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 -- start start 2024-03-24T00:31:27.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f006bd20 0x7fc6f00c71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:27.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6f006c950 0x7fc6f00ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:27.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6f015df80 0x7fc6f00aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f016af90 con 0x7fc6f006c950 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f016ae10 con 0x7fc6f006bd20 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6f7800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f016b110 con 0x7fc6f015df80 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6f006c950 0x7fc6f00ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6f006c950 0x7fc6f00ae930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45532/0 (socket says 172.21.15.40:45532) 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.762+0000 7fc6e7e00640 1 -- 172.21.15.40:0/2747532946 learned_addr learned my addr 172.21.15.40:0/2747532946 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6f5800640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6f015df80 0x7fc6f00aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6f4e00640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f006bd20 0x7fc6f00c71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6e7e00640 1 -- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6f015df80 msgr2=0x7fc6f00aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6e7e00640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6f015df80 0x7fc6f00aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6e7e00640 1 -- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f006bd20 msgr2=0x7fc6f00c71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6e7e00640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f006bd20 0x7fc6f00c71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6e7e00640 1 -- 172.21.15.40:0/2747532946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f00c76f0 con 0x7fc6f006c950 2024-03-24T00:31:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6f5800640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6f015df80 0x7fc6f00aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6e7e00640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6f006c950 0x7fc6f00ae930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc6d801abe0 tx=0x7fc6d801ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6ed600640 1 -- 172.21.15.40:0/2747532946 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d80abe60 con 0x7fc6f006c950 2024-03-24T00:31:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6ed600640 1 -- 172.21.15.40:0/2747532946 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d80be3f0 con 0x7fc6f006c950 2024-03-24T00:31:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6ed600640 1 -- 172.21.15.40:0/2747532946 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d80be7c0 con 0x7fc6f006c950 2024-03-24T00:31:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6f7800640 1 -- 172.21.15.40:0/2747532946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f00b2340 con 0x7fc6f006c950 2024-03-24T00:31:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6f7800640 1 -- 172.21.15.40:0/2747532946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f00b2900 con 0x7fc6f006c950 2024-03-24T00:31:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6f7800640 1 -- 172.21.15.40:0/2747532946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f01532a0 con 0x7fc6f006c950 2024-03-24T00:31:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6ed600640 1 -- 172.21.15.40:0/2747532946 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc6d80b43f0 con 0x7fc6f006c950 2024-03-24T00:31:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.766+0000 7fc6ed600640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6cc067660 0x7fc6cc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.770+0000 7fc6ed600640 1 -- 172.21.15.40:0/2747532946 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6d811f5c0 con 0x7fc6f006c950 2024-03-24T00:31:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.770+0000 7fc6f4e00640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6cc067660 0x7fc6cc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:27.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.770+0000 7fc6f4e00640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6cc067660 0x7fc6cc069b20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009a50 tx=0x7fc6e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:27.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.778+0000 7fc6ed600640 1 -- 172.21.15.40:0/2747532946 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc6f01532a0 con 0x7fc6f006c950 2024-03-24T00:31:27.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.938+0000 7fc6f7800640 1 -- 172.21.15.40:0/2747532946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc6f0068170 con 0x7fc6f006c950 2024-03-24T00:31:27.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.938+0000 7fc6ed600640 1 -- 172.21.15.40:0/2747532946 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc6d8016150 con 0x7fc6f006c950 2024-03-24T00:31:27.942 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.942+0000 7fc6f7800640 1 -- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6cc067660 msgr2=0x7fc6cc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.942+0000 7fc6f7800640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6cc067660 0x7fc6cc069b20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009a50 tx=0x7fc6e0008040 comp rx=0 tx=0).stop 2024-03-24T00:31:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.942+0000 7fc6f7800640 1 -- 172.21.15.40:0/2747532946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6f006c950 msgr2=0x7fc6f00ae930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.942+0000 7fc6f7800640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6f006c950 0x7fc6f00ae930 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc6d801abe0 tx=0x7fc6d801ad70 comp rx=0 tx=0).stop 2024-03-24T00:31:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.942+0000 7fc6f7800640 1 -- 172.21.15.40:0/2747532946 shutdown_connections 2024-03-24T00:31:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.942+0000 7fc6f7800640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6cc067660 0x7fc6cc069b20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.942+0000 7fc6f7800640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6f015df80 0x7fc6f00aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.942+0000 7fc6f7800640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6f006c950 0x7fc6f00ae930 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.942+0000 7fc6f7800640 1 --2- 172.21.15.40:0/2747532946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6f006bd20 0x7fc6f00c71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.942+0000 7fc6f7800640 1 -- 172.21.15.40:0/2747532946 >> 172.21.15.40:0/2747532946 conn(0x7fc6f006ecf0 msgr2=0x7fc6f0160050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.942+0000 7fc6f7800640 1 -- 172.21.15.40:0/2747532946 shutdown_connections 2024-03-24T00:31:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:27.942+0000 7fc6f7800640 1 -- 172.21.15.40:0/2747532946 wait complete. 2024-03-24T00:31:27.963 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:28.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd161e00640 1 Processor -- start 2024-03-24T00:31:28.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd161e00640 1 -- start start 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd161e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd15c15df80 0x7fd15c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd161e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c06d0f0 con 0x7fd15c06c950 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd161e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c06b280 con 0x7fd15c0d2270 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd161e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c06b890 con 0x7fd15c15df80 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd15a000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd15c06c950 0x7fd15c0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41108/0 (socket says 172.21.15.40:41108) 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd15a000640 1 -- 172.21.15.40:0/322850434 learned_addr learned my addr 172.21.15.40:0/322850434 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd15aa00640 1 --2- 172.21.15.40:0/322850434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd15c15df80 0x7fd15c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd159600640 1 -- 172.21.15.40:0/322850434 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314437967 0 0) 0x7fd15c06d0f0 con 0x7fd15c06c950 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd159600640 1 -- 172.21.15.40:0/322850434 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd140003530 con 0x7fd15c06c950 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd159600640 1 -- 172.21.15.40:0/322850434 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350183199 0 0) 0x7fd15c06b280 con 0x7fd15c0d2270 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd159600640 1 -- 172.21.15.40:0/322850434 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd15c06d0f0 con 0x7fd15c0d2270 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd15aa00640 1 -- 172.21.15.40:0/322850434 >> v1:172.21.15.53:6789/0 conn(0x7fd15c0d2270 legacy=0x7fd15c161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd15aa00640 1 -- 172.21.15.40:0/322850434 >> v1:172.21.15.40:6789/0 conn(0x7fd15c06c950 legacy=0x7fd15c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.118+0000 7fd15aa00640 1 -- 172.21.15.40:0/322850434 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c06ce10 con 0x7fd15c15df80 2024-03-24T00:31:28.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.122+0000 7fd15aa00640 1 --2- 172.21.15.40:0/322850434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd15c15df80 0x7fd15c070ff0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0097f0 tx=0x7fd14c01c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da9fbc5e6f943f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:28.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.122+0000 7fd159600640 1 -- 172.21.15.40:0/322850434 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0a7070 con 0x7fd15c15df80 2024-03-24T00:31:28.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.122+0000 7fd159600640 1 -- 172.21.15.40:0/322850434 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c0b0070 con 0x7fd15c15df80 2024-03-24T00:31:28.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.122+0000 7fd159600640 1 -- 172.21.15.40:0/322850434 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0ab870 con 0x7fd15c15df80 2024-03-24T00:31:28.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.122+0000 7fd161e00640 1 -- 172.21.15.40:0/322850434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd15c15df80 msgr2=0x7fd15c070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.122+0000 7fd161e00640 1 --2- 172.21.15.40:0/322850434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd15c15df80 0x7fd15c070ff0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd14c0097f0 tx=0x7fd14c01c4b0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.122+0000 7fd161e00640 1 -- 172.21.15.40:0/322850434 shutdown_connections 2024-03-24T00:31:28.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.122+0000 7fd161e00640 1 --2- 172.21.15.40:0/322850434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd15c15df80 0x7fd15c070ff0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.122+0000 7fd161e00640 1 -- 172.21.15.40:0/322850434 >> 172.21.15.40:0/322850434 conn(0x7fd15c06ecf0 msgr2=0x7fd15c06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:28.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.122+0000 7fd161e00640 1 -- 172.21.15.40:0/322850434 shutdown_connections 2024-03-24T00:31:28.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.122+0000 7fd161e00640 1 -- 172.21.15.40:0/322850434 wait complete. 2024-03-24T00:31:28.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd161e00640 1 Processor -- start 2024-03-24T00:31:28.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd161e00640 1 -- start start 2024-03-24T00:31:28.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd161e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd15c06c950 0x7fd15c154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd161e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd15c0d2270 0x7fd15c15cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd161e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd15c15df80 0x7fd15c0af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd161e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c16bfa0 con 0x7fd15c06c950 2024-03-24T00:31:28.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd161e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c16be20 con 0x7fd15c0d2270 2024-03-24T00:31:28.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd161e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c16c120 con 0x7fd15c15df80 2024-03-24T00:31:28.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd15c06c950 0x7fd15c154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd15c06c950 0x7fd15c154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45546/0 (socket says 172.21.15.40:45546) 2024-03-24T00:31:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15aa00640 1 -- 172.21.15.40:0/3489450053 learned_addr learned my addr 172.21.15.40:0/3489450053 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15a000640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd15c0d2270 0x7fd15c15cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15b400640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd15c15df80 0x7fd15c0af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15aa00640 1 -- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd15c15df80 msgr2=0x7fd15c0af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15aa00640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd15c15df80 0x7fd15c0af9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15aa00640 1 -- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd15c0d2270 msgr2=0x7fd15c15cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15aa00640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd15c0d2270 0x7fd15c15cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15aa00640 1 -- 172.21.15.40:0/3489450053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c1554b0 con 0x7fd15c06c950 2024-03-24T00:31:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15b400640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd15c15df80 0x7fd15c0af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15a000640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd15c0d2270 0x7fd15c15cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:28.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd15aa00640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd15c06c950 0x7fd15c154b90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd14c01a0e0 tx=0x7fd14c019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:28.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd153400640 1 -- 172.21.15.40:0/3489450053 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0abb30 con 0x7fd15c06c950 2024-03-24T00:31:28.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd153400640 1 -- 172.21.15.40:0/3489450053 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c0abcf0 con 0x7fd15c06c950 2024-03-24T00:31:28.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd153400640 1 -- 172.21.15.40:0/3489450053 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0be630 con 0x7fd15c06c950 2024-03-24T00:31:28.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd161e00640 1 -- 172.21.15.40:0/3489450053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c0aee30 con 0x7fd15c06c950 2024-03-24T00:31:28.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd161e00640 1 -- 172.21.15.40:0/3489450053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c0aeb50 con 0x7fd15c06c950 2024-03-24T00:31:28.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.126+0000 7fd161e00640 1 -- 172.21.15.40:0/3489450053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c068170 con 0x7fd15c06c950 2024-03-24T00:31:28.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.130+0000 7fd153400640 1 -- 172.21.15.40:0/3489450053 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd14c0be7d0 con 0x7fd15c06c950 2024-03-24T00:31:28.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.130+0000 7fd153400640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd13c067660 0x7fd13c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.130+0000 7fd153400640 1 -- 172.21.15.40:0/3489450053 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd14c11feb0 con 0x7fd15c06c950 2024-03-24T00:31:28.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.130+0000 7fd15a000640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd13c067660 0x7fd13c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.130+0000 7fd15a000640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd13c067660 0x7fd13c069b20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd15c1599c0 tx=0x7fd148009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:28.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.134+0000 7fd153400640 1 -- 172.21.15.40:0/3489450053 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd14c0160d0 con 0x7fd15c06c950 2024-03-24T00:31:28.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.330+0000 7fd161e00640 1 -- 172.21.15.40:0/3489450053 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd15c0cf9d0 con 0x7fd15c06c950 2024-03-24T00:31:28.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.330+0000 7fd153400640 1 -- 172.21.15.40:0/3489450053 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd14c0f0c40 con 0x7fd15c06c950 2024-03-24T00:31:28.334 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.334+0000 7fd161e00640 1 -- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd13c067660 msgr2=0x7fd13c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.334+0000 7fd161e00640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd13c067660 0x7fd13c069b20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd15c1599c0 tx=0x7fd148009040 comp rx=0 tx=0).stop 2024-03-24T00:31:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.334+0000 7fd161e00640 1 -- 172.21.15.40:0/3489450053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd15c06c950 msgr2=0x7fd15c154b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.334+0000 7fd161e00640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd15c06c950 0x7fd15c154b90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd14c01a0e0 tx=0x7fd14c019250 comp rx=0 tx=0).stop 2024-03-24T00:31:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.334+0000 7fd161e00640 1 -- 172.21.15.40:0/3489450053 shutdown_connections 2024-03-24T00:31:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.334+0000 7fd161e00640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd13c067660 0x7fd13c069b20 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.334+0000 7fd161e00640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd15c15df80 0x7fd15c0af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.334+0000 7fd161e00640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd15c0d2270 0x7fd15c15cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.334+0000 7fd161e00640 1 --2- 172.21.15.40:0/3489450053 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd15c06c950 0x7fd15c154b90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.334+0000 7fd161e00640 1 -- 172.21.15.40:0/3489450053 >> 172.21.15.40:0/3489450053 conn(0x7fd15c06ecf0 msgr2=0x7fd15c068fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.334+0000 7fd161e00640 1 -- 172.21.15.40:0/3489450053 shutdown_connections 2024-03-24T00:31:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.334+0000 7fd161e00640 1 -- 172.21.15.40:0/3489450053 wait complete. 2024-03-24T00:31:28.355 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:28.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a3c00640 1 Processor -- start 2024-03-24T00:31:28.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a3c00640 1 -- start start 2024-03-24T00:31:28.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a3c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f789c15b960 0x7f789c070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a3c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f789c0d16f0 0x7f789c0712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a3c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789c15f970 con 0x7f789c15bd40 2024-03-24T00:31:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a3c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789c160580 con 0x7f789c0d16f0 2024-03-24T00:31:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a3c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789c15fc50 con 0x7f789c15b960 2024-03-24T00:31:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f789be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f789c15bd40 0x7f789c0d0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41112/0 (socket says 172.21.15.40:41112) 2024-03-24T00:31:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f789be00640 1 -- 172.21.15.40:0/430584545 learned_addr learned my addr 172.21.15.40:0/430584545 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a1c00640 1 --2- 172.21.15.40:0/430584545 >> v2:172.21.15.40:3300/0 conn(0x7f789c0d16f0 0x7f789c0712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a1200640 1 --2- 172.21.15.40:0/430584545 >> v2:172.21.15.53:3300/0 conn(0x7f789c15b960 0x7f789c070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a1c00640 1 -- 172.21.15.40:0/430584545 >> v2:172.21.15.53:3300/0 conn(0x7f789c15b960 msgr2=0x7f789c070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a1c00640 1 --2- 172.21.15.40:0/430584545 >> v2:172.21.15.53:3300/0 conn(0x7f789c15b960 0x7f789c070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a1c00640 1 -- 172.21.15.40:0/430584545 >> v1:172.21.15.40:6789/0 conn(0x7f789c15bd40 legacy=0x7f789c0d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f78a1c00640 1 -- 172.21.15.40:0/430584545 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c15ff30 con 0x7f789c0d16f0 2024-03-24T00:31:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.498+0000 7f789b400640 1 -- 172.21.15.40:0/430584545 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325944288 0 0) 0x7f789c15f970 con 0x7f789c15bd40 2024-03-24T00:31:28.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f78a1c00640 1 --2- 172.21.15.40:0/430584545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f789c0d16f0 0x7f789c0712a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f789c0d1b30 tx=0x7f789401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e8343c2fdd2cf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:28.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f789b400640 1 -- 172.21.15.40:0/430584545 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78940b2070 con 0x7f789c0d16f0 2024-03-24T00:31:28.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f789b400640 1 -- 172.21.15.40:0/430584545 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78940a5900 con 0x7f789c0d16f0 2024-03-24T00:31:28.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f789b400640 1 -- 172.21.15.40:0/430584545 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78940a5cd0 con 0x7f789c0d16f0 2024-03-24T00:31:28.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f78a3c00640 1 -- 172.21.15.40:0/430584545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f789c0d16f0 msgr2=0x7f789c0712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f78a3c00640 1 --2- 172.21.15.40:0/430584545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f789c0d16f0 0x7f789c0712a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f789c0d1b30 tx=0x7f789401c930 comp rx=0 tx=0).stop 2024-03-24T00:31:28.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f78a3c00640 1 -- 172.21.15.40:0/430584545 shutdown_connections 2024-03-24T00:31:28.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f78a3c00640 1 --2- 172.21.15.40:0/430584545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f789c0d16f0 0x7f789c0712a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f78a3c00640 1 --2- 172.21.15.40:0/430584545 >> v2:172.21.15.53:3300/0 conn(0x7f789c15b960 0x7f789c070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f78a3c00640 1 -- 172.21.15.40:0/430584545 >> 172.21.15.40:0/430584545 conn(0x7f789c06ecf0 msgr2=0x7f789c06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:28.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f78a3c00640 1 -- 172.21.15.40:0/430584545 shutdown_connections 2024-03-24T00:31:28.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f78a3c00640 1 -- 172.21.15.40:0/430584545 wait complete. 2024-03-24T00:31:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f78a3c00640 1 Processor -- start 2024-03-24T00:31:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.502+0000 7f78a3c00640 1 -- start start 2024-03-24T00:31:28.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a3c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f789c0d16f0 0x7f789c146130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a3c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f789c15b960 0x7f789c14f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a3c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f789c15bd40 0x7f789c165e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a3c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c0ce580 con 0x7f789c0d16f0 2024-03-24T00:31:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a3c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c0ce400 con 0x7f789c15bd40 2024-03-24T00:31:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a3c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789c0ce700 con 0x7f789c15b960 2024-03-24T00:31:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f789c0d16f0 0x7f789c146130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f789c0d16f0 0x7f789c146130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45560/0 (socket says 172.21.15.40:45560) 2024-03-24T00:31:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a1200640 1 -- 172.21.15.40:0/3687488687 learned_addr learned my addr 172.21.15.40:0/3687488687 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a1c00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f789c15bd40 0x7f789c165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f789be00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f789c15b960 0x7f789c14f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a1200640 1 -- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f789c15b960 msgr2=0x7f789c14f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a1200640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f789c15b960 0x7f789c14f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a1200640 1 -- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f789c15bd40 msgr2=0x7f789c165e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a1200640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f789c15bd40 0x7f789c165e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a1200640 1 -- 172.21.15.40:0/3687488687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c146ef0 con 0x7f789c0d16f0 2024-03-24T00:31:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a1c00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f789c15bd40 0x7f789c165e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f789be00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f789c15b960 0x7f789c14f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a1200640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f789c0d16f0 0x7f789c146130 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f789000e930 tx=0x7f789000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f7898c00640 1 -- 172.21.15.40:0/3687488687 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890017070 con 0x7f789c0d16f0 2024-03-24T00:31:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f7898c00640 1 -- 172.21.15.40:0/3687488687 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789000ca00 con 0x7f789c0d16f0 2024-03-24T00:31:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f7898c00640 1 -- 172.21.15.40:0/3687488687 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789000cdb0 con 0x7f789c0d16f0 2024-03-24T00:31:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a3c00640 1 -- 172.21.15.40:0/3687488687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789c1471d0 con 0x7f789c0d16f0 2024-03-24T00:31:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a3c00640 1 -- 172.21.15.40:0/3687488687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789c147790 con 0x7f789c0d16f0 2024-03-24T00:31:28.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.506+0000 7f78a3c00640 1 -- 172.21.15.40:0/3687488687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789c0cf470 con 0x7f789c0d16f0 2024-03-24T00:31:28.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.510+0000 7f7898c00640 1 -- 172.21.15.40:0/3687488687 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f789001a070 con 0x7f789c0d16f0 2024-03-24T00:31:28.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.510+0000 7f7898c00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f787c067660 0x7f787c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.510+0000 7f7898c00640 1 -- 172.21.15.40:0/3687488687 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7890085ca0 con 0x7f789c0d16f0 2024-03-24T00:31:28.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.510+0000 7f789be00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f787c067660 0x7f787c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.510+0000 7f789be00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f787c067660 0x7f787c069b20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7888007fd0 tx=0x7f7888009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:28.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.514+0000 7f7898c00640 1 -- 172.21.15.40:0/3687488687 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f78900569d0 con 0x7f789c0d16f0 2024-03-24T00:31:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.766+0000 7f78a3c00640 1 -- 172.21.15.40:0/3687488687 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f789c068170 con 0x7f789c0d16f0 2024-03-24T00:31:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.766+0000 7f7898c00640 1 -- 172.21.15.40:0/3687488687 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f78900562d0 con 0x7f789c0d16f0 2024-03-24T00:31:28.771 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.770+0000 7f78a3c00640 1 -- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f787c067660 msgr2=0x7f787c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.770+0000 7f78a3c00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f787c067660 0x7f787c069b20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7888007fd0 tx=0x7f7888009040 comp rx=0 tx=0).stop 2024-03-24T00:31:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.770+0000 7f78a3c00640 1 -- 172.21.15.40:0/3687488687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f789c0d16f0 msgr2=0x7f789c146130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.770+0000 7f78a3c00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f789c0d16f0 0x7f789c146130 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f789000e930 tx=0x7f789000edf0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.770+0000 7f78a3c00640 1 -- 172.21.15.40:0/3687488687 shutdown_connections 2024-03-24T00:31:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.770+0000 7f78a3c00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f787c067660 0x7f787c069b20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.770+0000 7f78a3c00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f789c15bd40 0x7f789c165e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.770+0000 7f78a3c00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f789c15b960 0x7f789c14f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.770+0000 7f78a3c00640 1 --2- 172.21.15.40:0/3687488687 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f789c0d16f0 0x7f789c146130 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.770+0000 7f78a3c00640 1 -- 172.21.15.40:0/3687488687 >> 172.21.15.40:0/3687488687 conn(0x7f789c06ecf0 msgr2=0x7f789c0b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.770+0000 7f78a3c00640 1 -- 172.21.15.40:0/3687488687 shutdown_connections 2024-03-24T00:31:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.770+0000 7f78a3c00640 1 -- 172.21.15.40:0/3687488687 wait complete. 2024-03-24T00:31:28.792 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:28.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa889800640 1 Processor -- start 2024-03-24T00:31:28.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa889800640 1 -- start start 2024-03-24T00:31:28.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa889800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa88406e620 0x7fa8840b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa889800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa8840ccfa0 0x7fa8840cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa889800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa884070f60 con 0x7fa88406ea00 2024-03-24T00:31:28.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa889800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8840b4c00 con 0x7fa8840ccfa0 2024-03-24T00:31:28.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa889800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8840b3620 con 0x7fa88406e620 2024-03-24T00:31:28.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa883400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa8840ccfa0 0x7fa8840cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa87be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa88406ea00 0x7fa8840b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41126/0 (socket says 172.21.15.40:41126) 2024-03-24T00:31:28.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa87be00640 1 -- 172.21.15.40:0/139921576 learned_addr learned my addr 172.21.15.40:0/139921576 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:28.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa882a00640 1 --2- 172.21.15.40:0/139921576 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa88406e620 0x7fa8840b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa883400640 1 -- 172.21.15.40:0/139921576 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa88406e620 msgr2=0x7fa8840b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:28.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa883400640 1 --2- 172.21.15.40:0/139921576 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa88406e620 0x7fa8840b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa883400640 1 -- 172.21.15.40:0/139921576 >> v1:172.21.15.40:6789/0 conn(0x7fa88406ea00 legacy=0x7fa8840b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa883400640 1 -- 172.21.15.40:0/139921576 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8840b3900 con 0x7fa8840ccfa0 2024-03-24T00:31:28.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.930+0000 7fa882000640 1 -- 172.21.15.40:0/139921576 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973516485 0 0) 0x7fa884070f60 con 0x7fa88406ea00 2024-03-24T00:31:28.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.934+0000 7fa883400640 1 --2- 172.21.15.40:0/139921576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8840ccfa0 0x7fa8840cd3f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009d90 tx=0x7fa87c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2836edb3bbbd6c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:28.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.934+0000 7fa882000640 1 -- 172.21.15.40:0/139921576 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c0b2070 con 0x7fa8840ccfa0 2024-03-24T00:31:28.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.934+0000 7fa882000640 1 -- 172.21.15.40:0/139921576 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa87c0a5880 con 0x7fa8840ccfa0 2024-03-24T00:31:28.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.934+0000 7fa882000640 1 -- 172.21.15.40:0/139921576 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c0a5c50 con 0x7fa8840ccfa0 2024-03-24T00:31:28.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.934+0000 7fa889800640 1 -- 172.21.15.40:0/139921576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8840ccfa0 msgr2=0x7fa8840cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.934+0000 7fa889800640 1 --2- 172.21.15.40:0/139921576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8840ccfa0 0x7fa8840cd3f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009d90 tx=0x7fa87c01c8b0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.934+0000 7fa889800640 1 -- 172.21.15.40:0/139921576 shutdown_connections 2024-03-24T00:31:28.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.934+0000 7fa889800640 1 --2- 172.21.15.40:0/139921576 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8840ccfa0 0x7fa8840cd3f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.934+0000 7fa889800640 1 --2- 172.21.15.40:0/139921576 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa88406e620 0x7fa8840b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.934+0000 7fa889800640 1 -- 172.21.15.40:0/139921576 >> 172.21.15.40:0/139921576 conn(0x7fa8840b17b0 msgr2=0x7fa8840b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:28.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.934+0000 7fa889800640 1 -- 172.21.15.40:0/139921576 shutdown_connections 2024-03-24T00:31:28.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.934+0000 7fa889800640 1 -- 172.21.15.40:0/139921576 wait complete. 2024-03-24T00:31:28.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa889800640 1 Processor -- start 2024-03-24T00:31:28.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa889800640 1 -- start start 2024-03-24T00:31:28.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa889800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88406e620 0x7fa88415d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa889800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa88406ea00 0x7fa8840aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa889800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8840ccfa0 0x7fa884143680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa889800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa884144920 con 0x7fa88406e620 2024-03-24T00:31:28.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa889800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88406ee20 con 0x7fa8840ccfa0 2024-03-24T00:31:28.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa889800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88415c450 con 0x7fa88406ea00 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa882a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88406e620 0x7fa88415d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa882a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88406e620 0x7fa88415d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45568/0 (socket says 172.21.15.40:45568) 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa882a00640 1 -- 172.21.15.40:0/2235821586 learned_addr learned my addr 172.21.15.40:0/2235821586 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa87be00640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa88406ea00 0x7fa8840aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa883400640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8840ccfa0 0x7fa884143680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa882a00640 1 -- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa88406ea00 msgr2=0x7fa8840aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa882a00640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa88406ea00 0x7fa8840aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa882a00640 1 -- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8840ccfa0 msgr2=0x7fa884143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa882a00640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8840ccfa0 0x7fa884143680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa882a00640 1 -- 172.21.15.40:0/2235821586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88415c6f0 con 0x7fa88406e620 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa882a00640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88406e620 0x7fa88415d4b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa87400d800 tx=0x7fa87400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa87aa00640 1 -- 172.21.15.40:0/2235821586 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87400b6c0 con 0x7fa88406e620 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa87aa00640 1 -- 172.21.15.40:0/2235821586 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa874019070 con 0x7fa88406e620 2024-03-24T00:31:28.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa889800640 1 -- 172.21.15.40:0/2235821586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88415c9d0 con 0x7fa88406e620 2024-03-24T00:31:28.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa87aa00640 1 -- 172.21.15.40:0/2235821586 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87400ba60 con 0x7fa88406e620 2024-03-24T00:31:28.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.938+0000 7fa889800640 1 -- 172.21.15.40:0/2235821586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa88415cf90 con 0x7fa88406e620 2024-03-24T00:31:28.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.942+0000 7fa889800640 1 -- 172.21.15.40:0/2235821586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa884068170 con 0x7fa88406e620 2024-03-24T00:31:28.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.942+0000 7fa87aa00640 1 -- 172.21.15.40:0/2235821586 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa87400bc00 con 0x7fa88406e620 2024-03-24T00:31:28.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.942+0000 7fa87aa00640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa860067660 0x7fa860069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:28.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.942+0000 7fa87aa00640 1 -- 172.21.15.40:0/2235821586 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa874085d90 con 0x7fa88406e620 2024-03-24T00:31:28.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.942+0000 7fa87be00640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa860067660 0x7fa860069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:28.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.942+0000 7fa87be00640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa860067660 0x7fa860069b20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa88413b1b0 tx=0x7fa86c007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:28.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:28.946+0000 7fa87aa00640 1 -- 172.21.15.40:0/2235821586 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa874056900 con 0x7fa88406e620 2024-03-24T00:31:29.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.110+0000 7fa889800640 1 -- 172.21.15.40:0/2235821586 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa884072f50 con 0x7fa88406e620 2024-03-24T00:31:29.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.110+0000 7fa87aa00640 1 -- 172.21.15.40:0/2235821586 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7fa874056200 con 0x7fa88406e620 2024-03-24T00:31:29.115 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:29.115 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.114+0000 7fa889800640 1 -- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa860067660 msgr2=0x7fa860069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.114+0000 7fa889800640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa860067660 0x7fa860069b20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa88413b1b0 tx=0x7fa86c007d60 comp rx=0 tx=0).stop 2024-03-24T00:31:29.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.114+0000 7fa889800640 1 -- 172.21.15.40:0/2235821586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88406e620 msgr2=0x7fa88415d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.114+0000 7fa889800640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88406e620 0x7fa88415d4b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa87400d800 tx=0x7fa87400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.114+0000 7fa889800640 1 -- 172.21.15.40:0/2235821586 shutdown_connections 2024-03-24T00:31:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.114+0000 7fa889800640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa860067660 0x7fa860069b20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.114+0000 7fa889800640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8840ccfa0 0x7fa884143680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.114+0000 7fa889800640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa88406ea00 0x7fa8840aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.114+0000 7fa889800640 1 --2- 172.21.15.40:0/2235821586 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa88406e620 0x7fa88415d4b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.114+0000 7fa889800640 1 -- 172.21.15.40:0/2235821586 >> 172.21.15.40:0/2235821586 conn(0x7fa8840b17b0 msgr2=0x7fa88415ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.114+0000 7fa889800640 1 -- 172.21.15.40:0/2235821586 shutdown_connections 2024-03-24T00:31:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.114+0000 7fa889800640 1 -- 172.21.15.40:0/2235821586 wait complete. 2024-03-24T00:31:29.145 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:29.145 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:29.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.310+0000 7f819a600640 1 Processor -- start 2024-03-24T00:31:29.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.310+0000 7f819a600640 1 -- start start 2024-03-24T00:31:29.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.314+0000 7f819a600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f819406c950 0x7f8194070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:29.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.314+0000 7f819a600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819406d0f0 con 0x7f819415df80 2024-03-24T00:31:29.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.314+0000 7f819a600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819406b280 con 0x7f81940d2270 2024-03-24T00:31:29.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.314+0000 7f819a600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f819406b890 con 0x7f819406c950 2024-03-24T00:31:29.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.314+0000 7f8193400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f819415df80 0x7f81940d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41140/0 (socket says 172.21.15.40:41140) 2024-03-24T00:31:29.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.314+0000 7f8193400640 1 -- 172.21.15.40:0/1967568771 learned_addr learned my addr 172.21.15.40:0/1967568771 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:29.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.314+0000 7f8193e00640 1 --2- 172.21.15.40:0/1967568771 >> v2:172.21.15.40:3300/0 conn(0x7f819406c950 0x7f8194070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:29.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.314+0000 7f8192a00640 1 -- 172.21.15.40:0/1967568771 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066125303 0 0) 0x7f819406d0f0 con 0x7f819415df80 2024-03-24T00:31:29.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.314+0000 7f8193e00640 1 -- 172.21.15.40:0/1967568771 >> v1:172.21.15.53:6789/0 conn(0x7f81940d2270 legacy=0x7f8194161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.314+0000 7f8193e00640 1 -- 172.21.15.40:0/1967568771 >> v1:172.21.15.40:6789/0 conn(0x7f819415df80 legacy=0x7f81940d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.314+0000 7f8193e00640 1 -- 172.21.15.40:0/1967568771 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819406ce10 con 0x7f819406c950 2024-03-24T00:31:29.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f8193e00640 1 --2- 172.21.15.40:0/1967568771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819406c950 0x7f8194070ff0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f819406ba10 tx=0x7f818c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3e828adae09682 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:29.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f8192a00640 1 -- 172.21.15.40:0/1967568771 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c0b2070 con 0x7f819406c950 2024-03-24T00:31:29.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f8192a00640 1 -- 172.21.15.40:0/1967568771 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818c0a59a0 con 0x7f819406c950 2024-03-24T00:31:29.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f8192a00640 1 -- 172.21.15.40:0/1967568771 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c0a5d70 con 0x7f819406c950 2024-03-24T00:31:29.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f819a600640 1 -- 172.21.15.40:0/1967568771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819406c950 msgr2=0x7f8194070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f819a600640 1 --2- 172.21.15.40:0/1967568771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819406c950 0x7f8194070ff0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f819406ba10 tx=0x7f818c01c920 comp rx=0 tx=0).stop 2024-03-24T00:31:29.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f819a600640 1 -- 172.21.15.40:0/1967568771 shutdown_connections 2024-03-24T00:31:29.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f819a600640 1 --2- 172.21.15.40:0/1967568771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819406c950 0x7f8194070ff0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f819a600640 1 -- 172.21.15.40:0/1967568771 >> 172.21.15.40:0/1967568771 conn(0x7f819406ecf0 msgr2=0x7f819406f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:29.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f819a600640 1 -- 172.21.15.40:0/1967568771 shutdown_connections 2024-03-24T00:31:29.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f819a600640 1 -- 172.21.15.40:0/1967568771 wait complete. 2024-03-24T00:31:29.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f819a600640 1 Processor -- start 2024-03-24T00:31:29.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.318+0000 7f819a600640 1 -- start start 2024-03-24T00:31:29.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f819a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819406c950 0x7f8194154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:29.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f819a600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81940d2270 0x7f8194160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:29.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f819a600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f819415df80 0x7f8194160e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:29.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f819a600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819416bfa0 con 0x7f819406c950 2024-03-24T00:31:29.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f819a600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819416be20 con 0x7f81940d2270 2024-03-24T00:31:29.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f819a600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819416c120 con 0x7f819415df80 2024-03-24T00:31:29.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f8193400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819406c950 0x7f8194154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:29.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f8193400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819406c950 0x7f8194154960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45588/0 (socket says 172.21.15.40:45588) 2024-03-24T00:31:29.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f8193400640 1 -- 172.21.15.40:0/1328666637 learned_addr learned my addr 172.21.15.40:0/1328666637 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:29.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f818aa00640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81940d2270 0x7f8194160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:29.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f8193e00640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f819415df80 0x7f8194160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:29.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f8193400640 1 -- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f819415df80 msgr2=0x7f8194160e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f8193400640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f819415df80 0x7f8194160e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f8193400640 1 -- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81940d2270 msgr2=0x7f8194160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f8193400640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81940d2270 0x7f8194160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f8193400640 1 -- 172.21.15.40:0/1328666637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819415aff0 con 0x7f819406c950 2024-03-24T00:31:29.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f818aa00640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81940d2270 0x7f8194160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:29.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f8193e00640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f819415df80 0x7f8194160e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:29.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f8193400640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819406c950 0x7f8194154960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f818400c410 tx=0x7f818400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:29.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f818be00640 1 -- 172.21.15.40:0/1328666637 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81840074e0 con 0x7f819406c950 2024-03-24T00:31:29.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f818be00640 1 -- 172.21.15.40:0/1328666637 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81840047f0 con 0x7f819406c950 2024-03-24T00:31:29.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f819a600640 1 -- 172.21.15.40:0/1328666637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819415b2d0 con 0x7f819406c950 2024-03-24T00:31:29.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f818be00640 1 -- 172.21.15.40:0/1328666637 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184007680 con 0x7f819406c950 2024-03-24T00:31:29.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f819a600640 1 -- 172.21.15.40:0/1328666637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819415b890 con 0x7f819406c950 2024-03-24T00:31:29.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f819a600640 1 -- 172.21.15.40:0/1328666637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8194068170 con 0x7f819406c950 2024-03-24T00:31:29.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.322+0000 7f818be00640 1 -- 172.21.15.40:0/1328666637 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8184011070 con 0x7f819406c950 2024-03-24T00:31:29.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.326+0000 7f818be00640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8170067660 0x7f8170069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:29.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.326+0000 7f818be00640 1 -- 172.21.15.40:0/1328666637 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8184085530 con 0x7f819406c950 2024-03-24T00:31:29.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.326+0000 7f818aa00640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8170067660 0x7f8170069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:29.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.326+0000 7f818aa00640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8170067660 0x7f8170069b20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f81940c8860 tx=0x7f817c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:29.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.334+0000 7f818be00640 1 -- 172.21.15.40:0/1328666637 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f81840561b0 con 0x7f819406c950 2024-03-24T00:31:29.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.498+0000 7f819a600640 1 -- 172.21.15.40:0/1328666637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f81940cf9d0 con 0x7f819406c950 2024-03-24T00:31:29.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.498+0000 7f818be00640 1 -- 172.21.15.40:0/1328666637 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8184055ab0 con 0x7f819406c950 2024-03-24T00:31:29.503 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.502+0000 7f819a600640 1 -- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8170067660 msgr2=0x7f8170069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.502+0000 7f819a600640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8170067660 0x7f8170069b20 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f81940c8860 tx=0x7f817c009040 comp rx=0 tx=0).stop 2024-03-24T00:31:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.502+0000 7f819a600640 1 -- 172.21.15.40:0/1328666637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819406c950 msgr2=0x7f8194154960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.502+0000 7f819a600640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819406c950 0x7f8194154960 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f818400c410 tx=0x7f818400c8d0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.502+0000 7f819a600640 1 -- 172.21.15.40:0/1328666637 shutdown_connections 2024-03-24T00:31:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.502+0000 7f819a600640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8170067660 0x7f8170069b20 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.502+0000 7f819a600640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f819415df80 0x7f8194160e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.502+0000 7f819a600640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81940d2270 0x7f8194160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.502+0000 7f819a600640 1 --2- 172.21.15.40:0/1328666637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f819406c950 0x7f8194154960 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.502+0000 7f819a600640 1 -- 172.21.15.40:0/1328666637 >> 172.21.15.40:0/1328666637 conn(0x7f819406ecf0 msgr2=0x7f81940d1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.502+0000 7f819a600640 1 -- 172.21.15.40:0/1328666637 shutdown_connections 2024-03-24T00:31:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.502+0000 7f819a600640 1 -- 172.21.15.40:0/1328666637 wait complete. 2024-03-24T00:31:29.524 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:29.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7dba00640 1 Processor -- start 2024-03-24T00:31:29.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7dba00640 1 -- start start 2024-03-24T00:31:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7dba00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc7d406cb80 0x7fc7d40b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7dba00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7d40b4840 0x7fc7d4071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7dba00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d40b3260 con 0x7fc7d415ac60 2024-03-24T00:31:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7dba00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d40b3e70 con 0x7fc7d406cb80 2024-03-24T00:31:29.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7dba00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d40b3540 con 0x7fc7d40b4840 2024-03-24T00:31:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7d9000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc7d415ac60 0x7fc7d40b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41152/0 (socket says 172.21.15.40:41152) 2024-03-24T00:31:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7d9000640 1 -- 172.21.15.40:0/2684629643 learned_addr learned my addr 172.21.15.40:0/2684629643 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7cbe00640 1 --2- 172.21.15.40:0/2684629643 >> v2:172.21.15.40:3300/0 conn(0x7fc7d406cb80 0x7fc7d40b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7d9a00640 1 --2- 172.21.15.40:0/2684629643 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7d40b4840 0x7fc7d4071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7cbe00640 1 -- 172.21.15.40:0/2684629643 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7d40b4840 msgr2=0x7fc7d4071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7cbe00640 1 --2- 172.21.15.40:0/2684629643 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7d40b4840 0x7fc7d4071b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7cbe00640 1 -- 172.21.15.40:0/2684629643 >> v1:172.21.15.40:6789/0 conn(0x7fc7d415ac60 legacy=0x7fc7d40b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.670+0000 7fc7cbe00640 1 -- 172.21.15.40:0/2684629643 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d40b3820 con 0x7fc7d406cb80 2024-03-24T00:31:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.666+0000 7fc7d3e00640 1 -- 172.21.15.40:0/2684629643 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235856362 0 0) 0x7fc7d40b3260 con 0x7fc7d415ac60 2024-03-24T00:31:29.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.674+0000 7fc7cbe00640 1 --2- 172.21.15.40:0/2684629643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7d406cb80 0x7fc7d40b4300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc0097c0 tx=0x7fc7bc01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe92ddf0be20a16 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.674+0000 7fc7d3e00640 1 -- 172.21.15.40:0/2684629643 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc0b0070 con 0x7fc7d406cb80 2024-03-24T00:31:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.674+0000 7fc7d3e00640 1 -- 172.21.15.40:0/2684629643 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7bc0ab600 con 0x7fc7d406cb80 2024-03-24T00:31:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.674+0000 7fc7d3e00640 1 -- 172.21.15.40:0/2684629643 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7bc0ab9d0 con 0x7fc7d406cb80 2024-03-24T00:31:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.674+0000 7fc7dba00640 1 -- 172.21.15.40:0/2684629643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7d406cb80 msgr2=0x7fc7d40b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.674+0000 7fc7dba00640 1 --2- 172.21.15.40:0/2684629643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7d406cb80 0x7fc7d40b4300 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc0097c0 tx=0x7fc7bc01c410 comp rx=0 tx=0).stop 2024-03-24T00:31:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.674+0000 7fc7dba00640 1 -- 172.21.15.40:0/2684629643 shutdown_connections 2024-03-24T00:31:29.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.674+0000 7fc7dba00640 1 --2- 172.21.15.40:0/2684629643 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7d40b4840 0x7fc7d4071b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.674+0000 7fc7dba00640 1 --2- 172.21.15.40:0/2684629643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7d406cb80 0x7fc7d40b4300 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.674+0000 7fc7dba00640 1 -- 172.21.15.40:0/2684629643 >> 172.21.15.40:0/2684629643 conn(0x7fc7d40b1660 msgr2=0x7fc7d40b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.674+0000 7fc7dba00640 1 -- 172.21.15.40:0/2684629643 shutdown_connections 2024-03-24T00:31:29.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.674+0000 7fc7dba00640 1 -- 172.21.15.40:0/2684629643 wait complete. 2024-03-24T00:31:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7dba00640 1 Processor -- start 2024-03-24T00:31:29.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7dba00640 1 -- start start 2024-03-24T00:31:29.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7dba00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7d406cb80 0x7fc7d414f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:29.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7dba00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7d40b4840 0x7fc7d41479f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:29.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7dba00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7d415ac60 0x7fc7d4144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7dba00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d406d730 con 0x7fc7d415ac60 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7dba00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d406d5b0 con 0x7fc7d406cb80 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7dba00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d406d8b0 con 0x7fc7d40b4840 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7d415ac60 0x7fc7d4144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7d415ac60 0x7fc7d4144320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45600/0 (socket says 172.21.15.40:45600) 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d9a00640 1 -- 172.21.15.40:0/3854470130 learned_addr learned my addr 172.21.15.40:0/3854470130 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7cbe00640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7d40b4840 0x7fc7d41479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d9000640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7d406cb80 0x7fc7d414f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d9a00640 1 -- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7d40b4840 msgr2=0x7fc7d41479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d9a00640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7d40b4840 0x7fc7d41479f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d9a00640 1 -- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7d406cb80 msgr2=0x7fc7d414f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d9a00640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7d406cb80 0x7fc7d414f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d9a00640 1 -- 172.21.15.40:0/3854470130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d414fb90 con 0x7fc7d415ac60 2024-03-24T00:31:29.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d9a00640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7d415ac60 0x7fc7d4144320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc007e40 tx=0x7fc7cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:29.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d1600640 1 -- 172.21.15.40:0/3854470130 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc014620 con 0x7fc7d415ac60 2024-03-24T00:31:29.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d1600640 1 -- 172.21.15.40:0/3854470130 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc019070 con 0x7fc7d415ac60 2024-03-24T00:31:29.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7dba00640 1 -- 172.21.15.40:0/3854470130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d414fe70 con 0x7fc7d415ac60 2024-03-24T00:31:29.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.678+0000 7fc7d1600640 1 -- 172.21.15.40:0/3854470130 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc014a20 con 0x7fc7d415ac60 2024-03-24T00:31:29.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.682+0000 7fc7dba00640 1 -- 172.21.15.40:0/3854470130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d414e360 con 0x7fc7d415ac60 2024-03-24T00:31:29.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.682+0000 7fc7dba00640 1 -- 172.21.15.40:0/3854470130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d4150390 con 0x7fc7d415ac60 2024-03-24T00:31:29.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.682+0000 7fc7d1600640 1 -- 172.21.15.40:0/3854470130 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc7cc014bc0 con 0x7fc7d415ac60 2024-03-24T00:31:29.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.682+0000 7fc7d1600640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7b0067690 0x7fc7b0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:29.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.682+0000 7fc7d1600640 1 -- 172.21.15.40:0/3854470130 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7cc086220 con 0x7fc7d415ac60 2024-03-24T00:31:29.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.682+0000 7fc7d9000640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7b0067690 0x7fc7b0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:29.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.682+0000 7fc7d9000640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7b0067690 0x7fc7b0069b50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4000f70 tx=0x7fc7c4007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:29.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.690+0000 7fc7d1600640 1 -- 172.21.15.40:0/3854470130 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc7d4150390 con 0x7fc7d415ac60 2024-03-24T00:31:29.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.854+0000 7fc7dba00640 1 -- 172.21.15.40:0/3854470130 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc7d4068170 con 0x7fc7d415ac60 2024-03-24T00:31:29.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.854+0000 7fc7d1600640 1 -- 172.21.15.40:0/3854470130 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc7cc056e00 con 0x7fc7d415ac60 2024-03-24T00:31:29.859 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:29.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.858+0000 7fc7dba00640 1 -- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7b0067690 msgr2=0x7fc7b0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.858+0000 7fc7dba00640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7b0067690 0x7fc7b0069b50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4000f70 tx=0x7fc7c4007c10 comp rx=0 tx=0).stop 2024-03-24T00:31:29.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.858+0000 7fc7dba00640 1 -- 172.21.15.40:0/3854470130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7d415ac60 msgr2=0x7fc7d4144320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:29.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.858+0000 7fc7dba00640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7d415ac60 0x7fc7d4144320 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc007e40 tx=0x7fc7cc00e4b0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.858+0000 7fc7dba00640 1 -- 172.21.15.40:0/3854470130 shutdown_connections 2024-03-24T00:31:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.858+0000 7fc7dba00640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7b0067690 0x7fc7b0069b50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.858+0000 7fc7dba00640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7d415ac60 0x7fc7d4144320 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.858+0000 7fc7dba00640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7d40b4840 0x7fc7d41479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.858+0000 7fc7dba00640 1 --2- 172.21.15.40:0/3854470130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7d406cb80 0x7fc7d414f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.858+0000 7fc7dba00640 1 -- 172.21.15.40:0/3854470130 >> 172.21.15.40:0/3854470130 conn(0x7fc7d40b1660 msgr2=0x7fc7d40cfde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.858+0000 7fc7dba00640 1 -- 172.21.15.40:0/3854470130 shutdown_connections 2024-03-24T00:31:29.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:29.858+0000 7fc7dba00640 1 -- 172.21.15.40:0/3854470130 wait complete. 2024-03-24T00:31:29.879 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:30.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.018+0000 7f4809200640 1 Processor -- start 2024-03-24T00:31:30.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.022+0000 7f4809200640 1 -- start start 2024-03-24T00:31:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.022+0000 7f4809200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48040ae8e0 0x7f48040728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.022+0000 7f4809200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48040aed00 con 0x7f480415ffd0 2024-03-24T00:31:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.022+0000 7f4809200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4804071de0 con 0x7f480415bd00 2024-03-24T00:31:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.022+0000 7f4809200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48040cea50 con 0x7f48040ae8e0 2024-03-24T00:31:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.022+0000 7f4802800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48040ae8e0 0x7f48040728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.022+0000 7f4801400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f480415ffd0 0x7f480406cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41162/0 (socket says 172.21.15.40:41162) 2024-03-24T00:31:30.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.022+0000 7f4801400640 1 -- 172.21.15.40:0/1938278261 learned_addr learned my addr 172.21.15.40:0/1938278261 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.022+0000 7f4802800640 1 -- 172.21.15.40:0/1938278261 >> v1:172.21.15.53:6789/0 conn(0x7f480415bd00 legacy=0x7f480415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.022+0000 7f4802800640 1 -- 172.21.15.40:0/1938278261 >> v1:172.21.15.40:6789/0 conn(0x7f480415ffd0 legacy=0x7f480406cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.022+0000 7f4802800640 1 -- 172.21.15.40:0/1938278261 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48040ced30 con 0x7f48040ae8e0 2024-03-24T00:31:30.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.022+0000 7f4800a00640 1 -- 172.21.15.40:0/1938278261 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706985055 0 0) 0x7f48040aed00 con 0x7f480415ffd0 2024-03-24T00:31:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4802800640 1 --2- 172.21.15.40:0/1938278261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48040ae8e0 0x7f48040728d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f47fc009d90 tx=0x7f47fc01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9d1e5a15f5934a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4800a00640 1 -- 172.21.15.40:0/1938278261 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc0b2070 con 0x7f48040ae8e0 2024-03-24T00:31:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4800a00640 1 -- 172.21.15.40:0/1938278261 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47fc0a58c0 con 0x7f48040ae8e0 2024-03-24T00:31:30.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4800a00640 1 -- 172.21.15.40:0/1938278261 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc0a5c90 con 0x7f48040ae8e0 2024-03-24T00:31:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4809200640 1 -- 172.21.15.40:0/1938278261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48040ae8e0 msgr2=0x7f48040728d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4809200640 1 --2- 172.21.15.40:0/1938278261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48040ae8e0 0x7f48040728d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f47fc009d90 tx=0x7f47fc01c930 comp rx=0 tx=0).stop 2024-03-24T00:31:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4809200640 1 -- 172.21.15.40:0/1938278261 shutdown_connections 2024-03-24T00:31:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4809200640 1 --2- 172.21.15.40:0/1938278261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48040ae8e0 0x7f48040728d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4809200640 1 -- 172.21.15.40:0/1938278261 >> 172.21.15.40:0/1938278261 conn(0x7f48040b1660 msgr2=0x7f48040b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:30.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4809200640 1 -- 172.21.15.40:0/1938278261 shutdown_connections 2024-03-24T00:31:30.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4809200640 1 -- 172.21.15.40:0/1938278261 wait complete. 2024-03-24T00:31:30.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4809200640 1 Processor -- start 2024-03-24T00:31:30.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.026+0000 7f4809200640 1 -- start start 2024-03-24T00:31:30.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4809200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48040ae8e0 0x7f4804155dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4809200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f480415bd00 0x7f480414f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4809200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480415ffd0 0x7f48041542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4809200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480415c0e0 con 0x7f480415ffd0 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4809200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48040cf660 con 0x7f48040ae8e0 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4809200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f480416a420 con 0x7f480415bd00 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4802800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480415ffd0 0x7f48041542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4802800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480415ffd0 0x7f48041542b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45618/0 (socket says 172.21.15.40:45618) 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4802800640 1 -- 172.21.15.40:0/3424968332 learned_addr learned my addr 172.21.15.40:0/3424968332 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4801400640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f480415bd00 0x7f480414f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4801e00640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48040ae8e0 0x7f4804155dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4802800640 1 -- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f480415bd00 msgr2=0x7f480414f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4802800640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f480415bd00 0x7f480414f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4802800640 1 -- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48040ae8e0 msgr2=0x7f4804155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4802800640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48040ae8e0 0x7f4804155dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4802800640 1 -- 172.21.15.40:0/3424968332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4804156310 con 0x7f480415ffd0 2024-03-24T00:31:30.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4802800640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480415ffd0 0x7f48041542b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f47fc009d90 tx=0x7f47fc0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:30.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f47fa000640 1 -- 172.21.15.40:0/3424968332 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc0bd030 con 0x7f480415ffd0 2024-03-24T00:31:30.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f47fa000640 1 -- 172.21.15.40:0/3424968332 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47fc0b8530 con 0x7f480415ffd0 2024-03-24T00:31:30.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f47fa000640 1 -- 172.21.15.40:0/3424968332 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc0b2040 con 0x7f480415ffd0 2024-03-24T00:31:30.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4809200640 1 -- 172.21.15.40:0/3424968332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4804154d30 con 0x7f480415ffd0 2024-03-24T00:31:30.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4809200640 1 -- 172.21.15.40:0/3424968332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48041552f0 con 0x7f480415ffd0 2024-03-24T00:31:30.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f4809200640 1 -- 172.21.15.40:0/3424968332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4804068170 con 0x7f480415ffd0 2024-03-24T00:31:30.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.030+0000 7f47fa000640 1 -- 172.21.15.40:0/3424968332 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f47fc0c0070 con 0x7f480415ffd0 2024-03-24T00:31:30.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.034+0000 7f47fa000640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f47e4067660 0x7f47e4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.034+0000 7f47fa000640 1 -- 172.21.15.40:0/3424968332 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47fc120470 con 0x7f480415ffd0 2024-03-24T00:31:30.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.034+0000 7f4801e00640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f47e4067660 0x7f47e4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.034+0000 7f4801e00640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f47e4067660 0x7f47e4069b20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f47f4007fd0 tx=0x7f47f4007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:30.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.038+0000 7f47fa000640 1 -- 172.21.15.40:0/3424968332 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f47fc0c7070 con 0x7f480415ffd0 2024-03-24T00:31:30.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 -- 172.21.15.40:0/3424968332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f48041593a0 con 0x7f480415ffd0 2024-03-24T00:31:30.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f47fa000640 1 -- 172.21.15.40:0/3424968332 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f47fc016520 con 0x7f480415ffd0 2024-03-24T00:31:30.205 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:30.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 -- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f47e4067660 msgr2=0x7f47e4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f47e4067660 0x7f47e4069b20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f47f4007fd0 tx=0x7f47f4007e40 comp rx=0 tx=0).stop 2024-03-24T00:31:30.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 -- 172.21.15.40:0/3424968332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480415ffd0 msgr2=0x7f48041542b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480415ffd0 0x7f48041542b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f47fc009d90 tx=0x7f47fc0196d0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 -- 172.21.15.40:0/3424968332 shutdown_connections 2024-03-24T00:31:30.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f47e4067660 0x7f47e4069b20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480415ffd0 0x7f48041542b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f480415bd00 0x7f480414f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 --2- 172.21.15.40:0/3424968332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48040ae8e0 0x7f4804155dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 -- 172.21.15.40:0/3424968332 >> 172.21.15.40:0/3424968332 conn(0x7f48040b1660 msgr2=0x7f48040cfc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:30.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 -- 172.21.15.40:0/3424968332 shutdown_connections 2024-03-24T00:31:30.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.202+0000 7f4809200640 1 -- 172.21.15.40:0/3424968332 wait complete. 2024-03-24T00:31:30.227 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69d3c00640 1 Processor -- start 2024-03-24T00:31:30.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69d3c00640 1 -- start start 2024-03-24T00:31:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69d3c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f69cc141bc0 0x7f69cc0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69d3c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69cc0ce630 0x7f69cc071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69d3c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc070e00 con 0x7f69cc13ec80 2024-03-24T00:31:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69d3c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69cc0cee20 con 0x7f69cc141bc0 2024-03-24T00:31:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69d3c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0cf100 con 0x7f69cc0ce630 2024-03-24T00:31:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69d1200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f69cc13ec80 0x7f69cc1414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41178/0 (socket says 172.21.15.40:41178) 2024-03-24T00:31:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69d1200640 1 -- 172.21.15.40:0/890681154 learned_addr learned my addr 172.21.15.40:0/890681154 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69d1c00640 1 --2- 172.21.15.40:0/890681154 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69cc0ce630 0x7f69cc071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69c3e00640 1 --2- 172.21.15.40:0/890681154 >> v2:172.21.15.53:3300/0 conn(0x7f69cc141bc0 0x7f69cc0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406101534 0 0) 0x7f69cc070e00 con 0x7f69cc13ec80 2024-03-24T00:31:30.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.366+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b0003530 con 0x7f69cc13ec80 2024-03-24T00:31:30.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2841893708 0 0) 0x7f69b0003530 con 0x7f69cc13ec80 2024-03-24T00:31:30.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69cc0ce630 msgr2=0x7f69cc071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:30.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 --2- 172.21.15.40:0/890681154 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69cc0ce630 0x7f69cc071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 >> v2:172.21.15.53:3300/0 conn(0x7f69cc141bc0 msgr2=0x7f69cc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:30.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 --2- 172.21.15.40:0/890681154 >> v2:172.21.15.53:3300/0 conn(0x7f69cc141bc0 0x7f69cc0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc0cf3e0 con 0x7f69cc13ec80 2024-03-24T00:31:30.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f69bc001680 con 0x7f69cc13ec80 2024-03-24T00:31:30.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 >> v1:172.21.15.40:6789/0 conn(0x7f69cc13ec80 legacy=0x7f69cc1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 --2- 172.21.15.40:0/890681154 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69b00039c0 0x7f69b0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 --2- 172.21.15.40:0/890681154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69b00043b0 0x7f69b0004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 --2- 172.21.15.40:0/890681154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69b0004f00 0x7f69b000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0cee20 con 0x7f69b0004f00 2024-03-24T00:31:30.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0cf100 con 0x7f69b00043b0 2024-03-24T00:31:30.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b000bf10 con 0x7f69b00039c0 2024-03-24T00:31:30.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69d1c00640 1 --2- 172.21.15.40:0/890681154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69b0004f00 0x7f69b000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69d1200640 1 --2- 172.21.15.40:0/890681154 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69b00039c0 0x7f69b0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69c3e00640 1 --2- 172.21.15.40:0/890681154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69b00043b0 0x7f69b0004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69d1c00640 1 -- 172.21.15.40:0/890681154 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69b00039c0 msgr2=0x7f69b0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:30.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69d1c00640 1 --2- 172.21.15.40:0/890681154 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69b00039c0 0x7f69b0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69d1c00640 1 -- 172.21.15.40:0/890681154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69b00043b0 msgr2=0x7f69b0004890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:30.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69d1c00640 1 --2- 172.21.15.40:0/890681154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69b00043b0 0x7f69b0004890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69d1c00640 1 -- 172.21.15.40:0/890681154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b0003530 con 0x7f69b0004f00 2024-03-24T00:31:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.370+0000 7f69d1c00640 1 --2- 172.21.15.40:0/890681154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69b0004f00 0x7f69b000b9d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f69c4009f00 tx=0x7f69c401cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80dad86cbc046494 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c40b2070 con 0x7f69b0004f00 2024-03-24T00:31:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c40a5c10 con 0x7f69b0004f00 2024-03-24T00:31:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69cbe00640 1 -- 172.21.15.40:0/890681154 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c4009bd0 con 0x7f69b0004f00 2024-03-24T00:31:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 -- 172.21.15.40:0/890681154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69b0004f00 msgr2=0x7f69b000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 --2- 172.21.15.40:0/890681154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69b0004f00 0x7f69b000b9d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f69c4009f00 tx=0x7f69c401cbc0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d1c00640 1 -- 172.21.15.40:0/890681154 reap_dead start 2024-03-24T00:31:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 -- 172.21.15.40:0/890681154 shutdown_connections 2024-03-24T00:31:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 -- 172.21.15.40:0/890681154 >> 172.21.15.40:0/890681154 conn(0x7f69cc0b17b0 msgr2=0x7f69cc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 -- 172.21.15.40:0/890681154 shutdown_connections 2024-03-24T00:31:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 -- 172.21.15.40:0/890681154 wait complete. 2024-03-24T00:31:30.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 Processor -- start 2024-03-24T00:31:30.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 -- start start 2024-03-24T00:31:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69b0004f00 0x7f69cc156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69cc1588a0 0x7f69cc1f71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69cc158d20 0x7f69cc1f7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc071300 con 0x7f69cc158d20 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0ceac0 con 0x7f69b0004f00 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d3c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc071480 con 0x7f69cc1588a0 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69cc158d20 0x7f69cc1f7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69cc158d20 0x7f69cc1f7700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45640/0 (socket says 172.21.15.40:45640) 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d1c00640 1 -- 172.21.15.40:0/1257585089 learned_addr learned my addr 172.21.15.40:0/1257585089 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69c3e00640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69cc1588a0 0x7f69cc1f71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.374+0000 7f69d1200640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69b0004f00 0x7f69cc156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69d1c00640 1 -- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69cc1588a0 msgr2=0x7f69cc1f71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69d1c00640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69cc1588a0 0x7f69cc1f71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69d1c00640 1 -- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69b0004f00 msgr2=0x7f69cc156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69d1c00640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69b0004f00 0x7f69cc156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69d1c00640 1 -- 172.21.15.40:0/1257585089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc140e80 con 0x7f69cc158d20 2024-03-24T00:31:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69d1c00640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69cc158d20 0x7f69cc1f7700 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f69c40b0040 tx=0x7f69c4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69c9600640 1 -- 172.21.15.40:0/1257585089 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c40b2040 con 0x7f69cc158d20 2024-03-24T00:31:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69c9600640 1 -- 172.21.15.40:0/1257585089 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c40bc070 con 0x7f69cc158d20 2024-03-24T00:31:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69d3c00640 1 -- 172.21.15.40:0/1257585089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69cc141120 con 0x7f69cc158d20 2024-03-24T00:31:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69c9600640 1 -- 172.21.15.40:0/1257585089 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c4009bd0 con 0x7f69cc158d20 2024-03-24T00:31:30.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69d3c00640 1 -- 172.21.15.40:0/1257585089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69cc157030 con 0x7f69cc158d20 2024-03-24T00:31:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69d3c00640 1 -- 172.21.15.40:0/1257585089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69cc068170 con 0x7f69cc158d20 2024-03-24T00:31:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69c9600640 1 -- 172.21.15.40:0/1257585089 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f69c40c1460 con 0x7f69cc158d20 2024-03-24T00:31:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69c9600640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f69a8067660 0x7f69a8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.378+0000 7f69c9600640 1 -- 172.21.15.40:0/1257585089 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69c4120c90 con 0x7f69cc158d20 2024-03-24T00:31:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.382+0000 7f69d1200640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f69a8067660 0x7f69a8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.382+0000 7f69d1200640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f69a8067660 0x7f69a8069b20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0015a0 tx=0x7f69bc007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:30.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.386+0000 7f69c9600640 1 -- 172.21.15.40:0/1257585089 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f69c4016910 con 0x7f69cc158d20 2024-03-24T00:31:30.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.550+0000 7f69d3c00640 1 -- 172.21.15.40:0/1257585089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69cc072b90 con 0x7f69cc158d20 2024-03-24T00:31:30.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.550+0000 7f69c9600640 1 -- 172.21.15.40:0/1257585089 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7f69c40f13c0 con 0x7f69cc158d20 2024-03-24T00:31:30.555 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:30.555 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:30.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.554+0000 7f69d3c00640 1 -- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f69a8067660 msgr2=0x7f69a8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.554+0000 7f69d3c00640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f69a8067660 0x7f69a8069b20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0015a0 tx=0x7f69bc007900 comp rx=0 tx=0).stop 2024-03-24T00:31:30.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.554+0000 7f69d3c00640 1 -- 172.21.15.40:0/1257585089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69cc158d20 msgr2=0x7f69cc1f7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.554+0000 7f69d3c00640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69cc158d20 0x7f69cc1f7700 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f69c40b0040 tx=0x7f69c4009750 comp rx=0 tx=0).stop 2024-03-24T00:31:30.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.554+0000 7f69d3c00640 1 -- 172.21.15.40:0/1257585089 shutdown_connections 2024-03-24T00:31:30.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.554+0000 7f69d3c00640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f69a8067660 0x7f69a8069b20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.554+0000 7f69d3c00640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f69cc158d20 0x7f69cc1f7700 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.554+0000 7f69d3c00640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f69cc1588a0 0x7f69cc1f71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.554+0000 7f69d3c00640 1 --2- 172.21.15.40:0/1257585089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69b0004f00 0x7f69cc156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.554+0000 7f69d3c00640 1 -- 172.21.15.40:0/1257585089 >> 172.21.15.40:0/1257585089 conn(0x7f69cc0b17b0 msgr2=0x7f69cc15de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:30.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.554+0000 7f69d3c00640 1 -- 172.21.15.40:0/1257585089 shutdown_connections 2024-03-24T00:31:30.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.554+0000 7f69d3c00640 1 -- 172.21.15.40:0/1257585089 wait complete. 2024-03-24T00:31:30.577 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:30.579 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:30.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.718+0000 7f9f36600640 1 Processor -- start 2024-03-24T00:31:30.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.718+0000 7f9f36600640 1 -- start start 2024-03-24T00:31:30.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f36600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f3015b960 0x7f9f30070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f36600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f300d16f0 0x7f9f300712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f36600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3015f970 con 0x7f9f3015bd40 2024-03-24T00:31:30.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f36600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f30160580 con 0x7f9f3015b960 2024-03-24T00:31:30.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f36600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3015fc50 con 0x7f9f300d16f0 2024-03-24T00:31:30.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f2f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f3015b960 0x7f9f30070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f2f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9f3015b960 0x7f9f30070d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45654/0 (socket says 172.21.15.40:45654) 2024-03-24T00:31:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f2f400640 1 -- 172.21.15.40:0/1281141128 learned_addr learned my addr 172.21.15.40:0/1281141128 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f2fe00640 1 --2- 172.21.15.40:0/1281141128 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f300d16f0 0x7f9f300712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f2f400640 1 -- 172.21.15.40:0/1281141128 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f300d16f0 msgr2=0x7f9f300712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:30.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f2f400640 1 --2- 172.21.15.40:0/1281141128 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f300d16f0 0x7f9f300712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f2f400640 1 -- 172.21.15.40:0/1281141128 >> v1:172.21.15.53:6789/0 conn(0x7f9f3015bd40 legacy=0x7f9f300d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.722+0000 7f9f2f400640 1 -- 172.21.15.40:0/1281141128 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3015ff30 con 0x7f9f3015b960 2024-03-24T00:31:30.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f2f400640 1 --2- 172.21.15.40:0/1281141128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f3015b960 0x7f9f30070d60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009870 tx=0x7f9f2001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c906dcb61ef04d4d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f2ea00640 1 -- 172.21.15.40:0/1281141128 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f200a7070 con 0x7f9f3015b960 2024-03-24T00:31:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f2ea00640 1 -- 172.21.15.40:0/1281141128 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f200b0070 con 0x7f9f3015b960 2024-03-24T00:31:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f2ea00640 1 -- 172.21.15.40:0/1281141128 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f200ab890 con 0x7f9f3015b960 2024-03-24T00:31:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 -- 172.21.15.40:0/1281141128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f3015b960 msgr2=0x7f9f30070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 --2- 172.21.15.40:0/1281141128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f3015b960 0x7f9f30070d60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009870 tx=0x7f9f2001c560 comp rx=0 tx=0).stop 2024-03-24T00:31:30.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 -- 172.21.15.40:0/1281141128 shutdown_connections 2024-03-24T00:31:30.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 --2- 172.21.15.40:0/1281141128 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f300d16f0 0x7f9f300712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 --2- 172.21.15.40:0/1281141128 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f3015b960 0x7f9f30070d60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 -- 172.21.15.40:0/1281141128 >> 172.21.15.40:0/1281141128 conn(0x7f9f3006ecf0 msgr2=0x7f9f3006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:30.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 -- 172.21.15.40:0/1281141128 shutdown_connections 2024-03-24T00:31:30.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 -- 172.21.15.40:0/1281141128 wait complete. 2024-03-24T00:31:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 Processor -- start 2024-03-24T00:31:30.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 -- start start 2024-03-24T00:31:30.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f300d16f0 0x7f9f300c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f3015b960 0x7f9f301598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f3015bd40 0x7f9f30159e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300ce5a0 con 0x7f9f3015b960 2024-03-24T00:31:30.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300ce420 con 0x7f9f3015bd40 2024-03-24T00:31:30.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.726+0000 7f9f36600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f300ce720 con 0x7f9f300d16f0 2024-03-24T00:31:30.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f3015b960 0x7f9f301598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f2f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f300d16f0 0x7f9f300c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f2fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f3015bd40 0x7f9f30159e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f3015b960 0x7f9f301598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45666/0 (socket says 172.21.15.40:45666) 2024-03-24T00:31:30.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27e00640 1 -- 172.21.15.40:0/2670814859 learned_addr learned my addr 172.21.15.40:0/2670814859 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:30.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27e00640 1 -- 172.21.15.40:0/2670814859 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f300d16f0 msgr2=0x7f9f300c7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27e00640 1 --2- 172.21.15.40:0/2670814859 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f300d16f0 0x7f9f300c7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27e00640 1 -- 172.21.15.40:0/2670814859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f3015bd40 msgr2=0x7f9f30159e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27e00640 1 --2- 172.21.15.40:0/2670814859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f3015bd40 0x7f9f30159e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27e00640 1 -- 172.21.15.40:0/2670814859 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3015d010 con 0x7f9f3015b960 2024-03-24T00:31:30.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27e00640 1 --2- 172.21.15.40:0/2670814859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f3015b960 0x7f9f301598d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800c3a0 tx=0x7f9f1800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:30.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27400640 1 -- 172.21.15.40:0/2670814859 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f180052d0 con 0x7f9f3015b960 2024-03-24T00:31:30.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27400640 1 -- 172.21.15.40:0/2670814859 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f18005470 con 0x7f9f3015b960 2024-03-24T00:31:30.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27400640 1 -- 172.21.15.40:0/2670814859 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18007260 con 0x7f9f3015b960 2024-03-24T00:31:30.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f36600640 1 -- 172.21.15.40:0/2670814859 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f301530e0 con 0x7f9f3015b960 2024-03-24T00:31:30.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f36600640 1 -- 172.21.15.40:0/2670814859 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f301536a0 con 0x7f9f3015b960 2024-03-24T00:31:30.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f36600640 1 -- 172.21.15.40:0/2670814859 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f30068170 con 0x7f9f3015b960 2024-03-24T00:31:30.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.730+0000 7f9f27400640 1 -- 172.21.15.40:0/2670814859 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f9f18007400 con 0x7f9f3015b960 2024-03-24T00:31:30.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.734+0000 7f9f27400640 1 --2- 172.21.15.40:0/2670814859 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9f0c0676e0 0x7f9f0c069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:30.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.734+0000 7f9f27400640 1 -- 172.21.15.40:0/2670814859 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f18084c70 con 0x7f9f3015b960 2024-03-24T00:31:30.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.734+0000 7f9f2f400640 1 --2- 172.21.15.40:0/2670814859 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9f0c0676e0 0x7f9f0c069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:30.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.734+0000 7f9f2f400640 1 --2- 172.21.15.40:0/2670814859 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9f0c0676e0 0x7f9f0c069ba0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9f200af000 tx=0x7f9f200a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:30.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.742+0000 7f9f27400640 1 -- 172.21.15.40:0/2670814859 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9f18055280 con 0x7f9f3015b960 2024-03-24T00:31:30.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.902+0000 7f9f36600640 1 -- 172.21.15.40:0/2670814859 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f3015dd10 con 0x7f9f3015b960 2024-03-24T00:31:30.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.902+0000 7f9f27400640 1 -- 172.21.15.40:0/2670814859 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9f18059320 con 0x7f9f3015b960 2024-03-24T00:31:30.907 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:30.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.906+0000 7f9f36600640 1 -- 172.21.15.40:0/2670814859 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9f0c0676e0 msgr2=0x7f9f0c069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.906+0000 7f9f36600640 1 --2- 172.21.15.40:0/2670814859 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9f0c0676e0 0x7f9f0c069ba0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9f200af000 tx=0x7f9f200a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.906+0000 7f9f36600640 1 -- 172.21.15.40:0/2670814859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f3015b960 msgr2=0x7f9f301598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.906+0000 7f9f36600640 1 --2- 172.21.15.40:0/2670814859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f3015b960 0x7f9f301598d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800c3a0 tx=0x7f9f1800c860 comp rx=0 tx=0).stop 2024-03-24T00:31:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.906+0000 7f9f36600640 1 -- 172.21.15.40:0/2670814859 shutdown_connections 2024-03-24T00:31:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.906+0000 7f9f36600640 1 --2- 172.21.15.40:0/2670814859 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9f0c0676e0 0x7f9f0c069ba0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.906+0000 7f9f36600640 1 --2- 172.21.15.40:0/2670814859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f3015bd40 0x7f9f30159e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.906+0000 7f9f36600640 1 --2- 172.21.15.40:0/2670814859 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9f3015b960 0x7f9f301598d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.906+0000 7f9f36600640 1 --2- 172.21.15.40:0/2670814859 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f300d16f0 0x7f9f300c7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.906+0000 7f9f36600640 1 -- 172.21.15.40:0/2670814859 >> 172.21.15.40:0/2670814859 conn(0x7f9f3006ecf0 msgr2=0x7f9f300b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.906+0000 7f9f36600640 1 -- 172.21.15.40:0/2670814859 shutdown_connections 2024-03-24T00:31:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:30.906+0000 7f9f36600640 1 -- 172.21.15.40:0/2670814859 wait complete. 2024-03-24T00:31:30.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:31.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.070+0000 7fdf92400640 1 Processor -- start 2024-03-24T00:31:31.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.070+0000 7fdf92400640 1 -- start start 2024-03-24T00:31:31.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.070+0000 7fdf92400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf8c15bb10 0x7fdf8c06f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.070+0000 7fdf92400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdf8c067600 0x7fdf8c161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.070+0000 7fdf92400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf8c067c60 con 0x7fdf8c06f570 2024-03-24T00:31:31.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.070+0000 7fdf92400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf8c06f9d0 con 0x7fdf8c15bb10 2024-03-24T00:31:31.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.070+0000 7fdf92400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf8c15ef60 con 0x7fdf8c067600 2024-03-24T00:31:31.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.070+0000 7fdf8b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf8c15bb10 0x7fdf8c06f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.070+0000 7fdf8b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdf8c15bb10 0x7fdf8c06f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45670/0 (socket says 172.21.15.40:45670) 2024-03-24T00:31:31.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.070+0000 7fdf8b400640 1 -- 172.21.15.40:0/547194624 learned_addr learned my addr 172.21.15.40:0/547194624 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:31.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.070+0000 7fdf8be00640 1 --2- 172.21.15.40:0/547194624 >> v2:172.21.15.53:3300/0 conn(0x7fdf8c067600 0x7fdf8c161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.074+0000 7fdf8b400640 1 -- 172.21.15.40:0/547194624 >> v2:172.21.15.53:3300/0 conn(0x7fdf8c067600 msgr2=0x7fdf8c161bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:31.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.074+0000 7fdf8b400640 1 --2- 172.21.15.40:0/547194624 >> v2:172.21.15.53:3300/0 conn(0x7fdf8c067600 0x7fdf8c161bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.074+0000 7fdf8b400640 1 -- 172.21.15.40:0/547194624 >> v1:172.21.15.53:6789/0 conn(0x7fdf8c06f570 legacy=0x7fdf8c066ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.074+0000 7fdf8b400640 1 -- 172.21.15.40:0/547194624 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8c15e910 con 0x7fdf8c15bb10 2024-03-24T00:31:31.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.074+0000 7fdf8b400640 1 --2- 172.21.15.40:0/547194624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8c15bb10 0x7fdf8c06f030 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c071140 tx=0x7fdf7c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f082ba6a6c9829 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:31.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.074+0000 7fdf8aa00640 1 -- 172.21.15.40:0/547194624 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c0ab430 con 0x7fdf8c15bb10 2024-03-24T00:31:31.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.074+0000 7fdf8aa00640 1 -- 172.21.15.40:0/547194624 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7c0b0070 con 0x7fdf8c15bb10 2024-03-24T00:31:31.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.074+0000 7fdf8aa00640 1 -- 172.21.15.40:0/547194624 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c0ab7f0 con 0x7fdf8c15bb10 2024-03-24T00:31:31.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.074+0000 7fdf92400640 1 -- 172.21.15.40:0/547194624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8c15bb10 msgr2=0x7fdf8c06f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.074+0000 7fdf92400640 1 --2- 172.21.15.40:0/547194624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8c15bb10 0x7fdf8c06f030 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c071140 tx=0x7fdf7c01c520 comp rx=0 tx=0).stop 2024-03-24T00:31:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 -- 172.21.15.40:0/547194624 shutdown_connections 2024-03-24T00:31:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 --2- 172.21.15.40:0/547194624 >> v2:172.21.15.53:3300/0 conn(0x7fdf8c067600 0x7fdf8c161bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 --2- 172.21.15.40:0/547194624 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8c15bb10 0x7fdf8c06f030 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 -- 172.21.15.40:0/547194624 >> 172.21.15.40:0/547194624 conn(0x7fdf8c0b27c0 msgr2=0x7fdf8c0b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 -- 172.21.15.40:0/547194624 shutdown_connections 2024-03-24T00:31:31.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 -- 172.21.15.40:0/547194624 wait complete. 2024-03-24T00:31:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 Processor -- start 2024-03-24T00:31:31.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 -- start start 2024-03-24T00:31:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf8c067600 0x7fdf8c139180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf8c06f570 0x7fdf8c150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8c15bb10 0x7fdf8c071b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c15f570 con 0x7fdf8c15bb10 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c15f3f0 con 0x7fdf8c06f570 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf92400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c16af50 con 0x7fdf8c067600 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf8be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8c15bb10 0x7fdf8c071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf8be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8c15bb10 0x7fdf8c071b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45682/0 (socket says 172.21.15.40:45682) 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf8be00640 1 -- 172.21.15.40:0/2943045092 learned_addr learned my addr 172.21.15.40:0/2943045092 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf8b400640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf8c067600 0x7fdf8c139180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf82a00640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf8c06f570 0x7fdf8c150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf8be00640 1 -- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf8c067600 msgr2=0x7fdf8c139180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf8be00640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf8c067600 0x7fdf8c139180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf8be00640 1 -- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf8c06f570 msgr2=0x7fdf8c150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf8be00640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf8c06f570 0x7fdf8c150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf8be00640 1 -- 172.21.15.40:0/2943045092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8c1396c0 con 0x7fdf8c15bb10 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf8be00640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8c15bb10 0x7fdf8c071b90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdf84009d30 tx=0x7fdf8400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.078+0000 7fdf83e00640 1 -- 172.21.15.40:0/2943045092 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf84018020 con 0x7fdf8c15bb10 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.082+0000 7fdf83e00640 1 -- 172.21.15.40:0/2943045092 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf84014480 con 0x7fdf8c15bb10 2024-03-24T00:31:31.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.082+0000 7fdf83e00640 1 -- 172.21.15.40:0/2943045092 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf84014870 con 0x7fdf8c15bb10 2024-03-24T00:31:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.082+0000 7fdf92400640 1 -- 172.21.15.40:0/2943045092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8c14f0a0 con 0x7fdf8c15bb10 2024-03-24T00:31:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.082+0000 7fdf92400640 1 -- 172.21.15.40:0/2943045092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8c14f340 con 0x7fdf8c15bb10 2024-03-24T00:31:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.082+0000 7fdf92400640 1 -- 172.21.15.40:0/2943045092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8c068170 con 0x7fdf8c15bb10 2024-03-24T00:31:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.082+0000 7fdf83e00640 1 -- 172.21.15.40:0/2943045092 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fdf8401b020 con 0x7fdf8c15bb10 2024-03-24T00:31:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.082+0000 7fdf83e00640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdf68067660 0x7fdf68069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.082+0000 7fdf83e00640 1 -- 172.21.15.40:0/2943045092 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf840862e0 con 0x7fdf8c15bb10 2024-03-24T00:31:31.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.082+0000 7fdf8b400640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdf68067660 0x7fdf68069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.082+0000 7fdf8b400640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdf68067660 0x7fdf68069b20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c0af000 tx=0x7fdf7c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:31.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.086+0000 7fdf83e00640 1 -- 172.21.15.40:0/2943045092 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdf84022070 con 0x7fdf8c15bb10 2024-03-24T00:31:31.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.250+0000 7fdf92400640 1 -- 172.21.15.40:0/2943045092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdf8c15c1b0 con 0x7fdf8c15bb10 2024-03-24T00:31:31.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.250+0000 7fdf83e00640 1 -- 172.21.15.40:0/2943045092 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdf84010040 con 0x7fdf8c15bb10 2024-03-24T00:31:31.253 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:31.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.254+0000 7fdf92400640 1 -- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdf68067660 msgr2=0x7fdf68069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.254+0000 7fdf92400640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdf68067660 0x7fdf68069b20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c0af000 tx=0x7fdf7c0a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:31.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.254+0000 7fdf92400640 1 -- 172.21.15.40:0/2943045092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8c15bb10 msgr2=0x7fdf8c071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.254+0000 7fdf92400640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8c15bb10 0x7fdf8c071b90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdf84009d30 tx=0x7fdf8400c490 comp rx=0 tx=0).stop 2024-03-24T00:31:31.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.254+0000 7fdf92400640 1 -- 172.21.15.40:0/2943045092 shutdown_connections 2024-03-24T00:31:31.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.254+0000 7fdf92400640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdf68067660 0x7fdf68069b20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.254+0000 7fdf92400640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdf8c15bb10 0x7fdf8c071b90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.254+0000 7fdf92400640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf8c06f570 0x7fdf8c150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.254+0000 7fdf92400640 1 --2- 172.21.15.40:0/2943045092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf8c067600 0x7fdf8c139180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.254+0000 7fdf92400640 1 -- 172.21.15.40:0/2943045092 >> 172.21.15.40:0/2943045092 conn(0x7fdf8c0b27c0 msgr2=0x7fdf8c0b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:31.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.254+0000 7fdf92400640 1 -- 172.21.15.40:0/2943045092 shutdown_connections 2024-03-24T00:31:31.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.254+0000 7fdf92400640 1 -- 172.21.15.40:0/2943045092 wait complete. 2024-03-24T00:31:31.276 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:31.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe516800640 1 Processor -- start 2024-03-24T00:31:31.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe516800640 1 -- start start 2024-03-24T00:31:31.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe516800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe510145700 0x7fe5100725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe516800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5100cd320 0x7fe5100cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe516800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe510066e80 0x7fe510067310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe516800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5100cf8e0 con 0x7fe510066e80 2024-03-24T00:31:31.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe516800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5100ceff0 con 0x7fe510145700 2024-03-24T00:31:31.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe516800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5100cdb70 con 0x7fe5100cd320 2024-03-24T00:31:31.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe50fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe510066e80 0x7fe510067310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe50fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe510066e80 0x7fe510067310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45690/0 (socket says 172.21.15.40:45690) 2024-03-24T00:31:31.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe50fe00640 1 -- 172.21.15.40:0/1818807559 learned_addr learned my addr 172.21.15.40:0/1818807559 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:31.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe50f400640 1 --2- 172.21.15.40:0/1818807559 >> v2:172.21.15.53:3300/0 conn(0x7fe510145700 0x7fe5100725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe50ea00640 1 --2- 172.21.15.40:0/1818807559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5100cd320 0x7fe5100cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe50fe00640 1 -- 172.21.15.40:0/1818807559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5100cd320 msgr2=0x7fe5100cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:31.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe50fe00640 1 --2- 172.21.15.40:0/1818807559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5100cd320 0x7fe5100cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe50fe00640 1 -- 172.21.15.40:0/1818807559 >> v2:172.21.15.53:3300/0 conn(0x7fe510145700 msgr2=0x7fe5100725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:31.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe50fe00640 1 --2- 172.21.15.40:0/1818807559 >> v2:172.21.15.53:3300/0 conn(0x7fe510145700 0x7fe5100725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.418+0000 7fe50fe00640 1 -- 172.21.15.40:0/1818807559 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5100cd890 con 0x7fe510066e80 2024-03-24T00:31:31.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe50fe00640 1 --2- 172.21.15.40:0/1818807559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe510066e80 0x7fe510067310 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe510071700 tx=0x7fe50801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48876e17447ce8a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:31.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe50e000640 1 -- 172.21.15.40:0/1818807559 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5080a5760 con 0x7fe510066e80 2024-03-24T00:31:31.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe50e000640 1 -- 172.21.15.40:0/1818807559 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5080a5900 con 0x7fe510066e80 2024-03-24T00:31:31.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe50e000640 1 -- 172.21.15.40:0/1818807559 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5080a5cd0 con 0x7fe510066e80 2024-03-24T00:31:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe516800640 1 -- 172.21.15.40:0/1818807559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe510066e80 msgr2=0x7fe510067310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe516800640 1 --2- 172.21.15.40:0/1818807559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe510066e80 0x7fe510067310 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe510071700 tx=0x7fe50801c9f0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe516800640 1 -- 172.21.15.40:0/1818807559 shutdown_connections 2024-03-24T00:31:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe516800640 1 --2- 172.21.15.40:0/1818807559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe510066e80 0x7fe510067310 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe516800640 1 --2- 172.21.15.40:0/1818807559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5100cd320 0x7fe5100cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe516800640 1 --2- 172.21.15.40:0/1818807559 >> v2:172.21.15.53:3300/0 conn(0x7fe510145700 0x7fe5100725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe516800640 1 -- 172.21.15.40:0/1818807559 >> 172.21.15.40:0/1818807559 conn(0x7fe5100b27c0 msgr2=0x7fe5100b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe516800640 1 -- 172.21.15.40:0/1818807559 shutdown_connections 2024-03-24T00:31:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.422+0000 7fe516800640 1 -- 172.21.15.40:0/1818807559 wait complete. 2024-03-24T00:31:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe516800640 1 Processor -- start 2024-03-24T00:31:31.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe516800640 1 -- start start 2024-03-24T00:31:31.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe516800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe510066e80 0x7fe51015c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe516800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5100cd320 0x7fe510137c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe516800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe510145700 0x7fe510136200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe516800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5100d02b0 con 0x7fe5100cd320 2024-03-24T00:31:31.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe516800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51006e400 con 0x7fe510145700 2024-03-24T00:31:31.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe516800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5101404d0 con 0x7fe510066e80 2024-03-24T00:31:31.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe50ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5100cd320 0x7fe510137c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe50ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5100cd320 0x7fe510137c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45706/0 (socket says 172.21.15.40:45706) 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe50ea00640 1 -- 172.21.15.40:0/252858177 learned_addr learned my addr 172.21.15.40:0/252858177 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe50f400640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe510066e80 0x7fe51015c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe50fe00640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe510145700 0x7fe510136200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe50ea00640 1 -- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe510066e80 msgr2=0x7fe51015c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe50ea00640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe510066e80 0x7fe51015c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe50ea00640 1 -- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe510145700 msgr2=0x7fe510136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe50ea00640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe510145700 0x7fe510136200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe50ea00640 1 -- 172.21.15.40:0/252858177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5101407b0 con 0x7fe5100cd320 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe50ea00640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5100cd320 0x7fe510137c50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00d830 tx=0x7fe4fc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe507400640 1 -- 172.21.15.40:0/252858177 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc018030 con 0x7fe5100cd320 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe507400640 1 -- 172.21.15.40:0/252858177 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4fc00be10 con 0x7fe5100cd320 2024-03-24T00:31:31.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe507400640 1 -- 172.21.15.40:0/252858177 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc0098b0 con 0x7fe5100cd320 2024-03-24T00:31:31.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe516800640 1 -- 172.21.15.40:0/252858177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe510140a90 con 0x7fe5100cd320 2024-03-24T00:31:31.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe516800640 1 -- 172.21.15.40:0/252858177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe510138350 con 0x7fe5100cd320 2024-03-24T00:31:31.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.426+0000 7fe516800640 1 -- 172.21.15.40:0/252858177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe510068170 con 0x7fe5100cd320 2024-03-24T00:31:31.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.430+0000 7fe507400640 1 -- 172.21.15.40:0/252858177 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe4fc01b070 con 0x7fe5100cd320 2024-03-24T00:31:31.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.430+0000 7fe507400640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe4f0067660 0x7fe4f0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.430+0000 7fe507400640 1 -- 172.21.15.40:0/252858177 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4fc0859e0 con 0x7fe5100cd320 2024-03-24T00:31:31.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.430+0000 7fe50f400640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe4f0067660 0x7fe4f0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.430+0000 7fe50f400640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe4f0067660 0x7fe4f0069b20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe510134db0 tx=0x7fe500008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:31.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.434+0000 7fe507400640 1 -- 172.21.15.40:0/252858177 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe4fc055f70 con 0x7fe5100cd320 2024-03-24T00:31:31.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.594+0000 7fe516800640 1 -- 172.21.15.40:0/252858177 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe510072de0 con 0x7fe5100cd320 2024-03-24T00:31:31.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe507400640 1 -- 172.21.15.40:0/252858177 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe4fc05a010 con 0x7fe5100cd320 2024-03-24T00:31:31.600 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:31.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe516800640 1 -- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe4f0067660 msgr2=0x7fe4f0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe516800640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe4f0067660 0x7fe4f0069b20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe510134db0 tx=0x7fe500008040 comp rx=0 tx=0).stop 2024-03-24T00:31:31.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe516800640 1 -- 172.21.15.40:0/252858177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5100cd320 msgr2=0x7fe510137c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe516800640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5100cd320 0x7fe510137c50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00d830 tx=0x7fe4fc00dcf0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe516800640 1 -- 172.21.15.40:0/252858177 shutdown_connections 2024-03-24T00:31:31.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe516800640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe4f0067660 0x7fe4f0069b20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe516800640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe510145700 0x7fe510136200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe516800640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe5100cd320 0x7fe510137c50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe516800640 1 --2- 172.21.15.40:0/252858177 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe510066e80 0x7fe51015c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe516800640 1 -- 172.21.15.40:0/252858177 >> 172.21.15.40:0/252858177 conn(0x7fe5100b27c0 msgr2=0x7fe510143260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:31.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe516800640 1 -- 172.21.15.40:0/252858177 shutdown_connections 2024-03-24T00:31:31.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.598+0000 7fe516800640 1 -- 172.21.15.40:0/252858177 wait complete. 2024-03-24T00:31:31.622 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:31.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.758+0000 7f2e10a00640 1 Processor -- start 2024-03-24T00:31:31.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.758+0000 7f2e10a00640 1 -- start start 2024-03-24T00:31:31.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.758+0000 7f2e10a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e0c0b2910 0x7f2e0c0d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.758+0000 7f2e10a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2e0c0af170 0x7f2e0c165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e10a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0c0b1e50 con 0x7f2e0c15f600 2024-03-24T00:31:31.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e10a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e0c0b3630 con 0x7f2e0c0af170 2024-03-24T00:31:31.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e10a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c0b2d80 con 0x7f2e0c0b2910 2024-03-24T00:31:31.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e09e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2e0c15f600 0x7f2e0c0aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41192/0 (socket says 172.21.15.40:41192) 2024-03-24T00:31:31.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e09e00640 1 -- 172.21.15.40:0/1675095410 learned_addr learned my addr 172.21.15.40:0/1675095410 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:31.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e0a800640 1 --2- 172.21.15.40:0/1675095410 >> v2:172.21.15.40:3300/0 conn(0x7f2e0c0af170 0x7f2e0c165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e03e00640 1 --2- 172.21.15.40:0/1675095410 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e0c0b2910 0x7f2e0c0d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e09400640 1 -- 172.21.15.40:0/1675095410 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096177500 0 0) 0x7f2e0c0b1e50 con 0x7f2e0c15f600 2024-03-24T00:31:31.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e09400640 1 -- 172.21.15.40:0/1675095410 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df0003530 con 0x7f2e0c15f600 2024-03-24T00:31:31.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e0a800640 1 -- 172.21.15.40:0/1675095410 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e0c0b2910 msgr2=0x7f2e0c0d2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:31.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e0a800640 1 --2- 172.21.15.40:0/1675095410 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e0c0b2910 0x7f2e0c0d2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e0a800640 1 -- 172.21.15.40:0/1675095410 >> v1:172.21.15.40:6789/0 conn(0x7f2e0c15f600 legacy=0x7f2e0c0aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e0a800640 1 -- 172.21.15.40:0/1675095410 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c0b20f0 con 0x7f2e0c0af170 2024-03-24T00:31:31.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e03e00640 1 --2- 172.21.15.40:0/1675095410 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e0c0b2910 0x7f2e0c0d2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:31.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e0a800640 1 --2- 172.21.15.40:0/1675095410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c0af170 0x7f2e0c165e40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2e04009d90 tx=0x7f2e0401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab5e2edabb68192 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:31.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e09400640 1 -- 172.21.15.40:0/1675095410 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e040b4070 con 0x7f2e0c0af170 2024-03-24T00:31:31.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e09400640 1 -- 172.21.15.40:0/1675095410 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e040af600 con 0x7f2e0c0af170 2024-03-24T00:31:31.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.762+0000 7f2e09400640 1 -- 172.21.15.40:0/1675095410 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e040af9d0 con 0x7f2e0c0af170 2024-03-24T00:31:31.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 -- 172.21.15.40:0/1675095410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c0af170 msgr2=0x7f2e0c165e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 --2- 172.21.15.40:0/1675095410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c0af170 0x7f2e0c165e40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2e04009d90 tx=0x7f2e0401f150 comp rx=0 tx=0).stop 2024-03-24T00:31:31.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 -- 172.21.15.40:0/1675095410 shutdown_connections 2024-03-24T00:31:31.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 --2- 172.21.15.40:0/1675095410 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c0af170 0x7f2e0c165e40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 --2- 172.21.15.40:0/1675095410 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e0c0b2910 0x7f2e0c0d2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 -- 172.21.15.40:0/1675095410 >> 172.21.15.40:0/1675095410 conn(0x7f2e0c0d1d10 msgr2=0x7f2e0c0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:31.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 -- 172.21.15.40:0/1675095410 shutdown_connections 2024-03-24T00:31:31.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 -- 172.21.15.40:0/1675095410 wait complete. 2024-03-24T00:31:31.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 Processor -- start 2024-03-24T00:31:31.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 -- start start 2024-03-24T00:31:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e0c0af170 0x7f2e0c151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c0b2910 0x7f2e0c0b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e0c15f600 0x7f2e0c0b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c0b2fa0 con 0x7f2e0c0b2910 2024-03-24T00:31:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c0b2740 con 0x7f2e0c15f600 2024-03-24T00:31:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e10a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c15f9e0 con 0x7f2e0c0af170 2024-03-24T00:31:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e03e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c0b2910 0x7f2e0c0b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e03e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c0b2910 0x7f2e0c0b37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45716/0 (socket says 172.21.15.40:45716) 2024-03-24T00:31:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e03e00640 1 -- 172.21.15.40:0/3155213203 learned_addr learned my addr 172.21.15.40:0/3155213203 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e09e00640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e0c0af170 0x7f2e0c151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.766+0000 7f2e0a800640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e0c15f600 0x7f2e0c0b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e03e00640 1 -- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e0c0af170 msgr2=0x7f2e0c151120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e03e00640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e0c0af170 0x7f2e0c151120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e03e00640 1 -- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e0c15f600 msgr2=0x7f2e0c0b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e03e00640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e0c15f600 0x7f2e0c0b3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e03e00640 1 -- 172.21.15.40:0/3155213203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c0b3460 con 0x7f2e0c0b2910 2024-03-24T00:31:31.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e03e00640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c0b2910 0x7f2e0c0b37b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2df400e810 tx=0x7f2df400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:31.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e02000640 1 -- 172.21.15.40:0/3155213203 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df4017070 con 0x7f2e0c0b2910 2024-03-24T00:31:31.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e02000640 1 -- 172.21.15.40:0/3155213203 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df400c8d0 con 0x7f2e0c0b2910 2024-03-24T00:31:31.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e02000640 1 -- 172.21.15.40:0/3155213203 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df400f040 con 0x7f2e0c0b2910 2024-03-24T00:31:31.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e10a00640 1 -- 172.21.15.40:0/3155213203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0c15c4b0 con 0x7f2e0c0b2910 2024-03-24T00:31:31.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e10a00640 1 -- 172.21.15.40:0/3155213203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0c0b4320 con 0x7f2e0c0b2910 2024-03-24T00:31:31.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e10a00640 1 -- 172.21.15.40:0/3155213203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0c068170 con 0x7f2e0c0b2910 2024-03-24T00:31:31.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.770+0000 7f2e02000640 1 -- 172.21.15.40:0/3155213203 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2df401a070 con 0x7f2e0c0b2910 2024-03-24T00:31:31.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.774+0000 7f2e02000640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2de8067690 0x7f2de8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:31.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.774+0000 7f2e02000640 1 -- 172.21.15.40:0/3155213203 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2df4085440 con 0x7f2e0c0b2910 2024-03-24T00:31:31.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.774+0000 7f2e09e00640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2de8067690 0x7f2de8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:31.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.774+0000 7f2e09e00640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2de8067690 0x7f2de8069b50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc007fd0 tx=0x7f2dfc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:31.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.774+0000 7f2e02000640 1 -- 172.21.15.40:0/3155213203 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2df40559d0 con 0x7f2e0c0b2910 2024-03-24T00:31:31.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.938+0000 7f2e10a00640 1 -- 172.21.15.40:0/3155213203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e0c0d1960 con 0x7f2e0c0b2910 2024-03-24T00:31:31.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e02000640 1 -- 172.21.15.40:0/3155213203 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7f2df4059a70 con 0x7f2e0c0b2910 2024-03-24T00:31:31.944 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:31.944 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e10a00640 1 -- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2de8067690 msgr2=0x7f2de8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e10a00640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2de8067690 0x7f2de8069b50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc007fd0 tx=0x7f2dfc009040 comp rx=0 tx=0).stop 2024-03-24T00:31:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e10a00640 1 -- 172.21.15.40:0/3155213203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c0b2910 msgr2=0x7f2e0c0b37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e10a00640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c0b2910 0x7f2e0c0b37b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2df400e810 tx=0x7f2df400ecd0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e10a00640 1 -- 172.21.15.40:0/3155213203 shutdown_connections 2024-03-24T00:31:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e10a00640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2de8067690 0x7f2de8069b50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e10a00640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e0c15f600 0x7f2e0c0b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e10a00640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e0c0b2910 0x7f2e0c0b37b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e10a00640 1 --2- 172.21.15.40:0/3155213203 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e0c0af170 0x7f2e0c151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e10a00640 1 -- 172.21.15.40:0/3155213203 >> 172.21.15.40:0/3155213203 conn(0x7f2e0c0d1d10 msgr2=0x7f2e0c0ce6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:31.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e10a00640 1 -- 172.21.15.40:0/3155213203 shutdown_connections 2024-03-24T00:31:31.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:31.942+0000 7f2e10a00640 1 -- 172.21.15.40:0/3155213203 wait complete. 2024-03-24T00:31:31.965 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:31.965 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:32.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.110+0000 7fc1ad600640 1 Processor -- start 2024-03-24T00:31:32.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.110+0000 7fc1ad600640 1 -- start start 2024-03-24T00:31:32.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.110+0000 7fc1ad600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1a8145700 0x7fc1a80725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.110+0000 7fc1ad600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc1a80cd320 0x7fc1a80cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.110+0000 7fc1ad600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1a80ceff0 con 0x7fc1a8066e80 2024-03-24T00:31:32.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.110+0000 7fc1ad600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1a80cf8e0 con 0x7fc1a8145700 2024-03-24T00:31:32.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.110+0000 7fc1ad600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1a80cdb70 con 0x7fc1a80cd320 2024-03-24T00:31:32.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.110+0000 7fc1a6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1a8145700 0x7fc1a80725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.110+0000 7fc1a6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1a8145700 0x7fc1a80725c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45732/0 (socket says 172.21.15.40:45732) 2024-03-24T00:31:32.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.110+0000 7fc1a6a00640 1 -- 172.21.15.40:0/1108942119 learned_addr learned my addr 172.21.15.40:0/1108942119 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:32.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.110+0000 7fc19e000640 1 --2- 172.21.15.40:0/1108942119 >> v2:172.21.15.53:3300/0 conn(0x7fc1a80cd320 0x7fc1a80cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1a6a00640 1 -- 172.21.15.40:0/1108942119 >> v2:172.21.15.53:3300/0 conn(0x7fc1a80cd320 msgr2=0x7fc1a80cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:32.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1a6a00640 1 --2- 172.21.15.40:0/1108942119 >> v2:172.21.15.53:3300/0 conn(0x7fc1a80cd320 0x7fc1a80cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1a6a00640 1 -- 172.21.15.40:0/1108942119 >> v1:172.21.15.53:6789/0 conn(0x7fc1a8066e80 legacy=0x7fc1a806dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1a6a00640 1 -- 172.21.15.40:0/1108942119 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a80cd890 con 0x7fc1a8145700 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1a6a00640 1 --2- 172.21.15.40:0/1108942119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a8145700 0x7fc1a80725c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc1a80b3880 tx=0x7fc19801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b5b21e60ab1e2ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1a6000640 1 -- 172.21.15.40:0/1108942119 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1980ab600 con 0x7fc1a8145700 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1a6000640 1 -- 172.21.15.40:0/1108942119 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1980b0070 con 0x7fc1a8145700 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1a6000640 1 -- 172.21.15.40:0/1108942119 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1980ab9c0 con 0x7fc1a8145700 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1ad600640 1 -- 172.21.15.40:0/1108942119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a8145700 msgr2=0x7fc1a80725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1ad600640 1 --2- 172.21.15.40:0/1108942119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a8145700 0x7fc1a80725c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc1a80b3880 tx=0x7fc19801c4f0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1ad600640 1 -- 172.21.15.40:0/1108942119 shutdown_connections 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1ad600640 1 --2- 172.21.15.40:0/1108942119 >> v2:172.21.15.53:3300/0 conn(0x7fc1a80cd320 0x7fc1a80cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1ad600640 1 --2- 172.21.15.40:0/1108942119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a8145700 0x7fc1a80725c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.114+0000 7fc1ad600640 1 -- 172.21.15.40:0/1108942119 >> 172.21.15.40:0/1108942119 conn(0x7fc1a80b27c0 msgr2=0x7fc1a80b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1ad600640 1 -- 172.21.15.40:0/1108942119 shutdown_connections 2024-03-24T00:31:32.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1ad600640 1 -- 172.21.15.40:0/1108942119 wait complete. 2024-03-24T00:31:32.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1ad600640 1 Processor -- start 2024-03-24T00:31:32.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1ad600640 1 -- start start 2024-03-24T00:31:32.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1ad600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1a8066e80 0x7fc1a8159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1ad600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc1a80cd320 0x7fc1a815a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1ad600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a8145700 0x7fc1a813d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1ad600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a80d02b0 con 0x7fc1a8145700 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1ad600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a80676a0 con 0x7fc1a8066e80 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1ad600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1a8141140 con 0x7fc1a80cd320 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1a6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1a8066e80 0x7fc1a8159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1a7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a8145700 0x7fc1a813d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1a6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1a8066e80 0x7fc1a8159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:38472/0 (socket says 172.21.15.40:38472) 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1a6a00640 1 -- 172.21.15.40:0/1837863567 learned_addr learned my addr 172.21.15.40:0/1837863567 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc19e000640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc1a80cd320 0x7fc1a815a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1a7400640 1 -- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc1a80cd320 msgr2=0x7fc1a815a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1a7400640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc1a80cd320 0x7fc1a815a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1a7400640 1 -- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1a8066e80 msgr2=0x7fc1a8159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1a7400640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1a8066e80 0x7fc1a8159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1a7400640 1 -- 172.21.15.40:0/1837863567 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a815aa20 con 0x7fc1a8145700 2024-03-24T00:31:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1a6a00640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1a8066e80 0x7fc1a8159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:32.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.118+0000 7fc1a7400640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a8145700 0x7fc1a813d260 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000c700 tx=0x7fc1a000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:32.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.122+0000 7fc19f400640 1 -- 172.21.15.40:0/1837863567 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0003b80 con 0x7fc1a8145700 2024-03-24T00:31:32.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.122+0000 7fc19f400640 1 -- 172.21.15.40:0/1837863567 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a0003d20 con 0x7fc1a8145700 2024-03-24T00:31:32.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.122+0000 7fc1ad600640 1 -- 172.21.15.40:0/1837863567 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1a815ad00 con 0x7fc1a8145700 2024-03-24T00:31:32.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.122+0000 7fc19f400640 1 -- 172.21.15.40:0/1837863567 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a0005310 con 0x7fc1a8145700 2024-03-24T00:31:32.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.122+0000 7fc1ad600640 1 -- 172.21.15.40:0/1837863567 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1a815b2c0 con 0x7fc1a8145700 2024-03-24T00:31:32.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.122+0000 7fc1ad600640 1 -- 172.21.15.40:0/1837863567 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1a8068170 con 0x7fc1a8145700 2024-03-24T00:31:32.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.122+0000 7fc19f400640 1 -- 172.21.15.40:0/1837863567 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc1a00054b0 con 0x7fc1a8145700 2024-03-24T00:31:32.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.122+0000 7fc19f400640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc184067660 0x7fc184069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.122+0000 7fc19f400640 1 -- 172.21.15.40:0/1837863567 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1a00855d0 con 0x7fc1a8145700 2024-03-24T00:31:32.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.126+0000 7fc1a6a00640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc184067660 0x7fc184069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.126+0000 7fc1a6a00640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc184067660 0x7fc184069b20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc19801c530 tx=0x7fc1980a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.126+0000 7fc19f400640 1 -- 172.21.15.40:0/1837863567 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc1a0055b60 con 0x7fc1a8145700 2024-03-24T00:31:32.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.290+0000 7fc1ad600640 1 -- 172.21.15.40:0/1837863567 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1a8072de0 con 0x7fc1a8145700 2024-03-24T00:31:32.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.290+0000 7fc19f400640 1 -- 172.21.15.40:0/1837863567 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1a0059c00 con 0x7fc1a8145700 2024-03-24T00:31:32.294 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:32.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.294+0000 7fc1ad600640 1 -- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc184067660 msgr2=0x7fc184069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.294+0000 7fc1ad600640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc184067660 0x7fc184069b20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc19801c530 tx=0x7fc1980a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:32.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.294+0000 7fc1ad600640 1 -- 172.21.15.40:0/1837863567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a8145700 msgr2=0x7fc1a813d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.294+0000 7fc1ad600640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a8145700 0x7fc1a813d260 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc1a000c700 tx=0x7fc1a000cbc0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.294+0000 7fc1ad600640 1 -- 172.21.15.40:0/1837863567 shutdown_connections 2024-03-24T00:31:32.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.294+0000 7fc1ad600640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc184067660 0x7fc184069b20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.294+0000 7fc1ad600640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1a8145700 0x7fc1a813d260 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.294+0000 7fc1ad600640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc1a80cd320 0x7fc1a815a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.294+0000 7fc1ad600640 1 --2- 172.21.15.40:0/1837863567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1a8066e80 0x7fc1a8159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.294+0000 7fc1ad600640 1 -- 172.21.15.40:0/1837863567 >> 172.21.15.40:0/1837863567 conn(0x7fc1a80b27c0 msgr2=0x7fc1a813e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:32.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.294+0000 7fc1ad600640 1 -- 172.21.15.40:0/1837863567 shutdown_connections 2024-03-24T00:31:32.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.294+0000 7fc1ad600640 1 -- 172.21.15.40:0/1837863567 wait complete. 2024-03-24T00:31:32.317 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:32.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f6807c00640 1 Processor -- start 2024-03-24T00:31:32.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f6807c00640 1 -- start start 2024-03-24T00:31:32.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f6807c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f680015df80 0x7f6800070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f6807c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f680006c950 0x7f68000d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f6807c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f680006bd20 0x7f68000d20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f6807c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f680006b890 con 0x7f680006c950 2024-03-24T00:31:32.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f6807c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f680006d0f0 con 0x7f680015df80 2024-03-24T00:31:32.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f6807c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680006b280 con 0x7f680006bd20 2024-03-24T00:31:32.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f67f7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f680006c950 0x7f68000d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f67f7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f680006c950 0x7f68000d1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45754/0 (socket says 172.21.15.40:45754) 2024-03-24T00:31:32.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f67f7e00640 1 -- 172.21.15.40:0/347880422 learned_addr learned my addr 172.21.15.40:0/347880422 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:32.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f6805200640 1 --2- 172.21.15.40:0/347880422 >> v2:172.21.15.53:3300/0 conn(0x7f680015df80 0x7f6800070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f6805c00640 1 --2- 172.21.15.40:0/347880422 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f680006bd20 0x7f68000d20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f67f7e00640 1 -- 172.21.15.40:0/347880422 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f680006bd20 msgr2=0x7f68000d20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:32.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f67f7e00640 1 --2- 172.21.15.40:0/347880422 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f680006bd20 0x7f68000d20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f67f7e00640 1 -- 172.21.15.40:0/347880422 >> v2:172.21.15.53:3300/0 conn(0x7f680015df80 msgr2=0x7f6800070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:32.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f67f7e00640 1 --2- 172.21.15.40:0/347880422 >> v2:172.21.15.53:3300/0 conn(0x7f680015df80 0x7f6800070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.458+0000 7f67f7e00640 1 -- 172.21.15.40:0/347880422 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680006ce10 con 0x7f680006c950 2024-03-24T00:31:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f67f7e00640 1 --2- 172.21.15.40:0/347880422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680006c950 0x7f68000d1b60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f68000d2c90 tx=0x7f67e801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d520f1437709ba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f67ffe00640 1 -- 172.21.15.40:0/347880422 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e80a7070 con 0x7f680006c950 2024-03-24T00:31:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f67ffe00640 1 -- 172.21.15.40:0/347880422 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e80b0070 con 0x7f680006c950 2024-03-24T00:31:32.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f67ffe00640 1 -- 172.21.15.40:0/347880422 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e80ab890 con 0x7f680006c950 2024-03-24T00:31:32.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f6807c00640 1 -- 172.21.15.40:0/347880422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680006c950 msgr2=0x7f68000d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f6807c00640 1 --2- 172.21.15.40:0/347880422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680006c950 0x7f68000d1b60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f68000d2c90 tx=0x7f67e801c4f0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f6807c00640 1 -- 172.21.15.40:0/347880422 shutdown_connections 2024-03-24T00:31:32.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f6807c00640 1 --2- 172.21.15.40:0/347880422 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f680006bd20 0x7f68000d20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f6807c00640 1 --2- 172.21.15.40:0/347880422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680006c950 0x7f68000d1b60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f6807c00640 1 --2- 172.21.15.40:0/347880422 >> v2:172.21.15.53:3300/0 conn(0x7f680015df80 0x7f6800070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f6807c00640 1 -- 172.21.15.40:0/347880422 >> 172.21.15.40:0/347880422 conn(0x7f680006ecf0 msgr2=0x7f680006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:32.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f6807c00640 1 -- 172.21.15.40:0/347880422 shutdown_connections 2024-03-24T00:31:32.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.462+0000 7f6807c00640 1 -- 172.21.15.40:0/347880422 wait complete. 2024-03-24T00:31:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.466+0000 7f6807c00640 1 Processor -- start 2024-03-24T00:31:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.466+0000 7f6807c00640 1 -- start start 2024-03-24T00:31:32.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.466+0000 7f6807c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680006bd20 0x7f680014cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.466+0000 7f6807c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f680006c950 0x7f680015d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.466+0000 7f6807c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f680015df80 0x7f680014edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.466+0000 7f6807c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6800162460 con 0x7f680006bd20 2024-03-24T00:31:32.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.466+0000 7f6807c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68001622e0 con 0x7f680006c950 2024-03-24T00:31:32.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.466+0000 7f6807c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68001625e0 con 0x7f680015df80 2024-03-24T00:31:32.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.466+0000 7f6805200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680006bd20 0x7f680014cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.466+0000 7f6805200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680006bd20 0x7f680014cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45758/0 (socket says 172.21.15.40:45758) 2024-03-24T00:31:32.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.466+0000 7f6805200640 1 -- 172.21.15.40:0/1133409526 learned_addr learned my addr 172.21.15.40:0/1133409526 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:32.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f67f7e00640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f680006c950 0x7f680015d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f6805c00640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f680015df80 0x7f680014edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f6805200640 1 -- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f680015df80 msgr2=0x7f680014edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f6805200640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f680015df80 0x7f680014edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f6805200640 1 -- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f680006c950 msgr2=0x7f680015d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f6805200640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f680006c950 0x7f680015d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f6805200640 1 -- 172.21.15.40:0/1133409526 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680014dd20 con 0x7f680006bd20 2024-03-24T00:31:32.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f6805c00640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f680015df80 0x7f680014edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:32.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f67f7e00640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f680006c950 0x7f680015d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:32.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f6805200640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680006bd20 0x7f680014cdf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f67f000d5f0 tx=0x7f67f000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:32.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f67fd600640 1 -- 172.21.15.40:0/1133409526 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f00125f0 con 0x7f680006bd20 2024-03-24T00:31:32.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f67fd600640 1 -- 172.21.15.40:0/1133409526 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f0019070 con 0x7f680006bd20 2024-03-24T00:31:32.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f67fd600640 1 -- 172.21.15.40:0/1133409526 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f00129b0 con 0x7f680006bd20 2024-03-24T00:31:32.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f6807c00640 1 -- 172.21.15.40:0/1133409526 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680014e000 con 0x7f680006bd20 2024-03-24T00:31:32.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f6807c00640 1 -- 172.21.15.40:0/1133409526 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680014e5c0 con 0x7f680006bd20 2024-03-24T00:31:32.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f6807c00640 1 -- 172.21.15.40:0/1133409526 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6800068170 con 0x7f680006bd20 2024-03-24T00:31:32.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f67fd600640 1 -- 172.21.15.40:0/1133409526 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f67f0012b50 con 0x7f680006bd20 2024-03-24T00:31:32.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f67fd600640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f67dc067660 0x7f67dc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.470+0000 7f67fd600640 1 -- 172.21.15.40:0/1133409526 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67f0085c40 con 0x7f680006bd20 2024-03-24T00:31:32.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.474+0000 7f67f7e00640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f67dc067660 0x7f67dc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.474+0000 7f67f7e00640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f67dc067660 0x7f67dc069b20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f67e80af000 tx=0x7f67e80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:32.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.478+0000 7f67fd600640 1 -- 172.21.15.40:0/1133409526 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f67f00568b0 con 0x7f680006bd20 2024-03-24T00:31:32.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.642+0000 7f6807c00640 1 -- 172.21.15.40:0/1133409526 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f68000cf9d0 con 0x7f680006bd20 2024-03-24T00:31:32.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f67fd600640 1 -- 172.21.15.40:0/1133409526 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f67f00561b0 con 0x7f680006bd20 2024-03-24T00:31:32.648 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:32.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f6807c00640 1 -- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f67dc067660 msgr2=0x7f67dc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f6807c00640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f67dc067660 0x7f67dc069b20 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f67e80af000 tx=0x7f67e80a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:32.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f6807c00640 1 -- 172.21.15.40:0/1133409526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680006bd20 msgr2=0x7f680014cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f6807c00640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680006bd20 0x7f680014cdf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f67f000d5f0 tx=0x7f67f000dab0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f6807c00640 1 -- 172.21.15.40:0/1133409526 shutdown_connections 2024-03-24T00:31:32.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f6807c00640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f67dc067660 0x7f67dc069b20 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f6807c00640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f680015df80 0x7f680014edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f6807c00640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f680006c950 0x7f680015d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f6807c00640 1 --2- 172.21.15.40:0/1133409526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f680006bd20 0x7f680014cdf0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f6807c00640 1 -- 172.21.15.40:0/1133409526 >> 172.21.15.40:0/1133409526 conn(0x7f680006ecf0 msgr2=0x7f680015a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:32.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f6807c00640 1 -- 172.21.15.40:0/1133409526 shutdown_connections 2024-03-24T00:31:32.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.646+0000 7f6807c00640 1 -- 172.21.15.40:0/1133409526 wait complete. 2024-03-24T00:31:32.679 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:32.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.818+0000 7f84cc400640 1 Processor -- start 2024-03-24T00:31:32.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.818+0000 7f84cc400640 1 -- start start 2024-03-24T00:31:32.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84cc400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c40cd320 0x7f84c40cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84cc400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f84c4066e80 0x7f84c4067310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84cc400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c40ceff0 con 0x7f84c4145700 2024-03-24T00:31:32.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84cc400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c40cf8e0 con 0x7f84c4066e80 2024-03-24T00:31:32.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84cc400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40cdb70 con 0x7f84c40cd320 2024-03-24T00:31:32.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84c9000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c40cd320 0x7f84c40cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84ca400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f84c4066e80 0x7f84c4067310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84c9000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c40cd320 0x7f84c40cfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:59180/0 (socket says 172.21.15.40:59180) 2024-03-24T00:31:32.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84c9000640 1 -- 172.21.15.40:0/3130898345 learned_addr learned my addr 172.21.15.40:0/3130898345 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:32.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84c9a00640 1 --1- 172.21.15.40:0/3130898345 >> v1:172.21.15.53:6789/0 conn(0x7f84c4145700 0x7f84c40725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:37020/0 (socket says 172.21.15.40:37020) 2024-03-24T00:31:32.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84ca400640 1 -- 172.21.15.40:0/3130898345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c40cd320 msgr2=0x7f84c40cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:32.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84ca400640 1 --2- 172.21.15.40:0/3130898345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c40cd320 0x7f84c40cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84ca400640 1 -- 172.21.15.40:0/3130898345 >> v1:172.21.15.53:6789/0 conn(0x7f84c4145700 legacy=0x7f84c40725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84ca400640 1 -- 172.21.15.40:0/3130898345 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c40cd890 con 0x7f84c4066e80 2024-03-24T00:31:32.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84c3e00640 1 -- 172.21.15.40:0/3130898345 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783691541 0 0) 0x7f84c40ceff0 con 0x7f84c4145700 2024-03-24T00:31:32.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.822+0000 7f84c9000640 1 --2- 172.21.15.40:0/3130898345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c40cd320 0x7f84c40cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:32.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.826+0000 7f84ca400640 1 --2- 172.21.15.40:0/3130898345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c4066e80 0x7f84c4067310 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009d90 tx=0x7f84bc01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d934f3f66229180b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:32.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.826+0000 7f84c3e00640 1 -- 172.21.15.40:0/3130898345 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc0b2070 con 0x7f84c4066e80 2024-03-24T00:31:32.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84c3e00640 1 -- 172.21.15.40:0/3130898345 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc0a5890 con 0x7f84c4066e80 2024-03-24T00:31:32.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84c3e00640 1 -- 172.21.15.40:0/3130898345 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc0a5c60 con 0x7f84c4066e80 2024-03-24T00:31:32.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 -- 172.21.15.40:0/3130898345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c4066e80 msgr2=0x7f84c4067310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 --2- 172.21.15.40:0/3130898345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c4066e80 0x7f84c4067310 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009d90 tx=0x7f84bc01c8e0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 -- 172.21.15.40:0/3130898345 shutdown_connections 2024-03-24T00:31:32.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 --2- 172.21.15.40:0/3130898345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c4066e80 0x7f84c4067310 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 --2- 172.21.15.40:0/3130898345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c40cd320 0x7f84c40cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 -- 172.21.15.40:0/3130898345 >> 172.21.15.40:0/3130898345 conn(0x7f84c40b27c0 msgr2=0x7f84c40b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 -- 172.21.15.40:0/3130898345 shutdown_connections 2024-03-24T00:31:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 -- 172.21.15.40:0/3130898345 wait complete. 2024-03-24T00:31:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 Processor -- start 2024-03-24T00:31:32.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 -- start start 2024-03-24T00:31:32.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c4066e80 0x7f84c413c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c40cd320 0x7f84c413a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c4145700 0x7f84c413a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c406e580 con 0x7f84c4145700 2024-03-24T00:31:32.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c406e400 con 0x7f84c4066e80 2024-03-24T00:31:32.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84cc400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40b3bc0 con 0x7f84c40cd320 2024-03-24T00:31:32.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84ca400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c4145700 0x7f84c413a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84ca400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c4145700 0x7f84c413a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45762/0 (socket says 172.21.15.40:45762) 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84ca400640 1 -- 172.21.15.40:0/3101791127 learned_addr learned my addr 172.21.15.40:0/3101791127 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84c9a00640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c4066e80 0x7f84c413c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.830+0000 7f84c9000640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c40cd320 0x7f84c413a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84ca400640 1 -- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c40cd320 msgr2=0x7f84c413a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84ca400640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c40cd320 0x7f84c413a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84ca400640 1 -- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c4066e80 msgr2=0x7f84c413c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84ca400640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c4066e80 0x7f84c413c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84ca400640 1 -- 172.21.15.40:0/3101791127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c40d10a0 con 0x7f84c4145700 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84c9a00640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c4066e80 0x7f84c413c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84ca400640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c4145700 0x7f84c413a640 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0b0040 tx=0x7f84bc019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84c1600640 1 -- 172.21.15.40:0/3101791127 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc0bd020 con 0x7f84c4145700 2024-03-24T00:31:32.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84c1600640 1 -- 172.21.15.40:0/3101791127 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc0b8470 con 0x7f84c4145700 2024-03-24T00:31:32.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84c1600640 1 -- 172.21.15.40:0/3101791127 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc0b2040 con 0x7f84c4145700 2024-03-24T00:31:32.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84cc400640 1 -- 172.21.15.40:0/3101791127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c40d1380 con 0x7f84c4145700 2024-03-24T00:31:32.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84cc400640 1 -- 172.21.15.40:0/3101791127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c413af00 con 0x7f84c4145700 2024-03-24T00:31:32.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84cc400640 1 -- 172.21.15.40:0/3101791127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c4068170 con 0x7f84c4145700 2024-03-24T00:31:32.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84c1600640 1 -- 172.21.15.40:0/3101791127 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f84bc0c0020 con 0x7f84c4145700 2024-03-24T00:31:32.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.834+0000 7f84c1600640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f84a4067660 0x7f84a4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:32.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.838+0000 7f84c1600640 1 -- 172.21.15.40:0/3101791127 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84bc120920 con 0x7f84c4145700 2024-03-24T00:31:32.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.838+0000 7f84c9a00640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f84a4067660 0x7f84a4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:32.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.838+0000 7f84c9a00640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f84a4067660 0x7f84a4069b20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f84b8000f70 tx=0x7f84b8007cb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:32.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:32.842+0000 7f84c1600640 1 -- 172.21.15.40:0/3101791127 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f84bc0c7030 con 0x7f84c4145700 2024-03-24T00:31:33.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.002+0000 7f84cc400640 1 -- 172.21.15.40:0/3101791127 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f84c4072de0 con 0x7f84c4145700 2024-03-24T00:31:33.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.002+0000 7f84c1600640 1 -- 172.21.15.40:0/3101791127 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f84bc016560 con 0x7f84c4145700 2024-03-24T00:31:33.006 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:33.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.006+0000 7f84cc400640 1 -- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f84a4067660 msgr2=0x7f84a4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.006+0000 7f84cc400640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f84a4067660 0x7f84a4069b20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f84b8000f70 tx=0x7f84b8007cb0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.006+0000 7f84cc400640 1 -- 172.21.15.40:0/3101791127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c4145700 msgr2=0x7f84c413a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.006+0000 7f84cc400640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c4145700 0x7f84c413a640 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0b0040 tx=0x7f84bc019680 comp rx=0 tx=0).stop 2024-03-24T00:31:33.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.006+0000 7f84cc400640 1 -- 172.21.15.40:0/3101791127 shutdown_connections 2024-03-24T00:31:33.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.006+0000 7f84cc400640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f84a4067660 0x7f84a4069b20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.006+0000 7f84cc400640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c4145700 0x7f84c413a640 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.006+0000 7f84cc400640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c40cd320 0x7f84c413a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.006+0000 7f84cc400640 1 --2- 172.21.15.40:0/3101791127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c4066e80 0x7f84c413c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.006+0000 7f84cc400640 1 -- 172.21.15.40:0/3101791127 >> 172.21.15.40:0/3101791127 conn(0x7f84c40b27c0 msgr2=0x7f84c4160300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:33.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.006+0000 7f84cc400640 1 -- 172.21.15.40:0/3101791127 shutdown_connections 2024-03-24T00:31:33.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.006+0000 7f84cc400640 1 -- 172.21.15.40:0/3101791127 wait complete. 2024-03-24T00:31:33.064 INFO:tasks.cephfs_test_runner:Starting test: test_ephemeral_pin_dist_conf_off (tasks.cephfs.test_exports.TestEphemeralPins) That turning off ephemeral distributed pin config prevents distribution. 2024-03-24T00:31:33.064 INFO:tasks.cephfs_test_runner:test_ephemeral_pin_dist_conf_off (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:31:33.065 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.202+0000 7f8a53a00640 1 Processor -- start 2024-03-24T00:31:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.202+0000 7f8a53a00640 1 -- start start 2024-03-24T00:31:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.202+0000 7f8a53a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8a4c0b2aa0 0x7f8a4c0d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.202+0000 7f8a53a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4c0b2870 con 0x7f8a4c0d0a80 2024-03-24T00:31:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.202+0000 7f8a53a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4c0b3190 con 0x7f8a4c0b2490 2024-03-24T00:31:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.202+0000 7f8a53a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a4c06b6d0 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.202+0000 7f8a51a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8a4c0d0a80 0x7f8a4c0cc460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41204/0 (socket says 172.21.15.40:41204) 2024-03-24T00:31:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.202+0000 7f8a51a00640 1 -- 172.21.15.40:0/380315658 learned_addr learned my addr 172.21.15.40:0/380315658 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.202+0000 7f8a4be00640 1 --2- 172.21.15.40:0/380315658 >> v2:172.21.15.53:3300/0 conn(0x7f8a4c0b2aa0 0x7f8a4c0d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.206+0000 7f8a4b400640 1 -- 172.21.15.40:0/380315658 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059570762 0 0) 0x7f8a4c0b2870 con 0x7f8a4c0d0a80 2024-03-24T00:31:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.206+0000 7f8a4b400640 1 -- 172.21.15.40:0/380315658 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a34003530 con 0x7f8a4c0d0a80 2024-03-24T00:31:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.206+0000 7f8a4be00640 1 -- 172.21.15.40:0/380315658 >> v1:172.21.15.53:6789/0 conn(0x7f8a4c0b2490 legacy=0x7f8a4c0cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.206+0000 7f8a4be00640 1 -- 172.21.15.40:0/380315658 >> v1:172.21.15.40:6789/0 conn(0x7f8a4c0d0a80 legacy=0x7f8a4c0cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.206+0000 7f8a4be00640 1 -- 172.21.15.40:0/380315658 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c0b3470 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a4be00640 1 --2- 172.21.15.40:0/380315658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a4c0b2aa0 0x7f8a4c0d0540 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a380097c0 tx=0x7f8a3801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a5319f7c24fd6c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:33.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a4b400640 1 -- 172.21.15.40:0/380315658 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a380b1070 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a4b400640 1 -- 172.21.15.40:0/380315658 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a380ac410 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a4b400640 1 -- 172.21.15.40:0/380315658 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a380ac7e0 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a53a00640 1 -- 172.21.15.40:0/380315658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a4c0b2aa0 msgr2=0x7f8a4c0d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a53a00640 1 --2- 172.21.15.40:0/380315658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a4c0b2aa0 0x7f8a4c0d0540 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a380097c0 tx=0x7f8a3801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a53a00640 1 -- 172.21.15.40:0/380315658 shutdown_connections 2024-03-24T00:31:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a53a00640 1 --2- 172.21.15.40:0/380315658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a4c0b2aa0 0x7f8a4c0d0540 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a53a00640 1 -- 172.21.15.40:0/380315658 >> 172.21.15.40:0/380315658 conn(0x7f8a4c0d1d10 msgr2=0x7f8a4c0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a53a00640 1 -- 172.21.15.40:0/380315658 shutdown_connections 2024-03-24T00:31:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a53a00640 1 -- 172.21.15.40:0/380315658 wait complete. 2024-03-24T00:31:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a53a00640 1 Processor -- start 2024-03-24T00:31:33.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.210+0000 7f8a53a00640 1 -- start start 2024-03-24T00:31:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a53a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c0b2490 0x7f8a4c09f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a53a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a4c0b2aa0 0x7f8a4c160900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a53a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a4c0d0a80 0x7f8a4c1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a53a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c16bfd0 con 0x7f8a4c0b2490 2024-03-24T00:31:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a53a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0cd650 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a53a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c16c150 con 0x7f8a4c0d0a80 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a51000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c0b2490 0x7f8a4c09f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a4be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a4c0b2aa0 0x7f8a4c160900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a51a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a4c0d0a80 0x7f8a4c1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a4be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a4c0b2aa0 0x7f8a4c160900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:38530/0 (socket says 172.21.15.40:38530) 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a4be00640 1 -- 172.21.15.40:0/1201280345 learned_addr learned my addr 172.21.15.40:0/1201280345 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a51000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c0b2490 0x7f8a4c09f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45778/0 (socket says 172.21.15.40:45778) 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a4be00640 1 -- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a4c0d0a80 msgr2=0x7f8a4c1562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a4be00640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a4c0d0a80 0x7f8a4c1562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a4be00640 1 -- 172.21.15.40:0/1201280345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c0b2490 msgr2=0x7f8a4c09f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a4be00640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c0b2490 0x7f8a4c09f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a4be00640 1 -- 172.21.15.40:0/1201280345 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c15ab80 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a51000640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c0b2490 0x7f8a4c09f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:33.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a4be00640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a4c0b2aa0 0x7f8a4c160900 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8a380af040 tx=0x7f8a38019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a48c00640 1 -- 172.21.15.40:0/1201280345 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a380b9020 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a48c00640 1 -- 172.21.15.40:0/1201280345 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a380b5410 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a48c00640 1 -- 172.21.15.40:0/1201280345 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a380b1040 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a53a00640 1 -- 172.21.15.40:0/1201280345 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4c160e40 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a53a00640 1 -- 172.21.15.40:0/1201280345 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c15b100 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.214+0000 7f8a53a00640 1 -- 172.21.15.40:0/1201280345 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4c068170 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.218+0000 7f8a48c00640 1 -- 172.21.15.40:0/1201280345 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8a380bc020 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.218+0000 7f8a48c00640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a2c067690 0x7f8a2c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.218+0000 7f8a48c00640 1 -- 172.21.15.40:0/1201280345 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a3811f5e0 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.218+0000 7f8a51000640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a2c067690 0x7f8a2c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.218+0000 7f8a51000640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a2c067690 0x7f8a2c069b50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8a40001b60 tx=0x7f8a40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:33.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.222+0000 7f8a48c00640 1 -- 172.21.15.40:0/1201280345 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8a380c3030 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.386+0000 7f8a53a00640 1 -- 172.21.15.40:0/1201280345 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a4c159c50 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a48c00640 1 -- 172.21.15.40:0/1201280345 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7f8a38016090 con 0x7f8a4c0b2aa0 2024-03-24T00:31:33.392 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:33.393 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:33.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a53a00640 1 -- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a2c067690 msgr2=0x7f8a2c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a53a00640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a2c067690 0x7f8a2c069b50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8a40001b60 tx=0x7f8a40009040 comp rx=0 tx=0).stop 2024-03-24T00:31:33.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a53a00640 1 -- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a4c0b2aa0 msgr2=0x7f8a4c160900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a53a00640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a4c0b2aa0 0x7f8a4c160900 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8a380af040 tx=0x7f8a38019240 comp rx=0 tx=0).stop 2024-03-24T00:31:33.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a53a00640 1 -- 172.21.15.40:0/1201280345 shutdown_connections 2024-03-24T00:31:33.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a53a00640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a2c067690 0x7f8a2c069b50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a53a00640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a4c0d0a80 0x7f8a4c1562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a53a00640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a4c0b2aa0 0x7f8a4c160900 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a53a00640 1 --2- 172.21.15.40:0/1201280345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a4c0b2490 0x7f8a4c09f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a53a00640 1 -- 172.21.15.40:0/1201280345 >> 172.21.15.40:0/1201280345 conn(0x7f8a4c0d1d10 msgr2=0x7f8a4c06bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:33.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a53a00640 1 -- 172.21.15.40:0/1201280345 shutdown_connections 2024-03-24T00:31:33.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.390+0000 7f8a53a00640 1 -- 172.21.15.40:0/1201280345 wait complete. 2024-03-24T00:31:33.414 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:33.415 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:33.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.554+0000 7f7197200640 1 Processor -- start 2024-03-24T00:31:33.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.554+0000 7f7197200640 1 -- start start 2024-03-24T00:31:33.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7197200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f719806cd70 0x7f719806b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7197200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71980cf640 0x7f71980d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7197200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f71980ce360 0x7f71980ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7197200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f719806ed90 con 0x7f71980ce360 2024-03-24T00:31:33.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7197200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f719806f070 con 0x7f719806cd70 2024-03-24T00:31:33.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7197200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719806f350 con 0x7f71980cf640 2024-03-24T00:31:33.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7196800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f71980ce360 0x7f71980ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7196800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f71980ce360 0x7f71980ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45792/0 (socket says 172.21.15.40:45792) 2024-03-24T00:31:33.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7196800640 1 -- 172.21.15.40:0/401282059 learned_addr learned my addr 172.21.15.40:0/401282059 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:33.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7195e00640 1 --2- 172.21.15.40:0/401282059 >> v2:172.21.15.53:3300/0 conn(0x7f719806cd70 0x7f719806b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f718d400640 1 --2- 172.21.15.40:0/401282059 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71980cf640 0x7f71980d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7196800640 1 -- 172.21.15.40:0/401282059 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71980cf640 msgr2=0x7f71980d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:33.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7196800640 1 --2- 172.21.15.40:0/401282059 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71980cf640 0x7f71980d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7196800640 1 -- 172.21.15.40:0/401282059 >> v2:172.21.15.53:3300/0 conn(0x7f719806cd70 msgr2=0x7f719806b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:33.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7196800640 1 --2- 172.21.15.40:0/401282059 >> v2:172.21.15.53:3300/0 conn(0x7f719806cd70 0x7f719806b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.558+0000 7f7196800640 1 -- 172.21.15.40:0/401282059 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719806f630 con 0x7f71980ce360 2024-03-24T00:31:33.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7196800640 1 --2- 172.21.15.40:0/401282059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71980ce360 0x7f71980ce810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f71980b4550 tx=0x7f719001d450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5528546540d9b3ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:33.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7195400640 1 -- 172.21.15.40:0/401282059 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71900a5780 con 0x7f71980ce360 2024-03-24T00:31:33.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7195400640 1 -- 172.21.15.40:0/401282059 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71900a5920 con 0x7f71980ce360 2024-03-24T00:31:33.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7195400640 1 -- 172.21.15.40:0/401282059 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71900a5cf0 con 0x7f71980ce360 2024-03-24T00:31:33.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7197200640 1 -- 172.21.15.40:0/401282059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71980ce360 msgr2=0x7f71980ce810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7197200640 1 --2- 172.21.15.40:0/401282059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71980ce360 0x7f71980ce810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f71980b4550 tx=0x7f719001d450 comp rx=0 tx=0).stop 2024-03-24T00:31:33.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7197200640 1 -- 172.21.15.40:0/401282059 shutdown_connections 2024-03-24T00:31:33.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7197200640 1 --2- 172.21.15.40:0/401282059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71980ce360 0x7f71980ce810 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7197200640 1 --2- 172.21.15.40:0/401282059 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71980cf640 0x7f71980d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7197200640 1 --2- 172.21.15.40:0/401282059 >> v2:172.21.15.53:3300/0 conn(0x7f719806cd70 0x7f719806b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7197200640 1 -- 172.21.15.40:0/401282059 >> 172.21.15.40:0/401282059 conn(0x7f71980d19e0 msgr2=0x7f71980d1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:33.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7197200640 1 -- 172.21.15.40:0/401282059 shutdown_connections 2024-03-24T00:31:33.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.562+0000 7f7197200640 1 -- 172.21.15.40:0/401282059 wait complete. 2024-03-24T00:31:33.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7197200640 1 Processor -- start 2024-03-24T00:31:33.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7197200640 1 -- start start 2024-03-24T00:31:33.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7197200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f719806cd70 0x7f719814eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7197200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71980ce360 0x7f71980b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7197200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71980cf640 0x7f71980afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7197200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719815de60 con 0x7f71980cf640 2024-03-24T00:31:33.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7197200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719806d360 con 0x7f719806cd70 2024-03-24T00:31:33.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7197200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719815dfe0 con 0x7f71980ce360 2024-03-24T00:31:33.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7196800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71980cf640 0x7f71980afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7196800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71980cf640 0x7f71980afd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45794/0 (socket says 172.21.15.40:45794) 2024-03-24T00:31:33.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7196800640 1 -- 172.21.15.40:0/4135469635 learned_addr learned my addr 172.21.15.40:0/4135469635 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:33.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7195e00640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f719806cd70 0x7f719814eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f718d400640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71980ce360 0x7f71980b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7196800640 1 -- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71980ce360 msgr2=0x7f71980b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7196800640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71980ce360 0x7f71980b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7196800640 1 -- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f719806cd70 msgr2=0x7f719814eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7196800640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f719806cd70 0x7f719814eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7196800640 1 -- 172.21.15.40:0/4135469635 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71980c8e90 con 0x7f71980cf640 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7195e00640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f719806cd70 0x7f719814eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f718d400640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71980ce360 0x7f71980b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7196800640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71980cf640 0x7f71980afd40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7190009e20 tx=0x7f7190019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f718ea00640 1 -- 172.21.15.40:0/4135469635 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71900bc020 con 0x7f71980cf640 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f718ea00640 1 -- 172.21.15.40:0/4135469635 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71900b8670 con 0x7f71980cf640 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f718ea00640 1 -- 172.21.15.40:0/4135469635 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7190009510 con 0x7f71980cf640 2024-03-24T00:31:33.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7197200640 1 -- 172.21.15.40:0/4135469635 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71980c9170 con 0x7f71980cf640 2024-03-24T00:31:33.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.566+0000 7f7197200640 1 -- 172.21.15.40:0/4135469635 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71980c9730 con 0x7f71980cf640 2024-03-24T00:31:33.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.570+0000 7f7197200640 1 -- 172.21.15.40:0/4135469635 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f719804f410 con 0x7f71980cf640 2024-03-24T00:31:33.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.570+0000 7f718ea00640 1 -- 172.21.15.40:0/4135469635 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f71900bf020 con 0x7f71980cf640 2024-03-24T00:31:33.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.570+0000 7f718ea00640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7174067690 0x7f7174069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.570+0000 7f718ea00640 1 -- 172.21.15.40:0/4135469635 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71901207a0 con 0x7f71980cf640 2024-03-24T00:31:33.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.570+0000 7f7195e00640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7174067690 0x7f7174069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.570+0000 7f7195e00640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7174067690 0x7f7174069b50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7188009ae0 tx=0x7f7188008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:33.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.578+0000 7f718ea00640 1 -- 172.21.15.40:0/4135469635 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f71900c6030 con 0x7f71980cf640 2024-03-24T00:31:33.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 -- 172.21.15.40:0/4135469635 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f719806b430 con 0x7f71980cf640 2024-03-24T00:31:33.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f718ea00640 1 -- 172.21.15.40:0/4135469635 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f71900165e0 con 0x7f71980cf640 2024-03-24T00:31:33.745 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:33.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 -- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7174067690 msgr2=0x7f7174069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7174067690 0x7f7174069b50 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7188009ae0 tx=0x7f7188008040 comp rx=0 tx=0).stop 2024-03-24T00:31:33.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 -- 172.21.15.40:0/4135469635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71980cf640 msgr2=0x7f71980afd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71980cf640 0x7f71980afd40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7190009e20 tx=0x7f7190019790 comp rx=0 tx=0).stop 2024-03-24T00:31:33.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 -- 172.21.15.40:0/4135469635 shutdown_connections 2024-03-24T00:31:33.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7174067690 0x7f7174069b50 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71980cf640 0x7f71980afd40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71980ce360 0x7f71980b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 --2- 172.21.15.40:0/4135469635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f719806cd70 0x7f719814eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 -- 172.21.15.40:0/4135469635 >> 172.21.15.40:0/4135469635 conn(0x7f71980d19e0 msgr2=0x7f7198155b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:33.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 -- 172.21.15.40:0/4135469635 shutdown_connections 2024-03-24T00:31:33.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.742+0000 7f7197200640 1 -- 172.21.15.40:0/4135469635 wait complete. 2024-03-24T00:31:33.765 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:33.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.906+0000 7f343a200640 1 Processor -- start 2024-03-24T00:31:33.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.906+0000 7f343a200640 1 -- start start 2024-03-24T00:31:33.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.906+0000 7f343a200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f343415bd40 0x7f34340d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.906+0000 7f343a200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f34340d1520 0x7f34340cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.906+0000 7f343a200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343415f970 con 0x7f343415b960 2024-03-24T00:31:33.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.906+0000 7f343a200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3434160580 con 0x7f34340d1520 2024-03-24T00:31:33.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.906+0000 7f343a200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343415fc50 con 0x7f343415bd40 2024-03-24T00:31:33.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.906+0000 7f3433400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f343415b960 0x7f3434070d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41208/0 (socket says 172.21.15.40:41208) 2024-03-24T00:31:33.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.906+0000 7f3433400640 1 -- 172.21.15.40:0/2343795985 learned_addr learned my addr 172.21.15.40:0/2343795985 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:33.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.906+0000 7f342be00640 1 --2- 172.21.15.40:0/2343795985 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f343415bd40 0x7f34340d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.906+0000 7f3433e00640 1 --2- 172.21.15.40:0/2343795985 >> v2:172.21.15.53:3300/0 conn(0x7f34340d1520 0x7f34340cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.910+0000 7f3432a00640 1 -- 172.21.15.40:0/2343795985 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593755325 0 0) 0x7f343415f970 con 0x7f343415b960 2024-03-24T00:31:33.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.910+0000 7f3432a00640 1 -- 172.21.15.40:0/2343795985 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3418003530 con 0x7f343415b960 2024-03-24T00:31:33.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.910+0000 7f3433e00640 1 -- 172.21.15.40:0/2343795985 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f343415bd40 msgr2=0x7f34340d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:33.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.910+0000 7f3433e00640 1 --2- 172.21.15.40:0/2343795985 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f343415bd40 0x7f34340d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.910+0000 7f3433e00640 1 -- 172.21.15.40:0/2343795985 >> v1:172.21.15.40:6789/0 conn(0x7f343415b960 legacy=0x7f3434070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.910+0000 7f3433e00640 1 -- 172.21.15.40:0/2343795985 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343415ff30 con 0x7f34340d1520 2024-03-24T00:31:33.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.910+0000 7f342be00640 1 --2- 172.21.15.40:0/2343795985 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f343415bd40 0x7f34340d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:33.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.910+0000 7f3433e00640 1 --2- 172.21.15.40:0/2343795985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34340d1520 0x7f34340cda00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f342c009d90 tx=0x7f342c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38f132ec62c87b92 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:33.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.910+0000 7f3432a00640 1 -- 172.21.15.40:0/2343795985 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c0b2070 con 0x7f34340d1520 2024-03-24T00:31:33.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.910+0000 7f3432a00640 1 -- 172.21.15.40:0/2343795985 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342c0a5830 con 0x7f34340d1520 2024-03-24T00:31:33.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.910+0000 7f3432a00640 1 -- 172.21.15.40:0/2343795985 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342c0a5c00 con 0x7f34340d1520 2024-03-24T00:31:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 -- 172.21.15.40:0/2343795985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34340d1520 msgr2=0x7f34340cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 --2- 172.21.15.40:0/2343795985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34340d1520 0x7f34340cda00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f342c009d90 tx=0x7f342c01c970 comp rx=0 tx=0).stop 2024-03-24T00:31:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 -- 172.21.15.40:0/2343795985 shutdown_connections 2024-03-24T00:31:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 --2- 172.21.15.40:0/2343795985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34340d1520 0x7f34340cda00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 --2- 172.21.15.40:0/2343795985 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f343415bd40 0x7f34340d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 -- 172.21.15.40:0/2343795985 >> 172.21.15.40:0/2343795985 conn(0x7f343406ecf0 msgr2=0x7f343406f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 -- 172.21.15.40:0/2343795985 shutdown_connections 2024-03-24T00:31:33.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 -- 172.21.15.40:0/2343795985 wait complete. 2024-03-24T00:31:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 Processor -- start 2024-03-24T00:31:33.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 -- start start 2024-03-24T00:31:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34340d1520 0x7f343414ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f343415b960 0x7f34340cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f343415bd40 0x7f34341f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34340ceae0 con 0x7f343415b960 2024-03-24T00:31:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34340ce960 con 0x7f343415bd40 2024-03-24T00:31:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f343a200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34340cec60 con 0x7f34340d1520 2024-03-24T00:31:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f342be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f343415b960 0x7f34340cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f342be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f343415b960 0x7f34340cab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45796/0 (socket says 172.21.15.40:45796) 2024-03-24T00:31:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f342be00640 1 -- 172.21.15.40:0/1775702174 learned_addr learned my addr 172.21.15.40:0/1775702174 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f3433e00640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f343415bd40 0x7f34341f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f3433400640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34340d1520 0x7f343414ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f342be00640 1 -- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34340d1520 msgr2=0x7f343414ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f342be00640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34340d1520 0x7f343414ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f342be00640 1 -- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f343415bd40 msgr2=0x7f34341f7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f342be00640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f343415bd40 0x7f34341f7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f342be00640 1 -- 172.21.15.40:0/1775702174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3434150470 con 0x7f343415b960 2024-03-24T00:31:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.914+0000 7f3433400640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34340d1520 0x7f343414ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.918+0000 7f342be00640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f343415b960 0x7f34340cab40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f341c00e980 tx=0x7f341c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.918+0000 7f342b400640 1 -- 172.21.15.40:0/1775702174 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c017070 con 0x7f343415b960 2024-03-24T00:31:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.918+0000 7f342b400640 1 -- 172.21.15.40:0/1775702174 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c00ca00 con 0x7f343415b960 2024-03-24T00:31:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.918+0000 7f342b400640 1 -- 172.21.15.40:0/1775702174 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c00cdd0 con 0x7f343415b960 2024-03-24T00:31:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.918+0000 7f343a200640 1 -- 172.21.15.40:0/1775702174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34340ca760 con 0x7f343415b960 2024-03-24T00:31:33.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.918+0000 7f343a200640 1 -- 172.21.15.40:0/1775702174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34340ca1a0 con 0x7f343415b960 2024-03-24T00:31:33.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.918+0000 7f343a200640 1 -- 172.21.15.40:0/1775702174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3434068170 con 0x7f343415b960 2024-03-24T00:31:33.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.918+0000 7f342b400640 1 -- 172.21.15.40:0/1775702174 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f341c01a070 con 0x7f343415b960 2024-03-24T00:31:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.922+0000 7f342b400640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3410067690 0x7f3410069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.922+0000 7f342b400640 1 -- 172.21.15.40:0/1775702174 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f341c085790 con 0x7f343415b960 2024-03-24T00:31:33.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.922+0000 7f3433400640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3410067690 0x7f3410069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:33.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.922+0000 7f3433400640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3410067690 0x7f3410069b50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3424005890 tx=0x7f3424009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:33.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:33.926+0000 7f342b400640 1 -- 172.21.15.40:0/1775702174 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3434150470 con 0x7f343415b960 2024-03-24T00:31:34.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.094+0000 7f343a200640 1 -- 172.21.15.40:0/1775702174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f343415dc20 con 0x7f343415b960 2024-03-24T00:31:34.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.094+0000 7f342b400640 1 -- 172.21.15.40:0/1775702174 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f341c055d20 con 0x7f343415b960 2024-03-24T00:31:34.097 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:34.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.094+0000 7f343a200640 1 -- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3410067690 msgr2=0x7f3410069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.094+0000 7f343a200640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3410067690 0x7f3410069b50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3424005890 tx=0x7f3424009040 comp rx=0 tx=0).stop 2024-03-24T00:31:34.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.094+0000 7f343a200640 1 -- 172.21.15.40:0/1775702174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f343415b960 msgr2=0x7f34340cab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.094+0000 7f343a200640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f343415b960 0x7f34340cab40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f341c00e980 tx=0x7f341c00ee40 comp rx=0 tx=0).stop 2024-03-24T00:31:34.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.098+0000 7f343a200640 1 -- 172.21.15.40:0/1775702174 shutdown_connections 2024-03-24T00:31:34.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.098+0000 7f343a200640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3410067690 0x7f3410069b50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.098+0000 7f343a200640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f343415bd40 0x7f34341f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.098+0000 7f343a200640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f343415b960 0x7f34340cab40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.098+0000 7f343a200640 1 --2- 172.21.15.40:0/1775702174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34340d1520 0x7f343414ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.098+0000 7f343a200640 1 -- 172.21.15.40:0/1775702174 >> 172.21.15.40:0/1775702174 conn(0x7f343406ecf0 msgr2=0x7f3434150eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:34.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.098+0000 7f343a200640 1 -- 172.21.15.40:0/1775702174 shutdown_connections 2024-03-24T00:31:34.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.098+0000 7f343a200640 1 -- 172.21.15.40:0/1775702174 wait complete. 2024-03-24T00:31:34.119 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e7ba00640 1 Processor -- start 2024-03-24T00:31:34.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e7ba00640 1 -- start start 2024-03-24T00:31:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e7ba00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7e7415b960 0x7f7e74070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e7ba00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7415f970 con 0x7f7e740d16f0 2024-03-24T00:31:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e7ba00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e7415fc50 con 0x7f7e7415bd40 2024-03-24T00:31:34.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e7ba00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e74160580 con 0x7f7e7415b960 2024-03-24T00:31:34.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e79a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7e740d16f0 0x7f7e740cda00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41220/0 (socket says 172.21.15.40:41220) 2024-03-24T00:31:34.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e79a00640 1 -- 172.21.15.40:0/538771142 learned_addr learned my addr 172.21.15.40:0/538771142 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:34.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e79000640 1 --2- 172.21.15.40:0/538771142 >> v2:172.21.15.53:3300/0 conn(0x7f7e7415b960 0x7f7e74070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:34.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e73400640 1 -- 172.21.15.40:0/538771142 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601994006 0 0) 0x7f7e7415f970 con 0x7f7e740d16f0 2024-03-24T00:31:34.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e73400640 1 -- 172.21.15.40:0/538771142 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e5c003530 con 0x7f7e740d16f0 2024-03-24T00:31:34.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e73400640 1 -- 172.21.15.40:0/538771142 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471038480 0 0) 0x7f7e7415fc50 con 0x7f7e7415bd40 2024-03-24T00:31:34.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e79000640 1 -- 172.21.15.40:0/538771142 >> v1:172.21.15.53:6789/0 conn(0x7f7e7415bd40 legacy=0x7f7e740d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e79000640 1 -- 172.21.15.40:0/538771142 >> v1:172.21.15.40:6789/0 conn(0x7f7e740d16f0 legacy=0x7f7e740cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.262+0000 7f7e79000640 1 -- 172.21.15.40:0/538771142 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7415ff30 con 0x7f7e7415b960 2024-03-24T00:31:34.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.266+0000 7f7e79000640 1 --2- 172.21.15.40:0/538771142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e7415b960 0x7f7e74070d60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7e680097c0 tx=0x7f7e6801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=501c62822e9e5859 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:34.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.266+0000 7f7e73400640 1 -- 172.21.15.40:0/538771142 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e680b0070 con 0x7f7e7415b960 2024-03-24T00:31:34.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.266+0000 7f7e73400640 1 -- 172.21.15.40:0/538771142 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e680ab600 con 0x7f7e7415b960 2024-03-24T00:31:34.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.266+0000 7f7e73400640 1 -- 172.21.15.40:0/538771142 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e680ab9d0 con 0x7f7e7415b960 2024-03-24T00:31:34.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.266+0000 7f7e7ba00640 1 -- 172.21.15.40:0/538771142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e7415b960 msgr2=0x7f7e74070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.266+0000 7f7e7ba00640 1 --2- 172.21.15.40:0/538771142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e7415b960 0x7f7e74070d60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7e680097c0 tx=0x7f7e6801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.266+0000 7f7e7ba00640 1 -- 172.21.15.40:0/538771142 shutdown_connections 2024-03-24T00:31:34.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.266+0000 7f7e7ba00640 1 --2- 172.21.15.40:0/538771142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e7415b960 0x7f7e74070d60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.266+0000 7f7e7ba00640 1 -- 172.21.15.40:0/538771142 >> 172.21.15.40:0/538771142 conn(0x7f7e7406ecf0 msgr2=0x7f7e7406f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:34.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.266+0000 7f7e7ba00640 1 -- 172.21.15.40:0/538771142 shutdown_connections 2024-03-24T00:31:34.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.266+0000 7f7e7ba00640 1 -- 172.21.15.40:0/538771142 wait complete. 2024-03-24T00:31:34.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e7ba00640 1 Processor -- start 2024-03-24T00:31:34.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e7ba00640 1 -- start start 2024-03-24T00:31:34.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e7ba00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e740d16f0 0x7f7e741496e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:34.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e7ba00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e7415b960 0x7f7e7414e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:34.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e7ba00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e7415bd40 0x7f7e74145ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:34.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e7ba00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740ced30 con 0x7f7e740d16f0 2024-03-24T00:31:34.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e7ba00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e740cebb0 con 0x7f7e7415b960 2024-03-24T00:31:34.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e7ba00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e74169b80 con 0x7f7e7415bd40 2024-03-24T00:31:34.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e79000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e740d16f0 0x7f7e741496e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:34.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e79000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e740d16f0 0x7f7e741496e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45812/0 (socket says 172.21.15.40:45812) 2024-03-24T00:31:34.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e79000640 1 -- 172.21.15.40:0/270227328 learned_addr learned my addr 172.21.15.40:0/270227328 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:34.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e79a00640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e7415bd40 0x7f7e74145ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:34.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e73e00640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e7415b960 0x7f7e7414e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:34.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e79000640 1 -- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e7415bd40 msgr2=0x7f7e74145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e79000640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e7415bd40 0x7f7e74145ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e79000640 1 -- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e7415b960 msgr2=0x7f7e7414e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e79000640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e7415b960 0x7f7e7414e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e79000640 1 -- 172.21.15.40:0/270227328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e74165810 con 0x7f7e740d16f0 2024-03-24T00:31:34.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e79000640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e740d16f0 0x7f7e741496e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7e680097c0 tx=0x7f7e6801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:34.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e70c00640 1 -- 172.21.15.40:0/270227328 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e680bb020 con 0x7f7e740d16f0 2024-03-24T00:31:34.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e70c00640 1 -- 172.21.15.40:0/270227328 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e680b1040 con 0x7f7e740d16f0 2024-03-24T00:31:34.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e70c00640 1 -- 172.21.15.40:0/270227328 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e680b0050 con 0x7f7e740d16f0 2024-03-24T00:31:34.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e7ba00640 1 -- 172.21.15.40:0/270227328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e74165af0 con 0x7f7e740d16f0 2024-03-24T00:31:34.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e7ba00640 1 -- 172.21.15.40:0/270227328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e741660b0 con 0x7f7e740d16f0 2024-03-24T00:31:34.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.270+0000 7f7e7ba00640 1 -- 172.21.15.40:0/270227328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e74068170 con 0x7f7e740d16f0 2024-03-24T00:31:34.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.274+0000 7f7e70c00640 1 -- 172.21.15.40:0/270227328 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7e680be070 con 0x7f7e740d16f0 2024-03-24T00:31:34.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.274+0000 7f7e70c00640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e54067660 0x7f7e54069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:34.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.274+0000 7f7e70c00640 1 -- 172.21.15.40:0/270227328 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e6811f210 con 0x7f7e740d16f0 2024-03-24T00:31:34.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.274+0000 7f7e73e00640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e54067660 0x7f7e54069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:34.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.274+0000 7f7e73e00640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e54067660 0x7f7e54069b20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7e7406b640 tx=0x7f7e60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:34.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.278+0000 7f7e70c00640 1 -- 172.21.15.40:0/270227328 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7e68016090 con 0x7f7e740d16f0 2024-03-24T00:31:34.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.442+0000 7f7e7ba00640 1 -- 172.21.15.40:0/270227328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7e74070910 con 0x7f7e740d16f0 2024-03-24T00:31:34.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.442+0000 7f7e70c00640 1 -- 172.21.15.40:0/270227328 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7e680efe80 con 0x7f7e740d16f0 2024-03-24T00:31:34.447 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:34.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.446+0000 7f7e7ba00640 1 -- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e54067660 msgr2=0x7f7e54069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.446+0000 7f7e7ba00640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e54067660 0x7f7e54069b20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7e7406b640 tx=0x7f7e60009040 comp rx=0 tx=0).stop 2024-03-24T00:31:34.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.446+0000 7f7e7ba00640 1 -- 172.21.15.40:0/270227328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e740d16f0 msgr2=0x7f7e741496e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.446+0000 7f7e7ba00640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e740d16f0 0x7f7e741496e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7e680097c0 tx=0x7f7e6801ace0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.446+0000 7f7e7ba00640 1 -- 172.21.15.40:0/270227328 shutdown_connections 2024-03-24T00:31:34.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.446+0000 7f7e7ba00640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e54067660 0x7f7e54069b20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.446+0000 7f7e7ba00640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e7415bd40 0x7f7e74145ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.446+0000 7f7e7ba00640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e7415b960 0x7f7e7414e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.446+0000 7f7e7ba00640 1 --2- 172.21.15.40:0/270227328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e740d16f0 0x7f7e741496e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.446+0000 7f7e7ba00640 1 -- 172.21.15.40:0/270227328 >> 172.21.15.40:0/270227328 conn(0x7f7e7406ecf0 msgr2=0x7f7e740b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:34.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.446+0000 7f7e7ba00640 1 -- 172.21.15.40:0/270227328 shutdown_connections 2024-03-24T00:31:34.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.446+0000 7f7e7ba00640 1 -- 172.21.15.40:0/270227328 wait complete. 2024-03-24T00:31:34.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_conf_off' 2024-03-24T00:31:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.606+0000 7fb456400640 1 Processor -- start 2024-03-24T00:31:34.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.606+0000 7fb456400640 1 -- start start 2024-03-24T00:31:34.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.606+0000 7fb456400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb45015f600 0x7fb4500b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:34.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.610+0000 7fb456400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb4500af170 0x7fb450165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:34.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.610+0000 7fb456400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4500d2c70 con 0x7fb4500d2830 2024-03-24T00:31:34.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.610+0000 7fb456400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4500b3630 con 0x7fb4500af170 2024-03-24T00:31:34.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.610+0000 7fb456400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4500b2740 con 0x7fb45015f600 2024-03-24T00:31:34.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.610+0000 7fb44fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb4500af170 0x7fb450165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:34.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.610+0000 7fb44fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb4500af170 0x7fb450165e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45820/0 (socket says 172.21.15.40:45820) 2024-03-24T00:31:34.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.610+0000 7fb44fe00640 1 -- 172.21.15.40:0/3164932012 learned_addr learned my addr 172.21.15.40:0/3164932012 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:34.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.610+0000 7fb44f400640 1 --2- 172.21.15.40:0/3164932012 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb45015f600 0x7fb4500b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:34.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.610+0000 7fb44fe00640 1 -- 172.21.15.40:0/3164932012 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb45015f600 msgr2=0x7fb4500b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:34.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.610+0000 7fb44fe00640 1 --2- 172.21.15.40:0/3164932012 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb45015f600 0x7fb4500b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.610+0000 7fb44fe00640 1 -- 172.21.15.40:0/3164932012 >> v1:172.21.15.53:6789/0 conn(0x7fb4500d2830 legacy=0x7fb4500aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.610+0000 7fb44fe00640 1 -- 172.21.15.40:0/3164932012 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4500b1e50 con 0x7fb4500af170 2024-03-24T00:31:34.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.614+0000 7fb44fe00640 1 --2- 172.21.15.40:0/3164932012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4500af170 0x7fb450165e40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb448009e50 tx=0x7fb44801f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eafd16ff7009428 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:34.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.614+0000 7fb44ea00640 1 -- 172.21.15.40:0/3164932012 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4480b4070 con 0x7fb4500af170 2024-03-24T00:31:34.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.614+0000 7fb44ea00640 1 -- 172.21.15.40:0/3164932012 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4480af400 con 0x7fb4500af170 2024-03-24T00:31:34.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.614+0000 7fb44ea00640 1 -- 172.21.15.40:0/3164932012 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4480af7d0 con 0x7fb4500af170 2024-03-24T00:31:34.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.614+0000 7fb456400640 1 -- 172.21.15.40:0/3164932012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4500af170 msgr2=0x7fb450165e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.614+0000 7fb456400640 1 --2- 172.21.15.40:0/3164932012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4500af170 0x7fb450165e40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb448009e50 tx=0x7fb44801f210 comp rx=0 tx=0).stop 2024-03-24T00:31:34.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.614+0000 7fb456400640 1 -- 172.21.15.40:0/3164932012 shutdown_connections 2024-03-24T00:31:34.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.614+0000 7fb456400640 1 --2- 172.21.15.40:0/3164932012 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb4500af170 0x7fb450165e40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.614+0000 7fb456400640 1 --2- 172.21.15.40:0/3164932012 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb45015f600 0x7fb4500b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.614+0000 7fb456400640 1 -- 172.21.15.40:0/3164932012 >> 172.21.15.40:0/3164932012 conn(0x7fb4500d1d10 msgr2=0x7fb4500d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:34.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.614+0000 7fb456400640 1 -- 172.21.15.40:0/3164932012 shutdown_connections 2024-03-24T00:31:34.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.614+0000 7fb456400640 1 -- 172.21.15.40:0/3164932012 wait complete. 2024-03-24T00:31:34.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb456400640 1 Processor -- start 2024-03-24T00:31:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb456400640 1 -- start start 2024-03-24T00:31:34.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb456400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4500af170 0x7fb4501f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:34.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb456400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb4500d2830 0x7fb4500b5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb456400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb45015f600 0x7fb4501f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb456400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4500b2fa0 con 0x7fb45015f600 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb456400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4500b23d0 con 0x7fb4500af170 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb456400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb45015f9e0 con 0x7fb4500d2830 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb44fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb45015f600 0x7fb4501f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb44fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb45015f600 0x7fb4501f7640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45830/0 (socket says 172.21.15.40:45830) 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb44fe00640 1 -- 172.21.15.40:0/1305223800 learned_addr learned my addr 172.21.15.40:0/1305223800 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb44f400640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4500af170 0x7fb4501f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb447e00640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb4500d2830 0x7fb4500b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb44fe00640 1 -- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb4500d2830 msgr2=0x7fb4500b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb44fe00640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb4500d2830 0x7fb4500b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb44fe00640 1 -- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4500af170 msgr2=0x7fb4501f3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:34.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb44fe00640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4500af170 0x7fb4501f3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:34.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb44fe00640 1 -- 172.21.15.40:0/1305223800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4501f40c0 con 0x7fb45015f600 2024-03-24T00:31:34.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb44fe00640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb45015f600 0x7fb4501f7640 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb4480b2040 tx=0x7fb44801bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:34.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb447400640 1 -- 172.21.15.40:0/1305223800 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4480c0070 con 0x7fb45015f600 2024-03-24T00:31:34.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb447400640 1 -- 172.21.15.40:0/1305223800 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4480090d0 con 0x7fb45015f600 2024-03-24T00:31:34.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb456400640 1 -- 172.21.15.40:0/1305223800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4501f43a0 con 0x7fb45015f600 2024-03-24T00:31:34.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb447400640 1 -- 172.21.15.40:0/1305223800 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4480b4040 con 0x7fb45015f600 2024-03-24T00:31:34.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb456400640 1 -- 172.21.15.40:0/1305223800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4501f4960 con 0x7fb45015f600 2024-03-24T00:31:34.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.618+0000 7fb456400640 1 -- 172.21.15.40:0/1305223800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb450068170 con 0x7fb45015f600 2024-03-24T00:31:34.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.622+0000 7fb447400640 1 -- 172.21.15.40:0/1305223800 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb4480c3070 con 0x7fb45015f600 2024-03-24T00:31:34.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.622+0000 7fb447400640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb42c067660 0x7fb42c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:34.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.622+0000 7fb447400640 1 -- 172.21.15.40:0/1305223800 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb448122cb0 con 0x7fb45015f600 2024-03-24T00:31:34.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.622+0000 7fb44f400640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb42c067660 0x7fb42c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:34.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.622+0000 7fb44f400640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb42c067660 0x7fb42c069b20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb45009fd60 tx=0x7fb440009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:34.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.630+0000 7fb447400640 1 -- 172.21.15.40:0/1305223800 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb448018d70 con 0x7fb45015f600 2024-03-24T00:31:34.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:34.854+0000 7fb456400640 1 -- 172.21.15.40:0/1305223800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_conf_off"]} v 0) v1 -- 0x7fb4500b18c0 con 0x7fb45015f600 2024-03-24T00:31:35.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.362+0000 7fb447400640 1 -- 172.21.15.40:0/1305223800 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_conf_off"]}]=0 v48) v1 ==== 158+0+0 (secure 0 0 0) 0x7fb4480f33a0 con 0x7fb45015f600 2024-03-24T00:31:35.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.366+0000 7fb456400640 1 -- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb42c067660 msgr2=0x7fb42c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:35.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.366+0000 7fb456400640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb42c067660 0x7fb42c069b20 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb45009fd60 tx=0x7fb440009190 comp rx=0 tx=0).stop 2024-03-24T00:31:35.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.366+0000 7fb456400640 1 -- 172.21.15.40:0/1305223800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb45015f600 msgr2=0x7fb4501f7640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:35.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.366+0000 7fb456400640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb45015f600 0x7fb4501f7640 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb4480b2040 tx=0x7fb44801bfb0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.366+0000 7fb456400640 1 -- 172.21.15.40:0/1305223800 shutdown_connections 2024-03-24T00:31:35.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.366+0000 7fb456400640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb42c067660 0x7fb42c069b20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.366+0000 7fb456400640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb45015f600 0x7fb4501f7640 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.366+0000 7fb456400640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb4500d2830 0x7fb4500b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.366+0000 7fb456400640 1 --2- 172.21.15.40:0/1305223800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4500af170 0x7fb4501f3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.366+0000 7fb456400640 1 -- 172.21.15.40:0/1305223800 >> 172.21.15.40:0/1305223800 conn(0x7fb4500d1d10 msgr2=0x7fb4500ceb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:35.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.366+0000 7fb456400640 1 -- 172.21.15.40:0/1305223800 shutdown_connections 2024-03-24T00:31:35.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.366+0000 7fb456400640 1 -- 172.21.15.40:0/1305223800 wait complete. 2024-03-24T00:31:35.389 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:31:35.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f09a5200640 1 Processor -- start 2024-03-24T00:31:35.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f09a5200640 1 -- start start 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f09a5200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f09a015b960 0x7f09a0070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f09a5200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09a015bd40 0x7f09a00712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f09a5200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a015f970 con 0x7f09a00d0fe0 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f09a5200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a0160580 con 0x7f09a015b960 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f09a5200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a015fc50 con 0x7f09a015bd40 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f099de00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f09a015b960 0x7f09a0070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f099de00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f09a015b960 0x7f09a0070d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45834/0 (socket says 172.21.15.40:45834) 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f099de00640 1 -- 172.21.15.40:0/2054678722 learned_addr learned my addr 172.21.15.40:0/2054678722 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f0995400640 1 --2- 172.21.15.40:0/2054678722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09a015bd40 0x7f09a00712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f099de00640 1 -- 172.21.15.40:0/2054678722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09a015bd40 msgr2=0x7f09a00712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f099de00640 1 --2- 172.21.15.40:0/2054678722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09a015bd40 0x7f09a00712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f099de00640 1 -- 172.21.15.40:0/2054678722 >> v1:172.21.15.53:6789/0 conn(0x7f09a00d0fe0 legacy=0x7f09a00cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f099de00640 1 -- 172.21.15.40:0/2054678722 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a015ff30 con 0x7f09a015b960 2024-03-24T00:31:35.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.530+0000 7f0995400640 1 --2- 172.21.15.40:0/2054678722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09a015bd40 0x7f09a00712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:35.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.534+0000 7f099de00640 1 --2- 172.21.15.40:0/2054678722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a015b960 0x7f09a0070d60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0988009870 tx=0x7f098801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772ba29946d34f5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:35.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.534+0000 7f099d400640 1 -- 172.21.15.40:0/2054678722 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09880b2070 con 0x7f09a015b960 2024-03-24T00:31:35.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.534+0000 7f099d400640 1 -- 172.21.15.40:0/2054678722 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09880a9070 con 0x7f09a015b960 2024-03-24T00:31:35.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.534+0000 7f099d400640 1 -- 172.21.15.40:0/2054678722 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09880ad880 con 0x7f09a015b960 2024-03-24T00:31:35.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.534+0000 7f09a5200640 1 -- 172.21.15.40:0/2054678722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a015b960 msgr2=0x7f09a0070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:35.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.534+0000 7f09a5200640 1 --2- 172.21.15.40:0/2054678722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a015b960 0x7f09a0070d60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0988009870 tx=0x7f098801c560 comp rx=0 tx=0).stop 2024-03-24T00:31:35.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.534+0000 7f09a5200640 1 -- 172.21.15.40:0/2054678722 shutdown_connections 2024-03-24T00:31:35.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.534+0000 7f09a5200640 1 --2- 172.21.15.40:0/2054678722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09a015bd40 0x7f09a00712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.534+0000 7f09a5200640 1 --2- 172.21.15.40:0/2054678722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a015b960 0x7f09a0070d60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.534+0000 7f09a5200640 1 -- 172.21.15.40:0/2054678722 >> 172.21.15.40:0/2054678722 conn(0x7f09a006ecf0 msgr2=0x7f09a006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:35.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.534+0000 7f09a5200640 1 -- 172.21.15.40:0/2054678722 shutdown_connections 2024-03-24T00:31:35.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.534+0000 7f09a5200640 1 -- 172.21.15.40:0/2054678722 wait complete. 2024-03-24T00:31:35.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f09a5200640 1 Processor -- start 2024-03-24T00:31:35.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f09a5200640 1 -- start start 2024-03-24T00:31:35.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f09a5200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09a00d0fe0 0x7f09a0149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:35.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f09a5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a015bd40 0x7f09a00c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f09a5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a015c250 0x7f09a01524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f09a5200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a0168f00 con 0x7f09a015c250 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f09a5200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a0168d80 con 0x7f09a015bd40 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f09a5200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a0169080 con 0x7f09a00d0fe0 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f099e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a015c250 0x7f09a01524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f099e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a015c250 0x7f09a01524f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45846/0 (socket says 172.21.15.40:45846) 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f099e800640 1 -- 172.21.15.40:0/928637569 learned_addr learned my addr 172.21.15.40:0/928637569 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f099de00640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09a00d0fe0 0x7f09a0149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f0995400640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a015bd40 0x7f09a00c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f099e800640 1 -- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09a00d0fe0 msgr2=0x7f09a0149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f099e800640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09a00d0fe0 0x7f09a0149b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f099e800640 1 -- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a015bd40 msgr2=0x7f09a00c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f099e800640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a015bd40 0x7f09a00c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f099e800640 1 -- 172.21.15.40:0/928637569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a0156850 con 0x7f09a015c250 2024-03-24T00:31:35.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f0995400640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a015bd40 0x7f09a00c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:35.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f099e800640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a015c250 0x7f09a01524f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f099800c7a0 tx=0x7f099800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:35.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f0996a00640 1 -- 172.21.15.40:0/928637569 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0998003b80 con 0x7f09a015c250 2024-03-24T00:31:35.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f0996a00640 1 -- 172.21.15.40:0/928637569 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099800ce50 con 0x7f09a015c250 2024-03-24T00:31:35.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f0996a00640 1 -- 172.21.15.40:0/928637569 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099800a7f0 con 0x7f09a015c250 2024-03-24T00:31:35.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f09a5200640 1 -- 172.21.15.40:0/928637569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a00c7d20 con 0x7f09a015c250 2024-03-24T00:31:35.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.538+0000 7f09a5200640 1 -- 172.21.15.40:0/928637569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a015cb40 con 0x7f09a015c250 2024-03-24T00:31:35.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.542+0000 7f09a5200640 1 -- 172.21.15.40:0/928637569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a0068170 con 0x7f09a015c250 2024-03-24T00:31:35.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.542+0000 7f0996a00640 1 -- 172.21.15.40:0/928637569 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f099800a990 con 0x7f09a015c250 2024-03-24T00:31:35.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.542+0000 7f0996a00640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f097c067690 0x7f097c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:35.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.542+0000 7f0996a00640 1 -- 172.21.15.40:0/928637569 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0998085810 con 0x7f09a015c250 2024-03-24T00:31:35.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.542+0000 7f099de00640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f097c067690 0x7f097c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:35.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.542+0000 7f099de00640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f097c067690 0x7f097c069b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f09880af000 tx=0x7f09880a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:35.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.546+0000 7f0996a00640 1 -- 172.21.15.40:0/928637569 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f09980563f0 con 0x7f09a015c250 2024-03-24T00:31:35.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.734+0000 7f09a5200640 1 -- 172.21.15.40:0/928637569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f09a0166940 con 0x7f09a015c250 2024-03-24T00:31:35.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.738+0000 7f0996a00640 1 -- 172.21.15.40:0/928637569 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0998055cf0 con 0x7f09a015c250 2024-03-24T00:31:35.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.738+0000 7f09a5200640 1 -- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f097c067690 msgr2=0x7f097c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:35.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.738+0000 7f09a5200640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f097c067690 0x7f097c069b50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f09880af000 tx=0x7f09880a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:35.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.738+0000 7f09a5200640 1 -- 172.21.15.40:0/928637569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a015c250 msgr2=0x7f09a01524f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:35.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.738+0000 7f09a5200640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a015c250 0x7f09a01524f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f099800c7a0 tx=0x7f099800cc60 comp rx=0 tx=0).stop 2024-03-24T00:31:35.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.738+0000 7f09a5200640 1 -- 172.21.15.40:0/928637569 shutdown_connections 2024-03-24T00:31:35.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.738+0000 7f09a5200640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f097c067690 0x7f097c069b50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.738+0000 7f09a5200640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09a015c250 0x7f09a01524f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.738+0000 7f09a5200640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a015bd40 0x7f09a00c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.738+0000 7f09a5200640 1 --2- 172.21.15.40:0/928637569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09a00d0fe0 0x7f09a0149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:35.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.742+0000 7f09a5200640 1 -- 172.21.15.40:0/928637569 >> 172.21.15.40:0/928637569 conn(0x7f09a006ecf0 msgr2=0x7f09a0153b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:35.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.742+0000 7f09a5200640 1 -- 172.21.15.40:0/928637569 shutdown_connections 2024-03-24T00:31:35.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:35.742+0000 7f09a5200640 1 -- 172.21.15.40:0/928637569 wait complete. 2024-03-24T00:31:35.773 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:31:35.773 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:31:35.827 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:31:35.939 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:31:35.939 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:31:35.939 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:31:35.948 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:31:35.948 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:31:36.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:36.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1dffc00640 1 Processor -- start 2024-03-24T00:31:36.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1dffc00640 1 -- start start 2024-03-24T00:31:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1dffc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1df806e5a0 0x7f1df806e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1dffc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1df8070c80 0x7f1df806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1dffc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1df8071690 0x7f1df80b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1dffc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df806cdc0 con 0x7f1df8070c80 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1dffc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df806eec0 con 0x7f1df806e5a0 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1dffc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df8071160 con 0x7f1df8071690 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1defe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1df8070c80 0x7f1df806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1defe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1df8070c80 0x7f1df806db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45850/0 (socket says 172.21.15.40:45850) 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1defe00640 1 -- 172.21.15.40:0/42731772 learned_addr learned my addr 172.21.15.40:0/42731772 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1dfdc00640 1 --2- 172.21.15.40:0/42731772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1df8071690 0x7f1df80b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1dfd200640 1 --2- 172.21.15.40:0/42731772 >> v2:172.21.15.53:3300/0 conn(0x7f1df806e5a0 0x7f1df806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1defe00640 1 -- 172.21.15.40:0/42731772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1df8071690 msgr2=0x7f1df80b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1defe00640 1 --2- 172.21.15.40:0/42731772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1df8071690 0x7f1df80b33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1defe00640 1 -- 172.21.15.40:0/42731772 >> v2:172.21.15.53:3300/0 conn(0x7f1df806e5a0 msgr2=0x7f1df806e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1defe00640 1 --2- 172.21.15.40:0/42731772 >> v2:172.21.15.53:3300/0 conn(0x7f1df806e5a0 0x7f1df806e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1defe00640 1 -- 172.21.15.40:0/42731772 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df806d040 con 0x7f1df8070c80 2024-03-24T00:31:36.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.182+0000 7f1dfd200640 1 --2- 172.21.15.40:0/42731772 >> v2:172.21.15.53:3300/0 conn(0x7f1df806e5a0 0x7f1df806e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:36.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1defe00640 1 --2- 172.21.15.40:0/42731772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8070c80 0x7f1df806db20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1df80b3390 tx=0x7f1de001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f69120538a4a6b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:36.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1df7e00640 1 -- 172.21.15.40:0/42731772 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de00b0070 con 0x7f1df8070c80 2024-03-24T00:31:36.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1df7e00640 1 -- 172.21.15.40:0/42731772 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de00ab600 con 0x7f1df8070c80 2024-03-24T00:31:36.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1df7e00640 1 -- 172.21.15.40:0/42731772 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de00ab9d0 con 0x7f1df8070c80 2024-03-24T00:31:36.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1dffc00640 1 -- 172.21.15.40:0/42731772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8070c80 msgr2=0x7f1df806db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1dffc00640 1 --2- 172.21.15.40:0/42731772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8070c80 0x7f1df806db20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1df80b3390 tx=0x7f1de001c520 comp rx=0 tx=0).stop 2024-03-24T00:31:36.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1dffc00640 1 -- 172.21.15.40:0/42731772 shutdown_connections 2024-03-24T00:31:36.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1dffc00640 1 --2- 172.21.15.40:0/42731772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1df8071690 0x7f1df80b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1dffc00640 1 --2- 172.21.15.40:0/42731772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8070c80 0x7f1df806db20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1dffc00640 1 --2- 172.21.15.40:0/42731772 >> v2:172.21.15.53:3300/0 conn(0x7f1df806e5a0 0x7f1df806e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1dffc00640 1 -- 172.21.15.40:0/42731772 >> 172.21.15.40:0/42731772 conn(0x7f1df80b17b0 msgr2=0x7f1df80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:36.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1dffc00640 1 -- 172.21.15.40:0/42731772 shutdown_connections 2024-03-24T00:31:36.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1dffc00640 1 -- 172.21.15.40:0/42731772 wait complete. 2024-03-24T00:31:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1dffc00640 1 Processor -- start 2024-03-24T00:31:36.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.186+0000 7f1dffc00640 1 -- start start 2024-03-24T00:31:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1dffc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df806e5a0 0x7f1df8141700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1dffc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8070c80 0x7f1df80aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1dffc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1df8071690 0x7f1df81507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1dffc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df8155460 con 0x7f1df8070c80 2024-03-24T00:31:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1dffc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df81552e0 con 0x7f1df806e5a0 2024-03-24T00:31:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1dffc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df81623a0 con 0x7f1df8071690 2024-03-24T00:31:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1defe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8070c80 0x7f1df80aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1defe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8070c80 0x7f1df80aebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45866/0 (socket says 172.21.15.40:45866) 2024-03-24T00:31:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1defe00640 1 -- 172.21.15.40:0/3423989542 learned_addr learned my addr 172.21.15.40:0/3423989542 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1dfd200640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df806e5a0 0x7f1df8141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1dfdc00640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1df8071690 0x7f1df81507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1defe00640 1 -- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1df8071690 msgr2=0x7f1df81507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1defe00640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1df8071690 0x7f1df81507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1defe00640 1 -- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df806e5a0 msgr2=0x7f1df8141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1defe00640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df806e5a0 0x7f1df8141700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1defe00640 1 -- 172.21.15.40:0/3423989542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df81420b0 con 0x7f1df8070c80 2024-03-24T00:31:36.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1defe00640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8070c80 0x7f1df80aebd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1de001abd0 tx=0x7f1de00192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:36.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1df5600640 1 -- 172.21.15.40:0/3423989542 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de00b0040 con 0x7f1df8070c80 2024-03-24T00:31:36.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1df5600640 1 -- 172.21.15.40:0/3423989542 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de00abe50 con 0x7f1df8070c80 2024-03-24T00:31:36.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1df5600640 1 -- 172.21.15.40:0/3423989542 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de00b4640 con 0x7f1df8070c80 2024-03-24T00:31:36.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1dffc00640 1 -- 172.21.15.40:0/3423989542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df80adb30 con 0x7f1df8070c80 2024-03-24T00:31:36.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1dffc00640 1 -- 172.21.15.40:0/3423989542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df80ae0f0 con 0x7f1df8070c80 2024-03-24T00:31:36.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.190+0000 7f1dffc00640 1 -- 172.21.15.40:0/3423989542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df8068170 con 0x7f1df8070c80 2024-03-24T00:31:36.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.194+0000 7f1df5600640 1 -- 172.21.15.40:0/3423989542 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f1de00b47e0 con 0x7f1df8070c80 2024-03-24T00:31:36.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.194+0000 7f1df5600640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1dd4067690 0x7f1dd4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.194+0000 7f1df5600640 1 -- 172.21.15.40:0/3423989542 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1de011f320 con 0x7f1df8070c80 2024-03-24T00:31:36.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.194+0000 7f1dfd200640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1dd4067690 0x7f1dd4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.194+0000 7f1dfd200640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1dd4067690 0x7f1dd4069b50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1de800c880 tx=0x7f1de8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:36.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.198+0000 7f1df5600640 1 -- 172.21.15.40:0/3423989542 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1de0016110 con 0x7f1df8070c80 2024-03-24T00:31:36.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.366+0000 7f1dffc00640 1 -- 172.21.15.40:0/3423989542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1df80b2fa0 con 0x7f1df8070c80 2024-03-24T00:31:36.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.370+0000 7f1df5600640 1 -- 172.21.15.40:0/3423989542 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7f1de00eff90 con 0x7f1df8070c80 2024-03-24T00:31:36.374 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:36.374 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:36.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.374+0000 7f1dffc00640 1 -- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1dd4067690 msgr2=0x7f1dd4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.374+0000 7f1dffc00640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1dd4067690 0x7f1dd4069b50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1de800c880 tx=0x7f1de8008040 comp rx=0 tx=0).stop 2024-03-24T00:31:36.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.374+0000 7f1dffc00640 1 -- 172.21.15.40:0/3423989542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8070c80 msgr2=0x7f1df80aebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.374+0000 7f1dffc00640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8070c80 0x7f1df80aebd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1de001abd0 tx=0x7f1de00192c0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.374+0000 7f1dffc00640 1 -- 172.21.15.40:0/3423989542 shutdown_connections 2024-03-24T00:31:36.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.374+0000 7f1dffc00640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1dd4067690 0x7f1dd4069b50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.374+0000 7f1dffc00640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1df8071690 0x7f1df81507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.374+0000 7f1dffc00640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1df8070c80 0x7f1df80aebd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.374+0000 7f1dffc00640 1 --2- 172.21.15.40:0/3423989542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df806e5a0 0x7f1df8141700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.374+0000 7f1dffc00640 1 -- 172.21.15.40:0/3423989542 >> 172.21.15.40:0/3423989542 conn(0x7f1df80b17b0 msgr2=0x7f1df81522c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:36.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.374+0000 7f1dffc00640 1 -- 172.21.15.40:0/3423989542 shutdown_connections 2024-03-24T00:31:36.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.374+0000 7f1dffc00640 1 -- 172.21.15.40:0/3423989542 wait complete. 2024-03-24T00:31:36.395 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:36.395 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:36.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.533+0000 7fb876e00640 1 Processor -- start 2024-03-24T00:31:36.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.533+0000 7fb876e00640 1 -- start start 2024-03-24T00:31:36.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.533+0000 7fb876e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb8700d2270 0x7fb870070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb876e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb87006c950 0x7fb870161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb876e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87006d0f0 con 0x7fb87015df80 2024-03-24T00:31:36.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb876e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87006b890 con 0x7fb8700d2270 2024-03-24T00:31:36.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb876e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87006b280 con 0x7fb87006c950 2024-03-24T00:31:36.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb87015df80 0x7fb8700d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41222/0 (socket says 172.21.15.40:41222) 2024-03-24T00:31:36.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86fe00640 1 -- 172.21.15.40:0/2428045123 learned_addr learned my addr 172.21.15.40:0/2428045123 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:36.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb867400640 1 --2- 172.21.15.40:0/2428045123 >> v2:172.21.15.53:3300/0 conn(0x7fb8700d2270 0x7fb870070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb874e00640 1 --2- 172.21.15.40:0/2428045123 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb87006c950 0x7fb870161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439739275 0 0) 0x7fb87006d0f0 con 0x7fb87015df80 2024-03-24T00:31:36.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb854003530 con 0x7fb87015df80 2024-03-24T00:31:36.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 829793682 0 0) 0x7fb854003530 con 0x7fb87015df80 2024-03-24T00:31:36.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb87006c950 msgr2=0x7fb870161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:36.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 --2- 172.21.15.40:0/2428045123 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb87006c950 0x7fb870161ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 >> v2:172.21.15.53:3300/0 conn(0x7fb8700d2270 msgr2=0x7fb870070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:36.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 --2- 172.21.15.40:0/2428045123 >> v2:172.21.15.53:3300/0 conn(0x7fb8700d2270 0x7fb870070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87006ce10 con 0x7fb87015df80 2024-03-24T00:31:36.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fb860001680 con 0x7fb87015df80 2024-03-24T00:31:36.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 >> v1:172.21.15.40:6789/0 conn(0x7fb87015df80 legacy=0x7fb8700d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 --2- 172.21.15.40:0/2428045123 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb87006c950 0x7fb8540039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 --2- 172.21.15.40:0/2428045123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb854003f40 0x7fb854004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 --2- 172.21.15.40:0/2428045123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb854004d50 0x7fb8540051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87006b890 con 0x7fb854004d50 2024-03-24T00:31:36.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87006b280 con 0x7fb854003f40 2024-03-24T00:31:36.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85400b9d0 con 0x7fb87006c950 2024-03-24T00:31:36.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb874e00640 1 --2- 172.21.15.40:0/2428045123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb854004d50 0x7fb8540051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb86fe00640 1 --2- 172.21.15.40:0/2428045123 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb87006c950 0x7fb8540039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb867400640 1 --2- 172.21.15.40:0/2428045123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb854003f40 0x7fb854004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb874e00640 1 -- 172.21.15.40:0/2428045123 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb87006c950 msgr2=0x7fb8540039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:36.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb874e00640 1 --2- 172.21.15.40:0/2428045123 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb87006c950 0x7fb8540039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb874e00640 1 -- 172.21.15.40:0/2428045123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb854003f40 msgr2=0x7fb854004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:36.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb874e00640 1 --2- 172.21.15.40:0/2428045123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb854003f40 0x7fb854004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.537+0000 7fb874e00640 1 -- 172.21.15.40:0/2428045123 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb854003530 con 0x7fb854004d50 2024-03-24T00:31:36.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb874e00640 1 --2- 172.21.15.40:0/2428045123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb854004d50 0x7fb8540051d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb8700d2860 tx=0x7fb86801cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba4ab9ac16b5068 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:36.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8680b2070 con 0x7fb854004d50 2024-03-24T00:31:36.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8680a5c30 con 0x7fb854004d50 2024-03-24T00:31:36.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb86f400640 1 -- 172.21.15.40:0/2428045123 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb868009c10 con 0x7fb854004d50 2024-03-24T00:31:36.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb876e00640 1 -- 172.21.15.40:0/2428045123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb854004d50 msgr2=0x7fb8540051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb876e00640 1 --2- 172.21.15.40:0/2428045123 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb854004d50 0x7fb8540051d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb8700d2860 tx=0x7fb86801cbc0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb874e00640 1 -- 172.21.15.40:0/2428045123 reap_dead start 2024-03-24T00:31:36.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb876e00640 1 -- 172.21.15.40:0/2428045123 shutdown_connections 2024-03-24T00:31:36.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb876e00640 1 -- 172.21.15.40:0/2428045123 >> 172.21.15.40:0/2428045123 conn(0x7fb87006ecf0 msgr2=0x7fb87006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:36.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb876e00640 1 -- 172.21.15.40:0/2428045123 shutdown_connections 2024-03-24T00:31:36.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb876e00640 1 -- 172.21.15.40:0/2428045123 wait complete. 2024-03-24T00:31:36.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb876e00640 1 Processor -- start 2024-03-24T00:31:36.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.541+0000 7fb876e00640 1 -- start start 2024-03-24T00:31:36.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb876e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb854004d50 0x7fb87014ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb876e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8700ca910 0x7fb8700c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb876e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8700cad70 0x7fb8701e95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb876e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8700b12c0 con 0x7fb8700ca910 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb876e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb87015e360 con 0x7fb8700cad70 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb876e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87006bd20 con 0x7fb854004d50 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb867400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8700ca910 0x7fb8700c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb86fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb854004d50 0x7fb87014ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb867400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8700ca910 0x7fb8700c8ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45880/0 (socket says 172.21.15.40:45880) 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb867400640 1 -- 172.21.15.40:0/2007601187 learned_addr learned my addr 172.21.15.40:0/2007601187 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb874e00640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8700cad70 0x7fb8701e95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb867400640 1 -- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb854004d50 msgr2=0x7fb87014ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb867400640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb854004d50 0x7fb87014ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb867400640 1 -- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8700cad70 msgr2=0x7fb8701e95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb867400640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8700cad70 0x7fb8701e95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb867400640 1 -- 172.21.15.40:0/2007601187 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87015d400 con 0x7fb8700ca910 2024-03-24T00:31:36.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb867400640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8700ca910 0x7fb8700c8ca0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb858011a30 tx=0x7fb858011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:36.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb86cc00640 1 -- 172.21.15.40:0/2007601187 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85800fa90 con 0x7fb8700ca910 2024-03-24T00:31:36.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb86cc00640 1 -- 172.21.15.40:0/2007601187 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85800fc30 con 0x7fb8700ca910 2024-03-24T00:31:36.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb86cc00640 1 -- 172.21.15.40:0/2007601187 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb858014440 con 0x7fb8700ca910 2024-03-24T00:31:36.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb876e00640 1 -- 172.21.15.40:0/2007601187 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8701e5210 con 0x7fb8700ca910 2024-03-24T00:31:36.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb876e00640 1 -- 172.21.15.40:0/2007601187 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8701e57d0 con 0x7fb8700ca910 2024-03-24T00:31:36.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.545+0000 7fb876e00640 1 -- 172.21.15.40:0/2007601187 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8701e4fa0 con 0x7fb8700ca910 2024-03-24T00:31:36.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.549+0000 7fb86cc00640 1 -- 172.21.15.40:0/2007601187 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb858012040 con 0x7fb8700ca910 2024-03-24T00:31:36.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.549+0000 7fb86cc00640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb84c067660 0x7fb84c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.549+0000 7fb86cc00640 1 -- 172.21.15.40:0/2007601187 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb858085e70 con 0x7fb8700ca910 2024-03-24T00:31:36.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.549+0000 7fb86fe00640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb84c067660 0x7fb84c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.549+0000 7fb86fe00640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb84c067660 0x7fb84c069b20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb860001680 tx=0x7fb860007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:36.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.557+0000 7fb86cc00640 1 -- 172.21.15.40:0/2007601187 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb858019030 con 0x7fb8700ca910 2024-03-24T00:31:36.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.717+0000 7fb876e00640 1 -- 172.21.15.40:0/2007601187 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb870068170 con 0x7fb8700ca910 2024-03-24T00:31:36.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.717+0000 7fb86cc00640 1 -- 172.21.15.40:0/2007601187 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb858056b10 con 0x7fb8700ca910 2024-03-24T00:31:36.723 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.721+0000 7fb876e00640 1 -- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb84c067660 msgr2=0x7fb84c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.721+0000 7fb876e00640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb84c067660 0x7fb84c069b20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb860001680 tx=0x7fb860007900 comp rx=0 tx=0).stop 2024-03-24T00:31:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.721+0000 7fb876e00640 1 -- 172.21.15.40:0/2007601187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8700ca910 msgr2=0x7fb8700c8ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.721+0000 7fb876e00640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8700ca910 0x7fb8700c8ca0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb858011a30 tx=0x7fb858011e30 comp rx=0 tx=0).stop 2024-03-24T00:31:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.721+0000 7fb876e00640 1 -- 172.21.15.40:0/2007601187 shutdown_connections 2024-03-24T00:31:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.721+0000 7fb876e00640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb84c067660 0x7fb84c069b20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.721+0000 7fb876e00640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8700cad70 0x7fb8701e95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.721+0000 7fb876e00640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb8700ca910 0x7fb8700c8ca0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.721+0000 7fb876e00640 1 --2- 172.21.15.40:0/2007601187 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb854004d50 0x7fb87014ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.721+0000 7fb876e00640 1 -- 172.21.15.40:0/2007601187 >> 172.21.15.40:0/2007601187 conn(0x7fb87006ecf0 msgr2=0x7fb8700af420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.721+0000 7fb876e00640 1 -- 172.21.15.40:0/2007601187 shutdown_connections 2024-03-24T00:31:36.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.721+0000 7fb876e00640 1 -- 172.21.15.40:0/2007601187 wait complete. 2024-03-24T00:31:36.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:36.868 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:31:36.872 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:31:36.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e30c00640 1 Processor -- start 2024-03-24T00:31:36.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e30c00640 1 -- start start 2024-03-24T00:31:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e30c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e2c0d2270 0x7f1e2c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e30c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1e2c06c950 0x7f1e2c161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e30c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e2c06d0f0 con 0x7f1e2c15df80 2024-03-24T00:31:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e30c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e2c06b890 con 0x7f1e2c0d2270 2024-03-24T00:31:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e30c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e2c06b280 con 0x7f1e2c06c950 2024-03-24T00:31:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e21400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e2c0d2270 0x7f1e2c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e29e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1e2c15df80 0x7f1e2c0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41234/0 (socket says 172.21.15.40:41234) 2024-03-24T00:31:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e29e00640 1 -- 172.21.15.40:0/4210634658 learned_addr learned my addr 172.21.15.40:0/4210634658 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e2a800640 1 --2- 172.21.15.40:0/4210634658 >> v2:172.21.15.53:3300/0 conn(0x7f1e2c06c950 0x7f1e2c161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e21400640 1 -- 172.21.15.40:0/4210634658 >> v2:172.21.15.53:3300/0 conn(0x7f1e2c06c950 msgr2=0x7f1e2c161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:36.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e21400640 1 --2- 172.21.15.40:0/4210634658 >> v2:172.21.15.53:3300/0 conn(0x7f1e2c06c950 0x7f1e2c161ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e21400640 1 -- 172.21.15.40:0/4210634658 >> v1:172.21.15.40:6789/0 conn(0x7f1e2c15df80 legacy=0x7f1e2c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e21400640 1 -- 172.21.15.40:0/4210634658 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c06ce10 con 0x7f1e2c0d2270 2024-03-24T00:31:36.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.885+0000 7f1e2a800640 1 --2- 172.21.15.40:0/4210634658 >> v2:172.21.15.53:3300/0 conn(0x7f1e2c06c950 0x7f1e2c161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:36.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.889+0000 7f1e21400640 1 --2- 172.21.15.40:0/4210634658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c0d2270 0x7f1e2c070ff0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c0ce480 tx=0x7f1e1401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d34079650ecf715 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:36.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.889+0000 7f1e29400640 1 -- 172.21.15.40:0/4210634658 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e140b2070 con 0x7f1e2c0d2270 2024-03-24T00:31:36.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.889+0000 7f1e29400640 1 -- 172.21.15.40:0/4210634658 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e140a5980 con 0x7f1e2c0d2270 2024-03-24T00:31:36.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.889+0000 7f1e29400640 1 -- 172.21.15.40:0/4210634658 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e140a5d50 con 0x7f1e2c0d2270 2024-03-24T00:31:36.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.889+0000 7f1e30c00640 1 -- 172.21.15.40:0/4210634658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c0d2270 msgr2=0x7f1e2c070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.889+0000 7f1e30c00640 1 --2- 172.21.15.40:0/4210634658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c0d2270 0x7f1e2c070ff0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c0ce480 tx=0x7f1e1401c900 comp rx=0 tx=0).stop 2024-03-24T00:31:36.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.889+0000 7f1e30c00640 1 -- 172.21.15.40:0/4210634658 shutdown_connections 2024-03-24T00:31:36.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.889+0000 7f1e30c00640 1 --2- 172.21.15.40:0/4210634658 >> v2:172.21.15.53:3300/0 conn(0x7f1e2c06c950 0x7f1e2c161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.889+0000 7f1e30c00640 1 --2- 172.21.15.40:0/4210634658 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c0d2270 0x7f1e2c070ff0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.889+0000 7f1e30c00640 1 -- 172.21.15.40:0/4210634658 >> 172.21.15.40:0/4210634658 conn(0x7f1e2c06ecf0 msgr2=0x7f1e2c06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:36.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.889+0000 7f1e30c00640 1 -- 172.21.15.40:0/4210634658 shutdown_connections 2024-03-24T00:31:36.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.889+0000 7f1e30c00640 1 -- 172.21.15.40:0/4210634658 wait complete. 2024-03-24T00:31:36.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e30c00640 1 Processor -- start 2024-03-24T00:31:36.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e30c00640 1 -- start start 2024-03-24T00:31:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e30c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e2c06c950 0x7f1e2c0b6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e30c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c0d2270 0x7f1e2c0b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e30c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e2c15df80 0x7f1e2c0b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e30c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c16af90 con 0x7f1e2c0d2270 2024-03-24T00:31:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e30c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c16ae10 con 0x7f1e2c06c950 2024-03-24T00:31:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e30c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c16b110 con 0x7f1e2c15df80 2024-03-24T00:31:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e21400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c0d2270 0x7f1e2c0b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e21400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c0d2270 0x7f1e2c0b3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45900/0 (socket says 172.21.15.40:45900) 2024-03-24T00:31:36.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e21400640 1 -- 172.21.15.40:0/2869896451 learned_addr learned my addr 172.21.15.40:0/2869896451 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e21400640 1 -- 172.21.15.40:0/2869896451 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e2c15df80 msgr2=0x7f1e2c0b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e21400640 1 --2- 172.21.15.40:0/2869896451 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e2c15df80 0x7f1e2c0b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e21400640 1 -- 172.21.15.40:0/2869896451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e2c06c950 msgr2=0x7f1e2c0b6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e21400640 1 --2- 172.21.15.40:0/2869896451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e2c06c950 0x7f1e2c0b6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e21400640 1 -- 172.21.15.40:0/2869896451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2c0b47e0 con 0x7f1e2c0d2270 2024-03-24T00:31:36.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e21400640 1 --2- 172.21.15.40:0/2869896451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c0d2270 0x7f1e2c0b3a50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1e140b0040 tx=0x7f1e140196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e22a00640 1 -- 172.21.15.40:0/2869896451 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e140b2050 con 0x7f1e2c0d2270 2024-03-24T00:31:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e22a00640 1 -- 172.21.15.40:0/2869896451 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e140be070 con 0x7f1e2c0d2270 2024-03-24T00:31:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e22a00640 1 -- 172.21.15.40:0/2869896451 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e140adbb0 con 0x7f1e2c0d2270 2024-03-24T00:31:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e30c00640 1 -- 172.21.15.40:0/2869896451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2c0b4ac0 con 0x7f1e2c0d2270 2024-03-24T00:31:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.893+0000 7f1e30c00640 1 -- 172.21.15.40:0/2869896451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2c0b5080 con 0x7f1e2c0d2270 2024-03-24T00:31:36.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.897+0000 7f1e30c00640 1 -- 172.21.15.40:0/2869896451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2c068170 con 0x7f1e2c0d2270 2024-03-24T00:31:36.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.897+0000 7f1e22a00640 1 -- 172.21.15.40:0/2869896451 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f1e140c3460 con 0x7f1e2c0d2270 2024-03-24T00:31:36.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.897+0000 7f1e22a00640 1 --2- 172.21.15.40:0/2869896451 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1e08067660 0x7f1e08069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:36.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.897+0000 7f1e22a00640 1 -- 172.21.15.40:0/2869896451 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e14120e10 con 0x7f1e2c0d2270 2024-03-24T00:31:36.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.897+0000 7f1e29e00640 1 --2- 172.21.15.40:0/2869896451 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1e08067660 0x7f1e08069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:36.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.897+0000 7f1e29e00640 1 --2- 172.21.15.40:0/2869896451 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1e08067660 0x7f1e08069b20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c000f70 tx=0x7f1e1c007a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:36.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:36.901+0000 7f1e22a00640 1 -- 172.21.15.40:0/2869896451 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1e14124030 con 0x7f1e2c0d2270 2024-03-24T00:31:37.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.125+0000 7f1e30c00640 1 -- 172.21.15.40:0/2869896451 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1e2c0cf9d0 con 0x7f1e2c0d2270 2024-03-24T00:31:37.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.125+0000 7f1e22a00640 1 -- 172.21.15.40:0/2869896451 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1e14016520 con 0x7f1e2c0d2270 2024-03-24T00:31:37.129 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:37.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.129+0000 7f1e30c00640 1 -- 172.21.15.40:0/2869896451 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1e08067660 msgr2=0x7f1e08069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.129+0000 7f1e30c00640 1 --2- 172.21.15.40:0/2869896451 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1e08067660 0x7f1e08069b20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c000f70 tx=0x7f1e1c007a90 comp rx=0 tx=0).stop 2024-03-24T00:31:37.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.129+0000 7f1e30c00640 1 -- 172.21.15.40:0/2869896451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c0d2270 msgr2=0x7f1e2c0b3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.129+0000 7f1e30c00640 1 --2- 172.21.15.40:0/2869896451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c0d2270 0x7f1e2c0b3a50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1e140b0040 tx=0x7f1e140196a0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.129+0000 7f1e30c00640 1 -- 172.21.15.40:0/2869896451 shutdown_connections 2024-03-24T00:31:37.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.129+0000 7f1e30c00640 1 --2- 172.21.15.40:0/2869896451 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1e08067660 0x7f1e08069b20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.129+0000 7f1e30c00640 1 --2- 172.21.15.40:0/2869896451 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e2c15df80 0x7f1e2c0b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.129+0000 7f1e30c00640 1 --2- 172.21.15.40:0/2869896451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e2c0d2270 0x7f1e2c0b3a50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.129+0000 7f1e30c00640 1 --2- 172.21.15.40:0/2869896451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e2c06c950 0x7f1e2c0b6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.129+0000 7f1e30c00640 1 -- 172.21.15.40:0/2869896451 >> 172.21.15.40:0/2869896451 conn(0x7f1e2c06ecf0 msgr2=0x7f1e2c15fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:37.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.129+0000 7f1e30c00640 1 -- 172.21.15.40:0/2869896451 shutdown_connections 2024-03-24T00:31:37.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.129+0000 7f1e30c00640 1 -- 172.21.15.40:0/2869896451 wait complete. 2024-03-24T00:31:37.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:37.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc15400640 1 Processor -- start 2024-03-24T00:31:37.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc15400640 1 -- start start 2024-03-24T00:31:37.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc15400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc1006c950 0x7efc10070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:37.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc15400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1006d0f0 con 0x7efc100d2270 2024-03-24T00:31:37.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc15400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1006b280 con 0x7efc1015df80 2024-03-24T00:31:37.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc15400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1006b890 con 0x7efc1006c950 2024-03-24T00:31:37.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc07e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efc100d2270 0x7efc10161ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56300/0 (socket says 172.21.15.40:56300) 2024-03-24T00:31:37.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc07e00640 1 -- 172.21.15.40:0/80113554 learned_addr learned my addr 172.21.15.40:0/80113554 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:37.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc0f200640 1 --2- 172.21.15.40:0/80113554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc1006c950 0x7efc10070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:37.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc0de00640 1 -- 172.21.15.40:0/80113554 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204687506 0 0) 0x7efc1006d0f0 con 0x7efc100d2270 2024-03-24T00:31:37.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc0de00640 1 -- 172.21.15.40:0/80113554 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf4003530 con 0x7efc100d2270 2024-03-24T00:31:37.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc0de00640 1 -- 172.21.15.40:0/80113554 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062375842 0 0) 0x7efc1006b280 con 0x7efc1015df80 2024-03-24T00:31:37.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc0de00640 1 -- 172.21.15.40:0/80113554 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc1006d0f0 con 0x7efc1015df80 2024-03-24T00:31:37.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc0de00640 1 -- 172.21.15.40:0/80113554 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1193174508 0 0) 0x7efbf4003530 con 0x7efc100d2270 2024-03-24T00:31:37.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc0f200640 1 -- 172.21.15.40:0/80113554 >> v1:172.21.15.53:6789/0 conn(0x7efc1015df80 legacy=0x7efc100d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc0f200640 1 -- 172.21.15.40:0/80113554 >> v1:172.21.15.40:6789/0 conn(0x7efc100d2270 legacy=0x7efc10161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.301+0000 7efc0f200640 1 -- 172.21.15.40:0/80113554 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1006ce10 con 0x7efc1006c950 2024-03-24T00:31:37.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc0f200640 1 --2- 172.21.15.40:0/80113554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc1006c950 0x7efc10070ff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efc08009dc0 tx=0x7efc0801c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc3a5c16c6023882 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:37.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc0de00640 1 -- 172.21.15.40:0/80113554 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc080b2070 con 0x7efc1006c950 2024-03-24T00:31:37.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc0de00640 1 -- 172.21.15.40:0/80113554 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc080a5680 con 0x7efc1006c950 2024-03-24T00:31:37.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc0de00640 1 -- 172.21.15.40:0/80113554 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf8001680 con 0x7efc1006c950 2024-03-24T00:31:37.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc15400640 1 -- 172.21.15.40:0/80113554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc1006c950 msgr2=0x7efc10070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc15400640 1 --2- 172.21.15.40:0/80113554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc1006c950 0x7efc10070ff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efc08009dc0 tx=0x7efc0801c880 comp rx=0 tx=0).stop 2024-03-24T00:31:37.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc15400640 1 -- 172.21.15.40:0/80113554 shutdown_connections 2024-03-24T00:31:37.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc15400640 1 --2- 172.21.15.40:0/80113554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc1006c950 0x7efc10070ff0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc15400640 1 -- 172.21.15.40:0/80113554 >> 172.21.15.40:0/80113554 conn(0x7efc1006ecf0 msgr2=0x7efc1006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:37.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc15400640 1 -- 172.21.15.40:0/80113554 shutdown_connections 2024-03-24T00:31:37.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc15400640 1 -- 172.21.15.40:0/80113554 wait complete. 2024-03-24T00:31:37.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc15400640 1 Processor -- start 2024-03-24T00:31:37.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.309+0000 7efc15400640 1 -- start start 2024-03-24T00:31:37.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc15400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc1006c950 0x7efc100b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:37.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc15400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc100d2270 0x7efc10154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:37.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc15400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc1015df80 0x7efc10154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:37.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc15400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1016bfa0 con 0x7efc1015df80 2024-03-24T00:31:37.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc15400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1016be20 con 0x7efc1006c950 2024-03-24T00:31:37.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc15400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1016c120 con 0x7efc100d2270 2024-03-24T00:31:37.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc0f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc1015df80 0x7efc10154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:37.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc0f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc1015df80 0x7efc10154cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54620/0 (socket says 172.21.15.40:54620) 2024-03-24T00:31:37.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc0f200640 1 -- 172.21.15.40:0/57453505 learned_addr learned my addr 172.21.15.40:0/57453505 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:37.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc0e800640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc1006c950 0x7efc100b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:37.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc07e00640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc100d2270 0x7efc10154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:37.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc0f200640 1 -- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc100d2270 msgr2=0x7efc10154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc0f200640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc100d2270 0x7efc10154770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc0f200640 1 -- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc1006c950 msgr2=0x7efc100b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc0f200640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc1006c950 0x7efc100b62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc0f200640 1 -- 172.21.15.40:0/57453505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc100b44d0 con 0x7efc1015df80 2024-03-24T00:31:37.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc0f200640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc1015df80 0x7efc10154cb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efc0801af30 tx=0x7efc08019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:37.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc06a00640 1 -- 172.21.15.40:0/57453505 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc080b2070 con 0x7efc1015df80 2024-03-24T00:31:37.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc06a00640 1 -- 172.21.15.40:0/57453505 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc080bf070 con 0x7efc1015df80 2024-03-24T00:31:37.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc15400640 1 -- 172.21.15.40:0/57453505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc100b47b0 con 0x7efc1015df80 2024-03-24T00:31:37.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc06a00640 1 -- 172.21.15.40:0/57453505 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc080b9ad0 con 0x7efc1015df80 2024-03-24T00:31:37.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc15400640 1 -- 172.21.15.40:0/57453505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc100b4d70 con 0x7efc1015df80 2024-03-24T00:31:37.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc15400640 1 -- 172.21.15.40:0/57453505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc10068170 con 0x7efc1015df80 2024-03-24T00:31:37.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.313+0000 7efc06a00640 1 -- 172.21.15.40:0/57453505 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7efc080b9c70 con 0x7efc1015df80 2024-03-24T00:31:37.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.317+0000 7efc06a00640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efbec067660 0x7efbec069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:37.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.317+0000 7efc06a00640 1 -- 172.21.15.40:0/57453505 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc081207e0 con 0x7efc1015df80 2024-03-24T00:31:37.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.317+0000 7efc0e800640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efbec067660 0x7efbec069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:37.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.317+0000 7efc0e800640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efbec067660 0x7efbec069b20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efc00000f70 tx=0x7efc00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:37.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.321+0000 7efc06a00640 1 -- 172.21.15.40:0/57453505 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7efc080164a0 con 0x7efc1015df80 2024-03-24T00:31:37.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 -- 172.21.15.40:0/57453505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efc100cf9d0 con 0x7efc1015df80 2024-03-24T00:31:37.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc06a00640 1 -- 172.21.15.40:0/57453505 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7efc080abe20 con 0x7efc1015df80 2024-03-24T00:31:37.569 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:37.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 -- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efbec067660 msgr2=0x7efbec069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efbec067660 0x7efbec069b20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efc00000f70 tx=0x7efc00009040 comp rx=0 tx=0).stop 2024-03-24T00:31:37.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 -- 172.21.15.40:0/57453505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc1015df80 msgr2=0x7efc10154cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc1015df80 0x7efc10154cb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7efc0801af30 tx=0x7efc08019620 comp rx=0 tx=0).stop 2024-03-24T00:31:37.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 -- 172.21.15.40:0/57453505 shutdown_connections 2024-03-24T00:31:37.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efbec067660 0x7efbec069b20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc1015df80 0x7efc10154cb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc100d2270 0x7efc10154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 --2- 172.21.15.40:0/57453505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc1006c950 0x7efc100b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 -- 172.21.15.40:0/57453505 >> 172.21.15.40:0/57453505 conn(0x7efc1006ecf0 msgr2=0x7efc1015e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:37.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 -- 172.21.15.40:0/57453505 shutdown_connections 2024-03-24T00:31:37.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.565+0000 7efc15400640 1 -- 172.21.15.40:0/57453505 wait complete. 2024-03-24T00:31:37.600 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:31:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.737+0000 7f1e8a400640 1 Processor -- start 2024-03-24T00:31:37.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.737+0000 7f1e8a400640 1 -- start start 2024-03-24T00:31:37.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.737+0000 7f1e8a400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e8406f100 0x7f1e84070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:37.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.737+0000 7f1e8a400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e8406f4e0 0x7f1e84070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:37.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.737+0000 7f1e8a400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1e840aea60 0x7f1e840aee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.737+0000 7f1e8a400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e840d27b0 con 0x7f1e8406f4e0 2024-03-24T00:31:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.737+0000 7f1e8a400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e8406fa70 con 0x7f1e840aea60 2024-03-24T00:31:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.737+0000 7f1e8a400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8406fd50 con 0x7f1e8406f100 2024-03-24T00:31:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.737+0000 7f1e82a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e8406f4e0 0x7f1e84070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e82a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e8406f4e0 0x7f1e84070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54626/0 (socket says 172.21.15.40:54626) 2024-03-24T00:31:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e82a00640 1 -- 172.21.15.40:0/1808393463 learned_addr learned my addr 172.21.15.40:0/1808393463 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e83400640 1 --2- 172.21.15.40:0/1808393463 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e8406f100 0x7f1e84070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:37.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e83e00640 1 --2- 172.21.15.40:0/1808393463 >> v2:172.21.15.53:3300/0 conn(0x7f1e840aea60 0x7f1e840aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e82a00640 1 -- 172.21.15.40:0/1808393463 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e8406f100 msgr2=0x7f1e84070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e82a00640 1 --2- 172.21.15.40:0/1808393463 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e8406f100 0x7f1e84070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e82a00640 1 -- 172.21.15.40:0/1808393463 >> v2:172.21.15.53:3300/0 conn(0x7f1e840aea60 msgr2=0x7f1e840aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e82a00640 1 --2- 172.21.15.40:0/1808393463 >> v2:172.21.15.53:3300/0 conn(0x7f1e840aea60 0x7f1e840aee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e82a00640 1 -- 172.21.15.40:0/1808393463 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e840d03f0 con 0x7f1e8406f4e0 2024-03-24T00:31:37.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e83400640 1 --2- 172.21.15.40:0/1808393463 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e8406f100 0x7f1e84070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e82a00640 1 --2- 172.21.15.40:0/1808393463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e8406f4e0 0x7f1e84070030 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1e840cfcd0 tx=0x7f1e7001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f702b3c7c8cb1f52 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e82000640 1 -- 172.21.15.40:0/1808393463 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e700b0070 con 0x7f1e8406f4e0 2024-03-24T00:31:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e82000640 1 -- 172.21.15.40:0/1808393463 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e700ab600 con 0x7f1e8406f4e0 2024-03-24T00:31:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.741+0000 7f1e82000640 1 -- 172.21.15.40:0/1808393463 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e700ab9d0 con 0x7f1e8406f4e0 2024-03-24T00:31:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 -- 172.21.15.40:0/1808393463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e8406f4e0 msgr2=0x7f1e84070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 --2- 172.21.15.40:0/1808393463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e8406f4e0 0x7f1e84070030 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1e840cfcd0 tx=0x7f1e7001c520 comp rx=0 tx=0).stop 2024-03-24T00:31:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 -- 172.21.15.40:0/1808393463 shutdown_connections 2024-03-24T00:31:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 --2- 172.21.15.40:0/1808393463 >> v2:172.21.15.53:3300/0 conn(0x7f1e840aea60 0x7f1e840aee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 --2- 172.21.15.40:0/1808393463 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e8406f4e0 0x7f1e84070030 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 --2- 172.21.15.40:0/1808393463 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e8406f100 0x7f1e84070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 -- 172.21.15.40:0/1808393463 >> 172.21.15.40:0/1808393463 conn(0x7f1e840d1c90 msgr2=0x7f1e840d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 -- 172.21.15.40:0/1808393463 shutdown_connections 2024-03-24T00:31:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 -- 172.21.15.40:0/1808393463 wait complete. 2024-03-24T00:31:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 Processor -- start 2024-03-24T00:31:37.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 -- start start 2024-03-24T00:31:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e8406f100 0x7f1e84156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e8406f4e0 0x7f1e840c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e840aea60 0x7f1e840cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e84160ae0 con 0x7f1e8406f4e0 2024-03-24T00:31:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e84160960 con 0x7f1e840aea60 2024-03-24T00:31:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e8a400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e84160c60 con 0x7f1e8406f100 2024-03-24T00:31:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e82a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e8406f4e0 0x7f1e840c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e82a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e8406f4e0 0x7f1e840c8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54636/0 (socket says 172.21.15.40:54636) 2024-03-24T00:31:37.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e83400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e8406f100 0x7f1e84156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e82a00640 1 -- 172.21.15.40:0/3622874469 learned_addr learned my addr 172.21.15.40:0/3622874469 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e83e00640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e840aea60 0x7f1e840cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e82a00640 1 -- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e8406f100 msgr2=0x7f1e84156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e82a00640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e8406f100 0x7f1e84156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e82a00640 1 -- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e840aea60 msgr2=0x7f1e840cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e82a00640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e840aea60 0x7f1e840cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e82a00640 1 -- 172.21.15.40:0/3622874469 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e840c8390 con 0x7f1e8406f4e0 2024-03-24T00:31:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.745+0000 7f1e83e00640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e840aea60 0x7f1e840cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.749+0000 7f1e82a00640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e8406f4e0 0x7f1e840c8730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1e7001abd0 tx=0x7f1e700192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.749+0000 7f1e7b400640 1 -- 172.21.15.40:0/3622874469 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e700b9070 con 0x7f1e8406f4e0 2024-03-24T00:31:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.749+0000 7f1e7b400640 1 -- 172.21.15.40:0/3622874469 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e700b4400 con 0x7f1e8406f4e0 2024-03-24T00:31:37.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.749+0000 7f1e7b400640 1 -- 172.21.15.40:0/3622874469 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e700b0040 con 0x7f1e8406f4e0 2024-03-24T00:31:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.749+0000 7f1e8a400640 1 -- 172.21.15.40:0/3622874469 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e841572e0 con 0x7f1e8406f4e0 2024-03-24T00:31:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.749+0000 7f1e8a400640 1 -- 172.21.15.40:0/3622874469 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e841578a0 con 0x7f1e8406f4e0 2024-03-24T00:31:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.749+0000 7f1e8a400640 1 -- 172.21.15.40:0/3622874469 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e84068170 con 0x7f1e8406f4e0 2024-03-24T00:31:37.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.749+0000 7f1e7b400640 1 -- 172.21.15.40:0/3622874469 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f1e700b33a0 con 0x7f1e8406f4e0 2024-03-24T00:31:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.749+0000 7f1e7b400640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1e64067690 0x7f1e64069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.749+0000 7f1e7b400640 1 -- 172.21.15.40:0/3622874469 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e7011eac0 con 0x7f1e8406f4e0 2024-03-24T00:31:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.749+0000 7f1e83400640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1e64067690 0x7f1e64069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:37.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.753+0000 7f1e83400640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1e64067690 0x7f1e64069b50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1e740098d0 tx=0x7f1e74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:37.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.761+0000 7f1e7b400640 1 -- 172.21.15.40:0/3622874469 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1e70016110 con 0x7f1e8406f4e0 2024-03-24T00:31:37.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.933+0000 7f1e8a400640 1 -- 172.21.15.40:0/3622874469 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1e8406ddc0 con 0x7f1e8406f4e0 2024-03-24T00:31:37.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.933+0000 7f1e7b400640 1 -- 172.21.15.40:0/3622874469 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1e700ef0e0 con 0x7f1e8406f4e0 2024-03-24T00:31:37.939 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:31:37.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.937+0000 7f1e8a400640 1 -- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1e64067690 msgr2=0x7f1e64069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.937+0000 7f1e8a400640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1e64067690 0x7f1e64069b50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1e740098d0 tx=0x7f1e74008040 comp rx=0 tx=0).stop 2024-03-24T00:31:37.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.937+0000 7f1e8a400640 1 -- 172.21.15.40:0/3622874469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e8406f4e0 msgr2=0x7f1e840c8730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:37.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.937+0000 7f1e8a400640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e8406f4e0 0x7f1e840c8730 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1e7001abd0 tx=0x7f1e700192c0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.937+0000 7f1e8a400640 1 -- 172.21.15.40:0/3622874469 shutdown_connections 2024-03-24T00:31:37.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.937+0000 7f1e8a400640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1e64067690 0x7f1e64069b50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.937+0000 7f1e8a400640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e840aea60 0x7f1e840cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.937+0000 7f1e8a400640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e8406f4e0 0x7f1e840c8730 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.937+0000 7f1e8a400640 1 --2- 172.21.15.40:0/3622874469 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e8406f100 0x7f1e84156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:37.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.937+0000 7f1e8a400640 1 -- 172.21.15.40:0/3622874469 >> 172.21.15.40:0/3622874469 conn(0x7f1e840d1c90 msgr2=0x7f1e8406c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:37.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.937+0000 7f1e8a400640 1 -- 172.21.15.40:0/3622874469 shutdown_connections 2024-03-24T00:31:37.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:37.937+0000 7f1e8a400640 1 -- 172.21.15.40:0/3622874469 wait complete. 2024-03-24T00:31:37.961 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:31:37.961 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:31:37.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:38.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.145+0000 7f36a8e00640 1 Processor -- start 2024-03-24T00:31:38.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.145+0000 7f36a8e00640 1 -- start start 2024-03-24T00:31:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a8e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f36a40cdd20 0x7f36a4072520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a8e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a406f030 con 0x7f36a415e040 2024-03-24T00:31:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a8e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a4071480 con 0x7f36a406d900 2024-03-24T00:31:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a8e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a4071230 con 0x7f36a40cdd20 2024-03-24T00:31:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a3400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f36a415e040 0x7f36a406d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56316/0 (socket says 172.21.15.40:56316) 2024-03-24T00:31:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a3400640 1 -- 172.21.15.40:0/3570848907 learned_addr learned my addr 172.21.15.40:0/3570848907 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:38.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a3e00640 1 --2- 172.21.15.40:0/3570848907 >> v2:172.21.15.40:3300/0 conn(0x7f36a40cdd20 0x7f36a4072520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a2a00640 1 -- 172.21.15.40:0/3570848907 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939243354 0 0) 0x7f3694001120 con 0x7f36a415e040 2024-03-24T00:31:38.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a2a00640 1 -- 172.21.15.40:0/3570848907 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3688003530 con 0x7f36a415e040 2024-03-24T00:31:38.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a2a00640 1 -- 172.21.15.40:0/3570848907 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145094031 0 0) 0x7f36a4071480 con 0x7f36a406d900 2024-03-24T00:31:38.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a2a00640 1 -- 172.21.15.40:0/3570848907 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3694001120 con 0x7f36a406d900 2024-03-24T00:31:38.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a3e00640 1 -- 172.21.15.40:0/3570848907 >> v1:172.21.15.53:6789/0 conn(0x7f36a406d900 legacy=0x7f36a40ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a3e00640 1 -- 172.21.15.40:0/3570848907 >> v1:172.21.15.40:6789/0 conn(0x7f36a415e040 legacy=0x7f36a406d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a3e00640 1 -- 172.21.15.40:0/3570848907 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a4071760 con 0x7f36a40cdd20 2024-03-24T00:31:38.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.149+0000 7f36a3e00640 1 --2- 172.21.15.40:0/3570848907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a40cdd20 0x7f36a4072520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f369c009d90 tx=0x7f369c01f120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c2f5ae7c155dea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:38.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a2a00640 1 -- 172.21.15.40:0/3570848907 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c0b4070 con 0x7f36a40cdd20 2024-03-24T00:31:38.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a2a00640 1 -- 172.21.15.40:0/3570848907 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369c0af600 con 0x7f36a40cdd20 2024-03-24T00:31:38.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a2a00640 1 -- 172.21.15.40:0/3570848907 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c0af9d0 con 0x7f36a40cdd20 2024-03-24T00:31:38.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 -- 172.21.15.40:0/3570848907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a40cdd20 msgr2=0x7f36a4072520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 --2- 172.21.15.40:0/3570848907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a40cdd20 0x7f36a4072520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f369c009d90 tx=0x7f369c01f120 comp rx=0 tx=0).stop 2024-03-24T00:31:38.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 -- 172.21.15.40:0/3570848907 shutdown_connections 2024-03-24T00:31:38.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 --2- 172.21.15.40:0/3570848907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a40cdd20 0x7f36a4072520 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 -- 172.21.15.40:0/3570848907 >> 172.21.15.40:0/3570848907 conn(0x7f36a40b1c80 msgr2=0x7f36a40b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:38.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 -- 172.21.15.40:0/3570848907 shutdown_connections 2024-03-24T00:31:38.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 -- 172.21.15.40:0/3570848907 wait complete. 2024-03-24T00:31:38.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 Processor -- start 2024-03-24T00:31:38.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 -- start start 2024-03-24T00:31:38.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a406d900 0x7f36a414c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a40cdd20 0x7f36a40b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a415e040 0x7f36a40b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a4072000 con 0x7f36a406d900 2024-03-24T00:31:38.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a4071d20 con 0x7f36a40cdd20 2024-03-24T00:31:38.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.153+0000 7f36a8e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a40cd390 con 0x7f36a415e040 2024-03-24T00:31:38.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a406d900 0x7f36a414c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a406d900 0x7f36a414c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54648/0 (socket says 172.21.15.40:54648) 2024-03-24T00:31:38.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a3400640 1 -- 172.21.15.40:0/3327391504 learned_addr learned my addr 172.21.15.40:0/3327391504 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:38.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a3e00640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a415e040 0x7f36a40b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f369be00640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a40cdd20 0x7f36a40b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a3400640 1 -- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a415e040 msgr2=0x7f36a40b0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a3400640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a415e040 0x7f36a40b0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a3400640 1 -- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a40cdd20 msgr2=0x7f36a40b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a3400640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a40cdd20 0x7f36a40b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a3400640 1 -- 172.21.15.40:0/3327391504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a40c3d50 con 0x7f36a406d900 2024-03-24T00:31:38.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a3400640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a406d900 0x7f36a414c610 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f369400c6c0 tx=0x7f369400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:38.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f369b400640 1 -- 172.21.15.40:0/3327391504 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3694017070 con 0x7f36a406d900 2024-03-24T00:31:38.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f369b400640 1 -- 172.21.15.40:0/3327391504 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3694005ac0 con 0x7f36a406d900 2024-03-24T00:31:38.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f369b400640 1 -- 172.21.15.40:0/3327391504 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3694007070 con 0x7f36a406d900 2024-03-24T00:31:38.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a8e00640 1 -- 172.21.15.40:0/3327391504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a40c4030 con 0x7f36a406d900 2024-03-24T00:31:38.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a8e00640 1 -- 172.21.15.40:0/3327391504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a40c45f0 con 0x7f36a406d900 2024-03-24T00:31:38.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f36a8e00640 1 -- 172.21.15.40:0/3327391504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a409e2e0 con 0x7f36a406d900 2024-03-24T00:31:38.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.157+0000 7f369b400640 1 -- 172.21.15.40:0/3327391504 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f369401a070 con 0x7f36a406d900 2024-03-24T00:31:38.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.161+0000 7f369b400640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3680067690 0x7f3680069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.161+0000 7f369b400640 1 -- 172.21.15.40:0/3327391504 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3694084ca0 con 0x7f36a406d900 2024-03-24T00:31:38.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.161+0000 7f369be00640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3680067690 0x7f3680069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.161+0000 7f369be00640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3680067690 0x7f3680069b50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f36a40b22d0 tx=0x7f368c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:38.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.165+0000 7f369b400640 1 -- 172.21.15.40:0/3327391504 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f36a406f030 con 0x7f36a406d900 2024-03-24T00:31:38.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.329+0000 7f36a8e00640 1 -- 172.21.15.40:0/3327391504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36a415ad00 con 0x7f36a406d900 2024-03-24T00:31:38.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.329+0000 7f369b400640 1 -- 172.21.15.40:0/3327391504 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7f3694055360 con 0x7f36a406d900 2024-03-24T00:31:38.334 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:38.335 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.333+0000 7f36a8e00640 1 -- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3680067690 msgr2=0x7f3680069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.333+0000 7f36a8e00640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3680067690 0x7f3680069b50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f36a40b22d0 tx=0x7f368c009040 comp rx=0 tx=0).stop 2024-03-24T00:31:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.333+0000 7f36a8e00640 1 -- 172.21.15.40:0/3327391504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a406d900 msgr2=0x7f36a414c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.333+0000 7f36a8e00640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a406d900 0x7f36a414c610 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f369400c6c0 tx=0x7f369400cb80 comp rx=0 tx=0).stop 2024-03-24T00:31:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.333+0000 7f36a8e00640 1 -- 172.21.15.40:0/3327391504 shutdown_connections 2024-03-24T00:31:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.333+0000 7f36a8e00640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3680067690 0x7f3680069b50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.333+0000 7f36a8e00640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a415e040 0x7f36a40b0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.333+0000 7f36a8e00640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a40cdd20 0x7f36a40b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.333+0000 7f36a8e00640 1 --2- 172.21.15.40:0/3327391504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a406d900 0x7f36a414c610 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.333+0000 7f36a8e00640 1 -- 172.21.15.40:0/3327391504 >> 172.21.15.40:0/3327391504 conn(0x7f36a40b1c80 msgr2=0x7f36a406d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.333+0000 7f36a8e00640 1 -- 172.21.15.40:0/3327391504 shutdown_connections 2024-03-24T00:31:38.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.333+0000 7f36a8e00640 1 -- 172.21.15.40:0/3327391504 wait complete. 2024-03-24T00:31:38.355 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:38.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:38.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.493+0000 7f96f7c00640 1 Processor -- start 2024-03-24T00:31:38.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.493+0000 7f96f7c00640 1 -- start start 2024-03-24T00:31:38.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f7c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f96f015bd40 0x7f96f00d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f7c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f015f970 con 0x7f96f015b960 2024-03-24T00:31:38.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f7c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96f015fc50 con 0x7f96f00d1520 2024-03-24T00:31:38.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f7c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96f00d1990 con 0x7f96f015bd40 2024-03-24T00:31:38.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f5200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f96f015b960 0x7f96f0070d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56324/0 (socket says 172.21.15.40:56324) 2024-03-24T00:31:38.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f5200640 1 -- 172.21.15.40:0/3453717385 learned_addr learned my addr 172.21.15.40:0/3453717385 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:38.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115999136 0 0) 0x7f96f015f970 con 0x7f96f015b960 2024-03-24T00:31:38.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d4003530 con 0x7f96f015b960 2024-03-24T00:31:38.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96e7e00640 1 --2- 172.21.15.40:0/3453717385 >> v2:172.21.15.53:3300/0 conn(0x7f96f015bd40 0x7f96f00d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3136536372 0 0) 0x7f96d4003530 con 0x7f96f015b960 2024-03-24T00:31:38.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 >> v2:172.21.15.53:3300/0 conn(0x7f96f015bd40 msgr2=0x7f96f00d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:38.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 --2- 172.21.15.40:0/3453717385 >> v2:172.21.15.53:3300/0 conn(0x7f96f015bd40 0x7f96f00d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 >> v1:172.21.15.53:6789/0 conn(0x7f96f00d1520 legacy=0x7f96f00cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f015ff30 con 0x7f96f015b960 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f96e0001680 con 0x7f96f015b960 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 >> v1:172.21.15.40:6789/0 conn(0x7f96f015b960 legacy=0x7f96f0070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 --2- 172.21.15.40:0/3453717385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96d40039c0 0x7f96d4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 --2- 172.21.15.40:0/3453717385 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96d4004810 0x7f96d4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 --2- 172.21.15.40:0/3453717385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96d40051d0 0x7f96d400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f00d1990 con 0x7f96d40039c0 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d400c070 con 0x7f96d40051d0 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d400c270 con 0x7f96d4004810 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f5200640 1 --2- 172.21.15.40:0/3453717385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96d40039c0 0x7f96d4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96e7e00640 1 --2- 172.21.15.40:0/3453717385 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96d4004810 0x7f96d4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f5c00640 1 --2- 172.21.15.40:0/3453717385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96d40051d0 0x7f96d400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f5200640 1 -- 172.21.15.40:0/3453717385 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96d4004810 msgr2=0x7f96d4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f5200640 1 --2- 172.21.15.40:0/3453717385 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96d4004810 0x7f96d4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f5200640 1 -- 172.21.15.40:0/3453717385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96d40051d0 msgr2=0x7f96d400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:38.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f5200640 1 --2- 172.21.15.40:0/3453717385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96d40051d0 0x7f96d400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f5200640 1 -- 172.21.15.40:0/3453717385 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d4003530 con 0x7f96d40039c0 2024-03-24T00:31:38.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.497+0000 7f96f5c00640 1 -- 172.21.15.40:0/3453717385 reap_dead start 2024-03-24T00:31:38.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.501+0000 7f96f5200640 1 --2- 172.21.15.40:0/3453717385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96d40039c0 0x7f96d4003de0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f96e000c1c0 tx=0x7f96e001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2974755893b6e2e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:38.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.501+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0007d00 con 0x7f96d40039c0 2024-03-24T00:31:38.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.501+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96e00a9020 con 0x7f96d40039c0 2024-03-24T00:31:38.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.501+0000 7f96efe00640 1 -- 172.21.15.40:0/3453717385 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96e0005ae0 con 0x7f96d40039c0 2024-03-24T00:31:38.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.501+0000 7f96f7c00640 1 -- 172.21.15.40:0/3453717385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96d40039c0 msgr2=0x7f96d4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.501+0000 7f96f7c00640 1 --2- 172.21.15.40:0/3453717385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96d40039c0 0x7f96d4003de0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f96e000c1c0 tx=0x7f96e001cc60 comp rx=0 tx=0).stop 2024-03-24T00:31:38.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.501+0000 7f96f7c00640 1 -- 172.21.15.40:0/3453717385 shutdown_connections 2024-03-24T00:31:38.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.501+0000 7f96f7c00640 1 --2- 172.21.15.40:0/3453717385 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96d40039c0 0x7f96d4003de0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.501+0000 7f96f7c00640 1 -- 172.21.15.40:0/3453717385 >> 172.21.15.40:0/3453717385 conn(0x7f96f006ecf0 msgr2=0x7f96f006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:38.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.501+0000 7f96f7c00640 1 -- 172.21.15.40:0/3453717385 shutdown_connections 2024-03-24T00:31:38.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.501+0000 7f96f7c00640 1 -- 172.21.15.40:0/3453717385 wait complete. 2024-03-24T00:31:38.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96f7c00640 1 Processor -- start 2024-03-24T00:31:38.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96f7c00640 1 -- start start 2024-03-24T00:31:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96f7c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96d40039c0 0x7f96f00c7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96f7c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96f00c54c0 0x7f96f014f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96f7c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96f01fac60 0x7f96f01fb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96f7c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f00cfb30 con 0x7f96f00c54c0 2024-03-24T00:31:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96f7c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96f00cf9b0 con 0x7f96d40039c0 2024-03-24T00:31:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96f7c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96f00cfcb0 con 0x7f96f01fac60 2024-03-24T00:31:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96f00c54c0 0x7f96f014f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96f00c54c0 0x7f96f014f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54666/0 (socket says 172.21.15.40:54666) 2024-03-24T00:31:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96e7e00640 1 -- 172.21.15.40:0/947236424 learned_addr learned my addr 172.21.15.40:0/947236424 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96f5200640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96d40039c0 0x7f96f00c7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96f5c00640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96f01fac60 0x7f96f01fb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96e7e00640 1 -- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96f01fac60 msgr2=0x7f96f01fb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96e7e00640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96f01fac60 0x7f96f01fb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96e7e00640 1 -- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96d40039c0 msgr2=0x7f96f00c7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96e7e00640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96d40039c0 0x7f96f00c7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96e7e00640 1 -- 172.21.15.40:0/947236424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f00c6300 con 0x7f96f00c54c0 2024-03-24T00:31:38.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96e7e00640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96f00c54c0 0x7f96f014f030 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f96d8011ad0 tx=0x7f96d8011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:38.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96ed600640 1 -- 172.21.15.40:0/947236424 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d800fbb0 con 0x7f96f00c54c0 2024-03-24T00:31:38.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96ed600640 1 -- 172.21.15.40:0/947236424 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96d800fd50 con 0x7f96f00c54c0 2024-03-24T00:31:38.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96ed600640 1 -- 172.21.15.40:0/947236424 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d8014680 con 0x7f96f00c54c0 2024-03-24T00:31:38.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96f7c00640 1 -- 172.21.15.40:0/947236424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96f00c5a60 con 0x7f96f00c54c0 2024-03-24T00:31:38.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.505+0000 7f96f7c00640 1 -- 172.21.15.40:0/947236424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96f00c6020 con 0x7f96f00c54c0 2024-03-24T00:31:38.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.509+0000 7f96f7c00640 1 -- 172.21.15.40:0/947236424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96f0068170 con 0x7f96f00c54c0 2024-03-24T00:31:38.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.509+0000 7f96ed600640 1 -- 172.21.15.40:0/947236424 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f96d8012040 con 0x7f96f00c54c0 2024-03-24T00:31:38.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.509+0000 7f96ed600640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f96cc067690 0x7f96cc069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.509+0000 7f96ed600640 1 -- 172.21.15.40:0/947236424 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96d8085d30 con 0x7f96f00c54c0 2024-03-24T00:31:38.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.509+0000 7f96f5200640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f96cc067690 0x7f96cc069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.509+0000 7f96f5200640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f96cc067690 0x7f96cc069b50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f96e00aa000 tx=0x7f96f015fb40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:38.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.513+0000 7f96ed600640 1 -- 172.21.15.40:0/947236424 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f96d8019030 con 0x7f96f00c54c0 2024-03-24T00:31:38.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.677+0000 7f96f7c00640 1 -- 172.21.15.40:0/947236424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f96f0070910 con 0x7f96f00c54c0 2024-03-24T00:31:38.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.677+0000 7f96ed600640 1 -- 172.21.15.40:0/947236424 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f96d80563f0 con 0x7f96f00c54c0 2024-03-24T00:31:38.680 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:38.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.677+0000 7f96f7c00640 1 -- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f96cc067690 msgr2=0x7f96cc069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.677+0000 7f96f7c00640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f96cc067690 0x7f96cc069b50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f96e00aa000 tx=0x7f96f015fb40 comp rx=0 tx=0).stop 2024-03-24T00:31:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.677+0000 7f96f7c00640 1 -- 172.21.15.40:0/947236424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96f00c54c0 msgr2=0x7f96f014f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.677+0000 7f96f7c00640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96f00c54c0 0x7f96f014f030 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f96d8011ad0 tx=0x7f96d8011f90 comp rx=0 tx=0).stop 2024-03-24T00:31:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.677+0000 7f96f7c00640 1 -- 172.21.15.40:0/947236424 shutdown_connections 2024-03-24T00:31:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.677+0000 7f96f7c00640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f96cc067690 0x7f96cc069b50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.677+0000 7f96f7c00640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96f01fac60 0x7f96f01fb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.677+0000 7f96f7c00640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96f00c54c0 0x7f96f014f030 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.677+0000 7f96f7c00640 1 --2- 172.21.15.40:0/947236424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96d40039c0 0x7f96f00c7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.677+0000 7f96f7c00640 1 -- 172.21.15.40:0/947236424 >> 172.21.15.40:0/947236424 conn(0x7f96f006ecf0 msgr2=0x7f96f014f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.681+0000 7f96f7c00640 1 -- 172.21.15.40:0/947236424 shutdown_connections 2024-03-24T00:31:38.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.681+0000 7f96f7c00640 1 -- 172.21.15.40:0/947236424 wait complete. 2024-03-24T00:31:38.703 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:38.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.841+0000 7f2ea8800640 1 Processor -- start 2024-03-24T00:31:38.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.841+0000 7f2ea8800640 1 -- start start 2024-03-24T00:31:38.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.841+0000 7f2ea8800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2ea006c950 0x7f2ea0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.841+0000 7f2ea8800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea006d0f0 con 0x7f2ea00d2270 2024-03-24T00:31:38.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.845+0000 7f2ea8800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea006b280 con 0x7f2ea015df80 2024-03-24T00:31:38.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.845+0000 7f2ea8800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ea006b890 con 0x7f2ea006c950 2024-03-24T00:31:38.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.845+0000 7f2ea5400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2ea00d2270 0x7f2ea0161ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56330/0 (socket says 172.21.15.40:56330) 2024-03-24T00:31:38.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.845+0000 7f2ea5400640 1 -- 172.21.15.40:0/2080372599 learned_addr learned my addr 172.21.15.40:0/2080372599 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:38.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.845+0000 7f2ea6800640 1 --2- 172.21.15.40:0/2080372599 >> v2:172.21.15.53:3300/0 conn(0x7f2ea006c950 0x7f2ea0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.845+0000 7f2ea4a00640 1 -- 172.21.15.40:0/2080372599 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853104980 0 0) 0x7f2ea006d0f0 con 0x7f2ea00d2270 2024-03-24T00:31:38.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.845+0000 7f2ea4a00640 1 -- 172.21.15.40:0/2080372599 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e88003530 con 0x7f2ea00d2270 2024-03-24T00:31:38.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.845+0000 7f2ea6800640 1 -- 172.21.15.40:0/2080372599 >> v1:172.21.15.53:6789/0 conn(0x7f2ea015df80 legacy=0x7f2ea00d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.845+0000 7f2ea6800640 1 -- 172.21.15.40:0/2080372599 >> v1:172.21.15.40:6789/0 conn(0x7f2ea00d2270 legacy=0x7f2ea0161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.845+0000 7f2ea6800640 1 -- 172.21.15.40:0/2080372599 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea006ce10 con 0x7f2ea006c950 2024-03-24T00:31:38.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.845+0000 7f2ea6800640 1 --2- 172.21.15.40:0/2080372599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ea006c950 0x7f2ea0070ff0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2ea006ba10 tx=0x7f2e9c01c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=463564519dc602ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:38.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea4a00640 1 -- 172.21.15.40:0/2080372599 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c0b2070 con 0x7f2ea006c950 2024-03-24T00:31:38.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea4a00640 1 -- 172.21.15.40:0/2080372599 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9c0a59a0 con 0x7f2ea006c950 2024-03-24T00:31:38.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea4a00640 1 -- 172.21.15.40:0/2080372599 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c0a5d70 con 0x7f2ea006c950 2024-03-24T00:31:38.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 -- 172.21.15.40:0/2080372599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ea006c950 msgr2=0x7f2ea0070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 --2- 172.21.15.40:0/2080372599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ea006c950 0x7f2ea0070ff0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2ea006ba10 tx=0x7f2e9c01c920 comp rx=0 tx=0).stop 2024-03-24T00:31:38.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 -- 172.21.15.40:0/2080372599 shutdown_connections 2024-03-24T00:31:38.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 --2- 172.21.15.40:0/2080372599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ea006c950 0x7f2ea0070ff0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 -- 172.21.15.40:0/2080372599 >> 172.21.15.40:0/2080372599 conn(0x7f2ea006ecf0 msgr2=0x7f2ea006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:38.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 -- 172.21.15.40:0/2080372599 shutdown_connections 2024-03-24T00:31:38.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 -- 172.21.15.40:0/2080372599 wait complete. 2024-03-24T00:31:38.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 Processor -- start 2024-03-24T00:31:38.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 -- start start 2024-03-24T00:31:38.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea006c950 0x7f2ea015c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ea00d2270 0x7f2ea0160b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ea015df80 0x7f2ea0161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea016bfa0 con 0x7f2ea006c950 2024-03-24T00:31:38.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea016be20 con 0x7f2ea00d2270 2024-03-24T00:31:38.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea8800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea016c120 con 0x7f2ea015df80 2024-03-24T00:31:38.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea006c950 0x7f2ea015c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea006c950 0x7f2ea015c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54672/0 (socket says 172.21.15.40:54672) 2024-03-24T00:31:38.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea5e00640 1 -- 172.21.15.40:0/1377514386 learned_addr learned my addr 172.21.15.40:0/1377514386 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:38.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea6800640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ea015df80 0x7f2ea0161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.849+0000 7f2ea5400640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ea00d2270 0x7f2ea0160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2ea5e00640 1 -- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ea015df80 msgr2=0x7f2ea0161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2ea5e00640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ea015df80 0x7f2ea0161040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2ea5e00640 1 -- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ea00d2270 msgr2=0x7f2ea0160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:38.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2ea5e00640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ea00d2270 0x7f2ea0160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:38.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2ea5e00640 1 -- 172.21.15.40:0/1377514386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea015b560 con 0x7f2ea006c950 2024-03-24T00:31:38.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2ea5e00640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea006c950 0x7f2ea015c740 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400c3a0 tx=0x7f2e9400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:38.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2e9a000640 1 -- 172.21.15.40:0/1377514386 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e940052d0 con 0x7f2ea006c950 2024-03-24T00:31:38.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2e9a000640 1 -- 172.21.15.40:0/1377514386 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e94005470 con 0x7f2ea006c950 2024-03-24T00:31:38.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2e9a000640 1 -- 172.21.15.40:0/1377514386 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e94007260 con 0x7f2ea006c950 2024-03-24T00:31:38.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2ea8800640 1 -- 172.21.15.40:0/1377514386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea015a4c0 con 0x7f2ea006c950 2024-03-24T00:31:38.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2ea8800640 1 -- 172.21.15.40:0/1377514386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea015aa80 con 0x7f2ea006c950 2024-03-24T00:31:38.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2ea8800640 1 -- 172.21.15.40:0/1377514386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea0068170 con 0x7f2ea006c950 2024-03-24T00:31:38.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2e9a000640 1 -- 172.21.15.40:0/1377514386 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2e94007400 con 0x7f2ea006c950 2024-03-24T00:31:38.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.853+0000 7f2e9a000640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2e84067660 0x7f2e84069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:38.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.857+0000 7f2e9a000640 1 -- 172.21.15.40:0/1377514386 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e94084da0 con 0x7f2ea006c950 2024-03-24T00:31:38.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.857+0000 7f2ea5400640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2e84067660 0x7f2e84069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:38.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.857+0000 7f2ea5400640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2e84067660 0x7f2e84069b20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00b0500 tx=0x7f2e90009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:38.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:38.857+0000 7f2e9a000640 1 -- 172.21.15.40:0/1377514386 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2e94055980 con 0x7f2ea006c950 2024-03-24T00:31:39.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 -- 172.21.15.40:0/1377514386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2ea00cf9d0 con 0x7f2ea006c950 2024-03-24T00:31:39.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2e9a000640 1 -- 172.21.15.40:0/1377514386 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2e94055280 con 0x7f2ea006c950 2024-03-24T00:31:39.025 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:39.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 -- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2e84067660 msgr2=0x7f2e84069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2e84067660 0x7f2e84069b20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00b0500 tx=0x7f2e90009040 comp rx=0 tx=0).stop 2024-03-24T00:31:39.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 -- 172.21.15.40:0/1377514386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea006c950 msgr2=0x7f2ea015c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea006c950 0x7f2ea015c740 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400c3a0 tx=0x7f2e9400c860 comp rx=0 tx=0).stop 2024-03-24T00:31:39.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 -- 172.21.15.40:0/1377514386 shutdown_connections 2024-03-24T00:31:39.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2e84067660 0x7f2e84069b20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2ea015df80 0x7f2ea0161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ea00d2270 0x7f2ea0160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 --2- 172.21.15.40:0/1377514386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2ea006c950 0x7f2ea015c740 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 -- 172.21.15.40:0/1377514386 >> 172.21.15.40:0/1377514386 conn(0x7f2ea006ecf0 msgr2=0x7f2ea015fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:39.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 -- 172.21.15.40:0/1377514386 shutdown_connections 2024-03-24T00:31:39.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.021+0000 7f2ea8800640 1 -- 172.21.15.40:0/1377514386 wait complete. 2024-03-24T00:31:39.046 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:39.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.237+0000 7f141d200640 1 Processor -- start 2024-03-24T00:31:39.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.237+0000 7f141d200640 1 -- start start 2024-03-24T00:31:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.237+0000 7f141d200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14180b4330 0x7f14180ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.237+0000 7f141d200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14180cee20 con 0x7f14180b2360 2024-03-24T00:31:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.237+0000 7f141d200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14180cf090 con 0x7f14180cea00 2024-03-24T00:31:39.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.237+0000 7f141d200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14180d0f60 con 0x7f14180b4330 2024-03-24T00:31:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.241+0000 7f140fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14180b4330 0x7f14180ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.241+0000 7f140fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14180b4330 0x7f14180ce4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54682/0 (socket says 172.21.15.40:54682) 2024-03-24T00:31:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.241+0000 7f140fe00640 1 -- 172.21.15.40:0/4280079233 learned_addr learned my addr 172.21.15.40:0/4280079233 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.241+0000 7f1415e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f14180b2360 0x7f14180b2740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56342/0 (socket says 172.21.15.40:56342) 2024-03-24T00:31:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.241+0000 7f140fe00640 1 -- 172.21.15.40:0/4280079233 >> v1:172.21.15.53:6789/0 conn(0x7f14180cea00 legacy=0x7f14180cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:39.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.241+0000 7f140fe00640 1 -- 172.21.15.40:0/4280079233 >> v1:172.21.15.40:6789/0 conn(0x7f14180b2360 legacy=0x7f14180b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.241+0000 7f140fe00640 1 -- 172.21.15.40:0/4280079233 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14180cf330 con 0x7f14180b4330 2024-03-24T00:31:39.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.241+0000 7f1415400640 1 -- 172.21.15.40:0/4280079233 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654863541 0 0) 0x7f14180cee20 con 0x7f14180b2360 2024-03-24T00:31:39.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.241+0000 7f140fe00640 1 --2- 172.21.15.40:0/4280079233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14180b4330 0x7f14180ce4c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f14180cf260 tx=0x7f140001c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ba3417127d2549 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.241+0000 7f1415400640 1 -- 172.21.15.40:0/4280079233 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000b1070 con 0x7f14180b4330 2024-03-24T00:31:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.241+0000 7f1415400640 1 -- 172.21.15.40:0/4280079233 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14000ac410 con 0x7f14180b4330 2024-03-24T00:31:39.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.241+0000 7f1415400640 1 -- 172.21.15.40:0/4280079233 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14000ac7e0 con 0x7f14180b4330 2024-03-24T00:31:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.245+0000 7f141d200640 1 -- 172.21.15.40:0/4280079233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14180b4330 msgr2=0x7f14180ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.245+0000 7f141d200640 1 --2- 172.21.15.40:0/4280079233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14180b4330 0x7f14180ce4c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f14180cf260 tx=0x7f140001c3b0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.245+0000 7f141d200640 1 -- 172.21.15.40:0/4280079233 shutdown_connections 2024-03-24T00:31:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.245+0000 7f141d200640 1 --2- 172.21.15.40:0/4280079233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14180b4330 0x7f14180ce4c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.245+0000 7f141d200640 1 -- 172.21.15.40:0/4280079233 >> 172.21.15.40:0/4280079233 conn(0x7f14180b3810 msgr2=0x7f14180b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.245+0000 7f141d200640 1 -- 172.21.15.40:0/4280079233 shutdown_connections 2024-03-24T00:31:39.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.245+0000 7f141d200640 1 -- 172.21.15.40:0/4280079233 wait complete. 2024-03-24T00:31:39.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.245+0000 7f141d200640 1 Processor -- start 2024-03-24T00:31:39.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.245+0000 7f141d200640 1 -- start start 2024-03-24T00:31:39.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f141d200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14180b2360 0x7f14181501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:39.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f141d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14180b4330 0x7f14180b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:39.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f141d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14180cea00 0x7f141815cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:39.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f141d200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14180cd4e0 con 0x7f14180cea00 2024-03-24T00:31:39.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f141d200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14180cd360 con 0x7f14180b4330 2024-03-24T00:31:39.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f141d200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14180cd660 con 0x7f14180b2360 2024-03-24T00:31:39.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f1416800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14180cea00 0x7f141815cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:39.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f1416800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14180cea00 0x7f141815cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54690/0 (socket says 172.21.15.40:54690) 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f1416800640 1 -- 172.21.15.40:0/2530611698 learned_addr learned my addr 172.21.15.40:0/2530611698 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f140fe00640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14180b4330 0x7f14180b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f1415e00640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14180b2360 0x7f14181501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f1416800640 1 -- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14180b2360 msgr2=0x7f14181501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f1416800640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14180b2360 0x7f14181501b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f1416800640 1 -- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14180b4330 msgr2=0x7f14180b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f1416800640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14180b4330 0x7f14180b01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f1416800640 1 -- 172.21.15.40:0/2530611698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14180b0c60 con 0x7f14180cea00 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f1416800640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14180cea00 0x7f141815cc20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f141000b970 tx=0x7f141000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f140e000640 1 -- 172.21.15.40:0/2530611698 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1410019070 con 0x7f14180cea00 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f140e000640 1 -- 172.21.15.40:0/2530611698 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141000cb40 con 0x7f14180cea00 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f141d200640 1 -- 172.21.15.40:0/2530611698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14180b0f40 con 0x7f14180cea00 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f140e000640 1 -- 172.21.15.40:0/2530611698 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1410014440 con 0x7f14180cea00 2024-03-24T00:31:39.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f141d200640 1 -- 172.21.15.40:0/2530611698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14180c9090 con 0x7f14180cea00 2024-03-24T00:31:39.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.249+0000 7f141d200640 1 -- 172.21.15.40:0/2530611698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1418068170 con 0x7f14180cea00 2024-03-24T00:31:39.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.253+0000 7f140e000640 1 -- 172.21.15.40:0/2530611698 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f141001c070 con 0x7f14180cea00 2024-03-24T00:31:39.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.253+0000 7f140e000640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13f4067660 0x7f13f4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:39.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.253+0000 7f140e000640 1 -- 172.21.15.40:0/2530611698 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1410085460 con 0x7f14180cea00 2024-03-24T00:31:39.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.253+0000 7f1415e00640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13f4067660 0x7f13f4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:39.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.253+0000 7f1415e00640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13f4067660 0x7f13f4069b20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1408000f70 tx=0x7f1408007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:39.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.257+0000 7f140e000640 1 -- 172.21.15.40:0/2530611698 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1410056170 con 0x7f14180cea00 2024-03-24T00:31:39.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.421+0000 7f141d200640 1 -- 172.21.15.40:0/2530611698 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f14181506f0 con 0x7f14180cea00 2024-03-24T00:31:39.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.421+0000 7f140e000640 1 -- 172.21.15.40:0/2530611698 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1410055a70 con 0x7f14180cea00 2024-03-24T00:31:39.427 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:39.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.425+0000 7f141d200640 1 -- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13f4067660 msgr2=0x7f13f4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.425+0000 7f141d200640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13f4067660 0x7f13f4069b20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1408000f70 tx=0x7f1408007c10 comp rx=0 tx=0).stop 2024-03-24T00:31:39.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.425+0000 7f141d200640 1 -- 172.21.15.40:0/2530611698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14180cea00 msgr2=0x7f141815cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.425+0000 7f141d200640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14180cea00 0x7f141815cc20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f141000b970 tx=0x7f141000be30 comp rx=0 tx=0).stop 2024-03-24T00:31:39.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.425+0000 7f141d200640 1 -- 172.21.15.40:0/2530611698 shutdown_connections 2024-03-24T00:31:39.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.425+0000 7f141d200640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13f4067660 0x7f13f4069b20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.425+0000 7f141d200640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14180cea00 0x7f141815cc20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.425+0000 7f141d200640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14180b4330 0x7f14180b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.425+0000 7f141d200640 1 --2- 172.21.15.40:0/2530611698 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14180b2360 0x7f14181501b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.425+0000 7f141d200640 1 -- 172.21.15.40:0/2530611698 >> 172.21.15.40:0/2530611698 conn(0x7f14180b3810 msgr2=0x7f14180cc440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:39.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.425+0000 7f141d200640 1 -- 172.21.15.40:0/2530611698 shutdown_connections 2024-03-24T00:31:39.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.425+0000 7f141d200640 1 -- 172.21.15.40:0/2530611698 wait complete. 2024-03-24T00:31:39.449 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:31:39.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.585+0000 7fef57400640 1 Processor -- start 2024-03-24T00:31:39.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.585+0000 7fef57400640 1 -- start start 2024-03-24T00:31:39.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef57400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef500b2910 0x7fef500d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:39.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef57400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fef500af170 0x7fef50165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:39.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef57400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef500b1e50 con 0x7fef5015f600 2024-03-24T00:31:39.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef57400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef500b3630 con 0x7fef500af170 2024-03-24T00:31:39.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef57400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef500b2d80 con 0x7fef500b2910 2024-03-24T00:31:39.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef55400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fef500af170 0x7fef50165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:39.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef47e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef500b2910 0x7fef500d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:39.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef55400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fef500af170 0x7fef50165e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:41252/0 (socket says 172.21.15.40:41252) 2024-03-24T00:31:39.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef55400640 1 -- 172.21.15.40:0/1393029207 learned_addr learned my addr 172.21.15.40:0/1393029207 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:39.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef47e00640 1 -- 172.21.15.40:0/1393029207 >> v2:172.21.15.53:3300/0 conn(0x7fef500af170 msgr2=0x7fef50165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:39.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef47e00640 1 --2- 172.21.15.40:0/1393029207 >> v2:172.21.15.53:3300/0 conn(0x7fef500af170 0x7fef50165e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef47e00640 1 -- 172.21.15.40:0/1393029207 >> v1:172.21.15.53:6789/0 conn(0x7fef5015f600 legacy=0x7fef500aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef4fe00640 1 -- 172.21.15.40:0/1393029207 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738567725 0 0) 0x7fef500b1e50 con 0x7fef5015f600 2024-03-24T00:31:39.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.589+0000 7fef47e00640 1 -- 172.21.15.40:0/1393029207 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef500b20f0 con 0x7fef500b2910 2024-03-24T00:31:39.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.593+0000 7fef47e00640 1 --2- 172.21.15.40:0/1393029207 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef500b2910 0x7fef500d2830 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef380097c0 tx=0x7fef3801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d23c083bfd947d7b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:39.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.593+0000 7fef4fe00640 1 -- 172.21.15.40:0/1393029207 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef380b5070 con 0x7fef500b2910 2024-03-24T00:31:39.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.593+0000 7fef4fe00640 1 -- 172.21.15.40:0/1393029207 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef380a8990 con 0x7fef500b2910 2024-03-24T00:31:39.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.593+0000 7fef4fe00640 1 -- 172.21.15.40:0/1393029207 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef380a8d60 con 0x7fef500b2910 2024-03-24T00:31:39.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.593+0000 7fef57400640 1 -- 172.21.15.40:0/1393029207 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef500b2910 msgr2=0x7fef500d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.593+0000 7fef57400640 1 --2- 172.21.15.40:0/1393029207 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef500b2910 0x7fef500d2830 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fef380097c0 tx=0x7fef3801ec90 comp rx=0 tx=0).stop 2024-03-24T00:31:39.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.593+0000 7fef57400640 1 -- 172.21.15.40:0/1393029207 shutdown_connections 2024-03-24T00:31:39.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.593+0000 7fef57400640 1 --2- 172.21.15.40:0/1393029207 >> v2:172.21.15.53:3300/0 conn(0x7fef500af170 0x7fef50165e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.593+0000 7fef57400640 1 --2- 172.21.15.40:0/1393029207 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef500b2910 0x7fef500d2830 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.593+0000 7fef57400640 1 -- 172.21.15.40:0/1393029207 >> 172.21.15.40:0/1393029207 conn(0x7fef500d1d10 msgr2=0x7fef500d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:39.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.593+0000 7fef57400640 1 -- 172.21.15.40:0/1393029207 shutdown_connections 2024-03-24T00:31:39.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.593+0000 7fef57400640 1 -- 172.21.15.40:0/1393029207 wait complete. 2024-03-24T00:31:39.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef57400640 1 Processor -- start 2024-03-24T00:31:39.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef57400640 1 -- start start 2024-03-24T00:31:39.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef57400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef500af170 0x7fef500c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:39.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef57400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef500b2910 0x7fef5015b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:39.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef57400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef5015f600 0x7fef500ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:39.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef57400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef500b2fa0 con 0x7fef500b2910 2024-03-24T00:31:39.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef57400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef500b2740 con 0x7fef500af170 2024-03-24T00:31:39.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef57400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5015f9e0 con 0x7fef5015f600 2024-03-24T00:31:39.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef47e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef500b2910 0x7fef5015b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:39.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef47e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef500b2910 0x7fef5015b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54702/0 (socket says 172.21.15.40:54702) 2024-03-24T00:31:39.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef47e00640 1 -- 172.21.15.40:0/4207902828 learned_addr learned my addr 172.21.15.40:0/4207902828 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:39.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef54a00640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef500af170 0x7fef500c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:39.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef55400640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef5015f600 0x7fef500ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:39.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef47e00640 1 -- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef5015f600 msgr2=0x7fef500ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef47e00640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef5015f600 0x7fef500ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef47e00640 1 -- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef500af170 msgr2=0x7fef500c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef47e00640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef500af170 0x7fef500c6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef47e00640 1 -- 172.21.15.40:0/4207902828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef500c7300 con 0x7fef500b2910 2024-03-24T00:31:39.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef55400640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef5015f600 0x7fef500ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:39.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.597+0000 7fef47e00640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef500b2910 0x7fef5015b4e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fef380b3040 tx=0x7fef3801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:39.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.601+0000 7fef4d600640 1 -- 172.21.15.40:0/4207902828 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef380bf070 con 0x7fef500b2910 2024-03-24T00:31:39.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.601+0000 7fef4d600640 1 -- 172.21.15.40:0/4207902828 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef380b07a0 con 0x7fef500b2910 2024-03-24T00:31:39.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.601+0000 7fef57400640 1 -- 172.21.15.40:0/4207902828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef500c7580 con 0x7fef500b2910 2024-03-24T00:31:39.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.601+0000 7fef4d600640 1 -- 172.21.15.40:0/4207902828 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef380b5040 con 0x7fef500b2910 2024-03-24T00:31:39.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.601+0000 7fef57400640 1 -- 172.21.15.40:0/4207902828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef500c9ed0 con 0x7fef500b2910 2024-03-24T00:31:39.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.601+0000 7fef57400640 1 -- 172.21.15.40:0/4207902828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef50068170 con 0x7fef500b2910 2024-03-24T00:31:39.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.601+0000 7fef4d600640 1 -- 172.21.15.40:0/4207902828 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fef380c2070 con 0x7fef500b2910 2024-03-24T00:31:39.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.601+0000 7fef4d600640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fef2c067660 0x7fef2c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:39.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.601+0000 7fef4d600640 1 -- 172.21.15.40:0/4207902828 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef38122070 con 0x7fef500b2910 2024-03-24T00:31:39.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.605+0000 7fef54a00640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fef2c067660 0x7fef2c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:39.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.605+0000 7fef54a00640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fef2c067660 0x7fef2c069b20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fef501e9490 tx=0x7fef40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:39.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.605+0000 7fef4d600640 1 -- 172.21.15.40:0/4207902828 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fef500c7300 con 0x7fef500b2910 2024-03-24T00:31:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 -- 172.21.15.40:0/4207902828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fef50070930 con 0x7fef500b2910 2024-03-24T00:31:39.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef4d600640 1 -- 172.21.15.40:0/4207902828 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fef380187f0 con 0x7fef500b2910 2024-03-24T00:31:39.785 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:31:39.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 -- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fef2c067660 msgr2=0x7fef2c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fef2c067660 0x7fef2c069b20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fef501e9490 tx=0x7fef40009040 comp rx=0 tx=0).stop 2024-03-24T00:31:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 -- 172.21.15.40:0/4207902828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef500b2910 msgr2=0x7fef5015b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef500b2910 0x7fef5015b4e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fef380b3040 tx=0x7fef3801ba30 comp rx=0 tx=0).stop 2024-03-24T00:31:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 -- 172.21.15.40:0/4207902828 shutdown_connections 2024-03-24T00:31:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fef2c067660 0x7fef2c069b20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef5015f600 0x7fef500ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef500b2910 0x7fef5015b4e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 --2- 172.21.15.40:0/4207902828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef500af170 0x7fef500c6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 -- 172.21.15.40:0/4207902828 >> 172.21.15.40:0/4207902828 conn(0x7fef500d1d10 msgr2=0x7fef5015e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 -- 172.21.15.40:0/4207902828 shutdown_connections 2024-03-24T00:31:39.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:39.781+0000 7fef57400640 1 -- 172.21.15.40:0/4207902828 wait complete. 2024-03-24T00:31:39.813 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-03-24T00:31:39.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:39.814 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T00:31:39.846 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:31:39.846 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:31:39.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:39.846 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:31:39.904 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:31:39.904 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:31:39.914 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:31:39.914 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:31:39.965 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:40.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.105+0000 7f7040200640 1 Processor -- start 2024-03-24T00:31:40.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.105+0000 7f7040200640 1 -- start start 2024-03-24T00:31:40.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.105+0000 7f7040200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380b2910 0x7f70380d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.105+0000 7f7040200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380af170 0x7f7038165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.105+0000 7f7040200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380b1e50 con 0x7f703815f600 2024-03-24T00:31:40.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.105+0000 7f7040200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70380b3630 con 0x7f70380af170 2024-03-24T00:31:40.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.105+0000 7f7040200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70380b2d80 con 0x7f70380b2910 2024-03-24T00:31:40.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.105+0000 7f703ce00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380b2910 0x7f70380d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.105+0000 7f703e200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380af170 0x7f7038165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.105+0000 7f703d800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f703815f600 0x7f70380aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:55844/0 (socket says 172.21.15.40:55844) 2024-03-24T00:31:40.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.105+0000 7f703d800640 1 -- 172.21.15.40:0/1938353723 learned_addr learned my addr 172.21.15.40:0/1938353723 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:40.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f703ce00640 1 -- 172.21.15.40:0/1938353723 >> v2:172.21.15.53:3300/0 conn(0x7f70380af170 msgr2=0x7f7038165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:40.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f703ce00640 1 --2- 172.21.15.40:0/1938353723 >> v2:172.21.15.53:3300/0 conn(0x7f70380af170 0x7f7038165e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f703ce00640 1 -- 172.21.15.40:0/1938353723 >> v1:172.21.15.53:6789/0 conn(0x7f703815f600 legacy=0x7f70380aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f703ce00640 1 -- 172.21.15.40:0/1938353723 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380b20f0 con 0x7f70380b2910 2024-03-24T00:31:40.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f703e200640 1 --2- 172.21.15.40:0/1938353723 >> v2:172.21.15.53:3300/0 conn(0x7f70380af170 0x7f7038165e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:40.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f703ce00640 1 --2- 172.21.15.40:0/1938353723 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380b2910 0x7f70380d2830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f70240097c0 tx=0x7f702401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fa0d5b5d1dc477b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:40.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f7037e00640 1 -- 172.21.15.40:0/1938353723 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70240b5070 con 0x7f70380b2910 2024-03-24T00:31:40.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f7037e00640 1 -- 172.21.15.40:0/1938353723 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70240a8990 con 0x7f70380b2910 2024-03-24T00:31:40.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f7037e00640 1 -- 172.21.15.40:0/1938353723 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70240a8d60 con 0x7f70380b2910 2024-03-24T00:31:40.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f7040200640 1 -- 172.21.15.40:0/1938353723 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380b2910 msgr2=0x7f70380d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f7040200640 1 --2- 172.21.15.40:0/1938353723 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380b2910 0x7f70380d2830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f70240097c0 tx=0x7f702401ec90 comp rx=0 tx=0).stop 2024-03-24T00:31:40.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f7040200640 1 -- 172.21.15.40:0/1938353723 shutdown_connections 2024-03-24T00:31:40.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f7040200640 1 --2- 172.21.15.40:0/1938353723 >> v2:172.21.15.53:3300/0 conn(0x7f70380af170 0x7f7038165e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.109+0000 7f7040200640 1 --2- 172.21.15.40:0/1938353723 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380b2910 0x7f70380d2830 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f7040200640 1 -- 172.21.15.40:0/1938353723 >> 172.21.15.40:0/1938353723 conn(0x7f70380d1d10 msgr2=0x7f70380d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:40.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f7040200640 1 -- 172.21.15.40:0/1938353723 shutdown_connections 2024-03-24T00:31:40.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f7040200640 1 -- 172.21.15.40:0/1938353723 wait complete. 2024-03-24T00:31:40.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f7040200640 1 Processor -- start 2024-03-24T00:31:40.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f7040200640 1 -- start start 2024-03-24T00:31:40.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f7040200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70380af170 0x7f70380c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f7040200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380b2910 0x7f70380cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f7040200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f703815f600 0x7f70380ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f7040200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380b2fa0 con 0x7f703815f600 2024-03-24T00:31:40.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f7040200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380b2740 con 0x7f70380af170 2024-03-24T00:31:40.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f7040200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703815f9e0 con 0x7f70380b2910 2024-03-24T00:31:40.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f703e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f703815f600 0x7f70380ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f703e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f703815f600 0x7f70380ce1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54708/0 (socket says 172.21.15.40:54708) 2024-03-24T00:31:40.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f703e200640 1 -- 172.21.15.40:0/139089561 learned_addr learned my addr 172.21.15.40:0/139089561 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:40.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f703d800640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70380af170 0x7f70380c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f703ce00640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380b2910 0x7f70380cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f703e200640 1 -- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380b2910 msgr2=0x7f70380cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f703e200640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380b2910 0x7f70380cdca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f703e200640 1 -- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70380af170 msgr2=0x7f70380c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f703e200640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70380af170 0x7f70380c7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f703e200640 1 -- 172.21.15.40:0/139089561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703806bef0 con 0x7f703815f600 2024-03-24T00:31:40.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f703d800640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70380af170 0x7f70380c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:40.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.113+0000 7f703e200640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f703815f600 0x7f70380ce1e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f703000efc0 tx=0x7f7030009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:40.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.117+0000 7f7035600640 1 -- 172.21.15.40:0/139089561 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7030018030 con 0x7f703815f600 2024-03-24T00:31:40.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.117+0000 7f7035600640 1 -- 172.21.15.40:0/139089561 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703000cde0 con 0x7f703815f600 2024-03-24T00:31:40.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.117+0000 7f7035600640 1 -- 172.21.15.40:0/139089561 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7030003bc0 con 0x7f703815f600 2024-03-24T00:31:40.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.117+0000 7f7040200640 1 -- 172.21.15.40:0/139089561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703806c1d0 con 0x7f703815f600 2024-03-24T00:31:40.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.117+0000 7f7040200640 1 -- 172.21.15.40:0/139089561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70381f5010 con 0x7f703815f600 2024-03-24T00:31:40.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.117+0000 7f7040200640 1 -- 172.21.15.40:0/139089561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7038068170 con 0x7f703815f600 2024-03-24T00:31:40.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.117+0000 7f7035600640 1 -- 172.21.15.40:0/139089561 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f703001b070 con 0x7f703815f600 2024-03-24T00:31:40.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.121+0000 7f7035600640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7018067660 0x7f7018069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.121+0000 7f7035600640 1 -- 172.21.15.40:0/139089561 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7030086280 con 0x7f703815f600 2024-03-24T00:31:40.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.121+0000 7f703d800640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7018067660 0x7f7018069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.121+0000 7f703d800640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7018067660 0x7f7018069b20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f70380cf040 tx=0x7f702c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:40.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.125+0000 7f7035600640 1 -- 172.21.15.40:0/139089561 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7030010040 con 0x7f703815f600 2024-03-24T00:31:40.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.289+0000 7f7040200640 1 -- 172.21.15.40:0/139089561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f703815c810 con 0x7f703815f600 2024-03-24T00:31:40.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.289+0000 7f7035600640 1 -- 172.21.15.40:0/139089561 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7f7030056f00 con 0x7f703815f600 2024-03-24T00:31:40.293 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:40.294 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:40.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.293+0000 7f7040200640 1 -- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7018067660 msgr2=0x7f7018069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.293+0000 7f7040200640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7018067660 0x7f7018069b20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f70380cf040 tx=0x7f702c009040 comp rx=0 tx=0).stop 2024-03-24T00:31:40.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.293+0000 7f7040200640 1 -- 172.21.15.40:0/139089561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f703815f600 msgr2=0x7f70380ce1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.293+0000 7f7040200640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f703815f600 0x7f70380ce1e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f703000efc0 tx=0x7f7030009f90 comp rx=0 tx=0).stop 2024-03-24T00:31:40.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.293+0000 7f7040200640 1 -- 172.21.15.40:0/139089561 shutdown_connections 2024-03-24T00:31:40.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.293+0000 7f7040200640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7018067660 0x7f7018069b20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.293+0000 7f7040200640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f703815f600 0x7f70380ce1e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.293+0000 7f7040200640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380b2910 0x7f70380cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.293+0000 7f7040200640 1 --2- 172.21.15.40:0/139089561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70380af170 0x7f70380c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.293+0000 7f7040200640 1 -- 172.21.15.40:0/139089561 >> 172.21.15.40:0/139089561 conn(0x7f70380d1d10 msgr2=0x7f70380b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:40.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.293+0000 7f7040200640 1 -- 172.21.15.40:0/139089561 shutdown_connections 2024-03-24T00:31:40.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.293+0000 7f7040200640 1 -- 172.21.15.40:0/139089561 wait complete. 2024-03-24T00:31:40.316 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:40.316 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:40.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe101800640 1 Processor -- start 2024-03-24T00:31:40.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe101800640 1 -- start start 2024-03-24T00:31:40.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe101800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0fc0d2270 0x7fe0fc070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe101800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0fc06d0f0 con 0x7fe0fc15df80 2024-03-24T00:31:40.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe101800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0fc06b280 con 0x7fe0fc06c950 2024-03-24T00:31:40.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe101800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0fc06b890 con 0x7fe0fc0d2270 2024-03-24T00:31:40.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe0f3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0fc0d2270 0x7fe0fc070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe0faa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe0fc15df80 0x7fe0fc0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56344/0 (socket says 172.21.15.40:56344) 2024-03-24T00:31:40.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe0faa00640 1 -- 172.21.15.40:0/1758328581 learned_addr learned my addr 172.21.15.40:0/1758328581 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:40.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe0fa000640 1 -- 172.21.15.40:0/1758328581 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337250807 0 0) 0x7fe0fc06d0f0 con 0x7fe0fc15df80 2024-03-24T00:31:40.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe0fa000640 1 -- 172.21.15.40:0/1758328581 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e0003530 con 0x7fe0fc15df80 2024-03-24T00:31:40.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe0f3e00640 1 -- 172.21.15.40:0/1758328581 >> v1:172.21.15.53:6789/0 conn(0x7fe0fc06c950 legacy=0x7fe0fc161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:40.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe0f3e00640 1 -- 172.21.15.40:0/1758328581 >> v1:172.21.15.40:6789/0 conn(0x7fe0fc15df80 legacy=0x7fe0fc0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.457+0000 7fe0f3e00640 1 -- 172.21.15.40:0/1758328581 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0fc06ce10 con 0x7fe0fc0d2270 2024-03-24T00:31:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.461+0000 7fe0f3e00640 1 --2- 172.21.15.40:0/1758328581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0fc0d2270 0x7fe0fc070ff0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc06ba10 tx=0x7fe0e401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2695905d301bfcc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.461+0000 7fe0fa000640 1 -- 172.21.15.40:0/1758328581 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e40ab430 con 0x7fe0fc0d2270 2024-03-24T00:31:40.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.461+0000 7fe0fa000640 1 -- 172.21.15.40:0/1758328581 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e40b0070 con 0x7fe0fc0d2270 2024-03-24T00:31:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.461+0000 7fe0fa000640 1 -- 172.21.15.40:0/1758328581 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e40ab7f0 con 0x7fe0fc0d2270 2024-03-24T00:31:40.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.461+0000 7fe101800640 1 -- 172.21.15.40:0/1758328581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0fc0d2270 msgr2=0x7fe0fc070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.461+0000 7fe101800640 1 --2- 172.21.15.40:0/1758328581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0fc0d2270 0x7fe0fc070ff0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc06ba10 tx=0x7fe0e401c470 comp rx=0 tx=0).stop 2024-03-24T00:31:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.461+0000 7fe101800640 1 -- 172.21.15.40:0/1758328581 shutdown_connections 2024-03-24T00:31:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.461+0000 7fe101800640 1 --2- 172.21.15.40:0/1758328581 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0fc0d2270 0x7fe0fc070ff0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.461+0000 7fe101800640 1 -- 172.21.15.40:0/1758328581 >> 172.21.15.40:0/1758328581 conn(0x7fe0fc06ecf0 msgr2=0x7fe0fc06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.461+0000 7fe101800640 1 -- 172.21.15.40:0/1758328581 shutdown_connections 2024-03-24T00:31:40.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.461+0000 7fe101800640 1 -- 172.21.15.40:0/1758328581 wait complete. 2024-03-24T00:31:40.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe101800640 1 Processor -- start 2024-03-24T00:31:40.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe101800640 1 -- start start 2024-03-24T00:31:40.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe101800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0fc06c950 0x7fe0fc0b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe101800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0fc0d2270 0x7fe0fc0b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe101800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0fc15df80 0x7fe0fc0b62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe101800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc16bfa0 con 0x7fe0fc0d2270 2024-03-24T00:31:40.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe101800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc16be20 con 0x7fe0fc06c950 2024-03-24T00:31:40.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe101800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0fc16c120 con 0x7fe0fc15df80 2024-03-24T00:31:40.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0fc0d2270 0x7fe0fc0b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0fc0d2270 0x7fe0fc0b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54732/0 (socket says 172.21.15.40:54732) 2024-03-24T00:31:40.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0f3e00640 1 -- 172.21.15.40:0/3563925289 learned_addr learned my addr 172.21.15.40:0/3563925289 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:40.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0faa00640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0fc06c950 0x7fe0fc0b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0fb400640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0fc15df80 0x7fe0fc0b62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0f3e00640 1 -- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0fc15df80 msgr2=0x7fe0fc0b62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0f3e00640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0fc15df80 0x7fe0fc0b62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0f3e00640 1 -- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0fc06c950 msgr2=0x7fe0fc0b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0f3e00640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0fc06c950 0x7fe0fc0b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0f3e00640 1 -- 172.21.15.40:0/3563925289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0fc15c280 con 0x7fe0fc0d2270 2024-03-24T00:31:40.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0faa00640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0fc06c950 0x7fe0fc0b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:40.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0f3e00640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0fc0d2270 0x7fe0fc0b4a00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe0e40ad040 tx=0x7fe0e401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:40.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0f2a00640 1 -- 172.21.15.40:0/3563925289 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e40bf410 con 0x7fe0fc0d2270 2024-03-24T00:31:40.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0f2a00640 1 -- 172.21.15.40:0/3563925289 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e40bf5d0 con 0x7fe0fc0d2270 2024-03-24T00:31:40.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe101800640 1 -- 172.21.15.40:0/3563925289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0fc0b5010 con 0x7fe0fc0d2270 2024-03-24T00:31:40.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe0f2a00640 1 -- 172.21.15.40:0/3563925289 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e40bf9c0 con 0x7fe0fc0d2270 2024-03-24T00:31:40.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.465+0000 7fe101800640 1 -- 172.21.15.40:0/3563925289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0fc0b55d0 con 0x7fe0fc0d2270 2024-03-24T00:31:40.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.469+0000 7fe101800640 1 -- 172.21.15.40:0/3563925289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0fc068170 con 0x7fe0fc0d2270 2024-03-24T00:31:40.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.469+0000 7fe0f2a00640 1 -- 172.21.15.40:0/3563925289 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe0e40bd020 con 0x7fe0fc0d2270 2024-03-24T00:31:40.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.469+0000 7fe0f2a00640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe0d8067660 0x7fe0d8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.469+0000 7fe0f2a00640 1 -- 172.21.15.40:0/3563925289 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0e40a8030 con 0x7fe0fc0d2270 2024-03-24T00:31:40.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.469+0000 7fe0faa00640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe0d8067660 0x7fe0d8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.469+0000 7fe0faa00640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe0d8067660 0x7fe0d8069b20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0011f0 tx=0x7fe0ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:40.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.477+0000 7fe0f2a00640 1 -- 172.21.15.40:0/3563925289 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe0e40c4060 con 0x7fe0fc0d2270 2024-03-24T00:31:40.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.701+0000 7fe101800640 1 -- 172.21.15.40:0/3563925289 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe0fc169b40 con 0x7fe0fc0d2270 2024-03-24T00:31:40.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe0f2a00640 1 -- 172.21.15.40:0/3563925289 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe0e4016090 con 0x7fe0fc0d2270 2024-03-24T00:31:40.708 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:40.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe101800640 1 -- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe0d8067660 msgr2=0x7fe0d8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe101800640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe0d8067660 0x7fe0d8069b20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0011f0 tx=0x7fe0ec009040 comp rx=0 tx=0).stop 2024-03-24T00:31:40.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe101800640 1 -- 172.21.15.40:0/3563925289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0fc0d2270 msgr2=0x7fe0fc0b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe101800640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0fc0d2270 0x7fe0fc0b4a00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe0e40ad040 tx=0x7fe0e401acb0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe101800640 1 -- 172.21.15.40:0/3563925289 shutdown_connections 2024-03-24T00:31:40.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe101800640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe0d8067660 0x7fe0d8069b20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe101800640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0fc15df80 0x7fe0fc0b62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe101800640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0fc0d2270 0x7fe0fc0b4a00 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe101800640 1 --2- 172.21.15.40:0/3563925289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0fc06c950 0x7fe0fc0b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe101800640 1 -- 172.21.15.40:0/3563925289 >> 172.21.15.40:0/3563925289 conn(0x7fe0fc06ecf0 msgr2=0x7fe0fc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:40.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe101800640 1 -- 172.21.15.40:0/3563925289 shutdown_connections 2024-03-24T00:31:40.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.705+0000 7fe101800640 1 -- 172.21.15.40:0/3563925289 wait complete. 2024-03-24T00:31:40.732 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:40.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d3400640 1 Processor -- start 2024-03-24T00:31:40.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d3400640 1 -- start start 2024-03-24T00:31:40.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d3400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f96cc15df80 0x7f96cc070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d3400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96cc0d2270 0x7f96cc161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d3400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc06d0f0 con 0x7f96cc06c950 2024-03-24T00:31:40.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d3400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96cc06b890 con 0x7f96cc15df80 2024-03-24T00:31:40.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d3400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc06b280 con 0x7f96cc0d2270 2024-03-24T00:31:40.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d0a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f96cc15df80 0x7f96cc070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d0a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f96cc15df80 0x7f96cc070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54738/0 (socket says 172.21.15.40:54738) 2024-03-24T00:31:40.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d0a00640 1 -- 172.21.15.40:0/3438320441 learned_addr learned my addr 172.21.15.40:0/3438320441 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:40.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d1400640 1 --2- 172.21.15.40:0/3438320441 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96cc0d2270 0x7f96cc161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d0a00640 1 -- 172.21.15.40:0/3438320441 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96cc0d2270 msgr2=0x7f96cc161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:40.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d0a00640 1 --2- 172.21.15.40:0/3438320441 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96cc0d2270 0x7f96cc161ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d0a00640 1 -- 172.21.15.40:0/3438320441 >> v1:172.21.15.53:6789/0 conn(0x7f96cc06c950 legacy=0x7f96cc0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.873+0000 7f96d0a00640 1 -- 172.21.15.40:0/3438320441 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc06ce10 con 0x7f96cc15df80 2024-03-24T00:31:40.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96d0a00640 1 --2- 172.21.15.40:0/3438320441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96cc15df80 0x7f96cc070ff0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f96c0009870 tx=0x7f96c001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988d8f53894590a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:40.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96cb400640 1 -- 172.21.15.40:0/3438320441 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c00aa650 con 0x7f96cc15df80 2024-03-24T00:31:40.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96cb400640 1 -- 172.21.15.40:0/3438320441 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c00af070 con 0x7f96cc15df80 2024-03-24T00:31:40.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96cb400640 1 -- 172.21.15.40:0/3438320441 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c00aaa10 con 0x7f96cc15df80 2024-03-24T00:31:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96d3400640 1 -- 172.21.15.40:0/3438320441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96cc15df80 msgr2=0x7f96cc070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96d3400640 1 --2- 172.21.15.40:0/3438320441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96cc15df80 0x7f96cc070ff0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f96c0009870 tx=0x7f96c001c530 comp rx=0 tx=0).stop 2024-03-24T00:31:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96d3400640 1 -- 172.21.15.40:0/3438320441 shutdown_connections 2024-03-24T00:31:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96d3400640 1 --2- 172.21.15.40:0/3438320441 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96cc0d2270 0x7f96cc161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96d3400640 1 --2- 172.21.15.40:0/3438320441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96cc15df80 0x7f96cc070ff0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96d3400640 1 -- 172.21.15.40:0/3438320441 >> 172.21.15.40:0/3438320441 conn(0x7f96cc06ecf0 msgr2=0x7f96cc06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96d3400640 1 -- 172.21.15.40:0/3438320441 shutdown_connections 2024-03-24T00:31:40.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96d3400640 1 -- 172.21.15.40:0/3438320441 wait complete. 2024-03-24T00:31:40.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.877+0000 7f96d3400640 1 Processor -- start 2024-03-24T00:31:40.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d3400640 1 -- start start 2024-03-24T00:31:40.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d3400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96cc06c950 0x7f96cc150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96cc0d2270 0x7f96cc15aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96cc15df80 0x7f96cc15afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d3400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc16af90 con 0x7f96cc0d2270 2024-03-24T00:31:40.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d3400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc16ae10 con 0x7f96cc15df80 2024-03-24T00:31:40.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d3400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc16b110 con 0x7f96cc06c950 2024-03-24T00:31:40.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96cc0d2270 0x7f96cc15aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96cc0d2270 0x7f96cc15aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54744/0 (socket says 172.21.15.40:54744) 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96cbe00640 1 -- 172.21.15.40:0/2786346940 learned_addr learned my addr 172.21.15.40:0/2786346940 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d0a00640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96cc06c950 0x7f96cc150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d1400640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96cc15df80 0x7f96cc15afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96cbe00640 1 -- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96cc06c950 msgr2=0x7f96cc150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96cbe00640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96cc06c950 0x7f96cc150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96cbe00640 1 -- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96cc15df80 msgr2=0x7f96cc15afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96cbe00640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96cc15df80 0x7f96cc15afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96cbe00640 1 -- 172.21.15.40:0/2786346940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc153130 con 0x7f96cc0d2270 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d1400640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96cc15df80 0x7f96cc15afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96cbe00640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96cc0d2270 0x7f96cc15aa90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f96b800c370 tx=0x7f96b800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96c8c00640 1 -- 172.21.15.40:0/2786346940 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b80052d0 con 0x7f96cc0d2270 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96c8c00640 1 -- 172.21.15.40:0/2786346940 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b8005470 con 0x7f96cc0d2270 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d3400640 1 -- 172.21.15.40:0/2786346940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc156160 con 0x7f96cc0d2270 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96c8c00640 1 -- 172.21.15.40:0/2786346940 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8007260 con 0x7f96cc0d2270 2024-03-24T00:31:40.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d3400640 1 -- 172.21.15.40:0/2786346940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc156720 con 0x7f96cc0d2270 2024-03-24T00:31:40.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.881+0000 7f96d3400640 1 -- 172.21.15.40:0/2786346940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96cc068170 con 0x7f96cc0d2270 2024-03-24T00:31:40.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.885+0000 7f96c8c00640 1 -- 172.21.15.40:0/2786346940 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f96b8007400 con 0x7f96cc0d2270 2024-03-24T00:31:40.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.885+0000 7f96c8c00640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f96ac067660 0x7f96ac069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:40.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.885+0000 7f96c8c00640 1 -- 172.21.15.40:0/2786346940 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96b8084d00 con 0x7f96cc0d2270 2024-03-24T00:31:40.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.885+0000 7f96d0a00640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f96ac067660 0x7f96ac069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:40.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.885+0000 7f96d0a00640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f96ac067660 0x7f96ac069b20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f96c00ae000 tx=0x7f96c00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:40.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:40.889+0000 7f96c8c00640 1 -- 172.21.15.40:0/2786346940 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f96cc153130 con 0x7f96cc0d2270 2024-03-24T00:31:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 -- 172.21.15.40:0/2786346940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f96cc0cf9d0 con 0x7f96cc0d2270 2024-03-24T00:31:41.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96c8c00640 1 -- 172.21.15.40:0/2786346940 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f96b8055290 con 0x7f96cc0d2270 2024-03-24T00:31:41.057 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:41.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 -- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f96ac067660 msgr2=0x7f96ac069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f96ac067660 0x7f96ac069b20 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f96c00ae000 tx=0x7f96c00a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:41.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 -- 172.21.15.40:0/2786346940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96cc0d2270 msgr2=0x7f96cc15aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96cc0d2270 0x7f96cc15aa90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f96b800c370 tx=0x7f96b800c830 comp rx=0 tx=0).stop 2024-03-24T00:31:41.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 -- 172.21.15.40:0/2786346940 shutdown_connections 2024-03-24T00:31:41.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f96ac067660 0x7f96ac069b20 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96cc15df80 0x7f96cc15afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96cc0d2270 0x7f96cc15aa90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 --2- 172.21.15.40:0/2786346940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96cc06c950 0x7f96cc150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 -- 172.21.15.40:0/2786346940 >> 172.21.15.40:0/2786346940 conn(0x7f96cc06ecf0 msgr2=0x7f96cc160ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:41.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 -- 172.21.15.40:0/2786346940 shutdown_connections 2024-03-24T00:31:41.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.053+0000 7f96d3400640 1 -- 172.21.15.40:0/2786346940 wait complete. 2024-03-24T00:31:41.078 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:41.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.217+0000 7f05a3c00640 1 Processor -- start 2024-03-24T00:31:41.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.217+0000 7f05a3c00640 1 -- start start 2024-03-24T00:31:41.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a3c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f059c15b960 0x7f059c070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a3c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f059c15bd40 0x7f059c0712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a3c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f059c15f970 con 0x7f059c0d0fe0 2024-03-24T00:31:41.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a3c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059c160580 con 0x7f059c15b960 2024-03-24T00:31:41.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a3c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059c15fc50 con 0x7f059c15bd40 2024-03-24T00:31:41.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a1200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f059c15b960 0x7f059c070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a1c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f059c0d0fe0 0x7f059c0cf7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56346/0 (socket says 172.21.15.40:56346) 2024-03-24T00:31:41.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a1200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f059c15b960 0x7f059c070d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54760/0 (socket says 172.21.15.40:54760) 2024-03-24T00:31:41.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a1c00640 1 -- 172.21.15.40:0/3716679219 learned_addr learned my addr 172.21.15.40:0/3716679219 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:41.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f059be00640 1 --2- 172.21.15.40:0/3716679219 >> v2:172.21.15.53:3300/0 conn(0x7f059c15bd40 0x7f059c0712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a1200640 1 -- 172.21.15.40:0/3716679219 >> v2:172.21.15.53:3300/0 conn(0x7f059c15bd40 msgr2=0x7f059c0712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:41.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a1200640 1 --2- 172.21.15.40:0/3716679219 >> v2:172.21.15.53:3300/0 conn(0x7f059c15bd40 0x7f059c0712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a1200640 1 -- 172.21.15.40:0/3716679219 >> v1:172.21.15.40:6789/0 conn(0x7f059c0d0fe0 legacy=0x7f059c0cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a1200640 1 -- 172.21.15.40:0/3716679219 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c15ff30 con 0x7f059c15b960 2024-03-24T00:31:41.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f059b400640 1 -- 172.21.15.40:0/3716679219 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353378200 0 0) 0x7f059c15f970 con 0x7f059c0d0fe0 2024-03-24T00:31:41.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.221+0000 7f05a1200640 1 --2- 172.21.15.40:0/3716679219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f059c15b960 0x7f059c070d60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f059c15f730 tx=0x7f059001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2caa46258b1e2874 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:41.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f059b400640 1 -- 172.21.15.40:0/3716679219 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05900b1070 con 0x7f059c15b960 2024-03-24T00:31:41.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f059b400640 1 -- 172.21.15.40:0/3716679219 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05900ac5f0 con 0x7f059c15b960 2024-03-24T00:31:41.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f059b400640 1 -- 172.21.15.40:0/3716679219 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05900ac9c0 con 0x7f059c15b960 2024-03-24T00:31:41.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 -- 172.21.15.40:0/3716679219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f059c15b960 msgr2=0x7f059c070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 --2- 172.21.15.40:0/3716679219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f059c15b960 0x7f059c070d60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f059c15f730 tx=0x7f059001c480 comp rx=0 tx=0).stop 2024-03-24T00:31:41.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 -- 172.21.15.40:0/3716679219 shutdown_connections 2024-03-24T00:31:41.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 --2- 172.21.15.40:0/3716679219 >> v2:172.21.15.53:3300/0 conn(0x7f059c15bd40 0x7f059c0712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 --2- 172.21.15.40:0/3716679219 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f059c15b960 0x7f059c070d60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 -- 172.21.15.40:0/3716679219 >> 172.21.15.40:0/3716679219 conn(0x7f059c06ecf0 msgr2=0x7f059c06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:41.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 -- 172.21.15.40:0/3716679219 shutdown_connections 2024-03-24T00:31:41.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 -- 172.21.15.40:0/3716679219 wait complete. 2024-03-24T00:31:41.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 Processor -- start 2024-03-24T00:31:41.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 -- start start 2024-03-24T00:31:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f059c0d0fe0 0x7f059c15d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f059c15b960 0x7f059c156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c15bd40 0x7f059c156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c168f00 con 0x7f059c0d0fe0 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059c168d80 con 0x7f059c15bd40 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a3c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059c169080 con 0x7f059c15b960 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f059c0d0fe0 0x7f059c15d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f059c0d0fe0 0x7f059c15d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54764/0 (socket says 172.21.15.40:54764) 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.225+0000 7f05a1200640 1 -- 172.21.15.40:0/3223938819 learned_addr learned my addr 172.21.15.40:0/3223938819 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f05a1200640 1 -- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f059c15b960 msgr2=0x7f059c156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f05a1200640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f059c15b960 0x7f059c156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f05a1c00640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c15bd40 0x7f059c156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f05a1200640 1 -- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c15bd40 msgr2=0x7f059c156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f05a1200640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c15bd40 0x7f059c156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f05a1200640 1 -- 172.21.15.40:0/3223938819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c15db10 con 0x7f059c0d0fe0 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f05a1c00640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c15bd40 0x7f059c156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:41.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f05a1200640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f059c0d0fe0 0x7f059c15d470 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f05900af040 tx=0x7f059001acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:41.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f0598c00640 1 -- 172.21.15.40:0/3223938819 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05900bd070 con 0x7f059c0d0fe0 2024-03-24T00:31:41.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f0598c00640 1 -- 172.21.15.40:0/3223938819 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05900b2040 con 0x7f059c0d0fe0 2024-03-24T00:31:41.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f0598c00640 1 -- 172.21.15.40:0/3223938819 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05900b1050 con 0x7f059c0d0fe0 2024-03-24T00:31:41.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f05a3c00640 1 -- 172.21.15.40:0/3223938819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059c15ddf0 con 0x7f059c0d0fe0 2024-03-24T00:31:41.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f05a3c00640 1 -- 172.21.15.40:0/3223938819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059c15e3b0 con 0x7f059c0d0fe0 2024-03-24T00:31:41.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f05a3c00640 1 -- 172.21.15.40:0/3223938819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059c068170 con 0x7f059c0d0fe0 2024-03-24T00:31:41.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f0598c00640 1 -- 172.21.15.40:0/3223938819 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f059c0d1820 con 0x7f059c0d0fe0 2024-03-24T00:31:41.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f0598c00640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f057c067660 0x7f057c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.229+0000 7f0598c00640 1 -- 172.21.15.40:0/3223938819 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f059011f290 con 0x7f059c0d0fe0 2024-03-24T00:31:41.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.233+0000 7f059be00640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f057c067660 0x7f057c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.233+0000 7f059be00640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f057c067660 0x7f057c069b20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f059c0cfdd0 tx=0x7f0588008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:41.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.233+0000 7f0598c00640 1 -- 172.21.15.40:0/3223938819 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f05900a9070 con 0x7f059c0d0fe0 2024-03-24T00:31:41.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 -- 172.21.15.40:0/3223938819 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f059c15b190 con 0x7f059c0d0fe0 2024-03-24T00:31:41.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f0598c00640 1 -- 172.21.15.40:0/3223938819 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0590016070 con 0x7f059c0d0fe0 2024-03-24T00:31:41.405 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:41.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 -- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f057c067660 msgr2=0x7f057c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f057c067660 0x7f057c069b20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f059c0cfdd0 tx=0x7f0588008040 comp rx=0 tx=0).stop 2024-03-24T00:31:41.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 -- 172.21.15.40:0/3223938819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f059c0d0fe0 msgr2=0x7f059c15d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f059c0d0fe0 0x7f059c15d470 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f05900af040 tx=0x7f059001acc0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 -- 172.21.15.40:0/3223938819 shutdown_connections 2024-03-24T00:31:41.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f057c067660 0x7f057c069b20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f059c15bd40 0x7f059c156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f059c15b960 0x7f059c156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 --2- 172.21.15.40:0/3223938819 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f059c0d0fe0 0x7f059c15d470 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 -- 172.21.15.40:0/3223938819 >> 172.21.15.40:0/3223938819 conn(0x7f059c06ecf0 msgr2=0x7f059c149920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:41.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 -- 172.21.15.40:0/3223938819 shutdown_connections 2024-03-24T00:31:41.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.401+0000 7f05a3c00640 1 -- 172.21.15.40:0/3223938819 wait complete. 2024-03-24T00:31:41.426 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:31:41.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa72be00640 1 Processor -- start 2024-03-24T00:31:41.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa72be00640 1 -- start start 2024-03-24T00:31:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa72be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa72c0cea10 0x7fa72c0d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa72be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa72c0cfdc0 0x7fa72c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa72be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72c0cc190 con 0x7fa72c0ce630 2024-03-24T00:31:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa72be00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa72c0d1080 con 0x7fa72c0cea10 2024-03-24T00:31:41.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa72be00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c0d0da0 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa722000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa72c0cea10 0x7fa72c0d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa72aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa72c0ce630 0x7fa72c06b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56350/0 (socket says 172.21.15.40:56350) 2024-03-24T00:31:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa72aa00640 1 -- 172.21.15.40:0/3141070184 learned_addr learned my addr 172.21.15.40:0/3141070184 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa72b400640 1 --2- 172.21.15.40:0/3141070184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa72c0cfdc0 0x7fa72c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa722000640 1 -- 172.21.15.40:0/3141070184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa72c0cfdc0 msgr2=0x7fa72c06ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa722000640 1 --2- 172.21.15.40:0/3141070184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa72c0cfdc0 0x7fa72c06ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa722000640 1 -- 172.21.15.40:0/3141070184 >> v1:172.21.15.40:6789/0 conn(0x7fa72c0ce630 legacy=0x7fa72c06b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa722000640 1 -- 172.21.15.40:0/3141070184 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c0cc430 con 0x7fa72c0cea10 2024-03-24T00:31:41.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.565+0000 7fa72b400640 1 --2- 172.21.15.40:0/3141070184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa72c0cfdc0 0x7fa72c06ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:41.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.569+0000 7fa722000640 1 --2- 172.21.15.40:0/3141070184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa72c0cea10 0x7fa72c0d2590 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa714009d90 tx=0x7fa71401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ec95f2131fed1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:41.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.569+0000 7fa72a000640 1 -- 172.21.15.40:0/3141070184 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7140b4070 con 0x7fa72c0cea10 2024-03-24T00:31:41.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.569+0000 7fa72a000640 1 -- 172.21.15.40:0/3141070184 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7140af600 con 0x7fa72c0cea10 2024-03-24T00:31:41.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.569+0000 7fa72a000640 1 -- 172.21.15.40:0/3141070184 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7140af9d0 con 0x7fa72c0cea10 2024-03-24T00:31:41.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.569+0000 7fa72be00640 1 -- 172.21.15.40:0/3141070184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa72c0cea10 msgr2=0x7fa72c0d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.569+0000 7fa72be00640 1 --2- 172.21.15.40:0/3141070184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa72c0cea10 0x7fa72c0d2590 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa714009d90 tx=0x7fa71401f150 comp rx=0 tx=0).stop 2024-03-24T00:31:41.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.569+0000 7fa72be00640 1 -- 172.21.15.40:0/3141070184 shutdown_connections 2024-03-24T00:31:41.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.569+0000 7fa72be00640 1 --2- 172.21.15.40:0/3141070184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa72c0cfdc0 0x7fa72c06ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.569+0000 7fa72be00640 1 --2- 172.21.15.40:0/3141070184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa72c0cea10 0x7fa72c0d2590 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.569+0000 7fa72be00640 1 -- 172.21.15.40:0/3141070184 >> 172.21.15.40:0/3141070184 conn(0x7fa72c0d1aa0 msgr2=0x7fa72c0d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:41.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.569+0000 7fa72be00640 1 -- 172.21.15.40:0/3141070184 shutdown_connections 2024-03-24T00:31:41.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.569+0000 7fa72be00640 1 -- 172.21.15.40:0/3141070184 wait complete. 2024-03-24T00:31:41.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72be00640 1 Processor -- start 2024-03-24T00:31:41.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72be00640 1 -- start start 2024-03-24T00:31:41.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa72c0ce630 0x7fa72c1f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa72c0cea10 0x7fa72c14e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa72c0cfdc0 0x7fa72c15e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c06c640 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c06c4c0 con 0x7fa72c0ce630 2024-03-24T00:31:41.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72be00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c06bfb0 con 0x7fa72c0cea10 2024-03-24T00:31:41.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa72c0cfdc0 0x7fa72c15e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa72c0cfdc0 0x7fa72c15e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54786/0 (socket says 172.21.15.40:54786) 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72b400640 1 -- 172.21.15.40:0/325808356 learned_addr learned my addr 172.21.15.40:0/325808356 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72aa00640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa72c0ce630 0x7fa72c1f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa722000640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa72c0cea10 0x7fa72c14e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72b400640 1 -- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa72c0cea10 msgr2=0x7fa72c14e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72b400640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa72c0cea10 0x7fa72c14e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72b400640 1 -- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa72c0ce630 msgr2=0x7fa72c1f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72b400640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa72c0ce630 0x7fa72c1f3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72b400640 1 -- 172.21.15.40:0/325808356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72c1f4b60 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72b400640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa72c0cfdc0 0x7fa72c15e500 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa72400d800 tx=0x7fa72400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa723400640 1 -- 172.21.15.40:0/325808356 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7240096a0 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa723400640 1 -- 172.21.15.40:0/325808356 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa724019070 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa723400640 1 -- 172.21.15.40:0/325808356 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa724009a80 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72be00640 1 -- 172.21.15.40:0/325808356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72c1f42c0 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.573+0000 7fa72be00640 1 -- 172.21.15.40:0/325808356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72c1f4880 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.577+0000 7fa72be00640 1 -- 172.21.15.40:0/325808356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72c09fb10 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.577+0000 7fa723400640 1 -- 172.21.15.40:0/325808356 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa724009c20 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.577+0000 7fa723400640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa708067660 0x7fa708069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.577+0000 7fa723400640 1 -- 172.21.15.40:0/325808356 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa724085cf0 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.577+0000 7fa72aa00640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa708067660 0x7fa708069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.577+0000 7fa72aa00640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa708067660 0x7fa708069b20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa71c000f70 tx=0x7fa71c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:41.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.581+0000 7fa723400640 1 -- 172.21.15.40:0/325808356 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa724056960 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 -- 172.21.15.40:0/325808356 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa72c04f4f0 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa723400640 1 -- 172.21.15.40:0/325808356 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa724056260 con 0x7fa72c0cfdc0 2024-03-24T00:31:41.761 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:31:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 -- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa708067660 msgr2=0x7fa708069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa708067660 0x7fa708069b20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa71c000f70 tx=0x7fa71c009040 comp rx=0 tx=0).stop 2024-03-24T00:31:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 -- 172.21.15.40:0/325808356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa72c0cfdc0 msgr2=0x7fa72c15e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa72c0cfdc0 0x7fa72c15e500 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa72400d800 tx=0x7fa72400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 -- 172.21.15.40:0/325808356 shutdown_connections 2024-03-24T00:31:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa708067660 0x7fa708069b20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa72c0cfdc0 0x7fa72c15e500 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa72c0cea10 0x7fa72c14e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 --2- 172.21.15.40:0/325808356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa72c0ce630 0x7fa72c1f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 -- 172.21.15.40:0/325808356 >> 172.21.15.40:0/325808356 conn(0x7fa72c0d1aa0 msgr2=0x7fa72c15bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 -- 172.21.15.40:0/325808356 shutdown_connections 2024-03-24T00:31:41.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.757+0000 7fa72be00640 1 -- 172.21.15.40:0/325808356 wait complete. 2024-03-24T00:31:41.783 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:31:41.783 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:31:41.793 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:41.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.929+0000 7fd6cb200640 1 Processor -- start 2024-03-24T00:31:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.929+0000 7fd6cb200640 1 -- start start 2024-03-24T00:31:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6cb200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6c40b2910 0x7fd6c40d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6cb200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd6c40af170 0x7fd6c4165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6cb200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6c40b1e50 con 0x7fd6c415f600 2024-03-24T00:31:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6cb200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c40b3630 con 0x7fd6c40b2910 2024-03-24T00:31:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6cb200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6c40b2d80 con 0x7fd6c40af170 2024-03-24T00:31:41.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6bb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd6c40b2910 0x7fd6c40d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6c9200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd6c40af170 0x7fd6c4165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6c3e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd6c415f600 0x7fd6c40aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:55868/0 (socket says 172.21.15.40:55868) 2024-03-24T00:31:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6c9200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd6c40af170 0x7fd6c4165e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:41344/0 (socket says 172.21.15.40:41344) 2024-03-24T00:31:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6c3e00640 1 -- 172.21.15.40:0/1024891440 learned_addr learned my addr 172.21.15.40:0/1024891440 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6bb400640 1 -- 172.21.15.40:0/1024891440 >> v2:172.21.15.53:3300/0 conn(0x7fd6c40af170 msgr2=0x7fd6c4165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6bb400640 1 --2- 172.21.15.40:0/1024891440 >> v2:172.21.15.53:3300/0 conn(0x7fd6c40af170 0x7fd6c4165e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6bb400640 1 -- 172.21.15.40:0/1024891440 >> v1:172.21.15.53:6789/0 conn(0x7fd6c415f600 legacy=0x7fd6c40aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6c3400640 1 -- 172.21.15.40:0/1024891440 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623046204 0 0) 0x7fd6c40b1e50 con 0x7fd6c415f600 2024-03-24T00:31:41.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.933+0000 7fd6bb400640 1 -- 172.21.15.40:0/1024891440 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c40b20f0 con 0x7fd6c40b2910 2024-03-24T00:31:41.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.937+0000 7fd6bb400640 1 --2- 172.21.15.40:0/1024891440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40b2910 0x7fd6c40d2830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40b1d60 tx=0x7fd6ac01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf67064fac08b8e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.937+0000 7fd6c3400640 1 -- 172.21.15.40:0/1024891440 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac0b5070 con 0x7fd6c40b2910 2024-03-24T00:31:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.937+0000 7fd6c3400640 1 -- 172.21.15.40:0/1024891440 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ac0a8970 con 0x7fd6c40b2910 2024-03-24T00:31:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.937+0000 7fd6c3400640 1 -- 172.21.15.40:0/1024891440 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac0a8d40 con 0x7fd6c40b2910 2024-03-24T00:31:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.937+0000 7fd6cb200640 1 -- 172.21.15.40:0/1024891440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40b2910 msgr2=0x7fd6c40d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.937+0000 7fd6cb200640 1 --2- 172.21.15.40:0/1024891440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40b2910 0x7fd6c40d2830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd6c40b1d60 tx=0x7fd6ac01ec70 comp rx=0 tx=0).stop 2024-03-24T00:31:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.937+0000 7fd6cb200640 1 -- 172.21.15.40:0/1024891440 shutdown_connections 2024-03-24T00:31:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.937+0000 7fd6cb200640 1 --2- 172.21.15.40:0/1024891440 >> v2:172.21.15.53:3300/0 conn(0x7fd6c40af170 0x7fd6c4165e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.937+0000 7fd6cb200640 1 --2- 172.21.15.40:0/1024891440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c40b2910 0x7fd6c40d2830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.937+0000 7fd6cb200640 1 -- 172.21.15.40:0/1024891440 >> 172.21.15.40:0/1024891440 conn(0x7fd6c40d1d10 msgr2=0x7fd6c40d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.937+0000 7fd6cb200640 1 -- 172.21.15.40:0/1024891440 shutdown_connections 2024-03-24T00:31:41.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.937+0000 7fd6cb200640 1 -- 172.21.15.40:0/1024891440 wait complete. 2024-03-24T00:31:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6cb200640 1 Processor -- start 2024-03-24T00:31:41.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6cb200640 1 -- start start 2024-03-24T00:31:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6cb200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6c40af170 0x7fd6c40c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6cb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6c40b2910 0x7fd6c4152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6cb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c415f600 0x7fd6c40c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6cb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c40b2fa0 con 0x7fd6c415f600 2024-03-24T00:31:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6cb200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c40b2740 con 0x7fd6c40b2910 2024-03-24T00:31:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6cb200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c415f9e0 con 0x7fd6c40af170 2024-03-24T00:31:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c415f600 0x7fd6c40c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c415f600 0x7fd6c40c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54806/0 (socket says 172.21.15.40:54806) 2024-03-24T00:31:41.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c9200640 1 -- 172.21.15.40:0/4108699670 learned_addr learned my addr 172.21.15.40:0/4108699670 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:41.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6bb400640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6c40b2910 0x7fd6c4152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c3e00640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6c40af170 0x7fd6c40c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c9200640 1 -- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6c40af170 msgr2=0x7fd6c40c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c9200640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6c40af170 0x7fd6c40c8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c9200640 1 -- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6c40b2910 msgr2=0x7fd6c4152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:41.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c9200640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6c40b2910 0x7fd6c4152000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:41.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c9200640 1 -- 172.21.15.40:0/4108699670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c40cb5c0 con 0x7fd6c415f600 2024-03-24T00:31:41.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c3e00640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6c40af170 0x7fd6c40c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c9200640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c415f600 0x7fd6c40c8de0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00efc0 tx=0x7fd6bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c0c00640 1 -- 172.21.15.40:0/4108699670 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc018020 con 0x7fd6c415f600 2024-03-24T00:31:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c0c00640 1 -- 172.21.15.40:0/4108699670 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc014480 con 0x7fd6c415f600 2024-03-24T00:31:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6c0c00640 1 -- 172.21.15.40:0/4108699670 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc014870 con 0x7fd6c415f600 2024-03-24T00:31:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6cb200640 1 -- 172.21.15.40:0/4108699670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c40cb860 con 0x7fd6c415f600 2024-03-24T00:31:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.941+0000 7fd6cb200640 1 -- 172.21.15.40:0/4108699670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c40cbe20 con 0x7fd6c415f600 2024-03-24T00:31:41.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.945+0000 7fd6cb200640 1 -- 172.21.15.40:0/4108699670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6c4068170 con 0x7fd6c415f600 2024-03-24T00:31:41.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.945+0000 7fd6c0c00640 1 -- 172.21.15.40:0/4108699670 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd6bc01b020 con 0x7fd6c415f600 2024-03-24T00:31:41.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.945+0000 7fd6c0c00640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd6a0067660 0x7fd6a0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:41.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.945+0000 7fd6c0c00640 1 -- 172.21.15.40:0/4108699670 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6bc086620 con 0x7fd6c415f600 2024-03-24T00:31:41.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.945+0000 7fd6c3e00640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd6a0067660 0x7fd6a0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:41.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.945+0000 7fd6c3e00640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd6a0067660 0x7fd6a0069b20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4000f70 tx=0x7fd6b4007cb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:41.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:41.949+0000 7fd6c0c00640 1 -- 172.21.15.40:0/4108699670 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd6bc022030 con 0x7fd6c415f600 2024-03-24T00:31:42.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.113+0000 7fd6cb200640 1 -- 172.21.15.40:0/4108699670 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6c40d10a0 con 0x7fd6c415f600 2024-03-24T00:31:42.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.113+0000 7fd6c0c00640 1 -- 172.21.15.40:0/4108699670 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7fd6bc010040 con 0x7fd6c415f600 2024-03-24T00:31:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:42.120 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.117+0000 7fd6cb200640 1 -- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd6a0067660 msgr2=0x7fd6a0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.117+0000 7fd6cb200640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd6a0067660 0x7fd6a0069b20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4000f70 tx=0x7fd6b4007cb0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.117+0000 7fd6cb200640 1 -- 172.21.15.40:0/4108699670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c415f600 msgr2=0x7fd6c40c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.117+0000 7fd6cb200640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c415f600 0x7fd6c40c8de0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00efc0 tx=0x7fd6bc00c490 comp rx=0 tx=0).stop 2024-03-24T00:31:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.117+0000 7fd6cb200640 1 -- 172.21.15.40:0/4108699670 shutdown_connections 2024-03-24T00:31:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.117+0000 7fd6cb200640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd6a0067660 0x7fd6a0069b20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.117+0000 7fd6cb200640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd6c415f600 0x7fd6c40c8de0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.117+0000 7fd6cb200640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6c40b2910 0x7fd6c4152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.117+0000 7fd6cb200640 1 --2- 172.21.15.40:0/4108699670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6c40af170 0x7fd6c40c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.117+0000 7fd6cb200640 1 -- 172.21.15.40:0/4108699670 >> 172.21.15.40:0/4108699670 conn(0x7fd6c40d1d10 msgr2=0x7fd6c415e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.117+0000 7fd6cb200640 1 -- 172.21.15.40:0/4108699670 shutdown_connections 2024-03-24T00:31:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.117+0000 7fd6cb200640 1 -- 172.21.15.40:0/4108699670 wait complete. 2024-03-24T00:31:42.142 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:42.142 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:42.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.281+0000 7fe0ba200640 1 Processor -- start 2024-03-24T00:31:42.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.281+0000 7fe0ba200640 1 -- start start 2024-03-24T00:31:42.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0ba200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe0b415df30 0x7fe0b40b3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0ba200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0b40727f0 0x7fe0b4070cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0ba200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b40b3950 con 0x7fe0b4071210 2024-03-24T00:31:42.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0ba200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b40b3c30 con 0x7fe0b40727f0 2024-03-24T00:31:42.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0ba200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b40aeb40 con 0x7fe0b415df30 2024-03-24T00:31:42.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0abe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0b40727f0 0x7fe0b4070cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0abe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0b40727f0 0x7fe0b4070cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54816/0 (socket says 172.21.15.40:54816) 2024-03-24T00:31:42.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0b3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe0b4071210 0x7fe0b40ad8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56354/0 (socket says 172.21.15.40:56354) 2024-03-24T00:31:42.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0abe00640 1 -- 172.21.15.40:0/3749999173 learned_addr learned my addr 172.21.15.40:0/3749999173 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:42.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0b3400640 1 --2- 172.21.15.40:0/3749999173 >> v2:172.21.15.53:3300/0 conn(0x7fe0b415df30 0x7fe0b40b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0abe00640 1 -- 172.21.15.40:0/3749999173 >> v2:172.21.15.53:3300/0 conn(0x7fe0b415df30 msgr2=0x7fe0b40b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:42.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0abe00640 1 --2- 172.21.15.40:0/3749999173 >> v2:172.21.15.53:3300/0 conn(0x7fe0b415df30 0x7fe0b40b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0abe00640 1 -- 172.21.15.40:0/3749999173 >> v1:172.21.15.40:6789/0 conn(0x7fe0b4071210 legacy=0x7fe0b40ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0abe00640 1 -- 172.21.15.40:0/3749999173 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b40aee20 con 0x7fe0b40727f0 2024-03-24T00:31:42.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.285+0000 7fe0b2a00640 1 -- 172.21.15.40:0/3749999173 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033948895 0 0) 0x7fe0b40b3950 con 0x7fe0b4071210 2024-03-24T00:31:42.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0abe00640 1 --2- 172.21.15.40:0/3749999173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b40727f0 0x7fe0b4070cd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4062df0 tx=0x7fe09c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=954f665fca591c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:42.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0b2a00640 1 -- 172.21.15.40:0/3749999173 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0aa670 con 0x7fe0b40727f0 2024-03-24T00:31:42.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0b2a00640 1 -- 172.21.15.40:0/3749999173 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c0af070 con 0x7fe0b40727f0 2024-03-24T00:31:42.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0b2a00640 1 -- 172.21.15.40:0/3749999173 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0aaa30 con 0x7fe0b40727f0 2024-03-24T00:31:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 -- 172.21.15.40:0/3749999173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b40727f0 msgr2=0x7fe0b4070cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 --2- 172.21.15.40:0/3749999173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b40727f0 0x7fe0b4070cd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4062df0 tx=0x7fe09c01c4f0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 -- 172.21.15.40:0/3749999173 shutdown_connections 2024-03-24T00:31:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 --2- 172.21.15.40:0/3749999173 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b40727f0 0x7fe0b4070cd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 --2- 172.21.15.40:0/3749999173 >> v2:172.21.15.53:3300/0 conn(0x7fe0b415df30 0x7fe0b40b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 -- 172.21.15.40:0/3749999173 >> 172.21.15.40:0/3749999173 conn(0x7fe0b40b1660 msgr2=0x7fe0b40b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 -- 172.21.15.40:0/3749999173 shutdown_connections 2024-03-24T00:31:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 -- 172.21.15.40:0/3749999173 wait complete. 2024-03-24T00:31:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 Processor -- start 2024-03-24T00:31:42.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 -- start start 2024-03-24T00:31:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0b4071210 0x7fe0b41fb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b40727f0 0x7fe0b4152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0b415df30 0x7fe0b413d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b416ab10 con 0x7fe0b40727f0 2024-03-24T00:31:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b4071620 con 0x7fe0b415df30 2024-03-24T00:31:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.289+0000 7fe0ba200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b416ac90 con 0x7fe0b4071210 2024-03-24T00:31:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b40727f0 0x7fe0b4152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b40727f0 0x7fe0b4152150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54822/0 (socket says 172.21.15.40:54822) 2024-03-24T00:31:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0abe00640 1 -- 172.21.15.40:0/2220744201 learned_addr learned my addr 172.21.15.40:0/2220744201 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0b3400640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0b4071210 0x7fe0b41fb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0b3e00640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0b415df30 0x7fe0b413d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0abe00640 1 -- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0b4071210 msgr2=0x7fe0b41fb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0abe00640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0b4071210 0x7fe0b41fb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0abe00640 1 -- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0b415df30 msgr2=0x7fe0b413d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0abe00640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0b415df30 0x7fe0b413d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0abe00640 1 -- 172.21.15.40:0/2220744201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b4152690 con 0x7fe0b40727f0 2024-03-24T00:31:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0b3e00640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0b415df30 0x7fe0b413d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0abe00640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b40727f0 0x7fe0b4152150 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0ac040 tx=0x7fe09c01ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0ab400640 1 -- 172.21.15.40:0/2220744201 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0bd5f0 con 0x7fe0b40727f0 2024-03-24T00:31:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0ab400640 1 -- 172.21.15.40:0/2220744201 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c0bd790 con 0x7fe0b40727f0 2024-03-24T00:31:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0ab400640 1 -- 172.21.15.40:0/2220744201 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c0bdb60 con 0x7fe0b40727f0 2024-03-24T00:31:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0ba200640 1 -- 172.21.15.40:0/2220744201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b41573b0 con 0x7fe0b40727f0 2024-03-24T00:31:42.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0ba200640 1 -- 172.21.15.40:0/2220744201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b4157970 con 0x7fe0b40727f0 2024-03-24T00:31:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0ba200640 1 -- 172.21.15.40:0/2220744201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b4068170 con 0x7fe0b40727f0 2024-03-24T00:31:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0ab400640 1 -- 172.21.15.40:0/2220744201 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe09c0b33f0 con 0x7fe0b40727f0 2024-03-24T00:31:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.293+0000 7fe0ab400640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe090067660 0x7fe090069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.297+0000 7fe0ab400640 1 -- 172.21.15.40:0/2220744201 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe09c11fce0 con 0x7fe0b40727f0 2024-03-24T00:31:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.297+0000 7fe0b3400640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe090067660 0x7fe090069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.297+0000 7fe0b3400640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe090067660 0x7fe090069b20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4009ac0 tx=0x7fe0a4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:42.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.297+0000 7fe0ab400640 1 -- 172.21.15.40:0/2220744201 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe09c016110 con 0x7fe0b40727f0 2024-03-24T00:31:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.465+0000 7fe0ba200640 1 -- 172.21.15.40:0/2220744201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe0b40cf710 con 0x7fe0b40727f0 2024-03-24T00:31:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.465+0000 7fe0ab400640 1 -- 172.21.15.40:0/2220744201 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe09c0f0270 con 0x7fe0b40727f0 2024-03-24T00:31:42.470 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.469+0000 7fe0ba200640 1 -- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe090067660 msgr2=0x7fe090069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.469+0000 7fe0ba200640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe090067660 0x7fe090069b20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4009ac0 tx=0x7fe0a4009190 comp rx=0 tx=0).stop 2024-03-24T00:31:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.469+0000 7fe0ba200640 1 -- 172.21.15.40:0/2220744201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b40727f0 msgr2=0x7fe0b4152150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.469+0000 7fe0ba200640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b40727f0 0x7fe0b4152150 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0ac040 tx=0x7fe09c01ad30 comp rx=0 tx=0).stop 2024-03-24T00:31:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.469+0000 7fe0ba200640 1 -- 172.21.15.40:0/2220744201 shutdown_connections 2024-03-24T00:31:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.469+0000 7fe0ba200640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe090067660 0x7fe090069b20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.469+0000 7fe0ba200640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0b415df30 0x7fe0b413d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.469+0000 7fe0ba200640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0b40727f0 0x7fe0b4152150 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.469+0000 7fe0ba200640 1 --2- 172.21.15.40:0/2220744201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0b4071210 0x7fe0b41fb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.469+0000 7fe0ba200640 1 -- 172.21.15.40:0/2220744201 >> 172.21.15.40:0/2220744201 conn(0x7fe0b40b1660 msgr2=0x7fe0b415b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.469+0000 7fe0ba200640 1 -- 172.21.15.40:0/2220744201 shutdown_connections 2024-03-24T00:31:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.469+0000 7fe0ba200640 1 -- 172.21.15.40:0/2220744201 wait complete. 2024-03-24T00:31:42.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:42.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.629+0000 7fd7f5800640 1 Processor -- start 2024-03-24T00:31:42.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.629+0000 7fd7f5800640 1 -- start start 2024-03-24T00:31:42.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7f5800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd7f00aec70 0x7fd7f00ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7f5800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f0070ed0 con 0x7fd7f015a560 2024-03-24T00:31:42.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7f5800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f00cf240 con 0x7fd7f006d3a0 2024-03-24T00:31:42.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7f5800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f00cf520 con 0x7fd7f00aec70 2024-03-24T00:31:42.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7eea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd7f015a560 0x7fd7f00cc4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56364/0 (socket says 172.21.15.40:56364) 2024-03-24T00:31:42.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7eea00640 1 -- 172.21.15.40:0/640967044 learned_addr learned my addr 172.21.15.40:0/640967044 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:42.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7e7e00640 1 --2- 172.21.15.40:0/640967044 >> v2:172.21.15.53:3300/0 conn(0x7fd7f00aec70 0x7fd7f00ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7ee000640 1 -- 172.21.15.40:0/640967044 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624896415 0 0) 0x7fd7f0070ed0 con 0x7fd7f015a560 2024-03-24T00:31:42.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7ee000640 1 -- 172.21.15.40:0/640967044 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7d4003530 con 0x7fd7f015a560 2024-03-24T00:31:42.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7e7e00640 1 -- 172.21.15.40:0/640967044 >> v1:172.21.15.53:6789/0 conn(0x7fd7f006d3a0 legacy=0x7fd7f00b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7ee000640 1 -- 172.21.15.40:0/640967044 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420324708 0 0) 0x7fd7f00cf240 con 0x7fd7f006d3a0 2024-03-24T00:31:42.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7e7e00640 1 -- 172.21.15.40:0/640967044 >> v1:172.21.15.40:6789/0 conn(0x7fd7f015a560 legacy=0x7fd7f00cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.633+0000 7fd7e7e00640 1 -- 172.21.15.40:0/640967044 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f00b3aa0 con 0x7fd7f00aec70 2024-03-24T00:31:42.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.637+0000 7fd7e7e00640 1 --2- 172.21.15.40:0/640967044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7f00aec70 0x7fd7f00ae6b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80097c0 tx=0x7fd7d801c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3248810ba3477a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:42.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.637+0000 7fd7ee000640 1 -- 172.21.15.40:0/640967044 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d80b0070 con 0x7fd7f00aec70 2024-03-24T00:31:42.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.637+0000 7fd7ee000640 1 -- 172.21.15.40:0/640967044 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d80ab600 con 0x7fd7f00aec70 2024-03-24T00:31:42.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.637+0000 7fd7ee000640 1 -- 172.21.15.40:0/640967044 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d80ab9d0 con 0x7fd7f00aec70 2024-03-24T00:31:42.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.641+0000 7fd7f5800640 1 -- 172.21.15.40:0/640967044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7f00aec70 msgr2=0x7fd7f00ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.641+0000 7fd7f5800640 1 --2- 172.21.15.40:0/640967044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7f00aec70 0x7fd7f00ae6b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80097c0 tx=0x7fd7d801c380 comp rx=0 tx=0).stop 2024-03-24T00:31:42.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.641+0000 7fd7f5800640 1 -- 172.21.15.40:0/640967044 shutdown_connections 2024-03-24T00:31:42.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.641+0000 7fd7f5800640 1 --2- 172.21.15.40:0/640967044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7f00aec70 0x7fd7f00ae6b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.641+0000 7fd7f5800640 1 -- 172.21.15.40:0/640967044 >> 172.21.15.40:0/640967044 conn(0x7fd7f00b1660 msgr2=0x7fd7f00b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:42.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.641+0000 7fd7f5800640 1 -- 172.21.15.40:0/640967044 shutdown_connections 2024-03-24T00:31:42.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.641+0000 7fd7f5800640 1 -- 172.21.15.40:0/640967044 wait complete. 2024-03-24T00:31:42.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.641+0000 7fd7f5800640 1 Processor -- start 2024-03-24T00:31:42.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.641+0000 7fd7f5800640 1 -- start start 2024-03-24T00:31:42.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7f5800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7f006d3a0 0x7fd7f01513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7f5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7f00aec70 0x7fd7f01507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7f5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7f015a560 0x7fd7f0168270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7f5800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f015aac0 con 0x7fd7f015a560 2024-03-24T00:31:42.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7f5800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f015a940 con 0x7fd7f00aec70 2024-03-24T00:31:42.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7f5800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f0169380 con 0x7fd7f006d3a0 2024-03-24T00:31:42.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7ef400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7f015a560 0x7fd7f0168270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7ef400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7f015a560 0x7fd7f0168270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54838/0 (socket says 172.21.15.40:54838) 2024-03-24T00:31:42.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7e7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7f00aec70 0x7fd7f01507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7ef400640 1 -- 172.21.15.40:0/2142265568 learned_addr learned my addr 172.21.15.40:0/2142265568 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:42.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7eea00640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7f006d3a0 0x7fd7f01513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7ef400640 1 -- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7f006d3a0 msgr2=0x7fd7f01513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7ef400640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7f006d3a0 0x7fd7f01513b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7ef400640 1 -- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7f00aec70 msgr2=0x7fd7f01507c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7ef400640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7f00aec70 0x7fd7f01507c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7ef400640 1 -- 172.21.15.40:0/2142265568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f0168ff0 con 0x7fd7f015a560 2024-03-24T00:31:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7e7e00640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7f00aec70 0x7fd7f01507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7ef400640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7f015a560 0x7fd7f0168270 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800c7b0 tx=0x7fd7e800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7e6a00640 1 -- 172.21.15.40:0/2142265568 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e800ce60 con 0x7fd7f015a560 2024-03-24T00:31:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7e6a00640 1 -- 172.21.15.40:0/2142265568 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e8003db0 con 0x7fd7f015a560 2024-03-24T00:31:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7f5800640 1 -- 172.21.15.40:0/2142265568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f00c8ef0 con 0x7fd7f015a560 2024-03-24T00:31:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7e6a00640 1 -- 172.21.15.40:0/2142265568 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e80052b0 con 0x7fd7f015a560 2024-03-24T00:31:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7f5800640 1 -- 172.21.15.40:0/2142265568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f00c94b0 con 0x7fd7f015a560 2024-03-24T00:31:42.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.645+0000 7fd7f5800640 1 -- 172.21.15.40:0/2142265568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f00c8ce0 con 0x7fd7f015a560 2024-03-24T00:31:42.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.649+0000 7fd7e6a00640 1 -- 172.21.15.40:0/2142265568 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd7e8005450 con 0x7fd7f015a560 2024-03-24T00:31:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.649+0000 7fd7e6a00640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7cc067660 0x7fd7cc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.649+0000 7fd7e6a00640 1 -- 172.21.15.40:0/2142265568 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7e8085650 con 0x7fd7f015a560 2024-03-24T00:31:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.649+0000 7fd7eea00640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7cc067660 0x7fd7cc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.649+0000 7fd7eea00640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7cc067660 0x7fd7cc069b20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0007fd0 tx=0x7fd7e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:42.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.653+0000 7fd7e6a00640 1 -- 172.21.15.40:0/2142265568 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd7e8055c80 con 0x7fd7f015a560 2024-03-24T00:31:42.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 -- 172.21.15.40:0/2142265568 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd7f0068170 con 0x7fd7f015a560 2024-03-24T00:31:42.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7e6a00640 1 -- 172.21.15.40:0/2142265568 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd7e8059d20 con 0x7fd7f015a560 2024-03-24T00:31:42.820 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:42.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 -- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7cc067660 msgr2=0x7fd7cc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7cc067660 0x7fd7cc069b20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0007fd0 tx=0x7fd7e0009040 comp rx=0 tx=0).stop 2024-03-24T00:31:42.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 -- 172.21.15.40:0/2142265568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7f015a560 msgr2=0x7fd7f0168270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7f015a560 0x7fd7f0168270 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd7e800c7b0 tx=0x7fd7e800cc70 comp rx=0 tx=0).stop 2024-03-24T00:31:42.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 -- 172.21.15.40:0/2142265568 shutdown_connections 2024-03-24T00:31:42.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7cc067660 0x7fd7cc069b20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7f015a560 0x7fd7f0168270 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7f00aec70 0x7fd7f01507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 --2- 172.21.15.40:0/2142265568 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7f006d3a0 0x7fd7f01513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 -- 172.21.15.40:0/2142265568 >> 172.21.15.40:0/2142265568 conn(0x7fd7f00b1660 msgr2=0x7fd7f0153a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:42.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 -- 172.21.15.40:0/2142265568 shutdown_connections 2024-03-24T00:31:42.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.817+0000 7fd7f5800640 1 -- 172.21.15.40:0/2142265568 wait complete. 2024-03-24T00:31:42.842 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:42.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.981+0000 7fb385400640 1 Processor -- start 2024-03-24T00:31:42.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.981+0000 7fb385400640 1 -- start start 2024-03-24T00:31:42.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.981+0000 7fb385400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb38015df80 0x7fb380070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.981+0000 7fb385400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb3800d2270 0x7fb380161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.981+0000 7fb385400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38006d0f0 con 0x7fb38006c950 2024-03-24T00:31:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.981+0000 7fb385400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38006b890 con 0x7fb3800d2270 2024-03-24T00:31:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.981+0000 7fb385400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38006b280 con 0x7fb38015df80 2024-03-24T00:31:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.981+0000 7fb377e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb38006c950 0x7fb3800d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56366/0 (socket says 172.21.15.40:56366) 2024-03-24T00:31:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.981+0000 7fb377e00640 1 -- 172.21.15.40:0/3207688745 learned_addr learned my addr 172.21.15.40:0/3207688745 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.981+0000 7fb37f200640 1 --2- 172.21.15.40:0/3207688745 >> v2:172.21.15.53:3300/0 conn(0x7fb3800d2270 0x7fb380161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.981+0000 7fb37e800640 1 --2- 172.21.15.40:0/3207688745 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb38015df80 0x7fb380070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.981+0000 7fb37de00640 1 -- 172.21.15.40:0/3207688745 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864408678 0 0) 0x7fb38006d0f0 con 0x7fb38006c950 2024-03-24T00:31:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb37de00640 1 -- 172.21.15.40:0/3207688745 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb364003530 con 0x7fb38006c950 2024-03-24T00:31:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb37f200640 1 -- 172.21.15.40:0/3207688745 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb38015df80 msgr2=0x7fb380070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb37f200640 1 --2- 172.21.15.40:0/3207688745 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb38015df80 0x7fb380070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb37f200640 1 -- 172.21.15.40:0/3207688745 >> v1:172.21.15.40:6789/0 conn(0x7fb38006c950 legacy=0x7fb3800d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb37f200640 1 -- 172.21.15.40:0/3207688745 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38006ce10 con 0x7fb3800d2270 2024-03-24T00:31:42.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb37f200640 1 --2- 172.21.15.40:0/3207688745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3800d2270 0x7fb380161ad0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb378009d90 tx=0x7fb37801c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d447b14e0c9af85 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb37de00640 1 -- 172.21.15.40:0/3207688745 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3780b2070 con 0x7fb3800d2270 2024-03-24T00:31:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb37de00640 1 -- 172.21.15.40:0/3207688745 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3780a59c0 con 0x7fb3800d2270 2024-03-24T00:31:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb37de00640 1 -- 172.21.15.40:0/3207688745 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3780a5d90 con 0x7fb3800d2270 2024-03-24T00:31:42.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb385400640 1 -- 172.21.15.40:0/3207688745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3800d2270 msgr2=0x7fb380161ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb385400640 1 --2- 172.21.15.40:0/3207688745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3800d2270 0x7fb380161ad0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb378009d90 tx=0x7fb37801c940 comp rx=0 tx=0).stop 2024-03-24T00:31:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb385400640 1 -- 172.21.15.40:0/3207688745 shutdown_connections 2024-03-24T00:31:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb385400640 1 --2- 172.21.15.40:0/3207688745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3800d2270 0x7fb380161ad0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb385400640 1 --2- 172.21.15.40:0/3207688745 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb38015df80 0x7fb380070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.985+0000 7fb385400640 1 -- 172.21.15.40:0/3207688745 >> 172.21.15.40:0/3207688745 conn(0x7fb38006ecf0 msgr2=0x7fb38006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb385400640 1 -- 172.21.15.40:0/3207688745 shutdown_connections 2024-03-24T00:31:42.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb385400640 1 -- 172.21.15.40:0/3207688745 wait complete. 2024-03-24T00:31:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb385400640 1 Processor -- start 2024-03-24T00:31:42.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb385400640 1 -- start start 2024-03-24T00:31:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb385400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb38006c950 0x7fb38015f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb385400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3800d2270 0x7fb380150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb385400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb38015df80 0x7fb380150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb385400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38016af90 con 0x7fb38015df80 2024-03-24T00:31:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb385400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38016ae10 con 0x7fb38006c950 2024-03-24T00:31:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb385400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38016b110 con 0x7fb3800d2270 2024-03-24T00:31:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb37f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb38015df80 0x7fb380150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb37f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb38015df80 0x7fb380150d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54848/0 (socket says 172.21.15.40:54848) 2024-03-24T00:31:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb37f200640 1 -- 172.21.15.40:0/2482268050 learned_addr learned my addr 172.21.15.40:0/2482268050 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:42.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb37e800640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb38006c950 0x7fb38015f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:42.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb37f200640 1 -- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3800d2270 msgr2=0x7fb380150520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb37f200640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3800d2270 0x7fb380150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb37f200640 1 -- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb38006c950 msgr2=0x7fb38015f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb37f200640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb38006c950 0x7fb38015f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb37f200640 1 -- 172.21.15.40:0/2482268050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb380157d50 con 0x7fb38015df80 2024-03-24T00:31:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.989+0000 7fb37f200640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb38015df80 0x7fb380150d10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb3780b0040 tx=0x7fb3780196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.993+0000 7fb376a00640 1 -- 172.21.15.40:0/2482268050 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3780b2050 con 0x7fb38015df80 2024-03-24T00:31:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.993+0000 7fb376a00640 1 -- 172.21.15.40:0/2482268050 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3780be070 con 0x7fb38015df80 2024-03-24T00:31:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.993+0000 7fb376a00640 1 -- 172.21.15.40:0/2482268050 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3780ada70 con 0x7fb38015df80 2024-03-24T00:31:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.993+0000 7fb385400640 1 -- 172.21.15.40:0/2482268050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3800ada70 con 0x7fb38015df80 2024-03-24T00:31:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.993+0000 7fb385400640 1 -- 172.21.15.40:0/2482268050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb380150a60 con 0x7fb38015df80 2024-03-24T00:31:42.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.993+0000 7fb385400640 1 -- 172.21.15.40:0/2482268050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb380068170 con 0x7fb38015df80 2024-03-24T00:31:42.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.993+0000 7fb376a00640 1 -- 172.21.15.40:0/2482268050 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb3780adc10 con 0x7fb38015df80 2024-03-24T00:31:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.993+0000 7fb376a00640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb35c067660 0x7fb35c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.993+0000 7fb376a00640 1 -- 172.21.15.40:0/2482268050 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3781209f0 con 0x7fb38015df80 2024-03-24T00:31:42.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.997+0000 7fb37e800640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb35c067660 0x7fb35c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:43.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.997+0000 7fb37e800640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb35c067660 0x7fb35c069b20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb370009ac0 tx=0x7fb370009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:43.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:42.997+0000 7fb376a00640 1 -- 172.21.15.40:0/2482268050 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb378016560 con 0x7fb38015df80 2024-03-24T00:31:43.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 -- 172.21.15.40:0/2482268050 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb3800ce020 con 0x7fb38015df80 2024-03-24T00:31:43.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb376a00640 1 -- 172.21.15.40:0/2482268050 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb3780f1660 con 0x7fb38015df80 2024-03-24T00:31:43.165 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 -- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb35c067660 msgr2=0x7fb35c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb35c067660 0x7fb35c069b20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb370009ac0 tx=0x7fb370009190 comp rx=0 tx=0).stop 2024-03-24T00:31:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 -- 172.21.15.40:0/2482268050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb38015df80 msgr2=0x7fb380150d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb38015df80 0x7fb380150d10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb3780b0040 tx=0x7fb3780196e0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 -- 172.21.15.40:0/2482268050 shutdown_connections 2024-03-24T00:31:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb35c067660 0x7fb35c069b20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb38015df80 0x7fb380150d10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3800d2270 0x7fb380150520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 --2- 172.21.15.40:0/2482268050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb38006c950 0x7fb38015f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 -- 172.21.15.40:0/2482268050 >> 172.21.15.40:0/2482268050 conn(0x7fb38006ecf0 msgr2=0x7fb38006f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 -- 172.21.15.40:0/2482268050 shutdown_connections 2024-03-24T00:31:43.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.161+0000 7fb385400640 1 -- 172.21.15.40:0/2482268050 wait complete. 2024-03-24T00:31:43.196 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:31:43.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9078400640 1 Processor -- start 2024-03-24T00:31:43.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9078400640 1 -- start start 2024-03-24T00:31:43.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9078400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90700b2aa0 0x7f90700d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:43.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9078400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90700d0a80 0x7f90700cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:43.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9078400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90700b2870 con 0x7f90700b2490 2024-03-24T00:31:43.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9078400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907006b6d0 con 0x7f90700d0a80 2024-03-24T00:31:43.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9078400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90700b3190 con 0x7f90700b2aa0 2024-03-24T00:31:43.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9076400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90700d0a80 0x7f90700cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:43.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9076400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90700d0a80 0x7f90700cc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54864/0 (socket says 172.21.15.40:54864) 2024-03-24T00:31:43.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9076400640 1 -- 172.21.15.40:0/4228778788 learned_addr learned my addr 172.21.15.40:0/4228778788 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:43.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9075000640 1 --2- 172.21.15.40:0/4228778788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90700b2aa0 0x7f90700d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:43.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9076400640 1 -- 172.21.15.40:0/4228778788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90700b2aa0 msgr2=0x7f90700d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:43.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9076400640 1 --2- 172.21.15.40:0/4228778788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90700b2aa0 0x7f90700d0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9076400640 1 -- 172.21.15.40:0/4228778788 >> v1:172.21.15.53:6789/0 conn(0x7f90700b2490 legacy=0x7f90700cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:43.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.349+0000 7f9076400640 1 -- 172.21.15.40:0/4228778788 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90700b3470 con 0x7f90700d0a80 2024-03-24T00:31:43.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.353+0000 7f9076400640 1 --2- 172.21.15.40:0/4228778788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90700d0a80 0x7f90700cc460 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9068009e50 tx=0x7f906801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6d62cb0f2e19cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:43.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.353+0000 7f906fe00640 1 -- 172.21.15.40:0/4228778788 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90680a5810 con 0x7f90700d0a80 2024-03-24T00:31:43.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.353+0000 7f906fe00640 1 -- 172.21.15.40:0/4228778788 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90680a59b0 con 0x7f90700d0a80 2024-03-24T00:31:43.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.353+0000 7f906fe00640 1 -- 172.21.15.40:0/4228778788 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90680a5d80 con 0x7f90700d0a80 2024-03-24T00:31:43.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.353+0000 7f9078400640 1 -- 172.21.15.40:0/4228778788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90700d0a80 msgr2=0x7f90700cc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:43.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.353+0000 7f9078400640 1 --2- 172.21.15.40:0/4228778788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90700d0a80 0x7f90700cc460 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9068009e50 tx=0x7f906801ca30 comp rx=0 tx=0).stop 2024-03-24T00:31:43.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.353+0000 7f9078400640 1 -- 172.21.15.40:0/4228778788 shutdown_connections 2024-03-24T00:31:43.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.353+0000 7f9078400640 1 --2- 172.21.15.40:0/4228778788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90700b2aa0 0x7f90700d0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.353+0000 7f9078400640 1 --2- 172.21.15.40:0/4228778788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90700d0a80 0x7f90700cc460 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.353+0000 7f9078400640 1 -- 172.21.15.40:0/4228778788 >> 172.21.15.40:0/4228778788 conn(0x7f90700d1d10 msgr2=0x7f90700d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:43.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.353+0000 7f9078400640 1 -- 172.21.15.40:0/4228778788 shutdown_connections 2024-03-24T00:31:43.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.353+0000 7f9078400640 1 -- 172.21.15.40:0/4228778788 wait complete. 2024-03-24T00:31:43.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9078400640 1 Processor -- start 2024-03-24T00:31:43.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9078400640 1 -- start start 2024-03-24T00:31:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9078400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90700b2490 0x7f907015bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9078400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90700b2aa0 0x7f9070139740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9078400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90700d0a80 0x7f90701510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9078400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90700cd580 con 0x7f90700d0a80 2024-03-24T00:31:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9078400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90700cd400 con 0x7f90700b2490 2024-03-24T00:31:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9078400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90700cd700 con 0x7f90700b2aa0 2024-03-24T00:31:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9076400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90700d0a80 0x7f90701510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9076400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90700d0a80 0x7f90701510e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54866/0 (socket says 172.21.15.40:54866) 2024-03-24T00:31:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9076400640 1 -- 172.21.15.40:0/711775174 learned_addr learned my addr 172.21.15.40:0/711775174 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9075000640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90700b2aa0 0x7f9070139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9075a00640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90700b2490 0x7f907015bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:43.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9076400640 1 -- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90700b2aa0 msgr2=0x7f9070139740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9076400640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90700b2aa0 0x7f9070139740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9076400640 1 -- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90700b2490 msgr2=0x7f907015bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9076400640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90700b2490 0x7f907015bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9076400640 1 -- 172.21.15.40:0/711775174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907015f170 con 0x7f90700d0a80 2024-03-24T00:31:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9076400640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90700d0a80 0x7f90701510e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f90680af040 tx=0x7f90680197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f906d600640 1 -- 172.21.15.40:0/711775174 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90680b8680 con 0x7f90700d0a80 2024-03-24T00:31:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f906d600640 1 -- 172.21.15.40:0/711775174 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90680b8860 con 0x7f90700d0a80 2024-03-24T00:31:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9078400640 1 -- 172.21.15.40:0/711775174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907015c2b0 con 0x7f90700d0a80 2024-03-24T00:31:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f906d600640 1 -- 172.21.15.40:0/711775174 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90680b8c50 con 0x7f90700d0a80 2024-03-24T00:31:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.357+0000 7f9078400640 1 -- 172.21.15.40:0/711775174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907015f710 con 0x7f90700d0a80 2024-03-24T00:31:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.361+0000 7f9078400640 1 -- 172.21.15.40:0/711775174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9070068170 con 0x7f90700d0a80 2024-03-24T00:31:43.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.361+0000 7f906d600640 1 -- 172.21.15.40:0/711775174 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f90680c0020 con 0x7f90700d0a80 2024-03-24T00:31:43.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.361+0000 7f906d600640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9050067660 0x7f9050069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:43.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.361+0000 7f906d600640 1 -- 172.21.15.40:0/711775174 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90681208a0 con 0x7f90700d0a80 2024-03-24T00:31:43.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.361+0000 7f9075a00640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9050067660 0x7f9050069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:43.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.361+0000 7f9075a00640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9050067660 0x7f9050069b20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9064000f70 tx=0x7f9064009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:43.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.365+0000 7f906d600640 1 -- 172.21.15.40:0/711775174 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9068016620 con 0x7f90700d0a80 2024-03-24T00:31:43.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.637+0000 7f9078400640 1 -- 172.21.15.40:0/711775174 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f90700b0070 con 0x7f90700d0a80 2024-03-24T00:31:43.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.637+0000 7f906d600640 1 -- 172.21.15.40:0/711775174 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f90680f0e40 con 0x7f90700d0a80 2024-03-24T00:31:43.642 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:31:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.641+0000 7f9078400640 1 -- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9050067660 msgr2=0x7f9050069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.641+0000 7f9078400640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9050067660 0x7f9050069b20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9064000f70 tx=0x7f9064009040 comp rx=0 tx=0).stop 2024-03-24T00:31:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.641+0000 7f9078400640 1 -- 172.21.15.40:0/711775174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90700d0a80 msgr2=0x7f90701510e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.641+0000 7f9078400640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90700d0a80 0x7f90701510e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f90680af040 tx=0x7f90680197d0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.641+0000 7f9078400640 1 -- 172.21.15.40:0/711775174 shutdown_connections 2024-03-24T00:31:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.641+0000 7f9078400640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9050067660 0x7f9050069b20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.641+0000 7f9078400640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90700d0a80 0x7f90701510e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.641+0000 7f9078400640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90700b2aa0 0x7f9070139740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.641+0000 7f9078400640 1 --2- 172.21.15.40:0/711775174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90700b2490 0x7f907015bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:43.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.641+0000 7f9078400640 1 -- 172.21.15.40:0/711775174 >> 172.21.15.40:0/711775174 conn(0x7f90700d1d10 msgr2=0x7f90700aeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:43.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.641+0000 7f9078400640 1 -- 172.21.15.40:0/711775174 shutdown_connections 2024-03-24T00:31:43.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:43.641+0000 7f9078400640 1 -- 172.21.15.40:0/711775174 wait complete. 2024-03-24T00:31:43.664 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi053.front.sepia.ceph.com... 2024-03-24T00:31:43.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:43.665 DEBUG:teuthology.orchestra.run.smithi053:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-24T00:31:43.696 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:31:43.697 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:31:43.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:43.697 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T00:31:43.763 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:31:43.763 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:31:43.772 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:31:43.772 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:31:43.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:44.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f752ac00640 1 Processor -- start 2024-03-24T00:31:44.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f752ac00640 1 -- start start 2024-03-24T00:31:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f752ac00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f752415f600 0x7f75240b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f752ac00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75240d2830 0x7f75240aea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f752ac00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f75240aefa0 0x7f75240af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f752ac00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75240b3630 con 0x7f75240d2830 2024-03-24T00:31:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f752ac00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75240b2740 con 0x7f75240aefa0 2024-03-24T00:31:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f752ac00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75240d2c70 con 0x7f752415f600 2024-03-24T00:31:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f751b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75240d2830 0x7f75240aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f751b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75240d2830 0x7f75240aea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54872/0 (socket says 172.21.15.40:54872) 2024-03-24T00:31:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f751b400640 1 -- 172.21.15.40:0/3003516644 learned_addr learned my addr 172.21.15.40:0/3003516644 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f7523e00640 1 --2- 172.21.15.40:0/3003516644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f752415f600 0x7f75240b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f7528c00640 1 --2- 172.21.15.40:0/3003516644 >> v2:172.21.15.53:3300/0 conn(0x7f75240aefa0 0x7f75240af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f751b400640 1 -- 172.21.15.40:0/3003516644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f752415f600 msgr2=0x7f75240b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:44.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f751b400640 1 --2- 172.21.15.40:0/3003516644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f752415f600 0x7f75240b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f751b400640 1 -- 172.21.15.40:0/3003516644 >> v2:172.21.15.53:3300/0 conn(0x7f75240aefa0 msgr2=0x7f75240af3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:44.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f751b400640 1 --2- 172.21.15.40:0/3003516644 >> v2:172.21.15.53:3300/0 conn(0x7f75240aefa0 0x7f75240af3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.005+0000 7f751b400640 1 -- 172.21.15.40:0/3003516644 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75240b1e50 con 0x7f75240d2830 2024-03-24T00:31:44.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f751b400640 1 --2- 172.21.15.40:0/3003516644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75240d2830 0x7f75240aea60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7524071b30 tx=0x7f750c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6f537c4fea9c26 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:44.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f7523400640 1 -- 172.21.15.40:0/3003516644 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c0b5070 con 0x7f75240d2830 2024-03-24T00:31:44.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f7523400640 1 -- 172.21.15.40:0/3003516644 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750c0a89d0 con 0x7f75240d2830 2024-03-24T00:31:44.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f7523400640 1 -- 172.21.15.40:0/3003516644 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750c0a8da0 con 0x7f75240d2830 2024-03-24T00:31:44.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f752ac00640 1 -- 172.21.15.40:0/3003516644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75240d2830 msgr2=0x7f75240aea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f752ac00640 1 --2- 172.21.15.40:0/3003516644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75240d2830 0x7f75240aea60 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7524071b30 tx=0x7f750c01ed10 comp rx=0 tx=0).stop 2024-03-24T00:31:44.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f752ac00640 1 -- 172.21.15.40:0/3003516644 shutdown_connections 2024-03-24T00:31:44.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f752ac00640 1 --2- 172.21.15.40:0/3003516644 >> v2:172.21.15.53:3300/0 conn(0x7f75240aefa0 0x7f75240af3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f752ac00640 1 --2- 172.21.15.40:0/3003516644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75240d2830 0x7f75240aea60 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f752ac00640 1 --2- 172.21.15.40:0/3003516644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f752415f600 0x7f75240b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f752ac00640 1 -- 172.21.15.40:0/3003516644 >> 172.21.15.40:0/3003516644 conn(0x7f75240d1d10 msgr2=0x7f75240d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:44.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f752ac00640 1 -- 172.21.15.40:0/3003516644 shutdown_connections 2024-03-24T00:31:44.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.009+0000 7f752ac00640 1 -- 172.21.15.40:0/3003516644 wait complete. 2024-03-24T00:31:44.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f752ac00640 1 Processor -- start 2024-03-24T00:31:44.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f752ac00640 1 -- start start 2024-03-24T00:31:44.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f752ac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75240aefa0 0x7f7524159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f752ac00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75240d2830 0x7f75240c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f752ac00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f752415f600 0x7f75240c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f752ac00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75240b2fa0 con 0x7f75240aefa0 2024-03-24T00:31:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f752ac00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75240b23d0 con 0x7f75240d2830 2024-03-24T00:31:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f752ac00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752415f9e0 con 0x7f752415f600 2024-03-24T00:31:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7523e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75240aefa0 0x7f7524159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7523e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75240aefa0 0x7f7524159660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54886/0 (socket says 172.21.15.40:54886) 2024-03-24T00:31:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7523e00640 1 -- 172.21.15.40:0/2664917571 learned_addr learned my addr 172.21.15.40:0/2664917571 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7528c00640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f752415f600 0x7f75240c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f751b400640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75240d2830 0x7f75240c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7523e00640 1 -- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f752415f600 msgr2=0x7f75240c6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7523e00640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f752415f600 0x7f75240c6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7523e00640 1 -- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75240d2830 msgr2=0x7f75240c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7523e00640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75240d2830 0x7f75240c62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7523e00640 1 -- 172.21.15.40:0/2664917571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75240c7770 con 0x7f75240aefa0 2024-03-24T00:31:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7523e00640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75240aefa0 0x7f7524159660 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f751400e930 tx=0x7f751400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7520c00640 1 -- 172.21.15.40:0/2664917571 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7514017070 con 0x7f75240aefa0 2024-03-24T00:31:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7520c00640 1 -- 172.21.15.40:0/2664917571 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751400c9a0 con 0x7f75240aefa0 2024-03-24T00:31:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f752ac00640 1 -- 172.21.15.40:0/2664917571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75240c5810 con 0x7f75240aefa0 2024-03-24T00:31:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f7520c00640 1 -- 172.21.15.40:0/2664917571 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751400cd50 con 0x7f75240aefa0 2024-03-24T00:31:44.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f752ac00640 1 -- 172.21.15.40:0/2664917571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75240c5dd0 con 0x7f75240aefa0 2024-03-24T00:31:44.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.013+0000 7f752ac00640 1 -- 172.21.15.40:0/2664917571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7524068170 con 0x7f75240aefa0 2024-03-24T00:31:44.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.017+0000 7f7520c00640 1 -- 172.21.15.40:0/2664917571 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f751401a070 con 0x7f75240aefa0 2024-03-24T00:31:44.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.017+0000 7f7520c00640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7500067660 0x7f7500069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.017+0000 7f7520c00640 1 -- 172.21.15.40:0/2664917571 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7514085ca0 con 0x7f75240aefa0 2024-03-24T00:31:44.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.017+0000 7f751b400640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7500067660 0x7f7500069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.017+0000 7f751b400640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7500067660 0x7f7500069b20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f750c0b2000 tx=0x7f750c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:44.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.021+0000 7f7520c00640 1 -- 172.21.15.40:0/2664917571 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7514056260 con 0x7f75240aefa0 2024-03-24T00:31:44.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.273+0000 7f752ac00640 1 -- 172.21.15.40:0/2664917571 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75240d2190 con 0x7f75240aefa0 2024-03-24T00:31:44.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.273+0000 7f7520c00640 1 -- 172.21.15.40:0/2664917571 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7f751405a300 con 0x7f75240aefa0 2024-03-24T00:31:44.278 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:44.279 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.277+0000 7f752ac00640 1 -- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7500067660 msgr2=0x7f7500069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.277+0000 7f752ac00640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7500067660 0x7f7500069b20 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f750c0b2000 tx=0x7f750c009040 comp rx=0 tx=0).stop 2024-03-24T00:31:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.277+0000 7f752ac00640 1 -- 172.21.15.40:0/2664917571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75240aefa0 msgr2=0x7f7524159660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.277+0000 7f752ac00640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75240aefa0 0x7f7524159660 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f751400e930 tx=0x7f751400edf0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.277+0000 7f752ac00640 1 -- 172.21.15.40:0/2664917571 shutdown_connections 2024-03-24T00:31:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.277+0000 7f752ac00640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7500067660 0x7f7500069b20 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.277+0000 7f752ac00640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f752415f600 0x7f75240c6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.277+0000 7f752ac00640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75240d2830 0x7f75240c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.277+0000 7f752ac00640 1 --2- 172.21.15.40:0/2664917571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75240aefa0 0x7f7524159660 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.277+0000 7f752ac00640 1 -- 172.21.15.40:0/2664917571 >> 172.21.15.40:0/2664917571 conn(0x7f75240d1d10 msgr2=0x7f75240ce6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:44.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.277+0000 7f752ac00640 1 -- 172.21.15.40:0/2664917571 shutdown_connections 2024-03-24T00:31:44.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.277+0000 7f752ac00640 1 -- 172.21.15.40:0/2664917571 wait complete. 2024-03-24T00:31:44.300 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:44.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:44.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5f1000640 1 Processor -- start 2024-03-24T00:31:44.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5f1000640 1 -- start start 2024-03-24T00:31:44.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5f1000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5ec15bd00 0x7fa5ec0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5f1000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa5ec0728d0 0x7fa5ec0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5f1000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5ec072d60 con 0x7fa5ec15f8c0 2024-03-24T00:31:44.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5f1000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ec0cf790 con 0x7fa5ec15bd00 2024-03-24T00:31:44.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5f1000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5ec071de0 con 0x7fa5ec0728d0 2024-03-24T00:31:44.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5e9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5ec15bd00 0x7fa5ec0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5e1400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa5ec0728d0 0x7fa5ec0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5e9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5ec15bd00 0x7fa5ec0ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54898/0 (socket says 172.21.15.40:54898) 2024-03-24T00:31:44.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5e9e00640 1 -- 172.21.15.40:0/2005684853 learned_addr learned my addr 172.21.15.40:0/2005684853 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:44.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5e1400640 1 -- 172.21.15.40:0/2005684853 >> v2:172.21.15.40:3300/0 conn(0x7fa5ec15bd00 msgr2=0x7fa5ec0ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:44.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5e1400640 1 --2- 172.21.15.40:0/2005684853 >> v2:172.21.15.40:3300/0 conn(0x7fa5ec15bd00 0x7fa5ec0ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5e1400640 1 -- 172.21.15.40:0/2005684853 >> v1:172.21.15.53:6789/0 conn(0x7fa5ec15f8c0 legacy=0x7fa5ec15fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.441+0000 7fa5e1400640 1 -- 172.21.15.40:0/2005684853 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec0cea50 con 0x7fa5ec0728d0 2024-03-24T00:31:44.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.445+0000 7fa5e1400640 1 --2- 172.21.15.40:0/2005684853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5ec0728d0 0x7fa5ec0cf250 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec06ff40 tx=0x7fa5d401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a85bc71fa16f677d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.445+0000 7fa5e9400640 1 -- 172.21.15.40:0/2005684853 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d40ab650 con 0x7fa5ec0728d0 2024-03-24T00:31:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.445+0000 7fa5e9400640 1 -- 172.21.15.40:0/2005684853 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d40b0070 con 0x7fa5ec0728d0 2024-03-24T00:31:44.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.445+0000 7fa5e9400640 1 -- 172.21.15.40:0/2005684853 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d40aba10 con 0x7fa5ec0728d0 2024-03-24T00:31:44.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.445+0000 7fa5f1000640 1 -- 172.21.15.40:0/2005684853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5ec0728d0 msgr2=0x7fa5ec0cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.445+0000 7fa5f1000640 1 --2- 172.21.15.40:0/2005684853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5ec0728d0 0x7fa5ec0cf250 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec06ff40 tx=0x7fa5d401c450 comp rx=0 tx=0).stop 2024-03-24T00:31:44.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.445+0000 7fa5f1000640 1 -- 172.21.15.40:0/2005684853 shutdown_connections 2024-03-24T00:31:44.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.445+0000 7fa5f1000640 1 --2- 172.21.15.40:0/2005684853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5ec0728d0 0x7fa5ec0cf250 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.445+0000 7fa5f1000640 1 --2- 172.21.15.40:0/2005684853 >> v2:172.21.15.40:3300/0 conn(0x7fa5ec15bd00 0x7fa5ec0ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.445+0000 7fa5f1000640 1 -- 172.21.15.40:0/2005684853 >> 172.21.15.40:0/2005684853 conn(0x7fa5ec0b1660 msgr2=0x7fa5ec0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:44.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.445+0000 7fa5f1000640 1 -- 172.21.15.40:0/2005684853 shutdown_connections 2024-03-24T00:31:44.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.445+0000 7fa5f1000640 1 -- 172.21.15.40:0/2005684853 wait complete. 2024-03-24T00:31:44.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5f1000640 1 Processor -- start 2024-03-24T00:31:44.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5f1000640 1 -- start start 2024-03-24T00:31:44.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5f1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5ec0728d0 0x7fa5ec1448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5f1000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5ec15bd00 0x7fa5ec156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5f1000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5ec15f8c0 0x7fa5ec15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5f1000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec06d320 con 0x7fa5ec0728d0 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5f1000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec15c0e0 con 0x7fa5ec15f8c0 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5f1000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5ec06d4a0 con 0x7fa5ec15bd00 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5e9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5ec0728d0 0x7fa5ec1448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5e9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5ec0728d0 0x7fa5ec1448e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54902/0 (socket says 172.21.15.40:54902) 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5e9e00640 1 -- 172.21.15.40:0/2804499595 learned_addr learned my addr 172.21.15.40:0/2804499595 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5e9e00640 1 -- 172.21.15.40:0/2804499595 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5ec15bd00 msgr2=0x7fa5ec156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5e9e00640 1 --2- 172.21.15.40:0/2804499595 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5ec15bd00 0x7fa5ec156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5e9e00640 1 -- 172.21.15.40:0/2804499595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5ec15f8c0 msgr2=0x7fa5ec15a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5e9e00640 1 --2- 172.21.15.40:0/2804499595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5ec15f8c0 0x7fa5ec15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5e9e00640 1 -- 172.21.15.40:0/2804499595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec144e20 con 0x7fa5ec0728d0 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5e9e00640 1 --2- 172.21.15.40:0/2804499595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5ec0728d0 0x7fa5ec1448e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc00efc0 tx=0x7fa5dc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5e2a00640 1 -- 172.21.15.40:0/2804499595 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc00cda0 con 0x7fa5ec0728d0 2024-03-24T00:31:44.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5e2a00640 1 -- 172.21.15.40:0/2804499595 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5dc019070 con 0x7fa5ec0728d0 2024-03-24T00:31:44.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5f1000640 1 -- 172.21.15.40:0/2804499595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ec14e020 con 0x7fa5ec0728d0 2024-03-24T00:31:44.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5e2a00640 1 -- 172.21.15.40:0/2804499595 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5dc014680 con 0x7fa5ec0728d0 2024-03-24T00:31:44.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.449+0000 7fa5f1000640 1 -- 172.21.15.40:0/2804499595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5ec14e5e0 con 0x7fa5ec0728d0 2024-03-24T00:31:44.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.453+0000 7fa5f1000640 1 -- 172.21.15.40:0/2804499595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5ec068170 con 0x7fa5ec0728d0 2024-03-24T00:31:44.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.453+0000 7fa5e2a00640 1 -- 172.21.15.40:0/2804499595 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa5dc012050 con 0x7fa5ec0728d0 2024-03-24T00:31:44.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.453+0000 7fa5e2a00640 1 --2- 172.21.15.40:0/2804499595 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5c8067660 0x7fa5c8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.453+0000 7fa5e2a00640 1 -- 172.21.15.40:0/2804499595 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5dc085f30 con 0x7fa5ec0728d0 2024-03-24T00:31:44.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.453+0000 7fa5e1400640 1 --2- 172.21.15.40:0/2804499595 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5c8067660 0x7fa5c8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.457+0000 7fa5e1400640 1 --2- 172.21.15.40:0/2804499595 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5c8067660 0x7fa5c8069b20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40af000 tx=0x7fa5d4004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:44.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.457+0000 7fa5e2a00640 1 -- 172.21.15.40:0/2804499595 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa5dc010040 con 0x7fa5ec0728d0 2024-03-24T00:31:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.621+0000 7fa5f1000640 1 -- 172.21.15.40:0/2804499595 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5ec0aec00 con 0x7fa5ec0728d0 2024-03-24T00:31:44.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.621+0000 7fa5e2a00640 1 -- 172.21.15.40:0/2804499595 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5dc056b10 con 0x7fa5ec0728d0 2024-03-24T00:31:44.626 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.621+0000 7fa5f1000640 1 -- 172.21.15.40:0/2804499595 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5c8067660 msgr2=0x7fa5c8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.621+0000 7fa5f1000640 1 --2- 172.21.15.40:0/2804499595 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5c8067660 0x7fa5c8069b20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40af000 tx=0x7fa5d4004740 comp rx=0 tx=0).stop 2024-03-24T00:31:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.621+0000 7fa5f1000640 1 -- 172.21.15.40:0/2804499595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5ec0728d0 msgr2=0x7fa5ec1448e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.621+0000 7fa5f1000640 1 --2- 172.21.15.40:0/2804499595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5ec0728d0 0x7fa5ec1448e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa5dc00efc0 tx=0x7fa5dc00c490 comp rx=0 tx=0).stop 2024-03-24T00:31:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.621+0000 7fa5f1000640 1 -- 172.21.15.40:0/2804499595 shutdown_connections 2024-03-24T00:31:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.621+0000 7fa5f1000640 1 --2- 172.21.15.40:0/2804499595 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5c8067660 0x7fa5c8069b20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.621+0000 7fa5f1000640 1 --2- 172.21.15.40:0/2804499595 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5ec15f8c0 0x7fa5ec15a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.621+0000 7fa5f1000640 1 --2- 172.21.15.40:0/2804499595 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5ec15bd00 0x7fa5ec156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.621+0000 7fa5f1000640 1 --2- 172.21.15.40:0/2804499595 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5ec0728d0 0x7fa5ec1448e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.621+0000 7fa5f1000640 1 -- 172.21.15.40:0/2804499595 >> 172.21.15.40:0/2804499595 conn(0x7fa5ec0b1660 msgr2=0x7fa5ec1594b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.625+0000 7fa5f1000640 1 -- 172.21.15.40:0/2804499595 shutdown_connections 2024-03-24T00:31:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.625+0000 7fa5f1000640 1 -- 172.21.15.40:0/2804499595 wait complete. 2024-03-24T00:31:44.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:44.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.785+0000 7f5d83a00640 1 Processor -- start 2024-03-24T00:31:44.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.785+0000 7f5d83a00640 1 -- start start 2024-03-24T00:31:44.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.785+0000 7f5d83a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d7c0d2270 0x7f5d7c070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.785+0000 7f5d83a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c06d0f0 con 0x7f5d7c15df80 2024-03-24T00:31:44.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.785+0000 7f5d83a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c06b280 con 0x7f5d7c06c950 2024-03-24T00:31:44.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.785+0000 7f5d83a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7c06b890 con 0x7f5d7c0d2270 2024-03-24T00:31:44.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.785+0000 7f5d7be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d7c0d2270 0x7f5d7c070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.785+0000 7f5d81000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5d7c15df80 0x7f5d7c0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56372/0 (socket says 172.21.15.40:56372) 2024-03-24T00:31:44.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.785+0000 7f5d7be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d7c0d2270 0x7f5d7c070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54916/0 (socket says 172.21.15.40:54916) 2024-03-24T00:31:44.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.785+0000 7f5d81000640 1 -- 172.21.15.40:0/2004695775 learned_addr learned my addr 172.21.15.40:0/2004695775 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:44.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.789+0000 7f5d7be00640 1 -- 172.21.15.40:0/2004695775 >> v1:172.21.15.53:6789/0 conn(0x7f5d7c06c950 legacy=0x7f5d7c161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.789+0000 7f5d7be00640 1 -- 172.21.15.40:0/2004695775 >> v1:172.21.15.40:6789/0 conn(0x7f5d7c15df80 legacy=0x7f5d7c0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.789+0000 7f5d7be00640 1 -- 172.21.15.40:0/2004695775 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c06ce10 con 0x7f5d7c0d2270 2024-03-24T00:31:44.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.789+0000 7f5d7b400640 1 -- 172.21.15.40:0/2004695775 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255111006 0 0) 0x7f5d7c06d0f0 con 0x7f5d7c15df80 2024-03-24T00:31:44.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d7be00640 1 --2- 172.21.15.40:0/2004695775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d7c0d2270 0x7f5d7c070ff0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c06ba10 tx=0x7f5d6801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2271d4563490097d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:44.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d7b400640 1 -- 172.21.15.40:0/2004695775 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d680ab430 con 0x7f5d7c0d2270 2024-03-24T00:31:44.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d7b400640 1 -- 172.21.15.40:0/2004695775 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d680b0070 con 0x7f5d7c0d2270 2024-03-24T00:31:44.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d7b400640 1 -- 172.21.15.40:0/2004695775 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d680ab7f0 con 0x7f5d7c0d2270 2024-03-24T00:31:44.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d83a00640 1 -- 172.21.15.40:0/2004695775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d7c0d2270 msgr2=0x7f5d7c070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d83a00640 1 --2- 172.21.15.40:0/2004695775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d7c0d2270 0x7f5d7c070ff0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c06ba10 tx=0x7f5d6801c470 comp rx=0 tx=0).stop 2024-03-24T00:31:44.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d83a00640 1 -- 172.21.15.40:0/2004695775 shutdown_connections 2024-03-24T00:31:44.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d83a00640 1 --2- 172.21.15.40:0/2004695775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d7c0d2270 0x7f5d7c070ff0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d83a00640 1 -- 172.21.15.40:0/2004695775 >> 172.21.15.40:0/2004695775 conn(0x7f5d7c06ecf0 msgr2=0x7f5d7c06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:44.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d83a00640 1 -- 172.21.15.40:0/2004695775 shutdown_connections 2024-03-24T00:31:44.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d83a00640 1 -- 172.21.15.40:0/2004695775 wait complete. 2024-03-24T00:31:44.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d83a00640 1 Processor -- start 2024-03-24T00:31:44.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.793+0000 7f5d83a00640 1 -- start start 2024-03-24T00:31:44.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d83a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d7c06c950 0x7f5d7c14f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d83a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d7c0d2270 0x7f5d7c1fa580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d83a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d7c15df80 0x7f5d7c15b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d83a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c16bfa0 con 0x7f5d7c0d2270 2024-03-24T00:31:44.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d83a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c16be20 con 0x7f5d7c15df80 2024-03-24T00:31:44.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d83a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c16c120 con 0x7f5d7c06c950 2024-03-24T00:31:44.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d7be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d7c0d2270 0x7f5d7c1fa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d7be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d7c0d2270 0x7f5d7c1fa580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54920/0 (socket says 172.21.15.40:54920) 2024-03-24T00:31:44.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d7be00640 1 -- 172.21.15.40:0/844868018 learned_addr learned my addr 172.21.15.40:0/844868018 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:44.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d81000640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d7c06c950 0x7f5d7c14f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d81a00640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d7c15df80 0x7f5d7c15b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d7be00640 1 -- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d7c06c950 msgr2=0x7f5d7c14f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d7be00640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d7c06c950 0x7f5d7c14f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d7be00640 1 -- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d7c15df80 msgr2=0x7f5d7c15b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d7be00640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d7c15df80 0x7f5d7c15b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d7be00640 1 -- 172.21.15.40:0/844868018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c1faac0 con 0x7f5d7c0d2270 2024-03-24T00:31:44.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d7be00640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d7c0d2270 0x7f5d7c1fa580 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5d680ad040 tx=0x7f5d6801acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:44.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d78c00640 1 -- 172.21.15.40:0/844868018 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d680bf610 con 0x7f5d7c0d2270 2024-03-24T00:31:44.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d78c00640 1 -- 172.21.15.40:0/844868018 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d680bf7b0 con 0x7f5d7c0d2270 2024-03-24T00:31:44.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d78c00640 1 -- 172.21.15.40:0/844868018 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d680bfba0 con 0x7f5d7c0d2270 2024-03-24T00:31:44.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d83a00640 1 -- 172.21.15.40:0/844868018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c1fad60 con 0x7f5d7c0d2270 2024-03-24T00:31:44.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d83a00640 1 -- 172.21.15.40:0/844868018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c1fb320 con 0x7f5d7c0d2270 2024-03-24T00:31:44.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.797+0000 7f5d83a00640 1 -- 172.21.15.40:0/844868018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c068170 con 0x7f5d7c0d2270 2024-03-24T00:31:44.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.801+0000 7f5d78c00640 1 -- 172.21.15.40:0/844868018 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5d680bd020 con 0x7f5d7c0d2270 2024-03-24T00:31:44.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.801+0000 7f5d78c00640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5d5c067690 0x7f5d5c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:44.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.801+0000 7f5d78c00640 1 -- 172.21.15.40:0/844868018 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d680a8020 con 0x7f5d7c0d2270 2024-03-24T00:31:44.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.801+0000 7f5d81000640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5d5c067690 0x7f5d5c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:44.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.801+0000 7f5d81000640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5d5c067690 0x7f5d5c069b50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5d70007bc0 tx=0x7f5d70007b10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:44.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.805+0000 7f5d78c00640 1 -- 172.21.15.40:0/844868018 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5d68016090 con 0x7f5d7c0d2270 2024-03-24T00:31:44.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.965+0000 7f5d83a00640 1 -- 172.21.15.40:0/844868018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5d7c070910 con 0x7f5d7c0d2270 2024-03-24T00:31:44.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.965+0000 7f5d78c00640 1 -- 172.21.15.40:0/844868018 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5d680f0800 con 0x7f5d7c0d2270 2024-03-24T00:31:44.972 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:44.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.969+0000 7f5d83a00640 1 -- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5d5c067690 msgr2=0x7f5d5c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.969+0000 7f5d83a00640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5d5c067690 0x7f5d5c069b50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5d70007bc0 tx=0x7f5d70007b10 comp rx=0 tx=0).stop 2024-03-24T00:31:44.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.969+0000 7f5d83a00640 1 -- 172.21.15.40:0/844868018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d7c0d2270 msgr2=0x7f5d7c1fa580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:44.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.969+0000 7f5d83a00640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d7c0d2270 0x7f5d7c1fa580 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5d680ad040 tx=0x7f5d6801acb0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.969+0000 7f5d83a00640 1 -- 172.21.15.40:0/844868018 shutdown_connections 2024-03-24T00:31:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.969+0000 7f5d83a00640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5d5c067690 0x7f5d5c069b50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.969+0000 7f5d83a00640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d7c15df80 0x7f5d7c15b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.969+0000 7f5d83a00640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d7c0d2270 0x7f5d7c1fa580 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.969+0000 7f5d83a00640 1 --2- 172.21.15.40:0/844868018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d7c06c950 0x7f5d7c14f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.969+0000 7f5d83a00640 1 -- 172.21.15.40:0/844868018 >> 172.21.15.40:0/844868018 conn(0x7f5d7c06ecf0 msgr2=0x7f5d7c0b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.969+0000 7f5d83a00640 1 -- 172.21.15.40:0/844868018 shutdown_connections 2024-03-24T00:31:44.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:44.969+0000 7f5d83a00640 1 -- 172.21.15.40:0/844868018 wait complete. 2024-03-24T00:31:44.994 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:45.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.133+0000 7f90b7400640 1 Processor -- start 2024-03-24T00:31:45.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.133+0000 7f90b7400640 1 -- start start 2024-03-24T00:31:45.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.133+0000 7f90b7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90b015e110 0x7f90b0161b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.133+0000 7f90b7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f90b00b9880 0x7f90b00ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.133+0000 7f90b7400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b0162460 con 0x7f90b00cf020 2024-03-24T00:31:45.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.133+0000 7f90b7400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b00cf490 con 0x7f90b015e110 2024-03-24T00:31:45.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.133+0000 7f90b7400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b0162740 con 0x7f90b00b9880 2024-03-24T00:31:45.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.133+0000 7f90b4a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90b015e110 0x7f90b0161b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.133+0000 7f90b5400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f90b00cf020 0x7f90b00b5890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56382/0 (socket says 172.21.15.40:56382) 2024-03-24T00:31:45.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.133+0000 7f90b4a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90b015e110 0x7f90b0161b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54932/0 (socket says 172.21.15.40:54932) 2024-03-24T00:31:45.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.133+0000 7f90b5400640 1 -- 172.21.15.40:0/1630368437 learned_addr learned my addr 172.21.15.40:0/1630368437 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:45.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.133+0000 7f90afe00640 1 --2- 172.21.15.40:0/1630368437 >> v2:172.21.15.53:3300/0 conn(0x7f90b00b9880 0x7f90b00ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b4a00640 1 -- 172.21.15.40:0/1630368437 >> v2:172.21.15.53:3300/0 conn(0x7f90b00b9880 msgr2=0x7f90b00ceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:45.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b4a00640 1 --2- 172.21.15.40:0/1630368437 >> v2:172.21.15.53:3300/0 conn(0x7f90b00b9880 0x7f90b00ceae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b4a00640 1 -- 172.21.15.40:0/1630368437 >> v1:172.21.15.40:6789/0 conn(0x7f90b00cf020 legacy=0x7f90b00b5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b4a00640 1 -- 172.21.15.40:0/1630368437 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b0162a20 con 0x7f90b015e110 2024-03-24T00:31:45.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b4a00640 1 --2- 172.21.15.40:0/1630368437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b015e110 0x7f90b0161b60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f90b00d17e0 tx=0x7f90a401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a506ba69255853d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:45.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90af400640 1 -- 172.21.15.40:0/1630368437 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40ab650 con 0x7f90b015e110 2024-03-24T00:31:45.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90af400640 1 -- 172.21.15.40:0/1630368437 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a40b0070 con 0x7f90b015e110 2024-03-24T00:31:45.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90af400640 1 -- 172.21.15.40:0/1630368437 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40aba10 con 0x7f90b015e110 2024-03-24T00:31:45.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b7400640 1 -- 172.21.15.40:0/1630368437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b015e110 msgr2=0x7f90b0161b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b7400640 1 --2- 172.21.15.40:0/1630368437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b015e110 0x7f90b0161b60 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f90b00d17e0 tx=0x7f90a401c450 comp rx=0 tx=0).stop 2024-03-24T00:31:45.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b7400640 1 -- 172.21.15.40:0/1630368437 shutdown_connections 2024-03-24T00:31:45.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b7400640 1 --2- 172.21.15.40:0/1630368437 >> v2:172.21.15.53:3300/0 conn(0x7f90b00b9880 0x7f90b00ceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b7400640 1 --2- 172.21.15.40:0/1630368437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b015e110 0x7f90b0161b60 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b7400640 1 -- 172.21.15.40:0/1630368437 >> 172.21.15.40:0/1630368437 conn(0x7f90b0066e80 msgr2=0x7f90b0067260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:45.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b7400640 1 -- 172.21.15.40:0/1630368437 shutdown_connections 2024-03-24T00:31:45.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.137+0000 7f90b7400640 1 -- 172.21.15.40:0/1630368437 wait complete. 2024-03-24T00:31:45.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b7400640 1 Processor -- start 2024-03-24T00:31:45.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b7400640 1 -- start start 2024-03-24T00:31:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b00b9880 0x7f90b00cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b7400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90b00cf020 0x7f90b01581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90b015e110 0x7f90b00cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b7400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b016b080 con 0x7f90b00b9880 2024-03-24T00:31:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b7400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b016af00 con 0x7f90b015e110 2024-03-24T00:31:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b7400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b016b200 con 0x7f90b00cf020 2024-03-24T00:31:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b00b9880 0x7f90b00cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b00b9880 0x7f90b00cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54934/0 (socket says 172.21.15.40:54934) 2024-03-24T00:31:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b4a00640 1 -- 172.21.15.40:0/3392905013 learned_addr learned my addr 172.21.15.40:0/3392905013 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b5400640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90b015e110 0x7f90b00cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90afe00640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90b00cf020 0x7f90b01581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b4a00640 1 -- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90b00cf020 msgr2=0x7f90b01581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b4a00640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90b00cf020 0x7f90b01581b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b4a00640 1 -- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90b015e110 msgr2=0x7f90b00cb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b4a00640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90b015e110 0x7f90b00cb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b4a00640 1 -- 172.21.15.40:0/3392905013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b0156950 con 0x7f90b00b9880 2024-03-24T00:31:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90b4a00640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b00b9880 0x7f90b00cda90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f90a401ab00 tx=0x7f90a401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.141+0000 7f90acc00640 1 -- 172.21.15.40:0/3392905013 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40bf610 con 0x7f90b00b9880 2024-03-24T00:31:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.145+0000 7f90acc00640 1 -- 172.21.15.40:0/3392905013 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a40bf7b0 con 0x7f90b00b9880 2024-03-24T00:31:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.145+0000 7f90acc00640 1 -- 172.21.15.40:0/3392905013 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a40bfb60 con 0x7f90b00b9880 2024-03-24T00:31:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.145+0000 7f90b7400640 1 -- 172.21.15.40:0/3392905013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b0156c30 con 0x7f90b00b9880 2024-03-24T00:31:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.145+0000 7f90b7400640 1 -- 172.21.15.40:0/3392905013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b01571f0 con 0x7f90b00b9880 2024-03-24T00:31:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.145+0000 7f90b7400640 1 -- 172.21.15.40:0/3392905013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b0068170 con 0x7f90b00b9880 2024-03-24T00:31:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.145+0000 7f90acc00640 1 -- 172.21.15.40:0/3392905013 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f90a40bd020 con 0x7f90b00b9880 2024-03-24T00:31:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.145+0000 7f90acc00640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9090067660 0x7f9090069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.145+0000 7f90acc00640 1 -- 172.21.15.40:0/3392905013 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90a4120470 con 0x7f90b00b9880 2024-03-24T00:31:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.145+0000 7f90afe00640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9090067660 0x7f9090069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.145+0000 7f90afe00640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9090067660 0x7f9090069b20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f90b0156880 tx=0x7f909c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:45.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.149+0000 7f90acc00640 1 -- 172.21.15.40:0/3392905013 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f90a4016070 con 0x7f90b00b9880 2024-03-24T00:31:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 -- 172.21.15.40:0/3392905013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f90b00b3550 con 0x7f90b00b9880 2024-03-24T00:31:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90acc00640 1 -- 172.21.15.40:0/3392905013 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f90a40f0a00 con 0x7f90b00b9880 2024-03-24T00:31:45.337 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:45.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 -- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9090067660 msgr2=0x7f9090069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9090067660 0x7f9090069b20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f90b0156880 tx=0x7f909c008040 comp rx=0 tx=0).stop 2024-03-24T00:31:45.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 -- 172.21.15.40:0/3392905013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b00b9880 msgr2=0x7f90b00cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b00b9880 0x7f90b00cda90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f90a401ab00 tx=0x7f90a401ac90 comp rx=0 tx=0).stop 2024-03-24T00:31:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 -- 172.21.15.40:0/3392905013 shutdown_connections 2024-03-24T00:31:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9090067660 0x7f9090069b20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90b015e110 0x7f90b00cb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90b00cf020 0x7f90b01581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 --2- 172.21.15.40:0/3392905013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90b00b9880 0x7f90b00cda90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 -- 172.21.15.40:0/3392905013 >> 172.21.15.40:0/3392905013 conn(0x7f90b0066e80 msgr2=0x7f90b00b2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 -- 172.21.15.40:0/3392905013 shutdown_connections 2024-03-24T00:31:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.333+0000 7f90b7400640 1 -- 172.21.15.40:0/3392905013 wait complete. 2024-03-24T00:31:45.358 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:31:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.497+0000 7f68f3600640 1 Processor -- start 2024-03-24T00:31:45.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.497+0000 7f68f3600640 1 -- start start 2024-03-24T00:31:45.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.497+0000 7f68f3600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f68ec15e3a0 0x7f68ec0ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68f3600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ec0cfaa0 con 0x7f68ec15dfc0 2024-03-24T00:31:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68f3600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68ec0cfd80 con 0x7f68ec0cdaa0 2024-03-24T00:31:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68f3600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ec0cf870 con 0x7f68ec15e3a0 2024-03-24T00:31:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68e3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f68ec15e3a0 0x7f68ec0ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68f0c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f68ec15dfc0 0x7f68ec0cd390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56398/0 (socket says 172.21.15.40:56398) 2024-03-24T00:31:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68e3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f68ec15e3a0 0x7f68ec0ce690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54938/0 (socket says 172.21.15.40:54938) 2024-03-24T00:31:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68f0c00640 1 -- 172.21.15.40:0/2844048921 learned_addr learned my addr 172.21.15.40:0/2844048921 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:45.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68e3e00640 1 -- 172.21.15.40:0/2844048921 >> v1:172.21.15.53:6789/0 conn(0x7f68ec0cdaa0 legacy=0x7f68ec167fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68e3e00640 1 -- 172.21.15.40:0/2844048921 >> v1:172.21.15.40:6789/0 conn(0x7f68ec15dfc0 legacy=0x7f68ec0cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68e3e00640 1 -- 172.21.15.40:0/2844048921 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec0d0060 con 0x7f68ec15e3a0 2024-03-24T00:31:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68ebe00640 1 -- 172.21.15.40:0/2844048921 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748686835 0 0) 0x7f68ec0cfaa0 con 0x7f68ec15dfc0 2024-03-24T00:31:45.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68e3e00640 1 --2- 172.21.15.40:0/2844048921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68ec15e3a0 0x7f68ec0ce690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f68d40097c0 tx=0x7f68d401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e8768fd5e0694c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:45.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.501+0000 7f68ebe00640 1 -- 172.21.15.40:0/2844048921 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d40b0070 con 0x7f68ec15e3a0 2024-03-24T00:31:45.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68ebe00640 1 -- 172.21.15.40:0/2844048921 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d40ab600 con 0x7f68ec15e3a0 2024-03-24T00:31:45.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68ebe00640 1 -- 172.21.15.40:0/2844048921 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d40ab9d0 con 0x7f68ec15e3a0 2024-03-24T00:31:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 -- 172.21.15.40:0/2844048921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68ec15e3a0 msgr2=0x7f68ec0ce690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 --2- 172.21.15.40:0/2844048921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68ec15e3a0 0x7f68ec0ce690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f68d40097c0 tx=0x7f68d401c380 comp rx=0 tx=0).stop 2024-03-24T00:31:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 -- 172.21.15.40:0/2844048921 shutdown_connections 2024-03-24T00:31:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 --2- 172.21.15.40:0/2844048921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68ec15e3a0 0x7f68ec0ce690 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 -- 172.21.15.40:0/2844048921 >> 172.21.15.40:0/2844048921 conn(0x7f68ec0b37f0 msgr2=0x7f68ec0b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 -- 172.21.15.40:0/2844048921 shutdown_connections 2024-03-24T00:31:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 -- 172.21.15.40:0/2844048921 wait complete. 2024-03-24T00:31:45.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 Processor -- start 2024-03-24T00:31:45.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 -- start start 2024-03-24T00:31:45.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ec0cdaa0 0x7f68ec0b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68ec15dfc0 0x7f68ec157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68ec15e3a0 0x7f68ec158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec16c100 con 0x7f68ec15dfc0 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec0d0d50 con 0x7f68ec0cdaa0 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f3600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ec16c280 con 0x7f68ec15e3a0 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68ec15dfc0 0x7f68ec157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68ec15dfc0 0x7f68ec157bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54950/0 (socket says 172.21.15.40:54950) 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68e3e00640 1 -- 172.21.15.40:0/3015824 learned_addr learned my addr 172.21.15.40:0/3015824 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f0c00640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ec0cdaa0 0x7f68ec0b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.505+0000 7f68f1600640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68ec15e3a0 0x7f68ec158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68e3e00640 1 -- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68ec15e3a0 msgr2=0x7f68ec158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68e3e00640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68ec15e3a0 0x7f68ec158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68e3e00640 1 -- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ec0cdaa0 msgr2=0x7f68ec0b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68e3e00640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ec0cdaa0 0x7f68ec0b0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68e3e00640 1 -- 172.21.15.40:0/3015824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec0b6480 con 0x7f68ec15dfc0 2024-03-24T00:31:45.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68e3e00640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68ec15dfc0 0x7f68ec157bf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f68d40097c0 tx=0x7f68d401abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:45.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68e9600640 1 -- 172.21.15.40:0/3015824 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d40bc070 con 0x7f68ec15dfc0 2024-03-24T00:31:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68e9600640 1 -- 172.21.15.40:0/3015824 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68d40b1040 con 0x7f68ec15dfc0 2024-03-24T00:31:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68e9600640 1 -- 172.21.15.40:0/3015824 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68d40b0050 con 0x7f68ec15dfc0 2024-03-24T00:31:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68f3600640 1 -- 172.21.15.40:0/3015824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ec0b6760 con 0x7f68ec15dfc0 2024-03-24T00:31:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68f3600640 1 -- 172.21.15.40:0/3015824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ec0b6d20 con 0x7f68ec15dfc0 2024-03-24T00:31:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68f3600640 1 -- 172.21.15.40:0/3015824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ec0d2c00 con 0x7f68ec15dfc0 2024-03-24T00:31:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68e9600640 1 -- 172.21.15.40:0/3015824 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f68d4004230 con 0x7f68ec15dfc0 2024-03-24T00:31:45.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68e9600640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68c8067660 0x7f68c8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.509+0000 7f68e9600640 1 -- 172.21.15.40:0/3015824 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68d411f3b0 con 0x7f68ec15dfc0 2024-03-24T00:31:45.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.513+0000 7f68f0c00640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68c8067660 0x7f68c8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.513+0000 7f68f0c00640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68c8067660 0x7f68c8069b20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f68dc000f70 tx=0x7f68dc007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:45.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.513+0000 7f68e9600640 1 -- 172.21.15.40:0/3015824 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f68d4016090 con 0x7f68ec15dfc0 2024-03-24T00:31:45.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.689+0000 7f68f3600640 1 -- 172.21.15.40:0/3015824 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f68ec068170 con 0x7f68ec15dfc0 2024-03-24T00:31:45.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.689+0000 7f68e9600640 1 -- 172.21.15.40:0/3015824 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7f68d40efa70 con 0x7f68ec15dfc0 2024-03-24T00:31:45.694 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:31:45.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.693+0000 7f68f3600640 1 -- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68c8067660 msgr2=0x7f68c8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.693+0000 7f68f3600640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68c8067660 0x7f68c8069b20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f68dc000f70 tx=0x7f68dc007240 comp rx=0 tx=0).stop 2024-03-24T00:31:45.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.693+0000 7f68f3600640 1 -- 172.21.15.40:0/3015824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68ec15dfc0 msgr2=0x7f68ec157bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.693+0000 7f68f3600640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68ec15dfc0 0x7f68ec157bf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f68d40097c0 tx=0x7f68d401abc0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.693+0000 7f68f3600640 1 -- 172.21.15.40:0/3015824 shutdown_connections 2024-03-24T00:31:45.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.693+0000 7f68f3600640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68c8067660 0x7f68c8069b20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.693+0000 7f68f3600640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68ec15e3a0 0x7f68ec158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.693+0000 7f68f3600640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68ec15dfc0 0x7f68ec157bf0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.693+0000 7f68f3600640 1 --2- 172.21.15.40:0/3015824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ec0cdaa0 0x7f68ec0b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.693+0000 7f68f3600640 1 -- 172.21.15.40:0/3015824 >> 172.21.15.40:0/3015824 conn(0x7f68ec0b37f0 msgr2=0x7f68ec06ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:45.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.693+0000 7f68f3600640 1 -- 172.21.15.40:0/3015824 shutdown_connections 2024-03-24T00:31:45.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.693+0000 7f68f3600640 1 -- 172.21.15.40:0/3015824 wait complete. 2024-03-24T00:31:45.716 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:31:45.716 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:31:45.726 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:45.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.901+0000 7fbadfa00640 1 Processor -- start 2024-03-24T00:31:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadfa00640 1 -- start start 2024-03-24T00:31:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadfa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbad80cfdf0 0x7fbad80d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadfa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbad80b43a0 0x7fbad80d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadfa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbad815f090 0x7fbad815f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadfa00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad80b2bb0 con 0x7fbad815f090 2024-03-24T00:31:45.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadfa00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad80b2fe0 con 0x7fbad80b43a0 2024-03-24T00:31:45.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadfa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80b32a0 con 0x7fbad80cfdf0 2024-03-24T00:31:45.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadda00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbad815f090 0x7fbad815f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadda00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbad815f090 0x7fbad815f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54964/0 (socket says 172.21.15.40:54964) 2024-03-24T00:31:45.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadda00640 1 -- 172.21.15.40:0/1650694134 learned_addr learned my addr 172.21.15.40:0/1650694134 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:45.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadd000640 1 --2- 172.21.15.40:0/1650694134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbad80cfdf0 0x7fbad80d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbacfe00640 1 --2- 172.21.15.40:0/1650694134 >> v2:172.21.15.53:3300/0 conn(0x7fbad80b43a0 0x7fbad80d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadda00640 1 -- 172.21.15.40:0/1650694134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbad80cfdf0 msgr2=0x7fbad80d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadda00640 1 --2- 172.21.15.40:0/1650694134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbad80cfdf0 0x7fbad80d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadda00640 1 -- 172.21.15.40:0/1650694134 >> v2:172.21.15.53:3300/0 conn(0x7fbad80b43a0 msgr2=0x7fbad80d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadda00640 1 --2- 172.21.15.40:0/1650694134 >> v2:172.21.15.53:3300/0 conn(0x7fbad80b43a0 0x7fbad80d0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.905+0000 7fbadda00640 1 -- 172.21.15.40:0/1650694134 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad80af710 con 0x7fbad815f090 2024-03-24T00:31:45.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbadda00640 1 --2- 172.21.15.40:0/1650694134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad815f090 0x7fbad815f4e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbad80723b0 tx=0x7fbad001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7024f43553a7b33e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:45.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbad7e00640 1 -- 172.21.15.40:0/1650694134 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad00a5870 con 0x7fbad815f090 2024-03-24T00:31:45.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbad7e00640 1 -- 172.21.15.40:0/1650694134 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbad00a5a10 con 0x7fbad815f090 2024-03-24T00:31:45.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbad7e00640 1 -- 172.21.15.40:0/1650694134 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad00a5de0 con 0x7fbad815f090 2024-03-24T00:31:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbadfa00640 1 -- 172.21.15.40:0/1650694134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad815f090 msgr2=0x7fbad815f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbadfa00640 1 --2- 172.21.15.40:0/1650694134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad815f090 0x7fbad815f4e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbad80723b0 tx=0x7fbad001c9f0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbadfa00640 1 -- 172.21.15.40:0/1650694134 shutdown_connections 2024-03-24T00:31:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbadfa00640 1 --2- 172.21.15.40:0/1650694134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad815f090 0x7fbad815f4e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbadfa00640 1 --2- 172.21.15.40:0/1650694134 >> v2:172.21.15.53:3300/0 conn(0x7fbad80b43a0 0x7fbad80d0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbadfa00640 1 --2- 172.21.15.40:0/1650694134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbad80cfdf0 0x7fbad80d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbadfa00640 1 -- 172.21.15.40:0/1650694134 >> 172.21.15.40:0/1650694134 conn(0x7fbad80b3880 msgr2=0x7fbad80b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbadfa00640 1 -- 172.21.15.40:0/1650694134 shutdown_connections 2024-03-24T00:31:45.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.909+0000 7fbadfa00640 1 -- 172.21.15.40:0/1650694134 wait complete. 2024-03-24T00:31:45.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadfa00640 1 Processor -- start 2024-03-24T00:31:45.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadfa00640 1 -- start start 2024-03-24T00:31:45.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadfa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbad80b43a0 0x7fbad80c8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadfa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbad80cfdf0 0x7fbad80cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadfa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad815f090 0x7fbad80cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadfa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80b2020 con 0x7fbad815f090 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadfa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80b1ea0 con 0x7fbad80cfdf0 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadfa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad80b21a0 con 0x7fbad80b43a0 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadda00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad815f090 0x7fbad80cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbacfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbad80cfdf0 0x7fbad80cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadd000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbad80b43a0 0x7fbad80c8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadd000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbad80b43a0 0x7fbad80c8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:48866/0 (socket says 172.21.15.40:48866) 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadd000640 1 -- 172.21.15.40:0/1358157835 learned_addr learned my addr 172.21.15.40:0/1358157835 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbacfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbad80cfdf0 0x7fbad80cb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:41494/0 (socket says 172.21.15.40:41494) 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadd000640 1 -- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbad80cfdf0 msgr2=0x7fbad80cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadd000640 1 --2- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbad80cfdf0 0x7fbad80cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadd000640 1 -- 172.21.15.40:0/1358157835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad815f090 msgr2=0x7fbad80cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:45.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadd000640 1 --2- 172.21.15.40:0/1358157835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad815f090 0x7fbad80cb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:45.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadd000640 1 -- 172.21.15.40:0/1358157835 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad80d1880 con 0x7fbad80b43a0 2024-03-24T00:31:45.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadd000640 1 --2- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbad80b43a0 0x7fbad80c8d60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbac800e930 tx=0x7fbac800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbad5600640 1 -- 172.21.15.40:0/1358157835 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8017070 con 0x7fbad80b43a0 2024-03-24T00:31:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbad5600640 1 -- 172.21.15.40:0/1358157835 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac800ca00 con 0x7fbad80b43a0 2024-03-24T00:31:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadfa00640 1 -- 172.21.15.40:0/1358157835 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad80b75c0 con 0x7fbad80b43a0 2024-03-24T00:31:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbad5600640 1 -- 172.21.15.40:0/1358157835 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac800cdb0 con 0x7fbad80b43a0 2024-03-24T00:31:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.913+0000 7fbadfa00640 1 -- 172.21.15.40:0/1358157835 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad8157d70 con 0x7fbad80b43a0 2024-03-24T00:31:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.917+0000 7fbadfa00640 1 -- 172.21.15.40:0/1358157835 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad8068170 con 0x7fbad80b43a0 2024-03-24T00:31:45.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.917+0000 7fbad5600640 1 -- 172.21.15.40:0/1358157835 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbac801a070 con 0x7fbad80b43a0 2024-03-24T00:31:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.917+0000 7fbad5600640 1 --2- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbab4067660 0x7fbab4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.917+0000 7fbad5600640 1 -- 172.21.15.40:0/1358157835 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbac8085860 con 0x7fbad80b43a0 2024-03-24T00:31:45.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.917+0000 7fbacfe00640 1 --2- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbab4067660 0x7fbab4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:45.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.921+0000 7fbacfe00640 1 --2- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbab4067660 0x7fbab4069b20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbad8157150 tx=0x7fbac0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:45.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:45.925+0000 7fbad5600640 1 -- 172.21.15.40:0/1358157835 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbac8055f40 con 0x7fbad80b43a0 2024-03-24T00:31:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.121+0000 7fbadfa00640 1 -- 172.21.15.40:0/1358157835 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbad80b3fa0 con 0x7fbad80b43a0 2024-03-24T00:31:46.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.121+0000 7fbad5600640 1 -- 172.21.15.40:0/1358157835 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16545 (secure 0 0 0) 0x7fbac8059fe0 con 0x7fbad80b43a0 2024-03-24T00:31:46.126 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:46.127 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":20,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:01.034917+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:46.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.125+0000 7fbadfa00640 1 -- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbab4067660 msgr2=0x7fbab4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.125+0000 7fbadfa00640 1 --2- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbab4067660 0x7fbab4069b20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbad8157150 tx=0x7fbac0009190 comp rx=0 tx=0).stop 2024-03-24T00:31:46.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.125+0000 7fbadfa00640 1 -- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbad80b43a0 msgr2=0x7fbad80c8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.125+0000 7fbadfa00640 1 --2- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbad80b43a0 0x7fbad80c8d60 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbac800e930 tx=0x7fbac800edf0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.125+0000 7fbadfa00640 1 -- 172.21.15.40:0/1358157835 shutdown_connections 2024-03-24T00:31:46.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.125+0000 7fbadfa00640 1 --2- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbab4067660 0x7fbab4069b20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.125+0000 7fbadfa00640 1 --2- 172.21.15.40:0/1358157835 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbad815f090 0x7fbad80cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.125+0000 7fbadfa00640 1 --2- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbad80cfdf0 0x7fbad80cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.125+0000 7fbadfa00640 1 --2- 172.21.15.40:0/1358157835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbad80b43a0 0x7fbad80c8d60 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.125+0000 7fbadfa00640 1 -- 172.21.15.40:0/1358157835 >> 172.21.15.40:0/1358157835 conn(0x7fbad80b3880 msgr2=0x7fbad806ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:46.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.125+0000 7fbadfa00640 1 -- 172.21.15.40:0/1358157835 shutdown_connections 2024-03-24T00:31:46.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.125+0000 7fbadfa00640 1 -- 172.21.15.40:0/1358157835 wait complete. 2024-03-24T00:31:46.147 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:46.148 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:46.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.285+0000 7fd76da00640 1 Processor -- start 2024-03-24T00:31:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.285+0000 7fd76da00640 1 -- start start 2024-03-24T00:31:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd76da00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd76815b960 0x7fd768070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd76da00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd7680d16f0 0x7fd7680712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd76da00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76815f970 con 0x7fd76815bd40 2024-03-24T00:31:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd76da00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd768160580 con 0x7fd7680d16f0 2024-03-24T00:31:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd76da00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd76815fc50 con 0x7fd76815b960 2024-03-24T00:31:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd767400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd7680d16f0 0x7fd7680712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd75e000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd76815bd40 0x7fd7680d0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56410/0 (socket says 172.21.15.40:56410) 2024-03-24T00:31:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd75e000640 1 -- 172.21.15.40:0/4114277629 learned_addr learned my addr 172.21.15.40:0/4114277629 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd766a00640 1 --2- 172.21.15.40:0/4114277629 >> v2:172.21.15.53:3300/0 conn(0x7fd76815b960 0x7fd768070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd767400640 1 -- 172.21.15.40:0/4114277629 >> v2:172.21.15.53:3300/0 conn(0x7fd76815b960 msgr2=0x7fd768070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd767400640 1 --2- 172.21.15.40:0/4114277629 >> v2:172.21.15.53:3300/0 conn(0x7fd76815b960 0x7fd768070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd767400640 1 -- 172.21.15.40:0/4114277629 >> v1:172.21.15.40:6789/0 conn(0x7fd76815bd40 legacy=0x7fd7680d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd767400640 1 -- 172.21.15.40:0/4114277629 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76815ff30 con 0x7fd7680d16f0 2024-03-24T00:31:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.289+0000 7fd766000640 1 -- 172.21.15.40:0/4114277629 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583116124 0 0) 0x7fd76815f970 con 0x7fd76815bd40 2024-03-24T00:31:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd767400640 1 --2- 172.21.15.40:0/4114277629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7680d16f0 0x7fd7680712a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd7680d1b30 tx=0x7fd76001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9400b02b2283105 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd766000640 1 -- 172.21.15.40:0/4114277629 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7600b2070 con 0x7fd7680d16f0 2024-03-24T00:31:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd766000640 1 -- 172.21.15.40:0/4114277629 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7600a5900 con 0x7fd7680d16f0 2024-03-24T00:31:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd766000640 1 -- 172.21.15.40:0/4114277629 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7600a5cd0 con 0x7fd7680d16f0 2024-03-24T00:31:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd76da00640 1 -- 172.21.15.40:0/4114277629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7680d16f0 msgr2=0x7fd7680712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd76da00640 1 --2- 172.21.15.40:0/4114277629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7680d16f0 0x7fd7680712a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd7680d1b30 tx=0x7fd76001c930 comp rx=0 tx=0).stop 2024-03-24T00:31:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd76da00640 1 -- 172.21.15.40:0/4114277629 shutdown_connections 2024-03-24T00:31:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd76da00640 1 --2- 172.21.15.40:0/4114277629 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7680d16f0 0x7fd7680712a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd76da00640 1 --2- 172.21.15.40:0/4114277629 >> v2:172.21.15.53:3300/0 conn(0x7fd76815b960 0x7fd768070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd76da00640 1 -- 172.21.15.40:0/4114277629 >> 172.21.15.40:0/4114277629 conn(0x7fd76806ecf0 msgr2=0x7fd76806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd76da00640 1 -- 172.21.15.40:0/4114277629 shutdown_connections 2024-03-24T00:31:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd76da00640 1 -- 172.21.15.40:0/4114277629 wait complete. 2024-03-24T00:31:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd76da00640 1 Processor -- start 2024-03-24T00:31:46.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.293+0000 7fd76da00640 1 -- start start 2024-03-24T00:31:46.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd76da00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7680d16f0 0x7fd7680c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:46.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd76da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd76815b960 0x7fd7681453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd76da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76815bd40 0x7fd768145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd76da00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7680ce580 con 0x7fd76815bd40 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd76da00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7680ce400 con 0x7fd76815b960 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd76da00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7680ce700 con 0x7fd7680d16f0 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd767400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76815bd40 0x7fd768145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd767400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76815bd40 0x7fd768145910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54988/0 (socket says 172.21.15.40:54988) 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd767400640 1 -- 172.21.15.40:0/1612154554 learned_addr learned my addr 172.21.15.40:0/1612154554 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd766a00640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7680d16f0 0x7fd7680c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd75e000640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd76815b960 0x7fd7681453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd767400640 1 -- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7680d16f0 msgr2=0x7fd7680c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd767400640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7680d16f0 0x7fd7680c3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd767400640 1 -- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd76815b960 msgr2=0x7fd7681453d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd767400640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd76815b960 0x7fd7681453d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd767400640 1 -- 172.21.15.40:0/1612154554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7680b3370 con 0x7fd76815bd40 2024-03-24T00:31:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd75e000640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd76815b960 0x7fd7681453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd767400640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76815bd40 0x7fd768145910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd7600b0040 tx=0x7fd7600196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd75f400640 1 -- 172.21.15.40:0/1612154554 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7600b2040 con 0x7fd76815bd40 2024-03-24T00:31:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd75f400640 1 -- 172.21.15.40:0/1612154554 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7600a5700 con 0x7fd76815bd40 2024-03-24T00:31:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd75f400640 1 -- 172.21.15.40:0/1612154554 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7600ada70 con 0x7fd76815bd40 2024-03-24T00:31:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd76da00640 1 -- 172.21.15.40:0/1612154554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76815c650 con 0x7fd76815bd40 2024-03-24T00:31:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd76da00640 1 -- 172.21.15.40:0/1612154554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7681524d0 con 0x7fd76815bd40 2024-03-24T00:31:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd76da00640 1 -- 172.21.15.40:0/1612154554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd768068170 con 0x7fd76815bd40 2024-03-24T00:31:46.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.297+0000 7fd75f400640 1 -- 172.21.15.40:0/1612154554 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd7600c1020 con 0x7fd76815bd40 2024-03-24T00:31:46.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.301+0000 7fd75f400640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd744067660 0x7fd744069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:46.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.301+0000 7fd75f400640 1 -- 172.21.15.40:0/1612154554 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7601205c0 con 0x7fd76815bd40 2024-03-24T00:31:46.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.301+0000 7fd766a00640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd744067660 0x7fd744069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:46.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.301+0000 7fd766a00640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd744067660 0x7fd744069b20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd758007a60 tx=0x7fd758008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:46.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.305+0000 7fd75f400640 1 -- 172.21.15.40:0/1612154554 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd760016520 con 0x7fd76815bd40 2024-03-24T00:31:46.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.537+0000 7fd76da00640 1 -- 172.21.15.40:0/1612154554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd7680710d0 con 0x7fd76815bd40 2024-03-24T00:31:46.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.537+0000 7fd75f400640 1 -- 172.21.15.40:0/1612154554 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd7600f12c0 con 0x7fd76815bd40 2024-03-24T00:31:46.543 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:46.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.541+0000 7fd76da00640 1 -- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd744067660 msgr2=0x7fd744069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.541+0000 7fd76da00640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd744067660 0x7fd744069b20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd758007a60 tx=0x7fd758008040 comp rx=0 tx=0).stop 2024-03-24T00:31:46.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.541+0000 7fd76da00640 1 -- 172.21.15.40:0/1612154554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76815bd40 msgr2=0x7fd768145910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.541+0000 7fd76da00640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76815bd40 0x7fd768145910 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd7600b0040 tx=0x7fd7600196d0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.541+0000 7fd76da00640 1 -- 172.21.15.40:0/1612154554 shutdown_connections 2024-03-24T00:31:46.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.541+0000 7fd76da00640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd744067660 0x7fd744069b20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.541+0000 7fd76da00640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd76815bd40 0x7fd768145910 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.541+0000 7fd76da00640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd76815b960 0x7fd7681453d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.541+0000 7fd76da00640 1 --2- 172.21.15.40:0/1612154554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7680d16f0 0x7fd7680c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.541+0000 7fd76da00640 1 -- 172.21.15.40:0/1612154554 >> 172.21.15.40:0/1612154554 conn(0x7fd76806ecf0 msgr2=0x7fd768150b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:46.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.541+0000 7fd76da00640 1 -- 172.21.15.40:0/1612154554 shutdown_connections 2024-03-24T00:31:46.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.541+0000 7fd76da00640 1 -- 172.21.15.40:0/1612154554 wait complete. 2024-03-24T00:31:46.564 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:46.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.701+0000 7fc43f800640 1 Processor -- start 2024-03-24T00:31:46.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc43f800640 1 -- start start 2024-03-24T00:31:46.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc43f800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc43815bb10 0x7fc43806f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:46.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc43f800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc438067c60 con 0x7fc43806f570 2024-03-24T00:31:46.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc43f800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43815e910 con 0x7fc438067600 2024-03-24T00:31:46.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc43f800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43806f9d0 con 0x7fc43815bb10 2024-03-24T00:31:46.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc437e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc43806f570 0x7fc438066ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56420/0 (socket says 172.21.15.40:56420) 2024-03-24T00:31:46.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc437e00640 1 -- 172.21.15.40:0/2984860097 learned_addr learned my addr 172.21.15.40:0/2984860097 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:46.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc43ce00640 1 --2- 172.21.15.40:0/2984860097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc43815bb10 0x7fc43806f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc437400640 1 -- 172.21.15.40:0/2984860097 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2373599890 0 0) 0x7fc438067c60 con 0x7fc43806f570 2024-03-24T00:31:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc437400640 1 -- 172.21.15.40:0/2984860097 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc420003530 con 0x7fc43806f570 2024-03-24T00:31:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc437400640 1 -- 172.21.15.40:0/2984860097 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333043149 0 0) 0x7fc43815e910 con 0x7fc438067600 2024-03-24T00:31:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc437400640 1 -- 172.21.15.40:0/2984860097 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc438067c60 con 0x7fc438067600 2024-03-24T00:31:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc43ce00640 1 -- 172.21.15.40:0/2984860097 >> v1:172.21.15.53:6789/0 conn(0x7fc438067600 legacy=0x7fc438161bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc43ce00640 1 -- 172.21.15.40:0/2984860097 >> v1:172.21.15.40:6789/0 conn(0x7fc43806f570 legacy=0x7fc438066ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.705+0000 7fc43ce00640 1 -- 172.21.15.40:0/2984860097 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43815ebf0 con 0x7fc43815bb10 2024-03-24T00:31:46.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.709+0000 7fc43ce00640 1 --2- 172.21.15.40:0/2984860097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc43815bb10 0x7fc43806f030 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0097f0 tx=0x7fc42c01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=caa087d5b7ece48c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:46.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.709+0000 7fc437400640 1 -- 172.21.15.40:0/2984860097 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c0b0070 con 0x7fc43815bb10 2024-03-24T00:31:46.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.709+0000 7fc437400640 1 -- 172.21.15.40:0/2984860097 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c0ab600 con 0x7fc43815bb10 2024-03-24T00:31:46.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.709+0000 7fc437400640 1 -- 172.21.15.40:0/2984860097 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42c0ab9d0 con 0x7fc43815bb10 2024-03-24T00:31:46.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.709+0000 7fc43f800640 1 -- 172.21.15.40:0/2984860097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc43815bb10 msgr2=0x7fc43806f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.709+0000 7fc43f800640 1 --2- 172.21.15.40:0/2984860097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc43815bb10 0x7fc43806f030 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0097f0 tx=0x7fc42c01c4e0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.709+0000 7fc43f800640 1 -- 172.21.15.40:0/2984860097 shutdown_connections 2024-03-24T00:31:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.709+0000 7fc43f800640 1 --2- 172.21.15.40:0/2984860097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc43815bb10 0x7fc43806f030 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.709+0000 7fc43f800640 1 -- 172.21.15.40:0/2984860097 >> 172.21.15.40:0/2984860097 conn(0x7fc4380b27c0 msgr2=0x7fc4380b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.709+0000 7fc43f800640 1 -- 172.21.15.40:0/2984860097 shutdown_connections 2024-03-24T00:31:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.709+0000 7fc43f800640 1 -- 172.21.15.40:0/2984860097 wait complete. 2024-03-24T00:31:46.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.713+0000 7fc43f800640 1 Processor -- start 2024-03-24T00:31:46.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.713+0000 7fc43f800640 1 -- start start 2024-03-24T00:31:46.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.713+0000 7fc43f800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc438067600 0x7fc4381faab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:46.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.713+0000 7fc43f800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc43806f570 0x7fc43815ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:46.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.713+0000 7fc43f800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc43815bb10 0x7fc4381604a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:46.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.713+0000 7fc43f800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43816c0e0 con 0x7fc43806f570 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.713+0000 7fc43f800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43816bf60 con 0x7fc438067600 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.713+0000 7fc43f800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43816c260 con 0x7fc43815bb10 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc437e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc43806f570 0x7fc43815ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc437e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc43806f570 0x7fc43815ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54996/0 (socket says 172.21.15.40:54996) 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc437e00640 1 -- 172.21.15.40:0/2075979532 learned_addr learned my addr 172.21.15.40:0/2075979532 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc43d800640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc43815bb10 0x7fc4381604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc43ce00640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc438067600 0x7fc4381faab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc437e00640 1 -- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc43815bb10 msgr2=0x7fc4381604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc437e00640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc43815bb10 0x7fc4381604a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc437e00640 1 -- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc438067600 msgr2=0x7fc4381faab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc437e00640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc438067600 0x7fc4381faab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc437e00640 1 -- 172.21.15.40:0/2075979532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc438160ad0 con 0x7fc43806f570 2024-03-24T00:31:46.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc437e00640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc43806f570 0x7fc43815ff60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc42400c450 tx=0x7fc42400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:46.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc434c00640 1 -- 172.21.15.40:0/2075979532 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424017070 con 0x7fc43806f570 2024-03-24T00:31:46.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc434c00640 1 -- 172.21.15.40:0/2075979532 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc424005cf0 con 0x7fc43806f570 2024-03-24T00:31:46.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc434c00640 1 -- 172.21.15.40:0/2075979532 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4240072b0 con 0x7fc43806f570 2024-03-24T00:31:46.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc43f800640 1 -- 172.21.15.40:0/2075979532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc438160db0 con 0x7fc43806f570 2024-03-24T00:31:46.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc43f800640 1 -- 172.21.15.40:0/2075979532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4381507d0 con 0x7fc43806f570 2024-03-24T00:31:46.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc43f800640 1 -- 172.21.15.40:0/2075979532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4381509d0 con 0x7fc43806f570 2024-03-24T00:31:46.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.717+0000 7fc434c00640 1 -- 172.21.15.40:0/2075979532 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc42401a070 con 0x7fc43806f570 2024-03-24T00:31:46.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.721+0000 7fc434c00640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc418067660 0x7fc418069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:46.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.721+0000 7fc434c00640 1 -- 172.21.15.40:0/2075979532 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc424084b70 con 0x7fc43806f570 2024-03-24T00:31:46.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.721+0000 7fc43ce00640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc418067660 0x7fc418069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:46.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.721+0000 7fc43ce00640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc418067660 0x7fc418069b20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0ad000 tx=0x7fc42c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:46.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.725+0000 7fc434c00640 1 -- 172.21.15.40:0/2075979532 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc4381509d0 con 0x7fc43806f570 2024-03-24T00:31:46.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.885+0000 7fc43f800640 1 -- 172.21.15.40:0/2075979532 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc438152dc0 con 0x7fc43806f570 2024-03-24T00:31:46.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.885+0000 7fc434c00640 1 -- 172.21.15.40:0/2075979532 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc4240551a0 con 0x7fc43806f570 2024-03-24T00:31:46.890 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.889+0000 7fc43f800640 1 -- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc418067660 msgr2=0x7fc418069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.889+0000 7fc43f800640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc418067660 0x7fc418069b20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc42c0ad000 tx=0x7fc42c0a6000 comp rx=0 tx=0).stop 2024-03-24T00:31:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.889+0000 7fc43f800640 1 -- 172.21.15.40:0/2075979532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc43806f570 msgr2=0x7fc43815ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.889+0000 7fc43f800640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc43806f570 0x7fc43815ff60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc42400c450 tx=0x7fc42400c910 comp rx=0 tx=0).stop 2024-03-24T00:31:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.889+0000 7fc43f800640 1 -- 172.21.15.40:0/2075979532 shutdown_connections 2024-03-24T00:31:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.889+0000 7fc43f800640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc418067660 0x7fc418069b20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.889+0000 7fc43f800640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc43815bb10 0x7fc4381604a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.889+0000 7fc43f800640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc43806f570 0x7fc43815ff60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.889+0000 7fc43f800640 1 --2- 172.21.15.40:0/2075979532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc438067600 0x7fc4381faab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:46.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.889+0000 7fc43f800640 1 -- 172.21.15.40:0/2075979532 >> 172.21.15.40:0/2075979532 conn(0x7fc4380b27c0 msgr2=0x7fc43806dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:46.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.889+0000 7fc43f800640 1 -- 172.21.15.40:0/2075979532 shutdown_connections 2024-03-24T00:31:46.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:46.889+0000 7fc43f800640 1 -- 172.21.15.40:0/2075979532 wait complete. 2024-03-24T00:31:46.911 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:47.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.053+0000 7f13de800640 1 Processor -- start 2024-03-24T00:31:47.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.053+0000 7f13de800640 1 -- start start 2024-03-24T00:31:47.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.053+0000 7f13de800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f13d815f5b0 0x7f13d80cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.053+0000 7f13de800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f13d80d0750 0x7f13d8165b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.053+0000 7f13de800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d80b33e0 con 0x7f13d80cc4c0 2024-03-24T00:31:47.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.053+0000 7f13de800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d80afa90 con 0x7f13d815f5b0 2024-03-24T00:31:47.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.053+0000 7f13de800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80b22f0 con 0x7f13d80d0750 2024-03-24T00:31:47.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.053+0000 7f13dd400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f13d815f5b0 0x7f13d80cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.053+0000 7f13dca00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f13d80cc4c0 0x7f13d80d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42460/0 (socket says 172.21.15.40:42460) 2024-03-24T00:31:47.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.053+0000 7f13dd400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f13d815f5b0 0x7f13d80cfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33764/0 (socket says 172.21.15.40:33764) 2024-03-24T00:31:47.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.053+0000 7f13dca00640 1 -- 172.21.15.40:0/3013678221 learned_addr learned my addr 172.21.15.40:0/3013678221 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:47.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.053+0000 7f13dde00640 1 --2- 172.21.15.40:0/3013678221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f13d80d0750 0x7f13d8165b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.057+0000 7f13dd400640 1 -- 172.21.15.40:0/3013678221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f13d80d0750 msgr2=0x7f13d8165b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:47.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.057+0000 7f13dd400640 1 --2- 172.21.15.40:0/3013678221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f13d80d0750 0x7f13d8165b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.057+0000 7f13dd400640 1 -- 172.21.15.40:0/3013678221 >> v1:172.21.15.40:6789/0 conn(0x7f13d80cc4c0 legacy=0x7f13d80d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.057+0000 7f13dd400640 1 -- 172.21.15.40:0/3013678221 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d80af440 con 0x7f13d815f5b0 2024-03-24T00:31:47.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.057+0000 7f13dd400640 1 --2- 172.21.15.40:0/3013678221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d815f5b0 0x7f13d80cfb00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f13cc0097c0 tx=0x7f13cc01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56498e38950b64f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.057+0000 7f13d7e00640 1 -- 172.21.15.40:0/3013678221 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc0b0070 con 0x7f13d815f5b0 2024-03-24T00:31:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.057+0000 7f13d7e00640 1 -- 172.21.15.40:0/3013678221 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13cc0ab600 con 0x7f13d815f5b0 2024-03-24T00:31:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.057+0000 7f13d7e00640 1 -- 172.21.15.40:0/3013678221 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc0ab9d0 con 0x7f13d815f5b0 2024-03-24T00:31:47.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.057+0000 7f13de800640 1 -- 172.21.15.40:0/3013678221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d815f5b0 msgr2=0x7f13d80cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.057+0000 7f13de800640 1 --2- 172.21.15.40:0/3013678221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d815f5b0 0x7f13d80cfb00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f13cc0097c0 tx=0x7f13cc01c330 comp rx=0 tx=0).stop 2024-03-24T00:31:47.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 -- 172.21.15.40:0/3013678221 shutdown_connections 2024-03-24T00:31:47.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 --2- 172.21.15.40:0/3013678221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f13d80d0750 0x7f13d8165b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 --2- 172.21.15.40:0/3013678221 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d815f5b0 0x7f13d80cfb00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 -- 172.21.15.40:0/3013678221 >> 172.21.15.40:0/3013678221 conn(0x7f13d80b3620 msgr2=0x7f13d80b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:47.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 -- 172.21.15.40:0/3013678221 shutdown_connections 2024-03-24T00:31:47.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 -- 172.21.15.40:0/3013678221 wait complete. 2024-03-24T00:31:47.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 Processor -- start 2024-03-24T00:31:47.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 -- start start 2024-03-24T00:31:47.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13d80cc4c0 0x7f13d80b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f13d80d0750 0x7f13d81f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d815f5b0 0x7f13d81f2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80b20b0 con 0x7f13d815f5b0 2024-03-24T00:31:47.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80b1f30 con 0x7f13d80cc4c0 2024-03-24T00:31:47.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13de800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80b2730 con 0x7f13d80d0750 2024-03-24T00:31:47.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13dde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d815f5b0 0x7f13d81f2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13dde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d815f5b0 0x7f13d81f2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33776/0 (socket says 172.21.15.40:33776) 2024-03-24T00:31:47.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13dde00640 1 -- 172.21.15.40:0/1796970768 learned_addr learned my addr 172.21.15.40:0/1796970768 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:47.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13dca00640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f13d80d0750 0x7f13d81f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13dd400640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13d80cc4c0 0x7f13d80b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13dde00640 1 -- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f13d80d0750 msgr2=0x7f13d81f4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13dde00640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f13d80d0750 0x7f13d81f4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13dde00640 1 -- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13d80cc4c0 msgr2=0x7f13d80b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13dde00640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13d80cc4c0 0x7f13d80b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13dde00640 1 -- 172.21.15.40:0/1796970768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d80b5930 con 0x7f13d815f5b0 2024-03-24T00:31:47.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13dde00640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d815f5b0 0x7f13d81f2120 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f13d000efc0 tx=0x7f13d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:47.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13d5600640 1 -- 172.21.15.40:0/1796970768 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0018020 con 0x7f13d815f5b0 2024-03-24T00:31:47.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13d5600640 1 -- 172.21.15.40:0/1796970768 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d0014480 con 0x7f13d815f5b0 2024-03-24T00:31:47.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.061+0000 7f13d5600640 1 -- 172.21.15.40:0/1796970768 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0014870 con 0x7f13d815f5b0 2024-03-24T00:31:47.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.065+0000 7f13de800640 1 -- 172.21.15.40:0/1796970768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d80b5c10 con 0x7f13d815f5b0 2024-03-24T00:31:47.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.065+0000 7f13de800640 1 -- 172.21.15.40:0/1796970768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d80b61d0 con 0x7f13d815f5b0 2024-03-24T00:31:47.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.065+0000 7f13de800640 1 -- 172.21.15.40:0/1796970768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d804f410 con 0x7f13d815f5b0 2024-03-24T00:31:47.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.065+0000 7f13d5600640 1 -- 172.21.15.40:0/1796970768 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f13d001b020 con 0x7f13d815f5b0 2024-03-24T00:31:47.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.065+0000 7f13d5600640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13b8067660 0x7f13b8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.065+0000 7f13d5600640 1 -- 172.21.15.40:0/1796970768 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13d00868c0 con 0x7f13d815f5b0 2024-03-24T00:31:47.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.065+0000 7f13dd400640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13b8067660 0x7f13b8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.069+0000 7f13dd400640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13b8067660 0x7f13b8069b20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f13cc0ad000 tx=0x7f13cc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:47.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.077+0000 7f13d5600640 1 -- 172.21.15.40:0/1796970768 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f13d0022020 con 0x7f13d815f5b0 2024-03-24T00:31:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.237+0000 7f13de800640 1 -- 172.21.15.40:0/1796970768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f13d80b6570 con 0x7f13d815f5b0 2024-03-24T00:31:47.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13d5600640 1 -- 172.21.15.40:0/1796970768 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f13d0010040 con 0x7f13d815f5b0 2024-03-24T00:31:47.244 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13de800640 1 -- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13b8067660 msgr2=0x7f13b8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13de800640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13b8067660 0x7f13b8069b20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f13cc0ad000 tx=0x7f13cc0a6000 comp rx=0 tx=0).stop 2024-03-24T00:31:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13de800640 1 -- 172.21.15.40:0/1796970768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d815f5b0 msgr2=0x7f13d81f2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13de800640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d815f5b0 0x7f13d81f2120 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f13d000efc0 tx=0x7f13d000c490 comp rx=0 tx=0).stop 2024-03-24T00:31:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13de800640 1 -- 172.21.15.40:0/1796970768 shutdown_connections 2024-03-24T00:31:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13de800640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f13b8067660 0x7f13b8069b20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13de800640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f13d815f5b0 0x7f13d81f2120 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13de800640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f13d80d0750 0x7f13d81f4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13de800640 1 --2- 172.21.15.40:0/1796970768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13d80cc4c0 0x7f13d80b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13de800640 1 -- 172.21.15.40:0/1796970768 >> 172.21.15.40:0/1796970768 conn(0x7f13d80b3620 msgr2=0x7f13d815b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13de800640 1 -- 172.21.15.40:0/1796970768 shutdown_connections 2024-03-24T00:31:47.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.241+0000 7f13de800640 1 -- 172.21.15.40:0/1796970768 wait complete. 2024-03-24T00:31:47.275 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:31:47.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.413+0000 7f72a7000640 1 Processor -- start 2024-03-24T00:31:47.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.413+0000 7f72a7000640 1 -- start start 2024-03-24T00:31:47.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.413+0000 7f72a7000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72a015f600 0x7f72a00b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.413+0000 7f72a7000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f72a00af170 0x7f72a0165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.413+0000 7f72a7000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a00d2c70 con 0x7f72a00d2830 2024-03-24T00:31:47.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.413+0000 7f72a7000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a00b3630 con 0x7f72a00af170 2024-03-24T00:31:47.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.413+0000 7f72a7000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a00b2740 con 0x7f72a015f600 2024-03-24T00:31:47.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.413+0000 7f729f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f72a00d2830 0x7f72a00aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42472/0 (socket says 172.21.15.40:42472) 2024-03-24T00:31:47.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.413+0000 7f729f400640 1 -- 172.21.15.40:0/4141744237 learned_addr learned my addr 172.21.15.40:0/4141744237 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931970850 0 0) 0x7f72a00d2c70 con 0x7f72a00d2830 2024-03-24T00:31:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7284003530 con 0x7f72a00d2830 2024-03-24T00:31:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2059520097 0 0) 0x7f7284003530 con 0x7f72a00d2830 2024-03-24T00:31:47.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72a015f600 msgr2=0x7f72a00b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 --2- 172.21.15.40:0/4141744237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72a015f600 0x7f72a00b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 >> v2:172.21.15.53:3300/0 conn(0x7f72a00af170 msgr2=0x7f72a0165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 --2- 172.21.15.40:0/4141744237 >> v2:172.21.15.53:3300/0 conn(0x7f72a00af170 0x7f72a0165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a00b1e50 con 0x7f72a00d2830 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f728c001680 con 0x7f72a00d2830 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 >> v1:172.21.15.40:6789/0 conn(0x7f72a00d2830 legacy=0x7f72a00aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 --2- 172.21.15.40:0/4141744237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72840039c0 0x7f7284003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 --2- 172.21.15.40:0/4141744237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7284004810 0x7f7284004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 --2- 172.21.15.40:0/4141744237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72a015f600 0x7f728400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a00b3630 con 0x7f72840039c0 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a00b2740 con 0x7f7284004810 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7284005620 con 0x7f72a015f600 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729fe00640 1 --2- 172.21.15.40:0/4141744237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72840039c0 0x7f7284003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729fe00640 1 -- 172.21.15.40:0/4141744237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72a015f600 msgr2=0x7f728400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729fe00640 1 --2- 172.21.15.40:0/4141744237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72a015f600 0x7f728400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729fe00640 1 -- 172.21.15.40:0/4141744237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7284004810 msgr2=0x7f7284004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729fe00640 1 --2- 172.21.15.40:0/4141744237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7284004810 0x7f7284004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.417+0000 7f729fe00640 1 -- 172.21.15.40:0/4141744237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7284003530 con 0x7f72840039c0 2024-03-24T00:31:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.421+0000 7f729fe00640 1 --2- 172.21.15.40:0/4141744237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72840039c0 0x7f7284003de0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7290009890 tx=0x7f729001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aee0cfa9975956d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.421+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72900b0070 con 0x7f72840039c0 2024-03-24T00:31:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.421+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72900ab620 con 0x7f72840039c0 2024-03-24T00:31:47.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.421+0000 7f729ea00640 1 -- 172.21.15.40:0/4141744237 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7290009460 con 0x7f72840039c0 2024-03-24T00:31:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.421+0000 7f72a7000640 1 -- 172.21.15.40:0/4141744237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72840039c0 msgr2=0x7f7284003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.421+0000 7f72a7000640 1 --2- 172.21.15.40:0/4141744237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72840039c0 0x7f7284003de0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7290009890 tx=0x7f729001c3c0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.421+0000 7f72a5000640 1 -- 172.21.15.40:0/4141744237 reap_dead start 2024-03-24T00:31:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.421+0000 7f72a7000640 1 -- 172.21.15.40:0/4141744237 shutdown_connections 2024-03-24T00:31:47.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.421+0000 7f72a7000640 1 -- 172.21.15.40:0/4141744237 >> 172.21.15.40:0/4141744237 conn(0x7f72a00d1d10 msgr2=0x7f72a00d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.421+0000 7f72a7000640 1 -- 172.21.15.40:0/4141744237 shutdown_connections 2024-03-24T00:31:47.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.421+0000 7f72a7000640 1 -- 172.21.15.40:0/4141744237 wait complete. 2024-03-24T00:31:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f72a7000640 1 Processor -- start 2024-03-24T00:31:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f72a7000640 1 -- start start 2024-03-24T00:31:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f72a7000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72840039c0 0x7f72a015b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f72a7000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72a00cabe0 0x7f72a00c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f72a7000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72a00c6ca0 0x7f72a00c7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f72a7000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a00b2fa0 con 0x7f72a00cabe0 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f72a7000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a00b23d0 con 0x7f72a00c6ca0 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f72a7000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a015f9e0 con 0x7f72840039c0 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f729fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72840039c0 0x7f72a015b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f729fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72840039c0 0x7f72a015b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:47116/0 (socket says 172.21.15.40:47116) 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f729fe00640 1 -- 172.21.15.40:0/3442340025 learned_addr learned my addr 172.21.15.40:0/3442340025 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f72a5000640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72a00c6ca0 0x7f72a00c7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.425+0000 7f729f400640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72a00cabe0 0x7f72a00c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f729fe00640 1 -- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72a00c6ca0 msgr2=0x7f72a00c7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f729fe00640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72a00c6ca0 0x7f72a00c7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f729fe00640 1 -- 172.21.15.40:0/3442340025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72a00cabe0 msgr2=0x7f72a00c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f729fe00640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72a00cabe0 0x7f72a00c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f729fe00640 1 -- 172.21.15.40:0/3442340025 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a00c5f10 con 0x7f72840039c0 2024-03-24T00:31:47.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f729fe00640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72840039c0 0x7f72a015b330 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f729001c1f0 tx=0x7f7284005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:47.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f7297e00640 1 -- 172.21.15.40:0/3442340025 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72900bb070 con 0x7f72840039c0 2024-03-24T00:31:47.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f7297e00640 1 -- 172.21.15.40:0/3442340025 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72900b7460 con 0x7f72840039c0 2024-03-24T00:31:47.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f7297e00640 1 -- 172.21.15.40:0/3442340025 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72900b0040 con 0x7f72840039c0 2024-03-24T00:31:47.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f72a7000640 1 -- 172.21.15.40:0/3442340025 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a00c61f0 con 0x7f72840039c0 2024-03-24T00:31:47.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f72a7000640 1 -- 172.21.15.40:0/3442340025 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a00c67b0 con 0x7f72840039c0 2024-03-24T00:31:47.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f72a7000640 1 -- 172.21.15.40:0/3442340025 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a0068170 con 0x7f72840039c0 2024-03-24T00:31:47.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f7297e00640 1 -- 172.21.15.40:0/3442340025 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f72900be070 con 0x7f72840039c0 2024-03-24T00:31:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f7297e00640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7280067660 0x7f7280069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.429+0000 7f7297e00640 1 -- 172.21.15.40:0/3442340025 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f7290120230 con 0x7f72840039c0 2024-03-24T00:31:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.433+0000 7f729f400640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7280067660 0x7f7280069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.433+0000 7f729f400640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7280067660 0x7f7280069b20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f72a00cafe0 tx=0x7f728c00a9a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.433+0000 7f7297e00640 1 -- 172.21.15.40:0/3442340025 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f72900160d0 con 0x7f72840039c0 2024-03-24T00:31:47.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 -- 172.21.15.40:0/3442340025 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f72a0070c00 con 0x7f72840039c0 2024-03-24T00:31:47.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f7297e00640 1 -- 172.21.15.40:0/3442340025 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f72900f0970 con 0x7f72840039c0 2024-03-24T00:31:47.668 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:31:47.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 -- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7280067660 msgr2=0x7f7280069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7280067660 0x7f7280069b20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f72a00cafe0 tx=0x7f728c00a9a0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 -- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72840039c0 msgr2=0x7f72a015b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72840039c0 0x7f72a015b330 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f729001c1f0 tx=0x7f7284005840 comp rx=0 tx=0).stop 2024-03-24T00:31:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 -- 172.21.15.40:0/3442340025 shutdown_connections 2024-03-24T00:31:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7280067660 0x7f7280069b20 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72a00c6ca0 0x7f72a00c7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f72a00cabe0 0x7f72a00c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 --2- 172.21.15.40:0/3442340025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72840039c0 0x7f72a015b330 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 -- 172.21.15.40:0/3442340025 >> 172.21.15.40:0/3442340025 conn(0x7f72a00d1d10 msgr2=0x7f72a00b1360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 -- 172.21.15.40:0/3442340025 shutdown_connections 2024-03-24T00:31:47.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.665+0000 7f72a7000640 1 -- 172.21.15.40:0/3442340025 wait complete. 2024-03-24T00:31:47.689 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-03-24T00:31:47.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:47.689 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-24T00:31:47.712 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:31:47.712 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:31:47.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:47.712 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T00:31:47.765 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:31:47.766 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:31:47.776 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:31:47.776 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:31:47.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:47.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6ee000640 1 Processor -- start 2024-03-24T00:31:47.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6ee000640 1 -- start start 2024-03-24T00:31:47.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6ee000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc6e80b46c0 0x7fc6e80b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6ee000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc6e80b2ff0 0x7fc6e80ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6ee000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e8070f60 con 0x7fc6e806e620 2024-03-24T00:31:47.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6ee000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e80b3400 con 0x7fc6e80b2ff0 2024-03-24T00:31:47.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6ee000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6e80b3620 con 0x7fc6e80b46c0 2024-03-24T00:31:47.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6e7400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc6e806e620 0x7fc6e806ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42486/0 (socket says 172.21.15.40:42486) 2024-03-24T00:31:47.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6e7400640 1 -- 172.21.15.40:0/1825503076 learned_addr learned my addr 172.21.15.40:0/1825503076 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:47.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6e7e00640 1 --2- 172.21.15.40:0/1825503076 >> v2:172.21.15.40:3300/0 conn(0x7fc6e80b2ff0 0x7fc6e80ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6dea00640 1 --2- 172.21.15.40:0/1825503076 >> v2:172.21.15.53:3300/0 conn(0x7fc6e80b46c0 0x7fc6e80b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6e7e00640 1 -- 172.21.15.40:0/1825503076 >> v2:172.21.15.53:3300/0 conn(0x7fc6e80b46c0 msgr2=0x7fc6e80b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:47.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6e7e00640 1 --2- 172.21.15.40:0/1825503076 >> v2:172.21.15.53:3300/0 conn(0x7fc6e80b46c0 0x7fc6e80b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6e7e00640 1 -- 172.21.15.40:0/1825503076 >> v1:172.21.15.40:6789/0 conn(0x7fc6e806e620 legacy=0x7fc6e806ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6e7e00640 1 -- 172.21.15.40:0/1825503076 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e80b3900 con 0x7fc6e80b2ff0 2024-03-24T00:31:47.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.965+0000 7fc6e6a00640 1 -- 172.21.15.40:0/1825503076 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025125202 0 0) 0x7fc6e8070f60 con 0x7fc6e806e620 2024-03-24T00:31:47.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.969+0000 7fc6e7e00640 1 --2- 172.21.15.40:0/1825503076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6e80b2ff0 0x7fc6e80ccfa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc6e806cea0 tx=0x7fc6e001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8f8538a8ad050 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:47.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.969+0000 7fc6e6a00640 1 -- 172.21.15.40:0/1825503076 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e00b2070 con 0x7fc6e80b2ff0 2024-03-24T00:31:47.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.969+0000 7fc6e6a00640 1 -- 172.21.15.40:0/1825503076 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e00a5930 con 0x7fc6e80b2ff0 2024-03-24T00:31:47.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.969+0000 7fc6e6a00640 1 -- 172.21.15.40:0/1825503076 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e80cdcf0 con 0x7fc6e80b2ff0 2024-03-24T00:31:47.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.969+0000 7fc6ee000640 1 -- 172.21.15.40:0/1825503076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6e80b2ff0 msgr2=0x7fc6e80ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.969+0000 7fc6ee000640 1 --2- 172.21.15.40:0/1825503076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6e80b2ff0 0x7fc6e80ccfa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc6e806cea0 tx=0x7fc6e001c870 comp rx=0 tx=0).stop 2024-03-24T00:31:47.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.969+0000 7fc6ee000640 1 -- 172.21.15.40:0/1825503076 shutdown_connections 2024-03-24T00:31:47.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.969+0000 7fc6ee000640 1 --2- 172.21.15.40:0/1825503076 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6e80b2ff0 0x7fc6e80ccfa0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.969+0000 7fc6ee000640 1 --2- 172.21.15.40:0/1825503076 >> v2:172.21.15.53:3300/0 conn(0x7fc6e80b46c0 0x7fc6e80b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.969+0000 7fc6ee000640 1 -- 172.21.15.40:0/1825503076 >> 172.21.15.40:0/1825503076 conn(0x7fc6e80b17b0 msgr2=0x7fc6e80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:47.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.969+0000 7fc6ee000640 1 -- 172.21.15.40:0/1825503076 shutdown_connections 2024-03-24T00:31:47.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.969+0000 7fc6ee000640 1 -- 172.21.15.40:0/1825503076 wait complete. 2024-03-24T00:31:47.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6ee000640 1 Processor -- start 2024-03-24T00:31:47.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6ee000640 1 -- start start 2024-03-24T00:31:47.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6ee000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6e806e620 0x7fc6e81f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6ee000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e80b2ff0 0x7fc6e81f8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6ee000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6e80b46c0 0x7fc6e815bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6ee000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e8144920 con 0x7fc6e806e620 2024-03-24T00:31:47.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6ee000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e80b4ad0 con 0x7fc6e80b2ff0 2024-03-24T00:31:47.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6ee000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e81f88d0 con 0x7fc6e80b46c0 2024-03-24T00:31:47.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6e7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6e806e620 0x7fc6e81f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6e7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6e806e620 0x7fc6e81f55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33818/0 (socket says 172.21.15.40:33818) 2024-03-24T00:31:47.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6e7400640 1 -- 172.21.15.40:0/1379088800 learned_addr learned my addr 172.21.15.40:0/1379088800 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:47.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6e7e00640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6e80b46c0 0x7fc6e815bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6dea00640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e80b2ff0 0x7fc6e81f8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6e7400640 1 -- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6e80b46c0 msgr2=0x7fc6e815bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6e7400640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6e80b46c0 0x7fc6e815bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6e7400640 1 -- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e80b2ff0 msgr2=0x7fc6e81f8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:47.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6e7400640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e80b2ff0 0x7fc6e81f8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:47.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6e7400640 1 -- 172.21.15.40:0/1379088800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e8140c50 con 0x7fc6e806e620 2024-03-24T00:31:47.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6e7e00640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6e80b46c0 0x7fc6e815bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:47.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6dea00640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e80b2ff0 0x7fc6e81f8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:47.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6e7400640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6e806e620 0x7fc6e81f55c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8005890 tx=0x7fc6d800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:47.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6dfe00640 1 -- 172.21.15.40:0/1379088800 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8017070 con 0x7fc6e806e620 2024-03-24T00:31:47.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6dfe00640 1 -- 172.21.15.40:0/1379088800 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d8005d10 con 0x7fc6e806e620 2024-03-24T00:31:47.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6ee000640 1 -- 172.21.15.40:0/1379088800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e8140f30 con 0x7fc6e806e620 2024-03-24T00:31:47.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6dfe00640 1 -- 172.21.15.40:0/1379088800 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8007310 con 0x7fc6e806e620 2024-03-24T00:31:47.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.973+0000 7fc6ee000640 1 -- 172.21.15.40:0/1379088800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e813c570 con 0x7fc6e806e620 2024-03-24T00:31:47.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.977+0000 7fc6ee000640 1 -- 172.21.15.40:0/1379088800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e8068170 con 0x7fc6e806e620 2024-03-24T00:31:47.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.977+0000 7fc6dfe00640 1 -- 172.21.15.40:0/1379088800 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc6d801a070 con 0x7fc6e806e620 2024-03-24T00:31:47.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.977+0000 7fc6dfe00640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6c4067660 0x7fc6c4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:47.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.977+0000 7fc6dfe00640 1 -- 172.21.15.40:0/1379088800 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fc6d8084db0 con 0x7fc6e806e620 2024-03-24T00:31:47.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.977+0000 7fc6dea00640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6c4067660 0x7fc6c4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:47.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.977+0000 7fc6dea00640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6c4067660 0x7fc6c4069b20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc6e813de10 tx=0x7fc6d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:47.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:47.985+0000 7fc6dfe00640 1 -- 172.21.15.40:0/1379088800 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc6d8089070 con 0x7fc6e806e620 2024-03-24T00:31:48.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.153+0000 7fc6ee000640 1 -- 172.21.15.40:0/1379088800 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6e8072f50 con 0x7fc6e806e620 2024-03-24T00:31:48.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6dfe00640 1 -- 172.21.15.40:0/1379088800 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16813 (secure 0 0 0) 0x7fc6d80559a0 con 0x7fc6e806e620 2024-03-24T00:31:48.161 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:48.161 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":21,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:47.408228+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6ee000640 1 -- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6c4067660 msgr2=0x7fc6c4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6ee000640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6c4067660 0x7fc6c4069b20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc6e813de10 tx=0x7fc6d0008040 comp rx=0 tx=0).stop 2024-03-24T00:31:48.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6ee000640 1 -- 172.21.15.40:0/1379088800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6e806e620 msgr2=0x7fc6e81f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6ee000640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6e806e620 0x7fc6e81f55c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8005890 tx=0x7fc6d800c8e0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6ee000640 1 -- 172.21.15.40:0/1379088800 shutdown_connections 2024-03-24T00:31:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6ee000640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6c4067660 0x7fc6c4069b20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6ee000640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6e80b46c0 0x7fc6e815bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6ee000640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6e80b2ff0 0x7fc6e81f8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6ee000640 1 --2- 172.21.15.40:0/1379088800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6e806e620 0x7fc6e81f55c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6ee000640 1 -- 172.21.15.40:0/1379088800 >> 172.21.15.40:0/1379088800 conn(0x7fc6e80b17b0 msgr2=0x7fc6e8071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6ee000640 1 -- 172.21.15.40:0/1379088800 shutdown_connections 2024-03-24T00:31:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.157+0000 7fc6ee000640 1 -- 172.21.15.40:0/1379088800 wait complete. 2024-03-24T00:31:48.189 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:48.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:48.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefe2200640 1 Processor -- start 2024-03-24T00:31:48.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefe2200640 1 -- start start 2024-03-24T00:31:48.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefe2200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fefdc144560 0x7fefdc071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:48.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefe2200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefdc072500 con 0x7fefdc0b50a0 2024-03-24T00:31:48.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefe2200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefdc0727e0 con 0x7fefdc0b3c10 2024-03-24T00:31:48.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefe2200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefdc072ac0 con 0x7fefdc144560 2024-03-24T00:31:48.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefdbe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fefdc0b50a0 0x7fefdc164910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42500/0 (socket says 172.21.15.40:42500) 2024-03-24T00:31:48.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefdbe00640 1 -- 172.21.15.40:0/1936693329 learned_addr learned my addr 172.21.15.40:0/1936693329 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:48.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefdb400640 1 --2- 172.21.15.40:0/1936693329 >> v2:172.21.15.53:3300/0 conn(0x7fefdc144560 0x7fefdc071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:48.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefdaa00640 1 -- 172.21.15.40:0/1936693329 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999222346 0 0) 0x7fefdc072500 con 0x7fefdc0b50a0 2024-03-24T00:31:48.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefdaa00640 1 -- 172.21.15.40:0/1936693329 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc0005740 con 0x7fefdc0b50a0 2024-03-24T00:31:48.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefdaa00640 1 -- 172.21.15.40:0/1936693329 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790225969 0 0) 0x7fefdc0727e0 con 0x7fefdc0b3c10 2024-03-24T00:31:48.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefdaa00640 1 -- 172.21.15.40:0/1936693329 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefdc072500 con 0x7fefdc0b3c10 2024-03-24T00:31:48.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefdb400640 1 -- 172.21.15.40:0/1936693329 >> v1:172.21.15.53:6789/0 conn(0x7fefdc0b3c10 legacy=0x7fefdc0b4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefdb400640 1 -- 172.21.15.40:0/1936693329 >> v1:172.21.15.40:6789/0 conn(0x7fefdc0b50a0 legacy=0x7fefdc164910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.329+0000 7fefdb400640 1 -- 172.21.15.40:0/1936693329 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc072da0 con 0x7fefdc144560 2024-03-24T00:31:48.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.333+0000 7fefdb400640 1 --2- 172.21.15.40:0/1936693329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefdc144560 0x7fefdc071b90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fefcc0097f0 tx=0x7fefcc01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9af91b0a44dba1b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:48.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.333+0000 7fefdaa00640 1 -- 172.21.15.40:0/1936693329 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefcc0b1070 con 0x7fefdc144560 2024-03-24T00:31:48.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.333+0000 7fefdaa00640 1 -- 172.21.15.40:0/1936693329 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefcc0ac5f0 con 0x7fefdc144560 2024-03-24T00:31:48.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.333+0000 7fefdaa00640 1 -- 172.21.15.40:0/1936693329 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefcc0ac9c0 con 0x7fefdc144560 2024-03-24T00:31:48.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.333+0000 7fefe2200640 1 -- 172.21.15.40:0/1936693329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefdc144560 msgr2=0x7fefdc071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.333+0000 7fefe2200640 1 --2- 172.21.15.40:0/1936693329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefdc144560 0x7fefdc071b90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fefcc0097f0 tx=0x7fefcc01c470 comp rx=0 tx=0).stop 2024-03-24T00:31:48.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.333+0000 7fefe2200640 1 -- 172.21.15.40:0/1936693329 shutdown_connections 2024-03-24T00:31:48.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.333+0000 7fefe2200640 1 --2- 172.21.15.40:0/1936693329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefdc144560 0x7fefdc071b90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.333+0000 7fefe2200640 1 -- 172.21.15.40:0/1936693329 >> 172.21.15.40:0/1936693329 conn(0x7fefdc0b27c0 msgr2=0x7fefdc0b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:48.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.333+0000 7fefe2200640 1 -- 172.21.15.40:0/1936693329 shutdown_connections 2024-03-24T00:31:48.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.333+0000 7fefe2200640 1 -- 172.21.15.40:0/1936693329 wait complete. 2024-03-24T00:31:48.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefe2200640 1 Processor -- start 2024-03-24T00:31:48.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefe2200640 1 -- start start 2024-03-24T00:31:48.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefe2200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefdc0b3c10 0x7fefdc0d0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:48.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefe2200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc0b50a0 0x7fefdc1541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:48.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefe2200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fefdc144560 0x7fefdc154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:48.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefe2200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc067690 con 0x7fefdc0b50a0 2024-03-24T00:31:48.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefe2200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc0722e0 con 0x7fefdc0b3c10 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefe2200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefdc067810 con 0x7fefdc144560 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefd3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc0b50a0 0x7fefdc1541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefd3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc0b50a0 0x7fefdc1541d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33834/0 (socket says 172.21.15.40:33834) 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefd3e00640 1 -- 172.21.15.40:0/3259744158 learned_addr learned my addr 172.21.15.40:0/3259744158 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefdb400640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefdc0b3c10 0x7fefdc0d0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefdbe00640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fefdc144560 0x7fefdc154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefd3e00640 1 -- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fefdc144560 msgr2=0x7fefdc154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefd3e00640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fefdc144560 0x7fefdc154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefd3e00640 1 -- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefdc0b3c10 msgr2=0x7fefdc0d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefd3e00640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefdc0b3c10 0x7fefdc0d0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefd3e00640 1 -- 172.21.15.40:0/3259744158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefdc154c50 con 0x7fefdc0b50a0 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefd3e00640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc0b50a0 0x7fefdc1541d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fefc400c490 tx=0x7fefc400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefd3400640 1 -- 172.21.15.40:0/3259744158 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc40052d0 con 0x7fefdc0b50a0 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefd3400640 1 -- 172.21.15.40:0/3259744158 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc4005470 con 0x7fefdc0b50a0 2024-03-24T00:31:48.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefe2200640 1 -- 172.21.15.40:0/3259744158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefdc1fa000 con 0x7fefdc0b50a0 2024-03-24T00:31:48.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefd3400640 1 -- 172.21.15.40:0/3259744158 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc4012870 con 0x7fefdc0b50a0 2024-03-24T00:31:48.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.337+0000 7fefe2200640 1 -- 172.21.15.40:0/3259744158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefdc1fa5c0 con 0x7fefdc0b50a0 2024-03-24T00:31:48.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.341+0000 7fefe2200640 1 -- 172.21.15.40:0/3259744158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefdc068170 con 0x7fefdc0b50a0 2024-03-24T00:31:48.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.341+0000 7fefd3400640 1 -- 172.21.15.40:0/3259744158 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fefc4012a10 con 0x7fefdc0b50a0 2024-03-24T00:31:48.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.341+0000 7fefd3400640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fefb80676e0 0x7fefb8069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:48.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.341+0000 7fefd3400640 1 -- 172.21.15.40:0/3259744158 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fefc4085370 con 0x7fefdc0b50a0 2024-03-24T00:31:48.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.341+0000 7fefdb400640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fefb80676e0 0x7fefb8069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:48.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.345+0000 7fefdb400640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fefb80676e0 0x7fefb8069ba0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fefcc0ae000 tx=0x7fefcc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:48.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.345+0000 7fefd3400640 1 -- 172.21.15.40:0/3259744158 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fefc40560e0 con 0x7fefdc0b50a0 2024-03-24T00:31:48.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.513+0000 7fefe2200640 1 -- 172.21.15.40:0/3259744158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fefdc0b32b0 con 0x7fefdc0b50a0 2024-03-24T00:31:48.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.513+0000 7fefd3400640 1 -- 172.21.15.40:0/3259744158 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fefc40559e0 con 0x7fefdc0b50a0 2024-03-24T00:31:48.518 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.513+0000 7fefe2200640 1 -- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fefb80676e0 msgr2=0x7fefb8069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.513+0000 7fefe2200640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fefb80676e0 0x7fefb8069ba0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fefcc0ae000 tx=0x7fefcc0a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.513+0000 7fefe2200640 1 -- 172.21.15.40:0/3259744158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc0b50a0 msgr2=0x7fefdc1541d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.513+0000 7fefe2200640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc0b50a0 0x7fefdc1541d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fefc400c490 tx=0x7fefc400c950 comp rx=0 tx=0).stop 2024-03-24T00:31:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.517+0000 7fefe2200640 1 -- 172.21.15.40:0/3259744158 shutdown_connections 2024-03-24T00:31:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.517+0000 7fefe2200640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fefb80676e0 0x7fefb8069ba0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.517+0000 7fefe2200640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fefdc144560 0x7fefdc154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.517+0000 7fefe2200640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefdc0b50a0 0x7fefdc1541d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.517+0000 7fefe2200640 1 --2- 172.21.15.40:0/3259744158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefdc0b3c10 0x7fefdc0d0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.517+0000 7fefe2200640 1 -- 172.21.15.40:0/3259744158 >> 172.21.15.40:0/3259744158 conn(0x7fefdc0b27c0 msgr2=0x7fefdc1435a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:48.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.517+0000 7fefe2200640 1 -- 172.21.15.40:0/3259744158 shutdown_connections 2024-03-24T00:31:48.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.517+0000 7fefe2200640 1 -- 172.21.15.40:0/3259744158 wait complete. 2024-03-24T00:31:48.538 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:48.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.677+0000 7f2c68200640 1 Processor -- start 2024-03-24T00:31:48.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.677+0000 7f2c68200640 1 -- start start 2024-03-24T00:31:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.681+0000 7f2c68200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2c6015b960 0x7f2c60070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.681+0000 7f2c68200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6015f970 con 0x7f2c6015bd40 2024-03-24T00:31:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.681+0000 7f2c68200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6015fc50 con 0x7f2c600d16f0 2024-03-24T00:31:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.681+0000 7f2c68200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c60160580 con 0x7f2c6015b960 2024-03-24T00:31:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.681+0000 7f2c65800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2c6015b960 0x7f2c60070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.681+0000 7f2c64e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2c6015bd40 0x7f2c600d0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42516/0 (socket says 172.21.15.40:42516) 2024-03-24T00:31:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.681+0000 7f2c64e00640 1 -- 172.21.15.40:0/1560445740 learned_addr learned my addr 172.21.15.40:0/1560445740 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.681+0000 7f2c65800640 1 -- 172.21.15.40:0/1560445740 >> v1:172.21.15.53:6789/0 conn(0x7f2c600d16f0 legacy=0x7f2c600cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.681+0000 7f2c65800640 1 -- 172.21.15.40:0/1560445740 >> v1:172.21.15.40:6789/0 conn(0x7f2c6015bd40 legacy=0x7f2c600d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.681+0000 7f2c65800640 1 -- 172.21.15.40:0/1560445740 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6015ff30 con 0x7f2c6015b960 2024-03-24T00:31:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.685+0000 7f2c65800640 1 --2- 172.21.15.40:0/1560445740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c6015b960 0x7f2c60070d60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2c540097c0 tx=0x7f2c5401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c42e50fc68e79cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.685+0000 7f2c5fe00640 1 -- 172.21.15.40:0/1560445740 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c540b0070 con 0x7f2c6015b960 2024-03-24T00:31:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.685+0000 7f2c5fe00640 1 -- 172.21.15.40:0/1560445740 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c540ab600 con 0x7f2c6015b960 2024-03-24T00:31:48.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.685+0000 7f2c5fe00640 1 -- 172.21.15.40:0/1560445740 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c540ab9d0 con 0x7f2c6015b960 2024-03-24T00:31:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.685+0000 7f2c68200640 1 -- 172.21.15.40:0/1560445740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c6015b960 msgr2=0x7f2c60070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.685+0000 7f2c68200640 1 --2- 172.21.15.40:0/1560445740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c6015b960 0x7f2c60070d60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f2c540097c0 tx=0x7f2c5401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.685+0000 7f2c68200640 1 -- 172.21.15.40:0/1560445740 shutdown_connections 2024-03-24T00:31:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.685+0000 7f2c68200640 1 --2- 172.21.15.40:0/1560445740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c6015b960 0x7f2c60070d60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.685+0000 7f2c68200640 1 -- 172.21.15.40:0/1560445740 >> 172.21.15.40:0/1560445740 conn(0x7f2c6006ecf0 msgr2=0x7f2c6006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.685+0000 7f2c68200640 1 -- 172.21.15.40:0/1560445740 shutdown_connections 2024-03-24T00:31:48.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.685+0000 7f2c68200640 1 -- 172.21.15.40:0/1560445740 wait complete. 2024-03-24T00:31:48.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c68200640 1 Processor -- start 2024-03-24T00:31:48.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c68200640 1 -- start start 2024-03-24T00:31:48.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c68200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2c600d16f0 0x7f2c601467a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:48.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c68200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c6015b960 0x7f2c600c5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:48.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c68200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c6015bd40 0x7f2c600cb410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:48.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c68200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c600cecf0 con 0x7f2c6015b960 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c68200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c600ceb70 con 0x7f2c6015bd40 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c68200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c60169b80 con 0x7f2c600d16f0 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c64e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c6015b960 0x7f2c600c5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c64e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c6015b960 0x7f2c600c5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33844/0 (socket says 172.21.15.40:33844) 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c64e00640 1 -- 172.21.15.40:0/1255233893 learned_addr learned my addr 172.21.15.40:0/1255233893 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c65800640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2c600d16f0 0x7f2c601467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c66200640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c6015bd40 0x7f2c600cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c64e00640 1 -- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2c600d16f0 msgr2=0x7f2c601467a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c64e00640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2c600d16f0 0x7f2c601467a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c64e00640 1 -- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c6015bd40 msgr2=0x7f2c600cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c64e00640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c6015bd40 0x7f2c600cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c64e00640 1 -- 172.21.15.40:0/1255233893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c600d12c0 con 0x7f2c6015b960 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c65800640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2c600d16f0 0x7f2c601467a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:48.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c64e00640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c6015b960 0x7f2c600c5a90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00c430 tx=0x7f2c4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c5d600640 1 -- 172.21.15.40:0/1255233893 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c017070 con 0x7f2c6015b960 2024-03-24T00:31:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c5d600640 1 -- 172.21.15.40:0/1255233893 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c005cf0 con 0x7f2c6015b960 2024-03-24T00:31:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c68200640 1 -- 172.21.15.40:0/1255233893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c600c7a80 con 0x7f2c6015b960 2024-03-24T00:31:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c5d600640 1 -- 172.21.15.40:0/1255233893 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c0072d0 con 0x7f2c6015b960 2024-03-24T00:31:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.689+0000 7f2c68200640 1 -- 172.21.15.40:0/1255233893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c600c8040 con 0x7f2c6015b960 2024-03-24T00:31:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.693+0000 7f2c68200640 1 -- 172.21.15.40:0/1255233893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c60068170 con 0x7f2c6015b960 2024-03-24T00:31:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.693+0000 7f2c5d600640 1 -- 172.21.15.40:0/1255233893 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2c4c01a070 con 0x7f2c6015b960 2024-03-24T00:31:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.693+0000 7f2c5d600640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2c40067660 0x7f2c40069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:48.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.693+0000 7f2c5d600640 1 -- 172.21.15.40:0/1255233893 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f2c4c084c70 con 0x7f2c6015b960 2024-03-24T00:31:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.693+0000 7f2c65800640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2c40067660 0x7f2c40069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.697+0000 7f2c65800640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2c40067660 0x7f2c40069b20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2c540ad000 tx=0x7f2c540a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:48.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.697+0000 7f2c5d600640 1 -- 172.21.15.40:0/1255233893 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2c4c055190 con 0x7f2c6015b960 2024-03-24T00:31:48.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 -- 172.21.15.40:0/1255233893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2c6015b120 con 0x7f2c6015b960 2024-03-24T00:31:48.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c5d600640 1 -- 172.21.15.40:0/1255233893 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2c4c059230 con 0x7f2c6015b960 2024-03-24T00:31:48.873 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:48.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 -- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2c40067660 msgr2=0x7f2c40069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2c40067660 0x7f2c40069b20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2c540ad000 tx=0x7f2c540a6000 comp rx=0 tx=0).stop 2024-03-24T00:31:48.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 -- 172.21.15.40:0/1255233893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c6015b960 msgr2=0x7f2c600c5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:48.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c6015b960 0x7f2c600c5a90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00c430 tx=0x7f2c4c00c8f0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 -- 172.21.15.40:0/1255233893 shutdown_connections 2024-03-24T00:31:48.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2c40067660 0x7f2c40069b20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c6015bd40 0x7f2c600cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2c6015b960 0x7f2c600c5a90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 --2- 172.21.15.40:0/1255233893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2c600d16f0 0x7f2c601467a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:48.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 -- 172.21.15.40:0/1255233893 >> 172.21.15.40:0/1255233893 conn(0x7f2c6006ecf0 msgr2=0x7f2c6015d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:48.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 -- 172.21.15.40:0/1255233893 shutdown_connections 2024-03-24T00:31:48.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:48.869+0000 7f2c68200640 1 -- 172.21.15.40:0/1255233893 wait complete. 2024-03-24T00:31:48.893 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:49.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.037+0000 7f6a6d000640 1 Processor -- start 2024-03-24T00:31:49.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.037+0000 7f6a6d000640 1 -- start start 2024-03-24T00:31:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a6d000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815bb10 0x7f6a6806f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a6d000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a68067600 0x7f6a68161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a6d000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a68067c60 con 0x7f6a6806f570 2024-03-24T00:31:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a6d000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6806f9d0 con 0x7f6a68067600 2024-03-24T00:31:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a6d000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6815ef60 con 0x7f6a6815bb10 2024-03-24T00:31:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a66800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a68067600 0x7f6a68161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a66800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6a68067600 0x7f6a68161bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33846/0 (socket says 172.21.15.40:33846) 2024-03-24T00:31:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a66800640 1 -- 172.21.15.40:0/193715590 learned_addr learned my addr 172.21.15.40:0/193715590 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a5fe00640 1 --1- 172.21.15.40:0/193715590 >> v1:172.21.15.40:6789/0 conn(0x7f6a6806f570 0x7f6a68066ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42522/0 (socket says 172.21.15.40:42522) 2024-03-24T00:31:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a65e00640 1 --2- 172.21.15.40:0/193715590 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815bb10 0x7f6a6806f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a66800640 1 -- 172.21.15.40:0/193715590 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815bb10 msgr2=0x7f6a6806f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:49.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a66800640 1 --2- 172.21.15.40:0/193715590 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815bb10 0x7f6a6806f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a66800640 1 -- 172.21.15.40:0/193715590 >> v1:172.21.15.40:6789/0 conn(0x7f6a6806f570 legacy=0x7f6a68066ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.041+0000 7f6a66800640 1 -- 172.21.15.40:0/193715590 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6815e910 con 0x7f6a68067600 2024-03-24T00:31:49.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a66800640 1 --2- 172.21.15.40:0/193715590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68067600 0x7f6a68161bb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009e50 tx=0x7f6a6001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e811a2a41b56af96 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:49.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a65400640 1 -- 172.21.15.40:0/193715590 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a600a5810 con 0x7f6a68067600 2024-03-24T00:31:49.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a65400640 1 -- 172.21.15.40:0/193715590 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a600a59b0 con 0x7f6a68067600 2024-03-24T00:31:49.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a65400640 1 -- 172.21.15.40:0/193715590 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a600a5d80 con 0x7f6a68067600 2024-03-24T00:31:49.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 -- 172.21.15.40:0/193715590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68067600 msgr2=0x7f6a68161bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 --2- 172.21.15.40:0/193715590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68067600 0x7f6a68161bb0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009e50 tx=0x7f6a6001ca30 comp rx=0 tx=0).stop 2024-03-24T00:31:49.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 -- 172.21.15.40:0/193715590 shutdown_connections 2024-03-24T00:31:49.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 --2- 172.21.15.40:0/193715590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68067600 0x7f6a68161bb0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 --2- 172.21.15.40:0/193715590 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815bb10 0x7f6a6806f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 -- 172.21.15.40:0/193715590 >> 172.21.15.40:0/193715590 conn(0x7f6a680b27c0 msgr2=0x7f6a680b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 -- 172.21.15.40:0/193715590 shutdown_connections 2024-03-24T00:31:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 -- 172.21.15.40:0/193715590 wait complete. 2024-03-24T00:31:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 Processor -- start 2024-03-24T00:31:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 -- start start 2024-03-24T00:31:49.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68067600 0x7f6a6813b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a6806f570 0x7f6a68150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815bb10 0x7f6a6806dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6815f550 con 0x7f6a68067600 2024-03-24T00:31:49.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6815f3d0 con 0x7f6a6806f570 2024-03-24T00:31:49.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a6d000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6816af50 con 0x7f6a6815bb10 2024-03-24T00:31:49.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.045+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68067600 0x7f6a6813b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68067600 0x7f6a6813b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33848/0 (socket says 172.21.15.40:33848) 2024-03-24T00:31:49.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a65e00640 1 -- 172.21.15.40:0/563755323 learned_addr learned my addr 172.21.15.40:0/563755323 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:49.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a66800640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815bb10 0x7f6a6806dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a5fe00640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a6806f570 0x7f6a68150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a65e00640 1 -- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815bb10 msgr2=0x7f6a6806dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a65e00640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815bb10 0x7f6a6806dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a65e00640 1 -- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a6806f570 msgr2=0x7f6a68150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a65e00640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a6806f570 0x7f6a68150630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a65e00640 1 -- 172.21.15.40:0/563755323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680ae6a0 con 0x7f6a68067600 2024-03-24T00:31:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a65e00640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68067600 0x7f6a6813b0a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800e8a0 tx=0x7f6a5800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a5e000640 1 -- 172.21.15.40:0/563755323 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58017070 con 0x7f6a68067600 2024-03-24T00:31:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a5e000640 1 -- 172.21.15.40:0/563755323 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5800c940 con 0x7f6a68067600 2024-03-24T00:31:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a5e000640 1 -- 172.21.15.40:0/563755323 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5800ccf0 con 0x7f6a68067600 2024-03-24T00:31:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a6d000640 1 -- 172.21.15.40:0/563755323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6814fb50 con 0x7f6a68067600 2024-03-24T00:31:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a6d000640 1 -- 172.21.15.40:0/563755323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a68150b70 con 0x7f6a68067600 2024-03-24T00:31:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a6d000640 1 -- 172.21.15.40:0/563755323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6814f970 con 0x7f6a68067600 2024-03-24T00:31:49.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.049+0000 7f6a5e000640 1 -- 172.21.15.40:0/563755323 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6a5801a070 con 0x7f6a68067600 2024-03-24T00:31:49.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.053+0000 7f6a5e000640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a44067660 0x7f6a44069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.053+0000 7f6a5e000640 1 -- 172.21.15.40:0/563755323 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f6a58085cb0 con 0x7f6a68067600 2024-03-24T00:31:49.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.053+0000 7f6a5fe00640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a44067660 0x7f6a44069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.053+0000 7f6a5fe00640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a44067660 0x7f6a44069b20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6a68141f70 tx=0x7f6a50007cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.057+0000 7f6a5e000640 1 -- 172.21.15.40:0/563755323 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6a6814f970 con 0x7f6a68067600 2024-03-24T00:31:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 -- 172.21.15.40:0/563755323 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6a68068170 con 0x7f6a68067600 2024-03-24T00:31:49.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a5e000640 1 -- 172.21.15.40:0/563755323 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6a58056780 con 0x7f6a68067600 2024-03-24T00:31:49.224 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:49.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 -- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a44067660 msgr2=0x7f6a44069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a44067660 0x7f6a44069b20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6a68141f70 tx=0x7f6a50007cf0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 -- 172.21.15.40:0/563755323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68067600 msgr2=0x7f6a6813b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68067600 0x7f6a6813b0a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800e8a0 tx=0x7f6a5800eca0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 -- 172.21.15.40:0/563755323 shutdown_connections 2024-03-24T00:31:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a44067660 0x7f6a44069b20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815bb10 0x7f6a6806dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a6806f570 0x7f6a68150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 --2- 172.21.15.40:0/563755323 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a68067600 0x7f6a6813b0a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 -- 172.21.15.40:0/563755323 >> 172.21.15.40:0/563755323 conn(0x7f6a680b27c0 msgr2=0x7f6a680b1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 -- 172.21.15.40:0/563755323 shutdown_connections 2024-03-24T00:31:49.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.221+0000 7f6a6d000640 1 -- 172.21.15.40:0/563755323 wait complete. 2024-03-24T00:31:49.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:31:49.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.385+0000 7fe65e600640 1 Processor -- start 2024-03-24T00:31:49.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.385+0000 7fe65e600640 1 -- start start 2024-03-24T00:31:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.385+0000 7fe65e600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe65806e620 0x7fe6580b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.385+0000 7fe65e600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe6580ccfa0 0x7fe6580cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.385+0000 7fe65e600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe658070f60 con 0x7fe65806ea00 2024-03-24T00:31:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.385+0000 7fe65e600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6580b4c00 con 0x7fe6580ccfa0 2024-03-24T00:31:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.385+0000 7fe65e600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6580b3620 con 0x7fe65806e620 2024-03-24T00:31:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.385+0000 7fe64ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe65806ea00 0x7fe6580b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42528/0 (socket says 172.21.15.40:42528) 2024-03-24T00:31:49.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.385+0000 7fe64ea00640 1 -- 172.21.15.40:0/2281363970 learned_addr learned my addr 172.21.15.40:0/2281363970 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.385+0000 7fe657400640 1 --2- 172.21.15.40:0/2281363970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe65806e620 0x7fe6580b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.385+0000 7fe657e00640 1 --2- 172.21.15.40:0/2281363970 >> v2:172.21.15.53:3300/0 conn(0x7fe6580ccfa0 0x7fe6580cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.389+0000 7fe656a00640 1 -- 172.21.15.40:0/2281363970 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000128215 0 0) 0x7fe658070f60 con 0x7fe65806ea00 2024-03-24T00:31:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.389+0000 7fe656a00640 1 -- 172.21.15.40:0/2281363970 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe63c003530 con 0x7fe65806ea00 2024-03-24T00:31:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.389+0000 7fe657400640 1 -- 172.21.15.40:0/2281363970 >> v2:172.21.15.53:3300/0 conn(0x7fe6580ccfa0 msgr2=0x7fe6580cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:49.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.389+0000 7fe657400640 1 --2- 172.21.15.40:0/2281363970 >> v2:172.21.15.53:3300/0 conn(0x7fe6580ccfa0 0x7fe6580cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.389+0000 7fe657400640 1 -- 172.21.15.40:0/2281363970 >> v1:172.21.15.40:6789/0 conn(0x7fe65806ea00 legacy=0x7fe6580b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.389+0000 7fe657400640 1 -- 172.21.15.40:0/2281363970 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6580b3900 con 0x7fe65806e620 2024-03-24T00:31:49.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.389+0000 7fe657400640 1 --2- 172.21.15.40:0/2281363970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe65806e620 0x7fe6580b46c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe648009d90 tx=0x7fe64801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdbc959c5915248d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:49.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.389+0000 7fe656a00640 1 -- 172.21.15.40:0/2281363970 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6480b2070 con 0x7fe65806e620 2024-03-24T00:31:49.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.389+0000 7fe656a00640 1 -- 172.21.15.40:0/2281363970 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6480a5880 con 0x7fe65806e620 2024-03-24T00:31:49.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.389+0000 7fe656a00640 1 -- 172.21.15.40:0/2281363970 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6480a5c50 con 0x7fe65806e620 2024-03-24T00:31:49.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.393+0000 7fe65e600640 1 -- 172.21.15.40:0/2281363970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe65806e620 msgr2=0x7fe6580b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.393+0000 7fe65e600640 1 --2- 172.21.15.40:0/2281363970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe65806e620 0x7fe6580b46c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe648009d90 tx=0x7fe64801c8b0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.393+0000 7fe65e600640 1 -- 172.21.15.40:0/2281363970 shutdown_connections 2024-03-24T00:31:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.393+0000 7fe65e600640 1 --2- 172.21.15.40:0/2281363970 >> v2:172.21.15.53:3300/0 conn(0x7fe6580ccfa0 0x7fe6580cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.393+0000 7fe65e600640 1 --2- 172.21.15.40:0/2281363970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe65806e620 0x7fe6580b46c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.393+0000 7fe65e600640 1 -- 172.21.15.40:0/2281363970 >> 172.21.15.40:0/2281363970 conn(0x7fe6580b17b0 msgr2=0x7fe6580b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.393+0000 7fe65e600640 1 -- 172.21.15.40:0/2281363970 shutdown_connections 2024-03-24T00:31:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.393+0000 7fe65e600640 1 -- 172.21.15.40:0/2281363970 wait complete. 2024-03-24T00:31:49.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.393+0000 7fe65e600640 1 Processor -- start 2024-03-24T00:31:49.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.393+0000 7fe65e600640 1 -- start start 2024-03-24T00:31:49.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe65e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65806e620 0x7fe65815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe65e600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe65806ea00 0x7fe658071300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe65e600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6580ccfa0 0x7fe658071840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe65e600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe658144920 con 0x7fe65806e620 2024-03-24T00:31:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe65e600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65806ede0 con 0x7fe6580ccfa0 2024-03-24T00:31:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe65e600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6581fb150 con 0x7fe65806ea00 2024-03-24T00:31:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe657400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65806e620 0x7fe65815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe657400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65806e620 0x7fe65815a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33864/0 (socket says 172.21.15.40:33864) 2024-03-24T00:31:49.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe657400640 1 -- 172.21.15.40:0/3786066125 learned_addr learned my addr 172.21.15.40:0/3786066125 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe64ea00640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe65806ea00 0x7fe658071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe657e00640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6580ccfa0 0x7fe658071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe657400640 1 -- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe65806ea00 msgr2=0x7fe658071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe657400640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe65806ea00 0x7fe658071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe657400640 1 -- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6580ccfa0 msgr2=0x7fe658071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe657400640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6580ccfa0 0x7fe658071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe657400640 1 -- 172.21.15.40:0/3786066125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6581fb3a0 con 0x7fe65806e620 2024-03-24T00:31:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe657400640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65806e620 0x7fe65815a1f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe6480b0040 tx=0x7fe648019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe64fe00640 1 -- 172.21.15.40:0/3786066125 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6480bd020 con 0x7fe65806e620 2024-03-24T00:31:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe64fe00640 1 -- 172.21.15.40:0/3786066125 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6480b8470 con 0x7fe65806e620 2024-03-24T00:31:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe64fe00640 1 -- 172.21.15.40:0/3786066125 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6480b2040 con 0x7fe65806e620 2024-03-24T00:31:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe65e600640 1 -- 172.21.15.40:0/3786066125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6581fb680 con 0x7fe65806e620 2024-03-24T00:31:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe65e600640 1 -- 172.21.15.40:0/3786066125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6581fbc40 con 0x7fe65806e620 2024-03-24T00:31:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.397+0000 7fe65e600640 1 -- 172.21.15.40:0/3786066125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe658068170 con 0x7fe65806e620 2024-03-24T00:31:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.401+0000 7fe64fe00640 1 -- 172.21.15.40:0/3786066125 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe6480c0020 con 0x7fe65806e620 2024-03-24T00:31:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.401+0000 7fe64fe00640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe634067660 0x7fe634069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.401+0000 7fe64fe00640 1 -- 172.21.15.40:0/3786066125 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fe648120880 con 0x7fe65806e620 2024-03-24T00:31:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.401+0000 7fe64ea00640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe634067660 0x7fe634069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.401+0000 7fe64ea00640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe634067660 0x7fe634069b20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe6580cea60 tx=0x7fe640009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.409+0000 7fe64fe00640 1 -- 172.21.15.40:0/3786066125 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe6480c7030 con 0x7fe65806e620 2024-03-24T00:31:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.581+0000 7fe65e600640 1 -- 172.21.15.40:0/3786066125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe658072f50 con 0x7fe65806e620 2024-03-24T00:31:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.581+0000 7fe64fe00640 1 -- 172.21.15.40:0/3786066125 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe648016560 con 0x7fe65806e620 2024-03-24T00:31:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:31:49.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.585+0000 7fe65e600640 1 -- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe634067660 msgr2=0x7fe634069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.585+0000 7fe65e600640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe634067660 0x7fe634069b20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe6580cea60 tx=0x7fe640009040 comp rx=0 tx=0).stop 2024-03-24T00:31:49.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.585+0000 7fe65e600640 1 -- 172.21.15.40:0/3786066125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65806e620 msgr2=0x7fe65815a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.585+0000 7fe65e600640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65806e620 0x7fe65815a1f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe6480b0040 tx=0x7fe648019650 comp rx=0 tx=0).stop 2024-03-24T00:31:49.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.585+0000 7fe65e600640 1 -- 172.21.15.40:0/3786066125 shutdown_connections 2024-03-24T00:31:49.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.585+0000 7fe65e600640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe634067660 0x7fe634069b20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.585+0000 7fe65e600640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6580ccfa0 0x7fe658071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.585+0000 7fe65e600640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe65806ea00 0x7fe658071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.585+0000 7fe65e600640 1 --2- 172.21.15.40:0/3786066125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe65806e620 0x7fe65815a1f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.585+0000 7fe65e600640 1 -- 172.21.15.40:0/3786066125 >> 172.21.15.40:0/3786066125 conn(0x7fe6580b17b0 msgr2=0x7fe6580b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:49.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.585+0000 7fe65e600640 1 -- 172.21.15.40:0/3786066125 shutdown_connections 2024-03-24T00:31:49.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.585+0000 7fe65e600640 1 -- 172.21.15.40:0/3786066125 wait complete. 2024-03-24T00:31:49.609 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:31:49.609 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:31:49.619 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.753+0000 7f1b0d200640 1 Processor -- start 2024-03-24T00:31:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.753+0000 7f1b0d200640 1 -- start start 2024-03-24T00:31:49.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.757+0000 7f1b0d200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1b08072310 0x7f1b080ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.757+0000 7f1b0d200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b08070e00 con 0x7f1b0813ec80 2024-03-24T00:31:49.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.757+0000 7f1b0d200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b080cee20 con 0x7f1b08141bc0 2024-03-24T00:31:49.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.757+0000 7f1b0d200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b080cf100 con 0x7f1b08072310 2024-03-24T00:31:49.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.757+0000 7f1b06800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1b08072310 0x7f1b080ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.757+0000 7f1b06800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1b08072310 0x7f1b080ce800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33870/0 (socket says 172.21.15.40:33870) 2024-03-24T00:31:49.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.757+0000 7f1b06800640 1 -- 172.21.15.40:0/1970535471 learned_addr learned my addr 172.21.15.40:0/1970535471 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:49.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.757+0000 7f1b05e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1b0813ec80 0x7f1b081414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42534/0 (socket says 172.21.15.40:42534) 2024-03-24T00:31:49.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.757+0000 7f1b06800640 1 -- 172.21.15.40:0/1970535471 >> v1:172.21.15.53:6789/0 conn(0x7f1b08141bc0 legacy=0x7f1b080ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.757+0000 7f1b06800640 1 -- 172.21.15.40:0/1970535471 >> v1:172.21.15.40:6789/0 conn(0x7f1b0813ec80 legacy=0x7f1b081414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.757+0000 7f1b05400640 1 -- 172.21.15.40:0/1970535471 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931753613 0 0) 0x7f1b08070e00 con 0x7f1b0813ec80 2024-03-24T00:31:49.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.757+0000 7f1b06800640 1 -- 172.21.15.40:0/1970535471 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b080cf3e0 con 0x7f1b08072310 2024-03-24T00:31:49.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.761+0000 7f1b06800640 1 --2- 172.21.15.40:0/1970535471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b08072310 0x7f1b080ce800 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009e50 tx=0x7f1b0001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2228fb9458961b1e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:49.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.761+0000 7f1b05400640 1 -- 172.21.15.40:0/1970535471 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b000a5920 con 0x7f1b08072310 2024-03-24T00:31:49.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.761+0000 7f1b05400640 1 -- 172.21.15.40:0/1970535471 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b000a5ac0 con 0x7f1b08072310 2024-03-24T00:31:49.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.761+0000 7f1b05400640 1 -- 172.21.15.40:0/1970535471 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b000ad3f0 con 0x7f1b08072310 2024-03-24T00:31:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.761+0000 7f1b0d200640 1 -- 172.21.15.40:0/1970535471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b08072310 msgr2=0x7f1b080ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.761+0000 7f1b0d200640 1 --2- 172.21.15.40:0/1970535471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b08072310 0x7f1b080ce800 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1b00009e50 tx=0x7f1b0001ca30 comp rx=0 tx=0).stop 2024-03-24T00:31:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.761+0000 7f1b0d200640 1 -- 172.21.15.40:0/1970535471 shutdown_connections 2024-03-24T00:31:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.761+0000 7f1b0d200640 1 --2- 172.21.15.40:0/1970535471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b08072310 0x7f1b080ce800 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.761+0000 7f1b0d200640 1 -- 172.21.15.40:0/1970535471 >> 172.21.15.40:0/1970535471 conn(0x7f1b080b17b0 msgr2=0x7f1b080b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.761+0000 7f1b0d200640 1 -- 172.21.15.40:0/1970535471 shutdown_connections 2024-03-24T00:31:49.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.761+0000 7f1b0d200640 1 -- 172.21.15.40:0/1970535471 wait complete. 2024-03-24T00:31:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b0d200640 1 Processor -- start 2024-03-24T00:31:49.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b0d200640 1 -- start start 2024-03-24T00:31:49.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b0d200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b08072310 0x7f1b0815b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b0d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b0813ec80 0x7f1b0813da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b0d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b08141bc0 0x7f1b0813a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b0d200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b08169f00 con 0x7f1b08141bc0 2024-03-24T00:31:49.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b0d200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b08169d80 con 0x7f1b0813ec80 2024-03-24T00:31:49.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b0d200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0816a080 con 0x7f1b08072310 2024-03-24T00:31:49.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b06800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b08141bc0 0x7f1b0813a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b06800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b08141bc0 0x7f1b0813a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33884/0 (socket says 172.21.15.40:33884) 2024-03-24T00:31:49.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b06800640 1 -- 172.21.15.40:0/2565986536 learned_addr learned my addr 172.21.15.40:0/2565986536 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:49.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b05e00640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b08072310 0x7f1b0815b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1afd400640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b0813ec80 0x7f1b0813da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b06800640 1 -- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b08072310 msgr2=0x7f1b0815b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b06800640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b08072310 0x7f1b0815b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b06800640 1 -- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b0813ec80 msgr2=0x7f1b0813da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b06800640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b0813ec80 0x7f1b0813da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b06800640 1 -- 172.21.15.40:0/2565986536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0813df70 con 0x7f1b08141bc0 2024-03-24T00:31:49.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b06800640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b08141bc0 0x7f1b0813a2a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1b000af040 tx=0x7f1b000197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:49.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1afea00640 1 -- 172.21.15.40:0/2565986536 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b000bc020 con 0x7f1b08141bc0 2024-03-24T00:31:49.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1afea00640 1 -- 172.21.15.40:0/2565986536 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b000b8470 con 0x7f1b08141bc0 2024-03-24T00:31:49.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b0d200640 1 -- 172.21.15.40:0/2565986536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0815ba00 con 0x7f1b08141bc0 2024-03-24T00:31:49.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1afea00640 1 -- 172.21.15.40:0/2565986536 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b00009510 con 0x7f1b08141bc0 2024-03-24T00:31:49.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.765+0000 7f1b0d200640 1 -- 172.21.15.40:0/2565986536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0815bfc0 con 0x7f1b08141bc0 2024-03-24T00:31:49.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.769+0000 7f1b0d200640 1 -- 172.21.15.40:0/2565986536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b08068170 con 0x7f1b08141bc0 2024-03-24T00:31:49.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.769+0000 7f1afea00640 1 -- 172.21.15.40:0/2565986536 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f1b000bf020 con 0x7f1b08141bc0 2024-03-24T00:31:49.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.769+0000 7f1afea00640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1ae4067690 0x7f1ae4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:49.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.769+0000 7f1afea00640 1 -- 172.21.15.40:0/2565986536 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f1b001204d0 con 0x7f1b08141bc0 2024-03-24T00:31:49.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.769+0000 7f1b05e00640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1ae4067690 0x7f1ae4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:49.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.769+0000 7f1b05e00640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1ae4067690 0x7f1ae4069b50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1af8000f70 tx=0x7f1af8007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:49.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.773+0000 7f1afea00640 1 -- 172.21.15.40:0/2565986536 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1b000c7020 con 0x7f1b08141bc0 2024-03-24T00:31:49.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.937+0000 7f1b0d200640 1 -- 172.21.15.40:0/2565986536 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b08072b90 con 0x7f1b08141bc0 2024-03-24T00:31:49.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.937+0000 7f1afea00640 1 -- 172.21.15.40:0/2565986536 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16813 (secure 0 0 0) 0x7f1b00016620 con 0x7f1b08141bc0 2024-03-24T00:31:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:49.944 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":22,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:48.417423+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:49.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.941+0000 7f1b0d200640 1 -- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1ae4067690 msgr2=0x7f1ae4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.941+0000 7f1b0d200640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1ae4067690 0x7f1ae4069b50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1af8000f70 tx=0x7f1af8007240 comp rx=0 tx=0).stop 2024-03-24T00:31:49.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.941+0000 7f1b0d200640 1 -- 172.21.15.40:0/2565986536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b08141bc0 msgr2=0x7f1b0813a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:49.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.941+0000 7f1b0d200640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b08141bc0 0x7f1b0813a2a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1b000af040 tx=0x7f1b000197d0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.941+0000 7f1b0d200640 1 -- 172.21.15.40:0/2565986536 shutdown_connections 2024-03-24T00:31:49.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.941+0000 7f1b0d200640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1ae4067690 0x7f1ae4069b50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.941+0000 7f1b0d200640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1b08141bc0 0x7f1b0813a2a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.941+0000 7f1b0d200640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b0813ec80 0x7f1b0813da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.941+0000 7f1b0d200640 1 --2- 172.21.15.40:0/2565986536 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b08072310 0x7f1b0815b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:49.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.941+0000 7f1b0d200640 1 -- 172.21.15.40:0/2565986536 >> 172.21.15.40:0/2565986536 conn(0x7f1b080b17b0 msgr2=0x7f1b0814eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:49.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.941+0000 7f1b0d200640 1 -- 172.21.15.40:0/2565986536 shutdown_connections 2024-03-24T00:31:49.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:49.941+0000 7f1b0d200640 1 -- 172.21.15.40:0/2565986536 wait complete. 2024-03-24T00:31:49.966 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:49.967 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:50.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.105+0000 7f44f5800640 1 Processor -- start 2024-03-24T00:31:50.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44f5800640 1 -- start start 2024-03-24T00:31:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44f5800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f00d2270 0x7f44f0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44f5800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f006d0f0 con 0x7f44f015df80 2024-03-24T00:31:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44f5800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f006b280 con 0x7f44f006c950 2024-03-24T00:31:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44f5800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f006b890 con 0x7f44f00d2270 2024-03-24T00:31:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44eea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f44f015df80 0x7f44f00d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42540/0 (socket says 172.21.15.40:42540) 2024-03-24T00:31:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44eea00640 1 -- 172.21.15.40:0/1917220318 learned_addr learned my addr 172.21.15.40:0/1917220318 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44e7e00640 1 --2- 172.21.15.40:0/1917220318 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f00d2270 0x7f44f0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44ee000640 1 -- 172.21.15.40:0/1917220318 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215919474 0 0) 0x7f44f006b280 con 0x7f44f006c950 2024-03-24T00:31:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44ee000640 1 -- 172.21.15.40:0/1917220318 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d4003530 con 0x7f44f006c950 2024-03-24T00:31:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44ee000640 1 -- 172.21.15.40:0/1917220318 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502250179 0 0) 0x7f44f006d0f0 con 0x7f44f015df80 2024-03-24T00:31:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44ee000640 1 -- 172.21.15.40:0/1917220318 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f006b280 con 0x7f44f015df80 2024-03-24T00:31:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44e7e00640 1 -- 172.21.15.40:0/1917220318 >> v1:172.21.15.53:6789/0 conn(0x7f44f006c950 legacy=0x7f44f0161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44e7e00640 1 -- 172.21.15.40:0/1917220318 >> v1:172.21.15.40:6789/0 conn(0x7f44f015df80 legacy=0x7f44f00d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.109+0000 7f44e7e00640 1 -- 172.21.15.40:0/1917220318 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f006ce10 con 0x7f44f00d2270 2024-03-24T00:31:50.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.113+0000 7f44e7e00640 1 --2- 172.21.15.40:0/1917220318 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f00d2270 0x7f44f0070ff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f44d80097f0 tx=0x7f44d801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4926ab436ed25da5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:50.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.113+0000 7f44ee000640 1 -- 172.21.15.40:0/1917220318 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d80a7070 con 0x7f44f00d2270 2024-03-24T00:31:50.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.113+0000 7f44ee000640 1 -- 172.21.15.40:0/1917220318 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d80b0070 con 0x7f44f00d2270 2024-03-24T00:31:50.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.113+0000 7f44ee000640 1 -- 172.21.15.40:0/1917220318 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d80ab890 con 0x7f44f00d2270 2024-03-24T00:31:50.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.113+0000 7f44f5800640 1 -- 172.21.15.40:0/1917220318 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f00d2270 msgr2=0x7f44f0070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.113+0000 7f44f5800640 1 --2- 172.21.15.40:0/1917220318 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f00d2270 0x7f44f0070ff0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f44d80097f0 tx=0x7f44d801c4b0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.113+0000 7f44f5800640 1 -- 172.21.15.40:0/1917220318 shutdown_connections 2024-03-24T00:31:50.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.113+0000 7f44f5800640 1 --2- 172.21.15.40:0/1917220318 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f00d2270 0x7f44f0070ff0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.113+0000 7f44f5800640 1 -- 172.21.15.40:0/1917220318 >> 172.21.15.40:0/1917220318 conn(0x7f44f006ecf0 msgr2=0x7f44f006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:50.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.113+0000 7f44f5800640 1 -- 172.21.15.40:0/1917220318 shutdown_connections 2024-03-24T00:31:50.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.113+0000 7f44f5800640 1 -- 172.21.15.40:0/1917220318 wait complete. 2024-03-24T00:31:50.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44f5800640 1 Processor -- start 2024-03-24T00:31:50.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44f5800640 1 -- start start 2024-03-24T00:31:50.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44f5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f006c950 0x7f44f00c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44f5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f00d2270 0x7f44f0155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44f5800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f015df80 0x7f44f0157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44f5800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f016bfa0 con 0x7f44f006c950 2024-03-24T00:31:50.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44f5800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f016be20 con 0x7f44f00d2270 2024-03-24T00:31:50.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44f5800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f016c120 con 0x7f44f015df80 2024-03-24T00:31:50.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44eea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f006c950 0x7f44f00c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44eea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f006c950 0x7f44f00c78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33900/0 (socket says 172.21.15.40:33900) 2024-03-24T00:31:50.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44eea00640 1 -- 172.21.15.40:0/3659692175 learned_addr learned my addr 172.21.15.40:0/3659692175 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:50.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44e7e00640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f00d2270 0x7f44f0155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44ef400640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f015df80 0x7f44f0157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44eea00640 1 -- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f015df80 msgr2=0x7f44f0157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44eea00640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f015df80 0x7f44f0157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44eea00640 1 -- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f00d2270 msgr2=0x7f44f0155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44eea00640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f00d2270 0x7f44f0155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44eea00640 1 -- 172.21.15.40:0/3659692175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f00add60 con 0x7f44f006c950 2024-03-24T00:31:50.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44e7e00640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f00d2270 0x7f44f0155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:50.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44eea00640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f006c950 0x7f44f00c78a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f44e000c450 tx=0x7f44e000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:50.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.117+0000 7f44e6a00640 1 -- 172.21.15.40:0/3659692175 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0017070 con 0x7f44f006c950 2024-03-24T00:31:50.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.121+0000 7f44e6a00640 1 -- 172.21.15.40:0/3659692175 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e0005cf0 con 0x7f44f006c950 2024-03-24T00:31:50.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.121+0000 7f44e6a00640 1 -- 172.21.15.40:0/3659692175 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0007290 con 0x7f44f006c950 2024-03-24T00:31:50.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.121+0000 7f44f5800640 1 -- 172.21.15.40:0/3659692175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f00ae040 con 0x7f44f006c950 2024-03-24T00:31:50.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.121+0000 7f44f5800640 1 -- 172.21.15.40:0/3659692175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f00ae600 con 0x7f44f006c950 2024-03-24T00:31:50.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.121+0000 7f44f5800640 1 -- 172.21.15.40:0/3659692175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f0068170 con 0x7f44f006c950 2024-03-24T00:31:50.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.121+0000 7f44e6a00640 1 -- 172.21.15.40:0/3659692175 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f44e001a070 con 0x7f44f006c950 2024-03-24T00:31:50.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.121+0000 7f44e6a00640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44cc067660 0x7f44cc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.121+0000 7f44e6a00640 1 -- 172.21.15.40:0/3659692175 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f44e0084c90 con 0x7f44f006c950 2024-03-24T00:31:50.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.125+0000 7f44e7e00640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44cc067660 0x7f44cc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.125+0000 7f44e7e00640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44cc067660 0x7f44cc069b20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f44d80af000 tx=0x7f44d80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:50.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.125+0000 7f44e6a00640 1 -- 172.21.15.40:0/3659692175 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f44e0055240 con 0x7f44f006c950 2024-03-24T00:31:50.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.329+0000 7f44f5800640 1 -- 172.21.15.40:0/3659692175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44f00cf9d0 con 0x7f44f006c950 2024-03-24T00:31:50.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.329+0000 7f44e6a00640 1 -- 172.21.15.40:0/3659692175 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f44e00592e0 con 0x7f44f006c950 2024-03-24T00:31:50.334 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:50.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.333+0000 7f44f5800640 1 -- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44cc067660 msgr2=0x7f44cc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.333+0000 7f44f5800640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44cc067660 0x7f44cc069b20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f44d80af000 tx=0x7f44d80a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:50.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.333+0000 7f44f5800640 1 -- 172.21.15.40:0/3659692175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f006c950 msgr2=0x7f44f00c78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.333+0000 7f44f5800640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f006c950 0x7f44f00c78a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f44e000c450 tx=0x7f44e000c910 comp rx=0 tx=0).stop 2024-03-24T00:31:50.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.333+0000 7f44f5800640 1 -- 172.21.15.40:0/3659692175 shutdown_connections 2024-03-24T00:31:50.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.333+0000 7f44f5800640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44cc067660 0x7f44cc069b20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.333+0000 7f44f5800640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f015df80 0x7f44f0157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.333+0000 7f44f5800640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f00d2270 0x7f44f0155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.333+0000 7f44f5800640 1 --2- 172.21.15.40:0/3659692175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f006c950 0x7f44f00c78a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.333+0000 7f44f5800640 1 -- 172.21.15.40:0/3659692175 >> 172.21.15.40:0/3659692175 conn(0x7f44f006ecf0 msgr2=0x7f44f0160bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:50.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.333+0000 7f44f5800640 1 -- 172.21.15.40:0/3659692175 shutdown_connections 2024-03-24T00:31:50.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.333+0000 7f44f5800640 1 -- 172.21.15.40:0/3659692175 wait complete. 2024-03-24T00:31:50.357 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:50.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f913b400640 1 Processor -- start 2024-03-24T00:31:50.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f913b400640 1 -- start start 2024-03-24T00:31:50.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f913b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91340cf4e0 0x7f913406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f913b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9134072b60 0x7f91340b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f913b400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91340724a0 0x7f91340b31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f913b400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913406cfe0 con 0x7f91340cf4e0 2024-03-24T00:31:50.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f913b400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913406d880 con 0x7f9134072b60 2024-03-24T00:31:50.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f913b400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913415f2c0 con 0x7f91340724a0 2024-03-24T00:31:50.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f9138a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91340cf4e0 0x7f913406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f9133e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9134072b60 0x7f91340b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f9138a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91340cf4e0 0x7f913406dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33908/0 (socket says 172.21.15.40:33908) 2024-03-24T00:31:50.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f9139400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91340724a0 0x7f91340b31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f9138a00640 1 -- 172.21.15.40:0/3697844457 learned_addr learned my addr 172.21.15.40:0/3697844457 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:50.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f9138a00640 1 -- 172.21.15.40:0/3697844457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91340724a0 msgr2=0x7f91340b31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:50.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f9138a00640 1 --2- 172.21.15.40:0/3697844457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91340724a0 0x7f91340b31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f9138a00640 1 -- 172.21.15.40:0/3697844457 >> v2:172.21.15.53:3300/0 conn(0x7f9134072b60 msgr2=0x7f91340b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:50.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f9138a00640 1 --2- 172.21.15.40:0/3697844457 >> v2:172.21.15.53:3300/0 conn(0x7f9134072b60 0x7f91340b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f9138a00640 1 -- 172.21.15.40:0/3697844457 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913406d230 con 0x7f91340cf4e0 2024-03-24T00:31:50.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f9133e00640 1 --2- 172.21.15.40:0/3697844457 >> v2:172.21.15.53:3300/0 conn(0x7f9134072b60 0x7f91340b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:50.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.497+0000 7f9139400640 1 --2- 172.21.15.40:0/3697844457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91340724a0 0x7f91340b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:50.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f9138a00640 1 --2- 172.21.15.40:0/3697844457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91340cf4e0 0x7f913406dd10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9134073480 tx=0x7f912801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3078c027d0edf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:50.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f9133400640 1 -- 172.21.15.40:0/3697844457 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91280b5070 con 0x7f91340cf4e0 2024-03-24T00:31:50.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f9133400640 1 -- 172.21.15.40:0/3697844457 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91280a89d0 con 0x7f91340cf4e0 2024-03-24T00:31:50.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f9133400640 1 -- 172.21.15.40:0/3697844457 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91280a8da0 con 0x7f91340cf4e0 2024-03-24T00:31:50.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f913b400640 1 -- 172.21.15.40:0/3697844457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91340cf4e0 msgr2=0x7f913406dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f913b400640 1 --2- 172.21.15.40:0/3697844457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91340cf4e0 0x7f913406dd10 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9134073480 tx=0x7f912801ed10 comp rx=0 tx=0).stop 2024-03-24T00:31:50.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f913b400640 1 -- 172.21.15.40:0/3697844457 shutdown_connections 2024-03-24T00:31:50.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f913b400640 1 --2- 172.21.15.40:0/3697844457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91340724a0 0x7f91340b31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f913b400640 1 --2- 172.21.15.40:0/3697844457 >> v2:172.21.15.53:3300/0 conn(0x7f9134072b60 0x7f91340b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f913b400640 1 --2- 172.21.15.40:0/3697844457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91340cf4e0 0x7f913406dd10 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f913b400640 1 -- 172.21.15.40:0/3697844457 >> 172.21.15.40:0/3697844457 conn(0x7f91340b1660 msgr2=0x7f91340b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:50.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f913b400640 1 -- 172.21.15.40:0/3697844457 shutdown_connections 2024-03-24T00:31:50.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.501+0000 7f913b400640 1 -- 172.21.15.40:0/3697844457 wait complete. 2024-03-24T00:31:50.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f913b400640 1 Processor -- start 2024-03-24T00:31:50.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f913b400640 1 -- start start 2024-03-24T00:31:50.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f913b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91340724a0 0x7f9134144400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f913b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9134072b60 0x7f913415d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f913b400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91340cf4e0 0x7f913415d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f913b400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913415ef20 con 0x7f91340724a0 2024-03-24T00:31:50.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f913b400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913406e250 con 0x7f9134072b60 2024-03-24T00:31:50.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f913b400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913415f0a0 con 0x7f91340cf4e0 2024-03-24T00:31:50.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9138a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91340724a0 0x7f9134144400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9138a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91340724a0 0x7f9134144400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33924/0 (socket says 172.21.15.40:33924) 2024-03-24T00:31:50.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9138a00640 1 -- 172.21.15.40:0/625645373 learned_addr learned my addr 172.21.15.40:0/625645373 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:50.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9139400640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91340cf4e0 0x7f913415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9133e00640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9134072b60 0x7f913415d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9138a00640 1 -- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91340cf4e0 msgr2=0x7f913415d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9138a00640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91340cf4e0 0x7f913415d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9138a00640 1 -- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9134072b60 msgr2=0x7f913415d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9138a00640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9134072b60 0x7f913415d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9138a00640 1 -- 172.21.15.40:0/625645373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9134153330 con 0x7f91340724a0 2024-03-24T00:31:50.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9138a00640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91340724a0 0x7f9134144400 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f912801d3c0 tx=0x7f912801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:50.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9130c00640 1 -- 172.21.15.40:0/625645373 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91280b5070 con 0x7f91340724a0 2024-03-24T00:31:50.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9130c00640 1 -- 172.21.15.40:0/625645373 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91280b08f0 con 0x7f91340724a0 2024-03-24T00:31:50.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f913b400640 1 -- 172.21.15.40:0/625645373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9134154a30 con 0x7f91340724a0 2024-03-24T00:31:50.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f9130c00640 1 -- 172.21.15.40:0/625645373 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91280b0ce0 con 0x7f91340724a0 2024-03-24T00:31:50.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.505+0000 7f913b400640 1 -- 172.21.15.40:0/625645373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9134154ff0 con 0x7f91340724a0 2024-03-24T00:31:50.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.509+0000 7f913b400640 1 -- 172.21.15.40:0/625645373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9134068170 con 0x7f91340724a0 2024-03-24T00:31:50.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.509+0000 7f9130c00640 1 -- 172.21.15.40:0/625645373 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f91280c2060 con 0x7f91340724a0 2024-03-24T00:31:50.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.509+0000 7f9130c00640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9114067660 0x7f9114069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.509+0000 7f9130c00640 1 -- 172.21.15.40:0/625645373 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f9128122830 con 0x7f91340724a0 2024-03-24T00:31:50.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.509+0000 7f9133e00640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9114067660 0x7f9114069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.509+0000 7f9133e00640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9114067660 0x7f9114069b20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9120009580 tx=0x7f9120009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:50.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.513+0000 7f9130c00640 1 -- 172.21.15.40:0/625645373 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9134153330 con 0x7f91340724a0 2024-03-24T00:31:50.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 -- 172.21.15.40:0/625645373 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f91340b13d0 con 0x7f91340724a0 2024-03-24T00:31:50.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f9130c00640 1 -- 172.21.15.40:0/625645373 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9128018870 con 0x7f91340724a0 2024-03-24T00:31:50.681 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 -- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9114067660 msgr2=0x7f9114069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9114067660 0x7f9114069b20 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9120009580 tx=0x7f9120009190 comp rx=0 tx=0).stop 2024-03-24T00:31:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 -- 172.21.15.40:0/625645373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91340724a0 msgr2=0x7f9134144400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91340724a0 0x7f9134144400 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f912801d3c0 tx=0x7f912801bab0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 -- 172.21.15.40:0/625645373 shutdown_connections 2024-03-24T00:31:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9114067660 0x7f9114069b20 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91340cf4e0 0x7f913415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9134072b60 0x7f913415d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 --2- 172.21.15.40:0/625645373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91340724a0 0x7f9134144400 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 -- 172.21.15.40:0/625645373 >> 172.21.15.40:0/625645373 conn(0x7f91340b1660 msgr2=0x7f9134071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:50.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 -- 172.21.15.40:0/625645373 shutdown_connections 2024-03-24T00:31:50.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.677+0000 7f913b400640 1 -- 172.21.15.40:0/625645373 wait complete. 2024-03-24T00:31:50.702 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:50.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.841+0000 7f859ae00640 1 Processor -- start 2024-03-24T00:31:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.841+0000 7f859ae00640 1 -- start start 2024-03-24T00:31:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f859ae00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f859415e000 0x7f85940d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f859ae00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f85941628d0 0x7f85940b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f859ae00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f859415e3e0 con 0x7f859406f840 2024-03-24T00:31:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f859ae00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8594161ba0 con 0x7f859415e000 2024-03-24T00:31:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f859ae00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85940b5f10 con 0x7f85941628d0 2024-03-24T00:31:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f8593e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f859415e000 0x7f85940d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f8593e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f859415e000 0x7f85940d16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33930/0 (socket says 172.21.15.40:33930) 2024-03-24T00:31:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f8593e00640 1 -- 172.21.15.40:0/2126274717 learned_addr learned my addr 172.21.15.40:0/2126274717 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f8598e00640 1 --1- 172.21.15.40:0/2126274717 >> v1:172.21.15.40:6789/0 conn(0x7f859406f840 0x7f8594071c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42552/0 (socket says 172.21.15.40:42552) 2024-03-24T00:31:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f858be00640 1 --2- 172.21.15.40:0/2126274717 >> v2:172.21.15.53:3300/0 conn(0x7f85941628d0 0x7f85940b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f8593e00640 1 -- 172.21.15.40:0/2126274717 >> v2:172.21.15.53:3300/0 conn(0x7f85941628d0 msgr2=0x7f85940b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f8593e00640 1 --2- 172.21.15.40:0/2126274717 >> v2:172.21.15.53:3300/0 conn(0x7f85941628d0 0x7f85940b3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f8593e00640 1 -- 172.21.15.40:0/2126274717 >> v1:172.21.15.40:6789/0 conn(0x7f859406f840 legacy=0x7f8594071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f8593e00640 1 -- 172.21.15.40:0/2126274717 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8594161df0 con 0x7f859415e000 2024-03-24T00:31:50.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.845+0000 7f8593e00640 1 --2- 172.21.15.40:0/2126274717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f859415e000 0x7f85940d16d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f859406c5a0 tx=0x7f858401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=516a73f3021c375a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:50.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f8593400640 1 -- 172.21.15.40:0/2126274717 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85840b0070 con 0x7f859415e000 2024-03-24T00:31:50.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f8593400640 1 -- 172.21.15.40:0/2126274717 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85840ab600 con 0x7f859415e000 2024-03-24T00:31:50.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f8593400640 1 -- 172.21.15.40:0/2126274717 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85840ab9d0 con 0x7f859415e000 2024-03-24T00:31:50.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 -- 172.21.15.40:0/2126274717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f859415e000 msgr2=0x7f85940d16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 --2- 172.21.15.40:0/2126274717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f859415e000 0x7f85940d16d0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f859406c5a0 tx=0x7f858401c520 comp rx=0 tx=0).stop 2024-03-24T00:31:50.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 -- 172.21.15.40:0/2126274717 shutdown_connections 2024-03-24T00:31:50.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 --2- 172.21.15.40:0/2126274717 >> v2:172.21.15.53:3300/0 conn(0x7f85941628d0 0x7f85940b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 --2- 172.21.15.40:0/2126274717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f859415e000 0x7f85940d16d0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 -- 172.21.15.40:0/2126274717 >> 172.21.15.40:0/2126274717 conn(0x7f85940d0bb0 msgr2=0x7f85940d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:50.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 -- 172.21.15.40:0/2126274717 shutdown_connections 2024-03-24T00:31:50.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 -- 172.21.15.40:0/2126274717 wait complete. 2024-03-24T00:31:50.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 Processor -- start 2024-03-24T00:31:50.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 -- start start 2024-03-24T00:31:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f859406f840 0x7f859414d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f859415e000 0x7f8594152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85941628d0 0x7f8594152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859416b0c0 con 0x7f859406f840 2024-03-24T00:31:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859416af40 con 0x7f859415e000 2024-03-24T00:31:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f859ae00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f859416b240 con 0x7f85941628d0 2024-03-24T00:31:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f8593e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f859406f840 0x7f859414d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f8593e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f859406f840 0x7f859414d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33944/0 (socket says 172.21.15.40:33944) 2024-03-24T00:31:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.849+0000 7f8593e00640 1 -- 172.21.15.40:0/2844773399 learned_addr learned my addr 172.21.15.40:0/2844773399 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f858be00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f859415e000 0x7f8594152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f8598e00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85941628d0 0x7f8594152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f8593e00640 1 -- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85941628d0 msgr2=0x7f8594152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f8593e00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85941628d0 0x7f8594152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f8593e00640 1 -- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f859415e000 msgr2=0x7f8594152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:50.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f8593e00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f859415e000 0x7f8594152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:50.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f8593e00640 1 -- 172.21.15.40:0/2844773399 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85941513e0 con 0x7f859406f840 2024-03-24T00:31:50.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f8593e00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f859406f840 0x7f859414d240 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f85840ae040 tx=0x7f85840192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:50.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f8590c00640 1 -- 172.21.15.40:0/2844773399 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85840b8020 con 0x7f859406f840 2024-03-24T00:31:50.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f8590c00640 1 -- 172.21.15.40:0/2844773399 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85840b4610 con 0x7f859406f840 2024-03-24T00:31:50.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f859ae00640 1 -- 172.21.15.40:0/2844773399 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85941516c0 con 0x7f859406f840 2024-03-24T00:31:50.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f8590c00640 1 -- 172.21.15.40:0/2844773399 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85840b0040 con 0x7f859406f840 2024-03-24T00:31:50.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f859ae00640 1 -- 172.21.15.40:0/2844773399 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8594151c80 con 0x7f859406f840 2024-03-24T00:31:50.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.853+0000 7f859ae00640 1 -- 172.21.15.40:0/2844773399 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8594068170 con 0x7f859406f840 2024-03-24T00:31:50.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.857+0000 7f8590c00640 1 -- 172.21.15.40:0/2844773399 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f85840bb020 con 0x7f859406f840 2024-03-24T00:31:50.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.857+0000 7f8590c00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8570067690 0x7f8570069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:50.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.857+0000 7f8590c00640 1 -- 172.21.15.40:0/2844773399 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f858411f280 con 0x7f859406f840 2024-03-24T00:31:50.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.857+0000 7f858be00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8570067690 0x7f8570069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:50.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.857+0000 7f858be00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8570067690 0x7f8570069b50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f85940b13d0 tx=0x7f857c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:50.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:50.861+0000 7f8590c00640 1 -- 172.21.15.40:0/2844773399 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f85840c2030 con 0x7f859406f840 2024-03-24T00:31:51.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.025+0000 7f859ae00640 1 -- 172.21.15.40:0/2844773399 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8594067b90 con 0x7f859406f840 2024-03-24T00:31:51.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.025+0000 7f8590c00640 1 -- 172.21.15.40:0/2844773399 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8584016110 con 0x7f859406f840 2024-03-24T00:31:51.031 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:51.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.029+0000 7f859ae00640 1 -- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8570067690 msgr2=0x7f8570069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.029+0000 7f859ae00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8570067690 0x7f8570069b50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f85940b13d0 tx=0x7f857c008040 comp rx=0 tx=0).stop 2024-03-24T00:31:51.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.029+0000 7f859ae00640 1 -- 172.21.15.40:0/2844773399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f859406f840 msgr2=0x7f859414d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.029+0000 7f859ae00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f859406f840 0x7f859414d240 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f85840ae040 tx=0x7f85840192c0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.029+0000 7f859ae00640 1 -- 172.21.15.40:0/2844773399 shutdown_connections 2024-03-24T00:31:51.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.029+0000 7f859ae00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8570067690 0x7f8570069b50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.029+0000 7f859ae00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85941628d0 0x7f8594152eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.029+0000 7f859ae00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f859415e000 0x7f8594152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.029+0000 7f859ae00640 1 --2- 172.21.15.40:0/2844773399 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f859406f840 0x7f859414d240 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.029+0000 7f859ae00640 1 -- 172.21.15.40:0/2844773399 >> 172.21.15.40:0/2844773399 conn(0x7f85940d0bb0 msgr2=0x7f859406dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:51.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.029+0000 7f859ae00640 1 -- 172.21.15.40:0/2844773399 shutdown_connections 2024-03-24T00:31:51.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.029+0000 7f859ae00640 1 -- 172.21.15.40:0/2844773399 wait complete. 2024-03-24T00:31:51.053 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:31:51.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.189+0000 7f4d4b600640 1 Processor -- start 2024-03-24T00:31:51.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.189+0000 7f4d4b600640 1 -- start start 2024-03-24T00:31:51.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d4b600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d4406f100 0x7f4d44070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d4b600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4d4406f4e0 0x7f4d44070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d4b600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d440d27b0 con 0x7f4d440aea60 2024-03-24T00:31:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d4b600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d440b1490 con 0x7f4d4406f4e0 2024-03-24T00:31:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d4b600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4406fa70 con 0x7f4d4406f100 2024-03-24T00:31:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d43e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4d4406f4e0 0x7f4d44070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d49600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4d440aea60 0x7f4d440aee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42554/0 (socket says 172.21.15.40:42554) 2024-03-24T00:31:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d49600640 1 -- 172.21.15.40:0/2195751044 learned_addr learned my addr 172.21.15.40:0/2195751044 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d43e00640 1 --2- 172.21.15.40:0/2195751044 >> v2:172.21.15.40:3300/0 conn(0x7f4d4406f4e0 0x7f4d44070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33948/0 (socket says 172.21.15.40:33948) 2024-03-24T00:31:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d48c00640 1 --2- 172.21.15.40:0/2195751044 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d4406f100 0x7f4d44070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:51.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d43e00640 1 -- 172.21.15.40:0/2195751044 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d4406f100 msgr2=0x7f4d44070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:51.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d43e00640 1 --2- 172.21.15.40:0/2195751044 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d4406f100 0x7f4d44070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d43e00640 1 -- 172.21.15.40:0/2195751044 >> v1:172.21.15.40:6789/0 conn(0x7f4d440aea60 legacy=0x7f4d440aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d43e00640 1 -- 172.21.15.40:0/2195751044 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4406fd50 con 0x7f4d4406f4e0 2024-03-24T00:31:51.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.193+0000 7f4d43400640 1 -- 172.21.15.40:0/2195751044 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136784581 0 0) 0x7f4d440d27b0 con 0x7f4d440aea60 2024-03-24T00:31:51.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d43e00640 1 --2- 172.21.15.40:0/2195751044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4406f4e0 0x7f4d44070030 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4d300097c0 tx=0x7f4d3001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec32062d5fa2d611 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:51.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d43400640 1 -- 172.21.15.40:0/2195751044 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d300b0070 con 0x7f4d4406f4e0 2024-03-24T00:31:51.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d43400640 1 -- 172.21.15.40:0/2195751044 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d300ab600 con 0x7f4d4406f4e0 2024-03-24T00:31:51.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d43400640 1 -- 172.21.15.40:0/2195751044 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d300ab9d0 con 0x7f4d4406f4e0 2024-03-24T00:31:51.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 -- 172.21.15.40:0/2195751044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4406f4e0 msgr2=0x7f4d44070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 --2- 172.21.15.40:0/2195751044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4406f4e0 0x7f4d44070030 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4d300097c0 tx=0x7f4d3001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 -- 172.21.15.40:0/2195751044 shutdown_connections 2024-03-24T00:31:51.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 --2- 172.21.15.40:0/2195751044 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4406f4e0 0x7f4d44070030 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 --2- 172.21.15.40:0/2195751044 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d4406f100 0x7f4d44070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 -- 172.21.15.40:0/2195751044 >> 172.21.15.40:0/2195751044 conn(0x7f4d440d1c90 msgr2=0x7f4d440d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:51.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 -- 172.21.15.40:0/2195751044 shutdown_connections 2024-03-24T00:31:51.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 -- 172.21.15.40:0/2195751044 wait complete. 2024-03-24T00:31:51.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 Processor -- start 2024-03-24T00:31:51.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 -- start start 2024-03-24T00:31:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d4406f100 0x7f4d440c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4406f4e0 0x7f4d441520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d440aea60 0x7f4d440c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440b0e10 con 0x7f4d4406f4e0 2024-03-24T00:31:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440b0c90 con 0x7f4d440aea60 2024-03-24T00:31:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.197+0000 7f4d4b600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440b0f90 con 0x7f4d4406f100 2024-03-24T00:31:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d43e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4406f4e0 0x7f4d441520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d48c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d4406f100 0x7f4d440c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d43e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4406f4e0 0x7f4d441520b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33950/0 (socket says 172.21.15.40:33950) 2024-03-24T00:31:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d43e00640 1 -- 172.21.15.40:0/3185203741 learned_addr learned my addr 172.21.15.40:0/3185203741 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d49600640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d440aea60 0x7f4d440c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:51.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d48c00640 1 -- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d440aea60 msgr2=0x7f4d440c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d48c00640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d440aea60 0x7f4d440c9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d48c00640 1 -- 172.21.15.40:0/3185203741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4406f4e0 msgr2=0x7f4d441520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d48c00640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4406f4e0 0x7f4d441520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d48c00640 1 -- 172.21.15.40:0/3185203741 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d44159250 con 0x7f4d4406f100 2024-03-24T00:31:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d43e00640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4406f4e0 0x7f4d441520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d48c00640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d4406f100 0x7f4d440c6ae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800e870 tx=0x7f4d3800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d40c00640 1 -- 172.21.15.40:0/3185203741 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d38017070 con 0x7f4d4406f100 2024-03-24T00:31:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d40c00640 1 -- 172.21.15.40:0/3185203741 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3800c940 con 0x7f4d4406f100 2024-03-24T00:31:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d40c00640 1 -- 172.21.15.40:0/3185203741 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3800ccf0 con 0x7f4d4406f100 2024-03-24T00:31:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d4b600640 1 -- 172.21.15.40:0/3185203741 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d440c7110 con 0x7f4d4406f100 2024-03-24T00:31:51.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d4b600640 1 -- 172.21.15.40:0/3185203741 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d440c76d0 con 0x7f4d4406f100 2024-03-24T00:31:51.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d4b600640 1 -- 172.21.15.40:0/3185203741 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d44068170 con 0x7f4d4406f100 2024-03-24T00:31:51.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.201+0000 7f4d40c00640 1 -- 172.21.15.40:0/3185203741 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4d3801a070 con 0x7f4d4406f100 2024-03-24T00:31:51.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.205+0000 7f4d40c00640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4d24067690 0x7f4d24069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:51.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.205+0000 7f4d40c00640 1 -- 172.21.15.40:0/3185203741 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f4d38085c20 con 0x7f4d4406f100 2024-03-24T00:31:51.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.205+0000 7f4d43e00640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4d24067690 0x7f4d24069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:51.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.205+0000 7f4d43e00640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4d24067690 0x7f4d24069b50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4d300097c0 tx=0x7f4d30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:51.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.209+0000 7f4d40c00640 1 -- 172.21.15.40:0/3185203741 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4d38056130 con 0x7f4d4406f100 2024-03-24T00:31:51.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.385+0000 7f4d4b600640 1 -- 172.21.15.40:0/3185203741 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4d4406ddc0 con 0x7f4d4406f100 2024-03-24T00:31:51.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.385+0000 7f4d40c00640 1 -- 172.21.15.40:0/3185203741 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4d3805a1d0 con 0x7f4d4406f100 2024-03-24T00:31:51.391 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:31:51.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.389+0000 7f4d4b600640 1 -- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4d24067690 msgr2=0x7f4d24069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.389+0000 7f4d4b600640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4d24067690 0x7f4d24069b50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4d300097c0 tx=0x7f4d30008040 comp rx=0 tx=0).stop 2024-03-24T00:31:51.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.389+0000 7f4d4b600640 1 -- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d4406f100 msgr2=0x7f4d440c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.389+0000 7f4d4b600640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d4406f100 0x7f4d440c6ae0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800e870 tx=0x7f4d3800ed30 comp rx=0 tx=0).stop 2024-03-24T00:31:51.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.389+0000 7f4d4b600640 1 -- 172.21.15.40:0/3185203741 shutdown_connections 2024-03-24T00:31:51.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.389+0000 7f4d4b600640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4d24067690 0x7f4d24069b50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.389+0000 7f4d4b600640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d440aea60 0x7f4d440c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.389+0000 7f4d4b600640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d4406f4e0 0x7f4d441520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.389+0000 7f4d4b600640 1 --2- 172.21.15.40:0/3185203741 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d4406f100 0x7f4d440c6ae0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.389+0000 7f4d4b600640 1 -- 172.21.15.40:0/3185203741 >> 172.21.15.40:0/3185203741 conn(0x7f4d440d1c90 msgr2=0x7f4d440aee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:51.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.389+0000 7f4d4b600640 1 -- 172.21.15.40:0/3185203741 shutdown_connections 2024-03-24T00:31:51.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.389+0000 7f4d4b600640 1 -- 172.21.15.40:0/3185203741 wait complete. 2024-03-24T00:31:51.412 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi053.front.sepia.ceph.com... 2024-03-24T00:31:51.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:51.412 DEBUG:teuthology.orchestra.run.smithi053:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-24T00:31:51.437 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:31:51.437 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:31:51.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:31:51.438 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T00:31:51.489 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:31:51.489 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:31:51.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe064400640 1 Processor -- start 2024-03-24T00:31:51.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe064400640 1 -- start start 2024-03-24T00:31:51.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe064400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe05c15f600 0x7fe05c0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:51.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe064400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe05c0d2830 0x7fe05c0aea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:51.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe064400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c0b1e50 con 0x7fe05c0aefa0 2024-03-24T00:31:51.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe064400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe05c0b3630 con 0x7fe05c15f600 2024-03-24T00:31:51.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe064400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0d2ca0 con 0x7fe05c0d2830 2024-03-24T00:31:51.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe061a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe05c15f600 0x7fe05c0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:51.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe061a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe05c15f600 0x7fe05c0b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33958/0 (socket says 172.21.15.40:33958) 2024-03-24T00:31:51.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe061a00640 1 -- 172.21.15.40:0/2731045908 learned_addr learned my addr 172.21.15.40:0/2731045908 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:51.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe061000640 1 --2- 172.21.15.40:0/2731045908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe05c0d2830 0x7fe05c0aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:51.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe061a00640 1 -- 172.21.15.40:0/2731045908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe05c0d2830 msgr2=0x7fe05c0aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:51.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe061a00640 1 --2- 172.21.15.40:0/2731045908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe05c0d2830 0x7fe05c0aea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe061a00640 1 -- 172.21.15.40:0/2731045908 >> v1:172.21.15.53:6789/0 conn(0x7fe05c0aefa0 legacy=0x7fe05c164e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.625+0000 7fe061a00640 1 -- 172.21.15.40:0/2731045908 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c0b20f0 con 0x7fe05c15f600 2024-03-24T00:31:51.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.629+0000 7fe061a00640 1 --2- 172.21.15.40:0/2731045908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe05c15f600 0x7fe05c0b2910 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe050009870 tx=0x7fe05001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c452d62f23915db server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:51.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.629+0000 7fe05be00640 1 -- 172.21.15.40:0/2731045908 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0500b5070 con 0x7fe05c15f600 2024-03-24T00:31:51.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.629+0000 7fe05be00640 1 -- 172.21.15.40:0/2731045908 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0500a8a50 con 0x7fe05c15f600 2024-03-24T00:31:51.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.629+0000 7fe05be00640 1 -- 172.21.15.40:0/2731045908 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0500a8e20 con 0x7fe05c15f600 2024-03-24T00:31:51.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.629+0000 7fe064400640 1 -- 172.21.15.40:0/2731045908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe05c15f600 msgr2=0x7fe05c0b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.629+0000 7fe064400640 1 --2- 172.21.15.40:0/2731045908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe05c15f600 0x7fe05c0b2910 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe050009870 tx=0x7fe05001ed50 comp rx=0 tx=0).stop 2024-03-24T00:31:51.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.629+0000 7fe064400640 1 -- 172.21.15.40:0/2731045908 shutdown_connections 2024-03-24T00:31:51.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.629+0000 7fe064400640 1 --2- 172.21.15.40:0/2731045908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe05c0d2830 0x7fe05c0aea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.629+0000 7fe064400640 1 --2- 172.21.15.40:0/2731045908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe05c15f600 0x7fe05c0b2910 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.629+0000 7fe064400640 1 -- 172.21.15.40:0/2731045908 >> 172.21.15.40:0/2731045908 conn(0x7fe05c0d1d10 msgr2=0x7fe05c0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:51.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.629+0000 7fe064400640 1 -- 172.21.15.40:0/2731045908 shutdown_connections 2024-03-24T00:31:51.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe064400640 1 -- 172.21.15.40:0/2731045908 wait complete. 2024-03-24T00:31:51.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe064400640 1 Processor -- start 2024-03-24T00:31:51.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe064400640 1 -- start start 2024-03-24T00:31:51.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe064400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe05c0aefa0 0x7fe05c0ca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:51.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe064400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe05c0d2830 0x7fe05c09ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:51.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe064400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe05c15f600 0x7fe05c15db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:51.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe064400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0b2fa0 con 0x7fe05c15f600 2024-03-24T00:31:51.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe064400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0b2740 con 0x7fe05c0aefa0 2024-03-24T00:31:51.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe064400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c15f9e0 con 0x7fe05c0d2830 2024-03-24T00:31:51.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe062400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe05c15f600 0x7fe05c15db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:51.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe062400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe05c15f600 0x7fe05c15db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33970/0 (socket says 172.21.15.40:33970) 2024-03-24T00:31:51.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe062400640 1 -- 172.21.15.40:0/1525003603 learned_addr learned my addr 172.21.15.40:0/1525003603 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:51.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe061a00640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe05c0aefa0 0x7fe05c0ca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:51.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe061000640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe05c0d2830 0x7fe05c09ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:51.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe062400640 1 -- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe05c0d2830 msgr2=0x7fe05c09ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe062400640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe05c0d2830 0x7fe05c09ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe062400640 1 -- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe05c0aefa0 msgr2=0x7fe05c0ca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe062400640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe05c0aefa0 0x7fe05c0ca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe062400640 1 -- 172.21.15.40:0/1525003603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c0cbd40 con 0x7fe05c15f600 2024-03-24T00:31:51.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe062400640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe05c15f600 0x7fe05c15db80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe05400c7a0 tx=0x7fe05400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:51.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.633+0000 7fe059600640 1 -- 172.21.15.40:0/1525003603 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe054003b80 con 0x7fe05c15f600 2024-03-24T00:31:51.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.637+0000 7fe059600640 1 -- 172.21.15.40:0/1525003603 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe054003d20 con 0x7fe05c15f600 2024-03-24T00:31:51.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.637+0000 7fe064400640 1 -- 172.21.15.40:0/1525003603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe05c0cc020 con 0x7fe05c15f600 2024-03-24T00:31:51.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.637+0000 7fe059600640 1 -- 172.21.15.40:0/1525003603 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05400aa90 con 0x7fe05c15f600 2024-03-24T00:31:51.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.637+0000 7fe064400640 1 -- 172.21.15.40:0/1525003603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05c0ca1b0 con 0x7fe05c15f600 2024-03-24T00:31:51.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.637+0000 7fe064400640 1 -- 172.21.15.40:0/1525003603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05c068170 con 0x7fe05c15f600 2024-03-24T00:31:51.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.637+0000 7fe059600640 1 -- 172.21.15.40:0/1525003603 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe05400ac30 con 0x7fe05c15f600 2024-03-24T00:31:51.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.637+0000 7fe059600640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe03c067690 0x7fe03c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:51.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.637+0000 7fe059600640 1 -- 172.21.15.40:0/1525003603 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fe054086090 con 0x7fe05c15f600 2024-03-24T00:31:51.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.637+0000 7fe061a00640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe03c067690 0x7fe03c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:51.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.641+0000 7fe061a00640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe03c067690 0x7fe03c069b50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0500b2000 tx=0x7fe050009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:51.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.641+0000 7fe059600640 1 -- 172.21.15.40:0/1525003603 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe0540567b0 con 0x7fe05c15f600 2024-03-24T00:31:51.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.837+0000 7fe064400640 1 -- 172.21.15.40:0/1525003603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe05c0b1720 con 0x7fe05c15f600 2024-03-24T00:31:51.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe059600640 1 -- 172.21.15.40:0/1525003603 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4965 (secure 0 0 0) 0x7fe05405a850 con 0x7fe05c15f600 2024-03-24T00:31:51.844 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:51.844 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/501744669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":501744669},{"type":"v1","addr":"172.21.15.40:6833","nonce":501744669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/281354690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":281354690},{"type":"v1","addr":"172.21.15.53:6836","nonce":281354690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3092758821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3092758821},{"type":"v1","addr":"172.21.15.53:6837","nonce":3092758821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:30:59.427652+0000","modified":"2024-03-24T00:31:02.444298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4139303536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":4139303536},{"type":"v1","addr":"172.21.15.40:6835","nonce":4139303536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-24T00:31:51.844 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 8 2024-03-24T00:31:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe064400640 1 -- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe03c067690 msgr2=0x7fe03c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe064400640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe03c067690 0x7fe03c069b50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0500b2000 tx=0x7fe050009040 comp rx=0 tx=0).stop 2024-03-24T00:31:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe064400640 1 -- 172.21.15.40:0/1525003603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe05c15f600 msgr2=0x7fe05c15db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe064400640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe05c15f600 0x7fe05c15db80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe05400c7a0 tx=0x7fe05400cc60 comp rx=0 tx=0).stop 2024-03-24T00:31:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe064400640 1 -- 172.21.15.40:0/1525003603 shutdown_connections 2024-03-24T00:31:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe064400640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe03c067690 0x7fe03c069b50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe064400640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe05c15f600 0x7fe05c15db80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe064400640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe05c0d2830 0x7fe05c09ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe064400640 1 --2- 172.21.15.40:0/1525003603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe05c0aefa0 0x7fe05c0ca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:51.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe064400640 1 -- 172.21.15.40:0/1525003603 >> 172.21.15.40:0/1525003603 conn(0x7fe05c0d1d10 msgr2=0x7fe05c0cea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:51.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe064400640 1 -- 172.21.15.40:0/1525003603 shutdown_connections 2024-03-24T00:31:51.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:51.841+0000 7fe064400640 1 -- 172.21.15.40:0/1525003603 wait complete. 2024-03-24T00:31:51.865 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff189800640 1 Processor -- start 2024-03-24T00:31:52.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff189800640 1 -- start start 2024-03-24T00:31:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff189800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff1840b46c0 0x7ff1840b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff189800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff1840b2ff0 0x7ff1840ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff189800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff184070f60 con 0x7ff18406e620 2024-03-24T00:31:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff189800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1840b3400 con 0x7ff1840b2ff0 2024-03-24T00:31:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff189800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1840b3620 con 0x7ff1840b46c0 2024-03-24T00:31:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff183400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff1840b2ff0 0x7ff1840ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff182a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff18406e620 0x7ff18406ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42568/0 (socket says 172.21.15.40:42568) 2024-03-24T00:31:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff182a00640 1 -- 172.21.15.40:0/256587672 learned_addr learned my addr 172.21.15.40:0/256587672 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff17a000640 1 --2- 172.21.15.40:0/256587672 >> v2:172.21.15.53:3300/0 conn(0x7ff1840b46c0 0x7ff1840b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff183400640 1 -- 172.21.15.40:0/256587672 >> v2:172.21.15.53:3300/0 conn(0x7ff1840b46c0 msgr2=0x7ff1840b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:52.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff183400640 1 --2- 172.21.15.40:0/256587672 >> v2:172.21.15.53:3300/0 conn(0x7ff1840b46c0 0x7ff1840b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff183400640 1 -- 172.21.15.40:0/256587672 >> v1:172.21.15.40:6789/0 conn(0x7ff18406e620 legacy=0x7ff18406ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff183400640 1 -- 172.21.15.40:0/256587672 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840b3900 con 0x7ff1840b2ff0 2024-03-24T00:31:52.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.001+0000 7ff182000640 1 -- 172.21.15.40:0/256587672 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344152965 0 0) 0x7ff184070f60 con 0x7ff18406e620 2024-03-24T00:31:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.005+0000 7ff183400640 1 --2- 172.21.15.40:0/256587672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1840b2ff0 0x7ff1840ccfa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff18406cea0 tx=0x7ff17c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb1c47cf6eefb3c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.005+0000 7ff182000640 1 -- 172.21.15.40:0/256587672 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17c0b1070 con 0x7ff1840b2ff0 2024-03-24T00:31:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.005+0000 7ff182000640 1 -- 172.21.15.40:0/256587672 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17c0ac620 con 0x7ff1840b2ff0 2024-03-24T00:31:52.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.005+0000 7ff182000640 1 -- 172.21.15.40:0/256587672 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1840cdcf0 con 0x7ff1840b2ff0 2024-03-24T00:31:52.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.005+0000 7ff189800640 1 -- 172.21.15.40:0/256587672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1840b2ff0 msgr2=0x7ff1840ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.005+0000 7ff189800640 1 --2- 172.21.15.40:0/256587672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1840b2ff0 0x7ff1840ccfa0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff18406cea0 tx=0x7ff17c01c3c0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.005+0000 7ff189800640 1 -- 172.21.15.40:0/256587672 shutdown_connections 2024-03-24T00:31:52.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.005+0000 7ff189800640 1 --2- 172.21.15.40:0/256587672 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1840b2ff0 0x7ff1840ccfa0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.005+0000 7ff189800640 1 --2- 172.21.15.40:0/256587672 >> v2:172.21.15.53:3300/0 conn(0x7ff1840b46c0 0x7ff1840b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.005+0000 7ff189800640 1 -- 172.21.15.40:0/256587672 >> 172.21.15.40:0/256587672 conn(0x7ff1840b17b0 msgr2=0x7ff1840b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:52.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.005+0000 7ff189800640 1 -- 172.21.15.40:0/256587672 shutdown_connections 2024-03-24T00:31:52.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.005+0000 7ff189800640 1 -- 172.21.15.40:0/256587672 wait complete. 2024-03-24T00:31:52.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff189800640 1 Processor -- start 2024-03-24T00:31:52.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff189800640 1 -- start start 2024-03-24T00:31:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff189800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff18406e620 0x7ff184164850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff189800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1840b2ff0 0x7ff1841faad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff189800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1840b46c0 0x7ff1841fb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff189800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff184144920 con 0x7ff1840b2ff0 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff189800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840b4ad0 con 0x7ff1840b46c0 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff189800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff184164ee0 con 0x7ff18406e620 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff17a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1840b2ff0 0x7ff1841faad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff182a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff18406e620 0x7ff184164850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff17a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1840b2ff0 0x7ff1841faad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33988/0 (socket says 172.21.15.40:33988) 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff17a000640 1 -- 172.21.15.40:0/2183519035 learned_addr learned my addr 172.21.15.40:0/2183519035 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff183400640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1840b46c0 0x7ff1841fb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff17a000640 1 -- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff18406e620 msgr2=0x7ff184164850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff17a000640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff18406e620 0x7ff184164850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff17a000640 1 -- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1840b46c0 msgr2=0x7ff1841fb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff17a000640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1840b46c0 0x7ff1841fb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff17a000640 1 -- 172.21.15.40:0/2183519035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1841651c0 con 0x7ff1840b2ff0 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff183400640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1840b46c0 0x7ff1841fb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:52.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff17a000640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1840b2ff0 0x7ff1841faad0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00d800 tx=0x7ff16c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff17b400640 1 -- 172.21.15.40:0/2183519035 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c00b6c0 con 0x7ff1840b2ff0 2024-03-24T00:31:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff17b400640 1 -- 172.21.15.40:0/2183519035 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16c019070 con 0x7ff1840b2ff0 2024-03-24T00:31:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff189800640 1 -- 172.21.15.40:0/2183519035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1841654a0 con 0x7ff1840b2ff0 2024-03-24T00:31:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff17b400640 1 -- 172.21.15.40:0/2183519035 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c00ba60 con 0x7ff1840b2ff0 2024-03-24T00:31:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.009+0000 7ff189800640 1 -- 172.21.15.40:0/2183519035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1841f8740 con 0x7ff1840b2ff0 2024-03-24T00:31:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.013+0000 7ff189800640 1 -- 172.21.15.40:0/2183519035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff184068170 con 0x7ff1840b2ff0 2024-03-24T00:31:52.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.013+0000 7ff17b400640 1 -- 172.21.15.40:0/2183519035 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff16c00bc00 con 0x7ff1840b2ff0 2024-03-24T00:31:52.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.013+0000 7ff17b400640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff160067660 0x7ff160069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.013+0000 7ff17b400640 1 -- 172.21.15.40:0/2183519035 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7ff16c085ca0 con 0x7ff1840b2ff0 2024-03-24T00:31:52.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.013+0000 7ff182a00640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff160067660 0x7ff160069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.013+0000 7ff182a00640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff160067660 0x7ff160069b20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff1740012f0 tx=0x7ff174009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:52.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.017+0000 7ff17b400640 1 -- 172.21.15.40:0/2183519035 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff16c056250 con 0x7ff1840b2ff0 2024-03-24T00:31:52.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.181+0000 7ff189800640 1 -- 172.21.15.40:0/2183519035 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff184072f50 con 0x7ff1840b2ff0 2024-03-24T00:31:52.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.181+0000 7ff17b400640 1 -- 172.21.15.40:0/2183519035 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16813 (secure 0 0 0) 0x7ff16c05a2f0 con 0x7ff1840b2ff0 2024-03-24T00:31:52.187 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:52.187 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":22,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:48.417423+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:52.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.185+0000 7ff189800640 1 -- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff160067660 msgr2=0x7ff160069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.185+0000 7ff189800640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff160067660 0x7ff160069b20 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff1740012f0 tx=0x7ff174009040 comp rx=0 tx=0).stop 2024-03-24T00:31:52.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.185+0000 7ff189800640 1 -- 172.21.15.40:0/2183519035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1840b2ff0 msgr2=0x7ff1841faad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.185+0000 7ff189800640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1840b2ff0 0x7ff1841faad0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff16c00d800 tx=0x7ff16c00dcc0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.185+0000 7ff189800640 1 -- 172.21.15.40:0/2183519035 shutdown_connections 2024-03-24T00:31:52.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.185+0000 7ff189800640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff160067660 0x7ff160069b20 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.185+0000 7ff189800640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1840b46c0 0x7ff1841fb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.185+0000 7ff189800640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1840b2ff0 0x7ff1841faad0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.185+0000 7ff189800640 1 --2- 172.21.15.40:0/2183519035 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff18406e620 0x7ff184164850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.185+0000 7ff189800640 1 -- 172.21.15.40:0/2183519035 >> 172.21.15.40:0/2183519035 conn(0x7ff1840b17b0 msgr2=0x7ff1841417f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:52.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.185+0000 7ff189800640 1 -- 172.21.15.40:0/2183519035 shutdown_connections 2024-03-24T00:31:52.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.185+0000 7ff189800640 1 -- 172.21.15.40:0/2183519035 wait complete. 2024-03-24T00:31:52.209 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:30:57.724304+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:30:58.418950+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:31:52.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:31:52.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5aa200640 1 Processor -- start 2024-03-24T00:31:52.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5aa200640 1 -- start start 2024-03-24T00:31:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5aa200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5a415b960 0x7fb5a4070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5aa200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5a415bd40 0x7fb5a40712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5aa200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a415f970 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5aa200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a4160580 con 0x7fb5a415b960 2024-03-24T00:31:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5aa200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a415fc50 con 0x7fb5a415bd40 2024-03-24T00:31:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5a3400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5a415b960 0x7fb5a4070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5a3400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5a415b960 0x7fb5a4070d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33996/0 (socket says 172.21.15.40:33996) 2024-03-24T00:31:52.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5a3400640 1 -- 172.21.15.40:0/2609103577 learned_addr learned my addr 172.21.15.40:0/2609103577 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb59aa00640 1 --2- 172.21.15.40:0/2609103577 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5a415bd40 0x7fb5a40712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5a3400640 1 -- 172.21.15.40:0/2609103577 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5a415bd40 msgr2=0x7fb5a40712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5a3400640 1 --2- 172.21.15.40:0/2609103577 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5a415bd40 0x7fb5a40712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5a3400640 1 -- 172.21.15.40:0/2609103577 >> v1:172.21.15.53:6789/0 conn(0x7fb5a40d0fe0 legacy=0x7fb5a40cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.349+0000 7fb5a3400640 1 -- 172.21.15.40:0/2609103577 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a415ff30 con 0x7fb5a415b960 2024-03-24T00:31:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5a3400640 1 --2- 172.21.15.40:0/2609103577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a415b960 0x7fb5a4070d60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb594009870 tx=0x7fb59401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bd69021f7f18cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5a2a00640 1 -- 172.21.15.40:0/2609103577 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5940b2070 con 0x7fb5a415b960 2024-03-24T00:31:52.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5a2a00640 1 -- 172.21.15.40:0/2609103577 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5940a9070 con 0x7fb5a415b960 2024-03-24T00:31:52.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5a2a00640 1 -- 172.21.15.40:0/2609103577 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5940ad880 con 0x7fb5a415b960 2024-03-24T00:31:52.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5aa200640 1 -- 172.21.15.40:0/2609103577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a415b960 msgr2=0x7fb5a4070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5aa200640 1 --2- 172.21.15.40:0/2609103577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a415b960 0x7fb5a4070d60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb594009870 tx=0x7fb59401c560 comp rx=0 tx=0).stop 2024-03-24T00:31:52.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5aa200640 1 -- 172.21.15.40:0/2609103577 shutdown_connections 2024-03-24T00:31:52.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5aa200640 1 --2- 172.21.15.40:0/2609103577 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5a415bd40 0x7fb5a40712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5aa200640 1 --2- 172.21.15.40:0/2609103577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a415b960 0x7fb5a4070d60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5aa200640 1 -- 172.21.15.40:0/2609103577 >> 172.21.15.40:0/2609103577 conn(0x7fb5a406ecf0 msgr2=0x7fb5a406f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:52.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5aa200640 1 -- 172.21.15.40:0/2609103577 shutdown_connections 2024-03-24T00:31:52.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5aa200640 1 -- 172.21.15.40:0/2609103577 wait complete. 2024-03-24T00:31:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5aa200640 1 Processor -- start 2024-03-24T00:31:52.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.357+0000 7fb5aa200640 1 -- start start 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5aa200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5a40d0fe0 0x7fb5a40c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5aa200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a415b960 0x7fb5a414d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5aa200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5a415bd40 0x7fb5a40c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5aa200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a4168f00 con 0x7fb5a415b960 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5aa200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a4168d80 con 0x7fb5a415bd40 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5aa200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a4169080 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb59aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a415b960 0x7fb5a414d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb59aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a415b960 0x7fb5a414d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34006/0 (socket says 172.21.15.40:34006) 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5a3400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5a40d0fe0 0x7fb5a40c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb59aa00640 1 -- 172.21.15.40:0/3495330587 learned_addr learned my addr 172.21.15.40:0/3495330587 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5a3e00640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5a415bd40 0x7fb5a40c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5a3400640 1 -- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5a415bd40 msgr2=0x7fb5a40c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5a3400640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5a415bd40 0x7fb5a40c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5a3400640 1 -- 172.21.15.40:0/3495330587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a415b960 msgr2=0x7fb5a414d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5a3400640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a415b960 0x7fb5a414d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5a3400640 1 -- 172.21.15.40:0/3495330587 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a414d5e0 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb59aa00640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a415b960 0x7fb5a414d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5a3400640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5a40d0fe0 0x7fb5a40c5500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb5940ac000 tx=0x7fb59401ada0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb59be00640 1 -- 172.21.15.40:0/3495330587 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5940bc020 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb59be00640 1 -- 172.21.15.40:0/3495330587 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5940b8660 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb59be00640 1 -- 172.21.15.40:0/3495330587 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5940b2040 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5aa200640 1 -- 172.21.15.40:0/3495330587 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a40c5ae0 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5aa200640 1 -- 172.21.15.40:0/3495330587 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a40c60a0 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.361+0000 7fb5aa200640 1 -- 172.21.15.40:0/3495330587 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a4068170 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.365+0000 7fb59be00640 1 -- 172.21.15.40:0/3495330587 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb5940bf020 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.365+0000 7fb59be00640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb580067660 0x7fb580069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.365+0000 7fb59be00640 1 -- 172.21.15.40:0/3495330587 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fb59411fc70 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.365+0000 7fb59aa00640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb580067660 0x7fb580069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.365+0000 7fb59aa00640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb580067660 0x7fb580069b20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4155dc0 tx=0x7fb58c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:52.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.369+0000 7fb59be00640 1 -- 172.21.15.40:0/3495330587 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb5940c6030 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.529+0000 7fb5aa200640 1 -- 172.21.15.40:0/3495330587 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb5a4070910 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb59be00640 1 -- 172.21.15.40:0/3495330587 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb594016150 con 0x7fb5a40d0fe0 2024-03-24T00:31:52.536 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:31:52.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb5aa200640 1 -- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb580067660 msgr2=0x7fb580069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb5aa200640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb580067660 0x7fb580069b20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4155dc0 tx=0x7fb58c008040 comp rx=0 tx=0).stop 2024-03-24T00:31:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb5aa200640 1 -- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5a40d0fe0 msgr2=0x7fb5a40c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb5aa200640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5a40d0fe0 0x7fb5a40c5500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb5940ac000 tx=0x7fb59401ada0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb5aa200640 1 -- 172.21.15.40:0/3495330587 shutdown_connections 2024-03-24T00:31:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb5aa200640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb580067660 0x7fb580069b20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb5aa200640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5a415bd40 0x7fb5a40c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb5aa200640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5a415b960 0x7fb5a414d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb5aa200640 1 --2- 172.21.15.40:0/3495330587 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5a40d0fe0 0x7fb5a40c5500 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb5aa200640 1 -- 172.21.15.40:0/3495330587 >> 172.21.15.40:0/3495330587 conn(0x7fb5a406ecf0 msgr2=0x7fb5a406c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb5aa200640 1 -- 172.21.15.40:0/3495330587 shutdown_connections 2024-03-24T00:31:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.533+0000 7fb5aa200640 1 -- 172.21.15.40:0/3495330587 wait complete. 2024-03-24T00:31:52.557 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:31:52.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d96200640 1 Processor -- start 2024-03-24T00:31:52.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d96200640 1 -- start start 2024-03-24T00:31:52.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d96200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1d900d2270 0x7f1d90070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d96200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d9006c950 0x7f1d90161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d96200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9006d0f0 con 0x7f1d9015df80 2024-03-24T00:31:52.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d96200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9006b890 con 0x7f1d900d2270 2024-03-24T00:31:52.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d96200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9006b280 con 0x7f1d9006c950 2024-03-24T00:31:52.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d87e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1d900d2270 0x7f1d90070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d87e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1d900d2270 0x7f1d90070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34022/0 (socket says 172.21.15.40:34022) 2024-03-24T00:31:52.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d87e00640 1 -- 172.21.15.40:0/1843830374 learned_addr learned my addr 172.21.15.40:0/1843830374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d8fe00640 1 --2- 172.21.15.40:0/1843830374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d9006c950 0x7f1d90161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d87e00640 1 -- 172.21.15.40:0/1843830374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d9006c950 msgr2=0x7f1d90161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d87e00640 1 --2- 172.21.15.40:0/1843830374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d9006c950 0x7f1d90161ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d87e00640 1 -- 172.21.15.40:0/1843830374 >> v1:172.21.15.53:6789/0 conn(0x7f1d9015df80 legacy=0x7f1d900d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.697+0000 7f1d87e00640 1 -- 172.21.15.40:0/1843830374 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9006ce10 con 0x7f1d900d2270 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.701+0000 7f1d87e00640 1 --2- 172.21.15.40:0/1843830374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d900d2270 0x7f1d90070ff0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009870 tx=0x7f1d7801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abfde1d321f4fd9a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.701+0000 7f1d8ea00640 1 -- 172.21.15.40:0/1843830374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d780aa650 con 0x7f1d900d2270 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.701+0000 7f1d8ea00640 1 -- 172.21.15.40:0/1843830374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d780af070 con 0x7f1d900d2270 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.701+0000 7f1d8ea00640 1 -- 172.21.15.40:0/1843830374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d780aaa10 con 0x7f1d900d2270 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.701+0000 7f1d96200640 1 -- 172.21.15.40:0/1843830374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d900d2270 msgr2=0x7f1d90070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.701+0000 7f1d96200640 1 --2- 172.21.15.40:0/1843830374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d900d2270 0x7f1d90070ff0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1d78009870 tx=0x7f1d7801c530 comp rx=0 tx=0).stop 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.701+0000 7f1d96200640 1 -- 172.21.15.40:0/1843830374 shutdown_connections 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.701+0000 7f1d96200640 1 --2- 172.21.15.40:0/1843830374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d9006c950 0x7f1d90161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.701+0000 7f1d96200640 1 --2- 172.21.15.40:0/1843830374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d900d2270 0x7f1d90070ff0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.701+0000 7f1d96200640 1 -- 172.21.15.40:0/1843830374 >> 172.21.15.40:0/1843830374 conn(0x7f1d9006ecf0 msgr2=0x7f1d9006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.701+0000 7f1d96200640 1 -- 172.21.15.40:0/1843830374 shutdown_connections 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.701+0000 7f1d96200640 1 -- 172.21.15.40:0/1843830374 wait complete. 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d96200640 1 Processor -- start 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d96200640 1 -- start start 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d96200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d9006c950 0x7f1d9015bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d96200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d900d2270 0x7f1d900c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d96200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d9015df80 0x7f1d900b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d96200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9016af90 con 0x7f1d900d2270 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d96200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9016ae10 con 0x7f1d9006c950 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d96200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9016b110 con 0x7f1d9015df80 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d87e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d900d2270 0x7f1d900c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d87e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d900d2270 0x7f1d900c9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34038/0 (socket says 172.21.15.40:34038) 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d87e00640 1 -- 172.21.15.40:0/2760862293 learned_addr learned my addr 172.21.15.40:0/2760862293 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:52.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d8f400640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d9006c950 0x7f1d9015bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d8fe00640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d9015df80 0x7f1d900b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d87e00640 1 -- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d9015df80 msgr2=0x7f1d900b47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d87e00640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d9015df80 0x7f1d900b47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d87e00640 1 -- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d9006c950 msgr2=0x7f1d9015bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d87e00640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d9006c950 0x7f1d9015bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d87e00640 1 -- 172.21.15.40:0/2760862293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d900cc9f0 con 0x7f1d900d2270 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d8f400640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d9006c950 0x7f1d9015bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d87e00640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d900d2270 0x7f1d900c9680 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1d7801a160 tx=0x7f1d7801ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d87400640 1 -- 172.21.15.40:0/2760862293 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d780bd5f0 con 0x7f1d900d2270 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d87400640 1 -- 172.21.15.40:0/2760862293 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d780a6040 con 0x7f1d900d2270 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d96200640 1 -- 172.21.15.40:0/2760862293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d900b6d30 con 0x7f1d900d2270 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d87400640 1 -- 172.21.15.40:0/2760862293 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d780bd9d0 con 0x7f1d900d2270 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d96200640 1 -- 172.21.15.40:0/2760862293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d900cccb0 con 0x7f1d900d2270 2024-03-24T00:31:52.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.705+0000 7f1d96200640 1 -- 172.21.15.40:0/2760862293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d90068170 con 0x7f1d900d2270 2024-03-24T00:31:52.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.709+0000 7f1d87400640 1 -- 172.21.15.40:0/2760862293 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f1d780b33f0 con 0x7f1d900d2270 2024-03-24T00:31:52.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.709+0000 7f1d87400640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1d6c067660 0x7f1d6c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:52.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.709+0000 7f1d87400640 1 -- 172.21.15.40:0/2760862293 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f1d7811feb0 con 0x7f1d900d2270 2024-03-24T00:31:52.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.709+0000 7f1d8f400640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1d6c067660 0x7f1d6c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:52.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.709+0000 7f1d8f400640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1d6c067660 0x7f1d6c069b20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1d9015c700 tx=0x7f1d80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:52.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.717+0000 7f1d87400640 1 -- 172.21.15.40:0/2760862293 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1d780c3060 con 0x7f1d900d2270 2024-03-24T00:31:52.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.881+0000 7f1d96200640 1 -- 172.21.15.40:0/2760862293 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1d900cf9d0 con 0x7f1d900d2270 2024-03-24T00:31:52.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.881+0000 7f1d87400640 1 -- 172.21.15.40:0/2760862293 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1d78016150 con 0x7f1d900d2270 2024-03-24T00:31:52.887 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:31:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.885+0000 7f1d96200640 1 -- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1d6c067660 msgr2=0x7f1d6c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.885+0000 7f1d96200640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1d6c067660 0x7f1d6c069b20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1d9015c700 tx=0x7f1d80009040 comp rx=0 tx=0).stop 2024-03-24T00:31:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.885+0000 7f1d96200640 1 -- 172.21.15.40:0/2760862293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d900d2270 msgr2=0x7f1d900c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.885+0000 7f1d96200640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d900d2270 0x7f1d900c9680 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1d7801a160 tx=0x7f1d7801ad70 comp rx=0 tx=0).stop 2024-03-24T00:31:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.885+0000 7f1d96200640 1 -- 172.21.15.40:0/2760862293 shutdown_connections 2024-03-24T00:31:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.885+0000 7f1d96200640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1d6c067660 0x7f1d6c069b20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.885+0000 7f1d96200640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d9015df80 0x7f1d900b47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.885+0000 7f1d96200640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d900d2270 0x7f1d900c9680 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.885+0000 7f1d96200640 1 --2- 172.21.15.40:0/2760862293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d9006c950 0x7f1d9015bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:52.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.885+0000 7f1d96200640 1 -- 172.21.15.40:0/2760862293 >> 172.21.15.40:0/2760862293 conn(0x7f1d9006ecf0 msgr2=0x7f1d90160ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:52.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.885+0000 7f1d96200640 1 -- 172.21.15.40:0/2760862293 shutdown_connections 2024-03-24T00:31:52.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:52.885+0000 7f1d96200640 1 -- 172.21.15.40:0/2760862293 wait complete. 2024-03-24T00:31:52.907 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:31:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe205e00640 1 Processor -- start 2024-03-24T00:31:53.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe205e00640 1 -- start start 2024-03-24T00:31:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe205e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe20015bb10 0x7fe20006f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe205e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe20006f570 0x7fe200066ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe205e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe200067c60 con 0x7fe200067430 2024-03-24T00:31:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe205e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe20006f9d0 con 0x7fe20015bb10 2024-03-24T00:31:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe205e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20015ef60 con 0x7fe20006f570 2024-03-24T00:31:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe1ff400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe200067430 0x7fe200071b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42570/0 (socket says 172.21.15.40:42570) 2024-03-24T00:31:53.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe1ff400640 1 -- 172.21.15.40:0/2879953478 learned_addr learned my addr 172.21.15.40:0/2879953478 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe1fea00640 1 --2- 172.21.15.40:0/2879953478 >> v2:172.21.15.53:3300/0 conn(0x7fe20015bb10 0x7fe20006f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe1f7e00640 1 --2- 172.21.15.40:0/2879953478 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe20006f570 0x7fe200066ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe1fe000640 1 -- 172.21.15.40:0/2879953478 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455350623 0 0) 0x7fe200067c60 con 0x7fe200067430 2024-03-24T00:31:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe1fe000640 1 -- 172.21.15.40:0/2879953478 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1e4003530 con 0x7fe200067430 2024-03-24T00:31:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe1fea00640 1 -- 172.21.15.40:0/2879953478 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe20006f570 msgr2=0x7fe200066ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe1fea00640 1 --2- 172.21.15.40:0/2879953478 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe20006f570 0x7fe200066ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe1fea00640 1 -- 172.21.15.40:0/2879953478 >> v1:172.21.15.40:6789/0 conn(0x7fe200067430 legacy=0x7fe200071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe1fe000640 1 -- 172.21.15.40:0/2879953478 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 695072708 0 0) 0x7fe1e4003530 con 0x7fe200067430 2024-03-24T00:31:53.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.077+0000 7fe1fea00640 1 -- 172.21.15.40:0/2879953478 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20015e910 con 0x7fe20015bb10 2024-03-24T00:31:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.081+0000 7fe1fea00640 1 --2- 172.21.15.40:0/2879953478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe20015bb10 0x7fe20006f030 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00097c0 tx=0x7fe1f001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b17e74ae8445c932 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.081+0000 7fe1fe000640 1 -- 172.21.15.40:0/2879953478 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f00b0070 con 0x7fe20015bb10 2024-03-24T00:31:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.081+0000 7fe1fe000640 1 -- 172.21.15.40:0/2879953478 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f00ab400 con 0x7fe20015bb10 2024-03-24T00:31:53.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.081+0000 7fe1fe000640 1 -- 172.21.15.40:0/2879953478 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8001a00 con 0x7fe20015bb10 2024-03-24T00:31:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.081+0000 7fe205e00640 1 -- 172.21.15.40:0/2879953478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe20015bb10 msgr2=0x7fe20006f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.081+0000 7fe205e00640 1 --2- 172.21.15.40:0/2879953478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe20015bb10 0x7fe20006f030 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00097c0 tx=0x7fe1f001c3c0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.081+0000 7fe205e00640 1 -- 172.21.15.40:0/2879953478 shutdown_connections 2024-03-24T00:31:53.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.081+0000 7fe205e00640 1 --2- 172.21.15.40:0/2879953478 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe20006f570 0x7fe200066ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.081+0000 7fe205e00640 1 --2- 172.21.15.40:0/2879953478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe20015bb10 0x7fe20006f030 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.081+0000 7fe205e00640 1 -- 172.21.15.40:0/2879953478 >> 172.21.15.40:0/2879953478 conn(0x7fe2000b27c0 msgr2=0x7fe2000b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:53.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.081+0000 7fe205e00640 1 -- 172.21.15.40:0/2879953478 shutdown_connections 2024-03-24T00:31:53.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.081+0000 7fe205e00640 1 -- 172.21.15.40:0/2879953478 wait complete. 2024-03-24T00:31:53.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe205e00640 1 Processor -- start 2024-03-24T00:31:53.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe205e00640 1 -- start start 2024-03-24T00:31:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe205e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe200067430 0x7fe200160260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe205e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe20006f570 0x7fe2001607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe205e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe20015bb10 0x7fe20015c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe205e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe200067a20 con 0x7fe200067430 2024-03-24T00:31:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe205e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2000678a0 con 0x7fe20006f570 2024-03-24T00:31:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe205e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe20015f3f0 con 0x7fe20015bb10 2024-03-24T00:31:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1fea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe200067430 0x7fe200160260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1fea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe200067430 0x7fe200160260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34046/0 (socket says 172.21.15.40:34046) 2024-03-24T00:31:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1fea00640 1 -- 172.21.15.40:0/251823829 learned_addr learned my addr 172.21.15.40:0/251823829 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1f7e00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe20006f570 0x7fe2001607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1ff400640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe20015bb10 0x7fe20015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1fea00640 1 -- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe20015bb10 msgr2=0x7fe20015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1fea00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe20015bb10 0x7fe20015c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1fea00640 1 -- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe20006f570 msgr2=0x7fe2001607a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1fea00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe20006f570 0x7fe2001607a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1fea00640 1 -- 172.21.15.40:0/251823829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2001536e0 con 0x7fe200067430 2024-03-24T00:31:53.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1f7e00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe20006f570 0x7fe2001607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:53.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1ff400640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe20015bb10 0x7fe20015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:53.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1fea00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe200067430 0x7fe200160260 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00ae040 tx=0x7fe1f001ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:53.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1f7400640 1 -- 172.21.15.40:0/251823829 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f00b0050 con 0x7fe200067430 2024-03-24T00:31:53.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1f7400640 1 -- 172.21.15.40:0/251823829 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f00bc070 con 0x7fe200067430 2024-03-24T00:31:53.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe1f7400640 1 -- 172.21.15.40:0/251823829 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f00b78b0 con 0x7fe200067430 2024-03-24T00:31:53.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe205e00640 1 -- 172.21.15.40:0/251823829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2001539c0 con 0x7fe200067430 2024-03-24T00:31:53.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.085+0000 7fe205e00640 1 -- 172.21.15.40:0/251823829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe200153390 con 0x7fe200067430 2024-03-24T00:31:53.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.089+0000 7fe205e00640 1 -- 172.21.15.40:0/251823829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe200068170 con 0x7fe200067430 2024-03-24T00:31:53.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.089+0000 7fe1f7400640 1 -- 172.21.15.40:0/251823829 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe1f00b7a50 con 0x7fe200067430 2024-03-24T00:31:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.089+0000 7fe1f7400640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe1dc067660 0x7fe1dc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.089+0000 7fe1f7400640 1 -- 172.21.15.40:0/251823829 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fe1f01201a0 con 0x7fe200067430 2024-03-24T00:31:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.093+0000 7fe1f7e00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe1dc067660 0x7fe1dc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.093+0000 7fe1f7e00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe1dc067660 0x7fe1dc069b20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe200161740 tx=0x7fe1e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:53.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.097+0000 7fe1f7400640 1 -- 172.21.15.40:0/251823829 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe1f0015fb0 con 0x7fe200067430 2024-03-24T00:31:53.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.257+0000 7fe205e00640 1 -- 172.21.15.40:0/251823829 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe2000b3500 con 0x7fe200067430 2024-03-24T00:31:53.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe1f7400640 1 -- 172.21.15.40:0/251823829 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe1f00f0f50 con 0x7fe200067430 2024-03-24T00:31:53.264 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:31:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe205e00640 1 -- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe1dc067660 msgr2=0x7fe1dc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe205e00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe1dc067660 0x7fe1dc069b20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe200161740 tx=0x7fe1e8008040 comp rx=0 tx=0).stop 2024-03-24T00:31:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe205e00640 1 -- 172.21.15.40:0/251823829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe200067430 msgr2=0x7fe200160260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe205e00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe200067430 0x7fe200160260 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00ae040 tx=0x7fe1f001ac00 comp rx=0 tx=0).stop 2024-03-24T00:31:53.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe205e00640 1 -- 172.21.15.40:0/251823829 shutdown_connections 2024-03-24T00:31:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe205e00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe1dc067660 0x7fe1dc069b20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe205e00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe20015bb10 0x7fe20015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe205e00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe20006f570 0x7fe2001607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe205e00640 1 --2- 172.21.15.40:0/251823829 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe200067430 0x7fe200160260 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe205e00640 1 -- 172.21.15.40:0/251823829 >> 172.21.15.40:0/251823829 conn(0x7fe2000b27c0 msgr2=0x7fe2000b0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe205e00640 1 -- 172.21.15.40:0/251823829 shutdown_connections 2024-03-24T00:31:53.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.261+0000 7fe205e00640 1 -- 172.21.15.40:0/251823829 wait complete. 2024-03-24T00:31:53.285 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:31:53.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.421+0000 7f5b16800640 1 Processor -- start 2024-03-24T00:31:53.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.421+0000 7f5b16800640 1 -- start start 2024-03-24T00:31:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.421+0000 7f5b16800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b1015e0b0 0x7f5b100d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.421+0000 7f5b16800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b100cdf20 0x7f5b1006b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.421+0000 7f5b16800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b100d0e10 con 0x7f5b100cd3d0 2024-03-24T00:31:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.421+0000 7f5b16800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b100d10f0 con 0x7f5b100cdf20 2024-03-24T00:31:53.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.421+0000 7f5b16800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1006efd0 con 0x7f5b1015e0b0 2024-03-24T00:31:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.421+0000 7f5b0fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b100cdf20 0x7f5b1006b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.425+0000 7f5b0fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5b100cdf20 0x7f5b1006b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34060/0 (socket says 172.21.15.40:34060) 2024-03-24T00:31:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.425+0000 7f5b0fe00640 1 -- 172.21.15.40:0/578187799 learned_addr learned my addr 172.21.15.40:0/578187799 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.425+0000 7f5b0f400640 1 --2- 172.21.15.40:0/578187799 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b1015e0b0 0x7f5b100d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.425+0000 7f5b0fe00640 1 -- 172.21.15.40:0/578187799 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b1015e0b0 msgr2=0x7f5b100d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.425+0000 7f5b0fe00640 1 --2- 172.21.15.40:0/578187799 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b1015e0b0 0x7f5b100d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.425+0000 7f5b0fe00640 1 -- 172.21.15.40:0/578187799 >> v1:172.21.15.53:6789/0 conn(0x7f5b100cd3d0 legacy=0x7f5b100cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.425+0000 7f5b0fe00640 1 -- 172.21.15.40:0/578187799 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1006f2b0 con 0x7f5b100cdf20 2024-03-24T00:31:53.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.425+0000 7f5b0f400640 1 --2- 172.21.15.40:0/578187799 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b1015e0b0 0x7f5b100d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:53.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.425+0000 7f5b0fe00640 1 --2- 172.21.15.40:0/578187799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b100cdf20 0x7f5b1006b150 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009e50 tx=0x7f5b0801c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171700fd72285b1d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:53.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.425+0000 7f5b0ea00640 1 -- 172.21.15.40:0/578187799 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b080a64d0 con 0x7f5b100cdf20 2024-03-24T00:31:53.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.425+0000 7f5b0ea00640 1 -- 172.21.15.40:0/578187799 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b080a6670 con 0x7f5b100cdf20 2024-03-24T00:31:53.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.425+0000 7f5b0ea00640 1 -- 172.21.15.40:0/578187799 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b080a6a40 con 0x7f5b100cdf20 2024-03-24T00:31:53.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 -- 172.21.15.40:0/578187799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b100cdf20 msgr2=0x7f5b1006b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 --2- 172.21.15.40:0/578187799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b100cdf20 0x7f5b1006b150 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009e50 tx=0x7f5b0801c730 comp rx=0 tx=0).stop 2024-03-24T00:31:53.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 -- 172.21.15.40:0/578187799 shutdown_connections 2024-03-24T00:31:53.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 --2- 172.21.15.40:0/578187799 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b100cdf20 0x7f5b1006b150 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 --2- 172.21.15.40:0/578187799 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b1015e0b0 0x7f5b100d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 -- 172.21.15.40:0/578187799 >> 172.21.15.40:0/578187799 conn(0x7f5b100d1c90 msgr2=0x7f5b100d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:53.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 -- 172.21.15.40:0/578187799 shutdown_connections 2024-03-24T00:31:53.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 -- 172.21.15.40:0/578187799 wait complete. 2024-03-24T00:31:53.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 Processor -- start 2024-03-24T00:31:53.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 -- start start 2024-03-24T00:31:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b100cd3d0 0x7f5b10150b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b100cdf20 0x7f5b100cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b1015e0b0 0x7f5b100cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1006fc30 con 0x7f5b100cd3d0 2024-03-24T00:31:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1006fab0 con 0x7f5b1015e0b0 2024-03-24T00:31:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b16800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1006fdb0 con 0x7f5b100cdf20 2024-03-24T00:31:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b0f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b100cd3d0 0x7f5b10150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b0f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b100cd3d0 0x7f5b10150b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34072/0 (socket says 172.21.15.40:34072) 2024-03-24T00:31:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b0f400640 1 -- 172.21.15.40:0/1329292763 learned_addr learned my addr 172.21.15.40:0/1329292763 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b0fe00640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b1015e0b0 0x7f5b100cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b06a00640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b100cdf20 0x7f5b100cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b0f400640 1 -- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b100cdf20 msgr2=0x7f5b100cc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b0f400640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b100cdf20 0x7f5b100cc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b0f400640 1 -- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b1015e0b0 msgr2=0x7f5b100cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b0f400640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b1015e0b0 0x7f5b100cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b0f400640 1 -- 172.21.15.40:0/1329292763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b10151070 con 0x7f5b100cd3d0 2024-03-24T00:31:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b0fe00640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b1015e0b0 0x7f5b100cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.429+0000 7f5b0f400640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b100cd3d0 0x7f5b10150b30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000c820 tx=0x7f5b0000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.433+0000 7f5b07e00640 1 -- 172.21.15.40:0/1329292763 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00015070 con 0x7f5b100cd3d0 2024-03-24T00:31:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.433+0000 7f5b07e00640 1 -- 172.21.15.40:0/1329292763 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b00007a30 con 0x7f5b100cd3d0 2024-03-24T00:31:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.433+0000 7f5b07e00640 1 -- 172.21.15.40:0/1329292763 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00007e40 con 0x7f5b100cd3d0 2024-03-24T00:31:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.433+0000 7f5b16800640 1 -- 172.21.15.40:0/1329292763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b10158160 con 0x7f5b100cd3d0 2024-03-24T00:31:53.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.433+0000 7f5b16800640 1 -- 172.21.15.40:0/1329292763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b10158720 con 0x7f5b100cd3d0 2024-03-24T00:31:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.433+0000 7f5b16800640 1 -- 172.21.15.40:0/1329292763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b10068170 con 0x7f5b100cd3d0 2024-03-24T00:31:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.433+0000 7f5b07e00640 1 -- 172.21.15.40:0/1329292763 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5b00018070 con 0x7f5b100cd3d0 2024-03-24T00:31:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.433+0000 7f5b07e00640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5aec067660 0x7f5aec069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.433+0000 7f5b07e00640 1 -- 172.21.15.40:0/1329292763 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f5b00085ca0 con 0x7f5b100cd3d0 2024-03-24T00:31:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.433+0000 7f5b06a00640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5aec067660 0x7f5aec069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.437+0000 7f5b06a00640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5aec067660 0x7f5aec069b20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5b100cd3a0 tx=0x7f5af8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:53.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.437+0000 7f5b07e00640 1 -- 172.21.15.40:0/1329292763 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5b00056880 con 0x7f5b100cd3d0 2024-03-24T00:31:53.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 -- 172.21.15.40:0/1329292763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b1015a250 con 0x7f5b100cd3d0 2024-03-24T00:31:53.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b07e00640 1 -- 172.21.15.40:0/1329292763 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4965 (secure 0 0 0) 0x7f5b00056180 con 0x7f5b100cd3d0 2024-03-24T00:31:53.641 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:53.641 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/501744669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":501744669},{"type":"v1","addr":"172.21.15.40:6833","nonce":501744669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/281354690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":281354690},{"type":"v1","addr":"172.21.15.53:6836","nonce":281354690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3092758821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3092758821},{"type":"v1","addr":"172.21.15.53:6837","nonce":3092758821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:30:59.427652+0000","modified":"2024-03-24T00:31:02.444298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4139303536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":4139303536},{"type":"v1","addr":"172.21.15.40:6835","nonce":4139303536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-24T00:31:53.641 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 8 2024-03-24T00:31:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 -- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5aec067660 msgr2=0x7f5aec069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5aec067660 0x7f5aec069b20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5b100cd3a0 tx=0x7f5af8009040 comp rx=0 tx=0).stop 2024-03-24T00:31:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 -- 172.21.15.40:0/1329292763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b100cd3d0 msgr2=0x7f5b10150b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b100cd3d0 0x7f5b10150b30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000c820 tx=0x7f5b0000cce0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 -- 172.21.15.40:0/1329292763 shutdown_connections 2024-03-24T00:31:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5aec067660 0x7f5aec069b20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b1015e0b0 0x7f5b100cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b100cdf20 0x7f5b100cc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 --2- 172.21.15.40:0/1329292763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5b100cd3d0 0x7f5b10150b30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 -- 172.21.15.40:0/1329292763 >> 172.21.15.40:0/1329292763 conn(0x7f5b100d1c90 msgr2=0x7f5b100af390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 -- 172.21.15.40:0/1329292763 shutdown_connections 2024-03-24T00:31:53.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.637+0000 7f5b16800640 1 -- 172.21.15.40:0/1329292763 wait complete. 2024-03-24T00:31:53.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:53.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.797+0000 7f20e3800640 1 Processor -- start 2024-03-24T00:31:53.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20e3800640 1 -- start start 2024-03-24T00:31:53.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20e3800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20dc15f600 0x7f20dc0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20e3800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20dc0d2c70 con 0x7f20dc0d2830 2024-03-24T00:31:53.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20e3800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20dc0b1e50 con 0x7f20dc0af170 2024-03-24T00:31:53.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20e3800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc0b3630 con 0x7f20dc15f600 2024-03-24T00:31:53.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20dbe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f20dc0d2830 0x7f20dc0aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42586/0 (socket says 172.21.15.40:42586) 2024-03-24T00:31:53.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20dbe00640 1 -- 172.21.15.40:0/1217269545 learned_addr learned my addr 172.21.15.40:0/1217269545 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:53.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20e0e00640 1 --2- 172.21.15.40:0/1217269545 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20dc15f600 0x7f20dc0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20db400640 1 -- 172.21.15.40:0/1217269545 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814710440 0 0) 0x7f20dc0d2c70 con 0x7f20dc0d2830 2024-03-24T00:31:53.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20db400640 1 -- 172.21.15.40:0/1217269545 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20c4003530 con 0x7f20dc0d2830 2024-03-24T00:31:53.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20db400640 1 -- 172.21.15.40:0/1217269545 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736447910 0 0) 0x7f20dc0b1e50 con 0x7f20dc0af170 2024-03-24T00:31:53.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20db400640 1 -- 172.21.15.40:0/1217269545 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20dc0d2c70 con 0x7f20dc0af170 2024-03-24T00:31:53.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20e0e00640 1 -- 172.21.15.40:0/1217269545 >> v1:172.21.15.53:6789/0 conn(0x7f20dc0af170 legacy=0x7f20dc165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20e0e00640 1 -- 172.21.15.40:0/1217269545 >> v1:172.21.15.40:6789/0 conn(0x7f20dc0d2830 legacy=0x7f20dc0aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.801+0000 7f20e0e00640 1 -- 172.21.15.40:0/1217269545 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20dc0b20f0 con 0x7f20dc15f600 2024-03-24T00:31:53.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20e0e00640 1 --2- 172.21.15.40:0/1217269545 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20dc15f600 0x7f20dc0b2910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f20d00097f0 tx=0x7f20d001ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6a7e2ffd5b3b89d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:53.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20db400640 1 -- 172.21.15.40:0/1217269545 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d00b5070 con 0x7f20dc15f600 2024-03-24T00:31:53.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20db400640 1 -- 172.21.15.40:0/1217269545 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d00a89d0 con 0x7f20dc15f600 2024-03-24T00:31:53.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20db400640 1 -- 172.21.15.40:0/1217269545 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d00a8da0 con 0x7f20dc15f600 2024-03-24T00:31:53.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20e3800640 1 -- 172.21.15.40:0/1217269545 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20dc15f600 msgr2=0x7f20dc0b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20e3800640 1 --2- 172.21.15.40:0/1217269545 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20dc15f600 0x7f20dc0b2910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f20d00097f0 tx=0x7f20d001ecd0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20e3800640 1 -- 172.21.15.40:0/1217269545 shutdown_connections 2024-03-24T00:31:53.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20e3800640 1 --2- 172.21.15.40:0/1217269545 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20dc15f600 0x7f20dc0b2910 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20e3800640 1 -- 172.21.15.40:0/1217269545 >> 172.21.15.40:0/1217269545 conn(0x7f20dc0d1d10 msgr2=0x7f20dc0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:53.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20e3800640 1 -- 172.21.15.40:0/1217269545 shutdown_connections 2024-03-24T00:31:53.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20e3800640 1 -- 172.21.15.40:0/1217269545 wait complete. 2024-03-24T00:31:53.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20e3800640 1 Processor -- start 2024-03-24T00:31:53.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.805+0000 7f20e3800640 1 -- start start 2024-03-24T00:31:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e3800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20dc0af170 0x7f20dc152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e3800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20dc0d2830 0x7f20dc0c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e3800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20dc15f600 0x7f20dc06feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e3800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc0b2fa0 con 0x7f20dc0af170 2024-03-24T00:31:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e3800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc0b2740 con 0x7f20dc0d2830 2024-03-24T00:31:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e3800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20dc15f9e0 con 0x7f20dc15f600 2024-03-24T00:31:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e0e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20dc0af170 0x7f20dc152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e0e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20dc0af170 0x7f20dc152330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34082/0 (socket says 172.21.15.40:34082) 2024-03-24T00:31:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e0e00640 1 -- 172.21.15.40:0/1033177008 learned_addr learned my addr 172.21.15.40:0/1033177008 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20dbe00640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20dc0d2830 0x7f20dc0c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e1800640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20dc15f600 0x7f20dc06feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e0e00640 1 -- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20dc15f600 msgr2=0x7f20dc06feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e0e00640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20dc15f600 0x7f20dc06feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e0e00640 1 -- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20dc0d2830 msgr2=0x7f20dc0c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e0e00640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20dc0d2830 0x7f20dc0c9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e0e00640 1 -- 172.21.15.40:0/1033177008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20dc152870 con 0x7f20dc0af170 2024-03-24T00:31:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e1800640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20dc15f600 0x7f20dc06feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:31:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e0e00640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20dc0af170 0x7f20dc152330 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f20d001b8e0 tx=0x7f20d001ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20d8c00640 1 -- 172.21.15.40:0/1033177008 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d00bf070 con 0x7f20dc0af170 2024-03-24T00:31:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20d8c00640 1 -- 172.21.15.40:0/1033177008 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d00b0720 con 0x7f20dc0af170 2024-03-24T00:31:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e3800640 1 -- 172.21.15.40:0/1033177008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20dc156210 con 0x7f20dc0af170 2024-03-24T00:31:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20d8c00640 1 -- 172.21.15.40:0/1033177008 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d00b5040 con 0x7f20dc0af170 2024-03-24T00:31:53.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e3800640 1 -- 172.21.15.40:0/1033177008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20dc1567d0 con 0x7f20dc0af170 2024-03-24T00:31:53.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20e3800640 1 -- 172.21.15.40:0/1033177008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20dc06d710 con 0x7f20dc0af170 2024-03-24T00:31:53.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.809+0000 7f20d8c00640 1 -- 172.21.15.40:0/1033177008 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f20d00c2070 con 0x7f20dc0af170 2024-03-24T00:31:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.813+0000 7f20d8c00640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f20bc067690 0x7f20bc069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.813+0000 7f20d8c00640 1 -- 172.21.15.40:0/1033177008 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f20d01220a0 con 0x7f20dc0af170 2024-03-24T00:31:53.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.813+0000 7f20dbe00640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f20bc067690 0x7f20bc069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:53.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.813+0000 7f20dbe00640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f20bc067690 0x7f20bc069b50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f20dc0c9610 tx=0x7f20c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:53.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:53.817+0000 7f20d8c00640 1 -- 172.21.15.40:0/1033177008 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f20d0018830 con 0x7f20dc0af170 2024-03-24T00:31:54.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.061+0000 7f20e3800640 1 -- 172.21.15.40:0/1033177008 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20dc15dca0 con 0x7f20dc0af170 2024-03-24T00:31:54.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.061+0000 7f20d8c00640 1 -- 172.21.15.40:0/1033177008 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16813 (secure 0 0 0) 0x7f20d00f2c00 con 0x7f20dc0af170 2024-03-24T00:31:54.066 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:54.066 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":22,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:48.417423+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.065+0000 7f20e3800640 1 -- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f20bc067690 msgr2=0x7f20bc069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.065+0000 7f20e3800640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f20bc067690 0x7f20bc069b50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f20dc0c9610 tx=0x7f20c8009040 comp rx=0 tx=0).stop 2024-03-24T00:31:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.065+0000 7f20e3800640 1 -- 172.21.15.40:0/1033177008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20dc0af170 msgr2=0x7f20dc152330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.065+0000 7f20e3800640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20dc0af170 0x7f20dc152330 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f20d001b8e0 tx=0x7f20d001ba70 comp rx=0 tx=0).stop 2024-03-24T00:31:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.065+0000 7f20e3800640 1 -- 172.21.15.40:0/1033177008 shutdown_connections 2024-03-24T00:31:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.065+0000 7f20e3800640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f20bc067690 0x7f20bc069b50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.065+0000 7f20e3800640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20dc15f600 0x7f20dc06feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.065+0000 7f20e3800640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20dc0d2830 0x7f20dc0c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.065+0000 7f20e3800640 1 --2- 172.21.15.40:0/1033177008 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20dc0af170 0x7f20dc152330 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.065+0000 7f20e3800640 1 -- 172.21.15.40:0/1033177008 >> 172.21.15.40:0/1033177008 conn(0x7f20dc0d1d10 msgr2=0x7f20dc15d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:54.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.065+0000 7f20e3800640 1 -- 172.21.15.40:0/1033177008 shutdown_connections 2024-03-24T00:31:54.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.065+0000 7f20e3800640 1 -- 172.21.15.40:0/1033177008 wait complete. 2024-03-24T00:31:54.097 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:31:54.097 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:31:54.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f7346800640 1 Processor -- start 2024-03-24T00:31:54.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f7346800640 1 -- start start 2024-03-24T00:31:54.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f7346800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f73400cfd10 0x7f7340162430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f7346800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7340162b10 con 0x7f7340161890 2024-03-24T00:31:54.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f7346800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73400cf1d0 con 0x7f73401614b0 2024-03-24T00:31:54.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f7346800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73400cf540 con 0x7f73400cfd10 2024-03-24T00:31:54.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f733fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f73400cfd10 0x7f7340162430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f733ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7340161890 0x7f734006f050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42598/0 (socket says 172.21.15.40:42598) 2024-03-24T00:31:54.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f733ea00640 1 -- 172.21.15.40:0/4176372171 learned_addr learned my addr 172.21.15.40:0/4176372171 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:54.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f733e000640 1 -- 172.21.15.40:0/4176372171 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600400248 0 0) 0x7f7340162b10 con 0x7f7340161890 2024-03-24T00:31:54.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f733e000640 1 -- 172.21.15.40:0/4176372171 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7324003530 con 0x7f7340161890 2024-03-24T00:31:54.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f733e000640 1 -- 172.21.15.40:0/4176372171 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941101755 0 0) 0x7f73400cf1d0 con 0x7f73401614b0 2024-03-24T00:31:54.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f733fe00640 1 -- 172.21.15.40:0/4176372171 >> v1:172.21.15.53:6789/0 conn(0x7f73401614b0 legacy=0x7f7340161d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f733fe00640 1 -- 172.21.15.40:0/4176372171 >> v1:172.21.15.40:6789/0 conn(0x7f7340161890 legacy=0x7f734006f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.233+0000 7f733fe00640 1 -- 172.21.15.40:0/4176372171 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73400ceee0 con 0x7f73400cfd10 2024-03-24T00:31:54.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.237+0000 7f733fe00640 1 --2- 172.21.15.40:0/4176372171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73400cfd10 0x7f7340162430 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7338009d90 tx=0x7f733801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ee8c00b09c7bbc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:54.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.237+0000 7f733e000640 1 -- 172.21.15.40:0/4176372171 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73380b2070 con 0x7f73400cfd10 2024-03-24T00:31:54.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.237+0000 7f733e000640 1 -- 172.21.15.40:0/4176372171 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73380a5a50 con 0x7f73400cfd10 2024-03-24T00:31:54.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.237+0000 7f733e000640 1 -- 172.21.15.40:0/4176372171 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73380a5e20 con 0x7f73400cfd10 2024-03-24T00:31:54.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.237+0000 7f7346800640 1 -- 172.21.15.40:0/4176372171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73400cfd10 msgr2=0x7f7340162430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.237+0000 7f7346800640 1 --2- 172.21.15.40:0/4176372171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73400cfd10 0x7f7340162430 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7338009d90 tx=0x7f733801c970 comp rx=0 tx=0).stop 2024-03-24T00:31:54.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.237+0000 7f7346800640 1 -- 172.21.15.40:0/4176372171 shutdown_connections 2024-03-24T00:31:54.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.237+0000 7f7346800640 1 --2- 172.21.15.40:0/4176372171 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73400cfd10 0x7f7340162430 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.237+0000 7f7346800640 1 -- 172.21.15.40:0/4176372171 >> 172.21.15.40:0/4176372171 conn(0x7f73400d1c80 msgr2=0x7f73400d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.237+0000 7f7346800640 1 -- 172.21.15.40:0/4176372171 shutdown_connections 2024-03-24T00:31:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.237+0000 7f7346800640 1 -- 172.21.15.40:0/4176372171 wait complete. 2024-03-24T00:31:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7346800640 1 Processor -- start 2024-03-24T00:31:54.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7346800640 1 -- start start 2024-03-24T00:31:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7346800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73400cfd10 0x7f734014c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7346800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73401614b0 0x7f7340151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7346800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7340161890 0x7f7340150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7346800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734016c400 con 0x7f7340161890 2024-03-24T00:31:54.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7346800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734016c280 con 0x7f73401614b0 2024-03-24T00:31:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7346800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734016c580 con 0x7f73400cfd10 2024-03-24T00:31:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f733fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7340161890 0x7f7340150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f733fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7340161890 0x7f7340150ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34104/0 (socket says 172.21.15.40:34104) 2024-03-24T00:31:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f733fe00640 1 -- 172.21.15.40:0/3881708847 learned_addr learned my addr 172.21.15.40:0/3881708847 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f733f400640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73400cfd10 0x7f734014c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f733ea00640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73401614b0 0x7f7340151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f733fe00640 1 -- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73400cfd10 msgr2=0x7f734014c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f733fe00640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73400cfd10 0x7f734014c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f733fe00640 1 -- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73401614b0 msgr2=0x7f7340151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f733fe00640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73401614b0 0x7f7340151e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f733fe00640 1 -- 172.21.15.40:0/3881708847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7340156f60 con 0x7f7340161890 2024-03-24T00:31:54.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f733fe00640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7340161890 0x7f7340150ec0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f73380b0040 tx=0x7f7338019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:54.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7337400640 1 -- 172.21.15.40:0/3881708847 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73380c0070 con 0x7f7340161890 2024-03-24T00:31:54.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7337400640 1 -- 172.21.15.40:0/3881708847 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73380adaf0 con 0x7f7340161890 2024-03-24T00:31:54.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7337400640 1 -- 172.21.15.40:0/3881708847 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73380b2040 con 0x7f7340161890 2024-03-24T00:31:54.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7346800640 1 -- 172.21.15.40:0/3881708847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7340157240 con 0x7f7340161890 2024-03-24T00:31:54.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.241+0000 7f7346800640 1 -- 172.21.15.40:0/3881708847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7340157800 con 0x7f7340161890 2024-03-24T00:31:54.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.245+0000 7f7346800640 1 -- 172.21.15.40:0/3881708847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7340068170 con 0x7f7340161890 2024-03-24T00:31:54.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.245+0000 7f7337400640 1 -- 172.21.15.40:0/3881708847 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f73380039a0 con 0x7f7340161890 2024-03-24T00:31:54.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.245+0000 7f7337400640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7320067660 0x7f7320069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.245+0000 7f7337400640 1 -- 172.21.15.40:0/3881708847 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f73381203d0 con 0x7f7340161890 2024-03-24T00:31:54.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.245+0000 7f733f400640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7320067660 0x7f7320069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.245+0000 7f733f400640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7320067660 0x7f7320069b20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7330000f70 tx=0x7f7330009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:54.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.249+0000 7f7337400640 1 -- 172.21.15.40:0/3881708847 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7338016560 con 0x7f7340161890 2024-03-24T00:31:54.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.445+0000 7f7346800640 1 -- 172.21.15.40:0/3881708847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7340073070 con 0x7f7340161890 2024-03-24T00:31:54.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7337400640 1 -- 172.21.15.40:0/3881708847 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4965 (secure 0 0 0) 0x7f73380f08e0 con 0x7f7340161890 2024-03-24T00:31:54.453 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:54.453 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/501744669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":501744669},{"type":"v1","addr":"172.21.15.40:6833","nonce":501744669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/281354690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":281354690},{"type":"v1","addr":"172.21.15.53:6836","nonce":281354690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3092758821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3092758821},{"type":"v1","addr":"172.21.15.53:6837","nonce":3092758821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:30:59.427652+0000","modified":"2024-03-24T00:31:02.444298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4139303536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":4139303536},{"type":"v1","addr":"172.21.15.40:6835","nonce":4139303536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-24T00:31:54.453 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 8 2024-03-24T00:31:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7346800640 1 -- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7320067660 msgr2=0x7f7320069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7346800640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7320067660 0x7f7320069b20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7330000f70 tx=0x7f7330009040 comp rx=0 tx=0).stop 2024-03-24T00:31:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7346800640 1 -- 172.21.15.40:0/3881708847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7340161890 msgr2=0x7f7340150ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7346800640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7340161890 0x7f7340150ec0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f73380b0040 tx=0x7f7338019710 comp rx=0 tx=0).stop 2024-03-24T00:31:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7346800640 1 -- 172.21.15.40:0/3881708847 shutdown_connections 2024-03-24T00:31:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7346800640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7320067660 0x7f7320069b20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7346800640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7340161890 0x7f7340150ec0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7346800640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73401614b0 0x7f7340151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7346800640 1 --2- 172.21.15.40:0/3881708847 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73400cfd10 0x7f734014c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7346800640 1 -- 172.21.15.40:0/3881708847 >> 172.21.15.40:0/3881708847 conn(0x7f73400d1c80 msgr2=0x7f73400b6670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:54.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7346800640 1 -- 172.21.15.40:0/3881708847 shutdown_connections 2024-03-24T00:31:54.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.449+0000 7f7346800640 1 -- 172.21.15.40:0/3881708847 wait complete. 2024-03-24T00:31:54.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:31:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6b6000640 1 Processor -- start 2024-03-24T00:31:54.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6b6000640 1 -- start start 2024-03-24T00:31:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6b6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6b00b46c0 0x7ff6b00b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6b6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6b00b2ff0 0x7ff6b00ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6b6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b0070f60 con 0x7ff6b006e620 2024-03-24T00:31:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6b6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b00b3620 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6b6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b00b3900 con 0x7ff6b00b46c0 2024-03-24T00:31:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6afe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6b00b2ff0 0x7ff6b00ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6afe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6b00b2ff0 0x7ff6b00ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34118/0 (socket says 172.21.15.40:34118) 2024-03-24T00:31:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6aea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6b00b46c0 0x7ff6b00b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6afe00640 1 -- 172.21.15.40:0/247057722 learned_addr learned my addr 172.21.15.40:0/247057722 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:54.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6af400640 1 --1- 172.21.15.40:0/247057722 >> v1:172.21.15.40:6789/0 conn(0x7ff6b006e620 0x7ff6b006ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42612/0 (socket says 172.21.15.40:42612) 2024-03-24T00:31:54.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6afe00640 1 -- 172.21.15.40:0/247057722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6b00b46c0 msgr2=0x7ff6b00b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:54.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6afe00640 1 --2- 172.21.15.40:0/247057722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6b00b46c0 0x7ff6b00b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6afe00640 1 -- 172.21.15.40:0/247057722 >> v1:172.21.15.40:6789/0 conn(0x7ff6b006e620 legacy=0x7ff6b006ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6afe00640 1 -- 172.21.15.40:0/247057722 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b00b3be0 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.613+0000 7ff6ae000640 1 -- 172.21.15.40:0/247057722 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674900607 0 0) 0x7ff6b0070f60 con 0x7ff6b006e620 2024-03-24T00:31:54.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6afe00640 1 --2- 172.21.15.40:0/247057722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6b00b2ff0 0x7ff6b00ccfa0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009e50 tx=0x7ff6a801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a79c121f541d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:54.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6ae000640 1 -- 172.21.15.40:0/247057722 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b00cdcf0 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6ae000640 1 -- 172.21.15.40:0/247057722 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a80a5860 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6ae000640 1 -- 172.21.15.40:0/247057722 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a80a5c30 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6b6000640 1 -- 172.21.15.40:0/247057722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6b00b2ff0 msgr2=0x7ff6b00ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6b6000640 1 --2- 172.21.15.40:0/247057722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6b00b2ff0 0x7ff6b00ccfa0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8009e50 tx=0x7ff6a801c9a0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6b6000640 1 -- 172.21.15.40:0/247057722 shutdown_connections 2024-03-24T00:31:54.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6b6000640 1 --2- 172.21.15.40:0/247057722 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6b00b2ff0 0x7ff6b00ccfa0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6b6000640 1 --2- 172.21.15.40:0/247057722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6b00b46c0 0x7ff6b00b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6b6000640 1 -- 172.21.15.40:0/247057722 >> 172.21.15.40:0/247057722 conn(0x7ff6b00b17b0 msgr2=0x7ff6b00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:54.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6b6000640 1 -- 172.21.15.40:0/247057722 shutdown_connections 2024-03-24T00:31:54.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6b6000640 1 -- 172.21.15.40:0/247057722 wait complete. 2024-03-24T00:31:54.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.617+0000 7ff6b6000640 1 Processor -- start 2024-03-24T00:31:54.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6b6000640 1 -- start start 2024-03-24T00:31:54.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6b6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6b006e620 0x7ff6b015f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6b6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6b00b2ff0 0x7ff6b01383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6b6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b00b46c0 0x7ff6b0138f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6b6000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b0144920 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6b6000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b006d510 con 0x7ff6b00b46c0 2024-03-24T00:31:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6b6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b015bea0 con 0x7ff6b006e620 2024-03-24T00:31:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6aea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6b00b2ff0 0x7ff6b01383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6aea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6b00b2ff0 0x7ff6b01383d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34124/0 (socket says 172.21.15.40:34124) 2024-03-24T00:31:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6aea00640 1 -- 172.21.15.40:0/2073930676 learned_addr learned my addr 172.21.15.40:0/2073930676 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6afe00640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b00b46c0 0x7ff6b0138f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6af400640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6b006e620 0x7ff6b015f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6aea00640 1 -- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6b006e620 msgr2=0x7ff6b015f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6aea00640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6b006e620 0x7ff6b015f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6aea00640 1 -- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b00b46c0 msgr2=0x7ff6b0138f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6aea00640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b00b46c0 0x7ff6b0138f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6aea00640 1 -- 172.21.15.40:0/2073930676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b01441a0 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6afe00640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b00b46c0 0x7ff6b0138f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6aea00640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6b00b2ff0 0x7ff6b01383d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00e870 tx=0x7ff69c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6a7400640 1 -- 172.21.15.40:0/2073930676 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69c017070 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6a7400640 1 -- 172.21.15.40:0/2073930676 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff69c00c940 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6b6000640 1 -- 172.21.15.40:0/2073930676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b015c180 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6a7400640 1 -- 172.21.15.40:0/2073930676 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69c00ccf0 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6b6000640 1 -- 172.21.15.40:0/2073930676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b015c740 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.621+0000 7ff6b6000640 1 -- 172.21.15.40:0/2073930676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b0068170 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.625+0000 7ff6a7400640 1 -- 172.21.15.40:0/2073930676 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff69c01a070 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.625+0000 7ff6a7400640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff690067660 0x7ff690069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.625+0000 7ff6a7400640 1 -- 172.21.15.40:0/2073930676 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7ff69c085770 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.625+0000 7ff6af400640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff690067660 0x7ff690069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.625+0000 7ff6af400640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff690067660 0x7ff690069b20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0000f70 tx=0x7ff6a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:54.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.629+0000 7ff6a7400640 1 -- 172.21.15.40:0/2073930676 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff69c055d10 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.825+0000 7ff6b6000640 1 -- 172.21.15.40:0/2073930676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6b0072f50 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.825+0000 7ff6a7400640 1 -- 172.21.15.40:0/2073930676 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4965 (secure 0 0 0) 0x7ff69c059db0 con 0x7ff6b00b2ff0 2024-03-24T00:31:54.831 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:54.831 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/501744669","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":501744669},{"type":"v1","addr":"172.21.15.40:6833","nonce":501744669}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/281354690","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":281354690},{"type":"v1","addr":"172.21.15.53:6836","nonce":281354690}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/3092758821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3092758821},{"type":"v1","addr":"172.21.15.53:6837","nonce":3092758821}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:30:59.427652+0000","modified":"2024-03-24T00:31:02.444298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4264":{"gid":4264,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/4139303536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":4139303536},{"type":"v1","addr":"172.21.15.40:6835","nonce":4139303536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4264,"qdb_cluster":[4264]},"id":1}]} 2024-03-24T00:31:54.831 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 8 2024-03-24T00:31:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.829+0000 7ff6b6000640 1 -- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff690067660 msgr2=0x7ff690069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.829+0000 7ff6b6000640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff690067660 0x7ff690069b20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0000f70 tx=0x7ff6a0009040 comp rx=0 tx=0).stop 2024-03-24T00:31:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.829+0000 7ff6b6000640 1 -- 172.21.15.40:0/2073930676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6b00b2ff0 msgr2=0x7ff6b01383d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.829+0000 7ff6b6000640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6b00b2ff0 0x7ff6b01383d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff69c00e870 tx=0x7ff69c00ec70 comp rx=0 tx=0).stop 2024-03-24T00:31:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.829+0000 7ff6b6000640 1 -- 172.21.15.40:0/2073930676 shutdown_connections 2024-03-24T00:31:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.829+0000 7ff6b6000640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff690067660 0x7ff690069b20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.829+0000 7ff6b6000640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6b00b46c0 0x7ff6b0138f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.829+0000 7ff6b6000640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6b00b2ff0 0x7ff6b01383d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.829+0000 7ff6b6000640 1 --2- 172.21.15.40:0/2073930676 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6b006e620 0x7ff6b015f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.829+0000 7ff6b6000640 1 -- 172.21.15.40:0/2073930676 >> 172.21.15.40:0/2073930676 conn(0x7ff6b00b17b0 msgr2=0x7ff6b013c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.829+0000 7ff6b6000640 1 -- 172.21.15.40:0/2073930676 shutdown_connections 2024-03-24T00:31:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.829+0000 7ff6b6000640 1 -- 172.21.15.40:0/2073930676 wait complete. 2024-03-24T00:31:54.851 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:31:54.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11ce000640 1 Processor -- start 2024-03-24T00:31:54.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11ce000640 1 -- start start 2024-03-24T00:31:54.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11ce000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11c815f600 0x7f11c80b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11ce000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f11c80af170 0x7f11c8165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:54.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11ce000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c80d2c70 con 0x7f11c80d2830 2024-03-24T00:31:54.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11ce000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c80b3630 con 0x7f11c80af170 2024-03-24T00:31:54.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11ce000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c80b2740 con 0x7f11c815f600 2024-03-24T00:31:54.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11c7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f11c80af170 0x7f11c8165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11bfe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f11c80d2830 0x7f11c80aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:36934/0 (socket says 172.21.15.40:36934) 2024-03-24T00:31:54.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11bfe00640 1 -- 172.21.15.40:0/3225721575 learned_addr learned my addr 172.21.15.40:0/3225721575 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:54.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11c7400640 1 --2- 172.21.15.40:0/3225721575 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11c815f600 0x7f11c80b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:54.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11c7e00640 1 -- 172.21.15.40:0/3225721575 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11c815f600 msgr2=0x7f11c80b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:54.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11c7e00640 1 --2- 172.21.15.40:0/3225721575 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11c815f600 0x7f11c80b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11c7e00640 1 -- 172.21.15.40:0/3225721575 >> v1:172.21.15.53:6789/0 conn(0x7f11c80d2830 legacy=0x7f11c80aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11c6a00640 1 -- 172.21.15.40:0/3225721575 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128972168 0 0) 0x7f11c80d2c70 con 0x7f11c80d2830 2024-03-24T00:31:54.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11c7e00640 1 -- 172.21.15.40:0/3225721575 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c80b1e50 con 0x7f11c80af170 2024-03-24T00:31:54.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.989+0000 7f11c7400640 1 --2- 172.21.15.40:0/3225721575 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11c815f600 0x7f11c80b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:54.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.993+0000 7f11c7e00640 1 --2- 172.21.15.40:0/3225721575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11c80af170 0x7f11c8165e40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009d90 tx=0x7f11c001f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d599dc9d7a65a26 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:54.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.993+0000 7f11c6a00640 1 -- 172.21.15.40:0/3225721575 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c00b4070 con 0x7f11c80af170 2024-03-24T00:31:54.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.993+0000 7f11c6a00640 1 -- 172.21.15.40:0/3225721575 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c00af600 con 0x7f11c80af170 2024-03-24T00:31:54.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.993+0000 7f11c6a00640 1 -- 172.21.15.40:0/3225721575 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c00af9d0 con 0x7f11c80af170 2024-03-24T00:31:54.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.993+0000 7f11ce000640 1 -- 172.21.15.40:0/3225721575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11c80af170 msgr2=0x7f11c8165e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:54.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.993+0000 7f11ce000640 1 --2- 172.21.15.40:0/3225721575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11c80af170 0x7f11c8165e40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009d90 tx=0x7f11c001f150 comp rx=0 tx=0).stop 2024-03-24T00:31:54.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.993+0000 7f11ce000640 1 -- 172.21.15.40:0/3225721575 shutdown_connections 2024-03-24T00:31:54.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.993+0000 7f11ce000640 1 --2- 172.21.15.40:0/3225721575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11c80af170 0x7f11c8165e40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.993+0000 7f11ce000640 1 --2- 172.21.15.40:0/3225721575 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11c815f600 0x7f11c80b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:54.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.993+0000 7f11ce000640 1 -- 172.21.15.40:0/3225721575 >> 172.21.15.40:0/3225721575 conn(0x7f11c80d1d10 msgr2=0x7f11c80d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:54.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.993+0000 7f11ce000640 1 -- 172.21.15.40:0/3225721575 shutdown_connections 2024-03-24T00:31:54.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.993+0000 7f11ce000640 1 -- 172.21.15.40:0/3225721575 wait complete. 2024-03-24T00:31:55.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11ce000640 1 Processor -- start 2024-03-24T00:31:55.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11ce000640 1 -- start start 2024-03-24T00:31:55.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11ce000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11c80af170 0x7f11c8159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11ce000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11c80d2830 0x7f11c80c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11ce000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11c815f600 0x7f11c8070220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11ce000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c80b2fa0 con 0x7f11c80d2830 2024-03-24T00:31:55.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11ce000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c80b23d0 con 0x7f11c815f600 2024-03-24T00:31:55.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11ce000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c815f9e0 con 0x7f11c80af170 2024-03-24T00:31:55.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11bfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11c80d2830 0x7f11c80c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11bfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11c80d2830 0x7f11c80c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34126/0 (socket says 172.21.15.40:34126) 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11bfe00640 1 -- 172.21.15.40:0/3077970745 learned_addr learned my addr 172.21.15.40:0/3077970745 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11c7400640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11c80af170 0x7f11c8159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11c7e00640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11c815f600 0x7f11c8070220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11bfe00640 1 -- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11c80af170 msgr2=0x7f11c8159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11bfe00640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11c80af170 0x7f11c8159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11bfe00640 1 -- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11c815f600 msgr2=0x7f11c8070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11bfe00640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11c815f600 0x7f11c8070220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11bfe00640 1 -- 172.21.15.40:0/3077970745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c80ca080 con 0x7f11c80d2830 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11bfe00640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11c80d2830 0x7f11c80c7350 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f11b00011f0 tx=0x7f11b000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11bf400640 1 -- 172.21.15.40:0/3077970745 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0018070 con 0x7f11c80d2830 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11bf400640 1 -- 172.21.15.40:0/3077970745 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b00074e0 con 0x7f11c80d2830 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11bf400640 1 -- 172.21.15.40:0/3077970745 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b0004a10 con 0x7f11c80d2830 2024-03-24T00:31:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11ce000640 1 -- 172.21.15.40:0/3077970745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c80ca360 con 0x7f11c80d2830 2024-03-24T00:31:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:54.997+0000 7f11ce000640 1 -- 172.21.15.40:0/3077970745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c8159970 con 0x7f11c80d2830 2024-03-24T00:31:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.001+0000 7f11ce000640 1 -- 172.21.15.40:0/3077970745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c8068170 con 0x7f11c80d2830 2024-03-24T00:31:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.001+0000 7f11bf400640 1 -- 172.21.15.40:0/3077970745 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f11b0004bb0 con 0x7f11c80d2830 2024-03-24T00:31:55.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.001+0000 7f11bf400640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11a4067660 0x7f11a4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.001+0000 7f11bf400640 1 -- 172.21.15.40:0/3077970745 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7f11b0084d70 con 0x7f11c80d2830 2024-03-24T00:31:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.001+0000 7f11c7400640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11a4067660 0x7f11a4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.001+0000 7f11c7400640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11a4067660 0x7f11a4069b20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009580 tx=0x7f11b8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:55.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.005+0000 7f11bf400640 1 -- 172.21.15.40:0/3077970745 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f11b00551f0 con 0x7f11c80d2830 2024-03-24T00:31:55.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.173+0000 7f11ce000640 1 -- 172.21.15.40:0/3077970745 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11c80d2340 con 0x7f11c80d2830 2024-03-24T00:31:55.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.173+0000 7f11bf400640 1 -- 172.21.15.40:0/3077970745 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16813 (secure 0 0 0) 0x7f11b0059290 con 0x7f11c80d2830 2024-03-24T00:31:55.179 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:55.179 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":22,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:48.417423+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:30:57.724304+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:30:58.418950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.38","mappings":[{"from":5,"to":1}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:31:55.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.177+0000 7f11ce000640 1 -- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11a4067660 msgr2=0x7f11a4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.177+0000 7f11ce000640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11a4067660 0x7f11a4069b20 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f11b8009580 tx=0x7f11b8009190 comp rx=0 tx=0).stop 2024-03-24T00:31:55.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.177+0000 7f11ce000640 1 -- 172.21.15.40:0/3077970745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11c80d2830 msgr2=0x7f11c80c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.177+0000 7f11ce000640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11c80d2830 0x7f11c80c7350 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f11b00011f0 tx=0x7f11b000c9b0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.177+0000 7f11ce000640 1 -- 172.21.15.40:0/3077970745 shutdown_connections 2024-03-24T00:31:55.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.177+0000 7f11ce000640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11a4067660 0x7f11a4069b20 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.177+0000 7f11ce000640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11c815f600 0x7f11c8070220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.177+0000 7f11ce000640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11c80d2830 0x7f11c80c7350 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.177+0000 7f11ce000640 1 --2- 172.21.15.40:0/3077970745 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11c80af170 0x7f11c8159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.177+0000 7f11ce000640 1 -- 172.21.15.40:0/3077970745 >> 172.21.15.40:0/3077970745 conn(0x7f11c80d1d10 msgr2=0x7f11c815d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:55.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.177+0000 7f11ce000640 1 -- 172.21.15.40:0/3077970745 shutdown_connections 2024-03-24T00:31:55.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.177+0000 7f11ce000640 1 -- 172.21.15.40:0/3077970745 wait complete. 2024-03-24T00:31:55.201 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:31:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.337+0000 7fd640400640 1 Processor -- start 2024-03-24T00:31:55.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.337+0000 7fd640400640 1 -- start start 2024-03-24T00:31:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd640400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd63813ec80 0x7fd6381414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd640400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6381419f0 0x7fd6380ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd640400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd638070e00 con 0x7fd6380ce630 2024-03-24T00:31:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd640400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd638141e50 con 0x7fd63813ec80 2024-03-24T00:31:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd640400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380cee20 con 0x7fd6381419f0 2024-03-24T00:31:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd63da00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd63813ec80 0x7fd6381414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd63e400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd6380ce630 0x7fd63815f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42626/0 (socket says 172.21.15.40:42626) 2024-03-24T00:31:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd63e400640 1 -- 172.21.15.40:0/2004541087 learned_addr learned my addr 172.21.15.40:0/2004541087 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd63d000640 1 --2- 172.21.15.40:0/2004541087 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6381419f0 0x7fd6380ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd63da00640 1 -- 172.21.15.40:0/2004541087 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6381419f0 msgr2=0x7fd6380ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:55.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd63da00640 1 --2- 172.21.15.40:0/2004541087 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6381419f0 0x7fd6380ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd63da00640 1 -- 172.21.15.40:0/2004541087 >> v1:172.21.15.40:6789/0 conn(0x7fd6380ce630 legacy=0x7fd63815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.341+0000 7fd63da00640 1 -- 172.21.15.40:0/2004541087 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6380cf100 con 0x7fd63813ec80 2024-03-24T00:31:55.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd63da00640 1 --2- 172.21.15.40:0/2004541087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63813ec80 0x7fd6381414b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd62c0097c0 tx=0x7fd62c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4235420b643202b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:55.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd637e00640 1 -- 172.21.15.40:0/2004541087 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c0aa620 con 0x7fd63813ec80 2024-03-24T00:31:55.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd637e00640 1 -- 172.21.15.40:0/2004541087 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62c0af070 con 0x7fd63813ec80 2024-03-24T00:31:55.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd637e00640 1 -- 172.21.15.40:0/2004541087 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c0aa9e0 con 0x7fd63813ec80 2024-03-24T00:31:55.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd640400640 1 -- 172.21.15.40:0/2004541087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63813ec80 msgr2=0x7fd6381414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd640400640 1 --2- 172.21.15.40:0/2004541087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63813ec80 0x7fd6381414b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd62c0097c0 tx=0x7fd62c01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd640400640 1 -- 172.21.15.40:0/2004541087 shutdown_connections 2024-03-24T00:31:55.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd640400640 1 --2- 172.21.15.40:0/2004541087 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6381419f0 0x7fd6380ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd640400640 1 --2- 172.21.15.40:0/2004541087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63813ec80 0x7fd6381414b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd640400640 1 -- 172.21.15.40:0/2004541087 >> 172.21.15.40:0/2004541087 conn(0x7fd6380b17b0 msgr2=0x7fd6380b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:55.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd640400640 1 -- 172.21.15.40:0/2004541087 shutdown_connections 2024-03-24T00:31:55.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd640400640 1 -- 172.21.15.40:0/2004541087 wait complete. 2024-03-24T00:31:55.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.345+0000 7fd640400640 1 Processor -- start 2024-03-24T00:31:55.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd640400640 1 -- start start 2024-03-24T00:31:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd640400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6380ce630 0x7fd638139830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd640400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63813ec80 0x7fd63813d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd640400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6381419f0 0x7fd638137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd640400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd638168ec0 con 0x7fd63813ec80 2024-03-24T00:31:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd640400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd638168d40 con 0x7fd6381419f0 2024-03-24T00:31:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd640400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd638169040 con 0x7fd6380ce630 2024-03-24T00:31:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd63d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63813ec80 0x7fd63813d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd63d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63813ec80 0x7fd63813d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34144/0 (socket says 172.21.15.40:34144) 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd63d000640 1 -- 172.21.15.40:0/1254471020 learned_addr learned my addr 172.21.15.40:0/1254471020 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd63da00640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6380ce630 0x7fd638139830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd63e400640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6381419f0 0x7fd638137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd63d000640 1 -- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6380ce630 msgr2=0x7fd638139830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd63d000640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6380ce630 0x7fd638139830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd63d000640 1 -- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6381419f0 msgr2=0x7fd638137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd63d000640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6381419f0 0x7fd638137cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd63d000640 1 -- 172.21.15.40:0/1254471020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd638138790 con 0x7fd63813ec80 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd63e400640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6381419f0 0x7fd638137cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd63d000640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63813ec80 0x7fd63813d940 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd62400d830 tx=0x7fd62400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd635600640 1 -- 172.21.15.40:0/1254471020 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62400b660 con 0x7fd63813ec80 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd635600640 1 -- 172.21.15.40:0/1254471020 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd624019070 con 0x7fd63813ec80 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd635600640 1 -- 172.21.15.40:0/1254471020 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62400ba00 con 0x7fd63813ec80 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd640400640 1 -- 172.21.15.40:0/1254471020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd638138a70 con 0x7fd63813ec80 2024-03-24T00:31:55.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd640400640 1 -- 172.21.15.40:0/1254471020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd638139030 con 0x7fd63813ec80 2024-03-24T00:31:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.349+0000 7fd640400640 1 -- 172.21.15.40:0/1254471020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd638068170 con 0x7fd63813ec80 2024-03-24T00:31:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.353+0000 7fd635600640 1 -- 172.21.15.40:0/1254471020 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd62401e460 con 0x7fd63813ec80 2024-03-24T00:31:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.353+0000 7fd635600640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd618067660 0x7fd618069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.353+0000 7fd635600640 1 -- 172.21.15.40:0/1254471020 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5833+0+0 (secure 0 0 0) 0x7fd624085fd0 con 0x7fd63813ec80 2024-03-24T00:31:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.353+0000 7fd63da00640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd618067660 0x7fd618069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.353+0000 7fd63da00640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd618067660 0x7fd618069b20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd62c0ae000 tx=0x7fd62c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:55.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.357+0000 7fd635600640 1 -- 172.21.15.40:0/1254471020 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd624056b30 con 0x7fd63813ec80 2024-03-24T00:31:55.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.557+0000 7fd640400640 1 -- 172.21.15.40:0/1254471020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd638072b90 con 0x7fd63813ec80 2024-03-24T00:31:55.562 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-03-24T00:31:55.557+0000 7fa64e600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:31:55.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.557+0000 7fd635600640 1 -- 172.21.15.40:0/1254471020 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd624056430 con 0x7fd63813ec80 2024-03-24T00:31:55.564 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:31:55.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.561+0000 7fd640400640 1 -- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd618067660 msgr2=0x7fd618069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.561+0000 7fd640400640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd618067660 0x7fd618069b20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fd62c0ae000 tx=0x7fd62c0a7000 comp rx=0 tx=0).stop 2024-03-24T00:31:55.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.561+0000 7fd640400640 1 -- 172.21.15.40:0/1254471020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63813ec80 msgr2=0x7fd63813d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.561+0000 7fd640400640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63813ec80 0x7fd63813d940 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd62400d830 tx=0x7fd62400dcf0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.561+0000 7fd640400640 1 -- 172.21.15.40:0/1254471020 shutdown_connections 2024-03-24T00:31:55.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.561+0000 7fd640400640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd618067660 0x7fd618069b20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.561+0000 7fd640400640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6381419f0 0x7fd638137cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.561+0000 7fd640400640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd63813ec80 0x7fd63813d940 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.561+0000 7fd640400640 1 --2- 172.21.15.40:0/1254471020 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6380ce630 0x7fd638139830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.561+0000 7fd640400640 1 -- 172.21.15.40:0/1254471020 >> 172.21.15.40:0/1254471020 conn(0x7fd6380b17b0 msgr2=0x7fd63813f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:55.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.561+0000 7fd640400640 1 -- 172.21.15.40:0/1254471020 shutdown_connections 2024-03-24T00:31:55.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.561+0000 7fd640400640 1 -- 172.21.15.40:0/1254471020 wait complete. 2024-03-24T00:31:55.574 INFO:tasks.ceph.mds.c.smithi040.stderr: -3348> 2024-03-24T00:31:04.210+0000 7f6bcb800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:31:55.587 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:31:55.628 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:31:55.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.729+0000 7f013a400640 1 Processor -- start 2024-03-24T00:31:55.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f013a400640 1 -- start start 2024-03-24T00:31:55.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f013a400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f013415ac60 0x7f01340b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f013a400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01340b3260 con 0x7f0134071b00 2024-03-24T00:31:55.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f013a400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01340b3540 con 0x7f013406cb80 2024-03-24T00:31:55.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f013a400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01340b3e70 con 0x7f013415ac60 2024-03-24T00:31:55.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f0133e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0134071b00 0x7f01340ccd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42638/0 (socket says 172.21.15.40:42638) 2024-03-24T00:31:55.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f0133e00640 1 -- 172.21.15.40:0/3571512948 learned_addr learned my addr 172.21.15.40:0/3571512948 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:55.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f0133400640 1 --2- 172.21.15.40:0/3571512948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f013415ac60 0x7f01340b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f0132a00640 1 -- 172.21.15.40:0/3571512948 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391361297 0 0) 0x7f01340b3260 con 0x7f0134071b00 2024-03-24T00:31:55.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f0132a00640 1 -- 172.21.15.40:0/3571512948 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0118003530 con 0x7f0134071b00 2024-03-24T00:31:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f0132a00640 1 -- 172.21.15.40:0/3571512948 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335238026 0 0) 0x7f01340b3540 con 0x7f013406cb80 2024-03-24T00:31:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f0132a00640 1 -- 172.21.15.40:0/3571512948 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01340b3260 con 0x7f013406cb80 2024-03-24T00:31:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f0133400640 1 -- 172.21.15.40:0/3571512948 >> v1:172.21.15.53:6789/0 conn(0x7f013406cb80 legacy=0x7f01340b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f0133400640 1 -- 172.21.15.40:0/3571512948 >> v1:172.21.15.40:6789/0 conn(0x7f0134071b00 legacy=0x7f01340ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.733+0000 7f0133400640 1 -- 172.21.15.40:0/3571512948 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01340b3820 con 0x7f013415ac60 2024-03-24T00:31:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f0133400640 1 --2- 172.21.15.40:0/3571512948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f013415ac60 0x7f01340b2960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0124009dc0 tx=0x7f012401c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ec7453d7e092f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f0132a00640 1 -- 172.21.15.40:0/3571512948 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240b2070 con 0x7f013415ac60 2024-03-24T00:31:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f0132a00640 1 -- 172.21.15.40:0/3571512948 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01240a59b0 con 0x7f013415ac60 2024-03-24T00:31:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f0132a00640 1 -- 172.21.15.40:0/3571512948 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240a5d80 con 0x7f013415ac60 2024-03-24T00:31:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f013a400640 1 -- 172.21.15.40:0/3571512948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f013415ac60 msgr2=0x7f01340b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f013a400640 1 --2- 172.21.15.40:0/3571512948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f013415ac60 0x7f01340b2960 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0124009dc0 tx=0x7f012401c8d0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f013a400640 1 -- 172.21.15.40:0/3571512948 shutdown_connections 2024-03-24T00:31:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f013a400640 1 --2- 172.21.15.40:0/3571512948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f013415ac60 0x7f01340b2960 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f013a400640 1 -- 172.21.15.40:0/3571512948 >> 172.21.15.40:0/3571512948 conn(0x7f01340b1660 msgr2=0x7f01340b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f013a400640 1 -- 172.21.15.40:0/3571512948 shutdown_connections 2024-03-24T00:31:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f013a400640 1 -- 172.21.15.40:0/3571512948 wait complete. 2024-03-24T00:31:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f013a400640 1 Processor -- start 2024-03-24T00:31:55.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.737+0000 7f013a400640 1 -- start start 2024-03-24T00:31:55.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f013a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f013406cb80 0x7f01341432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f013a400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0134071b00 0x7f0134156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f013a400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f013415ac60 0x7f01340cf980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f013a400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01340b3020 con 0x7f013406cb80 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f013a400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01340b2ea0 con 0x7f013415ac60 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f013a400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013415b040 con 0x7f0134071b00 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f0133400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f013406cb80 0x7f01341432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f0133400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f013406cb80 0x7f01341432f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34168/0 (socket says 172.21.15.40:34168) 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f0133400640 1 -- 172.21.15.40:0/1663981290 learned_addr learned my addr 172.21.15.40:0/1663981290 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f012aa00640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0134071b00 0x7f0134156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f0133e00640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f013415ac60 0x7f01340cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f0133400640 1 -- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0134071b00 msgr2=0x7f0134156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f0133400640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0134071b00 0x7f0134156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f0133400640 1 -- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f013415ac60 msgr2=0x7f01340cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f0133400640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f013415ac60 0x7f01340cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f0133400640 1 -- 172.21.15.40:0/1663981290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0134155270 con 0x7f013406cb80 2024-03-24T00:31:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f0133400640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f013406cb80 0x7f01341432f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01240b0040 tx=0x7f0124019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:55.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f012be00640 1 -- 172.21.15.40:0/1663981290 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240bd020 con 0x7f013406cb80 2024-03-24T00:31:55.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f012be00640 1 -- 172.21.15.40:0/1663981290 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01240b8670 con 0x7f013406cb80 2024-03-24T00:31:55.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f013a400640 1 -- 172.21.15.40:0/1663981290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0134155550 con 0x7f013406cb80 2024-03-24T00:31:55.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f012be00640 1 -- 172.21.15.40:0/1663981290 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01240b2040 con 0x7f013406cb80 2024-03-24T00:31:55.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f013a400640 1 -- 172.21.15.40:0/1663981290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0134155b10 con 0x7f013406cb80 2024-03-24T00:31:55.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.741+0000 7f013a400640 1 -- 172.21.15.40:0/1663981290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0134068170 con 0x7f013406cb80 2024-03-24T00:31:55.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.745+0000 7f012be00640 1 -- 172.21.15.40:0/1663981290 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f01240c0020 con 0x7f013406cb80 2024-03-24T00:31:55.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.745+0000 7f012be00640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0110067660 0x7f0110069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:55.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.745+0000 7f012be00640 1 -- 172.21.15.40:0/1663981290 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5919+0+0 (secure 0 0 0) 0x7f0124120820 con 0x7f013406cb80 2024-03-24T00:31:55.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.745+0000 7f012aa00640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0110067660 0x7f0110069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:55.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.745+0000 7f012aa00640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0110067660 0x7f0110069b20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f013415e3e0 tx=0x7f011c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:55.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.749+0000 7f012be00640 1 -- 172.21.15.40:0/1663981290 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f01240c7030 con 0x7f013406cb80 2024-03-24T00:31:55.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:55.949+0000 7f013a400640 1 -- 172.21.15.40:0/1663981290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f01340b1410 con 0x7f013406cb80 2024-03-24T00:31:56.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.561+0000 7f012be00640 1 -- 172.21.15.40:0/1663981290 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0124016580 con 0x7f013406cb80 2024-03-24T00:31:56.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.565+0000 7f013a400640 1 -- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0110067660 msgr2=0x7f0110069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.565+0000 7f013a400640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0110067660 0x7f0110069b20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f013415e3e0 tx=0x7f011c009040 comp rx=0 tx=0).stop 2024-03-24T00:31:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.565+0000 7f013a400640 1 -- 172.21.15.40:0/1663981290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f013406cb80 msgr2=0x7f01341432f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.565+0000 7f013a400640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f013406cb80 0x7f01341432f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f01240b0040 tx=0x7f0124019670 comp rx=0 tx=0).stop 2024-03-24T00:31:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.565+0000 7f013a400640 1 -- 172.21.15.40:0/1663981290 shutdown_connections 2024-03-24T00:31:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.565+0000 7f013a400640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0110067660 0x7f0110069b20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.565+0000 7f013a400640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f013415ac60 0x7f01340cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.565+0000 7f013a400640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0134071b00 0x7f0134156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.565+0000 7f013a400640 1 --2- 172.21.15.40:0/1663981290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f013406cb80 0x7f01341432f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.565+0000 7f013a400640 1 -- 172.21.15.40:0/1663981290 >> 172.21.15.40:0/1663981290 conn(0x7f01340b1660 msgr2=0x7f01340ae6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.565+0000 7f013a400640 1 -- 172.21.15.40:0/1663981290 shutdown_connections 2024-03-24T00:31:56.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.565+0000 7f013a400640 1 -- 172.21.15.40:0/1663981290 wait complete. 2024-03-24T00:31:56.593 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:31:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.777+0000 7fe1ed000640 1 Processor -- start 2024-03-24T00:31:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.777+0000 7fe1ed000640 1 -- start start 2024-03-24T00:31:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1ed000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e815e000 0x7fe1e80d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1ed000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e815e3e0 con 0x7fe1e81628d0 2024-03-24T00:31:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1ed000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1e80b5f10 con 0x7fe1e80b42d0 2024-03-24T00:31:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1ed000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e8161ba0 con 0x7fe1e815e000 2024-03-24T00:31:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1dfe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe1e81628d0 0x7fe1e80b3bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42642/0 (socket says 172.21.15.40:42642) 2024-03-24T00:31:56.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1dfe00640 1 -- 172.21.15.40:0/1708145140 learned_addr learned my addr 172.21.15.40:0/1708145140 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5e00640 1 --2- 172.21.15.40:0/1708145140 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e815e000 0x7fe1e80d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053356578 0 0) 0x7fe1e815e3e0 con 0x7fe1e81628d0 2024-03-24T00:31:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1cc003530 con 0x7fe1e81628d0 2024-03-24T00:31:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108544553 0 0) 0x7fe1cc003530 con 0x7fe1e81628d0 2024-03-24T00:31:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e815e000 msgr2=0x7fe1e80d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 --2- 172.21.15.40:0/1708145140 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e815e000 0x7fe1e80d16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 >> v1:172.21.15.53:6789/0 conn(0x7fe1e80b42d0 legacy=0x7fe1e806dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e8161df0 con 0x7fe1e81628d0 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fe1d0001680 con 0x7fe1e81628d0 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 >> v1:172.21.15.40:6789/0 conn(0x7fe1e81628d0 legacy=0x7fe1e80b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 --2- 172.21.15.40:0/1708145140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0039f0 0x7fe1cc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 --2- 172.21.15.40:0/1708145140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1cc004810 0x7fe1cc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 --2- 172.21.15.40:0/1708145140 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e815e000 0x7fe1cc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e8161ba0 con 0x7fe1cc004810 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc00ba80 con 0x7fe1cc0039f0 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc00bc80 con 0x7fe1e815e000 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1dfe00640 1 --2- 172.21.15.40:0/1708145140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1cc004810 0x7fe1cc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e5e00640 1 --2- 172.21.15.40:0/1708145140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0039f0 0x7fe1cc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1e6800640 1 --2- 172.21.15.40:0/1708145140 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e815e000 0x7fe1cc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1dfe00640 1 -- 172.21.15.40:0/1708145140 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e815e000 msgr2=0x7fe1cc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1dfe00640 1 --2- 172.21.15.40:0/1708145140 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e815e000 0x7fe1cc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1dfe00640 1 -- 172.21.15.40:0/1708145140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0039f0 msgr2=0x7fe1cc003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1dfe00640 1 --2- 172.21.15.40:0/1708145140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1cc0039f0 0x7fe1cc003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.781+0000 7fe1dfe00640 1 -- 172.21.15.40:0/1708145140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc003530 con 0x7fe1cc004810 2024-03-24T00:31:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.785+0000 7fe1dfe00640 1 --2- 172.21.15.40:0/1708145140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1cc004810 0x7fe1cc004c90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000c190 tx=0x7fe1d001f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a491431667545c5a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.785+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0007d00 con 0x7fe1cc004810 2024-03-24T00:31:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.785+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d0005b00 con 0x7fe1cc004810 2024-03-24T00:31:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.785+0000 7fe1e5400640 1 -- 172.21.15.40:0/1708145140 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d0007070 con 0x7fe1cc004810 2024-03-24T00:31:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.785+0000 7fe1ed000640 1 -- 172.21.15.40:0/1708145140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1cc004810 msgr2=0x7fe1cc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.785+0000 7fe1ed000640 1 --2- 172.21.15.40:0/1708145140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1cc004810 0x7fe1cc004c90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe1d000c190 tx=0x7fe1d001f370 comp rx=0 tx=0).stop 2024-03-24T00:31:56.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.785+0000 7fe1e6800640 1 -- 172.21.15.40:0/1708145140 reap_dead start 2024-03-24T00:31:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.785+0000 7fe1ed000640 1 -- 172.21.15.40:0/1708145140 shutdown_connections 2024-03-24T00:31:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.785+0000 7fe1ed000640 1 -- 172.21.15.40:0/1708145140 >> 172.21.15.40:0/1708145140 conn(0x7fe1e80d0bb0 msgr2=0x7fe1e80d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.785+0000 7fe1ed000640 1 -- 172.21.15.40:0/1708145140 shutdown_connections 2024-03-24T00:31:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.785+0000 7fe1ed000640 1 -- 172.21.15.40:0/1708145140 wait complete. 2024-03-24T00:31:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1ed000640 1 Processor -- start 2024-03-24T00:31:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1ed000640 1 -- start start 2024-03-24T00:31:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1ed000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1cc004810 0x7fe1e8150db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1ed000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e8150510 0x7fe1e814f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1ed000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1e80af000 0x7fe1e80af480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1ed000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e80b5b30 con 0x7fe1cc004810 2024-03-24T00:31:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1ed000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e81623b0 con 0x7fe1e80af000 2024-03-24T00:31:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1ed000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e80b5cb0 con 0x7fe1e8150510 2024-03-24T00:31:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1e5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1cc004810 0x7fe1e8150db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1e5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1cc004810 0x7fe1e8150db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34178/0 (socket says 172.21.15.40:34178) 2024-03-24T00:31:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1e5e00640 1 -- 172.21.15.40:0/1885666114 learned_addr learned my addr 172.21.15.40:0/1885666114 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1e6800640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1e80af000 0x7fe1e80af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1dfe00640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e8150510 0x7fe1e814f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1e5e00640 1 -- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e8150510 msgr2=0x7fe1e814f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1e5e00640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e8150510 0x7fe1e814f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1e5e00640 1 -- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1e80af000 msgr2=0x7fe1e80af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1e5e00640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1e80af000 0x7fe1e80af480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1e5e00640 1 -- 172.21.15.40:0/1885666114 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e8150990 con 0x7fe1cc004810 2024-03-24T00:31:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1e5e00640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1cc004810 0x7fe1e8150db0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8011a00 tx=0x7fe1d8011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1de000640 1 -- 172.21.15.40:0/1885666114 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d800fb50 con 0x7fe1cc004810 2024-03-24T00:31:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1de000640 1 -- 172.21.15.40:0/1885666114 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d800fcf0 con 0x7fe1cc004810 2024-03-24T00:31:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1ed000640 1 -- 172.21.15.40:0/1885666114 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e8151490 con 0x7fe1cc004810 2024-03-24T00:31:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1de000640 1 -- 172.21.15.40:0/1885666114 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d800fea0 con 0x7fe1cc004810 2024-03-24T00:31:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1ed000640 1 -- 172.21.15.40:0/1885666114 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e8151a50 con 0x7fe1cc004810 2024-03-24T00:31:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.789+0000 7fe1ed000640 1 -- 172.21.15.40:0/1885666114 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e8068170 con 0x7fe1cc004810 2024-03-24T00:31:56.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.793+0000 7fe1de000640 1 -- 172.21.15.40:0/1885666114 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe1d8012040 con 0x7fe1cc004810 2024-03-24T00:31:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.793+0000 7fe1de000640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe1c4067690 0x7fe1c4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.793+0000 7fe1de000640 1 -- 172.21.15.40:0/1885666114 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5919+0+0 (secure 0 0 0) 0x7fe1d8085f80 con 0x7fe1cc004810 2024-03-24T00:31:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.793+0000 7fe1dfe00640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe1c4067690 0x7fe1c4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.793+0000 7fe1dfe00640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe1c4067690 0x7fe1c4069b50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00ae000 tx=0x7fe1d00b4040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:56.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.797+0000 7fe1de000640 1 -- 172.21.15.40:0/1885666114 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe1d8019030 con 0x7fe1cc004810 2024-03-24T00:31:56.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:56.961+0000 7fe1ed000640 1 -- 172.21.15.40:0/1885666114 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe1e8067b90 con 0x7fe1cc004810 2024-03-24T00:31:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.569+0000 7fe1de000640 1 -- 172.21.15.40:0/1885666114 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe1d8056500 con 0x7fe1cc004810 2024-03-24T00:31:57.572 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:31:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.569+0000 7fe1ed000640 1 -- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe1c4067690 msgr2=0x7fe1c4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.569+0000 7fe1ed000640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe1c4067690 0x7fe1c4069b50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00ae000 tx=0x7fe1d00b4040 comp rx=0 tx=0).stop 2024-03-24T00:31:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.569+0000 7fe1ed000640 1 -- 172.21.15.40:0/1885666114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1cc004810 msgr2=0x7fe1e8150db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.569+0000 7fe1ed000640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1cc004810 0x7fe1e8150db0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8011a00 tx=0x7fe1d8011ec0 comp rx=0 tx=0).stop 2024-03-24T00:31:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.569+0000 7fe1ed000640 1 -- 172.21.15.40:0/1885666114 shutdown_connections 2024-03-24T00:31:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.569+0000 7fe1ed000640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe1c4067690 0x7fe1c4069b50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.569+0000 7fe1ed000640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1e80af000 0x7fe1e80af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.569+0000 7fe1ed000640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1e8150510 0x7fe1e814f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.569+0000 7fe1ed000640 1 --2- 172.21.15.40:0/1885666114 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe1cc004810 0x7fe1e8150db0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.569+0000 7fe1ed000640 1 -- 172.21.15.40:0/1885666114 >> 172.21.15.40:0/1885666114 conn(0x7fe1e80d0bb0 msgr2=0x7fe1e806e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:57.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.573+0000 7fe1ed000640 1 -- 172.21.15.40:0/1885666114 shutdown_connections 2024-03-24T00:31:57.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.573+0000 7fe1ed000640 1 -- 172.21.15.40:0/1885666114 wait complete. 2024-03-24T00:31:57.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:31:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.745+0000 7f59d8400640 1 Processor -- start 2024-03-24T00:31:57.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.745+0000 7f59d8400640 1 -- start start 2024-03-24T00:31:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d8400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f59d015bd00 0x7f59d00ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d8400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d0072d60 con 0x7f59d00728d0 2024-03-24T00:31:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d8400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d0071de0 con 0x7f59d015ffd0 2024-03-24T00:31:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d8400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d00cea50 con 0x7f59d015bd00 2024-03-24T00:31:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d5000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f59d00728d0 0x7f59d015f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42558/0 (socket says 172.21.15.40:42558) 2024-03-24T00:31:57.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d5000640 1 -- 172.21.15.40:0/422777777 learned_addr learned my addr 172.21.15.40:0/422777777 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d5a00640 1 --2- 172.21.15.40:0/422777777 >> v2:172.21.15.53:3300/0 conn(0x7f59d015bd00 0x7f59d00ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462853910 0 0) 0x7f59d0072d60 con 0x7f59d00728d0 2024-03-24T00:31:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b8003530 con 0x7f59d00728d0 2024-03-24T00:31:57.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3103816395 0 0) 0x7f59b8003530 con 0x7f59d00728d0 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 >> v2:172.21.15.53:3300/0 conn(0x7f59d015bd00 msgr2=0x7f59d00ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 --2- 172.21.15.40:0/422777777 >> v2:172.21.15.53:3300/0 conn(0x7f59d015bd00 0x7f59d00ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 >> v1:172.21.15.53:6789/0 conn(0x7f59d015ffd0 legacy=0x7f59d006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d00ced30 con 0x7f59d00728d0 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f59bc001680 con 0x7f59d00728d0 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 >> v1:172.21.15.40:6789/0 conn(0x7f59d00728d0 legacy=0x7f59d015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 --2- 172.21.15.40:0/422777777 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59b80039c0 0x7f59b8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 --2- 172.21.15.40:0/422777777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b80043c0 0x7f59b8004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 --2- 172.21.15.40:0/422777777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59b8005090 0x7f59b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d00cea50 con 0x7f59b80043c0 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b800bff0 con 0x7f59b8005090 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b800c1f0 con 0x7f59b80039c0 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d5000640 1 --2- 172.21.15.40:0/422777777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b80043c0 0x7f59b8004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:57.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d5a00640 1 --2- 172.21.15.40:0/422777777 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59b80039c0 0x7f59b8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d6400640 1 --2- 172.21.15.40:0/422777777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59b8005090 0x7f59b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d5000640 1 -- 172.21.15.40:0/422777777 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59b80039c0 msgr2=0x7f59b8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d5000640 1 --2- 172.21.15.40:0/422777777 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59b80039c0 0x7f59b8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d5000640 1 -- 172.21.15.40:0/422777777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59b8005090 msgr2=0x7f59b800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d5000640 1 --2- 172.21.15.40:0/422777777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59b8005090 0x7f59b800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d5000640 1 -- 172.21.15.40:0/422777777 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b8003530 con 0x7f59b80043c0 2024-03-24T00:31:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d6400640 1 -- 172.21.15.40:0/422777777 reap_dead start 2024-03-24T00:31:57.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.749+0000 7f59d5a00640 1 --2- 172.21.15.40:0/422777777 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59b80039c0 0x7f59b8003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:57.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.753+0000 7f59d5000640 1 --2- 172.21.15.40:0/422777777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b80043c0 0x7f59b8004890 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00c210 tx=0x7f59bc01cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0319662ce53d01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:57.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.753+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59bc007cf0 con 0x7f59b80043c0 2024-03-24T00:31:57.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.753+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59bc0a9070 con 0x7f59b80043c0 2024-03-24T00:31:57.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.753+0000 7f59cfe00640 1 -- 172.21.15.40:0/422777777 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59bc005d80 con 0x7f59b80043c0 2024-03-24T00:31:57.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.753+0000 7f59d8400640 1 -- 172.21.15.40:0/422777777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b80043c0 msgr2=0x7f59b8004890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:57.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.753+0000 7f59d8400640 1 --2- 172.21.15.40:0/422777777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b80043c0 0x7f59b8004890 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f59bc00c210 tx=0x7f59bc01cbe0 comp rx=0 tx=0).stop 2024-03-24T00:31:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.753+0000 7f59d8400640 1 -- 172.21.15.40:0/422777777 shutdown_connections 2024-03-24T00:31:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.753+0000 7f59d8400640 1 --2- 172.21.15.40:0/422777777 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59b80043c0 0x7f59b8004890 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.753+0000 7f59d8400640 1 -- 172.21.15.40:0/422777777 >> 172.21.15.40:0/422777777 conn(0x7f59d00b1660 msgr2=0x7f59d00b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.753+0000 7f59d8400640 1 -- 172.21.15.40:0/422777777 shutdown_connections 2024-03-24T00:31:57.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.753+0000 7f59d8400640 1 -- 172.21.15.40:0/422777777 wait complete. 2024-03-24T00:31:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d8400640 1 Processor -- start 2024-03-24T00:31:57.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d8400640 1 -- start start 2024-03-24T00:31:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d8400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59b80043c0 0x7f59d014a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d8400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d00b3e80 0x7f59d01569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d8400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d00b4300 0x7f59d0156ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d8400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d006d4a0 con 0x7f59d00b4300 2024-03-24T00:31:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d8400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d006d320 con 0x7f59d00b3e80 2024-03-24T00:31:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d8400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d006d620 con 0x7f59b80043c0 2024-03-24T00:31:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d00b4300 0x7f59d0156ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d00b4300 0x7f59d0156ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48028/0 (socket says 172.21.15.40:48028) 2024-03-24T00:31:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d6400640 1 -- 172.21.15.40:0/1254378969 learned_addr learned my addr 172.21.15.40:0/1254378969 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d6400640 1 -- 172.21.15.40:0/1254378969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59b80043c0 msgr2=0x7f59d014a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d6400640 1 --2- 172.21.15.40:0/1254378969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59b80043c0 0x7f59d014a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:57.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d6400640 1 -- 172.21.15.40:0/1254378969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d00b3e80 msgr2=0x7f59d01569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d6400640 1 --2- 172.21.15.40:0/1254378969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d00b3e80 0x7f59d01569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d6400640 1 -- 172.21.15.40:0/1254378969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d015aa70 con 0x7f59d00b4300 2024-03-24T00:31:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d6400640 1 --2- 172.21.15.40:0/1254378969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d00b4300 0x7f59d0156ee0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f59d0067df0 tx=0x7f59c800eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59cd600640 1 -- 172.21.15.40:0/1254378969 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c8003980 con 0x7f59d00b4300 2024-03-24T00:31:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59cd600640 1 -- 172.21.15.40:0/1254378969 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c8018070 con 0x7f59d00b4300 2024-03-24T00:31:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59cd600640 1 -- 172.21.15.40:0/1254378969 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c8003d60 con 0x7f59d00b4300 2024-03-24T00:31:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d8400640 1 -- 172.21.15.40:0/1254378969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d015ad50 con 0x7f59d00b4300 2024-03-24T00:31:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d8400640 1 -- 172.21.15.40:0/1254378969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d0159a40 con 0x7f59d00b4300 2024-03-24T00:31:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.757+0000 7f59d8400640 1 -- 172.21.15.40:0/1254378969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d0068170 con 0x7f59d00b4300 2024-03-24T00:31:57.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.761+0000 7f59cd600640 1 -- 172.21.15.40:0/1254378969 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f59c80050d0 con 0x7f59d00b4300 2024-03-24T00:31:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.761+0000 7f59cd600640 1 --2- 172.21.15.40:0/1254378969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f59b0067690 0x7f59b0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.761+0000 7f59cd600640 1 -- 172.21.15.40:0/1254378969 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5481+0+0 (secure 0 0 0) 0x7f59c8084ff0 con 0x7f59d00b4300 2024-03-24T00:31:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.761+0000 7f59d5a00640 1 --2- 172.21.15.40:0/1254378969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f59b0067690 0x7f59b0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.761+0000 7f59d5a00640 1 --2- 172.21.15.40:0/1254378969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f59b0067690 0x7f59b0069b50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f59d00ae430 tx=0x7f59c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:57.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.765+0000 7f59cd600640 1 -- 172.21.15.40:0/1254378969 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f59c8021020 con 0x7f59d00b4300 2024-03-24T00:31:57.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:57.929+0000 7f59d8400640 1 -- 172.21.15.40:0/1254378969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f59d00716a0 con 0x7f59d00b4300 2024-03-24T00:31:58.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.573+0000 7f59cd600640 1 -- 172.21.15.40:0/1254378969 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7f59c8055cf0 con 0x7f59d00b4300 2024-03-24T00:31:58.579 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-03-24T00:31:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.577+0000 7f59d8400640 1 -- 172.21.15.40:0/1254378969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f59b0067690 msgr2=0x7f59b0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.577+0000 7f59d8400640 1 --2- 172.21.15.40:0/1254378969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f59b0067690 0x7f59b0069b50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f59d00ae430 tx=0x7f59c4008040 comp rx=0 tx=0).stop 2024-03-24T00:31:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.577+0000 7f59d8400640 1 -- 172.21.15.40:0/1254378969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d00b4300 msgr2=0x7f59d0156ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.577+0000 7f59d8400640 1 --2- 172.21.15.40:0/1254378969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d00b4300 0x7f59d0156ee0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f59d0067df0 tx=0x7f59c800eb50 comp rx=0 tx=0).stop 2024-03-24T00:31:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.577+0000 7f59d8400640 1 -- 172.21.15.40:0/1254378969 shutdown_connections 2024-03-24T00:31:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.577+0000 7f59d8400640 1 --2- 172.21.15.40:0/1254378969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f59b0067690 0x7f59b0069b50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.577+0000 7f59d8400640 1 --2- 172.21.15.40:0/1254378969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59d00b4300 0x7f59d0156ee0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.577+0000 7f59d8400640 1 --2- 172.21.15.40:0/1254378969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d00b3e80 0x7f59d01569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.577+0000 7f59d8400640 1 --2- 172.21.15.40:0/1254378969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59b80043c0 0x7f59d014a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.577+0000 7f59d8400640 1 -- 172.21.15.40:0/1254378969 >> 172.21.15.40:0/1254378969 conn(0x7f59d00b1660 msgr2=0x7f59d0070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.581+0000 7f59d8400640 1 -- 172.21.15.40:0/1254378969 shutdown_connections 2024-03-24T00:31:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.581+0000 7f59d8400640 1 -- 172.21.15.40:0/1254378969 wait complete. 2024-03-24T00:31:58.610 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:31:58.611 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:31:58.611 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:31:58.611 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:31:58.611 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:31:58.611 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:31:58.611 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:31:58.611 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:31:58.612 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:31:58.612 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:31:58.612 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:31:58.612 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:31:58.612 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:31:58.612 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:31:58.612 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:31:58.612 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:31:58.626 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:31:58.626 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:31:58.627 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:31:58.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:31:58.628 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:31:58.628 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:31:58.629 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:31:58.629 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:31:58.670 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:31:58.670 DEBUG:teuthology.parallel:result is None 2024-03-24T00:31:58.671 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:31:58.671 DEBUG:teuthology.parallel:result is None 2024-03-24T00:31:58.674 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:31:58.675 DEBUG:teuthology.parallel:result is None 2024-03-24T00:31:58.676 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:31:58.676 DEBUG:teuthology.parallel:result is None 2024-03-24T00:31:58.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:31:58.761 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:31:58.763 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:31:58.766 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:31:58.767 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:31:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.821+0000 7f75d8e00640 1 Processor -- start 2024-03-24T00:31:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.821+0000 7f75d8e00640 1 -- start start 2024-03-24T00:31:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d8e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f75d415f600 0x7f75d40b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d8e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40b1e50 con 0x7f75d40af170 2024-03-24T00:31:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d8e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40b20f0 con 0x7f75d40d2830 2024-03-24T00:31:58.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d8e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d40b3630 con 0x7f75d415f600 2024-03-24T00:31:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d2800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f75d40af170 0x7f75d4165e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42588/0 (socket says 172.21.15.40:42588) 2024-03-24T00:31:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d2800640 1 -- 172.21.15.40:0/2548919191 learned_addr learned my addr 172.21.15.40:0/2548919191 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d1e00640 1 --2- 172.21.15.40:0/2548919191 >> v2:172.21.15.53:3300/0 conn(0x7f75d415f600 0x7f75d40b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d1400640 1 -- 172.21.15.40:0/2548919191 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864341654 0 0) 0x7f75d40b1e50 con 0x7f75d40af170 2024-03-24T00:31:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d1400640 1 -- 172.21.15.40:0/2548919191 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8003530 con 0x7f75d40af170 2024-03-24T00:31:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d1400640 1 -- 172.21.15.40:0/2548919191 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109020615 0 0) 0x7f75d40b20f0 con 0x7f75d40d2830 2024-03-24T00:31:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d1400640 1 -- 172.21.15.40:0/2548919191 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d40b1e50 con 0x7f75d40d2830 2024-03-24T00:31:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d1e00640 1 -- 172.21.15.40:0/2548919191 >> v1:172.21.15.53:6789/0 conn(0x7f75d40d2830 legacy=0x7f75d40aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d1e00640 1 -- 172.21.15.40:0/2548919191 >> v1:172.21.15.40:6789/0 conn(0x7f75d40af170 legacy=0x7f75d4165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:58.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.825+0000 7f75d1e00640 1 -- 172.21.15.40:0/2548919191 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d40b23d0 con 0x7f75d415f600 2024-03-24T00:31:58.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d1e00640 1 --2- 172.21.15.40:0/2548919191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75d415f600 0x7f75d40b2910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f75c40097c0 tx=0x7f75c401ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=707d70e03506fc61 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:58.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d1400640 1 -- 172.21.15.40:0/2548919191 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c40b5070 con 0x7f75d415f600 2024-03-24T00:31:58.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d1400640 1 -- 172.21.15.40:0/2548919191 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c40a8990 con 0x7f75d415f600 2024-03-24T00:31:58.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d1400640 1 -- 172.21.15.40:0/2548919191 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c40a8d60 con 0x7f75d415f600 2024-03-24T00:31:58.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d8e00640 1 -- 172.21.15.40:0/2548919191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75d415f600 msgr2=0x7f75d40b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:58.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d8e00640 1 --2- 172.21.15.40:0/2548919191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75d415f600 0x7f75d40b2910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f75c40097c0 tx=0x7f75c401ec90 comp rx=0 tx=0).stop 2024-03-24T00:31:58.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d8e00640 1 -- 172.21.15.40:0/2548919191 shutdown_connections 2024-03-24T00:31:58.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d8e00640 1 --2- 172.21.15.40:0/2548919191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75d415f600 0x7f75d40b2910 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:58.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d8e00640 1 -- 172.21.15.40:0/2548919191 >> 172.21.15.40:0/2548919191 conn(0x7f75d40d1d10 msgr2=0x7f75d40d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:58.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d8e00640 1 -- 172.21.15.40:0/2548919191 shutdown_connections 2024-03-24T00:31:58.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d8e00640 1 -- 172.21.15.40:0/2548919191 wait complete. 2024-03-24T00:31:58.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d8e00640 1 Processor -- start 2024-03-24T00:31:58.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.829+0000 7f75d8e00640 1 -- start start 2024-03-24T00:31:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d8e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75d40af170 0x7f75d415b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d8e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75d40d2830 0x7f75d409f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75d415f600 0x7f75d406b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d8e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d40b2740 con 0x7f75d415f600 2024-03-24T00:31:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d8e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d40d2ca0 con 0x7f75d40d2830 2024-03-24T00:31:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d8e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d415f9e0 con 0x7f75d40af170 2024-03-24T00:31:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75d415f600 0x7f75d406b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75d415f600 0x7f75d406b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48082/0 (socket says 172.21.15.40:48082) 2024-03-24T00:31:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d2800640 1 -- 172.21.15.40:0/681160308 learned_addr learned my addr 172.21.15.40:0/681160308 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d1e00640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75d40af170 0x7f75d415b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75cbe00640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75d40d2830 0x7f75d409f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:58.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d2800640 1 -- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75d40af170 msgr2=0x7f75d415b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:58.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d2800640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75d40af170 0x7f75d415b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:58.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d2800640 1 -- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75d40d2830 msgr2=0x7f75d409f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:58.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d2800640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75d40d2830 0x7f75d409f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:58.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d2800640 1 -- 172.21.15.40:0/681160308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d409d1c0 con 0x7f75d415f600 2024-03-24T00:31:58.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75cbe00640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75d40d2830 0x7f75d409f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:31:58.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d2800640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75d415f600 0x7f75d406b150 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00cba0 tx=0x7f75cc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75ca000640 1 -- 172.21.15.40:0/681160308 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc017070 con 0x7f75d415f600 2024-03-24T00:31:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75ca000640 1 -- 172.21.15.40:0/681160308 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75cc00ace0 con 0x7f75d415f600 2024-03-24T00:31:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d8e00640 1 -- 172.21.15.40:0/681160308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d409d4a0 con 0x7f75d415f600 2024-03-24T00:31:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75ca000640 1 -- 172.21.15.40:0/681160308 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc012620 con 0x7f75d415f600 2024-03-24T00:31:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d8e00640 1 -- 172.21.15.40:0/681160308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d409cc20 con 0x7f75d415f600 2024-03-24T00:31:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75d8e00640 1 -- 172.21.15.40:0/681160308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d409cf40 con 0x7f75d415f600 2024-03-24T00:31:58.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.833+0000 7f75ca000640 1 -- 172.21.15.40:0/681160308 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f75cc01a070 con 0x7f75d415f600 2024-03-24T00:31:58.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.837+0000 7f75ca000640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f75b0067660 0x7f75b0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:58.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.837+0000 7f75ca000640 1 -- 172.21.15.40:0/681160308 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f75cc085340 con 0x7f75d415f600 2024-03-24T00:31:58.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.837+0000 7f75d1e00640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f75b0067660 0x7f75b0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:58.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.837+0000 7f75d1e00640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f75b0067660 0x7f75b0069b20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f75c40b2000 tx=0x7f75c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:58.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:58.841+0000 7f75ca000640 1 -- 172.21.15.40:0/681160308 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f75d409cf40 con 0x7f75d415f600 2024-03-24T00:31:59.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.017+0000 7f75d8e00640 1 -- 172.21.15.40:0/681160308 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f75d4068170 con 0x7f75d415f600 2024-03-24T00:31:59.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.585+0000 7f75ca000640 1 -- 172.21.15.40:0/681160308 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) v1 ==== 97+0+0 (secure 0 0 0) 0x7f75cc055bd0 con 0x7f75d415f600 2024-03-24T00:31:59.591 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-03-24T00:31:59.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.589+0000 7f75d8e00640 1 -- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f75b0067660 msgr2=0x7f75b0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:59.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.589+0000 7f75d8e00640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f75b0067660 0x7f75b0069b20 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f75c40b2000 tx=0x7f75c4008040 comp rx=0 tx=0).stop 2024-03-24T00:31:59.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.589+0000 7f75d8e00640 1 -- 172.21.15.40:0/681160308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75d415f600 msgr2=0x7f75d406b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:59.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.589+0000 7f75d8e00640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75d415f600 0x7f75d406b150 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00cba0 tx=0x7f75cc00a430 comp rx=0 tx=0).stop 2024-03-24T00:31:59.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.589+0000 7f75d8e00640 1 -- 172.21.15.40:0/681160308 shutdown_connections 2024-03-24T00:31:59.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.589+0000 7f75d8e00640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f75b0067660 0x7f75b0069b20 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:59.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.589+0000 7f75d8e00640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75d415f600 0x7f75d406b150 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:59.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.589+0000 7f75d8e00640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75d40d2830 0x7f75d409f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:59.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.589+0000 7f75d8e00640 1 --2- 172.21.15.40:0/681160308 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75d40af170 0x7f75d415b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:59.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.589+0000 7f75d8e00640 1 -- 172.21.15.40:0/681160308 >> 172.21.15.40:0/681160308 conn(0x7f75d40d1d10 msgr2=0x7f75d406bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:59.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.589+0000 7f75d8e00640 1 -- 172.21.15.40:0/681160308 shutdown_connections 2024-03-24T00:31:59.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.589+0000 7f75d8e00640 1 -- 172.21.15.40:0/681160308 wait complete. 2024-03-24T00:31:59.621 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:31:59.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce88800640 1 Processor -- start 2024-03-24T00:31:59.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce88800640 1 -- start start 2024-03-24T00:31:59.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce88800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 0x7fce8015ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce88800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fce80067590 0x7fce8006e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce88800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8015d3f0 con 0x7fce800ade40 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce88800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce80072900 con 0x7fce80067590 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce88800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8006fb90 con 0x7fce80145790 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce85400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fce800ade40 0x7fce80066e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42590/0 (socket says 172.21.15.40:42590) 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce85400640 1 -- 172.21.15.40:0/3758168086 learned_addr learned my addr 172.21.15.40:0/3758168086 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce85e00640 1 --2- 172.21.15.40:0/3758168086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 0x7fce8015ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce86800640 1 --2- 172.21.15.40:0/3758168086 >> v2:172.21.15.53:3300/0 conn(0x7fce80067590 0x7fce8006e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce84a00640 1 -- 172.21.15.40:0/3758168086 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951829419 0 0) 0x7fce70001120 con 0x7fce800ade40 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce84a00640 1 -- 172.21.15.40:0/3758168086 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce68003530 con 0x7fce800ade40 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce85e00640 1 -- 172.21.15.40:0/3758168086 >> v2:172.21.15.53:3300/0 conn(0x7fce80067590 msgr2=0x7fce8006e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce85e00640 1 --2- 172.21.15.40:0/3758168086 >> v2:172.21.15.53:3300/0 conn(0x7fce80067590 0x7fce8006e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:59.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce85e00640 1 -- 172.21.15.40:0/3758168086 >> v1:172.21.15.40:6789/0 conn(0x7fce800ade40 legacy=0x7fce80066e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:59.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce85e00640 1 -- 172.21.15.40:0/3758168086 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce80145b70 con 0x7fce80145790 2024-03-24T00:31:59.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.757+0000 7fce86800640 1 --2- 172.21.15.40:0/3758168086 >> v2:172.21.15.53:3300/0 conn(0x7fce80067590 0x7fce8006e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:59.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.761+0000 7fce85e00640 1 --2- 172.21.15.40:0/3758168086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 0x7fce8015ceb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fce740097c0 tx=0x7fce7401c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88def375270996c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:59.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.761+0000 7fce84a00640 1 -- 172.21.15.40:0/3758168086 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce740b0070 con 0x7fce80145790 2024-03-24T00:31:59.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.761+0000 7fce84a00640 1 -- 172.21.15.40:0/3758168086 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce740ab600 con 0x7fce80145790 2024-03-24T00:31:59.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.761+0000 7fce84a00640 1 -- 172.21.15.40:0/3758168086 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce740ab9d0 con 0x7fce80145790 2024-03-24T00:31:59.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.761+0000 7fce88800640 1 -- 172.21.15.40:0/3758168086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 msgr2=0x7fce8015ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:59.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.761+0000 7fce88800640 1 --2- 172.21.15.40:0/3758168086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 0x7fce8015ceb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fce740097c0 tx=0x7fce7401c410 comp rx=0 tx=0).stop 2024-03-24T00:31:59.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.761+0000 7fce88800640 1 -- 172.21.15.40:0/3758168086 shutdown_connections 2024-03-24T00:31:59.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.761+0000 7fce88800640 1 --2- 172.21.15.40:0/3758168086 >> v2:172.21.15.53:3300/0 conn(0x7fce80067590 0x7fce8006e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:59.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.761+0000 7fce88800640 1 --2- 172.21.15.40:0/3758168086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 0x7fce8015ceb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:59.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.761+0000 7fce88800640 1 -- 172.21.15.40:0/3758168086 >> 172.21.15.40:0/3758168086 conn(0x7fce8006b150 msgr2=0x7fce8006b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:31:59.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.761+0000 7fce88800640 1 -- 172.21.15.40:0/3758168086 shutdown_connections 2024-03-24T00:31:59.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.761+0000 7fce88800640 1 -- 172.21.15.40:0/3758168086 wait complete. 2024-03-24T00:31:59.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce88800640 1 Processor -- start 2024-03-24T00:31:59.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce88800640 1 -- start start 2024-03-24T00:31:59.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce88800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce80067590 0x7fce80135d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:59.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce88800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce800ade40 0x7fce8013dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:59.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce88800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 0x7fce800b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce88800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8006ee80 con 0x7fce800ade40 2024-03-24T00:31:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce88800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce80072e30 con 0x7fce80067590 2024-03-24T00:31:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce88800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8006f000 con 0x7fce80145790 2024-03-24T00:31:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce85400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce800ade40 0x7fce8013dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce85e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce80067590 0x7fce80135d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce86800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 0x7fce800b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce85400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce800ade40 0x7fce8013dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48088/0 (socket says 172.21.15.40:48088) 2024-03-24T00:31:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce85e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce80067590 0x7fce80135d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:45318/0 (socket says 172.21.15.40:45318) 2024-03-24T00:31:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce85400640 1 -- 172.21.15.40:0/897342102 learned_addr learned my addr 172.21.15.40:0/897342102 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:31:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce86800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 0x7fce800b0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:50436/0 (socket says 172.21.15.40:50436) 2024-03-24T00:31:59.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce86800640 1 -- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce80067590 msgr2=0x7fce80135d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce86800640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce80067590 0x7fce80135d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce86800640 1 -- 172.21.15.40:0/897342102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce800ade40 msgr2=0x7fce8013dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:31:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce86800640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce800ade40 0x7fce8013dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:31:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce86800640 1 -- 172.21.15.40:0/897342102 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce801406d0 con 0x7fce80145790 2024-03-24T00:31:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce85400640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce800ade40 0x7fce8013dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:31:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce86800640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 0x7fce800b0c80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fce7c009e50 tx=0x7fce7c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce7a000640 1 -- 172.21.15.40:0/897342102 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c018030 con 0x7fce80145790 2024-03-24T00:31:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce7a000640 1 -- 172.21.15.40:0/897342102 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c00cd90 con 0x7fce80145790 2024-03-24T00:31:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce88800640 1 -- 172.21.15.40:0/897342102 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce801409b0 con 0x7fce80145790 2024-03-24T00:31:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce7a000640 1 -- 172.21.15.40:0/897342102 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c003bc0 con 0x7fce80145790 2024-03-24T00:31:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce88800640 1 -- 172.21.15.40:0/897342102 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce80140f70 con 0x7fce80145790 2024-03-24T00:31:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.765+0000 7fce88800640 1 -- 172.21.15.40:0/897342102 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce80068170 con 0x7fce80145790 2024-03-24T00:31:59.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.769+0000 7fce7a000640 1 -- 172.21.15.40:0/897342102 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fce7c01b070 con 0x7fce80145790 2024-03-24T00:31:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.769+0000 7fce7a000640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce64067660 0x7fce64069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:31:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.769+0000 7fce7a000640 1 -- 172.21.15.40:0/897342102 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fce7c085e60 con 0x7fce80145790 2024-03-24T00:31:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.769+0000 7fce85e00640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce64067660 0x7fce64069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:31:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.769+0000 7fce85e00640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce64067660 0x7fce64069b20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fce740ad000 tx=0x7fce740a4f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:31:59.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.773+0000 7fce7a000640 1 -- 172.21.15.40:0/897342102 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fce7c010040 con 0x7fce80145790 2024-03-24T00:31:59.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.993+0000 7fce88800640 1 -- 172.21.15.40:0/897342102 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fce8013c9f0 con 0x7fce80145790 2024-03-24T00:31:59.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.993+0000 7fce7a000640 1 -- 172.21.15.40:0/897342102 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fce7c056740 con 0x7fce80145790 2024-03-24T00:31:59.999 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:31:59.999 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:31:59.999 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-03-24T00:31:59.999 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:31:59.999 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-03-24T00:31:59.999 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lCkU6EhAA0q21F90epsGPK5x17s9NLw==", 2024-03-24T00:31:59.999 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:31:59.999 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:31:59.999 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:31:59.999 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:31:59.999 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.000 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.000 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-03-24T00:32:00.000 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llkfOHhAAqKKFyEJzG0UxbLJMFPET8A==", 2024-03-24T00:32:00.000 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.000 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:00.000 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:32:00.000 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:32:00.000 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lKMmwGRAAAKaOSnhTRbHIpPKXXkAe+Q==", 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.002 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-03-24T00:32:00.002 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l7quKJhAA8QsquiFFmj5H1G5yck0FpA==", 2024-03-24T00:32:00.002 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.002 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:00.002 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:32:00.002 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:32:00.002 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.002 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.003 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-03-24T00:32:00.003 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAVdP9lVk3uBxAA9TC1OpeB5GIFlfJl5+ILBg==", 2024-03-24T00:32:00.003 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.003 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:00.003 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:00.003 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.003 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.003 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-03-24T00:32:00.004 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYdP9lUc1IDhAAPwCV9eqL7bFibkfMFt9wPw==", 2024-03-24T00:32:00.004 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.004 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:00.004 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:00.004 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.004 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.004 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-03-24T00:32:00.005 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAbdP9lwjyuFBAAJ1JAIAnLXAn03+lQPwwY9Q==", 2024-03-24T00:32:00.005 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.005 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:00.005 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:00.005 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.005 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.005 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-03-24T00:32:00.006 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAedP9lZ1bgGRAAuLAay48N7jXjj8/yaYwSqg==", 2024-03-24T00:32:00.006 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.006 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:00.006 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:00.006 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.006 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.007 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-03-24T00:32:00.007 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAidP9lvyk4JhAA8w4/XBYHDLoqJ1zop9sUiw==", 2024-03-24T00:32:00.007 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.007 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:00.008 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:00.008 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.008 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.009 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAldP9lcb9qLBAAXts6VU/4kVqUWS/WORV7zQ==", 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAodP9l8ipTMhAAjpagojh+woHyGvlQlIkHDw==", 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:00.010 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQArdP9lmGfpORAAPvow5nF/O7RLmuXEqNWENg==", 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-03-24T00:32:00.011 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw==", 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lpIZrNBAAzpGWZuQ6ndjNMAOmAsebvQ==", 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:32:00.012 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l6gXMMBAAd6Oe6S73hj/WfLXTsYiEZw==", 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llFHCOhAATtv8gcL6BHPnxB1dHrg6vw==", 2024-03-24T00:32:00.013 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAQdP9lmmd7KxAA4uk6aUKP6egYj6O1NgdadQ==", 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:32:00.014 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.bootstrap-mds", 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAxdP9lnVj2OhAA55qsSz77/2wVwCfkwv6u1w==", 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile bootstrap-mds" 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.bootstrap-mgr", 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAxdP9l6mL2OhAAaguNHZsbd+SzRC3XIh1+JQ==", 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-24T00:32:00.015 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.bootstrap-osd", 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAxdP9laGv2OhAA1pJsS7eEANfokuUfSpPChA==", 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile bootstrap-osd" 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.bootstrap-rbd", 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAxdP9lwHP2OhAAU5bwrzAZ3l0iksUTWhsAAw==", 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAxdP9lMHz2OhAAlyaSKHPqRftDEEIep5LMIA==", 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.bootstrap-rgw", 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAxdP9l/4T2OhAAke7nJKbwDHErMScgjrEwkA==", 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.017 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lebNrBxAAvbcCAJhDQn/5Uk1dJ01oHw==", 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lEnZyAxAAW5Lrm7LZPTVfLCOY6As1PQ==", 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:32:00.018 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:00.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:00.019 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-24T00:32:00.019 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:32:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.997+0000 7fce88800640 1 -- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce64067660 msgr2=0x7fce64069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.997+0000 7fce88800640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce64067660 0x7fce64069b20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fce740ad000 tx=0x7fce740a4f90 comp rx=0 tx=0).stop 2024-03-24T00:32:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.997+0000 7fce88800640 1 -- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 msgr2=0x7fce800b0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.997+0000 7fce88800640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 0x7fce800b0c80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fce7c009e50 tx=0x7fce7c00c490 comp rx=0 tx=0).stop 2024-03-24T00:32:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.997+0000 7fce88800640 1 -- 172.21.15.40:0/897342102 shutdown_connections 2024-03-24T00:32:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.997+0000 7fce88800640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce64067660 0x7fce64069b20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.997+0000 7fce88800640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce80145790 0x7fce800b0c80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.997+0000 7fce88800640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce800ade40 0x7fce8013dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.997+0000 7fce88800640 1 --2- 172.21.15.40:0/897342102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce80067590 0x7fce80135d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.997+0000 7fce88800640 1 -- 172.21.15.40:0/897342102 >> 172.21.15.40:0/897342102 conn(0x7fce8006b150 msgr2=0x7fce801444b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:00.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.997+0000 7fce88800640 1 -- 172.21.15.40:0/897342102 shutdown_connections 2024-03-24T00:32:00.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:31:59.997+0000 7fce88800640 1 -- 172.21.15.40:0/897342102 wait complete. 2024-03-24T00:32:00.022 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-24T00:32:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.157+0000 7fe673a00640 1 Processor -- start 2024-03-24T00:32:00.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.157+0000 7fe673a00640 1 -- start start 2024-03-24T00:32:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.161+0000 7fe673a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe66c141bc0 0x7fe66c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.161+0000 7fe673a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66c070e00 con 0x7fe66c13ec80 2024-03-24T00:32:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.161+0000 7fe673a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66c0cee20 con 0x7fe66c0ce630 2024-03-24T00:32:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.161+0000 7fe673a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe66c0cf100 con 0x7fe66c141bc0 2024-03-24T00:32:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.161+0000 7fe663e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe66c141bc0 0x7fe66c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.161+0000 7fe671000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe66c13ec80 0x7fe66c1414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42604/0 (socket says 172.21.15.40:42604) 2024-03-24T00:32:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.161+0000 7fe671000640 1 -- 172.21.15.40:0/2622827370 learned_addr learned my addr 172.21.15.40:0/2622827370 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.161+0000 7fe663e00640 1 -- 172.21.15.40:0/2622827370 >> v1:172.21.15.53:6789/0 conn(0x7fe66c0ce630 legacy=0x7fe66c071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.161+0000 7fe663e00640 1 -- 172.21.15.40:0/2622827370 >> v1:172.21.15.40:6789/0 conn(0x7fe66c13ec80 legacy=0x7fe66c1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.161+0000 7fe663e00640 1 -- 172.21.15.40:0/2622827370 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c0cf3e0 con 0x7fe66c141bc0 2024-03-24T00:32:00.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.161+0000 7fe66be00640 1 -- 172.21.15.40:0/2622827370 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503481177 0 0) 0x7fe66c070e00 con 0x7fe66c13ec80 2024-03-24T00:32:00.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.161+0000 7fe663e00640 1 --2- 172.21.15.40:0/2622827370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe66c141bc0 0x7fe66c0ce0f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe6540097c0 tx=0x7fe65401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6ace4dca72efe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:00.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe66be00640 1 -- 172.21.15.40:0/2622827370 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6540aa620 con 0x7fe66c141bc0 2024-03-24T00:32:00.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe66be00640 1 -- 172.21.15.40:0/2622827370 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6540af070 con 0x7fe66c141bc0 2024-03-24T00:32:00.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe66be00640 1 -- 172.21.15.40:0/2622827370 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6540aa9e0 con 0x7fe66c141bc0 2024-03-24T00:32:00.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 -- 172.21.15.40:0/2622827370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe66c141bc0 msgr2=0x7fe66c0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 --2- 172.21.15.40:0/2622827370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe66c141bc0 0x7fe66c0ce0f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe6540097c0 tx=0x7fe65401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 -- 172.21.15.40:0/2622827370 shutdown_connections 2024-03-24T00:32:00.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 --2- 172.21.15.40:0/2622827370 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe66c141bc0 0x7fe66c0ce0f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 -- 172.21.15.40:0/2622827370 >> 172.21.15.40:0/2622827370 conn(0x7fe66c0b17b0 msgr2=0x7fe66c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:00.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 -- 172.21.15.40:0/2622827370 shutdown_connections 2024-03-24T00:32:00.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 -- 172.21.15.40:0/2622827370 wait complete. 2024-03-24T00:32:00.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 Processor -- start 2024-03-24T00:32:00.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 -- start start 2024-03-24T00:32:00.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe66c0ce630 0x7fe66c157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe66c13ec80 0x7fe66c0b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe66c141bc0 0x7fe66c153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c0725f0 con 0x7fe66c13ec80 2024-03-24T00:32:00.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c072470 con 0x7fe66c141bc0 2024-03-24T00:32:00.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe673a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66c169d50 con 0x7fe66c0ce630 2024-03-24T00:32:00.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe663e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe66c13ec80 0x7fe66c0b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe671000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe66c0ce630 0x7fe66c157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe663e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe66c13ec80 0x7fe66c0b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48092/0 (socket says 172.21.15.40:48092) 2024-03-24T00:32:00.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe663e00640 1 -- 172.21.15.40:0/4138892435 learned_addr learned my addr 172.21.15.40:0/4138892435 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:00.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.165+0000 7fe671a00640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe66c141bc0 0x7fe66c153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe663e00640 1 -- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe66c0ce630 msgr2=0x7fe66c157730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe663e00640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe66c0ce630 0x7fe66c157730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe663e00640 1 -- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe66c141bc0 msgr2=0x7fe66c153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe663e00640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe66c141bc0 0x7fe66c153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe663e00640 1 -- 172.21.15.40:0/4138892435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66c158080 con 0x7fe66c13ec80 2024-03-24T00:32:00.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe663e00640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe66c13ec80 0x7fe66c0b07f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe6540190b0 tx=0x7fe654019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:00.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe669600640 1 -- 172.21.15.40:0/4138892435 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6540aadb0 con 0x7fe66c13ec80 2024-03-24T00:32:00.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe669600640 1 -- 172.21.15.40:0/4138892435 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6540be450 con 0x7fe66c13ec80 2024-03-24T00:32:00.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe669600640 1 -- 172.21.15.40:0/4138892435 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6540be840 con 0x7fe66c13ec80 2024-03-24T00:32:00.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe673a00640 1 -- 172.21.15.40:0/4138892435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe66c158360 con 0x7fe66c13ec80 2024-03-24T00:32:00.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe673a00640 1 -- 172.21.15.40:0/4138892435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe66c158920 con 0x7fe66c13ec80 2024-03-24T00:32:00.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe673a00640 1 -- 172.21.15.40:0/4138892435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66c068170 con 0x7fe66c13ec80 2024-03-24T00:32:00.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe669600640 1 -- 172.21.15.40:0/4138892435 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe6540b33f0 con 0x7fe66c13ec80 2024-03-24T00:32:00.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe669600640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe648067660 0x7fe648069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.169+0000 7fe669600640 1 -- 172.21.15.40:0/4138892435 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe65411f5f0 con 0x7fe66c13ec80 2024-03-24T00:32:00.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.173+0000 7fe671000640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe648067660 0x7fe648069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.173+0000 7fe671000640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe648067660 0x7fe648069b20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe65c007a90 tx=0x7fe65c0079e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:00.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.173+0000 7fe669600640 1 -- 172.21.15.40:0/4138892435 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe6540c5070 con 0x7fe66c13ec80 2024-03-24T00:32:00.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.393+0000 7fe673a00640 1 -- 172.21.15.40:0/4138892435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7fe66c0ce180 con 0x7fe66c13ec80 2024-03-24T00:32:00.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.393+0000 7fe669600640 1 -- 172.21.15.40:0/4138892435 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe654016090 con 0x7fe66c13ec80 2024-03-24T00:32:00.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.397+0000 7fe673a00640 1 -- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe648067660 msgr2=0x7fe648069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.397+0000 7fe673a00640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe648067660 0x7fe648069b20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe65c007a90 tx=0x7fe65c0079e0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.397+0000 7fe673a00640 1 -- 172.21.15.40:0/4138892435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe66c13ec80 msgr2=0x7fe66c0b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.397+0000 7fe673a00640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe66c13ec80 0x7fe66c0b07f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fe6540190b0 tx=0x7fe654019240 comp rx=0 tx=0).stop 2024-03-24T00:32:00.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.397+0000 7fe673a00640 1 -- 172.21.15.40:0/4138892435 shutdown_connections 2024-03-24T00:32:00.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.397+0000 7fe673a00640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe648067660 0x7fe648069b20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.397+0000 7fe673a00640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe66c141bc0 0x7fe66c153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.397+0000 7fe673a00640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe66c13ec80 0x7fe66c0b07f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.397+0000 7fe673a00640 1 --2- 172.21.15.40:0/4138892435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe66c0ce630 0x7fe66c157730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.397+0000 7fe673a00640 1 -- 172.21.15.40:0/4138892435 >> 172.21.15.40:0/4138892435 conn(0x7fe66c0b17b0 msgr2=0x7fe66c06d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:00.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.397+0000 7fe673a00640 1 -- 172.21.15.40:0/4138892435 shutdown_connections 2024-03-24T00:32:00.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.397+0000 7fe673a00640 1 -- 172.21.15.40:0/4138892435 wait complete. 2024-03-24T00:32:00.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-24T00:32:00.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.557+0000 7f55e3c00640 1 Processor -- start 2024-03-24T00:32:00.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.557+0000 7f55e3c00640 1 -- start start 2024-03-24T00:32:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55e3c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55dc0b46c0 0x7f55dc0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55e3c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f55dc0b2ff0 0x7f55dc0ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55e3c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc070f60 con 0x7f55dc06e620 2024-03-24T00:32:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55e3c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc0b3400 con 0x7f55dc0b46c0 2024-03-24T00:32:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55e3c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc0b3620 con 0x7f55dc0b2ff0 2024-03-24T00:32:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55d3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55dc0b46c0 0x7f55dc0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55e1200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f55dc06e620 0x7f55dc06ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42608/0 (socket says 172.21.15.40:42608) 2024-03-24T00:32:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55e1200640 1 -- 172.21.15.40:0/3264464317 learned_addr learned my addr 172.21.15.40:0/3264464317 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:00.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55e1c00640 1 --2- 172.21.15.40:0/3264464317 >> v2:172.21.15.53:3300/0 conn(0x7f55dc0b2ff0 0x7f55dc0ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55dbe00640 1 -- 172.21.15.40:0/3264464317 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054893603 0 0) 0x7f55dc070f60 con 0x7f55dc06e620 2024-03-24T00:32:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55dbe00640 1 -- 172.21.15.40:0/3264464317 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c0005740 con 0x7f55dc06e620 2024-03-24T00:32:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55d3e00640 1 -- 172.21.15.40:0/3264464317 >> v2:172.21.15.53:3300/0 conn(0x7f55dc0b2ff0 msgr2=0x7f55dc0ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55d3e00640 1 --2- 172.21.15.40:0/3264464317 >> v2:172.21.15.53:3300/0 conn(0x7f55dc0b2ff0 0x7f55dc0ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55d3e00640 1 -- 172.21.15.40:0/3264464317 >> v1:172.21.15.40:6789/0 conn(0x7f55dc06e620 legacy=0x7f55dc06ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.561+0000 7f55d3e00640 1 -- 172.21.15.40:0/3264464317 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0b3900 con 0x7f55dc0b46c0 2024-03-24T00:32:00.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55d3e00640 1 --2- 172.21.15.40:0/3264464317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0b46c0 0x7f55dc0b2ab0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0cf680 tx=0x7f55c401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545055ac7b116672 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:00.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55dbe00640 1 -- 172.21.15.40:0/3264464317 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c40b1070 con 0x7f55dc0b46c0 2024-03-24T00:32:00.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55dbe00640 1 -- 172.21.15.40:0/3264464317 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c40ac620 con 0x7f55dc0b46c0 2024-03-24T00:32:00.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55dbe00640 1 -- 172.21.15.40:0/3264464317 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55dc0cdcf0 con 0x7f55dc0b46c0 2024-03-24T00:32:00.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55e3c00640 1 -- 172.21.15.40:0/3264464317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0b46c0 msgr2=0x7f55dc0b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55e3c00640 1 --2- 172.21.15.40:0/3264464317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0b46c0 0x7f55dc0b2ab0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0cf680 tx=0x7f55c401c3c0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55e3c00640 1 -- 172.21.15.40:0/3264464317 shutdown_connections 2024-03-24T00:32:00.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55e3c00640 1 --2- 172.21.15.40:0/3264464317 >> v2:172.21.15.53:3300/0 conn(0x7f55dc0b2ff0 0x7f55dc0ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55e3c00640 1 --2- 172.21.15.40:0/3264464317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0b46c0 0x7f55dc0b2ab0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55e3c00640 1 -- 172.21.15.40:0/3264464317 >> 172.21.15.40:0/3264464317 conn(0x7f55dc0b17b0 msgr2=0x7f55dc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:00.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55e3c00640 1 -- 172.21.15.40:0/3264464317 shutdown_connections 2024-03-24T00:32:00.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55e3c00640 1 -- 172.21.15.40:0/3264464317 wait complete. 2024-03-24T00:32:00.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55e3c00640 1 Processor -- start 2024-03-24T00:32:00.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.565+0000 7f55e3c00640 1 -- start start 2024-03-24T00:32:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e3c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55dc06e620 0x7f55dc159750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e3c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55dc0b2ff0 0x7f55dc0b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e3c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0b46c0 0x7f55dc0b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e3c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc144920 con 0x7f55dc0b46c0 2024-03-24T00:32:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e3c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0b4b50 con 0x7f55dc0b2ff0 2024-03-24T00:32:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e3c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc141fc0 con 0x7f55dc06e620 2024-03-24T00:32:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0b46c0 0x7f55dc0b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0b46c0 0x7f55dc0b0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48108/0 (socket says 172.21.15.40:48108) 2024-03-24T00:32:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e1c00640 1 -- 172.21.15.40:0/1433713542 learned_addr learned my addr 172.21.15.40:0/1433713542 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:00.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55d3e00640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55dc0b2ff0 0x7f55dc0b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e1200640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55dc06e620 0x7f55dc159750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e1c00640 1 -- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55dc06e620 msgr2=0x7f55dc159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e1c00640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55dc06e620 0x7f55dc159750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e1c00640 1 -- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55dc0b2ff0 msgr2=0x7f55dc0b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e1c00640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55dc0b2ff0 0x7f55dc0b0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e1c00640 1 -- 172.21.15.40:0/1433713542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc1422a0 con 0x7f55dc0b46c0 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55d3e00640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55dc0b2ff0 0x7f55dc0b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e1200640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55dc06e620 0x7f55dc159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e1c00640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0b46c0 0x7f55dc0b0da0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f55d400efc0 tx=0x7f55d400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55d9600640 1 -- 172.21.15.40:0/1433713542 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4018020 con 0x7f55dc0b46c0 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55d9600640 1 -- 172.21.15.40:0/1433713542 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d4014480 con 0x7f55dc0b46c0 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55d9600640 1 -- 172.21.15.40:0/1433713542 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4014870 con 0x7f55dc0b46c0 2024-03-24T00:32:00.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e3c00640 1 -- 172.21.15.40:0/1433713542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc142580 con 0x7f55dc0b46c0 2024-03-24T00:32:00.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e3c00640 1 -- 172.21.15.40:0/1433713542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc142b40 con 0x7f55dc0b46c0 2024-03-24T00:32:00.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55e3c00640 1 -- 172.21.15.40:0/1433713542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55dc068170 con 0x7f55dc0b46c0 2024-03-24T00:32:00.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.569+0000 7f55d9600640 1 -- 172.21.15.40:0/1433713542 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f55d401b020 con 0x7f55dc0b46c0 2024-03-24T00:32:00.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.573+0000 7f55d9600640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f55b8067660 0x7f55b8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.573+0000 7f55d9600640 1 -- 172.21.15.40:0/1433713542 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f55d4086460 con 0x7f55dc0b46c0 2024-03-24T00:32:00.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.573+0000 7f55e1200640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f55b8067660 0x7f55b8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.573+0000 7f55e1200640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f55b8067660 0x7f55b8069b20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f55cc000f70 tx=0x7f55cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:00.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.573+0000 7f55d9600640 1 -- 172.21.15.40:0/1433713542 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f55d4022030 con 0x7f55dc0b46c0 2024-03-24T00:32:00.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.797+0000 7f55e3c00640 1 -- 172.21.15.40:0/1433713542 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f55dc072f50 con 0x7f55dc0b46c0 2024-03-24T00:32:00.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55d9600640 1 -- 172.21.15.40:0/1433713542 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f55d4010040 con 0x7f55dc0b46c0 2024-03-24T00:32:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55e3c00640 1 -- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f55b8067660 msgr2=0x7f55b8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55e3c00640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f55b8067660 0x7f55b8069b20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f55cc000f70 tx=0x7f55cc009040 comp rx=0 tx=0).stop 2024-03-24T00:32:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55e3c00640 1 -- 172.21.15.40:0/1433713542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0b46c0 msgr2=0x7f55dc0b0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55e3c00640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0b46c0 0x7f55dc0b0da0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f55d400efc0 tx=0x7f55d400c490 comp rx=0 tx=0).stop 2024-03-24T00:32:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55e3c00640 1 -- 172.21.15.40:0/1433713542 shutdown_connections 2024-03-24T00:32:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55e3c00640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f55b8067660 0x7f55b8069b20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55e3c00640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55dc0b46c0 0x7f55dc0b0da0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55e3c00640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55dc0b2ff0 0x7f55dc0b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55e3c00640 1 --2- 172.21.15.40:0/1433713542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55dc06e620 0x7f55dc159750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55e3c00640 1 -- 172.21.15.40:0/1433713542 >> 172.21.15.40:0/1433713542 conn(0x7f55dc0b17b0 msgr2=0x7f55dc0ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:00.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55e3c00640 1 -- 172.21.15.40:0/1433713542 shutdown_connections 2024-03-24T00:32:00.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.801+0000 7f55e3c00640 1 -- 172.21.15.40:0/1433713542 wait complete. 2024-03-24T00:32:00.827 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-24T00:32:00.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aac600640 1 Processor -- start 2024-03-24T00:32:00.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aac600640 1 -- start start 2024-03-24T00:32:00.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aac600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2aa406e620 0x7f2aa40b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aac600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2aa40ccfa0 0x7f2aa40cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aac600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2aa4070f60 con 0x7f2aa406ea00 2024-03-24T00:32:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aac600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa40b4c00 con 0x7f2aa40ccfa0 2024-03-24T00:32:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aac600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2aa40b3620 con 0x7f2aa406e620 2024-03-24T00:32:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aaa600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2aa40ccfa0 0x7f2aa40cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aaa600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2aa40ccfa0 0x7f2aa40cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48124/0 (socket says 172.21.15.40:48124) 2024-03-24T00:32:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aaa600640 1 -- 172.21.15.40:0/1785796668 learned_addr learned my addr 172.21.15.40:0/1785796668 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aa9c00640 1 --2- 172.21.15.40:0/1785796668 >> v2:172.21.15.53:3300/0 conn(0x7f2aa406e620 0x7f2aa40b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aaa600640 1 -- 172.21.15.40:0/1785796668 >> v2:172.21.15.53:3300/0 conn(0x7f2aa406e620 msgr2=0x7f2aa40b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aaa600640 1 --2- 172.21.15.40:0/1785796668 >> v2:172.21.15.53:3300/0 conn(0x7f2aa406e620 0x7f2aa40b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aaa600640 1 -- 172.21.15.40:0/1785796668 >> v1:172.21.15.53:6789/0 conn(0x7f2aa406ea00 legacy=0x7f2aa40b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aaa600640 1 -- 172.21.15.40:0/1785796668 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa40b3900 con 0x7f2aa40ccfa0 2024-03-24T00:32:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.965+0000 7f2aa9c00640 1 --2- 172.21.15.40:0/1785796668 >> v2:172.21.15.53:3300/0 conn(0x7f2aa406e620 0x7f2aa40b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:00.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.969+0000 7f2aaa600640 1 --2- 172.21.15.40:0/1785796668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aa40ccfa0 0x7f2aa40cd380 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2aa406c5f0 tx=0x7f2a9c01eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd7b9df5a7bd797 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:00.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.969+0000 7f2aa3e00640 1 -- 172.21.15.40:0/1785796668 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c00b2d0 con 0x7f2aa40ccfa0 2024-03-24T00:32:00.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.969+0000 7f2aa3e00640 1 -- 172.21.15.40:0/1785796668 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a9c00b470 con 0x7f2aa40ccfa0 2024-03-24T00:32:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.969+0000 7f2aa3e00640 1 -- 172.21.15.40:0/1785796668 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a9c0a7a30 con 0x7f2aa40ccfa0 2024-03-24T00:32:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.969+0000 7f2aac600640 1 -- 172.21.15.40:0/1785796668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aa40ccfa0 msgr2=0x7f2aa40cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.969+0000 7f2aac600640 1 --2- 172.21.15.40:0/1785796668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aa40ccfa0 0x7f2aa40cd380 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2aa406c5f0 tx=0x7f2a9c01eb30 comp rx=0 tx=0).stop 2024-03-24T00:32:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.969+0000 7f2aac600640 1 -- 172.21.15.40:0/1785796668 shutdown_connections 2024-03-24T00:32:00.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.969+0000 7f2aac600640 1 --2- 172.21.15.40:0/1785796668 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aa40ccfa0 0x7f2aa40cd380 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.969+0000 7f2aac600640 1 --2- 172.21.15.40:0/1785796668 >> v2:172.21.15.53:3300/0 conn(0x7f2aa406e620 0x7f2aa40b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.969+0000 7f2aac600640 1 -- 172.21.15.40:0/1785796668 >> 172.21.15.40:0/1785796668 conn(0x7f2aa40b17b0 msgr2=0x7f2aa40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.969+0000 7f2aac600640 1 -- 172.21.15.40:0/1785796668 shutdown_connections 2024-03-24T00:32:00.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.969+0000 7f2aac600640 1 -- 172.21.15.40:0/1785796668 wait complete. 2024-03-24T00:32:00.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aac600640 1 Processor -- start 2024-03-24T00:32:00.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aac600640 1 -- start start 2024-03-24T00:32:00.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aac600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2aa406e620 0x7f2aa413d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aac600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aa406ea00 0x7f2aa415b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aac600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2aa40ccfa0 0x7f2aa4159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aac600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa4144920 con 0x7f2aa406ea00 2024-03-24T00:32:00.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aac600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa406ee10 con 0x7f2aa40ccfa0 2024-03-24T00:32:00.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aac600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2aa41574e0 con 0x7f2aa406e620 2024-03-24T00:32:00.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aa406ea00 0x7f2aa415b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aa406ea00 0x7f2aa415b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48134/0 (socket says 172.21.15.40:48134) 2024-03-24T00:32:00.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa9200640 1 -- 172.21.15.40:0/306883066 learned_addr learned my addr 172.21.15.40:0/306883066 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa9c00640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2aa406e620 0x7f2aa413d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aaa600640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2aa40ccfa0 0x7f2aa4159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa9200640 1 -- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2aa406e620 msgr2=0x7f2aa413d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa9200640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2aa406e620 0x7f2aa413d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa9200640 1 -- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2aa40ccfa0 msgr2=0x7f2aa4159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa9200640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2aa40ccfa0 0x7f2aa4159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa9200640 1 -- 172.21.15.40:0/306883066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa41577c0 con 0x7f2aa406ea00 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa9200640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aa406ea00 0x7f2aa415b310 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2a90007fd0 tx=0x7f2a9000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa1600640 1 -- 172.21.15.40:0/306883066 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90017070 con 0x7f2aa406ea00 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa1600640 1 -- 172.21.15.40:0/306883066 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a900052d0 con 0x7f2aa406ea00 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aa1600640 1 -- 172.21.15.40:0/306883066 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a90003840 con 0x7f2aa406ea00 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aac600640 1 -- 172.21.15.40:0/306883066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2aa4157aa0 con 0x7f2aa406ea00 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aac600640 1 -- 172.21.15.40:0/306883066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2aa4158060 con 0x7f2aa406ea00 2024-03-24T00:32:00.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.973+0000 7f2aac600640 1 -- 172.21.15.40:0/306883066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2aa4068170 con 0x7f2aa406ea00 2024-03-24T00:32:00.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.977+0000 7f2aa1600640 1 -- 172.21.15.40:0/306883066 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2a9001a070 con 0x7f2aa406ea00 2024-03-24T00:32:00.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.977+0000 7f2aa1600640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a84067690 0x7f2a84069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:00.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.977+0000 7f2aa1600640 1 -- 172.21.15.40:0/306883066 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2a90084820 con 0x7f2aa406ea00 2024-03-24T00:32:00.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.977+0000 7f2aa9c00640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a84067690 0x7f2a84069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:00.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.977+0000 7f2aa9c00640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a84067690 0x7f2a84069b50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4165c40 tx=0x7f2a98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:00.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:00.981+0000 7f2aa1600640 1 -- 172.21.15.40:0/306883066 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2a90005480 con 0x7f2aa406ea00 2024-03-24T00:32:01.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.201+0000 7f2aac600640 1 -- 172.21.15.40:0/306883066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f2aa4072f50 con 0x7f2aa406ea00 2024-03-24T00:32:01.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.205+0000 7f2aa1600640 1 -- 172.21.15.40:0/306883066 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f2a90055190 con 0x7f2aa406ea00 2024-03-24T00:32:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.209+0000 7f2aac600640 1 -- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a84067690 msgr2=0x7f2a84069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.209+0000 7f2aac600640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a84067690 0x7f2a84069b50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4165c40 tx=0x7f2a98008040 comp rx=0 tx=0).stop 2024-03-24T00:32:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.209+0000 7f2aac600640 1 -- 172.21.15.40:0/306883066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aa406ea00 msgr2=0x7f2aa415b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.209+0000 7f2aac600640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aa406ea00 0x7f2aa415b310 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2a90007fd0 tx=0x7f2a9000c890 comp rx=0 tx=0).stop 2024-03-24T00:32:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.209+0000 7f2aac600640 1 -- 172.21.15.40:0/306883066 shutdown_connections 2024-03-24T00:32:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.209+0000 7f2aac600640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a84067690 0x7f2a84069b50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.209+0000 7f2aac600640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2aa40ccfa0 0x7f2aa4159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.209+0000 7f2aac600640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2aa406ea00 0x7f2aa415b310 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.209+0000 7f2aac600640 1 --2- 172.21.15.40:0/306883066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2aa406e620 0x7f2aa413d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.209+0000 7f2aac600640 1 -- 172.21.15.40:0/306883066 >> 172.21.15.40:0/306883066 conn(0x7f2aa40b17b0 msgr2=0x7f2aa40ae8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.209+0000 7f2aac600640 1 -- 172.21.15.40:0/306883066 shutdown_connections 2024-03-24T00:32:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.209+0000 7f2aac600640 1 -- 172.21.15.40:0/306883066 wait complete. 2024-03-24T00:32:01.232 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-24T00:32:01.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.369+0000 7fe6bb400640 1 Processor -- start 2024-03-24T00:32:01.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.369+0000 7fe6bb400640 1 -- start start 2024-03-24T00:32:01.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.369+0000 7fe6bb400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe6b40b46c0 0x7fe6b40b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:01.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.369+0000 7fe6bb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6b40b2ff0 0x7fe6b40ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:01.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6bb400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b4070f60 con 0x7fe6b406e620 2024-03-24T00:32:01.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6bb400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b40b3400 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6bb400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b40b3620 con 0x7fe6b40b46c0 2024-03-24T00:32:01.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6b9400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6b40b2ff0 0x7fe6b40ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:01.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6b8a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe6b406e620 0x7fe6b406ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42620/0 (socket says 172.21.15.40:42620) 2024-03-24T00:32:01.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6b8a00640 1 -- 172.21.15.40:0/12731038 learned_addr learned my addr 172.21.15.40:0/12731038 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:01.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6abe00640 1 --2- 172.21.15.40:0/12731038 >> v2:172.21.15.53:3300/0 conn(0x7fe6b40b46c0 0x7fe6b40b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:01.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6b9400640 1 -- 172.21.15.40:0/12731038 >> v2:172.21.15.53:3300/0 conn(0x7fe6b40b46c0 msgr2=0x7fe6b40b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:01.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6b9400640 1 --2- 172.21.15.40:0/12731038 >> v2:172.21.15.53:3300/0 conn(0x7fe6b40b46c0 0x7fe6b40b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6b9400640 1 -- 172.21.15.40:0/12731038 >> v1:172.21.15.40:6789/0 conn(0x7fe6b406e620 legacy=0x7fe6b406ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6b9400640 1 -- 172.21.15.40:0/12731038 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b40b3900 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6b9400640 1 --2- 172.21.15.40:0/12731038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b40b2ff0 0x7fe6b40ccfa0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe6b406cea0 tx=0x7fe6ac01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b7115252caa18a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:01.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6b3e00640 1 -- 172.21.15.40:0/12731038 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac0b2070 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6b3e00640 1 -- 172.21.15.40:0/12731038 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ac0a5930 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.373+0000 7fe6b3e00640 1 -- 172.21.15.40:0/12731038 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b40cdcf0 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 -- 172.21.15.40:0/12731038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b40b2ff0 msgr2=0x7fe6b40ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 --2- 172.21.15.40:0/12731038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b40b2ff0 0x7fe6b40ccfa0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe6b406cea0 tx=0x7fe6ac01c870 comp rx=0 tx=0).stop 2024-03-24T00:32:01.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 -- 172.21.15.40:0/12731038 shutdown_connections 2024-03-24T00:32:01.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 --2- 172.21.15.40:0/12731038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b40b2ff0 0x7fe6b40ccfa0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 --2- 172.21.15.40:0/12731038 >> v2:172.21.15.53:3300/0 conn(0x7fe6b40b46c0 0x7fe6b40b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 -- 172.21.15.40:0/12731038 >> 172.21.15.40:0/12731038 conn(0x7fe6b40b17b0 msgr2=0x7fe6b40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:01.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 -- 172.21.15.40:0/12731038 shutdown_connections 2024-03-24T00:32:01.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 -- 172.21.15.40:0/12731038 wait complete. 2024-03-24T00:32:01.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 Processor -- start 2024-03-24T00:32:01.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 -- start start 2024-03-24T00:32:01.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6b406e620 0x7fe6b4163e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:01.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b40b2ff0 0x7fe6b4159900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:01.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe6b40b46c0 0x7fe6b40cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:01.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4144920 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b40b4ad0 con 0x7fe6b406e620 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6bb400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4164590 con 0x7fe6b40b46c0 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b40b2ff0 0x7fe6b4159900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b40b2ff0 0x7fe6b4159900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48148/0 (socket says 172.21.15.40:48148) 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6abe00640 1 -- 172.21.15.40:0/3299040140 learned_addr learned my addr 172.21.15.40:0/3299040140 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6b8a00640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6b406e620 0x7fe6b4163e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6b9400640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe6b40b46c0 0x7fe6b40cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6abe00640 1 -- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe6b40b46c0 msgr2=0x7fe6b40cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6abe00640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe6b40b46c0 0x7fe6b40cdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6abe00640 1 -- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6b406e620 msgr2=0x7fe6b4163e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6abe00640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6b406e620 0x7fe6b4163e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6abe00640 1 -- 172.21.15.40:0/3299040140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b4164870 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6b8a00640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6b406e620 0x7fe6b4163e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.377+0000 7fe6b9400640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe6b40b46c0 0x7fe6b40cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:01.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.381+0000 7fe6abe00640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b40b2ff0 0x7fe6b4159900 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe69c00e930 tx=0x7fe69c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.381+0000 7fe6b1600640 1 -- 172.21.15.40:0/3299040140 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c017070 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.381+0000 7fe6b1600640 1 -- 172.21.15.40:0/3299040140 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69c00ca00 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.381+0000 7fe6b1600640 1 -- 172.21.15.40:0/3299040140 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c00cdb0 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.381+0000 7fe6bb400640 1 -- 172.21.15.40:0/3299040140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b4164b50 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.381+0000 7fe6bb400640 1 -- 172.21.15.40:0/3299040140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b4164e30 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.381+0000 7fe6bb400640 1 -- 172.21.15.40:0/3299040140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b4068170 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.381+0000 7fe6b1600640 1 -- 172.21.15.40:0/3299040140 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe69c01a070 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.381+0000 7fe6b1600640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe690067690 0x7fe690069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.381+0000 7fe6b1600640 1 -- 172.21.15.40:0/3299040140 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe69c0859c0 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.381+0000 7fe6b8a00640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe690067690 0x7fe690069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:01.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.385+0000 7fe6b8a00640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe690067690 0x7fe690069b50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4000f70 tx=0x7fe6a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:01.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.385+0000 7fe6b1600640 1 -- 172.21.15.40:0/3299040140 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe69c0569b0 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.605+0000 7fe6bb400640 1 -- 172.21.15.40:0/3299040140 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fe6b4072f50 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6b1600640 1 -- 172.21.15.40:0/3299040140 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe69c0562b0 con 0x7fe6b40b2ff0 2024-03-24T00:32:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6bb400640 1 -- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe690067690 msgr2=0x7fe690069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6bb400640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe690067690 0x7fe690069b50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4000f70 tx=0x7fe6a4009040 comp rx=0 tx=0).stop 2024-03-24T00:32:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6bb400640 1 -- 172.21.15.40:0/3299040140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b40b2ff0 msgr2=0x7fe6b4159900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6bb400640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b40b2ff0 0x7fe6b4159900 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe69c00e930 tx=0x7fe69c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6bb400640 1 -- 172.21.15.40:0/3299040140 shutdown_connections 2024-03-24T00:32:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6bb400640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe690067690 0x7fe690069b50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6bb400640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe6b40b46c0 0x7fe6b40cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6bb400640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6b40b2ff0 0x7fe6b4159900 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6bb400640 1 --2- 172.21.15.40:0/3299040140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6b406e620 0x7fe6b4163e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6bb400640 1 -- 172.21.15.40:0/3299040140 >> 172.21.15.40:0/3299040140 conn(0x7fe6b40b17b0 msgr2=0x7fe6b413f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:01.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6bb400640 1 -- 172.21.15.40:0/3299040140 shutdown_connections 2024-03-24T00:32:01.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.609+0000 7fe6bb400640 1 -- 172.21.15.40:0/3299040140 wait complete. 2024-03-24T00:32:01.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-24T00:32:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.769+0000 7ff818e00640 1 Processor -- start 2024-03-24T00:32:01.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff818e00640 1 -- start start 2024-03-24T00:32:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff818e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff8140ad8e0 0x7ff8140b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff818e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8140addc0 con 0x7ff8141477b0 2024-03-24T00:32:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff818e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8140ae0a0 con 0x7ff81406ec30 2024-03-24T00:32:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff818e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8140ae9d0 con 0x7ff8140ad8e0 2024-03-24T00:32:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff811e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff8141477b0 0x7ff81406e520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42632/0 (socket says 172.21.15.40:42632) 2024-03-24T00:32:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff811e00640 1 -- 172.21.15.40:0/3136640032 learned_addr learned my addr 172.21.15.40:0/3136640032 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff812800640 1 --2- 172.21.15.40:0/3136640032 >> v2:172.21.15.53:3300/0 conn(0x7ff8140ad8e0 0x7ff8140b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff811400640 1 -- 172.21.15.40:0/3136640032 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808938676 0 0) 0x7ff8140addc0 con 0x7ff8141477b0 2024-03-24T00:32:01.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff811400640 1 -- 172.21.15.40:0/3136640032 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f8003530 con 0x7ff8141477b0 2024-03-24T00:32:01.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff811400640 1 -- 172.21.15.40:0/3136640032 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2706195053 0 0) 0x7ff7f8003530 con 0x7ff8141477b0 2024-03-24T00:32:01.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff812800640 1 -- 172.21.15.40:0/3136640032 >> v1:172.21.15.53:6789/0 conn(0x7ff81406ec30 legacy=0x7ff8140b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff812800640 1 -- 172.21.15.40:0/3136640032 >> v1:172.21.15.40:6789/0 conn(0x7ff8141477b0 legacy=0x7ff81406e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.773+0000 7ff812800640 1 -- 172.21.15.40:0/3136640032 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8140ae380 con 0x7ff8140ad8e0 2024-03-24T00:32:01.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff812800640 1 --2- 172.21.15.40:0/3136640032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8140ad8e0 0x7ff8140b4710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0097c0 tx=0x7ff80c01c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c0235a81a19cf20 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:01.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff811400640 1 -- 172.21.15.40:0/3136640032 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c0b0070 con 0x7ff8140ad8e0 2024-03-24T00:32:01.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff811400640 1 -- 172.21.15.40:0/3136640032 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff80c0ab400 con 0x7ff8140ad8e0 2024-03-24T00:32:01.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff811400640 1 -- 172.21.15.40:0/3136640032 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff804001680 con 0x7ff8140ad8e0 2024-03-24T00:32:01.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff818e00640 1 -- 172.21.15.40:0/3136640032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8140ad8e0 msgr2=0x7ff8140b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff818e00640 1 --2- 172.21.15.40:0/3136640032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8140ad8e0 0x7ff8140b4710 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff80c0097c0 tx=0x7ff80c01c300 comp rx=0 tx=0).stop 2024-03-24T00:32:01.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff818e00640 1 -- 172.21.15.40:0/3136640032 shutdown_connections 2024-03-24T00:32:01.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff818e00640 1 --2- 172.21.15.40:0/3136640032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8140ad8e0 0x7ff8140b4710 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff818e00640 1 -- 172.21.15.40:0/3136640032 >> 172.21.15.40:0/3136640032 conn(0x7ff8140b1d40 msgr2=0x7ff8140b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:01.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff818e00640 1 -- 172.21.15.40:0/3136640032 shutdown_connections 2024-03-24T00:32:01.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff818e00640 1 -- 172.21.15.40:0/3136640032 wait complete. 2024-03-24T00:32:01.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff818e00640 1 Processor -- start 2024-03-24T00:32:01.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.777+0000 7ff818e00640 1 -- start start 2024-03-24T00:32:01.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff818e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff81406ec30 0x7ff8141faf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:01.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff818e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8140ad8e0 0x7ff8141417a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:01.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff818e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff8141477b0 0x7ff814141ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:01.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff818e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81416c360 con 0x7ff8140ad8e0 2024-03-24T00:32:01.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff818e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81416c1e0 con 0x7ff81406ec30 2024-03-24T00:32:01.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff818e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff81416c4e0 con 0x7ff8141477b0 2024-03-24T00:32:01.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff809400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8140ad8e0 0x7ff8141417a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:01.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff811e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff81406ec30 0x7ff8141faf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:01.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff809400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8140ad8e0 0x7ff8141417a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48158/0 (socket says 172.21.15.40:48158) 2024-03-24T00:32:01.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff809400640 1 -- 172.21.15.40:0/2076140041 learned_addr learned my addr 172.21.15.40:0/2076140041 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:01.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff812800640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff8141477b0 0x7ff814141ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:01.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff809400640 1 -- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff8141477b0 msgr2=0x7ff814141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff809400640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff8141477b0 0x7ff814141ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff809400640 1 -- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff81406ec30 msgr2=0x7ff8141faf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:01.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff809400640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff81406ec30 0x7ff8141faf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:01.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff809400640 1 -- 172.21.15.40:0/2076140041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8141fb8f0 con 0x7ff8140ad8e0 2024-03-24T00:32:01.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff809400640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8140ad8e0 0x7ff8141417a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00c7c0 tx=0x7ff7fc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:01.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff80aa00640 1 -- 172.21.15.40:0/2076140041 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc003b80 con 0x7ff8140ad8e0 2024-03-24T00:32:01.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff80aa00640 1 -- 172.21.15.40:0/2076140041 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc003d20 con 0x7ff8140ad8e0 2024-03-24T00:32:01.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff818e00640 1 -- 172.21.15.40:0/2076140041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8141fbbd0 con 0x7ff8140ad8e0 2024-03-24T00:32:01.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff80aa00640 1 -- 172.21.15.40:0/2076140041 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc00aa90 con 0x7ff8140ad8e0 2024-03-24T00:32:01.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff818e00640 1 -- 172.21.15.40:0/2076140041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8141fb4c0 con 0x7ff8140ad8e0 2024-03-24T00:32:01.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.781+0000 7ff818e00640 1 -- 172.21.15.40:0/2076140041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff814068170 con 0x7ff8140ad8e0 2024-03-24T00:32:01.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.785+0000 7ff80aa00640 1 -- 172.21.15.40:0/2076140041 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff7fc00ac30 con 0x7ff8140ad8e0 2024-03-24T00:32:01.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.785+0000 7ff80aa00640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff7f0067660 0x7ff7f0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:01.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.785+0000 7ff80aa00640 1 -- 172.21.15.40:0/2076140041 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff7fc085350 con 0x7ff8140ad8e0 2024-03-24T00:32:01.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.785+0000 7ff811e00640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff7f0067660 0x7ff7f0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:01.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.785+0000 7ff811e00640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff7f0067660 0x7ff7f0069b20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff804005890 tx=0x7ff804005800 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:01.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:01.789+0000 7ff80aa00640 1 -- 172.21.15.40:0/2076140041 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff7fc055d40 con 0x7ff8140ad8e0 2024-03-24T00:32:02.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.013+0000 7ff818e00640 1 -- 172.21.15.40:0/2076140041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7ff81413edf0 con 0x7ff8140ad8e0 2024-03-24T00:32:02.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.013+0000 7ff80aa00640 1 -- 172.21.15.40:0/2076140041 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff7fc059de0 con 0x7ff8140ad8e0 2024-03-24T00:32:02.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.017+0000 7ff818e00640 1 -- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff7f0067660 msgr2=0x7ff7f0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.017+0000 7ff818e00640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff7f0067660 0x7ff7f0069b20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff804005890 tx=0x7ff804005800 comp rx=0 tx=0).stop 2024-03-24T00:32:02.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.017+0000 7ff818e00640 1 -- 172.21.15.40:0/2076140041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8140ad8e0 msgr2=0x7ff8141417a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.017+0000 7ff818e00640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8140ad8e0 0x7ff8141417a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc00c7c0 tx=0x7ff7fc00cc80 comp rx=0 tx=0).stop 2024-03-24T00:32:02.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.017+0000 7ff818e00640 1 -- 172.21.15.40:0/2076140041 shutdown_connections 2024-03-24T00:32:02.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.017+0000 7ff818e00640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff7f0067660 0x7ff7f0069b20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.017+0000 7ff818e00640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff8141477b0 0x7ff814141ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.017+0000 7ff818e00640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff8140ad8e0 0x7ff8141417a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.017+0000 7ff818e00640 1 --2- 172.21.15.40:0/2076140041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff81406ec30 0x7ff8141faf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.017+0000 7ff818e00640 1 -- 172.21.15.40:0/2076140041 >> 172.21.15.40:0/2076140041 conn(0x7ff8140b1d40 msgr2=0x7ff814160d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:02.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.017+0000 7ff818e00640 1 -- 172.21.15.40:0/2076140041 shutdown_connections 2024-03-24T00:32:02.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.017+0000 7ff818e00640 1 -- 172.21.15.40:0/2076140041 wait complete. 2024-03-24T00:32:02.042 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-24T00:32:02.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.177+0000 7ff1b0e00640 1 Processor -- start 2024-03-24T00:32:02.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.177+0000 7ff1b0e00640 1 -- start start 2024-03-24T00:32:02.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.181+0000 7ff1b0e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff1ac13ec80 0x7ff1ac1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.181+0000 7ff1b0e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff1ac1419f0 0x7ff1ac0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.181+0000 7ff1b0e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ac070e00 con 0x7ff1ac0ce630 2024-03-24T00:32:02.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.181+0000 7ff1b0e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ac141e50 con 0x7ff1ac13ec80 2024-03-24T00:32:02.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.181+0000 7ff1b0e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ac0cee20 con 0x7ff1ac1419f0 2024-03-24T00:32:02.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.181+0000 7ff1a9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff1ac13ec80 0x7ff1ac1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.181+0000 7ff1a9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff1ac13ec80 0x7ff1ac1414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48168/0 (socket says 172.21.15.40:48168) 2024-03-24T00:32:02.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.181+0000 7ff1a9e00640 1 -- 172.21.15.40:0/2898777389 learned_addr learned my addr 172.21.15.40:0/2898777389 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.181+0000 7ff1a9e00640 1 -- 172.21.15.40:0/2898777389 >> v2:172.21.15.53:3300/0 conn(0x7ff1ac1419f0 msgr2=0x7ff1ac0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.181+0000 7ff1a9e00640 1 --2- 172.21.15.40:0/2898777389 >> v2:172.21.15.53:3300/0 conn(0x7ff1ac1419f0 0x7ff1ac0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.181+0000 7ff1a9e00640 1 -- 172.21.15.40:0/2898777389 >> v1:172.21.15.53:6789/0 conn(0x7ff1ac0ce630 legacy=0x7ff1ac15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:02.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.181+0000 7ff1a9e00640 1 -- 172.21.15.40:0/2898777389 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac0cf100 con 0x7ff1ac13ec80 2024-03-24T00:32:02.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1a9e00640 1 --2- 172.21.15.40:0/2898777389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ac13ec80 0x7ff1ac1414b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac04fa60 tx=0x7ff19c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94a9ceb9de1a002 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:02.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1a9400640 1 -- 172.21.15.40:0/2898777389 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0aa660 con 0x7ff1ac13ec80 2024-03-24T00:32:02.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1a9400640 1 -- 172.21.15.40:0/2898777389 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19c0af070 con 0x7ff1ac13ec80 2024-03-24T00:32:02.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1a9400640 1 -- 172.21.15.40:0/2898777389 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0aaa20 con 0x7ff1ac13ec80 2024-03-24T00:32:02.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1b0e00640 1 -- 172.21.15.40:0/2898777389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ac13ec80 msgr2=0x7ff1ac1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1b0e00640 1 --2- 172.21.15.40:0/2898777389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ac13ec80 0x7ff1ac1414b0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac04fa60 tx=0x7ff19c01c4e0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1b0e00640 1 -- 172.21.15.40:0/2898777389 shutdown_connections 2024-03-24T00:32:02.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1b0e00640 1 --2- 172.21.15.40:0/2898777389 >> v2:172.21.15.53:3300/0 conn(0x7ff1ac1419f0 0x7ff1ac0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1b0e00640 1 --2- 172.21.15.40:0/2898777389 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ac13ec80 0x7ff1ac1414b0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1b0e00640 1 -- 172.21.15.40:0/2898777389 >> 172.21.15.40:0/2898777389 conn(0x7ff1ac0b17b0 msgr2=0x7ff1ac0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:02.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1b0e00640 1 -- 172.21.15.40:0/2898777389 shutdown_connections 2024-03-24T00:32:02.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1b0e00640 1 -- 172.21.15.40:0/2898777389 wait complete. 2024-03-24T00:32:02.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1b0e00640 1 Processor -- start 2024-03-24T00:32:02.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.185+0000 7ff1b0e00640 1 -- start start 2024-03-24T00:32:02.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1b0e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ac0ce630 0x7ff1ac13d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1b0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1ac13ec80 0x7ff1ac14fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1b0e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1ac1419f0 0x7ff1ac150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1b0e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac168ec0 con 0x7ff1ac0ce630 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1b0e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac168d40 con 0x7ff1ac13ec80 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1b0e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac169040 con 0x7ff1ac1419f0 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ac0ce630 0x7ff1ac13d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ac0ce630 0x7ff1ac13d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48180/0 (socket says 172.21.15.40:48180) 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a9e00640 1 -- 172.21.15.40:0/1402182987 learned_addr learned my addr 172.21.15.40:0/1402182987 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a3e00640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1ac13ec80 0x7ff1ac14fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1aa800640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1ac1419f0 0x7ff1ac150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a9e00640 1 -- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1ac1419f0 msgr2=0x7ff1ac150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a9e00640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1ac1419f0 0x7ff1ac150090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a9e00640 1 -- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1ac13ec80 msgr2=0x7ff1ac14fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a9e00640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1ac13ec80 0x7ff1ac14fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a9e00640 1 -- 172.21.15.40:0/1402182987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac156760 con 0x7ff1ac0ce630 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1aa800640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1ac1419f0 0x7ff1ac150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:02.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a9e00640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ac0ce630 0x7ff1ac13d840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff19c0ac040 tx=0x7ff19c01ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:02.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a2000640 1 -- 172.21.15.40:0/1402182987 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0b3800 con 0x7ff1ac0ce630 2024-03-24T00:32:02.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a2000640 1 -- 172.21.15.40:0/1402182987 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19c0aae50 con 0x7ff1ac0ce630 2024-03-24T00:32:02.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1b0e00640 1 -- 172.21.15.40:0/1402182987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ac14d8a0 con 0x7ff1ac0ce630 2024-03-24T00:32:02.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a2000640 1 -- 172.21.15.40:0/1402182987 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c0b3c00 con 0x7ff1ac0ce630 2024-03-24T00:32:02.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1b0e00640 1 -- 172.21.15.40:0/1402182987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ac14de60 con 0x7ff1ac0ce630 2024-03-24T00:32:02.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1b0e00640 1 -- 172.21.15.40:0/1402182987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ac068170 con 0x7ff1ac0ce630 2024-03-24T00:32:02.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.189+0000 7ff1a2000640 1 -- 172.21.15.40:0/1402182987 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff19c0c6390 con 0x7ff1ac0ce630 2024-03-24T00:32:02.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.193+0000 7ff1a2000640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff188067690 0x7ff188069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.193+0000 7ff1a2000640 1 -- 172.21.15.40:0/1402182987 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff19c11fe50 con 0x7ff1ac0ce630 2024-03-24T00:32:02.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.193+0000 7ff1a3e00640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff188067690 0x7ff188069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.193+0000 7ff1a3e00640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff188067690 0x7ff188069b50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff194006fd0 tx=0x7ff194008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:02.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.197+0000 7ff1a2000640 1 -- 172.21.15.40:0/1402182987 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff19c0c3060 con 0x7ff1ac0ce630 2024-03-24T00:32:02.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.421+0000 7ff1b0e00640 1 -- 172.21.15.40:0/1402182987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7ff1ac13e410 con 0x7ff1ac0ce630 2024-03-24T00:32:02.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.421+0000 7ff1a2000640 1 -- 172.21.15.40:0/1402182987 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff19c0160d0 con 0x7ff1ac0ce630 2024-03-24T00:32:02.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.425+0000 7ff1b0e00640 1 -- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff188067690 msgr2=0x7ff188069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.425+0000 7ff1b0e00640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff188067690 0x7ff188069b50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff194006fd0 tx=0x7ff194008040 comp rx=0 tx=0).stop 2024-03-24T00:32:02.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.425+0000 7ff1b0e00640 1 -- 172.21.15.40:0/1402182987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ac0ce630 msgr2=0x7ff1ac13d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.425+0000 7ff1b0e00640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ac0ce630 0x7ff1ac13d840 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff19c0ac040 tx=0x7ff19c01ad20 comp rx=0 tx=0).stop 2024-03-24T00:32:02.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.425+0000 7ff1b0e00640 1 -- 172.21.15.40:0/1402182987 shutdown_connections 2024-03-24T00:32:02.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.425+0000 7ff1b0e00640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff188067690 0x7ff188069b50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.425+0000 7ff1b0e00640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1ac1419f0 0x7ff1ac150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.425+0000 7ff1b0e00640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1ac13ec80 0x7ff1ac14fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.425+0000 7ff1b0e00640 1 --2- 172.21.15.40:0/1402182987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1ac0ce630 0x7ff1ac13d840 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.425+0000 7ff1b0e00640 1 -- 172.21.15.40:0/1402182987 >> 172.21.15.40:0/1402182987 conn(0x7ff1ac0b17b0 msgr2=0x7ff1ac15e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:02.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.425+0000 7ff1b0e00640 1 -- 172.21.15.40:0/1402182987 shutdown_connections 2024-03-24T00:32:02.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.425+0000 7ff1b0e00640 1 -- 172.21.15.40:0/1402182987 wait complete. 2024-03-24T00:32:02.449 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:32:02.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5ad6400640 1 Processor -- start 2024-03-24T00:32:02.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5ad6400640 1 -- start start 2024-03-24T00:32:02.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5ad6400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5ad015ab80 0x7f5ad00d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5ad6400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00b4720 0x7f5ad00b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5ad6400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad00d1320 con 0x7f5ad00d0e30 2024-03-24T00:32:02.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5ad6400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad006e3f0 con 0x7f5ad015ab80 2024-03-24T00:32:02.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5ad6400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad00d1600 con 0x7f5ad00b4720 2024-03-24T00:32:02.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5ac7e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5ad00d0e30 0x7f5ad006e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42642/0 (socket says 172.21.15.40:42642) 2024-03-24T00:32:02.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5ac7e00640 1 -- 172.21.15.40:0/2668804253 learned_addr learned my addr 172.21.15.40:0/2668804253 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:02.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acfe00640 1 --2- 172.21.15.40:0/2668804253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00b4720 0x7f5ad00b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acf400640 1 --2- 172.21.15.40:0/2668804253 >> v2:172.21.15.53:3300/0 conn(0x7f5ad015ab80 0x7f5ad00d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980278899 0 0) 0x7f5ad00d1320 con 0x7f5ad00d0e30 2024-03-24T00:32:02.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab4003530 con 0x7f5ad00d0e30 2024-03-24T00:32:02.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 90352397 0 0) 0x7f5ab4003530 con 0x7f5ad00d0e30 2024-03-24T00:32:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00b4720 msgr2=0x7f5ad00b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 --2- 172.21.15.40:0/2668804253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00b4720 0x7f5ad00b4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 >> v2:172.21.15.53:3300/0 conn(0x7f5ad015ab80 msgr2=0x7f5ad00d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 --2- 172.21.15.40:0/2668804253 >> v2:172.21.15.53:3300/0 conn(0x7f5ad015ab80 0x7f5ad00d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad006e080 con 0x7f5ad00d0e30 2024-03-24T00:32:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f5ab80016a0 con 0x7f5ad00d0e30 2024-03-24T00:32:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 >> v1:172.21.15.40:6789/0 conn(0x7f5ad00d0e30 legacy=0x7f5ad006e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 --2- 172.21.15.40:0/2668804253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ab40039c0 0x7f5ab4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 --2- 172.21.15.40:0/2668804253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab4004810 0x7f5ab4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 --2- 172.21.15.40:0/2668804253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00b4720 0x7f5ab40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad006e3f0 con 0x7f5ab4004810 2024-03-24T00:32:02.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad00d1600 con 0x7f5ab40039c0 2024-03-24T00:32:02.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.585+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab400b9d0 con 0x7f5ad00b4720 2024-03-24T00:32:02.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.589+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/2668804253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab4004810 0x7f5ab4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.589+0000 7f5acf400640 1 --2- 172.21.15.40:0/2668804253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ab40039c0 0x7f5ab4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.589+0000 7f5acfe00640 1 --2- 172.21.15.40:0/2668804253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00b4720 0x7f5ab40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.589+0000 7f5ac7e00640 1 -- 172.21.15.40:0/2668804253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00b4720 msgr2=0x7f5ab40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:02.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.589+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/2668804253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00b4720 0x7f5ab40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.589+0000 7f5ac7e00640 1 -- 172.21.15.40:0/2668804253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ab40039c0 msgr2=0x7f5ab4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:02.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.589+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/2668804253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ab40039c0 0x7f5ab4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.589+0000 7f5ac7e00640 1 -- 172.21.15.40:0/2668804253 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab4003530 con 0x7f5ab4004810 2024-03-24T00:32:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.589+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/2668804253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab4004810 0x7f5ab4004c90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800c210 tx=0x7f5ab801f3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36da24edfbf5325c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.589+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8007d70 con 0x7f5ab4004810 2024-03-24T00:32:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.589+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab8005ae0 con 0x7f5ab4004810 2024-03-24T00:32:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.589+0000 7f5acea00640 1 -- 172.21.15.40:0/2668804253 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8007070 con 0x7f5ab4004810 2024-03-24T00:32:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 -- 172.21.15.40:0/2668804253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab4004810 msgr2=0x7f5ab4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 --2- 172.21.15.40:0/2668804253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ab4004810 0x7f5ab4004c90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800c210 tx=0x7f5ab801f3f0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5acfe00640 1 -- 172.21.15.40:0/2668804253 reap_dead start 2024-03-24T00:32:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 -- 172.21.15.40:0/2668804253 shutdown_connections 2024-03-24T00:32:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 -- 172.21.15.40:0/2668804253 >> 172.21.15.40:0/2668804253 conn(0x7f5ad006b150 msgr2=0x7f5ad006b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 -- 172.21.15.40:0/2668804253 shutdown_connections 2024-03-24T00:32:02.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 -- 172.21.15.40:0/2668804253 wait complete. 2024-03-24T00:32:02.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 Processor -- start 2024-03-24T00:32:02.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 -- start start 2024-03-24T00:32:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ab4004810 0x7f5ad015dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad014d6f0 0x7f5ad014c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad014ef10 0x7f5ad014fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad00b3fa0 con 0x7f5ad014d6f0 2024-03-24T00:32:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad00b3e20 con 0x7f5ad014ef10 2024-03-24T00:32:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ad6400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad006ef90 con 0x7f5ab4004810 2024-03-24T00:32:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ac7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad014d6f0 0x7f5ad014c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ac7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad014d6f0 0x7f5ad014c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48188/0 (socket says 172.21.15.40:48188) 2024-03-24T00:32:02.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ac7e00640 1 -- 172.21.15.40:0/278013534 learned_addr learned my addr 172.21.15.40:0/278013534 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5acf400640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ab4004810 0x7f5ad015dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5acfe00640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad014ef10 0x7f5ad014fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ac7e00640 1 -- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ab4004810 msgr2=0x7f5ad015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ab4004810 0x7f5ad015dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ac7e00640 1 -- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad014ef10 msgr2=0x7f5ad014fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad014ef10 0x7f5ad014fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ac7e00640 1 -- 172.21.15.40:0/278013534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad014de70 con 0x7f5ad014d6f0 2024-03-24T00:32:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5acf400640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ab4004810 0x7f5ad015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.593+0000 7f5ac7e00640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad014d6f0 0x7f5ad014c530 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5ab801f430 tx=0x7f5ab4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.597+0000 7f5ac7400640 1 -- 172.21.15.40:0/278013534 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8005d50 con 0x7f5ad014d6f0 2024-03-24T00:32:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.597+0000 7f5ac7400640 1 -- 172.21.15.40:0/278013534 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab80047f0 con 0x7f5ad014d6f0 2024-03-24T00:32:02.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.597+0000 7f5ac7400640 1 -- 172.21.15.40:0/278013534 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8004bc0 con 0x7f5ad014d6f0 2024-03-24T00:32:02.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.597+0000 7f5ad6400640 1 -- 172.21.15.40:0/278013534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad014e150 con 0x7f5ad014d6f0 2024-03-24T00:32:02.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.597+0000 7f5ad6400640 1 -- 172.21.15.40:0/278013534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad014e710 con 0x7f5ad014d6f0 2024-03-24T00:32:02.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.597+0000 7f5ad6400640 1 -- 172.21.15.40:0/278013534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad0068170 con 0x7f5ad014d6f0 2024-03-24T00:32:02.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.597+0000 7f5ac7400640 1 -- 172.21.15.40:0/278013534 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5ab80ba030 con 0x7f5ad014d6f0 2024-03-24T00:32:02.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.597+0000 7f5ac7400640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5aac067660 0x7f5aac069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.597+0000 7f5ac7400640 1 -- 172.21.15.40:0/278013534 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5ab8122040 con 0x7f5ad014d6f0 2024-03-24T00:32:02.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.597+0000 7f5acf400640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5aac067660 0x7f5aac069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.601+0000 7f5acf400640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5aac067660 0x7f5aac069b20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009be0 tx=0x7f5ac0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.605+0000 7f5ac7400640 1 -- 172.21.15.40:0/278013534 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5ab80c1030 con 0x7f5ad014d6f0 2024-03-24T00:32:02.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.773+0000 7f5ad6400640 1 -- 172.21.15.40:0/278013534 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ad014db70 con 0x7f5ad014d6f0 2024-03-24T00:32:02.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.773+0000 7f5ac7400640 1 -- 172.21.15.40:0/278013534 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+12831 (secure 0 0 0) 0x7f5ab8019100 con 0x7f5ad014d6f0 2024-03-24T00:32:02.779 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:02.780 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":29,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:31:59.599319+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:32:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.777+0000 7f5ad6400640 1 -- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5aac067660 msgr2=0x7f5aac069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.777+0000 7f5ad6400640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5aac067660 0x7f5aac069b20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009be0 tx=0x7f5ac0008040 comp rx=0 tx=0).stop 2024-03-24T00:32:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.777+0000 7f5ad6400640 1 -- 172.21.15.40:0/278013534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad014d6f0 msgr2=0x7f5ad014c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.777+0000 7f5ad6400640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad014d6f0 0x7f5ad014c530 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5ab801f430 tx=0x7f5ab4005840 comp rx=0 tx=0).stop 2024-03-24T00:32:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.777+0000 7f5ad6400640 1 -- 172.21.15.40:0/278013534 shutdown_connections 2024-03-24T00:32:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.777+0000 7f5ad6400640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5aac067660 0x7f5aac069b20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.777+0000 7f5ad6400640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad014ef10 0x7f5ad014fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.777+0000 7f5ad6400640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad014d6f0 0x7f5ad014c530 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.777+0000 7f5ad6400640 1 --2- 172.21.15.40:0/278013534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ab4004810 0x7f5ad015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.777+0000 7f5ad6400640 1 -- 172.21.15.40:0/278013534 >> 172.21.15.40:0/278013534 conn(0x7f5ad006b150 msgr2=0x7f5ad00b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.777+0000 7f5ad6400640 1 -- 172.21.15.40:0/278013534 shutdown_connections 2024-03-24T00:32:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.777+0000 7f5ad6400640 1 -- 172.21.15.40:0/278013534 wait complete. 2024-03-24T00:32:02.800 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:32:02.801 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:32:02.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56f400640 1 Processor -- start 2024-03-24T00:32:02.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56f400640 1 -- start start 2024-03-24T00:32:02.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb568145880 0x7fb56806dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56f400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb568066e80 0x7fb568067290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56f400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5680677d0 con 0x7fb56806f6f0 2024-03-24T00:32:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56f400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb568160080 con 0x7fb568145880 2024-03-24T00:32:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56f400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5681602c0 con 0x7fb568066e80 2024-03-24T00:32:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56ca00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb568145880 0x7fb56806dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb567e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb56806f6f0 0x7fb56815f970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42652/0 (socket says 172.21.15.40:42652) 2024-03-24T00:32:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb567e00640 1 -- 172.21.15.40:0/3699863502 learned_addr learned my addr 172.21.15.40:0/3699863502 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56d400640 1 --2- 172.21.15.40:0/3699863502 >> v2:172.21.15.53:3300/0 conn(0x7fb568066e80 0x7fb568067290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56ca00640 1 -- 172.21.15.40:0/3699863502 >> v2:172.21.15.53:3300/0 conn(0x7fb568066e80 msgr2=0x7fb568067290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56ca00640 1 --2- 172.21.15.40:0/3699863502 >> v2:172.21.15.53:3300/0 conn(0x7fb568066e80 0x7fb568067290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56ca00640 1 -- 172.21.15.40:0/3699863502 >> v1:172.21.15.40:6789/0 conn(0x7fb56806f6f0 legacy=0x7fb56815f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb56ca00640 1 -- 172.21.15.40:0/3699863502 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5681605a0 con 0x7fb568145880 2024-03-24T00:32:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.941+0000 7fb567400640 1 -- 172.21.15.40:0/3699863502 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594511096 0 0) 0x7fb554001120 con 0x7fb56806f6f0 2024-03-24T00:32:02.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb56ca00640 1 --2- 172.21.15.40:0/3699863502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568145880 0x7fb56806dcb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb568073450 tx=0x7fb55c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a70086340ad61f86 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:02.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb567400640 1 -- 172.21.15.40:0/3699863502 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c0ab650 con 0x7fb568145880 2024-03-24T00:32:02.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb567400640 1 -- 172.21.15.40:0/3699863502 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c0b0070 con 0x7fb568145880 2024-03-24T00:32:02.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb567400640 1 -- 172.21.15.40:0/3699863502 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c0aba10 con 0x7fb568145880 2024-03-24T00:32:02.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb56f400640 1 -- 172.21.15.40:0/3699863502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568145880 msgr2=0x7fb56806dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb56f400640 1 --2- 172.21.15.40:0/3699863502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568145880 0x7fb56806dcb0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb568073450 tx=0x7fb55c01c450 comp rx=0 tx=0).stop 2024-03-24T00:32:02.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb56f400640 1 -- 172.21.15.40:0/3699863502 shutdown_connections 2024-03-24T00:32:02.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb56f400640 1 --2- 172.21.15.40:0/3699863502 >> v2:172.21.15.53:3300/0 conn(0x7fb568066e80 0x7fb568067290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb56f400640 1 --2- 172.21.15.40:0/3699863502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568145880 0x7fb56806dcb0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb56f400640 1 -- 172.21.15.40:0/3699863502 >> 172.21.15.40:0/3699863502 conn(0x7fb5680d0bb0 msgr2=0x7fb5680d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:02.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb56f400640 1 -- 172.21.15.40:0/3699863502 shutdown_connections 2024-03-24T00:32:02.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb56f400640 1 -- 172.21.15.40:0/3699863502 wait complete. 2024-03-24T00:32:02.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.945+0000 7fb56f400640 1 Processor -- start 2024-03-24T00:32:02.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56f400640 1 -- start start 2024-03-24T00:32:02.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568066e80 0x7fb56813bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb56806f6f0 0x7fb568157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb568145880 0x7fb568157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56f400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680d03e0 con 0x7fb568066e80 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56f400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680d0260 con 0x7fb56806f6f0 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56f400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680d0560 con 0x7fb568145880 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568066e80 0x7fb56813bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568066e80 0x7fb56813bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48216/0 (socket says 172.21.15.40:48216) 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56ca00640 1 -- 172.21.15.40:0/1497052420 learned_addr learned my addr 172.21.15.40:0/1497052420 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb567e00640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb56806f6f0 0x7fb568157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56ca00640 1 -- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb568145880 msgr2=0x7fb568157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56d400640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb568145880 0x7fb568157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56ca00640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb568145880 0x7fb568157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56ca00640 1 -- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb56806f6f0 msgr2=0x7fb568157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56ca00640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb56806f6f0 0x7fb568157050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56ca00640 1 -- 172.21.15.40:0/1497052420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb568150c10 con 0x7fb568066e80 2024-03-24T00:32:02.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb567e00640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb56806f6f0 0x7fb568157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:02.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56d400640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb568145880 0x7fb568157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:02.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56ca00640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568066e80 0x7fb56813bbc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb55c019060 tx=0x7fb55c01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:02.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb564c00640 1 -- 172.21.15.40:0/1497052420 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c0bf610 con 0x7fb568066e80 2024-03-24T00:32:02.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb564c00640 1 -- 172.21.15.40:0/1497052420 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c0bf7d0 con 0x7fb568066e80 2024-03-24T00:32:02.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56f400640 1 -- 172.21.15.40:0/1497052420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb568150ef0 con 0x7fb568066e80 2024-03-24T00:32:02.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb564c00640 1 -- 172.21.15.40:0/1497052420 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c0bfbc0 con 0x7fb568066e80 2024-03-24T00:32:02.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56f400640 1 -- 172.21.15.40:0/1497052420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5681514b0 con 0x7fb568066e80 2024-03-24T00:32:02.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.949+0000 7fb56f400640 1 -- 172.21.15.40:0/1497052420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb568068170 con 0x7fb568066e80 2024-03-24T00:32:02.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.953+0000 7fb564c00640 1 -- 172.21.15.40:0/1497052420 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb55c0bd020 con 0x7fb568066e80 2024-03-24T00:32:02.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.953+0000 7fb564c00640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb548067660 0x7fb548069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:02.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.953+0000 7fb564c00640 1 -- 172.21.15.40:0/1497052420 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb55c11fde0 con 0x7fb568066e80 2024-03-24T00:32:02.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.953+0000 7fb567e00640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb548067660 0x7fb548069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:02.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.953+0000 7fb567e00640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb548067660 0x7fb548069b20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb5680d1a90 tx=0x7fb554009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:02.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:02.957+0000 7fb564c00640 1 -- 172.21.15.40:0/1497052420 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb55c016070 con 0x7fb568066e80 2024-03-24T00:32:03.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb56f400640 1 -- 172.21.15.40:0/1497052420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb568072f00 con 0x7fb568066e80 2024-03-24T00:32:03.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb564c00640 1 -- 172.21.15.40:0/1497052420 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb55c0f0d10 con 0x7fb568066e80 2024-03-24T00:32:03.126 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:32:03.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb56f400640 1 -- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb548067660 msgr2=0x7fb548069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:03.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb56f400640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb548067660 0x7fb548069b20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb5680d1a90 tx=0x7fb554009040 comp rx=0 tx=0).stop 2024-03-24T00:32:03.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb56f400640 1 -- 172.21.15.40:0/1497052420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568066e80 msgr2=0x7fb56813bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:03.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb56f400640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568066e80 0x7fb56813bbc0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb55c019060 tx=0x7fb55c01ac90 comp rx=0 tx=0).stop 2024-03-24T00:32:03.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb56f400640 1 -- 172.21.15.40:0/1497052420 shutdown_connections 2024-03-24T00:32:03.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb56f400640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb548067660 0x7fb548069b20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb56f400640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb568145880 0x7fb568157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb56f400640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb56806f6f0 0x7fb568157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb56f400640 1 --2- 172.21.15.40:0/1497052420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb568066e80 0x7fb56813bbc0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb56f400640 1 -- 172.21.15.40:0/1497052420 >> 172.21.15.40:0/1497052420 conn(0x7fb5680d0bb0 msgr2=0x7fb56806e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:03.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.121+0000 7fb56f400640 1 -- 172.21.15.40:0/1497052420 shutdown_connections 2024-03-24T00:32:03.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.125+0000 7fb56f400640 1 -- 172.21.15.40:0/1497052420 wait complete. 2024-03-24T00:32:03.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:32:03.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.281+0000 7f8f84e00640 1 Processor -- start 2024-03-24T00:32:03.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.281+0000 7f8f84e00640 1 -- start start 2024-03-24T00:32:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f84e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f8006e620 0x7f8f800b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f84e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f800ccfa0 0x7f8f800cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f84e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f80070f60 con 0x7f8f8006ea00 2024-03-24T00:32:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f84e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f800b4c00 con 0x7f8f800ccfa0 2024-03-24T00:32:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f84e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800b3620 con 0x7f8f8006e620 2024-03-24T00:32:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f7e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f800ccfa0 0x7f8f800cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f7e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f800ccfa0 0x7f8f800cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48232/0 (socket says 172.21.15.40:48232) 2024-03-24T00:32:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f7e800640 1 -- 172.21.15.40:0/1481310640 learned_addr learned my addr 172.21.15.40:0/1481310640 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f7de00640 1 --2- 172.21.15.40:0/1481310640 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f8006e620 0x7f8f800b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f7e800640 1 -- 172.21.15.40:0/1481310640 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f8006e620 msgr2=0x7f8f800b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:03.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f7e800640 1 --2- 172.21.15.40:0/1481310640 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f8006e620 0x7f8f800b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f7e800640 1 -- 172.21.15.40:0/1481310640 >> v1:172.21.15.53:6789/0 conn(0x7f8f8006ea00 legacy=0x7f8f800b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:03.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f7e800640 1 -- 172.21.15.40:0/1481310640 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f800b3900 con 0x7f8f800ccfa0 2024-03-24T00:32:03.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.285+0000 7f8f7de00640 1 --2- 172.21.15.40:0/1481310640 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f8006e620 0x7f8f800b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:03.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f7e800640 1 --2- 172.21.15.40:0/1481310640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f800ccfa0 0x7f8f800cd3f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009e50 tx=0x7f8f7801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1acf2bebc396030 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f7ca00640 1 -- 172.21.15.40:0/1481310640 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f780a5740 con 0x7f8f800ccfa0 2024-03-24T00:32:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f7ca00640 1 -- 172.21.15.40:0/1481310640 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f780a58e0 con 0x7f8f800ccfa0 2024-03-24T00:32:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f7ca00640 1 -- 172.21.15.40:0/1481310640 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f780a5cb0 con 0x7f8f800ccfa0 2024-03-24T00:32:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 -- 172.21.15.40:0/1481310640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f800ccfa0 msgr2=0x7f8f800cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 --2- 172.21.15.40:0/1481310640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f800ccfa0 0x7f8f800cd3f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009e50 tx=0x7f8f7801c970 comp rx=0 tx=0).stop 2024-03-24T00:32:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 -- 172.21.15.40:0/1481310640 shutdown_connections 2024-03-24T00:32:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 --2- 172.21.15.40:0/1481310640 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f800ccfa0 0x7f8f800cd3f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 --2- 172.21.15.40:0/1481310640 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f8006e620 0x7f8f800b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 -- 172.21.15.40:0/1481310640 >> 172.21.15.40:0/1481310640 conn(0x7f8f800b17b0 msgr2=0x7f8f800b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:03.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 -- 172.21.15.40:0/1481310640 shutdown_connections 2024-03-24T00:32:03.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 -- 172.21.15.40:0/1481310640 wait complete. 2024-03-24T00:32:03.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 Processor -- start 2024-03-24T00:32:03.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 -- start start 2024-03-24T00:32:03.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8006e620 0x7f8f8015b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:03.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f8006ea00 0x7f8f8013ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:03.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f800ccfa0 0x7f8f8013d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:03.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f80144920 con 0x7f8f8006e620 2024-03-24T00:32:03.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8006ede0 con 0x7f8f800ccfa0 2024-03-24T00:32:03.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.289+0000 7f8f84e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f80145ae0 con 0x7f8f8006ea00 2024-03-24T00:32:03.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8006e620 0x7f8f8015b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:03.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8006e620 0x7f8f8015b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48246/0 (socket says 172.21.15.40:48246) 2024-03-24T00:32:03.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7de00640 1 -- 172.21.15.40:0/159023407 learned_addr learned my addr 172.21.15.40:0/159023407 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7d400640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f8006ea00 0x7f8f8013ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7e800640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f800ccfa0 0x7f8f8013d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7de00640 1 -- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f8006ea00 msgr2=0x7f8f8013ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7de00640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f8006ea00 0x7f8f8013ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7de00640 1 -- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f800ccfa0 msgr2=0x7f8f8013d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7de00640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f800ccfa0 0x7f8f8013d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7de00640 1 -- 172.21.15.40:0/159023407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f80144c30 con 0x7f8f8006e620 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7d400640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f8006ea00 0x7f8f8013ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7e800640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f800ccfa0 0x7f8f8013d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f7de00640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8006e620 0x7f8f8015b480 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000d800 tx=0x7f8f7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f76000640 1 -- 172.21.15.40:0/159023407 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70018020 con 0x7f8f8006e620 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f76000640 1 -- 172.21.15.40:0/159023407 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f700096c0 con 0x7f8f8006e620 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f84e00640 1 -- 172.21.15.40:0/159023407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f80144ed0 con 0x7f8f8006e620 2024-03-24T00:32:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f76000640 1 -- 172.21.15.40:0/159023407 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70009ab0 con 0x7f8f8006e620 2024-03-24T00:32:03.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f84e00640 1 -- 172.21.15.40:0/159023407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f80145490 con 0x7f8f8006e620 2024-03-24T00:32:03.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f84e00640 1 -- 172.21.15.40:0/159023407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f80068170 con 0x7f8f8006e620 2024-03-24T00:32:03.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.293+0000 7f8f76000640 1 -- 172.21.15.40:0/159023407 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8f7001b020 con 0x7f8f8006e620 2024-03-24T00:32:03.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.297+0000 7f8f76000640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8f60067660 0x7f8f60069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:03.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.297+0000 7f8f76000640 1 -- 172.21.15.40:0/159023407 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8f70085b00 con 0x7f8f8006e620 2024-03-24T00:32:03.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.297+0000 7f8f7d400640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8f60067660 0x7f8f60069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:03.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.297+0000 7f8f7d400640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8f60067660 0x7f8f60069b20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8f800cf500 tx=0x7f8f6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:03.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.301+0000 7f8f76000640 1 -- 172.21.15.40:0/159023407 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8f70022020 con 0x7f8f8006e620 2024-03-24T00:32:03.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.461+0000 7f8f84e00640 1 -- 172.21.15.40:0/159023407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8f80072f50 con 0x7f8f8006e620 2024-03-24T00:32:03.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f76000640 1 -- 172.21.15.40:0/159023407 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8f70085e20 con 0x7f8f8006e620 2024-03-24T00:32:03.468 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:03.468 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-03-24T00:32:03.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:03.469 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-03-24T00:32:03.469 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-03-24T00:32:03.469 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:03.469 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-03-24T00:32:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f84e00640 1 -- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8f60067660 msgr2=0x7f8f60069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f84e00640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8f60067660 0x7f8f60069b20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8f800cf500 tx=0x7f8f6c009040 comp rx=0 tx=0).stop 2024-03-24T00:32:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f84e00640 1 -- 172.21.15.40:0/159023407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8006e620 msgr2=0x7f8f8015b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f84e00640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8006e620 0x7f8f8015b480 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000d800 tx=0x7f8f7000dcc0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f84e00640 1 -- 172.21.15.40:0/159023407 shutdown_connections 2024-03-24T00:32:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f84e00640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8f60067660 0x7f8f60069b20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f84e00640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f800ccfa0 0x7f8f8013d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f84e00640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f8006ea00 0x7f8f8013ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f84e00640 1 --2- 172.21.15.40:0/159023407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f8006e620 0x7f8f8015b480 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f84e00640 1 -- 172.21.15.40:0/159023407 >> 172.21.15.40:0/159023407 conn(0x7f8f800b17b0 msgr2=0x7f8f8006dde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f84e00640 1 -- 172.21.15.40:0/159023407 shutdown_connections 2024-03-24T00:32:03.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.465+0000 7f8f84e00640 1 -- 172.21.15.40:0/159023407 wait complete. 2024-03-24T00:32:03.490 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:32:03.490 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:32:03.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.629+0000 7f3f81a00640 1 Processor -- start 2024-03-24T00:32:03.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.629+0000 7f3f81a00640 1 -- start start 2024-03-24T00:32:03.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f81a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f7c15e000 0x7f3f7c0d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:03.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f81a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3f7c0b42d0 0x7f3f7c06dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:03.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f81a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f7c15e3e0 con 0x7f3f7c1628d0 2024-03-24T00:32:03.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f81a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f7c161ba0 con 0x7f3f7c0b42d0 2024-03-24T00:32:03.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f81a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c0b5f10 con 0x7f3f7c15e000 2024-03-24T00:32:03.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f7aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f7c15e000 0x7f3f7c0d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:03.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f7aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f7c15e000 0x7f3f7c0d16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:50576/0 (socket says 172.21.15.40:50576) 2024-03-24T00:32:03.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f7aa00640 1 -- 172.21.15.40:0/2942966138 learned_addr learned my addr 172.21.15.40:0/2942966138 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:03.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f7b400640 1 --2- 172.21.15.40:0/2942966138 >> v2:172.21.15.53:3300/0 conn(0x7f3f7c0b42d0 0x7f3f7c06dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:03.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f7aa00640 1 -- 172.21.15.40:0/2942966138 >> v2:172.21.15.53:3300/0 conn(0x7f3f7c0b42d0 msgr2=0x7f3f7c06dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:03.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f7aa00640 1 --2- 172.21.15.40:0/2942966138 >> v2:172.21.15.53:3300/0 conn(0x7f3f7c0b42d0 0x7f3f7c06dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f7aa00640 1 -- 172.21.15.40:0/2942966138 >> v1:172.21.15.53:6789/0 conn(0x7f3f7c1628d0 legacy=0x7f3f7c0b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:03.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.633+0000 7f3f7aa00640 1 -- 172.21.15.40:0/2942966138 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c161df0 con 0x7f3f7c15e000 2024-03-24T00:32:03.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f7aa00640 1 --2- 172.21.15.40:0/2942966138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f7c15e000 0x7f3f7c0d16d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009870 tx=0x7f3f6c01c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c33d491acc38d61 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:03.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f7a000640 1 -- 172.21.15.40:0/2942966138 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c0b0070 con 0x7f3f7c15e000 2024-03-24T00:32:03.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f7a000640 1 -- 172.21.15.40:0/2942966138 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6c0ab600 con 0x7f3f7c15e000 2024-03-24T00:32:03.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f7a000640 1 -- 172.21.15.40:0/2942966138 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c0ab9d0 con 0x7f3f7c15e000 2024-03-24T00:32:03.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f81a00640 1 -- 172.21.15.40:0/2942966138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f7c15e000 msgr2=0x7f3f7c0d16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:03.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f81a00640 1 --2- 172.21.15.40:0/2942966138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f7c15e000 0x7f3f7c0d16d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009870 tx=0x7f3f6c01c530 comp rx=0 tx=0).stop 2024-03-24T00:32:03.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f81a00640 1 -- 172.21.15.40:0/2942966138 shutdown_connections 2024-03-24T00:32:03.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f81a00640 1 --2- 172.21.15.40:0/2942966138 >> v2:172.21.15.53:3300/0 conn(0x7f3f7c0b42d0 0x7f3f7c06dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f81a00640 1 --2- 172.21.15.40:0/2942966138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f7c15e000 0x7f3f7c0d16d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f81a00640 1 -- 172.21.15.40:0/2942966138 >> 172.21.15.40:0/2942966138 conn(0x7f3f7c0d0bb0 msgr2=0x7f3f7c0d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:03.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f81a00640 1 -- 172.21.15.40:0/2942966138 shutdown_connections 2024-03-24T00:32:03.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f81a00640 1 -- 172.21.15.40:0/2942966138 wait complete. 2024-03-24T00:32:03.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f81a00640 1 Processor -- start 2024-03-24T00:32:03.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.637+0000 7f3f81a00640 1 -- start start 2024-03-24T00:32:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f81a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c0b42d0 0x7f3f7c152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f81a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f7c15e000 0x7f3f7c14a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f81a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f7c1628d0 0x7f3f7c14af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f81a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c16b0c0 con 0x7f3f7c0b42d0 2024-03-24T00:32:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f81a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c16af40 con 0x7f3f7c1628d0 2024-03-24T00:32:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f81a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7c16b240 con 0x7f3f7c15e000 2024-03-24T00:32:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f7aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c0b42d0 0x7f3f7c152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f7aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c0b42d0 0x7f3f7c152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48258/0 (socket says 172.21.15.40:48258) 2024-03-24T00:32:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f7aa00640 1 -- 172.21.15.40:0/1911563954 learned_addr learned my addr 172.21.15.40:0/1911563954 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f7b400640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f7c1628d0 0x7f3f7c14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f73e00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f7c15e000 0x7f3f7c14a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f7aa00640 1 -- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f7c15e000 msgr2=0x7f3f7c14a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f7aa00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f7c15e000 0x7f3f7c14a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f7aa00640 1 -- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f7c1628d0 msgr2=0x7f3f7c14af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f7aa00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f7c1628d0 0x7f3f7c14af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f7aa00640 1 -- 172.21.15.40:0/1911563954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7c151960 con 0x7f3f7c0b42d0 2024-03-24T00:32:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f7aa00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c0b42d0 0x7f3f7c152a00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c01abe0 tx=0x7f3f6c0192d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:03.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f72a00640 1 -- 172.21.15.40:0/1911563954 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c0ba070 con 0x7f3f7c0b42d0 2024-03-24T00:32:03.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f72a00640 1 -- 172.21.15.40:0/1911563954 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6c0b4420 con 0x7f3f7c0b42d0 2024-03-24T00:32:03.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f72a00640 1 -- 172.21.15.40:0/1911563954 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c0b0040 con 0x7f3f7c0b42d0 2024-03-24T00:32:03.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f81a00640 1 -- 172.21.15.40:0/1911563954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7c151c40 con 0x7f3f7c0b42d0 2024-03-24T00:32:03.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f81a00640 1 -- 172.21.15.40:0/1911563954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7c152200 con 0x7f3f7c0b42d0 2024-03-24T00:32:03.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f81a00640 1 -- 172.21.15.40:0/1911563954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7c068170 con 0x7f3f7c0b42d0 2024-03-24T00:32:03.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.641+0000 7f3f72a00640 1 -- 172.21.15.40:0/1911563954 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3f6c0b4810 con 0x7f3f7c0b42d0 2024-03-24T00:32:03.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.645+0000 7f3f72a00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f58067660 0x7f3f58069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:03.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.645+0000 7f3f72a00640 1 -- 172.21.15.40:0/1911563954 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3f6c11e750 con 0x7f3f7c0b42d0 2024-03-24T00:32:03.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.645+0000 7f3f73e00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f58067660 0x7f3f58069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:03.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.645+0000 7f3f73e00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f58067660 0x7f3f58069b20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c14df00 tx=0x7f3f64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:03.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.649+0000 7f3f72a00640 1 -- 172.21.15.40:0/1911563954 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3f6c016150 con 0x7f3f7c0b42d0 2024-03-24T00:32:03.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:03.813+0000 7f3f81a00640 1 -- 172.21.15.40:0/1911563954 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3f7c0aee80 con 0x7f3f7c0b42d0 2024-03-24T00:32:04.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.613+0000 7f3f72a00640 1 -- 172.21.15.40:0/1911563954 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3f6c0ef0d0 con 0x7f3f7c0b42d0 2024-03-24T00:32:04.619 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-03-24T00:32:04.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.617+0000 7f3f81a00640 1 -- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f58067660 msgr2=0x7f3f58069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.617+0000 7f3f81a00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f58067660 0x7f3f58069b20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c14df00 tx=0x7f3f64009040 comp rx=0 tx=0).stop 2024-03-24T00:32:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.617+0000 7f3f81a00640 1 -- 172.21.15.40:0/1911563954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c0b42d0 msgr2=0x7f3f7c152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.617+0000 7f3f81a00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c0b42d0 0x7f3f7c152a00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c01abe0 tx=0x7f3f6c0192d0 comp rx=0 tx=0).stop 2024-03-24T00:32:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.617+0000 7f3f81a00640 1 -- 172.21.15.40:0/1911563954 shutdown_connections 2024-03-24T00:32:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.617+0000 7f3f81a00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f58067660 0x7f3f58069b20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.617+0000 7f3f81a00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f7c1628d0 0x7f3f7c14af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.617+0000 7f3f81a00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f7c15e000 0x7f3f7c14a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.617+0000 7f3f81a00640 1 --2- 172.21.15.40:0/1911563954 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f7c0b42d0 0x7f3f7c152a00 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.617+0000 7f3f81a00640 1 -- 172.21.15.40:0/1911563954 >> 172.21.15.40:0/1911563954 conn(0x7f3f7c0d0bb0 msgr2=0x7f3f7c0ad970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.617+0000 7f3f81a00640 1 -- 172.21.15.40:0/1911563954 shutdown_connections 2024-03-24T00:32:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.617+0000 7f3f81a00640 1 -- 172.21.15.40:0/1911563954 wait complete. 2024-03-24T00:32:04.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:32:04.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f4892a00640 1 Processor -- start 2024-03-24T00:32:04.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f4892a00640 1 -- start start 2024-03-24T00:32:04.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f4892a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f488c0b3380 0x7f488c070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:04.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f4892a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488c15f2c0 con 0x7f488c072b60 2024-03-24T00:32:04.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f4892a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488c06d230 con 0x7f488c0cf4e0 2024-03-24T00:32:04.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f4892a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488c06cfe0 con 0x7f488c0b3380 2024-03-24T00:32:04.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f4883400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f488c072b60 0x7f488c0b2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42660/0 (socket says 172.21.15.40:42660) 2024-03-24T00:32:04.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f4883400640 1 -- 172.21.15.40:0/2928785232 learned_addr learned my addr 172.21.15.40:0/2928785232 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:04.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f4890a00640 1 --2- 172.21.15.40:0/2928785232 >> v2:172.21.15.53:3300/0 conn(0x7f488c0b3380 0x7f488c070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48072430 0 0) 0x7f488c15f2c0 con 0x7f488c072b60 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4870003530 con 0x7f488c072b60 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2667329139 0 0) 0x7f4870003530 con 0x7f488c072b60 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 >> v2:172.21.15.53:3300/0 conn(0x7f488c0b3380 msgr2=0x7f488c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 --2- 172.21.15.40:0/2928785232 >> v2:172.21.15.53:3300/0 conn(0x7f488c0b3380 0x7f488c070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 >> v1:172.21.15.53:6789/0 conn(0x7f488c0cf4e0 legacy=0x7f488c06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c06d510 con 0x7f488c072b60 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f4874001680 con 0x7f488c072b60 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 >> v1:172.21.15.40:6789/0 conn(0x7f488c072b60 legacy=0x7f488c0b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 --2- 172.21.15.40:0/2928785232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48700039c0 0x7f4870003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 --2- 172.21.15.40:0/2928785232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 0x7f4870004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 --2- 172.21.15.40:0/2928785232 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48700051d0 0x7f487000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c06cfe0 con 0x7f4870004810 2024-03-24T00:32:04.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487000c080 con 0x7f48700039c0 2024-03-24T00:32:04.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f487000c280 con 0x7f48700051d0 2024-03-24T00:32:04.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.793+0000 7f4883400640 1 --2- 172.21.15.40:0/2928785232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 0x7f4870004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:04.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f488be00640 1 --2- 172.21.15.40:0/2928785232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48700039c0 0x7f4870003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:04.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f4890a00640 1 --2- 172.21.15.40:0/2928785232 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48700051d0 0x7f487000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:04.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f4883400640 1 -- 172.21.15.40:0/2928785232 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48700051d0 msgr2=0x7f487000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:04.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f4883400640 1 --2- 172.21.15.40:0/2928785232 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48700051d0 0x7f487000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:04.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f4883400640 1 -- 172.21.15.40:0/2928785232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48700039c0 msgr2=0x7f4870003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:04.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f4883400640 1 --2- 172.21.15.40:0/2928785232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48700039c0 0x7f4870003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:04.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f4883400640 1 -- 172.21.15.40:0/2928785232 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4870003530 con 0x7f4870004810 2024-03-24T00:32:04.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f4890a00640 1 -- 172.21.15.40:0/2928785232 reap_dead start 2024-03-24T00:32:04.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f4883400640 1 --2- 172.21.15.40:0/2928785232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 0x7f4870004c90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f487400c190 tx=0x7f487401cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb2d3be820e70f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:04.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874007d00 con 0x7f4870004810 2024-03-24T00:32:04.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48740a9020 con 0x7f4870004810 2024-03-24T00:32:04.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f488b400640 1 -- 172.21.15.40:0/2928785232 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874005ae0 con 0x7f4870004810 2024-03-24T00:32:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f4892a00640 1 -- 172.21.15.40:0/2928785232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 msgr2=0x7f4870004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.797+0000 7f4892a00640 1 --2- 172.21.15.40:0/2928785232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 0x7f4870004c90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f487400c190 tx=0x7f487401cbc0 comp rx=0 tx=0).stop 2024-03-24T00:32:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 -- 172.21.15.40:0/2928785232 shutdown_connections 2024-03-24T00:32:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 --2- 172.21.15.40:0/2928785232 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 0x7f4870004c90 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:04.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 -- 172.21.15.40:0/2928785232 >> 172.21.15.40:0/2928785232 conn(0x7f488c0b1660 msgr2=0x7f488c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:04.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 -- 172.21.15.40:0/2928785232 shutdown_connections 2024-03-24T00:32:04.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 -- 172.21.15.40:0/2928785232 wait complete. 2024-03-24T00:32:04.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 Processor -- start 2024-03-24T00:32:04.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 -- start start 2024-03-24T00:32:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 0x7f488c156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f488c142ba0 0x7f488c142240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f488c154770 0x7f488c160790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c0724a0 con 0x7f4870004810 2024-03-24T00:32:04.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c06d880 con 0x7f488c154770 2024-03-24T00:32:04.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4892a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488c072620 con 0x7f488c142ba0 2024-03-24T00:32:04.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f488be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 0x7f488c156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:04.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f488be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 0x7f488c156f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48272/0 (socket says 172.21.15.40:48272) 2024-03-24T00:32:04.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f488be00640 1 -- 172.21.15.40:0/3384816237 learned_addr learned my addr 172.21.15.40:0/3384816237 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:04.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4883400640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f488c142ba0 0x7f488c142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:04.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4890a00640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f488c154770 0x7f488c160790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:04.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f488be00640 1 -- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f488c142ba0 msgr2=0x7f488c142240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:04.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f488be00640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f488c142ba0 0x7f488c142240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:04.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f488be00640 1 -- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f488c154770 msgr2=0x7f488c160790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:04.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f488be00640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f488c154770 0x7f488c160790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:04.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f488be00640 1 -- 172.21.15.40:0/3384816237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c157fc0 con 0x7f4870004810 2024-03-24T00:32:04.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f488be00640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 0x7f488c156f60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f487c00f770 tx=0x7f487c00fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:04.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.801+0000 7f4888c00640 1 -- 172.21.15.40:0/3384816237 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c0050d0 con 0x7f4870004810 2024-03-24T00:32:04.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.805+0000 7f4888c00640 1 -- 172.21.15.40:0/3384816237 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487c018070 con 0x7f4870004810 2024-03-24T00:32:04.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.805+0000 7f4888c00640 1 -- 172.21.15.40:0/3384816237 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c00d730 con 0x7f4870004810 2024-03-24T00:32:04.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.805+0000 7f4892a00640 1 -- 172.21.15.40:0/3384816237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488c1582a0 con 0x7f4870004810 2024-03-24T00:32:04.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.805+0000 7f4892a00640 1 -- 172.21.15.40:0/3384816237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488c158860 con 0x7f4870004810 2024-03-24T00:32:04.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.805+0000 7f4892a00640 1 -- 172.21.15.40:0/3384816237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c068170 con 0x7f4870004810 2024-03-24T00:32:04.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.805+0000 7f4888c00640 1 -- 172.21.15.40:0/3384816237 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f487c00d8d0 con 0x7f4870004810 2024-03-24T00:32:04.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.805+0000 7f4888c00640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4868067660 0x7f4868069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:04.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.805+0000 7f4888c00640 1 -- 172.21.15.40:0/3384816237 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f487c085d30 con 0x7f4870004810 2024-03-24T00:32:04.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.805+0000 7f4883400640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4868067660 0x7f4868069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:04.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.809+0000 7f4883400640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4868067660 0x7f4868069b20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f48740aa000 tx=0x7f488c070af0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:04.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.809+0000 7f4888c00640 1 -- 172.21.15.40:0/3384816237 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f487c0565b0 con 0x7f4870004810 2024-03-24T00:32:04.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:04.973+0000 7f4892a00640 1 -- 172.21.15.40:0/3384816237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f488c0b2ba0 con 0x7f4870004810 2024-03-24T00:32:05.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.621+0000 7f4888c00640 1 -- 172.21.15.40:0/3384816237 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) v1 ==== 170+0+0 (secure 0 0 0) 0x7f487c05a650 con 0x7f4870004810 2024-03-24T00:32:05.626 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-03-24T00:32:05.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.625+0000 7f4892a00640 1 -- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4868067660 msgr2=0x7f4868069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:05.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.625+0000 7f4892a00640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4868067660 0x7f4868069b20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f48740aa000 tx=0x7f488c070af0 comp rx=0 tx=0).stop 2024-03-24T00:32:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.625+0000 7f4892a00640 1 -- 172.21.15.40:0/3384816237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 msgr2=0x7f488c156f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.625+0000 7f4892a00640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 0x7f488c156f60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f487c00f770 tx=0x7f487c00fc30 comp rx=0 tx=0).stop 2024-03-24T00:32:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.625+0000 7f4892a00640 1 -- 172.21.15.40:0/3384816237 shutdown_connections 2024-03-24T00:32:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.625+0000 7f4892a00640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4868067660 0x7f4868069b20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.625+0000 7f4892a00640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f488c154770 0x7f488c160790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.625+0000 7f4892a00640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f488c142ba0 0x7f488c142240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.625+0000 7f4892a00640 1 --2- 172.21.15.40:0/3384816237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4870004810 0x7f488c156f60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.625+0000 7f4892a00640 1 -- 172.21.15.40:0/3384816237 >> 172.21.15.40:0/3384816237 conn(0x7f488c0b1660 msgr2=0x7f488c0aed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.625+0000 7f4892a00640 1 -- 172.21.15.40:0/3384816237 shutdown_connections 2024-03-24T00:32:05.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.625+0000 7f4892a00640 1 -- 172.21.15.40:0/3384816237 wait complete. 2024-03-24T00:32:05.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:32:05.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3e8400640 1 Processor -- start 2024-03-24T00:32:05.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3e8400640 1 -- start start 2024-03-24T00:32:05.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3e8400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3e0071b00 0x7fb3e0071f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:05.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3e8400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e00b3260 con 0x7fb3e015ac60 2024-03-24T00:32:05.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3e8400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e00b3540 con 0x7fb3e006cb80 2024-03-24T00:32:05.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3e8400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e0072490 con 0x7fb3e0071b00 2024-03-24T00:32:05.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3e6400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3e0071b00 0x7fb3e0071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:05.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3e5a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb3e015ac60 0x7fb3e00b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42662/0 (socket says 172.21.15.40:42662) 2024-03-24T00:32:05.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3e5a00640 1 -- 172.21.15.40:0/2924857038 learned_addr learned my addr 172.21.15.40:0/2924857038 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:05.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3e6400640 1 -- 172.21.15.40:0/2924857038 >> v1:172.21.15.53:6789/0 conn(0x7fb3e006cb80 legacy=0x7fb3e00b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:05.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3e6400640 1 -- 172.21.15.40:0/2924857038 >> v1:172.21.15.40:6789/0 conn(0x7fb3e015ac60 legacy=0x7fb3e00b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:05.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3dfe00640 1 -- 172.21.15.40:0/2924857038 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336234162 0 0) 0x7fb3e00b3260 con 0x7fb3e015ac60 2024-03-24T00:32:05.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.809+0000 7fb3e6400640 1 -- 172.21.15.40:0/2924857038 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e00b3820 con 0x7fb3e0071b00 2024-03-24T00:32:05.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3e6400640 1 --2- 172.21.15.40:0/2924857038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3e0071b00 0x7fb3e0071f50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009d90 tx=0x7fb3d801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edffd8d737ee7969 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:05.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3dfe00640 1 -- 172.21.15.40:0/2924857038 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d80b2070 con 0x7fb3e0071b00 2024-03-24T00:32:05.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3dfe00640 1 -- 172.21.15.40:0/2924857038 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d80a57a0 con 0x7fb3e0071b00 2024-03-24T00:32:05.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3dfe00640 1 -- 172.21.15.40:0/2924857038 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d80a5b70 con 0x7fb3e0071b00 2024-03-24T00:32:05.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3e8400640 1 -- 172.21.15.40:0/2924857038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3e0071b00 msgr2=0x7fb3e0071f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:05.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3e8400640 1 --2- 172.21.15.40:0/2924857038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3e0071b00 0x7fb3e0071f50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009d90 tx=0x7fb3d801c8e0 comp rx=0 tx=0).stop 2024-03-24T00:32:05.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3e8400640 1 -- 172.21.15.40:0/2924857038 shutdown_connections 2024-03-24T00:32:05.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3e8400640 1 --2- 172.21.15.40:0/2924857038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3e0071b00 0x7fb3e0071f50 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:05.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3e8400640 1 -- 172.21.15.40:0/2924857038 >> 172.21.15.40:0/2924857038 conn(0x7fb3e00b1660 msgr2=0x7fb3e00b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:05.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3e8400640 1 -- 172.21.15.40:0/2924857038 shutdown_connections 2024-03-24T00:32:05.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3e8400640 1 -- 172.21.15.40:0/2924857038 wait complete. 2024-03-24T00:32:05.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3e8400640 1 Processor -- start 2024-03-24T00:32:05.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.813+0000 7fb3e8400640 1 -- start start 2024-03-24T00:32:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e8400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e006cb80 0x7fb3e0150680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e8400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3e0071b00 0x7fb3e014ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e8400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3e015ac60 0x7fb3e00b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e8400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e0168a50 con 0x7fb3e0071b00 2024-03-24T00:32:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e8400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e01688d0 con 0x7fb3e006cb80 2024-03-24T00:32:05.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e8400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e0168bd0 con 0x7fb3e015ac60 2024-03-24T00:32:05.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3e0071b00 0x7fb3e014ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:05.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3e0071b00 0x7fb3e014ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48302/0 (socket says 172.21.15.40:48302) 2024-03-24T00:32:05.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e5000640 1 -- 172.21.15.40:0/2134439907 learned_addr learned my addr 172.21.15.40:0/2134439907 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:05.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e5a00640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e006cb80 0x7fb3e0150680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:05.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e6400640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3e015ac60 0x7fb3e00b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:05.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e5000640 1 -- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3e015ac60 msgr2=0x7fb3e00b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:05.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e5000640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3e015ac60 0x7fb3e00b1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:05.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e5000640 1 -- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e006cb80 msgr2=0x7fb3e0150680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:05.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e5000640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e006cb80 0x7fb3e0150680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:05.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e5000640 1 -- 172.21.15.40:0/2134439907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e0150c80 con 0x7fb3e0071b00 2024-03-24T00:32:05.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e5000640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3e0071b00 0x7fb3e014ebd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00c390 tx=0x7fb3cc00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3dd600640 1 -- 172.21.15.40:0/2134439907 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc017070 con 0x7fb3e0071b00 2024-03-24T00:32:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3dd600640 1 -- 172.21.15.40:0/2134439907 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3cc0052d0 con 0x7fb3e0071b00 2024-03-24T00:32:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e8400640 1 -- 172.21.15.40:0/2134439907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e01556d0 con 0x7fb3e0071b00 2024-03-24T00:32:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3dd600640 1 -- 172.21.15.40:0/2134439907 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc003820 con 0x7fb3e0071b00 2024-03-24T00:32:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e8400640 1 -- 172.21.15.40:0/2134439907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e0155c90 con 0x7fb3e0071b00 2024-03-24T00:32:05.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3e8400640 1 -- 172.21.15.40:0/2134439907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e0068170 con 0x7fb3e0071b00 2024-03-24T00:32:05.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.817+0000 7fb3dd600640 1 -- 172.21.15.40:0/2134439907 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb3cc01a070 con 0x7fb3e0071b00 2024-03-24T00:32:05.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.821+0000 7fb3dd600640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3c0067660 0x7fb3c0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:05.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.821+0000 7fb3dd600640 1 -- 172.21.15.40:0/2134439907 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fb3cc084b20 con 0x7fb3e0071b00 2024-03-24T00:32:05.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.821+0000 7fb3e5a00640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3c0067660 0x7fb3c0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:05.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.821+0000 7fb3e5a00640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3c0067660 0x7fb3c0069b20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4000f70 tx=0x7fb3d4007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:05.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:05.825+0000 7fb3dd600640 1 -- 172.21.15.40:0/2134439907 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb3cc005480 con 0x7fb3e0071b00 2024-03-24T00:32:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.021+0000 7fb3e8400640 1 -- 172.21.15.40:0/2134439907 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb3e0151db0 con 0x7fb3e0071b00 2024-03-24T00:32:06.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.637+0000 7fb3dd600640 1 -- 172.21.15.40:0/2134439907 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7fb3cc0551a0 con 0x7fb3e0071b00 2024-03-24T00:32:06.645 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 4 and data pool 5 2024-03-24T00:32:06.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.641+0000 7fb3e8400640 1 -- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3c0067660 msgr2=0x7fb3c0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:06.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.641+0000 7fb3e8400640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3c0067660 0x7fb3c0069b20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4000f70 tx=0x7fb3d4007240 comp rx=0 tx=0).stop 2024-03-24T00:32:06.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.641+0000 7fb3e8400640 1 -- 172.21.15.40:0/2134439907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3e0071b00 msgr2=0x7fb3e014ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:06.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.641+0000 7fb3e8400640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3e0071b00 0x7fb3e014ebd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00c390 tx=0x7fb3cc00c850 comp rx=0 tx=0).stop 2024-03-24T00:32:06.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.645+0000 7fb3e8400640 1 -- 172.21.15.40:0/2134439907 shutdown_connections 2024-03-24T00:32:06.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.645+0000 7fb3e8400640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3c0067660 0x7fb3c0069b20 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4000f70 tx=0x7fb3d4007240 comp rx=0 tx=0).stop 2024-03-24T00:32:06.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.645+0000 7fb3e8400640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3e015ac60 0x7fb3e00b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:06.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.645+0000 7fb3e8400640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3e0071b00 0x7fb3e014ebd0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:06.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.645+0000 7fb3e8400640 1 --2- 172.21.15.40:0/2134439907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e006cb80 0x7fb3e0150680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:06.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.645+0000 7fb3e8400640 1 -- 172.21.15.40:0/2134439907 >> 172.21.15.40:0/2134439907 conn(0x7fb3e00b1660 msgr2=0x7fb3e006d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:06.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.645+0000 7fb3e8400640 1 -- 172.21.15.40:0/2134439907 shutdown_connections 2024-03-24T00:32:06.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.649+0000 7fb3e8400640 1 -- 172.21.15.40:0/2134439907 wait complete. 2024-03-24T00:32:06.691 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:32:06.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.829+0000 7fdb51e00640 1 Processor -- start 2024-03-24T00:32:06.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.829+0000 7fdb51e00640 1 -- start start 2024-03-24T00:32:06.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb51e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb4c15f600 0x7fdb4c0b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:06.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb51e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb4c0af170 0x7fdb4c165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:06.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb51e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c0d2c70 con 0x7fdb4c0d2830 2024-03-24T00:32:06.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb51e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb4c0b3630 con 0x7fdb4c0af170 2024-03-24T00:32:06.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb51e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c0b2740 con 0x7fdb4c15f600 2024-03-24T00:32:06.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb4b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb4c0af170 0x7fdb4c165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:06.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb4b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb4c0af170 0x7fdb4c165e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48306/0 (socket says 172.21.15.40:48306) 2024-03-24T00:32:06.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb4b400640 1 -- 172.21.15.40:0/3842005636 learned_addr learned my addr 172.21.15.40:0/3842005636 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:06.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb4aa00640 1 --2- 172.21.15.40:0/3842005636 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb4c15f600 0x7fdb4c0b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:06.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb4b400640 1 -- 172.21.15.40:0/3842005636 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb4c15f600 msgr2=0x7fdb4c0b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:06.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb4b400640 1 --2- 172.21.15.40:0/3842005636 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb4c15f600 0x7fdb4c0b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:06.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb4b400640 1 -- 172.21.15.40:0/3842005636 >> v1:172.21.15.53:6789/0 conn(0x7fdb4c0d2830 legacy=0x7fdb4c0aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:06.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.833+0000 7fdb4b400640 1 -- 172.21.15.40:0/3842005636 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c0b1e50 con 0x7fdb4c0af170 2024-03-24T00:32:06.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb4b400640 1 --2- 172.21.15.40:0/3842005636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb4c0af170 0x7fdb4c165e40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdb44009e50 tx=0x7fdb4401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfc402423bd3809e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:06.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb49600640 1 -- 172.21.15.40:0/3842005636 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb440b4070 con 0x7fdb4c0af170 2024-03-24T00:32:06.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb49600640 1 -- 172.21.15.40:0/3842005636 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb440af400 con 0x7fdb4c0af170 2024-03-24T00:32:06.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb49600640 1 -- 172.21.15.40:0/3842005636 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb440af7d0 con 0x7fdb4c0af170 2024-03-24T00:32:06.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 -- 172.21.15.40:0/3842005636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb4c0af170 msgr2=0x7fdb4c165e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:06.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 --2- 172.21.15.40:0/3842005636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb4c0af170 0x7fdb4c165e40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdb44009e50 tx=0x7fdb4401f210 comp rx=0 tx=0).stop 2024-03-24T00:32:06.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 -- 172.21.15.40:0/3842005636 shutdown_connections 2024-03-24T00:32:06.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 --2- 172.21.15.40:0/3842005636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb4c0af170 0x7fdb4c165e40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:06.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 --2- 172.21.15.40:0/3842005636 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb4c15f600 0x7fdb4c0b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:06.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 -- 172.21.15.40:0/3842005636 >> 172.21.15.40:0/3842005636 conn(0x7fdb4c0d1d10 msgr2=0x7fdb4c0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:06.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 -- 172.21.15.40:0/3842005636 shutdown_connections 2024-03-24T00:32:06.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 -- 172.21.15.40:0/3842005636 wait complete. 2024-03-24T00:32:06.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 Processor -- start 2024-03-24T00:32:06.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 -- start start 2024-03-24T00:32:06.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb4c0af170 0x7fdb4c1f47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:06.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb4c0d2830 0x7fdb4c0c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:06.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb4c15f600 0x7fdb4c0c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:06.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c0b2fa0 con 0x7fdb4c15f600 2024-03-24T00:32:06.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c0b23d0 con 0x7fdb4c0af170 2024-03-24T00:32:06.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb51e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4c15f9e0 con 0x7fdb4c0d2830 2024-03-24T00:32:06.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb4b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb4c15f600 0x7fdb4c0c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:06.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb4b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb4c15f600 0x7fdb4c0c9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48322/0 (socket says 172.21.15.40:48322) 2024-03-24T00:32:06.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.837+0000 7fdb4b400640 1 -- 172.21.15.40:0/3097728556 learned_addr learned my addr 172.21.15.40:0/3097728556 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:06.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb4aa00640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb4c0af170 0x7fdb4c1f47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:06.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb4a000640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb4c0d2830 0x7fdb4c0c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:06.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb4b400640 1 -- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb4c0d2830 msgr2=0x7fdb4c0c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:06.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb4b400640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb4c0d2830 0x7fdb4c0c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:06.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb4b400640 1 -- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb4c0af170 msgr2=0x7fdb4c1f47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:06.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb4b400640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb4c0af170 0x7fdb4c1f47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:06.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb4b400640 1 -- 172.21.15.40:0/3097728556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c1f4170 con 0x7fdb4c15f600 2024-03-24T00:32:06.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb4b400640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb4c15f600 0x7fdb4c0c9d30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fdb440b2040 tx=0x7fdb4401bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:06.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb43400640 1 -- 172.21.15.40:0/3097728556 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb440c0070 con 0x7fdb4c15f600 2024-03-24T00:32:06.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb43400640 1 -- 172.21.15.40:0/3097728556 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb440090d0 con 0x7fdb4c15f600 2024-03-24T00:32:06.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb43400640 1 -- 172.21.15.40:0/3097728556 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb440b4040 con 0x7fdb4c15f600 2024-03-24T00:32:06.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb51e00640 1 -- 172.21.15.40:0/3097728556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb4c0ca400 con 0x7fdb4c15f600 2024-03-24T00:32:06.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb51e00640 1 -- 172.21.15.40:0/3097728556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4c0ca9c0 con 0x7fdb4c15f600 2024-03-24T00:32:06.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb51e00640 1 -- 172.21.15.40:0/3097728556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb4c068170 con 0x7fdb4c15f600 2024-03-24T00:32:06.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.841+0000 7fdb43400640 1 -- 172.21.15.40:0/3097728556 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fdb440c3070 con 0x7fdb4c15f600 2024-03-24T00:32:06.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.845+0000 7fdb43400640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb2c067690 0x7fdb2c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:06.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.845+0000 7fdb43400640 1 -- 172.21.15.40:0/3097728556 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb44122dc0 con 0x7fdb4c15f600 2024-03-24T00:32:06.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.845+0000 7fdb4aa00640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb2c067690 0x7fdb2c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:06.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.845+0000 7fdb4aa00640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb2c067690 0x7fdb2c069b50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009b60 tx=0x7fdb3c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:06.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:06.849+0000 7fdb43400640 1 -- 172.21.15.40:0/3097728556 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdb44018d70 con 0x7fdb4c15f600 2024-03-24T00:32:06.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:32:06.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:32:07.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.013+0000 7fdb51e00640 1 -- 172.21.15.40:0/3097728556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb4c0aea60 con 0x7fdb4c15f600 2024-03-24T00:32:07.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb43400640 1 -- 172.21.15.40:0/3097728556 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16622 (secure 0 0 0) 0x7fdb440f3380 con 0x7fdb4c15f600 2024-03-24T00:32:07.022 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:07.022 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":32,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:32:06.027640+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:03.863519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:05.024936+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:32:07.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb51e00640 1 -- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb2c067690 msgr2=0x7fdb2c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb51e00640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb2c067690 0x7fdb2c069b50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009b60 tx=0x7fdb3c009190 comp rx=0 tx=0).stop 2024-03-24T00:32:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb51e00640 1 -- 172.21.15.40:0/3097728556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb4c15f600 msgr2=0x7fdb4c0c9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb51e00640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb4c15f600 0x7fdb4c0c9d30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fdb440b2040 tx=0x7fdb4401bfb0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb51e00640 1 -- 172.21.15.40:0/3097728556 shutdown_connections 2024-03-24T00:32:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb51e00640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb2c067690 0x7fdb2c069b50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb51e00640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb4c15f600 0x7fdb4c0c9d30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb51e00640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb4c0d2830 0x7fdb4c0c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb51e00640 1 --2- 172.21.15.40:0/3097728556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb4c0af170 0x7fdb4c1f47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb51e00640 1 -- 172.21.15.40:0/3097728556 >> 172.21.15.40:0/3097728556 conn(0x7fdb4c0d1d10 msgr2=0x7fdb4c0d20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:07.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb51e00640 1 -- 172.21.15.40:0/3097728556 shutdown_connections 2024-03-24T00:32:07.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.017+0000 7fdb51e00640 1 -- 172.21.15.40:0/3097728556 wait complete. 2024-03-24T00:32:07.043 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:32:07.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3957e00640 1 Processor -- start 2024-03-24T00:32:07.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3957e00640 1 -- start start 2024-03-24T00:32:07.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3957e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f39580b41d0 0x7f3958162450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:07.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3957e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395806f180 0x7f3958067890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3957e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39580b5b80 con 0x7f395815dc80 2024-03-24T00:32:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3957e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39580b5950 con 0x7f39580b41d0 2024-03-24T00:32:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3957e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3958162990 con 0x7f395806f180 2024-03-24T00:32:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3957400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395806f180 0x7f3958067890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3956000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f39580b41d0 0x7f3958162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3956a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f395815dc80 0x7f39580b3ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:43870/0 (socket says 172.21.15.40:43870) 2024-03-24T00:32:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3956a00640 1 -- 172.21.15.40:0/3066157638 learned_addr learned my addr 172.21.15.40:0/3066157638 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3955600640 1 -- 172.21.15.40:0/3066157638 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804311626 0 0) 0x7f39580b5b80 con 0x7f395815dc80 2024-03-24T00:32:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3955600640 1 -- 172.21.15.40:0/3066157638 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f393c003530 con 0x7f395815dc80 2024-03-24T00:32:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3957400640 1 -- 172.21.15.40:0/3066157638 >> v2:172.21.15.53:3300/0 conn(0x7f39580b41d0 msgr2=0x7f3958162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3957400640 1 --2- 172.21.15.40:0/3066157638 >> v2:172.21.15.53:3300/0 conn(0x7f39580b41d0 0x7f3958162450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3957400640 1 -- 172.21.15.40:0/3066157638 >> v1:172.21.15.53:6789/0 conn(0x7f395815dc80 legacy=0x7f39580b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:07.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3957400640 1 -- 172.21.15.40:0/3066157638 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f395815f070 con 0x7f395806f180 2024-03-24T00:32:07.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.181+0000 7f3956000640 1 --2- 172.21.15.40:0/3066157638 >> v2:172.21.15.53:3300/0 conn(0x7f39580b41d0 0x7f3958162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:07.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.185+0000 7f3957400640 1 --2- 172.21.15.40:0/3066157638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395806f180 0x7f3958067890 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3950009d90 tx=0x7f395001c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=585700750e636208 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:07.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.185+0000 7f3955600640 1 -- 172.21.15.40:0/3066157638 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39500b2070 con 0x7f395806f180 2024-03-24T00:32:07.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.185+0000 7f3955600640 1 -- 172.21.15.40:0/3066157638 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39500a57a0 con 0x7f395806f180 2024-03-24T00:32:07.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.185+0000 7f3955600640 1 -- 172.21.15.40:0/3066157638 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39500a5b70 con 0x7f395806f180 2024-03-24T00:32:07.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.185+0000 7f3957e00640 1 -- 172.21.15.40:0/3066157638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395806f180 msgr2=0x7f3958067890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:07.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.185+0000 7f3957e00640 1 --2- 172.21.15.40:0/3066157638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395806f180 0x7f3958067890 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3950009d90 tx=0x7f395001c8e0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.185+0000 7f3957e00640 1 -- 172.21.15.40:0/3066157638 shutdown_connections 2024-03-24T00:32:07.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.185+0000 7f3957e00640 1 --2- 172.21.15.40:0/3066157638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395806f180 0x7f3958067890 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.185+0000 7f3957e00640 1 --2- 172.21.15.40:0/3066157638 >> v2:172.21.15.53:3300/0 conn(0x7f39580b41d0 0x7f3958162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.185+0000 7f3957e00640 1 -- 172.21.15.40:0/3066157638 >> 172.21.15.40:0/3066157638 conn(0x7f39580b27c0 msgr2=0x7f39580b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:07.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.185+0000 7f3957e00640 1 -- 172.21.15.40:0/3066157638 shutdown_connections 2024-03-24T00:32:07.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.185+0000 7f3957e00640 1 -- 172.21.15.40:0/3066157638 wait complete. 2024-03-24T00:32:07.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957e00640 1 Processor -- start 2024-03-24T00:32:07.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957e00640 1 -- start start 2024-03-24T00:32:07.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395806f180 0x7f3958157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:07.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39580b41d0 0x7f3958147740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:07.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f395815dc80 0x7f39580b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:07.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395815e9f0 con 0x7f395815dc80 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f395815e870 con 0x7f39580b41d0 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f395815eb70 con 0x7f395806f180 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f395815dc80 0x7f39580b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3956a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395806f180 0x7f3958157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f395815dc80 0x7f39580b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42334/0 (socket says 172.21.15.40:42334) 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957400640 1 -- 172.21.15.40:0/3591698194 learned_addr learned my addr 172.21.15.40:0/3591698194 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3956000640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39580b41d0 0x7f3958147740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957400640 1 -- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395806f180 msgr2=0x7f3958157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957400640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395806f180 0x7f3958157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957400640 1 -- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39580b41d0 msgr2=0x7f3958147740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957400640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39580b41d0 0x7f3958147740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957400640 1 -- 172.21.15.40:0/3591698194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3958158750 con 0x7f395815dc80 2024-03-24T00:32:07.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957400640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f395815dc80 0x7f39580b0400 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f39500b0040 tx=0x7f3950019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:07.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f394f400640 1 -- 172.21.15.40:0/3591698194 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39500b2050 con 0x7f395815dc80 2024-03-24T00:32:07.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f394f400640 1 -- 172.21.15.40:0/3591698194 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39500be070 con 0x7f395815dc80 2024-03-24T00:32:07.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957e00640 1 -- 172.21.15.40:0/3591698194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3958158a30 con 0x7f395815dc80 2024-03-24T00:32:07.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f394f400640 1 -- 172.21.15.40:0/3591698194 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39500ad890 con 0x7f395815dc80 2024-03-24T00:32:07.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.189+0000 7f3957e00640 1 -- 172.21.15.40:0/3591698194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39580cd7c0 con 0x7f395815dc80 2024-03-24T00:32:07.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.193+0000 7f3957e00640 1 -- 172.21.15.40:0/3591698194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39580735d0 con 0x7f395815dc80 2024-03-24T00:32:07.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.193+0000 7f394f400640 1 -- 172.21.15.40:0/3591698194 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f39500ada30 con 0x7f395815dc80 2024-03-24T00:32:07.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.193+0000 7f394f400640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3938067660 0x7f3938069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:07.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.193+0000 7f394f400640 1 -- 172.21.15.40:0/3591698194 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39501207e0 con 0x7f395815dc80 2024-03-24T00:32:07.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.193+0000 7f3956a00640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3938067660 0x7f3938069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:07.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.193+0000 7f3956a00640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3938067660 0x7f3938069b20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3948007fd0 tx=0x7f3948000bd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:07.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.197+0000 7f394f400640 1 -- 172.21.15.40:0/3591698194 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3950016560 con 0x7f395815dc80 2024-03-24T00:32:07.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.361+0000 7f3957e00640 1 -- 172.21.15.40:0/3591698194 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f395804f4f0 con 0x7f395815dc80 2024-03-24T00:32:07.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f394f400640 1 -- 172.21.15.40:0/3591698194 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16622 (secure 0 0 0) 0x7f39500f0e40 con 0x7f395815dc80 2024-03-24T00:32:07.369 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:07.370 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":32,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:32:06.027640+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:03.863519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:05.024936+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:32:07.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f3957e00640 1 -- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3938067660 msgr2=0x7f3938069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f3957e00640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3938067660 0x7f3938069b20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3948007fd0 tx=0x7f3948000bd0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f3957e00640 1 -- 172.21.15.40:0/3591698194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f395815dc80 msgr2=0x7f39580b0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f3957e00640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f395815dc80 0x7f39580b0400 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f39500b0040 tx=0x7f3950019680 comp rx=0 tx=0).stop 2024-03-24T00:32:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f3957e00640 1 -- 172.21.15.40:0/3591698194 shutdown_connections 2024-03-24T00:32:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f3957e00640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3938067660 0x7f3938069b20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f3957e00640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f395815dc80 0x7f39580b0400 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f3957e00640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39580b41d0 0x7f3958147740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f3957e00640 1 --2- 172.21.15.40:0/3591698194 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395806f180 0x7f3958157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f3957e00640 1 -- 172.21.15.40:0/3591698194 >> 172.21.15.40:0/3591698194 conn(0x7f39580b27c0 msgr2=0x7f395815fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f3957e00640 1 -- 172.21.15.40:0/3591698194 shutdown_connections 2024-03-24T00:32:07.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.365+0000 7f3957e00640 1 -- 172.21.15.40:0/3591698194 wait complete. 2024-03-24T00:32:07.392 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:32:07.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.529+0000 7f20ee600640 1 Processor -- start 2024-03-24T00:32:07.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.529+0000 7f20ee600640 1 -- start start 2024-03-24T00:32:07.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20ee600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f20e815df80 0x7f20e8070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:07.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20ee600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f20e80d2270 0x7f20e8161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:07.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20ee600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e806d0f0 con 0x7f20e806c950 2024-03-24T00:32:07.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20ee600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e806b890 con 0x7f20e80d2270 2024-03-24T00:32:07.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20ee600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e806b280 con 0x7f20e815df80 2024-03-24T00:32:07.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20e7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f20e80d2270 0x7f20e8161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:07.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20e7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f20e80d2270 0x7f20e8161ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42342/0 (socket says 172.21.15.40:42342) 2024-03-24T00:32:07.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20e7e00640 1 -- 172.21.15.40:0/223519071 learned_addr learned my addr 172.21.15.40:0/223519071 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:07.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20e7400640 1 --2- 172.21.15.40:0/223519071 >> v2:172.21.15.53:3300/0 conn(0x7f20e815df80 0x7f20e8070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:07.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20e7e00640 1 -- 172.21.15.40:0/223519071 >> v2:172.21.15.53:3300/0 conn(0x7f20e815df80 msgr2=0x7f20e8070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:07.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20e7e00640 1 --2- 172.21.15.40:0/223519071 >> v2:172.21.15.53:3300/0 conn(0x7f20e815df80 0x7f20e8070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20e7e00640 1 -- 172.21.15.40:0/223519071 >> v1:172.21.15.53:6789/0 conn(0x7f20e806c950 legacy=0x7f20e80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:07.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.533+0000 7f20e7e00640 1 -- 172.21.15.40:0/223519071 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e806ce10 con 0x7f20e80d2270 2024-03-24T00:32:07.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.537+0000 7f20e7e00640 1 --2- 172.21.15.40:0/223519071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e80d2270 0x7f20e8161ad0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f20e80d2c90 tx=0x7f20e001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca21fbf9d1d810f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.537+0000 7f20e6a00640 1 -- 172.21.15.40:0/223519071 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e00a5840 con 0x7f20e80d2270 2024-03-24T00:32:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.537+0000 7f20e6a00640 1 -- 172.21.15.40:0/223519071 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e00a59e0 con 0x7f20e80d2270 2024-03-24T00:32:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.537+0000 7f20e6a00640 1 -- 172.21.15.40:0/223519071 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e00a5db0 con 0x7f20e80d2270 2024-03-24T00:32:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.537+0000 7f20ee600640 1 -- 172.21.15.40:0/223519071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e80d2270 msgr2=0x7f20e8161ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:07.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.537+0000 7f20ee600640 1 --2- 172.21.15.40:0/223519071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e80d2270 0x7f20e8161ad0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f20e80d2c90 tx=0x7f20e001c9c0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.537+0000 7f20ee600640 1 -- 172.21.15.40:0/223519071 shutdown_connections 2024-03-24T00:32:07.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.537+0000 7f20ee600640 1 --2- 172.21.15.40:0/223519071 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e80d2270 0x7f20e8161ad0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.537+0000 7f20ee600640 1 --2- 172.21.15.40:0/223519071 >> v2:172.21.15.53:3300/0 conn(0x7f20e815df80 0x7f20e8070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.537+0000 7f20ee600640 1 -- 172.21.15.40:0/223519071 >> 172.21.15.40:0/223519071 conn(0x7f20e806ecf0 msgr2=0x7f20e806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:07.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.537+0000 7f20ee600640 1 -- 172.21.15.40:0/223519071 shutdown_connections 2024-03-24T00:32:07.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.537+0000 7f20ee600640 1 -- 172.21.15.40:0/223519071 wait complete. 2024-03-24T00:32:07.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20ee600640 1 Processor -- start 2024-03-24T00:32:07.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20ee600640 1 -- start start 2024-03-24T00:32:07.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20ee600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20e806c950 0x7f20e8151960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:07.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20ee600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e80d2270 0x7f20e80b1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:07.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20ee600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e815df80 0x7f20e80b1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:07.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20ee600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e816af90 con 0x7f20e815df80 2024-03-24T00:32:07.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20ee600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e816ae10 con 0x7f20e80d2270 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20ee600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e816b110 con 0x7f20e806c950 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e815df80 0x7f20e80b1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e815df80 0x7f20e80b1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42354/0 (socket says 172.21.15.40:42354) 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20e7e00640 1 -- 172.21.15.40:0/2466339588 learned_addr learned my addr 172.21.15.40:0/2466339588 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20e7400640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20e806c950 0x7f20e8151960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20dfe00640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e80d2270 0x7f20e80b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20e7e00640 1 -- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20e806c950 msgr2=0x7f20e8151960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20e7e00640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20e806c950 0x7f20e8151960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20e7e00640 1 -- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e80d2270 msgr2=0x7f20e80b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20e7e00640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e80d2270 0x7f20e80b1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20e7e00640 1 -- 172.21.15.40:0/2466339588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e80ada70 con 0x7f20e815df80 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20dfe00640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e80d2270 0x7f20e80b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20e7e00640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e815df80 0x7f20e80b1c70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f20e001a5f0 tx=0x7f20e0019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20df400640 1 -- 172.21.15.40:0/2466339588 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e00bc020 con 0x7f20e815df80 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20df400640 1 -- 172.21.15.40:0/2466339588 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e00b8670 con 0x7f20e815df80 2024-03-24T00:32:07.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20ee600640 1 -- 172.21.15.40:0/2466339588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e813e3c0 con 0x7f20e815df80 2024-03-24T00:32:07.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20df400640 1 -- 172.21.15.40:0/2466339588 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e00b8a40 con 0x7f20e815df80 2024-03-24T00:32:07.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.541+0000 7f20ee600640 1 -- 172.21.15.40:0/2466339588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e8151ea0 con 0x7f20e815df80 2024-03-24T00:32:07.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.545+0000 7f20ee600640 1 -- 172.21.15.40:0/2466339588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e8068170 con 0x7f20e815df80 2024-03-24T00:32:07.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.545+0000 7f20df400640 1 -- 172.21.15.40:0/2466339588 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f20e00bf020 con 0x7f20e815df80 2024-03-24T00:32:07.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.545+0000 7f20df400640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f20c4067660 0x7f20c4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:07.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.545+0000 7f20df400640 1 -- 172.21.15.40:0/2466339588 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20e0120a10 con 0x7f20e815df80 2024-03-24T00:32:07.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.545+0000 7f20e7400640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f20c4067660 0x7f20c4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:07.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.549+0000 7f20e7400640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f20c4067660 0x7f20c4069b20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f20e81fa470 tx=0x7f20d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:07.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.549+0000 7f20df400640 1 -- 172.21.15.40:0/2466339588 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f20e00c6030 con 0x7f20e815df80 2024-03-24T00:32:07.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:07.789+0000 7f20ee600640 1 -- 172.21.15.40:0/2466339588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f20e80cf9d0 con 0x7f20e815df80 2024-03-24T00:32:08.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.776+0000 7f20df400640 1 -- 172.21.15.40:0/2466339588 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f20e00165e0 con 0x7f20e815df80 2024-03-24T00:32:08.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.776+0000 7f20ee600640 1 -- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f20c4067660 msgr2=0x7f20c4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:08.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.776+0000 7f20ee600640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f20c4067660 0x7f20c4069b20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f20e81fa470 tx=0x7f20d8008040 comp rx=0 tx=0).stop 2024-03-24T00:32:08.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.776+0000 7f20ee600640 1 -- 172.21.15.40:0/2466339588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e815df80 msgr2=0x7f20e80b1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:08.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.776+0000 7f20ee600640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e815df80 0x7f20e80b1c70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f20e001a5f0 tx=0x7f20e0019760 comp rx=0 tx=0).stop 2024-03-24T00:32:08.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.776+0000 7f20ee600640 1 -- 172.21.15.40:0/2466339588 shutdown_connections 2024-03-24T00:32:08.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.776+0000 7f20ee600640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f20c4067660 0x7f20c4069b20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:08.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.776+0000 7f20ee600640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20e815df80 0x7f20e80b1c70 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.776+0000 7f20ee600640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20e80d2270 0x7f20e80b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.776+0000 7f20ee600640 1 --2- 172.21.15.40:0/2466339588 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f20e806c950 0x7f20e8151960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.776+0000 7f20ee600640 1 -- 172.21.15.40:0/2466339588 >> 172.21.15.40:0/2466339588 conn(0x7f20e806ecf0 msgr2=0x7f20e80d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.780+0000 7f20ee600640 1 -- 172.21.15.40:0/2466339588 shutdown_connections 2024-03-24T00:32:08.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.780+0000 7f20ee600640 1 -- 172.21.15.40:0/2466339588 wait complete. 2024-03-24T00:32:08.803 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:32:08.803 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:32:08.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.940+0000 7f120b800640 1 Processor -- start 2024-03-24T00:32:08.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.940+0000 7f120b800640 1 -- start start 2024-03-24T00:32:08.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f120b800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f120406dab0 0x7f12040ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:08.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f120b800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1204070ed0 con 0x7f12040aec70 2024-03-24T00:32:08.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f120b800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12040cf240 con 0x7f120415a560 2024-03-24T00:32:08.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f120b800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12040cf520 con 0x7f120406dab0 2024-03-24T00:32:08.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f1209800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f120406dab0 0x7f12040ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:08.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f1203e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f12040aec70 0x7f120406d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57898/0 (socket says 172.21.15.40:57898) 2024-03-24T00:32:08.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f1203e00640 1 -- 172.21.15.40:0/3130552301 learned_addr learned my addr 172.21.15.40:0/3130552301 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:08.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f1209800640 1 -- 172.21.15.40:0/3130552301 >> v1:172.21.15.53:6789/0 conn(0x7f120415a560 legacy=0x7f12040cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:08.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f1209800640 1 -- 172.21.15.40:0/3130552301 >> v1:172.21.15.40:6789/0 conn(0x7f12040aec70 legacy=0x7f120406d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:08.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f1203400640 1 -- 172.21.15.40:0/3130552301 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605976607 0 0) 0x7f1204070ed0 con 0x7f12040aec70 2024-03-24T00:32:08.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f1209800640 1 -- 172.21.15.40:0/3130552301 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12040b3aa0 con 0x7f120406dab0 2024-03-24T00:32:08.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f1209800640 1 --2- 172.21.15.40:0/3130552301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120406dab0 0x7f12040ae6b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009d90 tx=0x7f11fc01c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769a5930c4439d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:08.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f1203400640 1 -- 172.21.15.40:0/3130552301 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc0b2070 con 0x7f120406dab0 2024-03-24T00:32:08.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.944+0000 7f1203400640 1 -- 172.21.15.40:0/3130552301 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120415ab50 con 0x7f120406dab0 2024-03-24T00:32:08.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f1203400640 1 -- 172.21.15.40:0/3130552301 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc0a5b50 con 0x7f120406dab0 2024-03-24T00:32:08.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 -- 172.21.15.40:0/3130552301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120406dab0 msgr2=0x7f12040ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:08.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 --2- 172.21.15.40:0/3130552301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120406dab0 0x7f12040ae6b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009d90 tx=0x7f11fc01c850 comp rx=0 tx=0).stop 2024-03-24T00:32:08.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 -- 172.21.15.40:0/3130552301 shutdown_connections 2024-03-24T00:32:08.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 --2- 172.21.15.40:0/3130552301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120406dab0 0x7f12040ae6b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:08.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 -- 172.21.15.40:0/3130552301 >> 172.21.15.40:0/3130552301 conn(0x7f12040b1660 msgr2=0x7f12040b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:08.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 -- 172.21.15.40:0/3130552301 shutdown_connections 2024-03-24T00:32:08.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 -- 172.21.15.40:0/3130552301 wait complete. 2024-03-24T00:32:08.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 Processor -- start 2024-03-24T00:32:08.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 -- start start 2024-03-24T00:32:08.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f120406dab0 0x7f12041534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:08.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12040aec70 0x7f12041582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:08.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f120415a560 0x7f120414df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:08.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12040b4b60 con 0x7f12040aec70 2024-03-24T00:32:08.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12040b49e0 con 0x7f120415a560 2024-03-24T00:32:08.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f120b800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120406cb10 con 0x7f120406dab0 2024-03-24T00:32:08.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f1203e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12040aec70 0x7f12041582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:08.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f1203e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12040aec70 0x7f12041582a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42382/0 (socket says 172.21.15.40:42382) 2024-03-24T00:32:08.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f1203e00640 1 -- 172.21.15.40:0/1147394067 learned_addr learned my addr 172.21.15.40:0/1147394067 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:08.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f1208e00640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f120406dab0 0x7f12041534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:08.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.948+0000 7f1209800640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f120415a560 0x7f120414df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f1203e00640 1 -- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f120406dab0 msgr2=0x7f12041534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f1203e00640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f120406dab0 0x7f12041534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f1203e00640 1 -- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f120415a560 msgr2=0x7f120414df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f1203e00640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f120415a560 0x7f120414df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f1203e00640 1 -- 172.21.15.40:0/1147394067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1204153df0 con 0x7f12040aec70 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f1209800640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f120415a560 0x7f120414df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f1208e00640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f120406dab0 0x7f12041534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f1203e00640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12040aec70 0x7f12041582a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f11f000c4a0 tx=0x7f11f000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f1200c00640 1 -- 172.21.15.40:0/1147394067 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0018070 con 0x7f12040aec70 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f1200c00640 1 -- 172.21.15.40:0/1147394067 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f00074e0 con 0x7f12040aec70 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f120b800640 1 -- 172.21.15.40:0/1147394067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12041540d0 con 0x7f12040aec70 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f1200c00640 1 -- 172.21.15.40:0/1147394067 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f0004a30 con 0x7f12040aec70 2024-03-24T00:32:08.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f120b800640 1 -- 172.21.15.40:0/1147394067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1204154690 con 0x7f12040aec70 2024-03-24T00:32:08.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f120b800640 1 -- 172.21.15.40:0/1147394067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1204068170 con 0x7f12040aec70 2024-03-24T00:32:08.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.952+0000 7f1200c00640 1 -- 172.21.15.40:0/1147394067 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f11f001b070 con 0x7f12040aec70 2024-03-24T00:32:08.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.956+0000 7f1200c00640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11e4067660 0x7f11e4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:08.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.956+0000 7f1200c00640 1 -- 172.21.15.40:0/1147394067 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11f0084bd0 con 0x7f12040aec70 2024-03-24T00:32:08.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.956+0000 7f1208e00640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11e4067660 0x7f11e4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:08.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.956+0000 7f1208e00640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11e4067660 0x7f11e4069b20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f11f8000f70 tx=0x7f11f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:08.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:08.960+0000 7f1200c00640 1 -- 172.21.15.40:0/1147394067 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f11f0055840 con 0x7f12040aec70 2024-03-24T00:32:09.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.156+0000 7f120b800640 1 -- 172.21.15.40:0/1147394067 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1204071ba0 con 0x7f12040aec70 2024-03-24T00:32:09.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f1200c00640 1 -- 172.21.15.40:0/1147394067 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f11f005b020 con 0x7f12040aec70 2024-03-24T00:32:09.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f120b800640 1 -- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11e4067660 msgr2=0x7f11e4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:09.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f120b800640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11e4067660 0x7f11e4069b20 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f11f8000f70 tx=0x7f11f8009040 comp rx=0 tx=0).stop 2024-03-24T00:32:09.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f120b800640 1 -- 172.21.15.40:0/1147394067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12040aec70 msgr2=0x7f12041582a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:09.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f120b800640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12040aec70 0x7f12041582a0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f11f000c4a0 tx=0x7f11f000c960 comp rx=0 tx=0).stop 2024-03-24T00:32:09.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f120b800640 1 -- 172.21.15.40:0/1147394067 shutdown_connections 2024-03-24T00:32:09.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f120b800640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11e4067660 0x7f11e4069b20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:09.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f120b800640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f120415a560 0x7f120414df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:09.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f120b800640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12040aec70 0x7f12041582a0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:09.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f120b800640 1 --2- 172.21.15.40:0/1147394067 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f120406dab0 0x7f12041534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:09.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f120b800640 1 -- 172.21.15.40:0/1147394067 >> 172.21.15.40:0/1147394067 conn(0x7f12040b1660 msgr2=0x7f120415cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:09.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f120b800640 1 -- 172.21.15.40:0/1147394067 shutdown_connections 2024-03-24T00:32:09.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.784+0000 7f120b800640 1 -- 172.21.15.40:0/1147394067 wait complete. 2024-03-24T00:32:09.809 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:32:09.810 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:32:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.944+0000 7f4c2c800640 1 Processor -- start 2024-03-24T00:32:09.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.944+0000 7f4c2c800640 1 -- start start 2024-03-24T00:32:09.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.944+0000 7f4c2c800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c240cd470 0x7f4c24168fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:09.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.944+0000 7f4c2c800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240b2870 con 0x7f4c240b2490 2024-03-24T00:32:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.944+0000 7f4c2c800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240b3190 con 0x7f4c240b2aa0 2024-03-24T00:32:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.944+0000 7f4c2c800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2406b6d0 con 0x7f4c240cd470 2024-03-24T00:32:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.944+0000 7f4c29e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4c240b2490 0x7f4c240cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57904/0 (socket says 172.21.15.40:57904) 2024-03-24T00:32:09.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.944+0000 7f4c29e00640 1 -- 172.21.15.40:0/3799928740 learned_addr learned my addr 172.21.15.40:0/3799928740 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:09.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c2a800640 1 --2- 172.21.15.40:0/3799928740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c240cd470 0x7f4c24168fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:09.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c28a00640 1 -- 172.21.15.40:0/3799928740 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851632406 0 0) 0x7f4c240b2870 con 0x7f4c240b2490 2024-03-24T00:32:09.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c28a00640 1 -- 172.21.15.40:0/3799928740 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0c003530 con 0x7f4c240b2490 2024-03-24T00:32:09.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c28a00640 1 -- 172.21.15.40:0/3799928740 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199660473 0 0) 0x7f4c240b3190 con 0x7f4c240b2aa0 2024-03-24T00:32:09.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c28a00640 1 -- 172.21.15.40:0/3799928740 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c240b2870 con 0x7f4c240b2aa0 2024-03-24T00:32:09.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c2a800640 1 -- 172.21.15.40:0/3799928740 >> v1:172.21.15.53:6789/0 conn(0x7f4c240b2aa0 legacy=0x7f4c240d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:09.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c28a00640 1 -- 172.21.15.40:0/3799928740 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2269315905 0 0) 0x7f4c0c003530 con 0x7f4c240b2490 2024-03-24T00:32:09.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c2a800640 1 -- 172.21.15.40:0/3799928740 >> v1:172.21.15.40:6789/0 conn(0x7f4c240b2490 legacy=0x7f4c240cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:09.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c2a800640 1 -- 172.21.15.40:0/3799928740 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c240b3470 con 0x7f4c240cd470 2024-03-24T00:32:09.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c2a800640 1 --2- 172.21.15.40:0/3799928740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c240cd470 0x7f4c24168fb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4c20009dc0 tx=0x7f4c2001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3af443600940eb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:09.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c28a00640 1 -- 172.21.15.40:0/3799928740 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c200b2070 con 0x7f4c240cd470 2024-03-24T00:32:09.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c28a00640 1 -- 172.21.15.40:0/3799928740 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c200a5710 con 0x7f4c240cd470 2024-03-24T00:32:09.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.948+0000 7f4c28a00640 1 -- 172.21.15.40:0/3799928740 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c18001680 con 0x7f4c240cd470 2024-03-24T00:32:09.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 -- 172.21.15.40:0/3799928740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c240cd470 msgr2=0x7f4c24168fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:09.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 --2- 172.21.15.40:0/3799928740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c240cd470 0x7f4c24168fb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4c20009dc0 tx=0x7f4c2001c8b0 comp rx=0 tx=0).stop 2024-03-24T00:32:09.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 -- 172.21.15.40:0/3799928740 shutdown_connections 2024-03-24T00:32:09.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 --2- 172.21.15.40:0/3799928740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c240cd470 0x7f4c24168fb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:09.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 -- 172.21.15.40:0/3799928740 >> 172.21.15.40:0/3799928740 conn(0x7f4c240d1d10 msgr2=0x7f4c240d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:09.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 -- 172.21.15.40:0/3799928740 shutdown_connections 2024-03-24T00:32:09.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 -- 172.21.15.40:0/3799928740 wait complete. 2024-03-24T00:32:09.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 Processor -- start 2024-03-24T00:32:09.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 -- start start 2024-03-24T00:32:09.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c240b2490 0x7f4c2415d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:09.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c240b2aa0 0x7f4c240b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:09.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c240cd470 0x7f4c240b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:09.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2416c0a0 con 0x7f4c240b2aa0 2024-03-24T00:32:09.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2416bf20 con 0x7f4c240cd470 2024-03-24T00:32:09.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2c800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2416c220 con 0x7f4c240b2490 2024-03-24T00:32:09.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c29400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c240b2aa0 0x7f4c240b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:09.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c29400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c240b2aa0 0x7f4c240b6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42386/0 (socket says 172.21.15.40:42386) 2024-03-24T00:32:09.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c29400640 1 -- 172.21.15.40:0/2995000965 learned_addr learned my addr 172.21.15.40:0/2995000965 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:09.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c29e00640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c240b2490 0x7f4c2415d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:09.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c2a800640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c240cd470 0x7f4c240b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:09.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c29400640 1 -- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c240b2490 msgr2=0x7f4c2415d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:09.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c29400640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c240b2490 0x7f4c2415d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:09.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c29400640 1 -- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c240cd470 msgr2=0x7f4c240b6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:09.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c29400640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c240cd470 0x7f4c240b6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:09.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.952+0000 7f4c29400640 1 -- 172.21.15.40:0/2995000965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2415f9f0 con 0x7f4c240b2aa0 2024-03-24T00:32:09.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.956+0000 7f4c2a800640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c240cd470 0x7f4c240b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:09.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.956+0000 7f4c29400640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c240b2aa0 0x7f4c240b6a80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400c500 tx=0x7f4c1400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:09.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.956+0000 7f4c1e000640 1 -- 172.21.15.40:0/2995000965 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c140052d0 con 0x7f4c240b2aa0 2024-03-24T00:32:09.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.956+0000 7f4c1e000640 1 -- 172.21.15.40:0/2995000965 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c14005490 con 0x7f4c240b2aa0 2024-03-24T00:32:09.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.956+0000 7f4c2c800640 1 -- 172.21.15.40:0/2995000965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2415f710 con 0x7f4c240b2aa0 2024-03-24T00:32:09.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.956+0000 7f4c1e000640 1 -- 172.21.15.40:0/2995000965 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c14012850 con 0x7f4c240b2aa0 2024-03-24T00:32:09.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.956+0000 7f4c2c800640 1 -- 172.21.15.40:0/2995000965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c240b3b60 con 0x7f4c240b2aa0 2024-03-24T00:32:09.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.956+0000 7f4c2c800640 1 -- 172.21.15.40:0/2995000965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c24068170 con 0x7f4c240b2aa0 2024-03-24T00:32:09.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.956+0000 7f4c1e000640 1 -- 172.21.15.40:0/2995000965 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4c140129f0 con 0x7f4c240b2aa0 2024-03-24T00:32:09.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.960+0000 7f4c1e000640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4c08067660 0x7f4c08069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:09.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.960+0000 7f4c1e000640 1 -- 172.21.15.40:0/2995000965 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c14085670 con 0x7f4c240b2aa0 2024-03-24T00:32:09.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.960+0000 7f4c29e00640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4c08067660 0x7f4c08069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:09.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.960+0000 7f4c29e00640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4c08067660 0x7f4c08069b20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4c180011f0 tx=0x7f4c18009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:09.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:09.964+0000 7f4c1e000640 1 -- 172.21.15.40:0/2995000965 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4c14056260 con 0x7f4c240b2aa0 2024-03-24T00:32:10.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.164+0000 7f4c2c800640 1 -- 172.21.15.40:0/2995000965 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c2406ddc0 con 0x7f4c240b2aa0 2024-03-24T00:32:10.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.164+0000 7f4c1e000640 1 -- 172.21.15.40:0/2995000965 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f4c14055b60 con 0x7f4c240b2aa0 2024-03-24T00:32:10.171 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:10.171 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2069836835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2069836835},{"type":"v1","addr":"172.21.15.53:6836","nonce":2069836835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4856,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2159849028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":2159849028},{"type":"v1","addr":"172.21.15.53:6837","nonce":2159849028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2326157501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2326157501},{"type":"v1","addr":"172.21.15.40:6833","nonce":2326157501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:06.028086+0000","modified":"2024-03-24T00:32:09.162520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5229":{"gid":5229,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3050269371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3050269371},{"type":"v1","addr":"172.21.15.40:6835","nonce":3050269371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5229,"qdb_cluster":[5229]},"id":2}]} 2024-03-24T00:32:10.171 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 16 2024-03-24T00:32:10.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.168+0000 7f4c2c800640 1 -- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4c08067660 msgr2=0x7f4c08069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.168+0000 7f4c2c800640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4c08067660 0x7f4c08069b20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4c180011f0 tx=0x7f4c18009040 comp rx=0 tx=0).stop 2024-03-24T00:32:10.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.168+0000 7f4c2c800640 1 -- 172.21.15.40:0/2995000965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c240b2aa0 msgr2=0x7f4c240b6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.168+0000 7f4c2c800640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c240b2aa0 0x7f4c240b6a80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4c1400c500 tx=0x7f4c1400c9c0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.168+0000 7f4c2c800640 1 -- 172.21.15.40:0/2995000965 shutdown_connections 2024-03-24T00:32:10.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.168+0000 7f4c2c800640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4c08067660 0x7f4c08069b20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.168+0000 7f4c2c800640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c240cd470 0x7f4c240b6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.168+0000 7f4c2c800640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c240b2aa0 0x7f4c240b6a80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.168+0000 7f4c2c800640 1 --2- 172.21.15.40:0/2995000965 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c240b2490 0x7f4c2415d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.168+0000 7f4c2c800640 1 -- 172.21.15.40:0/2995000965 >> 172.21.15.40:0/2995000965 conn(0x7f4c240d1d10 msgr2=0x7f4c24070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:10.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.168+0000 7f4c2c800640 1 -- 172.21.15.40:0/2995000965 shutdown_connections 2024-03-24T00:32:10.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.168+0000 7f4c2c800640 1 -- 172.21.15.40:0/2995000965 wait complete. 2024-03-24T00:32:10.193 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:32:10.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dd600640 1 Processor -- start 2024-03-24T00:32:10.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dd600640 1 -- start start 2024-03-24T00:32:10.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dd600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f52d8160030 0x7f52d8070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:10.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dd600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52d806de00 0x7f52d8071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:10.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dd600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52d80cec10 con 0x7f52d8145520 2024-03-24T00:32:10.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dd600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d80ceeb0 con 0x7f52d806de00 2024-03-24T00:32:10.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dd600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d80cf190 con 0x7f52d8160030 2024-03-24T00:32:10.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dcc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52d806de00 0x7f52d8071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:10.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dcc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52d806de00 0x7f52d8071030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42388/0 (socket says 172.21.15.40:42388) 2024-03-24T00:32:10.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dcc00640 1 -- 172.21.15.40:0/3481617284 learned_addr learned my addr 172.21.15.40:0/3481617284 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:10.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52d7400640 1 --2- 172.21.15.40:0/3481617284 >> v2:172.21.15.53:3300/0 conn(0x7f52d8160030 0x7f52d8070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:10.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dcc00640 1 -- 172.21.15.40:0/3481617284 >> v2:172.21.15.53:3300/0 conn(0x7f52d8160030 msgr2=0x7f52d8070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:10.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dcc00640 1 --2- 172.21.15.40:0/3481617284 >> v2:172.21.15.53:3300/0 conn(0x7f52d8160030 0x7f52d8070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dcc00640 1 -- 172.21.15.40:0/3481617284 >> v1:172.21.15.53:6789/0 conn(0x7f52d8145520 legacy=0x7f52d815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.328+0000 7f52dcc00640 1 -- 172.21.15.40:0/3481617284 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d80cf470 con 0x7f52d806de00 2024-03-24T00:32:10.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52dcc00640 1 --2- 172.21.15.40:0/3481617284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52d806de00 0x7f52d8071030 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f52d806c4d0 tx=0x7f52d001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc66d15292d8c970 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:10.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52d6a00640 1 -- 172.21.15.40:0/3481617284 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d00a5840 con 0x7f52d806de00 2024-03-24T00:32:10.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52d6a00640 1 -- 172.21.15.40:0/3481617284 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d00a59e0 con 0x7f52d806de00 2024-03-24T00:32:10.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52d6a00640 1 -- 172.21.15.40:0/3481617284 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d00a5db0 con 0x7f52d806de00 2024-03-24T00:32:10.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52dd600640 1 -- 172.21.15.40:0/3481617284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52d806de00 msgr2=0x7f52d8071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52dd600640 1 --2- 172.21.15.40:0/3481617284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52d806de00 0x7f52d8071030 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f52d806c4d0 tx=0x7f52d001c960 comp rx=0 tx=0).stop 2024-03-24T00:32:10.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52dd600640 1 -- 172.21.15.40:0/3481617284 shutdown_connections 2024-03-24T00:32:10.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52dd600640 1 --2- 172.21.15.40:0/3481617284 >> v2:172.21.15.53:3300/0 conn(0x7f52d8160030 0x7f52d8070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52dd600640 1 --2- 172.21.15.40:0/3481617284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52d806de00 0x7f52d8071030 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52dd600640 1 -- 172.21.15.40:0/3481617284 >> 172.21.15.40:0/3481617284 conn(0x7f52d80b1740 msgr2=0x7f52d80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:10.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52dd600640 1 -- 172.21.15.40:0/3481617284 shutdown_connections 2024-03-24T00:32:10.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52dd600640 1 -- 172.21.15.40:0/3481617284 wait complete. 2024-03-24T00:32:10.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52dd600640 1 Processor -- start 2024-03-24T00:32:10.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.332+0000 7f52dd600640 1 -- start start 2024-03-24T00:32:10.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dd600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52d806de00 0x7f52d813ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:10.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dd600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52d8145520 0x7f52d81530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:10.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dd600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52d8160030 0x7f52d8158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:10.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dd600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d8145b10 con 0x7f52d8160030 2024-03-24T00:32:10.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dd600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d81607b0 con 0x7f52d8145520 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dd600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d8071d80 con 0x7f52d806de00 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dcc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52d8160030 0x7f52d8158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dcc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52d8160030 0x7f52d8158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42392/0 (socket says 172.21.15.40:42392) 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dcc00640 1 -- 172.21.15.40:0/124858978 learned_addr learned my addr 172.21.15.40:0/124858978 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52d7400640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52d8145520 0x7f52d81530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52d7e00640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52d806de00 0x7f52d813ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dcc00640 1 -- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52d806de00 msgr2=0x7f52d813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dcc00640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52d806de00 0x7f52d813ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dcc00640 1 -- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52d8145520 msgr2=0x7f52d81530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dcc00640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52d8145520 0x7f52d81530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dcc00640 1 -- 172.21.15.40:0/124858978 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d813b240 con 0x7f52d8160030 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dcc00640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52d8160030 0x7f52d8158340 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f52d00af040 tx=0x7f52d80cf960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52cfe00640 1 -- 172.21.15.40:0/124858978 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d00b8470 con 0x7f52d8160030 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52cfe00640 1 -- 172.21.15.40:0/124858978 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d00b8650 con 0x7f52d8160030 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dd600640 1 -- 172.21.15.40:0/124858978 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d8157580 con 0x7f52d8160030 2024-03-24T00:32:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52cfe00640 1 -- 172.21.15.40:0/124858978 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d00b8a40 con 0x7f52d8160030 2024-03-24T00:32:10.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dd600640 1 -- 172.21.15.40:0/124858978 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d815a3a0 con 0x7f52d8160030 2024-03-24T00:32:10.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.336+0000 7f52dd600640 1 -- 172.21.15.40:0/124858978 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d8073540 con 0x7f52d8160030 2024-03-24T00:32:10.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.340+0000 7f52cfe00640 1 -- 172.21.15.40:0/124858978 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f52d00c0020 con 0x7f52d8160030 2024-03-24T00:32:10.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.340+0000 7f52cfe00640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f52b8067690 0x7f52b8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:10.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.340+0000 7f52cfe00640 1 -- 172.21.15.40:0/124858978 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52d0120310 con 0x7f52d8160030 2024-03-24T00:32:10.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.340+0000 7f52d7e00640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f52b8067690 0x7f52b8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:10.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.340+0000 7f52d7e00640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f52b8067690 0x7f52b8069b50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f52c8001810 tx=0x7f52c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.344+0000 7f52cfe00640 1 -- 172.21.15.40:0/124858978 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f52d00165e0 con 0x7f52d8160030 2024-03-24T00:32:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.572+0000 7f52dd600640 1 -- 172.21.15.40:0/124858978 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52d804f4f0 con 0x7f52d8160030 2024-03-24T00:32:10.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.572+0000 7f52cfe00640 1 -- 172.21.15.40:0/124858978 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f52d00f0810 con 0x7f52d8160030 2024-03-24T00:32:10.579 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:10.579 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":33,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:32:07.623938+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:03.863519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:05.024936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:32:10.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.576+0000 7f52dd600640 1 -- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f52b8067690 msgr2=0x7f52b8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.576+0000 7f52dd600640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f52b8067690 0x7f52b8069b50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f52c8001810 tx=0x7f52c8009040 comp rx=0 tx=0).stop 2024-03-24T00:32:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.576+0000 7f52dd600640 1 -- 172.21.15.40:0/124858978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52d8160030 msgr2=0x7f52d8158340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.576+0000 7f52dd600640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52d8160030 0x7f52d8158340 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f52d00af040 tx=0x7f52d80cf960 comp rx=0 tx=0).stop 2024-03-24T00:32:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.576+0000 7f52dd600640 1 -- 172.21.15.40:0/124858978 shutdown_connections 2024-03-24T00:32:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.576+0000 7f52dd600640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f52b8067690 0x7f52b8069b50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.576+0000 7f52dd600640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52d8160030 0x7f52d8158340 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.576+0000 7f52dd600640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52d8145520 0x7f52d81530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.576+0000 7f52dd600640 1 --2- 172.21.15.40:0/124858978 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52d806de00 0x7f52d813ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.576+0000 7f52dd600640 1 -- 172.21.15.40:0/124858978 >> 172.21.15.40:0/124858978 conn(0x7f52d80b1740 msgr2=0x7f52d8154cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.576+0000 7f52dd600640 1 -- 172.21.15.40:0/124858978 shutdown_connections 2024-03-24T00:32:10.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.576+0000 7f52dd600640 1 -- 172.21.15.40:0/124858978 wait complete. 2024-03-24T00:32:10.600 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:32:10.600 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:32:10.601 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-03-24T00:32:10.596+0000 7f3bb5000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:32:10.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f069ec00640 1 Processor -- start 2024-03-24T00:32:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f069ec00640 1 -- start start 2024-03-24T00:32:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f069ec00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f06980d03f0 0x7f06980b2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:10.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f069ec00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06980b2ee0 con 0x7f069815f7e0 2024-03-24T00:32:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f069ec00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06980af570 con 0x7f06980d0010 2024-03-24T00:32:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f069ec00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06980af850 con 0x7f06980d03f0 2024-03-24T00:32:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f069cc00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f069815f7e0 0x7f0698165d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57916/0 (socket says 172.21.15.40:57916) 2024-03-24T00:32:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f069cc00640 1 -- 172.21.15.40:0/2355030185 learned_addr learned my addr 172.21.15.40:0/2355030185 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f0697400640 1 --2- 172.21.15.40:0/2355030185 >> v2:172.21.15.53:3300/0 conn(0x7f06980d03f0 0x7f06980b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f0696a00640 1 -- 172.21.15.40:0/2355030185 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411915861 0 0) 0x7f06980b2ee0 con 0x7f069815f7e0 2024-03-24T00:32:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f0696a00640 1 -- 172.21.15.40:0/2355030185 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f067c003530 con 0x7f069815f7e0 2024-03-24T00:32:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f0697400640 1 -- 172.21.15.40:0/2355030185 >> v1:172.21.15.53:6789/0 conn(0x7f06980d0010 legacy=0x7f069815f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f0697400640 1 -- 172.21.15.40:0/2355030185 >> v1:172.21.15.40:6789/0 conn(0x7f069815f7e0 legacy=0x7f0698165d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.736+0000 7f0697400640 1 -- 172.21.15.40:0/2355030185 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06980afb30 con 0x7f06980d03f0 2024-03-24T00:32:10.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.740+0000 7f0697400640 1 --2- 172.21.15.40:0/2355030185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06980d03f0 0x7f06980b2900 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f06840097c0 tx=0x7f068401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e62daff3ea6284e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:10.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.740+0000 7f0696a00640 1 -- 172.21.15.40:0/2355030185 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06840b0070 con 0x7f06980d03f0 2024-03-24T00:32:10.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.740+0000 7f0696a00640 1 -- 172.21.15.40:0/2355030185 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06840ab600 con 0x7f06980d03f0 2024-03-24T00:32:10.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.740+0000 7f0696a00640 1 -- 172.21.15.40:0/2355030185 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06840ab9d0 con 0x7f06980d03f0 2024-03-24T00:32:10.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.740+0000 7f069ec00640 1 -- 172.21.15.40:0/2355030185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06980d03f0 msgr2=0x7f06980b2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.740+0000 7f069ec00640 1 --2- 172.21.15.40:0/2355030185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06980d03f0 0x7f06980b2900 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f06840097c0 tx=0x7f068401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.740+0000 7f069ec00640 1 -- 172.21.15.40:0/2355030185 shutdown_connections 2024-03-24T00:32:10.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.740+0000 7f069ec00640 1 --2- 172.21.15.40:0/2355030185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06980d03f0 0x7f06980b2900 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.740+0000 7f069ec00640 1 -- 172.21.15.40:0/2355030185 >> 172.21.15.40:0/2355030185 conn(0x7f06980b38f0 msgr2=0x7f06980b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:10.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.740+0000 7f069ec00640 1 -- 172.21.15.40:0/2355030185 shutdown_connections 2024-03-24T00:32:10.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.740+0000 7f069ec00640 1 -- 172.21.15.40:0/2355030185 wait complete. 2024-03-24T00:32:10.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f069ec00640 1 Processor -- start 2024-03-24T00:32:10.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f069ec00640 1 -- start start 2024-03-24T00:32:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f069ec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06980d0010 0x7f069815b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f069ec00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06980d03f0 0x7f0698164090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f069ec00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f069815f7e0 0x7f069806f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f069ec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069816a060 con 0x7f06980d0010 2024-03-24T00:32:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f069ec00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0698169ee0 con 0x7f06980d03f0 2024-03-24T00:32:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f069ec00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069816a1e0 con 0x7f069815f7e0 2024-03-24T00:32:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f0697e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06980d0010 0x7f069815b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f0697e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06980d0010 0x7f069815b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42400/0 (socket says 172.21.15.40:42400) 2024-03-24T00:32:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f0697e00640 1 -- 172.21.15.40:0/2442717987 learned_addr learned my addr 172.21.15.40:0/2442717987 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f0697400640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06980d03f0 0x7f0698164090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f069cc00640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f069815f7e0 0x7f069806f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f0697e00640 1 -- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f069815f7e0 msgr2=0x7f069806f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f0697e00640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f069815f7e0 0x7f069806f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f0697e00640 1 -- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06980d03f0 msgr2=0x7f0698164090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f0697e00640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06980d03f0 0x7f0698164090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f0697e00640 1 -- 172.21.15.40:0/2442717987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069806b810 con 0x7f06980d0010 2024-03-24T00:32:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f069cc00640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f069815f7e0 0x7f069806f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f0697e00640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06980d0010 0x7f069815b1f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f068800c500 tx=0x7f068800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f068fe00640 1 -- 172.21.15.40:0/2442717987 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688016020 con 0x7f06980d0010 2024-03-24T00:32:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f068fe00640 1 -- 172.21.15.40:0/2442717987 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0688005220 con 0x7f06980d0010 2024-03-24T00:32:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f068fe00640 1 -- 172.21.15.40:0/2442717987 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0688007030 con 0x7f06980d0010 2024-03-24T00:32:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.744+0000 7f069ec00640 1 -- 172.21.15.40:0/2442717987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069806baf0 con 0x7f06980d0010 2024-03-24T00:32:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.748+0000 7f069ec00640 1 -- 172.21.15.40:0/2442717987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069806c0b0 con 0x7f06980d0010 2024-03-24T00:32:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.748+0000 7f069ec00640 1 -- 172.21.15.40:0/2442717987 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0698068170 con 0x7f06980d0010 2024-03-24T00:32:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.748+0000 7f068fe00640 1 -- 172.21.15.40:0/2442717987 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f06880071d0 con 0x7f06980d0010 2024-03-24T00:32:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.748+0000 7f068fe00640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0678067660 0x7f0678069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.748+0000 7f068fe00640 1 -- 172.21.15.40:0/2442717987 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f06880852c0 con 0x7f06980d0010 2024-03-24T00:32:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.748+0000 7f0697400640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0678067660 0x7f0678069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.748+0000 7f0697400640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0678067660 0x7f0678069b20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f06840ad000 tx=0x7f06840a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.752+0000 7f068fe00640 1 -- 172.21.15.40:0/2442717987 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0688020020 con 0x7f06980d0010 2024-03-24T00:32:10.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.912+0000 7f069ec00640 1 -- 172.21.15.40:0/2442717987 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f06980cc2b0 con 0x7f0678067660 2024-03-24T00:32:10.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.920+0000 7f068fe00640 1 -- 172.21.15.40:0/2442717987 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251931 (secure 0 0 0) 0x7f06980cc2b0 con 0x7f0678067660 2024-03-24T00:32:10.930 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:10.935 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:32:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.924+0000 7f069ec00640 1 -- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0678067660 msgr2=0x7f0678069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.924+0000 7f069ec00640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0678067660 0x7f0678069b20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f06840ad000 tx=0x7f06840a6000 comp rx=0 tx=0).stop 2024-03-24T00:32:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.924+0000 7f069ec00640 1 -- 172.21.15.40:0/2442717987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06980d0010 msgr2=0x7f069815b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.924+0000 7f069ec00640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06980d0010 0x7f069815b1f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f068800c500 tx=0x7f068800c9c0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.924+0000 7f069ec00640 1 -- 172.21.15.40:0/2442717987 shutdown_connections 2024-03-24T00:32:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.924+0000 7f069ec00640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0678067660 0x7f0678069b20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.924+0000 7f069ec00640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f069815f7e0 0x7f069806f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.924+0000 7f069ec00640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06980d03f0 0x7f0698164090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.924+0000 7f069ec00640 1 --2- 172.21.15.40:0/2442717987 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f06980d0010 0x7f069815b1f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.924+0000 7f069ec00640 1 -- 172.21.15.40:0/2442717987 >> 172.21.15.40:0/2442717987 conn(0x7f06980b38f0 msgr2=0x7f06980b1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.924+0000 7f069ec00640 1 -- 172.21.15.40:0/2442717987 shutdown_connections 2024-03-24T00:32:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:10.924+0000 7f069ec00640 1 -- 172.21.15.40:0/2442717987 wait complete. 2024-03-24T00:32:10.950 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-03-24T00:32:09.213351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236908,"kb_used_data":4088,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737812,"statfs":{"total":773094113280,"available":772851519488,"internally_reserved":0,"allocated":4186112,"data_stored":2333540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.591736"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.661362+0000","last_change":"2024-03-24T00:32:06.655127+0000","last_active":"2024-03-24T00:32:06.661362+0000","last_peered":"2024-03-24T00:32:06.661362+0000","last_clean":"2024-03-24T00:32:06.661362+0000","last_became_active":"2024-03-24T00:32:06.654824+0000","last_became_peered":"2024-03-24T00:32:06.654824+0000","last_unstale":"2024-03-24T00:32:06.661362+0000","last_undegraded":"2024-03-24T00:32:06.661362+0000","last_fullsized":"2024-03-24T00:32:06.661362+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:52:15.301289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.673987+0000","last_change":"2024-03-24T00:32:06.669569+0000","last_active":"2024-03-24T00:32:06.673987+0000","last_peered":"2024-03-24T00:32:06.673987+0000","last_clean":"2024-03-24T00:32:06.673987+0000","last_became_active":"2024-03-24T00:32:06.669204+0000","last_became_peered":"2024-03-24T00:32:06.669204+0000","last_unstale":"2024-03-24T00:32:06.673987+0000","last_undegraded":"2024-03-24T00:32:06.673987+0000","last_fullsized":"2024-03-24T00:32:06.673987+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:16:50.534597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730447+0000","last_change":"2024-03-24T00:32:06.666464+0000","last_active":"2024-03-24T00:32:07.730447+0000","last_peered":"2024-03-24T00:32:07.730447+0000","last_clean":"2024-03-24T00:32:07.730447+0000","last_became_active":"2024-03-24T00:32:06.666168+0000","last_became_peered":"2024-03-24T00:32:06.666168+0000","last_unstale":"2024-03-24T00:32:07.730447+0000","last_undegraded":"2024-03-24T00:32:07.730447+0000","last_fullsized":"2024-03-24T00:32:07.730447+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:33:54.537030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632533+0000","last_change":"2024-03-24T00:32:06.653589+0000","last_active":"2024-03-24T00:32:07.632533+0000","last_peered":"2024-03-24T00:32:07.632533+0000","last_clean":"2024-03-24T00:32:07.632533+0000","last_became_active":"2024-03-24T00:32:06.652742+0000","last_became_peered":"2024-03-24T00:32:06.652742+0000","last_unstale":"2024-03-24T00:32:07.632533+0000","last_undegraded":"2024-03-24T00:32:07.632533+0000","last_fullsized":"2024-03-24T00:32:07.632533+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:20.760035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.668412+0000","last_change":"2024-03-24T00:32:06.653736+0000","last_active":"2024-03-24T00:32:06.668412+0000","last_peered":"2024-03-24T00:32:06.668412+0000","last_clean":"2024-03-24T00:32:06.668412+0000","last_became_active":"2024-03-24T00:32:06.652654+0000","last_became_peered":"2024-03-24T00:32:06.652654+0000","last_unstale":"2024-03-24T00:32:06.668412+0000","last_undegraded":"2024-03-24T00:32:06.668412+0000","last_fullsized":"2024-03-24T00:32:06.668412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:04:48.930920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.673651+0000","last_change":"2024-03-24T00:32:06.660546+0000","last_active":"2024-03-24T00:32:06.673651+0000","last_peered":"2024-03-24T00:32:06.673651+0000","last_clean":"2024-03-24T00:32:06.673651+0000","last_became_active":"2024-03-24T00:32:06.656257+0000","last_became_peered":"2024-03-24T00:32:06.656257+0000","last_unstale":"2024-03-24T00:32:06.673651+0000","last_undegraded":"2024-03-24T00:32:06.673651+0000","last_fullsized":"2024-03-24T00:32:06.673651+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:53.548614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.663568+0000","last_change":"2024-03-24T00:32:06.657598+0000","last_active":"2024-03-24T00:32:06.663568+0000","last_peered":"2024-03-24T00:32:06.663568+0000","last_clean":"2024-03-24T00:32:06.663568+0000","last_became_active":"2024-03-24T00:32:06.657435+0000","last_became_peered":"2024-03-24T00:32:06.657435+0000","last_unstale":"2024-03-24T00:32:06.663568+0000","last_undegraded":"2024-03-24T00:32:06.663568+0000","last_fullsized":"2024-03-24T00:32:06.663568+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:08.356624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.657233+0000","last_change":"2024-03-24T00:32:06.651349+0000","last_active":"2024-03-24T00:32:06.657233+0000","last_peered":"2024-03-24T00:32:06.657233+0000","last_clean":"2024-03-24T00:32:06.657233+0000","last_became_active":"2024-03-24T00:32:06.648918+0000","last_became_peered":"2024-03-24T00:32:06.648918+0000","last_unstale":"2024-03-24T00:32:06.657233+0000","last_undegraded":"2024-03-24T00:32:06.657233+0000","last_fullsized":"2024-03-24T00:32:06.657233+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:01.983342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634535+0000","last_change":"2024-03-24T00:32:06.652465+0000","last_active":"2024-03-24T00:32:07.634535+0000","last_peered":"2024-03-24T00:32:07.634535+0000","last_clean":"2024-03-24T00:32:07.634535+0000","last_became_active":"2024-03-24T00:32:06.652195+0000","last_became_peered":"2024-03-24T00:32:06.652195+0000","last_unstale":"2024-03-24T00:32:07.634535+0000","last_undegraded":"2024-03-24T00:32:07.634535+0000","last_fullsized":"2024-03-24T00:32:07.634535+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:17.608787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634260+0000","last_change":"2024-03-24T00:32:06.670893+0000","last_active":"2024-03-24T00:32:07.634260+0000","last_peered":"2024-03-24T00:32:07.634260+0000","last_clean":"2024-03-24T00:32:07.634260+0000","last_became_active":"2024-03-24T00:32:06.670633+0000","last_became_peered":"2024-03-24T00:32:06.670633+0000","last_unstale":"2024-03-24T00:32:07.634260+0000","last_undegraded":"2024-03-24T00:32:07.634260+0000","last_fullsized":"2024-03-24T00:32:07.634260+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:42:24.998068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.666212+0000","last_change":"2024-03-24T00:32:06.664702+0000","last_active":"2024-03-24T00:32:06.666212+0000","last_peered":"2024-03-24T00:32:06.666212+0000","last_clean":"2024-03-24T00:32:06.666212+0000","last_became_active":"2024-03-24T00:32:06.664482+0000","last_became_peered":"2024-03-24T00:32:06.664482+0000","last_unstale":"2024-03-24T00:32:06.666212+0000","last_undegraded":"2024-03-24T00:32:06.666212+0000","last_fullsized":"2024-03-24T00:32:06.666212+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:42:24.998068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730610+0000","last_change":"2024-03-24T00:32:06.652174+0000","last_active":"2024-03-24T00:32:07.730610+0000","last_peered":"2024-03-24T00:32:07.730610+0000","last_clean":"2024-03-24T00:32:07.730610+0000","last_became_active":"2024-03-24T00:32:06.651473+0000","last_became_peered":"2024-03-24T00:32:06.651473+0000","last_unstale":"2024-03-24T00:32:07.730610+0000","last_undegraded":"2024-03-24T00:32:07.730610+0000","last_fullsized":"2024-03-24T00:32:07.730610+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:11.652686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.670845+0000","last_change":"2024-03-24T00:32:06.652928+0000","last_active":"2024-03-24T00:32:06.670845+0000","last_peered":"2024-03-24T00:32:06.670845+0000","last_clean":"2024-03-24T00:32:06.670845+0000","last_became_active":"2024-03-24T00:32:06.652412+0000","last_became_peered":"2024-03-24T00:32:06.652412+0000","last_unstale":"2024-03-24T00:32:06.670845+0000","last_undegraded":"2024-03-24T00:32:06.670845+0000","last_fullsized":"2024-03-24T00:32:06.670845+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:01.596065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.668677+0000","last_change":"2024-03-24T00:32:06.650673+0000","last_active":"2024-03-24T00:32:06.668677+0000","last_peered":"2024-03-24T00:32:06.668677+0000","last_clean":"2024-03-24T00:32:06.668677+0000","last_became_active":"2024-03-24T00:32:06.650341+0000","last_became_peered":"2024-03-24T00:32:06.650341+0000","last_unstale":"2024-03-24T00:32:06.668677+0000","last_undegraded":"2024-03-24T00:32:06.668677+0000","last_fullsized":"2024-03-24T00:32:06.668677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:40.821676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730244+0000","last_change":"2024-03-24T00:32:06.662635+0000","last_active":"2024-03-24T00:32:07.730244+0000","last_peered":"2024-03-24T00:32:07.730244+0000","last_clean":"2024-03-24T00:32:07.730244+0000","last_became_active":"2024-03-24T00:32:06.662368+0000","last_became_peered":"2024-03-24T00:32:06.662368+0000","last_unstale":"2024-03-24T00:32:07.730244+0000","last_undegraded":"2024-03-24T00:32:07.730244+0000","last_fullsized":"2024-03-24T00:32:07.730244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:01.853252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.663831+0000","last_change":"2024-03-24T00:32:06.658884+0000","last_active":"2024-03-24T00:32:06.663831+0000","last_peered":"2024-03-24T00:32:06.663831+0000","last_clean":"2024-03-24T00:32:06.663831+0000","last_became_active":"2024-03-24T00:32:06.656948+0000","last_became_peered":"2024-03-24T00:32:06.656948+0000","last_unstale":"2024-03-24T00:32:06.663831+0000","last_undegraded":"2024-03-24T00:32:06.663831+0000","last_fullsized":"2024-03-24T00:32:06.663831+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:20.760035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.653929+0000","last_change":"2024-03-24T00:32:06.648322+0000","last_active":"2024-03-24T00:32:06.653929+0000","last_peered":"2024-03-24T00:32:06.653929+0000","last_clean":"2024-03-24T00:32:06.653929+0000","last_became_active":"2024-03-24T00:32:06.647476+0000","last_became_peered":"2024-03-24T00:32:06.647476+0000","last_unstale":"2024-03-24T00:32:06.653929+0000","last_undegraded":"2024-03-24T00:32:06.653929+0000","last_fullsized":"2024-03-24T00:32:06.653929+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:58:22.347657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.672850+0000","last_change":"2024-03-24T00:32:06.652888+0000","last_active":"2024-03-24T00:32:06.672850+0000","last_peered":"2024-03-24T00:32:06.672850+0000","last_clean":"2024-03-24T00:32:06.672850+0000","last_became_active":"2024-03-24T00:32:06.652584+0000","last_became_peered":"2024-03-24T00:32:06.652584+0000","last_unstale":"2024-03-24T00:32:06.672850+0000","last_undegraded":"2024-03-24T00:32:06.672850+0000","last_fullsized":"2024-03-24T00:32:06.672850+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:34.123016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.673883+0000","last_change":"2024-03-24T00:32:06.669134+0000","last_active":"2024-03-24T00:32:06.673883+0000","last_peered":"2024-03-24T00:32:06.673883+0000","last_clean":"2024-03-24T00:32:06.673883+0000","last_became_active":"2024-03-24T00:32:06.668860+0000","last_became_peered":"2024-03-24T00:32:06.668860+0000","last_unstale":"2024-03-24T00:32:06.673883+0000","last_undegraded":"2024-03-24T00:32:06.673883+0000","last_fullsized":"2024-03-24T00:32:06.673883+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:01.853252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.668641+0000","last_change":"2024-03-24T00:32:06.657486+0000","last_active":"2024-03-24T00:32:06.668641+0000","last_peered":"2024-03-24T00:32:06.668641+0000","last_clean":"2024-03-24T00:32:06.668641+0000","last_became_active":"2024-03-24T00:32:06.656627+0000","last_became_peered":"2024-03-24T00:32:06.656627+0000","last_unstale":"2024-03-24T00:32:06.668641+0000","last_undegraded":"2024-03-24T00:32:06.668641+0000","last_fullsized":"2024-03-24T00:32:06.668641+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:17:26.737965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634065+0000","last_change":"2024-03-24T00:32:06.647201+0000","last_active":"2024-03-24T00:32:07.634065+0000","last_peered":"2024-03-24T00:32:07.634065+0000","last_clean":"2024-03-24T00:32:07.634065+0000","last_became_active":"2024-03-24T00:32:06.646847+0000","last_became_peered":"2024-03-24T00:32:06.646847+0000","last_unstale":"2024-03-24T00:32:07.634065+0000","last_undegraded":"2024-03-24T00:32:07.634065+0000","last_fullsized":"2024-03-24T00:32:07.634065+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:20.760035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633866+0000","last_change":"2024-03-24T00:32:06.636021+0000","last_active":"2024-03-24T00:32:07.633866+0000","last_peered":"2024-03-24T00:32:07.633866+0000","last_clean":"2024-03-24T00:32:07.633866+0000","last_became_active":"2024-03-24T00:32:06.635592+0000","last_became_peered":"2024-03-24T00:32:06.635592+0000","last_unstale":"2024-03-24T00:32:07.633866+0000","last_undegraded":"2024-03-24T00:32:07.633866+0000","last_fullsized":"2024-03-24T00:32:07.633866+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:42.860138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632682+0000","last_change":"2024-03-24T00:32:06.644546+0000","last_active":"2024-03-24T00:32:07.632682+0000","last_peered":"2024-03-24T00:32:07.632682+0000","last_clean":"2024-03-24T00:32:07.632682+0000","last_became_active":"2024-03-24T00:32:06.643991+0000","last_became_peered":"2024-03-24T00:32:06.643991+0000","last_unstale":"2024-03-24T00:32:07.632682+0000","last_undegraded":"2024-03-24T00:32:07.632682+0000","last_fullsized":"2024-03-24T00:32:07.632682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:58:22.347657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730210+0000","last_change":"2024-03-24T00:32:06.652544+0000","last_active":"2024-03-24T00:32:07.730210+0000","last_peered":"2024-03-24T00:32:07.730210+0000","last_clean":"2024-03-24T00:32:07.730210+0000","last_became_active":"2024-03-24T00:32:06.652197+0000","last_became_peered":"2024-03-24T00:32:06.652197+0000","last_unstale":"2024-03-24T00:32:07.730210+0000","last_undegraded":"2024-03-24T00:32:07.730210+0000","last_fullsized":"2024-03-24T00:32:07.730210+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:54.558428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634305+0000","last_change":"2024-03-24T00:32:06.654101+0000","last_active":"2024-03-24T00:32:07.634305+0000","last_peered":"2024-03-24T00:32:07.634305+0000","last_clean":"2024-03-24T00:32:07.634305+0000","last_became_active":"2024-03-24T00:32:06.651772+0000","last_became_peered":"2024-03-24T00:32:06.651772+0000","last_unstale":"2024-03-24T00:32:07.634305+0000","last_undegraded":"2024-03-24T00:32:07.634305+0000","last_fullsized":"2024-03-24T00:32:07.634305+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:34.123016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.652764+0000","last_change":"2024-03-24T00:32:06.642960+0000","last_active":"2024-03-24T00:32:06.652764+0000","last_peered":"2024-03-24T00:32:06.652764+0000","last_clean":"2024-03-24T00:32:06.652764+0000","last_became_active":"2024-03-24T00:32:06.642430+0000","last_became_peered":"2024-03-24T00:32:06.642430+0000","last_unstale":"2024-03-24T00:32:06.652764+0000","last_undegraded":"2024-03-24T00:32:06.652764+0000","last_fullsized":"2024-03-24T00:32:06.652764+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:20.760035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634285+0000","last_change":"2024-03-24T00:32:06.661594+0000","last_active":"2024-03-24T00:32:07.634285+0000","last_peered":"2024-03-24T00:32:07.634285+0000","last_clean":"2024-03-24T00:32:07.634285+0000","last_became_active":"2024-03-24T00:32:06.661038+0000","last_became_peered":"2024-03-24T00:32:06.661038+0000","last_unstale":"2024-03-24T00:32:07.634285+0000","last_undegraded":"2024-03-24T00:32:07.634285+0000","last_fullsized":"2024-03-24T00:32:07.634285+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:17.608787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730571+0000","last_change":"2024-03-24T00:32:06.652994+0000","last_active":"2024-03-24T00:32:07.730571+0000","last_peered":"2024-03-24T00:32:07.730571+0000","last_clean":"2024-03-24T00:32:07.730571+0000","last_became_active":"2024-03-24T00:32:06.652700+0000","last_became_peered":"2024-03-24T00:32:06.652700+0000","last_unstale":"2024-03-24T00:32:07.730571+0000","last_undegraded":"2024-03-24T00:32:07.730571+0000","last_fullsized":"2024-03-24T00:32:07.730571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:45.164296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633901+0000","last_change":"2024-03-24T00:32:06.636883+0000","last_active":"2024-03-24T00:32:07.633901+0000","last_peered":"2024-03-24T00:32:07.633901+0000","last_clean":"2024-03-24T00:32:07.633901+0000","last_became_active":"2024-03-24T00:32:06.636585+0000","last_became_peered":"2024-03-24T00:32:06.636585+0000","last_unstale":"2024-03-24T00:32:07.633901+0000","last_undegraded":"2024-03-24T00:32:07.633901+0000","last_fullsized":"2024-03-24T00:32:07.633901+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:08.356624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.656694+0000","last_change":"2024-03-24T00:32:06.649380+0000","last_active":"2024-03-24T00:32:06.656694+0000","last_peered":"2024-03-24T00:32:06.656694+0000","last_clean":"2024-03-24T00:32:06.656694+0000","last_became_active":"2024-03-24T00:32:06.649043+0000","last_became_peered":"2024-03-24T00:32:06.649043+0000","last_unstale":"2024-03-24T00:32:06.656694+0000","last_undegraded":"2024-03-24T00:32:06.656694+0000","last_fullsized":"2024-03-24T00:32:06.656694+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:43:57.884539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.661567+0000","last_change":"2024-03-24T00:32:06.656547+0000","last_active":"2024-03-24T00:32:06.661567+0000","last_peered":"2024-03-24T00:32:06.661567+0000","last_clean":"2024-03-24T00:32:06.661567+0000","last_became_active":"2024-03-24T00:32:06.655224+0000","last_became_peered":"2024-03-24T00:32:06.655224+0000","last_unstale":"2024-03-24T00:32:06.661567+0000","last_undegraded":"2024-03-24T00:32:06.661567+0000","last_fullsized":"2024-03-24T00:32:06.661567+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:52.375487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.638870+0000","last_change":"2024-03-24T00:32:06.636697+0000","last_active":"2024-03-24T00:32:06.638870+0000","last_peered":"2024-03-24T00:32:06.638870+0000","last_clean":"2024-03-24T00:32:06.638870+0000","last_became_active":"2024-03-24T00:32:06.636406+0000","last_became_peered":"2024-03-24T00:32:06.636406+0000","last_unstale":"2024-03-24T00:32:06.638870+0000","last_undegraded":"2024-03-24T00:32:06.638870+0000","last_fullsized":"2024-03-24T00:32:06.638870+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:13:03.982473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633079+0000","last_change":"2024-03-24T00:32:06.648061+0000","last_active":"2024-03-24T00:32:07.633079+0000","last_peered":"2024-03-24T00:32:07.633079+0000","last_clean":"2024-03-24T00:32:07.633079+0000","last_became_active":"2024-03-24T00:32:06.647712+0000","last_became_peered":"2024-03-24T00:32:06.647712+0000","last_unstale":"2024-03-24T00:32:07.633079+0000","last_undegraded":"2024-03-24T00:32:07.633079+0000","last_fullsized":"2024-03-24T00:32:07.633079+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:43:57.884539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.668089+0000","last_change":"2024-03-24T00:32:05.653072+0000","last_active":"2024-03-24T00:32:06.668089+0000","last_peered":"2024-03-24T00:32:06.668089+0000","last_clean":"2024-03-24T00:32:06.668089+0000","last_became_active":"2024-03-24T00:32:05.652553+0000","last_became_peered":"2024-03-24T00:32:05.652553+0000","last_unstale":"2024-03-24T00:32:06.668089+0000","last_undegraded":"2024-03-24T00:32:06.668089+0000","last_fullsized":"2024-03-24T00:32:06.668089+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:05.121138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.653471+0000","last_change":"2024-03-24T00:32:06.649765+0000","last_active":"2024-03-24T00:32:06.653471+0000","last_peered":"2024-03-24T00:32:06.653471+0000","last_clean":"2024-03-24T00:32:06.653471+0000","last_became_active":"2024-03-24T00:32:06.649486+0000","last_became_peered":"2024-03-24T00:32:06.649486+0000","last_unstale":"2024-03-24T00:32:06.653471+0000","last_undegraded":"2024-03-24T00:32:06.653471+0000","last_fullsized":"2024-03-24T00:32:06.653471+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:53:58.086058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633043+0000","last_change":"2024-03-24T00:32:05.644929+0000","last_active":"2024-03-24T00:32:07.633043+0000","last_peered":"2024-03-24T00:32:07.633043+0000","last_clean":"2024-03-24T00:32:07.633043+0000","last_became_active":"2024-03-24T00:32:05.644209+0000","last_became_peered":"2024-03-24T00:32:05.644209+0000","last_unstale":"2024-03-24T00:32:07.633043+0000","last_undegraded":"2024-03-24T00:32:07.633043+0000","last_fullsized":"2024-03-24T00:32:07.633043+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:41.419190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634244+0000","last_change":"2024-03-24T00:32:06.636949+0000","last_active":"2024-03-24T00:32:07.634244+0000","last_peered":"2024-03-24T00:32:07.634244+0000","last_clean":"2024-03-24T00:32:07.634244+0000","last_became_active":"2024-03-24T00:32:06.636611+0000","last_became_peered":"2024-03-24T00:32:06.636611+0000","last_unstale":"2024-03-24T00:32:07.634244+0000","last_undegraded":"2024-03-24T00:32:07.634244+0000","last_fullsized":"2024-03-24T00:32:07.634244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:42:24.998068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.677494+0000","last_change":"2024-03-24T00:32:05.645566+0000","last_active":"2024-03-24T00:32:06.677494+0000","last_peered":"2024-03-24T00:32:06.677494+0000","last_clean":"2024-03-24T00:32:06.677494+0000","last_became_active":"2024-03-24T00:32:05.644861+0000","last_became_peered":"2024-03-24T00:32:05.644861+0000","last_unstale":"2024-03-24T00:32:06.677494+0000","last_undegraded":"2024-03-24T00:32:06.677494+0000","last_fullsized":"2024-03-24T00:32:06.677494+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:41.849574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634360+0000","last_change":"2024-03-24T00:32:06.661387+0000","last_active":"2024-03-24T00:32:07.634360+0000","last_peered":"2024-03-24T00:32:07.634360+0000","last_clean":"2024-03-24T00:32:07.634360+0000","last_became_active":"2024-03-24T00:32:06.660732+0000","last_became_peered":"2024-03-24T00:32:06.660732+0000","last_unstale":"2024-03-24T00:32:07.634360+0000","last_undegraded":"2024-03-24T00:32:07.634360+0000","last_fullsized":"2024-03-24T00:32:07.634360+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:54.558428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.673101+0000","last_change":"2024-03-24T00:32:05.652339+0000","last_active":"2024-03-24T00:32:06.673101+0000","last_peered":"2024-03-24T00:32:06.673101+0000","last_clean":"2024-03-24T00:32:06.673101+0000","last_became_active":"2024-03-24T00:32:05.652003+0000","last_became_peered":"2024-03-24T00:32:05.652003+0000","last_unstale":"2024-03-24T00:32:06.673101+0000","last_undegraded":"2024-03-24T00:32:06.673101+0000","last_fullsized":"2024-03-24T00:32:06.673101+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:40.549119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.645951+0000","last_change":"2024-03-24T00:32:06.638036+0000","last_active":"2024-03-24T00:32:06.645951+0000","last_peered":"2024-03-24T00:32:06.645951+0000","last_clean":"2024-03-24T00:32:06.645951+0000","last_became_active":"2024-03-24T00:32:06.637735+0000","last_became_peered":"2024-03-24T00:32:06.637735+0000","last_unstale":"2024-03-24T00:32:06.645951+0000","last_undegraded":"2024-03-24T00:32:06.645951+0000","last_fullsized":"2024-03-24T00:32:06.645951+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:37.538781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633993+0000","last_change":"2024-03-24T00:32:05.648277+0000","last_active":"2024-03-24T00:32:07.633993+0000","last_peered":"2024-03-24T00:32:07.633993+0000","last_clean":"2024-03-24T00:32:07.633993+0000","last_became_active":"2024-03-24T00:32:05.647935+0000","last_became_peered":"2024-03-24T00:32:05.647935+0000","last_unstale":"2024-03-24T00:32:07.633993+0000","last_undegraded":"2024-03-24T00:32:07.633993+0000","last_fullsized":"2024-03-24T00:32:07.633993+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:04:47.920356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730202+0000","last_change":"2024-03-24T00:32:06.652842+0000","last_active":"2024-03-24T00:32:07.730202+0000","last_peered":"2024-03-24T00:32:07.730202+0000","last_clean":"2024-03-24T00:32:07.730202+0000","last_became_active":"2024-03-24T00:32:06.652516+0000","last_became_peered":"2024-03-24T00:32:06.652516+0000","last_unstale":"2024-03-24T00:32:07.730202+0000","last_undegraded":"2024-03-24T00:32:07.730202+0000","last_fullsized":"2024-03-24T00:32:07.730202+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:53.548614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633964+0000","last_change":"2024-03-24T00:32:05.646198+0000","last_active":"2024-03-24T00:32:07.633964+0000","last_peered":"2024-03-24T00:32:07.633964+0000","last_clean":"2024-03-24T00:32:07.633964+0000","last_became_active":"2024-03-24T00:32:05.645863+0000","last_became_peered":"2024-03-24T00:32:05.645863+0000","last_unstale":"2024-03-24T00:32:07.633964+0000","last_undegraded":"2024-03-24T00:32:07.633964+0000","last_fullsized":"2024-03-24T00:32:07.633964+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:41.419190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633781+0000","last_change":"2024-03-24T00:32:06.648159+0000","last_active":"2024-03-24T00:32:07.633781+0000","last_peered":"2024-03-24T00:32:07.633781+0000","last_clean":"2024-03-24T00:32:07.633781+0000","last_became_active":"2024-03-24T00:32:06.647071+0000","last_became_peered":"2024-03-24T00:32:06.647071+0000","last_unstale":"2024-03-24T00:32:07.633781+0000","last_undegraded":"2024-03-24T00:32:07.633781+0000","last_fullsized":"2024-03-24T00:32:07.633781+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:43:57.884539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.630900+0000","last_change":"2024-03-24T00:32:05.651447+0000","last_active":"2024-03-24T00:32:06.630900+0000","last_peered":"2024-03-24T00:32:06.630900+0000","last_clean":"2024-03-24T00:32:06.630900+0000","last_became_active":"2024-03-24T00:32:05.651118+0000","last_became_peered":"2024-03-24T00:32:05.651118+0000","last_unstale":"2024-03-24T00:32:06.630900+0000","last_undegraded":"2024-03-24T00:32:06.630900+0000","last_fullsized":"2024-03-24T00:32:06.630900+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:12:06.296848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632130+0000","last_change":"2024-03-24T00:32:06.661046+0000","last_active":"2024-03-24T00:32:07.632130+0000","last_peered":"2024-03-24T00:32:07.632130+0000","last_clean":"2024-03-24T00:32:07.632130+0000","last_became_active":"2024-03-24T00:32:06.660584+0000","last_became_peered":"2024-03-24T00:32:06.660584+0000","last_unstale":"2024-03-24T00:32:07.632130+0000","last_undegraded":"2024-03-24T00:32:07.632130+0000","last_fullsized":"2024-03-24T00:32:07.632130+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:42.860138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633707+0000","last_change":"2024-03-24T00:32:05.657062+0000","last_active":"2024-03-24T00:32:07.633707+0000","last_peered":"2024-03-24T00:32:07.633707+0000","last_clean":"2024-03-24T00:32:07.633707+0000","last_became_active":"2024-03-24T00:32:05.656678+0000","last_became_peered":"2024-03-24T00:32:05.656678+0000","last_unstale":"2024-03-24T00:32:07.633707+0000","last_undegraded":"2024-03-24T00:32:07.633707+0000","last_fullsized":"2024-03-24T00:32:07.633707+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:52:23.175141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730678+0000","last_change":"2024-03-24T00:32:06.659262+0000","last_active":"2024-03-24T00:32:07.730678+0000","last_peered":"2024-03-24T00:32:07.730678+0000","last_clean":"2024-03-24T00:32:07.730678+0000","last_became_active":"2024-03-24T00:32:06.659000+0000","last_became_peered":"2024-03-24T00:32:06.659000+0000","last_unstale":"2024-03-24T00:32:07.730678+0000","last_undegraded":"2024-03-24T00:32:07.730678+0000","last_fullsized":"2024-03-24T00:32:07.730678+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:30:39.174018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.631601+0000","last_change":"2024-03-24T00:32:05.646929+0000","last_active":"2024-03-24T00:32:06.631601+0000","last_peered":"2024-03-24T00:32:06.631601+0000","last_clean":"2024-03-24T00:32:06.631601+0000","last_became_active":"2024-03-24T00:32:05.646516+0000","last_became_peered":"2024-03-24T00:32:05.646516+0000","last_unstale":"2024-03-24T00:32:06.631601+0000","last_undegraded":"2024-03-24T00:32:06.631601+0000","last_fullsized":"2024-03-24T00:32:06.631601+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:43:56.873975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.670379+0000","last_change":"2024-03-24T00:32:06.651755+0000","last_active":"2024-03-24T00:32:06.670379+0000","last_peered":"2024-03-24T00:32:06.670379+0000","last_clean":"2024-03-24T00:32:06.670379+0000","last_became_active":"2024-03-24T00:32:06.651498+0000","last_became_peered":"2024-03-24T00:32:06.651498+0000","last_unstale":"2024-03-24T00:32:06.670379+0000","last_undegraded":"2024-03-24T00:32:06.670379+0000","last_fullsized":"2024-03-24T00:32:06.670379+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:17.608787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633566+0000","last_change":"2024-03-24T00:32:05.641208+0000","last_active":"2024-03-24T00:32:07.633566+0000","last_peered":"2024-03-24T00:32:07.633566+0000","last_clean":"2024-03-24T00:32:07.633566+0000","last_became_active":"2024-03-24T00:32:05.640888+0000","last_became_peered":"2024-03-24T00:32:05.640888+0000","last_unstale":"2024-03-24T00:32:07.633566+0000","last_undegraded":"2024-03-24T00:32:07.633566+0000","last_fullsized":"2024-03-24T00:32:07.633566+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:02.695616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.643297+0000","last_change":"2024-03-24T00:32:06.639883+0000","last_active":"2024-03-24T00:32:06.643297+0000","last_peered":"2024-03-24T00:32:06.643297+0000","last_clean":"2024-03-24T00:32:06.643297+0000","last_became_active":"2024-03-24T00:32:06.639588+0000","last_became_peered":"2024-03-24T00:32:06.639588+0000","last_unstale":"2024-03-24T00:32:06.643297+0000","last_undegraded":"2024-03-24T00:32:06.643297+0000","last_fullsized":"2024-03-24T00:32:06.643297+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:08.356624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730636+0000","last_change":"2024-03-24T00:32:05.651798+0000","last_active":"2024-03-24T00:32:07.730636+0000","last_peered":"2024-03-24T00:32:07.730636+0000","last_clean":"2024-03-24T00:32:07.730636+0000","last_became_active":"2024-03-24T00:32:05.650961+0000","last_became_peered":"2024-03-24T00:32:05.650961+0000","last_unstale":"2024-03-24T00:32:07.730636+0000","last_undegraded":"2024-03-24T00:32:07.730636+0000","last_fullsized":"2024-03-24T00:32:07.730636+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:36.528217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730444+0000","last_change":"2024-03-24T00:32:06.667468+0000","last_active":"2024-03-24T00:32:07.730444+0000","last_peered":"2024-03-24T00:32:07.730444+0000","last_clean":"2024-03-24T00:32:07.730444+0000","last_became_active":"2024-03-24T00:32:06.667075+0000","last_became_peered":"2024-03-24T00:32:06.667075+0000","last_unstale":"2024-03-24T00:32:07.730444+0000","last_undegraded":"2024-03-24T00:32:07.730444+0000","last_fullsized":"2024-03-24T00:32:07.730444+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:16:50.534597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.679863+0000","last_change":"2024-03-24T00:32:05.653205+0000","last_active":"2024-03-24T00:32:06.679863+0000","last_peered":"2024-03-24T00:32:06.679863+0000","last_clean":"2024-03-24T00:32:06.679863+0000","last_became_active":"2024-03-24T00:32:05.652963+0000","last_became_peered":"2024-03-24T00:32:05.652963+0000","last_unstale":"2024-03-24T00:32:06.679863+0000","last_undegraded":"2024-03-24T00:32:06.679863+0000","last_fullsized":"2024-03-24T00:32:06.679863+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:58:21.337093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632182+0000","last_change":"2024-03-24T00:32:06.654195+0000","last_active":"2024-03-24T00:32:07.632182+0000","last_peered":"2024-03-24T00:32:07.632182+0000","last_clean":"2024-03-24T00:32:07.632182+0000","last_became_active":"2024-03-24T00:32:06.653902+0000","last_became_peered":"2024-03-24T00:32:06.653902+0000","last_unstale":"2024-03-24T00:32:07.632182+0000","last_undegraded":"2024-03-24T00:32:07.632182+0000","last_fullsized":"2024-03-24T00:32:07.632182+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:08.356624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634076+0000","last_change":"2024-03-24T00:32:05.657149+0000","last_active":"2024-03-24T00:32:07.634076+0000","last_peered":"2024-03-24T00:32:07.634076+0000","last_clean":"2024-03-24T00:32:07.634076+0000","last_became_active":"2024-03-24T00:32:05.656636+0000","last_became_peered":"2024-03-24T00:32:05.656636+0000","last_unstale":"2024-03-24T00:32:07.634076+0000","last_undegraded":"2024-03-24T00:32:07.634076+0000","last_fullsized":"2024-03-24T00:32:07.634076+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:49.246320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.673582+0000","last_change":"2024-03-24T00:32:06.653499+0000","last_active":"2024-03-24T00:32:06.673582+0000","last_peered":"2024-03-24T00:32:06.673582+0000","last_clean":"2024-03-24T00:32:06.673582+0000","last_became_active":"2024-03-24T00:32:06.653243+0000","last_became_peered":"2024-03-24T00:32:06.653243+0000","last_unstale":"2024-03-24T00:32:06.673582+0000","last_undegraded":"2024-03-24T00:32:06.673582+0000","last_fullsized":"2024-03-24T00:32:06.673582+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:54.558428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632281+0000","last_change":"2024-03-24T00:32:05.648901+0000","last_active":"2024-03-24T00:32:07.632281+0000","last_peered":"2024-03-24T00:32:07.632281+0000","last_clean":"2024-03-24T00:32:07.632281+0000","last_became_active":"2024-03-24T00:32:05.647007+0000","last_became_peered":"2024-03-24T00:32:05.647007+0000","last_unstale":"2024-03-24T00:32:07.632281+0000","last_undegraded":"2024-03-24T00:32:07.632281+0000","last_fullsized":"2024-03-24T00:32:07.632281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:12.531540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.646114+0000","last_change":"2024-03-24T00:32:06.638414+0000","last_active":"2024-03-24T00:32:06.646114+0000","last_peered":"2024-03-24T00:32:06.646114+0000","last_clean":"2024-03-24T00:32:06.646114+0000","last_became_active":"2024-03-24T00:32:06.637451+0000","last_became_peered":"2024-03-24T00:32:06.637451+0000","last_unstale":"2024-03-24T00:32:06.646114+0000","last_undegraded":"2024-03-24T00:32:06.646114+0000","last_fullsized":"2024-03-24T00:32:06.646114+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:12:07.307412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.631378+0000","last_change":"2024-03-24T00:32:05.652002+0000","last_active":"2024-03-24T00:32:06.631378+0000","last_peered":"2024-03-24T00:32:06.631378+0000","last_clean":"2024-03-24T00:32:06.631378+0000","last_became_active":"2024-03-24T00:32:05.651746+0000","last_became_peered":"2024-03-24T00:32:05.651746+0000","last_unstale":"2024-03-24T00:32:06.631378+0000","last_undegraded":"2024-03-24T00:32:06.631378+0000","last_fullsized":"2024-03-24T00:32:06.631378+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:29.996970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.661547+0000","last_change":"2024-03-24T00:32:06.656314+0000","last_active":"2024-03-24T00:32:06.661547+0000","last_peered":"2024-03-24T00:32:06.661547+0000","last_clean":"2024-03-24T00:32:06.661547+0000","last_became_active":"2024-03-24T00:32:06.654864+0000","last_became_peered":"2024-03-24T00:32:06.654864+0000","last_unstale":"2024-03-24T00:32:06.661547+0000","last_undegraded":"2024-03-24T00:32:06.661547+0000","last_fullsized":"2024-03-24T00:32:06.661547+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:42.429755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.678041+0000","last_change":"2024-03-24T00:32:05.645461+0000","last_active":"2024-03-24T00:32:06.678041+0000","last_peered":"2024-03-24T00:32:06.678041+0000","last_clean":"2024-03-24T00:32:06.678041+0000","last_became_active":"2024-03-24T00:32:05.645153+0000","last_became_peered":"2024-03-24T00:32:05.645153+0000","last_unstale":"2024-03-24T00:32:06.678041+0000","last_undegraded":"2024-03-24T00:32:06.678041+0000","last_fullsized":"2024-03-24T00:32:06.678041+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:36.528217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634446+0000","last_change":"2024-03-24T00:32:06.656235+0000","last_active":"2024-03-24T00:32:07.634446+0000","last_peered":"2024-03-24T00:32:07.634446+0000","last_clean":"2024-03-24T00:32:07.634446+0000","last_became_active":"2024-03-24T00:32:06.655171+0000","last_became_peered":"2024-03-24T00:32:06.655171+0000","last_unstale":"2024-03-24T00:32:07.634446+0000","last_undegraded":"2024-03-24T00:32:07.634446+0000","last_fullsized":"2024-03-24T00:32:07.634446+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:40.821676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634398+0000","last_change":"2024-03-24T00:32:05.645160+0000","last_active":"2024-03-24T00:32:07.634398+0000","last_peered":"2024-03-24T00:32:07.634398+0000","last_clean":"2024-03-24T00:32:07.634398+0000","last_became_active":"2024-03-24T00:32:05.644878+0000","last_became_peered":"2024-03-24T00:32:05.644878+0000","last_unstale":"2024-03-24T00:32:07.634398+0000","last_undegraded":"2024-03-24T00:32:07.634398+0000","last_fullsized":"2024-03-24T00:32:07.634398+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:58:21.337093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730696+0000","last_change":"2024-03-24T00:32:06.653722+0000","last_active":"2024-03-24T00:32:07.730696+0000","last_peered":"2024-03-24T00:32:07.730696+0000","last_clean":"2024-03-24T00:32:07.730696+0000","last_became_active":"2024-03-24T00:32:06.653426+0000","last_became_peered":"2024-03-24T00:32:06.653426+0000","last_unstale":"2024-03-24T00:32:07.730696+0000","last_undegraded":"2024-03-24T00:32:07.730696+0000","last_fullsized":"2024-03-24T00:32:07.730696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:03.730126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634385+0000","last_change":"2024-03-24T00:32:05.657809+0000","last_active":"2024-03-24T00:32:07.634385+0000","last_peered":"2024-03-24T00:32:07.634385+0000","last_clean":"2024-03-24T00:32:07.634385+0000","last_became_active":"2024-03-24T00:32:05.657401+0000","last_became_peered":"2024-03-24T00:32:05.657401+0000","last_unstale":"2024-03-24T00:32:07.634385+0000","last_undegraded":"2024-03-24T00:32:07.634385+0000","last_fullsized":"2024-03-24T00:32:07.634385+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:05.121138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634270+0000","last_change":"2024-03-24T00:32:06.649734+0000","last_active":"2024-03-24T00:32:07.634270+0000","last_peered":"2024-03-24T00:32:07.634270+0000","last_clean":"2024-03-24T00:32:07.634270+0000","last_became_active":"2024-03-24T00:32:06.644029+0000","last_became_peered":"2024-03-24T00:32:06.644029+0000","last_unstale":"2024-03-24T00:32:07.634270+0000","last_undegraded":"2024-03-24T00:32:07.634270+0000","last_fullsized":"2024-03-24T00:32:07.634270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:40.821676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634380+0000","last_change":"2024-03-24T00:32:05.657590+0000","last_active":"2024-03-24T00:32:07.634380+0000","last_peered":"2024-03-24T00:32:07.634380+0000","last_clean":"2024-03-24T00:32:07.634380+0000","last_became_active":"2024-03-24T00:32:05.657315+0000","last_became_peered":"2024-03-24T00:32:05.657315+0000","last_unstale":"2024-03-24T00:32:07.634380+0000","last_undegraded":"2024-03-24T00:32:07.634380+0000","last_fullsized":"2024-03-24T00:32:07.634380+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:53:57.075494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.668376+0000","last_change":"2024-03-24T00:32:06.653459+0000","last_active":"2024-03-24T00:32:06.668376+0000","last_peered":"2024-03-24T00:32:06.668376+0000","last_clean":"2024-03-24T00:32:06.668376+0000","last_became_active":"2024-03-24T00:32:06.653252+0000","last_became_peered":"2024-03-24T00:32:06.653252+0000","last_unstale":"2024-03-24T00:32:06.668376+0000","last_undegraded":"2024-03-24T00:32:06.668376+0000","last_fullsized":"2024-03-24T00:32:06.668376+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:06.608312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634183+0000","last_change":"2024-03-24T00:32:05.646552+0000","last_active":"2024-03-24T00:32:07.634183+0000","last_peered":"2024-03-24T00:32:07.634183+0000","last_clean":"2024-03-24T00:32:07.634183+0000","last_became_active":"2024-03-24T00:32:05.646216+0000","last_became_peered":"2024-03-24T00:32:05.646216+0000","last_unstale":"2024-03-24T00:32:07.634183+0000","last_undegraded":"2024-03-24T00:32:07.634183+0000","last_fullsized":"2024-03-24T00:32:07.634183+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:17:25.727401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.666159+0000","last_change":"2024-03-24T00:32:06.647559+0000","last_active":"2024-03-24T00:32:06.666159+0000","last_peered":"2024-03-24T00:32:06.666159+0000","last_clean":"2024-03-24T00:32:06.666159+0000","last_became_active":"2024-03-24T00:32:06.647238+0000","last_became_peered":"2024-03-24T00:32:06.647238+0000","last_unstale":"2024-03-24T00:32:06.666159+0000","last_undegraded":"2024-03-24T00:32:06.666159+0000","last_fullsized":"2024-03-24T00:32:06.666159+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:33.350112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633395+0000","last_change":"2024-03-24T00:32:05.644751+0000","last_active":"2024-03-24T00:32:07.633395+0000","last_peered":"2024-03-24T00:32:07.633395+0000","last_clean":"2024-03-24T00:32:07.633395+0000","last_became_active":"2024-03-24T00:32:05.644467+0000","last_became_peered":"2024-03-24T00:32:05.644467+0000","last_unstale":"2024-03-24T00:32:07.633395+0000","last_undegraded":"2024-03-24T00:32:07.633395+0000","last_fullsized":"2024-03-24T00:32:07.633395+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:53:57.075494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.672955+0000","last_change":"2024-03-24T00:32:06.660627+0000","last_active":"2024-03-24T00:32:06.672955+0000","last_peered":"2024-03-24T00:32:06.672955+0000","last_clean":"2024-03-24T00:32:06.672955+0000","last_became_active":"2024-03-24T00:32:06.660052+0000","last_became_peered":"2024-03-24T00:32:06.660052+0000","last_unstale":"2024-03-24T00:32:06.672955+0000","last_undegraded":"2024-03-24T00:32:06.672955+0000","last_fullsized":"2024-03-24T00:32:06.672955+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:06.131702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.643285+0000","last_change":"2024-03-24T00:32:05.656238+0000","last_active":"2024-03-24T00:32:06.643285+0000","last_peered":"2024-03-24T00:32:06.643285+0000","last_clean":"2024-03-24T00:32:06.643285+0000","last_became_active":"2024-03-24T00:32:05.655859+0000","last_became_peered":"2024-03-24T00:32:05.655859+0000","last_unstale":"2024-03-24T00:32:06.643285+0000","last_undegraded":"2024-03-24T00:32:06.643285+0000","last_fullsized":"2024-03-24T00:32:06.643285+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:12.531540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.669514+0000","last_change":"2024-03-24T00:32:06.658344+0000","last_active":"2024-03-24T00:32:06.669514+0000","last_peered":"2024-03-24T00:32:06.669514+0000","last_clean":"2024-03-24T00:32:06.669514+0000","last_became_active":"2024-03-24T00:32:06.658079+0000","last_became_peered":"2024-03-24T00:32:06.658079+0000","last_unstale":"2024-03-24T00:32:06.669514+0000","last_undegraded":"2024-03-24T00:32:06.669514+0000","last_fullsized":"2024-03-24T00:32:06.669514+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:52:24.185706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633756+0000","last_change":"2024-03-24T00:32:05.649241+0000","last_active":"2024-03-24T00:32:07.633756+0000","last_peered":"2024-03-24T00:32:07.633756+0000","last_clean":"2024-03-24T00:32:07.633756+0000","last_became_active":"2024-03-24T00:32:05.648932+0000","last_became_peered":"2024-03-24T00:32:05.648932+0000","last_unstale":"2024-03-24T00:32:07.633756+0000","last_undegraded":"2024-03-24T00:32:07.633756+0000","last_fullsized":"2024-03-24T00:32:07.633756+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:12.531540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.663545+0000","last_change":"2024-03-24T00:32:06.657464+0000","last_active":"2024-03-24T00:32:06.663545+0000","last_peered":"2024-03-24T00:32:06.663545+0000","last_clean":"2024-03-24T00:32:06.663545+0000","last_became_active":"2024-03-24T00:32:06.657123+0000","last_became_peered":"2024-03-24T00:32:06.657123+0000","last_unstale":"2024-03-24T00:32:06.663545+0000","last_undegraded":"2024-03-24T00:32:06.663545+0000","last_fullsized":"2024-03-24T00:32:06.663545+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:42.860138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730127+0000","last_change":"2024-03-24T00:32:05.648699+0000","last_active":"2024-03-24T00:32:07.730127+0000","last_peered":"2024-03-24T00:32:07.730127+0000","last_clean":"2024-03-24T00:32:07.730127+0000","last_became_active":"2024-03-24T00:32:05.648370+0000","last_became_peered":"2024-03-24T00:32:05.648370+0000","last_unstale":"2024-03-24T00:32:07.730127+0000","last_undegraded":"2024-03-24T00:32:07.730127+0000","last_fullsized":"2024-03-24T00:32:07.730127+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:12:06.296848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.668319+0000","last_change":"2024-03-24T00:32:06.657811+0000","last_active":"2024-03-24T00:32:06.668319+0000","last_peered":"2024-03-24T00:32:06.668319+0000","last_clean":"2024-03-24T00:32:06.668319+0000","last_became_active":"2024-03-24T00:32:06.657517+0000","last_became_peered":"2024-03-24T00:32:06.657517+0000","last_unstale":"2024-03-24T00:32:06.668319+0000","last_undegraded":"2024-03-24T00:32:06.668319+0000","last_fullsized":"2024-03-24T00:32:06.668319+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:50.256884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634412+0000","last_change":"2024-03-24T00:32:05.650164+0000","last_active":"2024-03-24T00:32:07.634412+0000","last_peered":"2024-03-24T00:32:07.634412+0000","last_clean":"2024-03-24T00:32:07.634412+0000","last_became_active":"2024-03-24T00:32:05.649844+0000","last_became_peered":"2024-03-24T00:32:05.649844+0000","last_unstale":"2024-03-24T00:32:07.634412+0000","last_undegraded":"2024-03-24T00:32:07.634412+0000","last_fullsized":"2024-03-24T00:32:07.634412+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:29.996970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730714+0000","last_change":"2024-03-24T00:32:06.662089+0000","last_active":"2024-03-24T00:32:07.730714+0000","last_peered":"2024-03-24T00:32:07.730714+0000","last_clean":"2024-03-24T00:32:07.730714+0000","last_became_active":"2024-03-24T00:32:06.661764+0000","last_became_peered":"2024-03-24T00:32:06.661764+0000","last_unstale":"2024-03-24T00:32:07.730714+0000","last_undegraded":"2024-03-24T00:32:07.730714+0000","last_fullsized":"2024-03-24T00:32:07.730714+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:01:12.164533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.650070+0000","last_change":"2024-03-24T00:32:05.645138+0000","last_active":"2024-03-24T00:32:07.650070+0000","last_peered":"2024-03-24T00:32:07.650070+0000","last_clean":"2024-03-24T00:32:07.650070+0000","last_became_active":"2024-03-24T00:32:05.644628+0000","last_became_peered":"2024-03-24T00:32:05.644628+0000","last_unstale":"2024-03-24T00:32:07.650070+0000","last_undegraded":"2024-03-24T00:32:07.650070+0000","last_fullsized":"2024-03-24T00:32:07.650070+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:51.364923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632808+0000","last_change":"2024-03-24T00:32:06.670021+0000","last_active":"2024-03-24T00:32:07.632808+0000","last_peered":"2024-03-24T00:32:07.632808+0000","last_clean":"2024-03-24T00:32:07.632808+0000","last_became_active":"2024-03-24T00:32:06.669721+0000","last_became_peered":"2024-03-24T00:32:06.669721+0000","last_unstale":"2024-03-24T00:32:07.632808+0000","last_undegraded":"2024-03-24T00:32:07.632808+0000","last_fullsized":"2024-03-24T00:32:07.632808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:31.007534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634261+0000","last_change":"2024-03-24T00:32:05.649804+0000","last_active":"2024-03-24T00:32:07.634261+0000","last_peered":"2024-03-24T00:32:07.634261+0000","last_clean":"2024-03-24T00:32:07.634261+0000","last_became_active":"2024-03-24T00:32:05.649434+0000","last_became_peered":"2024-03-24T00:32:05.649434+0000","last_unstale":"2024-03-24T00:32:07.634261+0000","last_undegraded":"2024-03-24T00:32:07.634261+0000","last_fullsized":"2024-03-24T00:32:07.634261+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:41.419190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.665359+0000","last_change":"2024-03-24T00:32:06.658671+0000","last_active":"2024-03-24T00:32:06.665359+0000","last_peered":"2024-03-24T00:32:06.665359+0000","last_clean":"2024-03-24T00:32:06.665359+0000","last_became_active":"2024-03-24T00:32:06.658399+0000","last_became_peered":"2024-03-24T00:32:06.658399+0000","last_unstale":"2024-03-24T00:32:06.665359+0000","last_undegraded":"2024-03-24T00:32:06.665359+0000","last_fullsized":"2024-03-24T00:32:06.665359+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:03.706180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633473+0000","last_change":"2024-03-24T00:32:05.648874+0000","last_active":"2024-03-24T00:32:07.633473+0000","last_peered":"2024-03-24T00:32:07.633473+0000","last_clean":"2024-03-24T00:32:07.633473+0000","last_became_active":"2024-03-24T00:32:05.648513+0000","last_became_peered":"2024-03-24T00:32:05.648513+0000","last_unstale":"2024-03-24T00:32:07.633473+0000","last_undegraded":"2024-03-24T00:32:07.633473+0000","last_fullsized":"2024-03-24T00:32:07.633473+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:52:14.290725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633346+0000","last_change":"2024-03-24T00:32:06.655552+0000","last_active":"2024-03-24T00:32:07.633346+0000","last_peered":"2024-03-24T00:32:07.633346+0000","last_clean":"2024-03-24T00:32:07.633346+0000","last_became_active":"2024-03-24T00:32:06.655258+0000","last_became_peered":"2024-03-24T00:32:06.655258+0000","last_unstale":"2024-03-24T00:32:07.633346+0000","last_undegraded":"2024-03-24T00:32:07.633346+0000","last_fullsized":"2024-03-24T00:32:07.633346+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:01.983342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632368+0000","last_change":"2024-03-24T00:32:05.649402+0000","last_active":"2024-03-24T00:32:07.632368+0000","last_peered":"2024-03-24T00:32:07.632368+0000","last_clean":"2024-03-24T00:32:07.632368+0000","last_became_active":"2024-03-24T00:32:05.648624+0000","last_became_peered":"2024-03-24T00:32:05.648624+0000","last_unstale":"2024-03-24T00:32:07.632368+0000","last_undegraded":"2024-03-24T00:32:07.632368+0000","last_fullsized":"2024-03-24T00:32:07.632368+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:53:57.075494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.669670+0000","last_change":"2024-03-24T00:32:06.650297+0000","last_active":"2024-03-24T00:32:06.669670+0000","last_peered":"2024-03-24T00:32:06.669670+0000","last_clean":"2024-03-24T00:32:06.669670+0000","last_became_active":"2024-03-24T00:32:06.649951+0000","last_became_peered":"2024-03-24T00:32:06.649951+0000","last_unstale":"2024-03-24T00:32:06.669670+0000","last_undegraded":"2024-03-24T00:32:06.669670+0000","last_fullsized":"2024-03-24T00:32:06.669670+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:42:24.998068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633885+0000","last_change":"2024-03-24T00:32:05.652068+0000","last_active":"2024-03-24T00:32:07.633885+0000","last_peered":"2024-03-24T00:32:07.633885+0000","last_clean":"2024-03-24T00:32:07.633885+0000","last_became_active":"2024-03-24T00:32:05.651791+0000","last_became_peered":"2024-03-24T00:32:05.651791+0000","last_unstale":"2024-03-24T00:32:07.633885+0000","last_undegraded":"2024-03-24T00:32:07.633885+0000","last_fullsized":"2024-03-24T00:32:07.633885+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:52:14.290725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633805+0000","last_change":"2024-03-24T00:32:06.639843+0000","last_active":"2024-03-24T00:32:07.633805+0000","last_peered":"2024-03-24T00:32:07.633805+0000","last_clean":"2024-03-24T00:32:07.633805+0000","last_became_active":"2024-03-24T00:32:06.639546+0000","last_became_peered":"2024-03-24T00:32:06.639546+0000","last_unstale":"2024-03-24T00:32:07.633805+0000","last_undegraded":"2024-03-24T00:32:07.633805+0000","last_fullsized":"2024-03-24T00:32:07.633805+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:01.983342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.633080+0000","last_change":"2024-03-24T00:32:05.648650+0000","last_active":"2024-03-24T00:32:06.633080+0000","last_peered":"2024-03-24T00:32:06.633080+0000","last_clean":"2024-03-24T00:32:06.633080+0000","last_became_active":"2024-03-24T00:32:05.648347+0000","last_became_peered":"2024-03-24T00:32:05.648347+0000","last_unstale":"2024-03-24T00:32:06.633080+0000","last_undegraded":"2024-03-24T00:32:06.633080+0000","last_fullsized":"2024-03-24T00:32:06.633080+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:00.972778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.652978+0000","last_change":"2024-03-24T00:32:06.649251+0000","last_active":"2024-03-24T00:32:06.652978+0000","last_peered":"2024-03-24T00:32:06.652978+0000","last_clean":"2024-03-24T00:32:06.652978+0000","last_became_active":"2024-03-24T00:32:06.648934+0000","last_became_peered":"2024-03-24T00:32:06.648934+0000","last_unstale":"2024-03-24T00:32:06.652978+0000","last_undegraded":"2024-03-24T00:32:06.652978+0000","last_fullsized":"2024-03-24T00:32:06.652978+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:31.007534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.640256+0000","last_change":"2024-03-24T00:32:05.659056+0000","last_active":"2024-03-24T00:32:06.640256+0000","last_peered":"2024-03-24T00:32:06.640256+0000","last_clean":"2024-03-24T00:32:06.640256+0000","last_became_active":"2024-03-24T00:32:05.658712+0000","last_became_peered":"2024-03-24T00:32:05.658712+0000","last_unstale":"2024-03-24T00:32:06.640256+0000","last_undegraded":"2024-03-24T00:32:06.640256+0000","last_fullsized":"2024-03-24T00:32:06.640256+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:52:23.175141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":194,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.631943+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:32:06.631943+0000","last_peered":"2024-03-24T00:32:06.631943+0000","last_clean":"2024-03-24T00:32:06.631943+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:32:06.631943+0000","last_undegraded":"2024-03-24T00:32:06.631943+0000","last_fullsized":"2024-03-24T00:32:06.631943+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":730806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":368640,"data_stored":140964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":147320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":146752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":966656,"data_stored":737016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:32:10.955 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:32:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.128+0000 7f073f400640 1 Processor -- start 2024-03-24T00:32:11.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.128+0000 7f073f400640 1 -- start start 2024-03-24T00:32:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.132+0000 7f073f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f07380b2910 0x7f07380d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.132+0000 7f073f400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07380b1e50 con 0x7f07380af170 2024-03-24T00:32:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.132+0000 7f073f400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07380b20f0 con 0x7f073815f600 2024-03-24T00:32:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.132+0000 7f073f400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07380b3630 con 0x7f07380b2910 2024-03-24T00:32:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.132+0000 7f0737e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f07380b2910 0x7f07380d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.132+0000 7f0737e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f07380b2910 0x7f07380d2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42416/0 (socket says 172.21.15.40:42416) 2024-03-24T00:32:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.132+0000 7f0737e00640 1 -- 172.21.15.40:0/2412358599 learned_addr learned my addr 172.21.15.40:0/2412358599 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.132+0000 7f0737e00640 1 -- 172.21.15.40:0/2412358599 >> v1:172.21.15.53:6789/0 conn(0x7f073815f600 legacy=0x7f07380aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.132+0000 7f0737e00640 1 -- 172.21.15.40:0/2412358599 >> v1:172.21.15.40:6789/0 conn(0x7f07380af170 legacy=0x7f0738165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.132+0000 7f0737e00640 1 -- 172.21.15.40:0/2412358599 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07380b23d0 con 0x7f07380b2910 2024-03-24T00:32:11.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.132+0000 7f0737400640 1 -- 172.21.15.40:0/2412358599 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419760169 0 0) 0x7f07380b1e50 con 0x7f07380af170 2024-03-24T00:32:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.132+0000 7f0737e00640 1 --2- 172.21.15.40:0/2412358599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07380b2910 0x7f07380d2830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0724009870 tx=0x7f072401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26fe66d6742a28cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.136+0000 7f0737400640 1 -- 172.21.15.40:0/2412358599 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07240b5070 con 0x7f07380b2910 2024-03-24T00:32:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.136+0000 7f0737400640 1 -- 172.21.15.40:0/2412358599 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07240a8a50 con 0x7f07380b2910 2024-03-24T00:32:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.136+0000 7f0737400640 1 -- 172.21.15.40:0/2412358599 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07240a8e20 con 0x7f07380b2910 2024-03-24T00:32:11.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.136+0000 7f073f400640 1 -- 172.21.15.40:0/2412358599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07380b2910 msgr2=0x7f07380d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.136+0000 7f073f400640 1 --2- 172.21.15.40:0/2412358599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07380b2910 0x7f07380d2830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0724009870 tx=0x7f072401ed50 comp rx=0 tx=0).stop 2024-03-24T00:32:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.136+0000 7f073f400640 1 -- 172.21.15.40:0/2412358599 shutdown_connections 2024-03-24T00:32:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.136+0000 7f073f400640 1 --2- 172.21.15.40:0/2412358599 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07380b2910 0x7f07380d2830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.136+0000 7f073f400640 1 -- 172.21.15.40:0/2412358599 >> 172.21.15.40:0/2412358599 conn(0x7f07380d1d10 msgr2=0x7f07380d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.136+0000 7f073f400640 1 -- 172.21.15.40:0/2412358599 shutdown_connections 2024-03-24T00:32:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.136+0000 7f073f400640 1 -- 172.21.15.40:0/2412358599 wait complete. 2024-03-24T00:32:11.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.136+0000 7f073f400640 1 Processor -- start 2024-03-24T00:32:11.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.136+0000 7f073f400640 1 -- start start 2024-03-24T00:32:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f073f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380af170 0x7f07380c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f073f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07380b2910 0x7f07380cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f073f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f073815f600 0x7f07380cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f073f400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07380b2740 con 0x7f07380b2910 2024-03-24T00:32:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f073f400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07380b2d80 con 0x7f07380af170 2024-03-24T00:32:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f073f400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073815f9e0 con 0x7f073815f600 2024-03-24T00:32:11.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f0737e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07380b2910 0x7f07380cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f0737e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07380b2910 0x7f07380cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42424/0 (socket says 172.21.15.40:42424) 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f0737e00640 1 -- 172.21.15.40:0/2100195211 learned_addr learned my addr 172.21.15.40:0/2100195211 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f073ca00640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380af170 0x7f07380c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f073d400640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f073815f600 0x7f07380cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f0737e00640 1 -- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f073815f600 msgr2=0x7f07380cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f0737e00640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f073815f600 0x7f07380cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f0737e00640 1 -- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380af170 msgr2=0x7f07380c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f0737e00640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380af170 0x7f07380c5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f0737e00640 1 -- 172.21.15.40:0/2100195211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07380c6510 con 0x7f07380b2910 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f073d400640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f073815f600 0x7f07380cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f0737e00640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07380b2910 0x7f07380cd700 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f07240b3040 tx=0x7f072401baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f0734c00640 1 -- 172.21.15.40:0/2100195211 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07240c0070 con 0x7f07380b2910 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f0734c00640 1 -- 172.21.15.40:0/2100195211 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07240bb660 con 0x7f07380b2910 2024-03-24T00:32:11.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f073f400640 1 -- 172.21.15.40:0/2100195211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07380c6c30 con 0x7f07380b2910 2024-03-24T00:32:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f0734c00640 1 -- 172.21.15.40:0/2100195211 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07240b5070 con 0x7f07380b2910 2024-03-24T00:32:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f073f400640 1 -- 172.21.15.40:0/2100195211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07380c71f0 con 0x7f07380b2910 2024-03-24T00:32:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.140+0000 7f073f400640 1 -- 172.21.15.40:0/2100195211 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0738068170 con 0x7f07380b2910 2024-03-24T00:32:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.144+0000 7f0734c00640 1 -- 172.21.15.40:0/2100195211 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f07240c3070 con 0x7f07380b2910 2024-03-24T00:32:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.144+0000 7f0734c00640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0718067660 0x7f0718069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.144+0000 7f0734c00640 1 -- 172.21.15.40:0/2100195211 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0724122fb0 con 0x7f07380b2910 2024-03-24T00:32:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.144+0000 7f073ca00640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0718067660 0x7f0718069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.144+0000 7f073ca00640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0718067660 0x7f0718069b20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f072c007fd0 tx=0x7f072c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:11.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.152+0000 7f0734c00640 1 -- 172.21.15.40:0/2100195211 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f07380c6510 con 0x7f07380b2910 2024-03-24T00:32:11.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.312+0000 7f073f400640 1 -- 172.21.15.40:0/2100195211 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f073815e5b0 con 0x7f0718067660 2024-03-24T00:32:11.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.320+0000 7f0734c00640 1 -- 172.21.15.40:0/2100195211 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251931 (secure 0 0 0) 0x7f073815e5b0 con 0x7f0718067660 2024-03-24T00:32:11.328 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:11.330 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:32:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.324+0000 7f073f400640 1 -- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0718067660 msgr2=0x7f0718069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.324+0000 7f073f400640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0718067660 0x7f0718069b20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f072c007fd0 tx=0x7f072c009040 comp rx=0 tx=0).stop 2024-03-24T00:32:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.324+0000 7f073f400640 1 -- 172.21.15.40:0/2100195211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07380b2910 msgr2=0x7f07380cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.324+0000 7f073f400640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07380b2910 0x7f07380cd700 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f07240b3040 tx=0x7f072401baf0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.324+0000 7f073f400640 1 -- 172.21.15.40:0/2100195211 shutdown_connections 2024-03-24T00:32:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.324+0000 7f073f400640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0718067660 0x7f0718069b20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.324+0000 7f073f400640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f073815f600 0x7f07380cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.324+0000 7f073f400640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07380b2910 0x7f07380cd700 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.324+0000 7f073f400640 1 --2- 172.21.15.40:0/2100195211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07380af170 0x7f07380c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.324+0000 7f073f400640 1 -- 172.21.15.40:0/2100195211 >> 172.21.15.40:0/2100195211 conn(0x7f07380d1d10 msgr2=0x7f07380b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:11.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.324+0000 7f073f400640 1 -- 172.21.15.40:0/2100195211 shutdown_connections 2024-03-24T00:32:11.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.324+0000 7f073f400640 1 -- 172.21.15.40:0/2100195211 wait complete. 2024-03-24T00:32:11.350 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-03-24T00:32:11.213940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236908,"kb_used_data":4088,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737812,"statfs":{"total":773094113280,"available":772851519488,"internally_reserved":0,"allocated":4186112,"data_stored":2333540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":238276146},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001386"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.661362+0000","last_change":"2024-03-24T00:32:06.655127+0000","last_active":"2024-03-24T00:32:06.661362+0000","last_peered":"2024-03-24T00:32:06.661362+0000","last_clean":"2024-03-24T00:32:06.661362+0000","last_became_active":"2024-03-24T00:32:06.654824+0000","last_became_peered":"2024-03-24T00:32:06.654824+0000","last_unstale":"2024-03-24T00:32:06.661362+0000","last_undegraded":"2024-03-24T00:32:06.661362+0000","last_fullsized":"2024-03-24T00:32:06.661362+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:52:15.301289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.673987+0000","last_change":"2024-03-24T00:32:06.669569+0000","last_active":"2024-03-24T00:32:06.673987+0000","last_peered":"2024-03-24T00:32:06.673987+0000","last_clean":"2024-03-24T00:32:06.673987+0000","last_became_active":"2024-03-24T00:32:06.669204+0000","last_became_peered":"2024-03-24T00:32:06.669204+0000","last_unstale":"2024-03-24T00:32:06.673987+0000","last_undegraded":"2024-03-24T00:32:06.673987+0000","last_fullsized":"2024-03-24T00:32:06.673987+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:16:50.534597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730447+0000","last_change":"2024-03-24T00:32:06.666464+0000","last_active":"2024-03-24T00:32:07.730447+0000","last_peered":"2024-03-24T00:32:07.730447+0000","last_clean":"2024-03-24T00:32:07.730447+0000","last_became_active":"2024-03-24T00:32:06.666168+0000","last_became_peered":"2024-03-24T00:32:06.666168+0000","last_unstale":"2024-03-24T00:32:07.730447+0000","last_undegraded":"2024-03-24T00:32:07.730447+0000","last_fullsized":"2024-03-24T00:32:07.730447+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:33:54.537030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632533+0000","last_change":"2024-03-24T00:32:06.653589+0000","last_active":"2024-03-24T00:32:07.632533+0000","last_peered":"2024-03-24T00:32:07.632533+0000","last_clean":"2024-03-24T00:32:07.632533+0000","last_became_active":"2024-03-24T00:32:06.652742+0000","last_became_peered":"2024-03-24T00:32:06.652742+0000","last_unstale":"2024-03-24T00:32:07.632533+0000","last_undegraded":"2024-03-24T00:32:07.632533+0000","last_fullsized":"2024-03-24T00:32:07.632533+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:20.760035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.668412+0000","last_change":"2024-03-24T00:32:06.653736+0000","last_active":"2024-03-24T00:32:06.668412+0000","last_peered":"2024-03-24T00:32:06.668412+0000","last_clean":"2024-03-24T00:32:06.668412+0000","last_became_active":"2024-03-24T00:32:06.652654+0000","last_became_peered":"2024-03-24T00:32:06.652654+0000","last_unstale":"2024-03-24T00:32:06.668412+0000","last_undegraded":"2024-03-24T00:32:06.668412+0000","last_fullsized":"2024-03-24T00:32:06.668412+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:04:48.930920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.673651+0000","last_change":"2024-03-24T00:32:06.660546+0000","last_active":"2024-03-24T00:32:06.673651+0000","last_peered":"2024-03-24T00:32:06.673651+0000","last_clean":"2024-03-24T00:32:06.673651+0000","last_became_active":"2024-03-24T00:32:06.656257+0000","last_became_peered":"2024-03-24T00:32:06.656257+0000","last_unstale":"2024-03-24T00:32:06.673651+0000","last_undegraded":"2024-03-24T00:32:06.673651+0000","last_fullsized":"2024-03-24T00:32:06.673651+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:53.548614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.663568+0000","last_change":"2024-03-24T00:32:06.657598+0000","last_active":"2024-03-24T00:32:06.663568+0000","last_peered":"2024-03-24T00:32:06.663568+0000","last_clean":"2024-03-24T00:32:06.663568+0000","last_became_active":"2024-03-24T00:32:06.657435+0000","last_became_peered":"2024-03-24T00:32:06.657435+0000","last_unstale":"2024-03-24T00:32:06.663568+0000","last_undegraded":"2024-03-24T00:32:06.663568+0000","last_fullsized":"2024-03-24T00:32:06.663568+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:08.356624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.657233+0000","last_change":"2024-03-24T00:32:06.651349+0000","last_active":"2024-03-24T00:32:06.657233+0000","last_peered":"2024-03-24T00:32:06.657233+0000","last_clean":"2024-03-24T00:32:06.657233+0000","last_became_active":"2024-03-24T00:32:06.648918+0000","last_became_peered":"2024-03-24T00:32:06.648918+0000","last_unstale":"2024-03-24T00:32:06.657233+0000","last_undegraded":"2024-03-24T00:32:06.657233+0000","last_fullsized":"2024-03-24T00:32:06.657233+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:01.983342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634535+0000","last_change":"2024-03-24T00:32:06.652465+0000","last_active":"2024-03-24T00:32:07.634535+0000","last_peered":"2024-03-24T00:32:07.634535+0000","last_clean":"2024-03-24T00:32:07.634535+0000","last_became_active":"2024-03-24T00:32:06.652195+0000","last_became_peered":"2024-03-24T00:32:06.652195+0000","last_unstale":"2024-03-24T00:32:07.634535+0000","last_undegraded":"2024-03-24T00:32:07.634535+0000","last_fullsized":"2024-03-24T00:32:07.634535+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:17.608787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634260+0000","last_change":"2024-03-24T00:32:06.670893+0000","last_active":"2024-03-24T00:32:07.634260+0000","last_peered":"2024-03-24T00:32:07.634260+0000","last_clean":"2024-03-24T00:32:07.634260+0000","last_became_active":"2024-03-24T00:32:06.670633+0000","last_became_peered":"2024-03-24T00:32:06.670633+0000","last_unstale":"2024-03-24T00:32:07.634260+0000","last_undegraded":"2024-03-24T00:32:07.634260+0000","last_fullsized":"2024-03-24T00:32:07.634260+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:42:24.998068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.666212+0000","last_change":"2024-03-24T00:32:06.664702+0000","last_active":"2024-03-24T00:32:06.666212+0000","last_peered":"2024-03-24T00:32:06.666212+0000","last_clean":"2024-03-24T00:32:06.666212+0000","last_became_active":"2024-03-24T00:32:06.664482+0000","last_became_peered":"2024-03-24T00:32:06.664482+0000","last_unstale":"2024-03-24T00:32:06.666212+0000","last_undegraded":"2024-03-24T00:32:06.666212+0000","last_fullsized":"2024-03-24T00:32:06.666212+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:42:24.998068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730610+0000","last_change":"2024-03-24T00:32:06.652174+0000","last_active":"2024-03-24T00:32:07.730610+0000","last_peered":"2024-03-24T00:32:07.730610+0000","last_clean":"2024-03-24T00:32:07.730610+0000","last_became_active":"2024-03-24T00:32:06.651473+0000","last_became_peered":"2024-03-24T00:32:06.651473+0000","last_unstale":"2024-03-24T00:32:07.730610+0000","last_undegraded":"2024-03-24T00:32:07.730610+0000","last_fullsized":"2024-03-24T00:32:07.730610+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:11.652686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.670845+0000","last_change":"2024-03-24T00:32:06.652928+0000","last_active":"2024-03-24T00:32:06.670845+0000","last_peered":"2024-03-24T00:32:06.670845+0000","last_clean":"2024-03-24T00:32:06.670845+0000","last_became_active":"2024-03-24T00:32:06.652412+0000","last_became_peered":"2024-03-24T00:32:06.652412+0000","last_unstale":"2024-03-24T00:32:06.670845+0000","last_undegraded":"2024-03-24T00:32:06.670845+0000","last_fullsized":"2024-03-24T00:32:06.670845+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:46:01.596065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.668677+0000","last_change":"2024-03-24T00:32:06.650673+0000","last_active":"2024-03-24T00:32:06.668677+0000","last_peered":"2024-03-24T00:32:06.668677+0000","last_clean":"2024-03-24T00:32:06.668677+0000","last_became_active":"2024-03-24T00:32:06.650341+0000","last_became_peered":"2024-03-24T00:32:06.650341+0000","last_unstale":"2024-03-24T00:32:06.668677+0000","last_undegraded":"2024-03-24T00:32:06.668677+0000","last_fullsized":"2024-03-24T00:32:06.668677+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:40.821676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730244+0000","last_change":"2024-03-24T00:32:06.662635+0000","last_active":"2024-03-24T00:32:07.730244+0000","last_peered":"2024-03-24T00:32:07.730244+0000","last_clean":"2024-03-24T00:32:07.730244+0000","last_became_active":"2024-03-24T00:32:06.662368+0000","last_became_peered":"2024-03-24T00:32:06.662368+0000","last_unstale":"2024-03-24T00:32:07.730244+0000","last_undegraded":"2024-03-24T00:32:07.730244+0000","last_fullsized":"2024-03-24T00:32:07.730244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:01.853252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.663831+0000","last_change":"2024-03-24T00:32:06.658884+0000","last_active":"2024-03-24T00:32:06.663831+0000","last_peered":"2024-03-24T00:32:06.663831+0000","last_clean":"2024-03-24T00:32:06.663831+0000","last_became_active":"2024-03-24T00:32:06.656948+0000","last_became_peered":"2024-03-24T00:32:06.656948+0000","last_unstale":"2024-03-24T00:32:06.663831+0000","last_undegraded":"2024-03-24T00:32:06.663831+0000","last_fullsized":"2024-03-24T00:32:06.663831+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:20.760035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.653929+0000","last_change":"2024-03-24T00:32:06.648322+0000","last_active":"2024-03-24T00:32:06.653929+0000","last_peered":"2024-03-24T00:32:06.653929+0000","last_clean":"2024-03-24T00:32:06.653929+0000","last_became_active":"2024-03-24T00:32:06.647476+0000","last_became_peered":"2024-03-24T00:32:06.647476+0000","last_unstale":"2024-03-24T00:32:06.653929+0000","last_undegraded":"2024-03-24T00:32:06.653929+0000","last_fullsized":"2024-03-24T00:32:06.653929+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:58:22.347657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.672850+0000","last_change":"2024-03-24T00:32:06.652888+0000","last_active":"2024-03-24T00:32:06.672850+0000","last_peered":"2024-03-24T00:32:06.672850+0000","last_clean":"2024-03-24T00:32:06.672850+0000","last_became_active":"2024-03-24T00:32:06.652584+0000","last_became_peered":"2024-03-24T00:32:06.652584+0000","last_unstale":"2024-03-24T00:32:06.672850+0000","last_undegraded":"2024-03-24T00:32:06.672850+0000","last_fullsized":"2024-03-24T00:32:06.672850+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:34.123016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.673883+0000","last_change":"2024-03-24T00:32:06.669134+0000","last_active":"2024-03-24T00:32:06.673883+0000","last_peered":"2024-03-24T00:32:06.673883+0000","last_clean":"2024-03-24T00:32:06.673883+0000","last_became_active":"2024-03-24T00:32:06.668860+0000","last_became_peered":"2024-03-24T00:32:06.668860+0000","last_unstale":"2024-03-24T00:32:06.673883+0000","last_undegraded":"2024-03-24T00:32:06.673883+0000","last_fullsized":"2024-03-24T00:32:06.673883+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:01.853252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.668641+0000","last_change":"2024-03-24T00:32:06.657486+0000","last_active":"2024-03-24T00:32:06.668641+0000","last_peered":"2024-03-24T00:32:06.668641+0000","last_clean":"2024-03-24T00:32:06.668641+0000","last_became_active":"2024-03-24T00:32:06.656627+0000","last_became_peered":"2024-03-24T00:32:06.656627+0000","last_unstale":"2024-03-24T00:32:06.668641+0000","last_undegraded":"2024-03-24T00:32:06.668641+0000","last_fullsized":"2024-03-24T00:32:06.668641+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:17:26.737965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634065+0000","last_change":"2024-03-24T00:32:06.647201+0000","last_active":"2024-03-24T00:32:07.634065+0000","last_peered":"2024-03-24T00:32:07.634065+0000","last_clean":"2024-03-24T00:32:07.634065+0000","last_became_active":"2024-03-24T00:32:06.646847+0000","last_became_peered":"2024-03-24T00:32:06.646847+0000","last_unstale":"2024-03-24T00:32:07.634065+0000","last_undegraded":"2024-03-24T00:32:07.634065+0000","last_fullsized":"2024-03-24T00:32:07.634065+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:20.760035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633866+0000","last_change":"2024-03-24T00:32:06.636021+0000","last_active":"2024-03-24T00:32:07.633866+0000","last_peered":"2024-03-24T00:32:07.633866+0000","last_clean":"2024-03-24T00:32:07.633866+0000","last_became_active":"2024-03-24T00:32:06.635592+0000","last_became_peered":"2024-03-24T00:32:06.635592+0000","last_unstale":"2024-03-24T00:32:07.633866+0000","last_undegraded":"2024-03-24T00:32:07.633866+0000","last_fullsized":"2024-03-24T00:32:07.633866+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:42.860138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632682+0000","last_change":"2024-03-24T00:32:06.644546+0000","last_active":"2024-03-24T00:32:07.632682+0000","last_peered":"2024-03-24T00:32:07.632682+0000","last_clean":"2024-03-24T00:32:07.632682+0000","last_became_active":"2024-03-24T00:32:06.643991+0000","last_became_peered":"2024-03-24T00:32:06.643991+0000","last_unstale":"2024-03-24T00:32:07.632682+0000","last_undegraded":"2024-03-24T00:32:07.632682+0000","last_fullsized":"2024-03-24T00:32:07.632682+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:58:22.347657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730210+0000","last_change":"2024-03-24T00:32:06.652544+0000","last_active":"2024-03-24T00:32:07.730210+0000","last_peered":"2024-03-24T00:32:07.730210+0000","last_clean":"2024-03-24T00:32:07.730210+0000","last_became_active":"2024-03-24T00:32:06.652197+0000","last_became_peered":"2024-03-24T00:32:06.652197+0000","last_unstale":"2024-03-24T00:32:07.730210+0000","last_undegraded":"2024-03-24T00:32:07.730210+0000","last_fullsized":"2024-03-24T00:32:07.730210+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:54.558428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634305+0000","last_change":"2024-03-24T00:32:06.654101+0000","last_active":"2024-03-24T00:32:07.634305+0000","last_peered":"2024-03-24T00:32:07.634305+0000","last_clean":"2024-03-24T00:32:07.634305+0000","last_became_active":"2024-03-24T00:32:06.651772+0000","last_became_peered":"2024-03-24T00:32:06.651772+0000","last_unstale":"2024-03-24T00:32:07.634305+0000","last_undegraded":"2024-03-24T00:32:07.634305+0000","last_fullsized":"2024-03-24T00:32:07.634305+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:34.123016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.652764+0000","last_change":"2024-03-24T00:32:06.642960+0000","last_active":"2024-03-24T00:32:06.652764+0000","last_peered":"2024-03-24T00:32:06.652764+0000","last_clean":"2024-03-24T00:32:06.652764+0000","last_became_active":"2024-03-24T00:32:06.642430+0000","last_became_peered":"2024-03-24T00:32:06.642430+0000","last_unstale":"2024-03-24T00:32:06.652764+0000","last_undegraded":"2024-03-24T00:32:06.652764+0000","last_fullsized":"2024-03-24T00:32:06.652764+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:20.760035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634285+0000","last_change":"2024-03-24T00:32:06.661594+0000","last_active":"2024-03-24T00:32:07.634285+0000","last_peered":"2024-03-24T00:32:07.634285+0000","last_clean":"2024-03-24T00:32:07.634285+0000","last_became_active":"2024-03-24T00:32:06.661038+0000","last_became_peered":"2024-03-24T00:32:06.661038+0000","last_unstale":"2024-03-24T00:32:07.634285+0000","last_undegraded":"2024-03-24T00:32:07.634285+0000","last_fullsized":"2024-03-24T00:32:07.634285+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:17.608787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730571+0000","last_change":"2024-03-24T00:32:06.652994+0000","last_active":"2024-03-24T00:32:07.730571+0000","last_peered":"2024-03-24T00:32:07.730571+0000","last_clean":"2024-03-24T00:32:07.730571+0000","last_became_active":"2024-03-24T00:32:06.652700+0000","last_became_peered":"2024-03-24T00:32:06.652700+0000","last_unstale":"2024-03-24T00:32:07.730571+0000","last_undegraded":"2024-03-24T00:32:07.730571+0000","last_fullsized":"2024-03-24T00:32:07.730571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:12:45.164296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633901+0000","last_change":"2024-03-24T00:32:06.636883+0000","last_active":"2024-03-24T00:32:07.633901+0000","last_peered":"2024-03-24T00:32:07.633901+0000","last_clean":"2024-03-24T00:32:07.633901+0000","last_became_active":"2024-03-24T00:32:06.636585+0000","last_became_peered":"2024-03-24T00:32:06.636585+0000","last_unstale":"2024-03-24T00:32:07.633901+0000","last_undegraded":"2024-03-24T00:32:07.633901+0000","last_fullsized":"2024-03-24T00:32:07.633901+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:08.356624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.656694+0000","last_change":"2024-03-24T00:32:06.649380+0000","last_active":"2024-03-24T00:32:06.656694+0000","last_peered":"2024-03-24T00:32:06.656694+0000","last_clean":"2024-03-24T00:32:06.656694+0000","last_became_active":"2024-03-24T00:32:06.649043+0000","last_became_peered":"2024-03-24T00:32:06.649043+0000","last_unstale":"2024-03-24T00:32:06.656694+0000","last_undegraded":"2024-03-24T00:32:06.656694+0000","last_fullsized":"2024-03-24T00:32:06.656694+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:43:57.884539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.661567+0000","last_change":"2024-03-24T00:32:06.656547+0000","last_active":"2024-03-24T00:32:06.661567+0000","last_peered":"2024-03-24T00:32:06.661567+0000","last_clean":"2024-03-24T00:32:06.661567+0000","last_became_active":"2024-03-24T00:32:06.655224+0000","last_became_peered":"2024-03-24T00:32:06.655224+0000","last_unstale":"2024-03-24T00:32:06.661567+0000","last_undegraded":"2024-03-24T00:32:06.661567+0000","last_fullsized":"2024-03-24T00:32:06.661567+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:52.375487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.638870+0000","last_change":"2024-03-24T00:32:06.636697+0000","last_active":"2024-03-24T00:32:06.638870+0000","last_peered":"2024-03-24T00:32:06.638870+0000","last_clean":"2024-03-24T00:32:06.638870+0000","last_became_active":"2024-03-24T00:32:06.636406+0000","last_became_peered":"2024-03-24T00:32:06.636406+0000","last_unstale":"2024-03-24T00:32:06.638870+0000","last_undegraded":"2024-03-24T00:32:06.638870+0000","last_fullsized":"2024-03-24T00:32:06.638870+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:13:03.982473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633079+0000","last_change":"2024-03-24T00:32:06.648061+0000","last_active":"2024-03-24T00:32:07.633079+0000","last_peered":"2024-03-24T00:32:07.633079+0000","last_clean":"2024-03-24T00:32:07.633079+0000","last_became_active":"2024-03-24T00:32:06.647712+0000","last_became_peered":"2024-03-24T00:32:06.647712+0000","last_unstale":"2024-03-24T00:32:07.633079+0000","last_undegraded":"2024-03-24T00:32:07.633079+0000","last_fullsized":"2024-03-24T00:32:07.633079+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:43:57.884539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":16,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.668089+0000","last_change":"2024-03-24T00:32:05.653072+0000","last_active":"2024-03-24T00:32:06.668089+0000","last_peered":"2024-03-24T00:32:06.668089+0000","last_clean":"2024-03-24T00:32:06.668089+0000","last_became_active":"2024-03-24T00:32:05.652553+0000","last_became_peered":"2024-03-24T00:32:05.652553+0000","last_unstale":"2024-03-24T00:32:06.668089+0000","last_undegraded":"2024-03-24T00:32:06.668089+0000","last_fullsized":"2024-03-24T00:32:06.668089+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:05.121138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.653471+0000","last_change":"2024-03-24T00:32:06.649765+0000","last_active":"2024-03-24T00:32:06.653471+0000","last_peered":"2024-03-24T00:32:06.653471+0000","last_clean":"2024-03-24T00:32:06.653471+0000","last_became_active":"2024-03-24T00:32:06.649486+0000","last_became_peered":"2024-03-24T00:32:06.649486+0000","last_unstale":"2024-03-24T00:32:06.653471+0000","last_undegraded":"2024-03-24T00:32:06.653471+0000","last_fullsized":"2024-03-24T00:32:06.653471+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:53:58.086058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633043+0000","last_change":"2024-03-24T00:32:05.644929+0000","last_active":"2024-03-24T00:32:07.633043+0000","last_peered":"2024-03-24T00:32:07.633043+0000","last_clean":"2024-03-24T00:32:07.633043+0000","last_became_active":"2024-03-24T00:32:05.644209+0000","last_became_peered":"2024-03-24T00:32:05.644209+0000","last_unstale":"2024-03-24T00:32:07.633043+0000","last_undegraded":"2024-03-24T00:32:07.633043+0000","last_fullsized":"2024-03-24T00:32:07.633043+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:41.419190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634244+0000","last_change":"2024-03-24T00:32:06.636949+0000","last_active":"2024-03-24T00:32:07.634244+0000","last_peered":"2024-03-24T00:32:07.634244+0000","last_clean":"2024-03-24T00:32:07.634244+0000","last_became_active":"2024-03-24T00:32:06.636611+0000","last_became_peered":"2024-03-24T00:32:06.636611+0000","last_unstale":"2024-03-24T00:32:07.634244+0000","last_undegraded":"2024-03-24T00:32:07.634244+0000","last_fullsized":"2024-03-24T00:32:07.634244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:42:24.998068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.677494+0000","last_change":"2024-03-24T00:32:05.645566+0000","last_active":"2024-03-24T00:32:06.677494+0000","last_peered":"2024-03-24T00:32:06.677494+0000","last_clean":"2024-03-24T00:32:06.677494+0000","last_became_active":"2024-03-24T00:32:05.644861+0000","last_became_peered":"2024-03-24T00:32:05.644861+0000","last_unstale":"2024-03-24T00:32:06.677494+0000","last_undegraded":"2024-03-24T00:32:06.677494+0000","last_fullsized":"2024-03-24T00:32:06.677494+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:41.849574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634360+0000","last_change":"2024-03-24T00:32:06.661387+0000","last_active":"2024-03-24T00:32:07.634360+0000","last_peered":"2024-03-24T00:32:07.634360+0000","last_clean":"2024-03-24T00:32:07.634360+0000","last_became_active":"2024-03-24T00:32:06.660732+0000","last_became_peered":"2024-03-24T00:32:06.660732+0000","last_unstale":"2024-03-24T00:32:07.634360+0000","last_undegraded":"2024-03-24T00:32:07.634360+0000","last_fullsized":"2024-03-24T00:32:07.634360+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:54.558428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":14,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.673101+0000","last_change":"2024-03-24T00:32:05.652339+0000","last_active":"2024-03-24T00:32:06.673101+0000","last_peered":"2024-03-24T00:32:06.673101+0000","last_clean":"2024-03-24T00:32:06.673101+0000","last_became_active":"2024-03-24T00:32:05.652003+0000","last_became_peered":"2024-03-24T00:32:05.652003+0000","last_unstale":"2024-03-24T00:32:06.673101+0000","last_undegraded":"2024-03-24T00:32:06.673101+0000","last_fullsized":"2024-03-24T00:32:06.673101+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:40.549119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.645951+0000","last_change":"2024-03-24T00:32:06.638036+0000","last_active":"2024-03-24T00:32:06.645951+0000","last_peered":"2024-03-24T00:32:06.645951+0000","last_clean":"2024-03-24T00:32:06.645951+0000","last_became_active":"2024-03-24T00:32:06.637735+0000","last_became_peered":"2024-03-24T00:32:06.637735+0000","last_unstale":"2024-03-24T00:32:06.645951+0000","last_undegraded":"2024-03-24T00:32:06.645951+0000","last_fullsized":"2024-03-24T00:32:06.645951+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:37.538781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633993+0000","last_change":"2024-03-24T00:32:05.648277+0000","last_active":"2024-03-24T00:32:07.633993+0000","last_peered":"2024-03-24T00:32:07.633993+0000","last_clean":"2024-03-24T00:32:07.633993+0000","last_became_active":"2024-03-24T00:32:05.647935+0000","last_became_peered":"2024-03-24T00:32:05.647935+0000","last_unstale":"2024-03-24T00:32:07.633993+0000","last_undegraded":"2024-03-24T00:32:07.633993+0000","last_fullsized":"2024-03-24T00:32:07.633993+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:04:47.920356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730202+0000","last_change":"2024-03-24T00:32:06.652842+0000","last_active":"2024-03-24T00:32:07.730202+0000","last_peered":"2024-03-24T00:32:07.730202+0000","last_clean":"2024-03-24T00:32:07.730202+0000","last_became_active":"2024-03-24T00:32:06.652516+0000","last_became_peered":"2024-03-24T00:32:06.652516+0000","last_unstale":"2024-03-24T00:32:07.730202+0000","last_undegraded":"2024-03-24T00:32:07.730202+0000","last_fullsized":"2024-03-24T00:32:07.730202+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:53.548614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633964+0000","last_change":"2024-03-24T00:32:05.646198+0000","last_active":"2024-03-24T00:32:07.633964+0000","last_peered":"2024-03-24T00:32:07.633964+0000","last_clean":"2024-03-24T00:32:07.633964+0000","last_became_active":"2024-03-24T00:32:05.645863+0000","last_became_peered":"2024-03-24T00:32:05.645863+0000","last_unstale":"2024-03-24T00:32:07.633964+0000","last_undegraded":"2024-03-24T00:32:07.633964+0000","last_fullsized":"2024-03-24T00:32:07.633964+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:41.419190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633781+0000","last_change":"2024-03-24T00:32:06.648159+0000","last_active":"2024-03-24T00:32:07.633781+0000","last_peered":"2024-03-24T00:32:07.633781+0000","last_clean":"2024-03-24T00:32:07.633781+0000","last_became_active":"2024-03-24T00:32:06.647071+0000","last_became_peered":"2024-03-24T00:32:06.647071+0000","last_unstale":"2024-03-24T00:32:07.633781+0000","last_undegraded":"2024-03-24T00:32:07.633781+0000","last_fullsized":"2024-03-24T00:32:07.633781+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:43:57.884539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.630900+0000","last_change":"2024-03-24T00:32:05.651447+0000","last_active":"2024-03-24T00:32:06.630900+0000","last_peered":"2024-03-24T00:32:06.630900+0000","last_clean":"2024-03-24T00:32:06.630900+0000","last_became_active":"2024-03-24T00:32:05.651118+0000","last_became_peered":"2024-03-24T00:32:05.651118+0000","last_unstale":"2024-03-24T00:32:06.630900+0000","last_undegraded":"2024-03-24T00:32:06.630900+0000","last_fullsized":"2024-03-24T00:32:06.630900+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:12:06.296848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632130+0000","last_change":"2024-03-24T00:32:06.661046+0000","last_active":"2024-03-24T00:32:07.632130+0000","last_peered":"2024-03-24T00:32:07.632130+0000","last_clean":"2024-03-24T00:32:07.632130+0000","last_became_active":"2024-03-24T00:32:06.660584+0000","last_became_peered":"2024-03-24T00:32:06.660584+0000","last_unstale":"2024-03-24T00:32:07.632130+0000","last_undegraded":"2024-03-24T00:32:07.632130+0000","last_fullsized":"2024-03-24T00:32:07.632130+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:42.860138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633707+0000","last_change":"2024-03-24T00:32:05.657062+0000","last_active":"2024-03-24T00:32:07.633707+0000","last_peered":"2024-03-24T00:32:07.633707+0000","last_clean":"2024-03-24T00:32:07.633707+0000","last_became_active":"2024-03-24T00:32:05.656678+0000","last_became_peered":"2024-03-24T00:32:05.656678+0000","last_unstale":"2024-03-24T00:32:07.633707+0000","last_undegraded":"2024-03-24T00:32:07.633707+0000","last_fullsized":"2024-03-24T00:32:07.633707+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:52:23.175141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730678+0000","last_change":"2024-03-24T00:32:06.659262+0000","last_active":"2024-03-24T00:32:07.730678+0000","last_peered":"2024-03-24T00:32:07.730678+0000","last_clean":"2024-03-24T00:32:07.730678+0000","last_became_active":"2024-03-24T00:32:06.659000+0000","last_became_peered":"2024-03-24T00:32:06.659000+0000","last_unstale":"2024-03-24T00:32:07.730678+0000","last_undegraded":"2024-03-24T00:32:07.730678+0000","last_fullsized":"2024-03-24T00:32:07.730678+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:30:39.174018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.631601+0000","last_change":"2024-03-24T00:32:05.646929+0000","last_active":"2024-03-24T00:32:06.631601+0000","last_peered":"2024-03-24T00:32:06.631601+0000","last_clean":"2024-03-24T00:32:06.631601+0000","last_became_active":"2024-03-24T00:32:05.646516+0000","last_became_peered":"2024-03-24T00:32:05.646516+0000","last_unstale":"2024-03-24T00:32:06.631601+0000","last_undegraded":"2024-03-24T00:32:06.631601+0000","last_fullsized":"2024-03-24T00:32:06.631601+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:43:56.873975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.670379+0000","last_change":"2024-03-24T00:32:06.651755+0000","last_active":"2024-03-24T00:32:06.670379+0000","last_peered":"2024-03-24T00:32:06.670379+0000","last_clean":"2024-03-24T00:32:06.670379+0000","last_became_active":"2024-03-24T00:32:06.651498+0000","last_became_peered":"2024-03-24T00:32:06.651498+0000","last_unstale":"2024-03-24T00:32:06.670379+0000","last_undegraded":"2024-03-24T00:32:06.670379+0000","last_fullsized":"2024-03-24T00:32:06.670379+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:17.608787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633566+0000","last_change":"2024-03-24T00:32:05.641208+0000","last_active":"2024-03-24T00:32:07.633566+0000","last_peered":"2024-03-24T00:32:07.633566+0000","last_clean":"2024-03-24T00:32:07.633566+0000","last_became_active":"2024-03-24T00:32:05.640888+0000","last_became_peered":"2024-03-24T00:32:05.640888+0000","last_unstale":"2024-03-24T00:32:07.633566+0000","last_undegraded":"2024-03-24T00:32:07.633566+0000","last_fullsized":"2024-03-24T00:32:07.633566+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:02.695616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.643297+0000","last_change":"2024-03-24T00:32:06.639883+0000","last_active":"2024-03-24T00:32:06.643297+0000","last_peered":"2024-03-24T00:32:06.643297+0000","last_clean":"2024-03-24T00:32:06.643297+0000","last_became_active":"2024-03-24T00:32:06.639588+0000","last_became_peered":"2024-03-24T00:32:06.639588+0000","last_unstale":"2024-03-24T00:32:06.643297+0000","last_undegraded":"2024-03-24T00:32:06.643297+0000","last_fullsized":"2024-03-24T00:32:06.643297+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:08.356624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730636+0000","last_change":"2024-03-24T00:32:05.651798+0000","last_active":"2024-03-24T00:32:07.730636+0000","last_peered":"2024-03-24T00:32:07.730636+0000","last_clean":"2024-03-24T00:32:07.730636+0000","last_became_active":"2024-03-24T00:32:05.650961+0000","last_became_peered":"2024-03-24T00:32:05.650961+0000","last_unstale":"2024-03-24T00:32:07.730636+0000","last_undegraded":"2024-03-24T00:32:07.730636+0000","last_fullsized":"2024-03-24T00:32:07.730636+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:36.528217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730444+0000","last_change":"2024-03-24T00:32:06.667468+0000","last_active":"2024-03-24T00:32:07.730444+0000","last_peered":"2024-03-24T00:32:07.730444+0000","last_clean":"2024-03-24T00:32:07.730444+0000","last_became_active":"2024-03-24T00:32:06.667075+0000","last_became_peered":"2024-03-24T00:32:06.667075+0000","last_unstale":"2024-03-24T00:32:07.730444+0000","last_undegraded":"2024-03-24T00:32:07.730444+0000","last_fullsized":"2024-03-24T00:32:07.730444+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:16:50.534597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":18,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.679863+0000","last_change":"2024-03-24T00:32:05.653205+0000","last_active":"2024-03-24T00:32:06.679863+0000","last_peered":"2024-03-24T00:32:06.679863+0000","last_clean":"2024-03-24T00:32:06.679863+0000","last_became_active":"2024-03-24T00:32:05.652963+0000","last_became_peered":"2024-03-24T00:32:05.652963+0000","last_unstale":"2024-03-24T00:32:06.679863+0000","last_undegraded":"2024-03-24T00:32:06.679863+0000","last_fullsized":"2024-03-24T00:32:06.679863+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:58:21.337093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632182+0000","last_change":"2024-03-24T00:32:06.654195+0000","last_active":"2024-03-24T00:32:07.632182+0000","last_peered":"2024-03-24T00:32:07.632182+0000","last_clean":"2024-03-24T00:32:07.632182+0000","last_became_active":"2024-03-24T00:32:06.653902+0000","last_became_peered":"2024-03-24T00:32:06.653902+0000","last_unstale":"2024-03-24T00:32:07.632182+0000","last_undegraded":"2024-03-24T00:32:07.632182+0000","last_fullsized":"2024-03-24T00:32:07.632182+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:08.356624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634076+0000","last_change":"2024-03-24T00:32:05.657149+0000","last_active":"2024-03-24T00:32:07.634076+0000","last_peered":"2024-03-24T00:32:07.634076+0000","last_clean":"2024-03-24T00:32:07.634076+0000","last_became_active":"2024-03-24T00:32:05.656636+0000","last_became_peered":"2024-03-24T00:32:05.656636+0000","last_unstale":"2024-03-24T00:32:07.634076+0000","last_undegraded":"2024-03-24T00:32:07.634076+0000","last_fullsized":"2024-03-24T00:32:07.634076+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:49.246320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.673582+0000","last_change":"2024-03-24T00:32:06.653499+0000","last_active":"2024-03-24T00:32:06.673582+0000","last_peered":"2024-03-24T00:32:06.673582+0000","last_clean":"2024-03-24T00:32:06.673582+0000","last_became_active":"2024-03-24T00:32:06.653243+0000","last_became_peered":"2024-03-24T00:32:06.653243+0000","last_unstale":"2024-03-24T00:32:06.673582+0000","last_undegraded":"2024-03-24T00:32:06.673582+0000","last_fullsized":"2024-03-24T00:32:06.673582+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:02:54.558428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632281+0000","last_change":"2024-03-24T00:32:05.648901+0000","last_active":"2024-03-24T00:32:07.632281+0000","last_peered":"2024-03-24T00:32:07.632281+0000","last_clean":"2024-03-24T00:32:07.632281+0000","last_became_active":"2024-03-24T00:32:05.647007+0000","last_became_peered":"2024-03-24T00:32:05.647007+0000","last_unstale":"2024-03-24T00:32:07.632281+0000","last_undegraded":"2024-03-24T00:32:07.632281+0000","last_fullsized":"2024-03-24T00:32:07.632281+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:12.531540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.646114+0000","last_change":"2024-03-24T00:32:06.638414+0000","last_active":"2024-03-24T00:32:06.646114+0000","last_peered":"2024-03-24T00:32:06.646114+0000","last_clean":"2024-03-24T00:32:06.646114+0000","last_became_active":"2024-03-24T00:32:06.637451+0000","last_became_peered":"2024-03-24T00:32:06.637451+0000","last_unstale":"2024-03-24T00:32:06.646114+0000","last_undegraded":"2024-03-24T00:32:06.646114+0000","last_fullsized":"2024-03-24T00:32:06.646114+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:12:07.307412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.631378+0000","last_change":"2024-03-24T00:32:05.652002+0000","last_active":"2024-03-24T00:32:06.631378+0000","last_peered":"2024-03-24T00:32:06.631378+0000","last_clean":"2024-03-24T00:32:06.631378+0000","last_became_active":"2024-03-24T00:32:05.651746+0000","last_became_peered":"2024-03-24T00:32:05.651746+0000","last_unstale":"2024-03-24T00:32:06.631378+0000","last_undegraded":"2024-03-24T00:32:06.631378+0000","last_fullsized":"2024-03-24T00:32:06.631378+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:29.996970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.661547+0000","last_change":"2024-03-24T00:32:06.656314+0000","last_active":"2024-03-24T00:32:06.661547+0000","last_peered":"2024-03-24T00:32:06.661547+0000","last_clean":"2024-03-24T00:32:06.661547+0000","last_became_active":"2024-03-24T00:32:06.654864+0000","last_became_peered":"2024-03-24T00:32:06.654864+0000","last_unstale":"2024-03-24T00:32:06.661547+0000","last_undegraded":"2024-03-24T00:32:06.661547+0000","last_fullsized":"2024-03-24T00:32:06.661547+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:42.429755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":20,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.678041+0000","last_change":"2024-03-24T00:32:05.645461+0000","last_active":"2024-03-24T00:32:06.678041+0000","last_peered":"2024-03-24T00:32:06.678041+0000","last_clean":"2024-03-24T00:32:06.678041+0000","last_became_active":"2024-03-24T00:32:05.645153+0000","last_became_peered":"2024-03-24T00:32:05.645153+0000","last_unstale":"2024-03-24T00:32:06.678041+0000","last_undegraded":"2024-03-24T00:32:06.678041+0000","last_fullsized":"2024-03-24T00:32:06.678041+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:36.528217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634446+0000","last_change":"2024-03-24T00:32:06.656235+0000","last_active":"2024-03-24T00:32:07.634446+0000","last_peered":"2024-03-24T00:32:07.634446+0000","last_clean":"2024-03-24T00:32:07.634446+0000","last_became_active":"2024-03-24T00:32:06.655171+0000","last_became_peered":"2024-03-24T00:32:06.655171+0000","last_unstale":"2024-03-24T00:32:07.634446+0000","last_undegraded":"2024-03-24T00:32:07.634446+0000","last_fullsized":"2024-03-24T00:32:07.634446+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:40.821676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634398+0000","last_change":"2024-03-24T00:32:05.645160+0000","last_active":"2024-03-24T00:32:07.634398+0000","last_peered":"2024-03-24T00:32:07.634398+0000","last_clean":"2024-03-24T00:32:07.634398+0000","last_became_active":"2024-03-24T00:32:05.644878+0000","last_became_peered":"2024-03-24T00:32:05.644878+0000","last_unstale":"2024-03-24T00:32:07.634398+0000","last_undegraded":"2024-03-24T00:32:07.634398+0000","last_fullsized":"2024-03-24T00:32:07.634398+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:58:21.337093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730696+0000","last_change":"2024-03-24T00:32:06.653722+0000","last_active":"2024-03-24T00:32:07.730696+0000","last_peered":"2024-03-24T00:32:07.730696+0000","last_clean":"2024-03-24T00:32:07.730696+0000","last_became_active":"2024-03-24T00:32:06.653426+0000","last_became_peered":"2024-03-24T00:32:06.653426+0000","last_unstale":"2024-03-24T00:32:07.730696+0000","last_undegraded":"2024-03-24T00:32:07.730696+0000","last_fullsized":"2024-03-24T00:32:07.730696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:03.730126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634385+0000","last_change":"2024-03-24T00:32:05.657809+0000","last_active":"2024-03-24T00:32:07.634385+0000","last_peered":"2024-03-24T00:32:07.634385+0000","last_clean":"2024-03-24T00:32:07.634385+0000","last_became_active":"2024-03-24T00:32:05.657401+0000","last_became_peered":"2024-03-24T00:32:05.657401+0000","last_unstale":"2024-03-24T00:32:07.634385+0000","last_undegraded":"2024-03-24T00:32:07.634385+0000","last_fullsized":"2024-03-24T00:32:07.634385+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:05.121138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634270+0000","last_change":"2024-03-24T00:32:06.649734+0000","last_active":"2024-03-24T00:32:07.634270+0000","last_peered":"2024-03-24T00:32:07.634270+0000","last_clean":"2024-03-24T00:32:07.634270+0000","last_became_active":"2024-03-24T00:32:06.644029+0000","last_became_peered":"2024-03-24T00:32:06.644029+0000","last_unstale":"2024-03-24T00:32:07.634270+0000","last_undegraded":"2024-03-24T00:32:07.634270+0000","last_fullsized":"2024-03-24T00:32:07.634270+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:40.821676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634380+0000","last_change":"2024-03-24T00:32:05.657590+0000","last_active":"2024-03-24T00:32:07.634380+0000","last_peered":"2024-03-24T00:32:07.634380+0000","last_clean":"2024-03-24T00:32:07.634380+0000","last_became_active":"2024-03-24T00:32:05.657315+0000","last_became_peered":"2024-03-24T00:32:05.657315+0000","last_unstale":"2024-03-24T00:32:07.634380+0000","last_undegraded":"2024-03-24T00:32:07.634380+0000","last_fullsized":"2024-03-24T00:32:07.634380+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:53:57.075494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.668376+0000","last_change":"2024-03-24T00:32:06.653459+0000","last_active":"2024-03-24T00:32:06.668376+0000","last_peered":"2024-03-24T00:32:06.668376+0000","last_clean":"2024-03-24T00:32:06.668376+0000","last_became_active":"2024-03-24T00:32:06.653252+0000","last_became_peered":"2024-03-24T00:32:06.653252+0000","last_unstale":"2024-03-24T00:32:06.668376+0000","last_undegraded":"2024-03-24T00:32:06.668376+0000","last_fullsized":"2024-03-24T00:32:06.668376+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:06.608312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634183+0000","last_change":"2024-03-24T00:32:05.646552+0000","last_active":"2024-03-24T00:32:07.634183+0000","last_peered":"2024-03-24T00:32:07.634183+0000","last_clean":"2024-03-24T00:32:07.634183+0000","last_became_active":"2024-03-24T00:32:05.646216+0000","last_became_peered":"2024-03-24T00:32:05.646216+0000","last_unstale":"2024-03-24T00:32:07.634183+0000","last_undegraded":"2024-03-24T00:32:07.634183+0000","last_fullsized":"2024-03-24T00:32:07.634183+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:17:25.727401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.666159+0000","last_change":"2024-03-24T00:32:06.647559+0000","last_active":"2024-03-24T00:32:06.666159+0000","last_peered":"2024-03-24T00:32:06.666159+0000","last_clean":"2024-03-24T00:32:06.666159+0000","last_became_active":"2024-03-24T00:32:06.647238+0000","last_became_peered":"2024-03-24T00:32:06.647238+0000","last_unstale":"2024-03-24T00:32:06.666159+0000","last_undegraded":"2024-03-24T00:32:06.666159+0000","last_fullsized":"2024-03-24T00:32:06.666159+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:33.350112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633395+0000","last_change":"2024-03-24T00:32:05.644751+0000","last_active":"2024-03-24T00:32:07.633395+0000","last_peered":"2024-03-24T00:32:07.633395+0000","last_clean":"2024-03-24T00:32:07.633395+0000","last_became_active":"2024-03-24T00:32:05.644467+0000","last_became_peered":"2024-03-24T00:32:05.644467+0000","last_unstale":"2024-03-24T00:32:07.633395+0000","last_undegraded":"2024-03-24T00:32:07.633395+0000","last_fullsized":"2024-03-24T00:32:07.633395+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:53:57.075494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.672955+0000","last_change":"2024-03-24T00:32:06.660627+0000","last_active":"2024-03-24T00:32:06.672955+0000","last_peered":"2024-03-24T00:32:06.672955+0000","last_clean":"2024-03-24T00:32:06.672955+0000","last_became_active":"2024-03-24T00:32:06.660052+0000","last_became_peered":"2024-03-24T00:32:06.660052+0000","last_unstale":"2024-03-24T00:32:06.672955+0000","last_undegraded":"2024-03-24T00:32:06.672955+0000","last_fullsized":"2024-03-24T00:32:06.672955+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:06.131702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.643285+0000","last_change":"2024-03-24T00:32:05.656238+0000","last_active":"2024-03-24T00:32:06.643285+0000","last_peered":"2024-03-24T00:32:06.643285+0000","last_clean":"2024-03-24T00:32:06.643285+0000","last_became_active":"2024-03-24T00:32:05.655859+0000","last_became_peered":"2024-03-24T00:32:05.655859+0000","last_unstale":"2024-03-24T00:32:06.643285+0000","last_undegraded":"2024-03-24T00:32:06.643285+0000","last_fullsized":"2024-03-24T00:32:06.643285+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:12.531540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.669514+0000","last_change":"2024-03-24T00:32:06.658344+0000","last_active":"2024-03-24T00:32:06.669514+0000","last_peered":"2024-03-24T00:32:06.669514+0000","last_clean":"2024-03-24T00:32:06.669514+0000","last_became_active":"2024-03-24T00:32:06.658079+0000","last_became_peered":"2024-03-24T00:32:06.658079+0000","last_unstale":"2024-03-24T00:32:06.669514+0000","last_undegraded":"2024-03-24T00:32:06.669514+0000","last_fullsized":"2024-03-24T00:32:06.669514+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:52:24.185706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633756+0000","last_change":"2024-03-24T00:32:05.649241+0000","last_active":"2024-03-24T00:32:07.633756+0000","last_peered":"2024-03-24T00:32:07.633756+0000","last_clean":"2024-03-24T00:32:07.633756+0000","last_became_active":"2024-03-24T00:32:05.648932+0000","last_became_peered":"2024-03-24T00:32:05.648932+0000","last_unstale":"2024-03-24T00:32:07.633756+0000","last_undegraded":"2024-03-24T00:32:07.633756+0000","last_fullsized":"2024-03-24T00:32:07.633756+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:12.531540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.663545+0000","last_change":"2024-03-24T00:32:06.657464+0000","last_active":"2024-03-24T00:32:06.663545+0000","last_peered":"2024-03-24T00:32:06.663545+0000","last_clean":"2024-03-24T00:32:06.663545+0000","last_became_active":"2024-03-24T00:32:06.657123+0000","last_became_peered":"2024-03-24T00:32:06.657123+0000","last_unstale":"2024-03-24T00:32:06.663545+0000","last_undegraded":"2024-03-24T00:32:06.663545+0000","last_fullsized":"2024-03-24T00:32:06.663545+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:42.860138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730127+0000","last_change":"2024-03-24T00:32:05.648699+0000","last_active":"2024-03-24T00:32:07.730127+0000","last_peered":"2024-03-24T00:32:07.730127+0000","last_clean":"2024-03-24T00:32:07.730127+0000","last_became_active":"2024-03-24T00:32:05.648370+0000","last_became_peered":"2024-03-24T00:32:05.648370+0000","last_unstale":"2024-03-24T00:32:07.730127+0000","last_undegraded":"2024-03-24T00:32:07.730127+0000","last_fullsized":"2024-03-24T00:32:07.730127+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:12:06.296848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.668319+0000","last_change":"2024-03-24T00:32:06.657811+0000","last_active":"2024-03-24T00:32:06.668319+0000","last_peered":"2024-03-24T00:32:06.668319+0000","last_clean":"2024-03-24T00:32:06.668319+0000","last_became_active":"2024-03-24T00:32:06.657517+0000","last_became_peered":"2024-03-24T00:32:06.657517+0000","last_unstale":"2024-03-24T00:32:06.668319+0000","last_undegraded":"2024-03-24T00:32:06.668319+0000","last_fullsized":"2024-03-24T00:32:06.668319+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:28:50.256884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634412+0000","last_change":"2024-03-24T00:32:05.650164+0000","last_active":"2024-03-24T00:32:07.634412+0000","last_peered":"2024-03-24T00:32:07.634412+0000","last_clean":"2024-03-24T00:32:07.634412+0000","last_became_active":"2024-03-24T00:32:05.649844+0000","last_became_peered":"2024-03-24T00:32:05.649844+0000","last_unstale":"2024-03-24T00:32:07.634412+0000","last_undegraded":"2024-03-24T00:32:07.634412+0000","last_fullsized":"2024-03-24T00:32:07.634412+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:29.996970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.730714+0000","last_change":"2024-03-24T00:32:06.662089+0000","last_active":"2024-03-24T00:32:07.730714+0000","last_peered":"2024-03-24T00:32:07.730714+0000","last_clean":"2024-03-24T00:32:07.730714+0000","last_became_active":"2024-03-24T00:32:06.661764+0000","last_became_peered":"2024-03-24T00:32:06.661764+0000","last_unstale":"2024-03-24T00:32:07.730714+0000","last_undegraded":"2024-03-24T00:32:07.730714+0000","last_fullsized":"2024-03-24T00:32:07.730714+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:01:12.164533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.650070+0000","last_change":"2024-03-24T00:32:05.645138+0000","last_active":"2024-03-24T00:32:07.650070+0000","last_peered":"2024-03-24T00:32:07.650070+0000","last_clean":"2024-03-24T00:32:07.650070+0000","last_became_active":"2024-03-24T00:32:05.644628+0000","last_became_peered":"2024-03-24T00:32:05.644628+0000","last_unstale":"2024-03-24T00:32:07.650070+0000","last_undegraded":"2024-03-24T00:32:07.650070+0000","last_fullsized":"2024-03-24T00:32:07.650070+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:51.364923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632808+0000","last_change":"2024-03-24T00:32:06.670021+0000","last_active":"2024-03-24T00:32:07.632808+0000","last_peered":"2024-03-24T00:32:07.632808+0000","last_clean":"2024-03-24T00:32:07.632808+0000","last_became_active":"2024-03-24T00:32:06.669721+0000","last_became_peered":"2024-03-24T00:32:06.669721+0000","last_unstale":"2024-03-24T00:32:07.632808+0000","last_undegraded":"2024-03-24T00:32:07.632808+0000","last_fullsized":"2024-03-24T00:32:07.632808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:31.007534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.634261+0000","last_change":"2024-03-24T00:32:05.649804+0000","last_active":"2024-03-24T00:32:07.634261+0000","last_peered":"2024-03-24T00:32:07.634261+0000","last_clean":"2024-03-24T00:32:07.634261+0000","last_became_active":"2024-03-24T00:32:05.649434+0000","last_became_peered":"2024-03-24T00:32:05.649434+0000","last_unstale":"2024-03-24T00:32:07.634261+0000","last_undegraded":"2024-03-24T00:32:07.634261+0000","last_fullsized":"2024-03-24T00:32:07.634261+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:41.419190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.665359+0000","last_change":"2024-03-24T00:32:06.658671+0000","last_active":"2024-03-24T00:32:06.665359+0000","last_peered":"2024-03-24T00:32:06.665359+0000","last_clean":"2024-03-24T00:32:06.665359+0000","last_became_active":"2024-03-24T00:32:06.658399+0000","last_became_peered":"2024-03-24T00:32:06.658399+0000","last_unstale":"2024-03-24T00:32:06.665359+0000","last_undegraded":"2024-03-24T00:32:06.665359+0000","last_fullsized":"2024-03-24T00:32:06.665359+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:03.706180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633473+0000","last_change":"2024-03-24T00:32:05.648874+0000","last_active":"2024-03-24T00:32:07.633473+0000","last_peered":"2024-03-24T00:32:07.633473+0000","last_clean":"2024-03-24T00:32:07.633473+0000","last_became_active":"2024-03-24T00:32:05.648513+0000","last_became_peered":"2024-03-24T00:32:05.648513+0000","last_unstale":"2024-03-24T00:32:07.633473+0000","last_undegraded":"2024-03-24T00:32:07.633473+0000","last_fullsized":"2024-03-24T00:32:07.633473+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:52:14.290725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633346+0000","last_change":"2024-03-24T00:32:06.655552+0000","last_active":"2024-03-24T00:32:07.633346+0000","last_peered":"2024-03-24T00:32:07.633346+0000","last_clean":"2024-03-24T00:32:07.633346+0000","last_became_active":"2024-03-24T00:32:06.655258+0000","last_became_peered":"2024-03-24T00:32:06.655258+0000","last_unstale":"2024-03-24T00:32:07.633346+0000","last_undegraded":"2024-03-24T00:32:07.633346+0000","last_fullsized":"2024-03-24T00:32:07.633346+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:01.983342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.632368+0000","last_change":"2024-03-24T00:32:05.649402+0000","last_active":"2024-03-24T00:32:07.632368+0000","last_peered":"2024-03-24T00:32:07.632368+0000","last_clean":"2024-03-24T00:32:07.632368+0000","last_became_active":"2024-03-24T00:32:05.648624+0000","last_became_peered":"2024-03-24T00:32:05.648624+0000","last_unstale":"2024-03-24T00:32:07.632368+0000","last_undegraded":"2024-03-24T00:32:07.632368+0000","last_fullsized":"2024-03-24T00:32:07.632368+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:53:57.075494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.669670+0000","last_change":"2024-03-24T00:32:06.650297+0000","last_active":"2024-03-24T00:32:06.669670+0000","last_peered":"2024-03-24T00:32:06.669670+0000","last_clean":"2024-03-24T00:32:06.669670+0000","last_became_active":"2024-03-24T00:32:06.649951+0000","last_became_peered":"2024-03-24T00:32:06.649951+0000","last_unstale":"2024-03-24T00:32:06.669670+0000","last_undegraded":"2024-03-24T00:32:06.669670+0000","last_fullsized":"2024-03-24T00:32:06.669670+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:42:24.998068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633885+0000","last_change":"2024-03-24T00:32:05.652068+0000","last_active":"2024-03-24T00:32:07.633885+0000","last_peered":"2024-03-24T00:32:07.633885+0000","last_clean":"2024-03-24T00:32:07.633885+0000","last_became_active":"2024-03-24T00:32:05.651791+0000","last_became_peered":"2024-03-24T00:32:05.651791+0000","last_unstale":"2024-03-24T00:32:07.633885+0000","last_undegraded":"2024-03-24T00:32:07.633885+0000","last_fullsized":"2024-03-24T00:32:07.633885+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:52:14.290725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-24T00:32:07.633805+0000","last_change":"2024-03-24T00:32:06.639843+0000","last_active":"2024-03-24T00:32:07.633805+0000","last_peered":"2024-03-24T00:32:07.633805+0000","last_clean":"2024-03-24T00:32:07.633805+0000","last_became_active":"2024-03-24T00:32:06.639546+0000","last_became_peered":"2024-03-24T00:32:06.639546+0000","last_unstale":"2024-03-24T00:32:07.633805+0000","last_undegraded":"2024-03-24T00:32:07.633805+0000","last_fullsized":"2024-03-24T00:32:07.633805+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:01.983342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.633080+0000","last_change":"2024-03-24T00:32:05.648650+0000","last_active":"2024-03-24T00:32:06.633080+0000","last_peered":"2024-03-24T00:32:06.633080+0000","last_clean":"2024-03-24T00:32:06.633080+0000","last_became_active":"2024-03-24T00:32:05.648347+0000","last_became_peered":"2024-03-24T00:32:05.648347+0000","last_unstale":"2024-03-24T00:32:06.633080+0000","last_undegraded":"2024-03-24T00:32:06.633080+0000","last_fullsized":"2024-03-24T00:32:06.633080+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:00.972778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.652978+0000","last_change":"2024-03-24T00:32:06.649251+0000","last_active":"2024-03-24T00:32:06.652978+0000","last_peered":"2024-03-24T00:32:06.652978+0000","last_clean":"2024-03-24T00:32:06.652978+0000","last_became_active":"2024-03-24T00:32:06.648934+0000","last_became_peered":"2024-03-24T00:32:06.648934+0000","last_unstale":"2024-03-24T00:32:06.652978+0000","last_undegraded":"2024-03-24T00:32:06.652978+0000","last_fullsized":"2024-03-24T00:32:06.652978+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:05.619876+0000","last_clean_scrub_stamp":"2024-03-24T00:32:05.619876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:31.007534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.640256+0000","last_change":"2024-03-24T00:32:05.659056+0000","last_active":"2024-03-24T00:32:06.640256+0000","last_peered":"2024-03-24T00:32:06.640256+0000","last_clean":"2024-03-24T00:32:06.640256+0000","last_became_active":"2024-03-24T00:32:05.658712+0000","last_became_peered":"2024-03-24T00:32:05.658712+0000","last_unstale":"2024-03-24T00:32:06.640256+0000","last_undegraded":"2024-03-24T00:32:06.640256+0000","last_fullsized":"2024-03-24T00:32:06.640256+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:04.609312+0000","last_clean_scrub_stamp":"2024-03-24T00:32:04.609312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:52:23.175141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":194,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-03-24T00:32:06.631943+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:32:06.631943+0000","last_peered":"2024-03-24T00:32:06.631943+0000","last_clean":"2024-03-24T00:32:06.631943+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:32:06.631943+0000","last_undegraded":"2024-03-24T00:32:06.631943+0000","last_fullsized":"2024-03-24T00:32:06.631943+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32116,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339724,"statfs":{"total":96636764160,"available":96603877376,"internally_reserved":0,"allocated":958464,"data_stored":730806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31540,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340300,"statfs":{"total":96636764160,"available":96604467200,"internally_reserved":0,"allocated":368640,"data_stored":140964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":147320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":380928,"data_stored":146752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27388,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344452,"statfs":{"total":96636764160,"available":96608718848,"internally_reserved":0,"allocated":376832,"data_stored":141982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":364544,"data_stored":140418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":966656,"data_stored":737016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:32:11.355 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:32:11.355 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:32:11.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.532+0000 7f6301400640 1 Processor -- start 2024-03-24T00:32:11.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.532+0000 7f6301400640 1 -- start start 2024-03-24T00:32:11.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.532+0000 7f6301400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62fc15e110 0x7f62fc161b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.532+0000 7f6301400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f62fc0cf1f0 0x7f62fc0d2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.532+0000 7f6301400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62fc162460 con 0x7f62fc0b9880 2024-03-24T00:32:11.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.532+0000 7f6301400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62fc162740 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.532+0000 7f6301400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc162a20 con 0x7f62fc15e110 2024-03-24T00:32:11.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62fa800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62fc15e110 0x7f62fc161b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62fa800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62fc15e110 0x7f62fc161b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:42086/0 (socket says 172.21.15.40:42086) 2024-03-24T00:32:11.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62fa800640 1 -- 172.21.15.40:0/3930359734 learned_addr learned my addr 172.21.15.40:0/3930359734 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:11.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62fb200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f62fc0cf1f0 0x7f62fc0d2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62f9e00640 1 --1- 172.21.15.40:0/3930359734 >> v1:172.21.15.53:6789/0 conn(0x7f62fc0b9880 0x7f62fc0ceae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:43932/0 (socket says 172.21.15.40:43932) 2024-03-24T00:32:11.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62fa800640 1 -- 172.21.15.40:0/3930359734 >> v2:172.21.15.53:3300/0 conn(0x7f62fc0cf1f0 msgr2=0x7f62fc0d2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:11.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62fa800640 1 --2- 172.21.15.40:0/3930359734 >> v2:172.21.15.53:3300/0 conn(0x7f62fc0cf1f0 0x7f62fc0d2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62fa800640 1 -- 172.21.15.40:0/3930359734 >> v1:172.21.15.53:6789/0 conn(0x7f62fc0b9880 legacy=0x7f62fc0ceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62fa800640 1 -- 172.21.15.40:0/3930359734 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc0d2010 con 0x7f62fc15e110 2024-03-24T00:32:11.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62fb200640 1 --2- 172.21.15.40:0/3930359734 >> v2:172.21.15.53:3300/0 conn(0x7f62fc0cf1f0 0x7f62fc0d2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:11.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62fa800640 1 --2- 172.21.15.40:0/3930359734 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62fc15e110 0x7f62fc161b60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009870 tx=0x7f62ec01c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67c706e9d8132eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:11.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62f9400640 1 -- 172.21.15.40:0/3930359734 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec0ab640 con 0x7f62fc15e110 2024-03-24T00:32:11.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.536+0000 7f62f9400640 1 -- 172.21.15.40:0/3930359734 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62ec0b0070 con 0x7f62fc15e110 2024-03-24T00:32:11.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f62f9400640 1 -- 172.21.15.40:0/3930359734 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec0aba00 con 0x7f62fc15e110 2024-03-24T00:32:11.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 -- 172.21.15.40:0/3930359734 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62fc15e110 msgr2=0x7f62fc161b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 --2- 172.21.15.40:0/3930359734 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62fc15e110 0x7f62fc161b60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009870 tx=0x7f62ec01c530 comp rx=0 tx=0).stop 2024-03-24T00:32:11.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 -- 172.21.15.40:0/3930359734 shutdown_connections 2024-03-24T00:32:11.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 --2- 172.21.15.40:0/3930359734 >> v2:172.21.15.53:3300/0 conn(0x7f62fc0cf1f0 0x7f62fc0d2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 --2- 172.21.15.40:0/3930359734 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62fc15e110 0x7f62fc161b60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 -- 172.21.15.40:0/3930359734 >> 172.21.15.40:0/3930359734 conn(0x7f62fc066e80 msgr2=0x7f62fc067260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:11.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 -- 172.21.15.40:0/3930359734 shutdown_connections 2024-03-24T00:32:11.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 -- 172.21.15.40:0/3930359734 wait complete. 2024-03-24T00:32:11.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 Processor -- start 2024-03-24T00:32:11.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 -- start start 2024-03-24T00:32:11.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62fc0b9880 0x7f62fc157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62fc0cf1f0 0x7f62fc0cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62fc15e110 0x7f62fc1580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc0d38f0 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc0d3770 con 0x7f62fc15e110 2024-03-24T00:32:11.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f6301400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc0d3a70 con 0x7f62fc0b9880 2024-03-24T00:32:11.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f62f9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62fc0cf1f0 0x7f62fc0cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f62f9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62fc0cf1f0 0x7f62fc0cbb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42438/0 (socket says 172.21.15.40:42438) 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f62f9e00640 1 -- 172.21.15.40:0/3522398132 learned_addr learned my addr 172.21.15.40:0/3522398132 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f62fa800640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62fc0b9880 0x7f62fc157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.540+0000 7f62fb200640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62fc15e110 0x7f62fc1580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62f9e00640 1 -- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62fc0b9880 msgr2=0x7f62fc157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62f9e00640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62fc0b9880 0x7f62fc157760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62f9e00640 1 -- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62fc15e110 msgr2=0x7f62fc1580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62f9e00640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62fc15e110 0x7f62fc1580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62f9e00640 1 -- 172.21.15.40:0/3522398132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc0cc490 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62f9e00640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62fc0cf1f0 0x7f62fc0cbb80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f62e800c340 tx=0x7f62e800c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62f2a00640 1 -- 172.21.15.40:0/3522398132 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e80074e0 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62f2a00640 1 -- 172.21.15.40:0/3522398132 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e8007680 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f6301400640 1 -- 172.21.15.40:0/3522398132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62fc0cc7d0 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62f2a00640 1 -- 172.21.15.40:0/3522398132 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e8004a30 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f6301400640 1 -- 172.21.15.40:0/3522398132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62fc0cd010 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f6301400640 1 -- 172.21.15.40:0/3522398132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62fc0cd210 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62f2a00640 1 -- 172.21.15.40:0/3522398132 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f62e801a020 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62f2a00640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f62dc067660 0x7f62dc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62f2a00640 1 -- 172.21.15.40:0/3522398132 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62e8084f80 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.544+0000 7f62fa800640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f62dc067660 0x7f62dc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.548+0000 7f62fa800640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f62dc067660 0x7f62dc069b20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0af000 tx=0x7f62ec0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:11.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.548+0000 7f62f2a00640 1 -- 172.21.15.40:0/3522398132 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f62fc0cd210 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.772+0000 7f6301400640 1 -- 172.21.15.40:0/3522398132 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f62fc0b3550 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f62f2a00640 1 -- 172.21.15.40:0/3522398132 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f62fc0cc7d0 con 0x7f62fc0cf1f0 2024-03-24T00:32:11.781 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-03-24T00:32:11.781 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw== 2024-03-24T00:32:11.781 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-03-24T00:32:11.781 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-03-24T00:32:11.781 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:32:11.782 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-03-24T00:32:11.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f6301400640 1 -- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f62dc067660 msgr2=0x7f62dc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f6301400640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f62dc067660 0x7f62dc069b20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0af000 tx=0x7f62ec0a7000 comp rx=0 tx=0).stop 2024-03-24T00:32:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f6301400640 1 -- 172.21.15.40:0/3522398132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62fc0cf1f0 msgr2=0x7f62fc0cbb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f6301400640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62fc0cf1f0 0x7f62fc0cbb80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f62e800c340 tx=0x7f62e800c740 comp rx=0 tx=0).stop 2024-03-24T00:32:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f6301400640 1 -- 172.21.15.40:0/3522398132 shutdown_connections 2024-03-24T00:32:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f6301400640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f62dc067660 0x7f62dc069b20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f6301400640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62fc15e110 0x7f62fc1580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f6301400640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62fc0cf1f0 0x7f62fc0cbb80 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f6301400640 1 --2- 172.21.15.40:0/3522398132 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62fc0b9880 0x7f62fc157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f6301400640 1 -- 172.21.15.40:0/3522398132 >> 172.21.15.40:0/3522398132 conn(0x7f62fc066e80 msgr2=0x7f62fc0b5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f6301400640 1 -- 172.21.15.40:0/3522398132 shutdown_connections 2024-03-24T00:32:11.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.776+0000 7f6301400640 1 -- 172.21.15.40:0/3522398132 wait complete. 2024-03-24T00:32:11.804 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:32:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.936+0000 7f49c5200640 1 Processor -- start 2024-03-24T00:32:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49c5200640 1 -- start start 2024-03-24T00:32:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49c5200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49c0160030 0x7f49c0070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49c5200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f49c006de00 0x7f49c0071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49c5200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c00cec10 con 0x7f49c0145520 2024-03-24T00:32:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49c5200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c00ceeb0 con 0x7f49c006de00 2024-03-24T00:32:11.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49c5200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c00cf190 con 0x7f49c0160030 2024-03-24T00:32:11.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49bfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f49c006de00 0x7f49c0071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49bfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f49c006de00 0x7f49c0071030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42452/0 (socket says 172.21.15.40:42452) 2024-03-24T00:32:11.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49bfe00640 1 -- 172.21.15.40:0/2074264706 learned_addr learned my addr 172.21.15.40:0/2074264706 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:11.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49bf400640 1 --1- 172.21.15.40:0/2074264706 >> v1:172.21.15.53:6789/0 conn(0x7f49c0145520 0x7f49c015f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:43934/0 (socket says 172.21.15.40:43934) 2024-03-24T00:32:11.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49b6a00640 1 --2- 172.21.15.40:0/2074264706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49c0160030 0x7f49c0070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49bfe00640 1 -- 172.21.15.40:0/2074264706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49c0160030 msgr2=0x7f49c0070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:11.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49bfe00640 1 --2- 172.21.15.40:0/2074264706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49c0160030 0x7f49c0070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49bfe00640 1 -- 172.21.15.40:0/2074264706 >> v1:172.21.15.53:6789/0 conn(0x7f49c0145520 legacy=0x7f49c015f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.940+0000 7f49bfe00640 1 -- 172.21.15.40:0/2074264706 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00cf470 con 0x7f49c006de00 2024-03-24T00:32:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49bfe00640 1 --2- 172.21.15.40:0/2074264706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c006de00 0x7f49c0071030 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f49b8009870 tx=0x7f49b801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802cafa9ead0d283 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49bea00640 1 -- 172.21.15.40:0/2074264706 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b80aa650 con 0x7f49c006de00 2024-03-24T00:32:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49bea00640 1 -- 172.21.15.40:0/2074264706 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b80af070 con 0x7f49c006de00 2024-03-24T00:32:11.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49bea00640 1 -- 172.21.15.40:0/2074264706 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b80aaa10 con 0x7f49c006de00 2024-03-24T00:32:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49c5200640 1 -- 172.21.15.40:0/2074264706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c006de00 msgr2=0x7f49c0071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49c5200640 1 --2- 172.21.15.40:0/2074264706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c006de00 0x7f49c0071030 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f49b8009870 tx=0x7f49b801c4d0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49c5200640 1 -- 172.21.15.40:0/2074264706 shutdown_connections 2024-03-24T00:32:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49c5200640 1 --2- 172.21.15.40:0/2074264706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49c0160030 0x7f49c0070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49c5200640 1 --2- 172.21.15.40:0/2074264706 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c006de00 0x7f49c0071030 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49c5200640 1 -- 172.21.15.40:0/2074264706 >> 172.21.15.40:0/2074264706 conn(0x7f49c00b1740 msgr2=0x7f49c00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49c5200640 1 -- 172.21.15.40:0/2074264706 shutdown_connections 2024-03-24T00:32:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49c5200640 1 -- 172.21.15.40:0/2074264706 wait complete. 2024-03-24T00:32:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49c5200640 1 Processor -- start 2024-03-24T00:32:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.944+0000 7f49c5200640 1 -- start start 2024-03-24T00:32:11.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49c5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c006de00 0x7f49c0142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49c5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0145520 0x7f49c0138f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49c5200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49c0160030 0x7f49c0135170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49c5200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c0145b10 con 0x7f49c006de00 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49c5200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c01607b0 con 0x7f49c0145520 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49c5200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c0071d80 con 0x7f49c0160030 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49bf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c006de00 0x7f49c0142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49bf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c006de00 0x7f49c0142b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42458/0 (socket says 172.21.15.40:42458) 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49bf400640 1 -- 172.21.15.40:0/2016192202 learned_addr learned my addr 172.21.15.40:0/2016192202 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49b6a00640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0145520 0x7f49c0138f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49bfe00640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49c0160030 0x7f49c0135170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49bf400640 1 -- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49c0160030 msgr2=0x7f49c0135170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49bf400640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49c0160030 0x7f49c0135170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49bf400640 1 -- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0145520 msgr2=0x7f49c0138f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:11.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49bf400640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0145520 0x7f49c0138f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:11.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49bf400640 1 -- 172.21.15.40:0/2016192202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c0135a50 con 0x7f49c006de00 2024-03-24T00:32:11.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49bfe00640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49c0160030 0x7f49c0135170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:11.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49bf400640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c006de00 0x7f49c0142b20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f49b000c450 tx=0x7f49b000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:11.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49b7e00640 1 -- 172.21.15.40:0/2016192202 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b0018070 con 0x7f49c006de00 2024-03-24T00:32:11.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49b7e00640 1 -- 172.21.15.40:0/2016192202 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b0007480 con 0x7f49c006de00 2024-03-24T00:32:11.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49b7e00640 1 -- 172.21.15.40:0/2016192202 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b00047f0 con 0x7f49c006de00 2024-03-24T00:32:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49c5200640 1 -- 172.21.15.40:0/2016192202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c0135d30 con 0x7f49c006de00 2024-03-24T00:32:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49c5200640 1 -- 172.21.15.40:0/2016192202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c01362f0 con 0x7f49c006de00 2024-03-24T00:32:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.948+0000 7f49c5200640 1 -- 172.21.15.40:0/2016192202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c0073540 con 0x7f49c006de00 2024-03-24T00:32:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.952+0000 7f49b7e00640 1 -- 172.21.15.40:0/2016192202 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f49b001b070 con 0x7f49c006de00 2024-03-24T00:32:11.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.952+0000 7f49b7e00640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f499c067660 0x7f499c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:11.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.952+0000 7f49b7e00640 1 -- 172.21.15.40:0/2016192202 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49b0084b50 con 0x7f49c006de00 2024-03-24T00:32:11.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.952+0000 7f49b6a00640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f499c067660 0x7f499c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:11.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.952+0000 7f49b6a00640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f499c067660 0x7f499c069b20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f49c0137570 tx=0x7f49a800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:11.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:11.956+0000 7f49b7e00640 1 -- 172.21.15.40:0/2016192202 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f49b0055050 con 0x7f49c006de00 2024-03-24T00:32:12.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 -- 172.21.15.40:0/2016192202 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49c004f4f0 con 0x7f49c006de00 2024-03-24T00:32:12.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49b7e00640 1 -- 172.21.15.40:0/2016192202 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4977 (secure 0 0 0) 0x7f49b00590f0 con 0x7f49c006de00 2024-03-24T00:32:12.158 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:12.158 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2069836835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2069836835},{"type":"v1","addr":"172.21.15.53:6836","nonce":2069836835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4856,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2159849028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":2159849028},{"type":"v1","addr":"172.21.15.53:6837","nonce":2159849028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5226,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2326157501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2326157501},{"type":"v1","addr":"172.21.15.40:6833","nonce":2326157501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:06.028086+0000","modified":"2024-03-24T00:32:09.162520+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5229":{"gid":5229,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3050269371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3050269371},{"type":"v1","addr":"172.21.15.40:6835","nonce":3050269371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5229,"qdb_cluster":[5229]},"id":2}]} 2024-03-24T00:32:12.158 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 16 2024-03-24T00:32:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 -- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f499c067660 msgr2=0x7f499c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f499c067660 0x7f499c069b20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f49c0137570 tx=0x7f49a800b040 comp rx=0 tx=0).stop 2024-03-24T00:32:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 -- 172.21.15.40:0/2016192202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c006de00 msgr2=0x7f49c0142b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c006de00 0x7f49c0142b20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f49b000c450 tx=0x7f49b000c910 comp rx=0 tx=0).stop 2024-03-24T00:32:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 -- 172.21.15.40:0/2016192202 shutdown_connections 2024-03-24T00:32:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f499c067660 0x7f499c069b20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49c0160030 0x7f49c0135170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49c0145520 0x7f49c0138f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 --2- 172.21.15.40:0/2016192202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f49c006de00 0x7f49c0142b20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 -- 172.21.15.40:0/2016192202 >> 172.21.15.40:0/2016192202 conn(0x7f49c00b1740 msgr2=0x7f49c00b0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:12.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 -- 172.21.15.40:0/2016192202 shutdown_connections 2024-03-24T00:32:12.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.152+0000 7f49c5200640 1 -- 172.21.15.40:0/2016192202 wait complete. 2024-03-24T00:32:12.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:32:06.028086+0000', 'modified': '2024-03-24T00:32:09.162520+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5229}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5229': {'gid': 5229, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/3050269371', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 3050269371}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 3050269371}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5229, 'qdb_cluster': [5229]} 2024-03-24T00:32:12.581 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:32:12.582 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:32:12.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.716+0000 7f41a0000640 1 Processor -- start 2024-03-24T00:32:12.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.716+0000 7f41a0000640 1 -- start start 2024-03-24T00:32:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f41a0000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41980d0830 0x7f41980cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f41a0000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f419806b150 0x7f419806b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f41a0000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41980d0e10 con 0x7f419815e0b0 2024-03-24T00:32:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f41a0000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41980d10f0 con 0x7f419806b150 2024-03-24T00:32:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f41a0000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419806efd0 con 0x7f41980d0830 2024-03-24T00:32:12.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f419d600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f419815e0b0 0x7f41980cd3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57946/0 (socket says 172.21.15.40:57946) 2024-03-24T00:32:12.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f419d600640 1 -- 172.21.15.40:0/4145331459 learned_addr learned my addr 172.21.15.40:0/4145331459 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:12.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f419e000640 1 --2- 172.21.15.40:0/4145331459 >> v2:172.21.15.53:3300/0 conn(0x7f419806b150 0x7f419806b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f419cc00640 1 --2- 172.21.15.40:0/4145331459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41980d0830 0x7f41980cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f4197e00640 1 -- 172.21.15.40:0/4145331459 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692100022 0 0) 0x7f41980d0e10 con 0x7f419815e0b0 2024-03-24T00:32:12.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f4197e00640 1 -- 172.21.15.40:0/4145331459 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4180003530 con 0x7f419815e0b0 2024-03-24T00:32:12.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f419e000640 1 -- 172.21.15.40:0/4145331459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41980d0830 msgr2=0x7f41980cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:12.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f419e000640 1 --2- 172.21.15.40:0/4145331459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41980d0830 0x7f41980cdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f419e000640 1 -- 172.21.15.40:0/4145331459 >> v1:172.21.15.40:6789/0 conn(0x7f419815e0b0 legacy=0x7f41980cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.720+0000 7f419e000640 1 -- 172.21.15.40:0/4145331459 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419806f2b0 con 0x7f419806b150 2024-03-24T00:32:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f419e000640 1 --2- 172.21.15.40:0/4145331459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f419806b150 0x7f419806b580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4190009d90 tx=0x7f419001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47204d1996cba56d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f4197e00640 1 -- 172.21.15.40:0/4145331459 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41900b2070 con 0x7f419806b150 2024-03-24T00:32:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f4197e00640 1 -- 172.21.15.40:0/4145331459 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41900a5830 con 0x7f419806b150 2024-03-24T00:32:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f4197e00640 1 -- 172.21.15.40:0/4145331459 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41900a5c00 con 0x7f419806b150 2024-03-24T00:32:12.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f41a0000640 1 -- 172.21.15.40:0/4145331459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f419806b150 msgr2=0x7f419806b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f41a0000640 1 --2- 172.21.15.40:0/4145331459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f419806b150 0x7f419806b580 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4190009d90 tx=0x7f419001c970 comp rx=0 tx=0).stop 2024-03-24T00:32:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f41a0000640 1 -- 172.21.15.40:0/4145331459 shutdown_connections 2024-03-24T00:32:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f41a0000640 1 --2- 172.21.15.40:0/4145331459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f419806b150 0x7f419806b580 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f41a0000640 1 --2- 172.21.15.40:0/4145331459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41980d0830 0x7f41980cdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f41a0000640 1 -- 172.21.15.40:0/4145331459 >> 172.21.15.40:0/4145331459 conn(0x7f41980d1c90 msgr2=0x7f41980d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f41a0000640 1 -- 172.21.15.40:0/4145331459 shutdown_connections 2024-03-24T00:32:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f41a0000640 1 -- 172.21.15.40:0/4145331459 wait complete. 2024-03-24T00:32:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f41a0000640 1 Processor -- start 2024-03-24T00:32:12.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.724+0000 7f41a0000640 1 -- start start 2024-03-24T00:32:12.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f41a0000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f419806b150 0x7f419814a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f41a0000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41980d0830 0x7f41980cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f41a0000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f419815e0b0 0x7f41980cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f41a0000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419815e490 con 0x7f419815e0b0 2024-03-24T00:32:12.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f41a0000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41980ce020 con 0x7f419806b150 2024-03-24T00:32:12.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f41a0000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419815e610 con 0x7f41980d0830 2024-03-24T00:32:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f419e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f419815e0b0 0x7f41980cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f419e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f419815e0b0 0x7f41980cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42468/0 (socket says 172.21.15.40:42468) 2024-03-24T00:32:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f419e000640 1 -- 172.21.15.40:0/3469556844 learned_addr learned my addr 172.21.15.40:0/3469556844 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f419d600640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f419806b150 0x7f419814a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f419cc00640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41980d0830 0x7f41980cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f419e000640 1 -- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41980d0830 msgr2=0x7f41980cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f419e000640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41980d0830 0x7f41980cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f419e000640 1 -- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f419806b150 msgr2=0x7f419814a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f419e000640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f419806b150 0x7f419814a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f419e000640 1 -- 172.21.15.40:0/3469556844 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419814d050 con 0x7f419815e0b0 2024-03-24T00:32:12.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f419e000640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f419815e0b0 0x7f41980cccd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4190009d90 tx=0x7f4190019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:12.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f4195600640 1 -- 172.21.15.40:0/3469556844 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41900b2050 con 0x7f419815e0b0 2024-03-24T00:32:12.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f4195600640 1 -- 172.21.15.40:0/3469556844 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41900be070 con 0x7f419815e0b0 2024-03-24T00:32:12.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f4195600640 1 -- 172.21.15.40:0/3469556844 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41900ad890 con 0x7f419815e0b0 2024-03-24T00:32:12.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f41a0000640 1 -- 172.21.15.40:0/3469556844 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419814d330 con 0x7f419815e0b0 2024-03-24T00:32:12.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f41a0000640 1 -- 172.21.15.40:0/3469556844 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41980b5a50 con 0x7f419815e0b0 2024-03-24T00:32:12.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.728+0000 7f4195600640 1 -- 172.21.15.40:0/3469556844 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f41900ada30 con 0x7f419815e0b0 2024-03-24T00:32:12.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f4195600640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4178067660 0x7f4178069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f4195600640 1 -- 172.21.15.40:0/3469556844 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4190120900 con 0x7f419815e0b0 2024-03-24T00:32:12.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419d600640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4178067660 0x7f4178069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f41a3585000 1 Processor -- start 2024-03-24T00:32:12.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f41a3585000 1 -- start start 2024-03-24T00:32:12.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419d600640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4178067660 0x7f4178069b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f418c005890 tx=0x7f418c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:12.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f41a3585000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b815920 0x55a12b815d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f41a3585000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b816240 0x55a12b7cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f41a3585000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7cbff0 0x55a12b7cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f41a3585000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12b619680 con 0x55a12b815920 2024-03-24T00:32:12.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f41a3585000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12b5a0b00 con 0x55a12b816240 2024-03-24T00:32:12.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f41a3585000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a12b615720 con 0x55a12b7cbff0 2024-03-24T00:32:12.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b815920 0x55a12b815d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b815920 0x55a12b815d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42474/0 (socket says 172.21.15.40:42474) 2024-03-24T00:32:12.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419d600640 1 -- 172.21.15.40:0/1672713034 learned_addr learned my addr 172.21.15.40:0/1672713034 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:12.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419cc00640 1 --2- 172.21.15.40:0/1672713034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b816240 0x55a12b7cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419e000640 1 --2- 172.21.15.40:0/1672713034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7cbff0 0x55a12b7cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419d600640 1 -- 172.21.15.40:0/1672713034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7cbff0 msgr2=0x55a12b7cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:12.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419d600640 1 --2- 172.21.15.40:0/1672713034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7cbff0 0x55a12b7cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419d600640 1 -- 172.21.15.40:0/1672713034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b816240 msgr2=0x55a12b7cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:12.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419d600640 1 --2- 172.21.15.40:0/1672713034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b816240 0x55a12b7cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419d600640 1 -- 172.21.15.40:0/1672713034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a12b5f1940 con 0x55a12b815920 2024-03-24T00:32:12.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f419d600640 1 --2- 172.21.15.40:0/1672713034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b815920 0x55a12b815d00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f418c0058c0 tx=0x7f418c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ebece7d06ef444 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:12.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f418a000640 1 -- 172.21.15.40:0/1672713034 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c005e40 con 0x55a12b815920 2024-03-24T00:32:12.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f418a000640 1 -- 172.21.15.40:0/1672713034 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c00e7b0 con 0x55a12b815920 2024-03-24T00:32:12.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.732+0000 7f418a000640 1 -- 172.21.15.40:0/1672713034 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c00eb60 con 0x55a12b815920 2024-03-24T00:32:12.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 -- 172.21.15.40:0/1672713034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b815920 msgr2=0x55a12b815d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 --2- 172.21.15.40:0/1672713034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b815920 0x55a12b815d00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f418c0058c0 tx=0x7f418c011000 comp rx=0 tx=0).stop 2024-03-24T00:32:12.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 -- 172.21.15.40:0/1672713034 shutdown_connections 2024-03-24T00:32:12.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 --2- 172.21.15.40:0/1672713034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7cbff0 0x55a12b7cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 --2- 172.21.15.40:0/1672713034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b816240 0x55a12b7cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 --2- 172.21.15.40:0/1672713034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b815920 0x55a12b815d00 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 -- 172.21.15.40:0/1672713034 >> 172.21.15.40:0/1672713034 conn(0x55a12b7c7fa0 msgr2=0x55a12b7ca390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:12.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 -- 172.21.15.40:0/1672713034 shutdown_connections 2024-03-24T00:32:12.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 -- 172.21.15.40:0/1672713034 wait complete. 2024-03-24T00:32:12.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 -- start start 2024-03-24T00:32:12.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 Processor -- start 2024-03-24T00:32:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7cbff0 0x55a12b7fad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b815920 0x55a12b7fb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b816240 0x55a12b7fb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12b618990 con 0x55a12b816240 2024-03-24T00:32:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12b5d3ae0 con 0x55a12b7cbff0 2024-03-24T00:32:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a12b5b1750 con 0x55a12b815920 2024-03-24T00:32:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 10 client.0 _ll_register_callbacks cb 0x55a12b7b6ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:32:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f419e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b816240 0x55a12b7fb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f419e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b816240 0x55a12b7fb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42488/0 (socket says 172.21.15.40:42488) 2024-03-24T00:32:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f419e000640 1 -- 172.21.15.40:0/2396590310 learned_addr learned my addr 172.21.15.40:0/2396590310 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f419d600640 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7cbff0 0x55a12b7fad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f419cc00640 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b815920 0x55a12b7fb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f419e000640 1 -- 172.21.15.40:0/2396590310 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b815920 msgr2=0x55a12b7fb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f419e000640 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b815920 0x55a12b7fb280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f419e000640 1 -- 172.21.15.40:0/2396590310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7cbff0 msgr2=0x55a12b7fad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f419e000640 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7cbff0 0x55a12b7fad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f419e000640 1 -- 172.21.15.40:0/2396590310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a12b65ed80 con 0x55a12b816240 2024-03-24T00:32:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f419cc00640 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b815920 0x55a12b7fb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f419e000640 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b816240 0x55a12b7fb7c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f419001ce50 tx=0x7f41900f14e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248a189eba84e218 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f417ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:32:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f417ea00640 1 -- 172.21.15.40:0/2396590310 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41900f94f0 con 0x55a12b816240 2024-03-24T00:32:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f417ea00640 1 -- 172.21.15.40:0/2396590310 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41900ed450 con 0x55a12b816240 2024-03-24T00:32:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 -- 172.21.15.40:0/2396590310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f418c00e7b0 con 0x55a12b816240 2024-03-24T00:32:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f417ea00640 1 -- 172.21.15.40:0/2396590310 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f419010f120 con 0x55a12b816240 2024-03-24T00:32:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 10 client.5403 fetch_fsmap learned FSMap version 16 2024-03-24T00:32:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f41a3585000 1 -- 172.21.15.40:0/2396590310 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55a12b5a0d10 con 0x55a12b816240 2024-03-24T00:32:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.736+0000 7f417ea00640 1 -- 172.21.15.40:0/2396590310 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c00e7b0 con 0x55a12b816240 2024-03-24T00:32:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f417ea00640 1 -- 172.21.15.40:0/2396590310 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 16) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f41900f9060 con 0x55a12b816240 2024-03-24T00:32:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 10 client.5403 fetch_fsmap finished waiting for FSMap version 16 2024-03-24T00:32:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 10 client.5403 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:32:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 20 client.5403 populate_metadata read hostname 'smithi040' 2024-03-24T00:32:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x55a12b7fc030 0x55a12b827ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 4 client.5403 mds_command: new command op to 5229 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:32:12.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 -- 172.21.15.40:0/2396590310 --> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a12b5b4b60 con 0x55a12b7fc030 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f419d600640 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x55a12b7fc030 0x55a12b827ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f419d600640 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x55a12b7fc030 0x55a12b827ea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=103b53a30cf2e86b server_cookie=2ca860971219e1d in_seq=0 out_seq=0 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f417ea00640 10 client.5403 ms_handle_connect on v2:172.21.15.40:6834/3050269371 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f417ea00640 1 -- 172.21.15.40:0/2396590310 <== mds.0 v2:172.21.15.40:6834/3050269371 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a12b5b4b60 con 0x55a12b7fc030 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f417ea00640 10 client.5403 handle_command_reply: tid=0 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 client.5403 shutdown 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 -- 172.21.15.40:0/2396590310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b816240 msgr2=0x55a12b7fb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b816240 0x55a12b7fb7c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f419001ce50 tx=0x7f41900f14e0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 -- 172.21.15.40:0/2396590310 shutdown_connections 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x55a12b7fc030 0x55a12b827ea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b816240 0x55a12b7fb7c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b815920 0x55a12b7fb280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 --2- 172.21.15.40:0/2396590310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7cbff0 0x55a12b7fad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f417ea00640 0 client.5403 ms_handle_reset on v2:172.21.15.40:6834/3050269371 2024-03-24T00:32:12.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 -- 172.21.15.40:0/2396590310 >> 172.21.15.40:0/2396590310 conn(0x55a12b7c7fa0 msgr2=0x55a12b7cac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:12.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 -- 172.21.15.40:0/2396590310 shutdown_connections 2024-03-24T00:32:12.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 1 -- 172.21.15.40:0/2396590310 wait complete. 2024-03-24T00:32:12.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.740+0000 7f41a3585000 20 client.5403 trim_cache size 0 max 16384 2024-03-24T00:32:12.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 Processor -- start 2024-03-24T00:32:12.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 -- start start 2024-03-24T00:32:12.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7faed0 0x55a12b7cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7c7fa0 0x55a12b8144d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7cbff0 0x55a12b814a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4198203670 con 0x55a12b7faed0 2024-03-24T00:32:12.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41982027d0 con 0x55a12b7cbff0 2024-03-24T00:32:12.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419814c6d0 con 0x55a12b7c7fa0 2024-03-24T00:32:12.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f419d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7faed0 0x55a12b7cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f419d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7faed0 0x55a12b7cb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42504/0 (socket says 172.21.15.40:42504) 2024-03-24T00:32:12.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f419d600640 1 -- 172.21.15.40:0/97465738 learned_addr learned my addr 172.21.15.40:0/97465738 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:12.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f419d600640 1 -- 172.21.15.40:0/97465738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7c7fa0 msgr2=0x55a12b8144d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:12.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f419d600640 1 --2- 172.21.15.40:0/97465738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7c7fa0 0x55a12b8144d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f419d600640 1 -- 172.21.15.40:0/97465738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7cbff0 msgr2=0x55a12b814a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:12.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f419e000640 1 --2- 172.21.15.40:0/97465738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7cbff0 0x55a12b814a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f419d600640 1 --2- 172.21.15.40:0/97465738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7cbff0 0x55a12b814a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f419d600640 1 -- 172.21.15.40:0/97465738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a12b5e6540 con 0x55a12b7faed0 2024-03-24T00:32:12.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f419e000640 1 --2- 172.21.15.40:0/97465738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7cbff0 0x55a12b814a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:12.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f419d600640 1 --2- 172.21.15.40:0/97465738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7faed0 0x55a12b7cb7c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x55a12b2d5860 tx=0x7f418c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef56b8968fad06e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:12.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f4189600640 1 -- 172.21.15.40:0/97465738 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c007860 con 0x55a12b7faed0 2024-03-24T00:32:12.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f4189600640 1 -- 172.21.15.40:0/97465738 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c005e40 con 0x55a12b7faed0 2024-03-24T00:32:12.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f4189600640 1 -- 172.21.15.40:0/97465738 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c02f2b0 con 0x55a12b7faed0 2024-03-24T00:32:12.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 -- 172.21.15.40:0/97465738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7faed0 msgr2=0x55a12b7cb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 --2- 172.21.15.40:0/97465738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7faed0 0x55a12b7cb7c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x55a12b2d5860 tx=0x7f418c013000 comp rx=0 tx=0).stop 2024-03-24T00:32:12.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 -- 172.21.15.40:0/97465738 shutdown_connections 2024-03-24T00:32:12.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 --2- 172.21.15.40:0/97465738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7cbff0 0x55a12b814a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 --2- 172.21.15.40:0/97465738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7c7fa0 0x55a12b8144d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 --2- 172.21.15.40:0/97465738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7faed0 0x55a12b7cb7c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 -- 172.21.15.40:0/97465738 >> 172.21.15.40:0/97465738 conn(0x55a12b7ce750 msgr2=0x55a12b7f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 -- 172.21.15.40:0/97465738 shutdown_connections 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.776+0000 7f41a3585000 1 -- 172.21.15.40:0/97465738 wait complete. 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 1 -- start start 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 1 Processor -- start 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7c7fa0 0x55a12b828ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7cbff0 0x55a12b829430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7faed0 0x55a12b829970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a12b5c1a80 con 0x55a12b7cbff0 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419814ccd0 con 0x55a12b7c7fa0 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a12b6d6700 con 0x55a12b7faed0 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 10 client.0 _ll_register_callbacks cb 0x7f41900f0a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7cbff0 0x55a12b829430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7cbff0 0x55a12b829430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42514/0 (socket says 172.21.15.40:42514) 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419cc00640 1 -- 172.21.15.40:0/3822509858 learned_addr learned my addr 172.21.15.40:0/3822509858 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:12.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419d600640 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7c7fa0 0x55a12b828ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419cc00640 1 -- 172.21.15.40:0/3822509858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7faed0 msgr2=0x55a12b829970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419e000640 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7faed0 0x55a12b829970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419cc00640 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7faed0 0x55a12b829970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419cc00640 1 -- 172.21.15.40:0/3822509858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7c7fa0 msgr2=0x55a12b828ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419cc00640 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7c7fa0 0x55a12b828ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419cc00640 1 -- 172.21.15.40:0/3822509858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a12b5ad4b0 con 0x55a12b7cbff0 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419d600640 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7c7fa0 0x55a12b828ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419cc00640 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7cbff0 0x55a12b829430 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4184016650 tx=0x7f4184016b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17278ad46d793854 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f417ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f417ea00640 1 -- 172.21.15.40:0/3822509858 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184004230 con 0x55a12b7cbff0 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f417ea00640 1 -- 172.21.15.40:0/3822509858 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41840043d0 con 0x55a12b7cbff0 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 1 -- 172.21.15.40:0/3822509858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a12b626bc0 con 0x55a12b7cbff0 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f417ea00640 1 -- 172.21.15.40:0/3822509858 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4184029050 con 0x55a12b7cbff0 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 10 client.5409 fetch_fsmap learned FSMap version 16 2024-03-24T00:32:12.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 1 -- 172.21.15.40:0/3822509858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55a12b5c02f0 con 0x55a12b7cbff0 2024-03-24T00:32:12.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f417ea00640 1 -- 172.21.15.40:0/3822509858 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a12b626bc0 con 0x55a12b7cbff0 2024-03-24T00:32:12.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f417ea00640 1 -- 172.21.15.40:0/3822509858 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 16) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f418402c020 con 0x55a12b7cbff0 2024-03-24T00:32:12.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 10 client.5409 fetch_fsmap finished waiting for FSMap version 16 2024-03-24T00:32:12.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 10 client.5409 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:32:12.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 20 client.5409 populate_metadata read hostname 'smithi040' 2024-03-24T00:32:12.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x55a12b815920 0x55a12b8831f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 4 client.5409 mds_command: new command op to 5229 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:32:12.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f41a3585000 1 -- 172.21.15.40:0/3822509858 --> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a12b620f10 con 0x55a12b815920 2024-03-24T00:32:12.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.780+0000 7f419d600640 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x55a12b815920 0x55a12b8831f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f419d600640 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x55a12b815920 0x55a12b8831f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceed87d6dd2fade5 server_cookie=d2a065de7681b65b in_seq=0 out_seq=0 2024-03-24T00:32:12.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f417ea00640 10 client.5409 ms_handle_connect on v2:172.21.15.40:6834/3050269371 2024-03-24T00:32:12.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f417ea00640 1 -- 172.21.15.40:0/3822509858 <== mds.0 v2:172.21.15.40:6834/3050269371 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55a12b620f10 con 0x55a12b815920 2024-03-24T00:32:12.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f417ea00640 10 client.5409 handle_command_reply: tid=0 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a3585000 1 client.5409 shutdown 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a3585000 1 -- 172.21.15.40:0/3822509858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7cbff0 msgr2=0x55a12b829430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a3585000 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7cbff0 0x55a12b829430 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4184016650 tx=0x7f4184016b10 comp rx=0 tx=0).stop 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a3585000 1 -- 172.21.15.40:0/3822509858 shutdown_connections 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a3585000 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x55a12b815920 0x55a12b8831f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a3585000 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a12b7faed0 0x55a12b829970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a3585000 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a12b7cbff0 0x55a12b829430 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f417ea00640 0 client.5409 ms_handle_reset on v2:172.21.15.40:6834/3050269371 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a3585000 1 --2- 172.21.15.40:0/3822509858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a12b7c7fa0 0x55a12b828ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a3585000 1 -- 172.21.15.40:0/3822509858 >> 172.21.15.40:0/3822509858 conn(0x55a12b7ce750 msgr2=0x55a12b7fa190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a3585000 1 -- 172.21.15.40:0/3822509858 shutdown_connections 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a3585000 1 -- 172.21.15.40:0/3822509858 wait complete. 2024-03-24T00:32:12.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a3585000 20 client.5409 trim_cache size 0 max 16384 2024-03-24T00:32:12.794 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:32:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:32:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:32:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5229, 2024-03-24T00:32:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:32:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:32:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:32:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 6.1466524829999996, 2024-03-24T00:32:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 16, 2024-03-24T00:32:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 33, 2024-03-24T00:32:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 33, 2024-03-24T00:32:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 14.019891263 2024-03-24T00:32:12.795 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:32:12.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a0000640 1 -- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4178067660 msgr2=0x7f4178069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a0000640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4178067660 0x7f4178069b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f418c005890 tx=0x7f418c009040 comp rx=0 tx=0).stop 2024-03-24T00:32:12.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a0000640 1 -- 172.21.15.40:0/3469556844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f419815e0b0 msgr2=0x7f41980cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a0000640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f419815e0b0 0x7f41980cccd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4190009d90 tx=0x7f4190019710 comp rx=0 tx=0).stop 2024-03-24T00:32:12.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a0000640 1 -- 172.21.15.40:0/3469556844 shutdown_connections 2024-03-24T00:32:12.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a0000640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4178067660 0x7f4178069b20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a0000640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f419815e0b0 0x7f41980cccd0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a0000640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41980d0830 0x7f41980cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a0000640 1 --2- 172.21.15.40:0/3469556844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f419806b150 0x7f419814a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a0000640 1 -- 172.21.15.40:0/3469556844 >> 172.21.15.40:0/3469556844 conn(0x7f41980d1c90 msgr2=0x7f41980af730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:12.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a0000640 1 -- 172.21.15.40:0/3469556844 shutdown_connections 2024-03-24T00:32:12.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.784+0000 7f41a0000640 1 -- 172.21.15.40:0/3469556844 wait complete. 2024-03-24T00:32:12.819 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:32:12.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.952+0000 7f7890200640 1 Processor -- start 2024-03-24T00:32:12.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.952+0000 7f7890200640 1 -- start start 2024-03-24T00:32:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f7890200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f788806e5a0 0x7f788806e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f7890200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7888070c80 0x7f788806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f7890200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788806cdc0 con 0x7f7888071690 2024-03-24T00:32:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f7890200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788806eec0 con 0x7f788806e5a0 2024-03-24T00:32:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f7890200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7888071160 con 0x7f7888070c80 2024-03-24T00:32:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f788d800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f788806e5a0 0x7f788806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f788d800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f788806e5a0 0x7f788806e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42516/0 (socket says 172.21.15.40:42516) 2024-03-24T00:32:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f788d800640 1 -- 172.21.15.40:0/1688259973 learned_addr learned my addr 172.21.15.40:0/1688259973 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:12.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f788ce00640 1 --2- 172.21.15.40:0/1688259973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7888070c80 0x7f788806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f788d800640 1 -- 172.21.15.40:0/1688259973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7888070c80 msgr2=0x7f788806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:12.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f788d800640 1 --2- 172.21.15.40:0/1688259973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7888070c80 0x7f788806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f788d800640 1 -- 172.21.15.40:0/1688259973 >> v1:172.21.15.53:6789/0 conn(0x7f7888071690 legacy=0x7f78880b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.956+0000 7f788d800640 1 -- 172.21.15.40:0/1688259973 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788806d040 con 0x7f788806e5a0 2024-03-24T00:32:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f788d800640 1 --2- 172.21.15.40:0/1688259973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f788806e5a0 0x7f788806e980 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f787c009870 tx=0x7f787c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9469e5fb177b2fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7887e00640 1 -- 172.21.15.40:0/1688259973 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c0b2070 con 0x7f788806e5a0 2024-03-24T00:32:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7887e00640 1 -- 172.21.15.40:0/1688259973 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787c0a9070 con 0x7f788806e5a0 2024-03-24T00:32:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7887e00640 1 -- 172.21.15.40:0/1688259973 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c0ad880 con 0x7f788806e5a0 2024-03-24T00:32:12.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7890200640 1 -- 172.21.15.40:0/1688259973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f788806e5a0 msgr2=0x7f788806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7890200640 1 --2- 172.21.15.40:0/1688259973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f788806e5a0 0x7f788806e980 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f787c009870 tx=0x7f787c01c560 comp rx=0 tx=0).stop 2024-03-24T00:32:12.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7890200640 1 -- 172.21.15.40:0/1688259973 shutdown_connections 2024-03-24T00:32:12.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7890200640 1 --2- 172.21.15.40:0/1688259973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7888070c80 0x7f788806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7890200640 1 --2- 172.21.15.40:0/1688259973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f788806e5a0 0x7f788806e980 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7890200640 1 -- 172.21.15.40:0/1688259973 >> 172.21.15.40:0/1688259973 conn(0x7f78880b17b0 msgr2=0x7f78880b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7890200640 1 -- 172.21.15.40:0/1688259973 shutdown_connections 2024-03-24T00:32:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7890200640 1 -- 172.21.15.40:0/1688259973 wait complete. 2024-03-24T00:32:12.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7890200640 1 Processor -- start 2024-03-24T00:32:12.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.960+0000 7f7890200640 1 -- start start 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f7890200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f788806e5a0 0x7f7888145ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f7890200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7888070c80 0x7f78881fb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f7890200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7888071690 0x7f78881343a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f7890200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788816b8c0 con 0x7f7888070c80 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f7890200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788816b740 con 0x7f7888071690 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f7890200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788816ba40 con 0x7f788806e5a0 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f788ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7888070c80 0x7f78881fb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f788ce00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7888070c80 0x7f78881fb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42520/0 (socket says 172.21.15.40:42520) 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f788ce00640 1 -- 172.21.15.40:0/1400481406 learned_addr learned my addr 172.21.15.40:0/1400481406 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f788d800640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f788806e5a0 0x7f7888145ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f788e200640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7888071690 0x7f78881343a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f788ce00640 1 -- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f788806e5a0 msgr2=0x7f7888145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f788ce00640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f788806e5a0 0x7f7888145ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f788ce00640 1 -- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7888071690 msgr2=0x7f78881343a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f788ce00640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7888071690 0x7f78881343a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f788ce00640 1 -- 172.21.15.40:0/1400481406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888146220 con 0x7f7888070c80 2024-03-24T00:32:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f788ce00640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7888070c80 0x7f78881fb9e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f787400d620 tx=0x7f787400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f7885600640 1 -- 172.21.15.40:0/1400481406 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787400b620 con 0x7f7888070c80 2024-03-24T00:32:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f7885600640 1 -- 172.21.15.40:0/1400481406 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7874019070 con 0x7f7888070c80 2024-03-24T00:32:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f7890200640 1 -- 172.21.15.40:0/1400481406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7888134df0 con 0x7f7888070c80 2024-03-24T00:32:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f7885600640 1 -- 172.21.15.40:0/1400481406 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787400b9e0 con 0x7f7888070c80 2024-03-24T00:32:12.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f7890200640 1 -- 172.21.15.40:0/1400481406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78881fb1e0 con 0x7f7888070c80 2024-03-24T00:32:12.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.964+0000 7f7890200640 1 -- 172.21.15.40:0/1400481406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7888068170 con 0x7f7888070c80 2024-03-24T00:32:12.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.968+0000 7f7885600640 1 -- 172.21.15.40:0/1400481406 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f787400bb80 con 0x7f7888070c80 2024-03-24T00:32:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.968+0000 7f7885600640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7868067660 0x7f7868069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:12.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.968+0000 7f7885600640 1 -- 172.21.15.40:0/1400481406 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7874085aa0 con 0x7f7888070c80 2024-03-24T00:32:12.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.968+0000 7f788d800640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7868067660 0x7f7868069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:12.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.968+0000 7f788d800640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7868067660 0x7f7868069b20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f787c0af000 tx=0x7f787c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:12.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:12.972+0000 7f7885600640 1 -- 172.21.15.40:0/1400481406 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7888146220 con 0x7f7888070c80 2024-03-24T00:32:13.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.172+0000 7f7890200640 1 -- 172.21.15.40:0/1400481406 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7888168d00 con 0x7f7888070c80 2024-03-24T00:32:13.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.172+0000 7f7885600640 1 -- 172.21.15.40:0/1400481406 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7874056160 con 0x7f7888070c80 2024-03-24T00:32:13.180 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:32:13.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.176+0000 7f7890200640 1 -- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7868067660 msgr2=0x7f7868069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:13.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.176+0000 7f7890200640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7868067660 0x7f7868069b20 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f787c0af000 tx=0x7f787c008040 comp rx=0 tx=0).stop 2024-03-24T00:32:13.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.176+0000 7f7890200640 1 -- 172.21.15.40:0/1400481406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7888070c80 msgr2=0x7f78881fb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:13.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.176+0000 7f7890200640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7888070c80 0x7f78881fb9e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f787400d620 tx=0x7f787400dae0 comp rx=0 tx=0).stop 2024-03-24T00:32:13.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.176+0000 7f7890200640 1 -- 172.21.15.40:0/1400481406 shutdown_connections 2024-03-24T00:32:13.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.176+0000 7f7890200640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7868067660 0x7f7868069b20 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:13.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.176+0000 7f7890200640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7888071690 0x7f78881343a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:13.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.176+0000 7f7890200640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7888070c80 0x7f78881fb9e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:13.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.176+0000 7f7890200640 1 --2- 172.21.15.40:0/1400481406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f788806e5a0 0x7f7888145ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:13.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.176+0000 7f7890200640 1 -- 172.21.15.40:0/1400481406 >> 172.21.15.40:0/1400481406 conn(0x7f78880b17b0 msgr2=0x7f7888151000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:13.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.176+0000 7f7890200640 1 -- 172.21.15.40:0/1400481406 shutdown_connections 2024-03-24T00:32:13.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:13.176+0000 7f7890200640 1 -- 172.21.15.40:0/1400481406 wait complete. 2024-03-24T00:32:13.201 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:32:13.201 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:32:13.201 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-03-24T00:32:13.201 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:32:13.201 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:32:13.201 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:32:13.201 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:32:13.201 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:32:13.201 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:32:13.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:13.201 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-03-24T00:32:13.211 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:32:13.212 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:32:13.212 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:32:13.212 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:32:13.212 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-24T00:32:13.212 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:32:13.212 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:32:13.212 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:32:13.212 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-03-24T00:32:13.212 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:32:13.212 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:32:13.212 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-03-24T00:32:13.212 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:32:13.213 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:32:13.213 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-03-24T00:32:13.213 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-24T00:32:13.213 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42417sec preferred_lft 42417sec 2024-03-24T00:32:13.213 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2024-03-24T00:32:13.213 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:32:13.213 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:32:13.213 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:32:13.213 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-03-24T00:32:13.213 INFO:teuthology.orchestra.run.smithi040.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-24T00:32:13.213 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 02:42:9e:ab:82:f9 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:32:13.214 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-24T00:32:13.214 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:32:13.214 INFO:teuthology.orchestra.run.smithi040.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:32:13.214 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-24T00:32:13.214 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:32:13.214 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:32:13.214 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-24T00:32:13.214 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:32:13.214 INFO:teuthology.orchestra.run.smithi040.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:32:13.214 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:32:13.215 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-24T00:32:13.215 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:32:13.215 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:32:13.215 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:32:13.215 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-24T00:32:13.215 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:32:13.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:13.215 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:32:13.273 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-03-24T00:32:13.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:13.276 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-03-24T00:32:13.329 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-03-24T00:32:13.329 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:32:13.329 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-24T00:32:13.329 INFO:teuthology.orchestra.run.smithi040.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-24T00:32:13.329 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-24T00:32:13.329 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:32:13.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:13.330 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:32:13.330 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:32:13.330 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T00:32:13.330 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T00:32:13.330 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T00:32:13.330 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:32:13.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:13.421 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-24T00:32:13.430 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:32:13.430 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:32:13.431 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:32:13.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:13.431 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-03-24T00:32:13.491 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T00:32:13.491 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:32:13.492 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:32:13.544 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:32:13.544 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:32:13.544 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:32:13.595 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:32:13.659 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:32:13.709 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:32:13.709 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:32:13.709 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:32:13.709 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:32:13.709 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:32:13.709 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:32:13.710 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:32:13.711 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:32:13.711 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:32:13.711 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:32:13.711 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:32:13.711 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:32:13.711 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:32:13.711 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:32:13.711 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:32:13.711 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:32:13.711 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:32:13.711 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:32:13.712 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:32:13.712 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:32:13.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:13.713 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:32:13.763 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T00:32:13.764 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T00:32:13.810 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:32:13.831 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:32:13.887 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:32:13.887 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:32:13.887 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:32:13.887 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:32:13.887 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:32:13.887 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:32:13.887 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:32:13.887 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:32:13.888 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:32:13.889 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:32:13.889 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:32:13.889 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:32:13.889 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:32:13.889 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:32:13.889 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:32:13.889 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:32:13.889 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:32:13.889 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:32:13.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:13.890 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:32:13.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:2024-03-24T00:32:13.900+0000 7f280b977580 -1 init, newargv = 0x558e0b72a940 newargc=15 2024-03-24T00:32:13.906 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[37363]: starting ceph client 2024-03-24T00:32:13.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[37363]: starting fuse 2024-03-24T00:32:13.915 INFO:teuthology.orchestra.run.smithi040.stdout:43 2024-03-24T00:32:13.916 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-24T00:32:13.916 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:32:13.916 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-03-24T00:32:13.916 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-03-24T00:32:13.916 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-03-24T00:32:13.916 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-03-24T00:32:13.916 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:32:13.916 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-03-24T00:32:13.916 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-03-24T00:32:13.917 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:32:13.918 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:32:13.918 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.0")) 2024-03-24T00:32:13.918 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-03-24T00:32:14.042 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.0.37363.asok 2024-03-24T00:32:14.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:32:14.053 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.37363.asok 2024-03-24T00:32:14.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:14.054 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.37363.asok status 2024-03-24T00:32:14.228 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:32:14.228 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-03-24T00:32:14.228 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:32:14.228 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:32:14.228 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "0", 2024-03-24T00:32:14.228 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-03-24T00:32:14.228 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T00:32:14.228 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "37363", 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5424, 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5424 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 204086762 2024-03-24T00:32:14.229 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 204086762 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.5424 192.168.0.1:0/204086762", 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "192.168.0.1:0/204086762", 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 16, 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 33, 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-03-24T00:32:14.230 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-03-24T00:32:14.231 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:32:14.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:14.243 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-03-24T00:32:14.243 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T00:32:14.253 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-03-24T00:32:14.253 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T00:32:14.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:14.254 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:32:14.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random True 2024-03-24T00:32:14.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.492+0000 7f557cc00640 1 Processor -- start 2024-03-24T00:32:14.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.492+0000 7f557cc00640 1 -- start start 2024-03-24T00:32:14.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.492+0000 7f557cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55780b4330 0x7f55780ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:14.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.492+0000 7f557cc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55780cf090 con 0x7f55780b2360 2024-03-24T00:32:14.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.492+0000 7f557cc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55780cf330 con 0x7f55780cea00 2024-03-24T00:32:14.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.492+0000 7f557cc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55780d0f60 con 0x7f55780b4330 2024-03-24T00:32:14.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.492+0000 7f5575e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f55780b2360 0x7f55780b2740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57954/0 (socket says 172.21.15.40:57954) 2024-03-24T00:32:14.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.492+0000 7f5575e00640 1 -- 172.21.15.40:0/2540962112 learned_addr learned my addr 172.21.15.40:0/2540962112 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.492+0000 7f556fe00640 1 --2- 172.21.15.40:0/2540962112 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55780b4330 0x7f55780ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.496+0000 7f5575400640 1 -- 172.21.15.40:0/2540962112 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114486325 0 0) 0x7f55780cf090 con 0x7f55780b2360 2024-03-24T00:32:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.496+0000 7f5575400640 1 -- 172.21.15.40:0/2540962112 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f555c003530 con 0x7f55780b2360 2024-03-24T00:32:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.496+0000 7f5575400640 1 -- 172.21.15.40:0/2540962112 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436896091 0 0) 0x7f55780cf330 con 0x7f55780cea00 2024-03-24T00:32:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.496+0000 7f5575400640 1 -- 172.21.15.40:0/2540962112 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55780cf090 con 0x7f55780cea00 2024-03-24T00:32:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.496+0000 7f556fe00640 1 -- 172.21.15.40:0/2540962112 >> v1:172.21.15.53:6789/0 conn(0x7f55780cea00 legacy=0x7f55780cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.496+0000 7f556fe00640 1 -- 172.21.15.40:0/2540962112 >> v1:172.21.15.40:6789/0 conn(0x7f55780b2360 legacy=0x7f55780b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.496+0000 7f556fe00640 1 -- 172.21.15.40:0/2540962112 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55780d0910 con 0x7f55780b4330 2024-03-24T00:32:14.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.496+0000 7f556fe00640 1 --2- 172.21.15.40:0/2540962112 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55780b4330 0x7f55780ce4c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f55600097f0 tx=0x7f556001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7da43b69dbe233c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:14.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.496+0000 7f5575400640 1 -- 172.21.15.40:0/2540962112 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55600b0070 con 0x7f55780b4330 2024-03-24T00:32:14.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.496+0000 7f5575400640 1 -- 172.21.15.40:0/2540962112 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55780d19e0 con 0x7f55780b4330 2024-03-24T00:32:14.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.496+0000 7f5575400640 1 -- 172.21.15.40:0/2540962112 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55600ab820 con 0x7f55780b4330 2024-03-24T00:32:14.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 -- 172.21.15.40:0/2540962112 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55780b4330 msgr2=0x7f55780ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:14.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 --2- 172.21.15.40:0/2540962112 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55780b4330 0x7f55780ce4c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f55600097f0 tx=0x7f556001c3c0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 -- 172.21.15.40:0/2540962112 shutdown_connections 2024-03-24T00:32:14.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 --2- 172.21.15.40:0/2540962112 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55780b4330 0x7f55780ce4c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 -- 172.21.15.40:0/2540962112 >> 172.21.15.40:0/2540962112 conn(0x7f55780b3810 msgr2=0x7f55780b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:14.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 -- 172.21.15.40:0/2540962112 shutdown_connections 2024-03-24T00:32:14.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 -- 172.21.15.40:0/2540962112 wait complete. 2024-03-24T00:32:14.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 Processor -- start 2024-03-24T00:32:14.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 -- start start 2024-03-24T00:32:14.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55780b2360 0x7f55780b1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:14.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780b4330 0x7f55780aea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:14.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55780cea00 0x7f55780aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:14.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557816be20 con 0x7f55780b4330 2024-03-24T00:32:14.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55780cd630 con 0x7f55780b2360 2024-03-24T00:32:14.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f557cc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557816bfa0 con 0x7f55780cea00 2024-03-24T00:32:14.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f556fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780b4330 0x7f55780aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:14.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f556fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780b4330 0x7f55780aea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42532/0 (socket says 172.21.15.40:42532) 2024-03-24T00:32:14.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f556fe00640 1 -- 172.21.15.40:0/1743588062 learned_addr learned my addr 172.21.15.40:0/1743588062 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:14.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f5575e00640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55780b2360 0x7f55780b1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:14.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f5576800640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55780cea00 0x7f55780aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:14.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f556fe00640 1 -- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55780cea00 msgr2=0x7f55780aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:14.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f556fe00640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55780cea00 0x7f55780aefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f556fe00640 1 -- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55780b2360 msgr2=0x7f55780b1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:14.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f556fe00640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55780b2360 0x7f55780b1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.500+0000 7f556fe00640 1 -- 172.21.15.40:0/1743588062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55780b49a0 con 0x7f55780b4330 2024-03-24T00:32:14.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.504+0000 7f556fe00640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780b4330 0x7f55780aea80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5560018fd0 tx=0x7f5560019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:14.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.504+0000 7f556e000640 1 -- 172.21.15.40:0/1743588062 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55600ba070 con 0x7f55780b4330 2024-03-24T00:32:14.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.504+0000 7f556e000640 1 -- 172.21.15.40:0/1743588062 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55600abd40 con 0x7f55780b4330 2024-03-24T00:32:14.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.504+0000 7f556e000640 1 -- 172.21.15.40:0/1743588062 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55600b0040 con 0x7f55780b4330 2024-03-24T00:32:14.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.504+0000 7f557cc00640 1 -- 172.21.15.40:0/1743588062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55780b4c80 con 0x7f55780b4330 2024-03-24T00:32:14.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.504+0000 7f557cc00640 1 -- 172.21.15.40:0/1743588062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55780b5240 con 0x7f55780b4330 2024-03-24T00:32:14.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.504+0000 7f557cc00640 1 -- 172.21.15.40:0/1743588062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5578068170 con 0x7f55780b4330 2024-03-24T00:32:14.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.504+0000 7f556e000640 1 -- 172.21.15.40:0/1743588062 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f55600b4630 con 0x7f55780b4330 2024-03-24T00:32:14.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.504+0000 7f556e000640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5554067660 0x7f5554069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:14.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.504+0000 7f556e000640 1 -- 172.21.15.40:0/1743588062 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f556011e9d0 con 0x7f55780b4330 2024-03-24T00:32:14.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.504+0000 7f5575e00640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5554067660 0x7f5554069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:14.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.508+0000 7f5575e00640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5554067660 0x7f5554069b20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5568000f70 tx=0x7f5568009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:14.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.508+0000 7f556e000640 1 -- 172.21.15.40:0/1743588062 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f55600160d0 con 0x7f55780b4330 2024-03-24T00:32:14.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.676+0000 7f557cc00640 1 -- 172.21.15.40:0/1743588062 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random}] v 0) v1 -- 0x7f557806dd50 con 0x7f55780b4330 2024-03-24T00:32:14.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.680+0000 7f556e000640 1 -- 172.21.15.40:0/1743588062 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random}]=0 v11)=0 v11) v1 ==== 128+0+0 (secure 0 0 0) 0x7f55600ef530 con 0x7f55780b4330 2024-03-24T00:32:14.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.684+0000 7f557cc00640 1 -- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5554067660 msgr2=0x7f5554069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:14.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.684+0000 7f557cc00640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5554067660 0x7f5554069b20 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5568000f70 tx=0x7f5568009040 comp rx=0 tx=0).stop 2024-03-24T00:32:14.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.684+0000 7f557cc00640 1 -- 172.21.15.40:0/1743588062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780b4330 msgr2=0x7f55780aea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:14.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.684+0000 7f557cc00640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780b4330 0x7f55780aea80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5560018fd0 tx=0x7f5560019160 comp rx=0 tx=0).stop 2024-03-24T00:32:14.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.684+0000 7f557cc00640 1 -- 172.21.15.40:0/1743588062 shutdown_connections 2024-03-24T00:32:14.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.684+0000 7f557cc00640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5554067660 0x7f5554069b20 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.684+0000 7f557cc00640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55780cea00 0x7f55780aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.684+0000 7f557cc00640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780b4330 0x7f55780aea80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.684+0000 7f557cc00640 1 --2- 172.21.15.40:0/1743588062 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55780b2360 0x7f55780b1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.684+0000 7f557cc00640 1 -- 172.21.15.40:0/1743588062 >> 172.21.15.40:0/1743588062 conn(0x7f55780b3810 msgr2=0x7f557815d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:14.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.684+0000 7f557cc00640 1 -- 172.21.15.40:0/1743588062 shutdown_connections 2024-03-24T00:32:14.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.684+0000 7f557cc00640 1 -- 172.21.15.40:0/1743588062 wait complete. 2024-03-24T00:32:14.718 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_distributed True 2024-03-24T00:32:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a6b800640 1 Processor -- start 2024-03-24T00:32:14.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a6b800640 1 -- start start 2024-03-24T00:32:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a6b800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3a6415e170 0x7f3a6406f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a6b800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a640d0a10 0x7f3a6415f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a6b800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a640cfcb0 con 0x7f3a640d0e30 2024-03-24T00:32:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a6b800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a64070100 con 0x7f3a6415e170 2024-03-24T00:32:14.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a6b800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6406fe20 con 0x7f3a640d0a10 2024-03-24T00:32:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a63e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a640d0a10 0x7f3a6415f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a69800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3a640d0e30 0x7f3a64161b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:43956/0 (socket says 172.21.15.40:43956) 2024-03-24T00:32:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a69800640 1 -- 172.21.15.40:0/319380585 learned_addr learned my addr 172.21.15.40:0/319380585 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a68e00640 1 --2- 172.21.15.40:0/319380585 >> v2:172.21.15.53:3300/0 conn(0x7f3a6415e170 0x7f3a6406f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a63400640 1 -- 172.21.15.40:0/319380585 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031996977 0 0) 0x7f3a5c0014a0 con 0x7f3a640d0e30 2024-03-24T00:32:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a63400640 1 -- 172.21.15.40:0/319380585 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4c003530 con 0x7f3a640d0e30 2024-03-24T00:32:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a68e00640 1 -- 172.21.15.40:0/319380585 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a640d0a10 msgr2=0x7f3a6415f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:14.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a68e00640 1 --2- 172.21.15.40:0/319380585 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a640d0a10 0x7f3a6415f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a68e00640 1 -- 172.21.15.40:0/319380585 >> v1:172.21.15.53:6789/0 conn(0x7f3a640d0e30 legacy=0x7f3a64161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:14.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.856+0000 7f3a68e00640 1 -- 172.21.15.40:0/319380585 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a640cff30 con 0x7f3a6415e170 2024-03-24T00:32:14.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.860+0000 7f3a68e00640 1 --2- 172.21.15.40:0/319380585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a6415e170 0x7f3a6406f7b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3a580097c0 tx=0x7f3a5801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e924214ae1940ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:14.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.860+0000 7f3a63400640 1 -- 172.21.15.40:0/319380585 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a580aa600 con 0x7f3a6415e170 2024-03-24T00:32:14.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.860+0000 7f3a63400640 1 -- 172.21.15.40:0/319380585 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a580af070 con 0x7f3a6415e170 2024-03-24T00:32:14.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.860+0000 7f3a63400640 1 -- 172.21.15.40:0/319380585 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a580aa9c0 con 0x7f3a6415e170 2024-03-24T00:32:14.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.860+0000 7f3a6b800640 1 -- 172.21.15.40:0/319380585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a6415e170 msgr2=0x7f3a6406f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:14.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.860+0000 7f3a6b800640 1 --2- 172.21.15.40:0/319380585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a6415e170 0x7f3a6406f7b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3a580097c0 tx=0x7f3a5801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.860+0000 7f3a6b800640 1 -- 172.21.15.40:0/319380585 shutdown_connections 2024-03-24T00:32:14.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.860+0000 7f3a6b800640 1 --2- 172.21.15.40:0/319380585 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a640d0a10 0x7f3a6415f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.860+0000 7f3a6b800640 1 --2- 172.21.15.40:0/319380585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a6415e170 0x7f3a6406f7b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.860+0000 7f3a6b800640 1 -- 172.21.15.40:0/319380585 >> 172.21.15.40:0/319380585 conn(0x7f3a640b36a0 msgr2=0x7f3a640b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:14.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.860+0000 7f3a6b800640 1 -- 172.21.15.40:0/319380585 shutdown_connections 2024-03-24T00:32:14.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.860+0000 7f3a6b800640 1 -- 172.21.15.40:0/319380585 wait complete. 2024-03-24T00:32:14.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a6b800640 1 Processor -- start 2024-03-24T00:32:14.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a6b800640 1 -- start start 2024-03-24T00:32:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a6b800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a640d0a10 0x7f3a6415b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a6b800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a640d0e30 0x7f3a640b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a6b800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a6415e170 0x7f3a64152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a6b800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6416aeb0 con 0x7f3a6415e170 2024-03-24T00:32:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a6b800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a641602c0 con 0x7f3a640d0a10 2024-03-24T00:32:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a6b800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6416b030 con 0x7f3a640d0e30 2024-03-24T00:32:14.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a69800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a6415e170 0x7f3a64152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a69800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a6415e170 0x7f3a64152150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42540/0 (socket says 172.21.15.40:42540) 2024-03-24T00:32:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a69800640 1 -- 172.21.15.40:0/67249089 learned_addr learned my addr 172.21.15.40:0/67249089 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a68e00640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a640d0a10 0x7f3a6415b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:14.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a63e00640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a640d0e30 0x7f3a640b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a69800640 1 -- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a640d0e30 msgr2=0x7f3a640b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a69800640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a640d0e30 0x7f3a640b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a69800640 1 -- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a640d0a10 msgr2=0x7f3a6415b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a69800640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a640d0a10 0x7f3a6415b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a69800640 1 -- 172.21.15.40:0/67249089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a64152690 con 0x7f3a6415e170 2024-03-24T00:32:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a69800640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a6415e170 0x7f3a64152150 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00cb40 tx=0x7f3a5c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a60c00640 1 -- 172.21.15.40:0/67249089 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c017070 con 0x7f3a6415e170 2024-03-24T00:32:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a60c00640 1 -- 172.21.15.40:0/67249089 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5c00ac40 con 0x7f3a6415e170 2024-03-24T00:32:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a60c00640 1 -- 172.21.15.40:0/67249089 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c012400 con 0x7f3a6415e170 2024-03-24T00:32:14.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a6b800640 1 -- 172.21.15.40:0/67249089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6415b6c0 con 0x7f3a6415e170 2024-03-24T00:32:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.864+0000 7f3a6b800640 1 -- 172.21.15.40:0/67249089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6414e150 con 0x7f3a6415e170 2024-03-24T00:32:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.868+0000 7f3a6b800640 1 -- 172.21.15.40:0/67249089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a64068170 con 0x7f3a6415e170 2024-03-24T00:32:14.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.868+0000 7f3a60c00640 1 -- 172.21.15.40:0/67249089 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3a5c01a070 con 0x7f3a6415e170 2024-03-24T00:32:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.868+0000 7f3a60c00640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a44067690 0x7f3a44069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:14.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.868+0000 7f3a60c00640 1 -- 172.21.15.40:0/67249089 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a5c0855c0 con 0x7f3a6415e170 2024-03-24T00:32:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.868+0000 7f3a68e00640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a44067690 0x7f3a44069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:14.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.868+0000 7f3a68e00640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a44067690 0x7f3a44069b50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3a580ae000 tx=0x7f3a580a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:14.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:14.872+0000 7f3a60c00640 1 -- 172.21.15.40:0/67249089 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3a640cfcb0 con 0x7f3a6415e170 2024-03-24T00:32:15.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.040+0000 7f3a6b800640 1 -- 172.21.15.40:0/67249089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_distributed}] v 0) v1 -- 0x7f3a640c8a20 con 0x7f3a6415e170 2024-03-24T00:32:15.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a60c00640 1 -- 172.21.15.40:0/67249089 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_distributed}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7f3a5c055b50 con 0x7f3a6415e170 2024-03-24T00:32:15.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a6b800640 1 -- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a44067690 msgr2=0x7f3a44069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:15.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a6b800640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a44067690 0x7f3a44069b50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3a580ae000 tx=0x7f3a580a7000 comp rx=0 tx=0).stop 2024-03-24T00:32:15.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a6b800640 1 -- 172.21.15.40:0/67249089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a6415e170 msgr2=0x7f3a64152150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a6b800640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a6415e170 0x7f3a64152150 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c00cb40 tx=0x7f3a5c00a060 comp rx=0 tx=0).stop 2024-03-24T00:32:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a6b800640 1 -- 172.21.15.40:0/67249089 shutdown_connections 2024-03-24T00:32:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a6b800640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a44067690 0x7f3a44069b50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a6b800640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a6415e170 0x7f3a64152150 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a6b800640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a640d0e30 0x7f3a640b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a6b800640 1 --2- 172.21.15.40:0/67249089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a640d0a10 0x7f3a6415b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a6b800640 1 -- 172.21.15.40:0/67249089 >> 172.21.15.40:0/67249089 conn(0x7f3a640b36a0 msgr2=0x7f3a640b3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a6b800640 1 -- 172.21.15.40:0/67249089 shutdown_connections 2024-03-24T00:32:15.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.044+0000 7f3a6b800640 1 -- 172.21.15.40:0/67249089 wait complete. 2024-03-24T00:32:15.070 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random_max 1.0 2024-03-24T00:32:15.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1639a00640 1 Processor -- start 2024-03-24T00:32:15.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1639a00640 1 -- start start 2024-03-24T00:32:15.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1639a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1634145700 0x7f16340725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:15.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1639a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1634066e80 0x7f1634067310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:15.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1639a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16340ceff0 con 0x7f16340cd320 2024-03-24T00:32:15.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1639a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16340cf8e0 con 0x7f1634066e80 2024-03-24T00:32:15.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1639a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16340cdb70 con 0x7f1634145700 2024-03-24T00:32:15.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f162be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f16340cd320 0x7f16340cfd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57964/0 (socket says 172.21.15.40:57964) 2024-03-24T00:32:15.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f162be00640 1 -- 172.21.15.40:0/2154208274 learned_addr learned my addr 172.21.15.40:0/2154208274 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:15.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1633400640 1 --2- 172.21.15.40:0/2154208274 >> v2:172.21.15.40:3300/0 conn(0x7f1634066e80 0x7f1634067310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:15.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1632a00640 1 --2- 172.21.15.40:0/2154208274 >> v2:172.21.15.53:3300/0 conn(0x7f1634145700 0x7f16340725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:15.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1632000640 1 -- 172.21.15.40:0/2154208274 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100056279 0 0) 0x7f16340ceff0 con 0x7f16340cd320 2024-03-24T00:32:15.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1632000640 1 -- 172.21.15.40:0/2154208274 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1618003530 con 0x7f16340cd320 2024-03-24T00:32:15.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1633400640 1 -- 172.21.15.40:0/2154208274 >> v2:172.21.15.53:3300/0 conn(0x7f1634145700 msgr2=0x7f16340725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:15.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1633400640 1 --2- 172.21.15.40:0/2154208274 >> v2:172.21.15.53:3300/0 conn(0x7f1634145700 0x7f16340725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1633400640 1 -- 172.21.15.40:0/2154208274 >> v1:172.21.15.40:6789/0 conn(0x7f16340cd320 legacy=0x7f16340cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:15.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.208+0000 7f1633400640 1 -- 172.21.15.40:0/2154208274 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16340cd890 con 0x7f1634066e80 2024-03-24T00:32:15.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.212+0000 7f1633400640 1 --2- 172.21.15.40:0/2154208274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1634066e80 0x7f1634067310 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1634071700 tx=0x7f162c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c528b543d2894abc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:15.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.212+0000 7f1632000640 1 -- 172.21.15.40:0/2154208274 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c0b2070 con 0x7f1634066e80 2024-03-24T00:32:15.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.212+0000 7f1632000640 1 -- 172.21.15.40:0/2154208274 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c0a57f0 con 0x7f1634066e80 2024-03-24T00:32:15.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.212+0000 7f1632000640 1 -- 172.21.15.40:0/2154208274 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c0a5bc0 con 0x7f1634066e80 2024-03-24T00:32:15.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.212+0000 7f1639a00640 1 -- 172.21.15.40:0/2154208274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1634066e80 msgr2=0x7f1634067310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:15.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.212+0000 7f1639a00640 1 --2- 172.21.15.40:0/2154208274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1634066e80 0x7f1634067310 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f1634071700 tx=0x7f162c01c930 comp rx=0 tx=0).stop 2024-03-24T00:32:15.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.212+0000 7f1639a00640 1 -- 172.21.15.40:0/2154208274 shutdown_connections 2024-03-24T00:32:15.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.212+0000 7f1639a00640 1 --2- 172.21.15.40:0/2154208274 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1634066e80 0x7f1634067310 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.212+0000 7f1639a00640 1 --2- 172.21.15.40:0/2154208274 >> v2:172.21.15.53:3300/0 conn(0x7f1634145700 0x7f16340725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.212+0000 7f1639a00640 1 -- 172.21.15.40:0/2154208274 >> 172.21.15.40:0/2154208274 conn(0x7f16340b27c0 msgr2=0x7f16340b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:15.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.212+0000 7f1639a00640 1 -- 172.21.15.40:0/2154208274 shutdown_connections 2024-03-24T00:32:15.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.212+0000 7f1639a00640 1 -- 172.21.15.40:0/2154208274 wait complete. 2024-03-24T00:32:15.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f1639a00640 1 Processor -- start 2024-03-24T00:32:15.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f1639a00640 1 -- start start 2024-03-24T00:32:15.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f1639a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634066e80 0x7f1634150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:15.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f1639a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16340cd320 0x7f1634151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:15.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f1639a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1634145700 0x7f16341374e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:15.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f1639a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163406e580 con 0x7f1634145700 2024-03-24T00:32:15.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f1639a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163406e400 con 0x7f1634066e80 2024-03-24T00:32:15.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f1639a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1634168ce0 con 0x7f16340cd320 2024-03-24T00:32:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f1633400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1634145700 0x7f16341374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f1633400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1634145700 0x7f16341374e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42566/0 (socket says 172.21.15.40:42566) 2024-03-24T00:32:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f1633400640 1 -- 172.21.15.40:0/3482670676 learned_addr learned my addr 172.21.15.40:0/3482670676 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f162be00640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16340cd320 0x7f1634151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.216+0000 7f1632a00640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634066e80 0x7f1634150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f1633400640 1 -- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16340cd320 msgr2=0x7f1634151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f1633400640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16340cd320 0x7f1634151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f1633400640 1 -- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634066e80 msgr2=0x7f1634150810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f1633400640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634066e80 0x7f1634150810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f1633400640 1 -- 172.21.15.40:0/3482670676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1634150ee0 con 0x7f1634145700 2024-03-24T00:32:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f162be00640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16340cd320 0x7f1634151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:15.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f1633400640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1634145700 0x7f16341374e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f162c019540 tx=0x7f162c0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:15.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f162aa00640 1 -- 172.21.15.40:0/3482670676 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c0bd020 con 0x7f1634145700 2024-03-24T00:32:15.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f162aa00640 1 -- 172.21.15.40:0/3482670676 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c0b8470 con 0x7f1634145700 2024-03-24T00:32:15.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f1639a00640 1 -- 172.21.15.40:0/3482670676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16341511c0 con 0x7f1634145700 2024-03-24T00:32:15.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f162aa00640 1 -- 172.21.15.40:0/3482670676 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c0b2040 con 0x7f1634145700 2024-03-24T00:32:15.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f1639a00640 1 -- 172.21.15.40:0/3482670676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1634151780 con 0x7f1634145700 2024-03-24T00:32:15.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f1639a00640 1 -- 172.21.15.40:0/3482670676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1634068170 con 0x7f1634145700 2024-03-24T00:32:15.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.220+0000 7f162aa00640 1 -- 172.21.15.40:0/3482670676 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f162c0c0020 con 0x7f1634145700 2024-03-24T00:32:15.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.224+0000 7f162aa00640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1610067660 0x7f1610069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:15.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.224+0000 7f162aa00640 1 -- 172.21.15.40:0/3482670676 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f162c1207f0 con 0x7f1634145700 2024-03-24T00:32:15.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.224+0000 7f1632a00640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1610067660 0x7f1610069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:15.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.224+0000 7f1632a00640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1610067660 0x7f1610069b20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f16341372d0 tx=0x7f1624008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:15.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.228+0000 7f162aa00640 1 -- 172.21.15.40:0/3482670676 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f162c0c7030 con 0x7f1634145700 2024-03-24T00:32:15.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.396+0000 7f1639a00640 1 -- 172.21.15.40:0/3482670676 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random_max}] v 0) v1 -- 0x7f1634072de0 con 0x7f1634145700 2024-03-24T00:32:15.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.400+0000 7f162aa00640 1 -- 172.21.15.40:0/3482670676 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random_max}]=0 v13)=0 v13) v1 ==== 131+0+0 (secure 0 0 0) 0x7f162c016520 con 0x7f1634145700 2024-03-24T00:32:15.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.400+0000 7f1639a00640 1 -- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1610067660 msgr2=0x7f1610069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:15.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.400+0000 7f1639a00640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1610067660 0x7f1610069b20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f16341372d0 tx=0x7f1624008040 comp rx=0 tx=0).stop 2024-03-24T00:32:15.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.400+0000 7f1639a00640 1 -- 172.21.15.40:0/3482670676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1634145700 msgr2=0x7f16341374e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:15.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.400+0000 7f1639a00640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1634145700 0x7f16341374e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f162c019540 tx=0x7f162c0196d0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.404+0000 7f1639a00640 1 -- 172.21.15.40:0/3482670676 shutdown_connections 2024-03-24T00:32:15.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.404+0000 7f1639a00640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1610067660 0x7f1610069b20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.404+0000 7f1639a00640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1634145700 0x7f16341374e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.404+0000 7f1639a00640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16340cd320 0x7f1634151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.404+0000 7f1639a00640 1 --2- 172.21.15.40:0/3482670676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1634066e80 0x7f1634150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:15.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.404+0000 7f1639a00640 1 -- 172.21.15.40:0/3482670676 >> 172.21.15.40:0/3482670676 conn(0x7f16340b27c0 msgr2=0x7f163415d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:15.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.404+0000 7f1639a00640 1 -- 172.21.15.40:0/3482670676 shutdown_connections 2024-03-24T00:32:15.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:15.404+0000 7f1639a00640 1 -- 172.21.15.40:0/3482670676 wait complete. 2024-03-24T00:32:15.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:15.428 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:32:15.428 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:32:15.429 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:32:15.429 DEBUG:teuthology.orchestra.run.smithi040:> # Use up a random number of inode numbers so the ephemeral pinning is not the same every test. 2024-03-24T00:32:15.429 DEBUG:teuthology.orchestra.run.smithi040:> mkdir .inode_number_thrash 2024-03-24T00:32:15.429 DEBUG:teuthology.orchestra.run.smithi040:> count=$((RANDOM % 1024)) 2024-03-24T00:32:15.429 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < count; i++)); do touch .inode_number_thrash/$i; done 2024-03-24T00:32:15.429 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf .inode_number_thrash 2024-03-24T00:32:15.429 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:32:17.842 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-24T00:32:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.984+0000 7f1d25800640 1 Processor -- start 2024-03-24T00:32:17.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.984+0000 7f1d25800640 1 -- start start 2024-03-24T00:32:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.984+0000 7f1d25800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1d2015bd40 0x7f1d200d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.984+0000 7f1d25800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d200d1520 0x7f1d200cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.984+0000 7f1d25800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2015f970 con 0x7f1d2015b960 2024-03-24T00:32:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.984+0000 7f1d25800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d20160580 con 0x7f1d2015bd40 2024-03-24T00:32:17.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.984+0000 7f1d25800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2015fc50 con 0x7f1d200d1520 2024-03-24T00:32:17.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.988+0000 7f1d1f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d200d1520 0x7f1d200cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:17.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.988+0000 7f1d1f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d200d1520 0x7f1d200cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:55080/0 (socket says 172.21.15.40:55080) 2024-03-24T00:32:17.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.988+0000 7f1d1f400640 1 -- 172.21.15.40:0/4277693896 learned_addr learned my addr 172.21.15.40:0/4277693896 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:17.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.988+0000 7f1d16000640 1 --2- 172.21.15.40:0/4277693896 >> v2:172.21.15.53:3300/0 conn(0x7f1d2015bd40 0x7f1d200d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:17.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.988+0000 7f1d1e000640 1 -- 172.21.15.40:0/4277693896 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796237039 0 0) 0x7f1d2015f970 con 0x7f1d2015b960 2024-03-24T00:32:17.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.988+0000 7f1d1e000640 1 -- 172.21.15.40:0/4277693896 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d04003530 con 0x7f1d2015b960 2024-03-24T00:32:17.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.988+0000 7f1d1f400640 1 -- 172.21.15.40:0/4277693896 >> v2:172.21.15.53:3300/0 conn(0x7f1d2015bd40 msgr2=0x7f1d200d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.988+0000 7f1d1f400640 1 --2- 172.21.15.40:0/4277693896 >> v2:172.21.15.53:3300/0 conn(0x7f1d2015bd40 0x7f1d200d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.988+0000 7f1d1f400640 1 -- 172.21.15.40:0/4277693896 >> v1:172.21.15.53:6789/0 conn(0x7f1d2015b960 legacy=0x7f1d20070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.988+0000 7f1d1f400640 1 -- 172.21.15.40:0/4277693896 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2015ff30 con 0x7f1d200d1520 2024-03-24T00:32:17.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.988+0000 7f1d16000640 1 --2- 172.21.15.40:0/4277693896 >> v2:172.21.15.53:3300/0 conn(0x7f1d2015bd40 0x7f1d200d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:17.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.988+0000 7f1d1f400640 1 --2- 172.21.15.40:0/4277693896 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d200d1520 0x7f1d200cda00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009e50 tx=0x7f1d1801ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aeb25522411fc236 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:17.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d1e000640 1 -- 172.21.15.40:0/4277693896 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d180a5700 con 0x7f1d200d1520 2024-03-24T00:32:17.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d1e000640 1 -- 172.21.15.40:0/4277693896 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d180a58a0 con 0x7f1d200d1520 2024-03-24T00:32:17.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d1e000640 1 -- 172.21.15.40:0/4277693896 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d180a5c70 con 0x7f1d200d1520 2024-03-24T00:32:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 -- 172.21.15.40:0/4277693896 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d200d1520 msgr2=0x7f1d200cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 --2- 172.21.15.40:0/4277693896 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d200d1520 0x7f1d200cda00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1d18009e50 tx=0x7f1d1801ca30 comp rx=0 tx=0).stop 2024-03-24T00:32:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 -- 172.21.15.40:0/4277693896 shutdown_connections 2024-03-24T00:32:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 --2- 172.21.15.40:0/4277693896 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d200d1520 0x7f1d200cda00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 --2- 172.21.15.40:0/4277693896 >> v2:172.21.15.53:3300/0 conn(0x7f1d2015bd40 0x7f1d200d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 -- 172.21.15.40:0/4277693896 >> 172.21.15.40:0/4277693896 conn(0x7f1d2006ecf0 msgr2=0x7f1d2006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 -- 172.21.15.40:0/4277693896 shutdown_connections 2024-03-24T00:32:17.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 -- 172.21.15.40:0/4277693896 wait complete. 2024-03-24T00:32:17.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 Processor -- start 2024-03-24T00:32:17.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 -- start start 2024-03-24T00:32:17.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d200d1520 0x7f1d201520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:17.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d2015b960 0x7f1d2006c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:17.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d2015bd40 0x7f1d2006c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:17.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d200ceb20 con 0x7f1d2015b960 2024-03-24T00:32:17.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d200ce9a0 con 0x7f1d2015bd40 2024-03-24T00:32:17.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d25800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d200ceca0 con 0x7f1d200d1520 2024-03-24T00:32:17.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d16000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d2015b960 0x7f1d2006c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:17.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d16000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d2015b960 0x7f1d2006c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46240/0 (socket says 172.21.15.40:46240) 2024-03-24T00:32:17.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.992+0000 7f1d16000640 1 -- 172.21.15.40:0/1012631083 learned_addr learned my addr 172.21.15.40:0/1012631083 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:17.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d1ea00640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d200d1520 0x7f1d201520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:17.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d1f400640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d2015bd40 0x7f1d2006c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:17.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d16000640 1 -- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d200d1520 msgr2=0x7f1d201520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:17.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d16000640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d200d1520 0x7f1d201520a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:17.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d16000640 1 -- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d2015bd40 msgr2=0x7f1d2006c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:17.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d16000640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d2015bd40 0x7f1d2006c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:17.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d16000640 1 -- 172.21.15.40:0/1012631083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d20150d10 con 0x7f1d2015b960 2024-03-24T00:32:17.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d16000640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d2015b960 0x7f1d2006c1a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800ebe0 tx=0x7f1d0800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:17.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d17400640 1 -- 172.21.15.40:0/1012631083 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08017070 con 0x7f1d2015b960 2024-03-24T00:32:17.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d17400640 1 -- 172.21.15.40:0/1012631083 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0800cc30 con 0x7f1d2015b960 2024-03-24T00:32:17.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d17400640 1 -- 172.21.15.40:0/1012631083 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08013470 con 0x7f1d2015b960 2024-03-24T00:32:17.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d25800640 1 -- 172.21.15.40:0/1012631083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d20150ff0 con 0x7f1d2015b960 2024-03-24T00:32:17.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d25800640 1 -- 172.21.15.40:0/1012631083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d201515b0 con 0x7f1d2015b960 2024-03-24T00:32:17.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d25800640 1 -- 172.21.15.40:0/1012631083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d20068170 con 0x7f1d2015b960 2024-03-24T00:32:17.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:17.996+0000 7f1d17400640 1 -- 172.21.15.40:0/1012631083 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f1d0801a070 con 0x7f1d2015b960 2024-03-24T00:32:18.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.000+0000 7f1d17400640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1cfc067660 0x7f1cfc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:18.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.000+0000 7f1d17400640 1 -- 172.21.15.40:0/1012631083 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d08085b10 con 0x7f1d2015b960 2024-03-24T00:32:18.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.000+0000 7f1d1ea00640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1cfc067660 0x7f1cfc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:18.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.000+0000 7f1d1ea00640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1cfc067660 0x7f1cfc069b20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1d10007fd0 tx=0x7f1d10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:18.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.004+0000 7f1d17400640 1 -- 172.21.15.40:0/1012631083 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1d08056820 con 0x7f1d2015b960 2024-03-24T00:32:18.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.200+0000 7f1d25800640 1 -- 172.21.15.40:0/1012631083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f1d20070910 con 0x7f1d2015b960 2024-03-24T00:32:18.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.272+0000 7f1d17400640 1 -- 172.21.15.40:0/1012631083 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v17) v1 ==== 105+0+0 (secure 0 0 0) 0x7f1d08056120 con 0x7f1d2015b960 2024-03-24T00:32:18.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.276+0000 7f1d25800640 1 -- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1cfc067660 msgr2=0x7f1cfc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:18.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.276+0000 7f1d25800640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1cfc067660 0x7f1cfc069b20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1d10007fd0 tx=0x7f1d10009040 comp rx=0 tx=0).stop 2024-03-24T00:32:18.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.276+0000 7f1d25800640 1 -- 172.21.15.40:0/1012631083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d2015b960 msgr2=0x7f1d2006c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:18.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.276+0000 7f1d25800640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d2015b960 0x7f1d2006c1a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800ebe0 tx=0x7f1d0800c520 comp rx=0 tx=0).stop 2024-03-24T00:32:18.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.276+0000 7f1d25800640 1 -- 172.21.15.40:0/1012631083 shutdown_connections 2024-03-24T00:32:18.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.276+0000 7f1d25800640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1cfc067660 0x7f1cfc069b20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.276+0000 7f1d25800640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d2015bd40 0x7f1d2006c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.276+0000 7f1d25800640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1d2015b960 0x7f1d2006c1a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.276+0000 7f1d25800640 1 --2- 172.21.15.40:0/1012631083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1d200d1520 0x7f1d201520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.276+0000 7f1d25800640 1 -- 172.21.15.40:0/1012631083 >> 172.21.15.40:0/1012631083 conn(0x7f1d2006ecf0 msgr2=0x7f1d20156d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:18.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.276+0000 7f1d25800640 1 -- 172.21.15.40:0/1012631083 shutdown_connections 2024-03-24T00:32:18.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.276+0000 7f1d25800640 1 -- 172.21.15.40:0/1012631083 wait complete. 2024-03-24T00:32:18.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:32:18.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff273200640 1 Processor -- start 2024-03-24T00:32:18.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff273200640 1 -- start start 2024-03-24T00:32:18.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff273200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff26c0b46c0 0x7ff26c0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:18.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff273200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff26c0b2ff0 0x7ff26c0ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:18.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff273200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26c070f60 con 0x7ff26c06e620 2024-03-24T00:32:18.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff273200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff26c0b3400 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff273200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff26c0b3620 con 0x7ff26c0b46c0 2024-03-24T00:32:18.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff271200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff26c0b2ff0 0x7ff26c0ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:18.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff26be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff26c06e620 0x7ff26c06ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54082/0 (socket says 172.21.15.40:54082) 2024-03-24T00:32:18.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff26be00640 1 -- 172.21.15.40:0/727638074 learned_addr learned my addr 172.21.15.40:0/727638074 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:18.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff263e00640 1 --2- 172.21.15.40:0/727638074 >> v2:172.21.15.53:3300/0 conn(0x7ff26c0b46c0 0x7ff26c0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:18.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff271200640 1 -- 172.21.15.40:0/727638074 >> v2:172.21.15.53:3300/0 conn(0x7ff26c0b46c0 msgr2=0x7ff26c0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:18.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff271200640 1 --2- 172.21.15.40:0/727638074 >> v2:172.21.15.53:3300/0 conn(0x7ff26c0b46c0 0x7ff26c0b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff271200640 1 -- 172.21.15.40:0/727638074 >> v1:172.21.15.40:6789/0 conn(0x7ff26c06e620 legacy=0x7ff26c06ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:18.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff271200640 1 -- 172.21.15.40:0/727638074 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c0b3900 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.444+0000 7ff26b400640 1 -- 172.21.15.40:0/727638074 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396564271 0 0) 0x7ff26c070f60 con 0x7ff26c06e620 2024-03-24T00:32:18.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.448+0000 7ff271200640 1 --2- 172.21.15.40:0/727638074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff26c0b2ff0 0x7ff26c0ccfa0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff26c06cea0 tx=0x7ff26401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81eb50127800ac02 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:18.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.448+0000 7ff26b400640 1 -- 172.21.15.40:0/727638074 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2640b2070 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.448+0000 7ff26b400640 1 -- 172.21.15.40:0/727638074 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2640a5930 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.448+0000 7ff26b400640 1 -- 172.21.15.40:0/727638074 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c0cdcf0 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.448+0000 7ff273200640 1 -- 172.21.15.40:0/727638074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff26c0b2ff0 msgr2=0x7ff26c0ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:18.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.448+0000 7ff273200640 1 --2- 172.21.15.40:0/727638074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff26c0b2ff0 0x7ff26c0ccfa0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff26c06cea0 tx=0x7ff26401c870 comp rx=0 tx=0).stop 2024-03-24T00:32:18.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.448+0000 7ff273200640 1 -- 172.21.15.40:0/727638074 shutdown_connections 2024-03-24T00:32:18.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.448+0000 7ff273200640 1 --2- 172.21.15.40:0/727638074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff26c0b2ff0 0x7ff26c0ccfa0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.448+0000 7ff273200640 1 --2- 172.21.15.40:0/727638074 >> v2:172.21.15.53:3300/0 conn(0x7ff26c0b46c0 0x7ff26c0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.448+0000 7ff273200640 1 -- 172.21.15.40:0/727638074 >> 172.21.15.40:0/727638074 conn(0x7ff26c0b17b0 msgr2=0x7ff26c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:18.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.448+0000 7ff273200640 1 -- 172.21.15.40:0/727638074 shutdown_connections 2024-03-24T00:32:18.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.448+0000 7ff273200640 1 -- 172.21.15.40:0/727638074 wait complete. 2024-03-24T00:32:18.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff273200640 1 Processor -- start 2024-03-24T00:32:18.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff273200640 1 -- start start 2024-03-24T00:32:18.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff273200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff26c06e620 0x7ff26c1f74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:18.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff273200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff26c0b2ff0 0x7ff26c0cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:18.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff273200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff26c0b46c0 0x7ff26c13f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:18.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff273200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c144920 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff273200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c0b4ad0 con 0x7ff26c0b46c0 2024-03-24T00:32:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff273200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26c15c890 con 0x7ff26c06e620 2024-03-24T00:32:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff263e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff26c0b2ff0 0x7ff26c0cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff26be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff26c06e620 0x7ff26c1f74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff271200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff26c0b46c0 0x7ff26c13f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff263e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff26c0b2ff0 0x7ff26c0cbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46266/0 (socket says 172.21.15.40:46266) 2024-03-24T00:32:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff263e00640 1 -- 172.21.15.40:0/4269464477 learned_addr learned my addr 172.21.15.40:0/4269464477 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff263e00640 1 -- 172.21.15.40:0/4269464477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff26c06e620 msgr2=0x7ff26c1f74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff263e00640 1 --2- 172.21.15.40:0/4269464477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff26c06e620 0x7ff26c1f74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff263e00640 1 -- 172.21.15.40:0/4269464477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff26c0b46c0 msgr2=0x7ff26c13f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff263e00640 1 --2- 172.21.15.40:0/4269464477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff26c0b46c0 0x7ff26c13f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff263e00640 1 -- 172.21.15.40:0/4269464477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c15bc80 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff263e00640 1 --2- 172.21.15.40:0/4269464477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff26c0b2ff0 0x7ff26c0cbb20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff25400d800 tx=0x7ff25400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:18.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff268c00640 1 -- 172.21.15.40:0/4269464477 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25400b6c0 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff268c00640 1 -- 172.21.15.40:0/4269464477 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff254019070 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff268c00640 1 -- 172.21.15.40:0/4269464477 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25400ba60 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff273200640 1 -- 172.21.15.40:0/4269464477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26c15bf60 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.452+0000 7ff273200640 1 -- 172.21.15.40:0/4269464477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26c15c520 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.456+0000 7ff273200640 1 -- 172.21.15.40:0/4269464477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff26c068170 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.456+0000 7ff268c00640 1 -- 172.21.15.40:0/4269464477 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff25400bc00 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.456+0000 7ff268c00640 1 --2- 172.21.15.40:0/4269464477 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff248067690 0x7ff248069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.456+0000 7ff268c00640 1 -- 172.21.15.40:0/4269464477 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff254085c60 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.456+0000 7ff26be00640 1 --2- 172.21.15.40:0/4269464477 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff248067690 0x7ff248069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:18.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.456+0000 7ff26be00640 1 --2- 172.21.15.40:0/4269464477 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff248067690 0x7ff248069b50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff25c000f70 tx=0x7ff25c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:18.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.460+0000 7ff268c00640 1 -- 172.21.15.40:0/4269464477 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff2540561f0 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.664+0000 7ff273200640 1 -- 172.21.15.40:0/4269464477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff26c072f50 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.664+0000 7ff268c00640 1 -- 172.21.15.40:0/4269464477 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4993 (secure 0 0 0) 0x7ff25405c030 con 0x7ff26c0b2ff0 2024-03-24T00:32:18.667 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:18.667 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2069836835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2069836835},{"type":"v1","addr":"172.21.15.53:6836","nonce":2069836835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":4856,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2159849028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":2159849028},{"type":"v1","addr":"172.21.15.53:6837","nonce":2159849028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:06.028086+0000","modified":"2024-03-24T00:32:18.276965+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":5229,"mds_1":5226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5226":{"gid":5226,"name":"a","rank":1,"incarnation":18,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6833/2326157501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2326157501},{"type":"v1","addr":"172.21.15.40:6833","nonce":2326157501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5229":{"gid":5229,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3050269371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3050269371},{"type":"v1","addr":"172.21.15.40:6835","nonce":3050269371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5229,"qdb_cluster":[5229]},"id":2}]} 2024-03-24T00:32:18.667 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 18 2024-03-24T00:32:18.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.668+0000 7ff273200640 1 -- 172.21.15.40:0/4269464477 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff248067690 msgr2=0x7ff248069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:18.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.668+0000 7ff273200640 1 --2- 172.21.15.40:0/4269464477 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff248067690 0x7ff248069b50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff25c000f70 tx=0x7ff25c009040 comp rx=0 tx=0).stop 2024-03-24T00:32:18.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.668+0000 7ff273200640 1 -- 172.21.15.40:0/4269464477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff26c0b2ff0 msgr2=0x7ff26c0cbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:18.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.668+0000 7ff273200640 1 --2- 172.21.15.40:0/4269464477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff26c0b2ff0 0x7ff26c0cbb20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff25400d800 tx=0x7ff25400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.668+0000 7ff273200640 1 -- 172.21.15.40:0/4269464477 shutdown_connections 2024-03-24T00:32:18.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.668+0000 7ff273200640 1 --2- 172.21.15.40:0/4269464477 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff248067690 0x7ff248069b50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.668+0000 7ff273200640 1 --2- 172.21.15.40:0/4269464477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff26c0b46c0 0x7ff26c13f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.668+0000 7ff273200640 1 --2- 172.21.15.40:0/4269464477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff26c0b2ff0 0x7ff26c0cbb20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.668+0000 7ff273200640 1 --2- 172.21.15.40:0/4269464477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff26c06e620 0x7ff26c1f74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:18.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.668+0000 7ff273200640 1 -- 172.21.15.40:0/4269464477 >> 172.21.15.40:0/4269464477 conn(0x7ff26c0b17b0 msgr2=0x7ff26c0cf030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:18.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.668+0000 7ff273200640 1 -- 172.21.15.40:0/4269464477 shutdown_connections 2024-03-24T00:32:18.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:18.668+0000 7ff273200640 1 -- 172.21.15.40:0/4269464477 wait complete. 2024-03-24T00:32:19.095 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:32:06.028086+0000', 'modified': '2024-03-24T00:32:18.276965+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 5229, 'mds_1': 5226}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5226': {'gid': 5226, 'name': 'a', 'rank': 1, 'incarnation': 18, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6833/2326157501', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 2326157501}, {'type': 'v1', 'addr': '172.21.15.40:6833', 'nonce': 2326157501}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5229': {'gid': 5229, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/3050269371', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 3050269371}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 3050269371}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5229, 'qdb_cluster': [5229]} 2024-03-24T00:32:19.095 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5226:up:creating 2024-03-24T00:32:20.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:32:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f368e600640 1 Processor -- start 2024-03-24T00:32:20.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f368e600640 1 -- start start 2024-03-24T00:32:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f368e600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f368806c8d0 0x7f36880b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f368e600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36880d0600 con 0x7f36880ced20 2024-03-24T00:32:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f368e600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36880d08e0 con 0x7f3688161880 2024-03-24T00:32:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f368e600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3688161c60 con 0x7f368806c8d0 2024-03-24T00:32:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f3686a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f36880ced20 0x7f36880cc950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54088/0 (socket says 172.21.15.40:54088) 2024-03-24T00:32:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f3686a00640 1 -- 172.21.15.40:0/2595756903 learned_addr learned my addr 172.21.15.40:0/2595756903 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:20.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f3687e00640 1 --2- 172.21.15.40:0/2595756903 >> v2:172.21.15.53:3300/0 conn(0x7f368806c8d0 0x7f36880b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:20.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f3686000640 1 -- 172.21.15.40:0/2595756903 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100592309 0 0) 0x7f36880d0600 con 0x7f36880ced20 2024-03-24T00:32:20.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f3686000640 1 -- 172.21.15.40:0/2595756903 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f366c003530 con 0x7f36880ced20 2024-03-24T00:32:20.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f3686000640 1 -- 172.21.15.40:0/2595756903 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250262593 0 0) 0x7f36880d08e0 con 0x7f3688161880 2024-03-24T00:32:20.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f3686000640 1 -- 172.21.15.40:0/2595756903 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36880d0600 con 0x7f3688161880 2024-03-24T00:32:20.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f3687e00640 1 -- 172.21.15.40:0/2595756903 >> v1:172.21.15.53:6789/0 conn(0x7f3688161880 legacy=0x7f36880ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:20.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f3686000640 1 -- 172.21.15.40:0/2595756903 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1090378154 0 0) 0x7f366c003530 con 0x7f36880ced20 2024-03-24T00:32:20.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f3687e00640 1 -- 172.21.15.40:0/2595756903 >> v1:172.21.15.40:6789/0 conn(0x7f36880ced20 legacy=0x7f36880cc950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:20.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.236+0000 7f3687e00640 1 -- 172.21.15.40:0/2595756903 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36880d0bc0 con 0x7f368806c8d0 2024-03-24T00:32:20.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.240+0000 7f3687e00640 1 --2- 172.21.15.40:0/2595756903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f368806c8d0 0x7f36880b2d50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3680009d90 tx=0x7f368001c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f55bc5cbe2c25ec8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:20.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.240+0000 7f3686000640 1 -- 172.21.15.40:0/2595756903 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3674001680 con 0x7f368806c8d0 2024-03-24T00:32:20.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.240+0000 7f3686000640 1 -- 172.21.15.40:0/2595756903 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36800a6450 con 0x7f368806c8d0 2024-03-24T00:32:20.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.240+0000 7f3686000640 1 -- 172.21.15.40:0/2595756903 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36800a6820 con 0x7f368806c8d0 2024-03-24T00:32:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.240+0000 7f368e600640 1 -- 172.21.15.40:0/2595756903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f368806c8d0 msgr2=0x7f36880b2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.240+0000 7f368e600640 1 --2- 172.21.15.40:0/2595756903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f368806c8d0 0x7f36880b2d50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3680009d90 tx=0x7f368001c800 comp rx=0 tx=0).stop 2024-03-24T00:32:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.240+0000 7f368e600640 1 -- 172.21.15.40:0/2595756903 shutdown_connections 2024-03-24T00:32:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.240+0000 7f368e600640 1 --2- 172.21.15.40:0/2595756903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f368806c8d0 0x7f36880b2d50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.240+0000 7f368e600640 1 -- 172.21.15.40:0/2595756903 >> 172.21.15.40:0/2595756903 conn(0x7f36880b38f0 msgr2=0x7f36880b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.240+0000 7f368e600640 1 -- 172.21.15.40:0/2595756903 shutdown_connections 2024-03-24T00:32:20.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.240+0000 7f368e600640 1 -- 172.21.15.40:0/2595756903 wait complete. 2024-03-24T00:32:20.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f368e600640 1 Processor -- start 2024-03-24T00:32:20.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f368e600640 1 -- start start 2024-03-24T00:32:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f368e600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f368806c8d0 0x7f3688153850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:20.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f368e600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36880ced20 0x7f36880c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:20.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f368e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3688161880 0x7f368815cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:20.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f368e600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36880cdc80 con 0x7f3688161880 2024-03-24T00:32:20.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f368e600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36880cdb00 con 0x7f368806c8d0 2024-03-24T00:32:20.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f368e600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f368806b430 con 0x7f36880ced20 2024-03-24T00:32:20.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f3687e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3688161880 0x7f368815cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:20.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f3686a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36880ced20 0x7f36880c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:20.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f3687e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3688161880 0x7f368815cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46274/0 (socket says 172.21.15.40:46274) 2024-03-24T00:32:20.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f3687e00640 1 -- 172.21.15.40:0/3119911711 learned_addr learned my addr 172.21.15.40:0/3119911711 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:20.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f3687400640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f368806c8d0 0x7f3688153850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:20.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f3687e00640 1 -- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36880ced20 msgr2=0x7f36880c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:20.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f3687e00640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36880ced20 0x7f36880c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:20.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f3687e00640 1 -- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f368806c8d0 msgr2=0x7f3688153850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:20.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f3687e00640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f368806c8d0 0x7f3688153850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:20.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f3687e00640 1 -- 172.21.15.40:0/3119911711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36880d2cc0 con 0x7f3688161880 2024-03-24T00:32:20.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f3686a00640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36880ced20 0x7f36880c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:20.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f3687e00640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3688161880 0x7f368815cc20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f368001cce0 tx=0x7f36800195a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:20.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f367f400640 1 -- 172.21.15.40:0/3119911711 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36800bf070 con 0x7f3688161880 2024-03-24T00:32:20.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f367f400640 1 -- 172.21.15.40:0/3119911711 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36800a7640 con 0x7f3688161880 2024-03-24T00:32:20.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f367f400640 1 -- 172.21.15.40:0/3119911711 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3680009450 con 0x7f3688161880 2024-03-24T00:32:20.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f368e600640 1 -- 172.21.15.40:0/3119911711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36880c63b0 con 0x7f3688161880 2024-03-24T00:32:20.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.244+0000 7f368e600640 1 -- 172.21.15.40:0/3119911711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36880c6970 con 0x7f3688161880 2024-03-24T00:32:20.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.248+0000 7f368e600640 1 -- 172.21.15.40:0/3119911711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3688068170 con 0x7f3688161880 2024-03-24T00:32:20.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.248+0000 7f367f400640 1 -- 172.21.15.40:0/3119911711 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f36800b0400 con 0x7f3688161880 2024-03-24T00:32:20.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.248+0000 7f367f400640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3668067660 0x7f3668069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:20.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.248+0000 7f367f400640 1 -- 172.21.15.40:0/3119911711 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f368011f920 con 0x7f3688161880 2024-03-24T00:32:20.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.248+0000 7f3687400640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3668067660 0x7f3668069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:20.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.252+0000 7f3687400640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3668067660 0x7f3668069b20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f36780011f0 tx=0x7f3678009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:20.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.252+0000 7f367f400640 1 -- 172.21.15.40:0/3119911711 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f36800ac030 con 0x7f3688161880 2024-03-24T00:32:20.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.452+0000 7f368e600640 1 -- 172.21.15.40:0/3119911711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3688159da0 con 0x7f3688161880 2024-03-24T00:32:20.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.452+0000 7f367f400640 1 -- 172.21.15.40:0/3119911711 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5015 (secure 0 0 0) 0x7f3680016480 con 0x7f3688161880 2024-03-24T00:32:20.456 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:20.456 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2069836835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2069836835},{"type":"v1","addr":"172.21.15.53:6836","nonce":2069836835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:06.028086+0000","modified":"2024-03-24T00:32:20.288717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5229,"mds_1":5226,"mds_2":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"b","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.53:6837/2159849028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":2159849028},{"type":"v1","addr":"172.21.15.53:6837","nonce":2159849028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5226":{"gid":5226,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6833/2326157501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2326157501},{"type":"v1","addr":"172.21.15.40:6833","nonce":2326157501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5229":{"gid":5229,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3050269371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3050269371},{"type":"v1","addr":"172.21.15.40:6835","nonce":3050269371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5229,"qdb_cluster":[4856,5226,5229]},"id":2}]} 2024-03-24T00:32:20.456 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 21 2024-03-24T00:32:20.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.456+0000 7f368e600640 1 -- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3668067660 msgr2=0x7f3668069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:20.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.456+0000 7f368e600640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3668067660 0x7f3668069b20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f36780011f0 tx=0x7f3678009040 comp rx=0 tx=0).stop 2024-03-24T00:32:20.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.456+0000 7f368e600640 1 -- 172.21.15.40:0/3119911711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3688161880 msgr2=0x7f368815cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:20.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.456+0000 7f368e600640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3688161880 0x7f368815cc20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f368001cce0 tx=0x7f36800195a0 comp rx=0 tx=0).stop 2024-03-24T00:32:20.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.456+0000 7f368e600640 1 -- 172.21.15.40:0/3119911711 shutdown_connections 2024-03-24T00:32:20.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.456+0000 7f368e600640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3668067660 0x7f3668069b20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:20.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.456+0000 7f368e600640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3688161880 0x7f368815cc20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:20.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.456+0000 7f368e600640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36880ced20 0x7f36880c7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:20.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.456+0000 7f368e600640 1 --2- 172.21.15.40:0/3119911711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f368806c8d0 0x7f3688153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:20.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.456+0000 7f368e600640 1 -- 172.21.15.40:0/3119911711 >> 172.21.15.40:0/3119911711 conn(0x7f36880b38f0 msgr2=0x7f36880d1d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:20.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.456+0000 7f368e600640 1 -- 172.21.15.40:0/3119911711 shutdown_connections 2024-03-24T00:32:20.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:20.456+0000 7f368e600640 1 -- 172.21.15.40:0/3119911711 wait complete. 2024-03-24T00:32:20.601 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-03-24T00:32:20.600+0000 7f3bb5000640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:32:20.602 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-03-24T00:32:20.600+0000 7f3bb5000640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-24T00:32:20.880 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:32:06.028086+0000', 'modified': '2024-03-24T00:32:20.288717+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 5229, 'mds_1': 5226, 'mds_2': 4856}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4856': {'gid': 4856, 'name': 'b', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.53:6837/2159849028', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6835', 'nonce': 2159849028}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 2159849028}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5226': {'gid': 5226, 'name': 'a', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.40:6833/2326157501', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 2326157501}, {'type': 'v1', 'addr': '172.21.15.40:6833', 'nonce': 2326157501}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5229': {'gid': 5229, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/3050269371', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 3050269371}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 3050269371}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5229, 'qdb_cluster': [4856, 5226, 5229]} 2024-03-24T00:32:20.880 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T00:32:20.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:32:21.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.020+0000 7f73eae00640 1 Processor -- start 2024-03-24T00:32:21.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.020+0000 7f73eae00640 1 -- start start 2024-03-24T00:32:21.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73eae00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73e40d2830 0x7f73e4166e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73eae00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e40af880 con 0x7f73e415f600 2024-03-24T00:32:21.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73eae00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e40b1e50 con 0x7f73e40b2910 2024-03-24T00:32:21.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73eae00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e40b3630 con 0x7f73e40d2830 2024-03-24T00:32:21.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73dbe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f73e40b2910 0x7f73e40af170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:56310/0 (socket says 172.21.15.40:56310) 2024-03-24T00:32:21.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73e3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f73e415f600 0x7f73e40aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54100/0 (socket says 172.21.15.40:54100) 2024-03-24T00:32:21.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73dbe00640 1 -- 172.21.15.40:0/257063128 learned_addr learned my addr 172.21.15.40:0/257063128 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:21.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73e8e00640 1 --2- 172.21.15.40:0/257063128 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73e40d2830 0x7f73e4166e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73e3400640 1 -- 172.21.15.40:0/257063128 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875262999 0 0) 0x7f73e40b1e50 con 0x7f73e40b2910 2024-03-24T00:32:21.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73e3400640 1 -- 172.21.15.40:0/257063128 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73c8003530 con 0x7f73e40b2910 2024-03-24T00:32:21.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73e3400640 1 -- 172.21.15.40:0/257063128 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496997629 0 0) 0x7f73e40af880 con 0x7f73e415f600 2024-03-24T00:32:21.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73e3400640 1 -- 172.21.15.40:0/257063128 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73e40b1e50 con 0x7f73e415f600 2024-03-24T00:32:21.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73e8e00640 1 -- 172.21.15.40:0/257063128 >> v1:172.21.15.53:6789/0 conn(0x7f73e40b2910 legacy=0x7f73e40af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73e8e00640 1 -- 172.21.15.40:0/257063128 >> v1:172.21.15.40:6789/0 conn(0x7f73e415f600 legacy=0x7f73e40aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.024+0000 7f73e8e00640 1 -- 172.21.15.40:0/257063128 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e40d2c40 con 0x7f73e40d2830 2024-03-24T00:32:21.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73e8e00640 1 --2- 172.21.15.40:0/257063128 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73e40d2830 0x7f73e4166e50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009dc0 tx=0x7f73dc01f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9c87d4cac1518f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73e3400640 1 -- 172.21.15.40:0/257063128 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc0b4070 con 0x7f73e40d2830 2024-03-24T00:32:21.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73e3400640 1 -- 172.21.15.40:0/257063128 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73dc0af600 con 0x7f73e40d2830 2024-03-24T00:32:21.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73e3400640 1 -- 172.21.15.40:0/257063128 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc0af9d0 con 0x7f73e40d2830 2024-03-24T00:32:21.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 -- 172.21.15.40:0/257063128 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73e40d2830 msgr2=0x7f73e4166e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 --2- 172.21.15.40:0/257063128 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73e40d2830 0x7f73e4166e50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009dc0 tx=0x7f73dc01f170 comp rx=0 tx=0).stop 2024-03-24T00:32:21.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 -- 172.21.15.40:0/257063128 shutdown_connections 2024-03-24T00:32:21.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 --2- 172.21.15.40:0/257063128 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73e40d2830 0x7f73e4166e50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 -- 172.21.15.40:0/257063128 >> 172.21.15.40:0/257063128 conn(0x7f73e40d1d10 msgr2=0x7f73e40d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 -- 172.21.15.40:0/257063128 shutdown_connections 2024-03-24T00:32:21.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 -- 172.21.15.40:0/257063128 wait complete. 2024-03-24T00:32:21.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 Processor -- start 2024-03-24T00:32:21.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 -- start start 2024-03-24T00:32:21.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73e40b2910 0x7f73e40c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73e40d2830 0x7f73e40c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73e415f600 0x7f73e40c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e40b2740 con 0x7f73e40b2910 2024-03-24T00:32:21.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e40b2d80 con 0x7f73e40d2830 2024-03-24T00:32:21.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.028+0000 7f73eae00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e40b23d0 con 0x7f73e415f600 2024-03-24T00:32:21.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73e40b2910 0x7f73e40c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73e40b2910 0x7f73e40c6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46286/0 (socket says 172.21.15.40:46286) 2024-03-24T00:32:21.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e3e00640 1 -- 172.21.15.40:0/2412206403 learned_addr learned my addr 172.21.15.40:0/2412206403 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73dbe00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73e40d2830 0x7f73e40c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e8e00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73e415f600 0x7f73e40c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e3e00640 1 -- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73e415f600 msgr2=0x7f73e40c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e3e00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73e415f600 0x7f73e40c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e3e00640 1 -- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73e40d2830 msgr2=0x7f73e40c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e3e00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73e40d2830 0x7f73e40c8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e3e00640 1 -- 172.21.15.40:0/2412206403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e40c55c0 con 0x7f73e40b2910 2024-03-24T00:32:21.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e3e00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73e40b2910 0x7f73e40c6900 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f73d400c390 tx=0x7f73d400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e0c00640 1 -- 172.21.15.40:0/2412206403 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d40074e0 con 0x7f73e40b2910 2024-03-24T00:32:21.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e0c00640 1 -- 172.21.15.40:0/2412206403 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d40047f0 con 0x7f73e40b2910 2024-03-24T00:32:21.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73eae00640 1 -- 172.21.15.40:0/2412206403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e40c5860 con 0x7f73e40b2910 2024-03-24T00:32:21.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e0c00640 1 -- 172.21.15.40:0/2412206403 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d40076a0 con 0x7f73e40b2910 2024-03-24T00:32:21.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73eae00640 1 -- 172.21.15.40:0/2412206403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e40c5e20 con 0x7f73e40b2910 2024-03-24T00:32:21.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.032+0000 7f73e0c00640 1 -- 172.21.15.40:0/2412206403 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f73d401a020 con 0x7f73e40b2910 2024-03-24T00:32:21.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.036+0000 7f73e0c00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f73c0067660 0x7f73c0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.036+0000 7f73e0c00640 1 -- 172.21.15.40:0/2412206403 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73d40850d0 con 0x7f73e40b2910 2024-03-24T00:32:21.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.036+0000 7f73dbe00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f73c0067660 0x7f73c0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.036+0000 7f73dbe00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f73c0067660 0x7f73c0069b20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f73e40b1700 tx=0x7f73cc007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.036+0000 7f73ee34e000 1 Processor -- start 2024-03-24T00:32:21.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.036+0000 7f73ee34e000 1 -- start start 2024-03-24T00:32:21.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.036+0000 7f73ee34e000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 0x55d41cad9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.036+0000 7f73ee34e000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cada240 0x55d41ca8fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.036+0000 7f73ee34e000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41ca8fff0 0x55d41ca903d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.036+0000 7f73ee34e000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41c8dd680 con 0x55d41ca8fff0 2024-03-24T00:32:21.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.036+0000 7f73ee34e000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41c864b00 con 0x55d41cad9920 2024-03-24T00:32:21.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.036+0000 7f73ee34e000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d41c8d9720 con 0x55d41cada240 2024-03-24T00:32:21.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73e3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 0x55d41cad9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73e8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41ca8fff0 0x55d41ca903d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73e3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 0x55d41cad9d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:59390/0 (socket says 172.21.15.40:59390) 2024-03-24T00:32:21.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73e3e00640 1 -- 172.21.15.40:0/1427134233 learned_addr learned my addr 172.21.15.40:0/1427134233 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73dbe00640 1 --2- 172.21.15.40:0/1427134233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cada240 0x55d41ca8fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73e3e00640 1 -- 172.21.15.40:0/1427134233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cada240 msgr2=0x55d41ca8fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73e3e00640 1 --2- 172.21.15.40:0/1427134233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cada240 0x55d41ca8fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73e3e00640 1 -- 172.21.15.40:0/1427134233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41ca8fff0 msgr2=0x55d41ca903d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73e3e00640 1 --2- 172.21.15.40:0/1427134233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41ca8fff0 0x55d41ca903d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73e3e00640 1 -- 172.21.15.40:0/1427134233 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d41c8b5940 con 0x55d41cad9920 2024-03-24T00:32:21.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73e3e00640 1 --2- 172.21.15.40:0/1427134233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 0x55d41cad9d00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73d4051a40 tx=0x7f73d40597c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4a0be1917c11868 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73d8c00640 1 -- 172.21.15.40:0/1427134233 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4051c50 con 0x55d41cad9920 2024-03-24T00:32:21.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73d8c00640 1 -- 172.21.15.40:0/1427134233 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d4004be0 con 0x55d41cad9920 2024-03-24T00:32:21.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73d8c00640 1 -- 172.21.15.40:0/1427134233 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4055840 con 0x55d41cad9920 2024-03-24T00:32:21.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 -- 172.21.15.40:0/1427134233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 msgr2=0x55d41cad9d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 --2- 172.21.15.40:0/1427134233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 0x55d41cad9d00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73d4051a40 tx=0x7f73d40597c0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 -- 172.21.15.40:0/1427134233 shutdown_connections 2024-03-24T00:32:21.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 --2- 172.21.15.40:0/1427134233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41ca8fff0 0x55d41ca903d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 --2- 172.21.15.40:0/1427134233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cada240 0x55d41ca8fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 --2- 172.21.15.40:0/1427134233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 0x55d41cad9d00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 -- 172.21.15.40:0/1427134233 >> 172.21.15.40:0/1427134233 conn(0x55d41ca8bfa0 msgr2=0x55d41ca8e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 -- 172.21.15.40:0/1427134233 shutdown_connections 2024-03-24T00:32:21.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 -- 172.21.15.40:0/1427134233 wait complete. 2024-03-24T00:32:21.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 -- start start 2024-03-24T00:32:21.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 Processor -- start 2024-03-24T00:32:21.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 0x55d41cabed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cad9920 0x55d41cabf280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41cabf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41c8dc990 con 0x55d41cada240 2024-03-24T00:32:21.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41c897ae0 con 0x55d41ca8fff0 2024-03-24T00:32:21.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.040+0000 7f73ee34e000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d41c875750 con 0x55d41cad9920 2024-03-24T00:32:21.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 10 client.0 _ll_register_callbacks cb 0x55d41ca7aff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:32:21.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73dbe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cad9920 0x55d41cabf280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73e3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 0x55d41cabed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73e8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41cabf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73e3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 0x55d41cabed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:59400/0 (socket says 172.21.15.40:59400) 2024-03-24T00:32:21.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73e3e00640 1 -- 172.21.15.40:0/867739414 learned_addr learned my addr 172.21.15.40:0/867739414 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73e3e00640 1 -- 172.21.15.40:0/867739414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cad9920 msgr2=0x55d41cabf280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73e3e00640 1 --2- 172.21.15.40:0/867739414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cad9920 0x55d41cabf280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73e3e00640 1 -- 172.21.15.40:0/867739414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 msgr2=0x55d41cabf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73e3e00640 1 --2- 172.21.15.40:0/867739414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41cabf7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73e3e00640 1 -- 172.21.15.40:0/867739414 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d41c922d80 con 0x55d41ca8fff0 2024-03-24T00:32:21.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73e8e00640 1 --2- 172.21.15.40:0/867739414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41cabf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:21.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73e3e00640 1 --2- 172.21.15.40:0/867739414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 0x55d41cabed40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f73d4055520 tx=0x7f73d4000bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=273bc4a36ce72d4e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73d1600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-24T00:32:21.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73d1600640 1 -- 172.21.15.40:0/867739414 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4051c50 con 0x55d41ca8fff0 2024-03-24T00:32:21.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73d1600640 1 -- 172.21.15.40:0/867739414 <== mon.1 v2:172.21.15.53:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73d4072380 con 0x55d41ca8fff0 2024-03-24T00:32:21.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 1 -- 172.21.15.40:0/867739414 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73d4004be0 con 0x55d41ca8fff0 2024-03-24T00:32:21.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73d1600640 1 -- 172.21.15.40:0/867739414 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73d4051df0 con 0x55d41ca8fff0 2024-03-24T00:32:21.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 10 client.5365 fetch_fsmap learned FSMap version 21 2024-03-24T00:32:21.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 1 -- 172.21.15.40:0/867739414 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55d41c864d10 con 0x55d41ca8fff0 2024-03-24T00:32:21.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73d1600640 1 -- 172.21.15.40:0/867739414 <== mon.1 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d4004be0 con 0x55d41ca8fff0 2024-03-24T00:32:21.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73d1600640 1 -- 172.21.15.40:0/867739414 <== mon.1 v2:172.21.15.53:3300/0 5 ==== fsmap(e 21) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f73d40830a0 con 0x55d41ca8fff0 2024-03-24T00:32:21.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 10 client.5365 fetch_fsmap finished waiting for FSMap version 21 2024-03-24T00:32:21.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 10 client.5365 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:32:21.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 20 client.5365 populate_metadata read hostname 'smithi040' 2024-03-24T00:32:21.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 1 --2- 172.21.15.40:0/867739414 >> [v2:172.21.15.53:6835/2159849028,v1:172.21.15.53:6837/2159849028] conn(0x55d41cac0060 0x55d41caebea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 4 client.5365 mds_command: new command op to 4856 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:32:21.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 1 -- 172.21.15.40:0/867739414 --> [v2:172.21.15.53:6835/2159849028,v1:172.21.15.53:6837/2159849028] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d41c878b60 con 0x55d41cac0060 2024-03-24T00:32:21.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73dbe00640 1 --2- 172.21.15.40:0/867739414 >> [v2:172.21.15.53:6835/2159849028,v1:172.21.15.53:6837/2159849028] conn(0x55d41cac0060 0x55d41caebea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73dbe00640 1 --2- 172.21.15.40:0/867739414 >> [v2:172.21.15.53:6835/2159849028,v1:172.21.15.53:6837/2159849028] conn(0x55d41cac0060 0x55d41caebea0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2217864f6487e61a server_cookie=f77d88cac74d3bae in_seq=0 out_seq=0 2024-03-24T00:32:21.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73d1600640 10 client.5365 ms_handle_connect on v2:172.21.15.53:6835/2159849028 2024-03-24T00:32:21.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73d1600640 1 -- 172.21.15.40:0/867739414 <== mds.2 v2:172.21.15.53:6835/2159849028 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d41c878b60 con 0x55d41cac0060 2024-03-24T00:32:21.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73d1600640 10 client.5365 handle_command_reply: tid=0 2024-03-24T00:32:21.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 1 client.5365 shutdown 2024-03-24T00:32:21.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 1 -- 172.21.15.40:0/867739414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 msgr2=0x55d41cabed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.044+0000 7f73ee34e000 1 --2- 172.21.15.40:0/867739414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 0x55d41cabed40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f73d4055520 tx=0x7f73d4000bd0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.048+0000 7f73ee34e000 1 -- 172.21.15.40:0/867739414 shutdown_connections 2024-03-24T00:32:21.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.048+0000 7f73ee34e000 1 --2- 172.21.15.40:0/867739414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41cabf7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.048+0000 7f73ee34e000 1 --2- 172.21.15.40:0/867739414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cad9920 0x55d41cabf280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.048+0000 7f73ee34e000 1 --2- 172.21.15.40:0/867739414 >> [v2:172.21.15.53:6835/2159849028,v1:172.21.15.53:6837/2159849028] conn(0x55d41cac0060 0x55d41caebea0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.048+0000 7f73ee34e000 1 --2- 172.21.15.40:0/867739414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 0x55d41cabed40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.048+0000 7f73d1600640 0 client.5365 ms_handle_reset on v2:172.21.15.53:6835/2159849028 2024-03-24T00:32:21.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.048+0000 7f73ee34e000 1 -- 172.21.15.40:0/867739414 >> 172.21.15.40:0/867739414 conn(0x55d41ca8bfa0 msgr2=0x55d41ca8ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.048+0000 7f73ee34e000 1 -- 172.21.15.40:0/867739414 shutdown_connections 2024-03-24T00:32:21.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.048+0000 7f73ee34e000 1 -- 172.21.15.40:0/867739414 wait complete. 2024-03-24T00:32:21.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.048+0000 7f73ee34e000 20 client.5365 trim_cache size 0 max 16384 2024-03-24T00:32:21.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 Processor -- start 2024-03-24T00:32:21.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 -- start start 2024-03-24T00:32:21.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 0x55d41cad8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cad9920 0x55d41cabd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41cabde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e41ffbf0 con 0x55d41cada240 2024-03-24T00:32:21.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e4151700 con 0x55d41ca8fff0 2024-03-24T00:32:21.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d41c885a80 con 0x55d41cad9920 2024-03-24T00:32:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73e3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 0x55d41cad8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73e8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41cabde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73e3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 0x55d41cad8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:59414/0 (socket says 172.21.15.40:59414) 2024-03-24T00:32:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73e3e00640 1 -- 172.21.15.40:0/493885517 learned_addr learned my addr 172.21.15.40:0/493885517 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73dbe00640 1 --2- 172.21.15.40:0/493885517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cad9920 0x55d41cabd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73e3e00640 1 -- 172.21.15.40:0/493885517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cad9920 msgr2=0x55d41cabd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73e3e00640 1 --2- 172.21.15.40:0/493885517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cad9920 0x55d41cabd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73e3e00640 1 -- 172.21.15.40:0/493885517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 msgr2=0x55d41cabde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73e3e00640 1 --2- 172.21.15.40:0/493885517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41cabde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73e3e00640 1 -- 172.21.15.40:0/493885517 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d41c8aa540 con 0x55d41ca8fff0 2024-03-24T00:32:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73e3e00640 1 --2- 172.21.15.40:0/493885517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 0x55d41cad8e20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f73d400cd50 tx=0x7f73d406b070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e97e26130c0ccb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73d3e00640 1 -- 172.21.15.40:0/493885517 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4004be0 con 0x55d41ca8fff0 2024-03-24T00:32:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73d3e00640 1 -- 172.21.15.40:0/493885517 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d406d370 con 0x55d41ca8fff0 2024-03-24T00:32:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73d3e00640 1 -- 172.21.15.40:0/493885517 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d4072180 con 0x55d41ca8fff0 2024-03-24T00:32:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 -- 172.21.15.40:0/493885517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 msgr2=0x55d41cad8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 --2- 172.21.15.40:0/493885517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 0x55d41cad8e20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f73d400cd50 tx=0x7f73d406b070 comp rx=0 tx=0).stop 2024-03-24T00:32:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 -- 172.21.15.40:0/493885517 shutdown_connections 2024-03-24T00:32:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 --2- 172.21.15.40:0/493885517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41cabde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 --2- 172.21.15.40:0/493885517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41cad9920 0x55d41cabd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 --2- 172.21.15.40:0/493885517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41ca8fff0 0x55d41cad8e20 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 -- 172.21.15.40:0/493885517 >> 172.21.15.40:0/493885517 conn(0x55d41ca8bfa0 msgr2=0x55d41cad8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 -- 172.21.15.40:0/493885517 shutdown_connections 2024-03-24T00:32:21.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.064+0000 7f73ee34e000 1 -- 172.21.15.40:0/493885517 wait complete. 2024-03-24T00:32:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 1 -- start start 2024-03-24T00:32:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 1 Processor -- start 2024-03-24T00:32:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41ca8fff0 0x55d41cabced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 0x55d41caecef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41caed430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41c875750 con 0x55d41cada240 2024-03-24T00:32:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d41c99a700 con 0x55d41cad9920 2024-03-24T00:32:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d41c897ae0 con 0x55d41ca8fff0 2024-03-24T00:32:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 10 client.0 _ll_register_callbacks cb 0x7f73c80048f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:32:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73e8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41caed430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73e8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41caed430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46336/0 (socket says 172.21.15.40:46336) 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73e8e00640 1 -- 172.21.15.40:0/1211174025 learned_addr learned my addr 172.21.15.40:0/1211174025 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73dbe00640 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 0x55d41caecef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73e3e00640 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41ca8fff0 0x55d41cabced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73dbe00640 1 -- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41ca8fff0 msgr2=0x55d41cabced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73dbe00640 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41ca8fff0 0x55d41cabced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73dbe00640 1 -- 172.21.15.40:0/1211174025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 msgr2=0x55d41caed430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73dbe00640 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41caed430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73dbe00640 1 -- 172.21.15.40:0/1211174025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d41c8714b0 con 0x55d41cad9920 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73e8e00640 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41caed430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73dbe00640 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 0x55d41caecef0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f73e415bc60 tx=0x7f73cc00a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53b358fff8641674 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73d1600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73d1600640 1 -- 172.21.15.40:0/1211174025 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc001700 con 0x55d41cad9920 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73d1600640 1 -- 172.21.15.40:0/1211174025 <== mon.1 v2:172.21.15.53:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73cc0018a0 con 0x55d41cad9920 2024-03-24T00:32:21.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 1 -- 172.21.15.40:0/1211174025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73d406d370 con 0x55d41cad9920 2024-03-24T00:32:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73d1600640 1 -- 172.21.15.40:0/1211174025 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73cc038050 con 0x55d41cad9920 2024-03-24T00:32:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 10 client.5371 fetch_fsmap learned FSMap version 21 2024-03-24T00:32:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 1 -- 172.21.15.40:0/1211174025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x55d41c8842f0 con 0x55d41cad9920 2024-03-24T00:32:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73d1600640 1 -- 172.21.15.40:0/1211174025 <== mon.1 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d406d370 con 0x55d41cad9920 2024-03-24T00:32:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73d1600640 1 -- 172.21.15.40:0/1211174025 <== mon.1 v2:172.21.15.53:3300/0 5 ==== fsmap(e 21) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f73cc03b020 con 0x55d41cad9920 2024-03-24T00:32:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 10 client.5371 fetch_fsmap finished waiting for FSMap version 21 2024-03-24T00:32:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 10 client.5371 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:32:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 20 client.5371 populate_metadata read hostname 'smithi040' 2024-03-24T00:32:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:6835/2159849028,v1:172.21.15.53:6837/2159849028] conn(0x55d41cb3e5c0 0x55d41cb3e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 4 client.5371 mds_command: new command op to 4856 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:32:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73ee34e000 1 -- 172.21.15.40:0/1211174025 --> [v2:172.21.15.53:6835/2159849028,v1:172.21.15.53:6837/2159849028] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d41c8eabc0 con 0x55d41cb3e5c0 2024-03-24T00:32:21.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.068+0000 7f73e3e00640 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:6835/2159849028,v1:172.21.15.53:6837/2159849028] conn(0x55d41cb3e5c0 0x55d41cb3e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73e3e00640 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:6835/2159849028,v1:172.21.15.53:6837/2159849028] conn(0x55d41cb3e5c0 0x55d41cb3e9a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4cbd8eaf2ed020e6 server_cookie=c04fd6e9f210f1bf in_seq=0 out_seq=0 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73d1600640 10 client.5371 ms_handle_connect on v2:172.21.15.53:6835/2159849028 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73d1600640 1 -- 172.21.15.40:0/1211174025 <== mds.2 v2:172.21.15.53:6835/2159849028 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55d41c8eabc0 con 0x55d41cb3e5c0 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73d1600640 10 client.5371 handle_command_reply: tid=0 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73ee34e000 1 client.5371 shutdown 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73ee34e000 1 -- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 msgr2=0x55d41caecef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73ee34e000 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 0x55d41caecef0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f73e415bc60 tx=0x7f73cc00a000 comp rx=0 tx=0).stop 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73ee34e000 1 -- 172.21.15.40:0/1211174025 shutdown_connections 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73ee34e000 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d41cada240 0x55d41caed430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73ee34e000 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:6835/2159849028,v1:172.21.15.53:6837/2159849028] conn(0x55d41cb3e5c0 0x55d41cb3e9a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73ee34e000 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d41cad9920 0x55d41caecef0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73ee34e000 1 --2- 172.21.15.40:0/1211174025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d41ca8fff0 0x55d41cabced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73d1600640 0 client.5371 ms_handle_reset on v2:172.21.15.53:6835/2159849028 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73ee34e000 1 -- 172.21.15.40:0/1211174025 >> 172.21.15.40:0/1211174025 conn(0x55d41ca8bfa0 msgr2=0x55d41cad8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73ee34e000 1 -- 172.21.15.40:0/1211174025 shutdown_connections 2024-03-24T00:32:21.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73ee34e000 1 -- 172.21.15.40:0/1211174025 wait complete. 2024-03-24T00:32:21.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73ee34e000 20 client.5371 trim_cache size 0 max 16384 2024-03-24T00:32:21.077 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:32:21.077 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:32:21.077 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-03-24T00:32:21.077 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 4856, 2024-03-24T00:32:21.077 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:32:21.077 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:32:21.077 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:32:21.077 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 1.7842204719999999, 2024-03-24T00:32:21.077 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 21, 2024-03-24T00:32:21.078 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 33, 2024-03-24T00:32:21.078 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 33, 2024-03-24T00:32:21.078 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 22.300926904000001 2024-03-24T00:32:21.078 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:32:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73eae00640 1 -- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f73c0067660 msgr2=0x7f73c0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73eae00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f73c0067660 0x7f73c0069b20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f73e40b1700 tx=0x7f73cc007d40 comp rx=0 tx=0).stop 2024-03-24T00:32:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73eae00640 1 -- 172.21.15.40:0/2412206403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73e40b2910 msgr2=0x7f73e40c6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73eae00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73e40b2910 0x7f73e40c6900 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f73d400c390 tx=0x7f73d400c850 comp rx=0 tx=0).stop 2024-03-24T00:32:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73eae00640 1 -- 172.21.15.40:0/2412206403 shutdown_connections 2024-03-24T00:32:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73eae00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f73c0067660 0x7f73c0069b20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73eae00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73e415f600 0x7f73e40c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73eae00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73e40d2830 0x7f73e40c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73eae00640 1 --2- 172.21.15.40:0/2412206403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73e40b2910 0x7f73e40c6900 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73eae00640 1 -- 172.21.15.40:0/2412206403 >> 172.21.15.40:0/2412206403 conn(0x7f73e40d1d10 msgr2=0x7f73e406b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73eae00640 1 -- 172.21.15.40:0/2412206403 shutdown_connections 2024-03-24T00:32:21.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.072+0000 7f73eae00640 1 -- 172.21.15.40:0/2412206403 wait complete. 2024-03-24T00:32:21.099 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:32:21.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3edf000640 1 Processor -- start 2024-03-24T00:32:21.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3edf000640 1 -- start start 2024-03-24T00:32:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3edf000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ed8066e80 0x7f3ed8067300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3edf000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ed806de50 0x7f3ed806e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3edf000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ed80af600 con 0x7f3ed8156320 2024-03-24T00:32:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3edf000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ed80b0470 con 0x7f3ed806de50 2024-03-24T00:32:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3edf000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed80af8a0 con 0x7f3ed8066e80 2024-03-24T00:32:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3ed7e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3ed8156320 0x7f3ed8072920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54106/0 (socket says 172.21.15.40:54106) 2024-03-24T00:32:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3ed7e00640 1 -- 172.21.15.40:0/2926776063 learned_addr learned my addr 172.21.15.40:0/2926776063 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/2926776063 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ed8066e80 0x7f3ed8067300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3edd000640 1 --2- 172.21.15.40:0/2926776063 >> v2:172.21.15.40:3300/0 conn(0x7f3ed806de50 0x7f3ed806e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3ecfe00640 1 -- 172.21.15.40:0/2926776063 >> v2:172.21.15.40:3300/0 conn(0x7f3ed806de50 msgr2=0x7f3ed806e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/2926776063 >> v2:172.21.15.40:3300/0 conn(0x7f3ed806de50 0x7f3ed806e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3ecfe00640 1 -- 172.21.15.40:0/2926776063 >> v1:172.21.15.40:6789/0 conn(0x7f3ed8156320 legacy=0x7f3ed8072920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3ecfe00640 1 -- 172.21.15.40:0/2926776063 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed80afb40 con 0x7f3ed8066e80 2024-03-24T00:32:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.240+0000 7f3ed7400640 1 -- 172.21.15.40:0/2926776063 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3100505443 0 0) 0x7f3ed80af600 con 0x7f3ed8156320 2024-03-24T00:32:21.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.244+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/2926776063 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ed8066e80 0x7f3ed8067300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00097c0 tx=0x7f3ec001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b69f65265ad8b925 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.244+0000 7f3ed7400640 1 -- 172.21.15.40:0/2926776063 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00b0070 con 0x7f3ed8066e80 2024-03-24T00:32:21.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.244+0000 7f3ed7400640 1 -- 172.21.15.40:0/2926776063 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec00ab400 con 0x7f3ed8066e80 2024-03-24T00:32:21.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.244+0000 7f3ed7400640 1 -- 172.21.15.40:0/2926776063 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00ab7d0 con 0x7f3ed8066e80 2024-03-24T00:32:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.244+0000 7f3edf000640 1 -- 172.21.15.40:0/2926776063 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ed8066e80 msgr2=0x7f3ed8067300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.244+0000 7f3edf000640 1 --2- 172.21.15.40:0/2926776063 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ed8066e80 0x7f3ed8067300 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00097c0 tx=0x7f3ec001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.244+0000 7f3edf000640 1 -- 172.21.15.40:0/2926776063 shutdown_connections 2024-03-24T00:32:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.244+0000 7f3edf000640 1 --2- 172.21.15.40:0/2926776063 >> v2:172.21.15.40:3300/0 conn(0x7f3ed806de50 0x7f3ed806e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.244+0000 7f3edf000640 1 --2- 172.21.15.40:0/2926776063 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ed8066e80 0x7f3ed8067300 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.244+0000 7f3edf000640 1 -- 172.21.15.40:0/2926776063 >> 172.21.15.40:0/2926776063 conn(0x7f3ed806b150 msgr2=0x7f3ed806b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.244+0000 7f3edf000640 1 -- 172.21.15.40:0/2926776063 shutdown_connections 2024-03-24T00:32:21.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.244+0000 7f3edf000640 1 -- 172.21.15.40:0/2926776063 wait complete. 2024-03-24T00:32:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edf000640 1 Processor -- start 2024-03-24T00:32:21.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edf000640 1 -- start start 2024-03-24T00:32:21.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edf000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ed8066e80 0x7f3ed80d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edf000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ed806de50 0x7f3ed80b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edf000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed8156320 0x7f3ed809ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edf000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed80ae080 con 0x7f3ed8156320 2024-03-24T00:32:21.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edf000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed80adf00 con 0x7f3ed806de50 2024-03-24T00:32:21.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edf000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed8166cd0 con 0x7f3ed8066e80 2024-03-24T00:32:21.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edd000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed8156320 0x7f3ed809ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edd000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed8156320 0x7f3ed809ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46352/0 (socket says 172.21.15.40:46352) 2024-03-24T00:32:21.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edd000640 1 -- 172.21.15.40:0/3215537555 learned_addr learned my addr 172.21.15.40:0/3215537555 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ed806de50 0x7f3ed80b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3ed7e00640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ed8066e80 0x7f3ed80d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3ecfe00640 1 -- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ed8066e80 msgr2=0x7f3ed80d0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ed8066e80 0x7f3ed80d0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3ecfe00640 1 -- 172.21.15.40:0/3215537555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed8156320 msgr2=0x7f3ed809ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed8156320 0x7f3ed809ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3ecfe00640 1 -- 172.21.15.40:0/3215537555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed80d11b0 con 0x7f3ed806de50 2024-03-24T00:32:21.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ed806de50 0x7f3ed80b08d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00ab000 tx=0x7f3ec0019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3ed4c00640 1 -- 172.21.15.40:0/3215537555 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00b0040 con 0x7f3ed806de50 2024-03-24T00:32:21.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3ed4c00640 1 -- 172.21.15.40:0/3215537555 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec00b9070 con 0x7f3ed806de50 2024-03-24T00:32:21.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edf000640 1 -- 172.21.15.40:0/3215537555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed80b1390 con 0x7f3ed806de50 2024-03-24T00:32:21.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3ed4c00640 1 -- 172.21.15.40:0/3215537555 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00b4870 con 0x7f3ed806de50 2024-03-24T00:32:21.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.248+0000 7f3edf000640 1 -- 172.21.15.40:0/3215537555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed80b1950 con 0x7f3ed806de50 2024-03-24T00:32:21.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ed4c00640 1 -- 172.21.15.40:0/3215537555 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3ec00b4a10 con 0x7f3ed806de50 2024-03-24T00:32:21.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ee2442000 1 Processor -- start 2024-03-24T00:32:21.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ed4c00640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3eb4067660 0x7f3eb4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ee2442000 1 -- start start 2024-03-24T00:32:21.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ed4c00640 1 -- 172.21.15.40:0/3215537555 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ec011f540 con 0x7f3ed806de50 2024-03-24T00:32:21.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3edd000640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3eb4067660 0x7f3eb4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ee2442000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b250920 0x561a6b250d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ee2442000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b251240 0x561a6b206ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ee2442000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b2073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ee2442000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a6b054680 con 0x561a6b250920 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ee2442000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a6afdbb00 con 0x561a6b206ff0 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ee2442000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a6b050720 con 0x561a6b251240 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3edd000640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3eb4067660 0x7f3eb4069b20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000dfd0 tx=0x7f3ed000f040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ecfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b251240 0x561a6b206ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ed7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b250920 0x561a6b250d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ecfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b251240 0x561a6b206ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:55180/0 (socket says 172.21.15.40:55180) 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3edd000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b2073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.252+0000 7f3ecfe00640 1 -- 172.21.15.40:0/785370332 learned_addr learned my addr 172.21.15.40:0/785370332 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ecfe00640 1 -- 172.21.15.40:0/785370332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 msgr2=0x561a6b2073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/785370332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b2073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ecfe00640 1 -- 172.21.15.40:0/785370332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b250920 msgr2=0x561a6b250d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/785370332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b250920 0x561a6b250d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ecfe00640 1 -- 172.21.15.40:0/785370332 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a6b02c940 con 0x561a6b251240 2024-03-24T00:32:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/785370332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b251240 0x561a6b206ab0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00ae040 tx=0x7f3ec00ef430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ece58f0af291bf50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3eccc00640 1 -- 172.21.15.40:0/785370332 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00f8130 con 0x561a6b251240 2024-03-24T00:32:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3eccc00640 1 -- 172.21.15.40:0/785370332 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec00eddd0 con 0x561a6b251240 2024-03-24T00:32:21.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3eccc00640 1 -- 172.21.15.40:0/785370332 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00f7ca0 con 0x561a6b251240 2024-03-24T00:32:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 -- 172.21.15.40:0/785370332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b251240 msgr2=0x561a6b206ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 --2- 172.21.15.40:0/785370332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b251240 0x561a6b206ab0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00ae040 tx=0x7f3ec00ef430 comp rx=0 tx=0).stop 2024-03-24T00:32:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 -- 172.21.15.40:0/785370332 shutdown_connections 2024-03-24T00:32:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 --2- 172.21.15.40:0/785370332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b2073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 --2- 172.21.15.40:0/785370332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b251240 0x561a6b206ab0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 --2- 172.21.15.40:0/785370332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b250920 0x561a6b250d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 -- 172.21.15.40:0/785370332 >> 172.21.15.40:0/785370332 conn(0x561a6b202fa0 msgr2=0x561a6b205390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 -- 172.21.15.40:0/785370332 shutdown_connections 2024-03-24T00:32:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 -- 172.21.15.40:0/785370332 wait complete. 2024-03-24T00:32:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 -- start start 2024-03-24T00:32:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 Processor -- start 2024-03-24T00:32:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b235d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b250920 0x561a6b236280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b251240 0x561a6b2367c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a6b053990 con 0x561a6b250920 2024-03-24T00:32:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a6b00eae0 con 0x561a6b206ff0 2024-03-24T00:32:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a6afec750 con 0x561a6b251240 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ee2442000 10 client.0 _ll_register_callbacks cb 0x561a6b1f1ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ed7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b235d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ecfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b250920 0x561a6b236280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ed7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b235d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:59436/0 (socket says 172.21.15.40:59436) 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3edd000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b251240 0x561a6b2367c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.256+0000 7f3ed7e00640 1 -- 172.21.15.40:0/3859985447 learned_addr learned my addr 172.21.15.40:0/3859985447 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ecfe00640 1 -- 172.21.15.40:0/3859985447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b251240 msgr2=0x561a6b2367c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/3859985447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b251240 0x561a6b2367c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ecfe00640 1 -- 172.21.15.40:0/3859985447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 msgr2=0x561a6b235d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/3859985447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b235d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ecfe00640 1 -- 172.21.15.40:0/3859985447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a6b099d80 con 0x561a6b250920 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ed7e00640 1 --2- 172.21.15.40:0/3859985447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b235d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:21.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/3859985447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b250920 0x561a6b236280 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00ef7d0 tx=0x7f3ec0000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6db8d4b34fee8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ec5600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ec5600640 1 -- 172.21.15.40:0/3859985447 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec00f7ea0 con 0x561a6b250920 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ec5600640 1 -- 172.21.15.40:0/3859985447 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ec00eef20 con 0x561a6b250920 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 -- 172.21.15.40:0/3859985447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ec00eddd0 con 0x561a6b250920 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ec5600640 1 -- 172.21.15.40:0/3859985447 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ec00f8130 con 0x561a6b250920 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 10 client.5493 fetch_fsmap learned FSMap version 21 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 -- 172.21.15.40:0/3859985447 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x561a6afdbd10 con 0x561a6b250920 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ec5600640 1 -- 172.21.15.40:0/3859985447 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec00eddd0 con 0x561a6b250920 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ec5600640 1 -- 172.21.15.40:0/3859985447 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 21) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f3ec00b3070 con 0x561a6b250920 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 10 client.5493 fetch_fsmap finished waiting for FSMap version 21 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 10 client.5493 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 20 client.5493 populate_metadata read hostname 'smithi040' 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 --2- 172.21.15.40:0/3859985447 >> [v2:172.21.15.40:6832/2326157501,v1:172.21.15.40:6833/2326157501] conn(0x561a6b237090 0x561a6b262ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 4 client.5493 mds_command: new command op to 5226 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:32:21.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 -- 172.21.15.40:0/3859985447 --> [v2:172.21.15.40:6832/2326157501,v1:172.21.15.40:6833/2326157501] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561a6afefb60 con 0x561a6b237090 2024-03-24T00:32:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ed7e00640 1 --2- 172.21.15.40:0/3859985447 >> [v2:172.21.15.40:6832/2326157501,v1:172.21.15.40:6833/2326157501] conn(0x561a6b237090 0x561a6b262ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ed7e00640 1 --2- 172.21.15.40:0/3859985447 >> [v2:172.21.15.40:6832/2326157501,v1:172.21.15.40:6833/2326157501] conn(0x561a6b237090 0x561a6b262ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f3ea0b3e456c07e6 server_cookie=5e3f245f0ddec961 in_seq=0 out_seq=0 2024-03-24T00:32:21.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ec5600640 10 client.5493 ms_handle_connect on v2:172.21.15.40:6832/2326157501 2024-03-24T00:32:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ec5600640 1 -- 172.21.15.40:0/3859985447 <== mds.1 v2:172.21.15.40:6832/2326157501 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561a6afefb60 con 0x561a6b237090 2024-03-24T00:32:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ec5600640 10 client.5493 handle_command_reply: tid=0 2024-03-24T00:32:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 client.5493 shutdown 2024-03-24T00:32:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 -- 172.21.15.40:0/3859985447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b250920 msgr2=0x561a6b236280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 --2- 172.21.15.40:0/3859985447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b250920 0x561a6b236280 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00ef7d0 tx=0x7f3ec0000bd0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 -- 172.21.15.40:0/3859985447 shutdown_connections 2024-03-24T00:32:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 --2- 172.21.15.40:0/3859985447 >> [v2:172.21.15.40:6832/2326157501,v1:172.21.15.40:6833/2326157501] conn(0x561a6b237090 0x561a6b262ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 --2- 172.21.15.40:0/3859985447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b251240 0x561a6b2367c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 --2- 172.21.15.40:0/3859985447 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b250920 0x561a6b236280 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 --2- 172.21.15.40:0/3859985447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b235d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 -- 172.21.15.40:0/3859985447 >> 172.21.15.40:0/3859985447 conn(0x561a6b202fa0 msgr2=0x561a6b205c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ec5600640 0 client.5493 ms_handle_reset on v2:172.21.15.40:6832/2326157501 2024-03-24T00:32:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 -- 172.21.15.40:0/3859985447 shutdown_connections 2024-03-24T00:32:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.260+0000 7f3ee2442000 1 -- 172.21.15.40:0/3859985447 wait complete. 2024-03-24T00:32:21.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.264+0000 7f3ee2442000 20 client.5493 trim_cache size 0 max 16384 2024-03-24T00:32:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 Processor -- start 2024-03-24T00:32:21.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 -- start start 2024-03-24T00:32:21.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b234900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b250920 0x561a6b234e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b251240 0x561a6b2b2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed81fe550 con 0x561a6b251240 2024-03-24T00:32:21.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed809d1f0 con 0x561a6b206ff0 2024-03-24T00:32:21.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a6affca80 con 0x561a6b250920 2024-03-24T00:32:21.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ed7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b234900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ecfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b250920 0x561a6b234e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ed7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b234900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:59444/0 (socket says 172.21.15.40:59444) 2024-03-24T00:32:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ed7e00640 1 -- 172.21.15.40:0/2751593319 learned_addr learned my addr 172.21.15.40:0/2751593319 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3edd000640 1 --2- 172.21.15.40:0/2751593319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b251240 0x561a6b2b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ed7e00640 1 -- 172.21.15.40:0/2751593319 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b250920 msgr2=0x561a6b234e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ed7e00640 1 --2- 172.21.15.40:0/2751593319 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b250920 0x561a6b234e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ed7e00640 1 -- 172.21.15.40:0/2751593319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b251240 msgr2=0x561a6b2b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ed7e00640 1 --2- 172.21.15.40:0/2751593319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b251240 0x561a6b2b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ed7e00640 1 -- 172.21.15.40:0/2751593319 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a6b021540 con 0x561a6b206ff0 2024-03-24T00:32:21.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3edd000640 1 --2- 172.21.15.40:0/2751593319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b251240 0x561a6b2b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:21.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ed7e00640 1 --2- 172.21.15.40:0/2751593319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b234900 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80011f0 tx=0x7f3ec8006040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8570f7736149b4a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ec7e00640 1 -- 172.21.15.40:0/2751593319 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec801d1d0 con 0x561a6b206ff0 2024-03-24T00:32:21.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ec7e00640 1 -- 172.21.15.40:0/2751593319 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec801dcf0 con 0x561a6b206ff0 2024-03-24T00:32:21.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ec7e00640 1 -- 172.21.15.40:0/2751593319 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec8016640 con 0x561a6b206ff0 2024-03-24T00:32:21.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 -- 172.21.15.40:0/2751593319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 msgr2=0x561a6b234900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 --2- 172.21.15.40:0/2751593319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b234900 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80011f0 tx=0x7f3ec8006040 comp rx=0 tx=0).stop 2024-03-24T00:32:21.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 -- 172.21.15.40:0/2751593319 shutdown_connections 2024-03-24T00:32:21.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 --2- 172.21.15.40:0/2751593319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b251240 0x561a6b2b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 --2- 172.21.15.40:0/2751593319 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b250920 0x561a6b234e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 --2- 172.21.15.40:0/2751593319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b206ff0 0x561a6b234900 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 -- 172.21.15.40:0/2751593319 >> 172.21.15.40:0/2751593319 conn(0x561a6b202fa0 msgr2=0x561a6b25df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.292+0000 7f3ee2442000 1 -- 172.21.15.40:0/2751593319 shutdown_connections 2024-03-24T00:32:21.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 1 -- 172.21.15.40:0/2751593319 wait complete. 2024-03-24T00:32:21.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 1 -- start start 2024-03-24T00:32:21.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 1 Processor -- start 2024-03-24T00:32:21.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b206ff0 0x561a6b263ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b250920 0x561a6b264430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b251240 0x561a6b264970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a6afec750 con 0x561a6b206ff0 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a6b111700 con 0x561a6b251240 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561a6b053990 con 0x561a6b250920 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 10 client.0 _ll_register_callbacks cb 0x7f3ec80070e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ed7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b206ff0 0x561a6b263ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ed7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b206ff0 0x561a6b263ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46376/0 (socket says 172.21.15.40:46376) 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ed7e00640 1 -- 172.21.15.40:0/184425632 learned_addr learned my addr 172.21.15.40:0/184425632 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b250920 0x561a6b264430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3edd000640 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b251240 0x561a6b264970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ed7e00640 1 -- 172.21.15.40:0/184425632 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b250920 msgr2=0x561a6b264430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ed7e00640 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b250920 0x561a6b264430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ed7e00640 1 -- 172.21.15.40:0/184425632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b251240 msgr2=0x561a6b264970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ed7e00640 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b251240 0x561a6b264970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ed7e00640 1 -- 172.21.15.40:0/184425632 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a6afe84b0 con 0x561a6b206ff0 2024-03-24T00:32:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ed7e00640 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b206ff0 0x561a6b263ef0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8007830 tx=0x7f3ec801d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92fb1911fa6bd783 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ec5600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:32:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ec5600640 1 -- 172.21.15.40:0/184425632 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec802c060 con 0x561a6b206ff0 2024-03-24T00:32:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ec5600640 1 -- 172.21.15.40:0/184425632 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ec8016830 con 0x561a6b206ff0 2024-03-24T00:32:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 1 -- 172.21.15.40:0/184425632 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ec801dcf0 con 0x561a6b206ff0 2024-03-24T00:32:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ec5600640 1 -- 172.21.15.40:0/184425632 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ec801a070 con 0x561a6b206ff0 2024-03-24T00:32:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 10 client.5499 fetch_fsmap learned FSMap version 21 2024-03-24T00:32:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ee2442000 1 -- 172.21.15.40:0/184425632 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x561a6affb2f0 con 0x561a6b206ff0 2024-03-24T00:32:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ec5600640 1 -- 172.21.15.40:0/184425632 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec801dcf0 con 0x561a6b206ff0 2024-03-24T00:32:21.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.296+0000 7f3ec5600640 1 -- 172.21.15.40:0/184425632 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 21) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f3ec8035080 con 0x561a6b206ff0 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 10 client.5499 fetch_fsmap finished waiting for FSMap version 21 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 10 client.5499 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 20 client.5499 populate_metadata read hostname 'smithi040' 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.40:6832/2326157501,v1:172.21.15.40:6833/2326157501] conn(0x561a6b2bc1e0 0x561a6b2be5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 4 client.5499 mds_command: new command op to 5226 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 -- 172.21.15.40:0/184425632 --> [v2:172.21.15.40:6832/2326157501,v1:172.21.15.40:6833/2326157501] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561a6b061bc0 con 0x561a6b2bc1e0 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.40:6832/2326157501,v1:172.21.15.40:6833/2326157501] conn(0x561a6b2bc1e0 0x561a6b2be5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ecfe00640 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.40:6832/2326157501,v1:172.21.15.40:6833/2326157501] conn(0x561a6b2bc1e0 0x561a6b2be5d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d58268757329bdf6 server_cookie=68e07176aaf22393 in_seq=0 out_seq=0 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ec5600640 10 client.5499 ms_handle_connect on v2:172.21.15.40:6832/2326157501 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ec5600640 1 -- 172.21.15.40:0/184425632 <== mds.1 v2:172.21.15.40:6832/2326157501 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x561a6b061bc0 con 0x561a6b2bc1e0 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ec5600640 10 client.5499 handle_command_reply: tid=0 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 client.5499 shutdown 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 -- 172.21.15.40:0/184425632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b206ff0 msgr2=0x561a6b263ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b206ff0 0x561a6b263ef0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8007830 tx=0x7f3ec801d4b0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 -- 172.21.15.40:0/184425632 shutdown_connections 2024-03-24T00:32:21.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.40:6832/2326157501,v1:172.21.15.40:6833/2326157501] conn(0x561a6b2bc1e0 0x561a6b2be5d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561a6b251240 0x561a6b264970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561a6b250920 0x561a6b264430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 --2- 172.21.15.40:0/184425632 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561a6b206ff0 0x561a6b263ef0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 -- 172.21.15.40:0/184425632 >> 172.21.15.40:0/184425632 conn(0x561a6b202fa0 msgr2=0x561a6b25d680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ec5600640 0 client.5499 ms_handle_reset on v2:172.21.15.40:6832/2326157501 2024-03-24T00:32:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 -- 172.21.15.40:0/184425632 shutdown_connections 2024-03-24T00:32:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 1 -- 172.21.15.40:0/184425632 wait complete. 2024-03-24T00:32:21.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3ee2442000 20 client.5499 trim_cache size 0 max 16384 2024-03-24T00:32:21.304 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:32:21.304 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:32:21.304 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-03-24T00:32:21.304 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5226, 2024-03-24T00:32:21.305 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:32:21.305 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:32:21.305 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:32:21.305 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.020645611, 2024-03-24T00:32:21.305 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 21, 2024-03-24T00:32:21.305 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 33, 2024-03-24T00:32:21.305 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 33, 2024-03-24T00:32:21.305 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 22.535580914000001 2024-03-24T00:32:21.305 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:32:21.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3edf000640 1 -- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3eb4067660 msgr2=0x7f3eb4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3edf000640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3eb4067660 0x7f3eb4069b20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000dfd0 tx=0x7f3ed000f040 comp rx=0 tx=0).stop 2024-03-24T00:32:21.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3edf000640 1 -- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ed806de50 msgr2=0x7f3ed80b08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3edf000640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ed806de50 0x7f3ed80b08d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3ec00ab000 tx=0x7f3ec0019240 comp rx=0 tx=0).stop 2024-03-24T00:32:21.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3edf000640 1 -- 172.21.15.40:0/3215537555 shutdown_connections 2024-03-24T00:32:21.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3edf000640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3eb4067660 0x7f3eb4069b20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3edf000640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ed8156320 0x7f3ed809ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3edf000640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ed806de50 0x7f3ed80b08d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3edf000640 1 --2- 172.21.15.40:0/3215537555 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ed8066e80 0x7f3ed80d0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3edf000640 1 -- 172.21.15.40:0/3215537555 >> 172.21.15.40:0/3215537555 conn(0x7f3ed806b150 msgr2=0x7f3ed815c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3edf000640 1 -- 172.21.15.40:0/3215537555 shutdown_connections 2024-03-24T00:32:21.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.300+0000 7f3edf000640 1 -- 172.21.15.40:0/3215537555 wait complete. 2024-03-24T00:32:21.329 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:32:21.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.468+0000 7fddf4000640 1 Processor -- start 2024-03-24T00:32:21.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.468+0000 7fddf4000640 1 -- start start 2024-03-24T00:32:21.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.472+0000 7fddf4000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fddec13ec80 0x7fddec1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.472+0000 7fddf4000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddec070e00 con 0x7fddec1419f0 2024-03-24T00:32:21.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.472+0000 7fddf4000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddec0cee20 con 0x7fddec0ce800 2024-03-24T00:32:21.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.472+0000 7fddf4000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddec141e50 con 0x7fddec13ec80 2024-03-24T00:32:21.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.472+0000 7fddf0c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fddec1419f0 0x7fddec0ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54122/0 (socket says 172.21.15.40:54122) 2024-03-24T00:32:21.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.472+0000 7fddf0c00640 1 -- 172.21.15.40:0/1988789952 learned_addr learned my addr 172.21.15.40:0/1988789952 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:21.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.472+0000 7fddf1600640 1 --2- 172.21.15.40:0/1988789952 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fddec13ec80 0x7fddec1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.472+0000 7fddebe00640 1 -- 172.21.15.40:0/1988789952 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388397713 0 0) 0x7fddec070e00 con 0x7fddec1419f0 2024-03-24T00:32:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.472+0000 7fddf1600640 1 -- 172.21.15.40:0/1988789952 >> v1:172.21.15.53:6789/0 conn(0x7fddec0ce800 legacy=0x7fddec071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.472+0000 7fddf1600640 1 -- 172.21.15.40:0/1988789952 >> v1:172.21.15.40:6789/0 conn(0x7fddec1419f0 legacy=0x7fddec0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.472+0000 7fddf1600640 1 -- 172.21.15.40:0/1988789952 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec0cf100 con 0x7fddec13ec80 2024-03-24T00:32:21.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.476+0000 7fddf1600640 1 --2- 172.21.15.40:0/1988789952 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fddec13ec80 0x7fddec1414b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009770 tx=0x7fdde001c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3b3dca7dbeb3a9e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.476+0000 7fddebe00640 1 -- 172.21.15.40:0/1988789952 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00b0070 con 0x7fddec13ec80 2024-03-24T00:32:21.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.476+0000 7fddebe00640 1 -- 172.21.15.40:0/1988789952 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde00ab600 con 0x7fddec13ec80 2024-03-24T00:32:21.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.476+0000 7fddebe00640 1 -- 172.21.15.40:0/1988789952 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00ab9d0 con 0x7fddec13ec80 2024-03-24T00:32:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.476+0000 7fddf4000640 1 -- 172.21.15.40:0/1988789952 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fddec13ec80 msgr2=0x7fddec1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.476+0000 7fddf4000640 1 --2- 172.21.15.40:0/1988789952 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fddec13ec80 0x7fddec1414b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdde0009770 tx=0x7fdde001c310 comp rx=0 tx=0).stop 2024-03-24T00:32:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.476+0000 7fddf4000640 1 -- 172.21.15.40:0/1988789952 shutdown_connections 2024-03-24T00:32:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.476+0000 7fddf4000640 1 --2- 172.21.15.40:0/1988789952 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fddec13ec80 0x7fddec1414b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.476+0000 7fddf4000640 1 -- 172.21.15.40:0/1988789952 >> 172.21.15.40:0/1988789952 conn(0x7fddec0b17b0 msgr2=0x7fddec0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.476+0000 7fddf4000640 1 -- 172.21.15.40:0/1988789952 shutdown_connections 2024-03-24T00:32:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.476+0000 7fddf4000640 1 -- 172.21.15.40:0/1988789952 wait complete. 2024-03-24T00:32:21.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf4000640 1 Processor -- start 2024-03-24T00:32:21.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf4000640 1 -- start start 2024-03-24T00:32:21.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf4000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddec0ce800 0x7fddec1592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf4000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fddec13ec80 0x7fddec1fb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf4000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddec1419f0 0x7fddec1e5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf4000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec071a10 con 0x7fddec0ce800 2024-03-24T00:32:21.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf4000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec13f060 con 0x7fddec1419f0 2024-03-24T00:32:21.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf4000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddec071b90 con 0x7fddec13ec80 2024-03-24T00:32:21.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddec0ce800 0x7fddec1592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddec0ce800 0x7fddec1592b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46388/0 (socket says 172.21.15.40:46388) 2024-03-24T00:32:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf1600640 1 -- 172.21.15.40:0/1891353617 learned_addr learned my addr 172.21.15.40:0/1891353617 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf0c00640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fddec13ec80 0x7fddec1fb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf2000640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddec1419f0 0x7fddec1e5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf1600640 1 -- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fddec13ec80 msgr2=0x7fddec1fb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf1600640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fddec13ec80 0x7fddec1fb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf1600640 1 -- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddec1419f0 msgr2=0x7fddec1e5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf1600640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddec1419f0 0x7fddec1e5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf1600640 1 -- 172.21.15.40:0/1891353617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec13ded0 con 0x7fddec0ce800 2024-03-24T00:32:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf1600640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddec0ce800 0x7fddec1592b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdde00ae040 tx=0x7fdde001ab50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fdde9600640 1 -- 172.21.15.40:0/1891353617 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00b0050 con 0x7fddec0ce800 2024-03-24T00:32:21.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fdde9600640 1 -- 172.21.15.40:0/1891353617 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde00bc070 con 0x7fddec0ce800 2024-03-24T00:32:21.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fdde9600640 1 -- 172.21.15.40:0/1891353617 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00aae40 con 0x7fddec0ce800 2024-03-24T00:32:21.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf4000640 1 -- 172.21.15.40:0/1891353617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddec1598c0 con 0x7fddec0ce800 2024-03-24T00:32:21.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.480+0000 7fddf4000640 1 -- 172.21.15.40:0/1891353617 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddec156d30 con 0x7fddec0ce800 2024-03-24T00:32:21.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.484+0000 7fdde9600640 1 -- 172.21.15.40:0/1891353617 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fdde00ab4c0 con 0x7fddec0ce800 2024-03-24T00:32:21.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.484+0000 7fddf758a000 1 Processor -- start 2024-03-24T00:32:21.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.484+0000 7fddf758a000 1 -- start start 2024-03-24T00:32:21.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.484+0000 7fdde9600640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fddcc067660 0x7fddcc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.484+0000 7fdde9600640 1 -- 172.21.15.40:0/1891353617 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdde01204b0 con 0x7fddec0ce800 2024-03-24T00:32:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf2000640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fddcc067660 0x7fddcc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d4bb920 0x559d0d4bbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d4bc240 0x559d0d471ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d471ff0 0x559d0d4723d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0d2bf680 con 0x559d0d471ff0 2024-03-24T00:32:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0d246b00 con 0x559d0d4bb920 2024-03-24T00:32:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d0d2bb720 con 0x559d0d4bc240 2024-03-24T00:32:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf1600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d4bb920 0x559d0d4bbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d471ff0 0x559d0d4723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf0c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d4bc240 0x559d0d471ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf1600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d4bb920 0x559d0d4bbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:59456/0 (socket says 172.21.15.40:59456) 2024-03-24T00:32:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf1600640 1 -- 172.21.15.40:0/3857306340 learned_addr learned my addr 172.21.15.40:0/3857306340 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf2000640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fddcc067660 0x7fddcc069b20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009fd0 tx=0x7fdde4009dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf1600640 1 -- 172.21.15.40:0/3857306340 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d4bc240 msgr2=0x559d0d471ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf1600640 1 --2- 172.21.15.40:0/3857306340 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d4bc240 0x559d0d471ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf1600640 1 -- 172.21.15.40:0/3857306340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d471ff0 msgr2=0x559d0d4723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf1600640 1 --2- 172.21.15.40:0/3857306340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d471ff0 0x559d0d4723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf1600640 1 -- 172.21.15.40:0/3857306340 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d0d297940 con 0x559d0d4bb920 2024-03-24T00:32:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf1600640 1 --2- 172.21.15.40:0/3857306340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d4bb920 0x559d0d4bbd00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdde00f09e0 tx=0x7fdde00190b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e7af8101c7e4251 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddde000640 1 -- 172.21.15.40:0/3857306340 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00f4af0 con 0x559d0d4bb920 2024-03-24T00:32:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddde000640 1 -- 172.21.15.40:0/3857306340 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde00b1040 con 0x559d0d4bb920 2024-03-24T00:32:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddde000640 1 -- 172.21.15.40:0/3857306340 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00f90b0 con 0x559d0d4bb920 2024-03-24T00:32:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 -- 172.21.15.40:0/3857306340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d4bb920 msgr2=0x559d0d4bbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 --2- 172.21.15.40:0/3857306340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d4bb920 0x559d0d4bbd00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdde00f09e0 tx=0x7fdde00190b0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 -- 172.21.15.40:0/3857306340 shutdown_connections 2024-03-24T00:32:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 --2- 172.21.15.40:0/3857306340 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d471ff0 0x559d0d4723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 --2- 172.21.15.40:0/3857306340 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d4bc240 0x559d0d471ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 --2- 172.21.15.40:0/3857306340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d4bb920 0x559d0d4bbd00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 -- 172.21.15.40:0/3857306340 >> 172.21.15.40:0/3857306340 conn(0x559d0d46dfa0 msgr2=0x559d0d470390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 -- 172.21.15.40:0/3857306340 shutdown_connections 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 -- 172.21.15.40:0/3857306340 wait complete. 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 -- start start 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.488+0000 7fddf758a000 1 Processor -- start 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 0x559d0d4a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d4bb920 0x559d0d4a1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bc240 0x559d0d4a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0d2be990 con 0x559d0d4bc240 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0d279ae0 con 0x559d0d471ff0 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d0d257750 con 0x559d0d4bb920 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 10 client.0 _ll_register_callbacks cb 0x559d0d45cff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bc240 0x559d0d4a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bc240 0x559d0d4a17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46396/0 (socket says 172.21.15.40:46396) 2024-03-24T00:32:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf2000640 1 -- 172.21.15.40:0/1037613806 learned_addr learned my addr 172.21.15.40:0/1037613806 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf0c00640 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d4bb920 0x559d0d4a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf1600640 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 0x559d0d4a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf2000640 1 -- 172.21.15.40:0/1037613806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d4bb920 msgr2=0x559d0d4a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf2000640 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d4bb920 0x559d0d4a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf2000640 1 -- 172.21.15.40:0/1037613806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 msgr2=0x559d0d4a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf2000640 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 0x559d0d4a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf2000640 1 -- 172.21.15.40:0/1037613806 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d0d304d80 con 0x559d0d4bc240 2024-03-24T00:32:21.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf2000640 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bc240 0x559d0d4a17c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fdde40174e0 tx=0x7fdde4001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a46358d65f0fce server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddd2a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddd2a00640 1 -- 172.21.15.40:0/1037613806 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4027070 con 0x559d0d4bc240 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 1 -- 172.21.15.40:0/1037613806 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdde00b1040 con 0x559d0d4bc240 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddd2a00640 1 -- 172.21.15.40:0/1037613806 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdde4003980 con 0x559d0d4bc240 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddd2a00640 1 -- 172.21.15.40:0/1037613806 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdde4031050 con 0x559d0d4bc240 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 10 client.5511 fetch_fsmap learned FSMap version 21 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 1 -- 172.21.15.40:0/1037613806 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x559d0d246d10 con 0x559d0d4bc240 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddd2a00640 1 -- 172.21.15.40:0/1037613806 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde00b1040 con 0x559d0d4bc240 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddd2a00640 1 -- 172.21.15.40:0/1037613806 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 21) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fdde4034020 con 0x559d0d4bc240 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 10 client.5511 fetch_fsmap finished waiting for FSMap version 21 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 10 client.5511 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 20 client.5511 populate_metadata read hostname 'smithi040' 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x559d0d4a2090 0x559d0d4cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 4 client.5511 mds_command: new command op to 5229 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf758a000 1 -- 172.21.15.40:0/1037613806 --> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d0d25ab60 con 0x559d0d4a2090 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf1600640 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x559d0d4a2090 0x559d0d4cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddf1600640 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x559d0d4a2090 0x559d0d4cdea0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e5e4a56c751218 server_cookie=e98f28b8c53bf08a in_seq=0 out_seq=0 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.492+0000 7fddd2a00640 10 client.5511 ms_handle_connect on v2:172.21.15.40:6834/3050269371 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddd2a00640 1 -- 172.21.15.40:0/1037613806 <== mds.0 v2:172.21.15.40:6834/3050269371 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559d0d25ab60 con 0x559d0d4a2090 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddd2a00640 10 client.5511 handle_command_reply: tid=0 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddf758a000 1 client.5511 shutdown 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddf758a000 1 -- 172.21.15.40:0/1037613806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bc240 msgr2=0x559d0d4a17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddf758a000 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bc240 0x559d0d4a17c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fdde40174e0 tx=0x7fdde4001260 comp rx=0 tx=0).stop 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddf758a000 1 -- 172.21.15.40:0/1037613806 shutdown_connections 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddf758a000 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x559d0d4a2090 0x559d0d4cdea0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddf758a000 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bc240 0x559d0d4a17c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddf758a000 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d4bb920 0x559d0d4a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddf758a000 1 --2- 172.21.15.40:0/1037613806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 0x559d0d4a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddf758a000 1 -- 172.21.15.40:0/1037613806 >> 172.21.15.40:0/1037613806 conn(0x559d0d46dfa0 msgr2=0x559d0d470c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddd2a00640 0 client.5511 ms_handle_reset on v2:172.21.15.40:6834/3050269371 2024-03-24T00:32:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddf758a000 1 -- 172.21.15.40:0/1037613806 shutdown_connections 2024-03-24T00:32:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddf758a000 1 -- 172.21.15.40:0/1037613806 wait complete. 2024-03-24T00:32:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.496+0000 7fddf758a000 20 client.5511 trim_cache size 0 max 16384 2024-03-24T00:32:21.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 Processor -- start 2024-03-24T00:32:21.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 -- start start 2024-03-24T00:32:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d46dfa0 0x559d0d4719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 0x559d0d49f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 0x559d0d49fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec2044f0 con 0x559d0d4bb920 2024-03-24T00:32:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec203890 con 0x559d0d471ff0 2024-03-24T00:32:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddec203e90 con 0x559d0d46dfa0 2024-03-24T00:32:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 0x559d0d49fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf1600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d46dfa0 0x559d0d4719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 0x559d0d49fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46400/0 (socket says 172.21.15.40:46400) 2024-03-24T00:32:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf2000640 1 -- 172.21.15.40:0/3571292735 learned_addr learned my addr 172.21.15.40:0/3571292735 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf0c00640 1 --2- 172.21.15.40:0/3571292735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 0x559d0d49f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf1600640 1 -- 172.21.15.40:0/3571292735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 msgr2=0x559d0d49f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf1600640 1 --2- 172.21.15.40:0/3571292735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 0x559d0d49f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf1600640 1 -- 172.21.15.40:0/3571292735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 msgr2=0x559d0d49fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf1600640 1 --2- 172.21.15.40:0/3571292735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 0x559d0d49fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf1600640 1 -- 172.21.15.40:0/3571292735 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d0d4761d0 con 0x559d0d46dfa0 2024-03-24T00:32:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf2000640 1 --2- 172.21.15.40:0/3571292735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 0x559d0d49fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf1600640 1 --2- 172.21.15.40:0/3571292735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d46dfa0 0x559d0d4719d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdde00f0750 tx=0x7fdde00c2000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cfd4a3abe3467ea server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fdddd600640 1 -- 172.21.15.40:0/3571292735 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde00f90b0 con 0x559d0d46dfa0 2024-03-24T00:32:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fdddd600640 1 -- 172.21.15.40:0/3571292735 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde00eed50 con 0x559d0d46dfa0 2024-03-24T00:32:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fdddd600640 1 -- 172.21.15.40:0/3571292735 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde01200b0 con 0x559d0d46dfa0 2024-03-24T00:32:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 -- 172.21.15.40:0/3571292735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d46dfa0 msgr2=0x559d0d4719d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 --2- 172.21.15.40:0/3571292735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d46dfa0 0x559d0d4719d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdde00f0750 tx=0x7fdde00c2000 comp rx=0 tx=0).stop 2024-03-24T00:32:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 -- 172.21.15.40:0/3571292735 shutdown_connections 2024-03-24T00:32:21.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 --2- 172.21.15.40:0/3571292735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 0x559d0d49fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 --2- 172.21.15.40:0/3571292735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 0x559d0d49f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 --2- 172.21.15.40:0/3571292735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d46dfa0 0x559d0d4719d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.512+0000 7fddf758a000 1 -- 172.21.15.40:0/3571292735 >> 172.21.15.40:0/3571292735 conn(0x559d0d4a0ed0 msgr2=0x559d0d4c8f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 1 -- 172.21.15.40:0/3571292735 shutdown_connections 2024-03-24T00:32:21.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 1 -- 172.21.15.40:0/3571292735 wait complete. 2024-03-24T00:32:21.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 1 -- start start 2024-03-24T00:32:21.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 1 Processor -- start 2024-03-24T00:32:21.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d46dfa0 0x559d0d5211b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 0x559d0d523700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 0x559d0d525c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0d267a80 con 0x559d0d4bb920 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddec1fc950 con 0x559d0d471ff0 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d0d37c700 con 0x559d0d46dfa0 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 10 client.0 _ll_register_callbacks cb 0x7fdde4016530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 0x559d0d525c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf1600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d46dfa0 0x559d0d5211b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 0x559d0d525c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46414/0 (socket says 172.21.15.40:46414) 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 0x559d0d523700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf2000640 1 -- 172.21.15.40:0/796282372 learned_addr learned my addr 172.21.15.40:0/796282372 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf2000640 1 -- 172.21.15.40:0/796282372 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d46dfa0 msgr2=0x559d0d5211b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf2000640 1 --2- 172.21.15.40:0/796282372 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d46dfa0 0x559d0d5211b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf2000640 1 -- 172.21.15.40:0/796282372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 msgr2=0x559d0d523700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf2000640 1 --2- 172.21.15.40:0/796282372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 0x559d0d523700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf2000640 1 -- 172.21.15.40:0/796282372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d0d28c540 con 0x559d0d4bb920 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf2000640 1 --2- 172.21.15.40:0/796282372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 0x559d0d525c50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdde4008cc0 tx=0x7fdde4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2718680ecaf7ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddd2a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddd2a00640 1 -- 172.21.15.40:0/796282372 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4027050 con 0x559d0d4bb920 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddd2a00640 1 -- 172.21.15.40:0/796282372 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdde4015b50 con 0x559d0d4bb920 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 1 -- 172.21.15.40:0/796282372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdde00eed50 con 0x559d0d4bb920 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddd2a00640 1 -- 172.21.15.40:0/796282372 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdde4044050 con 0x559d0d4bb920 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 10 client.5517 fetch_fsmap learned FSMap version 21 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddf758a000 1 -- 172.21.15.40:0/796282372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=21,monmap=2+}) v3 -- 0x559d0d2534b0 con 0x559d0d4bb920 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.516+0000 7fddd2a00640 1 -- 172.21.15.40:0/796282372 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde00eed50 con 0x559d0d4bb920 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddd2a00640 1 -- 172.21.15.40:0/796282372 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 21) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fdde4047020 con 0x559d0d4bb920 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 10 client.5517 fetch_fsmap finished waiting for FSMap version 21 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 10 client.5517 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 20 client.5517 populate_metadata read hostname 'smithi040' 2024-03-24T00:32:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 --2- 172.21.15.40:0/796282372 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x559d0d4bc240 0x559d0d5281a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:21.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 4 client.5517 mds_command: new command op to 5229 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:32:21.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 -- 172.21.15.40:0/796282372 --> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559d0d2ccbc0 con 0x559d0d4bc240 2024-03-24T00:32:21.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf1600640 1 --2- 172.21.15.40:0/796282372 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x559d0d4bc240 0x559d0d5281a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:21.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddd2a00640 10 client.5517 ms_handle_connect on v2:172.21.15.40:6834/3050269371 2024-03-24T00:32:21.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf1600640 1 --2- 172.21.15.40:0/796282372 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x559d0d4bc240 0x559d0d5281a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71dbc337de6dd339 server_cookie=ba05b5a8b19ec9db in_seq=0 out_seq=0 2024-03-24T00:32:21.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddd2a00640 1 -- 172.21.15.40:0/796282372 <== mds.0 v2:172.21.15.40:6834/3050269371 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x559d0d2ccbc0 con 0x559d0d4bc240 2024-03-24T00:32:21.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddd2a00640 10 client.5517 handle_command_reply: tid=0 2024-03-24T00:32:21.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 client.5517 shutdown 2024-03-24T00:32:21.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 -- 172.21.15.40:0/796282372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 msgr2=0x559d0d525c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 --2- 172.21.15.40:0/796282372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 0x559d0d525c50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fdde4008cc0 tx=0x7fdde4017000 comp rx=0 tx=0).stop 2024-03-24T00:32:21.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 -- 172.21.15.40:0/796282372 shutdown_connections 2024-03-24T00:32:21.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 --2- 172.21.15.40:0/796282372 >> [v2:172.21.15.40:6834/3050269371,v1:172.21.15.40:6835/3050269371] conn(0x559d0d4bc240 0x559d0d5281a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 --2- 172.21.15.40:0/796282372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x559d0d4bb920 0x559d0d525c50 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 --2- 172.21.15.40:0/796282372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x559d0d471ff0 0x559d0d523700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 --2- 172.21.15.40:0/796282372 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x559d0d46dfa0 0x559d0d5211b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 -- 172.21.15.40:0/796282372 >> 172.21.15.40:0/796282372 conn(0x559d0d4a0ed0 msgr2=0x559d0d4c8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddd2a00640 0 client.5517 ms_handle_reset on v2:172.21.15.40:6834/3050269371 2024-03-24T00:32:21.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 -- 172.21.15.40:0/796282372 shutdown_connections 2024-03-24T00:32:21.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 1 -- 172.21.15.40:0/796282372 wait complete. 2024-03-24T00:32:21.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf758a000 20 client.5517 trim_cache size 0 max 16384 2024-03-24T00:32:21.525 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:32:21.525 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:32:21.525 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:32:21.525 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5229, 2024-03-24T00:32:21.526 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:32:21.526 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:32:21.526 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:32:21.526 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 14.879063724, 2024-03-24T00:32:21.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 21, 2024-03-24T00:32:21.526 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 33, 2024-03-24T00:32:21.526 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 33, 2024-03-24T00:32:21.526 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 22.752303424000001 2024-03-24T00:32:21.526 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:32:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf4000640 1 -- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fddcc067660 msgr2=0x7fddcc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf4000640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fddcc067660 0x7fddcc069b20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009fd0 tx=0x7fdde4009dc0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf4000640 1 -- 172.21.15.40:0/1891353617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddec0ce800 msgr2=0x7fddec1592b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf4000640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddec0ce800 0x7fddec1592b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fdde00ae040 tx=0x7fdde001ab50 comp rx=0 tx=0).stop 2024-03-24T00:32:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf4000640 1 -- 172.21.15.40:0/1891353617 shutdown_connections 2024-03-24T00:32:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf4000640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fddcc067660 0x7fddcc069b20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf4000640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddec1419f0 0x7fddec1e5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf4000640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fddec13ec80 0x7fddec1fb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf4000640 1 --2- 172.21.15.40:0/1891353617 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fddec0ce800 0x7fddec1592b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf4000640 1 -- 172.21.15.40:0/1891353617 >> 172.21.15.40:0/1891353617 conn(0x7fddec0b17b0 msgr2=0x7fddec0b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf4000640 1 -- 172.21.15.40:0/1891353617 shutdown_connections 2024-03-24T00:32:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:21.520+0000 7fddf4000640 1 -- 172.21.15.40:0/1891353617 wait complete. 2024-03-24T00:32:21.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:21.550 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:32:21.550 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:32:21.551 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p tree 2024-03-24T00:32:21.551 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:32:21.551 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:32:21.551 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:32:21.551 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < 100; i++)); do 2024-03-24T00:32:21.551 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p "tree/$i" 2024-03-24T00:32:21.551 DEBUG:teuthology.orchestra.run.smithi040:> echo file > "tree/$i/file" 2024-03-24T00:32:21.551 DEBUG:teuthology.orchestra.run.smithi040:> done 2024-03-24T00:32:21.551 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:32:21.561 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree 2024-03-24T00:32:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i = 0 )) 2024-03-24T00:32:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/0 2024-03-24T00:32:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/1 2024-03-24T00:32:21.592 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.595 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.595 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.595 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/2 2024-03-24T00:32:21.602 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.605 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.605 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.605 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/3 2024-03-24T00:32:21.612 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/4 2024-03-24T00:32:21.622 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.625 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.625 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.625 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/5 2024-03-24T00:32:21.632 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.635 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.635 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.635 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/6 2024-03-24T00:32:21.641 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.644 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/7 2024-03-24T00:32:21.652 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.656 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.656 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.656 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/8 2024-03-24T00:32:21.663 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.666 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.666 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.666 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/9 2024-03-24T00:32:21.673 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.677 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.677 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.677 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/10 2024-03-24T00:32:21.684 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.687 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.687 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.687 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/11 2024-03-24T00:32:21.694 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.697 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.697 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.697 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/12 2024-03-24T00:32:21.704 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.706 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.706 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.707 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/13 2024-03-24T00:32:21.713 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.715 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.716 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.716 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/14 2024-03-24T00:32:21.723 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.725 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.726 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.726 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/15 2024-03-24T00:32:21.733 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.736 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.736 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.736 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/16 2024-03-24T00:32:21.743 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.746 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.746 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.746 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/17 2024-03-24T00:32:21.753 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.756 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.756 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.756 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/18 2024-03-24T00:32:21.763 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.766 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.766 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.766 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/19 2024-03-24T00:32:21.773 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.776 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.776 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/20 2024-03-24T00:32:21.783 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/21 2024-03-24T00:32:21.791 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.794 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.794 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.794 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/22 2024-03-24T00:32:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.804 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/23 2024-03-24T00:32:21.810 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/24 2024-03-24T00:32:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/25 2024-03-24T00:32:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.830 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.830 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.830 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/26 2024-03-24T00:32:21.837 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.841 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.841 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.841 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/27 2024-03-24T00:32:21.848 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.851 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.851 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.851 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/28 2024-03-24T00:32:21.858 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/29 2024-03-24T00:32:21.866 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/30 2024-03-24T00:32:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/31 2024-03-24T00:32:21.885 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.889 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/32 2024-03-24T00:32:21.897 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.899 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.900 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.900 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/33 2024-03-24T00:32:21.907 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.909 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.910 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.910 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/34 2024-03-24T00:32:21.916 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.920 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/35 2024-03-24T00:32:21.927 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/36 2024-03-24T00:32:21.937 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.940 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.940 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.940 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/37 2024-03-24T00:32:21.947 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.950 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.950 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.950 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/38 2024-03-24T00:32:21.956 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.959 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.959 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.960 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/39 2024-03-24T00:32:21.967 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.970 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.970 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.970 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/40 2024-03-24T00:32:21.977 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/41 2024-03-24T00:32:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:21.990 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/42 2024-03-24T00:32:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/43 2024-03-24T00:32:22.007 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.010 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.010 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.010 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/44 2024-03-24T00:32:22.017 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/45 2024-03-24T00:32:22.027 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.030 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/46 2024-03-24T00:32:22.037 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.040 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.040 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.040 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/47 2024-03-24T00:32:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/48 2024-03-24T00:32:22.058 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.061 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/49 2024-03-24T00:32:22.069 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.071 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.071 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.071 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/50 2024-03-24T00:32:22.079 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.082 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.082 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.082 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/51 2024-03-24T00:32:22.088 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.091 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.091 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.091 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/52 2024-03-24T00:32:22.098 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.100 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.101 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.101 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/53 2024-03-24T00:32:22.108 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.110 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.111 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.111 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/54 2024-03-24T00:32:22.118 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.120 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.120 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.121 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/55 2024-03-24T00:32:22.127 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.130 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.130 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.130 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/56 2024-03-24T00:32:22.137 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.140 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.140 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.140 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/57 2024-03-24T00:32:22.147 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.151 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/58 2024-03-24T00:32:22.157 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.160 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.160 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.160 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/59 2024-03-24T00:32:22.166 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.170 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.170 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.170 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/60 2024-03-24T00:32:22.176 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.179 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.179 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.179 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/61 2024-03-24T00:32:22.185 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.187 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/62 2024-03-24T00:32:22.194 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.197 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/63 2024-03-24T00:32:22.205 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.208 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.208 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.208 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/64 2024-03-24T00:32:22.215 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.218 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.218 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.218 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/65 2024-03-24T00:32:22.224 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.227 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.227 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.227 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/66 2024-03-24T00:32:22.234 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/67 2024-03-24T00:32:22.243 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.245 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.245 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.246 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/68 2024-03-24T00:32:22.253 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.256 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.256 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.256 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/69 2024-03-24T00:32:22.262 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.265 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.265 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.265 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/70 2024-03-24T00:32:22.271 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.273 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.274 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.274 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/71 2024-03-24T00:32:22.280 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.283 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.283 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.283 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/72 2024-03-24T00:32:22.290 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.292 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.292 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.293 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/73 2024-03-24T00:32:22.300 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.303 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.303 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.303 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/74 2024-03-24T00:32:22.310 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.312 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.312 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.312 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/75 2024-03-24T00:32:22.319 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.321 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.321 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.321 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/76 2024-03-24T00:32:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.330 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.330 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.330 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/77 2024-03-24T00:32:22.338 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.341 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.341 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.341 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/78 2024-03-24T00:32:22.348 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.351 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/79 2024-03-24T00:32:22.357 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.360 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.360 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.360 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/80 2024-03-24T00:32:22.367 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.369 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.369 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.370 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/81 2024-03-24T00:32:22.376 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.379 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.379 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.379 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/82 2024-03-24T00:32:22.385 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.388 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.388 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.388 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/83 2024-03-24T00:32:22.395 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.397 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.398 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.398 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/84 2024-03-24T00:32:22.405 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.408 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.408 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.408 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/85 2024-03-24T00:32:22.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.418 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.418 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.418 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/86 2024-03-24T00:32:22.425 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.427 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.427 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.428 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/87 2024-03-24T00:32:22.434 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.438 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.438 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.438 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/88 2024-03-24T00:32:22.444 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.447 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/89 2024-03-24T00:32:22.453 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.456 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/90 2024-03-24T00:32:22.463 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.465 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.465 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.466 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/91 2024-03-24T00:32:22.472 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.474 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.474 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.474 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/92 2024-03-24T00:32:22.481 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.483 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.483 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.484 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/93 2024-03-24T00:32:22.491 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.493 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.493 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.493 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/94 2024-03-24T00:32:22.500 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.503 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.503 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.503 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/95 2024-03-24T00:32:22.510 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.513 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.513 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.513 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/96 2024-03-24T00:32:22.520 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.522 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.522 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.522 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/97 2024-03-24T00:32:22.529 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.532 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.532 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.532 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/98 2024-03-24T00:32:22.538 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.541 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.541 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.541 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/99 2024-03-24T00:32:22.548 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:32:22.550 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:32:22.550 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:32:22.552 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_distributed False 2024-03-24T00:32:22.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.740+0000 7f9c7c400640 1 Processor -- start 2024-03-24T00:32:22.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.740+0000 7f9c7c400640 1 -- start start 2024-03-24T00:32:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c7c400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9c7415ac60 0x7f9c740b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c7c400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9c74071b00 0x7f9c74071f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c7c400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c740b3260 con 0x7f9c7406cb80 2024-03-24T00:32:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c7c400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c74072490 con 0x7f9c74071b00 2024-03-24T00:32:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c7c400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c740b3e70 con 0x7f9c7415ac60 2024-03-24T00:32:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c7a400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9c74071b00 0x7f9c74071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c79a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9c7415ac60 0x7f9c740b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c79000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9c7406cb80 0x7f9c740b4300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:54132/0 (socket says 172.21.15.40:54132) 2024-03-24T00:32:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c79000640 1 -- 172.21.15.40:0/2055879316 learned_addr learned my addr 172.21.15.40:0/2055879316 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c7a400640 1 -- 172.21.15.40:0/2055879316 >> v2:172.21.15.53:3300/0 conn(0x7f9c7415ac60 msgr2=0x7f9c740b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c7a400640 1 --2- 172.21.15.40:0/2055879316 >> v2:172.21.15.53:3300/0 conn(0x7f9c7415ac60 0x7f9c740b2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:22.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c7a400640 1 -- 172.21.15.40:0/2055879316 >> v1:172.21.15.40:6789/0 conn(0x7f9c7406cb80 legacy=0x7f9c740b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:22.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.744+0000 7f9c7a400640 1 -- 172.21.15.40:0/2055879316 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c740b3540 con 0x7f9c74071b00 2024-03-24T00:32:22.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.748+0000 7f9c7a400640 1 --2- 172.21.15.40:0/2055879316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c74071b00 0x7f9c74071f50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9c7406ff40 tx=0x7f9c6c01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a682e1e51e8fc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:22.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.748+0000 7f9c73e00640 1 -- 172.21.15.40:0/2055879316 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c0b2070 con 0x7f9c74071b00 2024-03-24T00:32:22.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.748+0000 7f9c73e00640 1 -- 172.21.15.40:0/2055879316 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6c0a5930 con 0x7f9c74071b00 2024-03-24T00:32:22.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.748+0000 7f9c73e00640 1 -- 172.21.15.40:0/2055879316 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c0a5d00 con 0x7f9c74071b00 2024-03-24T00:32:22.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.748+0000 7f9c7c400640 1 -- 172.21.15.40:0/2055879316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c74071b00 msgr2=0x7f9c74071f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:22.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.748+0000 7f9c7c400640 1 --2- 172.21.15.40:0/2055879316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c74071b00 0x7f9c74071f50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f9c7406ff40 tx=0x7f9c6c01c870 comp rx=0 tx=0).stop 2024-03-24T00:32:22.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.748+0000 7f9c7c400640 1 -- 172.21.15.40:0/2055879316 shutdown_connections 2024-03-24T00:32:22.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.748+0000 7f9c7c400640 1 --2- 172.21.15.40:0/2055879316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c74071b00 0x7f9c74071f50 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:22.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.748+0000 7f9c7c400640 1 --2- 172.21.15.40:0/2055879316 >> v2:172.21.15.53:3300/0 conn(0x7f9c7415ac60 0x7f9c740b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:22.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.748+0000 7f9c7c400640 1 -- 172.21.15.40:0/2055879316 >> 172.21.15.40:0/2055879316 conn(0x7f9c740b1660 msgr2=0x7f9c740b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:22.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.748+0000 7f9c7c400640 1 -- 172.21.15.40:0/2055879316 shutdown_connections 2024-03-24T00:32:22.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.748+0000 7f9c7c400640 1 -- 172.21.15.40:0/2055879316 wait complete. 2024-03-24T00:32:22.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.752+0000 7f9c7c400640 1 Processor -- start 2024-03-24T00:32:22.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.752+0000 7f9c7c400640 1 -- start start 2024-03-24T00:32:22.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.752+0000 7f9c7c400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7406cb80 0x7f9c741393e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:22.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.752+0000 7f9c7c400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c74071b00 0x7f9c74156aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:22.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.752+0000 7f9c7c400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c7415ac60 0x7f9c740ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:22.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.752+0000 7f9c7c400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c740b3020 con 0x7f9c7406cb80 2024-03-24T00:32:22.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.752+0000 7f9c7c400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c740b2ea0 con 0x7f9c7415ac60 2024-03-24T00:32:22.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.752+0000 7f9c7c400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7415b040 con 0x7f9c74071b00 2024-03-24T00:32:22.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.752+0000 7f9c79a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7406cb80 0x7f9c741393e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:22.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.752+0000 7f9c79a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7406cb80 0x7f9c741393e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46424/0 (socket says 172.21.15.40:46424) 2024-03-24T00:32:22.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.752+0000 7f9c79a00640 1 -- 172.21.15.40:0/3492247997 learned_addr learned my addr 172.21.15.40:0/3492247997 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:22.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c79000640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c74071b00 0x7f9c74156aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:22.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c7a400640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c7415ac60 0x7f9c740ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:22.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c79a00640 1 -- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c74071b00 msgr2=0x7f9c74156aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:22.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c79a00640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c74071b00 0x7f9c74156aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:22.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c79a00640 1 -- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c7415ac60 msgr2=0x7f9c740ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:22.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c79a00640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c7415ac60 0x7f9c740ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:22.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c79a00640 1 -- 172.21.15.40:0/3492247997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c74150680 con 0x7f9c7406cb80 2024-03-24T00:32:22.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c79a00640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7406cb80 0x7f9c741393e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9c6800e930 tx=0x7f9c6800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:22.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c71600640 1 -- 172.21.15.40:0/3492247997 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c68017070 con 0x7f9c7406cb80 2024-03-24T00:32:22.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c71600640 1 -- 172.21.15.40:0/3492247997 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6800ca00 con 0x7f9c7406cb80 2024-03-24T00:32:22.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c7c400640 1 -- 172.21.15.40:0/3492247997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c74150960 con 0x7f9c7406cb80 2024-03-24T00:32:22.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c71600640 1 -- 172.21.15.40:0/3492247997 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6800cdb0 con 0x7f9c7406cb80 2024-03-24T00:32:22.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c7c400640 1 -- 172.21.15.40:0/3492247997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c74150f20 con 0x7f9c7406cb80 2024-03-24T00:32:22.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.756+0000 7f9c7c400640 1 -- 172.21.15.40:0/3492247997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c74068170 con 0x7f9c7406cb80 2024-03-24T00:32:22.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.760+0000 7f9c71600640 1 -- 172.21.15.40:0/3492247997 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f9c6801a070 con 0x7f9c7406cb80 2024-03-24T00:32:22.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.760+0000 7f9c71600640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c54067660 0x7f9c54069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:22.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.760+0000 7f9c71600640 1 -- 172.21.15.40:0/3492247997 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c68085c00 con 0x7f9c7406cb80 2024-03-24T00:32:22.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.760+0000 7f9c79000640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c54067660 0x7f9c54069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:22.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.760+0000 7f9c79000640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c54067660 0x7f9c54069b20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9c740b27e0 tx=0x7f9c60007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:22.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.768+0000 7f9c71600640 1 -- 172.21.15.40:0/3492247997 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9c680562e0 con 0x7f9c7406cb80 2024-03-24T00:32:22.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.932+0000 7f9c7c400640 1 -- 172.21.15.40:0/3492247997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_distributed}] v 0) v1 -- 0x7f9c740b4480 con 0x7f9c7406cb80 2024-03-24T00:32:22.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.932+0000 7f9c71600640 1 -- 172.21.15.40:0/3492247997 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_distributed}]=0 v14)=0 v14) v1 ==== 134+0+0 (secure 0 0 0) 0x7f9c6805a380 con 0x7f9c7406cb80 2024-03-24T00:32:22.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.936+0000 7f9c7c400640 1 -- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c54067660 msgr2=0x7f9c54069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:22.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.936+0000 7f9c7c400640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c54067660 0x7f9c54069b20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9c740b27e0 tx=0x7f9c60007e90 comp rx=0 tx=0).stop 2024-03-24T00:32:22.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.936+0000 7f9c7c400640 1 -- 172.21.15.40:0/3492247997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7406cb80 msgr2=0x7f9c741393e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:22.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.936+0000 7f9c7c400640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7406cb80 0x7f9c741393e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9c6800e930 tx=0x7f9c6800edf0 comp rx=0 tx=0).stop 2024-03-24T00:32:22.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.936+0000 7f9c7c400640 1 -- 172.21.15.40:0/3492247997 shutdown_connections 2024-03-24T00:32:22.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.936+0000 7f9c7c400640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c54067660 0x7f9c54069b20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:22.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.936+0000 7f9c7c400640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c7415ac60 0x7f9c740ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:22.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.936+0000 7f9c7c400640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c74071b00 0x7f9c74156aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:22.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.936+0000 7f9c7c400640 1 --2- 172.21.15.40:0/3492247997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c7406cb80 0x7f9c741393e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:22.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.936+0000 7f9c7c400640 1 -- 172.21.15.40:0/3492247997 >> 172.21.15.40:0/3492247997 conn(0x7f9c740b1660 msgr2=0x7f9c74158e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:22.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.936+0000 7f9c7c400640 1 -- 172.21.15.40:0/3492247997 shutdown_connections 2024-03-24T00:32:22.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:22.936+0000 7f9c7c400640 1 -- 172.21.15.40:0/3492247997 wait complete. 2024-03-24T00:32:22.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:22.958 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin.distributed -v 1 tree) 2024-03-24T00:32:36.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:32:36.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:32:37.984 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-24T00:32:38.144 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"1267","version":"1267","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"355","version":"355","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-24T00:32:38.146 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "1267", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "1267" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "355", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "355" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:32:38.146 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:32:38.147 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:32:38.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:32:38.271 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:32:38.271 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:32:38.271 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:32:38.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:32:38.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e83e00640 1 Processor -- start 2024-03-24T00:32:38.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e83e00640 1 -- start start 2024-03-24T00:32:38.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e83e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e84145350 0x7f3e840cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:38.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e83e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e840cfb30 con 0x7f3e84145730 2024-03-24T00:32:38.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e83e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e840cfe10 con 0x7f3e84066e80 2024-03-24T00:32:38.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e83e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e840d00f0 con 0x7f3e84145350 2024-03-24T00:32:38.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e7be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3e84145730 0x7f3e840b6660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38576/0 (socket says 172.21.15.40:38576) 2024-03-24T00:32:38.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e7be00640 1 -- 172.21.15.40:0/77120493 learned_addr learned my addr 172.21.15.40:0/77120493 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:38.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e82a00640 1 --2- 172.21.15.40:0/77120493 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e84145350 0x7f3e840cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:38.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e82000640 1 -- 172.21.15.40:0/77120493 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537072408 0 0) 0x7f3e840cfb30 con 0x7f3e84145730 2024-03-24T00:32:38.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e82000640 1 -- 172.21.15.40:0/77120493 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e68003530 con 0x7f3e84145730 2024-03-24T00:32:38.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e82000640 1 -- 172.21.15.40:0/77120493 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974225518 0 0) 0x7f3e840cfe10 con 0x7f3e84066e80 2024-03-24T00:32:38.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e82000640 1 -- 172.21.15.40:0/77120493 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e840cfb30 con 0x7f3e84066e80 2024-03-24T00:32:38.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e82a00640 1 -- 172.21.15.40:0/77120493 >> v1:172.21.15.53:6789/0 conn(0x7f3e84066e80 legacy=0x7f3e840b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:38.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e82a00640 1 -- 172.21.15.40:0/77120493 >> v1:172.21.15.40:6789/0 conn(0x7f3e84145730 legacy=0x7f3e840b6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:38.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.460+0000 7f3e82a00640 1 -- 172.21.15.40:0/77120493 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e840d03d0 con 0x7f3e84145350 2024-03-24T00:32:38.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.464+0000 7f3e82a00640 1 --2- 172.21.15.40:0/77120493 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e84145350 0x7f3e840cd6c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3e740097f0 tx=0x7f3e7401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=200ae91d5871dd8c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:38.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.464+0000 7f3e82000640 1 -- 172.21.15.40:0/77120493 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e740b0070 con 0x7f3e84145350 2024-03-24T00:32:38.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.464+0000 7f3e82000640 1 -- 172.21.15.40:0/77120493 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e740ab600 con 0x7f3e84145350 2024-03-24T00:32:38.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.464+0000 7f3e82000640 1 -- 172.21.15.40:0/77120493 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e740ab9d0 con 0x7f3e84145350 2024-03-24T00:32:38.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.464+0000 7f3e83e00640 1 -- 172.21.15.40:0/77120493 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e84145350 msgr2=0x7f3e840cd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:38.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.464+0000 7f3e83e00640 1 --2- 172.21.15.40:0/77120493 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e84145350 0x7f3e840cd6c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3e740097f0 tx=0x7f3e7401c420 comp rx=0 tx=0).stop 2024-03-24T00:32:38.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.464+0000 7f3e83e00640 1 -- 172.21.15.40:0/77120493 shutdown_connections 2024-03-24T00:32:38.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.464+0000 7f3e83e00640 1 --2- 172.21.15.40:0/77120493 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e84145350 0x7f3e840cd6c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.464+0000 7f3e83e00640 1 -- 172.21.15.40:0/77120493 >> 172.21.15.40:0/77120493 conn(0x7f3e840b2810 msgr2=0x7f3e840b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:38.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.464+0000 7f3e83e00640 1 -- 172.21.15.40:0/77120493 shutdown_connections 2024-03-24T00:32:38.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.464+0000 7f3e83e00640 1 -- 172.21.15.40:0/77120493 wait complete. 2024-03-24T00:32:38.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e83e00640 1 Processor -- start 2024-03-24T00:32:38.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e83e00640 1 -- start start 2024-03-24T00:32:38.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e83e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e84066e80 0x7f3e84158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:38.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e83e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e84145350 0x7f3e841413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:38.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e83e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e84145730 0x7f3e8415d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:38.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e83e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e840b4fb0 con 0x7f3e84145350 2024-03-24T00:32:38.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e83e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e840b4e30 con 0x7f3e84145730 2024-03-24T00:32:38.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e83e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e84169b10 con 0x7f3e84066e80 2024-03-24T00:32:38.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e7be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e84145350 0x7f3e841413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:38.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e82a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e84066e80 0x7f3e84158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:38.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e7be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e84145350 0x7f3e841413b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49092/0 (socket says 172.21.15.40:49092) 2024-03-24T00:32:38.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e7be00640 1 -- 172.21.15.40:0/869841664 learned_addr learned my addr 172.21.15.40:0/869841664 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:38.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e83400640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e84145730 0x7f3e8415d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:38.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e82a00640 1 -- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e84145730 msgr2=0x7f3e8415d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:38.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e82a00640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e84145730 0x7f3e8415d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e82a00640 1 -- 172.21.15.40:0/869841664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e84145350 msgr2=0x7f3e841413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:38.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e82a00640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e84145350 0x7f3e841413b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e82a00640 1 -- 172.21.15.40:0/869841664 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e84141e70 con 0x7f3e84066e80 2024-03-24T00:32:38.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.468+0000 7f3e82a00640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e84066e80 0x7f3e84158990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3e840d19f0 tx=0x7f3e7401ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:38.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.472+0000 7f3e7aa00640 1 -- 172.21.15.40:0/869841664 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e740b0040 con 0x7f3e84066e80 2024-03-24T00:32:38.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.472+0000 7f3e7aa00640 1 -- 172.21.15.40:0/869841664 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e740b9070 con 0x7f3e84066e80 2024-03-24T00:32:38.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.472+0000 7f3e7aa00640 1 -- 172.21.15.40:0/869841664 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e740b46a0 con 0x7f3e84066e80 2024-03-24T00:32:38.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.472+0000 7f3e83e00640 1 -- 172.21.15.40:0/869841664 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e84142150 con 0x7f3e84066e80 2024-03-24T00:32:38.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.472+0000 7f3e83e00640 1 -- 172.21.15.40:0/869841664 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e84158ed0 con 0x7f3e84066e80 2024-03-24T00:32:38.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.472+0000 7f3e83e00640 1 -- 172.21.15.40:0/869841664 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e840733f0 con 0x7f3e84066e80 2024-03-24T00:32:38.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.472+0000 7f3e7aa00640 1 -- 172.21.15.40:0/869841664 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3e74004020 con 0x7f3e84066e80 2024-03-24T00:32:38.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.472+0000 7f3e7aa00640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e6006baa0 0x7f3e6006df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:38.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.472+0000 7f3e7aa00640 1 -- 172.21.15.40:0/869841664 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e7411ef50 con 0x7f3e84066e80 2024-03-24T00:32:38.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.472+0000 7f3e7be00640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e6006baa0 0x7f3e6006df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:38.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.476+0000 7f3e7be00640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e6006baa0 0x7f3e6006df60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3e841f92e0 tx=0x7f3e6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:38.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.476+0000 7f3e7aa00640 1 -- 172.21.15.40:0/869841664 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3e740160d0 con 0x7f3e84066e80 2024-03-24T00:32:38.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.644+0000 7f3e83e00640 1 -- 172.21.15.40:0/869841664 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e84067af0 con 0x7f3e84066e80 2024-03-24T00:32:38.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.644+0000 7f3e7aa00640 1 -- 172.21.15.40:0/869841664 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f3e740ef570 con 0x7f3e84066e80 2024-03-24T00:32:38.647 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:38.647 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":33,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:32:07.623938+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:03.863519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:05.024936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:32:38.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.648+0000 7f3e83e00640 1 -- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e6006baa0 msgr2=0x7f3e6006df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:38.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.648+0000 7f3e83e00640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e6006baa0 0x7f3e6006df60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3e841f92e0 tx=0x7f3e6c009040 comp rx=0 tx=0).stop 2024-03-24T00:32:38.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.648+0000 7f3e83e00640 1 -- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e84066e80 msgr2=0x7f3e84158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:38.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.648+0000 7f3e83e00640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e84066e80 0x7f3e84158990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3e840d19f0 tx=0x7f3e7401ac60 comp rx=0 tx=0).stop 2024-03-24T00:32:38.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.648+0000 7f3e83e00640 1 -- 172.21.15.40:0/869841664 shutdown_connections 2024-03-24T00:32:38.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.648+0000 7f3e83e00640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e6006baa0 0x7f3e6006df60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.648+0000 7f3e83e00640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e84145730 0x7f3e8415d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.648+0000 7f3e83e00640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e84145350 0x7f3e841413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.648+0000 7f3e83e00640 1 --2- 172.21.15.40:0/869841664 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e84066e80 0x7f3e84158990 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.648+0000 7f3e83e00640 1 -- 172.21.15.40:0/869841664 >> 172.21.15.40:0/869841664 conn(0x7f3e840b2810 msgr2=0x7f3e8413cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:38.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.648+0000 7f3e83e00640 1 -- 172.21.15.40:0/869841664 shutdown_connections 2024-03-24T00:32:38.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.648+0000 7f3e83e00640 1 -- 172.21.15.40:0/869841664 wait complete. 2024-03-24T00:32:38.673 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:32:03.863519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:32:05.024936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:32:38.673 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:32:38.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.816+0000 7fb013000640 1 Processor -- start 2024-03-24T00:32:38.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.816+0000 7fb013000640 1 -- start start 2024-03-24T00:32:38.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb013000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb00c15b960 0x7fb00c070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:38.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb013000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb00c0d16f0 0x7fb00c0712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:38.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb013000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb00c15f970 con 0x7fb00c15bd40 2024-03-24T00:32:38.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb013000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb00c160580 con 0x7fb00c0d16f0 2024-03-24T00:32:38.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb013000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c15fc50 con 0x7fb00c15b960 2024-03-24T00:32:38.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb011000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb00c0d16f0 0x7fb00c0712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:38.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb003400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb00c15bd40 0x7fb00c0d0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38582/0 (socket says 172.21.15.40:38582) 2024-03-24T00:32:38.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb003400640 1 -- 172.21.15.40:0/350519967 learned_addr learned my addr 172.21.15.40:0/350519967 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:38.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb00be00640 1 --2- 172.21.15.40:0/350519967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb00c15b960 0x7fb00c070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:38.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb011000640 1 -- 172.21.15.40:0/350519967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb00c15b960 msgr2=0x7fb00c070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:38.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb011000640 1 --2- 172.21.15.40:0/350519967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb00c15b960 0x7fb00c070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb011000640 1 -- 172.21.15.40:0/350519967 >> v1:172.21.15.40:6789/0 conn(0x7fb00c15bd40 legacy=0x7fb00c0d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:38.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb00b400640 1 -- 172.21.15.40:0/350519967 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452175558 0 0) 0x7fb00c15f970 con 0x7fb00c15bd40 2024-03-24T00:32:38.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.820+0000 7fb011000640 1 -- 172.21.15.40:0/350519967 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c15ff30 con 0x7fb00c0d16f0 2024-03-24T00:32:38.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.824+0000 7fb011000640 1 --2- 172.21.15.40:0/350519967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb00c0d16f0 0x7fb00c0712a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb0040097c0 tx=0x7fb00401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c0337823e5a0a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:38.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.824+0000 7fb00b400640 1 -- 172.21.15.40:0/350519967 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0040b1070 con 0x7fb00c0d16f0 2024-03-24T00:32:38.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.824+0000 7fb00b400640 1 -- 172.21.15.40:0/350519967 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0040ac410 con 0x7fb00c0d16f0 2024-03-24T00:32:38.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.824+0000 7fb00b400640 1 -- 172.21.15.40:0/350519967 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0040ac7e0 con 0x7fb00c0d16f0 2024-03-24T00:32:38.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.824+0000 7fb013000640 1 -- 172.21.15.40:0/350519967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb00c0d16f0 msgr2=0x7fb00c0712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:38.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.824+0000 7fb013000640 1 --2- 172.21.15.40:0/350519967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb00c0d16f0 0x7fb00c0712a0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb0040097c0 tx=0x7fb00401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.824+0000 7fb013000640 1 -- 172.21.15.40:0/350519967 shutdown_connections 2024-03-24T00:32:38.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.824+0000 7fb013000640 1 --2- 172.21.15.40:0/350519967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb00c0d16f0 0x7fb00c0712a0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.824+0000 7fb013000640 1 --2- 172.21.15.40:0/350519967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb00c15b960 0x7fb00c070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.824+0000 7fb013000640 1 -- 172.21.15.40:0/350519967 >> 172.21.15.40:0/350519967 conn(0x7fb00c06ecf0 msgr2=0x7fb00c06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:38.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.824+0000 7fb013000640 1 -- 172.21.15.40:0/350519967 shutdown_connections 2024-03-24T00:32:38.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.824+0000 7fb013000640 1 -- 172.21.15.40:0/350519967 wait complete. 2024-03-24T00:32:38.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb013000640 1 Processor -- start 2024-03-24T00:32:38.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb013000640 1 -- start start 2024-03-24T00:32:38.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb013000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb00c0d16f0 0x7fb00c06ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:38.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb013000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb00c15b960 0x7fb00c1464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:38.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb013000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb00c15bd40 0x7fb00c146a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:38.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb013000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0ce580 con 0x7fb00c0d16f0 2024-03-24T00:32:38.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb013000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0ce400 con 0x7fb00c15b960 2024-03-24T00:32:38.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb013000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c0ce700 con 0x7fb00c15bd40 2024-03-24T00:32:38.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb00be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb00c0d16f0 0x7fb00c06ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:38.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb00be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb00c0d16f0 0x7fb00c06ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49116/0 (socket says 172.21.15.40:49116) 2024-03-24T00:32:38.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb00be00640 1 -- 172.21.15.40:0/2195486157 learned_addr learned my addr 172.21.15.40:0/2195486157 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:38.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb003400640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb00c15b960 0x7fb00c1464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb011000640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb00c15bd40 0x7fb00c146a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb00be00640 1 -- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb00c15bd40 msgr2=0x7fb00c146a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb00be00640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb00c15bd40 0x7fb00c146a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb00be00640 1 -- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb00c15b960 msgr2=0x7fb00c1464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb00be00640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb00c15b960 0x7fb00c1464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb00be00640 1 -- 172.21.15.40:0/2195486157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c154480 con 0x7fb00c0d16f0 2024-03-24T00:32:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb00be00640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb00c0d16f0 0x7fb00c06ccb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7faffc00e960 tx=0x7faffc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb008c00640 1 -- 172.21.15.40:0/2195486157 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc017070 con 0x7fb00c0d16f0 2024-03-24T00:32:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb008c00640 1 -- 172.21.15.40:0/2195486157 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faffc00ca00 con 0x7fb00c0d16f0 2024-03-24T00:32:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb008c00640 1 -- 172.21.15.40:0/2195486157 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faffc00cdb0 con 0x7fb00c0d16f0 2024-03-24T00:32:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb013000640 1 -- 172.21.15.40:0/2195486157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00c0c7350 con 0x7fb00c0d16f0 2024-03-24T00:32:38.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb013000640 1 -- 172.21.15.40:0/2195486157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00c0c7910 con 0x7fb00c0d16f0 2024-03-24T00:32:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.828+0000 7fb013000640 1 -- 172.21.15.40:0/2195486157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00c068170 con 0x7fb00c0d16f0 2024-03-24T00:32:38.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.832+0000 7fb008c00640 1 -- 172.21.15.40:0/2195486157 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7faffc01a070 con 0x7fb00c0d16f0 2024-03-24T00:32:38.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.832+0000 7fb008c00640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafe8067660 0x7fafe8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:38.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.832+0000 7fb008c00640 1 -- 172.21.15.40:0/2195486157 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faffc0858b0 con 0x7fb00c0d16f0 2024-03-24T00:32:38.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.832+0000 7fb003400640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafe8067660 0x7fafe8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:38.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.832+0000 7fb003400640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafe8067660 0x7fafe8069b20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb00c15b340 tx=0x7faff4001690 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:38.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:38.836+0000 7fb008c00640 1 -- 172.21.15.40:0/2195486157 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb00c154480 con 0x7fb00c0d16f0 2024-03-24T00:32:39.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.000+0000 7fb013000640 1 -- 172.21.15.40:0/2195486157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb00c15ddc0 con 0x7fb00c0d16f0 2024-03-24T00:32:39.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.000+0000 7fb008c00640 1 -- 172.21.15.40:0/2195486157 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7faffc055ed0 con 0x7fb00c0d16f0 2024-03-24T00:32:39.003 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:32:39.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.004+0000 7fb013000640 1 -- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafe8067660 msgr2=0x7fafe8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.004+0000 7fb013000640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafe8067660 0x7fafe8069b20 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb00c15b340 tx=0x7faff4001690 comp rx=0 tx=0).stop 2024-03-24T00:32:39.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.004+0000 7fb013000640 1 -- 172.21.15.40:0/2195486157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb00c0d16f0 msgr2=0x7fb00c06ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.004+0000 7fb013000640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb00c0d16f0 0x7fb00c06ccb0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7faffc00e960 tx=0x7faffc00ee20 comp rx=0 tx=0).stop 2024-03-24T00:32:39.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.004+0000 7fb013000640 1 -- 172.21.15.40:0/2195486157 shutdown_connections 2024-03-24T00:32:39.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.004+0000 7fb013000640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafe8067660 0x7fafe8069b20 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.004+0000 7fb013000640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb00c15bd40 0x7fb00c146a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.004+0000 7fb013000640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb00c15b960 0x7fb00c1464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.004+0000 7fb013000640 1 --2- 172.21.15.40:0/2195486157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb00c0d16f0 0x7fb00c06ccb0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.004+0000 7fb013000640 1 -- 172.21.15.40:0/2195486157 >> 172.21.15.40:0/2195486157 conn(0x7fb00c06ecf0 msgr2=0x7fb00c150ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:39.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.004+0000 7fb013000640 1 -- 172.21.15.40:0/2195486157 shutdown_connections 2024-03-24T00:32:39.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.004+0000 7fb013000640 1 -- 172.21.15.40:0/2195486157 wait complete. 2024-03-24T00:32:39.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:32:39.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.176+0000 7f10e6600640 1 Processor -- start 2024-03-24T00:32:39.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.176+0000 7f10e6600640 1 -- start start 2024-03-24T00:32:39.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.176+0000 7f10e6600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f10e015ffd0 0x7f10e00ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.176+0000 7f10e6600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f10e00728d0 0x7f10e00cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.176+0000 7f10e6600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10e0071de0 con 0x7f10e015bd00 2024-03-24T00:32:39.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.176+0000 7f10e6600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e00cf790 con 0x7f10e00728d0 2024-03-24T00:32:39.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.176+0000 7f10e6600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10e00cea50 con 0x7f10e015ffd0 2024-03-24T00:32:39.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.176+0000 7f10df400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f10e015bd00 0x7f10e015f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38590/0 (socket says 172.21.15.40:38590) 2024-03-24T00:32:39.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.176+0000 7f10df400640 1 -- 172.21.15.40:0/1924466412 learned_addr learned my addr 172.21.15.40:0/1924466412 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:39.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.176+0000 7f10dea00640 1 --2- 172.21.15.40:0/1924466412 >> v2:172.21.15.53:3300/0 conn(0x7f10e015ffd0 0x7f10e00ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.176+0000 7f10dfe00640 1 --2- 172.21.15.40:0/1924466412 >> v2:172.21.15.40:3300/0 conn(0x7f10e00728d0 0x7f10e00cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.180+0000 7f10dea00640 1 -- 172.21.15.40:0/1924466412 >> v2:172.21.15.40:3300/0 conn(0x7f10e00728d0 msgr2=0x7f10e00cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:39.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.180+0000 7f10dea00640 1 --2- 172.21.15.40:0/1924466412 >> v2:172.21.15.40:3300/0 conn(0x7f10e00728d0 0x7f10e00cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.180+0000 7f10dea00640 1 -- 172.21.15.40:0/1924466412 >> v1:172.21.15.40:6789/0 conn(0x7f10e015bd00 legacy=0x7f10e015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.180+0000 7f10de000640 1 -- 172.21.15.40:0/1924466412 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642807872 0 0) 0x7f10e0071de0 con 0x7f10e015bd00 2024-03-24T00:32:39.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.180+0000 7f10dea00640 1 -- 172.21.15.40:0/1924466412 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e00ced30 con 0x7f10e015ffd0 2024-03-24T00:32:39.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.180+0000 7f10dfe00640 1 --2- 172.21.15.40:0/1924466412 >> v2:172.21.15.40:3300/0 conn(0x7f10e00728d0 0x7f10e00cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:39.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.180+0000 7f10dea00640 1 --2- 172.21.15.40:0/1924466412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10e015ffd0 0x7f10e00ae8e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f10e006ff40 tx=0x7f10cc01c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=974a750f556f2bb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:39.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.180+0000 7f10de000640 1 -- 172.21.15.40:0/1924466412 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc0ab430 con 0x7f10e015ffd0 2024-03-24T00:32:39.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.180+0000 7f10de000640 1 -- 172.21.15.40:0/1924466412 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc0b0070 con 0x7f10e015ffd0 2024-03-24T00:32:39.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.180+0000 7f10de000640 1 -- 172.21.15.40:0/1924466412 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc0ab7f0 con 0x7f10e015ffd0 2024-03-24T00:32:39.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 -- 172.21.15.40:0/1924466412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10e015ffd0 msgr2=0x7f10e00ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 --2- 172.21.15.40:0/1924466412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10e015ffd0 0x7f10e00ae8e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f10e006ff40 tx=0x7f10cc01c450 comp rx=0 tx=0).stop 2024-03-24T00:32:39.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 -- 172.21.15.40:0/1924466412 shutdown_connections 2024-03-24T00:32:39.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 --2- 172.21.15.40:0/1924466412 >> v2:172.21.15.40:3300/0 conn(0x7f10e00728d0 0x7f10e00cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 --2- 172.21.15.40:0/1924466412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10e015ffd0 0x7f10e00ae8e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 -- 172.21.15.40:0/1924466412 >> 172.21.15.40:0/1924466412 conn(0x7f10e00b1660 msgr2=0x7f10e00b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:39.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 -- 172.21.15.40:0/1924466412 shutdown_connections 2024-03-24T00:32:39.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 -- 172.21.15.40:0/1924466412 wait complete. 2024-03-24T00:32:39.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 Processor -- start 2024-03-24T00:32:39.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 -- start start 2024-03-24T00:32:39.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f10e00728d0 0x7f10e015edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e015bd00 0x7f10e015cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10e015ffd0 0x7f10e014d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e006d320 con 0x7f10e015bd00 2024-03-24T00:32:39.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e015c0e0 con 0x7f10e015ffd0 2024-03-24T00:32:39.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10e6600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10e006d4a0 con 0x7f10e00728d0 2024-03-24T00:32:39.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10dea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e015bd00 0x7f10e015cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10dea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e015bd00 0x7f10e015cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49140/0 (socket says 172.21.15.40:49140) 2024-03-24T00:32:39.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10dea00640 1 -- 172.21.15.40:0/1582107763 learned_addr learned my addr 172.21.15.40:0/1582107763 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:39.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10df400640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f10e00728d0 0x7f10e015edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10dfe00640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10e015ffd0 0x7f10e014d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10dea00640 1 -- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f10e00728d0 msgr2=0x7f10e015edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10dea00640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f10e00728d0 0x7f10e015edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10dea00640 1 -- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10e015ffd0 msgr2=0x7f10e014d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10dea00640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10e015ffd0 0x7f10e014d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10dea00640 1 -- 172.21.15.40:0/1582107763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e0154da0 con 0x7f10e015bd00 2024-03-24T00:32:39.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.184+0000 7f10dfe00640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10e015ffd0 0x7f10e014d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:39.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.188+0000 7f10dea00640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e015bd00 0x7f10e015cdb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f10cc0ad040 tx=0x7f10cc01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:39.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.188+0000 7f10d7400640 1 -- 172.21.15.40:0/1582107763 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc006e20 con 0x7f10e015bd00 2024-03-24T00:32:39.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.188+0000 7f10d7400640 1 -- 172.21.15.40:0/1582107763 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc0bf630 con 0x7f10e015bd00 2024-03-24T00:32:39.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.188+0000 7f10e6600640 1 -- 172.21.15.40:0/1582107763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e0155040 con 0x7f10e015bd00 2024-03-24T00:32:39.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.188+0000 7f10d7400640 1 -- 172.21.15.40:0/1582107763 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc0bfa20 con 0x7f10e015bd00 2024-03-24T00:32:39.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.188+0000 7f10e6600640 1 -- 172.21.15.40:0/1582107763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e0155600 con 0x7f10e015bd00 2024-03-24T00:32:39.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.188+0000 7f10d4c00640 1 -- 172.21.15.40:0/1582107763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e0068170 con 0x7f10e015bd00 2024-03-24T00:32:39.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.188+0000 7f10d7400640 1 -- 172.21.15.40:0/1582107763 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f10cc0bd020 con 0x7f10e015bd00 2024-03-24T00:32:39.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.192+0000 7f10d7400640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f10c0067660 0x7f10c0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.192+0000 7f10d7400640 1 -- 172.21.15.40:0/1582107763 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10cc0a8030 con 0x7f10e015bd00 2024-03-24T00:32:39.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.192+0000 7f10df400640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f10c0067660 0x7f10c0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.192+0000 7f10df400640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f10c0067660 0x7f10c0069b20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f10d0007fd0 tx=0x7f10d0007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:39.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.196+0000 7f10d7400640 1 -- 172.21.15.40:0/1582107763 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f10cc0c4060 con 0x7f10e015bd00 2024-03-24T00:32:39.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d4c00640 1 -- 172.21.15.40:0/1582107763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f10e01598c0 con 0x7f10e015bd00 2024-03-24T00:32:39.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d7400640 1 -- 172.21.15.40:0/1582107763 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7f10cc016070 con 0x7f10e015bd00 2024-03-24T00:32:39.362 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:32:39.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d4c00640 1 -- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f10c0067660 msgr2=0x7f10c0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d4c00640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f10c0067660 0x7f10c0069b20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f10d0007fd0 tx=0x7f10d0007d60 comp rx=0 tx=0).stop 2024-03-24T00:32:39.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d4c00640 1 -- 172.21.15.40:0/1582107763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e015bd00 msgr2=0x7f10e015cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d4c00640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e015bd00 0x7f10e015cdb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f10cc0ad040 tx=0x7f10cc01ac90 comp rx=0 tx=0).stop 2024-03-24T00:32:39.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d4c00640 1 -- 172.21.15.40:0/1582107763 shutdown_connections 2024-03-24T00:32:39.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d4c00640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f10c0067660 0x7f10c0069b20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d4c00640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10e015ffd0 0x7f10e014d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d4c00640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f10e015bd00 0x7f10e015cdb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d4c00640 1 --2- 172.21.15.40:0/1582107763 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f10e00728d0 0x7f10e015edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d4c00640 1 -- 172.21.15.40:0/1582107763 >> 172.21.15.40:0/1582107763 conn(0x7f10e00b1660 msgr2=0x7f10e0158b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:39.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.360+0000 7f10d4c00640 1 -- 172.21.15.40:0/1582107763 shutdown_connections 2024-03-24T00:32:39.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.364+0000 7f10d4c00640 1 -- 172.21.15.40:0/1582107763 wait complete. 2024-03-24T00:32:39.383 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:32:39.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.524+0000 7fae61000640 1 Processor -- start 2024-03-24T00:32:39.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.524+0000 7fae61000640 1 -- start start 2024-03-24T00:32:39.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae61000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fae5c15a560 0x7fae5c0ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae61000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fae5c0aec70 0x7fae5c0cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae61000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae5c06d3a0 0x7fae5c06d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae61000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae5c0cca30 con 0x7fae5c15a560 2024-03-24T00:32:39.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae61000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae5c070ed0 con 0x7fae5c0aec70 2024-03-24T00:32:39.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae61000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c0cf240 con 0x7fae5c06d3a0 2024-03-24T00:32:39.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae59e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fae5c15a560 0x7fae5c0ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae5a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae5c06d3a0 0x7fae5c06d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae59400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fae5c0aec70 0x7fae5c0cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae59e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fae5c15a560 0x7fae5c0ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49148/0 (socket says 172.21.15.40:49148) 2024-03-24T00:32:39.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae5a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae5c06d3a0 0x7fae5c06d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:46944/0 (socket says 172.21.15.40:46944) 2024-03-24T00:32:39.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae59e00640 1 -- 172.21.15.40:0/3630211453 learned_addr learned my addr 172.21.15.40:0/3630211453 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:39.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae59400640 1 -- 172.21.15.40:0/3630211453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae5c06d3a0 msgr2=0x7fae5c06d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:39.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae59400640 1 --2- 172.21.15.40:0/3630211453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae5c06d3a0 0x7fae5c06d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae59400640 1 -- 172.21.15.40:0/3630211453 >> v2:172.21.15.40:3300/0 conn(0x7fae5c15a560 msgr2=0x7fae5c0ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:39.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae59400640 1 --2- 172.21.15.40:0/3630211453 >> v2:172.21.15.40:3300/0 conn(0x7fae5c15a560 0x7fae5c0ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae59400640 1 -- 172.21.15.40:0/3630211453 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c0cf520 con 0x7fae5c0aec70 2024-03-24T00:32:39.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.528+0000 7fae59400640 1 --2- 172.21.15.40:0/3630211453 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae5c0aec70 0x7fae5c0cc4f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fae5c067dc0 tx=0x7fae4801c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b9a269bfc51ad1e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:39.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae58a00640 1 -- 172.21.15.40:0/3630211453 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae480b0070 con 0x7fae5c0aec70 2024-03-24T00:32:39.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae58a00640 1 -- 172.21.15.40:0/3630211453 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae480ab600 con 0x7fae5c0aec70 2024-03-24T00:32:39.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae58a00640 1 -- 172.21.15.40:0/3630211453 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae480ab9d0 con 0x7fae5c0aec70 2024-03-24T00:32:39.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae61000640 1 -- 172.21.15.40:0/3630211453 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae5c0aec70 msgr2=0x7fae5c0cc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae61000640 1 --2- 172.21.15.40:0/3630211453 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae5c0aec70 0x7fae5c0cc4f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fae5c067dc0 tx=0x7fae4801c3f0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae61000640 1 -- 172.21.15.40:0/3630211453 shutdown_connections 2024-03-24T00:32:39.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae61000640 1 --2- 172.21.15.40:0/3630211453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae5c06d3a0 0x7fae5c06d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae61000640 1 --2- 172.21.15.40:0/3630211453 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae5c0aec70 0x7fae5c0cc4f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae61000640 1 --2- 172.21.15.40:0/3630211453 >> v2:172.21.15.40:3300/0 conn(0x7fae5c15a560 0x7fae5c0ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae61000640 1 -- 172.21.15.40:0/3630211453 >> 172.21.15.40:0/3630211453 conn(0x7fae5c0b1660 msgr2=0x7fae5c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:39.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae61000640 1 -- 172.21.15.40:0/3630211453 shutdown_connections 2024-03-24T00:32:39.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae61000640 1 -- 172.21.15.40:0/3630211453 wait complete. 2024-03-24T00:32:39.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae61000640 1 Processor -- start 2024-03-24T00:32:39.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.532+0000 7fae61000640 1 -- start start 2024-03-24T00:32:39.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae61000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae5c06d3a0 0x7fae5c0b0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae61000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae5c0aec70 0x7fae5c0cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae61000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae5c15a560 0x7fae5c1532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae61000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c0b4140 con 0x7fae5c0aec70 2024-03-24T00:32:39.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae61000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c0b3fc0 con 0x7fae5c15a560 2024-03-24T00:32:39.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae61000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c0b42c0 con 0x7fae5c06d3a0 2024-03-24T00:32:39.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae59400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae5c0aec70 0x7fae5c0cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae59400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae5c0aec70 0x7fae5c0cb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49156/0 (socket says 172.21.15.40:49156) 2024-03-24T00:32:39.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae59400640 1 -- 172.21.15.40:0/546544738 learned_addr learned my addr 172.21.15.40:0/546544738 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:39.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae59e00640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae5c06d3a0 0x7fae5c0b0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae5a800640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae5c15a560 0x7fae5c1532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae59400640 1 -- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae5c06d3a0 msgr2=0x7fae5c0b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae59400640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae5c06d3a0 0x7fae5c0b0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae59400640 1 -- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae5c15a560 msgr2=0x7fae5c1532e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae59400640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae5c15a560 0x7fae5c1532e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae59400640 1 -- 172.21.15.40:0/546544738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c0b0110 con 0x7fae5c0aec70 2024-03-24T00:32:39.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae59400640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae5c0aec70 0x7fae5c0cb3c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fae4801cc90 tx=0x7fae48019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:39.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae52000640 1 -- 172.21.15.40:0/546544738 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae480b0040 con 0x7fae5c0aec70 2024-03-24T00:32:39.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae52000640 1 -- 172.21.15.40:0/546544738 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae480b9070 con 0x7fae5c0aec70 2024-03-24T00:32:39.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae52000640 1 -- 172.21.15.40:0/546544738 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae480b4650 con 0x7fae5c0aec70 2024-03-24T00:32:39.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae61000640 1 -- 172.21.15.40:0/546544738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5c0c4c20 con 0x7fae5c0aec70 2024-03-24T00:32:39.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae61000640 1 -- 172.21.15.40:0/546544738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5c0c48d0 con 0x7fae5c0aec70 2024-03-24T00:32:39.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.536+0000 7fae61000640 1 -- 172.21.15.40:0/546544738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5c068170 con 0x7fae5c0aec70 2024-03-24T00:32:39.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.540+0000 7fae52000640 1 -- 172.21.15.40:0/546544738 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fae480b47f0 con 0x7fae5c0aec70 2024-03-24T00:32:39.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.540+0000 7fae52000640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fae3c067660 0x7fae3c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.540+0000 7fae52000640 1 -- 172.21.15.40:0/546544738 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae4811f4c0 con 0x7fae5c0aec70 2024-03-24T00:32:39.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.540+0000 7fae59e00640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fae3c067660 0x7fae3c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.540+0000 7fae59e00640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fae3c067660 0x7fae3c069b20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fae4c009a30 tx=0x7fae4c009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:39.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.544+0000 7fae52000640 1 -- 172.21.15.40:0/546544738 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fae48016070 con 0x7fae5c0aec70 2024-03-24T00:32:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.704+0000 7fae61000640 1 -- 172.21.15.40:0/546544738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fae5c071ba0 con 0x7fae5c0aec70 2024-03-24T00:32:39.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae52000640 1 -- 172.21.15.40:0/546544738 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7fae480efaf0 con 0x7fae5c0aec70 2024-03-24T00:32:39.708 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:32:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae61000640 1 -- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fae3c067660 msgr2=0x7fae3c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae61000640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fae3c067660 0x7fae3c069b20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fae4c009a30 tx=0x7fae4c009240 comp rx=0 tx=0).stop 2024-03-24T00:32:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae61000640 1 -- 172.21.15.40:0/546544738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae5c0aec70 msgr2=0x7fae5c0cb3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae61000640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae5c0aec70 0x7fae5c0cb3c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fae4801cc90 tx=0x7fae48019190 comp rx=0 tx=0).stop 2024-03-24T00:32:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae61000640 1 -- 172.21.15.40:0/546544738 shutdown_connections 2024-03-24T00:32:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae61000640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fae3c067660 0x7fae3c069b20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae61000640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fae5c15a560 0x7fae5c1532e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae61000640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fae5c0aec70 0x7fae5c0cb3c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae61000640 1 --2- 172.21.15.40:0/546544738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fae5c06d3a0 0x7fae5c0b0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae61000640 1 -- 172.21.15.40:0/546544738 >> 172.21.15.40:0/546544738 conn(0x7fae5c0b1660 msgr2=0x7fae5c158840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae61000640 1 -- 172.21.15.40:0/546544738 shutdown_connections 2024-03-24T00:32:39.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.708+0000 7fae61000640 1 -- 172.21.15.40:0/546544738 wait complete. 2024-03-24T00:32:39.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:32:39.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.868+0000 7fa545e00640 1 Processor -- start 2024-03-24T00:32:39.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.868+0000 7fa545e00640 1 -- start start 2024-03-24T00:32:39.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa545e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa5400b46c0 0x7fa5400b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa545e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5400b2ff0 0x7fa5400ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa545e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa540070f60 con 0x7fa54006e620 2024-03-24T00:32:39.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa545e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5400b3620 con 0x7fa5400b46c0 2024-03-24T00:32:39.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa545e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400b3900 con 0x7fa5400b2ff0 2024-03-24T00:32:39.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa53ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa54006e620 0x7fa54006ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38606/0 (socket says 172.21.15.40:38606) 2024-03-24T00:32:39.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa53ea00640 1 -- 172.21.15.40:0/1577683494 learned_addr learned my addr 172.21.15.40:0/1577683494 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:39.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa537e00640 1 --2- 172.21.15.40:0/1577683494 >> v2:172.21.15.40:3300/0 conn(0x7fa5400b46c0 0x7fa5400b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa53f400640 1 --2- 172.21.15.40:0/1577683494 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5400b2ff0 0x7fa5400ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa53e000640 1 -- 172.21.15.40:0/1577683494 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824234353 0 0) 0x7fa540070f60 con 0x7fa54006e620 2024-03-24T00:32:39.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa53e000640 1 -- 172.21.15.40:0/1577683494 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa524005740 con 0x7fa54006e620 2024-03-24T00:32:39.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa53f400640 1 -- 172.21.15.40:0/1577683494 >> v2:172.21.15.40:3300/0 conn(0x7fa5400b46c0 msgr2=0x7fa5400b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:39.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa53f400640 1 --2- 172.21.15.40:0/1577683494 >> v2:172.21.15.40:3300/0 conn(0x7fa5400b46c0 0x7fa5400b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa53f400640 1 -- 172.21.15.40:0/1577683494 >> v1:172.21.15.40:6789/0 conn(0x7fa54006e620 legacy=0x7fa54006ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.872+0000 7fa53f400640 1 -- 172.21.15.40:0/1577683494 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5400b3be0 con 0x7fa5400b2ff0 2024-03-24T00:32:39.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.876+0000 7fa53f400640 1 --2- 172.21.15.40:0/1577683494 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5400b2ff0 0x7fa5400ccfa0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa538009d90 tx=0x7fa53801c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41a6a16b3ec5b3de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:39.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.876+0000 7fa53e000640 1 -- 172.21.15.40:0/1577683494 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5380b2070 con 0x7fa5400b2ff0 2024-03-24T00:32:39.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.876+0000 7fa53e000640 1 -- 172.21.15.40:0/1577683494 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5380a5890 con 0x7fa5400b2ff0 2024-03-24T00:32:39.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.876+0000 7fa53e000640 1 -- 172.21.15.40:0/1577683494 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5400cdcf0 con 0x7fa5400b2ff0 2024-03-24T00:32:39.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.876+0000 7fa545e00640 1 -- 172.21.15.40:0/1577683494 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5400b2ff0 msgr2=0x7fa5400ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.876+0000 7fa545e00640 1 --2- 172.21.15.40:0/1577683494 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5400b2ff0 0x7fa5400ccfa0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa538009d90 tx=0x7fa53801c8e0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.876+0000 7fa545e00640 1 -- 172.21.15.40:0/1577683494 shutdown_connections 2024-03-24T00:32:39.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.876+0000 7fa545e00640 1 --2- 172.21.15.40:0/1577683494 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5400b2ff0 0x7fa5400ccfa0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.876+0000 7fa545e00640 1 --2- 172.21.15.40:0/1577683494 >> v2:172.21.15.40:3300/0 conn(0x7fa5400b46c0 0x7fa5400b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.876+0000 7fa545e00640 1 -- 172.21.15.40:0/1577683494 >> 172.21.15.40:0/1577683494 conn(0x7fa5400b17b0 msgr2=0x7fa5400b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:39.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.876+0000 7fa545e00640 1 -- 172.21.15.40:0/1577683494 shutdown_connections 2024-03-24T00:32:39.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.876+0000 7fa545e00640 1 -- 172.21.15.40:0/1577683494 wait complete. 2024-03-24T00:32:39.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.880+0000 7fa545e00640 1 Processor -- start 2024-03-24T00:32:39.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.880+0000 7fa545e00640 1 -- start start 2024-03-24T00:32:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa545e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa54006e620 0x7fa540165040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa545e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5400b2ff0 0x7fa54014e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa545e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5400b46c0 0x7fa54013dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa545e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa540144920 con 0x7fa5400b2ff0 2024-03-24T00:32:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa545e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54006d510 con 0x7fa54006e620 2024-03-24T00:32:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa545e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5400cc330 con 0x7fa5400b46c0 2024-03-24T00:32:39.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa537e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5400b2ff0 0x7fa54014e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa53ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa54006e620 0x7fa540165040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa53f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5400b46c0 0x7fa54013dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa537e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5400b2ff0 0x7fa54014e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49174/0 (socket says 172.21.15.40:49174) 2024-03-24T00:32:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa537e00640 1 -- 172.21.15.40:0/23315119 learned_addr learned my addr 172.21.15.40:0/23315119 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa53ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa54006e620 0x7fa540165040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:52292/0 (socket says 172.21.15.40:52292) 2024-03-24T00:32:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa537e00640 1 -- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5400b46c0 msgr2=0x7fa54013dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa537e00640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5400b46c0 0x7fa54013dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa537e00640 1 -- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa54006e620 msgr2=0x7fa540165040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa537e00640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa54006e620 0x7fa540165040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa537e00640 1 -- 172.21.15.40:0/23315119 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54014d470 con 0x7fa5400b2ff0 2024-03-24T00:32:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa53ea00640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa54006e620 0x7fa540165040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:39.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa537e00640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5400b2ff0 0x7fa54014e510 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa52800e870 tx=0x7fa52800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:39.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa537400640 1 -- 172.21.15.40:0/23315119 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa528017070 con 0x7fa5400b2ff0 2024-03-24T00:32:39.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa537400640 1 -- 172.21.15.40:0/23315119 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa52800c940 con 0x7fa5400b2ff0 2024-03-24T00:32:39.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa537400640 1 -- 172.21.15.40:0/23315119 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa52800ccf0 con 0x7fa5400b2ff0 2024-03-24T00:32:39.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa545e00640 1 -- 172.21.15.40:0/23315119 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54014d750 con 0x7fa5400b2ff0 2024-03-24T00:32:39.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa545e00640 1 -- 172.21.15.40:0/23315119 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54014dd10 con 0x7fa5400b2ff0 2024-03-24T00:32:39.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.884+0000 7fa545e00640 1 -- 172.21.15.40:0/23315119 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa540068170 con 0x7fa5400b2ff0 2024-03-24T00:32:39.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.888+0000 7fa537400640 1 -- 172.21.15.40:0/23315119 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa52801a070 con 0x7fa5400b2ff0 2024-03-24T00:32:39.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.888+0000 7fa537400640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa51c067690 0x7fa51c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:39.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.888+0000 7fa537400640 1 -- 172.21.15.40:0/23315119 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa528085ad0 con 0x7fa5400b2ff0 2024-03-24T00:32:39.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.888+0000 7fa53ea00640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa51c067690 0x7fa51c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:39.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.888+0000 7fa53ea00640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa51c067690 0x7fa51c069b50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa530005890 tx=0x7fa530009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:39.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:39.892+0000 7fa537400640 1 -- 172.21.15.40:0/23315119 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa528056060 con 0x7fa5400b2ff0 2024-03-24T00:32:40.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.104+0000 7fa545e00640 1 -- 172.21.15.40:0/23315119 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa540072f50 con 0x7fa5400b2ff0 2024-03-24T00:32:40.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.104+0000 7fa537400640 1 -- 172.21.15.40:0/23315119 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v33) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa52805a100 con 0x7fa5400b2ff0 2024-03-24T00:32:40.109 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:32:40.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.108+0000 7fa545e00640 1 -- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa51c067690 msgr2=0x7fa51c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:40.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.108+0000 7fa545e00640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa51c067690 0x7fa51c069b50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa530005890 tx=0x7fa530009040 comp rx=0 tx=0).stop 2024-03-24T00:32:40.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.108+0000 7fa545e00640 1 -- 172.21.15.40:0/23315119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5400b2ff0 msgr2=0x7fa54014e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:40.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.108+0000 7fa545e00640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5400b2ff0 0x7fa54014e510 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa52800e870 tx=0x7fa52800ec70 comp rx=0 tx=0).stop 2024-03-24T00:32:40.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.108+0000 7fa545e00640 1 -- 172.21.15.40:0/23315119 shutdown_connections 2024-03-24T00:32:40.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.108+0000 7fa545e00640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa51c067690 0x7fa51c069b50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:40.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.108+0000 7fa545e00640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5400b46c0 0x7fa54013dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:40.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.108+0000 7fa545e00640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5400b2ff0 0x7fa54014e510 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:40.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.108+0000 7fa545e00640 1 --2- 172.21.15.40:0/23315119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa54006e620 0x7fa540165040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:40.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.108+0000 7fa545e00640 1 -- 172.21.15.40:0/23315119 >> 172.21.15.40:0/23315119 conn(0x7fa5400b17b0 msgr2=0x7fa54015ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:40.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.108+0000 7fa545e00640 1 -- 172.21.15.40:0/23315119 shutdown_connections 2024-03-24T00:32:40.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.108+0000 7fa545e00640 1 -- 172.21.15.40:0/23315119 wait complete. 2024-03-24T00:32:40.130 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-03-24T00:32:40.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:40.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T00:32:40.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[37363]: fuse finished with error 0 and tester_r 0 2024-03-24T00:32:40.153 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:32:40.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:40.153 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:32:40.316 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:32:40.316 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:32:40.324 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:32:40.325 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:32:40.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:40.325 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T00:32:40.375 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T00:32:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:32:40.376 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:32:40.376 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:32:40.386 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:32:40.386 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:32:40.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:40.386 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T00:32:40.436 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T00:32:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:32:40.437 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:32:40.437 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:32:40.445 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:32:40.445 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:32:40.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:32:40.445 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T00:32:40.495 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T00:32:40.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:32:40.496 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:32:40.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb86e200640 1 Processor -- start 2024-03-24T00:32:40.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb86e200640 1 -- start start 2024-03-24T00:32:40.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb86e200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb8680d0830 0x7fb8680cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:40.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb86e200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8680d0e10 con 0x7fb86806b150 2024-03-24T00:32:40.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb86e200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86806efd0 con 0x7fb86815e0b0 2024-03-24T00:32:40.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb86e200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8680d10f0 con 0x7fb8680d0830 2024-03-24T00:32:40.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb867e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb86806b150 0x7fb86806b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38620/0 (socket says 172.21.15.40:38620) 2024-03-24T00:32:40.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb867e00640 1 -- 172.21.15.40:0/2652240630 learned_addr learned my addr 172.21.15.40:0/2652240630 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:40.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb85ea00640 1 --2- 172.21.15.40:0/2652240630 >> v2:172.21.15.53:3300/0 conn(0x7fb8680d0830 0x7fb8680cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:40.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb866a00640 1 -- 172.21.15.40:0/2652240630 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079822370 0 0) 0x7fb8680d0e10 con 0x7fb86806b150 2024-03-24T00:32:40.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb866a00640 1 -- 172.21.15.40:0/2652240630 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84c003530 con 0x7fb86806b150 2024-03-24T00:32:40.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb866a00640 1 -- 172.21.15.40:0/2652240630 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181105579 0 0) 0x7fb86806efd0 con 0x7fb86815e0b0 2024-03-24T00:32:40.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb85ea00640 1 -- 172.21.15.40:0/2652240630 >> v1:172.21.15.53:6789/0 conn(0x7fb86815e0b0 legacy=0x7fb8680cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:40.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb85ea00640 1 -- 172.21.15.40:0/2652240630 >> v1:172.21.15.40:6789/0 conn(0x7fb86806b150 legacy=0x7fb86806b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:40.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.636+0000 7fb85ea00640 1 -- 172.21.15.40:0/2652240630 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86806f2b0 con 0x7fb8680d0830 2024-03-24T00:32:40.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb85ea00640 1 --2- 172.21.15.40:0/2652240630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8680d0830 0x7fb8680cdae0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb8500097c0 tx=0x7fb85001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5df852a43c13780a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:40.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb866a00640 1 -- 172.21.15.40:0/2652240630 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8500b1070 con 0x7fb8680d0830 2024-03-24T00:32:40.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb866a00640 1 -- 172.21.15.40:0/2652240630 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8500ac410 con 0x7fb8680d0830 2024-03-24T00:32:40.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb866a00640 1 -- 172.21.15.40:0/2652240630 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8500ac7c0 con 0x7fb8680d0830 2024-03-24T00:32:40.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb86e200640 1 -- 172.21.15.40:0/2652240630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8680d0830 msgr2=0x7fb8680cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:40.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb86e200640 1 --2- 172.21.15.40:0/2652240630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8680d0830 0x7fb8680cdae0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb8500097c0 tx=0x7fb85001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:32:40.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb86e200640 1 -- 172.21.15.40:0/2652240630 shutdown_connections 2024-03-24T00:32:40.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb86e200640 1 --2- 172.21.15.40:0/2652240630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8680d0830 0x7fb8680cdae0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:40.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb86e200640 1 -- 172.21.15.40:0/2652240630 >> 172.21.15.40:0/2652240630 conn(0x7fb8680d1c90 msgr2=0x7fb8680d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:40.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb86e200640 1 -- 172.21.15.40:0/2652240630 shutdown_connections 2024-03-24T00:32:40.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb86e200640 1 -- 172.21.15.40:0/2652240630 wait complete. 2024-03-24T00:32:40.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb86e200640 1 Processor -- start 2024-03-24T00:32:40.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.644+0000 7fb86e200640 1 -- start start 2024-03-24T00:32:40.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb86e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb86806b150 0x7fb868151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:40.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb86e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8680d0830 0x7fb8680afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:40.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb86e200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb86815e0b0 0x7fb8680b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:40.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb86e200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86815e610 con 0x7fb86806b150 2024-03-24T00:32:40.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb86e200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86815e490 con 0x7fb8680d0830 2024-03-24T00:32:40.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb86e200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb868070ac0 con 0x7fb86815e0b0 2024-03-24T00:32:40.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb867400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb86806b150 0x7fb868151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:40.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb867400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb86806b150 0x7fb868151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49184/0 (socket says 172.21.15.40:49184) 2024-03-24T00:32:40.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb867e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb86815e0b0 0x7fb8680b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:40.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb867400640 1 -- 172.21.15.40:0/3179716031 learned_addr learned my addr 172.21.15.40:0/3179716031 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:40.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb85ea00640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8680d0830 0x7fb8680afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:40.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb867400640 1 -- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb86815e0b0 msgr2=0x7fb8680b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:40.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb867400640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb86815e0b0 0x7fb8680b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:40.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb867400640 1 -- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8680d0830 msgr2=0x7fb8680afe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:40.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb867400640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8680d0830 0x7fb8680afe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:40.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb867400640 1 -- 172.21.15.40:0/3179716031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8680c5ad0 con 0x7fb86806b150 2024-03-24T00:32:40.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb867e00640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb86815e0b0 0x7fb8680b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:40.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb85ea00640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8680d0830 0x7fb8680afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:40.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb867400640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb86806b150 0x7fb868151e60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb85800c7a0 tx=0x7fb85800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:40.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb85fe00640 1 -- 172.21.15.40:0/3179716031 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb858017070 con 0x7fb86806b150 2024-03-24T00:32:40.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb85fe00640 1 -- 172.21.15.40:0/3179716031 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb858003db0 con 0x7fb86806b150 2024-03-24T00:32:40.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb85fe00640 1 -- 172.21.15.40:0/3179716031 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb858005270 con 0x7fb86806b150 2024-03-24T00:32:40.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb86e200640 1 -- 172.21.15.40:0/3179716031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb868158b30 con 0x7fb86806b150 2024-03-24T00:32:40.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb86e200640 1 -- 172.21.15.40:0/3179716031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8680b1a00 con 0x7fb86806b150 2024-03-24T00:32:40.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.648+0000 7fb86e200640 1 -- 172.21.15.40:0/3179716031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb868068170 con 0x7fb86806b150 2024-03-24T00:32:40.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.652+0000 7fb85fe00640 1 -- 172.21.15.40:0/3179716031 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb85801a070 con 0x7fb86806b150 2024-03-24T00:32:40.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.652+0000 7fb85fe00640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb844067660 0x7fb844069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:40.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.652+0000 7fb85fe00640 1 -- 172.21.15.40:0/3179716031 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb858084de0 con 0x7fb86806b150 2024-03-24T00:32:40.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.652+0000 7fb85ea00640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb844067660 0x7fb844069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:40.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.652+0000 7fb85ea00640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb844067660 0x7fb844069b20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb8500ae000 tx=0x7fb850008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:40.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.660+0000 7fb85fe00640 1 -- 172.21.15.40:0/3179716031 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb858055400 con 0x7fb86806b150 2024-03-24T00:32:40.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.856+0000 7fb86e200640 1 -- 172.21.15.40:0/3179716031 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8680cfdf0 con 0x7fb86806b150 2024-03-24T00:32:40.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb85fe00640 1 -- 172.21.15.40:0/3179716031 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5015 (secure 0 0 0) 0x7fb8580594a0 con 0x7fb86806b150 2024-03-24T00:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:40.861 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2069836835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2069836835},{"type":"v1","addr":"172.21.15.53:6836","nonce":2069836835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:06.028086+0000","modified":"2024-03-24T00:32:20.288717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5229,"mds_1":5226,"mds_2":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"b","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.53:6837/2159849028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":2159849028},{"type":"v1","addr":"172.21.15.53:6837","nonce":2159849028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5226":{"gid":5226,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6833/2326157501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2326157501},{"type":"v1","addr":"172.21.15.40:6833","nonce":2326157501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5229":{"gid":5229,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3050269371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3050269371},{"type":"v1","addr":"172.21.15.40:6835","nonce":3050269371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5229,"qdb_cluster":[4856,5226,5229]},"id":2}]} 2024-03-24T00:32:40.861 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 21 2024-03-24T00:32:40.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb86e200640 1 -- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb844067660 msgr2=0x7fb844069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:40.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb86e200640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb844067660 0x7fb844069b20 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb8500ae000 tx=0x7fb850008040 comp rx=0 tx=0).stop 2024-03-24T00:32:40.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb86e200640 1 -- 172.21.15.40:0/3179716031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb86806b150 msgr2=0x7fb868151e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:40.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb86e200640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb86806b150 0x7fb868151e60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fb85800c7a0 tx=0x7fb85800cc60 comp rx=0 tx=0).stop 2024-03-24T00:32:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb86e200640 1 -- 172.21.15.40:0/3179716031 shutdown_connections 2024-03-24T00:32:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb86e200640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb844067660 0x7fb844069b20 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb86e200640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb86815e0b0 0x7fb8680b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb86e200640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8680d0830 0x7fb8680afe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb86e200640 1 --2- 172.21.15.40:0/3179716031 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb86806b150 0x7fb868151e60 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb86e200640 1 -- 172.21.15.40:0/3179716031 >> 172.21.15.40:0/3179716031 conn(0x7fb8680d1c90 msgr2=0x7fb8680b6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb86e200640 1 -- 172.21.15.40:0/3179716031 shutdown_connections 2024-03-24T00:32:40.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:40.860+0000 7fb86e200640 1 -- 172.21.15.40:0/3179716031 wait complete. 2024-03-24T00:32:40.891 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:32:41.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4dd600640 1 Processor -- start 2024-03-24T00:32:41.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4dd600640 1 -- start start 2024-03-24T00:32:41.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4dd600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe4d8072310 0x7fe4d80ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4dd600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d8070e00 con 0x7fe4d8141bc0 2024-03-24T00:32:41.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4dd600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d80cee20 con 0x7fe4d813ec80 2024-03-24T00:32:41.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4dd600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d80cf100 con 0x7fe4d8072310 2024-03-24T00:32:41.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4d6000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe4d8141bc0 0x7fe4d80ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38632/0 (socket says 172.21.15.40:38632) 2024-03-24T00:32:41.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4d6000640 1 -- 172.21.15.40:0/758577387 learned_addr learned my addr 172.21.15.40:0/758577387 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:41.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4d7400640 1 --2- 172.21.15.40:0/758577387 >> v2:172.21.15.53:3300/0 conn(0x7fe4d8072310 0x7fe4d80ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4d5600640 1 -- 172.21.15.40:0/758577387 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919443291 0 0) 0x7fe4d8070e00 con 0x7fe4d8141bc0 2024-03-24T00:32:41.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4d5600640 1 -- 172.21.15.40:0/758577387 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4bc003530 con 0x7fe4d8141bc0 2024-03-24T00:32:41.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4d5600640 1 -- 172.21.15.40:0/758577387 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972090698 0 0) 0x7fe4d80cee20 con 0x7fe4d813ec80 2024-03-24T00:32:41.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4d7400640 1 -- 172.21.15.40:0/758577387 >> v1:172.21.15.53:6789/0 conn(0x7fe4d813ec80 legacy=0x7fe4d81414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4d7400640 1 -- 172.21.15.40:0/758577387 >> v1:172.21.15.40:6789/0 conn(0x7fe4d8141bc0 legacy=0x7fe4d80ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.044+0000 7fe4d7400640 1 -- 172.21.15.40:0/758577387 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d80cf3e0 con 0x7fe4d8072310 2024-03-24T00:32:41.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.048+0000 7fe4d7400640 1 --2- 172.21.15.40:0/758577387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4d8072310 0x7fe4d80ce800 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009d90 tx=0x7fe4d001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e099911c01a8fe9d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:41.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.048+0000 7fe4d5600640 1 -- 172.21.15.40:0/758577387 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d00b2070 con 0x7fe4d8072310 2024-03-24T00:32:41.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.048+0000 7fe4d5600640 1 -- 172.21.15.40:0/758577387 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d00a5a50 con 0x7fe4d8072310 2024-03-24T00:32:41.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.048+0000 7fe4d5600640 1 -- 172.21.15.40:0/758577387 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d00a5e20 con 0x7fe4d8072310 2024-03-24T00:32:41.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.048+0000 7fe4dd600640 1 -- 172.21.15.40:0/758577387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4d8072310 msgr2=0x7fe4d80ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.048+0000 7fe4dd600640 1 --2- 172.21.15.40:0/758577387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4d8072310 0x7fe4d80ce800 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009d90 tx=0x7fe4d001c970 comp rx=0 tx=0).stop 2024-03-24T00:32:41.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.048+0000 7fe4dd600640 1 -- 172.21.15.40:0/758577387 shutdown_connections 2024-03-24T00:32:41.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.048+0000 7fe4dd600640 1 --2- 172.21.15.40:0/758577387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4d8072310 0x7fe4d80ce800 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.048+0000 7fe4dd600640 1 -- 172.21.15.40:0/758577387 >> 172.21.15.40:0/758577387 conn(0x7fe4d80b17b0 msgr2=0x7fe4d80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:41.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.048+0000 7fe4dd600640 1 -- 172.21.15.40:0/758577387 shutdown_connections 2024-03-24T00:32:41.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.048+0000 7fe4dd600640 1 -- 172.21.15.40:0/758577387 wait complete. 2024-03-24T00:32:41.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4dd600640 1 Processor -- start 2024-03-24T00:32:41.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4dd600640 1 -- start start 2024-03-24T00:32:41.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4dd600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4d8072310 0x7fe4d8139e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4dd600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d813ec80 0x7fe4d8071300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4dd600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4d8141bc0 0x7fe4d8071840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4dd600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d8169f00 con 0x7fe4d813ec80 2024-03-24T00:32:41.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4dd600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d8169d80 con 0x7fe4d8141bc0 2024-03-24T00:32:41.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4dd600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d816a080 con 0x7fe4d8072310 2024-03-24T00:32:41.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d813ec80 0x7fe4d8071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4d8141bc0 0x7fe4d8071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d6a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4d8072310 0x7fe4d8139e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4d8141bc0 0x7fe4d8071840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:52328/0 (socket says 172.21.15.40:52328) 2024-03-24T00:32:41.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d813ec80 0x7fe4d8071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49194/0 (socket says 172.21.15.40:49194) 2024-03-24T00:32:41.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d7400640 1 -- 172.21.15.40:0/3237674334 learned_addr learned my addr 172.21.15.40:0/3237674334 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d7400640 1 -- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4d8072310 msgr2=0x7fe4d8139e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d7400640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4d8072310 0x7fe4d8139e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d7400640 1 -- 172.21.15.40:0/3237674334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d813ec80 msgr2=0x7fe4d8071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d7400640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d813ec80 0x7fe4d8071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d7400640 1 -- 172.21.15.40:0/3237674334 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d813a3c0 con 0x7fe4d8141bc0 2024-03-24T00:32:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d6000640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d813ec80 0x7fe4d8071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4d7400640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4d8141bc0 0x7fe4d8071840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009d90 tx=0x7fe4d0019710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4cea00640 1 -- 172.21.15.40:0/3237674334 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d00b2040 con 0x7fe4d8141bc0 2024-03-24T00:32:41.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4cea00640 1 -- 172.21.15.40:0/3237674334 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d00adaf0 con 0x7fe4d8141bc0 2024-03-24T00:32:41.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4cea00640 1 -- 172.21.15.40:0/3237674334 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d0009450 con 0x7fe4d8141bc0 2024-03-24T00:32:41.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4dd600640 1 -- 172.21.15.40:0/3237674334 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d8156c80 con 0x7fe4d8141bc0 2024-03-24T00:32:41.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.052+0000 7fe4dd600640 1 -- 172.21.15.40:0/3237674334 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d8157240 con 0x7fe4d8141bc0 2024-03-24T00:32:41.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.056+0000 7fe4dd600640 1 -- 172.21.15.40:0/3237674334 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d8068170 con 0x7fe4d8141bc0 2024-03-24T00:32:41.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.056+0000 7fe4cea00640 1 -- 172.21.15.40:0/3237674334 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe4d00039a0 con 0x7fe4d8141bc0 2024-03-24T00:32:41.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.056+0000 7fe4cea00640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe4b8067660 0x7fe4b8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.056+0000 7fe4cea00640 1 -- 172.21.15.40:0/3237674334 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4d0120a10 con 0x7fe4d8141bc0 2024-03-24T00:32:41.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.056+0000 7fe4d6a00640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe4b8067660 0x7fe4b8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.056+0000 7fe4d6a00640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe4b8067660 0x7fe4b8069b20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8000f70 tx=0x7fe4c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:41.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.060+0000 7fe4cea00640 1 -- 172.21.15.40:0/3237674334 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe4d0016560 con 0x7fe4d8141bc0 2024-03-24T00:32:41.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.224+0000 7fe4dd600640 1 -- 172.21.15.40:0/3237674334 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4d8072b90 con 0x7fe4d8141bc0 2024-03-24T00:32:41.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4cea00640 1 -- 172.21.15.40:0/3237674334 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16609 (secure 0 0 0) 0x7fe4d00f1150 con 0x7fe4d8141bc0 2024-03-24T00:32:41.229 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:41.230 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":33,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:32:07.623938+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:03.863519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:05.024936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:32:41.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4dd600640 1 -- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe4b8067660 msgr2=0x7fe4b8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4dd600640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe4b8067660 0x7fe4b8069b20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe4c8000f70 tx=0x7fe4c8009040 comp rx=0 tx=0).stop 2024-03-24T00:32:41.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4dd600640 1 -- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4d8141bc0 msgr2=0x7fe4d8071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4dd600640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4d8141bc0 0x7fe4d8071840 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe4d0009d90 tx=0x7fe4d0019710 comp rx=0 tx=0).stop 2024-03-24T00:32:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4dd600640 1 -- 172.21.15.40:0/3237674334 shutdown_connections 2024-03-24T00:32:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4dd600640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe4b8067660 0x7fe4b8069b20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4dd600640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4d8141bc0 0x7fe4d8071840 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4dd600640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4d813ec80 0x7fe4d8071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4dd600640 1 --2- 172.21.15.40:0/3237674334 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4d8072310 0x7fe4d8139e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4dd600640 1 -- 172.21.15.40:0/3237674334 >> 172.21.15.40:0/3237674334 conn(0x7fe4d80b17b0 msgr2=0x7fe4d815ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4dd600640 1 -- 172.21.15.40:0/3237674334 shutdown_connections 2024-03-24T00:32:41.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.228+0000 7fe4dd600640 1 -- 172.21.15.40:0/3237674334 wait complete. 2024-03-24T00:32:41.260 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:32:03.863519+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:32:05.024936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:32:41.260 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:32:41.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.408+0000 7fab19600640 1 Processor -- start 2024-03-24T00:32:41.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.408+0000 7fab19600640 1 -- start start 2024-03-24T00:32:41.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.408+0000 7fab19600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fab1415bd00 0x7fab140ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.408+0000 7fab19600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab1415ffd0 0x7fab140cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.408+0000 7fab19600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab14072d60 con 0x7fab140728d0 2024-03-24T00:32:41.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.408+0000 7fab19600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab140cf790 con 0x7fab1415bd00 2024-03-24T00:32:41.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.408+0000 7fab19600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab14071de0 con 0x7fab1415ffd0 2024-03-24T00:32:41.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.408+0000 7fab12a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fab1415bd00 0x7fab140ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.408+0000 7fab0be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fab140728d0 0x7fab1415f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38646/0 (socket says 172.21.15.40:38646) 2024-03-24T00:32:41.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.408+0000 7fab0be00640 1 -- 172.21.15.40:0/204800875 learned_addr learned my addr 172.21.15.40:0/204800875 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:41.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.408+0000 7fab13400640 1 --2- 172.21.15.40:0/204800875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab1415ffd0 0x7fab140cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.412+0000 7fab12000640 1 -- 172.21.15.40:0/204800875 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265236206 0 0) 0x7fab14072d60 con 0x7fab140728d0 2024-03-24T00:32:41.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.412+0000 7fab12a00640 1 -- 172.21.15.40:0/204800875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab1415ffd0 msgr2=0x7fab140cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:41.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.412+0000 7fab12a00640 1 --2- 172.21.15.40:0/204800875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab1415ffd0 0x7fab140cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.412+0000 7fab12a00640 1 -- 172.21.15.40:0/204800875 >> v1:172.21.15.40:6789/0 conn(0x7fab140728d0 legacy=0x7fab1415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.412+0000 7fab12a00640 1 -- 172.21.15.40:0/204800875 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab140cea50 con 0x7fab1415bd00 2024-03-24T00:32:41.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab12a00640 1 --2- 172.21.15.40:0/204800875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1415bd00 0x7fab140ae8e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fab040097c0 tx=0x7fab0401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95972c559964f633 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:41.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab12000640 1 -- 172.21.15.40:0/204800875 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab040a7070 con 0x7fab1415bd00 2024-03-24T00:32:41.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab12000640 1 -- 172.21.15.40:0/204800875 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab040b0070 con 0x7fab1415bd00 2024-03-24T00:32:41.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab12000640 1 -- 172.21.15.40:0/204800875 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab040ab890 con 0x7fab1415bd00 2024-03-24T00:32:41.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab19600640 1 -- 172.21.15.40:0/204800875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1415bd00 msgr2=0x7fab140ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab19600640 1 --2- 172.21.15.40:0/204800875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1415bd00 0x7fab140ae8e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fab040097c0 tx=0x7fab0401c470 comp rx=0 tx=0).stop 2024-03-24T00:32:41.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab19600640 1 -- 172.21.15.40:0/204800875 shutdown_connections 2024-03-24T00:32:41.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab19600640 1 --2- 172.21.15.40:0/204800875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab1415ffd0 0x7fab140cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab19600640 1 --2- 172.21.15.40:0/204800875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1415bd00 0x7fab140ae8e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab19600640 1 -- 172.21.15.40:0/204800875 >> 172.21.15.40:0/204800875 conn(0x7fab140b1660 msgr2=0x7fab140b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:41.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab19600640 1 -- 172.21.15.40:0/204800875 shutdown_connections 2024-03-24T00:32:41.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab19600640 1 -- 172.21.15.40:0/204800875 wait complete. 2024-03-24T00:32:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab19600640 1 Processor -- start 2024-03-24T00:32:41.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.416+0000 7fab19600640 1 -- start start 2024-03-24T00:32:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab19600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab140728d0 0x7fab140b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab19600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab1415bd00 0x7fab1414e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab19600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1415ffd0 0x7fab1414ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab19600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1406d320 con 0x7fab140728d0 2024-03-24T00:32:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab19600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1415c0e0 con 0x7fab1415ffd0 2024-03-24T00:32:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab19600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab1406d4a0 con 0x7fab1415bd00 2024-03-24T00:32:41.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab12a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab140728d0 0x7fab140b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab12a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab140728d0 0x7fab140b2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49202/0 (socket says 172.21.15.40:49202) 2024-03-24T00:32:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab12a00640 1 -- 172.21.15.40:0/1350743780 learned_addr learned my addr 172.21.15.40:0/1350743780 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab0be00640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab1415bd00 0x7fab1414e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab13400640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1415ffd0 0x7fab1414ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab12a00640 1 -- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab1415bd00 msgr2=0x7fab1414e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab12a00640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab1415bd00 0x7fab1414e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab12a00640 1 -- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1415ffd0 msgr2=0x7fab1414ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab12a00640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1415ffd0 0x7fab1414ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab12a00640 1 -- 172.21.15.40:0/1350743780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab141391f0 con 0x7fab140728d0 2024-03-24T00:32:41.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab12a00640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab140728d0 0x7fab140b2a60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fab040ad040 tx=0x7fab0401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab0aa00640 1 -- 172.21.15.40:0/1350743780 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab040abe60 con 0x7fab140728d0 2024-03-24T00:32:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab0aa00640 1 -- 172.21.15.40:0/1350743780 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab04004220 con 0x7fab140728d0 2024-03-24T00:32:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab19600640 1 -- 172.21.15.40:0/1350743780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab141394d0 con 0x7fab140728d0 2024-03-24T00:32:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab0aa00640 1 -- 172.21.15.40:0/1350743780 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab04003620 con 0x7fab140728d0 2024-03-24T00:32:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab19600640 1 -- 172.21.15.40:0/1350743780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab14139a90 con 0x7fab140728d0 2024-03-24T00:32:41.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.420+0000 7fab19600640 1 -- 172.21.15.40:0/1350743780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab14068170 con 0x7fab140728d0 2024-03-24T00:32:41.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.424+0000 7fab0aa00640 1 -- 172.21.15.40:0/1350743780 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fab040037c0 con 0x7fab140728d0 2024-03-24T00:32:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.424+0000 7fab0aa00640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faaf0067660 0x7faaf0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.424+0000 7fab0aa00640 1 -- 172.21.15.40:0/1350743780 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab0411ffb0 con 0x7fab140728d0 2024-03-24T00:32:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.424+0000 7fab0be00640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faaf0067660 0x7faaf0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.424+0000 7fab0be00640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faaf0067660 0x7faaf0069b20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fab14150020 tx=0x7faafc007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:41.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.428+0000 7fab0aa00640 1 -- 172.21.15.40:0/1350743780 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fab04016090 con 0x7fab140728d0 2024-03-24T00:32:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.592+0000 7fab19600640 1 -- 172.21.15.40:0/1350743780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab1415e6b0 con 0x7fab140728d0 2024-03-24T00:32:41.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.592+0000 7fab0aa00640 1 -- 172.21.15.40:0/1350743780 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v33) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab040f0670 con 0x7fab140728d0 2024-03-24T00:32:41.595 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:32:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.596+0000 7fab19600640 1 -- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faaf0067660 msgr2=0x7faaf0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.596+0000 7fab19600640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faaf0067660 0x7faaf0069b20 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fab14150020 tx=0x7faafc007d60 comp rx=0 tx=0).stop 2024-03-24T00:32:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.596+0000 7fab19600640 1 -- 172.21.15.40:0/1350743780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab140728d0 msgr2=0x7fab140b2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.596+0000 7fab19600640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab140728d0 0x7fab140b2a60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fab040ad040 tx=0x7fab0401acb0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.596+0000 7fab19600640 1 -- 172.21.15.40:0/1350743780 shutdown_connections 2024-03-24T00:32:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.596+0000 7fab19600640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faaf0067660 0x7faaf0069b20 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.596+0000 7fab19600640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab1415ffd0 0x7fab1414ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.596+0000 7fab19600640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab1415bd00 0x7fab1414e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.596+0000 7fab19600640 1 --2- 172.21.15.40:0/1350743780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab140728d0 0x7fab140b2a60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.596+0000 7fab19600640 1 -- 172.21.15.40:0/1350743780 >> 172.21.15.40:0/1350743780 conn(0x7fab140b1660 msgr2=0x7fab14154b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.596+0000 7fab19600640 1 -- 172.21.15.40:0/1350743780 shutdown_connections 2024-03-24T00:32:41.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.596+0000 7fab19600640 1 -- 172.21.15.40:0/1350743780 wait complete. 2024-03-24T00:32:41.616 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:32:41.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.756+0000 7f9340c00640 1 Processor -- start 2024-03-24T00:32:41.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.756+0000 7f9340c00640 1 -- start start 2024-03-24T00:32:41.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.760+0000 7f9340c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f933c15e350 0x7f933c0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.760+0000 7f9340c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0734e0 con 0x7f933c0c9580 2024-03-24T00:32:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.760+0000 7f9340c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f933c0d0030 con 0x7f933c0ad8f0 2024-03-24T00:32:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.760+0000 7f9340c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933c0cd660 con 0x7f933c15e350 2024-03-24T00:32:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.760+0000 7f9339e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f933c15e350 0x7f933c0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.760+0000 7f9333e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f933c0c9580 0x7f933c0ad1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38648/0 (socket says 172.21.15.40:38648) 2024-03-24T00:32:41.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.760+0000 7f9333e00640 1 -- 172.21.15.40:0/2447970127 learned_addr learned my addr 172.21.15.40:0/2447970127 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.760+0000 7f9339e00640 1 -- 172.21.15.40:0/2447970127 >> v1:172.21.15.53:6789/0 conn(0x7f933c0ad8f0 legacy=0x7f933c168d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.760+0000 7f9339e00640 1 -- 172.21.15.40:0/2447970127 >> v1:172.21.15.40:6789/0 conn(0x7f933c0c9580 legacy=0x7f933c0ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.760+0000 7f9339e00640 1 -- 172.21.15.40:0/2447970127 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c0745a0 con 0x7f933c15e350 2024-03-24T00:32:41.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.760+0000 7f9339400640 1 -- 172.21.15.40:0/2447970127 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241400821 0 0) 0x7f933c0d0030 con 0x7f933c0ad8f0 2024-03-24T00:32:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9339e00640 1 --2- 172.21.15.40:0/2447970127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f933c15e350 0x7f933c0cfaf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f932c0097c0 tx=0x7f932c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20542037e4ce5b00 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9339400640 1 -- 172.21.15.40:0/2447970127 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c0b1070 con 0x7f933c15e350 2024-03-24T00:32:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9339400640 1 -- 172.21.15.40:0/2447970127 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932c0ac410 con 0x7f933c15e350 2024-03-24T00:32:41.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9339400640 1 -- 172.21.15.40:0/2447970127 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c0ac7e0 con 0x7f933c15e350 2024-03-24T00:32:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9340c00640 1 -- 172.21.15.40:0/2447970127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f933c15e350 msgr2=0x7f933c0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9340c00640 1 --2- 172.21.15.40:0/2447970127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f933c15e350 0x7f933c0cfaf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f932c0097c0 tx=0x7f932c01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9340c00640 1 -- 172.21.15.40:0/2447970127 shutdown_connections 2024-03-24T00:32:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9340c00640 1 --2- 172.21.15.40:0/2447970127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f933c15e350 0x7f933c0cfaf0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9340c00640 1 -- 172.21.15.40:0/2447970127 >> 172.21.15.40:0/2447970127 conn(0x7f933c072870 msgr2=0x7f933c072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:41.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9340c00640 1 -- 172.21.15.40:0/2447970127 shutdown_connections 2024-03-24T00:32:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9340c00640 1 -- 172.21.15.40:0/2447970127 wait complete. 2024-03-24T00:32:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9340c00640 1 Processor -- start 2024-03-24T00:32:41.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.764+0000 7f9340c00640 1 -- start start 2024-03-24T00:32:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9340c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f933c0ad8f0 0x7f933c06de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9340c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f933c0c9580 0x7f933c152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9340c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f933c15e350 0x7f933c0b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9340c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0cda30 con 0x7f933c0ad8f0 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9340c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0716b0 con 0x7f933c0c9580 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9340c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f933c0cdbb0 con 0x7f933c15e350 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9339e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f933c0ad8f0 0x7f933c06de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9339e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f933c0ad8f0 0x7f933c06de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49208/0 (socket says 172.21.15.40:49208) 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9339e00640 1 -- 172.21.15.40:0/52486142 learned_addr learned my addr 172.21.15.40:0/52486142 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9333e00640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f933c0c9580 0x7f933c152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f933a800640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f933c15e350 0x7f933c0b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9339e00640 1 -- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f933c15e350 msgr2=0x7f933c0b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9339e00640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f933c15e350 0x7f933c0b5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9339e00640 1 -- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f933c0c9580 msgr2=0x7f933c152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9339e00640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f933c0c9580 0x7f933c152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9339e00640 1 -- 172.21.15.40:0/52486142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933c0cbac0 con 0x7f933c0ad8f0 2024-03-24T00:32:41.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9339e00640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f933c0ad8f0 0x7f933c06de30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f932c0190b0 tx=0x7f932c019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9332000640 1 -- 172.21.15.40:0/52486142 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c0b1040 con 0x7f933c0ad8f0 2024-03-24T00:32:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9332000640 1 -- 172.21.15.40:0/52486142 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932c0bb070 con 0x7f933c0ad8f0 2024-03-24T00:32:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9332000640 1 -- 172.21.15.40:0/52486142 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c0b5630 con 0x7f933c0ad8f0 2024-03-24T00:32:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9340c00640 1 -- 172.21.15.40:0/52486142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933c0cbda0 con 0x7f933c0ad8f0 2024-03-24T00:32:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9340c00640 1 -- 172.21.15.40:0/52486142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c0cc360 con 0x7f933c0ad8f0 2024-03-24T00:32:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9340c00640 1 -- 172.21.15.40:0/52486142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933c068170 con 0x7f933c0ad8f0 2024-03-24T00:32:41.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.768+0000 7f9332000640 1 -- 172.21.15.40:0/52486142 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f932c0b57d0 con 0x7f933c0ad8f0 2024-03-24T00:32:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.772+0000 7f9332000640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9318067660 0x7f9318069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.772+0000 7f9332000640 1 -- 172.21.15.40:0/52486142 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f932c11f250 con 0x7f933c0ad8f0 2024-03-24T00:32:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.772+0000 7f9333e00640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9318067660 0x7f9318069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:41.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.772+0000 7f9333e00640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9318067660 0x7f9318069b20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f933c15e0c0 tx=0x7f9324009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:41.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.776+0000 7f9332000640 1 -- 172.21.15.40:0/52486142 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f932c016090 con 0x7f933c0ad8f0 2024-03-24T00:32:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.936+0000 7f9340c00640 1 -- 172.21.15.40:0/52486142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f933c159540 con 0x7f933c0ad8f0 2024-03-24T00:32:41.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.936+0000 7f9332000640 1 -- 172.21.15.40:0/52486142 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+11 (secure 0 0 0) 0x7f932c0ef7e0 con 0x7f933c0ad8f0 2024-03-24T00:32:41.940 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:32:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.940+0000 7f9340c00640 1 -- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9318067660 msgr2=0x7f9318069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.940+0000 7f9340c00640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9318067660 0x7f9318069b20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f933c15e0c0 tx=0x7f9324009040 comp rx=0 tx=0).stop 2024-03-24T00:32:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.940+0000 7f9340c00640 1 -- 172.21.15.40:0/52486142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f933c0ad8f0 msgr2=0x7f933c06de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.940+0000 7f9340c00640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f933c0ad8f0 0x7f933c06de30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f932c0190b0 tx=0x7f932c019240 comp rx=0 tx=0).stop 2024-03-24T00:32:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.940+0000 7f9340c00640 1 -- 172.21.15.40:0/52486142 shutdown_connections 2024-03-24T00:32:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.940+0000 7f9340c00640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9318067660 0x7f9318069b20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.940+0000 7f9340c00640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f933c15e350 0x7f933c0b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.940+0000 7f9340c00640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f933c0c9580 0x7f933c152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.940+0000 7f9340c00640 1 --2- 172.21.15.40:0/52486142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f933c0ad8f0 0x7f933c06de30 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.940+0000 7f9340c00640 1 -- 172.21.15.40:0/52486142 >> 172.21.15.40:0/52486142 conn(0x7f933c072870 msgr2=0x7f933c169b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.940+0000 7f9340c00640 1 -- 172.21.15.40:0/52486142 shutdown_connections 2024-03-24T00:32:41.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:41.940+0000 7f9340c00640 1 -- 172.21.15.40:0/52486142 wait complete. 2024-03-24T00:32:41.962 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:32:42.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.112+0000 7f2687200640 1 Processor -- start 2024-03-24T00:32:42.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.112+0000 7f2687200640 1 -- start start 2024-03-24T00:32:42.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f2687200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f26801628d0 0x7f26800d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f2687200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26800b42d0 0x7f268006dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f2687200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268015e3e0 con 0x7f268015e000 2024-03-24T00:32:42.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f2687200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2680161ba0 con 0x7f26801628d0 2024-03-24T00:32:42.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f2687200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26800b5f10 con 0x7f26800b42d0 2024-03-24T00:32:42.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f267fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f268015e000 0x7f26800b3bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:45972/0 (socket says 172.21.15.40:45972) 2024-03-24T00:32:42.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f267fe00640 1 -- 172.21.15.40:0/1553902460 learned_addr learned my addr 172.21.15.40:0/1553902460 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:42.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f2685200640 1 --2- 172.21.15.40:0/1553902460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26800b42d0 0x7f268006dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f2677e00640 1 --2- 172.21.15.40:0/1553902460 >> v2:172.21.15.40:3300/0 conn(0x7f26801628d0 0x7f26800d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f267f400640 1 -- 172.21.15.40:0/1553902460 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274209363 0 0) 0x7f268015e3e0 con 0x7f268015e000 2024-03-24T00:32:42.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f267f400640 1 -- 172.21.15.40:0/1553902460 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2664003530 con 0x7f268015e000 2024-03-24T00:32:42.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f2685200640 1 -- 172.21.15.40:0/1553902460 >> v2:172.21.15.40:3300/0 conn(0x7f26801628d0 msgr2=0x7f26800d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:42.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f2685200640 1 --2- 172.21.15.40:0/1553902460 >> v2:172.21.15.40:3300/0 conn(0x7f26801628d0 0x7f26800d16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f2685200640 1 -- 172.21.15.40:0/1553902460 >> v1:172.21.15.53:6789/0 conn(0x7f268015e000 legacy=0x7f26800b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.116+0000 7f2685200640 1 -- 172.21.15.40:0/1553902460 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680161df0 con 0x7f26800b42d0 2024-03-24T00:32:42.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f2685200640 1 --2- 172.21.15.40:0/1553902460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26800b42d0 0x7f268006dcb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2678009d90 tx=0x7f267801c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f738b168792f916e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:42.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f267f400640 1 -- 172.21.15.40:0/1553902460 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26780ac070 con 0x7f26800b42d0 2024-03-24T00:32:42.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f267f400640 1 -- 172.21.15.40:0/1553902460 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26780a6640 con 0x7f26800b42d0 2024-03-24T00:32:42.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f267f400640 1 -- 172.21.15.40:0/1553902460 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26780a6a10 con 0x7f26800b42d0 2024-03-24T00:32:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f2687200640 1 -- 172.21.15.40:0/1553902460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26800b42d0 msgr2=0x7f268006dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f2687200640 1 --2- 172.21.15.40:0/1553902460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26800b42d0 0x7f268006dcb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2678009d90 tx=0x7f267801c940 comp rx=0 tx=0).stop 2024-03-24T00:32:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f2687200640 1 -- 172.21.15.40:0/1553902460 shutdown_connections 2024-03-24T00:32:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f2687200640 1 --2- 172.21.15.40:0/1553902460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26800b42d0 0x7f268006dcb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f2687200640 1 --2- 172.21.15.40:0/1553902460 >> v2:172.21.15.40:3300/0 conn(0x7f26801628d0 0x7f26800d16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f2687200640 1 -- 172.21.15.40:0/1553902460 >> 172.21.15.40:0/1553902460 conn(0x7f26800d0bb0 msgr2=0x7f26800d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f2687200640 1 -- 172.21.15.40:0/1553902460 shutdown_connections 2024-03-24T00:32:42.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f2687200640 1 -- 172.21.15.40:0/1553902460 wait complete. 2024-03-24T00:32:42.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f2687200640 1 Processor -- start 2024-03-24T00:32:42.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.120+0000 7f2687200640 1 -- start start 2024-03-24T00:32:42.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f2687200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26800b42d0 0x7f268015bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f2687200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f268015e000 0x7f268015bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f2687200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26801628d0 0x7f268015e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f2687200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268016b0c0 con 0x7f26800b42d0 2024-03-24T00:32:42.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f2687200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268016af40 con 0x7f26801628d0 2024-03-24T00:32:42.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f2687200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268016b240 con 0x7f268015e000 2024-03-24T00:32:42.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26800b42d0 0x7f268015bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26800b42d0 0x7f268015bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49226/0 (socket says 172.21.15.40:49226) 2024-03-24T00:32:42.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267fe00640 1 -- 172.21.15.40:0/3562068659 learned_addr learned my addr 172.21.15.40:0/3562068659 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:42.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f2685200640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26801628d0 0x7f268015e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f2677e00640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f268015e000 0x7f268015bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267fe00640 1 -- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f268015e000 msgr2=0x7f268015bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267fe00640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f268015e000 0x7f268015bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267fe00640 1 -- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26801628d0 msgr2=0x7f268015e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267fe00640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26801628d0 0x7f268015e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267fe00640 1 -- 172.21.15.40:0/3562068659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680150e10 con 0x7f26800b42d0 2024-03-24T00:32:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267fe00640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26800b42d0 0x7f268015bab0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f267000c450 tx=0x7f267000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267cc00640 1 -- 172.21.15.40:0/3562068659 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2670017070 con 0x7f26800b42d0 2024-03-24T00:32:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267cc00640 1 -- 172.21.15.40:0/3562068659 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2670005cf0 con 0x7f26800b42d0 2024-03-24T00:32:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267cc00640 1 -- 172.21.15.40:0/3562068659 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2670007290 con 0x7f26800b42d0 2024-03-24T00:32:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f2687200640 1 -- 172.21.15.40:0/3562068659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268014f6f0 con 0x7f26800b42d0 2024-03-24T00:32:42.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f2687200640 1 -- 172.21.15.40:0/3562068659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268014fcb0 con 0x7f26800b42d0 2024-03-24T00:32:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f2687200640 1 -- 172.21.15.40:0/3562068659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2680068170 con 0x7f26800b42d0 2024-03-24T00:32:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.124+0000 7f267cc00640 1 -- 172.21.15.40:0/3562068659 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f267001a070 con 0x7f26800b42d0 2024-03-24T00:32:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.128+0000 7f267cc00640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f265c067690 0x7f265c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.128+0000 7f267cc00640 1 -- 172.21.15.40:0/3562068659 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2670084c00 con 0x7f26800b42d0 2024-03-24T00:32:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.128+0000 7f2677e00640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f265c067690 0x7f265c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.128+0000 7f2677e00640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f265c067690 0x7f265c069b50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f26800cfd80 tx=0x7f2668009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:42.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.132+0000 7f267cc00640 1 -- 172.21.15.40:0/3562068659 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2670055190 con 0x7f26800b42d0 2024-03-24T00:32:42.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.292+0000 7f2687200640 1 -- 172.21.15.40:0/3562068659 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f26801611f0 con 0x7f26800b42d0 2024-03-24T00:32:42.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f267cc00640 1 -- 172.21.15.40:0/3562068659 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2670059230 con 0x7f26800b42d0 2024-03-24T00:32:42.297 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:32:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f2687200640 1 -- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f265c067690 msgr2=0x7f265c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f2687200640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f265c067690 0x7f265c069b50 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f26800cfd80 tx=0x7f2668009190 comp rx=0 tx=0).stop 2024-03-24T00:32:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f2687200640 1 -- 172.21.15.40:0/3562068659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26800b42d0 msgr2=0x7f268015bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f2687200640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26800b42d0 0x7f268015bab0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f267000c450 tx=0x7f267000c910 comp rx=0 tx=0).stop 2024-03-24T00:32:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f2687200640 1 -- 172.21.15.40:0/3562068659 shutdown_connections 2024-03-24T00:32:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f2687200640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f265c067690 0x7f265c069b50 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f2687200640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26801628d0 0x7f268015e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f2687200640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f268015e000 0x7f268015bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f2687200640 1 --2- 172.21.15.40:0/3562068659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f26800b42d0 0x7f268015bab0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f2687200640 1 -- 172.21.15.40:0/3562068659 >> 172.21.15.40:0/3562068659 conn(0x7f26800d0bb0 msgr2=0x7f26800cfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:42.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f2687200640 1 -- 172.21.15.40:0/3562068659 shutdown_connections 2024-03-24T00:32:42.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.296+0000 7f2687200640 1 -- 172.21.15.40:0/3562068659 wait complete. 2024-03-24T00:32:42.317 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:32:42.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.456+0000 7fab3d000640 1 Processor -- start 2024-03-24T00:32:42.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.456+0000 7fab3d000640 1 -- start start 2024-03-24T00:32:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.456+0000 7fab3d000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab3806e620 0x7fab380b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.456+0000 7fab3d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fab380ccfa0 0x7fab380cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.456+0000 7fab3d000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab38070f60 con 0x7fab3806ea00 2024-03-24T00:32:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.456+0000 7fab3d000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab380b4c00 con 0x7fab3806e620 2024-03-24T00:32:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.456+0000 7fab3d000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab380b3620 con 0x7fab380ccfa0 2024-03-24T00:32:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.456+0000 7fab35e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab3806e620 0x7fab380b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.456+0000 7fab35e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab3806e620 0x7fab380b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49238/0 (socket says 172.21.15.40:49238) 2024-03-24T00:32:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.456+0000 7fab35e00640 1 -- 172.21.15.40:0/3002121524 learned_addr learned my addr 172.21.15.40:0/3002121524 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.456+0000 7fab2d400640 1 --1- 172.21.15.40:0/3002121524 >> v1:172.21.15.40:6789/0 conn(0x7fab3806ea00 0x7fab380b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38660/0 (socket says 172.21.15.40:38660) 2024-03-24T00:32:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.456+0000 7fab36800640 1 --2- 172.21.15.40:0/3002121524 >> v2:172.21.15.53:3300/0 conn(0x7fab380ccfa0 0x7fab380cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.460+0000 7fab35e00640 1 -- 172.21.15.40:0/3002121524 >> v2:172.21.15.53:3300/0 conn(0x7fab380ccfa0 msgr2=0x7fab380cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.460+0000 7fab35e00640 1 --2- 172.21.15.40:0/3002121524 >> v2:172.21.15.53:3300/0 conn(0x7fab380ccfa0 0x7fab380cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.460+0000 7fab35e00640 1 -- 172.21.15.40:0/3002121524 >> v1:172.21.15.40:6789/0 conn(0x7fab3806ea00 legacy=0x7fab380b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.460+0000 7fab35e00640 1 -- 172.21.15.40:0/3002121524 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab380b3900 con 0x7fab3806e620 2024-03-24T00:32:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.460+0000 7fab35400640 1 -- 172.21.15.40:0/3002121524 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516524816 0 0) 0x7fab38070f60 con 0x7fab3806ea00 2024-03-24T00:32:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.460+0000 7fab36800640 1 --2- 172.21.15.40:0/3002121524 >> v2:172.21.15.53:3300/0 conn(0x7fab380ccfa0 0x7fab380cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab35e00640 1 --2- 172.21.15.40:0/3002121524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab3806e620 0x7fab380b46c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fab38067e30 tx=0x7fab2801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97e3c400f52592a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab35400640 1 -- 172.21.15.40:0/3002121524 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab280b0070 con 0x7fab3806e620 2024-03-24T00:32:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab35400640 1 -- 172.21.15.40:0/3002121524 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab280ab600 con 0x7fab3806e620 2024-03-24T00:32:42.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab35400640 1 -- 172.21.15.40:0/3002121524 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab280ab9d0 con 0x7fab3806e620 2024-03-24T00:32:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab3d000640 1 -- 172.21.15.40:0/3002121524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab3806e620 msgr2=0x7fab380b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab3d000640 1 --2- 172.21.15.40:0/3002121524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab3806e620 0x7fab380b46c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fab38067e30 tx=0x7fab2801c460 comp rx=0 tx=0).stop 2024-03-24T00:32:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab3d000640 1 -- 172.21.15.40:0/3002121524 shutdown_connections 2024-03-24T00:32:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab3d000640 1 --2- 172.21.15.40:0/3002121524 >> v2:172.21.15.53:3300/0 conn(0x7fab380ccfa0 0x7fab380cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab3d000640 1 --2- 172.21.15.40:0/3002121524 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab3806e620 0x7fab380b46c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab3d000640 1 -- 172.21.15.40:0/3002121524 >> 172.21.15.40:0/3002121524 conn(0x7fab380b17b0 msgr2=0x7fab380b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab3d000640 1 -- 172.21.15.40:0/3002121524 shutdown_connections 2024-03-24T00:32:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab3d000640 1 -- 172.21.15.40:0/3002121524 wait complete. 2024-03-24T00:32:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab3d000640 1 Processor -- start 2024-03-24T00:32:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.464+0000 7fab3d000640 1 -- start start 2024-03-24T00:32:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab3d000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab3806e620 0x7fab381f71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab3d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab3806ea00 0x7fab38137b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab3d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab380ccfa0 0x7fab38158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab3d000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3806efd0 con 0x7fab3806ea00 2024-03-24T00:32:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab3d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3806ee50 con 0x7fab380ccfa0 2024-03-24T00:32:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab3d000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab38144920 con 0x7fab3806e620 2024-03-24T00:32:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab2d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab3806ea00 0x7fab38137b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab36800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab380ccfa0 0x7fab38158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab35e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab3806e620 0x7fab381f71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab36800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab380ccfa0 0x7fab38158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:52388/0 (socket says 172.21.15.40:52388) 2024-03-24T00:32:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab36800640 1 -- 172.21.15.40:0/255098660 learned_addr learned my addr 172.21.15.40:0/255098660 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab36800640 1 -- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab3806e620 msgr2=0x7fab381f71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab36800640 1 --2- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab3806e620 0x7fab381f71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab36800640 1 -- 172.21.15.40:0/255098660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab3806ea00 msgr2=0x7fab38137b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab36800640 1 --2- 172.21.15.40:0/255098660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab3806ea00 0x7fab38137b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab36800640 1 -- 172.21.15.40:0/255098660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab38154cd0 con 0x7fab380ccfa0 2024-03-24T00:32:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab36800640 1 --2- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab380ccfa0 0x7fab38158440 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fab30009f00 tx=0x7fab3000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:42.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab2ea00640 1 -- 172.21.15.40:0/255098660 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30018030 con 0x7fab380ccfa0 2024-03-24T00:32:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab2ea00640 1 -- 172.21.15.40:0/255098660 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3000ce50 con 0x7fab380ccfa0 2024-03-24T00:32:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab3d000640 1 -- 172.21.15.40:0/255098660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab381f7710 con 0x7fab380ccfa0 2024-03-24T00:32:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab2ea00640 1 -- 172.21.15.40:0/255098660 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab30003bc0 con 0x7fab380ccfa0 2024-03-24T00:32:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab3d000640 1 -- 172.21.15.40:0/255098660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3815b420 con 0x7fab380ccfa0 2024-03-24T00:32:42.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.468+0000 7fab3d000640 1 -- 172.21.15.40:0/255098660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab38068170 con 0x7fab380ccfa0 2024-03-24T00:32:42.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.472+0000 7fab2ea00640 1 -- 172.21.15.40:0/255098660 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fab3001b070 con 0x7fab380ccfa0 2024-03-24T00:32:42.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.472+0000 7fab2ea00640 1 --2- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab14067660 0x7fab14069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.472+0000 7fab2ea00640 1 -- 172.21.15.40:0/255098660 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab30086220 con 0x7fab380ccfa0 2024-03-24T00:32:42.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.472+0000 7fab35e00640 1 --2- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab14067660 0x7fab14069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.472+0000 7fab35e00640 1 --2- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab14067660 0x7fab14069b20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fab280ad000 tx=0x7fab28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:42.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.476+0000 7fab2ea00640 1 -- 172.21.15.40:0/255098660 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fab30010040 con 0x7fab380ccfa0 2024-03-24T00:32:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.676+0000 7fab3d000640 1 -- 172.21.15.40:0/255098660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab3815dd30 con 0x7fab380ccfa0 2024-03-24T00:32:42.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.676+0000 7fab2ea00640 1 -- 172.21.15.40:0/255098660 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5015 (secure 0 0 0) 0x7fab300567b0 con 0x7fab380ccfa0 2024-03-24T00:32:42.678 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:42.679 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2069836835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2069836835},{"type":"v1","addr":"172.21.15.53:6836","nonce":2069836835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:06.028086+0000","modified":"2024-03-24T00:32:20.288717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5229,"mds_1":5226,"mds_2":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"b","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.53:6837/2159849028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":2159849028},{"type":"v1","addr":"172.21.15.53:6837","nonce":2159849028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5226":{"gid":5226,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6833/2326157501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2326157501},{"type":"v1","addr":"172.21.15.40:6833","nonce":2326157501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5229":{"gid":5229,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3050269371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3050269371},{"type":"v1","addr":"172.21.15.40:6835","nonce":3050269371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5229,"qdb_cluster":[4856,5226,5229]},"id":2}]} 2024-03-24T00:32:42.679 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 21 2024-03-24T00:32:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.680+0000 7fab3d000640 1 -- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab14067660 msgr2=0x7fab14069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.680+0000 7fab3d000640 1 --2- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab14067660 0x7fab14069b20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fab280ad000 tx=0x7fab28008040 comp rx=0 tx=0).stop 2024-03-24T00:32:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.680+0000 7fab3d000640 1 -- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab380ccfa0 msgr2=0x7fab38158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.680+0000 7fab3d000640 1 --2- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab380ccfa0 0x7fab38158440 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fab30009f00 tx=0x7fab3000c490 comp rx=0 tx=0).stop 2024-03-24T00:32:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.680+0000 7fab3d000640 1 -- 172.21.15.40:0/255098660 shutdown_connections 2024-03-24T00:32:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.680+0000 7fab3d000640 1 --2- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab14067660 0x7fab14069b20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.680+0000 7fab3d000640 1 --2- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab380ccfa0 0x7fab38158440 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.680+0000 7fab3d000640 1 --2- 172.21.15.40:0/255098660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab3806ea00 0x7fab38137b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.680+0000 7fab3d000640 1 --2- 172.21.15.40:0/255098660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab3806e620 0x7fab381f71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.680+0000 7fab3d000640 1 -- 172.21.15.40:0/255098660 >> 172.21.15.40:0/255098660 conn(0x7fab380b17b0 msgr2=0x7fab380afae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:42.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.680+0000 7fab3d000640 1 -- 172.21.15.40:0/255098660 shutdown_connections 2024-03-24T00:32:42.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.680+0000 7fab3d000640 1 -- 172.21.15.40:0/255098660 wait complete. 2024-03-24T00:32:42.701 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:32:42.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.840+0000 7f8204000640 1 Processor -- start 2024-03-24T00:32:42.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.840+0000 7f8204000640 1 -- start start 2024-03-24T00:32:42.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.844+0000 7f8204000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81fc13ec80 0x7f81fc1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.844+0000 7f8204000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81fc070e00 con 0x7f81fc1419f0 2024-03-24T00:32:42.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.844+0000 7f8204000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81fc0cee20 con 0x7f81fc0ce800 2024-03-24T00:32:42.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.844+0000 7f8204000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc141e50 con 0x7f81fc13ec80 2024-03-24T00:32:42.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.844+0000 7f8201600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81fc13ec80 0x7f81fc1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.844+0000 7f8202000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f81fc0ce800 0x7f81fc071300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:45996/0 (socket says 172.21.15.40:45996) 2024-03-24T00:32:42.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.844+0000 7f8202000640 1 -- 172.21.15.40:0/3319601853 learned_addr learned my addr 172.21.15.40:0/3319601853 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:42.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.844+0000 7f8200c00640 1 --1- 172.21.15.40:0/3319601853 >> v1:172.21.15.40:6789/0 conn(0x7f81fc1419f0 0x7f81fc0ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38670/0 (socket says 172.21.15.40:38670) 2024-03-24T00:32:42.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.844+0000 7f8201600640 1 -- 172.21.15.40:0/3319601853 >> v1:172.21.15.53:6789/0 conn(0x7f81fc0ce800 legacy=0x7f81fc071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.844+0000 7f8201600640 1 -- 172.21.15.40:0/3319601853 >> v1:172.21.15.40:6789/0 conn(0x7f81fc1419f0 legacy=0x7f81fc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.844+0000 7f81fbe00640 1 -- 172.21.15.40:0/3319601853 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592746519 0 0) 0x7f81fc0cee20 con 0x7f81fc0ce800 2024-03-24T00:32:42.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.844+0000 7f8201600640 1 -- 172.21.15.40:0/3319601853 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81fc0cf100 con 0x7f81fc13ec80 2024-03-24T00:32:42.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8201600640 1 --2- 172.21.15.40:0/3319601853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81fc13ec80 0x7f81fc1414b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f81f00097f0 tx=0x7f81f001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61f90e05442c492f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:42.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f81fbe00640 1 -- 172.21.15.40:0/3319601853 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f00aa660 con 0x7f81fc13ec80 2024-03-24T00:32:42.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f81fbe00640 1 -- 172.21.15.40:0/3319601853 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f00af070 con 0x7f81fc13ec80 2024-03-24T00:32:42.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f81fbe00640 1 -- 172.21.15.40:0/3319601853 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f00aaa20 con 0x7f81fc13ec80 2024-03-24T00:32:42.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 -- 172.21.15.40:0/3319601853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81fc13ec80 msgr2=0x7f81fc1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 --2- 172.21.15.40:0/3319601853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81fc13ec80 0x7f81fc1414b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f81f00097f0 tx=0x7f81f001c4e0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 -- 172.21.15.40:0/3319601853 shutdown_connections 2024-03-24T00:32:42.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 --2- 172.21.15.40:0/3319601853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81fc13ec80 0x7f81fc1414b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 -- 172.21.15.40:0/3319601853 >> 172.21.15.40:0/3319601853 conn(0x7f81fc0b17b0 msgr2=0x7f81fc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:42.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 -- 172.21.15.40:0/3319601853 shutdown_connections 2024-03-24T00:32:42.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 -- 172.21.15.40:0/3319601853 wait complete. 2024-03-24T00:32:42.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 Processor -- start 2024-03-24T00:32:42.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 -- start start 2024-03-24T00:32:42.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81fc0ce800 0x7f81fc14db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81fc13ec80 0x7f81fc158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81fc1419f0 0x7f81fc14ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc0725f0 con 0x7f81fc1419f0 2024-03-24T00:32:42.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc072470 con 0x7f81fc13ec80 2024-03-24T00:32:42.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8204000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81fc169d50 con 0x7f81fc0ce800 2024-03-24T00:32:42.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8200c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81fc13ec80 0x7f81fc158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8201600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81fc0ce800 0x7f81fc14db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8202000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81fc1419f0 0x7f81fc14ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8200c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81fc13ec80 0x7f81fc158240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:52402/0 (socket says 172.21.15.40:52402) 2024-03-24T00:32:42.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8201600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81fc0ce800 0x7f81fc14db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:47068/0 (socket says 172.21.15.40:47068) 2024-03-24T00:32:42.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.848+0000 7f8200c00640 1 -- 172.21.15.40:0/3448019451 learned_addr learned my addr 172.21.15.40:0/3448019451 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:42.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f8200c00640 1 -- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81fc0ce800 msgr2=0x7f81fc14db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f8200c00640 1 --2- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81fc0ce800 0x7f81fc14db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f8200c00640 1 -- 172.21.15.40:0/3448019451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81fc1419f0 msgr2=0x7f81fc14ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:42.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f8200c00640 1 --2- 172.21.15.40:0/3448019451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81fc1419f0 0x7f81fc14ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:42.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f8200c00640 1 -- 172.21.15.40:0/3448019451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81fc0b4410 con 0x7f81fc13ec80 2024-03-24T00:32:42.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f8200c00640 1 --2- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81fc13ec80 0x7f81fc158240 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f81e800c490 tx=0x7f81e800c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:42.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f81f9600640 1 -- 172.21.15.40:0/3448019451 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8018070 con 0x7f81fc13ec80 2024-03-24T00:32:42.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f81f9600640 1 -- 172.21.15.40:0/3448019451 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e8007480 con 0x7f81fc13ec80 2024-03-24T00:32:42.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f81f9600640 1 -- 172.21.15.40:0/3448019451 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8004a10 con 0x7f81fc13ec80 2024-03-24T00:32:42.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f8204000640 1 -- 172.21.15.40:0/3448019451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81fc157480 con 0x7f81fc13ec80 2024-03-24T00:32:42.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f8204000640 1 -- 172.21.15.40:0/3448019451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81fc157a40 con 0x7f81fc13ec80 2024-03-24T00:32:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f8204000640 1 -- 172.21.15.40:0/3448019451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81fc068170 con 0x7f81fc13ec80 2024-03-24T00:32:42.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f81f9600640 1 -- 172.21.15.40:0/3448019451 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f81e8004bb0 con 0x7f81fc13ec80 2024-03-24T00:32:42.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f81f9600640 1 --2- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81dc067660 0x7f81dc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:42.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.852+0000 7f81f9600640 1 -- 172.21.15.40:0/3448019451 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81e8084af0 con 0x7f81fc13ec80 2024-03-24T00:32:42.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.856+0000 7f8201600640 1 --2- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81dc067660 0x7f81dc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:42.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.856+0000 7f8201600640 1 --2- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81dc067660 0x7f81dc069b20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f81f00ae000 tx=0x7f81f00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:42.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:42.856+0000 7f81f9600640 1 -- 172.21.15.40:0/3448019451 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f81e8055110 con 0x7f81fc13ec80 2024-03-24T00:32:43.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.036+0000 7f8204000640 1 -- 172.21.15.40:0/3448019451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81fc072b90 con 0x7f81fc13ec80 2024-03-24T00:32:43.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.036+0000 7f81f9600640 1 -- 172.21.15.40:0/3448019451 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f81e80591b0 con 0x7f81fc13ec80 2024-03-24T00:32:43.039 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:43.040 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":33,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:32:07.623938+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:03.863519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:05.024936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:32:43.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.040+0000 7f8204000640 1 -- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81dc067660 msgr2=0x7f81dc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.040+0000 7f8204000640 1 --2- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81dc067660 0x7f81dc069b20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f81f00ae000 tx=0x7f81f00a7000 comp rx=0 tx=0).stop 2024-03-24T00:32:43.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.040+0000 7f8204000640 1 -- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81fc13ec80 msgr2=0x7f81fc158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.040+0000 7f8204000640 1 --2- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81fc13ec80 0x7f81fc158240 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f81e800c490 tx=0x7f81e800c950 comp rx=0 tx=0).stop 2024-03-24T00:32:43.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.040+0000 7f8204000640 1 -- 172.21.15.40:0/3448019451 shutdown_connections 2024-03-24T00:32:43.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.040+0000 7f8204000640 1 --2- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81dc067660 0x7f81dc069b20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.040+0000 7f8204000640 1 --2- 172.21.15.40:0/3448019451 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81fc1419f0 0x7f81fc14ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.040+0000 7f8204000640 1 --2- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81fc13ec80 0x7f81fc158240 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.040+0000 7f8204000640 1 --2- 172.21.15.40:0/3448019451 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81fc0ce800 0x7f81fc14db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.040+0000 7f8204000640 1 -- 172.21.15.40:0/3448019451 >> 172.21.15.40:0/3448019451 conn(0x7f81fc0b17b0 msgr2=0x7f81fc071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:43.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.040+0000 7f8204000640 1 -- 172.21.15.40:0/3448019451 shutdown_connections 2024-03-24T00:32:43.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.040+0000 7f8204000640 1 -- 172.21.15.40:0/3448019451 wait complete. 2024-03-24T00:32:43.071 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:32:43.071 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:32:43.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe828a00640 1 Processor -- start 2024-03-24T00:32:43.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe828a00640 1 -- start start 2024-03-24T00:32:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe828a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe82415bca0 0x7fe8240d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe828a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe8240ceba0 0x7fe8240d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe828a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8240d0780 con 0x7fe82415c080 2024-03-24T00:32:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe828a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8240b4a90 con 0x7fe82415bca0 2024-03-24T00:32:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe828a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8240b4d70 con 0x7fe8240ceba0 2024-03-24T00:32:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe822800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe82415bca0 0x7fe8240d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe821e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe82415c080 0x7fe8240ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:46014/0 (socket says 172.21.15.40:46014) 2024-03-24T00:32:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe821e00640 1 -- 172.21.15.40:0/854682961 learned_addr learned my addr 172.21.15.40:0/854682961 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe823200640 1 --2- 172.21.15.40:0/854682961 >> v2:172.21.15.53:3300/0 conn(0x7fe8240ceba0 0x7fe8240d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe821400640 1 -- 172.21.15.40:0/854682961 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 554973208 0 0) 0x7fe8240d0780 con 0x7fe82415c080 2024-03-24T00:32:43.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe821400640 1 -- 172.21.15.40:0/854682961 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe808003530 con 0x7fe82415c080 2024-03-24T00:32:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe822800640 1 -- 172.21.15.40:0/854682961 >> v2:172.21.15.53:3300/0 conn(0x7fe8240ceba0 msgr2=0x7fe8240d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe822800640 1 --2- 172.21.15.40:0/854682961 >> v2:172.21.15.53:3300/0 conn(0x7fe8240ceba0 0x7fe8240d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe822800640 1 -- 172.21.15.40:0/854682961 >> v1:172.21.15.53:6789/0 conn(0x7fe82415c080 legacy=0x7fe8240ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe822800640 1 -- 172.21.15.40:0/854682961 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8240b5050 con 0x7fe82415bca0 2024-03-24T00:32:43.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.212+0000 7fe823200640 1 --2- 172.21.15.40:0/854682961 >> v2:172.21.15.53:3300/0 conn(0x7fe8240ceba0 0x7fe8240d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:43.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.216+0000 7fe822800640 1 --2- 172.21.15.40:0/854682961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82415bca0 0x7fe8240d2590 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe824072300 tx=0x7fe81401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632603e59cfd3ec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:43.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.216+0000 7fe821400640 1 -- 172.21.15.40:0/854682961 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8140b0070 con 0x7fe82415bca0 2024-03-24T00:32:43.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.216+0000 7fe821400640 1 -- 172.21.15.40:0/854682961 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8140ab600 con 0x7fe82415bca0 2024-03-24T00:32:43.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.216+0000 7fe821400640 1 -- 172.21.15.40:0/854682961 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8140ab9b0 con 0x7fe82415bca0 2024-03-24T00:32:43.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.216+0000 7fe828a00640 1 -- 172.21.15.40:0/854682961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82415bca0 msgr2=0x7fe8240d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.216+0000 7fe828a00640 1 --2- 172.21.15.40:0/854682961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82415bca0 0x7fe8240d2590 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe824072300 tx=0x7fe81401c480 comp rx=0 tx=0).stop 2024-03-24T00:32:43.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.216+0000 7fe828a00640 1 -- 172.21.15.40:0/854682961 shutdown_connections 2024-03-24T00:32:43.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.216+0000 7fe828a00640 1 --2- 172.21.15.40:0/854682961 >> v2:172.21.15.53:3300/0 conn(0x7fe8240ceba0 0x7fe8240d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.216+0000 7fe828a00640 1 --2- 172.21.15.40:0/854682961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82415bca0 0x7fe8240d2590 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.216+0000 7fe828a00640 1 -- 172.21.15.40:0/854682961 >> 172.21.15.40:0/854682961 conn(0x7fe8240d1aa0 msgr2=0x7fe8240d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:43.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.216+0000 7fe828a00640 1 -- 172.21.15.40:0/854682961 shutdown_connections 2024-03-24T00:32:43.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.216+0000 7fe828a00640 1 -- 172.21.15.40:0/854682961 wait complete. 2024-03-24T00:32:43.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe828a00640 1 Processor -- start 2024-03-24T00:32:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe828a00640 1 -- start start 2024-03-24T00:32:43.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe828a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8240ceba0 0x7fe8240c9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:43.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe828a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe82415bca0 0x7fe824150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:43.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe828a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82415c080 0x7fe82415a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe828a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8240b6110 con 0x7fe82415c080 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe828a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8240b5f90 con 0x7fe82415bca0 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe828a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8240b6290 con 0x7fe8240ceba0 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe823200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82415c080 0x7fe82415a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe822800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8240ceba0 0x7fe8240c9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe823200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82415c080 0x7fe82415a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49276/0 (socket says 172.21.15.40:49276) 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe823200640 1 -- 172.21.15.40:0/1834367759 learned_addr learned my addr 172.21.15.40:0/1834367759 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe821e00640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe82415bca0 0x7fe824150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe822800640 1 -- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe82415bca0 msgr2=0x7fe824150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe822800640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe82415bca0 0x7fe824150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe822800640 1 -- 172.21.15.40:0/1834367759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82415c080 msgr2=0x7fe82415a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe822800640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82415c080 0x7fe82415a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe822800640 1 -- 172.21.15.40:0/1834367759 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8240ca4b0 con 0x7fe8240ceba0 2024-03-24T00:32:43.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe822800640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8240ceba0 0x7fe8240c9f70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe8240b4690 tx=0x7fe814019220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:43.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe81aa00640 1 -- 172.21.15.40:0/1834367759 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8140b0040 con 0x7fe8240ceba0 2024-03-24T00:32:43.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe81aa00640 1 -- 172.21.15.40:0/1834367759 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8140b9070 con 0x7fe8240ceba0 2024-03-24T00:32:43.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe81aa00640 1 -- 172.21.15.40:0/1834367759 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe814006e20 con 0x7fe8240ceba0 2024-03-24T00:32:43.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe828a00640 1 -- 172.21.15.40:0/1834367759 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8241593e0 con 0x7fe8240ceba0 2024-03-24T00:32:43.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe828a00640 1 -- 172.21.15.40:0/1834367759 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8241599a0 con 0x7fe8240ceba0 2024-03-24T00:32:43.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.220+0000 7fe828a00640 1 -- 172.21.15.40:0/1834367759 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe82409fb10 con 0x7fe8240ceba0 2024-03-24T00:32:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.224+0000 7fe81aa00640 1 -- 172.21.15.40:0/1834367759 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe8140aa9d0 con 0x7fe8240ceba0 2024-03-24T00:32:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.224+0000 7fe81aa00640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe804067660 0x7fe804069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.224+0000 7fe81aa00640 1 -- 172.21.15.40:0/1834367759 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe81411fb30 con 0x7fe8240ceba0 2024-03-24T00:32:43.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.224+0000 7fe821e00640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe804067660 0x7fe804069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:43.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.224+0000 7fe821e00640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe804067660 0x7fe804069b20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe8241f9db0 tx=0x7fe810000bd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:43.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.232+0000 7fe81aa00640 1 -- 172.21.15.40:0/1834367759 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe814016070 con 0x7fe8240ceba0 2024-03-24T00:32:43.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.428+0000 7fe828a00640 1 -- 172.21.15.40:0/1834367759 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8240d1880 con 0x7fe8240ceba0 2024-03-24T00:32:43.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.428+0000 7fe81aa00640 1 -- 172.21.15.40:0/1834367759 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5015 (secure 0 0 0) 0x7fe8140f0270 con 0x7fe8240ceba0 2024-03-24T00:32:43.431 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:43.431 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2069836835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2069836835},{"type":"v1","addr":"172.21.15.53:6836","nonce":2069836835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:06.028086+0000","modified":"2024-03-24T00:32:20.288717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5229,"mds_1":5226,"mds_2":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"b","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.53:6837/2159849028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":2159849028},{"type":"v1","addr":"172.21.15.53:6837","nonce":2159849028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5226":{"gid":5226,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6833/2326157501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2326157501},{"type":"v1","addr":"172.21.15.40:6833","nonce":2326157501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5229":{"gid":5229,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3050269371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3050269371},{"type":"v1","addr":"172.21.15.40:6835","nonce":3050269371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5229,"qdb_cluster":[4856,5226,5229]},"id":2}]} 2024-03-24T00:32:43.431 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 21 2024-03-24T00:32:43.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.432+0000 7fe828a00640 1 -- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe804067660 msgr2=0x7fe804069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.432+0000 7fe828a00640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe804067660 0x7fe804069b20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe8241f9db0 tx=0x7fe810000bd0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.432+0000 7fe828a00640 1 -- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8240ceba0 msgr2=0x7fe8240c9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.432+0000 7fe828a00640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8240ceba0 0x7fe8240c9f70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe8240b4690 tx=0x7fe814019220 comp rx=0 tx=0).stop 2024-03-24T00:32:43.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.432+0000 7fe828a00640 1 -- 172.21.15.40:0/1834367759 shutdown_connections 2024-03-24T00:32:43.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.432+0000 7fe828a00640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe804067660 0x7fe804069b20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.432+0000 7fe828a00640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe82415c080 0x7fe82415a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.432+0000 7fe828a00640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe82415bca0 0x7fe824150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.432+0000 7fe828a00640 1 --2- 172.21.15.40:0/1834367759 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8240ceba0 0x7fe8240c9f70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.432+0000 7fe828a00640 1 -- 172.21.15.40:0/1834367759 >> 172.21.15.40:0/1834367759 conn(0x7fe8240d1aa0 msgr2=0x7fe82406eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:43.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.432+0000 7fe828a00640 1 -- 172.21.15.40:0/1834367759 shutdown_connections 2024-03-24T00:32:43.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.432+0000 7fe828a00640 1 -- 172.21.15.40:0/1834367759 wait complete. 2024-03-24T00:32:43.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:32:43.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef3600640 1 Processor -- start 2024-03-24T00:32:43.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef3600640 1 -- start start 2024-03-24T00:32:43.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef3600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4eec15e0b0 0x7f4eec0b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:43.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef3600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4eec0b4b80 0x7f4eec0cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:43.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef3600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eec0cd3d0 con 0x7f4eec06b600 2024-03-24T00:32:43.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef3600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eec15e490 con 0x7f4eec15e0b0 2024-03-24T00:32:43.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef3600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec0cdfe0 con 0x7f4eec0b4b80 2024-03-24T00:32:43.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef0c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4eec15e0b0 0x7f4eec0b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef0c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4eec15e0b0 0x7f4eec0b43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49282/0 (socket says 172.21.15.40:49282) 2024-03-24T00:32:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef0c00640 1 -- 172.21.15.40:0/2408506255 learned_addr learned my addr 172.21.15.40:0/2408506255 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4eebe00640 1 --2- 172.21.15.40:0/2408506255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4eec0b4b80 0x7f4eec0cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef1600640 1 --1- 172.21.15.40:0/2408506255 >> v1:172.21.15.40:6789/0 conn(0x7f4eec06b600 0x7f4eec0b6ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38678/0 (socket says 172.21.15.40:38678) 2024-03-24T00:32:43.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef0c00640 1 -- 172.21.15.40:0/2408506255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4eec0b4b80 msgr2=0x7f4eec0cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef0c00640 1 --2- 172.21.15.40:0/2408506255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4eec0b4b80 0x7f4eec0cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef0c00640 1 -- 172.21.15.40:0/2408506255 >> v1:172.21.15.40:6789/0 conn(0x7f4eec06b600 legacy=0x7f4eec0b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4ef0c00640 1 -- 172.21.15.40:0/2408506255 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eec0cd6b0 con 0x7f4eec15e0b0 2024-03-24T00:32:43.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.600+0000 7f4eebe00640 1 --2- 172.21.15.40:0/2408506255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4eec0b4b80 0x7f4eec0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:43.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4ef0c00640 1 --2- 172.21.15.40:0/2408506255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eec15e0b0 0x7f4eec0b43a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009870 tx=0x7f4ee001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=370f66814be8a1b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:43.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4eeb400640 1 -- 172.21.15.40:0/2408506255 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00b5070 con 0x7f4eec15e0b0 2024-03-24T00:32:43.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4eeb400640 1 -- 172.21.15.40:0/2408506255 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee00a8a50 con 0x7f4eec15e0b0 2024-03-24T00:32:43.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4eeb400640 1 -- 172.21.15.40:0/2408506255 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00a8e20 con 0x7f4eec15e0b0 2024-03-24T00:32:43.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4ef3600640 1 -- 172.21.15.40:0/2408506255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eec15e0b0 msgr2=0x7f4eec0b43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4ef3600640 1 --2- 172.21.15.40:0/2408506255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eec15e0b0 0x7f4eec0b43a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009870 tx=0x7f4ee001ed50 comp rx=0 tx=0).stop 2024-03-24T00:32:43.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4ef3600640 1 -- 172.21.15.40:0/2408506255 shutdown_connections 2024-03-24T00:32:43.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4ef3600640 1 --2- 172.21.15.40:0/2408506255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4eec0b4b80 0x7f4eec0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4ef3600640 1 --2- 172.21.15.40:0/2408506255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eec15e0b0 0x7f4eec0b43a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4ef3600640 1 -- 172.21.15.40:0/2408506255 >> 172.21.15.40:0/2408506255 conn(0x7f4eec0b3880 msgr2=0x7f4eec0b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:43.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4ef3600640 1 -- 172.21.15.40:0/2408506255 shutdown_connections 2024-03-24T00:32:43.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4ef3600640 1 -- 172.21.15.40:0/2408506255 wait complete. 2024-03-24T00:32:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4ef3600640 1 Processor -- start 2024-03-24T00:32:43.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.608+0000 7f4ef3600640 1 -- start start 2024-03-24T00:32:43.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef3600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4eec06b600 0x7f4eec1f8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:43.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef3600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4eec0b4b80 0x7f4eec0ca420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:43.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef3600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eec15e0b0 0x7f4eec1f7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:43.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef3600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec06c480 con 0x7f4eec15e0b0 2024-03-24T00:32:43.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef3600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec0cdc70 con 0x7f4eec06b600 2024-03-24T00:32:43.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef3600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec06bab0 con 0x7f4eec0b4b80 2024-03-24T00:32:43.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eec15e0b0 0x7f4eec1f7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:43.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4eec06b600 0x7f4eec1f8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:43.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4eebe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4eec0b4b80 0x7f4eec0ca420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:43.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eec15e0b0 0x7f4eec1f7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49294/0 (socket says 172.21.15.40:49294) 2024-03-24T00:32:43.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef1600640 1 -- 172.21.15.40:0/1564104045 learned_addr learned my addr 172.21.15.40:0/1564104045 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:43.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef1600640 1 -- 172.21.15.40:0/1564104045 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4eec0b4b80 msgr2=0x7f4eec0ca420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef1600640 1 --2- 172.21.15.40:0/1564104045 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4eec0b4b80 0x7f4eec0ca420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef1600640 1 -- 172.21.15.40:0/1564104045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4eec06b600 msgr2=0x7f4eec1f8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef1600640 1 --2- 172.21.15.40:0/1564104045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4eec06b600 0x7f4eec1f8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef1600640 1 -- 172.21.15.40:0/1564104045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eec0cace0 con 0x7f4eec15e0b0 2024-03-24T00:32:43.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef1600640 1 --2- 172.21.15.40:0/1564104045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eec15e0b0 0x7f4eec1f7840 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400c7a0 tx=0x7f4ee400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:43.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ee8c00640 1 -- 172.21.15.40:0/1564104045 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4017070 con 0x7f4eec15e0b0 2024-03-24T00:32:43.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ee8c00640 1 -- 172.21.15.40:0/1564104045 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee400a640 con 0x7f4eec15e0b0 2024-03-24T00:32:43.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef3600640 1 -- 172.21.15.40:0/1564104045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eec0cafc0 con 0x7f4eec15e0b0 2024-03-24T00:32:43.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ee8c00640 1 -- 172.21.15.40:0/1564104045 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee400a9f0 con 0x7f4eec15e0b0 2024-03-24T00:32:43.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef3600640 1 -- 172.21.15.40:0/1564104045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eec0cb580 con 0x7f4eec15e0b0 2024-03-24T00:32:43.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ef3600640 1 -- 172.21.15.40:0/1564104045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eec068170 con 0x7f4eec15e0b0 2024-03-24T00:32:43.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.612+0000 7f4ee8c00640 1 -- 172.21.15.40:0/1564104045 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4ee401a070 con 0x7f4eec15e0b0 2024-03-24T00:32:43.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.616+0000 7f4ee8c00640 1 --2- 172.21.15.40:0/1564104045 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ecc067660 0x7f4ecc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:43.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.616+0000 7f4ee8c00640 1 -- 172.21.15.40:0/1564104045 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ee4084f10 con 0x7f4eec15e0b0 2024-03-24T00:32:43.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.616+0000 7f4ef0c00640 1 --2- 172.21.15.40:0/1564104045 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ecc067660 0x7f4ecc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:43.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.616+0000 7f4ef0c00640 1 --2- 172.21.15.40:0/1564104045 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ecc067660 0x7f4ecc069b20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4ee00b2000 tx=0x7f4ee0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:43.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.620+0000 7f4ee8c00640 1 -- 172.21.15.40:0/1564104045 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4ee40554a0 con 0x7f4eec15e0b0 2024-03-24T00:32:43.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 -- 172.21.15.40:0/1564104045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4eec0d0790 con 0x7f4eec15e0b0 2024-03-24T00:32:43.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ee8c00640 1 -- 172.21.15.40:0/1564104045 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+5015 (secure 0 0 0) 0x7f4ee4059540 con 0x7f4eec15e0b0 2024-03-24T00:32:43.834 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:43.834 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2069836835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2069836835},{"type":"v1","addr":"172.21.15.53:6836","nonce":2069836835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:06.028086+0000","modified":"2024-03-24T00:32:20.288717+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5229,"mds_1":5226,"mds_2":4856},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4856":{"gid":4856,"name":"b","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.53:6837/2159849028","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":2159849028},{"type":"v1","addr":"172.21.15.53:6837","nonce":2159849028}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5226":{"gid":5226,"name":"a","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6833/2326157501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2326157501},{"type":"v1","addr":"172.21.15.40:6833","nonce":2326157501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5229":{"gid":5229,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/3050269371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":3050269371},{"type":"v1","addr":"172.21.15.40:6835","nonce":3050269371}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5229,"qdb_cluster":[4856,5226,5229]},"id":2}]} 2024-03-24T00:32:43.834 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 21 2024-03-24T00:32:43.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 -- 172.21.15.40:0/1564104045 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ecc067660 msgr2=0x7f4ecc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 --2- 172.21.15.40:0/1564104045 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ecc067660 0x7f4ecc069b20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4ee00b2000 tx=0x7f4ee0009040 comp rx=0 tx=0).stop 2024-03-24T00:32:43.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 -- 172.21.15.40:0/1564104045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eec15e0b0 msgr2=0x7f4eec1f7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:43.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 --2- 172.21.15.40:0/1564104045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eec15e0b0 0x7f4eec1f7840 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4ee400c7a0 tx=0x7f4ee400cc60 comp rx=0 tx=0).stop 2024-03-24T00:32:43.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 -- 172.21.15.40:0/1564104045 shutdown_connections 2024-03-24T00:32:43.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 --2- 172.21.15.40:0/1564104045 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ecc067660 0x7f4ecc069b20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 --2- 172.21.15.40:0/1564104045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4eec15e0b0 0x7f4eec1f7840 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 --2- 172.21.15.40:0/1564104045 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4eec0b4b80 0x7f4eec0ca420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 --2- 172.21.15.40:0/1564104045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4eec06b600 0x7f4eec1f8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:43.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 -- 172.21.15.40:0/1564104045 >> 172.21.15.40:0/1564104045 conn(0x7f4eec0b3880 msgr2=0x7f4eec0cfce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:43.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 -- 172.21.15.40:0/1564104045 shutdown_connections 2024-03-24T00:32:43.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.832+0000 7f4ef3600640 1 -- 172.21.15.40:0/1564104045 wait complete. 2024-03-24T00:32:43.855 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:32:43.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.996+0000 7f21b0000640 1 Processor -- start 2024-03-24T00:32:43.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:43.996+0000 7f21b0000640 1 -- start start 2024-03-24T00:32:44.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21b0000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f21a80b46c0 0x7f21a80b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21b0000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a8070f60 con 0x7f21a806e620 2024-03-24T00:32:44.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21b0000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a80b3620 con 0x7f21a80b2ff0 2024-03-24T00:32:44.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21b0000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a80b3400 con 0x7f21a80b46c0 2024-03-24T00:32:44.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21ad600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f21a806e620 0x7f21a806ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38680/0 (socket says 172.21.15.40:38680) 2024-03-24T00:32:44.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21ad600640 1 -- 172.21.15.40:0/3672124896 learned_addr learned my addr 172.21.15.40:0/3672124896 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:44.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21acc00640 1 --2- 172.21.15.40:0/3672124896 >> v2:172.21.15.53:3300/0 conn(0x7f21a80b46c0 0x7f21a80b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21a7e00640 1 -- 172.21.15.40:0/3672124896 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193814366 0 0) 0x7f21a80b3620 con 0x7f21a80b2ff0 2024-03-24T00:32:44.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21a7e00640 1 -- 172.21.15.40:0/3672124896 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2190005740 con 0x7f21a80b2ff0 2024-03-24T00:32:44.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21a7e00640 1 -- 172.21.15.40:0/3672124896 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757327054 0 0) 0x7f21a8070f60 con 0x7f21a806e620 2024-03-24T00:32:44.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21a7e00640 1 -- 172.21.15.40:0/3672124896 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21a80b3620 con 0x7f21a806e620 2024-03-24T00:32:44.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21acc00640 1 -- 172.21.15.40:0/3672124896 >> v1:172.21.15.53:6789/0 conn(0x7f21a80b2ff0 legacy=0x7f21a80ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21acc00640 1 -- 172.21.15.40:0/3672124896 >> v1:172.21.15.40:6789/0 conn(0x7f21a806e620 legacy=0x7f21a806ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.000+0000 7f21acc00640 1 -- 172.21.15.40:0/3672124896 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a80b3900 con 0x7f21a80b46c0 2024-03-24T00:32:44.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21acc00640 1 --2- 172.21.15.40:0/3672124896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a80b46c0 0x7f21a80b2ab0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f21940097c0 tx=0x7f219401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5fdd7e560de553e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:44.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21a7e00640 1 -- 172.21.15.40:0/3672124896 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21940b0070 con 0x7f21a80b46c0 2024-03-24T00:32:44.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21a7e00640 1 -- 172.21.15.40:0/3672124896 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21940ab600 con 0x7f21a80b46c0 2024-03-24T00:32:44.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21a7e00640 1 -- 172.21.15.40:0/3672124896 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21940ab9d0 con 0x7f21a80b46c0 2024-03-24T00:32:44.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21b0000640 1 -- 172.21.15.40:0/3672124896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a80b46c0 msgr2=0x7f21a80b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21b0000640 1 --2- 172.21.15.40:0/3672124896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a80b46c0 0x7f21a80b2ab0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f21940097c0 tx=0x7f219401c3e0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21b0000640 1 -- 172.21.15.40:0/3672124896 shutdown_connections 2024-03-24T00:32:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21b0000640 1 --2- 172.21.15.40:0/3672124896 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a80b46c0 0x7f21a80b2ab0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21b0000640 1 -- 172.21.15.40:0/3672124896 >> 172.21.15.40:0/3672124896 conn(0x7f21a80b17b0 msgr2=0x7f21a80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21b0000640 1 -- 172.21.15.40:0/3672124896 shutdown_connections 2024-03-24T00:32:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21b0000640 1 -- 172.21.15.40:0/3672124896 wait complete. 2024-03-24T00:32:44.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21b0000640 1 Processor -- start 2024-03-24T00:32:44.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.004+0000 7f21b0000640 1 -- start start 2024-03-24T00:32:44.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21b0000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a806e620 0x7f21a81f31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21b0000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21a80b2ff0 0x7f21a813c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21b0000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21a80b46c0 0x7f21a80ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21b0000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a8144920 con 0x7f21a80b2ff0 2024-03-24T00:32:44.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21b0000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80b4b50 con 0x7f21a806e620 2024-03-24T00:32:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21b0000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80aabd0 con 0x7f21a80b46c0 2024-03-24T00:32:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21acc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21a80b2ff0 0x7f21a813c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21ad600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a806e620 0x7f21a81f31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21acc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21a80b2ff0 0x7f21a813c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49308/0 (socket says 172.21.15.40:49308) 2024-03-24T00:32:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21acc00640 1 -- 172.21.15.40:0/4017564678 learned_addr learned my addr 172.21.15.40:0/4017564678 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:44.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21ae000640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21a80b46c0 0x7f21a80ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21ad600640 1 -- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21a80b46c0 msgr2=0x7f21a80ae5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21ad600640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21a80b46c0 0x7f21a80ae5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21ad600640 1 -- 172.21.15.40:0/4017564678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21a80b2ff0 msgr2=0x7f21a813c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21ad600640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21a80b2ff0 0x7f21a813c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21ad600640 1 -- 172.21.15.40:0/4017564678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a80af3e0 con 0x7f21a806e620 2024-03-24T00:32:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21acc00640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21a80b2ff0 0x7f21a813c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21ad600640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a806e620 0x7f21a81f31d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f219c000f70 tx=0x7f219c00c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21a5600640 1 -- 172.21.15.40:0/4017564678 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219c017070 con 0x7f21a806e620 2024-03-24T00:32:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21a5600640 1 -- 172.21.15.40:0/4017564678 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219c005d10 con 0x7f21a806e620 2024-03-24T00:32:44.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21b0000640 1 -- 172.21.15.40:0/4017564678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a80af100 con 0x7f21a806e620 2024-03-24T00:32:44.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21a5600640 1 -- 172.21.15.40:0/4017564678 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219c007310 con 0x7f21a806e620 2024-03-24T00:32:44.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21b0000640 1 -- 172.21.15.40:0/4017564678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a80aa860 con 0x7f21a806e620 2024-03-24T00:32:44.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.008+0000 7f21b0000640 1 -- 172.21.15.40:0/4017564678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a8068170 con 0x7f21a806e620 2024-03-24T00:32:44.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.012+0000 7f21a5600640 1 -- 172.21.15.40:0/4017564678 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f219c01a070 con 0x7f21a806e620 2024-03-24T00:32:44.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.012+0000 7f21a5600640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2188067660 0x7f2188069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.012+0000 7f21a5600640 1 -- 172.21.15.40:0/4017564678 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f219c084bb0 con 0x7f21a806e620 2024-03-24T00:32:44.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.012+0000 7f21acc00640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2188067660 0x7f2188069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.012+0000 7f21acc00640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2188067660 0x7f2188069b20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f21940097c0 tx=0x7f2194008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:44.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.016+0000 7f21a5600640 1 -- 172.21.15.40:0/4017564678 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f219c0551d0 con 0x7f21a806e620 2024-03-24T00:32:44.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.180+0000 7f21b0000640 1 -- 172.21.15.40:0/4017564678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21a8072f50 con 0x7f21a806e620 2024-03-24T00:32:44.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.180+0000 7f21a5600640 1 -- 172.21.15.40:0/4017564678 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f219c059270 con 0x7f21a806e620 2024-03-24T00:32:44.184 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:44.184 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":33,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:32:07.623938+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:03.863519+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:05.024936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:32:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.184+0000 7f21b0000640 1 -- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2188067660 msgr2=0x7f2188069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.184+0000 7f21b0000640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2188067660 0x7f2188069b20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f21940097c0 tx=0x7f2194008040 comp rx=0 tx=0).stop 2024-03-24T00:32:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.184+0000 7f21b0000640 1 -- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a806e620 msgr2=0x7f21a81f31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.184+0000 7f21b0000640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a806e620 0x7f21a81f31d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f219c000f70 tx=0x7f219c00c890 comp rx=0 tx=0).stop 2024-03-24T00:32:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.184+0000 7f21b0000640 1 -- 172.21.15.40:0/4017564678 shutdown_connections 2024-03-24T00:32:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.184+0000 7f21b0000640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2188067660 0x7f2188069b20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.184+0000 7f21b0000640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21a80b46c0 0x7f21a80ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.184+0000 7f21b0000640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21a80b2ff0 0x7f21a813c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.184+0000 7f21b0000640 1 --2- 172.21.15.40:0/4017564678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a806e620 0x7f21a81f31d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.184+0000 7f21b0000640 1 -- 172.21.15.40:0/4017564678 >> 172.21.15.40:0/4017564678 conn(0x7f21a80b17b0 msgr2=0x7f21a80720a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.184+0000 7f21b0000640 1 -- 172.21.15.40:0/4017564678 shutdown_connections 2024-03-24T00:32:44.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.184+0000 7f21b0000640 1 -- 172.21.15.40:0/4017564678 wait complete. 2024-03-24T00:32:44.207 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:32:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefb2c00640 1 Processor -- start 2024-03-24T00:32:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefb2c00640 1 -- start start 2024-03-24T00:32:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefb2c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fefac06e840 0x7fefac0b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefb2c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fefac06ec20 0x7fefac071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefb2c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefac070d20 con 0x7fefac06d510 2024-03-24T00:32:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefb2c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefac15b820 con 0x7fefac06ec20 2024-03-24T00:32:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefb2c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefac070fc0 con 0x7fefac06e840 2024-03-24T00:32:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefab400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fefac06ec20 0x7fefac071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefabe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fefac06e840 0x7fefac0b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefb0c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fefac06d510 0x7fefac06d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38696/0 (socket says 172.21.15.40:38696) 2024-03-24T00:32:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefb0c00640 1 -- 172.21.15.40:0/1080803704 learned_addr learned my addr 172.21.15.40:0/1080803704 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefabe00640 1 -- 172.21.15.40:0/1080803704 >> v2:172.21.15.40:3300/0 conn(0x7fefac06ec20 msgr2=0x7fefac071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefabe00640 1 --2- 172.21.15.40:0/1080803704 >> v2:172.21.15.40:3300/0 conn(0x7fefac06ec20 0x7fefac071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefabe00640 1 -- 172.21.15.40:0/1080803704 >> v1:172.21.15.40:6789/0 conn(0x7fefac06d510 legacy=0x7fefac06d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.348+0000 7fefabe00640 1 -- 172.21.15.40:0/1080803704 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac0712a0 con 0x7fefac06e840 2024-03-24T00:32:44.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.356+0000 7fefabe00640 1 --2- 172.21.15.40:0/1080803704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefac06e840 0x7fefac0b4a00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fefac06cac0 tx=0x7fef9c01d730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c0f07aa4d44d538 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:44.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.356+0000 7fefaaa00640 1 -- 172.21.15.40:0/1080803704 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c00b040 con 0x7fefac06e840 2024-03-24T00:32:44.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.356+0000 7fefaaa00640 1 -- 172.21.15.40:0/1080803704 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef9c0a76a0 con 0x7fefac06e840 2024-03-24T00:32:44.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.356+0000 7fefaaa00640 1 -- 172.21.15.40:0/1080803704 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9c0a7a70 con 0x7fefac06e840 2024-03-24T00:32:44.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.356+0000 7fefb2c00640 1 -- 172.21.15.40:0/1080803704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefac06e840 msgr2=0x7fefac0b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.356+0000 7fefb2c00640 1 --2- 172.21.15.40:0/1080803704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefac06e840 0x7fefac0b4a00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fefac06cac0 tx=0x7fef9c01d730 comp rx=0 tx=0).stop 2024-03-24T00:32:44.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.356+0000 7fefb2c00640 1 -- 172.21.15.40:0/1080803704 shutdown_connections 2024-03-24T00:32:44.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.356+0000 7fefb2c00640 1 --2- 172.21.15.40:0/1080803704 >> v2:172.21.15.40:3300/0 conn(0x7fefac06ec20 0x7fefac071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.356+0000 7fefb2c00640 1 --2- 172.21.15.40:0/1080803704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefac06e840 0x7fefac0b4a00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.356+0000 7fefb2c00640 1 -- 172.21.15.40:0/1080803704 >> 172.21.15.40:0/1080803704 conn(0x7fefac0b17b0 msgr2=0x7fefac0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:44.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.356+0000 7fefb2c00640 1 -- 172.21.15.40:0/1080803704 shutdown_connections 2024-03-24T00:32:44.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.356+0000 7fefb2c00640 1 -- 172.21.15.40:0/1080803704 wait complete. 2024-03-24T00:32:44.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb2c00640 1 Processor -- start 2024-03-24T00:32:44.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb2c00640 1 -- start start 2024-03-24T00:32:44.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb2c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fefac06d510 0x7fefac14c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb2c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefac06e840 0x7fefac147630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb2c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefac06ec20 0x7fefac0cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb2c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac15e5f0 con 0x7fefac06ec20 2024-03-24T00:32:44.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb2c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac15e470 con 0x7fefac06e840 2024-03-24T00:32:44.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb2c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefac15e770 con 0x7fefac06d510 2024-03-24T00:32:44.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefac06ec20 0x7fefac0cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefac06ec20 0x7fefac0cd500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49332/0 (socket says 172.21.15.40:49332) 2024-03-24T00:32:44.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefabe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fefac06d510 0x7fefac14c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb0c00640 1 -- 172.21.15.40:0/981128138 learned_addr learned my addr 172.21.15.40:0/981128138 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:44.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefab400640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefac06e840 0x7fefac147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb0c00640 1 -- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fefac06d510 msgr2=0x7fefac14c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb0c00640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fefac06d510 0x7fefac14c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb0c00640 1 -- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefac06e840 msgr2=0x7fefac147630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb0c00640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefac06e840 0x7fefac147630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb0c00640 1 -- 172.21.15.40:0/981128138 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac153640 con 0x7fefac06ec20 2024-03-24T00:32:44.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefab400640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefac06e840 0x7fefac147630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:44.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb0c00640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefac06ec20 0x7fefac0cd500 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fefa400cbf0 tx=0x7fefa400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:44.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefa3e00640 1 -- 172.21.15.40:0/981128138 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4017070 con 0x7fefac06ec20 2024-03-24T00:32:44.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefa3e00640 1 -- 172.21.15.40:0/981128138 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefa400ad40 con 0x7fefac06ec20 2024-03-24T00:32:44.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb2c00640 1 -- 172.21.15.40:0/981128138 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac149630 con 0x7fefac06ec20 2024-03-24T00:32:44.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefa3e00640 1 -- 172.21.15.40:0/981128138 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefa4012600 con 0x7fefac06ec20 2024-03-24T00:32:44.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.360+0000 7fefb2c00640 1 -- 172.21.15.40:0/981128138 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefac153be0 con 0x7fefac06ec20 2024-03-24T00:32:44.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.364+0000 7fefb2c00640 1 -- 172.21.15.40:0/981128138 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefac068170 con 0x7fefac06ec20 2024-03-24T00:32:44.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.364+0000 7fefa3e00640 1 -- 172.21.15.40:0/981128138 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fefa401a070 con 0x7fefac06ec20 2024-03-24T00:32:44.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.364+0000 7fefa3e00640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fef8c067660 0x7fef8c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.364+0000 7fefa3e00640 1 -- 172.21.15.40:0/981128138 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fefa4085560 con 0x7fefac06ec20 2024-03-24T00:32:44.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.364+0000 7fefabe00640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fef8c067660 0x7fef8c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.364+0000 7fefabe00640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fef8c067660 0x7fef8c069b20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0b0000 tx=0x7fef9c00c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:44.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.372+0000 7fefa3e00640 1 -- 172.21.15.40:0/981128138 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fefa4055af0 con 0x7fefac06ec20 2024-03-24T00:32:44.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.568+0000 7fefb2c00640 1 -- 172.21.15.40:0/981128138 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fefac0b2fe0 con 0x7fefac06ec20 2024-03-24T00:32:44.572 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-03-24T00:32:44.572+0000 7fa64e600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:32:44.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.572+0000 7fefa3e00640 1 -- 172.21.15.40:0/981128138 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v22) v1 ==== 161+0+0 (secure 0 0 0) 0x7fefa4059b90 con 0x7fefac06ec20 2024-03-24T00:32:44.574 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:32:44.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.580+0000 7fefb2c00640 1 -- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fef8c067660 msgr2=0x7fef8c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.580+0000 7fefb2c00640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fef8c067660 0x7fef8c069b20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0b0000 tx=0x7fef9c00c040 comp rx=0 tx=0).stop 2024-03-24T00:32:44.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.580+0000 7fefb2c00640 1 -- 172.21.15.40:0/981128138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefac06ec20 msgr2=0x7fefac0cd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.580+0000 7fefb2c00640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefac06ec20 0x7fefac0cd500 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fefa400cbf0 tx=0x7fefa400a430 comp rx=0 tx=0).stop 2024-03-24T00:32:44.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.580+0000 7fefb2c00640 1 -- 172.21.15.40:0/981128138 shutdown_connections 2024-03-24T00:32:44.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.580+0000 7fefb2c00640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fef8c067660 0x7fef8c069b20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.580+0000 7fefb2c00640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fefac06ec20 0x7fefac0cd500 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.580+0000 7fefb2c00640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fefac06e840 0x7fefac147630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.580+0000 7fefb2c00640 1 --2- 172.21.15.40:0/981128138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fefac06d510 0x7fefac14c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.580+0000 7fefb2c00640 1 -- 172.21.15.40:0/981128138 >> 172.21.15.40:0/981128138 conn(0x7fefac0b17b0 msgr2=0x7fefac14b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:44.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.580+0000 7fefb2c00640 1 -- 172.21.15.40:0/981128138 shutdown_connections 2024-03-24T00:32:44.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.580+0000 7fefb2c00640 1 -- 172.21.15.40:0/981128138 wait complete. 2024-03-24T00:32:44.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:32:44.623 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:32:44.632 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:32:44.648 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:32:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12bc000640 1 Processor -- start 2024-03-24T00:32:44.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12bc000640 1 -- start start 2024-03-24T00:32:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12bc000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f12b415bd00 0x7f12b40ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12bc000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f12b415ffd0 0x7f12b40cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12bc000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12b4072d60 con 0x7f12b40728d0 2024-03-24T00:32:44.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12bc000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b40cf790 con 0x7f12b415bd00 2024-03-24T00:32:44.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12bc000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12b4071de0 con 0x7f12b415ffd0 2024-03-24T00:32:44.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12b9600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f12b415bd00 0x7f12b40ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12b8c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f12b40728d0 0x7f12b415f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38728/0 (socket says 172.21.15.40:38728) 2024-03-24T00:32:44.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12b8c00640 1 -- 172.21.15.40:0/1220528913 learned_addr learned my addr 172.21.15.40:0/1220528913 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:44.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12ba000640 1 --2- 172.21.15.40:0/1220528913 >> v2:172.21.15.53:3300/0 conn(0x7f12b415ffd0 0x7f12b40cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12b9600640 1 -- 172.21.15.40:0/1220528913 >> v2:172.21.15.53:3300/0 conn(0x7f12b415ffd0 msgr2=0x7f12b40cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:44.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12b9600640 1 --2- 172.21.15.40:0/1220528913 >> v2:172.21.15.53:3300/0 conn(0x7f12b415ffd0 0x7f12b40cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12b9600640 1 -- 172.21.15.40:0/1220528913 >> v1:172.21.15.40:6789/0 conn(0x7f12b40728d0 legacy=0x7f12b415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12b9600640 1 -- 172.21.15.40:0/1220528913 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b40cea50 con 0x7f12b415bd00 2024-03-24T00:32:44.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.752+0000 7f12b3e00640 1 -- 172.21.15.40:0/1220528913 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975589542 0 0) 0x7f12b4072d60 con 0x7f12b40728d0 2024-03-24T00:32:44.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12b9600640 1 --2- 172.21.15.40:0/1220528913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12b415bd00 0x7f12b40ae8e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f12b4070a80 tx=0x7f12a801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3fabbab54b2251 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:44.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12b3e00640 1 -- 172.21.15.40:0/1220528913 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a80ab650 con 0x7f12b415bd00 2024-03-24T00:32:44.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12b3e00640 1 -- 172.21.15.40:0/1220528913 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a80b0070 con 0x7f12b415bd00 2024-03-24T00:32:44.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12b3e00640 1 -- 172.21.15.40:0/1220528913 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a80aba10 con 0x7f12b415bd00 2024-03-24T00:32:44.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12bc000640 1 -- 172.21.15.40:0/1220528913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12b415bd00 msgr2=0x7f12b40ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12bc000640 1 --2- 172.21.15.40:0/1220528913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12b415bd00 0x7f12b40ae8e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f12b4070a80 tx=0x7f12a801c450 comp rx=0 tx=0).stop 2024-03-24T00:32:44.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12bc000640 1 -- 172.21.15.40:0/1220528913 shutdown_connections 2024-03-24T00:32:44.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12bc000640 1 --2- 172.21.15.40:0/1220528913 >> v2:172.21.15.53:3300/0 conn(0x7f12b415ffd0 0x7f12b40cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12bc000640 1 --2- 172.21.15.40:0/1220528913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12b415bd00 0x7f12b40ae8e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12bc000640 1 -- 172.21.15.40:0/1220528913 >> 172.21.15.40:0/1220528913 conn(0x7f12b40b1660 msgr2=0x7f12b40b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:44.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12bc000640 1 -- 172.21.15.40:0/1220528913 shutdown_connections 2024-03-24T00:32:44.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12bc000640 1 -- 172.21.15.40:0/1220528913 wait complete. 2024-03-24T00:32:44.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.756+0000 7f12bc000640 1 Processor -- start 2024-03-24T00:32:44.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12bc000640 1 -- start start 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12bc000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12b40728d0 0x7f12b4159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12bc000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12b415bd00 0x7f12b413e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12bc000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12b415ffd0 0x7f12b41415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12bc000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b406d320 con 0x7f12b40728d0 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12bc000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b415c0e0 con 0x7f12b415bd00 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12bc000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12b406d4a0 con 0x7f12b415ffd0 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12b40728d0 0x7f12b4159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12b40728d0 0x7f12b4159530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49368/0 (socket says 172.21.15.40:49368) 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b9600640 1 -- 172.21.15.40:0/2719748157 learned_addr learned my addr 172.21.15.40:0/2719748157 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12ba000640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12b415ffd0 0x7f12b41415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b8c00640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12b415bd00 0x7f12b413e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b9600640 1 -- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12b415ffd0 msgr2=0x7f12b41415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b9600640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12b415ffd0 0x7f12b41415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b9600640 1 -- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12b415bd00 msgr2=0x7f12b413e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b9600640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12b415bd00 0x7f12b413e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b9600640 1 -- 172.21.15.40:0/2719748157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4141b00 con 0x7f12b40728d0 2024-03-24T00:32:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b8c00640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12b415bd00 0x7f12b413e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b9600640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12b40728d0 0x7f12b4159530 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f12a801a080 tx=0x7f12a801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b1600640 1 -- 172.21.15.40:0/2719748157 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8004230 con 0x7f12b40728d0 2024-03-24T00:32:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b1600640 1 -- 172.21.15.40:0/2719748157 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a80043d0 con 0x7f12b40728d0 2024-03-24T00:32:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12b1600640 1 -- 172.21.15.40:0/2719748157 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a8003820 con 0x7f12b40728d0 2024-03-24T00:32:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12bc000640 1 -- 172.21.15.40:0/2719748157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b4159b60 con 0x7f12b40728d0 2024-03-24T00:32:44.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12bc000640 1 -- 172.21.15.40:0/2719748157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b415a120 con 0x7f12b40728d0 2024-03-24T00:32:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.760+0000 7f12bc000640 1 -- 172.21.15.40:0/2719748157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b4068170 con 0x7f12b40728d0 2024-03-24T00:32:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.764+0000 7f12b1600640 1 -- 172.21.15.40:0/2719748157 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f12a80039c0 con 0x7f12b40728d0 2024-03-24T00:32:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.764+0000 7f12b1600640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1294067660 0x7f1294069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:44.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.764+0000 7f12b1600640 1 -- 172.21.15.40:0/2719748157 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f12a81205b0 con 0x7f12b40728d0 2024-03-24T00:32:44.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.764+0000 7f12b8c00640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1294067660 0x7f1294069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:44.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.764+0000 7f12b8c00640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1294067660 0x7f1294069b20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f12a0007fd0 tx=0x7f12a0007c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:44.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.768+0000 7f12b1600640 1 -- 172.21.15.40:0/2719748157 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f12a8016070 con 0x7f12b40728d0 2024-03-24T00:32:44.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:44.968+0000 7f12bc000640 1 -- 172.21.15.40:0/2719748157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f12b406e870 con 0x7f12b40728d0 2024-03-24T00:32:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.572+0000 7f12b1600640 1 -- 172.21.15.40:0/2719748157 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f12a80f1090 con 0x7f12b40728d0 2024-03-24T00:32:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.576+0000 7f12bc000640 1 -- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1294067660 msgr2=0x7f1294069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.576+0000 7f12bc000640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1294067660 0x7f1294069b20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f12a0007fd0 tx=0x7f12a0007c20 comp rx=0 tx=0).stop 2024-03-24T00:32:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.576+0000 7f12bc000640 1 -- 172.21.15.40:0/2719748157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12b40728d0 msgr2=0x7f12b4159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.576+0000 7f12bc000640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12b40728d0 0x7f12b4159530 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f12a801a080 tx=0x7f12a801ac90 comp rx=0 tx=0).stop 2024-03-24T00:32:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.576+0000 7f12bc000640 1 -- 172.21.15.40:0/2719748157 shutdown_connections 2024-03-24T00:32:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.576+0000 7f12bc000640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1294067660 0x7f1294069b20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.576+0000 7f12bc000640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12b415ffd0 0x7f12b41415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.576+0000 7f12bc000640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12b415bd00 0x7f12b413e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.576+0000 7f12bc000640 1 --2- 172.21.15.40:0/2719748157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12b40728d0 0x7f12b4159530 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.576+0000 7f12bc000640 1 -- 172.21.15.40:0/2719748157 >> 172.21.15.40:0/2719748157 conn(0x7f12b40b1660 msgr2=0x7f12b40b3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:45.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.576+0000 7f12bc000640 1 -- 172.21.15.40:0/2719748157 shutdown_connections 2024-03-24T00:32:45.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.576+0000 7f12bc000640 1 -- 172.21.15.40:0/2719748157 wait complete. 2024-03-24T00:32:45.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:32:45.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f762dc00640 1 Processor -- start 2024-03-24T00:32:45.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f762dc00640 1 -- start start 2024-03-24T00:32:45.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f762dc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f762815e350 0x7f76280cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:45.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f762dc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f76280c9580 0x7f76280c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:45.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f762dc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76280734e0 con 0x7f76280ad1e0 2024-03-24T00:32:45.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f762dc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76280cd660 con 0x7f76280c9580 2024-03-24T00:32:45.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f762dc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76280c9f40 con 0x7f762815e350 2024-03-24T00:32:45.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f7626a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f762815e350 0x7f76280cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:45.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f761e000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f76280c9580 0x7f76280c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:45.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f7627400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f76280ad1e0 0x7f76280ad610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:46106/0 (socket says 172.21.15.40:46106) 2024-03-24T00:32:45.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f761e000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f76280c9580 0x7f76280c9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:52524/0 (socket says 172.21.15.40:52524) 2024-03-24T00:32:45.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f7627400640 1 -- 172.21.15.40:0/2563187368 learned_addr learned my addr 172.21.15.40:0/2563187368 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:45.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f7626a00640 1 -- 172.21.15.40:0/2563187368 >> v2:172.21.15.53:3300/0 conn(0x7f76280c9580 msgr2=0x7f76280c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:45.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f7626a00640 1 --2- 172.21.15.40:0/2563187368 >> v2:172.21.15.53:3300/0 conn(0x7f76280c9580 0x7f76280c9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:45.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f7626a00640 1 -- 172.21.15.40:0/2563187368 >> v1:172.21.15.53:6789/0 conn(0x7f76280ad1e0 legacy=0x7f76280ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:45.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f7626a00640 1 -- 172.21.15.40:0/2563187368 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76280d0030 con 0x7f762815e350 2024-03-24T00:32:45.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.792+0000 7f761e000640 1 --2- 172.21.15.40:0/2563187368 >> v2:172.21.15.53:3300/0 conn(0x7f76280c9580 0x7f76280c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:45.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.796+0000 7f7626a00640 1 --2- 172.21.15.40:0/2563187368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f762815e350 0x7f76280cfaf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f76280d0200 tx=0x7f761801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8789e7223f258c66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:45.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.796+0000 7f7626000640 1 -- 172.21.15.40:0/2563187368 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76180b1070 con 0x7f762815e350 2024-03-24T00:32:45.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.796+0000 7f7626000640 1 -- 172.21.15.40:0/2563187368 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76180ac410 con 0x7f762815e350 2024-03-24T00:32:45.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.796+0000 7f7626000640 1 -- 172.21.15.40:0/2563187368 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76180ac7e0 con 0x7f762815e350 2024-03-24T00:32:45.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.796+0000 7f762dc00640 1 -- 172.21.15.40:0/2563187368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f762815e350 msgr2=0x7f76280cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:45.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.796+0000 7f762dc00640 1 --2- 172.21.15.40:0/2563187368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f762815e350 0x7f76280cfaf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f76280d0200 tx=0x7f761801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:32:45.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.796+0000 7f762dc00640 1 -- 172.21.15.40:0/2563187368 shutdown_connections 2024-03-24T00:32:45.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.796+0000 7f762dc00640 1 --2- 172.21.15.40:0/2563187368 >> v2:172.21.15.53:3300/0 conn(0x7f76280c9580 0x7f76280c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:45.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.796+0000 7f762dc00640 1 --2- 172.21.15.40:0/2563187368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f762815e350 0x7f76280cfaf0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:45.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.796+0000 7f762dc00640 1 -- 172.21.15.40:0/2563187368 >> 172.21.15.40:0/2563187368 conn(0x7f7628072870 msgr2=0x7f7628072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:45.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.796+0000 7f762dc00640 1 -- 172.21.15.40:0/2563187368 shutdown_connections 2024-03-24T00:32:45.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.796+0000 7f762dc00640 1 -- 172.21.15.40:0/2563187368 wait complete. 2024-03-24T00:32:45.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f762dc00640 1 Processor -- start 2024-03-24T00:32:45.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f762dc00640 1 -- start start 2024-03-24T00:32:45.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f762dc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76280ad1e0 0x7f7628155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:45.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f762dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76280c9580 0x7f76281507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:45.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f762dc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f762815e350 0x7f762806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:45.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f762dc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76280716b0 con 0x7f76280c9580 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f762dc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7628071fe0 con 0x7f76280ad1e0 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f762dc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76280cda30 con 0x7f762815e350 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f761e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76280c9580 0x7f76281507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f761e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76280c9580 0x7f76281507c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49374/0 (socket says 172.21.15.40:49374) 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f761e000640 1 -- 172.21.15.40:0/3719915546 learned_addr learned my addr 172.21.15.40:0/3719915546 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f7627400640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f762815e350 0x7f762806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f7626a00640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76280ad1e0 0x7f7628155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f761e000640 1 -- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f762815e350 msgr2=0x7f762806d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f761e000640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f762815e350 0x7f762806d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f761e000640 1 -- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76280ad1e0 msgr2=0x7f7628155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f761e000640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76280ad1e0 0x7f7628155580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f761e000640 1 -- 172.21.15.40:0/3719915546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76280b36c0 con 0x7f76280c9580 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f7626a00640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76280ad1e0 0x7f7628155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:45.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f761e000640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76280c9580 0x7f76281507c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f761000ec10 tx=0x7f761000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:45.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f761f400640 1 -- 172.21.15.40:0/3719915546 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610017070 con 0x7f76280c9580 2024-03-24T00:32:45.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f761f400640 1 -- 172.21.15.40:0/3719915546 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761000cc30 con 0x7f76280c9580 2024-03-24T00:32:45.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f762dc00640 1 -- 172.21.15.40:0/3719915546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76280b39a0 con 0x7f76280c9580 2024-03-24T00:32:45.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f761f400640 1 -- 172.21.15.40:0/3719915546 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610013470 con 0x7f76280c9580 2024-03-24T00:32:45.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.800+0000 7f762dc00640 1 -- 172.21.15.40:0/3719915546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76280b3f60 con 0x7f76280c9580 2024-03-24T00:32:45.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.804+0000 7f762dc00640 1 -- 172.21.15.40:0/3719915546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7628068170 con 0x7f76280c9580 2024-03-24T00:32:45.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.804+0000 7f761f400640 1 -- 172.21.15.40:0/3719915546 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f761001a070 con 0x7f76280c9580 2024-03-24T00:32:45.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.804+0000 7f761f400640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7604067690 0x7f7604069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:45.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.804+0000 7f761f400640 1 -- 172.21.15.40:0/3719915546 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f7610085e30 con 0x7f76280c9580 2024-03-24T00:32:45.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.804+0000 7f7626a00640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7604067690 0x7f7604069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:45.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.804+0000 7f7626a00640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7604067690 0x7f7604069b50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f76180ae000 tx=0x7f7618008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:45.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.808+0000 7f761f400640 1 -- 172.21.15.40:0/3719915546 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7610056880 con 0x7f76280c9580 2024-03-24T00:32:45.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:45.972+0000 7f762dc00640 1 -- 172.21.15.40:0/3719915546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7628156cd0 con 0x7f76280c9580 2024-03-24T00:32:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.576+0000 7f761f400640 1 -- 172.21.15.40:0/3719915546 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7610056180 con 0x7f76280c9580 2024-03-24T00:32:46.578 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:32:46.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.580+0000 7f762dc00640 1 -- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7604067690 msgr2=0x7f7604069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.580+0000 7f762dc00640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7604067690 0x7f7604069b50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f76180ae000 tx=0x7f7618008040 comp rx=0 tx=0).stop 2024-03-24T00:32:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.580+0000 7f762dc00640 1 -- 172.21.15.40:0/3719915546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76280c9580 msgr2=0x7f76281507c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.580+0000 7f762dc00640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76280c9580 0x7f76281507c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f761000ec10 tx=0x7f761000c520 comp rx=0 tx=0).stop 2024-03-24T00:32:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.580+0000 7f762dc00640 1 -- 172.21.15.40:0/3719915546 shutdown_connections 2024-03-24T00:32:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.580+0000 7f762dc00640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7604067690 0x7f7604069b50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.580+0000 7f762dc00640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f762815e350 0x7f762806d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.580+0000 7f762dc00640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f76280c9580 0x7f76281507c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.580+0000 7f762dc00640 1 --2- 172.21.15.40:0/3719915546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76280ad1e0 0x7f7628155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.580+0000 7f762dc00640 1 -- 172.21.15.40:0/3719915546 >> 172.21.15.40:0/3719915546 conn(0x7f7628072870 msgr2=0x7f76281571f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.580+0000 7f762dc00640 1 -- 172.21.15.40:0/3719915546 shutdown_connections 2024-03-24T00:32:46.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.580+0000 7f762dc00640 1 -- 172.21.15.40:0/3719915546 wait complete. 2024-03-24T00:32:46.604 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:32:46.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.748+0000 7fbc82400640 1 Processor -- start 2024-03-24T00:32:46.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.748+0000 7fbc82400640 1 -- start start 2024-03-24T00:32:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc82400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c166c10 0x7fbc7c169000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc82400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7c0b14b0 con 0x7fbc7c06b150 2024-03-24T00:32:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc82400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7c0b1790 con 0x7fbc7c06b530 2024-03-24T00:32:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc82400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c0b20c0 con 0x7fbc7c166c10 2024-03-24T00:32:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbc7c06b150 0x7fbc7c0d09c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38742/0 (socket says 172.21.15.40:38742) 2024-03-24T00:32:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7b400640 1 -- 172.21.15.40:0/3615903481 learned_addr learned my addr 172.21.15.40:0/3615903481 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7be00640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c166c10 0x7fbc7c169000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3574967286 0 0) 0x7fbc7c0b14b0 con 0x7fbc7c06b150 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc60003530 con 0x7fbc7c06b150 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237419207 0 0) 0x7fbc7c0b1790 con 0x7fbc7c06b530 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc7c0b14b0 con 0x7fbc7c06b530 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 6544235 0 0) 0x7fbc60003530 con 0x7fbc7c06b150 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c166c10 msgr2=0x7fbc7c169000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c166c10 0x7fbc7c169000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 >> v1:172.21.15.53:6789/0 conn(0x7fbc7c06b530 legacy=0x7fbc7c06b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c0b1a70 con 0x7fbc7c06b150 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fbc6c001680 con 0x7fbc7c06b150 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 >> v1:172.21.15.40:6789/0 conn(0x7fbc7c06b150 legacy=0x7fbc7c0d09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc60003b80 0x7fbc60003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:46.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c166c10 0x7fbc60004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc60004d50 0x7fbc600051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c0b20c0 con 0x7fbc60003b80 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6000ba80 con 0x7fbc60004d50 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6000bc80 con 0x7fbc7c166c10 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7b400640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc60003b80 0x7fbc60003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7be00640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc60004d50 0x7fbc600051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc73e00640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c166c10 0x7fbc60004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7b400640 1 -- 172.21.15.40:0/3615903481 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c166c10 msgr2=0x7fbc60004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7b400640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c166c10 0x7fbc60004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7b400640 1 -- 172.21.15.40:0/3615903481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc60004d50 msgr2=0x7fbc600051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7b400640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc60004d50 0x7fbc600051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7b400640 1 -- 172.21.15.40:0/3615903481 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc60003530 con 0x7fbc60003b80 2024-03-24T00:32:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.752+0000 7fbc7be00640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc60004d50 0x7fbc600051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.756+0000 7fbc7b400640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc60003b80 0x7fbc60003f80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00c0e0 tx=0x7fbc6c01cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31f130695b7eb186 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.756+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c007c50 con 0x7fbc60003b80 2024-03-24T00:32:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.756+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6c0a9070 con 0x7fbc60003b80 2024-03-24T00:32:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.756+0000 7fbc7aa00640 1 -- 172.21.15.40:0/3615903481 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c005d60 con 0x7fbc60003b80 2024-03-24T00:32:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.756+0000 7fbc82400640 1 -- 172.21.15.40:0/3615903481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc60003b80 msgr2=0x7fbc60003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.756+0000 7fbc82400640 1 --2- 172.21.15.40:0/3615903481 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc60003b80 0x7fbc60003f80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00c0e0 tx=0x7fbc6c01cb60 comp rx=0 tx=0).stop 2024-03-24T00:32:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.756+0000 7fbc7be00640 1 -- 172.21.15.40:0/3615903481 reap_dead start 2024-03-24T00:32:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.756+0000 7fbc82400640 1 -- 172.21.15.40:0/3615903481 shutdown_connections 2024-03-24T00:32:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.756+0000 7fbc82400640 1 -- 172.21.15.40:0/3615903481 >> 172.21.15.40:0/3615903481 conn(0x7fbc7c0b36a0 msgr2=0x7fbc7c0b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.756+0000 7fbc82400640 1 -- 172.21.15.40:0/3615903481 shutdown_connections 2024-03-24T00:32:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.756+0000 7fbc82400640 1 -- 172.21.15.40:0/3615903481 wait complete. 2024-03-24T00:32:46.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.756+0000 7fbc82400640 1 Processor -- start 2024-03-24T00:32:46.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc82400640 1 -- start start 2024-03-24T00:32:46.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc82400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc60003b80 0x7fbc7c153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:46.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc82400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc7c0aec80 0x7fbc7c0af100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:46.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc82400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c157420 0x7fbc7c1578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:46.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc82400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c0aeb00 con 0x7fbc60003b80 2024-03-24T00:32:46.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc82400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c0ae980 con 0x7fbc7c0aec80 2024-03-24T00:32:46.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc82400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7c169810 con 0x7fbc7c157420 2024-03-24T00:32:46.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc7b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc60003b80 0x7fbc7c153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:46.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc7b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc60003b80 0x7fbc7c153ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49388/0 (socket says 172.21.15.40:49388) 2024-03-24T00:32:46.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc7b400640 1 -- 172.21.15.40:0/972954779 learned_addr learned my addr 172.21.15.40:0/972954779 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:46.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc7be00640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c157420 0x7fbc7c1578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc73e00640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc7c0aec80 0x7fbc7c0af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc7b400640 1 -- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c157420 msgr2=0x7fbc7c1578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc7b400640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c157420 0x7fbc7c1578a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc7b400640 1 -- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc7c0aec80 msgr2=0x7fbc7c0af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc7b400640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc7c0aec80 0x7fbc7c0af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc7b400640 1 -- 172.21.15.40:0/972954779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c157de0 con 0x7fbc60003b80 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc7be00640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c157420 0x7fbc7c1578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc73e00640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc7c0aec80 0x7fbc7c0af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc7b400640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc60003b80 0x7fbc7c153ce0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00c0e0 tx=0x7fbc60005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc73400640 1 -- 172.21.15.40:0/972954779 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c001480 con 0x7fbc60003b80 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc73400640 1 -- 172.21.15.40:0/972954779 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc6c001620 con 0x7fbc60003b80 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc73400640 1 -- 172.21.15.40:0/972954779 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0a5950 con 0x7fbc60003b80 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc82400640 1 -- 172.21.15.40:0/972954779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc7c154220 con 0x7fbc60003b80 2024-03-24T00:32:46.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc82400640 1 -- 172.21.15.40:0/972954779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7c0cc640 con 0x7fbc60003b80 2024-03-24T00:32:46.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.760+0000 7fbc82400640 1 -- 172.21.15.40:0/972954779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc7c068170 con 0x7fbc60003b80 2024-03-24T00:32:46.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.764+0000 7fbc73400640 1 -- 172.21.15.40:0/972954779 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbc6c0a5af0 con 0x7fbc60003b80 2024-03-24T00:32:46.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.764+0000 7fbc73400640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc58067660 0x7fbc58069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:46.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.764+0000 7fbc73400640 1 -- 172.21.15.40:0/972954779 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5529+0+0 (secure 0 0 0) 0x7fbc6c11edc0 con 0x7fbc60003b80 2024-03-24T00:32:46.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.764+0000 7fbc73e00640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc58067660 0x7fbc58069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:46.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.764+0000 7fbc73e00640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc58067660 0x7fbc58069b20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbc64007fd0 tx=0x7fbc64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:46.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.768+0000 7fbc73400640 1 -- 172.21.15.40:0/972954779 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbc6c0c0030 con 0x7fbc60003b80 2024-03-24T00:32:46.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:46.932+0000 7fbc82400640 1 -- 172.21.15.40:0/972954779 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbc7c0d0c40 con 0x7fbc60003b80 2024-03-24T00:32:47.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.576+0000 7fbc73400640 1 -- 172.21.15.40:0/972954779 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbc6c016870 con 0x7fbc60003b80 2024-03-24T00:32:47.580 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-03-24T00:32:47.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.580+0000 7fbc82400640 1 -- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc58067660 msgr2=0x7fbc58069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:47.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.580+0000 7fbc82400640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc58067660 0x7fbc58069b20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbc64007fd0 tx=0x7fbc64009040 comp rx=0 tx=0).stop 2024-03-24T00:32:47.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.580+0000 7fbc82400640 1 -- 172.21.15.40:0/972954779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc60003b80 msgr2=0x7fbc7c153ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:47.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.580+0000 7fbc82400640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc60003b80 0x7fbc7c153ce0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c00c0e0 tx=0x7fbc60005840 comp rx=0 tx=0).stop 2024-03-24T00:32:47.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.580+0000 7fbc82400640 1 -- 172.21.15.40:0/972954779 shutdown_connections 2024-03-24T00:32:47.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.580+0000 7fbc82400640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc58067660 0x7fbc58069b20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.580+0000 7fbc82400640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc7c157420 0x7fbc7c1578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.580+0000 7fbc82400640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc7c0aec80 0x7fbc7c0af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.580+0000 7fbc82400640 1 --2- 172.21.15.40:0/972954779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc60003b80 0x7fbc7c153ce0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.580+0000 7fbc82400640 1 -- 172.21.15.40:0/972954779 >> 172.21.15.40:0/972954779 conn(0x7fbc7c0b36a0 msgr2=0x7fbc7c06edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:47.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.580+0000 7fbc82400640 1 -- 172.21.15.40:0/972954779 shutdown_connections 2024-03-24T00:32:47.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.580+0000 7fbc82400640 1 -- 172.21.15.40:0/972954779 wait complete. 2024-03-24T00:32:47.606 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random 2024-03-24T00:32:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f0973600640 1 Processor -- start 2024-03-24T00:32:47.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f0973600640 1 -- start start 2024-03-24T00:32:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f0973600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f096c0b2960 0x7f096c0727d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f0973600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f096c0b39e0 0x7f096c0b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f0973600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f096c0b4790 con 0x7f096c0b3280 2024-03-24T00:32:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f0973600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f096c071000 con 0x7f096c0b39e0 2024-03-24T00:32:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f0973600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096c0b4a70 con 0x7f096c0b2960 2024-03-24T00:32:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f096be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f096c0b39e0 0x7f096c0b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:47.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f0971600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f096c0b3280 0x7f096c06cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56618/0 (socket says 172.21.15.40:56618) 2024-03-24T00:32:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f0971600640 1 -- 172.21.15.40:0/1731320231 learned_addr learned my addr 172.21.15.40:0/1731320231 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f0970c00640 1 --2- 172.21.15.40:0/1731320231 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f096c0b2960 0x7f096c0727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f096be00640 1 -- 172.21.15.40:0/1731320231 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f096c0b2960 msgr2=0x7f096c0727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f096be00640 1 --2- 172.21.15.40:0/1731320231 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f096c0b2960 0x7f096c0727d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f096be00640 1 -- 172.21.15.40:0/1731320231 >> v1:172.21.15.40:6789/0 conn(0x7f096c0b3280 legacy=0x7f096c06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:47.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f096be00640 1 -- 172.21.15.40:0/1731320231 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096c070c90 con 0x7f096c0b39e0 2024-03-24T00:32:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f096b400640 1 -- 172.21.15.40:0/1731320231 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284761781 0 0) 0x7f096c0b4790 con 0x7f096c0b3280 2024-03-24T00:32:47.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.748+0000 7f0970c00640 1 --2- 172.21.15.40:0/1731320231 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f096c0b2960 0x7f096c0727d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:47.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f096be00640 1 --2- 172.21.15.40:0/1731320231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c0b39e0 0x7f096c0b2d40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f09580097c0 tx=0x7f095801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70a82a369e6ca6b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f096b400640 1 -- 172.21.15.40:0/1731320231 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09580b0070 con 0x7f096c0b39e0 2024-03-24T00:32:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f096b400640 1 -- 172.21.15.40:0/1731320231 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09580ab600 con 0x7f096c0b39e0 2024-03-24T00:32:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f096b400640 1 -- 172.21.15.40:0/1731320231 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09580ab9d0 con 0x7f096c0b39e0 2024-03-24T00:32:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f0973600640 1 -- 172.21.15.40:0/1731320231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c0b39e0 msgr2=0x7f096c0b2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:47.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f0973600640 1 --2- 172.21.15.40:0/1731320231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c0b39e0 0x7f096c0b2d40 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f09580097c0 tx=0x7f095801c380 comp rx=0 tx=0).stop 2024-03-24T00:32:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f0973600640 1 -- 172.21.15.40:0/1731320231 shutdown_connections 2024-03-24T00:32:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f0973600640 1 --2- 172.21.15.40:0/1731320231 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c0b39e0 0x7f096c0b2d40 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f0973600640 1 --2- 172.21.15.40:0/1731320231 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f096c0b2960 0x7f096c0727d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f0973600640 1 -- 172.21.15.40:0/1731320231 >> 172.21.15.40:0/1731320231 conn(0x7f096c0b1660 msgr2=0x7f096c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f0973600640 1 -- 172.21.15.40:0/1731320231 shutdown_connections 2024-03-24T00:32:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f0973600640 1 -- 172.21.15.40:0/1731320231 wait complete. 2024-03-24T00:32:47.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f0973600640 1 Processor -- start 2024-03-24T00:32:47.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.752+0000 7f0973600640 1 -- start start 2024-03-24T00:32:47.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0973600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f096c0b2960 0x7f096c164ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:47.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0973600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c0b3280 0x7f096c152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0973600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f096c0b39e0 0x7f096c0afde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0973600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c06ddc0 con 0x7f096c0b3280 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0973600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096c06dc40 con 0x7f096c0b39e0 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0973600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096c16ee50 con 0x7f096c0b2960 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f096be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c0b3280 0x7f096c152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f096be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c0b3280 0x7f096c152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42628/0 (socket says 172.21.15.40:42628) 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f096be00640 1 -- 172.21.15.40:0/631960830 learned_addr learned my addr 172.21.15.40:0/631960830 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0970c00640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f096c0b2960 0x7f096c164ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0971600640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f096c0b39e0 0x7f096c0afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f096be00640 1 -- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f096c0b2960 msgr2=0x7f096c164ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f096be00640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f096c0b2960 0x7f096c164ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f096be00640 1 -- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f096c0b39e0 msgr2=0x7f096c0afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f096be00640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f096c0b39e0 0x7f096c0afde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f096be00640 1 -- 172.21.15.40:0/631960830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096c153220 con 0x7f096c0b3280 2024-03-24T00:32:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0970c00640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f096c0b2960 0x7f096c164ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0971600640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f096c0b39e0 0x7f096c0afde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f096be00640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c0b3280 0x7f096c152ce0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0958019fb0 tx=0x7f0958019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0968c00640 1 -- 172.21.15.40:0/631960830 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09580b0040 con 0x7f096c0b3280 2024-03-24T00:32:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0968c00640 1 -- 172.21.15.40:0/631960830 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09580b43f0 con 0x7f096c0b3280 2024-03-24T00:32:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0973600640 1 -- 172.21.15.40:0/631960830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f096c15af40 con 0x7f096c0b3280 2024-03-24T00:32:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0968c00640 1 -- 172.21.15.40:0/631960830 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09580b47c0 con 0x7f096c0b3280 2024-03-24T00:32:47.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0973600640 1 -- 172.21.15.40:0/631960830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096c15a6a0 con 0x7f096c0b3280 2024-03-24T00:32:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.756+0000 7f0973600640 1 -- 172.21.15.40:0/631960830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f096c068170 con 0x7f096c0b3280 2024-03-24T00:32:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.760+0000 7f0968c00640 1 -- 172.21.15.40:0/631960830 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f09580b4960 con 0x7f096c0b3280 2024-03-24T00:32:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.760+0000 7f0968c00640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f094c067660 0x7f094c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.760+0000 7f0968c00640 1 -- 172.21.15.40:0/631960830 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f095811f110 con 0x7f096c0b3280 2024-03-24T00:32:47.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.760+0000 7f0970c00640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f094c067660 0x7f094c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.760+0000 7f0970c00640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f094c067660 0x7f094c069b20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0960009980 tx=0x7f0960008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:47.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.764+0000 7f0968c00640 1 -- 172.21.15.40:0/631960830 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0958016090 con 0x7f096c0b3280 2024-03-24T00:32:47.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.924+0000 7f0973600640 1 -- 172.21.15.40:0/631960830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"} v 0) v1 -- 0x7f096c1565e0 con 0x7f096c0b3280 2024-03-24T00:32:47.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.928+0000 7f0968c00640 1 -- 172.21.15.40:0/631960830 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"}]=0 v15) v1 ==== 110+0+0 (secure 0 0 0) 0x7f09580effd0 con 0x7f096c0b3280 2024-03-24T00:32:47.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.928+0000 7f0973600640 1 -- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f094c067660 msgr2=0x7f094c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:47.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.928+0000 7f0973600640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f094c067660 0x7f094c069b20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0960009980 tx=0x7f0960008040 comp rx=0 tx=0).stop 2024-03-24T00:32:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.928+0000 7f0973600640 1 -- 172.21.15.40:0/631960830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c0b3280 msgr2=0x7f096c152ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.928+0000 7f0973600640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c0b3280 0x7f096c152ce0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0958019fb0 tx=0x7f0958019120 comp rx=0 tx=0).stop 2024-03-24T00:32:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.928+0000 7f0973600640 1 -- 172.21.15.40:0/631960830 shutdown_connections 2024-03-24T00:32:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.928+0000 7f0973600640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f094c067660 0x7f094c069b20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.928+0000 7f0973600640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f096c0b39e0 0x7f096c0afde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.928+0000 7f0973600640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f096c0b3280 0x7f096c152ce0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.928+0000 7f0973600640 1 --2- 172.21.15.40:0/631960830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f096c0b2960 0x7f096c164ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.928+0000 7f0973600640 1 -- 172.21.15.40:0/631960830 >> 172.21.15.40:0/631960830 conn(0x7f096c0b1660 msgr2=0x7f096c1596b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.932+0000 7f0973600640 1 -- 172.21.15.40:0/631960830 shutdown_connections 2024-03-24T00:32:47.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:47.932+0000 7f0973600640 1 -- 172.21.15.40:0/631960830 wait complete. 2024-03-24T00:32:47.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_distributed 2024-03-24T00:32:48.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc645e00640 1 Processor -- start 2024-03-24T00:32:48.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc645e00640 1 -- start start 2024-03-24T00:32:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc645e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc64015e000 0x7fc6400d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc645e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6400b42d0 0x7fc64006dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc645e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64015e3e0 con 0x7fc6401628d0 2024-03-24T00:32:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc645e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc640161ba0 con 0x7fc64015e000 2024-03-24T00:32:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc645e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6400b5f10 con 0x7fc6400b42d0 2024-03-24T00:32:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc63ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc64015e000 0x7fc6400d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc636000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc6401628d0 0x7fc6400b3bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56620/0 (socket says 172.21.15.40:56620) 2024-03-24T00:32:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc636000640 1 -- 172.21.15.40:0/827080638 learned_addr learned my addr 172.21.15.40:0/827080638 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:48.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc63f400640 1 --2- 172.21.15.40:0/827080638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6400b42d0 0x7fc64006dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc63ea00640 1 -- 172.21.15.40:0/827080638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6400b42d0 msgr2=0x7fc64006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc63ea00640 1 --2- 172.21.15.40:0/827080638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6400b42d0 0x7fc64006dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc63ea00640 1 -- 172.21.15.40:0/827080638 >> v1:172.21.15.40:6789/0 conn(0x7fc6401628d0 legacy=0x7fc6400b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc63ea00640 1 -- 172.21.15.40:0/827080638 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc640161df0 con 0x7fc64015e000 2024-03-24T00:32:48.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.092+0000 7fc63e000640 1 -- 172.21.15.40:0/827080638 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506166880 0 0) 0x7fc64015e3e0 con 0x7fc6401628d0 2024-03-24T00:32:48.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.096+0000 7fc63ea00640 1 --2- 172.21.15.40:0/827080638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64015e000 0x7fc6400d16d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc6280097c0 tx=0x7fc62801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e964ee1f1d0a7380 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:48.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.096+0000 7fc63e000640 1 -- 172.21.15.40:0/827080638 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6280b0070 con 0x7fc64015e000 2024-03-24T00:32:48.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.096+0000 7fc63e000640 1 -- 172.21.15.40:0/827080638 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6280ab600 con 0x7fc64015e000 2024-03-24T00:32:48.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.096+0000 7fc63e000640 1 -- 172.21.15.40:0/827080638 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6280ab9d0 con 0x7fc64015e000 2024-03-24T00:32:48.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.096+0000 7fc645e00640 1 -- 172.21.15.40:0/827080638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64015e000 msgr2=0x7fc6400d16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.096+0000 7fc645e00640 1 --2- 172.21.15.40:0/827080638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64015e000 0x7fc6400d16d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc6280097c0 tx=0x7fc62801c470 comp rx=0 tx=0).stop 2024-03-24T00:32:48.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.096+0000 7fc645e00640 1 -- 172.21.15.40:0/827080638 shutdown_connections 2024-03-24T00:32:48.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.096+0000 7fc645e00640 1 --2- 172.21.15.40:0/827080638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6400b42d0 0x7fc64006dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.096+0000 7fc645e00640 1 --2- 172.21.15.40:0/827080638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64015e000 0x7fc6400d16d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.096+0000 7fc645e00640 1 -- 172.21.15.40:0/827080638 >> 172.21.15.40:0/827080638 conn(0x7fc6400d0bb0 msgr2=0x7fc6400d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:48.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.096+0000 7fc645e00640 1 -- 172.21.15.40:0/827080638 shutdown_connections 2024-03-24T00:32:48.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.096+0000 7fc645e00640 1 -- 172.21.15.40:0/827080638 wait complete. 2024-03-24T00:32:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc645e00640 1 Processor -- start 2024-03-24T00:32:48.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc645e00640 1 -- start start 2024-03-24T00:32:48.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc645e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6400b42d0 0x7fc640149150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc645e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64015e000 0x7fc6400af480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc645e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6401628d0 0x7fc6400af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc645e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64016b0c0 con 0x7fc64015e000 2024-03-24T00:32:48.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc645e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc64016af40 con 0x7fc6401628d0 2024-03-24T00:32:48.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc645e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64016b240 con 0x7fc6400b42d0 2024-03-24T00:32:48.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc636000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64015e000 0x7fc6400af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc636000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64015e000 0x7fc6400af480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42638/0 (socket says 172.21.15.40:42638) 2024-03-24T00:32:48.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc636000640 1 -- 172.21.15.40:0/909742178 learned_addr learned my addr 172.21.15.40:0/909742178 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:48.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc63f400640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6401628d0 0x7fc6400af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc63ea00640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6400b42d0 0x7fc640149150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc636000640 1 -- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6400b42d0 msgr2=0x7fc640149150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc636000640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6400b42d0 0x7fc640149150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc636000640 1 -- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6401628d0 msgr2=0x7fc6400af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc636000640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6401628d0 0x7fc6400af9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc636000640 1 -- 172.21.15.40:0/909742178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64015e480 con 0x7fc64015e000 2024-03-24T00:32:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc636000640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64015e000 0x7fc6400af480 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c480 tx=0x7fc63000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc637e00640 1 -- 172.21.15.40:0/909742178 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630018070 con 0x7fc64015e000 2024-03-24T00:32:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc637e00640 1 -- 172.21.15.40:0/909742178 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc630007500 con 0x7fc64015e000 2024-03-24T00:32:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc637e00640 1 -- 172.21.15.40:0/909742178 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630004a30 con 0x7fc64015e000 2024-03-24T00:32:48.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc645e00640 1 -- 172.21.15.40:0/909742178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc64015e720 con 0x7fc64015e000 2024-03-24T00:32:48.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.100+0000 7fc645e00640 1 -- 172.21.15.40:0/909742178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc64015ece0 con 0x7fc64015e000 2024-03-24T00:32:48.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.104+0000 7fc645e00640 1 -- 172.21.15.40:0/909742178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc640068170 con 0x7fc64015e000 2024-03-24T00:32:48.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.104+0000 7fc637e00640 1 -- 172.21.15.40:0/909742178 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc630004bd0 con 0x7fc64015e000 2024-03-24T00:32:48.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.104+0000 7fc637e00640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc61c067660 0x7fc61c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.104+0000 7fc637e00640 1 -- 172.21.15.40:0/909742178 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fc630084910 con 0x7fc64015e000 2024-03-24T00:32:48.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.104+0000 7fc63ea00640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc61c067660 0x7fc61c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.104+0000 7fc63ea00640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc61c067660 0x7fc61c069b20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc6280ad000 tx=0x7fc628008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:48.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.108+0000 7fc637e00640 1 -- 172.21.15.40:0/909742178 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc6300557d0 con 0x7fc64015e000 2024-03-24T00:32:48.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.276+0000 7fc645e00640 1 -- 172.21.15.40:0/909742178 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"} v 0) v1 -- 0x7fc640067b90 con 0x7fc64015e000 2024-03-24T00:32:48.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.276+0000 7fc637e00640 1 -- 172.21.15.40:0/909742178 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"}]=0 v16) v1 ==== 115+0+0 (secure 0 0 0) 0x7fc6300550d0 con 0x7fc64015e000 2024-03-24T00:32:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.280+0000 7fc645e00640 1 -- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc61c067660 msgr2=0x7fc61c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.280+0000 7fc645e00640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc61c067660 0x7fc61c069b20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc6280ad000 tx=0x7fc628008040 comp rx=0 tx=0).stop 2024-03-24T00:32:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.280+0000 7fc645e00640 1 -- 172.21.15.40:0/909742178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64015e000 msgr2=0x7fc6400af480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.280+0000 7fc645e00640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64015e000 0x7fc6400af480 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c480 tx=0x7fc63000c940 comp rx=0 tx=0).stop 2024-03-24T00:32:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.280+0000 7fc645e00640 1 -- 172.21.15.40:0/909742178 shutdown_connections 2024-03-24T00:32:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.280+0000 7fc645e00640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc61c067660 0x7fc61c069b20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.280+0000 7fc645e00640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6401628d0 0x7fc6400af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.280+0000 7fc645e00640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc64015e000 0x7fc6400af480 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.280+0000 7fc645e00640 1 --2- 172.21.15.40:0/909742178 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6400b42d0 0x7fc640149150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.280+0000 7fc645e00640 1 -- 172.21.15.40:0/909742178 >> 172.21.15.40:0/909742178 conn(0x7fc6400d0bb0 msgr2=0x7fc640149e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.280+0000 7fc645e00640 1 -- 172.21.15.40:0/909742178 shutdown_connections 2024-03-24T00:32:48.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.280+0000 7fc645e00640 1 -- 172.21.15.40:0/909742178 wait complete. 2024-03-24T00:32:48.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random_max 2024-03-24T00:32:48.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe076000640 1 Processor -- start 2024-03-24T00:32:48.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe076000640 1 -- start start 2024-03-24T00:32:48.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe076000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe07015e000 0x7fe0700d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe076000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0701628d0 0x7fe0700b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe076000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe07006f840 0x7fe0700b4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe076000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe070161ba0 con 0x7fe0701628d0 2024-03-24T00:32:48.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe076000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe07015e3e0 con 0x7fe07015e000 2024-03-24T00:32:48.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe076000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0700b5f10 con 0x7fe07006f840 2024-03-24T00:32:48.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe066a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0701628d0 0x7fe0700b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe066a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0701628d0 0x7fe0700b3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42646/0 (socket says 172.21.15.40:42646) 2024-03-24T00:32:48.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe066a00640 1 -- 172.21.15.40:0/1239738453 learned_addr learned my addr 172.21.15.40:0/1239738453 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:48.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe06fe00640 1 --2- 172.21.15.40:0/1239738453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe07006f840 0x7fe0700b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe06f400640 1 --2- 172.21.15.40:0/1239738453 >> v2:172.21.15.53:3300/0 conn(0x7fe07015e000 0x7fe0700d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe066a00640 1 -- 172.21.15.40:0/1239738453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe07006f840 msgr2=0x7fe0700b4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:48.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe066a00640 1 --2- 172.21.15.40:0/1239738453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe07006f840 0x7fe0700b4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe066a00640 1 -- 172.21.15.40:0/1239738453 >> v2:172.21.15.53:3300/0 conn(0x7fe07015e000 msgr2=0x7fe0700d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:48.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe066a00640 1 --2- 172.21.15.40:0/1239738453 >> v2:172.21.15.53:3300/0 conn(0x7fe07015e000 0x7fe0700d16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe066a00640 1 -- 172.21.15.40:0/1239738453 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe070161df0 con 0x7fe0701628d0 2024-03-24T00:32:48.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe06fe00640 1 --2- 172.21.15.40:0/1239738453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe07006f840 0x7fe0700b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:48.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.452+0000 7fe06f400640 1 --2- 172.21.15.40:0/1239738453 >> v2:172.21.15.53:3300/0 conn(0x7fe07015e000 0x7fe0700d16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:48.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe066a00640 1 --2- 172.21.15.40:0/1239738453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0701628d0 0x7fe0700b3bc0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe0700b6120 tx=0x7fe05801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f37da5022dc58479 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:48.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe06ea00640 1 -- 172.21.15.40:0/1239738453 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0580aa610 con 0x7fe0701628d0 2024-03-24T00:32:48.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe06ea00640 1 -- 172.21.15.40:0/1239738453 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0580af070 con 0x7fe0701628d0 2024-03-24T00:32:48.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe06ea00640 1 -- 172.21.15.40:0/1239738453 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0580aa9d0 con 0x7fe0701628d0 2024-03-24T00:32:48.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe076000640 1 -- 172.21.15.40:0/1239738453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0701628d0 msgr2=0x7fe0700b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe076000640 1 --2- 172.21.15.40:0/1239738453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0701628d0 0x7fe0700b3bc0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe0700b6120 tx=0x7fe05801c4f0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe076000640 1 -- 172.21.15.40:0/1239738453 shutdown_connections 2024-03-24T00:32:48.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe076000640 1 --2- 172.21.15.40:0/1239738453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe07006f840 0x7fe0700b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe076000640 1 --2- 172.21.15.40:0/1239738453 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0701628d0 0x7fe0700b3bc0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe076000640 1 --2- 172.21.15.40:0/1239738453 >> v2:172.21.15.53:3300/0 conn(0x7fe07015e000 0x7fe0700d16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe076000640 1 -- 172.21.15.40:0/1239738453 >> 172.21.15.40:0/1239738453 conn(0x7fe0700d0bb0 msgr2=0x7fe0700d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:48.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe076000640 1 -- 172.21.15.40:0/1239738453 shutdown_connections 2024-03-24T00:32:48.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.456+0000 7fe076000640 1 -- 172.21.15.40:0/1239738453 wait complete. 2024-03-24T00:32:48.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe076000640 1 Processor -- start 2024-03-24T00:32:48.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe076000640 1 -- start start 2024-03-24T00:32:48.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe076000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe07006f840 0x7fe07015e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe076000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe07015e000 0x7fe070141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe076000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0701628d0 0x7fe07015bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe076000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0700725e0 con 0x7fe07006f840 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe076000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe070072460 con 0x7fe0701628d0 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe076000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe070072760 con 0x7fe07015e000 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe06f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe07006f840 0x7fe07015e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe06f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe07006f840 0x7fe07015e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42656/0 (socket says 172.21.15.40:42656) 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe06f400640 1 -- 172.21.15.40:0/3614225997 learned_addr learned my addr 172.21.15.40:0/3614225997 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe06fe00640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0701628d0 0x7fe07015bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe066a00640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe07015e000 0x7fe070141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe06f400640 1 -- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe07015e000 msgr2=0x7fe070141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe06f400640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe07015e000 0x7fe070141640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe06f400640 1 -- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0701628d0 msgr2=0x7fe07015bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe06f400640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0701628d0 0x7fe07015bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe06f400640 1 -- 172.21.15.40:0/3614225997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07015c760 con 0x7fe07006f840 2024-03-24T00:32:48.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe06f400640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe07006f840 0x7fe07015e6f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe06000c7c0 tx=0x7fe06000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:48.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe067e00640 1 -- 172.21.15.40:0/3614225997 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe060007a30 con 0x7fe07006f840 2024-03-24T00:32:48.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe067e00640 1 -- 172.21.15.40:0/3614225997 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe060007bd0 con 0x7fe07006f840 2024-03-24T00:32:48.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe067e00640 1 -- 172.21.15.40:0/3614225997 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0600113f0 con 0x7fe07006f840 2024-03-24T00:32:48.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe076000640 1 -- 172.21.15.40:0/3614225997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07015ca40 con 0x7fe07006f840 2024-03-24T00:32:48.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.460+0000 7fe076000640 1 -- 172.21.15.40:0/3614225997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07015f2e0 con 0x7fe07006f840 2024-03-24T00:32:48.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.464+0000 7fe076000640 1 -- 172.21.15.40:0/3614225997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe070068170 con 0x7fe07006f840 2024-03-24T00:32:48.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.464+0000 7fe067e00640 1 -- 172.21.15.40:0/3614225997 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe060011590 con 0x7fe07006f840 2024-03-24T00:32:48.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.464+0000 7fe067e00640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe04c067660 0x7fe04c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.464+0000 7fe067e00640 1 -- 172.21.15.40:0/3614225997 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fe060085920 con 0x7fe07006f840 2024-03-24T00:32:48.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.464+0000 7fe066a00640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe04c067660 0x7fe04c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.464+0000 7fe066a00640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe04c067660 0x7fe04c069b20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe0580ae000 tx=0x7fe0580a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:48.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.468+0000 7fe067e00640 1 -- 172.21.15.40:0/3614225997 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe060015020 con 0x7fe07006f840 2024-03-24T00:32:48.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.636+0000 7fe076000640 1 -- 172.21.15.40:0/3614225997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"} v 0) v1 -- 0x7fe070157e40 con 0x7fe07006f840 2024-03-24T00:32:48.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe067e00640 1 -- 172.21.15.40:0/3614225997 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"}]=0 v17) v1 ==== 114+0+0 (secure 0 0 0) 0x7fe060056070 con 0x7fe07006f840 2024-03-24T00:32:48.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe076000640 1 -- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe04c067660 msgr2=0x7fe04c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe076000640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe04c067660 0x7fe04c069b20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe0580ae000 tx=0x7fe0580a7000 comp rx=0 tx=0).stop 2024-03-24T00:32:48.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe076000640 1 -- 172.21.15.40:0/3614225997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe07006f840 msgr2=0x7fe07015e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe076000640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe07006f840 0x7fe07015e6f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fe06000c7c0 tx=0x7fe06000cc80 comp rx=0 tx=0).stop 2024-03-24T00:32:48.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe076000640 1 -- 172.21.15.40:0/3614225997 shutdown_connections 2024-03-24T00:32:48.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe076000640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe04c067660 0x7fe04c069b20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe076000640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0701628d0 0x7fe07015bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe076000640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe07015e000 0x7fe070141640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe076000640 1 --2- 172.21.15.40:0/3614225997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe07006f840 0x7fe07015e6f0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe076000640 1 -- 172.21.15.40:0/3614225997 >> 172.21.15.40:0/3614225997 conn(0x7fe0700d0bb0 msgr2=0x7fe0700d0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:48.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe076000640 1 -- 172.21.15.40:0/3614225997 shutdown_connections 2024-03-24T00:32:48.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.640+0000 7fe076000640 1 -- 172.21.15.40:0/3614225997 wait complete. 2024-03-24T00:32:48.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:32:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6ac9400640 1 Processor -- start 2024-03-24T00:32:48.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6ac9400640 1 -- start start 2024-03-24T00:32:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6ac9400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ac4144ec0 0x7f6ac40cfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6ac9400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ac4145450 0x7f6ac40b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6ac9400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac41605b0 con 0x7f6ac40b62f0 2024-03-24T00:32:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6ac9400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac41452a0 con 0x7f6ac4145450 2024-03-24T00:32:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6ac9400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac40d0840 con 0x7f6ac4144ec0 2024-03-24T00:32:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6abb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ac4145450 0x7f6ac40b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6abb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ac4145450 0x7f6ac40b5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42670/0 (socket says 172.21.15.40:42670) 2024-03-24T00:32:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6ac8a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6ac40b62f0 0x7f6ac4066e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56636/0 (socket says 172.21.15.40:56636) 2024-03-24T00:32:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6abb400640 1 -- 172.21.15.40:0/1639355973 learned_addr learned my addr 172.21.15.40:0/1639355973 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:48.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6ac3e00640 1 --2- 172.21.15.40:0/1639355973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ac4144ec0 0x7f6ac40cfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6abb400640 1 -- 172.21.15.40:0/1639355973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ac4144ec0 msgr2=0x7f6ac40cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6abb400640 1 --2- 172.21.15.40:0/1639355973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ac4144ec0 0x7f6ac40cfef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6abb400640 1 -- 172.21.15.40:0/1639355973 >> v1:172.21.15.40:6789/0 conn(0x7f6ac40b62f0 legacy=0x7f6ac4066e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6abb400640 1 -- 172.21.15.40:0/1639355973 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac415f7f0 con 0x7f6ac4145450 2024-03-24T00:32:48.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.804+0000 7f6ac3400640 1 -- 172.21.15.40:0/1639355973 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543499676 0 0) 0x7f6ac41605b0 con 0x7f6ac40b62f0 2024-03-24T00:32:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.808+0000 7f6abb400640 1 --2- 172.21.15.40:0/1639355973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ac4145450 0x7f6ac40b5db0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009870 tx=0x7f6ab401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252a56a1e892589a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.808+0000 7f6ac3400640 1 -- 172.21.15.40:0/1639355973 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab40b0070 con 0x7f6ac4145450 2024-03-24T00:32:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.808+0000 7f6ac3400640 1 -- 172.21.15.40:0/1639355973 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab40ab600 con 0x7f6ac4145450 2024-03-24T00:32:48.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.808+0000 7f6ac3400640 1 -- 172.21.15.40:0/1639355973 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab40ab9d0 con 0x7f6ac4145450 2024-03-24T00:32:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.808+0000 7f6ac9400640 1 -- 172.21.15.40:0/1639355973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ac4145450 msgr2=0x7f6ac40b5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.808+0000 7f6ac9400640 1 --2- 172.21.15.40:0/1639355973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ac4145450 0x7f6ac40b5db0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009870 tx=0x7f6ab401c4d0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.808+0000 7f6ac9400640 1 -- 172.21.15.40:0/1639355973 shutdown_connections 2024-03-24T00:32:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.808+0000 7f6ac9400640 1 --2- 172.21.15.40:0/1639355973 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ac4145450 0x7f6ac40b5db0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.808+0000 7f6ac9400640 1 --2- 172.21.15.40:0/1639355973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ac4144ec0 0x7f6ac40cfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.808+0000 7f6ac9400640 1 -- 172.21.15.40:0/1639355973 >> 172.21.15.40:0/1639355973 conn(0x7f6ac4071a40 msgr2=0x7f6ac4071e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.808+0000 7f6ac9400640 1 -- 172.21.15.40:0/1639355973 shutdown_connections 2024-03-24T00:32:48.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.808+0000 7f6ac9400640 1 -- 172.21.15.40:0/1639355973 wait complete. 2024-03-24T00:32:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac9400640 1 Processor -- start 2024-03-24T00:32:48.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac9400640 1 -- start start 2024-03-24T00:32:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ac40b62f0 0x7f6ac415b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ac4144ec0 0x7f6ac413fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac9400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ac4145450 0x7f6ac4140430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac9400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac40ce9c0 con 0x7f6ac4144ec0 2024-03-24T00:32:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac9400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac406fb10 con 0x7f6ac40b62f0 2024-03-24T00:32:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac9400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac40d0430 con 0x7f6ac4145450 2024-03-24T00:32:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6abb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ac4144ec0 0x7f6ac413fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6abb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ac4144ec0 0x7f6ac413fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42680/0 (socket says 172.21.15.40:42680) 2024-03-24T00:32:48.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6abb400640 1 -- 172.21.15.40:0/3468108731 learned_addr learned my addr 172.21.15.40:0/3468108731 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac8a00640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ac4145450 0x7f6ac4140430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac3e00640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ac40b62f0 0x7f6ac415b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6abb400640 1 -- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ac4145450 msgr2=0x7f6ac4140430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6abb400640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ac4145450 0x7f6ac4140430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6abb400640 1 -- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ac40b62f0 msgr2=0x7f6ac415b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6abb400640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ac40b62f0 0x7f6ac415b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6abb400640 1 -- 172.21.15.40:0/3468108731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac40ae2c0 con 0x7f6ac4144ec0 2024-03-24T00:32:48.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6abb400640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ac4144ec0 0x7f6ac413fef0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009870 tx=0x7f6ab401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:48.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac0c00640 1 -- 172.21.15.40:0/3468108731 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab40b0040 con 0x7f6ac4144ec0 2024-03-24T00:32:48.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac0c00640 1 -- 172.21.15.40:0/3468108731 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab40abe50 con 0x7f6ac4144ec0 2024-03-24T00:32:48.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac0c00640 1 -- 172.21.15.40:0/3468108731 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4004220 con 0x7f6ac4144ec0 2024-03-24T00:32:48.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac9400640 1 -- 172.21.15.40:0/3468108731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac40ae540 con 0x7f6ac4144ec0 2024-03-24T00:32:48.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.812+0000 7f6ac9400640 1 -- 172.21.15.40:0/3468108731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac40aeb00 con 0x7f6ac4144ec0 2024-03-24T00:32:48.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.816+0000 7f6ac9400640 1 -- 172.21.15.40:0/3468108731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac404f410 con 0x7f6ac4144ec0 2024-03-24T00:32:48.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.816+0000 7f6ac0c00640 1 -- 172.21.15.40:0/3468108731 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6ab40be060 con 0x7f6ac4144ec0 2024-03-24T00:32:48.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.816+0000 7f6ac0c00640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6aa0067660 0x7f6aa0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:48.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.816+0000 7f6ac0c00640 1 -- 172.21.15.40:0/3468108731 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f6ab411edc0 con 0x7f6ac4144ec0 2024-03-24T00:32:48.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.816+0000 7f6ac3e00640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6aa0067660 0x7f6aa0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:48.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.816+0000 7f6ac3e00640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6aa0067660 0x7f6aa0069b20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6aac009a50 tx=0x7f6aac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:48.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.820+0000 7f6ac0c00640 1 -- 172.21.15.40:0/3468108731 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6ac40ae2c0 con 0x7f6ac4144ec0 2024-03-24T00:32:48.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.984+0000 7f6ac9400640 1 -- 172.21.15.40:0/3468108731 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6ac4072ae0 con 0x7f6ac4144ec0 2024-03-24T00:32:48.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac0c00640 1 -- 172.21.15.40:0/3468108731 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6ab4016150 con 0x7f6ac4144ec0 2024-03-24T00:32:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac9400640 1 -- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6aa0067660 msgr2=0x7f6aa0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac9400640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6aa0067660 0x7f6aa0069b20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6aac009a50 tx=0x7f6aac008040 comp rx=0 tx=0).stop 2024-03-24T00:32:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac9400640 1 -- 172.21.15.40:0/3468108731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ac4144ec0 msgr2=0x7f6ac413fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac9400640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ac4144ec0 0x7f6ac413fef0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009870 tx=0x7f6ab401ad10 comp rx=0 tx=0).stop 2024-03-24T00:32:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac9400640 1 -- 172.21.15.40:0/3468108731 shutdown_connections 2024-03-24T00:32:48.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac9400640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6aa0067660 0x7f6aa0069b20 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac9400640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ac4145450 0x7f6ac4140430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac9400640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ac4144ec0 0x7f6ac413fef0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac9400640 1 --2- 172.21.15.40:0/3468108731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ac40b62f0 0x7f6ac415b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac9400640 1 -- 172.21.15.40:0/3468108731 >> 172.21.15.40:0/3468108731 conn(0x7f6ac4071a40 msgr2=0x7f6ac4071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac9400640 1 -- 172.21.15.40:0/3468108731 shutdown_connections 2024-03-24T00:32:48.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:48.988+0000 7f6ac9400640 1 -- 172.21.15.40:0/3468108731 wait complete. 2024-03-24T00:32:49.011 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_conf_off' 2024-03-24T00:32:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.148+0000 7fa14ae00640 1 Processor -- start 2024-03-24T00:32:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.148+0000 7fa14ae00640 1 -- start start 2024-03-24T00:32:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa14ae00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa1440cdaa0 0x7fa144168fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa14ae00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1440cfaa0 con 0x7fa14415dfc0 2024-03-24T00:32:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa14ae00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1440cfd80 con 0x7fa14415e3a0 2024-03-24T00:32:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa14ae00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1440cf870 con 0x7fa1440cdaa0 2024-03-24T00:32:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa14415dfc0 0x7fa1440cd390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56644/0 (socket says 172.21.15.40:56644) 2024-03-24T00:32:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143e00640 1 -- 172.21.15.40:0/2046211045 learned_addr learned my addr 172.21.15.40:0/2046211045 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:49.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa148e00640 1 --2- 172.21.15.40:0/2046211045 >> v2:172.21.15.53:3300/0 conn(0x7fa1440cdaa0 0x7fa144168fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358384354 0 0) 0x7fa1440cfaa0 con 0x7fa14415dfc0 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa128003530 con 0x7fa14415dfc0 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238515667 0 0) 0x7fa1440cfd80 con 0x7fa14415e3a0 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1440cfaa0 con 0x7fa14415e3a0 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3449510630 0 0) 0x7fa128003530 con 0x7fa14415dfc0 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 >> v2:172.21.15.53:3300/0 conn(0x7fa1440cdaa0 msgr2=0x7fa144168fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 --2- 172.21.15.40:0/2046211045 >> v2:172.21.15.53:3300/0 conn(0x7fa1440cdaa0 0x7fa144168fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 >> v1:172.21.15.53:6789/0 conn(0x7fa14415e3a0 legacy=0x7fa1440ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1440d0060 con 0x7fa14415dfc0 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fa134001a00 con 0x7fa14415dfc0 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa148e00640 1 --2- 172.21.15.40:0/2046211045 >> v2:172.21.15.53:3300/0 conn(0x7fa1440cdaa0 0x7fa144168fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 >> v1:172.21.15.40:6789/0 conn(0x7fa14415dfc0 legacy=0x7fa1440cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 --2- 172.21.15.40:0/2046211045 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa128003b50 0x7fa128003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 --2- 172.21.15.40:0/2046211045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 0x7fa128004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 --2- 172.21.15.40:0/2046211045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa128005240 0x7fa12800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12800c170 con 0x7fa128004810 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12800c320 con 0x7fa128005240 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa12800c520 con 0x7fa128003b50 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa13b400640 1 --2- 172.21.15.40:0/2046211045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 0x7fa128004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa143e00640 1 --2- 172.21.15.40:0/2046211045 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa128003b50 0x7fa128003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa148e00640 1 --2- 172.21.15.40:0/2046211045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa128005240 0x7fa12800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa13b400640 1 -- 172.21.15.40:0/2046211045 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa128003b50 msgr2=0x7fa128003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa13b400640 1 --2- 172.21.15.40:0/2046211045 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa128003b50 0x7fa128003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa13b400640 1 -- 172.21.15.40:0/2046211045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa128005240 msgr2=0x7fa12800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa13b400640 1 --2- 172.21.15.40:0/2046211045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa128005240 0x7fa12800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa13b400640 1 -- 172.21.15.40:0/2046211045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa128003530 con 0x7fa128004810 2024-03-24T00:32:49.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.152+0000 7fa148e00640 1 -- 172.21.15.40:0/2046211045 reap_dead start 2024-03-24T00:32:49.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.156+0000 7fa13b400640 1 --2- 172.21.15.40:0/2046211045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 0x7fa128004c90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0018d0 tx=0x7fa12c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a39a22ef2d3d99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:49.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.156+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c001520 con 0x7fa128004810 2024-03-24T00:32:49.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.156+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12c0052f0 con 0x7fa128004810 2024-03-24T00:32:49.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.156+0000 7fa143400640 1 -- 172.21.15.40:0/2046211045 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c003840 con 0x7fa128004810 2024-03-24T00:32:49.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.156+0000 7fa14ae00640 1 -- 172.21.15.40:0/2046211045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 msgr2=0x7fa128004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.156+0000 7fa14ae00640 1 --2- 172.21.15.40:0/2046211045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 0x7fa128004c90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0018d0 tx=0x7fa12c01c520 comp rx=0 tx=0).stop 2024-03-24T00:32:49.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.156+0000 7fa14ae00640 1 -- 172.21.15.40:0/2046211045 shutdown_connections 2024-03-24T00:32:49.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.156+0000 7fa14ae00640 1 --2- 172.21.15.40:0/2046211045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 0x7fa128004c90 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.156+0000 7fa14ae00640 1 -- 172.21.15.40:0/2046211045 >> 172.21.15.40:0/2046211045 conn(0x7fa1440b37f0 msgr2=0x7fa1440b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:49.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.156+0000 7fa14ae00640 1 -- 172.21.15.40:0/2046211045 shutdown_connections 2024-03-24T00:32:49.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.156+0000 7fa14ae00640 1 -- 172.21.15.40:0/2046211045 wait complete. 2024-03-24T00:32:49.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa14ae00640 1 Processor -- start 2024-03-24T00:32:49.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa14ae00640 1 -- start start 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa14ae00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 0x7fa144156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa14ae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa14406bcb0 0x7fa14406c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa14ae00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa144160210 0x7fa1441606c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa14ae00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa144161c80 con 0x7fa128004810 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa14ae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1440d0d50 con 0x7fa14406bcb0 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa14ae00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa144161e00 con 0x7fa144160210 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa143e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 0x7fa144156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa143e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 0x7fa144156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42692/0 (socket says 172.21.15.40:42692) 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa143e00640 1 -- 172.21.15.40:0/1293173433 learned_addr learned my addr 172.21.15.40:0/1293173433 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa148e00640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa144160210 0x7fa1441606c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa13b400640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa14406bcb0 0x7fa14406c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa143e00640 1 -- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa144160210 msgr2=0x7fa1441606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa143e00640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa144160210 0x7fa1441606c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa143e00640 1 -- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa14406bcb0 msgr2=0x7fa14406c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa143e00640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa14406bcb0 0x7fa14406c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa143e00640 1 -- 172.21.15.40:0/1293173433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa144160c00 con 0x7fa128004810 2024-03-24T00:32:49.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa143e00640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 0x7fa144156300 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa1340077c0 tx=0x7fa13400a310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:49.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa140c00640 1 -- 172.21.15.40:0/1293173433 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa134017020 con 0x7fa128004810 2024-03-24T00:32:49.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa140c00640 1 -- 172.21.15.40:0/1293173433 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa13400ad50 con 0x7fa128004810 2024-03-24T00:32:49.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa14ae00640 1 -- 172.21.15.40:0/1293173433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1441faa70 con 0x7fa128004810 2024-03-24T00:32:49.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa140c00640 1 -- 172.21.15.40:0/1293173433 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa134013630 con 0x7fa128004810 2024-03-24T00:32:49.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa14ae00640 1 -- 172.21.15.40:0/1293173433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1441fb030 con 0x7fa128004810 2024-03-24T00:32:49.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.160+0000 7fa14ae00640 1 -- 172.21.15.40:0/1293173433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa144068170 con 0x7fa128004810 2024-03-24T00:32:49.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.164+0000 7fa140c00640 1 -- 172.21.15.40:0/1293173433 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa13401a020 con 0x7fa128004810 2024-03-24T00:32:49.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.164+0000 7fa140c00640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa120067690 0x7fa120069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.164+0000 7fa140c00640 1 -- 172.21.15.40:0/1293173433 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fa134086080 con 0x7fa128004810 2024-03-24T00:32:49.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.164+0000 7fa13b400640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa120067690 0x7fa120069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.164+0000 7fa13b400640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa120067690 0x7fa120069b50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0ac000 tx=0x7fa12c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:49.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.168+0000 7fa140c00640 1 -- 172.21.15.40:0/1293173433 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa134021020 con 0x7fa128004810 2024-03-24T00:32:49.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.396+0000 7fa14ae00640 1 -- 172.21.15.40:0/1293173433 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_conf_off"]} v 0) v1 -- 0x7fa14406d280 con 0x7fa128004810 2024-03-24T00:32:49.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.580+0000 7fa140c00640 1 -- 172.21.15.40:0/1293173433 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_conf_off"]}]=0 v112) v1 ==== 155+0+0 (secure 0 0 0) 0x7fa134056f40 con 0x7fa128004810 2024-03-24T00:32:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.584+0000 7fa14ae00640 1 -- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa120067690 msgr2=0x7fa120069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.584+0000 7fa14ae00640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa120067690 0x7fa120069b50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0ac000 tx=0x7fa12c009040 comp rx=0 tx=0).stop 2024-03-24T00:32:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.584+0000 7fa14ae00640 1 -- 172.21.15.40:0/1293173433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 msgr2=0x7fa144156300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.584+0000 7fa14ae00640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 0x7fa144156300 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa1340077c0 tx=0x7fa13400a310 comp rx=0 tx=0).stop 2024-03-24T00:32:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.584+0000 7fa14ae00640 1 -- 172.21.15.40:0/1293173433 shutdown_connections 2024-03-24T00:32:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.584+0000 7fa14ae00640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa120067690 0x7fa120069b50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.584+0000 7fa14ae00640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa144160210 0x7fa1441606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.584+0000 7fa14ae00640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa14406bcb0 0x7fa14406c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.584+0000 7fa14ae00640 1 --2- 172.21.15.40:0/1293173433 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa128004810 0x7fa144156300 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.584+0000 7fa14ae00640 1 -- 172.21.15.40:0/1293173433 >> 172.21.15.40:0/1293173433 conn(0x7fa1440b37f0 msgr2=0x7fa1440d10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.584+0000 7fa14ae00640 1 -- 172.21.15.40:0/1293173433 shutdown_connections 2024-03-24T00:32:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.584+0000 7fa14ae00640 1 -- 172.21.15.40:0/1293173433 wait complete. 2024-03-24T00:32:49.605 INFO:tasks.cephfs_test_runner:That turning off ephemeral distributed pin config prevents distribution. ... ok 2024-03-24T00:32:49.605 INFO:tasks.cephfs_test_runner:Starting test: test_ephemeral_pin_dist_failover (tasks.cephfs.test_exports.TestEphemeralPins) That MDS failover does not cause unnecessary migrations. 2024-03-24T00:32:49.605 INFO:tasks.cephfs_test_runner:test_ephemeral_pin_dist_failover (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:32:49.606 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:32:49.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.744+0000 7ffada400640 1 Processor -- start 2024-03-24T00:32:49.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.744+0000 7ffada400640 1 -- start start 2024-03-24T00:32:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.744+0000 7ffada400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffad413ec80 0x7ffad41414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.744+0000 7ffada400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffad41419f0 0x7ffad40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.744+0000 7ffada400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad4070e00 con 0x7ffad40ce630 2024-03-24T00:32:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffada400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad4141e50 con 0x7ffad413ec80 2024-03-24T00:32:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffada400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad40cee20 con 0x7ffad41419f0 2024-03-24T00:32:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad3400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffad413ec80 0x7ffad41414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ffad40ce630 0x7ffad415f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56650/0 (socket says 172.21.15.40:56650) 2024-03-24T00:32:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad3e00640 1 -- 172.21.15.40:0/1647115967 learned_addr learned my addr 172.21.15.40:0/1647115967 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad3400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffad413ec80 0x7ffad41414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42708/0 (socket says 172.21.15.40:42708) 2024-03-24T00:32:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffacbe00640 1 --2- 172.21.15.40:0/1647115967 >> v2:172.21.15.53:3300/0 conn(0x7ffad41419f0 0x7ffad40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad3400640 1 -- 172.21.15.40:0/1647115967 >> v2:172.21.15.53:3300/0 conn(0x7ffad41419f0 msgr2=0x7ffad40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad3400640 1 --2- 172.21.15.40:0/1647115967 >> v2:172.21.15.53:3300/0 conn(0x7ffad41419f0 0x7ffad40ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad3400640 1 -- 172.21.15.40:0/1647115967 >> v1:172.21.15.40:6789/0 conn(0x7ffad40ce630 legacy=0x7ffad415f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad3400640 1 -- 172.21.15.40:0/1647115967 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad40cf100 con 0x7ffad413ec80 2024-03-24T00:32:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad2a00640 1 -- 172.21.15.40:0/1647115967 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605983445 0 0) 0x7ffad4070e00 con 0x7ffad40ce630 2024-03-24T00:32:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad3400640 1 --2- 172.21.15.40:0/1647115967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffad413ec80 0x7ffad41414b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ffad404fa60 tx=0x7ffac401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d123f4215a551d33 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad2a00640 1 -- 172.21.15.40:0/1647115967 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac40aa600 con 0x7ffad413ec80 2024-03-24T00:32:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad2a00640 1 -- 172.21.15.40:0/1647115967 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac40af070 con 0x7ffad413ec80 2024-03-24T00:32:49.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.748+0000 7ffad2a00640 1 -- 172.21.15.40:0/1647115967 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac40aa9c0 con 0x7ffad413ec80 2024-03-24T00:32:49.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 -- 172.21.15.40:0/1647115967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffad413ec80 msgr2=0x7ffad41414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 --2- 172.21.15.40:0/1647115967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffad413ec80 0x7ffad41414b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ffad404fa60 tx=0x7ffac401c480 comp rx=0 tx=0).stop 2024-03-24T00:32:49.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 -- 172.21.15.40:0/1647115967 shutdown_connections 2024-03-24T00:32:49.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 --2- 172.21.15.40:0/1647115967 >> v2:172.21.15.53:3300/0 conn(0x7ffad41419f0 0x7ffad40ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 --2- 172.21.15.40:0/1647115967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffad413ec80 0x7ffad41414b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 -- 172.21.15.40:0/1647115967 >> 172.21.15.40:0/1647115967 conn(0x7ffad40b17b0 msgr2=0x7ffad40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:49.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 -- 172.21.15.40:0/1647115967 shutdown_connections 2024-03-24T00:32:49.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 -- 172.21.15.40:0/1647115967 wait complete. 2024-03-24T00:32:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 Processor -- start 2024-03-24T00:32:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 -- start start 2024-03-24T00:32:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffad40ce630 0x7ffad413b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffad413ec80 0x7ffad414eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffad41419f0 0x7ffad413d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad4168ec0 con 0x7ffad40ce630 2024-03-24T00:32:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad4168d40 con 0x7ffad41419f0 2024-03-24T00:32:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffada400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad4169040 con 0x7ffad413ec80 2024-03-24T00:32:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffad3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffad40ce630 0x7ffad413b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffad3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffad40ce630 0x7ffad413b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42710/0 (socket says 172.21.15.40:42710) 2024-03-24T00:32:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffad3400640 1 -- 172.21.15.40:0/915704368 learned_addr learned my addr 172.21.15.40:0/915704368 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffacbe00640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffad413ec80 0x7ffad414eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffad3e00640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffad41419f0 0x7ffad413d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffad3400640 1 -- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffad413ec80 msgr2=0x7ffad414eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffad3400640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffad413ec80 0x7ffad414eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffad3400640 1 -- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffad41419f0 msgr2=0x7ffad413d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffad3400640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffad41419f0 0x7ffad413d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffad3400640 1 -- 172.21.15.40:0/915704368 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad4157690 con 0x7ffad40ce630 2024-03-24T00:32:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffacbe00640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffad413ec80 0x7ffad414eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffad3e00640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffad41419f0 0x7ffad413d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.752+0000 7ffad3400640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffad40ce630 0x7ffad413b0c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ffac4019090 tx=0x7ffac401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.756+0000 7ffacb400640 1 -- 172.21.15.40:0/915704368 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac40be610 con 0x7ffad40ce630 2024-03-24T00:32:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.756+0000 7ffacb400640 1 -- 172.21.15.40:0/915704368 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac40be7b0 con 0x7ffad40ce630 2024-03-24T00:32:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.756+0000 7ffacb400640 1 -- 172.21.15.40:0/915704368 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac40beb80 con 0x7ffad40ce630 2024-03-24T00:32:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.756+0000 7ffada400640 1 -- 172.21.15.40:0/915704368 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad413ba30 con 0x7ffad40ce630 2024-03-24T00:32:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.756+0000 7ffada400640 1 -- 172.21.15.40:0/915704368 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad413bff0 con 0x7ffad40ce630 2024-03-24T00:32:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.756+0000 7ffada400640 1 -- 172.21.15.40:0/915704368 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad4068170 con 0x7ffad40ce630 2024-03-24T00:32:49.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.756+0000 7ffacb400640 1 -- 172.21.15.40:0/915704368 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ffac40bc020 con 0x7ffad40ce630 2024-03-24T00:32:49.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.756+0000 7ffacb400640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffab0067690 0x7ffab0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:49.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.756+0000 7ffacb400640 1 -- 172.21.15.40:0/915704368 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5115+0+0 (secure 0 0 0) 0x7ffac411fe30 con 0x7ffad40ce630 2024-03-24T00:32:49.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.756+0000 7ffacbe00640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffab0067690 0x7ffab0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:49.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.760+0000 7ffacbe00640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffab0067690 0x7ffab0069b50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ffad413e2f0 tx=0x7ffabc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:49.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.760+0000 7ffacb400640 1 -- 172.21.15.40:0/915704368 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ffac4016070 con 0x7ffad40ce630 2024-03-24T00:32:49.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.924+0000 7ffada400640 1 -- 172.21.15.40:0/915704368 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffad4072b90 con 0x7ffad40ce630 2024-03-24T00:32:49.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.924+0000 7ffacb400640 1 -- 172.21.15.40:0/915704368 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+13220 (secure 0 0 0) 0x7ffac40f0e10 con 0x7ffad40ce630 2024-03-24T00:32:49.928 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:49.928 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":36,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:32:47.578081+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6832/2326157501":"2024-03-25T00:32:44.572497+0000","172.21.15.40:6835/3050269371":"2024-03-25T00:32:44.572508+0000","172.21.15.40:6834/3050269371":"2024-03-25T00:32:44.572508+0000","172.21.15.40:6833/2326157501":"2024-03-25T00:32:44.572497+0000","172.21.15.53:6837/2159849028":"2024-03-25T00:32:44.572478+0000","172.21.15.53:6835/2159849028":"2024-03-25T00:32:44.572478+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:32:49.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.928+0000 7ffada400640 1 -- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffab0067690 msgr2=0x7ffab0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.928+0000 7ffada400640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffab0067690 0x7ffab0069b50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ffad413e2f0 tx=0x7ffabc008040 comp rx=0 tx=0).stop 2024-03-24T00:32:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.928+0000 7ffada400640 1 -- 172.21.15.40:0/915704368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffad40ce630 msgr2=0x7ffad413b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.928+0000 7ffada400640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffad40ce630 0x7ffad413b0c0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7ffac4019090 tx=0x7ffac401acc0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.928+0000 7ffada400640 1 -- 172.21.15.40:0/915704368 shutdown_connections 2024-03-24T00:32:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.928+0000 7ffada400640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffab0067690 0x7ffab0069b50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.928+0000 7ffada400640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffad41419f0 0x7ffad413d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.928+0000 7ffada400640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffad413ec80 0x7ffad414eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.928+0000 7ffada400640 1 --2- 172.21.15.40:0/915704368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffad40ce630 0x7ffad413b0c0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.928+0000 7ffada400640 1 -- 172.21.15.40:0/915704368 >> 172.21.15.40:0/915704368 conn(0x7ffad40b17b0 msgr2=0x7ffad4071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.928+0000 7ffada400640 1 -- 172.21.15.40:0/915704368 shutdown_connections 2024-03-24T00:32:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:49.928+0000 7ffada400640 1 -- 172.21.15.40:0/915704368 wait complete. 2024-03-24T00:32:49.950 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:32:49.950 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:32:50.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.092+0000 7fe7baa00640 1 Processor -- start 2024-03-24T00:32:50.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.092+0000 7fe7baa00640 1 -- start start 2024-03-24T00:32:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.092+0000 7fe7baa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe7b415bb10 0x7fe7b406f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.092+0000 7fe7baa00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b4067c60 con 0x7fe7b406f570 2024-03-24T00:32:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.092+0000 7fe7baa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b415e910 con 0x7fe7b4067600 2024-03-24T00:32:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.092+0000 7fe7baa00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b406f9d0 con 0x7fe7b415bb10 2024-03-24T00:32:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.096+0000 7fe7b3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe7b415bb10 0x7fe7b406f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.096+0000 7fe7b3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe7b415bb10 0x7fe7b406f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42714/0 (socket says 172.21.15.40:42714) 2024-03-24T00:32:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.096+0000 7fe7b3e00640 1 -- 172.21.15.40:0/2119502550 learned_addr learned my addr 172.21.15.40:0/2119502550 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.096+0000 7fe7abe00640 1 --1- 172.21.15.40:0/2119502550 >> v1:172.21.15.40:6789/0 conn(0x7fe7b406f570 0x7fe7b4066ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56662/0 (socket says 172.21.15.40:56662) 2024-03-24T00:32:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.096+0000 7fe7b3e00640 1 -- 172.21.15.40:0/2119502550 >> v1:172.21.15.53:6789/0 conn(0x7fe7b4067600 legacy=0x7fe7b4161bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.096+0000 7fe7b3e00640 1 -- 172.21.15.40:0/2119502550 >> v1:172.21.15.40:6789/0 conn(0x7fe7b406f570 legacy=0x7fe7b4066ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.096+0000 7fe7b3e00640 1 -- 172.21.15.40:0/2119502550 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b415ebf0 con 0x7fe7b415bb10 2024-03-24T00:32:50.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.096+0000 7fe7b3e00640 1 --2- 172.21.15.40:0/2119502550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7b415bb10 0x7fe7b406f030 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009870 tx=0x7fe7a401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece540e6ca7e7b42 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:50.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.096+0000 7fe7b3400640 1 -- 172.21.15.40:0/2119502550 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40b2070 con 0x7fe7b415bb10 2024-03-24T00:32:50.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.096+0000 7fe7b3400640 1 -- 172.21.15.40:0/2119502550 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a40a9070 con 0x7fe7b415bb10 2024-03-24T00:32:50.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.096+0000 7fe7b3400640 1 -- 172.21.15.40:0/2119502550 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40ad880 con 0x7fe7b415bb10 2024-03-24T00:32:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 -- 172.21.15.40:0/2119502550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7b415bb10 msgr2=0x7fe7b406f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 --2- 172.21.15.40:0/2119502550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7b415bb10 0x7fe7b406f030 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009870 tx=0x7fe7a401c560 comp rx=0 tx=0).stop 2024-03-24T00:32:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 -- 172.21.15.40:0/2119502550 shutdown_connections 2024-03-24T00:32:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 --2- 172.21.15.40:0/2119502550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7b415bb10 0x7fe7b406f030 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 -- 172.21.15.40:0/2119502550 >> 172.21.15.40:0/2119502550 conn(0x7fe7b40b27c0 msgr2=0x7fe7b40b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 -- 172.21.15.40:0/2119502550 shutdown_connections 2024-03-24T00:32:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 -- 172.21.15.40:0/2119502550 wait complete. 2024-03-24T00:32:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 Processor -- start 2024-03-24T00:32:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 -- start start 2024-03-24T00:32:50.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7b4067600 0x7fe7b414f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:50.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe7b406f570 0x7fe7b414c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:50.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b415bb10 0x7fe7b406dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:50.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b416c0e0 con 0x7fe7b4067600 2024-03-24T00:32:50.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b416bf60 con 0x7fe7b415bb10 2024-03-24T00:32:50.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7baa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b416c260 con 0x7fe7b406f570 2024-03-24T00:32:50.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7b4067600 0x7fe7b414f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:50.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7b4067600 0x7fe7b414f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42718/0 (socket says 172.21.15.40:42718) 2024-03-24T00:32:50.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7b3e00640 1 -- 172.21.15.40:0/1352681273 learned_addr learned my addr 172.21.15.40:0/1352681273 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7abe00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe7b406f570 0x7fe7b414c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7b8a00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b415bb10 0x7fe7b406dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7b3e00640 1 -- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe7b406f570 msgr2=0x7fe7b414c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7b3e00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe7b406f570 0x7fe7b414c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7b3e00640 1 -- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b415bb10 msgr2=0x7fe7b406dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7b3e00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b415bb10 0x7fe7b406dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.100+0000 7fe7b3e00640 1 -- 172.21.15.40:0/1352681273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b4154bb0 con 0x7fe7b4067600 2024-03-24T00:32:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.104+0000 7fe7b3e00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7b4067600 0x7fe7b414f4c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40b0040 tx=0x7fe7a4019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.104+0000 7fe7b0c00640 1 -- 172.21.15.40:0/1352681273 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40b2040 con 0x7fe7b4067600 2024-03-24T00:32:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.104+0000 7fe7b0c00640 1 -- 172.21.15.40:0/1352681273 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a40bb070 con 0x7fe7b4067600 2024-03-24T00:32:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.104+0000 7fe7b0c00640 1 -- 172.21.15.40:0/1352681273 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40b6680 con 0x7fe7b4067600 2024-03-24T00:32:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.104+0000 7fe7baa00640 1 -- 172.21.15.40:0/1352681273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b40aead0 con 0x7fe7b4067600 2024-03-24T00:32:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.104+0000 7fe7baa00640 1 -- 172.21.15.40:0/1352681273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b40af5b0 con 0x7fe7b4067600 2024-03-24T00:32:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.104+0000 7fe7baa00640 1 -- 172.21.15.40:0/1352681273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b4068170 con 0x7fe7b4067600 2024-03-24T00:32:50.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.104+0000 7fe7b0c00640 1 -- 172.21.15.40:0/1352681273 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe7a40ab070 con 0x7fe7b4067600 2024-03-24T00:32:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.104+0000 7fe7b0c00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe790067660 0x7fe790069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.104+0000 7fe7b0c00640 1 -- 172.21.15.40:0/1352681273 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fe7a411f0f0 con 0x7fe7b4067600 2024-03-24T00:32:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.104+0000 7fe7abe00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe790067660 0x7fe790069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.108+0000 7fe7abe00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe790067660 0x7fe790069b20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe7b41501b0 tx=0x7fe79c007e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:50.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.108+0000 7fe7b0c00640 1 -- 172.21.15.40:0/1352681273 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe7a4016150 con 0x7fe7b4067600 2024-03-24T00:32:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.272+0000 7fe7baa00640 1 -- 172.21.15.40:0/1352681273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7b40af190 con 0x7fe7b4067600 2024-03-24T00:32:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.272+0000 7fe7b0c00640 1 -- 172.21.15.40:0/1352681273 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe7a40eff20 con 0x7fe7b4067600 2024-03-24T00:32:50.276 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:32:50.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.276+0000 7fe7baa00640 1 -- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe790067660 msgr2=0x7fe790069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:50.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.276+0000 7fe7baa00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe790067660 0x7fe790069b20 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe7b41501b0 tx=0x7fe79c007e50 comp rx=0 tx=0).stop 2024-03-24T00:32:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.276+0000 7fe7baa00640 1 -- 172.21.15.40:0/1352681273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7b4067600 msgr2=0x7fe7b414f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.276+0000 7fe7baa00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7b4067600 0x7fe7b414f4c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe7a40b0040 tx=0x7fe7a4019300 comp rx=0 tx=0).stop 2024-03-24T00:32:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.276+0000 7fe7baa00640 1 -- 172.21.15.40:0/1352681273 shutdown_connections 2024-03-24T00:32:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.276+0000 7fe7baa00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe790067660 0x7fe790069b20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.276+0000 7fe7baa00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b415bb10 0x7fe7b406dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.276+0000 7fe7baa00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe7b406f570 0x7fe7b414c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.276+0000 7fe7baa00640 1 --2- 172.21.15.40:0/1352681273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7b4067600 0x7fe7b414f4c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.276+0000 7fe7baa00640 1 -- 172.21.15.40:0/1352681273 >> 172.21.15.40:0/1352681273 conn(0x7fe7b40b27c0 msgr2=0x7fe7b4153cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.276+0000 7fe7baa00640 1 -- 172.21.15.40:0/1352681273 shutdown_connections 2024-03-24T00:32:50.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.276+0000 7fe7baa00640 1 -- 172.21.15.40:0/1352681273 wait complete. 2024-03-24T00:32:50.298 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_failover' 2024-03-24T00:32:50.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.436+0000 7f40b1400640 1 Processor -- start 2024-03-24T00:32:50.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.436+0000 7f40b1400640 1 -- start start 2024-03-24T00:32:50.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40b1400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f40ac06fab0 0x7f40ac06fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:50.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40b1400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ac0d0e10 con 0x7f40ac15e0b0 2024-03-24T00:32:50.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40b1400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ac06efd0 con 0x7f40ac0d0830 2024-03-24T00:32:50.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40b1400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ac0d10f0 con 0x7f40ac06fab0 2024-03-24T00:32:50.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40aa800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f40ac15e0b0 0x7f40ac0cd3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56666/0 (socket says 172.21.15.40:56666) 2024-03-24T00:32:50.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40aa800640 1 -- 172.21.15.40:0/4221994546 learned_addr learned my addr 172.21.15.40:0/4221994546 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:50.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40ab200640 1 --2- 172.21.15.40:0/4221994546 >> v2:172.21.15.53:3300/0 conn(0x7f40ac06fab0 0x7f40ac06fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:50.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40a9e00640 1 -- 172.21.15.40:0/4221994546 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430785553 0 0) 0x7f40ac0d0e10 con 0x7f40ac15e0b0 2024-03-24T00:32:50.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40a9e00640 1 -- 172.21.15.40:0/4221994546 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4090003530 con 0x7f40ac15e0b0 2024-03-24T00:32:50.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40a9e00640 1 -- 172.21.15.40:0/4221994546 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82248379 0 0) 0x7f40ac06efd0 con 0x7f40ac0d0830 2024-03-24T00:32:50.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40a9e00640 1 -- 172.21.15.40:0/4221994546 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40ac0d0e10 con 0x7f40ac0d0830 2024-03-24T00:32:50.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40ab200640 1 -- 172.21.15.40:0/4221994546 >> v1:172.21.15.53:6789/0 conn(0x7f40ac0d0830 legacy=0x7f40ac0cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:50.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40ab200640 1 -- 172.21.15.40:0/4221994546 >> v1:172.21.15.40:6789/0 conn(0x7f40ac15e0b0 legacy=0x7f40ac0cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:50.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40a9e00640 1 -- 172.21.15.40:0/4221994546 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 198719116 0 0) 0x7f4090003530 con 0x7f40ac15e0b0 2024-03-24T00:32:50.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.440+0000 7f40ab200640 1 -- 172.21.15.40:0/4221994546 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac06f2b0 con 0x7f40ac06fab0 2024-03-24T00:32:50.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40ab200640 1 --2- 172.21.15.40:0/4221994546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac06fab0 0x7f40ac06fec0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f40a4009d90 tx=0x7f40a401c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89d67f7a1b3ea032 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:50.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40a9e00640 1 -- 172.21.15.40:0/4221994546 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c0016a0 con 0x7f40ac06fab0 2024-03-24T00:32:50.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40a9e00640 1 -- 172.21.15.40:0/4221994546 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a40a64e0 con 0x7f40ac06fab0 2024-03-24T00:32:50.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40a9e00640 1 -- 172.21.15.40:0/4221994546 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a40a68b0 con 0x7f40ac06fab0 2024-03-24T00:32:50.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40b1400640 1 -- 172.21.15.40:0/4221994546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac06fab0 msgr2=0x7f40ac06fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:50.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40b1400640 1 --2- 172.21.15.40:0/4221994546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac06fab0 0x7f40ac06fec0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f40a4009d90 tx=0x7f40a401c890 comp rx=0 tx=0).stop 2024-03-24T00:32:50.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40b1400640 1 -- 172.21.15.40:0/4221994546 shutdown_connections 2024-03-24T00:32:50.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40b1400640 1 --2- 172.21.15.40:0/4221994546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac06fab0 0x7f40ac06fec0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:50.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40b1400640 1 -- 172.21.15.40:0/4221994546 >> 172.21.15.40:0/4221994546 conn(0x7f40ac0d1c90 msgr2=0x7f40ac0d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:50.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40b1400640 1 -- 172.21.15.40:0/4221994546 shutdown_connections 2024-03-24T00:32:50.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40b1400640 1 -- 172.21.15.40:0/4221994546 wait complete. 2024-03-24T00:32:50.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40b1400640 1 Processor -- start 2024-03-24T00:32:50.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.444+0000 7f40b1400640 1 -- start start 2024-03-24T00:32:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40b1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40ac06fab0 0x7f40ac1567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40b1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0d0830 0x7f40ac0cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40b1400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f40ac15e0b0 0x7f40ac15c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40b1400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac16bec0 con 0x7f40ac06fab0 2024-03-24T00:32:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40b1400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac16bd40 con 0x7f40ac0d0830 2024-03-24T00:32:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40b1400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac16c040 con 0x7f40ac15e0b0 2024-03-24T00:32:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40aa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40ac06fab0 0x7f40ac1567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40aa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40ac06fab0 0x7f40ac1567d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42722/0 (socket says 172.21.15.40:42722) 2024-03-24T00:32:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40aa800640 1 -- 172.21.15.40:0/410513967 learned_addr learned my addr 172.21.15.40:0/410513967 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40ab200640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f40ac15e0b0 0x7f40ac15c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40a1e00640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0d0830 0x7f40ac0cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40aa800640 1 -- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f40ac15e0b0 msgr2=0x7f40ac15c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40aa800640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f40ac15e0b0 0x7f40ac15c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40aa800640 1 -- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0d0830 msgr2=0x7f40ac0cbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40aa800640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0d0830 0x7f40ac0cbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40aa800640 1 -- 172.21.15.40:0/410513967 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac0c81e0 con 0x7f40ac06fab0 2024-03-24T00:32:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40aa800640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40ac06fab0 0x7f40ac1567d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f409c00c700 tx=0x7f409c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40a3400640 1 -- 172.21.15.40:0/410513967 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c018070 con 0x7f40ac06fab0 2024-03-24T00:32:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40a3400640 1 -- 172.21.15.40:0/410513967 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c0072d0 con 0x7f40ac06fab0 2024-03-24T00:32:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40a3400640 1 -- 172.21.15.40:0/410513967 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c007680 con 0x7f40ac06fab0 2024-03-24T00:32:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40b1400640 1 -- 172.21.15.40:0/410513967 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ac0c84c0 con 0x7f40ac06fab0 2024-03-24T00:32:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40b1400640 1 -- 172.21.15.40:0/410513967 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ac156d10 con 0x7f40ac06fab0 2024-03-24T00:32:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40b1400640 1 -- 172.21.15.40:0/410513967 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ac068170 con 0x7f40ac06fab0 2024-03-24T00:32:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.448+0000 7f40a3400640 1 -- 172.21.15.40:0/410513967 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f409c0047f0 con 0x7f40ac06fab0 2024-03-24T00:32:50.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.452+0000 7f40a3400640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4088067660 0x7f4088069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:50.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.452+0000 7f40a3400640 1 -- 172.21.15.40:0/410513967 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f409c084b60 con 0x7f40ac06fab0 2024-03-24T00:32:50.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.452+0000 7f40a1e00640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4088067660 0x7f4088069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:50.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.452+0000 7f40a1e00640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4088067660 0x7f4088069b20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f40ac159490 tx=0x7f4094009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:50.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.456+0000 7f40a3400640 1 -- 172.21.15.40:0/410513967 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f409c021080 con 0x7f40ac06fab0 2024-03-24T00:32:50.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:50.676+0000 7f40b1400640 1 -- 172.21.15.40:0/410513967 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_failover"]} v 0) v1 -- 0x7f40ac074480 con 0x7f40ac06fab0 2024-03-24T00:32:51.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.588+0000 7f40a3400640 1 -- 172.21.15.40:0/410513967 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_failover"]}]=0 v114) v1 ==== 158+0+0 (secure 0 0 0) 0x7f409c055550 con 0x7f40ac06fab0 2024-03-24T00:32:51.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.592+0000 7f40b1400640 1 -- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4088067660 msgr2=0x7f4088069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.592+0000 7f40b1400640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4088067660 0x7f4088069b20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f40ac159490 tx=0x7f4094009040 comp rx=0 tx=0).stop 2024-03-24T00:32:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.592+0000 7f40b1400640 1 -- 172.21.15.40:0/410513967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40ac06fab0 msgr2=0x7f40ac1567d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.592+0000 7f40b1400640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40ac06fab0 0x7f40ac1567d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f409c00c700 tx=0x7f409c00cbc0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.592+0000 7f40b1400640 1 -- 172.21.15.40:0/410513967 shutdown_connections 2024-03-24T00:32:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.592+0000 7f40b1400640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4088067660 0x7f4088069b20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.592+0000 7f40b1400640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f40ac15e0b0 0x7f40ac15c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.592+0000 7f40b1400640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40ac0d0830 0x7f40ac0cbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.592+0000 7f40b1400640 1 --2- 172.21.15.40:0/410513967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f40ac06fab0 0x7f40ac1567d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.592+0000 7f40b1400640 1 -- 172.21.15.40:0/410513967 >> 172.21.15.40:0/410513967 conn(0x7f40ac0d1c90 msgr2=0x7f40ac06b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.592+0000 7f40b1400640 1 -- 172.21.15.40:0/410513967 shutdown_connections 2024-03-24T00:32:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.592+0000 7f40b1400640 1 -- 172.21.15.40:0/410513967 wait complete. 2024-03-24T00:32:51.620 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:32:51.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.768+0000 7fdef1000640 1 Processor -- start 2024-03-24T00:32:51.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.768+0000 7fdef1000640 1 -- start start 2024-03-24T00:32:51.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdef1000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdeec15ffd0 0x7fdeec0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:51.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdef1000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec071de0 con 0x7fdeec0728d0 2024-03-24T00:32:51.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdef1000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeec0cea50 con 0x7fdeec15bd00 2024-03-24T00:32:51.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdef1000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec0ced30 con 0x7fdeec15ffd0 2024-03-24T00:32:51.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdeea800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdeec0728d0 0x7fdeec06cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56676/0 (socket says 172.21.15.40:56676) 2024-03-24T00:32:51.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdeea800640 1 -- 172.21.15.40:0/573104902 learned_addr learned my addr 172.21.15.40:0/573104902 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:51.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee1400640 1 --2- 172.21.15.40:0/573104902 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdeec15ffd0 0x7fdeec0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:51.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849300887 0 0) 0x7fdeec071de0 con 0x7fdeec0728d0 2024-03-24T00:32:51.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fded0003530 con 0x7fdeec0728d0 2024-03-24T00:32:51.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2746288596 0 0) 0x7fded0003530 con 0x7fdeec0728d0 2024-03-24T00:32:51.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdeec15ffd0 msgr2=0x7fdeec0ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 --2- 172.21.15.40:0/573104902 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdeec15ffd0 0x7fdeec0ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 >> v1:172.21.15.53:6789/0 conn(0x7fdeec15bd00 legacy=0x7fdeec15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec071b00 con 0x7fdeec0728d0 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fdee4001680 con 0x7fdeec0728d0 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 >> v1:172.21.15.40:6789/0 conn(0x7fdeec0728d0 legacy=0x7fdeec06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 --2- 172.21.15.40:0/573104902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fded00039f0 0x7fded0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 --2- 172.21.15.40:0/573104902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fded0004810 0x7fded0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 --2- 172.21.15.40:0/573104902 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdeec15ffd0 0x7fded00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec0ced30 con 0x7fded0004810 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded000ba80 con 0x7fded00039f0 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded000bc80 con 0x7fdeec15ffd0 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee1400640 1 --2- 172.21.15.40:0/573104902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fded0004810 0x7fded0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee9e00640 1 --2- 172.21.15.40:0/573104902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fded00039f0 0x7fded0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdeea800640 1 --2- 172.21.15.40:0/573104902 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdeec15ffd0 0x7fded00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:51.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee1400640 1 -- 172.21.15.40:0/573104902 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdeec15ffd0 msgr2=0x7fded00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:51.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee1400640 1 --2- 172.21.15.40:0/573104902 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdeec15ffd0 0x7fded00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee1400640 1 -- 172.21.15.40:0/573104902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fded00039f0 msgr2=0x7fded0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:51.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee1400640 1 --2- 172.21.15.40:0/573104902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fded00039f0 0x7fded0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.772+0000 7fdee1400640 1 -- 172.21.15.40:0/573104902 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded0003530 con 0x7fded0004810 2024-03-24T00:32:51.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.776+0000 7fdee1400640 1 --2- 172.21.15.40:0/573104902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fded0004810 0x7fded0004c90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fded4009ab0 tx=0x7fded401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59605fffa6fd2247 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:51.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.776+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded40b2070 con 0x7fded0004810 2024-03-24T00:32:51.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.776+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded40a5510 con 0x7fded0004810 2024-03-24T00:32:51.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.776+0000 7fdee9400640 1 -- 172.21.15.40:0/573104902 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded40096b0 con 0x7fded0004810 2024-03-24T00:32:51.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.776+0000 7fdef1000640 1 -- 172.21.15.40:0/573104902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fded0004810 msgr2=0x7fded0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.776+0000 7fdef1000640 1 --2- 172.21.15.40:0/573104902 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fded0004810 0x7fded0004c90 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fded4009ab0 tx=0x7fded401ca30 comp rx=0 tx=0).stop 2024-03-24T00:32:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.776+0000 7fdeea800640 1 -- 172.21.15.40:0/573104902 reap_dead start 2024-03-24T00:32:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.776+0000 7fdef1000640 1 -- 172.21.15.40:0/573104902 shutdown_connections 2024-03-24T00:32:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.776+0000 7fdef1000640 1 -- 172.21.15.40:0/573104902 >> 172.21.15.40:0/573104902 conn(0x7fdeec0b1660 msgr2=0x7fdeec0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.776+0000 7fdef1000640 1 -- 172.21.15.40:0/573104902 shutdown_connections 2024-03-24T00:32:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.776+0000 7fdef1000640 1 -- 172.21.15.40:0/573104902 wait complete. 2024-03-24T00:32:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.776+0000 7fdef1000640 1 Processor -- start 2024-03-24T00:32:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdef1000640 1 -- start start 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdef1000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fded0004810 0x7fdeec14c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdef1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec0b3440 0x7fdeec14ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdef1000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeec155880 0x7fdeec155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdef1000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec15c0e0 con 0x7fdeec0b3440 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdef1000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec072d00 con 0x7fdeec155880 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdef1000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeec0ae8e0 con 0x7fded0004810 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec0b3440 0x7fdeec14ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec0b3440 0x7fdeec14ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42726/0 (socket says 172.21.15.40:42726) 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee1400640 1 -- 172.21.15.40:0/2401286305 learned_addr learned my addr 172.21.15.40:0/2401286305 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdeea800640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeec155880 0x7fdeec155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee9e00640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fded0004810 0x7fdeec14c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee1400640 1 -- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fded0004810 msgr2=0x7fdeec14c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee1400640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fded0004810 0x7fdeec14c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee1400640 1 -- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeec155880 msgr2=0x7fdeec155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee1400640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeec155880 0x7fdeec155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee1400640 1 -- 172.21.15.40:0/2401286305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeec0b3950 con 0x7fdeec0b3440 2024-03-24T00:32:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee1400640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec0b3440 0x7fdeec14ced0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fded401c860 tx=0x7fded0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee2a00640 1 -- 172.21.15.40:0/2401286305 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded40bc020 con 0x7fdeec0b3440 2024-03-24T00:32:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee2a00640 1 -- 172.21.15.40:0/2401286305 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded4004020 con 0x7fdeec0b3440 2024-03-24T00:32:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdee2a00640 1 -- 172.21.15.40:0/2401286305 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded40b2040 con 0x7fdeec0b3440 2024-03-24T00:32:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdef1000640 1 -- 172.21.15.40:0/2401286305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeec0b3c30 con 0x7fdeec0b3440 2024-03-24T00:32:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdef1000640 1 -- 172.21.15.40:0/2401286305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeec14db10 con 0x7fdeec0b3440 2024-03-24T00:32:51.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.780+0000 7fdef1000640 1 -- 172.21.15.40:0/2401286305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeec068170 con 0x7fdeec0b3440 2024-03-24T00:32:51.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.784+0000 7fdee2a00640 1 -- 172.21.15.40:0/2401286305 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fded4003620 con 0x7fdeec0b3440 2024-03-24T00:32:51.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.784+0000 7fdee2a00640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdec8067690 0x7fdec8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:51.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.784+0000 7fdee2a00640 1 -- 172.21.15.40:0/2401286305 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fded411faa0 con 0x7fdeec0b3440 2024-03-24T00:32:51.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.784+0000 7fdee9e00640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdec8067690 0x7fdec8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:51.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.784+0000 7fdee9e00640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdec8067690 0x7fdec8069b50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdedc005210 tx=0x7fdedc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:51.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.788+0000 7fdee2a00640 1 -- 172.21.15.40:0/2401286305 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fded40c6020 con 0x7fdeec0b3440 2024-03-24T00:32:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.948+0000 7fdef1000640 1 -- 172.21.15.40:0/2401286305 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdeec154050 con 0x7fdeec0b3440 2024-03-24T00:32:51.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.952+0000 7fdee2a00640 1 -- 172.21.15.40:0/2401286305 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v19)=0 v19) v1 ==== 122+0+0 (secure 0 0 0) 0x7fded4016740 con 0x7fdeec0b3440 2024-03-24T00:32:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.956+0000 7fdef1000640 1 -- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdec8067690 msgr2=0x7fdec8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.956+0000 7fdef1000640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdec8067690 0x7fdec8069b50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdedc005210 tx=0x7fdedc009040 comp rx=0 tx=0).stop 2024-03-24T00:32:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.956+0000 7fdef1000640 1 -- 172.21.15.40:0/2401286305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec0b3440 msgr2=0x7fdeec14ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.956+0000 7fdef1000640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec0b3440 0x7fdeec14ced0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fded401c860 tx=0x7fded0005840 comp rx=0 tx=0).stop 2024-03-24T00:32:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.956+0000 7fdef1000640 1 -- 172.21.15.40:0/2401286305 shutdown_connections 2024-03-24T00:32:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.956+0000 7fdef1000640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdec8067690 0x7fdec8069b50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.956+0000 7fdef1000640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeec155880 0x7fdeec155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.956+0000 7fdef1000640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdeec0b3440 0x7fdeec14ced0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.956+0000 7fdef1000640 1 --2- 172.21.15.40:0/2401286305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fded0004810 0x7fdeec14c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.956+0000 7fdef1000640 1 -- 172.21.15.40:0/2401286305 >> 172.21.15.40:0/2401286305 conn(0x7fdeec0b1660 msgr2=0x7fdeec0b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.956+0000 7fdef1000640 1 -- 172.21.15.40:0/2401286305 shutdown_connections 2024-03-24T00:32:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:51.956+0000 7fdef1000640 1 -- 172.21.15.40:0/2401286305 wait complete. 2024-03-24T00:32:51.978 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:32:51.978 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:32:52.063 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:32:52.171 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:32:52.171 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:32:52.171 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:32:52.180 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:32:52.180 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:32:52.180 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:32:52.190 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:32:52.190 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:32:52.190 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:32:52.232 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:32:52.232 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:32:52.232 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:32:52.244 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:32:52.244 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:32:52.245 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:32:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4ab2800640 1 Processor -- start 2024-03-24T00:32:52.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4ab2800640 1 -- start start 2024-03-24T00:32:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4ab2800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4aac13ec80 0x7f4aac1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4ab2800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4aac0ce800 0x7f4aac071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4ab2800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac070e00 con 0x7f4aac1419f0 2024-03-24T00:32:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4ab2800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aac141e50 con 0x7f4aac0ce800 2024-03-24T00:32:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4ab2800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aac0cee20 con 0x7f4aac13ec80 2024-03-24T00:32:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4aabe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4aac0ce800 0x7f4aac071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4aabe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4aac0ce800 0x7f4aac071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42740/0 (socket says 172.21.15.40:42740) 2024-03-24T00:32:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4aabe00640 1 -- 172.21.15.40:0/4114872765 learned_addr learned my addr 172.21.15.40:0/4114872765 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4aab400640 1 --2- 172.21.15.40:0/4114872765 >> v2:172.21.15.53:3300/0 conn(0x7f4aac13ec80 0x7f4aac1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4aabe00640 1 -- 172.21.15.40:0/4114872765 >> v2:172.21.15.53:3300/0 conn(0x7f4aac13ec80 msgr2=0x7f4aac1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4aabe00640 1 --2- 172.21.15.40:0/4114872765 >> v2:172.21.15.53:3300/0 conn(0x7f4aac13ec80 0x7f4aac1414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4aabe00640 1 -- 172.21.15.40:0/4114872765 >> v1:172.21.15.53:6789/0 conn(0x7f4aac1419f0 legacy=0x7f4aac0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4aabe00640 1 -- 172.21.15.40:0/4114872765 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac0cf100 con 0x7f4aac0ce800 2024-03-24T00:32:52.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.416+0000 7f4aab400640 1 --2- 172.21.15.40:0/4114872765 >> v2:172.21.15.53:3300/0 conn(0x7f4aac13ec80 0x7f4aac1414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:52.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.420+0000 7f4aabe00640 1 --2- 172.21.15.40:0/4114872765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aac0ce800 0x7f4aac071300 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4aac06b170 tx=0x7f4aa401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b221584a0dd4274c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:52.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.420+0000 7f4aaaa00640 1 -- 172.21.15.40:0/4114872765 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa40a58e0 con 0x7f4aac0ce800 2024-03-24T00:32:52.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.420+0000 7f4aaaa00640 1 -- 172.21.15.40:0/4114872765 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa40a5a80 con 0x7f4aac0ce800 2024-03-24T00:32:52.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.420+0000 7f4aaaa00640 1 -- 172.21.15.40:0/4114872765 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa40a5e50 con 0x7f4aac0ce800 2024-03-24T00:32:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.420+0000 7f4ab2800640 1 -- 172.21.15.40:0/4114872765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aac0ce800 msgr2=0x7f4aac071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.420+0000 7f4ab2800640 1 --2- 172.21.15.40:0/4114872765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aac0ce800 0x7f4aac071300 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f4aac06b170 tx=0x7f4aa401c9f0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.420+0000 7f4ab2800640 1 -- 172.21.15.40:0/4114872765 shutdown_connections 2024-03-24T00:32:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.420+0000 7f4ab2800640 1 --2- 172.21.15.40:0/4114872765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aac0ce800 0x7f4aac071300 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.420+0000 7f4ab2800640 1 --2- 172.21.15.40:0/4114872765 >> v2:172.21.15.53:3300/0 conn(0x7f4aac13ec80 0x7f4aac1414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.420+0000 7f4ab2800640 1 -- 172.21.15.40:0/4114872765 >> 172.21.15.40:0/4114872765 conn(0x7f4aac0b17b0 msgr2=0x7f4aac0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.420+0000 7f4ab2800640 1 -- 172.21.15.40:0/4114872765 shutdown_connections 2024-03-24T00:32:52.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.420+0000 7f4ab2800640 1 -- 172.21.15.40:0/4114872765 wait complete. 2024-03-24T00:32:52.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4ab2800640 1 Processor -- start 2024-03-24T00:32:52.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4ab2800640 1 -- start start 2024-03-24T00:32:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4ab2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac0ce800 0x7f4aac159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4ab2800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4aac13ec80 0x7f4aac06d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4ab2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aac1419f0 0x7f4aac06d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4ab2800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0723e0 con 0x7f4aac1419f0 2024-03-24T00:32:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4ab2800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac072260 con 0x7f4aac0ce800 2024-03-24T00:32:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4ab2800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac072560 con 0x7f4aac13ec80 2024-03-24T00:32:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aabe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aac1419f0 0x7f4aac06d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aabe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aac1419f0 0x7f4aac06d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42742/0 (socket says 172.21.15.40:42742) 2024-03-24T00:32:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aabe00640 1 -- 172.21.15.40:0/2410714203 learned_addr learned my addr 172.21.15.40:0/2410714203 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aab400640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac0ce800 0x7f4aac159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aa2a00640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4aac13ec80 0x7f4aac06d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:52.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aabe00640 1 -- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4aac13ec80 msgr2=0x7f4aac06d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aabe00640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4aac13ec80 0x7f4aac06d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aabe00640 1 -- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac0ce800 msgr2=0x7f4aac159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aabe00640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac0ce800 0x7f4aac159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aabe00640 1 -- 172.21.15.40:0/2410714203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac06e0d0 con 0x7f4aac1419f0 2024-03-24T00:32:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aabe00640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aac1419f0 0x7f4aac06d9d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40af040 tx=0x7f4aa4019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aa3e00640 1 -- 172.21.15.40:0/2410714203 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa40bc020 con 0x7f4aac1419f0 2024-03-24T00:32:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aa3e00640 1 -- 172.21.15.40:0/2410714203 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa4007e20 con 0x7f4aac1419f0 2024-03-24T00:32:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4aa3e00640 1 -- 172.21.15.40:0/2410714203 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa40a58e0 con 0x7f4aac1419f0 2024-03-24T00:32:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4ab2800640 1 -- 172.21.15.40:0/2410714203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aac06e3b0 con 0x7f4aac1419f0 2024-03-24T00:32:52.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4ab2800640 1 -- 172.21.15.40:0/2410714203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aac157a40 con 0x7f4aac1419f0 2024-03-24T00:32:52.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.424+0000 7f4ab2800640 1 -- 172.21.15.40:0/2410714203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aac068170 con 0x7f4aac1419f0 2024-03-24T00:32:52.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.428+0000 7f4aa3e00640 1 -- 172.21.15.40:0/2410714203 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4aa40bf020 con 0x7f4aac1419f0 2024-03-24T00:32:52.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.428+0000 7f4aa3e00640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4a88067660 0x7f4a88069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:52.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.428+0000 7f4aa3e00640 1 -- 172.21.15.40:0/2410714203 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f4aa4120860 con 0x7f4aac1419f0 2024-03-24T00:32:52.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.428+0000 7f4aab400640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4a88067660 0x7f4a88069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:52.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.428+0000 7f4aab400640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4a88067660 0x7f4a88069b20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c006e90 tx=0x7f4a9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:52.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.432+0000 7f4aa3e00640 1 -- 172.21.15.40:0/2410714203 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4aa40c6030 con 0x7f4aac1419f0 2024-03-24T00:32:52.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.628+0000 7f4ab2800640 1 -- 172.21.15.40:0/2410714203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4aac072b90 con 0x7f4aac1419f0 2024-03-24T00:32:52.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.628+0000 7f4aa3e00640 1 -- 172.21.15.40:0/2410714203 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f4aa40f7020 con 0x7f4aac1419f0 2024-03-24T00:32:52.632 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:52.632 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":23,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4853,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2069836835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2069836835},{"type":"v1","addr":"172.21.15.53:6836","nonce":2069836835}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5123,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2591078360","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":2591078360},{"type":"v1","addr":"172.21.15.53:6837","nonce":2591078360}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5126,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/3509292321","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":3509292321},{"type":"v1","addr":"172.21.15.40:6833","nonce":3509292321}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1476230307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1476230307},{"type":"v1","addr":"172.21.15.40:6835","nonce":1476230307}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[]} 2024-03-24T00:32:52.633 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 23 2024-03-24T00:32:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.632+0000 7f4ab2800640 1 -- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4a88067660 msgr2=0x7f4a88069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.632+0000 7f4ab2800640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4a88067660 0x7f4a88069b20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c006e90 tx=0x7f4a9c008040 comp rx=0 tx=0).stop 2024-03-24T00:32:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.632+0000 7f4ab2800640 1 -- 172.21.15.40:0/2410714203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aac1419f0 msgr2=0x7f4aac06d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.632+0000 7f4ab2800640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aac1419f0 0x7f4aac06d9d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40af040 tx=0x7f4aa4019790 comp rx=0 tx=0).stop 2024-03-24T00:32:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.632+0000 7f4ab2800640 1 -- 172.21.15.40:0/2410714203 shutdown_connections 2024-03-24T00:32:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.632+0000 7f4ab2800640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4a88067660 0x7f4a88069b20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.632+0000 7f4ab2800640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4aac1419f0 0x7f4aac06d9d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.632+0000 7f4ab2800640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4aac13ec80 0x7f4aac06d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.632+0000 7f4ab2800640 1 --2- 172.21.15.40:0/2410714203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4aac0ce800 0x7f4aac159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.632+0000 7f4ab2800640 1 -- 172.21.15.40:0/2410714203 >> 172.21.15.40:0/2410714203 conn(0x7f4aac0b17b0 msgr2=0x7f4aac0b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.632+0000 7f4ab2800640 1 -- 172.21.15.40:0/2410714203 shutdown_connections 2024-03-24T00:32:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.632+0000 7f4ab2800640 1 -- 172.21.15.40:0/2410714203 wait complete. 2024-03-24T00:32:52.654 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:32:52.654 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:32:52.654 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:32:52.654 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:32:52.654 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:32:52.654 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:32:52.655 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:32:52.655 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:32:52.655 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:32:52.655 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:32:52.655 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:32:52.655 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:32:52.655 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:32:52.655 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:32:52.656 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:32:52.656 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:32:52.670 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:32:52.671 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:32:52.671 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:32:52.671 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:32:52.672 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:32:52.673 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:32:52.673 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:32:52.674 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:32:52.715 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:32:52.715 DEBUG:teuthology.parallel:result is None 2024-03-24T00:32:52.716 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:32:52.716 DEBUG:teuthology.parallel:result is None 2024-03-24T00:32:52.719 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:32:52.719 DEBUG:teuthology.parallel:result is None 2024-03-24T00:32:52.721 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:32:52.721 DEBUG:teuthology.parallel:result is None 2024-03-24T00:32:52.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:32:52.802 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:32:52.803 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:32:52.814 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:32:52.814 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:32:52.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.864+0000 7fca0b200640 1 Processor -- start 2024-03-24T00:32:52.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.864+0000 7fca0b200640 1 -- start start 2024-03-24T00:32:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca0b200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fca0406e620 0x7fca040b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca0b200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca040ccfa0 0x7fca040cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca0b200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca04070f60 con 0x7fca0406ea00 2024-03-24T00:32:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca0b200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca040b4c00 con 0x7fca0406e620 2024-03-24T00:32:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca0b200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca040b3620 con 0x7fca040ccfa0 2024-03-24T00:32:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca09200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca040ccfa0 0x7fca040cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca03e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fca0406e620 0x7fca040b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca03e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fca0406e620 0x7fca040b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:35748/0 (socket says 172.21.15.40:35748) 2024-03-24T00:32:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca03e00640 1 -- 172.21.15.40:0/3492051158 learned_addr learned my addr 172.21.15.40:0/3492051158 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca09200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca040ccfa0 0x7fca040cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:50654/0 (socket says 172.21.15.40:50654) 2024-03-24T00:32:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca03e00640 1 -- 172.21.15.40:0/3492051158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca040ccfa0 msgr2=0x7fca040cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca03e00640 1 --2- 172.21.15.40:0/3492051158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca040ccfa0 0x7fca040cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca03e00640 1 -- 172.21.15.40:0/3492051158 >> v1:172.21.15.53:6789/0 conn(0x7fca0406ea00 legacy=0x7fca040b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca03e00640 1 -- 172.21.15.40:0/3492051158 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca040b3900 con 0x7fca0406e620 2024-03-24T00:32:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.868+0000 7fca03400640 1 -- 172.21.15.40:0/3492051158 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225614649 0 0) 0x7fca04070f60 con 0x7fca0406ea00 2024-03-24T00:32:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca03e00640 1 --2- 172.21.15.40:0/3492051158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca0406e620 0x7fca040b46c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400bb10 tx=0x7fc9f401eb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d104048c79809d00 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca03400640 1 -- 172.21.15.40:0/3492051158 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f40b4070 con 0x7fca0406e620 2024-03-24T00:32:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca03400640 1 -- 172.21.15.40:0/3492051158 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f40a7a50 con 0x7fca0406e620 2024-03-24T00:32:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca03400640 1 -- 172.21.15.40:0/3492051158 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f40a7e20 con 0x7fca0406e620 2024-03-24T00:32:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca0b200640 1 -- 172.21.15.40:0/3492051158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca0406e620 msgr2=0x7fca040b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca0b200640 1 --2- 172.21.15.40:0/3492051158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca0406e620 0x7fca040b46c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc9f400bb10 tx=0x7fc9f401eb70 comp rx=0 tx=0).stop 2024-03-24T00:32:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca0b200640 1 -- 172.21.15.40:0/3492051158 shutdown_connections 2024-03-24T00:32:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca0b200640 1 --2- 172.21.15.40:0/3492051158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca040ccfa0 0x7fca040cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca0b200640 1 --2- 172.21.15.40:0/3492051158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca0406e620 0x7fca040b46c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca0b200640 1 -- 172.21.15.40:0/3492051158 >> 172.21.15.40:0/3492051158 conn(0x7fca040b17b0 msgr2=0x7fca040b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca0b200640 1 -- 172.21.15.40:0/3492051158 shutdown_connections 2024-03-24T00:32:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca0b200640 1 -- 172.21.15.40:0/3492051158 wait complete. 2024-03-24T00:32:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca0b200640 1 Processor -- start 2024-03-24T00:32:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.872+0000 7fca0b200640 1 -- start start 2024-03-24T00:32:52.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca0b200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca0406e620 0x7fca041f8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:52.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca0b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca0406ea00 0x7fca0415c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:52.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca0b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca040ccfa0 0x7fca0415a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:52.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca0b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0406efd0 con 0x7fca0406ea00 2024-03-24T00:32:52.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca0b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0406ee50 con 0x7fca040ccfa0 2024-03-24T00:32:52.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca0b200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca04144920 con 0x7fca0406e620 2024-03-24T00:32:52.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fc9fb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca0406ea00 0x7fca0415c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fc9fb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca0406ea00 0x7fca0415c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42770/0 (socket says 172.21.15.40:42770) 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fc9fb400640 1 -- 172.21.15.40:0/60234882 learned_addr learned my addr 172.21.15.40:0/60234882 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca09200640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca040ccfa0 0x7fca0415a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca03e00640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca0406e620 0x7fca041f8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fc9fb400640 1 -- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca0406e620 msgr2=0x7fca041f8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fc9fb400640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca0406e620 0x7fca041f8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fc9fb400640 1 -- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca040ccfa0 msgr2=0x7fca0415a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fc9fb400640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca040ccfa0 0x7fca0415a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fc9fb400640 1 -- 172.21.15.40:0/60234882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0415cba0 con 0x7fca0406ea00 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca09200640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca040ccfa0 0x7fca0415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fc9fb400640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca0406ea00 0x7fca0415c660 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00c480 tx=0x7fc9ec00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca00c00640 1 -- 172.21.15.40:0/60234882 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec0052d0 con 0x7fca0406ea00 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca00c00640 1 -- 172.21.15.40:0/60234882 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9ec005470 con 0x7fca0406ea00 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca00c00640 1 -- 172.21.15.40:0/60234882 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec012870 con 0x7fca0406ea00 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca0b200640 1 -- 172.21.15.40:0/60234882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0415ada0 con 0x7fca0406ea00 2024-03-24T00:32:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca0b200640 1 -- 172.21.15.40:0/60234882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca041fb620 con 0x7fca0406ea00 2024-03-24T00:32:52.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca0b200640 1 -- 172.21.15.40:0/60234882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca04068170 con 0x7fca0406ea00 2024-03-24T00:32:52.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.876+0000 7fca00c00640 1 -- 172.21.15.40:0/60234882 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc9ec012a10 con 0x7fca0406ea00 2024-03-24T00:32:52.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.880+0000 7fca00c00640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc9e0067690 0x7fc9e0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:52.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.880+0000 7fca00c00640 1 -- 172.21.15.40:0/60234882 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fc9ec085240 con 0x7fca0406ea00 2024-03-24T00:32:52.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.880+0000 7fca03e00640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc9e0067690 0x7fc9e0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:52.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.880+0000 7fca03e00640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc9e0067690 0x7fc9e0069b50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc9f40b1000 tx=0x7fc9f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:52.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:52.884+0000 7fca00c00640 1 -- 172.21.15.40:0/60234882 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc9ec0561a0 con 0x7fca0406ea00 2024-03-24T00:32:53.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.056+0000 7fca0b200640 1 -- 172.21.15.40:0/60234882 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fca04072f50 con 0x7fca0406ea00 2024-03-24T00:32:53.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.604+0000 7fca00c00640 1 -- 172.21.15.40:0/60234882 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v38) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc9ec055aa0 con 0x7fca0406ea00 2024-03-24T00:32:53.605 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-03-24T00:32:53.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.604+0000 7fca0b200640 1 -- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc9e0067690 msgr2=0x7fc9e0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.604+0000 7fca0b200640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc9e0067690 0x7fc9e0069b50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc9f40b1000 tx=0x7fc9f4008040 comp rx=0 tx=0).stop 2024-03-24T00:32:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.604+0000 7fca0b200640 1 -- 172.21.15.40:0/60234882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca0406ea00 msgr2=0x7fca0415c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.604+0000 7fca0b200640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca0406ea00 0x7fca0415c660 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00c480 tx=0x7fc9ec00c940 comp rx=0 tx=0).stop 2024-03-24T00:32:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.604+0000 7fca0b200640 1 -- 172.21.15.40:0/60234882 shutdown_connections 2024-03-24T00:32:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.608+0000 7fca0b200640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc9e0067690 0x7fc9e0069b50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.608+0000 7fca0b200640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca040ccfa0 0x7fca0415a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.608+0000 7fca0b200640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca0406ea00 0x7fca0415c660 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.608+0000 7fca0b200640 1 --2- 172.21.15.40:0/60234882 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca0406e620 0x7fca041f8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.608+0000 7fca0b200640 1 -- 172.21.15.40:0/60234882 >> 172.21.15.40:0/60234882 conn(0x7fca040b17b0 msgr2=0x7fca0413f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.608+0000 7fca0b200640 1 -- 172.21.15.40:0/60234882 shutdown_connections 2024-03-24T00:32:53.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.608+0000 7fca0b200640 1 -- 172.21.15.40:0/60234882 wait complete. 2024-03-24T00:32:53.631 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:32:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb7bc00640 1 Processor -- start 2024-03-24T00:32:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb7bc00640 1 -- start start 2024-03-24T00:32:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb7bc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7feb74141bc0 0x7feb740ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb7bc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feb740ce630 0x7feb74071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb7bc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb74070e00 con 0x7feb7413ec80 2024-03-24T00:32:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb7bc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb740cee20 con 0x7feb740ce630 2024-03-24T00:32:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb7bc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb740cf100 con 0x7feb74141bc0 2024-03-24T00:32:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb79c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feb740ce630 0x7feb74071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb79c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feb740ce630 0x7feb74071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42784/0 (socket says 172.21.15.40:42784) 2024-03-24T00:32:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb79c00640 1 -- 172.21.15.40:0/1160857921 learned_addr learned my addr 172.21.15.40:0/1160857921 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb73e00640 1 --2- 172.21.15.40:0/1160857921 >> v2:172.21.15.53:3300/0 conn(0x7feb74141bc0 0x7feb740ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb79c00640 1 -- 172.21.15.40:0/1160857921 >> v2:172.21.15.53:3300/0 conn(0x7feb74141bc0 msgr2=0x7feb740ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb79c00640 1 --2- 172.21.15.40:0/1160857921 >> v2:172.21.15.53:3300/0 conn(0x7feb74141bc0 0x7feb740ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb79c00640 1 -- 172.21.15.40:0/1160857921 >> v1:172.21.15.53:6789/0 conn(0x7feb7413ec80 legacy=0x7feb741414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:53.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.772+0000 7feb79c00640 1 -- 172.21.15.40:0/1160857921 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb740cf3e0 con 0x7feb740ce630 2024-03-24T00:32:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.776+0000 7feb79c00640 1 --2- 172.21.15.40:0/1160857921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb740ce630 0x7feb74071300 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7feb7404fa60 tx=0x7feb6c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7d7db6ff9b277bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.776+0000 7feb73400640 1 -- 172.21.15.40:0/1160857921 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0a58e0 con 0x7feb740ce630 2024-03-24T00:32:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.776+0000 7feb73400640 1 -- 172.21.15.40:0/1160857921 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c0a5a80 con 0x7feb740ce630 2024-03-24T00:32:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.776+0000 7feb73400640 1 -- 172.21.15.40:0/1160857921 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0a5e50 con 0x7feb740ce630 2024-03-24T00:32:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.776+0000 7feb7bc00640 1 -- 172.21.15.40:0/1160857921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb740ce630 msgr2=0x7feb74071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.776+0000 7feb7bc00640 1 --2- 172.21.15.40:0/1160857921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb740ce630 0x7feb74071300 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7feb7404fa60 tx=0x7feb6c01c9f0 comp rx=0 tx=0).stop 2024-03-24T00:32:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.776+0000 7feb7bc00640 1 -- 172.21.15.40:0/1160857921 shutdown_connections 2024-03-24T00:32:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.776+0000 7feb7bc00640 1 --2- 172.21.15.40:0/1160857921 >> v2:172.21.15.53:3300/0 conn(0x7feb74141bc0 0x7feb740ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.776+0000 7feb7bc00640 1 --2- 172.21.15.40:0/1160857921 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb740ce630 0x7feb74071300 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.776+0000 7feb7bc00640 1 -- 172.21.15.40:0/1160857921 >> 172.21.15.40:0/1160857921 conn(0x7feb740b17b0 msgr2=0x7feb740b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.776+0000 7feb7bc00640 1 -- 172.21.15.40:0/1160857921 shutdown_connections 2024-03-24T00:32:53.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.776+0000 7feb7bc00640 1 -- 172.21.15.40:0/1160857921 wait complete. 2024-03-24T00:32:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb7bc00640 1 Processor -- start 2024-03-24T00:32:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb7bc00640 1 -- start start 2024-03-24T00:32:53.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb7bc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb740ce630 0x7feb74155ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb7bc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb7413ec80 0x7feb7415a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb7bc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb74141bc0 0x7feb7413de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb7bc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb740723e0 con 0x7feb74141bc0 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb7bc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb74072260 con 0x7feb740ce630 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb7bc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb74072560 con 0x7feb7413ec80 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb79c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb74141bc0 0x7feb7413de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb79c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb74141bc0 0x7feb7413de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42796/0 (socket says 172.21.15.40:42796) 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb79c00640 1 -- 172.21.15.40:0/2476146281 learned_addr learned my addr 172.21.15.40:0/2476146281 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb79200640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb740ce630 0x7feb74155ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb73e00640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb7413ec80 0x7feb7415a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb79c00640 1 -- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb7413ec80 msgr2=0x7feb7415a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb79c00640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb7413ec80 0x7feb7415a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb79c00640 1 -- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb740ce630 msgr2=0x7feb74155ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb79c00640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb740ce630 0x7feb74155ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:53.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb79c00640 1 -- 172.21.15.40:0/2476146281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb741560b0 con 0x7feb74141bc0 2024-03-24T00:32:53.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb73e00640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb7413ec80 0x7feb7415a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:53.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb79c00640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb74141bc0 0x7feb7413de00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0af040 tx=0x7feb6c019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:53.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb70c00640 1 -- 172.21.15.40:0/2476146281 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c0a58e0 con 0x7feb74141bc0 2024-03-24T00:32:53.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb70c00640 1 -- 172.21.15.40:0/2476146281 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c0be070 con 0x7feb74141bc0 2024-03-24T00:32:53.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb7bc00640 1 -- 172.21.15.40:0/2476146281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb74156390 con 0x7feb74141bc0 2024-03-24T00:32:53.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb70c00640 1 -- 172.21.15.40:0/2476146281 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c009510 con 0x7feb74141bc0 2024-03-24T00:32:53.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb7bc00640 1 -- 172.21.15.40:0/2476146281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7415e760 con 0x7feb74141bc0 2024-03-24T00:32:53.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.780+0000 7feb7bc00640 1 -- 172.21.15.40:0/2476146281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb74068170 con 0x7feb74141bc0 2024-03-24T00:32:53.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.784+0000 7feb70c00640 1 -- 172.21.15.40:0/2476146281 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7feb6c0b8470 con 0x7feb74141bc0 2024-03-24T00:32:53.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.784+0000 7feb70c00640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb54067660 0x7feb54069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:53.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.784+0000 7feb70c00640 1 -- 172.21.15.40:0/2476146281 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feb6c120820 con 0x7feb74141bc0 2024-03-24T00:32:53.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.784+0000 7feb79200640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb54067660 0x7feb54069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:53.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.784+0000 7feb79200640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb54067660 0x7feb54069b20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feb74159a80 tx=0x7feb68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:53.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:53.788+0000 7feb70c00640 1 -- 172.21.15.40:0/2476146281 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7feb6c0165e0 con 0x7feb74141bc0 2024-03-24T00:32:54.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb7bc00640 1 -- 172.21.15.40:0/2476146281 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7feb740b1f10 con 0x7feb74141bc0 2024-03-24T00:32:54.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb70c00640 1 -- 172.21.15.40:0/2476146281 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7feb6c0f1870 con 0x7feb74141bc0 2024-03-24T00:32:54.010 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:54.010 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:32:54.010 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-03-24T00:32:54.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.010 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-03-24T00:32:54.010 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lCkU6EhAA0q21F90epsGPK5x17s9NLw==", 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llkfOHhAAqKKFyEJzG0UxbLJMFPET8A==", 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:32:54.011 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lKMmwGRAAAKaOSnhTRbHIpPKXXkAe+Q==", 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.012 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l7quKJhAA8QsquiFFmj5H1G5yck0FpA==", 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAVdP9lVk3uBxAA9TC1OpeB5GIFlfJl5+ILBg==", 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:54.013 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYdP9lUc1IDhAAPwCV9eqL7bFibkfMFt9wPw==", 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAbdP9lwjyuFBAAJ1JAIAnLXAn03+lQPwwY9Q==", 2024-03-24T00:32:54.014 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAedP9lZ1bgGRAAuLAay48N7jXjj8/yaYwSqg==", 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAidP9lvyk4JhAA8w4/XBYHDLoqJ1zop9sUiw==", 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAldP9lcb9qLBAAXts6VU/4kVqUWS/WORV7zQ==", 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.017 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.017 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-03-24T00:32:54.017 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAodP9l8ipTMhAAjpagojh+woHyGvlQlIkHDw==", 2024-03-24T00:32:54.017 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.017 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:54.017 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:54.017 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.017 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQArdP9lmGfpORAAPvow5nF/O7RLmuXEqNWENg==", 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw==", 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:54.018 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow r", 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lpIZrNBAAzpGWZuQ6ndjNMAOmAsebvQ==", 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.019 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-03-24T00:32:54.020 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l6gXMMBAAd6Oe6S73hj/WfLXTsYiEZw==", 2024-03-24T00:32:54.020 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.021 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:54.021 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:32:54.021 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:32:54.021 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:32:54.021 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.021 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llFHCOhAATtv8gcL6BHPnxB1dHrg6vw==", 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAQdP9lmmd7KxAA4uk6aUKP6egYj6O1NgdadQ==", 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:32:54.022 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lebNrBxAAvbcCAJhDQn/5Uk1dJ01oHw==", 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-03-24T00:32:54.023 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lEnZyAxAAW5Lrm7LZPTVfLCOY6As1PQ==", 2024-03-24T00:32:54.024 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:32:54.024 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:32:54.024 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:32:54.024 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:32:54.024 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.024 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:54.024 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-24T00:32:54.024 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:32:54.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb7bc00640 1 -- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb54067660 msgr2=0x7feb54069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb7bc00640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb54067660 0x7feb54069b20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7feb74159a80 tx=0x7feb68009040 comp rx=0 tx=0).stop 2024-03-24T00:32:54.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb7bc00640 1 -- 172.21.15.40:0/2476146281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb74141bc0 msgr2=0x7feb7413de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb7bc00640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb74141bc0 0x7feb7413de00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0af040 tx=0x7feb6c019790 comp rx=0 tx=0).stop 2024-03-24T00:32:54.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb7bc00640 1 -- 172.21.15.40:0/2476146281 shutdown_connections 2024-03-24T00:32:54.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb7bc00640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb54067660 0x7feb54069b20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb7bc00640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb74141bc0 0x7feb7413de00 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb7bc00640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb7413ec80 0x7feb7415a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb7bc00640 1 --2- 172.21.15.40:0/2476146281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb740ce630 0x7feb74155ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb7bc00640 1 -- 172.21.15.40:0/2476146281 >> 172.21.15.40:0/2476146281 conn(0x7feb740b17b0 msgr2=0x7feb7406e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:54.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.008+0000 7feb7bc00640 1 -- 172.21.15.40:0/2476146281 shutdown_connections 2024-03-24T00:32:54.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.012+0000 7feb7bc00640 1 -- 172.21.15.40:0/2476146281 wait complete. 2024-03-24T00:32:54.032 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:32:54.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14d2e00640 1 Processor -- start 2024-03-24T00:32:54.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14d2e00640 1 -- start start 2024-03-24T00:32:54.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14d2e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc0b5c40 0x7f14cc161b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14d2e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14cc072b30 con 0x7f14cc147830 2024-03-24T00:32:54.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14d2e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14cc1622e0 con 0x7f14cc066e80 2024-03-24T00:32:54.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14d2e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc1620b0 con 0x7f14cc0b5c40 2024-03-24T00:32:54.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cbe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f14cc147830 0x7f14cc072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56702/0 (socket says 172.21.15.40:56702) 2024-03-24T00:32:54.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cbe00640 1 -- 172.21.15.40:0/1306704180 learned_addr learned my addr 172.21.15.40:0/1306704180 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:54.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14c3400640 1 --2- 172.21.15.40:0/1306704180 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc0b5c40 0x7f14cc161b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069751515 0 0) 0x7f14cc072b30 con 0x7f14cc147830 2024-03-24T00:32:54.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14b0003530 con 0x7f14cc147830 2024-03-24T00:32:54.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 188801943 0 0) 0x7f14b0003530 con 0x7f14cc147830 2024-03-24T00:32:54.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc0b5c40 msgr2=0x7f14cc161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:54.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 --2- 172.21.15.40:0/1306704180 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc0b5c40 0x7f14cc161b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 >> v1:172.21.15.53:6789/0 conn(0x7f14cc066e80 legacy=0x7f14cc06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14cc1625c0 con 0x7f14cc147830 2024-03-24T00:32:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f14bc001a00 con 0x7f14cc147830 2024-03-24T00:32:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 >> v1:172.21.15.40:6789/0 conn(0x7f14cc147830 legacy=0x7f14cc072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 --2- 172.21.15.40:0/1306704180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14b00039f0 0x7f14b0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 --2- 172.21.15.40:0/1306704180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14b00043a0 0x7f14b0004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 --2- 172.21.15.40:0/1306704180 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc0b5c40 0x7f14b0005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b000ba80 con 0x7f14b00043a0 2024-03-24T00:32:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b000bc00 con 0x7f14b00039f0 2024-03-24T00:32:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b000be00 con 0x7f14cc0b5c40 2024-03-24T00:32:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14c3400640 1 --2- 172.21.15.40:0/1306704180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14b00043a0 0x7f14b0004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14cbe00640 1 --2- 172.21.15.40:0/1306704180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14b00039f0 0x7f14b0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.172+0000 7f14d0e00640 1 --2- 172.21.15.40:0/1306704180 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc0b5c40 0x7f14b0005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14c3400640 1 -- 172.21.15.40:0/1306704180 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc0b5c40 msgr2=0x7f14b0005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:54.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14c3400640 1 --2- 172.21.15.40:0/1306704180 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc0b5c40 0x7f14b0005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14c3400640 1 -- 172.21.15.40:0/1306704180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14b00039f0 msgr2=0x7f14b0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:54.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14c3400640 1 --2- 172.21.15.40:0/1306704180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14b00039f0 0x7f14b0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14c3400640 1 -- 172.21.15.40:0/1306704180 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b0003530 con 0x7f14b00043a0 2024-03-24T00:32:54.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14c3400640 1 --2- 172.21.15.40:0/1306704180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14b00043a0 0x7f14b0004870 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009ae0 tx=0x7f14b401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47c88d9a60a45d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:54.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b40b2070 con 0x7f14b00043a0 2024-03-24T00:32:54.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b40a5660 con 0x7f14b00043a0 2024-03-24T00:32:54.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14cb400640 1 -- 172.21.15.40:0/1306704180 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b40096b0 con 0x7f14b00043a0 2024-03-24T00:32:54.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14d2e00640 1 -- 172.21.15.40:0/1306704180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14b00043a0 msgr2=0x7f14b0004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14d2e00640 1 --2- 172.21.15.40:0/1306704180 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14b00043a0 0x7f14b0004870 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f14b4009ae0 tx=0x7f14b401ca70 comp rx=0 tx=0).stop 2024-03-24T00:32:54.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14d0e00640 1 -- 172.21.15.40:0/1306704180 reap_dead start 2024-03-24T00:32:54.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14d2e00640 1 -- 172.21.15.40:0/1306704180 shutdown_connections 2024-03-24T00:32:54.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14d2e00640 1 -- 172.21.15.40:0/1306704180 >> 172.21.15.40:0/1306704180 conn(0x7f14cc0d0c20 msgr2=0x7f14cc0d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:54.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.176+0000 7f14d2e00640 1 -- 172.21.15.40:0/1306704180 shutdown_connections 2024-03-24T00:32:54.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14d2e00640 1 -- 172.21.15.40:0/1306704180 wait complete. 2024-03-24T00:32:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14d2e00640 1 Processor -- start 2024-03-24T00:32:54.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14d2e00640 1 -- start start 2024-03-24T00:32:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14d2e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14b00043a0 0x7f14cc1603a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14d2e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc146300 0x7f14cc146780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14d2e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14cc15d370 0x7f14cc15d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14d2e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc071f00 con 0x7f14b00043a0 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14d2e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc071cb0 con 0x7f14cc15d370 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14d2e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc0b6080 con 0x7f14cc146300 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14b00043a0 0x7f14cc1603a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14b00043a0 0x7f14cc1603a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42816/0 (socket says 172.21.15.40:42816) 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14cbe00640 1 -- 172.21.15.40:0/1354714619 learned_addr learned my addr 172.21.15.40:0/1354714619 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14c3400640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc146300 0x7f14cc146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14d0e00640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14cc15d370 0x7f14cc15d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14cbe00640 1 -- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc146300 msgr2=0x7f14cc146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14cbe00640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc146300 0x7f14cc146780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14cbe00640 1 -- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14cc15d370 msgr2=0x7f14cc15d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14cbe00640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14cc15d370 0x7f14cc15d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14cbe00640 1 -- 172.21.15.40:0/1354714619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14cc15dd60 con 0x7f14b00043a0 2024-03-24T00:32:54.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14cbe00640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14b00043a0 0x7f14cc1603a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f14bc007770 tx=0x7f14bc00a360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:54.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.180+0000 7f14c8c00640 1 -- 172.21.15.40:0/1354714619 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc017020 con 0x7f14b00043a0 2024-03-24T00:32:54.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.184+0000 7f14c8c00640 1 -- 172.21.15.40:0/1354714619 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14bc00ada0 con 0x7f14b00043a0 2024-03-24T00:32:54.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.184+0000 7f14c8c00640 1 -- 172.21.15.40:0/1354714619 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14bc013630 con 0x7f14b00043a0 2024-03-24T00:32:54.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.184+0000 7f14d2e00640 1 -- 172.21.15.40:0/1354714619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14cc141000 con 0x7f14b00043a0 2024-03-24T00:32:54.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.184+0000 7f14d2e00640 1 -- 172.21.15.40:0/1354714619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14cc1415c0 con 0x7f14b00043a0 2024-03-24T00:32:54.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.184+0000 7f14d2e00640 1 -- 172.21.15.40:0/1354714619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14cc068170 con 0x7f14b00043a0 2024-03-24T00:32:54.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.184+0000 7f14c8c00640 1 -- 172.21.15.40:0/1354714619 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f14bc01a020 con 0x7f14b00043a0 2024-03-24T00:32:54.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.184+0000 7f14c8c00640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f14a8067660 0x7f14a8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.184+0000 7f14c8c00640 1 -- 172.21.15.40:0/1354714619 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f14bc085e30 con 0x7f14b00043a0 2024-03-24T00:32:54.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.184+0000 7f14c3400640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f14a8067660 0x7f14a8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.188+0000 7f14c3400640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f14a8067660 0x7f14a8069b20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f14b40af000 tx=0x7f14b40046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:54.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.188+0000 7f14c8c00640 1 -- 172.21.15.40:0/1354714619 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f14bc021020 con 0x7f14b00043a0 2024-03-24T00:32:54.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.356+0000 7f14d2e00640 1 -- 172.21.15.40:0/1354714619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14cc141d40 con 0x7f14b00043a0 2024-03-24T00:32:54.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.356+0000 7f14c8c00640 1 -- 172.21.15.40:0/1354714619 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+12831 (secure 0 0 0) 0x7f14bc0567a0 con 0x7f14b00043a0 2024-03-24T00:32:54.360 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:54.360 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":40,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:32:53.612125+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:32:54.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.360+0000 7f14d2e00640 1 -- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f14a8067660 msgr2=0x7f14a8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.360+0000 7f14d2e00640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f14a8067660 0x7f14a8069b20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f14b40af000 tx=0x7f14b40046d0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.360+0000 7f14d2e00640 1 -- 172.21.15.40:0/1354714619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14b00043a0 msgr2=0x7f14cc1603a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.360+0000 7f14d2e00640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14b00043a0 0x7f14cc1603a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f14bc007770 tx=0x7f14bc00a360 comp rx=0 tx=0).stop 2024-03-24T00:32:54.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.360+0000 7f14d2e00640 1 -- 172.21.15.40:0/1354714619 shutdown_connections 2024-03-24T00:32:54.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.360+0000 7f14d2e00640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f14a8067660 0x7f14a8069b20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.360+0000 7f14d2e00640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14cc15d370 0x7f14cc15d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.360+0000 7f14d2e00640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14cc146300 0x7f14cc146780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.360+0000 7f14d2e00640 1 --2- 172.21.15.40:0/1354714619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14b00043a0 0x7f14cc1603a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.360+0000 7f14d2e00640 1 -- 172.21.15.40:0/1354714619 >> 172.21.15.40:0/1354714619 conn(0x7f14cc0d0c20 msgr2=0x7f14cc0b08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:54.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.360+0000 7f14d2e00640 1 -- 172.21.15.40:0/1354714619 shutdown_connections 2024-03-24T00:32:54.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.360+0000 7f14d2e00640 1 -- 172.21.15.40:0/1354714619 wait complete. 2024-03-24T00:32:54.381 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:32:54.382 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:32:54.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf9400640 1 Processor -- start 2024-03-24T00:32:54.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf9400640 1 -- start start 2024-03-24T00:32:54.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf9400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf40d2270 0x7fcaf4070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf9400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcaf406c950 0x7fcaf4161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf9400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf406d0f0 con 0x7fcaf415df80 2024-03-24T00:32:54.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf9400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf406b890 con 0x7fcaf406c950 2024-03-24T00:32:54.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf9400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf406b280 con 0x7fcaf40d2270 2024-03-24T00:32:54.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf3200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcaf406c950 0x7fcaf4161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf2800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcaf415df80 0x7fcaf40d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56708/0 (socket says 172.21.15.40:56708) 2024-03-24T00:32:54.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf2800640 1 -- 172.21.15.40:0/739426136 learned_addr learned my addr 172.21.15.40:0/739426136 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:54.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcae9e00640 1 --2- 172.21.15.40:0/739426136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf40d2270 0x7fcaf4070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf3200640 1 -- 172.21.15.40:0/739426136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf40d2270 msgr2=0x7fcaf4070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:54.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf3200640 1 --2- 172.21.15.40:0/739426136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf40d2270 0x7fcaf4070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf3200640 1 -- 172.21.15.40:0/739426136 >> v1:172.21.15.40:6789/0 conn(0x7fcaf415df80 legacy=0x7fcaf40d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.524+0000 7fcaf3200640 1 -- 172.21.15.40:0/739426136 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf406ce10 con 0x7fcaf406c950 2024-03-24T00:32:54.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.528+0000 7fcaf3200640 1 --2- 172.21.15.40:0/739426136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf406c950 0x7fcaf4161ad0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcaec009d90 tx=0x7fcaec01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5e638534378ce6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:54.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.528+0000 7fcaf1e00640 1 -- 172.21.15.40:0/739426136 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec0b2070 con 0x7fcaf406c950 2024-03-24T00:32:54.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.528+0000 7fcaf1e00640 1 -- 172.21.15.40:0/739426136 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaec0a59c0 con 0x7fcaf406c950 2024-03-24T00:32:54.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.528+0000 7fcaf1e00640 1 -- 172.21.15.40:0/739426136 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec0a5d90 con 0x7fcaf406c950 2024-03-24T00:32:54.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.528+0000 7fcaf9400640 1 -- 172.21.15.40:0/739426136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf406c950 msgr2=0x7fcaf4161ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.528+0000 7fcaf9400640 1 --2- 172.21.15.40:0/739426136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf406c950 0x7fcaf4161ad0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fcaec009d90 tx=0x7fcaec01c940 comp rx=0 tx=0).stop 2024-03-24T00:32:54.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.528+0000 7fcaf9400640 1 -- 172.21.15.40:0/739426136 shutdown_connections 2024-03-24T00:32:54.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.528+0000 7fcaf9400640 1 --2- 172.21.15.40:0/739426136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf406c950 0x7fcaf4161ad0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.528+0000 7fcaf9400640 1 --2- 172.21.15.40:0/739426136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf40d2270 0x7fcaf4070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.528+0000 7fcaf9400640 1 -- 172.21.15.40:0/739426136 >> 172.21.15.40:0/739426136 conn(0x7fcaf406ecf0 msgr2=0x7fcaf406f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:54.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.528+0000 7fcaf9400640 1 -- 172.21.15.40:0/739426136 shutdown_connections 2024-03-24T00:32:54.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.528+0000 7fcaf9400640 1 -- 172.21.15.40:0/739426136 wait complete. 2024-03-24T00:32:54.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf9400640 1 Processor -- start 2024-03-24T00:32:54.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf9400640 1 -- start start 2024-03-24T00:32:54.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf9400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf406c950 0x7fcaf4157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf40d2270 0x7fcaf4156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf415df80 0x7fcaf4151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf9400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf416af90 con 0x7fcaf40d2270 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf9400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf416ae10 con 0x7fcaf415df80 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf9400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf416b110 con 0x7fcaf406c950 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcae9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf40d2270 0x7fcaf4156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcae9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf40d2270 0x7fcaf4156c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42842/0 (socket says 172.21.15.40:42842) 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcae9e00640 1 -- 172.21.15.40:0/2044769066 learned_addr learned my addr 172.21.15.40:0/2044769066 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf2800640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf406c950 0x7fcaf4157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcae9e00640 1 -- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf406c950 msgr2=0x7fcaf4157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcae9e00640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf406c950 0x7fcaf4157300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf3200640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf415df80 0x7fcaf4151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcae9e00640 1 -- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf415df80 msgr2=0x7fcaf4151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcae9e00640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf415df80 0x7fcaf4151da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcae9e00640 1 -- 172.21.15.40:0/2044769066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf4150fe0 con 0x7fcaf40d2270 2024-03-24T00:32:54.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf2800640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf406c950 0x7fcaf4157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:54.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf3200640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf415df80 0x7fcaf4151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:54.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcae9e00640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf40d2270 0x7fcaf4156c50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00d830 tx=0x7fcadc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:54.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaeb400640 1 -- 172.21.15.40:0/2044769066 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcadc00b6c0 con 0x7fcaf40d2270 2024-03-24T00:32:54.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaeb400640 1 -- 172.21.15.40:0/2044769066 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcadc019070 con 0x7fcaf40d2270 2024-03-24T00:32:54.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaeb400640 1 -- 172.21.15.40:0/2044769066 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcadc00ba60 con 0x7fcaf40d2270 2024-03-24T00:32:54.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf9400640 1 -- 172.21.15.40:0/2044769066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf41512c0 con 0x7fcaf40d2270 2024-03-24T00:32:54.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.532+0000 7fcaf9400640 1 -- 172.21.15.40:0/2044769066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf4157a00 con 0x7fcaf40d2270 2024-03-24T00:32:54.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.536+0000 7fcaf9400640 1 -- 172.21.15.40:0/2044769066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf4068170 con 0x7fcaf40d2270 2024-03-24T00:32:54.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.536+0000 7fcaeb400640 1 -- 172.21.15.40:0/2044769066 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcadc01e460 con 0x7fcaf40d2270 2024-03-24T00:32:54.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.536+0000 7fcaeb400640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcad0067660 0x7fcad0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.536+0000 7fcaeb400640 1 -- 172.21.15.40:0/2044769066 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcadc085cd0 con 0x7fcaf40d2270 2024-03-24T00:32:54.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.536+0000 7fcaf2800640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcad0067660 0x7fcad0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.536+0000 7fcaf2800640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcad0067660 0x7fcad0069b20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcae4007fd0 tx=0x7fcae4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:54.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.540+0000 7fcaeb400640 1 -- 172.21.15.40:0/2044769066 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcadc056c00 con 0x7fcaf40d2270 2024-03-24T00:32:54.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 -- 172.21.15.40:0/2044769066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcaf40cf9d0 con 0x7fcaf40d2270 2024-03-24T00:32:54.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaeb400640 1 -- 172.21.15.40:0/2044769066 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcadc056500 con 0x7fcaf40d2270 2024-03-24T00:32:54.710 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:32:54.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 -- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcad0067660 msgr2=0x7fcad0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcad0067660 0x7fcad0069b20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcae4007fd0 tx=0x7fcae4009040 comp rx=0 tx=0).stop 2024-03-24T00:32:54.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 -- 172.21.15.40:0/2044769066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf40d2270 msgr2=0x7fcaf4156c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf40d2270 0x7fcaf4156c50 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00d830 tx=0x7fcadc00dcf0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 -- 172.21.15.40:0/2044769066 shutdown_connections 2024-03-24T00:32:54.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcad0067660 0x7fcad0069b20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf415df80 0x7fcaf4151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf40d2270 0x7fcaf4156c50 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 --2- 172.21.15.40:0/2044769066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf406c950 0x7fcaf4157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 -- 172.21.15.40:0/2044769066 >> 172.21.15.40:0/2044769066 conn(0x7fcaf406ecf0 msgr2=0x7fcaf415f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:54.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 -- 172.21.15.40:0/2044769066 shutdown_connections 2024-03-24T00:32:54.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.708+0000 7fcaf9400640 1 -- 172.21.15.40:0/2044769066 wait complete. 2024-03-24T00:32:54.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:32:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.868+0000 7f6ab7a00640 1 Processor -- start 2024-03-24T00:32:54.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.868+0000 7f6ab7a00640 1 -- start start 2024-03-24T00:32:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab7a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6ab0141bc0 0x7f6ab00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab7a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ab00ce630 0x7f6ab0071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab7a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab0070e00 con 0x7f6ab013ec80 2024-03-24T00:32:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab7a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab00cee20 con 0x7f6ab00ce630 2024-03-24T00:32:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab7a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab00cf100 con 0x7f6ab0141bc0 2024-03-24T00:32:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab5a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ab00ce630 0x7f6ab0071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab5a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ab00ce630 0x7f6ab0071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42854/0 (socket says 172.21.15.40:42854) 2024-03-24T00:32:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab5a00640 1 -- 172.21.15.40:0/1686009843 learned_addr learned my addr 172.21.15.40:0/1686009843 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:54.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6aafe00640 1 --2- 172.21.15.40:0/1686009843 >> v2:172.21.15.53:3300/0 conn(0x7f6ab0141bc0 0x7f6ab00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab5a00640 1 -- 172.21.15.40:0/1686009843 >> v2:172.21.15.53:3300/0 conn(0x7f6ab0141bc0 msgr2=0x7f6ab00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:54.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab5a00640 1 --2- 172.21.15.40:0/1686009843 >> v2:172.21.15.53:3300/0 conn(0x7f6ab0141bc0 0x7f6ab00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab5a00640 1 -- 172.21.15.40:0/1686009843 >> v1:172.21.15.53:6789/0 conn(0x7f6ab013ec80 legacy=0x7f6ab01414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab5a00640 1 -- 172.21.15.40:0/1686009843 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab00cf3e0 con 0x7f6ab00ce630 2024-03-24T00:32:54.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.872+0000 7f6ab5a00640 1 --2- 172.21.15.40:0/1686009843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab00ce630 0x7f6ab0071300 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6ab004fa60 tx=0x7f6aa801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6cf14aa2faf3b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:54.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6aaf400640 1 -- 172.21.15.40:0/1686009843 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80a58e0 con 0x7f6ab00ce630 2024-03-24T00:32:54.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6aaf400640 1 -- 172.21.15.40:0/1686009843 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa80a5a80 con 0x7f6ab00ce630 2024-03-24T00:32:54.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6aaf400640 1 -- 172.21.15.40:0/1686009843 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80a5e50 con 0x7f6ab00ce630 2024-03-24T00:32:54.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 -- 172.21.15.40:0/1686009843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab00ce630 msgr2=0x7f6ab0071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 --2- 172.21.15.40:0/1686009843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab00ce630 0x7f6ab0071300 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f6ab004fa60 tx=0x7f6aa801c9f0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 -- 172.21.15.40:0/1686009843 shutdown_connections 2024-03-24T00:32:54.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 --2- 172.21.15.40:0/1686009843 >> v2:172.21.15.53:3300/0 conn(0x7f6ab0141bc0 0x7f6ab00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 --2- 172.21.15.40:0/1686009843 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab00ce630 0x7f6ab0071300 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 -- 172.21.15.40:0/1686009843 >> 172.21.15.40:0/1686009843 conn(0x7f6ab00b17b0 msgr2=0x7f6ab00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:54.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 -- 172.21.15.40:0/1686009843 shutdown_connections 2024-03-24T00:32:54.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 -- 172.21.15.40:0/1686009843 wait complete. 2024-03-24T00:32:54.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 Processor -- start 2024-03-24T00:32:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 -- start start 2024-03-24T00:32:54.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab00ce630 0x7f6ab015c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab013ec80 0x7f6ab015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab0141bc0 0x7f6ab013db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab00723e0 con 0x7f6ab00ce630 2024-03-24T00:32:54.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0072260 con 0x7f6ab013ec80 2024-03-24T00:32:54.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab7a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab0072560 con 0x7f6ab0141bc0 2024-03-24T00:32:54.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab00ce630 0x7f6ab015c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab00ce630 0x7f6ab015c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42864/0 (socket says 172.21.15.40:42864) 2024-03-24T00:32:54.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab5000640 1 -- 172.21.15.40:0/810023155 learned_addr learned my addr 172.21.15.40:0/810023155 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:54.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6ab5a00640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab0141bc0 0x7f6ab013db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.876+0000 7f6aafe00640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab013ec80 0x7f6ab015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6ab5000640 1 -- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab0141bc0 msgr2=0x7f6ab013db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6ab5000640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab0141bc0 0x7f6ab013db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6ab5000640 1 -- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab013ec80 msgr2=0x7f6ab015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6ab5000640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab013ec80 0x7f6ab015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6ab5000640 1 -- 172.21.15.40:0/810023155 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab006da30 con 0x7f6ab00ce630 2024-03-24T00:32:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6ab5000640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab00ce630 0x7f6ab015c390 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400c500 tx=0x7f6aa400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6aacc00640 1 -- 172.21.15.40:0/810023155 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4016020 con 0x7f6ab00ce630 2024-03-24T00:32:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6aacc00640 1 -- 172.21.15.40:0/810023155 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa4005220 con 0x7f6ab00ce630 2024-03-24T00:32:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6aacc00640 1 -- 172.21.15.40:0/810023155 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4007030 con 0x7f6ab00ce630 2024-03-24T00:32:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6ab7a00640 1 -- 172.21.15.40:0/810023155 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab015a100 con 0x7f6ab00ce630 2024-03-24T00:32:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6ab7a00640 1 -- 172.21.15.40:0/810023155 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab015a6c0 con 0x7f6ab00ce630 2024-03-24T00:32:54.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6ab7a00640 1 -- 172.21.15.40:0/810023155 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab015a8c0 con 0x7f6ab00ce630 2024-03-24T00:32:54.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6aacc00640 1 -- 172.21.15.40:0/810023155 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6aa40071f0 con 0x7f6ab00ce630 2024-03-24T00:32:54.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6aacc00640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a90067690 0x7f6a90069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:54.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.880+0000 7f6aacc00640 1 -- 172.21.15.40:0/810023155 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6aa4084e80 con 0x7f6ab00ce630 2024-03-24T00:32:54.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.884+0000 7f6aafe00640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a90067690 0x7f6a90069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:54.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.884+0000 7f6aafe00640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a90067690 0x7f6a90069b50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6ab01564b0 tx=0x7f6a9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:54.884+0000 7f6aacc00640 1 -- 172.21.15.40:0/810023155 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6aa4020020 con 0x7f6ab00ce630 2024-03-24T00:32:55.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 -- 172.21.15.40:0/810023155 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6ab0068170 con 0x7f6ab00ce630 2024-03-24T00:32:55.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6aacc00640 1 -- 172.21.15.40:0/810023155 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v40) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6aa4055770 con 0x7f6ab00ce630 2024-03-24T00:32:55.053 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:55.053 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-03-24T00:32:55.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:32:55.053 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-03-24T00:32:55.054 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-03-24T00:32:55.054 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:32:55.054 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-03-24T00:32:55.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 -- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a90067690 msgr2=0x7f6a90069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:55.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a90067690 0x7f6a90069b50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6ab01564b0 tx=0x7f6a9c008040 comp rx=0 tx=0).stop 2024-03-24T00:32:55.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 -- 172.21.15.40:0/810023155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab00ce630 msgr2=0x7f6ab015c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:55.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab00ce630 0x7f6ab015c390 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400c500 tx=0x7f6aa400c9c0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 -- 172.21.15.40:0/810023155 shutdown_connections 2024-03-24T00:32:55.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a90067690 0x7f6a90069b50 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab0141bc0 0x7f6ab013db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab013ec80 0x7f6ab015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 --2- 172.21.15.40:0/810023155 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ab00ce630 0x7f6ab015c390 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 -- 172.21.15.40:0/810023155 >> 172.21.15.40:0/810023155 conn(0x7f6ab00b17b0 msgr2=0x7f6ab015ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:55.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 -- 172.21.15.40:0/810023155 shutdown_connections 2024-03-24T00:32:55.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.052+0000 7f6ab7a00640 1 -- 172.21.15.40:0/810023155 wait complete. 2024-03-24T00:32:55.074 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:32:55.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:32:55.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.216+0000 7f1207800640 1 Processor -- start 2024-03-24T00:32:55.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.216+0000 7f1207800640 1 -- start start 2024-03-24T00:32:55.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.216+0000 7f1207800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f120006cb80 0x7f12000b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:55.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.216+0000 7f1207800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12000b4840 0x7f1200071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:55.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.216+0000 7f1207800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12000b3260 con 0x7f120015ac60 2024-03-24T00:32:55.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.216+0000 7f1207800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12000b3e70 con 0x7f120006cb80 2024-03-24T00:32:55.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.216+0000 7f1207800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12000b3540 con 0x7f12000b4840 2024-03-24T00:32:55.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.216+0000 7f11ffe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f120006cb80 0x7f12000b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:55.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.216+0000 7f1204e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f120015ac60 0x7f12000b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56722/0 (socket says 172.21.15.40:56722) 2024-03-24T00:32:55.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.216+0000 7f1204e00640 1 -- 172.21.15.40:0/519176972 learned_addr learned my addr 172.21.15.40:0/519176972 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:55.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.216+0000 7f1205800640 1 --2- 172.21.15.40:0/519176972 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12000b4840 0x7f1200071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:55.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.220+0000 7f11ffe00640 1 -- 172.21.15.40:0/519176972 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12000b4840 msgr2=0x7f1200071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:55.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.220+0000 7f11ffe00640 1 --2- 172.21.15.40:0/519176972 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12000b4840 0x7f1200071b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.220+0000 7f11ffe00640 1 -- 172.21.15.40:0/519176972 >> v1:172.21.15.40:6789/0 conn(0x7f120015ac60 legacy=0x7f12000b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:55.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.220+0000 7f11ffe00640 1 -- 172.21.15.40:0/519176972 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12000b3820 con 0x7f120006cb80 2024-03-24T00:32:55.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.220+0000 7f11ffe00640 1 --2- 172.21.15.40:0/519176972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120006cb80 0x7f12000b4300 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0097c0 tx=0x7f11ec01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a0d8be5f6828574 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:55.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.220+0000 7f11ff400640 1 -- 172.21.15.40:0/519176972 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec0b0070 con 0x7f120006cb80 2024-03-24T00:32:55.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.220+0000 7f11ff400640 1 -- 172.21.15.40:0/519176972 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec0ab600 con 0x7f120006cb80 2024-03-24T00:32:55.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.220+0000 7f11ff400640 1 -- 172.21.15.40:0/519176972 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec0ab9d0 con 0x7f120006cb80 2024-03-24T00:32:55.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 -- 172.21.15.40:0/519176972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120006cb80 msgr2=0x7f12000b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:55.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 --2- 172.21.15.40:0/519176972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120006cb80 0x7f12000b4300 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0097c0 tx=0x7f11ec01c410 comp rx=0 tx=0).stop 2024-03-24T00:32:55.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 -- 172.21.15.40:0/519176972 shutdown_connections 2024-03-24T00:32:55.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 --2- 172.21.15.40:0/519176972 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12000b4840 0x7f1200071b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 --2- 172.21.15.40:0/519176972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120006cb80 0x7f12000b4300 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 -- 172.21.15.40:0/519176972 >> 172.21.15.40:0/519176972 conn(0x7f12000b1660 msgr2=0x7f12000b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:55.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 -- 172.21.15.40:0/519176972 shutdown_connections 2024-03-24T00:32:55.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 -- 172.21.15.40:0/519176972 wait complete. 2024-03-24T00:32:55.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 Processor -- start 2024-03-24T00:32:55.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 -- start start 2024-03-24T00:32:55.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120006cb80 0x7f1200149370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:55.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12000b4840 0x7f1200156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:55.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f120015ac60 0x7f1200157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:55.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1200167c50 con 0x7f120006cb80 2024-03-24T00:32:55.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1200167ad0 con 0x7f120015ac60 2024-03-24T00:32:55.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1207800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1200167dd0 con 0x7f12000b4840 2024-03-24T00:32:55.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1204e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120006cb80 0x7f1200149370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:55.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1204e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120006cb80 0x7f1200149370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42872/0 (socket says 172.21.15.40:42872) 2024-03-24T00:32:55.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f11ffe00640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12000b4840 0x7f1200156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:55.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1204e00640 1 -- 172.21.15.40:0/1392103111 learned_addr learned my addr 172.21.15.40:0/1392103111 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:55.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1205800640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f120015ac60 0x7f1200157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:55.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1204e00640 1 -- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12000b4840 msgr2=0x7f1200156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:55.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1204e00640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12000b4840 0x7f1200156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1204e00640 1 -- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f120015ac60 msgr2=0x7f1200157300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:55.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1204e00640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f120015ac60 0x7f1200157300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1204e00640 1 -- 172.21.15.40:0/1392103111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12000c5550 con 0x7f120006cb80 2024-03-24T00:32:55.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.224+0000 7f1205800640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f120015ac60 0x7f1200157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:55.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.228+0000 7f1204e00640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120006cb80 0x7f1200149370 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f11f400c3a0 tx=0x7f11f400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:55.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.228+0000 7f11fcc00640 1 -- 172.21.15.40:0/1392103111 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4017070 con 0x7f120006cb80 2024-03-24T00:32:55.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.228+0000 7f11fcc00640 1 -- 172.21.15.40:0/1392103111 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f4005cf0 con 0x7f120006cb80 2024-03-24T00:32:55.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.228+0000 7f11fcc00640 1 -- 172.21.15.40:0/1392103111 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f40072f0 con 0x7f120006cb80 2024-03-24T00:32:55.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.228+0000 7f1207800640 1 -- 172.21.15.40:0/1392103111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12000c5830 con 0x7f120006cb80 2024-03-24T00:32:55.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.228+0000 7f1207800640 1 -- 172.21.15.40:0/1392103111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12000c5df0 con 0x7f120006cb80 2024-03-24T00:32:55.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.228+0000 7f1207800640 1 -- 172.21.15.40:0/1392103111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1200068170 con 0x7f120006cb80 2024-03-24T00:32:55.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.228+0000 7f11fcc00640 1 -- 172.21.15.40:0/1392103111 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f11f401a070 con 0x7f120006cb80 2024-03-24T00:32:55.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.228+0000 7f11fcc00640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11e0067690 0x7f11e0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:55.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.228+0000 7f11fcc00640 1 -- 172.21.15.40:0/1392103111 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f11f4084810 con 0x7f120006cb80 2024-03-24T00:32:55.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.228+0000 7f11ffe00640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11e0067690 0x7f11e0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:55.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.232+0000 7f11ffe00640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11e0067690 0x7f11e0069b50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0ad000 tx=0x7f11ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:55.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.232+0000 7f11fcc00640 1 -- 172.21.15.40:0/1392103111 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f11f4055180 con 0x7f120006cb80 2024-03-24T00:32:55.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.400+0000 7f1207800640 1 -- 172.21.15.40:0/1392103111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f12000726f0 con 0x7f120006cb80 2024-03-24T00:32:55.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.612+0000 7f11fcc00640 1 -- 172.21.15.40:0/1392103111 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7f11f4059220 con 0x7f120006cb80 2024-03-24T00:32:55.615 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-03-24T00:32:55.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.616+0000 7f1207800640 1 -- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11e0067690 msgr2=0x7f11e0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:55.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.616+0000 7f1207800640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11e0067690 0x7f11e0069b50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0ad000 tx=0x7f11ec008040 comp rx=0 tx=0).stop 2024-03-24T00:32:55.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.616+0000 7f1207800640 1 -- 172.21.15.40:0/1392103111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120006cb80 msgr2=0x7f1200149370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:55.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.616+0000 7f1207800640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120006cb80 0x7f1200149370 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f11f400c3a0 tx=0x7f11f400c860 comp rx=0 tx=0).stop 2024-03-24T00:32:55.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.616+0000 7f1207800640 1 -- 172.21.15.40:0/1392103111 shutdown_connections 2024-03-24T00:32:55.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.616+0000 7f1207800640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f11e0067690 0x7f11e0069b50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.616+0000 7f1207800640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f120015ac60 0x7f1200157300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.616+0000 7f1207800640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12000b4840 0x7f1200156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.616+0000 7f1207800640 1 --2- 172.21.15.40:0/1392103111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f120006cb80 0x7f1200149370 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.616+0000 7f1207800640 1 -- 172.21.15.40:0/1392103111 >> 172.21.15.40:0/1392103111 conn(0x7f12000b1660 msgr2=0x7f12000b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:55.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.616+0000 7f1207800640 1 -- 172.21.15.40:0/1392103111 shutdown_connections 2024-03-24T00:32:55.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.616+0000 7f1207800640 1 -- 172.21.15.40:0/1392103111 wait complete. 2024-03-24T00:32:55.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:32:55.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f1591600640 1 Processor -- start 2024-03-24T00:32:55.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f1591600640 1 -- start start 2024-03-24T00:32:55.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f1591600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f158c15bd00 0x7f158c0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:55.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f1591600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f158c15ffd0 0x7f158c0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:55.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f1591600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158c072d60 con 0x7f158c0728d0 2024-03-24T00:32:55.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f1591600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f158c0cf790 con 0x7f158c15bd00 2024-03-24T00:32:55.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f1591600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f158c071de0 con 0x7f158c15ffd0 2024-03-24T00:32:55.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f158aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f158c15bd00 0x7f158c0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:55.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f158b400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f158c15ffd0 0x7f158c0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:55.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f158b400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f158c15ffd0 0x7f158c0cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:50752/0 (socket says 172.21.15.40:50752) 2024-03-24T00:32:55.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f158b400640 1 -- 172.21.15.40:0/199408070 learned_addr learned my addr 172.21.15.40:0/199408070 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:55.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f158a000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f158c0728d0 0x7f158c15f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:43134/0 (socket says 172.21.15.40:43134) 2024-03-24T00:32:55.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f158b400640 1 -- 172.21.15.40:0/199408070 >> v2:172.21.15.53:3300/0 conn(0x7f158c15bd00 msgr2=0x7f158c0ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:55.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f158b400640 1 --2- 172.21.15.40:0/199408070 >> v2:172.21.15.53:3300/0 conn(0x7f158c15bd00 0x7f158c0ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f158b400640 1 -- 172.21.15.40:0/199408070 >> v1:172.21.15.53:6789/0 conn(0x7f158c0728d0 legacy=0x7f158c15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:55.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.788+0000 7f158b400640 1 -- 172.21.15.40:0/199408070 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158c0cea50 con 0x7f158c15ffd0 2024-03-24T00:32:55.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.792+0000 7f158b400640 1 --2- 172.21.15.40:0/199408070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f158c15ffd0 0x7f158c0cf250 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1584009e50 tx=0x7f158401ca00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6da2da68073009 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:55.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.792+0000 7f1589600640 1 -- 172.21.15.40:0/199408070 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15840a58d0 con 0x7f158c15ffd0 2024-03-24T00:32:55.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.792+0000 7f1589600640 1 -- 172.21.15.40:0/199408070 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15840a5a70 con 0x7f158c15ffd0 2024-03-24T00:32:55.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.792+0000 7f1589600640 1 -- 172.21.15.40:0/199408070 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15840a5e40 con 0x7f158c15ffd0 2024-03-24T00:32:55.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.792+0000 7f1591600640 1 -- 172.21.15.40:0/199408070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f158c15ffd0 msgr2=0x7f158c0cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:55.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.792+0000 7f1591600640 1 --2- 172.21.15.40:0/199408070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f158c15ffd0 0x7f158c0cf250 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1584009e50 tx=0x7f158401ca00 comp rx=0 tx=0).stop 2024-03-24T00:32:55.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.792+0000 7f1591600640 1 -- 172.21.15.40:0/199408070 shutdown_connections 2024-03-24T00:32:55.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.792+0000 7f1591600640 1 --2- 172.21.15.40:0/199408070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f158c15ffd0 0x7f158c0cf250 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.792+0000 7f1591600640 1 --2- 172.21.15.40:0/199408070 >> v2:172.21.15.53:3300/0 conn(0x7f158c15bd00 0x7f158c0ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.792+0000 7f1591600640 1 -- 172.21.15.40:0/199408070 >> 172.21.15.40:0/199408070 conn(0x7f158c0b1660 msgr2=0x7f158c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:55.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.792+0000 7f1591600640 1 -- 172.21.15.40:0/199408070 shutdown_connections 2024-03-24T00:32:55.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.792+0000 7f1591600640 1 -- 172.21.15.40:0/199408070 wait complete. 2024-03-24T00:32:55.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1591600640 1 Processor -- start 2024-03-24T00:32:55.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1591600640 1 -- start start 2024-03-24T00:32:55.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1591600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f158c0728d0 0x7f158c14edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:55.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1591600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f158c15bd00 0x7f158c156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:55.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1591600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f158c15ffd0 0x7f158c0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:55.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1591600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158c06d320 con 0x7f158c0728d0 2024-03-24T00:32:55.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1591600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158c15c0e0 con 0x7f158c15ffd0 2024-03-24T00:32:55.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1591600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f158c06d4a0 con 0x7f158c15bd00 2024-03-24T00:32:55.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f158aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f158c0728d0 0x7f158c14edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:55.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f158aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f158c0728d0 0x7f158c14edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42884/0 (socket says 172.21.15.40:42884) 2024-03-24T00:32:55.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f158aa00640 1 -- 172.21.15.40:0/3011126913 learned_addr learned my addr 172.21.15.40:0/3011126913 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f158a000640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f158c15bd00 0x7f158c156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f158b400640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f158c15ffd0 0x7f158c0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f158aa00640 1 -- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f158c15bd00 msgr2=0x7f158c156020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f158aa00640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f158c15bd00 0x7f158c156020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f158aa00640 1 -- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f158c15ffd0 msgr2=0x7f158c0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f158aa00640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f158c15ffd0 0x7f158c0ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f158aa00640 1 -- 172.21.15.40:0/3011126913 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158c153160 con 0x7f158c0728d0 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f158aa00640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f158c0728d0 0x7f158c14edc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f157c00e960 tx=0x7f157c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1582a00640 1 -- 172.21.15.40:0/3011126913 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c017070 con 0x7f158c0728d0 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1582a00640 1 -- 172.21.15.40:0/3011126913 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157c00ca00 con 0x7f158c0728d0 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1582a00640 1 -- 172.21.15.40:0/3011126913 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c00cdb0 con 0x7f158c0728d0 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1591600640 1 -- 172.21.15.40:0/3011126913 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158c153440 con 0x7f158c0728d0 2024-03-24T00:32:55.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1591600640 1 -- 172.21.15.40:0/3011126913 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158c152e10 con 0x7f158c0728d0 2024-03-24T00:32:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.796+0000 7f1591600640 1 -- 172.21.15.40:0/3011126913 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f158c068170 con 0x7f158c0728d0 2024-03-24T00:32:55.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.800+0000 7f1582a00640 1 -- 172.21.15.40:0/3011126913 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f157c01a070 con 0x7f158c0728d0 2024-03-24T00:32:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.800+0000 7f1582a00640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f156c067660 0x7f156c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.800+0000 7f1582a00640 1 -- 172.21.15.40:0/3011126913 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f157c085a40 con 0x7f158c0728d0 2024-03-24T00:32:55.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.800+0000 7f158a000640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f156c067660 0x7f156c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:55.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.800+0000 7f158a000640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f156c067660 0x7f156c069b20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f158c1575c0 tx=0x7f1578007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:55.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.804+0000 7f1582a00640 1 -- 172.21.15.40:0/3011126913 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f157c056240 con 0x7f158c0728d0 2024-03-24T00:32:55.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:55.968+0000 7f1591600640 1 -- 172.21.15.40:0/3011126913 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f158c15f0e0 con 0x7f158c0728d0 2024-03-24T00:32:56.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.620+0000 7f1582a00640 1 -- 172.21.15.40:0/3011126913 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v42) v1 ==== 170+0+0 (secure 0 0 0) 0x7f157c05a2e0 con 0x7f158c0728d0 2024-03-24T00:32:56.623 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-03-24T00:32:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.624+0000 7f1591600640 1 -- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f156c067660 msgr2=0x7f156c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.624+0000 7f1591600640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f156c067660 0x7f156c069b20 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f158c1575c0 tx=0x7f1578007e90 comp rx=0 tx=0).stop 2024-03-24T00:32:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.624+0000 7f1591600640 1 -- 172.21.15.40:0/3011126913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f158c0728d0 msgr2=0x7f158c14edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.624+0000 7f1591600640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f158c0728d0 0x7f158c14edc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f157c00e960 tx=0x7f157c00ee20 comp rx=0 tx=0).stop 2024-03-24T00:32:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.624+0000 7f1591600640 1 -- 172.21.15.40:0/3011126913 shutdown_connections 2024-03-24T00:32:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.624+0000 7f1591600640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f156c067660 0x7f156c069b20 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:56.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.624+0000 7f1591600640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f158c15ffd0 0x7f158c0ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.624+0000 7f1591600640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f158c15bd00 0x7f158c156020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.624+0000 7f1591600640 1 --2- 172.21.15.40:0/3011126913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f158c0728d0 0x7f158c14edc0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.624+0000 7f1591600640 1 -- 172.21.15.40:0/3011126913 >> 172.21.15.40:0/3011126913 conn(0x7f158c0b1660 msgr2=0x7f158c1581c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:56.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.624+0000 7f1591600640 1 -- 172.21.15.40:0/3011126913 shutdown_connections 2024-03-24T00:32:56.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.624+0000 7f1591600640 1 -- 172.21.15.40:0/3011126913 wait complete. 2024-03-24T00:32:56.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:32:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.804+0000 7f1a1fe00640 1 Processor -- start 2024-03-24T00:32:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.804+0000 7f1a1fe00640 1 -- start start 2024-03-24T00:32:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a2015dc80 0x7f1a200b9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a200ce870 0x7f1a200ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1fe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a200cf240 con 0x7f1a200b6c10 2024-03-24T00:32:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1fe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a200b8050 con 0x7f1a200ce870 2024-03-24T00:32:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a200b7440 con 0x7f1a2015dc80 2024-03-24T00:32:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a200ce870 0x7f1a200ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a200ce870 0x7f1a200ced00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42896/0 (socket says 172.21.15.40:42896) 2024-03-24T00:32:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1f400640 1 -- 172.21.15.40:0/1737336222 learned_addr learned my addr 172.21.15.40:0/1737336222 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1ea00640 1 --2- 172.21.15.40:0/1737336222 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a2015dc80 0x7f1a200b9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1f400640 1 -- 172.21.15.40:0/1737336222 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a2015dc80 msgr2=0x7f1a200b9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1f400640 1 --2- 172.21.15.40:0/1737336222 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a2015dc80 0x7f1a200b9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1f400640 1 -- 172.21.15.40:0/1737336222 >> v1:172.21.15.40:6789/0 conn(0x7f1a200b6c10 legacy=0x7f1a200d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.808+0000 7f1a1f400640 1 -- 172.21.15.40:0/1737336222 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a200b7720 con 0x7f1a200ce870 2024-03-24T00:32:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1f400640 1 --2- 172.21.15.40:0/1737336222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a200ce870 0x7f1a200ced00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009e50 tx=0x7f1a1801f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d247980deb84863b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1e000640 1 -- 172.21.15.40:0/1737336222 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a180b4070 con 0x7f1a200ce870 2024-03-24T00:32:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1e000640 1 -- 172.21.15.40:0/1737336222 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a180af400 con 0x7f1a200ce870 2024-03-24T00:32:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1e000640 1 -- 172.21.15.40:0/1737336222 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a180af7d0 con 0x7f1a200ce870 2024-03-24T00:32:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 -- 172.21.15.40:0/1737336222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a200ce870 msgr2=0x7f1a200ced00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:56.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 --2- 172.21.15.40:0/1737336222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a200ce870 0x7f1a200ced00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009e50 tx=0x7f1a1801f210 comp rx=0 tx=0).stop 2024-03-24T00:32:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 -- 172.21.15.40:0/1737336222 shutdown_connections 2024-03-24T00:32:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 --2- 172.21.15.40:0/1737336222 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a200ce870 0x7f1a200ced00 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 --2- 172.21.15.40:0/1737336222 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a2015dc80 0x7f1a200b9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 -- 172.21.15.40:0/1737336222 >> 172.21.15.40:0/1737336222 conn(0x7f1a20071b70 msgr2=0x7f1a20071f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 -- 172.21.15.40:0/1737336222 shutdown_connections 2024-03-24T00:32:56.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 -- 172.21.15.40:0/1737336222 wait complete. 2024-03-24T00:32:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 Processor -- start 2024-03-24T00:32:56.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 -- start start 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a200b6c10 0x7f1a20153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a200ce870 0x7f1a2015f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a2015dc80 0x7f1a200c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2015e060 con 0x7f1a200ce870 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a200b7ce0 con 0x7f1a200b6c10 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a1fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a200b7010 con 0x7f1a2015dc80 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.812+0000 7f1a16000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a200ce870 0x7f1a2015f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a16000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a200ce870 0x7f1a2015f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42908/0 (socket says 172.21.15.40:42908) 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a16000640 1 -- 172.21.15.40:0/2591878051 learned_addr learned my addr 172.21.15.40:0/2591878051 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a1ea00640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a200b6c10 0x7f1a20153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a1f400640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a2015dc80 0x7f1a200c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a16000640 1 -- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a2015dc80 msgr2=0x7f1a200c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:56.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a16000640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a2015dc80 0x7f1a200c8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a16000640 1 -- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a200b6c10 msgr2=0x7f1a20153cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a16000640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a200b6c10 0x7f1a20153cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a16000640 1 -- 172.21.15.40:0/2591878051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a200c7bd0 con 0x7f1a200ce870 2024-03-24T00:32:56.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a16000640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a200ce870 0x7f1a2015f930 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800c420 tx=0x7f1a0800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a17400640 1 -- 172.21.15.40:0/2591878051 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a080052d0 con 0x7f1a200ce870 2024-03-24T00:32:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a17400640 1 -- 172.21.15.40:0/2591878051 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a08005470 con 0x7f1a200ce870 2024-03-24T00:32:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a1fe00640 1 -- 172.21.15.40:0/2591878051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a200c7eb0 con 0x7f1a200ce870 2024-03-24T00:32:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a17400640 1 -- 172.21.15.40:0/2591878051 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a08007260 con 0x7f1a200ce870 2024-03-24T00:32:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a1fe00640 1 -- 172.21.15.40:0/2591878051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a200c8470 con 0x7f1a200ce870 2024-03-24T00:32:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a1fe00640 1 -- 172.21.15.40:0/2591878051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a2004f410 con 0x7f1a200ce870 2024-03-24T00:32:56.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.816+0000 7f1a17400640 1 -- 172.21.15.40:0/2591878051 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f1a08007400 con 0x7f1a200ce870 2024-03-24T00:32:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.820+0000 7f1a17400640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f19fc067660 0x7f19fc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:56.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.820+0000 7f1a17400640 1 -- 172.21.15.40:0/2591878051 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f1a080853e0 con 0x7f1a200ce870 2024-03-24T00:32:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.820+0000 7f1a1ea00640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f19fc067660 0x7f19fc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:56.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.820+0000 7f1a1ea00640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f19fc067660 0x7f19fc069b20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1a100079c0 tx=0x7f1a10008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:56.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:56.824+0000 7f1a17400640 1 -- 172.21.15.40:0/2591878051 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1a08055be0 con 0x7f1a200ce870 2024-03-24T00:32:57.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.020+0000 7f1a1fe00640 1 -- 172.21.15.40:0/2591878051 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1a20161210 con 0x7f1a200ce870 2024-03-24T00:32:57.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.632+0000 7f1a17400640 1 -- 172.21.15.40:0/2591878051 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v26) v1 ==== 172+0+0 (secure 0 0 0) 0x7f1a08059c80 con 0x7f1a200ce870 2024-03-24T00:32:57.638 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 6 and data pool 7 2024-03-24T00:32:57.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.648+0000 7f1a1fe00640 1 -- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f19fc067660 msgr2=0x7f19fc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:57.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.648+0000 7f1a1fe00640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f19fc067660 0x7f19fc069b20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f1a100079c0 tx=0x7f1a10008040 comp rx=0 tx=0).stop 2024-03-24T00:32:57.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.648+0000 7f1a1fe00640 1 -- 172.21.15.40:0/2591878051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a200ce870 msgr2=0x7f1a2015f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:57.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.648+0000 7f1a1fe00640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a200ce870 0x7f1a2015f930 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800c420 tx=0x7f1a0800c8e0 comp rx=0 tx=0).stop 2024-03-24T00:32:57.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.656+0000 7f1a1fe00640 1 -- 172.21.15.40:0/2591878051 shutdown_connections 2024-03-24T00:32:57.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.656+0000 7f1a1fe00640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f19fc067660 0x7f19fc069b20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:57.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.656+0000 7f1a1fe00640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a2015dc80 0x7f1a200c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:57.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.656+0000 7f1a1fe00640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a200ce870 0x7f1a2015f930 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:57.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.656+0000 7f1a1fe00640 1 --2- 172.21.15.40:0/2591878051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a200b6c10 0x7f1a20153cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:57.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.656+0000 7f1a1fe00640 1 -- 172.21.15.40:0/2591878051 >> 172.21.15.40:0/2591878051 conn(0x7f1a20071b70 msgr2=0x7f1a200d4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:57.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.660+0000 7f1a1fe00640 1 -- 172.21.15.40:0/2591878051 shutdown_connections 2024-03-24T00:32:57.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.660+0000 7f1a1fe00640 1 -- 172.21.15.40:0/2591878051 wait complete. 2024-03-24T00:32:57.695 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:32:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa096c00640 1 Processor -- start 2024-03-24T00:32:57.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa096c00640 1 -- start start 2024-03-24T00:32:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa096c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa09013ec80 0x7fa0901414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa096c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa0901419f0 0x7fa0900ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa096c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa090070e00 con 0x7fa0900ce630 2024-03-24T00:32:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa096c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa090141e50 con 0x7fa0901419f0 2024-03-24T00:32:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa096c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0900cee20 con 0x7fa09013ec80 2024-03-24T00:32:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa087e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa0901419f0 0x7fa0900ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa087e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa0901419f0 0x7fa0900ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49524/0 (socket says 172.21.15.40:49524) 2024-03-24T00:32:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa087e00640 1 -- 172.21.15.40:0/3913502217 learned_addr learned my addr 172.21.15.40:0/3913502217 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa08fe00640 1 --2- 172.21.15.40:0/3913502217 >> v2:172.21.15.53:3300/0 conn(0x7fa09013ec80 0x7fa0901414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa087e00640 1 -- 172.21.15.40:0/3913502217 >> v2:172.21.15.53:3300/0 conn(0x7fa09013ec80 msgr2=0x7fa0901414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:57.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa087e00640 1 --2- 172.21.15.40:0/3913502217 >> v2:172.21.15.53:3300/0 conn(0x7fa09013ec80 0x7fa0901414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:57.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.836+0000 7fa087e00640 1 -- 172.21.15.40:0/3913502217 >> v1:172.21.15.53:6789/0 conn(0x7fa0900ce630 legacy=0x7fa09015f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:57.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa087e00640 1 -- 172.21.15.40:0/3913502217 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0900cf100 con 0x7fa0901419f0 2024-03-24T00:32:57.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa087e00640 1 --2- 172.21.15.40:0/3913502217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0901419f0 0x7fa0900ce0f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa09006b170 tx=0x7fa07801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c18fbe3883c686 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:57.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa08f400640 1 -- 172.21.15.40:0/3913502217 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0780a9400 con 0x7fa0901419f0 2024-03-24T00:32:57.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa08f400640 1 -- 172.21.15.40:0/3913502217 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0780a95a0 con 0x7fa0901419f0 2024-03-24T00:32:57.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa08f400640 1 -- 172.21.15.40:0/3913502217 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0780a9970 con 0x7fa0901419f0 2024-03-24T00:32:57.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa096c00640 1 -- 172.21.15.40:0/3913502217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0901419f0 msgr2=0x7fa0900ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:57.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa096c00640 1 --2- 172.21.15.40:0/3913502217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0901419f0 0x7fa0900ce0f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fa09006b170 tx=0x7fa07801c520 comp rx=0 tx=0).stop 2024-03-24T00:32:57.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa096c00640 1 -- 172.21.15.40:0/3913502217 shutdown_connections 2024-03-24T00:32:57.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa096c00640 1 --2- 172.21.15.40:0/3913502217 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0901419f0 0x7fa0900ce0f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:57.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa096c00640 1 --2- 172.21.15.40:0/3913502217 >> v2:172.21.15.53:3300/0 conn(0x7fa09013ec80 0x7fa0901414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:57.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa096c00640 1 -- 172.21.15.40:0/3913502217 >> 172.21.15.40:0/3913502217 conn(0x7fa0900b17b0 msgr2=0x7fa0900b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:57.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa096c00640 1 -- 172.21.15.40:0/3913502217 shutdown_connections 2024-03-24T00:32:57.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.840+0000 7fa096c00640 1 -- 172.21.15.40:0/3913502217 wait complete. 2024-03-24T00:32:57.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa096c00640 1 Processor -- start 2024-03-24T00:32:57.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa096c00640 1 -- start start 2024-03-24T00:32:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa096c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0900ce630 0x7fa0901556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa096c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa09013ec80 0x7fa090137d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa096c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0901419f0 0x7fa090138290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:57.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa096c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa090168ec0 con 0x7fa0900ce630 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa096c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa090168d40 con 0x7fa0901419f0 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa096c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa090169040 con 0x7fa09013ec80 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa08fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0900ce630 0x7fa0901556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa08fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0900ce630 0x7fa0901556a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49532/0 (socket says 172.21.15.40:49532) 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa08fe00640 1 -- 172.21.15.40:0/688152286 learned_addr learned my addr 172.21.15.40:0/688152286 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa087e00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa09013ec80 0x7fa090137d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa094c00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0901419f0 0x7fa090138290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa08fe00640 1 -- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa09013ec80 msgr2=0x7fa090137d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa08fe00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa09013ec80 0x7fa090137d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa08fe00640 1 -- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0901419f0 msgr2=0x7fa090138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa08fe00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0901419f0 0x7fa090138290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa08fe00640 1 -- 172.21.15.40:0/688152286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09013d630 con 0x7fa0900ce630 2024-03-24T00:32:57.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa08fe00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0900ce630 0x7fa0901556a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa08000d830 tx=0x7fa08000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa08cc00640 1 -- 172.21.15.40:0/688152286 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08000b6c0 con 0x7fa0900ce630 2024-03-24T00:32:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa08cc00640 1 -- 172.21.15.40:0/688152286 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa080019070 con 0x7fa0900ce630 2024-03-24T00:32:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa08cc00640 1 -- 172.21.15.40:0/688152286 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08000bae0 con 0x7fa0900ce630 2024-03-24T00:32:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa096c00640 1 -- 172.21.15.40:0/688152286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09013d910 con 0x7fa0900ce630 2024-03-24T00:32:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.844+0000 7fa096c00640 1 -- 172.21.15.40:0/688152286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa090156680 con 0x7fa0900ce630 2024-03-24T00:32:57.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.848+0000 7fa096c00640 1 -- 172.21.15.40:0/688152286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa090068170 con 0x7fa0900ce630 2024-03-24T00:32:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.848+0000 7fa08cc00640 1 -- 172.21.15.40:0/688152286 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa08001e460 con 0x7fa0900ce630 2024-03-24T00:32:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.848+0000 7fa08cc00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa06c067660 0x7fa06c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:57.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.848+0000 7fa08cc00640 1 -- 172.21.15.40:0/688152286 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa080085f50 con 0x7fa0900ce630 2024-03-24T00:32:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.848+0000 7fa087e00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa06c067660 0x7fa06c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:57.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.848+0000 7fa087e00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa06c067660 0x7fa06c069b20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa0780ae000 tx=0x7fa0780a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:57.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:57.852+0000 7fa08cc00640 1 -- 172.21.15.40:0/688152286 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa080056bc0 con 0x7fa0900ce630 2024-03-24T00:32:58.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.016+0000 7fa096c00640 1 -- 172.21.15.40:0/688152286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa09006ed30 con 0x7fa0900ce630 2024-03-24T00:32:58.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.016+0000 7fa08cc00640 1 -- 172.21.15.40:0/688152286 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+16566 (secure 0 0 0) 0x7fa0800564c0 con 0x7fa0900ce630 2024-03-24T00:32:58.020 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:58.020 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":43,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:32:57.024187+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:55.446633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:56.017210+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:32:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.020+0000 7fa096c00640 1 -- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa06c067660 msgr2=0x7fa06c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.020+0000 7fa096c00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa06c067660 0x7fa06c069b20 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa0780ae000 tx=0x7fa0780a7000 comp rx=0 tx=0).stop 2024-03-24T00:32:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.020+0000 7fa096c00640 1 -- 172.21.15.40:0/688152286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0900ce630 msgr2=0x7fa0901556a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.020+0000 7fa096c00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0900ce630 0x7fa0901556a0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fa08000d830 tx=0x7fa08000dcf0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.020+0000 7fa096c00640 1 -- 172.21.15.40:0/688152286 shutdown_connections 2024-03-24T00:32:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.020+0000 7fa096c00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa06c067660 0x7fa06c069b20 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.020+0000 7fa096c00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0901419f0 0x7fa090138290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.020+0000 7fa096c00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa09013ec80 0x7fa090137d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.020+0000 7fa096c00640 1 --2- 172.21.15.40:0/688152286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0900ce630 0x7fa0901556a0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.020+0000 7fa096c00640 1 -- 172.21.15.40:0/688152286 >> 172.21.15.40:0/688152286 conn(0x7fa0900b17b0 msgr2=0x7fa0900b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.020+0000 7fa096c00640 1 -- 172.21.15.40:0/688152286 shutdown_connections 2024-03-24T00:32:58.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.020+0000 7fa096c00640 1 -- 172.21.15.40:0/688152286 wait complete. 2024-03-24T00:32:58.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:32:58.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.180+0000 7f81c5200640 1 Processor -- start 2024-03-24T00:32:58.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.180+0000 7f81c5200640 1 -- start start 2024-03-24T00:32:58.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.180+0000 7f81c5200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81c00b46c0 0x7f81c00b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:58.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.180+0000 7f81c5200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c0070f60 con 0x7f81c006e620 2024-03-24T00:32:58.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.180+0000 7f81c5200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c00b3620 con 0x7f81c00b2ff0 2024-03-24T00:32:58.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.180+0000 7f81c5200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c00b3400 con 0x7f81c00b46c0 2024-03-24T00:32:58.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.180+0000 7f81b7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81c00b46c0 0x7f81c00b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:58.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.180+0000 7f81bde00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f81c006e620 0x7f81c006ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38928/0 (socket says 172.21.15.40:38928) 2024-03-24T00:32:58.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.180+0000 7f81bde00640 1 -- 172.21.15.40:0/4100566550 learned_addr learned my addr 172.21.15.40:0/4100566550 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:58.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.180+0000 7f81b7e00640 1 -- 172.21.15.40:0/4100566550 >> v1:172.21.15.53:6789/0 conn(0x7f81c00b2ff0 legacy=0x7f81c00ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:58.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.180+0000 7f81b7e00640 1 -- 172.21.15.40:0/4100566550 >> v1:172.21.15.40:6789/0 conn(0x7f81c006e620 legacy=0x7f81c006ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.180+0000 7f81b7e00640 1 -- 172.21.15.40:0/4100566550 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c00b3900 con 0x7f81c00b46c0 2024-03-24T00:32:58.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.184+0000 7f81b7e00640 1 --2- 172.21.15.40:0/4100566550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81c00b46c0 0x7f81c00b2ab0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f81a80097c0 tx=0x7f81a801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6c7107013093f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:58.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.184+0000 7f81bd400640 1 -- 172.21.15.40:0/4100566550 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a80b1070 con 0x7f81c00b46c0 2024-03-24T00:32:58.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.184+0000 7f81bd400640 1 -- 172.21.15.40:0/4100566550 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a80ac600 con 0x7f81c00b46c0 2024-03-24T00:32:58.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.184+0000 7f81bd400640 1 -- 172.21.15.40:0/4100566550 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a80ac9d0 con 0x7f81c00b46c0 2024-03-24T00:32:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.184+0000 7f81c5200640 1 -- 172.21.15.40:0/4100566550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81c00b46c0 msgr2=0x7f81c00b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.184+0000 7f81c5200640 1 --2- 172.21.15.40:0/4100566550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81c00b46c0 0x7f81c00b2ab0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f81a80097c0 tx=0x7f81a801c3a0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.184+0000 7f81c5200640 1 -- 172.21.15.40:0/4100566550 shutdown_connections 2024-03-24T00:32:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.184+0000 7f81c5200640 1 --2- 172.21.15.40:0/4100566550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81c00b46c0 0x7f81c00b2ab0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.184+0000 7f81c5200640 1 -- 172.21.15.40:0/4100566550 >> 172.21.15.40:0/4100566550 conn(0x7f81c00b17b0 msgr2=0x7f81c00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.184+0000 7f81c5200640 1 -- 172.21.15.40:0/4100566550 shutdown_connections 2024-03-24T00:32:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.184+0000 7f81c5200640 1 -- 172.21.15.40:0/4100566550 wait complete. 2024-03-24T00:32:58.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81c5200640 1 Processor -- start 2024-03-24T00:32:58.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81c5200640 1 -- start start 2024-03-24T00:32:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81c5200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81c006e620 0x7f81c015a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81c5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81c00b2ff0 0x7f81c015e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81c5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81c00b46c0 0x7f81c015ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81c5200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c0144920 con 0x7f81c00b2ff0 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81c5200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c00b4b50 con 0x7f81c00b46c0 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81c5200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c013e820 con 0x7f81c006e620 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81b7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81c00b2ff0 0x7f81c015e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81b7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81c00b2ff0 0x7f81c015e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49550/0 (socket says 172.21.15.40:49550) 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81b7e00640 1 -- 172.21.15.40:0/1944951413 learned_addr learned my addr 172.21.15.40:0/1944951413 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81bde00640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81c006e620 0x7f81c015a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81be800640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81c00b46c0 0x7f81c015ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81b7e00640 1 -- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81c006e620 msgr2=0x7f81c015a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81b7e00640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81c006e620 0x7f81c015a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81b7e00640 1 -- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81c00b46c0 msgr2=0x7f81c015ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81b7e00640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81c00b46c0 0x7f81c015ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81b7e00640 1 -- 172.21.15.40:0/1944951413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c013d780 con 0x7f81c00b2ff0 2024-03-24T00:32:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81be800640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81c00b46c0 0x7f81c015ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:32:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81b7e00640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81c00b2ff0 0x7f81c015e900 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f81a8018fb0 tx=0x7f81a801abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81b6000640 1 -- 172.21.15.40:0/1944951413 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a80b1040 con 0x7f81c00b2ff0 2024-03-24T00:32:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81b6000640 1 -- 172.21.15.40:0/1944951413 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a80bc070 con 0x7f81c00b2ff0 2024-03-24T00:32:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81c5200640 1 -- 172.21.15.40:0/1944951413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c013da60 con 0x7f81c00b2ff0 2024-03-24T00:32:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81b6000640 1 -- 172.21.15.40:0/1944951413 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a80b78b0 con 0x7f81c00b2ff0 2024-03-24T00:32:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81c5200640 1 -- 172.21.15.40:0/1944951413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c013e020 con 0x7f81c00b2ff0 2024-03-24T00:32:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.188+0000 7f81c5200640 1 -- 172.21.15.40:0/1944951413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c0068170 con 0x7f81c00b2ff0 2024-03-24T00:32:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.192+0000 7f81b6000640 1 -- 172.21.15.40:0/1944951413 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f81a80ac4c0 con 0x7f81c00b2ff0 2024-03-24T00:32:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.192+0000 7f81b6000640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f819c067660 0x7f819c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.192+0000 7f81b6000640 1 -- 172.21.15.40:0/1944951413 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81a811fb60 con 0x7f81c00b2ff0 2024-03-24T00:32:58.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.192+0000 7f81bde00640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f819c067660 0x7f819c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:58.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.192+0000 7f81bde00640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f819c067660 0x7f819c069b20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f81b0000f70 tx=0x7f81b0007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:58.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.196+0000 7f81b6000640 1 -- 172.21.15.40:0/1944951413 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f81a8016050 con 0x7f81c00b2ff0 2024-03-24T00:32:58.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.360+0000 7f81c5200640 1 -- 172.21.15.40:0/1944951413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81c0072f50 con 0x7f81c00b2ff0 2024-03-24T00:32:58.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81b6000640 1 -- 172.21.15.40:0/1944951413 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f81a80f0870 con 0x7f81c00b2ff0 2024-03-24T00:32:58.364 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:32:58.365 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":43,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:32:57.024187+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:55.446633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:56.017210+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:32:58.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81c5200640 1 -- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f819c067660 msgr2=0x7f819c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:58.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81c5200640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f819c067660 0x7f819c069b20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f81b0000f70 tx=0x7f81b0007e90 comp rx=0 tx=0).stop 2024-03-24T00:32:58.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81c5200640 1 -- 172.21.15.40:0/1944951413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81c00b2ff0 msgr2=0x7f81c015e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:58.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81c5200640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81c00b2ff0 0x7f81c015e900 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f81a8018fb0 tx=0x7f81a801abe0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81c5200640 1 -- 172.21.15.40:0/1944951413 shutdown_connections 2024-03-24T00:32:58.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81c5200640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f819c067660 0x7f819c069b20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81c5200640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81c00b46c0 0x7f81c015ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81c5200640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81c00b2ff0 0x7f81c015e900 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81c5200640 1 --2- 172.21.15.40:0/1944951413 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81c006e620 0x7f81c015a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81c5200640 1 -- 172.21.15.40:0/1944951413 >> 172.21.15.40:0/1944951413 conn(0x7f81c00b17b0 msgr2=0x7f81c013cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:58.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81c5200640 1 -- 172.21.15.40:0/1944951413 shutdown_connections 2024-03-24T00:32:58.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.364+0000 7f81c5200640 1 -- 172.21.15.40:0/1944951413 wait complete. 2024-03-24T00:32:58.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:32:58.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.524+0000 7f53bb200640 1 Processor -- start 2024-03-24T00:32:58.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53bb200640 1 -- start start 2024-03-24T00:32:58.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53bb200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53b415bd00 0x7f53b40ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:58.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53bb200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f53b40728d0 0x7f53b40cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:58.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53bb200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b4072d60 con 0x7f53b415f8c0 2024-03-24T00:32:58.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53bb200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b40cf790 con 0x7f53b415bd00 2024-03-24T00:32:58.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53bb200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b4071de0 con 0x7f53b40728d0 2024-03-24T00:32:58.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53b3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53b415bd00 0x7f53b40ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:58.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53b3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53b415bd00 0x7f53b40ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49564/0 (socket says 172.21.15.40:49564) 2024-03-24T00:32:58.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53b3e00640 1 -- 172.21.15.40:0/3640976376 learned_addr learned my addr 172.21.15.40:0/3640976376 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:58.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53abe00640 1 --2- 172.21.15.40:0/3640976376 >> v2:172.21.15.53:3300/0 conn(0x7f53b40728d0 0x7f53b40cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:58.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53b3e00640 1 -- 172.21.15.40:0/3640976376 >> v2:172.21.15.53:3300/0 conn(0x7f53b40728d0 msgr2=0x7f53b40cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:58.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53b3e00640 1 --2- 172.21.15.40:0/3640976376 >> v2:172.21.15.53:3300/0 conn(0x7f53b40728d0 0x7f53b40cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53b3e00640 1 -- 172.21.15.40:0/3640976376 >> v1:172.21.15.53:6789/0 conn(0x7f53b415f8c0 legacy=0x7f53b415fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:58.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53b3e00640 1 -- 172.21.15.40:0/3640976376 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b40cea50 con 0x7f53b415bd00 2024-03-24T00:32:58.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.528+0000 7f53abe00640 1 --2- 172.21.15.40:0/3640976376 >> v2:172.21.15.53:3300/0 conn(0x7f53b40728d0 0x7f53b40cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53b3e00640 1 --2- 172.21.15.40:0/3640976376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53b415bd00 0x7f53b40ae8e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f53b406ff40 tx=0x7f53a401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48ab95fccf8ce32 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53b3400640 1 -- 172.21.15.40:0/3640976376 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a40aa610 con 0x7f53b415bd00 2024-03-24T00:32:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53b3400640 1 -- 172.21.15.40:0/3640976376 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a40af070 con 0x7f53b415bd00 2024-03-24T00:32:58.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53b3400640 1 -- 172.21.15.40:0/3640976376 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a40aa9d0 con 0x7f53b415bd00 2024-03-24T00:32:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53bb200640 1 -- 172.21.15.40:0/3640976376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53b415bd00 msgr2=0x7f53b40ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53bb200640 1 --2- 172.21.15.40:0/3640976376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53b415bd00 0x7f53b40ae8e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f53b406ff40 tx=0x7f53a401c4b0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53bb200640 1 -- 172.21.15.40:0/3640976376 shutdown_connections 2024-03-24T00:32:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53bb200640 1 --2- 172.21.15.40:0/3640976376 >> v2:172.21.15.53:3300/0 conn(0x7f53b40728d0 0x7f53b40cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53bb200640 1 --2- 172.21.15.40:0/3640976376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53b415bd00 0x7f53b40ae8e0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53bb200640 1 -- 172.21.15.40:0/3640976376 >> 172.21.15.40:0/3640976376 conn(0x7f53b40b1660 msgr2=0x7f53b40b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:58.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53bb200640 1 -- 172.21.15.40:0/3640976376 shutdown_connections 2024-03-24T00:32:58.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53bb200640 1 -- 172.21.15.40:0/3640976376 wait complete. 2024-03-24T00:32:58.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53bb200640 1 Processor -- start 2024-03-24T00:32:58.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.532+0000 7f53bb200640 1 -- start start 2024-03-24T00:32:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53bb200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53b40728d0 0x7f53b415d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53bb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53b415bd00 0x7f53b4156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53bb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53b415f8c0 0x7f53b4149640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53bb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b406d320 con 0x7f53b415f8c0 2024-03-24T00:32:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53bb200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b415c0e0 con 0x7f53b415bd00 2024-03-24T00:32:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53bb200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b406d4a0 con 0x7f53b40728d0 2024-03-24T00:32:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53b415f8c0 0x7f53b4149640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53b415f8c0 0x7f53b4149640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49574/0 (socket says 172.21.15.40:49574) 2024-03-24T00:32:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b9200640 1 -- 172.21.15.40:0/4147402233 learned_addr learned my addr 172.21.15.40:0/4147402233 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53abe00640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53b415bd00 0x7f53b4156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:58.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b3e00640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53b40728d0 0x7f53b415d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b9200640 1 -- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53b40728d0 msgr2=0x7f53b415d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b9200640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53b40728d0 0x7f53b415d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b9200640 1 -- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53b415bd00 msgr2=0x7f53b4156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b9200640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53b415bd00 0x7f53b4156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b9200640 1 -- 172.21.15.40:0/4147402233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b415c300 con 0x7f53b415f8c0 2024-03-24T00:32:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b9200640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53b415f8c0 0x7f53b4149640 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00b9a0 tx=0x7f53ac00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b0c00640 1 -- 172.21.15.40:0/4147402233 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac019070 con 0x7f53b415f8c0 2024-03-24T00:32:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b0c00640 1 -- 172.21.15.40:0/4147402233 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53ac00cba0 con 0x7f53b415f8c0 2024-03-24T00:32:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53bb200640 1 -- 172.21.15.40:0/4147402233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b415d8a0 con 0x7f53b415f8c0 2024-03-24T00:32:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b0c00640 1 -- 172.21.15.40:0/4147402233 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ac014440 con 0x7f53b415f8c0 2024-03-24T00:32:58.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53bb200640 1 -- 172.21.15.40:0/4147402233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b415c880 con 0x7f53b415f8c0 2024-03-24T00:32:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53bb200640 1 -- 172.21.15.40:0/4147402233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b4068170 con 0x7f53b415f8c0 2024-03-24T00:32:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.536+0000 7f53b0c00640 1 -- 172.21.15.40:0/4147402233 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f53ac01c070 con 0x7f53b415f8c0 2024-03-24T00:32:58.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.540+0000 7f53b0c00640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5390067660 0x7f5390069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.540+0000 7f53b0c00640 1 -- 172.21.15.40:0/4147402233 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53ac0854a0 con 0x7f53b415f8c0 2024-03-24T00:32:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.540+0000 7f53b3e00640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5390067660 0x7f5390069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:58.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.540+0000 7f53b3e00640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5390067660 0x7f5390069b20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f53a40ae000 tx=0x7f53a40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.544+0000 7f53b0c00640 1 -- 172.21.15.40:0/4147402233 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f53ac055a30 con 0x7f53b415f8c0 2024-03-24T00:32:58.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:58.744+0000 7f53bb200640 1 -- 172.21.15.40:0/4147402233 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f53b4152d20 con 0x7f53b415f8c0 2024-03-24T00:32:59.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.640+0000 7f53b0c00640 1 -- 172.21.15.40:0/4147402233 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v29) v1 ==== 118+0+0 (secure 0 0 0) 0x7f53ac059ad0 con 0x7f53b415f8c0 2024-03-24T00:32:59.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.644+0000 7f53bb200640 1 -- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5390067660 msgr2=0x7f5390069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:59.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.644+0000 7f53bb200640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5390067660 0x7f5390069b20 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f53a40ae000 tx=0x7f53a40a7000 comp rx=0 tx=0).stop 2024-03-24T00:32:59.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.644+0000 7f53bb200640 1 -- 172.21.15.40:0/4147402233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53b415f8c0 msgr2=0x7f53b4149640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:59.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.644+0000 7f53bb200640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53b415f8c0 0x7f53b4149640 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00b9a0 tx=0x7f53ac00be60 comp rx=0 tx=0).stop 2024-03-24T00:32:59.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.644+0000 7f53bb200640 1 -- 172.21.15.40:0/4147402233 shutdown_connections 2024-03-24T00:32:59.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.644+0000 7f53bb200640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5390067660 0x7f5390069b20 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:59.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.644+0000 7f53bb200640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53b415f8c0 0x7f53b4149640 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:59.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.644+0000 7f53bb200640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53b415bd00 0x7f53b4156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:59.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.644+0000 7f53bb200640 1 --2- 172.21.15.40:0/4147402233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53b40728d0 0x7f53b415d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:59.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.644+0000 7f53bb200640 1 -- 172.21.15.40:0/4147402233 >> 172.21.15.40:0/4147402233 conn(0x7f53b40b1660 msgr2=0x7f53b40d0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:59.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.644+0000 7f53bb200640 1 -- 172.21.15.40:0/4147402233 shutdown_connections 2024-03-24T00:32:59.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.644+0000 7f53bb200640 1 -- 172.21.15.40:0/4147402233 wait complete. 2024-03-24T00:32:59.669 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:32:59.670 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:32:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f6327a00640 1 Processor -- start 2024-03-24T00:32:59.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f6327a00640 1 -- start start 2024-03-24T00:32:59.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f6327a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6320071b00 0x7f6320071f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:59.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f6327a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63200b3260 con 0x7f632006cb80 2024-03-24T00:32:59.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f6327a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63200b3540 con 0x7f632015ac60 2024-03-24T00:32:59.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f6327a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6320072490 con 0x7f6320071b00 2024-03-24T00:32:59.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f632006cb80 0x7f63200b4300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38940/0 (socket says 172.21.15.40:38940) 2024-03-24T00:32:59.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631fe00640 1 -- 172.21.15.40:0/522690905 learned_addr learned my addr 172.21.15.40:0/522690905 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:32:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f6325a00640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6320071b00 0x7f6320071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068562737 0 0) 0x7f63200b3260 con 0x7f632006cb80 2024-03-24T00:32:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6308003530 con 0x7f632006cb80 2024-03-24T00:32:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850051835 0 0) 0x7f63200b3540 con 0x7f632015ac60 2024-03-24T00:32:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63200b3260 con 0x7f632015ac60 2024-03-24T00:32:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3244869165 0 0) 0x7f6308003530 con 0x7f632006cb80 2024-03-24T00:32:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6320071b00 msgr2=0x7f6320071f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:59.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6320071b00 0x7f6320071f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 >> v1:172.21.15.53:6789/0 conn(0x7f632015ac60 legacy=0x7f63200b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63200b3820 con 0x7f632006cb80 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f630c001680 con 0x7f632006cb80 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 >> v1:172.21.15.40:6789/0 conn(0x7f632006cb80 legacy=0x7f63200b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6320071b00 0x7f6308003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6308004130 0x7f6308004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6308004d50 0x7f63080051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6320072490 con 0x7f6308004130 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630800bb90 con 0x7f6308004d50 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630800bd90 con 0x7f6320071b00 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f6325000640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6320071b00 0x7f6308003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.812+0000 7f631fe00640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6308004130 0x7f6308004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.816+0000 7f6325a00640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6308004d50 0x7f63080051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:59.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.816+0000 7f631fe00640 1 -- 172.21.15.40:0/522690905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6320071b00 msgr2=0x7f6308003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.816+0000 7f631fe00640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6320071b00 0x7f6308003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.816+0000 7f631fe00640 1 -- 172.21.15.40:0/522690905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6308004d50 msgr2=0x7f63080051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:32:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.816+0000 7f631fe00640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6308004d50 0x7f63080051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.816+0000 7f631fe00640 1 -- 172.21.15.40:0/522690905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6308003530 con 0x7f6308004130 2024-03-24T00:32:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.816+0000 7f6325000640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6320071b00 0x7f6308003b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:32:59.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.816+0000 7f631fe00640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6308004130 0x7f6308004810 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f630c007fa0 tx=0x7f630c01cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d44275455fb798 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:59.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.816+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c007bf0 con 0x7f6308004130 2024-03-24T00:32:59.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.816+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630c0a9070 con 0x7f6308004130 2024-03-24T00:32:59.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.816+0000 7f631f400640 1 -- 172.21.15.40:0/522690905 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c005d80 con 0x7f6308004130 2024-03-24T00:32:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 -- 172.21.15.40:0/522690905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6308004130 msgr2=0x7f6308004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 --2- 172.21.15.40:0/522690905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6308004130 0x7f6308004810 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f630c007fa0 tx=0x7f630c01cb00 comp rx=0 tx=0).stop 2024-03-24T00:32:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6325a00640 1 -- 172.21.15.40:0/522690905 reap_dead start 2024-03-24T00:32:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 -- 172.21.15.40:0/522690905 shutdown_connections 2024-03-24T00:32:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 -- 172.21.15.40:0/522690905 >> 172.21.15.40:0/522690905 conn(0x7f63200b1660 msgr2=0x7f63200b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:32:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 -- 172.21.15.40:0/522690905 shutdown_connections 2024-03-24T00:32:59.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 -- 172.21.15.40:0/522690905 wait complete. 2024-03-24T00:32:59.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 Processor -- start 2024-03-24T00:32:59.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 -- start start 2024-03-24T00:32:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6308004130 0x7f632014c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f632014f1a0 0x7f632014f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:59.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f632015d4d0 0x7f632015d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63200cd560 con 0x7f6308004130 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63200b3e70 con 0x7f632014f1a0 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6327a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63200cd6e0 con 0x7f632015d4d0 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6325000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6308004130 0x7f632014c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6325000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6308004130 0x7f632014c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49592/0 (socket says 172.21.15.40:49592) 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.820+0000 7f6325000640 1 -- 172.21.15.40:0/3612951083 learned_addr learned my addr 172.21.15.40:0/3612951083 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f6325a00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f632015d4d0 0x7f632015d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f6325000640 1 -- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f632015d4d0 msgr2=0x7f632015d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f631fe00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f632014f1a0 0x7f632014f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f6325000640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f632015d4d0 0x7f632015d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f6325000640 1 -- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f632014f1a0 msgr2=0x7f632014f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f6325000640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f632014f1a0 0x7f632014f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:32:59.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f6325000640 1 -- 172.21.15.40:0/3612951083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632014d8a0 con 0x7f6308004130 2024-03-24T00:32:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f6325a00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f632015d4d0 0x7f632015d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f631fe00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f632014f1a0 0x7f632014f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:32:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f6325000640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6308004130 0x7f632014c0e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f631400fae0 tx=0x7f6314001d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f631cc00640 1 -- 172.21.15.40:0/3612951083 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63140052e0 con 0x7f6308004130 2024-03-24T00:32:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f631cc00640 1 -- 172.21.15.40:0/3612951083 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6314018070 con 0x7f6308004130 2024-03-24T00:32:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f631cc00640 1 -- 172.21.15.40:0/3612951083 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631400d9b0 con 0x7f6308004130 2024-03-24T00:32:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f6327a00640 1 -- 172.21.15.40:0/3612951083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632014db20 con 0x7f6308004130 2024-03-24T00:32:59.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f6327a00640 1 -- 172.21.15.40:0/3612951083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632014e0e0 con 0x7f6308004130 2024-03-24T00:32:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f6327a00640 1 -- 172.21.15.40:0/3612951083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6320068170 con 0x7f6308004130 2024-03-24T00:32:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f631cc00640 1 -- 172.21.15.40:0/3612951083 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f631400db50 con 0x7f6308004130 2024-03-24T00:32:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f631cc00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6300067690 0x7f6300069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:32:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f631cc00640 1 -- 172.21.15.40:0/3612951083 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6314085a50 con 0x7f6308004130 2024-03-24T00:32:59.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.824+0000 7f631fe00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6300067690 0x7f6300069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:32:59.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.828+0000 7f631fe00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6300067690 0x7f6300069b50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f630c0ab000 tx=0x7f630c0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:32:59.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:32:59.836+0000 7f631cc00640 1 -- 172.21.15.40:0/3612951083 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f632014d8a0 con 0x7f6308004130 2024-03-24T00:33:00.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.032+0000 7f6327a00640 1 -- 172.21.15.40:0/3612951083 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f632015a2d0 con 0x7f6308004130 2024-03-24T00:33:00.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.648+0000 7f631cc00640 1 -- 172.21.15.40:0/3612951083 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f631401a310 con 0x7f6308004130 2024-03-24T00:33:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.652+0000 7f6327a00640 1 -- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6300067690 msgr2=0x7f6300069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.652+0000 7f6327a00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6300067690 0x7f6300069b50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f630c0ab000 tx=0x7f630c0a7070 comp rx=0 tx=0).stop 2024-03-24T00:33:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.652+0000 7f6327a00640 1 -- 172.21.15.40:0/3612951083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6308004130 msgr2=0x7f632014c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.652+0000 7f6327a00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6308004130 0x7f632014c0e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f631400fae0 tx=0x7f6314001d60 comp rx=0 tx=0).stop 2024-03-24T00:33:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.652+0000 7f6327a00640 1 -- 172.21.15.40:0/3612951083 shutdown_connections 2024-03-24T00:33:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.652+0000 7f6327a00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6300067690 0x7f6300069b50 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:00.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.652+0000 7f6327a00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f632015d4d0 0x7f632015d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.652+0000 7f6327a00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f632014f1a0 0x7f632014f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.652+0000 7f6327a00640 1 --2- 172.21.15.40:0/3612951083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6308004130 0x7f632014c0e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.652+0000 7f6327a00640 1 -- 172.21.15.40:0/3612951083 >> 172.21.15.40:0/3612951083 conn(0x7f63200b1660 msgr2=0x7f63200cdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.652+0000 7f6327a00640 1 -- 172.21.15.40:0/3612951083 shutdown_connections 2024-03-24T00:33:00.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.652+0000 7f6327a00640 1 -- 172.21.15.40:0/3612951083 wait complete. 2024-03-24T00:33:00.672 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:33:00.672 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:00.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfd5000640 1 Processor -- start 2024-03-24T00:33:00.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfd5000640 1 -- start start 2024-03-24T00:33:00.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfd5000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdfd0160030 0x7fdfd0070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:00.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfd5000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd00cec10 con 0x7fdfd0145520 2024-03-24T00:33:00.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfd5000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfd00ceeb0 con 0x7fdfd006de00 2024-03-24T00:33:00.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfd5000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfd00cf190 con 0x7fdfd0160030 2024-03-24T00:33:00.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcf400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdfd0145520 0x7fdfd015f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38946/0 (socket says 172.21.15.40:38946) 2024-03-24T00:33:00.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcf400640 1 -- 172.21.15.40:0/3888974594 learned_addr learned my addr 172.21.15.40:0/3888974594 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:00.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284481278 0 0) 0x7fdfd00cec10 con 0x7fdfd0145520 2024-03-24T00:33:00.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb4003530 con 0x7fdfd0145520 2024-03-24T00:33:00.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcea00640 1 --2- 172.21.15.40:0/3888974594 >> v2:172.21.15.53:3300/0 conn(0x7fdfd0160030 0x7fdfd0070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1788189492 0 0) 0x7fdfb4003530 con 0x7fdfd0145520 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 >> v2:172.21.15.53:3300/0 conn(0x7fdfd0160030 msgr2=0x7fdfd0070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 --2- 172.21.15.40:0/3888974594 >> v2:172.21.15.53:3300/0 conn(0x7fdfd0160030 0x7fdfd0070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 >> v1:172.21.15.53:6789/0 conn(0x7fdfd006de00 legacy=0x7fdfd0071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd00cf470 con 0x7fdfd0145520 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fdfc0001680 con 0x7fdfd0145520 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 >> v1:172.21.15.40:6789/0 conn(0x7fdfd0145520 legacy=0x7fdfd015f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 --2- 172.21.15.40:0/3888974594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfb4003a10 0x7fdfb4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 --2- 172.21.15.40:0/3888974594 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfb4004810 0x7fdfb4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 --2- 172.21.15.40:0/3888974594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb4005220 0x7fdfb400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd00cf190 con 0x7fdfb4003a10 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb400c170 con 0x7fdfb4005220 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb400c370 con 0x7fdfb4004810 2024-03-24T00:33:00.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcf400640 1 --2- 172.21.15.40:0/3888974594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfb4003a10 0x7fdfb4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:00.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcea00640 1 --2- 172.21.15.40:0/3888974594 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfb4004810 0x7fdfb4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:00.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcfe00640 1 --2- 172.21.15.40:0/3888974594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb4005220 0x7fdfb400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:00.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcf400640 1 -- 172.21.15.40:0/3888974594 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfb4004810 msgr2=0x7fdfb4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:00.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcf400640 1 --2- 172.21.15.40:0/3888974594 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfb4004810 0x7fdfb4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:00.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcf400640 1 -- 172.21.15.40:0/3888974594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb4005220 msgr2=0x7fdfb400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:00.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcf400640 1 --2- 172.21.15.40:0/3888974594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb4005220 0x7fdfb400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:00.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcf400640 1 -- 172.21.15.40:0/3888974594 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfb4003530 con 0x7fdfb4003a10 2024-03-24T00:33:00.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcfe00640 1 --2- 172.21.15.40:0/3888974594 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfb4005220 0x7fdfb400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:33:00.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.812+0000 7fdfcfe00640 1 -- 172.21.15.40:0/3888974594 reap_dead start 2024-03-24T00:33:00.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.816+0000 7fdfcf400640 1 --2- 172.21.15.40:0/3888974594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfb4003a10 0x7fdfb4003e30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000c0e0 tx=0x7fdfc001cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80c67d0acb994d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:00.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.816+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0007c50 con 0x7fdfb4003a10 2024-03-24T00:33:00.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.816+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc00a9020 con 0x7fdfb4003a10 2024-03-24T00:33:00.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.816+0000 7fdfce000640 1 -- 172.21.15.40:0/3888974594 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0005ae0 con 0x7fdfb4003a10 2024-03-24T00:33:00.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.816+0000 7fdfd5000640 1 -- 172.21.15.40:0/3888974594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfb4003a10 msgr2=0x7fdfb4003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:00.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.816+0000 7fdfd5000640 1 --2- 172.21.15.40:0/3888974594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfb4003a10 0x7fdfb4003e30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fdfc000c0e0 tx=0x7fdfc001cb60 comp rx=0 tx=0).stop 2024-03-24T00:33:00.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.816+0000 7fdfd5000640 1 -- 172.21.15.40:0/3888974594 shutdown_connections 2024-03-24T00:33:00.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.816+0000 7fdfd5000640 1 --2- 172.21.15.40:0/3888974594 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfb4003a10 0x7fdfb4003e30 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:00.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.816+0000 7fdfd5000640 1 -- 172.21.15.40:0/3888974594 >> 172.21.15.40:0/3888974594 conn(0x7fdfd00b1740 msgr2=0x7fdfd00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:00.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.816+0000 7fdfd5000640 1 -- 172.21.15.40:0/3888974594 shutdown_connections 2024-03-24T00:33:00.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.816+0000 7fdfd5000640 1 -- 172.21.15.40:0/3888974594 wait complete. 2024-03-24T00:33:00.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfd5000640 1 Processor -- start 2024-03-24T00:33:00.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfd5000640 1 -- start start 2024-03-24T00:33:00.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfd5000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfb4003a10 0x7fdfd015edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:00.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfd5000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd0159e30 0x7fdfd015a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:00.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfd5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd01580c0 0x7fdfd0158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:00.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfd5000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd01605a0 con 0x7fdfd01580c0 2024-03-24T00:33:00.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfd5000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd0145b10 con 0x7fdfd0159e30 2024-03-24T00:33:00.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfd5000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd0160720 con 0x7fdfb4003a10 2024-03-24T00:33:00.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd01580c0 0x7fdfd0158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:00.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd01580c0 0x7fdfd0158540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49614/0 (socket says 172.21.15.40:49614) 2024-03-24T00:33:00.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcfe00640 1 -- 172.21.15.40:0/1661269094 learned_addr learned my addr 172.21.15.40:0/1661269094 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcf400640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfb4003a10 0x7fdfd015edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcea00640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd0159e30 0x7fdfd015a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcfe00640 1 -- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfb4003a10 msgr2=0x7fdfd015edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcfe00640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfb4003a10 0x7fdfd015edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcfe00640 1 -- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd0159e30 msgr2=0x7fdfd015a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcfe00640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd0159e30 0x7fdfd015a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcfe00640 1 -- 172.21.15.40:0/1661269094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd015f2f0 con 0x7fdfd01580c0 2024-03-24T00:33:00.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcf400640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfb4003a10 0x7fdfd015edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:33:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcea00640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd0159e30 0x7fdfd015a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:33:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfcfe00640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd01580c0 0x7fdfd0158540 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdfd006e290 tx=0x7fdfc800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfc7400640 1 -- 172.21.15.40:0/1661269094 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8018070 con 0x7fdfd01580c0 2024-03-24T00:33:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfc7400640 1 -- 172.21.15.40:0/1661269094 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfc8003db0 con 0x7fdfd01580c0 2024-03-24T00:33:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfd5000640 1 -- 172.21.15.40:0/1661269094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd0159670 con 0x7fdfd01580c0 2024-03-24T00:33:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfc7400640 1 -- 172.21.15.40:0/1661269094 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc80052b0 con 0x7fdfd01580c0 2024-03-24T00:33:00.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.820+0000 7fdfd5000640 1 -- 172.21.15.40:0/1661269094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd015cf60 con 0x7fdfd01580c0 2024-03-24T00:33:00.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.824+0000 7fdfd5000640 1 -- 172.21.15.40:0/1661269094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd0073540 con 0x7fdfd01580c0 2024-03-24T00:33:00.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.824+0000 7fdfc7400640 1 -- 172.21.15.40:0/1661269094 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fdfc801b070 con 0x7fdfd01580c0 2024-03-24T00:33:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.824+0000 7fdfc7400640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdfb0067660 0x7fdfb0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.824+0000 7fdfc7400640 1 -- 172.21.15.40:0/1661269094 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdfc8084f90 con 0x7fdfd01580c0 2024-03-24T00:33:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.824+0000 7fdfcf400640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdfb0067660 0x7fdfb0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.824+0000 7fdfcf400640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdfb0067660 0x7fdfb0069b20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00aa000 tx=0x7fdfd00cf640 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:00.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:00.828+0000 7fdfc7400640 1 -- 172.21.15.40:0/1661269094 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdfc8055c00 con 0x7fdfd01580c0 2024-03-24T00:33:01.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.024+0000 7fdfd5000640 1 -- 172.21.15.40:0/1661269094 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfd004f4f0 con 0x7fdfd01580c0 2024-03-24T00:33:01.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.024+0000 7fdfc7400640 1 -- 172.21.15.40:0/1661269094 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fdfc8055500 con 0x7fdfd01580c0 2024-03-24T00:33:01.027 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:01.027 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5192,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5198,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/1600357100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1600357100},{"type":"v1","addr":"172.21.15.53:6836","nonce":1600357100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:00.034888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5763]},"id":3}]} 2024-03-24T00:33:01.027 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 30 2024-03-24T00:33:01.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.028+0000 7fdfd5000640 1 -- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdfb0067660 msgr2=0x7fdfb0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.028+0000 7fdfd5000640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdfb0067660 0x7fdfb0069b20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00aa000 tx=0x7fdfd00cf640 comp rx=0 tx=0).stop 2024-03-24T00:33:01.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.028+0000 7fdfd5000640 1 -- 172.21.15.40:0/1661269094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd01580c0 msgr2=0x7fdfd0158540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.028+0000 7fdfd5000640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd01580c0 0x7fdfd0158540 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fdfd006e290 tx=0x7fdfc800d9d0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.028+0000 7fdfd5000640 1 -- 172.21.15.40:0/1661269094 shutdown_connections 2024-03-24T00:33:01.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.028+0000 7fdfd5000640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdfb0067660 0x7fdfb0069b20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.028+0000 7fdfd5000640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdfd01580c0 0x7fdfd0158540 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.028+0000 7fdfd5000640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdfd0159e30 0x7fdfd015a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.028+0000 7fdfd5000640 1 --2- 172.21.15.40:0/1661269094 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdfb4003a10 0x7fdfd015edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.028+0000 7fdfd5000640 1 -- 172.21.15.40:0/1661269094 >> 172.21.15.40:0/1661269094 conn(0x7fdfd00b1740 msgr2=0x7fdfd00715e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:01.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.028+0000 7fdfd5000640 1 -- 172.21.15.40:0/1661269094 shutdown_connections 2024-03-24T00:33:01.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.028+0000 7fdfd5000640 1 -- 172.21.15.40:0/1661269094 wait complete. 2024-03-24T00:33:01.059 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:33:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.196+0000 7facba000640 1 Processor -- start 2024-03-24T00:33:01.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.196+0000 7facba000640 1 -- start start 2024-03-24T00:33:01.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.196+0000 7facba000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facb40b2aa0 0x7facb40d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.196+0000 7facba000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7facb40d0a80 0x7facb40cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.196+0000 7facba000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb40b2870 con 0x7facb40b2490 2024-03-24T00:33:01.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.196+0000 7facba000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb406b6d0 con 0x7facb40d0a80 2024-03-24T00:33:01.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.196+0000 7facba000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb40b3190 con 0x7facb40b2aa0 2024-03-24T00:33:01.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.196+0000 7facb3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7facb40d0a80 0x7facb40cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.200+0000 7facb3400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7facb40b2490 0x7facb40cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38950/0 (socket says 172.21.15.40:38950) 2024-03-24T00:33:01.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.200+0000 7facb3400640 1 -- 172.21.15.40:0/3471949800 learned_addr learned my addr 172.21.15.40:0/3471949800 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:01.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.200+0000 7facabe00640 1 --2- 172.21.15.40:0/3471949800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facb40b2aa0 0x7facb40d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.200+0000 7facb3e00640 1 -- 172.21.15.40:0/3471949800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facb40b2aa0 msgr2=0x7facb40d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:01.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.200+0000 7facb3e00640 1 --2- 172.21.15.40:0/3471949800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facb40b2aa0 0x7facb40d0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.200+0000 7facb3e00640 1 -- 172.21.15.40:0/3471949800 >> v1:172.21.15.40:6789/0 conn(0x7facb40b2490 legacy=0x7facb40cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.200+0000 7facb2a00640 1 -- 172.21.15.40:0/3471949800 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602844884 0 0) 0x7facb40b2870 con 0x7facb40b2490 2024-03-24T00:33:01.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.200+0000 7facb3e00640 1 -- 172.21.15.40:0/3471949800 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb40b3470 con 0x7facb40d0a80 2024-03-24T00:33:01.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.204+0000 7facb3e00640 1 --2- 172.21.15.40:0/3471949800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facb40d0a80 0x7facb40cc460 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7facac009d90 tx=0x7facac01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885714e9f8e22109 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:01.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.204+0000 7facb2a00640 1 -- 172.21.15.40:0/3471949800 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac0b2070 con 0x7facb40d0a80 2024-03-24T00:33:01.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.204+0000 7facb2a00640 1 -- 172.21.15.40:0/3471949800 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac0a5940 con 0x7facb40d0a80 2024-03-24T00:33:01.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.204+0000 7facb2a00640 1 -- 172.21.15.40:0/3471949800 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac0a5d10 con 0x7facb40d0a80 2024-03-24T00:33:01.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.204+0000 7facba000640 1 -- 172.21.15.40:0/3471949800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facb40d0a80 msgr2=0x7facb40cc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.204+0000 7facba000640 1 --2- 172.21.15.40:0/3471949800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facb40d0a80 0x7facb40cc460 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7facac009d90 tx=0x7facac01c970 comp rx=0 tx=0).stop 2024-03-24T00:33:01.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.204+0000 7facba000640 1 -- 172.21.15.40:0/3471949800 shutdown_connections 2024-03-24T00:33:01.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.204+0000 7facba000640 1 --2- 172.21.15.40:0/3471949800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facb40d0a80 0x7facb40cc460 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.204+0000 7facba000640 1 --2- 172.21.15.40:0/3471949800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facb40b2aa0 0x7facb40d0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.204+0000 7facba000640 1 -- 172.21.15.40:0/3471949800 >> 172.21.15.40:0/3471949800 conn(0x7facb40d1d10 msgr2=0x7facb40d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:01.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.204+0000 7facba000640 1 -- 172.21.15.40:0/3471949800 shutdown_connections 2024-03-24T00:33:01.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.204+0000 7facba000640 1 -- 172.21.15.40:0/3471949800 wait complete. 2024-03-24T00:33:01.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facba000640 1 Processor -- start 2024-03-24T00:33:01.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facba000640 1 -- start start 2024-03-24T00:33:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facba000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facb40b2490 0x7facb409efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facba000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facb40b2aa0 0x7facb40b6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facba000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facb40d0a80 0x7facb40b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facba000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb40cd540 con 0x7facb40d0a80 2024-03-24T00:33:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facba000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb40cd3c0 con 0x7facb40b2aa0 2024-03-24T00:33:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facba000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb40cd6c0 con 0x7facb40b2490 2024-03-24T00:33:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facb3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facb40d0a80 0x7facb40b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facb3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facb40d0a80 0x7facb40b6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49636/0 (socket says 172.21.15.40:49636) 2024-03-24T00:33:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facb3e00640 1 -- 172.21.15.40:0/3737371546 learned_addr learned my addr 172.21.15.40:0/3737371546 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:01.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facb3400640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facb40b2490 0x7facb409efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facabe00640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facb40b2aa0 0x7facb40b6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facb3e00640 1 -- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facb40b2490 msgr2=0x7facb409efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facb3e00640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facb40b2490 0x7facb409efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facb3e00640 1 -- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facb40b2aa0 msgr2=0x7facb40b6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facb3e00640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facb40b2aa0 0x7facb40b6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facb3e00640 1 -- 172.21.15.40:0/3737371546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb40b71b0 con 0x7facb40d0a80 2024-03-24T00:33:01.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facabe00640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facb40b2aa0 0x7facb40b6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:01.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facb3e00640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facb40d0a80 0x7facb40b6bc0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7facac009d90 tx=0x7facac019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:01.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facab400640 1 -- 172.21.15.40:0/3737371546 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac0b2040 con 0x7facb40d0a80 2024-03-24T00:33:01.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facab400640 1 -- 172.21.15.40:0/3737371546 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac0a5740 con 0x7facb40d0a80 2024-03-24T00:33:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facab400640 1 -- 172.21.15.40:0/3737371546 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac0adab0 con 0x7facb40d0a80 2024-03-24T00:33:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facba000640 1 -- 172.21.15.40:0/3737371546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb40b4990 con 0x7facb40d0a80 2024-03-24T00:33:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facba000640 1 -- 172.21.15.40:0/3737371546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb40b4f50 con 0x7facb40d0a80 2024-03-24T00:33:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.208+0000 7facba000640 1 -- 172.21.15.40:0/3737371546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb4068170 con 0x7facb40d0a80 2024-03-24T00:33:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.212+0000 7facab400640 1 -- 172.21.15.40:0/3737371546 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7facac0c1020 con 0x7facb40d0a80 2024-03-24T00:33:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.212+0000 7facab400640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fac90067660 0x7fac90069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.212+0000 7facab400640 1 -- 172.21.15.40:0/3737371546 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7facac120360 con 0x7facb40d0a80 2024-03-24T00:33:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.212+0000 7facb3400640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fac90067660 0x7fac90069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.212+0000 7facb3400640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fac90067660 0x7fac90069b20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7faca4000f70 tx=0x7faca4007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.216+0000 7facab400640 1 -- 172.21.15.40:0/3737371546 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7facac0c8020 con 0x7facb40d0a80 2024-03-24T00:33:01.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.380+0000 7facba000640 1 -- 172.21.15.40:0/3737371546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facb409f510 con 0x7facb40d0a80 2024-03-24T00:33:01.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.380+0000 7facab400640 1 -- 172.21.15.40:0/3737371546 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+16553 (secure 0 0 0) 0x7facac016560 con 0x7facb40d0a80 2024-03-24T00:33:01.384 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:01.384 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":44,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:32:58.633867+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:55.446633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:56.017210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:33:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.384+0000 7facba000640 1 -- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fac90067660 msgr2=0x7fac90069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.384+0000 7facba000640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fac90067660 0x7fac90069b20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7faca4000f70 tx=0x7faca4007df0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.384+0000 7facba000640 1 -- 172.21.15.40:0/3737371546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facb40d0a80 msgr2=0x7facb40b6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.384+0000 7facba000640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facb40d0a80 0x7facb40b6bc0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7facac009d90 tx=0x7facac019710 comp rx=0 tx=0).stop 2024-03-24T00:33:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.384+0000 7facba000640 1 -- 172.21.15.40:0/3737371546 shutdown_connections 2024-03-24T00:33:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.384+0000 7facba000640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fac90067660 0x7fac90069b20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.384+0000 7facba000640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facb40d0a80 0x7facb40b6bc0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.384+0000 7facba000640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facb40b2aa0 0x7facb40b6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.384+0000 7facba000640 1 --2- 172.21.15.40:0/3737371546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facb40b2490 0x7facb409efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.384+0000 7facba000640 1 -- 172.21.15.40:0/3737371546 >> 172.21.15.40:0/3737371546 conn(0x7facb40d1d10 msgr2=0x7facb40afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.384+0000 7facba000640 1 -- 172.21.15.40:0/3737371546 shutdown_connections 2024-03-24T00:33:01.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.384+0000 7facba000640 1 -- 172.21.15.40:0/3737371546 wait complete. 2024-03-24T00:33:01.412 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:33:01.412 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:33:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe31600640 1 Processor -- start 2024-03-24T00:33:01.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe31600640 1 -- start start 2024-03-24T00:33:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe31600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe2c13ec80 0x7efe2c1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe31600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efe2c0ce800 0x7efe2c071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe31600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2c070e00 con 0x7efe2c1419f0 2024-03-24T00:33:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe31600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2c141e50 con 0x7efe2c13ec80 2024-03-24T00:33:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe31600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2c0cee20 con 0x7efe2c0ce800 2024-03-24T00:33:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe2aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe2c13ec80 0x7efe2c1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe2aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe2c13ec80 0x7efe2c1414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49640/0 (socket says 172.21.15.40:49640) 2024-03-24T00:33:01.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe2aa00640 1 -- 172.21.15.40:0/471704331 learned_addr learned my addr 172.21.15.40:0/471704331 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:01.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe2b400640 1 --2- 172.21.15.40:0/471704331 >> v2:172.21.15.53:3300/0 conn(0x7efe2c0ce800 0x7efe2c071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe2aa00640 1 -- 172.21.15.40:0/471704331 >> v2:172.21.15.53:3300/0 conn(0x7efe2c0ce800 msgr2=0x7efe2c071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:01.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe2aa00640 1 --2- 172.21.15.40:0/471704331 >> v2:172.21.15.53:3300/0 conn(0x7efe2c0ce800 0x7efe2c071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe2aa00640 1 -- 172.21.15.40:0/471704331 >> v1:172.21.15.53:6789/0 conn(0x7efe2c1419f0 legacy=0x7efe2c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:01.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe2aa00640 1 -- 172.21.15.40:0/471704331 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c0cf100 con 0x7efe2c13ec80 2024-03-24T00:33:01.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.552+0000 7efe2b400640 1 --2- 172.21.15.40:0/471704331 >> v2:172.21.15.53:3300/0 conn(0x7efe2c0ce800 0x7efe2c071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:33:01.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.556+0000 7efe2aa00640 1 --2- 172.21.15.40:0/471704331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c13ec80 0x7efe2c1414b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7efe2c067e60 tx=0x7efe1c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b9b7625c87d6dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:01.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.556+0000 7efe2a000640 1 -- 172.21.15.40:0/471704331 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c0aa660 con 0x7efe2c13ec80 2024-03-24T00:33:01.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.556+0000 7efe2a000640 1 -- 172.21.15.40:0/471704331 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe1c0af070 con 0x7efe2c13ec80 2024-03-24T00:33:01.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.556+0000 7efe2a000640 1 -- 172.21.15.40:0/471704331 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe1c0aaa20 con 0x7efe2c13ec80 2024-03-24T00:33:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.556+0000 7efe31600640 1 -- 172.21.15.40:0/471704331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c13ec80 msgr2=0x7efe2c1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.556+0000 7efe31600640 1 --2- 172.21.15.40:0/471704331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c13ec80 0x7efe2c1414b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7efe2c067e60 tx=0x7efe1c01c4e0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.556+0000 7efe31600640 1 -- 172.21.15.40:0/471704331 shutdown_connections 2024-03-24T00:33:01.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.556+0000 7efe31600640 1 --2- 172.21.15.40:0/471704331 >> v2:172.21.15.53:3300/0 conn(0x7efe2c0ce800 0x7efe2c071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.556+0000 7efe31600640 1 --2- 172.21.15.40:0/471704331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c13ec80 0x7efe2c1414b0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.556+0000 7efe31600640 1 -- 172.21.15.40:0/471704331 >> 172.21.15.40:0/471704331 conn(0x7efe2c0b17b0 msgr2=0x7efe2c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:01.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.556+0000 7efe31600640 1 -- 172.21.15.40:0/471704331 shutdown_connections 2024-03-24T00:33:01.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.556+0000 7efe31600640 1 -- 172.21.15.40:0/471704331 wait complete. 2024-03-24T00:33:01.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe31600640 1 Processor -- start 2024-03-24T00:33:01.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe31600640 1 -- start start 2024-03-24T00:33:01.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe31600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe2c0ce800 0x7efe2c13a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe31600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c13ec80 0x7efe2c156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe31600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe2c1419f0 0x7efe2c06dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe31600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c0723e0 con 0x7efe2c13ec80 2024-03-24T00:33:01.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe31600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c072260 con 0x7efe2c0ce800 2024-03-24T00:33:01.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe31600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c072560 con 0x7efe2c1419f0 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe23e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c13ec80 0x7efe2c156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe23e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c13ec80 0x7efe2c156a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49654/0 (socket says 172.21.15.40:49654) 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe23e00640 1 -- 172.21.15.40:0/2685304345 learned_addr learned my addr 172.21.15.40:0/2685304345 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe2aa00640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe2c0ce800 0x7efe2c13a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe2b400640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe2c1419f0 0x7efe2c06dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe23e00640 1 -- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe2c1419f0 msgr2=0x7efe2c06dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe23e00640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe2c1419f0 0x7efe2c06dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe23e00640 1 -- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe2c0ce800 msgr2=0x7efe2c13a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe23e00640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe2c0ce800 0x7efe2c13a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe23e00640 1 -- 172.21.15.40:0/2685304345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c13c1e0 con 0x7efe2c13ec80 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe23e00640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c13ec80 0x7efe2c156a70 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c4c0 tx=0x7efe14001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe22a00640 1 -- 172.21.15.40:0/2685304345 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14017070 con 0x7efe2c13ec80 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe22a00640 1 -- 172.21.15.40:0/2685304345 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe14003600 con 0x7efe2c13ec80 2024-03-24T00:33:01.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe22a00640 1 -- 172.21.15.40:0/2685304345 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe140039f0 con 0x7efe2c13ec80 2024-03-24T00:33:01.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe31600640 1 -- 172.21.15.40:0/2685304345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2c06e9a0 con 0x7efe2c13ec80 2024-03-24T00:33:01.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe31600640 1 -- 172.21.15.40:0/2685304345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2c13c480 con 0x7efe2c13ec80 2024-03-24T00:33:01.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.560+0000 7efe31600640 1 -- 172.21.15.40:0/2685304345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2c068170 con 0x7efe2c13ec80 2024-03-24T00:33:01.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.564+0000 7efe22a00640 1 -- 172.21.15.40:0/2685304345 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7efe14003b90 con 0x7efe2c13ec80 2024-03-24T00:33:01.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.564+0000 7efe22a00640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe08067660 0x7efe08069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.564+0000 7efe22a00640 1 -- 172.21.15.40:0/2685304345 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe14084bf0 con 0x7efe2c13ec80 2024-03-24T00:33:01.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.564+0000 7efe2aa00640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe08067660 0x7efe08069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.564+0000 7efe2aa00640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe08067660 0x7efe08069b20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7efe1c0ae000 tx=0x7efe1c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:01.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.568+0000 7efe22a00640 1 -- 172.21.15.40:0/2685304345 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7efe14055200 con 0x7efe2c13ec80 2024-03-24T00:33:01.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.756+0000 7efe31600640 1 -- 172.21.15.40:0/2685304345 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe2c072b90 con 0x7efe08067660 2024-03-24T00:33:01.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.764+0000 7efe22a00640 1 -- 172.21.15.40:0/2685304345 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251890 (secure 0 0 0) 0x7efe2c072b90 con 0x7efe08067660 2024-03-24T00:33:01.769 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:01.770 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:33:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.768+0000 7efe31600640 1 -- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe08067660 msgr2=0x7efe08069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.768+0000 7efe31600640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe08067660 0x7efe08069b20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7efe1c0ae000 tx=0x7efe1c0a7000 comp rx=0 tx=0).stop 2024-03-24T00:33:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.768+0000 7efe31600640 1 -- 172.21.15.40:0/2685304345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c13ec80 msgr2=0x7efe2c156a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.768+0000 7efe31600640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c13ec80 0x7efe2c156a70 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c4c0 tx=0x7efe14001e80 comp rx=0 tx=0).stop 2024-03-24T00:33:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.768+0000 7efe31600640 1 -- 172.21.15.40:0/2685304345 shutdown_connections 2024-03-24T00:33:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.768+0000 7efe31600640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe08067660 0x7efe08069b20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.768+0000 7efe31600640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe2c1419f0 0x7efe2c06dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.768+0000 7efe31600640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe2c13ec80 0x7efe2c156a70 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.768+0000 7efe31600640 1 --2- 172.21.15.40:0/2685304345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe2c0ce800 0x7efe2c13a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.768+0000 7efe31600640 1 -- 172.21.15.40:0/2685304345 >> 172.21.15.40:0/2685304345 conn(0x7efe2c0b17b0 msgr2=0x7efe2c13b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.768+0000 7efe31600640 1 -- 172.21.15.40:0/2685304345 shutdown_connections 2024-03-24T00:33:01.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.768+0000 7efe31600640 1 -- 172.21.15.40:0/2685304345 wait complete. 2024-03-24T00:33:01.788 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-03-24T00:33:01.222956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":250424,"kb_used_data":5136,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754724296,"statfs":{"total":773094113280,"available":772837679104,"internally_reserved":0,"allocated":5259264,"data_stored":2813545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.604997"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.645247+0000","last_change":"2024-03-24T00:32:57.641375+0000","last_active":"2024-03-24T00:32:57.645247+0000","last_peered":"2024-03-24T00:32:57.645247+0000","last_clean":"2024-03-24T00:32:57.645247+0000","last_became_active":"2024-03-24T00:32:57.640908+0000","last_became_peered":"2024-03-24T00:32:57.640908+0000","last_unstale":"2024-03-24T00:32:57.645247+0000","last_undegraded":"2024-03-24T00:32:57.645247+0000","last_fullsized":"2024-03-24T00:32:57.645247+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:09.134460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.653996+0000","last_change":"2024-03-24T00:32:57.649460+0000","last_active":"2024-03-24T00:32:57.653996+0000","last_peered":"2024-03-24T00:32:57.653996+0000","last_clean":"2024-03-24T00:32:57.653996+0000","last_became_active":"2024-03-24T00:32:57.649131+0000","last_became_peered":"2024-03-24T00:32:57.649131+0000","last_unstale":"2024-03-24T00:32:57.653996+0000","last_undegraded":"2024-03-24T00:32:57.653996+0000","last_fullsized":"2024-03-24T00:32:57.653996+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:12.978906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.648860+0000","last_change":"2024-03-24T00:32:57.646333+0000","last_active":"2024-03-24T00:32:57.648860+0000","last_peered":"2024-03-24T00:32:57.648860+0000","last_clean":"2024-03-24T00:32:57.648860+0000","last_became_active":"2024-03-24T00:32:57.644248+0000","last_became_peered":"2024-03-24T00:32:57.644248+0000","last_unstale":"2024-03-24T00:32:57.648860+0000","last_undegraded":"2024-03-24T00:32:57.648860+0000","last_fullsized":"2024-03-24T00:32:57.648860+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:26:03.923852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.662357+0000","last_change":"2024-03-24T00:32:57.657093+0000","last_active":"2024-03-24T00:32:57.662357+0000","last_peered":"2024-03-24T00:32:57.662357+0000","last_clean":"2024-03-24T00:32:57.662357+0000","last_became_active":"2024-03-24T00:32:57.656164+0000","last_became_peered":"2024-03-24T00:32:57.656164+0000","last_unstale":"2024-03-24T00:32:57.662357+0000","last_undegraded":"2024-03-24T00:32:57.662357+0000","last_fullsized":"2024-03-24T00:32:57.662357+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:45.475352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.660126+0000","last_change":"2024-03-24T00:32:57.658758+0000","last_active":"2024-03-24T00:32:57.660126+0000","last_peered":"2024-03-24T00:32:57.660126+0000","last_clean":"2024-03-24T00:32:57.660126+0000","last_became_active":"2024-03-24T00:32:57.658330+0000","last_became_peered":"2024-03-24T00:32:57.658330+0000","last_unstale":"2024-03-24T00:32:57.660126+0000","last_undegraded":"2024-03-24T00:32:57.660126+0000","last_fullsized":"2024-03-24T00:32:57.660126+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:22.931165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.654050+0000","last_change":"2024-03-24T00:32:57.651105+0000","last_active":"2024-03-24T00:32:57.654050+0000","last_peered":"2024-03-24T00:32:57.654050+0000","last_clean":"2024-03-24T00:32:57.654050+0000","last_became_active":"2024-03-24T00:32:57.650682+0000","last_became_peered":"2024-03-24T00:32:57.650682+0000","last_unstale":"2024-03-24T00:32:57.654050+0000","last_undegraded":"2024-03-24T00:32:57.654050+0000","last_fullsized":"2024-03-24T00:32:57.654050+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:56.094715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.641049+0000","last_change":"2024-03-24T00:32:57.637115+0000","last_active":"2024-03-24T00:32:57.641049+0000","last_peered":"2024-03-24T00:32:57.641049+0000","last_clean":"2024-03-24T00:32:57.641049+0000","last_became_active":"2024-03-24T00:32:57.636825+0000","last_became_peered":"2024-03-24T00:32:57.636825+0000","last_unstale":"2024-03-24T00:32:57.641049+0000","last_undegraded":"2024-03-24T00:32:57.641049+0000","last_fullsized":"2024-03-24T00:32:57.641049+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:05:49.377023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.665091+0000","last_change":"2024-03-24T00:32:57.660992+0000","last_active":"2024-03-24T00:32:57.665091+0000","last_peered":"2024-03-24T00:32:57.665091+0000","last_clean":"2024-03-24T00:32:57.665091+0000","last_became_active":"2024-03-24T00:32:57.660660+0000","last_became_peered":"2024-03-24T00:32:57.660660+0000","last_unstale":"2024-03-24T00:32:57.665091+0000","last_undegraded":"2024-03-24T00:32:57.665091+0000","last_fullsized":"2024-03-24T00:32:57.665091+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:26:03.923852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.669693+0000","last_change":"2024-03-24T00:32:57.661704+0000","last_active":"2024-03-24T00:32:57.669693+0000","last_peered":"2024-03-24T00:32:57.669693+0000","last_clean":"2024-03-24T00:32:57.669693+0000","last_became_active":"2024-03-24T00:32:57.660878+0000","last_became_peered":"2024-03-24T00:32:57.660878+0000","last_unstale":"2024-03-24T00:32:57.669693+0000","last_undegraded":"2024-03-24T00:32:57.669693+0000","last_fullsized":"2024-03-24T00:32:57.669693+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:55:12.531151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.669723+0000","last_change":"2024-03-24T00:32:57.661393+0000","last_active":"2024-03-24T00:32:57.669723+0000","last_peered":"2024-03-24T00:32:57.669723+0000","last_clean":"2024-03-24T00:32:57.669723+0000","last_became_active":"2024-03-24T00:32:57.660855+0000","last_became_peered":"2024-03-24T00:32:57.660855+0000","last_unstale":"2024-03-24T00:32:57.669723+0000","last_undegraded":"2024-03-24T00:32:57.669723+0000","last_fullsized":"2024-03-24T00:32:57.669723+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:04.853544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.670319+0000","last_change":"2024-03-24T00:32:57.663817+0000","last_active":"2024-03-24T00:32:57.670319+0000","last_peered":"2024-03-24T00:32:57.670319+0000","last_clean":"2024-03-24T00:32:57.670319+0000","last_became_active":"2024-03-24T00:32:57.663507+0000","last_became_peered":"2024-03-24T00:32:57.663507+0000","last_unstale":"2024-03-24T00:32:57.670319+0000","last_undegraded":"2024-03-24T00:32:57.670319+0000","last_fullsized":"2024-03-24T00:32:57.670319+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:50:28.064185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.655135+0000","last_change":"2024-03-24T00:32:57.653528+0000","last_active":"2024-03-24T00:32:57.655135+0000","last_peered":"2024-03-24T00:32:57.655135+0000","last_clean":"2024-03-24T00:32:57.655135+0000","last_became_active":"2024-03-24T00:32:57.652773+0000","last_became_peered":"2024-03-24T00:32:57.652773+0000","last_unstale":"2024-03-24T00:32:57.655135+0000","last_undegraded":"2024-03-24T00:32:57.655135+0000","last_fullsized":"2024-03-24T00:32:57.655135+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:05:49.377023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.652707+0000","last_change":"2024-03-24T00:32:57.650413+0000","last_active":"2024-03-24T00:32:57.652707+0000","last_peered":"2024-03-24T00:32:57.652707+0000","last_clean":"2024-03-24T00:32:57.652707+0000","last_became_active":"2024-03-24T00:32:57.650136+0000","last_became_peered":"2024-03-24T00:32:57.650136+0000","last_unstale":"2024-03-24T00:32:57.652707+0000","last_undegraded":"2024-03-24T00:32:57.652707+0000","last_fullsized":"2024-03-24T00:32:57.652707+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:02.649025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.665400+0000","last_change":"2024-03-24T00:32:57.661156+0000","last_active":"2024-03-24T00:32:57.665400+0000","last_peered":"2024-03-24T00:32:57.665400+0000","last_clean":"2024-03-24T00:32:57.665400+0000","last_became_active":"2024-03-24T00:32:57.660816+0000","last_became_peered":"2024-03-24T00:32:57.660816+0000","last_unstale":"2024-03-24T00:32:57.665400+0000","last_undegraded":"2024-03-24T00:32:57.665400+0000","last_fullsized":"2024-03-24T00:32:57.665400+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:00.542240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.654400+0000","last_change":"2024-03-24T00:32:57.649676+0000","last_active":"2024-03-24T00:32:57.654400+0000","last_peered":"2024-03-24T00:32:57.654400+0000","last_clean":"2024-03-24T00:32:57.654400+0000","last_became_active":"2024-03-24T00:32:57.649035+0000","last_became_peered":"2024-03-24T00:32:57.649035+0000","last_unstale":"2024-03-24T00:32:57.654400+0000","last_undegraded":"2024-03-24T00:32:57.654400+0000","last_fullsized":"2024-03-24T00:32:57.654400+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:35.153143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.655363+0000","last_change":"2024-03-24T00:32:57.651779+0000","last_active":"2024-03-24T00:32:57.655363+0000","last_peered":"2024-03-24T00:32:57.655363+0000","last_clean":"2024-03-24T00:32:57.655363+0000","last_became_active":"2024-03-24T00:32:57.651396+0000","last_became_peered":"2024-03-24T00:32:57.651396+0000","last_unstale":"2024-03-24T00:32:57.655363+0000","last_undegraded":"2024-03-24T00:32:57.655363+0000","last_fullsized":"2024-03-24T00:32:57.655363+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:31:30.170356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.651302+0000","last_change":"2024-03-24T00:32:57.649098+0000","last_active":"2024-03-24T00:32:57.651302+0000","last_peered":"2024-03-24T00:32:57.651302+0000","last_clean":"2024-03-24T00:32:57.651302+0000","last_became_active":"2024-03-24T00:32:57.647750+0000","last_became_peered":"2024-03-24T00:32:57.647750+0000","last_unstale":"2024-03-24T00:32:57.651302+0000","last_undegraded":"2024-03-24T00:32:57.651302+0000","last_fullsized":"2024-03-24T00:32:57.651302+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:39.125653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.649810+0000","last_change":"2024-03-24T00:32:57.647483+0000","last_active":"2024-03-24T00:32:57.649810+0000","last_peered":"2024-03-24T00:32:57.649810+0000","last_clean":"2024-03-24T00:32:57.649810+0000","last_became_active":"2024-03-24T00:32:57.647177+0000","last_became_peered":"2024-03-24T00:32:57.647177+0000","last_unstale":"2024-03-24T00:32:57.649810+0000","last_undegraded":"2024-03-24T00:32:57.649810+0000","last_fullsized":"2024-03-24T00:32:57.649810+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:04.853544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.667780+0000","last_change":"2024-03-24T00:32:57.664933+0000","last_active":"2024-03-24T00:32:57.667780+0000","last_peered":"2024-03-24T00:32:57.667780+0000","last_clean":"2024-03-24T00:32:57.667780+0000","last_became_active":"2024-03-24T00:32:57.664380+0000","last_became_peered":"2024-03-24T00:32:57.664380+0000","last_unstale":"2024-03-24T00:32:57.667780+0000","last_undegraded":"2024-03-24T00:32:57.667780+0000","last_fullsized":"2024-03-24T00:32:57.667780+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:33.948237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.663136+0000","last_change":"2024-03-24T00:32:57.651927+0000","last_active":"2024-03-24T00:32:57.663136+0000","last_peered":"2024-03-24T00:32:57.663136+0000","last_clean":"2024-03-24T00:32:57.663136+0000","last_became_active":"2024-03-24T00:32:57.651626+0000","last_became_peered":"2024-03-24T00:32:57.651626+0000","last_unstale":"2024-03-24T00:32:57.663136+0000","last_undegraded":"2024-03-24T00:32:57.663136+0000","last_fullsized":"2024-03-24T00:32:57.663136+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:31:30.170356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.656321+0000","last_change":"2024-03-24T00:32:57.653668+0000","last_active":"2024-03-24T00:32:57.656321+0000","last_peered":"2024-03-24T00:32:57.656321+0000","last_clean":"2024-03-24T00:32:57.656321+0000","last_became_active":"2024-03-24T00:32:57.653293+0000","last_became_peered":"2024-03-24T00:32:57.653293+0000","last_unstale":"2024-03-24T00:32:57.656321+0000","last_undegraded":"2024-03-24T00:32:57.656321+0000","last_fullsized":"2024-03-24T00:32:57.656321+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:58:10.847742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.660003+0000","last_change":"2024-03-24T00:32:57.657083+0000","last_active":"2024-03-24T00:32:57.660003+0000","last_peered":"2024-03-24T00:32:57.660003+0000","last_clean":"2024-03-24T00:32:57.660003+0000","last_became_active":"2024-03-24T00:32:57.656748+0000","last_became_peered":"2024-03-24T00:32:57.656748+0000","last_unstale":"2024-03-24T00:32:57.660003+0000","last_undegraded":"2024-03-24T00:32:57.660003+0000","last_fullsized":"2024-03-24T00:32:57.660003+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:42:45.402081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.653433+0000","last_change":"2024-03-24T00:32:57.651501+0000","last_active":"2024-03-24T00:32:57.653433+0000","last_peered":"2024-03-24T00:32:57.653433+0000","last_clean":"2024-03-24T00:32:57.653433+0000","last_became_active":"2024-03-24T00:32:57.651150+0000","last_became_peered":"2024-03-24T00:32:57.651150+0000","last_unstale":"2024-03-24T00:32:57.653433+0000","last_undegraded":"2024-03-24T00:32:57.653433+0000","last_fullsized":"2024-03-24T00:32:57.653433+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:36.160634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.655338+0000","last_change":"2024-03-24T00:32:57.650989+0000","last_active":"2024-03-24T00:32:57.655338+0000","last_peered":"2024-03-24T00:32:57.655338+0000","last_clean":"2024-03-24T00:32:57.655338+0000","last_became_active":"2024-03-24T00:32:57.650040+0000","last_became_peered":"2024-03-24T00:32:57.650040+0000","last_unstale":"2024-03-24T00:32:57.655338+0000","last_undegraded":"2024-03-24T00:32:57.655338+0000","last_fullsized":"2024-03-24T00:32:57.655338+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:45.533368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.666499+0000","last_change":"2024-03-24T00:32:57.663460+0000","last_active":"2024-03-24T00:32:57.666499+0000","last_peered":"2024-03-24T00:32:57.666499+0000","last_clean":"2024-03-24T00:32:57.666499+0000","last_became_active":"2024-03-24T00:32:57.663195+0000","last_became_peered":"2024-03-24T00:32:57.663195+0000","last_unstale":"2024-03-24T00:32:57.666499+0000","last_undegraded":"2024-03-24T00:32:57.666499+0000","last_fullsized":"2024-03-24T00:32:57.666499+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:38.092584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.658207+0000","last_change":"2024-03-24T00:32:57.655701+0000","last_active":"2024-03-24T00:32:57.658207+0000","last_peered":"2024-03-24T00:32:57.658207+0000","last_clean":"2024-03-24T00:32:57.658207+0000","last_became_active":"2024-03-24T00:32:57.655431+0000","last_became_peered":"2024-03-24T00:32:57.655431+0000","last_unstale":"2024-03-24T00:32:57.658207+0000","last_undegraded":"2024-03-24T00:32:57.658207+0000","last_fullsized":"2024-03-24T00:32:57.658207+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:54.726464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.655306+0000","last_change":"2024-03-24T00:32:57.653565+0000","last_active":"2024-03-24T00:32:57.655306+0000","last_peered":"2024-03-24T00:32:57.655306+0000","last_clean":"2024-03-24T00:32:57.655306+0000","last_became_active":"2024-03-24T00:32:57.653290+0000","last_became_peered":"2024-03-24T00:32:57.653290+0000","last_unstale":"2024-03-24T00:32:57.655306+0000","last_undegraded":"2024-03-24T00:32:57.655306+0000","last_fullsized":"2024-03-24T00:32:57.655306+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:55:12.531151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.655157+0000","last_change":"2024-03-24T00:32:57.653742+0000","last_active":"2024-03-24T00:32:57.655157+0000","last_peered":"2024-03-24T00:32:57.655157+0000","last_clean":"2024-03-24T00:32:57.655157+0000","last_became_active":"2024-03-24T00:32:57.653233+0000","last_became_peered":"2024-03-24T00:32:57.653233+0000","last_unstale":"2024-03-24T00:32:57.655157+0000","last_undegraded":"2024-03-24T00:32:57.655157+0000","last_fullsized":"2024-03-24T00:32:57.655157+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:09.134460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.668840+0000","last_change":"2024-03-24T00:32:57.667815+0000","last_active":"2024-03-24T00:32:57.668840+0000","last_peered":"2024-03-24T00:32:57.668840+0000","last_clean":"2024-03-24T00:32:57.668840+0000","last_became_active":"2024-03-24T00:32:57.667058+0000","last_became_peered":"2024-03-24T00:32:57.667058+0000","last_unstale":"2024-03-24T00:32:57.668840+0000","last_undegraded":"2024-03-24T00:32:57.668840+0000","last_fullsized":"2024-03-24T00:32:57.668840+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:10.928568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.645613+0000","last_change":"2024-03-24T00:32:57.643273+0000","last_active":"2024-03-24T00:32:57.645613+0000","last_peered":"2024-03-24T00:32:57.645613+0000","last_clean":"2024-03-24T00:32:57.645613+0000","last_became_active":"2024-03-24T00:32:57.642982+0000","last_became_peered":"2024-03-24T00:32:57.642982+0000","last_unstale":"2024-03-24T00:32:57.645613+0000","last_undegraded":"2024-03-24T00:32:57.645613+0000","last_fullsized":"2024-03-24T00:32:57.645613+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:45.475352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.662868+0000","last_change":"2024-03-24T00:32:57.656565+0000","last_active":"2024-03-24T00:32:57.662868+0000","last_peered":"2024-03-24T00:32:57.662868+0000","last_clean":"2024-03-24T00:32:57.662868+0000","last_became_active":"2024-03-24T00:32:57.654772+0000","last_became_peered":"2024-03-24T00:32:57.654772+0000","last_unstale":"2024-03-24T00:32:57.662868+0000","last_undegraded":"2024-03-24T00:32:57.662868+0000","last_fullsized":"2024-03-24T00:32:57.662868+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:25:42.325973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.648465+0000","last_change":"2024-03-24T00:32:57.645048+0000","last_active":"2024-03-24T00:32:57.648465+0000","last_peered":"2024-03-24T00:32:57.648465+0000","last_clean":"2024-03-24T00:32:57.648465+0000","last_became_active":"2024-03-24T00:32:57.644749+0000","last_became_peered":"2024-03-24T00:32:57.644749+0000","last_unstale":"2024-03-24T00:32:57.648465+0000","last_undegraded":"2024-03-24T00:32:57.648465+0000","last_fullsized":"2024-03-24T00:32:57.648465+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:36.160634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.658651+0000","last_change":"2024-03-24T00:32:57.654420+0000","last_active":"2024-03-24T00:32:57.658651+0000","last_peered":"2024-03-24T00:32:57.658651+0000","last_clean":"2024-03-24T00:32:57.658651+0000","last_became_active":"2024-03-24T00:32:57.654050+0000","last_became_peered":"2024-03-24T00:32:57.654050+0000","last_unstale":"2024-03-24T00:32:57.658651+0000","last_undegraded":"2024-03-24T00:32:57.658651+0000","last_fullsized":"2024-03-24T00:32:57.658651+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:45.475352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"43'4","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.675870+0000","last_change":"2024-03-24T00:32:56.641164+0000","last_active":"2024-03-24T00:32:57.675870+0000","last_peered":"2024-03-24T00:32:57.675870+0000","last_clean":"2024-03-24T00:32:57.675870+0000","last_became_active":"2024-03-24T00:32:56.640607+0000","last_became_peered":"2024-03-24T00:32:56.640607+0000","last_unstale":"2024-03-24T00:32:57.675870+0000","last_undegraded":"2024-03-24T00:32:57.675870+0000","last_fullsized":"2024-03-24T00:32:57.675870+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:23:40.773929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.660996+0000","last_change":"2024-03-24T00:32:57.658865+0000","last_active":"2024-03-24T00:32:57.660996+0000","last_peered":"2024-03-24T00:32:57.660996+0000","last_clean":"2024-03-24T00:32:57.660996+0000","last_became_active":"2024-03-24T00:32:57.656527+0000","last_became_peered":"2024-03-24T00:32:57.656527+0000","last_unstale":"2024-03-24T00:32:57.660996+0000","last_undegraded":"2024-03-24T00:32:57.660996+0000","last_fullsized":"2024-03-24T00:32:57.660996+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:02:03.160871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.666900+0000","last_change":"2024-03-24T00:32:56.644143+0000","last_active":"2024-03-24T00:32:57.666900+0000","last_peered":"2024-03-24T00:32:57.666900+0000","last_clean":"2024-03-24T00:32:57.666900+0000","last_became_active":"2024-03-24T00:32:56.643856+0000","last_became_peered":"2024-03-24T00:32:56.643856+0000","last_unstale":"2024-03-24T00:32:57.666900+0000","last_undegraded":"2024-03-24T00:32:57.666900+0000","last_fullsized":"2024-03-24T00:32:57.666900+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:07.226658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.659224+0000","last_change":"2024-03-24T00:32:57.654822+0000","last_active":"2024-03-24T00:32:57.659224+0000","last_peered":"2024-03-24T00:32:57.659224+0000","last_clean":"2024-03-24T00:32:57.659224+0000","last_became_active":"2024-03-24T00:32:57.654519+0000","last_became_peered":"2024-03-24T00:32:57.654519+0000","last_unstale":"2024-03-24T00:32:57.659224+0000","last_undegraded":"2024-03-24T00:32:57.659224+0000","last_fullsized":"2024-03-24T00:32:57.659224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:19.011696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.674076+0000","last_change":"2024-03-24T00:32:56.636221+0000","last_active":"2024-03-24T00:32:57.674076+0000","last_peered":"2024-03-24T00:32:57.674076+0000","last_clean":"2024-03-24T00:32:57.674076+0000","last_became_active":"2024-03-24T00:32:56.635925+0000","last_became_peered":"2024-03-24T00:32:56.635925+0000","last_unstale":"2024-03-24T00:32:57.674076+0000","last_undegraded":"2024-03-24T00:32:57.674076+0000","last_fullsized":"2024-03-24T00:32:57.674076+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:07.226658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.663416+0000","last_change":"2024-03-24T00:32:57.656393+0000","last_active":"2024-03-24T00:32:57.663416+0000","last_peered":"2024-03-24T00:32:57.663416+0000","last_clean":"2024-03-24T00:32:57.663416+0000","last_became_active":"2024-03-24T00:32:57.656088+0000","last_became_peered":"2024-03-24T00:32:57.656088+0000","last_unstale":"2024-03-24T00:32:57.663416+0000","last_undegraded":"2024-03-24T00:32:57.663416+0000","last_fullsized":"2024-03-24T00:32:57.663416+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:26:03.923852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.668642+0000","last_change":"2024-03-24T00:32:56.649789+0000","last_active":"2024-03-24T00:32:57.668642+0000","last_peered":"2024-03-24T00:32:57.668642+0000","last_clean":"2024-03-24T00:32:57.668642+0000","last_became_active":"2024-03-24T00:32:56.646468+0000","last_became_peered":"2024-03-24T00:32:56.646468+0000","last_unstale":"2024-03-24T00:32:57.668642+0000","last_undegraded":"2024-03-24T00:32:57.668642+0000","last_fullsized":"2024-03-24T00:32:57.668642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:50:27.055732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.659097+0000","last_change":"2024-03-24T00:32:57.657752+0000","last_active":"2024-03-24T00:32:57.659097+0000","last_peered":"2024-03-24T00:32:57.659097+0000","last_clean":"2024-03-24T00:32:57.659097+0000","last_became_active":"2024-03-24T00:32:57.657393+0000","last_became_peered":"2024-03-24T00:32:57.657393+0000","last_unstale":"2024-03-24T00:32:57.659097+0000","last_undegraded":"2024-03-24T00:32:57.659097+0000","last_fullsized":"2024-03-24T00:32:57.659097+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:50:28.064185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.674954+0000","last_change":"2024-03-24T00:32:56.642339+0000","last_active":"2024-03-24T00:32:57.674954+0000","last_peered":"2024-03-24T00:32:57.674954+0000","last_clean":"2024-03-24T00:32:57.674954+0000","last_became_active":"2024-03-24T00:32:56.641854+0000","last_became_peered":"2024-03-24T00:32:56.641854+0000","last_unstale":"2024-03-24T00:32:57.674954+0000","last_undegraded":"2024-03-24T00:32:57.674954+0000","last_fullsized":"2024-03-24T00:32:57.674954+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:21.922712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.671893+0000","last_change":"2024-03-24T00:32:57.661384+0000","last_active":"2024-03-24T00:32:57.671893+0000","last_peered":"2024-03-24T00:32:57.671893+0000","last_clean":"2024-03-24T00:32:57.671893+0000","last_became_active":"2024-03-24T00:32:57.661027+0000","last_became_peered":"2024-03-24T00:32:57.661027+0000","last_unstale":"2024-03-24T00:32:57.671893+0000","last_undegraded":"2024-03-24T00:32:57.671893+0000","last_fullsized":"2024-03-24T00:32:57.671893+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:33.356685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.667640+0000","last_change":"2024-03-24T00:32:56.645601+0000","last_active":"2024-03-24T00:32:57.667640+0000","last_peered":"2024-03-24T00:32:57.667640+0000","last_clean":"2024-03-24T00:32:57.667640+0000","last_became_active":"2024-03-24T00:32:56.645202+0000","last_became_peered":"2024-03-24T00:32:56.645202+0000","last_unstale":"2024-03-24T00:32:57.667640+0000","last_undegraded":"2024-03-24T00:32:57.667640+0000","last_fullsized":"2024-03-24T00:32:57.667640+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:07.913989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.659089+0000","last_change":"2024-03-24T00:32:57.655853+0000","last_active":"2024-03-24T00:32:57.659089+0000","last_peered":"2024-03-24T00:32:57.659089+0000","last_clean":"2024-03-24T00:32:57.659089+0000","last_became_active":"2024-03-24T00:32:57.655584+0000","last_became_peered":"2024-03-24T00:32:57.655584+0000","last_unstale":"2024-03-24T00:32:57.659089+0000","last_undegraded":"2024-03-24T00:32:57.659089+0000","last_fullsized":"2024-03-24T00:32:57.659089+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:01.187105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.645600+0000","last_change":"2024-03-24T00:32:56.638440+0000","last_active":"2024-03-24T00:32:57.645600+0000","last_peered":"2024-03-24T00:32:57.645600+0000","last_clean":"2024-03-24T00:32:57.645600+0000","last_became_active":"2024-03-24T00:32:56.638066+0000","last_became_peered":"2024-03-24T00:32:56.638066+0000","last_unstale":"2024-03-24T00:32:57.645600+0000","last_undegraded":"2024-03-24T00:32:57.645600+0000","last_fullsized":"2024-03-24T00:32:57.645600+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:55:11.522698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.653453+0000","last_change":"2024-03-24T00:32:57.651737+0000","last_active":"2024-03-24T00:32:57.653453+0000","last_peered":"2024-03-24T00:32:57.653453+0000","last_clean":"2024-03-24T00:32:57.653453+0000","last_became_active":"2024-03-24T00:32:57.651343+0000","last_became_peered":"2024-03-24T00:32:57.651343+0000","last_unstale":"2024-03-24T00:32:57.653453+0000","last_undegraded":"2024-03-24T00:32:57.653453+0000","last_fullsized":"2024-03-24T00:32:57.653453+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:02.649025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.636951+0000","last_change":"2024-03-24T00:32:56.650015+0000","last_active":"2024-03-24T00:32:57.636951+0000","last_peered":"2024-03-24T00:32:57.636951+0000","last_clean":"2024-03-24T00:32:57.636951+0000","last_became_active":"2024-03-24T00:32:56.648189+0000","last_became_peered":"2024-03-24T00:32:56.648189+0000","last_unstale":"2024-03-24T00:32:57.636951+0000","last_undegraded":"2024-03-24T00:32:57.636951+0000","last_fullsized":"2024-03-24T00:32:57.636951+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:13.894010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.649784+0000","last_change":"2024-03-24T00:32:57.647415+0000","last_active":"2024-03-24T00:32:57.649784+0000","last_peered":"2024-03-24T00:32:57.649784+0000","last_clean":"2024-03-24T00:32:57.649784+0000","last_became_active":"2024-03-24T00:32:57.647119+0000","last_became_peered":"2024-03-24T00:32:57.647119+0000","last_unstale":"2024-03-24T00:32:57.649784+0000","last_undegraded":"2024-03-24T00:32:57.649784+0000","last_fullsized":"2024-03-24T00:32:57.649784+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:00.542240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.635512+0000","last_change":"2024-03-24T00:32:56.652751+0000","last_active":"2024-03-24T00:32:57.635512+0000","last_peered":"2024-03-24T00:32:57.635512+0000","last_clean":"2024-03-24T00:32:57.635512+0000","last_became_active":"2024-03-24T00:32:56.651848+0000","last_became_peered":"2024-03-24T00:32:56.651848+0000","last_unstale":"2024-03-24T00:32:57.635512+0000","last_undegraded":"2024-03-24T00:32:57.635512+0000","last_fullsized":"2024-03-24T00:32:57.635512+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:42:44.393628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.650894+0000","last_change":"2024-03-24T00:32:57.648136+0000","last_active":"2024-03-24T00:32:57.650894+0000","last_peered":"2024-03-24T00:32:57.650894+0000","last_clean":"2024-03-24T00:32:57.650894+0000","last_became_active":"2024-03-24T00:32:57.647750+0000","last_became_peered":"2024-03-24T00:32:57.647750+0000","last_unstale":"2024-03-24T00:32:57.650894+0000","last_undegraded":"2024-03-24T00:32:57.650894+0000","last_fullsized":"2024-03-24T00:32:57.650894+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:07.244689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.641330+0000","last_change":"2024-03-24T00:32:56.645946+0000","last_active":"2024-03-24T00:32:57.641330+0000","last_peered":"2024-03-24T00:32:57.641330+0000","last_clean":"2024-03-24T00:32:57.641330+0000","last_became_active":"2024-03-24T00:32:56.645628+0000","last_became_peered":"2024-03-24T00:32:56.645628+0000","last_unstale":"2024-03-24T00:32:57.641330+0000","last_undegraded":"2024-03-24T00:32:57.641330+0000","last_fullsized":"2024-03-24T00:32:57.641330+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:06.226473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.673723+0000","last_change":"2024-03-24T00:32:57.669261+0000","last_active":"2024-03-24T00:32:57.673723+0000","last_peered":"2024-03-24T00:32:57.673723+0000","last_clean":"2024-03-24T00:32:57.673723+0000","last_became_active":"2024-03-24T00:32:57.668969+0000","last_became_peered":"2024-03-24T00:32:57.668969+0000","last_unstale":"2024-03-24T00:32:57.673723+0000","last_undegraded":"2024-03-24T00:32:57.673723+0000","last_fullsized":"2024-03-24T00:32:57.673723+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:53.944849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"43'3","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.669107+0000","last_change":"2024-03-24T00:32:56.647429+0000","last_active":"2024-03-24T00:32:57.669107+0000","last_peered":"2024-03-24T00:32:57.669107+0000","last_clean":"2024-03-24T00:32:57.669107+0000","last_became_active":"2024-03-24T00:32:56.646236+0000","last_became_peered":"2024-03-24T00:32:56.646236+0000","last_unstale":"2024-03-24T00:32:57.669107+0000","last_undegraded":"2024-03-24T00:32:57.669107+0000","last_fullsized":"2024-03-24T00:32:57.669107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:57:28.581587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.645907+0000","last_change":"2024-03-24T00:32:57.641810+0000","last_active":"2024-03-24T00:32:57.645907+0000","last_peered":"2024-03-24T00:32:57.645907+0000","last_clean":"2024-03-24T00:32:57.645907+0000","last_became_active":"2024-03-24T00:32:57.641364+0000","last_became_peered":"2024-03-24T00:32:57.641364+0000","last_unstale":"2024-03-24T00:32:57.645907+0000","last_undegraded":"2024-03-24T00:32:57.645907+0000","last_fullsized":"2024-03-24T00:32:57.645907+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:19.011696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"43'6","reported_seq":18,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.677534+0000","last_change":"2024-03-24T00:32:56.647948+0000","last_active":"2024-03-24T00:32:57.677534+0000","last_peered":"2024-03-24T00:32:57.677534+0000","last_clean":"2024-03-24T00:32:57.677534+0000","last_became_active":"2024-03-24T00:32:56.647533+0000","last_became_peered":"2024-03-24T00:32:56.647533+0000","last_unstale":"2024-03-24T00:32:57.677534+0000","last_undegraded":"2024-03-24T00:32:57.677534+0000","last_fullsized":"2024-03-24T00:32:57.677534+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:17:40.522483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.666800+0000","last_change":"2024-03-24T00:32:57.664651+0000","last_active":"2024-03-24T00:32:57.666800+0000","last_peered":"2024-03-24T00:32:57.666800+0000","last_clean":"2024-03-24T00:32:57.666800+0000","last_became_active":"2024-03-24T00:32:57.664307+0000","last_became_peered":"2024-03-24T00:32:57.664307+0000","last_unstale":"2024-03-24T00:32:57.666800+0000","last_undegraded":"2024-03-24T00:32:57.666800+0000","last_fullsized":"2024-03-24T00:32:57.666800+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:12.842264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.673923+0000","last_change":"2024-03-24T00:32:56.642994+0000","last_active":"2024-03-24T00:32:57.673923+0000","last_peered":"2024-03-24T00:32:57.673923+0000","last_clean":"2024-03-24T00:32:57.673923+0000","last_became_active":"2024-03-24T00:32:56.642702+0000","last_became_peered":"2024-03-24T00:32:56.642702+0000","last_unstale":"2024-03-24T00:32:57.673923+0000","last_undegraded":"2024-03-24T00:32:57.673923+0000","last_fullsized":"2024-03-24T00:32:57.673923+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:24.547880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.657421+0000","last_change":"2024-03-24T00:32:57.650284+0000","last_active":"2024-03-24T00:32:57.657421+0000","last_peered":"2024-03-24T00:32:57.657421+0000","last_clean":"2024-03-24T00:32:57.657421+0000","last_became_active":"2024-03-24T00:32:57.650030+0000","last_became_peered":"2024-03-24T00:32:57.650030+0000","last_unstale":"2024-03-24T00:32:57.657421+0000","last_undegraded":"2024-03-24T00:32:57.657421+0000","last_fullsized":"2024-03-24T00:32:57.657421+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:17:49.038615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.633573+0000","last_change":"2024-03-24T00:32:56.651558+0000","last_active":"2024-03-24T00:32:57.633573+0000","last_peered":"2024-03-24T00:32:57.633573+0000","last_clean":"2024-03-24T00:32:57.633573+0000","last_became_active":"2024-03-24T00:32:56.651271+0000","last_became_peered":"2024-03-24T00:32:56.651271+0000","last_unstale":"2024-03-24T00:32:57.633573+0000","last_undegraded":"2024-03-24T00:32:57.633573+0000","last_fullsized":"2024-03-24T00:32:57.633573+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:59.451437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.656351+0000","last_change":"2024-03-24T00:32:57.653532+0000","last_active":"2024-03-24T00:32:57.656351+0000","last_peered":"2024-03-24T00:32:57.656351+0000","last_clean":"2024-03-24T00:32:57.656351+0000","last_became_active":"2024-03-24T00:32:57.653107+0000","last_became_peered":"2024-03-24T00:32:57.653107+0000","last_unstale":"2024-03-24T00:32:57.656351+0000","last_undegraded":"2024-03-24T00:32:57.656351+0000","last_fullsized":"2024-03-24T00:32:57.656351+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:33.312475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.644043+0000","last_change":"2024-03-24T00:32:56.649216+0000","last_active":"2024-03-24T00:32:57.644043+0000","last_peered":"2024-03-24T00:32:57.644043+0000","last_clean":"2024-03-24T00:32:57.644043+0000","last_became_active":"2024-03-24T00:32:56.648949+0000","last_became_peered":"2024-03-24T00:32:56.648949+0000","last_unstale":"2024-03-24T00:32:57.644043+0000","last_undegraded":"2024-03-24T00:32:57.644043+0000","last_fullsized":"2024-03-24T00:32:57.644043+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:18.003243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.658198+0000","last_change":"2024-03-24T00:32:57.655907+0000","last_active":"2024-03-24T00:32:57.658198+0000","last_peered":"2024-03-24T00:32:57.658198+0000","last_clean":"2024-03-24T00:32:57.658198+0000","last_became_active":"2024-03-24T00:32:57.655342+0000","last_became_peered":"2024-03-24T00:32:57.655342+0000","last_unstale":"2024-03-24T00:32:57.658198+0000","last_undegraded":"2024-03-24T00:32:57.658198+0000","last_fullsized":"2024-03-24T00:32:57.658198+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:25.556333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"43'8","reported_seq":20,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.673861+0000","last_change":"2024-03-24T00:32:56.649467+0000","last_active":"2024-03-24T00:32:57.673861+0000","last_peered":"2024-03-24T00:32:57.673861+0000","last_clean":"2024-03-24T00:32:57.673861+0000","last_became_active":"2024-03-24T00:32:56.649196+0000","last_became_peered":"2024-03-24T00:32:56.649196+0000","last_unstale":"2024-03-24T00:32:57.673861+0000","last_undegraded":"2024-03-24T00:32:57.673861+0000","last_fullsized":"2024-03-24T00:32:57.673861+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:32.939784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.658231+0000","last_change":"2024-03-24T00:32:57.656170+0000","last_active":"2024-03-24T00:32:57.658231+0000","last_peered":"2024-03-24T00:32:57.658231+0000","last_clean":"2024-03-24T00:32:57.658231+0000","last_became_active":"2024-03-24T00:32:57.655896+0000","last_became_peered":"2024-03-24T00:32:57.655896+0000","last_unstale":"2024-03-24T00:32:57.658231+0000","last_undegraded":"2024-03-24T00:32:57.658231+0000","last_fullsized":"2024-03-24T00:32:57.658231+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:02:03.160871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.676474+0000","last_change":"2024-03-24T00:32:56.646580+0000","last_active":"2024-03-24T00:32:57.676474+0000","last_peered":"2024-03-24T00:32:57.676474+0000","last_clean":"2024-03-24T00:32:57.676474+0000","last_became_active":"2024-03-24T00:32:56.646267+0000","last_became_peered":"2024-03-24T00:32:56.646267+0000","last_unstale":"2024-03-24T00:32:57.676474+0000","last_undegraded":"2024-03-24T00:32:57.676474+0000","last_fullsized":"2024-03-24T00:32:57.676474+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:51.841137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.655924+0000","last_change":"2024-03-24T00:32:57.646920+0000","last_active":"2024-03-24T00:32:57.655924+0000","last_peered":"2024-03-24T00:32:57.655924+0000","last_clean":"2024-03-24T00:32:57.655924+0000","last_became_active":"2024-03-24T00:32:57.646583+0000","last_became_peered":"2024-03-24T00:32:57.646583+0000","last_unstale":"2024-03-24T00:32:57.655924+0000","last_undegraded":"2024-03-24T00:32:57.655924+0000","last_fullsized":"2024-03-24T00:32:57.655924+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:42.847452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.673662+0000","last_change":"2024-03-24T00:32:56.641393+0000","last_active":"2024-03-24T00:32:57.673662+0000","last_peered":"2024-03-24T00:32:57.673662+0000","last_clean":"2024-03-24T00:32:57.673662+0000","last_became_active":"2024-03-24T00:32:56.641055+0000","last_became_peered":"2024-03-24T00:32:56.641055+0000","last_unstale":"2024-03-24T00:32:57.673662+0000","last_undegraded":"2024-03-24T00:32:57.673662+0000","last_fullsized":"2024-03-24T00:32:57.673662+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:00.178652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.660349+0000","last_change":"2024-03-24T00:32:57.657007+0000","last_active":"2024-03-24T00:32:57.660349+0000","last_peered":"2024-03-24T00:32:57.660349+0000","last_clean":"2024-03-24T00:32:57.660349+0000","last_became_active":"2024-03-24T00:32:57.656716+0000","last_became_peered":"2024-03-24T00:32:57.656716+0000","last_unstale":"2024-03-24T00:32:57.660349+0000","last_undegraded":"2024-03-24T00:32:57.660349+0000","last_fullsized":"2024-03-24T00:32:57.660349+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:54.726464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.675504+0000","last_change":"2024-03-24T00:32:56.637709+0000","last_active":"2024-03-24T00:32:57.675504+0000","last_peered":"2024-03-24T00:32:57.675504+0000","last_clean":"2024-03-24T00:32:57.675504+0000","last_became_active":"2024-03-24T00:32:56.637386+0000","last_became_peered":"2024-03-24T00:32:56.637386+0000","last_unstale":"2024-03-24T00:32:57.675504+0000","last_undegraded":"2024-03-24T00:32:57.675504+0000","last_fullsized":"2024-03-24T00:32:57.675504+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:59.451437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.666832+0000","last_change":"2024-03-24T00:32:57.664680+0000","last_active":"2024-03-24T00:32:57.666832+0000","last_peered":"2024-03-24T00:32:57.666832+0000","last_clean":"2024-03-24T00:32:57.666832+0000","last_became_active":"2024-03-24T00:32:57.663919+0000","last_became_peered":"2024-03-24T00:32:57.663919+0000","last_unstale":"2024-03-24T00:32:57.666832+0000","last_undegraded":"2024-03-24T00:32:57.666832+0000","last_fullsized":"2024-03-24T00:32:57.666832+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:14.902463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.644792+0000","last_change":"2024-03-24T00:32:56.651295+0000","last_active":"2024-03-24T00:32:57.644792+0000","last_peered":"2024-03-24T00:32:57.644792+0000","last_clean":"2024-03-24T00:32:57.644792+0000","last_became_active":"2024-03-24T00:32:56.647097+0000","last_became_peered":"2024-03-24T00:32:56.647097+0000","last_unstale":"2024-03-24T00:32:57.644792+0000","last_undegraded":"2024-03-24T00:32:57.644792+0000","last_fullsized":"2024-03-24T00:32:57.644792+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:06.236237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.672324+0000","last_change":"2024-03-24T00:32:57.667931+0000","last_active":"2024-03-24T00:32:57.672324+0000","last_peered":"2024-03-24T00:32:57.672324+0000","last_clean":"2024-03-24T00:32:57.672324+0000","last_became_active":"2024-03-24T00:32:57.667640+0000","last_became_peered":"2024-03-24T00:32:57.667640+0000","last_unstale":"2024-03-24T00:32:57.672324+0000","last_undegraded":"2024-03-24T00:32:57.672324+0000","last_fullsized":"2024-03-24T00:32:57.672324+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:54:45.618722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.676277+0000","last_change":"2024-03-24T00:32:56.642107+0000","last_active":"2024-03-24T00:32:57.676277+0000","last_peered":"2024-03-24T00:32:57.676277+0000","last_clean":"2024-03-24T00:32:57.676277+0000","last_became_active":"2024-03-24T00:32:56.641494+0000","last_became_peered":"2024-03-24T00:32:56.641494+0000","last_unstale":"2024-03-24T00:32:57.676277+0000","last_undegraded":"2024-03-24T00:32:57.676277+0000","last_fullsized":"2024-03-24T00:32:57.676277+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:56.812668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.661415+0000","last_change":"2024-03-24T00:32:57.651891+0000","last_active":"2024-03-24T00:32:57.661415+0000","last_peered":"2024-03-24T00:32:57.661415+0000","last_clean":"2024-03-24T00:32:57.661415+0000","last_became_active":"2024-03-24T00:32:57.651599+0000","last_became_peered":"2024-03-24T00:32:57.651599+0000","last_unstale":"2024-03-24T00:32:57.661415+0000","last_undegraded":"2024-03-24T00:32:57.661415+0000","last_fullsized":"2024-03-24T00:32:57.661415+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:42:45.402081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.647390+0000","last_change":"2024-03-24T00:32:56.645211+0000","last_active":"2024-03-24T00:32:57.647390+0000","last_peered":"2024-03-24T00:32:57.647390+0000","last_clean":"2024-03-24T00:32:57.647390+0000","last_became_active":"2024-03-24T00:32:56.644497+0000","last_became_peered":"2024-03-24T00:32:56.644497+0000","last_unstale":"2024-03-24T00:32:57.647390+0000","last_undegraded":"2024-03-24T00:32:57.647390+0000","last_fullsized":"2024-03-24T00:32:57.647390+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:50:27.055732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.644578+0000","last_change":"2024-03-24T00:32:57.639445+0000","last_active":"2024-03-24T00:32:57.644578+0000","last_peered":"2024-03-24T00:32:57.644578+0000","last_clean":"2024-03-24T00:32:57.644578+0000","last_became_active":"2024-03-24T00:32:57.639154+0000","last_became_peered":"2024-03-24T00:32:57.639154+0000","last_unstale":"2024-03-24T00:32:57.644578+0000","last_undegraded":"2024-03-24T00:32:57.644578+0000","last_fullsized":"2024-03-24T00:32:57.644578+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:45.533368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.643489+0000","last_change":"2024-03-24T00:32:56.653106+0000","last_active":"2024-03-24T00:32:57.643489+0000","last_peered":"2024-03-24T00:32:57.643489+0000","last_clean":"2024-03-24T00:32:57.643489+0000","last_became_active":"2024-03-24T00:32:56.652932+0000","last_became_peered":"2024-03-24T00:32:56.652932+0000","last_unstale":"2024-03-24T00:32:57.643489+0000","last_undegraded":"2024-03-24T00:32:57.643489+0000","last_fullsized":"2024-03-24T00:32:57.643489+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:31.302381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.659279+0000","last_change":"2024-03-24T00:32:57.647770+0000","last_active":"2024-03-24T00:32:57.659279+0000","last_peered":"2024-03-24T00:32:57.659279+0000","last_clean":"2024-03-24T00:32:57.659279+0000","last_became_active":"2024-03-24T00:32:57.647481+0000","last_became_peered":"2024-03-24T00:32:57.647481+0000","last_unstale":"2024-03-24T00:32:57.659279+0000","last_undegraded":"2024-03-24T00:32:57.659279+0000","last_fullsized":"2024-03-24T00:32:57.659279+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:08.922442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.632533+0000","last_change":"2024-03-24T00:32:56.647756+0000","last_active":"2024-03-24T00:32:57.632533+0000","last_peered":"2024-03-24T00:32:57.632533+0000","last_clean":"2024-03-24T00:32:57.632533+0000","last_became_active":"2024-03-24T00:32:56.647349+0000","last_became_peered":"2024-03-24T00:32:56.647349+0000","last_unstale":"2024-03-24T00:32:57.632533+0000","last_undegraded":"2024-03-24T00:32:57.632533+0000","last_fullsized":"2024-03-24T00:32:57.632533+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:55:11.522698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.666732+0000","last_change":"2024-03-24T00:32:57.664624+0000","last_active":"2024-03-24T00:32:57.666732+0000","last_peered":"2024-03-24T00:32:57.666732+0000","last_clean":"2024-03-24T00:32:57.666732+0000","last_became_active":"2024-03-24T00:32:57.664190+0000","last_became_peered":"2024-03-24T00:32:57.664190+0000","last_unstale":"2024-03-24T00:32:57.666732+0000","last_undegraded":"2024-03-24T00:32:57.666732+0000","last_fullsized":"2024-03-24T00:32:57.666732+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:55:47.192626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.644964+0000","last_change":"2024-03-24T00:32:56.644674+0000","last_active":"2024-03-24T00:32:57.644964+0000","last_peered":"2024-03-24T00:32:57.644964+0000","last_clean":"2024-03-24T00:32:57.644964+0000","last_became_active":"2024-03-24T00:32:56.644372+0000","last_became_peered":"2024-03-24T00:32:56.644372+0000","last_unstale":"2024-03-24T00:32:57.644964+0000","last_undegraded":"2024-03-24T00:32:57.644964+0000","last_fullsized":"2024-03-24T00:32:57.644964+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:03.399082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.659125+0000","last_change":"2024-03-24T00:32:57.654013+0000","last_active":"2024-03-24T00:32:57.659125+0000","last_peered":"2024-03-24T00:32:57.659125+0000","last_clean":"2024-03-24T00:32:57.659125+0000","last_became_active":"2024-03-24T00:32:57.653703+0000","last_became_peered":"2024-03-24T00:32:57.653703+0000","last_unstale":"2024-03-24T00:32:57.659125+0000","last_undegraded":"2024-03-24T00:32:57.659125+0000","last_fullsized":"2024-03-24T00:32:57.659125+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:08.235111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.634219+0000","last_change":"2024-03-24T00:32:56.652565+0000","last_active":"2024-03-24T00:32:57.634219+0000","last_peered":"2024-03-24T00:32:57.634219+0000","last_clean":"2024-03-24T00:32:57.634219+0000","last_became_active":"2024-03-24T00:32:56.652288+0000","last_became_peered":"2024-03-24T00:32:56.652288+0000","last_unstale":"2024-03-24T00:32:57.634219+0000","last_undegraded":"2024-03-24T00:32:57.634219+0000","last_fullsized":"2024-03-24T00:32:57.634219+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:11.970453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.646855+0000","last_change":"2024-03-24T00:32:57.644718+0000","last_active":"2024-03-24T00:32:57.646855+0000","last_peered":"2024-03-24T00:32:57.646855+0000","last_clean":"2024-03-24T00:32:57.646855+0000","last_became_active":"2024-03-24T00:32:57.644469+0000","last_became_peered":"2024-03-24T00:32:57.644469+0000","last_unstale":"2024-03-24T00:32:57.646855+0000","last_undegraded":"2024-03-24T00:32:57.646855+0000","last_fullsized":"2024-03-24T00:32:57.646855+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:22.931165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.632095+0000","last_change":"2024-03-24T00:32:56.635338+0000","last_active":"2024-03-24T00:32:57.632095+0000","last_peered":"2024-03-24T00:32:57.632095+0000","last_clean":"2024-03-24T00:32:57.632095+0000","last_became_active":"2024-03-24T00:32:56.635087+0000","last_became_peered":"2024-03-24T00:32:56.635087+0000","last_unstale":"2024-03-24T00:32:57.632095+0000","last_undegraded":"2024-03-24T00:32:57.632095+0000","last_fullsized":"2024-03-24T00:32:57.632095+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:17:48.030162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.654030+0000","last_change":"2024-03-24T00:32:57.646622+0000","last_active":"2024-03-24T00:32:57.654030+0000","last_peered":"2024-03-24T00:32:57.654030+0000","last_clean":"2024-03-24T00:32:57.654030+0000","last_became_active":"2024-03-24T00:32:57.646197+0000","last_became_peered":"2024-03-24T00:32:57.646197+0000","last_unstale":"2024-03-24T00:32:57.654030+0000","last_undegraded":"2024-03-24T00:32:57.654030+0000","last_fullsized":"2024-03-24T00:32:57.654030+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:28:22.693763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.632991+0000","last_change":"2024-03-24T00:32:56.648030+0000","last_active":"2024-03-24T00:32:57.632991+0000","last_peered":"2024-03-24T00:32:57.632991+0000","last_clean":"2024-03-24T00:32:57.632991+0000","last_became_active":"2024-03-24T00:32:56.647557+0000","last_became_peered":"2024-03-24T00:32:56.647557+0000","last_unstale":"2024-03-24T00:32:57.632991+0000","last_undegraded":"2024-03-24T00:32:57.632991+0000","last_fullsized":"2024-03-24T00:32:57.632991+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:32.348232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.671352+0000","last_change":"2024-03-24T00:32:57.669507+0000","last_active":"2024-03-24T00:32:57.671352+0000","last_peered":"2024-03-24T00:32:57.671352+0000","last_clean":"2024-03-24T00:32:57.671352+0000","last_became_active":"2024-03-24T00:32:57.669120+0000","last_became_peered":"2024-03-24T00:32:57.669120+0000","last_unstale":"2024-03-24T00:32:57.671352+0000","last_undegraded":"2024-03-24T00:32:57.671352+0000","last_fullsized":"2024-03-24T00:32:57.671352+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:03.270143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.634327+0000","last_change":"2024-03-24T00:32:56.649287+0000","last_active":"2024-03-24T00:32:57.634327+0000","last_peered":"2024-03-24T00:32:57.634327+0000","last_clean":"2024-03-24T00:32:57.634327+0000","last_became_active":"2024-03-24T00:32:56.648928+0000","last_became_peered":"2024-03-24T00:32:56.648928+0000","last_unstale":"2024-03-24T00:32:57.634327+0000","last_undegraded":"2024-03-24T00:32:57.634327+0000","last_fullsized":"2024-03-24T00:32:57.634327+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:17:48.030162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.648101+0000","last_change":"2024-03-24T00:32:57.645819+0000","last_active":"2024-03-24T00:32:57.648101+0000","last_peered":"2024-03-24T00:32:57.648101+0000","last_clean":"2024-03-24T00:32:57.648101+0000","last_became_active":"2024-03-24T00:32:57.643963+0000","last_became_peered":"2024-03-24T00:32:57.643963+0000","last_unstale":"2024-03-24T00:32:57.648101+0000","last_undegraded":"2024-03-24T00:32:57.648101+0000","last_fullsized":"2024-03-24T00:32:57.648101+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:33.356685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.632524+0000","last_change":"2024-03-24T00:32:56.644266+0000","last_active":"2024-03-24T00:32:57.632524+0000","last_peered":"2024-03-24T00:32:57.632524+0000","last_clean":"2024-03-24T00:32:57.632524+0000","last_became_active":"2024-03-24T00:32:56.643959+0000","last_became_peered":"2024-03-24T00:32:56.643959+0000","last_unstale":"2024-03-24T00:32:57.632524+0000","last_undegraded":"2024-03-24T00:32:57.632524+0000","last_fullsized":"2024-03-24T00:32:57.632524+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:00.178652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.656129+0000","last_change":"2024-03-24T00:32:57.654428+0000","last_active":"2024-03-24T00:32:57.656129+0000","last_peered":"2024-03-24T00:32:57.656129+0000","last_clean":"2024-03-24T00:32:57.656129+0000","last_became_active":"2024-03-24T00:32:57.651529+0000","last_became_peered":"2024-03-24T00:32:57.651529+0000","last_unstale":"2024-03-24T00:32:57.656129+0000","last_undegraded":"2024-03-24T00:32:57.656129+0000","last_fullsized":"2024-03-24T00:32:57.656129+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:33.356685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.637316+0000","last_change":"2024-03-24T00:32:56.646192+0000","last_active":"2024-03-24T00:32:57.637316+0000","last_peered":"2024-03-24T00:32:57.637316+0000","last_clean":"2024-03-24T00:32:57.637316+0000","last_became_active":"2024-03-24T00:32:56.645876+0000","last_became_peered":"2024-03-24T00:32:56.645876+0000","last_unstale":"2024-03-24T00:32:57.637316+0000","last_undegraded":"2024-03-24T00:32:57.637316+0000","last_fullsized":"2024-03-24T00:32:57.637316+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:09.920115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.669747+0000","last_change":"2024-03-24T00:32:57.661552+0000","last_active":"2024-03-24T00:32:57.669747+0000","last_peered":"2024-03-24T00:32:57.669747+0000","last_clean":"2024-03-24T00:32:57.669747+0000","last_became_active":"2024-03-24T00:32:57.660943+0000","last_became_peered":"2024-03-24T00:32:57.660943+0000","last_unstale":"2024-03-24T00:32:57.669747+0000","last_undegraded":"2024-03-24T00:32:57.669747+0000","last_fullsized":"2024-03-24T00:32:57.669747+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:39.125653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.666376+0000","last_change":"2024-03-24T00:32:56.648195+0000","last_active":"2024-03-24T00:32:57.666376+0000","last_peered":"2024-03-24T00:32:57.666376+0000","last_clean":"2024-03-24T00:32:57.666376+0000","last_became_active":"2024-03-24T00:32:56.647589+0000","last_became_peered":"2024-03-24T00:32:56.647589+0000","last_unstale":"2024-03-24T00:32:57.666376+0000","last_undegraded":"2024-03-24T00:32:57.666376+0000","last_fullsized":"2024-03-24T00:32:57.666376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:57:28.581587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.632639+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:32:57.632639+0000","last_peered":"2024-03-24T00:32:57.632639+0000","last_clean":"2024-03-24T00:32:57.632639+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:32:57.632639+0000","last_undegraded":"2024-03-24T00:32:57.632639+0000","last_fullsized":"2024-03-24T00:32:57.632639+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1101824,"data_stored":794998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":205212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":507904,"data_stored":204610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":507904,"data_stored":204630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":516096,"data_stored":205178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":206174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":204610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":788133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:33:01.792 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:33:01.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd651e00640 1 Processor -- start 2024-03-24T00:33:01.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd651e00640 1 -- start start 2024-03-24T00:33:01.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd651e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd64c15aa50 0x7fd64c0ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd651e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64c15ae30 0x7fd64c071600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd651e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64c0b3420 con 0x7fd64c0722e0 2024-03-24T00:33:01.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd651e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd64c070af0 con 0x7fd64c15aa50 2024-03-24T00:33:01.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd651e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c0b3a10 con 0x7fd64c15ae30 2024-03-24T00:33:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd650a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd64c15aa50 0x7fd64c0ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd651400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64c15ae30 0x7fd64c071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd650a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd64c15aa50 0x7fd64c0ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:35074/0 (socket says 172.21.15.40:35074) 2024-03-24T00:33:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd64be00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd64c0722e0 0x7fd64c06cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:54086/0 (socket says 172.21.15.40:54086) 2024-03-24T00:33:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd650a00640 1 -- 172.21.15.40:0/1951570539 learned_addr learned my addr 172.21.15.40:0/1951570539 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd651400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64c15ae30 0x7fd64c071600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:44128/0 (socket says 172.21.15.40:44128) 2024-03-24T00:33:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd650a00640 1 -- 172.21.15.40:0/1951570539 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64c15ae30 msgr2=0x7fd64c071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd650a00640 1 --2- 172.21.15.40:0/1951570539 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64c15ae30 0x7fd64c071600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd650a00640 1 -- 172.21.15.40:0/1951570539 >> v1:172.21.15.53:6789/0 conn(0x7fd64c0722e0 legacy=0x7fd64c06cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd650a00640 1 -- 172.21.15.40:0/1951570539 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c070d40 con 0x7fd64c15aa50 2024-03-24T00:33:01.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.972+0000 7fd64b400640 1 -- 172.21.15.40:0/1951570539 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682712571 0 0) 0x7fd638001170 con 0x7fd64c0722e0 2024-03-24T00:33:01.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.976+0000 7fd650a00640 1 --2- 172.21.15.40:0/1951570539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd64c15aa50 0x7fd64c0ccde0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd640009870 tx=0x7fd64001c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=115ca209ebb2485a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.976+0000 7fd64b400640 1 -- 172.21.15.40:0/1951570539 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6400b0070 con 0x7fd64c15aa50 2024-03-24T00:33:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.976+0000 7fd64b400640 1 -- 172.21.15.40:0/1951570539 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6400ab600 con 0x7fd64c15aa50 2024-03-24T00:33:01.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.976+0000 7fd64b400640 1 -- 172.21.15.40:0/1951570539 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6400ab9d0 con 0x7fd64c15aa50 2024-03-24T00:33:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.976+0000 7fd651e00640 1 -- 172.21.15.40:0/1951570539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd64c15aa50 msgr2=0x7fd64c0ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.976+0000 7fd651e00640 1 --2- 172.21.15.40:0/1951570539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd64c15aa50 0x7fd64c0ccde0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd640009870 tx=0x7fd64001c4d0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.976+0000 7fd651e00640 1 -- 172.21.15.40:0/1951570539 shutdown_connections 2024-03-24T00:33:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.976+0000 7fd651e00640 1 --2- 172.21.15.40:0/1951570539 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64c15ae30 0x7fd64c071600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.976+0000 7fd651e00640 1 --2- 172.21.15.40:0/1951570539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd64c15aa50 0x7fd64c0ccde0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.976+0000 7fd651e00640 1 -- 172.21.15.40:0/1951570539 >> 172.21.15.40:0/1951570539 conn(0x7fd64c0b1740 msgr2=0x7fd64c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.976+0000 7fd651e00640 1 -- 172.21.15.40:0/1951570539 shutdown_connections 2024-03-24T00:33:01.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.976+0000 7fd651e00640 1 -- 172.21.15.40:0/1951570539 wait complete. 2024-03-24T00:33:01.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd651e00640 1 Processor -- start 2024-03-24T00:33:01.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd651e00640 1 -- start start 2024-03-24T00:33:01.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd651e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64c0722e0 0x7fd64c1402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd651e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd64c15aa50 0x7fd64c147fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd651e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd64c15ae30 0x7fd64c1484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd651e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c0b4a40 con 0x7fd64c15aa50 2024-03-24T00:33:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd651e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c0b48c0 con 0x7fd64c15ae30 2024-03-24T00:33:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd651e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c0b4bc0 con 0x7fd64c0722e0 2024-03-24T00:33:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd64be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd64c15aa50 0x7fd64c147fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd64be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd64c15aa50 0x7fd64c147fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49658/0 (socket says 172.21.15.40:49658) 2024-03-24T00:33:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd64be00640 1 -- 172.21.15.40:0/802724504 learned_addr learned my addr 172.21.15.40:0/802724504 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd650a00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64c0722e0 0x7fd64c1402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd651400640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd64c15ae30 0x7fd64c1484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd650a00640 1 -- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd64c15ae30 msgr2=0x7fd64c1484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd650a00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd64c15ae30 0x7fd64c1484f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd650a00640 1 -- 172.21.15.40:0/802724504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd64c15aa50 msgr2=0x7fd64c147fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd650a00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd64c15aa50 0x7fd64c147fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd650a00640 1 -- 172.21.15.40:0/802724504 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c0cec00 con 0x7fd64c0722e0 2024-03-24T00:33:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd64be00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd64c15aa50 0x7fd64c147fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd650a00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64c0722e0 0x7fd64c1402c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd6400ab000 tx=0x7fd64001ad10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd648c00640 1 -- 172.21.15.40:0/802724504 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6400ba020 con 0x7fd64c0722e0 2024-03-24T00:33:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd648c00640 1 -- 172.21.15.40:0/802724504 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6400b6420 con 0x7fd64c0722e0 2024-03-24T00:33:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd648c00640 1 -- 172.21.15.40:0/802724504 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6400b0040 con 0x7fd64c0722e0 2024-03-24T00:33:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd651e00640 1 -- 172.21.15.40:0/802724504 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64c0ce680 con 0x7fd64c0722e0 2024-03-24T00:33:01.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd651e00640 1 -- 172.21.15.40:0/802724504 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64c0cf570 con 0x7fd64c0722e0 2024-03-24T00:33:01.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.980+0000 7fd651e00640 1 -- 172.21.15.40:0/802724504 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64c073540 con 0x7fd64c0722e0 2024-03-24T00:33:01.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.984+0000 7fd648c00640 1 -- 172.21.15.40:0/802724504 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd6400bd020 con 0x7fd64c0722e0 2024-03-24T00:33:01.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.984+0000 7fd648c00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd62c067660 0x7fd62c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:01.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.984+0000 7fd648c00640 1 -- 172.21.15.40:0/802724504 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd64011fc20 con 0x7fd64c0722e0 2024-03-24T00:33:01.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.984+0000 7fd64be00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd62c067660 0x7fd62c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:01.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.984+0000 7fd64be00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd62c067660 0x7fd62c069b20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd64c149ba0 tx=0x7fd638007bb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:01.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:01.988+0000 7fd648c00640 1 -- 172.21.15.40:0/802724504 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd6400c4030 con 0x7fd64c0722e0 2024-03-24T00:33:02.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.148+0000 7fd651e00640 1 -- 172.21.15.40:0/802724504 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd64c04f4f0 con 0x7fd62c067660 2024-03-24T00:33:02.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.152+0000 7fd648c00640 1 -- 172.21.15.40:0/802724504 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251890 (secure 0 0 0) 0x7fd64c04f4f0 con 0x7fd62c067660 2024-03-24T00:33:02.158 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:02.160 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:33:02.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.156+0000 7fd651e00640 1 -- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd62c067660 msgr2=0x7fd62c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.156+0000 7fd651e00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd62c067660 0x7fd62c069b20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd64c149ba0 tx=0x7fd638007bb0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.156+0000 7fd651e00640 1 -- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64c0722e0 msgr2=0x7fd64c1402c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.156+0000 7fd651e00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64c0722e0 0x7fd64c1402c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd6400ab000 tx=0x7fd64001ad10 comp rx=0 tx=0).stop 2024-03-24T00:33:02.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.156+0000 7fd651e00640 1 -- 172.21.15.40:0/802724504 shutdown_connections 2024-03-24T00:33:02.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.156+0000 7fd651e00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd62c067660 0x7fd62c069b20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.156+0000 7fd651e00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd64c15ae30 0x7fd64c1484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.156+0000 7fd651e00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd64c15aa50 0x7fd64c147fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.156+0000 7fd651e00640 1 --2- 172.21.15.40:0/802724504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64c0722e0 0x7fd64c1402c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.156+0000 7fd651e00640 1 -- 172.21.15.40:0/802724504 >> 172.21.15.40:0/802724504 conn(0x7fd64c0b1740 msgr2=0x7fd64c0b0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:02.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.156+0000 7fd651e00640 1 -- 172.21.15.40:0/802724504 shutdown_connections 2024-03-24T00:33:02.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.156+0000 7fd651e00640 1 -- 172.21.15.40:0/802724504 wait complete. 2024-03-24T00:33:02.179 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-03-24T00:33:01.222956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":250424,"kb_used_data":5136,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754724296,"statfs":{"total":773094113280,"available":772837679104,"internally_reserved":0,"allocated":5259264,"data_stored":2813545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":251055666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.604997"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.645247+0000","last_change":"2024-03-24T00:32:57.641375+0000","last_active":"2024-03-24T00:32:57.645247+0000","last_peered":"2024-03-24T00:32:57.645247+0000","last_clean":"2024-03-24T00:32:57.645247+0000","last_became_active":"2024-03-24T00:32:57.640908+0000","last_became_peered":"2024-03-24T00:32:57.640908+0000","last_unstale":"2024-03-24T00:32:57.645247+0000","last_undegraded":"2024-03-24T00:32:57.645247+0000","last_fullsized":"2024-03-24T00:32:57.645247+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:09.134460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.653996+0000","last_change":"2024-03-24T00:32:57.649460+0000","last_active":"2024-03-24T00:32:57.653996+0000","last_peered":"2024-03-24T00:32:57.653996+0000","last_clean":"2024-03-24T00:32:57.653996+0000","last_became_active":"2024-03-24T00:32:57.649131+0000","last_became_peered":"2024-03-24T00:32:57.649131+0000","last_unstale":"2024-03-24T00:32:57.653996+0000","last_undegraded":"2024-03-24T00:32:57.653996+0000","last_fullsized":"2024-03-24T00:32:57.653996+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:12.978906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.648860+0000","last_change":"2024-03-24T00:32:57.646333+0000","last_active":"2024-03-24T00:32:57.648860+0000","last_peered":"2024-03-24T00:32:57.648860+0000","last_clean":"2024-03-24T00:32:57.648860+0000","last_became_active":"2024-03-24T00:32:57.644248+0000","last_became_peered":"2024-03-24T00:32:57.644248+0000","last_unstale":"2024-03-24T00:32:57.648860+0000","last_undegraded":"2024-03-24T00:32:57.648860+0000","last_fullsized":"2024-03-24T00:32:57.648860+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:26:03.923852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.662357+0000","last_change":"2024-03-24T00:32:57.657093+0000","last_active":"2024-03-24T00:32:57.662357+0000","last_peered":"2024-03-24T00:32:57.662357+0000","last_clean":"2024-03-24T00:32:57.662357+0000","last_became_active":"2024-03-24T00:32:57.656164+0000","last_became_peered":"2024-03-24T00:32:57.656164+0000","last_unstale":"2024-03-24T00:32:57.662357+0000","last_undegraded":"2024-03-24T00:32:57.662357+0000","last_fullsized":"2024-03-24T00:32:57.662357+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:45.475352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.660126+0000","last_change":"2024-03-24T00:32:57.658758+0000","last_active":"2024-03-24T00:32:57.660126+0000","last_peered":"2024-03-24T00:32:57.660126+0000","last_clean":"2024-03-24T00:32:57.660126+0000","last_became_active":"2024-03-24T00:32:57.658330+0000","last_became_peered":"2024-03-24T00:32:57.658330+0000","last_unstale":"2024-03-24T00:32:57.660126+0000","last_undegraded":"2024-03-24T00:32:57.660126+0000","last_fullsized":"2024-03-24T00:32:57.660126+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:22.931165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.654050+0000","last_change":"2024-03-24T00:32:57.651105+0000","last_active":"2024-03-24T00:32:57.654050+0000","last_peered":"2024-03-24T00:32:57.654050+0000","last_clean":"2024-03-24T00:32:57.654050+0000","last_became_active":"2024-03-24T00:32:57.650682+0000","last_became_peered":"2024-03-24T00:32:57.650682+0000","last_unstale":"2024-03-24T00:32:57.654050+0000","last_undegraded":"2024-03-24T00:32:57.654050+0000","last_fullsized":"2024-03-24T00:32:57.654050+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:56.094715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.641049+0000","last_change":"2024-03-24T00:32:57.637115+0000","last_active":"2024-03-24T00:32:57.641049+0000","last_peered":"2024-03-24T00:32:57.641049+0000","last_clean":"2024-03-24T00:32:57.641049+0000","last_became_active":"2024-03-24T00:32:57.636825+0000","last_became_peered":"2024-03-24T00:32:57.636825+0000","last_unstale":"2024-03-24T00:32:57.641049+0000","last_undegraded":"2024-03-24T00:32:57.641049+0000","last_fullsized":"2024-03-24T00:32:57.641049+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:05:49.377023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.665091+0000","last_change":"2024-03-24T00:32:57.660992+0000","last_active":"2024-03-24T00:32:57.665091+0000","last_peered":"2024-03-24T00:32:57.665091+0000","last_clean":"2024-03-24T00:32:57.665091+0000","last_became_active":"2024-03-24T00:32:57.660660+0000","last_became_peered":"2024-03-24T00:32:57.660660+0000","last_unstale":"2024-03-24T00:32:57.665091+0000","last_undegraded":"2024-03-24T00:32:57.665091+0000","last_fullsized":"2024-03-24T00:32:57.665091+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:26:03.923852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.669693+0000","last_change":"2024-03-24T00:32:57.661704+0000","last_active":"2024-03-24T00:32:57.669693+0000","last_peered":"2024-03-24T00:32:57.669693+0000","last_clean":"2024-03-24T00:32:57.669693+0000","last_became_active":"2024-03-24T00:32:57.660878+0000","last_became_peered":"2024-03-24T00:32:57.660878+0000","last_unstale":"2024-03-24T00:32:57.669693+0000","last_undegraded":"2024-03-24T00:32:57.669693+0000","last_fullsized":"2024-03-24T00:32:57.669693+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:55:12.531151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.669723+0000","last_change":"2024-03-24T00:32:57.661393+0000","last_active":"2024-03-24T00:32:57.669723+0000","last_peered":"2024-03-24T00:32:57.669723+0000","last_clean":"2024-03-24T00:32:57.669723+0000","last_became_active":"2024-03-24T00:32:57.660855+0000","last_became_peered":"2024-03-24T00:32:57.660855+0000","last_unstale":"2024-03-24T00:32:57.669723+0000","last_undegraded":"2024-03-24T00:32:57.669723+0000","last_fullsized":"2024-03-24T00:32:57.669723+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:04.853544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.670319+0000","last_change":"2024-03-24T00:32:57.663817+0000","last_active":"2024-03-24T00:32:57.670319+0000","last_peered":"2024-03-24T00:32:57.670319+0000","last_clean":"2024-03-24T00:32:57.670319+0000","last_became_active":"2024-03-24T00:32:57.663507+0000","last_became_peered":"2024-03-24T00:32:57.663507+0000","last_unstale":"2024-03-24T00:32:57.670319+0000","last_undegraded":"2024-03-24T00:32:57.670319+0000","last_fullsized":"2024-03-24T00:32:57.670319+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:50:28.064185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.655135+0000","last_change":"2024-03-24T00:32:57.653528+0000","last_active":"2024-03-24T00:32:57.655135+0000","last_peered":"2024-03-24T00:32:57.655135+0000","last_clean":"2024-03-24T00:32:57.655135+0000","last_became_active":"2024-03-24T00:32:57.652773+0000","last_became_peered":"2024-03-24T00:32:57.652773+0000","last_unstale":"2024-03-24T00:32:57.655135+0000","last_undegraded":"2024-03-24T00:32:57.655135+0000","last_fullsized":"2024-03-24T00:32:57.655135+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:05:49.377023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.652707+0000","last_change":"2024-03-24T00:32:57.650413+0000","last_active":"2024-03-24T00:32:57.652707+0000","last_peered":"2024-03-24T00:32:57.652707+0000","last_clean":"2024-03-24T00:32:57.652707+0000","last_became_active":"2024-03-24T00:32:57.650136+0000","last_became_peered":"2024-03-24T00:32:57.650136+0000","last_unstale":"2024-03-24T00:32:57.652707+0000","last_undegraded":"2024-03-24T00:32:57.652707+0000","last_fullsized":"2024-03-24T00:32:57.652707+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:02.649025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.665400+0000","last_change":"2024-03-24T00:32:57.661156+0000","last_active":"2024-03-24T00:32:57.665400+0000","last_peered":"2024-03-24T00:32:57.665400+0000","last_clean":"2024-03-24T00:32:57.665400+0000","last_became_active":"2024-03-24T00:32:57.660816+0000","last_became_peered":"2024-03-24T00:32:57.660816+0000","last_unstale":"2024-03-24T00:32:57.665400+0000","last_undegraded":"2024-03-24T00:32:57.665400+0000","last_fullsized":"2024-03-24T00:32:57.665400+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:00.542240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.654400+0000","last_change":"2024-03-24T00:32:57.649676+0000","last_active":"2024-03-24T00:32:57.654400+0000","last_peered":"2024-03-24T00:32:57.654400+0000","last_clean":"2024-03-24T00:32:57.654400+0000","last_became_active":"2024-03-24T00:32:57.649035+0000","last_became_peered":"2024-03-24T00:32:57.649035+0000","last_unstale":"2024-03-24T00:32:57.654400+0000","last_undegraded":"2024-03-24T00:32:57.654400+0000","last_fullsized":"2024-03-24T00:32:57.654400+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:35.153143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.655363+0000","last_change":"2024-03-24T00:32:57.651779+0000","last_active":"2024-03-24T00:32:57.655363+0000","last_peered":"2024-03-24T00:32:57.655363+0000","last_clean":"2024-03-24T00:32:57.655363+0000","last_became_active":"2024-03-24T00:32:57.651396+0000","last_became_peered":"2024-03-24T00:32:57.651396+0000","last_unstale":"2024-03-24T00:32:57.655363+0000","last_undegraded":"2024-03-24T00:32:57.655363+0000","last_fullsized":"2024-03-24T00:32:57.655363+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:31:30.170356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.651302+0000","last_change":"2024-03-24T00:32:57.649098+0000","last_active":"2024-03-24T00:32:57.651302+0000","last_peered":"2024-03-24T00:32:57.651302+0000","last_clean":"2024-03-24T00:32:57.651302+0000","last_became_active":"2024-03-24T00:32:57.647750+0000","last_became_peered":"2024-03-24T00:32:57.647750+0000","last_unstale":"2024-03-24T00:32:57.651302+0000","last_undegraded":"2024-03-24T00:32:57.651302+0000","last_fullsized":"2024-03-24T00:32:57.651302+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:39.125653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.649810+0000","last_change":"2024-03-24T00:32:57.647483+0000","last_active":"2024-03-24T00:32:57.649810+0000","last_peered":"2024-03-24T00:32:57.649810+0000","last_clean":"2024-03-24T00:32:57.649810+0000","last_became_active":"2024-03-24T00:32:57.647177+0000","last_became_peered":"2024-03-24T00:32:57.647177+0000","last_unstale":"2024-03-24T00:32:57.649810+0000","last_undegraded":"2024-03-24T00:32:57.649810+0000","last_fullsized":"2024-03-24T00:32:57.649810+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:04.853544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.667780+0000","last_change":"2024-03-24T00:32:57.664933+0000","last_active":"2024-03-24T00:32:57.667780+0000","last_peered":"2024-03-24T00:32:57.667780+0000","last_clean":"2024-03-24T00:32:57.667780+0000","last_became_active":"2024-03-24T00:32:57.664380+0000","last_became_peered":"2024-03-24T00:32:57.664380+0000","last_unstale":"2024-03-24T00:32:57.667780+0000","last_undegraded":"2024-03-24T00:32:57.667780+0000","last_fullsized":"2024-03-24T00:32:57.667780+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:33.948237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.663136+0000","last_change":"2024-03-24T00:32:57.651927+0000","last_active":"2024-03-24T00:32:57.663136+0000","last_peered":"2024-03-24T00:32:57.663136+0000","last_clean":"2024-03-24T00:32:57.663136+0000","last_became_active":"2024-03-24T00:32:57.651626+0000","last_became_peered":"2024-03-24T00:32:57.651626+0000","last_unstale":"2024-03-24T00:32:57.663136+0000","last_undegraded":"2024-03-24T00:32:57.663136+0000","last_fullsized":"2024-03-24T00:32:57.663136+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:31:30.170356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.656321+0000","last_change":"2024-03-24T00:32:57.653668+0000","last_active":"2024-03-24T00:32:57.656321+0000","last_peered":"2024-03-24T00:32:57.656321+0000","last_clean":"2024-03-24T00:32:57.656321+0000","last_became_active":"2024-03-24T00:32:57.653293+0000","last_became_peered":"2024-03-24T00:32:57.653293+0000","last_unstale":"2024-03-24T00:32:57.656321+0000","last_undegraded":"2024-03-24T00:32:57.656321+0000","last_fullsized":"2024-03-24T00:32:57.656321+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:58:10.847742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.660003+0000","last_change":"2024-03-24T00:32:57.657083+0000","last_active":"2024-03-24T00:32:57.660003+0000","last_peered":"2024-03-24T00:32:57.660003+0000","last_clean":"2024-03-24T00:32:57.660003+0000","last_became_active":"2024-03-24T00:32:57.656748+0000","last_became_peered":"2024-03-24T00:32:57.656748+0000","last_unstale":"2024-03-24T00:32:57.660003+0000","last_undegraded":"2024-03-24T00:32:57.660003+0000","last_fullsized":"2024-03-24T00:32:57.660003+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:42:45.402081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.653433+0000","last_change":"2024-03-24T00:32:57.651501+0000","last_active":"2024-03-24T00:32:57.653433+0000","last_peered":"2024-03-24T00:32:57.653433+0000","last_clean":"2024-03-24T00:32:57.653433+0000","last_became_active":"2024-03-24T00:32:57.651150+0000","last_became_peered":"2024-03-24T00:32:57.651150+0000","last_unstale":"2024-03-24T00:32:57.653433+0000","last_undegraded":"2024-03-24T00:32:57.653433+0000","last_fullsized":"2024-03-24T00:32:57.653433+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:36.160634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.655338+0000","last_change":"2024-03-24T00:32:57.650989+0000","last_active":"2024-03-24T00:32:57.655338+0000","last_peered":"2024-03-24T00:32:57.655338+0000","last_clean":"2024-03-24T00:32:57.655338+0000","last_became_active":"2024-03-24T00:32:57.650040+0000","last_became_peered":"2024-03-24T00:32:57.650040+0000","last_unstale":"2024-03-24T00:32:57.655338+0000","last_undegraded":"2024-03-24T00:32:57.655338+0000","last_fullsized":"2024-03-24T00:32:57.655338+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:45.533368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.666499+0000","last_change":"2024-03-24T00:32:57.663460+0000","last_active":"2024-03-24T00:32:57.666499+0000","last_peered":"2024-03-24T00:32:57.666499+0000","last_clean":"2024-03-24T00:32:57.666499+0000","last_became_active":"2024-03-24T00:32:57.663195+0000","last_became_peered":"2024-03-24T00:32:57.663195+0000","last_unstale":"2024-03-24T00:32:57.666499+0000","last_undegraded":"2024-03-24T00:32:57.666499+0000","last_fullsized":"2024-03-24T00:32:57.666499+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:38.092584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.658207+0000","last_change":"2024-03-24T00:32:57.655701+0000","last_active":"2024-03-24T00:32:57.658207+0000","last_peered":"2024-03-24T00:32:57.658207+0000","last_clean":"2024-03-24T00:32:57.658207+0000","last_became_active":"2024-03-24T00:32:57.655431+0000","last_became_peered":"2024-03-24T00:32:57.655431+0000","last_unstale":"2024-03-24T00:32:57.658207+0000","last_undegraded":"2024-03-24T00:32:57.658207+0000","last_fullsized":"2024-03-24T00:32:57.658207+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:54.726464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.655306+0000","last_change":"2024-03-24T00:32:57.653565+0000","last_active":"2024-03-24T00:32:57.655306+0000","last_peered":"2024-03-24T00:32:57.655306+0000","last_clean":"2024-03-24T00:32:57.655306+0000","last_became_active":"2024-03-24T00:32:57.653290+0000","last_became_peered":"2024-03-24T00:32:57.653290+0000","last_unstale":"2024-03-24T00:32:57.655306+0000","last_undegraded":"2024-03-24T00:32:57.655306+0000","last_fullsized":"2024-03-24T00:32:57.655306+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:55:12.531151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.655157+0000","last_change":"2024-03-24T00:32:57.653742+0000","last_active":"2024-03-24T00:32:57.655157+0000","last_peered":"2024-03-24T00:32:57.655157+0000","last_clean":"2024-03-24T00:32:57.655157+0000","last_became_active":"2024-03-24T00:32:57.653233+0000","last_became_peered":"2024-03-24T00:32:57.653233+0000","last_unstale":"2024-03-24T00:32:57.655157+0000","last_undegraded":"2024-03-24T00:32:57.655157+0000","last_fullsized":"2024-03-24T00:32:57.655157+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:09.134460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.668840+0000","last_change":"2024-03-24T00:32:57.667815+0000","last_active":"2024-03-24T00:32:57.668840+0000","last_peered":"2024-03-24T00:32:57.668840+0000","last_clean":"2024-03-24T00:32:57.668840+0000","last_became_active":"2024-03-24T00:32:57.667058+0000","last_became_peered":"2024-03-24T00:32:57.667058+0000","last_unstale":"2024-03-24T00:32:57.668840+0000","last_undegraded":"2024-03-24T00:32:57.668840+0000","last_fullsized":"2024-03-24T00:32:57.668840+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:10.928568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.645613+0000","last_change":"2024-03-24T00:32:57.643273+0000","last_active":"2024-03-24T00:32:57.645613+0000","last_peered":"2024-03-24T00:32:57.645613+0000","last_clean":"2024-03-24T00:32:57.645613+0000","last_became_active":"2024-03-24T00:32:57.642982+0000","last_became_peered":"2024-03-24T00:32:57.642982+0000","last_unstale":"2024-03-24T00:32:57.645613+0000","last_undegraded":"2024-03-24T00:32:57.645613+0000","last_fullsized":"2024-03-24T00:32:57.645613+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:45.475352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.662868+0000","last_change":"2024-03-24T00:32:57.656565+0000","last_active":"2024-03-24T00:32:57.662868+0000","last_peered":"2024-03-24T00:32:57.662868+0000","last_clean":"2024-03-24T00:32:57.662868+0000","last_became_active":"2024-03-24T00:32:57.654772+0000","last_became_peered":"2024-03-24T00:32:57.654772+0000","last_unstale":"2024-03-24T00:32:57.662868+0000","last_undegraded":"2024-03-24T00:32:57.662868+0000","last_fullsized":"2024-03-24T00:32:57.662868+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:25:42.325973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.648465+0000","last_change":"2024-03-24T00:32:57.645048+0000","last_active":"2024-03-24T00:32:57.648465+0000","last_peered":"2024-03-24T00:32:57.648465+0000","last_clean":"2024-03-24T00:32:57.648465+0000","last_became_active":"2024-03-24T00:32:57.644749+0000","last_became_peered":"2024-03-24T00:32:57.644749+0000","last_unstale":"2024-03-24T00:32:57.648465+0000","last_undegraded":"2024-03-24T00:32:57.648465+0000","last_fullsized":"2024-03-24T00:32:57.648465+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:13:36.160634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.658651+0000","last_change":"2024-03-24T00:32:57.654420+0000","last_active":"2024-03-24T00:32:57.658651+0000","last_peered":"2024-03-24T00:32:57.658651+0000","last_clean":"2024-03-24T00:32:57.658651+0000","last_became_active":"2024-03-24T00:32:57.654050+0000","last_became_peered":"2024-03-24T00:32:57.654050+0000","last_unstale":"2024-03-24T00:32:57.658651+0000","last_undegraded":"2024-03-24T00:32:57.658651+0000","last_fullsized":"2024-03-24T00:32:57.658651+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:59:45.475352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"43'4","reported_seq":16,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.675870+0000","last_change":"2024-03-24T00:32:56.641164+0000","last_active":"2024-03-24T00:32:57.675870+0000","last_peered":"2024-03-24T00:32:57.675870+0000","last_clean":"2024-03-24T00:32:57.675870+0000","last_became_active":"2024-03-24T00:32:56.640607+0000","last_became_peered":"2024-03-24T00:32:56.640607+0000","last_unstale":"2024-03-24T00:32:57.675870+0000","last_undegraded":"2024-03-24T00:32:57.675870+0000","last_fullsized":"2024-03-24T00:32:57.675870+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:23:40.773929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.660996+0000","last_change":"2024-03-24T00:32:57.658865+0000","last_active":"2024-03-24T00:32:57.660996+0000","last_peered":"2024-03-24T00:32:57.660996+0000","last_clean":"2024-03-24T00:32:57.660996+0000","last_became_active":"2024-03-24T00:32:57.656527+0000","last_became_peered":"2024-03-24T00:32:57.656527+0000","last_unstale":"2024-03-24T00:32:57.660996+0000","last_undegraded":"2024-03-24T00:32:57.660996+0000","last_fullsized":"2024-03-24T00:32:57.660996+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:02:03.160871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.666900+0000","last_change":"2024-03-24T00:32:56.644143+0000","last_active":"2024-03-24T00:32:57.666900+0000","last_peered":"2024-03-24T00:32:57.666900+0000","last_clean":"2024-03-24T00:32:57.666900+0000","last_became_active":"2024-03-24T00:32:56.643856+0000","last_became_peered":"2024-03-24T00:32:56.643856+0000","last_unstale":"2024-03-24T00:32:57.666900+0000","last_undegraded":"2024-03-24T00:32:57.666900+0000","last_fullsized":"2024-03-24T00:32:57.666900+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:07.226658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.659224+0000","last_change":"2024-03-24T00:32:57.654822+0000","last_active":"2024-03-24T00:32:57.659224+0000","last_peered":"2024-03-24T00:32:57.659224+0000","last_clean":"2024-03-24T00:32:57.659224+0000","last_became_active":"2024-03-24T00:32:57.654519+0000","last_became_peered":"2024-03-24T00:32:57.654519+0000","last_unstale":"2024-03-24T00:32:57.659224+0000","last_undegraded":"2024-03-24T00:32:57.659224+0000","last_fullsized":"2024-03-24T00:32:57.659224+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:19.011696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.674076+0000","last_change":"2024-03-24T00:32:56.636221+0000","last_active":"2024-03-24T00:32:57.674076+0000","last_peered":"2024-03-24T00:32:57.674076+0000","last_clean":"2024-03-24T00:32:57.674076+0000","last_became_active":"2024-03-24T00:32:56.635925+0000","last_became_peered":"2024-03-24T00:32:56.635925+0000","last_unstale":"2024-03-24T00:32:57.674076+0000","last_undegraded":"2024-03-24T00:32:57.674076+0000","last_fullsized":"2024-03-24T00:32:57.674076+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:07.226658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.663416+0000","last_change":"2024-03-24T00:32:57.656393+0000","last_active":"2024-03-24T00:32:57.663416+0000","last_peered":"2024-03-24T00:32:57.663416+0000","last_clean":"2024-03-24T00:32:57.663416+0000","last_became_active":"2024-03-24T00:32:57.656088+0000","last_became_peered":"2024-03-24T00:32:57.656088+0000","last_unstale":"2024-03-24T00:32:57.663416+0000","last_undegraded":"2024-03-24T00:32:57.663416+0000","last_fullsized":"2024-03-24T00:32:57.663416+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:26:03.923852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.668642+0000","last_change":"2024-03-24T00:32:56.649789+0000","last_active":"2024-03-24T00:32:57.668642+0000","last_peered":"2024-03-24T00:32:57.668642+0000","last_clean":"2024-03-24T00:32:57.668642+0000","last_became_active":"2024-03-24T00:32:56.646468+0000","last_became_peered":"2024-03-24T00:32:56.646468+0000","last_unstale":"2024-03-24T00:32:57.668642+0000","last_undegraded":"2024-03-24T00:32:57.668642+0000","last_fullsized":"2024-03-24T00:32:57.668642+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:50:27.055732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.659097+0000","last_change":"2024-03-24T00:32:57.657752+0000","last_active":"2024-03-24T00:32:57.659097+0000","last_peered":"2024-03-24T00:32:57.659097+0000","last_clean":"2024-03-24T00:32:57.659097+0000","last_became_active":"2024-03-24T00:32:57.657393+0000","last_became_peered":"2024-03-24T00:32:57.657393+0000","last_unstale":"2024-03-24T00:32:57.659097+0000","last_undegraded":"2024-03-24T00:32:57.659097+0000","last_fullsized":"2024-03-24T00:32:57.659097+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:50:28.064185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.674954+0000","last_change":"2024-03-24T00:32:56.642339+0000","last_active":"2024-03-24T00:32:57.674954+0000","last_peered":"2024-03-24T00:32:57.674954+0000","last_clean":"2024-03-24T00:32:57.674954+0000","last_became_active":"2024-03-24T00:32:56.641854+0000","last_became_peered":"2024-03-24T00:32:56.641854+0000","last_unstale":"2024-03-24T00:32:57.674954+0000","last_undegraded":"2024-03-24T00:32:57.674954+0000","last_fullsized":"2024-03-24T00:32:57.674954+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:21.922712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.671893+0000","last_change":"2024-03-24T00:32:57.661384+0000","last_active":"2024-03-24T00:32:57.671893+0000","last_peered":"2024-03-24T00:32:57.671893+0000","last_clean":"2024-03-24T00:32:57.671893+0000","last_became_active":"2024-03-24T00:32:57.661027+0000","last_became_peered":"2024-03-24T00:32:57.661027+0000","last_unstale":"2024-03-24T00:32:57.671893+0000","last_undegraded":"2024-03-24T00:32:57.671893+0000","last_fullsized":"2024-03-24T00:32:57.671893+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:33.356685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.667640+0000","last_change":"2024-03-24T00:32:56.645601+0000","last_active":"2024-03-24T00:32:57.667640+0000","last_peered":"2024-03-24T00:32:57.667640+0000","last_clean":"2024-03-24T00:32:57.667640+0000","last_became_active":"2024-03-24T00:32:56.645202+0000","last_became_peered":"2024-03-24T00:32:56.645202+0000","last_unstale":"2024-03-24T00:32:57.667640+0000","last_undegraded":"2024-03-24T00:32:57.667640+0000","last_fullsized":"2024-03-24T00:32:57.667640+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:07.913989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.659089+0000","last_change":"2024-03-24T00:32:57.655853+0000","last_active":"2024-03-24T00:32:57.659089+0000","last_peered":"2024-03-24T00:32:57.659089+0000","last_clean":"2024-03-24T00:32:57.659089+0000","last_became_active":"2024-03-24T00:32:57.655584+0000","last_became_peered":"2024-03-24T00:32:57.655584+0000","last_unstale":"2024-03-24T00:32:57.659089+0000","last_undegraded":"2024-03-24T00:32:57.659089+0000","last_fullsized":"2024-03-24T00:32:57.659089+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:01.187105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.645600+0000","last_change":"2024-03-24T00:32:56.638440+0000","last_active":"2024-03-24T00:32:57.645600+0000","last_peered":"2024-03-24T00:32:57.645600+0000","last_clean":"2024-03-24T00:32:57.645600+0000","last_became_active":"2024-03-24T00:32:56.638066+0000","last_became_peered":"2024-03-24T00:32:56.638066+0000","last_unstale":"2024-03-24T00:32:57.645600+0000","last_undegraded":"2024-03-24T00:32:57.645600+0000","last_fullsized":"2024-03-24T00:32:57.645600+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:55:11.522698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.653453+0000","last_change":"2024-03-24T00:32:57.651737+0000","last_active":"2024-03-24T00:32:57.653453+0000","last_peered":"2024-03-24T00:32:57.653453+0000","last_clean":"2024-03-24T00:32:57.653453+0000","last_became_active":"2024-03-24T00:32:57.651343+0000","last_became_peered":"2024-03-24T00:32:57.651343+0000","last_unstale":"2024-03-24T00:32:57.653453+0000","last_undegraded":"2024-03-24T00:32:57.653453+0000","last_fullsized":"2024-03-24T00:32:57.653453+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:03:02.649025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.636951+0000","last_change":"2024-03-24T00:32:56.650015+0000","last_active":"2024-03-24T00:32:57.636951+0000","last_peered":"2024-03-24T00:32:57.636951+0000","last_clean":"2024-03-24T00:32:57.636951+0000","last_became_active":"2024-03-24T00:32:56.648189+0000","last_became_peered":"2024-03-24T00:32:56.648189+0000","last_unstale":"2024-03-24T00:32:57.636951+0000","last_undegraded":"2024-03-24T00:32:57.636951+0000","last_fullsized":"2024-03-24T00:32:57.636951+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:13.894010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.649784+0000","last_change":"2024-03-24T00:32:57.647415+0000","last_active":"2024-03-24T00:32:57.649784+0000","last_peered":"2024-03-24T00:32:57.649784+0000","last_clean":"2024-03-24T00:32:57.649784+0000","last_became_active":"2024-03-24T00:32:57.647119+0000","last_became_peered":"2024-03-24T00:32:57.647119+0000","last_unstale":"2024-03-24T00:32:57.649784+0000","last_undegraded":"2024-03-24T00:32:57.649784+0000","last_fullsized":"2024-03-24T00:32:57.649784+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:00.542240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.635512+0000","last_change":"2024-03-24T00:32:56.652751+0000","last_active":"2024-03-24T00:32:57.635512+0000","last_peered":"2024-03-24T00:32:57.635512+0000","last_clean":"2024-03-24T00:32:57.635512+0000","last_became_active":"2024-03-24T00:32:56.651848+0000","last_became_peered":"2024-03-24T00:32:56.651848+0000","last_unstale":"2024-03-24T00:32:57.635512+0000","last_undegraded":"2024-03-24T00:32:57.635512+0000","last_fullsized":"2024-03-24T00:32:57.635512+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:42:44.393628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.650894+0000","last_change":"2024-03-24T00:32:57.648136+0000","last_active":"2024-03-24T00:32:57.650894+0000","last_peered":"2024-03-24T00:32:57.650894+0000","last_clean":"2024-03-24T00:32:57.650894+0000","last_became_active":"2024-03-24T00:32:57.647750+0000","last_became_peered":"2024-03-24T00:32:57.647750+0000","last_unstale":"2024-03-24T00:32:57.650894+0000","last_undegraded":"2024-03-24T00:32:57.650894+0000","last_fullsized":"2024-03-24T00:32:57.650894+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:07.244689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.641330+0000","last_change":"2024-03-24T00:32:56.645946+0000","last_active":"2024-03-24T00:32:57.641330+0000","last_peered":"2024-03-24T00:32:57.641330+0000","last_clean":"2024-03-24T00:32:57.641330+0000","last_became_active":"2024-03-24T00:32:56.645628+0000","last_became_peered":"2024-03-24T00:32:56.645628+0000","last_unstale":"2024-03-24T00:32:57.641330+0000","last_undegraded":"2024-03-24T00:32:57.641330+0000","last_fullsized":"2024-03-24T00:32:57.641330+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:06.226473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.673723+0000","last_change":"2024-03-24T00:32:57.669261+0000","last_active":"2024-03-24T00:32:57.673723+0000","last_peered":"2024-03-24T00:32:57.673723+0000","last_clean":"2024-03-24T00:32:57.673723+0000","last_became_active":"2024-03-24T00:32:57.668969+0000","last_became_peered":"2024-03-24T00:32:57.668969+0000","last_unstale":"2024-03-24T00:32:57.673723+0000","last_undegraded":"2024-03-24T00:32:57.673723+0000","last_fullsized":"2024-03-24T00:32:57.673723+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:53.944849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"43'3","reported_seq":15,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.669107+0000","last_change":"2024-03-24T00:32:56.647429+0000","last_active":"2024-03-24T00:32:57.669107+0000","last_peered":"2024-03-24T00:32:57.669107+0000","last_clean":"2024-03-24T00:32:57.669107+0000","last_became_active":"2024-03-24T00:32:56.646236+0000","last_became_peered":"2024-03-24T00:32:56.646236+0000","last_unstale":"2024-03-24T00:32:57.669107+0000","last_undegraded":"2024-03-24T00:32:57.669107+0000","last_fullsized":"2024-03-24T00:32:57.669107+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:57:28.581587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.645907+0000","last_change":"2024-03-24T00:32:57.641810+0000","last_active":"2024-03-24T00:32:57.645907+0000","last_peered":"2024-03-24T00:32:57.645907+0000","last_clean":"2024-03-24T00:32:57.645907+0000","last_became_active":"2024-03-24T00:32:57.641364+0000","last_became_peered":"2024-03-24T00:32:57.641364+0000","last_unstale":"2024-03-24T00:32:57.645907+0000","last_undegraded":"2024-03-24T00:32:57.645907+0000","last_fullsized":"2024-03-24T00:32:57.645907+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:19.011696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"43'6","reported_seq":18,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.677534+0000","last_change":"2024-03-24T00:32:56.647948+0000","last_active":"2024-03-24T00:32:57.677534+0000","last_peered":"2024-03-24T00:32:57.677534+0000","last_clean":"2024-03-24T00:32:57.677534+0000","last_became_active":"2024-03-24T00:32:56.647533+0000","last_became_peered":"2024-03-24T00:32:56.647533+0000","last_unstale":"2024-03-24T00:32:57.677534+0000","last_undegraded":"2024-03-24T00:32:57.677534+0000","last_fullsized":"2024-03-24T00:32:57.677534+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:17:40.522483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.666800+0000","last_change":"2024-03-24T00:32:57.664651+0000","last_active":"2024-03-24T00:32:57.666800+0000","last_peered":"2024-03-24T00:32:57.666800+0000","last_clean":"2024-03-24T00:32:57.666800+0000","last_became_active":"2024-03-24T00:32:57.664307+0000","last_became_peered":"2024-03-24T00:32:57.664307+0000","last_unstale":"2024-03-24T00:32:57.666800+0000","last_undegraded":"2024-03-24T00:32:57.666800+0000","last_fullsized":"2024-03-24T00:32:57.666800+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:22:12.842264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.673923+0000","last_change":"2024-03-24T00:32:56.642994+0000","last_active":"2024-03-24T00:32:57.673923+0000","last_peered":"2024-03-24T00:32:57.673923+0000","last_clean":"2024-03-24T00:32:57.673923+0000","last_became_active":"2024-03-24T00:32:56.642702+0000","last_became_peered":"2024-03-24T00:32:56.642702+0000","last_unstale":"2024-03-24T00:32:57.673923+0000","last_undegraded":"2024-03-24T00:32:57.673923+0000","last_fullsized":"2024-03-24T00:32:57.673923+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:24.547880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.657421+0000","last_change":"2024-03-24T00:32:57.650284+0000","last_active":"2024-03-24T00:32:57.657421+0000","last_peered":"2024-03-24T00:32:57.657421+0000","last_clean":"2024-03-24T00:32:57.657421+0000","last_became_active":"2024-03-24T00:32:57.650030+0000","last_became_peered":"2024-03-24T00:32:57.650030+0000","last_unstale":"2024-03-24T00:32:57.657421+0000","last_undegraded":"2024-03-24T00:32:57.657421+0000","last_fullsized":"2024-03-24T00:32:57.657421+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:17:49.038615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.633573+0000","last_change":"2024-03-24T00:32:56.651558+0000","last_active":"2024-03-24T00:32:57.633573+0000","last_peered":"2024-03-24T00:32:57.633573+0000","last_clean":"2024-03-24T00:32:57.633573+0000","last_became_active":"2024-03-24T00:32:56.651271+0000","last_became_peered":"2024-03-24T00:32:56.651271+0000","last_unstale":"2024-03-24T00:32:57.633573+0000","last_undegraded":"2024-03-24T00:32:57.633573+0000","last_fullsized":"2024-03-24T00:32:57.633573+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:59.451437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.656351+0000","last_change":"2024-03-24T00:32:57.653532+0000","last_active":"2024-03-24T00:32:57.656351+0000","last_peered":"2024-03-24T00:32:57.656351+0000","last_clean":"2024-03-24T00:32:57.656351+0000","last_became_active":"2024-03-24T00:32:57.653107+0000","last_became_peered":"2024-03-24T00:32:57.653107+0000","last_unstale":"2024-03-24T00:32:57.656351+0000","last_undegraded":"2024-03-24T00:32:57.656351+0000","last_fullsized":"2024-03-24T00:32:57.656351+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:33.312475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.644043+0000","last_change":"2024-03-24T00:32:56.649216+0000","last_active":"2024-03-24T00:32:57.644043+0000","last_peered":"2024-03-24T00:32:57.644043+0000","last_clean":"2024-03-24T00:32:57.644043+0000","last_became_active":"2024-03-24T00:32:56.648949+0000","last_became_peered":"2024-03-24T00:32:56.648949+0000","last_unstale":"2024-03-24T00:32:57.644043+0000","last_undegraded":"2024-03-24T00:32:57.644043+0000","last_fullsized":"2024-03-24T00:32:57.644043+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:18.003243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.658198+0000","last_change":"2024-03-24T00:32:57.655907+0000","last_active":"2024-03-24T00:32:57.658198+0000","last_peered":"2024-03-24T00:32:57.658198+0000","last_clean":"2024-03-24T00:32:57.658198+0000","last_became_active":"2024-03-24T00:32:57.655342+0000","last_became_peered":"2024-03-24T00:32:57.655342+0000","last_unstale":"2024-03-24T00:32:57.658198+0000","last_undegraded":"2024-03-24T00:32:57.658198+0000","last_fullsized":"2024-03-24T00:32:57.658198+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:34:25.556333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"43'8","reported_seq":20,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.673861+0000","last_change":"2024-03-24T00:32:56.649467+0000","last_active":"2024-03-24T00:32:57.673861+0000","last_peered":"2024-03-24T00:32:57.673861+0000","last_clean":"2024-03-24T00:32:57.673861+0000","last_became_active":"2024-03-24T00:32:56.649196+0000","last_became_peered":"2024-03-24T00:32:56.649196+0000","last_unstale":"2024-03-24T00:32:57.673861+0000","last_undegraded":"2024-03-24T00:32:57.673861+0000","last_fullsized":"2024-03-24T00:32:57.673861+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:10:32.939784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.658231+0000","last_change":"2024-03-24T00:32:57.656170+0000","last_active":"2024-03-24T00:32:57.658231+0000","last_peered":"2024-03-24T00:32:57.658231+0000","last_clean":"2024-03-24T00:32:57.658231+0000","last_became_active":"2024-03-24T00:32:57.655896+0000","last_became_peered":"2024-03-24T00:32:57.655896+0000","last_unstale":"2024-03-24T00:32:57.658231+0000","last_undegraded":"2024-03-24T00:32:57.658231+0000","last_fullsized":"2024-03-24T00:32:57.658231+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:02:03.160871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.676474+0000","last_change":"2024-03-24T00:32:56.646580+0000","last_active":"2024-03-24T00:32:57.676474+0000","last_peered":"2024-03-24T00:32:57.676474+0000","last_clean":"2024-03-24T00:32:57.676474+0000","last_became_active":"2024-03-24T00:32:56.646267+0000","last_became_peered":"2024-03-24T00:32:56.646267+0000","last_unstale":"2024-03-24T00:32:57.676474+0000","last_undegraded":"2024-03-24T00:32:57.676474+0000","last_fullsized":"2024-03-24T00:32:57.676474+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:40:51.841137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.655924+0000","last_change":"2024-03-24T00:32:57.646920+0000","last_active":"2024-03-24T00:32:57.655924+0000","last_peered":"2024-03-24T00:32:57.655924+0000","last_clean":"2024-03-24T00:32:57.655924+0000","last_became_active":"2024-03-24T00:32:57.646583+0000","last_became_peered":"2024-03-24T00:32:57.646583+0000","last_unstale":"2024-03-24T00:32:57.655924+0000","last_undegraded":"2024-03-24T00:32:57.655924+0000","last_fullsized":"2024-03-24T00:32:57.655924+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:42.847452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.673662+0000","last_change":"2024-03-24T00:32:56.641393+0000","last_active":"2024-03-24T00:32:57.673662+0000","last_peered":"2024-03-24T00:32:57.673662+0000","last_clean":"2024-03-24T00:32:57.673662+0000","last_became_active":"2024-03-24T00:32:56.641055+0000","last_became_peered":"2024-03-24T00:32:56.641055+0000","last_unstale":"2024-03-24T00:32:57.673662+0000","last_undegraded":"2024-03-24T00:32:57.673662+0000","last_fullsized":"2024-03-24T00:32:57.673662+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:00.178652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.660349+0000","last_change":"2024-03-24T00:32:57.657007+0000","last_active":"2024-03-24T00:32:57.660349+0000","last_peered":"2024-03-24T00:32:57.660349+0000","last_clean":"2024-03-24T00:32:57.660349+0000","last_became_active":"2024-03-24T00:32:57.656716+0000","last_became_peered":"2024-03-24T00:32:57.656716+0000","last_unstale":"2024-03-24T00:32:57.660349+0000","last_undegraded":"2024-03-24T00:32:57.660349+0000","last_fullsized":"2024-03-24T00:32:57.660349+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:54.726464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"43'1","reported_seq":13,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.675504+0000","last_change":"2024-03-24T00:32:56.637709+0000","last_active":"2024-03-24T00:32:57.675504+0000","last_peered":"2024-03-24T00:32:57.675504+0000","last_clean":"2024-03-24T00:32:57.675504+0000","last_became_active":"2024-03-24T00:32:56.637386+0000","last_became_peered":"2024-03-24T00:32:56.637386+0000","last_unstale":"2024-03-24T00:32:57.675504+0000","last_undegraded":"2024-03-24T00:32:57.675504+0000","last_fullsized":"2024-03-24T00:32:57.675504+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:59.451437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.666832+0000","last_change":"2024-03-24T00:32:57.664680+0000","last_active":"2024-03-24T00:32:57.666832+0000","last_peered":"2024-03-24T00:32:57.666832+0000","last_clean":"2024-03-24T00:32:57.666832+0000","last_became_active":"2024-03-24T00:32:57.663919+0000","last_became_peered":"2024-03-24T00:32:57.663919+0000","last_unstale":"2024-03-24T00:32:57.666832+0000","last_undegraded":"2024-03-24T00:32:57.666832+0000","last_fullsized":"2024-03-24T00:32:57.666832+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:14.902463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.644792+0000","last_change":"2024-03-24T00:32:56.651295+0000","last_active":"2024-03-24T00:32:57.644792+0000","last_peered":"2024-03-24T00:32:57.644792+0000","last_clean":"2024-03-24T00:32:57.644792+0000","last_became_active":"2024-03-24T00:32:56.647097+0000","last_became_peered":"2024-03-24T00:32:56.647097+0000","last_unstale":"2024-03-24T00:32:57.644792+0000","last_undegraded":"2024-03-24T00:32:57.644792+0000","last_fullsized":"2024-03-24T00:32:57.644792+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:06.236237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.672324+0000","last_change":"2024-03-24T00:32:57.667931+0000","last_active":"2024-03-24T00:32:57.672324+0000","last_peered":"2024-03-24T00:32:57.672324+0000","last_clean":"2024-03-24T00:32:57.672324+0000","last_became_active":"2024-03-24T00:32:57.667640+0000","last_became_peered":"2024-03-24T00:32:57.667640+0000","last_unstale":"2024-03-24T00:32:57.672324+0000","last_undegraded":"2024-03-24T00:32:57.672324+0000","last_fullsized":"2024-03-24T00:32:57.672324+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:54:45.618722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.676277+0000","last_change":"2024-03-24T00:32:56.642107+0000","last_active":"2024-03-24T00:32:57.676277+0000","last_peered":"2024-03-24T00:32:57.676277+0000","last_clean":"2024-03-24T00:32:57.676277+0000","last_became_active":"2024-03-24T00:32:56.641494+0000","last_became_peered":"2024-03-24T00:32:56.641494+0000","last_unstale":"2024-03-24T00:32:57.676277+0000","last_undegraded":"2024-03-24T00:32:57.676277+0000","last_fullsized":"2024-03-24T00:32:57.676277+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:56.812668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.661415+0000","last_change":"2024-03-24T00:32:57.651891+0000","last_active":"2024-03-24T00:32:57.661415+0000","last_peered":"2024-03-24T00:32:57.661415+0000","last_clean":"2024-03-24T00:32:57.661415+0000","last_became_active":"2024-03-24T00:32:57.651599+0000","last_became_peered":"2024-03-24T00:32:57.651599+0000","last_unstale":"2024-03-24T00:32:57.661415+0000","last_undegraded":"2024-03-24T00:32:57.661415+0000","last_fullsized":"2024-03-24T00:32:57.661415+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:42:45.402081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.647390+0000","last_change":"2024-03-24T00:32:56.645211+0000","last_active":"2024-03-24T00:32:57.647390+0000","last_peered":"2024-03-24T00:32:57.647390+0000","last_clean":"2024-03-24T00:32:57.647390+0000","last_became_active":"2024-03-24T00:32:56.644497+0000","last_became_peered":"2024-03-24T00:32:56.644497+0000","last_unstale":"2024-03-24T00:32:57.647390+0000","last_undegraded":"2024-03-24T00:32:57.647390+0000","last_fullsized":"2024-03-24T00:32:57.647390+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:50:27.055732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.644578+0000","last_change":"2024-03-24T00:32:57.639445+0000","last_active":"2024-03-24T00:32:57.644578+0000","last_peered":"2024-03-24T00:32:57.644578+0000","last_clean":"2024-03-24T00:32:57.644578+0000","last_became_active":"2024-03-24T00:32:57.639154+0000","last_became_peered":"2024-03-24T00:32:57.639154+0000","last_unstale":"2024-03-24T00:32:57.644578+0000","last_undegraded":"2024-03-24T00:32:57.644578+0000","last_fullsized":"2024-03-24T00:32:57.644578+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:45.533368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.643489+0000","last_change":"2024-03-24T00:32:56.653106+0000","last_active":"2024-03-24T00:32:57.643489+0000","last_peered":"2024-03-24T00:32:57.643489+0000","last_clean":"2024-03-24T00:32:57.643489+0000","last_became_active":"2024-03-24T00:32:56.652932+0000","last_became_peered":"2024-03-24T00:32:56.652932+0000","last_unstale":"2024-03-24T00:32:57.643489+0000","last_undegraded":"2024-03-24T00:32:57.643489+0000","last_fullsized":"2024-03-24T00:32:57.643489+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:51:31.302381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.659279+0000","last_change":"2024-03-24T00:32:57.647770+0000","last_active":"2024-03-24T00:32:57.659279+0000","last_peered":"2024-03-24T00:32:57.659279+0000","last_clean":"2024-03-24T00:32:57.659279+0000","last_became_active":"2024-03-24T00:32:57.647481+0000","last_became_peered":"2024-03-24T00:32:57.647481+0000","last_unstale":"2024-03-24T00:32:57.659279+0000","last_undegraded":"2024-03-24T00:32:57.659279+0000","last_fullsized":"2024-03-24T00:32:57.659279+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:08.922442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.632533+0000","last_change":"2024-03-24T00:32:56.647756+0000","last_active":"2024-03-24T00:32:57.632533+0000","last_peered":"2024-03-24T00:32:57.632533+0000","last_clean":"2024-03-24T00:32:57.632533+0000","last_became_active":"2024-03-24T00:32:56.647349+0000","last_became_peered":"2024-03-24T00:32:56.647349+0000","last_unstale":"2024-03-24T00:32:57.632533+0000","last_undegraded":"2024-03-24T00:32:57.632533+0000","last_fullsized":"2024-03-24T00:32:57.632533+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:55:11.522698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.666732+0000","last_change":"2024-03-24T00:32:57.664624+0000","last_active":"2024-03-24T00:32:57.666732+0000","last_peered":"2024-03-24T00:32:57.666732+0000","last_clean":"2024-03-24T00:32:57.666732+0000","last_became_active":"2024-03-24T00:32:57.664190+0000","last_became_peered":"2024-03-24T00:32:57.664190+0000","last_unstale":"2024-03-24T00:32:57.666732+0000","last_undegraded":"2024-03-24T00:32:57.666732+0000","last_fullsized":"2024-03-24T00:32:57.666732+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:55:47.192626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.644964+0000","last_change":"2024-03-24T00:32:56.644674+0000","last_active":"2024-03-24T00:32:57.644964+0000","last_peered":"2024-03-24T00:32:57.644964+0000","last_clean":"2024-03-24T00:32:57.644964+0000","last_became_active":"2024-03-24T00:32:56.644372+0000","last_became_peered":"2024-03-24T00:32:56.644372+0000","last_unstale":"2024-03-24T00:32:57.644964+0000","last_undegraded":"2024-03-24T00:32:57.644964+0000","last_fullsized":"2024-03-24T00:32:57.644964+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:03.399082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.659125+0000","last_change":"2024-03-24T00:32:57.654013+0000","last_active":"2024-03-24T00:32:57.659125+0000","last_peered":"2024-03-24T00:32:57.659125+0000","last_clean":"2024-03-24T00:32:57.659125+0000","last_became_active":"2024-03-24T00:32:57.653703+0000","last_became_peered":"2024-03-24T00:32:57.653703+0000","last_unstale":"2024-03-24T00:32:57.659125+0000","last_undegraded":"2024-03-24T00:32:57.659125+0000","last_fullsized":"2024-03-24T00:32:57.659125+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:08.235111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.634219+0000","last_change":"2024-03-24T00:32:56.652565+0000","last_active":"2024-03-24T00:32:57.634219+0000","last_peered":"2024-03-24T00:32:57.634219+0000","last_clean":"2024-03-24T00:32:57.634219+0000","last_became_active":"2024-03-24T00:32:56.652288+0000","last_became_peered":"2024-03-24T00:32:56.652288+0000","last_unstale":"2024-03-24T00:32:57.634219+0000","last_undegraded":"2024-03-24T00:32:57.634219+0000","last_fullsized":"2024-03-24T00:32:57.634219+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:11.970453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.646855+0000","last_change":"2024-03-24T00:32:57.644718+0000","last_active":"2024-03-24T00:32:57.646855+0000","last_peered":"2024-03-24T00:32:57.646855+0000","last_clean":"2024-03-24T00:32:57.646855+0000","last_became_active":"2024-03-24T00:32:57.644469+0000","last_became_peered":"2024-03-24T00:32:57.644469+0000","last_unstale":"2024-03-24T00:32:57.646855+0000","last_undegraded":"2024-03-24T00:32:57.646855+0000","last_fullsized":"2024-03-24T00:32:57.646855+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:04:22.931165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.632095+0000","last_change":"2024-03-24T00:32:56.635338+0000","last_active":"2024-03-24T00:32:57.632095+0000","last_peered":"2024-03-24T00:32:57.632095+0000","last_clean":"2024-03-24T00:32:57.632095+0000","last_became_active":"2024-03-24T00:32:56.635087+0000","last_became_peered":"2024-03-24T00:32:56.635087+0000","last_unstale":"2024-03-24T00:32:57.632095+0000","last_undegraded":"2024-03-24T00:32:57.632095+0000","last_fullsized":"2024-03-24T00:32:57.632095+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:17:48.030162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.654030+0000","last_change":"2024-03-24T00:32:57.646622+0000","last_active":"2024-03-24T00:32:57.654030+0000","last_peered":"2024-03-24T00:32:57.654030+0000","last_clean":"2024-03-24T00:32:57.654030+0000","last_became_active":"2024-03-24T00:32:57.646197+0000","last_became_peered":"2024-03-24T00:32:57.646197+0000","last_unstale":"2024-03-24T00:32:57.654030+0000","last_undegraded":"2024-03-24T00:32:57.654030+0000","last_fullsized":"2024-03-24T00:32:57.654030+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:28:22.693763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.632991+0000","last_change":"2024-03-24T00:32:56.648030+0000","last_active":"2024-03-24T00:32:57.632991+0000","last_peered":"2024-03-24T00:32:57.632991+0000","last_clean":"2024-03-24T00:32:57.632991+0000","last_became_active":"2024-03-24T00:32:56.647557+0000","last_became_peered":"2024-03-24T00:32:56.647557+0000","last_unstale":"2024-03-24T00:32:57.632991+0000","last_undegraded":"2024-03-24T00:32:57.632991+0000","last_fullsized":"2024-03-24T00:32:57.632991+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:32.348232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.671352+0000","last_change":"2024-03-24T00:32:57.669507+0000","last_active":"2024-03-24T00:32:57.671352+0000","last_peered":"2024-03-24T00:32:57.671352+0000","last_clean":"2024-03-24T00:32:57.671352+0000","last_became_active":"2024-03-24T00:32:57.669120+0000","last_became_peered":"2024-03-24T00:32:57.669120+0000","last_unstale":"2024-03-24T00:32:57.671352+0000","last_undegraded":"2024-03-24T00:32:57.671352+0000","last_fullsized":"2024-03-24T00:32:57.671352+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:03.270143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.634327+0000","last_change":"2024-03-24T00:32:56.649287+0000","last_active":"2024-03-24T00:32:57.634327+0000","last_peered":"2024-03-24T00:32:57.634327+0000","last_clean":"2024-03-24T00:32:57.634327+0000","last_became_active":"2024-03-24T00:32:56.648928+0000","last_became_peered":"2024-03-24T00:32:56.648928+0000","last_unstale":"2024-03-24T00:32:57.634327+0000","last_undegraded":"2024-03-24T00:32:57.634327+0000","last_fullsized":"2024-03-24T00:32:57.634327+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:17:48.030162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.648101+0000","last_change":"2024-03-24T00:32:57.645819+0000","last_active":"2024-03-24T00:32:57.648101+0000","last_peered":"2024-03-24T00:32:57.648101+0000","last_clean":"2024-03-24T00:32:57.648101+0000","last_became_active":"2024-03-24T00:32:57.643963+0000","last_became_peered":"2024-03-24T00:32:57.643963+0000","last_unstale":"2024-03-24T00:32:57.648101+0000","last_undegraded":"2024-03-24T00:32:57.648101+0000","last_fullsized":"2024-03-24T00:32:57.648101+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:33.356685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.632524+0000","last_change":"2024-03-24T00:32:56.644266+0000","last_active":"2024-03-24T00:32:57.632524+0000","last_peered":"2024-03-24T00:32:57.632524+0000","last_clean":"2024-03-24T00:32:57.632524+0000","last_became_active":"2024-03-24T00:32:56.643959+0000","last_became_peered":"2024-03-24T00:32:56.643959+0000","last_unstale":"2024-03-24T00:32:57.632524+0000","last_undegraded":"2024-03-24T00:32:57.632524+0000","last_fullsized":"2024-03-24T00:32:57.632524+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:00.178652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.656129+0000","last_change":"2024-03-24T00:32:57.654428+0000","last_active":"2024-03-24T00:32:57.656129+0000","last_peered":"2024-03-24T00:32:57.656129+0000","last_clean":"2024-03-24T00:32:57.656129+0000","last_became_active":"2024-03-24T00:32:57.651529+0000","last_became_peered":"2024-03-24T00:32:57.651529+0000","last_unstale":"2024-03-24T00:32:57.656129+0000","last_undegraded":"2024-03-24T00:32:57.656129+0000","last_fullsized":"2024-03-24T00:32:57.656129+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:33.356685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.637316+0000","last_change":"2024-03-24T00:32:56.646192+0000","last_active":"2024-03-24T00:32:57.637316+0000","last_peered":"2024-03-24T00:32:57.637316+0000","last_clean":"2024-03-24T00:32:57.637316+0000","last_became_active":"2024-03-24T00:32:56.645876+0000","last_became_peered":"2024-03-24T00:32:56.645876+0000","last_unstale":"2024-03-24T00:32:57.637316+0000","last_undegraded":"2024-03-24T00:32:57.637316+0000","last_fullsized":"2024-03-24T00:32:57.637316+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:57:09.920115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.669747+0000","last_change":"2024-03-24T00:32:57.661552+0000","last_active":"2024-03-24T00:32:57.669747+0000","last_peered":"2024-03-24T00:32:57.669747+0000","last_clean":"2024-03-24T00:32:57.669747+0000","last_became_active":"2024-03-24T00:32:57.660943+0000","last_became_peered":"2024-03-24T00:32:57.660943+0000","last_unstale":"2024-03-24T00:32:57.669747+0000","last_undegraded":"2024-03-24T00:32:57.669747+0000","last_fullsized":"2024-03-24T00:32:57.669747+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":42,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:56.616214+0000","last_clean_scrub_stamp":"2024-03-24T00:32:56.616214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:39.125653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"43'2","reported_seq":14,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.666376+0000","last_change":"2024-03-24T00:32:56.648195+0000","last_active":"2024-03-24T00:32:57.666376+0000","last_peered":"2024-03-24T00:32:57.666376+0000","last_clean":"2024-03-24T00:32:57.666376+0000","last_became_active":"2024-03-24T00:32:56.647589+0000","last_became_peered":"2024-03-24T00:32:56.647589+0000","last_unstale":"2024-03-24T00:32:57.666376+0000","last_undegraded":"2024-03-24T00:32:57.666376+0000","last_fullsized":"2024-03-24T00:32:57.666376+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":41,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:32:55.607762+0000","last_clean_scrub_stamp":"2024-03-24T00:32:55.607762+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:57:28.581587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":217,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-03-24T00:32:57.632639+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:32:57.632639+0000","last_peered":"2024-03-24T00:32:57.632639+0000","last_clean":"2024-03-24T00:32:57.632639+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:32:57.632639+0000","last_undegraded":"2024-03-24T00:32:57.632639+0000","last_fullsized":"2024-03-24T00:32:57.632639+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1101824,"data_stored":794998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":205212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":507904,"data_stored":204610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":507904,"data_stored":204630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":516096,"data_stored":205178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":206174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":507904,"data_stored":204610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":788133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:33:02.183 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:33:02.183 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:33:02.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.364+0000 7f1170200640 1 Processor -- start 2024-03-24T00:33:02.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.364+0000 7f1170200640 1 -- start start 2024-03-24T00:33:02.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.364+0000 7f1170200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f11680cf4e0 0x7f116806dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:02.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.364+0000 7f1170200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11680b3380 0x7f1168070ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:02.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.364+0000 7f1170200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116815f2c0 con 0x7f1168072b60 2024-03-24T00:33:02.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.364+0000 7f1170200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f116806cfe0 con 0x7f11680cf4e0 2024-03-24T00:33:02.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.364+0000 7f1170200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116806d880 con 0x7f11680b3380 2024-03-24T00:33:02.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.364+0000 7f116d800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f11680cf4e0 0x7f116806dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:02.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.364+0000 7f116ce00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1168072b60 0x7f11680b2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38952/0 (socket says 172.21.15.40:38952) 2024-03-24T00:33:02.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.364+0000 7f116ce00640 1 -- 172.21.15.40:0/150712753 learned_addr learned my addr 172.21.15.40:0/150712753 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:02.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.364+0000 7f116e200640 1 --2- 172.21.15.40:0/150712753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11680b3380 0x7f1168070ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:02.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f116d800640 1 -- 172.21.15.40:0/150712753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11680b3380 msgr2=0x7f1168070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:02.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f116d800640 1 --2- 172.21.15.40:0/150712753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11680b3380 0x7f1168070ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f116d800640 1 -- 172.21.15.40:0/150712753 >> v1:172.21.15.40:6789/0 conn(0x7f1168072b60 legacy=0x7f11680b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f116d800640 1 -- 172.21.15.40:0/150712753 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116806d230 con 0x7f11680cf4e0 2024-03-24T00:33:02.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f116d800640 1 --2- 172.21.15.40:0/150712753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11680cf4e0 0x7f116806dd10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f115c0097c0 tx=0x7f115c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9696a0b09bf138e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:02.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f1167e00640 1 -- 172.21.15.40:0/150712753 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c0b5070 con 0x7f11680cf4e0 2024-03-24T00:33:02.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f1167e00640 1 -- 172.21.15.40:0/150712753 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115c0a8990 con 0x7f11680cf4e0 2024-03-24T00:33:02.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f1167e00640 1 -- 172.21.15.40:0/150712753 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c0a8d60 con 0x7f11680cf4e0 2024-03-24T00:33:02.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f1170200640 1 -- 172.21.15.40:0/150712753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11680cf4e0 msgr2=0x7f116806dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f1170200640 1 --2- 172.21.15.40:0/150712753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11680cf4e0 0x7f116806dd10 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f115c0097c0 tx=0x7f115c01ec90 comp rx=0 tx=0).stop 2024-03-24T00:33:02.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f1170200640 1 -- 172.21.15.40:0/150712753 shutdown_connections 2024-03-24T00:33:02.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f1170200640 1 --2- 172.21.15.40:0/150712753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11680b3380 0x7f1168070ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f1170200640 1 --2- 172.21.15.40:0/150712753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f11680cf4e0 0x7f116806dd10 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f1170200640 1 -- 172.21.15.40:0/150712753 >> 172.21.15.40:0/150712753 conn(0x7f11680b1660 msgr2=0x7f11680b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:02.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f1170200640 1 -- 172.21.15.40:0/150712753 shutdown_connections 2024-03-24T00:33:02.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.368+0000 7f1170200640 1 -- 172.21.15.40:0/150712753 wait complete. 2024-03-24T00:33:02.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f1170200640 1 Processor -- start 2024-03-24T00:33:02.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f1170200640 1 -- start start 2024-03-24T00:33:02.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f1170200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1168072b60 0x7f1168150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:02.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f1170200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11680b3380 0x7f11681fc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:02.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f1170200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11680cf4e0 0x7f11681f8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f1170200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116815ef20 con 0x7f1168072b60 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f1170200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116806e250 con 0x7f11680b3380 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f1170200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116815f0a0 con 0x7f11680cf4e0 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f116d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1168072b60 0x7f1168150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f116d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1168072b60 0x7f1168150700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49672/0 (socket says 172.21.15.40:49672) 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f116d800640 1 -- 172.21.15.40:0/1526841452 learned_addr learned my addr 172.21.15.40:0/1526841452 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f116ce00640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11680b3380 0x7f11681fc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f116e200640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11680cf4e0 0x7f11681f8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f116d800640 1 -- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11680cf4e0 msgr2=0x7f11681f8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f116d800640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11680cf4e0 0x7f11681f8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f116d800640 1 -- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11680b3380 msgr2=0x7f11681fc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f116d800640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11680b3380 0x7f11681fc410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f116d800640 1 -- 172.21.15.40:0/1526841452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11681fc950 con 0x7f1168072b60 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.372+0000 7f116d800640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1168072b60 0x7f1168150700 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f115c01c8c0 tx=0x7f115c01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.376+0000 7f1165600640 1 -- 172.21.15.40:0/1526841452 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c0bf070 con 0x7f1168072b60 2024-03-24T00:33:02.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.376+0000 7f1165600640 1 -- 172.21.15.40:0/1526841452 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f115c0b07a0 con 0x7f1168072b60 2024-03-24T00:33:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.376+0000 7f1170200640 1 -- 172.21.15.40:0/1526841452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11681f8e60 con 0x7f1168072b60 2024-03-24T00:33:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.376+0000 7f1165600640 1 -- 172.21.15.40:0/1526841452 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f115c0b5040 con 0x7f1168072b60 2024-03-24T00:33:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.376+0000 7f1170200640 1 -- 172.21.15.40:0/1526841452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11681fbc10 con 0x7f1168072b60 2024-03-24T00:33:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.376+0000 7f1170200640 1 -- 172.21.15.40:0/1526841452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1168068170 con 0x7f1168072b60 2024-03-24T00:33:02.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.376+0000 7f1165600640 1 -- 172.21.15.40:0/1526841452 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f115c0c2070 con 0x7f1168072b60 2024-03-24T00:33:02.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.376+0000 7f1165600640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1148067660 0x7f1148069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:02.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.376+0000 7f1165600640 1 -- 172.21.15.40:0/1526841452 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f115c121fa0 con 0x7f1168072b60 2024-03-24T00:33:02.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.376+0000 7f116ce00640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1148067660 0x7f1148069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:02.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.376+0000 7f116ce00640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1148067660 0x7f1148069b20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f11680b1330 tx=0x7f1154009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.380+0000 7f1165600640 1 -- 172.21.15.40:0/1526841452 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f11681fc950 con 0x7f1168072b60 2024-03-24T00:33:02.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.600+0000 7f1170200640 1 -- 172.21.15.40:0/1526841452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f11681f9380 con 0x7f1168072b60 2024-03-24T00:33:02.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.604+0000 7f1165600640 1 -- 172.21.15.40:0/1526841452 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f115c0187f0 con 0x7f1168072b60 2024-03-24T00:33:02.606 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-03-24T00:33:02.606 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw== 2024-03-24T00:33:02.606 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-03-24T00:33:02.606 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-03-24T00:33:02.607 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:33:02.607 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-03-24T00:33:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.604+0000 7f1170200640 1 -- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1148067660 msgr2=0x7f1148069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.604+0000 7f1170200640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1148067660 0x7f1148069b20 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f11680b1330 tx=0x7f1154009040 comp rx=0 tx=0).stop 2024-03-24T00:33:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.604+0000 7f1170200640 1 -- 172.21.15.40:0/1526841452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1168072b60 msgr2=0x7f1168150700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.604+0000 7f1170200640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1168072b60 0x7f1168150700 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f115c01c8c0 tx=0x7f115c01ba30 comp rx=0 tx=0).stop 2024-03-24T00:33:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.604+0000 7f1170200640 1 -- 172.21.15.40:0/1526841452 shutdown_connections 2024-03-24T00:33:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.604+0000 7f1170200640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1148067660 0x7f1148069b20 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.604+0000 7f1170200640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11680cf4e0 0x7f11681f8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.604+0000 7f1170200640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11680b3380 0x7f11681fc410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.604+0000 7f1170200640 1 --2- 172.21.15.40:0/1526841452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1168072b60 0x7f1168150700 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.604+0000 7f1170200640 1 -- 172.21.15.40:0/1526841452 >> 172.21.15.40:0/1526841452 conn(0x7f11680b1660 msgr2=0x7f11680ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.608+0000 7f1170200640 1 -- 172.21.15.40:0/1526841452 shutdown_connections 2024-03-24T00:33:02.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.608+0000 7f1170200640 1 -- 172.21.15.40:0/1526841452 wait complete. 2024-03-24T00:33:02.628 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:02.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.768+0000 7fe0a9200640 1 Processor -- start 2024-03-24T00:33:02.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.768+0000 7fe0a9200640 1 -- start start 2024-03-24T00:33:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.768+0000 7fe0a9200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0a40d0830 0x7fe0a40cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.768+0000 7fe0a9200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a40d0e10 con 0x7fe0a406b150 2024-03-24T00:33:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.768+0000 7fe0a9200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a406efd0 con 0x7fe0a415e0b0 2024-03-24T00:33:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.768+0000 7fe0a9200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0a40d10f0 con 0x7fe0a40d0830 2024-03-24T00:33:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.768+0000 7fe09be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0a40d0830 0x7fe0a40cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.768+0000 7fe0a2800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe0a406b150 0x7fe0a406b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38954/0 (socket says 172.21.15.40:38954) 2024-03-24T00:33:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.768+0000 7fe09be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe0a40d0830 0x7fe0a40cdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49676/0 (socket says 172.21.15.40:49676) 2024-03-24T00:33:02.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.768+0000 7fe0a2800640 1 -- 172.21.15.40:0/375465246 learned_addr learned my addr 172.21.15.40:0/375465246 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:02.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.772+0000 7fe09be00640 1 -- 172.21.15.40:0/375465246 >> v1:172.21.15.53:6789/0 conn(0x7fe0a415e0b0 legacy=0x7fe0a40cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.772+0000 7fe09be00640 1 -- 172.21.15.40:0/375465246 >> v1:172.21.15.40:6789/0 conn(0x7fe0a406b150 legacy=0x7fe0a406b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.772+0000 7fe09be00640 1 -- 172.21.15.40:0/375465246 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a406f2b0 con 0x7fe0a40d0830 2024-03-24T00:33:02.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.772+0000 7fe0a1400640 1 -- 172.21.15.40:0/375465246 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3776317324 0 0) 0x7fe0a40d0e10 con 0x7fe0a406b150 2024-03-24T00:33:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.772+0000 7fe09be00640 1 --2- 172.21.15.40:0/375465246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0a40d0830 0x7fe0a40cdae0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0097c0 tx=0x7fe08c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ebaa8a62643bdde server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:02.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.772+0000 7fe0a1400640 1 -- 172.21.15.40:0/375465246 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0b1070 con 0x7fe0a40d0830 2024-03-24T00:33:02.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.772+0000 7fe0a1400640 1 -- 172.21.15.40:0/375465246 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08c0ac410 con 0x7fe0a40d0830 2024-03-24T00:33:02.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.772+0000 7fe0a1400640 1 -- 172.21.15.40:0/375465246 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08c0ac7e0 con 0x7fe0a40d0830 2024-03-24T00:33:02.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 -- 172.21.15.40:0/375465246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0a40d0830 msgr2=0x7fe0a40cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 --2- 172.21.15.40:0/375465246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0a40d0830 0x7fe0a40cdae0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0097c0 tx=0x7fe08c01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 -- 172.21.15.40:0/375465246 shutdown_connections 2024-03-24T00:33:02.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 --2- 172.21.15.40:0/375465246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0a40d0830 0x7fe0a40cdae0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 -- 172.21.15.40:0/375465246 >> 172.21.15.40:0/375465246 conn(0x7fe0a40d1c90 msgr2=0x7fe0a40d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:02.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 -- 172.21.15.40:0/375465246 shutdown_connections 2024-03-24T00:33:02.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 -- 172.21.15.40:0/375465246 wait complete. 2024-03-24T00:33:02.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 Processor -- start 2024-03-24T00:33:02.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 -- start start 2024-03-24T00:33:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0a406b150 0x7fe0a40c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0a40d0830 0x7fe0a40c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0a415e0b0 0x7fe0a40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a415e610 con 0x7fe0a406b150 2024-03-24T00:33:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a415e490 con 0x7fe0a40d0830 2024-03-24T00:33:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a9200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a4070ac0 con 0x7fe0a415e0b0 2024-03-24T00:33:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0a406b150 0x7fe0a40c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0a406b150 0x7fe0a40c7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49688/0 (socket says 172.21.15.40:49688) 2024-03-24T00:33:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a1e00640 1 -- 172.21.15.40:0/2559509179 learned_addr learned my addr 172.21.15.40:0/2559509179 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe09be00640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0a40d0830 0x7fe0a40c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:02.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a2800640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0a415e0b0 0x7fe0a40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a1e00640 1 -- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0a415e0b0 msgr2=0x7fe0a40b4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a1e00640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0a415e0b0 0x7fe0a40b4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a1e00640 1 -- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0a40d0830 msgr2=0x7fe0a40c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a1e00640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0a40d0830 0x7fe0a40c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.776+0000 7fe0a1e00640 1 -- 172.21.15.40:0/2559509179 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a40c8500 con 0x7fe0a406b150 2024-03-24T00:33:02.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.780+0000 7fe0a1e00640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0a406b150 0x7fe0a40c7f20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe09400c3f0 tx=0x7fe09400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.780+0000 7fe09a000640 1 -- 172.21.15.40:0/2559509179 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094017070 con 0x7fe0a406b150 2024-03-24T00:33:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.780+0000 7fe09a000640 1 -- 172.21.15.40:0/2559509179 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0940052d0 con 0x7fe0a406b150 2024-03-24T00:33:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.780+0000 7fe09a000640 1 -- 172.21.15.40:0/2559509179 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe094003840 con 0x7fe0a406b150 2024-03-24T00:33:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.780+0000 7fe0a9200640 1 -- 172.21.15.40:0/2559509179 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a40b5860 con 0x7fe0a406b150 2024-03-24T00:33:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.780+0000 7fe0a9200640 1 -- 172.21.15.40:0/2559509179 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a4150fa0 con 0x7fe0a406b150 2024-03-24T00:33:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.780+0000 7fe0a9200640 1 -- 172.21.15.40:0/2559509179 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a4068170 con 0x7fe0a406b150 2024-03-24T00:33:02.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.780+0000 7fe09a000640 1 -- 172.21.15.40:0/2559509179 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe09401a070 con 0x7fe0a406b150 2024-03-24T00:33:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.780+0000 7fe09a000640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe080067660 0x7fe080069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.780+0000 7fe09a000640 1 -- 172.21.15.40:0/2559509179 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe094084be0 con 0x7fe0a406b150 2024-03-24T00:33:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.780+0000 7fe09be00640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe080067660 0x7fe080069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:02.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.784+0000 7fe09be00640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe080067660 0x7fe080069b20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0ae000 tx=0x7fe08c0a6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:02.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.784+0000 7fe09a000640 1 -- 172.21.15.40:0/2559509179 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe094005480 con 0x7fe0a406b150 2024-03-24T00:33:02.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.980+0000 7fe0a9200640 1 -- 172.21.15.40:0/2559509179 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0a40cf130 con 0x7fe0a406b150 2024-03-24T00:33:02.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe09a000640 1 -- 172.21.15.40:0/2559509179 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fe094055200 con 0x7fe0a406b150 2024-03-24T00:33:02.985 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:02.985 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5192,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5198,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5626,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/1600357100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1600357100},{"type":"v1","addr":"172.21.15.53:6836","nonce":1600357100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:00.034888+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5763},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5763]},"id":3}]} 2024-03-24T00:33:02.985 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 30 2024-03-24T00:33:02.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe0a9200640 1 -- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe080067660 msgr2=0x7fe080069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe0a9200640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe080067660 0x7fe080069b20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe08c0ae000 tx=0x7fe08c0a6040 comp rx=0 tx=0).stop 2024-03-24T00:33:02.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe0a9200640 1 -- 172.21.15.40:0/2559509179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0a406b150 msgr2=0x7fe0a40c7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:02.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe0a9200640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0a406b150 0x7fe0a40c7f20 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe09400c3f0 tx=0x7fe09400c8b0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe0a9200640 1 -- 172.21.15.40:0/2559509179 shutdown_connections 2024-03-24T00:33:02.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe0a9200640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe080067660 0x7fe080069b20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe0a9200640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe0a415e0b0 0x7fe0a40b4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe0a9200640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0a40d0830 0x7fe0a40c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe0a9200640 1 --2- 172.21.15.40:0/2559509179 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe0a406b150 0x7fe0a40c7f20 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:02.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe0a9200640 1 -- 172.21.15.40:0/2559509179 >> 172.21.15.40:0/2559509179 conn(0x7fe0a40d1c90 msgr2=0x7fe0a40aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:02.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe0a9200640 1 -- 172.21.15.40:0/2559509179 shutdown_connections 2024-03-24T00:33:02.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:02.984+0000 7fe0a9200640 1 -- 172.21.15.40:0/2559509179 wait complete. 2024-03-24T00:33:03.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:32:57.024524+0000', 'modified': '2024-03-24T00:33:00.034888+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5763}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5763': {'gid': 5763, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6834/224223662', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 224223662}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 224223662}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5763, 'qdb_cluster': [5763]} 2024-03-24T00:33:03.409 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:33:03.410 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:33:03.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.548+0000 7fe4a1c00640 1 Processor -- start 2024-03-24T00:33:03.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.548+0000 7fe4a1c00640 1 -- start start 2024-03-24T00:33:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.548+0000 7fe4a1c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe49c145760 0x7fe49c06f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe4a1c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe49c0cf200 0x7fe49c0cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe4a1c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49c072bc0 con 0x7fe49c072750 2024-03-24T00:33:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe4a1c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49c0cd630 con 0x7fe49c145760 2024-03-24T00:33:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe4a1c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49c0cd910 con 0x7fe49c0cf200 2024-03-24T00:33:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe49aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe49c145760 0x7fe49c06f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe49aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe49c145760 0x7fe49c06f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49702/0 (socket says 172.21.15.40:49702) 2024-03-24T00:33:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe49aa00640 1 -- 172.21.15.40:0/4157886182 learned_addr learned my addr 172.21.15.40:0/4157886182 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:03.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe49b400640 1 --2- 172.21.15.40:0/4157886182 >> v2:172.21.15.53:3300/0 conn(0x7fe49c0cf200 0x7fe49c0cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe49aa00640 1 -- 172.21.15.40:0/4157886182 >> v2:172.21.15.53:3300/0 conn(0x7fe49c0cf200 msgr2=0x7fe49c0cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:03.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe49aa00640 1 --2- 172.21.15.40:0/4157886182 >> v2:172.21.15.53:3300/0 conn(0x7fe49c0cf200 0x7fe49c0cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe49aa00640 1 -- 172.21.15.40:0/4157886182 >> v1:172.21.15.53:6789/0 conn(0x7fe49c072750 legacy=0x7fe49c06e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe49aa00640 1 -- 172.21.15.40:0/4157886182 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49c0cdbf0 con 0x7fe49c145760 2024-03-24T00:33:03.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.552+0000 7fe49aa00640 1 --2- 172.21.15.40:0/4157886182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe49c145760 0x7fe49c06f580 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe49c06fce0 tx=0x7fe48c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98bdea2f55f33c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:03.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe49a000640 1 -- 172.21.15.40:0/4157886182 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c0b0070 con 0x7fe49c145760 2024-03-24T00:33:03.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe49a000640 1 -- 172.21.15.40:0/4157886182 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48c0ab600 con 0x7fe49c145760 2024-03-24T00:33:03.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe49a000640 1 -- 172.21.15.40:0/4157886182 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48c0ab9d0 con 0x7fe49c145760 2024-03-24T00:33:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 -- 172.21.15.40:0/4157886182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe49c145760 msgr2=0x7fe49c06f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 --2- 172.21.15.40:0/4157886182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe49c145760 0x7fe49c06f580 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe49c06fce0 tx=0x7fe48c01c520 comp rx=0 tx=0).stop 2024-03-24T00:33:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 -- 172.21.15.40:0/4157886182 shutdown_connections 2024-03-24T00:33:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 --2- 172.21.15.40:0/4157886182 >> v2:172.21.15.53:3300/0 conn(0x7fe49c0cf200 0x7fe49c0cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 --2- 172.21.15.40:0/4157886182 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe49c145760 0x7fe49c06f580 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 -- 172.21.15.40:0/4157886182 >> 172.21.15.40:0/4157886182 conn(0x7fe49c0d0c20 msgr2=0x7fe49c0d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 -- 172.21.15.40:0/4157886182 shutdown_connections 2024-03-24T00:33:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 -- 172.21.15.40:0/4157886182 wait complete. 2024-03-24T00:33:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 Processor -- start 2024-03-24T00:33:03.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 -- start start 2024-03-24T00:33:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe49c072750 0x7fe49c165e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe49c0cf200 0x7fe49c15e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe49c145760 0x7fe49c15edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c06eda0 con 0x7fe49c0cf200 2024-03-24T00:33:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c06ec20 con 0x7fe49c145760 2024-03-24T00:33:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe4a1c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c168a60 con 0x7fe49c072750 2024-03-24T00:33:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe492000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe49c0cf200 0x7fe49c15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe492000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe49c0cf200 0x7fe49c15e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49710/0 (socket says 172.21.15.40:49710) 2024-03-24T00:33:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe492000640 1 -- 172.21.15.40:0/1908528897 learned_addr learned my addr 172.21.15.40:0/1908528897 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe49aa00640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe49c072750 0x7fe49c165e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.556+0000 7fe49b400640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe49c145760 0x7fe49c15edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe492000640 1 -- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe49c072750 msgr2=0x7fe49c165e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe492000640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe49c072750 0x7fe49c165e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe492000640 1 -- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe49c145760 msgr2=0x7fe49c15edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe492000640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe49c145760 0x7fe49c15edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe492000640 1 -- 172.21.15.40:0/1908528897 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49c1642c0 con 0x7fe49c0cf200 2024-03-24T00:33:03.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe492000640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe49c0cf200 0x7fe49c15e870 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe48400c3a0 tx=0x7fe48400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:03.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe493400640 1 -- 172.21.15.40:0/1908528897 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4840052d0 con 0x7fe49c0cf200 2024-03-24T00:33:03.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe493400640 1 -- 172.21.15.40:0/1908528897 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe484005470 con 0x7fe49c0cf200 2024-03-24T00:33:03.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe4a1c00640 1 -- 172.21.15.40:0/1908528897 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49c164560 con 0x7fe49c0cf200 2024-03-24T00:33:03.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe493400640 1 -- 172.21.15.40:0/1908528897 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484007260 con 0x7fe49c0cf200 2024-03-24T00:33:03.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe4a1c00640 1 -- 172.21.15.40:0/1908528897 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49c164b20 con 0x7fe49c0cf200 2024-03-24T00:33:03.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe493400640 1 -- 172.21.15.40:0/1908528897 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe484007400 con 0x7fe49c0cf200 2024-03-24T00:33:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.560+0000 7fe493400640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe478067690 0x7fe478069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe493400640 1 -- 172.21.15.40:0/1908528897 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe484084d30 con 0x7fe49c0cf200 2024-03-24T00:33:03.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 Processor -- start 2024-03-24T00:33:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49aa00640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe478067690 0x7fe478069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 -- start start 2024-03-24T00:33:03.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49aa00640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe478067690 0x7fe478069b50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0ad000 tx=0x7fe48c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:03.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b86a5d80 0x55c6b86a6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a66a0 0x55c6b865bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b865c400 0x55c6b865c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b84a5680 con 0x55c6b865c400 2024-03-24T00:33:03.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b842cb00 con 0x55c6b86a5d80 2024-03-24T00:33:03.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6b84a1720 con 0x55c6b86a66a0 2024-03-24T00:33:03.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b865c400 0x55c6b865c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b86a5d80 0x55c6b86a6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b865c400 0x55c6b865c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49718/0 (socket says 172.21.15.40:49718) 2024-03-24T00:33:03.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49b400640 1 -- 172.21.15.40:0/2024890898 learned_addr learned my addr 172.21.15.40:0/2024890898 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:03.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe492000640 1 --2- 172.21.15.40:0/2024890898 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a66a0 0x55c6b865bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49b400640 1 -- 172.21.15.40:0/2024890898 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a66a0 msgr2=0x55c6b865bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:03.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49b400640 1 --2- 172.21.15.40:0/2024890898 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a66a0 0x55c6b865bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49b400640 1 -- 172.21.15.40:0/2024890898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b86a5d80 msgr2=0x55c6b86a6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:03.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49b400640 1 --2- 172.21.15.40:0/2024890898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b86a5d80 0x55c6b86a6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49b400640 1 -- 172.21.15.40:0/2024890898 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6b847d940 con 0x55c6b865c400 2024-03-24T00:33:03.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49aa00640 1 --2- 172.21.15.40:0/2024890898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b86a5d80 0x55c6b86a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:03.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe49b400640 1 --2- 172.21.15.40:0/2024890898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b865c400 0x55c6b865c7e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe494011550 tx=0x7fe494009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f733d406fa807a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe48b400640 1 -- 172.21.15.40:0/2024890898 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe494015840 con 0x55c6b865c400 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe48b400640 1 -- 172.21.15.40:0/2024890898 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4940159e0 con 0x55c6b865c400 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe48b400640 1 -- 172.21.15.40:0/2024890898 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe494015db0 con 0x55c6b865c400 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 -- 172.21.15.40:0/2024890898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b865c400 msgr2=0x55c6b865c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 --2- 172.21.15.40:0/2024890898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b865c400 0x55c6b865c7e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe494011550 tx=0x7fe494009750 comp rx=0 tx=0).stop 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 -- 172.21.15.40:0/2024890898 shutdown_connections 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 --2- 172.21.15.40:0/2024890898 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b865c400 0x55c6b865c7e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 --2- 172.21.15.40:0/2024890898 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a66a0 0x55c6b865bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 --2- 172.21.15.40:0/2024890898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b86a5d80 0x55c6b86a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 -- 172.21.15.40:0/2024890898 >> 172.21.15.40:0/2024890898 conn(0x55c6b8653fa0 msgr2=0x55c6b8656390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 -- 172.21.15.40:0/2024890898 shutdown_connections 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 -- 172.21.15.40:0/2024890898 wait complete. 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.564+0000 7fe4a5072000 1 -- start start 2024-03-24T00:33:03.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 1 Processor -- start 2024-03-24T00:33:03.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b865c400 0x55c6b86a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 0x55c6b86624c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b86a66a0 0x55c6b8662a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b8562700 con 0x55c6b86a5d80 2024-03-24T00:33:03.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b845fae0 con 0x55c6b86a66a0 2024-03-24T00:33:03.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6b843d750 con 0x55c6b865c400 2024-03-24T00:33:03.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 10 client.0 _ll_register_callbacks cb 0x55c6b8642ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:03.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe492000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 0x55c6b86624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe492000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 0x55c6b86624c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49730/0 (socket says 172.21.15.40:49730) 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe492000640 1 -- 172.21.15.40:0/1893800372 learned_addr learned my addr 172.21.15.40:0/1893800372 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe49aa00640 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b865c400 0x55c6b86a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe49b400640 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b86a66a0 0x55c6b8662a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe492000640 1 -- 172.21.15.40:0/1893800372 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b865c400 msgr2=0x55c6b86a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe492000640 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b865c400 0x55c6b86a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe492000640 1 -- 172.21.15.40:0/1893800372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b86a66a0 msgr2=0x55c6b8662a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe492000640 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b86a66a0 0x55c6b8662a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe492000640 1 -- 172.21.15.40:0/1893800372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6b84ead80 con 0x55c6b86a5d80 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe492000640 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 0x55c6b86624c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe484083100 tx=0x7fe4840552c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c7c85a4747fab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe47fe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe47fe00640 1 -- 172.21.15.40:0/1893800372 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48405d920 con 0x55c6b86a5d80 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe47fe00640 1 -- 172.21.15.40:0/1893800372 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe484054710 con 0x55c6b86a5d80 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 1 -- 172.21.15.40:0/1893800372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4940159e0 con 0x55c6b86a5d80 2024-03-24T00:33:03.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe47fe00640 1 -- 172.21.15.40:0/1893800372 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe484054910 con 0x55c6b86a5d80 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 10 client.5907 fetch_fsmap learned FSMap version 30 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 1 -- 172.21.15.40:0/1893800372 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x55c6b842cd10 con 0x55c6b86a5d80 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe47fe00640 1 -- 172.21.15.40:0/1893800372 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4940159e0 con 0x55c6b86a5d80 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe47fe00640 1 -- 172.21.15.40:0/1893800372 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe48407cb20 con 0x55c6b86a5d80 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 10 client.5907 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 10 client.5907 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 20 client.5907 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c6b8663740 0x55c6b86b8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 4 client.5907 mds_command: new command op to 5763 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe4a5072000 1 -- 172.21.15.40:0/1893800372 --> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6b8440b60 con 0x55c6b8663740 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe49aa00640 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c6b8663740 0x55c6b86b8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe47fe00640 10 client.5907 ms_handle_connect on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:03.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.568+0000 7fe49aa00640 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c6b8663740 0x55c6b86b8040 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64df2b195b91eff0 server_cookie=93d85bd1f38963e6 in_seq=0 out_seq=0 2024-03-24T00:33:03.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe47fe00640 1 -- 172.21.15.40:0/1893800372 <== mds.0 v2:172.21.15.40:6832/224223662 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c6b8440b60 con 0x55c6b8663740 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe47fe00640 10 client.5907 handle_command_reply: tid=0 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe4a5072000 1 client.5907 shutdown 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe4a5072000 1 -- 172.21.15.40:0/1893800372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 msgr2=0x55c6b86624c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe4a5072000 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 0x55c6b86624c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe484083100 tx=0x7fe4840552c0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe4a5072000 1 -- 172.21.15.40:0/1893800372 shutdown_connections 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe4a5072000 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c6b8663740 0x55c6b86b8040 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe4a5072000 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b86a66a0 0x55c6b8662a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe4a5072000 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 0x55c6b86624c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe4a5072000 1 --2- 172.21.15.40:0/1893800372 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b865c400 0x55c6b86a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe4a5072000 1 -- 172.21.15.40:0/1893800372 >> 172.21.15.40:0/1893800372 conn(0x55c6b8653fa0 msgr2=0x55c6b865d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe47fe00640 0 client.5907 ms_handle_reset on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe4a5072000 1 -- 172.21.15.40:0/1893800372 shutdown_connections 2024-03-24T00:33:03.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe4a5072000 1 -- 172.21.15.40:0/1893800372 wait complete. 2024-03-24T00:33:03.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.572+0000 7fe4a5072000 20 client.5907 trim_cache size 0 max 16384 2024-03-24T00:33:03.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe4a5072000 1 Processor -- start 2024-03-24T00:33:03.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe4a5072000 1 -- start start 2024-03-24T00:33:03.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe4a5072000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b865c400 0x55c6b8672df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe4a5072000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a5d80 0x55c6b8686010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe4a5072000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a66a0 0x55c6b8686550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe4a5072000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0b5790 con 0x55c6b86a66a0 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe4a5072000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0b4930 con 0x55c6b865c400 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe4a5072000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0b3d30 con 0x55c6b86a5d80 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe49b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a66a0 0x55c6b8686550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe49b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a66a0 0x55c6b8686550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49732/0 (socket says 172.21.15.40:49732) 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe49b400640 1 -- 172.21.15.40:0/4276322449 learned_addr learned my addr 172.21.15.40:0/4276322449 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe49aa00640 1 --2- 172.21.15.40:0/4276322449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b865c400 0x55c6b8672df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe492000640 1 --2- 172.21.15.40:0/4276322449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a5d80 0x55c6b8686010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe49b400640 1 -- 172.21.15.40:0/4276322449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a5d80 msgr2=0x55c6b8686010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe49b400640 1 --2- 172.21.15.40:0/4276322449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a5d80 0x55c6b8686010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe49b400640 1 -- 172.21.15.40:0/4276322449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b865c400 msgr2=0x55c6b8672df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe49b400640 1 --2- 172.21.15.40:0/4276322449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b865c400 0x55c6b8672df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe49b400640 1 -- 172.21.15.40:0/4276322449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6b8472540 con 0x55c6b86a66a0 2024-03-24T00:33:03.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe492000640 1 --2- 172.21.15.40:0/4276322449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a5d80 0x55c6b8686010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:03.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe49b400640 1 --2- 172.21.15.40:0/4276322449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a66a0 0x55c6b8686550 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe494000f90 tx=0x7fe494017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c7bc3b170fe734 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:03.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe48aa00640 1 -- 172.21.15.40:0/4276322449 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49402e070 con 0x55c6b86a66a0 2024-03-24T00:33:03.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe48aa00640 1 -- 172.21.15.40:0/4276322449 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe494003e10 con 0x55c6b86a66a0 2024-03-24T00:33:03.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.588+0000 7fe48aa00640 1 -- 172.21.15.40:0/4276322449 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe494015880 con 0x55c6b86a66a0 2024-03-24T00:33:03.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 -- 172.21.15.40:0/4276322449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a66a0 msgr2=0x55c6b8686550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 --2- 172.21.15.40:0/4276322449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a66a0 0x55c6b8686550 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe494000f90 tx=0x7fe494017000 comp rx=0 tx=0).stop 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 -- 172.21.15.40:0/4276322449 shutdown_connections 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 --2- 172.21.15.40:0/4276322449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a66a0 0x55c6b8686550 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 --2- 172.21.15.40:0/4276322449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a5d80 0x55c6b8686010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 --2- 172.21.15.40:0/4276322449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b865c400 0x55c6b8672df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 -- 172.21.15.40:0/4276322449 >> 172.21.15.40:0/4276322449 conn(0x55c6b8653fa0 msgr2=0x55c6b8685900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 -- 172.21.15.40:0/4276322449 shutdown_connections 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 -- 172.21.15.40:0/4276322449 wait complete. 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 -- start start 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 Processor -- start 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b865c400 0x55c6b870b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 0x55c6b870d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a66a0 0x55c6b870fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b844da80 con 0x55c6b86a5d80 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49c0b4330 con 0x55c6b865c400 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6b844be20 con 0x55c6b86a66a0 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 10 client.0 _ll_register_callbacks cb 0x7fe48c0ae070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe492000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 0x55c6b870d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe492000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 0x55c6b870d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49742/0 (socket says 172.21.15.40:49742) 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe492000640 1 -- 172.21.15.40:0/1447436783 learned_addr learned my addr 172.21.15.40:0/1447436783 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe49aa00640 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b865c400 0x55c6b870b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe49b400640 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a66a0 0x55c6b870fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe492000640 1 -- 172.21.15.40:0/1447436783 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a66a0 msgr2=0x55c6b870fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe492000640 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a66a0 0x55c6b870fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe492000640 1 -- 172.21.15.40:0/1447436783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b865c400 msgr2=0x55c6b870b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe492000640 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b865c400 0x55c6b870b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe492000640 1 -- 172.21.15.40:0/1447436783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6b84394b0 con 0x55c6b86a5d80 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe49aa00640 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b865c400 0x55c6b870b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:03.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe492000640 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 0x55c6b870d8a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe48400cd60 tx=0x7fe484083090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcabbd232b71499b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:03.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe47fe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe47fe00640 1 -- 172.21.15.40:0/1447436783 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484073040 con 0x55c6b86a5d80 2024-03-24T00:33:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe47fe00640 1 -- 172.21.15.40:0/1447436783 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe48405b510 con 0x55c6b86a5d80 2024-03-24T00:33:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 -- 172.21.15.40:0/1447436783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe494003e10 con 0x55c6b86a5d80 2024-03-24T00:33:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe47fe00640 1 -- 172.21.15.40:0/1447436783 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe48405d920 con 0x55c6b86a5d80 2024-03-24T00:33:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 10 client.5913 fetch_fsmap learned FSMap version 30 2024-03-24T00:33:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.592+0000 7fe4a5072000 1 -- 172.21.15.40:0/1447436783 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x55c6b844c2f0 con 0x55c6b86a5d80 2024-03-24T00:33:03.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe47fe00640 1 -- 172.21.15.40:0/1447436783 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe494003e10 con 0x55c6b86a5d80 2024-03-24T00:33:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe47fe00640 1 -- 172.21.15.40:0/1447436783 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 30) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fe484051010 con 0x55c6b86a5d80 2024-03-24T00:33:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 10 client.5913 fetch_fsmap finished waiting for FSMap version 30 2024-03-24T00:33:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 10 client.5913 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:33:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 20 client.5913 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c6b8710330 0x55c6b8712720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 4 client.5913 mds_command: new command op to 5763 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:33:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 -- 172.21.15.40:0/1447436783 --> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c6b84b2bc0 con 0x55c6b8710330 2024-03-24T00:33:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe49aa00640 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c6b8710330 0x55c6b8712720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe49aa00640 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c6b8710330 0x55c6b8712720 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44a6296edf3cd274 server_cookie=956e35ce97669e03 in_seq=0 out_seq=0 2024-03-24T00:33:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe47fe00640 10 client.5913 ms_handle_connect on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:03.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe47fe00640 1 -- 172.21.15.40:0/1447436783 <== mds.0 v2:172.21.15.40:6832/224223662 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c6b84b2bc0 con 0x55c6b8710330 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe47fe00640 10 client.5913 handle_command_reply: tid=0 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 client.5913 shutdown 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 -- 172.21.15.40:0/1447436783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 msgr2=0x55c6b870d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 0x55c6b870d8a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe48400cd60 tx=0x7fe484083090 comp rx=0 tx=0).stop 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 -- 172.21.15.40:0/1447436783 shutdown_connections 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c6b8710330 0x55c6b8712720 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b86a66a0 0x55c6b870fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b86a5d80 0x55c6b870d8a0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 --2- 172.21.15.40:0/1447436783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b865c400 0x55c6b870b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 -- 172.21.15.40:0/1447436783 >> 172.21.15.40:0/1447436783 conn(0x55c6b8653fa0 msgr2=0x55c6b8686190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe47fe00640 0 client.5913 ms_handle_reset on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 -- 172.21.15.40:0/1447436783 shutdown_connections 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 1 -- 172.21.15.40:0/1447436783 wait complete. 2024-03-24T00:33:03.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a5072000 20 client.5913 trim_cache size 0 max 16384 2024-03-24T00:33:03.603 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:33:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:33:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5763, 2024-03-24T00:33:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:33:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:33:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:33:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 5.9561983630000004, 2024-03-24T00:33:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 30, 2024-03-24T00:33:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 44, 2024-03-24T00:33:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 44, 2024-03-24T00:33:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 10.791694143999999 2024-03-24T00:33:03.604 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:03.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a1c00640 1 -- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe478067690 msgr2=0x7fe478069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a1c00640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe478067690 0x7fe478069b50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0ad000 tx=0x7fe48c008040 comp rx=0 tx=0).stop 2024-03-24T00:33:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a1c00640 1 -- 172.21.15.40:0/1908528897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe49c0cf200 msgr2=0x7fe49c15e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a1c00640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe49c0cf200 0x7fe49c15e870 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe48400c3a0 tx=0x7fe48400c860 comp rx=0 tx=0).stop 2024-03-24T00:33:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a1c00640 1 -- 172.21.15.40:0/1908528897 shutdown_connections 2024-03-24T00:33:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a1c00640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe478067690 0x7fe478069b50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a1c00640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe49c145760 0x7fe49c15edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a1c00640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe49c0cf200 0x7fe49c15e870 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a1c00640 1 --2- 172.21.15.40:0/1908528897 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe49c072750 0x7fe49c165e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a1c00640 1 -- 172.21.15.40:0/1908528897 >> 172.21.15.40:0/1908528897 conn(0x7fe49c0d0c20 msgr2=0x7fe49c13e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a1c00640 1 -- 172.21.15.40:0/1908528897 shutdown_connections 2024-03-24T00:33:03.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.596+0000 7fe4a1c00640 1 -- 172.21.15.40:0/1908528897 wait complete. 2024-03-24T00:33:03.607 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-03-24T00:33:03.604+0000 7f2af2400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:33:03.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:33:03.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c72e00640 1 Processor -- start 2024-03-24T00:33:03.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c72e00640 1 -- start start 2024-03-24T00:33:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c72e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c6c06f100 0x7f0c6c070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c72e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c6c06f4e0 0x7f0c6c070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c72e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c6c0d27b0 con 0x7f0c6c0aea60 2024-03-24T00:33:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c72e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c6c0b1490 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c72e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c06fa70 con 0x7f0c6c06f100 2024-03-24T00:33:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c63400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c6c06f4e0 0x7f0c6c070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c63400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c6c06f4e0 0x7f0c6c070030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49750/0 (socket says 172.21.15.40:49750) 2024-03-24T00:33:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c63400640 1 -- 172.21.15.40:0/2569132912 learned_addr learned my addr 172.21.15.40:0/2569132912 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c6be00640 1 --2- 172.21.15.40:0/2569132912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c6c06f100 0x7f0c6c070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c63400640 1 -- 172.21.15.40:0/2569132912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c6c06f100 msgr2=0x7f0c6c070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:03.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c63400640 1 --2- 172.21.15.40:0/2569132912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c6c06f100 0x7f0c6c070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c63400640 1 -- 172.21.15.40:0/2569132912 >> v1:172.21.15.40:6789/0 conn(0x7f0c6c0aea60 legacy=0x7f0c6c0aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.760+0000 7f0c63400640 1 -- 172.21.15.40:0/2569132912 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c06fd50 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.764+0000 7f0c63400640 1 --2- 172.21.15.40:0/2569132912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c06f4e0 0x7f0c6c070030 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0c54009870 tx=0x7f0c5401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8146498a5632fd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:03.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.764+0000 7f0c6b400640 1 -- 172.21.15.40:0/2569132912 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c540b1070 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.764+0000 7f0c6b400640 1 -- 172.21.15.40:0/2569132912 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c540ac410 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.764+0000 7f0c6b400640 1 -- 172.21.15.40:0/2569132912 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c540ac7e0 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.764+0000 7f0c72e00640 1 -- 172.21.15.40:0/2569132912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c06f4e0 msgr2=0x7f0c6c070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.764+0000 7f0c72e00640 1 --2- 172.21.15.40:0/2569132912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c06f4e0 0x7f0c6c070030 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0c54009870 tx=0x7f0c5401c560 comp rx=0 tx=0).stop 2024-03-24T00:33:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.764+0000 7f0c72e00640 1 -- 172.21.15.40:0/2569132912 shutdown_connections 2024-03-24T00:33:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.764+0000 7f0c72e00640 1 --2- 172.21.15.40:0/2569132912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c06f4e0 0x7f0c6c070030 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.764+0000 7f0c72e00640 1 --2- 172.21.15.40:0/2569132912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c6c06f100 0x7f0c6c070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.764+0000 7f0c72e00640 1 -- 172.21.15.40:0/2569132912 >> 172.21.15.40:0/2569132912 conn(0x7f0c6c0d1c90 msgr2=0x7f0c6c0d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.764+0000 7f0c72e00640 1 -- 172.21.15.40:0/2569132912 shutdown_connections 2024-03-24T00:33:03.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.764+0000 7f0c72e00640 1 -- 172.21.15.40:0/2569132912 wait complete. 2024-03-24T00:33:03.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c72e00640 1 Processor -- start 2024-03-24T00:33:03.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c72e00640 1 -- start start 2024-03-24T00:33:03.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c72e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c06f100 0x7f0c6c0c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c72e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c06f4e0 0x7f0c6c156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c72e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c6c0aea60 0x7f0c6c09bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c72e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c0b0e10 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c72e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c0b0c90 con 0x7f0c6c06f100 2024-03-24T00:33:03.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c72e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c0b0f90 con 0x7f0c6c0aea60 2024-03-24T00:33:03.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c06f4e0 0x7f0c6c156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c06f4e0 0x7f0c6c156d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49756/0 (socket says 172.21.15.40:49756) 2024-03-24T00:33:03.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c63400640 1 -- 172.21.15.40:0/2528813878 learned_addr learned my addr 172.21.15.40:0/2528813878 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:03.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c6be00640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c06f100 0x7f0c6c0c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c70e00640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c6c0aea60 0x7f0c6c09bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c63400640 1 -- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c6c0aea60 msgr2=0x7f0c6c09bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c63400640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c6c0aea60 0x7f0c6c09bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c63400640 1 -- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c06f100 msgr2=0x7f0c6c0c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c63400640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c06f100 0x7f0c6c0c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c63400640 1 -- 172.21.15.40:0/2528813878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c0c6650 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c63400640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c06f4e0 0x7f0c6c156d20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0c5401ac10 tx=0x7f0c54019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c68c00640 1 -- 172.21.15.40:0/2528813878 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c540bc070 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c68c00640 1 -- 172.21.15.40:0/2528813878 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c540ab5c0 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c72e00640 1 -- 172.21.15.40:0/2528813878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6c0c6930 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.768+0000 7f0c68c00640 1 -- 172.21.15.40:0/2528813878 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c540b1070 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.772+0000 7f0c72e00640 1 -- 172.21.15.40:0/2528813878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6c0c6ef0 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.772+0000 7f0c72e00640 1 -- 172.21.15.40:0/2528813878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c6c068170 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.772+0000 7f0c68c00640 1 -- 172.21.15.40:0/2528813878 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f0c540bf070 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.772+0000 7f0c68c00640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c48067660 0x7f0c48069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.772+0000 7f0c68c00640 1 -- 172.21.15.40:0/2528813878 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c5411f2f0 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.772+0000 7f0c6be00640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c48067660 0x7f0c48069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:03.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.772+0000 7f0c6be00640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c48067660 0x7f0c48069b20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0c5df0 tx=0x7f0c5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:03.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.776+0000 7f0c68c00640 1 -- 172.21.15.40:0/2528813878 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0c54016150 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.988+0000 7f0c72e00640 1 -- 172.21.15.40:0/2528813878 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0c6c15f6d0 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.988+0000 7f0c68c00640 1 -- 172.21.15.40:0/2528813878 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0c540ef910 con 0x7f0c6c06f4e0 2024-03-24T00:33:03.994 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:33:03.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.992+0000 7f0c72e00640 1 -- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c48067660 msgr2=0x7f0c48069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.992+0000 7f0c72e00640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c48067660 0x7f0c48069b20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0c5df0 tx=0x7f0c5c008040 comp rx=0 tx=0).stop 2024-03-24T00:33:03.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.992+0000 7f0c72e00640 1 -- 172.21.15.40:0/2528813878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c06f4e0 msgr2=0x7f0c6c156d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:03.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.992+0000 7f0c72e00640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c06f4e0 0x7f0c6c156d20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0c5401ac10 tx=0x7f0c54019300 comp rx=0 tx=0).stop 2024-03-24T00:33:03.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.992+0000 7f0c72e00640 1 -- 172.21.15.40:0/2528813878 shutdown_connections 2024-03-24T00:33:03.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.992+0000 7f0c72e00640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c48067660 0x7f0c48069b20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.992+0000 7f0c72e00640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c6c0aea60 0x7f0c6c09bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.992+0000 7f0c72e00640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c6c06f4e0 0x7f0c6c156d20 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.992+0000 7f0c72e00640 1 --2- 172.21.15.40:0/2528813878 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c6c06f100 0x7f0c6c0c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:03.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.992+0000 7f0c72e00640 1 -- 172.21.15.40:0/2528813878 >> 172.21.15.40:0/2528813878 conn(0x7f0c6c0d1c90 msgr2=0x7f0c6c06ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:03.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.992+0000 7f0c72e00640 1 -- 172.21.15.40:0/2528813878 shutdown_connections 2024-03-24T00:33:03.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:03.992+0000 7f0c72e00640 1 -- 172.21.15.40:0/2528813878 wait complete. 2024-03-24T00:33:04.012 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:33:04.012 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:33:04.012 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-03-24T00:33:04.012 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:33:04.012 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:33:04.012 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:33:04.013 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:33:04.013 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:33:04.013 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:33:04.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:04.013 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-03-24T00:33:04.022 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:33:04.022 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:33:04.022 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-24T00:33:04.023 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42366sec preferred_lft 42366sec 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 02:42:9e:ab:82:f9 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:33:04.024 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-24T00:33:04.025 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:33:04.025 INFO:teuthology.orchestra.run.smithi040.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:33:04.025 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:33:04.025 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-24T00:33:04.025 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:33:04.025 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:33:04.025 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:33:04.025 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-24T00:33:04.025 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:33:04.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:04.025 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:33:04.086 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-03-24T00:33:04.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:04.088 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-03-24T00:33:04.145 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-03-24T00:33:04.145 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:33:04.145 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-24T00:33:04.145 INFO:teuthology.orchestra.run.smithi040.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-24T00:33:04.145 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-24T00:33:04.145 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:33:04.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:04.146 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:33:04.146 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:33:04.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T00:33:04.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T00:33:04.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T00:33:04.146 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:33:04.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:04.238 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-24T00:33:04.246 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:33:04.247 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:33:04.247 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:33:04.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:04.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-03-24T00:33:04.312 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T00:33:04.312 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:33:04.312 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:33:04.368 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:33:04.368 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:33:04.368 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:33:04.420 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:33:04.480 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:33:04.533 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:33:04.533 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:33:04.533 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:33:04.533 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:33:04.533 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:33:04.533 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:33:04.533 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:33:04.533 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:33:04.534 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:33:04.535 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:33:04.535 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:33:04.535 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:33:04.535 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:33:04.535 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:33:04.535 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:33:04.535 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:33:04.535 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:33:04.535 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:33:04.535 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:33:04.535 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:33:04.535 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:33:04.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:04.536 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:33:04.587 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T00:33:04.587 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T00:33:04.631 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:33:04.654 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:33:04.705 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:33:04.706 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:33:04.706 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:33:04.706 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:33:04.706 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:33:04.706 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:33:04.706 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:33:04.706 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:33:04.706 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:33:04.706 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:33:04.706 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:33:04.706 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:33:04.706 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:33:04.707 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:33:04.708 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:33:04.708 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:33:04.708 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:33:04.708 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:33:04.708 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:33:04.708 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:33:04.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:04.709 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:33:04.730 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:2024-03-24T00:33:04.728+0000 7f513209b580 -1 init, newargv = 0x55eb818ee970 newargc=15 2024-03-24T00:33:04.731 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[39495]: starting ceph client 2024-03-24T00:33:04.738 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[39495]: starting fuse 2024-03-24T00:33:04.739 INFO:teuthology.orchestra.run.smithi040.stdout:43 2024-03-24T00:33:04.740 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-03-24T00:33:04.740 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.0")) 2024-03-24T00:33:04.741 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-03-24T00:33:04.862 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.0.39495.asok 2024-03-24T00:33:04.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:33:04.873 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.39495.asok 2024-03-24T00:33:04.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:04.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.39495.asok status 2024-03-24T00:33:05.053 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "0", 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "39495", 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5758, 2024-03-24T00:33:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 5758 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2679601456 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2679601456 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.5758 172.21.15.40:0/2679601456", 2024-03-24T00:33:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "172.21.15.40:0/2679601456", 2024-03-24T00:33:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-03-24T00:33:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 30, 2024-03-24T00:33:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 44, 2024-03-24T00:33:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:33:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-03-24T00:33:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-03-24T00:33:05.056 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:05.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:05.069 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-03-24T00:33:05.069 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T00:33:05.078 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-03-24T00:33:05.078 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T00:33:05.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:05.079 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:33:05.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random True 2024-03-24T00:33:05.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.324+0000 7fafcdc00640 1 Processor -- start 2024-03-24T00:33:05.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.324+0000 7fafcdc00640 1 -- start start 2024-03-24T00:33:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafcdc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fafc815b960 0x7fafc8070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafcdc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc80d16f0 0x7fafc80712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafcdc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc815f970 con 0x7fafc815bd40 2024-03-24T00:33:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafcdc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc8160580 con 0x7fafc815b960 2024-03-24T00:33:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafcdc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc815fc50 con 0x7fafc80d16f0 2024-03-24T00:33:05.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafc6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fafc815b960 0x7fafc8070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafbfe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fafc815bd40 0x7fafc80d0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38964/0 (socket says 172.21.15.40:38964) 2024-03-24T00:33:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafbfe00640 1 -- 172.21.15.40:0/93770242 learned_addr learned my addr 172.21.15.40:0/93770242 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafc7400640 1 --2- 172.21.15.40:0/93770242 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc80d16f0 0x7fafc80712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafc6a00640 1 -- 172.21.15.40:0/93770242 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc80d16f0 msgr2=0x7fafc80712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafc6a00640 1 --2- 172.21.15.40:0/93770242 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc80d16f0 0x7fafc80712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafc6a00640 1 -- 172.21.15.40:0/93770242 >> v1:172.21.15.40:6789/0 conn(0x7fafc815bd40 legacy=0x7fafc80d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafc6a00640 1 -- 172.21.15.40:0/93770242 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc815ff30 con 0x7fafc815b960 2024-03-24T00:33:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.328+0000 7fafc6a00640 1 --2- 172.21.15.40:0/93770242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc815b960 0x7fafc8070d60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fafb80097c0 tx=0x7fafb801c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c9f48d5e85140ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafc6000640 1 -- 172.21.15.40:0/93770242 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb80b0070 con 0x7fafc815b960 2024-03-24T00:33:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafc6000640 1 -- 172.21.15.40:0/93770242 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb80ab600 con 0x7fafc815b960 2024-03-24T00:33:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafc6000640 1 -- 172.21.15.40:0/93770242 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb80ab9d0 con 0x7fafc815b960 2024-03-24T00:33:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 -- 172.21.15.40:0/93770242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc815b960 msgr2=0x7fafc8070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 --2- 172.21.15.40:0/93770242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc815b960 0x7fafc8070d60 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fafb80097c0 tx=0x7fafb801c330 comp rx=0 tx=0).stop 2024-03-24T00:33:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 -- 172.21.15.40:0/93770242 shutdown_connections 2024-03-24T00:33:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 --2- 172.21.15.40:0/93770242 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc80d16f0 0x7fafc80712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 --2- 172.21.15.40:0/93770242 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc815b960 0x7fafc8070d60 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 -- 172.21.15.40:0/93770242 >> 172.21.15.40:0/93770242 conn(0x7fafc806ecf0 msgr2=0x7fafc806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 -- 172.21.15.40:0/93770242 shutdown_connections 2024-03-24T00:33:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 -- 172.21.15.40:0/93770242 wait complete. 2024-03-24T00:33:05.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 Processor -- start 2024-03-24T00:33:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 -- start start 2024-03-24T00:33:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc80d16f0 0x7fafc80c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc815b960 0x7fafc814bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:05.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc815bd40 0x7fafc815d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc80cdb80 con 0x7fafc815bd40 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc80cda00 con 0x7fafc80d16f0 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafcdc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc80cdd00 con 0x7fafc815b960 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafc7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc815bd40 0x7fafc815d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafc6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc80d16f0 0x7fafc80c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafc6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc80d16f0 0x7fafc80c68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:35184/0 (socket says 172.21.15.40:35184) 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.332+0000 7fafc6a00640 1 -- 172.21.15.40:0/2449612329 learned_addr learned my addr 172.21.15.40:0/2449612329 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafbfe00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc815b960 0x7fafc814bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafc6a00640 1 -- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc815b960 msgr2=0x7fafc814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafc6a00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc815b960 0x7fafc814bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafc6a00640 1 -- 172.21.15.40:0/2449612329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc815bd40 msgr2=0x7fafc815d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafc6a00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc815bd40 0x7fafc815d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafc6a00640 1 -- 172.21.15.40:0/2449612329 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc80cc580 con 0x7fafc80d16f0 2024-03-24T00:33:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafc6a00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc80d16f0 0x7fafc80c68a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fafb80097c0 tx=0x7fafb80190d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafbea00640 1 -- 172.21.15.40:0/2449612329 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb80b0040 con 0x7fafc80d16f0 2024-03-24T00:33:05.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafbea00640 1 -- 172.21.15.40:0/2449612329 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb80b9070 con 0x7fafc80d16f0 2024-03-24T00:33:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafcdc00640 1 -- 172.21.15.40:0/2449612329 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc80c6ff0 con 0x7fafc80d16f0 2024-03-24T00:33:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafbea00640 1 -- 172.21.15.40:0/2449612329 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb80aa9f0 con 0x7fafc80d16f0 2024-03-24T00:33:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafcdc00640 1 -- 172.21.15.40:0/2449612329 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc80c75b0 con 0x7fafc80d16f0 2024-03-24T00:33:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafcdc00640 1 -- 172.21.15.40:0/2449612329 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc8068170 con 0x7fafc80d16f0 2024-03-24T00:33:05.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.336+0000 7fafbea00640 1 -- 172.21.15.40:0/2449612329 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fafb80aab90 con 0x7fafc80d16f0 2024-03-24T00:33:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.340+0000 7fafbea00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafa4067660 0x7fafa4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.340+0000 7fafbea00640 1 -- 172.21.15.40:0/2449612329 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafb811fc20 con 0x7fafc80d16f0 2024-03-24T00:33:05.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.340+0000 7fafbfe00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafa4067660 0x7fafa4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:05.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.340+0000 7fafbfe00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafa4067660 0x7fafa4069b20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009fd0 tx=0x7fafb000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:05.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.344+0000 7fafbea00640 1 -- 172.21.15.40:0/2449612329 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fafb8015fb0 con 0x7fafc80d16f0 2024-03-24T00:33:05.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.504+0000 7fafcdc00640 1 -- 172.21.15.40:0/2449612329 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random}] v 0) v1 -- 0x7fafc815b120 con 0x7fafc80d16f0 2024-03-24T00:33:05.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.512+0000 7fafbea00640 1 -- 172.21.15.40:0/2449612329 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random}]=0 v20)=0 v20) v1 ==== 128+0+0 (secure 0 0 0) 0x7fafb80f0230 con 0x7fafc80d16f0 2024-03-24T00:33:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.512+0000 7fafcdc00640 1 -- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafa4067660 msgr2=0x7fafa4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.512+0000 7fafcdc00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafa4067660 0x7fafa4069b20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fafb0009fd0 tx=0x7fafb000b040 comp rx=0 tx=0).stop 2024-03-24T00:33:05.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.512+0000 7fafcdc00640 1 -- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc80d16f0 msgr2=0x7fafc80c68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.512+0000 7fafcdc00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc80d16f0 0x7fafc80c68a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fafb80097c0 tx=0x7fafb80190d0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.512+0000 7fafcdc00640 1 -- 172.21.15.40:0/2449612329 shutdown_connections 2024-03-24T00:33:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.512+0000 7fafcdc00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafa4067660 0x7fafa4069b20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.512+0000 7fafcdc00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc815bd40 0x7fafc815d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.512+0000 7fafcdc00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc815b960 0x7fafc814bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.512+0000 7fafcdc00640 1 --2- 172.21.15.40:0/2449612329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc80d16f0 0x7fafc80c68a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.512+0000 7fafcdc00640 1 -- 172.21.15.40:0/2449612329 >> 172.21.15.40:0/2449612329 conn(0x7fafc806ecf0 msgr2=0x7fafc806bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.516+0000 7fafcdc00640 1 -- 172.21.15.40:0/2449612329 shutdown_connections 2024-03-24T00:33:05.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.516+0000 7fafcdc00640 1 -- 172.21.15.40:0/2449612329 wait complete. 2024-03-24T00:33:05.535 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_distributed True 2024-03-24T00:33:05.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff69a800640 1 Processor -- start 2024-03-24T00:33:05.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff69a800640 1 -- start start 2024-03-24T00:33:05.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff69a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff69415bb10 0x7ff69406f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:05.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff69a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff69406f570 0x7ff694066ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:05.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff69a800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff694067c60 con 0x7ff694067430 2024-03-24T00:33:05.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff69a800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff69406f9d0 con 0x7ff69406f570 2024-03-24T00:33:05.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff69a800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69415ef60 con 0x7ff69415bb10 2024-03-24T00:33:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff693e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff694067430 0x7ff694071b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38980/0 (socket says 172.21.15.40:38980) 2024-03-24T00:33:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff693e00640 1 -- 172.21.15.40:0/1431800549 learned_addr learned my addr 172.21.15.40:0/1431800549 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff68aa00640 1 --2- 172.21.15.40:0/1431800549 >> v2:172.21.15.40:3300/0 conn(0x7ff69406f570 0x7ff694066ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff693400640 1 --2- 172.21.15.40:0/1431800549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff69415bb10 0x7ff69406f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff68aa00640 1 -- 172.21.15.40:0/1431800549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff69415bb10 msgr2=0x7ff69406f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff68aa00640 1 --2- 172.21.15.40:0/1431800549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff69415bb10 0x7ff69406f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff68aa00640 1 -- 172.21.15.40:0/1431800549 >> v1:172.21.15.40:6789/0 conn(0x7ff694067430 legacy=0x7ff694071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff68aa00640 1 -- 172.21.15.40:0/1431800549 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69415e910 con 0x7ff69406f570 2024-03-24T00:33:05.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff692a00640 1 -- 172.21.15.40:0/1431800549 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238210797 0 0) 0x7ff694067c60 con 0x7ff694067430 2024-03-24T00:33:05.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.688+0000 7ff693400640 1 --2- 172.21.15.40:0/1431800549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff69415bb10 0x7ff69406f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:33:05.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.692+0000 7ff68aa00640 1 --2- 172.21.15.40:0/1431800549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff69406f570 0x7ff694066ef0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0097c0 tx=0x7ff67c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf36e9c59face5c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:05.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.692+0000 7ff692a00640 1 -- 172.21.15.40:0/1431800549 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c0b0070 con 0x7ff69406f570 2024-03-24T00:33:05.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.692+0000 7ff692a00640 1 -- 172.21.15.40:0/1431800549 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff67c0ab600 con 0x7ff69406f570 2024-03-24T00:33:05.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.692+0000 7ff692a00640 1 -- 172.21.15.40:0/1431800549 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c0ab9d0 con 0x7ff69406f570 2024-03-24T00:33:05.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.692+0000 7ff69a800640 1 -- 172.21.15.40:0/1431800549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff69406f570 msgr2=0x7ff694066ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.692+0000 7ff69a800640 1 --2- 172.21.15.40:0/1431800549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff69406f570 0x7ff694066ef0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0097c0 tx=0x7ff67c01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.692+0000 7ff69a800640 1 -- 172.21.15.40:0/1431800549 shutdown_connections 2024-03-24T00:33:05.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.692+0000 7ff69a800640 1 --2- 172.21.15.40:0/1431800549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff69406f570 0x7ff694066ef0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.692+0000 7ff69a800640 1 --2- 172.21.15.40:0/1431800549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff69415bb10 0x7ff69406f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.692+0000 7ff69a800640 1 -- 172.21.15.40:0/1431800549 >> 172.21.15.40:0/1431800549 conn(0x7ff6940b27c0 msgr2=0x7ff6940b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:05.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.692+0000 7ff69a800640 1 -- 172.21.15.40:0/1431800549 shutdown_connections 2024-03-24T00:33:05.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.692+0000 7ff69a800640 1 -- 172.21.15.40:0/1431800549 wait complete. 2024-03-24T00:33:05.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff69a800640 1 Processor -- start 2024-03-24T00:33:05.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff69a800640 1 -- start start 2024-03-24T00:33:05.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff69a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff694067430 0x7ff69415afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:05.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff69a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff69406f570 0x7ff69415f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:05.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff69a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff69415bb10 0x7ff69414f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:05.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff69a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6940679e0 con 0x7ff694067430 2024-03-24T00:33:05.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff69a800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff694067860 con 0x7ff69406f570 2024-03-24T00:33:05.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff69a800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff69415f3d0 con 0x7ff69415bb10 2024-03-24T00:33:05.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff693400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff694067430 0x7ff69415afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:05.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff693400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff694067430 0x7ff69415afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49802/0 (socket says 172.21.15.40:49802) 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff693400640 1 -- 172.21.15.40:0/4035883048 learned_addr learned my addr 172.21.15.40:0/4035883048 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff693e00640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff69415bb10 0x7ff69414f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff68aa00640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff69406f570 0x7ff69415f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff693400640 1 -- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff69415bb10 msgr2=0x7ff69414f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff693400640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff69415bb10 0x7ff69414f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff693400640 1 -- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff69406f570 msgr2=0x7ff69415f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff693400640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff69406f570 0x7ff69415f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff693400640 1 -- 172.21.15.40:0/4035883048 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6941615f0 con 0x7ff694067430 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff68aa00640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff69406f570 0x7ff69415f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff693400640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff694067430 0x7ff69415afc0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff68400c7f0 tx=0x7ff68400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff68be00640 1 -- 172.21.15.40:0/4035883048 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684015070 con 0x7ff694067430 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff68be00640 1 -- 172.21.15.40:0/4035883048 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6840077c0 con 0x7ff694067430 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff68be00640 1 -- 172.21.15.40:0/4035883048 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684007bb0 con 0x7ff694067430 2024-03-24T00:33:05.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff69a800640 1 -- 172.21.15.40:0/4035883048 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69415dbe0 con 0x7ff694067430 2024-03-24T00:33:05.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.696+0000 7ff69a800640 1 -- 172.21.15.40:0/4035883048 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69414ece0 con 0x7ff694067430 2024-03-24T00:33:05.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.700+0000 7ff69a800640 1 -- 172.21.15.40:0/4035883048 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff694068170 con 0x7ff694067430 2024-03-24T00:33:05.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.700+0000 7ff68be00640 1 -- 172.21.15.40:0/4035883048 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff684018070 con 0x7ff694067430 2024-03-24T00:33:05.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.700+0000 7ff68be00640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff670067660 0x7ff670069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:05.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.700+0000 7ff68be00640 1 -- 172.21.15.40:0/4035883048 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff684085a40 con 0x7ff694067430 2024-03-24T00:33:05.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.700+0000 7ff68aa00640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff670067660 0x7ff670069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:05.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.704+0000 7ff68aa00640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff670067660 0x7ff670069b20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0ad000 tx=0x7ff67c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:05.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.704+0000 7ff68be00640 1 -- 172.21.15.40:0/4035883048 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff684056620 con 0x7ff694067430 2024-03-24T00:33:05.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.868+0000 7ff69a800640 1 -- 172.21.15.40:0/4035883048 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_distributed}] v 0) v1 -- 0x7ff6940b3500 con 0x7ff694067430 2024-03-24T00:33:05.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.872+0000 7ff68be00640 1 -- 172.21.15.40:0/4035883048 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_distributed}]=0 v21)=0 v21) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff684055f20 con 0x7ff694067430 2024-03-24T00:33:05.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.876+0000 7ff69a800640 1 -- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff670067660 msgr2=0x7ff670069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.876+0000 7ff69a800640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff670067660 0x7ff670069b20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0ad000 tx=0x7ff67c0a6000 comp rx=0 tx=0).stop 2024-03-24T00:33:05.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.876+0000 7ff69a800640 1 -- 172.21.15.40:0/4035883048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff694067430 msgr2=0x7ff69415afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:05.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.876+0000 7ff69a800640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff694067430 0x7ff69415afc0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff68400c7f0 tx=0x7ff68400ccb0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.876+0000 7ff69a800640 1 -- 172.21.15.40:0/4035883048 shutdown_connections 2024-03-24T00:33:05.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.876+0000 7ff69a800640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff670067660 0x7ff670069b20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.876+0000 7ff69a800640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff69415bb10 0x7ff69414f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.876+0000 7ff69a800640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff69406f570 0x7ff69415f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.876+0000 7ff69a800640 1 --2- 172.21.15.40:0/4035883048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff694067430 0x7ff69415afc0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:05.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.876+0000 7ff69a800640 1 -- 172.21.15.40:0/4035883048 >> 172.21.15.40:0/4035883048 conn(0x7ff6940b27c0 msgr2=0x7ff6941514c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:05.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.876+0000 7ff69a800640 1 -- 172.21.15.40:0/4035883048 shutdown_connections 2024-03-24T00:33:05.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:05.876+0000 7ff69a800640 1 -- 172.21.15.40:0/4035883048 wait complete. 2024-03-24T00:33:05.897 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random_max 1.0 2024-03-24T00:33:06.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f298d400640 1 Processor -- start 2024-03-24T00:33:06.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f298d400640 1 -- start start 2024-03-24T00:33:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f298d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f298815e3d0 0x7f29880af500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f298d400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2988071730 0x7f298815edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f298d400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2988072cf0 con 0x7f29880b05a0 2024-03-24T00:33:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f298d400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2988072fb0 con 0x7f298815e3d0 2024-03-24T00:33:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f298d400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2988073290 con 0x7f2988071730 2024-03-24T00:33:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f297fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f298815e3d0 0x7f29880af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f2987e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f29880b05a0 0x7f29880b0980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:38986/0 (socket says 172.21.15.40:38986) 2024-03-24T00:33:06.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f2987e00640 1 -- 172.21.15.40:0/1456530458 learned_addr learned my addr 172.21.15.40:0/1456530458 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:06.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f298ca00640 1 --2- 172.21.15.40:0/1456530458 >> v2:172.21.15.53:3300/0 conn(0x7f2988071730 0x7f298815edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:06.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f297fe00640 1 -- 172.21.15.40:0/1456530458 >> v2:172.21.15.53:3300/0 conn(0x7f2988071730 msgr2=0x7f298815edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:06.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f2987400640 1 -- 172.21.15.40:0/1456530458 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394114461 0 0) 0x7f2988072cf0 con 0x7f29880b05a0 2024-03-24T00:33:06.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f297fe00640 1 --2- 172.21.15.40:0/1456530458 >> v2:172.21.15.53:3300/0 conn(0x7f2988071730 0x7f298815edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:06.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f297fe00640 1 -- 172.21.15.40:0/1456530458 >> v1:172.21.15.40:6789/0 conn(0x7f29880b05a0 legacy=0x7f29880b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:06.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.040+0000 7f297fe00640 1 -- 172.21.15.40:0/1456530458 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298815e890 con 0x7f298815e3d0 2024-03-24T00:33:06.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.044+0000 7f297fe00640 1 --2- 172.21.15.40:0/1456530458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f298815e3d0 0x7f29880af500 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f298806c9d0 tx=0x7f297001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89103a1cec6ffb85 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.044+0000 7f2987400640 1 -- 172.21.15.40:0/1456530458 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29700b0070 con 0x7f298815e3d0 2024-03-24T00:33:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.044+0000 7f2987400640 1 -- 172.21.15.40:0/1456530458 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29700ab600 con 0x7f298815e3d0 2024-03-24T00:33:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.044+0000 7f2987400640 1 -- 172.21.15.40:0/1456530458 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29700ab9d0 con 0x7f298815e3d0 2024-03-24T00:33:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.044+0000 7f298d400640 1 -- 172.21.15.40:0/1456530458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f298815e3d0 msgr2=0x7f29880af500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.044+0000 7f298d400640 1 --2- 172.21.15.40:0/1456530458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f298815e3d0 0x7f29880af500 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f298806c9d0 tx=0x7f297001c480 comp rx=0 tx=0).stop 2024-03-24T00:33:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.044+0000 7f298d400640 1 -- 172.21.15.40:0/1456530458 shutdown_connections 2024-03-24T00:33:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.044+0000 7f298d400640 1 --2- 172.21.15.40:0/1456530458 >> v2:172.21.15.53:3300/0 conn(0x7f2988071730 0x7f298815edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.044+0000 7f298d400640 1 --2- 172.21.15.40:0/1456530458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f298815e3d0 0x7f29880af500 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.044+0000 7f298d400640 1 -- 172.21.15.40:0/1456530458 >> 172.21.15.40:0/1456530458 conn(0x7f29880b57a0 msgr2=0x7f29880725e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.044+0000 7f298d400640 1 -- 172.21.15.40:0/1456530458 shutdown_connections 2024-03-24T00:33:06.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.044+0000 7f298d400640 1 -- 172.21.15.40:0/1456530458 wait complete. 2024-03-24T00:33:06.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f298d400640 1 Processor -- start 2024-03-24T00:33:06.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f298d400640 1 -- start start 2024-03-24T00:33:06.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f298d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988071730 0x7f298814a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:06.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f298d400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29880b05a0 0x7f29880af500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:06.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f298d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f298815e3d0 0x7f29880afa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:06.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f298d400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298816b2e0 con 0x7f2988071730 2024-03-24T00:33:06.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f298d400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298816b160 con 0x7f298815e3d0 2024-03-24T00:33:06.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f298d400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298816b460 con 0x7f29880b05a0 2024-03-24T00:33:06.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f2987e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988071730 0x7f298814a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:06.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f2987e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988071730 0x7f298814a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49818/0 (socket says 172.21.15.40:49818) 2024-03-24T00:33:06.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f2987e00640 1 -- 172.21.15.40:0/913743141 learned_addr learned my addr 172.21.15.40:0/913743141 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:06.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f298ca00640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f298815e3d0 0x7f29880afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:06.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f297fe00640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29880b05a0 0x7f29880af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:06.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f2987e00640 1 -- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29880b05a0 msgr2=0x7f29880af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f2987e00640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29880b05a0 0x7f29880af500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f2987e00640 1 -- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f298815e3d0 msgr2=0x7f29880afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f2987e00640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f298815e3d0 0x7f29880afa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f2987e00640 1 -- 172.21.15.40:0/913743141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2988153700 con 0x7f2988071730 2024-03-24T00:33:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f297fe00640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29880b05a0 0x7f29880af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.048+0000 7f298ca00640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f298815e3d0 0x7f29880afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.052+0000 7f2987e00640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988071730 0x7f298814a170 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f297800c4b0 tx=0x7f297800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.052+0000 7f2984c00640 1 -- 172.21.15.40:0/913743141 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978018070 con 0x7f2988071730 2024-03-24T00:33:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.052+0000 7f2984c00640 1 -- 172.21.15.40:0/913743141 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2978007500 con 0x7f2988071730 2024-03-24T00:33:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.052+0000 7f298d400640 1 -- 172.21.15.40:0/913743141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29881539c0 con 0x7f2988071730 2024-03-24T00:33:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.052+0000 7f2984c00640 1 -- 172.21.15.40:0/913743141 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978004a30 con 0x7f2988071730 2024-03-24T00:33:06.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.052+0000 7f298d400640 1 -- 172.21.15.40:0/913743141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298814abe0 con 0x7f2988071730 2024-03-24T00:33:06.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.052+0000 7f298d400640 1 -- 172.21.15.40:0/913743141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298814f6d0 con 0x7f2988071730 2024-03-24T00:33:06.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.052+0000 7f2984c00640 1 -- 172.21.15.40:0/913743141 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f297801b070 con 0x7f2988071730 2024-03-24T00:33:06.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.052+0000 7f2984c00640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2964067660 0x7f2964069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:06.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.052+0000 7f2984c00640 1 -- 172.21.15.40:0/913743141 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2978084ac0 con 0x7f2988071730 2024-03-24T00:33:06.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.052+0000 7f297fe00640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2964067660 0x7f2964069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:06.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.056+0000 7f297fe00640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2964067660 0x7f2964069b20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f29700ad000 tx=0x7f29700a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:06.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.064+0000 7f2984c00640 1 -- 172.21.15.40:0/913743141 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f298814f6d0 con 0x7f2988071730 2024-03-24T00:33:06.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.224+0000 7f298d400640 1 -- 172.21.15.40:0/913743141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random_max}] v 0) v1 -- 0x7f298804f410 con 0x7f2988071730 2024-03-24T00:33:06.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.228+0000 7f2984c00640 1 -- 172.21.15.40:0/913743141 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random_max}]=0 v22)=0 v22) v1 ==== 131+0+0 (secure 0 0 0) 0x7f29780557d0 con 0x7f2988071730 2024-03-24T00:33:06.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.228+0000 7f298d400640 1 -- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2964067660 msgr2=0x7f2964069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.228+0000 7f298d400640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2964067660 0x7f2964069b20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f29700ad000 tx=0x7f29700a6000 comp rx=0 tx=0).stop 2024-03-24T00:33:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.228+0000 7f298d400640 1 -- 172.21.15.40:0/913743141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988071730 msgr2=0x7f298814a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.228+0000 7f298d400640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988071730 0x7f298814a170 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f297800c4b0 tx=0x7f297800c970 comp rx=0 tx=0).stop 2024-03-24T00:33:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.232+0000 7f298d400640 1 -- 172.21.15.40:0/913743141 shutdown_connections 2024-03-24T00:33:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.232+0000 7f298d400640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2964067660 0x7f2964069b20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.232+0000 7f298d400640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f298815e3d0 0x7f29880afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.232+0000 7f298d400640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29880b05a0 0x7f29880af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.232+0000 7f298d400640 1 --2- 172.21.15.40:0/913743141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2988071730 0x7f298814a170 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.232+0000 7f298d400640 1 -- 172.21.15.40:0/913743141 >> 172.21.15.40:0/913743141 conn(0x7f29880b57a0 msgr2=0x7f29880cf1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.232+0000 7f298d400640 1 -- 172.21.15.40:0/913743141 shutdown_connections 2024-03-24T00:33:06.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:06.232+0000 7f298d400640 1 -- 172.21.15.40:0/913743141 wait complete. 2024-03-24T00:33:06.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:06.253 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:33:06.253 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:33:06.253 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:33:06.253 DEBUG:teuthology.orchestra.run.smithi040:> # Use up a random number of inode numbers so the ephemeral pinning is not the same every test. 2024-03-24T00:33:06.253 DEBUG:teuthology.orchestra.run.smithi040:> mkdir .inode_number_thrash 2024-03-24T00:33:06.253 DEBUG:teuthology.orchestra.run.smithi040:> count=$((RANDOM % 1024)) 2024-03-24T00:33:06.253 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < count; i++)); do touch .inode_number_thrash/$i; done 2024-03-24T00:33:06.253 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf .inode_number_thrash 2024-03-24T00:33:06.253 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:33:07.060 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:33:07.064 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:33:09.115 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-24T00:33:09.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a2a000640 1 Processor -- start 2024-03-24T00:33:09.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a2a000640 1 -- start start 2024-03-24T00:33:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a2a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3a240aec70 0x7f3a240ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a2a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3a2406d3a0 0x7f3a2406d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a2a000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a24070ed0 con 0x7f3a2415a560 2024-03-24T00:33:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a2a000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a240cf240 con 0x7f3a240aec70 2024-03-24T00:33:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a2a000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a240cf520 con 0x7f3a2406d3a0 2024-03-24T00:33:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a23400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3a2415a560 0x7f3a240cc4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35758/0 (socket says 172.21.15.40:35758) 2024-03-24T00:33:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a23400640 1 -- 172.21.15.40:0/3178325058 learned_addr learned my addr 172.21.15.40:0/3178325058 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a1aa00640 1 --2- 172.21.15.40:0/3178325058 >> v2:172.21.15.40:3300/0 conn(0x7f3a240aec70 0x7f3a240ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a23e00640 1 --2- 172.21.15.40:0/3178325058 >> v2:172.21.15.53:3300/0 conn(0x7f3a2406d3a0 0x7f3a2406d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a1aa00640 1 -- 172.21.15.40:0/3178325058 >> v2:172.21.15.53:3300/0 conn(0x7f3a2406d3a0 msgr2=0x7f3a2406d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:09.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a1aa00640 1 --2- 172.21.15.40:0/3178325058 >> v2:172.21.15.53:3300/0 conn(0x7f3a2406d3a0 0x7f3a2406d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a1aa00640 1 -- 172.21.15.40:0/3178325058 >> v1:172.21.15.40:6789/0 conn(0x7f3a2415a560 legacy=0x7f3a240cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:09.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.332+0000 7f3a1aa00640 1 -- 172.21.15.40:0/3178325058 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a240b3aa0 con 0x7f3a240aec70 2024-03-24T00:33:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.340+0000 7f3a1aa00640 1 --2- 172.21.15.40:0/3178325058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a240aec70 0x7f3a240ae6b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3a24070a80 tx=0x7f3a0c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8bc0cf4410496a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.340+0000 7f3a22a00640 1 -- 172.21.15.40:0/3178325058 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0b0070 con 0x7f3a240aec70 2024-03-24T00:33:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.340+0000 7f3a22a00640 1 -- 172.21.15.40:0/3178325058 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c0ab600 con 0x7f3a240aec70 2024-03-24T00:33:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.340+0000 7f3a22a00640 1 -- 172.21.15.40:0/3178325058 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0ab9d0 con 0x7f3a240aec70 2024-03-24T00:33:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.340+0000 7f3a2a000640 1 -- 172.21.15.40:0/3178325058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a240aec70 msgr2=0x7f3a240ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.340+0000 7f3a2a000640 1 --2- 172.21.15.40:0/3178325058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a240aec70 0x7f3a240ae6b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3a24070a80 tx=0x7f3a0c01c360 comp rx=0 tx=0).stop 2024-03-24T00:33:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.340+0000 7f3a2a000640 1 -- 172.21.15.40:0/3178325058 shutdown_connections 2024-03-24T00:33:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.340+0000 7f3a2a000640 1 --2- 172.21.15.40:0/3178325058 >> v2:172.21.15.53:3300/0 conn(0x7f3a2406d3a0 0x7f3a2406d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.340+0000 7f3a2a000640 1 --2- 172.21.15.40:0/3178325058 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a240aec70 0x7f3a240ae6b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.340+0000 7f3a2a000640 1 -- 172.21.15.40:0/3178325058 >> 172.21.15.40:0/3178325058 conn(0x7f3a240b1660 msgr2=0x7f3a240b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.340+0000 7f3a2a000640 1 -- 172.21.15.40:0/3178325058 shutdown_connections 2024-03-24T00:33:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.340+0000 7f3a2a000640 1 -- 172.21.15.40:0/3178325058 wait complete. 2024-03-24T00:33:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a2a000640 1 Processor -- start 2024-03-24T00:33:09.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a2a000640 1 -- start start 2024-03-24T00:33:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a2a000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a2406d3a0 0x7f3a24155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a2a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a240aec70 0x7f3a24151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a2a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a2415a560 0x7f3a24152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a2a000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240b4350 con 0x7f3a240aec70 2024-03-24T00:33:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a2a000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240b41d0 con 0x7f3a2415a560 2024-03-24T00:33:09.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a2a000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a240b44d0 con 0x7f3a2406d3a0 2024-03-24T00:33:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a1aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a240aec70 0x7f3a24151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a1aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a240aec70 0x7f3a24151c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51972/0 (socket says 172.21.15.40:51972) 2024-03-24T00:33:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a1aa00640 1 -- 172.21.15.40:0/2483689047 learned_addr learned my addr 172.21.15.40:0/2483689047 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a23e00640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a2415a560 0x7f3a24152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a23400640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a2406d3a0 0x7f3a24155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a1aa00640 1 -- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a2406d3a0 msgr2=0x7f3a24155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a1aa00640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a2406d3a0 0x7f3a24155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a1aa00640 1 -- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a2415a560 msgr2=0x7f3a24152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a1aa00640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a2415a560 0x7f3a24152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a1aa00640 1 -- 172.21.15.40:0/2483689047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a240ad980 con 0x7f3a240aec70 2024-03-24T00:33:09.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a1aa00640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a240aec70 0x7f3a24151c20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c019f90 tx=0x7f3a0c01aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a1be00640 1 -- 172.21.15.40:0/2483689047 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0b0050 con 0x7f3a240aec70 2024-03-24T00:33:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a1be00640 1 -- 172.21.15.40:0/2483689047 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c0bc070 con 0x7f3a240aec70 2024-03-24T00:33:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a2a000640 1 -- 172.21.15.40:0/2483689047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a240adc60 con 0x7f3a240aec70 2024-03-24T00:33:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a1be00640 1 -- 172.21.15.40:0/2483689047 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0b5830 con 0x7f3a240aec70 2024-03-24T00:33:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.344+0000 7f3a2a000640 1 -- 172.21.15.40:0/2483689047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a24150770 con 0x7f3a240aec70 2024-03-24T00:33:09.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.348+0000 7f3a2a000640 1 -- 172.21.15.40:0/2483689047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a24068170 con 0x7f3a240aec70 2024-03-24T00:33:09.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.348+0000 7f3a1be00640 1 -- 172.21.15.40:0/2483689047 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3a0c0b59d0 con 0x7f3a240aec70 2024-03-24T00:33:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.348+0000 7f3a1be00640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a00067690 0x7f3a00069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:09.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.348+0000 7f3a1be00640 1 -- 172.21.15.40:0/2483689047 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a0c120060 con 0x7f3a240aec70 2024-03-24T00:33:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.348+0000 7f3a23400640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a00067690 0x7f3a00069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:09.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.352+0000 7f3a23400640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a00067690 0x7f3a00069b50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3a240ce890 tx=0x7f3a14007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:09.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.352+0000 7f3a1be00640 1 -- 172.21.15.40:0/2483689047 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3a0c016070 con 0x7f3a240aec70 2024-03-24T00:33:09.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:09.552+0000 7f3a2a000640 1 -- 172.21.15.40:0/2483689047 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f3a2406ecb0 con 0x7f3a240aec70 2024-03-24T00:33:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.520+0000 7f3a1be00640 1 -- 172.21.15.40:0/2483689047 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v31) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3a0c0f0710 con 0x7f3a240aec70 2024-03-24T00:33:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.520+0000 7f3a2a000640 1 -- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a00067690 msgr2=0x7f3a00069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.520+0000 7f3a2a000640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a00067690 0x7f3a00069b50 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3a240ce890 tx=0x7f3a14007e90 comp rx=0 tx=0).stop 2024-03-24T00:33:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.520+0000 7f3a2a000640 1 -- 172.21.15.40:0/2483689047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a240aec70 msgr2=0x7f3a24151c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.520+0000 7f3a2a000640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a240aec70 0x7f3a24151c20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c019f90 tx=0x7f3a0c01aba0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.520+0000 7f3a2a000640 1 -- 172.21.15.40:0/2483689047 shutdown_connections 2024-03-24T00:33:10.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.520+0000 7f3a2a000640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a00067690 0x7f3a00069b50 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.520+0000 7f3a2a000640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a2415a560 0x7f3a24152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.520+0000 7f3a2a000640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a240aec70 0x7f3a24151c20 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.520+0000 7f3a2a000640 1 --2- 172.21.15.40:0/2483689047 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a2406d3a0 0x7f3a24155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.520+0000 7f3a2a000640 1 -- 172.21.15.40:0/2483689047 >> 172.21.15.40:0/2483689047 conn(0x7f3a240b1660 msgr2=0x7f3a24071190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.524+0000 7f3a2a000640 1 -- 172.21.15.40:0/2483689047 shutdown_connections 2024-03-24T00:33:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.524+0000 7f3a2a000640 1 -- 172.21.15.40:0/2483689047 wait complete. 2024-03-24T00:33:10.547 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:10.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faaa5800640 1 Processor -- start 2024-03-24T00:33:10.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faaa5800640 1 -- start start 2024-03-24T00:33:10.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faaa5800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faaa0147530 0x7faaa006cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:10.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faaa5800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 0x7faaa0147910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:10.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faaa5800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa01485f0 con 0x7faaa0147e50 2024-03-24T00:33:10.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faaa5800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa00b3030 con 0x7faaa0147530 2024-03-24T00:33:10.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faaa5800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa006d600 con 0x7faaa006d1c0 2024-03-24T00:33:10.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faa9ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faaa0147530 0x7faaa006cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:10.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faa9ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faaa0147530 0x7faaa006cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51976/0 (socket says 172.21.15.40:51976) 2024-03-24T00:33:10.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faa9ea00640 1 -- 172.21.15.40:0/459732365 learned_addr learned my addr 172.21.15.40:0/459732365 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:10.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faa9e000640 1 --2- 172.21.15.40:0/459732365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 0x7faaa0147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:10.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faa9ea00640 1 -- 172.21.15.40:0/459732365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 msgr2=0x7faaa0147910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:10.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faa9ea00640 1 --2- 172.21.15.40:0/459732365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 0x7faaa0147910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faa9ea00640 1 -- 172.21.15.40:0/459732365 >> v1:172.21.15.53:6789/0 conn(0x7faaa0147e50 legacy=0x7faaa0166060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:10.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faa9ea00640 1 -- 172.21.15.40:0/459732365 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa01488d0 con 0x7faaa0147530 2024-03-24T00:33:10.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.704+0000 7faa9e000640 1 --2- 172.21.15.40:0/459732365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 0x7faaa0147910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:10.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.712+0000 7faa9ea00640 1 --2- 172.21.15.40:0/459732365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faaa0147530 0x7faaa006cc80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7faa88009870 tx=0x7faa8801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d20fabc6bd8a82c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:10.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.712+0000 7faa9d600640 1 -- 172.21.15.40:0/459732365 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa880b0070 con 0x7faaa0147530 2024-03-24T00:33:10.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.712+0000 7faa9d600640 1 -- 172.21.15.40:0/459732365 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa880ab600 con 0x7faaa0147530 2024-03-24T00:33:10.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.712+0000 7faa9d600640 1 -- 172.21.15.40:0/459732365 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa880ab9d0 con 0x7faaa0147530 2024-03-24T00:33:10.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.712+0000 7faaa5800640 1 -- 172.21.15.40:0/459732365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faaa0147530 msgr2=0x7faaa006cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:10.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.712+0000 7faaa5800640 1 --2- 172.21.15.40:0/459732365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faaa0147530 0x7faaa006cc80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7faa88009870 tx=0x7faa8801c560 comp rx=0 tx=0).stop 2024-03-24T00:33:10.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.712+0000 7faaa5800640 1 -- 172.21.15.40:0/459732365 shutdown_connections 2024-03-24T00:33:10.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.712+0000 7faaa5800640 1 --2- 172.21.15.40:0/459732365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 0x7faaa0147910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.712+0000 7faaa5800640 1 --2- 172.21.15.40:0/459732365 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faaa0147530 0x7faaa006cc80 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.712+0000 7faaa5800640 1 -- 172.21.15.40:0/459732365 >> 172.21.15.40:0/459732365 conn(0x7faaa00b17b0 msgr2=0x7faaa00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:10.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.712+0000 7faaa5800640 1 -- 172.21.15.40:0/459732365 shutdown_connections 2024-03-24T00:33:10.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.712+0000 7faaa5800640 1 -- 172.21.15.40:0/459732365 wait complete. 2024-03-24T00:33:10.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faaa5800640 1 Processor -- start 2024-03-24T00:33:10.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faaa5800640 1 -- start start 2024-03-24T00:33:10.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faaa5800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 0x7faaa0165560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:10.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faaa5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faaa0147530 0x7faaa01636b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:10.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faaa5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaa0147e50 0x7faaa015df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:10.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faaa5800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa00b4650 con 0x7faaa0147530 2024-03-24T00:33:10.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faaa5800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa00b44d0 con 0x7faaa0147e50 2024-03-24T00:33:10.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faaa5800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa016f400 con 0x7faaa006d1c0 2024-03-24T00:33:10.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faa9ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 0x7faaa0165560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:10.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faa9ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 0x7faaa0165560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:43018/0 (socket says 172.21.15.40:43018) 2024-03-24T00:33:10.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faa9ea00640 1 -- 172.21.15.40:0/2675072466 learned_addr learned my addr 172.21.15.40:0/2675072466 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:10.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faa9f400640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaa0147e50 0x7faaa015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:10.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faa9ea00640 1 -- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaa0147e50 msgr2=0x7faaa015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:10.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faa9ea00640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaa0147e50 0x7faaa015df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faa9ea00640 1 -- 172.21.15.40:0/2675072466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faaa0147530 msgr2=0x7faaa01636b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:10.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faa9ea00640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faaa0147530 0x7faaa01636b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faa9ea00640 1 -- 172.21.15.40:0/2675072466 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa00719c0 con 0x7faaa006d1c0 2024-03-24T00:33:10.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faa9f400640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaa0147e50 0x7faaa015df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:10.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.716+0000 7faa9ea00640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 0x7faaa0165560 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faa8801a190 tx=0x7faa88019300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:10.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.720+0000 7faa96a00640 1 -- 172.21.15.40:0/2675072466 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa880b0050 con 0x7faaa006d1c0 2024-03-24T00:33:10.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.720+0000 7faa96a00640 1 -- 172.21.15.40:0/2675072466 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa880bb070 con 0x7faaa006d1c0 2024-03-24T00:33:10.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.720+0000 7faa96a00640 1 -- 172.21.15.40:0/2675072466 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa880b5830 con 0x7faaa006d1c0 2024-03-24T00:33:10.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.720+0000 7faaa5800640 1 -- 172.21.15.40:0/2675072466 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa0071ca0 con 0x7faaa006d1c0 2024-03-24T00:33:10.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.720+0000 7faaa5800640 1 -- 172.21.15.40:0/2675072466 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa0072260 con 0x7faaa006d1c0 2024-03-24T00:33:10.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.720+0000 7faaa5800640 1 -- 172.21.15.40:0/2675072466 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa0072640 con 0x7faaa006d1c0 2024-03-24T00:33:10.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.724+0000 7faa96a00640 1 -- 172.21.15.40:0/2675072466 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7faa880b59d0 con 0x7faaa006d1c0 2024-03-24T00:33:10.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.724+0000 7faa96a00640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faa7c0676b0 0x7faa7c069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:10.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.724+0000 7faa9e000640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faa7c0676b0 0x7faa7c069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:10.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.724+0000 7faa9e000640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faa7c0676b0 0x7faa7c069b70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faaa015dd60 tx=0x7faa9000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:10.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.728+0000 7faa96a00640 1 -- 172.21.15.40:0/2675072466 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa880ee700 con 0x7faaa006d1c0 2024-03-24T00:33:10.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.728+0000 7faa96a00640 1 -- 172.21.15.40:0/2675072466 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7faaa0072640 con 0x7faaa006d1c0 2024-03-24T00:33:10.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 -- 172.21.15.40:0/2675072466 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faaa0068170 con 0x7faaa006d1c0 2024-03-24T00:33:10.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faa96a00640 1 -- 172.21.15.40:0/2675072466 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4990 (secure 0 0 0) 0x7faa88016150 con 0x7faaa006d1c0 2024-03-24T00:33:10.929 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:10.929 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":32,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5192,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25},{"gid":5198,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:10.523943+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":5763,"mds_1":5626},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5626":{"gid":5626,"name":"d","rank":1,"incarnation":32,"state":"up:creating","state_seq":1,"addr":"172.21.15.53:6836/1600357100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1600357100},{"type":"v1","addr":"172.21.15.53:6836","nonce":1600357100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5763]},"id":3}]} 2024-03-24T00:33:10.929 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 32 2024-03-24T00:33:10.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 -- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faa7c0676b0 msgr2=0x7faa7c069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:10.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faa7c0676b0 0x7faa7c069b70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7faaa015dd60 tx=0x7faa9000b040 comp rx=0 tx=0).stop 2024-03-24T00:33:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 -- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 msgr2=0x7faaa0165560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 0x7faaa0165560 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faa8801a190 tx=0x7faa88019300 comp rx=0 tx=0).stop 2024-03-24T00:33:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 -- 172.21.15.40:0/2675072466 shutdown_connections 2024-03-24T00:33:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faa7c0676b0 0x7faa7c069b70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faaa0147e50 0x7faaa015df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faaa0147530 0x7faaa01636b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 --2- 172.21.15.40:0/2675072466 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faaa006d1c0 0x7faaa0165560 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 -- 172.21.15.40:0/2675072466 >> 172.21.15.40:0/2675072466 conn(0x7faaa00b17b0 msgr2=0x7faaa0161cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 -- 172.21.15.40:0/2675072466 shutdown_connections 2024-03-24T00:33:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:10.928+0000 7faaa5800640 1 -- 172.21.15.40:0/2675072466 wait complete. 2024-03-24T00:33:11.355 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 32, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:32:57.024524+0000', 'modified': '2024-03-24T00:33:10.523943+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 5763, 'mds_1': 5626}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5626': {'gid': 5626, 'name': 'd', 'rank': 1, 'incarnation': 32, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.53:6836/1600357100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 1600357100}, {'type': 'v1', 'addr': '172.21.15.53:6836', 'nonce': 1600357100}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5763': {'gid': 5763, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6834/224223662', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 224223662}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 224223662}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5763, 'qdb_cluster': [5763]} 2024-03-24T00:33:11.355 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5626:up:creating 2024-03-24T00:33:12.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc849c00640 1 Processor -- start 2024-03-24T00:33:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc849c00640 1 -- start start 2024-03-24T00:33:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc849c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc84406e620 0x7fc8440b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc849c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84406ea00 0x7fc8440b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc849c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc844070f60 con 0x7fc8440b2ff0 2024-03-24T00:33:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc849c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8440b4c00 con 0x7fc84406e620 2024-03-24T00:33:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc849c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8440b3620 con 0x7fc84406ea00 2024-03-24T00:33:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc842a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc84406e620 0x7fc8440b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc842000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84406ea00 0x7fc8440b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc843400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc8440b2ff0 0x7fc8440ccfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:34348/0 (socket says 172.21.15.40:34348) 2024-03-24T00:33:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc843400640 1 -- 172.21.15.40:0/3645105436 learned_addr learned my addr 172.21.15.40:0/3645105436 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc842a00640 1 -- 172.21.15.40:0/3645105436 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84406ea00 msgr2=0x7fc8440b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc842a00640 1 --2- 172.21.15.40:0/3645105436 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84406ea00 0x7fc8440b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc842a00640 1 -- 172.21.15.40:0/3645105436 >> v1:172.21.15.53:6789/0 conn(0x7fc8440b2ff0 legacy=0x7fc8440ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.496+0000 7fc842a00640 1 -- 172.21.15.40:0/3645105436 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8440b3900 con 0x7fc84406e620 2024-03-24T00:33:12.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc842a00640 1 --2- 172.21.15.40:0/3645105436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc84406e620 0x7fc8440b46c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc8340097c0 tx=0x7fc83401e620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3217cc01284631f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:12.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc841600640 1 -- 172.21.15.40:0/3645105436 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8340ad620 con 0x7fc84406e620 2024-03-24T00:33:12.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc841600640 1 -- 172.21.15.40:0/3645105436 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8340b2070 con 0x7fc84406e620 2024-03-24T00:33:12.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc841600640 1 -- 172.21.15.40:0/3645105436 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8340ad9e0 con 0x7fc84406e620 2024-03-24T00:33:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc849c00640 1 -- 172.21.15.40:0/3645105436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc84406e620 msgr2=0x7fc8440b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc849c00640 1 --2- 172.21.15.40:0/3645105436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc84406e620 0x7fc8440b46c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc8340097c0 tx=0x7fc83401e620 comp rx=0 tx=0).stop 2024-03-24T00:33:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc849c00640 1 -- 172.21.15.40:0/3645105436 shutdown_connections 2024-03-24T00:33:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc849c00640 1 --2- 172.21.15.40:0/3645105436 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84406ea00 0x7fc8440b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc849c00640 1 --2- 172.21.15.40:0/3645105436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc84406e620 0x7fc8440b46c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc849c00640 1 -- 172.21.15.40:0/3645105436 >> 172.21.15.40:0/3645105436 conn(0x7fc8440b17b0 msgr2=0x7fc8440b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc849c00640 1 -- 172.21.15.40:0/3645105436 shutdown_connections 2024-03-24T00:33:12.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc849c00640 1 -- 172.21.15.40:0/3645105436 wait complete. 2024-03-24T00:33:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.500+0000 7fc849c00640 1 Processor -- start 2024-03-24T00:33:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc849c00640 1 -- start start 2024-03-24T00:33:12.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc849c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc84406e620 0x7fc84415a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:12.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc849c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84406ea00 0x7fc844141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:12.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc849c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8440b2ff0 0x7fc844141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:12.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc849c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc844144920 con 0x7fc84406e620 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc849c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84406ede0 con 0x7fc8440b2ff0 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc849c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc844141d90 con 0x7fc84406ea00 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc842a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc84406e620 0x7fc84415a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc842a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc84406e620 0x7fc84415a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52002/0 (socket says 172.21.15.40:52002) 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc842a00640 1 -- 172.21.15.40:0/4015826938 learned_addr learned my addr 172.21.15.40:0/4015826938 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc843400640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8440b2ff0 0x7fc844141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc842000640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84406ea00 0x7fc844141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc842a00640 1 -- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84406ea00 msgr2=0x7fc844141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc842a00640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84406ea00 0x7fc844141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc842a00640 1 -- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8440b2ff0 msgr2=0x7fc844141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc842a00640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8440b2ff0 0x7fc844141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc842a00640 1 -- 172.21.15.40:0/4015826938 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc844142060 con 0x7fc84406e620 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc842a00640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc84406e620 0x7fc84415a310 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc8340af040 tx=0x7fc83401ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:12.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc83aa00640 1 -- 172.21.15.40:0/4015826938 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8340bc070 con 0x7fc84406e620 2024-03-24T00:33:12.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc83aa00640 1 -- 172.21.15.40:0/4015826938 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8340adb30 con 0x7fc84406e620 2024-03-24T00:33:12.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc83aa00640 1 -- 172.21.15.40:0/4015826938 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8340b6420 con 0x7fc84406e620 2024-03-24T00:33:12.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc849c00640 1 -- 172.21.15.40:0/4015826938 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc844142340 con 0x7fc84406e620 2024-03-24T00:33:12.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc849c00640 1 -- 172.21.15.40:0/4015826938 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc844142900 con 0x7fc84406e620 2024-03-24T00:33:12.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.504+0000 7fc849c00640 1 -- 172.21.15.40:0/4015826938 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc844068170 con 0x7fc84406e620 2024-03-24T00:33:12.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.508+0000 7fc83aa00640 1 -- 172.21.15.40:0/4015826938 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc8340bf070 con 0x7fc84406e620 2024-03-24T00:33:12.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.508+0000 7fc83aa00640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc824067690 0x7fc824069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:12.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.508+0000 7fc83aa00640 1 -- 172.21.15.40:0/4015826938 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc83411f720 con 0x7fc84406e620 2024-03-24T00:33:12.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.508+0000 7fc842000640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc824067690 0x7fc824069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:12.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.508+0000 7fc842000640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc824067690 0x7fc824069b50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc8440ae7c0 tx=0x7fc830009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:12.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.512+0000 7fc83aa00640 1 -- 172.21.15.40:0/4015826938 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc8340182a0 con 0x7fc84406e620 2024-03-24T00:33:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 -- 172.21.15.40:0/4015826938 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc844072f50 con 0x7fc84406e620 2024-03-24T00:33:12.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc83aa00640 1 -- 172.21.15.40:0/4015826938 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+5012 (secure 0 0 0) 0x7fc8340efdd0 con 0x7fc84406e620 2024-03-24T00:33:12.726 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:12.726 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5192,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:12.534259+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5763,"mds_1":5626,"mds_2":5198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5198":{"gid":5198,"name":"b","rank":2,"incarnation":34,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5626":{"gid":5626,"name":"d","rank":1,"incarnation":32,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6836/1600357100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1600357100},{"type":"v1","addr":"172.21.15.53:6836","nonce":1600357100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5198,5626,5763]},"id":3}]} 2024-03-24T00:33:12.726 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 35 2024-03-24T00:33:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 -- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc824067690 msgr2=0x7fc824069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc824067690 0x7fc824069b50 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc8440ae7c0 tx=0x7fc830009190 comp rx=0 tx=0).stop 2024-03-24T00:33:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 -- 172.21.15.40:0/4015826938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc84406e620 msgr2=0x7fc84415a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc84406e620 0x7fc84415a310 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fc8340af040 tx=0x7fc83401ce60 comp rx=0 tx=0).stop 2024-03-24T00:33:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 -- 172.21.15.40:0/4015826938 shutdown_connections 2024-03-24T00:33:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc824067690 0x7fc824069b50 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8440b2ff0 0x7fc844141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84406ea00 0x7fc844141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 --2- 172.21.15.40:0/4015826938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc84406e620 0x7fc84415a310 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 -- 172.21.15.40:0/4015826938 >> 172.21.15.40:0/4015826938 conn(0x7fc8440b17b0 msgr2=0x7fc8440cde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 -- 172.21.15.40:0/4015826938 shutdown_connections 2024-03-24T00:33:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:12.723+0000 7fc849c00640 1 -- 172.21.15.40:0/4015826938 wait complete. 2024-03-24T00:33:13.149 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:32:57.024524+0000', 'modified': '2024-03-24T00:33:12.534259+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 5763, 'mds_1': 5626, 'mds_2': 5198}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5198': {'gid': 5198, 'name': 'b', 'rank': 2, 'incarnation': 34, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.53:6837/1453842342', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6835', 'nonce': 1453842342}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 1453842342}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5626': {'gid': 5626, 'name': 'd', 'rank': 1, 'incarnation': 32, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.53:6836/1600357100', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 1600357100}, {'type': 'v1', 'addr': '172.21.15.53:6836', 'nonce': 1600357100}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5763': {'gid': 5763, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6834/224223662', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 224223662}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 224223662}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5763, 'qdb_cluster': [5198, 5626, 5763]} 2024-03-24T00:33:13.149 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T00:33:13.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:33:13.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.287+0000 7fc5de200640 1 Processor -- start 2024-03-24T00:33:13.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.287+0000 7fc5de200640 1 -- start start 2024-03-24T00:33:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5de200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc5d8147830 0x7fc5d80b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5de200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d8072b30 con 0x7fc5d8161b70 2024-03-24T00:33:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5de200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d81622e0 con 0x7fc5d8066e80 2024-03-24T00:33:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5de200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8071cb0 con 0x7fc5d8147830 2024-03-24T00:33:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5d6a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc5d8161b70 0x7fc5d8072420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35774/0 (socket says 172.21.15.40:35774) 2024-03-24T00:33:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5d6a00640 1 -- 172.21.15.40:0/725231093 learned_addr learned my addr 172.21.15.40:0/725231093 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5d7400640 1 --2- 172.21.15.40:0/725231093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc5d8147830 0x7fc5d80b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5d6000640 1 -- 172.21.15.40:0/725231093 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365313062 0 0) 0x7fc5d8072b30 con 0x7fc5d8161b70 2024-03-24T00:33:13.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5d6000640 1 -- 172.21.15.40:0/725231093 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc003530 con 0x7fc5d8161b70 2024-03-24T00:33:13.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5d6000640 1 -- 172.21.15.40:0/725231093 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330726690 0 0) 0x7fc5d81622e0 con 0x7fc5d8066e80 2024-03-24T00:33:13.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5d6000640 1 -- 172.21.15.40:0/725231093 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d8072b30 con 0x7fc5d8066e80 2024-03-24T00:33:13.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5d7400640 1 -- 172.21.15.40:0/725231093 >> v1:172.21.15.53:6789/0 conn(0x7fc5d8066e80 legacy=0x7fc5d806dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5d7400640 1 -- 172.21.15.40:0/725231093 >> v1:172.21.15.40:6789/0 conn(0x7fc5d8161b70 legacy=0x7fc5d8072420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.291+0000 7fc5d7400640 1 -- 172.21.15.40:0/725231093 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d81625c0 con 0x7fc5d8147830 2024-03-24T00:33:13.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5d7400640 1 --2- 172.21.15.40:0/725231093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc5d8147830 0x7fc5d80b5c40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80097f0 tx=0x7fc5c801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75401b23b26d04ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5d6000640 1 -- 172.21.15.40:0/725231093 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80b0070 con 0x7fc5d8147830 2024-03-24T00:33:13.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5d6000640 1 -- 172.21.15.40:0/725231093 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c80ab600 con 0x7fc5d8147830 2024-03-24T00:33:13.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5d6000640 1 -- 172.21.15.40:0/725231093 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80ab9d0 con 0x7fc5d8147830 2024-03-24T00:33:13.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5de200640 1 -- 172.21.15.40:0/725231093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc5d8147830 msgr2=0x7fc5d80b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5de200640 1 --2- 172.21.15.40:0/725231093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc5d8147830 0x7fc5d80b5c40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80097f0 tx=0x7fc5c801c450 comp rx=0 tx=0).stop 2024-03-24T00:33:13.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5de200640 1 -- 172.21.15.40:0/725231093 shutdown_connections 2024-03-24T00:33:13.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5de200640 1 --2- 172.21.15.40:0/725231093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc5d8147830 0x7fc5d80b5c40 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5de200640 1 -- 172.21.15.40:0/725231093 >> 172.21.15.40:0/725231093 conn(0x7fc5d80d0c20 msgr2=0x7fc5d80d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5de200640 1 -- 172.21.15.40:0/725231093 shutdown_connections 2024-03-24T00:33:13.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5de200640 1 -- 172.21.15.40:0/725231093 wait complete. 2024-03-24T00:33:13.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5de200640 1 Processor -- start 2024-03-24T00:33:13.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.295+0000 7fc5de200640 1 -- start start 2024-03-24T00:33:13.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5de200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5d8066e80 0x7fc5d81425d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5de200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc5d8147830 0x7fc5d80af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5de200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d8161b70 0x7fc5d8143de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5de200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d80676a0 con 0x7fc5d8066e80 2024-03-24T00:33:13.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5de200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8161f50 con 0x7fc5d8161b70 2024-03-24T00:33:13.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5de200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8142140 con 0x7fc5d8147830 2024-03-24T00:33:13.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5d7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5d8066e80 0x7fc5d81425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5d7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5d8066e80 0x7fc5d81425d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52018/0 (socket says 172.21.15.40:52018) 2024-03-24T00:33:13.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5d7400640 1 -- 172.21.15.40:0/1963002988 learned_addr learned my addr 172.21.15.40:0/1963002988 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5d7e00640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d8161b70 0x7fc5d8143de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc5d8147830 0x7fc5d80af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5d7400640 1 -- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc5d8147830 msgr2=0x7fc5d80af2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5d7400640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc5d8147830 0x7fc5d80af2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5d7400640 1 -- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d8161b70 msgr2=0x7fc5d8143de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5d7400640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d8161b70 0x7fc5d8143de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5d7400640 1 -- 172.21.15.40:0/1963002988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8141810 con 0x7fc5d8066e80 2024-03-24T00:33:13.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc5d8147830 0x7fc5d80af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:33:13.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5d7400640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5d8066e80 0x7fc5d81425d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8019060 tx=0x7fc5c80191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5cf400640 1 -- 172.21.15.40:0/1963002988 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80b9020 con 0x7fc5d8066e80 2024-03-24T00:33:13.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5cf400640 1 -- 172.21.15.40:0/1963002988 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c80b5480 con 0x7fc5d8066e80 2024-03-24T00:33:13.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5de200640 1 -- 172.21.15.40:0/1963002988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d8141af0 con 0x7fc5d8066e80 2024-03-24T00:33:13.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5cf400640 1 -- 172.21.15.40:0/1963002988 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80b0040 con 0x7fc5d8066e80 2024-03-24T00:33:13.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5de200640 1 -- 172.21.15.40:0/1963002988 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d815aa10 con 0x7fc5d8066e80 2024-03-24T00:33:13.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.299+0000 7fc5cf400640 1 -- 172.21.15.40:0/1963002988 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc5c80bc020 con 0x7fc5d8066e80 2024-03-24T00:33:13.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5cf400640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc5b8067660 0x7fc5b8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5cf400640 1 -- 172.21.15.40:0/1963002988 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5c811f910 con 0x7fc5d8066e80 2024-03-24T00:33:13.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc5b8067660 0x7fc5b8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5e16b0000 1 Processor -- start 2024-03-24T00:33:13.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5e16b0000 1 -- start start 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc5b8067660 0x7fc5b8069b20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc5d80b3e60 tx=0x7fc5c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5e16b0000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc8920 0x560328dc8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5e16b0000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328dc9240 0x560328d7eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5e16b0000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328d7eff0 0x560328d7f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5e16b0000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560328bcc680 con 0x560328dc9240 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5e16b0000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x560328b53b00 con 0x560328dc8920 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5e16b0000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x560328bc8720 con 0x560328d7eff0 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328dc9240 0x560328d7eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328dc9240 0x560328d7eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52026/0 (socket says 172.21.15.40:52026) 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d6a00640 1 -- 172.21.15.40:0/3642031559 learned_addr learned my addr 172.21.15.40:0/3642031559 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d7400640 1 --2- 172.21.15.40:0/3642031559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc8920 0x560328dc8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d7e00640 1 --2- 172.21.15.40:0/3642031559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328d7eff0 0x560328d7f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d6a00640 1 -- 172.21.15.40:0/3642031559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328d7eff0 msgr2=0x560328d7f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/3642031559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328d7eff0 0x560328d7f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d6a00640 1 -- 172.21.15.40:0/3642031559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc8920 msgr2=0x560328dc8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/3642031559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc8920 0x560328dc8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d6a00640 1 -- 172.21.15.40:0/3642031559 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560328ba4940 con 0x560328dc9240 2024-03-24T00:33:13.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d7400640 1 --2- 172.21.15.40:0/3642031559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc8920 0x560328dc8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:13.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/3642031559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328dc9240 0x560328d7eab0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4005b70 tx=0x7fc5c4010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de81e70dcacc68c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5c3e00640 1 -- 172.21.15.40:0/3642031559 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c400db50 con 0x560328dc9240 2024-03-24T00:33:13.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5c3e00640 1 -- 172.21.15.40:0/3642031559 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c400dcf0 con 0x560328dc9240 2024-03-24T00:33:13.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.303+0000 7fc5c3e00640 1 -- 172.21.15.40:0/3642031559 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c400dea0 con 0x560328dc9240 2024-03-24T00:33:13.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3642031559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328dc9240 msgr2=0x560328d7eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/3642031559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328dc9240 0x560328d7eab0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc5c4005b70 tx=0x7fc5c4010000 comp rx=0 tx=0).stop 2024-03-24T00:33:13.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3642031559 shutdown_connections 2024-03-24T00:33:13.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/3642031559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328d7eff0 0x560328d7f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/3642031559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328dc9240 0x560328d7eab0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/3642031559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc8920 0x560328dc8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3642031559 >> 172.21.15.40:0/3642031559 conn(0x560328d7afa0 msgr2=0x560328d7d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3642031559 shutdown_connections 2024-03-24T00:33:13.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3642031559 wait complete. 2024-03-24T00:33:13.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 -- start start 2024-03-24T00:33:13.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 Processor -- start 2024-03-24T00:33:13.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dadd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 0x560328dae280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328dae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560328bcb990 con 0x560328d7eff0 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x560328b86ae0 con 0x560328dc9240 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x560328b64750 con 0x560328dc8920 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 10 client.0 _ll_register_callbacks cb 0x560328d69ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5d7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dadd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5d7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dadd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52032/0 (socket says 172.21.15.40:52032) 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5d7400640 1 -- 172.21.15.40:0/153466751 learned_addr learned my addr 172.21.15.40:0/153466751 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 0x560328dae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5d7e00640 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328dae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5d7400640 1 -- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 msgr2=0x560328dae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5d7400640 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 0x560328dae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5d7400640 1 -- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 msgr2=0x560328dae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5d7400640 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328dae7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5d7400640 1 -- 172.21.15.40:0/153466751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560328c11d80 con 0x560328d7eff0 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5d7e00640 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328dae7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:13.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5d7400640 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dadd40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80efd00 tx=0x7fc5c80ab600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4532e5248bf937a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5c0c00640 1 -- 172.21.15.40:0/153466751 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c810c060 con 0x560328d7eff0 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5c0c00640 1 -- 172.21.15.40:0/153466751 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5c810e0b0 con 0x560328d7eff0 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 -- 172.21.15.40:0/153466751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc5c400dcf0 con 0x560328d7eff0 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5c0c00640 1 -- 172.21.15.40:0/153466751 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5c811d050 con 0x560328d7eff0 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 10 client.5979 fetch_fsmap learned FSMap version 35 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.307+0000 7fc5e16b0000 1 -- 172.21.15.40:0/153466751 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x560328b53d10 con 0x560328d7eff0 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5c0c00640 1 -- 172.21.15.40:0/153466751 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c400dcf0 con 0x560328d7eff0 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5c0c00640 1 -- 172.21.15.40:0/153466751 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fc5c80effd0 con 0x560328d7eff0 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 10 client.5979 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 10 client.5979 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 20 client.5979 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x560328daf030 0x560328ddaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 4 client.5979 mds_command: new command op to 5198 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:13.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 -- 172.21.15.40:0/153466751 --> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560328b67b60 con 0x560328daf030 2024-03-24T00:33:13.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x560328daf030 0x560328ddaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5c0c00640 10 client.5979 ms_handle_connect on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:13.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x560328daf030 0x560328ddaea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=636d09c909885d55 server_cookie=afddc2635095dcd9 in_seq=0 out_seq=0 2024-03-24T00:33:13.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5c0c00640 1 -- 172.21.15.40:0/153466751 <== mds.2 v2:172.21.15.53:6835/1453842342 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560328b67b60 con 0x560328daf030 2024-03-24T00:33:13.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5c0c00640 10 client.5979 handle_command_reply: tid=0 2024-03-24T00:33:13.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 client.5979 shutdown 2024-03-24T00:33:13.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 -- 172.21.15.40:0/153466751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 msgr2=0x560328dadd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dadd40 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80efd00 tx=0x7fc5c80ab600 comp rx=0 tx=0).stop 2024-03-24T00:33:13.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 -- 172.21.15.40:0/153466751 shutdown_connections 2024-03-24T00:33:13.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x560328daf030 0x560328ddaea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328dae7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 0x560328dae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5c0c00640 0 client.5979 ms_handle_reset on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:13.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/153466751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dadd40 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 -- 172.21.15.40:0/153466751 >> 172.21.15.40:0/153466751 conn(0x560328d7afa0 msgr2=0x560328d7dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 -- 172.21.15.40:0/153466751 shutdown_connections 2024-03-24T00:33:13.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 1 -- 172.21.15.40:0/153466751 wait complete. 2024-03-24T00:33:13.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.311+0000 7fc5e16b0000 20 client.5979 trim_cache size 0 max 16384 2024-03-24T00:33:13.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.327+0000 7fc5e16b0000 1 Processor -- start 2024-03-24T00:33:13.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.327+0000 7fc5e16b0000 1 -- start start 2024-03-24T00:33:13.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dabef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 0x560328dc74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328dc7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8203640 con 0x560328d7eff0 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8202b80 con 0x560328dc9240 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8202fe0 con 0x560328dc8920 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5d7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dabef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5d7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dabef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52038/0 (socket says 172.21.15.40:52038) 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5d7400640 1 -- 172.21.15.40:0/648428403 learned_addr learned my addr 172.21.15.40:0/648428403 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/648428403 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 0x560328dc74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5d7e00640 1 --2- 172.21.15.40:0/648428403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328dc7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5d7400640 1 -- 172.21.15.40:0/648428403 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 msgr2=0x560328dc74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5d7400640 1 --2- 172.21.15.40:0/648428403 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 0x560328dc74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5d7400640 1 -- 172.21.15.40:0/648428403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 msgr2=0x560328dc7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5d7400640 1 --2- 172.21.15.40:0/648428403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328dc7a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5d7400640 1 -- 172.21.15.40:0/648428403 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560328b99540 con 0x560328d7eff0 2024-03-24T00:33:13.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5d7400640 1 --2- 172.21.15.40:0/648428403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dabef0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8097e10 tx=0x7fc5c80b7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af3a5dd584e976a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5c3400640 1 -- 172.21.15.40:0/648428403 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80ef6d0 con 0x560328d7eff0 2024-03-24T00:33:13.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5c3400640 1 -- 172.21.15.40:0/648428403 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c80f8480 con 0x560328d7eff0 2024-03-24T00:33:13.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5c3400640 1 -- 172.21.15.40:0/648428403 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80f3f10 con 0x560328d7eff0 2024-03-24T00:33:13.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 -- 172.21.15.40:0/648428403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 msgr2=0x560328dabef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/648428403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dabef0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8097e10 tx=0x7fc5c80b7000 comp rx=0 tx=0).stop 2024-03-24T00:33:13.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 -- 172.21.15.40:0/648428403 shutdown_connections 2024-03-24T00:33:13.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/648428403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328dc7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/648428403 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 0x560328dc74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/648428403 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dabef0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 -- 172.21.15.40:0/648428403 >> 172.21.15.40:0/648428403 conn(0x560328d7afa0 msgr2=0x560328dab5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 -- 172.21.15.40:0/648428403 shutdown_connections 2024-03-24T00:33:13.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 -- 172.21.15.40:0/648428403 wait complete. 2024-03-24T00:33:13.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 -- start start 2024-03-24T00:33:13.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 Processor -- start 2024-03-24T00:33:13.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dc7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 0x560328ddbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328ddc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560328b74a80 con 0x560328d7eff0 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d813f540 con 0x560328dc9240 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x560328c89700 con 0x560328dc8920 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.331+0000 7fc5e16b0000 10 client.0 _ll_register_callbacks cb 0x7fc5c80ef8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dc7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dc7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52040/0 (socket says 172.21.15.40:52040) 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d7400640 1 -- 172.21.15.40:0/3465223711 learned_addr learned my addr 172.21.15.40:0/3465223711 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d7e00640 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328ddc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 0x560328ddbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d7400640 1 -- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 msgr2=0x560328ddbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d7400640 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 0x560328ddbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d7400640 1 -- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 msgr2=0x560328ddc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d7400640 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328ddc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d7400640 1 -- 172.21.15.40:0/3465223711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560328b604b0 con 0x560328d7eff0 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d7e00640 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328ddc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d7400640 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dc7dd0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80efd00 tx=0x7fc5c811dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6998ac73a662a872 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5c0c00640 1 -- 172.21.15.40:0/3465223711 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c80ef6d0 con 0x560328d7eff0 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5c0c00640 1 -- 172.21.15.40:0/3465223711 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc5c80f7ff0 con 0x560328d7eff0 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3465223711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc5c80f8480 con 0x560328d7eff0 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5c0c00640 1 -- 172.21.15.40:0/3465223711 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5c810e2c0 con 0x560328d7eff0 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 10 client.5985 fetch_fsmap learned FSMap version 35 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3465223711 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x560328b732f0 con 0x560328d7eff0 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5c0c00640 1 -- 172.21.15.40:0/3465223711 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c80f8480 con 0x560328d7eff0 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5c0c00640 1 -- 172.21.15.40:0/3465223711 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fc5c8115070 con 0x560328d7eff0 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 10 client.5985 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 10 client.5985 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 20 client.5985 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:13.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x560328e2d5c0 0x560328e2d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 4 client.5985 mds_command: new command op to 5198 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:33:13.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3465223711 --> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560328bd9bc0 con 0x560328e2d5c0 2024-03-24T00:33:13.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x560328e2d5c0 0x560328e2d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5d6a00640 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x560328e2d5c0 0x560328e2d9a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b70a9508d355788c server_cookie=9e76820d72d0ee7d in_seq=0 out_seq=0 2024-03-24T00:33:13.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5c0c00640 10 client.5985 ms_handle_connect on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:13.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5c0c00640 1 -- 172.21.15.40:0/3465223711 <== mds.2 v2:172.21.15.53:6835/1453842342 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x560328bd9bc0 con 0x560328e2d5c0 2024-03-24T00:33:13.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5c0c00640 10 client.5985 handle_command_reply: tid=0 2024-03-24T00:33:13.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 client.5985 shutdown 2024-03-24T00:33:13.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3465223711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 msgr2=0x560328dc7dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dc7dd0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc5c80efd00 tx=0x7fc5c811dc70 comp rx=0 tx=0).stop 2024-03-24T00:33:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3465223711 shutdown_connections 2024-03-24T00:33:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x560328e2d5c0 0x560328e2d9a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560328dc9240 0x560328ddc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560328dc8920 0x560328ddbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 --2- 172.21.15.40:0/3465223711 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560328d7eff0 0x560328dc7dd0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5c0c00640 0 client.5985 ms_handle_reset on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3465223711 >> 172.21.15.40:0/3465223711 conn(0x560328d7afa0 msgr2=0x560328dad190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3465223711 shutdown_connections 2024-03-24T00:33:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 1 -- 172.21.15.40:0/3465223711 wait complete. 2024-03-24T00:33:13.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5e16b0000 20 client.5985 trim_cache size 0 max 16384 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5198, 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 1.803589412, 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 35, 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 44, 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 44, 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 20.520546276000001 2024-03-24T00:33:13.344 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:13.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5de200640 1 -- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc5b8067660 msgr2=0x7fc5b8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5de200640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc5b8067660 0x7fc5b8069b20 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc5d80b3e60 tx=0x7fc5c4009040 comp rx=0 tx=0).stop 2024-03-24T00:33:13.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5de200640 1 -- 172.21.15.40:0/1963002988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5d8066e80 msgr2=0x7fc5d81425d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5de200640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5d8066e80 0x7fc5d81425d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8019060 tx=0x7fc5c80191f0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5de200640 1 -- 172.21.15.40:0/1963002988 shutdown_connections 2024-03-24T00:33:13.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5de200640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc5b8067660 0x7fc5b8069b20 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5de200640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5d8161b70 0x7fc5d8143de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5de200640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc5d8147830 0x7fc5d80af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5de200640 1 --2- 172.21.15.40:0/1963002988 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc5d8066e80 0x7fc5d81425d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.335+0000 7fc5de200640 1 -- 172.21.15.40:0/1963002988 >> 172.21.15.40:0/1963002988 conn(0x7fc5d80d0c20 msgr2=0x7fc5d806f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.339+0000 7fc5de200640 1 -- 172.21.15.40:0/1963002988 shutdown_connections 2024-03-24T00:33:13.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.339+0000 7fc5de200640 1 -- 172.21.15.40:0/1963002988 wait complete. 2024-03-24T00:33:13.359 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T00:33:13.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.495+0000 7f4938800640 1 Processor -- start 2024-03-24T00:33:13.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.495+0000 7f4938800640 1 -- start start 2024-03-24T00:33:13.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4938800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f49300b2910 0x7f49300d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4938800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300af170 0x7f4930165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4938800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49300b1e50 con 0x7f493015f600 2024-03-24T00:33:13.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4938800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49300b3630 con 0x7f49300b2910 2024-03-24T00:33:13.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4938800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49300b2d80 con 0x7f49300af170 2024-03-24T00:33:13.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4935e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f493015f600 0x7f49300aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35782/0 (socket says 172.21.15.40:35782) 2024-03-24T00:33:13.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4935e00640 1 -- 172.21.15.40:0/3254955287 learned_addr learned my addr 172.21.15.40:0/3254955287 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:13.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4935400640 1 --2- 172.21.15.40:0/3254955287 >> v2:172.21.15.53:3300/0 conn(0x7f49300b2910 0x7f49300d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4936800640 1 --2- 172.21.15.40:0/3254955287 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300af170 0x7f4930165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4934a00640 1 -- 172.21.15.40:0/3254955287 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233578843 0 0) 0x7f49300b1e50 con 0x7f493015f600 2024-03-24T00:33:13.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4934a00640 1 -- 172.21.15.40:0/3254955287 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4918003530 con 0x7f493015f600 2024-03-24T00:33:13.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4936800640 1 -- 172.21.15.40:0/3254955287 >> v2:172.21.15.53:3300/0 conn(0x7f49300b2910 msgr2=0x7f49300d2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4936800640 1 --2- 172.21.15.40:0/3254955287 >> v2:172.21.15.53:3300/0 conn(0x7f49300b2910 0x7f49300d2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4936800640 1 -- 172.21.15.40:0/3254955287 >> v1:172.21.15.40:6789/0 conn(0x7f493015f600 legacy=0x7f49300aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.499+0000 7f4936800640 1 -- 172.21.15.40:0/3254955287 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49300b20f0 con 0x7f49300af170 2024-03-24T00:33:13.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4936800640 1 --2- 172.21.15.40:0/3254955287 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300af170 0x7f4930165e40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f492c009d90 tx=0x7f492c01f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed9b639632e45138 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4934a00640 1 -- 172.21.15.40:0/3254955287 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c0b4070 con 0x7f49300af170 2024-03-24T00:33:13.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4934a00640 1 -- 172.21.15.40:0/3254955287 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492c0af600 con 0x7f49300af170 2024-03-24T00:33:13.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4934a00640 1 -- 172.21.15.40:0/3254955287 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c0af9d0 con 0x7f49300af170 2024-03-24T00:33:13.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4938800640 1 -- 172.21.15.40:0/3254955287 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300af170 msgr2=0x7f4930165e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4938800640 1 --2- 172.21.15.40:0/3254955287 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300af170 0x7f4930165e40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f492c009d90 tx=0x7f492c01f150 comp rx=0 tx=0).stop 2024-03-24T00:33:13.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4938800640 1 -- 172.21.15.40:0/3254955287 shutdown_connections 2024-03-24T00:33:13.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4938800640 1 --2- 172.21.15.40:0/3254955287 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300af170 0x7f4930165e40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4938800640 1 --2- 172.21.15.40:0/3254955287 >> v2:172.21.15.53:3300/0 conn(0x7f49300b2910 0x7f49300d2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4938800640 1 -- 172.21.15.40:0/3254955287 >> 172.21.15.40:0/3254955287 conn(0x7f49300d1d10 msgr2=0x7f49300d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4938800640 1 -- 172.21.15.40:0/3254955287 shutdown_connections 2024-03-24T00:33:13.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4938800640 1 -- 172.21.15.40:0/3254955287 wait complete. 2024-03-24T00:33:13.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4938800640 1 Processor -- start 2024-03-24T00:33:13.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.503+0000 7f4938800640 1 -- start start 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4938800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49300af170 0x7f49300b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4938800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300b2910 0x7f493006c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4938800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f493015f600 0x7f49301f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4938800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49300b2fa0 con 0x7f493015f600 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4938800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49300b2740 con 0x7f49300af170 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4938800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f493015f9e0 con 0x7f49300b2910 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4936800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f493015f600 0x7f49301f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4935400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300b2910 0x7f493006c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4935e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49300af170 0x7f49300b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4935400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300b2910 0x7f493006c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:43098/0 (socket says 172.21.15.40:43098) 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4935400640 1 -- 172.21.15.40:0/2498845998 learned_addr learned my addr 172.21.15.40:0/2498845998 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4935400640 1 -- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49300af170 msgr2=0x7f49300b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4935400640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49300af170 0x7f49300b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4935400640 1 -- 172.21.15.40:0/2498845998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f493015f600 msgr2=0x7f49301f7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4935400640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f493015f600 0x7f49301f7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4935400640 1 -- 172.21.15.40:0/2498845998 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49300cab30 con 0x7f49300b2910 2024-03-24T00:33:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4936800640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f493015f600 0x7f49301f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4935400640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300b2910 0x7f493006c560 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f492000e930 tx=0x7f492000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f492a000640 1 -- 172.21.15.40:0/2498845998 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4920017070 con 0x7f49300b2910 2024-03-24T00:33:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f492a000640 1 -- 172.21.15.40:0/2498845998 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492000ca00 con 0x7f49300b2910 2024-03-24T00:33:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f492a000640 1 -- 172.21.15.40:0/2498845998 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492000cdb0 con 0x7f49300b2910 2024-03-24T00:33:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4938800640 1 -- 172.21.15.40:0/2498845998 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49301f8040 con 0x7f49300b2910 2024-03-24T00:33:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.507+0000 7f4938800640 1 -- 172.21.15.40:0/2498845998 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49300cadd0 con 0x7f49300b2910 2024-03-24T00:33:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f492a000640 1 -- 172.21.15.40:0/2498845998 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f492001a070 con 0x7f49300b2910 2024-03-24T00:33:13.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f492a000640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4914067660 0x7f4914069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f492a000640 1 -- 172.21.15.40:0/2498845998 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4920085830 con 0x7f49300b2910 2024-03-24T00:33:13.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4935e00640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4914067660 0x7f4914069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f493bcda000 1 Processor -- start 2024-03-24T00:33:13.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f493bcda000 1 -- start start 2024-03-24T00:33:13.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4935e00640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4914067660 0x7f4914069b20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4924007fd0 tx=0x7f4924009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f493bcda000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9c7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f493bcda000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 0x55b5ac97dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f493bcda000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac97e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f493bcda000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ac7cb680 con 0x55b5ac97dff0 2024-03-24T00:33:13.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f493bcda000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ac752b00 con 0x55b5ac9c7920 2024-03-24T00:33:13.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f493bcda000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5ac7c7720 con 0x55b5ac9c8240 2024-03-24T00:33:13.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4936800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac97e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4935e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9c7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4936800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac97e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52068/0 (socket says 172.21.15.40:52068) 2024-03-24T00:33:13.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4936800640 1 -- 172.21.15.40:0/3130216030 learned_addr learned my addr 172.21.15.40:0/3130216030 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4935400640 1 --2- 172.21.15.40:0/3130216030 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 0x55b5ac97dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4936800640 1 -- 172.21.15.40:0/3130216030 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 msgr2=0x55b5ac97dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4936800640 1 --2- 172.21.15.40:0/3130216030 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 0x55b5ac97dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4936800640 1 -- 172.21.15.40:0/3130216030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 msgr2=0x55b5ac9c7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4936800640 1 --2- 172.21.15.40:0/3130216030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9c7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4936800640 1 -- 172.21.15.40:0/3130216030 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5ac7a3940 con 0x55b5ac97dff0 2024-03-24T00:33:13.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f4936800640 1 --2- 172.21.15.40:0/3130216030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac97e3d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f492c009d90 tx=0x7f492c004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8563e669adcaeb5d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.511+0000 7f491ea00640 1 -- 172.21.15.40:0/3130216030 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c0b4040 con 0x55b5ac97dff0 2024-03-24T00:33:13.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f491ea00640 1 -- 172.21.15.40:0/3130216030 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492c0afb60 con 0x55b5ac97dff0 2024-03-24T00:33:13.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f491ea00640 1 -- 172.21.15.40:0/3130216030 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c0bb460 con 0x55b5ac97dff0 2024-03-24T00:33:13.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 -- 172.21.15.40:0/3130216030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 msgr2=0x55b5ac97e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 --2- 172.21.15.40:0/3130216030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac97e3d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f492c009d90 tx=0x7f492c004ac0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 -- 172.21.15.40:0/3130216030 shutdown_connections 2024-03-24T00:33:13.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 --2- 172.21.15.40:0/3130216030 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac97e3d0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 --2- 172.21.15.40:0/3130216030 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 0x55b5ac97dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 --2- 172.21.15.40:0/3130216030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9c7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 -- 172.21.15.40:0/3130216030 >> 172.21.15.40:0/3130216030 conn(0x55b5ac979fa0 msgr2=0x55b5ac97c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 -- 172.21.15.40:0/3130216030 shutdown_connections 2024-03-24T00:33:13.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 -- 172.21.15.40:0/3130216030 wait complete. 2024-03-24T00:33:13.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 -- start start 2024-03-24T00:33:13.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 Processor -- start 2024-03-24T00:33:13.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac9acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9ad280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 0x55b5ac9ad7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ac7ca990 con 0x55b5ac97dff0 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ac785ae0 con 0x55b5ac9c7920 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5ac763750 con 0x55b5ac9c8240 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.515+0000 7f493bcda000 10 client.0 _ll_register_callbacks cb 0x55b5ac968ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac9acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac9acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52084/0 (socket says 172.21.15.40:52084) 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935e00640 1 -- 172.21.15.40:0/3738454087 learned_addr learned my addr 172.21.15.40:0/3738454087 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4936800640 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 0x55b5ac9ad7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935400640 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935e00640 1 -- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 msgr2=0x55b5ac9ad7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935e00640 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 0x55b5ac9ad7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935e00640 1 -- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 msgr2=0x55b5ac9ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935e00640 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9ad280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935e00640 1 -- 172.21.15.40:0/3738454087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5ac810d80 con 0x55b5ac97dff0 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935400640 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935e00640 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac9acd40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4924015770 tx=0x7f4924000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b71b88ee0660c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4913400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4913400640 1 -- 172.21.15.40:0/3738454087 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924013800 con 0x55b5ac97dff0 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4913400640 1 -- 172.21.15.40:0/3738454087 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4924010040 con 0x55b5ac97dff0 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 -- 172.21.15.40:0/3738454087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f492c0afb60 con 0x55b5ac97dff0 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4913400640 1 -- 172.21.15.40:0/3738454087 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f492402e050 con 0x55b5ac97dff0 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 10 client.5997 fetch_fsmap learned FSMap version 35 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 -- 172.21.15.40:0/3738454087 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55b5ac752d10 con 0x55b5ac97dff0 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4913400640 1 -- 172.21.15.40:0/3738454087 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492c0afb60 con 0x55b5ac97dff0 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4913400640 1 -- 172.21.15.40:0/3738454087 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f4924031020 con 0x55b5ac97dff0 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 10 client.5997 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 10 client.5997 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:33:13.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 20 client.5997 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x55b5ac9ae090 0x55b5ac9d9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 4 client.5997 mds_command: new command op to 5626 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 -- 172.21.15.40:0/3738454087 --> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5ac766b60 con 0x55b5ac9ae090 2024-03-24T00:33:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935400640 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x55b5ac9ae090 0x55b5ac9d9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4935400640 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x55b5ac9ae090 0x55b5ac9d9ea0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f2c0ce76d5665e8a server_cookie=7547b8671a19c351 in_seq=0 out_seq=0 2024-03-24T00:33:13.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4913400640 10 client.5997 ms_handle_connect on v2:172.21.15.53:6834/1600357100 2024-03-24T00:33:13.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4913400640 1 -- 172.21.15.40:0/3738454087 <== mds.1 v2:172.21.15.53:6834/1600357100 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b5ac766b60 con 0x55b5ac9ae090 2024-03-24T00:33:13.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f4913400640 10 client.5997 handle_command_reply: tid=0 2024-03-24T00:33:13.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 client.5997 shutdown 2024-03-24T00:33:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 -- 172.21.15.40:0/3738454087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 msgr2=0x55b5ac9acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac9acd40 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4924015770 tx=0x7f4924000bd0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 -- 172.21.15.40:0/3738454087 shutdown_connections 2024-03-24T00:33:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x55b5ac9ae090 0x55b5ac9d9ea0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 0x55b5ac9ad7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 --2- 172.21.15.40:0/3738454087 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac9acd40 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.519+0000 7f493bcda000 1 -- 172.21.15.40:0/3738454087 >> 172.21.15.40:0/3738454087 conn(0x55b5ac979fa0 msgr2=0x55b5ac97cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.523+0000 7f4913400640 0 client.5997 ms_handle_reset on v2:172.21.15.53:6834/1600357100 2024-03-24T00:33:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.523+0000 7f493bcda000 1 -- 172.21.15.40:0/3738454087 shutdown_connections 2024-03-24T00:33:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.523+0000 7f493bcda000 1 -- 172.21.15.40:0/3738454087 wait complete. 2024-03-24T00:33:13.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.523+0000 7f493bcda000 20 client.5997 trim_cache size 0 max 16384 2024-03-24T00:33:13.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.551+0000 7f493bcda000 1 Processor -- start 2024-03-24T00:33:13.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 -- start start 2024-03-24T00:33:13.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac9c6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9c6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 0x55b5aca29170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49302000a0 con 0x55b5ac97dff0 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49301ff5a0 con 0x55b5ac9c7920 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49301ffa80 con 0x55b5ac9c8240 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f4935e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac9c6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f4935e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac9c6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52086/0 (socket says 172.21.15.40:52086) 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f4935e00640 1 -- 172.21.15.40:0/1256050191 learned_addr learned my addr 172.21.15.40:0/1256050191 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f4935400640 1 --2- 172.21.15.40:0/1256050191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9c6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f4936800640 1 --2- 172.21.15.40:0/1256050191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 0x55b5aca29170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f4935e00640 1 -- 172.21.15.40:0/1256050191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 msgr2=0x55b5aca29170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f4935e00640 1 --2- 172.21.15.40:0/1256050191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 0x55b5aca29170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f4935e00640 1 -- 172.21.15.40:0/1256050191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 msgr2=0x55b5ac9c6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f4935e00640 1 --2- 172.21.15.40:0/1256050191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9c6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f4935e00640 1 -- 172.21.15.40:0/1256050191 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5ac798540 con 0x55b5ac97dff0 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f4935e00640 1 --2- 172.21.15.40:0/1256050191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac9c6530 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4924015740 tx=0x7f4924020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dab90975e97bf91 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f491e000640 1 -- 172.21.15.40:0/1256050191 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924013a00 con 0x55b5ac97dff0 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f491e000640 1 -- 172.21.15.40:0/1256050191 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4924026070 con 0x55b5ac97dff0 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f491e000640 1 -- 172.21.15.40:0/1256050191 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924021630 con 0x55b5ac97dff0 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 -- 172.21.15.40:0/1256050191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 msgr2=0x55b5ac9c6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 --2- 172.21.15.40:0/1256050191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac9c6530 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4924015740 tx=0x7f4924020000 comp rx=0 tx=0).stop 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 -- 172.21.15.40:0/1256050191 shutdown_connections 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 --2- 172.21.15.40:0/1256050191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c8240 0x55b5aca29170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 --2- 172.21.15.40:0/1256050191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac9c7920 0x55b5ac9c6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 --2- 172.21.15.40:0/1256050191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac97dff0 0x55b5ac9c6530 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 -- 172.21.15.40:0/1256050191 >> 172.21.15.40:0/1256050191 conn(0x55b5ac979fa0 msgr2=0x55b5ac9d4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 -- 172.21.15.40:0/1256050191 shutdown_connections 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 -- 172.21.15.40:0/1256050191 wait complete. 2024-03-24T00:33:13.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.555+0000 7f493bcda000 1 -- start start 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 1 Processor -- start 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac97dff0 0x55b5ac9daef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c7920 0x55b5ac9db430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac9c8240 0x55b5ac9db970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5ac773a80 con 0x55b5ac9c8240 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49301f4880 con 0x55b5ac97dff0 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5ac888700 con 0x55b5ac9c7920 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 10 client.0 _ll_register_callbacks cb 0x7f49240015d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4936800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac9c8240 0x55b5ac9db970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4936800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac9c8240 0x55b5ac9db970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52088/0 (socket says 172.21.15.40:52088) 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4936800640 1 -- 172.21.15.40:0/3715267176 learned_addr learned my addr 172.21.15.40:0/3715267176 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4935400640 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c7920 0x55b5ac9db430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4935e00640 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac97dff0 0x55b5ac9daef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4936800640 1 -- 172.21.15.40:0/3715267176 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c7920 msgr2=0x55b5ac9db430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4936800640 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c7920 0x55b5ac9db430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4936800640 1 -- 172.21.15.40:0/3715267176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac97dff0 msgr2=0x55b5ac9daef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4936800640 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac97dff0 0x55b5ac9daef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4936800640 1 -- 172.21.15.40:0/3715267176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5ac75f4b0 con 0x55b5ac9c8240 2024-03-24T00:33:13.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4936800640 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac9c8240 0x55b5ac9db970 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f492c004b40 tx=0x7f492c01d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8827796e0e8ebb00 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4913400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:13.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4913400640 1 -- 172.21.15.40:0/3715267176 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492c0b4050 con 0x55b5ac9c8240 2024-03-24T00:33:13.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4913400640 1 -- 172.21.15.40:0/3715267176 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f492c0af480 con 0x55b5ac9c8240 2024-03-24T00:33:13.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 1 -- 172.21.15.40:0/3715267176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4924026070 con 0x55b5ac9c8240 2024-03-24T00:33:13.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4913400640 1 -- 172.21.15.40:0/3715267176 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f492c0039a0 con 0x55b5ac9c8240 2024-03-24T00:33:13.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 10 client.6003 fetch_fsmap learned FSMap version 35 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 1 -- 172.21.15.40:0/3715267176 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55b5ac7722f0 con 0x55b5ac9c8240 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4913400640 1 -- 172.21.15.40:0/3715267176 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4924026070 con 0x55b5ac9c8240 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4913400640 1 -- 172.21.15.40:0/3715267176 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f492c0db020 con 0x55b5ac9c8240 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 10 client.6003 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 10 client.6003 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 20 client.6003 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x55b5ac9dbeb0 0x55b5ac9dc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 4 client.6003 mds_command: new command op to 5626 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f493bcda000 1 -- 172.21.15.40:0/3715267176 --> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b5ac7d8bc0 con 0x55b5ac9dbeb0 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.559+0000 7f4935e00640 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x55b5ac9dbeb0 0x55b5ac9dc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4935e00640 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x55b5ac9dbeb0 0x55b5ac9dc290 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cfff532bc760f490 server_cookie=a3ec379ec2c0e4a6 in_seq=0 out_seq=0 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4913400640 10 client.6003 ms_handle_connect on v2:172.21.15.53:6834/1600357100 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4913400640 1 -- 172.21.15.40:0/3715267176 <== mds.1 v2:172.21.15.53:6834/1600357100 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55b5ac7d8bc0 con 0x55b5ac9dbeb0 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4913400640 10 client.6003 handle_command_reply: tid=0 2024-03-24T00:33:13.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f493bcda000 1 client.6003 shutdown 2024-03-24T00:33:13.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f493bcda000 1 -- 172.21.15.40:0/3715267176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac9c8240 msgr2=0x55b5ac9db970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f493bcda000 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac9c8240 0x55b5ac9db970 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f492c004b40 tx=0x7f492c01d990 comp rx=0 tx=0).stop 2024-03-24T00:33:13.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f493bcda000 1 -- 172.21.15.40:0/3715267176 shutdown_connections 2024-03-24T00:33:13.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f493bcda000 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x55b5ac9dbeb0 0x55b5ac9dc290 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f493bcda000 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b5ac9c8240 0x55b5ac9db970 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f493bcda000 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b5ac9c7920 0x55b5ac9db430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f493bcda000 1 --2- 172.21.15.40:0/3715267176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b5ac97dff0 0x55b5ac9daef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4913400640 0 client.6003 ms_handle_reset on v2:172.21.15.53:6834/1600357100 2024-03-24T00:33:13.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f493bcda000 1 -- 172.21.15.40:0/3715267176 >> 172.21.15.40:0/3715267176 conn(0x55b5ac979fa0 msgr2=0x55b5ac9aa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f493bcda000 1 -- 172.21.15.40:0/3715267176 shutdown_connections 2024-03-24T00:33:13.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f493bcda000 1 -- 172.21.15.40:0/3715267176 wait complete. 2024-03-24T00:33:13.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f493bcda000 20 client.6003 trim_cache size 0 max 16384 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5626, 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.0371559170000002, 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 35, 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 44, 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 44, 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 20.746708623 2024-03-24T00:33:13.568 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:13.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4938800640 1 -- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4914067660 msgr2=0x7f4914069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4938800640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4914067660 0x7f4914069b20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4924007fd0 tx=0x7f4924009040 comp rx=0 tx=0).stop 2024-03-24T00:33:13.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4938800640 1 -- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300b2910 msgr2=0x7f493006c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4938800640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300b2910 0x7f493006c560 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f492000e930 tx=0x7f492000edf0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4938800640 1 -- 172.21.15.40:0/2498845998 shutdown_connections 2024-03-24T00:33:13.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4938800640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4914067660 0x7f4914069b20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4938800640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f493015f600 0x7f49301f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4938800640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f49300b2910 0x7f493006c560 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4938800640 1 --2- 172.21.15.40:0/2498845998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49300af170 0x7f49300b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4938800640 1 -- 172.21.15.40:0/2498845998 >> 172.21.15.40:0/2498845998 conn(0x7f49300d1d10 msgr2=0x7f493015b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4938800640 1 -- 172.21.15.40:0/2498845998 shutdown_connections 2024-03-24T00:33:13.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.563+0000 7f4938800640 1 -- 172.21.15.40:0/2498845998 wait complete. 2024-03-24T00:33:13.593 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:33:13.608 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-03-24T00:33:13.607+0000 7f2af2400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:33:13.608 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-03-24T00:33:13.607+0000 7f2af2400640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-24T00:33:13.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.735+0000 7f6457e00640 1 Processor -- start 2024-03-24T00:33:13.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.735+0000 7f6457e00640 1 -- start start 2024-03-24T00:33:13.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6457e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f64500b2910 0x7f64500d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6457e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64500af170 0x7f6450165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6457e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64500b1e50 con 0x7f645015f600 2024-03-24T00:33:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6457e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64500b3630 con 0x7f64500b2910 2024-03-24T00:33:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6457e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64500b2d80 con 0x7f64500af170 2024-03-24T00:33:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6455400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f645015f600 0x7f64500aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:35794/0 (socket says 172.21.15.40:35794) 2024-03-24T00:33:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6455400640 1 -- 172.21.15.40:0/3264941061 learned_addr learned my addr 172.21.15.40:0/3264941061 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6454a00640 1 --2- 172.21.15.40:0/3264941061 >> v2:172.21.15.53:3300/0 conn(0x7f64500b2910 0x7f64500d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6455e00640 1 --2- 172.21.15.40:0/3264941061 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64500af170 0x7f6450165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f644fe00640 1 -- 172.21.15.40:0/3264941061 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499060996 0 0) 0x7f64500b1e50 con 0x7f645015f600 2024-03-24T00:33:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f644fe00640 1 -- 172.21.15.40:0/3264941061 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6438003530 con 0x7f645015f600 2024-03-24T00:33:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6454a00640 1 -- 172.21.15.40:0/3264941061 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64500af170 msgr2=0x7f6450165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6454a00640 1 --2- 172.21.15.40:0/3264941061 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64500af170 0x7f6450165e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6454a00640 1 -- 172.21.15.40:0/3264941061 >> v1:172.21.15.40:6789/0 conn(0x7f645015f600 legacy=0x7f64500aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.739+0000 7f6454a00640 1 -- 172.21.15.40:0/3264941061 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64500b20f0 con 0x7f64500b2910 2024-03-24T00:33:13.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f6454a00640 1 --2- 172.21.15.40:0/3264941061 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64500b2910 0x7f64500d2830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f643c0097c0 tx=0x7f643c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4abac11daacf3e47 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f644fe00640 1 -- 172.21.15.40:0/3264941061 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c0b5070 con 0x7f64500b2910 2024-03-24T00:33:13.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f644fe00640 1 -- 172.21.15.40:0/3264941061 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c0a8990 con 0x7f64500b2910 2024-03-24T00:33:13.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f644fe00640 1 -- 172.21.15.40:0/3264941061 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c0a8d40 con 0x7f64500b2910 2024-03-24T00:33:13.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f6457e00640 1 -- 172.21.15.40:0/3264941061 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64500b2910 msgr2=0x7f64500d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f6457e00640 1 --2- 172.21.15.40:0/3264941061 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64500b2910 0x7f64500d2830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f643c0097c0 tx=0x7f643c01ec90 comp rx=0 tx=0).stop 2024-03-24T00:33:13.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f6457e00640 1 -- 172.21.15.40:0/3264941061 shutdown_connections 2024-03-24T00:33:13.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f6457e00640 1 --2- 172.21.15.40:0/3264941061 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64500af170 0x7f6450165e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f6457e00640 1 --2- 172.21.15.40:0/3264941061 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64500b2910 0x7f64500d2830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f6457e00640 1 -- 172.21.15.40:0/3264941061 >> 172.21.15.40:0/3264941061 conn(0x7f64500d1d10 msgr2=0x7f64500d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f6457e00640 1 -- 172.21.15.40:0/3264941061 shutdown_connections 2024-03-24T00:33:13.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f6457e00640 1 -- 172.21.15.40:0/3264941061 wait complete. 2024-03-24T00:33:13.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f6457e00640 1 Processor -- start 2024-03-24T00:33:13.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.743+0000 7f6457e00640 1 -- start start 2024-03-24T00:33:13.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6457e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64500af170 0x7f64501f39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6457e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64500b2910 0x7f64501f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6457e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645015f600 0x7f645006b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6457e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64500b2fa0 con 0x7f64500af170 2024-03-24T00:33:13.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6457e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64500b2740 con 0x7f645015f600 2024-03-24T00:33:13.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6457e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645015f9e0 con 0x7f64500b2910 2024-03-24T00:33:13.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6455400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64500af170 0x7f64501f39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6455400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64500af170 0x7f64501f39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52100/0 (socket says 172.21.15.40:52100) 2024-03-24T00:33:13.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6455400640 1 -- 172.21.15.40:0/1121173429 learned_addr learned my addr 172.21.15.40:0/1121173429 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6455e00640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645015f600 0x7f645006b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6454a00640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64500b2910 0x7f64501f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6455400640 1 -- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64500b2910 msgr2=0x7f64501f4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6455400640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64500b2910 0x7f64501f4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6455400640 1 -- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645015f600 msgr2=0x7f645006b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6455400640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645015f600 0x7f645006b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6455400640 1 -- 172.21.15.40:0/1121173429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64501f42e0 con 0x7f64500af170 2024-03-24T00:33:13.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6455400640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64500af170 0x7f64501f39e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f644400c3c0 tx=0x7f644400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f644d600640 1 -- 172.21.15.40:0/1121173429 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444017070 con 0x7f64500af170 2024-03-24T00:33:13.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f644d600640 1 -- 172.21.15.40:0/1121173429 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6444005cf0 con 0x7f64500af170 2024-03-24T00:33:13.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6457e00640 1 -- 172.21.15.40:0/1121173429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64501f45c0 con 0x7f64500af170 2024-03-24T00:33:13.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f644d600640 1 -- 172.21.15.40:0/1121173429 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64440072f0 con 0x7f64500af170 2024-03-24T00:33:13.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f6457e00640 1 -- 172.21.15.40:0/1121173429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645006b150 con 0x7f64500af170 2024-03-24T00:33:13.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.747+0000 7f644d600640 1 -- 172.21.15.40:0/1121173429 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f644401a070 con 0x7f64500af170 2024-03-24T00:33:13.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f644d600640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6430067660 0x7f6430069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f644d600640 1 -- 172.21.15.40:0/1121173429 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6444084c50 con 0x7f64500af170 2024-03-24T00:33:13.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6454a00640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6430067660 0x7f6430069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 Processor -- start 2024-03-24T00:33:13.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 -- start start 2024-03-24T00:33:13.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6454a00640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6430067660 0x7f6430069b20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f643c0b2000 tx=0x7f643c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 0x55c021b0cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021b0d240 0x55c021ac2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021ac2ff0 0x55c021ac33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c021910680 con 0x55c021b0c920 2024-03-24T00:33:13.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c021897b00 con 0x55c021b0d240 2024-03-24T00:33:13.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c02190c720 con 0x55c021ac2ff0 2024-03-24T00:33:13.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6455400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 0x55c021b0cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6455400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 0x55c021b0cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52106/0 (socket says 172.21.15.40:52106) 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6455400640 1 -- 172.21.15.40:0/3892153471 learned_addr learned my addr 172.21.15.40:0/3892153471 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6455e00640 1 --2- 172.21.15.40:0/3892153471 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021ac2ff0 0x55c021ac33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6454a00640 1 --2- 172.21.15.40:0/3892153471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021b0d240 0x55c021ac2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6455400640 1 -- 172.21.15.40:0/3892153471 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021ac2ff0 msgr2=0x55c021ac33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6455400640 1 --2- 172.21.15.40:0/3892153471 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021ac2ff0 0x55c021ac33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6455400640 1 -- 172.21.15.40:0/3892153471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021b0d240 msgr2=0x55c021ac2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6455400640 1 --2- 172.21.15.40:0/3892153471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021b0d240 0x55c021ac2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6455400640 1 -- 172.21.15.40:0/3892153471 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0218e8940 con 0x55c021b0c920 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6455400640 1 --2- 172.21.15.40:0/3892153471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 0x55c021b0cd00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6444051650 tx=0x7f64440550f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84fe50d814e43a66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6442000640 1 -- 172.21.15.40:0/3892153471 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64440534f0 con 0x55c021b0c920 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6442000640 1 -- 172.21.15.40:0/3892153471 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6444051860 con 0x55c021b0c920 2024-03-24T00:33:13.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f6442000640 1 -- 172.21.15.40:0/3892153471 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64440076d0 con 0x55c021b0c920 2024-03-24T00:33:13.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 -- 172.21.15.40:0/3892153471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 msgr2=0x55c021b0cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 --2- 172.21.15.40:0/3892153471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 0x55c021b0cd00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6444051650 tx=0x7f64440550f0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 -- 172.21.15.40:0/3892153471 shutdown_connections 2024-03-24T00:33:13.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 --2- 172.21.15.40:0/3892153471 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021ac2ff0 0x55c021ac33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 --2- 172.21.15.40:0/3892153471 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021b0d240 0x55c021ac2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 --2- 172.21.15.40:0/3892153471 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 0x55c021b0cd00 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 -- 172.21.15.40:0/3892153471 >> 172.21.15.40:0/3892153471 conn(0x55c021abefa0 msgr2=0x55c021ac1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 -- 172.21.15.40:0/3892153471 shutdown_connections 2024-03-24T00:33:13.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.751+0000 7f645b2be000 1 -- 172.21.15.40:0/3892153471 wait complete. 2024-03-24T00:33:13.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 1 -- start start 2024-03-24T00:33:13.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 1 Processor -- start 2024-03-24T00:33:13.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021ac2ff0 0x55c021af1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 0x55c021af2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021b0d240 0x55c021af27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c02190f990 con 0x55c021b0c920 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0218caae0 con 0x55c021ac2ff0 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c0218a8750 con 0x55c021b0d240 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 10 client.0 _ll_register_callbacks cb 0x55c021aadff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6454a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 0x55c021af2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6454a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 0x55c021af2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52116/0 (socket says 172.21.15.40:52116) 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6455e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021b0d240 0x55c021af27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6454a00640 1 -- 172.21.15.40:0/998688620 learned_addr learned my addr 172.21.15.40:0/998688620 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6455400640 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021ac2ff0 0x55c021af1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6454a00640 1 -- 172.21.15.40:0/998688620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021b0d240 msgr2=0x55c021af27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6454a00640 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021b0d240 0x55c021af27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6454a00640 1 -- 172.21.15.40:0/998688620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021ac2ff0 msgr2=0x55c021af1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6454a00640 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021ac2ff0 0x55c021af1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6454a00640 1 -- 172.21.15.40:0/998688620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c021955d80 con 0x55c021b0c920 2024-03-24T00:33:13.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6455e00640 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021b0d240 0x55c021af27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:13.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6454a00640 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 0x55c021af2280 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f643c008010 tx=0x7f643c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f3d442de1d72ca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6436a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:13.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6436a00640 1 -- 172.21.15.40:0/998688620 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c0ce060 con 0x55c021b0c920 2024-03-24T00:33:13.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 1 -- 172.21.15.40:0/998688620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6444051860 con 0x55c021b0c920 2024-03-24T00:33:13.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6436a00640 1 -- 172.21.15.40:0/998688620 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f643c0bbc40 con 0x55c021b0c920 2024-03-24T00:33:13.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6436a00640 1 -- 172.21.15.40:0/998688620 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f643c0d4050 con 0x55c021b0c920 2024-03-24T00:33:13.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 10 client.6015 fetch_fsmap learned FSMap version 35 2024-03-24T00:33:13.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 1 -- 172.21.15.40:0/998688620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55c021897d10 con 0x55c021b0c920 2024-03-24T00:33:13.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6436a00640 1 -- 172.21.15.40:0/998688620 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6444051860 con 0x55c021b0c920 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f6436a00640 1 -- 172.21.15.40:0/998688620 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f643c0d7020 con 0x55c021b0c920 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 10 client.6015 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 10 client.6015 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 20 client.6015 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c021af3090 0x55c021b1eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 4 client.6015 mds_command: new command op to 5763 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.755+0000 7f645b2be000 1 -- 172.21.15.40:0/998688620 --> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c0218abb60 con 0x55c021af3090 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f6455400640 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c021af3090 0x55c021b1eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f6455400640 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c021af3090 0x55c021b1eea0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbcdd7631ebafb18 server_cookie=74bf365d18190de1 in_seq=0 out_seq=0 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f6436a00640 10 client.6015 ms_handle_connect on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f6436a00640 1 -- 172.21.15.40:0/998688620 <== mds.0 v2:172.21.15.40:6832/224223662 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c0218abb60 con 0x55c021af3090 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f6436a00640 10 client.6015 handle_command_reply: tid=0 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f645b2be000 1 client.6015 shutdown 2024-03-24T00:33:13.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f645b2be000 1 -- 172.21.15.40:0/998688620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 msgr2=0x55c021af2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f645b2be000 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 0x55c021af2280 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f643c008010 tx=0x7f643c004740 comp rx=0 tx=0).stop 2024-03-24T00:33:13.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f645b2be000 1 -- 172.21.15.40:0/998688620 shutdown_connections 2024-03-24T00:33:13.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f645b2be000 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c021af3090 0x55c021b1eea0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f645b2be000 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021b0d240 0x55c021af27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f645b2be000 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021b0c920 0x55c021af2280 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f645b2be000 1 --2- 172.21.15.40:0/998688620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021ac2ff0 0x55c021af1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f6436a00640 0 client.6015 ms_handle_reset on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:13.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f645b2be000 1 -- 172.21.15.40:0/998688620 >> 172.21.15.40:0/998688620 conn(0x55c021abefa0 msgr2=0x55c021ac1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f645b2be000 1 -- 172.21.15.40:0/998688620 shutdown_connections 2024-03-24T00:33:13.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f645b2be000 1 -- 172.21.15.40:0/998688620 wait complete. 2024-03-24T00:33:13.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.759+0000 7f645b2be000 20 client.6015 trim_cache size 0 max 16384 2024-03-24T00:33:13.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f645b2be000 1 Processor -- start 2024-03-24T00:33:13.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f645b2be000 1 -- start start 2024-03-24T00:33:13.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f645b2be000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021af1ed0 0x55c021ac27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f645b2be000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021abefa0 0x55c021b0b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f645b2be000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021ac2ff0 0x55c021b0ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f645b2be000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64501fff40 con 0x55c021af1ed0 2024-03-24T00:33:13.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f645b2be000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64501ff2e0 con 0x55c021abefa0 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f645b2be000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64501ff8e0 con 0x55c021ac2ff0 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f6455400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021af1ed0 0x55c021ac27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f6455400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021af1ed0 0x55c021ac27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52120/0 (socket says 172.21.15.40:52120) 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f6454a00640 1 --2- 172.21.15.40:0/2573492554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021abefa0 0x55c021b0b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f6455400640 1 -- 172.21.15.40:0/2573492554 learned_addr learned my addr 172.21.15.40:0/2573492554 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f6455e00640 1 --2- 172.21.15.40:0/2573492554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021ac2ff0 0x55c021b0ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f6455400640 1 -- 172.21.15.40:0/2573492554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021ac2ff0 msgr2=0x55c021b0ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f6455400640 1 --2- 172.21.15.40:0/2573492554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021ac2ff0 0x55c021b0ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f6455400640 1 -- 172.21.15.40:0/2573492554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021abefa0 msgr2=0x55c021b0b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f6455400640 1 --2- 172.21.15.40:0/2573492554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021abefa0 0x55c021b0b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f6455400640 1 -- 172.21.15.40:0/2573492554 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c021ac71c0 con 0x55c021af1ed0 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.775+0000 7f6454a00640 1 --2- 172.21.15.40:0/2573492554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021abefa0 0x55c021b0b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6455400640 1 --2- 172.21.15.40:0/2573492554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021af1ed0 0x55c021ac27c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6444059450 tx=0x7f6444014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ca819c74dee59f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6441600640 1 -- 172.21.15.40:0/2573492554 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444054fc0 con 0x55c021af1ed0 2024-03-24T00:33:13.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6441600640 1 -- 172.21.15.40:0/2573492554 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6444080070 con 0x55c021af1ed0 2024-03-24T00:33:13.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6441600640 1 -- 172.21.15.40:0/2573492554 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64440076b0 con 0x55c021af1ed0 2024-03-24T00:33:13.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 -- 172.21.15.40:0/2573492554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021af1ed0 msgr2=0x55c021ac27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 --2- 172.21.15.40:0/2573492554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021af1ed0 0x55c021ac27c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6444059450 tx=0x7f6444014000 comp rx=0 tx=0).stop 2024-03-24T00:33:13.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 -- 172.21.15.40:0/2573492554 shutdown_connections 2024-03-24T00:33:13.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 --2- 172.21.15.40:0/2573492554 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021ac2ff0 0x55c021b0ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 --2- 172.21.15.40:0/2573492554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021abefa0 0x55c021b0b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 --2- 172.21.15.40:0/2573492554 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021af1ed0 0x55c021ac27c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 -- 172.21.15.40:0/2573492554 >> 172.21.15.40:0/2573492554 conn(0x55c021ac5750 msgr2=0x55c021aef5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 -- 172.21.15.40:0/2573492554 shutdown_connections 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 -- 172.21.15.40:0/2573492554 wait complete. 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 -- start start 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 Processor -- start 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021abefa0 0x55c021b202e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021ac2ff0 0x55c021b20820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021af1ed0 0x55c021b20d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0218b8a80 con 0x55c021ac2ff0 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64500c9050 con 0x55c021abefa0 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c0219cd700 con 0x55c021af1ed0 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f645b2be000 10 client.0 _ll_register_callbacks cb 0x7f64480095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:13.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6454a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021ac2ff0 0x55c021b20820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6455400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021abefa0 0x55c021b202e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6454a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021ac2ff0 0x55c021b20820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52136/0 (socket says 172.21.15.40:52136) 2024-03-24T00:33:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6454a00640 1 -- 172.21.15.40:0/3229512955 learned_addr learned my addr 172.21.15.40:0/3229512955 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6455e00640 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021af1ed0 0x55c021b20d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6454a00640 1 -- 172.21.15.40:0/3229512955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021af1ed0 msgr2=0x55c021b20d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6454a00640 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021af1ed0 0x55c021b20d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6454a00640 1 -- 172.21.15.40:0/3229512955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021abefa0 msgr2=0x55c021b202e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6454a00640 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021abefa0 0x55c021b202e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6454a00640 1 -- 172.21.15.40:0/3229512955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c0218dd540 con 0x55c021ac2ff0 2024-03-24T00:33:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6455400640 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021abefa0 0x55c021b202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6454a00640 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021ac2ff0 0x55c021b20820 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f643c01b8a0 tx=0x7f643c01d4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ae78cfac41c46e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:13.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.779+0000 7f6436a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:13.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6436a00640 1 -- 172.21.15.40:0/3229512955 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c0b5040 con 0x55c021ac2ff0 2024-03-24T00:33:13.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6436a00640 1 -- 172.21.15.40:0/3229512955 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f643c0c9830 con 0x55c021ac2ff0 2024-03-24T00:33:13.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 -- 172.21.15.40:0/3229512955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6444080070 con 0x55c021ac2ff0 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6436a00640 1 -- 172.21.15.40:0/3229512955 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f643c0c9a30 con 0x55c021ac2ff0 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 10 client.6021 fetch_fsmap learned FSMap version 35 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 -- 172.21.15.40:0/3229512955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55c0218a44b0 con 0x55c021ac2ff0 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6436a00640 1 -- 172.21.15.40:0/3229512955 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6444080070 con 0x55c021ac2ff0 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6436a00640 1 -- 172.21.15.40:0/3229512955 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 35) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f643c0e8020 con 0x55c021ac2ff0 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 10 client.6021 fetch_fsmap finished waiting for FSMap version 35 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 10 client.6021 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 20 client.6021 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c021b0c920 0x55c021b7a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 4 client.6021 mds_command: new command op to 5763 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 -- 172.21.15.40:0/3229512955 --> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c02191dbc0 con 0x55c021b0c920 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6455400640 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c021b0c920 0x55c021b7a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6455400640 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c021b0c920 0x55c021b7a1f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4083325b48b26813 server_cookie=fd724e6d41308a43 in_seq=0 out_seq=0 2024-03-24T00:33:13.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6436a00640 10 client.6021 ms_handle_connect on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6436a00640 1 -- 172.21.15.40:0/3229512955 <== mds.0 v2:172.21.15.40:6832/224223662 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c02191dbc0 con 0x55c021b0c920 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6436a00640 10 client.6021 handle_command_reply: tid=0 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 client.6021 shutdown 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 -- 172.21.15.40:0/3229512955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021ac2ff0 msgr2=0x55c021b20820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021ac2ff0 0x55c021b20820 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f643c01b8a0 tx=0x7f643c01d4d0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 -- 172.21.15.40:0/3229512955 shutdown_connections 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55c021b0c920 0x55c021b7a1f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c021af1ed0 0x55c021b20d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c021ac2ff0 0x55c021b20820 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 --2- 172.21.15.40:0/3229512955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c021abefa0 0x55c021b202e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6436a00640 0 client.6021 ms_handle_reset on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 -- 172.21.15.40:0/3229512955 >> 172.21.15.40:0/3229512955 conn(0x55c021ac5750 msgr2=0x55c021aefe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 -- 172.21.15.40:0/3229512955 shutdown_connections 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 1 -- 172.21.15.40:0/3229512955 wait complete. 2024-03-24T00:33:13.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f645b2be000 20 client.6021 trim_cache size 0 max 16384 2024-03-24T00:33:13.791 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:13.791 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:33:13.791 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:33:13.791 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5763, 2024-03-24T00:33:13.791 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:33:13.791 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:33:13.791 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:33:13.791 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 16.145073491000002, 2024-03-24T00:33:13.791 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 35, 2024-03-24T00:33:13.791 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 44, 2024-03-24T00:33:13.792 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 44, 2024-03-24T00:33:13.792 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 20.980570133000001 2024-03-24T00:33:13.792 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:13.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6457e00640 1 -- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6430067660 msgr2=0x7f6430069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6457e00640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6430067660 0x7f6430069b20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f643c0b2000 tx=0x7f643c008040 comp rx=0 tx=0).stop 2024-03-24T00:33:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6457e00640 1 -- 172.21.15.40:0/1121173429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64500af170 msgr2=0x7f64501f39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6457e00640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64500af170 0x7f64501f39e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f644400c3c0 tx=0x7f644400c880 comp rx=0 tx=0).stop 2024-03-24T00:33:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6457e00640 1 -- 172.21.15.40:0/1121173429 shutdown_connections 2024-03-24T00:33:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6457e00640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6430067660 0x7f6430069b20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6457e00640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645015f600 0x7f645006b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6457e00640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64500b2910 0x7f64501f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6457e00640 1 --2- 172.21.15.40:0/1121173429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64500af170 0x7f64501f39e0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6457e00640 1 -- 172.21.15.40:0/1121173429 >> 172.21.15.40:0/1121173429 conn(0x7f64500d1d10 msgr2=0x7f645015c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6457e00640 1 -- 172.21.15.40:0/1121173429 shutdown_connections 2024-03-24T00:33:13.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:13.783+0000 7f6457e00640 1 -- 172.21.15.40:0/1121173429 wait complete. 2024-03-24T00:33:13.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:13.806 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:33:13.806 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:33:13.806 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p tree 2024-03-24T00:33:13.806 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:33:13.806 DEBUG:teuthology.orchestra.run.smithi040:> setfattr -n ceph.dir.pin.distributed -v 1 tree 2024-03-24T00:33:13.806 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:33:13.806 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < 100; i++)); do 2024-03-24T00:33:13.806 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p "tree/$i" 2024-03-24T00:33:13.807 DEBUG:teuthology.orchestra.run.smithi040:> echo file > "tree/$i/file" 2024-03-24T00:33:13.807 DEBUG:teuthology.orchestra.run.smithi040:> done 2024-03-24T00:33:13.807 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:33:13.858 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree 2024-03-24T00:33:13.862 INFO:teuthology.orchestra.run.smithi040.stderr:+ setfattr -n ceph.dir.pin.distributed -v 1 tree 2024-03-24T00:33:13.866 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i = 0 )) 2024-03-24T00:33:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/0 2024-03-24T00:33:13.872 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.879 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.879 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.879 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/1 2024-03-24T00:33:13.889 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/2 2024-03-24T00:33:13.899 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/3 2024-03-24T00:33:13.908 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.911 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.911 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.911 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/4 2024-03-24T00:33:13.918 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.921 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.921 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.921 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/5 2024-03-24T00:33:13.928 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/6 2024-03-24T00:33:13.936 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.938 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.938 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.938 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/7 2024-03-24T00:33:13.945 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.948 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.948 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.948 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/8 2024-03-24T00:33:13.955 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.957 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.957 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.957 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/9 2024-03-24T00:33:13.963 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.966 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.966 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.966 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/10 2024-03-24T00:33:13.972 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.975 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.975 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.975 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/11 2024-03-24T00:33:13.981 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.984 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.984 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.984 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/12 2024-03-24T00:33:13.991 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:13.994 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:13.994 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:13.994 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/13 2024-03-24T00:33:14.000 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.002 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.002 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.002 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/14 2024-03-24T00:33:14.008 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.011 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.011 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.011 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/15 2024-03-24T00:33:14.018 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.021 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.022 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.022 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/16 2024-03-24T00:33:14.030 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.030 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.030 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.030 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/17 2024-03-24T00:33:14.037 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.039 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.040 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.040 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/18 2024-03-24T00:33:14.046 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.049 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.049 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.049 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/19 2024-03-24T00:33:14.055 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.058 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.058 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.058 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/20 2024-03-24T00:33:14.064 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.066 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.067 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.067 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/21 2024-03-24T00:33:14.073 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.076 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/22 2024-03-24T00:33:14.082 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.085 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.086 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/23 2024-03-24T00:33:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.095 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/24 2024-03-24T00:33:14.101 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.103 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/25 2024-03-24T00:33:14.110 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.112 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.112 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.112 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/26 2024-03-24T00:33:14.119 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.122 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/27 2024-03-24T00:33:14.128 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.131 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.132 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.132 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/28 2024-03-24T00:33:14.138 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.141 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.141 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.141 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/29 2024-03-24T00:33:14.148 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.150 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.150 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.151 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/30 2024-03-24T00:33:14.158 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.161 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.161 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.161 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/31 2024-03-24T00:33:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.171 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.171 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.171 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/32 2024-03-24T00:33:14.178 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/33 2024-03-24T00:33:14.186 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.189 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/34 2024-03-24T00:33:14.195 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/35 2024-03-24T00:33:14.204 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.207 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.207 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/36 2024-03-24T00:33:14.213 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.216 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.216 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.216 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/37 2024-03-24T00:33:14.222 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.225 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.225 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.225 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/38 2024-03-24T00:33:14.234 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.234 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.234 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.234 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/39 2024-03-24T00:33:14.240 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.242 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.242 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.242 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/40 2024-03-24T00:33:14.249 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.252 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.252 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.252 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/41 2024-03-24T00:33:14.259 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.261 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.261 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.261 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/42 2024-03-24T00:33:14.267 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.270 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/43 2024-03-24T00:33:14.277 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.280 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/44 2024-03-24T00:33:14.286 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.289 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.289 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.289 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/45 2024-03-24T00:33:14.295 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.298 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.299 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.299 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/46 2024-03-24T00:33:14.305 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.308 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/47 2024-03-24T00:33:14.314 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.316 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.316 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.316 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/48 2024-03-24T00:33:14.322 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.324 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.324 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.325 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/49 2024-03-24T00:33:14.330 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.332 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.333 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/50 2024-03-24T00:33:14.339 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.342 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.342 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.342 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/51 2024-03-24T00:33:14.349 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.351 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.351 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.351 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/52 2024-03-24T00:33:14.359 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.361 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.361 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.361 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/53 2024-03-24T00:33:14.367 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.370 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.370 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.370 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/54 2024-03-24T00:33:14.376 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.379 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.379 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.379 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/55 2024-03-24T00:33:14.385 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.387 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.387 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.387 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/56 2024-03-24T00:33:14.393 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/57 2024-03-24T00:33:14.402 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.405 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.405 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.405 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/58 2024-03-24T00:33:14.411 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/59 2024-03-24T00:33:14.420 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.423 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.423 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.423 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/60 2024-03-24T00:33:14.429 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.431 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.431 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.431 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/61 2024-03-24T00:33:14.438 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.440 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/62 2024-03-24T00:33:14.447 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.449 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.449 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.450 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/63 2024-03-24T00:33:14.456 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.458 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.458 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.458 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/64 2024-03-24T00:33:14.465 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.467 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.467 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.467 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/65 2024-03-24T00:33:14.473 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.476 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.476 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.476 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/66 2024-03-24T00:33:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.485 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.485 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.485 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/67 2024-03-24T00:33:14.492 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.495 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.495 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.495 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/68 2024-03-24T00:33:14.501 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.503 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.504 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.504 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/69 2024-03-24T00:33:14.510 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.512 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.513 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.513 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/70 2024-03-24T00:33:14.519 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.522 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.522 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.522 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/71 2024-03-24T00:33:14.528 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.530 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.530 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.530 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/72 2024-03-24T00:33:14.537 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.539 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.539 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.539 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/73 2024-03-24T00:33:14.547 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.549 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.549 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.549 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/74 2024-03-24T00:33:14.556 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.559 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.559 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.559 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/75 2024-03-24T00:33:14.566 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.569 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.569 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.569 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/76 2024-03-24T00:33:14.576 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.579 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.580 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.580 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/77 2024-03-24T00:33:14.587 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.590 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.590 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.590 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/78 2024-03-24T00:33:14.597 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.599 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.599 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.599 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/79 2024-03-24T00:33:14.606 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.609 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.609 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.609 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/80 2024-03-24T00:33:14.615 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.618 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.618 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.618 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/81 2024-03-24T00:33:14.624 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.627 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.627 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.627 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/82 2024-03-24T00:33:14.633 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.636 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.636 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.636 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/83 2024-03-24T00:33:14.643 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.645 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.645 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.645 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/84 2024-03-24T00:33:14.652 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.655 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.655 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.655 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/85 2024-03-24T00:33:14.662 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.665 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.665 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.665 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/86 2024-03-24T00:33:14.672 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.675 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/87 2024-03-24T00:33:14.682 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.685 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/88 2024-03-24T00:33:14.692 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.694 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.695 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.695 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/89 2024-03-24T00:33:14.700 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.703 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.703 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.703 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/90 2024-03-24T00:33:14.709 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.711 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.712 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/91 2024-03-24T00:33:14.718 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.720 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.721 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/92 2024-03-24T00:33:14.727 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.729 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.729 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.729 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/93 2024-03-24T00:33:14.736 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.738 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.738 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.738 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/94 2024-03-24T00:33:14.745 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.748 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/95 2024-03-24T00:33:14.754 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.758 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.758 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.758 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/96 2024-03-24T00:33:14.763 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.765 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/97 2024-03-24T00:33:14.772 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.775 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.775 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.775 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/98 2024-03-24T00:33:14.781 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.784 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.784 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.785 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/99 2024-03-24T00:33:14.791 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:33:14.794 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:33:14.794 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:33:14.795 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:33:14.954 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:33:14.954 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:33:14.955 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:33:15.114 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:33:15.115 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:33:15.115 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:33:15.275 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"1626","version":"1624","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"477","version":"477","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-24T00:33:15.276 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "1626", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "1624" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "477", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "477" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:33:15.276 INFO:tasks.cephfs.cephfs_test_case:len=0 [] 2024-03-24T00:33:20.278 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:33:20.438 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:33:20.438 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:33:20.439 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:33:20.600 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:33:20.601 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:33:20.601 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:33:20.769 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"1632","version":"1628","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":2,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"477","version":"477","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-24T00:33:20.769 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 2, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "1632", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "1628" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "477", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "477" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:33:20.770 INFO:tasks.cephfs.cephfs_test_case:len=0 [] 2024-03-24T00:33:25.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:33:25.930 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"1635","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":3},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x100000002e1.101*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x100000002e1.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x100000002e1.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-24T00:33:25.931 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 3 }, "snapid_first": 2, "states": [], "version": "1635" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002e1.101*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "422", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002e1.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "430", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002e1.001*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "428", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:33:25.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:33:26.093 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"1635","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000002e1.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-24T00:33:26.094 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "1635" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x100000002e1.111*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "416", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:33:26.094 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:33:26.253 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"1635","version":"1635","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1,"2":3}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"477","version":"477","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x100000002e1.000*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x100000002e1.100*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"waiter":0,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000002e1.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x100000002e1.101*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x100000002e1.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x100000002e1.010*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x100000002e1.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x100000002e1.110*","snapid_first":2,"projected_version":"442","version":"442","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-24T00:33:26.254 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1, "2": 3 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "1635", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "1635" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "477", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "477" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x100000002e1.000*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x100000002e1.100*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x100000002e1.111*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "416", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002e1.101*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "422", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002e1.011*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "430", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x100000002e1.010*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002e1.001*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "428", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x100000002e1.110*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "442", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "442" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:33:26.255 INFO:tasks.cephfs.cephfs_test_case:len=8 [{'is_auth': True, 'auth_first': 2, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 2, 'dir': {'path': '/tree', 'dirfrag': '0x100000002e1.101*', 'snapid_first': 2, 'projected_version': '422', 'version': '422', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '2', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [2, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 2, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 2, 'dir': {'path': '/tree', 'dirfrag': '0x100000002e1.011*', 'snapid_first': 2, 'projected_version': '430', 'version': '430', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '2', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [2, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 2, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 2, 'dir': {'path': '/tree', 'dirfrag': '0x100000002e1.001*', 'snapid_first': 2, 'projected_version': '428', 'version': '428', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '2', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [2, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x100000002e1.111*', 'snapid_first': 2, 'projected_version': '416', 'version': '416', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x100000002e1.000*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x100000002e1.100*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'waiter': 0, 'authpin': 0}, 'nref': 3}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x100000002e1.010*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x100000002e1.110*', 'snapid_first': 2, 'projected_version': '442', 'version': '442', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}] 2024-03-24T00:33:26.255 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:26.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.391+0000 7f1fe4a00640 1 Processor -- start 2024-03-24T00:33:26.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.391+0000 7f1fe4a00640 1 -- start start 2024-03-24T00:33:26.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fe4a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1fe013ec80 0x7f1fe01414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fe4a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1fe01419f0 0x7f1fe00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fe4a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe0070e00 con 0x7f1fe00ce630 2024-03-24T00:33:26.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fe4a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe0141e50 con 0x7f1fe013ec80 2024-03-24T00:33:26.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fe4a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00cee20 con 0x7f1fe01419f0 2024-03-24T00:33:26.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1fe01419f0 0x7f1fe00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fde800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1fe00ce630 0x7f1fe015f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:52722/0 (socket says 172.21.15.40:52722) 2024-03-24T00:33:26.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fde800640 1 -- 172.21.15.40:0/4022344961 learned_addr learned my addr 172.21.15.40:0/4022344961 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:26.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fdde00640 1 --2- 172.21.15.40:0/4022344961 >> v2:172.21.15.53:3300/0 conn(0x7f1fe013ec80 0x7f1fe01414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fdd400640 1 -- 172.21.15.40:0/4022344961 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626798945 0 0) 0x7f1fe0070e00 con 0x7f1fe00ce630 2024-03-24T00:33:26.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fdd400640 1 -- 172.21.15.40:0/4022344961 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fc4003530 con 0x7f1fe00ce630 2024-03-24T00:33:26.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fd7e00640 1 -- 172.21.15.40:0/4022344961 >> v2:172.21.15.53:3300/0 conn(0x7f1fe013ec80 msgr2=0x7f1fe01414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:26.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fd7e00640 1 --2- 172.21.15.40:0/4022344961 >> v2:172.21.15.53:3300/0 conn(0x7f1fe013ec80 0x7f1fe01414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fd7e00640 1 -- 172.21.15.40:0/4022344961 >> v1:172.21.15.53:6789/0 conn(0x7f1fe00ce630 legacy=0x7f1fe015f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fd7e00640 1 -- 172.21.15.40:0/4022344961 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00cf100 con 0x7f1fe01419f0 2024-03-24T00:33:26.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.395+0000 7f1fdde00640 1 --2- 172.21.15.40:0/4022344961 >> v2:172.21.15.53:3300/0 conn(0x7f1fe013ec80 0x7f1fe01414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:26.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fd7e00640 1 --2- 172.21.15.40:0/4022344961 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1fe01419f0 0x7f1fe00ce0f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80097c0 tx=0x7f1fc801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b759251c1dc4d75a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:26.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fdd400640 1 -- 172.21.15.40:0/4022344961 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc80aa620 con 0x7f1fe01419f0 2024-03-24T00:33:26.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fdd400640 1 -- 172.21.15.40:0/4022344961 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc80af070 con 0x7f1fe01419f0 2024-03-24T00:33:26.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fdd400640 1 -- 172.21.15.40:0/4022344961 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc80aa9c0 con 0x7f1fe01419f0 2024-03-24T00:33:26.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fe4a00640 1 -- 172.21.15.40:0/4022344961 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1fe01419f0 msgr2=0x7f1fe00ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fe4a00640 1 --2- 172.21.15.40:0/4022344961 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1fe01419f0 0x7f1fe00ce0f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80097c0 tx=0x7f1fc801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fe4a00640 1 -- 172.21.15.40:0/4022344961 shutdown_connections 2024-03-24T00:33:26.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fe4a00640 1 --2- 172.21.15.40:0/4022344961 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1fe01419f0 0x7f1fe00ce0f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fe4a00640 1 --2- 172.21.15.40:0/4022344961 >> v2:172.21.15.53:3300/0 conn(0x7f1fe013ec80 0x7f1fe01414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fe4a00640 1 -- 172.21.15.40:0/4022344961 >> 172.21.15.40:0/4022344961 conn(0x7f1fe00b17b0 msgr2=0x7f1fe00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:26.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fe4a00640 1 -- 172.21.15.40:0/4022344961 shutdown_connections 2024-03-24T00:33:26.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fe4a00640 1 -- 172.21.15.40:0/4022344961 wait complete. 2024-03-24T00:33:26.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fe4a00640 1 Processor -- start 2024-03-24T00:33:26.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.399+0000 7f1fe4a00640 1 -- start start 2024-03-24T00:33:26.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fe4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fe00ce630 0x7f1fe013a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fe4a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe013ec80 0x7f1fe0141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fe4a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1fe01419f0 0x7f1fe0139910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fe4a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe0168ec0 con 0x7f1fe00ce630 2024-03-24T00:33:26.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fe4a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe0168d40 con 0x7f1fe013ec80 2024-03-24T00:33:26.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fe4a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe0169040 con 0x7f1fe01419f0 2024-03-24T00:33:26.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fdde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fe00ce630 0x7f1fe013a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe013ec80 0x7f1fe0141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe013ec80 0x7f1fe0141170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:53366/0 (socket says 172.21.15.40:53366) 2024-03-24T00:33:26.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fde800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1fe01419f0 0x7f1fe0139910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd7e00640 1 -- 172.21.15.40:0/2719688712 learned_addr learned my addr 172.21.15.40:0/2719688712 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:26.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd7e00640 1 -- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1fe01419f0 msgr2=0x7f1fe0139910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd7e00640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1fe01419f0 0x7f1fe0139910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd7e00640 1 -- 172.21.15.40:0/2719688712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fe00ce630 msgr2=0x7f1fe013a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd7e00640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fe00ce630 0x7f1fe013a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd7e00640 1 -- 172.21.15.40:0/2719688712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe01400d0 con 0x7f1fe013ec80 2024-03-24T00:33:26.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fdde00640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fe00ce630 0x7f1fe013a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:26.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd7e00640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe013ec80 0x7f1fe0141170 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1fc801cd40 tx=0x7f1fc8019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:26.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd6000640 1 -- 172.21.15.40:0/2719688712 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8003820 con 0x7f1fe013ec80 2024-03-24T00:33:26.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd6000640 1 -- 172.21.15.40:0/2719688712 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc80039c0 con 0x7f1fe013ec80 2024-03-24T00:33:26.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd6000640 1 -- 172.21.15.40:0/2719688712 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8003d70 con 0x7f1fe013ec80 2024-03-24T00:33:26.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fe4a00640 1 -- 172.21.15.40:0/2719688712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe01403b0 con 0x7f1fe013ec80 2024-03-24T00:33:26.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fe4a00640 1 -- 172.21.15.40:0/2719688712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe0140970 con 0x7f1fe013ec80 2024-03-24T00:33:26.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fe4a00640 1 -- 172.21.15.40:0/2719688712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe0068170 con 0x7f1fe013ec80 2024-03-24T00:33:26.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.403+0000 7f1fd6000640 1 -- 172.21.15.40:0/2719688712 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f1fc80bf480 con 0x7f1fe013ec80 2024-03-24T00:33:26.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.407+0000 7f1fd6000640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1fbc067660 0x7f1fbc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.407+0000 7f1fd6000640 1 -- 172.21.15.40:0/2719688712 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fc8120070 con 0x7f1fe013ec80 2024-03-24T00:33:26.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.407+0000 7f1fdde00640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1fbc067660 0x7f1fbc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.407+0000 7f1fdde00640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1fbc067660 0x7f1fbc069b20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009580 tx=0x7f1fd0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:26.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.411+0000 7f1fd6000640 1 -- 172.21.15.40:0/2719688712 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1fc8016090 con 0x7f1fe013ec80 2024-03-24T00:33:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.607+0000 7f1fe4a00640 1 -- 172.21.15.40:0/2719688712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fe0072b90 con 0x7f1fe013ec80 2024-03-24T00:33:26.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.607+0000 7f1fd6000640 1 -- 172.21.15.40:0/2719688712 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+5015 (secure 0 0 0) 0x7f1fc80f0720 con 0x7f1fe013ec80 2024-03-24T00:33:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:26.610 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":36,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5192,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:22.810241+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5763,"mds_1":5626,"mds_2":5198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5198":{"gid":5198,"name":"b","rank":2,"incarnation":34,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5626":{"gid":5626,"name":"d","rank":1,"incarnation":32,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6836/1600357100","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1600357100},{"type":"v1","addr":"172.21.15.53:6836","nonce":1600357100}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5763,5626,5198]},"id":3}]} 2024-03-24T00:33:26.610 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 36 2024-03-24T00:33:26.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.607+0000 7f1fe4a00640 1 -- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1fbc067660 msgr2=0x7f1fbc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.607+0000 7f1fe4a00640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1fbc067660 0x7f1fbc069b20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009580 tx=0x7f1fd0009190 comp rx=0 tx=0).stop 2024-03-24T00:33:26.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.607+0000 7f1fe4a00640 1 -- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe013ec80 msgr2=0x7f1fe0141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.607+0000 7f1fe4a00640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe013ec80 0x7f1fe0141170 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1fc801cd40 tx=0x7f1fc8019240 comp rx=0 tx=0).stop 2024-03-24T00:33:26.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.607+0000 7f1fe4a00640 1 -- 172.21.15.40:0/2719688712 shutdown_connections 2024-03-24T00:33:26.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.607+0000 7f1fe4a00640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1fbc067660 0x7f1fbc069b20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.607+0000 7f1fe4a00640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1fe01419f0 0x7f1fe0139910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.607+0000 7f1fe4a00640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe013ec80 0x7f1fe0141170 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.607+0000 7f1fe4a00640 1 --2- 172.21.15.40:0/2719688712 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1fe00ce630 0x7f1fe013a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.607+0000 7f1fe4a00640 1 -- 172.21.15.40:0/2719688712 >> 172.21.15.40:0/2719688712 conn(0x7f1fe00b17b0 msgr2=0x7f1fe00b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:26.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.611+0000 7f1fe4a00640 1 -- 172.21.15.40:0/2719688712 shutdown_connections 2024-03-24T00:33:26.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.611+0000 7f1fe4a00640 1 -- 172.21.15.40:0/2719688712 wait complete. 2024-03-24T00:33:26.631 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:33:26.631 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:2 perf dump 2024-03-24T00:33:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83b6a00640 1 Processor -- start 2024-03-24T00:33:26.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83b6a00640 1 -- start start 2024-03-24T00:33:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83b6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f83b00d2270 0x7f83b0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83b6a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83b006c950 0x7f83b0161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83b6a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b006d0f0 con 0x7f83b015df80 2024-03-24T00:33:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83b6a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b006b890 con 0x7f83b00d2270 2024-03-24T00:33:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83b6a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b006b280 con 0x7f83b006c950 2024-03-24T00:33:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83a7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f83b00d2270 0x7f83b0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83a7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f83b00d2270 0x7f83b0070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37114/0 (socket says 172.21.15.40:37114) 2024-03-24T00:33:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83a7e00640 1 -- 172.21.15.40:0/1975362316 learned_addr learned my addr 172.21.15.40:0/1975362316 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83b4a00640 1 --2- 172.21.15.40:0/1975362316 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83b006c950 0x7f83b0161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83a7e00640 1 -- 172.21.15.40:0/1975362316 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83b006c950 msgr2=0x7f83b0161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83a7e00640 1 --2- 172.21.15.40:0/1975362316 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83b006c950 0x7f83b0161ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83a7e00640 1 -- 172.21.15.40:0/1975362316 >> v1:172.21.15.53:6789/0 conn(0x7f83b015df80 legacy=0x7f83b00d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.771+0000 7f83a7e00640 1 -- 172.21.15.40:0/1975362316 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b006ce10 con 0x7f83b00d2270 2024-03-24T00:33:26.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.775+0000 7f83a7e00640 1 --2- 172.21.15.40:0/1975362316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b00d2270 0x7f83b0070ff0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f8398009870 tx=0x7f839801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e537225778a9d49 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.775+0000 7f83af400640 1 -- 172.21.15.40:0/1975362316 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980aa650 con 0x7f83b00d2270 2024-03-24T00:33:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.775+0000 7f83af400640 1 -- 172.21.15.40:0/1975362316 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83980af070 con 0x7f83b00d2270 2024-03-24T00:33:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.775+0000 7f83af400640 1 -- 172.21.15.40:0/1975362316 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980aaa10 con 0x7f83b00d2270 2024-03-24T00:33:26.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.775+0000 7f83b6a00640 1 -- 172.21.15.40:0/1975362316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b00d2270 msgr2=0x7f83b0070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.775+0000 7f83b6a00640 1 --2- 172.21.15.40:0/1975362316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b00d2270 0x7f83b0070ff0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f8398009870 tx=0x7f839801c530 comp rx=0 tx=0).stop 2024-03-24T00:33:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.775+0000 7f83b6a00640 1 -- 172.21.15.40:0/1975362316 shutdown_connections 2024-03-24T00:33:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.775+0000 7f83b6a00640 1 --2- 172.21.15.40:0/1975362316 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83b006c950 0x7f83b0161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.775+0000 7f83b6a00640 1 --2- 172.21.15.40:0/1975362316 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b00d2270 0x7f83b0070ff0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.775+0000 7f83b6a00640 1 -- 172.21.15.40:0/1975362316 >> 172.21.15.40:0/1975362316 conn(0x7f83b006ecf0 msgr2=0x7f83b006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.775+0000 7f83b6a00640 1 -- 172.21.15.40:0/1975362316 shutdown_connections 2024-03-24T00:33:26.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.775+0000 7f83b6a00640 1 -- 172.21.15.40:0/1975362316 wait complete. 2024-03-24T00:33:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.779+0000 7f83b6a00640 1 Processor -- start 2024-03-24T00:33:26.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.779+0000 7f83b6a00640 1 -- start start 2024-03-24T00:33:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.779+0000 7f83b6a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83b006c950 0x7f83b015a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.779+0000 7f83b6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b00d2270 0x7f83b01faac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.779+0000 7f83b6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b015df80 0x7f83b01563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.779+0000 7f83b6a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b016af90 con 0x7f83b00d2270 2024-03-24T00:33:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.779+0000 7f83b6a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b016ae10 con 0x7f83b015df80 2024-03-24T00:33:26.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.779+0000 7f83b6a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b016b110 con 0x7f83b006c950 2024-03-24T00:33:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.779+0000 7f83a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b00d2270 0x7f83b01faac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83afe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83b006c950 0x7f83b015a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b00d2270 0x7f83b01faac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37126/0 (socket says 172.21.15.40:37126) 2024-03-24T00:33:26.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83a7e00640 1 -- 172.21.15.40:0/1408216836 learned_addr learned my addr 172.21.15.40:0/1408216836 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83b4a00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b015df80 0x7f83b01563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83a7e00640 1 -- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83b006c950 msgr2=0x7f83b015a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83a7e00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83b006c950 0x7f83b015a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83a7e00640 1 -- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b015df80 msgr2=0x7f83b01563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83a7e00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b015df80 0x7f83b01563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83a7e00640 1 -- 172.21.15.40:0/1408216836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00b37a0 con 0x7f83b00d2270 2024-03-24T00:33:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83a7e00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b00d2270 0x7f83b01faac0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f839801a160 tx=0x7f839801ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83acc00640 1 -- 172.21.15.40:0/1408216836 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980bd5f0 con 0x7f83b00d2270 2024-03-24T00:33:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83acc00640 1 -- 172.21.15.40:0/1408216836 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83980a6040 con 0x7f83b00d2270 2024-03-24T00:33:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83b6a00640 1 -- 172.21.15.40:0/1408216836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b01fb2f0 con 0x7f83b00d2270 2024-03-24T00:33:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83acc00640 1 -- 172.21.15.40:0/1408216836 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980bd9d0 con 0x7f83b00d2270 2024-03-24T00:33:26.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83b6a00640 1 -- 172.21.15.40:0/1408216836 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b00b3a60 con 0x7f83b00d2270 2024-03-24T00:33:26.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.783+0000 7f83acc00640 1 -- 172.21.15.40:0/1408216836 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f83980b33f0 con 0x7f83b00d2270 2024-03-24T00:33:26.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 Processor -- start 2024-03-24T00:33:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 -- start start 2024-03-24T00:33:26.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83acc00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f838c067660 0x7f838c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83acc00640 1 -- 172.21.15.40:0/1408216836 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f839811fe80 con 0x7f83b00d2270 2024-03-24T00:33:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b4a00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f838c067660 0x7f838c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc68970 0x55895cc68d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc69290 0x55895cc1ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc1f380 0x55895cc1f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895ca6d680 con 0x55895cc69290 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895ca0ab70 con 0x55895cc68970 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55895ca69720 con 0x55895cc1f380 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc69290 0x55895cc1ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc69290 0x55895cc1ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37128/0 (socket says 172.21.15.40:37128) 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83afe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc68970 0x55895cc68d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83a7e00640 1 -- 172.21.15.40:0/2457464939 learned_addr learned my addr 172.21.15.40:0/2457464939 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b4a00640 1 --2- 172.21.15.40:0/2457464939 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc1f380 0x55895cc1f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b4a00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f838c067660 0x7f838c069b20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f83b00ce4a0 tx=0x7f83a8009820 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83a7e00640 1 -- 172.21.15.40:0/2457464939 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc1f380 msgr2=0x55895cc1f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83a7e00640 1 --2- 172.21.15.40:0/2457464939 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc1f380 0x55895cc1f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83a7e00640 1 -- 172.21.15.40:0/2457464939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc68970 msgr2=0x55895cc68d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:26.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83a7e00640 1 --2- 172.21.15.40:0/2457464939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc68970 0x55895cc68d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83a7e00640 1 -- 172.21.15.40:0/2457464939 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55895ca0adc0 con 0x55895cc69290 2024-03-24T00:33:26.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83a7e00640 1 --2- 172.21.15.40:0/2457464939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc69290 0x55895cc1ee40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f839811e460 tx=0x7f83980192d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5044454d2333c830 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83a4c00640 1 -- 172.21.15.40:0/2457464939 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980f45e0 con 0x55895cc69290 2024-03-24T00:33:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83a4c00640 1 -- 172.21.15.40:0/2457464939 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83980af050 con 0x55895cc69290 2024-03-24T00:33:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83a4c00640 1 -- 172.21.15.40:0/2457464939 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980f8ba0 con 0x55895cc69290 2024-03-24T00:33:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 -- 172.21.15.40:0/2457464939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc69290 msgr2=0x55895cc1ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/2457464939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc69290 0x55895cc1ee40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f839811e460 tx=0x7f83980192d0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 -- 172.21.15.40:0/2457464939 shutdown_connections 2024-03-24T00:33:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/2457464939 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc1f380 0x55895cc1f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/2457464939 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc69290 0x55895cc1ee40 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/2457464939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc68970 0x55895cc68d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 -- 172.21.15.40:0/2457464939 >> 172.21.15.40:0/2457464939 conn(0x55895cc1b330 msgr2=0x55895cc1d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 -- 172.21.15.40:0/2457464939 shutdown_connections 2024-03-24T00:33:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.787+0000 7f83b9f7f000 1 -- 172.21.15.40:0/2457464939 wait complete. 2024-03-24T00:33:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83b9f7f000 1 -- start start 2024-03-24T00:33:26.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83b9f7f000 1 Processor -- start 2024-03-24T00:33:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83b9f7f000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 0x55895cc241d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83b9f7f000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc68970 0x55895cc24710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83b9f7f000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc69290 0x55895cc55e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83b9f7f000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895cb38c80 con 0x55895cc1f380 2024-03-24T00:33:26.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83b9f7f000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895ca2b3f0 con 0x55895cc68970 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83b9f7f000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55895ca6d680 con 0x55895cc69290 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83b9f7f000 10 client.0 _ll_register_callbacks cb 0x55895cc0a680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83afe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 0x55895cc241d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83afe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 0x55895cc241d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37136/0 (socket says 172.21.15.40:37136) 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83afe00640 1 -- 172.21.15.40:0/1609059482 learned_addr learned my addr 172.21.15.40:0/1609059482 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83b4a00640 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc69290 0x55895cc55e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83a7e00640 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc68970 0x55895cc24710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83afe00640 1 -- 172.21.15.40:0/1609059482 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc69290 msgr2=0x55895cc55e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83afe00640 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc69290 0x55895cc55e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83afe00640 1 -- 172.21.15.40:0/1609059482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc68970 msgr2=0x55895cc24710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83afe00640 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc68970 0x55895cc24710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83afe00640 1 -- 172.21.15.40:0/1609059482 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55895ca2e6e0 con 0x55895cc1f380 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f83afe00640 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 0x55895cc241d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f83a00122f0 tx=0x7f83a00127b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bea9a613ed0f362 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.791+0000 7f839d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:26.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f839d600640 1 -- 172.21.15.40:0/1609059482 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a0019020 con 0x55895cc1f380 2024-03-24T00:33:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f839d600640 1 -- 172.21.15.40:0/1609059482 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83a00052e0 con 0x55895cc1f380 2024-03-24T00:33:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1609059482 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f83980af050 con 0x55895cc1f380 2024-03-24T00:33:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f839d600640 1 -- 172.21.15.40:0/1609059482 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f83a0023050 con 0x55895cc1f380 2024-03-24T00:33:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 10 client.6036 fetch_fsmap learned FSMap version 36 2024-03-24T00:33:26.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1609059482 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55895ca284a0 con 0x55895cc1f380 2024-03-24T00:33:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f839d600640 1 -- 172.21.15.40:0/1609059482 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83980af050 con 0x55895cc1f380 2024-03-24T00:33:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f839d600640 1 -- 172.21.15.40:0/1609059482 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 36) v1 ==== 3362+0+0 (secure 0 0 0) 0x7f83a0026020 con 0x55895cc1f380 2024-03-24T00:33:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 10 client.6036 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T00:33:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 10 client.6036 resolve_mds: resolved 3:2 to role '3:2' aka daemon mds.b 2024-03-24T00:33:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 20 client.6036 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x55895cc55810 0x55895cc56810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 4 client.6036 mds_command: new command op to 5198 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1609059482 --> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55895ca08cf0 con 0x55895cc55810 2024-03-24T00:33:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83a7e00640 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x55895cc55810 0x55895cc56810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83a7e00640 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x55895cc55810 0x55895cc56810 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4a550a5900e1db92 server_cookie=939322a8f346e2c5 in_seq=0 out_seq=0 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f839d600640 10 client.6036 ms_handle_connect on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f839d600640 1 -- 172.21.15.40:0/1609059482 <== mds.2 v2:172.21.15.53:6835/1453842342 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55895ca08cf0 con 0x55895cc55810 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f839d600640 10 client.6036 handle_command_reply: tid=0 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 client.6036 shutdown 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1609059482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 msgr2=0x55895cc241d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 0x55895cc241d0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f83a00122f0 tx=0x7f83a00127b0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1609059482 shutdown_connections 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x55895cc55810 0x55895cc56810 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc69290 0x55895cc55e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc68970 0x55895cc24710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1609059482 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 0x55895cc241d0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1609059482 >> 172.21.15.40:0/1609059482 conn(0x55895cc1b330 msgr2=0x55895cc1dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f839d600640 0 client.6036 ms_handle_reset on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1609059482 shutdown_connections 2024-03-24T00:33:26.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1609059482 wait complete. 2024-03-24T00:33:26.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.795+0000 7f83b9f7f000 20 client.6036 trim_cache size 0 max 16384 2024-03-24T00:33:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83b9f7f000 1 Processor -- start 2024-03-24T00:33:26.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83b9f7f000 1 -- start start 2024-03-24T00:33:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83b9f7f000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc1b330 0x55895cc1d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83b9f7f000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 0x55895cc1dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83b9f7f000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc68970 0x55895cc1e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83b9f7f000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b02036d0 con 0x55895cc1f380 2024-03-24T00:33:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83b9f7f000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00b7950 con 0x55895cc1b330 2024-03-24T00:33:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83b9f7f000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b0203070 con 0x55895cc68970 2024-03-24T00:33:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 0x55895cc1dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 0x55895cc1dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37146/0 (socket says 172.21.15.40:37146) 2024-03-24T00:33:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83a7e00640 1 -- 172.21.15.40:0/1753146869 learned_addr learned my addr 172.21.15.40:0/1753146869 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83afe00640 1 --2- 172.21.15.40:0/1753146869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc1b330 0x55895cc1d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83b4a00640 1 --2- 172.21.15.40:0/1753146869 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc68970 0x55895cc1e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83a7e00640 1 -- 172.21.15.40:0/1753146869 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc68970 msgr2=0x55895cc1e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83a7e00640 1 --2- 172.21.15.40:0/1753146869 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc68970 0x55895cc1e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83a7e00640 1 -- 172.21.15.40:0/1753146869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc1b330 msgr2=0x55895cc1d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83a7e00640 1 --2- 172.21.15.40:0/1753146869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc1b330 0x55895cc1d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.811+0000 7f83a7e00640 1 -- 172.21.15.40:0/1753146869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55895caee8b0 con 0x55895cc1f380 2024-03-24T00:33:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83a7e00640 1 --2- 172.21.15.40:0/1753146869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 0x55895cc1dc60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8398098020 tx=0x7f8398009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7faa2a61fae23186 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f839fe00640 1 -- 172.21.15.40:0/1753146869 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980ecbb0 con 0x55895cc1f380 2024-03-24T00:33:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f839fe00640 1 -- 172.21.15.40:0/1753146869 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83980af040 con 0x55895cc1f380 2024-03-24T00:33:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f839fe00640 1 -- 172.21.15.40:0/1753146869 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980ecd50 con 0x55895cc1f380 2024-03-24T00:33:26.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1753146869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 msgr2=0x55895cc1dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1753146869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 0x55895cc1dc60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8398098020 tx=0x7f8398009240 comp rx=0 tx=0).stop 2024-03-24T00:33:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1753146869 shutdown_connections 2024-03-24T00:33:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1753146869 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc68970 0x55895cc1e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1753146869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc1f380 0x55895cc1dc60 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1753146869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc1b330 0x55895cc1d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1753146869 >> 172.21.15.40:0/1753146869 conn(0x55895cc24590 msgr2=0x55895cc54980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1753146869 shutdown_connections 2024-03-24T00:33:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1753146869 wait complete. 2024-03-24T00:33:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 -- start start 2024-03-24T00:33:26.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 Processor -- start 2024-03-24T00:33:26.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc1b330 0x55895cc7bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc1f380 0x55895cc7c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc68970 0x55895cc7c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55895ca71cb0 con 0x55895cc68970 2024-03-24T00:33:26.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00b6ab0 con 0x55895cc1f380 2024-03-24T00:33:26.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55895cc552a0 con 0x55895cc1b330 2024-03-24T00:33:26.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b9f7f000 10 client.0 _ll_register_callbacks cb 0x7f83a0001320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:26.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc68970 0x55895cc7c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83afe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc1b330 0x55895cc7bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc68970 0x55895cc7c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37152/0 (socket says 172.21.15.40:37152) 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b4a00640 1 -- 172.21.15.40:0/1061501528 learned_addr learned my addr 172.21.15.40:0/1061501528 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83a7e00640 1 --2- 172.21.15.40:0/1061501528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc1f380 0x55895cc7c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b4a00640 1 -- 172.21.15.40:0/1061501528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc1b330 msgr2=0x55895cc7bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b4a00640 1 --2- 172.21.15.40:0/1061501528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc1b330 0x55895cc7bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b4a00640 1 -- 172.21.15.40:0/1061501528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc1f380 msgr2=0x55895cc7c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b4a00640 1 --2- 172.21.15.40:0/1061501528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc1f380 0x55895cc7c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.815+0000 7f83b4a00640 1 -- 172.21.15.40:0/1061501528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55895cac5fd0 con 0x55895cc68970 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b4a00640 1 --2- 172.21.15.40:0/1061501528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc68970 0x55895cc7c5d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f83a8017aa0 tx=0x7f83a8004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8c5ca529fd9161 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f839d600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f839d600640 1 -- 172.21.15.40:0/1061501528 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8015c30 con 0x55895cc68970 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f839d600640 1 -- 172.21.15.40:0/1061501528 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83a8015dd0 con 0x55895cc68970 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1061501528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f83980af040 con 0x55895cc68970 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f839d600640 1 -- 172.21.15.40:0/1061501528 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f83a8035050 con 0x55895cc68970 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 10 client.6042 fetch_fsmap learned FSMap version 36 2024-03-24T00:33:26.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1061501528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55895c7291c0 con 0x55895cc68970 2024-03-24T00:33:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f839d600640 1 -- 172.21.15.40:0/1061501528 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83980af040 con 0x55895cc68970 2024-03-24T00:33:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f839d600640 1 -- 172.21.15.40:0/1061501528 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 36) v1 ==== 3362+0+0 (secure 0 0 0) 0x7f83a8038020 con 0x55895cc68970 2024-03-24T00:33:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 10 client.6042 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T00:33:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 10 client.6042 resolve_mds: resolved 3:2 to role '3:2' aka daemon mds.b 2024-03-24T00:33:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 20 client.6042 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1061501528 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x55895cc69290 0x55895ccd5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 4 client.6042 mds_command: new command op to 5198 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-03-24T00:33:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1061501528 --> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x55895ca7abc0 con 0x55895cc69290 2024-03-24T00:33:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83afe00640 1 --2- 172.21.15.40:0/1061501528 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x55895cc69290 0x55895ccd5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83afe00640 1 --2- 172.21.15.40:0/1061501528 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x55895cc69290 0x55895ccd5820 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b38a26c697c98e3b server_cookie=9bd5e7d9817bbcc3 in_seq=0 out_seq=0 2024-03-24T00:33:26.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f839d600640 10 client.6042 ms_handle_connect on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f839d600640 1 -- 172.21.15.40:0/1061501528 <== mds.2 v2:172.21.15.53:6835/1453842342 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17649 (crc 0 0 0) 0x55895ca7abc0 con 0x55895cc69290 2024-03-24T00:33:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f839d600640 10 client.6042 handle_command_reply: tid=0 2024-03-24T00:33:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 client.6042 shutdown 2024-03-24T00:33:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1061501528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc68970 msgr2=0x55895cc7c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1061501528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc68970 0x55895cc7c5d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f83a8017aa0 tx=0x7f83a8004ac0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1061501528 shutdown_connections 2024-03-24T00:33:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1061501528 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x55895cc69290 0x55895ccd5820 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1061501528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55895cc68970 0x55895cc7c5d0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1061501528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55895cc1f380 0x55895cc7c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f839d600640 0 client.6042 ms_handle_reset on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:26.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 --2- 172.21.15.40:0/1061501528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55895cc1b330 0x55895cc7bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1061501528 >> 172.21.15.40:0/1061501528 conn(0x55895cc24590 msgr2=0x55895cc66e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1061501528 shutdown_connections 2024-03-24T00:33:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 1 -- 172.21.15.40:0/1061501528 wait complete. 2024-03-24T00:33:26.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b9f7f000 20 client.6042 trim_cache size 0 max 16384 2024-03-24T00:33:26.827 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:33:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 15, 2024-03-24T00:33:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 14, 2024-03-24T00:33:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 4373, 2024-03-24T00:33:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 34075, 2024-03-24T00:33:26.827 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 10, 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.013332771, 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000367956, 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.008741956, 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000719097, 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 13, 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.020925343, 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001609641 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000258, 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000129 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.828 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1568, 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 448 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 49, 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 129, 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 105380, 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 128130, 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 6, 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.013721284, 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.004610060, 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.008713506, 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000623964, 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:26.829 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 127, 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.025797535, 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000203130 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 127, 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000086019, 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000677 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 94880 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 81, 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 149, 2024-03-24T00:33:26.830 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 118222, 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 133410, 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 7, 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.012840673, 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.003955935, 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.011252827, 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000722880, 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 149, 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.030062303, 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000201760 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:26.831 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 13, 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000003223, 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000247 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 112144, 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 12672 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 46, 2024-03-24T00:33:26.832 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.003299416, 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000071726 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000008470, 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000004235 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 0, 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 0, 2024-03-24T00:33:26.833 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 11, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 10, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 0, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 10, 2024-03-24T00:33:26.834 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 10, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 0, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 0, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 0, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 1, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 3, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 0, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 3, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-03-24T00:33:26.835 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 3, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 74, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-03-24T00:33:26.836 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 0, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 0, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 0, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 0, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 0 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-03-24T00:33:26.837 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 4, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-03-24T00:33:26.838 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 0, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 8, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 8, 2024-03-24T00:33:26.839 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 2, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 2, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 2, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 4194304, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 4250433, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 4, 2024-03-24T00:33:26.840 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.009498684, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002374671 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 88, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 88, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 0, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 52, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 53, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 1, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 85, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 85, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 0, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 74, 2024-03-24T00:33:26.841 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 74, 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 0, 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 48928, 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 223536 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 0, 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 0, 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 3, 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.844 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.845 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.846 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.847 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.848 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.849 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-03-24T00:33:26.850 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-03-24T00:33:26.851 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.852 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 1, 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 1, 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 0, 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 1, 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 0, 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 0, 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 0, 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:33:26.853 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-24T00:33:26.854 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-24T00:33:26.855 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 220631, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 57, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-24T00:33:26.856 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 5312, 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 75, 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 50040, 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 198, 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-03-24T00:33:26.857 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 39, 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 68445, 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 33, 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 39, 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 39, 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.335991357, 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.008615163 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 39, 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 62 2024-03-24T00:33:26.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 39, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 0, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 39, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 0, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 10, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 0, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 4, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 5, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2024-03-24T00:33:26.859 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 6, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 0, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 20, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-03-24T00:33:26.860 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-03-24T00:33:26.861 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 0, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 0, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 7, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 7, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 16, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 0, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 1 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.862 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 0, 2024-03-24T00:33:26.863 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 141, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 211006, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 141, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 141, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 211006, 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:26.864 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 39, 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 68445, 2024-03-24T00:33:26.865 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 39, 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 68445, 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:26.866 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 39, 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 39, 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 39, 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 39, 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:26.867 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x558fa1946780": { 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 183, 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-24T00:33:26.868 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 8, 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 56129, 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 8, 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 4, 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 55946, 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:26.869 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b6a00640 1 -- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f838c067660 msgr2=0x7f838c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b6a00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f838c067660 0x7f838c069b20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f83b00ce4a0 tx=0x7f83a8009820 comp rx=0 tx=0).stop 2024-03-24T00:33:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b6a00640 1 -- 172.21.15.40:0/1408216836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b00d2270 msgr2=0x7f83b01faac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b6a00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b00d2270 0x7f83b01faac0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f839801a160 tx=0x7f839801ad70 comp rx=0 tx=0).stop 2024-03-24T00:33:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b6a00640 1 -- 172.21.15.40:0/1408216836 shutdown_connections 2024-03-24T00:33:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b6a00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f838c067660 0x7f838c069b20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b6a00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b015df80 0x7f83b01563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b6a00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f83b00d2270 0x7f83b01faac0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b6a00640 1 --2- 172.21.15.40:0/1408216836 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83b006c950 0x7f83b015a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.819+0000 7f83b6a00640 1 -- 172.21.15.40:0/1408216836 >> 172.21.15.40:0/1408216836 conn(0x7f83b006ecf0 msgr2=0x7f83b006f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.823+0000 7f83b6a00640 1 -- 172.21.15.40:0/1408216836 shutdown_connections 2024-03-24T00:33:26.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:26.823+0000 7f83b6a00640 1 -- 172.21.15.40:0/1408216836 wait complete. 2024-03-24T00:33:26.872 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:33:26.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:1 perf dump 2024-03-24T00:33:27.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.011+0000 7f68dbc00640 1 Processor -- start 2024-03-24T00:33:27.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.011+0000 7f68dbc00640 1 -- start start 2024-03-24T00:33:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68dbc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f68d40cd320 0x7f68d40cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68dbc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f68d4066e80 0x7f68d4067310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68dbc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d40ceff0 con 0x7f68d4145700 2024-03-24T00:33:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68dbc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d40cf8e0 con 0x7f68d4066e80 2024-03-24T00:33:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68dbc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d40cdb70 con 0x7f68d40cd320 2024-03-24T00:33:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68d9c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f68d4066e80 0x7f68d4067310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68d9c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f68d4066e80 0x7f68d4067310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44436/0 (socket says 172.21.15.40:44436) 2024-03-24T00:33:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68d9200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f68d4145700 0x7f68d40725c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41488/0 (socket says 172.21.15.40:41488) 2024-03-24T00:33:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68d3e00640 1 --2- 172.21.15.40:0/182409396 >> v2:172.21.15.53:3300/0 conn(0x7f68d40cd320 0x7f68d40cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68d9c00640 1 -- 172.21.15.40:0/182409396 learned_addr learned my addr 172.21.15.40:0/182409396 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68d9c00640 1 -- 172.21.15.40:0/182409396 >> v2:172.21.15.53:3300/0 conn(0x7f68d40cd320 msgr2=0x7f68d40cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68d9c00640 1 --2- 172.21.15.40:0/182409396 >> v2:172.21.15.53:3300/0 conn(0x7f68d40cd320 0x7f68d40cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68d9c00640 1 -- 172.21.15.40:0/182409396 >> v1:172.21.15.40:6789/0 conn(0x7f68d4145700 legacy=0x7f68d40725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.015+0000 7f68d9c00640 1 -- 172.21.15.40:0/182409396 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d40cd890 con 0x7f68d4066e80 2024-03-24T00:33:27.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68d9c00640 1 --2- 172.21.15.40:0/182409396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d4066e80 0x7f68d4067310 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f68d40b3880 tx=0x7f68cc01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4cc2a8abccd3a6b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68d3400640 1 -- 172.21.15.40:0/182409396 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0a5710 con 0x7f68d4066e80 2024-03-24T00:33:27.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68d3400640 1 -- 172.21.15.40:0/182409396 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc0a58b0 con 0x7f68d4066e80 2024-03-24T00:33:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68d3400640 1 -- 172.21.15.40:0/182409396 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0a5c80 con 0x7f68d4066e80 2024-03-24T00:33:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68dbc00640 1 -- 172.21.15.40:0/182409396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d4066e80 msgr2=0x7f68d4067310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68dbc00640 1 --2- 172.21.15.40:0/182409396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d4066e80 0x7f68d4067310 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f68d40b3880 tx=0x7f68cc01c960 comp rx=0 tx=0).stop 2024-03-24T00:33:27.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68dbc00640 1 -- 172.21.15.40:0/182409396 shutdown_connections 2024-03-24T00:33:27.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68dbc00640 1 --2- 172.21.15.40:0/182409396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d4066e80 0x7f68d4067310 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68dbc00640 1 --2- 172.21.15.40:0/182409396 >> v2:172.21.15.53:3300/0 conn(0x7f68d40cd320 0x7f68d40cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68dbc00640 1 -- 172.21.15.40:0/182409396 >> 172.21.15.40:0/182409396 conn(0x7f68d40b27c0 msgr2=0x7f68d40b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68dbc00640 1 -- 172.21.15.40:0/182409396 shutdown_connections 2024-03-24T00:33:27.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68dbc00640 1 -- 172.21.15.40:0/182409396 wait complete. 2024-03-24T00:33:27.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68dbc00640 1 Processor -- start 2024-03-24T00:33:27.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.019+0000 7f68dbc00640 1 -- start start 2024-03-24T00:33:27.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68dbc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68d4066e80 0x7f68d415b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68dbc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68d40cd320 0x7f68d415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68dbc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d4145700 0x7f68d4137bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68dbc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d406e580 con 0x7f68d4145700 2024-03-24T00:33:27.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68dbc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d406e400 con 0x7f68d4066e80 2024-03-24T00:33:27.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68dbc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d4168ce0 con 0x7f68d40cd320 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d4145700 0x7f68d4137bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d4145700 0x7f68d4137bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44442/0 (socket says 172.21.15.40:44442) 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d9c00640 1 -- 172.21.15.40:0/3560430815 learned_addr learned my addr 172.21.15.40:0/3560430815 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d3e00640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68d40cd320 0x7f68d415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d9200640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68d4066e80 0x7f68d415b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d9c00640 1 -- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68d40cd320 msgr2=0x7f68d415b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d9c00640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68d40cd320 0x7f68d415b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d9c00640 1 -- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68d4066e80 msgr2=0x7f68d415b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d9c00640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68d4066e80 0x7f68d415b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d9c00640 1 -- 172.21.15.40:0/3560430815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d413b1a0 con 0x7f68d4145700 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d9c00640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d4145700 0x7f68d4137bd0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f68cc019570 tx=0x7f68cc019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d0c00640 1 -- 172.21.15.40:0/3560430815 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0bc020 con 0x7f68d4145700 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d0c00640 1 -- 172.21.15.40:0/3560430815 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc0b8670 con 0x7f68d4145700 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68d0c00640 1 -- 172.21.15.40:0/3560430815 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc009510 con 0x7f68d4145700 2024-03-24T00:33:27.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68dbc00640 1 -- 172.21.15.40:0/3560430815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d40b4750 con 0x7f68d4145700 2024-03-24T00:33:27.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.023+0000 7f68dbc00640 1 -- 172.21.15.40:0/3560430815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d413ae30 con 0x7f68d4145700 2024-03-24T00:33:27.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d0c00640 1 -- 172.21.15.40:0/3560430815 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f68cc0bf020 con 0x7f68d4145700 2024-03-24T00:33:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d0c00640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68b4067660 0x7f68b4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d0c00640 1 -- 172.21.15.40:0/3560430815 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68cc120710 con 0x7f68d4145700 2024-03-24T00:33:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d9200640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68b4067660 0x7f68b4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68df00a000 1 Processor -- start 2024-03-24T00:33:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68df00a000 1 -- start start 2024-03-24T00:33:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d9200640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68b4067660 0x7f68b4069b20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f68d41fa240 tx=0x7f68c8007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68df00a000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7c970 0x564370d7cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68df00a000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7d290 0x564370d32e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68df00a000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370d33760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68df00a000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564370b81680 con 0x564370d33380 2024-03-24T00:33:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68df00a000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x564370b1eb70 con 0x564370d7c970 2024-03-24T00:33:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68df00a000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x564370b7d720 con 0x564370d7d290 2024-03-24T00:33:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370d33760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7d290 0x564370d32e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370d33760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44458/0 (socket says 172.21.15.40:44458) 2024-03-24T00:33:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7d290 0x564370d32e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:34584/0 (socket says 172.21.15.40:34584) 2024-03-24T00:33:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d9c00640 1 -- 172.21.15.40:0/1609928098 learned_addr learned my addr 172.21.15.40:0/1609928098 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d9200640 1 --2- 172.21.15.40:0/1609928098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7c970 0x564370d7cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d9c00640 1 -- 172.21.15.40:0/1609928098 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7d290 msgr2=0x564370d32e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d9c00640 1 --2- 172.21.15.40:0/1609928098 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7d290 0x564370d32e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d9c00640 1 -- 172.21.15.40:0/1609928098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7c970 msgr2=0x564370d7cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d9c00640 1 --2- 172.21.15.40:0/1609928098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7c970 0x564370d7cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d9c00640 1 -- 172.21.15.40:0/1609928098 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564370b1edc0 con 0x564370d33380 2024-03-24T00:33:27.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.027+0000 7f68d9200640 1 --2- 172.21.15.40:0/1609928098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7c970 0x564370d7cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:27.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68d9c00640 1 --2- 172.21.15.40:0/1609928098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370d33760 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0f0d50 tx=0x7f68cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b36de4b71b621d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68c5600640 1 -- 172.21.15.40:0/1609928098 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0f4e60 con 0x564370d33380 2024-03-24T00:33:27.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68c5600640 1 -- 172.21.15.40:0/1609928098 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc0f9420 con 0x564370d33380 2024-03-24T00:33:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68c5600640 1 -- 172.21.15.40:0/1609928098 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0a5810 con 0x564370d33380 2024-03-24T00:33:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 -- 172.21.15.40:0/1609928098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 msgr2=0x564370d33760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 --2- 172.21.15.40:0/1609928098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370d33760 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0f0d50 tx=0x7f68cc009bb0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 -- 172.21.15.40:0/1609928098 shutdown_connections 2024-03-24T00:33:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 --2- 172.21.15.40:0/1609928098 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370d33760 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 --2- 172.21.15.40:0/1609928098 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7d290 0x564370d32e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 --2- 172.21.15.40:0/1609928098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7c970 0x564370d7cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 -- 172.21.15.40:0/1609928098 >> 172.21.15.40:0/1609928098 conn(0x564370d2f330 msgr2=0x564370d31720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 -- 172.21.15.40:0/1609928098 shutdown_connections 2024-03-24T00:33:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 -- 172.21.15.40:0/1609928098 wait complete. 2024-03-24T00:33:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 -- start start 2024-03-24T00:33:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 Processor -- start 2024-03-24T00:33:27.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370d38570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7c970 0x564370d3d930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7d290 0x564370d3de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564370c4cc80 con 0x564370d33380 2024-03-24T00:33:27.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x564370b3f3f0 con 0x564370d7d290 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x564370b81680 con 0x564370d7c970 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68df00a000 10 client.0 _ll_register_callbacks cb 0x564370d1e680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68d9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370d38570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68d3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7c970 0x564370d3d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68d9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370d38570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44472/0 (socket says 172.21.15.40:44472) 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68d3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7c970 0x564370d3d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:34588/0 (socket says 172.21.15.40:34588) 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68d9200640 1 -- 172.21.15.40:0/1956120899 learned_addr learned my addr 172.21.15.40:0/1956120899 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68d9c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7d290 0x564370d3de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68d9200640 1 -- 172.21.15.40:0/1956120899 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7c970 msgr2=0x564370d3d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68d9200640 1 --2- 172.21.15.40:0/1956120899 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7c970 0x564370d3d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68d9200640 1 -- 172.21.15.40:0/1956120899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7d290 msgr2=0x564370d3de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68d9200640 1 --2- 172.21.15.40:0/1956120899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7d290 0x564370d3de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.031+0000 7f68d9200640 1 -- 172.21.15.40:0/1956120899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564370b426e0 con 0x564370d33380 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68d9200640 1 --2- 172.21.15.40:0/1956120899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370d38570 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f68c80133f0 tx=0x7f68c8000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab155d5962f4850d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68ba000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68ba000640 1 -- 172.21.15.40:0/1956120899 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8027060 con 0x564370d33380 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 -- 172.21.15.40:0/1956120899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f68cc0f9420 con 0x564370d33380 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68ba000640 1 -- 172.21.15.40:0/1956120899 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68c8022ab0 con 0x564370d33380 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68ba000640 1 -- 172.21.15.40:0/1956120899 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68c8013e30 con 0x564370d33380 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 10 client.6057 fetch_fsmap learned FSMap version 36 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 -- 172.21.15.40:0/1956120899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x564370b3c4a0 con 0x564370d33380 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68ba000640 1 -- 172.21.15.40:0/1956120899 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc0f9420 con 0x564370d33380 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68ba000640 1 -- 172.21.15.40:0/1956120899 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 36) v1 ==== 3362+0+0 (secure 0 0 0) 0x7f68c8022cb0 con 0x564370d33380 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 10 client.6057 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 10 client.6057 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.d 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 20 client.6057 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 --2- 172.21.15.40:0/1956120899 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x564370d82fb0 0x564370d83390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 4 client.6057 mds_command: new command op to 5626 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 -- 172.21.15.40:0/1956120899 --> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564370b1ccf0 con 0x564370d82fb0 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68d3e00640 1 --2- 172.21.15.40:0/1956120899 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x564370d82fb0 0x564370d83390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68d3e00640 1 --2- 172.21.15.40:0/1956120899 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x564370d82fb0 0x564370d83390 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=95f8d5ee2f6947d2 server_cookie=c81411d29aef3c11 in_seq=0 out_seq=0 2024-03-24T00:33:27.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68ba000640 10 client.6057 ms_handle_connect on v2:172.21.15.53:6834/1600357100 2024-03-24T00:33:27.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68ba000640 1 -- 172.21.15.40:0/1956120899 <== mds.1 v2:172.21.15.53:6834/1600357100 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564370b1ccf0 con 0x564370d82fb0 2024-03-24T00:33:27.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68ba000640 10 client.6057 handle_command_reply: tid=0 2024-03-24T00:33:27.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 client.6057 shutdown 2024-03-24T00:33:27.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 -- 172.21.15.40:0/1956120899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 msgr2=0x564370d38570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 --2- 172.21.15.40:0/1956120899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370d38570 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f68c80133f0 tx=0x7f68c8000bd0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 -- 172.21.15.40:0/1956120899 shutdown_connections 2024-03-24T00:33:27.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 --2- 172.21.15.40:0/1956120899 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x564370d82fb0 0x564370d83390 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 --2- 172.21.15.40:0/1956120899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7d290 0x564370d3de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 --2- 172.21.15.40:0/1956120899 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7c970 0x564370d3d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 --2- 172.21.15.40:0/1956120899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370d38570 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 -- 172.21.15.40:0/1956120899 >> 172.21.15.40:0/1956120899 conn(0x564370d2f330 msgr2=0x564370d31fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68ba000640 0 client.6057 ms_handle_reset on v2:172.21.15.53:6834/1600357100 2024-03-24T00:33:27.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 -- 172.21.15.40:0/1956120899 shutdown_connections 2024-03-24T00:33:27.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 1 -- 172.21.15.40:0/1956120899 wait complete. 2024-03-24T00:33:27.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.035+0000 7f68df00a000 20 client.6057 trim_cache size 0 max 16384 2024-03-24T00:33:27.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68df00a000 1 Processor -- start 2024-03-24T00:33:27.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68df00a000 1 -- start start 2024-03-24T00:33:27.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68df00a000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d33380 0x564370d92b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68df00a000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7c970 0x564370d32730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68df00a000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d7d290 0x564370d32c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68df00a000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d4203070 con 0x564370d7d290 2024-03-24T00:33:27.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68df00a000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d42023d0 con 0x564370d33380 2024-03-24T00:33:27.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68df00a000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d42029d0 con 0x564370d7c970 2024-03-24T00:33:27.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68d9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d7d290 0x564370d32c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68d9c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d7d290 0x564370d32c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44486/0 (socket says 172.21.15.40:44486) 2024-03-24T00:33:27.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68d9c00640 1 -- 172.21.15.40:0/1829861787 learned_addr learned my addr 172.21.15.40:0/1829861787 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68d9200640 1 --2- 172.21.15.40:0/1829861787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d33380 0x564370d92b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68d3e00640 1 --2- 172.21.15.40:0/1829861787 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7c970 0x564370d32730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68d9c00640 1 -- 172.21.15.40:0/1829861787 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7c970 msgr2=0x564370d32730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.051+0000 7f68d9c00640 1 --2- 172.21.15.40:0/1829861787 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7c970 0x564370d32730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9c00640 1 -- 172.21.15.40:0/1829861787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d33380 msgr2=0x564370d92b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9c00640 1 --2- 172.21.15.40:0/1829861787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d33380 0x564370d92b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9c00640 1 -- 172.21.15.40:0/1829861787 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564370d7d0c0 con 0x564370d7d290 2024-03-24T00:33:27.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9c00640 1 --2- 172.21.15.40:0/1829861787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d7d290 0x564370d32c70 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f68cc01ced0 tx=0x7f68cc0f3070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807df278177dce08 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68c4c00640 1 -- 172.21.15.40:0/1829861787 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0f9420 con 0x564370d7d290 2024-03-24T00:33:27.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68c4c00640 1 -- 172.21.15.40:0/1829861787 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc0f4e60 con 0x564370d7d290 2024-03-24T00:33:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68c4c00640 1 -- 172.21.15.40:0/1829861787 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0a5810 con 0x564370d7d290 2024-03-24T00:33:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 -- 172.21.15.40:0/1829861787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d7d290 msgr2=0x564370d32c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 --2- 172.21.15.40:0/1829861787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d7d290 0x564370d32c70 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f68cc01ced0 tx=0x7f68cc0f3070 comp rx=0 tx=0).stop 2024-03-24T00:33:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 -- 172.21.15.40:0/1829861787 shutdown_connections 2024-03-24T00:33:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 --2- 172.21.15.40:0/1829861787 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d7d290 0x564370d32c70 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 --2- 172.21.15.40:0/1829861787 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7c970 0x564370d32730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 --2- 172.21.15.40:0/1829861787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d33380 0x564370d92b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 -- 172.21.15.40:0/1829861787 >> 172.21.15.40:0/1829861787 conn(0x564370d2f330 msgr2=0x564370d92060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 -- 172.21.15.40:0/1829861787 shutdown_connections 2024-03-24T00:33:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 -- 172.21.15.40:0/1829861787 wait complete. 2024-03-24T00:33:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 -- start start 2024-03-24T00:33:27.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 Processor -- start 2024-03-24T00:33:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370de1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7c970 0x564370de19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7d290 0x564370d876c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564370c02420 con 0x564370d33380 2024-03-24T00:33:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d4201530 con 0x564370d7c970 2024-03-24T00:33:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x564370b81680 con 0x564370d7d290 2024-03-24T00:33:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68df00a000 10 client.0 _ll_register_callbacks cb 0x7f68c80047f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370de1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7d290 0x564370d876c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7c970 0x564370de19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370de1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44488/0 (socket says 172.21.15.40:44488) 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9200640 1 -- 172.21.15.40:0/1871957844 learned_addr learned my addr 172.21.15.40:0/1871957844 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9200640 1 -- 172.21.15.40:0/1871957844 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7d290 msgr2=0x564370d876c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9200640 1 --2- 172.21.15.40:0/1871957844 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7d290 0x564370d876c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9200640 1 -- 172.21.15.40:0/1871957844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7c970 msgr2=0x564370de19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9200640 1 --2- 172.21.15.40:0/1871957844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7c970 0x564370de19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.055+0000 7f68d9200640 1 -- 172.21.15.40:0/1871957844 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564370c028b0 con 0x564370d33380 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68d9200640 1 --2- 172.21.15.40:0/1871957844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370de1480 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f68c8013fd0 tx=0x7f68c800a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4edb0e6ee3f2525 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68ba000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68ba000640 1 -- 172.21.15.40:0/1871957844 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c8013bc0 con 0x564370d33380 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68ba000640 1 -- 172.21.15.40:0/1871957844 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68c8010040 con 0x564370d33380 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 -- 172.21.15.40:0/1871957844 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f68cc0f4e60 con 0x564370d33380 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68ba000640 1 -- 172.21.15.40:0/1871957844 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68c8044050 con 0x564370d33380 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 10 client.6063 fetch_fsmap learned FSMap version 36 2024-03-24T00:33:27.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 -- 172.21.15.40:0/1871957844 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x564370bd9fd0 con 0x564370d33380 2024-03-24T00:33:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68ba000640 1 -- 172.21.15.40:0/1871957844 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc0f4e60 con 0x564370d33380 2024-03-24T00:33:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68ba000640 1 -- 172.21.15.40:0/1871957844 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 36) v1 ==== 3362+0+0 (secure 0 0 0) 0x7f68c8047020 con 0x564370d33380 2024-03-24T00:33:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 10 client.6063 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T00:33:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 10 client.6063 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.d 2024-03-24T00:33:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 20 client.6063 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 --2- 172.21.15.40:0/1871957844 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x564370d88210 0x564370d88620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 4 client.6063 mds_command: new command op to 5626 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-03-24T00:33:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 -- 172.21.15.40:0/1871957844 --> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x564370b8ebc0 con 0x564370d88210 2024-03-24T00:33:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68d3e00640 1 --2- 172.21.15.40:0/1871957844 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x564370d88210 0x564370d88620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68d3e00640 1 --2- 172.21.15.40:0/1871957844 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x564370d88210 0x564370d88620 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b5fd89d5d8d521ba server_cookie=bd3a91e82b9af600 in_seq=0 out_seq=0 2024-03-24T00:33:27.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68ba000640 10 client.6063 ms_handle_connect on v2:172.21.15.53:6834/1600357100 2024-03-24T00:33:27.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68ba000640 1 -- 172.21.15.40:0/1871957844 <== mds.1 v2:172.21.15.53:6834/1600357100 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17637 (crc 0 0 0) 0x564370b8ebc0 con 0x564370d88210 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68ba000640 10 client.6063 handle_command_reply: tid=0 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 client.6063 shutdown 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 -- 172.21.15.40:0/1871957844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 msgr2=0x564370de1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 --2- 172.21.15.40:0/1871957844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370de1480 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f68c8013fd0 tx=0x7f68c800a000 comp rx=0 tx=0).stop 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 -- 172.21.15.40:0/1871957844 shutdown_connections 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 --2- 172.21.15.40:0/1871957844 >> [v2:172.21.15.53:6834/1600357100,v1:172.21.15.53:6836/1600357100] conn(0x564370d88210 0x564370d88620 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 --2- 172.21.15.40:0/1871957844 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564370d7d290 0x564370d876c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 --2- 172.21.15.40:0/1871957844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564370d7c970 0x564370de19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68ba000640 0 client.6063 ms_handle_reset on v2:172.21.15.53:6834/1600357100 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 --2- 172.21.15.40:0/1871957844 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564370d33380 0x564370de1480 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 -- 172.21.15.40:0/1871957844 >> 172.21.15.40:0/1871957844 conn(0x564370d2f330 msgr2=0x564370d928f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 -- 172.21.15.40:0/1871957844 shutdown_connections 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 1 -- 172.21.15.40:0/1871957844 wait complete. 2024-03-24T00:33:27.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.059+0000 7f68df00a000 20 client.6063 trim_cache size 0 max 16384 2024-03-24T00:33:27.067 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:27.067 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:33:27.067 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 10, 2024-03-24T00:33:27.067 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 11, 2024-03-24T00:33:27.067 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 2517, 2024-03-24T00:33:27.067 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 26128, 2024-03-24T00:33:27.067 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 9, 2024-03-24T00:33:27.067 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-03-24T00:33:27.067 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.007581588, 2024-03-24T00:33:27.067 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000370883, 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.003725540, 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000496947, 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 10, 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.013334376, 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001333437 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000126, 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000063 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.068 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 100, 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 108, 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 150159, 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 54900, 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 5, 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.014220731, 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.003931057, 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.012976124, 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000715538, 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 110, 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.048187955, 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000438072 2024-03-24T00:33:27.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 63, 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000010912, 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000173 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 115248, 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 12336 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 21, 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 46, 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 5744, 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 129742, 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 9, 2024-03-24T00:33:27.070 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.011996128, 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.001487683, 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.009752923, 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000335037, 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 42, 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.023116024, 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000550381 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 8, 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000001470, 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000183 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1728, 2024-03-24T00:33:27.071 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 95328 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 48, 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.001892817, 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000039433 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-03-24T00:33:27.072 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000021322, 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000010661 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 0, 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 0, 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-03-24T00:33:27.073 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 11, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 10, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 0, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 10, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 10, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 0, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 0, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 0, 2024-03-24T00:33:27.074 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 1, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 1, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 0, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 1, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 0, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 0, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 1, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 14, 2024-03-24T00:33:27.075 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-03-24T00:33:27.076 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-03-24T00:33:27.076 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-03-24T00:33:27.076 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 0, 2024-03-24T00:33:27.076 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 0, 2024-03-24T00:33:27.076 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 0, 2024-03-24T00:33:27.076 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T00:33:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-03-24T00:33:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T00:33:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-03-24T00:33:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-03-24T00:33:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T00:33:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T00:33:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-03-24T00:33:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-03-24T00:33:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T00:33:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 0, 2024-03-24T00:33:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T00:33:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 0, 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 0, 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 0, 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 0 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 2, 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 0, 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-03-24T00:33:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2024-03-24T00:33:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 4, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 4, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 2, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 2, 2024-03-24T00:33:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 2, 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 4194304, 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 4206195, 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.005125357, 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002562678 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 28, 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 28, 2024-03-24T00:33:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 0, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 20, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 21, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 1, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 25, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 25, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 0, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 14, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 14, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 0, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 43040, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 223536 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 0, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-03-24T00:33:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 0, 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 3, 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-03-24T00:33:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-03-24T00:33:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 1, 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 1, 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 0, 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 1, 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 0, 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 0, 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 0, 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-03-24T00:33:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:33:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:33:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 212583, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 54, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:33:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 5312, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 75, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 50040, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 198, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-24T00:33:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 37, 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 22523, 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 27, 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 37, 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 37, 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.331991732, 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.008972749 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-03-24T00:33:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 37, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 60 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 37, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 0, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 37, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 0, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 10, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 0, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 2, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 5, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-03-24T00:33:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 6, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 20, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-03-24T00:33:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-03-24T00:33:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 0, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 0, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 7, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 7, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 16, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 0, 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 1 2024-03-24T00:33:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2024-03-24T00:33:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 0, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 127, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 142841, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 127, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 127, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 142841, 2024-03-24T00:33:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 37, 2024-03-24T00:33:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 22523, 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 37, 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 22523, 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 37, 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 37, 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 37, 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 37, 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-24T00:33:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x5642f7ec06e0": { 2024-03-24T00:33:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 61, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 4, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 11891, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 4, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 2, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 11830, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.110 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.063+0000 7f68dbc00640 1 -- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68b4067660 msgr2=0x7f68b4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.063+0000 7f68dbc00640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68b4067660 0x7f68b4069b20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f68d41fa240 tx=0x7f68c8007e90 comp rx=0 tx=0).stop 2024-03-24T00:33:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.063+0000 7f68dbc00640 1 -- 172.21.15.40:0/3560430815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d4145700 msgr2=0x7f68d4137bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.063+0000 7f68dbc00640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d4145700 0x7f68d4137bd0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f68cc019570 tx=0x7f68cc019700 comp rx=0 tx=0).stop 2024-03-24T00:33:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.063+0000 7f68dbc00640 1 -- 172.21.15.40:0/3560430815 shutdown_connections 2024-03-24T00:33:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.063+0000 7f68dbc00640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68b4067660 0x7f68b4069b20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.063+0000 7f68dbc00640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d4145700 0x7f68d4137bd0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.063+0000 7f68dbc00640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68d40cd320 0x7f68d415b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.063+0000 7f68dbc00640 1 --2- 172.21.15.40:0/3560430815 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68d4066e80 0x7f68d415b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.063+0000 7f68dbc00640 1 -- 172.21.15.40:0/3560430815 >> 172.21.15.40:0/3560430815 conn(0x7f68d40b27c0 msgr2=0x7f68d40b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.063+0000 7f68dbc00640 1 -- 172.21.15.40:0/3560430815 shutdown_connections 2024-03-24T00:33:27.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.063+0000 7f68dbc00640 1 -- 172.21.15.40:0/3560430815 wait complete. 2024-03-24T00:33:27.112 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:33:27.112 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 perf dump 2024-03-24T00:33:27.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.251+0000 7fd7e1000640 1 Processor -- start 2024-03-24T00:33:27.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.251+0000 7fd7e1000640 1 -- start start 2024-03-24T00:33:27.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7e1000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7dc0d2270 0x7fd7dc070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7e1000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc06d0f0 con 0x7fd7dc15df80 2024-03-24T00:33:27.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7e1000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7dc06b280 con 0x7fd7dc06c950 2024-03-24T00:33:27.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7e1000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc06b890 con 0x7fd7dc0d2270 2024-03-24T00:33:27.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7d9e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd7dc15df80 0x7fd7dc0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41490/0 (socket says 172.21.15.40:41490) 2024-03-24T00:33:27.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7d9e00640 1 -- 172.21.15.40:0/8323541 learned_addr learned my addr 172.21.15.40:0/8323541 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:27.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/8323541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7dc0d2270 0x7fd7dc070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7d9400640 1 -- 172.21.15.40:0/8323541 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053982533 0 0) 0x7fd7dc06d0f0 con 0x7fd7dc15df80 2024-03-24T00:33:27.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7d9400640 1 -- 172.21.15.40:0/8323541 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7c0003530 con 0x7fd7dc15df80 2024-03-24T00:33:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7d9400640 1 -- 172.21.15.40:0/8323541 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117980874 0 0) 0x7fd7dc06b280 con 0x7fd7dc06c950 2024-03-24T00:33:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7d9400640 1 -- 172.21.15.40:0/8323541 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7dc06d0f0 con 0x7fd7dc06c950 2024-03-24T00:33:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7d3e00640 1 -- 172.21.15.40:0/8323541 >> v1:172.21.15.53:6789/0 conn(0x7fd7dc06c950 legacy=0x7fd7dc161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7d3e00640 1 -- 172.21.15.40:0/8323541 >> v1:172.21.15.40:6789/0 conn(0x7fd7dc15df80 legacy=0x7fd7dc0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.255+0000 7fd7d3e00640 1 -- 172.21.15.40:0/8323541 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc06ce10 con 0x7fd7dc0d2270 2024-03-24T00:33:27.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.259+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/8323541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7dc0d2270 0x7fd7dc070ff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40097f0 tx=0x7fd7c401c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1cbe553db4d5be9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.259+0000 7fd7d9400640 1 -- 172.21.15.40:0/8323541 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40a7070 con 0x7fd7dc0d2270 2024-03-24T00:33:27.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.259+0000 7fd7d9400640 1 -- 172.21.15.40:0/8323541 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c40b0070 con 0x7fd7dc0d2270 2024-03-24T00:33:27.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.259+0000 7fd7d9400640 1 -- 172.21.15.40:0/8323541 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40ab870 con 0x7fd7dc0d2270 2024-03-24T00:33:27.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.259+0000 7fd7e1000640 1 -- 172.21.15.40:0/8323541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7dc0d2270 msgr2=0x7fd7dc070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.259+0000 7fd7e1000640 1 --2- 172.21.15.40:0/8323541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7dc0d2270 0x7fd7dc070ff0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40097f0 tx=0x7fd7c401c4b0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.259+0000 7fd7e1000640 1 -- 172.21.15.40:0/8323541 shutdown_connections 2024-03-24T00:33:27.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.259+0000 7fd7e1000640 1 --2- 172.21.15.40:0/8323541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7dc0d2270 0x7fd7dc070ff0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.259+0000 7fd7e1000640 1 -- 172.21.15.40:0/8323541 >> 172.21.15.40:0/8323541 conn(0x7fd7dc06ecf0 msgr2=0x7fd7dc06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.259+0000 7fd7e1000640 1 -- 172.21.15.40:0/8323541 shutdown_connections 2024-03-24T00:33:27.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.259+0000 7fd7e1000640 1 -- 172.21.15.40:0/8323541 wait complete. 2024-03-24T00:33:27.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7e1000640 1 Processor -- start 2024-03-24T00:33:27.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7e1000640 1 -- start start 2024-03-24T00:33:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7e1000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7dc06c950 0x7fd7dc155460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7e1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7dc0d2270 0x7fd7dc0ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7e1000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7dc15df80 0x7fd7dc0b01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7e1000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc16bfa0 con 0x7fd7dc0d2270 2024-03-24T00:33:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7e1000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc16be20 con 0x7fd7dc15df80 2024-03-24T00:33:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7e1000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc16c120 con 0x7fd7dc06c950 2024-03-24T00:33:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7dc0d2270 0x7fd7dc0ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7dc0d2270 0x7fd7dc0ae930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44494/0 (socket says 172.21.15.40:44494) 2024-03-24T00:33:27.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d3e00640 1 -- 172.21.15.40:0/2793090374 learned_addr learned my addr 172.21.15.40:0/2793090374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d9e00640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7dc06c950 0x7fd7dc155460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7da800640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7dc15df80 0x7fd7dc0b01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d3e00640 1 -- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7dc06c950 msgr2=0x7fd7dc155460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7dc06c950 0x7fd7dc155460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d3e00640 1 -- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7dc15df80 msgr2=0x7fd7dc0b01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7dc15df80 0x7fd7dc0b01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d3e00640 1 -- 172.21.15.40:0/2793090374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7dc0aee70 con 0x7fd7dc0d2270 2024-03-24T00:33:27.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7dc0d2270 0x7fd7dc0ae930 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40097f0 tx=0x7fd7c401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d2000640 1 -- 172.21.15.40:0/2793090374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4006e20 con 0x7fd7dc0d2270 2024-03-24T00:33:27.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d2000640 1 -- 172.21.15.40:0/2793090374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c40abe50 con 0x7fd7dc0d2270 2024-03-24T00:33:27.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7e1000640 1 -- 172.21.15.40:0/2793090374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7dc0af110 con 0x7fd7dc0d2270 2024-03-24T00:33:27.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7d2000640 1 -- 172.21.15.40:0/2793090374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40bf810 con 0x7fd7dc0d2270 2024-03-24T00:33:27.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.263+0000 7fd7e1000640 1 -- 172.21.15.40:0/2793090374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc0af6d0 con 0x7fd7dc0d2270 2024-03-24T00:33:27.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7d2000640 1 -- 172.21.15.40:0/2793090374 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd7c40bd020 con 0x7fd7dc0d2270 2024-03-24T00:33:27.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7d2000640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7b8067660 0x7fd7b8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7d2000640 1 -- 172.21.15.40:0/2793090374 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7c411fe50 con 0x7fd7dc0d2270 2024-03-24T00:33:27.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7d9e00640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7b8067660 0x7fd7b8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7e449e000 1 Processor -- start 2024-03-24T00:33:27.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7e449e000 1 -- start start 2024-03-24T00:33:27.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7d9e00640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7b8067660 0x7fd7b8069b20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc007fd0 tx=0x7fd7cc007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7e449e000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e85f970 0x55ff5e85fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7e449e000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e860290 0x55ff5e815e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7e449e000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e816380 0x55ff5e816760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7e449e000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff5e664680 con 0x55ff5e85f970 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7e449e000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff5e601b70 con 0x55ff5e860290 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7e449e000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ff5e660720 con 0x55ff5e816380 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e85f970 0x55ff5e85fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7d3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e860290 0x55ff5e815e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7d3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e860290 0x55ff5e815e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:42484/0 (socket says 172.21.15.40:42484) 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7d3e00640 1 -- 172.21.15.40:0/1186341361 learned_addr learned my addr 172.21.15.40:0/1186341361 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.267+0000 7fd7da800640 1 --2- 172.21.15.40:0/1186341361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e816380 0x55ff5e816760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d3e00640 1 -- 172.21.15.40:0/1186341361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e816380 msgr2=0x55ff5e816760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/1186341361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e816380 0x55ff5e816760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d3e00640 1 -- 172.21.15.40:0/1186341361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e85f970 msgr2=0x55ff5e85fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/1186341361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e85f970 0x55ff5e85fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d3e00640 1 -- 172.21.15.40:0/1186341361 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff5e601dc0 con 0x55ff5e860290 2024-03-24T00:33:27.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/1186341361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e860290 0x55ff5e815e40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40ec840 tx=0x7fd7c4019250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8d1e986b7311554 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7caa00640 1 -- 172.21.15.40:0/1186341361 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40ee480 con 0x55ff5e860290 2024-03-24T00:33:27.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7caa00640 1 -- 172.21.15.40:0/1186341361 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c40b0070 con 0x55ff5e860290 2024-03-24T00:33:27.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7caa00640 1 -- 172.21.15.40:0/1186341361 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40eca50 con 0x55ff5e860290 2024-03-24T00:33:27.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 -- 172.21.15.40:0/1186341361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e860290 msgr2=0x55ff5e815e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 --2- 172.21.15.40:0/1186341361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e860290 0x55ff5e815e40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40ec840 tx=0x7fd7c4019250 comp rx=0 tx=0).stop 2024-03-24T00:33:27.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 -- 172.21.15.40:0/1186341361 shutdown_connections 2024-03-24T00:33:27.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 --2- 172.21.15.40:0/1186341361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e816380 0x55ff5e816760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 --2- 172.21.15.40:0/1186341361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e860290 0x55ff5e815e40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 --2- 172.21.15.40:0/1186341361 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e85f970 0x55ff5e85fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 -- 172.21.15.40:0/1186341361 >> 172.21.15.40:0/1186341361 conn(0x55ff5e812330 msgr2=0x55ff5e814720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 -- 172.21.15.40:0/1186341361 shutdown_connections 2024-03-24T00:33:27.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 -- 172.21.15.40:0/1186341361 wait complete. 2024-03-24T00:33:27.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 -- start start 2024-03-24T00:33:27.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 Processor -- start 2024-03-24T00:33:27.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 0x55ff5e81b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e85f970 0x55ff5e820930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e860290 0x55ff5e820e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff5e72fc80 con 0x55ff5e816380 2024-03-24T00:33:27.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff5e6223f0 con 0x55ff5e85f970 2024-03-24T00:33:27.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ff5e668cb0 con 0x55ff5e860290 2024-03-24T00:33:27.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7e449e000 10 client.0 _ll_register_callbacks cb 0x55ff5e801680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:27.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 0x55ff5e81b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 0x55ff5e81b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44504/0 (socket says 172.21.15.40:44504) 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d9e00640 1 -- 172.21.15.40:0/48236725 learned_addr learned my addr 172.21.15.40:0/48236725 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e85f970 0x55ff5e820930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7da800640 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e860290 0x55ff5e820e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d9e00640 1 -- 172.21.15.40:0/48236725 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e860290 msgr2=0x55ff5e820e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d9e00640 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e860290 0x55ff5e820e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d9e00640 1 -- 172.21.15.40:0/48236725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e85f970 msgr2=0x55ff5e820930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d9e00640 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e85f970 0x55ff5e820930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.271+0000 7fd7d9e00640 1 -- 172.21.15.40:0/48236725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff5e6256e0 con 0x55ff5e816380 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7d9e00640 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 0x55ff5e81b560 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0146d0 tx=0x7fd7cc001160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23bc3c7da7957f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7bf400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7bf400640 1 -- 172.21.15.40:0/48236725 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc025020 con 0x55ff5e816380 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 -- 172.21.15.40:0/48236725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7c40b0070 con 0x55ff5e816380 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7bf400640 1 -- 172.21.15.40:0/48236725 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7cc013700 con 0x55ff5e816380 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7bf400640 1 -- 172.21.15.40:0/48236725 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7cc02f050 con 0x55ff5e816380 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 10 client.6075 fetch_fsmap learned FSMap version 36 2024-03-24T00:33:27.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 -- 172.21.15.40:0/48236725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55ff5e61f4a0 con 0x55ff5e816380 2024-03-24T00:33:27.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7bf400640 1 -- 172.21.15.40:0/48236725 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c40b0070 con 0x55ff5e816380 2024-03-24T00:33:27.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7bf400640 1 -- 172.21.15.40:0/48236725 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 36) v1 ==== 3362+0+0 (secure 0 0 0) 0x7fd7cc032020 con 0x55ff5e816380 2024-03-24T00:33:27.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 10 client.6075 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T00:33:27.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 10 client.6075 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-24T00:33:27.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 20 client.6075 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:27.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55ff5e865fb0 0x55ff5e8663c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 4 client.6075 mds_command: new command op to 5763 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:27.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 -- 172.21.15.40:0/48236725 --> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ff5e5ffcf0 con 0x55ff5e865fb0 2024-03-24T00:33:27.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55ff5e865fb0 0x55ff5e8663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7bf400640 10 client.6075 ms_handle_connect on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:27.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55ff5e865fb0 0x55ff5e8663c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e122103802291141 server_cookie=96502df97ab65c59 in_seq=0 out_seq=0 2024-03-24T00:33:27.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7bf400640 1 -- 172.21.15.40:0/48236725 <== mds.0 v2:172.21.15.40:6832/224223662 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ff5e5ffcf0 con 0x55ff5e865fb0 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7bf400640 10 client.6075 handle_command_reply: tid=0 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 client.6075 shutdown 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 -- 172.21.15.40:0/48236725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 msgr2=0x55ff5e81b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 0x55ff5e81b560 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0146d0 tx=0x7fd7cc001160 comp rx=0 tx=0).stop 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 -- 172.21.15.40:0/48236725 shutdown_connections 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55ff5e865fb0 0x55ff5e8663c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e860290 0x55ff5e820e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e85f970 0x55ff5e820930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7bf400640 0 client.6075 ms_handle_reset on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 --2- 172.21.15.40:0/48236725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 0x55ff5e81b560 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 -- 172.21.15.40:0/48236725 >> 172.21.15.40:0/48236725 conn(0x55ff5e812330 msgr2=0x55ff5e814fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 -- 172.21.15.40:0/48236725 shutdown_connections 2024-03-24T00:33:27.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 1 -- 172.21.15.40:0/48236725 wait complete. 2024-03-24T00:33:27.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.275+0000 7fd7e449e000 20 client.6075 trim_cache size 0 max 16384 2024-03-24T00:33:27.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7e449e000 1 Processor -- start 2024-03-24T00:33:27.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7e449e000 1 -- start start 2024-03-24T00:33:27.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7e449e000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 0x55ff5e875b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7e449e000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e85f970 0x55ff5e815730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7e449e000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e860290 0x55ff5e815c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7e449e000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc203290 con 0x55ff5e816380 2024-03-24T00:33:27.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7e449e000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc1546b0 con 0x55ff5e860290 2024-03-24T00:33:27.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7e449e000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc154cb0 con 0x55ff5e85f970 2024-03-24T00:33:27.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 0x55ff5e875b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7d9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 0x55ff5e875b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44506/0 (socket says 172.21.15.40:44506) 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7d9e00640 1 -- 172.21.15.40:0/3391316911 learned_addr learned my addr 172.21.15.40:0/3391316911 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/3391316911 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e85f970 0x55ff5e815730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7da800640 1 --2- 172.21.15.40:0/3391316911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e860290 0x55ff5e815c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7d9e00640 1 -- 172.21.15.40:0/3391316911 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e85f970 msgr2=0x55ff5e815730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7d9e00640 1 --2- 172.21.15.40:0/3391316911 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e85f970 0x55ff5e815730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7d9e00640 1 -- 172.21.15.40:0/3391316911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e860290 msgr2=0x55ff5e815c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7d9e00640 1 --2- 172.21.15.40:0/3391316911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e860290 0x55ff5e815c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7d9e00640 1 -- 172.21.15.40:0/3391316911 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff5e81afa0 con 0x55ff5e816380 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/3391316911 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e85f970 0x55ff5e815730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7d9e00640 1 --2- 172.21.15.40:0/3391316911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 0x55ff5e875b90 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc001210 tx=0x7fd7cc01b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5443df1e16cc35b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.291+0000 7fd7ca000640 1 -- 172.21.15.40:0/3391316911 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc025020 con 0x55ff5e816380 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7ca000640 1 -- 172.21.15.40:0/3391316911 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7cc01b070 con 0x55ff5e816380 2024-03-24T00:33:27.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7ca000640 1 -- 172.21.15.40:0/3391316911 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc013ca0 con 0x55ff5e816380 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 -- 172.21.15.40:0/3391316911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 msgr2=0x55ff5e875b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 --2- 172.21.15.40:0/3391316911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 0x55ff5e875b90 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc001210 tx=0x7fd7cc01b000 comp rx=0 tx=0).stop 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 -- 172.21.15.40:0/3391316911 shutdown_connections 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 --2- 172.21.15.40:0/3391316911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e860290 0x55ff5e815c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 --2- 172.21.15.40:0/3391316911 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e85f970 0x55ff5e815730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 --2- 172.21.15.40:0/3391316911 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e816380 0x55ff5e875b90 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 -- 172.21.15.40:0/3391316911 >> 172.21.15.40:0/3391316911 conn(0x55ff5e812330 msgr2=0x55ff5e875060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 -- 172.21.15.40:0/3391316911 shutdown_connections 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 -- 172.21.15.40:0/3391316911 wait complete. 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 -- start start 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 Processor -- start 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e816380 0x55ff5e8c4480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e85f970 0x55ff5e8c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e860290 0x55ff5e86a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff5e8121b0 con 0x55ff5e85f970 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc156910 con 0x55ff5e816380 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ff5e81f0f0 con 0x55ff5e860290 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 10 client.0 _ll_register_callbacks cb 0x7fd7cc006c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e85f970 0x55ff5e8c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7d9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e816380 0x55ff5e8c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7d9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e816380 0x55ff5e8c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:42504/0 (socket says 172.21.15.40:42504) 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7d9e00640 1 -- 172.21.15.40:0/4010511606 learned_addr learned my addr 172.21.15.40:0/4010511606 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7da800640 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e860290 0x55ff5e86a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7d3e00640 1 -- 172.21.15.40:0/4010511606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e860290 msgr2=0x55ff5e86a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e860290 0x55ff5e86a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7d3e00640 1 -- 172.21.15.40:0/4010511606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e816380 msgr2=0x55ff5e8c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e816380 0x55ff5e8c4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7d3e00640 1 -- 172.21.15.40:0/4010511606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff5e8600c0 con 0x55ff5e85f970 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7d9e00640 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e816380 0x55ff5e8c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7d3e00640 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e85f970 0x55ff5e8c49c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd7c401ca20 tx=0x7fd7c401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712d90fd085f1c78 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7bf400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7bf400640 1 -- 172.21.15.40:0/4010511606 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c40f4150 con 0x55ff5e85f970 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7bf400640 1 -- 172.21.15.40:0/4010511606 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7c40eca50 con 0x55ff5e85f970 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.295+0000 7fd7e449e000 1 -- 172.21.15.40:0/4010511606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7cc01b070 con 0x55ff5e85f970 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7bf400640 1 -- 172.21.15.40:0/4010511606 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7c40f85b0 con 0x55ff5e85f970 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 10 client.6081 fetch_fsmap learned FSMap version 36 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 -- 172.21.15.40:0/4010511606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55ff5e6e58b0 con 0x55ff5e85f970 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7bf400640 1 -- 172.21.15.40:0/4010511606 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7cc01b070 con 0x55ff5e85f970 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7bf400640 1 -- 172.21.15.40:0/4010511606 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 36) v1 ==== 3362+0+0 (secure 0 0 0) 0x7fd7c40c7030 con 0x55ff5e85f970 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 10 client.6081 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 10 client.6081 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-24T00:33:27.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 20 client.6081 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:27.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55ff5e86b050 0x55ff5e86b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 4 client.6081 mds_command: new command op to 5763 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-03-24T00:33:27.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 -- 172.21.15.40:0/4010511606 --> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x55ff5e671bc0 con 0x55ff5e86b050 2024-03-24T00:33:27.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7d9e00640 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55ff5e86b050 0x55ff5e86b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7d9e00640 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55ff5e86b050 0x55ff5e86b460 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee3aecfd84520047 server_cookie=3a4bcb8e231381ca in_seq=0 out_seq=0 2024-03-24T00:33:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7bf400640 10 client.6081 ms_handle_connect on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7bf400640 1 -- 172.21.15.40:0/4010511606 <== mds.0 v2:172.21.15.40:6832/224223662 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17822 (crc 0 0 0) 0x55ff5e671bc0 con 0x55ff5e86b050 2024-03-24T00:33:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7bf400640 10 client.6081 handle_command_reply: tid=0 2024-03-24T00:33:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 client.6081 shutdown 2024-03-24T00:33:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 -- 172.21.15.40:0/4010511606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e85f970 msgr2=0x55ff5e8c49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e85f970 0x55ff5e8c49c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fd7c401ca20 tx=0x7fd7c401c4b0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 -- 172.21.15.40:0/4010511606 shutdown_connections 2024-03-24T00:33:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55ff5e86b050 0x55ff5e86b460 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ff5e860290 0x55ff5e86a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ff5e85f970 0x55ff5e8c49c0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 --2- 172.21.15.40:0/4010511606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ff5e816380 0x55ff5e8c4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7bf400640 0 client.6081 ms_handle_reset on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 -- 172.21.15.40:0/4010511606 >> 172.21.15.40:0/4010511606 conn(0x55ff5e812330 msgr2=0x55ff5e8758f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 -- 172.21.15.40:0/4010511606 shutdown_connections 2024-03-24T00:33:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 1 -- 172.21.15.40:0/4010511606 wait complete. 2024-03-24T00:33:27.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e449e000 20 client.6081 trim_cache size 0 max 16384 2024-03-24T00:33:27.306 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:27.306 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:33:27.306 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 957, 2024-03-24T00:33:27.306 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 939, 2024-03-24T00:33:27.306 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 241559, 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 2794731, 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 8, 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 3, 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.095952129, 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.039823009, 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.028648871, 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.019142478, 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 937, 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.165797252, 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000176944 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 71, 2024-03-24T00:33:27.307 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000009122, 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000128 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2145, 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 3681, 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 836244, 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 1741696, 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 13, 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 8, 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.334500822, 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.178050220, 2024-03-24T00:33:27.308 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.189874312, 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.003389923, 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3678, 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.289888638, 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000078816 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 5188, 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.001220450, 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000235 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 125424, 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 14736 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 257, 2024-03-24T00:33:27.309 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 192, 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 66409, 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 285509, 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 7, 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.024451097, 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.008526348, 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.010751947, 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.004392369, 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 194, 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.186234938, 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000959973 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:27.310 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 133, 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000015427, 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000115 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 68384 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1474, 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.846588198, 2024-03-24T00:33:27.311 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000574347 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 364, 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.028083014, 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000077151 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 1506, 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 1506, 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1506, 2024-03-24T00:33:27.312 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 2.158336002, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001433158 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 21, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 1, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 212, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 0, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 212, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-03-24T00:33:27.313 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 212, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 88, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 202, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 202, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 2, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 3005, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 1194, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 0, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-03-24T00:33:27.314 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 4, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 88, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 0, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 99, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 55, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 0, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-03-24T00:33:27.315 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 339, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 573, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 439, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 336, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 3, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 913 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-03-24T00:33:27.316 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 0, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 6, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 237, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 237, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-03-24T00:33:27.317 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 4, 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 1, 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 1265, 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-03-24T00:33:27.318 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 1265, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 3, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 3, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 2, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 4194304, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 6658784, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 823, 2024-03-24T00:33:27.319 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 1.561652349, 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001897511 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 126, 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 451, 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 325, 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 76, 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 122, 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 46, 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 124, 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 685, 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 561, 2024-03-24T00:33:27.320 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 114, 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 439, 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 325, 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 56372, 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 231728 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 2180, 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 1506, 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 8, 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 336, 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.716311475, 2024-03-24T00:33:27.321 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002131879 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 16, 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.003846698, 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000240418 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.322 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 811, 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.144534233, 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000178217 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.323 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.324 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 102, 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.280717748, 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002752134 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-03-24T00:33:27.325 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.012218083, 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.006109041 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-03-24T00:33:27.326 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.327 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1, 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.004737418, 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.004737418 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-03-24T00:33:27.328 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1, 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000240407, 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000240407 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.329 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1, 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000494896, 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000494896 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 236, 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.995235044, 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.004217097 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-03-24T00:33:27.331 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 1, 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 1, 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 0, 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 1, 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 1408, 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 1408, 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 18, 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:33:27.332 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:33:27.333 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-24T00:33:27.334 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 1477470, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 403, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 88, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 1, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 5312, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 75, 2024-03-24T00:33:27.335 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 1054957, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 5850, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 1144, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 2636358, 2024-03-24T00:33:27.336 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 37, 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 1144, 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1144, 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 3.199972327, 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002797178 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1144, 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 1194 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 1144, 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 7, 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 1137, 2024-03-24T00:33:27.337 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 0, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 11, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 7, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 830, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 11, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 1, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 249, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-03-24T00:33:27.338 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 22, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-03-24T00:33:27.339 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 44, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 1, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 8, 2024-03-24T00:33:27.340 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 8, 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 53, 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 0, 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 10 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-03-24T00:33:27.341 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 10, 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 10, 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 237, 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 237, 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 3354, 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 837412, 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:27.342 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 3354, 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 3354, 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 837412, 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-24T00:33:27.343 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 1144, 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 2660295, 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 1144, 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 2660295, 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.344 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 1144, 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 1144, 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 1144, 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1144, 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.345 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 237, 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 23937, 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 237, 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:27.346 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 7, 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 23937, 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x559c68bcc780": { 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 1265, 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 2464480, 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:27.347 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 1265, 2024-03-24T00:33:27.348 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:27.348 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:27.348 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 823, 2024-03-24T00:33:27.348 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 2464480, 2024-03-24T00:33:27.348 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:27.348 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:27.348 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:27.348 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:27.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.348 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:27.348 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:27.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e1000640 1 -- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7b8067660 msgr2=0x7fd7b8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e1000640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7b8067660 0x7fd7b8069b20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc007fd0 tx=0x7fd7cc007d80 comp rx=0 tx=0).stop 2024-03-24T00:33:27.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e1000640 1 -- 172.21.15.40:0/2793090374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7dc0d2270 msgr2=0x7fd7dc0ae930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e1000640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7dc0d2270 0x7fd7dc0ae930 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40097f0 tx=0x7fd7c401acf0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e1000640 1 -- 172.21.15.40:0/2793090374 shutdown_connections 2024-03-24T00:33:27.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e1000640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7b8067660 0x7fd7b8069b20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e1000640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7dc15df80 0x7fd7dc0b01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e1000640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7dc0d2270 0x7fd7dc0ae930 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e1000640 1 --2- 172.21.15.40:0/2793090374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7dc06c950 0x7fd7dc155460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e1000640 1 -- 172.21.15.40:0/2793090374 >> 172.21.15.40:0/2793090374 conn(0x7fd7dc06ecf0 msgr2=0x7fd7dc160ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e1000640 1 -- 172.21.15.40:0/2793090374 shutdown_connections 2024-03-24T00:33:27.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.299+0000 7fd7e1000640 1 -- 172.21.15.40:0/2793090374 wait complete. 2024-03-24T00:33:27.351 INFO:tasks.cephfs.test_exports:export stats: [(0, 4), (1, 0), (2, 0)] 2024-03-24T00:33:27.351 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail 3:1 2024-03-24T00:33:27.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.487+0000 7f0813800640 1 Processor -- start 2024-03-24T00:33:27.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.487+0000 7f0813800640 1 -- start start 2024-03-24T00:33:27.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0813800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c06e620 0x7f080c0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0813800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f080c0ccfa0 0x7f080c0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0813800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c070f60 con 0x7f080c06ea00 2024-03-24T00:33:27.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0813800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f080c0b4c00 con 0x7f080c0ccfa0 2024-03-24T00:33:27.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0813800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c0b3620 con 0x7f080c06e620 2024-03-24T00:33:27.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0811800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f080c0ccfa0 0x7f080c0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0810e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c06e620 0x7f080c0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0811800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f080c0ccfa0 0x7f080c0cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44526/0 (socket says 172.21.15.40:44526) 2024-03-24T00:33:27.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0811800640 1 -- 172.21.15.40:0/2249662393 learned_addr learned my addr 172.21.15.40:0/2249662393 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f080be00640 1 --1- 172.21.15.40:0/2249662393 >> v1:172.21.15.40:6789/0 conn(0x7f080c06ea00 0x7f080c0b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41494/0 (socket says 172.21.15.40:41494) 2024-03-24T00:33:27.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0810e00640 1 -- 172.21.15.40:0/2249662393 >> v2:172.21.15.40:3300/0 conn(0x7f080c0ccfa0 msgr2=0x7f080c0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:27.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0810e00640 1 --2- 172.21.15.40:0/2249662393 >> v2:172.21.15.40:3300/0 conn(0x7f080c0ccfa0 0x7f080c0cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0810e00640 1 -- 172.21.15.40:0/2249662393 >> v1:172.21.15.40:6789/0 conn(0x7f080c06ea00 legacy=0x7f080c0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.491+0000 7f0810e00640 1 -- 172.21.15.40:0/2249662393 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c0b3900 con 0x7f080c06e620 2024-03-24T00:33:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.495+0000 7f0810e00640 1 --2- 172.21.15.40:0/2249662393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c06e620 0x7f080c0b46c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f08000097c0 tx=0x7f080001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76a1d2ec99b7bfcb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.495+0000 7f080b400640 1 -- 172.21.15.40:0/2249662393 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08000b0070 con 0x7f080c06e620 2024-03-24T00:33:27.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.495+0000 7f080b400640 1 -- 172.21.15.40:0/2249662393 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08000ab600 con 0x7f080c06e620 2024-03-24T00:33:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.495+0000 7f080b400640 1 -- 172.21.15.40:0/2249662393 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08000ab9d0 con 0x7f080c06e620 2024-03-24T00:33:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.495+0000 7f0813800640 1 -- 172.21.15.40:0/2249662393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c06e620 msgr2=0x7f080c0b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.495+0000 7f0813800640 1 --2- 172.21.15.40:0/2249662393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c06e620 0x7f080c0b46c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f08000097c0 tx=0x7f080001c3e0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.495+0000 7f0813800640 1 -- 172.21.15.40:0/2249662393 shutdown_connections 2024-03-24T00:33:27.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.495+0000 7f0813800640 1 --2- 172.21.15.40:0/2249662393 >> v2:172.21.15.40:3300/0 conn(0x7f080c0ccfa0 0x7f080c0cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.495+0000 7f0813800640 1 --2- 172.21.15.40:0/2249662393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c06e620 0x7f080c0b46c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.495+0000 7f0813800640 1 -- 172.21.15.40:0/2249662393 >> 172.21.15.40:0/2249662393 conn(0x7f080c0b17b0 msgr2=0x7f080c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0813800640 1 -- 172.21.15.40:0/2249662393 shutdown_connections 2024-03-24T00:33:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0813800640 1 -- 172.21.15.40:0/2249662393 wait complete. 2024-03-24T00:33:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0813800640 1 Processor -- start 2024-03-24T00:33:27.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0813800640 1 -- start start 2024-03-24T00:33:27.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0813800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c06e620 0x7f080c1f7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0813800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c06ea00 0x7f080c1552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0813800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f080c0ccfa0 0x7f080c0aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0813800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c144920 con 0x7f080c06ea00 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0813800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c06ede0 con 0x7f080c0ccfa0 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0813800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c15b390 con 0x7f080c06e620 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f080be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c06ea00 0x7f080c1552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f080be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c06ea00 0x7f080c1552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44538/0 (socket says 172.21.15.40:44538) 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f080be00640 1 -- 172.21.15.40:0/3681517016 learned_addr learned my addr 172.21.15.40:0/3681517016 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0811800640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f080c0ccfa0 0x7f080c0aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0810e00640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c06e620 0x7f080c1f7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f080be00640 1 -- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c06e620 msgr2=0x7f080c1f7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f080be00640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c06e620 0x7f080c1f7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f080be00640 1 -- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f080c0ccfa0 msgr2=0x7f080c0aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f080be00640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f080c0ccfa0 0x7f080c0aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f080be00640 1 -- 172.21.15.40:0/3681517016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c15b670 con 0x7f080c06ea00 2024-03-24T00:33:27.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f080be00640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c06ea00 0x7f080c1552d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f07f800c420 tx=0x7f07f800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0808c00640 1 -- 172.21.15.40:0/3681517016 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f80074e0 con 0x7f080c06ea00 2024-03-24T00:33:27.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0808c00640 1 -- 172.21.15.40:0/3681517016 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f8007680 con 0x7f080c06ea00 2024-03-24T00:33:27.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0808c00640 1 -- 172.21.15.40:0/3681517016 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f8004a30 con 0x7f080c06ea00 2024-03-24T00:33:27.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.499+0000 7f0813800640 1 -- 172.21.15.40:0/3681517016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080c15b950 con 0x7f080c06ea00 2024-03-24T00:33:27.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.503+0000 7f0813800640 1 -- 172.21.15.40:0/3681517016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c15bf10 con 0x7f080c06ea00 2024-03-24T00:33:27.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.503+0000 7f0813800640 1 -- 172.21.15.40:0/3681517016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080c1f8380 con 0x7f080c06ea00 2024-03-24T00:33:27.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.503+0000 7f0808c00640 1 -- 172.21.15.40:0/3681517016 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f07f801a020 con 0x7f080c06ea00 2024-03-24T00:33:27.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.503+0000 7f0808c00640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07ec067660 0x7f07ec069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:27.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.503+0000 7f0808c00640 1 -- 172.21.15.40:0/3681517016 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07f8084f20 con 0x7f080c06ea00 2024-03-24T00:33:27.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.503+0000 7f0810e00640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07ec067660 0x7f07ec069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:27.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.507+0000 7f0810e00640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07ec067660 0x7f07ec069b20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f08000ad000 tx=0x7f08000a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:27.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.507+0000 7f0808c00640 1 -- 172.21.15.40:0/3681517016 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f07f80555e0 con 0x7f080c06ea00 2024-03-24T00:33:27.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.875+0000 7f0813800640 1 -- 172.21.15.40:0/3681517016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "3:1"} v 0) v1 -- 0x7f080c068170 con 0x7f080c06ea00 2024-03-24T00:33:27.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.879+0000 7f0808c00640 1 -- 172.21.15.40:0/3681517016 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "3:1"}]=0 failed mds gid 5626 v37) v1 ==== 97+0+0 (secure 0 0 0) 0x7f07f8059680 con 0x7f080c06ea00 2024-03-24T00:33:27.881 INFO:teuthology.orchestra.run.smithi040.stderr:failed mds gid 5626 2024-03-24T00:33:27.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.879+0000 7f0813800640 1 -- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07ec067660 msgr2=0x7f07ec069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.879+0000 7f0813800640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07ec067660 0x7f07ec069b20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f08000ad000 tx=0x7f08000a6000 comp rx=0 tx=0).stop 2024-03-24T00:33:27.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.879+0000 7f0813800640 1 -- 172.21.15.40:0/3681517016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c06ea00 msgr2=0x7f080c1552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:27.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.879+0000 7f0813800640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c06ea00 0x7f080c1552d0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f07f800c420 tx=0x7f07f800c8e0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.883+0000 7f0813800640 1 -- 172.21.15.40:0/3681517016 shutdown_connections 2024-03-24T00:33:27.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.883+0000 7f0813800640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07ec067660 0x7f07ec069b20 secure :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f08000ad000 tx=0x7f08000a6000 comp rx=0 tx=0).stop 2024-03-24T00:33:27.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.883+0000 7f0813800640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f080c0ccfa0 0x7f080c0aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.883+0000 7f0813800640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c06ea00 0x7f080c1552d0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.883+0000 7f0813800640 1 --2- 172.21.15.40:0/3681517016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c06e620 0x7f080c1f7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:27.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.883+0000 7f0813800640 1 -- 172.21.15.40:0/3681517016 >> 172.21.15.40:0/3681517016 conn(0x7f080c0b17b0 msgr2=0x7f080c0b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:27.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.883+0000 7f0813800640 1 -- 172.21.15.40:0/3681517016 shutdown_connections 2024-03-24T00:33:27.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:27.883+0000 7f0813800640 1 -- 172.21.15.40:0/3681517016 wait complete. 2024-03-24T00:33:27.912 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:27.951 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:33:28.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa09000640 1 Processor -- start 2024-03-24T00:33:28.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa09000640 1 -- start start 2024-03-24T00:33:28.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa09000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffa04145760 0x7ffa0406f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:28.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa09000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffa04072750 0x7ffa0406e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:28.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa09000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa04072bc0 con 0x7ffa0406ea50 2024-03-24T00:33:28.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa09000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa040cd630 con 0x7ffa04145760 2024-03-24T00:33:28.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa09000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa040cd910 con 0x7ffa04072750 2024-03-24T00:33:28.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa01e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffa04145760 0x7ffa0406f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:28.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ff9fbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffa04072750 0x7ffa0406e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:28.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa01e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffa04145760 0x7ffa0406f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44552/0 (socket says 172.21.15.40:44552) 2024-03-24T00:33:28.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa01e00640 1 -- 172.21.15.40:0/1246362925 learned_addr learned my addr 172.21.15.40:0/1246362925 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:28.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa01e00640 1 -- 172.21.15.40:0/1246362925 >> v2:172.21.15.53:3300/0 conn(0x7ffa04072750 msgr2=0x7ffa0406e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:28.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa01e00640 1 --2- 172.21.15.40:0/1246362925 >> v2:172.21.15.53:3300/0 conn(0x7ffa04072750 0x7ffa0406e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:28.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa01e00640 1 -- 172.21.15.40:0/1246362925 >> v1:172.21.15.53:6789/0 conn(0x7ffa0406ea50 legacy=0x7ffa040cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:28.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ffa01e00640 1 -- 172.21.15.40:0/1246362925 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa040cdbf0 con 0x7ffa04145760 2024-03-24T00:33:28.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.051+0000 7ff9fbe00640 1 --2- 172.21.15.40:0/1246362925 >> v2:172.21.15.53:3300/0 conn(0x7ffa04072750 0x7ffa0406e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:28.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.055+0000 7ffa01e00640 1 --2- 172.21.15.40:0/1246362925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa04145760 0x7ffa0406f580 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ffa04071c40 tx=0x7ff9f401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e83ddd3eca354e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:28.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.055+0000 7ffa01400640 1 -- 172.21.15.40:0/1246362925 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f40ab430 con 0x7ffa04145760 2024-03-24T00:33:28.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.055+0000 7ffa01400640 1 -- 172.21.15.40:0/1246362925 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f40b0070 con 0x7ffa04145760 2024-03-24T00:33:28.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.055+0000 7ffa01400640 1 -- 172.21.15.40:0/1246362925 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f40ab7f0 con 0x7ffa04145760 2024-03-24T00:33:28.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.055+0000 7ffa09000640 1 -- 172.21.15.40:0/1246362925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa04145760 msgr2=0x7ffa0406f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:28.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.055+0000 7ffa09000640 1 --2- 172.21.15.40:0/1246362925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa04145760 0x7ffa0406f580 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ffa04071c40 tx=0x7ff9f401c520 comp rx=0 tx=0).stop 2024-03-24T00:33:28.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.055+0000 7ffa09000640 1 -- 172.21.15.40:0/1246362925 shutdown_connections 2024-03-24T00:33:28.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.055+0000 7ffa09000640 1 --2- 172.21.15.40:0/1246362925 >> v2:172.21.15.53:3300/0 conn(0x7ffa04072750 0x7ffa0406e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:28.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.055+0000 7ffa09000640 1 --2- 172.21.15.40:0/1246362925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa04145760 0x7ffa0406f580 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:28.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.055+0000 7ffa09000640 1 -- 172.21.15.40:0/1246362925 >> 172.21.15.40:0/1246362925 conn(0x7ffa040d0c20 msgr2=0x7ffa040d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:28.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.055+0000 7ffa09000640 1 -- 172.21.15.40:0/1246362925 shutdown_connections 2024-03-24T00:33:28.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.055+0000 7ffa09000640 1 -- 172.21.15.40:0/1246362925 wait complete. 2024-03-24T00:33:28.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.059+0000 7ffa09000640 1 Processor -- start 2024-03-24T00:33:28.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.059+0000 7ffa09000640 1 -- start start 2024-03-24T00:33:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.059+0000 7ffa09000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa0406ea50 0x7ffa0413b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.059+0000 7ffa09000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa04072750 0x7ffa041642a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ffa09000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa04145760 0x7ffa04140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ffa09000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa040cfa90 con 0x7ffa04145760 2024-03-24T00:33:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ffa09000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa040cf910 con 0x7ffa04072750 2024-03-24T00:33:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ffa09000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa04168a60 con 0x7ffa0406ea50 2024-03-24T00:33:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ffa01e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa0406ea50 0x7ffa0413b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ff9fbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa04072750 0x7ffa041642a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ffa01e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa0406ea50 0x7ffa0413b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:34690/0 (socket says 172.21.15.40:34690) 2024-03-24T00:33:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ffa01e00640 1 -- 172.21.15.40:0/4162026309 learned_addr learned my addr 172.21.15.40:0/4162026309 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ff9fbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa04072750 0x7ffa041642a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:42530/0 (socket says 172.21.15.40:42530) 2024-03-24T00:33:28.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ffa02800640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa04145760 0x7ffa04140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:28.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ff9fbe00640 1 -- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa0406ea50 msgr2=0x7ffa0413b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:28.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ff9fbe00640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa0406ea50 0x7ffa0413b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:28.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ff9fbe00640 1 -- 172.21.15.40:0/4162026309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa04145760 msgr2=0x7ffa04140060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:28.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ff9fbe00640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa04145760 0x7ffa04140060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:28.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ff9fbe00640 1 -- 172.21.15.40:0/4162026309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa04164cd0 con 0x7ffa04072750 2024-03-24T00:33:28.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ffa02800640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa04145760 0x7ffa04140060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ff9fbe00640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa04072750 0x7ffa041642a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00e9b0 tx=0x7ff9ec00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ff9fa000640 1 -- 172.21.15.40:0/4162026309 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec017070 con 0x7ffa04072750 2024-03-24T00:33:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ff9fa000640 1 -- 172.21.15.40:0/4162026309 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ec00c9a0 con 0x7ffa04072750 2024-03-24T00:33:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ffa09000640 1 -- 172.21.15.40:0/4162026309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa04164fb0 con 0x7ffa04072750 2024-03-24T00:33:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ff9fa000640 1 -- 172.21.15.40:0/4162026309 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec00cd90 con 0x7ffa04072750 2024-03-24T00:33:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ffa09000640 1 -- 172.21.15.40:0/4162026309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa04165570 con 0x7ffa04072750 2024-03-24T00:33:28.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ffa09000640 1 -- 172.21.15.40:0/4162026309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa04068170 con 0x7ffa04072750 2024-03-24T00:33:28.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.063+0000 7ff9fa000640 1 -- 172.21.15.40:0/4162026309 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff9ec01a070 con 0x7ffa04072750 2024-03-24T00:33:28.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.067+0000 7ff9fa000640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff9e0067660 0x7ff9e0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:28.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.067+0000 7ff9fa000640 1 -- 172.21.15.40:0/4162026309 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff9ec085be0 con 0x7ffa04072750 2024-03-24T00:33:28.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.067+0000 7ffa01e00640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff9e0067660 0x7ff9e0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:28.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.067+0000 7ffa01e00640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff9e0067660 0x7ff9e0069b20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff9f401c560 tx=0x7ff9f40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:28.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.071+0000 7ff9fa000640 1 -- 172.21.15.40:0/4162026309 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff9ec056080 con 0x7ffa04072750 2024-03-24T00:33:28.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.267+0000 7ffa09000640 1 -- 172.21.15.40:0/4162026309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa040cf3f0 con 0x7ffa04072750 2024-03-24T00:33:28.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.267+0000 7ff9fa000640 1 -- 172.21.15.40:0/4162026309 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+4221 (secure 0 0 0) 0x7ff9ec05a120 con 0x7ffa04072750 2024-03-24T00:33:28.271 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:28.271 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":38,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:27.884567+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5763,"mds_1":5192,"mds_2":5198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"c","rank":1,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5198":{"gid":5198,"name":"b","rank":2,"incarnation":34,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5763,5198]},"id":3}]} 2024-03-24T00:33:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 38 2024-03-24T00:33:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.271+0000 7ffa09000640 1 -- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff9e0067660 msgr2=0x7ff9e0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.271+0000 7ffa09000640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff9e0067660 0x7ff9e0069b20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff9f401c560 tx=0x7ff9f40a7000 comp rx=0 tx=0).stop 2024-03-24T00:33:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.271+0000 7ffa09000640 1 -- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa04072750 msgr2=0x7ffa041642a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.271+0000 7ffa09000640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa04072750 0x7ffa041642a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec00e9b0 tx=0x7ff9ec00ee70 comp rx=0 tx=0).stop 2024-03-24T00:33:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.271+0000 7ffa09000640 1 -- 172.21.15.40:0/4162026309 shutdown_connections 2024-03-24T00:33:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.271+0000 7ffa09000640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff9e0067660 0x7ff9e0069b20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.271+0000 7ffa09000640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa04145760 0x7ffa04140060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.271+0000 7ffa09000640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa04072750 0x7ffa041642a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.271+0000 7ffa09000640 1 --2- 172.21.15.40:0/4162026309 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa0406ea50 0x7ffa0413b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.271+0000 7ffa09000640 1 -- 172.21.15.40:0/4162026309 >> 172.21.15.40:0/4162026309 conn(0x7ffa040d0c20 msgr2=0x7ffa04144a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.271+0000 7ffa09000640 1 -- 172.21.15.40:0/4162026309 shutdown_connections 2024-03-24T00:33:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:28.271+0000 7ffa09000640 1 -- 172.21.15.40:0/4162026309 wait complete. 2024-03-24T00:33:28.694 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 38, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:32:57.024524+0000', 'modified': '2024-03-24T00:33:27.884567+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 45, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 5763, 'mds_1': 5192, 'mds_2': 5198}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5192': {'gid': 5192, 'name': 'c', 'rank': 1, 'incarnation': 38, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.40:6835/2128553323', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 2128553323}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 2128553323}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5198': {'gid': 5198, 'name': 'b', 'rank': 2, 'incarnation': 34, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.53:6837/1453842342', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6835', 'nonce': 1453842342}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 1453842342}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5763': {'gid': 5763, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6834/224223662', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 224223662}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 224223662}]}, 'join_fscid': -1, 'export_targets': [1, 2], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5763, 'qdb_cluster': [5763, 5198]} 2024-03-24T00:33:28.694 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5192:up:replay 2024-03-24T00:33:29.695 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.831+0000 7f7a3c600640 1 Processor -- start 2024-03-24T00:33:29.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.831+0000 7f7a3c600640 1 -- start start 2024-03-24T00:33:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a3c600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a340b2490 0x7f7a340cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a3c600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a340d0a80 0x7f7a340cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:29.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a3c600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a340b2870 con 0x7f7a340b2aa0 2024-03-24T00:33:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a3c600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a3406b6d0 con 0x7f7a340d0a80 2024-03-24T00:33:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a3c600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340b3190 con 0x7f7a340b2490 2024-03-24T00:33:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a3a600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a340d0a80 0x7f7a340cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a39200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7a340b2aa0 0x7f7a340d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41510/0 (socket says 172.21.15.40:41510) 2024-03-24T00:33:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a39200640 1 -- 172.21.15.40:0/4095461548 learned_addr learned my addr 172.21.15.40:0/4095461548 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a39c00640 1 --2- 172.21.15.40:0/4095461548 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a340b2490 0x7f7a340cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a3a600640 1 -- 172.21.15.40:0/4095461548 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a340b2490 msgr2=0x7f7a340cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:29.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a3a600640 1 --2- 172.21.15.40:0/4095461548 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a340b2490 0x7f7a340cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:29.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a3a600640 1 -- 172.21.15.40:0/4095461548 >> v1:172.21.15.40:6789/0 conn(0x7f7a340b2aa0 legacy=0x7f7a340d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:29.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a33e00640 1 -- 172.21.15.40:0/4095461548 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257997892 0 0) 0x7f7a340b2870 con 0x7f7a340b2aa0 2024-03-24T00:33:29.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.835+0000 7f7a3a600640 1 -- 172.21.15.40:0/4095461548 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a340b3470 con 0x7f7a340d0a80 2024-03-24T00:33:29.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.839+0000 7f7a3a600640 1 --2- 172.21.15.40:0/4095461548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a340d0a80 0x7f7a340cc460 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c009d90 tx=0x7f7a2c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2c985060d5ea68 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.839+0000 7f7a33e00640 1 -- 172.21.15.40:0/4095461548 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0b2070 con 0x7f7a340d0a80 2024-03-24T00:33:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.839+0000 7f7a33e00640 1 -- 172.21.15.40:0/4095461548 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c0a5940 con 0x7f7a340d0a80 2024-03-24T00:33:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.839+0000 7f7a33e00640 1 -- 172.21.15.40:0/4095461548 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0a5d10 con 0x7f7a340d0a80 2024-03-24T00:33:29.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.839+0000 7f7a3c600640 1 -- 172.21.15.40:0/4095461548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a340d0a80 msgr2=0x7f7a340cc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:29.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.839+0000 7f7a3c600640 1 --2- 172.21.15.40:0/4095461548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a340d0a80 0x7f7a340cc460 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c009d90 tx=0x7f7a2c01c970 comp rx=0 tx=0).stop 2024-03-24T00:33:29.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.839+0000 7f7a3c600640 1 -- 172.21.15.40:0/4095461548 shutdown_connections 2024-03-24T00:33:29.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.839+0000 7f7a3c600640 1 --2- 172.21.15.40:0/4095461548 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a340d0a80 0x7f7a340cc460 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:29.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.839+0000 7f7a3c600640 1 --2- 172.21.15.40:0/4095461548 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a340b2490 0x7f7a340cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:29.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.839+0000 7f7a3c600640 1 -- 172.21.15.40:0/4095461548 >> 172.21.15.40:0/4095461548 conn(0x7f7a340d1d10 msgr2=0x7f7a340d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:29.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.839+0000 7f7a3c600640 1 -- 172.21.15.40:0/4095461548 shutdown_connections 2024-03-24T00:33:29.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.839+0000 7f7a3c600640 1 -- 172.21.15.40:0/4095461548 wait complete. 2024-03-24T00:33:29.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.843+0000 7f7a3c600640 1 Processor -- start 2024-03-24T00:33:29.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.843+0000 7f7a3c600640 1 -- start start 2024-03-24T00:33:29.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.843+0000 7f7a3c600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a340b2490 0x7f7a340cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:29.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.843+0000 7f7a3c600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a340b2aa0 0x7f7a340cb860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:29.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.843+0000 7f7a3c600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a340d0a80 0x7f7a340b1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:29.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.843+0000 7f7a3c600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340cd540 con 0x7f7a340b2aa0 2024-03-24T00:33:29.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.843+0000 7f7a3c600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340cd3c0 con 0x7f7a340d0a80 2024-03-24T00:33:29.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.843+0000 7f7a3c600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a340cd6c0 con 0x7f7a340b2490 2024-03-24T00:33:29.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.843+0000 7f7a39200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a340b2aa0 0x7f7a340cb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:29.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a39200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a340b2aa0 0x7f7a340cb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44578/0 (socket says 172.21.15.40:44578) 2024-03-24T00:33:29.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a39200640 1 -- 172.21.15.40:0/2678113064 learned_addr learned my addr 172.21.15.40:0/2678113064 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:29.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a39c00640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a340b2490 0x7f7a340cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:29.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a3a600640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a340d0a80 0x7f7a340b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:29.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a39200640 1 -- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a340b2490 msgr2=0x7f7a340cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:29.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a39200640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a340b2490 0x7f7a340cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:29.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a39200640 1 -- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a340d0a80 msgr2=0x7f7a340b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:29.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a39200640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a340d0a80 0x7f7a340b1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:29.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a39200640 1 -- 172.21.15.40:0/2678113064 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a340cbda0 con 0x7f7a340b2aa0 2024-03-24T00:33:29.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a39c00640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a340b2490 0x7f7a340cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:29.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a39200640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a340b2aa0 0x7f7a340cb860 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f7a200011f0 tx=0x7f7a2000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:29.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a31600640 1 -- 172.21.15.40:0/2678113064 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a20018070 con 0x7f7a340b2aa0 2024-03-24T00:33:29.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a31600640 1 -- 172.21.15.40:0/2678113064 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a200074e0 con 0x7f7a340b2aa0 2024-03-24T00:33:29.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a3c600640 1 -- 172.21.15.40:0/2678113064 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a34159a40 con 0x7f7a340b2aa0 2024-03-24T00:33:29.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a31600640 1 -- 172.21.15.40:0/2678113064 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a20004a10 con 0x7f7a340b2aa0 2024-03-24T00:33:29.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a3c600640 1 -- 172.21.15.40:0/2678113064 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a340c90c0 con 0x7f7a340b2aa0 2024-03-24T00:33:29.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a3c600640 1 -- 172.21.15.40:0/2678113064 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a34068170 con 0x7f7a340b2aa0 2024-03-24T00:33:29.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.847+0000 7f7a31600640 1 -- 172.21.15.40:0/2678113064 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7a2001b070 con 0x7f7a340b2aa0 2024-03-24T00:33:29.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.851+0000 7f7a31600640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a14067660 0x7f7a14069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:29.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.851+0000 7f7a31600640 1 -- 172.21.15.40:0/2678113064 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7a20084ca0 con 0x7f7a340b2aa0 2024-03-24T00:33:29.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.851+0000 7f7a39c00640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a14067660 0x7f7a14069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:29.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.851+0000 7f7a39c00640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a14067660 0x7f7a14069b20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7a340ca4f0 tx=0x7f7a28009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:29.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:29.855+0000 7f7a31600640 1 -- 172.21.15.40:0/2678113064 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7a200551d0 con 0x7f7a340b2aa0 2024-03-24T00:33:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.051+0000 7f7a3c600640 1 -- 172.21.15.40:0/2678113064 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a3406f290 con 0x7f7a340b2aa0 2024-03-24T00:33:30.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a31600640 1 -- 172.21.15.40:0/2678113064 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5012 (secure 0 0 0) 0x7f7a20059270 con 0x7f7a340b2aa0 2024-03-24T00:33:30.057 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:30.057 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/374985095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":374985095},{"type":"v1","addr":"172.21.15.53:6836","nonce":374985095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:28.894697+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5763,"mds_1":5192,"mds_2":5198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"c","rank":1,"incarnation":38,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5198":{"gid":5198,"name":"b","rank":2,"incarnation":34,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5763,5198]},"id":3}]} 2024-03-24T00:33:30.057 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 40 2024-03-24T00:33:30.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a3c600640 1 -- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a14067660 msgr2=0x7f7a14069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:30.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a3c600640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a14067660 0x7f7a14069b20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7a340ca4f0 tx=0x7f7a28009190 comp rx=0 tx=0).stop 2024-03-24T00:33:30.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a3c600640 1 -- 172.21.15.40:0/2678113064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a340b2aa0 msgr2=0x7f7a340cb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:30.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a3c600640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a340b2aa0 0x7f7a340cb860 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f7a200011f0 tx=0x7f7a2000c9b0 comp rx=0 tx=0).stop 2024-03-24T00:33:30.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a3c600640 1 -- 172.21.15.40:0/2678113064 shutdown_connections 2024-03-24T00:33:30.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a3c600640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a14067660 0x7f7a14069b20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:30.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a3c600640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a340d0a80 0x7f7a340b1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:30.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a3c600640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a340b2aa0 0x7f7a340cb860 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:30.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a3c600640 1 --2- 172.21.15.40:0/2678113064 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a340b2490 0x7f7a340cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:30.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a3c600640 1 -- 172.21.15.40:0/2678113064 >> 172.21.15.40:0/2678113064 conn(0x7f7a340d1d10 msgr2=0x7f7a340cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:30.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a3c600640 1 -- 172.21.15.40:0/2678113064 shutdown_connections 2024-03-24T00:33:30.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:30.055+0000 7f7a3c600640 1 -- 172.21.15.40:0/2678113064 wait complete. 2024-03-24T00:33:30.483 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:32:57.024524+0000', 'modified': '2024-03-24T00:33:28.894697+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 45, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 5763, 'mds_1': 5192, 'mds_2': 5198}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5192': {'gid': 5192, 'name': 'c', 'rank': 1, 'incarnation': 38, 'state': 'up:reconnect', 'state_seq': 11, 'addr': '172.21.15.40:6835/2128553323', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 2128553323}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 2128553323}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5198': {'gid': 5198, 'name': 'b', 'rank': 2, 'incarnation': 34, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.53:6837/1453842342', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6835', 'nonce': 1453842342}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 1453842342}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5763': {'gid': 5763, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6834/224223662', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 224223662}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 224223662}]}, 'join_fscid': -1, 'export_targets': [1, 2], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5763, 'qdb_cluster': [5763, 5198]} 2024-03-24T00:33:30.483 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5192:up:reconnect 2024-03-24T00:33:31.484 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:31.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe52200640 1 Processor -- start 2024-03-24T00:33:31.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe52200640 1 -- start start 2024-03-24T00:33:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe52200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efe4c13ec80 0x7efe4c1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe52200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe4c1419f0 0x7efe4c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe52200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c070e00 con 0x7efe4c0ce630 2024-03-24T00:33:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe52200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe4c141e50 con 0x7efe4c1419f0 2024-03-24T00:33:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe52200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe4c0cee20 con 0x7efe4c13ec80 2024-03-24T00:33:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe42a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe4c1419f0 0x7efe4c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe4be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efe4c0ce630 0x7efe4c15f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41522/0 (socket says 172.21.15.40:41522) 2024-03-24T00:33:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe4be00640 1 -- 172.21.15.40:0/873885874 learned_addr learned my addr 172.21.15.40:0/873885874 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe4b400640 1 --2- 172.21.15.40:0/873885874 >> v2:172.21.15.53:3300/0 conn(0x7efe4c13ec80 0x7efe4c1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe4aa00640 1 -- 172.21.15.40:0/873885874 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299623159 0 0) 0x7efe4c070e00 con 0x7efe4c0ce630 2024-03-24T00:33:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe4b400640 1 -- 172.21.15.40:0/873885874 >> v2:172.21.15.40:3300/0 conn(0x7efe4c1419f0 msgr2=0x7efe4c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe4b400640 1 --2- 172.21.15.40:0/873885874 >> v2:172.21.15.40:3300/0 conn(0x7efe4c1419f0 0x7efe4c0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe4b400640 1 -- 172.21.15.40:0/873885874 >> v1:172.21.15.40:6789/0 conn(0x7efe4c0ce630 legacy=0x7efe4c15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe4b400640 1 -- 172.21.15.40:0/873885874 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c0cf100 con 0x7efe4c13ec80 2024-03-24T00:33:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.623+0000 7efe42a00640 1 --2- 172.21.15.40:0/873885874 >> v2:172.21.15.40:3300/0 conn(0x7efe4c1419f0 0x7efe4c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:31.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.627+0000 7efe4b400640 1 --2- 172.21.15.40:0/873885874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe4c13ec80 0x7efe4c1414b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7efe4c06b170 tx=0x7efe3c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2ea5a3203e96359 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:31.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.627+0000 7efe4aa00640 1 -- 172.21.15.40:0/873885874 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c0aa600 con 0x7efe4c13ec80 2024-03-24T00:33:31.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.627+0000 7efe4aa00640 1 -- 172.21.15.40:0/873885874 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3c0af070 con 0x7efe4c13ec80 2024-03-24T00:33:31.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.627+0000 7efe4aa00640 1 -- 172.21.15.40:0/873885874 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c0aa9c0 con 0x7efe4c13ec80 2024-03-24T00:33:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.627+0000 7efe52200640 1 -- 172.21.15.40:0/873885874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe4c13ec80 msgr2=0x7efe4c1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.627+0000 7efe52200640 1 --2- 172.21.15.40:0/873885874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe4c13ec80 0x7efe4c1414b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7efe4c06b170 tx=0x7efe3c01c480 comp rx=0 tx=0).stop 2024-03-24T00:33:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.627+0000 7efe52200640 1 -- 172.21.15.40:0/873885874 shutdown_connections 2024-03-24T00:33:31.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.627+0000 7efe52200640 1 --2- 172.21.15.40:0/873885874 >> v2:172.21.15.40:3300/0 conn(0x7efe4c1419f0 0x7efe4c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:31.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.627+0000 7efe52200640 1 --2- 172.21.15.40:0/873885874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe4c13ec80 0x7efe4c1414b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:31.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.627+0000 7efe52200640 1 -- 172.21.15.40:0/873885874 >> 172.21.15.40:0/873885874 conn(0x7efe4c0b17b0 msgr2=0x7efe4c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:31.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.627+0000 7efe52200640 1 -- 172.21.15.40:0/873885874 shutdown_connections 2024-03-24T00:33:31.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.627+0000 7efe52200640 1 -- 172.21.15.40:0/873885874 wait complete. 2024-03-24T00:33:31.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe52200640 1 Processor -- start 2024-03-24T00:33:31.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe52200640 1 -- start start 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe52200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe4c0ce630 0x7efe4c13d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe52200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe4c13ec80 0x7efe4c154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe52200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe4c1419f0 0x7efe4c158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe52200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c168ec0 con 0x7efe4c1419f0 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe52200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c168d40 con 0x7efe4c13ec80 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe52200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c169040 con 0x7efe4c0ce630 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe4b400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe4c0ce630 0x7efe4c13d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe42a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe4c13ec80 0x7efe4c154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe4be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe4c1419f0 0x7efe4c158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe4b400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe4c0ce630 0x7efe4c13d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:34724/0 (socket says 172.21.15.40:34724) 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe42a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe4c13ec80 0x7efe4c154880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:42540/0 (socket says 172.21.15.40:42540) 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe4b400640 1 -- 172.21.15.40:0/2257817874 learned_addr learned my addr 172.21.15.40:0/2257817874 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:31.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe42a00640 1 -- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe4c0ce630 msgr2=0x7efe4c13d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe42a00640 1 --2- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe4c0ce630 0x7efe4c13d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe42a00640 1 -- 172.21.15.40:0/2257817874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe4c1419f0 msgr2=0x7efe4c158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe42a00640 1 --2- 172.21.15.40:0/2257817874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe4c1419f0 0x7efe4c158de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe42a00640 1 -- 172.21.15.40:0/2257817874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c13e080 con 0x7efe4c13ec80 2024-03-24T00:33:31.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe42a00640 1 --2- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe4c13ec80 0x7efe4c154880 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7efe3400e8e0 tx=0x7efe3400eda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe43e00640 1 -- 172.21.15.40:0/2257817874 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe34017070 con 0x7efe4c13ec80 2024-03-24T00:33:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe43e00640 1 -- 172.21.15.40:0/2257817874 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3400c8e0 con 0x7efe4c13ec80 2024-03-24T00:33:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe43e00640 1 -- 172.21.15.40:0/2257817874 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3400ccd0 con 0x7efe4c13ec80 2024-03-24T00:33:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe52200640 1 -- 172.21.15.40:0/2257817874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe4c13d610 con 0x7efe4c13ec80 2024-03-24T00:33:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.631+0000 7efe52200640 1 -- 172.21.15.40:0/2257817874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4c153970 con 0x7efe4c13ec80 2024-03-24T00:33:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.635+0000 7efe52200640 1 -- 172.21.15.40:0/2257817874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4c068170 con 0x7efe4c13ec80 2024-03-24T00:33:31.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.635+0000 7efe43e00640 1 -- 172.21.15.40:0/2257817874 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7efe3401a070 con 0x7efe4c13ec80 2024-03-24T00:33:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.635+0000 7efe43e00640 1 --2- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe28067660 0x7efe28069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.635+0000 7efe43e00640 1 -- 172.21.15.40:0/2257817874 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efe34085730 con 0x7efe4c13ec80 2024-03-24T00:33:31.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.635+0000 7efe4b400640 1 --2- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe28067660 0x7efe28069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:31.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.635+0000 7efe4b400640 1 --2- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe28067660 0x7efe28069b20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efe3c01a0b0 tx=0x7efe3c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:31.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.643+0000 7efe43e00640 1 -- 172.21.15.40:0/2257817874 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7efe34055cf0 con 0x7efe4c13ec80 2024-03-24T00:33:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.839+0000 7efe52200640 1 -- 172.21.15.40:0/2257817874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe4c06cc80 con 0x7efe4c13ec80 2024-03-24T00:33:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.839+0000 7efe43e00640 1 -- 172.21.15.40:0/2257817874 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5009 (secure 0 0 0) 0x7efe34059d90 con 0x7efe4c13ec80 2024-03-24T00:33:31.844 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:31.844 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/374985095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":374985095},{"type":"v1","addr":"172.21.15.53:6836","nonce":374985095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:29.903030+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5763,"mds_1":5192,"mds_2":5198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"c","rank":1,"incarnation":38,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5198":{"gid":5198,"name":"b","rank":2,"incarnation":34,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5198,5763]},"id":3}]} 2024-03-24T00:33:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 41 2024-03-24T00:33:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.843+0000 7efe52200640 1 -- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe28067660 msgr2=0x7efe28069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.843+0000 7efe52200640 1 --2- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe28067660 0x7efe28069b20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7efe3c01a0b0 tx=0x7efe3c0a7000 comp rx=0 tx=0).stop 2024-03-24T00:33:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.843+0000 7efe52200640 1 -- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe4c13ec80 msgr2=0x7efe4c154880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.843+0000 7efe52200640 1 --2- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe4c13ec80 0x7efe4c154880 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7efe3400e8e0 tx=0x7efe3400eda0 comp rx=0 tx=0).stop 2024-03-24T00:33:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.843+0000 7efe52200640 1 -- 172.21.15.40:0/2257817874 shutdown_connections 2024-03-24T00:33:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.843+0000 7efe52200640 1 --2- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe28067660 0x7efe28069b20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.843+0000 7efe52200640 1 --2- 172.21.15.40:0/2257817874 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe4c1419f0 0x7efe4c158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.843+0000 7efe52200640 1 --2- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe4c13ec80 0x7efe4c154880 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.843+0000 7efe52200640 1 --2- 172.21.15.40:0/2257817874 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe4c0ce630 0x7efe4c13d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.843+0000 7efe52200640 1 -- 172.21.15.40:0/2257817874 >> 172.21.15.40:0/2257817874 conn(0x7efe4c0b17b0 msgr2=0x7efe4c155840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.843+0000 7efe52200640 1 -- 172.21.15.40:0/2257817874 shutdown_connections 2024-03-24T00:33:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:31.843+0000 7efe52200640 1 -- 172.21.15.40:0/2257817874 wait complete. 2024-03-24T00:33:32.267 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:32:57.024524+0000', 'modified': '2024-03-24T00:33:29.903030+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 45, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 5763, 'mds_1': 5192, 'mds_2': 5198}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5192': {'gid': 5192, 'name': 'c', 'rank': 1, 'incarnation': 38, 'state': 'up:rejoin', 'state_seq': 12, 'addr': '172.21.15.40:6835/2128553323', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 2128553323}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 2128553323}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5198': {'gid': 5198, 'name': 'b', 'rank': 2, 'incarnation': 34, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.53:6837/1453842342', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6835', 'nonce': 1453842342}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 1453842342}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5763': {'gid': 5763, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6834/224223662', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 224223662}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 224223662}]}, 'join_fscid': -1, 'export_targets': [1, 2], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5763, 'qdb_cluster': [5198, 5763]} 2024-03-24T00:33:32.267 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5192:up:rejoin 2024-03-24T00:33:33.269 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:33.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.407+0000 7fb37b800640 1 Processor -- start 2024-03-24T00:33:33.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.407+0000 7fb37b800640 1 -- start start 2024-03-24T00:33:33.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb37b800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb37413ec80 0x7fb3741414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:33.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb37b800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3741419f0 0x7fb3740ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:33.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb37b800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb3740ce630 0x7fb37415f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:33.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb37b800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb374141e50 con 0x7fb37413ec80 2024-03-24T00:33:33.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb37b800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb374070e00 con 0x7fb3740ce630 2024-03-24T00:33:33.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb37b800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3740cee20 con 0x7fb3741419f0 2024-03-24T00:33:33.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb378e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb37413ec80 0x7fb3741414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:33.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb378e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb37413ec80 0x7fb3741414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44608/0 (socket says 172.21.15.40:44608) 2024-03-24T00:33:33.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb378e00640 1 -- 172.21.15.40:0/2145686230 learned_addr learned my addr 172.21.15.40:0/2145686230 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:33.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb373e00640 1 --2- 172.21.15.40:0/2145686230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3741419f0 0x7fb3740ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:33.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb379800640 1 --2- 172.21.15.40:0/2145686230 >> v2:172.21.15.53:3300/0 conn(0x7fb3740ce630 0x7fb37415f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:33.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb378e00640 1 -- 172.21.15.40:0/2145686230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3741419f0 msgr2=0x7fb3740ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:33.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb378e00640 1 --2- 172.21.15.40:0/2145686230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3741419f0 0x7fb3740ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:33.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb378e00640 1 -- 172.21.15.40:0/2145686230 >> v2:172.21.15.53:3300/0 conn(0x7fb3740ce630 msgr2=0x7fb37415f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:33.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb378e00640 1 --2- 172.21.15.40:0/2145686230 >> v2:172.21.15.53:3300/0 conn(0x7fb3740ce630 0x7fb37415f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:33.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.411+0000 7fb378e00640 1 -- 172.21.15.40:0/2145686230 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3740cf100 con 0x7fb37413ec80 2024-03-24T00:33:33.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb378e00640 1 --2- 172.21.15.40:0/2145686230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb37413ec80 0x7fb3741414b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb3740b2b60 tx=0x7fb36801c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b1a5f221461e17 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:33.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb373400640 1 -- 172.21.15.40:0/2145686230 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3680a55d0 con 0x7fb37413ec80 2024-03-24T00:33:33.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb373400640 1 -- 172.21.15.40:0/2145686230 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3680a5770 con 0x7fb37413ec80 2024-03-24T00:33:33.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb373400640 1 -- 172.21.15.40:0/2145686230 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3680a5b40 con 0x7fb37413ec80 2024-03-24T00:33:33.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb37b800640 1 -- 172.21.15.40:0/2145686230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb37413ec80 msgr2=0x7fb3741414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:33.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb37b800640 1 --2- 172.21.15.40:0/2145686230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb37413ec80 0x7fb3741414b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb3740b2b60 tx=0x7fb36801c730 comp rx=0 tx=0).stop 2024-03-24T00:33:33.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb37b800640 1 -- 172.21.15.40:0/2145686230 shutdown_connections 2024-03-24T00:33:33.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb37b800640 1 --2- 172.21.15.40:0/2145686230 >> v2:172.21.15.53:3300/0 conn(0x7fb3740ce630 0x7fb37415f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:33.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb37b800640 1 --2- 172.21.15.40:0/2145686230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3741419f0 0x7fb3740ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:33.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb37b800640 1 --2- 172.21.15.40:0/2145686230 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb37413ec80 0x7fb3741414b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:33.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb37b800640 1 -- 172.21.15.40:0/2145686230 >> 172.21.15.40:0/2145686230 conn(0x7fb3740b17b0 msgr2=0x7fb3740b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:33.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb37b800640 1 -- 172.21.15.40:0/2145686230 shutdown_connections 2024-03-24T00:33:33.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb37b800640 1 -- 172.21.15.40:0/2145686230 wait complete. 2024-03-24T00:33:33.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb37b800640 1 Processor -- start 2024-03-24T00:33:33.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.415+0000 7fb37b800640 1 -- start start 2024-03-24T00:33:33.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb37b800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3740ce630 0x7fb37413fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:33.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb37b800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb37413ec80 0x7fb37413d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:33.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb37b800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3741419f0 0x7fb374140800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:33.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb37b800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374071480 con 0x7fb3741419f0 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb37b800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374071300 con 0x7fb3740ce630 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb37b800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb374071600 con 0x7fb37413ec80 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb378e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3740ce630 0x7fb37413fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb379800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3741419f0 0x7fb374140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb378e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3740ce630 0x7fb37413fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:42570/0 (socket says 172.21.15.40:42570) 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb378e00640 1 -- 172.21.15.40:0/3944793520 learned_addr learned my addr 172.21.15.40:0/3944793520 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb373e00640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb37413ec80 0x7fb37413d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb378e00640 1 -- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb37413ec80 msgr2=0x7fb37413d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb378e00640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb37413ec80 0x7fb37413d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb378e00640 1 -- 172.21.15.40:0/3944793520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3741419f0 msgr2=0x7fb374140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb378e00640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3741419f0 0x7fb374140800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb378e00640 1 -- 172.21.15.40:0/3944793520 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37413d8a0 con 0x7fb3740ce630 2024-03-24T00:33:33.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb379800640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3741419f0 0x7fb374140800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:33.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb378e00640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3740ce630 0x7fb37413fd50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb3680af040 tx=0x7fb3680194d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:33.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb370c00640 1 -- 172.21.15.40:0/3944793520 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3680bc020 con 0x7fb3740ce630 2024-03-24T00:33:33.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb370c00640 1 -- 172.21.15.40:0/3944793520 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3680b8690 con 0x7fb3740ce630 2024-03-24T00:33:33.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb37b800640 1 -- 172.21.15.40:0/3944793520 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37415d850 con 0x7fb3740ce630 2024-03-24T00:33:33.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb370c00640 1 -- 172.21.15.40:0/3944793520 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368009510 con 0x7fb3740ce630 2024-03-24T00:33:33.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb37b800640 1 -- 172.21.15.40:0/3944793520 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb37413f270 con 0x7fb3740ce630 2024-03-24T00:33:33.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.419+0000 7fb37b800640 1 -- 172.21.15.40:0/3944793520 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb374068170 con 0x7fb3740ce630 2024-03-24T00:33:33.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.423+0000 7fb370c00640 1 -- 172.21.15.40:0/3944793520 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb3680bf020 con 0x7fb3740ce630 2024-03-24T00:33:33.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.423+0000 7fb370c00640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb354067660 0x7fb354069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:33.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.423+0000 7fb370c00640 1 -- 172.21.15.40:0/3944793520 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb3681203f0 con 0x7fb3740ce630 2024-03-24T00:33:33.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.423+0000 7fb373e00640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb354067660 0x7fb354069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:33.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.423+0000 7fb373e00640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb354067660 0x7fb354069b20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb3741410f0 tx=0x7fb360009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:33.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.431+0000 7fb370c00640 1 -- 172.21.15.40:0/3944793520 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb3680c6030 con 0x7fb3740ce630 2024-03-24T00:33:33.609 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-03-24T00:33:33.607+0000 7f2af2400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:33:33.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.635+0000 7fb37b800640 1 -- 172.21.15.40:0/3944793520 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb374072b90 con 0x7fb3740ce630 2024-03-24T00:33:33.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.635+0000 7fb370c00640 1 -- 172.21.15.40:0/3944793520 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5014 (secure 0 0 0) 0x7fb37413d8a0 con 0x7fb3740ce630 2024-03-24T00:33:33.642 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:33.642 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":42,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/374985095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":374985095},{"type":"v1","addr":"172.21.15.53:6836","nonce":374985095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:31.903823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5763,"mds_1":5192,"mds_2":5198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"c","rank":1,"incarnation":38,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5198":{"gid":5198,"name":"b","rank":2,"incarnation":34,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5192,5198,5763]},"id":3}]} 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 42 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.639+0000 7fb37b800640 1 -- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb354067660 msgr2=0x7fb354069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.639+0000 7fb37b800640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb354067660 0x7fb354069b20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb3741410f0 tx=0x7fb360009190 comp rx=0 tx=0).stop 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.639+0000 7fb37b800640 1 -- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3740ce630 msgr2=0x7fb37413fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.639+0000 7fb37b800640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3740ce630 0x7fb37413fd50 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb3680af040 tx=0x7fb3680194d0 comp rx=0 tx=0).stop 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.639+0000 7fb37b800640 1 -- 172.21.15.40:0/3944793520 shutdown_connections 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.639+0000 7fb37b800640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb354067660 0x7fb354069b20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.639+0000 7fb37b800640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3741419f0 0x7fb374140800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.639+0000 7fb37b800640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb37413ec80 0x7fb37413d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.639+0000 7fb37b800640 1 --2- 172.21.15.40:0/3944793520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3740ce630 0x7fb37413fd50 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.639+0000 7fb37b800640 1 -- 172.21.15.40:0/3944793520 >> 172.21.15.40:0/3944793520 conn(0x7fb3740b17b0 msgr2=0x7fb37406cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.639+0000 7fb37b800640 1 -- 172.21.15.40:0/3944793520 shutdown_connections 2024-03-24T00:33:33.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:33.639+0000 7fb37b800640 1 -- 172.21.15.40:0/3944793520 wait complete. 2024-03-24T00:33:34.067 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 42, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:32:57.024524+0000', 'modified': '2024-03-24T00:33:31.903823+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 45, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 5763, 'mds_1': 5192, 'mds_2': 5198}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5192': {'gid': 5192, 'name': 'c', 'rank': 1, 'incarnation': 38, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.40:6835/2128553323', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 2128553323}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 2128553323}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5198': {'gid': 5198, 'name': 'b', 'rank': 2, 'incarnation': 34, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.53:6837/1453842342', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6835', 'nonce': 1453842342}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 1453842342}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_5763': {'gid': 5763, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6834/224223662', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 224223662}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 224223662}]}, 'join_fscid': -1, 'export_targets': [1, 2], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5763, 'qdb_cluster': [5192, 5198, 5763]} 2024-03-24T00:33:34.067 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T00:33:34.067 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:33:34.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb8200640 1 Processor -- start 2024-03-24T00:33:34.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb8200640 1 -- start start 2024-03-24T00:33:34.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb8200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efcb015dfc0 0x7efcb00cd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb8200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efcb015e3a0 0x7efcb00cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb8200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efcb00d0b40 0x7efcb00ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb8200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb00cf870 con 0x7efcb015e3a0 2024-03-24T00:33:34.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb8200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb00cfaa0 con 0x7efcb015dfc0 2024-03-24T00:33:34.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb8200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb00cfd80 con 0x7efcb00d0b40 2024-03-24T00:33:34.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb4e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efcb015e3a0 0x7efcb00cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb6200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efcb00d0b40 0x7efcb00ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb5800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efcb015dfc0 0x7efcb00cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb4e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efcb015e3a0 0x7efcb00cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44638/0 (socket says 172.21.15.40:44638) 2024-03-24T00:33:34.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb4e00640 1 -- 172.21.15.40:0/1348410995 learned_addr learned my addr 172.21.15.40:0/1348410995 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb4e00640 1 -- 172.21.15.40:0/1348410995 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efcb00d0b40 msgr2=0x7efcb00ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb4e00640 1 --2- 172.21.15.40:0/1348410995 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efcb00d0b40 0x7efcb00ce690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb4e00640 1 -- 172.21.15.40:0/1348410995 >> v2:172.21.15.53:3300/0 conn(0x7efcb015dfc0 msgr2=0x7efcb00cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb4e00640 1 --2- 172.21.15.40:0/1348410995 >> v2:172.21.15.53:3300/0 conn(0x7efcb015dfc0 0x7efcb00cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb4e00640 1 -- 172.21.15.40:0/1348410995 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb00d0060 con 0x7efcb015e3a0 2024-03-24T00:33:34.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.207+0000 7efcb6200640 1 --2- 172.21.15.40:0/1348410995 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efcb00d0b40 0x7efcb00ce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:34.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcb4e00640 1 --2- 172.21.15.40:0/1348410995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb015e3a0 0x7efcb00cd8d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7efcb00720f0 tx=0x7efc9c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277010c4befa120d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcafe00640 1 -- 172.21.15.40:0/1348410995 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c0b0070 con 0x7efcb015e3a0 2024-03-24T00:33:34.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcafe00640 1 -- 172.21.15.40:0/1348410995 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9c0a4e50 con 0x7efcb015e3a0 2024-03-24T00:33:34.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcafe00640 1 -- 172.21.15.40:0/1348410995 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c0ab820 con 0x7efcb015e3a0 2024-03-24T00:33:34.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcb8200640 1 -- 172.21.15.40:0/1348410995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb015e3a0 msgr2=0x7efcb00cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcb8200640 1 --2- 172.21.15.40:0/1348410995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb015e3a0 0x7efcb00cd8d0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7efcb00720f0 tx=0x7efc9c01c400 comp rx=0 tx=0).stop 2024-03-24T00:33:34.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcb8200640 1 -- 172.21.15.40:0/1348410995 shutdown_connections 2024-03-24T00:33:34.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcb8200640 1 --2- 172.21.15.40:0/1348410995 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efcb00d0b40 0x7efcb00ce690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcb8200640 1 --2- 172.21.15.40:0/1348410995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb015e3a0 0x7efcb00cd8d0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcb8200640 1 --2- 172.21.15.40:0/1348410995 >> v2:172.21.15.53:3300/0 conn(0x7efcb015dfc0 0x7efcb00cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcb8200640 1 -- 172.21.15.40:0/1348410995 >> 172.21.15.40:0/1348410995 conn(0x7efcb00b37f0 msgr2=0x7efcb00b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcb8200640 1 -- 172.21.15.40:0/1348410995 shutdown_connections 2024-03-24T00:33:34.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.211+0000 7efcb8200640 1 -- 172.21.15.40:0/1348410995 wait complete. 2024-03-24T00:33:34.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb8200640 1 Processor -- start 2024-03-24T00:33:34.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb8200640 1 -- start start 2024-03-24T00:33:34.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb8200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb00d0b40 0x7efcb014dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb8200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efcb015dfc0 0x7efcb00cb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb8200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efcb015e3a0 0x7efcb015cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb8200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb0162820 con 0x7efcb00d0b40 2024-03-24T00:33:34.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb8200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb01626a0 con 0x7efcb015dfc0 2024-03-24T00:33:34.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb8200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb01629a0 con 0x7efcb015e3a0 2024-03-24T00:33:34.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb00d0b40 0x7efcb014dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb4e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efcb015dfc0 0x7efcb00cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb00d0b40 0x7efcb014dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44642/0 (socket says 172.21.15.40:44642) 2024-03-24T00:33:34.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb5800640 1 -- 172.21.15.40:0/2089445034 learned_addr learned my addr 172.21.15.40:0/2089445034 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb6200640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efcb015e3a0 0x7efcb015cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb5800640 1 -- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efcb015e3a0 msgr2=0x7efcb015cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb5800640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efcb015e3a0 0x7efcb015cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb5800640 1 -- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efcb015dfc0 msgr2=0x7efcb00cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb5800640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efcb015dfc0 0x7efcb00cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb5800640 1 -- 172.21.15.40:0/2089445034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb00b6fa0 con 0x7efcb00d0b40 2024-03-24T00:33:34.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb5800640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb00d0b40 0x7efcb014dfc0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7efca400e870 tx=0x7efca400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcad600640 1 -- 172.21.15.40:0/2089445034 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4017070 con 0x7efcb00d0b40 2024-03-24T00:33:34.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcad600640 1 -- 172.21.15.40:0/2089445034 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca400c730 con 0x7efcb00d0b40 2024-03-24T00:33:34.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcad600640 1 -- 172.21.15.40:0/2089445034 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca400cae0 con 0x7efcb00d0b40 2024-03-24T00:33:34.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb8200640 1 -- 172.21.15.40:0/2089445034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb00b7280 con 0x7efcb00d0b40 2024-03-24T00:33:34.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.215+0000 7efcb8200640 1 -- 172.21.15.40:0/2089445034 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb00b7840 con 0x7efcb00d0b40 2024-03-24T00:33:34.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcad600640 1 -- 172.21.15.40:0/2089445034 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7efca401a070 con 0x7efcb00d0b40 2024-03-24T00:33:34.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcad600640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efc90067660 0x7efc90069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcad600640 1 -- 172.21.15.40:0/2089445034 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efca4085a90 con 0x7efcb00d0b40 2024-03-24T00:33:34.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcb4e00640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efc90067660 0x7efc90069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcbb792000 1 Processor -- start 2024-03-24T00:33:34.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcbb792000 1 -- start start 2024-03-24T00:33:34.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcb4e00640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efc90067660 0x7efc90069b20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0ad000 tx=0x7efc9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcbb792000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 0x5560cecb7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcbb792000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cecb8240 0x5560cec6dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcbb792000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec6dff0 0x5560cec6e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcbb792000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560ceabb680 con 0x5560cecb7920 2024-03-24T00:33:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcbb792000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560cea42b00 con 0x5560cec6dff0 2024-03-24T00:33:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcbb792000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5560ceab7720 con 0x5560cecb8240 2024-03-24T00:33:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcb5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 0x5560cecb7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcb4e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cecb8240 0x5560cec6dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcb5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 0x5560cecb7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44650/0 (socket says 172.21.15.40:44650) 2024-03-24T00:33:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcb5800640 1 -- 172.21.15.40:0/509898309 learned_addr learned my addr 172.21.15.40:0/509898309 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.219+0000 7efcb6200640 1 --2- 172.21.15.40:0/509898309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec6dff0 0x5560cec6e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 -- 172.21.15.40:0/509898309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec6dff0 msgr2=0x5560cec6e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 --2- 172.21.15.40:0/509898309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec6dff0 0x5560cec6e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 -- 172.21.15.40:0/509898309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 msgr2=0x5560cecb7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 --2- 172.21.15.40:0/509898309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 0x5560cecb7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 -- 172.21.15.40:0/509898309 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560cea93940 con 0x5560cecb8240 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb5800640 1 --2- 172.21.15.40:0/509898309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 0x5560cecb7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 --2- 172.21.15.40:0/509898309 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cecb8240 0x5560cec6dab0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0ab000 tx=0x7efc9c0a6000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ab5ec48b0838be server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efca2000640 1 -- 172.21.15.40:0/509898309 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c0c7070 con 0x5560cecb8240 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efca2000640 1 -- 172.21.15.40:0/509898309 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9c0b1040 con 0x5560cecb8240 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efca2000640 1 -- 172.21.15.40:0/509898309 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c0b5400 con 0x5560cecb8240 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 -- 172.21.15.40:0/509898309 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cecb8240 msgr2=0x5560cec6dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 --2- 172.21.15.40:0/509898309 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cecb8240 0x5560cec6dab0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0ab000 tx=0x7efc9c0a6000 comp rx=0 tx=0).stop 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 -- 172.21.15.40:0/509898309 shutdown_connections 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 --2- 172.21.15.40:0/509898309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec6dff0 0x5560cec6e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 --2- 172.21.15.40:0/509898309 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cecb8240 0x5560cec6dab0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 --2- 172.21.15.40:0/509898309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 0x5560cecb7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 -- 172.21.15.40:0/509898309 >> 172.21.15.40:0/509898309 conn(0x5560cec69fa0 msgr2=0x5560cec6c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 -- 172.21.15.40:0/509898309 shutdown_connections 2024-03-24T00:33:34.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 -- 172.21.15.40:0/509898309 wait complete. 2024-03-24T00:33:34.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 -- start start 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 Processor -- start 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560cec9cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 0x5560cec9d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cecb8240 0x5560cec9d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560ceaba990 con 0x5560cecb7920 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560cea75ae0 con 0x5560cecb8240 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5560cea53750 con 0x5560cec6dff0 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcbb792000 10 client.0 _ll_register_callbacks cb 0x5560cec58ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 0x5560cec9d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 0x5560cec9d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44652/0 (socket says 172.21.15.40:44652) 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 -- 172.21.15.40:0/3226796884 learned_addr learned my addr 172.21.15.40:0/3226796884 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb5800640 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560cec9cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb6200640 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cecb8240 0x5560cec9d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 -- 172.21.15.40:0/3226796884 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 msgr2=0x5560cec9cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560cec9cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 -- 172.21.15.40:0/3226796884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cecb8240 msgr2=0x5560cec9d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cecb8240 0x5560cec9d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.223+0000 7efcb4e00640 1 -- 172.21.15.40:0/3226796884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560ceb00d80 con 0x5560cecb7920 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcb4e00640 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 0x5560cec9d280 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7efc9c006db0 tx=0x7efc9c01ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3988a2058ece91 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efc96a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efc96a00640 1 -- 172.21.15.40:0/3226796884 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c0c7050 con 0x5560cecb7920 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efc96a00640 1 -- 172.21.15.40:0/3226796884 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efc9c0ab890 con 0x5560cecb7920 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 -- 172.21.15.40:0/3226796884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efc9c0b1040 con 0x5560cecb7920 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efc96a00640 1 -- 172.21.15.40:0/3226796884 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc9c0d8050 con 0x5560cecb7920 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 10 client.6138 fetch_fsmap learned FSMap version 42 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 -- 172.21.15.40:0/3226796884 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x5560cea42d10 con 0x5560cecb7920 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efc96a00640 1 -- 172.21.15.40:0/3226796884 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9c0b1040 con 0x5560cecb7920 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efc96a00640 1 -- 172.21.15.40:0/3226796884 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) v1 ==== 3362+0+0 (secure 0 0 0) 0x7efc9c0db020 con 0x5560cecb7920 2024-03-24T00:33:34.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 10 client.6138 fetch_fsmap finished waiting for FSMap version 42 2024-03-24T00:33:34.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 10 client.6138 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:33:34.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 20 client.6138 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:34.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x5560cec9e090 0x5560cecc9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 4 client.6138 mds_command: new command op to 5192 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:34.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 -- 172.21.15.40:0/3226796884 --> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560cea56b60 con 0x5560cec9e090 2024-03-24T00:33:34.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcb5800640 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x5560cec9e090 0x5560cecc9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcb5800640 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x5560cec9e090 0x5560cecc9ea0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=39b4ad76a2fec7e4 server_cookie=df6cbb983970b940 in_seq=0 out_seq=0 2024-03-24T00:33:34.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efc96a00640 10 client.6138 ms_handle_connect on v2:172.21.15.40:6833/2128553323 2024-03-24T00:33:34.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efc96a00640 1 -- 172.21.15.40:0/3226796884 <== mds.1 v2:172.21.15.40:6833/2128553323 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5560cea56b60 con 0x5560cec9e090 2024-03-24T00:33:34.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efc96a00640 10 client.6138 handle_command_reply: tid=0 2024-03-24T00:33:34.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 client.6138 shutdown 2024-03-24T00:33:34.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 -- 172.21.15.40:0/3226796884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 msgr2=0x5560cec9d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 0x5560cec9d280 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7efc9c006db0 tx=0x7efc9c01ac40 comp rx=0 tx=0).stop 2024-03-24T00:33:34.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 -- 172.21.15.40:0/3226796884 shutdown_connections 2024-03-24T00:33:34.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x5560cec9e090 0x5560cecc9ea0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cecb8240 0x5560cec9d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cecb7920 0x5560cec9d280 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 --2- 172.21.15.40:0/3226796884 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560cec9cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efcbb792000 1 -- 172.21.15.40:0/3226796884 >> 172.21.15.40:0/3226796884 conn(0x5560cec69fa0 msgr2=0x5560cec6cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.227+0000 7efc96a00640 0 client.6138 ms_handle_reset on v2:172.21.15.40:6833/2128553323 2024-03-24T00:33:34.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.231+0000 7efcbb792000 1 -- 172.21.15.40:0/3226796884 shutdown_connections 2024-03-24T00:33:34.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.231+0000 7efcbb792000 1 -- 172.21.15.40:0/3226796884 wait complete. 2024-03-24T00:33:34.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.231+0000 7efcbb792000 20 client.6138 trim_cache size 0 max 16384 2024-03-24T00:33:34.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 Processor -- start 2024-03-24T00:33:34.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 -- start start 2024-03-24T00:33:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 0x5560cecb6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec69fa0 0x5560cecb6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560cec6c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb0203630 con 0x5560cec9ced0 2024-03-24T00:33:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb0202a50 con 0x5560cec69fa0 2024-03-24T00:33:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb0202fb0 con 0x5560cec6dff0 2024-03-24T00:33:34.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcb5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 0x5560cecb6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcb5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 0x5560cecb6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44654/0 (socket says 172.21.15.40:44654) 2024-03-24T00:33:34.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcb5800640 1 -- 172.21.15.40:0/1415443682 learned_addr learned my addr 172.21.15.40:0/1415443682 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcb4e00640 1 --2- 172.21.15.40:0/1415443682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec69fa0 0x5560cecb6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcb6200640 1 --2- 172.21.15.40:0/1415443682 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560cec6c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcb5800640 1 -- 172.21.15.40:0/1415443682 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 msgr2=0x5560cec6c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcb5800640 1 --2- 172.21.15.40:0/1415443682 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560cec6c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcb5800640 1 -- 172.21.15.40:0/1415443682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec69fa0 msgr2=0x5560cecb6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcb5800640 1 --2- 172.21.15.40:0/1415443682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec69fa0 0x5560cecb6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcb5800640 1 -- 172.21.15.40:0/1415443682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560cea88540 con 0x5560cec9ced0 2024-03-24T00:33:34.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcb5800640 1 --2- 172.21.15.40:0/1415443682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 0x5560cecb6530 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7efca405a220 tx=0x7efca401e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee5252c5042a828a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efca1600640 1 -- 172.21.15.40:0/1415443682 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4021040 con 0x5560cec9ced0 2024-03-24T00:33:34.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efca1600640 1 -- 172.21.15.40:0/1415443682 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca40742e0 con 0x5560cec9ced0 2024-03-24T00:33:34.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efca1600640 1 -- 172.21.15.40:0/1415443682 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca400cc80 con 0x5560cec9ced0 2024-03-24T00:33:34.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 -- 172.21.15.40:0/1415443682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 msgr2=0x5560cecb6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 --2- 172.21.15.40:0/1415443682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 0x5560cecb6530 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7efca405a220 tx=0x7efca401e040 comp rx=0 tx=0).stop 2024-03-24T00:33:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 -- 172.21.15.40:0/1415443682 shutdown_connections 2024-03-24T00:33:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 --2- 172.21.15.40:0/1415443682 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560cec6c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 --2- 172.21.15.40:0/1415443682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec69fa0 0x5560cecb6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 --2- 172.21.15.40:0/1415443682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 0x5560cecb6530 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 -- 172.21.15.40:0/1415443682 >> 172.21.15.40:0/1415443682 conn(0x5560cec70750 msgr2=0x5560cec9a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 -- 172.21.15.40:0/1415443682 shutdown_connections 2024-03-24T00:33:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.247+0000 7efcbb792000 1 -- 172.21.15.40:0/1415443682 wait complete. 2024-03-24T00:33:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 1 -- start start 2024-03-24T00:33:34.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 1 Processor -- start 2024-03-24T00:33:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec69fa0 0x5560ceccaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560ceccb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 0x5560ceccb970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560ceb78700 con 0x5560cec9ced0 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb00c7520 con 0x5560cec69fa0 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5560ceaba990 con 0x5560cec6dff0 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 10 client.0 _ll_register_callbacks cb 0x7efc9c0c5070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb6200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 0x5560ceccb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb4e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560ceccb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec69fa0 0x5560ceccaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb4e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560ceccb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:34810/0 (socket says 172.21.15.40:34810) 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec69fa0 0x5560ceccaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:42616/0 (socket says 172.21.15.40:42616) 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb4e00640 1 -- 172.21.15.40:0/3979077228 learned_addr learned my addr 172.21.15.40:0/3979077228 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb5800640 1 -- 172.21.15.40:0/3979077228 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 msgr2=0x5560ceccb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb5800640 1 --2- 172.21.15.40:0/3979077228 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560ceccb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb5800640 1 -- 172.21.15.40:0/3979077228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 msgr2=0x5560ceccb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb5800640 1 --2- 172.21.15.40:0/3979077228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 0x5560ceccb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb5800640 1 -- 172.21.15.40:0/3979077228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560cea4f4b0 con 0x5560cec69fa0 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb6200640 1 --2- 172.21.15.40:0/3979077228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 0x5560ceccb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb5800640 1 --2- 172.21.15.40:0/3979077228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec69fa0 0x5560ceccaef0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7efca4083ec0 tx=0x7efca405c040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d28584e8ad9ecd23 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efc96a00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efc96a00640 1 -- 172.21.15.40:0/3979077228 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4021040 con 0x5560cec69fa0 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efc96a00640 1 -- 172.21.15.40:0/3979077228 <== mon.1 v2:172.21.15.53:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efca405c0b0 con 0x5560cec69fa0 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 1 -- 172.21.15.40:0/3979077228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efca40742e0 con 0x5560cec69fa0 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efc96a00640 1 -- 172.21.15.40:0/3979077228 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7efca4084090 con 0x5560cec69fa0 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 10 client.5956 fetch_fsmap learned FSMap version 42 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 1 -- 172.21.15.40:0/3979077228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x5560cea622f0 con 0x5560cec69fa0 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efc96a00640 1 -- 172.21.15.40:0/3979077228 <== mon.1 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca40742e0 con 0x5560cec69fa0 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efc96a00640 1 -- 172.21.15.40:0/3979077228 <== mon.1 v2:172.21.15.53:3300/0 5 ==== fsmap(e 42) v1 ==== 3362+0+0 (secure 0 0 0) 0x7efca407d300 con 0x5560cec69fa0 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 10 client.5956 fetch_fsmap finished waiting for FSMap version 42 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 10 client.5956 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 20 client.5956 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:34.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 1 --2- 172.21.15.40:0/3979077228 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x5560cecb7920 0x5560ceccc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 4 client.5956 mds_command: new command op to 5192 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:33:34.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcbb792000 1 -- 172.21.15.40:0/3979077228 --> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5560ceac8bc0 con 0x5560cecb7920 2024-03-24T00:33:34.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb4e00640 1 --2- 172.21.15.40:0/3979077228 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x5560cecb7920 0x5560ceccc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efcb4e00640 1 --2- 172.21.15.40:0/3979077228 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x5560cecb7920 0x5560ceccc300 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=24014fca94365844 server_cookie=7a18261da67de29 in_seq=0 out_seq=0 2024-03-24T00:33:34.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.251+0000 7efc96a00640 10 client.5956 ms_handle_connect on v2:172.21.15.40:6833/2128553323 2024-03-24T00:33:34.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efc96a00640 1 -- 172.21.15.40:0/3979077228 <== mds.1 v2:172.21.15.40:6833/2128553323 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5560ceac8bc0 con 0x5560cecb7920 2024-03-24T00:33:34.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efc96a00640 10 client.5956 handle_command_reply: tid=0 2024-03-24T00:33:34.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcbb792000 1 client.5956 shutdown 2024-03-24T00:33:34.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcbb792000 1 -- 172.21.15.40:0/3979077228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec69fa0 msgr2=0x5560ceccaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcbb792000 1 --2- 172.21.15.40:0/3979077228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec69fa0 0x5560ceccaef0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7efca4083ec0 tx=0x7efca405c040 comp rx=0 tx=0).stop 2024-03-24T00:33:34.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcbb792000 1 -- 172.21.15.40:0/3979077228 shutdown_connections 2024-03-24T00:33:34.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcbb792000 1 --2- 172.21.15.40:0/3979077228 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x5560cecb7920 0x5560ceccc300 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcbb792000 1 --2- 172.21.15.40:0/3979077228 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5560cec9ced0 0x5560ceccb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcbb792000 1 --2- 172.21.15.40:0/3979077228 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5560cec6dff0 0x5560ceccb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efc96a00640 0 client.5956 ms_handle_reset on v2:172.21.15.40:6833/2128553323 2024-03-24T00:33:34.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcbb792000 1 --2- 172.21.15.40:0/3979077228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5560cec69fa0 0x5560ceccaef0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcbb792000 1 -- 172.21.15.40:0/3979077228 >> 172.21.15.40:0/3979077228 conn(0x5560cec70750 msgr2=0x5560cec9c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcbb792000 1 -- 172.21.15.40:0/3979077228 shutdown_connections 2024-03-24T00:33:34.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcbb792000 1 -- 172.21.15.40:0/3979077228 wait complete. 2024-03-24T00:33:34.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcbb792000 20 client.5956 trim_cache size 0 max 16384 2024-03-24T00:33:34.260 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:34.260 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:33:34.260 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-03-24T00:33:34.260 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5192, 2024-03-24T00:33:34.260 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:33:34.260 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:33:34.261 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:33:34.261 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 6.367054972, 2024-03-24T00:33:34.261 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 42, 2024-03-24T00:33:34.261 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 45, 2024-03-24T00:33:34.261 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 45, 2024-03-24T00:33:34.261 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 41.448569374999998 2024-03-24T00:33:34.261 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcb8200640 1 -- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efc90067660 msgr2=0x7efc90069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcb8200640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efc90067660 0x7efc90069b20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7efc9c0ad000 tx=0x7efc9c008040 comp rx=0 tx=0).stop 2024-03-24T00:33:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcb8200640 1 -- 172.21.15.40:0/2089445034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb00d0b40 msgr2=0x7efcb014dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcb8200640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb00d0b40 0x7efcb014dfc0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7efca400e870 tx=0x7efca400ec70 comp rx=0 tx=0).stop 2024-03-24T00:33:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcb8200640 1 -- 172.21.15.40:0/2089445034 shutdown_connections 2024-03-24T00:33:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcb8200640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efc90067660 0x7efc90069b20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcb8200640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efcb015e3a0 0x7efcb015cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcb8200640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efcb015dfc0 0x7efcb00cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcb8200640 1 --2- 172.21.15.40:0/2089445034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efcb00d0b40 0x7efcb014dfc0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcb8200640 1 -- 172.21.15.40:0/2089445034 >> 172.21.15.40:0/2089445034 conn(0x7efcb00b37f0 msgr2=0x7efcb00d1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcb8200640 1 -- 172.21.15.40:0/2089445034 shutdown_connections 2024-03-24T00:33:34.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.255+0000 7efcb8200640 1 -- 172.21.15.40:0/2089445034 wait complete. 2024-03-24T00:33:34.277 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:33:34.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.487+0000 7f6238a00640 1 Processor -- start 2024-03-24T00:33:34.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.487+0000 7f6238a00640 1 -- start start 2024-03-24T00:33:34.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.487+0000 7f6238a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f623415aa80 0x7f62340cb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.487+0000 7f6238a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6234072310 0x7f623406e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.487+0000 7f6238a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62340cbcd0 con 0x7f623415ae60 2024-03-24T00:33:34.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.487+0000 7f6238a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62340cad40 con 0x7f623415aa80 2024-03-24T00:33:34.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.487+0000 7f6238a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623406e5c0 con 0x7f6234072310 2024-03-24T00:33:34.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.491+0000 7f6233400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f623415aa80 0x7f62340cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.491+0000 7f6233400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f623415aa80 0x7f62340cb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44670/0 (socket says 172.21.15.40:44670) 2024-03-24T00:33:34.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.491+0000 7f6233400640 1 -- 172.21.15.40:0/1894256216 learned_addr learned my addr 172.21.15.40:0/1894256216 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.491+0000 7f6233e00640 1 --2- 172.21.15.40:0/1894256216 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6234072310 0x7f623406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.491+0000 7f6233400640 1 -- 172.21.15.40:0/1894256216 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6234072310 msgr2=0x7f623406e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.491+0000 7f6233400640 1 --2- 172.21.15.40:0/1894256216 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6234072310 0x7f623406e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.491+0000 7f6233400640 1 -- 172.21.15.40:0/1894256216 >> v1:172.21.15.53:6789/0 conn(0x7f623415ae60 legacy=0x7f623406d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.491+0000 7f6232a00640 1 -- 172.21.15.40:0/1894256216 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655688883 0 0) 0x7f62340cbcd0 con 0x7f623415ae60 2024-03-24T00:33:34.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.491+0000 7f6233400640 1 -- 172.21.15.40:0/1894256216 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62340cd8b0 con 0x7f623415aa80 2024-03-24T00:33:34.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.491+0000 7f6233400640 1 --2- 172.21.15.40:0/1894256216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623415aa80 0x7f62340cb790 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6224009870 tx=0x7f622401e6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312baff911adfeb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.491+0000 7f6232a00640 1 -- 172.21.15.40:0/1894256216 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62240ac650 con 0x7f623415aa80 2024-03-24T00:33:34.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.491+0000 7f6232a00640 1 -- 172.21.15.40:0/1894256216 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62240b1070 con 0x7f623415aa80 2024-03-24T00:33:34.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6232a00640 1 -- 172.21.15.40:0/1894256216 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62240aca10 con 0x7f623415aa80 2024-03-24T00:33:34.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 -- 172.21.15.40:0/1894256216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623415aa80 msgr2=0x7f62340cb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 --2- 172.21.15.40:0/1894256216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623415aa80 0x7f62340cb790 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6224009870 tx=0x7f622401e6e0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 -- 172.21.15.40:0/1894256216 shutdown_connections 2024-03-24T00:33:34.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 --2- 172.21.15.40:0/1894256216 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6234072310 0x7f623406e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 --2- 172.21.15.40:0/1894256216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623415aa80 0x7f62340cb790 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 -- 172.21.15.40:0/1894256216 >> 172.21.15.40:0/1894256216 conn(0x7f62340cfce0 msgr2=0x7f62340d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 -- 172.21.15.40:0/1894256216 shutdown_connections 2024-03-24T00:33:34.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 -- 172.21.15.40:0/1894256216 wait complete. 2024-03-24T00:33:34.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 Processor -- start 2024-03-24T00:33:34.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 -- start start 2024-03-24T00:33:34.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6234072310 0x7f623414ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f623415aa80 0x7f623414e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623415ae60 0x7f62340ad8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623415f170 con 0x7f623415ae60 2024-03-24T00:33:34.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f623415eff0 con 0x7f6234072310 2024-03-24T00:33:34.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6238a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f623415f2f0 con 0x7f623415aa80 2024-03-24T00:33:34.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6233400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6234072310 0x7f623414ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f622be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f623415aa80 0x7f623414e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6233e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623415ae60 0x7f62340ad8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.495+0000 7f6233400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6234072310 0x7f623414ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:42620/0 (socket says 172.21.15.40:42620) 2024-03-24T00:33:34.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f6233400640 1 -- 172.21.15.40:0/3982661401 learned_addr learned my addr 172.21.15.40:0/3982661401 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f6233400640 1 -- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f623415aa80 msgr2=0x7f623414e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f6233400640 1 --2- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f623415aa80 0x7f623414e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f6233400640 1 -- 172.21.15.40:0/3982661401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623415ae60 msgr2=0x7f62340ad8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f6233400640 1 --2- 172.21.15.40:0/3982661401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623415ae60 0x7f62340ad8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f6233400640 1 -- 172.21.15.40:0/3982661401 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6234155570 con 0x7f6234072310 2024-03-24T00:33:34.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f6233400640 1 --2- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6234072310 0x7f623414ecd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f62240ae040 tx=0x7f622401cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f622b400640 1 -- 172.21.15.40:0/3982661401 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62240bb070 con 0x7f6234072310 2024-03-24T00:33:34.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f622b400640 1 -- 172.21.15.40:0/3982661401 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62240a8040 con 0x7f6234072310 2024-03-24T00:33:34.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f622b400640 1 -- 172.21.15.40:0/3982661401 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62240b78a0 con 0x7f6234072310 2024-03-24T00:33:34.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f6238a00640 1 -- 172.21.15.40:0/3982661401 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6234154fb0 con 0x7f6234072310 2024-03-24T00:33:34.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f6238a00640 1 -- 172.21.15.40:0/3982661401 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f623415b3f0 con 0x7f6234072310 2024-03-24T00:33:34.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.499+0000 7f622b400640 1 -- 172.21.15.40:0/3982661401 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f62240be070 con 0x7f6234072310 2024-03-24T00:33:34.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f622b400640 1 --2- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6210067660 0x7f6210069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f622b400640 1 -- 172.21.15.40:0/3982661401 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f622411fe70 con 0x7f6234072310 2024-03-24T00:33:34.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f622be00640 1 --2- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6210067660 0x7f6210069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f623d4c3000 1 Processor -- start 2024-03-24T00:33:34.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f623d4c3000 1 -- start start 2024-03-24T00:33:34.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f622be00640 1 --2- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6210067660 0x7f6210069b20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f62340c6830 tx=0x7f621c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f623d4c3000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec456d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f623d4c3000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 0x5564ec40cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f623d4c3000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec40d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f623d4c3000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ec25a680 con 0x5564ec457240 2024-03-24T00:33:34.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f623d4c3000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ec1e1b00 con 0x5564ec456920 2024-03-24T00:33:34.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f623d4c3000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5564ec256720 con 0x5564ec40cff0 2024-03-24T00:33:34.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f622be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 0x5564ec40cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f622be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 0x5564ec40cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44694/0 (socket says 172.21.15.40:44694) 2024-03-24T00:33:34.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f622be00640 1 -- 172.21.15.40:0/98948212 learned_addr learned my addr 172.21.15.40:0/98948212 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f6233400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec456d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f6233e00640 1 --2- 172.21.15.40:0/98948212 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec40d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f6233400640 1 -- 172.21.15.40:0/98948212 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 msgr2=0x5564ec40d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f6233400640 1 --2- 172.21.15.40:0/98948212 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec40d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f6233400640 1 -- 172.21.15.40:0/98948212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 msgr2=0x5564ec40cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f6233400640 1 --2- 172.21.15.40:0/98948212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 0x5564ec40cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f6233400640 1 -- 172.21.15.40:0/98948212 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564ec232940 con 0x5564ec456920 2024-03-24T00:33:34.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f6233400640 1 --2- 172.21.15.40:0/98948212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec456d00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f622411e390 tx=0x7f62240eff80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b72b9bd843803dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f6223e00640 1 -- 172.21.15.40:0/98948212 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62240c5070 con 0x5564ec456920 2024-03-24T00:33:34.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f6223e00640 1 -- 172.21.15.40:0/98948212 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62240b1040 con 0x5564ec456920 2024-03-24T00:33:34.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.503+0000 7f6223e00640 1 -- 172.21.15.40:0/98948212 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62240f44e0 con 0x5564ec456920 2024-03-24T00:33:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 -- 172.21.15.40:0/98948212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 msgr2=0x5564ec456d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 --2- 172.21.15.40:0/98948212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec456d00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f622411e390 tx=0x7f62240eff80 comp rx=0 tx=0).stop 2024-03-24T00:33:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 -- 172.21.15.40:0/98948212 shutdown_connections 2024-03-24T00:33:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 --2- 172.21.15.40:0/98948212 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec40d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 --2- 172.21.15.40:0/98948212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 0x5564ec40cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 --2- 172.21.15.40:0/98948212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec456d00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 -- 172.21.15.40:0/98948212 >> 172.21.15.40:0/98948212 conn(0x5564ec408fa0 msgr2=0x5564ec40b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 -- 172.21.15.40:0/98948212 shutdown_connections 2024-03-24T00:33:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 -- 172.21.15.40:0/98948212 wait complete. 2024-03-24T00:33:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 -- start start 2024-03-24T00:33:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 Processor -- start 2024-03-24T00:33:34.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec43bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec43c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 0x5564ec43c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ec259990 con 0x5564ec457240 2024-03-24T00:33:34.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ec214ae0 con 0x5564ec456920 2024-03-24T00:33:34.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5564ec1f2750 con 0x5564ec40cff0 2024-03-24T00:33:34.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 10 client.0 _ll_register_callbacks cb 0x5564ec3f7ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:34.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6233e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 0x5564ec43c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6233e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 0x5564ec43c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44708/0 (socket says 172.21.15.40:44708) 2024-03-24T00:33:34.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6233e00640 1 -- 172.21.15.40:0/65300573 learned_addr learned my addr 172.21.15.40:0/65300573 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6233400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec43bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f622be00640 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec43c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6233e00640 1 -- 172.21.15.40:0/65300573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 msgr2=0x5564ec43bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6233e00640 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec43bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6233e00640 1 -- 172.21.15.40:0/65300573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 msgr2=0x5564ec43c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6233e00640 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec43c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6233e00640 1 -- 172.21.15.40:0/65300573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564ec29fd80 con 0x5564ec457240 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6233400640 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec43bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6233e00640 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 0x5564ec43c7c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f622c014e70 tx=0x7f622c012670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86de2728aa7dd8f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6220c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6220c00640 1 -- 172.21.15.40:0/65300573 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622c01e600 con 0x5564ec457240 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6220c00640 1 -- 172.21.15.40:0/65300573 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f622c016400 con 0x5564ec457240 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 -- 172.21.15.40:0/65300573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f62240b1040 con 0x5564ec457240 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f6220c00640 1 -- 172.21.15.40:0/65300573 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f622c01e7a0 con 0x5564ec457240 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 10 client.6156 fetch_fsmap learned FSMap version 42 2024-03-24T00:33:34.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.507+0000 7f623d4c3000 1 -- 172.21.15.40:0/65300573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x5564ec1e1d10 con 0x5564ec457240 2024-03-24T00:33:34.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f6220c00640 1 -- 172.21.15.40:0/65300573 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62240b1040 con 0x5564ec457240 2024-03-24T00:33:34.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f6220c00640 1 -- 172.21.15.40:0/65300573 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) v1 ==== 3362+0+0 (secure 0 0 0) 0x7f622c01ebd0 con 0x5564ec457240 2024-03-24T00:33:34.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 10 client.6156 fetch_fsmap finished waiting for FSMap version 42 2024-03-24T00:33:34.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 10 client.6156 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:33:34.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 20 client.6156 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:34.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x5564ec43d060 0x5564ec468ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 4 client.6156 mds_command: new command op to 5198 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:34.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 -- 172.21.15.40:0/65300573 --> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564ec1f5b60 con 0x5564ec43d060 2024-03-24T00:33:34.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f6233400640 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x5564ec43d060 0x5564ec468ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f6233400640 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x5564ec43d060 0x5564ec468ea0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f8de091d0edc4acb server_cookie=4c420973d5ee6c5a in_seq=0 out_seq=0 2024-03-24T00:33:34.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f6220c00640 10 client.6156 ms_handle_connect on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:34.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f6220c00640 1 -- 172.21.15.40:0/65300573 <== mds.2 v2:172.21.15.53:6835/1453842342 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5564ec1f5b60 con 0x5564ec43d060 2024-03-24T00:33:34.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f6220c00640 10 client.6156 handle_command_reply: tid=0 2024-03-24T00:33:34.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 client.6156 shutdown 2024-03-24T00:33:34.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 -- 172.21.15.40:0/65300573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 msgr2=0x5564ec43c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 0x5564ec43c7c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f622c014e70 tx=0x7f622c012670 comp rx=0 tx=0).stop 2024-03-24T00:33:34.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 -- 172.21.15.40:0/65300573 shutdown_connections 2024-03-24T00:33:34.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x5564ec43d060 0x5564ec468ea0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec457240 0x5564ec43c7c0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec43c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 --2- 172.21.15.40:0/65300573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec43bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f6220c00640 0 client.6156 ms_handle_reset on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:34.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 -- 172.21.15.40:0/65300573 >> 172.21.15.40:0/65300573 conn(0x5564ec408fa0 msgr2=0x5564ec40bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 -- 172.21.15.40:0/65300573 shutdown_connections 2024-03-24T00:33:34.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 1 -- 172.21.15.40:0/65300573 wait complete. 2024-03-24T00:33:34.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.511+0000 7f623d4c3000 20 client.6156 trim_cache size 0 max 16384 2024-03-24T00:33:34.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.527+0000 7f623d4c3000 1 Processor -- start 2024-03-24T00:33:34.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.527+0000 7f623d4c3000 1 -- start start 2024-03-24T00:33:34.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec455e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec456920 0x5564ec4395d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec457240 0x5564ec439b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6234202dd0 con 0x5564ec456920 2024-03-24T00:33:34.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62342020b0 con 0x5564ec457240 2024-03-24T00:33:34.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6234202710 con 0x5564ec40cff0 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f622be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec456920 0x5564ec4395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f622be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec456920 0x5564ec4395d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44724/0 (socket says 172.21.15.40:44724) 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f622be00640 1 -- 172.21.15.40:0/3729634526 learned_addr learned my addr 172.21.15.40:0/3729634526 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f6233400640 1 --2- 172.21.15.40:0/3729634526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec455e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f6233e00640 1 --2- 172.21.15.40:0/3729634526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec457240 0x5564ec439b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f622be00640 1 -- 172.21.15.40:0/3729634526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 msgr2=0x5564ec455e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f622be00640 1 --2- 172.21.15.40:0/3729634526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec455e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f622be00640 1 -- 172.21.15.40:0/3729634526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec457240 msgr2=0x5564ec439b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f622be00640 1 --2- 172.21.15.40:0/3729634526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec457240 0x5564ec439b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f622be00640 1 -- 172.21.15.40:0/3729634526 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564ec227540 con 0x5564ec456920 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f622be00640 1 --2- 172.21.15.40:0/3729634526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec456920 0x5564ec4395d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f621c00fb90 tx=0x7f621c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c67a4037ee3f4c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f6223400640 1 -- 172.21.15.40:0/3729634526 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c014d60 con 0x5564ec456920 2024-03-24T00:33:34.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f6223400640 1 -- 172.21.15.40:0/3729634526 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621c01b040 con 0x5564ec456920 2024-03-24T00:33:34.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f6223400640 1 -- 172.21.15.40:0/3729634526 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f621c023650 con 0x5564ec456920 2024-03-24T00:33:34.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 -- 172.21.15.40:0/3729634526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec456920 msgr2=0x5564ec4395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 --2- 172.21.15.40:0/3729634526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec456920 0x5564ec4395d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f621c00fb90 tx=0x7f621c012000 comp rx=0 tx=0).stop 2024-03-24T00:33:34.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 -- 172.21.15.40:0/3729634526 shutdown_connections 2024-03-24T00:33:34.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 --2- 172.21.15.40:0/3729634526 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec457240 0x5564ec439b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 --2- 172.21.15.40:0/3729634526 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec456920 0x5564ec4395d0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 --2- 172.21.15.40:0/3729634526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec40cff0 0x5564ec455e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 -- 172.21.15.40:0/3729634526 >> 172.21.15.40:0/3729634526 conn(0x5564ec408fa0 msgr2=0x5564ec455500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 -- 172.21.15.40:0/3729634526 shutdown_connections 2024-03-24T00:33:34.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 -- 172.21.15.40:0/3729634526 wait complete. 2024-03-24T00:33:34.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 -- start start 2024-03-24T00:33:34.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.531+0000 7f623d4c3000 1 Processor -- start 2024-03-24T00:33:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec40cff0 0x5564ec469ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec46a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec457240 0x5564ec46a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564ec317700 con 0x5564ec40cff0 2024-03-24T00:33:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6234201240 con 0x5564ec456920 2024-03-24T00:33:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5564ec408e20 con 0x5564ec457240 2024-03-24T00:33:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 10 client.0 _ll_register_callbacks cb 0x7f622c009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6233400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec40cff0 0x5564ec469ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6233400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec40cff0 0x5564ec469ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44738/0 (socket says 172.21.15.40:44738) 2024-03-24T00:33:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6233400640 1 -- 172.21.15.40:0/1284764041 learned_addr learned my addr 172.21.15.40:0/1284764041 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f622be00640 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec46a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6233e00640 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec457240 0x5564ec46a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6233400640 1 -- 172.21.15.40:0/1284764041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec457240 msgr2=0x5564ec46a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6233400640 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec457240 0x5564ec46a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6233400640 1 -- 172.21.15.40:0/1284764041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 msgr2=0x5564ec46a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6233400640 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec46a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6233400640 1 -- 172.21.15.40:0/1284764041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564ec1ee4b0 con 0x5564ec40cff0 2024-03-24T00:33:34.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6233400640 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec40cff0 0x5564ec469ef0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f62240f03d0 tx=0x7f622401e6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d197390e77ec512b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6220c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:34.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6220c00640 1 -- 172.21.15.40:0/1284764041 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62240f44e0 con 0x5564ec40cff0 2024-03-24T00:33:34.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6220c00640 1 -- 172.21.15.40:0/1284764041 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f62241081c0 con 0x5564ec40cff0 2024-03-24T00:33:34.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 1 -- 172.21.15.40:0/1284764041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f621c01b040 con 0x5564ec40cff0 2024-03-24T00:33:34.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6220c00640 1 -- 172.21.15.40:0/1284764041 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62240f40d0 con 0x5564ec40cff0 2024-03-24T00:33:34.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 10 client.6162 fetch_fsmap learned FSMap version 42 2024-03-24T00:33:34.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 1 -- 172.21.15.40:0/1284764041 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x5564ec2012f0 con 0x5564ec40cff0 2024-03-24T00:33:34.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6220c00640 1 -- 172.21.15.40:0/1284764041 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f621c01b040 con 0x5564ec40cff0 2024-03-24T00:33:34.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6220c00640 1 -- 172.21.15.40:0/1284764041 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) v1 ==== 3362+0+0 (secure 0 0 0) 0x7f62240ef890 con 0x5564ec40cff0 2024-03-24T00:33:34.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 10 client.6162 fetch_fsmap finished waiting for FSMap version 42 2024-03-24T00:33:34.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 10 client.6162 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:33:34.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 20 client.6162 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:34.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x5564ec46aeb0 0x5564ec46b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 4 client.6162 mds_command: new command op to 5198 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:33:34.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f623d4c3000 1 -- 172.21.15.40:0/1284764041 --> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5564ec267bc0 con 0x5564ec46aeb0 2024-03-24T00:33:34.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f622be00640 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x5564ec46aeb0 0x5564ec46b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f622be00640 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x5564ec46aeb0 0x5564ec46b290 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=37a11d4b8690d6af server_cookie=7f4e417ed824b68a in_seq=0 out_seq=0 2024-03-24T00:33:34.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.535+0000 7f6220c00640 10 client.6162 ms_handle_connect on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:34.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6220c00640 1 -- 172.21.15.40:0/1284764041 <== mds.2 v2:172.21.15.53:6835/1453842342 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5564ec267bc0 con 0x5564ec46aeb0 2024-03-24T00:33:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6220c00640 10 client.6162 handle_command_reply: tid=0 2024-03-24T00:33:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f623d4c3000 1 client.6162 shutdown 2024-03-24T00:33:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f623d4c3000 1 -- 172.21.15.40:0/1284764041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec40cff0 msgr2=0x5564ec469ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f623d4c3000 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec40cff0 0x5564ec469ef0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f62240f03d0 tx=0x7f622401e6e0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f623d4c3000 1 -- 172.21.15.40:0/1284764041 shutdown_connections 2024-03-24T00:33:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f623d4c3000 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x5564ec46aeb0 0x5564ec46b290 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f623d4c3000 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5564ec457240 0x5564ec46a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f623d4c3000 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5564ec456920 0x5564ec46a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f623d4c3000 1 --2- 172.21.15.40:0/1284764041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5564ec40cff0 0x5564ec469ef0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f623d4c3000 1 -- 172.21.15.40:0/1284764041 >> 172.21.15.40:0/1284764041 conn(0x5564ec408fa0 msgr2=0x5564ec455d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6220c00640 0 client.6162 ms_handle_reset on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:34.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f623d4c3000 1 -- 172.21.15.40:0/1284764041 shutdown_connections 2024-03-24T00:33:34.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f623d4c3000 1 -- 172.21.15.40:0/1284764041 wait complete. 2024-03-24T00:33:34.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f623d4c3000 20 client.6162 trim_cache size 0 max 16384 2024-03-24T00:33:34.545 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:34.545 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:33:34.545 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-03-24T00:33:34.545 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5198, 2024-03-24T00:33:34.545 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:33:34.545 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:33:34.545 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:33:34.546 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 23.004719735999998, 2024-03-24T00:33:34.546 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 42, 2024-03-24T00:33:34.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 45, 2024-03-24T00:33:34.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 45, 2024-03-24T00:33:34.546 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 41.721674876000002 2024-03-24T00:33:34.546 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6238a00640 1 -- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6210067660 msgr2=0x7f6210069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6238a00640 1 --2- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6210067660 0x7f6210069b20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f62340c6830 tx=0x7f621c009040 comp rx=0 tx=0).stop 2024-03-24T00:33:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6238a00640 1 -- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6234072310 msgr2=0x7f623414ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6238a00640 1 --2- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6234072310 0x7f623414ecd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f62240ae040 tx=0x7f622401cf20 comp rx=0 tx=0).stop 2024-03-24T00:33:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6238a00640 1 -- 172.21.15.40:0/3982661401 shutdown_connections 2024-03-24T00:33:34.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6238a00640 1 --2- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6210067660 0x7f6210069b20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6238a00640 1 --2- 172.21.15.40:0/3982661401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623415ae60 0x7f62340ad8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6238a00640 1 --2- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f623415aa80 0x7f623414e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6238a00640 1 --2- 172.21.15.40:0/3982661401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6234072310 0x7f623414ecd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6238a00640 1 -- 172.21.15.40:0/3982661401 >> 172.21.15.40:0/3982661401 conn(0x7f62340cfce0 msgr2=0x7f62340afe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6238a00640 1 -- 172.21.15.40:0/3982661401 shutdown_connections 2024-03-24T00:33:34.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.539+0000 7f6238a00640 1 -- 172.21.15.40:0/3982661401 wait complete. 2024-03-24T00:33:34.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:33:34.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.707+0000 7f2352800640 1 Processor -- start 2024-03-24T00:33:34.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.707+0000 7f2352800640 1 -- start start 2024-03-24T00:33:34.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.707+0000 7f2352800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f234c141bc0 0x7f234c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f2352800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f234c0ce630 0x7f234c071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f2352800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234c070e00 con 0x7f234c13ec80 2024-03-24T00:33:34.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f2352800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f234c0cee20 con 0x7f234c0ce630 2024-03-24T00:33:34.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f2352800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f234c0cf100 con 0x7f234c141bc0 2024-03-24T00:33:34.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f234b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f234c13ec80 0x7f234c1414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:41524/0 (socket says 172.21.15.40:41524) 2024-03-24T00:33:34.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f234b400640 1 -- 172.21.15.40:0/4235270153 learned_addr learned my addr 172.21.15.40:0/4235270153 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:34.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f234be00640 1 --2- 172.21.15.40:0/4235270153 >> v2:172.21.15.40:3300/0 conn(0x7f234c0ce630 0x7f234c071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f2343e00640 1 --2- 172.21.15.40:0/4235270153 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f234c141bc0 0x7f234c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f234be00640 1 -- 172.21.15.40:0/4235270153 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f234c141bc0 msgr2=0x7f234c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f234be00640 1 --2- 172.21.15.40:0/4235270153 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f234c141bc0 0x7f234c0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f234be00640 1 -- 172.21.15.40:0/4235270153 >> v1:172.21.15.40:6789/0 conn(0x7f234c13ec80 legacy=0x7f234c1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f234be00640 1 -- 172.21.15.40:0/4235270153 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c0cf3e0 con 0x7f234c0ce630 2024-03-24T00:33:34.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f2343e00640 1 --2- 172.21.15.40:0/4235270153 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f234c141bc0 0x7f234c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:34.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f234be00640 1 --2- 172.21.15.40:0/4235270153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f234c0ce630 0x7f234c071300 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2344009d90 tx=0x7f234401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a55def94e37c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.711+0000 7f234aa00640 1 -- 172.21.15.40:0/4235270153 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23440b2070 con 0x7f234c0ce630 2024-03-24T00:33:34.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.715+0000 7f234aa00640 1 -- 172.21.15.40:0/4235270153 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23440a5a50 con 0x7f234c0ce630 2024-03-24T00:33:34.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.715+0000 7f234aa00640 1 -- 172.21.15.40:0/4235270153 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23440a5e20 con 0x7f234c0ce630 2024-03-24T00:33:34.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.715+0000 7f2352800640 1 -- 172.21.15.40:0/4235270153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f234c0ce630 msgr2=0x7f234c071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.715+0000 7f2352800640 1 --2- 172.21.15.40:0/4235270153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f234c0ce630 0x7f234c071300 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f2344009d90 tx=0x7f234401c970 comp rx=0 tx=0).stop 2024-03-24T00:33:34.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.715+0000 7f2352800640 1 -- 172.21.15.40:0/4235270153 shutdown_connections 2024-03-24T00:33:34.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.715+0000 7f2352800640 1 --2- 172.21.15.40:0/4235270153 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f234c0ce630 0x7f234c071300 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.715+0000 7f2352800640 1 --2- 172.21.15.40:0/4235270153 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f234c141bc0 0x7f234c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.715+0000 7f2352800640 1 -- 172.21.15.40:0/4235270153 >> 172.21.15.40:0/4235270153 conn(0x7f234c0b17b0 msgr2=0x7f234c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.715+0000 7f2352800640 1 -- 172.21.15.40:0/4235270153 shutdown_connections 2024-03-24T00:33:34.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.715+0000 7f2352800640 1 -- 172.21.15.40:0/4235270153 wait complete. 2024-03-24T00:33:34.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.715+0000 7f2352800640 1 Processor -- start 2024-03-24T00:33:34.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.715+0000 7f2352800640 1 -- start start 2024-03-24T00:33:34.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2352800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f234c0ce630 0x7f234c1f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2352800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f234c13ec80 0x7f234c13d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2352800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f234c141bc0 0x7f234c155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2352800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234c0723e0 con 0x7f234c13ec80 2024-03-24T00:33:34.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2352800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234c072260 con 0x7f234c0ce630 2024-03-24T00:33:34.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2352800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f234c072560 con 0x7f234c141bc0 2024-03-24T00:33:34.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2343e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f234c13ec80 0x7f234c13d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2343e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f234c13ec80 0x7f234c13d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44752/0 (socket says 172.21.15.40:44752) 2024-03-24T00:33:34.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2343e00640 1 -- 172.21.15.40:0/2735371701 learned_addr learned my addr 172.21.15.40:0/2735371701 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f234b400640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f234c0ce630 0x7f234c1f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f234be00640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f234c141bc0 0x7f234c155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2343e00640 1 -- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f234c141bc0 msgr2=0x7f234c155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2343e00640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f234c141bc0 0x7f234c155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2343e00640 1 -- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f234c0ce630 msgr2=0x7f234c1f6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2343e00640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f234c0ce630 0x7f234c1f6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2343e00640 1 -- 172.21.15.40:0/2735371701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c1f74e0 con 0x7f234c13ec80 2024-03-24T00:33:34.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2343e00640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f234c13ec80 0x7f234c13d3a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f233400e980 tx=0x7f233400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2343400640 1 -- 172.21.15.40:0/2735371701 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334017070 con 0x7f234c13ec80 2024-03-24T00:33:34.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2343400640 1 -- 172.21.15.40:0/2735371701 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233400ca00 con 0x7f234c13ec80 2024-03-24T00:33:34.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2352800640 1 -- 172.21.15.40:0/2735371701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f234c1f77c0 con 0x7f234c13ec80 2024-03-24T00:33:34.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2343400640 1 -- 172.21.15.40:0/2735371701 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233400cdd0 con 0x7f234c13ec80 2024-03-24T00:33:34.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.719+0000 7f2352800640 1 -- 172.21.15.40:0/2735371701 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f234c06d1a0 con 0x7f234c13ec80 2024-03-24T00:33:34.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.723+0000 7f2343400640 1 -- 172.21.15.40:0/2735371701 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f233401a070 con 0x7f234c13ec80 2024-03-24T00:33:34.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.723+0000 7f2343400640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2328067660 0x7f2328069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.723+0000 7f2343400640 1 -- 172.21.15.40:0/2735371701 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2334085b80 con 0x7f234c13ec80 2024-03-24T00:33:34.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.723+0000 7f234b400640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2328067660 0x7f2328069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.723+0000 7f234b400640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2328067660 0x7f2328069b20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f234c06d130 tx=0x7f233c007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.723+0000 7f2355d5c000 1 Processor -- start 2024-03-24T00:33:34.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 -- start start 2024-03-24T00:33:34.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd37d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fcedab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fcedff0 0x55d65fcee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d65fb3b680 con 0x55d65fd37920 2024-03-24T00:33:34.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d65fac2b00 con 0x55d65fcedff0 2024-03-24T00:33:34.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d65fb37720 con 0x55d65fd38240 2024-03-24T00:33:34.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f234b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd37d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2343e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fcedab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2343e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fcedab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:34880/0 (socket says 172.21.15.40:34880) 2024-03-24T00:33:34.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2343e00640 1 -- 172.21.15.40:0/386821601 learned_addr learned my addr 172.21.15.40:0/386821601 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f234be00640 1 --2- 172.21.15.40:0/386821601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fcedff0 0x55d65fcee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2343e00640 1 -- 172.21.15.40:0/386821601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fcedff0 msgr2=0x55d65fcee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2343e00640 1 --2- 172.21.15.40:0/386821601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fcedff0 0x55d65fcee3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2343e00640 1 -- 172.21.15.40:0/386821601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 msgr2=0x55d65fd37d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2343e00640 1 --2- 172.21.15.40:0/386821601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd37d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2343e00640 1 -- 172.21.15.40:0/386821601 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d65fb13940 con 0x55d65fd38240 2024-03-24T00:33:34.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2343e00640 1 --2- 172.21.15.40:0/386821601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fcedab0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f233405e1e0 tx=0x7f2334055b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1a96aeb4cb2d220 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f233be00640 1 -- 172.21.15.40:0/386821601 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334021040 con 0x55d65fd38240 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f233be00640 1 -- 172.21.15.40:0/386821601 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2334054150 con 0x55d65fd38240 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f233be00640 1 -- 172.21.15.40:0/386821601 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334052720 con 0x55d65fd38240 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 -- 172.21.15.40:0/386821601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 msgr2=0x55d65fcedab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 --2- 172.21.15.40:0/386821601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fcedab0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f233405e1e0 tx=0x7f2334055b40 comp rx=0 tx=0).stop 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 -- 172.21.15.40:0/386821601 shutdown_connections 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 --2- 172.21.15.40:0/386821601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fcedff0 0x55d65fcee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 --2- 172.21.15.40:0/386821601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fcedab0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 --2- 172.21.15.40:0/386821601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd37d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 -- 172.21.15.40:0/386821601 >> 172.21.15.40:0/386821601 conn(0x55d65fce9fa0 msgr2=0x55d65fcec390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 -- 172.21.15.40:0/386821601 shutdown_connections 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.727+0000 7f2355d5c000 1 -- 172.21.15.40:0/386821601 wait complete. 2024-03-24T00:33:34.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f2355d5c000 1 -- start start 2024-03-24T00:33:34.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f2355d5c000 1 Processor -- start 2024-03-24T00:33:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f2355d5c000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fcedff0 0x55d65fd1cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f2355d5c000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd1d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f2355d5c000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fd38240 0x55d65fd1d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f2355d5c000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d65fb3a990 con 0x55d65fd37920 2024-03-24T00:33:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f2355d5c000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d65faf5ae0 con 0x55d65fd38240 2024-03-24T00:33:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f2355d5c000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d65fad3750 con 0x55d65fcedff0 2024-03-24T00:33:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f2355d5c000 10 client.0 _ll_register_callbacks cb 0x55d65fcd8ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f234b400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fcedff0 0x55d65fd1cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f2343e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd1d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f234b400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fcedff0 0x55d65fd1cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:34890/0 (socket says 172.21.15.40:34890) 2024-03-24T00:33:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f234b400640 1 -- 172.21.15.40:0/2456603284 learned_addr learned my addr 172.21.15.40:0/2456603284 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f234be00640 1 --2- 172.21.15.40:0/2456603284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fd38240 0x55d65fd1d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f2343e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd1d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44770/0 (socket says 172.21.15.40:44770) 2024-03-24T00:33:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f234b400640 1 -- 172.21.15.40:0/2456603284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fd38240 msgr2=0x55d65fd1d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f234b400640 1 --2- 172.21.15.40:0/2456603284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fd38240 0x55d65fd1d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f234b400640 1 -- 172.21.15.40:0/2456603284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 msgr2=0x55d65fd1d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f234b400640 1 --2- 172.21.15.40:0/2456603284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd1d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.731+0000 7f234b400640 1 -- 172.21.15.40:0/2456603284 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d65fb80d80 con 0x55d65fcedff0 2024-03-24T00:33:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f234b400640 1 --2- 172.21.15.40:0/2456603284 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fcedff0 0x55d65fd1cd40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f233c013780 tx=0x7f233c000bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd69e3f87047e568 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2338c00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:33:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2338c00640 1 -- 172.21.15.40:0/2456603284 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c020640 con 0x55d65fcedff0 2024-03-24T00:33:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2338c00640 1 -- 172.21.15.40:0/2456603284 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f233c0207e0 con 0x55d65fcedff0 2024-03-24T00:33:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 -- 172.21.15.40:0/2456603284 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2334054150 con 0x55d65fcedff0 2024-03-24T00:33:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2338c00640 1 -- 172.21.15.40:0/2456603284 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f233c02e050 con 0x55d65fcedff0 2024-03-24T00:33:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 10 client.5510 fetch_fsmap learned FSMap version 42 2024-03-24T00:33:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 -- 172.21.15.40:0/2456603284 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55d65fac2d10 con 0x55d65fcedff0 2024-03-24T00:33:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2338c00640 1 -- 172.21.15.40:0/2456603284 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2334054150 con 0x55d65fcedff0 2024-03-24T00:33:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2338c00640 1 -- 172.21.15.40:0/2456603284 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 42) v1 ==== 3362+0+0 (secure 0 0 0) 0x7f233c031020 con 0x55d65fcedff0 2024-03-24T00:33:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 10 client.5510 fetch_fsmap finished waiting for FSMap version 42 2024-03-24T00:33:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 10 client.5510 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:33:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 20 client.5510 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 --2- 172.21.15.40:0/2456603284 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55d65fd1e060 0x55d65fd49ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 4 client.5510 mds_command: new command op to 5763 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 -- 172.21.15.40:0/2456603284 --> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d65fad6b60 con 0x55d65fd1e060 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2343e00640 1 --2- 172.21.15.40:0/2456603284 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55d65fd1e060 0x55d65fd49ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2343e00640 1 --2- 172.21.15.40:0/2456603284 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55d65fd1e060 0x55d65fd49ea0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ef7a2918bc7d0c0 server_cookie=4cfaa9b7054f0975 in_seq=0 out_seq=0 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2338c00640 10 client.5510 ms_handle_connect on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2338c00640 1 -- 172.21.15.40:0/2456603284 <== mds.0 v2:172.21.15.40:6832/224223662 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d65fad6b60 con 0x55d65fd1e060 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2338c00640 10 client.5510 handle_command_reply: tid=0 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 client.5510 shutdown 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 -- 172.21.15.40:0/2456603284 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fcedff0 msgr2=0x55d65fd1cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 --2- 172.21.15.40:0/2456603284 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fcedff0 0x55d65fd1cd40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f233c013780 tx=0x7f233c000bd0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 -- 172.21.15.40:0/2456603284 shutdown_connections 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 --2- 172.21.15.40:0/2456603284 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55d65fd1e060 0x55d65fd49ea0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 --2- 172.21.15.40:0/2456603284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fd38240 0x55d65fd1d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 --2- 172.21.15.40:0/2456603284 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd1d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2338c00640 0 client.5510 ms_handle_reset on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 --2- 172.21.15.40:0/2456603284 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fcedff0 0x55d65fd1cd40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 -- 172.21.15.40:0/2456603284 >> 172.21.15.40:0/2456603284 conn(0x55d65fce9fa0 msgr2=0x55d65fcecc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 -- 172.21.15.40:0/2456603284 shutdown_connections 2024-03-24T00:33:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 1 -- 172.21.15.40:0/2456603284 wait complete. 2024-03-24T00:33:34.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.735+0000 7f2355d5c000 20 client.5510 trim_cache size 0 max 16384 2024-03-24T00:33:34.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 Processor -- start 2024-03-24T00:33:34.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 -- start start 2024-03-24T00:33:34.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fcedff0 0x55d65fd36e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd1b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fd1be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234c2033e0 con 0x55d65fd37920 2024-03-24T00:33:34.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234c202700 con 0x55d65fcedff0 2024-03-24T00:33:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f234c202d20 con 0x55d65fd38240 2024-03-24T00:33:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2343e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd1b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2343e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd1b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44780/0 (socket says 172.21.15.40:44780) 2024-03-24T00:33:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2343e00640 1 -- 172.21.15.40:0/1289645913 learned_addr learned my addr 172.21.15.40:0/1289645913 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f234b400640 1 --2- 172.21.15.40:0/1289645913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fcedff0 0x55d65fd36e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f234be00640 1 --2- 172.21.15.40:0/1289645913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fd1be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2343e00640 1 -- 172.21.15.40:0/1289645913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 msgr2=0x55d65fd1be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2343e00640 1 --2- 172.21.15.40:0/1289645913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fd1be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2343e00640 1 -- 172.21.15.40:0/1289645913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fcedff0 msgr2=0x55d65fd36e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2343e00640 1 --2- 172.21.15.40:0/1289645913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fcedff0 0x55d65fd36e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2343e00640 1 -- 172.21.15.40:0/1289645913 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d65fb08540 con 0x55d65fd37920 2024-03-24T00:33:34.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2343e00640 1 --2- 172.21.15.40:0/1289645913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd1b900 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f233405a310 tx=0x7f233401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc1c03d9564cf44 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f233b400640 1 -- 172.21.15.40:0/1289645913 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334021040 con 0x55d65fd37920 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f233b400640 1 -- 172.21.15.40:0/1289645913 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2334052720 con 0x55d65fd37920 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f233b400640 1 -- 172.21.15.40:0/1289645913 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23340744f0 con 0x55d65fd37920 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 -- 172.21.15.40:0/1289645913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 msgr2=0x55d65fd1b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 --2- 172.21.15.40:0/1289645913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd1b900 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f233405a310 tx=0x7f233401e000 comp rx=0 tx=0).stop 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 -- 172.21.15.40:0/1289645913 shutdown_connections 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 --2- 172.21.15.40:0/1289645913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fd1be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 --2- 172.21.15.40:0/1289645913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fd37920 0x55d65fd1b900 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 --2- 172.21.15.40:0/1289645913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fcedff0 0x55d65fd36e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 -- 172.21.15.40:0/1289645913 >> 172.21.15.40:0/1289645913 conn(0x55d65fce9fa0 msgr2=0x55d65fd36500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 -- 172.21.15.40:0/1289645913 shutdown_connections 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.755+0000 7f2355d5c000 1 -- 172.21.15.40:0/1289645913 wait complete. 2024-03-24T00:33:34.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 1 -- start start 2024-03-24T00:33:34.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 1 Processor -- start 2024-03-24T00:33:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fcedff0 0x55d65fd1c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fd37920 0x55d65fd4aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fd4b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d65fbf8700 con 0x55d65fcedff0 2024-03-24T00:33:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234c0b04b0 con 0x55d65fd37920 2024-03-24T00:33:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d65fae3a80 con 0x55d65fd38240 2024-03-24T00:33:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 10 client.0 _ll_register_callbacks cb 0x7f233c001470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f234b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fcedff0 0x55d65fd1c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f234b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fcedff0 0x55d65fd1c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:44794/0 (socket says 172.21.15.40:44794) 2024-03-24T00:33:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f234b400640 1 -- 172.21.15.40:0/2969820961 learned_addr learned my addr 172.21.15.40:0/2969820961 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:34.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2343e00640 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fd37920 0x55d65fd4aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f234be00640 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fd4b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f234b400640 1 -- 172.21.15.40:0/2969820961 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 msgr2=0x55d65fd4b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f234b400640 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fd4b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f234b400640 1 -- 172.21.15.40:0/2969820961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fd37920 msgr2=0x55d65fd4aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f234b400640 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fd37920 0x55d65fd4aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f234b400640 1 -- 172.21.15.40:0/2969820961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d65facf4b0 con 0x55d65fcedff0 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2343e00640 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fd37920 0x55d65fd4aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f234be00640 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fd4b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f234b400640 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fcedff0 0x55d65fd1c200 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f233c0047f0 tx=0x7f233c00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=add0e73d24b9ce1d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2338c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2338c00640 1 -- 172.21.15.40:0/2969820961 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233c001c30 con 0x55d65fcedff0 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2338c00640 1 -- 172.21.15.40:0/2969820961 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f233c010070 con 0x55d65fcedff0 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 1 -- 172.21.15.40:0/2969820961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2334052720 con 0x55d65fcedff0 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2338c00640 1 -- 172.21.15.40:0/2969820961 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f233c045050 con 0x55d65fcedff0 2024-03-24T00:33:34.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 10 client.6183 fetch_fsmap learned FSMap version 42 2024-03-24T00:33:34.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 1 -- 172.21.15.40:0/2969820961 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55d65fae22f0 con 0x55d65fcedff0 2024-03-24T00:33:34.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2338c00640 1 -- 172.21.15.40:0/2969820961 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2334052720 con 0x55d65fcedff0 2024-03-24T00:33:34.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2338c00640 1 -- 172.21.15.40:0/2969820961 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) v1 ==== 3362+0+0 (secure 0 0 0) 0x7f233c031040 con 0x55d65fcedff0 2024-03-24T00:33:34.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 10 client.6183 fetch_fsmap finished waiting for FSMap version 42 2024-03-24T00:33:34.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 10 client.6183 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:33:34.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 20 client.6183 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:34.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55d65fd4b970 0x55d65fd9c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:34.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 4 client.6183 mds_command: new command op to 5763 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:33:34.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.759+0000 7f2355d5c000 1 -- 172.21.15.40:0/2969820961 --> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d65fb48bc0 con 0x55d65fd4b970 2024-03-24T00:33:34.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2343e00640 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55d65fd4b970 0x55d65fd9c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:34.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2338c00640 10 client.6183 ms_handle_connect on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2343e00640 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55d65fd4b970 0x55d65fd9c3b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e8052243b22360e server_cookie=d0097f677ef4304d in_seq=0 out_seq=0 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2338c00640 1 -- 172.21.15.40:0/2969820961 <== mds.0 v2:172.21.15.40:6832/224223662 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55d65fb48bc0 con 0x55d65fd4b970 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2338c00640 10 client.6183 handle_command_reply: tid=0 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2355d5c000 1 client.6183 shutdown 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2355d5c000 1 -- 172.21.15.40:0/2969820961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fcedff0 msgr2=0x55d65fd1c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2355d5c000 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fcedff0 0x55d65fd1c200 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f233c0047f0 tx=0x7f233c00a000 comp rx=0 tx=0).stop 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2355d5c000 1 -- 172.21.15.40:0/2969820961 shutdown_connections 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2355d5c000 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55d65fd4b970 0x55d65fd9c3b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2355d5c000 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d65fd38240 0x55d65fd4b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2355d5c000 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d65fd37920 0x55d65fd4aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2355d5c000 1 --2- 172.21.15.40:0/2969820961 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d65fcedff0 0x55d65fd1c200 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2355d5c000 1 -- 172.21.15.40:0/2969820961 >> 172.21.15.40:0/2969820961 conn(0x55d65fce9fa0 msgr2=0x55d65fd36d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2338c00640 0 client.6183 ms_handle_reset on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2355d5c000 1 -- 172.21.15.40:0/2969820961 shutdown_connections 2024-03-24T00:33:34.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2355d5c000 1 -- 172.21.15.40:0/2969820961 wait complete. 2024-03-24T00:33:34.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2355d5c000 20 client.6183 trim_cache size 0 max 16384 2024-03-24T00:33:34.769 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:34.769 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:33:34.769 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:33:34.769 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 5763, 2024-03-24T00:33:34.769 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:33:34.769 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:33:34.769 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:33:34.770 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 37.123713174999999, 2024-03-24T00:33:34.770 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 42, 2024-03-24T00:33:34.770 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 45, 2024-03-24T00:33:34.770 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 45, 2024-03-24T00:33:34.770 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 41.959210134999999 2024-03-24T00:33:34.770 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:34.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2352800640 1 -- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2328067660 msgr2=0x7f2328069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2352800640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2328067660 0x7f2328069b20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f234c06d130 tx=0x7f233c007e90 comp rx=0 tx=0).stop 2024-03-24T00:33:34.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2352800640 1 -- 172.21.15.40:0/2735371701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f234c13ec80 msgr2=0x7f234c13d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:34.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2352800640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f234c13ec80 0x7f234c13d3a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f233400e980 tx=0x7f233400ee40 comp rx=0 tx=0).stop 2024-03-24T00:33:34.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2352800640 1 -- 172.21.15.40:0/2735371701 shutdown_connections 2024-03-24T00:33:34.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2352800640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2328067660 0x7f2328069b20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2352800640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f234c141bc0 0x7f234c155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2352800640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f234c13ec80 0x7f234c13d3a0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2352800640 1 --2- 172.21.15.40:0/2735371701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f234c0ce630 0x7f234c1f6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:34.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2352800640 1 -- 172.21.15.40:0/2735371701 >> 172.21.15.40:0/2735371701 conn(0x7f234c0b17b0 msgr2=0x7f234c0b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:34.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2352800640 1 -- 172.21.15.40:0/2735371701 shutdown_connections 2024-03-24T00:33:34.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:34.763+0000 7f2352800640 1 -- 172.21.15.40:0/2735371701 wait complete. 2024-03-24T00:33:37.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:33:37.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:33:44.786 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:44.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8dfd600640 1 Processor -- start 2024-03-24T00:33:44.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8dfd600640 1 -- start start 2024-03-24T00:33:44.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8dfd600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80d03a0 0x7f8df80cf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:44.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8dfd600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8df80b2c90 0x7f8df80cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:44.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8dfd600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df80ceed0 con 0x7f8df80d0780 2024-03-24T00:33:44.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8dfd600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df80d2650 con 0x7f8df80b2c90 2024-03-24T00:33:44.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8dfd600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df80d2930 con 0x7f8df80d03a0 2024-03-24T00:33:44.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8def400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8df80d0780 0x7f8df80b49b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57658/0 (socket says 172.21.15.40:57658) 2024-03-24T00:33:44.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8def400640 1 -- 172.21.15.40:0/163570893 learned_addr learned my addr 172.21.15.40:0/163570893 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:44.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8df7e00640 1 --2- 172.21.15.40:0/163570893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80d03a0 0x7f8df80cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:44.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8dfcc00640 1 --2- 172.21.15.40:0/163570893 >> v2:172.21.15.53:3300/0 conn(0x7f8df80b2c90 0x7f8df80cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:44.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8df7e00640 1 -- 172.21.15.40:0/163570893 >> v2:172.21.15.53:3300/0 conn(0x7f8df80b2c90 msgr2=0x7f8df80cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:44.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8df7e00640 1 --2- 172.21.15.40:0/163570893 >> v2:172.21.15.53:3300/0 conn(0x7f8df80b2c90 0x7f8df80cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:44.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8df7e00640 1 -- 172.21.15.40:0/163570893 >> v1:172.21.15.40:6789/0 conn(0x7f8df80d0780 legacy=0x7f8df80b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:44.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8df7400640 1 -- 172.21.15.40:0/163570893 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600991343 0 0) 0x7f8df80ceed0 con 0x7f8df80d0780 2024-03-24T00:33:44.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8df7e00640 1 -- 172.21.15.40:0/163570893 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df80ccac0 con 0x7f8df80d03a0 2024-03-24T00:33:44.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.927+0000 7f8dfcc00640 1 --2- 172.21.15.40:0/163570893 >> v2:172.21.15.53:3300/0 conn(0x7f8df80b2c90 0x7f8df80cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:44.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.931+0000 7f8df7e00640 1 --2- 172.21.15.40:0/163570893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80d03a0 0x7f8df80cf940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009d90 tx=0x7f8de801c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23070af93aa73e34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:44.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.931+0000 7f8df7400640 1 -- 172.21.15.40:0/163570893 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de80b2070 con 0x7f8df80d03a0 2024-03-24T00:33:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.931+0000 7f8df7400640 1 -- 172.21.15.40:0/163570893 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de80a58b0 con 0x7f8df80d03a0 2024-03-24T00:33:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.931+0000 7f8df7400640 1 -- 172.21.15.40:0/163570893 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de80a5c80 con 0x7f8df80d03a0 2024-03-24T00:33:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.931+0000 7f8dfd600640 1 -- 172.21.15.40:0/163570893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80d03a0 msgr2=0x7f8df80cf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.931+0000 7f8dfd600640 1 --2- 172.21.15.40:0/163570893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80d03a0 0x7f8df80cf940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8de8009d90 tx=0x7f8de801c8e0 comp rx=0 tx=0).stop 2024-03-24T00:33:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.931+0000 7f8dfd600640 1 -- 172.21.15.40:0/163570893 shutdown_connections 2024-03-24T00:33:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.931+0000 7f8dfd600640 1 --2- 172.21.15.40:0/163570893 >> v2:172.21.15.53:3300/0 conn(0x7f8df80b2c90 0x7f8df80cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.931+0000 7f8dfd600640 1 --2- 172.21.15.40:0/163570893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80d03a0 0x7f8df80cf940 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.931+0000 7f8dfd600640 1 -- 172.21.15.40:0/163570893 >> 172.21.15.40:0/163570893 conn(0x7f8df80d1b60 msgr2=0x7f8df80d1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.931+0000 7f8dfd600640 1 -- 172.21.15.40:0/163570893 shutdown_connections 2024-03-24T00:33:44.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.931+0000 7f8dfd600640 1 -- 172.21.15.40:0/163570893 wait complete. 2024-03-24T00:33:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8dfd600640 1 Processor -- start 2024-03-24T00:33:44.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8dfd600640 1 -- start start 2024-03-24T00:33:44.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8dfd600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80b2c90 0x7f8df815acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:44.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8dfd600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8df80d03a0 0x7f8df8144990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:44.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8dfd600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8df80d0780 0x7f8df8144ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8dfd600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df815c4e0 con 0x7f8df80d0780 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8dfd600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df815c360 con 0x7f8df80d03a0 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8dfd600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df815c660 con 0x7f8df80b2c90 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8dfcc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8df80d0780 0x7f8df8144ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8df7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80b2c90 0x7f8df815acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8dfcc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8df80d0780 0x7f8df8144ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48764/0 (socket says 172.21.15.40:48764) 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8dfcc00640 1 -- 172.21.15.40:0/3760117359 learned_addr learned my addr 172.21.15.40:0/3760117359 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8df7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80b2c90 0x7f8df815acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:48468/0 (socket says 172.21.15.40:48468) 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8def400640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8df80d03a0 0x7f8df8144990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8df7e00640 1 -- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8df80d03a0 msgr2=0x7f8df8144990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8df7e00640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8df80d03a0 0x7f8df8144990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8df7e00640 1 -- 172.21.15.40:0/3760117359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8df80d0780 msgr2=0x7f8df8144ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8df7e00640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8df80d0780 0x7f8df8144ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8df7e00640 1 -- 172.21.15.40:0/3760117359 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df806bef0 con 0x7f8df80b2c90 2024-03-24T00:33:44.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8df7e00640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80b2c90 0x7f8df815acd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8de80ac000 tx=0x7f8de8019680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8df4c00640 1 -- 172.21.15.40:0/3760117359 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de80bd020 con 0x7f8df80b2c90 2024-03-24T00:33:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8df4c00640 1 -- 172.21.15.40:0/3760117359 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de80b3040 con 0x7f8df80b2c90 2024-03-24T00:33:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.935+0000 7f8dfd600640 1 -- 172.21.15.40:0/3760117359 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df806c1d0 con 0x7f8df80b2c90 2024-03-24T00:33:44.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.939+0000 7f8df4c00640 1 -- 172.21.15.40:0/3760117359 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de80b2050 con 0x7f8df80b2c90 2024-03-24T00:33:44.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.939+0000 7f8dfd600640 1 -- 172.21.15.40:0/3760117359 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df806c790 con 0x7f8df80b2c90 2024-03-24T00:33:44.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.939+0000 7f8dfd600640 1 -- 172.21.15.40:0/3760117359 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df809fba0 con 0x7f8df80b2c90 2024-03-24T00:33:44.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.939+0000 7f8df4c00640 1 -- 172.21.15.40:0/3760117359 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8de80c0070 con 0x7f8df80b2c90 2024-03-24T00:33:44.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.939+0000 7f8df4c00640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8dd4067660 0x7f8dd4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:44.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.939+0000 7f8df4c00640 1 -- 172.21.15.40:0/3760117359 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8de811fb20 con 0x7f8df80b2c90 2024-03-24T00:33:44.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.939+0000 7f8def400640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8dd4067660 0x7f8dd4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:44.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.943+0000 7f8def400640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8dd4067660 0x7f8dd4069b20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8df809fb70 tx=0x7f8de0007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:44.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:44.947+0000 7f8df4c00640 1 -- 172.21.15.40:0/3760117359 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8de8016560 con 0x7f8df80b2c90 2024-03-24T00:33:45.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.143+0000 7f8dfd600640 1 -- 172.21.15.40:0/3760117359 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8df804f4f0 con 0x7f8df80b2c90 2024-03-24T00:33:45.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.143+0000 7f8df4c00640 1 -- 172.21.15.40:0/3760117359 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f8de80f0060 con 0x7f8df80b2c90 2024-03-24T00:33:45.148 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:45.148 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":42,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/374985095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":374985095},{"type":"v1","addr":"172.21.15.53:6836","nonce":374985095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:31.903823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5763,"mds_1":5192,"mds_2":5198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"c","rank":1,"incarnation":38,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5198":{"gid":5198,"name":"b","rank":2,"incarnation":34,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5192,5198,5763]},"id":3}]} 2024-03-24T00:33:45.149 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 42 2024-03-24T00:33:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.147+0000 7f8dfd600640 1 -- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8dd4067660 msgr2=0x7f8dd4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.147+0000 7f8dfd600640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8dd4067660 0x7f8dd4069b20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8df809fb70 tx=0x7f8de0007df0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.147+0000 7f8dfd600640 1 -- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80b2c90 msgr2=0x7f8df815acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.147+0000 7f8dfd600640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80b2c90 0x7f8df815acd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8de80ac000 tx=0x7f8de8019680 comp rx=0 tx=0).stop 2024-03-24T00:33:45.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.147+0000 7f8dfd600640 1 -- 172.21.15.40:0/3760117359 shutdown_connections 2024-03-24T00:33:45.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.147+0000 7f8dfd600640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8dd4067660 0x7f8dd4069b20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.147+0000 7f8dfd600640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8df80d0780 0x7f8df8144ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.147+0000 7f8dfd600640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8df80d03a0 0x7f8df8144990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.147+0000 7f8dfd600640 1 --2- 172.21.15.40:0/3760117359 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8df80b2c90 0x7f8df815acd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.147+0000 7f8dfd600640 1 -- 172.21.15.40:0/3760117359 >> 172.21.15.40:0/3760117359 conn(0x7f8df80d1b60 msgr2=0x7f8df8070110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.147+0000 7f8dfd600640 1 -- 172.21.15.40:0/3760117359 shutdown_connections 2024-03-24T00:33:45.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.147+0000 7f8dfd600640 1 -- 172.21.15.40:0/3760117359 wait complete. 2024-03-24T00:33:45.169 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:33:45.170 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:1 perf dump 2024-03-24T00:33:45.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc7a00640 1 Processor -- start 2024-03-24T00:33:45.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc7a00640 1 -- start start 2024-03-24T00:33:45.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc7a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbbc015b960 0x7fbbc0070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc7a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc015bd40 0x7fbbc00712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc7a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc015f970 con 0x7fbbc00d0fe0 2024-03-24T00:33:45.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc7a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc0160580 con 0x7fbbc015b960 2024-03-24T00:33:45.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc7a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc015fc50 con 0x7fbbc015bd40 2024-03-24T00:33:45.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc5000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbbc015b960 0x7fbbc0070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc5000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbbc015b960 0x7fbbc0070d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48780/0 (socket says 172.21.15.40:48780) 2024-03-24T00:33:45.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc5000640 1 -- 172.21.15.40:0/1320611226 learned_addr learned my addr 172.21.15.40:0/1320611226 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/1320611226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc015bd40 0x7fbbc00712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc5000640 1 -- 172.21.15.40:0/1320611226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc015bd40 msgr2=0x7fbbc00712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc5000640 1 --2- 172.21.15.40:0/1320611226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc015bd40 0x7fbbc00712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc5000640 1 -- 172.21.15.40:0/1320611226 >> v1:172.21.15.53:6789/0 conn(0x7fbbc00d0fe0 legacy=0x7fbbc00cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.311+0000 7fbbc5000640 1 -- 172.21.15.40:0/1320611226 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc015ff30 con 0x7fbbc015b960 2024-03-24T00:33:45.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.315+0000 7fbbc5000640 1 --2- 172.21.15.40:0/1320611226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc015b960 0x7fbbc0070d60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009870 tx=0x7fbbb401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f3e8756676a3e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.315+0000 7fbbbf400640 1 -- 172.21.15.40:0/1320611226 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb40b2070 con 0x7fbbc015b960 2024-03-24T00:33:45.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.315+0000 7fbbbf400640 1 -- 172.21.15.40:0/1320611226 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb40a9070 con 0x7fbbc015b960 2024-03-24T00:33:45.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.315+0000 7fbbbf400640 1 -- 172.21.15.40:0/1320611226 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb40ad880 con 0x7fbbc015b960 2024-03-24T00:33:45.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.315+0000 7fbbc7a00640 1 -- 172.21.15.40:0/1320611226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc015b960 msgr2=0x7fbbc0070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.315+0000 7fbbc7a00640 1 --2- 172.21.15.40:0/1320611226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc015b960 0x7fbbc0070d60 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009870 tx=0x7fbbb401c560 comp rx=0 tx=0).stop 2024-03-24T00:33:45.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.315+0000 7fbbc7a00640 1 -- 172.21.15.40:0/1320611226 shutdown_connections 2024-03-24T00:33:45.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.315+0000 7fbbc7a00640 1 --2- 172.21.15.40:0/1320611226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc015bd40 0x7fbbc00712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.315+0000 7fbbc7a00640 1 --2- 172.21.15.40:0/1320611226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc015b960 0x7fbbc0070d60 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.315+0000 7fbbc7a00640 1 -- 172.21.15.40:0/1320611226 >> 172.21.15.40:0/1320611226 conn(0x7fbbc006ecf0 msgr2=0x7fbbc006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.315+0000 7fbbc7a00640 1 -- 172.21.15.40:0/1320611226 shutdown_connections 2024-03-24T00:33:45.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.315+0000 7fbbc7a00640 1 -- 172.21.15.40:0/1320611226 wait complete. 2024-03-24T00:33:45.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc7a00640 1 Processor -- start 2024-03-24T00:33:45.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc7a00640 1 -- start start 2024-03-24T00:33:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc7a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc00d0fe0 0x7fbbc006c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc7a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc015b960 0x7fbbc0147d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc7a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc015bd40 0x7fbbc00c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc7a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc0168f00 con 0x7fbbc00d0fe0 2024-03-24T00:33:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc7a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc0168d80 con 0x7fbbc015bd40 2024-03-24T00:33:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc7a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc0169080 con 0x7fbbc015b960 2024-03-24T00:33:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc00d0fe0 0x7fbbc006c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc00d0fe0 0x7fbbc006c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48790/0 (socket says 172.21.15.40:48790) 2024-03-24T00:33:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc5000640 1 -- 172.21.15.40:0/773211413 learned_addr learned my addr 172.21.15.40:0/773211413 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc015b960 0x7fbbc0147d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc5a00640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc015bd40 0x7fbbc00c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc5000640 1 -- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc015b960 msgr2=0x7fbbc0147d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc5000640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc015b960 0x7fbbc0147d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc5000640 1 -- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc015bd40 msgr2=0x7fbbc00c5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc5000640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc015bd40 0x7fbbc00c5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc5000640 1 -- 172.21.15.40:0/773211413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc00c6720 con 0x7fbbc00d0fe0 2024-03-24T00:33:45.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc5000640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc00d0fe0 0x7fbbc006c500 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbbb401a190 tx=0x7fbbb401ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbbcc00640 1 -- 172.21.15.40:0/773211413 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb40bc020 con 0x7fbbc00d0fe0 2024-03-24T00:33:45.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbbcc00640 1 -- 172.21.15.40:0/773211413 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb40b8420 con 0x7fbbc00d0fe0 2024-03-24T00:33:45.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc7a00640 1 -- 172.21.15.40:0/773211413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc00c7f30 con 0x7fbbc00d0fe0 2024-03-24T00:33:45.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbbcc00640 1 -- 172.21.15.40:0/773211413 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb40b2040 con 0x7fbbc00d0fe0 2024-03-24T00:33:45.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.319+0000 7fbbc7a00640 1 -- 172.21.15.40:0/773211413 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc00c6440 con 0x7fbbc00d0fe0 2024-03-24T00:33:45.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.323+0000 7fbbbcc00640 1 -- 172.21.15.40:0/773211413 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbbb40bf020 con 0x7fbbc00d0fe0 2024-03-24T00:33:45.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.323+0000 7fbbcaf9e000 1 Processor -- start 2024-03-24T00:33:45.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.323+0000 7fbbbcc00640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbba0067660 0x7fbba0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.323+0000 7fbbcaf9e000 1 -- start start 2024-03-24T00:33:45.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.323+0000 7fbbbcc00640 1 -- 172.21.15.40:0/773211413 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbbb411fbd0 con 0x7fbbc00d0fe0 2024-03-24T00:33:45.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.323+0000 7fbbc5a00640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbba0067660 0x7fbba0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e5f970 0x561e55e5fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e15e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e16380 0x561e55e16760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e55c64680 con 0x561e55e16380 2024-03-24T00:33:45.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e55c01b70 con 0x561e55e60290 2024-03-24T00:33:45.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561e55c60720 con 0x561e55e5f970 2024-03-24T00:33:45.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbbfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e15e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbc5000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e5f970 0x561e55e5fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbbfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e15e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:41034/0 (socket says 172.21.15.40:41034) 2024-03-24T00:33:45.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbc5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e16380 0x561e55e16760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbbfe00640 1 -- 172.21.15.40:0/3398404339 learned_addr learned my addr 172.21.15.40:0/3398404339 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbc5a00640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbba0067660 0x7fbba0069b20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80012f0 tx=0x7fbbb8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbbfe00640 1 -- 172.21.15.40:0/3398404339 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e5f970 msgr2=0x561e55e5fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/3398404339 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e5f970 0x561e55e5fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbbfe00640 1 -- 172.21.15.40:0/3398404339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e16380 msgr2=0x561e55e16760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/3398404339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e16380 0x561e55e16760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbbfe00640 1 -- 172.21.15.40:0/3398404339 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e55c01dc0 con 0x561e55e60290 2024-03-24T00:33:45.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/3398404339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e15e40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009ae0 tx=0x7fbbac0112c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e22ffa47eef6334 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbb1600640 1 -- 172.21.15.40:0/3398404339 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac019070 con 0x561e55e60290 2024-03-24T00:33:45.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbb1600640 1 -- 172.21.15.40:0/3398404339 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbac011d00 con 0x561e55e60290 2024-03-24T00:33:45.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbb1600640 1 -- 172.21.15.40:0/3398404339 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac004230 con 0x561e55e60290 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3398404339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 msgr2=0x561e55e15e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/3398404339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e15e40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009ae0 tx=0x7fbbac0112c0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3398404339 shutdown_connections 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/3398404339 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e16380 0x561e55e16760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/3398404339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e15e40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/3398404339 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e5f970 0x561e55e5fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3398404339 >> 172.21.15.40:0/3398404339 conn(0x561e55e12330 msgr2=0x561e55e14720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3398404339 shutdown_connections 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3398404339 wait complete. 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 -- start start 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.327+0000 7fbbcaf9e000 1 Processor -- start 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e16380 0x561e55e1b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e5f970 0x561e55e20930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e20e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e55d2fc80 con 0x561e55e5f970 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e55c223f0 con 0x561e55e60290 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561e55e1f0f0 con 0x561e55e16380 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 10 client.0 _ll_register_callbacks cb 0x561e55e01680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e5f970 0x561e55e20930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbc5000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e16380 0x561e55e1b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e5f970 0x561e55e20930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48802/0 (socket says 172.21.15.40:48802) 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbbfe00640 1 -- 172.21.15.40:0/3117781322 learned_addr learned my addr 172.21.15.40:0/3117781322 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbc5a00640 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e20e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbbfe00640 1 -- 172.21.15.40:0/3117781322 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e16380 msgr2=0x561e55e1b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e16380 0x561e55e1b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbbfe00640 1 -- 172.21.15.40:0/3117781322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 msgr2=0x561e55e20e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e20e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbbfe00640 1 -- 172.21.15.40:0/3117781322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e55c256e0 con 0x561e55e5f970 2024-03-24T00:33:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbc5000640 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e16380 0x561e55e1b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e5f970 0x561e55e20930 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x561e559c8460 tx=0x7fbbac004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12701f9d3496cef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbba6000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbba6000640 1 -- 172.21.15.40:0/3117781322 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac019040 con 0x561e55e5f970 2024-03-24T00:33:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbba6000640 1 -- 172.21.15.40:0/3117781322 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbbac003830 con 0x561e55e5f970 2024-03-24T00:33:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3117781322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbbac011d00 con 0x561e55e5f970 2024-03-24T00:33:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbba6000640 1 -- 172.21.15.40:0/3117781322 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbac02f050 con 0x561e55e5f970 2024-03-24T00:33:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 10 client.6201 fetch_fsmap learned FSMap version 42 2024-03-24T00:33:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3117781322 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x561e55c1f4a0 con 0x561e55e5f970 2024-03-24T00:33:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbba6000640 1 -- 172.21.15.40:0/3117781322 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbac011d00 con 0x561e55e5f970 2024-03-24T00:33:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbba6000640 1 -- 172.21.15.40:0/3117781322 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) v1 ==== 3362+0+0 (secure 0 0 0) 0x7fbbac032020 con 0x561e55e5f970 2024-03-24T00:33:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 10 client.6201 fetch_fsmap finished waiting for FSMap version 42 2024-03-24T00:33:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 10 client.6201 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.c 2024-03-24T00:33:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 20 client.6201 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x561e55e65fb0 0x561e55e663c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 4 client.6201 mds_command: new command op to 5192 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3117781322 --> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e55bffcf0 con 0x561e55e65fb0 2024-03-24T00:33:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbc5000640 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x561e55e65fb0 0x561e55e663c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbbc5000640 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x561e55e65fb0 0x561e55e663c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=376d47757128849e server_cookie=b553720c9d7e244c in_seq=0 out_seq=0 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.331+0000 7fbba6000640 10 client.6201 ms_handle_connect on v2:172.21.15.40:6833/2128553323 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbba6000640 1 -- 172.21.15.40:0/3117781322 <== mds.1 v2:172.21.15.40:6833/2128553323 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561e55bffcf0 con 0x561e55e65fb0 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbba6000640 10 client.6201 handle_command_reply: tid=0 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbbcaf9e000 1 client.6201 shutdown 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3117781322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e5f970 msgr2=0x561e55e20930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e5f970 0x561e55e20930 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x561e559c8460 tx=0x7fbbac004740 comp rx=0 tx=0).stop 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3117781322 shutdown_connections 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x561e55e65fb0 0x561e55e663c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e20e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e5f970 0x561e55e20930 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/3117781322 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e16380 0x561e55e1b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3117781322 >> 172.21.15.40:0/3117781322 conn(0x561e55e12330 msgr2=0x561e55e14fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbba6000640 0 client.6201 ms_handle_reset on v2:172.21.15.40:6833/2128553323 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3117781322 shutdown_connections 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/3117781322 wait complete. 2024-03-24T00:33:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.335+0000 7fbbcaf9e000 20 client.6201 trim_cache size 0 max 16384 2024-03-24T00:33:45.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbcaf9e000 1 Processor -- start 2024-03-24T00:33:45.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbcaf9e000 1 -- start start 2024-03-24T00:33:45.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbcaf9e000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e16380 0x561e55e15730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbcaf9e000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e5f970 0x561e55e15c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbcaf9e000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e60290 0x561e55e75770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbcaf9e000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc0203b90 con 0x561e55e60290 2024-03-24T00:33:45.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbcaf9e000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc015ab70 con 0x561e55e5f970 2024-03-24T00:33:45.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbcaf9e000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc02034d0 con 0x561e55e16380 2024-03-24T00:33:45.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbbfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e5f970 0x561e55e15c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbc5000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e16380 0x561e55e15730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbc5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e60290 0x561e55e75770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbbfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e5f970 0x561e55e15c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:41046/0 (socket says 172.21.15.40:41046) 2024-03-24T00:33:45.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbbfe00640 1 -- 172.21.15.40:0/544991583 learned_addr learned my addr 172.21.15.40:0/544991583 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbbfe00640 1 -- 172.21.15.40:0/544991583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e16380 msgr2=0x561e55e15730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/544991583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e16380 0x561e55e15730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbbfe00640 1 -- 172.21.15.40:0/544991583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e60290 msgr2=0x561e55e75770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/544991583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e60290 0x561e55e75770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbbfe00640 1 -- 172.21.15.40:0/544991583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e55e600c0 con 0x561e55e5f970 2024-03-24T00:33:45.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbc5000640 1 --2- 172.21.15.40:0/544991583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e16380 0x561e55e15730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:45.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/544991583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e5f970 0x561e55e15c70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbbac029d50 tx=0x7fbbac024000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de3f55b4a72dd9c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbb0c00640 1 -- 172.21.15.40:0/544991583 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac019040 con 0x561e55e5f970 2024-03-24T00:33:45.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbb0c00640 1 -- 172.21.15.40:0/544991583 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbac000e50 con 0x561e55e5f970 2024-03-24T00:33:45.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.363+0000 7fbbb0c00640 1 -- 172.21.15.40:0/544991583 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac022040 con 0x561e55e5f970 2024-03-24T00:33:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/544991583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e5f970 msgr2=0x561e55e15c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/544991583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e5f970 0x561e55e15c70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbbac029d50 tx=0x7fbbac024000 comp rx=0 tx=0).stop 2024-03-24T00:33:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/544991583 shutdown_connections 2024-03-24T00:33:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/544991583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e60290 0x561e55e75770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/544991583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e5f970 0x561e55e15c70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/544991583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e16380 0x561e55e15730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/544991583 >> 172.21.15.40:0/544991583 conn(0x561e55e12330 msgr2=0x561e55e75060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/544991583 shutdown_connections 2024-03-24T00:33:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/544991583 wait complete. 2024-03-24T00:33:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 -- start start 2024-03-24T00:33:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 Processor -- start 2024-03-24T00:33:45.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e16380 0x561e55e6a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e5f970 0x561e55e6a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e6ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e55e121b0 con 0x561e55e16380 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc00c9a90 con 0x561e55e60290 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561e55d2fc80 con 0x561e55e5f970 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 10 client.0 _ll_register_callbacks cb 0x7fbbb8001b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbc5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e16380 0x561e55e6a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbc5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e16380 0x561e55e6a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48820/0 (socket says 172.21.15.40:48820) 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbc5000640 1 -- 172.21.15.40:0/1955693763 learned_addr learned my addr 172.21.15.40:0/1955693763 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e5f970 0x561e55e6a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbc5a00640 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e6ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbc5000640 1 -- 172.21.15.40:0/1955693763 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e5f970 msgr2=0x561e55e6a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbc5000640 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e5f970 0x561e55e6a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbc5000640 1 -- 172.21.15.40:0/1955693763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 msgr2=0x561e55e6ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbc5000640 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e6ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbc5000640 1 -- 172.21.15.40:0/1955693763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e55ce58b0 con 0x561e55e16380 2024-03-24T00:33:45.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbc5000640 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e16380 0x561e55e6a2e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fbbb401ca40 tx=0x7fbbb40a4f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f341688bc96ba820 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbba6000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:45.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbba6000640 1 -- 172.21.15.40:0/1955693763 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb410e440 con 0x561e55e16380 2024-03-24T00:33:45.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbba6000640 1 -- 172.21.15.40:0/1955693763 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbbb40ef5e0 con 0x561e55e16380 2024-03-24T00:33:45.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/1955693763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbbac000e50 con 0x561e55e16380 2024-03-24T00:33:45.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbba6000640 1 -- 172.21.15.40:0/1955693763 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbb410e5e0 con 0x561e55e16380 2024-03-24T00:33:45.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 10 client.6207 fetch_fsmap learned FSMap version 42 2024-03-24T00:33:45.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/1955693763 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x561e55cbcfd0 con 0x561e55e16380 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbba6000640 1 -- 172.21.15.40:0/1955693763 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbac000e50 con 0x561e55e16380 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.367+0000 7fbba6000640 1 -- 172.21.15.40:0/1955693763 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) v1 ==== 3362+0+0 (secure 0 0 0) 0x7fbbb40b5030 con 0x561e55e16380 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 10 client.6207 fetch_fsmap finished waiting for FSMap version 42 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 10 client.6207 resolve_mds: resolved 3:1 to role '3:1' aka daemon mds.c 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 20 client.6207 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x561e55e6b720 0x561e55ecbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 4 client.6207 mds_command: new command op to 5192 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/1955693763 --> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x561e55c71bc0 con 0x561e55e6b720 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x561e55e6b720 0x561e55ecbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbbfe00640 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x561e55e6b720 0x561e55ecbe50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=90a76d0e7d36519d server_cookie=472d0db02108547b in_seq=0 out_seq=0 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbba6000640 10 client.6207 ms_handle_connect on v2:172.21.15.40:6833/2128553323 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbba6000640 1 -- 172.21.15.40:0/1955693763 <== mds.1 v2:172.21.15.40:6833/2128553323 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17636 (crc 0 0 0) 0x561e55c71bc0 con 0x561e55e6b720 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbba6000640 10 client.6207 handle_command_reply: tid=0 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 client.6207 shutdown 2024-03-24T00:33:45.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/1955693763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e16380 msgr2=0x561e55e6a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e16380 0x561e55e6a2e0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fbbb401ca40 tx=0x7fbbb40a4f90 comp rx=0 tx=0).stop 2024-03-24T00:33:45.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/1955693763 shutdown_connections 2024-03-24T00:33:45.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.40:6833/2128553323,v1:172.21.15.40:6835/2128553323] conn(0x561e55e6b720 0x561e55ecbe50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e55e60290 0x561e55e6ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561e55e5f970 0x561e55e6a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbba6000640 0 client.6207 ms_handle_reset on v2:172.21.15.40:6833/2128553323 2024-03-24T00:33:45.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 --2- 172.21.15.40:0/1955693763 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561e55e16380 0x561e55e6a2e0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/1955693763 >> 172.21.15.40:0/1955693763 conn(0x561e55e12330 msgr2=0x561e55e758f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/1955693763 shutdown_connections 2024-03-24T00:33:45.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 1 -- 172.21.15.40:0/1955693763 wait complete. 2024-03-24T00:33:45.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbcaf9e000 20 client.6207 trim_cache size 0 max 16384 2024-03-24T00:33:45.377 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:45.378 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:33:45.378 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 21, 2024-03-24T00:33:45.378 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 22, 2024-03-24T00:33:45.378 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 21366, 2024-03-24T00:33:45.378 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 7366, 2024-03-24T00:33:45.378 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 10, 2024-03-24T00:33:45.378 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-03-24T00:33:45.378 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.010525358, 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000803897, 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.004632346, 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000476964, 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 21, 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.005326858, 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000253659 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 4, 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000839, 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000209 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.379 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 73, 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 61, 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 139609, 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 71139, 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 9, 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 7, 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.014874735, 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.002905688, 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.013061310, 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000726596, 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:45.380 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 61, 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.016499977, 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000270491 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 33, 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000004841, 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000146 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 115712, 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 12736 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 16, 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 29, 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 10656, 2024-03-24T00:33:45.381 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 74907, 2024-03-24T00:33:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 6, 2024-03-24T00:33:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-03-24T00:33:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.005462142, 2024-03-24T00:33:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.001306548, 2024-03-24T00:33:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.003816627, 2024-03-24T00:33:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000198824, 2024-03-24T00:33:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 27, 2024-03-24T00:33:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.008445605, 2024-03-24T00:33:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000312800 2024-03-24T00:33:45.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 8, 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000001267, 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000158 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 65808 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-24T00:33:45.383 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 44, 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.003941453, 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000089578 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 4, 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000103184, 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000025796 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 0, 2024-03-24T00:33:45.384 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 0, 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 11, 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 10, 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 0, 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 25, 2024-03-24T00:33:45.385 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 7, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 18, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 25, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 0, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 0, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 0, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 3, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 0, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 0, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 0, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-03-24T00:33:45.386 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 100, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 500, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-03-24T00:33:45.387 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 0, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 0, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 0, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 1, 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 0 2024-03-24T00:33:45.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-03-24T00:33:45.389 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 0, 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 1, 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-03-24T00:33:45.390 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 5, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 0, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 2, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 2, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 4194304, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 4206311, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 4206134, 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-03-24T00:33:45.391 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1, 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.002256145, 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002256145 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 4 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 28, 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 28, 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 0, 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 20, 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 20, 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 0, 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 25, 2024-03-24T00:33:45.392 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 25, 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 0, 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 0, 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 7, 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 7, 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 42664, 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 239920 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 0, 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 0, 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 0, 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-03-24T00:33:45.393 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-03-24T00:33:45.394 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.395 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.396 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-03-24T00:33:45.397 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.398 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.402 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.403 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 1, 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 1, 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 0, 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 1, 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 0, 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 0, 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 896, 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-24T00:33:45.404 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:33:45.405 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-24T00:33:45.406 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 245047, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 70, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:33:45.407 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 5544, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 90, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 103432, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 355, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-03-24T00:33:45.408 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 42, 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 756, 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 0, 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 42, 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 42, 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.055999515, 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001333321 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 42, 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 87 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.409 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 42, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 33, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 9, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 3, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 0, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 6, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 1, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 1, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 1, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2024-03-24T00:33:45.410 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 5, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 22, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-03-24T00:33:45.411 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-03-24T00:33:45.412 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 0, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 0, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 7, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 7, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 1, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 46, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 1 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 0, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-03-24T00:33:45.413 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 0, 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.414 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 105, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 158771, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 105, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 105, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 158771, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.415 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 42, 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 12399, 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 42, 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 12399, 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.416 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 42, 2024-03-24T00:33:45.417 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 42, 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 42, 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 42, 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:45.418 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x55effcbbca00": { 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.419 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 1, 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 116, 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 1, 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 1, 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 116, 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.420 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:45.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbc7a00640 1 -- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbba0067660 msgr2=0x7fbba0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbc7a00640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbba0067660 0x7fbba0069b20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80012f0 tx=0x7fbbb8009040 comp rx=0 tx=0).stop 2024-03-24T00:33:45.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbc7a00640 1 -- 172.21.15.40:0/773211413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc00d0fe0 msgr2=0x7fbbc006c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbc7a00640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc00d0fe0 0x7fbbc006c500 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbbb401a190 tx=0x7fbbb401ada0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbc7a00640 1 -- 172.21.15.40:0/773211413 shutdown_connections 2024-03-24T00:33:45.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbc7a00640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbba0067660 0x7fbba0069b20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbc7a00640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc015bd40 0x7fbbc00c5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbc7a00640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc015b960 0x7fbbc0147d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbc7a00640 1 --2- 172.21.15.40:0/773211413 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbc00d0fe0 0x7fbbc006c500 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbc7a00640 1 -- 172.21.15.40:0/773211413 >> 172.21.15.40:0/773211413 conn(0x7fbbc006ecf0 msgr2=0x7fbbc015c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbc7a00640 1 -- 172.21.15.40:0/773211413 shutdown_connections 2024-03-24T00:33:45.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.371+0000 7fbbc7a00640 1 -- 172.21.15.40:0/773211413 wait complete. 2024-03-24T00:33:45.423 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:33:45.423 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:2 perf dump 2024-03-24T00:33:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.571+0000 7f9b01200640 1 Processor -- start 2024-03-24T00:33:45.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.571+0000 7f9b01200640 1 -- start start 2024-03-24T00:33:45.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9b01200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9afc15ffd0 0x7f9afc0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9b01200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9afc0728d0 0x7f9afc0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9b01200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9afc072d40 con 0x7f9afc15bd00 2024-03-24T00:33:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9b01200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9afc0cf790 con 0x7f9afc15ffd0 2024-03-24T00:33:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9b01200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc071de0 con 0x7f9afc0728d0 2024-03-24T00:33:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9af9e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9afc15bd00 0x7f9afc15f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57666/0 (socket says 172.21.15.40:57666) 2024-03-24T00:33:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9af9e00640 1 -- 172.21.15.40:0/2257801963 learned_addr learned my addr 172.21.15.40:0/2257801963 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9af3e00640 1 --2- 172.21.15.40:0/2257801963 >> v2:172.21.15.53:3300/0 conn(0x7f9afc15ffd0 0x7f9afc0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9afa800640 1 --2- 172.21.15.40:0/2257801963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9afc0728d0 0x7f9afc0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9af3e00640 1 -- 172.21.15.40:0/2257801963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9afc0728d0 msgr2=0x7f9afc0cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9af3e00640 1 --2- 172.21.15.40:0/2257801963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9afc0728d0 0x7f9afc0cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9af9400640 1 -- 172.21.15.40:0/2257801963 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475243821 0 0) 0x7f9afc072d40 con 0x7f9afc15bd00 2024-03-24T00:33:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9af3e00640 1 -- 172.21.15.40:0/2257801963 >> v1:172.21.15.40:6789/0 conn(0x7f9afc15bd00 legacy=0x7f9afc15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.575+0000 7f9af3e00640 1 -- 172.21.15.40:0/2257801963 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc0cea50 con 0x7f9afc15ffd0 2024-03-24T00:33:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.579+0000 7f9af3e00640 1 --2- 172.21.15.40:0/2257801963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9afc15ffd0 0x7f9afc0ae8e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40097c0 tx=0x7f9ae401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b440718714c56ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.579+0000 7f9af9400640 1 -- 172.21.15.40:0/2257801963 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae40a7070 con 0x7f9afc15ffd0 2024-03-24T00:33:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.579+0000 7f9af9400640 1 -- 172.21.15.40:0/2257801963 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae40b0070 con 0x7f9afc15ffd0 2024-03-24T00:33:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.579+0000 7f9af9400640 1 -- 172.21.15.40:0/2257801963 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae40ab890 con 0x7f9afc15ffd0 2024-03-24T00:33:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.579+0000 7f9b01200640 1 -- 172.21.15.40:0/2257801963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9afc15ffd0 msgr2=0x7f9afc0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.579+0000 7f9b01200640 1 --2- 172.21.15.40:0/2257801963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9afc15ffd0 0x7f9afc0ae8e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40097c0 tx=0x7f9ae401c470 comp rx=0 tx=0).stop 2024-03-24T00:33:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.579+0000 7f9b01200640 1 -- 172.21.15.40:0/2257801963 shutdown_connections 2024-03-24T00:33:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.579+0000 7f9b01200640 1 --2- 172.21.15.40:0/2257801963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9afc0728d0 0x7f9afc0cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.579+0000 7f9b01200640 1 --2- 172.21.15.40:0/2257801963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9afc15ffd0 0x7f9afc0ae8e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.579+0000 7f9b01200640 1 -- 172.21.15.40:0/2257801963 >> 172.21.15.40:0/2257801963 conn(0x7f9afc0b1660 msgr2=0x7f9afc0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.579+0000 7f9b01200640 1 -- 172.21.15.40:0/2257801963 shutdown_connections 2024-03-24T00:33:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.579+0000 7f9b01200640 1 -- 172.21.15.40:0/2257801963 wait complete. 2024-03-24T00:33:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9b01200640 1 Processor -- start 2024-03-24T00:33:45.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9b01200640 1 -- start start 2024-03-24T00:33:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9b01200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9afc0728d0 0x7f9afc148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9b01200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9afc15bd00 0x7f9afc15c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9b01200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9afc15ffd0 0x7f9afc1540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9b01200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc06d320 con 0x7f9afc15ffd0 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9b01200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc15c0e0 con 0x7f9afc15bd00 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9b01200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc06d4a0 con 0x7f9afc0728d0 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af9e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9afc0728d0 0x7f9afc148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9afa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9afc15ffd0 0x7f9afc1540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af9e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9afc0728d0 0x7f9afc148130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:48528/0 (socket says 172.21.15.40:48528) 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af9e00640 1 -- 172.21.15.40:0/893259220 learned_addr learned my addr 172.21.15.40:0/893259220 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9afc15bd00 0x7f9afc15c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af9e00640 1 -- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9afc15bd00 msgr2=0x7f9afc15c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af9e00640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9afc15bd00 0x7f9afc15c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af9e00640 1 -- 172.21.15.40:0/893259220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9afc15ffd0 msgr2=0x7f9afc1540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af9e00640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9afc15ffd0 0x7f9afc1540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af9e00640 1 -- 172.21.15.40:0/893259220 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc14fc60 con 0x7f9afc0728d0 2024-03-24T00:33:45.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af3e00640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9afc15bd00 0x7f9afc15c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af9e00640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9afc0728d0 0x7f9afc148130 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00c550 tx=0x7f9aec00ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af2000640 1 -- 172.21.15.40:0/893259220 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec017070 con 0x7f9afc0728d0 2024-03-24T00:33:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af2000640 1 -- 172.21.15.40:0/893259220 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec005d10 con 0x7f9afc0728d0 2024-03-24T00:33:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9af2000640 1 -- 172.21.15.40:0/893259220 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec007330 con 0x7f9afc0728d0 2024-03-24T00:33:45.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.583+0000 7f9b01200640 1 -- 172.21.15.40:0/893259220 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9afc14ff20 con 0x7f9afc0728d0 2024-03-24T00:33:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.587+0000 7f9b01200640 1 -- 172.21.15.40:0/893259220 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9afc1504e0 con 0x7f9afc0728d0 2024-03-24T00:33:45.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.587+0000 7f9af2000640 1 -- 172.21.15.40:0/893259220 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f9aec01a070 con 0x7f9afc0728d0 2024-03-24T00:33:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.587+0000 7f9af2000640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ad8067660 0x7f9ad8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.587+0000 7f9af2000640 1 -- 172.21.15.40:0/893259220 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9aec084d00 con 0x7f9afc0728d0 2024-03-24T00:33:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.587+0000 7f9af3e00640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ad8067660 0x7f9ad8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.587+0000 7f9b047b2000 1 Processor -- start 2024-03-24T00:33:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.587+0000 7f9b047b2000 1 -- start start 2024-03-24T00:33:45.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9af3e00640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ad8067660 0x7f9ad8069b20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40af000 tx=0x7f9ae40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 0x556aecf77d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 0x556aecf2de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf2e380 0x556aecf2e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556aecd7c680 con 0x556aecf77970 2024-03-24T00:33:45.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x556aecd19b70 con 0x556aecf78290 2024-03-24T00:33:45.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x556aecd78720 con 0x556aecf2e380 2024-03-24T00:33:45.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9af9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 0x556aecf77d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9af9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 0x556aecf77d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48838/0 (socket says 172.21.15.40:48838) 2024-03-24T00:33:45.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9af9e00640 1 -- 172.21.15.40:0/4039666635 learned_addr learned my addr 172.21.15.40:0/4039666635 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9af3e00640 1 --2- 172.21.15.40:0/4039666635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 0x556aecf2de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9afa800640 1 --2- 172.21.15.40:0/4039666635 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf2e380 0x556aecf2e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9af9e00640 1 -- 172.21.15.40:0/4039666635 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf2e380 msgr2=0x556aecf2e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9af9e00640 1 --2- 172.21.15.40:0/4039666635 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf2e380 0x556aecf2e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9af9e00640 1 -- 172.21.15.40:0/4039666635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 msgr2=0x556aecf2de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9af9e00640 1 --2- 172.21.15.40:0/4039666635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 0x556aecf2de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9af9e00640 1 -- 172.21.15.40:0/4039666635 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556aecd19dc0 con 0x556aecf77970 2024-03-24T00:33:45.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9af9e00640 1 --2- 172.21.15.40:0/4039666635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 0x556aecf77d50 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9aec051720 tx=0x7f9aec055400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879e6549c1a03a66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9aeaa00640 1 -- 172.21.15.40:0/4039666635 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec0535c0 con 0x556aecf77970 2024-03-24T00:33:45.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9aeaa00640 1 -- 172.21.15.40:0/4039666635 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec051930 con 0x556aecf77970 2024-03-24T00:33:45.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9aeaa00640 1 -- 172.21.15.40:0/4039666635 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec055720 con 0x556aecf77970 2024-03-24T00:33:45.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 -- 172.21.15.40:0/4039666635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 msgr2=0x556aecf77d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 --2- 172.21.15.40:0/4039666635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 0x556aecf77d50 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9aec051720 tx=0x7f9aec055400 comp rx=0 tx=0).stop 2024-03-24T00:33:45.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 -- 172.21.15.40:0/4039666635 shutdown_connections 2024-03-24T00:33:45.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 --2- 172.21.15.40:0/4039666635 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf2e380 0x556aecf2e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 --2- 172.21.15.40:0/4039666635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 0x556aecf2de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 --2- 172.21.15.40:0/4039666635 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 0x556aecf77d50 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 -- 172.21.15.40:0/4039666635 >> 172.21.15.40:0/4039666635 conn(0x556aecf2a330 msgr2=0x556aecf2c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 -- 172.21.15.40:0/4039666635 shutdown_connections 2024-03-24T00:33:45.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 -- 172.21.15.40:0/4039666635 wait complete. 2024-03-24T00:33:45.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 -- start start 2024-03-24T00:33:45.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.591+0000 7f9b047b2000 1 Processor -- start 2024-03-24T00:33:45.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf2e380 0x556aecf331d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf77970 0x556aecf33710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 0x556aecf64e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556aece47c80 con 0x556aecf2e380 2024-03-24T00:33:45.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x556aecd3a3f0 con 0x556aecf78290 2024-03-24T00:33:45.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x556aecdfd420 con 0x556aecf77970 2024-03-24T00:33:45.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 10 client.0 _ll_register_callbacks cb 0x556aecf19680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:45.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9af9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf2e380 0x556aecf331d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9af9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf2e380 0x556aecf331d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48854/0 (socket says 172.21.15.40:48854) 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9af3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf77970 0x556aecf33710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9af9e00640 1 -- 172.21.15.40:0/2169953717 learned_addr learned my addr 172.21.15.40:0/2169953717 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9afa800640 1 --2- 172.21.15.40:0/2169953717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 0x556aecf64e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9af9e00640 1 -- 172.21.15.40:0/2169953717 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf77970 msgr2=0x556aecf33710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9af9e00640 1 --2- 172.21.15.40:0/2169953717 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf77970 0x556aecf33710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9af9e00640 1 -- 172.21.15.40:0/2169953717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 msgr2=0x556aecf64e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9af9e00640 1 --2- 172.21.15.40:0/2169953717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 0x556aecf64e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9af9e00640 1 -- 172.21.15.40:0/2169953717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556aecd3d6e0 con 0x556aecf2e380 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9af9e00640 1 --2- 172.21.15.40:0/2169953717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf2e380 0x556aecf331d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9aec051720 tx=0x7f9aec000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78aa4f6a87e0384d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9adf400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9adf400640 1 -- 172.21.15.40:0/2169953717 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec05d690 con 0x556aecf2e380 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9adf400640 1 -- 172.21.15.40:0/2169953717 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9aec054710 con 0x556aecf2e380 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 1 -- 172.21.15.40:0/2169953717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9aec051930 con 0x556aecf2e380 2024-03-24T00:33:45.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9adf400640 1 -- 172.21.15.40:0/2169953717 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9aec054910 con 0x556aecf2e380 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 10 client.6219 fetch_fsmap learned FSMap version 42 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 1 -- 172.21.15.40:0/2169953717 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x556aecd374a0 con 0x556aecf2e380 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9adf400640 1 -- 172.21.15.40:0/2169953717 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec051930 con 0x556aecf2e380 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9adf400640 1 -- 172.21.15.40:0/2169953717 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 42) v1 ==== 3362+0+0 (secure 0 0 0) 0x7f9aec05db70 con 0x556aecf2e380 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 10 client.6219 fetch_fsmap finished waiting for FSMap version 42 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 10 client.6219 resolve_mds: resolved 3:2 to role '3:2' aka daemon mds.b 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 20 client.6219 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 1 --2- 172.21.15.40:0/2169953717 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x556aecf64810 0x556aecf65810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 4 client.6219 mds_command: new command op to 5198 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9b047b2000 1 -- 172.21.15.40:0/2169953717 --> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556aecd17cf0 con 0x556aecf64810 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9af3e00640 1 --2- 172.21.15.40:0/2169953717 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x556aecf64810 0x556aecf65810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9adf400640 10 client.6219 ms_handle_connect on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:45.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.595+0000 7f9af3e00640 1 --2- 172.21.15.40:0/2169953717 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x556aecf64810 0x556aecf65810 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=32e0be3e3651b10e server_cookie=ca2d5b8226f35973 in_seq=0 out_seq=0 2024-03-24T00:33:45.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9adf400640 1 -- 172.21.15.40:0/2169953717 <== mds.2 v2:172.21.15.53:6835/1453842342 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556aecd17cf0 con 0x556aecf64810 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9adf400640 10 client.6219 handle_command_reply: tid=0 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9b047b2000 1 client.6219 shutdown 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9b047b2000 1 -- 172.21.15.40:0/2169953717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf2e380 msgr2=0x556aecf331d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9b047b2000 1 --2- 172.21.15.40:0/2169953717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf2e380 0x556aecf331d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9aec051720 tx=0x7f9aec000ee0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9b047b2000 1 -- 172.21.15.40:0/2169953717 shutdown_connections 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9b047b2000 1 --2- 172.21.15.40:0/2169953717 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x556aecf64810 0x556aecf65810 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9b047b2000 1 --2- 172.21.15.40:0/2169953717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 0x556aecf64e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9b047b2000 1 --2- 172.21.15.40:0/2169953717 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf77970 0x556aecf33710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9b047b2000 1 --2- 172.21.15.40:0/2169953717 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf2e380 0x556aecf331d0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9adf400640 0 client.6219 ms_handle_reset on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9b047b2000 1 -- 172.21.15.40:0/2169953717 >> 172.21.15.40:0/2169953717 conn(0x556aecf2a330 msgr2=0x556aecf2cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9b047b2000 1 -- 172.21.15.40:0/2169953717 shutdown_connections 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9b047b2000 1 -- 172.21.15.40:0/2169953717 wait complete. 2024-03-24T00:33:45.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.599+0000 7f9b047b2000 20 client.6219 trim_cache size 0 max 16384 2024-03-24T00:33:45.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9b047b2000 1 Processor -- start 2024-03-24T00:33:45.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9b047b2000 1 -- start start 2024-03-24T00:33:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9b047b2000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf2e380 0x556aecf2d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9b047b2000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 0x556aecf2dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9b047b2000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 0x556aecf88a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9b047b2000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc202e10 con 0x556aecf77970 2024-03-24T00:33:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9b047b2000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc0b2f60 con 0x556aecf78290 2024-03-24T00:33:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9b047b2000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc2027a0 con 0x556aecf2e380 2024-03-24T00:33:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9af3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 0x556aecf2dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9af9e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf2e380 0x556aecf2d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9af3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 0x556aecf2dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48868/0 (socket says 172.21.15.40:48868) 2024-03-24T00:33:45.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9af3e00640 1 -- 172.21.15.40:0/4087993780 learned_addr learned my addr 172.21.15.40:0/4087993780 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9afa800640 1 --2- 172.21.15.40:0/4087993780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 0x556aecf88a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9af3e00640 1 -- 172.21.15.40:0/4087993780 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf2e380 msgr2=0x556aecf2d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9af3e00640 1 --2- 172.21.15.40:0/4087993780 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf2e380 0x556aecf2d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9af3e00640 1 -- 172.21.15.40:0/4087993780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 msgr2=0x556aecf88a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9af3e00640 1 --2- 172.21.15.40:0/4087993780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 0x556aecf88a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9af3e00640 1 -- 172.21.15.40:0/4087993780 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556aecdfd8b0 con 0x556aecf77970 2024-03-24T00:33:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9af3e00640 1 --2- 172.21.15.40:0/4087993780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 0x556aecf2dc70 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9ae401a0a0 tx=0x7f9ae4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=943797b44ab59b41 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9aea000640 1 -- 172.21.15.40:0/4087993780 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae40bc070 con 0x556aecf77970 2024-03-24T00:33:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9aea000640 1 -- 172.21.15.40:0/4087993780 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae40b0040 con 0x556aecf77970 2024-03-24T00:33:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.623+0000 7f9aea000640 1 -- 172.21.15.40:0/4087993780 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae40ce310 con 0x556aecf77970 2024-03-24T00:33:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 -- 172.21.15.40:0/4087993780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 msgr2=0x556aecf2dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 --2- 172.21.15.40:0/4087993780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 0x556aecf2dc70 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9ae401a0a0 tx=0x7f9ae4008040 comp rx=0 tx=0).stop 2024-03-24T00:33:45.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 -- 172.21.15.40:0/4087993780 shutdown_connections 2024-03-24T00:33:45.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 --2- 172.21.15.40:0/4087993780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf78290 0x556aecf88a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 --2- 172.21.15.40:0/4087993780 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf77970 0x556aecf2dc70 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 --2- 172.21.15.40:0/4087993780 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf2e380 0x556aecf2d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 -- 172.21.15.40:0/4087993780 >> 172.21.15.40:0/4087993780 conn(0x556aecf2a330 msgr2=0x556aecf63980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 -- 172.21.15.40:0/4087993780 shutdown_connections 2024-03-24T00:33:45.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 -- 172.21.15.40:0/4087993780 wait complete. 2024-03-24T00:33:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 -- start start 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 Processor -- start 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf2e380 0x556aecf8bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf77970 0x556aecf8c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf78290 0x556aecf8c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x556aecf2a1b0 con 0x556aecf78290 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc151380 con 0x556aecf2e380 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x556aecdfd420 con 0x556aecf77970 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 10 client.0 _ll_register_callbacks cb 0x7f9ae4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9afa800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf78290 0x556aecf8c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9af9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf2e380 0x556aecf8bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9af3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf77970 0x556aecf8c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9af9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf2e380 0x556aecf8bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:41094/0 (socket says 172.21.15.40:41094) 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9af9e00640 1 -- 172.21.15.40:0/439064366 learned_addr learned my addr 172.21.15.40:0/439064366 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9af3e00640 1 -- 172.21.15.40:0/439064366 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf2e380 msgr2=0x556aecf8bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9af3e00640 1 --2- 172.21.15.40:0/439064366 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf2e380 0x556aecf8bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9af3e00640 1 -- 172.21.15.40:0/439064366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf78290 msgr2=0x556aecf8c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9af3e00640 1 --2- 172.21.15.40:0/439064366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf78290 0x556aecf8c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9af3e00640 1 -- 172.21.15.40:0/439064366 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556aecdd4fd0 con 0x556aecf77970 2024-03-24T00:33:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9af3e00640 1 --2- 172.21.15.40:0/439064366 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf77970 0x556aecf8c220 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9ae401c4b0 tx=0x7f9ae40046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efb49180b0aa98a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9adf400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:33:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9adf400640 1 -- 172.21.15.40:0/439064366 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae40bc040 con 0x556aecf77970 2024-03-24T00:33:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9adf400640 1 -- 172.21.15.40:0/439064366 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9ae40d7b20 con 0x556aecf77970 2024-03-24T00:33:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.627+0000 7f9b047b2000 1 -- 172.21.15.40:0/439064366 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ae40b0040 con 0x556aecf77970 2024-03-24T00:33:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9adf400640 1 -- 172.21.15.40:0/439064366 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ae40d7d20 con 0x556aecf77970 2024-03-24T00:33:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 10 client.5558 fetch_fsmap learned FSMap version 42 2024-03-24T00:33:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 -- 172.21.15.40:0/439064366 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x556aeca381c0 con 0x556aecf77970 2024-03-24T00:33:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9adf400640 1 -- 172.21.15.40:0/439064366 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ae40b0040 con 0x556aecf77970 2024-03-24T00:33:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9adf400640 1 -- 172.21.15.40:0/439064366 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 42) v1 ==== 3362+0+0 (secure 0 0 0) 0x7f9ae40ca040 con 0x556aecf77970 2024-03-24T00:33:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 10 client.5558 fetch_fsmap finished waiting for FSMap version 42 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 10 client.5558 resolve_mds: resolved 3:2 to role '3:2' aka daemon mds.b 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 20 client.5558 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 --2- 172.21.15.40:0/439064366 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x556aecf8cca0 0x556aecf8d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 4 client.5558 mds_command: new command op to 5198 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 -- 172.21.15.40:0/439064366 --> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x556aecd89bc0 con 0x556aecf8cca0 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9af9e00640 1 --2- 172.21.15.40:0/439064366 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x556aecf8cca0 0x556aecf8d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9af9e00640 1 --2- 172.21.15.40:0/439064366 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x556aecf8cca0 0x556aecf8d0b0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e2ec60380d943317 server_cookie=c0366bebe8f80a39 in_seq=0 out_seq=0 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9adf400640 10 client.5558 ms_handle_connect on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9adf400640 1 -- 172.21.15.40:0/439064366 <== mds.2 v2:172.21.15.53:6835/1453842342 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17660 (crc 0 0 0) 0x556aecd89bc0 con 0x556aecf8cca0 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9adf400640 10 client.5558 handle_command_reply: tid=0 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 client.5558 shutdown 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 -- 172.21.15.40:0/439064366 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf77970 msgr2=0x556aecf8c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 --2- 172.21.15.40:0/439064366 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf77970 0x556aecf8c220 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9ae401c4b0 tx=0x7f9ae40046d0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 -- 172.21.15.40:0/439064366 shutdown_connections 2024-03-24T00:33:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 --2- 172.21.15.40:0/439064366 >> [v2:172.21.15.53:6835/1453842342,v1:172.21.15.53:6837/1453842342] conn(0x556aecf8cca0 0x556aecf8d0b0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 --2- 172.21.15.40:0/439064366 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x556aecf78290 0x556aecf8c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 --2- 172.21.15.40:0/439064366 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556aecf77970 0x556aecf8c220 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9adf400640 0 client.5558 ms_handle_reset on v2:172.21.15.53:6835/1453842342 2024-03-24T00:33:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 --2- 172.21.15.40:0/439064366 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556aecf2e380 0x556aecf8bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 -- 172.21.15.40:0/439064366 >> 172.21.15.40:0/439064366 conn(0x556aecf2a330 msgr2=0x556aecf2c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 -- 172.21.15.40:0/439064366 shutdown_connections 2024-03-24T00:33:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 1 -- 172.21.15.40:0/439064366 wait complete. 2024-03-24T00:33:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b047b2000 20 client.5558 trim_cache size 0 max 16384 2024-03-24T00:33:45.637 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:45.637 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:33:45.637 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 21, 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 19, 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 9986, 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 57629, 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 11, 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 1, 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.014623397, 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.000595252, 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.009423627, 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000741932, 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 20, 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.022864557, 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001143227 2024-03-24T00:33:45.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 9, 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000001254, 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000139 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 1568, 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 448 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 68, 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 171, 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 110449, 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 191351, 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 10, 2024-03-24T00:33:45.639 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 6, 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.017503813, 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.005973857, 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.010856403, 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000684781, 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 169, 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.031906140, 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000188793 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 172, 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000093737, 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000544 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.640 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 101984 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 94, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 158, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 134676, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 137207, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 8, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.014099295, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.004293125, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.012085882, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.000832516, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 156, 2024-03-24T00:33:45.641 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.030378007, 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000194730 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 13, 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000003223, 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000247 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 128352, 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 15968 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.642 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 49, 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.003308622, 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000067522 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000008470, 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000004235 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-03-24T00:33:45.643 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 0, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 0, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 11, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 0, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 85, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 0, 2024-03-24T00:33:45.644 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 85, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 85, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 0, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 74, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 74, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 5, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 3, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 0, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 3, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 0, 2024-03-24T00:33:45.645 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 0, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 3, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 74, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 100, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 500, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 0, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-03-24T00:33:45.646 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 0 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-03-24T00:33:45.647 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 4, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-03-24T00:33:45.648 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 0, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 0, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 8, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 8, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-03-24T00:33:45.649 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 2, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 2, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 2, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 4194304, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 4250433, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 5, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.011821970, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002364394 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2024-03-24T00:33:45.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 88, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 88, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 0, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 52, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 53, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 1, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 85, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 85, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 0, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 74, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 74, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 0, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 50080, 2024-03-24T00:33:45.651 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 223536 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 0, 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 0, 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 3, 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.652 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.653 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.654 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.655 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.656 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.657 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.658 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.659 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.661 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 1, 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 1, 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 0, 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 1, 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 0, 2024-03-24T00:33:45.662 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 0, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 910, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:33:45.663 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-24T00:33:45.664 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 228839, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 63, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-24T00:33:45.665 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 5544, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 90, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 342584, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 1621, 2024-03-24T00:33:45.666 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 41, 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 68718, 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 33, 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 41, 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 41, 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.339991280, 2024-03-24T00:33:45.667 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.008292470 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 41, 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 64 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 41, 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 0, 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 41, 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 0, 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 10, 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 0, 2024-03-24T00:33:45.668 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 5, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 6, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 6, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 20, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-03-24T00:33:45.669 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-03-24T00:33:45.670 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 45, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 1, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 7, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 7, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 16, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 0, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 1 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.671 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 0, 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 0, 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 0, 2024-03-24T00:33:45.672 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 179, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 234143, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 179, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 179, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 234143, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.673 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 41, 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 68718, 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 41, 2024-03-24T00:33:45.674 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 68718, 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:45.675 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 41, 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 41, 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 41, 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 41, 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:45.676 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 0, 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 0, 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x558fa1946780": { 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.677 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 8, 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 56129, 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 8, 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 5, 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 56129, 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.678 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b01200640 1 -- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ad8067660 msgr2=0x7f9ad8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b01200640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ad8067660 0x7f9ad8069b20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9ae40af000 tx=0x7f9ae40a7000 comp rx=0 tx=0).stop 2024-03-24T00:33:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b01200640 1 -- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9afc0728d0 msgr2=0x7f9afc148130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b01200640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9afc0728d0 0x7f9afc148130 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9aec00c550 tx=0x7f9aec00ca10 comp rx=0 tx=0).stop 2024-03-24T00:33:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b01200640 1 -- 172.21.15.40:0/893259220 shutdown_connections 2024-03-24T00:33:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b01200640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ad8067660 0x7f9ad8069b20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b01200640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9afc15ffd0 0x7f9afc1540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b01200640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9afc15bd00 0x7f9afc15c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b01200640 1 --2- 172.21.15.40:0/893259220 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9afc0728d0 0x7f9afc148130 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b01200640 1 -- 172.21.15.40:0/893259220 >> 172.21.15.40:0/893259220 conn(0x7f9afc0b1660 msgr2=0x7f9afc0cde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b01200640 1 -- 172.21.15.40:0/893259220 shutdown_connections 2024-03-24T00:33:45.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.631+0000 7f9b01200640 1 -- 172.21.15.40:0/893259220 wait complete. 2024-03-24T00:33:45.681 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:33:45.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 perf dump 2024-03-24T00:33:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.855+0000 7f5fd2600640 1 Processor -- start 2024-03-24T00:33:45.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.855+0000 7f5fd2600640 1 -- start start 2024-03-24T00:33:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fd2600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5fcc15ffd0 0x7f5fcc0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fd2600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fcc0728d0 0x7f5fcc0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fd2600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fcc072d40 con 0x7f5fcc15bd00 2024-03-24T00:33:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fd2600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fcc0cf790 con 0x7f5fcc15ffd0 2024-03-24T00:33:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fd2600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc071de0 con 0x7f5fcc0728d0 2024-03-24T00:33:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fcbe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fcc0728d0 0x7f5fcc0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fcaa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5fcc15ffd0 0x7f5fcc0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fcbe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fcc0728d0 0x7f5fcc0cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:48606/0 (socket says 172.21.15.40:48606) 2024-03-24T00:33:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fcbe00640 1 -- 172.21.15.40:0/2262594196 learned_addr learned my addr 172.21.15.40:0/2262594196 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fcbe00640 1 -- 172.21.15.40:0/2262594196 >> v2:172.21.15.53:3300/0 conn(0x7f5fcc15ffd0 msgr2=0x7f5fcc0ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fcbe00640 1 --2- 172.21.15.40:0/2262594196 >> v2:172.21.15.53:3300/0 conn(0x7f5fcc15ffd0 0x7f5fcc0ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fcbe00640 1 -- 172.21.15.40:0/2262594196 >> v1:172.21.15.53:6789/0 conn(0x7f5fcc15bd00 legacy=0x7f5fcc15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fcbe00640 1 -- 172.21.15.40:0/2262594196 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc0cea50 con 0x7f5fcc0728d0 2024-03-24T00:33:45.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.859+0000 7f5fcaa00640 1 --2- 172.21.15.40:0/2262594196 >> v2:172.21.15.53:3300/0 conn(0x7f5fcc15ffd0 0x7f5fcc0ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:45.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.863+0000 7f5fcbe00640 1 --2- 172.21.15.40:0/2262594196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fcc0728d0 0x7f5fcc0cf250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4009e50 tx=0x7f5fc401ca00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f7804c203225040 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.863+0000 7f5fca000640 1 -- 172.21.15.40:0/2262594196 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc40a58d0 con 0x7f5fcc0728d0 2024-03-24T00:33:45.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.863+0000 7f5fca000640 1 -- 172.21.15.40:0/2262594196 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fc40a5a70 con 0x7f5fcc0728d0 2024-03-24T00:33:45.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.863+0000 7f5fca000640 1 -- 172.21.15.40:0/2262594196 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc40a5e40 con 0x7f5fcc0728d0 2024-03-24T00:33:45.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.863+0000 7f5fd2600640 1 -- 172.21.15.40:0/2262594196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fcc0728d0 msgr2=0x7f5fcc0cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.863+0000 7f5fd2600640 1 --2- 172.21.15.40:0/2262594196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fcc0728d0 0x7f5fcc0cf250 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5fc4009e50 tx=0x7f5fc401ca00 comp rx=0 tx=0).stop 2024-03-24T00:33:45.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.863+0000 7f5fd2600640 1 -- 172.21.15.40:0/2262594196 shutdown_connections 2024-03-24T00:33:45.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.863+0000 7f5fd2600640 1 --2- 172.21.15.40:0/2262594196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fcc0728d0 0x7f5fcc0cf250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.863+0000 7f5fd2600640 1 --2- 172.21.15.40:0/2262594196 >> v2:172.21.15.53:3300/0 conn(0x7f5fcc15ffd0 0x7f5fcc0ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.863+0000 7f5fd2600640 1 -- 172.21.15.40:0/2262594196 >> 172.21.15.40:0/2262594196 conn(0x7f5fcc0b1660 msgr2=0x7f5fcc0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.863+0000 7f5fd2600640 1 -- 172.21.15.40:0/2262594196 shutdown_connections 2024-03-24T00:33:45.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.863+0000 7f5fd2600640 1 -- 172.21.15.40:0/2262594196 wait complete. 2024-03-24T00:33:45.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fd2600640 1 Processor -- start 2024-03-24T00:33:45.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fd2600640 1 -- start start 2024-03-24T00:33:45.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fd2600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fcc0728d0 0x7f5fcc14fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fd2600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fcc15bd00 0x7f5fcc1584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fd2600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fcc15ffd0 0x7f5fcc0b3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fd2600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc06d320 con 0x7f5fcc0728d0 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fd2600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc15c0e0 con 0x7f5fcc15bd00 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fd2600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc06d4a0 con 0x7f5fcc15ffd0 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fcb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fcc0728d0 0x7f5fcc14fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fcaa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fcc15bd00 0x7f5fcc1584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fcb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fcc0728d0 0x7f5fcc14fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48882/0 (socket says 172.21.15.40:48882) 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fcb400640 1 -- 172.21.15.40:0/1904211653 learned_addr learned my addr 172.21.15.40:0/1904211653 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fcbe00640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fcc15ffd0 0x7f5fcc0b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fcaa00640 1 -- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fcc15ffd0 msgr2=0x7f5fcc0b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fcaa00640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fcc15ffd0 0x7f5fcc0b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fcaa00640 1 -- 172.21.15.40:0/1904211653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fcc0728d0 msgr2=0x7f5fcc14fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fcaa00640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fcc0728d0 0x7f5fcc14fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fcaa00640 1 -- 172.21.15.40:0/1904211653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc0b4980 con 0x7f5fcc15bd00 2024-03-24T00:33:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fcaa00640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fcc15bd00 0x7f5fcc1584a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800e8e0 tx=0x7f5fb800eda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fc3400640 1 -- 172.21.15.40:0/1904211653 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8017070 con 0x7f5fcc15bd00 2024-03-24T00:33:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fc3400640 1 -- 172.21.15.40:0/1904211653 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb800c8e0 con 0x7f5fcc15bd00 2024-03-24T00:33:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fc3400640 1 -- 172.21.15.40:0/1904211653 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fb800ccd0 con 0x7f5fcc15bd00 2024-03-24T00:33:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fd2600640 1 -- 172.21.15.40:0/1904211653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fcc0b43c0 con 0x7f5fcc15bd00 2024-03-24T00:33:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.867+0000 7f5fd2600640 1 -- 172.21.15.40:0/1904211653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fcc0afcc0 con 0x7f5fcc15bd00 2024-03-24T00:33:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fc3400640 1 -- 172.21.15.40:0/1904211653 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5fb801a070 con 0x7f5fcc15bd00 2024-03-24T00:33:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fd5ab8000 1 Processor -- start 2024-03-24T00:33:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fd5ab8000 1 -- start start 2024-03-24T00:33:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fc3400640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5fac067660 0x7f5fac069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fc3400640 1 -- 172.21.15.40:0/1904211653 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5fb8085b10 con 0x7f5fcc15bd00 2024-03-24T00:33:45.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fcbe00640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5fac067660 0x7f5fac069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fd5ab8000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe306970 0x55fcfe306d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fd5ab8000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe307290 0x55fcfe2bce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fd5ab8000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe2bd380 0x55fcfe2bd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fd5ab8000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcfe10b680 con 0x55fcfe2bd380 2024-03-24T00:33:45.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fd5ab8000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcfe0a8b70 con 0x55fcfe307290 2024-03-24T00:33:45.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fd5ab8000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fcfe107720 con 0x55fcfe306970 2024-03-24T00:33:45.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.871+0000 7f5fcb400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe306970 0x55fcfe306d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fcb400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe306970 0x55fcfe306d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:48614/0 (socket says 172.21.15.40:48614) 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fcaa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe307290 0x55fcfe2bce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fcb400640 1 -- 172.21.15.40:0/1704525934 learned_addr learned my addr 172.21.15.40:0/1704525934 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fcbe00640 1 --2- 172.21.15.40:0/1704525934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe2bd380 0x55fcfe2bd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fcb400640 1 -- 172.21.15.40:0/1704525934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe307290 msgr2=0x55fcfe2bce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fcb400640 1 --2- 172.21.15.40:0/1704525934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe307290 0x55fcfe2bce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fcb400640 1 -- 172.21.15.40:0/1704525934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe2bd380 msgr2=0x55fcfe2bd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fcb400640 1 --2- 172.21.15.40:0/1704525934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe2bd380 0x55fcfe2bd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fcb400640 1 -- 172.21.15.40:0/1704525934 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fcfe0a8dc0 con 0x55fcfe306970 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fcbe00640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5fac067660 0x7f5fac069b20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40ae000 tx=0x7f5fc4004ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fcaa00640 1 --2- 172.21.15.40:0/1704525934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe307290 0x55fcfe2bce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fcb400640 1 --2- 172.21.15.40:0/1704525934 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe306970 0x55fcfe306d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc001960 tx=0x7f5fbc007e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0293671e2c98960 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fb7e00640 1 -- 172.21.15.40:0/1704525934 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc0052e0 con 0x55fcfe306970 2024-03-24T00:33:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fb7e00640 1 -- 172.21.15.40:0/1704525934 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc018070 con 0x55fcfe306970 2024-03-24T00:33:45.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fb7e00640 1 -- 172.21.15.40:0/1704525934 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc00faf0 con 0x55fcfe306970 2024-03-24T00:33:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/1704525934 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe306970 msgr2=0x55fcfe306d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/1704525934 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe306970 0x55fcfe306d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc001960 tx=0x7f5fbc007e70 comp rx=0 tx=0).stop 2024-03-24T00:33:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/1704525934 shutdown_connections 2024-03-24T00:33:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/1704525934 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe2bd380 0x55fcfe2bd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/1704525934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe307290 0x55fcfe2bce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/1704525934 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe306970 0x55fcfe306d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/1704525934 >> 172.21.15.40:0/1704525934 conn(0x55fcfe2b9330 msgr2=0x55fcfe2bb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/1704525934 shutdown_connections 2024-03-24T00:33:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/1704525934 wait complete. 2024-03-24T00:33:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fd5ab8000 1 -- start start 2024-03-24T00:33:45.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.875+0000 7f5fd5ab8000 1 Processor -- start 2024-03-24T00:33:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe2bd380 0x55fcfe2c2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe306970 0x55fcfe2c7930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe307290 0x55fcfe2c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcfe1d6c80 con 0x55fcfe306970 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcfe0c93f0 con 0x55fcfe307290 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fcfe10fcb0 con 0x55fcfe2bd380 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 10 client.0 _ll_register_callbacks cb 0x55fcfe2a8680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe306970 0x55fcfe2c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcb400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe2bd380 0x55fcfe2c2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcb400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe2bd380 0x55fcfe2c2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:48622/0 (socket says 172.21.15.40:48622) 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcb400640 1 -- 172.21.15.40:0/2086034338 learned_addr learned my addr 172.21.15.40:0/2086034338 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcbe00640 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe307290 0x55fcfe2c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcb400640 1 -- 172.21.15.40:0/2086034338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe307290 msgr2=0x55fcfe2c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcb400640 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe307290 0x55fcfe2c7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcb400640 1 -- 172.21.15.40:0/2086034338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe306970 msgr2=0x55fcfe2c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcb400640 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe306970 0x55fcfe2c7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcb400640 1 -- 172.21.15.40:0/2086034338 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fcfe0cc6e0 con 0x55fcfe2bd380 2024-03-24T00:33:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcaa00640 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe306970 0x55fcfe2c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcb400640 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe2bd380 0x55fcfe2c2540 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc015040 tx=0x7f5fbc000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3472e7f3f5e9a1a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fb4c00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:33:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fb4c00640 1 -- 172.21.15.40:0/2086034338 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc003800 con 0x55fcfe2bd380 2024-03-24T00:33:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fb4c00640 1 -- 172.21.15.40:0/2086034338 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5fbc0039a0 con 0x55fcfe2bd380 2024-03-24T00:33:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2086034338 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5fbc018070 con 0x55fcfe2bd380 2024-03-24T00:33:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fb4c00640 1 -- 172.21.15.40:0/2086034338 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fbc02d050 con 0x55fcfe2bd380 2024-03-24T00:33:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 10 client.5570 fetch_fsmap learned FSMap version 42 2024-03-24T00:33:45.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2086034338 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55fcfe0c64a0 con 0x55fcfe2bd380 2024-03-24T00:33:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fb4c00640 1 -- 172.21.15.40:0/2086034338 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc018070 con 0x55fcfe2bd380 2024-03-24T00:33:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fb4c00640 1 -- 172.21.15.40:0/2086034338 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 42) v1 ==== 3362+0+0 (secure 0 0 0) 0x7f5fbc030020 con 0x55fcfe2bd380 2024-03-24T00:33:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 10 client.5570 fetch_fsmap finished waiting for FSMap version 42 2024-03-24T00:33:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 10 client.5570 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-24T00:33:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 20 client.5570 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55fcfe30cfb0 0x55fcfe30d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 4 client.5570 mds_command: new command op to 5763 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:33:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2086034338 --> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fcfe0a6cf0 con 0x55fcfe30cfb0 2024-03-24T00:33:45.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.879+0000 7f5fcaa00640 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55fcfe30cfb0 0x55fcfe30d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fcaa00640 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55fcfe30cfb0 0x55fcfe30d390 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d3f5e42818df86f server_cookie=dd6b0f05c9a5dd58 in_seq=0 out_seq=0 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fb4c00640 10 client.5570 ms_handle_connect on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fb4c00640 1 -- 172.21.15.40:0/2086034338 <== mds.0 v2:172.21.15.40:6832/224223662 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fcfe0a6cf0 con 0x55fcfe30cfb0 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fb4c00640 10 client.5570 handle_command_reply: tid=0 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fd5ab8000 1 client.5570 shutdown 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2086034338 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe2bd380 msgr2=0x55fcfe2c2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe2bd380 0x55fcfe2c2540 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc015040 tx=0x7f5fbc000ee0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2086034338 shutdown_connections 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55fcfe30cfb0 0x55fcfe30d390 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe307290 0x55fcfe2c7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe306970 0x55fcfe2c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/2086034338 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe2bd380 0x55fcfe2c2540 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fb4c00640 0 client.5570 ms_handle_reset on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2086034338 >> 172.21.15.40:0/2086034338 conn(0x55fcfe2b9330 msgr2=0x55fcfe2bbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2086034338 shutdown_connections 2024-03-24T00:33:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2086034338 wait complete. 2024-03-24T00:33:45.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.883+0000 7f5fd5ab8000 20 client.5570 trim_cache size 0 max 16384 2024-03-24T00:33:45.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 Processor -- start 2024-03-24T00:33:45.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 -- start start 2024-03-24T00:33:45.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe2bd380 0x55fcfe31cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe306970 0x55fcfe2bc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe307290 0x55fcfe2bcc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc2030d0 con 0x55fcfe2bd380 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc15b1f0 con 0x55fcfe306970 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc202a10 con 0x55fcfe307290 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fcb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe2bd380 0x55fcfe31cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fcb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe2bd380 0x55fcfe31cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48914/0 (socket says 172.21.15.40:48914) 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fcb400640 1 -- 172.21.15.40:0/1113125867 learned_addr learned my addr 172.21.15.40:0/1113125867 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fcaa00640 1 --2- 172.21.15.40:0/1113125867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe306970 0x55fcfe2bc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fcbe00640 1 --2- 172.21.15.40:0/1113125867 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe307290 0x55fcfe2bcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fcb400640 1 -- 172.21.15.40:0/1113125867 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe307290 msgr2=0x55fcfe2bcc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fcb400640 1 --2- 172.21.15.40:0/1113125867 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe307290 0x55fcfe2bcc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fcb400640 1 -- 172.21.15.40:0/1113125867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe306970 msgr2=0x55fcfe2bc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fcb400640 1 --2- 172.21.15.40:0/1113125867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe306970 0x55fcfe2bc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fcb400640 1 -- 172.21.15.40:0/1113125867 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fcfe3070c0 con 0x55fcfe2bd380 2024-03-24T00:33:45.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fcbe00640 1 --2- 172.21.15.40:0/1113125867 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe307290 0x55fcfe2bcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fcb400640 1 --2- 172.21.15.40:0/1113125867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe2bd380 0x55fcfe31cb90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc007eb0 tx=0x7f5fbc01f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a85cb2921a5608b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fb7400640 1 -- 172.21.15.40:0/1113125867 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc036070 con 0x55fcfe2bd380 2024-03-24T00:33:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fb7400640 1 -- 172.21.15.40:0/1113125867 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc018040 con 0x55fcfe2bd380 2024-03-24T00:33:45.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fb7400640 1 -- 172.21.15.40:0/1113125867 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc003860 con 0x55fcfe2bd380 2024-03-24T00:33:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/1113125867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe2bd380 msgr2=0x55fcfe31cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/1113125867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe2bd380 0x55fcfe31cb90 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc007eb0 tx=0x7f5fbc01f070 comp rx=0 tx=0).stop 2024-03-24T00:33:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/1113125867 shutdown_connections 2024-03-24T00:33:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/1113125867 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe307290 0x55fcfe2bcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/1113125867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe306970 0x55fcfe2bc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/1113125867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe2bd380 0x55fcfe31cb90 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/1113125867 >> 172.21.15.40:0/1113125867 conn(0x55fcfe2b9330 msgr2=0x55fcfe31c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/1113125867 shutdown_connections 2024-03-24T00:33:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.899+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/1113125867 wait complete. 2024-03-24T00:33:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 1 -- start start 2024-03-24T00:33:45.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 1 Processor -- start 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe2bd380 0x55fcfe36b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe306970 0x55fcfe36b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe307290 0x55fcfe3116c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcfe2b91b0 con 0x55fcfe306970 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc15a350 con 0x55fcfe2bd380 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fcfe10fcb0 con 0x55fcfe307290 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 10 client.0 _ll_register_callbacks cb 0x7f5fc40bc050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fcb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe2bd380 0x55fcfe36b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fcaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe306970 0x55fcfe36b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fcb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe2bd380 0x55fcfe36b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:41166/0 (socket says 172.21.15.40:41166) 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fcb400640 1 -- 172.21.15.40:0/2386594652 learned_addr learned my addr 172.21.15.40:0/2386594652 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fcbe00640 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe307290 0x55fcfe3116c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fcb400640 1 -- 172.21.15.40:0/2386594652 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe307290 msgr2=0x55fcfe3116c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fcb400640 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe307290 0x55fcfe3116c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fcb400640 1 -- 172.21.15.40:0/2386594652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe306970 msgr2=0x55fcfe36b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fcb400640 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe306970 0x55fcfe36b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fcb400640 1 -- 172.21.15.40:0/2386594652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fcfe18c8b0 con 0x55fcfe2bd380 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fcaa00640 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe306970 0x55fcfe36b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fcb400640 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe2bd380 0x55fcfe36b410 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc034040 tx=0x7f5fbc009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=386e97fb67be5e62 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fb4c00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fb4c00640 1 -- 172.21.15.40:0/2386594652 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fbc036040 con 0x55fcfe2bd380 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fb4c00640 1 -- 172.21.15.40:0/2386594652 <== mon.1 v2:172.21.15.53:3300/0 2 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5fbc03cb40 con 0x55fcfe2bd380 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2386594652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5fbc018040 con 0x55fcfe2bd380 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fb4c00640 1 -- 172.21.15.40:0/2386594652 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fbc03cd40 con 0x55fcfe2bd380 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 10 client.6052 fetch_fsmap learned FSMap version 42 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2386594652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x55fcfe163fd0 con 0x55fcfe2bd380 2024-03-24T00:33:45.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fb4c00640 1 -- 172.21.15.40:0/2386594652 <== mon.1 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fbc018040 con 0x55fcfe2bd380 2024-03-24T00:33:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fb4c00640 1 -- 172.21.15.40:0/2386594652 <== mon.1 v2:172.21.15.53:3300/0 5 ==== fsmap(e 42) v1 ==== 3362+0+0 (secure 0 0 0) 0x7f5fbc046020 con 0x55fcfe2bd380 2024-03-24T00:33:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 10 client.6052 fetch_fsmap finished waiting for FSMap version 42 2024-03-24T00:33:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 10 client.6052 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-03-24T00:33:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 20 client.6052 populate_metadata read hostname 'smithi040' 2024-03-24T00:33:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55fcfe312020 0x55fcfe312430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 4 client.6052 mds_command: new command op to 5763 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-03-24T00:33:45.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.903+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2386594652 --> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x55fcfe118bc0 con 0x55fcfe312020 2024-03-24T00:33:45.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fcaa00640 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55fcfe312020 0x55fcfe312430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fb4c00640 10 client.6052 ms_handle_connect on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fcaa00640 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55fcfe312020 0x55fcfe312430 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=143f4afe3dac3f7d server_cookie=660629a8251ab375 in_seq=0 out_seq=0 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fb4c00640 1 -- 172.21.15.40:0/2386594652 <== mds.0 v2:172.21.15.40:6832/224223662 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17823 (crc 0 0 0) 0x55fcfe118bc0 con 0x55fcfe312020 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fb4c00640 10 client.6052 handle_command_reply: tid=0 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd5ab8000 1 client.6052 shutdown 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2386594652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe2bd380 msgr2=0x55fcfe36b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe2bd380 0x55fcfe36b410 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc034040 tx=0x7f5fbc009040 comp rx=0 tx=0).stop 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2386594652 shutdown_connections 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.40:6832/224223662,v1:172.21.15.40:6834/224223662] conn(0x55fcfe312020 0x55fcfe312430 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fcfe307290 0x55fcfe3116c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55fcfe306970 0x55fcfe36b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fb4c00640 0 client.6052 ms_handle_reset on v2:172.21.15.40:6832/224223662 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd5ab8000 1 --2- 172.21.15.40:0/2386594652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fcfe2bd380 0x55fcfe36b410 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2386594652 >> 172.21.15.40:0/2386594652 conn(0x55fcfe2b9330 msgr2=0x55fcfe31c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2386594652 shutdown_connections 2024-03-24T00:33:45.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd5ab8000 1 -- 172.21.15.40:0/2386594652 wait complete. 2024-03-24T00:33:45.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd5ab8000 20 client.6052 trim_cache size 0 max 16384 2024-03-24T00:33:45.913 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:33:45.913 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T00:33:45.913 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 964, 2024-03-24T00:33:45.913 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 947, 2024-03-24T00:33:45.913 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 243069, 2024-03-24T00:33:45.913 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 2841059, 2024-03-24T00:33:45.913 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 13, 2024-03-24T00:33:45.913 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 7, 2024-03-24T00:33:45.913 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.099912627, 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.040207992, 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.031315837, 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.019233311, 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 942, 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.165933289, 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000176149 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 76, 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000009750, 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000128 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T00:33:45.914 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 2193, 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 3702, 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 869717, 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 1774044, 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 14, 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 5, 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.340432548, 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.180524430, 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.192535518, 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.004314873, 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 3702, 2024-03-24T00:33:45.915 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.295126394, 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000079720 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 5237, 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.001227847, 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000234 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 141632, 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 18032 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_messages": 292, 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages": 211, 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_bytes": 78581, 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_bytes": 297171, 2024-03-24T00:33:45.916 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_created_connections": 7, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_active_connections": 4, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_total_time": 0.027178465, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_send_time": 0.009656209, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_recv_time": 0.012580805, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_running_fast_dispatch_time": 0.004752858, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 213, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.186851175, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000877235 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_handle_ack_lat": { 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 181, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000021941, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000121 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-24T00:33:45.917 INFO:teuthology.orchestra.run.smithi040.stdout: "msgr_send_encrypted_bytes": 75552 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: "cct": { 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: "total_workers": 1, 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: "unhealthy_workers": 0 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-MDSRank": { 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1477, 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.846906108, 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000573396 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.918 INFO:teuthology.orchestra.run.smithi040.stdout: "finisher-PurgeQueue": { 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "queue_len": 0, 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "complete_latency": { 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 364, 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.028083014, 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000077151 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": { 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "request": 1506, 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "reply": 1506, 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "reply_latency": { 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1506, 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 2.158336002, 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001433158 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_reply": 0, 2024-03-24T00:33:45.919 INFO:teuthology.orchestra.run.smithi040.stdout: "forward": 0, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_complete": 0, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_fetch_keys": 0, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_commit": 21, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_split": 1, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_merge": 0, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes": 124, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_top": 0, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_bottom": 124, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pin_tail": 0, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_pinned": 124, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_expired": 88, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "inodes_with_caps": 114, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": 114, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "subtrees": 10, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse": 3005, 2024-03-24T00:33:45.920 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_hit": 1194, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_forward": 0, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_discover": 0, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_dir_fetch": 0, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_remote_ino": 0, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "traverse_lock": 0, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "load_cent": 0, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "q": 0, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "exported": 4, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "exported_inodes": 88, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "imported": 0, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "imported_inodes": 0, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_dir_fetch": 0, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_backtrace_fetch": 0, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "openino_peer_discover": 0, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rfiles": 100, 2024-03-24T00:33:45.921 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rbytes": 500, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "root_rsnaps": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_set_tag": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_inotable_repaired": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_inodes": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "scrub_file_inodes": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_inode_file_caps": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_revoke": 339, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_grant": 573, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T00:33:45.922 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps": 439, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_caps_dirty": 336, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_cap_release": 3, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "process_request_cap_release": 913 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache": { 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update": 0, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_update_receipt": 0, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_try_discover": 0, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_send_discover": 0, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "dir_handle_discover": 6, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays": 0, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_delayed": 0, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "num_strays_enqueuing": 0, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_created": 237, 2024-03-24T00:33:45.923 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_enqueued": 237, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_reintegrated": 0, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "strays_migrated": 0, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_processing": 0, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_enqueued": 0, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "num_recovering_prioritized": 0, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_started": 0, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "recovery_completed": 0, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_exportdir": 4, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_flush": 0, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragmentdir": 1, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_fragstats": 0, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "ireq_inodestats": 0 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log": { 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "evlrg": 0, 2024-03-24T00:33:45.924 INFO:teuthology.orchestra.run.smithi040.stdout: "evadd": 1266, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "evex": 0, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "evtrm": 0, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "ev": 1266, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "evexg": 0, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "evexd": 0, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "segadd": 3, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "segex": 0, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "segtrm": 0, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "seg": 3, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "segmjr": 2, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "segexg": 0, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "segexd": 0, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "expos": 4194304, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "wrpos": 6660441, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "rdpos": 0, 2024-03-24T00:33:45.925 INFO:teuthology.orchestra.run.smithi040.stdout: "jlat": { 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 824, 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 1.563898253, 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.001897934 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: "replayed": 0 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mem": { 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: "ino": 126, 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: "ino+": 451, 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: "ino-": 325, 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: "dir": 76, 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: "dir+": 122, 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: "dir-": 46, 2024-03-24T00:33:45.926 INFO:teuthology.orchestra.run.smithi040.stdout: "dn": 124, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "dn+": 685, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "dn-": 561, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "cap": 114, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "cap+": 439, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "cap-": 325, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "rss": 56500, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "heap": 231728 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_server": { 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_client_request": 2180, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "dispatch_server_request": 0, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_request": 1506, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_client_session": 9, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "handle_peer_request": 0, 2024-03-24T00:33:45.927 INFO:teuthology.orchestra.run.smithi040.stdout: "req_create_latency": { 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 336, 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.716311475, 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002131879 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getattr_latency": { 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 16, 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.003846698, 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000240418 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getfilelock_latency": { 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "req_link_latency": { 2024-03-24T00:33:45.928 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookup_latency": { 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 811, 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.144534233, 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000178217 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookuphash_latency": { 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupino_latency": { 2024-03-24T00:33:45.929 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupname_latency": { 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupparent_latency": { 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lookupsnap_latency": { 2024-03-24T00:33:45.930 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: "req_lssnap_latency": { 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mkdir_latency": { 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 102, 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.280717748, 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002752134 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mknod_latency": { 2024-03-24T00:33:45.931 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "req_mksnap_latency": { 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "req_open_latency": { 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "req_readdir_latency": { 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 2, 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.012218083, 2024-03-24T00:33:45.932 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.006109041 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rename_latency": { 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: "req_renamesnap_latency": { 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: "req_snapdiff_latency": { 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.933 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmdir_latency": { 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1, 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.004737418, 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.004737418 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmsnap_latency": { 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: "req_rmxattr_latency": { 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.934 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setattr_latency": { 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1, 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000240407, 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000240407 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setdirlayout_latency": { 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setfilelock_latency": { 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.935 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setlayout_latency": { 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: "req_setxattr_latency": { 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1, 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000494896, 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000494896 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: "req_symlink_latency": { 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.936 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: "req_unlink_latency": { 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 236, 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.995235044, 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.004217097 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_revoke_eviction": 0, 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: "cap_acquisition_throttle": 0, 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: "req_getvxattr_latency": { 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessions": { 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "session_count": 1, 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "session_add": 1, 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "session_remove": 0, 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_open": 1, 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "sessions_stale": 0, 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "total_load": 1408, 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "average_load": 1408, 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "avg_session_uptime": 38, 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "mdthresh_evicted": 0 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool": { 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_bytes": 0, 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "bloom_filter_items": 0, 2024-03-24T00:33:45.938 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_items": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_data_items": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_other_items": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_bytes": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_items": 0, 2024-03-24T00:33:45.939 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_bytes": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blob_items": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_items": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_bytes": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_txc_items": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_bytes": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_writing_items": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_bytes": 0, 2024-03-24T00:33:45.940 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_items": 0, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_reader_items": 0, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_file_writer_items": 0, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_bytes": 241526, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_anon_items": 89, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_bytes": 88, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "buffer_meta_items": 1, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bytes": 0, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_items": 0, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_bytes": 0, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mapbl_items": 0, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_bytes": 0, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pglog_items": 0, 2024-03-24T00:33:45.941 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_bytes": 5544, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_items": 90, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_mapping_items": 0, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_bytes": 0, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "pgmap_items": 0, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_bytes": 501797, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_co_items": 2971, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_bytes": 0, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_1_items": 0, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_bytes": 0, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "unittest_2_items": 0 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter": { 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "op_active": 0, 2024-03-24T00:33:45.942 INFO:teuthology.orchestra.run.smithi040.stdout: "op_laggy": 0, 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send": 1146, 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "op_send_bytes": 2638105, 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "op_resend": 37, 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "op_reply": 1146, 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "op_latency": { 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1146, 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 3.199972327, 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.002792296 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "op_inflight": 0, 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "oplen_avg": { 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 1146, 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 1196 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.943 INFO:teuthology.orchestra.run.smithi040.stdout: "op": 1146, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "op_r": 7, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "op_w": 1139, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "op_rmw": 0, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "op_pg": 0, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_stat": 0, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_create": 11, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_read": 7, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_write": 831, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writefull": 12, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_writesame": 0, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_append": 0, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_zero": 0, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_truncate": 1, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_delete": 249, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_mapext": 0, 2024-03-24T00:33:45.944 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_sparse_read": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_clonerange": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_getxattr": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_setxattr": 22, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_cmpxattr": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_rmxattr": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_resetxattrs": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_call": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_watch": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_notify": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_pgls_filter": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "osdop_other": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_active": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_send": 0, 2024-03-24T00:33:45.945 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_resend": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "linger_ping": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_active": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_send": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "poolop_resend": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_active": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_send": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "poolstat_resend": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_active": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_send": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "statfs_resend": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "command_active": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "command_send": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "command_resend": 0, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "map_epoch": 45, 2024-03-24T00:33:45.946 INFO:teuthology.orchestra.run.smithi040.stdout: "map_full": 0, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "map_inc": 2, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_sessions": 8, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_open": 8, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_session_close": 0, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_laggy": 0, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_wr": 53, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_rd": 0, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_del": 10 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "oft": { 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_objs": 1, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_kv_pairs": 0, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_updates": 0, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "omap_total_removes": 0 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.947 INFO:teuthology.orchestra.run.smithi040.stdout: "osd-slow-ops": { 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: "slow_ops_count": 0 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: "purge_queue": { 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops": 0, 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_ops_high_water": 10, 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing": 0, 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executing_high_water": 10, 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed_ops": 237, 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_executed": 237, 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: "pq_item_in_journal": 0 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.948 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 3444, 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 875888, 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 3444, 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 3444, 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 875888, 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_bytes": { 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 104857600, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 1146, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 2662042, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 1146, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 2662042, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.950 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-objecter_ops": { 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 1024, 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 0, 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 0, 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 0, 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 1146, 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 1146, 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 1146, 2024-03-24T00:33:45.951 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 1146, 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle": { 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 237, 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 23937, 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 237, 2024-03-24T00:33:45.952 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 7, 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 23937, 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "throttle-write_buf_throttle-0x559c68bcc780": { 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "val": 0, 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "max": 3758096384, 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "get_started": 0, 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "get": 1266, 2024-03-24T00:33:45.953 INFO:teuthology.orchestra.run.smithi040.stdout: "get_sum": 2466137, 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_fail": 0, 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout: "get_or_fail_success": 1266, 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout: "take": 0, 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout: "take_sum": 0, 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout: "put": 824, 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout: "put_sum": 2466137, 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout: "wait": { 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout: "avgcount": 0, 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout: "sum": 0.000000000, 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout: "avgtime": 0.000000000 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:33:45.954 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:33:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd2600640 1 -- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5fac067660 msgr2=0x7f5fac069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd2600640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5fac067660 0x7f5fac069b20 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5fc40ae000 tx=0x7f5fc4004ac0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd2600640 1 -- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fcc15bd00 msgr2=0x7f5fcc1584a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd2600640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fcc15bd00 0x7f5fcc1584a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5fb800e8e0 tx=0x7f5fb800eda0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd2600640 1 -- 172.21.15.40:0/1904211653 shutdown_connections 2024-03-24T00:33:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd2600640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5fac067660 0x7f5fac069b20 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd2600640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fcc15ffd0 0x7f5fcc0b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd2600640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fcc15bd00 0x7f5fcc1584a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd2600640 1 --2- 172.21.15.40:0/1904211653 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5fcc0728d0 0x7f5fcc14fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd2600640 1 -- 172.21.15.40:0/1904211653 >> 172.21.15.40:0/1904211653 conn(0x7f5fcc0b1660 msgr2=0x7f5fcc1552c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd2600640 1 -- 172.21.15.40:0/1904211653 shutdown_connections 2024-03-24T00:33:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:45.907+0000 7f5fd2600640 1 -- 172.21.15.40:0/1904211653 wait complete. 2024-03-24T00:33:45.957 INFO:tasks.cephfs.test_exports:export stats: [(0, 4), (1, 0), (2, 0)] 2024-03-24T00:33:45.957 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:33:45.958 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:33:46.022 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:33:46.071 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:33:46.071 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:33:46.071 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:33:46.081 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:33:46.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.259+0000 7f8216200640 1 Processor -- start 2024-03-24T00:33:46.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.259+0000 7f8216200640 1 -- start start 2024-03-24T00:33:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.259+0000 7f8216200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f821006e620 0x7f82100b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.259+0000 7f8216200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82100ccfa0 0x7f82100cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.259+0000 7f8216200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8210070f60 con 0x7f821006ea00 2024-03-24T00:33:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.259+0000 7f8216200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82100b4c00 con 0x7f821006e620 2024-03-24T00:33:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.259+0000 7f8216200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82100b3620 con 0x7f82100ccfa0 2024-03-24T00:33:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f8207e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f821006ea00 0x7f82100b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57682/0 (socket says 172.21.15.40:57682) 2024-03-24T00:33:46.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f8207e00640 1 -- 172.21.15.40:0/2282395399 learned_addr learned my addr 172.21.15.40:0/2282395399 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f820fe00640 1 --2- 172.21.15.40:0/2282395399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82100ccfa0 0x7f82100cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f820f400640 1 --2- 172.21.15.40:0/2282395399 >> v2:172.21.15.53:3300/0 conn(0x7f821006e620 0x7f82100b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f820fe00640 1 -- 172.21.15.40:0/2282395399 >> v2:172.21.15.53:3300/0 conn(0x7f821006e620 msgr2=0x7f82100b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f820fe00640 1 --2- 172.21.15.40:0/2282395399 >> v2:172.21.15.53:3300/0 conn(0x7f821006e620 0x7f82100b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f820fe00640 1 -- 172.21.15.40:0/2282395399 >> v1:172.21.15.40:6789/0 conn(0x7f821006ea00 legacy=0x7f82100b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:46.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f820ea00640 1 -- 172.21.15.40:0/2282395399 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956488974 0 0) 0x7f8210070f60 con 0x7f821006ea00 2024-03-24T00:33:46.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f820fe00640 1 -- 172.21.15.40:0/2282395399 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82100b3900 con 0x7f82100ccfa0 2024-03-24T00:33:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f820fe00640 1 --2- 172.21.15.40:0/2282395399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82100ccfa0 0x7f82100cd3d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8208009d90 tx=0x7f820801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68e919f91bada0d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f820ea00640 1 -- 172.21.15.40:0/2282395399 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82080b2070 con 0x7f82100ccfa0 2024-03-24T00:33:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f820ea00640 1 -- 172.21.15.40:0/2282395399 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82080a5880 con 0x7f82100ccfa0 2024-03-24T00:33:46.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.263+0000 7f820ea00640 1 -- 172.21.15.40:0/2282395399 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82080a5c50 con 0x7f82100ccfa0 2024-03-24T00:33:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 -- 172.21.15.40:0/2282395399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82100ccfa0 msgr2=0x7f82100cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 --2- 172.21.15.40:0/2282395399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82100ccfa0 0x7f82100cd3d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8208009d90 tx=0x7f820801c8b0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 -- 172.21.15.40:0/2282395399 shutdown_connections 2024-03-24T00:33:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 --2- 172.21.15.40:0/2282395399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82100ccfa0 0x7f82100cd3d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 --2- 172.21.15.40:0/2282395399 >> v2:172.21.15.53:3300/0 conn(0x7f821006e620 0x7f82100b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 -- 172.21.15.40:0/2282395399 >> 172.21.15.40:0/2282395399 conn(0x7f82100b17b0 msgr2=0x7f82100b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 -- 172.21.15.40:0/2282395399 shutdown_connections 2024-03-24T00:33:46.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 -- 172.21.15.40:0/2282395399 wait complete. 2024-03-24T00:33:46.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 Processor -- start 2024-03-24T00:33:46.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 -- start start 2024-03-24T00:33:46.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821006e620 0x7f821013d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:46.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f821006ea00 0x7f821015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:46.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82100ccfa0 0x7f821015c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:46.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821006efd0 con 0x7f821006e620 2024-03-24T00:33:46.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821006ee50 con 0x7f82100ccfa0 2024-03-24T00:33:46.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8216200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8210144920 con 0x7f821006ea00 2024-03-24T00:33:46.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f820f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821006e620 0x7f821013d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:46.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f820fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82100ccfa0 0x7f821015c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:46.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f8207e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f821006ea00 0x7f821015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:46.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f820f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821006e620 0x7f821013d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48942/0 (socket says 172.21.15.40:48942) 2024-03-24T00:33:46.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f820f400640 1 -- 172.21.15.40:0/3040766268 learned_addr learned my addr 172.21.15.40:0/3040766268 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:46.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f820fe00640 1 -- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f821006ea00 msgr2=0x7f821015c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:46.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.267+0000 7f820fe00640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f821006ea00 0x7f821015c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f820fe00640 1 -- 172.21.15.40:0/3040766268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821006e620 msgr2=0x7f821013d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:46.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f820fe00640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821006e620 0x7f821013d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f820fe00640 1 -- 172.21.15.40:0/3040766268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821015a860 con 0x7f82100ccfa0 2024-03-24T00:33:46.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f820f400640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821006e620 0x7f821013d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:46.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f820fe00640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82100ccfa0 0x7f821015c880 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f821006b1b0 tx=0x7f8208019650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:46.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f8207400640 1 -- 172.21.15.40:0/3040766268 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82080b2050 con 0x7f82100ccfa0 2024-03-24T00:33:46.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f8207400640 1 -- 172.21.15.40:0/3040766268 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82080be070 con 0x7f82100ccfa0 2024-03-24T00:33:46.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f8207400640 1 -- 172.21.15.40:0/3040766268 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82080ad850 con 0x7f82100ccfa0 2024-03-24T00:33:46.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f8216200640 1 -- 172.21.15.40:0/3040766268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821013d9d0 con 0x7f82100ccfa0 2024-03-24T00:33:46.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f8216200640 1 -- 172.21.15.40:0/3040766268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82101f6000 con 0x7f82100ccfa0 2024-03-24T00:33:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f8216200640 1 -- 172.21.15.40:0/3040766268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8210068170 con 0x7f82100ccfa0 2024-03-24T00:33:46.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f8207400640 1 -- 172.21.15.40:0/3040766268 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f82080ad9f0 con 0x7f82100ccfa0 2024-03-24T00:33:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f8207400640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81ec067660 0x7f81ec069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.271+0000 7f8207400640 1 -- 172.21.15.40:0/3040766268 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8208120bb0 con 0x7f82100ccfa0 2024-03-24T00:33:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.275+0000 7f820f400640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81ec067660 0x7f81ec069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:46.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.275+0000 7f820f400640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81ec067660 0x7f81ec069b20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8200011310 tx=0x7f8200009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:46.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.275+0000 7f8207400640 1 -- 172.21.15.40:0/3040766268 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f821015a860 con 0x7f82100ccfa0 2024-03-24T00:33:46.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.571+0000 7f8216200640 1 -- 172.21.15.40:0/3040766268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f821015e650 con 0x7f82100ccfa0 2024-03-24T00:33:46.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8207400640 1 -- 172.21.15.40:0/3040766268 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+16682 (secure 0 0 0) 0x7f8208016560 con 0x7f82100ccfa0 2024-03-24T00:33:46.577 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:46.577 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":45,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:33:27.880059+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:55.446633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:56.017210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6836/1600357100":"2024-03-25T00:33:27.880035+0000","172.21.15.53:6834/1600357100":"2024-03-25T00:33:27.880035+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:33:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8216200640 1 -- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81ec067660 msgr2=0x7f81ec069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8216200640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81ec067660 0x7f81ec069b20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8200011310 tx=0x7f8200009190 comp rx=0 tx=0).stop 2024-03-24T00:33:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8216200640 1 -- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82100ccfa0 msgr2=0x7f821015c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8216200640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82100ccfa0 0x7f821015c880 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f821006b1b0 tx=0x7f8208019650 comp rx=0 tx=0).stop 2024-03-24T00:33:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8216200640 1 -- 172.21.15.40:0/3040766268 shutdown_connections 2024-03-24T00:33:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8216200640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81ec067660 0x7f81ec069b20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8216200640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82100ccfa0 0x7f821015c880 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8216200640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f821006ea00 0x7f821015c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8216200640 1 --2- 172.21.15.40:0/3040766268 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821006e620 0x7f821013d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8216200640 1 -- 172.21.15.40:0/3040766268 >> 172.21.15.40:0/3040766268 conn(0x7f82100b17b0 msgr2=0x7f82101416f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8216200640 1 -- 172.21.15.40:0/3040766268 shutdown_connections 2024-03-24T00:33:46.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.575+0000 7f8216200640 1 -- 172.21.15.40:0/3040766268 wait complete. 2024-03-24T00:33:46.598 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:32:55.446633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:32:56.017210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:33:46.598 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:33:46.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd57200640 1 Processor -- start 2024-03-24T00:33:46.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd57200640 1 -- start start 2024-03-24T00:33:46.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd57200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd500b0bd0 0x7fdd500cd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:46.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd57200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdd500d0070 0x7fdd500d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:46.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd57200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd500cf2e0 con 0x7fdd5015e0e0 2024-03-24T00:33:46.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd57200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd500cd0e0 con 0x7fdd500d0070 2024-03-24T00:33:46.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd57200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd500cc7b0 con 0x7fdd500b0bd0 2024-03-24T00:33:46.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd55e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdd5015e0e0 0x7fdd500ceab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57684/0 (socket says 172.21.15.40:57684) 2024-03-24T00:33:46.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd55e00640 1 -- 172.21.15.40:0/1792004967 learned_addr learned my addr 172.21.15.40:0/1792004967 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:46.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd56800640 1 --2- 172.21.15.40:0/1792004967 >> v2:172.21.15.53:3300/0 conn(0x7fdd500d0070 0x7fdd500d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:46.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd55400640 1 --2- 172.21.15.40:0/1792004967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd500b0bd0 0x7fdd500cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:46.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd54a00640 1 -- 172.21.15.40:0/1792004967 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693005543 0 0) 0x7fdd500cf2e0 con 0x7fdd5015e0e0 2024-03-24T00:33:46.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd54a00640 1 -- 172.21.15.40:0/1792004967 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd38003530 con 0x7fdd5015e0e0 2024-03-24T00:33:46.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd56800640 1 -- 172.21.15.40:0/1792004967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd500b0bd0 msgr2=0x7fdd500cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:46.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd56800640 1 --2- 172.21.15.40:0/1792004967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd500b0bd0 0x7fdd500cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd56800640 1 -- 172.21.15.40:0/1792004967 >> v1:172.21.15.40:6789/0 conn(0x7fdd5015e0e0 legacy=0x7fdd500ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:46.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.743+0000 7fdd56800640 1 -- 172.21.15.40:0/1792004967 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500cca90 con 0x7fdd500d0070 2024-03-24T00:33:46.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.747+0000 7fdd56800640 1 --2- 172.21.15.40:0/1792004967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd500d0070 0x7fdd500d0500 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c009d90 tx=0x7fdd4c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85f608e19fac4aa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:46.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.747+0000 7fdd54a00640 1 -- 172.21.15.40:0/1792004967 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c0b2070 con 0x7fdd500d0070 2024-03-24T00:33:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.747+0000 7fdd54a00640 1 -- 172.21.15.40:0/1792004967 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4c0a5940 con 0x7fdd500d0070 2024-03-24T00:33:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.747+0000 7fdd54a00640 1 -- 172.21.15.40:0/1792004967 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c0a5cf0 con 0x7fdd500d0070 2024-03-24T00:33:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 -- 172.21.15.40:0/1792004967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd500d0070 msgr2=0x7fdd500d0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 --2- 172.21.15.40:0/1792004967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd500d0070 0x7fdd500d0500 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c009d90 tx=0x7fdd4c01c970 comp rx=0 tx=0).stop 2024-03-24T00:33:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 -- 172.21.15.40:0/1792004967 shutdown_connections 2024-03-24T00:33:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 --2- 172.21.15.40:0/1792004967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd500d0070 0x7fdd500d0500 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 --2- 172.21.15.40:0/1792004967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd500b0bd0 0x7fdd500cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 -- 172.21.15.40:0/1792004967 >> 172.21.15.40:0/1792004967 conn(0x7fdd500b3620 msgr2=0x7fdd500b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 -- 172.21.15.40:0/1792004967 shutdown_connections 2024-03-24T00:33:46.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 -- 172.21.15.40:0/1792004967 wait complete. 2024-03-24T00:33:46.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 Processor -- start 2024-03-24T00:33:46.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 -- start start 2024-03-24T00:33:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd500b0bd0 0x7fdd500c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd500d0070 0x7fdd50158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd5015e0e0 0x7fdd501590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5006bcf0 con 0x7fdd500b0bd0 2024-03-24T00:33:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5006bb70 con 0x7fdd500d0070 2024-03-24T00:33:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd57200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd500c66a0 con 0x7fdd5015e0e0 2024-03-24T00:33:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd55e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd500b0bd0 0x7fdd500c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd55e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd500b0bd0 0x7fdd500c8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48948/0 (socket says 172.21.15.40:48948) 2024-03-24T00:33:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd55400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd500d0070 0x7fdd50158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd55e00640 1 -- 172.21.15.40:0/2428922246 learned_addr learned my addr 172.21.15.40:0/2428922246 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.751+0000 7fdd56800640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd5015e0e0 0x7fdd501590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:46.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd55e00640 1 -- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd5015e0e0 msgr2=0x7fdd501590a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd55e00640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd5015e0e0 0x7fdd501590a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd55e00640 1 -- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd500d0070 msgr2=0x7fdd50158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd55e00640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd500d0070 0x7fdd50158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd55e00640 1 -- 172.21.15.40:0/2428922246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500c6980 con 0x7fdd500b0bd0 2024-03-24T00:33:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd55400640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd500d0070 0x7fdd50158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd55e00640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd500b0bd0 0x7fdd500c8b60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400c450 tx=0x7fdd4400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd4a000640 1 -- 172.21.15.40:0/2428922246 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44017070 con 0x7fdd500b0bd0 2024-03-24T00:33:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd4a000640 1 -- 172.21.15.40:0/2428922246 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd44005cf0 con 0x7fdd500b0bd0 2024-03-24T00:33:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd4a000640 1 -- 172.21.15.40:0/2428922246 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44007290 con 0x7fdd500b0bd0 2024-03-24T00:33:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd57200640 1 -- 172.21.15.40:0/2428922246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd500c6c60 con 0x7fdd500b0bd0 2024-03-24T00:33:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd57200640 1 -- 172.21.15.40:0/2428922246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd500c7220 con 0x7fdd500b0bd0 2024-03-24T00:33:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd57200640 1 -- 172.21.15.40:0/2428922246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5004f410 con 0x7fdd500b0bd0 2024-03-24T00:33:46.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.755+0000 7fdd4a000640 1 -- 172.21.15.40:0/2428922246 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fdd4401a070 con 0x7fdd500b0bd0 2024-03-24T00:33:46.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.759+0000 7fdd4a000640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd34067660 0x7fdd34069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:46.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.759+0000 7fdd4a000640 1 -- 172.21.15.40:0/2428922246 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdd44084d10 con 0x7fdd500b0bd0 2024-03-24T00:33:46.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.759+0000 7fdd55400640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd34067660 0x7fdd34069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:46.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.759+0000 7fdd55400640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd34067660 0x7fdd34069b20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdd500cec60 tx=0x7fdd40009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:46.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.763+0000 7fdd4a000640 1 -- 172.21.15.40:0/2428922246 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdd4408a020 con 0x7fdd500b0bd0 2024-03-24T00:33:46.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 -- 172.21.15.40:0/2428922246 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdd500b3400 con 0x7fdd500b0bd0 2024-03-24T00:33:46.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd4a000640 1 -- 172.21.15.40:0/2428922246 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd44055240 con 0x7fdd500b0bd0 2024-03-24T00:33:46.929 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:33:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 -- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd34067660 msgr2=0x7fdd34069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd34067660 0x7fdd34069b20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdd500cec60 tx=0x7fdd40009190 comp rx=0 tx=0).stop 2024-03-24T00:33:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 -- 172.21.15.40:0/2428922246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd500b0bd0 msgr2=0x7fdd500c8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd500b0bd0 0x7fdd500c8b60 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400c450 tx=0x7fdd4400c910 comp rx=0 tx=0).stop 2024-03-24T00:33:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 -- 172.21.15.40:0/2428922246 shutdown_connections 2024-03-24T00:33:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd34067660 0x7fdd34069b20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd5015e0e0 0x7fdd501590a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd500d0070 0x7fdd50158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 --2- 172.21.15.40:0/2428922246 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd500b0bd0 0x7fdd500c8b60 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 -- 172.21.15.40:0/2428922246 >> 172.21.15.40:0/2428922246 conn(0x7fdd500b3620 msgr2=0x7fdd500ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 -- 172.21.15.40:0/2428922246 shutdown_connections 2024-03-24T00:33:46.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:46.927+0000 7fdd57200640 1 -- 172.21.15.40:0/2428922246 wait complete. 2024-03-24T00:33:46.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:33:47.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.091+0000 7f63e4e00640 1 Processor -- start 2024-03-24T00:33:47.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.091+0000 7f63e4e00640 1 -- start start 2024-03-24T00:33:47.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.095+0000 7f63e4e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63e015dde0 0x7f63e00ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.095+0000 7f63e4e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e00adfe0 con 0x7f63e01619c0 2024-03-24T00:33:47.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.095+0000 7f63e4e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e00ae2c0 con 0x7f63e00b4780 2024-03-24T00:33:47.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.095+0000 7f63e4e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e006e2b0 con 0x7f63e015dde0 2024-03-24T00:33:47.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.095+0000 7f63dde00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63e015dde0 0x7f63e00ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.095+0000 7f63d5400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f63e01619c0 0x7f63e00b4070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60828/0 (socket says 172.21.15.40:60828) 2024-03-24T00:33:47.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.095+0000 7f63d5400640 1 -- 172.21.15.40:0/359252938 learned_addr learned my addr 172.21.15.40:0/359252938 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:47.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.095+0000 7f63dd400640 1 -- 172.21.15.40:0/359252938 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858978221 0 0) 0x7f63e00ae2c0 con 0x7f63e00b4780 2024-03-24T00:33:47.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.095+0000 7f63dde00640 1 -- 172.21.15.40:0/359252938 >> v1:172.21.15.53:6789/0 conn(0x7f63e00b4780 legacy=0x7f63e006ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.095+0000 7f63dde00640 1 -- 172.21.15.40:0/359252938 >> v1:172.21.15.40:6789/0 conn(0x7f63e01619c0 legacy=0x7f63e00b4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.095+0000 7f63dde00640 1 -- 172.21.15.40:0/359252938 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e00ae5a0 con 0x7f63e015dde0 2024-03-24T00:33:47.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.099+0000 7f63dde00640 1 --2- 172.21.15.40:0/359252938 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63e015dde0 0x7f63e00ad8e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f63d00097f0 tx=0x7f63d001c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35afe73a882c80c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.099+0000 7f63dd400640 1 -- 172.21.15.40:0/359252938 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00b1070 con 0x7f63e015dde0 2024-03-24T00:33:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.099+0000 7f63dd400640 1 -- 172.21.15.40:0/359252938 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d00ac640 con 0x7f63e015dde0 2024-03-24T00:33:47.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.099+0000 7f63dd400640 1 -- 172.21.15.40:0/359252938 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00aca10 con 0x7f63e015dde0 2024-03-24T00:33:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.099+0000 7f63e4e00640 1 -- 172.21.15.40:0/359252938 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63e015dde0 msgr2=0x7f63e00ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.099+0000 7f63e4e00640 1 --2- 172.21.15.40:0/359252938 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63e015dde0 0x7f63e00ad8e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f63d00097f0 tx=0x7f63d001c370 comp rx=0 tx=0).stop 2024-03-24T00:33:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.099+0000 7f63e4e00640 1 -- 172.21.15.40:0/359252938 shutdown_connections 2024-03-24T00:33:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.099+0000 7f63e4e00640 1 --2- 172.21.15.40:0/359252938 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63e015dde0 0x7f63e00ad8e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.099+0000 7f63e4e00640 1 -- 172.21.15.40:0/359252938 >> 172.21.15.40:0/359252938 conn(0x7f63e00b1c60 msgr2=0x7f63e00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.099+0000 7f63e4e00640 1 -- 172.21.15.40:0/359252938 shutdown_connections 2024-03-24T00:33:47.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.099+0000 7f63e4e00640 1 -- 172.21.15.40:0/359252938 wait complete. 2024-03-24T00:33:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63e4e00640 1 Processor -- start 2024-03-24T00:33:47.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63e4e00640 1 -- start start 2024-03-24T00:33:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63e4e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00b4780 0x7f63e00c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63e4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e015dde0 0x7f63e00c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63e4e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63e01619c0 0x7f63e015f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63e4e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e015f1d0 con 0x7f63e015dde0 2024-03-24T00:33:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63e4e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e00ae880 con 0x7f63e00b4780 2024-03-24T00:33:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63e4e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e015f590 con 0x7f63e01619c0 2024-03-24T00:33:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63dde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00b4780 0x7f63e00c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63dde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00b4780 0x7f63e00c67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:35808/0 (socket says 172.21.15.40:35808) 2024-03-24T00:33:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63dde00640 1 -- 172.21.15.40:0/2981840719 learned_addr learned my addr 172.21.15.40:0/2981840719 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63d5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e015dde0 0x7f63e00c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63de800640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63e01619c0 0x7f63e015f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63dde00640 1 -- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63e01619c0 msgr2=0x7f63e015f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63dde00640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63e01619c0 0x7f63e015f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63dde00640 1 -- 172.21.15.40:0/2981840719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e015dde0 msgr2=0x7f63e00c4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63dde00640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e015dde0 0x7f63e00c4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63dde00640 1 -- 172.21.15.40:0/2981840719 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e00c5700 con 0x7f63e00b4780 2024-03-24T00:33:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63de800640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63e01619c0 0x7f63e015f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.103+0000 7f63dde00640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00b4780 0x7f63e00c67a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f63d0019fa0 tx=0x7f63d001abb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.107+0000 7f63d6a00640 1 -- 172.21.15.40:0/2981840719 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00b1050 con 0x7f63e00b4780 2024-03-24T00:33:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.107+0000 7f63d6a00640 1 -- 172.21.15.40:0/2981840719 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d00bd070 con 0x7f63e00b4780 2024-03-24T00:33:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.107+0000 7f63e4e00640 1 -- 172.21.15.40:0/2981840719 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e00c59e0 con 0x7f63e00b4780 2024-03-24T00:33:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.107+0000 7f63d6a00640 1 -- 172.21.15.40:0/2981840719 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d00aade0 con 0x7f63e00b4780 2024-03-24T00:33:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.107+0000 7f63e4e00640 1 -- 172.21.15.40:0/2981840719 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e00c5fa0 con 0x7f63e00b4780 2024-03-24T00:33:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.107+0000 7f63e4e00640 1 -- 172.21.15.40:0/2981840719 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e0068170 con 0x7f63e00b4780 2024-03-24T00:33:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.107+0000 7f63d6a00640 1 -- 172.21.15.40:0/2981840719 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f63d00c2460 con 0x7f63e00b4780 2024-03-24T00:33:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.107+0000 7f63d6a00640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f63bc067660 0x7f63bc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.107+0000 7f63d6a00640 1 -- 172.21.15.40:0/2981840719 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f63d0120510 con 0x7f63e00b4780 2024-03-24T00:33:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.107+0000 7f63d5400640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f63bc067660 0x7f63bc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.111+0000 7f63d5400640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f63bc067660 0x7f63bc069b20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f63e014fe60 tx=0x7f63c800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:47.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.115+0000 7f63d6a00640 1 -- 172.21.15.40:0/2981840719 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f63d0015ff0 con 0x7f63e00b4780 2024-03-24T00:33:47.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.275+0000 7f63e4e00640 1 -- 172.21.15.40:0/2981840719 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f63e0161730 con 0x7f63e00b4780 2024-03-24T00:33:47.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.275+0000 7f63d6a00640 1 -- 172.21.15.40:0/2981840719 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v45) v1 ==== 104+0+11 (secure 0 0 0) 0x7f63d00f0b70 con 0x7f63e00b4780 2024-03-24T00:33:47.280 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:33:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.279+0000 7f63e4e00640 1 -- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f63bc067660 msgr2=0x7f63bc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.279+0000 7f63e4e00640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f63bc067660 0x7f63bc069b20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f63e014fe60 tx=0x7f63c800b040 comp rx=0 tx=0).stop 2024-03-24T00:33:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.279+0000 7f63e4e00640 1 -- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00b4780 msgr2=0x7f63e00c67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.279+0000 7f63e4e00640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00b4780 0x7f63e00c67a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f63d0019fa0 tx=0x7f63d001abb0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.279+0000 7f63e4e00640 1 -- 172.21.15.40:0/2981840719 shutdown_connections 2024-03-24T00:33:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.279+0000 7f63e4e00640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f63bc067660 0x7f63bc069b20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.279+0000 7f63e4e00640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f63e01619c0 0x7f63e015f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.279+0000 7f63e4e00640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f63e015dde0 0x7f63e00c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.279+0000 7f63e4e00640 1 --2- 172.21.15.40:0/2981840719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00b4780 0x7f63e00c67a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.279+0000 7f63e4e00640 1 -- 172.21.15.40:0/2981840719 >> 172.21.15.40:0/2981840719 conn(0x7f63e00b1c60 msgr2=0x7f63e00b2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.279+0000 7f63e4e00640 1 -- 172.21.15.40:0/2981840719 shutdown_connections 2024-03-24T00:33:47.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.279+0000 7f63e4e00640 1 -- 172.21.15.40:0/2981840719 wait complete. 2024-03-24T00:33:47.303 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:33:47.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.443+0000 7fa8e8800640 1 Processor -- start 2024-03-24T00:33:47.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.443+0000 7fa8e8800640 1 -- start start 2024-03-24T00:33:47.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e8800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa8e015bd00 0x7fa8e00ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e8800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8e015ffd0 0x7fa8e00cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e8800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e0072d60 con 0x7fa8e00728d0 2024-03-24T00:33:47.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e8800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8e00cf790 con 0x7fa8e015bd00 2024-03-24T00:33:47.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e8800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e0071de0 con 0x7fa8e015ffd0 2024-03-24T00:33:47.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e5400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa8e00728d0 0x7fa8e015f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60830/0 (socket says 172.21.15.40:60830) 2024-03-24T00:33:47.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e5400640 1 -- 172.21.15.40:0/188249817 learned_addr learned my addr 172.21.15.40:0/188249817 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:47.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e5e00640 1 --2- 172.21.15.40:0/188249817 >> v2:172.21.15.53:3300/0 conn(0x7fa8e015bd00 0x7fa8e00ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e6800640 1 --2- 172.21.15.40:0/188249817 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8e015ffd0 0x7fa8e00cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e6800640 1 -- 172.21.15.40:0/188249817 >> v2:172.21.15.53:3300/0 conn(0x7fa8e015bd00 msgr2=0x7fa8e00ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:47.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e6800640 1 --2- 172.21.15.40:0/188249817 >> v2:172.21.15.53:3300/0 conn(0x7fa8e015bd00 0x7fa8e00ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e4a00640 1 -- 172.21.15.40:0/188249817 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558605604 0 0) 0x7fa8e0072d60 con 0x7fa8e00728d0 2024-03-24T00:33:47.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e6800640 1 -- 172.21.15.40:0/188249817 >> v1:172.21.15.40:6789/0 conn(0x7fa8e00728d0 legacy=0x7fa8e015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.447+0000 7fa8e6800640 1 -- 172.21.15.40:0/188249817 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e00cea50 con 0x7fa8e015ffd0 2024-03-24T00:33:47.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.451+0000 7fa8e6800640 1 --2- 172.21.15.40:0/188249817 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8e015ffd0 0x7fa8e00cf250 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc009d90 tx=0x7fa8dc01c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11d4edd91d431fab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:47.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.451+0000 7fa8e4a00640 1 -- 172.21.15.40:0/188249817 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc0b2070 con 0x7fa8e015ffd0 2024-03-24T00:33:47.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.451+0000 7fa8e4a00640 1 -- 172.21.15.40:0/188249817 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc0a5a00 con 0x7fa8e015ffd0 2024-03-24T00:33:47.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.451+0000 7fa8e4a00640 1 -- 172.21.15.40:0/188249817 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc0a5dd0 con 0x7fa8e015ffd0 2024-03-24T00:33:47.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.451+0000 7fa8e8800640 1 -- 172.21.15.40:0/188249817 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8e015ffd0 msgr2=0x7fa8e00cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.451+0000 7fa8e8800640 1 --2- 172.21.15.40:0/188249817 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8e015ffd0 0x7fa8e00cf250 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc009d90 tx=0x7fa8dc01c940 comp rx=0 tx=0).stop 2024-03-24T00:33:47.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.451+0000 7fa8e8800640 1 -- 172.21.15.40:0/188249817 shutdown_connections 2024-03-24T00:33:47.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.451+0000 7fa8e8800640 1 --2- 172.21.15.40:0/188249817 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8e015ffd0 0x7fa8e00cf250 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.451+0000 7fa8e8800640 1 --2- 172.21.15.40:0/188249817 >> v2:172.21.15.53:3300/0 conn(0x7fa8e015bd00 0x7fa8e00ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.451+0000 7fa8e8800640 1 -- 172.21.15.40:0/188249817 >> 172.21.15.40:0/188249817 conn(0x7fa8e00b1660 msgr2=0x7fa8e00b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:47.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.451+0000 7fa8e8800640 1 -- 172.21.15.40:0/188249817 shutdown_connections 2024-03-24T00:33:47.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.451+0000 7fa8e8800640 1 -- 172.21.15.40:0/188249817 wait complete. 2024-03-24T00:33:47.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e8800640 1 Processor -- start 2024-03-24T00:33:47.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e8800640 1 -- start start 2024-03-24T00:33:47.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e8800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8e00728d0 0x7fa8e014fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e8800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8e015bd00 0x7fa8e015b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e8800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8e015ffd0 0x7fa8e0148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e8800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e006d320 con 0x7fa8e00728d0 2024-03-24T00:33:47.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e8800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e015c0e0 con 0x7fa8e015bd00 2024-03-24T00:33:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e8800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e006d4a0 con 0x7fa8e015ffd0 2024-03-24T00:33:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8e00728d0 0x7fa8e014fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8e00728d0 0x7fa8e014fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40512/0 (socket says 172.21.15.40:40512) 2024-03-24T00:33:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e5e00640 1 -- 172.21.15.40:0/3923437915 learned_addr learned my addr 172.21.15.40:0/3923437915 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e6800640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8e015ffd0 0x7fa8e0148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e5400640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8e015bd00 0x7fa8e015b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e6800640 1 -- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8e015bd00 msgr2=0x7fa8e015b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e6800640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8e015bd00 0x7fa8e015b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e6800640 1 -- 172.21.15.40:0/3923437915 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8e00728d0 msgr2=0x7fa8e014fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e6800640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8e00728d0 0x7fa8e014fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e6800640 1 -- 172.21.15.40:0/3923437915 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e01569c0 con 0x7fa8e015ffd0 2024-03-24T00:33:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e6800640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8e015ffd0 0x7fa8e0148990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00d0a30 tx=0x7fa8dc0196e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8da000640 1 -- 172.21.15.40:0/3923437915 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc0b2040 con 0x7fa8e015ffd0 2024-03-24T00:33:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8da000640 1 -- 172.21.15.40:0/3923437915 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8dc0bf070 con 0x7fa8e015ffd0 2024-03-24T00:33:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e8800640 1 -- 172.21.15.40:0/3923437915 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e0156ca0 con 0x7fa8e015ffd0 2024-03-24T00:33:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8da000640 1 -- 172.21.15.40:0/3923437915 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8dc0b8870 con 0x7fa8e015ffd0 2024-03-24T00:33:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.455+0000 7fa8e8800640 1 -- 172.21.15.40:0/3923437915 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e0156670 con 0x7fa8e015ffd0 2024-03-24T00:33:47.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.459+0000 7fa8e8800640 1 -- 172.21.15.40:0/3923437915 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e0156ea0 con 0x7fa8e015ffd0 2024-03-24T00:33:47.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.459+0000 7fa8da000640 1 -- 172.21.15.40:0/3923437915 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa8dc0b8a10 con 0x7fa8e015ffd0 2024-03-24T00:33:47.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.459+0000 7fa8da000640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa8c4067660 0x7fa8c4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.459+0000 7fa8da000640 1 -- 172.21.15.40:0/3923437915 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa8dc120b50 con 0x7fa8e015ffd0 2024-03-24T00:33:47.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.459+0000 7fa8e5e00640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa8c4067660 0x7fa8c4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.459+0000 7fa8e5e00640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa8c4067660 0x7fa8c4069b20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00d0580 tx=0x7fa8d4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:47.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.463+0000 7fa8da000640 1 -- 172.21.15.40:0/3923437915 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa8dc016560 con 0x7fa8e015ffd0 2024-03-24T00:33:47.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.627+0000 7fa8e8800640 1 -- 172.21.15.40:0/3923437915 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa8e0068170 con 0x7fa8e015ffd0 2024-03-24T00:33:47.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.627+0000 7fa8da000640 1 -- 172.21.15.40:0/3923437915 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v45) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa8dc0b8d00 con 0x7fa8e015ffd0 2024-03-24T00:33:47.630 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:33:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.627+0000 7fa8e8800640 1 -- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa8c4067660 msgr2=0x7fa8c4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.627+0000 7fa8e8800640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa8c4067660 0x7fa8c4069b20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00d0580 tx=0x7fa8d4009190 comp rx=0 tx=0).stop 2024-03-24T00:33:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.627+0000 7fa8e8800640 1 -- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8e015ffd0 msgr2=0x7fa8e0148990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.627+0000 7fa8e8800640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8e015ffd0 0x7fa8e0148990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa8e00d0a30 tx=0x7fa8dc0196e0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.631+0000 7fa8e8800640 1 -- 172.21.15.40:0/3923437915 shutdown_connections 2024-03-24T00:33:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.631+0000 7fa8e8800640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa8c4067660 0x7fa8c4069b20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.631+0000 7fa8e8800640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8e015ffd0 0x7fa8e0148990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.631+0000 7fa8e8800640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8e015bd00 0x7fa8e015b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.631+0000 7fa8e8800640 1 --2- 172.21.15.40:0/3923437915 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa8e00728d0 0x7fa8e014fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.631+0000 7fa8e8800640 1 -- 172.21.15.40:0/3923437915 >> 172.21.15.40:0/3923437915 conn(0x7fa8e00b1660 msgr2=0x7fa8e0158aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.631+0000 7fa8e8800640 1 -- 172.21.15.40:0/3923437915 shutdown_connections 2024-03-24T00:33:47.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.631+0000 7fa8e8800640 1 -- 172.21.15.40:0/3923437915 wait complete. 2024-03-24T00:33:47.652 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:33:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa069800640 1 Processor -- start 2024-03-24T00:33:47.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa069800640 1 -- start start 2024-03-24T00:33:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa069800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa0640d0010 0x7fa0640b2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa069800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0640b2ee0 con 0x7fa06415f7e0 2024-03-24T00:33:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa069800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0640af570 con 0x7fa0640d03f0 2024-03-24T00:33:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa069800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0640af850 con 0x7fa0640d0010 2024-03-24T00:33:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa062a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa0640d0010 0x7fa0640b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa062a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa0640d0010 0x7fa0640b2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40518/0 (socket says 172.21.15.40:40518) 2024-03-24T00:33:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa063400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa06415f7e0 0x7fa064165d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60844/0 (socket says 172.21.15.40:60844) 2024-03-24T00:33:47.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa062a00640 1 -- 172.21.15.40:0/1581138661 learned_addr learned my addr 172.21.15.40:0/1581138661 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa062a00640 1 -- 172.21.15.40:0/1581138661 >> v1:172.21.15.53:6789/0 conn(0x7fa0640d03f0 legacy=0x7fa06415f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa061600640 1 -- 172.21.15.40:0/1581138661 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3142750189 0 0) 0x7fa0640af570 con 0x7fa0640d03f0 2024-03-24T00:33:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa062a00640 1 -- 172.21.15.40:0/1581138661 >> v1:172.21.15.40:6789/0 conn(0x7fa06415f7e0 legacy=0x7fa064165d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.807+0000 7fa062a00640 1 -- 172.21.15.40:0/1581138661 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0640afb30 con 0x7fa0640d0010 2024-03-24T00:33:47.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.811+0000 7fa062a00640 1 --2- 172.21.15.40:0/1581138661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0640d0010 0x7fa0640b2900 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa054009870 tx=0x7fa05401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f74080d6fa9712e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:47.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.811+0000 7fa061600640 1 -- 172.21.15.40:0/1581138661 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0540b1070 con 0x7fa0640d0010 2024-03-24T00:33:47.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.811+0000 7fa061600640 1 -- 172.21.15.40:0/1581138661 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0540ac5f0 con 0x7fa0640d0010 2024-03-24T00:33:47.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.811+0000 7fa061600640 1 -- 172.21.15.40:0/1581138661 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0540ac9c0 con 0x7fa0640d0010 2024-03-24T00:33:47.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.811+0000 7fa069800640 1 -- 172.21.15.40:0/1581138661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0640d0010 msgr2=0x7fa0640b2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.811+0000 7fa069800640 1 --2- 172.21.15.40:0/1581138661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0640d0010 0x7fa0640b2900 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa054009870 tx=0x7fa05401c3f0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.811+0000 7fa069800640 1 -- 172.21.15.40:0/1581138661 shutdown_connections 2024-03-24T00:33:47.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.811+0000 7fa069800640 1 --2- 172.21.15.40:0/1581138661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0640d0010 0x7fa0640b2900 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.811+0000 7fa069800640 1 -- 172.21.15.40:0/1581138661 >> 172.21.15.40:0/1581138661 conn(0x7fa0640b38f0 msgr2=0x7fa0640b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:47.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.811+0000 7fa069800640 1 -- 172.21.15.40:0/1581138661 shutdown_connections 2024-03-24T00:33:47.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.811+0000 7fa069800640 1 -- 172.21.15.40:0/1581138661 wait complete. 2024-03-24T00:33:47.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.815+0000 7fa069800640 1 Processor -- start 2024-03-24T00:33:47.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.815+0000 7fa069800640 1 -- start start 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.815+0000 7fa069800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0640d0010 0x7fa06414e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.815+0000 7fa069800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0640d03f0 0x7fa0640b7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.815+0000 7fa069800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa06415f7e0 0x7fa0640cde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.815+0000 7fa069800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640b24c0 con 0x7fa0640d03f0 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.815+0000 7fa069800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640b2340 con 0x7fa0640d0010 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.815+0000 7fa069800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0640b2640 con 0x7fa06415f7e0 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa062000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0640d03f0 0x7fa0640b7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa062a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0640d0010 0x7fa06414e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa062a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0640d0010 0x7fa06414e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:35826/0 (socket says 172.21.15.40:35826) 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa062a00640 1 -- 172.21.15.40:0/146608438 learned_addr learned my addr 172.21.15.40:0/146608438 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa063400640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa06415f7e0 0x7fa0640cde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa062a00640 1 -- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa06415f7e0 msgr2=0x7fa0640cde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa062a00640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa06415f7e0 0x7fa0640cde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa062a00640 1 -- 172.21.15.40:0/146608438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0640d03f0 msgr2=0x7fa0640b7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:47.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa062a00640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0640d03f0 0x7fa0640b7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:47.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa062a00640 1 -- 172.21.15.40:0/146608438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0640b4910 con 0x7fa0640d0010 2024-03-24T00:33:47.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa063400640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa06415f7e0 0x7fa0640cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:47.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa062a00640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0640d0010 0x7fa06414e390 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa0540af040 tx=0x7fa054019190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:47.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa05aa00640 1 -- 172.21.15.40:0/146608438 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0540b1040 con 0x7fa0640d0010 2024-03-24T00:33:47.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa05aa00640 1 -- 172.21.15.40:0/146608438 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0540ba070 con 0x7fa0640d0010 2024-03-24T00:33:47.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa069800640 1 -- 172.21.15.40:0/146608438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0640b4bf0 con 0x7fa0640d0010 2024-03-24T00:33:47.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa05aa00640 1 -- 172.21.15.40:0/146608438 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0540ab980 con 0x7fa0640d0010 2024-03-24T00:33:47.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa069800640 1 -- 172.21.15.40:0/146608438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0640b51b0 con 0x7fa0640d0010 2024-03-24T00:33:47.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa069800640 1 -- 172.21.15.40:0/146608438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa064068170 con 0x7fa0640d0010 2024-03-24T00:33:47.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.819+0000 7fa05aa00640 1 -- 172.21.15.40:0/146608438 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa0540abb20 con 0x7fa0640d0010 2024-03-24T00:33:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.823+0000 7fa05aa00640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa044067660 0x7fa044069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.823+0000 7fa05aa00640 1 -- 172.21.15.40:0/146608438 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa05411fd30 con 0x7fa0640d0010 2024-03-24T00:33:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.823+0000 7fa062000640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa044067660 0x7fa044069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:47.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.823+0000 7fa062000640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa044067660 0x7fa044069b20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa0640d2d60 tx=0x7fa050009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:47.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:47.823+0000 7fa05aa00640 1 -- 172.21.15.40:0/146608438 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa0540a9070 con 0x7fa0640d0010 2024-03-24T00:33:48.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.015+0000 7fa069800640 1 -- 172.21.15.40:0/146608438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa06406de30 con 0x7fa0640d0010 2024-03-24T00:33:48.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.015+0000 7fa05aa00640 1 -- 172.21.15.40:0/146608438 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v46) v1 ==== 80+0+122 (secure 0 0 0) 0x7fa054016070 con 0x7fa0640d0010 2024-03-24T00:33:48.018 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.53:6836/1600357100 2024-03-25T00:33:27.880035+0000 2024-03-24T00:33:48.018 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.53:6834/1600357100 2024-03-25T00:33:27.880035+0000 2024-03-24T00:33:48.019 INFO:teuthology.orchestra.run.smithi040.stderr:listed 2 entries 2024-03-24T00:33:48.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.019+0000 7fa069800640 1 -- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa044067660 msgr2=0x7fa044069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.019+0000 7fa069800640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa044067660 0x7fa044069b20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fa0640d2d60 tx=0x7fa050009040 comp rx=0 tx=0).stop 2024-03-24T00:33:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.019+0000 7fa069800640 1 -- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0640d0010 msgr2=0x7fa06414e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.019+0000 7fa069800640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0640d0010 0x7fa06414e390 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa0540af040 tx=0x7fa054019190 comp rx=0 tx=0).stop 2024-03-24T00:33:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.019+0000 7fa069800640 1 -- 172.21.15.40:0/146608438 shutdown_connections 2024-03-24T00:33:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.019+0000 7fa069800640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa044067660 0x7fa044069b20 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.019+0000 7fa069800640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa06415f7e0 0x7fa0640cde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.019+0000 7fa069800640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa0640d03f0 0x7fa0640b7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.019+0000 7fa069800640 1 --2- 172.21.15.40:0/146608438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0640d0010 0x7fa06414e390 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.019+0000 7fa069800640 1 -- 172.21.15.40:0/146608438 >> 172.21.15.40:0/146608438 conn(0x7fa0640b38f0 msgr2=0x7fa06415e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.019+0000 7fa069800640 1 -- 172.21.15.40:0/146608438 shutdown_connections 2024-03-24T00:33:48.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.019+0000 7fa069800640 1 -- 172.21.15.40:0/146608438 wait complete. 2024-03-24T00:33:48.047 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-03-24T00:33:48.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:48.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T00:33:48.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[39495]: fuse finished with error 0 and tester_r 0 2024-03-24T00:33:48.071 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:33:48.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:48.071 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:33:48.160 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:33:48.160 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:33:48.169 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:33:48.169 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:33:48.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:48.169 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T00:33:48.219 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T00:33:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:33:48.220 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:33:48.220 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:33:48.229 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:33:48.229 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:33:48.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:48.229 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T00:33:48.279 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T00:33:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:33:48.280 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:33:48.280 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:33:48.289 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:33:48.289 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:33:48.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:33:48.289 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T00:33:48.339 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T00:33:48.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:33:48.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:48.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.479+0000 7f3450e00640 1 Processor -- start 2024-03-24T00:33:48.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.479+0000 7f3450e00640 1 -- start start 2024-03-24T00:33:48.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.479+0000 7f3450e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f344c156320 0x7f344c072920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:48.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.479+0000 7f3450e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344c160760 con 0x7f344c066e80 2024-03-24T00:33:48.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.479+0000 7f3450e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344c0af600 con 0x7f344c06ee60 2024-03-24T00:33:48.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.479+0000 7f3450e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344c0b0470 con 0x7f344c156320 2024-03-24T00:33:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.479+0000 7f3441400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f344c066e80 0x7f344c15f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60848/0 (socket says 172.21.15.40:60848) 2024-03-24T00:33:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.479+0000 7f3441400640 1 -- 172.21.15.40:0/3211394102 learned_addr learned my addr 172.21.15.40:0/3211394102 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.479+0000 7f3449e00640 1 --2- 172.21.15.40:0/3211394102 >> v2:172.21.15.53:3300/0 conn(0x7f344c156320 0x7f344c072920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3449e00640 1 -- 172.21.15.40:0/3211394102 >> v1:172.21.15.53:6789/0 conn(0x7f344c06ee60 legacy=0x7f344c160050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3449e00640 1 -- 172.21.15.40:0/3211394102 >> v1:172.21.15.40:6789/0 conn(0x7f344c066e80 legacy=0x7f344c15f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.479+0000 7f3449400640 1 -- 172.21.15.40:0/3211394102 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798899923 0 0) 0x7f344c0af600 con 0x7f344c06ee60 2024-03-24T00:33:48.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3449e00640 1 -- 172.21.15.40:0/3211394102 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c0af8a0 con 0x7f344c156320 2024-03-24T00:33:48.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3449e00640 1 --2- 172.21.15.40:0/3211394102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f344c156320 0x7f344c072920 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f344c0b05f0 tx=0x7f343c01c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f72d1d8a092bf791 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3449400640 1 -- 172.21.15.40:0/3211394102 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c0b2070 con 0x7f344c156320 2024-03-24T00:33:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3449400640 1 -- 172.21.15.40:0/3211394102 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343c0a57b0 con 0x7f344c156320 2024-03-24T00:33:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3449400640 1 -- 172.21.15.40:0/3211394102 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c0a5b80 con 0x7f344c156320 2024-03-24T00:33:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3450e00640 1 -- 172.21.15.40:0/3211394102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f344c156320 msgr2=0x7f344c072920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3450e00640 1 --2- 172.21.15.40:0/3211394102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f344c156320 0x7f344c072920 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f344c0b05f0 tx=0x7f343c01c750 comp rx=0 tx=0).stop 2024-03-24T00:33:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3450e00640 1 -- 172.21.15.40:0/3211394102 shutdown_connections 2024-03-24T00:33:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3450e00640 1 --2- 172.21.15.40:0/3211394102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f344c156320 0x7f344c072920 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3450e00640 1 -- 172.21.15.40:0/3211394102 >> 172.21.15.40:0/3211394102 conn(0x7f344c06b150 msgr2=0x7f344c06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3450e00640 1 -- 172.21.15.40:0/3211394102 shutdown_connections 2024-03-24T00:33:48.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.483+0000 7f3450e00640 1 -- 172.21.15.40:0/3211394102 wait complete. 2024-03-24T00:33:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3450e00640 1 Processor -- start 2024-03-24T00:33:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3450e00640 1 -- start start 2024-03-24T00:33:48.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3450e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f344c066e80 0x7f344c1517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:48.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3450e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f344c06ee60 0x7f344c09b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:48.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3450e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f344c156320 0x7f344c15ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3450e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0b0100 con 0x7f344c06ee60 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3450e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0afe20 con 0x7f344c066e80 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3450e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344c156700 con 0x7f344c156320 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3441400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f344c06ee60 0x7f344c09b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3449e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f344c066e80 0x7f344c1517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3441400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f344c06ee60 0x7f344c09b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40522/0 (socket says 172.21.15.40:40522) 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3449e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f344c066e80 0x7f344c1517b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:35846/0 (socket says 172.21.15.40:35846) 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3441400640 1 -- 172.21.15.40:0/1820426506 learned_addr learned my addr 172.21.15.40:0/1820426506 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f344a800640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f344c156320 0x7f344c15ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3449e00640 1 -- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f344c156320 msgr2=0x7f344c15ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3449e00640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f344c156320 0x7f344c15ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3449e00640 1 -- 172.21.15.40:0/1820426506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f344c06ee60 msgr2=0x7f344c09b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3449e00640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f344c06ee60 0x7f344c09b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3449e00640 1 -- 172.21.15.40:0/1820426506 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c151cf0 con 0x7f344c066e80 2024-03-24T00:33:48.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3441400640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f344c06ee60 0x7f344c09b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:48.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.487+0000 7f3449e00640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f344c066e80 0x7f344c1517b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f343c0ac000 tx=0x7f343c0194f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.491+0000 7f3442a00640 1 -- 172.21.15.40:0/1820426506 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c0bd020 con 0x7f344c066e80 2024-03-24T00:33:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.491+0000 7f3442a00640 1 -- 172.21.15.40:0/1820426506 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343c0adb70 con 0x7f344c066e80 2024-03-24T00:33:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.491+0000 7f3442a00640 1 -- 172.21.15.40:0/1820426506 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c0b2040 con 0x7f344c066e80 2024-03-24T00:33:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.491+0000 7f3450e00640 1 -- 172.21.15.40:0/1820426506 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344c09d200 con 0x7f344c066e80 2024-03-24T00:33:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.491+0000 7f3450e00640 1 -- 172.21.15.40:0/1820426506 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344c09ad50 con 0x7f344c066e80 2024-03-24T00:33:48.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.491+0000 7f3450e00640 1 -- 172.21.15.40:0/1820426506 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344c068170 con 0x7f344c066e80 2024-03-24T00:33:48.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.491+0000 7f3442a00640 1 -- 172.21.15.40:0/1820426506 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f343c0c0020 con 0x7f344c066e80 2024-03-24T00:33:48.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.491+0000 7f3442a00640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3428067660 0x7f3428069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:48.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.491+0000 7f3442a00640 1 -- 172.21.15.40:0/1820426506 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f343c120be0 con 0x7f344c066e80 2024-03-24T00:33:48.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.491+0000 7f3441400640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3428067660 0x7f3428069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:48.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.491+0000 7f3441400640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3428067660 0x7f3428069b20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f344c09c280 tx=0x7f343400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:48.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.495+0000 7f3442a00640 1 -- 172.21.15.40:0/1820426506 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f343c0c7030 con 0x7f344c066e80 2024-03-24T00:33:48.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 -- 172.21.15.40:0/1820426506 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f344c072220 con 0x7f344c066e80 2024-03-24T00:33:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3442a00640 1 -- 172.21.15.40:0/1820426506 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f343c016460 con 0x7f344c066e80 2024-03-24T00:33:48.697 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:48.697 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":42,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/374985095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":374985095},{"type":"v1","addr":"172.21.15.53:6836","nonce":374985095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:31.903823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5763,"mds_1":5192,"mds_2":5198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"c","rank":1,"incarnation":38,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5198":{"gid":5198,"name":"b","rank":2,"incarnation":34,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5192,5198,5763]},"id":3}]} 2024-03-24T00:33:48.697 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 42 2024-03-24T00:33:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 -- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3428067660 msgr2=0x7f3428069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3428067660 0x7f3428069b20 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f344c09c280 tx=0x7f343400b040 comp rx=0 tx=0).stop 2024-03-24T00:33:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 -- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f344c066e80 msgr2=0x7f344c1517b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f344c066e80 0x7f344c1517b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f343c0ac000 tx=0x7f343c0194f0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 -- 172.21.15.40:0/1820426506 shutdown_connections 2024-03-24T00:33:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3428067660 0x7f3428069b20 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f344c156320 0x7f344c15ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f344c06ee60 0x7f344c09b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 --2- 172.21.15.40:0/1820426506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f344c066e80 0x7f344c1517b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 -- 172.21.15.40:0/1820426506 >> 172.21.15.40:0/1820426506 conn(0x7f344c06b150 msgr2=0x7f344c15c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 -- 172.21.15.40:0/1820426506 shutdown_connections 2024-03-24T00:33:48.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.695+0000 7f3450e00640 1 -- 172.21.15.40:0/1820426506 wait complete. 2024-03-24T00:33:48.719 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:33:48.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.927+0000 7f84ce600640 1 Processor -- start 2024-03-24T00:33:48.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.927+0000 7f84ce600640 1 -- start start 2024-03-24T00:33:48.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84ce600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c8147830 0x7f84c80b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:48.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84ce600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f84c8161b70 0x7f84c8072420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:48.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84ce600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c81622e0 con 0x7f84c8072960 2024-03-24T00:33:48.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84ce600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84c8071cb0 con 0x7f84c8161b70 2024-03-24T00:33:48.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84ce600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c81625c0 con 0x7f84c8147830 2024-03-24T00:33:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84c6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f84c8161b70 0x7f84c8072420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84c7400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c8147830 0x7f84c80b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84c6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f84c8161b70 0x7f84c8072420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40538/0 (socket says 172.21.15.40:40538) 2024-03-24T00:33:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84c6a00640 1 -- 172.21.15.40:0/217787157 learned_addr learned my addr 172.21.15.40:0/217787157 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84c6000640 1 -- 172.21.15.40:0/217787157 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926604893 0 0) 0x7f84c81622e0 con 0x7f84c8072960 2024-03-24T00:33:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84c6a00640 1 -- 172.21.15.40:0/217787157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c8147830 msgr2=0x7f84c80b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:48.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84c6a00640 1 --2- 172.21.15.40:0/217787157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c8147830 0x7f84c80b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84c6a00640 1 -- 172.21.15.40:0/217787157 >> v1:172.21.15.53:6789/0 conn(0x7f84c8072960 legacy=0x7f84c8066e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84c6a00640 1 -- 172.21.15.40:0/217787157 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c81628a0 con 0x7f84c8161b70 2024-03-24T00:33:48.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.931+0000 7f84c7400640 1 --2- 172.21.15.40:0/217787157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c8147830 0x7f84c80b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:48.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.935+0000 7f84c6a00640 1 --2- 172.21.15.40:0/217787157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c8161b70 0x7f84c8072420 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009870 tx=0x7f84b401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9c68b8accd528e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:48.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.935+0000 7f84c6000640 1 -- 172.21.15.40:0/217787157 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b40b2070 con 0x7f84c8161b70 2024-03-24T00:33:48.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.935+0000 7f84c6000640 1 -- 172.21.15.40:0/217787157 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b40a9070 con 0x7f84c8161b70 2024-03-24T00:33:48.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.935+0000 7f84c6000640 1 -- 172.21.15.40:0/217787157 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b40ad880 con 0x7f84c8161b70 2024-03-24T00:33:48.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.939+0000 7f84ce600640 1 -- 172.21.15.40:0/217787157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c8161b70 msgr2=0x7f84c8072420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.939+0000 7f84ce600640 1 --2- 172.21.15.40:0/217787157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c8161b70 0x7f84c8072420 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f84b4009870 tx=0x7f84b401c4d0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.939+0000 7f84ce600640 1 -- 172.21.15.40:0/217787157 shutdown_connections 2024-03-24T00:33:48.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.939+0000 7f84ce600640 1 --2- 172.21.15.40:0/217787157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c8161b70 0x7f84c8072420 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.939+0000 7f84ce600640 1 --2- 172.21.15.40:0/217787157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c8147830 0x7f84c80b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.939+0000 7f84ce600640 1 -- 172.21.15.40:0/217787157 >> 172.21.15.40:0/217787157 conn(0x7f84c80d0c20 msgr2=0x7f84c80d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:48.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.939+0000 7f84ce600640 1 -- 172.21.15.40:0/217787157 shutdown_connections 2024-03-24T00:33:48.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.939+0000 7f84ce600640 1 -- 172.21.15.40:0/217787157 wait complete. 2024-03-24T00:33:48.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.939+0000 7f84ce600640 1 Processor -- start 2024-03-24T00:33:48.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.939+0000 7f84ce600640 1 -- start start 2024-03-24T00:33:48.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84ce600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c8147830 0x7f84c813e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:48.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84ce600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c8161b70 0x7f84c813ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:48.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84ce600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80cedc0 0x7f84c815d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:48.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84ce600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c806e6a0 con 0x7f84c8147830 2024-03-24T00:33:48.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84ce600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c806e520 con 0x7f84c80cedc0 2024-03-24T00:33:48.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84ce600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c8067590 con 0x7f84c8161b70 2024-03-24T00:33:48.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84c7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c8147830 0x7f84c813e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:48.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84c7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c8147830 0x7f84c813e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40546/0 (socket says 172.21.15.40:40546) 2024-03-24T00:33:48.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84c7400640 1 -- 172.21.15.40:0/2961105213 learned_addr learned my addr 172.21.15.40:0/2961105213 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:48.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84c7e00640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80cedc0 0x7f84c815d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:48.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84c6a00640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c8161b70 0x7f84c813ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:48.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84c7e00640 1 -- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c8161b70 msgr2=0x7f84c813ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84c7e00640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c8161b70 0x7f84c813ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84c7e00640 1 -- 172.21.15.40:0/2961105213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c8147830 msgr2=0x7f84c813e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:48.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84c7e00640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c8147830 0x7f84c813e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:48.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84c7e00640 1 -- 172.21.15.40:0/2961105213 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c80cf250 con 0x7f84c80cedc0 2024-03-24T00:33:48.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84c7e00640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80cedc0 0x7f84c815d550 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f84c00012f0 tx=0x7f84c000cbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:48.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84bf400640 1 -- 172.21.15.40:0/2961105213 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c0003b80 con 0x7f84c80cedc0 2024-03-24T00:33:48.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84bf400640 1 -- 172.21.15.40:0/2961105213 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c0003d40 con 0x7f84c80cedc0 2024-03-24T00:33:48.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84bf400640 1 -- 172.21.15.40:0/2961105213 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c00052b0 con 0x7f84c80cedc0 2024-03-24T00:33:48.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84ce600640 1 -- 172.21.15.40:0/2961105213 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c80cf530 con 0x7f84c80cedc0 2024-03-24T00:33:48.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.943+0000 7f84ce600640 1 -- 172.21.15.40:0/2961105213 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c80cfaf0 con 0x7f84c80cedc0 2024-03-24T00:33:48.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.947+0000 7f84ce600640 1 -- 172.21.15.40:0/2961105213 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84c8068170 con 0x7f84c80cedc0 2024-03-24T00:33:48.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.947+0000 7f84bf400640 1 -- 172.21.15.40:0/2961105213 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f84c0005450 con 0x7f84c80cedc0 2024-03-24T00:33:48.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.947+0000 7f84bf400640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f84a80676b0 0x7f84a8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:48.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.947+0000 7f84bf400640 1 -- 172.21.15.40:0/2961105213 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f84c0085430 con 0x7f84c80cedc0 2024-03-24T00:33:48.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.947+0000 7f84c7400640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f84a80676b0 0x7f84a8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:48.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.951+0000 7f84c7400640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f84a80676b0 0x7f84a8069b70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009580 tx=0x7f84b8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:48.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:48.955+0000 7f84bf400640 1 -- 172.21.15.40:0/2961105213 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f84c0055b40 con 0x7f84c80cedc0 2024-03-24T00:33:49.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.119+0000 7f84ce600640 1 -- 172.21.15.40:0/2961105213 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84c8067ca0 con 0x7f84c80cedc0 2024-03-24T00:33:49.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84bf400640 1 -- 172.21.15.40:0/2961105213 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16773 (secure 0 0 0) 0x7f84c0059be0 con 0x7f84c80cedc0 2024-03-24T00:33:49.126 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:49.126 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":47,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:33:48.936722+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:55.446633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:56.017210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.37","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6836/1600357100":"2024-03-25T00:33:27.880035+0000","172.21.15.53:6834/1600357100":"2024-03-25T00:33:27.880035+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:33:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84ce600640 1 -- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f84a80676b0 msgr2=0x7f84a8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84ce600640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f84a80676b0 0x7f84a8069b70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f84b8009580 tx=0x7f84b8009190 comp rx=0 tx=0).stop 2024-03-24T00:33:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84ce600640 1 -- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80cedc0 msgr2=0x7f84c815d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84ce600640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80cedc0 0x7f84c815d550 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f84c00012f0 tx=0x7f84c000cbe0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84ce600640 1 -- 172.21.15.40:0/2961105213 shutdown_connections 2024-03-24T00:33:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84ce600640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f84a80676b0 0x7f84a8069b70 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84ce600640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84c80cedc0 0x7f84c815d550 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84ce600640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84c8161b70 0x7f84c813ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84ce600640 1 --2- 172.21.15.40:0/2961105213 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f84c8147830 0x7f84c813e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84ce600640 1 -- 172.21.15.40:0/2961105213 >> 172.21.15.40:0/2961105213 conn(0x7f84c80d0c20 msgr2=0x7f84c80b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:49.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84ce600640 1 -- 172.21.15.40:0/2961105213 shutdown_connections 2024-03-24T00:33:49.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.123+0000 7f84ce600640 1 -- 172.21.15.40:0/2961105213 wait complete. 2024-03-24T00:33:49.149 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:32:55.446633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '43', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:32:56.017210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '44', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:33:49.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:33:49.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.291+0000 7f53d5600640 1 Processor -- start 2024-03-24T00:33:49.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.291+0000 7f53d5600640 1 -- start start 2024-03-24T00:33:49.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.291+0000 7f53d5600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53d015e000 0x7f53d00d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:49.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.291+0000 7f53d5600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53d01628d0 0x7f53d00b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:49.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.291+0000 7f53d5600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d015e3e0 con 0x7f53d006f840 2024-03-24T00:33:49.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.291+0000 7f53d5600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d0161ba0 con 0x7f53d015e000 2024-03-24T00:33:49.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.291+0000 7f53d5600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d00b5f10 con 0x7f53d01628d0 2024-03-24T00:33:49.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.291+0000 7f53cea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53d015e000 0x7f53d00d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:49.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.291+0000 7f53cf400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f53d006f840 0x7f53d0071c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60854/0 (socket says 172.21.15.40:60854) 2024-03-24T00:33:49.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.291+0000 7f53cf400640 1 -- 172.21.15.40:0/3555290823 learned_addr learned my addr 172.21.15.40:0/3555290823 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:49.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.291+0000 7f53ce000640 1 --2- 172.21.15.40:0/3555290823 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53d01628d0 0x7f53d00b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:49.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53cea00640 1 -- 172.21.15.40:0/3555290823 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53d01628d0 msgr2=0x7f53d00b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:49.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53cea00640 1 --2- 172.21.15.40:0/3555290823 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53d01628d0 0x7f53d00b3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53cea00640 1 -- 172.21.15.40:0/3555290823 >> v1:172.21.15.40:6789/0 conn(0x7f53d006f840 legacy=0x7f53d0071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53cea00640 1 -- 172.21.15.40:0/3555290823 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d0161df0 con 0x7f53d015e000 2024-03-24T00:33:49.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53ce000640 1 --2- 172.21.15.40:0/3555290823 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53d01628d0 0x7f53d00b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:49.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53cea00640 1 --2- 172.21.15.40:0/3555290823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d015e000 0x7f53d00d16d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f53c00097c0 tx=0x7f53c001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdaf4e1e89aa4851 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:49.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53cd600640 1 -- 172.21.15.40:0/3555290823 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c00b0070 con 0x7f53d015e000 2024-03-24T00:33:49.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53cd600640 1 -- 172.21.15.40:0/3555290823 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c00ab600 con 0x7f53d015e000 2024-03-24T00:33:49.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53cd600640 1 -- 172.21.15.40:0/3555290823 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c00ab9d0 con 0x7f53d015e000 2024-03-24T00:33:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53d5600640 1 -- 172.21.15.40:0/3555290823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d015e000 msgr2=0x7f53d00d16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53d5600640 1 --2- 172.21.15.40:0/3555290823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d015e000 0x7f53d00d16d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f53c00097c0 tx=0x7f53c001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53d5600640 1 -- 172.21.15.40:0/3555290823 shutdown_connections 2024-03-24T00:33:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53d5600640 1 --2- 172.21.15.40:0/3555290823 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53d01628d0 0x7f53d00b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53d5600640 1 --2- 172.21.15.40:0/3555290823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d015e000 0x7f53d00d16d0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53d5600640 1 -- 172.21.15.40:0/3555290823 >> 172.21.15.40:0/3555290823 conn(0x7f53d00d0bb0 msgr2=0x7f53d00d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53d5600640 1 -- 172.21.15.40:0/3555290823 shutdown_connections 2024-03-24T00:33:49.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.295+0000 7f53d5600640 1 -- 172.21.15.40:0/3555290823 wait complete. 2024-03-24T00:33:49.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.299+0000 7f53d5600640 1 Processor -- start 2024-03-24T00:33:49.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.299+0000 7f53d5600640 1 -- start start 2024-03-24T00:33:49.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53d5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d006f840 0x7f53d014de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:49.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53d5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53d015e000 0x7f53d013e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:49.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53d5600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53d01628d0 0x7f53d015e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:49.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53d5600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d016b0c0 con 0x7f53d006f840 2024-03-24T00:33:49.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53d5600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d016af40 con 0x7f53d015e000 2024-03-24T00:33:49.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53d5600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d016b240 con 0x7f53d01628d0 2024-03-24T00:33:49.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53cea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d006f840 0x7f53d014de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:49.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53cea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d006f840 0x7f53d014de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40576/0 (socket says 172.21.15.40:40576) 2024-03-24T00:33:49.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53cea00640 1 -- 172.21.15.40:0/1166392620 learned_addr learned my addr 172.21.15.40:0/1166392620 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:49.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53ce000640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53d015e000 0x7f53d013e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:49.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53cf400640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53d01628d0 0x7f53d015e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:49.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53cea00640 1 -- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53d01628d0 msgr2=0x7f53d015e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53cea00640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53d01628d0 0x7f53d015e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53cea00640 1 -- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53d015e000 msgr2=0x7f53d013e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53cea00640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53d015e000 0x7f53d013e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53cea00640 1 -- 172.21.15.40:0/1166392620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d0151570 con 0x7f53d006f840 2024-03-24T00:33:49.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53cf400640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53d01628d0 0x7f53d015e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:49.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53cea00640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d006f840 0x7f53d014de50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f53c00097c0 tx=0x7f53c0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:49.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53c6a00640 1 -- 172.21.15.40:0/1166392620 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c00ba070 con 0x7f53d006f840 2024-03-24T00:33:49.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53c6a00640 1 -- 172.21.15.40:0/1166392620 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c00b43f0 con 0x7f53d006f840 2024-03-24T00:33:49.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53c6a00640 1 -- 172.21.15.40:0/1166392620 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c00b0040 con 0x7f53d006f840 2024-03-24T00:33:49.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53d5600640 1 -- 172.21.15.40:0/1166392620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d0151850 con 0x7f53d006f840 2024-03-24T00:33:49.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53d5600640 1 -- 172.21.15.40:0/1166392620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d0151e10 con 0x7f53d006f840 2024-03-24T00:33:49.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53d5600640 1 -- 172.21.15.40:0/1166392620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d0068170 con 0x7f53d006f840 2024-03-24T00:33:49.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.303+0000 7f53c6a00640 1 -- 172.21.15.40:0/1166392620 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f53c00b47c0 con 0x7f53d006f840 2024-03-24T00:33:49.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.307+0000 7f53c6a00640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53b0067660 0x7f53b0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:49.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.307+0000 7f53c6a00640 1 -- 172.21.15.40:0/1166392620 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f53c011ea50 con 0x7f53d006f840 2024-03-24T00:33:49.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.307+0000 7f53ce000640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53b0067660 0x7f53b0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:49.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.307+0000 7f53ce000640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53b0067660 0x7f53b0069b20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009580 tx=0x7f53bc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:49.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.311+0000 7f53c6a00640 1 -- 172.21.15.40:0/1166392620 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f53c0016090 con 0x7f53d006f840 2024-03-24T00:33:49.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53d5600640 1 -- 172.21.15.40:0/1166392620 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53d013e9b0 con 0x7f53d006f840 2024-03-24T00:33:49.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53c6a00640 1 -- 172.21.15.40:0/1166392620 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7f53c00eef50 con 0x7f53d006f840 2024-03-24T00:33:49.533 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:33:49.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53d5600640 1 -- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53b0067660 msgr2=0x7f53b0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53d5600640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53b0067660 0x7f53b0069b20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f53bc009580 tx=0x7f53bc009190 comp rx=0 tx=0).stop 2024-03-24T00:33:49.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53d5600640 1 -- 172.21.15.40:0/1166392620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d006f840 msgr2=0x7f53d014de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53d5600640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d006f840 0x7f53d014de50 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f53c00097c0 tx=0x7f53c0019240 comp rx=0 tx=0).stop 2024-03-24T00:33:49.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53d5600640 1 -- 172.21.15.40:0/1166392620 shutdown_connections 2024-03-24T00:33:49.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53d5600640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53b0067660 0x7f53b0069b20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53d5600640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53d01628d0 0x7f53d015e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53d5600640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53d015e000 0x7f53d013e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53d5600640 1 --2- 172.21.15.40:0/1166392620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53d006f840 0x7f53d014de50 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53d5600640 1 -- 172.21.15.40:0/1166392620 >> 172.21.15.40:0/1166392620 conn(0x7f53d00d0bb0 msgr2=0x7f53d0066e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.531+0000 7f53d5600640 1 -- 172.21.15.40:0/1166392620 shutdown_connections 2024-03-24T00:33:49.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.535+0000 7f53d5600640 1 -- 172.21.15.40:0/1166392620 wait complete. 2024-03-24T00:33:49.556 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:33:49.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.707+0000 7fad4d400640 1 Processor -- start 2024-03-24T00:33:49.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.707+0000 7fad4d400640 1 -- start start 2024-03-24T00:33:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.707+0000 7fad4d400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fad4815ffd0 0x7fad480ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.707+0000 7fad4d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad480728d0 0x7fad480cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.707+0000 7fad4d400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad48071de0 con 0x7fad4815bd00 2024-03-24T00:33:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.707+0000 7fad4d400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad480cf790 con 0x7fad480728d0 2024-03-24T00:33:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.707+0000 7fad4d400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad480cea50 con 0x7fad4815ffd0 2024-03-24T00:33:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad47200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad480728d0 0x7fad480cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad3de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fad4815ffd0 0x7fad480ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad46800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fad4815bd00 0x7fad4815f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:35348/0 (socket says 172.21.15.40:35348) 2024-03-24T00:33:49.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad47200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fad480728d0 0x7fad480cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40578/0 (socket says 172.21.15.40:40578) 2024-03-24T00:33:49.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad46800640 1 -- 172.21.15.40:0/2180857610 learned_addr learned my addr 172.21.15.40:0/2180857610 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:49.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad3de00640 1 -- 172.21.15.40:0/2180857610 >> v2:172.21.15.40:3300/0 conn(0x7fad480728d0 msgr2=0x7fad480cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:49.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad3de00640 1 --2- 172.21.15.40:0/2180857610 >> v2:172.21.15.40:3300/0 conn(0x7fad480728d0 0x7fad480cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad3de00640 1 -- 172.21.15.40:0/2180857610 >> v1:172.21.15.53:6789/0 conn(0x7fad4815bd00 legacy=0x7fad4815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad3de00640 1 -- 172.21.15.40:0/2180857610 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad480ced30 con 0x7fad4815ffd0 2024-03-24T00:33:49.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad3de00640 1 --2- 172.21.15.40:0/2180857610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad4815ffd0 0x7fad480ae8e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fad4806ff40 tx=0x7fad3001c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c3a4a523ce37b05 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:49.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad45e00640 1 -- 172.21.15.40:0/2180857610 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad300b2070 con 0x7fad4815ffd0 2024-03-24T00:33:49.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad45e00640 1 -- 172.21.15.40:0/2180857610 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad300a59a0 con 0x7fad4815ffd0 2024-03-24T00:33:49.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.711+0000 7fad45e00640 1 -- 172.21.15.40:0/2180857610 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad300a5d70 con 0x7fad4815ffd0 2024-03-24T00:33:49.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 -- 172.21.15.40:0/2180857610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad4815ffd0 msgr2=0x7fad480ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 --2- 172.21.15.40:0/2180857610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad4815ffd0 0x7fad480ae8e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fad4806ff40 tx=0x7fad3001c900 comp rx=0 tx=0).stop 2024-03-24T00:33:49.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 -- 172.21.15.40:0/2180857610 shutdown_connections 2024-03-24T00:33:49.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 --2- 172.21.15.40:0/2180857610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad4815ffd0 0x7fad480ae8e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 --2- 172.21.15.40:0/2180857610 >> v2:172.21.15.40:3300/0 conn(0x7fad480728d0 0x7fad480cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 -- 172.21.15.40:0/2180857610 >> 172.21.15.40:0/2180857610 conn(0x7fad480b1660 msgr2=0x7fad480b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:49.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 -- 172.21.15.40:0/2180857610 shutdown_connections 2024-03-24T00:33:49.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 -- 172.21.15.40:0/2180857610 wait complete. 2024-03-24T00:33:49.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 Processor -- start 2024-03-24T00:33:49.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 -- start start 2024-03-24T00:33:49.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad480728d0 0x7fad481485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:49.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad4815bd00 0x7fad48154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:49.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad4815ffd0 0x7fad480affc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:49.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4806d320 con 0x7fad4815ffd0 2024-03-24T00:33:49.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4815c0e0 con 0x7fad480728d0 2024-03-24T00:33:49.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad4d400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4806d4a0 con 0x7fad4815bd00 2024-03-24T00:33:49.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad47200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad4815ffd0 0x7fad480affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:49.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad46800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad480728d0 0x7fad481485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:49.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad47200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad4815ffd0 0x7fad480affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40580/0 (socket says 172.21.15.40:40580) 2024-03-24T00:33:49.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad47200640 1 -- 172.21.15.40:0/1094114791 learned_addr learned my addr 172.21.15.40:0/1094114791 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:49.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad3de00640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad4815bd00 0x7fad48154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:49.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad47200640 1 -- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad4815bd00 msgr2=0x7fad48154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad47200640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad4815bd00 0x7fad48154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad47200640 1 -- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad480728d0 msgr2=0x7fad481485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad47200640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad480728d0 0x7fad481485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad47200640 1 -- 172.21.15.40:0/1094114791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad480b27b0 con 0x7fad4815ffd0 2024-03-24T00:33:49.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad46800640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad480728d0 0x7fad481485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:49.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.715+0000 7fad47200640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad4815ffd0 0x7fad480affc0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fad4000e960 tx=0x7fad4000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:49.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.719+0000 7fad3f400640 1 -- 172.21.15.40:0/1094114791 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad40017070 con 0x7fad4815ffd0 2024-03-24T00:33:49.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.719+0000 7fad3f400640 1 -- 172.21.15.40:0/1094114791 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad4000c9a0 con 0x7fad4815ffd0 2024-03-24T00:33:49.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.719+0000 7fad4d400640 1 -- 172.21.15.40:0/1094114791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad480b2a90 con 0x7fad4815ffd0 2024-03-24T00:33:49.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.719+0000 7fad3f400640 1 -- 172.21.15.40:0/1094114791 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad4000cd70 con 0x7fad4815ffd0 2024-03-24T00:33:49.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.719+0000 7fad4d400640 1 -- 172.21.15.40:0/1094114791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad480b3050 con 0x7fad4815ffd0 2024-03-24T00:33:49.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.719+0000 7fad4d400640 1 -- 172.21.15.40:0/1094114791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad48068170 con 0x7fad4815ffd0 2024-03-24T00:33:49.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.719+0000 7fad3f400640 1 -- 172.21.15.40:0/1094114791 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fad4001a070 con 0x7fad4815ffd0 2024-03-24T00:33:49.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.723+0000 7fad3f400640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fad24067660 0x7fad24069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:49.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.723+0000 7fad3f400640 1 -- 172.21.15.40:0/1094114791 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fad400861b0 con 0x7fad4815ffd0 2024-03-24T00:33:49.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.723+0000 7fad46800640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fad24067660 0x7fad24069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:49.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.723+0000 7fad46800640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fad24067660 0x7fad24069b20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fad38000f70 tx=0x7fad38007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:49.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.723+0000 7fad3f400640 1 -- 172.21.15.40:0/1094114791 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fad40056620 con 0x7fad4815ffd0 2024-03-24T00:33:49.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.887+0000 7fad4d400640 1 -- 172.21.15.40:0/1094114791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fad4806e870 con 0x7fad4815ffd0 2024-03-24T00:33:49.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.887+0000 7fad3f400640 1 -- 172.21.15.40:0/1094114791 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v47) v1 ==== 104+0+11 (secure 0 0 0) 0x7fad4005a6c0 con 0x7fad4815ffd0 2024-03-24T00:33:49.890 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:33:49.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.891+0000 7fad4d400640 1 -- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fad24067660 msgr2=0x7fad24069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.891+0000 7fad4d400640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fad24067660 0x7fad24069b20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fad38000f70 tx=0x7fad38007e90 comp rx=0 tx=0).stop 2024-03-24T00:33:49.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.891+0000 7fad4d400640 1 -- 172.21.15.40:0/1094114791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad4815ffd0 msgr2=0x7fad480affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:49.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.891+0000 7fad4d400640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad4815ffd0 0x7fad480affc0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fad4000e960 tx=0x7fad4000ee20 comp rx=0 tx=0).stop 2024-03-24T00:33:49.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.891+0000 7fad4d400640 1 -- 172.21.15.40:0/1094114791 shutdown_connections 2024-03-24T00:33:49.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.891+0000 7fad4d400640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fad24067660 0x7fad24069b20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.891+0000 7fad4d400640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fad4815ffd0 0x7fad480affc0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.891+0000 7fad4d400640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad4815bd00 0x7fad48154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.891+0000 7fad4d400640 1 --2- 172.21.15.40:0/1094114791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad480728d0 0x7fad481485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:49.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.891+0000 7fad4d400640 1 -- 172.21.15.40:0/1094114791 >> 172.21.15.40:0/1094114791 conn(0x7fad480b1660 msgr2=0x7fad480cf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:49.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.891+0000 7fad4d400640 1 -- 172.21.15.40:0/1094114791 shutdown_connections 2024-03-24T00:33:49.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:49.891+0000 7fad4d400640 1 -- 172.21.15.40:0/1094114791 wait complete. 2024-03-24T00:33:49.912 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:33:50.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.083+0000 7f0132400640 1 Processor -- start 2024-03-24T00:33:50.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.083+0000 7f0132400640 1 -- start start 2024-03-24T00:33:50.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.083+0000 7f0132400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f012c15bd00 0x7f012c0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.083+0000 7f0132400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c072d30 con 0x7f012c15ffd0 2024-03-24T00:33:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.083+0000 7f0132400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c071de0 con 0x7f012c0728d0 2024-03-24T00:33:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.083+0000 7f0132400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012c0cea50 con 0x7f012c15bd00 2024-03-24T00:33:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.083+0000 7f012aa00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f012c0728d0 0x7f012c15f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:35356/0 (socket says 172.21.15.40:35356) 2024-03-24T00:33:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.083+0000 7f012aa00640 1 -- 172.21.15.40:0/312175553 learned_addr learned my addr 172.21.15.40:0/312175553 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:50.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.083+0000 7f012b400640 1 --2- 172.21.15.40:0/312175553 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f012c15bd00 0x7f012c0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.087+0000 7f012a000640 1 -- 172.21.15.40:0/312175553 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280261162 0 0) 0x7f012c071de0 con 0x7f012c0728d0 2024-03-24T00:33:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.087+0000 7f012a000640 1 -- 172.21.15.40:0/312175553 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0110003530 con 0x7f012c0728d0 2024-03-24T00:33:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.087+0000 7f012a000640 1 -- 172.21.15.40:0/312175553 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909381625 0 0) 0x7f012c072d30 con 0x7f012c15ffd0 2024-03-24T00:33:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.087+0000 7f012a000640 1 -- 172.21.15.40:0/312175553 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f012c071de0 con 0x7f012c15ffd0 2024-03-24T00:33:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.087+0000 7f012b400640 1 -- 172.21.15.40:0/312175553 >> v1:172.21.15.53:6789/0 conn(0x7f012c0728d0 legacy=0x7f012c15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.087+0000 7f012b400640 1 -- 172.21.15.40:0/312175553 >> v1:172.21.15.40:6789/0 conn(0x7f012c15ffd0 legacy=0x7f012c06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.087+0000 7f012b400640 1 -- 172.21.15.40:0/312175553 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c0ced30 con 0x7f012c15bd00 2024-03-24T00:33:50.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.087+0000 7f012b400640 1 --2- 172.21.15.40:0/312175553 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f012c15bd00 0x7f012c0ae8e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f011c0097f0 tx=0x7f011c01c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86b3245a1b64e5ba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:50.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.087+0000 7f012a000640 1 -- 172.21.15.40:0/312175553 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c0aa5f0 con 0x7f012c15bd00 2024-03-24T00:33:50.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.087+0000 7f012a000640 1 -- 172.21.15.40:0/312175553 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c0af070 con 0x7f012c15bd00 2024-03-24T00:33:50.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.087+0000 7f012a000640 1 -- 172.21.15.40:0/312175553 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c0aa9b0 con 0x7f012c15bd00 2024-03-24T00:33:50.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 -- 172.21.15.40:0/312175553 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f012c15bd00 msgr2=0x7f012c0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 --2- 172.21.15.40:0/312175553 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f012c15bd00 0x7f012c0ae8e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f011c0097f0 tx=0x7f011c01c4b0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 -- 172.21.15.40:0/312175553 shutdown_connections 2024-03-24T00:33:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 --2- 172.21.15.40:0/312175553 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f012c15bd00 0x7f012c0ae8e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 -- 172.21.15.40:0/312175553 >> 172.21.15.40:0/312175553 conn(0x7f012c0b1660 msgr2=0x7f012c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 -- 172.21.15.40:0/312175553 shutdown_connections 2024-03-24T00:33:50.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 -- 172.21.15.40:0/312175553 wait complete. 2024-03-24T00:33:50.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 Processor -- start 2024-03-24T00:33:50.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 -- start start 2024-03-24T00:33:50.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f012c0728d0 0x7f012c147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f012c15bd00 0x7f012c149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f012c15ffd0 0x7f012c1473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012c15c0e0 con 0x7f012c15ffd0 2024-03-24T00:33:50.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012c0cf660 con 0x7f012c0728d0 2024-03-24T00:33:50.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f0132400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012c16a420 con 0x7f012c15bd00 2024-03-24T00:33:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f012b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f012c0728d0 0x7f012c147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f012aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f012c15bd00 0x7f012c149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f012be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f012c15ffd0 0x7f012c1473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f012b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f012c0728d0 0x7f012c147bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:35876/0 (socket says 172.21.15.40:35876) 2024-03-24T00:33:50.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.091+0000 7f012b400640 1 -- 172.21.15.40:0/3709784795 learned_addr learned my addr 172.21.15.40:0/3709784795 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f012b400640 1 -- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f012c15bd00 msgr2=0x7f012c149a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f012b400640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f012c15bd00 0x7f012c149a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f012b400640 1 -- 172.21.15.40:0/3709784795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f012c15ffd0 msgr2=0x7f012c1473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f012b400640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f012c15ffd0 0x7f012c1473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f012b400640 1 -- 172.21.15.40:0/3709784795 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c151530 con 0x7f012c0728d0 2024-03-24T00:33:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f012aa00640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f012c15bd00 0x7f012c149a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:50.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f012b400640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f012c0728d0 0x7f012c147bd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f011c0097f0 tx=0x7f011c01acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:50.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f0123400640 1 -- 172.21.15.40:0/3709784795 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c0be410 con 0x7f012c0728d0 2024-03-24T00:33:50.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f0123400640 1 -- 172.21.15.40:0/3709784795 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c0be5b0 con 0x7f012c0728d0 2024-03-24T00:33:50.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f0123400640 1 -- 172.21.15.40:0/3709784795 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c0be9a0 con 0x7f012c0728d0 2024-03-24T00:33:50.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f0132400640 1 -- 172.21.15.40:0/3709784795 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012c151810 con 0x7f012c0728d0 2024-03-24T00:33:50.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f0132400640 1 -- 172.21.15.40:0/3709784795 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f012c151dd0 con 0x7f012c0728d0 2024-03-24T00:33:50.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f0132400640 1 -- 172.21.15.40:0/3709784795 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012c068170 con 0x7f012c0728d0 2024-03-24T00:33:50.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.095+0000 7f0123400640 1 -- 172.21.15.40:0/3709784795 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f011c0bc020 con 0x7f012c0728d0 2024-03-24T00:33:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.099+0000 7f0123400640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f010c067660 0x7f010c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.099+0000 7f0123400640 1 -- 172.21.15.40:0/3709784795 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f011c120270 con 0x7f012c0728d0 2024-03-24T00:33:50.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.099+0000 7f012aa00640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f010c067660 0x7f010c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.099+0000 7f012aa00640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f010c067660 0x7f010c069b20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f012c14c4e0 tx=0x7f0118007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.107+0000 7f0123400640 1 -- 172.21.15.40:0/3709784795 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f011c0c3060 con 0x7f012c0728d0 2024-03-24T00:33:50.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.267+0000 7f0132400640 1 -- 172.21.15.40:0/3709784795 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f012c06e870 con 0x7f012c0728d0 2024-03-24T00:33:50.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.267+0000 7f0123400640 1 -- 172.21.15.40:0/3709784795 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v47) v1 ==== 100+0+11 (secure 0 0 0) 0x7f011c0160d0 con 0x7f012c0728d0 2024-03-24T00:33:50.272 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:33:50.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.271+0000 7f0132400640 1 -- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f010c067660 msgr2=0x7f010c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.271+0000 7f0132400640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f010c067660 0x7f010c069b20 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f012c14c4e0 tx=0x7f0118007d80 comp rx=0 tx=0).stop 2024-03-24T00:33:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.271+0000 7f0132400640 1 -- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f012c0728d0 msgr2=0x7f012c147bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.271+0000 7f0132400640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f012c0728d0 0x7f012c147bd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f011c0097f0 tx=0x7f011c01acf0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.271+0000 7f0132400640 1 -- 172.21.15.40:0/3709784795 shutdown_connections 2024-03-24T00:33:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.271+0000 7f0132400640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f010c067660 0x7f010c069b20 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.271+0000 7f0132400640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f012c15ffd0 0x7f012c1473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.271+0000 7f0132400640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f012c15bd00 0x7f012c149a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.271+0000 7f0132400640 1 --2- 172.21.15.40:0/3709784795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f012c0728d0 0x7f012c147bd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.271+0000 7f0132400640 1 -- 172.21.15.40:0/3709784795 >> 172.21.15.40:0/3709784795 conn(0x7f012c0b1660 msgr2=0x7f012c0cf7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.271+0000 7f0132400640 1 -- 172.21.15.40:0/3709784795 shutdown_connections 2024-03-24T00:33:50.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.271+0000 7f0132400640 1 -- 172.21.15.40:0/3709784795 wait complete. 2024-03-24T00:33:50.303 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:50.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7fef06400640 1 Processor -- start 2024-03-24T00:33:50.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7fef06400640 1 -- start start 2024-03-24T00:33:50.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7fef06400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fef0006e5a0 0x7fef0006e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7fef06400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef00070c80 0x7fef0006db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7fef06400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef00071690 0x7fef000b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7fef06400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0006cdc0 con 0x7fef00071690 2024-03-24T00:33:50.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7fef06400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0006eec0 con 0x7fef0006e5a0 2024-03-24T00:33:50.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7fef06400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef00071160 con 0x7fef00070c80 2024-03-24T00:33:50.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7feeff400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fef0006e5a0 0x7fef0006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7feeffe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef00071690 0x7fef000b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7feeff400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fef0006e5a0 0x7fef0006e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:35884/0 (socket says 172.21.15.40:35884) 2024-03-24T00:33:50.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7feef7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef00070c80 0x7fef0006db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7feeff400640 1 -- 172.21.15.40:0/3361557457 learned_addr learned my addr 172.21.15.40:0/3361557457 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:50.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7feeffe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fef00071690 0x7fef000b33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40590/0 (socket says 172.21.15.40:40590) 2024-03-24T00:33:50.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7feeff400640 1 -- 172.21.15.40:0/3361557457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef00070c80 msgr2=0x7fef0006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:50.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7feeff400640 1 --2- 172.21.15.40:0/3361557457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef00070c80 0x7fef0006db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7feeff400640 1 -- 172.21.15.40:0/3361557457 >> v2:172.21.15.40:3300/0 conn(0x7fef00071690 msgr2=0x7fef000b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:50.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7feeff400640 1 --2- 172.21.15.40:0/3361557457 >> v2:172.21.15.40:3300/0 conn(0x7fef00071690 0x7fef000b33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7feeff400640 1 -- 172.21.15.40:0/3361557457 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0006d040 con 0x7fef0006e5a0 2024-03-24T00:33:50.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.443+0000 7feeffe00640 1 --2- 172.21.15.40:0/3361557457 >> v2:172.21.15.40:3300/0 conn(0x7fef00071690 0x7fef000b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7feeff400640 1 --2- 172.21.15.40:0/3361557457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0006e5a0 0x7fef0006e980 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fef000b3390 tx=0x7feef001c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a122e80970df182 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7feefea00640 1 -- 172.21.15.40:0/3361557457 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef00b0070 con 0x7fef0006e5a0 2024-03-24T00:33:50.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7feefea00640 1 -- 172.21.15.40:0/3361557457 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef00ab600 con 0x7fef0006e5a0 2024-03-24T00:33:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7feefea00640 1 -- 172.21.15.40:0/3361557457 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef00ab9d0 con 0x7fef0006e5a0 2024-03-24T00:33:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7fef06400640 1 -- 172.21.15.40:0/3361557457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0006e5a0 msgr2=0x7fef0006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7fef06400640 1 --2- 172.21.15.40:0/3361557457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0006e5a0 0x7fef0006e980 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fef000b3390 tx=0x7feef001c520 comp rx=0 tx=0).stop 2024-03-24T00:33:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7fef06400640 1 -- 172.21.15.40:0/3361557457 shutdown_connections 2024-03-24T00:33:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7fef06400640 1 --2- 172.21.15.40:0/3361557457 >> v2:172.21.15.40:3300/0 conn(0x7fef00071690 0x7fef000b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7fef06400640 1 --2- 172.21.15.40:0/3361557457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef00070c80 0x7fef0006db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7fef06400640 1 --2- 172.21.15.40:0/3361557457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0006e5a0 0x7fef0006e980 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7fef06400640 1 -- 172.21.15.40:0/3361557457 >> 172.21.15.40:0/3361557457 conn(0x7fef000b17b0 msgr2=0x7fef000b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7fef06400640 1 -- 172.21.15.40:0/3361557457 shutdown_connections 2024-03-24T00:33:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.447+0000 7fef06400640 1 -- 172.21.15.40:0/3361557457 wait complete. 2024-03-24T00:33:50.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7fef06400640 1 Processor -- start 2024-03-24T00:33:50.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7fef06400640 1 -- start start 2024-03-24T00:33:50.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7fef06400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0006e5a0 0x7fef00151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7fef06400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef00070c80 0x7fef001568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7fef06400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00071690 0x7fef00152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7fef06400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef00155460 con 0x7fef00071690 2024-03-24T00:33:50.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7fef06400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef001552e0 con 0x7fef0006e5a0 2024-03-24T00:33:50.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7fef06400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef001623a0 con 0x7fef00070c80 2024-03-24T00:33:50.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feef7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef00070c80 0x7fef001568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feeffe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00071690 0x7fef00152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feeff400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0006e5a0 0x7fef00151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feeff400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0006e5a0 0x7fef00151c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:35892/0 (socket says 172.21.15.40:35892) 2024-03-24T00:33:50.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feef7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef00070c80 0x7fef001568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:51704/0 (socket says 172.21.15.40:51704) 2024-03-24T00:33:50.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feeffe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00071690 0x7fef00152280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40596/0 (socket says 172.21.15.40:40596) 2024-03-24T00:33:50.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feeff400640 1 -- 172.21.15.40:0/181170960 learned_addr learned my addr 172.21.15.40:0/181170960 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:50.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feeff400640 1 -- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef00070c80 msgr2=0x7fef001568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feeff400640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef00070c80 0x7fef001568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feeff400640 1 -- 172.21.15.40:0/181170960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00071690 msgr2=0x7fef00152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feeff400640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00071690 0x7fef00152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feeff400640 1 -- 172.21.15.40:0/181170960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef00157ee0 con 0x7fef0006e5a0 2024-03-24T00:33:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feeffe00640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00071690 0x7fef00152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feeff400640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0006e5a0 0x7fef00151c90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7feef00ae040 tx=0x7feef00192c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feef7400640 1 -- 172.21.15.40:0/181170960 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef00b0040 con 0x7fef0006e5a0 2024-03-24T00:33:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feef7400640 1 -- 172.21.15.40:0/181170960 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feef00b4420 con 0x7fef0006e5a0 2024-03-24T00:33:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7fef06400640 1 -- 172.21.15.40:0/181170960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef00157920 con 0x7fef0006e5a0 2024-03-24T00:33:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7feef7400640 1 -- 172.21.15.40:0/181170960 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef00b4810 con 0x7fef0006e5a0 2024-03-24T00:33:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.451+0000 7fef06400640 1 -- 172.21.15.40:0/181170960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef001560e0 con 0x7fef0006e5a0 2024-03-24T00:33:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.455+0000 7fef06400640 1 -- 172.21.15.40:0/181170960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef00068170 con 0x7fef0006e5a0 2024-03-24T00:33:50.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.455+0000 7feef7400640 1 -- 172.21.15.40:0/181170960 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7feef00b49b0 con 0x7fef0006e5a0 2024-03-24T00:33:50.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.455+0000 7feef7400640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feedc067690 0x7feedc069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.455+0000 7feef7400640 1 -- 172.21.15.40:0/181170960 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5853+0+0 (secure 0 0 0) 0x7feef011f440 con 0x7fef0006e5a0 2024-03-24T00:33:50.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.455+0000 7feef7e00640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feedc067690 0x7feedc069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.459+0000 7feef7e00640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feedc067690 0x7feedc069b50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fef001544b0 tx=0x7feee8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:50.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.463+0000 7feef7400640 1 -- 172.21.15.40:0/181170960 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7feef0016110 con 0x7fef0006e5a0 2024-03-24T00:33:50.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.659+0000 7fef06400640 1 -- 172.21.15.40:0/181170960 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef000cf4e0 con 0x7fef0006e5a0 2024-03-24T00:33:50.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.659+0000 7feef7400640 1 -- 172.21.15.40:0/181170960 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5014 (secure 0 0 0) 0x7feef00f0020 con 0x7fef0006e5a0 2024-03-24T00:33:50.663 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:50.663 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":42,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/374985095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":374985095},{"type":"v1","addr":"172.21.15.53:6836","nonce":374985095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:31.903823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5763,"mds_1":5192,"mds_2":5198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"c","rank":1,"incarnation":38,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5198":{"gid":5198,"name":"b","rank":2,"incarnation":34,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5192,5198,5763]},"id":3}]} 2024-03-24T00:33:50.663 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 42 2024-03-24T00:33:50.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.663+0000 7fef06400640 1 -- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feedc067690 msgr2=0x7feedc069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.663+0000 7fef06400640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feedc067690 0x7feedc069b50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fef001544b0 tx=0x7feee8009190 comp rx=0 tx=0).stop 2024-03-24T00:33:50.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.663+0000 7fef06400640 1 -- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0006e5a0 msgr2=0x7fef00151c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.663+0000 7fef06400640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0006e5a0 0x7fef00151c90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7feef00ae040 tx=0x7feef00192c0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.663+0000 7fef06400640 1 -- 172.21.15.40:0/181170960 shutdown_connections 2024-03-24T00:33:50.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.663+0000 7fef06400640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feedc067690 0x7feedc069b50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.663+0000 7fef06400640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fef00071690 0x7fef00152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.663+0000 7fef06400640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fef00070c80 0x7fef001568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.663+0000 7fef06400640 1 --2- 172.21.15.40:0/181170960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0006e5a0 0x7fef00151c90 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.663+0000 7fef06400640 1 -- 172.21.15.40:0/181170960 >> 172.21.15.40:0/181170960 conn(0x7fef000b17b0 msgr2=0x7fef0015cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:50.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.663+0000 7fef06400640 1 -- 172.21.15.40:0/181170960 shutdown_connections 2024-03-24T00:33:50.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.663+0000 7fef06400640 1 -- 172.21.15.40:0/181170960 wait complete. 2024-03-24T00:33:50.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:33:50.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02fe00640 1 Processor -- start 2024-03-24T00:33:50.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02fe00640 1 -- start start 2024-03-24T00:33:50.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0300cea10 0x7fb0300d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02fe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03006c180 con 0x7fb0300cfdc0 2024-03-24T00:33:50.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0300cc190 con 0x7fb0300ce630 2024-03-24T00:33:50.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0300d1080 con 0x7fb0300cea10 2024-03-24T00:33:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb0300cfdc0 0x7fb03006ba70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60880/0 (socket says 172.21.15.40:60880) 2024-03-24T00:33:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02e000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0300cea10 0x7fb0300d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02ea00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb0300ce630 0x7fb03006b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:35368/0 (socket says 172.21.15.40:35368) 2024-03-24T00:33:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02f400640 1 -- 172.21.15.40:0/1618870932 learned_addr learned my addr 172.21.15.40:0/1618870932 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02e000640 1 -- 172.21.15.40:0/1618870932 >> v1:172.21.15.53:6789/0 conn(0x7fb0300ce630 legacy=0x7fb03006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02d600640 1 -- 172.21.15.40:0/1618870932 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86028682 0 0) 0x7fb0300cc190 con 0x7fb0300ce630 2024-03-24T00:33:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02e000640 1 -- 172.21.15.40:0/1618870932 >> v1:172.21.15.40:6789/0 conn(0x7fb0300cfdc0 legacy=0x7fb03006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.835+0000 7fb02e000640 1 -- 172.21.15.40:0/1618870932 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0300d0da0 con 0x7fb0300cea10 2024-03-24T00:33:50.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02e000640 1 --2- 172.21.15.40:0/1618870932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0300cea10 0x7fb0300d2590 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0097f0 tx=0x7fb01c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ce67e324d34b165 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:50.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02d600640 1 -- 172.21.15.40:0/1618870932 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c0b1070 con 0x7fb0300cea10 2024-03-24T00:33:50.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02d600640 1 -- 172.21.15.40:0/1618870932 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c0ac600 con 0x7fb0300cea10 2024-03-24T00:33:50.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02d600640 1 -- 172.21.15.40:0/1618870932 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c0ac9d0 con 0x7fb0300cea10 2024-03-24T00:33:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02fe00640 1 -- 172.21.15.40:0/1618870932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0300cea10 msgr2=0x7fb0300d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02fe00640 1 --2- 172.21.15.40:0/1618870932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0300cea10 0x7fb0300d2590 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0097f0 tx=0x7fb01c01c400 comp rx=0 tx=0).stop 2024-03-24T00:33:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02fe00640 1 -- 172.21.15.40:0/1618870932 shutdown_connections 2024-03-24T00:33:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02fe00640 1 --2- 172.21.15.40:0/1618870932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0300cea10 0x7fb0300d2590 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02fe00640 1 -- 172.21.15.40:0/1618870932 >> 172.21.15.40:0/1618870932 conn(0x7fb0300d1aa0 msgr2=0x7fb0300d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02fe00640 1 -- 172.21.15.40:0/1618870932 shutdown_connections 2024-03-24T00:33:50.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02fe00640 1 -- 172.21.15.40:0/1618870932 wait complete. 2024-03-24T00:33:50.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02fe00640 1 Processor -- start 2024-03-24T00:33:50.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.839+0000 7fb02fe00640 1 -- start start 2024-03-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0300ce630 0x7fb0300c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0300cea10 0x7fb0301f4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0300cfdc0 0x7fb0300c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03006c4c0 con 0x7fb0300cea10 2024-03-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0300cc710 con 0x7fb0300ce630 2024-03-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03006c640 con 0x7fb0300cfdc0 2024-03-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0300cea10 0x7fb0301f4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0300ce630 0x7fb0300c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0300cea10 0x7fb0301f4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40600/0 (socket says 172.21.15.40:40600) 2024-03-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02e000640 1 -- 172.21.15.40:0/3028871944 learned_addr learned my addr 172.21.15.40:0/3028871944 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02f400640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0300cfdc0 0x7fb0300c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02e000640 1 -- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0300cfdc0 msgr2=0x7fb0300c7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02e000640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0300cfdc0 0x7fb0300c7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02e000640 1 -- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0300ce630 msgr2=0x7fb0300c9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:50.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02e000640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0300ce630 0x7fb0300c9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02e000640 1 -- 172.21.15.40:0/3028871944 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0300ca020 con 0x7fb0300cea10 2024-03-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02e000640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0300cea10 0x7fb0301f4500 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0097f0 tx=0x7fb01c01ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb026a00640 1 -- 172.21.15.40:0/3028871944 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c0b1040 con 0x7fb0300cea10 2024-03-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb026a00640 1 -- 172.21.15.40:0/3028871944 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c0bd070 con 0x7fb0300cea10 2024-03-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02fe00640 1 -- 172.21.15.40:0/3028871944 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0300ca300 con 0x7fb0300cea10 2024-03-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb026a00640 1 -- 172.21.15.40:0/3028871944 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c0abc10 con 0x7fb0300cea10 2024-03-24T00:33:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02fe00640 1 -- 172.21.15.40:0/3028871944 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0300ca8c0 con 0x7fb0300cea10 2024-03-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb02fe00640 1 -- 172.21.15.40:0/3028871944 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03009fb10 con 0x7fb0300cea10 2024-03-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.843+0000 7fb026a00640 1 -- 172.21.15.40:0/3028871944 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb01c0ac420 con 0x7fb0300cea10 2024-03-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.847+0000 7fb026a00640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb010067660 0x7fb010069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.847+0000 7fb026a00640 1 -- 172.21.15.40:0/3028871944 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb01c1201a0 con 0x7fb0300cea10 2024-03-24T00:33:50.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.847+0000 7fb02ea00640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb010067660 0x7fb010069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:50.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.847+0000 7fb02ea00640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb010067660 0x7fb010069b20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb020000f70 tx=0x7fb020007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:50.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:50.851+0000 7fb026a00640 1 -- 172.21.15.40:0/3028871944 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb01c015ff0 con 0x7fb0300cea10 2024-03-24T00:33:51.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.015+0000 7fb02fe00640 1 -- 172.21.15.40:0/3028871944 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb03004f4f0 con 0x7fb0300cea10 2024-03-24T00:33:51.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.015+0000 7fb026a00640 1 -- 172.21.15.40:0/3028871944 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16773 (secure 0 0 0) 0x7fb01c0f0850 con 0x7fb0300cea10 2024-03-24T00:33:51.018 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:51.018 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":47,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:33:48.936722+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:55.446633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:56.017210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.37","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6836/1600357100":"2024-03-25T00:33:27.880035+0000","172.21.15.53:6834/1600357100":"2024-03-25T00:33:27.880035+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:33:51.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.015+0000 7fb02fe00640 1 -- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb010067660 msgr2=0x7fb010069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.015+0000 7fb02fe00640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb010067660 0x7fb010069b20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb020000f70 tx=0x7fb020007df0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.015+0000 7fb02fe00640 1 -- 172.21.15.40:0/3028871944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0300cea10 msgr2=0x7fb0301f4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.015+0000 7fb02fe00640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0300cea10 0x7fb0301f4500 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb01c0097f0 tx=0x7fb01c01ac40 comp rx=0 tx=0).stop 2024-03-24T00:33:51.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.019+0000 7fb02fe00640 1 -- 172.21.15.40:0/3028871944 shutdown_connections 2024-03-24T00:33:51.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.019+0000 7fb02fe00640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb010067660 0x7fb010069b20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.019+0000 7fb02fe00640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0300cfdc0 0x7fb0300c7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.019+0000 7fb02fe00640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0300cea10 0x7fb0301f4500 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.019+0000 7fb02fe00640 1 --2- 172.21.15.40:0/3028871944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0300ce630 0x7fb0300c9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.019+0000 7fb02fe00640 1 -- 172.21.15.40:0/3028871944 >> 172.21.15.40:0/3028871944 conn(0x7fb0300d1aa0 msgr2=0x7fb0300703a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:51.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.019+0000 7fb02fe00640 1 -- 172.21.15.40:0/3028871944 shutdown_connections 2024-03-24T00:33:51.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.019+0000 7fb02fe00640 1 -- 172.21.15.40:0/3028871944 wait complete. 2024-03-24T00:33:51.045 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:33:51.045 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:51.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.183+0000 7fd3baa00640 1 Processor -- start 2024-03-24T00:33:51.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.183+0000 7fd3baa00640 1 -- start start 2024-03-24T00:33:51.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3baa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd3b40cce80 0x7fd3b40cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3baa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd3b40afc50 0x7fd3b40cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3baa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b40cd7a0 con 0x7fd3b406d5c0 2024-03-24T00:33:51.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3baa00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b40cde10 con 0x7fd3b40afc50 2024-03-24T00:33:51.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3baa00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3b40ce0f0 con 0x7fd3b40cce80 2024-03-24T00:33:51.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3abe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd3b40afc50 0x7fd3b40cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3abe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd3b40afc50 0x7fd3b40cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40612/0 (socket says 172.21.15.40:40612) 2024-03-24T00:33:51.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3b3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd3b40cce80 0x7fd3b40cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3abe00640 1 -- 172.21.15.40:0/1555653083 learned_addr learned my addr 172.21.15.40:0/1555653083 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:51.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3abe00640 1 -- 172.21.15.40:0/1555653083 >> v2:172.21.15.53:3300/0 conn(0x7fd3b40cce80 msgr2=0x7fd3b40cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:51.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3abe00640 1 --2- 172.21.15.40:0/1555653083 >> v2:172.21.15.53:3300/0 conn(0x7fd3b40cce80 0x7fd3b40cfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3abe00640 1 -- 172.21.15.40:0/1555653083 >> v1:172.21.15.53:6789/0 conn(0x7fd3b406d5c0 legacy=0x7fd3b415ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3abe00640 1 -- 172.21.15.40:0/1555653083 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b40ce3d0 con 0x7fd3b40afc50 2024-03-24T00:33:51.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3abe00640 1 --2- 172.21.15.40:0/1555653083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b40afc50 0x7fd3b40cd260 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40729d0 tx=0x7fd39c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af5bc12ea487093 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:51.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.187+0000 7fd3b3400640 1 -- 172.21.15.40:0/1555653083 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c0b1070 con 0x7fd3b40afc50 2024-03-24T00:33:51.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3b3400640 1 -- 172.21.15.40:0/1555653083 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39c0ac600 con 0x7fd3b40afc50 2024-03-24T00:33:51.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3b3400640 1 -- 172.21.15.40:0/1555653083 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39c0ac9d0 con 0x7fd3b40afc50 2024-03-24T00:33:51.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 -- 172.21.15.40:0/1555653083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b40afc50 msgr2=0x7fd3b40cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 --2- 172.21.15.40:0/1555653083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b40afc50 0x7fd3b40cd260 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40729d0 tx=0x7fd39c01c490 comp rx=0 tx=0).stop 2024-03-24T00:33:51.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 -- 172.21.15.40:0/1555653083 shutdown_connections 2024-03-24T00:33:51.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 --2- 172.21.15.40:0/1555653083 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b40afc50 0x7fd3b40cd260 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 --2- 172.21.15.40:0/1555653083 >> v2:172.21.15.53:3300/0 conn(0x7fd3b40cce80 0x7fd3b40cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 -- 172.21.15.40:0/1555653083 >> 172.21.15.40:0/1555653083 conn(0x7fd3b40b21a0 msgr2=0x7fd3b40b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:51.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 -- 172.21.15.40:0/1555653083 shutdown_connections 2024-03-24T00:33:51.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 -- 172.21.15.40:0/1555653083 wait complete. 2024-03-24T00:33:51.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 Processor -- start 2024-03-24T00:33:51.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 -- start start 2024-03-24T00:33:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b406d5c0 0x7fd3b415b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b40afc50 0x7fd3b41f7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3b40cce80 0x7fd3b4151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b4159eb0 con 0x7fd3b40afc50 2024-03-24T00:33:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b4159d30 con 0x7fd3b40cce80 2024-03-24T00:33:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3baa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b41695e0 con 0x7fd3b406d5c0 2024-03-24T00:33:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b40afc50 0x7fd3b41f7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b40afc50 0x7fd3b41f7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40622/0 (socket says 172.21.15.40:40622) 2024-03-24T00:33:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3abe00640 1 -- 172.21.15.40:0/1438453138 learned_addr learned my addr 172.21.15.40:0/1438453138 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3b3e00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b406d5c0 0x7fd3b415b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.191+0000 7fd3b8a00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3b40cce80 0x7fd3b4151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3b3e00640 1 -- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3b40cce80 msgr2=0x7fd3b4151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3b3e00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3b40cce80 0x7fd3b4151f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3b3e00640 1 -- 172.21.15.40:0/1438453138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b40afc50 msgr2=0x7fd3b41f7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3b3e00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b40afc50 0x7fd3b41f7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3b3e00640 1 -- 172.21.15.40:0/1438453138 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b41524b0 con 0x7fd3b406d5c0 2024-03-24T00:33:51.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3abe00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b40afc50 0x7fd3b41f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:51.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3b3e00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b406d5c0 0x7fd3b415b1e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400e8a0 tx=0x7fd3a400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:51.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3b0c00640 1 -- 172.21.15.40:0/1438453138 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4017070 con 0x7fd3b406d5c0 2024-03-24T00:33:51.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3b0c00640 1 -- 172.21.15.40:0/1438453138 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a400c940 con 0x7fd3b406d5c0 2024-03-24T00:33:51.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3baa00640 1 -- 172.21.15.40:0/1438453138 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b4158ce0 con 0x7fd3b406d5c0 2024-03-24T00:33:51.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3b0c00640 1 -- 172.21.15.40:0/1438453138 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a400ccf0 con 0x7fd3b406d5c0 2024-03-24T00:33:51.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3baa00640 1 -- 172.21.15.40:0/1438453138 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b4158200 con 0x7fd3b406d5c0 2024-03-24T00:33:51.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3baa00640 1 -- 172.21.15.40:0/1438453138 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b4068170 con 0x7fd3b406d5c0 2024-03-24T00:33:51.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3b0c00640 1 -- 172.21.15.40:0/1438453138 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd3a401a070 con 0x7fd3b406d5c0 2024-03-24T00:33:51.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3b0c00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd390067660 0x7fd390069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3b0c00640 1 -- 172.21.15.40:0/1438453138 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd3a4085980 con 0x7fd3b406d5c0 2024-03-24T00:33:51.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.195+0000 7fd3abe00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd390067660 0x7fd390069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.199+0000 7fd3abe00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd390067660 0x7fd390069b20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd39c000c10 tx=0x7fd39c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.203+0000 7fd3b0c00640 1 -- 172.21.15.40:0/1438453138 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd3a4055e80 con 0x7fd3b406d5c0 2024-03-24T00:33:51.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.395+0000 7fd3baa00640 1 -- 172.21.15.40:0/1438453138 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3b4074a00 con 0x7fd3b406d5c0 2024-03-24T00:33:51.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.399+0000 7fd3b0c00640 1 -- 172.21.15.40:0/1438453138 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5014 (secure 0 0 0) 0x7fd3a4059f20 con 0x7fd3b406d5c0 2024-03-24T00:33:51.401 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:51.401 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":42,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/374985095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":374985095},{"type":"v1","addr":"172.21.15.53:6836","nonce":374985095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:31.903823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5763,"mds_1":5192,"mds_2":5198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"c","rank":1,"incarnation":38,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5198":{"gid":5198,"name":"b","rank":2,"incarnation":34,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5192,5198,5763]},"id":3}]} 2024-03-24T00:33:51.401 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 42 2024-03-24T00:33:51.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.399+0000 7fd3baa00640 1 -- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd390067660 msgr2=0x7fd390069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.399+0000 7fd3baa00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd390067660 0x7fd390069b20 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd39c000c10 tx=0x7fd39c0a7000 comp rx=0 tx=0).stop 2024-03-24T00:33:51.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.399+0000 7fd3baa00640 1 -- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b406d5c0 msgr2=0x7fd3b415b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.399+0000 7fd3baa00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b406d5c0 0x7fd3b415b1e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400e8a0 tx=0x7fd3a400ed60 comp rx=0 tx=0).stop 2024-03-24T00:33:51.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.399+0000 7fd3baa00640 1 -- 172.21.15.40:0/1438453138 shutdown_connections 2024-03-24T00:33:51.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.399+0000 7fd3baa00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd390067660 0x7fd390069b20 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.399+0000 7fd3baa00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3b40cce80 0x7fd3b4151f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.399+0000 7fd3baa00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3b40afc50 0x7fd3b41f7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.399+0000 7fd3baa00640 1 --2- 172.21.15.40:0/1438453138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b406d5c0 0x7fd3b415b1e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.399+0000 7fd3baa00640 1 -- 172.21.15.40:0/1438453138 >> 172.21.15.40:0/1438453138 conn(0x7fd3b40b21a0 msgr2=0x7fd3b40ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:51.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.403+0000 7fd3baa00640 1 -- 172.21.15.40:0/1438453138 shutdown_connections 2024-03-24T00:33:51.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.403+0000 7fd3baa00640 1 -- 172.21.15.40:0/1438453138 wait complete. 2024-03-24T00:33:51.424 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:33:51.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f0857800640 1 Processor -- start 2024-03-24T00:33:51.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f0857800640 1 -- start start 2024-03-24T00:33:51.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f0857800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f085006e620 0x7f08500b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f0857800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0850070f60 con 0x7f085006ea00 2024-03-24T00:33:51.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f0857800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08500b3620 con 0x7f08500ccfa0 2024-03-24T00:33:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f0857800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08500b4c00 con 0x7f085006e620 2024-03-24T00:33:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f084fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f085006ea00 0x7f08500b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60890/0 (socket says 172.21.15.40:60890) 2024-03-24T00:33:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f084fe00640 1 -- 172.21.15.40:0/3427888239 learned_addr learned my addr 172.21.15.40:0/3427888239 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f0854e00640 1 --2- 172.21.15.40:0/3427888239 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f085006e620 0x7f08500b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f0854e00640 1 -- 172.21.15.40:0/3427888239 >> v1:172.21.15.53:6789/0 conn(0x7f08500ccfa0 legacy=0x7f08500cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f0854e00640 1 -- 172.21.15.40:0/3427888239 >> v1:172.21.15.40:6789/0 conn(0x7f085006ea00 legacy=0x7f08500b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f084f400640 1 -- 172.21.15.40:0/3427888239 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285675128 0 0) 0x7f08500b3620 con 0x7f08500ccfa0 2024-03-24T00:33:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.563+0000 7f0854e00640 1 -- 172.21.15.40:0/3427888239 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500b3900 con 0x7f085006e620 2024-03-24T00:33:51.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.567+0000 7f0854e00640 1 --2- 172.21.15.40:0/3427888239 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f085006e620 0x7f08500b46c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f08440097f0 tx=0x7f084401c2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=237cb888147b9703 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:51.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.567+0000 7f084f400640 1 -- 172.21.15.40:0/3427888239 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08440b0070 con 0x7f085006e620 2024-03-24T00:33:51.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.567+0000 7f084f400640 1 -- 172.21.15.40:0/3427888239 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08440ab600 con 0x7f085006e620 2024-03-24T00:33:51.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.567+0000 7f084f400640 1 -- 172.21.15.40:0/3427888239 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08440ab9d0 con 0x7f085006e620 2024-03-24T00:33:51.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.567+0000 7f0857800640 1 -- 172.21.15.40:0/3427888239 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f085006e620 msgr2=0x7f08500b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.567+0000 7f0857800640 1 --2- 172.21.15.40:0/3427888239 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f085006e620 0x7f08500b46c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f08440097f0 tx=0x7f084401c2b0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.567+0000 7f0857800640 1 -- 172.21.15.40:0/3427888239 shutdown_connections 2024-03-24T00:33:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.567+0000 7f0857800640 1 --2- 172.21.15.40:0/3427888239 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f085006e620 0x7f08500b46c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.567+0000 7f0857800640 1 -- 172.21.15.40:0/3427888239 >> 172.21.15.40:0/3427888239 conn(0x7f08500b17b0 msgr2=0x7f08500b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.567+0000 7f0857800640 1 -- 172.21.15.40:0/3427888239 shutdown_connections 2024-03-24T00:33:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.567+0000 7f0857800640 1 -- 172.21.15.40:0/3427888239 wait complete. 2024-03-24T00:33:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.567+0000 7f0857800640 1 Processor -- start 2024-03-24T00:33:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0857800640 1 -- start start 2024-03-24T00:33:51.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0857800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f085006e620 0x7f08500ad2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0857800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f085006ea00 0x7f085015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0857800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08500ccfa0 0x7f08501426a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0857800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0850144920 con 0x7f085006e620 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0857800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085006ee20 con 0x7f08500ccfa0 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0857800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08500ad910 con 0x7f085006ea00 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0854e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f085006e620 0x7f08500ad2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0854e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f085006e620 0x7f08500ad2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40628/0 (socket says 172.21.15.40:40628) 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0854e00640 1 -- 172.21.15.40:0/2027396867 learned_addr learned my addr 172.21.15.40:0/2027396867 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f084fe00640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f085006ea00 0x7f085015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0855800640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08500ccfa0 0x7f08501426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0854e00640 1 -- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f085006ea00 msgr2=0x7f085015a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0854e00640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f085006ea00 0x7f085015a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0854e00640 1 -- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08500ccfa0 msgr2=0x7f08501426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0854e00640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08500ccfa0 0x7f08501426a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0854e00640 1 -- 172.21.15.40:0/2027396867 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f085015f820 con 0x7f085006e620 2024-03-24T00:33:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0854e00640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f085006e620 0x7f08500ad2c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0844019ee0 tx=0x7f084401aaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f084cc00640 1 -- 172.21.15.40:0/2027396867 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08440b0050 con 0x7f085006e620 2024-03-24T00:33:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f084cc00640 1 -- 172.21.15.40:0/2027396867 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08440bc070 con 0x7f085006e620 2024-03-24T00:33:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f084cc00640 1 -- 172.21.15.40:0/2027396867 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08440b5a70 con 0x7f085006e620 2024-03-24T00:33:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0857800640 1 -- 172.21.15.40:0/2027396867 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08500adbf0 con 0x7f085006e620 2024-03-24T00:33:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0857800640 1 -- 172.21.15.40:0/2027396867 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08500ae1b0 con 0x7f085006e620 2024-03-24T00:33:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.571+0000 7f0857800640 1 -- 172.21.15.40:0/2027396867 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0850068170 con 0x7f085006e620 2024-03-24T00:33:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.575+0000 7f084cc00640 1 -- 172.21.15.40:0/2027396867 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f08440ab480 con 0x7f085006e620 2024-03-24T00:33:51.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.575+0000 7f084cc00640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0830067660 0x7f0830069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.575+0000 7f084cc00640 1 -- 172.21.15.40:0/2027396867 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f08441201c0 con 0x7f085006e620 2024-03-24T00:33:51.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.575+0000 7f084fe00640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0830067660 0x7f0830069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.575+0000 7f084fe00640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0830067660 0x7f0830069b20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0850071320 tx=0x7f083c007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:51.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.579+0000 7f084cc00640 1 -- 172.21.15.40:0/2027396867 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0844015ff0 con 0x7f085006e620 2024-03-24T00:33:51.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 -- 172.21.15.40:0/2027396867 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0850072f50 con 0x7f085006e620 2024-03-24T00:33:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f084cc00640 1 -- 172.21.15.40:0/2027396867 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f08440f06c0 con 0x7f085006e620 2024-03-24T00:33:51.781 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:51.781 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":42,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/374985095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":374985095},{"type":"v1","addr":"172.21.15.53:6836","nonce":374985095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:32:57.024524+0000","modified":"2024-03-24T00:33:31.903823+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5763,"mds_1":5192,"mds_2":5198},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5192":{"gid":5192,"name":"c","rank":1,"incarnation":38,"state":"up:active","state_seq":13,"addr":"172.21.15.40:6835/2128553323","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2128553323},{"type":"v1","addr":"172.21.15.40:6835","nonce":2128553323}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5198":{"gid":5198,"name":"b","rank":2,"incarnation":34,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/1453842342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1453842342},{"type":"v1","addr":"172.21.15.53:6837","nonce":1453842342}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5763":{"gid":5763,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6834/224223662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":224223662},{"type":"v1","addr":"172.21.15.40:6834","nonce":224223662}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5763,"qdb_cluster":[5192,5198,5763]},"id":3}]} 2024-03-24T00:33:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 42 2024-03-24T00:33:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 -- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0830067660 msgr2=0x7f0830069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0830067660 0x7f0830069b20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0850071320 tx=0x7f083c007e10 comp rx=0 tx=0).stop 2024-03-24T00:33:51.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 -- 172.21.15.40:0/2027396867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f085006e620 msgr2=0x7f08500ad2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f085006e620 0x7f08500ad2c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f0844019ee0 tx=0x7f084401aaf0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 -- 172.21.15.40:0/2027396867 shutdown_connections 2024-03-24T00:33:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0830067660 0x7f0830069b20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08500ccfa0 0x7f08501426a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f085006ea00 0x7f085015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 --2- 172.21.15.40:0/2027396867 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f085006e620 0x7f08500ad2c0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 -- 172.21.15.40:0/2027396867 >> 172.21.15.40:0/2027396867 conn(0x7f08500b17b0 msgr2=0x7f08500b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:51.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 -- 172.21.15.40:0/2027396867 shutdown_connections 2024-03-24T00:33:51.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.779+0000 7f0857800640 1 -- 172.21.15.40:0/2027396867 wait complete. 2024-03-24T00:33:51.803 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:33:51.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.943+0000 7fdd89c00640 1 Processor -- start 2024-03-24T00:33:51.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.943+0000 7fdd89c00640 1 -- start start 2024-03-24T00:33:51.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.943+0000 7fdd89c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd8415e0b0 0x7fdd840d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.943+0000 7fdd89c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdd840cdf20 0x7fdd8406b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.943+0000 7fdd89c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd840d0e10 con 0x7fdd840cd3d0 2024-03-24T00:33:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.943+0000 7fdd89c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd840d10f0 con 0x7fdd840cdf20 2024-03-24T00:33:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.943+0000 7fdd89c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8406efd0 con 0x7fdd8415e0b0 2024-03-24T00:33:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.943+0000 7fdd82a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd8415e0b0 0x7fdd840d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.943+0000 7fdd83400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdd840cdf20 0x7fdd8406b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.943+0000 7fdd7be00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fdd840cd3d0 0x7fdd840cd810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:35378/0 (socket says 172.21.15.40:35378) 2024-03-24T00:33:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.943+0000 7fdd82a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd8415e0b0 0x7fdd840d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:51752/0 (socket says 172.21.15.40:51752) 2024-03-24T00:33:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.943+0000 7fdd7be00640 1 -- 172.21.15.40:0/4138093400 learned_addr learned my addr 172.21.15.40:0/4138093400 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:51.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.947+0000 7fdd82a00640 1 -- 172.21.15.40:0/4138093400 >> v2:172.21.15.53:3300/0 conn(0x7fdd840cdf20 msgr2=0x7fdd8406b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:51.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.947+0000 7fdd82a00640 1 --2- 172.21.15.40:0/4138093400 >> v2:172.21.15.53:3300/0 conn(0x7fdd840cdf20 0x7fdd8406b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.947+0000 7fdd82a00640 1 -- 172.21.15.40:0/4138093400 >> v1:172.21.15.53:6789/0 conn(0x7fdd840cd3d0 legacy=0x7fdd840cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.947+0000 7fdd82a00640 1 -- 172.21.15.40:0/4138093400 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8406f2b0 con 0x7fdd8415e0b0 2024-03-24T00:33:51.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.947+0000 7fdd83400640 1 --2- 172.21.15.40:0/4138093400 >> v2:172.21.15.53:3300/0 conn(0x7fdd840cdf20 0x7fdd8406b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.947+0000 7fdd82a00640 1 --2- 172.21.15.40:0/4138093400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd8415e0b0 0x7fdd840d0830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdd740097c0 tx=0x7fdd7401c1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7d7c7adef907ba7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.947+0000 7fdd82000640 1 -- 172.21.15.40:0/4138093400 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd740b0070 con 0x7fdd8415e0b0 2024-03-24T00:33:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.947+0000 7fdd82000640 1 -- 172.21.15.40:0/4138093400 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd740ab600 con 0x7fdd8415e0b0 2024-03-24T00:33:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.947+0000 7fdd82000640 1 -- 172.21.15.40:0/4138093400 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd740ab9d0 con 0x7fdd8415e0b0 2024-03-24T00:33:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 -- 172.21.15.40:0/4138093400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd8415e0b0 msgr2=0x7fdd840d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 --2- 172.21.15.40:0/4138093400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd8415e0b0 0x7fdd840d0830 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdd740097c0 tx=0x7fdd7401c1e0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 -- 172.21.15.40:0/4138093400 shutdown_connections 2024-03-24T00:33:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 --2- 172.21.15.40:0/4138093400 >> v2:172.21.15.53:3300/0 conn(0x7fdd840cdf20 0x7fdd8406b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 --2- 172.21.15.40:0/4138093400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd8415e0b0 0x7fdd840d0830 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 -- 172.21.15.40:0/4138093400 >> 172.21.15.40:0/4138093400 conn(0x7fdd840d1c90 msgr2=0x7fdd840d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:51.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 -- 172.21.15.40:0/4138093400 shutdown_connections 2024-03-24T00:33:51.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 -- 172.21.15.40:0/4138093400 wait complete. 2024-03-24T00:33:51.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 Processor -- start 2024-03-24T00:33:51.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 -- start start 2024-03-24T00:33:51.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd840cd3d0 0x7fdd84154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd840cdf20 0x7fdd8415e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd8415e0b0 0x7fdd841593c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8406fc30 con 0x7fdd8415e0b0 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8406fab0 con 0x7fdd840cdf20 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.951+0000 7fdd89c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8406fdb0 con 0x7fdd840cd3d0 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd82a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd840cd3d0 0x7fdd84154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd82a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd840cd3d0 0x7fdd84154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:51768/0 (socket says 172.21.15.40:51768) 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd82a00640 1 -- 172.21.15.40:0/3774673295 learned_addr learned my addr 172.21.15.40:0/3774673295 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd7be00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd840cdf20 0x7fdd8415e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd83400640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd8415e0b0 0x7fdd841593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd82a00640 1 -- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd840cdf20 msgr2=0x7fdd8415e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd82a00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd840cdf20 0x7fdd8415e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd82a00640 1 -- 172.21.15.40:0/3774673295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd8415e0b0 msgr2=0x7fdd841593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd82a00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd8415e0b0 0x7fdd841593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd82a00640 1 -- 172.21.15.40:0/3774673295 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd841552d0 con 0x7fdd840cd3d0 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd82a00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd840cd3d0 0x7fdd84154d90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdd74019e10 tx=0x7fdd74018f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd7aa00640 1 -- 172.21.15.40:0/3774673295 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd740b0040 con 0x7fdd840cd3d0 2024-03-24T00:33:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd7aa00640 1 -- 172.21.15.40:0/3774673295 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd740b9070 con 0x7fdd840cd3d0 2024-03-24T00:33:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd7aa00640 1 -- 172.21.15.40:0/3774673295 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd74006e20 con 0x7fdd840cd3d0 2024-03-24T00:33:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd89c00640 1 -- 172.21.15.40:0/3774673295 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd840c58f0 con 0x7fdd840cd3d0 2024-03-24T00:33:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd89c00640 1 -- 172.21.15.40:0/3774673295 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd840c5b90 con 0x7fdd840cd3d0 2024-03-24T00:33:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd89c00640 1 -- 172.21.15.40:0/3774673295 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd84068170 con 0x7fdd840cd3d0 2024-03-24T00:33:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.955+0000 7fdd7aa00640 1 -- 172.21.15.40:0/3774673295 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fdd740b4650 con 0x7fdd840cd3d0 2024-03-24T00:33:51.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.959+0000 7fdd7aa00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd60067660 0x7fdd60069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:51.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.959+0000 7fdd7aa00640 1 -- 172.21.15.40:0/3774673295 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fdd7411f4b0 con 0x7fdd840cd3d0 2024-03-24T00:33:51.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.959+0000 7fdd7be00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd60067660 0x7fdd60069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:51.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.959+0000 7fdd7be00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd60067660 0x7fdd60069b20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdd841559a0 tx=0x7fdd6c007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:51.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:51.963+0000 7fdd7aa00640 1 -- 172.21.15.40:0/3774673295 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdd74016090 con 0x7fdd840cd3d0 2024-03-24T00:33:52.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.131+0000 7fdd89c00640 1 -- 172.21.15.40:0/3774673295 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd840d23f0 con 0x7fdd840cd3d0 2024-03-24T00:33:52.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.131+0000 7fdd7aa00640 1 -- 172.21.15.40:0/3774673295 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16773 (secure 0 0 0) 0x7fdd740f0090 con 0x7fdd840cd3d0 2024-03-24T00:33:52.137 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:52.137 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":47,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:33:48.936722+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:32:55.446633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-03-24T00:32:56.017210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"44","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.37","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6836/1600357100":"2024-03-25T00:33:27.880035+0000","172.21.15.53:6834/1600357100":"2024-03-25T00:33:27.880035+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:33:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.135+0000 7fdd89c00640 1 -- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd60067660 msgr2=0x7fdd60069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.135+0000 7fdd89c00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd60067660 0x7fdd60069b20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fdd841559a0 tx=0x7fdd6c007e90 comp rx=0 tx=0).stop 2024-03-24T00:33:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.135+0000 7fdd89c00640 1 -- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd840cd3d0 msgr2=0x7fdd84154d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:52.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.135+0000 7fdd89c00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd840cd3d0 0x7fdd84154d90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdd74019e10 tx=0x7fdd74018f80 comp rx=0 tx=0).stop 2024-03-24T00:33:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.135+0000 7fdd89c00640 1 -- 172.21.15.40:0/3774673295 shutdown_connections 2024-03-24T00:33:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.135+0000 7fdd89c00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdd60067660 0x7fdd60069b20 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.135+0000 7fdd89c00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdd8415e0b0 0x7fdd841593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.135+0000 7fdd89c00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd840cdf20 0x7fdd8415e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.135+0000 7fdd89c00640 1 --2- 172.21.15.40:0/3774673295 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdd840cd3d0 0x7fdd84154d90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.135+0000 7fdd89c00640 1 -- 172.21.15.40:0/3774673295 >> 172.21.15.40:0/3774673295 conn(0x7fdd840d1c90 msgr2=0x7fdd84155c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:52.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.135+0000 7fdd89c00640 1 -- 172.21.15.40:0/3774673295 shutdown_connections 2024-03-24T00:33:52.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.135+0000 7fdd89c00640 1 -- 172.21.15.40:0/3774673295 wait complete. 2024-03-24T00:33:52.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:33:52.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.307+0000 7feab8800640 1 Processor -- start 2024-03-24T00:33:52.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.307+0000 7feab8800640 1 -- start start 2024-03-24T00:33:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.307+0000 7feab8800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feab013ec80 0x7feab01414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.307+0000 7feab8800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7feab00ce800 0x7feab0071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.307+0000 7feab8800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab0070e00 con 0x7feab01419f0 2024-03-24T00:33:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.307+0000 7feab8800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab0141e50 con 0x7feab013ec80 2024-03-24T00:33:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.307+0000 7feab8800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab00cee20 con 0x7feab00ce800 2024-03-24T00:33:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.307+0000 7feab5e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feab013ec80 0x7feab01414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.307+0000 7feab5400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7feab01419f0 0x7feab00ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60902/0 (socket says 172.21.15.40:60902) 2024-03-24T00:33:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.307+0000 7feab5e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feab013ec80 0x7feab01414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40654/0 (socket says 172.21.15.40:40654) 2024-03-24T00:33:52.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.311+0000 7feab5400640 1 -- 172.21.15.40:0/3981364509 learned_addr learned my addr 172.21.15.40:0/3981364509 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.307+0000 7feab6800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7feab00ce800 0x7feab0071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.311+0000 7feab5e00640 1 -- 172.21.15.40:0/3981364509 >> v2:172.21.15.53:3300/0 conn(0x7feab00ce800 msgr2=0x7feab0071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.311+0000 7feab5e00640 1 --2- 172.21.15.40:0/3981364509 >> v2:172.21.15.53:3300/0 conn(0x7feab00ce800 0x7feab0071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.311+0000 7feab5e00640 1 -- 172.21.15.40:0/3981364509 >> v1:172.21.15.40:6789/0 conn(0x7feab01419f0 legacy=0x7feab00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.311+0000 7feab5e00640 1 -- 172.21.15.40:0/3981364509 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab00cf100 con 0x7feab013ec80 2024-03-24T00:33:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.311+0000 7feab4a00640 1 -- 172.21.15.40:0/3981364509 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666465446 0 0) 0x7feab0070e00 con 0x7feab01419f0 2024-03-24T00:33:52.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.311+0000 7feab5e00640 1 --2- 172.21.15.40:0/3981364509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feab013ec80 0x7feab01414b0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7feab0067e60 tx=0x7feaa401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c27ebdc113c61c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:52.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.311+0000 7feab4a00640 1 -- 172.21.15.40:0/3981364509 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa40aa400 con 0x7feab013ec80 2024-03-24T00:33:52.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.311+0000 7feab4a00640 1 -- 172.21.15.40:0/3981364509 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa40af070 con 0x7feab013ec80 2024-03-24T00:33:52.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.311+0000 7feab4a00640 1 -- 172.21.15.40:0/3981364509 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa40aa7c0 con 0x7feab013ec80 2024-03-24T00:33:52.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 -- 172.21.15.40:0/3981364509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feab013ec80 msgr2=0x7feab01414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:52.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 --2- 172.21.15.40:0/3981364509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feab013ec80 0x7feab01414b0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7feab0067e60 tx=0x7feaa401c480 comp rx=0 tx=0).stop 2024-03-24T00:33:52.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 -- 172.21.15.40:0/3981364509 shutdown_connections 2024-03-24T00:33:52.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 --2- 172.21.15.40:0/3981364509 >> v2:172.21.15.53:3300/0 conn(0x7feab00ce800 0x7feab0071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 --2- 172.21.15.40:0/3981364509 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feab013ec80 0x7feab01414b0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 -- 172.21.15.40:0/3981364509 >> 172.21.15.40:0/3981364509 conn(0x7feab00b17b0 msgr2=0x7feab00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:52.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 -- 172.21.15.40:0/3981364509 shutdown_connections 2024-03-24T00:33:52.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 -- 172.21.15.40:0/3981364509 wait complete. 2024-03-24T00:33:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 Processor -- start 2024-03-24T00:33:52.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 -- start start 2024-03-24T00:33:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feab00ce800 0x7feab00c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feab013ec80 0x7feab015aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feab01419f0 0x7feab015b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab00723e0 con 0x7feab00ce800 2024-03-24T00:33:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab0072260 con 0x7feab013ec80 2024-03-24T00:33:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab8800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab0072560 con 0x7feab01419f0 2024-03-24T00:33:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feab00ce800 0x7feab00c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feab00ce800 0x7feab00c4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40660/0 (socket says 172.21.15.40:40660) 2024-03-24T00:33:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab5e00640 1 -- 172.21.15.40:0/616664376 learned_addr learned my addr 172.21.15.40:0/616664376 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab5400640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feab013ec80 0x7feab015aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:52.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab6800640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feab01419f0 0x7feab015b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab5e00640 1 -- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feab01419f0 msgr2=0x7feab015b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab5e00640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feab01419f0 0x7feab015b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab5e00640 1 -- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feab013ec80 msgr2=0x7feab015aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab5e00640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feab013ec80 0x7feab015aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.315+0000 7feab5e00640 1 -- 172.21.15.40:0/616664376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab015ccf0 con 0x7feab00ce800 2024-03-24T00:33:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.319+0000 7feab5e00640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feab00ce800 0x7feab00c4c80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7feaa40ac040 tx=0x7feaa401acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.319+0000 7feaaa000640 1 -- 172.21.15.40:0/616664376 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa40be410 con 0x7feab00ce800 2024-03-24T00:33:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.319+0000 7feaaa000640 1 -- 172.21.15.40:0/616664376 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa40be5d0 con 0x7feab00ce800 2024-03-24T00:33:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.319+0000 7feab8800640 1 -- 172.21.15.40:0/616664376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab015cfd0 con 0x7feab00ce800 2024-03-24T00:33:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.319+0000 7feaaa000640 1 -- 172.21.15.40:0/616664376 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa40be9c0 con 0x7feab00ce800 2024-03-24T00:33:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.319+0000 7feab8800640 1 -- 172.21.15.40:0/616664376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab015d590 con 0x7feab00ce800 2024-03-24T00:33:52.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.319+0000 7feab8800640 1 -- 172.21.15.40:0/616664376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab0068170 con 0x7feab00ce800 2024-03-24T00:33:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.319+0000 7feaaa000640 1 -- 172.21.15.40:0/616664376 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7feaa40bc020 con 0x7feab00ce800 2024-03-24T00:33:52.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.323+0000 7feaaa000640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fea94067660 0x7fea94069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.323+0000 7feaaa000640 1 -- 172.21.15.40:0/616664376 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5853+0+0 (secure 0 0 0) 0x7feaa411fcf0 con 0x7feab00ce800 2024-03-24T00:33:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.323+0000 7feab5400640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fea94067660 0x7fea94069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:52.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.323+0000 7feab5400640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fea94067660 0x7fea94069b20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7feab013b9c0 tx=0x7feaa0007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:52.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.331+0000 7feaaa000640 1 -- 172.21.15.40:0/616664376 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7feaa4124070 con 0x7feab00ce800 2024-03-24T00:33:52.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.531+0000 7feab8800640 1 -- 172.21.15.40:0/616664376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7feab0070e00 con 0x7feab00ce800 2024-03-24T00:33:52.533 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-03-24T00:33:52.531+0000 7fa64e600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:33:52.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feaaa000640 1 -- 172.21.15.40:0/616664376 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7feaa4016070 con 0x7feab00ce800 2024-03-24T00:33:52.536 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:33:52.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feab8800640 1 -- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fea94067660 msgr2=0x7fea94069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:52.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feab8800640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fea94067660 0x7fea94069b20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7feab013b9c0 tx=0x7feaa0007d60 comp rx=0 tx=0).stop 2024-03-24T00:33:52.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feab8800640 1 -- 172.21.15.40:0/616664376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feab00ce800 msgr2=0x7feab00c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:52.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feab8800640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feab00ce800 0x7feab00c4c80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7feaa40ac040 tx=0x7feaa401acc0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feab8800640 1 -- 172.21.15.40:0/616664376 shutdown_connections 2024-03-24T00:33:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feab8800640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fea94067660 0x7fea94069b20 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feab8800640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feab01419f0 0x7feab015b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feab8800640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feab013ec80 0x7feab015aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feab8800640 1 --2- 172.21.15.40:0/616664376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feab00ce800 0x7feab00c4c80 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feab8800640 1 -- 172.21.15.40:0/616664376 >> 172.21.15.40:0/616664376 conn(0x7feab00b17b0 msgr2=0x7feab01410c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:52.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feab8800640 1 -- 172.21.15.40:0/616664376 shutdown_connections 2024-03-24T00:33:52.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.535+0000 7feab8800640 1 -- 172.21.15.40:0/616664376 wait complete. 2024-03-24T00:33:52.563 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:33:52.564 INFO:tasks.ceph.mds.a.smithi040.stderr: -2374> 2024-03-24T00:33:33.607+0000 7f2af2400640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:33:52.600 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:33:52.604 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:33:52.616 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:33:52.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.711+0000 7f1a44800640 1 Processor -- start 2024-03-24T00:33:52.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.711+0000 7f1a44800640 1 -- start start 2024-03-24T00:33:52.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a44800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a3c15ac60 0x7f1a3c0b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a44800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1a3c071b00 0x7f1a3c071f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a44800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a3c0b3260 con 0x7f1a3c06cb80 2024-03-24T00:33:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a44800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a3c072490 con 0x7f1a3c15ac60 2024-03-24T00:33:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a44800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a3c0b3e70 con 0x7f1a3c071b00 2024-03-24T00:33:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a41e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a3c15ac60 0x7f1a3c0b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a41e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1a3c15ac60 0x7f1a3c0b2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40704/0 (socket says 172.21.15.40:40704) 2024-03-24T00:33:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a41e00640 1 -- 172.21.15.40:0/2928929449 learned_addr learned my addr 172.21.15.40:0/2928929449 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a42800640 1 --2- 172.21.15.40:0/2928929449 >> v2:172.21.15.53:3300/0 conn(0x7f1a3c071b00 0x7f1a3c071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a41e00640 1 -- 172.21.15.40:0/2928929449 >> v2:172.21.15.53:3300/0 conn(0x7f1a3c071b00 msgr2=0x7f1a3c071f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a41e00640 1 --2- 172.21.15.40:0/2928929449 >> v2:172.21.15.53:3300/0 conn(0x7f1a3c071b00 0x7f1a3c071f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a41e00640 1 -- 172.21.15.40:0/2928929449 >> v1:172.21.15.53:6789/0 conn(0x7f1a3c06cb80 legacy=0x7f1a3c0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:52.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.715+0000 7f1a41e00640 1 -- 172.21.15.40:0/2928929449 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c0b3540 con 0x7f1a3c15ac60 2024-03-24T00:33:52.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a41e00640 1 --2- 172.21.15.40:0/2928929449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a3c15ac60 0x7f1a3c0b2960 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c06d370 tx=0x7f1a3001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19da7b04461b38f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a40a00640 1 -- 172.21.15.40:0/2928929449 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a300ab660 con 0x7f1a3c15ac60 2024-03-24T00:33:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a40a00640 1 -- 172.21.15.40:0/2928929449 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a300b0070 con 0x7f1a3c15ac60 2024-03-24T00:33:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a40a00640 1 -- 172.21.15.40:0/2928929449 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a300aba20 con 0x7f1a3c15ac60 2024-03-24T00:33:52.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a44800640 1 -- 172.21.15.40:0/2928929449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a3c15ac60 msgr2=0x7f1a3c0b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a44800640 1 --2- 172.21.15.40:0/2928929449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a3c15ac60 0x7f1a3c0b2960 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c06d370 tx=0x7f1a3001c460 comp rx=0 tx=0).stop 2024-03-24T00:33:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a44800640 1 -- 172.21.15.40:0/2928929449 shutdown_connections 2024-03-24T00:33:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a44800640 1 --2- 172.21.15.40:0/2928929449 >> v2:172.21.15.53:3300/0 conn(0x7f1a3c071b00 0x7f1a3c071f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a44800640 1 --2- 172.21.15.40:0/2928929449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a3c15ac60 0x7f1a3c0b2960 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a44800640 1 -- 172.21.15.40:0/2928929449 >> 172.21.15.40:0/2928929449 conn(0x7f1a3c0b1660 msgr2=0x7f1a3c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a44800640 1 -- 172.21.15.40:0/2928929449 shutdown_connections 2024-03-24T00:33:52.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a44800640 1 -- 172.21.15.40:0/2928929449 wait complete. 2024-03-24T00:33:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a44800640 1 Processor -- start 2024-03-24T00:33:52.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.719+0000 7f1a44800640 1 -- start start 2024-03-24T00:33:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a44800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a3c06cb80 0x7f1a3c1587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a44800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a3c071b00 0x7f1a3c1597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a44800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a3c15ac60 0x7f1a3c1fa530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a44800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c0b3020 con 0x7f1a3c06cb80 2024-03-24T00:33:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a44800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c0b2ea0 con 0x7f1a3c071b00 2024-03-24T00:33:52.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a44800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c15b040 con 0x7f1a3c15ac60 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a41e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a3c06cb80 0x7f1a3c1587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a41e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a3c06cb80 0x7f1a3c1587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40714/0 (socket says 172.21.15.40:40714) 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a41e00640 1 -- 172.21.15.40:0/4098431718 learned_addr learned my addr 172.21.15.40:0/4098431718 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a42800640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a3c15ac60 0x7f1a3c1fa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a41400640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a3c071b00 0x7f1a3c1597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a41e00640 1 -- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a3c15ac60 msgr2=0x7f1a3c1fa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a41e00640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a3c15ac60 0x7f1a3c1fa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a41e00640 1 -- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a3c071b00 msgr2=0x7f1a3c1597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a41e00640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a3c071b00 0x7f1a3c1597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a41e00640 1 -- 172.21.15.40:0/4098431718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c0cf240 con 0x7f1a3c06cb80 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a41e00640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a3c06cb80 0x7f1a3c1587a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1a300ad040 tx=0x7f1a3001aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a36000640 1 -- 172.21.15.40:0/4098431718 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a300be5f0 con 0x7f1a3c06cb80 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a36000640 1 -- 172.21.15.40:0/4098431718 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a300be7b0 con 0x7f1a3c06cb80 2024-03-24T00:33:52.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a36000640 1 -- 172.21.15.40:0/4098431718 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a300beba0 con 0x7f1a3c06cb80 2024-03-24T00:33:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a44800640 1 -- 172.21.15.40:0/4098431718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3c158e10 con 0x7f1a3c06cb80 2024-03-24T00:33:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a44800640 1 -- 172.21.15.40:0/4098431718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3c0cf7c0 con 0x7f1a3c06cb80 2024-03-24T00:33:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a44800640 1 -- 172.21.15.40:0/4098431718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3c068170 con 0x7f1a3c06cb80 2024-03-24T00:33:52.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.723+0000 7f1a36000640 1 -- 172.21.15.40:0/4098431718 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f1a300b43f0 con 0x7f1a3c06cb80 2024-03-24T00:33:52.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.727+0000 7f1a36000640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1a20067660 0x7f1a20069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:52.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.727+0000 7f1a36000640 1 -- 172.21.15.40:0/4098431718 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6111+0+0 (secure 0 0 0) 0x7f1a30120650 con 0x7f1a3c06cb80 2024-03-24T00:33:52.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.727+0000 7f1a41400640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1a20067660 0x7f1a20069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:52.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.727+0000 7f1a41400640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1a20067660 0x7f1a20069b20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c154da0 tx=0x7f1a2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:52.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.731+0000 7f1a36000640 1 -- 172.21.15.40:0/4098431718 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1a30016110 con 0x7f1a3c06cb80 2024-03-24T00:33:52.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:52.931+0000 7f1a44800640 1 -- 172.21.15.40:0/4098431718 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1a3c0af2c0 con 0x7f1a3c06cb80 2024-03-24T00:33:53.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.535+0000 7f1a36000640 1 -- 172.21.15.40:0/4098431718 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v44) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1a300f1250 con 0x7f1a3c06cb80 2024-03-24T00:33:53.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.539+0000 7f1a44800640 1 -- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1a20067660 msgr2=0x7f1a20069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:53.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.539+0000 7f1a44800640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1a20067660 0x7f1a20069b20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c154da0 tx=0x7f1a2c009040 comp rx=0 tx=0).stop 2024-03-24T00:33:53.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.539+0000 7f1a44800640 1 -- 172.21.15.40:0/4098431718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a3c06cb80 msgr2=0x7f1a3c1587a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:53.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.539+0000 7f1a44800640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a3c06cb80 0x7f1a3c1587a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f1a300ad040 tx=0x7f1a3001aca0 comp rx=0 tx=0).stop 2024-03-24T00:33:53.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.539+0000 7f1a44800640 1 -- 172.21.15.40:0/4098431718 shutdown_connections 2024-03-24T00:33:53.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.539+0000 7f1a44800640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1a20067660 0x7f1a20069b20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:53.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.539+0000 7f1a44800640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a3c15ac60 0x7f1a3c1fa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:53.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.539+0000 7f1a44800640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a3c071b00 0x7f1a3c1597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:53.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.539+0000 7f1a44800640 1 --2- 172.21.15.40:0/4098431718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1a3c06cb80 0x7f1a3c1587a0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:53.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.539+0000 7f1a44800640 1 -- 172.21.15.40:0/4098431718 >> 172.21.15.40:0/4098431718 conn(0x7f1a3c0b1660 msgr2=0x7f1a3c0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:53.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.539+0000 7f1a44800640 1 -- 172.21.15.40:0/4098431718 shutdown_connections 2024-03-24T00:33:53.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.539+0000 7f1a44800640 1 -- 172.21.15.40:0/4098431718 wait complete. 2024-03-24T00:33:53.565 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:33:53.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.759+0000 7f2f64200640 1 Processor -- start 2024-03-24T00:33:53.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.759+0000 7f2f64200640 1 -- start start 2024-03-24T00:33:53.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f64200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c15e350 0x7f2f5c0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f64200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2f5c0c9580 0x7f2f5c0c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f64200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f5c0734e0 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f64200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5c0cd660 con 0x7f2f5c0c9580 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f64200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c0c9f40 con 0x7f2f5c15e350 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f61800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c15e350 0x7f2f5c0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f60e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2f5c0c9580 0x7f2f5c0c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f61800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c15e350 0x7f2f5c0cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:51818/0 (socket says 172.21.15.40:51818) 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f61800640 1 -- 172.21.15.40:0/3239826112 learned_addr learned my addr 172.21.15.40:0/3239826112 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f62200640 1 --1- 172.21.15.40:0/3239826112 >> v1:172.21.15.53:6789/0 conn(0x7f2f5c0ad1e0 0x7f2f5c0ad610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:35412/0 (socket says 172.21.15.40:35412) 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f61800640 1 -- 172.21.15.40:0/3239826112 >> v2:172.21.15.53:3300/0 conn(0x7f2f5c0c9580 msgr2=0x7f2f5c0c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f61800640 1 --2- 172.21.15.40:0/3239826112 >> v2:172.21.15.53:3300/0 conn(0x7f2f5c0c9580 0x7f2f5c0c9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f61800640 1 -- 172.21.15.40:0/3239826112 >> v1:172.21.15.53:6789/0 conn(0x7f2f5c0ad1e0 legacy=0x7f2f5c0ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:53.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f61800640 1 -- 172.21.15.40:0/3239826112 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c0d0030 con 0x7f2f5c15e350 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f60e00640 1 --2- 172.21.15.40:0/3239826112 >> v2:172.21.15.53:3300/0 conn(0x7f2f5c0c9580 0x7f2f5c0c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.763+0000 7f2f5be00640 1 -- 172.21.15.40:0/3239826112 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200322336 0 0) 0x7f2f5c0734e0 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f61800640 1 --2- 172.21.15.40:0/3239826112 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c15e350 0x7f2f5c0cfaf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0d0200 tx=0x7f2f5001c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd7098ff60d9c7dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f5be00640 1 -- 172.21.15.40:0/3239826112 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500ab650 con 0x7f2f5c15e350 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f5be00640 1 -- 172.21.15.40:0/3239826112 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f500b0070 con 0x7f2f5c15e350 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f5be00640 1 -- 172.21.15.40:0/3239826112 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500aba10 con 0x7f2f5c15e350 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f64200640 1 -- 172.21.15.40:0/3239826112 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c15e350 msgr2=0x7f2f5c0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f64200640 1 --2- 172.21.15.40:0/3239826112 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c15e350 0x7f2f5c0cfaf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0d0200 tx=0x7f2f5001c540 comp rx=0 tx=0).stop 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f64200640 1 -- 172.21.15.40:0/3239826112 shutdown_connections 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f64200640 1 --2- 172.21.15.40:0/3239826112 >> v2:172.21.15.53:3300/0 conn(0x7f2f5c0c9580 0x7f2f5c0c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f64200640 1 --2- 172.21.15.40:0/3239826112 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c15e350 0x7f2f5c0cfaf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f64200640 1 -- 172.21.15.40:0/3239826112 >> 172.21.15.40:0/3239826112 conn(0x7f2f5c072870 msgr2=0x7f2f5c072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f64200640 1 -- 172.21.15.40:0/3239826112 shutdown_connections 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f64200640 1 -- 172.21.15.40:0/3239826112 wait complete. 2024-03-24T00:33:53.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f64200640 1 Processor -- start 2024-03-24T00:33:53.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.767+0000 7f2f64200640 1 -- start start 2024-03-24T00:33:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f64200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f5c0ad1e0 0x7f2f5c154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f64200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c0c9580 0x7f2f5c152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f64200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f5c15e350 0x7f2f5c152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f64200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c0716b0 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f64200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c071fe0 con 0x7f2f5c15e350 2024-03-24T00:33:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f64200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5c0cda30 con 0x7f2f5c0c9580 2024-03-24T00:33:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f61800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f5c0ad1e0 0x7f2f5c154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f61800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f5c0ad1e0 0x7f2f5c154120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40724/0 (socket says 172.21.15.40:40724) 2024-03-24T00:33:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f61800640 1 -- 172.21.15.40:0/2558126583 learned_addr learned my addr 172.21.15.40:0/2558126583 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f60e00640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c0c9580 0x7f2f5c152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:53.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f62200640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f5c15e350 0x7f2f5c152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f61800640 1 -- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c0c9580 msgr2=0x7f2f5c152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f61800640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c0c9580 0x7f2f5c152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f61800640 1 -- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f5c15e350 msgr2=0x7f2f5c152a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f61800640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f5c15e350 0x7f2f5c152a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f61800640 1 -- 172.21.15.40:0/2558126583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c154a00 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f60e00640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c0c9580 0x7f2f5c152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f62200640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f5c15e350 0x7f2f5c152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f61800640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f5c0ad1e0 0x7f2f5c154120 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2f5001a170 tx=0x7f2f5001ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f59600640 1 -- 172.21.15.40:0/2558126583 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500b4800 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f59600640 1 -- 172.21.15.40:0/2558126583 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f500b49a0 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f59600640 1 -- 172.21.15.40:0/2558126583 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500b4d90 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f64200640 1 -- 172.21.15.40:0/2558126583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f5c154ce0 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f64200640 1 -- 172.21.15.40:0/2558126583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f5c1552a0 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f64200640 1 -- 172.21.15.40:0/2558126583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f5c068170 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.771+0000 7f2f59600640 1 -- 172.21.15.40:0/2558126583 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2f500b3380 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.775+0000 7f2f59600640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f3c067660 0x7f2f3c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.775+0000 7f2f59600640 1 -- 172.21.15.40:0/2558126583 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6111+0+0 (secure 0 0 0) 0x7f2f5011ff60 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.775+0000 7f2f60e00640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f3c067660 0x7f2f3c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:53.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.775+0000 7f2f60e00640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f3c067660 0x7f2f3c069b20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0af880 tx=0x7f2f48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:53.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.779+0000 7f2f59600640 1 -- 172.21.15.40:0/2558126583 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2f50016130 con 0x7f2f5c0ad1e0 2024-03-24T00:33:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:53.943+0000 7f2f64200640 1 -- 172.21.15.40:0/2558126583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2f5c06f220 con 0x7f2f5c0ad1e0 2024-03-24T00:33:54.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.539+0000 7f2f59600640 1 -- 172.21.15.40:0/2558126583 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v49) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2f500f0510 con 0x7f2f5c0ad1e0 2024-03-24T00:33:54.541 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:33:54.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.539+0000 7f2f64200640 1 -- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f3c067660 msgr2=0x7f2f3c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:54.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.539+0000 7f2f64200640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f3c067660 0x7f2f3c069b20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0af880 tx=0x7f2f48008040 comp rx=0 tx=0).stop 2024-03-24T00:33:54.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.539+0000 7f2f64200640 1 -- 172.21.15.40:0/2558126583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f5c0ad1e0 msgr2=0x7f2f5c154120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:54.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.539+0000 7f2f64200640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f5c0ad1e0 0x7f2f5c154120 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2f5001a170 tx=0x7f2f5001ad80 comp rx=0 tx=0).stop 2024-03-24T00:33:54.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.543+0000 7f2f64200640 1 -- 172.21.15.40:0/2558126583 shutdown_connections 2024-03-24T00:33:54.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.543+0000 7f2f64200640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f3c067660 0x7f2f3c069b20 secure :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0af880 tx=0x7f2f48008040 comp rx=0 tx=0).stop 2024-03-24T00:33:54.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.543+0000 7f2f64200640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f5c15e350 0x7f2f5c152a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:54.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.543+0000 7f2f64200640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2f5c0c9580 0x7f2f5c152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:54.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.543+0000 7f2f64200640 1 --2- 172.21.15.40:0/2558126583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2f5c0ad1e0 0x7f2f5c154120 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:54.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.543+0000 7f2f64200640 1 -- 172.21.15.40:0/2558126583 >> 172.21.15.40:0/2558126583 conn(0x7f2f5c072870 msgr2=0x7f2f5c06d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:54.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.543+0000 7f2f64200640 1 -- 172.21.15.40:0/2558126583 shutdown_connections 2024-03-24T00:33:54.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.543+0000 7f2f64200640 1 -- 172.21.15.40:0/2558126583 wait complete. 2024-03-24T00:33:54.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:33:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.715+0000 7fa787200640 1 Processor -- start 2024-03-24T00:33:54.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa787200640 1 -- start start 2024-03-24T00:33:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa787200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815bde0 0x7fa7880ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa787200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7880cdfd0 con 0x7fa78815c1c0 2024-03-24T00:33:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa787200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7880cda30 con 0x7fa7880704c0 2024-03-24T00:33:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa787200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7880cdcf0 con 0x7fa78815bde0 2024-03-24T00:33:54.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa77fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa78815c1c0 0x7fa7880ce700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60920/0 (socket says 172.21.15.40:60920) 2024-03-24T00:33:54.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa77fe00640 1 -- 172.21.15.40:0/3916841875 learned_addr learned my addr 172.21.15.40:0/3916841875 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:54.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785e00640 1 --2- 172.21.15.40:0/3916841875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815bde0 0x7fa7880ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:54.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192029671 0 0) 0x7fa7880cdfd0 con 0x7fa78815c1c0 2024-03-24T00:33:54.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa76c003530 con 0x7fa78815c1c0 2024-03-24T00:33:54.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496195377 0 0) 0x7fa7880cda30 con 0x7fa7880704c0 2024-03-24T00:33:54.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7880cdfd0 con 0x7fa7880704c0 2024-03-24T00:33:54.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2205780244 0 0) 0x7fa76c003530 con 0x7fa78815c1c0 2024-03-24T00:33:54.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815bde0 msgr2=0x7fa7880ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:54.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 --2- 172.21.15.40:0/3916841875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815bde0 0x7fa7880ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:54.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 >> v1:172.21.15.53:6789/0 conn(0x7fa7880704c0 legacy=0x7fa7880cc410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78806ecb0 con 0x7fa78815c1c0 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fa770001680 con 0x7fa78815c1c0 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 >> v1:172.21.15.40:6789/0 conn(0x7fa78815c1c0 legacy=0x7fa7880ce700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 --2- 172.21.15.40:0/3916841875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa76c003bf0 0x7fa76c003ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 --2- 172.21.15.40:0/3916841875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815bde0 0x7fa76c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 --2- 172.21.15.40:0/3916841875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa76c004d50 0x7fa76c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7880cdcf0 con 0x7fa76c004d50 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c00bb00 con 0x7fa76c003bf0 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa76c00bd00 con 0x7fa78815bde0 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa786800640 1 --2- 172.21.15.40:0/3916841875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa76c004d50 0x7fa76c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa785e00640 1 --2- 172.21.15.40:0/3916841875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa76c003bf0 0x7fa76c003ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa77fe00640 1 --2- 172.21.15.40:0/3916841875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815bde0 0x7fa76c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa786800640 1 -- 172.21.15.40:0/3916841875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815bde0 msgr2=0x7fa76c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:54.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa786800640 1 --2- 172.21.15.40:0/3916841875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815bde0 0x7fa76c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:54.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa786800640 1 -- 172.21.15.40:0/3916841875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa76c003bf0 msgr2=0x7fa76c003ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:54.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa786800640 1 --2- 172.21.15.40:0/3916841875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa76c003bf0 0x7fa76c003ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:54.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.719+0000 7fa786800640 1 -- 172.21.15.40:0/3916841875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c003530 con 0x7fa76c004d50 2024-03-24T00:33:54.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.723+0000 7fa786800640 1 --2- 172.21.15.40:0/3916841875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa76c004d50 0x7fa76c0051d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fa780001fd0 tx=0x7fa78001c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23dd2ce38be80c59 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:54.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.723+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780001c20 con 0x7fa76c004d50 2024-03-24T00:33:54.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.723+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa780003ba0 con 0x7fa76c004d50 2024-03-24T00:33:54.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.723+0000 7fa785400640 1 -- 172.21.15.40:0/3916841875 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780005080 con 0x7fa76c004d50 2024-03-24T00:33:54.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.723+0000 7fa787200640 1 -- 172.21.15.40:0/3916841875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa76c004d50 msgr2=0x7fa76c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:54.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.723+0000 7fa787200640 1 --2- 172.21.15.40:0/3916841875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa76c004d50 0x7fa76c0051d0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fa780001fd0 tx=0x7fa78001c720 comp rx=0 tx=0).stop 2024-03-24T00:33:54.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.723+0000 7fa786800640 1 -- 172.21.15.40:0/3916841875 reap_dead start 2024-03-24T00:33:54.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.723+0000 7fa787200640 1 -- 172.21.15.40:0/3916841875 shutdown_connections 2024-03-24T00:33:54.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.723+0000 7fa787200640 1 -- 172.21.15.40:0/3916841875 >> 172.21.15.40:0/3916841875 conn(0x7fa7880d19e0 msgr2=0x7fa7880d1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:54.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.723+0000 7fa787200640 1 -- 172.21.15.40:0/3916841875 shutdown_connections 2024-03-24T00:33:54.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.723+0000 7fa787200640 1 -- 172.21.15.40:0/3916841875 wait complete. 2024-03-24T00:33:54.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa787200640 1 Processor -- start 2024-03-24T00:33:54.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa787200640 1 -- start start 2024-03-24T00:33:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa787200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa76c004d50 0x7fa788150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa787200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815e690 0x7fa78815eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa787200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7880b6b60 0x7fa7880b6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa787200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7880ce340 con 0x7fa7880b6b60 2024-03-24T00:33:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa787200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7880ce1c0 con 0x7fa76c004d50 2024-03-24T00:33:54.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa787200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7880ce4c0 con 0x7fa78815e690 2024-03-24T00:33:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa786800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7880b6b60 0x7fa7880b6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa786800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7880b6b60 0x7fa7880b6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40734/0 (socket says 172.21.15.40:40734) 2024-03-24T00:33:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa786800640 1 -- 172.21.15.40:0/4279896446 learned_addr learned my addr 172.21.15.40:0/4279896446 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa785e00640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa76c004d50 0x7fa788150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa77fe00640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815e690 0x7fa78815eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa786800640 1 -- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815e690 msgr2=0x7fa78815eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa786800640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815e690 0x7fa78815eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa786800640 1 -- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa76c004d50 msgr2=0x7fa788150f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa786800640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa76c004d50 0x7fa788150f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa786800640 1 -- 172.21.15.40:0/4279896446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7881514a0 con 0x7fa7880b6b60 2024-03-24T00:33:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa785e00640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa76c004d50 0x7fa788150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:54.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa786800640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7880b6b60 0x7fa7880b6fe0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fa7800ab040 tx=0x7fa7800194c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa77e000640 1 -- 172.21.15.40:0/4279896446 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7800bc070 con 0x7fa7880b6b60 2024-03-24T00:33:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa77e000640 1 -- 172.21.15.40:0/4279896446 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa780001c00 con 0x7fa7880b6b60 2024-03-24T00:33:54.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa77e000640 1 -- 172.21.15.40:0/4279896446 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780005570 con 0x7fa7880b6b60 2024-03-24T00:33:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa787200640 1 -- 172.21.15.40:0/4279896446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78815c640 con 0x7fa7880b6b60 2024-03-24T00:33:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa787200640 1 -- 172.21.15.40:0/4279896446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78815cc00 con 0x7fa7880b6b60 2024-03-24T00:33:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.727+0000 7fa787200640 1 -- 172.21.15.40:0/4279896446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78804f410 con 0x7fa7880b6b60 2024-03-24T00:33:54.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.731+0000 7fa77e000640 1 -- 172.21.15.40:0/4279896446 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa7800bf070 con 0x7fa7880b6b60 2024-03-24T00:33:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.731+0000 7fa77e000640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa764067690 0x7fa764069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.731+0000 7fa77e000640 1 -- 172.21.15.40:0/4279896446 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fa78011ef10 con 0x7fa7880b6b60 2024-03-24T00:33:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.731+0000 7fa785e00640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa764067690 0x7fa764069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:54.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.731+0000 7fa785e00640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa764067690 0x7fa764069b50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa778009be0 tx=0x7fa778008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:54.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.735+0000 7fa77e000640 1 -- 172.21.15.40:0/4279896446 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa7800ad020 con 0x7fa7880b6b60 2024-03-24T00:33:54.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:54.899+0000 7fa787200640 1 -- 172.21.15.40:0/4279896446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa78815b8c0 con 0x7fa7880b6b60 2024-03-24T00:33:55.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.539+0000 7fa77e000640 1 -- 172.21.15.40:0/4279896446 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v50) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa770007e20 con 0x7fa7880b6b60 2024-03-24T00:33:55.543 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-03-24T00:33:55.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.543+0000 7fa787200640 1 -- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa764067690 msgr2=0x7fa764069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.543+0000 7fa787200640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa764067690 0x7fa764069b50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa778009be0 tx=0x7fa778008040 comp rx=0 tx=0).stop 2024-03-24T00:33:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.543+0000 7fa787200640 1 -- 172.21.15.40:0/4279896446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7880b6b60 msgr2=0x7fa7880b6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.543+0000 7fa787200640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7880b6b60 0x7fa7880b6fe0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fa7800ab040 tx=0x7fa7800194c0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.543+0000 7fa787200640 1 -- 172.21.15.40:0/4279896446 shutdown_connections 2024-03-24T00:33:55.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.543+0000 7fa787200640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa764067690 0x7fa764069b50 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.543+0000 7fa787200640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa7880b6b60 0x7fa7880b6fe0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.543+0000 7fa787200640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa78815e690 0x7fa78815eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.543+0000 7fa787200640 1 --2- 172.21.15.40:0/4279896446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa76c004d50 0x7fa788150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.543+0000 7fa787200640 1 -- 172.21.15.40:0/4279896446 >> 172.21.15.40:0/4279896446 conn(0x7fa7880d19e0 msgr2=0x7fa7880b07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.543+0000 7fa787200640 1 -- 172.21.15.40:0/4279896446 shutdown_connections 2024-03-24T00:33:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.543+0000 7fa787200640 1 -- 172.21.15.40:0/4279896446 wait complete. 2024-03-24T00:33:55.573 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random 2024-03-24T00:33:55.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f7010a00640 1 Processor -- start 2024-03-24T00:33:55.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f7010a00640 1 -- start start 2024-03-24T00:33:55.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f7010a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f700c0ce070 0x7f700c0cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:55.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f7010a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f700c15ce60 0x7f700c0ce450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f7010a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700c0d24e0 con 0x7f700c0cca10 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f7010a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f700c15c2c0 con 0x7f700c0ce070 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f7010a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700c0d27c0 con 0x7f700c15ce60 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f700b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f700c0ce070 0x7f700c0cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f7003e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f700c0cca10 0x7f700c15c750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60928/0 (socket says 172.21.15.40:60928) 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f700b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f700c0ce070 0x7f700c0cc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40744/0 (socket says 172.21.15.40:40744) 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f7003e00640 1 -- 172.21.15.40:0/2800055057 learned_addr learned my addr 172.21.15.40:0/2800055057 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f700be00640 1 --2- 172.21.15.40:0/2800055057 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f700c15ce60 0x7f700c0ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f700b400640 1 -- 172.21.15.40:0/2800055057 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f700c15ce60 msgr2=0x7f700c0ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f700b400640 1 --2- 172.21.15.40:0/2800055057 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f700c15ce60 0x7f700c0ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f700b400640 1 -- 172.21.15.40:0/2800055057 >> v1:172.21.15.40:6789/0 conn(0x7f700c0cca10 legacy=0x7f700c15c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f700b400640 1 -- 172.21.15.40:0/2800055057 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c0cf590 con 0x7f700c0ce070 2024-03-24T00:33:55.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.727+0000 7f700be00640 1 --2- 172.21.15.40:0/2800055057 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f700c15ce60 0x7f700c0ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:55.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.731+0000 7f700b400640 1 --2- 172.21.15.40:0/2800055057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f700c0ce070 0x7f700c0cc4d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0097c0 tx=0x7f6ffc01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be521b078793b8ba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:55.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.731+0000 7f700aa00640 1 -- 172.21.15.40:0/2800055057 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0aa5f0 con 0x7f700c0ce070 2024-03-24T00:33:55.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.731+0000 7f700aa00640 1 -- 172.21.15.40:0/2800055057 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc0af070 con 0x7f700c0ce070 2024-03-24T00:33:55.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.731+0000 7f700aa00640 1 -- 172.21.15.40:0/2800055057 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0aa9b0 con 0x7f700c0ce070 2024-03-24T00:33:55.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.731+0000 7f7010a00640 1 -- 172.21.15.40:0/2800055057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f700c0ce070 msgr2=0x7f700c0cc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:55.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.731+0000 7f7010a00640 1 --2- 172.21.15.40:0/2800055057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f700c0ce070 0x7f700c0cc4d0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0097c0 tx=0x7f6ffc01c470 comp rx=0 tx=0).stop 2024-03-24T00:33:55.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.731+0000 7f7010a00640 1 -- 172.21.15.40:0/2800055057 shutdown_connections 2024-03-24T00:33:55.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.731+0000 7f7010a00640 1 --2- 172.21.15.40:0/2800055057 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f700c15ce60 0x7f700c0ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.731+0000 7f7010a00640 1 --2- 172.21.15.40:0/2800055057 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f700c0ce070 0x7f700c0cc4d0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.731+0000 7f7010a00640 1 -- 172.21.15.40:0/2800055057 >> 172.21.15.40:0/2800055057 conn(0x7f700c0d19c0 msgr2=0x7f700c0d1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:55.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.731+0000 7f7010a00640 1 -- 172.21.15.40:0/2800055057 shutdown_connections 2024-03-24T00:33:55.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.731+0000 7f7010a00640 1 -- 172.21.15.40:0/2800055057 wait complete. 2024-03-24T00:33:55.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7010a00640 1 Processor -- start 2024-03-24T00:33:55.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7010a00640 1 -- start start 2024-03-24T00:33:55.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7010a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f700c0cca10 0x7f700c1fa7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7010a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f700c0ce070 0x7f700c0b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7010a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f700c15ce60 0x7f700c06c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7010a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c15c070 con 0x7f700c0cca10 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7010a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700c15bef0 con 0x7f700c15ce60 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7010a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700c169280 con 0x7f700c0ce070 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f700b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f700c0cca10 0x7f700c1fa7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f700b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f700c0cca10 0x7f700c1fa7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40760/0 (socket says 172.21.15.40:40760) 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f700b400640 1 -- 172.21.15.40:0/986606412 learned_addr learned my addr 172.21.15.40:0/986606412 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f700b400640 1 -- 172.21.15.40:0/986606412 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f700c0ce070 msgr2=0x7f700c0b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f700b400640 1 --2- 172.21.15.40:0/986606412 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f700c0ce070 0x7f700c0b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f700b400640 1 -- 172.21.15.40:0/986606412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f700c15ce60 msgr2=0x7f700c06c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f700b400640 1 --2- 172.21.15.40:0/986606412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f700c15ce60 0x7f700c06c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f700b400640 1 -- 172.21.15.40:0/986606412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700c0b6fc0 con 0x7f700c0cca10 2024-03-24T00:33:55.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f700b400640 1 --2- 172.21.15.40:0/986606412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f700c0cca10 0x7f700c1fa7b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0ac040 tx=0x7f6ffc01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7003400640 1 -- 172.21.15.40:0/986606412 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0be610 con 0x7f700c0cca10 2024-03-24T00:33:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7003400640 1 -- 172.21.15.40:0/986606412 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc0be7f0 con 0x7f700c0cca10 2024-03-24T00:33:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7003400640 1 -- 172.21.15.40:0/986606412 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0bebe0 con 0x7f700c0cca10 2024-03-24T00:33:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7010a00640 1 -- 172.21.15.40:0/986606412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700c0b72a0 con 0x7f700c0cca10 2024-03-24T00:33:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7010a00640 1 -- 172.21.15.40:0/986606412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700c0ae9c0 con 0x7f700c0cca10 2024-03-24T00:33:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.735+0000 7f7010a00640 1 -- 172.21.15.40:0/986606412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f700c04f410 con 0x7f700c0cca10 2024-03-24T00:33:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.739+0000 7f7003400640 1 -- 172.21.15.40:0/986606412 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6ffc0bc020 con 0x7f700c0cca10 2024-03-24T00:33:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.739+0000 7f7003400640 1 --2- 172.21.15.40:0/986606412 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fe8067660 0x7f6fe8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.739+0000 7f7003400640 1 -- 172.21.15.40:0/986606412 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f6ffc11fcb0 con 0x7f700c0cca10 2024-03-24T00:33:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.739+0000 7f7003e00640 1 --2- 172.21.15.40:0/986606412 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fe8067660 0x7f6fe8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:55.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.739+0000 7f7003e00640 1 --2- 172.21.15.40:0/986606412 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fe8067660 0x7f6fe8069b20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4000f70 tx=0x7f6ff4007a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:55.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.743+0000 7f7003400640 1 -- 172.21.15.40:0/986606412 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6ffc016090 con 0x7f700c0cca10 2024-03-24T00:33:55.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.907+0000 7f7010a00640 1 -- 172.21.15.40:0/986606412 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"} v 0) v1 -- 0x7f700c0aeb90 con 0x7f700c0cca10 2024-03-24T00:33:55.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7003400640 1 -- 172.21.15.40:0/986606412 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"}]=0 v23) v1 ==== 110+0+0 (secure 0 0 0) 0x7f6ffc0f0b10 con 0x7f700c0cca10 2024-03-24T00:33:55.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7010a00640 1 -- 172.21.15.40:0/986606412 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fe8067660 msgr2=0x7f6fe8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:55.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7010a00640 1 --2- 172.21.15.40:0/986606412 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fe8067660 0x7f6fe8069b20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4000f70 tx=0x7f6ff4007a90 comp rx=0 tx=0).stop 2024-03-24T00:33:55.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7010a00640 1 -- 172.21.15.40:0/986606412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f700c0cca10 msgr2=0x7f700c1fa7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:55.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7010a00640 1 --2- 172.21.15.40:0/986606412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f700c0cca10 0x7f700c1fa7b0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0ac040 tx=0x7f6ffc01acb0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7010a00640 1 -- 172.21.15.40:0/986606412 shutdown_connections 2024-03-24T00:33:55.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7010a00640 1 --2- 172.21.15.40:0/986606412 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fe8067660 0x7f6fe8069b20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7010a00640 1 --2- 172.21.15.40:0/986606412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f700c15ce60 0x7f700c06c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7010a00640 1 --2- 172.21.15.40:0/986606412 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f700c0ce070 0x7f700c0b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7010a00640 1 --2- 172.21.15.40:0/986606412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f700c0cca10 0x7f700c1fa7b0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:55.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7010a00640 1 -- 172.21.15.40:0/986606412 >> 172.21.15.40:0/986606412 conn(0x7f700c0d19c0 msgr2=0x7f700c070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:55.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7010a00640 1 -- 172.21.15.40:0/986606412 shutdown_connections 2024-03-24T00:33:55.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:55.911+0000 7f7010a00640 1 -- 172.21.15.40:0/986606412 wait complete. 2024-03-24T00:33:55.934 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_distributed 2024-03-24T00:33:56.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.075+0000 7fab75200640 1 Processor -- start 2024-03-24T00:33:56.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.075+0000 7fab75200640 1 -- start start 2024-03-24T00:33:56.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.075+0000 7fab75200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fab7015ffd0 0x7fab700ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.075+0000 7fab75200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab700728d0 0x7fab700cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.075+0000 7fab75200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab70072d40 con 0x7fab7015bd00 2024-03-24T00:33:56.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.075+0000 7fab75200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab700cf790 con 0x7fab7015ffd0 2024-03-24T00:33:56.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.075+0000 7fab75200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab70071de0 con 0x7fab700728d0 2024-03-24T00:33:56.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.075+0000 7fab6e800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab700728d0 0x7fab700cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.075+0000 7fab65400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fab7015ffd0 0x7fab700ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.075+0000 7fab6de00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fab7015bd00 0x7fab7015f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:35422/0 (socket says 172.21.15.40:35422) 2024-03-24T00:33:56.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.075+0000 7fab6de00640 1 -- 172.21.15.40:0/1847770 learned_addr learned my addr 172.21.15.40:0/1847770 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:56.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.079+0000 7fab65400640 1 -- 172.21.15.40:0/1847770 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab700728d0 msgr2=0x7fab700cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:56.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.079+0000 7fab65400640 1 --2- 172.21.15.40:0/1847770 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab700728d0 0x7fab700cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.079+0000 7fab65400640 1 -- 172.21.15.40:0/1847770 >> v1:172.21.15.53:6789/0 conn(0x7fab7015bd00 legacy=0x7fab7015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.079+0000 7fab65400640 1 -- 172.21.15.40:0/1847770 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab700cea50 con 0x7fab7015ffd0 2024-03-24T00:33:56.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.079+0000 7fab6e800640 1 --2- 172.21.15.40:0/1847770 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab700728d0 0x7fab700cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:56.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.079+0000 7fab65400640 1 --2- 172.21.15.40:0/1847770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7015ffd0 0x7fab700ae8e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fab580097c0 tx=0x7fab5801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1defc2bdd7f5e548 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:56.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.079+0000 7fab6d400640 1 -- 172.21.15.40:0/1847770 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab580a7070 con 0x7fab7015ffd0 2024-03-24T00:33:56.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.079+0000 7fab6d400640 1 -- 172.21.15.40:0/1847770 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab580b0070 con 0x7fab7015ffd0 2024-03-24T00:33:56.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.079+0000 7fab6d400640 1 -- 172.21.15.40:0/1847770 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab580ab890 con 0x7fab7015ffd0 2024-03-24T00:33:56.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.079+0000 7fab75200640 1 -- 172.21.15.40:0/1847770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7015ffd0 msgr2=0x7fab700ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.079+0000 7fab75200640 1 --2- 172.21.15.40:0/1847770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7015ffd0 0x7fab700ae8e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fab580097c0 tx=0x7fab5801c470 comp rx=0 tx=0).stop 2024-03-24T00:33:56.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 -- 172.21.15.40:0/1847770 shutdown_connections 2024-03-24T00:33:56.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 --2- 172.21.15.40:0/1847770 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab700728d0 0x7fab700cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 --2- 172.21.15.40:0/1847770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7015ffd0 0x7fab700ae8e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 -- 172.21.15.40:0/1847770 >> 172.21.15.40:0/1847770 conn(0x7fab700b1660 msgr2=0x7fab700b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:56.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 -- 172.21.15.40:0/1847770 shutdown_connections 2024-03-24T00:33:56.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 -- 172.21.15.40:0/1847770 wait complete. 2024-03-24T00:33:56.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 Processor -- start 2024-03-24T00:33:56.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 -- start start 2024-03-24T00:33:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab700728d0 0x7fab7014d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7015bd00 0x7fab7014dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab7015ffd0 0x7fab7015dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7006d320 con 0x7fab7015ffd0 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7015c0e0 con 0x7fab7015bd00 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab75200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7006d4a0 con 0x7fab700728d0 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab6e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab7015ffd0 0x7fab7015dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab6e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab7015ffd0 0x7fab7015dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40766/0 (socket says 172.21.15.40:40766) 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab6de00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab700728d0 0x7fab7014d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab6e800640 1 -- 172.21.15.40:0/1845830402 learned_addr learned my addr 172.21.15.40:0/1845830402 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab65400640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7015bd00 0x7fab7014dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab6e800640 1 -- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab700728d0 msgr2=0x7fab7014d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab6e800640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab700728d0 0x7fab7014d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab6e800640 1 -- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7015bd00 msgr2=0x7fab7014dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab6e800640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7015bd00 0x7fab7014dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab6e800640 1 -- 172.21.15.40:0/1845830402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7014b660 con 0x7fab7015ffd0 2024-03-24T00:33:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.083+0000 7fab6e800640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab7015ffd0 0x7fab7015dc60 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fab68009f00 tx=0x7fab6800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.087+0000 7fab66a00640 1 -- 172.21.15.40:0/1845830402 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68018030 con 0x7fab7015ffd0 2024-03-24T00:33:56.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.087+0000 7fab66a00640 1 -- 172.21.15.40:0/1845830402 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6800ce50 con 0x7fab7015ffd0 2024-03-24T00:33:56.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.087+0000 7fab66a00640 1 -- 172.21.15.40:0/1845830402 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68003bc0 con 0x7fab7015ffd0 2024-03-24T00:33:56.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.087+0000 7fab75200640 1 -- 172.21.15.40:0/1845830402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7014b940 con 0x7fab7015ffd0 2024-03-24T00:33:56.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.087+0000 7fab75200640 1 -- 172.21.15.40:0/1845830402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7014bf00 con 0x7fab7015ffd0 2024-03-24T00:33:56.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.087+0000 7fab75200640 1 -- 172.21.15.40:0/1845830402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab70068170 con 0x7fab7015ffd0 2024-03-24T00:33:56.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.087+0000 7fab66a00640 1 -- 172.21.15.40:0/1845830402 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fab6801b070 con 0x7fab7015ffd0 2024-03-24T00:33:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.087+0000 7fab66a00640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab4c067660 0x7fab4c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.087+0000 7fab66a00640 1 -- 172.21.15.40:0/1845830402 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fab680860f0 con 0x7fab7015ffd0 2024-03-24T00:33:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.087+0000 7fab6de00640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab4c067660 0x7fab4c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.091+0000 7fab6de00640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab4c067660 0x7fab4c069b20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fab60000f70 tx=0x7fab60007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:56.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.091+0000 7fab66a00640 1 -- 172.21.15.40:0/1845830402 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fab68010040 con 0x7fab7015ffd0 2024-03-24T00:33:56.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.255+0000 7fab75200640 1 -- 172.21.15.40:0/1845830402 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"} v 0) v1 -- 0x7fab70156e80 con 0x7fab7015ffd0 2024-03-24T00:33:56.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab66a00640 1 -- 172.21.15.40:0/1845830402 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"}]=0 v24) v1 ==== 115+0+0 (secure 0 0 0) 0x7fab680567e0 con 0x7fab7015ffd0 2024-03-24T00:33:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab75200640 1 -- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab4c067660 msgr2=0x7fab4c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab75200640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab4c067660 0x7fab4c069b20 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fab60000f70 tx=0x7fab60007e90 comp rx=0 tx=0).stop 2024-03-24T00:33:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab75200640 1 -- 172.21.15.40:0/1845830402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab7015ffd0 msgr2=0x7fab7015dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab75200640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab7015ffd0 0x7fab7015dc60 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fab68009f00 tx=0x7fab6800c490 comp rx=0 tx=0).stop 2024-03-24T00:33:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab75200640 1 -- 172.21.15.40:0/1845830402 shutdown_connections 2024-03-24T00:33:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab75200640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab4c067660 0x7fab4c069b20 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab75200640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab7015ffd0 0x7fab7015dc60 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab75200640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7015bd00 0x7fab7014dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab75200640 1 --2- 172.21.15.40:0/1845830402 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab700728d0 0x7fab7014d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab75200640 1 -- 172.21.15.40:0/1845830402 >> 172.21.15.40:0/1845830402 conn(0x7fab700b1660 msgr2=0x7fab70153b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:56.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab75200640 1 -- 172.21.15.40:0/1845830402 shutdown_connections 2024-03-24T00:33:56.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.263+0000 7fab75200640 1 -- 172.21.15.40:0/1845830402 wait complete. 2024-03-24T00:33:56.286 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random_max 2024-03-24T00:33:56.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.427+0000 7f03a3600640 1 Processor -- start 2024-03-24T00:33:56.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.427+0000 7f03a3600640 1 -- start start 2024-03-24T00:33:56.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.427+0000 7f03a3600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f039c0cf4e0 0x7f039c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.427+0000 7f03a3600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c15f2c0 con 0x7f039c0b3380 2024-03-24T00:33:56.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.427+0000 7f03a3600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c06d230 con 0x7f039c072b60 2024-03-24T00:33:56.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.427+0000 7f03a3600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f039c06cfe0 con 0x7f039c0cf4e0 2024-03-24T00:33:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.427+0000 7f03a0c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f039c0cf4e0 0x7f039c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.427+0000 7f03a1600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f039c0b3380 0x7f039c070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60936/0 (socket says 172.21.15.40:60936) 2024-03-24T00:33:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.427+0000 7f03a1600640 1 -- 172.21.15.40:0/2387544400 learned_addr learned my addr 172.21.15.40:0/2387544400 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.427+0000 7f03a0c00640 1 -- 172.21.15.40:0/2387544400 >> v1:172.21.15.53:6789/0 conn(0x7f039c072b60 legacy=0x7f039c0b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.427+0000 7f03a0c00640 1 -- 172.21.15.40:0/2387544400 >> v1:172.21.15.40:6789/0 conn(0x7f039c0b3380 legacy=0x7f039c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.427+0000 7f03a0c00640 1 -- 172.21.15.40:0/2387544400 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c06d510 con 0x7f039c0cf4e0 2024-03-24T00:33:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f03a0c00640 1 --2- 172.21.15.40:0/2387544400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0cf4e0 0x7f039c06dd10 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f03900097c0 tx=0x7f039001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=159bd882fed7f6af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f039b400640 1 -- 172.21.15.40:0/2387544400 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03900b5070 con 0x7f039c0cf4e0 2024-03-24T00:33:56.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f039b400640 1 -- 172.21.15.40:0/2387544400 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03900a8950 con 0x7f039c0cf4e0 2024-03-24T00:33:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f039b400640 1 -- 172.21.15.40:0/2387544400 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03900a8d20 con 0x7f039c0cf4e0 2024-03-24T00:33:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f03a3600640 1 -- 172.21.15.40:0/2387544400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0cf4e0 msgr2=0x7f039c06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f03a3600640 1 --2- 172.21.15.40:0/2387544400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0cf4e0 0x7f039c06dd10 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f03900097c0 tx=0x7f039001ec50 comp rx=0 tx=0).stop 2024-03-24T00:33:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f03a3600640 1 -- 172.21.15.40:0/2387544400 shutdown_connections 2024-03-24T00:33:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f03a3600640 1 --2- 172.21.15.40:0/2387544400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0cf4e0 0x7f039c06dd10 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f03a3600640 1 -- 172.21.15.40:0/2387544400 >> 172.21.15.40:0/2387544400 conn(0x7f039c0b1660 msgr2=0x7f039c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f03a3600640 1 -- 172.21.15.40:0/2387544400 shutdown_connections 2024-03-24T00:33:56.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f03a3600640 1 -- 172.21.15.40:0/2387544400 wait complete. 2024-03-24T00:33:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f03a3600640 1 Processor -- start 2024-03-24T00:33:56.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.431+0000 7f03a3600640 1 -- start start 2024-03-24T00:33:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f03a3600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f039c072b60 0x7f039c14e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f03a3600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0b3380 0x7f039c0b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f03a3600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f039c0cf4e0 0x7f039c0b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f03a3600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c15ef20 con 0x7f039c0b3380 2024-03-24T00:33:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f03a3600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c06d880 con 0x7f039c072b60 2024-03-24T00:33:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f03a3600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f039c15f0a0 con 0x7f039c0cf4e0 2024-03-24T00:33:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f039be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0b3380 0x7f039c0b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f039be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0b3380 0x7f039c0b04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40780/0 (socket says 172.21.15.40:40780) 2024-03-24T00:33:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f039be00640 1 -- 172.21.15.40:0/82619673 learned_addr learned my addr 172.21.15.40:0/82619673 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f03a0c00640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f039c072b60 0x7f039c14e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f03a1600640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f039c0cf4e0 0x7f039c0b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f039be00640 1 -- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f039c0cf4e0 msgr2=0x7f039c0b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f039be00640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f039c0cf4e0 0x7f039c0b09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f039be00640 1 -- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f039c072b60 msgr2=0x7f039c14e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f039be00640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f039c072b60 0x7f039c14e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f039be00640 1 -- 172.21.15.40:0/82619673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c1e79d0 con 0x7f039c0b3380 2024-03-24T00:33:56.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f03a1600640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f039c0cf4e0 0x7f039c0b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f039be00640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0b3380 0x7f039c0b04a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f038800c410 tx=0x7f038800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f0398c00640 1 -- 172.21.15.40:0/82619673 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0388011790 con 0x7f039c0b3380 2024-03-24T00:33:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f0398c00640 1 -- 172.21.15.40:0/82619673 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0388017070 con 0x7f039c0b3380 2024-03-24T00:33:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f0398c00640 1 -- 172.21.15.40:0/82619673 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0388011bb0 con 0x7f039c0b3380 2024-03-24T00:33:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f03a3600640 1 -- 172.21.15.40:0/82619673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f039c15ced0 con 0x7f039c0b3380 2024-03-24T00:33:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f03a3600640 1 -- 172.21.15.40:0/82619673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039c15c350 con 0x7f039c0b3380 2024-03-24T00:33:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.435+0000 7f03a3600640 1 -- 172.21.15.40:0/82619673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f039c068170 con 0x7f039c0b3380 2024-03-24T00:33:56.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.439+0000 7f0398c00640 1 -- 172.21.15.40:0/82619673 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f038801c460 con 0x7f039c0b3380 2024-03-24T00:33:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.439+0000 7f0398c00640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f037c067660 0x7f037c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.439+0000 7f0398c00640 1 -- 172.21.15.40:0/82619673 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f0388085240 con 0x7f039c0b3380 2024-03-24T00:33:56.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.439+0000 7f03a0c00640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f037c067660 0x7f037c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.439+0000 7f03a0c00640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f037c067660 0x7f037c069b20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f03900b2000 tx=0x7f0390008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:56.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.443+0000 7f0398c00640 1 -- 172.21.15.40:0/82619673 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0388021020 con 0x7f039c0b3380 2024-03-24T00:33:56.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.607+0000 7f03a3600640 1 -- 172.21.15.40:0/82619673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"} v 0) v1 -- 0x7f039c06e9a0 con 0x7f039c0b3380 2024-03-24T00:33:56.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f0398c00640 1 -- 172.21.15.40:0/82619673 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"}]=0 v25) v1 ==== 114+0+0 (secure 0 0 0) 0x7f039c1e79d0 con 0x7f039c0b3380 2024-03-24T00:33:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f03a3600640 1 -- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f037c067660 msgr2=0x7f037c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f03a3600640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f037c067660 0x7f037c069b20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f03900b2000 tx=0x7f0390008040 comp rx=0 tx=0).stop 2024-03-24T00:33:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f03a3600640 1 -- 172.21.15.40:0/82619673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0b3380 msgr2=0x7f039c0b04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f03a3600640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0b3380 0x7f039c0b04a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f038800c410 tx=0x7f038800c8d0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f03a3600640 1 -- 172.21.15.40:0/82619673 shutdown_connections 2024-03-24T00:33:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f03a3600640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f037c067660 0x7f037c069b20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f03a3600640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f039c0cf4e0 0x7f039c0b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f03a3600640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f039c0b3380 0x7f039c0b04a0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f03a3600640 1 --2- 172.21.15.40:0/82619673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f039c072b60 0x7f039c14e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f03a3600640 1 -- 172.21.15.40:0/82619673 >> 172.21.15.40:0/82619673 conn(0x7f039c0b1660 msgr2=0x7f039c072360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f03a3600640 1 -- 172.21.15.40:0/82619673 shutdown_connections 2024-03-24T00:33:56.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.615+0000 7f03a3600640 1 -- 172.21.15.40:0/82619673 wait complete. 2024-03-24T00:33:56.639 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:33:56.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f8360200640 1 Processor -- start 2024-03-24T00:33:56.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f8360200640 1 -- start start 2024-03-24T00:33:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f8360200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835815a560 0x7f83580ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f8360200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f835806d3a0 0x7f835806d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f8360200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8358070ed0 con 0x7f83580aec70 2024-03-24T00:33:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f8360200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83580cf240 con 0x7f835806d3a0 2024-03-24T00:33:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f8360200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83580cf520 con 0x7f835815a560 2024-03-24T00:33:56.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f835ce00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f83580aec70 0x7f83580cc4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60946/0 (socket says 172.21.15.40:60946) 2024-03-24T00:33:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f835ce00640 1 -- 172.21.15.40:0/2941927684 learned_addr learned my addr 172.21.15.40:0/2941927684 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f835e200640 1 --2- 172.21.15.40:0/2941927684 >> v2:172.21.15.40:3300/0 conn(0x7f835806d3a0 0x7f835806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f835d800640 1 --2- 172.21.15.40:0/2941927684 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835815a560 0x7f83580ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f835e200640 1 -- 172.21.15.40:0/2941927684 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835815a560 msgr2=0x7f83580ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f835e200640 1 --2- 172.21.15.40:0/2941927684 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835815a560 0x7f83580ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f835e200640 1 -- 172.21.15.40:0/2941927684 >> v1:172.21.15.40:6789/0 conn(0x7f83580aec70 legacy=0x7f83580cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.779+0000 7f835e200640 1 -- 172.21.15.40:0/2941927684 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83580b3aa0 con 0x7f835806d3a0 2024-03-24T00:33:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.783+0000 7f835e200640 1 --2- 172.21.15.40:0/2941927684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835806d3a0 0x7f835806d850 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8350009dc0 tx=0x7f835001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583af1bfaae050 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.783+0000 7f8357e00640 1 -- 172.21.15.40:0/2941927684 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83500ac070 con 0x7f835806d3a0 2024-03-24T00:33:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.783+0000 7f8357e00640 1 -- 172.21.15.40:0/2941927684 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83500a6680 con 0x7f835806d3a0 2024-03-24T00:33:56.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.783+0000 7f8357e00640 1 -- 172.21.15.40:0/2941927684 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83500a6a50 con 0x7f835806d3a0 2024-03-24T00:33:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.783+0000 7f8360200640 1 -- 172.21.15.40:0/2941927684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835806d3a0 msgr2=0x7f835806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.783+0000 7f8360200640 1 --2- 172.21.15.40:0/2941927684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835806d3a0 0x7f835806d850 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8350009dc0 tx=0x7f835001c830 comp rx=0 tx=0).stop 2024-03-24T00:33:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.783+0000 7f8360200640 1 -- 172.21.15.40:0/2941927684 shutdown_connections 2024-03-24T00:33:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.783+0000 7f8360200640 1 --2- 172.21.15.40:0/2941927684 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835806d3a0 0x7f835806d850 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.783+0000 7f8360200640 1 --2- 172.21.15.40:0/2941927684 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835815a560 0x7f83580ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.783+0000 7f8360200640 1 -- 172.21.15.40:0/2941927684 >> 172.21.15.40:0/2941927684 conn(0x7f83580b1660 msgr2=0x7f83580b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.783+0000 7f8360200640 1 -- 172.21.15.40:0/2941927684 shutdown_connections 2024-03-24T00:33:56.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.783+0000 7f8360200640 1 -- 172.21.15.40:0/2941927684 wait complete. 2024-03-24T00:33:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8360200640 1 Processor -- start 2024-03-24T00:33:56.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8360200640 1 -- start start 2024-03-24T00:33:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8360200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835806d3a0 0x7f83580c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8360200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83580aec70 0x7f83580ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8360200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835815a560 0x7f83580ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8360200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83580b4350 con 0x7f835806d3a0 2024-03-24T00:33:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8360200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83580b41d0 con 0x7f83580aec70 2024-03-24T00:33:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8360200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83580b44d0 con 0x7f835815a560 2024-03-24T00:33:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f835d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835806d3a0 0x7f83580c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f835d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835806d3a0 0x7f83580c6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40804/0 (socket says 172.21.15.40:40804) 2024-03-24T00:33:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f835d800640 1 -- 172.21.15.40:0/974147438 learned_addr learned my addr 172.21.15.40:0/974147438 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:56.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f835d800640 1 -- 172.21.15.40:0/974147438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835815a560 msgr2=0x7f83580ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f835d800640 1 --2- 172.21.15.40:0/974147438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835815a560 0x7f83580ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f835d800640 1 -- 172.21.15.40:0/974147438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83580aec70 msgr2=0x7f83580ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f835d800640 1 --2- 172.21.15.40:0/974147438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83580aec70 0x7f83580ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f835d800640 1 -- 172.21.15.40:0/974147438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83580c52e0 con 0x7f835806d3a0 2024-03-24T00:33:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f835d800640 1 --2- 172.21.15.40:0/974147438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835806d3a0 0x7f83580c6360 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f834c00d5f0 tx=0x7f834c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8355600640 1 -- 172.21.15.40:0/974147438 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c0125f0 con 0x7f835806d3a0 2024-03-24T00:33:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8355600640 1 -- 172.21.15.40:0/974147438 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f834c019070 con 0x7f835806d3a0 2024-03-24T00:33:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8355600640 1 -- 172.21.15.40:0/974147438 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f834c0129d0 con 0x7f835806d3a0 2024-03-24T00:33:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8360200640 1 -- 172.21.15.40:0/974147438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83580c55a0 con 0x7f835806d3a0 2024-03-24T00:33:56.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.787+0000 7f8360200640 1 -- 172.21.15.40:0/974147438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83580c5b60 con 0x7f835806d3a0 2024-03-24T00:33:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.791+0000 7f8360200640 1 -- 172.21.15.40:0/974147438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8358068170 con 0x7f835806d3a0 2024-03-24T00:33:56.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.791+0000 7f8355600640 1 -- 172.21.15.40:0/974147438 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f834c012b70 con 0x7f835806d3a0 2024-03-24T00:33:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.791+0000 7f8355600640 1 --2- 172.21.15.40:0/974147438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8338067660 0x7f8338069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.791+0000 7f8355600640 1 -- 172.21.15.40:0/974147438 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f834c085b10 con 0x7f835806d3a0 2024-03-24T00:33:56.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.791+0000 7f835ce00640 1 --2- 172.21.15.40:0/974147438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8338067660 0x7f8338069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.791+0000 7f835ce00640 1 --2- 172.21.15.40:0/974147438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8338067660 0x7f8338069b20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8344000f70 tx=0x7f8344007a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.795+0000 7f8355600640 1 -- 172.21.15.40:0/974147438 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f834c011340 con 0x7f835806d3a0 2024-03-24T00:33:56.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.959+0000 7f8360200640 1 -- 172.21.15.40:0/974147438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f83580ae690 con 0x7f835806d3a0 2024-03-24T00:33:56.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8355600640 1 -- 172.21.15.40:0/974147438 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7f834c056970 con 0x7f835806d3a0 2024-03-24T00:33:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8360200640 1 -- 172.21.15.40:0/974147438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8338067660 msgr2=0x7f8338069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8360200640 1 --2- 172.21.15.40:0/974147438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8338067660 0x7f8338069b20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8344000f70 tx=0x7f8344007a90 comp rx=0 tx=0).stop 2024-03-24T00:33:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8360200640 1 -- 172.21.15.40:0/974147438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835806d3a0 msgr2=0x7f83580c6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8360200640 1 --2- 172.21.15.40:0/974147438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835806d3a0 0x7f83580c6360 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f834c00d5f0 tx=0x7f834c00dab0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8360200640 1 -- 172.21.15.40:0/974147438 shutdown_connections 2024-03-24T00:33:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8360200640 1 --2- 172.21.15.40:0/974147438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8338067660 0x7f8338069b20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8360200640 1 --2- 172.21.15.40:0/974147438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835815a560 0x7f83580ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8360200640 1 --2- 172.21.15.40:0/974147438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83580aec70 0x7f83580ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8360200640 1 --2- 172.21.15.40:0/974147438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835806d3a0 0x7f83580c6360 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8360200640 1 -- 172.21.15.40:0/974147438 >> 172.21.15.40:0/974147438 conn(0x7f83580b1660 msgr2=0x7f83580b2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8360200640 1 -- 172.21.15.40:0/974147438 shutdown_connections 2024-03-24T00:33:56.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:56.967+0000 7f8360200640 1 -- 172.21.15.40:0/974147438 wait complete. 2024-03-24T00:33:56.989 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_failover' 2024-03-24T00:33:57.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9e5800640 1 Processor -- start 2024-03-24T00:33:57.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9e5800640 1 -- start start 2024-03-24T00:33:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9e5800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9e006e620 0x7fd9e00b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9e5800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e0070f60 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9e5800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e00b3620 con 0x7fd9e006ea00 2024-03-24T00:33:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9e5800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e00b4c00 con 0x7fd9e006e620 2024-03-24T00:33:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9dea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd9e006e620 0x7fd9e00b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9df400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd9e00ccfa0 0x7fd9e00cd380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60440/0 (socket says 172.21.15.40:60440) 2024-03-24T00:33:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9df400640 1 -- 172.21.15.40:0/3427542341 learned_addr learned my addr 172.21.15.40:0/3427542341 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:57.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9dea00640 1 -- 172.21.15.40:0/3427542341 >> v1:172.21.15.53:6789/0 conn(0x7fd9e006ea00 legacy=0x7fd9e00b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:57.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9dea00640 1 -- 172.21.15.40:0/3427542341 >> v1:172.21.15.40:6789/0 conn(0x7fd9e00ccfa0 legacy=0x7fd9e00cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9de000640 1 -- 172.21.15.40:0/3427542341 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583688402 0 0) 0x7fd9e0070f60 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.127+0000 7fd9dea00640 1 -- 172.21.15.40:0/3427542341 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e00b3900 con 0x7fd9e006e620 2024-03-24T00:33:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.131+0000 7fd9dea00640 1 --2- 172.21.15.40:0/3427542341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e006e620 0x7fd9e00b46c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00097c0 tx=0x7fd9d001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b4c4c4e2f1c00e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.131+0000 7fd9de000640 1 -- 172.21.15.40:0/3427542341 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d00b1070 con 0x7fd9e006e620 2024-03-24T00:33:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.131+0000 7fd9de000640 1 -- 172.21.15.40:0/3427542341 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d00ac600 con 0x7fd9e006e620 2024-03-24T00:33:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.131+0000 7fd9de000640 1 -- 172.21.15.40:0/3427542341 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d00ac9d0 con 0x7fd9e006e620 2024-03-24T00:33:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.131+0000 7fd9e5800640 1 -- 172.21.15.40:0/3427542341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e006e620 msgr2=0x7fd9e00b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.131+0000 7fd9e5800640 1 --2- 172.21.15.40:0/3427542341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e006e620 0x7fd9e00b46c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00097c0 tx=0x7fd9d001c3a0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.131+0000 7fd9e5800640 1 -- 172.21.15.40:0/3427542341 shutdown_connections 2024-03-24T00:33:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.131+0000 7fd9e5800640 1 --2- 172.21.15.40:0/3427542341 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e006e620 0x7fd9e00b46c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.131+0000 7fd9e5800640 1 -- 172.21.15.40:0/3427542341 >> 172.21.15.40:0/3427542341 conn(0x7fd9e00b17b0 msgr2=0x7fd9e00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.131+0000 7fd9e5800640 1 -- 172.21.15.40:0/3427542341 shutdown_connections 2024-03-24T00:33:57.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.131+0000 7fd9e5800640 1 -- 172.21.15.40:0/3427542341 wait complete. 2024-03-24T00:33:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9e5800640 1 Processor -- start 2024-03-24T00:33:57.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9e5800640 1 -- start start 2024-03-24T00:33:57.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9e5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9e006e620 0x7fd9e0159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9e5800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9e006ea00 0x7fd9e01fa010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9e5800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e00ccfa0 0x7fd9e00aaed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9e5800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e0144920 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9e5800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e006ee90 con 0x7fd9e006e620 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9e5800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e01f8f70 con 0x7fd9e006ea00 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9df400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e00ccfa0 0x7fd9e00aaed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9df400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e00ccfa0 0x7fd9e00aaed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38804/0 (socket says 172.21.15.40:38804) 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9df400640 1 -- 172.21.15.40:0/3380877175 learned_addr learned my addr 172.21.15.40:0/3380877175 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9dea00640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9e006e620 0x7fd9e0159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9d6000640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9e006ea00 0x7fd9e01fa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9df400640 1 -- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9e006ea00 msgr2=0x7fd9e01fa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9df400640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9e006ea00 0x7fd9e01fa010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9df400640 1 -- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9e006e620 msgr2=0x7fd9e0159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9df400640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9e006e620 0x7fd9e0159f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9df400640 1 -- 172.21.15.40:0/3380877175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e01f9250 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9d6000640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9e006ea00 0x7fd9e01fa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:33:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9df400640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e00ccfa0 0x7fd9e00aaed0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800caf0 tx=0x7fd9d80021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9d7400640 1 -- 172.21.15.40:0/3380877175 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d800ac00 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9d7400640 1 -- 172.21.15.40:0/3380877175 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d800ada0 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9e5800640 1 -- 172.21.15.40:0/3380877175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e01f9530 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9d7400640 1 -- 172.21.15.40:0/3380877175 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8012640 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.135+0000 7fd9e5800640 1 -- 172.21.15.40:0/3380877175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e01f9af0 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.139+0000 7fd9e5800640 1 -- 172.21.15.40:0/3380877175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e0068170 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.139+0000 7fd9d7400640 1 -- 172.21.15.40:0/3380877175 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd9d80127e0 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.139+0000 7fd9d7400640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd9bc067660 0x7fd9bc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.139+0000 7fd9d7400640 1 -- 172.21.15.40:0/3380877175 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fd9d8085d70 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.139+0000 7fd9dea00640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd9bc067660 0x7fd9bc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:57.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.139+0000 7fd9dea00640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd9bc067660 0x7fd9bc069b20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00ae000 tx=0x7fd9d00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:57.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.143+0000 7fd9d7400640 1 -- 172.21.15.40:0/3380877175 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd9d8056b40 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.363+0000 7fd9e5800640 1 -- 172.21.15.40:0/3380877175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_failover"]} v 0) v1 -- 0x7fd9e0072f50 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.543+0000 7fd9d7400640 1 -- 172.21.15.40:0/3380877175 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_failover"]}]=0 v165) v1 ==== 155+0+0 (secure 0 0 0) 0x7fd9d8056440 con 0x7fd9e00ccfa0 2024-03-24T00:33:57.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.547+0000 7fd9e5800640 1 -- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd9bc067660 msgr2=0x7fd9bc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.547+0000 7fd9e5800640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd9bc067660 0x7fd9bc069b20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00ae000 tx=0x7fd9d00a7000 comp rx=0 tx=0).stop 2024-03-24T00:33:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.547+0000 7fd9e5800640 1 -- 172.21.15.40:0/3380877175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e00ccfa0 msgr2=0x7fd9e00aaed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.547+0000 7fd9e5800640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e00ccfa0 0x7fd9e00aaed0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800caf0 tx=0x7fd9d80021d0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.547+0000 7fd9e5800640 1 -- 172.21.15.40:0/3380877175 shutdown_connections 2024-03-24T00:33:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.547+0000 7fd9e5800640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd9bc067660 0x7fd9bc069b20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.547+0000 7fd9e5800640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9e00ccfa0 0x7fd9e00aaed0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.547+0000 7fd9e5800640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9e006ea00 0x7fd9e01fa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.547+0000 7fd9e5800640 1 --2- 172.21.15.40:0/3380877175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9e006e620 0x7fd9e0159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.547+0000 7fd9e5800640 1 -- 172.21.15.40:0/3380877175 >> 172.21.15.40:0/3380877175 conn(0x7fd9e00b17b0 msgr2=0x7fd9e015db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.547+0000 7fd9e5800640 1 -- 172.21.15.40:0/3380877175 shutdown_connections 2024-03-24T00:33:57.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.547+0000 7fd9e5800640 1 -- 172.21.15.40:0/3380877175 wait complete. 2024-03-24T00:33:57.569 INFO:tasks.cephfs_test_runner:That MDS failover does not cause unnecessary migrations. ... ok 2024-03-24T00:33:57.570 INFO:tasks.cephfs_test_runner:Starting test: test_ephemeral_pin_dist_off (tasks.cephfs.test_exports.TestEphemeralPins) That turning off ephemeral distributed pin merges subtrees. 2024-03-24T00:33:57.570 INFO:tasks.cephfs_test_runner:test_ephemeral_pin_dist_off (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:33:57.570 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:33:57.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.707+0000 7f5c3c400640 1 Processor -- start 2024-03-24T00:33:57.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.707+0000 7f5c3c400640 1 -- start start 2024-03-24T00:33:57.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.711+0000 7f5c3c400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5c34072310 0x7f5c340ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.711+0000 7f5c3c400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c34070e00 con 0x7f5c34141bc0 2024-03-24T00:33:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.711+0000 7f5c3c400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c340cee20 con 0x7f5c3413ec80 2024-03-24T00:33:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.711+0000 7f5c3c400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c340cf100 con 0x7f5c34072310 2024-03-24T00:33:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.711+0000 7f5c3a400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5c34072310 0x7f5c340ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.711+0000 7f5c3a400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5c34072310 0x7f5c340ce800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38818/0 (socket says 172.21.15.40:38818) 2024-03-24T00:33:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.711+0000 7f5c3a400640 1 -- 172.21.15.40:0/4037925197 learned_addr learned my addr 172.21.15.40:0/4037925197 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.711+0000 7f5c39000640 1 --1- 172.21.15.40:0/4037925197 >> v1:172.21.15.40:6789/0 conn(0x7f5c34141bc0 0x7f5c340ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60450/0 (socket says 172.21.15.40:60450) 2024-03-24T00:33:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.711+0000 7f5c3a400640 1 -- 172.21.15.40:0/4037925197 >> v1:172.21.15.53:6789/0 conn(0x7f5c3413ec80 legacy=0x7f5c341414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.711+0000 7f5c3a400640 1 -- 172.21.15.40:0/4037925197 >> v1:172.21.15.40:6789/0 conn(0x7f5c34141bc0 legacy=0x7f5c340ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.711+0000 7f5c3a400640 1 -- 172.21.15.40:0/4037925197 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c340cf3e0 con 0x7f5c34072310 2024-03-24T00:33:57.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c3a400640 1 --2- 172.21.15.40:0/4037925197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c34072310 0x7f5c340ce800 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c009e50 tx=0x7f5c2c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c7e5cf8c6569b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c33e00640 1 -- 172.21.15.40:0/4037925197 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c0a5920 con 0x7f5c34072310 2024-03-24T00:33:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c33e00640 1 -- 172.21.15.40:0/4037925197 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2c0a5ac0 con 0x7f5c34072310 2024-03-24T00:33:57.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c33e00640 1 -- 172.21.15.40:0/4037925197 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c0ad3f0 con 0x7f5c34072310 2024-03-24T00:33:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c3c400640 1 -- 172.21.15.40:0/4037925197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c34072310 msgr2=0x7f5c340ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c3c400640 1 --2- 172.21.15.40:0/4037925197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c34072310 0x7f5c340ce800 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c009e50 tx=0x7f5c2c01ca30 comp rx=0 tx=0).stop 2024-03-24T00:33:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c3c400640 1 -- 172.21.15.40:0/4037925197 shutdown_connections 2024-03-24T00:33:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c3c400640 1 --2- 172.21.15.40:0/4037925197 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c34072310 0x7f5c340ce800 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c3c400640 1 -- 172.21.15.40:0/4037925197 >> 172.21.15.40:0/4037925197 conn(0x7f5c340b17b0 msgr2=0x7f5c340b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c3c400640 1 -- 172.21.15.40:0/4037925197 shutdown_connections 2024-03-24T00:33:57.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c3c400640 1 -- 172.21.15.40:0/4037925197 wait complete. 2024-03-24T00:33:57.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c3c400640 1 Processor -- start 2024-03-24T00:33:57.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c3c400640 1 -- start start 2024-03-24T00:33:57.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c3c400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c34072310 0x7f5c341fa5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:57.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.715+0000 7f5c3c400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3413ec80 0x7f5c3415e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:57.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c3c400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c34141bc0 0x7f5c3415ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:57.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c3c400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c34169f00 con 0x7f5c3413ec80 2024-03-24T00:33:57.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c3c400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c34169d80 con 0x7f5c34072310 2024-03-24T00:33:57.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c3c400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3416a080 con 0x7f5c34141bc0 2024-03-24T00:33:57.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c39000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3413ec80 0x7f5c3415e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:57.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c39000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3413ec80 0x7f5c3415e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38824/0 (socket says 172.21.15.40:38824) 2024-03-24T00:33:57.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c39000640 1 -- 172.21.15.40:0/2624739475 learned_addr learned my addr 172.21.15.40:0/2624739475 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:57.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c39a00640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c34072310 0x7f5c341fa5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:57.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c3a400640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c34141bc0 0x7f5c3415ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:57.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c39000640 1 -- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c34141bc0 msgr2=0x7f5c3415ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c39000640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c34141bc0 0x7f5c3415ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c39000640 1 -- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c34072310 msgr2=0x7f5c341fa5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c39000640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c34072310 0x7f5c341fa5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c39000640 1 -- 172.21.15.40:0/2624739475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3406dc00 con 0x7f5c3413ec80 2024-03-24T00:33:57.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c39000640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3413ec80 0x7f5c3415e910 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5c2000c490 tx=0x7f5c2000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:57.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c31600640 1 -- 172.21.15.40:0/2624739475 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20018070 con 0x7f5c3413ec80 2024-03-24T00:33:57.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c31600640 1 -- 172.21.15.40:0/2624739475 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c20007480 con 0x7f5c3413ec80 2024-03-24T00:33:57.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c3c400640 1 -- 172.21.15.40:0/2624739475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3406dee0 con 0x7f5c3413ec80 2024-03-24T00:33:57.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c31600640 1 -- 172.21.15.40:0/2624739475 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20004a10 con 0x7f5c3413ec80 2024-03-24T00:33:57.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c3c400640 1 -- 172.21.15.40:0/2624739475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3406e4a0 con 0x7f5c3413ec80 2024-03-24T00:33:57.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c3c400640 1 -- 172.21.15.40:0/2624739475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c34068170 con 0x7f5c3413ec80 2024-03-24T00:33:57.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.719+0000 7f5c31600640 1 -- 172.21.15.40:0/2624739475 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5c2001b070 con 0x7f5c3413ec80 2024-03-24T00:33:57.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.723+0000 7f5c31600640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5c14067690 0x7f5c14069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:57.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.723+0000 7f5c31600640 1 -- 172.21.15.40:0/2624739475 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f5c20084940 con 0x7f5c3413ec80 2024-03-24T00:33:57.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.723+0000 7f5c39a00640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5c14067690 0x7f5c14069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:57.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.723+0000 7f5c39a00640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5c14067690 0x7f5c14069b50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5c28000f70 tx=0x7f5c28009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:57.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.723+0000 7f5c31600640 1 -- 172.21.15.40:0/2624739475 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5c200557a0 con 0x7f5c3413ec80 2024-03-24T00:33:57.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.887+0000 7f5c3c400640 1 -- 172.21.15.40:0/2624739475 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c34072b90 con 0x7f5c3413ec80 2024-03-24T00:33:57.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c31600640 1 -- 172.21.15.40:0/2624739475 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+13348 (secure 0 0 0) 0x7f5c200550a0 con 0x7f5c3413ec80 2024-03-24T00:33:57.892 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:33:57.892 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":50,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:33:55.540890+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6832/224223662":"2024-03-25T00:33:52.533441+0000","172.21.15.53:6837/1453842342":"2024-03-25T00:33:52.533434+0000","172.21.15.40:6834/224223662":"2024-03-25T00:33:52.533441+0000","172.21.15.53:6835/1453842342":"2024-03-25T00:33:52.533434+0000","172.21.15.40:6833/2128553323":"2024-03-25T00:33:52.533401+0000","172.21.15.53:6836/1600357100":"2024-03-25T00:33:27.880035+0000","172.21.15.40:6835/2128553323":"2024-03-25T00:33:52.533401+0000","172.21.15.53:6834/1600357100":"2024-03-25T00:33:27.880035+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:33:57.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c3c400640 1 -- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5c14067690 msgr2=0x7f5c14069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c3c400640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5c14067690 0x7f5c14069b50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5c28000f70 tx=0x7f5c28009040 comp rx=0 tx=0).stop 2024-03-24T00:33:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c3c400640 1 -- 172.21.15.40:0/2624739475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3413ec80 msgr2=0x7f5c3415e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c3c400640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3413ec80 0x7f5c3415e910 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5c2000c490 tx=0x7f5c2000c950 comp rx=0 tx=0).stop 2024-03-24T00:33:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c3c400640 1 -- 172.21.15.40:0/2624739475 shutdown_connections 2024-03-24T00:33:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c3c400640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5c14067690 0x7f5c14069b50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c3c400640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c34141bc0 0x7f5c3415ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c3c400640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5c3413ec80 0x7f5c3415e910 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c3c400640 1 --2- 172.21.15.40:0/2624739475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c34072310 0x7f5c341fa5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c3c400640 1 -- 172.21.15.40:0/2624739475 >> 172.21.15.40:0/2624739475 conn(0x7f5c340b17b0 msgr2=0x7f5c34071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c3c400640 1 -- 172.21.15.40:0/2624739475 shutdown_connections 2024-03-24T00:33:57.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:57.891+0000 7f5c3c400640 1 -- 172.21.15.40:0/2624739475 wait complete. 2024-03-24T00:33:57.913 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:33:57.913 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:33:58.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f313f600640 1 Processor -- start 2024-03-24T00:33:58.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f313f600640 1 -- start start 2024-03-24T00:33:58.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f313f600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f313815ac60 0x7f31380b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f313f600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f313806cb80 0x7f31380b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f313f600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31380b3260 con 0x7f31380b4840 2024-03-24T00:33:58.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f313f600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31380b3e70 con 0x7f313806cb80 2024-03-24T00:33:58.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f313f600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31380b3540 con 0x7f313815ac60 2024-03-24T00:33:58.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f313d600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f31380b4840 0x7f31380ccd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60466/0 (socket says 172.21.15.40:60466) 2024-03-24T00:33:58.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f313d600640 1 -- 172.21.15.40:0/1227542740 learned_addr learned my addr 172.21.15.40:0/1227542740 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:58.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137e00640 1 --2- 172.21.15.40:0/1227542740 >> v2:172.21.15.53:3300/0 conn(0x7f313806cb80 0x7f31380b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f313cc00640 1 --2- 172.21.15.40:0/1227542740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f313815ac60 0x7f31380b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870165831 0 0) 0x7f31380b3260 con 0x7f31380b4840 2024-03-24T00:33:58.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3120003530 con 0x7f31380b4840 2024-03-24T00:33:58.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3969117298 0 0) 0x7f3120003530 con 0x7f31380b4840 2024-03-24T00:33:58.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f313815ac60 msgr2=0x7f31380b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:58.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 --2- 172.21.15.40:0/1227542740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f313815ac60 0x7f31380b2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 >> v2:172.21.15.53:3300/0 conn(0x7f313806cb80 msgr2=0x7f31380b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 --2- 172.21.15.40:0/1227542740 >> v2:172.21.15.53:3300/0 conn(0x7f313806cb80 0x7f31380b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31380b3820 con 0x7f31380b4840 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f3130001a00 con 0x7f31380b4840 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 >> v1:172.21.15.40:6789/0 conn(0x7f31380b4840 legacy=0x7f31380ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 --2- 172.21.15.40:0/1227542740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31200039c0 0x7f3120003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 --2- 172.21.15.40:0/1227542740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3120004810 0x7f3120004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 --2- 172.21.15.40:0/1227542740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f313815ac60 0x7f31200051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31380b3e70 con 0x7f3120004810 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31380b3540 con 0x7f31200039c0 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312000ba20 con 0x7f313815ac60 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137e00640 1 --2- 172.21.15.40:0/1227542740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3120004810 0x7f3120004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f313cc00640 1 --2- 172.21.15.40:0/1227542740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31200039c0 0x7f3120003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f313d600640 1 --2- 172.21.15.40:0/1227542740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f313815ac60 0x7f31200051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137e00640 1 -- 172.21.15.40:0/1227542740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f313815ac60 msgr2=0x7f31200051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:58.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137e00640 1 --2- 172.21.15.40:0/1227542740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f313815ac60 0x7f31200051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137e00640 1 -- 172.21.15.40:0/1227542740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31200039c0 msgr2=0x7f3120003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:58.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137e00640 1 --2- 172.21.15.40:0/1227542740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31200039c0 0x7f3120003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.055+0000 7f3137e00640 1 -- 172.21.15.40:0/1227542740 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3120003530 con 0x7f3120004810 2024-03-24T00:33:58.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.059+0000 7f3137e00640 1 --2- 172.21.15.40:0/1227542740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3120004810 0x7f3120004c90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3124009ae0 tx=0x7f312401ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4732f1c5a7a42c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:58.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.059+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31240b2070 con 0x7f3120004810 2024-03-24T00:33:58.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.059+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31240a5550 con 0x7f3120004810 2024-03-24T00:33:58.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.059+0000 7f3137400640 1 -- 172.21.15.40:0/1227542740 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31240096b0 con 0x7f3120004810 2024-03-24T00:33:58.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.059+0000 7f313f600640 1 -- 172.21.15.40:0/1227542740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3120004810 msgr2=0x7f3120004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:58.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.059+0000 7f313f600640 1 --2- 172.21.15.40:0/1227542740 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3120004810 0x7f3120004c90 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f3124009ae0 tx=0x7f312401ca40 comp rx=0 tx=0).stop 2024-03-24T00:33:58.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.059+0000 7f313d600640 1 -- 172.21.15.40:0/1227542740 reap_dead start 2024-03-24T00:33:58.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.059+0000 7f313f600640 1 -- 172.21.15.40:0/1227542740 shutdown_connections 2024-03-24T00:33:58.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.059+0000 7f313f600640 1 -- 172.21.15.40:0/1227542740 >> 172.21.15.40:0/1227542740 conn(0x7f31380b1660 msgr2=0x7f31380b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:58.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.059+0000 7f313f600640 1 -- 172.21.15.40:0/1227542740 shutdown_connections 2024-03-24T00:33:58.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.059+0000 7f313f600640 1 -- 172.21.15.40:0/1227542740 wait complete. 2024-03-24T00:33:58.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f313f600640 1 Processor -- start 2024-03-24T00:33:58.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f313f600640 1 -- start start 2024-03-24T00:33:58.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f313f600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3120004810 0x7f3138141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f313f600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31380ced80 0x7f313815b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f313f600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cf1c0 0x7f313815b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f313f600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3138071e90 con 0x7f31380ced80 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f313f600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3138071d10 con 0x7f31380cf1c0 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f313f600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3138072010 con 0x7f3120004810 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f3137e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31380ced80 0x7f313815b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f3137e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31380ced80 0x7f313815b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38830/0 (socket says 172.21.15.40:38830) 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f3137e00640 1 -- 172.21.15.40:0/2693406979 learned_addr learned my addr 172.21.15.40:0/2693406979 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f313cc00640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3120004810 0x7f3138141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f313d600640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cf1c0 0x7f313815b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f3137e00640 1 -- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3120004810 msgr2=0x7f3138141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f3137e00640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3120004810 0x7f3138141570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f3137e00640 1 -- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cf1c0 msgr2=0x7f313815b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f3137e00640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cf1c0 0x7f313815b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f3137e00640 1 -- 172.21.15.40:0/2693406979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f313815c1a0 con 0x7f31380ced80 2024-03-24T00:33:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f3137e00640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31380ced80 0x7f313815b370 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3124009ae0 tx=0x7f3120005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:58.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f3134c00640 1 -- 172.21.15.40:0/2693406979 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31240b2040 con 0x7f31380ced80 2024-03-24T00:33:58.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f3134c00640 1 -- 172.21.15.40:0/2693406979 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312400f000 con 0x7f31380ced80 2024-03-24T00:33:58.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f313f600640 1 -- 172.21.15.40:0/2693406979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f313815bec0 con 0x7f31380ced80 2024-03-24T00:33:58.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f3134c00640 1 -- 172.21.15.40:0/2693406979 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3124004020 con 0x7f31380ced80 2024-03-24T00:33:58.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.063+0000 7f313f600640 1 -- 172.21.15.40:0/2693406979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3138140d70 con 0x7f31380ced80 2024-03-24T00:33:58.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.067+0000 7f313f600640 1 -- 172.21.15.40:0/2693406979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3138068170 con 0x7f31380ced80 2024-03-24T00:33:58.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.067+0000 7f3134c00640 1 -- 172.21.15.40:0/2693406979 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f31240c0020 con 0x7f31380ced80 2024-03-24T00:33:58.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.067+0000 7f3134c00640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f31180675a0 0x7f3118069a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.067+0000 7f3134c00640 1 -- 172.21.15.40:0/2693406979 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f312411f730 con 0x7f31380ced80 2024-03-24T00:33:58.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.067+0000 7f313cc00640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f31180675a0 0x7f3118069a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.067+0000 7f313cc00640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f31180675a0 0x7f3118069a60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f312c009a40 tx=0x7f312c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:58.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.071+0000 7f3134c00640 1 -- 172.21.15.40:0/2693406979 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f31240167c0 con 0x7f31380ced80 2024-03-24T00:33:58.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 -- 172.21.15.40:0/2693406979 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31380cbc90 con 0x7f31380ced80 2024-03-24T00:33:58.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f3134c00640 1 -- 172.21.15.40:0/2693406979 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31240f0470 con 0x7f31380ced80 2024-03-24T00:33:58.237 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:33:58.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 -- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f31180675a0 msgr2=0x7f3118069a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:58.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f31180675a0 0x7f3118069a60 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f312c009a40 tx=0x7f312c008040 comp rx=0 tx=0).stop 2024-03-24T00:33:58.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 -- 172.21.15.40:0/2693406979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31380ced80 msgr2=0x7f313815b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:58.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31380ced80 0x7f313815b370 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3124009ae0 tx=0x7f3120005840 comp rx=0 tx=0).stop 2024-03-24T00:33:58.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 -- 172.21.15.40:0/2693406979 shutdown_connections 2024-03-24T00:33:58.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f31180675a0 0x7f3118069a60 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cf1c0 0x7f313815b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f31380ced80 0x7f313815b370 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 --2- 172.21.15.40:0/2693406979 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3120004810 0x7f3138141570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 -- 172.21.15.40:0/2693406979 >> 172.21.15.40:0/2693406979 conn(0x7f31380b1660 msgr2=0x7f31380af800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:58.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 -- 172.21.15.40:0/2693406979 shutdown_connections 2024-03-24T00:33:58.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.235+0000 7f313f600640 1 -- 172.21.15.40:0/2693406979 wait complete. 2024-03-24T00:33:58.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_off' 2024-03-24T00:33:58.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.395+0000 7f4842200640 1 Processor -- start 2024-03-24T00:33:58.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.395+0000 7f4842200640 1 -- start start 2024-03-24T00:33:58.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.395+0000 7f4842200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f483c13ec80 0x7f483c1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.395+0000 7f4842200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c070e00 con 0x7f483c0ce800 2024-03-24T00:33:58.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.395+0000 7f4842200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483c0cee20 con 0x7f483c1419f0 2024-03-24T00:33:58.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.395+0000 7f4842200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c141e50 con 0x7f483c13ec80 2024-03-24T00:33:58.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.395+0000 7f483be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f483c0ce800 0x7f483c071300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60468/0 (socket says 172.21.15.40:60468) 2024-03-24T00:33:58.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.395+0000 7f483be00640 1 -- 172.21.15.40:0/3210393853 learned_addr learned my addr 172.21.15.40:0/3210393853 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:58.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483b400640 1 --2- 172.21.15.40:0/3210393853 >> v2:172.21.15.53:3300/0 conn(0x7f483c13ec80 0x7f483c1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1243732157 0 0) 0x7f483c070e00 con 0x7f483c0ce800 2024-03-24T00:33:58.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4820003530 con 0x7f483c0ce800 2024-03-24T00:33:58.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220386517 0 0) 0x7f483c0cee20 con 0x7f483c1419f0 2024-03-24T00:33:58.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f483c070e00 con 0x7f483c1419f0 2024-03-24T00:33:58.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 372718022 0 0) 0x7f4820003530 con 0x7f483c0ce800 2024-03-24T00:33:58.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 >> v2:172.21.15.53:3300/0 conn(0x7f483c13ec80 msgr2=0x7f483c1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:58.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 --2- 172.21.15.40:0/3210393853 >> v2:172.21.15.53:3300/0 conn(0x7f483c13ec80 0x7f483c1414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 >> v1:172.21.15.53:6789/0 conn(0x7f483c1419f0 legacy=0x7f483c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:58.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c0cf100 con 0x7f483c0ce800 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f4834001a00 con 0x7f483c0ce800 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 >> v1:172.21.15.40:6789/0 conn(0x7f483c0ce800 legacy=0x7f483c071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 --2- 172.21.15.40:0/3210393853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4820003b50 0x7f4820003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 --2- 172.21.15.40:0/3210393853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4820004810 0x7f4820004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 --2- 172.21.15.40:0/3210393853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48200051d0 0x7f482000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482000c020 con 0x7f4820004810 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482000c1d0 con 0x7f4820003b50 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482000c3d0 con 0x7f48200051d0 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f4832a00640 1 --2- 172.21.15.40:0/3210393853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4820004810 0x7f4820004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483b400640 1 --2- 172.21.15.40:0/3210393853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4820003b50 0x7f4820003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483be00640 1 --2- 172.21.15.40:0/3210393853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48200051d0 0x7f482000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483b400640 1 -- 172.21.15.40:0/3210393853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48200051d0 msgr2=0x7f482000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:58.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483b400640 1 --2- 172.21.15.40:0/3210393853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48200051d0 0x7f482000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483b400640 1 -- 172.21.15.40:0/3210393853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4820004810 msgr2=0x7f4820004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:58.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483b400640 1 --2- 172.21.15.40:0/3210393853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4820004810 0x7f4820004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483b400640 1 -- 172.21.15.40:0/3210393853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4820003530 con 0x7f4820003b50 2024-03-24T00:33:58.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f4832a00640 1 --2- 172.21.15.40:0/3210393853 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4820004810 0x7f4820004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:33:58.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.399+0000 7f483be00640 1 -- 172.21.15.40:0/3210393853 reap_dead start 2024-03-24T00:33:58.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f483b400640 1 --2- 172.21.15.40:0/3210393853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4820003b50 0x7f4820003f50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f482c009ae0 tx=0x7f482c01ca50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4da456fd8a063404 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:58.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c0b2070 con 0x7f4820003b50 2024-03-24T00:33:58.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482c0a5530 con 0x7f4820003b50 2024-03-24T00:33:58.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f483aa00640 1 -- 172.21.15.40:0/3210393853 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c0096b0 con 0x7f4820003b50 2024-03-24T00:33:58.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f4842200640 1 -- 172.21.15.40:0/3210393853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4820003b50 msgr2=0x7f4820003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:58.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f4842200640 1 --2- 172.21.15.40:0/3210393853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4820003b50 0x7f4820003f50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f482c009ae0 tx=0x7f482c01ca50 comp rx=0 tx=0).stop 2024-03-24T00:33:58.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f4842200640 1 -- 172.21.15.40:0/3210393853 shutdown_connections 2024-03-24T00:33:58.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f4842200640 1 --2- 172.21.15.40:0/3210393853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4820003b50 0x7f4820003f50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f4842200640 1 -- 172.21.15.40:0/3210393853 >> 172.21.15.40:0/3210393853 conn(0x7f483c0b17b0 msgr2=0x7f483c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:58.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f4842200640 1 -- 172.21.15.40:0/3210393853 shutdown_connections 2024-03-24T00:33:58.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f4842200640 1 -- 172.21.15.40:0/3210393853 wait complete. 2024-03-24T00:33:58.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f4842200640 1 Processor -- start 2024-03-24T00:33:58.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.403+0000 7f4842200640 1 -- start start 2024-03-24T00:33:58.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4842200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4820003b50 0x7f483c13a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4842200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f483c13d290 0x7f483c13d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4842200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f483c13b0e0 0x7f483c13b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4842200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c1414b0 con 0x7f483c13d290 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4842200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c13f060 con 0x7f483c13b0e0 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4842200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c141630 con 0x7f4820003b50 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4832a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f483c13d290 0x7f483c13d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4832a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f483c13d290 0x7f483c13d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38852/0 (socket says 172.21.15.40:38852) 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4832a00640 1 -- 172.21.15.40:0/2438225466 learned_addr learned my addr 172.21.15.40:0/2438225466 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f483be00640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f483c13b0e0 0x7f483c13b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f483b400640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4820003b50 0x7f483c13a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4832a00640 1 -- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4820003b50 msgr2=0x7f483c13a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4832a00640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4820003b50 0x7f483c13a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4832a00640 1 -- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f483c13b0e0 msgr2=0x7f483c13b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4832a00640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f483c13b0e0 0x7f483c13b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4832a00640 1 -- 172.21.15.40:0/2438225466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c13a9a0 con 0x7f483c13d290 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4832a00640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f483c13d290 0x7f483c13d710 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f48240128d0 tx=0x7f4824012cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:58.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4833e00640 1 -- 172.21.15.40:0/2438225466 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824001520 con 0x7f483c13d290 2024-03-24T00:33:58.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4833e00640 1 -- 172.21.15.40:0/2438225466 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482401b070 con 0x7f483c13d290 2024-03-24T00:33:58.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4842200640 1 -- 172.21.15.40:0/2438225466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c06ce20 con 0x7f483c13d290 2024-03-24T00:33:58.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4833e00640 1 -- 172.21.15.40:0/2438225466 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4824010d90 con 0x7f483c13d290 2024-03-24T00:33:58.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4842200640 1 -- 172.21.15.40:0/2438225466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c13e0c0 con 0x7f483c13d290 2024-03-24T00:33:58.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.407+0000 7f4842200640 1 -- 172.21.15.40:0/2438225466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483c068170 con 0x7f483c13d290 2024-03-24T00:33:58.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.411+0000 7f4833e00640 1 -- 172.21.15.40:0/2438225466 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4824016480 con 0x7f483c13d290 2024-03-24T00:33:58.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.411+0000 7f4833e00640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4818067660 0x7f4818069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:58.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.411+0000 7f4833e00640 1 -- 172.21.15.40:0/2438225466 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f4824088b90 con 0x7f483c13d290 2024-03-24T00:33:58.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.411+0000 7f483b400640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4818067660 0x7f4818069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:58.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.411+0000 7f483b400640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4818067660 0x7f4818069b20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f482c0af000 tx=0x7f482c0046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:58.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.415+0000 7f4833e00640 1 -- 172.21.15.40:0/2438225466 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4824059a00 con 0x7f483c13d290 2024-03-24T00:33:58.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:58.635+0000 7f4842200640 1 -- 172.21.15.40:0/2438225466 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_off"]} v 0) v1 -- 0x7f483c0ce1d0 con 0x7f483c13d290 2024-03-24T00:33:59.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.551+0000 7f4833e00640 1 -- 172.21.15.40:0/2438225466 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_off"]}]=0 v167) v1 ==== 153+0+0 (secure 0 0 0) 0x7f4824059300 con 0x7f483c13d290 2024-03-24T00:33:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.555+0000 7f4842200640 1 -- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4818067660 msgr2=0x7f4818069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.555+0000 7f4842200640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4818067660 0x7f4818069b20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f482c0af000 tx=0x7f482c0046d0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.555+0000 7f4842200640 1 -- 172.21.15.40:0/2438225466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f483c13d290 msgr2=0x7f483c13d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.555+0000 7f4842200640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f483c13d290 0x7f483c13d710 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f48240128d0 tx=0x7f4824012cd0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.555+0000 7f4842200640 1 -- 172.21.15.40:0/2438225466 shutdown_connections 2024-03-24T00:33:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.555+0000 7f4842200640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4818067660 0x7f4818069b20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.555+0000 7f4842200640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f483c13b0e0 0x7f483c13b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.555+0000 7f4842200640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f483c13d290 0x7f483c13d710 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.555+0000 7f4842200640 1 --2- 172.21.15.40:0/2438225466 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4820003b50 0x7f483c13a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.555+0000 7f4842200640 1 -- 172.21.15.40:0/2438225466 >> 172.21.15.40:0/2438225466 conn(0x7f483c0b17b0 msgr2=0x7f483c0cce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.555+0000 7f4842200640 1 -- 172.21.15.40:0/2438225466 shutdown_connections 2024-03-24T00:33:59.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.555+0000 7f4842200640 1 -- 172.21.15.40:0/2438225466 wait complete. 2024-03-24T00:33:59.580 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:33:59.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.727+0000 7f28f2400640 1 Processor -- start 2024-03-24T00:33:59.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.727+0000 7f28f2400640 1 -- start start 2024-03-24T00:33:59.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.727+0000 7f28f2400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f28ec164a80 0x7f28ec166e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:59.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.727+0000 7f28f2400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ec0cc910 con 0x7f28ec15f760 2024-03-24T00:33:59.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.727+0000 7f28f2400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ec0ccbf0 con 0x7f28ec0b4290 2024-03-24T00:33:59.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.727+0000 7f28f2400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28ec0cced0 con 0x7f28ec164a80 2024-03-24T00:33:59.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.727+0000 7f28f1000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f28ec15f760 0x7f28ec15fb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60474/0 (socket says 172.21.15.40:60474) 2024-03-24T00:33:59.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.727+0000 7f28f1000640 1 -- 172.21.15.40:0/1071930429 learned_addr learned my addr 172.21.15.40:0/1071930429 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:33:59.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28f1a00640 1 --2- 172.21.15.40:0/1071930429 >> v2:172.21.15.53:3300/0 conn(0x7f28ec164a80 0x7f28ec166e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:59.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345253872 0 0) 0x7f28ec0cc910 con 0x7f28ec15f760 2024-03-24T00:33:59.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28d4003530 con 0x7f28ec15f760 2024-03-24T00:33:59.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 903227027 0 0) 0x7f28d4003530 con 0x7f28ec15f760 2024-03-24T00:33:59.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 >> v2:172.21.15.53:3300/0 conn(0x7f28ec164a80 msgr2=0x7f28ec166e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:59.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 --2- 172.21.15.40:0/1071930429 >> v2:172.21.15.53:3300/0 conn(0x7f28ec164a80 0x7f28ec166e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 >> v1:172.21.15.53:6789/0 conn(0x7f28ec0b4290 legacy=0x7f28ec0cf660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:59.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec0cd1b0 con 0x7f28ec15f760 2024-03-24T00:33:59.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f28e0001680 con 0x7f28ec15f760 2024-03-24T00:33:59.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 >> v1:172.21.15.40:6789/0 conn(0x7f28ec15f760 legacy=0x7f28ec15fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:59.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 --2- 172.21.15.40:0/1071930429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28d40039c0 0x7f28d4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:59.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 --2- 172.21.15.40:0/1071930429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28d4004810 0x7f28d4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 --2- 172.21.15.40:0/1071930429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28d40051d0 0x7f28d400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec0cced0 con 0x7f28d4004810 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28d400c070 con 0x7f28d40039c0 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28d400c270 con 0x7f28d40051d0 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28ebe00640 1 --2- 172.21.15.40:0/1071930429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28d4004810 0x7f28d4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28f1000640 1 --2- 172.21.15.40:0/1071930429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28d40039c0 0x7f28d4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28f1a00640 1 --2- 172.21.15.40:0/1071930429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28d40051d0 0x7f28d400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28ebe00640 1 -- 172.21.15.40:0/1071930429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28d40051d0 msgr2=0x7f28d400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28ebe00640 1 --2- 172.21.15.40:0/1071930429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28d40051d0 0x7f28d400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28ebe00640 1 -- 172.21.15.40:0/1071930429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28d40039c0 msgr2=0x7f28d4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28ebe00640 1 --2- 172.21.15.40:0/1071930429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28d40039c0 0x7f28d4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28ebe00640 1 -- 172.21.15.40:0/1071930429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d4003530 con 0x7f28d4004810 2024-03-24T00:33:59.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.731+0000 7f28f1a00640 1 -- 172.21.15.40:0/1071930429 reap_dead start 2024-03-24T00:33:59.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28ebe00640 1 --2- 172.21.15.40:0/1071930429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28d4004810 0x7f28d4004c90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f28d8001750 tx=0x7f28d801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e75b120bfa931aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:59.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d80013a0 con 0x7f28d4004810 2024-03-24T00:33:59.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d80052f0 con 0x7f28d4004810 2024-03-24T00:33:59.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28eb400640 1 -- 172.21.15.40:0/1071930429 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d8003840 con 0x7f28d4004810 2024-03-24T00:33:59.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 -- 172.21.15.40:0/1071930429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28d4004810 msgr2=0x7f28d4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:59.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 --2- 172.21.15.40:0/1071930429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28d4004810 0x7f28d4004c90 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f28d8001750 tx=0x7f28d801c3c0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 -- 172.21.15.40:0/1071930429 shutdown_connections 2024-03-24T00:33:59.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 --2- 172.21.15.40:0/1071930429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28d4004810 0x7f28d4004c90 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 -- 172.21.15.40:0/1071930429 >> 172.21.15.40:0/1071930429 conn(0x7f28ec0b3620 msgr2=0x7f28ec0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:59.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 -- 172.21.15.40:0/1071930429 shutdown_connections 2024-03-24T00:33:59.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 -- 172.21.15.40:0/1071930429 wait complete. 2024-03-24T00:33:59.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 Processor -- start 2024-03-24T00:33:59.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 -- start start 2024-03-24T00:33:59.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28d4004810 0x7f28ec1569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:59.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec152950 0x7f28ec158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:59.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28ec0b2330 0x7f28ec0b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:59.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec0cc6b0 con 0x7f28ec0b2330 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec0cc530 con 0x7f28ec152950 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.735+0000 7f28f2400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec0cd6d0 con 0x7f28d4004810 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28ec0b2330 0x7f28ec0b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28ec0b2330 0x7f28ec0b27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38876/0 (socket says 172.21.15.40:38876) 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f1a00640 1 -- 172.21.15.40:0/852934515 learned_addr learned my addr 172.21.15.40:0/852934515 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28ebe00640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec152950 0x7f28ec158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f1000640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28d4004810 0x7f28ec1569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f1a00640 1 -- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28d4004810 msgr2=0x7f28ec1569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f1a00640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28d4004810 0x7f28ec1569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f1a00640 1 -- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec152950 msgr2=0x7f28ec158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f1a00640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec152950 0x7f28ec158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f1a00640 1 -- 172.21.15.40:0/852934515 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec0ae930 con 0x7f28ec0b2330 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f1000640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28d4004810 0x7f28ec1569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:33:59.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f1a00640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28ec0b2330 0x7f28ec0b27e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f28ec071590 tx=0x7f28e40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:59.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28e8c00640 1 -- 172.21.15.40:0/852934515 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e4011e10 con 0x7f28ec0b2330 2024-03-24T00:33:59.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28e8c00640 1 -- 172.21.15.40:0/852934515 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e401b070 con 0x7f28ec0b2330 2024-03-24T00:33:59.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f2400640 1 -- 172.21.15.40:0/852934515 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ec158ac0 con 0x7f28ec0b2330 2024-03-24T00:33:59.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28e8c00640 1 -- 172.21.15.40:0/852934515 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e40039a0 con 0x7f28ec0b2330 2024-03-24T00:33:59.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f2400640 1 -- 172.21.15.40:0/852934515 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ec0aeba0 con 0x7f28ec0b2330 2024-03-24T00:33:59.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28f2400640 1 -- 172.21.15.40:0/852934515 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ec04f410 con 0x7f28ec0b2330 2024-03-24T00:33:59.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28e8c00640 1 -- 172.21.15.40:0/852934515 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f28e4003b60 con 0x7f28ec0b2330 2024-03-24T00:33:59.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28e8c00640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f28cc067690 0x7f28cc069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:33:59.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.739+0000 7f28e8c00640 1 -- 172.21.15.40:0/852934515 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f28e4085de0 con 0x7f28ec0b2330 2024-03-24T00:33:59.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.743+0000 7f28f1000640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f28cc067690 0x7f28cc069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:33:59.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.743+0000 7f28f1000640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f28cc067690 0x7f28cc069b50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f28e0007d10 tx=0x7f28e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:33:59.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.747+0000 7f28e8c00640 1 -- 172.21.15.40:0/852934515 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f28e4056bb0 con 0x7f28ec0b2330 2024-03-24T00:33:59.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.907+0000 7f28f2400640 1 -- 172.21.15.40:0/852934515 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f28ec072dc0 con 0x7f28ec0b2330 2024-03-24T00:33:59.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.911+0000 7f28e8c00640 1 -- 172.21.15.40:0/852934515 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v27)=0 v27) v1 ==== 122+0+0 (secure 0 0 0) 0x7f28e40564b0 con 0x7f28ec0b2330 2024-03-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.915+0000 7f28f2400640 1 -- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f28cc067690 msgr2=0x7f28cc069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.915+0000 7f28f2400640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f28cc067690 0x7f28cc069b50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f28e0007d10 tx=0x7f28e0009040 comp rx=0 tx=0).stop 2024-03-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.915+0000 7f28f2400640 1 -- 172.21.15.40:0/852934515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28ec0b2330 msgr2=0x7f28ec0b27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.915+0000 7f28f2400640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28ec0b2330 0x7f28ec0b27e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f28ec071590 tx=0x7f28e40115a0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.915+0000 7f28f2400640 1 -- 172.21.15.40:0/852934515 shutdown_connections 2024-03-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.915+0000 7f28f2400640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f28cc067690 0x7f28cc069b50 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.915+0000 7f28f2400640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28ec0b2330 0x7f28ec0b27e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.915+0000 7f28f2400640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28ec152950 0x7f28ec158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.915+0000 7f28f2400640 1 --2- 172.21.15.40:0/852934515 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28d4004810 0x7f28ec1569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.915+0000 7f28f2400640 1 -- 172.21.15.40:0/852934515 >> 172.21.15.40:0/852934515 conn(0x7f28ec0b3620 msgr2=0x7f28ec155e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:33:59.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.915+0000 7f28f2400640 1 -- 172.21.15.40:0/852934515 shutdown_connections 2024-03-24T00:33:59.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:33:59.915+0000 7f28f2400640 1 -- 172.21.15.40:0/852934515 wait complete. 2024-03-24T00:33:59.936 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:33:59.937 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:33:59.975 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:34:00.047 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:34:00.047 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:34:00.047 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:34:00.056 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:34:00.057 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:34:00.057 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:34:00.066 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:34:00.066 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:34:00.066 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:34:00.108 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:34:00.108 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:34:00.108 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:34:00.120 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:34:00.120 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:34:00.120 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:34:00.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.287+0000 7fa361a00640 1 Processor -- start 2024-03-24T00:34:00.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.287+0000 7fa361a00640 1 -- start start 2024-03-24T00:34:00.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.287+0000 7fa361a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa35c06e620 0x7fa35c0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:00.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa361a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c070f60 con 0x7fa35c0ccfa0 2024-03-24T00:34:00.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa361a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35c0b3620 con 0x7fa35c06ea00 2024-03-24T00:34:00.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa361a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa35c0b4c00 con 0x7fa35c06e620 2024-03-24T00:34:00.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa35aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa35c06e620 0x7fa35c0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa35b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa35c0ccfa0 0x7fa35c0cd380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60482/0 (socket says 172.21.15.40:60482) 2024-03-24T00:34:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa35b400640 1 -- 172.21.15.40:0/508760759 learned_addr learned my addr 172.21.15.40:0/508760759 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa35aa00640 1 -- 172.21.15.40:0/508760759 >> v1:172.21.15.53:6789/0 conn(0x7fa35c06ea00 legacy=0x7fa35c0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa35aa00640 1 -- 172.21.15.40:0/508760759 >> v1:172.21.15.40:6789/0 conn(0x7fa35c0ccfa0 legacy=0x7fa35c0cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:00.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa35aa00640 1 -- 172.21.15.40:0/508760759 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c0b3900 con 0x7fa35c06e620 2024-03-24T00:34:00.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa35a000640 1 -- 172.21.15.40:0/508760759 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441640681 0 0) 0x7fa35c070f60 con 0x7fa35c0ccfa0 2024-03-24T00:34:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa35aa00640 1 --2- 172.21.15.40:0/508760759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa35c06e620 0x7fa35c0b46c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa34c0097c0 tx=0x7fa34c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46009afd8b7dd42e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa35a000640 1 -- 172.21.15.40:0/508760759 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c0b1070 con 0x7fa35c06e620 2024-03-24T00:34:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa35a000640 1 -- 172.21.15.40:0/508760759 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34c0a8050 con 0x7fa35c06e620 2024-03-24T00:34:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.291+0000 7fa35a000640 1 -- 172.21.15.40:0/508760759 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c0ac860 con 0x7fa35c06e620 2024-03-24T00:34:00.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 -- 172.21.15.40:0/508760759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa35c06e620 msgr2=0x7fa35c0b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 --2- 172.21.15.40:0/508760759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa35c06e620 0x7fa35c0b46c0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa34c0097c0 tx=0x7fa34c01c3e0 comp rx=0 tx=0).stop 2024-03-24T00:34:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 -- 172.21.15.40:0/508760759 shutdown_connections 2024-03-24T00:34:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 --2- 172.21.15.40:0/508760759 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa35c06e620 0x7fa35c0b46c0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 -- 172.21.15.40:0/508760759 >> 172.21.15.40:0/508760759 conn(0x7fa35c0b17b0 msgr2=0x7fa35c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 -- 172.21.15.40:0/508760759 shutdown_connections 2024-03-24T00:34:00.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 -- 172.21.15.40:0/508760759 wait complete. 2024-03-24T00:34:00.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 Processor -- start 2024-03-24T00:34:00.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 -- start start 2024-03-24T00:34:00.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa35c06e620 0x7fa35c13ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa35c06ea00 0x7fa35c159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa35c0ccfa0 0x7fa35c15a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c144920 con 0x7fa35c06e620 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c06ee10 con 0x7fa35c06ea00 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa361a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa35c137a60 con 0x7fa35c0ccfa0 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa35aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa35c06e620 0x7fa35c13ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa35aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa35c06e620 0x7fa35c13ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38896/0 (socket says 172.21.15.40:38896) 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa35aa00640 1 -- 172.21.15.40:0/958546597 learned_addr learned my addr 172.21.15.40:0/958546597 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa353e00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa35c06ea00 0x7fa35c159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa35b400640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa35c0ccfa0 0x7fa35c15a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa35aa00640 1 -- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa35c0ccfa0 msgr2=0x7fa35c15a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa35aa00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa35c0ccfa0 0x7fa35c15a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa35aa00640 1 -- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa35c06ea00 msgr2=0x7fa35c159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa35aa00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa35c06ea00 0x7fa35c159f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa35aa00640 1 -- 172.21.15.40:0/958546597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c136e50 con 0x7fa35c06e620 2024-03-24T00:34:00.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.295+0000 7fa353e00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa35c06ea00 0x7fa35c159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:00.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.299+0000 7fa35aa00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa35c06e620 0x7fa35c13ded0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa34c0af040 tx=0x7fa34c01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:00.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.299+0000 7fa352a00640 1 -- 172.21.15.40:0/958546597 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c0be070 con 0x7fa35c06e620 2024-03-24T00:34:00.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.299+0000 7fa352a00640 1 -- 172.21.15.40:0/958546597 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34c004030 con 0x7fa35c06e620 2024-03-24T00:34:00.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.299+0000 7fa361a00640 1 -- 172.21.15.40:0/958546597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa35c137130 con 0x7fa35c06e620 2024-03-24T00:34:00.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.299+0000 7fa352a00640 1 -- 172.21.15.40:0/958546597 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34c0b1040 con 0x7fa35c06e620 2024-03-24T00:34:00.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.299+0000 7fa361a00640 1 -- 172.21.15.40:0/958546597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35c1376f0 con 0x7fa35c06e620 2024-03-24T00:34:00.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.299+0000 7fa361a00640 1 -- 172.21.15.40:0/958546597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa35c068170 con 0x7fa35c06e620 2024-03-24T00:34:00.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.299+0000 7fa352a00640 1 -- 172.21.15.40:0/958546597 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa34c0c1070 con 0x7fa35c06e620 2024-03-24T00:34:00.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.299+0000 7fa352a00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa338067660 0x7fa338069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:00.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.299+0000 7fa352a00640 1 -- 172.21.15.40:0/958546597 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5201+0+0 (secure 0 0 0) 0x7fa34c11f310 con 0x7fa35c06e620 2024-03-24T00:34:00.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.299+0000 7fa353e00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa338067660 0x7fa338069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:00.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.303+0000 7fa353e00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa338067660 0x7fa338069b20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa35c13efb0 tx=0x7fa344009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:00.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.303+0000 7fa352a00640 1 -- 172.21.15.40:0/958546597 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa34c016090 con 0x7fa35c06e620 2024-03-24T00:34:00.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 -- 172.21.15.40:0/958546597 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa35c137c10 con 0x7fa35c06e620 2024-03-24T00:34:00.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa352a00640 1 -- 172.21.15.40:0/958546597 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+3693 (secure 0 0 0) 0x7fa35c137c10 con 0x7fa35c06e620 2024-03-24T00:34:00.505 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:00.506 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":44,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5444,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/374985095","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":374985095},{"type":"v1","addr":"172.21.15.53:6836","nonce":374985095}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39},{"gid":5660,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1426823282","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1426823282},{"type":"v1","addr":"172.21.15.53:6837","nonce":1426823282}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6360,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/641116783","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":641116783},{"type":"v1","addr":"172.21.15.40:6833","nonce":641116783}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6366,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2832393784","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":2832393784},{"type":"v1","addr":"172.21.15.40:6835","nonce":2832393784}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44}],"filesystems":[]} 2024-03-24T00:34:00.506 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 44 2024-03-24T00:34:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 -- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa338067660 msgr2=0x7fa338069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa338067660 0x7fa338069b20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fa35c13efb0 tx=0x7fa344009040 comp rx=0 tx=0).stop 2024-03-24T00:34:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 -- 172.21.15.40:0/958546597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa35c06e620 msgr2=0x7fa35c13ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa35c06e620 0x7fa35c13ded0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fa34c0af040 tx=0x7fa34c01ac20 comp rx=0 tx=0).stop 2024-03-24T00:34:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 -- 172.21.15.40:0/958546597 shutdown_connections 2024-03-24T00:34:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa338067660 0x7fa338069b20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa35c0ccfa0 0x7fa35c15a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa35c06ea00 0x7fa35c159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 --2- 172.21.15.40:0/958546597 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa35c06e620 0x7fa35c13ded0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 -- 172.21.15.40:0/958546597 >> 172.21.15.40:0/958546597 conn(0x7fa35c0b17b0 msgr2=0x7fa35c13f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 -- 172.21.15.40:0/958546597 shutdown_connections 2024-03-24T00:34:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:00.503+0000 7fa361a00640 1 -- 172.21.15.40:0/958546597 wait complete. 2024-03-24T00:34:00.527 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:34:00.527 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:34:00.527 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:34:00.528 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:34:00.528 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:34:00.528 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:34:00.528 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:34:00.528 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:34:00.528 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:34:00.529 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:34:00.529 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:34:00.529 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:34:00.529 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:34:00.529 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:34:00.529 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:34:00.529 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:34:00.540 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:34:00.540 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:34:00.541 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:34:00.541 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:34:00.546 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:34:00.546 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:34:00.592 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:34:00.592 DEBUG:teuthology.parallel:result is None 2024-03-24T00:34:00.593 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:34:00.593 DEBUG:teuthology.parallel:result is None 2024-03-24T00:34:00.595 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:34:00.595 DEBUG:teuthology.parallel:result is None 2024-03-24T00:34:00.678 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:34:00.678 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:34:00.686 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:34:05.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-24T00:34:06.547 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:34:06.547 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:34:06.551 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:34:06.551 DEBUG:teuthology.parallel:result is None 2024-03-24T00:34:06.551 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:34:06.649 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:34:06.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.703+0000 7f152aa00640 1 Processor -- start 2024-03-24T00:34:06.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.703+0000 7f152aa00640 1 -- start start 2024-03-24T00:34:06.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.703+0000 7f152aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f152406e620 0x7f15240b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:06.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.703+0000 7f152aa00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1524070f60 con 0x7f152406ea00 2024-03-24T00:34:06.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.703+0000 7f152aa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15240b3620 con 0x7f15240ccfa0 2024-03-24T00:34:06.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.703+0000 7f152aa00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15240b4c00 con 0x7f152406e620 2024-03-24T00:34:06.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.703+0000 7f151be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f152406ea00 0x7f15240b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60496/0 (socket says 172.21.15.40:60496) 2024-03-24T00:34:06.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.703+0000 7f151be00640 1 -- 172.21.15.40:0/2395050628 learned_addr learned my addr 172.21.15.40:0/2395050628 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:06.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.703+0000 7f1523e00640 1 --2- 172.21.15.40:0/2395050628 >> v2:172.21.15.40:3300/0 conn(0x7f152406e620 0x7f15240b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:06.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f1523e00640 1 -- 172.21.15.40:0/2395050628 >> v1:172.21.15.53:6789/0 conn(0x7f15240ccfa0 legacy=0x7f15240cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:06.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f1523e00640 1 -- 172.21.15.40:0/2395050628 >> v1:172.21.15.40:6789/0 conn(0x7f152406ea00 legacy=0x7f15240b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:06.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f1523e00640 1 -- 172.21.15.40:0/2395050628 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15240b3900 con 0x7f152406e620 2024-03-24T00:34:06.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f1523e00640 1 --2- 172.21.15.40:0/2395050628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152406e620 0x7f15240b46c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f15140097c0 tx=0x7f151401e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5405dd7d4a36f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:06.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f1523400640 1 -- 172.21.15.40:0/2395050628 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15140ac660 con 0x7f152406e620 2024-03-24T00:34:06.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f1523400640 1 -- 172.21.15.40:0/2395050628 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15140b1070 con 0x7f152406e620 2024-03-24T00:34:06.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f1523400640 1 -- 172.21.15.40:0/2395050628 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15140aca20 con 0x7f152406e620 2024-03-24T00:34:06.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f152aa00640 1 -- 172.21.15.40:0/2395050628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152406e620 msgr2=0x7f15240b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:06.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f152aa00640 1 --2- 172.21.15.40:0/2395050628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152406e620 0x7f15240b46c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f15140097c0 tx=0x7f151401e5f0 comp rx=0 tx=0).stop 2024-03-24T00:34:06.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f152aa00640 1 -- 172.21.15.40:0/2395050628 shutdown_connections 2024-03-24T00:34:06.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f152aa00640 1 --2- 172.21.15.40:0/2395050628 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152406e620 0x7f15240b46c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:06.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f152aa00640 1 -- 172.21.15.40:0/2395050628 >> 172.21.15.40:0/2395050628 conn(0x7f15240b17b0 msgr2=0x7f15240b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:06.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f152aa00640 1 -- 172.21.15.40:0/2395050628 shutdown_connections 2024-03-24T00:34:06.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.707+0000 7f152aa00640 1 -- 172.21.15.40:0/2395050628 wait complete. 2024-03-24T00:34:06.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f152aa00640 1 Processor -- start 2024-03-24T00:34:06.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f152aa00640 1 -- start start 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f152aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152406e620 0x7f1524136e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f152aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152406ea00 0x7f152415a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f152aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f15240ccfa0 0x7f15241f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f152aa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1524144920 con 0x7f152406ea00 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f152aa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152406ee10 con 0x7f152406e620 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f152aa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1524137360 con 0x7f15240ccfa0 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f151be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152406ea00 0x7f152415a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f151be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152406ea00 0x7f152415a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:38942/0 (socket says 172.21.15.40:38942) 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f151be00640 1 -- 172.21.15.40:0/700630043 learned_addr learned my addr 172.21.15.40:0/700630043 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f1523e00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152406e620 0x7f1524136e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f1528a00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f15240ccfa0 0x7f15241f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f151be00640 1 -- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f15240ccfa0 msgr2=0x7f15241f2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:06.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f151be00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f15240ccfa0 0x7f15241f2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:06.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f151be00640 1 -- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152406e620 msgr2=0x7f1524136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:06.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f151be00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152406e620 0x7f1524136e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:06.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f151be00640 1 -- 172.21.15.40:0/700630043 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15240a9a90 con 0x7f152406ea00 2024-03-24T00:34:06.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f151be00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152406ea00 0x7f152415a760 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f150c005890 tx=0x7f150c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:06.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f1520c00640 1 -- 172.21.15.40:0/700630043 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c017070 con 0x7f152406ea00 2024-03-24T00:34:06.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f1520c00640 1 -- 172.21.15.40:0/700630043 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f150c005cf0 con 0x7f152406ea00 2024-03-24T00:34:06.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f152aa00640 1 -- 172.21.15.40:0/700630043 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152415c610 con 0x7f152406ea00 2024-03-24T00:34:06.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.711+0000 7f1520c00640 1 -- 172.21.15.40:0/700630043 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f150c0072b0 con 0x7f152406ea00 2024-03-24T00:34:06.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.715+0000 7f152aa00640 1 -- 172.21.15.40:0/700630043 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f152415aca0 con 0x7f152406ea00 2024-03-24T00:34:06.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.715+0000 7f152aa00640 1 -- 172.21.15.40:0/700630043 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1524068170 con 0x7f152406ea00 2024-03-24T00:34:06.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.715+0000 7f1520c00640 1 -- 172.21.15.40:0/700630043 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f150c01a070 con 0x7f152406ea00 2024-03-24T00:34:06.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.715+0000 7f1520c00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1500067660 0x7f1500069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:06.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.715+0000 7f1520c00640 1 -- 172.21.15.40:0/700630043 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5201+0+0 (secure 0 0 0) 0x7f150c0849a0 con 0x7f152406ea00 2024-03-24T00:34:06.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.715+0000 7f1523e00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1500067660 0x7f1500069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:06.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.715+0000 7f1523e00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1500067660 0x7f1500069b20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f15140b0000 tx=0x7f15140a9000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:06.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.719+0000 7f1520c00640 1 -- 172.21.15.40:0/700630043 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f150c0551b0 con 0x7f152406ea00 2024-03-24T00:34:06.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:06.895+0000 7f152aa00640 1 -- 172.21.15.40:0/700630043 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1524072f50 con 0x7f152406ea00 2024-03-24T00:34:07.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:34:07.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:34:08.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.583+0000 7f1520c00640 1 -- 172.21.15.40:0/700630043 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v55) v1 ==== 97+0+0 (secure 0 0 0) 0x7f150c059250 con 0x7f152406ea00 2024-03-24T00:34:08.586 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-03-24T00:34:08.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.587+0000 7f152aa00640 1 -- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1500067660 msgr2=0x7f1500069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:08.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.587+0000 7f152aa00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1500067660 0x7f1500069b20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f15140b0000 tx=0x7f15140a9000 comp rx=0 tx=0).stop 2024-03-24T00:34:08.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.587+0000 7f152aa00640 1 -- 172.21.15.40:0/700630043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152406ea00 msgr2=0x7f152415a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:08.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.587+0000 7f152aa00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152406ea00 0x7f152415a760 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f150c005890 tx=0x7f150c00c960 comp rx=0 tx=0).stop 2024-03-24T00:34:08.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.587+0000 7f152aa00640 1 -- 172.21.15.40:0/700630043 shutdown_connections 2024-03-24T00:34:08.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.587+0000 7f152aa00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1500067660 0x7f1500069b20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:08.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.587+0000 7f152aa00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f15240ccfa0 0x7f15241f2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:08.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.587+0000 7f152aa00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f152406ea00 0x7f152415a760 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:08.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.587+0000 7f152aa00640 1 --2- 172.21.15.40:0/700630043 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f152406e620 0x7f1524136e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:08.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.587+0000 7f152aa00640 1 -- 172.21.15.40:0/700630043 >> 172.21.15.40:0/700630043 conn(0x7f15240b17b0 msgr2=0x7f1524166270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:08.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.587+0000 7f152aa00640 1 -- 172.21.15.40:0/700630043 shutdown_connections 2024-03-24T00:34:08.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.587+0000 7f152aa00640 1 -- 172.21.15.40:0/700630043 wait complete. 2024-03-24T00:34:08.613 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.755+0000 7f006ca00640 1 Processor -- start 2024-03-24T00:34:08.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.755+0000 7f006ca00640 1 -- start start 2024-03-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f006ca00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0068141bc0 0x7f00680ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:08.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f006ca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00680ce630 0x7f0068071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f006ca00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0068070e00 con 0x7f006813ec80 2024-03-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f006ca00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00680cee20 con 0x7f00680ce630 2024-03-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f006ca00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00680cf100 con 0x7f0068141bc0 2024-03-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f006813ec80 0x7f00681414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53666/0 (socket says 172.21.15.40:53666) 2024-03-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065e00640 1 -- 172.21.15.40:0/2565222725 learned_addr learned my addr 172.21.15.40:0/2565222725 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f005fe00640 1 --2- 172.21.15.40:0/2565222725 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0068141bc0 0x7f00680ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:08.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0066800640 1 --2- 172.21.15.40:0/2565222725 >> v2:172.21.15.53:3300/0 conn(0x7f00680ce630 0x7f0068071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477087941 0 0) 0x7f0068070e00 con 0x7f006813ec80 2024-03-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f004c003530 con 0x7f006813ec80 2024-03-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3680708502 0 0) 0x7f004c003530 con 0x7f006813ec80 2024-03-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0068141bc0 msgr2=0x7f00680ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 --2- 172.21.15.40:0/2565222725 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0068141bc0 0x7f00680ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 >> v2:172.21.15.53:3300/0 conn(0x7f00680ce630 msgr2=0x7f0068071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:08.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 --2- 172.21.15.40:0/2565222725 >> v2:172.21.15.53:3300/0 conn(0x7f00680ce630 0x7f0068071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00680cf3e0 con 0x7f006813ec80 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0066800640 1 --2- 172.21.15.40:0/2565222725 >> v2:172.21.15.53:3300/0 conn(0x7f00680ce630 0x7f0068071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f0058001680 con 0x7f006813ec80 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 >> v1:172.21.15.40:6789/0 conn(0x7f006813ec80 legacy=0x7f00681414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 --2- 172.21.15.40:0/2565222725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f004c0039c0 0x7f004c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 --2- 172.21.15.40:0/2565222725 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0068141bc0 0x7f004c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 --2- 172.21.15.40:0/2565222725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f004c004d50 0x7f004c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00680cee20 con 0x7f004c004d50 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00680cf100 con 0x7f004c0039c0 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f004c00b9d0 con 0x7f0068141bc0 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0066800640 1 --2- 172.21.15.40:0/2565222725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f004c004d50 0x7f004c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0065e00640 1 --2- 172.21.15.40:0/2565222725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f004c0039c0 0x7f004c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f005fe00640 1 --2- 172.21.15.40:0/2565222725 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0068141bc0 0x7f004c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0066800640 1 -- 172.21.15.40:0/2565222725 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0068141bc0 msgr2=0x7f004c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:08.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0066800640 1 --2- 172.21.15.40:0/2565222725 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0068141bc0 0x7f004c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:08.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0066800640 1 -- 172.21.15.40:0/2565222725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f004c0039c0 msgr2=0x7f004c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:08.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0066800640 1 --2- 172.21.15.40:0/2565222725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f004c0039c0 0x7f004c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:08.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.759+0000 7f0066800640 1 -- 172.21.15.40:0/2565222725 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c003530 con 0x7f004c004d50 2024-03-24T00:34:08.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f0066800640 1 --2- 172.21.15.40:0/2565222725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f004c004d50 0x7f004c0051d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f006000e350 tx=0x7f006001ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5f10d285185a1d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:08.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00600b2070 con 0x7f004c004d50 2024-03-24T00:34:08.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00600ad440 con 0x7f004c004d50 2024-03-24T00:34:08.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f0065400640 1 -- 172.21.15.40:0/2565222725 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060009de0 con 0x7f004c004d50 2024-03-24T00:34:08.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f006ca00640 1 -- 172.21.15.40:0/2565222725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f004c004d50 msgr2=0x7f004c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:08.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f006ca00640 1 --2- 172.21.15.40:0/2565222725 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f004c004d50 0x7f004c0051d0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f006000e350 tx=0x7f006001ce80 comp rx=0 tx=0).stop 2024-03-24T00:34:08.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f0066800640 1 -- 172.21.15.40:0/2565222725 reap_dead start 2024-03-24T00:34:08.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f006ca00640 1 -- 172.21.15.40:0/2565222725 shutdown_connections 2024-03-24T00:34:08.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f006ca00640 1 -- 172.21.15.40:0/2565222725 >> 172.21.15.40:0/2565222725 conn(0x7f00680b17b0 msgr2=0x7f00680b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:08.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f006ca00640 1 -- 172.21.15.40:0/2565222725 shutdown_connections 2024-03-24T00:34:08.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f006ca00640 1 -- 172.21.15.40:0/2565222725 wait complete. 2024-03-24T00:34:08.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f006ca00640 1 Processor -- start 2024-03-24T00:34:08.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.763+0000 7f006ca00640 1 -- start start 2024-03-24T00:34:08.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f006ca00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f006813a130 0x7f006815e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:08.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f006ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0068156a50 0x7f0068156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:08.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f006ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f006815c120 0x7f006815c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:08.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f006ca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00680ce0f0 con 0x7f006815c120 2024-03-24T00:34:08.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f006ca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00680ceac0 con 0x7f0068156a50 2024-03-24T00:34:08.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f006ca00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00680ce270 con 0x7f006813a130 2024-03-24T00:34:08.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f0066800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f006815c120 0x7f006815c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:08.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f0066800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f006815c120 0x7f006815c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50050/0 (socket says 172.21.15.40:50050) 2024-03-24T00:34:08.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f0066800640 1 -- 172.21.15.40:0/283161016 learned_addr learned my addr 172.21.15.40:0/283161016 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:08.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f0065e00640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f006813a130 0x7f006815e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:08.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f005fe00640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0068156a50 0x7f0068156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:08.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f0066800640 1 -- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f006813a130 msgr2=0x7f006815e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:08.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f0066800640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f006813a130 0x7f006815e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:08.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f0066800640 1 -- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0068156a50 msgr2=0x7f0068156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:08.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f0066800640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0068156a50 0x7f0068156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:08.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f0066800640 1 -- 172.21.15.40:0/283161016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006813cd30 con 0x7f006815c120 2024-03-24T00:34:08.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f0065e00640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f006813a130 0x7f006815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:08.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f0066800640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f006815c120 0x7f006815c5d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0060002240 tx=0x7f0060009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:08.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f005e000640 1 -- 172.21.15.40:0/283161016 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00600b2040 con 0x7f006815c120 2024-03-24T00:34:08.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f005e000640 1 -- 172.21.15.40:0/283161016 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00600aed90 con 0x7f006815c120 2024-03-24T00:34:08.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f005e000640 1 -- 172.21.15.40:0/283161016 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0060009de0 con 0x7f006815c120 2024-03-24T00:34:08.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f006ca00640 1 -- 172.21.15.40:0/283161016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006813d010 con 0x7f006815c120 2024-03-24T00:34:08.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f006ca00640 1 -- 172.21.15.40:0/283161016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006813a510 con 0x7f006815c120 2024-03-24T00:34:08.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f006ca00640 1 -- 172.21.15.40:0/283161016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0068068170 con 0x7f006815c120 2024-03-24T00:34:08.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.767+0000 7f005e000640 1 -- 172.21.15.40:0/283161016 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f00600be020 con 0x7f006815c120 2024-03-24T00:34:08.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.771+0000 7f005e000640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0044067660 0x7f0044069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:08.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.771+0000 7f005e000640 1 -- 172.21.15.40:0/283161016 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f006011f750 con 0x7f006815c120 2024-03-24T00:34:08.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.771+0000 7f0065e00640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0044067660 0x7f0044069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:08.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.771+0000 7f0065e00640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0044067660 0x7f0044069b20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0058000c10 tx=0x7f0058009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:08.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.775+0000 7f005e000640 1 -- 172.21.15.40:0/283161016 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0060016b90 con 0x7f006815c120 2024-03-24T00:34:08.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 -- 172.21.15.40:0/283161016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0068072b90 con 0x7f006815c120 2024-03-24T00:34:08.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f005e000640 1 -- 172.21.15.40:0/283161016 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f00600f05f0 con 0x7f006815c120 2024-03-24T00:34:08.998 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:08.998 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:34:08.998 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-03-24T00:34:08.998 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:08.998 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-03-24T00:34:08.998 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lCkU6EhAA0q21F90epsGPK5x17s9NLw==", 2024-03-24T00:34:08.998 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:08.998 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:34:08.998 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llkfOHhAAqKKFyEJzG0UxbLJMFPET8A==", 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:08.999 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-03-24T00:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lKMmwGRAAAKaOSnhTRbHIpPKXXkAe+Q==", 2024-03-24T00:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-03-24T00:34:09.000 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l7quKJhAA8QsquiFFmj5H1G5yck0FpA==", 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAVdP9lVk3uBxAA9TC1OpeB5GIFlfJl5+ILBg==", 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:34:09.001 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:34:09.002 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.002 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.002 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-03-24T00:34:09.002 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYdP9lUc1IDhAAPwCV9eqL7bFibkfMFt9wPw==", 2024-03-24T00:34:09.002 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.002 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:34:09.002 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:34:09.002 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.002 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAbdP9lwjyuFBAAJ1JAIAnLXAn03+lQPwwY9Q==", 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAedP9lZ1bgGRAAuLAay48N7jXjj8/yaYwSqg==", 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.003 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:34:09.004 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:34:09.004 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.004 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.004 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-03-24T00:34:09.004 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAidP9lvyk4JhAA8w4/XBYHDLoqJ1zop9sUiw==", 2024-03-24T00:34:09.004 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.004 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:34:09.004 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:34:09.004 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.004 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAldP9lcb9qLBAAXts6VU/4kVqUWS/WORV7zQ==", 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAodP9l8ipTMhAAjpagojh+woHyGvlQlIkHDw==", 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.005 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQArdP9lmGfpORAAPvow5nF/O7RLmuXEqNWENg==", 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw==", 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow r", 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lpIZrNBAAzpGWZuQ6ndjNMAOmAsebvQ==", 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:34:09.007 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l6gXMMBAAd6Oe6S73hj/WfLXTsYiEZw==", 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llFHCOhAATtv8gcL6BHPnxB1dHrg6vw==", 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAQdP9lmmd7KxAA4uk6aUKP6egYj6O1NgdadQ==", 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:34:09.009 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lebNrBxAAvbcCAJhDQn/5Uk1dJ01oHw==", 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:09.010 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-03-24T00:34:09.011 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lEnZyAxAAW5Lrm7LZPTVfLCOY6As1PQ==", 2024-03-24T00:34:09.011 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:34:09.011 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:34:09.011 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:34:09.011 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:34:09.011 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.011 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:09.011 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-24T00:34:09.011 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:34:09.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 -- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0044067660 msgr2=0x7f0044069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0044067660 0x7f0044069b20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f0058000c10 tx=0x7f0058009040 comp rx=0 tx=0).stop 2024-03-24T00:34:09.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 -- 172.21.15.40:0/283161016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f006815c120 msgr2=0x7f006815c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f006815c120 0x7f006815c5d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0060002240 tx=0x7f0060009750 comp rx=0 tx=0).stop 2024-03-24T00:34:09.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 -- 172.21.15.40:0/283161016 shutdown_connections 2024-03-24T00:34:09.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0044067660 0x7f0044069b20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f006815c120 0x7f006815c5d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0068156a50 0x7f0068156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 --2- 172.21.15.40:0/283161016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f006813a130 0x7f006815e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 -- 172.21.15.40:0/283161016 >> 172.21.15.40:0/283161016 conn(0x7f00680b17b0 msgr2=0x7f00681408c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:09.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 -- 172.21.15.40:0/283161016 shutdown_connections 2024-03-24T00:34:09.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:08.995+0000 7f006ca00640 1 -- 172.21.15.40:0/283161016 wait complete. 2024-03-24T00:34:09.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:34:09.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb788800640 1 Processor -- start 2024-03-24T00:34:09.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb788800640 1 -- start start 2024-03-24T00:34:09.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb788800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb7800b2aa0 0x7fb7800d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb788800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7800b2870 con 0x7fb7800d0a80 2024-03-24T00:34:09.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb788800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7800b3190 con 0x7fb7800b2490 2024-03-24T00:34:09.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb788800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78006b6d0 con 0x7fb7800b2aa0 2024-03-24T00:34:09.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb785400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb7800b2aa0 0x7fb7800d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb786800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb7800d0a80 0x7fb7800cc460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53668/0 (socket says 172.21.15.40:53668) 2024-03-24T00:34:09.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb786800640 1 -- 172.21.15.40:0/1972969025 learned_addr learned my addr 172.21.15.40:0/1972969025 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:09.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb785400640 1 -- 172.21.15.40:0/1972969025 >> v1:172.21.15.53:6789/0 conn(0x7fb7800b2490 legacy=0x7fb7800cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb785400640 1 -- 172.21.15.40:0/1972969025 >> v1:172.21.15.40:6789/0 conn(0x7fb7800d0a80 legacy=0x7fb7800cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb784a00640 1 -- 172.21.15.40:0/1972969025 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003827886 0 0) 0x7fb7800b2870 con 0x7fb7800d0a80 2024-03-24T00:34:09.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.159+0000 7fb785400640 1 -- 172.21.15.40:0/1972969025 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7800b3470 con 0x7fb7800b2aa0 2024-03-24T00:34:09.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.163+0000 7fb785400640 1 --2- 172.21.15.40:0/1972969025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7800b2aa0 0x7fb7800d0540 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb770009740 tx=0x7fb77001c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13dbff2ed12d269c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:09.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.163+0000 7fb784a00640 1 -- 172.21.15.40:0/1972969025 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700b0070 con 0x7fb7800b2aa0 2024-03-24T00:34:09.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.163+0000 7fb784a00640 1 -- 172.21.15.40:0/1972969025 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7700ab600 con 0x7fb7800b2aa0 2024-03-24T00:34:09.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.163+0000 7fb784a00640 1 -- 172.21.15.40:0/1972969025 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700ab9d0 con 0x7fb7800b2aa0 2024-03-24T00:34:09.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.163+0000 7fb788800640 1 -- 172.21.15.40:0/1972969025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7800b2aa0 msgr2=0x7fb7800d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.163+0000 7fb788800640 1 --2- 172.21.15.40:0/1972969025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7800b2aa0 0x7fb7800d0540 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fb770009740 tx=0x7fb77001c2d0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.163+0000 7fb788800640 1 -- 172.21.15.40:0/1972969025 shutdown_connections 2024-03-24T00:34:09.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.163+0000 7fb788800640 1 --2- 172.21.15.40:0/1972969025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7800b2aa0 0x7fb7800d0540 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.163+0000 7fb788800640 1 -- 172.21.15.40:0/1972969025 >> 172.21.15.40:0/1972969025 conn(0x7fb7800d1d10 msgr2=0x7fb7800d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:09.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.163+0000 7fb788800640 1 -- 172.21.15.40:0/1972969025 shutdown_connections 2024-03-24T00:34:09.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.163+0000 7fb788800640 1 -- 172.21.15.40:0/1972969025 wait complete. 2024-03-24T00:34:09.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb788800640 1 Processor -- start 2024-03-24T00:34:09.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb788800640 1 -- start start 2024-03-24T00:34:09.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb788800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7800b2490 0x7fb7800c8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb788800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7800b2aa0 0x7fb7800c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb788800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7800d0a80 0x7fb780156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb788800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78016bfd0 con 0x7fb7800d0a80 2024-03-24T00:34:09.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb788800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7800cd650 con 0x7fb7800b2490 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb788800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78016c150 con 0x7fb7800b2aa0 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb786800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7800d0a80 0x7fb780156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb786800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7800d0a80 0x7fb780156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50056/0 (socket says 172.21.15.40:50056) 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb786800640 1 -- 172.21.15.40:0/3278581142 learned_addr learned my addr 172.21.15.40:0/3278581142 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb785e00640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7800b2490 0x7fb7800c8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb785400640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7800b2aa0 0x7fb7800c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb786800640 1 -- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7800b2aa0 msgr2=0x7fb7800c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb786800640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7800b2aa0 0x7fb7800c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb786800640 1 -- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7800b2490 msgr2=0x7fb7800c8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb786800640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7800b2490 0x7fb7800c8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb786800640 1 -- 172.21.15.40:0/3278581142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7800b18f0 con 0x7fb7800d0a80 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb785400640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7800b2aa0 0x7fb7800c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb785e00640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7800b2490 0x7fb7800c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:09.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb786800640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7800d0a80 0x7fb780156fd0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00c7d0 tx=0x7fb77c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:09.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb77a000640 1 -- 172.21.15.40:0/3278581142 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c018070 con 0x7fb7800d0a80 2024-03-24T00:34:09.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb77a000640 1 -- 172.21.15.40:0/3278581142 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77c00a640 con 0x7fb7800d0a80 2024-03-24T00:34:09.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb77a000640 1 -- 172.21.15.40:0/3278581142 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c00aa70 con 0x7fb7800d0a80 2024-03-24T00:34:09.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb788800640 1 -- 172.21.15.40:0/3278581142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7800b1610 con 0x7fb7800d0a80 2024-03-24T00:34:09.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.167+0000 7fb788800640 1 -- 172.21.15.40:0/3278581142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7800aede0 con 0x7fb7800d0a80 2024-03-24T00:34:09.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.171+0000 7fb788800640 1 -- 172.21.15.40:0/3278581142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb780068170 con 0x7fb7800d0a80 2024-03-24T00:34:09.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.171+0000 7fb77a000640 1 -- 172.21.15.40:0/3278581142 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb77c00ac10 con 0x7fb7800d0a80 2024-03-24T00:34:09.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.171+0000 7fb77a000640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb764067660 0x7fb764069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.171+0000 7fb77a000640 1 -- 172.21.15.40:0/3278581142 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb77c084c30 con 0x7fb7800d0a80 2024-03-24T00:34:09.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.171+0000 7fb785e00640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb764067660 0x7fb764069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.171+0000 7fb785e00640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb764067660 0x7fb764069b20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb774000f70 tx=0x7fb774009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:09.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.175+0000 7fb77a000640 1 -- 172.21.15.40:0/3278581142 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb77c055b60 con 0x7fb7800d0a80 2024-03-24T00:34:09.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.339+0000 7fb788800640 1 -- 172.21.15.40:0/3278581142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb780070750 con 0x7fb7800d0a80 2024-03-24T00:34:09.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.339+0000 7fb77a000640 1 -- 172.21.15.40:0/3278581142 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+12831 (secure 0 0 0) 0x7fb77c055460 con 0x7fb7800d0a80 2024-03-24T00:34:09.342 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:09.342 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":55,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:34:08.583743+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:34:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.339+0000 7fb788800640 1 -- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb764067660 msgr2=0x7fb764069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.339+0000 7fb788800640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb764067660 0x7fb764069b20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb774000f70 tx=0x7fb774009040 comp rx=0 tx=0).stop 2024-03-24T00:34:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.339+0000 7fb788800640 1 -- 172.21.15.40:0/3278581142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7800d0a80 msgr2=0x7fb780156fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.339+0000 7fb788800640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7800d0a80 0x7fb780156fd0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00c7d0 tx=0x7fb77c00cc90 comp rx=0 tx=0).stop 2024-03-24T00:34:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.339+0000 7fb788800640 1 -- 172.21.15.40:0/3278581142 shutdown_connections 2024-03-24T00:34:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.339+0000 7fb788800640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb764067660 0x7fb764069b20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.339+0000 7fb788800640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7800d0a80 0x7fb780156fd0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.339+0000 7fb788800640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7800b2aa0 0x7fb7800c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.339+0000 7fb788800640 1 --2- 172.21.15.40:0/3278581142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7800b2490 0x7fb7800c8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.339+0000 7fb788800640 1 -- 172.21.15.40:0/3278581142 >> 172.21.15.40:0/3278581142 conn(0x7fb7800d1d10 msgr2=0x7fb7800c9120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:09.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.343+0000 7fb788800640 1 -- 172.21.15.40:0/3278581142 shutdown_connections 2024-03-24T00:34:09.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.343+0000 7fb788800640 1 -- 172.21.15.40:0/3278581142 wait complete. 2024-03-24T00:34:09.364 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:34:09.364 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:34:09.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f91c00640 1 Processor -- start 2024-03-24T00:34:09.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f91c00640 1 -- start start 2024-03-24T00:34:09.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f91c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5f8c0cf4e0 0x7f5f8c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f91c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5f8c072b60 0x7f5f8c0b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f91c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f8c15f2c0 con 0x7f5f8c0724a0 2024-03-24T00:34:09.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f91c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f8c06cfe0 con 0x7f5f8c0cf4e0 2024-03-24T00:34:09.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f91c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f8c06d880 con 0x7f5f8c072b60 2024-03-24T00:34:09.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f8aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5f8c0cf4e0 0x7f5f8c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f8b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5f8c0724a0 0x7f5f8c070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53672/0 (socket says 172.21.15.40:53672) 2024-03-24T00:34:09.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f8b400640 1 -- 172.21.15.40:0/1851749395 learned_addr learned my addr 172.21.15.40:0/1851749395 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:09.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f82000640 1 --2- 172.21.15.40:0/1851749395 >> v2:172.21.15.53:3300/0 conn(0x7f5f8c072b60 0x7f5f8c0b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f8aa00640 1 -- 172.21.15.40:0/1851749395 >> v2:172.21.15.53:3300/0 conn(0x7f5f8c072b60 msgr2=0x7f5f8c0b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:09.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f8aa00640 1 --2- 172.21.15.40:0/1851749395 >> v2:172.21.15.53:3300/0 conn(0x7f5f8c072b60 0x7f5f8c0b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f8aa00640 1 -- 172.21.15.40:0/1851749395 >> v1:172.21.15.40:6789/0 conn(0x7f5f8c0724a0 legacy=0x7f5f8c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.507+0000 7f5f8aa00640 1 -- 172.21.15.40:0/1851749395 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c06d230 con 0x7f5f8c0cf4e0 2024-03-24T00:34:09.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.511+0000 7f5f8aa00640 1 --2- 172.21.15.40:0/1851749395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f8c0cf4e0 0x7f5f8c06dd10 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c073480 tx=0x7f5f7c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb370fe54b249ba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:09.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.511+0000 7f5f8a000640 1 -- 172.21.15.40:0/1851749395 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c0b5070 con 0x7f5f8c0cf4e0 2024-03-24T00:34:09.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.511+0000 7f5f8a000640 1 -- 172.21.15.40:0/1851749395 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7c0a8970 con 0x7f5f8c0cf4e0 2024-03-24T00:34:09.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.511+0000 7f5f8a000640 1 -- 172.21.15.40:0/1851749395 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c0a8d40 con 0x7f5f8c0cf4e0 2024-03-24T00:34:09.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.511+0000 7f5f91c00640 1 -- 172.21.15.40:0/1851749395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f8c0cf4e0 msgr2=0x7f5f8c06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.511+0000 7f5f91c00640 1 --2- 172.21.15.40:0/1851749395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f8c0cf4e0 0x7f5f8c06dd10 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c073480 tx=0x7f5f7c01ec70 comp rx=0 tx=0).stop 2024-03-24T00:34:09.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.511+0000 7f5f91c00640 1 -- 172.21.15.40:0/1851749395 shutdown_connections 2024-03-24T00:34:09.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.511+0000 7f5f91c00640 1 --2- 172.21.15.40:0/1851749395 >> v2:172.21.15.53:3300/0 conn(0x7f5f8c072b60 0x7f5f8c0b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.511+0000 7f5f91c00640 1 --2- 172.21.15.40:0/1851749395 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f8c0cf4e0 0x7f5f8c06dd10 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.511+0000 7f5f91c00640 1 -- 172.21.15.40:0/1851749395 >> 172.21.15.40:0/1851749395 conn(0x7f5f8c0b1660 msgr2=0x7f5f8c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:09.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.511+0000 7f5f91c00640 1 -- 172.21.15.40:0/1851749395 shutdown_connections 2024-03-24T00:34:09.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.511+0000 7f5f91c00640 1 -- 172.21.15.40:0/1851749395 wait complete. 2024-03-24T00:34:09.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f91c00640 1 Processor -- start 2024-03-24T00:34:09.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f91c00640 1 -- start start 2024-03-24T00:34:09.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f91c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f8c0724a0 0x7f5f8c152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f91c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f8c072b60 0x7f5f8c071d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f91c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f8c0cf4e0 0x7f5f8c15d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f91c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c15ef20 con 0x7f5f8c072b60 2024-03-24T00:34:09.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f91c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c06e250 con 0x7f5f8c0724a0 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f91c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c15f0a0 con 0x7f5f8c0cf4e0 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f82000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f8c072b60 0x7f5f8c071d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f82000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f8c072b60 0x7f5f8c071d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50070/0 (socket says 172.21.15.40:50070) 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f82000640 1 -- 172.21.15.40:0/4223277601 learned_addr learned my addr 172.21.15.40:0/4223277601 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f8aa00640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f8c0724a0 0x7f5f8c152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f82000640 1 -- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f8c0cf4e0 msgr2=0x7f5f8c15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f8b400640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f8c0cf4e0 0x7f5f8c15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f82000640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f8c0cf4e0 0x7f5f8c15d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f82000640 1 -- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f8c0724a0 msgr2=0x7f5f8c152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f82000640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f8c0724a0 0x7f5f8c152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f82000640 1 -- 172.21.15.40:0/4223277601 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c153d60 con 0x7f5f8c072b60 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f8aa00640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f8c0724a0 0x7f5f8c152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:34:09.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f8b400640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f8c0cf4e0 0x7f5f8c15d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:34:09.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f82000640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f8c072b60 0x7f5f8c071d50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5f7400d7d0 tx=0x7f5f7400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:09.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f83400640 1 -- 172.21.15.40:0/4223277601 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7400b6c0 con 0x7f5f8c072b60 2024-03-24T00:34:09.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f83400640 1 -- 172.21.15.40:0/4223277601 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f74019070 con 0x7f5f8c072b60 2024-03-24T00:34:09.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f83400640 1 -- 172.21.15.40:0/4223277601 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7400baa0 con 0x7f5f8c072b60 2024-03-24T00:34:09.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f91c00640 1 -- 172.21.15.40:0/4223277601 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8c153fe0 con 0x7f5f8c072b60 2024-03-24T00:34:09.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.515+0000 7f5f91c00640 1 -- 172.21.15.40:0/4223277601 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8c160540 con 0x7f5f8c072b60 2024-03-24T00:34:09.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.519+0000 7f5f91c00640 1 -- 172.21.15.40:0/4223277601 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8c1543d0 con 0x7f5f8c072b60 2024-03-24T00:34:09.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.519+0000 7f5f83400640 1 -- 172.21.15.40:0/4223277601 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5f7400bc40 con 0x7f5f8c072b60 2024-03-24T00:34:09.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.519+0000 7f5f83400640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f68067660 0x7f5f68069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.519+0000 7f5f83400640 1 -- 172.21.15.40:0/4223277601 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5f74085a80 con 0x7f5f8c072b60 2024-03-24T00:34:09.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.519+0000 7f5f8aa00640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f68067660 0x7f5f68069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.519+0000 7f5f8aa00640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f68067660 0x7f5f68069b20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c0b2000 tx=0x7f5f7c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:09.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.523+0000 7f5f83400640 1 -- 172.21.15.40:0/4223277601 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5f74087e20 con 0x7f5f8c072b60 2024-03-24T00:34:09.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.691+0000 7f5f91c00640 1 -- 172.21.15.40:0/4223277601 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5f8c068170 con 0x7f5f8c072b60 2024-03-24T00:34:09.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.691+0000 7f5f83400640 1 -- 172.21.15.40:0/4223277601 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5f8c153d60 con 0x7f5f8c072b60 2024-03-24T00:34:09.694 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:34:09.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.691+0000 7f5f91c00640 1 -- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f68067660 msgr2=0x7f5f68069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.691+0000 7f5f91c00640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f68067660 0x7f5f68069b20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c0b2000 tx=0x7f5f7c0aa070 comp rx=0 tx=0).stop 2024-03-24T00:34:09.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.691+0000 7f5f91c00640 1 -- 172.21.15.40:0/4223277601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f8c072b60 msgr2=0x7f5f8c071d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.691+0000 7f5f91c00640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f8c072b60 0x7f5f8c071d50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f5f7400d7d0 tx=0x7f5f7400dc90 comp rx=0 tx=0).stop 2024-03-24T00:34:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.695+0000 7f5f91c00640 1 -- 172.21.15.40:0/4223277601 shutdown_connections 2024-03-24T00:34:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.695+0000 7f5f91c00640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f68067660 0x7f5f68069b20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.695+0000 7f5f91c00640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f8c0cf4e0 0x7f5f8c15d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.695+0000 7f5f91c00640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f8c072b60 0x7f5f8c071d50 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.695+0000 7f5f91c00640 1 --2- 172.21.15.40:0/4223277601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f8c0724a0 0x7f5f8c152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.695+0000 7f5f91c00640 1 -- 172.21.15.40:0/4223277601 >> 172.21.15.40:0/4223277601 conn(0x7f5f8c0b1660 msgr2=0x7f5f8c15a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.695+0000 7f5f91c00640 1 -- 172.21.15.40:0/4223277601 shutdown_connections 2024-03-24T00:34:09.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.695+0000 7f5f91c00640 1 -- 172.21.15.40:0/4223277601 wait complete. 2024-03-24T00:34:09.715 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.851+0000 7f7a27a00640 1 Processor -- start 2024-03-24T00:34:09.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.851+0000 7f7a27a00640 1 -- start start 2024-03-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.855+0000 7f7a27a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a2006e620 0x7f7a200b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.855+0000 7f7a27a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a20070f60 con 0x7f7a2006ea00 2024-03-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.855+0000 7f7a27a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a200b3620 con 0x7f7a200ccfa0 2024-03-24T00:34:09.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.855+0000 7f7a27a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a200b4c00 con 0x7f7a2006e620 2024-03-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.855+0000 7f7a25000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a2006e620 0x7f7a200b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.855+0000 7f7a1fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7a2006ea00 0x7f7a200b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53684/0 (socket says 172.21.15.40:53684) 2024-03-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.855+0000 7f7a1fe00640 1 -- 172.21.15.40:0/227512184 learned_addr learned my addr 172.21.15.40:0/227512184 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.855+0000 7f7a25000640 1 -- 172.21.15.40:0/227512184 >> v1:172.21.15.53:6789/0 conn(0x7f7a200ccfa0 legacy=0x7f7a200cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.855+0000 7f7a25000640 1 -- 172.21.15.40:0/227512184 >> v1:172.21.15.40:6789/0 conn(0x7f7a2006ea00 legacy=0x7f7a200b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.855+0000 7f7a1f400640 1 -- 172.21.15.40:0/227512184 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310550716 0 0) 0x7f7a20070f60 con 0x7f7a2006ea00 2024-03-24T00:34:09.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.855+0000 7f7a25000640 1 -- 172.21.15.40:0/227512184 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a200b3900 con 0x7f7a2006e620 2024-03-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.855+0000 7f7a25000640 1 --2- 172.21.15.40:0/227512184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2006e620 0x7f7a200b46c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f7a140097c0 tx=0x7f7a1401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8494131b9896ebfc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a1f400640 1 -- 172.21.15.40:0/227512184 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a140b1070 con 0x7f7a2006e620 2024-03-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a1f400640 1 -- 172.21.15.40:0/227512184 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a140a8050 con 0x7f7a2006e620 2024-03-24T00:34:09.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a1f400640 1 -- 172.21.15.40:0/227512184 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a140ac860 con 0x7f7a2006e620 2024-03-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 -- 172.21.15.40:0/227512184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2006e620 msgr2=0x7f7a200b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 --2- 172.21.15.40:0/227512184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2006e620 0x7f7a200b46c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f7a140097c0 tx=0x7f7a1401c3e0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 -- 172.21.15.40:0/227512184 shutdown_connections 2024-03-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 --2- 172.21.15.40:0/227512184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a2006e620 0x7f7a200b46c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 -- 172.21.15.40:0/227512184 >> 172.21.15.40:0/227512184 conn(0x7f7a200b17b0 msgr2=0x7f7a200b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 -- 172.21.15.40:0/227512184 shutdown_connections 2024-03-24T00:34:09.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 -- 172.21.15.40:0/227512184 wait complete. 2024-03-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 Processor -- start 2024-03-24T00:34:09.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 -- start start 2024-03-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a2006e620 0x7f7a2013a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a2006ea00 0x7f7a20134b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200ccfa0 0x7f7a201350a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2006efd0 con 0x7f7a200ccfa0 2024-03-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2006ee50 con 0x7f7a2006e620 2024-03-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a27a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a20144920 con 0x7f7a2006ea00 2024-03-24T00:34:09.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a25a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200ccfa0 0x7f7a201350a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a25a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200ccfa0 0x7f7a201350a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50088/0 (socket says 172.21.15.40:50088) 2024-03-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a25a00640 1 -- 172.21.15.40:0/2731054815 learned_addr learned my addr 172.21.15.40:0/2731054815 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a25000640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a2006e620 0x7f7a2013a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.859+0000 7f7a1fe00640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a2006ea00 0x7f7a20134b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a25a00640 1 -- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a2006ea00 msgr2=0x7f7a20134b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a25a00640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a2006ea00 0x7f7a20134b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a25a00640 1 -- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a2006e620 msgr2=0x7f7a2013a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a25a00640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a2006e620 0x7f7a2013a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a25a00640 1 -- 172.21.15.40:0/2731054815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2015ac50 con 0x7f7a200ccfa0 2024-03-24T00:34:09.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a25a00640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200ccfa0 0x7f7a201350a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7a1800c730 tx=0x7f7a1800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a1cc00640 1 -- 172.21.15.40:0/2731054815 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18003b80 con 0x7f7a200ccfa0 2024-03-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a1cc00640 1 -- 172.21.15.40:0/2731054815 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a18003d20 con 0x7f7a200ccfa0 2024-03-24T00:34:09.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a1cc00640 1 -- 172.21.15.40:0/2731054815 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1800aa30 con 0x7f7a200ccfa0 2024-03-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a27a00640 1 -- 172.21.15.40:0/2731054815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2015b500 con 0x7f7a200ccfa0 2024-03-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a27a00640 1 -- 172.21.15.40:0/2731054815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2015ba80 con 0x7f7a200ccfa0 2024-03-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a27a00640 1 -- 172.21.15.40:0/2731054815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a20068170 con 0x7f7a200ccfa0 2024-03-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a1cc00640 1 -- 172.21.15.40:0/2731054815 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7a1800abd0 con 0x7f7a200ccfa0 2024-03-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a1cc00640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a00067660 0x7f7a00069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:09.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a1cc00640 1 -- 172.21.15.40:0/2731054815 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7a18085320 con 0x7f7a200ccfa0 2024-03-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.863+0000 7f7a25000640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a00067660 0x7f7a00069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:09.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.867+0000 7f7a25000640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a00067660 0x7f7a00069b20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7a140ae000 tx=0x7f7a14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:09.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:09.867+0000 7f7a1cc00640 1 -- 172.21.15.40:0/2731054815 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7a18055c00 con 0x7f7a200ccfa0 2024-03-24T00:34:10.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.031+0000 7f7a27a00640 1 -- 172.21.15.40:0/2731054815 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7a20072f50 con 0x7f7a200ccfa0 2024-03-24T00:34:10.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.031+0000 7f7a1cc00640 1 -- 172.21.15.40:0/2731054815 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v55) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7a18059ca0 con 0x7f7a200ccfa0 2024-03-24T00:34:10.035 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:10.035 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-03-24T00:34:10.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:34:10.035 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-03-24T00:34:10.035 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-03-24T00:34:10.035 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:10.036 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-03-24T00:34:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.035+0000 7f7a27a00640 1 -- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a00067660 msgr2=0x7f7a00069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.035+0000 7f7a27a00640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a00067660 0x7f7a00069b20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7a140ae000 tx=0x7f7a14008040 comp rx=0 tx=0).stop 2024-03-24T00:34:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.035+0000 7f7a27a00640 1 -- 172.21.15.40:0/2731054815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200ccfa0 msgr2=0x7f7a201350a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.035+0000 7f7a27a00640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200ccfa0 0x7f7a201350a0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f7a1800c730 tx=0x7f7a1800cbf0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.035+0000 7f7a27a00640 1 -- 172.21.15.40:0/2731054815 shutdown_connections 2024-03-24T00:34:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.035+0000 7f7a27a00640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a00067660 0x7f7a00069b20 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.035+0000 7f7a27a00640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200ccfa0 0x7f7a201350a0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.035+0000 7f7a27a00640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a2006ea00 0x7f7a20134b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.035+0000 7f7a27a00640 1 --2- 172.21.15.40:0/2731054815 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a2006e620 0x7f7a2013a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.035+0000 7f7a27a00640 1 -- 172.21.15.40:0/2731054815 >> 172.21.15.40:0/2731054815 conn(0x7f7a200b17b0 msgr2=0x7f7a2006d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.035+0000 7f7a27a00640 1 -- 172.21.15.40:0/2731054815 shutdown_connections 2024-03-24T00:34:10.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.035+0000 7f7a27a00640 1 -- 172.21.15.40:0/2731054815 wait complete. 2024-03-24T00:34:10.056 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:34:10.056 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:34:10.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97fb800640 1 Processor -- start 2024-03-24T00:34:10.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97fb800640 1 -- start start 2024-03-24T00:34:10.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97fb800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f97f415f970 0x7f97f415fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97fb800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f97f40cdfe0 0x7f97f415c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97fb800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f40cc550 con 0x7f97f40cefb0 2024-03-24T00:34:10.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97fb800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f40ccb40 con 0x7f97f40cdfe0 2024-03-24T00:34:10.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97fb800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f40d0a10 con 0x7f97f415f970 2024-03-24T00:34:10.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97f9800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f97f40cdfe0 0x7f97f415c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97ebe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f97f40cefb0 0x7f97f415c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53696/0 (socket says 172.21.15.40:53696) 2024-03-24T00:34:10.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97ebe00640 1 -- 172.21.15.40:0/1787097075 learned_addr learned my addr 172.21.15.40:0/1787097075 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:10.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97f8e00640 1 --2- 172.21.15.40:0/1787097075 >> v2:172.21.15.53:3300/0 conn(0x7f97f415f970 0x7f97f415fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97f3e00640 1 -- 172.21.15.40:0/1787097075 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442045990 0 0) 0x7f97dc001120 con 0x7f97f40cefb0 2024-03-24T00:34:10.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97f3e00640 1 -- 172.21.15.40:0/1787097075 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97d8003530 con 0x7f97f40cefb0 2024-03-24T00:34:10.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97f9800640 1 -- 172.21.15.40:0/1787097075 >> v2:172.21.15.53:3300/0 conn(0x7f97f415f970 msgr2=0x7f97f415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:10.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97f9800640 1 --2- 172.21.15.40:0/1787097075 >> v2:172.21.15.53:3300/0 conn(0x7f97f415f970 0x7f97f415fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97f9800640 1 -- 172.21.15.40:0/1787097075 >> v1:172.21.15.40:6789/0 conn(0x7f97f40cefb0 legacy=0x7f97f415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.199+0000 7f97f9800640 1 -- 172.21.15.40:0/1787097075 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f4160290 con 0x7f97f40cdfe0 2024-03-24T00:34:10.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.203+0000 7f97f9800640 1 --2- 172.21.15.40:0/1787097075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97f40cdfe0 0x7f97f415c860 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f97f40cccc0 tx=0x7f97ec01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2170c7d060d9d7b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:10.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.203+0000 7f97f3e00640 1 -- 172.21.15.40:0/1787097075 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec0b2070 con 0x7f97f40cdfe0 2024-03-24T00:34:10.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.203+0000 7f97f3e00640 1 -- 172.21.15.40:0/1787097075 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ec0a5930 con 0x7f97f40cdfe0 2024-03-24T00:34:10.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.203+0000 7f97f3e00640 1 -- 172.21.15.40:0/1787097075 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec0a5d00 con 0x7f97f40cdfe0 2024-03-24T00:34:10.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.203+0000 7f97fb800640 1 -- 172.21.15.40:0/1787097075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97f40cdfe0 msgr2=0x7f97f415c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.203+0000 7f97fb800640 1 --2- 172.21.15.40:0/1787097075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97f40cdfe0 0x7f97f415c860 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f97f40cccc0 tx=0x7f97ec01c870 comp rx=0 tx=0).stop 2024-03-24T00:34:10.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.203+0000 7f97fb800640 1 -- 172.21.15.40:0/1787097075 shutdown_connections 2024-03-24T00:34:10.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.203+0000 7f97fb800640 1 --2- 172.21.15.40:0/1787097075 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97f40cdfe0 0x7f97f415c860 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.203+0000 7f97fb800640 1 --2- 172.21.15.40:0/1787097075 >> v2:172.21.15.53:3300/0 conn(0x7f97f415f970 0x7f97f415fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.203+0000 7f97fb800640 1 -- 172.21.15.40:0/1787097075 >> 172.21.15.40:0/1787097075 conn(0x7f97f40b36a0 msgr2=0x7f97f40b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:10.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.203+0000 7f97fb800640 1 -- 172.21.15.40:0/1787097075 shutdown_connections 2024-03-24T00:34:10.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.203+0000 7f97fb800640 1 -- 172.21.15.40:0/1787097075 wait complete. 2024-03-24T00:34:10.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97fb800640 1 Processor -- start 2024-03-24T00:34:10.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97fb800640 1 -- start start 2024-03-24T00:34:10.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97fb800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f97f40cdfe0 0x7f97f415e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97fb800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97f40cefb0 0x7f97f406c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97fb800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97f415f970 0x7f97f40af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97fb800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f415d1b0 con 0x7f97f415f970 2024-03-24T00:34:10.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97fb800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f40cc7d0 con 0x7f97f40cefb0 2024-03-24T00:34:10.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97fb800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f41606c0 con 0x7f97f40cdfe0 2024-03-24T00:34:10.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97f415f970 0x7f97f40af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97f415f970 0x7f97f40af9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50116/0 (socket says 172.21.15.40:50116) 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f9800640 1 -- 172.21.15.40:0/3517504813 learned_addr learned my addr 172.21.15.40:0/3517504813 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f8e00640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f97f40cdfe0 0x7f97f415e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97ebe00640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97f40cefb0 0x7f97f406c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f9800640 1 -- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f97f40cdfe0 msgr2=0x7f97f415e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f9800640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f97f40cdfe0 0x7f97f415e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f9800640 1 -- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97f40cefb0 msgr2=0x7f97f406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f9800640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97f40cefb0 0x7f97f406c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f9800640 1 -- 172.21.15.40:0/3517504813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f415ed90 con 0x7f97f415f970 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f8e00640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f97f40cdfe0 0x7f97f415e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f9800640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97f415f970 0x7f97f40af9b0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f97ec019480 tx=0x7f97ec019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f1600640 1 -- 172.21.15.40:0/3517504813 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec0b2050 con 0x7f97f415f970 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f1600640 1 -- 172.21.15.40:0/3517504813 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ec0be070 con 0x7f97f415f970 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97fb800640 1 -- 172.21.15.40:0/3517504813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f415f070 con 0x7f97f415f970 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97f1600640 1 -- 172.21.15.40:0/3517504813 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec0ada50 con 0x7f97f415f970 2024-03-24T00:34:10.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.207+0000 7f97fb800640 1 -- 172.21.15.40:0/3517504813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f415f630 con 0x7f97f415f970 2024-03-24T00:34:10.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.211+0000 7f97fb800640 1 -- 172.21.15.40:0/3517504813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f4068170 con 0x7f97f415f970 2024-03-24T00:34:10.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.211+0000 7f97f1600640 1 -- 172.21.15.40:0/3517504813 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f97ec0a57b0 con 0x7f97f415f970 2024-03-24T00:34:10.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.211+0000 7f97f1600640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f97d0067660 0x7f97d0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.211+0000 7f97f1600640 1 -- 172.21.15.40:0/3517504813 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f97ec120710 con 0x7f97f415f970 2024-03-24T00:34:10.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.211+0000 7f97f8e00640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f97d0067660 0x7f97d0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.211+0000 7f97f8e00640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f97d0067660 0x7f97d0069b20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009b10 tx=0x7f97e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:10.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.215+0000 7f97f1600640 1 -- 172.21.15.40:0/3517504813 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f97ec016520 con 0x7f97f415f970 2024-03-24T00:34:10.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.379+0000 7f97fb800640 1 -- 172.21.15.40:0/3517504813 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f97f41604a0 con 0x7f97f415f970 2024-03-24T00:34:10.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.599+0000 7f97f1600640 1 -- 172.21.15.40:0/3517504813 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v56) v1 ==== 138+0+0 (secure 0 0 0) 0x7f97ec0f1890 con 0x7f97f415f970 2024-03-24T00:34:10.602 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-03-24T00:34:10.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.603+0000 7f97fb800640 1 -- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f97d0067660 msgr2=0x7f97d0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.603+0000 7f97fb800640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f97d0067660 0x7f97d0069b20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f97e4009b10 tx=0x7f97e4008040 comp rx=0 tx=0).stop 2024-03-24T00:34:10.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.603+0000 7f97fb800640 1 -- 172.21.15.40:0/3517504813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97f415f970 msgr2=0x7f97f40af9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.603+0000 7f97fb800640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97f415f970 0x7f97f40af9b0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f97ec019480 tx=0x7f97ec019610 comp rx=0 tx=0).stop 2024-03-24T00:34:10.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.603+0000 7f97fb800640 1 -- 172.21.15.40:0/3517504813 shutdown_connections 2024-03-24T00:34:10.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.603+0000 7f97fb800640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f97d0067660 0x7f97d0069b20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.603+0000 7f97fb800640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f97f415f970 0x7f97f40af9b0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.603+0000 7f97fb800640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97f40cefb0 0x7f97f406c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.603+0000 7f97fb800640 1 --2- 172.21.15.40:0/3517504813 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f97f40cdfe0 0x7f97f415e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.603+0000 7f97fb800640 1 -- 172.21.15.40:0/3517504813 >> 172.21.15.40:0/3517504813 conn(0x7f97f40b36a0 msgr2=0x7f97f4159df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:10.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.603+0000 7f97fb800640 1 -- 172.21.15.40:0/3517504813 shutdown_connections 2024-03-24T00:34:10.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.603+0000 7f97fb800640 1 -- 172.21.15.40:0/3517504813 wait complete. 2024-03-24T00:34:10.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:34:10.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4eca400640 1 Processor -- start 2024-03-24T00:34:10.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4eca400640 1 -- start start 2024-03-24T00:34:10.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4eca400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4ec415e350 0x7f4ec40cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4eca400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec40734e0 con 0x7f4ec40c9580 2024-03-24T00:34:10.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4eca400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec40d0030 con 0x7f4ec40ad8f0 2024-03-24T00:34:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4eca400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec40cd660 con 0x7f4ec415e350 2024-03-24T00:34:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4ebaa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4ec40c9580 0x7f4ec40ad1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53698/0 (socket says 172.21.15.40:53698) 2024-03-24T00:34:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4ebaa00640 1 -- 172.21.15.40:0/3798325743 learned_addr learned my addr 172.21.15.40:0/3798325743 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4ec3400640 1 --2- 172.21.15.40:0/3798325743 >> v2:172.21.15.53:3300/0 conn(0x7f4ec415e350 0x7f4ec40cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4ec2a00640 1 -- 172.21.15.40:0/3798325743 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625309616 0 0) 0x7f4ec40734e0 con 0x7f4ec40c9580 2024-03-24T00:34:10.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4ec2a00640 1 -- 172.21.15.40:0/3798325743 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea8003530 con 0x7f4ec40c9580 2024-03-24T00:34:10.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4ec3400640 1 -- 172.21.15.40:0/3798325743 >> v1:172.21.15.53:6789/0 conn(0x7f4ec40ad8f0 legacy=0x7f4ec4168d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4ec3400640 1 -- 172.21.15.40:0/3798325743 >> v1:172.21.15.40:6789/0 conn(0x7f4ec40c9580 legacy=0x7f4ec40ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.775+0000 7f4ec3400640 1 -- 172.21.15.40:0/3798325743 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec40745a0 con 0x7f4ec415e350 2024-03-24T00:34:10.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.779+0000 7f4ec3400640 1 --2- 172.21.15.40:0/3798325743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ec415e350 0x7f4ec40cfaf0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009d90 tx=0x7f4eb401c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5658b064580f894 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:10.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.779+0000 7f4ec2a00640 1 -- 172.21.15.40:0/3798325743 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb40b2070 con 0x7f4ec415e350 2024-03-24T00:34:10.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.779+0000 7f4ec2a00640 1 -- 172.21.15.40:0/3798325743 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb40a5940 con 0x7f4ec415e350 2024-03-24T00:34:10.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.779+0000 7f4ec2a00640 1 -- 172.21.15.40:0/3798325743 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb40a5d10 con 0x7f4ec415e350 2024-03-24T00:34:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.779+0000 7f4eca400640 1 -- 172.21.15.40:0/3798325743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ec415e350 msgr2=0x7f4ec40cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.779+0000 7f4eca400640 1 --2- 172.21.15.40:0/3798325743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ec415e350 0x7f4ec40cfaf0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009d90 tx=0x7f4eb401c970 comp rx=0 tx=0).stop 2024-03-24T00:34:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.779+0000 7f4eca400640 1 -- 172.21.15.40:0/3798325743 shutdown_connections 2024-03-24T00:34:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.779+0000 7f4eca400640 1 --2- 172.21.15.40:0/3798325743 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ec415e350 0x7f4ec40cfaf0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.779+0000 7f4eca400640 1 -- 172.21.15.40:0/3798325743 >> 172.21.15.40:0/3798325743 conn(0x7f4ec4072870 msgr2=0x7f4ec4072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.779+0000 7f4eca400640 1 -- 172.21.15.40:0/3798325743 shutdown_connections 2024-03-24T00:34:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.779+0000 7f4eca400640 1 -- 172.21.15.40:0/3798325743 wait complete. 2024-03-24T00:34:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4eca400640 1 Processor -- start 2024-03-24T00:34:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4eca400640 1 -- start start 2024-03-24T00:34:10.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4eca400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ec40ad8f0 0x7f4ec40b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4eca400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ec40c9580 0x7f4ec40cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4eca400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec415e350 0x7f4ec4154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4eca400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec40cda30 con 0x7f4ec415e350 2024-03-24T00:34:10.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4eca400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec40716b0 con 0x7f4ec40ad8f0 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4eca400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec40cdbb0 con 0x7f4ec40c9580 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ec3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec415e350 0x7f4ec4154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ec3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec415e350 0x7f4ec4154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50132/0 (socket says 172.21.15.40:50132) 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ec3e00640 1 -- 172.21.15.40:0/1137190157 learned_addr learned my addr 172.21.15.40:0/1137190157 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ebaa00640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ec40c9580 0x7f4ec40cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ec3400640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ec40ad8f0 0x7f4ec40b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ec3e00640 1 -- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ec40c9580 msgr2=0x7f4ec40cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ec3e00640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ec40c9580 0x7f4ec40cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ec3e00640 1 -- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ec40ad8f0 msgr2=0x7f4ec40b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ec3e00640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ec40ad8f0 0x7f4ec40b4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ec3e00640 1 -- 172.21.15.40:0/1137190157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec40cbaa0 con 0x7f4ec415e350 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ec3400640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ec40ad8f0 0x7f4ec40b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ebaa00640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ec40c9580 0x7f4ec40cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:10.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ec3e00640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec415e350 0x7f4ec4154250 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00c3f0 tx=0x7f4ebc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1137190157 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc017070 con 0x7f4ec415e350 2024-03-24T00:34:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1137190157 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc0052d0 con 0x7f4ec415e350 2024-03-24T00:34:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1137190157 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc003860 con 0x7f4ec415e350 2024-03-24T00:34:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4eca400640 1 -- 172.21.15.40:0/1137190157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec40cbd80 con 0x7f4ec415e350 2024-03-24T00:34:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.783+0000 7f4eca400640 1 -- 172.21.15.40:0/1137190157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec40caa00 con 0x7f4ec415e350 2024-03-24T00:34:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.787+0000 7f4eca400640 1 -- 172.21.15.40:0/1137190157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec4068170 con 0x7f4ec415e350 2024-03-24T00:34:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.787+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1137190157 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4ebc01a070 con 0x7f4ec415e350 2024-03-24T00:34:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.787+0000 7f4ebbe00640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ea0067660 0x7f4ea0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.787+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1137190157 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f4ebc084f10 con 0x7f4ec415e350 2024-03-24T00:34:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.787+0000 7f4ec3400640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ea0067660 0x7f4ea0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.787+0000 7f4ec3400640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ea0067660 0x7f4ea0069b20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40af000 tx=0x7f4eb40a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.791+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1137190157 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4ebc021020 con 0x7f4ec415e350 2024-03-24T00:34:10.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:10.951+0000 7f4eca400640 1 -- 172.21.15.40:0/1137190157 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4ec40cfe10 con 0x7f4ec415e350 2024-03-24T00:34:11.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.607+0000 7f4ebbe00640 1 -- 172.21.15.40:0/1137190157 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v57) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4ebc005480 con 0x7f4ec415e350 2024-03-24T00:34:11.610 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-03-24T00:34:11.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.611+0000 7f4eca400640 1 -- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ea0067660 msgr2=0x7f4ea0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:11.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.611+0000 7f4eca400640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ea0067660 0x7f4ea0069b20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4eb40af000 tx=0x7f4eb40a7070 comp rx=0 tx=0).stop 2024-03-24T00:34:11.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.611+0000 7f4eca400640 1 -- 172.21.15.40:0/1137190157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec415e350 msgr2=0x7f4ec4154250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.611+0000 7f4eca400640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec415e350 0x7f4ec4154250 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc00c3f0 tx=0x7f4ebc00c8b0 comp rx=0 tx=0).stop 2024-03-24T00:34:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.611+0000 7f4eca400640 1 -- 172.21.15.40:0/1137190157 shutdown_connections 2024-03-24T00:34:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.611+0000 7f4eca400640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4ea0067660 0x7f4ea0069b20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.611+0000 7f4eca400640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ec415e350 0x7f4ec4154250 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.611+0000 7f4eca400640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ec40c9580 0x7f4ec40cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.611+0000 7f4eca400640 1 --2- 172.21.15.40:0/1137190157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ec40ad8f0 0x7f4ec40b4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.611+0000 7f4eca400640 1 -- 172.21.15.40:0/1137190157 >> 172.21.15.40:0/1137190157 conn(0x7f4ec4072870 msgr2=0x7f4ec40ccdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.611+0000 7f4eca400640 1 -- 172.21.15.40:0/1137190157 shutdown_connections 2024-03-24T00:34:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.611+0000 7f4eca400640 1 -- 172.21.15.40:0/1137190157 wait complete. 2024-03-24T00:34:11.656 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:34:11.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.799+0000 7f3361800640 1 Processor -- start 2024-03-24T00:34:11.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.799+0000 7f3361800640 1 -- start start 2024-03-24T00:34:11.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.799+0000 7f3361800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f335c0cf4e0 0x7f335c06dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:11.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.799+0000 7f3361800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f335c072b60 0x7f335c0b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:11.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.799+0000 7f3361800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f335c15f2c0 con 0x7f335c0724a0 2024-03-24T00:34:11.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.799+0000 7f3361800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f335c06cfe0 con 0x7f335c0cf4e0 2024-03-24T00:34:11.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.799+0000 7f3361800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335c06d880 con 0x7f335c072b60 2024-03-24T00:34:11.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.799+0000 7f335aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f335c0cf4e0 0x7f335c06dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:11.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.799+0000 7f335b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f335c0724a0 0x7f335c070ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53712/0 (socket says 172.21.15.40:53712) 2024-03-24T00:34:11.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.799+0000 7f335b400640 1 -- 172.21.15.40:0/3118792357 learned_addr learned my addr 172.21.15.40:0/3118792357 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:11.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.799+0000 7f3352000640 1 --2- 172.21.15.40:0/3118792357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f335c072b60 0x7f335c0b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:11.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f335aa00640 1 -- 172.21.15.40:0/3118792357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f335c072b60 msgr2=0x7f335c0b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:11.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f335aa00640 1 --2- 172.21.15.40:0/3118792357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f335c072b60 0x7f335c0b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:11.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f335aa00640 1 -- 172.21.15.40:0/3118792357 >> v1:172.21.15.40:6789/0 conn(0x7f335c0724a0 legacy=0x7f335c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:11.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f335aa00640 1 -- 172.21.15.40:0/3118792357 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c06d230 con 0x7f335c0cf4e0 2024-03-24T00:34:11.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f335a000640 1 -- 172.21.15.40:0/3118792357 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259749012 0 0) 0x7f335c15f2c0 con 0x7f335c0724a0 2024-03-24T00:34:11.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f335aa00640 1 --2- 172.21.15.40:0/3118792357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f335c0cf4e0 0x7f335c06dd10 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f334c009d90 tx=0x7f334c01f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d78a578218d92c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:11.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f335a000640 1 -- 172.21.15.40:0/3118792357 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c0b4070 con 0x7f335c0cf4e0 2024-03-24T00:34:11.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f335a000640 1 -- 172.21.15.40:0/3118792357 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334c0af600 con 0x7f335c0cf4e0 2024-03-24T00:34:11.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f335a000640 1 -- 172.21.15.40:0/3118792357 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c0af9d0 con 0x7f335c0cf4e0 2024-03-24T00:34:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f3361800640 1 -- 172.21.15.40:0/3118792357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f335c0cf4e0 msgr2=0x7f335c06dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f3361800640 1 --2- 172.21.15.40:0/3118792357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f335c0cf4e0 0x7f335c06dd10 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f334c009d90 tx=0x7f334c01f150 comp rx=0 tx=0).stop 2024-03-24T00:34:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f3361800640 1 -- 172.21.15.40:0/3118792357 shutdown_connections 2024-03-24T00:34:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f3361800640 1 --2- 172.21.15.40:0/3118792357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f335c072b60 0x7f335c0b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f3361800640 1 --2- 172.21.15.40:0/3118792357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f335c0cf4e0 0x7f335c06dd10 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.803+0000 7f3361800640 1 -- 172.21.15.40:0/3118792357 >> 172.21.15.40:0/3118792357 conn(0x7f335c0b1660 msgr2=0x7f335c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f3361800640 1 -- 172.21.15.40:0/3118792357 shutdown_connections 2024-03-24T00:34:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f3361800640 1 -- 172.21.15.40:0/3118792357 wait complete. 2024-03-24T00:34:11.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f3361800640 1 Processor -- start 2024-03-24T00:34:11.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f3361800640 1 -- start start 2024-03-24T00:34:11.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f3361800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f335c0724a0 0x7f335c1e86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:11.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f3361800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f335c072b60 0x7f335c154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:11.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f3361800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0cf4e0 0x7f335c1503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:11.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f3361800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c15ef20 con 0x7f335c0724a0 2024-03-24T00:34:11.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f3361800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335c06e250 con 0x7f335c0cf4e0 2024-03-24T00:34:11.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f3361800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335c15f0a0 con 0x7f335c072b60 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f335aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f335c0724a0 0x7f335c1e86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f335aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f335c0724a0 0x7f335c1e86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50152/0 (socket says 172.21.15.40:50152) 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f335aa00640 1 -- 172.21.15.40:0/3084457097 learned_addr learned my addr 172.21.15.40:0/3084457097 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f3352000640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f335c072b60 0x7f335c154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f335b400640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0cf4e0 0x7f335c1503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f335aa00640 1 -- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f335c072b60 msgr2=0x7f335c154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f335aa00640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f335c072b60 0x7f335c154770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f335aa00640 1 -- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0cf4e0 msgr2=0x7f335c1503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f335aa00640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0cf4e0 0x7f335c1503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f335aa00640 1 -- 172.21.15.40:0/3084457097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c150900 con 0x7f335c0724a0 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f335b400640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0cf4e0 0x7f335c1503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.807+0000 7f335aa00640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f335c0724a0 0x7f335c1e86b0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f334c0b2040 tx=0x7f334c01bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.811+0000 7f3353400640 1 -- 172.21.15.40:0/3084457097 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c0b4040 con 0x7f335c0724a0 2024-03-24T00:34:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.811+0000 7f3353400640 1 -- 172.21.15.40:0/3084457097 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334c0c1070 con 0x7f335c0724a0 2024-03-24T00:34:11.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.811+0000 7f3353400640 1 -- 172.21.15.40:0/3084457097 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c009450 con 0x7f335c0724a0 2024-03-24T00:34:11.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.811+0000 7f3361800640 1 -- 172.21.15.40:0/3084457097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f335c154ee0 con 0x7f335c0724a0 2024-03-24T00:34:11.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.811+0000 7f3361800640 1 -- 172.21.15.40:0/3084457097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f335c1554a0 con 0x7f335c0724a0 2024-03-24T00:34:11.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.811+0000 7f3361800640 1 -- 172.21.15.40:0/3084457097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f335c068170 con 0x7f335c0724a0 2024-03-24T00:34:11.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.811+0000 7f3353400640 1 -- 172.21.15.40:0/3084457097 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f334c0ba670 con 0x7f335c0724a0 2024-03-24T00:34:11.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.811+0000 7f3353400640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3338067660 0x7f3338069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:11.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.811+0000 7f3353400640 1 -- 172.21.15.40:0/3084457097 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f334c124250 con 0x7f335c0724a0 2024-03-24T00:34:11.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.811+0000 7f3352000640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3338067660 0x7f3338069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.815+0000 7f3352000640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3338067660 0x7f3338069b20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f335c160f50 tx=0x7f3344008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:11.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:11.819+0000 7f3353400640 1 -- 172.21.15.40:0/3084457097 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f334c018cb0 con 0x7f335c0724a0 2024-03-24T00:34:12.011 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-24T00:34:12.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.015+0000 7f3361800640 1 -- 172.21.15.40:0/3084457097 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f335c162670 con 0x7f335c0724a0 2024-03-24T00:34:12.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.607+0000 7f3353400640 1 -- 172.21.15.40:0/3084457097 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v48) v1 ==== 172+0+0 (secure 0 0 0) 0x7f334c0f4fb0 con 0x7f335c0724a0 2024-03-24T00:34:12.612 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 8 and data pool 9 2024-03-24T00:34:12.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.611+0000 7f3361800640 1 -- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3338067660 msgr2=0x7f3338069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:12.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.611+0000 7f3361800640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3338067660 0x7f3338069b20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f335c160f50 tx=0x7f3344008040 comp rx=0 tx=0).stop 2024-03-24T00:34:12.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.611+0000 7f3361800640 1 -- 172.21.15.40:0/3084457097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f335c0724a0 msgr2=0x7f335c1e86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:12.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.611+0000 7f3361800640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f335c0724a0 0x7f335c1e86b0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f334c0b2040 tx=0x7f334c01bef0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.611+0000 7f3361800640 1 -- 172.21.15.40:0/3084457097 shutdown_connections 2024-03-24T00:34:12.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.611+0000 7f3361800640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3338067660 0x7f3338069b20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.611+0000 7f3361800640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335c0cf4e0 0x7f335c1503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.611+0000 7f3361800640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f335c072b60 0x7f335c154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.611+0000 7f3361800640 1 --2- 172.21.15.40:0/3084457097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f335c0724a0 0x7f335c1e86b0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.611+0000 7f3361800640 1 -- 172.21.15.40:0/3084457097 >> 172.21.15.40:0/3084457097 conn(0x7f335c0b1660 msgr2=0x7f335c06e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:12.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.615+0000 7f3361800640 1 -- 172.21.15.40:0/3084457097 shutdown_connections 2024-03-24T00:34:12.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.615+0000 7f3361800640 1 -- 172.21.15.40:0/3084457097 wait complete. 2024-03-24T00:34:12.661 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:34:12.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.799+0000 7feb90000640 1 Processor -- start 2024-03-24T00:34:12.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.799+0000 7feb90000640 1 -- start start 2024-03-24T00:34:12.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.799+0000 7feb90000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7feb880b46c0 0x7feb880b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:12.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb90000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7feb880b2ff0 0x7feb880ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:12.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb90000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb88070f60 con 0x7feb8806e620 2024-03-24T00:34:12.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb90000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb880b3400 con 0x7feb880b2ff0 2024-03-24T00:34:12.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb90000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb880b3620 con 0x7feb880b46c0 2024-03-24T00:34:12.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb8d600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7feb8806e620 0x7feb8806ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53720/0 (socket says 172.21.15.40:53720) 2024-03-24T00:34:12.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb8d600640 1 -- 172.21.15.40:0/1533965203 learned_addr learned my addr 172.21.15.40:0/1533965203 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:12.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb8e000640 1 --2- 172.21.15.40:0/1533965203 >> v2:172.21.15.40:3300/0 conn(0x7feb880b2ff0 0x7feb880ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:12.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb8cc00640 1 --2- 172.21.15.40:0/1533965203 >> v2:172.21.15.53:3300/0 conn(0x7feb880b46c0 0x7feb880b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:12.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb87e00640 1 -- 172.21.15.40:0/1533965203 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009145589 0 0) 0x7feb88070f60 con 0x7feb8806e620 2024-03-24T00:34:12.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb87e00640 1 -- 172.21.15.40:0/1533965203 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb70003530 con 0x7feb8806e620 2024-03-24T00:34:12.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb8e000640 1 -- 172.21.15.40:0/1533965203 >> v2:172.21.15.53:3300/0 conn(0x7feb880b46c0 msgr2=0x7feb880b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:12.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb8e000640 1 --2- 172.21.15.40:0/1533965203 >> v2:172.21.15.53:3300/0 conn(0x7feb880b46c0 0x7feb880b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb8e000640 1 -- 172.21.15.40:0/1533965203 >> v1:172.21.15.40:6789/0 conn(0x7feb8806e620 legacy=0x7feb8806ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:12.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb8e000640 1 -- 172.21.15.40:0/1533965203 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb880b3900 con 0x7feb880b2ff0 2024-03-24T00:34:12.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.803+0000 7feb8e000640 1 --2- 172.21.15.40:0/1533965203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb880b2ff0 0x7feb880ccfa0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7feb8806cea0 tx=0x7feb8001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=317b78ca7c814017 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:12.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb87e00640 1 -- 172.21.15.40:0/1533965203 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800b2070 con 0x7feb880b2ff0 2024-03-24T00:34:12.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb87e00640 1 -- 172.21.15.40:0/1533965203 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb800a5930 con 0x7feb880b2ff0 2024-03-24T00:34:12.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb87e00640 1 -- 172.21.15.40:0/1533965203 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb880cdcf0 con 0x7feb880b2ff0 2024-03-24T00:34:12.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 -- 172.21.15.40:0/1533965203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb880b2ff0 msgr2=0x7feb880ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:12.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 --2- 172.21.15.40:0/1533965203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb880b2ff0 0x7feb880ccfa0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7feb8806cea0 tx=0x7feb8001c870 comp rx=0 tx=0).stop 2024-03-24T00:34:12.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 -- 172.21.15.40:0/1533965203 shutdown_connections 2024-03-24T00:34:12.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 --2- 172.21.15.40:0/1533965203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb880b2ff0 0x7feb880ccfa0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 --2- 172.21.15.40:0/1533965203 >> v2:172.21.15.53:3300/0 conn(0x7feb880b46c0 0x7feb880b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 -- 172.21.15.40:0/1533965203 >> 172.21.15.40:0/1533965203 conn(0x7feb880b17b0 msgr2=0x7feb880b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:12.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 -- 172.21.15.40:0/1533965203 shutdown_connections 2024-03-24T00:34:12.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 -- 172.21.15.40:0/1533965203 wait complete. 2024-03-24T00:34:12.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 Processor -- start 2024-03-24T00:34:12.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 -- start start 2024-03-24T00:34:12.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb8806e620 0x7feb88159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:12.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb880b2ff0 0x7feb881412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:12.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb880b46c0 0x7feb88141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:12.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb88144920 con 0x7feb880b46c0 2024-03-24T00:34:12.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb880b4ad0 con 0x7feb880b2ff0 2024-03-24T00:34:12.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb90000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8813afd0 con 0x7feb8806e620 2024-03-24T00:34:12.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb8e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb880b46c0 0x7feb88141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:12.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb8e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb880b46c0 0x7feb88141810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50164/0 (socket says 172.21.15.40:50164) 2024-03-24T00:34:12.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.807+0000 7feb8e000640 1 -- 172.21.15.40:0/1208460499 learned_addr learned my addr 172.21.15.40:0/1208460499 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:12.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb8cc00640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb880b2ff0 0x7feb881412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:12.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb8d600640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb8806e620 0x7feb88159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:12.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb8e000640 1 -- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb8806e620 msgr2=0x7feb88159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:12.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb8e000640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb8806e620 0x7feb88159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb8e000640 1 -- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb880b2ff0 msgr2=0x7feb881412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:12.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb8e000640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb880b2ff0 0x7feb881412d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb8e000640 1 -- 172.21.15.40:0/1208460499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8815a4f0 con 0x7feb880b46c0 2024-03-24T00:34:12.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb8d600640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb8806e620 0x7feb88159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:34:12.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb8e000640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb880b46c0 0x7feb88141810 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7feb800b0040 tx=0x7feb80019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:12.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb85600640 1 -- 172.21.15.40:0/1208460499 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800b2050 con 0x7feb880b46c0 2024-03-24T00:34:12.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb85600640 1 -- 172.21.15.40:0/1208460499 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb800be070 con 0x7feb880b46c0 2024-03-24T00:34:12.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb85600640 1 -- 172.21.15.40:0/1208460499 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800ada70 con 0x7feb880b46c0 2024-03-24T00:34:12.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb90000640 1 -- 172.21.15.40:0/1208460499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8813c310 con 0x7feb880b46c0 2024-03-24T00:34:12.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb90000640 1 -- 172.21.15.40:0/1208460499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8813a6a0 con 0x7feb880b46c0 2024-03-24T00:34:12.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb90000640 1 -- 172.21.15.40:0/1208460499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb88068170 con 0x7feb880b46c0 2024-03-24T00:34:12.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb85600640 1 -- 172.21.15.40:0/1208460499 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7feb800a57b0 con 0x7feb880b46c0 2024-03-24T00:34:12.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.811+0000 7feb85600640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb68067660 0x7feb68069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:12.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.815+0000 7feb85600640 1 -- 172.21.15.40:0/1208460499 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb80120b20 con 0x7feb880b46c0 2024-03-24T00:34:12.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.815+0000 7feb8d600640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb68067660 0x7feb68069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:12.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.815+0000 7feb8d600640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb68067660 0x7feb68069b20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7feb7c000f70 tx=0x7feb7c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:12.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.819+0000 7feb85600640 1 -- 172.21.15.40:0/1208460499 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7feb80016520 con 0x7feb880b46c0 2024-03-24T00:34:12.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.979+0000 7feb90000640 1 -- 172.21.15.40:0/1208460499 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb88072f50 con 0x7feb880b46c0 2024-03-24T00:34:12.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb85600640 1 -- 172.21.15.40:0/1208460499 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16618 (secure 0 0 0) 0x7feb800f18b0 con 0x7feb880b46c0 2024-03-24T00:34:12.985 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:12.985 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":58,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:34:12.018666+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:34:10.428162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-24T00:34:11.001094+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:34:12.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb90000640 1 -- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb68067660 msgr2=0x7feb68069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:12.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb90000640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb68067660 0x7feb68069b20 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7feb7c000f70 tx=0x7feb7c009040 comp rx=0 tx=0).stop 2024-03-24T00:34:12.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb90000640 1 -- 172.21.15.40:0/1208460499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb880b46c0 msgr2=0x7feb88141810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:12.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb90000640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb880b46c0 0x7feb88141810 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7feb800b0040 tx=0x7feb80019610 comp rx=0 tx=0).stop 2024-03-24T00:34:12.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb90000640 1 -- 172.21.15.40:0/1208460499 shutdown_connections 2024-03-24T00:34:12.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb90000640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb68067660 0x7feb68069b20 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb90000640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feb880b46c0 0x7feb88141810 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb90000640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb880b2ff0 0x7feb881412d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb90000640 1 --2- 172.21.15.40:0/1208460499 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feb8806e620 0x7feb88159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:12.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb90000640 1 -- 172.21.15.40:0/1208460499 >> 172.21.15.40:0/1208460499 conn(0x7feb880b17b0 msgr2=0x7feb8806d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:12.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb90000640 1 -- 172.21.15.40:0/1208460499 shutdown_connections 2024-03-24T00:34:12.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:12.983+0000 7feb90000640 1 -- 172.21.15.40:0/1208460499 wait complete. 2024-03-24T00:34:13.006 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:34:13.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.143+0000 7f37ad400640 1 Processor -- start 2024-03-24T00:34:13.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.143+0000 7f37ad400640 1 -- start start 2024-03-24T00:34:13.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.143+0000 7f37ad400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37a806e620 0x7f37a80b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.143+0000 7f37ad400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f37a806ea00 0x7f37a80b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.143+0000 7f37ad400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a8070f60 con 0x7f37a80b2ff0 2024-03-24T00:34:13.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.143+0000 7f37ad400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a80b4c00 con 0x7f37a806ea00 2024-03-24T00:34:13.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.143+0000 7f37ad400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a80b3620 con 0x7f37a806e620 2024-03-24T00:34:13.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a7200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f37a80b2ff0 0x7f37a80ccfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53736/0 (socket says 172.21.15.40:53736) 2024-03-24T00:34:13.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a7200640 1 -- 172.21.15.40:0/3053371680 learned_addr learned my addr 172.21.15.40:0/3053371680 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a6800640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37a806e620 0x7f37a80b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f379fe00640 1 --2- 172.21.15.40:0/3053371680 >> v2:172.21.15.53:3300/0 conn(0x7f37a806ea00 0x7f37a80b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237111075 0 0) 0x7f37a8070f60 con 0x7f37a80b2ff0 2024-03-24T00:34:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f378c003530 con 0x7f37a80b2ff0 2024-03-24T00:34:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2301023859 0 0) 0x7f378c003530 con 0x7f37a80b2ff0 2024-03-24T00:34:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37a806e620 msgr2=0x7f37a80b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37a806e620 0x7f37a80b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 >> v2:172.21.15.53:3300/0 conn(0x7f37a806ea00 msgr2=0x7f37a80b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 --2- 172.21.15.40:0/3053371680 >> v2:172.21.15.53:3300/0 conn(0x7f37a806ea00 0x7f37a80b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a80b3900 con 0x7f37a80b2ff0 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f37a0001a00 con 0x7f37a80b2ff0 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f379fe00640 1 --2- 172.21.15.40:0/3053371680 >> v2:172.21.15.53:3300/0 conn(0x7f37a806ea00 0x7f37a80b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 >> v1:172.21.15.40:6789/0 conn(0x7f37a80b2ff0 legacy=0x7f37a80ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c0039c0 0x7f378c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f378c004810 0x7f378c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37a806e620 0x7f378c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a80b4c00 con 0x7f378c004810 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a80b3620 con 0x7f378c0039c0 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378c00ba20 con 0x7f37a806e620 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f379fe00640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f378c004810 0x7f378c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a6800640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c0039c0 0x7f378c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a7200640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37a806e620 0x7f378c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f379fe00640 1 -- 172.21.15.40:0/3053371680 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37a806e620 msgr2=0x7f378c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f379fe00640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37a806e620 0x7f378c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f379fe00640 1 -- 172.21.15.40:0/3053371680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c0039c0 msgr2=0x7f378c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f379fe00640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c0039c0 0x7f378c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f379fe00640 1 -- 172.21.15.40:0/3053371680 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c003530 con 0x7f378c004810 2024-03-24T00:34:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.147+0000 7f37a6800640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c0039c0 0x7f378c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:13.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f379fe00640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f378c004810 0x7f378c004c90 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3790009ce0 tx=0x7f379001cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e035ee9a68b37ea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:13.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37900b2070 con 0x7f378c004810 2024-03-24T00:34:13.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37900a5630 con 0x7f378c004810 2024-03-24T00:34:13.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f37a5e00640 1 -- 172.21.15.40:0/3053371680 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790009880 con 0x7f378c004810 2024-03-24T00:34:13.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f37ad400640 1 -- 172.21.15.40:0/3053371680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f378c004810 msgr2=0x7f378c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:13.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f37ad400640 1 --2- 172.21.15.40:0/3053371680 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f378c004810 0x7f378c004c90 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3790009ce0 tx=0x7f379001cc70 comp rx=0 tx=0).stop 2024-03-24T00:34:13.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f37a7200640 1 -- 172.21.15.40:0/3053371680 reap_dead start 2024-03-24T00:34:13.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f37ad400640 1 -- 172.21.15.40:0/3053371680 shutdown_connections 2024-03-24T00:34:13.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f37ad400640 1 -- 172.21.15.40:0/3053371680 >> 172.21.15.40:0/3053371680 conn(0x7f37a80b17b0 msgr2=0x7f37a80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:13.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f37ad400640 1 -- 172.21.15.40:0/3053371680 shutdown_connections 2024-03-24T00:34:13.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f37ad400640 1 -- 172.21.15.40:0/3053371680 wait complete. 2024-03-24T00:34:13.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f37ad400640 1 Processor -- start 2024-03-24T00:34:13.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.151+0000 7f37ad400640 1 -- start start 2024-03-24T00:34:13.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f37ad400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c004810 0x7f37a81331a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f37ad400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37a81eb190 0x7f37a8163c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f37ad400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37a81eb610 0x7f37a81641a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f37ad400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a8144920 con 0x7f37a81eb190 2024-03-24T00:34:13.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f37ad400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a806ede0 con 0x7f378c004810 2024-03-24T00:34:13.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f37ad400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a81ebb70 con 0x7f37a81eb610 2024-03-24T00:34:13.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37a81eb190 0x7f37a8163c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37a81eb190 0x7f37a8163c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50172/0 (socket says 172.21.15.40:50172) 2024-03-24T00:34:13.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379fe00640 1 -- 172.21.15.40:0/5899823 learned_addr learned my addr 172.21.15.40:0/5899823 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:13.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379fe00640 1 -- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37a81eb610 msgr2=0x7f37a81641a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:13.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379fe00640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37a81eb610 0x7f37a81641a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379fe00640 1 -- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c004810 msgr2=0x7f37a81331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:13.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f37a6800640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c004810 0x7f37a81331a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379fe00640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c004810 0x7f37a81331a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379fe00640 1 -- 172.21.15.40:0/5899823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a81ebe50 con 0x7f37a81eb190 2024-03-24T00:34:13.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f37a6800640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c004810 0x7f37a81331a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:34:13.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379fe00640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37a81eb190 0x7f37a8163c60 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3790009ce0 tx=0x7f378c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:13.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379ea00640 1 -- 172.21.15.40:0/5899823 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37900bc030 con 0x7f37a81eb190 2024-03-24T00:34:13.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379ea00640 1 -- 172.21.15.40:0/5899823 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37900a5de0 con 0x7f37a81eb190 2024-03-24T00:34:13.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f37ad400640 1 -- 172.21.15.40:0/5899823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a81ec130 con 0x7f37a81eb190 2024-03-24T00:34:13.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379ea00640 1 -- 172.21.15.40:0/5899823 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37900b2040 con 0x7f37a81eb190 2024-03-24T00:34:13.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f37ad400640 1 -- 172.21.15.40:0/5899823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a81ecc10 con 0x7f37a81eb190 2024-03-24T00:34:13.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f37ad400640 1 -- 172.21.15.40:0/5899823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a8068170 con 0x7f37a81eb190 2024-03-24T00:34:13.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.155+0000 7f379ea00640 1 -- 172.21.15.40:0/5899823 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f37900bf070 con 0x7f37a81eb190 2024-03-24T00:34:13.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.159+0000 7f379ea00640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3784067690 0x7f3784069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.159+0000 7f379ea00640 1 -- 172.21.15.40:0/5899823 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f379011f5b0 con 0x7f37a81eb190 2024-03-24T00:34:13.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.159+0000 7f37a6800640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3784067690 0x7f3784069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.159+0000 7f37a6800640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3784067690 0x7f3784069b50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3798009b20 tx=0x7f3798008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:13.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.163+0000 7f379ea00640 1 -- 172.21.15.40:0/5899823 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f37900c7020 con 0x7f37a81eb190 2024-03-24T00:34:13.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.327+0000 7f37ad400640 1 -- 172.21.15.40:0/5899823 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37a8072f50 con 0x7f37a81eb190 2024-03-24T00:34:13.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.327+0000 7f379ea00640 1 -- 172.21.15.40:0/5899823 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+16618 (secure 0 0 0) 0x7f37900169c0 con 0x7f37a81eb190 2024-03-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:13.331 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":58,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:34:12.018666+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:34:10.428162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-24T00:34:11.001094+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.331+0000 7f37ad400640 1 -- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3784067690 msgr2=0x7f3784069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.331+0000 7f37ad400640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3784067690 0x7f3784069b50 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3798009b20 tx=0x7f3798008040 comp rx=0 tx=0).stop 2024-03-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.331+0000 7f37ad400640 1 -- 172.21.15.40:0/5899823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37a81eb190 msgr2=0x7f37a8163c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.331+0000 7f37ad400640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37a81eb190 0x7f37a8163c60 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3790009ce0 tx=0x7f378c005840 comp rx=0 tx=0).stop 2024-03-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.331+0000 7f37ad400640 1 -- 172.21.15.40:0/5899823 shutdown_connections 2024-03-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.331+0000 7f37ad400640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3784067690 0x7f3784069b50 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.331+0000 7f37ad400640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37a81eb610 0x7f37a81641a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.331+0000 7f37ad400640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37a81eb190 0x7f37a8163c60 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.331+0000 7f37ad400640 1 --2- 172.21.15.40:0/5899823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c004810 0x7f37a81331a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.331+0000 7f37ad400640 1 -- 172.21.15.40:0/5899823 >> 172.21.15.40:0/5899823 conn(0x7f37a80b17b0 msgr2=0x7f37a815ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:13.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.331+0000 7f37ad400640 1 -- 172.21.15.40:0/5899823 shutdown_connections 2024-03-24T00:34:13.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.331+0000 7f37ad400640 1 -- 172.21.15.40:0/5899823 wait complete. 2024-03-24T00:34:13.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:34:13.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f569ec00640 1 Processor -- start 2024-03-24T00:34:13.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f569ec00640 1 -- start start 2024-03-24T00:34:13.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f569ec00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f569806cb80 0x7f56980b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f569ec00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56980b4840 0x7f5698071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f569ec00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56980b3260 con 0x7f569815ac60 2024-03-24T00:34:13.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f569ec00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56980b3e70 con 0x7f569806cb80 2024-03-24T00:34:13.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f569ec00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56980b3540 con 0x7f56980b4840 2024-03-24T00:34:13.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5697e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f569815ac60 0x7f56980b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53750/0 (socket says 172.21.15.40:53750) 2024-03-24T00:34:13.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5697e00640 1 -- 172.21.15.40:0/3676502368 learned_addr learned my addr 172.21.15.40:0/3676502368 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:13.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5697400640 1 --2- 172.21.15.40:0/3676502368 >> v2:172.21.15.53:3300/0 conn(0x7f569806cb80 0x7f56980b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f569cc00640 1 --2- 172.21.15.40:0/3676502368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56980b4840 0x7f5698071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643465231 0 0) 0x7f56980b3260 con 0x7f569815ac60 2024-03-24T00:34:13.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f567c003530 con 0x7f569815ac60 2024-03-24T00:34:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2237833724 0 0) 0x7f567c003530 con 0x7f569815ac60 2024-03-24T00:34:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56980b4840 msgr2=0x7f5698071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 --2- 172.21.15.40:0/3676502368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56980b4840 0x7f5698071b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 >> v2:172.21.15.53:3300/0 conn(0x7f569806cb80 msgr2=0x7f56980b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 --2- 172.21.15.40:0/3676502368 >> v2:172.21.15.53:3300/0 conn(0x7f569806cb80 0x7f56980b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56980b3820 con 0x7f569815ac60 2024-03-24T00:34:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f5688001680 con 0x7f569815ac60 2024-03-24T00:34:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5697400640 1 --2- 172.21.15.40:0/3676502368 >> v2:172.21.15.53:3300/0 conn(0x7f569806cb80 0x7f56980b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 >> v1:172.21.15.40:6789/0 conn(0x7f569815ac60 legacy=0x7f56980b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 --2- 172.21.15.40:0/3676502368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56980b4840 0x7f567c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 --2- 172.21.15.40:0/3676502368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f567c003f40 0x7f567c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 --2- 172.21.15.40:0/3676502368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f567c004d50 0x7f567c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56980b3e70 con 0x7f567c003f40 2024-03-24T00:34:13.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56980b3540 con 0x7f567c004d50 2024-03-24T00:34:13.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c00dc30 con 0x7f56980b4840 2024-03-24T00:34:13.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5697400640 1 --2- 172.21.15.40:0/3676502368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f567c003f40 0x7f567c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5697e00640 1 --2- 172.21.15.40:0/3676502368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56980b4840 0x7f567c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f569cc00640 1 --2- 172.21.15.40:0/3676502368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f567c004d50 0x7f567c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5697400640 1 -- 172.21.15.40:0/3676502368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56980b4840 msgr2=0x7f567c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:13.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5697400640 1 --2- 172.21.15.40:0/3676502368 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56980b4840 0x7f567c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5697400640 1 -- 172.21.15.40:0/3676502368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f567c004d50 msgr2=0x7f567c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:13.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5697400640 1 --2- 172.21.15.40:0/3676502368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f567c004d50 0x7f567c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.495+0000 7f5697400640 1 -- 172.21.15.40:0/3676502368 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c003530 con 0x7f567c003f40 2024-03-24T00:34:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.499+0000 7f5697400640 1 --2- 172.21.15.40:0/3676502368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f567c003f40 0x7f567c004810 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f5684009cb0 tx=0x7f568401cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bee277331094941 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.499+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56840b2070 con 0x7f567c003f40 2024-03-24T00:34:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.499+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56840a5890 con 0x7f567c003f40 2024-03-24T00:34:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.499+0000 7f5696a00640 1 -- 172.21.15.40:0/3676502368 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684009880 con 0x7f567c003f40 2024-03-24T00:34:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.499+0000 7f569ec00640 1 -- 172.21.15.40:0/3676502368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f567c003f40 msgr2=0x7f567c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.499+0000 7f569ec00640 1 --2- 172.21.15.40:0/3676502368 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f567c003f40 0x7f567c004810 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f5684009cb0 tx=0x7f568401cc50 comp rx=0 tx=0).stop 2024-03-24T00:34:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.499+0000 7f569cc00640 1 -- 172.21.15.40:0/3676502368 reap_dead start 2024-03-24T00:34:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.499+0000 7f569ec00640 1 -- 172.21.15.40:0/3676502368 shutdown_connections 2024-03-24T00:34:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.499+0000 7f569ec00640 1 -- 172.21.15.40:0/3676502368 >> 172.21.15.40:0/3676502368 conn(0x7f56980b1660 msgr2=0x7f56980b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:13.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.499+0000 7f569ec00640 1 -- 172.21.15.40:0/3676502368 shutdown_connections 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.499+0000 7f569ec00640 1 -- 172.21.15.40:0/3676502368 wait complete. 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569ec00640 1 Processor -- start 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569ec00640 1 -- start start 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569ec00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f567c003f40 0x7f56980ce620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569ec00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56981f79f0 0x7f56981f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569ec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56981f70e0 0x7f569814f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569ec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56980cd650 con 0x7f56981f70e0 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569ec00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56980cd4d0 con 0x7f567c003f40 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569ec00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56980cd7d0 con 0x7f56981f79f0 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56981f70e0 0x7f569814f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56981f70e0 0x7f569814f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50186/0 (socket says 172.21.15.40:50186) 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569cc00640 1 -- 172.21.15.40:0/3747290490 learned_addr learned my addr 172.21.15.40:0/3747290490 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f5697e00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f567c003f40 0x7f56980ce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f5697400640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56981f79f0 0x7f56981f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569cc00640 1 -- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56981f79f0 msgr2=0x7f56981f3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569cc00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56981f79f0 0x7f56981f3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569cc00640 1 -- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f567c003f40 msgr2=0x7f56980ce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569cc00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f567c003f40 0x7f56980ce620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569cc00640 1 -- 172.21.15.40:0/3747290490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569814d450 con 0x7f56981f70e0 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f5697400640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56981f79f0 0x7f56981f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569cc00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56981f70e0 0x7f569814f540 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5698072ba0 tx=0x7f569000f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f568fe00640 1 -- 172.21.15.40:0/3747290490 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569000fce0 con 0x7f56981f70e0 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f568fe00640 1 -- 172.21.15.40:0/3747290490 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569001d070 con 0x7f56981f70e0 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f568fe00640 1 -- 172.21.15.40:0/3747290490 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5690015670 con 0x7f56981f70e0 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569ec00640 1 -- 172.21.15.40:0/3747290490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569814d730 con 0x7f56981f70e0 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.503+0000 7f569ec00640 1 -- 172.21.15.40:0/3747290490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569814dcf0 con 0x7f56981f70e0 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.507+0000 7f569ec00640 1 -- 172.21.15.40:0/3747290490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5698068170 con 0x7f56981f70e0 2024-03-24T00:34:13.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.507+0000 7f568fe00640 1 -- 172.21.15.40:0/3747290490 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5690015810 con 0x7f56981f70e0 2024-03-24T00:34:13.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.507+0000 7f568fe00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56780676e0 0x7f5678069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:13.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.507+0000 7f568fe00640 1 -- 172.21.15.40:0/3747290490 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5690085f90 con 0x7f56981f70e0 2024-03-24T00:34:13.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.507+0000 7f5697e00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56780676e0 0x7f5678069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:13.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.507+0000 7f5697e00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56780676e0 0x7f5678069ba0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5688000c10 tx=0x7f5688007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:13.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.511+0000 7f568fe00640 1 -- 172.21.15.40:0/3747290490 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5690010040 con 0x7f56981f70e0 2024-03-24T00:34:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:13.711+0000 7f569ec00640 1 -- 172.21.15.40:0/3747290490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5698071c30 con 0x7f56981f70e0 2024-03-24T00:34:14.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.635+0000 7f568fe00640 1 -- 172.21.15.40:0/3747290490 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v51) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5690056cf0 con 0x7f56981f70e0 2024-03-24T00:34:14.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.639+0000 7f569ec00640 1 -- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56780676e0 msgr2=0x7f5678069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:14.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.639+0000 7f569ec00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56780676e0 0x7f5678069ba0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5688000c10 tx=0x7f5688007900 comp rx=0 tx=0).stop 2024-03-24T00:34:14.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.639+0000 7f569ec00640 1 -- 172.21.15.40:0/3747290490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56981f70e0 msgr2=0x7f569814f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:14.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.639+0000 7f569ec00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56981f70e0 0x7f569814f540 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f5698072ba0 tx=0x7f569000f1b0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.639+0000 7f569ec00640 1 -- 172.21.15.40:0/3747290490 shutdown_connections 2024-03-24T00:34:14.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.639+0000 7f569ec00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56780676e0 0x7f5678069ba0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.639+0000 7f569ec00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56981f70e0 0x7f569814f540 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.639+0000 7f569ec00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56981f79f0 0x7f56981f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.639+0000 7f569ec00640 1 --2- 172.21.15.40:0/3747290490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f567c003f40 0x7f56980ce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.639+0000 7f569ec00640 1 -- 172.21.15.40:0/3747290490 >> 172.21.15.40:0/3747290490 conn(0x7f56980b1660 msgr2=0x7f56980aee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:14.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.639+0000 7f569ec00640 1 -- 172.21.15.40:0/3747290490 shutdown_connections 2024-03-24T00:34:14.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.639+0000 7f569ec00640 1 -- 172.21.15.40:0/3747290490 wait complete. 2024-03-24T00:34:14.665 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:34:14.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:34:14.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e31400640 1 Processor -- start 2024-03-24T00:34:14.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e31400640 1 -- start start 2024-03-24T00:34:14.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e31400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2e2c15c0e0 0x7f2e2c074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:14.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e31400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e2c0afa60 0x7f2e2c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:14.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e31400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2e2c073360 0x7f2e2c0aea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:14.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e31400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2c0cdb80 con 0x7f2e2c15c0e0 2024-03-24T00:34:14.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e31400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2c0b2df0 con 0x7f2e2c073360 2024-03-24T00:34:14.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e31400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0cd8a0 con 0x7f2e2c0afa60 2024-03-24T00:34:14.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e2a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2e2c15c0e0 0x7f2e2c074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:14.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e2a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2e2c15c0e0 0x7f2e2c074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50190/0 (socket says 172.21.15.40:50190) 2024-03-24T00:34:14.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e2a800640 1 -- 172.21.15.40:0/2807042397 learned_addr learned my addr 172.21.15.40:0/2807042397 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:14.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e2b200640 1 --2- 172.21.15.40:0/2807042397 >> v2:172.21.15.53:3300/0 conn(0x7f2e2c073360 0x7f2e2c0aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:14.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e21e00640 1 --2- 172.21.15.40:0/2807042397 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e2c0afa60 0x7f2e2c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:14.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e2a800640 1 -- 172.21.15.40:0/2807042397 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e2c0afa60 msgr2=0x7f2e2c0ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:14.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e2a800640 1 --2- 172.21.15.40:0/2807042397 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e2c0afa60 0x7f2e2c0ae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e2a800640 1 -- 172.21.15.40:0/2807042397 >> v2:172.21.15.53:3300/0 conn(0x7f2e2c073360 msgr2=0x7f2e2c0aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:14.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e2a800640 1 --2- 172.21.15.40:0/2807042397 >> v2:172.21.15.53:3300/0 conn(0x7f2e2c073360 0x7f2e2c0aea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.811+0000 7f2e2a800640 1 -- 172.21.15.40:0/2807042397 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c0add20 con 0x7f2e2c15c0e0 2024-03-24T00:34:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e2a800640 1 --2- 172.21.15.40:0/2807042397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2c15c0e0 0x7f2e2c074420 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c072760 tx=0x7f2e1c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b0af6244d53292 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:14.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e29e00640 1 -- 172.21.15.40:0/2807042397 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c0b0070 con 0x7f2e2c15c0e0 2024-03-24T00:34:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e29e00640 1 -- 172.21.15.40:0/2807042397 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1c0ab600 con 0x7f2e2c15c0e0 2024-03-24T00:34:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e29e00640 1 -- 172.21.15.40:0/2807042397 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c0ab9d0 con 0x7f2e2c15c0e0 2024-03-24T00:34:14.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e31400640 1 -- 172.21.15.40:0/2807042397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2c15c0e0 msgr2=0x7f2e2c074420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e31400640 1 --2- 172.21.15.40:0/2807042397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2c15c0e0 0x7f2e2c074420 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c072760 tx=0x7f2e1c01c3d0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e31400640 1 -- 172.21.15.40:0/2807042397 shutdown_connections 2024-03-24T00:34:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e31400640 1 --2- 172.21.15.40:0/2807042397 >> v2:172.21.15.53:3300/0 conn(0x7f2e2c073360 0x7f2e2c0aea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e31400640 1 --2- 172.21.15.40:0/2807042397 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e2c0afa60 0x7f2e2c0ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e31400640 1 --2- 172.21.15.40:0/2807042397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2c15c0e0 0x7f2e2c074420 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e31400640 1 -- 172.21.15.40:0/2807042397 >> 172.21.15.40:0/2807042397 conn(0x7f2e2c072870 msgr2=0x7f2e2c072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e31400640 1 -- 172.21.15.40:0/2807042397 shutdown_connections 2024-03-24T00:34:14.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.815+0000 7f2e31400640 1 -- 172.21.15.40:0/2807042397 wait complete. 2024-03-24T00:34:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e31400640 1 Processor -- start 2024-03-24T00:34:14.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e31400640 1 -- start start 2024-03-24T00:34:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e31400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2c073360 0x7f2e2c13f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:14.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e31400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e2c0afa60 0x7f2e2c0b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e31400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c15c0e0 0x7f2e2c15f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e31400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0cc540 con 0x7f2e2c073360 2024-03-24T00:34:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e31400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0cc3c0 con 0x7f2e2c15c0e0 2024-03-24T00:34:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e31400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0cc6c0 con 0x7f2e2c0afa60 2024-03-24T00:34:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e2a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2c073360 0x7f2e2c13f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e2a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2c073360 0x7f2e2c13f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50202/0 (socket says 172.21.15.40:50202) 2024-03-24T00:34:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e2a800640 1 -- 172.21.15.40:0/1049356781 learned_addr learned my addr 172.21.15.40:0/1049356781 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e2b200640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c15c0e0 0x7f2e2c15f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e21e00640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e2c0afa60 0x7f2e2c0b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e2a800640 1 -- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e2c0afa60 msgr2=0x7f2e2c0b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:14.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e2a800640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e2c0afa60 0x7f2e2c0b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e2a800640 1 -- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c15c0e0 msgr2=0x7f2e2c15f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e2a800640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c15c0e0 0x7f2e2c15f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e2a800640 1 -- 172.21.15.40:0/1049356781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c0b5e00 con 0x7f2e2c073360 2024-03-24T00:34:14.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e2a800640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2c073360 0x7f2e2c13f5b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0ae040 tx=0x7f2e1c019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e23400640 1 -- 172.21.15.40:0/1049356781 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c0b0040 con 0x7f2e2c073360 2024-03-24T00:34:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e23400640 1 -- 172.21.15.40:0/1049356781 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1c0b9070 con 0x7f2e2c073360 2024-03-24T00:34:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e23400640 1 -- 172.21.15.40:0/1049356781 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c0b48a0 con 0x7f2e2c073360 2024-03-24T00:34:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e31400640 1 -- 172.21.15.40:0/1049356781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2c15e270 con 0x7f2e2c073360 2024-03-24T00:34:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e31400640 1 -- 172.21.15.40:0/1049356781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c15e830 con 0x7f2e2c073360 2024-03-24T00:34:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.819+0000 7f2e31400640 1 -- 172.21.15.40:0/1049356781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2c068170 con 0x7f2e2c073360 2024-03-24T00:34:14.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.823+0000 7f2e23400640 1 -- 172.21.15.40:0/1049356781 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2e1c0ab4c0 con 0x7f2e2c073360 2024-03-24T00:34:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.823+0000 7f2e23400640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2e08067660 0x7f2e08069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.823+0000 7f2e23400640 1 -- 172.21.15.40:0/1049356781 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e1c11f600 con 0x7f2e2c073360 2024-03-24T00:34:14.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.823+0000 7f2e21e00640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2e08067660 0x7f2e08069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:14.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.823+0000 7f2e21e00640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2e08067660 0x7f2e08069b20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0cb640 tx=0x7f2e14009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:14.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:14.827+0000 7f2e23400640 1 -- 172.21.15.40:0/1049356781 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2e1c016110 con 0x7f2e2c073360 2024-03-24T00:34:15.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.027+0000 7f2e31400640 1 -- 172.21.15.40:0/1049356781 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2e2c0af6e0 con 0x7f2e2c073360 2024-03-24T00:34:15.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e23400640 1 -- 172.21.15.40:0/1049356781 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2e1c0efb90 con 0x7f2e2c073360 2024-03-24T00:34:15.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e31400640 1 -- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2e08067660 msgr2=0x7f2e08069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:15.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e31400640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2e08067660 0x7f2e08069b20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c0cb640 tx=0x7f2e14009190 comp rx=0 tx=0).stop 2024-03-24T00:34:15.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e31400640 1 -- 172.21.15.40:0/1049356781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2c073360 msgr2=0x7f2e2c13f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:15.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e31400640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2c073360 0x7f2e2c13f5b0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0ae040 tx=0x7f2e1c019170 comp rx=0 tx=0).stop 2024-03-24T00:34:15.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e31400640 1 -- 172.21.15.40:0/1049356781 shutdown_connections 2024-03-24T00:34:15.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e31400640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2e08067660 0x7f2e08069b20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:15.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e31400640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c15c0e0 0x7f2e2c15f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:15.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e31400640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e2c0afa60 0x7f2e2c0b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:15.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e31400640 1 --2- 172.21.15.40:0/1049356781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2e2c073360 0x7f2e2c13f5b0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:15.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e31400640 1 -- 172.21.15.40:0/1049356781 >> 172.21.15.40:0/1049356781 conn(0x7f2e2c072870 msgr2=0x7f2e2c06db30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:15.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e31400640 1 -- 172.21.15.40:0/1049356781 shutdown_connections 2024-03-24T00:34:15.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.643+0000 7f2e31400640 1 -- 172.21.15.40:0/1049356781 wait complete. 2024-03-24T00:34:15.672 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:34:15.672 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:34:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcabe200640 1 Processor -- start 2024-03-24T00:34:15.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcabe200640 1 -- start start 2024-03-24T00:34:15.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcabe200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcab815f600 0x7fcab80b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:15.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcabe200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcab80d2830 0x7fcab80aea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:15.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcabe200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcab80aefa0 0x7fcab80af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:15.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcabe200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab80b3630 con 0x7fcab80aefa0 2024-03-24T00:34:15.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcabe200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab80b2740 con 0x7fcab80d2830 2024-03-24T00:34:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcabe200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab80d2c70 con 0x7fcab815f600 2024-03-24T00:34:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcab7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcab80aefa0 0x7fcab80af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcaafe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcab80d2830 0x7fcab80aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcab7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcab80aefa0 0x7fcab80af3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50204/0 (socket says 172.21.15.40:50204) 2024-03-24T00:34:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcab7e00640 1 -- 172.21.15.40:0/3765258794 learned_addr learned my addr 172.21.15.40:0/3765258794 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcab7400640 1 --2- 172.21.15.40:0/3765258794 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcab815f600 0x7fcab80b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcab7e00640 1 -- 172.21.15.40:0/3765258794 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcab815f600 msgr2=0x7fcab80b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcab7e00640 1 --2- 172.21.15.40:0/3765258794 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcab815f600 0x7fcab80b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcab7e00640 1 -- 172.21.15.40:0/3765258794 >> v2:172.21.15.53:3300/0 conn(0x7fcab80d2830 msgr2=0x7fcab80aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcab7e00640 1 --2- 172.21.15.40:0/3765258794 >> v2:172.21.15.53:3300/0 conn(0x7fcab80d2830 0x7fcab80aea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcab7e00640 1 -- 172.21.15.40:0/3765258794 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab80b1e50 con 0x7fcab80aefa0 2024-03-24T00:34:15.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.815+0000 7fcaafe00640 1 --2- 172.21.15.40:0/3765258794 >> v2:172.21.15.53:3300/0 conn(0x7fcab80d2830 0x7fcab80aea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:15.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcab7e00640 1 --2- 172.21.15.40:0/3765258794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcab80aefa0 0x7fcab80af3d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcab80723f0 tx=0x7fcab001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a7bc0465b66fca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:15.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcab6a00640 1 -- 172.21.15.40:0/3765258794 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab00b4070 con 0x7fcab80aefa0 2024-03-24T00:34:15.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcab6a00640 1 -- 172.21.15.40:0/3765258794 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcab00af400 con 0x7fcab80aefa0 2024-03-24T00:34:15.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcab6a00640 1 -- 172.21.15.40:0/3765258794 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab00af7d0 con 0x7fcab80aefa0 2024-03-24T00:34:15.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcabe200640 1 -- 172.21.15.40:0/3765258794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcab80aefa0 msgr2=0x7fcab80af3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:15.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcabe200640 1 --2- 172.21.15.40:0/3765258794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcab80aefa0 0x7fcab80af3d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcab80723f0 tx=0x7fcab001f1e0 comp rx=0 tx=0).stop 2024-03-24T00:34:15.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcabe200640 1 -- 172.21.15.40:0/3765258794 shutdown_connections 2024-03-24T00:34:15.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcabe200640 1 --2- 172.21.15.40:0/3765258794 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcab80aefa0 0x7fcab80af3d0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:15.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcabe200640 1 --2- 172.21.15.40:0/3765258794 >> v2:172.21.15.53:3300/0 conn(0x7fcab80d2830 0x7fcab80aea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:15.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcabe200640 1 --2- 172.21.15.40:0/3765258794 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcab815f600 0x7fcab80b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:15.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcabe200640 1 -- 172.21.15.40:0/3765258794 >> 172.21.15.40:0/3765258794 conn(0x7fcab80d1d10 msgr2=0x7fcab80d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:15.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcabe200640 1 -- 172.21.15.40:0/3765258794 shutdown_connections 2024-03-24T00:34:15.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.819+0000 7fcabe200640 1 -- 172.21.15.40:0/3765258794 wait complete. 2024-03-24T00:34:15.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcabe200640 1 Processor -- start 2024-03-24T00:34:15.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcabe200640 1 -- start start 2024-03-24T00:34:15.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcabe200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcab80aefa0 0x7fcab815a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:15.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcabe200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcab80d2830 0x7fcab80b7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:15.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcabe200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcab815f600 0x7fcab80cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:15.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcabe200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab80b2fa0 con 0x7fcab80d2830 2024-03-24T00:34:15.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcabe200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab80b23d0 con 0x7fcab80aefa0 2024-03-24T00:34:15.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcabe200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab815f9e0 con 0x7fcab815f600 2024-03-24T00:34:15.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcaafe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcab80d2830 0x7fcab80b7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:15.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcab7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcab80aefa0 0x7fcab815a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:15.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcaafe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcab80d2830 0x7fcab80b7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50210/0 (socket says 172.21.15.40:50210) 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcaafe00640 1 -- 172.21.15.40:0/1882041148 learned_addr learned my addr 172.21.15.40:0/1882041148 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcab7e00640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcab815f600 0x7fcab80cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcab7400640 1 -- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcab815f600 msgr2=0x7fcab80cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcab7400640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcab815f600 0x7fcab80cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcab7400640 1 -- 172.21.15.40:0/1882041148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcab80d2830 msgr2=0x7fcab80b7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcab7400640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcab80d2830 0x7fcab80b7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcab7400640 1 -- 172.21.15.40:0/1882041148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab80c6510 con 0x7fcab80aefa0 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcaafe00640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcab80d2830 0x7fcab80b7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcab7e00640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcab815f600 0x7fcab80cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcab7400640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcab80aefa0 0x7fcab815a290 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fcaa800e870 tx=0x7fcaa800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcaaf400640 1 -- 172.21.15.40:0/1882041148 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa8017070 con 0x7fcab80aefa0 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcaaf400640 1 -- 172.21.15.40:0/1882041148 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa800c940 con 0x7fcab80aefa0 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcaaf400640 1 -- 172.21.15.40:0/1882041148 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa800ccf0 con 0x7fcab80aefa0 2024-03-24T00:34:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcabe200640 1 -- 172.21.15.40:0/1882041148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab80c5f50 con 0x7fcab80aefa0 2024-03-24T00:34:15.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.823+0000 7fcabe200640 1 -- 172.21.15.40:0/1882041148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab80c8630 con 0x7fcab80aefa0 2024-03-24T00:34:15.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.827+0000 7fcabe200640 1 -- 172.21.15.40:0/1882041148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab8068170 con 0x7fcab80aefa0 2024-03-24T00:34:15.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.827+0000 7fcaaf400640 1 -- 172.21.15.40:0/1882041148 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcaa801a070 con 0x7fcab80aefa0 2024-03-24T00:34:15.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.827+0000 7fcaaf400640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca94067660 0x7fca94069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:15.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.827+0000 7fcaaf400640 1 -- 172.21.15.40:0/1882041148 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcaa8085790 con 0x7fcab80aefa0 2024-03-24T00:34:15.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.827+0000 7fcaafe00640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca94067660 0x7fca94069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:15.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.827+0000 7fcaafe00640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca94067660 0x7fca94069b20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0009580 tx=0x7fcaa0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:15.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:15.831+0000 7fcaaf400640 1 -- 172.21.15.40:0/1882041148 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcaa8055db0 con 0x7fcab80aefa0 2024-03-24T00:34:16.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 -- 172.21.15.40:0/1882041148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcab80ca0a0 con 0x7fcab80aefa0 2024-03-24T00:34:16.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcaaf400640 1 -- 172.21.15.40:0/1882041148 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4974 (secure 0 0 0) 0x7fcaa8059e50 con 0x7fcab80aefa0 2024-03-24T00:34:16.029 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:16.029 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":52,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6277,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1633817617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1633817617},{"type":"v1","addr":"172.21.15.53:6837","nonce":1633817617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6468,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1278515461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1278515461},{"type":"v1","addr":"172.21.15.40:6834","nonce":1278515461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":6471,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1312292485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1312292485},{"type":"v1","addr":"172.21.15.40:6835","nonce":1312292485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:34:12.019118+0000","modified":"2024-03-24T00:34:15.031104+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6474},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6474":{"gid":6474,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/501079401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":501079401},{"type":"v1","addr":"172.21.15.53:6835","nonce":501079401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6474,"qdb_cluster":[6474]},"id":4}]} 2024-03-24T00:34:16.029 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 52 2024-03-24T00:34:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 -- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca94067660 msgr2=0x7fca94069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca94067660 0x7fca94069b20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0009580 tx=0x7fcaa0009190 comp rx=0 tx=0).stop 2024-03-24T00:34:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 -- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcab80aefa0 msgr2=0x7fcab815a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcab80aefa0 0x7fcab815a290 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fcaa800e870 tx=0x7fcaa800ed30 comp rx=0 tx=0).stop 2024-03-24T00:34:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 -- 172.21.15.40:0/1882041148 shutdown_connections 2024-03-24T00:34:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca94067660 0x7fca94069b20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcab815f600 0x7fcab80cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcab80d2830 0x7fcab80b7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 --2- 172.21.15.40:0/1882041148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcab80aefa0 0x7fcab815a290 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 -- 172.21.15.40:0/1882041148 >> 172.21.15.40:0/1882041148 conn(0x7fcab80d1d10 msgr2=0x7fcab815afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:16.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 -- 172.21.15.40:0/1882041148 shutdown_connections 2024-03-24T00:34:16.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.027+0000 7fcabe200640 1 -- 172.21.15.40:0/1882041148 wait complete. 2024-03-24T00:34:16.050 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:34:16.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dca400640 1 Processor -- start 2024-03-24T00:34:16.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dca400640 1 -- start start 2024-03-24T00:34:16.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dca400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0dc4072310 0x7f0dc40ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dca400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc4070e00 con 0x7f0dc4141bc0 2024-03-24T00:34:16.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dca400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc40cee20 con 0x7f0dc413ec80 2024-03-24T00:34:16.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dca400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc40cf100 con 0x7f0dc4072310 2024-03-24T00:34:16.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dc2a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0dc4141bc0 0x7f0dc40ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53760/0 (socket says 172.21.15.40:53760) 2024-03-24T00:34:16.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dc3400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0dc413ec80 0x7f0dc41414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:33748/0 (socket says 172.21.15.40:33748) 2024-03-24T00:34:16.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dc2a00640 1 -- 172.21.15.40:0/143115467 learned_addr learned my addr 172.21.15.40:0/143115467 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:16.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dc3e00640 1 --2- 172.21.15.40:0/143115467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0dc4072310 0x7f0dc40ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dc2000640 1 -- 172.21.15.40:0/143115467 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707291136 0 0) 0x7f0dc40cee20 con 0x7f0dc413ec80 2024-03-24T00:34:16.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dc2000640 1 -- 172.21.15.40:0/143115467 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da8003530 con 0x7f0dc413ec80 2024-03-24T00:34:16.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dc2000640 1 -- 172.21.15.40:0/143115467 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267009904 0 0) 0x7f0dc4070e00 con 0x7f0dc4141bc0 2024-03-24T00:34:16.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dc2000640 1 -- 172.21.15.40:0/143115467 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc40cee20 con 0x7f0dc4141bc0 2024-03-24T00:34:16.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.187+0000 7f0dc3e00640 1 -- 172.21.15.40:0/143115467 >> v1:172.21.15.53:6789/0 conn(0x7f0dc413ec80 legacy=0x7f0dc41414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dc3e00640 1 -- 172.21.15.40:0/143115467 >> v1:172.21.15.40:6789/0 conn(0x7f0dc4141bc0 legacy=0x7f0dc40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dc3e00640 1 -- 172.21.15.40:0/143115467 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc40cf3e0 con 0x7f0dc4072310 2024-03-24T00:34:16.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dc3e00640 1 --2- 172.21.15.40:0/143115467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0dc4072310 0x7f0dc40ce800 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009dc0 tx=0x7f0dbc01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aee021923dfa2ac5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:16.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dc2000640 1 -- 172.21.15.40:0/143115467 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0b2070 con 0x7f0dc4072310 2024-03-24T00:34:16.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dc2000640 1 -- 172.21.15.40:0/143115467 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc0a5a70 con 0x7f0dc4072310 2024-03-24T00:34:16.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dc2000640 1 -- 172.21.15.40:0/143115467 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0a5e40 con 0x7f0dc4072310 2024-03-24T00:34:16.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dca400640 1 -- 172.21.15.40:0/143115467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0dc4072310 msgr2=0x7f0dc40ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dca400640 1 --2- 172.21.15.40:0/143115467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0dc4072310 0x7f0dc40ce800 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009dc0 tx=0x7f0dbc01c990 comp rx=0 tx=0).stop 2024-03-24T00:34:16.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dca400640 1 -- 172.21.15.40:0/143115467 shutdown_connections 2024-03-24T00:34:16.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dca400640 1 --2- 172.21.15.40:0/143115467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0dc4072310 0x7f0dc40ce800 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dca400640 1 -- 172.21.15.40:0/143115467 >> 172.21.15.40:0/143115467 conn(0x7f0dc40b17b0 msgr2=0x7f0dc40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:16.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dca400640 1 -- 172.21.15.40:0/143115467 shutdown_connections 2024-03-24T00:34:16.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.191+0000 7f0dca400640 1 -- 172.21.15.40:0/143115467 wait complete. 2024-03-24T00:34:16.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dca400640 1 Processor -- start 2024-03-24T00:34:16.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dca400640 1 -- start start 2024-03-24T00:34:16.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dca400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dc4072310 0x7f0dc413d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dca400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0dc413ec80 0x7f0dc4129480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dca400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0dc4141bc0 0x7f0dc414edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dca400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc4169f00 con 0x7f0dc413ec80 2024-03-24T00:34:16.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dca400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc4169d80 con 0x7f0dc4072310 2024-03-24T00:34:16.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dca400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc416a080 con 0x7f0dc4141bc0 2024-03-24T00:34:16.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dc2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0dc413ec80 0x7f0dc4129480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dc3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dc4072310 0x7f0dc413d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dc2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0dc413ec80 0x7f0dc4129480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50218/0 (socket says 172.21.15.40:50218) 2024-03-24T00:34:16.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dc2a00640 1 -- 172.21.15.40:0/3716775498 learned_addr learned my addr 172.21.15.40:0/3716775498 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:16.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dc3e00640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0dc4141bc0 0x7f0dc414edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dc3400640 1 -- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0dc4141bc0 msgr2=0x7f0dc414edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dc3400640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0dc4141bc0 0x7f0dc414edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dc3400640 1 -- 172.21.15.40:0/3716775498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0dc413ec80 msgr2=0x7f0dc4129480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dc3400640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0dc413ec80 0x7f0dc4129480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dc3400640 1 -- 172.21.15.40:0/3716775498 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc4129ed0 con 0x7f0dc4072310 2024-03-24T00:34:16.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dc3400640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dc4072310 0x7f0dc413d680 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0db400c400 tx=0x7f0db400c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:16.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dbb400640 1 -- 172.21.15.40:0/3716775498 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db40074e0 con 0x7f0dc4072310 2024-03-24T00:34:16.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dbb400640 1 -- 172.21.15.40:0/3716775498 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db40047f0 con 0x7f0dc4072310 2024-03-24T00:34:16.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.195+0000 7f0dbb400640 1 -- 172.21.15.40:0/3716775498 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4004be0 con 0x7f0dc4072310 2024-03-24T00:34:16.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.199+0000 7f0dca400640 1 -- 172.21.15.40:0/3716775498 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc412a1b0 con 0x7f0dc4072310 2024-03-24T00:34:16.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.199+0000 7f0dca400640 1 -- 172.21.15.40:0/3716775498 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc412a770 con 0x7f0dc4072310 2024-03-24T00:34:16.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.199+0000 7f0dca400640 1 -- 172.21.15.40:0/3716775498 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc4068170 con 0x7f0dc4072310 2024-03-24T00:34:16.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.199+0000 7f0dbb400640 1 -- 172.21.15.40:0/3716775498 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f0db401a020 con 0x7f0dc4072310 2024-03-24T00:34:16.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.199+0000 7f0dbb400640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0da4067660 0x7f0da4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.199+0000 7f0dbb400640 1 -- 172.21.15.40:0/3716775498 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0db4084ff0 con 0x7f0dc4072310 2024-03-24T00:34:16.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.199+0000 7f0dc2a00640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0da4067660 0x7f0da4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.199+0000 7f0dc2a00640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0da4067660 0x7f0da4069b20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0dc415ebd0 tx=0x7f0db0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:16.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.203+0000 7f0dbb400640 1 -- 172.21.15.40:0/3716775498 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0db40556d0 con 0x7f0dc4072310 2024-03-24T00:34:16.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.367+0000 7f0dca400640 1 -- 172.21.15.40:0/3716775498 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dc414e890 con 0x7f0dc4072310 2024-03-24T00:34:16.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.367+0000 7f0dbb400640 1 -- 172.21.15.40:0/3716775498 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f0db4059770 con 0x7f0dc4072310 2024-03-24T00:34:16.371 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:16.371 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":59,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:34:13.623804+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:34:10.428162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-24T00:34:11.001094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:34:16.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.371+0000 7f0dca400640 1 -- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0da4067660 msgr2=0x7f0da4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.371+0000 7f0dca400640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0da4067660 0x7f0da4069b20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f0dc415ebd0 tx=0x7f0db0009040 comp rx=0 tx=0).stop 2024-03-24T00:34:16.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.371+0000 7f0dca400640 1 -- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dc4072310 msgr2=0x7f0dc413d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.371+0000 7f0dca400640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dc4072310 0x7f0dc413d680 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0db400c400 tx=0x7f0db400c8c0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.371+0000 7f0dca400640 1 -- 172.21.15.40:0/3716775498 shutdown_connections 2024-03-24T00:34:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.371+0000 7f0dca400640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0da4067660 0x7f0da4069b20 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.371+0000 7f0dca400640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0dc4141bc0 0x7f0dc414edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.371+0000 7f0dca400640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0dc413ec80 0x7f0dc4129480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.371+0000 7f0dca400640 1 --2- 172.21.15.40:0/3716775498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dc4072310 0x7f0dc413d680 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.371+0000 7f0dca400640 1 -- 172.21.15.40:0/3716775498 >> 172.21.15.40:0/3716775498 conn(0x7f0dc40b17b0 msgr2=0x7f0dc40b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.371+0000 7f0dca400640 1 -- 172.21.15.40:0/3716775498 shutdown_connections 2024-03-24T00:34:16.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.371+0000 7f0dca400640 1 -- 172.21.15.40:0/3716775498 wait complete. 2024-03-24T00:34:16.394 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:34:16.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:34:16.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.531+0000 7f38aea00640 1 Processor -- start 2024-03-24T00:34:16.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.531+0000 7f38aea00640 1 -- start start 2024-03-24T00:34:16.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f38aea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a815e3a0 0x7f38a80ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f38aea00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a80cfaa0 con 0x7f38a80cdaa0 2024-03-24T00:34:16.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f38aea00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a80cfd80 con 0x7f38a815dfc0 2024-03-24T00:34:16.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f38aea00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80cf870 con 0x7f38a815e3a0 2024-03-24T00:34:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f38aca00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f38a80cdaa0 0x7f38a8167fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53764/0 (socket says 172.21.15.40:53764) 2024-03-24T00:34:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f38aca00640 1 -- 172.21.15.40:0/193295459 learned_addr learned my addr 172.21.15.40:0/193295459 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f38a7e00640 1 --1- 172.21.15.40:0/193295459 >> v1:172.21.15.53:6789/0 conn(0x7f38a815dfc0 0x7f38a80cd390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:33758/0 (socket says 172.21.15.40:33758) 2024-03-24T00:34:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f389f400640 1 --2- 172.21.15.40:0/193295459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a815e3a0 0x7f38a80ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f38a7400640 1 -- 172.21.15.40:0/193295459 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424482554 0 0) 0x7f38a80cfaa0 con 0x7f38a80cdaa0 2024-03-24T00:34:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f38a7400640 1 -- 172.21.15.40:0/193295459 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f388c003530 con 0x7f38a80cdaa0 2024-03-24T00:34:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f38a7400640 1 -- 172.21.15.40:0/193295459 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588240612 0 0) 0x7f38a80cfd80 con 0x7f38a815dfc0 2024-03-24T00:34:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f38a7400640 1 -- 172.21.15.40:0/193295459 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a80cfaa0 con 0x7f38a815dfc0 2024-03-24T00:34:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f389f400640 1 -- 172.21.15.40:0/193295459 >> v1:172.21.15.53:6789/0 conn(0x7f38a815dfc0 legacy=0x7f38a80cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f389f400640 1 -- 172.21.15.40:0/193295459 >> v1:172.21.15.40:6789/0 conn(0x7f38a80cdaa0 legacy=0x7f38a8167fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f389f400640 1 -- 172.21.15.40:0/193295459 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a80d0060 con 0x7f38a815e3a0 2024-03-24T00:34:16.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.535+0000 7f389f400640 1 --2- 172.21.15.40:0/193295459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a815e3a0 0x7f38a80ce690 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f38900097f0 tx=0x7f389001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82b0a033a90afafc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:16.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38a7400640 1 -- 172.21.15.40:0/193295459 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38900b0070 con 0x7f38a815e3a0 2024-03-24T00:34:16.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38a7400640 1 -- 172.21.15.40:0/193295459 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38900a4e50 con 0x7f38a815e3a0 2024-03-24T00:34:16.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38a7400640 1 -- 172.21.15.40:0/193295459 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38900ab820 con 0x7f38a815e3a0 2024-03-24T00:34:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 -- 172.21.15.40:0/193295459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a815e3a0 msgr2=0x7f38a80ce690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 --2- 172.21.15.40:0/193295459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a815e3a0 0x7f38a80ce690 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f38900097f0 tx=0x7f389001c3c0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 -- 172.21.15.40:0/193295459 shutdown_connections 2024-03-24T00:34:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 --2- 172.21.15.40:0/193295459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a815e3a0 0x7f38a80ce690 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 -- 172.21.15.40:0/193295459 >> 172.21.15.40:0/193295459 conn(0x7f38a80b37f0 msgr2=0x7f38a80b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 -- 172.21.15.40:0/193295459 shutdown_connections 2024-03-24T00:34:16.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 -- 172.21.15.40:0/193295459 wait complete. 2024-03-24T00:34:16.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 Processor -- start 2024-03-24T00:34:16.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 -- start start 2024-03-24T00:34:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a80cdaa0 0x7f38a8154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a815dfc0 0x7f38a8149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a815e3a0 0x7f38a80af4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a816c100 con 0x7f38a80cdaa0 2024-03-24T00:34:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80d0d50 con 0x7f38a815e3a0 2024-03-24T00:34:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38aea00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a816c280 con 0x7f38a815dfc0 2024-03-24T00:34:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a80cdaa0 0x7f38a8154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a80cdaa0 0x7f38a8154ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50234/0 (socket says 172.21.15.40:50234) 2024-03-24T00:34:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f38a7e00640 1 -- 172.21.15.40:0/2287759839 learned_addr learned my addr 172.21.15.40:0/2287759839 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.539+0000 7f389f400640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a815dfc0 0x7f38a8149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38aca00640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a815e3a0 0x7f38a80af4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38a7e00640 1 -- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a815dfc0 msgr2=0x7f38a8149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38a7e00640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a815dfc0 0x7f38a8149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38a7e00640 1 -- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a815e3a0 msgr2=0x7f38a80af4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38a7e00640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a815e3a0 0x7f38a80af4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38a7e00640 1 -- 172.21.15.40:0/2287759839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a8154040 con 0x7f38a80cdaa0 2024-03-24T00:34:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38a7e00640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a80cdaa0 0x7f38a8154ae0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f389800c420 tx=0x7f389800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38a4c00640 1 -- 172.21.15.40:0/2287759839 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38980074e0 con 0x7f38a80cdaa0 2024-03-24T00:34:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38a4c00640 1 -- 172.21.15.40:0/2287759839 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38980047f0 con 0x7f38a80cdaa0 2024-03-24T00:34:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38a4c00640 1 -- 172.21.15.40:0/2287759839 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38980076a0 con 0x7f38a80cdaa0 2024-03-24T00:34:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38aea00640 1 -- 172.21.15.40:0/2287759839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a80aef60 con 0x7f38a80cdaa0 2024-03-24T00:34:16.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38aea00640 1 -- 172.21.15.40:0/2287759839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a80af240 con 0x7f38a80cdaa0 2024-03-24T00:34:16.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38aea00640 1 -- 172.21.15.40:0/2287759839 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a8068170 con 0x7f38a80cdaa0 2024-03-24T00:34:16.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38a4c00640 1 -- 172.21.15.40:0/2287759839 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f389801a020 con 0x7f38a80cdaa0 2024-03-24T00:34:16.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38a4c00640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3884067660 0x7f3884069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.543+0000 7f38a4c00640 1 -- 172.21.15.40:0/2287759839 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3898085090 con 0x7f38a80cdaa0 2024-03-24T00:34:16.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.547+0000 7f389f400640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3884067660 0x7f3884069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.547+0000 7f389f400640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3884067660 0x7f3884069b20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f38900ad000 tx=0x7f38900a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:16.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.547+0000 7f38a4c00640 1 -- 172.21.15.40:0/2287759839 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3898055620 con 0x7f38a80cdaa0 2024-03-24T00:34:16.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.707+0000 7f38aea00640 1 -- 172.21.15.40:0/2287759839 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f38a80b7c20 con 0x7f3884067660 2024-03-24T00:34:16.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.723+0000 7f38a4c00640 1 -- 172.21.15.40:0/2287759839 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251887 (secure 0 0 0) 0x7f38a80b7c20 con 0x7f3884067660 2024-03-24T00:34:16.728 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:16.730 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:34:16.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.727+0000 7f38aea00640 1 -- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3884067660 msgr2=0x7f3884069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.727+0000 7f38aea00640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3884067660 0x7f3884069b20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f38900ad000 tx=0x7f38900a6000 comp rx=0 tx=0).stop 2024-03-24T00:34:16.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.727+0000 7f38aea00640 1 -- 172.21.15.40:0/2287759839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a80cdaa0 msgr2=0x7f38a8154ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.727+0000 7f38aea00640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a80cdaa0 0x7f38a8154ae0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f389800c420 tx=0x7f389800c8e0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.727+0000 7f38aea00640 1 -- 172.21.15.40:0/2287759839 shutdown_connections 2024-03-24T00:34:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.727+0000 7f38aea00640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3884067660 0x7f3884069b20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.727+0000 7f38aea00640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a815e3a0 0x7f38a80af4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.727+0000 7f38aea00640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a815dfc0 0x7f38a8149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.727+0000 7f38aea00640 1 --2- 172.21.15.40:0/2287759839 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38a80cdaa0 0x7f38a8154ae0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.727+0000 7f38aea00640 1 -- 172.21.15.40:0/2287759839 >> 172.21.15.40:0/2287759839 conn(0x7f38a80b37f0 msgr2=0x7f38a8158ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.727+0000 7f38aea00640 1 -- 172.21.15.40:0/2287759839 shutdown_connections 2024-03-24T00:34:16.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.727+0000 7f38aea00640 1 -- 172.21.15.40:0/2287759839 wait complete. 2024-03-24T00:34:16.748 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-03-24T00:34:15.237086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260268,"kb_used_data":6636,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754714452,"statfs":{"total":773094113280,"available":772827598848,"internally_reserved":0,"allocated":6795264,"data_stored":3525755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.632136"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641391+0000","last_change":"2024-03-24T00:34:12.639810+0000","last_active":"2024-03-24T00:34:12.641391+0000","last_peered":"2024-03-24T00:34:12.641391+0000","last_clean":"2024-03-24T00:34:12.641391+0000","last_became_active":"2024-03-24T00:34:12.639540+0000","last_became_peered":"2024-03-24T00:34:12.639540+0000","last_unstale":"2024-03-24T00:34:12.641391+0000","last_undegraded":"2024-03-24T00:34:12.641391+0000","last_fullsized":"2024-03-24T00:34:12.641391+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:03:12.808236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.637648+0000","last_change":"2024-03-24T00:34:12.635073+0000","last_active":"2024-03-24T00:34:12.637648+0000","last_peered":"2024-03-24T00:34:12.637648+0000","last_clean":"2024-03-24T00:34:12.637648+0000","last_became_active":"2024-03-24T00:34:12.634805+0000","last_became_peered":"2024-03-24T00:34:12.634805+0000","last_unstale":"2024-03-24T00:34:12.637648+0000","last_undegraded":"2024-03-24T00:34:12.637648+0000","last_fullsized":"2024-03-24T00:34:12.637648+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:34.398454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.651737+0000","last_change":"2024-03-24T00:34:12.649659+0000","last_active":"2024-03-24T00:34:12.651737+0000","last_peered":"2024-03-24T00:34:12.651737+0000","last_clean":"2024-03-24T00:34:12.651737+0000","last_became_active":"2024-03-24T00:34:12.649195+0000","last_became_peered":"2024-03-24T00:34:12.649195+0000","last_unstale":"2024-03-24T00:34:12.651737+0000","last_undegraded":"2024-03-24T00:34:12.651737+0000","last_fullsized":"2024-03-24T00:34:12.651737+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:42.944475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641251+0000","last_change":"2024-03-24T00:34:12.632651+0000","last_active":"2024-03-24T00:34:12.641251+0000","last_peered":"2024-03-24T00:34:12.641251+0000","last_clean":"2024-03-24T00:34:12.641251+0000","last_became_active":"2024-03-24T00:34:12.632369+0000","last_became_peered":"2024-03-24T00:34:12.632369+0000","last_unstale":"2024-03-24T00:34:12.641251+0000","last_undegraded":"2024-03-24T00:34:12.641251+0000","last_fullsized":"2024-03-24T00:34:12.641251+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:42.944475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.632542+0000","last_change":"2024-03-24T00:34:12.627518+0000","last_active":"2024-03-24T00:34:12.632542+0000","last_peered":"2024-03-24T00:34:12.632542+0000","last_clean":"2024-03-24T00:34:12.632542+0000","last_became_active":"2024-03-24T00:34:12.627230+0000","last_became_peered":"2024-03-24T00:34:12.627230+0000","last_unstale":"2024-03-24T00:34:12.632542+0000","last_undegraded":"2024-03-24T00:34:12.632542+0000","last_fullsized":"2024-03-24T00:34:12.632542+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:40:22.231804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.656771+0000","last_change":"2024-03-24T00:34:12.653958+0000","last_active":"2024-03-24T00:34:12.656771+0000","last_peered":"2024-03-24T00:34:12.656771+0000","last_clean":"2024-03-24T00:34:12.656771+0000","last_became_active":"2024-03-24T00:34:12.653671+0000","last_became_peered":"2024-03-24T00:34:12.653671+0000","last_unstale":"2024-03-24T00:34:12.656771+0000","last_undegraded":"2024-03-24T00:34:12.656771+0000","last_fullsized":"2024-03-24T00:34:12.656771+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:34.322294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.647131+0000","last_change":"2024-03-24T00:34:12.637902+0000","last_active":"2024-03-24T00:34:12.647131+0000","last_peered":"2024-03-24T00:34:12.647131+0000","last_clean":"2024-03-24T00:34:12.647131+0000","last_became_active":"2024-03-24T00:34:12.637266+0000","last_became_peered":"2024-03-24T00:34:12.637266+0000","last_unstale":"2024-03-24T00:34:12.647131+0000","last_undegraded":"2024-03-24T00:34:12.647131+0000","last_fullsized":"2024-03-24T00:34:12.647131+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:22:49.803500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.642885+0000","last_change":"2024-03-24T00:34:12.636304+0000","last_active":"2024-03-24T00:34:12.642885+0000","last_peered":"2024-03-24T00:34:12.642885+0000","last_clean":"2024-03-24T00:34:12.642885+0000","last_became_active":"2024-03-24T00:34:12.635975+0000","last_became_peered":"2024-03-24T00:34:12.635975+0000","last_unstale":"2024-03-24T00:34:12.642885+0000","last_undegraded":"2024-03-24T00:34:12.642885+0000","last_fullsized":"2024-03-24T00:34:12.642885+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:20:06.851983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.654287+0000","last_change":"2024-03-24T00:34:12.653549+0000","last_active":"2024-03-24T00:34:12.654287+0000","last_peered":"2024-03-24T00:34:12.654287+0000","last_clean":"2024-03-24T00:34:12.654287+0000","last_became_active":"2024-03-24T00:34:12.653201+0000","last_became_peered":"2024-03-24T00:34:12.653201+0000","last_unstale":"2024-03-24T00:34:12.654287+0000","last_undegraded":"2024-03-24T00:34:12.654287+0000","last_fullsized":"2024-03-24T00:34:12.654287+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:18.631243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.630392+0000","last_change":"2024-03-24T00:34:12.628664+0000","last_active":"2024-03-24T00:34:12.630392+0000","last_peered":"2024-03-24T00:34:12.630392+0000","last_clean":"2024-03-24T00:34:12.630392+0000","last_became_active":"2024-03-24T00:34:12.628363+0000","last_became_peered":"2024-03-24T00:34:12.628363+0000","last_unstale":"2024-03-24T00:34:12.630392+0000","last_undegraded":"2024-03-24T00:34:12.630392+0000","last_fullsized":"2024-03-24T00:34:12.630392+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:26.884066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.643087+0000","last_change":"2024-03-24T00:34:12.639146+0000","last_active":"2024-03-24T00:34:12.643087+0000","last_peered":"2024-03-24T00:34:12.643087+0000","last_clean":"2024-03-24T00:34:12.643087+0000","last_became_active":"2024-03-24T00:34:12.638873+0000","last_became_peered":"2024-03-24T00:34:12.638873+0000","last_unstale":"2024-03-24T00:34:12.643087+0000","last_undegraded":"2024-03-24T00:34:12.643087+0000","last_fullsized":"2024-03-24T00:34:12.643087+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:30.727741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.630305+0000","last_change":"2024-03-24T00:34:12.628970+0000","last_active":"2024-03-24T00:34:12.630305+0000","last_peered":"2024-03-24T00:34:12.630305+0000","last_clean":"2024-03-24T00:34:12.630305+0000","last_became_active":"2024-03-24T00:34:12.628618+0000","last_became_peered":"2024-03-24T00:34:12.628618+0000","last_unstale":"2024-03-24T00:34:12.630305+0000","last_undegraded":"2024-03-24T00:34:12.630305+0000","last_fullsized":"2024-03-24T00:34:12.630305+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:04.494411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.652407+0000","last_change":"2024-03-24T00:34:12.646984+0000","last_active":"2024-03-24T00:34:12.652407+0000","last_peered":"2024-03-24T00:34:12.652407+0000","last_clean":"2024-03-24T00:34:12.652407+0000","last_became_active":"2024-03-24T00:34:12.646453+0000","last_became_peered":"2024-03-24T00:34:12.646453+0000","last_unstale":"2024-03-24T00:34:12.652407+0000","last_undegraded":"2024-03-24T00:34:12.652407+0000","last_fullsized":"2024-03-24T00:34:12.652407+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:26.884066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.647263+0000","last_change":"2024-03-24T00:34:12.640749+0000","last_active":"2024-03-24T00:34:12.647263+0000","last_peered":"2024-03-24T00:34:12.647263+0000","last_clean":"2024-03-24T00:34:12.647263+0000","last_became_active":"2024-03-24T00:34:12.640287+0000","last_became_peered":"2024-03-24T00:34:12.640287+0000","last_unstale":"2024-03-24T00:34:12.647263+0000","last_undegraded":"2024-03-24T00:34:12.647263+0000","last_fullsized":"2024-03-24T00:34:12.647263+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:19.306192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.644791+0000","last_change":"2024-03-24T00:34:12.635211+0000","last_active":"2024-03-24T00:34:12.644791+0000","last_peered":"2024-03-24T00:34:12.644791+0000","last_clean":"2024-03-24T00:34:12.644791+0000","last_became_active":"2024-03-24T00:34:12.634736+0000","last_became_peered":"2024-03-24T00:34:12.634736+0000","last_unstale":"2024-03-24T00:34:12.644791+0000","last_undegraded":"2024-03-24T00:34:12.644791+0000","last_fullsized":"2024-03-24T00:34:12.644791+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:12:28.260973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.656822+0000","last_change":"2024-03-24T00:34:12.654963+0000","last_active":"2024-03-24T00:34:12.656822+0000","last_peered":"2024-03-24T00:34:12.656822+0000","last_clean":"2024-03-24T00:34:12.656822+0000","last_became_active":"2024-03-24T00:34:12.654655+0000","last_became_peered":"2024-03-24T00:34:12.654655+0000","last_unstale":"2024-03-24T00:34:12.656822+0000","last_undegraded":"2024-03-24T00:34:12.656822+0000","last_fullsized":"2024-03-24T00:34:12.656822+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:27.215945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.643035+0000","last_change":"2024-03-24T00:34:12.637403+0000","last_active":"2024-03-24T00:34:12.643035+0000","last_peered":"2024-03-24T00:34:12.643035+0000","last_clean":"2024-03-24T00:34:12.643035+0000","last_became_active":"2024-03-24T00:34:12.637088+0000","last_became_peered":"2024-03-24T00:34:12.637088+0000","last_unstale":"2024-03-24T00:34:12.643035+0000","last_undegraded":"2024-03-24T00:34:12.643035+0000","last_fullsized":"2024-03-24T00:34:12.643035+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:29.388659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.653709+0000","last_change":"2024-03-24T00:34:12.651405+0000","last_active":"2024-03-24T00:34:12.653709+0000","last_peered":"2024-03-24T00:34:12.653709+0000","last_clean":"2024-03-24T00:34:12.653709+0000","last_became_active":"2024-03-24T00:34:12.651070+0000","last_became_peered":"2024-03-24T00:34:12.651070+0000","last_unstale":"2024-03-24T00:34:12.653709+0000","last_undegraded":"2024-03-24T00:34:12.653709+0000","last_fullsized":"2024-03-24T00:34:12.653709+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:45:48.010797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641444+0000","last_change":"2024-03-24T00:34:12.639846+0000","last_active":"2024-03-24T00:34:12.641444+0000","last_peered":"2024-03-24T00:34:12.641444+0000","last_clean":"2024-03-24T00:34:12.641444+0000","last_became_active":"2024-03-24T00:34:12.639593+0000","last_became_peered":"2024-03-24T00:34:12.639593+0000","last_unstale":"2024-03-24T00:34:12.641444+0000","last_undegraded":"2024-03-24T00:34:12.641444+0000","last_fullsized":"2024-03-24T00:34:12.641444+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:56.769496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.652706+0000","last_change":"2024-03-24T00:34:12.648194+0000","last_active":"2024-03-24T00:34:12.652706+0000","last_peered":"2024-03-24T00:34:12.652706+0000","last_clean":"2024-03-24T00:34:12.652706+0000","last_became_active":"2024-03-24T00:34:12.647893+0000","last_became_peered":"2024-03-24T00:34:12.647893+0000","last_unstale":"2024-03-24T00:34:12.652706+0000","last_undegraded":"2024-03-24T00:34:12.652706+0000","last_fullsized":"2024-03-24T00:34:12.652706+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:32:56.873303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.646365+0000","last_change":"2024-03-24T00:34:12.633174+0000","last_active":"2024-03-24T00:34:12.646365+0000","last_peered":"2024-03-24T00:34:12.646365+0000","last_clean":"2024-03-24T00:34:12.646365+0000","last_became_active":"2024-03-24T00:34:12.632921+0000","last_became_peered":"2024-03-24T00:34:12.632921+0000","last_unstale":"2024-03-24T00:34:12.646365+0000","last_undegraded":"2024-03-24T00:34:12.646365+0000","last_fullsized":"2024-03-24T00:34:12.646365+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:32.862791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.646832+0000","last_change":"2024-03-24T00:34:12.636922+0000","last_active":"2024-03-24T00:34:12.646832+0000","last_peered":"2024-03-24T00:34:12.646832+0000","last_clean":"2024-03-24T00:34:12.646832+0000","last_became_active":"2024-03-24T00:34:12.636616+0000","last_became_peered":"2024-03-24T00:34:12.636616+0000","last_unstale":"2024-03-24T00:34:12.646832+0000","last_undegraded":"2024-03-24T00:34:12.646832+0000","last_fullsized":"2024-03-24T00:34:12.646832+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:01.684593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.650735+0000","last_change":"2024-03-24T00:34:12.646731+0000","last_active":"2024-03-24T00:34:12.650735+0000","last_peered":"2024-03-24T00:34:12.650735+0000","last_clean":"2024-03-24T00:34:12.650735+0000","last_became_active":"2024-03-24T00:34:12.643397+0000","last_became_peered":"2024-03-24T00:34:12.643397+0000","last_unstale":"2024-03-24T00:34:12.650735+0000","last_undegraded":"2024-03-24T00:34:12.650735+0000","last_fullsized":"2024-03-24T00:34:12.650735+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:01.823045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.652500+0000","last_change":"2024-03-24T00:34:12.646918+0000","last_active":"2024-03-24T00:34:12.652500+0000","last_peered":"2024-03-24T00:34:12.652500+0000","last_clean":"2024-03-24T00:34:12.652500+0000","last_became_active":"2024-03-24T00:34:12.646571+0000","last_became_peered":"2024-03-24T00:34:12.646571+0000","last_unstale":"2024-03-24T00:34:12.652500+0000","last_undegraded":"2024-03-24T00:34:12.652500+0000","last_fullsized":"2024-03-24T00:34:12.652500+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:50:54.993763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.634678+0000","last_change":"2024-03-24T00:34:12.629682+0000","last_active":"2024-03-24T00:34:12.634678+0000","last_peered":"2024-03-24T00:34:12.634678+0000","last_clean":"2024-03-24T00:34:12.634678+0000","last_became_active":"2024-03-24T00:34:12.629353+0000","last_became_peered":"2024-03-24T00:34:12.629353+0000","last_unstale":"2024-03-24T00:34:12.634678+0000","last_undegraded":"2024-03-24T00:34:12.634678+0000","last_fullsized":"2024-03-24T00:34:12.634678+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:35:40.543447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.650306+0000","last_change":"2024-03-24T00:34:12.648586+0000","last_active":"2024-03-24T00:34:12.650306+0000","last_peered":"2024-03-24T00:34:12.650306+0000","last_clean":"2024-03-24T00:34:12.650306+0000","last_became_active":"2024-03-24T00:34:12.648291+0000","last_became_peered":"2024-03-24T00:34:12.648291+0000","last_unstale":"2024-03-24T00:34:12.650306+0000","last_undegraded":"2024-03-24T00:34:12.650306+0000","last_fullsized":"2024-03-24T00:34:12.650306+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:43.825491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.656800+0000","last_change":"2024-03-24T00:34:12.654345+0000","last_active":"2024-03-24T00:34:12.656800+0000","last_peered":"2024-03-24T00:34:12.656800+0000","last_clean":"2024-03-24T00:34:12.656800+0000","last_became_active":"2024-03-24T00:34:12.653139+0000","last_became_peered":"2024-03-24T00:34:12.653139+0000","last_unstale":"2024-03-24T00:34:12.656800+0000","last_undegraded":"2024-03-24T00:34:12.656800+0000","last_fullsized":"2024-03-24T00:34:12.656800+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:42.435900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.634632+0000","last_change":"2024-03-24T00:34:12.629865+0000","last_active":"2024-03-24T00:34:12.634632+0000","last_peered":"2024-03-24T00:34:12.634632+0000","last_clean":"2024-03-24T00:34:12.634632+0000","last_became_active":"2024-03-24T00:34:12.628899+0000","last_became_peered":"2024-03-24T00:34:12.628899+0000","last_unstale":"2024-03-24T00:34:12.634632+0000","last_undegraded":"2024-03-24T00:34:12.634632+0000","last_fullsized":"2024-03-24T00:34:12.634632+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:19.394650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.643228+0000","last_change":"2024-03-24T00:34:12.638095+0000","last_active":"2024-03-24T00:34:12.643228+0000","last_peered":"2024-03-24T00:34:12.643228+0000","last_clean":"2024-03-24T00:34:12.643228+0000","last_became_active":"2024-03-24T00:34:12.637256+0000","last_became_peered":"2024-03-24T00:34:12.637256+0000","last_unstale":"2024-03-24T00:34:12.643228+0000","last_undegraded":"2024-03-24T00:34:12.643228+0000","last_fullsized":"2024-03-24T00:34:12.643228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:10:44.804304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.650575+0000","last_change":"2024-03-24T00:34:12.644570+0000","last_active":"2024-03-24T00:34:12.650575+0000","last_peered":"2024-03-24T00:34:12.650575+0000","last_clean":"2024-03-24T00:34:12.650575+0000","last_became_active":"2024-03-24T00:34:12.644314+0000","last_became_peered":"2024-03-24T00:34:12.644314+0000","last_unstale":"2024-03-24T00:34:12.650575+0000","last_undegraded":"2024-03-24T00:34:12.650575+0000","last_fullsized":"2024-03-24T00:34:12.650575+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:49.371076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.630144+0000","last_change":"2024-03-24T00:34:12.625778+0000","last_active":"2024-03-24T00:34:12.630144+0000","last_peered":"2024-03-24T00:34:12.630144+0000","last_clean":"2024-03-24T00:34:12.630144+0000","last_became_active":"2024-03-24T00:34:12.625389+0000","last_became_peered":"2024-03-24T00:34:12.625389+0000","last_unstale":"2024-03-24T00:34:12.630144+0000","last_undegraded":"2024-03-24T00:34:12.630144+0000","last_fullsized":"2024-03-24T00:34:12.630144+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:37.918280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.656376+0000","last_change":"2024-03-24T00:34:12.652924+0000","last_active":"2024-03-24T00:34:12.656376+0000","last_peered":"2024-03-24T00:34:12.656376+0000","last_clean":"2024-03-24T00:34:12.656376+0000","last_became_active":"2024-03-24T00:34:12.651457+0000","last_became_peered":"2024-03-24T00:34:12.651457+0000","last_unstale":"2024-03-24T00:34:12.656376+0000","last_undegraded":"2024-03-24T00:34:12.656376+0000","last_fullsized":"2024-03-24T00:34:12.656376+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:43.825491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.651179+0000","last_change":"2024-03-24T00:34:12.647285+0000","last_active":"2024-03-24T00:34:12.651179+0000","last_peered":"2024-03-24T00:34:12.651179+0000","last_clean":"2024-03-24T00:34:12.651179+0000","last_became_active":"2024-03-24T00:34:12.647054+0000","last_became_peered":"2024-03-24T00:34:12.647054+0000","last_unstale":"2024-03-24T00:34:12.651179+0000","last_undegraded":"2024-03-24T00:34:12.651179+0000","last_fullsized":"2024-03-24T00:34:12.651179+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:01.823045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"58'4","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.656341+0000","last_change":"2024-03-24T00:34:11.635411+0000","last_active":"2024-03-24T00:34:12.656341+0000","last_peered":"2024-03-24T00:34:12.656341+0000","last_clean":"2024-03-24T00:34:12.656341+0000","last_became_active":"2024-03-24T00:34:11.635088+0000","last_became_peered":"2024-03-24T00:34:11.635088+0000","last_unstale":"2024-03-24T00:34:12.656341+0000","last_undegraded":"2024-03-24T00:34:12.656341+0000","last_fullsized":"2024-03-24T00:34:12.656341+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:17.777645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.637586+0000","last_change":"2024-03-24T00:34:12.633501+0000","last_active":"2024-03-24T00:34:12.637586+0000","last_peered":"2024-03-24T00:34:12.637586+0000","last_clean":"2024-03-24T00:34:12.637586+0000","last_became_active":"2024-03-24T00:34:12.633232+0000","last_became_peered":"2024-03-24T00:34:12.633232+0000","last_unstale":"2024-03-24T00:34:12.637586+0000","last_undegraded":"2024-03-24T00:34:12.637586+0000","last_fullsized":"2024-03-24T00:34:12.637586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:08:26.441734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.649761+0000","last_change":"2024-03-24T00:34:11.633628+0000","last_active":"2024-03-24T00:34:12.649761+0000","last_peered":"2024-03-24T00:34:12.649761+0000","last_clean":"2024-03-24T00:34:12.649761+0000","last_became_active":"2024-03-24T00:34:11.633358+0000","last_became_peered":"2024-03-24T00:34:11.633358+0000","last_unstale":"2024-03-24T00:34:12.649761+0000","last_undegraded":"2024-03-24T00:34:12.649761+0000","last_fullsized":"2024-03-24T00:34:12.649761+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:26.208918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.656313+0000","last_change":"2024-03-24T00:34:12.651300+0000","last_active":"2024-03-24T00:34:12.656313+0000","last_peered":"2024-03-24T00:34:12.656313+0000","last_clean":"2024-03-24T00:34:12.656313+0000","last_became_active":"2024-03-24T00:34:12.650101+0000","last_became_peered":"2024-03-24T00:34:12.650101+0000","last_unstale":"2024-03-24T00:34:12.656313+0000","last_undegraded":"2024-03-24T00:34:12.656313+0000","last_fullsized":"2024-03-24T00:34:12.656313+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:45:48.010797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.653176+0000","last_change":"2024-03-24T00:34:11.639622+0000","last_active":"2024-03-24T00:34:12.653176+0000","last_peered":"2024-03-24T00:34:12.653176+0000","last_clean":"2024-03-24T00:34:12.653176+0000","last_became_active":"2024-03-24T00:34:11.639347+0000","last_became_peered":"2024-03-24T00:34:11.639347+0000","last_unstale":"2024-03-24T00:34:12.653176+0000","last_undegraded":"2024-03-24T00:34:12.653176+0000","last_fullsized":"2024-03-24T00:34:12.653176+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:22.665321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.654845+0000","last_change":"2024-03-24T00:34:12.650537+0000","last_active":"2024-03-24T00:34:12.654845+0000","last_peered":"2024-03-24T00:34:12.654845+0000","last_clean":"2024-03-24T00:34:12.654845+0000","last_became_active":"2024-03-24T00:34:12.648402+0000","last_became_peered":"2024-03-24T00:34:12.648402+0000","last_unstale":"2024-03-24T00:34:12.654845+0000","last_undegraded":"2024-03-24T00:34:12.654845+0000","last_fullsized":"2024-03-24T00:34:12.654845+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:34.956494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.649864+0000","last_change":"2024-03-24T00:34:11.631927+0000","last_active":"2024-03-24T00:34:12.649864+0000","last_peered":"2024-03-24T00:34:12.649864+0000","last_clean":"2024-03-24T00:34:12.649864+0000","last_became_active":"2024-03-24T00:34:11.631553+0000","last_became_peered":"2024-03-24T00:34:11.631553+0000","last_unstale":"2024-03-24T00:34:12.649864+0000","last_undegraded":"2024-03-24T00:34:12.649864+0000","last_fullsized":"2024-03-24T00:34:12.649864+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:18.299165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.646855+0000","last_change":"2024-03-24T00:34:12.638309+0000","last_active":"2024-03-24T00:34:12.646855+0000","last_peered":"2024-03-24T00:34:12.646855+0000","last_clean":"2024-03-24T00:34:12.646855+0000","last_became_active":"2024-03-24T00:34:12.638032+0000","last_became_peered":"2024-03-24T00:34:12.638032+0000","last_unstale":"2024-03-24T00:34:12.646855+0000","last_undegraded":"2024-03-24T00:34:12.646855+0000","last_fullsized":"2024-03-24T00:34:12.646855+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:55:30.175598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.650706+0000","last_change":"2024-03-24T00:34:11.634138+0000","last_active":"2024-03-24T00:34:12.650706+0000","last_peered":"2024-03-24T00:34:12.650706+0000","last_clean":"2024-03-24T00:34:12.650706+0000","last_became_active":"2024-03-24T00:34:11.633644+0000","last_became_peered":"2024-03-24T00:34:11.633644+0000","last_unstale":"2024-03-24T00:34:12.650706+0000","last_undegraded":"2024-03-24T00:34:12.650706+0000","last_fullsized":"2024-03-24T00:34:12.650706+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:44:33.132978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.654865+0000","last_change":"2024-03-24T00:34:12.649115+0000","last_active":"2024-03-24T00:34:12.654865+0000","last_peered":"2024-03-24T00:34:12.654865+0000","last_clean":"2024-03-24T00:34:12.654865+0000","last_became_active":"2024-03-24T00:34:12.648431+0000","last_became_peered":"2024-03-24T00:34:12.648431+0000","last_unstale":"2024-03-24T00:34:12.654865+0000","last_undegraded":"2024-03-24T00:34:12.654865+0000","last_fullsized":"2024-03-24T00:34:12.654865+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:37.203531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.654527+0000","last_change":"2024-03-24T00:34:11.636952+0000","last_active":"2024-03-24T00:34:12.654527+0000","last_peered":"2024-03-24T00:34:12.654527+0000","last_clean":"2024-03-24T00:34:12.654527+0000","last_became_active":"2024-03-24T00:34:11.636584+0000","last_became_peered":"2024-03-24T00:34:11.636584+0000","last_unstale":"2024-03-24T00:34:12.654527+0000","last_undegraded":"2024-03-24T00:34:12.654527+0000","last_fullsized":"2024-03-24T00:34:12.654527+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:48.364049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.637699+0000","last_change":"2024-03-24T00:34:12.633770+0000","last_active":"2024-03-24T00:34:12.637699+0000","last_peered":"2024-03-24T00:34:12.637699+0000","last_clean":"2024-03-24T00:34:12.637699+0000","last_became_active":"2024-03-24T00:34:12.631944+0000","last_became_peered":"2024-03-24T00:34:12.631944+0000","last_unstale":"2024-03-24T00:34:12.637699+0000","last_undegraded":"2024-03-24T00:34:12.637699+0000","last_fullsized":"2024-03-24T00:34:12.637699+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:06.028431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.625483+0000","last_change":"2024-03-24T00:34:11.636320+0000","last_active":"2024-03-24T00:34:12.625483+0000","last_peered":"2024-03-24T00:34:12.625483+0000","last_clean":"2024-03-24T00:34:12.625483+0000","last_became_active":"2024-03-24T00:34:11.635965+0000","last_became_peered":"2024-03-24T00:34:11.635965+0000","last_unstale":"2024-03-24T00:34:12.625483+0000","last_undegraded":"2024-03-24T00:34:12.625483+0000","last_fullsized":"2024-03-24T00:34:12.625483+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:31.855764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.637615+0000","last_change":"2024-03-24T00:34:12.634803+0000","last_active":"2024-03-24T00:34:12.637615+0000","last_peered":"2024-03-24T00:34:12.637615+0000","last_clean":"2024-03-24T00:34:12.637615+0000","last_became_active":"2024-03-24T00:34:12.633695+0000","last_became_peered":"2024-03-24T00:34:12.633695+0000","last_unstale":"2024-03-24T00:34:12.637615+0000","last_undegraded":"2024-03-24T00:34:12.637615+0000","last_fullsized":"2024-03-24T00:34:12.637615+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:55:20.612662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.638128+0000","last_change":"2024-03-24T00:34:11.629752+0000","last_active":"2024-03-24T00:34:12.638128+0000","last_peered":"2024-03-24T00:34:12.638128+0000","last_clean":"2024-03-24T00:34:12.638128+0000","last_became_active":"2024-03-24T00:34:11.629443+0000","last_became_peered":"2024-03-24T00:34:11.629443+0000","last_unstale":"2024-03-24T00:34:12.638128+0000","last_undegraded":"2024-03-24T00:34:12.638128+0000","last_fullsized":"2024-03-24T00:34:12.638128+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:37.882421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.647223+0000","last_change":"2024-03-24T00:34:12.639235+0000","last_active":"2024-03-24T00:34:12.647223+0000","last_peered":"2024-03-24T00:34:12.647223+0000","last_clean":"2024-03-24T00:34:12.647223+0000","last_became_active":"2024-03-24T00:34:12.638898+0000","last_became_peered":"2024-03-24T00:34:12.638898+0000","last_unstale":"2024-03-24T00:34:12.647223+0000","last_undegraded":"2024-03-24T00:34:12.647223+0000","last_fullsized":"2024-03-24T00:34:12.647223+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:00:57.594800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.620715+0000","last_change":"2024-03-24T00:34:11.641975+0000","last_active":"2024-03-24T00:34:12.620715+0000","last_peered":"2024-03-24T00:34:12.620715+0000","last_clean":"2024-03-24T00:34:12.620715+0000","last_became_active":"2024-03-24T00:34:11.641574+0000","last_became_peered":"2024-03-24T00:34:11.641574+0000","last_unstale":"2024-03-24T00:34:12.620715+0000","last_undegraded":"2024-03-24T00:34:12.620715+0000","last_fullsized":"2024-03-24T00:34:12.620715+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:39.087307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.642772+0000","last_change":"2024-03-24T00:34:12.632580+0000","last_active":"2024-03-24T00:34:12.642772+0000","last_peered":"2024-03-24T00:34:12.642772+0000","last_clean":"2024-03-24T00:34:12.642772+0000","last_became_active":"2024-03-24T00:34:12.632309+0000","last_became_peered":"2024-03-24T00:34:12.632309+0000","last_unstale":"2024-03-24T00:34:12.642772+0000","last_undegraded":"2024-03-24T00:34:12.642772+0000","last_fullsized":"2024-03-24T00:34:12.642772+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:15:00.368589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.624363+0000","last_change":"2024-03-24T00:34:11.638279+0000","last_active":"2024-03-24T00:34:12.624363+0000","last_peered":"2024-03-24T00:34:12.624363+0000","last_clean":"2024-03-24T00:34:12.624363+0000","last_became_active":"2024-03-24T00:34:11.638019+0000","last_became_peered":"2024-03-24T00:34:11.638019+0000","last_unstale":"2024-03-24T00:34:12.624363+0000","last_undegraded":"2024-03-24T00:34:12.624363+0000","last_fullsized":"2024-03-24T00:34:12.624363+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:46:12.594425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.649441+0000","last_change":"2024-03-24T00:34:12.645730+0000","last_active":"2024-03-24T00:34:12.649441+0000","last_peered":"2024-03-24T00:34:12.649441+0000","last_clean":"2024-03-24T00:34:12.649441+0000","last_became_active":"2024-03-24T00:34:12.645344+0000","last_became_peered":"2024-03-24T00:34:12.645344+0000","last_unstale":"2024-03-24T00:34:12.649441+0000","last_undegraded":"2024-03-24T00:34:12.649441+0000","last_fullsized":"2024-03-24T00:34:12.649441+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:10:42.804405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"58'3","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.650738+0000","last_change":"2024-03-24T00:34:11.629548+0000","last_active":"2024-03-24T00:34:12.650738+0000","last_peered":"2024-03-24T00:34:12.650738+0000","last_clean":"2024-03-24T00:34:12.650738+0000","last_became_active":"2024-03-24T00:34:11.629262+0000","last_became_peered":"2024-03-24T00:34:11.629262+0000","last_unstale":"2024-03-24T00:34:12.650738+0000","last_undegraded":"2024-03-24T00:34:12.650738+0000","last_fullsized":"2024-03-24T00:34:12.650738+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:18.299165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.637199+0000","last_change":"2024-03-24T00:34:12.628833+0000","last_active":"2024-03-24T00:34:12.637199+0000","last_peered":"2024-03-24T00:34:12.637199+0000","last_clean":"2024-03-24T00:34:12.637199+0000","last_became_active":"2024-03-24T00:34:12.628523+0000","last_became_peered":"2024-03-24T00:34:12.628523+0000","last_unstale":"2024-03-24T00:34:12.637199+0000","last_undegraded":"2024-03-24T00:34:12.637199+0000","last_fullsized":"2024-03-24T00:34:12.637199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:18.631243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"58'6","reported_seq":18,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.658082+0000","last_change":"2024-03-24T00:34:11.641867+0000","last_active":"2024-03-24T00:34:12.658082+0000","last_peered":"2024-03-24T00:34:12.658082+0000","last_clean":"2024-03-24T00:34:12.658082+0000","last_became_active":"2024-03-24T00:34:11.641468+0000","last_became_peered":"2024-03-24T00:34:11.641468+0000","last_unstale":"2024-03-24T00:34:12.658082+0000","last_undegraded":"2024-03-24T00:34:12.658082+0000","last_fullsized":"2024-03-24T00:34:12.658082+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:04.192880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.636586+0000","last_change":"2024-03-24T00:34:12.631235+0000","last_active":"2024-03-24T00:34:12.636586+0000","last_peered":"2024-03-24T00:34:12.636586+0000","last_clean":"2024-03-24T00:34:12.636586+0000","last_became_active":"2024-03-24T00:34:12.630587+0000","last_became_peered":"2024-03-24T00:34:12.630587+0000","last_unstale":"2024-03-24T00:34:12.636586+0000","last_undegraded":"2024-03-24T00:34:12.636586+0000","last_fullsized":"2024-03-24T00:34:12.636586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:49.371076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.640679+0000","last_change":"2024-03-24T00:34:11.634013+0000","last_active":"2024-03-24T00:34:12.640679+0000","last_peered":"2024-03-24T00:34:12.640679+0000","last_clean":"2024-03-24T00:34:12.640679+0000","last_became_active":"2024-03-24T00:34:11.633712+0000","last_became_peered":"2024-03-24T00:34:11.633712+0000","last_unstale":"2024-03-24T00:34:12.640679+0000","last_undegraded":"2024-03-24T00:34:12.640679+0000","last_fullsized":"2024-03-24T00:34:12.640679+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:07.720435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.638984+0000","last_change":"2024-03-24T00:34:12.631027+0000","last_active":"2024-03-24T00:34:12.638984+0000","last_peered":"2024-03-24T00:34:12.638984+0000","last_clean":"2024-03-24T00:34:12.638984+0000","last_became_active":"2024-03-24T00:34:12.630743+0000","last_became_peered":"2024-03-24T00:34:12.630743+0000","last_unstale":"2024-03-24T00:34:12.638984+0000","last_undegraded":"2024-03-24T00:34:12.638984+0000","last_fullsized":"2024-03-24T00:34:12.638984+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:18.784673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.623506+0000","last_change":"2024-03-24T00:34:11.630910+0000","last_active":"2024-03-24T00:34:12.623506+0000","last_peered":"2024-03-24T00:34:12.623506+0000","last_clean":"2024-03-24T00:34:12.623506+0000","last_became_active":"2024-03-24T00:34:11.630593+0000","last_became_peered":"2024-03-24T00:34:11.630593+0000","last_unstale":"2024-03-24T00:34:12.623506+0000","last_undegraded":"2024-03-24T00:34:12.623506+0000","last_fullsized":"2024-03-24T00:34:12.623506+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:09.962122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.647444+0000","last_change":"2024-03-24T00:34:12.641608+0000","last_active":"2024-03-24T00:34:12.647444+0000","last_peered":"2024-03-24T00:34:12.647444+0000","last_clean":"2024-03-24T00:34:12.647444+0000","last_became_active":"2024-03-24T00:34:12.640410+0000","last_became_peered":"2024-03-24T00:34:12.640410+0000","last_unstale":"2024-03-24T00:34:12.647444+0000","last_undegraded":"2024-03-24T00:34:12.647444+0000","last_fullsized":"2024-03-24T00:34:12.647444+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:23.672349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.625454+0000","last_change":"2024-03-24T00:34:11.630895+0000","last_active":"2024-03-24T00:34:12.625454+0000","last_peered":"2024-03-24T00:34:12.625454+0000","last_clean":"2024-03-24T00:34:12.625454+0000","last_became_active":"2024-03-24T00:34:11.630063+0000","last_became_peered":"2024-03-24T00:34:11.630063+0000","last_unstale":"2024-03-24T00:34:12.625454+0000","last_undegraded":"2024-03-24T00:34:12.625454+0000","last_fullsized":"2024-03-24T00:34:12.625454+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:59.361561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.650712+0000","last_change":"2024-03-24T00:34:12.645396+0000","last_active":"2024-03-24T00:34:12.650712+0000","last_peered":"2024-03-24T00:34:12.650712+0000","last_clean":"2024-03-24T00:34:12.650712+0000","last_became_active":"2024-03-24T00:34:12.643067+0000","last_became_peered":"2024-03-24T00:34:12.643067+0000","last_unstale":"2024-03-24T00:34:12.650712+0000","last_undegraded":"2024-03-24T00:34:12.650712+0000","last_fullsized":"2024-03-24T00:34:12.650712+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:39.516699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"58'8","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.653086+0000","last_change":"2024-03-24T00:34:11.632399+0000","last_active":"2024-03-24T00:34:12.653086+0000","last_peered":"2024-03-24T00:34:12.653086+0000","last_clean":"2024-03-24T00:34:12.653086+0000","last_became_active":"2024-03-24T00:34:11.632126+0000","last_became_peered":"2024-03-24T00:34:11.632126+0000","last_unstale":"2024-03-24T00:34:12.653086+0000","last_undegraded":"2024-03-24T00:34:12.653086+0000","last_fullsized":"2024-03-24T00:34:12.653086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:00:56.587773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.652346+0000","last_change":"2024-03-24T00:34:12.649908+0000","last_active":"2024-03-24T00:34:12.652346+0000","last_peered":"2024-03-24T00:34:12.652346+0000","last_clean":"2024-03-24T00:34:12.652346+0000","last_became_active":"2024-03-24T00:34:12.647786+0000","last_became_peered":"2024-03-24T00:34:12.647786+0000","last_unstale":"2024-03-24T00:34:12.652346+0000","last_undegraded":"2024-03-24T00:34:12.652346+0000","last_fullsized":"2024-03-24T00:34:12.652346+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:34.956494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.639902+0000","last_change":"2024-03-24T00:34:11.634357+0000","last_active":"2024-03-24T00:34:12.639902+0000","last_peered":"2024-03-24T00:34:12.639902+0000","last_clean":"2024-03-24T00:34:12.639902+0000","last_became_active":"2024-03-24T00:34:11.634072+0000","last_became_peered":"2024-03-24T00:34:11.634072+0000","last_unstale":"2024-03-24T00:34:12.639902+0000","last_undegraded":"2024-03-24T00:34:12.639902+0000","last_fullsized":"2024-03-24T00:34:12.639902+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:34:41.047302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.643374+0000","last_change":"2024-03-24T00:34:12.638710+0000","last_active":"2024-03-24T00:34:12.643374+0000","last_peered":"2024-03-24T00:34:12.643374+0000","last_clean":"2024-03-24T00:34:12.643374+0000","last_became_active":"2024-03-24T00:34:12.638395+0000","last_became_peered":"2024-03-24T00:34:12.638395+0000","last_unstale":"2024-03-24T00:34:12.643374+0000","last_undegraded":"2024-03-24T00:34:12.643374+0000","last_fullsized":"2024-03-24T00:34:12.643374+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:45:48.010797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.653532+0000","last_change":"2024-03-24T00:34:11.639569+0000","last_active":"2024-03-24T00:34:12.653532+0000","last_peered":"2024-03-24T00:34:12.653532+0000","last_clean":"2024-03-24T00:34:12.653532+0000","last_became_active":"2024-03-24T00:34:11.639306+0000","last_became_peered":"2024-03-24T00:34:11.639306+0000","last_unstale":"2024-03-24T00:34:12.653532+0000","last_undegraded":"2024-03-24T00:34:12.653532+0000","last_fullsized":"2024-03-24T00:34:12.653532+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:17.624215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.654862+0000","last_change":"2024-03-24T00:34:12.650325+0000","last_active":"2024-03-24T00:34:12.654862+0000","last_peered":"2024-03-24T00:34:12.654862+0000","last_clean":"2024-03-24T00:34:12.654862+0000","last_became_active":"2024-03-24T00:34:12.650033+0000","last_became_peered":"2024-03-24T00:34:12.650033+0000","last_unstale":"2024-03-24T00:34:12.654862+0000","last_undegraded":"2024-03-24T00:34:12.654862+0000","last_fullsized":"2024-03-24T00:34:12.654862+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:44.272866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.649544+0000","last_change":"2024-03-24T00:34:11.631227+0000","last_active":"2024-03-24T00:34:12.649544+0000","last_peered":"2024-03-24T00:34:12.649544+0000","last_clean":"2024-03-24T00:34:12.649544+0000","last_became_active":"2024-03-24T00:34:11.630881+0000","last_became_peered":"2024-03-24T00:34:11.630881+0000","last_unstale":"2024-03-24T00:34:12.649544+0000","last_undegraded":"2024-03-24T00:34:12.649544+0000","last_fullsized":"2024-03-24T00:34:12.649544+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:17.777645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.633859+0000","last_change":"2024-03-24T00:34:12.628634+0000","last_active":"2024-03-24T00:34:12.633859+0000","last_peered":"2024-03-24T00:34:12.633859+0000","last_clean":"2024-03-24T00:34:12.633859+0000","last_became_active":"2024-03-24T00:34:12.628377+0000","last_became_peered":"2024-03-24T00:34:12.628377+0000","last_unstale":"2024-03-24T00:34:12.633859+0000","last_undegraded":"2024-03-24T00:34:12.633859+0000","last_fullsized":"2024-03-24T00:34:12.633859+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:23.672349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.643389+0000","last_change":"2024-03-24T00:34:11.636086+0000","last_active":"2024-03-24T00:34:12.643389+0000","last_peered":"2024-03-24T00:34:12.643389+0000","last_clean":"2024-03-24T00:34:12.643389+0000","last_became_active":"2024-03-24T00:34:11.635800+0000","last_became_peered":"2024-03-24T00:34:11.635800+0000","last_unstale":"2024-03-24T00:34:12.643389+0000","last_undegraded":"2024-03-24T00:34:12.643389+0000","last_fullsized":"2024-03-24T00:34:12.643389+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:31.855764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641738+0000","last_change":"2024-03-24T00:34:12.631462+0000","last_active":"2024-03-24T00:34:12.641738+0000","last_peered":"2024-03-24T00:34:12.641738+0000","last_clean":"2024-03-24T00:34:12.641738+0000","last_became_active":"2024-03-24T00:34:12.630972+0000","last_became_peered":"2024-03-24T00:34:12.630972+0000","last_unstale":"2024-03-24T00:34:12.641738+0000","last_undegraded":"2024-03-24T00:34:12.641738+0000","last_fullsized":"2024-03-24T00:34:12.641738+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:39.516699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.651967+0000","last_change":"2024-03-24T00:34:11.632863+0000","last_active":"2024-03-24T00:34:12.651967+0000","last_peered":"2024-03-24T00:34:12.651967+0000","last_clean":"2024-03-24T00:34:12.651967+0000","last_became_active":"2024-03-24T00:34:11.632425+0000","last_became_peered":"2024-03-24T00:34:11.632425+0000","last_unstale":"2024-03-24T00:34:12.651967+0000","last_undegraded":"2024-03-24T00:34:12.651967+0000","last_fullsized":"2024-03-24T00:34:12.651967+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:16.733803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.651471+0000","last_change":"2024-03-24T00:34:12.645022+0000","last_active":"2024-03-24T00:34:12.651471+0000","last_peered":"2024-03-24T00:34:12.651471+0000","last_clean":"2024-03-24T00:34:12.651471+0000","last_became_active":"2024-03-24T00:34:12.644737+0000","last_became_peered":"2024-03-24T00:34:12.644737+0000","last_unstale":"2024-03-24T00:34:12.651471+0000","last_undegraded":"2024-03-24T00:34:12.651471+0000","last_fullsized":"2024-03-24T00:34:12.651471+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:04.494411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.622745+0000","last_change":"2024-03-24T00:34:11.637661+0000","last_active":"2024-03-24T00:34:12.622745+0000","last_peered":"2024-03-24T00:34:12.622745+0000","last_clean":"2024-03-24T00:34:12.622745+0000","last_became_active":"2024-03-24T00:34:11.637118+0000","last_became_peered":"2024-03-24T00:34:11.637118+0000","last_unstale":"2024-03-24T00:34:12.622745+0000","last_undegraded":"2024-03-24T00:34:12.622745+0000","last_fullsized":"2024-03-24T00:34:12.622745+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:05.021404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.654299+0000","last_change":"2024-03-24T00:34:12.650455+0000","last_active":"2024-03-24T00:34:12.654299+0000","last_peered":"2024-03-24T00:34:12.654299+0000","last_clean":"2024-03-24T00:34:12.654299+0000","last_became_active":"2024-03-24T00:34:12.650186+0000","last_became_peered":"2024-03-24T00:34:12.650186+0000","last_unstale":"2024-03-24T00:34:12.654299+0000","last_undegraded":"2024-03-24T00:34:12.654299+0000","last_fullsized":"2024-03-24T00:34:12.654299+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:42.944475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.625138+0000","last_change":"2024-03-24T00:34:11.637350+0000","last_active":"2024-03-24T00:34:12.625138+0000","last_peered":"2024-03-24T00:34:12.625138+0000","last_clean":"2024-03-24T00:34:12.625138+0000","last_became_active":"2024-03-24T00:34:11.637088+0000","last_became_peered":"2024-03-24T00:34:11.637088+0000","last_unstale":"2024-03-24T00:34:12.625138+0000","last_undegraded":"2024-03-24T00:34:12.625138+0000","last_fullsized":"2024-03-24T00:34:12.625138+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:22.665321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641659+0000","last_change":"2024-03-24T00:34:12.632330+0000","last_active":"2024-03-24T00:34:12.641659+0000","last_peered":"2024-03-24T00:34:12.641659+0000","last_clean":"2024-03-24T00:34:12.641659+0000","last_became_active":"2024-03-24T00:34:12.631976+0000","last_became_peered":"2024-03-24T00:34:12.631976+0000","last_unstale":"2024-03-24T00:34:12.641659+0000","last_undegraded":"2024-03-24T00:34:12.641659+0000","last_fullsized":"2024-03-24T00:34:12.641659+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:01.823045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.646335+0000","last_change":"2024-03-24T00:34:11.626734+0000","last_active":"2024-03-24T00:34:12.646335+0000","last_peered":"2024-03-24T00:34:12.646335+0000","last_clean":"2024-03-24T00:34:12.646335+0000","last_became_active":"2024-03-24T00:34:11.624092+0000","last_became_peered":"2024-03-24T00:34:11.624092+0000","last_unstale":"2024-03-24T00:34:12.646335+0000","last_undegraded":"2024-03-24T00:34:12.646335+0000","last_fullsized":"2024-03-24T00:34:12.646335+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:50:53.986736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641308+0000","last_change":"2024-03-24T00:34:12.634750+0000","last_active":"2024-03-24T00:34:12.641308+0000","last_peered":"2024-03-24T00:34:12.641308+0000","last_clean":"2024-03-24T00:34:12.641308+0000","last_became_active":"2024-03-24T00:34:12.634451+0000","last_became_peered":"2024-03-24T00:34:12.634451+0000","last_unstale":"2024-03-24T00:34:12.641308+0000","last_undegraded":"2024-03-24T00:34:12.641308+0000","last_fullsized":"2024-03-24T00:34:12.641308+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:34.956494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641839+0000","last_change":"2024-03-24T00:34:11.634704+0000","last_active":"2024-03-24T00:34:12.641839+0000","last_peered":"2024-03-24T00:34:12.641839+0000","last_clean":"2024-03-24T00:34:12.641839+0000","last_became_active":"2024-03-24T00:34:11.634339+0000","last_became_peered":"2024-03-24T00:34:11.634339+0000","last_unstale":"2024-03-24T00:34:12.641839+0000","last_undegraded":"2024-03-24T00:34:12.641839+0000","last_fullsized":"2024-03-24T00:34:12.641839+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:59.361561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.649418+0000","last_change":"2024-03-24T00:34:12.645562+0000","last_active":"2024-03-24T00:34:12.649418+0000","last_peered":"2024-03-24T00:34:12.649418+0000","last_clean":"2024-03-24T00:34:12.649418+0000","last_became_active":"2024-03-24T00:34:12.644988+0000","last_became_peered":"2024-03-24T00:34:12.644988+0000","last_unstale":"2024-03-24T00:34:12.649418+0000","last_undegraded":"2024-03-24T00:34:12.649418+0000","last_fullsized":"2024-03-24T00:34:12.649418+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:07:15.742707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.623404+0000","last_change":"2024-03-24T00:34:11.629779+0000","last_active":"2024-03-24T00:34:12.623404+0000","last_peered":"2024-03-24T00:34:12.623404+0000","last_clean":"2024-03-24T00:34:12.623404+0000","last_became_active":"2024-03-24T00:34:11.629391+0000","last_became_peered":"2024-03-24T00:34:11.629391+0000","last_unstale":"2024-03-24T00:34:12.623404+0000","last_undegraded":"2024-03-24T00:34:12.623404+0000","last_fullsized":"2024-03-24T00:34:12.623404+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:11.477318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.651193+0000","last_change":"2024-03-24T00:34:12.647184+0000","last_active":"2024-03-24T00:34:12.651193+0000","last_peered":"2024-03-24T00:34:12.651193+0000","last_clean":"2024-03-24T00:34:12.651193+0000","last_became_active":"2024-03-24T00:34:12.646855+0000","last_became_peered":"2024-03-24T00:34:12.646855+0000","last_unstale":"2024-03-24T00:34:12.651193+0000","last_undegraded":"2024-03-24T00:34:12.651193+0000","last_fullsized":"2024-03-24T00:34:12.651193+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:39.516699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.625290+0000","last_change":"2024-03-24T00:34:11.636495+0000","last_active":"2024-03-24T00:34:12.625290+0000","last_peered":"2024-03-24T00:34:12.625290+0000","last_clean":"2024-03-24T00:34:12.625290+0000","last_became_active":"2024-03-24T00:34:11.636125+0000","last_became_peered":"2024-03-24T00:34:11.636125+0000","last_unstale":"2024-03-24T00:34:12.625290+0000","last_undegraded":"2024-03-24T00:34:12.625290+0000","last_fullsized":"2024-03-24T00:34:12.625290+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:17.624215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.638461+0000","last_change":"2024-03-24T00:34:12.636338+0000","last_active":"2024-03-24T00:34:12.638461+0000","last_peered":"2024-03-24T00:34:12.638461+0000","last_clean":"2024-03-24T00:34:12.638461+0000","last_became_active":"2024-03-24T00:34:12.636009+0000","last_became_peered":"2024-03-24T00:34:12.636009+0000","last_unstale":"2024-03-24T00:34:12.638461+0000","last_undegraded":"2024-03-24T00:34:12.638461+0000","last_fullsized":"2024-03-24T00:34:12.638461+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:58.812833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.622739+0000","last_change":"2024-03-24T00:34:11.629175+0000","last_active":"2024-03-24T00:34:12.622739+0000","last_peered":"2024-03-24T00:34:12.622739+0000","last_clean":"2024-03-24T00:34:12.622739+0000","last_became_active":"2024-03-24T00:34:11.628884+0000","last_became_peered":"2024-03-24T00:34:11.628884+0000","last_unstale":"2024-03-24T00:34:12.622739+0000","last_undegraded":"2024-03-24T00:34:12.622739+0000","last_fullsized":"2024-03-24T00:34:12.622739+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:09.962122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.649821+0000","last_change":"2024-03-24T00:34:12.642381+0000","last_active":"2024-03-24T00:34:12.649821+0000","last_peered":"2024-03-24T00:34:12.649821+0000","last_clean":"2024-03-24T00:34:12.649821+0000","last_became_active":"2024-03-24T00:34:12.642017+0000","last_became_peered":"2024-03-24T00:34:12.642017+0000","last_unstale":"2024-03-24T00:34:12.649821+0000","last_undegraded":"2024-03-24T00:34:12.649821+0000","last_fullsized":"2024-03-24T00:34:12.649821+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:04.494411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.640569+0000","last_change":"2024-03-24T00:34:11.640663+0000","last_active":"2024-03-24T00:34:12.640569+0000","last_peered":"2024-03-24T00:34:12.640569+0000","last_clean":"2024-03-24T00:34:12.640569+0000","last_became_active":"2024-03-24T00:34:11.640331+0000","last_became_peered":"2024-03-24T00:34:11.640331+0000","last_unstale":"2024-03-24T00:34:12.640569+0000","last_undegraded":"2024-03-24T00:34:12.640569+0000","last_fullsized":"2024-03-24T00:34:12.640569+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:39.087307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.635253+0000","last_change":"2024-03-24T00:34:12.629894+0000","last_active":"2024-03-24T00:34:12.635253+0000","last_peered":"2024-03-24T00:34:12.635253+0000","last_clean":"2024-03-24T00:34:12.635253+0000","last_became_active":"2024-03-24T00:34:12.629615+0000","last_became_peered":"2024-03-24T00:34:12.629615+0000","last_unstale":"2024-03-24T00:34:12.635253+0000","last_undegraded":"2024-03-24T00:34:12.635253+0000","last_fullsized":"2024-03-24T00:34:12.635253+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:42:42.768882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.622739+0000","last_change":"2024-03-24T00:34:11.639715+0000","last_active":"2024-03-24T00:34:12.622739+0000","last_peered":"2024-03-24T00:34:12.622739+0000","last_clean":"2024-03-24T00:34:12.622739+0000","last_became_active":"2024-03-24T00:34:11.639093+0000","last_became_peered":"2024-03-24T00:34:11.639093+0000","last_unstale":"2024-03-24T00:34:12.622739+0000","last_undegraded":"2024-03-24T00:34:12.622739+0000","last_fullsized":"2024-03-24T00:34:12.622739+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:11.477318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.635973+0000","last_change":"2024-03-24T00:34:12.631422+0000","last_active":"2024-03-24T00:34:12.635973+0000","last_peered":"2024-03-24T00:34:12.635973+0000","last_clean":"2024-03-24T00:34:12.635973+0000","last_became_active":"2024-03-24T00:34:12.631111+0000","last_became_peered":"2024-03-24T00:34:12.631111+0000","last_unstale":"2024-03-24T00:34:12.635973+0000","last_undegraded":"2024-03-24T00:34:12.635973+0000","last_fullsized":"2024-03-24T00:34:12.635973+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:13.255425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.624212+0000","last_change":"2024-03-24T00:34:11.634811+0000","last_active":"2024-03-24T00:34:12.624212+0000","last_peered":"2024-03-24T00:34:12.624212+0000","last_clean":"2024-03-24T00:34:12.624212+0000","last_became_active":"2024-03-24T00:34:11.634504+0000","last_became_peered":"2024-03-24T00:34:11.634504+0000","last_unstale":"2024-03-24T00:34:12.624212+0000","last_undegraded":"2024-03-24T00:34:12.624212+0000","last_fullsized":"2024-03-24T00:34:12.624212+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:53:48.596023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.652084+0000","last_change":"2024-03-24T00:34:12.648047+0000","last_active":"2024-03-24T00:34:12.652084+0000","last_peered":"2024-03-24T00:34:12.652084+0000","last_clean":"2024-03-24T00:34:12.652084+0000","last_became_active":"2024-03-24T00:34:12.647723+0000","last_became_peered":"2024-03-24T00:34:12.647723+0000","last_unstale":"2024-03-24T00:34:12.652084+0000","last_undegraded":"2024-03-24T00:34:12.652084+0000","last_fullsized":"2024-03-24T00:34:12.652084+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:31:37.091502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.623134+0000","last_change":"2024-03-24T00:34:11.633822+0000","last_active":"2024-03-24T00:34:12.623134+0000","last_peered":"2024-03-24T00:34:12.623134+0000","last_clean":"2024-03-24T00:34:12.623134+0000","last_became_active":"2024-03-24T00:34:11.633543+0000","last_became_peered":"2024-03-24T00:34:11.633543+0000","last_unstale":"2024-03-24T00:34:12.623134+0000","last_undegraded":"2024-03-24T00:34:12.623134+0000","last_fullsized":"2024-03-24T00:34:12.623134+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:48:56.784074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":247,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.617548+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:34:12.617548+0000","last_peered":"2024-03-24T00:34:12.617548+0000","last_clean":"2024-03-24T00:34:12.617548+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:34:12.617548+0000","last_undegraded":"2024-03-24T00:34:12.617548+0000","last_fullsized":"2024-03-24T00:34:12.617548+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1298432,"data_stored":884174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":293646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":704512,"data_stored":294298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36032,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335808,"statfs":{"total":96636764160,"available":96599867392,"internally_reserved":0,"allocated":708608,"data_stored":295258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":293702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":704512,"data_stored":293702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":700416,"data_stored":293736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32428,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339412,"statfs":{"total":96636764160,"available":96603557888,"internally_reserved":0,"allocated":1277952,"data_stored":877239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:34:16.753 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:34:16.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.927+0000 7ff028000640 1 Processor -- start 2024-03-24T00:34:16.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.927+0000 7ff028000640 1 -- start start 2024-03-24T00:34:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff028000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff02006e620 0x7ff0200b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff028000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff02006ea00 0x7ff0200b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff028000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff020070f60 con 0x7ff0200b2ff0 2024-03-24T00:34:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff028000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0200b4c00 con 0x7ff02006e620 2024-03-24T00:34:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff028000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0200b3620 con 0x7ff02006ea00 2024-03-24T00:34:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff025600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff02006e620 0x7ff0200b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff025600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff02006e620 0x7ff0200b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50240/0 (socket says 172.21.15.40:50240) 2024-03-24T00:34:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff025600640 1 -- 172.21.15.40:0/921859333 learned_addr learned my addr 172.21.15.40:0/921859333 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff024c00640 1 --2- 172.21.15.40:0/921859333 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff02006ea00 0x7ff0200b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff025600640 1 -- 172.21.15.40:0/921859333 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff02006ea00 msgr2=0x7ff0200b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff025600640 1 --2- 172.21.15.40:0/921859333 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff02006ea00 0x7ff0200b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff025600640 1 -- 172.21.15.40:0/921859333 >> v1:172.21.15.53:6789/0 conn(0x7ff0200b2ff0 legacy=0x7ff0200ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff025600640 1 -- 172.21.15.40:0/921859333 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0200b3900 con 0x7ff02006e620 2024-03-24T00:34:16.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.931+0000 7ff025600640 1 --2- 172.21.15.40:0/921859333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff02006e620 0x7ff0200b46c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff014009870 tx=0x7ff01401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65f781cea0f9e207 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:16.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff01fe00640 1 -- 172.21.15.40:0/921859333 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0140b1070 con 0x7ff02006e620 2024-03-24T00:34:16.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff01fe00640 1 -- 172.21.15.40:0/921859333 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0140ac620 con 0x7ff02006e620 2024-03-24T00:34:16.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff01fe00640 1 -- 172.21.15.40:0/921859333 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0140ac9f0 con 0x7ff02006e620 2024-03-24T00:34:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 -- 172.21.15.40:0/921859333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff02006e620 msgr2=0x7ff0200b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 --2- 172.21.15.40:0/921859333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff02006e620 0x7ff0200b46c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff014009870 tx=0x7ff01401c4d0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 -- 172.21.15.40:0/921859333 shutdown_connections 2024-03-24T00:34:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 --2- 172.21.15.40:0/921859333 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff02006ea00 0x7ff0200b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 --2- 172.21.15.40:0/921859333 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff02006e620 0x7ff0200b46c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 -- 172.21.15.40:0/921859333 >> 172.21.15.40:0/921859333 conn(0x7ff0200b17b0 msgr2=0x7ff0200b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 -- 172.21.15.40:0/921859333 shutdown_connections 2024-03-24T00:34:16.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 -- 172.21.15.40:0/921859333 wait complete. 2024-03-24T00:34:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 Processor -- start 2024-03-24T00:34:16.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 -- start start 2024-03-24T00:34:16.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff02006e620 0x7ff020143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff02006ea00 0x7ff0201597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0200b2ff0 0x7ff02015d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff020144920 con 0x7ff0200b2ff0 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02006ede0 con 0x7ff02006ea00 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff028000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff020143620 con 0x7ff02006e620 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff026000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0200b2ff0 0x7ff02015d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff024c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff02006ea00 0x7ff0201597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff026000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0200b2ff0 0x7ff02015d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:50254/0 (socket says 172.21.15.40:50254) 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff026000640 1 -- 172.21.15.40:0/1233424647 learned_addr learned my addr 172.21.15.40:0/1233424647 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.935+0000 7ff025600640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff02006e620 0x7ff020143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff026000640 1 -- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff02006e620 msgr2=0x7ff020143ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff026000640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff02006e620 0x7ff020143ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff026000640 1 -- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff02006ea00 msgr2=0x7ff0201597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff026000640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff02006ea00 0x7ff0201597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:16.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff026000640 1 -- 172.21.15.40:0/1233424647 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff020143ff0 con 0x7ff0200b2ff0 2024-03-24T00:34:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff026000640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0200b2ff0 0x7ff02015d540 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff01800c730 tx=0x7ff01800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff01d600640 1 -- 172.21.15.40:0/1233424647 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff018003b80 con 0x7ff0200b2ff0 2024-03-24T00:34:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff01d600640 1 -- 172.21.15.40:0/1233424647 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff018003d20 con 0x7ff0200b2ff0 2024-03-24T00:34:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff01d600640 1 -- 172.21.15.40:0/1233424647 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01800aa30 con 0x7ff0200b2ff0 2024-03-24T00:34:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff028000640 1 -- 172.21.15.40:0/1233424647 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff020142a50 con 0x7ff0200b2ff0 2024-03-24T00:34:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff028000640 1 -- 172.21.15.40:0/1233424647 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff020142fd0 con 0x7ff0200b2ff0 2024-03-24T00:34:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff028000640 1 -- 172.21.15.40:0/1233424647 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff020068170 con 0x7ff0200b2ff0 2024-03-24T00:34:16.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff01d600640 1 -- 172.21.15.40:0/1233424647 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff01800abd0 con 0x7ff0200b2ff0 2024-03-24T00:34:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff01d600640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff000067690 0x7ff000069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:16.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff01d600640 1 -- 172.21.15.40:0/1233424647 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff018085630 con 0x7ff0200b2ff0 2024-03-24T00:34:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.939+0000 7ff025600640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff000067690 0x7ff000069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:16.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.943+0000 7ff025600640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff000067690 0x7ff000069b50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff0140ae000 tx=0x7ff0140a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:16.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:16.943+0000 7ff01d600640 1 -- 172.21.15.40:0/1233424647 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff018055bd0 con 0x7ff0200b2ff0 2024-03-24T00:34:17.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.107+0000 7ff028000640 1 -- 172.21.15.40:0/1233424647 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff020072f50 con 0x7ff000067690 2024-03-24T00:34:17.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.111+0000 7ff01d600640 1 -- 172.21.15.40:0/1233424647 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251887 (secure 0 0 0) 0x7ff020072f50 con 0x7ff000067690 2024-03-24T00:34:17.118 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:17.119 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:34:17.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.115+0000 7ff028000640 1 -- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff000067690 msgr2=0x7ff000069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.115+0000 7ff028000640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff000067690 0x7ff000069b50 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff0140ae000 tx=0x7ff0140a7000 comp rx=0 tx=0).stop 2024-03-24T00:34:17.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.115+0000 7ff028000640 1 -- 172.21.15.40:0/1233424647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0200b2ff0 msgr2=0x7ff02015d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.115+0000 7ff028000640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0200b2ff0 0x7ff02015d540 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff01800c730 tx=0x7ff01800cbf0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.115+0000 7ff028000640 1 -- 172.21.15.40:0/1233424647 shutdown_connections 2024-03-24T00:34:17.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.115+0000 7ff028000640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff000067690 0x7ff000069b50 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.115+0000 7ff028000640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff0200b2ff0 0x7ff02015d540 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.115+0000 7ff028000640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff02006ea00 0x7ff0201597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.115+0000 7ff028000640 1 --2- 172.21.15.40:0/1233424647 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff02006e620 0x7ff020143ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.115+0000 7ff028000640 1 -- 172.21.15.40:0/1233424647 >> 172.21.15.40:0/1233424647 conn(0x7ff0200b17b0 msgr2=0x7ff0200b0390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:17.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.115+0000 7ff028000640 1 -- 172.21.15.40:0/1233424647 shutdown_connections 2024-03-24T00:34:17.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.115+0000 7ff028000640 1 -- 172.21.15.40:0/1233424647 wait complete. 2024-03-24T00:34:17.138 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-03-24T00:34:15.237086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260268,"kb_used_data":6636,"kb_used_omap":12,"kb_used_meta":253491,"kb_avail":754714452,"statfs":{"total":773094113280,"available":772827598848,"internally_reserved":0,"allocated":6795264,"data_stored":3525755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":259575346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.632136"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641391+0000","last_change":"2024-03-24T00:34:12.639810+0000","last_active":"2024-03-24T00:34:12.641391+0000","last_peered":"2024-03-24T00:34:12.641391+0000","last_clean":"2024-03-24T00:34:12.641391+0000","last_became_active":"2024-03-24T00:34:12.639540+0000","last_became_peered":"2024-03-24T00:34:12.639540+0000","last_unstale":"2024-03-24T00:34:12.641391+0000","last_undegraded":"2024-03-24T00:34:12.641391+0000","last_fullsized":"2024-03-24T00:34:12.641391+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:03:12.808236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.637648+0000","last_change":"2024-03-24T00:34:12.635073+0000","last_active":"2024-03-24T00:34:12.637648+0000","last_peered":"2024-03-24T00:34:12.637648+0000","last_clean":"2024-03-24T00:34:12.637648+0000","last_became_active":"2024-03-24T00:34:12.634805+0000","last_became_peered":"2024-03-24T00:34:12.634805+0000","last_unstale":"2024-03-24T00:34:12.637648+0000","last_undegraded":"2024-03-24T00:34:12.637648+0000","last_fullsized":"2024-03-24T00:34:12.637648+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:34.398454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.651737+0000","last_change":"2024-03-24T00:34:12.649659+0000","last_active":"2024-03-24T00:34:12.651737+0000","last_peered":"2024-03-24T00:34:12.651737+0000","last_clean":"2024-03-24T00:34:12.651737+0000","last_became_active":"2024-03-24T00:34:12.649195+0000","last_became_peered":"2024-03-24T00:34:12.649195+0000","last_unstale":"2024-03-24T00:34:12.651737+0000","last_undegraded":"2024-03-24T00:34:12.651737+0000","last_fullsized":"2024-03-24T00:34:12.651737+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:42.944475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641251+0000","last_change":"2024-03-24T00:34:12.632651+0000","last_active":"2024-03-24T00:34:12.641251+0000","last_peered":"2024-03-24T00:34:12.641251+0000","last_clean":"2024-03-24T00:34:12.641251+0000","last_became_active":"2024-03-24T00:34:12.632369+0000","last_became_peered":"2024-03-24T00:34:12.632369+0000","last_unstale":"2024-03-24T00:34:12.641251+0000","last_undegraded":"2024-03-24T00:34:12.641251+0000","last_fullsized":"2024-03-24T00:34:12.641251+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:42.944475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.632542+0000","last_change":"2024-03-24T00:34:12.627518+0000","last_active":"2024-03-24T00:34:12.632542+0000","last_peered":"2024-03-24T00:34:12.632542+0000","last_clean":"2024-03-24T00:34:12.632542+0000","last_became_active":"2024-03-24T00:34:12.627230+0000","last_became_peered":"2024-03-24T00:34:12.627230+0000","last_unstale":"2024-03-24T00:34:12.632542+0000","last_undegraded":"2024-03-24T00:34:12.632542+0000","last_fullsized":"2024-03-24T00:34:12.632542+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:40:22.231804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.656771+0000","last_change":"2024-03-24T00:34:12.653958+0000","last_active":"2024-03-24T00:34:12.656771+0000","last_peered":"2024-03-24T00:34:12.656771+0000","last_clean":"2024-03-24T00:34:12.656771+0000","last_became_active":"2024-03-24T00:34:12.653671+0000","last_became_peered":"2024-03-24T00:34:12.653671+0000","last_unstale":"2024-03-24T00:34:12.656771+0000","last_undegraded":"2024-03-24T00:34:12.656771+0000","last_fullsized":"2024-03-24T00:34:12.656771+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:40:34.322294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.647131+0000","last_change":"2024-03-24T00:34:12.637902+0000","last_active":"2024-03-24T00:34:12.647131+0000","last_peered":"2024-03-24T00:34:12.647131+0000","last_clean":"2024-03-24T00:34:12.647131+0000","last_became_active":"2024-03-24T00:34:12.637266+0000","last_became_peered":"2024-03-24T00:34:12.637266+0000","last_unstale":"2024-03-24T00:34:12.647131+0000","last_undegraded":"2024-03-24T00:34:12.647131+0000","last_fullsized":"2024-03-24T00:34:12.647131+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:22:49.803500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.642885+0000","last_change":"2024-03-24T00:34:12.636304+0000","last_active":"2024-03-24T00:34:12.642885+0000","last_peered":"2024-03-24T00:34:12.642885+0000","last_clean":"2024-03-24T00:34:12.642885+0000","last_became_active":"2024-03-24T00:34:12.635975+0000","last_became_peered":"2024-03-24T00:34:12.635975+0000","last_unstale":"2024-03-24T00:34:12.642885+0000","last_undegraded":"2024-03-24T00:34:12.642885+0000","last_fullsized":"2024-03-24T00:34:12.642885+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:20:06.851983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.654287+0000","last_change":"2024-03-24T00:34:12.653549+0000","last_active":"2024-03-24T00:34:12.654287+0000","last_peered":"2024-03-24T00:34:12.654287+0000","last_clean":"2024-03-24T00:34:12.654287+0000","last_became_active":"2024-03-24T00:34:12.653201+0000","last_became_peered":"2024-03-24T00:34:12.653201+0000","last_unstale":"2024-03-24T00:34:12.654287+0000","last_undegraded":"2024-03-24T00:34:12.654287+0000","last_fullsized":"2024-03-24T00:34:12.654287+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:18.631243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.630392+0000","last_change":"2024-03-24T00:34:12.628664+0000","last_active":"2024-03-24T00:34:12.630392+0000","last_peered":"2024-03-24T00:34:12.630392+0000","last_clean":"2024-03-24T00:34:12.630392+0000","last_became_active":"2024-03-24T00:34:12.628363+0000","last_became_peered":"2024-03-24T00:34:12.628363+0000","last_unstale":"2024-03-24T00:34:12.630392+0000","last_undegraded":"2024-03-24T00:34:12.630392+0000","last_fullsized":"2024-03-24T00:34:12.630392+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:26.884066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.643087+0000","last_change":"2024-03-24T00:34:12.639146+0000","last_active":"2024-03-24T00:34:12.643087+0000","last_peered":"2024-03-24T00:34:12.643087+0000","last_clean":"2024-03-24T00:34:12.643087+0000","last_became_active":"2024-03-24T00:34:12.638873+0000","last_became_peered":"2024-03-24T00:34:12.638873+0000","last_unstale":"2024-03-24T00:34:12.643087+0000","last_undegraded":"2024-03-24T00:34:12.643087+0000","last_fullsized":"2024-03-24T00:34:12.643087+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:30.727741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.630305+0000","last_change":"2024-03-24T00:34:12.628970+0000","last_active":"2024-03-24T00:34:12.630305+0000","last_peered":"2024-03-24T00:34:12.630305+0000","last_clean":"2024-03-24T00:34:12.630305+0000","last_became_active":"2024-03-24T00:34:12.628618+0000","last_became_peered":"2024-03-24T00:34:12.628618+0000","last_unstale":"2024-03-24T00:34:12.630305+0000","last_undegraded":"2024-03-24T00:34:12.630305+0000","last_fullsized":"2024-03-24T00:34:12.630305+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:04.494411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.652407+0000","last_change":"2024-03-24T00:34:12.646984+0000","last_active":"2024-03-24T00:34:12.652407+0000","last_peered":"2024-03-24T00:34:12.652407+0000","last_clean":"2024-03-24T00:34:12.652407+0000","last_became_active":"2024-03-24T00:34:12.646453+0000","last_became_peered":"2024-03-24T00:34:12.646453+0000","last_unstale":"2024-03-24T00:34:12.652407+0000","last_undegraded":"2024-03-24T00:34:12.652407+0000","last_fullsized":"2024-03-24T00:34:12.652407+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:26.884066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.647263+0000","last_change":"2024-03-24T00:34:12.640749+0000","last_active":"2024-03-24T00:34:12.647263+0000","last_peered":"2024-03-24T00:34:12.647263+0000","last_clean":"2024-03-24T00:34:12.647263+0000","last_became_active":"2024-03-24T00:34:12.640287+0000","last_became_peered":"2024-03-24T00:34:12.640287+0000","last_unstale":"2024-03-24T00:34:12.647263+0000","last_undegraded":"2024-03-24T00:34:12.647263+0000","last_fullsized":"2024-03-24T00:34:12.647263+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:19.306192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.644791+0000","last_change":"2024-03-24T00:34:12.635211+0000","last_active":"2024-03-24T00:34:12.644791+0000","last_peered":"2024-03-24T00:34:12.644791+0000","last_clean":"2024-03-24T00:34:12.644791+0000","last_became_active":"2024-03-24T00:34:12.634736+0000","last_became_peered":"2024-03-24T00:34:12.634736+0000","last_unstale":"2024-03-24T00:34:12.644791+0000","last_undegraded":"2024-03-24T00:34:12.644791+0000","last_fullsized":"2024-03-24T00:34:12.644791+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:12:28.260973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.656822+0000","last_change":"2024-03-24T00:34:12.654963+0000","last_active":"2024-03-24T00:34:12.656822+0000","last_peered":"2024-03-24T00:34:12.656822+0000","last_clean":"2024-03-24T00:34:12.656822+0000","last_became_active":"2024-03-24T00:34:12.654655+0000","last_became_peered":"2024-03-24T00:34:12.654655+0000","last_unstale":"2024-03-24T00:34:12.656822+0000","last_undegraded":"2024-03-24T00:34:12.656822+0000","last_fullsized":"2024-03-24T00:34:12.656822+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:27.215945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.643035+0000","last_change":"2024-03-24T00:34:12.637403+0000","last_active":"2024-03-24T00:34:12.643035+0000","last_peered":"2024-03-24T00:34:12.643035+0000","last_clean":"2024-03-24T00:34:12.643035+0000","last_became_active":"2024-03-24T00:34:12.637088+0000","last_became_peered":"2024-03-24T00:34:12.637088+0000","last_unstale":"2024-03-24T00:34:12.643035+0000","last_undegraded":"2024-03-24T00:34:12.643035+0000","last_fullsized":"2024-03-24T00:34:12.643035+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:30:29.388659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.653709+0000","last_change":"2024-03-24T00:34:12.651405+0000","last_active":"2024-03-24T00:34:12.653709+0000","last_peered":"2024-03-24T00:34:12.653709+0000","last_clean":"2024-03-24T00:34:12.653709+0000","last_became_active":"2024-03-24T00:34:12.651070+0000","last_became_peered":"2024-03-24T00:34:12.651070+0000","last_unstale":"2024-03-24T00:34:12.653709+0000","last_undegraded":"2024-03-24T00:34:12.653709+0000","last_fullsized":"2024-03-24T00:34:12.653709+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:45:48.010797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641444+0000","last_change":"2024-03-24T00:34:12.639846+0000","last_active":"2024-03-24T00:34:12.641444+0000","last_peered":"2024-03-24T00:34:12.641444+0000","last_clean":"2024-03-24T00:34:12.641444+0000","last_became_active":"2024-03-24T00:34:12.639593+0000","last_became_peered":"2024-03-24T00:34:12.639593+0000","last_unstale":"2024-03-24T00:34:12.641444+0000","last_undegraded":"2024-03-24T00:34:12.641444+0000","last_fullsized":"2024-03-24T00:34:12.641444+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:24:56.769496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.652706+0000","last_change":"2024-03-24T00:34:12.648194+0000","last_active":"2024-03-24T00:34:12.652706+0000","last_peered":"2024-03-24T00:34:12.652706+0000","last_clean":"2024-03-24T00:34:12.652706+0000","last_became_active":"2024-03-24T00:34:12.647893+0000","last_became_peered":"2024-03-24T00:34:12.647893+0000","last_unstale":"2024-03-24T00:34:12.652706+0000","last_undegraded":"2024-03-24T00:34:12.652706+0000","last_fullsized":"2024-03-24T00:34:12.652706+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:32:56.873303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.646365+0000","last_change":"2024-03-24T00:34:12.633174+0000","last_active":"2024-03-24T00:34:12.646365+0000","last_peered":"2024-03-24T00:34:12.646365+0000","last_clean":"2024-03-24T00:34:12.646365+0000","last_became_active":"2024-03-24T00:34:12.632921+0000","last_became_peered":"2024-03-24T00:34:12.632921+0000","last_unstale":"2024-03-24T00:34:12.646365+0000","last_undegraded":"2024-03-24T00:34:12.646365+0000","last_fullsized":"2024-03-24T00:34:12.646365+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:32.862791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.646832+0000","last_change":"2024-03-24T00:34:12.636922+0000","last_active":"2024-03-24T00:34:12.646832+0000","last_peered":"2024-03-24T00:34:12.646832+0000","last_clean":"2024-03-24T00:34:12.646832+0000","last_became_active":"2024-03-24T00:34:12.636616+0000","last_became_peered":"2024-03-24T00:34:12.636616+0000","last_unstale":"2024-03-24T00:34:12.646832+0000","last_undegraded":"2024-03-24T00:34:12.646832+0000","last_fullsized":"2024-03-24T00:34:12.646832+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:42:01.684593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.650735+0000","last_change":"2024-03-24T00:34:12.646731+0000","last_active":"2024-03-24T00:34:12.650735+0000","last_peered":"2024-03-24T00:34:12.650735+0000","last_clean":"2024-03-24T00:34:12.650735+0000","last_became_active":"2024-03-24T00:34:12.643397+0000","last_became_peered":"2024-03-24T00:34:12.643397+0000","last_unstale":"2024-03-24T00:34:12.650735+0000","last_undegraded":"2024-03-24T00:34:12.650735+0000","last_fullsized":"2024-03-24T00:34:12.650735+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:01.823045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.652500+0000","last_change":"2024-03-24T00:34:12.646918+0000","last_active":"2024-03-24T00:34:12.652500+0000","last_peered":"2024-03-24T00:34:12.652500+0000","last_clean":"2024-03-24T00:34:12.652500+0000","last_became_active":"2024-03-24T00:34:12.646571+0000","last_became_peered":"2024-03-24T00:34:12.646571+0000","last_unstale":"2024-03-24T00:34:12.652500+0000","last_undegraded":"2024-03-24T00:34:12.652500+0000","last_fullsized":"2024-03-24T00:34:12.652500+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:50:54.993763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.634678+0000","last_change":"2024-03-24T00:34:12.629682+0000","last_active":"2024-03-24T00:34:12.634678+0000","last_peered":"2024-03-24T00:34:12.634678+0000","last_clean":"2024-03-24T00:34:12.634678+0000","last_became_active":"2024-03-24T00:34:12.629353+0000","last_became_peered":"2024-03-24T00:34:12.629353+0000","last_unstale":"2024-03-24T00:34:12.634678+0000","last_undegraded":"2024-03-24T00:34:12.634678+0000","last_fullsized":"2024-03-24T00:34:12.634678+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:35:40.543447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.650306+0000","last_change":"2024-03-24T00:34:12.648586+0000","last_active":"2024-03-24T00:34:12.650306+0000","last_peered":"2024-03-24T00:34:12.650306+0000","last_clean":"2024-03-24T00:34:12.650306+0000","last_became_active":"2024-03-24T00:34:12.648291+0000","last_became_peered":"2024-03-24T00:34:12.648291+0000","last_unstale":"2024-03-24T00:34:12.650306+0000","last_undegraded":"2024-03-24T00:34:12.650306+0000","last_fullsized":"2024-03-24T00:34:12.650306+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:43.825491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.656800+0000","last_change":"2024-03-24T00:34:12.654345+0000","last_active":"2024-03-24T00:34:12.656800+0000","last_peered":"2024-03-24T00:34:12.656800+0000","last_clean":"2024-03-24T00:34:12.656800+0000","last_became_active":"2024-03-24T00:34:12.653139+0000","last_became_peered":"2024-03-24T00:34:12.653139+0000","last_unstale":"2024-03-24T00:34:12.656800+0000","last_undegraded":"2024-03-24T00:34:12.656800+0000","last_fullsized":"2024-03-24T00:34:12.656800+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:42.435900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.634632+0000","last_change":"2024-03-24T00:34:12.629865+0000","last_active":"2024-03-24T00:34:12.634632+0000","last_peered":"2024-03-24T00:34:12.634632+0000","last_clean":"2024-03-24T00:34:12.634632+0000","last_became_active":"2024-03-24T00:34:12.628899+0000","last_became_peered":"2024-03-24T00:34:12.628899+0000","last_unstale":"2024-03-24T00:34:12.634632+0000","last_undegraded":"2024-03-24T00:34:12.634632+0000","last_fullsized":"2024-03-24T00:34:12.634632+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:19.394650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.643228+0000","last_change":"2024-03-24T00:34:12.638095+0000","last_active":"2024-03-24T00:34:12.643228+0000","last_peered":"2024-03-24T00:34:12.643228+0000","last_clean":"2024-03-24T00:34:12.643228+0000","last_became_active":"2024-03-24T00:34:12.637256+0000","last_became_peered":"2024-03-24T00:34:12.637256+0000","last_unstale":"2024-03-24T00:34:12.643228+0000","last_undegraded":"2024-03-24T00:34:12.643228+0000","last_fullsized":"2024-03-24T00:34:12.643228+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:10:44.804304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.650575+0000","last_change":"2024-03-24T00:34:12.644570+0000","last_active":"2024-03-24T00:34:12.650575+0000","last_peered":"2024-03-24T00:34:12.650575+0000","last_clean":"2024-03-24T00:34:12.650575+0000","last_became_active":"2024-03-24T00:34:12.644314+0000","last_became_peered":"2024-03-24T00:34:12.644314+0000","last_unstale":"2024-03-24T00:34:12.650575+0000","last_undegraded":"2024-03-24T00:34:12.650575+0000","last_fullsized":"2024-03-24T00:34:12.650575+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:49.371076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.630144+0000","last_change":"2024-03-24T00:34:12.625778+0000","last_active":"2024-03-24T00:34:12.630144+0000","last_peered":"2024-03-24T00:34:12.630144+0000","last_clean":"2024-03-24T00:34:12.630144+0000","last_became_active":"2024-03-24T00:34:12.625389+0000","last_became_peered":"2024-03-24T00:34:12.625389+0000","last_unstale":"2024-03-24T00:34:12.630144+0000","last_undegraded":"2024-03-24T00:34:12.630144+0000","last_fullsized":"2024-03-24T00:34:12.630144+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:05:37.918280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.656376+0000","last_change":"2024-03-24T00:34:12.652924+0000","last_active":"2024-03-24T00:34:12.656376+0000","last_peered":"2024-03-24T00:34:12.656376+0000","last_clean":"2024-03-24T00:34:12.656376+0000","last_became_active":"2024-03-24T00:34:12.651457+0000","last_became_peered":"2024-03-24T00:34:12.651457+0000","last_unstale":"2024-03-24T00:34:12.656376+0000","last_undegraded":"2024-03-24T00:34:12.656376+0000","last_fullsized":"2024-03-24T00:34:12.656376+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:43.825491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.651179+0000","last_change":"2024-03-24T00:34:12.647285+0000","last_active":"2024-03-24T00:34:12.651179+0000","last_peered":"2024-03-24T00:34:12.651179+0000","last_clean":"2024-03-24T00:34:12.651179+0000","last_became_active":"2024-03-24T00:34:12.647054+0000","last_became_peered":"2024-03-24T00:34:12.647054+0000","last_unstale":"2024-03-24T00:34:12.651179+0000","last_undegraded":"2024-03-24T00:34:12.651179+0000","last_fullsized":"2024-03-24T00:34:12.651179+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:01.823045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"58'4","reported_seq":16,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.656341+0000","last_change":"2024-03-24T00:34:11.635411+0000","last_active":"2024-03-24T00:34:12.656341+0000","last_peered":"2024-03-24T00:34:12.656341+0000","last_clean":"2024-03-24T00:34:12.656341+0000","last_became_active":"2024-03-24T00:34:11.635088+0000","last_became_peered":"2024-03-24T00:34:11.635088+0000","last_unstale":"2024-03-24T00:34:12.656341+0000","last_undegraded":"2024-03-24T00:34:12.656341+0000","last_fullsized":"2024-03-24T00:34:12.656341+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:17.777645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.637586+0000","last_change":"2024-03-24T00:34:12.633501+0000","last_active":"2024-03-24T00:34:12.637586+0000","last_peered":"2024-03-24T00:34:12.637586+0000","last_clean":"2024-03-24T00:34:12.637586+0000","last_became_active":"2024-03-24T00:34:12.633232+0000","last_became_peered":"2024-03-24T00:34:12.633232+0000","last_unstale":"2024-03-24T00:34:12.637586+0000","last_undegraded":"2024-03-24T00:34:12.637586+0000","last_fullsized":"2024-03-24T00:34:12.637586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:08:26.441734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.649761+0000","last_change":"2024-03-24T00:34:11.633628+0000","last_active":"2024-03-24T00:34:12.649761+0000","last_peered":"2024-03-24T00:34:12.649761+0000","last_clean":"2024-03-24T00:34:12.649761+0000","last_became_active":"2024-03-24T00:34:11.633358+0000","last_became_peered":"2024-03-24T00:34:11.633358+0000","last_unstale":"2024-03-24T00:34:12.649761+0000","last_undegraded":"2024-03-24T00:34:12.649761+0000","last_fullsized":"2024-03-24T00:34:12.649761+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:00:26.208918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.656313+0000","last_change":"2024-03-24T00:34:12.651300+0000","last_active":"2024-03-24T00:34:12.656313+0000","last_peered":"2024-03-24T00:34:12.656313+0000","last_clean":"2024-03-24T00:34:12.656313+0000","last_became_active":"2024-03-24T00:34:12.650101+0000","last_became_peered":"2024-03-24T00:34:12.650101+0000","last_unstale":"2024-03-24T00:34:12.656313+0000","last_undegraded":"2024-03-24T00:34:12.656313+0000","last_fullsized":"2024-03-24T00:34:12.656313+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:45:48.010797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.653176+0000","last_change":"2024-03-24T00:34:11.639622+0000","last_active":"2024-03-24T00:34:12.653176+0000","last_peered":"2024-03-24T00:34:12.653176+0000","last_clean":"2024-03-24T00:34:12.653176+0000","last_became_active":"2024-03-24T00:34:11.639347+0000","last_became_peered":"2024-03-24T00:34:11.639347+0000","last_unstale":"2024-03-24T00:34:12.653176+0000","last_undegraded":"2024-03-24T00:34:12.653176+0000","last_fullsized":"2024-03-24T00:34:12.653176+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:22.665321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.654845+0000","last_change":"2024-03-24T00:34:12.650537+0000","last_active":"2024-03-24T00:34:12.654845+0000","last_peered":"2024-03-24T00:34:12.654845+0000","last_clean":"2024-03-24T00:34:12.654845+0000","last_became_active":"2024-03-24T00:34:12.648402+0000","last_became_peered":"2024-03-24T00:34:12.648402+0000","last_unstale":"2024-03-24T00:34:12.654845+0000","last_undegraded":"2024-03-24T00:34:12.654845+0000","last_fullsized":"2024-03-24T00:34:12.654845+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:34.956494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.649864+0000","last_change":"2024-03-24T00:34:11.631927+0000","last_active":"2024-03-24T00:34:12.649864+0000","last_peered":"2024-03-24T00:34:12.649864+0000","last_clean":"2024-03-24T00:34:12.649864+0000","last_became_active":"2024-03-24T00:34:11.631553+0000","last_became_peered":"2024-03-24T00:34:11.631553+0000","last_unstale":"2024-03-24T00:34:12.649864+0000","last_undegraded":"2024-03-24T00:34:12.649864+0000","last_fullsized":"2024-03-24T00:34:12.649864+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:18.299165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.646855+0000","last_change":"2024-03-24T00:34:12.638309+0000","last_active":"2024-03-24T00:34:12.646855+0000","last_peered":"2024-03-24T00:34:12.646855+0000","last_clean":"2024-03-24T00:34:12.646855+0000","last_became_active":"2024-03-24T00:34:12.638032+0000","last_became_peered":"2024-03-24T00:34:12.638032+0000","last_unstale":"2024-03-24T00:34:12.646855+0000","last_undegraded":"2024-03-24T00:34:12.646855+0000","last_fullsized":"2024-03-24T00:34:12.646855+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:55:30.175598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.650706+0000","last_change":"2024-03-24T00:34:11.634138+0000","last_active":"2024-03-24T00:34:12.650706+0000","last_peered":"2024-03-24T00:34:12.650706+0000","last_clean":"2024-03-24T00:34:12.650706+0000","last_became_active":"2024-03-24T00:34:11.633644+0000","last_became_peered":"2024-03-24T00:34:11.633644+0000","last_unstale":"2024-03-24T00:34:12.650706+0000","last_undegraded":"2024-03-24T00:34:12.650706+0000","last_fullsized":"2024-03-24T00:34:12.650706+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:44:33.132978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.654865+0000","last_change":"2024-03-24T00:34:12.649115+0000","last_active":"2024-03-24T00:34:12.654865+0000","last_peered":"2024-03-24T00:34:12.654865+0000","last_clean":"2024-03-24T00:34:12.654865+0000","last_became_active":"2024-03-24T00:34:12.648431+0000","last_became_peered":"2024-03-24T00:34:12.648431+0000","last_unstale":"2024-03-24T00:34:12.654865+0000","last_undegraded":"2024-03-24T00:34:12.654865+0000","last_fullsized":"2024-03-24T00:34:12.654865+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:37.203531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.654527+0000","last_change":"2024-03-24T00:34:11.636952+0000","last_active":"2024-03-24T00:34:12.654527+0000","last_peered":"2024-03-24T00:34:12.654527+0000","last_clean":"2024-03-24T00:34:12.654527+0000","last_became_active":"2024-03-24T00:34:11.636584+0000","last_became_peered":"2024-03-24T00:34:11.636584+0000","last_unstale":"2024-03-24T00:34:12.654527+0000","last_undegraded":"2024-03-24T00:34:12.654527+0000","last_fullsized":"2024-03-24T00:34:12.654527+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:48.364049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.637699+0000","last_change":"2024-03-24T00:34:12.633770+0000","last_active":"2024-03-24T00:34:12.637699+0000","last_peered":"2024-03-24T00:34:12.637699+0000","last_clean":"2024-03-24T00:34:12.637699+0000","last_became_active":"2024-03-24T00:34:12.631944+0000","last_became_peered":"2024-03-24T00:34:12.631944+0000","last_unstale":"2024-03-24T00:34:12.637699+0000","last_undegraded":"2024-03-24T00:34:12.637699+0000","last_fullsized":"2024-03-24T00:34:12.637699+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:06.028431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.625483+0000","last_change":"2024-03-24T00:34:11.636320+0000","last_active":"2024-03-24T00:34:12.625483+0000","last_peered":"2024-03-24T00:34:12.625483+0000","last_clean":"2024-03-24T00:34:12.625483+0000","last_became_active":"2024-03-24T00:34:11.635965+0000","last_became_peered":"2024-03-24T00:34:11.635965+0000","last_unstale":"2024-03-24T00:34:12.625483+0000","last_undegraded":"2024-03-24T00:34:12.625483+0000","last_fullsized":"2024-03-24T00:34:12.625483+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:31.855764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.637615+0000","last_change":"2024-03-24T00:34:12.634803+0000","last_active":"2024-03-24T00:34:12.637615+0000","last_peered":"2024-03-24T00:34:12.637615+0000","last_clean":"2024-03-24T00:34:12.637615+0000","last_became_active":"2024-03-24T00:34:12.633695+0000","last_became_peered":"2024-03-24T00:34:12.633695+0000","last_unstale":"2024-03-24T00:34:12.637615+0000","last_undegraded":"2024-03-24T00:34:12.637615+0000","last_fullsized":"2024-03-24T00:34:12.637615+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:55:20.612662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.638128+0000","last_change":"2024-03-24T00:34:11.629752+0000","last_active":"2024-03-24T00:34:12.638128+0000","last_peered":"2024-03-24T00:34:12.638128+0000","last_clean":"2024-03-24T00:34:12.638128+0000","last_became_active":"2024-03-24T00:34:11.629443+0000","last_became_peered":"2024-03-24T00:34:11.629443+0000","last_unstale":"2024-03-24T00:34:12.638128+0000","last_undegraded":"2024-03-24T00:34:12.638128+0000","last_fullsized":"2024-03-24T00:34:12.638128+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:18:37.882421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.647223+0000","last_change":"2024-03-24T00:34:12.639235+0000","last_active":"2024-03-24T00:34:12.647223+0000","last_peered":"2024-03-24T00:34:12.647223+0000","last_clean":"2024-03-24T00:34:12.647223+0000","last_became_active":"2024-03-24T00:34:12.638898+0000","last_became_peered":"2024-03-24T00:34:12.638898+0000","last_unstale":"2024-03-24T00:34:12.647223+0000","last_undegraded":"2024-03-24T00:34:12.647223+0000","last_fullsized":"2024-03-24T00:34:12.647223+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:00:57.594800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.620715+0000","last_change":"2024-03-24T00:34:11.641975+0000","last_active":"2024-03-24T00:34:12.620715+0000","last_peered":"2024-03-24T00:34:12.620715+0000","last_clean":"2024-03-24T00:34:12.620715+0000","last_became_active":"2024-03-24T00:34:11.641574+0000","last_became_peered":"2024-03-24T00:34:11.641574+0000","last_unstale":"2024-03-24T00:34:12.620715+0000","last_undegraded":"2024-03-24T00:34:12.620715+0000","last_fullsized":"2024-03-24T00:34:12.620715+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:39.087307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.642772+0000","last_change":"2024-03-24T00:34:12.632580+0000","last_active":"2024-03-24T00:34:12.642772+0000","last_peered":"2024-03-24T00:34:12.642772+0000","last_clean":"2024-03-24T00:34:12.642772+0000","last_became_active":"2024-03-24T00:34:12.632309+0000","last_became_peered":"2024-03-24T00:34:12.632309+0000","last_unstale":"2024-03-24T00:34:12.642772+0000","last_undegraded":"2024-03-24T00:34:12.642772+0000","last_fullsized":"2024-03-24T00:34:12.642772+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:15:00.368589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.624363+0000","last_change":"2024-03-24T00:34:11.638279+0000","last_active":"2024-03-24T00:34:12.624363+0000","last_peered":"2024-03-24T00:34:12.624363+0000","last_clean":"2024-03-24T00:34:12.624363+0000","last_became_active":"2024-03-24T00:34:11.638019+0000","last_became_peered":"2024-03-24T00:34:11.638019+0000","last_unstale":"2024-03-24T00:34:12.624363+0000","last_undegraded":"2024-03-24T00:34:12.624363+0000","last_fullsized":"2024-03-24T00:34:12.624363+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:46:12.594425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.649441+0000","last_change":"2024-03-24T00:34:12.645730+0000","last_active":"2024-03-24T00:34:12.649441+0000","last_peered":"2024-03-24T00:34:12.649441+0000","last_clean":"2024-03-24T00:34:12.649441+0000","last_became_active":"2024-03-24T00:34:12.645344+0000","last_became_peered":"2024-03-24T00:34:12.645344+0000","last_unstale":"2024-03-24T00:34:12.649441+0000","last_undegraded":"2024-03-24T00:34:12.649441+0000","last_fullsized":"2024-03-24T00:34:12.649441+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:10:42.804405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"58'3","reported_seq":15,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.650738+0000","last_change":"2024-03-24T00:34:11.629548+0000","last_active":"2024-03-24T00:34:12.650738+0000","last_peered":"2024-03-24T00:34:12.650738+0000","last_clean":"2024-03-24T00:34:12.650738+0000","last_became_active":"2024-03-24T00:34:11.629262+0000","last_became_peered":"2024-03-24T00:34:11.629262+0000","last_unstale":"2024-03-24T00:34:12.650738+0000","last_undegraded":"2024-03-24T00:34:12.650738+0000","last_fullsized":"2024-03-24T00:34:12.650738+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:18.299165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.637199+0000","last_change":"2024-03-24T00:34:12.628833+0000","last_active":"2024-03-24T00:34:12.637199+0000","last_peered":"2024-03-24T00:34:12.637199+0000","last_clean":"2024-03-24T00:34:12.637199+0000","last_became_active":"2024-03-24T00:34:12.628523+0000","last_became_peered":"2024-03-24T00:34:12.628523+0000","last_unstale":"2024-03-24T00:34:12.637199+0000","last_undegraded":"2024-03-24T00:34:12.637199+0000","last_fullsized":"2024-03-24T00:34:12.637199+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:18.631243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"58'6","reported_seq":18,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.658082+0000","last_change":"2024-03-24T00:34:11.641867+0000","last_active":"2024-03-24T00:34:12.658082+0000","last_peered":"2024-03-24T00:34:12.658082+0000","last_clean":"2024-03-24T00:34:12.658082+0000","last_became_active":"2024-03-24T00:34:11.641468+0000","last_became_peered":"2024-03-24T00:34:11.641468+0000","last_unstale":"2024-03-24T00:34:12.658082+0000","last_undegraded":"2024-03-24T00:34:12.658082+0000","last_fullsized":"2024-03-24T00:34:12.658082+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:04.192880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.636586+0000","last_change":"2024-03-24T00:34:12.631235+0000","last_active":"2024-03-24T00:34:12.636586+0000","last_peered":"2024-03-24T00:34:12.636586+0000","last_clean":"2024-03-24T00:34:12.636586+0000","last_became_active":"2024-03-24T00:34:12.630587+0000","last_became_peered":"2024-03-24T00:34:12.630587+0000","last_unstale":"2024-03-24T00:34:12.636586+0000","last_undegraded":"2024-03-24T00:34:12.636586+0000","last_fullsized":"2024-03-24T00:34:12.636586+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:36:49.371076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.640679+0000","last_change":"2024-03-24T00:34:11.634013+0000","last_active":"2024-03-24T00:34:12.640679+0000","last_peered":"2024-03-24T00:34:12.640679+0000","last_clean":"2024-03-24T00:34:12.640679+0000","last_became_active":"2024-03-24T00:34:11.633712+0000","last_became_peered":"2024-03-24T00:34:11.633712+0000","last_unstale":"2024-03-24T00:34:12.640679+0000","last_undegraded":"2024-03-24T00:34:12.640679+0000","last_fullsized":"2024-03-24T00:34:12.640679+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:07.720435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.638984+0000","last_change":"2024-03-24T00:34:12.631027+0000","last_active":"2024-03-24T00:34:12.638984+0000","last_peered":"2024-03-24T00:34:12.638984+0000","last_clean":"2024-03-24T00:34:12.638984+0000","last_became_active":"2024-03-24T00:34:12.630743+0000","last_became_peered":"2024-03-24T00:34:12.630743+0000","last_unstale":"2024-03-24T00:34:12.638984+0000","last_undegraded":"2024-03-24T00:34:12.638984+0000","last_fullsized":"2024-03-24T00:34:12.638984+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:18.784673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.623506+0000","last_change":"2024-03-24T00:34:11.630910+0000","last_active":"2024-03-24T00:34:12.623506+0000","last_peered":"2024-03-24T00:34:12.623506+0000","last_clean":"2024-03-24T00:34:12.623506+0000","last_became_active":"2024-03-24T00:34:11.630593+0000","last_became_peered":"2024-03-24T00:34:11.630593+0000","last_unstale":"2024-03-24T00:34:12.623506+0000","last_undegraded":"2024-03-24T00:34:12.623506+0000","last_fullsized":"2024-03-24T00:34:12.623506+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:09.962122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.647444+0000","last_change":"2024-03-24T00:34:12.641608+0000","last_active":"2024-03-24T00:34:12.647444+0000","last_peered":"2024-03-24T00:34:12.647444+0000","last_clean":"2024-03-24T00:34:12.647444+0000","last_became_active":"2024-03-24T00:34:12.640410+0000","last_became_peered":"2024-03-24T00:34:12.640410+0000","last_unstale":"2024-03-24T00:34:12.647444+0000","last_undegraded":"2024-03-24T00:34:12.647444+0000","last_fullsized":"2024-03-24T00:34:12.647444+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:23.672349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.625454+0000","last_change":"2024-03-24T00:34:11.630895+0000","last_active":"2024-03-24T00:34:12.625454+0000","last_peered":"2024-03-24T00:34:12.625454+0000","last_clean":"2024-03-24T00:34:12.625454+0000","last_became_active":"2024-03-24T00:34:11.630063+0000","last_became_peered":"2024-03-24T00:34:11.630063+0000","last_unstale":"2024-03-24T00:34:12.625454+0000","last_undegraded":"2024-03-24T00:34:12.625454+0000","last_fullsized":"2024-03-24T00:34:12.625454+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:59.361561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.650712+0000","last_change":"2024-03-24T00:34:12.645396+0000","last_active":"2024-03-24T00:34:12.650712+0000","last_peered":"2024-03-24T00:34:12.650712+0000","last_clean":"2024-03-24T00:34:12.650712+0000","last_became_active":"2024-03-24T00:34:12.643067+0000","last_became_peered":"2024-03-24T00:34:12.643067+0000","last_unstale":"2024-03-24T00:34:12.650712+0000","last_undegraded":"2024-03-24T00:34:12.650712+0000","last_fullsized":"2024-03-24T00:34:12.650712+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:39.516699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"58'8","reported_seq":20,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.653086+0000","last_change":"2024-03-24T00:34:11.632399+0000","last_active":"2024-03-24T00:34:12.653086+0000","last_peered":"2024-03-24T00:34:12.653086+0000","last_clean":"2024-03-24T00:34:12.653086+0000","last_became_active":"2024-03-24T00:34:11.632126+0000","last_became_peered":"2024-03-24T00:34:11.632126+0000","last_unstale":"2024-03-24T00:34:12.653086+0000","last_undegraded":"2024-03-24T00:34:12.653086+0000","last_fullsized":"2024-03-24T00:34:12.653086+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:00:56.587773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.652346+0000","last_change":"2024-03-24T00:34:12.649908+0000","last_active":"2024-03-24T00:34:12.652346+0000","last_peered":"2024-03-24T00:34:12.652346+0000","last_clean":"2024-03-24T00:34:12.652346+0000","last_became_active":"2024-03-24T00:34:12.647786+0000","last_became_peered":"2024-03-24T00:34:12.647786+0000","last_unstale":"2024-03-24T00:34:12.652346+0000","last_undegraded":"2024-03-24T00:34:12.652346+0000","last_fullsized":"2024-03-24T00:34:12.652346+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:34.956494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.639902+0000","last_change":"2024-03-24T00:34:11.634357+0000","last_active":"2024-03-24T00:34:12.639902+0000","last_peered":"2024-03-24T00:34:12.639902+0000","last_clean":"2024-03-24T00:34:12.639902+0000","last_became_active":"2024-03-24T00:34:11.634072+0000","last_became_peered":"2024-03-24T00:34:11.634072+0000","last_unstale":"2024-03-24T00:34:12.639902+0000","last_undegraded":"2024-03-24T00:34:12.639902+0000","last_fullsized":"2024-03-24T00:34:12.639902+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:34:41.047302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.643374+0000","last_change":"2024-03-24T00:34:12.638710+0000","last_active":"2024-03-24T00:34:12.643374+0000","last_peered":"2024-03-24T00:34:12.643374+0000","last_clean":"2024-03-24T00:34:12.643374+0000","last_became_active":"2024-03-24T00:34:12.638395+0000","last_became_peered":"2024-03-24T00:34:12.638395+0000","last_unstale":"2024-03-24T00:34:12.643374+0000","last_undegraded":"2024-03-24T00:34:12.643374+0000","last_fullsized":"2024-03-24T00:34:12.643374+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:45:48.010797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.653532+0000","last_change":"2024-03-24T00:34:11.639569+0000","last_active":"2024-03-24T00:34:12.653532+0000","last_peered":"2024-03-24T00:34:12.653532+0000","last_clean":"2024-03-24T00:34:12.653532+0000","last_became_active":"2024-03-24T00:34:11.639306+0000","last_became_peered":"2024-03-24T00:34:11.639306+0000","last_unstale":"2024-03-24T00:34:12.653532+0000","last_undegraded":"2024-03-24T00:34:12.653532+0000","last_fullsized":"2024-03-24T00:34:12.653532+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:17.624215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.654862+0000","last_change":"2024-03-24T00:34:12.650325+0000","last_active":"2024-03-24T00:34:12.654862+0000","last_peered":"2024-03-24T00:34:12.654862+0000","last_clean":"2024-03-24T00:34:12.654862+0000","last_became_active":"2024-03-24T00:34:12.650033+0000","last_became_peered":"2024-03-24T00:34:12.650033+0000","last_unstale":"2024-03-24T00:34:12.654862+0000","last_undegraded":"2024-03-24T00:34:12.654862+0000","last_fullsized":"2024-03-24T00:34:12.654862+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:47:44.272866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"58'1","reported_seq":13,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.649544+0000","last_change":"2024-03-24T00:34:11.631227+0000","last_active":"2024-03-24T00:34:12.649544+0000","last_peered":"2024-03-24T00:34:12.649544+0000","last_clean":"2024-03-24T00:34:12.649544+0000","last_became_active":"2024-03-24T00:34:11.630881+0000","last_became_peered":"2024-03-24T00:34:11.630881+0000","last_unstale":"2024-03-24T00:34:12.649544+0000","last_undegraded":"2024-03-24T00:34:12.649544+0000","last_fullsized":"2024-03-24T00:34:12.649544+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:50:17.777645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.633859+0000","last_change":"2024-03-24T00:34:12.628634+0000","last_active":"2024-03-24T00:34:12.633859+0000","last_peered":"2024-03-24T00:34:12.633859+0000","last_clean":"2024-03-24T00:34:12.633859+0000","last_became_active":"2024-03-24T00:34:12.628377+0000","last_became_peered":"2024-03-24T00:34:12.628377+0000","last_unstale":"2024-03-24T00:34:12.633859+0000","last_undegraded":"2024-03-24T00:34:12.633859+0000","last_fullsized":"2024-03-24T00:34:12.633859+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:23.672349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.643389+0000","last_change":"2024-03-24T00:34:11.636086+0000","last_active":"2024-03-24T00:34:12.643389+0000","last_peered":"2024-03-24T00:34:12.643389+0000","last_clean":"2024-03-24T00:34:12.643389+0000","last_became_active":"2024-03-24T00:34:11.635800+0000","last_became_peered":"2024-03-24T00:34:11.635800+0000","last_unstale":"2024-03-24T00:34:12.643389+0000","last_undegraded":"2024-03-24T00:34:12.643389+0000","last_fullsized":"2024-03-24T00:34:12.643389+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:19:31.855764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641738+0000","last_change":"2024-03-24T00:34:12.631462+0000","last_active":"2024-03-24T00:34:12.641738+0000","last_peered":"2024-03-24T00:34:12.641738+0000","last_clean":"2024-03-24T00:34:12.641738+0000","last_became_active":"2024-03-24T00:34:12.630972+0000","last_became_peered":"2024-03-24T00:34:12.630972+0000","last_unstale":"2024-03-24T00:34:12.641738+0000","last_undegraded":"2024-03-24T00:34:12.641738+0000","last_fullsized":"2024-03-24T00:34:12.641738+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:39.516699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.651967+0000","last_change":"2024-03-24T00:34:11.632863+0000","last_active":"2024-03-24T00:34:12.651967+0000","last_peered":"2024-03-24T00:34:12.651967+0000","last_clean":"2024-03-24T00:34:12.651967+0000","last_became_active":"2024-03-24T00:34:11.632425+0000","last_became_peered":"2024-03-24T00:34:11.632425+0000","last_unstale":"2024-03-24T00:34:12.651967+0000","last_undegraded":"2024-03-24T00:34:12.651967+0000","last_fullsized":"2024-03-24T00:34:12.651967+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:16.733803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.651471+0000","last_change":"2024-03-24T00:34:12.645022+0000","last_active":"2024-03-24T00:34:12.651471+0000","last_peered":"2024-03-24T00:34:12.651471+0000","last_clean":"2024-03-24T00:34:12.651471+0000","last_became_active":"2024-03-24T00:34:12.644737+0000","last_became_peered":"2024-03-24T00:34:12.644737+0000","last_unstale":"2024-03-24T00:34:12.651471+0000","last_undegraded":"2024-03-24T00:34:12.651471+0000","last_fullsized":"2024-03-24T00:34:12.651471+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:04.494411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.622745+0000","last_change":"2024-03-24T00:34:11.637661+0000","last_active":"2024-03-24T00:34:12.622745+0000","last_peered":"2024-03-24T00:34:12.622745+0000","last_clean":"2024-03-24T00:34:12.622745+0000","last_became_active":"2024-03-24T00:34:11.637118+0000","last_became_peered":"2024-03-24T00:34:11.637118+0000","last_unstale":"2024-03-24T00:34:12.622745+0000","last_undegraded":"2024-03-24T00:34:12.622745+0000","last_fullsized":"2024-03-24T00:34:12.622745+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:05.021404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.654299+0000","last_change":"2024-03-24T00:34:12.650455+0000","last_active":"2024-03-24T00:34:12.654299+0000","last_peered":"2024-03-24T00:34:12.654299+0000","last_clean":"2024-03-24T00:34:12.654299+0000","last_became_active":"2024-03-24T00:34:12.650186+0000","last_became_peered":"2024-03-24T00:34:12.650186+0000","last_unstale":"2024-03-24T00:34:12.654299+0000","last_undegraded":"2024-03-24T00:34:12.654299+0000","last_fullsized":"2024-03-24T00:34:12.654299+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:42.944475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.625138+0000","last_change":"2024-03-24T00:34:11.637350+0000","last_active":"2024-03-24T00:34:12.625138+0000","last_peered":"2024-03-24T00:34:12.625138+0000","last_clean":"2024-03-24T00:34:12.625138+0000","last_became_active":"2024-03-24T00:34:11.637088+0000","last_became_peered":"2024-03-24T00:34:11.637088+0000","last_unstale":"2024-03-24T00:34:12.625138+0000","last_undegraded":"2024-03-24T00:34:12.625138+0000","last_fullsized":"2024-03-24T00:34:12.625138+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:22.665321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641659+0000","last_change":"2024-03-24T00:34:12.632330+0000","last_active":"2024-03-24T00:34:12.641659+0000","last_peered":"2024-03-24T00:34:12.641659+0000","last_clean":"2024-03-24T00:34:12.641659+0000","last_became_active":"2024-03-24T00:34:12.631976+0000","last_became_peered":"2024-03-24T00:34:12.631976+0000","last_unstale":"2024-03-24T00:34:12.641659+0000","last_undegraded":"2024-03-24T00:34:12.641659+0000","last_fullsized":"2024-03-24T00:34:12.641659+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:31:01.823045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"58'2","reported_seq":14,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.646335+0000","last_change":"2024-03-24T00:34:11.626734+0000","last_active":"2024-03-24T00:34:12.646335+0000","last_peered":"2024-03-24T00:34:12.646335+0000","last_clean":"2024-03-24T00:34:12.646335+0000","last_became_active":"2024-03-24T00:34:11.624092+0000","last_became_peered":"2024-03-24T00:34:11.624092+0000","last_unstale":"2024-03-24T00:34:12.646335+0000","last_undegraded":"2024-03-24T00:34:12.646335+0000","last_fullsized":"2024-03-24T00:34:12.646335+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:50:53.986736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641308+0000","last_change":"2024-03-24T00:34:12.634750+0000","last_active":"2024-03-24T00:34:12.641308+0000","last_peered":"2024-03-24T00:34:12.641308+0000","last_clean":"2024-03-24T00:34:12.641308+0000","last_became_active":"2024-03-24T00:34:12.634451+0000","last_became_peered":"2024-03-24T00:34:12.634451+0000","last_unstale":"2024-03-24T00:34:12.641308+0000","last_undegraded":"2024-03-24T00:34:12.641308+0000","last_fullsized":"2024-03-24T00:34:12.641308+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:49:34.956494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.641839+0000","last_change":"2024-03-24T00:34:11.634704+0000","last_active":"2024-03-24T00:34:12.641839+0000","last_peered":"2024-03-24T00:34:12.641839+0000","last_clean":"2024-03-24T00:34:12.641839+0000","last_became_active":"2024-03-24T00:34:11.634339+0000","last_became_peered":"2024-03-24T00:34:11.634339+0000","last_unstale":"2024-03-24T00:34:12.641839+0000","last_undegraded":"2024-03-24T00:34:12.641839+0000","last_fullsized":"2024-03-24T00:34:12.641839+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:14:59.361561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.649418+0000","last_change":"2024-03-24T00:34:12.645562+0000","last_active":"2024-03-24T00:34:12.649418+0000","last_peered":"2024-03-24T00:34:12.649418+0000","last_clean":"2024-03-24T00:34:12.649418+0000","last_became_active":"2024-03-24T00:34:12.644988+0000","last_became_peered":"2024-03-24T00:34:12.644988+0000","last_unstale":"2024-03-24T00:34:12.649418+0000","last_undegraded":"2024-03-24T00:34:12.649418+0000","last_fullsized":"2024-03-24T00:34:12.649418+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:07:15.742707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.623404+0000","last_change":"2024-03-24T00:34:11.629779+0000","last_active":"2024-03-24T00:34:12.623404+0000","last_peered":"2024-03-24T00:34:12.623404+0000","last_clean":"2024-03-24T00:34:12.623404+0000","last_became_active":"2024-03-24T00:34:11.629391+0000","last_became_peered":"2024-03-24T00:34:11.629391+0000","last_unstale":"2024-03-24T00:34:12.623404+0000","last_undegraded":"2024-03-24T00:34:12.623404+0000","last_fullsized":"2024-03-24T00:34:12.623404+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:11.477318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.651193+0000","last_change":"2024-03-24T00:34:12.647184+0000","last_active":"2024-03-24T00:34:12.651193+0000","last_peered":"2024-03-24T00:34:12.651193+0000","last_clean":"2024-03-24T00:34:12.651193+0000","last_became_active":"2024-03-24T00:34:12.646855+0000","last_became_peered":"2024-03-24T00:34:12.646855+0000","last_unstale":"2024-03-24T00:34:12.651193+0000","last_undegraded":"2024-03-24T00:34:12.651193+0000","last_fullsized":"2024-03-24T00:34:12.651193+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:29:39.516699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.625290+0000","last_change":"2024-03-24T00:34:11.636495+0000","last_active":"2024-03-24T00:34:12.625290+0000","last_peered":"2024-03-24T00:34:12.625290+0000","last_clean":"2024-03-24T00:34:12.625290+0000","last_became_active":"2024-03-24T00:34:11.636125+0000","last_became_peered":"2024-03-24T00:34:11.636125+0000","last_unstale":"2024-03-24T00:34:12.625290+0000","last_undegraded":"2024-03-24T00:34:12.625290+0000","last_fullsized":"2024-03-24T00:34:12.625290+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:17.624215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.638461+0000","last_change":"2024-03-24T00:34:12.636338+0000","last_active":"2024-03-24T00:34:12.638461+0000","last_peered":"2024-03-24T00:34:12.638461+0000","last_clean":"2024-03-24T00:34:12.638461+0000","last_became_active":"2024-03-24T00:34:12.636009+0000","last_became_peered":"2024-03-24T00:34:12.636009+0000","last_unstale":"2024-03-24T00:34:12.638461+0000","last_undegraded":"2024-03-24T00:34:12.638461+0000","last_fullsized":"2024-03-24T00:34:12.638461+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:43:58.812833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.622739+0000","last_change":"2024-03-24T00:34:11.629175+0000","last_active":"2024-03-24T00:34:12.622739+0000","last_peered":"2024-03-24T00:34:12.622739+0000","last_clean":"2024-03-24T00:34:12.622739+0000","last_became_active":"2024-03-24T00:34:11.628884+0000","last_became_peered":"2024-03-24T00:34:11.628884+0000","last_unstale":"2024-03-24T00:34:12.622739+0000","last_undegraded":"2024-03-24T00:34:12.622739+0000","last_fullsized":"2024-03-24T00:34:12.622739+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:25:09.962122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.649821+0000","last_change":"2024-03-24T00:34:12.642381+0000","last_active":"2024-03-24T00:34:12.649821+0000","last_peered":"2024-03-24T00:34:12.649821+0000","last_clean":"2024-03-24T00:34:12.649821+0000","last_became_active":"2024-03-24T00:34:12.642017+0000","last_became_peered":"2024-03-24T00:34:12.642017+0000","last_unstale":"2024-03-24T00:34:12.649821+0000","last_undegraded":"2024-03-24T00:34:12.649821+0000","last_fullsized":"2024-03-24T00:34:12.649821+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:04.494411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.640569+0000","last_change":"2024-03-24T00:34:11.640663+0000","last_active":"2024-03-24T00:34:12.640569+0000","last_peered":"2024-03-24T00:34:12.640569+0000","last_clean":"2024-03-24T00:34:12.640569+0000","last_became_active":"2024-03-24T00:34:11.640331+0000","last_became_peered":"2024-03-24T00:34:11.640331+0000","last_unstale":"2024-03-24T00:34:12.640569+0000","last_undegraded":"2024-03-24T00:34:12.640569+0000","last_fullsized":"2024-03-24T00:34:12.640569+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:39.087307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.635253+0000","last_change":"2024-03-24T00:34:12.629894+0000","last_active":"2024-03-24T00:34:12.635253+0000","last_peered":"2024-03-24T00:34:12.635253+0000","last_clean":"2024-03-24T00:34:12.635253+0000","last_became_active":"2024-03-24T00:34:12.629615+0000","last_became_peered":"2024-03-24T00:34:12.629615+0000","last_unstale":"2024-03-24T00:34:12.635253+0000","last_undegraded":"2024-03-24T00:34:12.635253+0000","last_fullsized":"2024-03-24T00:34:12.635253+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:42:42.768882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.622739+0000","last_change":"2024-03-24T00:34:11.639715+0000","last_active":"2024-03-24T00:34:12.622739+0000","last_peered":"2024-03-24T00:34:12.622739+0000","last_clean":"2024-03-24T00:34:12.622739+0000","last_became_active":"2024-03-24T00:34:11.639093+0000","last_became_peered":"2024-03-24T00:34:11.639093+0000","last_unstale":"2024-03-24T00:34:12.622739+0000","last_undegraded":"2024-03-24T00:34:12.622739+0000","last_fullsized":"2024-03-24T00:34:12.622739+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:04:11.477318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.635973+0000","last_change":"2024-03-24T00:34:12.631422+0000","last_active":"2024-03-24T00:34:12.635973+0000","last_peered":"2024-03-24T00:34:12.635973+0000","last_clean":"2024-03-24T00:34:12.635973+0000","last_became_active":"2024-03-24T00:34:12.631111+0000","last_became_peered":"2024-03-24T00:34:12.631111+0000","last_unstale":"2024-03-24T00:34:12.635973+0000","last_undegraded":"2024-03-24T00:34:12.635973+0000","last_fullsized":"2024-03-24T00:34:12.635973+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:13.255425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.624212+0000","last_change":"2024-03-24T00:34:11.634811+0000","last_active":"2024-03-24T00:34:12.624212+0000","last_peered":"2024-03-24T00:34:12.624212+0000","last_clean":"2024-03-24T00:34:12.624212+0000","last_became_active":"2024-03-24T00:34:11.634504+0000","last_became_peered":"2024-03-24T00:34:11.634504+0000","last_unstale":"2024-03-24T00:34:12.624212+0000","last_undegraded":"2024-03-24T00:34:12.624212+0000","last_fullsized":"2024-03-24T00:34:12.624212+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:53:48.596023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.652084+0000","last_change":"2024-03-24T00:34:12.648047+0000","last_active":"2024-03-24T00:34:12.652084+0000","last_peered":"2024-03-24T00:34:12.652084+0000","last_clean":"2024-03-24T00:34:12.652084+0000","last_became_active":"2024-03-24T00:34:12.647723+0000","last_became_peered":"2024-03-24T00:34:12.647723+0000","last_unstale":"2024-03-24T00:34:12.652084+0000","last_undegraded":"2024-03-24T00:34:12.652084+0000","last_fullsized":"2024-03-24T00:34:12.652084+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:11.603329+0000","last_clean_scrub_stamp":"2024-03-24T00:34:11.603329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:31:37.091502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.623134+0000","last_change":"2024-03-24T00:34:11.633822+0000","last_active":"2024-03-24T00:34:12.623134+0000","last_peered":"2024-03-24T00:34:12.623134+0000","last_clean":"2024-03-24T00:34:12.623134+0000","last_became_active":"2024-03-24T00:34:11.633543+0000","last_became_peered":"2024-03-24T00:34:11.633543+0000","last_unstale":"2024-03-24T00:34:12.623134+0000","last_undegraded":"2024-03-24T00:34:12.623134+0000","last_fullsized":"2024-03-24T00:34:12.623134+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:34:10.596302+0000","last_clean_scrub_stamp":"2024-03-24T00:34:10.596302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:48:56.784074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":247,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-03-24T00:34:12.617548+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:34:12.617548+0000","last_peered":"2024-03-24T00:34:12.617548+0000","last_clean":"2024-03-24T00:34:12.617548+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:34:12.617548+0000","last_undegraded":"2024-03-24T00:34:12.617548+0000","last_fullsized":"2024-03-24T00:34:12.617548+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1298432,"data_stored":884174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":696320,"data_stored":293646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31876,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339964,"statfs":{"total":96636764160,"available":96604123136,"internally_reserved":0,"allocated":704512,"data_stored":294298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36032,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335808,"statfs":{"total":96636764160,"available":96599867392,"internally_reserved":0,"allocated":708608,"data_stored":295258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":704512,"data_stored":293702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":704512,"data_stored":293702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":700416,"data_stored":293736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32428,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339412,"statfs":{"total":96636764160,"available":96603557888,"internally_reserved":0,"allocated":1277952,"data_stored":877239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:34:17.142 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:34:17.143 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:34:17.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.323+0000 7f484da00640 1 Processor -- start 2024-03-24T00:34:17.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.323+0000 7f484da00640 1 -- start start 2024-03-24T00:34:17.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.323+0000 7f484da00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f484806cb80 0x7f48480b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:17.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.323+0000 7f484da00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480b3260 con 0x7f484815ac60 2024-03-24T00:34:17.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.323+0000 7f484da00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48480b3540 con 0x7f48480b4840 2024-03-24T00:34:17.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.323+0000 7f484da00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48480b3e70 con 0x7f484806cb80 2024-03-24T00:34:17.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f4846a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f484815ac60 0x7f48480b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37654/0 (socket says 172.21.15.40:37654) 2024-03-24T00:34:17.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f4846a00640 1 -- 172.21.15.40:0/228533714 learned_addr learned my addr 172.21.15.40:0/228533714 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:17.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f483e000640 1 --2- 172.21.15.40:0/228533714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f484806cb80 0x7f48480b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:17.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f4846000640 1 -- 172.21.15.40:0/228533714 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983906744 0 0) 0x7f48480b3260 con 0x7f484815ac60 2024-03-24T00:34:17.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f4846000640 1 -- 172.21.15.40:0/228533714 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f482c003530 con 0x7f484815ac60 2024-03-24T00:34:17.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f4846000640 1 -- 172.21.15.40:0/228533714 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966200369 0 0) 0x7f48480b3540 con 0x7f48480b4840 2024-03-24T00:34:17.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f483e000640 1 -- 172.21.15.40:0/228533714 >> v1:172.21.15.53:6789/0 conn(0x7f48480b4840 legacy=0x7f4848071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f483e000640 1 -- 172.21.15.40:0/228533714 >> v1:172.21.15.40:6789/0 conn(0x7f484815ac60 legacy=0x7f48480b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f483e000640 1 -- 172.21.15.40:0/228533714 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48480b3820 con 0x7f484806cb80 2024-03-24T00:34:17.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f483e000640 1 --2- 172.21.15.40:0/228533714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f484806cb80 0x7f48480b4300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f48300097f0 tx=0x7f483001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfab08c12cef025a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:17.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f4846000640 1 -- 172.21.15.40:0/228533714 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48300b2070 con 0x7f484806cb80 2024-03-24T00:34:17.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f4846000640 1 -- 172.21.15.40:0/228533714 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48300a9070 con 0x7f484806cb80 2024-03-24T00:34:17.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.327+0000 7f4846000640 1 -- 172.21.15.40:0/228533714 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48300ad880 con 0x7f484806cb80 2024-03-24T00:34:17.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 -- 172.21.15.40:0/228533714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f484806cb80 msgr2=0x7f48480b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 --2- 172.21.15.40:0/228533714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f484806cb80 0x7f48480b4300 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f48300097f0 tx=0x7f483001c450 comp rx=0 tx=0).stop 2024-03-24T00:34:17.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 -- 172.21.15.40:0/228533714 shutdown_connections 2024-03-24T00:34:17.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 --2- 172.21.15.40:0/228533714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f484806cb80 0x7f48480b4300 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 -- 172.21.15.40:0/228533714 >> 172.21.15.40:0/228533714 conn(0x7f48480b1660 msgr2=0x7f48480b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:17.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 -- 172.21.15.40:0/228533714 shutdown_connections 2024-03-24T00:34:17.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 -- 172.21.15.40:0/228533714 wait complete. 2024-03-24T00:34:17.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 Processor -- start 2024-03-24T00:34:17.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 -- start start 2024-03-24T00:34:17.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f484806cb80 0x7f484813c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48480b4840 0x7f48481587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f484815ac60 0x7f48480afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4848072210 con 0x7f48480b4840 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48480cd7b0 con 0x7f484806cb80 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f484da00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4848072390 con 0x7f484815ac60 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f483e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48480b4840 0x7f48481587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f4846a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f484806cb80 0x7f484813c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f483e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48480b4840 0x7f48481587e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46916/0 (socket says 172.21.15.40:46916) 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f483e000640 1 -- 172.21.15.40:0/93081304 learned_addr learned my addr 172.21.15.40:0/93081304 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f4847400640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f484815ac60 0x7f48480afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f483e000640 1 -- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f484815ac60 msgr2=0x7f48480afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f483e000640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f484815ac60 0x7f48480afd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f483e000640 1 -- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f484806cb80 msgr2=0x7f484813c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f483e000640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f484806cb80 0x7f484813c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.331+0000 7f483e000640 1 -- 172.21.15.40:0/93081304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4848158d20 con 0x7f48480b4840 2024-03-24T00:34:17.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.335+0000 7f483e000640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48480b4840 0x7f48481587e0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f4830009740 tx=0x7f483001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:17.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.335+0000 7f483f400640 1 -- 172.21.15.40:0/93081304 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48300bd020 con 0x7f48480b4840 2024-03-24T00:34:17.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.335+0000 7f483f400640 1 -- 172.21.15.40:0/93081304 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48300b3040 con 0x7f48480b4840 2024-03-24T00:34:17.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.335+0000 7f484da00640 1 -- 172.21.15.40:0/93081304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4848158ff0 con 0x7f48480b4840 2024-03-24T00:34:17.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.335+0000 7f483f400640 1 -- 172.21.15.40:0/93081304 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48300b2050 con 0x7f48480b4840 2024-03-24T00:34:17.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.335+0000 7f484da00640 1 -- 172.21.15.40:0/93081304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48480cedb0 con 0x7f48480b4840 2024-03-24T00:34:17.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.335+0000 7f484da00640 1 -- 172.21.15.40:0/93081304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4848068170 con 0x7f48480b4840 2024-03-24T00:34:17.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.335+0000 7f483f400640 1 -- 172.21.15.40:0/93081304 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f48300c0070 con 0x7f48480b4840 2024-03-24T00:34:17.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.335+0000 7f483f400640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4824067660 0x7f4824069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:17.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.335+0000 7f483f400640 1 -- 172.21.15.40:0/93081304 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f483011ec70 con 0x7f48480b4840 2024-03-24T00:34:17.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.335+0000 7f4846a00640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4824067660 0x7f4824069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:17.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.335+0000 7f4846a00640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4824067660 0x7f4824069b20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4838007fd0 tx=0x7f4838009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:17.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.339+0000 7f483f400640 1 -- 172.21.15.40:0/93081304 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f48300160d0 con 0x7f48480b4840 2024-03-24T00:34:17.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.559+0000 7f484da00640 1 -- 172.21.15.40:0/93081304 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f48480af820 con 0x7f48480b4840 2024-03-24T00:34:17.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.559+0000 7f483f400640 1 -- 172.21.15.40:0/93081304 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f48300ef290 con 0x7f48480b4840 2024-03-24T00:34:17.563 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-03-24T00:34:17.564 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw== 2024-03-24T00:34:17.564 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-03-24T00:34:17.564 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-03-24T00:34:17.564 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:34:17.564 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-03-24T00:34:17.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.563+0000 7f484da00640 1 -- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4824067660 msgr2=0x7f4824069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.563+0000 7f484da00640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4824067660 0x7f4824069b20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4838007fd0 tx=0x7f4838009040 comp rx=0 tx=0).stop 2024-03-24T00:34:17.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.563+0000 7f484da00640 1 -- 172.21.15.40:0/93081304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48480b4840 msgr2=0x7f48481587e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.563+0000 7f484da00640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48480b4840 0x7f48481587e0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f4830009740 tx=0x7f483001ac90 comp rx=0 tx=0).stop 2024-03-24T00:34:17.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.563+0000 7f484da00640 1 -- 172.21.15.40:0/93081304 shutdown_connections 2024-03-24T00:34:17.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.563+0000 7f484da00640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4824067660 0x7f4824069b20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.563+0000 7f484da00640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f484815ac60 0x7f48480afd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.563+0000 7f484da00640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48480b4840 0x7f48481587e0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.563+0000 7f484da00640 1 --2- 172.21.15.40:0/93081304 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f484806cb80 0x7f484813c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.563+0000 7f484da00640 1 -- 172.21.15.40:0/93081304 >> 172.21.15.40:0/93081304 conn(0x7f48480b1660 msgr2=0x7f48481531d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:17.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.563+0000 7f484da00640 1 -- 172.21.15.40:0/93081304 shutdown_connections 2024-03-24T00:34:17.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.563+0000 7f484da00640 1 -- 172.21.15.40:0/93081304 wait complete. 2024-03-24T00:34:17.578 INFO:tasks.ceph.mds.d.smithi053.stderr:2024-03-24T00:34:17.574+0000 7f1c21a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:34:17.585 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:34:17.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.723+0000 7f28aa000640 1 Processor -- start 2024-03-24T00:34:17.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.723+0000 7f28aa000640 1 -- start start 2024-03-24T00:34:17.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.723+0000 7f28aa000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f28a4072310 0x7f28a40ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:17.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.723+0000 7f28aa000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a4070e00 con 0x7f28a413ec80 2024-03-24T00:34:17.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.723+0000 7f28aa000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a40cee20 con 0x7f28a4141bc0 2024-03-24T00:34:17.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.723+0000 7f28aa000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a40cf100 con 0x7f28a4072310 2024-03-24T00:34:17.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.723+0000 7f28a3400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f28a413ec80 0x7f28a41414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37670/0 (socket says 172.21.15.40:37670) 2024-03-24T00:34:17.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.723+0000 7f28a3400640 1 -- 172.21.15.40:0/3600185373 learned_addr learned my addr 172.21.15.40:0/3600185373 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:17.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.723+0000 7f28a3e00640 1 --2- 172.21.15.40:0/3600185373 >> v2:172.21.15.53:3300/0 conn(0x7f28a4072310 0x7f28a40ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:17.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.727+0000 7f28a3e00640 1 -- 172.21.15.40:0/3600185373 >> v1:172.21.15.53:6789/0 conn(0x7f28a4141bc0 legacy=0x7f28a40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.727+0000 7f28a3e00640 1 -- 172.21.15.40:0/3600185373 >> v1:172.21.15.40:6789/0 conn(0x7f28a413ec80 legacy=0x7f28a41414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.727+0000 7f28a3e00640 1 -- 172.21.15.40:0/3600185373 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a40cf3e0 con 0x7f28a4072310 2024-03-24T00:34:17.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.727+0000 7f28a2a00640 1 -- 172.21.15.40:0/3600185373 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849917519 0 0) 0x7f28a4070e00 con 0x7f28a413ec80 2024-03-24T00:34:17.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.727+0000 7f28a3e00640 1 --2- 172.21.15.40:0/3600185373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28a4072310 0x7f28a40ce800 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f289c009d90 tx=0x7f289c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=112ef4d47edbd634 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:17.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.727+0000 7f28a2a00640 1 -- 172.21.15.40:0/3600185373 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c0b2070 con 0x7f28a4072310 2024-03-24T00:34:17.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.727+0000 7f28a2a00640 1 -- 172.21.15.40:0/3600185373 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289c0a5a50 con 0x7f28a4072310 2024-03-24T00:34:17.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.727+0000 7f28a2a00640 1 -- 172.21.15.40:0/3600185373 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c0a5e20 con 0x7f28a4072310 2024-03-24T00:34:17.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.727+0000 7f28aa000640 1 -- 172.21.15.40:0/3600185373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28a4072310 msgr2=0x7f28a40ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.727+0000 7f28aa000640 1 --2- 172.21.15.40:0/3600185373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28a4072310 0x7f28a40ce800 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f289c009d90 tx=0x7f289c01c970 comp rx=0 tx=0).stop 2024-03-24T00:34:17.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 -- 172.21.15.40:0/3600185373 shutdown_connections 2024-03-24T00:34:17.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 --2- 172.21.15.40:0/3600185373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28a4072310 0x7f28a40ce800 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 -- 172.21.15.40:0/3600185373 >> 172.21.15.40:0/3600185373 conn(0x7f28a40b17b0 msgr2=0x7f28a40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:17.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 -- 172.21.15.40:0/3600185373 shutdown_connections 2024-03-24T00:34:17.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 -- 172.21.15.40:0/3600185373 wait complete. 2024-03-24T00:34:17.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 Processor -- start 2024-03-24T00:34:17.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 -- start start 2024-03-24T00:34:17.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28a4072310 0x7f28a413d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:17.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a413ec80 0x7f28a41577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:17.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28a4141bc0 0x7f28a413cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:17.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a4169f00 con 0x7f28a413ec80 2024-03-24T00:34:17.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a4169d80 con 0x7f28a4072310 2024-03-24T00:34:17.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28aa000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a416a080 con 0x7f28a4141bc0 2024-03-24T00:34:17.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f289aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a413ec80 0x7f28a41577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:17.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28a3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28a4072310 0x7f28a413d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:17.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28a3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28a4141bc0 0x7f28a413cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:17.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28a3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28a4072310 0x7f28a413d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:44724/0 (socket says 172.21.15.40:44724) 2024-03-24T00:34:17.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f289aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a413ec80 0x7f28a41577b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46930/0 (socket says 172.21.15.40:46930) 2024-03-24T00:34:17.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28a3400640 1 -- 172.21.15.40:0/177751735 learned_addr learned my addr 172.21.15.40:0/177751735 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:17.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f289aa00640 1 -- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28a4141bc0 msgr2=0x7f28a413cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f289aa00640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28a4141bc0 0x7f28a413cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f289aa00640 1 -- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28a4072310 msgr2=0x7f28a413d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f289aa00640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28a4072310 0x7f28a413d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f289aa00640 1 -- 172.21.15.40:0/177751735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a4157e40 con 0x7f28a413ec80 2024-03-24T00:34:17.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f28a3e00640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28a4141bc0 0x7f28a413cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:17.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.731+0000 7f289aa00640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a413ec80 0x7f28a41577b0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f288c000f70 tx=0x7f288c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:17.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.735+0000 7f289be00640 1 -- 172.21.15.40:0/177751735 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c017070 con 0x7f28a413ec80 2024-03-24T00:34:17.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.735+0000 7f289be00640 1 -- 172.21.15.40:0/177751735 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288c0052d0 con 0x7f28a413ec80 2024-03-24T00:34:17.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.735+0000 7f28aa000640 1 -- 172.21.15.40:0/177751735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a4158120 con 0x7f28a413ec80 2024-03-24T00:34:17.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.735+0000 7f289be00640 1 -- 172.21.15.40:0/177751735 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c003840 con 0x7f28a413ec80 2024-03-24T00:34:17.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.735+0000 7f28aa000640 1 -- 172.21.15.40:0/177751735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a406cb80 con 0x7f28a413ec80 2024-03-24T00:34:17.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.735+0000 7f28aa000640 1 -- 172.21.15.40:0/177751735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a4068170 con 0x7f28a413ec80 2024-03-24T00:34:17.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.735+0000 7f289be00640 1 -- 172.21.15.40:0/177751735 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f288c01a070 con 0x7f28a413ec80 2024-03-24T00:34:17.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.735+0000 7f289be00640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2880067660 0x7f2880069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:17.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.735+0000 7f289be00640 1 -- 172.21.15.40:0/177751735 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f288c084ab0 con 0x7f28a413ec80 2024-03-24T00:34:17.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.735+0000 7f28a3400640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2880067660 0x7f2880069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:17.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.735+0000 7f28a3400640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2880067660 0x7f2880069b20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2894007b20 tx=0x7f2894007a70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:17.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.739+0000 7f289be00640 1 -- 172.21.15.40:0/177751735 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f288c005480 con 0x7f28a413ec80 2024-03-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.939+0000 7f28aa000640 1 -- 172.21.15.40:0/177751735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28a406d020 con 0x7f28a413ec80 2024-03-24T00:34:17.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.939+0000 7f289be00640 1 -- 172.21.15.40:0/177751735 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+4974 (secure 0 0 0) 0x7f288c055170 con 0x7f28a413ec80 2024-03-24T00:34:17.944 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:17.945 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":52,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6277,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1633817617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1633817617},{"type":"v1","addr":"172.21.15.53:6837","nonce":1633817617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6468,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1278515461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1278515461},{"type":"v1","addr":"172.21.15.40:6834","nonce":1278515461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46},{"gid":6471,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1312292485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1312292485},{"type":"v1","addr":"172.21.15.40:6835","nonce":1312292485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":45}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:34:12.019118+0000","modified":"2024-03-24T00:34:15.031104+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6474},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6474":{"gid":6474,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/501079401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":501079401},{"type":"v1","addr":"172.21.15.53:6835","nonce":501079401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6474,"qdb_cluster":[6474]},"id":4}]} 2024-03-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 52 2024-03-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.943+0000 7f28aa000640 1 -- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2880067660 msgr2=0x7f2880069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.943+0000 7f28aa000640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2880067660 0x7f2880069b20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2894007b20 tx=0x7f2894007a70 comp rx=0 tx=0).stop 2024-03-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.943+0000 7f28aa000640 1 -- 172.21.15.40:0/177751735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a413ec80 msgr2=0x7f28a41577b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:17.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.943+0000 7f28aa000640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a413ec80 0x7f28a41577b0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f288c000f70 tx=0x7f288c00c890 comp rx=0 tx=0).stop 2024-03-24T00:34:17.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.943+0000 7f28aa000640 1 -- 172.21.15.40:0/177751735 shutdown_connections 2024-03-24T00:34:17.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.943+0000 7f28aa000640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2880067660 0x7f2880069b20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.943+0000 7f28aa000640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28a4141bc0 0x7f28a413cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.943+0000 7f28aa000640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28a413ec80 0x7f28a41577b0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.943+0000 7f28aa000640 1 --2- 172.21.15.40:0/177751735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28a4072310 0x7f28a413d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:17.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.943+0000 7f28aa000640 1 -- 172.21.15.40:0/177751735 >> 172.21.15.40:0/177751735 conn(0x7f28a40b17b0 msgr2=0x7f28a4071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:17.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.943+0000 7f28aa000640 1 -- 172.21.15.40:0/177751735 shutdown_connections 2024-03-24T00:34:17.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:17.943+0000 7f28aa000640 1 -- 172.21.15.40:0/177751735 wait complete. 2024-03-24T00:34:18.368 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 52, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:34:12.019118+0000', 'modified': '2024-03-24T00:34:15.031104+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6474}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6474': {'gid': 6474, 'name': 'd', 'rank': 0, 'incarnation': 48, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.53:6835/501079401', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 501079401}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 501079401}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6474, 'qdb_cluster': [6474]} 2024-03-24T00:34:18.368 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:34:18.368 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T00:34:18.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.507+0000 7fd080a00640 1 Processor -- start 2024-03-24T00:34:18.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.507+0000 7fd080a00640 1 -- start start 2024-03-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd080a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd07c0cfcc0 0x7fd07c0b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd080a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd07c071c70 0x7fd07c0aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd080a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c0b4ae0 con 0x7fd07c0b4460 2024-03-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd080a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07c15e9b0 con 0x7fd07c0cfcc0 2024-03-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd080a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c0b4dc0 con 0x7fd07c071c70 2024-03-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd079e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd07c0cfcc0 0x7fd07c0b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd07a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd07c071c70 0x7fd07c0aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd07a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd07c071c70 0x7fd07c0aef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:58984/0 (socket says 172.21.15.40:58984) 2024-03-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd07a800640 1 -- 172.21.15.40:0/1573386260 learned_addr learned my addr 172.21.15.40:0/1573386260 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd079e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd07c0cfcc0 0x7fd07c0b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:44740/0 (socket says 172.21.15.40:44740) 2024-03-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd07a800640 1 -- 172.21.15.40:0/1573386260 >> v2:172.21.15.53:3300/0 conn(0x7fd07c0cfcc0 msgr2=0x7fd07c0b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd07a800640 1 --2- 172.21.15.40:0/1573386260 >> v2:172.21.15.53:3300/0 conn(0x7fd07c0cfcc0 0x7fd07c0b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd07a800640 1 -- 172.21.15.40:0/1573386260 >> v1:172.21.15.53:6789/0 conn(0x7fd07c0b4460 legacy=0x7fd07c071560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.511+0000 7fd07a800640 1 -- 172.21.15.40:0/1573386260 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c0ae740 con 0x7fd07c071c70 2024-03-24T00:34:18.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd07a800640 1 --2- 172.21.15.40:0/1573386260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd07c071c70 0x7fd07c0aef40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd074009870 tx=0x7fd07401c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acd48f7cbae0edf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd079400640 1 -- 172.21.15.40:0/1573386260 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0740a7070 con 0x7fd07c071c70 2024-03-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd079400640 1 -- 172.21.15.40:0/1573386260 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0740b0070 con 0x7fd07c071c70 2024-03-24T00:34:18.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd079400640 1 -- 172.21.15.40:0/1573386260 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0740ab890 con 0x7fd07c071c70 2024-03-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd080a00640 1 -- 172.21.15.40:0/1573386260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd07c071c70 msgr2=0x7fd07c0aef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd080a00640 1 --2- 172.21.15.40:0/1573386260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd07c071c70 0x7fd07c0aef40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd074009870 tx=0x7fd07401c560 comp rx=0 tx=0).stop 2024-03-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd080a00640 1 -- 172.21.15.40:0/1573386260 shutdown_connections 2024-03-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd080a00640 1 --2- 172.21.15.40:0/1573386260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd07c071c70 0x7fd07c0aef40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd080a00640 1 --2- 172.21.15.40:0/1573386260 >> v2:172.21.15.53:3300/0 conn(0x7fd07c0cfcc0 0x7fd07c0b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd080a00640 1 -- 172.21.15.40:0/1573386260 >> 172.21.15.40:0/1573386260 conn(0x7fd07c0b21a0 msgr2=0x7fd07c0b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd080a00640 1 -- 172.21.15.40:0/1573386260 shutdown_connections 2024-03-24T00:34:18.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd080a00640 1 -- 172.21.15.40:0/1573386260 wait complete. 2024-03-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd080a00640 1 Processor -- start 2024-03-24T00:34:18.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.515+0000 7fd080a00640 1 -- start start 2024-03-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd080a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd07c071c70 0x7fd07c142bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd080a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd07c0b4460 0x7fd07c1541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd080a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd07c0cfcc0 0x7fd07c156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd080a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c0d0220 con 0x7fd07c071c70 2024-03-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd080a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c0d00a0 con 0x7fd07c0cfcc0 2024-03-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd080a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c0afc90 con 0x7fd07c0b4460 2024-03-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd079e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd07c071c70 0x7fd07c142bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd079e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd07c071c70 0x7fd07c142bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46940/0 (socket says 172.21.15.40:46940) 2024-03-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd079e00640 1 -- 172.21.15.40:0/3463632945 learned_addr learned my addr 172.21.15.40:0/3463632945 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd071400640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd07c0b4460 0x7fd07c1541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd07a800640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd07c0cfcc0 0x7fd07c156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd079e00640 1 -- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd07c0b4460 msgr2=0x7fd07c1541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd079e00640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd07c0b4460 0x7fd07c1541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd079e00640 1 -- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd07c0cfcc0 msgr2=0x7fd07c156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd079e00640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd07c0cfcc0 0x7fd07c156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd079e00640 1 -- 172.21.15.40:0/3463632945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c0b0b70 con 0x7fd07c071c70 2024-03-24T00:34:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd079e00640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd07c071c70 0x7fd07c142bc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd06c009d10 tx=0x7fd06c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd072a00640 1 -- 172.21.15.40:0/3463632945 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c018020 con 0x7fd07c071c70 2024-03-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd072a00640 1 -- 172.21.15.40:0/3463632945 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c014480 con 0x7fd07c071c70 2024-03-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd072a00640 1 -- 172.21.15.40:0/3463632945 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c014830 con 0x7fd07c071c70 2024-03-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd080a00640 1 -- 172.21.15.40:0/3463632945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07c160950 con 0x7fd07c071c70 2024-03-24T00:34:18.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.519+0000 7fd080a00640 1 -- 172.21.15.40:0/3463632945 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07c1472f0 con 0x7fd07c071c70 2024-03-24T00:34:18.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd072a00640 1 -- 172.21.15.40:0/3463632945 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd06c01b020 con 0x7fd07c071c70 2024-03-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd072a00640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd058067690 0x7fd058069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd072a00640 1 -- 172.21.15.40:0/3463632945 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd06c0861d0 con 0x7fd07c071c70 2024-03-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd083ede000 1 Processor -- start 2024-03-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd083ede000 1 -- start start 2024-03-24T00:34:18.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd071400640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd058067690 0x7fd058069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd071400640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd058067690 0x7fd058069b50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd07c160500 tx=0x7fd064009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd083ede000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82794bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd083ede000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c82794c240 0x55c827901ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd083ede000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c827901ff0 0x55c8279023d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd083ede000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c82774f680 con 0x55c82794b920 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd083ede000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8276d6b00 con 0x55c827901ff0 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd083ede000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c82774b720 con 0x55c82794c240 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd079e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82794bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd079e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82794bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46946/0 (socket says 172.21.15.40:46946) 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd079e00640 1 -- 172.21.15.40:0/3035596504 learned_addr learned my addr 172.21.15.40:0/3035596504 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd071400640 1 --2- 172.21.15.40:0/3035596504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c82794c240 0x55c827901ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd07a800640 1 --2- 172.21.15.40:0/3035596504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c827901ff0 0x55c8279023d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd079e00640 1 -- 172.21.15.40:0/3035596504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c82794c240 msgr2=0x55c827901ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd079e00640 1 --2- 172.21.15.40:0/3035596504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c82794c240 0x55c827901ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd079e00640 1 -- 172.21.15.40:0/3035596504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c827901ff0 msgr2=0x55c8279023d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd079e00640 1 --2- 172.21.15.40:0/3035596504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c827901ff0 0x55c8279023d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd079e00640 1 -- 172.21.15.40:0/3035596504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c827727940 con 0x55c82794b920 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd079e00640 1 --2- 172.21.15.40:0/3035596504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82794bd00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd06c05e890 tx=0x7fd06c056760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0f8ddf861abe5e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd06aa00640 1 -- 172.21.15.40:0/3035596504 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c082070 con 0x55c82794b920 2024-03-24T00:34:18.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd06aa00640 1 -- 172.21.15.40:0/3035596504 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c05edc0 con 0x55c82794b920 2024-03-24T00:34:18.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.523+0000 7fd06aa00640 1 -- 172.21.15.40:0/3035596504 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c06e070 con 0x55c82794b920 2024-03-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 -- 172.21.15.40:0/3035596504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 msgr2=0x55c82794bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 --2- 172.21.15.40:0/3035596504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82794bd00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd06c05e890 tx=0x7fd06c056760 comp rx=0 tx=0).stop 2024-03-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 -- 172.21.15.40:0/3035596504 shutdown_connections 2024-03-24T00:34:18.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 --2- 172.21.15.40:0/3035596504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c827901ff0 0x55c8279023d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 --2- 172.21.15.40:0/3035596504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c82794c240 0x55c827901ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 --2- 172.21.15.40:0/3035596504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82794bd00 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 -- 172.21.15.40:0/3035596504 >> 172.21.15.40:0/3035596504 conn(0x55c8278fdfa0 msgr2=0x55c827900390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 -- 172.21.15.40:0/3035596504 shutdown_connections 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 -- 172.21.15.40:0/3035596504 wait complete. 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 -- start start 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 Processor -- start 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c827901ff0 0x55c827930d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c827931280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c82794c240 0x55c8279317c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c82774e990 con 0x55c82794b920 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c827709ae0 con 0x55c827901ff0 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c8276e7750 con 0x55c82794c240 2024-03-24T00:34:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 10 client.0 _ll_register_callbacks cb 0x55c8278ecff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd071400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c827931280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd079e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c827901ff0 0x55c827930d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd07a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c82794c240 0x55c8279317c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd079e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c827901ff0 0x55c827930d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:44766/0 (socket says 172.21.15.40:44766) 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd079e00640 1 -- 172.21.15.40:0/1939141775 learned_addr learned my addr 172.21.15.40:0/1939141775 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd071400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c827931280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46958/0 (socket says 172.21.15.40:46958) 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd071400640 1 -- 172.21.15.40:0/1939141775 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c82794c240 msgr2=0x55c8279317c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd071400640 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c82794c240 0x55c8279317c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd071400640 1 -- 172.21.15.40:0/1939141775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c827901ff0 msgr2=0x55c827930d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd071400640 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c827901ff0 0x55c827930d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd071400640 1 -- 172.21.15.40:0/1939141775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c827794d80 con 0x55c82794b920 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd079e00640 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c827901ff0 0x55c827930d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd07a800640 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c82794c240 0x55c8279317c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd071400640 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c827931280 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fd064000f70 tx=0x7fd064000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eefbbd89e6ec64d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd05f400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:34:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd05f400640 1 -- 172.21.15.40:0/1939141775 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd064025020 con 0x55c82794b920 2024-03-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 -- 172.21.15.40:0/1939141775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd06c05edc0 con 0x55c82794b920 2024-03-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd05f400640 1 -- 172.21.15.40:0/1939141775 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd064021610 con 0x55c82794b920 2024-03-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd05f400640 1 -- 172.21.15.40:0/1939141775 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd06402f050 con 0x55c82794b920 2024-03-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 10 client.6618 fetch_fsmap learned FSMap version 52 2024-03-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd083ede000 1 -- 172.21.15.40:0/1939141775 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) v3 -- 0x55c8276d6d10 con 0x55c82794b920 2024-03-24T00:34:18.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd05f400640 1 -- 172.21.15.40:0/1939141775 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c05edc0 con 0x55c82794b920 2024-03-24T00:34:18.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.527+0000 7fd05f400640 1 -- 172.21.15.40:0/1939141775 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 52) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd064032020 con 0x55c82794b920 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 10 client.6618 fetch_fsmap finished waiting for FSMap version 52 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 10 client.6618 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 20 client.6618 populate_metadata read hostname 'smithi040' 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c827932030 0x55c82795dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 4 client.6618 mds_command: new command op to 6474 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 -- 172.21.15.40:0/1939141775 --> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c8276eab60 con 0x55c827932030 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd079e00640 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c827932030 0x55c82795dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd079e00640 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c827932030 0x55c82795dea0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16729ecb24666f4 server_cookie=c1d046739a44cf2e in_seq=0 out_seq=0 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd05f400640 10 client.6618 ms_handle_connect on v2:172.21.15.53:6834/501079401 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd05f400640 1 -- 172.21.15.40:0/1939141775 <== mds.0 v2:172.21.15.53:6834/501079401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c8276eab60 con 0x55c827932030 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd05f400640 10 client.6618 handle_command_reply: tid=0 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 client.6618 shutdown 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 -- 172.21.15.40:0/1939141775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 msgr2=0x55c827931280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c827931280 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fd064000f70 tx=0x7fd064000ee0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 -- 172.21.15.40:0/1939141775 shutdown_connections 2024-03-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c827932030 0x55c82795dea0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c82794c240 0x55c8279317c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c827931280 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 --2- 172.21.15.40:0/1939141775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c827901ff0 0x55c827930d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 -- 172.21.15.40:0/1939141775 >> 172.21.15.40:0/1939141775 conn(0x55c8278fdfa0 msgr2=0x55c827900c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd05f400640 0 client.6618 ms_handle_reset on v2:172.21.15.53:6834/501079401 2024-03-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 -- 172.21.15.40:0/1939141775 shutdown_connections 2024-03-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 1 -- 172.21.15.40:0/1939141775 wait complete. 2024-03-24T00:34:18.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.531+0000 7fd083ede000 20 client.6618 trim_cache size 0 max 16384 2024-03-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.547+0000 7fd083ede000 1 Processor -- start 2024-03-24T00:34:18.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.547+0000 7fd083ede000 1 -- start start 2024-03-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.547+0000 7fd083ede000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c827901ff0 0x55c82792f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.547+0000 7fd083ede000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82792fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.547+0000 7fd083ede000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c82794c240 0x55c82794a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.547+0000 7fd083ede000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c2031f0 con 0x55c82794b920 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.547+0000 7fd083ede000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c153520 con 0x55c82794c240 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.547+0000 7fd083ede000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c15f790 con 0x55c827901ff0 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd071400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82792fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd079e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c827901ff0 0x55c82792f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd071400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82792fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46960/0 (socket says 172.21.15.40:46960) 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd071400640 1 -- 172.21.15.40:0/2891895531 learned_addr learned my addr 172.21.15.40:0/2891895531 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd07a800640 1 --2- 172.21.15.40:0/2891895531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c82794c240 0x55c82794a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd071400640 1 -- 172.21.15.40:0/2891895531 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c827901ff0 msgr2=0x55c82792f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd071400640 1 --2- 172.21.15.40:0/2891895531 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c827901ff0 0x55c82792f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd071400640 1 -- 172.21.15.40:0/2891895531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c82794c240 msgr2=0x55c82794a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd071400640 1 --2- 172.21.15.40:0/2891895531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c82794c240 0x55c82794a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd071400640 1 -- 172.21.15.40:0/2891895531 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c82771c540 con 0x55c82794b920 2024-03-24T00:34:18.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd071400640 1 --2- 172.21.15.40:0/2891895531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82792fe40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fd064001000 tx=0x7fd064021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f93c7d3a7f94f433 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:18.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd06a000640 1 -- 172.21.15.40:0/2891895531 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd064025070 con 0x55c82794b920 2024-03-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd06a000640 1 -- 172.21.15.40:0/2891895531 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06401b070 con 0x55c82794b920 2024-03-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd06a000640 1 -- 172.21.15.40:0/2891895531 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd064021de0 con 0x55c82794b920 2024-03-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 -- 172.21.15.40:0/2891895531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 msgr2=0x55c82792fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 --2- 172.21.15.40:0/2891895531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82792fe40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fd064001000 tx=0x7fd064021000 comp rx=0 tx=0).stop 2024-03-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 -- 172.21.15.40:0/2891895531 shutdown_connections 2024-03-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 --2- 172.21.15.40:0/2891895531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c82794c240 0x55c82794a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 --2- 172.21.15.40:0/2891895531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82792fe40 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 --2- 172.21.15.40:0/2891895531 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c827901ff0 0x55c82792f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 -- 172.21.15.40:0/2891895531 >> 172.21.15.40:0/2891895531 conn(0x55c8278fdfa0 msgr2=0x55c827958f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 -- 172.21.15.40:0/2891895531 shutdown_connections 2024-03-24T00:34:18.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 -- 172.21.15.40:0/2891895531 wait complete. 2024-03-24T00:34:18.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 -- start start 2024-03-24T00:34:18.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 Processor -- start 2024-03-24T00:34:18.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c827901ff0 0x55c827901690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82795eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c82794c240 0x55c82795f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8276f7a80 con 0x55c82794b920 2024-03-24T00:34:18.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c152f20 con 0x55c82794c240 2024-03-24T00:34:18.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c82780c700 con 0x55c827901ff0 2024-03-24T00:34:18.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd083ede000 10 client.0 _ll_register_callbacks cb 0x7fd064001dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd071400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82795eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd071400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82795eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46974/0 (socket says 172.21.15.40:46974) 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd071400640 1 -- 172.21.15.40:0/4175612424 learned_addr learned my addr 172.21.15.40:0/4175612424 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd079e00640 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c827901ff0 0x55c827901690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.551+0000 7fd07a800640 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c82794c240 0x55c82795f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd079e00640 1 -- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c82794c240 msgr2=0x55c82795f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd079e00640 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c82794c240 0x55c82795f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd079e00640 1 -- 172.21.15.40:0/4175612424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 msgr2=0x55c82795eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd079e00640 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82795eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd079e00640 1 -- 172.21.15.40:0/4175612424 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8276e34b0 con 0x55c827901ff0 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd071400640 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82795eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd079e00640 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c827901ff0 0x55c827901690 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd06c086160 tx=0x7fd06c000f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b99d768abb3030c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:18.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd05f400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd05f400640 1 -- 172.21.15.40:0/4175612424 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c082040 con 0x55c827901ff0 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 -- 172.21.15.40:0/4175612424 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd06401b070 con 0x55c827901ff0 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd05f400640 1 -- 172.21.15.40:0/4175612424 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd06c054ae0 con 0x55c827901ff0 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd05f400640 1 -- 172.21.15.40:0/4175612424 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd06c06e040 con 0x55c827901ff0 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 10 client.5819 fetch_fsmap learned FSMap version 52 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 -- 172.21.15.40:0/4175612424 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=52,monmap=2+}) v3 -- 0x55c8276f62f0 con 0x55c827901ff0 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd05f400640 1 -- 172.21.15.40:0/4175612424 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06401b070 con 0x55c827901ff0 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd05f400640 1 -- 172.21.15.40:0/4175612424 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 52) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fd06c055bb0 con 0x55c827901ff0 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 10 client.5819 fetch_fsmap finished waiting for FSMap version 52 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 10 client.5819 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 20 client.5819 populate_metadata read hostname 'smithi040' 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c8279b05c0 0x55c8279b09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 4 client.5819 mds_command: new command op to 6474 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 -- 172.21.15.40:0/4175612424 --> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c82775cbc0 con 0x55c8279b05c0 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd071400640 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c8279b05c0 0x55c8279b09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd071400640 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c8279b05c0 0x55c8279b09a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ffcee3bad05f3cf server_cookie=db7a847623365f60 in_seq=0 out_seq=0 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd05f400640 10 client.5819 ms_handle_connect on v2:172.21.15.53:6834/501079401 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd05f400640 1 -- 172.21.15.40:0/4175612424 <== mds.0 v2:172.21.15.53:6834/501079401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c82775cbc0 con 0x55c8279b05c0 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd05f400640 10 client.5819 handle_command_reply: tid=0 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 client.5819 shutdown 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 -- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c827901ff0 msgr2=0x55c827901690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c827901ff0 0x55c827901690 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd06c086160 tx=0x7fd06c000f50 comp rx=0 tx=0).stop 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 -- 172.21.15.40:0/4175612424 shutdown_connections 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c8279b05c0 0x55c8279b09a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c82794c240 0x55c82795f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd05f400640 0 client.5819 ms_handle_reset on v2:172.21.15.53:6834/501079401 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c82794b920 0x55c82795eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 --2- 172.21.15.40:0/4175612424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c827901ff0 0x55c827901690 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 -- 172.21.15.40:0/4175612424 >> 172.21.15.40:0/4175612424 conn(0x55c8278fdfa0 msgr2=0x55c827958680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 -- 172.21.15.40:0/4175612424 shutdown_connections 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 1 -- 172.21.15.40:0/4175612424 wait complete. 2024-03-24T00:34:18.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd083ede000 20 client.5819 trim_cache size 0 max 16384 2024-03-24T00:34:18.566 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:34:18.566 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:34:18.566 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:34:18.566 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6474, 2024-03-24T00:34:18.566 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:34:18.566 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:34:18.566 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:34:18.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 5.9454599779999997, 2024-03-24T00:34:18.566 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 52, 2024-03-24T00:34:18.566 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 59, 2024-03-24T00:34:18.567 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 59, 2024-03-24T00:34:18.567 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 17.868426539000001 2024-03-24T00:34:18.567 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:34:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd080a00640 1 -- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd058067690 msgr2=0x7fd058069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd080a00640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd058067690 0x7fd058069b50 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd07c160500 tx=0x7fd064009040 comp rx=0 tx=0).stop 2024-03-24T00:34:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd080a00640 1 -- 172.21.15.40:0/3463632945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd07c071c70 msgr2=0x7fd07c142bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd080a00640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd07c071c70 0x7fd07c142bc0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd06c009d10 tx=0x7fd06c00c490 comp rx=0 tx=0).stop 2024-03-24T00:34:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd080a00640 1 -- 172.21.15.40:0/3463632945 shutdown_connections 2024-03-24T00:34:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd080a00640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd058067690 0x7fd058069b50 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd080a00640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd07c0cfcc0 0x7fd07c156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd080a00640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd07c0b4460 0x7fd07c1541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd080a00640 1 --2- 172.21.15.40:0/3463632945 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd07c071c70 0x7fd07c142bc0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd080a00640 1 -- 172.21.15.40:0/3463632945 >> 172.21.15.40:0/3463632945 conn(0x7fd07c0b21a0 msgr2=0x7fd07c157d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd080a00640 1 -- 172.21.15.40:0/3463632945 shutdown_connections 2024-03-24T00:34:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.555+0000 7fd080a00640 1 -- 172.21.15.40:0/3463632945 wait complete. 2024-03-24T00:34:18.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:34:18.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.715+0000 7f2b6f600640 1 Processor -- start 2024-03-24T00:34:18.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.715+0000 7f2b6f600640 1 -- start start 2024-03-24T00:34:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.715+0000 7f2b6f600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b680cfcc0 0x7f2b680b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.715+0000 7f2b6f600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2b68071c70 0x7f2b680aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.715+0000 7f2b6f600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b680b4ae0 con 0x7f2b680b4460 2024-03-24T00:34:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.715+0000 7f2b6f600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b6815e9b0 con 0x7f2b68071c70 2024-03-24T00:34:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.715+0000 7f2b6f600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680b4dc0 con 0x7f2b680cfcc0 2024-03-24T00:34:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.715+0000 7f2b6cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b680cfcc0 0x7f2b680b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.715+0000 7f2b6d600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2b68071c70 0x7f2b680aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.719+0000 7f2b67e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2b680b4460 0x7f2b68071560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:52232/0 (socket says 172.21.15.40:52232) 2024-03-24T00:34:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.719+0000 7f2b6cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b680cfcc0 0x7f2b680b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:59042/0 (socket says 172.21.15.40:59042) 2024-03-24T00:34:18.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.719+0000 7f2b67e00640 1 -- 172.21.15.40:0/806291393 learned_addr learned my addr 172.21.15.40:0/806291393 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:18.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.719+0000 7f2b6d600640 1 -- 172.21.15.40:0/806291393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b680cfcc0 msgr2=0x7f2b680b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:18.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.719+0000 7f2b6d600640 1 --2- 172.21.15.40:0/806291393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b680cfcc0 0x7f2b680b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.719+0000 7f2b6d600640 1 -- 172.21.15.40:0/806291393 >> v1:172.21.15.53:6789/0 conn(0x7f2b680b4460 legacy=0x7f2b68071560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.719+0000 7f2b6d600640 1 -- 172.21.15.40:0/806291393 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b680ae740 con 0x7f2b68071c70 2024-03-24T00:34:18.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.719+0000 7f2b67400640 1 -- 172.21.15.40:0/806291393 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341253683 0 0) 0x7f2b680b4ae0 con 0x7f2b680b4460 2024-03-24T00:34:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.719+0000 7f2b6d600640 1 --2- 172.21.15.40:0/806291393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b68071c70 0x7f2b680aef40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009d90 tx=0x7f2b6001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd5d1591fef55558 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:18.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.719+0000 7f2b67400640 1 -- 172.21.15.40:0/806291393 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b600b2070 con 0x7f2b68071c70 2024-03-24T00:34:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.719+0000 7f2b67400640 1 -- 172.21.15.40:0/806291393 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b600a5940 con 0x7f2b68071c70 2024-03-24T00:34:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.719+0000 7f2b67400640 1 -- 172.21.15.40:0/806291393 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b600a5d10 con 0x7f2b68071c70 2024-03-24T00:34:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 -- 172.21.15.40:0/806291393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b68071c70 msgr2=0x7f2b680aef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 --2- 172.21.15.40:0/806291393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b68071c70 0x7f2b680aef40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f2b60009d90 tx=0x7f2b6001c970 comp rx=0 tx=0).stop 2024-03-24T00:34:18.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 -- 172.21.15.40:0/806291393 shutdown_connections 2024-03-24T00:34:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 --2- 172.21.15.40:0/806291393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b68071c70 0x7f2b680aef40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 --2- 172.21.15.40:0/806291393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b680cfcc0 0x7f2b680b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 -- 172.21.15.40:0/806291393 >> 172.21.15.40:0/806291393 conn(0x7f2b680b21a0 msgr2=0x7f2b680b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 -- 172.21.15.40:0/806291393 shutdown_connections 2024-03-24T00:34:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 -- 172.21.15.40:0/806291393 wait complete. 2024-03-24T00:34:18.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 Processor -- start 2024-03-24T00:34:18.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 -- start start 2024-03-24T00:34:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b68071c70 0x7f2b68153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b680b4460 0x7f2b68154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b680cfcc0 0x7f2b68150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680d0220 con 0x7f2b680b4460 2024-03-24T00:34:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680d00a0 con 0x7f2b680cfcc0 2024-03-24T00:34:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6f600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680afc90 con 0x7f2b68071c70 2024-03-24T00:34:18.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b680b4460 0x7f2b68154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b680cfcc0 0x7f2b68150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b68071c70 0x7f2b68153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b680b4460 0x7f2b68154bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46984/0 (socket says 172.21.15.40:46984) 2024-03-24T00:34:18.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b67e00640 1 -- 172.21.15.40:0/2651699801 learned_addr learned my addr 172.21.15.40:0/2651699801 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6d600640 1 -- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b68071c70 msgr2=0x7f2b68153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6d600640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b68071c70 0x7f2b68153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6d600640 1 -- 172.21.15.40:0/2651699801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b680b4460 msgr2=0x7f2b68154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6d600640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b680b4460 0x7f2b68154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b6d600640 1 -- 172.21.15.40:0/2651699801 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b68155410 con 0x7f2b680cfcc0 2024-03-24T00:34:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.723+0000 7f2b67e00640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b680b4460 0x7f2b68154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:18.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.727+0000 7f2b6d600640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b680cfcc0 0x7f2b68150120 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2b600ac000 tx=0x7f2b60019710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.727+0000 7f2b64c00640 1 -- 172.21.15.40:0/2651699801 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b600b2050 con 0x7f2b680cfcc0 2024-03-24T00:34:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.727+0000 7f2b64c00640 1 -- 172.21.15.40:0/2651699801 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b600b3040 con 0x7f2b680cfcc0 2024-03-24T00:34:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.727+0000 7f2b64c00640 1 -- 172.21.15.40:0/2651699801 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b600ad850 con 0x7f2b680cfcc0 2024-03-24T00:34:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.727+0000 7f2b6f600640 1 -- 172.21.15.40:0/2651699801 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b68153690 con 0x7f2b680cfcc0 2024-03-24T00:34:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.727+0000 7f2b6f600640 1 -- 172.21.15.40:0/2651699801 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b68152670 con 0x7f2b680cfcc0 2024-03-24T00:34:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.727+0000 7f2b6f600640 1 -- 172.21.15.40:0/2651699801 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b68068170 con 0x7f2b680cfcc0 2024-03-24T00:34:18.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.727+0000 7f2b64c00640 1 -- 172.21.15.40:0/2651699801 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2b600c0020 con 0x7f2b680cfcc0 2024-03-24T00:34:18.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.727+0000 7f2b64c00640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2b48067660 0x7f2b48069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:18.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.727+0000 7f2b64c00640 1 -- 172.21.15.40:0/2651699801 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b60120050 con 0x7f2b680cfcc0 2024-03-24T00:34:18.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.727+0000 7f2b6cc00640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2b48067660 0x7f2b48069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:18.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.731+0000 7f2b6cc00640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2b48067660 0x7f2b48069b20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2b68168b40 tx=0x7f2b5c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:18.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.731+0000 7f2b64c00640 1 -- 172.21.15.40:0/2651699801 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2b60016560 con 0x7f2b680cfcc0 2024-03-24T00:34:18.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.931+0000 7f2b6f600640 1 -- 172.21.15.40:0/2651699801 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f2b6806dcc0 con 0x7f2b680cfcc0 2024-03-24T00:34:18.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.931+0000 7f2b64c00640 1 -- 172.21.15.40:0/2651699801 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v52) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2b600f05e0 con 0x7f2b680cfcc0 2024-03-24T00:34:18.936 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:34:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.935+0000 7f2b6f600640 1 -- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2b48067660 msgr2=0x7f2b48069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.935+0000 7f2b6f600640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2b48067660 0x7f2b48069b20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2b68168b40 tx=0x7f2b5c009190 comp rx=0 tx=0).stop 2024-03-24T00:34:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.935+0000 7f2b6f600640 1 -- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b680cfcc0 msgr2=0x7f2b68150120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.935+0000 7f2b6f600640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b680cfcc0 0x7f2b68150120 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2b600ac000 tx=0x7f2b60019710 comp rx=0 tx=0).stop 2024-03-24T00:34:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.935+0000 7f2b6f600640 1 -- 172.21.15.40:0/2651699801 shutdown_connections 2024-03-24T00:34:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.935+0000 7f2b6f600640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2b48067660 0x7f2b48069b20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.935+0000 7f2b6f600640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b680cfcc0 0x7f2b68150120 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.935+0000 7f2b6f600640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b680b4460 0x7f2b68154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.935+0000 7f2b6f600640 1 --2- 172.21.15.40:0/2651699801 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b68071c70 0x7f2b68153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:18.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.935+0000 7f2b6f600640 1 -- 172.21.15.40:0/2651699801 >> 172.21.15.40:0/2651699801 conn(0x7f2b680b21a0 msgr2=0x7f2b681519d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:18.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.935+0000 7f2b6f600640 1 -- 172.21.15.40:0/2651699801 shutdown_connections 2024-03-24T00:34:18.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:18.935+0000 7f2b6f600640 1 -- 172.21.15.40:0/2651699801 wait complete. 2024-03-24T00:34:18.957 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:34:18.957 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:34:18.957 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-03-24T00:34:18.957 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:34:18.957 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:34:18.957 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:34:18.957 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:34:18.957 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:34:18.957 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:34:18.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:18.958 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-03-24T00:34:18.970 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:34:18.970 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:34:18.970 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:34:18.971 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:34:18.971 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-24T00:34:18.971 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:34:18.971 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:34:18.971 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:34:18.971 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-03-24T00:34:18.971 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:34:18.971 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42291sec preferred_lft 42291sec 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 02:42:9e:ab:82:f9 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:34:18.972 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-24T00:34:18.973 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:34:18.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:18.974 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:34:19.029 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-03-24T00:34:19.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:19.032 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-03-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-03-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:34:19.085 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi040.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-24T00:34:19.086 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:34:19.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:19.086 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:34:19.086 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:34:19.086 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T00:34:19.086 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T00:34:19.086 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T00:34:19.086 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:34:19.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:19.177 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-24T00:34:19.186 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:34:19.186 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:34:19.187 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:34:19.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:19.187 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-03-24T00:34:19.247 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T00:34:19.247 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:34:19.248 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:34:19.299 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:34:19.300 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:34:19.300 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:34:19.351 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:34:19.412 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:34:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:34:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:34:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:34:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:34:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:34:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:34:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:34:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:34:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:34:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:34:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:34:19.465 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:34:19.466 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:34:19.467 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:34:19.467 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:34:19.467 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:34:19.467 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:34:19.467 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:34:19.467 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:34:19.467 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:34:19.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:19.468 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:34:19.519 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T00:34:19.519 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T00:34:19.562 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:34:19.582 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:34:19.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:34:19.638 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:34:19.638 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:34:19.638 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:34:19.638 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:34:19.638 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:34:19.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:34:19.638 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:34:19.638 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:34:19.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:34:19.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:34:19.639 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:34:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:34:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:34:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:34:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:34:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:34:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:34:19.640 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:34:19.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:19.641 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:34:19.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:2024-03-24T00:34:19.659+0000 7fe8dbbc6580 -1 init, newargv = 0x56552dbc2a10 newargc=15 2024-03-24T00:34:19.660 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[42300]: starting ceph client 2024-03-24T00:34:19.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[42300]: starting fuse 2024-03-24T00:34:19.669 INFO:teuthology.orchestra.run.smithi040.stdout:43 2024-03-24T00:34:19.669 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-24T00:34:19.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:34:19.669 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-03-24T00:34:19.669 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-03-24T00:34:19.669 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-03-24T00:34:19.669 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:34:19.670 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-03-24T00:34:19.671 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-03-24T00:34:19.671 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-03-24T00:34:19.671 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:34:19.671 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:34:19.671 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.0")) 2024-03-24T00:34:19.671 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-03-24T00:34:19.794 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.0.42300.asok 2024-03-24T00:34:19.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:34:19.805 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.42300.asok 2024-03-24T00:34:19.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:19.805 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.42300.asok status 2024-03-24T00:34:19.979 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:34:19.979 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-03-24T00:34:19.979 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "0", 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "42300", 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6412, 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 6412 2024-03-24T00:34:19.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2903200941 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "172.21.15.40:0", 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2903200941 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.6412 172.21.15.40:0/2903200941", 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "172.21.15.40:0/2903200941", 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-03-24T00:34:19.981 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 52, 2024-03-24T00:34:19.982 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 59, 2024-03-24T00:34:19.982 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:34:19.982 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-03-24T00:34:19.982 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-03-24T00:34:19.982 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:34:19.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:19.995 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-03-24T00:34:19.995 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T00:34:20.005 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-03-24T00:34:20.006 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T00:34:20.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:20.006 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:34:20.067 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random True 2024-03-24T00:34:20.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c03e00640 1 Processor -- start 2024-03-24T00:34:20.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c03e00640 1 -- start start 2024-03-24T00:34:20.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c03e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5bfc15bd40 0x7f5bfc0d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c03e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5bfc0d1520 0x7f5bfc0cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c03e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc15f970 con 0x7f5bfc15b960 2024-03-24T00:34:20.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c03e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bfc0d1990 con 0x7f5bfc0d1520 2024-03-24T00:34:20.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c03e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bfc160580 con 0x7f5bfc15bd40 2024-03-24T00:34:20.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c01400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5bfc15b960 0x7f5bfc070d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37680/0 (socket says 172.21.15.40:37680) 2024-03-24T00:34:20.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c01400640 1 -- 172.21.15.40:0/2163257821 learned_addr learned my addr 172.21.15.40:0/2163257821 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:20.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c01e00640 1 --2- 172.21.15.40:0/2163257821 >> v2:172.21.15.40:3300/0 conn(0x7f5bfc0d1520 0x7f5bfc0cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c00a00640 1 --2- 172.21.15.40:0/2163257821 >> v2:172.21.15.53:3300/0 conn(0x7f5bfc15bd40 0x7f5bfc0d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5bfbe00640 1 -- 172.21.15.40:0/2163257821 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849298848 0 0) 0x7f5bfc15f970 con 0x7f5bfc15b960 2024-03-24T00:34:20.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5bfbe00640 1 -- 172.21.15.40:0/2163257821 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be4003530 con 0x7f5bfc15b960 2024-03-24T00:34:20.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c01e00640 1 -- 172.21.15.40:0/2163257821 >> v2:172.21.15.53:3300/0 conn(0x7f5bfc15bd40 msgr2=0x7f5bfc0d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:20.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c01e00640 1 --2- 172.21.15.40:0/2163257821 >> v2:172.21.15.53:3300/0 conn(0x7f5bfc15bd40 0x7f5bfc0d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c01e00640 1 -- 172.21.15.40:0/2163257821 >> v1:172.21.15.40:6789/0 conn(0x7f5bfc15b960 legacy=0x7f5bfc070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c01e00640 1 -- 172.21.15.40:0/2163257821 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc15fc50 con 0x7f5bfc0d1520 2024-03-24T00:34:20.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.251+0000 7f5c00a00640 1 --2- 172.21.15.40:0/2163257821 >> v2:172.21.15.53:3300/0 conn(0x7f5bfc15bd40 0x7f5bfc0d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:20.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.255+0000 7f5c01e00640 1 --2- 172.21.15.40:0/2163257821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bfc0d1520 0x7f5bfc0cda00 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc15f730 tx=0x7f5bf401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98dd443169dd7327 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:20.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.255+0000 7f5bfbe00640 1 -- 172.21.15.40:0/2163257821 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf40b2070 con 0x7f5bfc0d1520 2024-03-24T00:34:20.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.255+0000 7f5bfbe00640 1 -- 172.21.15.40:0/2163257821 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf40a58e0 con 0x7f5bfc0d1520 2024-03-24T00:34:20.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.255+0000 7f5bfbe00640 1 -- 172.21.15.40:0/2163257821 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf40a5cb0 con 0x7f5bfc0d1520 2024-03-24T00:34:20.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.255+0000 7f5c03e00640 1 -- 172.21.15.40:0/2163257821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bfc0d1520 msgr2=0x7f5bfc0cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.255+0000 7f5c03e00640 1 --2- 172.21.15.40:0/2163257821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bfc0d1520 0x7f5bfc0cda00 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc15f730 tx=0x7f5bf401c930 comp rx=0 tx=0).stop 2024-03-24T00:34:20.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.255+0000 7f5c03e00640 1 -- 172.21.15.40:0/2163257821 shutdown_connections 2024-03-24T00:34:20.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.255+0000 7f5c03e00640 1 --2- 172.21.15.40:0/2163257821 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bfc0d1520 0x7f5bfc0cda00 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.255+0000 7f5c03e00640 1 --2- 172.21.15.40:0/2163257821 >> v2:172.21.15.53:3300/0 conn(0x7f5bfc15bd40 0x7f5bfc0d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.255+0000 7f5c03e00640 1 -- 172.21.15.40:0/2163257821 >> 172.21.15.40:0/2163257821 conn(0x7f5bfc06ecf0 msgr2=0x7f5bfc06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:20.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.255+0000 7f5c03e00640 1 -- 172.21.15.40:0/2163257821 shutdown_connections 2024-03-24T00:34:20.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.255+0000 7f5c03e00640 1 -- 172.21.15.40:0/2163257821 wait complete. 2024-03-24T00:34:20.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c03e00640 1 Processor -- start 2024-03-24T00:34:20.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c03e00640 1 -- start start 2024-03-24T00:34:20.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c03e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bfc0d1520 0x7f5bfc0cba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c03e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5bfc15b960 0x7f5bfc0c4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c03e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bfc15bd40 0x7f5bfc151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c03e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0ce8d0 con 0x7f5bfc15bd40 2024-03-24T00:34:20.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c03e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0ce750 con 0x7f5bfc0d1520 2024-03-24T00:34:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c03e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0cea50 con 0x7f5bfc15b960 2024-03-24T00:34:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c01e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bfc15bd40 0x7f5bfc151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c01400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bfc0d1520 0x7f5bfc0cba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c01e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bfc15bd40 0x7f5bfc151df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47004/0 (socket says 172.21.15.40:47004) 2024-03-24T00:34:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c01e00640 1 -- 172.21.15.40:0/4097045422 learned_addr learned my addr 172.21.15.40:0/4097045422 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c00a00640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5bfc15b960 0x7f5bfc0c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c01e00640 1 -- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5bfc15b960 msgr2=0x7f5bfc0c4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c01e00640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5bfc15b960 0x7f5bfc0c4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c01e00640 1 -- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bfc0d1520 msgr2=0x7f5bfc0cba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c01e00640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bfc0d1520 0x7f5bfc0cba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c01e00640 1 -- 172.21.15.40:0/4097045422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc152540 con 0x7f5bfc15bd40 2024-03-24T00:34:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c01e00640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bfc15bd40 0x7f5bfc151df0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5bf401a560 tx=0x7f5bf40196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:20.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5bf9600640 1 -- 172.21.15.40:0/4097045422 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf40b2050 con 0x7f5bfc15bd40 2024-03-24T00:34:20.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5bf9600640 1 -- 172.21.15.40:0/4097045422 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf40be070 con 0x7f5bfc15bd40 2024-03-24T00:34:20.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c03e00640 1 -- 172.21.15.40:0/4097045422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc152820 con 0x7f5bfc15bd40 2024-03-24T00:34:20.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5bf9600640 1 -- 172.21.15.40:0/4097045422 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf40ada90 con 0x7f5bfc15bd40 2024-03-24T00:34:20.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.259+0000 7f5c03e00640 1 -- 172.21.15.40:0/4097045422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc06b1f0 con 0x7f5bfc15bd40 2024-03-24T00:34:20.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.263+0000 7f5c03e00640 1 -- 172.21.15.40:0/4097045422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc068170 con 0x7f5bfc15bd40 2024-03-24T00:34:20.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.263+0000 7f5bf9600640 1 -- 172.21.15.40:0/4097045422 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5bf40a5760 con 0x7f5bfc15bd40 2024-03-24T00:34:20.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.263+0000 7f5bf9600640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5bdc067660 0x7f5bdc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.263+0000 7f5bf9600640 1 -- 172.21.15.40:0/4097045422 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5bf41208b0 con 0x7f5bfc15bd40 2024-03-24T00:34:20.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.263+0000 7f5c01400640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5bdc067660 0x7f5bdc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.263+0000 7f5c01400640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5bdc067660 0x7f5bdc069b20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0007fd0 tx=0x7f5bf0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:20.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.267+0000 7f5bf9600640 1 -- 172.21.15.40:0/4097045422 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5bf4016520 con 0x7f5bfc15bd40 2024-03-24T00:34:20.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.431+0000 7f5c03e00640 1 -- 172.21.15.40:0/4097045422 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random}] v 0) v1 -- 0x7f5bfc0c4c60 con 0x7f5bfc15bd40 2024-03-24T00:34:20.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5bf9600640 1 -- 172.21.15.40:0/4097045422 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random}]=0 v28)=0 v28) v1 ==== 128+0+0 (secure 0 0 0) 0x7f5bf40f0e40 con 0x7f5bfc15bd40 2024-03-24T00:34:20.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5c03e00640 1 -- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5bdc067660 msgr2=0x7f5bdc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5c03e00640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5bdc067660 0x7f5bdc069b20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0007fd0 tx=0x7f5bf0009040 comp rx=0 tx=0).stop 2024-03-24T00:34:20.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5c03e00640 1 -- 172.21.15.40:0/4097045422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bfc15bd40 msgr2=0x7f5bfc151df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5c03e00640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bfc15bd40 0x7f5bfc151df0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5bf401a560 tx=0x7f5bf40196d0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5c03e00640 1 -- 172.21.15.40:0/4097045422 shutdown_connections 2024-03-24T00:34:20.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5c03e00640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5bdc067660 0x7f5bdc069b20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5c03e00640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5bfc15bd40 0x7f5bfc151df0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5c03e00640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5bfc15b960 0x7f5bfc0c4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5c03e00640 1 --2- 172.21.15.40:0/4097045422 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bfc0d1520 0x7f5bfc0cba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5c03e00640 1 -- 172.21.15.40:0/4097045422 >> 172.21.15.40:0/4097045422 conn(0x7f5bfc06ecf0 msgr2=0x7f5bfc15cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:20.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5c03e00640 1 -- 172.21.15.40:0/4097045422 shutdown_connections 2024-03-24T00:34:20.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.435+0000 7f5c03e00640 1 -- 172.21.15.40:0/4097045422 wait complete. 2024-03-24T00:34:20.458 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_distributed True 2024-03-24T00:34:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b4400640 1 Processor -- start 2024-03-24T00:34:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b4400640 1 -- start start 2024-03-24T00:34:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b4400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f89ac15e350 0x7f89ac0cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b4400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f89ac0c9580 0x7f89ac0c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b4400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ac0734e0 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b4400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac0cd660 con 0x7f89ac0c9580 2024-03-24T00:34:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b4400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac0c9f70 con 0x7f89ac15e350 2024-03-24T00:34:20.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b2400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f89ac0ad1e0 0x7f89ac0ad5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37686/0 (socket says 172.21.15.40:37686) 2024-03-24T00:34:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b2400640 1 -- 172.21.15.40:0/2401323177 learned_addr learned my addr 172.21.15.40:0/2401323177 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b1a00640 1 --2- 172.21.15.40:0/2401323177 >> v2:172.21.15.53:3300/0 conn(0x7f89ac15e350 0x7f89ac0cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b1000640 1 --2- 172.21.15.40:0/2401323177 >> v2:172.21.15.40:3300/0 conn(0x7f89ac0c9580 0x7f89ac0c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b1a00640 1 -- 172.21.15.40:0/2401323177 >> v2:172.21.15.40:3300/0 conn(0x7f89ac0c9580 msgr2=0x7f89ac0c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b1a00640 1 --2- 172.21.15.40:0/2401323177 >> v2:172.21.15.40:3300/0 conn(0x7f89ac0c9580 0x7f89ac0c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b1a00640 1 -- 172.21.15.40:0/2401323177 >> v1:172.21.15.40:6789/0 conn(0x7f89ac0ad1e0 legacy=0x7f89ac0ad5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89b1a00640 1 -- 172.21.15.40:0/2401323177 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac0d0030 con 0x7f89ac15e350 2024-03-24T00:34:20.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.599+0000 7f89abe00640 1 -- 172.21.15.40:0/2401323177 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318868699 0 0) 0x7f89ac0734e0 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.603+0000 7f89b1a00640 1 --2- 172.21.15.40:0/2401323177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ac15e350 0x7f89ac0cfaf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f89ac070d30 tx=0x7f89a001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fade167729b57882 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:20.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.603+0000 7f89abe00640 1 -- 172.21.15.40:0/2401323177 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a00b1070 con 0x7f89ac15e350 2024-03-24T00:34:20.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.603+0000 7f89abe00640 1 -- 172.21.15.40:0/2401323177 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a00ac5f0 con 0x7f89ac15e350 2024-03-24T00:34:20.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.603+0000 7f89abe00640 1 -- 172.21.15.40:0/2401323177 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a00ac9c0 con 0x7f89ac15e350 2024-03-24T00:34:20.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.603+0000 7f89b4400640 1 -- 172.21.15.40:0/2401323177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ac15e350 msgr2=0x7f89ac0cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.603+0000 7f89b4400640 1 --2- 172.21.15.40:0/2401323177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ac15e350 0x7f89ac0cfaf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f89ac070d30 tx=0x7f89a001c480 comp rx=0 tx=0).stop 2024-03-24T00:34:20.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.603+0000 7f89b4400640 1 -- 172.21.15.40:0/2401323177 shutdown_connections 2024-03-24T00:34:20.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.603+0000 7f89b4400640 1 --2- 172.21.15.40:0/2401323177 >> v2:172.21.15.40:3300/0 conn(0x7f89ac0c9580 0x7f89ac0c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.603+0000 7f89b4400640 1 --2- 172.21.15.40:0/2401323177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ac15e350 0x7f89ac0cfaf0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.603+0000 7f89b4400640 1 -- 172.21.15.40:0/2401323177 >> 172.21.15.40:0/2401323177 conn(0x7f89ac072870 msgr2=0x7f89ac072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:20.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.603+0000 7f89b4400640 1 -- 172.21.15.40:0/2401323177 shutdown_connections 2024-03-24T00:34:20.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.603+0000 7f89b4400640 1 -- 172.21.15.40:0/2401323177 wait complete. 2024-03-24T00:34:20.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b4400640 1 Processor -- start 2024-03-24T00:34:20.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b4400640 1 -- start start 2024-03-24T00:34:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b4400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ac0ad1e0 0x7f89ac15b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b4400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f89ac0c9580 0x7f89ac161160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b4400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ac15e350 0x7f89ac153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b4400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac0716b0 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b4400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac071fe0 con 0x7f89ac15e350 2024-03-24T00:34:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b4400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac0cda30 con 0x7f89ac0c9580 2024-03-24T00:34:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ac0ad1e0 0x7f89ac15b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ac0ad1e0 0x7f89ac15b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47022/0 (socket says 172.21.15.40:47022) 2024-03-24T00:34:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b1a00640 1 -- 172.21.15.40:0/682586001 learned_addr learned my addr 172.21.15.40:0/682586001 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:20.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b1000640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f89ac0c9580 0x7f89ac161160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b2400640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ac15e350 0x7f89ac153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b1a00640 1 -- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f89ac0c9580 msgr2=0x7f89ac161160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b1a00640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f89ac0c9580 0x7f89ac161160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b1a00640 1 -- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ac15e350 msgr2=0x7f89ac153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b1a00640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ac15e350 0x7f89ac153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b1a00640 1 -- 172.21.15.40:0/682586001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac15bd60 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b1a00640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ac0ad1e0 0x7f89ac15b780 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f89a0000c10 tx=0x7f89a001acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89a9600640 1 -- 172.21.15.40:0/682586001 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a00bd070 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89a9600640 1 -- 172.21.15.40:0/682586001 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a00b2040 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b4400640 1 -- 172.21.15.40:0/682586001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ac15c040 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89a9600640 1 -- 172.21.15.40:0/682586001 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a00b1050 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b4400640 1 -- 172.21.15.40:0/682586001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ac15c600 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.607+0000 7f89b4400640 1 -- 172.21.15.40:0/682586001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac074c30 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.611+0000 7f89a9600640 1 -- 172.21.15.40:0/682586001 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f89a00c0070 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.611+0000 7f89a9600640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f898c067660 0x7f898c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.611+0000 7f89a9600640 1 -- 172.21.15.40:0/682586001 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89a011f270 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.611+0000 7f89b1000640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f898c067660 0x7f898c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.611+0000 7f89b1000640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f898c067660 0x7f898c069b20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f89ac15fe00 tx=0x7f8998008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:20.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.615+0000 7f89a9600640 1 -- 172.21.15.40:0/682586001 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f89a00a9070 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.783+0000 7f89b4400640 1 -- 172.21.15.40:0/682586001 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_distributed}] v 0) v1 -- 0x7f89ac068170 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.787+0000 7f89a9600640 1 -- 172.21.15.40:0/682586001 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_distributed}]=0 v29)=0 v29) v1 ==== 133+0+0 (secure 0 0 0) 0x7f89a0016070 con 0x7f89ac0ad1e0 2024-03-24T00:34:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.791+0000 7f89b4400640 1 -- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f898c067660 msgr2=0x7f898c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.791+0000 7f89b4400640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f898c067660 0x7f898c069b20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f89ac15fe00 tx=0x7f8998008040 comp rx=0 tx=0).stop 2024-03-24T00:34:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.791+0000 7f89b4400640 1 -- 172.21.15.40:0/682586001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ac0ad1e0 msgr2=0x7f89ac15b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.791+0000 7f89b4400640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ac0ad1e0 0x7f89ac15b780 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f89a0000c10 tx=0x7f89a001acc0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.791+0000 7f89b4400640 1 -- 172.21.15.40:0/682586001 shutdown_connections 2024-03-24T00:34:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.791+0000 7f89b4400640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f898c067660 0x7f898c069b20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.791+0000 7f89b4400640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89ac15e350 0x7f89ac153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.791+0000 7f89b4400640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f89ac0c9580 0x7f89ac161160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.791+0000 7f89b4400640 1 --2- 172.21.15.40:0/682586001 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f89ac0ad1e0 0x7f89ac15b780 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.791+0000 7f89b4400640 1 -- 172.21.15.40:0/682586001 >> 172.21.15.40:0/682586001 conn(0x7f89ac072870 msgr2=0x7f89ac0cefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.791+0000 7f89b4400640 1 -- 172.21.15.40:0/682586001 shutdown_connections 2024-03-24T00:34:20.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.791+0000 7f89b4400640 1 -- 172.21.15.40:0/682586001 wait complete. 2024-03-24T00:34:20.813 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random_max 1.0 2024-03-24T00:34:20.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e07800640 1 Processor -- start 2024-03-24T00:34:20.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e07800640 1 -- start start 2024-03-24T00:34:20.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e07800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e0015ac60 0x7f1e000b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e07800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1e0006cb80 0x7f1e000b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e07800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e000b3260 con 0x7f1e000b4840 2024-03-24T00:34:20.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e07800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e000b3e70 con 0x7f1e0015ac60 2024-03-24T00:34:20.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e07800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e000b3540 con 0x7f1e0006cb80 2024-03-24T00:34:20.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e05800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1e000b4840 0x7f1e000ccd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37696/0 (socket says 172.21.15.40:37696) 2024-03-24T00:34:20.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e05800640 1 -- 172.21.15.40:0/2174517111 learned_addr learned my addr 172.21.15.40:0/2174517111 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:20.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e04e00640 1 --2- 172.21.15.40:0/2174517111 >> v2:172.21.15.40:3300/0 conn(0x7f1e0015ac60 0x7f1e000b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1dffe00640 1 --2- 172.21.15.40:0/2174517111 >> v2:172.21.15.53:3300/0 conn(0x7f1e0006cb80 0x7f1e000b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e04e00640 1 -- 172.21.15.40:0/2174517111 >> v2:172.21.15.53:3300/0 conn(0x7f1e0006cb80 msgr2=0x7f1e000b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:20.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e04e00640 1 --2- 172.21.15.40:0/2174517111 >> v2:172.21.15.53:3300/0 conn(0x7f1e0006cb80 0x7f1e000b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e04e00640 1 -- 172.21.15.40:0/2174517111 >> v1:172.21.15.40:6789/0 conn(0x7f1e000b4840 legacy=0x7f1e000ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1e04e00640 1 -- 172.21.15.40:0/2174517111 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e000b3820 con 0x7f1e0015ac60 2024-03-24T00:34:20.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.955+0000 7f1dff400640 1 -- 172.21.15.40:0/2174517111 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290976345 0 0) 0x7f1e000b3260 con 0x7f1e000b4840 2024-03-24T00:34:20.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.959+0000 7f1e04e00640 1 --2- 172.21.15.40:0/2174517111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0015ac60 0x7f1e000b2960 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1e0015e7d0 tx=0x7f1df401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ff33e889585544 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:20.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.959+0000 7f1dff400640 1 -- 172.21.15.40:0/2174517111 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df40b0070 con 0x7f1e0015ac60 2024-03-24T00:34:20.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.959+0000 7f1dff400640 1 -- 172.21.15.40:0/2174517111 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df40ab600 con 0x7f1e0015ac60 2024-03-24T00:34:20.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.959+0000 7f1dff400640 1 -- 172.21.15.40:0/2174517111 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df40ab9d0 con 0x7f1e0015ac60 2024-03-24T00:34:20.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.959+0000 7f1e07800640 1 -- 172.21.15.40:0/2174517111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0015ac60 msgr2=0x7f1e000b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.959+0000 7f1e07800640 1 --2- 172.21.15.40:0/2174517111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0015ac60 0x7f1e000b2960 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1e0015e7d0 tx=0x7f1df401c3c0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.959+0000 7f1e07800640 1 -- 172.21.15.40:0/2174517111 shutdown_connections 2024-03-24T00:34:20.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.959+0000 7f1e07800640 1 --2- 172.21.15.40:0/2174517111 >> v2:172.21.15.53:3300/0 conn(0x7f1e0006cb80 0x7f1e000b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.959+0000 7f1e07800640 1 --2- 172.21.15.40:0/2174517111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0015ac60 0x7f1e000b2960 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.959+0000 7f1e07800640 1 -- 172.21.15.40:0/2174517111 >> 172.21.15.40:0/2174517111 conn(0x7f1e000b1660 msgr2=0x7f1e000b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:20.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.959+0000 7f1e07800640 1 -- 172.21.15.40:0/2174517111 shutdown_connections 2024-03-24T00:34:20.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.959+0000 7f1e07800640 1 -- 172.21.15.40:0/2174517111 wait complete. 2024-03-24T00:34:20.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e07800640 1 Processor -- start 2024-03-24T00:34:20.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e07800640 1 -- start start 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e07800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0006cb80 0x7f1e00135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e07800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e000b4840 0x7f1e00146d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e07800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e0015ac60 0x7f1e001570d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e07800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e00167a40 con 0x7f1e000b4840 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e07800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e001678c0 con 0x7f1e0015ac60 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e07800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e00167bc0 con 0x7f1e0006cb80 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1dffe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e000b4840 0x7f1e00146d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1dffe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e000b4840 0x7f1e00146d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47044/0 (socket says 172.21.15.40:47044) 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1dffe00640 1 -- 172.21.15.40:0/2070408932 learned_addr learned my addr 172.21.15.40:0/2070408932 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e04e00640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0006cb80 0x7f1e00135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e05800640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e0015ac60 0x7f1e001570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1dffe00640 1 -- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0006cb80 msgr2=0x7f1e00135fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1dffe00640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0006cb80 0x7f1e00135fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1dffe00640 1 -- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e0015ac60 msgr2=0x7f1e001570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:20.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1dffe00640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e0015ac60 0x7f1e001570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:20.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1dffe00640 1 -- 172.21.15.40:0/2070408932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0015e260 con 0x7f1e000b4840 2024-03-24T00:34:20.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e05800640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e0015ac60 0x7f1e001570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:20.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1dffe00640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e000b4840 0x7f1e00146d60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00e900 tx=0x7f1dec00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:20.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1dfcc00640 1 -- 172.21.15.40:0/2070408932 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec017070 con 0x7f1e000b4840 2024-03-24T00:34:20.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1dfcc00640 1 -- 172.21.15.40:0/2070408932 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dec00ca00 con 0x7f1e000b4840 2024-03-24T00:34:20.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1dfcc00640 1 -- 172.21.15.40:0/2070408932 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec00cdf0 con 0x7f1e000b4840 2024-03-24T00:34:20.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e07800640 1 -- 172.21.15.40:0/2070408932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e00147730 con 0x7f1e000b4840 2024-03-24T00:34:20.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e07800640 1 -- 172.21.15.40:0/2070408932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0015df80 con 0x7f1e000b4840 2024-03-24T00:34:20.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.963+0000 7f1e07800640 1 -- 172.21.15.40:0/2070408932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e00068170 con 0x7f1e000b4840 2024-03-24T00:34:20.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.967+0000 7f1dfcc00640 1 -- 172.21.15.40:0/2070408932 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f1dec01a070 con 0x7f1e000b4840 2024-03-24T00:34:20.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.967+0000 7f1dfcc00640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1de0067660 0x7f1de0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:20.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.967+0000 7f1dfcc00640 1 -- 172.21.15.40:0/2070408932 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1dec0858a0 con 0x7f1e000b4840 2024-03-24T00:34:20.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.967+0000 7f1e04e00640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1de0067660 0x7f1de0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:20.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.967+0000 7f1e04e00640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1de0067660 0x7f1de0069b20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1df40ad000 tx=0x7f1df40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:20.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:20.971+0000 7f1dfcc00640 1 -- 172.21.15.40:0/2070408932 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1dec055e30 con 0x7f1e000b4840 2024-03-24T00:34:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.135+0000 7f1e07800640 1 -- 172.21.15.40:0/2070408932 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random_max}] v 0) v1 -- 0x7f1e00152ff0 con 0x7f1e000b4840 2024-03-24T00:34:21.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1dfcc00640 1 -- 172.21.15.40:0/2070408932 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random_max}]=0 v30)=0 v30) v1 ==== 131+0+0 (secure 0 0 0) 0x7f1dec059ed0 con 0x7f1e000b4840 2024-03-24T00:34:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1e07800640 1 -- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1de0067660 msgr2=0x7f1de0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1e07800640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1de0067660 0x7f1de0069b20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1df40ad000 tx=0x7f1df40a6000 comp rx=0 tx=0).stop 2024-03-24T00:34:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1e07800640 1 -- 172.21.15.40:0/2070408932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e000b4840 msgr2=0x7f1e00146d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1e07800640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e000b4840 0x7f1e00146d60 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f1dec00e900 tx=0x7f1dec00edc0 comp rx=0 tx=0).stop 2024-03-24T00:34:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1e07800640 1 -- 172.21.15.40:0/2070408932 shutdown_connections 2024-03-24T00:34:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1e07800640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1de0067660 0x7f1de0069b20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1e07800640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e0015ac60 0x7f1e001570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1e07800640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e000b4840 0x7f1e00146d60 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1e07800640 1 --2- 172.21.15.40:0/2070408932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0006cb80 0x7f1e00135fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1e07800640 1 -- 172.21.15.40:0/2070408932 >> 172.21.15.40:0/2070408932 conn(0x7f1e000b1660 msgr2=0x7f1e00155e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1e07800640 1 -- 172.21.15.40:0/2070408932 shutdown_connections 2024-03-24T00:34:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:21.139+0000 7f1e07800640 1 -- 172.21.15.40:0/2070408932 wait complete. 2024-03-24T00:34:21.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:21.162 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:34:21.162 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:34:21.162 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:34:21.162 DEBUG:teuthology.orchestra.run.smithi040:> # Use up a random number of inode numbers so the ephemeral pinning is not the same every test. 2024-03-24T00:34:21.162 DEBUG:teuthology.orchestra.run.smithi040:> mkdir .inode_number_thrash 2024-03-24T00:34:21.162 DEBUG:teuthology.orchestra.run.smithi040:> count=$((RANDOM % 1024)) 2024-03-24T00:34:21.162 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < count; i++)); do touch .inode_number_thrash/$i; done 2024-03-24T00:34:21.163 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf .inode_number_thrash 2024-03-24T00:34:21.163 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:34:23.622 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-24T00:34:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e67e00640 1 Processor -- start 2024-03-24T00:34:23.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e67e00640 1 -- start start 2024-03-24T00:34:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e67e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6815ab50 0x7f5e680aefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e67e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5e6806e2c0 0x7f5e680cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e67e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e680b1770 con 0x7f5e680aea50 2024-03-24T00:34:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e67e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e680b2340 con 0x7f5e6806e2c0 2024-03-24T00:34:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e67e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e680b1a10 con 0x7f5e6815ab50 2024-03-24T00:34:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e5fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5e680aea50 0x7f5e6806dbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37704/0 (socket says 172.21.15.40:37704) 2024-03-24T00:34:23.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e5fe00640 1 -- 172.21.15.40:0/569465468 learned_addr learned my addr 172.21.15.40:0/569465468 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:23.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e66a00640 1 --2- 172.21.15.40:0/569465468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6815ab50 0x7f5e680aefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:23.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e67400640 1 --2- 172.21.15.40:0/569465468 >> v2:172.21.15.53:3300/0 conn(0x7f5e6806e2c0 0x7f5e680cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:23.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e66000640 1 -- 172.21.15.40:0/569465468 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796456849 0 0) 0x7f5e680b1770 con 0x7f5e680aea50 2024-03-24T00:34:23.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.763+0000 7f5e66000640 1 -- 172.21.15.40:0/569465468 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e4c003530 con 0x7f5e680aea50 2024-03-24T00:34:23.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e66a00640 1 -- 172.21.15.40:0/569465468 >> v2:172.21.15.53:3300/0 conn(0x7f5e6806e2c0 msgr2=0x7f5e680cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:23.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e66a00640 1 --2- 172.21.15.40:0/569465468 >> v2:172.21.15.53:3300/0 conn(0x7f5e6806e2c0 0x7f5e680cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:23.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e66a00640 1 -- 172.21.15.40:0/569465468 >> v1:172.21.15.40:6789/0 conn(0x7f5e680aea50 legacy=0x7f5e6806dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:23.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e66a00640 1 -- 172.21.15.40:0/569465468 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e680b1cf0 con 0x7f5e6815ab50 2024-03-24T00:34:23.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e67400640 1 --2- 172.21.15.40:0/569465468 >> v2:172.21.15.53:3300/0 conn(0x7f5e6806e2c0 0x7f5e680cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e66a00640 1 --2- 172.21.15.40:0/569465468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6815ab50 0x7f5e680aefa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5e580097c0 tx=0x7f5e5801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=527ff78bd4ab8db1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e66000640 1 -- 172.21.15.40:0/569465468 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e580b0070 con 0x7f5e6815ab50 2024-03-24T00:34:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e66000640 1 -- 172.21.15.40:0/569465468 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e580ab600 con 0x7f5e6815ab50 2024-03-24T00:34:23.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e66000640 1 -- 172.21.15.40:0/569465468 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e580ab9d0 con 0x7f5e6815ab50 2024-03-24T00:34:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e67e00640 1 -- 172.21.15.40:0/569465468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6815ab50 msgr2=0x7f5e680aefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e67e00640 1 --2- 172.21.15.40:0/569465468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6815ab50 0x7f5e680aefa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5e580097c0 tx=0x7f5e5801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:34:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e67e00640 1 -- 172.21.15.40:0/569465468 shutdown_connections 2024-03-24T00:34:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e67e00640 1 --2- 172.21.15.40:0/569465468 >> v2:172.21.15.53:3300/0 conn(0x7f5e6806e2c0 0x7f5e680cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e67e00640 1 --2- 172.21.15.40:0/569465468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6815ab50 0x7f5e680aefa0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e67e00640 1 -- 172.21.15.40:0/569465468 >> 172.21.15.40:0/569465468 conn(0x7f5e68066e80 msgr2=0x7f5e68067290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e67e00640 1 -- 172.21.15.40:0/569465468 shutdown_connections 2024-03-24T00:34:23.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.767+0000 7f5e67e00640 1 -- 172.21.15.40:0/569465468 wait complete. 2024-03-24T00:34:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e67e00640 1 Processor -- start 2024-03-24T00:34:23.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e67e00640 1 -- start start 2024-03-24T00:34:23.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e67e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e6806e2c0 0x7f5e681454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:23.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e680aea50 0x7f5e6815bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:23.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e67e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6815ab50 0x7f5e680ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:23.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e67e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6815f360 con 0x7f5e680aea50 2024-03-24T00:34:23.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e67e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6815f1e0 con 0x7f5e6806e2c0 2024-03-24T00:34:23.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e67e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6815f4e0 con 0x7f5e6815ab50 2024-03-24T00:34:23.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e680aea50 0x7f5e6815bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:23.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e66a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e6806e2c0 0x7f5e681454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:23.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e66a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e6806e2c0 0x7f5e681454b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:44874/0 (socket says 172.21.15.40:44874) 2024-03-24T00:34:23.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e66a00640 1 -- 172.21.15.40:0/4066160328 learned_addr learned my addr 172.21.15.40:0/4066160328 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:23.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e67400640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6815ab50 0x7f5e680ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:23.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e66a00640 1 -- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6815ab50 msgr2=0x7f5e680ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:23.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e66a00640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6815ab50 0x7f5e680ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:23.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e66a00640 1 -- 172.21.15.40:0/4066160328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e680aea50 msgr2=0x7f5e6815bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:23.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e66a00640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e680aea50 0x7f5e6815bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:23.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e66a00640 1 -- 172.21.15.40:0/4066160328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e680ce4d0 con 0x7f5e6806e2c0 2024-03-24T00:34:23.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e66a00640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e6806e2c0 0x7f5e681454b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5e5801a0d0 tx=0x7f5e5801ace0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:23.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e5ea00640 1 -- 172.21.15.40:0/4066160328 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e580b0050 con 0x7f5e6806e2c0 2024-03-24T00:34:23.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e5ea00640 1 -- 172.21.15.40:0/4066160328 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e580bc070 con 0x7f5e6806e2c0 2024-03-24T00:34:23.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e5ea00640 1 -- 172.21.15.40:0/4066160328 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58006e20 con 0x7f5e6806e2c0 2024-03-24T00:34:23.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.771+0000 7f5e67e00640 1 -- 172.21.15.40:0/4066160328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e680ce7b0 con 0x7f5e6806e2c0 2024-03-24T00:34:23.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.775+0000 7f5e67e00640 1 -- 172.21.15.40:0/4066160328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6815c5c0 con 0x7f5e6806e2c0 2024-03-24T00:34:23.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.775+0000 7f5e67e00640 1 -- 172.21.15.40:0/4066160328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6804f410 con 0x7f5e6806e2c0 2024-03-24T00:34:23.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.775+0000 7f5e5ea00640 1 -- 172.21.15.40:0/4066160328 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5e580b5830 con 0x7f5e6806e2c0 2024-03-24T00:34:23.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.775+0000 7f5e5ea00640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e44067660 0x7f5e44069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:23.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.775+0000 7f5e5ea00640 1 -- 172.21.15.40:0/4066160328 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e5811fde0 con 0x7f5e6806e2c0 2024-03-24T00:34:23.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.775+0000 7f5e5fe00640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e44067660 0x7f5e44069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:23.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.775+0000 7f5e5fe00640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e44067660 0x7f5e44069b20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5e50007fd0 tx=0x7f5e50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:23.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.779+0000 7f5e5ea00640 1 -- 172.21.15.40:0/4066160328 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5e58016090 con 0x7f5e6806e2c0 2024-03-24T00:34:23.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:23.983+0000 7f5e67e00640 1 -- 172.21.15.40:0/4066160328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f5e6806dd50 con 0x7f5e6806e2c0 2024-03-24T00:34:24.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e5ea00640 1 -- 172.21.15.40:0/4066160328 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v53) v1 ==== 105+0+0 (secure 0 0 0) 0x7f5e580f0b70 con 0x7f5e6806e2c0 2024-03-24T00:34:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e67e00640 1 -- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e44067660 msgr2=0x7f5e44069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e67e00640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e44067660 0x7f5e44069b20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5e50007fd0 tx=0x7f5e50009040 comp rx=0 tx=0).stop 2024-03-24T00:34:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e67e00640 1 -- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e6806e2c0 msgr2=0x7f5e681454b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e67e00640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e6806e2c0 0x7f5e681454b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5e5801a0d0 tx=0x7f5e5801ace0 comp rx=0 tx=0).stop 2024-03-24T00:34:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e67e00640 1 -- 172.21.15.40:0/4066160328 shutdown_connections 2024-03-24T00:34:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e67e00640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e44067660 0x7f5e44069b20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e67e00640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6815ab50 0x7f5e680ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e67e00640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e680aea50 0x7f5e6815bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e67e00640 1 --2- 172.21.15.40:0/4066160328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e6806e2c0 0x7f5e681454b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e67e00640 1 -- 172.21.15.40:0/4066160328 >> 172.21.15.40:0/4066160328 conn(0x7f5e68066e80 msgr2=0x7f5e680b0fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e67e00640 1 -- 172.21.15.40:0/4066160328 shutdown_connections 2024-03-24T00:34:24.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.799+0000 7f5e67e00640 1 -- 172.21.15.40:0/4066160328 wait complete. 2024-03-24T00:34:24.832 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:34:24.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccca800640 1 Processor -- start 2024-03-24T00:34:24.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccca800640 1 -- start start 2024-03-24T00:34:24.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccca800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fccc40ceba0 0x7fccc40d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:24.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccca800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fccc415c080 0x7fccc40d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:24.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccca800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc40d0780 con 0x7fccc415bca0 2024-03-24T00:34:24.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccca800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc40b4a90 con 0x7fccc40ceba0 2024-03-24T00:34:24.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccca800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc40b4d70 con 0x7fccc415c080 2024-03-24T00:34:24.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccc9400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fccc415bca0 0x7fccc40ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37710/0 (socket says 172.21.15.40:37710) 2024-03-24T00:34:24.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccc9400640 1 -- 172.21.15.40:0/1434506776 learned_addr learned my addr 172.21.15.40:0/1434506776 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:24.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccc8a00640 1 --2- 172.21.15.40:0/1434506776 >> v2:172.21.15.40:3300/0 conn(0x7fccc40ceba0 0x7fccc40d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:24.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccc9e00640 1 --2- 172.21.15.40:0/1434506776 >> v2:172.21.15.53:3300/0 conn(0x7fccc415c080 0x7fccc40d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:24.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccc3e00640 1 -- 172.21.15.40:0/1434506776 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415806201 0 0) 0x7fccc40d0780 con 0x7fccc415bca0 2024-03-24T00:34:24.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccc3e00640 1 -- 172.21.15.40:0/1434506776 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccac003530 con 0x7fccc415bca0 2024-03-24T00:34:24.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccc8a00640 1 -- 172.21.15.40:0/1434506776 >> v2:172.21.15.53:3300/0 conn(0x7fccc415c080 msgr2=0x7fccc40d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:24.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccc8a00640 1 --2- 172.21.15.40:0/1434506776 >> v2:172.21.15.53:3300/0 conn(0x7fccc415c080 0x7fccc40d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:24.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccc8a00640 1 -- 172.21.15.40:0/1434506776 >> v1:172.21.15.40:6789/0 conn(0x7fccc415bca0 legacy=0x7fccc40ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:24.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.975+0000 7fccc8a00640 1 -- 172.21.15.40:0/1434506776 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc40b5050 con 0x7fccc40ceba0 2024-03-24T00:34:24.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.979+0000 7fccc8a00640 1 --2- 172.21.15.40:0/1434506776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc40ceba0 0x7fccc40d2590 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fccc4074fd0 tx=0x7fccb001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5657581da9a602 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:24.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.979+0000 7fccc3e00640 1 -- 172.21.15.40:0/1434506776 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb00b0070 con 0x7fccc40ceba0 2024-03-24T00:34:24.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.979+0000 7fccc3e00640 1 -- 172.21.15.40:0/1434506776 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb00ab600 con 0x7fccc40ceba0 2024-03-24T00:34:24.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.979+0000 7fccc3e00640 1 -- 172.21.15.40:0/1434506776 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb00ab9d0 con 0x7fccc40ceba0 2024-03-24T00:34:24.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.979+0000 7fccca800640 1 -- 172.21.15.40:0/1434506776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc40ceba0 msgr2=0x7fccc40d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:24.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.979+0000 7fccca800640 1 --2- 172.21.15.40:0/1434506776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc40ceba0 0x7fccc40d2590 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fccc4074fd0 tx=0x7fccb001c480 comp rx=0 tx=0).stop 2024-03-24T00:34:24.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.979+0000 7fccca800640 1 -- 172.21.15.40:0/1434506776 shutdown_connections 2024-03-24T00:34:24.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.979+0000 7fccca800640 1 --2- 172.21.15.40:0/1434506776 >> v2:172.21.15.53:3300/0 conn(0x7fccc415c080 0x7fccc40d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:24.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.979+0000 7fccca800640 1 --2- 172.21.15.40:0/1434506776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc40ceba0 0x7fccc40d2590 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:24.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.979+0000 7fccca800640 1 -- 172.21.15.40:0/1434506776 >> 172.21.15.40:0/1434506776 conn(0x7fccc40d1aa0 msgr2=0x7fccc40d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:24.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.979+0000 7fccca800640 1 -- 172.21.15.40:0/1434506776 shutdown_connections 2024-03-24T00:34:24.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.979+0000 7fccca800640 1 -- 172.21.15.40:0/1434506776 wait complete. 2024-03-24T00:34:24.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccca800640 1 Processor -- start 2024-03-24T00:34:24.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccca800640 1 -- start start 2024-03-24T00:34:24.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccca800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc40ceba0 0x7fccc414c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:24.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccca800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fccc415bca0 0x7fccc40c8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:24.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccca800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccc415c080 0x7fccc40c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:24.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccca800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc40b6110 con 0x7fccc40ceba0 2024-03-24T00:34:24.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccca800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc40b5f90 con 0x7fccc415c080 2024-03-24T00:34:24.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccca800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc40b6290 con 0x7fccc415bca0 2024-03-24T00:34:24.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc40ceba0 0x7fccc414c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:24.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc40ceba0 0x7fccc414c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47084/0 (socket says 172.21.15.40:47084) 2024-03-24T00:34:24.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc9400640 1 -- 172.21.15.40:0/341938917 learned_addr learned my addr 172.21.15.40:0/341938917 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:24.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc9e00640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccc415c080 0x7fccc40c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:24.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc8a00640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fccc415bca0 0x7fccc40c8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:24.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc9400640 1 -- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fccc415bca0 msgr2=0x7fccc40c8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:24.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc9400640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fccc415bca0 0x7fccc40c8b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:24.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc9400640 1 -- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccc415c080 msgr2=0x7fccc40c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:24.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc9400640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccc415c080 0x7fccc40c90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:24.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc9400640 1 -- 172.21.15.40:0/341938917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc414cb80 con 0x7fccc40ceba0 2024-03-24T00:34:24.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc9400640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc40ceba0 0x7fccc414c640 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fccb800c450 tx=0x7fccb800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:24.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc1600640 1 -- 172.21.15.40:0/341938917 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8017070 con 0x7fccc40ceba0 2024-03-24T00:34:24.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc1600640 1 -- 172.21.15.40:0/341938917 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb8005cf0 con 0x7fccc40ceba0 2024-03-24T00:34:24.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccc1600640 1 -- 172.21.15.40:0/341938917 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8007290 con 0x7fccc40ceba0 2024-03-24T00:34:24.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccca800640 1 -- 172.21.15.40:0/341938917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc41578c0 con 0x7fccc40ceba0 2024-03-24T00:34:24.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccca800640 1 -- 172.21.15.40:0/341938917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc4157e80 con 0x7fccc40ceba0 2024-03-24T00:34:24.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.983+0000 7fccca800640 1 -- 172.21.15.40:0/341938917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc409fb10 con 0x7fccc40ceba0 2024-03-24T00:34:24.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.987+0000 7fccc1600640 1 -- 172.21.15.40:0/341938917 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fccb801a070 con 0x7fccc40ceba0 2024-03-24T00:34:24.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.987+0000 7fccc1600640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcca4067660 0x7fcca4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:24.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.987+0000 7fccc1600640 1 -- 172.21.15.40:0/341938917 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccb8084bf0 con 0x7fccc40ceba0 2024-03-24T00:34:24.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.987+0000 7fccc8a00640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcca4067660 0x7fcca4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:24.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.987+0000 7fccc8a00640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcca4067660 0x7fcca4069b20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fccb00ad000 tx=0x7fccb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:24.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:24.991+0000 7fccc1600640 1 -- 172.21.15.40:0/341938917 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fccb8055220 con 0x7fccc40ceba0 2024-03-24T00:34:25.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.187+0000 7fccca800640 1 -- 172.21.15.40:0/341938917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccc40d1880 con 0x7fccc40ceba0 2024-03-24T00:34:25.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.187+0000 7fccc1600640 1 -- 172.21.15.40:0/341938917 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+4990 (secure 0 0 0) 0x7fccb80592c0 con 0x7fccc40ceba0 2024-03-24T00:34:25.190 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:25.191 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":54,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6277,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1633817617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1633817617},{"type":"v1","addr":"172.21.15.53:6837","nonce":1633817617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6468,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1278515461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1278515461},{"type":"v1","addr":"172.21.15.40:6834","nonce":1278515461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:34:12.019118+0000","modified":"2024-03-24T00:34:24.801096+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":6474,"mds_1":6471},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6471":{"gid":6471,"name":"c","rank":1,"incarnation":54,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6835/1312292485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1312292485},{"type":"v1","addr":"172.21.15.40:6835","nonce":1312292485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6474":{"gid":6474,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/501079401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":501079401},{"type":"v1","addr":"172.21.15.53:6835","nonce":501079401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6474,"qdb_cluster":[6474]},"id":4}]} 2024-03-24T00:34:25.191 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 54 2024-03-24T00:34:25.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.191+0000 7fccca800640 1 -- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcca4067660 msgr2=0x7fcca4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:25.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.191+0000 7fccca800640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcca4067660 0x7fcca4069b20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fccb00ad000 tx=0x7fccb0008040 comp rx=0 tx=0).stop 2024-03-24T00:34:25.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.191+0000 7fccca800640 1 -- 172.21.15.40:0/341938917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc40ceba0 msgr2=0x7fccc414c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:25.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.191+0000 7fccca800640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc40ceba0 0x7fccc414c640 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fccb800c450 tx=0x7fccb800c910 comp rx=0 tx=0).stop 2024-03-24T00:34:25.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.191+0000 7fccca800640 1 -- 172.21.15.40:0/341938917 shutdown_connections 2024-03-24T00:34:25.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.191+0000 7fccca800640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcca4067660 0x7fcca4069b20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:25.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.191+0000 7fccca800640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccc415c080 0x7fccc40c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:25.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.191+0000 7fccca800640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fccc415bca0 0x7fccc40c8b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:25.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.191+0000 7fccca800640 1 --2- 172.21.15.40:0/341938917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fccc40ceba0 0x7fccc414c640 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:25.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.191+0000 7fccca800640 1 -- 172.21.15.40:0/341938917 >> 172.21.15.40:0/341938917 conn(0x7fccc40d1aa0 msgr2=0x7fccc415a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:25.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.191+0000 7fccca800640 1 -- 172.21.15.40:0/341938917 shutdown_connections 2024-03-24T00:34:25.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:25.191+0000 7fccca800640 1 -- 172.21.15.40:0/341938917 wait complete. 2024-03-24T00:34:25.615 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 54, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:34:12.019118+0000', 'modified': '2024-03-24T00:34:24.801096+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 6474, 'mds_1': 6471}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6471': {'gid': 6471, 'name': 'c', 'rank': 1, 'incarnation': 54, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6835/1312292485', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 1312292485}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 1312292485}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_6474': {'gid': 6474, 'name': 'd', 'rank': 0, 'incarnation': 48, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.53:6835/501079401', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 501079401}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 501079401}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6474, 'qdb_cluster': [6474]} 2024-03-24T00:34:25.615 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6471:up:creating 2024-03-24T00:34:26.616 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:34:26.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1b5400640 1 Processor -- start 2024-03-24T00:34:26.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1b5400640 1 -- start start 2024-03-24T00:34:26.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1b5400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1b013ec80 0x7fc1b01414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1b5400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc1b01419f0 0x7fc1b00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1b5400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b0070e00 con 0x7fc1b00ce630 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1b5400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b0141e50 con 0x7fc1b013ec80 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1b5400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b00cee20 con 0x7fc1b01419f0 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1ae800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1b013ec80 0x7fc1b01414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1ae800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc1b013ec80 0x7fc1b01414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47100/0 (socket says 172.21.15.40:47100) 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1ae800640 1 -- 172.21.15.40:0/184326072 learned_addr learned my addr 172.21.15.40:0/184326072 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1ade00640 1 --2- 172.21.15.40:0/184326072 >> v2:172.21.15.53:3300/0 conn(0x7fc1b01419f0 0x7fc1b00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1ae800640 1 -- 172.21.15.40:0/184326072 >> v2:172.21.15.53:3300/0 conn(0x7fc1b01419f0 msgr2=0x7fc1b00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1ae800640 1 --2- 172.21.15.40:0/184326072 >> v2:172.21.15.53:3300/0 conn(0x7fc1b01419f0 0x7fc1b00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1ae800640 1 -- 172.21.15.40:0/184326072 >> v1:172.21.15.53:6789/0 conn(0x7fc1b00ce630 legacy=0x7fc1b015f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1ae800640 1 -- 172.21.15.40:0/184326072 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00cf100 con 0x7fc1b013ec80 2024-03-24T00:34:26.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.755+0000 7fc1ade00640 1 --2- 172.21.15.40:0/184326072 >> v2:172.21.15.53:3300/0 conn(0x7fc1b01419f0 0x7fc1b00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:34:26.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.759+0000 7fc1ae800640 1 --2- 172.21.15.40:0/184326072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b013ec80 0x7fc1b01414b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc1b004fa60 tx=0x7fc1a001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e1565d0055efa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.759+0000 7fc1ad400640 1 -- 172.21.15.40:0/184326072 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a00a9400 con 0x7fc1b013ec80 2024-03-24T00:34:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.759+0000 7fc1ad400640 1 -- 172.21.15.40:0/184326072 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a00a95a0 con 0x7fc1b013ec80 2024-03-24T00:34:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.759+0000 7fc1ad400640 1 -- 172.21.15.40:0/184326072 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a00a9970 con 0x7fc1b013ec80 2024-03-24T00:34:26.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.759+0000 7fc1b5400640 1 -- 172.21.15.40:0/184326072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b013ec80 msgr2=0x7fc1b01414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.759+0000 7fc1b5400640 1 --2- 172.21.15.40:0/184326072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b013ec80 0x7fc1b01414b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fc1b004fa60 tx=0x7fc1a001c520 comp rx=0 tx=0).stop 2024-03-24T00:34:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.759+0000 7fc1b5400640 1 -- 172.21.15.40:0/184326072 shutdown_connections 2024-03-24T00:34:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.759+0000 7fc1b5400640 1 --2- 172.21.15.40:0/184326072 >> v2:172.21.15.53:3300/0 conn(0x7fc1b01419f0 0x7fc1b00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.759+0000 7fc1b5400640 1 --2- 172.21.15.40:0/184326072 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b013ec80 0x7fc1b01414b0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.759+0000 7fc1b5400640 1 -- 172.21.15.40:0/184326072 >> 172.21.15.40:0/184326072 conn(0x7fc1b00b17b0 msgr2=0x7fc1b00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.759+0000 7fc1b5400640 1 -- 172.21.15.40:0/184326072 shutdown_connections 2024-03-24T00:34:26.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.759+0000 7fc1b5400640 1 -- 172.21.15.40:0/184326072 wait complete. 2024-03-24T00:34:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1b5400640 1 Processor -- start 2024-03-24T00:34:26.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1b5400640 1 -- start start 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1b5400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc1b00ce630 0x7fc1b0153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1b5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1b013ec80 0x7fc1b013dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1b5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b01419f0 0x7fc1b0157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1b5400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b0168ec0 con 0x7fc1b01419f0 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1b5400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b0168d40 con 0x7fc1b013ec80 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1b5400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b0169040 con 0x7fc1b00ce630 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1af200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b01419f0 0x7fc1b0157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1af200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b01419f0 0x7fc1b0157040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47104/0 (socket says 172.21.15.40:47104) 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1af200640 1 -- 172.21.15.40:0/2624347414 learned_addr learned my addr 172.21.15.40:0/2624347414 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1ade00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1b013ec80 0x7fc1b013dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1ae800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc1b00ce630 0x7fc1b0153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1af200640 1 -- 172.21.15.40:0/2624347414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc1b00ce630 msgr2=0x7fc1b0153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1af200640 1 --2- 172.21.15.40:0/2624347414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc1b00ce630 0x7fc1b0153040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:26.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1af200640 1 -- 172.21.15.40:0/2624347414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1b013ec80 msgr2=0x7fc1b013dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1af200640 1 --2- 172.21.15.40:0/2624347414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1b013ec80 0x7fc1b013dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:26.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1af200640 1 -- 172.21.15.40:0/2624347414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b01549a0 con 0x7fc1b01419f0 2024-03-24T00:34:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1af200640 1 --2- 172.21.15.40:0/2624347414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b01419f0 0x7fc1b0157040 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800c770 tx=0x7fc1a800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1a6a00640 1 -- 172.21.15.40:0/2624347414 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8003b80 con 0x7fc1b01419f0 2024-03-24T00:34:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1a6a00640 1 -- 172.21.15.40:0/2624347414 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a8003d20 con 0x7fc1b01419f0 2024-03-24T00:34:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1a6a00640 1 -- 172.21.15.40:0/2624347414 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a800aa90 con 0x7fc1b01419f0 2024-03-24T00:34:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1b5400640 1 -- 172.21.15.40:0/2624347414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1b0153900 con 0x7fc1b01419f0 2024-03-24T00:34:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1b5400640 1 -- 172.21.15.40:0/2624347414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b0153ec0 con 0x7fc1b01419f0 2024-03-24T00:34:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.763+0000 7fc1b5400640 1 -- 172.21.15.40:0/2624347414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b0068170 con 0x7fc1b01419f0 2024-03-24T00:34:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.767+0000 7fc1a6a00640 1 -- 172.21.15.40:0/2624347414 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc1a800ac30 con 0x7fc1b01419f0 2024-03-24T00:34:26.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.767+0000 7fc1a6a00640 1 --2- 172.21.15.40:0/2624347414 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc190067660 0x7fc190069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:26.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.767+0000 7fc1a6a00640 1 -- 172.21.15.40:0/2624347414 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1a8085710 con 0x7fc1b01419f0 2024-03-24T00:34:26.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.767+0000 7fc1ae800640 1 --2- 172.21.15.40:0/2624347414 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc190067660 0x7fc190069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:26.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.767+0000 7fc1ae800640 1 --2- 172.21.15.40:0/2624347414 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc190067660 0x7fc190069b20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc1a00ae000 tx=0x7fc1a00b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:26.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.775+0000 7fc1a6a00640 1 -- 172.21.15.40:0/2624347414 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc1a8055ca0 con 0x7fc1b01419f0 2024-03-24T00:34:26.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.975+0000 7fc1b5400640 1 -- 172.21.15.40:0/2624347414 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1b01563c0 con 0x7fc1b01419f0 2024-03-24T00:34:26.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.975+0000 7fc1a6a00640 1 -- 172.21.15.40:0/2624347414 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+5012 (secure 0 0 0) 0x7fc1a8059d40 con 0x7fc1b01419f0 2024-03-24T00:34:26.979 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:26.979 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":57,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6277,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1633817617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1633817617},{"type":"v1","addr":"172.21.15.53:6837","nonce":1633817617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:34:12.019118+0000","modified":"2024-03-24T00:34:26.809653+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6474,"mds_1":6471,"mds_2":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":2,"incarnation":56,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6834/1278515461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1278515461},{"type":"v1","addr":"172.21.15.40:6834","nonce":1278515461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6471":{"gid":6471,"name":"c","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6835/1312292485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1312292485},{"type":"v1","addr":"172.21.15.40:6835","nonce":1312292485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6474":{"gid":6474,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/501079401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":501079401},{"type":"v1","addr":"172.21.15.53:6835","nonce":501079401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6474,"qdb_cluster":[6468,6471,6474]},"id":4}]} 2024-03-24T00:34:26.980 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 57 2024-03-24T00:34:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.979+0000 7fc1b5400640 1 -- 172.21.15.40:0/2624347414 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc190067660 msgr2=0x7fc190069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.979+0000 7fc1b5400640 1 --2- 172.21.15.40:0/2624347414 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc190067660 0x7fc190069b20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc1a00ae000 tx=0x7fc1a00b3040 comp rx=0 tx=0).stop 2024-03-24T00:34:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.979+0000 7fc1b5400640 1 -- 172.21.15.40:0/2624347414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b01419f0 msgr2=0x7fc1b0157040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.979+0000 7fc1b5400640 1 --2- 172.21.15.40:0/2624347414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b01419f0 0x7fc1b0157040 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800c770 tx=0x7fc1a800cc30 comp rx=0 tx=0).stop 2024-03-24T00:34:26.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.979+0000 7fc1b5400640 1 -- 172.21.15.40:0/2624347414 shutdown_connections 2024-03-24T00:34:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.979+0000 7fc1b5400640 1 --2- 172.21.15.40:0/2624347414 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc190067660 0x7fc190069b20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.979+0000 7fc1b5400640 1 --2- 172.21.15.40:0/2624347414 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc1b01419f0 0x7fc1b0157040 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:26.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.979+0000 7fc1b5400640 1 --2- 172.21.15.40:0/2624347414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1b013ec80 0x7fc1b013dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.979+0000 7fc1b5400640 1 --2- 172.21.15.40:0/2624347414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc1b00ce630 0x7fc1b0153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.979+0000 7fc1b5400640 1 -- 172.21.15.40:0/2624347414 >> 172.21.15.40:0/2624347414 conn(0x7fc1b00b17b0 msgr2=0x7fc1b01409b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:26.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.979+0000 7fc1b5400640 1 -- 172.21.15.40:0/2624347414 shutdown_connections 2024-03-24T00:34:26.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:26.979+0000 7fc1b5400640 1 -- 172.21.15.40:0/2624347414 wait complete. 2024-03-24T00:34:27.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:34:12.019118+0000', 'modified': '2024-03-24T00:34:26.809653+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 6474, 'mds_1': 6471, 'mds_2': 6468}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6468': {'gid': 6468, 'name': 'a', 'rank': 2, 'incarnation': 56, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.40:6834/1278515461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 1278515461}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 1278515461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_6471': {'gid': 6471, 'name': 'c', 'rank': 1, 'incarnation': 54, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.40:6835/1312292485', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 1312292485}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 1312292485}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_6474': {'gid': 6474, 'name': 'd', 'rank': 0, 'incarnation': 48, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.53:6835/501079401', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 501079401}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 501079401}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6474, 'qdb_cluster': [6468, 6471, 6474]} 2024-03-24T00:34:27.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T00:34:27.403 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01fe00640 1 Processor -- start 2024-03-24T00:34:27.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01fe00640 1 -- start start 2024-03-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa01813ec80 0x7fa0181414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa0180ce800 0x7fa018071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa018070e00 con 0x7fa0181419f0 2024-03-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01fe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa018141e50 con 0x7fa0180ce800 2024-03-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0180cee20 con 0x7fa01813ec80 2024-03-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01ca00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa0181419f0 0x7fa0180ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:59460/0 (socket says 172.21.15.40:59460) 2024-03-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01d400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa01813ec80 0x7fa0181414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01ca00640 1 -- 172.21.15.40:0/1930671623 learned_addr learned my addr 172.21.15.40:0/1930671623 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:27.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01de00640 1 --2- 172.21.15.40:0/1930671623 >> v2:172.21.15.53:3300/0 conn(0x7fa0180ce800 0x7fa018071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01d400640 1 -- 172.21.15.40:0/1930671623 >> v2:172.21.15.53:3300/0 conn(0x7fa0180ce800 msgr2=0x7fa018071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa017e00640 1 -- 172.21.15.40:0/1930671623 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472472747 0 0) 0x7fa018070e00 con 0x7fa0181419f0 2024-03-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01d400640 1 --2- 172.21.15.40:0/1930671623 >> v2:172.21.15.53:3300/0 conn(0x7fa0180ce800 0x7fa018071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01d400640 1 -- 172.21.15.40:0/1930671623 >> v1:172.21.15.53:6789/0 conn(0x7fa0181419f0 legacy=0x7fa0180ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.543+0000 7fa01d400640 1 -- 172.21.15.40:0/1930671623 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0180cf100 con 0x7fa01813ec80 2024-03-24T00:34:27.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.547+0000 7fa01d400640 1 --2- 172.21.15.40:0/1930671623 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa01813ec80 0x7fa0181414b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0097c0 tx=0x7fa00c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=125bfdb2e16a433a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.547+0000 7fa017e00640 1 -- 172.21.15.40:0/1930671623 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c0aa620 con 0x7fa01813ec80 2024-03-24T00:34:27.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.547+0000 7fa017e00640 1 -- 172.21.15.40:0/1930671623 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c0af070 con 0x7fa01813ec80 2024-03-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.547+0000 7fa017e00640 1 -- 172.21.15.40:0/1930671623 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c0aa9e0 con 0x7fa01813ec80 2024-03-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.547+0000 7fa01fe00640 1 -- 172.21.15.40:0/1930671623 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa01813ec80 msgr2=0x7fa0181414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.547+0000 7fa01fe00640 1 --2- 172.21.15.40:0/1930671623 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa01813ec80 0x7fa0181414b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa00c0097c0 tx=0x7fa00c01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.547+0000 7fa01fe00640 1 -- 172.21.15.40:0/1930671623 shutdown_connections 2024-03-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.547+0000 7fa01fe00640 1 --2- 172.21.15.40:0/1930671623 >> v2:172.21.15.53:3300/0 conn(0x7fa0180ce800 0x7fa018071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.547+0000 7fa01fe00640 1 --2- 172.21.15.40:0/1930671623 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa01813ec80 0x7fa0181414b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.547+0000 7fa01fe00640 1 -- 172.21.15.40:0/1930671623 >> 172.21.15.40:0/1930671623 conn(0x7fa0180b17b0 msgr2=0x7fa0180b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.547+0000 7fa01fe00640 1 -- 172.21.15.40:0/1930671623 shutdown_connections 2024-03-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.547+0000 7fa01fe00640 1 -- 172.21.15.40:0/1930671623 wait complete. 2024-03-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.551+0000 7fa01fe00640 1 Processor -- start 2024-03-24T00:34:27.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.551+0000 7fa01fe00640 1 -- start start 2024-03-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.551+0000 7fa01fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa0180ce800 0x7fa01813e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.551+0000 7fa01fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01813ec80 0x7fa01806e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.551+0000 7fa01fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0181419f0 0x7fa018140350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.551+0000 7fa01fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0180723e0 con 0x7fa01813ec80 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.551+0000 7fa01fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa018072260 con 0x7fa0181419f0 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.551+0000 7fa01fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa018072560 con 0x7fa0180ce800 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01d400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa0180ce800 0x7fa01813e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01813ec80 0x7fa01806e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01d400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa0180ce800 0x7fa01813e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:56780/0 (socket says 172.21.15.40:56780) 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01d400640 1 -- 172.21.15.40:0/2563385749 learned_addr learned my addr 172.21.15.40:0/2563385749 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01de00640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0181419f0 0x7fa018140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01ca00640 1 -- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa0180ce800 msgr2=0x7fa01813e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01ca00640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa0180ce800 0x7fa01813e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01ca00640 1 -- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0181419f0 msgr2=0x7fa018140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01ca00640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0181419f0 0x7fa018140350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01ca00640 1 -- 172.21.15.40:0/2563385749 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0180b2850 con 0x7fa01813ec80 2024-03-24T00:34:27.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01de00640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0181419f0 0x7fa018140350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01ca00640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01813ec80 0x7fa01806e010 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fa00400c460 tx=0x7fa00400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa015600640 1 -- 172.21.15.40:0/2563385749 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004018070 con 0x7fa01813ec80 2024-03-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa015600640 1 -- 172.21.15.40:0/2563385749 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa004007500 con 0x7fa01813ec80 2024-03-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01fe00640 1 -- 172.21.15.40:0/2563385749 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0180b2b30 con 0x7fa01813ec80 2024-03-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa015600640 1 -- 172.21.15.40:0/2563385749 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004004a30 con 0x7fa01813ec80 2024-03-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa01fe00640 1 -- 172.21.15.40:0/2563385749 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0180b30f0 con 0x7fa01813ec80 2024-03-24T00:34:27.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.555+0000 7fa015600640 1 -- 172.21.15.40:0/2563385749 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa00401b070 con 0x7fa01813ec80 2024-03-24T00:34:27.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa015600640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ff8067660 0x7f9ff8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa015600640 1 -- 172.21.15.40:0/2563385749 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa004084b50 con 0x7fa01813ec80 2024-03-24T00:34:27.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01d400640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ff8067660 0x7f9ff8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa023344000 1 Processor -- start 2024-03-24T00:34:27.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa023344000 1 -- start start 2024-03-24T00:34:27.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01d400640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ff8067660 0x7f9ff8069b20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa00c000c10 tx=0x7fa00c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa023344000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e5c6920 0x55af4e5c6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa023344000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 0x55af4e57cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa023344000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e57cff0 0x55af4e57d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa023344000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4e3ca680 con 0x55af4e5c7240 2024-03-24T00:34:27.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa023344000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4e351b00 con 0x55af4e5c6920 2024-03-24T00:34:27.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa023344000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af4e3c6720 con 0x55af4e57cff0 2024-03-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 0x55af4e57cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e5c6920 0x55af4e5c6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 0x55af4e57cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36972/0 (socket says 172.21.15.40:36972) 2024-03-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01ca00640 1 -- 172.21.15.40:0/280425190 learned_addr learned my addr 172.21.15.40:0/280425190 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01de00640 1 --2- 172.21.15.40:0/280425190 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e57cff0 0x55af4e57d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01ca00640 1 -- 172.21.15.40:0/280425190 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e57cff0 msgr2=0x55af4e57d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01ca00640 1 --2- 172.21.15.40:0/280425190 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e57cff0 0x55af4e57d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01ca00640 1 -- 172.21.15.40:0/280425190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e5c6920 msgr2=0x55af4e5c6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01ca00640 1 --2- 172.21.15.40:0/280425190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e5c6920 0x55af4e5c6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01ca00640 1 -- 172.21.15.40:0/280425190 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af4e3a2940 con 0x55af4e5c7240 2024-03-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01d400640 1 --2- 172.21.15.40:0/280425190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e5c6920 0x55af4e5c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:27.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa01ca00640 1 --2- 172.21.15.40:0/280425190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 0x55af4e57cab0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fa004054ea0 tx=0x7fa0040590e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=169479c6013bde0c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa00a000640 1 -- 172.21.15.40:0/280425190 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa004081070 con 0x55af4e5c7240 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa00a000640 1 -- 172.21.15.40:0/280425190 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0040534a0 con 0x55af4e5c7240 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.559+0000 7fa00a000640 1 -- 172.21.15.40:0/280425190 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00405d370 con 0x55af4e5c7240 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 -- 172.21.15.40:0/280425190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 msgr2=0x55af4e57cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 --2- 172.21.15.40:0/280425190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 0x55af4e57cab0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fa004054ea0 tx=0x7fa0040590e0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 -- 172.21.15.40:0/280425190 shutdown_connections 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 --2- 172.21.15.40:0/280425190 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e57cff0 0x55af4e57d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 --2- 172.21.15.40:0/280425190 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 0x55af4e57cab0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 --2- 172.21.15.40:0/280425190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e5c6920 0x55af4e5c6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 -- 172.21.15.40:0/280425190 >> 172.21.15.40:0/280425190 conn(0x55af4e578fa0 msgr2=0x55af4e57b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 -- 172.21.15.40:0/280425190 shutdown_connections 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 -- 172.21.15.40:0/280425190 wait complete. 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 -- start start 2024-03-24T00:34:27.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 Processor -- start 2024-03-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e57cff0 0x55af4e5abd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e5c6920 0x55af4e5ac280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 0x55af4e5ac7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4e3c9990 con 0x55af4e5c7240 2024-03-24T00:34:27.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4e384ae0 con 0x55af4e57cff0 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af4e362750 con 0x55af4e5c6920 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 10 client.0 _ll_register_callbacks cb 0x55af4e567ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa01de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 0x55af4e5ac7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa01d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e57cff0 0x55af4e5abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa01ca00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e5c6920 0x55af4e5ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa01d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e57cff0 0x55af4e5abd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:54712/0 (socket says 172.21.15.40:54712) 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa01d400640 1 -- 172.21.15.40:0/3286386070 learned_addr learned my addr 172.21.15.40:0/3286386070 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa01de00640 1 -- 172.21.15.40:0/3286386070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e5c6920 msgr2=0x55af4e5ac280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa01de00640 1 --2- 172.21.15.40:0/3286386070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e5c6920 0x55af4e5ac280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa01de00640 1 -- 172.21.15.40:0/3286386070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e57cff0 msgr2=0x55af4e5abd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa01de00640 1 --2- 172.21.15.40:0/3286386070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e57cff0 0x55af4e5abd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa01de00640 1 -- 172.21.15.40:0/3286386070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af4e40fd80 con 0x55af4e5c7240 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa01d400640 1 --2- 172.21.15.40:0/3286386070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e57cff0 0x55af4e5abd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa01de00640 1 --2- 172.21.15.40:0/3286386070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 0x55af4e5ac7c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa01000ef60 tx=0x7fa0100147b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a87137009ffb44d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7f9ffea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7f9ffea00640 1 -- 172.21.15.40:0/3286386070 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010003db0 con 0x55af4e5c7240 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7fa023344000 1 -- 172.21.15.40:0/3286386070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0040534a0 con 0x55af4e5c7240 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.563+0000 7f9ffea00640 1 -- 172.21.15.40:0/3286386070 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa010004030 con 0x55af4e5c7240 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7f9ffea00640 1 -- 172.21.15.40:0/3286386070 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa010027050 con 0x55af4e5c7240 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 10 client.6690 fetch_fsmap learned FSMap version 57 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 -- 172.21.15.40:0/3286386070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x55af4e351d10 con 0x55af4e5c7240 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7f9ffea00640 1 -- 172.21.15.40:0/3286386070 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0040534a0 con 0x55af4e5c7240 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7f9ffea00640 1 -- 172.21.15.40:0/3286386070 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 57) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fa010004230 con 0x55af4e5c7240 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 10 client.6690 fetch_fsmap finished waiting for FSMap version 57 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 10 client.6690 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 20 client.6690 populate_metadata read hostname 'smithi040' 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 --2- 172.21.15.40:0/3286386070 >> [v2:172.21.15.40:6832/1278515461,v1:172.21.15.40:6834/1278515461] conn(0x55af4e5ad090 0x55af4e5d8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 4 client.6690 mds_command: new command op to 6468 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 -- 172.21.15.40:0/3286386070 --> [v2:172.21.15.40:6832/1278515461,v1:172.21.15.40:6834/1278515461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55af4e365b60 con 0x55af4e5ad090 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa01d400640 1 --2- 172.21.15.40:0/3286386070 >> [v2:172.21.15.40:6832/1278515461,v1:172.21.15.40:6834/1278515461] conn(0x55af4e5ad090 0x55af4e5d8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa01d400640 1 --2- 172.21.15.40:0/3286386070 >> [v2:172.21.15.40:6832/1278515461,v1:172.21.15.40:6834/1278515461] conn(0x55af4e5ad090 0x55af4e5d8ea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7fc0356448e42b3 server_cookie=517e06370b9d4b16 in_seq=0 out_seq=0 2024-03-24T00:34:27.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7f9ffea00640 10 client.6690 ms_handle_connect on v2:172.21.15.40:6832/1278515461 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7f9ffea00640 1 -- 172.21.15.40:0/3286386070 <== mds.2 v2:172.21.15.40:6832/1278515461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55af4e365b60 con 0x55af4e5ad090 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7f9ffea00640 10 client.6690 handle_command_reply: tid=0 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 client.6690 shutdown 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 -- 172.21.15.40:0/3286386070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 msgr2=0x55af4e5ac7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 --2- 172.21.15.40:0/3286386070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 0x55af4e5ac7c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fa01000ef60 tx=0x7fa0100147b0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 -- 172.21.15.40:0/3286386070 shutdown_connections 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 --2- 172.21.15.40:0/3286386070 >> [v2:172.21.15.40:6832/1278515461,v1:172.21.15.40:6834/1278515461] conn(0x55af4e5ad090 0x55af4e5d8ea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 --2- 172.21.15.40:0/3286386070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e5c7240 0x55af4e5ac7c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 --2- 172.21.15.40:0/3286386070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e5c6920 0x55af4e5ac280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 --2- 172.21.15.40:0/3286386070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e57cff0 0x55af4e5abd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 -- 172.21.15.40:0/3286386070 >> 172.21.15.40:0/3286386070 conn(0x55af4e578fa0 msgr2=0x55af4e57bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7f9ffea00640 0 client.6690 ms_handle_reset on v2:172.21.15.40:6832/1278515461 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 -- 172.21.15.40:0/3286386070 shutdown_connections 2024-03-24T00:34:27.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 1 -- 172.21.15.40:0/3286386070 wait complete. 2024-03-24T00:34:27.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.567+0000 7fa023344000 20 client.6690 trim_cache size 0 max 16384 2024-03-24T00:34:27.578 INFO:tasks.ceph.mds.d.smithi053.stderr:2024-03-24T00:34:27.578+0000 7f1c21a00640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:34:27.578 INFO:tasks.ceph.mds.d.smithi053.stderr:2024-03-24T00:34:27.578+0000 7f1c21a00640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-24T00:34:27.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.583+0000 7fa023344000 1 Processor -- start 2024-03-24T00:34:27.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.583+0000 7fa023344000 1 -- start start 2024-03-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e5abed0 0x55af4e5c5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e578fa0 0x55af4e5c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e57cff0 0x55af4e57b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa018202eb0 con 0x55af4e57cff0 2024-03-24T00:34:27.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01806da10 con 0x55af4e578fa0 2024-03-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0180b4610 con 0x55af4e5abed0 2024-03-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa01d400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e5abed0 0x55af4e5c5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa01de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e57cff0 0x55af4e57b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa01ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e578fa0 0x55af4e5c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa01d400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e5abed0 0x55af4e5c5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:56802/0 (socket says 172.21.15.40:56802) 2024-03-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa01d400640 1 -- 172.21.15.40:0/2126828660 learned_addr learned my addr 172.21.15.40:0/2126828660 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa01de00640 1 -- 172.21.15.40:0/2126828660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e5abed0 msgr2=0x55af4e5c5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa01de00640 1 --2- 172.21.15.40:0/2126828660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e5abed0 0x55af4e5c5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa01de00640 1 -- 172.21.15.40:0/2126828660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e578fa0 msgr2=0x55af4e5c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa01de00640 1 --2- 172.21.15.40:0/2126828660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e578fa0 0x55af4e5c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa01de00640 1 -- 172.21.15.40:0/2126828660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55af4e397540 con 0x55af4e57cff0 2024-03-24T00:34:27.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa01ca00640 1 --2- 172.21.15.40:0/2126828660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e578fa0 0x55af4e5c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:27.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa01de00640 1 --2- 172.21.15.40:0/2126828660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e57cff0 0x55af4e57b390 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa01000ef60 tx=0x7fa010008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d599ce70f31ceb41 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa009600640 1 -- 172.21.15.40:0/2126828660 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01002f070 con 0x55af4e57cff0 2024-03-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa009600640 1 -- 172.21.15.40:0/2126828660 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa010003db0 con 0x55af4e57cff0 2024-03-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa009600640 1 -- 172.21.15.40:0/2126828660 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010004160 con 0x55af4e57cff0 2024-03-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 -- 172.21.15.40:0/2126828660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e57cff0 msgr2=0x55af4e57b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 --2- 172.21.15.40:0/2126828660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e57cff0 0x55af4e57b390 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa01000ef60 tx=0x7fa010008000 comp rx=0 tx=0).stop 2024-03-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 -- 172.21.15.40:0/2126828660 shutdown_connections 2024-03-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 --2- 172.21.15.40:0/2126828660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e57cff0 0x55af4e57b390 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 --2- 172.21.15.40:0/2126828660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e578fa0 0x55af4e5c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 --2- 172.21.15.40:0/2126828660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e5abed0 0x55af4e5c5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 -- 172.21.15.40:0/2126828660 >> 172.21.15.40:0/2126828660 conn(0x55af4e57f750 msgr2=0x55af4e5a95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 -- 172.21.15.40:0/2126828660 shutdown_connections 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 -- 172.21.15.40:0/2126828660 wait complete. 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 -- start start 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.587+0000 7fa023344000 1 Processor -- start 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e578fa0 0x55af4e5d9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e57cff0 0x55af4e5da430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e5abed0 0x55af4e5da970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55af4e372a80 con 0x55af4e578fa0 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa01806d410 con 0x55af4e5abed0 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55af4e487700 con 0x55af4e57cff0 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 10 client.0 _ll_register_callbacks cb 0x7fa01000e410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e578fa0 0x55af4e5d9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e578fa0 0x55af4e5d9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36998/0 (socket says 172.21.15.40:36998) 2024-03-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01ca00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e57cff0 0x55af4e5da430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01d400640 1 -- 172.21.15.40:0/2822773349 learned_addr learned my addr 172.21.15.40:0/2822773349 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01de00640 1 --2- 172.21.15.40:0/2822773349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e5abed0 0x55af4e5da970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01d400640 1 -- 172.21.15.40:0/2822773349 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e57cff0 msgr2=0x55af4e5da430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01d400640 1 --2- 172.21.15.40:0/2822773349 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e57cff0 0x55af4e5da430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01d400640 1 -- 172.21.15.40:0/2822773349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e5abed0 msgr2=0x55af4e5da970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01d400640 1 --2- 172.21.15.40:0/2822773349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e5abed0 0x55af4e5da970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01d400640 1 -- 172.21.15.40:0/2822773349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55af4e35e4b0 con 0x55af4e578fa0 2024-03-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01d400640 1 --2- 172.21.15.40:0/2822773349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e578fa0 0x55af4e5d9ef0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fa00c01c980 tx=0x7fa00c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b19bec0020ec78b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7f9ffea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7f9ffea00640 1 -- 172.21.15.40:0/2822773349 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c003cf0 con 0x55af4e578fa0 2024-03-24T00:34:27.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7f9ffea00640 1 -- 172.21.15.40:0/2822773349 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa00c0d7aa0 con 0x55af4e578fa0 2024-03-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 -- 172.21.15.40:0/2822773349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa010003db0 con 0x55af4e578fa0 2024-03-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7f9ffea00640 1 -- 172.21.15.40:0/2822773349 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa00c0d7ca0 con 0x55af4e578fa0 2024-03-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 10 client.6696 fetch_fsmap learned FSMap version 57 2024-03-24T00:34:27.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 -- 172.21.15.40:0/2822773349 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x55af4e3712f0 con 0x55af4e578fa0 2024-03-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7f9ffea00640 1 -- 172.21.15.40:0/2822773349 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa010003db0 con 0x55af4e578fa0 2024-03-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7f9ffea00640 1 -- 172.21.15.40:0/2822773349 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 57) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fa00c0bc040 con 0x55af4e578fa0 2024-03-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 10 client.6696 fetch_fsmap finished waiting for FSMap version 57 2024-03-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 10 client.6696 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 20 client.6696 populate_metadata read hostname 'smithi040' 2024-03-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 --2- 172.21.15.40:0/2822773349 >> [v2:172.21.15.40:6832/1278515461,v1:172.21.15.40:6834/1278515461] conn(0x55af4e5c6920 0x55af4e6341f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 4 client.6696 mds_command: new command op to 6468 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 -- 172.21.15.40:0/2822773349 --> [v2:172.21.15.40:6832/1278515461,v1:172.21.15.40:6834/1278515461] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55af4e3d7bc0 con 0x55af4e5c6920 2024-03-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01ca00640 1 --2- 172.21.15.40:0/2822773349 >> [v2:172.21.15.40:6832/1278515461,v1:172.21.15.40:6834/1278515461] conn(0x55af4e5c6920 0x55af4e6341f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7f9ffea00640 10 client.6696 ms_handle_connect on v2:172.21.15.40:6832/1278515461 2024-03-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa01ca00640 1 --2- 172.21.15.40:0/2822773349 >> [v2:172.21.15.40:6832/1278515461,v1:172.21.15.40:6834/1278515461] conn(0x55af4e5c6920 0x55af4e6341f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=40b76e2b12672269 server_cookie=5054073237fa0372 in_seq=0 out_seq=0 2024-03-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7f9ffea00640 1 -- 172.21.15.40:0/2822773349 <== mds.2 v2:172.21.15.40:6832/1278515461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+318 (crc 0 0 0) 0x55af4e3d7bc0 con 0x55af4e5c6920 2024-03-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7f9ffea00640 10 client.6696 handle_command_reply: tid=0 2024-03-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 client.6696 shutdown 2024-03-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 -- 172.21.15.40:0/2822773349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e578fa0 msgr2=0x55af4e5d9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 --2- 172.21.15.40:0/2822773349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e578fa0 0x55af4e5d9ef0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fa00c01c980 tx=0x7fa00c0046d0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 -- 172.21.15.40:0/2822773349 shutdown_connections 2024-03-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 --2- 172.21.15.40:0/2822773349 >> [v2:172.21.15.40:6832/1278515461,v1:172.21.15.40:6834/1278515461] conn(0x55af4e5c6920 0x55af4e6341f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 --2- 172.21.15.40:0/2822773349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55af4e5abed0 0x55af4e5da970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 --2- 172.21.15.40:0/2822773349 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55af4e57cff0 0x55af4e5da430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 --2- 172.21.15.40:0/2822773349 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55af4e578fa0 0x55af4e5d9ef0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7fa023344000 1 -- 172.21.15.40:0/2822773349 >> 172.21.15.40:0/2822773349 conn(0x55af4e57f750 msgr2=0x55af4e5ab190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.591+0000 7f9ffea00640 0 client.6696 ms_handle_reset on v2:172.21.15.40:6832/1278515461 2024-03-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa023344000 1 -- 172.21.15.40:0/2822773349 shutdown_connections 2024-03-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa023344000 1 -- 172.21.15.40:0/2822773349 wait complete. 2024-03-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa023344000 20 client.6696 trim_cache size 0 max 16384 2024-03-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:34:27.601 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-03-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6468, 2024-03-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 1.786451947, 2024-03-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 57, 2024-03-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 59, 2024-03-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 59, 2024-03-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 26.914558355 2024-03-24T00:34:27.602 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa01fe00640 1 -- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ff8067660 msgr2=0x7f9ff8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa01fe00640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ff8067660 0x7f9ff8069b20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa00c000c10 tx=0x7fa00c0a7000 comp rx=0 tx=0).stop 2024-03-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa01fe00640 1 -- 172.21.15.40:0/2563385749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01813ec80 msgr2=0x7fa01806e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa01fe00640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01813ec80 0x7fa01806e010 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fa00400c460 tx=0x7fa00400c920 comp rx=0 tx=0).stop 2024-03-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa01fe00640 1 -- 172.21.15.40:0/2563385749 shutdown_connections 2024-03-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa01fe00640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ff8067660 0x7f9ff8069b20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa01fe00640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0181419f0 0x7fa018140350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa01fe00640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01813ec80 0x7fa01806e010 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa01fe00640 1 --2- 172.21.15.40:0/2563385749 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa0180ce800 0x7fa01813e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa01fe00640 1 -- 172.21.15.40:0/2563385749 >> 172.21.15.40:0/2563385749 conn(0x7fa0180b17b0 msgr2=0x7fa018159fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa01fe00640 1 -- 172.21.15.40:0/2563385749 shutdown_connections 2024-03-24T00:34:27.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.595+0000 7fa01fe00640 1 -- 172.21.15.40:0/2563385749 wait complete. 2024-03-24T00:34:27.615 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:34:27.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.755+0000 7fb7f2400640 1 Processor -- start 2024-03-24T00:34:27.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.755+0000 7fb7f2400640 1 -- start start 2024-03-24T00:34:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.759+0000 7fb7f2400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7ec13ec80 0x7fb7ec1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.759+0000 7fb7f2400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ec070e00 con 0x7fb7ec1419f0 2024-03-24T00:34:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.759+0000 7fb7f2400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ec0cee20 con 0x7fb7ec0ce800 2024-03-24T00:34:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.759+0000 7fb7f2400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec141e50 con 0x7fb7ec13ec80 2024-03-24T00:34:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.759+0000 7fb7eb400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7ec13ec80 0x7fb7ec1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.759+0000 7fb7ebe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb7ec0ce800 0x7fb7ec071300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:59474/0 (socket says 172.21.15.40:59474) 2024-03-24T00:34:27.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.759+0000 7fb7ebe00640 1 -- 172.21.15.40:0/1198141434 learned_addr learned my addr 172.21.15.40:0/1198141434 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:27.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.759+0000 7fb7eb400640 1 -- 172.21.15.40:0/1198141434 >> v1:172.21.15.53:6789/0 conn(0x7fb7ec0ce800 legacy=0x7fb7ec071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.759+0000 7fb7eb400640 1 -- 172.21.15.40:0/1198141434 >> v1:172.21.15.40:6789/0 conn(0x7fb7ec1419f0 legacy=0x7fb7ec0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.759+0000 7fb7eaa00640 1 -- 172.21.15.40:0/1198141434 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989946288 0 0) 0x7fb7ec0cee20 con 0x7fb7ec0ce800 2024-03-24T00:34:27.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.759+0000 7fb7eb400640 1 -- 172.21.15.40:0/1198141434 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec0cf100 con 0x7fb7ec13ec80 2024-03-24T00:34:27.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7eb400640 1 --2- 172.21.15.40:0/1198141434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7ec13ec80 0x7fb7ec1414b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0097f0 tx=0x7fb7dc01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b80b7b30135df5d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7eaa00640 1 -- 172.21.15.40:0/1198141434 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0aa660 con 0x7fb7ec13ec80 2024-03-24T00:34:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7eaa00640 1 -- 172.21.15.40:0/1198141434 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc0af070 con 0x7fb7ec13ec80 2024-03-24T00:34:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7eaa00640 1 -- 172.21.15.40:0/1198141434 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0aaa20 con 0x7fb7ec13ec80 2024-03-24T00:34:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 -- 172.21.15.40:0/1198141434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7ec13ec80 msgr2=0x7fb7ec1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 --2- 172.21.15.40:0/1198141434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7ec13ec80 0x7fb7ec1414b0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0097f0 tx=0x7fb7dc01c4e0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 -- 172.21.15.40:0/1198141434 shutdown_connections 2024-03-24T00:34:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 --2- 172.21.15.40:0/1198141434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7ec13ec80 0x7fb7ec1414b0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 -- 172.21.15.40:0/1198141434 >> 172.21.15.40:0/1198141434 conn(0x7fb7ec0b17b0 msgr2=0x7fb7ec0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 -- 172.21.15.40:0/1198141434 shutdown_connections 2024-03-24T00:34:27.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 -- 172.21.15.40:0/1198141434 wait complete. 2024-03-24T00:34:27.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 Processor -- start 2024-03-24T00:34:27.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 -- start start 2024-03-24T00:34:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7ec0ce800 0x7fb7ec14db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7ec13ec80 0x7fb7ec158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7ec1419f0 0x7fb7ec14ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0725f0 con 0x7fb7ec0ce800 2024-03-24T00:34:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec072470 con 0x7fb7ec1419f0 2024-03-24T00:34:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.763+0000 7fb7f2400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec169d50 con 0x7fb7ec13ec80 2024-03-24T00:34:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7eb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7ec0ce800 0x7fb7ec14db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7eb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7ec0ce800 0x7fb7ec14db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37004/0 (socket says 172.21.15.40:37004) 2024-03-24T00:34:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7eb400640 1 -- 172.21.15.40:0/1539178675 learned_addr learned my addr 172.21.15.40:0/1539178675 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7ebe00640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7ec1419f0 0x7fb7ec14ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7ec13ec80 0x7fb7ec158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7eb400640 1 -- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7ec13ec80 msgr2=0x7fb7ec158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7eb400640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7ec13ec80 0x7fb7ec158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7eb400640 1 -- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7ec1419f0 msgr2=0x7fb7ec14ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7eb400640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7ec1419f0 0x7fb7ec14ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7eb400640 1 -- 172.21.15.40:0/1539178675 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec0b4410 con 0x7fb7ec0ce800 2024-03-24T00:34:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7ebe00640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7ec1419f0 0x7fb7ec14ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:27.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7eb400640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7ec0ce800 0x7fb7ec14db40 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0ac040 tx=0x7fb7dc01ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7e3400640 1 -- 172.21.15.40:0/1539178675 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0be610 con 0x7fb7ec0ce800 2024-03-24T00:34:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7e3400640 1 -- 172.21.15.40:0/1539178675 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc0be7f0 con 0x7fb7ec0ce800 2024-03-24T00:34:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7e3400640 1 -- 172.21.15.40:0/1539178675 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0bebe0 con 0x7fb7ec0ce800 2024-03-24T00:34:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7f2400640 1 -- 172.21.15.40:0/1539178675 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ec157480 con 0x7fb7ec0ce800 2024-03-24T00:34:27.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.767+0000 7fb7f2400640 1 -- 172.21.15.40:0/1539178675 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ec157a40 con 0x7fb7ec0ce800 2024-03-24T00:34:27.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7e3400640 1 -- 172.21.15.40:0/1539178675 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb7dc0bc020 con 0x7fb7ec0ce800 2024-03-24T00:34:27.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7e3400640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb7c8067660 0x7fb7c8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7e3400640 1 -- 172.21.15.40:0/1539178675 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7dc11ffe0 con 0x7fb7ec0ce800 2024-03-24T00:34:27.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb7c8067660 0x7fb7c8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb7c8067660 0x7fb7c8069b20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec15ecf0 tx=0x7fb7d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7f59a6000 1 Processor -- start 2024-03-24T00:34:27.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7f59a6000 1 -- start start 2024-03-24T00:34:27.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7f59a6000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f7337920 0x55b6f7337d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7f59a6000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f7338240 0x55b6f72edab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7f59a6000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f72edff0 0x55b6f72ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7f59a6000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6f713b680 con 0x55b6f7337920 2024-03-24T00:34:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7f59a6000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6f70c2b00 con 0x55b6f72edff0 2024-03-24T00:34:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.771+0000 7fb7f59a6000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6f7137720 con 0x55b6f7338240 2024-03-24T00:34:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7eb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f7337920 0x55b6f7337d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7eb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f7337920 0x55b6f7337d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37010/0 (socket says 172.21.15.40:37010) 2024-03-24T00:34:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7eb400640 1 -- 172.21.15.40:0/1000165528 learned_addr learned my addr 172.21.15.40:0/1000165528 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/1000165528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f7338240 0x55b6f72edab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7ebe00640 1 --2- 172.21.15.40:0/1000165528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f72edff0 0x55b6f72ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7eb400640 1 -- 172.21.15.40:0/1000165528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f7338240 msgr2=0x55b6f72edab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7eb400640 1 --2- 172.21.15.40:0/1000165528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f7338240 0x55b6f72edab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7eb400640 1 -- 172.21.15.40:0/1000165528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f72edff0 msgr2=0x55b6f72ee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7eb400640 1 --2- 172.21.15.40:0/1000165528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f72edff0 0x55b6f72ee3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7eb400640 1 -- 172.21.15.40:0/1000165528 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6f7113940 con 0x55b6f7337920 2024-03-24T00:34:27.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7eb400640 1 --2- 172.21.15.40:0/1000165528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f7337920 0x55b6f7337d00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0190f0 tx=0x7fb7dc0ecb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee012b1ce56a13a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7dbe00640 1 -- 172.21.15.40:0/1000165528 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0c3070 con 0x55b6f7337920 2024-03-24T00:34:27.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7dbe00640 1 -- 172.21.15.40:0/1000165528 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc0af040 con 0x55b6f7337920 2024-03-24T00:34:27.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7dbe00640 1 -- 172.21.15.40:0/1000165528 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0f4800 con 0x55b6f7337920 2024-03-24T00:34:27.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7f59a6000 1 -- 172.21.15.40:0/1000165528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f7337920 msgr2=0x55b6f7337d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/1000165528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f7337920 0x55b6f7337d00 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0190f0 tx=0x7fb7dc0ecb30 comp rx=0 tx=0).stop 2024-03-24T00:34:27.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7f59a6000 1 -- 172.21.15.40:0/1000165528 shutdown_connections 2024-03-24T00:34:27.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/1000165528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f72edff0 0x55b6f72ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/1000165528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f7338240 0x55b6f72edab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/1000165528 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f7337920 0x55b6f7337d00 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7f59a6000 1 -- 172.21.15.40:0/1000165528 >> 172.21.15.40:0/1000165528 conn(0x55b6f72e9fa0 msgr2=0x55b6f72ec390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7f59a6000 1 -- 172.21.15.40:0/1000165528 shutdown_connections 2024-03-24T00:34:27.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7f59a6000 1 -- 172.21.15.40:0/1000165528 wait complete. 2024-03-24T00:34:27.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7f59a6000 1 -- start start 2024-03-24T00:34:27.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.775+0000 7fb7f59a6000 1 Processor -- start 2024-03-24T00:34:27.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 0x55b6f731cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f7337920 0x55b6f731d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f7338240 0x55b6f731d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6f713a990 con 0x55b6f72edff0 2024-03-24T00:34:27.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6f70f5ae0 con 0x55b6f7337920 2024-03-24T00:34:27.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6f70d3750 con 0x55b6f7338240 2024-03-24T00:34:27.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 10 client.0 _ll_register_callbacks cb 0x55b6f72d8ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:34:27.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7eb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 0x55b6f731cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7eb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 0x55b6f731cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37026/0 (socket says 172.21.15.40:37026) 2024-03-24T00:34:27.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7eb400640 1 -- 172.21.15.40:0/498984218 learned_addr learned my addr 172.21.15.40:0/498984218 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:27.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f7337920 0x55b6f731d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7ebe00640 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f7338240 0x55b6f731d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7eb400640 1 -- 172.21.15.40:0/498984218 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f7338240 msgr2=0x55b6f731d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7eb400640 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f7338240 0x55b6f731d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7eb400640 1 -- 172.21.15.40:0/498984218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f7337920 msgr2=0x55b6f731d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7eb400640 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f7337920 0x55b6f731d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7eb400640 1 -- 172.21.15.40:0/498984218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b6f7180d80 con 0x55b6f72edff0 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7eb400640 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 0x55b6f731cd40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0f0460 tx=0x7fb7dc11e410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac91143a9f4c869c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7d8c00640 1 -- 172.21.15.40:0/498984218 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0f8b30 con 0x55b6f72edff0 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7d8c00640 1 -- 172.21.15.40:0/498984218 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7dc0efbb0 con 0x55b6f72edff0 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 1 -- 172.21.15.40:0/498984218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7dc0af040 con 0x55b6f72edff0 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7d8c00640 1 -- 172.21.15.40:0/498984218 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7dc0f4800 con 0x55b6f72edff0 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 10 client.6708 fetch_fsmap learned FSMap version 57 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 1 -- 172.21.15.40:0/498984218 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x55b6f70c2d10 con 0x55b6f72edff0 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7d8c00640 1 -- 172.21.15.40:0/498984218 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc0af040 con 0x55b6f72edff0 2024-03-24T00:34:27.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7d8c00640 1 -- 172.21.15.40:0/498984218 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 57) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fb7dc0ec4b0 con 0x55b6f72edff0 2024-03-24T00:34:27.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 10 client.6708 fetch_fsmap finished waiting for FSMap version 57 2024-03-24T00:34:27.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 10 client.6708 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:34:27.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 20 client.6708 populate_metadata read hostname 'smithi040' 2024-03-24T00:34:27.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.40:6833/1312292485,v1:172.21.15.40:6835/1312292485] conn(0x55b6f731e090 0x55b6f7349ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 4 client.6708 mds_command: new command op to 6471 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:34:27.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7f59a6000 1 -- 172.21.15.40:0/498984218 --> [v2:172.21.15.40:6833/1312292485,v1:172.21.15.40:6835/1312292485] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b6f70d6b60 con 0x55b6f731e090 2024-03-24T00:34:27.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.40:6833/1312292485,v1:172.21.15.40:6835/1312292485] conn(0x55b6f731e090 0x55b6f7349ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.40:6833/1312292485,v1:172.21.15.40:6835/1312292485] conn(0x55b6f731e090 0x55b6f7349ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9259bd58b9d7a864 server_cookie=b821d15a87ea9db9 in_seq=0 out_seq=0 2024-03-24T00:34:27.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.779+0000 7fb7d8c00640 10 client.6708 ms_handle_connect on v2:172.21.15.40:6833/1312292485 2024-03-24T00:34:27.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7d8c00640 1 -- 172.21.15.40:0/498984218 <== mds.1 v2:172.21.15.40:6833/1312292485 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b6f70d6b60 con 0x55b6f731e090 2024-03-24T00:34:27.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7d8c00640 10 client.6708 handle_command_reply: tid=0 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7f59a6000 1 client.6708 shutdown 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7f59a6000 1 -- 172.21.15.40:0/498984218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 msgr2=0x55b6f731cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 0x55b6f731cd40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0f0460 tx=0x7fb7dc11e410 comp rx=0 tx=0).stop 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7f59a6000 1 -- 172.21.15.40:0/498984218 shutdown_connections 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.40:6833/1312292485,v1:172.21.15.40:6835/1312292485] conn(0x55b6f731e090 0x55b6f7349ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f7338240 0x55b6f731d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f7337920 0x55b6f731d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/498984218 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 0x55b6f731cd40 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7f59a6000 1 -- 172.21.15.40:0/498984218 >> 172.21.15.40:0/498984218 conn(0x55b6f72e9fa0 msgr2=0x55b6f72ecc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7d8c00640 0 client.6708 ms_handle_reset on v2:172.21.15.40:6833/1312292485 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7f59a6000 1 -- 172.21.15.40:0/498984218 shutdown_connections 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7f59a6000 1 -- 172.21.15.40:0/498984218 wait complete. 2024-03-24T00:34:27.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.783+0000 7fb7f59a6000 20 client.6708 trim_cache size 0 max 16384 2024-03-24T00:34:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7f59a6000 1 Processor -- start 2024-03-24T00:34:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7f59a6000 1 -- start start 2024-03-24T00:34:27.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7f59a6000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f731ced0 0x55b6f7336150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7f59a6000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f72e9fa0 0x55b6f7336690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7f59a6000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f72edff0 0x55b6f72ec390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7f59a6000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec202270 con 0x55b6f731ced0 2024-03-24T00:34:27.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7f59a6000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec202870 con 0x55b6f72e9fa0 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7f59a6000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0b4150 con 0x55b6f72edff0 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7eb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f731ced0 0x55b6f7336150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7e3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f72e9fa0 0x55b6f7336690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7e3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f72e9fa0 0x55b6f7336690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:54778/0 (socket says 172.21.15.40:54778) 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7e3e00640 1 -- 172.21.15.40:0/501764257 learned_addr learned my addr 172.21.15.40:0/501764257 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7ebe00640 1 --2- 172.21.15.40:0/501764257 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f72edff0 0x55b6f72ec390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7e3e00640 1 -- 172.21.15.40:0/501764257 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f72edff0 msgr2=0x55b6f72ec390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/501764257 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f72edff0 0x55b6f72ec390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7e3e00640 1 -- 172.21.15.40:0/501764257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f731ced0 msgr2=0x55b6f7336150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/501764257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f731ced0 0x55b6f7336150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.799+0000 7fb7e3e00640 1 -- 172.21.15.40:0/501764257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b6f72f21c0 con 0x55b6f72e9fa0 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/501764257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f72e9fa0 0x55b6f7336690 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb7d40013a0 tx=0x7fb7d400a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f932d079097955c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7db400640 1 -- 172.21.15.40:0/501764257 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d400fea0 con 0x55b6f72e9fa0 2024-03-24T00:34:27.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7db400640 1 -- 172.21.15.40:0/501764257 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d401b040 con 0x55b6f72e9fa0 2024-03-24T00:34:27.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7db400640 1 -- 172.21.15.40:0/501764257 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d402a2d0 con 0x55b6f72e9fa0 2024-03-24T00:34:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 -- 172.21.15.40:0/501764257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f72e9fa0 msgr2=0x55b6f7336690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/501764257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f72e9fa0 0x55b6f7336690 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb7d40013a0 tx=0x7fb7d400a000 comp rx=0 tx=0).stop 2024-03-24T00:34:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 -- 172.21.15.40:0/501764257 shutdown_connections 2024-03-24T00:34:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/501764257 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f72edff0 0x55b6f72ec390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/501764257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f72e9fa0 0x55b6f7336690 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/501764257 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f731ced0 0x55b6f7336150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 -- 172.21.15.40:0/501764257 >> 172.21.15.40:0/501764257 conn(0x55b6f72f0750 msgr2=0x55b6f731b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 -- 172.21.15.40:0/501764257 shutdown_connections 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 -- 172.21.15.40:0/501764257 wait complete. 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 -- start start 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 Processor -- start 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f72e9fa0 0x55b6f734aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 0x55b6f734b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f731ced0 0x55b6f734b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6f71f8700 con 0x55b6f72edff0 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b6f70e3a80 con 0x55b6f731ced0 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6f70f5ae0 con 0x55b6f72e9fa0 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7f59a6000 10 client.0 _ll_register_callbacks cb 0x7fb7e4004c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 0x55b6f734b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 0x55b6f734b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37036/0 (socket says 172.21.15.40:37036) 2024-03-24T00:34:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7e3e00640 1 -- 172.21.15.40:0/2788313922 learned_addr learned my addr 172.21.15.40:0/2788313922 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7eb400640 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f72e9fa0 0x55b6f734aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7ebe00640 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f731ced0 0x55b6f734b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7e3e00640 1 -- 172.21.15.40:0/2788313922 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f72e9fa0 msgr2=0x55b6f734aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f72e9fa0 0x55b6f734aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7e3e00640 1 -- 172.21.15.40:0/2788313922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f731ced0 msgr2=0x55b6f734b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f731ced0 0x55b6f734b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7e3e00640 1 -- 172.21.15.40:0/2788313922 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b6f7108540 con 0x55b6f72edff0 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7e3e00640 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 0x55b6f734b430 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x55b6f7157970 tx=0x7fb7d4001130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24c922cf8f2b036 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.803+0000 7fb7d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7d8c00640 1 -- 172.21.15.40:0/2788313922 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4023720 con 0x55b6f72edff0 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7d8c00640 1 -- 172.21.15.40:0/2788313922 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb7d40238c0 con 0x55b6f72edff0 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 -- 172.21.15.40:0/2788313922 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7d401b040 con 0x55b6f72edff0 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7d8c00640 1 -- 172.21.15.40:0/2788313922 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7d4023ae0 con 0x55b6f72edff0 2024-03-24T00:34:27.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 10 client.6714 fetch_fsmap learned FSMap version 57 2024-03-24T00:34:27.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 -- 172.21.15.40:0/2788313922 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x55b6f70cf4b0 con 0x55b6f72edff0 2024-03-24T00:34:27.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7d8c00640 1 -- 172.21.15.40:0/2788313922 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d401b040 con 0x55b6f72edff0 2024-03-24T00:34:27.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7d8c00640 1 -- 172.21.15.40:0/2788313922 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 57) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fb7d4040020 con 0x55b6f72edff0 2024-03-24T00:34:27.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 10 client.6714 fetch_fsmap finished waiting for FSMap version 57 2024-03-24T00:34:27.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 10 client.6714 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:34:27.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 20 client.6714 populate_metadata read hostname 'smithi040' 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.40:6833/1312292485,v1:172.21.15.40:6835/1312292485] conn(0x55b6f7337920 0x55b6f734c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 4 client.6714 mds_command: new command op to 6471 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 -- 172.21.15.40:0/2788313922 --> [v2:172.21.15.40:6833/1312292485,v1:172.21.15.40:6835/1312292485] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b6f7148bc0 con 0x55b6f7337920 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7eb400640 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.40:6833/1312292485,v1:172.21.15.40:6835/1312292485] conn(0x55b6f7337920 0x55b6f734c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7d8c00640 10 client.6714 ms_handle_connect on v2:172.21.15.40:6833/1312292485 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7eb400640 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.40:6833/1312292485,v1:172.21.15.40:6835/1312292485] conn(0x55b6f7337920 0x55b6f734c300 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a0abf2816ecece49 server_cookie=33050d1fd52de2e2 in_seq=0 out_seq=0 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7d8c00640 1 -- 172.21.15.40:0/2788313922 <== mds.1 v2:172.21.15.40:6833/1312292485 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+323 (crc 0 0 0) 0x55b6f7148bc0 con 0x55b6f7337920 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7d8c00640 10 client.6714 handle_command_reply: tid=0 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 client.6714 shutdown 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 -- 172.21.15.40:0/2788313922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 msgr2=0x55b6f734b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 0x55b6f734b430 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x55b6f7157970 tx=0x7fb7d4001130 comp rx=0 tx=0).stop 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 -- 172.21.15.40:0/2788313922 shutdown_connections 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6f731ced0 0x55b6f734b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6f72edff0 0x55b6f734b430 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.40:6833/1312292485,v1:172.21.15.40:6835/1312292485] conn(0x55b6f7337920 0x55b6f734c300 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 --2- 172.21.15.40:0/2788313922 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b6f72e9fa0 0x55b6f734aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 -- 172.21.15.40:0/2788313922 >> 172.21.15.40:0/2788313922 conn(0x55b6f72f0750 msgr2=0x55b6f731c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7d8c00640 0 client.6714 ms_handle_reset on v2:172.21.15.40:6833/1312292485 2024-03-24T00:34:27.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 -- 172.21.15.40:0/2788313922 shutdown_connections 2024-03-24T00:34:27.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 1 -- 172.21.15.40:0/2788313922 wait complete. 2024-03-24T00:34:27.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f59a6000 20 client.6714 trim_cache size 0 max 16384 2024-03-24T00:34:27.814 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:34:27.814 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:34:27.814 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-03-24T00:34:27.814 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6471, 2024-03-24T00:34:27.815 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:34:27.815 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:34:27.815 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:34:27.815 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.00757304, 2024-03-24T00:34:27.815 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 57, 2024-03-24T00:34:27.815 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 59, 2024-03-24T00:34:27.815 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 59, 2024-03-24T00:34:27.815 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 27.128782438000002 2024-03-24T00:34:27.815 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:34:27.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f2400640 1 -- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb7c8067660 msgr2=0x7fb7c8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f2400640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb7c8067660 0x7fb7c8069b20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec15ecf0 tx=0x7fb7d4009040 comp rx=0 tx=0).stop 2024-03-24T00:34:27.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f2400640 1 -- 172.21.15.40:0/1539178675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7ec0ce800 msgr2=0x7fb7ec14db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f2400640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7ec0ce800 0x7fb7ec14db40 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc0ac040 tx=0x7fb7dc01ad20 comp rx=0 tx=0).stop 2024-03-24T00:34:27.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f2400640 1 -- 172.21.15.40:0/1539178675 shutdown_connections 2024-03-24T00:34:27.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f2400640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb7c8067660 0x7fb7c8069b20 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f2400640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7ec1419f0 0x7fb7ec14ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f2400640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb7ec13ec80 0x7fb7ec158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f2400640 1 --2- 172.21.15.40:0/1539178675 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb7ec0ce800 0x7fb7ec14db40 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.807+0000 7fb7f2400640 1 -- 172.21.15.40:0/1539178675 >> 172.21.15.40:0/1539178675 conn(0x7fb7ec0b17b0 msgr2=0x7fb7ec071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.811+0000 7fb7f2400640 1 -- 172.21.15.40:0/1539178675 shutdown_connections 2024-03-24T00:34:27.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.811+0000 7fb7f2400640 1 -- 172.21.15.40:0/1539178675 wait complete. 2024-03-24T00:34:27.839 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T00:34:27.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a28e00640 1 Processor -- start 2024-03-24T00:34:27.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a28e00640 1 -- start start 2024-03-24T00:34:27.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a28e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415e0b0 0x7f2a240b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a28e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a240b4b80 0x7f2a240cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a28e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a240cd3d0 con 0x7f2a2406b600 2024-03-24T00:34:27.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a28e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2415e490 con 0x7f2a240b4b80 2024-03-24T00:34:27.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a28e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a240cdfe0 con 0x7f2a2415e0b0 2024-03-24T00:34:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a19400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a240b4b80 0x7f2a240cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a21e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415e0b0 0x7f2a240b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a22800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2a2406b600 0x7f2a240b6ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:59488/0 (socket says 172.21.15.40:59488) 2024-03-24T00:34:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a22800640 1 -- 172.21.15.40:0/577568586 learned_addr learned my addr 172.21.15.40:0/577568586 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a21e00640 1 -- 172.21.15.40:0/577568586 >> v2:172.21.15.40:3300/0 conn(0x7f2a240b4b80 msgr2=0x7f2a240cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a21e00640 1 --2- 172.21.15.40:0/577568586 >> v2:172.21.15.40:3300/0 conn(0x7f2a240b4b80 0x7f2a240cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a21e00640 1 -- 172.21.15.40:0/577568586 >> v1:172.21.15.53:6789/0 conn(0x7f2a2406b600 legacy=0x7f2a240b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a21e00640 1 -- 172.21.15.40:0/577568586 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a240cd6b0 con 0x7f2a2415e0b0 2024-03-24T00:34:27.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.979+0000 7f2a21400640 1 -- 172.21.15.40:0/577568586 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392962500 0 0) 0x7f2a240cd3d0 con 0x7f2a2406b600 2024-03-24T00:34:27.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a19400640 1 --2- 172.21.15.40:0/577568586 >> v2:172.21.15.40:3300/0 conn(0x7f2a240b4b80 0x7f2a240cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:27.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a21e00640 1 --2- 172.21.15.40:0/577568586 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415e0b0 0x7f2a240b43a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009d90 tx=0x7f2a1401f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cbe555b34b2de16 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a21400640 1 -- 172.21.15.40:0/577568586 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a140b4070 con 0x7f2a2415e0b0 2024-03-24T00:34:27.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a21400640 1 -- 172.21.15.40:0/577568586 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a140af600 con 0x7f2a2415e0b0 2024-03-24T00:34:27.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a21400640 1 -- 172.21.15.40:0/577568586 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a140af9d0 con 0x7f2a2415e0b0 2024-03-24T00:34:27.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a28e00640 1 -- 172.21.15.40:0/577568586 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415e0b0 msgr2=0x7f2a240b43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a28e00640 1 --2- 172.21.15.40:0/577568586 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415e0b0 0x7f2a240b43a0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2a14009d90 tx=0x7f2a1401f150 comp rx=0 tx=0).stop 2024-03-24T00:34:27.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a28e00640 1 -- 172.21.15.40:0/577568586 shutdown_connections 2024-03-24T00:34:27.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a28e00640 1 --2- 172.21.15.40:0/577568586 >> v2:172.21.15.40:3300/0 conn(0x7f2a240b4b80 0x7f2a240cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a28e00640 1 --2- 172.21.15.40:0/577568586 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415e0b0 0x7f2a240b43a0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a28e00640 1 -- 172.21.15.40:0/577568586 >> 172.21.15.40:0/577568586 conn(0x7f2a240b3880 msgr2=0x7f2a240b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:27.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a28e00640 1 -- 172.21.15.40:0/577568586 shutdown_connections 2024-03-24T00:34:27.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.983+0000 7f2a28e00640 1 -- 172.21.15.40:0/577568586 wait complete. 2024-03-24T00:34:27.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a28e00640 1 Processor -- start 2024-03-24T00:34:27.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a28e00640 1 -- start start 2024-03-24T00:34:27.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a28e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2406b600 0x7f2a241f4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a28e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a240b4b80 0x7f2a240c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a28e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a2415e0b0 0x7f2a2415fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a28e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2406c480 con 0x7f2a2415e0b0 2024-03-24T00:34:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a28e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a240cdc70 con 0x7f2a240b4b80 2024-03-24T00:34:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a28e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2406bab0 con 0x7f2a2406b600 2024-03-24T00:34:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a21e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2406b600 0x7f2a241f4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a22800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a2415e0b0 0x7f2a2415fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a21e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2406b600 0x7f2a241f4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:56878/0 (socket says 172.21.15.40:56878) 2024-03-24T00:34:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a21e00640 1 -- 172.21.15.40:0/958863570 learned_addr learned my addr 172.21.15.40:0/958863570 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:27.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a19400640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a240b4b80 0x7f2a240c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a21e00640 1 -- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a240b4b80 msgr2=0x7f2a240c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a21e00640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a240b4b80 0x7f2a240c7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a21e00640 1 -- 172.21.15.40:0/958863570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a2415e0b0 msgr2=0x7f2a2415fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a21e00640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a2415e0b0 0x7f2a2415fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a21e00640 1 -- 172.21.15.40:0/958863570 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a240c90e0 con 0x7f2a2406b600 2024-03-24T00:34:27.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a22800640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a2415e0b0 0x7f2a2415fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:27.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a19400640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a240b4b80 0x7f2a240c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:27.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a21e00640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2406b600 0x7f2a241f4480 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2a1401cd80 tx=0x7f2a1401bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a1aa00640 1 -- 172.21.15.40:0/958863570 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a140b4040 con 0x7f2a2406b600 2024-03-24T00:34:27.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a1aa00640 1 -- 172.21.15.40:0/958863570 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a140c1070 con 0x7f2a2406b600 2024-03-24T00:34:27.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a28e00640 1 -- 172.21.15.40:0/958863570 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a240c93c0 con 0x7f2a2406b600 2024-03-24T00:34:27.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.987+0000 7f2a1aa00640 1 -- 172.21.15.40:0/958863570 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14009450 con 0x7f2a2406b600 2024-03-24T00:34:27.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.991+0000 7f2a28e00640 1 -- 172.21.15.40:0/958863570 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a240c9980 con 0x7f2a2406b600 2024-03-24T00:34:27.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.991+0000 7f2a1aa00640 1 -- 172.21.15.40:0/958863570 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2a140ba690 con 0x7f2a2406b600 2024-03-24T00:34:27.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.991+0000 7f2a1aa00640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a00067660 0x7f2a00069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.991+0000 7f2a1aa00640 1 -- 172.21.15.40:0/958863570 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a14124060 con 0x7f2a2406b600 2024-03-24T00:34:27.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.991+0000 7f2a19400640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a00067660 0x7f2a00069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.991+0000 7f2a2c210000 1 Processor -- start 2024-03-24T00:34:27.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.991+0000 7f2a2c210000 1 -- start start 2024-03-24T00:34:27.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.991+0000 7f2a19400640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a00067660 0x7f2a00069b20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2a240b2c70 tx=0x7f2a0c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c738129920 0x55c738129d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c73812a240 0x55c7380dfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c7380e03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:27.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c737f2d680 con 0x55c7380dfff0 2024-03-24T00:34:27.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c737eb4b00 con 0x55c738129920 2024-03-24T00:34:27.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c737f29720 con 0x55c73812a240 2024-03-24T00:34:27.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a22800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c7380e03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a19400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c73812a240 0x55c7380dfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a21e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c738129920 0x55c738129d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:27.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a22800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c7380e03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37064/0 (socket says 172.21.15.40:37064) 2024-03-24T00:34:27.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a22800640 1 -- 172.21.15.40:0/4285737375 learned_addr learned my addr 172.21.15.40:0/4285737375 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:27.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a19400640 1 -- 172.21.15.40:0/4285737375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c738129920 msgr2=0x55c738129d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a19400640 1 --2- 172.21.15.40:0/4285737375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c738129920 0x55c738129d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a19400640 1 -- 172.21.15.40:0/4285737375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 msgr2=0x55c7380e03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a19400640 1 --2- 172.21.15.40:0/4285737375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c7380e03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a19400640 1 -- 172.21.15.40:0/4285737375 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c737f05940 con 0x55c73812a240 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a22800640 1 --2- 172.21.15.40:0/4285737375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c7380e03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a19400640 1 --2- 172.21.15.40:0/4285737375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c73812a240 0x55c7380dfab0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2a24160770 tx=0x7f2a0c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6f34a42c1563729 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a12a00640 1 -- 172.21.15.40:0/4285737375 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c025070 con 0x55c73812a240 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a12a00640 1 -- 172.21.15.40:0/4285737375 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0c01b070 con 0x55c73812a240 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a12a00640 1 -- 172.21.15.40:0/4285737375 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c0215f0 con 0x55c73812a240 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 -- 172.21.15.40:0/4285737375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c73812a240 msgr2=0x55c7380dfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 --2- 172.21.15.40:0/4285737375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c73812a240 0x55c7380dfab0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2a24160770 tx=0x7f2a0c014000 comp rx=0 tx=0).stop 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 -- 172.21.15.40:0/4285737375 shutdown_connections 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 --2- 172.21.15.40:0/4285737375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c7380e03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 --2- 172.21.15.40:0/4285737375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c73812a240 0x55c7380dfab0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 --2- 172.21.15.40:0/4285737375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c738129920 0x55c738129d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 -- 172.21.15.40:0/4285737375 >> 172.21.15.40:0/4285737375 conn(0x55c7380dbfa0 msgr2=0x55c7380de390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:28.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 -- 172.21.15.40:0/4285737375 shutdown_connections 2024-03-24T00:34:28.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 -- 172.21.15.40:0/4285737375 wait complete. 2024-03-24T00:34:28.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 -- start start 2024-03-24T00:34:28.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.995+0000 7f2a2c210000 1 Processor -- start 2024-03-24T00:34:28.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c7380dfff0 0x55c73810ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:28.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c738129920 0x55c73810f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73812a240 0x55c73810f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c737f2c990 con 0x55c73812a240 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c737ee7ae0 con 0x55c7380dfff0 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c737ec5750 con 0x55c738129920 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 10 client.0 _ll_register_callbacks cb 0x55c7380caff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a22800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73812a240 0x55c73810f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a21e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c7380dfff0 0x55c73810ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a19400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c738129920 0x55c73810f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a22800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73812a240 0x55c73810f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37072/0 (socket says 172.21.15.40:37072) 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a22800640 1 -- 172.21.15.40:0/3749702105 learned_addr learned my addr 172.21.15.40:0/3749702105 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a21e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c7380dfff0 0x55c73810ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:54818/0 (socket says 172.21.15.40:54818) 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a19400640 1 -- 172.21.15.40:0/3749702105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c7380dfff0 msgr2=0x55c73810ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a19400640 1 --2- 172.21.15.40:0/3749702105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c7380dfff0 0x55c73810ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a19400640 1 -- 172.21.15.40:0/3749702105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73812a240 msgr2=0x55c73810f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:28.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a19400640 1 --2- 172.21.15.40:0/3749702105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73812a240 0x55c73810f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a19400640 1 -- 172.21.15.40:0/3749702105 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c737f72d80 con 0x55c738129920 2024-03-24T00:34:28.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a19400640 1 --2- 172.21.15.40:0/3749702105 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c738129920 0x55c73810f280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c024000 tx=0x7f2a0c004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c466dea069ab9432 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:28.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a07400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:34:28.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a07400640 1 -- 172.21.15.40:0/3749702105 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c025040 con 0x55c738129920 2024-03-24T00:34:28.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a07400640 1 -- 172.21.15.40:0/3749702105 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a0c0333f0 con 0x55c738129920 2024-03-24T00:34:28.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 1 -- 172.21.15.40:0/3749702105 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a0c01b070 con 0x55c738129920 2024-03-24T00:34:28.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a07400640 1 -- 172.21.15.40:0/3749702105 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a0c0059a0 con 0x55c738129920 2024-03-24T00:34:28.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 10 client.5900 fetch_fsmap learned FSMap version 57 2024-03-24T00:34:28.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 1 -- 172.21.15.40:0/3749702105 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x55c737eb4d10 con 0x55c738129920 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a07400640 1 -- 172.21.15.40:0/3749702105 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0c01b070 con 0x55c738129920 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a07400640 1 -- 172.21.15.40:0/3749702105 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 57) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f2a0c03d080 con 0x55c738129920 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 10 client.5900 fetch_fsmap finished waiting for FSMap version 57 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 10 client.5900 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 20 client.5900 populate_metadata read hostname 'smithi040' 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 1 --2- 172.21.15.40:0/3749702105 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c738110060 0x55c73813bea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 4 client.5900 mds_command: new command op to 6474 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a2c210000 1 -- 172.21.15.40:0/3749702105 --> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c737ec8b60 con 0x55c738110060 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a21e00640 1 --2- 172.21.15.40:0/3749702105 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c738110060 0x55c73813bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a21e00640 1 --2- 172.21.15.40:0/3749702105 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c738110060 0x55c73813bea0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26db73da07b8b62c server_cookie=fb2a719cd26c0227 in_seq=0 out_seq=0 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:27.999+0000 7f2a07400640 10 client.5900 ms_handle_connect on v2:172.21.15.53:6834/501079401 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a07400640 1 -- 172.21.15.40:0/3749702105 <== mds.0 v2:172.21.15.53:6834/501079401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c737ec8b60 con 0x55c738110060 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a07400640 10 client.5900 handle_command_reply: tid=0 2024-03-24T00:34:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a2c210000 1 client.5900 shutdown 2024-03-24T00:34:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a2c210000 1 -- 172.21.15.40:0/3749702105 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c738129920 msgr2=0x55c73810f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a2c210000 1 --2- 172.21.15.40:0/3749702105 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c738129920 0x55c73810f280 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c024000 tx=0x7f2a0c004740 comp rx=0 tx=0).stop 2024-03-24T00:34:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a2c210000 1 -- 172.21.15.40:0/3749702105 shutdown_connections 2024-03-24T00:34:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a2c210000 1 --2- 172.21.15.40:0/3749702105 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c738110060 0x55c73813bea0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a2c210000 1 --2- 172.21.15.40:0/3749702105 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c73812a240 0x55c73810f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a2c210000 1 --2- 172.21.15.40:0/3749702105 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c738129920 0x55c73810f280 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a2c210000 1 --2- 172.21.15.40:0/3749702105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c7380dfff0 0x55c73810ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a07400640 0 client.5900 ms_handle_reset on v2:172.21.15.53:6834/501079401 2024-03-24T00:34:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a2c210000 1 -- 172.21.15.40:0/3749702105 >> 172.21.15.40:0/3749702105 conn(0x55c7380dbfa0 msgr2=0x55c7380dec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a2c210000 1 -- 172.21.15.40:0/3749702105 shutdown_connections 2024-03-24T00:34:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a2c210000 1 -- 172.21.15.40:0/3749702105 wait complete. 2024-03-24T00:34:28.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.003+0000 7f2a2c210000 20 client.5900 trim_cache size 0 max 16384 2024-03-24T00:34:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 Processor -- start 2024-03-24T00:34:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 -- start start 2024-03-24T00:34:28.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c73810eed0 0x55c7380df7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c7380dbfa0 0x55c73810d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c73810de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a24200080 con 0x55c7380dfff0 2024-03-24T00:34:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a241ff420 con 0x55c73810eed0 2024-03-24T00:34:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a241ffa20 con 0x55c7380dbfa0 2024-03-24T00:34:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a22800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c73810de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a21e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c73810eed0 0x55c7380df7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a21e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c73810eed0 0x55c7380df7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:54834/0 (socket says 172.21.15.40:54834) 2024-03-24T00:34:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a21e00640 1 -- 172.21.15.40:0/346814378 learned_addr learned my addr 172.21.15.40:0/346814378 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a19400640 1 --2- 172.21.15.40:0/346814378 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c7380dbfa0 0x55c73810d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a21e00640 1 -- 172.21.15.40:0/346814378 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c7380dbfa0 msgr2=0x55c73810d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a21e00640 1 --2- 172.21.15.40:0/346814378 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c7380dbfa0 0x55c73810d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a21e00640 1 -- 172.21.15.40:0/346814378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 msgr2=0x55c73810de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a21e00640 1 --2- 172.21.15.40:0/346814378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c73810de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a21e00640 1 -- 172.21.15.40:0/346814378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7380e4200 con 0x55c73810eed0 2024-03-24T00:34:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a22800640 1 --2- 172.21.15.40:0/346814378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c73810de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a21e00640 1 --2- 172.21.15.40:0/346814378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c73810eed0 0x55c7380df7c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2a1401d800 tx=0x7f2a140bd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fbf172296140bbf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a12000640 1 -- 172.21.15.40:0/346814378 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1411f030 con 0x55c73810eed0 2024-03-24T00:34:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a12000640 1 -- 172.21.15.40:0/346814378 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a140f2720 con 0x55c73810eed0 2024-03-24T00:34:28.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a12000640 1 -- 172.21.15.40:0/346814378 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a140f2980 con 0x55c73810eed0 2024-03-24T00:34:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 -- 172.21.15.40:0/346814378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c73810eed0 msgr2=0x55c7380df7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 --2- 172.21.15.40:0/346814378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c73810eed0 0x55c7380df7c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2a1401d800 tx=0x7f2a140bd040 comp rx=0 tx=0).stop 2024-03-24T00:34:28.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 -- 172.21.15.40:0/346814378 shutdown_connections 2024-03-24T00:34:28.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 --2- 172.21.15.40:0/346814378 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c73810de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 --2- 172.21.15.40:0/346814378 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c7380dbfa0 0x55c73810d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 --2- 172.21.15.40:0/346814378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c73810eed0 0x55c7380df7c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 -- 172.21.15.40:0/346814378 >> 172.21.15.40:0/346814378 conn(0x55c7380e2750 msgr2=0x55c738128500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:28.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 -- 172.21.15.40:0/346814378 shutdown_connections 2024-03-24T00:34:28.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.035+0000 7f2a2c210000 1 -- 172.21.15.40:0/346814378 wait complete. 2024-03-24T00:34:28.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a2c210000 1 -- start start 2024-03-24T00:34:28.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a2c210000 1 Processor -- start 2024-03-24T00:34:28.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a2c210000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c7380dbfa0 0x55c73813cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a2c210000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c73813d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a2c210000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c73810eed0 0x55c73813d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a2c210000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c737ed5a80 con 0x55c7380dfff0 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a2c210000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2415a480 con 0x55c7380dbfa0 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a2c210000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c737fea700 con 0x55c73810eed0 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a2c210000 10 client.0 _ll_register_callbacks cb 0x7f2a1c007060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a19400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c73813d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a19400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c73813d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37098/0 (socket says 172.21.15.40:37098) 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a19400640 1 -- 172.21.15.40:0/16515999 learned_addr learned my addr 172.21.15.40:0/16515999 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a22800640 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c73810eed0 0x55c73813d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a21e00640 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c7380dbfa0 0x55c73813cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a19400640 1 -- 172.21.15.40:0/16515999 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c73810eed0 msgr2=0x55c73813d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:28.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a19400640 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c73810eed0 0x55c73813d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a19400640 1 -- 172.21.15.40:0/16515999 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c7380dbfa0 msgr2=0x55c73813cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a19400640 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c7380dbfa0 0x55c73813cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a19400640 1 -- 172.21.15.40:0/16515999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c737efa540 con 0x55c7380dfff0 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a19400640 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c73813d430 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c0337a0 tx=0x7f2a0c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb0e5bcb6b227863 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a07400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.039+0000 7f2a07400640 1 -- 172.21.15.40:0/16515999 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c025040 con 0x55c7380dfff0 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a07400640 1 -- 172.21.15.40:0/16515999 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a0c045680 con 0x55c7380dfff0 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 -- 172.21.15.40:0/16515999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2a140f2720 con 0x55c7380dfff0 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a07400640 1 -- 172.21.15.40:0/16515999 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a0c04d050 con 0x55c7380dfff0 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 10 client.6732 fetch_fsmap learned FSMap version 57 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 -- 172.21.15.40:0/16515999 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x55c737ec14b0 con 0x55c7380dfff0 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a07400640 1 -- 172.21.15.40:0/16515999 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a140f2720 con 0x55c7380dfff0 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a07400640 1 -- 172.21.15.40:0/16515999 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 57) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f2a0c050020 con 0x55c7380dfff0 2024-03-24T00:34:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 10 client.6732 fetch_fsmap finished waiting for FSMap version 57 2024-03-24T00:34:28.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 10 client.6732 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:34:28.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 20 client.6732 populate_metadata read hostname 'smithi040' 2024-03-24T00:34:28.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c738129920 0x55c73813e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:28.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 4 client.6732 mds_command: new command op to 6474 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:34:28.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 -- 172.21.15.40:0/16515999 --> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c737f3abc0 con 0x55c738129920 2024-03-24T00:34:28.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a21e00640 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c738129920 0x55c73813e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:28.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a21e00640 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c738129920 0x55c73813e300 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e47aa592ce4d426 server_cookie=54453fd04bad80cf in_seq=0 out_seq=0 2024-03-24T00:34:28.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a07400640 10 client.6732 ms_handle_connect on v2:172.21.15.53:6834/501079401 2024-03-24T00:34:28.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a07400640 1 -- 172.21.15.40:0/16515999 <== mds.0 v2:172.21.15.53:6834/501079401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c737f3abc0 con 0x55c738129920 2024-03-24T00:34:28.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a07400640 10 client.6732 handle_command_reply: tid=0 2024-03-24T00:34:28.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 client.6732 shutdown 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 -- 172.21.15.40:0/16515999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 msgr2=0x55c73813d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c73813d430 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c0337a0 tx=0x7f2a0c018000 comp rx=0 tx=0).stop 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 -- 172.21.15.40:0/16515999 shutdown_connections 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.53:6834/501079401,v1:172.21.15.53:6835/501079401] conn(0x55c738129920 0x55c73813e300 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c73810eed0 0x55c73813d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c7380dfff0 0x55c73813d430 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 --2- 172.21.15.40:0/16515999 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c7380dbfa0 0x55c73813cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 -- 172.21.15.40:0/16515999 >> 172.21.15.40:0/16515999 conn(0x55c7380e2750 msgr2=0x55c738128d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a07400640 0 client.6732 ms_handle_reset on v2:172.21.15.53:6834/501079401 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 -- 172.21.15.40:0/16515999 shutdown_connections 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 1 -- 172.21.15.40:0/16515999 wait complete. 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a2c210000 20 client.6732 trim_cache size 0 max 16384 2024-03-24T00:34:28.049 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:34:28.050 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:34:28.050 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:34:28.050 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6474, 2024-03-24T00:34:28.050 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:34:28.050 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:34:28.050 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:34:28.050 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 15.432940995999999, 2024-03-24T00:34:28.050 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 57, 2024-03-24T00:34:28.050 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 59, 2024-03-24T00:34:28.050 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 59, 2024-03-24T00:34:28.051 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 27.355908361000001 2024-03-24T00:34:28.051 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:34:28.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a28e00640 1 -- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a00067660 msgr2=0x7f2a00069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:28.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a28e00640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a00067660 0x7f2a00069b20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2a240b2c70 tx=0x7f2a0c009190 comp rx=0 tx=0).stop 2024-03-24T00:34:28.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a28e00640 1 -- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2406b600 msgr2=0x7f2a241f4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:28.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a28e00640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2406b600 0x7f2a241f4480 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2a1401cd80 tx=0x7f2a1401bef0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a28e00640 1 -- 172.21.15.40:0/958863570 shutdown_connections 2024-03-24T00:34:28.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a28e00640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a00067660 0x7f2a00069b20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a28e00640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a2415e0b0 0x7f2a2415fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a28e00640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a240b4b80 0x7f2a240c7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a28e00640 1 --2- 172.21.15.40:0/958863570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2406b600 0x7f2a241f4480 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:28.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a28e00640 1 -- 172.21.15.40:0/958863570 >> 172.21.15.40:0/958863570 conn(0x7f2a240b3880 msgr2=0x7f2a2406fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:28.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a28e00640 1 -- 172.21.15.40:0/958863570 shutdown_connections 2024-03-24T00:34:28.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:28.043+0000 7f2a28e00640 1 -- 172.21.15.40:0/958863570 wait complete. 2024-03-24T00:34:28.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:28.066 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:34:28.066 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:34:28.066 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p tree 2024-03-24T00:34:28.066 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:34:28.066 DEBUG:teuthology.orchestra.run.smithi040:> setfattr -n ceph.dir.pin.distributed -v 1 tree 2024-03-24T00:34:28.066 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:34:28.066 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < 100; i++)); do 2024-03-24T00:34:28.066 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p "tree/$i" 2024-03-24T00:34:28.066 DEBUG:teuthology.orchestra.run.smithi040:> echo file > "tree/$i/file" 2024-03-24T00:34:28.066 DEBUG:teuthology.orchestra.run.smithi040:> done 2024-03-24T00:34:28.066 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:34:28.077 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree 2024-03-24T00:34:28.081 INFO:teuthology.orchestra.run.smithi040.stderr:+ setfattr -n ceph.dir.pin.distributed -v 1 tree 2024-03-24T00:34:28.085 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i = 0 )) 2024-03-24T00:34:28.085 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.085 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/0 2024-03-24T00:34:28.090 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.097 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.097 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.097 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/1 2024-03-24T00:34:28.106 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.109 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.109 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.109 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/2 2024-03-24T00:34:28.116 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.118 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.119 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.119 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/3 2024-03-24T00:34:28.125 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.128 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.128 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.128 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/4 2024-03-24T00:34:28.134 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.138 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.138 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.138 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/5 2024-03-24T00:34:28.144 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.147 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.147 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.147 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/6 2024-03-24T00:34:28.154 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.157 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.157 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.157 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/7 2024-03-24T00:34:28.166 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.166 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.167 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.167 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/8 2024-03-24T00:34:28.173 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.176 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.176 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.176 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/9 2024-03-24T00:34:28.183 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.187 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.187 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.187 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/10 2024-03-24T00:34:28.193 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.196 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.196 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.197 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/11 2024-03-24T00:34:28.203 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.206 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.206 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.206 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/12 2024-03-24T00:34:28.213 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.216 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.216 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.216 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/13 2024-03-24T00:34:28.223 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/14 2024-03-24T00:34:28.234 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.237 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/15 2024-03-24T00:34:28.244 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/16 2024-03-24T00:34:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/17 2024-03-24T00:34:28.263 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.265 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.265 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.265 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/18 2024-03-24T00:34:28.272 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.275 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.275 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.275 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/19 2024-03-24T00:34:28.282 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.285 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.285 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.285 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/20 2024-03-24T00:34:28.292 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/21 2024-03-24T00:34:28.302 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.304 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.304 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.304 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/22 2024-03-24T00:34:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.314 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.314 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.314 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/23 2024-03-24T00:34:28.321 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.324 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.324 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.324 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/24 2024-03-24T00:34:28.331 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.333 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.333 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.333 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/25 2024-03-24T00:34:28.340 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.342 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.342 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.343 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/26 2024-03-24T00:34:28.349 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.352 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.352 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.352 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/27 2024-03-24T00:34:28.358 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.361 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.361 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.361 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/28 2024-03-24T00:34:28.370 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.370 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.371 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.371 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/29 2024-03-24T00:34:28.376 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.379 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.379 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.379 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/30 2024-03-24T00:34:28.386 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.388 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.388 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.388 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/31 2024-03-24T00:34:28.393 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.397 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.397 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/32 2024-03-24T00:34:28.402 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.405 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.405 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.405 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/33 2024-03-24T00:34:28.412 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/34 2024-03-24T00:34:28.421 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.423 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.423 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.423 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/35 2024-03-24T00:34:28.430 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.433 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.433 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.433 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/36 2024-03-24T00:34:28.439 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.442 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/37 2024-03-24T00:34:28.448 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.450 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.450 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.450 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/38 2024-03-24T00:34:28.456 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/39 2024-03-24T00:34:28.465 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.468 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.468 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.468 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/40 2024-03-24T00:34:28.475 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.478 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.478 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.478 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/41 2024-03-24T00:34:28.484 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.487 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.487 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.487 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/42 2024-03-24T00:34:28.494 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.497 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.497 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.497 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/43 2024-03-24T00:34:28.504 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/44 2024-03-24T00:34:28.514 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.517 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.518 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.518 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/45 2024-03-24T00:34:28.525 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.528 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.528 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.529 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/46 2024-03-24T00:34:28.535 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.538 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.538 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.539 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/47 2024-03-24T00:34:28.545 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.549 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.549 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.549 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/48 2024-03-24T00:34:28.555 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.558 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/49 2024-03-24T00:34:28.564 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.567 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.567 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.567 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/50 2024-03-24T00:34:28.574 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.576 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.577 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.577 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/51 2024-03-24T00:34:28.584 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.587 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.587 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.588 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/52 2024-03-24T00:34:28.595 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.598 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/53 2024-03-24T00:34:28.604 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.609 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.609 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.609 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/54 2024-03-24T00:34:28.614 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.617 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.617 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.617 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/55 2024-03-24T00:34:28.624 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/56 2024-03-24T00:34:28.634 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.637 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.637 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.637 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/57 2024-03-24T00:34:28.645 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.648 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/58 2024-03-24T00:34:28.656 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.659 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/59 2024-03-24T00:34:28.665 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.668 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/60 2024-03-24T00:34:28.675 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.678 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.678 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.678 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/61 2024-03-24T00:34:28.684 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.687 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/62 2024-03-24T00:34:28.694 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.696 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/63 2024-03-24T00:34:28.702 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.705 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/64 2024-03-24T00:34:28.712 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.714 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.715 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/65 2024-03-24T00:34:28.721 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/66 2024-03-24T00:34:28.730 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.733 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/67 2024-03-24T00:34:28.739 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.742 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/68 2024-03-24T00:34:28.748 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.751 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/69 2024-03-24T00:34:28.757 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/70 2024-03-24T00:34:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/71 2024-03-24T00:34:28.776 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.779 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.779 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.779 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/72 2024-03-24T00:34:28.786 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.789 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.789 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.789 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/73 2024-03-24T00:34:28.797 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.800 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.800 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.800 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/74 2024-03-24T00:34:28.807 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.810 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.810 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.810 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/75 2024-03-24T00:34:28.818 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.820 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.820 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.820 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/76 2024-03-24T00:34:28.827 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/77 2024-03-24T00:34:28.836 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.839 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.839 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.839 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/78 2024-03-24T00:34:28.845 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/79 2024-03-24T00:34:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.858 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/80 2024-03-24T00:34:28.864 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.867 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.867 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.867 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/81 2024-03-24T00:34:28.873 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.875 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.875 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.875 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/82 2024-03-24T00:34:28.882 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.884 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.884 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.884 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/83 2024-03-24T00:34:28.891 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.894 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/84 2024-03-24T00:34:28.899 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/85 2024-03-24T00:34:28.909 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.911 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.911 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.911 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/86 2024-03-24T00:34:28.917 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.920 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.920 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.920 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/87 2024-03-24T00:34:28.927 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.929 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.929 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/88 2024-03-24T00:34:28.935 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.938 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.938 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.938 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/89 2024-03-24T00:34:28.945 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.947 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.947 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.947 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/90 2024-03-24T00:34:28.954 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.957 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.957 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.957 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/91 2024-03-24T00:34:28.963 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.966 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.966 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.966 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/92 2024-03-24T00:34:28.973 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.976 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.977 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.977 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/93 2024-03-24T00:34:28.983 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.986 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.986 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.986 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/94 2024-03-24T00:34:28.993 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:28.996 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:28.996 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:28.997 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/95 2024-03-24T00:34:29.003 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:29.006 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:29.007 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:29.007 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/96 2024-03-24T00:34:29.013 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:29.015 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:29.016 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:29.016 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/97 2024-03-24T00:34:29.022 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:29.025 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:29.026 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:29.026 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/98 2024-03-24T00:34:29.032 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:29.035 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:29.035 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:29.036 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/99 2024-03-24T00:34:29.042 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:34:29.045 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:34:29.045 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:34:29.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:34:29.248 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:34:29.248 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:34:29.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-24T00:34:29.431 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:34:29.432 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:34:29.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:34:29.591 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"1488","version":"1486","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"429","version":"429","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-24T00:34:29.592 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "1488", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "1486" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "429", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "429" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:34:29.592 INFO:tasks.cephfs.cephfs_test_case:len=0 [] 2024-03-24T00:34:34.593 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:34:34.755 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:34:34.755 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:34:34.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-24T00:34:34.915 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:34:34.916 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:34:34.916 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:34:35.076 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"1494","version":"1490","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":2,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"431","version":"431","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-24T00:34:35.076 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 2, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "1494", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "1490" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "431", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "431" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:34:35.077 INFO:tasks.cephfs.cephfs_test_case:len=0 [] 2024-03-24T00:34:37.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:34:37.243 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:34:40.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:34:40.237 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"1497","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":3},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x100000002ca.110*","snapid_first":2,"projected_version":"442","version":"442","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x100000002ca.101*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x100000002ca.010*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-24T00:34:40.238 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 3 }, "snapid_first": 2, "states": [], "version": "1497" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002ca.110*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "442", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "442" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002ca.101*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "422", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002ca.010*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:34:40.238 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-24T00:34:40.398 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"1497","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":3},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000002ca.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000002ca.100*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000002ca.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:34:40.398 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 3 }, "snapid_first": 2, "states": [], "version": "1497" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x100000002ca.111*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "416", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x100000002ca.100*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x100000002ca.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "430", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:34:40.399 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:34:40.638 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"1497","version":"1497","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":3,"2":3}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"431","version":"431","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000002ca.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000002ca.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x100000002ca.000*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x100000002ca.101*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x100000002ca.010*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x100000002ca.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000002ca.100*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x100000002ca.110*","snapid_first":2,"projected_version":"442","version":"442","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-03-24T00:34:40.639 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 3, "2": 3 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "1497", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "1497" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "431", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "431" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x100000002ca.011*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "430", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x100000002ca.111*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "416", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x100000002ca.000*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002ca.101*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "422", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002ca.010*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x100000002ca.001*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "428", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x100000002ca.100*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002ca.110*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "442", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "442" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-24T00:34:40.639 INFO:tasks.cephfs.cephfs_test_case:len=8 [{'is_auth': True, 'auth_first': 2, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 2, 'dir': {'path': '/tree', 'dirfrag': '0x100000002ca.110*', 'snapid_first': 2, 'projected_version': '442', 'version': '442', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '2', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [2, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 2, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 2, 'dir': {'path': '/tree', 'dirfrag': '0x100000002ca.101*', 'snapid_first': 2, 'projected_version': '422', 'version': '422', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '2', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [2, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 2, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 2, 'dir': {'path': '/tree', 'dirfrag': '0x100000002ca.010*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '2', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [2, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x100000002ca.111*', 'snapid_first': 2, 'projected_version': '416', 'version': '416', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x100000002ca.100*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x100000002ca.011*', 'snapid_first': 2, 'projected_version': '430', 'version': '430', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x100000002ca.000*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x100000002ca.001*', 'snapid_first': 2, 'projected_version': '428', 'version': '428', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}] 2024-03-24T00:34:40.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:40.640 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin.distributed -v 0 tree) 2024-03-24T00:34:55.654 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:34:55.853 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"1500","version":"1500","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":3,"2":3}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"431","version":"431","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"/tree","dirfrag":"0x100000002ca.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"/tree","dirfrag":"0x100000002ca.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"/tree","dirfrag":"0x100000002ca.101*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"/tree","dirfrag":"0x100000002ca.010*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"/tree","dirfrag":"0x100000002ca.100*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"/tree","dirfrag":"0x100000002ca.110*","snapid_first":2,"projected_version":"442","version":"442","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-03-24T00:34:55.854 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 3, "2": 3 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "1500", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "1500" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "431", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "431" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x100000002ca.011*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "430", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "430" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x100000002ca.111*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "416", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "416" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002ca.101*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "422", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "422" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002ca.010*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x100000002ca.100*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x100000002ca.110*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "442", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "442" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-24T00:34:55.855 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:34:55.855 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:34:55.904 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:34:55.974 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:34:55.975 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:34:55.975 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:34:55.984 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:34:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f3656400640 1 Processor -- start 2024-03-24T00:34:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f3656400640 1 -- start start 2024-03-24T00:34:56.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f3656400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3650141bc0 0x7f36500ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f3656400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500ce630 0x7f3650071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f3656400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3650070e00 con 0x7f365013ec80 2024-03-24T00:34:56.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f3656400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36500cee20 con 0x7f3650141bc0 2024-03-24T00:34:56.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f3656400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36500cf100 con 0x7f36500ce630 2024-03-24T00:34:56.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f364f400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f365013ec80 0x7f36501414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:42286/0 (socket says 172.21.15.40:42286) 2024-03-24T00:34:56.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f364fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500ce630 0x7f3650071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f364f400640 1 -- 172.21.15.40:0/1192099125 learned_addr learned my addr 172.21.15.40:0/1192099125 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:56.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f3647e00640 1 --2- 172.21.15.40:0/1192099125 >> v2:172.21.15.40:3300/0 conn(0x7f3650141bc0 0x7f36500ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f364ea00640 1 -- 172.21.15.40:0/1192099125 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035242390 0 0) 0x7f3650070e00 con 0x7f365013ec80 2024-03-24T00:34:56.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f364ea00640 1 -- 172.21.15.40:0/1192099125 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3634003530 con 0x7f365013ec80 2024-03-24T00:34:56.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f364fe00640 1 -- 172.21.15.40:0/1192099125 >> v2:172.21.15.40:3300/0 conn(0x7f3650141bc0 msgr2=0x7f36500ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:56.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f364fe00640 1 --2- 172.21.15.40:0/1192099125 >> v2:172.21.15.40:3300/0 conn(0x7f3650141bc0 0x7f36500ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f364fe00640 1 -- 172.21.15.40:0/1192099125 >> v1:172.21.15.53:6789/0 conn(0x7f365013ec80 legacy=0x7f36501414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.167+0000 7f364fe00640 1 -- 172.21.15.40:0/1192099125 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36500cf3e0 con 0x7f36500ce630 2024-03-24T00:34:56.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f364fe00640 1 --2- 172.21.15.40:0/1192099125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500ce630 0x7f3650071300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3648009d90 tx=0x7f364801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff86cb775718cf49 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:56.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f364ea00640 1 -- 172.21.15.40:0/1192099125 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36480b2070 con 0x7f36500ce630 2024-03-24T00:34:56.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f364ea00640 1 -- 172.21.15.40:0/1192099125 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36480a5a50 con 0x7f36500ce630 2024-03-24T00:34:56.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f364ea00640 1 -- 172.21.15.40:0/1192099125 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36480a5e20 con 0x7f36500ce630 2024-03-24T00:34:56.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f3656400640 1 -- 172.21.15.40:0/1192099125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500ce630 msgr2=0x7f3650071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f3656400640 1 --2- 172.21.15.40:0/1192099125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500ce630 0x7f3650071300 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3648009d90 tx=0x7f364801c970 comp rx=0 tx=0).stop 2024-03-24T00:34:56.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f3656400640 1 -- 172.21.15.40:0/1192099125 shutdown_connections 2024-03-24T00:34:56.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f3656400640 1 --2- 172.21.15.40:0/1192099125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36500ce630 0x7f3650071300 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f3656400640 1 --2- 172.21.15.40:0/1192099125 >> v2:172.21.15.40:3300/0 conn(0x7f3650141bc0 0x7f36500ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f3656400640 1 -- 172.21.15.40:0/1192099125 >> 172.21.15.40:0/1192099125 conn(0x7f36500b17b0 msgr2=0x7f36500b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:56.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f3656400640 1 -- 172.21.15.40:0/1192099125 shutdown_connections 2024-03-24T00:34:56.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f3656400640 1 -- 172.21.15.40:0/1192099125 wait complete. 2024-03-24T00:34:56.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.171+0000 7f3656400640 1 Processor -- start 2024-03-24T00:34:56.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3656400640 1 -- start start 2024-03-24T00:34:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3656400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36500ce630 0x7f36501563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3656400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f365013ec80 0x7f365015b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3656400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3650141bc0 0x7f365015b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3656400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36500723e0 con 0x7f3650141bc0 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3656400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3650072260 con 0x7f36500ce630 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3656400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3650072560 con 0x7f365013ec80 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f364fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3650141bc0 0x7f365015b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f364f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36500ce630 0x7f36501563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3647e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f365013ec80 0x7f365015b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f364fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3650141bc0 0x7f365015b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33810/0 (socket says 172.21.15.40:33810) 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f364fe00640 1 -- 172.21.15.40:0/2410300538 learned_addr learned my addr 172.21.15.40:0/2410300538 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f364fe00640 1 -- 172.21.15.40:0/2410300538 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f365013ec80 msgr2=0x7f365015b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f364fe00640 1 --2- 172.21.15.40:0/2410300538 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f365013ec80 0x7f365015b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f364fe00640 1 -- 172.21.15.40:0/2410300538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36500ce630 msgr2=0x7f36501563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f364fe00640 1 --2- 172.21.15.40:0/2410300538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36500ce630 0x7f36501563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f364fe00640 1 -- 172.21.15.40:0/2410300538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3650159970 con 0x7f3650141bc0 2024-03-24T00:34:56.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f364fe00640 1 --2- 172.21.15.40:0/2410300538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3650141bc0 0x7f365015b850 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f364801b020 tx=0x7f3648019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3647400640 1 -- 172.21.15.40:0/2410300538 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36480b2050 con 0x7f3650141bc0 2024-03-24T00:34:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3647400640 1 -- 172.21.15.40:0/2410300538 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36480be070 con 0x7f3650141bc0 2024-03-24T00:34:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3647400640 1 -- 172.21.15.40:0/2410300538 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3648009450 con 0x7f3650141bc0 2024-03-24T00:34:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3656400640 1 -- 172.21.15.40:0/2410300538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f365012c240 con 0x7f3650141bc0 2024-03-24T00:34:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3656400640 1 -- 172.21.15.40:0/2410300538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f365012c800 con 0x7f3650141bc0 2024-03-24T00:34:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.175+0000 7f3656400640 1 -- 172.21.15.40:0/2410300538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3650068170 con 0x7f3650141bc0 2024-03-24T00:34:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.179+0000 7f3647400640 1 -- 172.21.15.40:0/2410300538 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f36480c3460 con 0x7f3650141bc0 2024-03-24T00:34:56.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.179+0000 7f3647400640 1 --2- 172.21.15.40:0/2410300538 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f362c067660 0x7f362c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.179+0000 7f3647400640 1 -- 172.21.15.40:0/2410300538 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3648120b80 con 0x7f3650141bc0 2024-03-24T00:34:56.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.179+0000 7f364f400640 1 --2- 172.21.15.40:0/2410300538 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f362c067660 0x7f362c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.179+0000 7f364f400640 1 --2- 172.21.15.40:0/2410300538 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f362c067660 0x7f362c069b20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3640007fd0 tx=0x7f3640009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:56.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.183+0000 7f3647400640 1 -- 172.21.15.40:0/2410300538 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3648016560 con 0x7f3650141bc0 2024-03-24T00:34:56.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.347+0000 7f3656400640 1 -- 172.21.15.40:0/2410300538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3650072b90 con 0x7f3650141bc0 2024-03-24T00:34:56.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.347+0000 7f3647400640 1 -- 172.21.15.40:0/2410300538 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f36480f12c0 con 0x7f3650141bc0 2024-03-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:56.351 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":59,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:34:13.623804+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:34:10.428162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-24T00:34:11.001094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.351+0000 7f3656400640 1 -- 172.21.15.40:0/2410300538 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f362c067660 msgr2=0x7f362c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.351+0000 7f3656400640 1 --2- 172.21.15.40:0/2410300538 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f362c067660 0x7f362c069b20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3640007fd0 tx=0x7f3640009040 comp rx=0 tx=0).stop 2024-03-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.351+0000 7f3656400640 1 -- 172.21.15.40:0/2410300538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3650141bc0 msgr2=0x7f365015b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.351+0000 7f3656400640 1 --2- 172.21.15.40:0/2410300538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3650141bc0 0x7f365015b850 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f364801b020 tx=0x7f3648019710 comp rx=0 tx=0).stop 2024-03-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.351+0000 7f3656400640 1 -- 172.21.15.40:0/2410300538 shutdown_connections 2024-03-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.351+0000 7f3656400640 1 --2- 172.21.15.40:0/2410300538 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f362c067660 0x7f362c069b20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.351+0000 7f3656400640 1 --2- 172.21.15.40:0/2410300538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3650141bc0 0x7f365015b850 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.351+0000 7f3656400640 1 --2- 172.21.15.40:0/2410300538 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f365013ec80 0x7f365015b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.351+0000 7f3656400640 1 --2- 172.21.15.40:0/2410300538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36500ce630 0x7f36501563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.351+0000 7f3656400640 1 -- 172.21.15.40:0/2410300538 >> 172.21.15.40:0/2410300538 conn(0x7f36500b17b0 msgr2=0x7f365015e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.351+0000 7f3656400640 1 -- 172.21.15.40:0/2410300538 shutdown_connections 2024-03-24T00:34:56.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.351+0000 7f3656400640 1 -- 172.21.15.40:0/2410300538 wait complete. 2024-03-24T00:34:56.372 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:34:10.428162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:34:11.001094+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:34:56.373 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:34:56.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1fa00640 1 Processor -- start 2024-03-24T00:34:56.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1fa00640 1 -- start start 2024-03-24T00:34:56.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1fa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a1815bd00 0x7f9a180ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1fa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9a1815ffd0 0x7f9a180cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1fa00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a18072d30 con 0x7f9a180728d0 2024-03-24T00:34:56.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1fa00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a180cf790 con 0x7f9a1815ffd0 2024-03-24T00:34:56.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1fa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a18071de0 con 0x7f9a1815bd00 2024-03-24T00:34:56.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a17e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9a180728d0 0x7f9a1815f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39582/0 (socket says 172.21.15.40:39582) 2024-03-24T00:34:56.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a17e00640 1 -- 172.21.15.40:0/2621843392 learned_addr learned my addr 172.21.15.40:0/2621843392 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:56.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1d000640 1 --2- 172.21.15.40:0/2621843392 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a1815bd00 0x7f9a180ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1da00640 1 --2- 172.21.15.40:0/2621843392 >> v2:172.21.15.53:3300/0 conn(0x7f9a1815ffd0 0x7f9a180cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1d000640 1 -- 172.21.15.40:0/2621843392 >> v2:172.21.15.53:3300/0 conn(0x7f9a1815ffd0 msgr2=0x7f9a180cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:56.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1d000640 1 --2- 172.21.15.40:0/2621843392 >> v2:172.21.15.53:3300/0 conn(0x7f9a1815ffd0 0x7f9a180cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1d000640 1 -- 172.21.15.40:0/2621843392 >> v1:172.21.15.40:6789/0 conn(0x7f9a180728d0 legacy=0x7f9a1815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a17400640 1 -- 172.21.15.40:0/2621843392 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809655697 0 0) 0x7f9a18072d30 con 0x7f9a180728d0 2024-03-24T00:34:56.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1d000640 1 -- 172.21.15.40:0/2621843392 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a180cea50 con 0x7f9a1815bd00 2024-03-24T00:34:56.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.515+0000 7f9a1da00640 1 --2- 172.21.15.40:0/2621843392 >> v2:172.21.15.53:3300/0 conn(0x7f9a1815ffd0 0x7f9a180cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:56.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.519+0000 7f9a1d000640 1 --2- 172.21.15.40:0/2621843392 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a1815bd00 0x7f9a180ae8e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c0097c0 tx=0x7f9a0c01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5197bbe3bb035633 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:56.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.519+0000 7f9a17400640 1 -- 172.21.15.40:0/2621843392 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c0a7070 con 0x7f9a1815bd00 2024-03-24T00:34:56.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.519+0000 7f9a17400640 1 -- 172.21.15.40:0/2621843392 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0c0b0070 con 0x7f9a1815bd00 2024-03-24T00:34:56.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.519+0000 7f9a17400640 1 -- 172.21.15.40:0/2621843392 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0c0ab890 con 0x7f9a1815bd00 2024-03-24T00:34:56.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.519+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2621843392 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a1815bd00 msgr2=0x7f9a180ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.519+0000 7f9a1fa00640 1 --2- 172.21.15.40:0/2621843392 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a1815bd00 0x7f9a180ae8e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c0097c0 tx=0x7f9a0c01c470 comp rx=0 tx=0).stop 2024-03-24T00:34:56.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.519+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2621843392 shutdown_connections 2024-03-24T00:34:56.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.519+0000 7f9a1fa00640 1 --2- 172.21.15.40:0/2621843392 >> v2:172.21.15.53:3300/0 conn(0x7f9a1815ffd0 0x7f9a180cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.519+0000 7f9a1fa00640 1 --2- 172.21.15.40:0/2621843392 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a1815bd00 0x7f9a180ae8e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.519+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2621843392 >> 172.21.15.40:0/2621843392 conn(0x7f9a180b1660 msgr2=0x7f9a180b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:56.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.519+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2621843392 shutdown_connections 2024-03-24T00:34:56.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.519+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2621843392 wait complete. 2024-03-24T00:34:56.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1fa00640 1 Processor -- start 2024-03-24T00:34:56.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1fa00640 1 -- start start 2024-03-24T00:34:56.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1fa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a180728d0 0x7f9a181417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1fa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a1815bd00 0x7f9a18142490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1fa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a1815ffd0 0x7f9a1813e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1fa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1806d320 con 0x7f9a180728d0 2024-03-24T00:34:56.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1fa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1815c0e0 con 0x7f9a1815bd00 2024-03-24T00:34:56.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1fa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1806d4a0 con 0x7f9a1815ffd0 2024-03-24T00:34:56.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a180728d0 0x7f9a181417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a17e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a1815bd00 0x7f9a18142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a180728d0 0x7f9a181417e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33812/0 (socket says 172.21.15.40:33812) 2024-03-24T00:34:56.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1d000640 1 -- 172.21.15.40:0/2148654959 learned_addr learned my addr 172.21.15.40:0/2148654959 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:56.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1da00640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a1815ffd0 0x7f9a1813e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a17e00640 1 -- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a1815ffd0 msgr2=0x7f9a1813e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a17e00640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a1815ffd0 0x7f9a1813e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a17e00640 1 -- 172.21.15.40:0/2148654959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a180728d0 msgr2=0x7f9a181417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a17e00640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a180728d0 0x7f9a181417e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a17e00640 1 -- 172.21.15.40:0/2148654959 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1814f8e0 con 0x7f9a1815bd00 2024-03-24T00:34:56.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a17e00640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a1815bd00 0x7f9a18142490 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9a0400c460 tx=0x7f9a0400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:56.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a14c00640 1 -- 172.21.15.40:0/2148654959 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a040074e0 con 0x7f9a1815bd00 2024-03-24T00:34:56.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a14c00640 1 -- 172.21.15.40:0/2148654959 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a040047f0 con 0x7f9a1815bd00 2024-03-24T00:34:56.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2148654959 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1814fbc0 con 0x7f9a1815bd00 2024-03-24T00:34:56.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a14c00640 1 -- 172.21.15.40:0/2148654959 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a04004be0 con 0x7f9a1815bd00 2024-03-24T00:34:56.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.523+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2148654959 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a18143520 con 0x7f9a1815bd00 2024-03-24T00:34:56.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.527+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2148654959 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a18068170 con 0x7f9a1815bd00 2024-03-24T00:34:56.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.527+0000 7f9a14c00640 1 -- 172.21.15.40:0/2148654959 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f9a04011070 con 0x7f9a1815bd00 2024-03-24T00:34:56.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.527+0000 7f9a14c00640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f99f8067660 0x7f99f8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.527+0000 7f9a14c00640 1 -- 172.21.15.40:0/2148654959 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a040853a0 con 0x7f9a1815bd00 2024-03-24T00:34:56.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.527+0000 7f9a1d000640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f99f8067660 0x7f99f8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.527+0000 7f9a1d000640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f99f8067660 0x7f99f8069b20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c01ab20 tx=0x7f9a0c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:56.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.531+0000 7f9a14c00640 1 -- 172.21.15.40:0/2148654959 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9a040560d0 con 0x7f9a1815bd00 2024-03-24T00:34:56.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.695+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2148654959 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9a1806e870 con 0x7f9a1815bd00 2024-03-24T00:34:56.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.695+0000 7f9a14c00640 1 -- 172.21.15.40:0/2148654959 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9a040559d0 con 0x7f9a1815bd00 2024-03-24T00:34:56.699 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:34:56.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.699+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f99f8067660 msgr2=0x7f99f8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.699+0000 7f9a1fa00640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f99f8067660 0x7f99f8069b20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c01ab20 tx=0x7f9a0c0a7000 comp rx=0 tx=0).stop 2024-03-24T00:34:56.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.699+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a1815bd00 msgr2=0x7f9a18142490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.699+0000 7f9a1fa00640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a1815bd00 0x7f9a18142490 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f9a0400c460 tx=0x7f9a0400c860 comp rx=0 tx=0).stop 2024-03-24T00:34:56.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.699+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2148654959 shutdown_connections 2024-03-24T00:34:56.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.699+0000 7f9a1fa00640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f99f8067660 0x7f99f8069b20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.699+0000 7f9a1fa00640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a1815ffd0 0x7f9a1813e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.699+0000 7f9a1fa00640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a1815bd00 0x7f9a18142490 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.699+0000 7f9a1fa00640 1 --2- 172.21.15.40:0/2148654959 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a180728d0 0x7f9a181417e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.699+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2148654959 >> 172.21.15.40:0/2148654959 conn(0x7f9a180b1660 msgr2=0x7f9a180cff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:56.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.699+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2148654959 shutdown_connections 2024-03-24T00:34:56.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.699+0000 7f9a1fa00640 1 -- 172.21.15.40:0/2148654959 wait complete. 2024-03-24T00:34:56.719 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:34:56.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.863+0000 7fd120400640 1 Processor -- start 2024-03-24T00:34:56.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.863+0000 7fd120400640 1 -- start start 2024-03-24T00:34:56.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.863+0000 7fd120400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd11815bd40 0x7fd1180d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.863+0000 7fd120400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd1180d1520 0x7fd1180cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.863+0000 7fd120400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11815f970 con 0x7fd11815b960 2024-03-24T00:34:56.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.863+0000 7fd120400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd118160580 con 0x7fd11815bd40 2024-03-24T00:34:56.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.863+0000 7fd120400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11815fc50 con 0x7fd1180d1520 2024-03-24T00:34:56.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.863+0000 7fd11e400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd1180d1520 0x7fd1180cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.863+0000 7fd11da00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd11815b960 0x7fd118070d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:42296/0 (socket says 172.21.15.40:42296) 2024-03-24T00:34:56.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.863+0000 7fd11da00640 1 -- 172.21.15.40:0/3418033291 learned_addr learned my addr 172.21.15.40:0/3418033291 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:56.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.863+0000 7fd11d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd11815bd40 0x7fd1180d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.867+0000 7fd117e00640 1 -- 172.21.15.40:0/3418033291 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051154035 0 0) 0x7fd11815f970 con 0x7fd11815b960 2024-03-24T00:34:56.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.867+0000 7fd11e400640 1 -- 172.21.15.40:0/3418033291 >> v2:172.21.15.53:3300/0 conn(0x7fd11815bd40 msgr2=0x7fd1180d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:56.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.867+0000 7fd11e400640 1 --2- 172.21.15.40:0/3418033291 >> v2:172.21.15.53:3300/0 conn(0x7fd11815bd40 0x7fd1180d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.867+0000 7fd11e400640 1 -- 172.21.15.40:0/3418033291 >> v1:172.21.15.53:6789/0 conn(0x7fd11815b960 legacy=0x7fd118070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.867+0000 7fd11e400640 1 -- 172.21.15.40:0/3418033291 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11815ff30 con 0x7fd1180d1520 2024-03-24T00:34:56.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.867+0000 7fd11d000640 1 --2- 172.21.15.40:0/3418033291 >> v2:172.21.15.53:3300/0 conn(0x7fd11815bd40 0x7fd1180d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.867+0000 7fd11e400640 1 --2- 172.21.15.40:0/3418033291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd1180d1520 0x7fd1180cda00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd110009d90 tx=0x7fd11001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1f831d413413abc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.867+0000 7fd117e00640 1 -- 172.21.15.40:0/3418033291 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1100b2070 con 0x7fd1180d1520 2024-03-24T00:34:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.867+0000 7fd117e00640 1 -- 172.21.15.40:0/3418033291 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1100a5830 con 0x7fd1180d1520 2024-03-24T00:34:56.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.867+0000 7fd117e00640 1 -- 172.21.15.40:0/3418033291 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1100a5c00 con 0x7fd1180d1520 2024-03-24T00:34:56.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.867+0000 7fd120400640 1 -- 172.21.15.40:0/3418033291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd1180d1520 msgr2=0x7fd1180cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.867+0000 7fd120400640 1 --2- 172.21.15.40:0/3418033291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd1180d1520 0x7fd1180cda00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd110009d90 tx=0x7fd11001c970 comp rx=0 tx=0).stop 2024-03-24T00:34:56.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 -- 172.21.15.40:0/3418033291 shutdown_connections 2024-03-24T00:34:56.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 --2- 172.21.15.40:0/3418033291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd1180d1520 0x7fd1180cda00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 --2- 172.21.15.40:0/3418033291 >> v2:172.21.15.53:3300/0 conn(0x7fd11815bd40 0x7fd1180d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 -- 172.21.15.40:0/3418033291 >> 172.21.15.40:0/3418033291 conn(0x7fd11806ecf0 msgr2=0x7fd11806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 -- 172.21.15.40:0/3418033291 shutdown_connections 2024-03-24T00:34:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 -- 172.21.15.40:0/3418033291 wait complete. 2024-03-24T00:34:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 Processor -- start 2024-03-24T00:34:56.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 -- start start 2024-03-24T00:34:56.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1180d1520 0x7fd11806c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd11815b960 0x7fd1180c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11815bd40 0x7fd11806cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1180ceb20 con 0x7fd1180d1520 2024-03-24T00:34:56.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1180ce9a0 con 0x7fd11815bd40 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd120400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1180ceca0 con 0x7fd11815b960 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd11da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1180d1520 0x7fd11806c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd11d000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd11815b960 0x7fd1180c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd11da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1180d1520 0x7fd11806c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:33814/0 (socket says 172.21.15.40:33814) 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd11da00640 1 -- 172.21.15.40:0/3978404154 learned_addr learned my addr 172.21.15.40:0/3978404154 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd11e400640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11815bd40 0x7fd11806cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd11da00640 1 -- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd11815b960 msgr2=0x7fd1180c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd11da00640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd11815b960 0x7fd1180c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd11da00640 1 -- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11815bd40 msgr2=0x7fd11806cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd11da00640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11815bd40 0x7fd11806cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd11da00640 1 -- 172.21.15.40:0/3978404154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11806bc00 con 0x7fd1180d1520 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd11d000640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd11815b960 0x7fd1180c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:56.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.871+0000 7fd11da00640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1180d1520 0x7fd11806c0e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00c4d0 tx=0x7fd10c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.875+0000 7fd115600640 1 -- 172.21.15.40:0/3978404154 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c018070 con 0x7fd1180d1520 2024-03-24T00:34:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.875+0000 7fd115600640 1 -- 172.21.15.40:0/3978404154 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10c0074e0 con 0x7fd1180d1520 2024-03-24T00:34:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.875+0000 7fd120400640 1 -- 172.21.15.40:0/3978404154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11814fcc0 con 0x7fd1180d1520 2024-03-24T00:34:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.875+0000 7fd115600640 1 -- 172.21.15.40:0/3978404154 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c004a30 con 0x7fd1180d1520 2024-03-24T00:34:56.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.875+0000 7fd120400640 1 -- 172.21.15.40:0/3978404154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1180c9330 con 0x7fd1180d1520 2024-03-24T00:34:56.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.875+0000 7fd120400640 1 -- 172.21.15.40:0/3978404154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd118068170 con 0x7fd1180d1520 2024-03-24T00:34:56.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.875+0000 7fd115600640 1 -- 172.21.15.40:0/3978404154 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd10c01b070 con 0x7fd1180d1520 2024-03-24T00:34:56.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.875+0000 7fd115600640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd0f8067660 0x7fd0f8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:56.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.875+0000 7fd115600640 1 -- 172.21.15.40:0/3978404154 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd10c084b60 con 0x7fd1180d1520 2024-03-24T00:34:56.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.879+0000 7fd11d000640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd0f8067660 0x7fd0f8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:56.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.879+0000 7fd11d000640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd0f8067660 0x7fd0f8069b20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd1180ca520 tx=0x7fd104009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:56.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:56.879+0000 7fd115600640 1 -- 172.21.15.40:0/3978404154 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd10c055190 con 0x7fd1180d1520 2024-03-24T00:34:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 -- 172.21.15.40:0/3978404154 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd11806bdd0 con 0x7fd1180d1520 2024-03-24T00:34:57.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd115600640 1 -- 172.21.15.40:0/3978404154 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd10c059230 con 0x7fd1180d1520 2024-03-24T00:34:57.049 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:34:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 -- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd0f8067660 msgr2=0x7fd0f8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd0f8067660 0x7fd0f8069b20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fd1180ca520 tx=0x7fd104009190 comp rx=0 tx=0).stop 2024-03-24T00:34:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 -- 172.21.15.40:0/3978404154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1180d1520 msgr2=0x7fd11806c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1180d1520 0x7fd11806c0e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00c4d0 tx=0x7fd10c00c990 comp rx=0 tx=0).stop 2024-03-24T00:34:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 -- 172.21.15.40:0/3978404154 shutdown_connections 2024-03-24T00:34:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd0f8067660 0x7fd0f8069b20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd11815bd40 0x7fd11806cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd11815b960 0x7fd1180c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 --2- 172.21.15.40:0/3978404154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd1180d1520 0x7fd11806c0e0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 -- 172.21.15.40:0/3978404154 >> 172.21.15.40:0/3978404154 conn(0x7fd11806ecf0 msgr2=0x7fd11815cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 -- 172.21.15.40:0/3978404154 shutdown_connections 2024-03-24T00:34:57.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.047+0000 7fd120400640 1 -- 172.21.15.40:0/3978404154 wait complete. 2024-03-24T00:34:57.070 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:34:57.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.211+0000 7fc2f4e00640 1 Processor -- start 2024-03-24T00:34:57.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.211+0000 7fc2f4e00640 1 -- start start 2024-03-24T00:34:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.211+0000 7fc2f4e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc2f015bd00 0x7fc2f00ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:57.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.211+0000 7fc2f4e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f00728d0 0x7fc2f00cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:57.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.211+0000 7fc2f4e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f0072d30 con 0x7fc2f015f8c0 2024-03-24T00:34:57.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.211+0000 7fc2f4e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f00cf790 con 0x7fc2f015bd00 2024-03-24T00:34:57.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.211+0000 7fc2f4e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0071de0 con 0x7fc2f00728d0 2024-03-24T00:34:57.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.211+0000 7fc2ee800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc2f015f8c0 0x7fc2f006cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56376/0 (socket says 172.21.15.40:56376) 2024-03-24T00:34:57.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.211+0000 7fc2ede00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc2f015bd00 0x7fc2f00ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:57.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.211+0000 7fc2ee800640 1 -- 172.21.15.40:0/330996988 learned_addr learned my addr 172.21.15.40:0/330996988 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:57.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.211+0000 7fc2ed400640 1 --2- 172.21.15.40:0/330996988 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f00728d0 0x7fc2f00cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:57.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.215+0000 7fc2ede00640 1 -- 172.21.15.40:0/330996988 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f00728d0 msgr2=0x7fc2f00cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:57.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.215+0000 7fc2ede00640 1 --2- 172.21.15.40:0/330996988 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f00728d0 0x7fc2f00cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.215+0000 7fc2ede00640 1 -- 172.21.15.40:0/330996988 >> v1:172.21.15.40:6789/0 conn(0x7fc2f015f8c0 legacy=0x7fc2f006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.215+0000 7fc2ede00640 1 -- 172.21.15.40:0/330996988 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f00cea50 con 0x7fc2f015bd00 2024-03-24T00:34:57.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.215+0000 7fc2ede00640 1 --2- 172.21.15.40:0/330996988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f015bd00 0x7fc2f00ae8e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc2e00097c0 tx=0x7fc2e001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3893c5b4f8e382a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:57.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.215+0000 7fc2eca00640 1 -- 172.21.15.40:0/330996988 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e00a7070 con 0x7fc2f015bd00 2024-03-24T00:34:57.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.215+0000 7fc2eca00640 1 -- 172.21.15.40:0/330996988 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e00b0070 con 0x7fc2f015bd00 2024-03-24T00:34:57.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.215+0000 7fc2eca00640 1 -- 172.21.15.40:0/330996988 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e00ab690 con 0x7fc2f015bd00 2024-03-24T00:34:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.215+0000 7fc2f4e00640 1 -- 172.21.15.40:0/330996988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f015bd00 msgr2=0x7fc2f00ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.215+0000 7fc2f4e00640 1 --2- 172.21.15.40:0/330996988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f015bd00 0x7fc2f00ae8e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc2e00097c0 tx=0x7fc2e001c470 comp rx=0 tx=0).stop 2024-03-24T00:34:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.215+0000 7fc2f4e00640 1 -- 172.21.15.40:0/330996988 shutdown_connections 2024-03-24T00:34:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.215+0000 7fc2f4e00640 1 --2- 172.21.15.40:0/330996988 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f00728d0 0x7fc2f00cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2f4e00640 1 --2- 172.21.15.40:0/330996988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f015bd00 0x7fc2f00ae8e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2f4e00640 1 -- 172.21.15.40:0/330996988 >> 172.21.15.40:0/330996988 conn(0x7fc2f00b1660 msgr2=0x7fc2f00b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2f4e00640 1 -- 172.21.15.40:0/330996988 shutdown_connections 2024-03-24T00:34:57.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2f4e00640 1 -- 172.21.15.40:0/330996988 wait complete. 2024-03-24T00:34:57.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2f4e00640 1 Processor -- start 2024-03-24T00:34:57.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2f4e00640 1 -- start start 2024-03-24T00:34:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2f4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f00728d0 0x7fc2f014a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2f4e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f015bd00 0x7fc2f00a9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:57.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2f4e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f015f8c0 0x7fc2f00a9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2f4e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f015c0e0 con 0x7fc2f00728d0 2024-03-24T00:34:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2f4e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f006d630 con 0x7fc2f015bd00 2024-03-24T00:34:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2f4e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f01600e0 con 0x7fc2f015f8c0 2024-03-24T00:34:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2ede00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f00728d0 0x7fc2f014a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2ed400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f015bd00 0x7fc2f00a9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2ede00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f00728d0 0x7fc2f014a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46478/0 (socket says 172.21.15.40:46478) 2024-03-24T00:34:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2ede00640 1 -- 172.21.15.40:0/2640746933 learned_addr learned my addr 172.21.15.40:0/2640746933 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2ee800640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f015f8c0 0x7fc2f00a9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:57.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2ed400640 1 -- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f015f8c0 msgr2=0x7fc2f00a9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2ed400640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f015f8c0 0x7fc2f00a9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2ed400640 1 -- 172.21.15.40:0/2640746933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f00728d0 msgr2=0x7fc2f014a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2ed400640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f00728d0 0x7fc2f014a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2ed400640 1 -- 172.21.15.40:0/2640746933 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f014ba60 con 0x7fc2f015bd00 2024-03-24T00:34:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2ed400640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f015bd00 0x7fc2f00a9990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00e870 tx=0x7fc2dc00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.219+0000 7fc2e6000640 1 -- 172.21.15.40:0/2640746933 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc017070 con 0x7fc2f015bd00 2024-03-24T00:34:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.223+0000 7fc2e6000640 1 -- 172.21.15.40:0/2640746933 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc00c940 con 0x7fc2f015bd00 2024-03-24T00:34:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.223+0000 7fc2e6000640 1 -- 172.21.15.40:0/2640746933 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc00ccf0 con 0x7fc2f015bd00 2024-03-24T00:34:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.223+0000 7fc2f4e00640 1 -- 172.21.15.40:0/2640746933 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f014bd40 con 0x7fc2f015bd00 2024-03-24T00:34:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.223+0000 7fc2f4e00640 1 -- 172.21.15.40:0/2640746933 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f014c300 con 0x7fc2f015bd00 2024-03-24T00:34:57.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.223+0000 7fc2f4e00640 1 -- 172.21.15.40:0/2640746933 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f0068170 con 0x7fc2f015bd00 2024-03-24T00:34:57.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.223+0000 7fc2e6000640 1 -- 172.21.15.40:0/2640746933 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc2dc01a070 con 0x7fc2f015bd00 2024-03-24T00:34:57.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.223+0000 7fc2e6000640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc2d0067660 0x7fc2d0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:57.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.223+0000 7fc2e6000640 1 -- 172.21.15.40:0/2640746933 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2dc085780 con 0x7fc2f015bd00 2024-03-24T00:34:57.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.227+0000 7fc2ede00640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc2d0067660 0x7fc2d0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:57.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.227+0000 7fc2ede00640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc2d0067660 0x7fc2d0069b20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc2e001ab20 tx=0x7fc2e00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:57.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.231+0000 7fc2e6000640 1 -- 172.21.15.40:0/2640746933 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc2dc055db0 con 0x7fc2f015bd00 2024-03-24T00:34:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.391+0000 7fc2f4e00640 1 -- 172.21.15.40:0/2640746933 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc2f00b2820 con 0x7fc2f015bd00 2024-03-24T00:34:57.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.391+0000 7fc2e6000640 1 -- 172.21.15.40:0/2640746933 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc2dc059e50 con 0x7fc2f015bd00 2024-03-24T00:34:57.395 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:34:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.395+0000 7fc2f4e00640 1 -- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc2d0067660 msgr2=0x7fc2d0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.395+0000 7fc2f4e00640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc2d0067660 0x7fc2d0069b20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc2e001ab20 tx=0x7fc2e00a7000 comp rx=0 tx=0).stop 2024-03-24T00:34:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.395+0000 7fc2f4e00640 1 -- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f015bd00 msgr2=0x7fc2f00a9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.395+0000 7fc2f4e00640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f015bd00 0x7fc2f00a9990 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc00e870 tx=0x7fc2dc00ed30 comp rx=0 tx=0).stop 2024-03-24T00:34:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.395+0000 7fc2f4e00640 1 -- 172.21.15.40:0/2640746933 shutdown_connections 2024-03-24T00:34:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.395+0000 7fc2f4e00640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc2d0067660 0x7fc2d0069b20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.395+0000 7fc2f4e00640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f015f8c0 0x7fc2f00a9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.395+0000 7fc2f4e00640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f015bd00 0x7fc2f00a9990 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.395+0000 7fc2f4e00640 1 --2- 172.21.15.40:0/2640746933 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f00728d0 0x7fc2f014a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.395+0000 7fc2f4e00640 1 -- 172.21.15.40:0/2640746933 >> 172.21.15.40:0/2640746933 conn(0x7fc2f00b1660 msgr2=0x7fc2f00b0c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.395+0000 7fc2f4e00640 1 -- 172.21.15.40:0/2640746933 shutdown_connections 2024-03-24T00:34:57.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.395+0000 7fc2f4e00640 1 -- 172.21.15.40:0/2640746933 wait complete. 2024-03-24T00:34:57.426 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:34:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.563+0000 7fc7e9000640 1 Processor -- start 2024-03-24T00:34:57.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.563+0000 7fc7e9000640 1 -- start start 2024-03-24T00:34:57.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7e9000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 0x7fc7e40cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:57.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7e9000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc7e406b150 0x7fc7e406b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7e9000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e40d0e10 con 0x7fc7e415e0b0 2024-03-24T00:34:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7e9000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e40d10f0 con 0x7fc7e406b150 2024-03-24T00:34:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7e9000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e406efd0 con 0x7fc7e40d0830 2024-03-24T00:34:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7e1e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc7e415e0b0 0x7fc7e40cd3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56388/0 (socket says 172.21.15.40:56388) 2024-03-24T00:34:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7e1e00640 1 -- 172.21.15.40:0/985424414 learned_addr learned my addr 172.21.15.40:0/985424414 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7dbe00640 1 --2- 172.21.15.40:0/985424414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 0x7fc7e40cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7e2800640 1 --2- 172.21.15.40:0/985424414 >> v2:172.21.15.40:3300/0 conn(0x7fc7e406b150 0x7fc7e406b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7e1400640 1 -- 172.21.15.40:0/985424414 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213590187 0 0) 0x7fc7e40d0e10 con 0x7fc7e415e0b0 2024-03-24T00:34:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7e1400640 1 -- 172.21.15.40:0/985424414 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c8003530 con 0x7fc7e415e0b0 2024-03-24T00:34:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7dbe00640 1 -- 172.21.15.40:0/985424414 >> v2:172.21.15.40:3300/0 conn(0x7fc7e406b150 msgr2=0x7fc7e406b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7dbe00640 1 --2- 172.21.15.40:0/985424414 >> v2:172.21.15.40:3300/0 conn(0x7fc7e406b150 0x7fc7e406b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7dbe00640 1 -- 172.21.15.40:0/985424414 >> v1:172.21.15.40:6789/0 conn(0x7fc7e415e0b0 legacy=0x7fc7e40cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.567+0000 7fc7dbe00640 1 -- 172.21.15.40:0/985424414 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e406f2b0 con 0x7fc7e40d0830 2024-03-24T00:34:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.571+0000 7fc7dbe00640 1 --2- 172.21.15.40:0/985424414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 0x7fc7e40cdae0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097c0 tx=0x7fc7cc01c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f73cba5c8aef03fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.571+0000 7fc7e1400640 1 -- 172.21.15.40:0/985424414 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0b0070 con 0x7fc7e40d0830 2024-03-24T00:34:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.571+0000 7fc7e1400640 1 -- 172.21.15.40:0/985424414 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc0ab600 con 0x7fc7e40d0830 2024-03-24T00:34:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.571+0000 7fc7e1400640 1 -- 172.21.15.40:0/985424414 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0ab9d0 con 0x7fc7e40d0830 2024-03-24T00:34:57.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.571+0000 7fc7e9000640 1 -- 172.21.15.40:0/985424414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 msgr2=0x7fc7e40cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.571+0000 7fc7e9000640 1 --2- 172.21.15.40:0/985424414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 0x7fc7e40cdae0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097c0 tx=0x7fc7cc01c4c0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.571+0000 7fc7e9000640 1 -- 172.21.15.40:0/985424414 shutdown_connections 2024-03-24T00:34:57.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.571+0000 7fc7e9000640 1 --2- 172.21.15.40:0/985424414 >> v2:172.21.15.40:3300/0 conn(0x7fc7e406b150 0x7fc7e406b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.571+0000 7fc7e9000640 1 --2- 172.21.15.40:0/985424414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 0x7fc7e40cdae0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.571+0000 7fc7e9000640 1 -- 172.21.15.40:0/985424414 >> 172.21.15.40:0/985424414 conn(0x7fc7e40d1c90 msgr2=0x7fc7e40d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:57.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.571+0000 7fc7e9000640 1 -- 172.21.15.40:0/985424414 shutdown_connections 2024-03-24T00:34:57.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.571+0000 7fc7e9000640 1 -- 172.21.15.40:0/985424414 wait complete. 2024-03-24T00:34:57.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7e9000640 1 Processor -- start 2024-03-24T00:34:57.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7e9000640 1 -- start start 2024-03-24T00:34:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7e9000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7e406b150 0x7fc7e40b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7e9000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 0x7fc7e40c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7e9000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7e415e0b0 0x7fc7e414a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7e9000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e415e490 con 0x7fc7e415e0b0 2024-03-24T00:34:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7e9000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e40ce020 con 0x7fc7e406b150 2024-03-24T00:34:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7e9000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e415e610 con 0x7fc7e40d0830 2024-03-24T00:34:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7e2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7e415e0b0 0x7fc7e414a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:57.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7dbe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 0x7fc7e40c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7dbe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 0x7fc7e40c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:49074/0 (socket says 172.21.15.40:49074) 2024-03-24T00:34:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7dbe00640 1 -- 172.21.15.40:0/179779793 learned_addr learned my addr 172.21.15.40:0/179779793 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7e1e00640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7e406b150 0x7fc7e40b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7dbe00640 1 -- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7e406b150 msgr2=0x7fc7e40b3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7dbe00640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7e406b150 0x7fc7e40b3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7dbe00640 1 -- 172.21.15.40:0/179779793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7e415e0b0 msgr2=0x7fc7e414a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7dbe00640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7e415e0b0 0x7fc7e414a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7dbe00640 1 -- 172.21.15.40:0/179779793 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e40c9cb0 con 0x7fc7e40d0830 2024-03-24T00:34:57.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7dbe00640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 0x7fc7e40c9770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0ae040 tx=0x7fc7cc019260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:57.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7da000640 1 -- 172.21.15.40:0/179779793 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0b8020 con 0x7fc7e40d0830 2024-03-24T00:34:57.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7da000640 1 -- 172.21.15.40:0/179779793 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc0b4410 con 0x7fc7e40d0830 2024-03-24T00:34:57.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7da000640 1 -- 172.21.15.40:0/179779793 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc0b0040 con 0x7fc7e40d0830 2024-03-24T00:34:57.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.575+0000 7fc7e9000640 1 -- 172.21.15.40:0/179779793 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e40c9f80 con 0x7fc7e40d0830 2024-03-24T00:34:57.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.579+0000 7fc7e9000640 1 -- 172.21.15.40:0/179779793 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e40ca540 con 0x7fc7e40d0830 2024-03-24T00:34:57.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.579+0000 7fc7e9000640 1 -- 172.21.15.40:0/179779793 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e4068170 con 0x7fc7e40d0830 2024-03-24T00:34:57.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.579+0000 7fc7da000640 1 -- 172.21.15.40:0/179779793 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc7cc0bb020 con 0x7fc7e40d0830 2024-03-24T00:34:57.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.579+0000 7fc7da000640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7c0067690 0x7fc7c0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:57.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.579+0000 7fc7da000640 1 -- 172.21.15.40:0/179779793 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7cc11f160 con 0x7fc7e40d0830 2024-03-24T00:34:57.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.579+0000 7fc7e1e00640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7c0067690 0x7fc7c0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:57.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.583+0000 7fc7e1e00640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7c0067690 0x7fc7c0069b50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4007fd0 tx=0x7fc7d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:57.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.587+0000 7fc7da000640 1 -- 172.21.15.40:0/179779793 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc7cc0c2030 con 0x7fc7e40d0830 2024-03-24T00:34:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.759+0000 7fc7e9000640 1 -- 172.21.15.40:0/179779793 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc7e406ddc0 con 0x7fc7e40d0830 2024-03-24T00:34:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.759+0000 7fc7da000640 1 -- 172.21.15.40:0/179779793 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v59) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc7cc0160b0 con 0x7fc7e40d0830 2024-03-24T00:34:57.763 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:34:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.763+0000 7fc7e9000640 1 -- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7c0067690 msgr2=0x7fc7c0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.763+0000 7fc7e9000640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7c0067690 0x7fc7c0069b50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4007fd0 tx=0x7fc7d4009040 comp rx=0 tx=0).stop 2024-03-24T00:34:57.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.763+0000 7fc7e9000640 1 -- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 msgr2=0x7fc7e40c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.763+0000 7fc7e9000640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 0x7fc7e40c9770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0ae040 tx=0x7fc7cc019260 comp rx=0 tx=0).stop 2024-03-24T00:34:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.763+0000 7fc7e9000640 1 -- 172.21.15.40:0/179779793 shutdown_connections 2024-03-24T00:34:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.763+0000 7fc7e9000640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7c0067690 0x7fc7c0069b50 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.763+0000 7fc7e9000640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7e415e0b0 0x7fc7e414a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.763+0000 7fc7e9000640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7e40d0830 0x7fc7e40c9770 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.763+0000 7fc7e9000640 1 --2- 172.21.15.40:0/179779793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7e406b150 0x7fc7e40b3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.763+0000 7fc7e9000640 1 -- 172.21.15.40:0/179779793 >> 172.21.15.40:0/179779793 conn(0x7fc7e40d1c90 msgr2=0x7fc7e40b6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.763+0000 7fc7e9000640 1 -- 172.21.15.40:0/179779793 shutdown_connections 2024-03-24T00:34:57.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:57.763+0000 7fc7e9000640 1 -- 172.21.15.40:0/179779793 wait complete. 2024-03-24T00:34:57.785 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-03-24T00:34:57.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:57.785 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T00:34:57.804 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[42300]: fuse finished with error 0 and tester_r 0 2024-03-24T00:34:57.809 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:34:57.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:57.809 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:34:58.002 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:34:58.002 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:34:58.011 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:34:58.011 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:34:58.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:58.011 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T00:34:58.061 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T00:34:58.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:34:58.062 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:34:58.062 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:34:58.071 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:34:58.071 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:34:58.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:58.071 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T00:34:58.123 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T00:34:58.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:34:58.124 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:34:58.125 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:34:58.132 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:34:58.133 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:34:58.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:34:58.133 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T00:34:58.183 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T00:34:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:34:58.184 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:34:58.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.323+0000 7f46d3200640 1 Processor -- start 2024-03-24T00:34:58.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.323+0000 7f46d3200640 1 -- start start 2024-03-24T00:34:58.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46d3200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc06e5a0 0x7f46cc06e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46d3200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f46cc070c80 0x7f46cc06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46d3200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46cc06cdc0 con 0x7f46cc071690 2024-03-24T00:34:58.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46d3200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46cc06eec0 con 0x7f46cc070c80 2024-03-24T00:34:58.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46d3200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc071160 con 0x7f46cc06e5a0 2024-03-24T00:34:58.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46d1200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f46cc071690 0x7f46cc0b33c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56392/0 (socket says 172.21.15.40:56392) 2024-03-24T00:34:58.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46d1200640 1 -- 172.21.15.40:0/2079480626 learned_addr learned my addr 172.21.15.40:0/2079480626 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:58.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46cbe00640 1 --2- 172.21.15.40:0/2079480626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc06e5a0 0x7f46cc06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46c3e00640 1 --2- 172.21.15.40:0/2079480626 >> v2:172.21.15.53:3300/0 conn(0x7f46cc070c80 0x7f46cc06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46cbe00640 1 -- 172.21.15.40:0/2079480626 >> v2:172.21.15.53:3300/0 conn(0x7f46cc070c80 msgr2=0x7f46cc06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:58.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46cbe00640 1 --2- 172.21.15.40:0/2079480626 >> v2:172.21.15.53:3300/0 conn(0x7f46cc070c80 0x7f46cc06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46cbe00640 1 -- 172.21.15.40:0/2079480626 >> v1:172.21.15.40:6789/0 conn(0x7f46cc071690 legacy=0x7f46cc0b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46cbe00640 1 -- 172.21.15.40:0/2079480626 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc06d040 con 0x7f46cc06e5a0 2024-03-24T00:34:58.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46cb400640 1 -- 172.21.15.40:0/2079480626 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883433509 0 0) 0x7f46c40014a0 con 0x7f46cc071690 2024-03-24T00:34:58.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.327+0000 7f46cbe00640 1 --2- 172.21.15.40:0/2079480626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc06e5a0 0x7f46cc06e980 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0097c0 tx=0x7f46bc01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c62bbb31a993649 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:58.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46cb400640 1 -- 172.21.15.40:0/2079480626 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc0b0070 con 0x7f46cc06e5a0 2024-03-24T00:34:58.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46cb400640 1 -- 172.21.15.40:0/2079480626 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46bc0ab600 con 0x7f46cc06e5a0 2024-03-24T00:34:58.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46cb400640 1 -- 172.21.15.40:0/2079480626 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc0ab9d0 con 0x7f46cc06e5a0 2024-03-24T00:34:58.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 -- 172.21.15.40:0/2079480626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc06e5a0 msgr2=0x7f46cc06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 --2- 172.21.15.40:0/2079480626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc06e5a0 0x7f46cc06e980 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0097c0 tx=0x7f46bc01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 -- 172.21.15.40:0/2079480626 shutdown_connections 2024-03-24T00:34:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 --2- 172.21.15.40:0/2079480626 >> v2:172.21.15.53:3300/0 conn(0x7f46cc070c80 0x7f46cc06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 --2- 172.21.15.40:0/2079480626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc06e5a0 0x7f46cc06e980 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 -- 172.21.15.40:0/2079480626 >> 172.21.15.40:0/2079480626 conn(0x7f46cc0b17b0 msgr2=0x7f46cc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 -- 172.21.15.40:0/2079480626 shutdown_connections 2024-03-24T00:34:58.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 -- 172.21.15.40:0/2079480626 wait complete. 2024-03-24T00:34:58.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 Processor -- start 2024-03-24T00:34:58.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 -- start start 2024-03-24T00:34:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46cc06e5a0 0x7f46cc144210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc070c80 0x7f46cc151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46cc071690 0x7f46cc151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc16b8c0 con 0x7f46cc06e5a0 2024-03-24T00:34:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc16b740 con 0x7f46cc071690 2024-03-24T00:34:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d3200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc16ba40 con 0x7f46cc070c80 2024-03-24T00:34:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46c3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc070c80 0x7f46cc151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46cc06e5a0 0x7f46cc144210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46c3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc070c80 0x7f46cc151130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:49098/0 (socket says 172.21.15.40:49098) 2024-03-24T00:34:58.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46c3e00640 1 -- 172.21.15.40:0/1911422651 learned_addr learned my addr 172.21.15.40:0/1911422651 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:58.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.331+0000 7f46d1200640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46cc071690 0x7f46cc151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46c3e00640 1 -- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46cc071690 msgr2=0x7f46cc151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46c3e00640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46cc071690 0x7f46cc151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46c3e00640 1 -- 172.21.15.40:0/1911422651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46cc06e5a0 msgr2=0x7f46cc144210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46c3e00640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46cc06e5a0 0x7f46cc144210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46c3e00640 1 -- 172.21.15.40:0/1911422651 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc144a90 con 0x7f46cc070c80 2024-03-24T00:34:58.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46c3e00640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc070c80 0x7f46cc151130 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f46b400e870 tx=0x7f46b400ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:58.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46c8c00640 1 -- 172.21.15.40:0/1911422651 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b4017070 con 0x7f46cc070c80 2024-03-24T00:34:58.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46c8c00640 1 -- 172.21.15.40:0/1911422651 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b400c940 con 0x7f46cc070c80 2024-03-24T00:34:58.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46c8c00640 1 -- 172.21.15.40:0/1911422651 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b400ccf0 con 0x7f46cc070c80 2024-03-24T00:34:58.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46d3200640 1 -- 172.21.15.40:0/1911422651 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc0af950 con 0x7f46cc070c80 2024-03-24T00:34:58.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46d3200640 1 -- 172.21.15.40:0/1911422651 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc0aff10 con 0x7f46cc070c80 2024-03-24T00:34:58.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46d3200640 1 -- 172.21.15.40:0/1911422651 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc0af6c0 con 0x7f46cc070c80 2024-03-24T00:34:58.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46c8c00640 1 -- 172.21.15.40:0/1911422651 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f46b401a070 con 0x7f46cc070c80 2024-03-24T00:34:58.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46c8c00640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46a8067660 0x7f46a8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.335+0000 7f46c8c00640 1 -- 172.21.15.40:0/1911422651 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f46b4085ab0 con 0x7f46cc070c80 2024-03-24T00:34:58.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.339+0000 7f46cbe00640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46a8067660 0x7f46a8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.339+0000 7f46cbe00640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46a8067660 0x7f46a8069b20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0ad000 tx=0x7f46bc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:58.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.343+0000 7f46c8c00640 1 -- 172.21.15.40:0/1911422651 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f46b40560d0 con 0x7f46cc070c80 2024-03-24T00:34:58.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 -- 172.21.15.40:0/1911422651 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46cc0cf4e0 con 0x7f46cc070c80 2024-03-24T00:34:58.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46c8c00640 1 -- 172.21.15.40:0/1911422651 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5015 (secure 0 0 0) 0x7f46b405a170 con 0x7f46cc070c80 2024-03-24T00:34:58.545 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:58.545 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":58,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6277,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1633817617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1633817617},{"type":"v1","addr":"172.21.15.53:6837","nonce":1633817617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:34:12.019118+0000","modified":"2024-03-24T00:34:35.695160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6474,"mds_1":6471,"mds_2":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":2,"incarnation":56,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6834/1278515461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1278515461},{"type":"v1","addr":"172.21.15.40:6834","nonce":1278515461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6471":{"gid":6471,"name":"c","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6835/1312292485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1312292485},{"type":"v1","addr":"172.21.15.40:6835","nonce":1312292485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6474":{"gid":6474,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/501079401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":501079401},{"type":"v1","addr":"172.21.15.53:6835","nonce":501079401}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6474,"qdb_cluster":[6474,6471,6468]},"id":4}]} 2024-03-24T00:34:58.546 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 58 2024-03-24T00:34:58.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 -- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46a8067660 msgr2=0x7f46a8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46a8067660 0x7f46a8069b20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f46bc0ad000 tx=0x7f46bc0a6000 comp rx=0 tx=0).stop 2024-03-24T00:34:58.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 -- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc070c80 msgr2=0x7f46cc151130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc070c80 0x7f46cc151130 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f46b400e870 tx=0x7f46b400ec70 comp rx=0 tx=0).stop 2024-03-24T00:34:58.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 -- 172.21.15.40:0/1911422651 shutdown_connections 2024-03-24T00:34:58.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46a8067660 0x7f46a8069b20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46cc071690 0x7f46cc151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46cc070c80 0x7f46cc151130 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 --2- 172.21.15.40:0/1911422651 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46cc06e5a0 0x7f46cc144210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 -- 172.21.15.40:0/1911422651 >> 172.21.15.40:0/1911422651 conn(0x7f46cc0b17b0 msgr2=0x7f46cc0ae780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:58.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 -- 172.21.15.40:0/1911422651 shutdown_connections 2024-03-24T00:34:58.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.543+0000 7f46d3200640 1 -- 172.21.15.40:0/1911422651 wait complete. 2024-03-24T00:34:58.566 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:34:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a9e000640 1 Processor -- start 2024-03-24T00:34:58.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a9e000640 1 -- start start 2024-03-24T00:34:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a9e000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5a9813ec80 0x7f5a981414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a9e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a980ce800 0x7f5a98071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a9e000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a98070e00 con 0x7f5a981419f0 2024-03-24T00:34:58.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a9e000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a98141e50 con 0x7f5a980ce800 2024-03-24T00:34:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a9e000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a980cee20 con 0x7f5a9813ec80 2024-03-24T00:34:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a97400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5a9813ec80 0x7f5a981414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a97e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5a980ce800 0x7f5a98071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a8ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5a981419f0 0x7f5a980ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56396/0 (socket says 172.21.15.40:56396) 2024-03-24T00:34:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a8ea00640 1 -- 172.21.15.40:0/2198496903 learned_addr learned my addr 172.21.15.40:0/2198496903 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a97400640 1 -- 172.21.15.40:0/2198496903 >> v2:172.21.15.40:3300/0 conn(0x7f5a980ce800 msgr2=0x7f5a98071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a97400640 1 --2- 172.21.15.40:0/2198496903 >> v2:172.21.15.40:3300/0 conn(0x7f5a980ce800 0x7f5a98071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a97400640 1 -- 172.21.15.40:0/2198496903 >> v1:172.21.15.40:6789/0 conn(0x7f5a981419f0 legacy=0x7f5a980ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.707+0000 7f5a97400640 1 -- 172.21.15.40:0/2198496903 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a980cf100 con 0x7f5a9813ec80 2024-03-24T00:34:58.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.711+0000 7f5a97400640 1 --2- 172.21.15.40:0/2198496903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a9813ec80 0x7f5a981414b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5a9806b170 tx=0x7f5a8801c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e99102ad384d0eb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.711+0000 7f5a96a00640 1 -- 172.21.15.40:0/2198496903 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a880b2070 con 0x7f5a9813ec80 2024-03-24T00:34:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.711+0000 7f5a96a00640 1 -- 172.21.15.40:0/2198496903 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a880a5a10 con 0x7f5a9813ec80 2024-03-24T00:34:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.711+0000 7f5a96a00640 1 -- 172.21.15.40:0/2198496903 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a880a5de0 con 0x7f5a9813ec80 2024-03-24T00:34:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.711+0000 7f5a9e000640 1 -- 172.21.15.40:0/2198496903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a9813ec80 msgr2=0x7f5a981414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.711+0000 7f5a9e000640 1 --2- 172.21.15.40:0/2198496903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a9813ec80 0x7f5a981414b0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5a9806b170 tx=0x7f5a8801c930 comp rx=0 tx=0).stop 2024-03-24T00:34:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.711+0000 7f5a9e000640 1 -- 172.21.15.40:0/2198496903 shutdown_connections 2024-03-24T00:34:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.711+0000 7f5a9e000640 1 --2- 172.21.15.40:0/2198496903 >> v2:172.21.15.40:3300/0 conn(0x7f5a980ce800 0x7f5a98071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.711+0000 7f5a9e000640 1 --2- 172.21.15.40:0/2198496903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a9813ec80 0x7f5a981414b0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.711+0000 7f5a9e000640 1 -- 172.21.15.40:0/2198496903 >> 172.21.15.40:0/2198496903 conn(0x7f5a980b17b0 msgr2=0x7f5a980b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.711+0000 7f5a9e000640 1 -- 172.21.15.40:0/2198496903 shutdown_connections 2024-03-24T00:34:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.711+0000 7f5a9e000640 1 -- 172.21.15.40:0/2198496903 wait complete. 2024-03-24T00:34:58.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a9e000640 1 Processor -- start 2024-03-24T00:34:58.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a9e000640 1 -- start start 2024-03-24T00:34:58.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a9e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a980ce800 0x7f5a98151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a9e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a9813ec80 0x7f5a98154c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a9e000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a981419f0 0x7f5a981551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a9e000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a980723e0 con 0x7f5a9813ec80 2024-03-24T00:34:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a9e000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a98072260 con 0x7f5a980ce800 2024-03-24T00:34:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a9e000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a98072560 con 0x7f5a981419f0 2024-03-24T00:34:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a97400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a980ce800 0x7f5a98151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a8ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a9813ec80 0x7f5a98154c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a97e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a981419f0 0x7f5a981551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a97400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a980ce800 0x7f5a98151080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:42974/0 (socket says 172.21.15.40:42974) 2024-03-24T00:34:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a97400640 1 -- 172.21.15.40:0/2534126248 learned_addr learned my addr 172.21.15.40:0/2534126248 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:58.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a97e00640 1 -- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a980ce800 msgr2=0x7f5a98151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a97e00640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a980ce800 0x7f5a98151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a97e00640 1 -- 172.21.15.40:0/2534126248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a9813ec80 msgr2=0x7f5a98154c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a97e00640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a9813ec80 0x7f5a98154c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a97e00640 1 -- 172.21.15.40:0/2534126248 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9813bfb0 con 0x7f5a981419f0 2024-03-24T00:34:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a97400640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a980ce800 0x7f5a98151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a97e00640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a981419f0 0x7f5a981551c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5a9000e7f0 tx=0x7f5a9000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a8fe00640 1 -- 172.21.15.40:0/2534126248 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a90017070 con 0x7f5a981419f0 2024-03-24T00:34:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a8fe00640 1 -- 172.21.15.40:0/2534126248 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9000c6d0 con 0x7f5a981419f0 2024-03-24T00:34:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a8fe00640 1 -- 172.21.15.40:0/2534126248 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9000caa0 con 0x7f5a981419f0 2024-03-24T00:34:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a9e000640 1 -- 172.21.15.40:0/2534126248 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9813c290 con 0x7f5a981419f0 2024-03-24T00:34:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.715+0000 7f5a9e000640 1 -- 172.21.15.40:0/2534126248 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9813c850 con 0x7f5a981419f0 2024-03-24T00:34:58.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.719+0000 7f5a9e000640 1 -- 172.21.15.40:0/2534126248 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a98068170 con 0x7f5a981419f0 2024-03-24T00:34:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.719+0000 7f5a8fe00640 1 -- 172.21.15.40:0/2534126248 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5a9001a070 con 0x7f5a981419f0 2024-03-24T00:34:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.719+0000 7f5a8fe00640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5a74067660 0x7f5a74069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.719+0000 7f5a8fe00640 1 -- 172.21.15.40:0/2534126248 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a90085670 con 0x7f5a981419f0 2024-03-24T00:34:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.719+0000 7f5a97400640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5a74067660 0x7f5a74069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.719+0000 7f5a97400640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5a74067660 0x7f5a74069b20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5a88000f90 tx=0x7f5a880a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:58.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.723+0000 7f5a8fe00640 1 -- 172.21.15.40:0/2534126248 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5a90055c00 con 0x7f5a981419f0 2024-03-24T00:34:58.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.887+0000 7f5a9e000640 1 -- 172.21.15.40:0/2534126248 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a980b35f0 con 0x7f5a981419f0 2024-03-24T00:34:58.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a8fe00640 1 -- 172.21.15.40:0/2534126248 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f5a90059ca0 con 0x7f5a981419f0 2024-03-24T00:34:58.892 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:34:58.892 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":59,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:34:13.623804+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:34:10.428162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-24T00:34:11.001094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:34:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a9e000640 1 -- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5a74067660 msgr2=0x7f5a74069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a9e000640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5a74067660 0x7f5a74069b20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5a88000f90 tx=0x7f5a880a7070 comp rx=0 tx=0).stop 2024-03-24T00:34:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a9e000640 1 -- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a981419f0 msgr2=0x7f5a981551c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a9e000640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a981419f0 0x7f5a981551c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5a9000e7f0 tx=0x7f5a9000ecb0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a9e000640 1 -- 172.21.15.40:0/2534126248 shutdown_connections 2024-03-24T00:34:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a9e000640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5a74067660 0x7f5a74069b20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a9e000640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a981419f0 0x7f5a981551c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a9e000640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5a9813ec80 0x7f5a98154c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a9e000640 1 --2- 172.21.15.40:0/2534126248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a980ce800 0x7f5a98151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:58.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a9e000640 1 -- 172.21.15.40:0/2534126248 >> 172.21.15.40:0/2534126248 conn(0x7f5a980b17b0 msgr2=0x7f5a98156290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:58.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a9e000640 1 -- 172.21.15.40:0/2534126248 shutdown_connections 2024-03-24T00:34:58.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:58.891+0000 7f5a9e000640 1 -- 172.21.15.40:0/2534126248 wait complete. 2024-03-24T00:34:58.923 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:34:10.428162+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:34:11.001094+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '59', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:34:58.923 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:34:59.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.063+0000 7f7535400640 1 Processor -- start 2024-03-24T00:34:59.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.063+0000 7f7535400640 1 -- start start 2024-03-24T00:34:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.067+0000 7f7535400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75300cc4c0 0x7f75300d0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.067+0000 7f7535400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75300b33e0 con 0x7f75300d0750 2024-03-24T00:34:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.067+0000 7f7535400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75300b22f0 con 0x7f753015f5b0 2024-03-24T00:34:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.067+0000 7f7535400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75300afa90 con 0x7f75300cc4c0 2024-03-24T00:34:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.067+0000 7f7527e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75300cc4c0 0x7f75300d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.067+0000 7f752fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f753015f5b0 0x7f75300cfb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:49360/0 (socket says 172.21.15.40:49360) 2024-03-24T00:34:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.067+0000 7f752fe00640 1 -- 172.21.15.40:0/1770862447 learned_addr learned my addr 172.21.15.40:0/1770862447 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.067+0000 7f7527e00640 1 -- 172.21.15.40:0/1770862447 >> v1:172.21.15.53:6789/0 conn(0x7f753015f5b0 legacy=0x7f75300cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.067+0000 7f7527e00640 1 -- 172.21.15.40:0/1770862447 >> v1:172.21.15.40:6789/0 conn(0x7f75300d0750 legacy=0x7f7530165b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.067+0000 7f7527e00640 1 -- 172.21.15.40:0/1770862447 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75300af440 con 0x7f75300cc4c0 2024-03-24T00:34:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f7527e00640 1 --2- 172.21.15.40:0/1770862447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75300cc4c0 0x7f75300d0210 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f75180097f0 tx=0x7f751801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea8f1e7b89fbc4ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f752f400640 1 -- 172.21.15.40:0/1770862447 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75180b0070 con 0x7f75300cc4c0 2024-03-24T00:34:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f752f400640 1 -- 172.21.15.40:0/1770862447 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75180ab600 con 0x7f75300cc4c0 2024-03-24T00:34:59.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f752f400640 1 -- 172.21.15.40:0/1770862447 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75180ab9d0 con 0x7f75300cc4c0 2024-03-24T00:34:59.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f7535400640 1 -- 172.21.15.40:0/1770862447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75300cc4c0 msgr2=0x7f75300d0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f7535400640 1 --2- 172.21.15.40:0/1770862447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75300cc4c0 0x7f75300d0210 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f75180097f0 tx=0x7f751801c4e0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f7535400640 1 -- 172.21.15.40:0/1770862447 shutdown_connections 2024-03-24T00:34:59.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f7535400640 1 --2- 172.21.15.40:0/1770862447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75300cc4c0 0x7f75300d0210 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f7535400640 1 -- 172.21.15.40:0/1770862447 >> 172.21.15.40:0/1770862447 conn(0x7f75300b3620 msgr2=0x7f75300b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:59.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f7535400640 1 -- 172.21.15.40:0/1770862447 shutdown_connections 2024-03-24T00:34:59.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f7535400640 1 -- 172.21.15.40:0/1770862447 wait complete. 2024-03-24T00:34:59.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f7535400640 1 Processor -- start 2024-03-24T00:34:59.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.071+0000 7f7535400640 1 -- start start 2024-03-24T00:34:59.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f7535400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75300cc4c0 0x7f75300b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f7535400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75300d0750 0x7f75300b64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f7535400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f753015f5b0 0x7f75300b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f7535400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7530169e20 con 0x7f75300cc4c0 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f7535400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7530169ca0 con 0x7f75300d0750 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f7535400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7530169fa0 con 0x7f753015f5b0 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75300cc4c0 0x7f75300b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f7527e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75300d0750 0x7f75300b64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75300cc4c0 0x7f75300b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46522/0 (socket says 172.21.15.40:46522) 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752fe00640 1 -- 172.21.15.40:0/1206250137 learned_addr learned my addr 172.21.15.40:0/1206250137 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f7534a00640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f753015f5b0 0x7f75300b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752fe00640 1 -- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f753015f5b0 msgr2=0x7f75300b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752fe00640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f753015f5b0 0x7f75300b69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752fe00640 1 -- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75300d0750 msgr2=0x7f75300b64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752fe00640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75300d0750 0x7f75300b64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752fe00640 1 -- 172.21.15.40:0/1206250137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75300b7210 con 0x7f75300cc4c0 2024-03-24T00:34:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f7527e00640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75300d0750 0x7f75300b64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752fe00640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75300cc4c0 0x7f75300b46d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f752000c400 tx=0x7f752000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752cc00640 1 -- 172.21.15.40:0/1206250137 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520017070 con 0x7f75300cc4c0 2024-03-24T00:34:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752cc00640 1 -- 172.21.15.40:0/1206250137 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7520005cf0 con 0x7f75300cc4c0 2024-03-24T00:34:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752cc00640 1 -- 172.21.15.40:0/1206250137 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75200072d0 con 0x7f75300cc4c0 2024-03-24T00:34:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f7535400640 1 -- 172.21.15.40:0/1206250137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75300b74f0 con 0x7f75300cc4c0 2024-03-24T00:34:59.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f7535400640 1 -- 172.21.15.40:0/1206250137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75300c8fe0 con 0x7f75300cc4c0 2024-03-24T00:34:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f7535400640 1 -- 172.21.15.40:0/1206250137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f753004f410 con 0x7f75300cc4c0 2024-03-24T00:34:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.075+0000 7f752cc00640 1 -- 172.21.15.40:0/1206250137 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f752001a070 con 0x7f75300cc4c0 2024-03-24T00:34:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.079+0000 7f752cc00640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f750c067660 0x7f750c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.079+0000 7f752cc00640 1 -- 172.21.15.40:0/1206250137 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7520084c00 con 0x7f75300cc4c0 2024-03-24T00:34:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.079+0000 7f7527e00640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f750c067660 0x7f750c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.079+0000 7f7527e00640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f750c067660 0x7f750c069b20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f75180ad000 tx=0x7f7518008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:59.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.083+0000 7f752cc00640 1 -- 172.21.15.40:0/1206250137 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7520055190 con 0x7f75300cc4c0 2024-03-24T00:34:59.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.247+0000 7f7535400640 1 -- 172.21.15.40:0/1206250137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f75300b6f30 con 0x7f75300cc4c0 2024-03-24T00:34:59.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.247+0000 7f752cc00640 1 -- 172.21.15.40:0/1206250137 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7520059230 con 0x7f75300cc4c0 2024-03-24T00:34:59.250 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:34:59.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.247+0000 7f7535400640 1 -- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f750c067660 msgr2=0x7f750c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.247+0000 7f7535400640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f750c067660 0x7f750c069b20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f75180ad000 tx=0x7f7518008040 comp rx=0 tx=0).stop 2024-03-24T00:34:59.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.247+0000 7f7535400640 1 -- 172.21.15.40:0/1206250137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75300cc4c0 msgr2=0x7f75300b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.247+0000 7f7535400640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75300cc4c0 0x7f75300b46d0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f752000c400 tx=0x7f752000c8c0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.247+0000 7f7535400640 1 -- 172.21.15.40:0/1206250137 shutdown_connections 2024-03-24T00:34:59.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.247+0000 7f7535400640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f750c067660 0x7f750c069b20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.247+0000 7f7535400640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f753015f5b0 0x7f75300b69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.247+0000 7f7535400640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75300d0750 0x7f75300b64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.247+0000 7f7535400640 1 --2- 172.21.15.40:0/1206250137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75300cc4c0 0x7f75300b46d0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.247+0000 7f7535400640 1 -- 172.21.15.40:0/1206250137 >> 172.21.15.40:0/1206250137 conn(0x7f75300b3620 msgr2=0x7f75300aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:59.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.251+0000 7f7535400640 1 -- 172.21.15.40:0/1206250137 shutdown_connections 2024-03-24T00:34:59.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.251+0000 7f7535400640 1 -- 172.21.15.40:0/1206250137 wait complete. 2024-03-24T00:34:59.271 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:34:59.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.411+0000 7f5f96000640 1 Processor -- start 2024-03-24T00:34:59.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.411+0000 7f5f96000640 1 -- start start 2024-03-24T00:34:59.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.411+0000 7f5f96000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5f9015f970 0x7f5f9015fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f96000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5f900cefb0 0x7f5f9015c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f96000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f900cc550 con 0x7f5f9015c690 2024-03-24T00:34:59.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f96000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f900ccb40 con 0x7f5f9015f970 2024-03-24T00:34:59.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f96000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f900d0a10 con 0x7f5f900cefb0 2024-03-24T00:34:59.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f8f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5f9015f970 0x7f5f9015fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f8fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5f9015c690 0x7f5f9015caa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56414/0 (socket says 172.21.15.40:56414) 2024-03-24T00:34:59.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f87e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5f900cefb0 0x7f5f9015c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f8fe00640 1 -- 172.21.15.40:0/1916779091 learned_addr learned my addr 172.21.15.40:0/1916779091 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:34:59.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f87e00640 1 -- 172.21.15.40:0/1916779091 >> v2:172.21.15.40:3300/0 conn(0x7f5f9015f970 msgr2=0x7f5f9015fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:59.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f87e00640 1 --2- 172.21.15.40:0/1916779091 >> v2:172.21.15.40:3300/0 conn(0x7f5f9015f970 0x7f5f9015fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f87e00640 1 -- 172.21.15.40:0/1916779091 >> v1:172.21.15.40:6789/0 conn(0x7f5f9015c690 legacy=0x7f5f9015caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f87e00640 1 -- 172.21.15.40:0/1916779091 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f90160290 con 0x7f5f900cefb0 2024-03-24T00:34:59.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f87e00640 1 --2- 172.21.15.40:0/1916779091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f900cefb0 0x7f5f9015c150 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5f900cccc0 tx=0x7f5f7801c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=188737994889f9af server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:59.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.415+0000 7f5f8ea00640 1 -- 172.21.15.40:0/1916779091 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f780b1070 con 0x7f5f900cefb0 2024-03-24T00:34:59.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f8ea00640 1 -- 172.21.15.40:0/1916779091 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f780ac600 con 0x7f5f900cefb0 2024-03-24T00:34:59.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f8ea00640 1 -- 172.21.15.40:0/1916779091 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f780ac9d0 con 0x7f5f900cefb0 2024-03-24T00:34:59.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 -- 172.21.15.40:0/1916779091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f900cefb0 msgr2=0x7f5f9015c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 --2- 172.21.15.40:0/1916779091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f900cefb0 0x7f5f9015c150 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5f900cccc0 tx=0x7f5f7801c3a0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 -- 172.21.15.40:0/1916779091 shutdown_connections 2024-03-24T00:34:59.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 --2- 172.21.15.40:0/1916779091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f900cefb0 0x7f5f9015c150 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 --2- 172.21.15.40:0/1916779091 >> v2:172.21.15.40:3300/0 conn(0x7f5f9015f970 0x7f5f9015fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 -- 172.21.15.40:0/1916779091 >> 172.21.15.40:0/1916779091 conn(0x7f5f900b36a0 msgr2=0x7f5f900b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:59.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 -- 172.21.15.40:0/1916779091 shutdown_connections 2024-03-24T00:34:59.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 -- 172.21.15.40:0/1916779091 wait complete. 2024-03-24T00:34:59.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 Processor -- start 2024-03-24T00:34:59.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 -- start start 2024-03-24T00:34:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f900cefb0 0x7f5f9015a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9015c690 0x7f5f900b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f9015f970 0x7f5f90144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9015d1b0 con 0x7f5f9015f970 2024-03-24T00:34:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f900cc7d0 con 0x7f5f900cefb0 2024-03-24T00:34:59.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f96000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f901606c0 con 0x7f5f9015c690 2024-03-24T00:34:59.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f87e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9015c690 0x7f5f900b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f8f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f900cefb0 0x7f5f9015a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f87e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9015c690 0x7f5f900b0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:49152/0 (socket says 172.21.15.40:49152) 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f8fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f9015f970 0x7f5f90144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.419+0000 7f5f87e00640 1 -- 172.21.15.40:0/2358207601 learned_addr learned my addr 172.21.15.40:0/2358207601 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f8f400640 1 -- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9015c690 msgr2=0x7f5f900b0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f8f400640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9015c690 0x7f5f900b0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f8f400640 1 -- 172.21.15.40:0/2358207601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f9015f970 msgr2=0x7f5f90144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f8f400640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f9015f970 0x7f5f90144770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f8f400640 1 -- 172.21.15.40:0/2358207601 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f90144cb0 con 0x7f5f900cefb0 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f87e00640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9015c690 0x7f5f900b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f8f400640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f900cefb0 0x7f5f9015a930 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5f8000e870 tx=0x7f5f8000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f87400640 1 -- 172.21.15.40:0/2358207601 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f80017070 con 0x7f5f900cefb0 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f87400640 1 -- 172.21.15.40:0/2358207601 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8000ee60 con 0x7f5f900cefb0 2024-03-24T00:34:59.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f96000640 1 -- 172.21.15.40:0/2358207601 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9015b030 con 0x7f5f900cefb0 2024-03-24T00:34:59.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f87400640 1 -- 172.21.15.40:0/2358207601 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8000cb60 con 0x7f5f900cefb0 2024-03-24T00:34:59.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f96000640 1 -- 172.21.15.40:0/2358207601 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9006b520 con 0x7f5f900cefb0 2024-03-24T00:34:59.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f96000640 1 -- 172.21.15.40:0/2358207601 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f90068170 con 0x7f5f900cefb0 2024-03-24T00:34:59.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f87400640 1 -- 172.21.15.40:0/2358207601 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5f8001a070 con 0x7f5f900cefb0 2024-03-24T00:34:59.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f87400640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f6c067660 0x7f5f6c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f87400640 1 -- 172.21.15.40:0/2358207601 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f80085690 con 0x7f5f900cefb0 2024-03-24T00:34:59.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.423+0000 7f5f87e00640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f6c067660 0x7f5f6c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.427+0000 7f5f87e00640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f6c067660 0x7f5f6c069b20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5f78000c10 tx=0x7f5f780a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:59.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.427+0000 7f5f87400640 1 -- 172.21.15.40:0/2358207601 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5f80055cb0 con 0x7f5f900cefb0 2024-03-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f96000640 1 -- 172.21.15.40:0/2358207601 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5f900d2190 con 0x7f5f900cefb0 2024-03-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f87400640 1 -- 172.21.15.40:0/2358207601 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v59) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5f80059d50 con 0x7f5f900cefb0 2024-03-24T00:34:59.593 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f96000640 1 -- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f6c067660 msgr2=0x7f5f6c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f96000640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f6c067660 0x7f5f6c069b20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5f78000c10 tx=0x7f5f780a7000 comp rx=0 tx=0).stop 2024-03-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f96000640 1 -- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f900cefb0 msgr2=0x7f5f9015a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f96000640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f900cefb0 0x7f5f9015a930 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5f8000e870 tx=0x7f5f8000ed30 comp rx=0 tx=0).stop 2024-03-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f96000640 1 -- 172.21.15.40:0/2358207601 shutdown_connections 2024-03-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f96000640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f6c067660 0x7f5f6c069b20 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f96000640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f9015f970 0x7f5f90144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f96000640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9015c690 0x7f5f900b0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f96000640 1 --2- 172.21.15.40:0/2358207601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f900cefb0 0x7f5f9015a930 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f96000640 1 -- 172.21.15.40:0/2358207601 >> 172.21.15.40:0/2358207601 conn(0x7f5f900b36a0 msgr2=0x7f5f900ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:59.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.591+0000 7f5f96000640 1 -- 172.21.15.40:0/2358207601 shutdown_connections 2024-03-24T00:34:59.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.595+0000 7f5f96000640 1 -- 172.21.15.40:0/2358207601 wait complete. 2024-03-24T00:34:59.623 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:34:59.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.763+0000 7f887de00640 1 Processor -- start 2024-03-24T00:34:59.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.763+0000 7f887de00640 1 -- start start 2024-03-24T00:34:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f887de00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f887815bd00 0x7f88780ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f887de00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f88780728d0 0x7f88780cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f887de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f887815f8c0 0x7f887815fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f887de00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88780cf790 con 0x7f887815bd00 2024-03-24T00:34:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f887de00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8878160250 con 0x7f887815f8c0 2024-03-24T00:34:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f887de00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8878072d30 con 0x7f88780728d0 2024-03-24T00:34:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f8876a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f887815bd00 0x7f88780ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f8876a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f887815bd00 0x7f88780ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46546/0 (socket says 172.21.15.40:46546) 2024-03-24T00:34:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f8876a00640 1 -- 172.21.15.40:0/3776966671 learned_addr learned my addr 172.21.15.40:0/3776966671 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f886fe00640 1 --2- 172.21.15.40:0/3776966671 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f88780728d0 0x7f88780cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f8877400640 1 --2- 172.21.15.40:0/3776966671 >> v2:172.21.15.53:3300/0 conn(0x7f887815f8c0 0x7f887815fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f8876a00640 1 -- 172.21.15.40:0/3776966671 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f88780728d0 msgr2=0x7f88780cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:59.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f8876a00640 1 --2- 172.21.15.40:0/3776966671 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f88780728d0 0x7f88780cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f8876a00640 1 -- 172.21.15.40:0/3776966671 >> v2:172.21.15.53:3300/0 conn(0x7f887815f8c0 msgr2=0x7f887815fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:34:59.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f8876a00640 1 --2- 172.21.15.40:0/3776966671 >> v2:172.21.15.53:3300/0 conn(0x7f887815f8c0 0x7f887815fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.767+0000 7f8876a00640 1 -- 172.21.15.40:0/3776966671 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8878071de0 con 0x7f887815bd00 2024-03-24T00:34:59.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f8876a00640 1 --2- 172.21.15.40:0/3776966671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f887815bd00 0x7f88780ae8e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f8878073320 tx=0x7f886801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54208844f724933 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:59.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f8876000640 1 -- 172.21.15.40:0/3776966671 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88680aa610 con 0x7f887815bd00 2024-03-24T00:34:59.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f8876000640 1 -- 172.21.15.40:0/3776966671 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88680af070 con 0x7f887815bd00 2024-03-24T00:34:59.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f8876000640 1 -- 172.21.15.40:0/3776966671 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88680aa9d0 con 0x7f887815bd00 2024-03-24T00:34:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f887de00640 1 -- 172.21.15.40:0/3776966671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f887815bd00 msgr2=0x7f88780ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f887de00640 1 --2- 172.21.15.40:0/3776966671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f887815bd00 0x7f88780ae8e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f8878073320 tx=0x7f886801c4f0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f887de00640 1 -- 172.21.15.40:0/3776966671 shutdown_connections 2024-03-24T00:34:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f887de00640 1 --2- 172.21.15.40:0/3776966671 >> v2:172.21.15.53:3300/0 conn(0x7f887815f8c0 0x7f887815fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f887de00640 1 --2- 172.21.15.40:0/3776966671 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f88780728d0 0x7f88780cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f887de00640 1 --2- 172.21.15.40:0/3776966671 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f887815bd00 0x7f88780ae8e0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f887de00640 1 -- 172.21.15.40:0/3776966671 >> 172.21.15.40:0/3776966671 conn(0x7f88780b1660 msgr2=0x7f88780b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f887de00640 1 -- 172.21.15.40:0/3776966671 shutdown_connections 2024-03-24T00:34:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.771+0000 7f887de00640 1 -- 172.21.15.40:0/3776966671 wait complete. 2024-03-24T00:34:59.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f887de00640 1 Processor -- start 2024-03-24T00:34:59.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f887de00640 1 -- start start 2024-03-24T00:34:59.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f887de00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f88780728d0 0x7f887815e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f887de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f887815bd00 0x7f887814dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f887de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f887815f8c0 0x7f88781535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f887de00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887806d320 con 0x7f887815f8c0 2024-03-24T00:34:59.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f887de00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887815c0e0 con 0x7f887815bd00 2024-03-24T00:34:59.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f887de00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887806d4a0 con 0x7f88780728d0 2024-03-24T00:34:59.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f886fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f887815bd00 0x7f887814dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f8877400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f887815f8c0 0x7f88781535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f8876a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f88780728d0 0x7f887815e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f886fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f887815bd00 0x7f887814dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:42996/0 (socket says 172.21.15.40:42996) 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f886fe00640 1 -- 172.21.15.40:0/2874268935 learned_addr learned my addr 172.21.15.40:0/2874268935 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f8876a00640 1 -- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f887815bd00 msgr2=0x7f887814dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f8876a00640 1 --2- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f887815bd00 0x7f887814dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f8876a00640 1 -- 172.21.15.40:0/2874268935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f887815f8c0 msgr2=0x7f88781535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f8876a00640 1 --2- 172.21.15.40:0/2874268935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f887815f8c0 0x7f88781535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f8876a00640 1 -- 172.21.15.40:0/2874268935 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88781f9c60 con 0x7f88780728d0 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f8876a00640 1 --2- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f88780728d0 0x7f887815e470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f88680ab000 tx=0x7f886801ad30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f886f400640 1 -- 172.21.15.40:0/2874268935 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88680bd5f0 con 0x7f88780728d0 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f886f400640 1 -- 172.21.15.40:0/2874268935 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88680bd7b0 con 0x7f88780728d0 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f886f400640 1 -- 172.21.15.40:0/2874268935 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88680bdba0 con 0x7f88780728d0 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f887de00640 1 -- 172.21.15.40:0/2874268935 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f887815c720 con 0x7f88780728d0 2024-03-24T00:34:59.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.775+0000 7f887de00640 1 -- 172.21.15.40:0/2874268935 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887815cce0 con 0x7f88780728d0 2024-03-24T00:34:59.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.779+0000 7f887de00640 1 -- 172.21.15.40:0/2874268935 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88781f9e30 con 0x7f88780728d0 2024-03-24T00:34:59.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.779+0000 7f886f400640 1 -- 172.21.15.40:0/2874268935 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f88680b33f0 con 0x7f88780728d0 2024-03-24T00:34:59.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.779+0000 7f886f400640 1 --2- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8854067660 0x7f8854069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:34:59.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.779+0000 7f886f400640 1 -- 172.21.15.40:0/2874268935 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f886811fcc0 con 0x7f88780728d0 2024-03-24T00:34:59.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.779+0000 7f886fe00640 1 --2- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8854067660 0x7f8854069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:34:59.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.783+0000 7f886fe00640 1 --2- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8854067660 0x7f8854069b20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f88780b2d00 tx=0x7f8860009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:34:59.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.783+0000 7f886f400640 1 -- 172.21.15.40:0/2874268935 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8868126080 con 0x7f88780728d0 2024-03-24T00:34:59.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.947+0000 7f887de00640 1 -- 172.21.15.40:0/2874268935 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8878068170 con 0x7f88780728d0 2024-03-24T00:34:59.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.947+0000 7f886f400640 1 -- 172.21.15.40:0/2874268935 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v59) v1 ==== 100+0+11 (secure 0 0 0) 0x7f88680c3060 con 0x7f88780728d0 2024-03-24T00:34:59.951 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:34:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.951+0000 7f887de00640 1 -- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8854067660 msgr2=0x7f8854069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.951+0000 7f887de00640 1 --2- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8854067660 0x7f8854069b20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f88780b2d00 tx=0x7f8860009190 comp rx=0 tx=0).stop 2024-03-24T00:34:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.951+0000 7f887de00640 1 -- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f88780728d0 msgr2=0x7f887815e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:34:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.951+0000 7f887de00640 1 --2- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f88780728d0 0x7f887815e470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f88680ab000 tx=0x7f886801ad30 comp rx=0 tx=0).stop 2024-03-24T00:34:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.951+0000 7f887de00640 1 -- 172.21.15.40:0/2874268935 shutdown_connections 2024-03-24T00:34:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.951+0000 7f887de00640 1 --2- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8854067660 0x7f8854069b20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.951+0000 7f887de00640 1 --2- 172.21.15.40:0/2874268935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f887815f8c0 0x7f88781535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.951+0000 7f887de00640 1 --2- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f887815bd00 0x7f887814dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.951+0000 7f887de00640 1 --2- 172.21.15.40:0/2874268935 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f88780728d0 0x7f887815e470 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:34:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.951+0000 7f887de00640 1 -- 172.21.15.40:0/2874268935 >> 172.21.15.40:0/2874268935 conn(0x7f88780b1660 msgr2=0x7f88780b36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:34:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.951+0000 7f887de00640 1 -- 172.21.15.40:0/2874268935 shutdown_connections 2024-03-24T00:34:59.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:34:59.951+0000 7f887de00640 1 -- 172.21.15.40:0/2874268935 wait complete. 2024-03-24T00:34:59.981 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:00.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865f200640 1 Processor -- start 2024-03-24T00:35:00.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865f200640 1 -- start start 2024-03-24T00:35:00.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865f200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f86580ceba0 0x7f86580d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865f200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86580d0780 con 0x7f865815c080 2024-03-24T00:35:00.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865f200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86580b4a90 con 0x7f865815bca0 2024-03-24T00:35:00.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865f200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86580b4d70 con 0x7f86580ceba0 2024-03-24T00:35:00.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865e800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f865815c080 0x7f86580b5570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56420/0 (socket says 172.21.15.40:56420) 2024-03-24T00:35:00.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865e800640 1 -- 172.21.15.40:0/996616748 learned_addr learned my addr 172.21.15.40:0/996616748 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:00.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865d400640 1 --2- 172.21.15.40:0/996616748 >> v2:172.21.15.53:3300/0 conn(0x7f86580ceba0 0x7f86580d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865ca00640 1 -- 172.21.15.40:0/996616748 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181905843 0 0) 0x7f86580d0780 con 0x7f865815c080 2024-03-24T00:35:00.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865ca00640 1 -- 172.21.15.40:0/996616748 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8640003530 con 0x7f865815c080 2024-03-24T00:35:00.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865d400640 1 -- 172.21.15.40:0/996616748 >> v1:172.21.15.53:6789/0 conn(0x7f865815bca0 legacy=0x7f86580ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865ca00640 1 -- 172.21.15.40:0/996616748 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044165398 0 0) 0x7f86580b4a90 con 0x7f865815bca0 2024-03-24T00:35:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865d400640 1 -- 172.21.15.40:0/996616748 >> v1:172.21.15.40:6789/0 conn(0x7f865815c080 legacy=0x7f86580b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.123+0000 7f865d400640 1 -- 172.21.15.40:0/996616748 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580b5050 con 0x7f86580ceba0 2024-03-24T00:35:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865d400640 1 --2- 172.21.15.40:0/996616748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86580ceba0 0x7f86580d2590 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f86480097c0 tx=0x7f864801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=309f0f43f2caead4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865ca00640 1 -- 172.21.15.40:0/996616748 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86480b1070 con 0x7f86580ceba0 2024-03-24T00:35:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865ca00640 1 -- 172.21.15.40:0/996616748 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86480ac410 con 0x7f86580ceba0 2024-03-24T00:35:00.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865ca00640 1 -- 172.21.15.40:0/996616748 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86480ac7e0 con 0x7f86580ceba0 2024-03-24T00:35:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865f200640 1 -- 172.21.15.40:0/996616748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86580ceba0 msgr2=0x7f86580d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865f200640 1 --2- 172.21.15.40:0/996616748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86580ceba0 0x7f86580d2590 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f86480097c0 tx=0x7f864801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865f200640 1 -- 172.21.15.40:0/996616748 shutdown_connections 2024-03-24T00:35:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865f200640 1 --2- 172.21.15.40:0/996616748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86580ceba0 0x7f86580d2590 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865f200640 1 -- 172.21.15.40:0/996616748 >> 172.21.15.40:0/996616748 conn(0x7f86580d1aa0 msgr2=0x7f86580d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865f200640 1 -- 172.21.15.40:0/996616748 shutdown_connections 2024-03-24T00:35:00.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865f200640 1 -- 172.21.15.40:0/996616748 wait complete. 2024-03-24T00:35:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865f200640 1 Processor -- start 2024-03-24T00:35:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.127+0000 7f865f200640 1 -- start start 2024-03-24T00:35:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86580ceba0 0x7f865815a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865f200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f865815bca0 0x7f865815dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f865815c080 0x7f865815e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86580b68a0 con 0x7f86580ceba0 2024-03-24T00:35:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86580b6720 con 0x7f865815c080 2024-03-24T00:35:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865f200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8658169a50 con 0x7f865815bca0 2024-03-24T00:35:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86580ceba0 0x7f865815a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86580ceba0 0x7f865815a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46578/0 (socket says 172.21.15.40:46578) 2024-03-24T00:35:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865de00640 1 -- 172.21.15.40:0/3004397760 learned_addr learned my addr 172.21.15.40:0/3004397760 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865d400640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f865815bca0 0x7f865815dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865e800640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f865815c080 0x7f865815e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865de00640 1 -- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f865815bca0 msgr2=0x7f865815dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865de00640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f865815bca0 0x7f865815dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865de00640 1 -- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f865815c080 msgr2=0x7f865815e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865de00640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f865815c080 0x7f865815e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865de00640 1 -- 172.21.15.40:0/3004397760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580c8c10 con 0x7f86580ceba0 2024-03-24T00:35:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865e800640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f865815c080 0x7f865815e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865de00640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86580ceba0 0x7f865815a760 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f864c00c480 tx=0x7f864c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f8652000640 1 -- 172.21.15.40:0/3004397760 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c005480 con 0x7f86580ceba0 2024-03-24T00:35:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f8652000640 1 -- 172.21.15.40:0/3004397760 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c003600 con 0x7f86580ceba0 2024-03-24T00:35:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f8652000640 1 -- 172.21.15.40:0/3004397760 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c0039f0 con 0x7f86580ceba0 2024-03-24T00:35:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865f200640 1 -- 172.21.15.40:0/3004397760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86581415f0 con 0x7f86580ceba0 2024-03-24T00:35:00.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865f200640 1 -- 172.21.15.40:0/3004397760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f865815aca0 con 0x7f86580ceba0 2024-03-24T00:35:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.131+0000 7f865f200640 1 -- 172.21.15.40:0/3004397760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f865809fb10 con 0x7f86580ceba0 2024-03-24T00:35:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.135+0000 7f8652000640 1 -- 172.21.15.40:0/3004397760 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f864c003b90 con 0x7f86580ceba0 2024-03-24T00:35:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.135+0000 7f8652000640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f863c067660 0x7f863c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.135+0000 7f8652000640 1 -- 172.21.15.40:0/3004397760 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f864c0854c0 con 0x7f86580ceba0 2024-03-24T00:35:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.135+0000 7f865d400640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f863c067660 0x7f863c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.135+0000 7f865d400640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f863c067660 0x7f863c069b20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f86480ae000 tx=0x7f86480a6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:00.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.139+0000 7f8652000640 1 -- 172.21.15.40:0/3004397760 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f864c056130 con 0x7f86580ceba0 2024-03-24T00:35:00.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.335+0000 7f865f200640 1 -- 172.21.15.40:0/3004397760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f865804f4f0 con 0x7f86580ceba0 2024-03-24T00:35:00.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f8652000640 1 -- 172.21.15.40:0/3004397760 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5015 (secure 0 0 0) 0x7f864c055a30 con 0x7f86580ceba0 2024-03-24T00:35:00.340 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:00.341 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":58,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6277,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1633817617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1633817617},{"type":"v1","addr":"172.21.15.53:6837","nonce":1633817617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:34:12.019118+0000","modified":"2024-03-24T00:34:35.695160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6474,"mds_1":6471,"mds_2":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":2,"incarnation":56,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6834/1278515461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1278515461},{"type":"v1","addr":"172.21.15.40:6834","nonce":1278515461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6471":{"gid":6471,"name":"c","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6835/1312292485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1312292485},{"type":"v1","addr":"172.21.15.40:6835","nonce":1312292485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6474":{"gid":6474,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/501079401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":501079401},{"type":"v1","addr":"172.21.15.53:6835","nonce":501079401}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6474,"qdb_cluster":[6474,6471,6468]},"id":4}]} 2024-03-24T00:35:00.341 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 58 2024-03-24T00:35:00.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f865f200640 1 -- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f863c067660 msgr2=0x7f863c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f865f200640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f863c067660 0x7f863c069b20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f86480ae000 tx=0x7f86480a6040 comp rx=0 tx=0).stop 2024-03-24T00:35:00.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f865f200640 1 -- 172.21.15.40:0/3004397760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86580ceba0 msgr2=0x7f865815a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f865f200640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86580ceba0 0x7f865815a760 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f864c00c480 tx=0x7f864c00c940 comp rx=0 tx=0).stop 2024-03-24T00:35:00.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f865f200640 1 -- 172.21.15.40:0/3004397760 shutdown_connections 2024-03-24T00:35:00.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f865f200640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f863c067660 0x7f863c069b20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f865f200640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f865815c080 0x7f865815e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f865f200640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f865815bca0 0x7f865815dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f865f200640 1 --2- 172.21.15.40:0/3004397760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86580ceba0 0x7f865815a760 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f865f200640 1 -- 172.21.15.40:0/3004397760 >> 172.21.15.40:0/3004397760 conn(0x7f86580d1aa0 msgr2=0x7f86580b70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:00.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f865f200640 1 -- 172.21.15.40:0/3004397760 shutdown_connections 2024-03-24T00:35:00.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.339+0000 7f865f200640 1 -- 172.21.15.40:0/3004397760 wait complete. 2024-03-24T00:35:00.370 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:00.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c3600640 1 Processor -- start 2024-03-24T00:35:00.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c3600640 1 -- start start 2024-03-24T00:35:00.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c3600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5bc15f350 0x7fd5bc0cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c3600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5bc070970 0x7fd5bc0d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c3600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd5bc0d27a0 0x7fd5bc06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c3600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5bc160500 con 0x7fd5bc0d27a0 2024-03-24T00:35:00.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c3600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5bc160a30 con 0x7fd5bc070970 2024-03-24T00:35:00.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c3600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc070650 con 0x7fd5bc15f350 2024-03-24T00:35:00.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c1600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd5bc0d27a0 0x7fd5bc06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5bbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5bc070970 0x7fd5bc0d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c0c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5bc15f350 0x7fd5bc0cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c0c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5bc15f350 0x7fd5bc0cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:49188/0 (socket says 172.21.15.40:49188) 2024-03-24T00:35:00.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5bbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5bc070970 0x7fd5bc0d0230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:43024/0 (socket says 172.21.15.40:43024) 2024-03-24T00:35:00.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c0c00640 1 -- 172.21.15.40:0/3905258285 learned_addr learned my addr 172.21.15.40:0/3905258285 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:00.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c0c00640 1 -- 172.21.15.40:0/3905258285 >> v2:172.21.15.53:3300/0 conn(0x7fd5bc070970 msgr2=0x7fd5bc0d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:00.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c0c00640 1 --2- 172.21.15.40:0/3905258285 >> v2:172.21.15.53:3300/0 conn(0x7fd5bc070970 0x7fd5bc0d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c0c00640 1 -- 172.21.15.40:0/3905258285 >> v2:172.21.15.40:3300/0 conn(0x7fd5bc0d27a0 msgr2=0x7fd5bc06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:00.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c0c00640 1 --2- 172.21.15.40:0/3905258285 >> v2:172.21.15.40:3300/0 conn(0x7fd5bc0d27a0 0x7fd5bc06ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.511+0000 7fd5c0c00640 1 -- 172.21.15.40:0/3905258285 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5bc0d0a50 con 0x7fd5bc15f350 2024-03-24T00:35:00.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5c0c00640 1 --2- 172.21.15.40:0/3905258285 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5bc15f350 0x7fd5bc0cfcf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc068e70 tx=0x7fd5b001c110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98775789244d122f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5bb400640 1 -- 172.21.15.40:0/3905258285 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b00b0070 con 0x7fd5bc15f350 2024-03-24T00:35:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5bb400640 1 -- 172.21.15.40:0/3905258285 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b00ab600 con 0x7fd5bc15f350 2024-03-24T00:35:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5bb400640 1 -- 172.21.15.40:0/3905258285 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b00ab9d0 con 0x7fd5bc15f350 2024-03-24T00:35:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5c3600640 1 -- 172.21.15.40:0/3905258285 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5bc15f350 msgr2=0x7fd5bc0cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5c3600640 1 --2- 172.21.15.40:0/3905258285 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5bc15f350 0x7fd5bc0cfcf0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc068e70 tx=0x7fd5b001c110 comp rx=0 tx=0).stop 2024-03-24T00:35:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5c3600640 1 -- 172.21.15.40:0/3905258285 shutdown_connections 2024-03-24T00:35:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5c3600640 1 --2- 172.21.15.40:0/3905258285 >> v2:172.21.15.40:3300/0 conn(0x7fd5bc0d27a0 0x7fd5bc06ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5c3600640 1 --2- 172.21.15.40:0/3905258285 >> v2:172.21.15.53:3300/0 conn(0x7fd5bc070970 0x7fd5bc0d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5c3600640 1 --2- 172.21.15.40:0/3905258285 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5bc15f350 0x7fd5bc0cfcf0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5c3600640 1 -- 172.21.15.40:0/3905258285 >> 172.21.15.40:0/3905258285 conn(0x7fd5bc0d1c80 msgr2=0x7fd5bc0d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5c3600640 1 -- 172.21.15.40:0/3905258285 shutdown_connections 2024-03-24T00:35:00.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.515+0000 7fd5c3600640 1 -- 172.21.15.40:0/3905258285 wait complete. 2024-03-24T00:35:00.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c3600640 1 Processor -- start 2024-03-24T00:35:00.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c3600640 1 -- start start 2024-03-24T00:35:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c3600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5bc070970 0x7fd5bc0b7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c3600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5bc0d27a0 0x7fd5bc159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c3600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5bc15f350 0x7fd5bc152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c3600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc15f8b0 con 0x7fd5bc15f350 2024-03-24T00:35:00.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c3600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc15f730 con 0x7fd5bc070970 2024-03-24T00:35:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c3600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5bc0aecd0 con 0x7fd5bc0d27a0 2024-03-24T00:35:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5bc15f350 0x7fd5bc152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5bc070970 0x7fd5bc0b7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5bc15f350 0x7fd5bc152930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46588/0 (socket says 172.21.15.40:46588) 2024-03-24T00:35:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5bbe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5bc0d27a0 0x7fd5bc159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c1600640 1 -- 172.21.15.40:0/3053065660 learned_addr learned my addr 172.21.15.40:0/3053065660 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c1600640 1 -- 172.21.15.40:0/3053065660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5bc0d27a0 msgr2=0x7fd5bc159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c1600640 1 --2- 172.21.15.40:0/3053065660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5bc0d27a0 0x7fd5bc159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c1600640 1 -- 172.21.15.40:0/3053065660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5bc070970 msgr2=0x7fd5bc0b7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c1600640 1 --2- 172.21.15.40:0/3053065660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5bc070970 0x7fd5bc0b7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c1600640 1 -- 172.21.15.40:0/3053065660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5bc0ca670 con 0x7fd5bc15f350 2024-03-24T00:35:00.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c1600640 1 --2- 172.21.15.40:0/3053065660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5bc15f350 0x7fd5bc152930 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400efc0 tx=0x7fd5b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5b8c00640 1 -- 172.21.15.40:0/3053065660 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4018020 con 0x7fd5bc15f350 2024-03-24T00:35:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5b8c00640 1 -- 172.21.15.40:0/3053065660 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b4014480 con 0x7fd5bc15f350 2024-03-24T00:35:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5b8c00640 1 -- 172.21.15.40:0/3053065660 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4014870 con 0x7fd5bc15f350 2024-03-24T00:35:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c3600640 1 -- 172.21.15.40:0/3053065660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5bc0ca950 con 0x7fd5bc15f350 2024-03-24T00:35:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.519+0000 7fd5c3600640 1 -- 172.21.15.40:0/3053065660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5bc0caf10 con 0x7fd5bc15f350 2024-03-24T00:35:00.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.523+0000 7fd5c3600640 1 -- 172.21.15.40:0/3053065660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5bc068170 con 0x7fd5bc15f350 2024-03-24T00:35:00.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.523+0000 7fd5b8c00640 1 -- 172.21.15.40:0/3053065660 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd5b401b020 con 0x7fd5bc15f350 2024-03-24T00:35:00.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.523+0000 7fd5b8c00640 1 --2- 172.21.15.40:0/3053065660 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd59c067660 0x7fd59c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.523+0000 7fd5b8c00640 1 -- 172.21.15.40:0/3053065660 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5b40865e0 con 0x7fd5bc15f350 2024-03-24T00:35:00.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.523+0000 7fd5c0c00640 1 --2- 172.21.15.40:0/3053065660 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd59c067660 0x7fd59c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.523+0000 7fd5c0c00640 1 --2- 172.21.15.40:0/3053065660 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd59c067660 0x7fd59c069b20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00ad000 tx=0x7fd5b00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:00.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.527+0000 7fd5b8c00640 1 -- 172.21.15.40:0/3053065660 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd5b4022030 con 0x7fd5bc15f350 2024-03-24T00:35:00.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.699+0000 7fd5c3600640 1 -- 172.21.15.40:0/3053065660 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5bc073070 con 0x7fd5bc15f350 2024-03-24T00:35:00.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.699+0000 7fd5b8c00640 1 -- 172.21.15.40:0/3053065660 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16605 (secure 0 0 0) 0x7fd5b4010040 con 0x7fd5bc15f350 2024-03-24T00:35:00.701 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:00.702 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":59,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:34:13.623804+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:34:10.428162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-24T00:34:11.001094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.699+0000 7fd5c3600640 1 -- 172.21.15.40:0/3053065660 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd59c067660 msgr2=0x7fd59c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.699+0000 7fd5c3600640 1 --2- 172.21.15.40:0/3053065660 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd59c067660 0x7fd59c069b20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00ad000 tx=0x7fd5b00a6000 comp rx=0 tx=0).stop 2024-03-24T00:35:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.699+0000 7fd5c3600640 1 -- 172.21.15.40:0/3053065660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5bc15f350 msgr2=0x7fd5bc152930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.699+0000 7fd5c3600640 1 --2- 172.21.15.40:0/3053065660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5bc15f350 0x7fd5bc152930 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400efc0 tx=0x7fd5b400c490 comp rx=0 tx=0).stop 2024-03-24T00:35:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.699+0000 7fd5c3600640 1 -- 172.21.15.40:0/3053065660 shutdown_connections 2024-03-24T00:35:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.699+0000 7fd5c3600640 1 --2- 172.21.15.40:0/3053065660 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd59c067660 0x7fd59c069b20 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.699+0000 7fd5c3600640 1 --2- 172.21.15.40:0/3053065660 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5bc15f350 0x7fd5bc152930 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.699+0000 7fd5c3600640 1 --2- 172.21.15.40:0/3053065660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5bc0d27a0 0x7fd5bc159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.699+0000 7fd5c3600640 1 --2- 172.21.15.40:0/3053065660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5bc070970 0x7fd5bc0b7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.699+0000 7fd5c3600640 1 -- 172.21.15.40:0/3053065660 >> 172.21.15.40:0/3053065660 conn(0x7fd5bc0d1c80 msgr2=0x7fd5bc0c6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.703+0000 7fd5c3600640 1 -- 172.21.15.40:0/3053065660 shutdown_connections 2024-03-24T00:35:00.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.703+0000 7fd5c3600640 1 -- 172.21.15.40:0/3053065660 wait complete. 2024-03-24T00:35:00.731 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:35:00.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe98c00640 1 Processor -- start 2024-03-24T00:35:00.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe98c00640 1 -- start start 2024-03-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe98c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbe9406e620 0x7fbe940b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe98c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe940ccfa0 0x7fbe940cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe98c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe94070f60 con 0x7fbe9406ea00 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe98c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe940b4c00 con 0x7fbe940ccfa0 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe98c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe940b3620 con 0x7fbe9406e620 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe92800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbe940ccfa0 0x7fbe940cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe91400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbe9406ea00 0x7fbe940b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56434/0 (socket says 172.21.15.40:56434) 2024-03-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe91400640 1 -- 172.21.15.40:0/2157438261 learned_addr learned my addr 172.21.15.40:0/2157438261 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe91e00640 1 --2- 172.21.15.40:0/2157438261 >> v2:172.21.15.53:3300/0 conn(0x7fbe9406e620 0x7fbe940b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe92800640 1 -- 172.21.15.40:0/2157438261 >> v2:172.21.15.53:3300/0 conn(0x7fbe9406e620 msgr2=0x7fbe940b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe90a00640 1 -- 172.21.15.40:0/2157438261 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156753680 0 0) 0x7fbe94070f60 con 0x7fbe9406ea00 2024-03-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe92800640 1 --2- 172.21.15.40:0/2157438261 >> v2:172.21.15.53:3300/0 conn(0x7fbe9406e620 0x7fbe940b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe92800640 1 -- 172.21.15.40:0/2157438261 >> v1:172.21.15.40:6789/0 conn(0x7fbe9406ea00 legacy=0x7fbe940b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe92800640 1 -- 172.21.15.40:0/2157438261 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe940b3900 con 0x7fbe940ccfa0 2024-03-24T00:35:00.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.871+0000 7fbe91e00640 1 --2- 172.21.15.40:0/2157438261 >> v2:172.21.15.53:3300/0 conn(0x7fbe9406e620 0x7fbe940b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe92800640 1 --2- 172.21.15.40:0/2157438261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe940ccfa0 0x7fbe940cd380 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fbe9406c5f0 tx=0x7fbe8c01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec370cef738c102a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe90a00640 1 -- 172.21.15.40:0/2157438261 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c0b2070 con 0x7fbe940ccfa0 2024-03-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe90a00640 1 -- 172.21.15.40:0/2157438261 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8c0a5840 con 0x7fbe940ccfa0 2024-03-24T00:35:00.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe90a00640 1 -- 172.21.15.40:0/2157438261 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c0a5c10 con 0x7fbe940ccfa0 2024-03-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe98c00640 1 -- 172.21.15.40:0/2157438261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe940ccfa0 msgr2=0x7fbe940cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe98c00640 1 --2- 172.21.15.40:0/2157438261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe940ccfa0 0x7fbe940cd380 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fbe9406c5f0 tx=0x7fbe8c01c870 comp rx=0 tx=0).stop 2024-03-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe98c00640 1 -- 172.21.15.40:0/2157438261 shutdown_connections 2024-03-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe98c00640 1 --2- 172.21.15.40:0/2157438261 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe940ccfa0 0x7fbe940cd380 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe98c00640 1 --2- 172.21.15.40:0/2157438261 >> v2:172.21.15.53:3300/0 conn(0x7fbe9406e620 0x7fbe940b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe98c00640 1 -- 172.21.15.40:0/2157438261 >> 172.21.15.40:0/2157438261 conn(0x7fbe940b17b0 msgr2=0x7fbe940b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe98c00640 1 -- 172.21.15.40:0/2157438261 shutdown_connections 2024-03-24T00:35:00.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe98c00640 1 -- 172.21.15.40:0/2157438261 wait complete. 2024-03-24T00:35:00.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe98c00640 1 Processor -- start 2024-03-24T00:35:00.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe98c00640 1 -- start start 2024-03-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.879+0000 7fbe98c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406e620 0x7fbe94143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe98c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9406ea00 0x7fbe9415d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe98c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe940ccfa0 0x7fbe94140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe98c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9406efd0 con 0x7fbe9406e620 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe98c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9406ee50 con 0x7fbe9406ea00 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe98c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe94144920 con 0x7fbe940ccfa0 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe91e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406e620 0x7fbe94143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe91e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406e620 0x7fbe94143570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46608/0 (socket says 172.21.15.40:46608) 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe91e00640 1 -- 172.21.15.40:0/2114664428 learned_addr learned my addr 172.21.15.40:0/2114664428 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe91400640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9406ea00 0x7fbe9415d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe92800640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe940ccfa0 0x7fbe94140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe91e00640 1 -- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe940ccfa0 msgr2=0x7fbe94140ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe91e00640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe940ccfa0 0x7fbe94140ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe91e00640 1 -- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9406ea00 msgr2=0x7fbe9415d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe91e00640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9406ea00 0x7fbe9415d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe91e00640 1 -- 172.21.15.40:0/2114664428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe941594e0 con 0x7fbe9406e620 2024-03-24T00:35:00.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe91e00640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406e620 0x7fbe94143570 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400e980 tx=0x7fbe8400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe8a000640 1 -- 172.21.15.40:0/2114664428 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe84017070 con 0x7fbe9406e620 2024-03-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe8a000640 1 -- 172.21.15.40:0/2114664428 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8400ca00 con 0x7fbe9406e620 2024-03-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe98c00640 1 -- 172.21.15.40:0/2114664428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9415dce0 con 0x7fbe9406e620 2024-03-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe8a000640 1 -- 172.21.15.40:0/2114664428 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8400cdd0 con 0x7fbe9406e620 2024-03-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe98c00640 1 -- 172.21.15.40:0/2114664428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe94159780 con 0x7fbe9406e620 2024-03-24T00:35:00.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe98c00640 1 -- 172.21.15.40:0/2114664428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe94068170 con 0x7fbe9406e620 2024-03-24T00:35:00.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.883+0000 7fbe8a000640 1 -- 172.21.15.40:0/2114664428 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbe8401a070 con 0x7fbe9406e620 2024-03-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.887+0000 7fbe8a000640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbe74067660 0x7fbe74069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.887+0000 7fbe8a000640 1 -- 172.21.15.40:0/2114664428 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe840857a0 con 0x7fbe9406e620 2024-03-24T00:35:00.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.887+0000 7fbe91400640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbe74067660 0x7fbe74069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:00.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.887+0000 7fbe91400640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbe74067660 0x7fbe74069b20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbe940cd660 tx=0x7fbe80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:00.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:00.895+0000 7fbe8a000640 1 -- 172.21.15.40:0/2114664428 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbe84055d30 con 0x7fbe9406e620 2024-03-24T00:35:01.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.095+0000 7fbe98c00640 1 -- 172.21.15.40:0/2114664428 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe94072f50 con 0x7fbe9406e620 2024-03-24T00:35:01.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.095+0000 7fbe8a000640 1 -- 172.21.15.40:0/2114664428 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5015 (secure 0 0 0) 0x7fbe84059dd0 con 0x7fbe9406e620 2024-03-24T00:35:01.099 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:01.099 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":58,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6277,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1633817617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1633817617},{"type":"v1","addr":"172.21.15.53:6837","nonce":1633817617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:34:12.019118+0000","modified":"2024-03-24T00:34:35.695160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6474,"mds_1":6471,"mds_2":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":2,"incarnation":56,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6834/1278515461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1278515461},{"type":"v1","addr":"172.21.15.40:6834","nonce":1278515461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6471":{"gid":6471,"name":"c","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6835/1312292485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1312292485},{"type":"v1","addr":"172.21.15.40:6835","nonce":1312292485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6474":{"gid":6474,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/501079401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":501079401},{"type":"v1","addr":"172.21.15.53:6835","nonce":501079401}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6474,"qdb_cluster":[6474,6471,6468]},"id":4}]} 2024-03-24T00:35:01.099 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 58 2024-03-24T00:35:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.099+0000 7fbe98c00640 1 -- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbe74067660 msgr2=0x7fbe74069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.099+0000 7fbe98c00640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbe74067660 0x7fbe74069b20 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbe940cd660 tx=0x7fbe80009040 comp rx=0 tx=0).stop 2024-03-24T00:35:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.099+0000 7fbe98c00640 1 -- 172.21.15.40:0/2114664428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406e620 msgr2=0x7fbe94143570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.099+0000 7fbe98c00640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406e620 0x7fbe94143570 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400e980 tx=0x7fbe8400ee40 comp rx=0 tx=0).stop 2024-03-24T00:35:01.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.099+0000 7fbe98c00640 1 -- 172.21.15.40:0/2114664428 shutdown_connections 2024-03-24T00:35:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.099+0000 7fbe98c00640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbe74067660 0x7fbe74069b20 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.099+0000 7fbe98c00640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe940ccfa0 0x7fbe94140ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.099+0000 7fbe98c00640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe9406ea00 0x7fbe9415d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.099+0000 7fbe98c00640 1 --2- 172.21.15.40:0/2114664428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe9406e620 0x7fbe94143570 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.099+0000 7fbe98c00640 1 -- 172.21.15.40:0/2114664428 >> 172.21.15.40:0/2114664428 conn(0x7fbe940b17b0 msgr2=0x7fbe940cde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.099+0000 7fbe98c00640 1 -- 172.21.15.40:0/2114664428 shutdown_connections 2024-03-24T00:35:01.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.099+0000 7fbe98c00640 1 -- 172.21.15.40:0/2114664428 wait complete. 2024-03-24T00:35:01.120 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:01.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f8625c00640 1 Processor -- start 2024-03-24T00:35:01.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f8625c00640 1 -- start start 2024-03-24T00:35:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f8625c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f862013ec80 0x7f86201414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f8625c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8620070e00 con 0x7f86200ce800 2024-03-24T00:35:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f8625c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86200cee20 con 0x7f86201419f0 2024-03-24T00:35:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f8625c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8620141e50 con 0x7f862013ec80 2024-03-24T00:35:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f861f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f86200ce800 0x7f8620071300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56444/0 (socket says 172.21.15.40:56444) 2024-03-24T00:35:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f861f400640 1 -- 172.21.15.40:0/1353536475 learned_addr learned my addr 172.21.15.40:0/1353536475 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:01.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f861ea00640 1 --2- 172.21.15.40:0/1353536475 >> v2:172.21.15.53:3300/0 conn(0x7f862013ec80 0x7f86201414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f861e000640 1 -- 172.21.15.40:0/1353536475 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548633503 0 0) 0x7f8620070e00 con 0x7f86200ce800 2024-03-24T00:35:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f861e000640 1 -- 172.21.15.40:0/1353536475 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8604003530 con 0x7f86200ce800 2024-03-24T00:35:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f861ea00640 1 -- 172.21.15.40:0/1353536475 >> v1:172.21.15.53:6789/0 conn(0x7f86201419f0 legacy=0x7f86200ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f861e000640 1 -- 172.21.15.40:0/1353536475 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637329111 0 0) 0x7f86200cee20 con 0x7f86201419f0 2024-03-24T00:35:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f861ea00640 1 -- 172.21.15.40:0/1353536475 >> v1:172.21.15.40:6789/0 conn(0x7f86200ce800 legacy=0x7f8620071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.259+0000 7f861ea00640 1 -- 172.21.15.40:0/1353536475 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86200cf100 con 0x7f862013ec80 2024-03-24T00:35:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.263+0000 7f861ea00640 1 --2- 172.21.15.40:0/1353536475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f862013ec80 0x7f86201414b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f86100097c0 tx=0x7f861001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7993d00cc8328fb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.263+0000 7f861e000640 1 -- 172.21.15.40:0/1353536475 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86100aa620 con 0x7f862013ec80 2024-03-24T00:35:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.263+0000 7f861e000640 1 -- 172.21.15.40:0/1353536475 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86100af070 con 0x7f862013ec80 2024-03-24T00:35:01.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.263+0000 7f861e000640 1 -- 172.21.15.40:0/1353536475 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86100aa9e0 con 0x7f862013ec80 2024-03-24T00:35:01.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 -- 172.21.15.40:0/1353536475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f862013ec80 msgr2=0x7f86201414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 --2- 172.21.15.40:0/1353536475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f862013ec80 0x7f86201414b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f86100097c0 tx=0x7f861001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 -- 172.21.15.40:0/1353536475 shutdown_connections 2024-03-24T00:35:01.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 --2- 172.21.15.40:0/1353536475 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f862013ec80 0x7f86201414b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 -- 172.21.15.40:0/1353536475 >> 172.21.15.40:0/1353536475 conn(0x7f86200b17b0 msgr2=0x7f86200b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:01.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 -- 172.21.15.40:0/1353536475 shutdown_connections 2024-03-24T00:35:01.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 -- 172.21.15.40:0/1353536475 wait complete. 2024-03-24T00:35:01.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 Processor -- start 2024-03-24T00:35:01.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 -- start start 2024-03-24T00:35:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86200ce800 0x7f862013bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f862013ec80 0x7f86201574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86201419f0 0x7f862013b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86200725b0 con 0x7f86201419f0 2024-03-24T00:35:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8620072430 con 0x7f86200ce800 2024-03-24T00:35:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8625c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8620169d50 con 0x7f862013ec80 2024-03-24T00:35:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f861f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86201419f0 0x7f862013b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8617e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f862013ec80 0x7f86201574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:01.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f861ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86200ce800 0x7f862013bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:01.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f861ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86200ce800 0x7f862013bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:43052/0 (socket says 172.21.15.40:43052) 2024-03-24T00:35:01.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f861ea00640 1 -- 172.21.15.40:0/2208232892 learned_addr learned my addr 172.21.15.40:0/2208232892 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:01.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f8617e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f862013ec80 0x7f86201574d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:49202/0 (socket says 172.21.15.40:49202) 2024-03-24T00:35:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f861ea00640 1 -- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f862013ec80 msgr2=0x7f86201574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f861ea00640 1 --2- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f862013ec80 0x7f86201574d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f861ea00640 1 -- 172.21.15.40:0/2208232892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86201419f0 msgr2=0x7f862013b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f861ea00640 1 --2- 172.21.15.40:0/2208232892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86201419f0 0x7f862013b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.267+0000 7f861ea00640 1 -- 172.21.15.40:0/2208232892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862014ed10 con 0x7f86200ce800 2024-03-24T00:35:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.271+0000 7f861ea00640 1 --2- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86200ce800 0x7f862013bea0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f86100ac040 tx=0x7f861001ace0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.271+0000 7f8616a00640 1 -- 172.21.15.40:0/2208232892 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86100aac00 con 0x7f86200ce800 2024-03-24T00:35:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.271+0000 7f8616a00640 1 -- 172.21.15.40:0/2208232892 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86100aada0 con 0x7f86200ce800 2024-03-24T00:35:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.271+0000 7f8616a00640 1 -- 172.21.15.40:0/2208232892 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86100be650 con 0x7f86200ce800 2024-03-24T00:35:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.271+0000 7f8625c00640 1 -- 172.21.15.40:0/2208232892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f862014eff0 con 0x7f86200ce800 2024-03-24T00:35:01.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.271+0000 7f8625c00640 1 -- 172.21.15.40:0/2208232892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8620157a10 con 0x7f86200ce800 2024-03-24T00:35:01.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.271+0000 7f8625c00640 1 -- 172.21.15.40:0/2208232892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8620068170 con 0x7f86200ce800 2024-03-24T00:35:01.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.271+0000 7f8616a00640 1 -- 172.21.15.40:0/2208232892 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f86100bc020 con 0x7f86200ce800 2024-03-24T00:35:01.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.275+0000 7f8616a00640 1 --2- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f85fc067660 0x7f85fc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:01.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.275+0000 7f8616a00640 1 -- 172.21.15.40:0/2208232892 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f861011f840 con 0x7f86200ce800 2024-03-24T00:35:01.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.275+0000 7f8617e00640 1 --2- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f85fc067660 0x7f85fc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:01.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.275+0000 7f8617e00640 1 --2- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f85fc067660 0x7f85fc069b20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f862006f770 tx=0x7f8608009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:01.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.279+0000 7f8616a00640 1 -- 172.21.15.40:0/2208232892 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8610016090 con 0x7f86200ce800 2024-03-24T00:35:01.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.475+0000 7f8625c00640 1 -- 172.21.15.40:0/2208232892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8620072b90 con 0x7f86200ce800 2024-03-24T00:35:01.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.475+0000 7f8616a00640 1 -- 172.21.15.40:0/2208232892 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+5015 (secure 0 0 0) 0x7f86100eff90 con 0x7f86200ce800 2024-03-24T00:35:01.479 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:01.479 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":58,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6277,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1633817617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1633817617},{"type":"v1","addr":"172.21.15.53:6837","nonce":1633817617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47}],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:34:12.019118+0000","modified":"2024-03-24T00:34:35.695160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6474,"mds_1":6471,"mds_2":6468},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6468":{"gid":6468,"name":"a","rank":2,"incarnation":56,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6834/1278515461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1278515461},{"type":"v1","addr":"172.21.15.40:6834","nonce":1278515461}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6471":{"gid":6471,"name":"c","rank":1,"incarnation":54,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6835/1312292485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1312292485},{"type":"v1","addr":"172.21.15.40:6835","nonce":1312292485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6474":{"gid":6474,"name":"d","rank":0,"incarnation":48,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/501079401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":501079401},{"type":"v1","addr":"172.21.15.53:6835","nonce":501079401}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6474,"qdb_cluster":[6474,6471,6468]},"id":4}]} 2024-03-24T00:35:01.479 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 58 2024-03-24T00:35:01.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.479+0000 7f8625c00640 1 -- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f85fc067660 msgr2=0x7f85fc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.479+0000 7f8625c00640 1 --2- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f85fc067660 0x7f85fc069b20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f862006f770 tx=0x7f8608009040 comp rx=0 tx=0).stop 2024-03-24T00:35:01.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.479+0000 7f8625c00640 1 -- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86200ce800 msgr2=0x7f862013bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.479+0000 7f8625c00640 1 --2- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86200ce800 0x7f862013bea0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f86100ac040 tx=0x7f861001ace0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.479+0000 7f8625c00640 1 -- 172.21.15.40:0/2208232892 shutdown_connections 2024-03-24T00:35:01.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.479+0000 7f8625c00640 1 --2- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f85fc067660 0x7f85fc069b20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.479+0000 7f8625c00640 1 --2- 172.21.15.40:0/2208232892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86201419f0 0x7f862013b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.479+0000 7f8625c00640 1 --2- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f862013ec80 0x7f86201574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.479+0000 7f8625c00640 1 --2- 172.21.15.40:0/2208232892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86200ce800 0x7f862013bea0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.479+0000 7f8625c00640 1 -- 172.21.15.40:0/2208232892 >> 172.21.15.40:0/2208232892 conn(0x7f86200b17b0 msgr2=0x7f8620071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:01.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.479+0000 7f8625c00640 1 -- 172.21.15.40:0/2208232892 shutdown_connections 2024-03-24T00:35:01.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.479+0000 7f8625c00640 1 -- 172.21.15.40:0/2208232892 wait complete. 2024-03-24T00:35:01.502 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:01.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.639+0000 7fc2fda00640 1 Processor -- start 2024-03-24T00:35:01.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.639+0000 7fc2fda00640 1 -- start start 2024-03-24T00:35:01.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.639+0000 7fc2fda00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc2f80cfdf0 0x7fc2f80d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:01.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.639+0000 7fc2fda00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc2f815f7a0 0x7fc2f80d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:01.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.639+0000 7fc2fda00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f80af710 con 0x7fc2f80b43a0 2024-03-24T00:35:01.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.639+0000 7fc2fda00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f80b2bb0 con 0x7fc2f80cfdf0 2024-03-24T00:35:01.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.639+0000 7fc2fda00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f80b26a0 con 0x7fc2f815f7a0 2024-03-24T00:35:01.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.643+0000 7fc2f6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc2f80cfdf0 0x7fc2f80d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:01.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.643+0000 7fc2f7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc2f815f7a0 0x7fc2f80d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:01.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.643+0000 7fc2f6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc2f80cfdf0 0x7fc2f80d01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46636/0 (socket says 172.21.15.40:46636) 2024-03-24T00:35:01.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.643+0000 7fc2f6a00640 1 -- 172.21.15.40:0/2265622675 learned_addr learned my addr 172.21.15.40:0/2265622675 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:01.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.643+0000 7fc2f7400640 1 -- 172.21.15.40:0/2265622675 >> v2:172.21.15.40:3300/0 conn(0x7fc2f80cfdf0 msgr2=0x7fc2f80d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:01.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.643+0000 7fc2f7400640 1 --2- 172.21.15.40:0/2265622675 >> v2:172.21.15.40:3300/0 conn(0x7fc2f80cfdf0 0x7fc2f80d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.643+0000 7fc2f7400640 1 -- 172.21.15.40:0/2265622675 >> v1:172.21.15.53:6789/0 conn(0x7fc2f80b43a0 legacy=0x7fc2f815f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.643+0000 7fc2f7400640 1 -- 172.21.15.40:0/2265622675 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f80b34f0 con 0x7fc2f815f7a0 2024-03-24T00:35:01.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.643+0000 7fc2f7400640 1 --2- 172.21.15.40:0/2265622675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f815f7a0 0x7fc2f80d0710 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8073290 tx=0x7fc2f001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78b3e3797168b1ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:01.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.643+0000 7fc2f6000640 1 -- 172.21.15.40:0/2265622675 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00b0070 con 0x7fc2f815f7a0 2024-03-24T00:35:01.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.643+0000 7fc2f6000640 1 -- 172.21.15.40:0/2265622675 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f00ab600 con 0x7fc2f815f7a0 2024-03-24T00:35:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.643+0000 7fc2f6000640 1 -- 172.21.15.40:0/2265622675 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f00ab9d0 con 0x7fc2f815f7a0 2024-03-24T00:35:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 -- 172.21.15.40:0/2265622675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f815f7a0 msgr2=0x7fc2f80d0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 --2- 172.21.15.40:0/2265622675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f815f7a0 0x7fc2f80d0710 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc2f8073290 tx=0x7fc2f001c480 comp rx=0 tx=0).stop 2024-03-24T00:35:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 -- 172.21.15.40:0/2265622675 shutdown_connections 2024-03-24T00:35:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 --2- 172.21.15.40:0/2265622675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f815f7a0 0x7fc2f80d0710 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 --2- 172.21.15.40:0/2265622675 >> v2:172.21.15.40:3300/0 conn(0x7fc2f80cfdf0 0x7fc2f80d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 -- 172.21.15.40:0/2265622675 >> 172.21.15.40:0/2265622675 conn(0x7fc2f80b3880 msgr2=0x7fc2f80b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 -- 172.21.15.40:0/2265622675 shutdown_connections 2024-03-24T00:35:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 -- 172.21.15.40:0/2265622675 wait complete. 2024-03-24T00:35:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 Processor -- start 2024-03-24T00:35:01.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 -- start start 2024-03-24T00:35:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f80b43a0 0x7fc2f8159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f80cfdf0 0x7fc2f80cdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f815f7a0 0x7fc2f80cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f80b0790 con 0x7fc2f80b43a0 2024-03-24T00:35:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f80b0610 con 0x7fc2f80cfdf0 2024-03-24T00:35:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2fda00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f80b0910 con 0x7fc2f815f7a0 2024-03-24T00:35:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f80b43a0 0x7fc2f8159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f80b43a0 0x7fc2f8159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46640/0 (socket says 172.21.15.40:46640) 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2f6a00640 1 -- 172.21.15.40:0/3875214584 learned_addr learned my addr 172.21.15.40:0/3875214584 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2ee000640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f80cfdf0 0x7fc2f80cdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2f7400640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f815f7a0 0x7fc2f80cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2f6a00640 1 -- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f815f7a0 msgr2=0x7fc2f80cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2f6a00640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f815f7a0 0x7fc2f80cdfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2f6a00640 1 -- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f80cfdf0 msgr2=0x7fc2f80cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2f6a00640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f80cfdf0 0x7fc2f80cdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.647+0000 7fc2f6a00640 1 -- 172.21.15.40:0/3875214584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f80c5f60 con 0x7fc2f80b43a0 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.651+0000 7fc2f6a00640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f80b43a0 0x7fc2f8159b00 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009d90 tx=0x7fc2e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.651+0000 7fc2ef400640 1 -- 172.21.15.40:0/3875214584 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e800cda0 con 0x7fc2f80b43a0 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.651+0000 7fc2ef400640 1 -- 172.21.15.40:0/3875214584 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e8019070 con 0x7fc2f80b43a0 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.651+0000 7fc2ef400640 1 -- 172.21.15.40:0/3875214584 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8014680 con 0x7fc2f80b43a0 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.651+0000 7fc2fda00640 1 -- 172.21.15.40:0/3875214584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f80c6240 con 0x7fc2f80b43a0 2024-03-24T00:35:01.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.651+0000 7fc2fda00640 1 -- 172.21.15.40:0/3875214584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f80c6800 con 0x7fc2f80b43a0 2024-03-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.651+0000 7fc2e7e00640 1 -- 172.21.15.40:0/3875214584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f8068170 con 0x7fc2f80b43a0 2024-03-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.651+0000 7fc2ef400640 1 -- 172.21.15.40:0/3875214584 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc2e8012050 con 0x7fc2f80b43a0 2024-03-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.651+0000 7fc2ef400640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc2d4067660 0x7fc2d4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:01.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.651+0000 7fc2ef400640 1 -- 172.21.15.40:0/3875214584 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2e8085d20 con 0x7fc2f80b43a0 2024-03-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.655+0000 7fc2ee000640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc2d4067660 0x7fc2d4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:01.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.655+0000 7fc2ee000640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc2d4067660 0x7fc2d4069b20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc2f815ad60 tx=0x7fc2e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:01.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.659+0000 7fc2ef400640 1 -- 172.21.15.40:0/3875214584 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc2e8010040 con 0x7fc2f80b43a0 2024-03-24T00:35:01.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.819+0000 7fc2e7e00640 1 -- 172.21.15.40:0/3875214584 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2f8166960 con 0x7fc2f80b43a0 2024-03-24T00:35:01.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.819+0000 7fc2ef400640 1 -- 172.21.15.40:0/3875214584 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16605 (secure 0 0 0) 0x7fc2e80562b0 con 0x7fc2f80b43a0 2024-03-24T00:35:01.824 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:01.824 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":59,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:34:13.623804+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:34:10.428162+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-03-24T00:34:11.001094+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.823+0000 7fc2e7e00640 1 -- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc2d4067660 msgr2=0x7fc2d4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.823+0000 7fc2e7e00640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc2d4067660 0x7fc2d4069b20 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc2f815ad60 tx=0x7fc2e0009040 comp rx=0 tx=0).stop 2024-03-24T00:35:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.823+0000 7fc2e7e00640 1 -- 172.21.15.40:0/3875214584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f80b43a0 msgr2=0x7fc2f8159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.823+0000 7fc2e7e00640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f80b43a0 0x7fc2f8159b00 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8009d90 tx=0x7fc2e800c490 comp rx=0 tx=0).stop 2024-03-24T00:35:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.823+0000 7fc2e7e00640 1 -- 172.21.15.40:0/3875214584 shutdown_connections 2024-03-24T00:35:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.823+0000 7fc2e7e00640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc2d4067660 0x7fc2d4069b20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.823+0000 7fc2e7e00640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2f815f7a0 0x7fc2f80cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.823+0000 7fc2e7e00640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2f80cfdf0 0x7fc2f80cdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.823+0000 7fc2e7e00640 1 --2- 172.21.15.40:0/3875214584 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc2f80b43a0 0x7fc2f8159b00 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.823+0000 7fc2e7e00640 1 -- 172.21.15.40:0/3875214584 >> 172.21.15.40:0/3875214584 conn(0x7fc2f80b3880 msgr2=0x7fc2f80d1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.823+0000 7fc2e7e00640 1 -- 172.21.15.40:0/3875214584 shutdown_connections 2024-03-24T00:35:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.823+0000 7fc2e7e00640 1 -- 172.21.15.40:0/3875214584 wait complete. 2024-03-24T00:35:01.855 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:35:02.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.995+0000 7f8bc6000640 1 Processor -- start 2024-03-24T00:35:02.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.995+0000 7f8bc6000640 1 -- start start 2024-03-24T00:35:02.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bc6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8bc00b0ed0 0x7f8bc0070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:02.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bc6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc0160e10 con 0x7f8bc0070250 2024-03-24T00:35:02.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bc6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc00af420 con 0x7f8bc00d01d0 2024-03-24T00:35:02.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bc6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc00d27d0 con 0x7f8bc00b0ed0 2024-03-24T00:35:02.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bbea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8bc0070250 0x7f8bc0160700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56450/0 (socket says 172.21.15.40:56450) 2024-03-24T00:35:02.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bbea00640 1 -- 172.21.15.40:0/2530204343 learned_addr learned my addr 172.21.15.40:0/2530204343 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:02.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bbfe00640 1 --2- 172.21.15.40:0/2530204343 >> v2:172.21.15.53:3300/0 conn(0x7f8bc00b0ed0 0x7f8bc0070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:02.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bbe000640 1 -- 172.21.15.40:0/2530204343 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935655432 0 0) 0x7f8bc0160e10 con 0x7f8bc0070250 2024-03-24T00:35:02.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bbe000640 1 -- 172.21.15.40:0/2530204343 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ba4003530 con 0x7f8bc0070250 2024-03-24T00:35:02.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bbe000640 1 -- 172.21.15.40:0/2530204343 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2711365233 0 0) 0x7f8bc00af420 con 0x7f8bc00d01d0 2024-03-24T00:35:02.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bbe000640 1 -- 172.21.15.40:0/2530204343 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bc0160e10 con 0x7f8bc00d01d0 2024-03-24T00:35:02.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bbfe00640 1 -- 172.21.15.40:0/2530204343 >> v1:172.21.15.53:6789/0 conn(0x7f8bc00d01d0 legacy=0x7f8bc015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:02.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bbfe00640 1 -- 172.21.15.40:0/2530204343 >> v1:172.21.15.40:6789/0 conn(0x7f8bc0070250 legacy=0x7f8bc0160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:02.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:01.999+0000 7f8bbfe00640 1 -- 172.21.15.40:0/2530204343 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc00d2a20 con 0x7f8bc00b0ed0 2024-03-24T00:35:02.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bbfe00640 1 --2- 172.21.15.40:0/2530204343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bc00b0ed0 0x7f8bc0070b20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009d90 tx=0x7f8bb801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a464d95f2067b8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:02.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bbe000640 1 -- 172.21.15.40:0/2530204343 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb80b2070 con 0x7f8bc00b0ed0 2024-03-24T00:35:02.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bbe000640 1 -- 172.21.15.40:0/2530204343 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc0070080 con 0x7f8bc00b0ed0 2024-03-24T00:35:02.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bbe000640 1 -- 172.21.15.40:0/2530204343 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb80a5b60 con 0x7f8bc00b0ed0 2024-03-24T00:35:02.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bc6000640 1 -- 172.21.15.40:0/2530204343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bc00b0ed0 msgr2=0x7f8bc0070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:02.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bc6000640 1 --2- 172.21.15.40:0/2530204343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bc00b0ed0 0x7f8bc0070b20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009d90 tx=0x7f8bb801c970 comp rx=0 tx=0).stop 2024-03-24T00:35:02.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bc6000640 1 -- 172.21.15.40:0/2530204343 shutdown_connections 2024-03-24T00:35:02.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bc6000640 1 --2- 172.21.15.40:0/2530204343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bc00b0ed0 0x7f8bc0070b20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:02.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bc6000640 1 -- 172.21.15.40:0/2530204343 >> 172.21.15.40:0/2530204343 conn(0x7f8bc00d1cb0 msgr2=0x7f8bc00d20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:02.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bc6000640 1 -- 172.21.15.40:0/2530204343 shutdown_connections 2024-03-24T00:35:02.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bc6000640 1 -- 172.21.15.40:0/2530204343 wait complete. 2024-03-24T00:35:02.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bc6000640 1 Processor -- start 2024-03-24T00:35:02.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.003+0000 7f8bc6000640 1 -- start start 2024-03-24T00:35:02.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bc6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bc00b0ed0 0x7f8bc0153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:02.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bc6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bc00d01d0 0x7f8bc0153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:02.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bc6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8bc0166140 0x7f8bc0157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:02.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bc6000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc0169e60 con 0x7f8bc00d01d0 2024-03-24T00:35:02.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bc6000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc0169ce0 con 0x7f8bc00b0ed0 2024-03-24T00:35:02.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bc6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc0169fe0 con 0x7f8bc0166140 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bbea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bc00d01d0 0x7f8bc0153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bbea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bc00d01d0 0x7f8bc0153370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46642/0 (socket says 172.21.15.40:46642) 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bbea00640 1 -- 172.21.15.40:0/3173352766 learned_addr learned my addr 172.21.15.40:0/3173352766 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bbf400640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bc00b0ed0 0x7f8bc0153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bbfe00640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8bc0166140 0x7f8bc0157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bbea00640 1 -- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8bc0166140 msgr2=0x7f8bc0157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bbea00640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8bc0166140 0x7f8bc0157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bbea00640 1 -- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bc00b0ed0 msgr2=0x7f8bc0153fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bbea00640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bc00b0ed0 0x7f8bc0153fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bbea00640 1 -- 172.21.15.40:0/3173352766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc01538b0 con 0x7f8bc00d01d0 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bbea00640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bc00d01d0 0x7f8bc0153370 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f8bac00c420 tx=0x7f8bac00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bb7400640 1 -- 172.21.15.40:0/3173352766 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac017070 con 0x7f8bc00d01d0 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bb7400640 1 -- 172.21.15.40:0/3173352766 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac005cf0 con 0x7f8bc00d01d0 2024-03-24T00:35:02.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bc6000640 1 -- 172.21.15.40:0/3173352766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc006ccb0 con 0x7f8bc00d01d0 2024-03-24T00:35:02.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bb7400640 1 -- 172.21.15.40:0/3173352766 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac0072d0 con 0x7f8bc00d01d0 2024-03-24T00:35:02.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bc6000640 1 -- 172.21.15.40:0/3173352766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc0165660 con 0x7f8bc00d01d0 2024-03-24T00:35:02.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bc6000640 1 -- 172.21.15.40:0/3173352766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc0068170 con 0x7f8bc00d01d0 2024-03-24T00:35:02.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.007+0000 7f8bb7400640 1 -- 172.21.15.40:0/3173352766 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8bac01a070 con 0x7f8bc00d01d0 2024-03-24T00:35:02.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.011+0000 7f8bb7400640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8ba0067690 0x7f8ba0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:02.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.011+0000 7f8bb7400640 1 -- 172.21.15.40:0/3173352766 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8bac084af0 con 0x7f8bc00d01d0 2024-03-24T00:35:02.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.011+0000 7f8bbf400640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8ba0067690 0x7f8ba0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:02.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.011+0000 7f8bbf400640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8ba0067690 0x7f8ba0069b50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0007fd0 tx=0x7f8bb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:02.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.019+0000 7f8bb7400640 1 -- 172.21.15.40:0/3173352766 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8bac0551b0 con 0x7f8bc00d01d0 2024-03-24T00:35:02.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.215+0000 7f8bc6000640 1 -- 172.21.15.40:0/3173352766 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8bc0155560 con 0x7f8bc00d01d0 2024-03-24T00:35:02.218 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-03-24T00:35:02.215+0000 7fa64e600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:35:02.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.219+0000 7f8bb7400640 1 -- 172.21.15.40:0/3173352766 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v59) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8bac059250 con 0x7f8bc00d01d0 2024-03-24T00:35:02.220 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:35:02.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.219+0000 7f8bc6000640 1 -- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8ba0067690 msgr2=0x7f8ba0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:02.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.219+0000 7f8bc6000640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8ba0067690 0x7f8ba0069b50 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0007fd0 tx=0x7f8bb0009040 comp rx=0 tx=0).stop 2024-03-24T00:35:02.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.219+0000 7f8bc6000640 1 -- 172.21.15.40:0/3173352766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bc00d01d0 msgr2=0x7f8bc0153370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:02.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.219+0000 7f8bc6000640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bc00d01d0 0x7f8bc0153370 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f8bac00c420 tx=0x7f8bac00c8e0 comp rx=0 tx=0).stop 2024-03-24T00:35:02.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.223+0000 7f8bc6000640 1 -- 172.21.15.40:0/3173352766 shutdown_connections 2024-03-24T00:35:02.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.223+0000 7f8bc6000640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8ba0067690 0x7f8ba0069b50 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:02.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.223+0000 7f8bc6000640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8bc0166140 0x7f8bc0157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:02.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.223+0000 7f8bc6000640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8bc00d01d0 0x7f8bc0153370 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:02.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.223+0000 7f8bc6000640 1 --2- 172.21.15.40:0/3173352766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bc00b0ed0 0x7f8bc0153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:02.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.223+0000 7f8bc6000640 1 -- 172.21.15.40:0/3173352766 >> 172.21.15.40:0/3173352766 conn(0x7f8bc00d1cb0 msgr2=0x7f8bc00d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:02.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.223+0000 7f8bc6000640 1 -- 172.21.15.40:0/3173352766 shutdown_connections 2024-03-24T00:35:02.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.223+0000 7f8bc6000640 1 -- 172.21.15.40:0/3173352766 wait complete. 2024-03-24T00:35:02.254 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:35:02.288 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:35:02.290 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:35:02.291 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:35:02.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.399+0000 7f8896000640 1 Processor -- start 2024-03-24T00:35:02.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.399+0000 7f8896000640 1 -- start start 2024-03-24T00:35:02.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.399+0000 7f8896000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f889015ac60 0x7f88900b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:02.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.399+0000 7f8896000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88900b3260 con 0x7f8890071b00 2024-03-24T00:35:02.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.399+0000 7f8896000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88900b3540 con 0x7f889006cb80 2024-03-24T00:35:02.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.399+0000 7f8896000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88900b3e70 con 0x7f889015ac60 2024-03-24T00:35:02.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.399+0000 7f888f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f889015ac60 0x7f88900b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:02.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.403+0000 7f888f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f889015ac60 0x7f88900b2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46666/0 (socket says 172.21.15.40:46666) 2024-03-24T00:35:02.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.403+0000 7f888fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8890071b00 0x7f88900ccd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56484/0 (socket says 172.21.15.40:56484) 2024-03-24T00:35:02.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.403+0000 7f888f400640 1 -- 172.21.15.40:0/1588858412 learned_addr learned my addr 172.21.15.40:0/1588858412 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:02.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.403+0000 7f888f400640 1 -- 172.21.15.40:0/1588858412 >> v1:172.21.15.53:6789/0 conn(0x7f889006cb80 legacy=0x7f88900b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:02.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.403+0000 7f888f400640 1 -- 172.21.15.40:0/1588858412 >> v1:172.21.15.40:6789/0 conn(0x7f8890071b00 legacy=0x7f88900ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:02.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.403+0000 7f888f400640 1 -- 172.21.15.40:0/1588858412 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88900b3820 con 0x7f889015ac60 2024-03-24T00:35:02.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.403+0000 7f888f400640 1 --2- 172.21.15.40:0/1588858412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f889015ac60 0x7f88900b2960 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f8880009e50 tx=0x7f888001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f0dbf91a33596d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:02.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.403+0000 7f888ea00640 1 -- 172.21.15.40:0/1588858412 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800a5850 con 0x7f889015ac60 2024-03-24T00:35:02.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.403+0000 7f888ea00640 1 -- 172.21.15.40:0/1588858412 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88800a59f0 con 0x7f889015ac60 2024-03-24T00:35:02.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.403+0000 7f888ea00640 1 -- 172.21.15.40:0/1588858412 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800a5dc0 con 0x7f889015ac60 2024-03-24T00:35:02.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 -- 172.21.15.40:0/1588858412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f889015ac60 msgr2=0x7f88900b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:02.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 --2- 172.21.15.40:0/1588858412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f889015ac60 0x7f88900b2960 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f8880009e50 tx=0x7f888001c970 comp rx=0 tx=0).stop 2024-03-24T00:35:02.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 -- 172.21.15.40:0/1588858412 shutdown_connections 2024-03-24T00:35:02.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 --2- 172.21.15.40:0/1588858412 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f889015ac60 0x7f88900b2960 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:02.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 -- 172.21.15.40:0/1588858412 >> 172.21.15.40:0/1588858412 conn(0x7f88900b1660 msgr2=0x7f88900b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:02.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 -- 172.21.15.40:0/1588858412 shutdown_connections 2024-03-24T00:35:02.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 -- 172.21.15.40:0/1588858412 wait complete. 2024-03-24T00:35:02.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 Processor -- start 2024-03-24T00:35:02.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 -- start start 2024-03-24T00:35:02.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f889006cb80 0x7f8890148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:02.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8890071b00 0x7f889015d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:02.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f889015ac60 0x7f8890158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88900b3020 con 0x7f8890071b00 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88900b2ea0 con 0x7f889015ac60 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8896000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f889015b040 con 0x7f889006cb80 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f888f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f889006cb80 0x7f8890148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f8886a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8890071b00 0x7f889015d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f888f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f889006cb80 0x7f8890148d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:49264/0 (socket says 172.21.15.40:49264) 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f888f400640 1 -- 172.21.15.40:0/4268057212 learned_addr learned my addr 172.21.15.40:0/4268057212 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f888fe00640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f889015ac60 0x7f8890158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f888f400640 1 -- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f889015ac60 msgr2=0x7f8890158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f888f400640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f889015ac60 0x7f8890158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f888f400640 1 -- 172.21.15.40:0/4268057212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8890071b00 msgr2=0x7f889015d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f888f400640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8890071b00 0x7f889015d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f888f400640 1 -- 172.21.15.40:0/4268057212 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88901492a0 con 0x7f889006cb80 2024-03-24T00:35:02.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.407+0000 7f888f400640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f889006cb80 0x7f8890148d60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8880019580 tx=0x7f8880019710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.411+0000 7f8887e00640 1 -- 172.21.15.40:0/4268057212 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800bc020 con 0x7f889006cb80 2024-03-24T00:35:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.411+0000 7f8887e00640 1 -- 172.21.15.40:0/4268057212 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88800b8670 con 0x7f889006cb80 2024-03-24T00:35:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.411+0000 7f8887e00640 1 -- 172.21.15.40:0/4268057212 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88800b8a60 con 0x7f889006cb80 2024-03-24T00:35:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.411+0000 7f8896000640 1 -- 172.21.15.40:0/4268057212 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889015c6d0 con 0x7f889006cb80 2024-03-24T00:35:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.411+0000 7f8896000640 1 -- 172.21.15.40:0/4268057212 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889015cc50 con 0x7f889006cb80 2024-03-24T00:35:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.411+0000 7f8896000640 1 -- 172.21.15.40:0/4268057212 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8890068170 con 0x7f889006cb80 2024-03-24T00:35:02.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.411+0000 7f8887e00640 1 -- 172.21.15.40:0/4268057212 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f88800bf020 con 0x7f889006cb80 2024-03-24T00:35:02.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.411+0000 7f8887e00640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f886c067690 0x7f886c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:02.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.411+0000 7f8887e00640 1 -- 172.21.15.40:0/4268057212 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f8880120e60 con 0x7f889006cb80 2024-03-24T00:35:02.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.411+0000 7f8886a00640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f886c067690 0x7f886c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:02.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.411+0000 7f8886a00640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f886c067690 0x7f886c069b50 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f88901fa990 tx=0x7f8878009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:02.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.415+0000 7f8887e00640 1 -- 172.21.15.40:0/4268057212 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f88901492a0 con 0x7f889006cb80 2024-03-24T00:35:02.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:02.611+0000 7f8896000640 1 -- 172.21.15.40:0/4268057212 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f88901499f0 con 0x7f889006cb80 2024-03-24T00:35:03.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.223+0000 7f8887e00640 1 -- 172.21.15.40:0/4268057212 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v60) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8880016620 con 0x7f889006cb80 2024-03-24T00:35:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.227+0000 7f8896000640 1 -- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f886c067690 msgr2=0x7f886c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:03.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.227+0000 7f8896000640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f886c067690 0x7f886c069b50 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f88901fa990 tx=0x7f8878009040 comp rx=0 tx=0).stop 2024-03-24T00:35:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.227+0000 7f8896000640 1 -- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f889006cb80 msgr2=0x7f8890148d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.227+0000 7f8896000640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f889006cb80 0x7f8890148d60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8880019580 tx=0x7f8880019710 comp rx=0 tx=0).stop 2024-03-24T00:35:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.227+0000 7f8896000640 1 -- 172.21.15.40:0/4268057212 shutdown_connections 2024-03-24T00:35:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.227+0000 7f8896000640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f886c067690 0x7f886c069b50 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.227+0000 7f8896000640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f889015ac60 0x7f8890158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.227+0000 7f8896000640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8890071b00 0x7f889015d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.227+0000 7f8896000640 1 --2- 172.21.15.40:0/4268057212 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f889006cb80 0x7f8890148d60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.227+0000 7f8896000640 1 -- 172.21.15.40:0/4268057212 >> 172.21.15.40:0/4268057212 conn(0x7f88900b1660 msgr2=0x7f88900cede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.227+0000 7f8896000640 1 -- 172.21.15.40:0/4268057212 shutdown_connections 2024-03-24T00:35:03.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.227+0000 7f8896000640 1 -- 172.21.15.40:0/4268057212 wait complete. 2024-03-24T00:35:03.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:35:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.439+0000 7fb3fd800640 1 Processor -- start 2024-03-24T00:35:03.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.439+0000 7fb3fd800640 1 -- start start 2024-03-24T00:35:03.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3fd800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f815ffd0 0x7fb3f80ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:03.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3fd800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f8071de0 con 0x7fb3f80728d0 2024-03-24T00:35:03.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3fd800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f80cea50 con 0x7fb3f815bd00 2024-03-24T00:35:03.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3fd800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f80ced30 con 0x7fb3f815ffd0 2024-03-24T00:35:03.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f7400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb3f80728d0 0x7fb3f806cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56494/0 (socket says 172.21.15.40:56494) 2024-03-24T00:35:03.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f7400640 1 -- 172.21.15.40:0/1643660738 learned_addr learned my addr 172.21.15.40:0/1643660738 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3efe00640 1 --2- 172.21.15.40:0/1643660738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f815ffd0 0x7fb3f80ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880347694 0 0) 0x7fb3f8071de0 con 0x7fb3f80728d0 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3dc003530 con 0x7fb3f80728d0 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1570134822 0 0) 0x7fb3dc003530 con 0x7fb3f80728d0 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f815ffd0 msgr2=0x7fb3f80ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 --2- 172.21.15.40:0/1643660738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f815ffd0 0x7fb3f80ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 >> v1:172.21.15.53:6789/0 conn(0x7fb3f815bd00 legacy=0x7fb3f815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f8071b00 con 0x7fb3f80728d0 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fb3f0001a00 con 0x7fb3f80728d0 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 >> v1:172.21.15.40:6789/0 conn(0x7fb3f80728d0 legacy=0x7fb3f806cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 --2- 172.21.15.40:0/1643660738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f815ffd0 0x7fb3dc0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 --2- 172.21.15.40:0/1643660738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3dc003f70 0x7fb3dc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 --2- 172.21.15.40:0/1643660738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3dc004d50 0x7fb3dc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f80ced30 con 0x7fb3dc004d50 2024-03-24T00:35:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc00bab0 con 0x7fb3dc003f70 2024-03-24T00:35:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc00bcb0 con 0x7fb3f815ffd0 2024-03-24T00:35:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f7400640 1 --2- 172.21.15.40:0/1643660738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3dc004d50 0x7fb3dc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f6a00640 1 --2- 172.21.15.40:0/1643660738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f815ffd0 0x7fb3dc0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3efe00640 1 --2- 172.21.15.40:0/1643660738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3dc003f70 0x7fb3dc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f7400640 1 -- 172.21.15.40:0/1643660738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f815ffd0 msgr2=0x7fb3dc0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f7400640 1 --2- 172.21.15.40:0/1643660738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f815ffd0 0x7fb3dc0039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f7400640 1 -- 172.21.15.40:0/1643660738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3dc003f70 msgr2=0x7fb3dc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f7400640 1 --2- 172.21.15.40:0/1643660738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3dc003f70 0x7fb3dc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.443+0000 7fb3f7400640 1 -- 172.21.15.40:0/1643660738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc003530 con 0x7fb3dc004d50 2024-03-24T00:35:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.447+0000 7fb3f7400640 1 --2- 172.21.15.40:0/1643660738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3dc004d50 0x7fb3dc0051d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000a840 tx=0x7fb3f001d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1cc661e3d64050 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.447+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f000a460 con 0x7fb3dc004d50 2024-03-24T00:35:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.447+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f00ad650 con 0x7fb3dc004d50 2024-03-24T00:35:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.447+0000 7fb3f6000640 1 -- 172.21.15.40:0/1643660738 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f00ada80 con 0x7fb3dc004d50 2024-03-24T00:35:03.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.447+0000 7fb3fd800640 1 -- 172.21.15.40:0/1643660738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3dc004d50 msgr2=0x7fb3dc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:03.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.447+0000 7fb3fd800640 1 --2- 172.21.15.40:0/1643660738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3dc004d50 0x7fb3dc0051d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000a840 tx=0x7fb3f001d360 comp rx=0 tx=0).stop 2024-03-24T00:35:03.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.447+0000 7fb3f7400640 1 -- 172.21.15.40:0/1643660738 reap_dead start 2024-03-24T00:35:03.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.447+0000 7fb3fd800640 1 -- 172.21.15.40:0/1643660738 shutdown_connections 2024-03-24T00:35:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.447+0000 7fb3fd800640 1 -- 172.21.15.40:0/1643660738 >> 172.21.15.40:0/1643660738 conn(0x7fb3f80b1660 msgr2=0x7fb3f80b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.447+0000 7fb3fd800640 1 -- 172.21.15.40:0/1643660738 shutdown_connections 2024-03-24T00:35:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.447+0000 7fb3fd800640 1 -- 172.21.15.40:0/1643660738 wait complete. 2024-03-24T00:35:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.447+0000 7fb3fd800640 1 Processor -- start 2024-03-24T00:35:03.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3fd800640 1 -- start start 2024-03-24T00:35:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3fd800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3dc004d50 0x7fb3f8145410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3fd800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f81e6ec0 0x7fb3f81e7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3fd800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f80b33e0 0x7fb3f80b3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3fd800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f815c0e0 con 0x7fb3dc004d50 2024-03-24T00:35:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3fd800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f8072d00 con 0x7fb3f80b33e0 2024-03-24T00:35:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3fd800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f80ae8e0 con 0x7fb3f81e6ec0 2024-03-24T00:35:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3dc004d50 0x7fb3f8145410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3dc004d50 0x7fb3f8145410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46714/0 (socket says 172.21.15.40:46714) 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3f6a00640 1 -- 172.21.15.40:0/1364937846 learned_addr learned my addr 172.21.15.40:0/1364937846 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3f6a00640 1 -- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f81e6ec0 msgr2=0x7fb3f81e7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3f6a00640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f81e6ec0 0x7fb3f81e7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3f6a00640 1 -- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f80b33e0 msgr2=0x7fb3f80b3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3f7400640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f80b33e0 0x7fb3f80b3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3f6a00640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f80b33e0 0x7fb3f80b3860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3f6a00640 1 -- 172.21.15.40:0/1364937846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f8148e00 con 0x7fb3dc004d50 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3f6a00640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3dc004d50 0x7fb3f8145410 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800f7f0 tx=0x7fb3e800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3eea00640 1 -- 172.21.15.40:0/1364937846 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e80050d0 con 0x7fb3dc004d50 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3eea00640 1 -- 172.21.15.40:0/1364937846 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e8018070 con 0x7fb3dc004d50 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3eea00640 1 -- 172.21.15.40:0/1364937846 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e800d730 con 0x7fb3dc004d50 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3fd800640 1 -- 172.21.15.40:0/1364937846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f81e7a30 con 0x7fb3dc004d50 2024-03-24T00:35:03.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3fd800640 1 -- 172.21.15.40:0/1364937846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f81e7ff0 con 0x7fb3dc004d50 2024-03-24T00:35:03.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.451+0000 7fb3fd800640 1 -- 172.21.15.40:0/1364937846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f8068170 con 0x7fb3dc004d50 2024-03-24T00:35:03.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.455+0000 7fb3eea00640 1 -- 172.21.15.40:0/1364937846 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb3e800d8d0 con 0x7fb3dc004d50 2024-03-24T00:35:03.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.455+0000 7fb3eea00640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3d40676e0 0x7fb3d4069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:03.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.455+0000 7fb3eea00640 1 -- 172.21.15.40:0/1364937846 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb3e80857f0 con 0x7fb3dc004d50 2024-03-24T00:35:03.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.455+0000 7fb3efe00640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3d40676e0 0x7fb3d4069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.455+0000 7fb3efe00640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3d40676e0 0x7fb3d4069ba0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80b1260 tx=0x7fb3e00095f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:03.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.459+0000 7fb3eea00640 1 -- 172.21.15.40:0/1364937846 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb3e80563e0 con 0x7fb3dc004d50 2024-03-24T00:35:03.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:03.627+0000 7fb3fd800640 1 -- 172.21.15.40:0/1364937846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb3f80b44b0 con 0x7fb3dc004d50 2024-03-24T00:35:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.227+0000 7fb3eea00640 1 -- 172.21.15.40:0/1364937846 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v61) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb3e8055ce0 con 0x7fb3dc004d50 2024-03-24T00:35:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:35:04.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.231+0000 7fb3fd800640 1 -- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3d40676e0 msgr2=0x7fb3d4069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:04.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.231+0000 7fb3fd800640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3d40676e0 0x7fb3d4069ba0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb3f80b1260 tx=0x7fb3e00095f0 comp rx=0 tx=0).stop 2024-03-24T00:35:04.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.231+0000 7fb3fd800640 1 -- 172.21.15.40:0/1364937846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3dc004d50 msgr2=0x7fb3f8145410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.231+0000 7fb3fd800640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3dc004d50 0x7fb3f8145410 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb3e800f7f0 tx=0x7fb3e800fcb0 comp rx=0 tx=0).stop 2024-03-24T00:35:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.231+0000 7fb3fd800640 1 -- 172.21.15.40:0/1364937846 shutdown_connections 2024-03-24T00:35:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.231+0000 7fb3fd800640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3d40676e0 0x7fb3d4069ba0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.231+0000 7fb3fd800640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f80b33e0 0x7fb3f80b3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.231+0000 7fb3fd800640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f81e6ec0 0x7fb3f81e7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.231+0000 7fb3fd800640 1 --2- 172.21.15.40:0/1364937846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3dc004d50 0x7fb3f8145410 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.231+0000 7fb3fd800640 1 -- 172.21.15.40:0/1364937846 >> 172.21.15.40:0/1364937846 conn(0x7fb3f80b1660 msgr2=0x7fb3f80cde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.231+0000 7fb3fd800640 1 -- 172.21.15.40:0/1364937846 shutdown_connections 2024-03-24T00:35:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.231+0000 7fb3fd800640 1 -- 172.21.15.40:0/1364937846 wait complete. 2024-03-24T00:35:04.257 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:35:04.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.399+0000 7f9c2aa00640 1 Processor -- start 2024-03-24T00:35:04.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.399+0000 7f9c2aa00640 1 -- start start 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.399+0000 7f9c2aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9c2415bd00 0x7f9c240ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.399+0000 7f9c2aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9c240728d0 0x7f9c240cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.399+0000 7f9c2aa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c24072d60 con 0x7f9c2415f8c0 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c2aa00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c240cf790 con 0x7f9c240728d0 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c2aa00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c24071de0 con 0x7f9c2415bd00 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c1be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9c240728d0 0x7f9c240cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c1be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9c240728d0 0x7f9c240cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46724/0 (socket says 172.21.15.40:46724) 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c1be00640 1 -- 172.21.15.40:0/726348904 learned_addr learned my addr 172.21.15.40:0/726348904 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c23e00640 1 --2- 172.21.15.40:0/726348904 >> v2:172.21.15.53:3300/0 conn(0x7f9c2415bd00 0x7f9c240ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c1be00640 1 -- 172.21.15.40:0/726348904 >> v2:172.21.15.53:3300/0 conn(0x7f9c2415bd00 msgr2=0x7f9c240ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c1be00640 1 --2- 172.21.15.40:0/726348904 >> v2:172.21.15.53:3300/0 conn(0x7f9c2415bd00 0x7f9c240ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c1be00640 1 -- 172.21.15.40:0/726348904 >> v1:172.21.15.53:6789/0 conn(0x7f9c2415f8c0 legacy=0x7f9c2415fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:04.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c1be00640 1 -- 172.21.15.40:0/726348904 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c240cea50 con 0x7f9c240728d0 2024-03-24T00:35:04.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c1be00640 1 --2- 172.21.15.40:0/726348904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c240728d0 0x7f9c240cf250 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f9c240733c0 tx=0x7f9c0c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb3ee3d94f009c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:04.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c23400640 1 -- 172.21.15.40:0/726348904 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c0aa650 con 0x7f9c240728d0 2024-03-24T00:35:04.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c23400640 1 -- 172.21.15.40:0/726348904 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0c0af070 con 0x7f9c240728d0 2024-03-24T00:35:04.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.403+0000 7f9c23400640 1 -- 172.21.15.40:0/726348904 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c0aaa10 con 0x7f9c240728d0 2024-03-24T00:35:04.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 -- 172.21.15.40:0/726348904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c240728d0 msgr2=0x7f9c240cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:04.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 --2- 172.21.15.40:0/726348904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c240728d0 0x7f9c240cf250 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f9c240733c0 tx=0x7f9c0c01c4f0 comp rx=0 tx=0).stop 2024-03-24T00:35:04.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 -- 172.21.15.40:0/726348904 shutdown_connections 2024-03-24T00:35:04.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 --2- 172.21.15.40:0/726348904 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c240728d0 0x7f9c240cf250 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:04.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 --2- 172.21.15.40:0/726348904 >> v2:172.21.15.53:3300/0 conn(0x7f9c2415bd00 0x7f9c240ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:04.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 -- 172.21.15.40:0/726348904 >> 172.21.15.40:0/726348904 conn(0x7f9c240b1660 msgr2=0x7f9c240b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:04.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 -- 172.21.15.40:0/726348904 shutdown_connections 2024-03-24T00:35:04.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 -- 172.21.15.40:0/726348904 wait complete. 2024-03-24T00:35:04.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 Processor -- start 2024-03-24T00:35:04.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 -- start start 2024-03-24T00:35:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c240728d0 0x7f9c2414f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2415bd00 0x7f9c2415a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c2415f8c0 0x7f9c24151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2406d320 con 0x7f9c2415f8c0 2024-03-24T00:35:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2415c0e0 con 0x7f9c2415bd00 2024-03-24T00:35:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c2aa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2406d4a0 con 0x7f9c240728d0 2024-03-24T00:35:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c28a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c2415f8c0 0x7f9c24151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c28a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c2415f8c0 0x7f9c24151020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46728/0 (socket says 172.21.15.40:46728) 2024-03-24T00:35:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c28a00640 1 -- 172.21.15.40:0/2360723046 learned_addr learned my addr 172.21.15.40:0/2360723046 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c1be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2415bd00 0x7f9c2415a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.407+0000 7f9c23e00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c240728d0 0x7f9c2414f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:04.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c28a00640 1 -- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c240728d0 msgr2=0x7f9c2414f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c28a00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c240728d0 0x7f9c2414f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c28a00640 1 -- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2415bd00 msgr2=0x7f9c2415a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c28a00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2415bd00 0x7f9c2415a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c28a00640 1 -- 172.21.15.40:0/2360723046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c240af830 con 0x7f9c2415f8c0 2024-03-24T00:35:04.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c28a00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c2415f8c0 0x7f9c24151020 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c00c760 tx=0x7f9c1c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c20c00640 1 -- 172.21.15.40:0/2360723046 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c003b80 con 0x7f9c2415f8c0 2024-03-24T00:35:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c20c00640 1 -- 172.21.15.40:0/2360723046 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c003d20 con 0x7f9c2415f8c0 2024-03-24T00:35:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c2aa00640 1 -- 172.21.15.40:0/2360723046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c24159b80 con 0x7f9c2415f8c0 2024-03-24T00:35:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c20c00640 1 -- 172.21.15.40:0/2360723046 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c00aa30 con 0x7f9c2415f8c0 2024-03-24T00:35:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c2aa00640 1 -- 172.21.15.40:0/2360723046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c24154d70 con 0x7f9c2415f8c0 2024-03-24T00:35:04.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c2aa00640 1 -- 172.21.15.40:0/2360723046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c24068170 con 0x7f9c2415f8c0 2024-03-24T00:35:04.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c20c00640 1 -- 172.21.15.40:0/2360723046 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f9c1c00abd0 con 0x7f9c2415f8c0 2024-03-24T00:35:04.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c20c00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c00067660 0x7f9c00069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:04.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c20c00640 1 -- 172.21.15.40:0/2360723046 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5529+0+0 (secure 0 0 0) 0x7f9c1c085550 con 0x7f9c2415f8c0 2024-03-24T00:35:04.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.411+0000 7f9c23e00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c00067660 0x7f9c00069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:04.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.415+0000 7f9c23e00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c00067660 0x7f9c00069b20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9c14007a60 tx=0x7f9c14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:04.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.415+0000 7f9c20c00640 1 -- 172.21.15.40:0/2360723046 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9c1c055c30 con 0x7f9c2415f8c0 2024-03-24T00:35:04.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:04.583+0000 7f9c2aa00640 1 -- 172.21.15.40:0/2360723046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9c2406e870 con 0x7f9c2415f8c0 2024-03-24T00:35:05.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.231+0000 7f9c20c00640 1 -- 172.21.15.40:0/2360723046 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v62) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9c1c059cd0 con 0x7f9c2415f8c0 2024-03-24T00:35:05.234 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-03-24T00:35:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.235+0000 7f9c2aa00640 1 -- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c00067660 msgr2=0x7f9c00069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.235+0000 7f9c2aa00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c00067660 0x7f9c00069b20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9c14007a60 tx=0x7f9c14008040 comp rx=0 tx=0).stop 2024-03-24T00:35:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.235+0000 7f9c2aa00640 1 -- 172.21.15.40:0/2360723046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c2415f8c0 msgr2=0x7f9c24151020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.235+0000 7f9c2aa00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c2415f8c0 0x7f9c24151020 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c00c760 tx=0x7f9c1c00cc20 comp rx=0 tx=0).stop 2024-03-24T00:35:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.235+0000 7f9c2aa00640 1 -- 172.21.15.40:0/2360723046 shutdown_connections 2024-03-24T00:35:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.235+0000 7f9c2aa00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c00067660 0x7f9c00069b20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.235+0000 7f9c2aa00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c2415f8c0 0x7f9c24151020 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.235+0000 7f9c2aa00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2415bd00 0x7f9c2415a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.235+0000 7f9c2aa00640 1 --2- 172.21.15.40:0/2360723046 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c240728d0 0x7f9c2414f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.235+0000 7f9c2aa00640 1 -- 172.21.15.40:0/2360723046 >> 172.21.15.40:0/2360723046 conn(0x7f9c240b1660 msgr2=0x7f9c240cf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.235+0000 7f9c2aa00640 1 -- 172.21.15.40:0/2360723046 shutdown_connections 2024-03-24T00:35:05.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.235+0000 7f9c2aa00640 1 -- 172.21.15.40:0/2360723046 wait complete. 2024-03-24T00:35:05.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random 2024-03-24T00:35:05.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.399+0000 7f4d3d600640 1 Processor -- start 2024-03-24T00:35:05.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.399+0000 7f4d3d600640 1 -- start start 2024-03-24T00:35:05.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d3d600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4d380cea30 0x7f4d3806b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d3d600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3806c270 0x7f4d3815d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d3d600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d380cee10 con 0x7f4d380b4040 2024-03-24T00:35:05.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d3d600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d380cfc30 con 0x7f4d380cea30 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d3d600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d380cfed0 con 0x7f4d3806c270 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d2f400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4d380b4040 0x7f4d3806bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56510/0 (socket says 172.21.15.40:56510) 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d2f400640 1 -- 172.21.15.40:0/654084917 learned_addr learned my addr 172.21.15.40:0/654084917 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37e00640 1 --2- 172.21.15.40:0/654084917 >> v2:172.21.15.53:3300/0 conn(0x7f4d380cea30 0x7f4d3806b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d3cc00640 1 --2- 172.21.15.40:0/654084917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3806c270 0x7f4d3815d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414514498 0 0) 0x7f4d380cee10 con 0x7f4d380b4040 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d1c003530 con 0x7f4d380b4040 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1755730674 0 0) 0x7f4d1c003530 con 0x7f4d380b4040 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3806c270 msgr2=0x7f4d3815d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 --2- 172.21.15.40:0/654084917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3806c270 0x7f4d3815d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 >> v2:172.21.15.53:3300/0 conn(0x7f4d380cea30 msgr2=0x7f4d3806b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 --2- 172.21.15.40:0/654084917 >> v2:172.21.15.53:3300/0 conn(0x7f4d380cea30 0x7f4d3806b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d380d01b0 con 0x7f4d380b4040 2024-03-24T00:35:05.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f4d200016a0 con 0x7f4d380b4040 2024-03-24T00:35:05.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 >> v1:172.21.15.40:6789/0 conn(0x7f4d380b4040 legacy=0x7f4d3806bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 --2- 172.21.15.40:0/654084917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3806c270 0x7f4d1c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 --2- 172.21.15.40:0/654084917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d1c003f40 0x7f4d1c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 --2- 172.21.15.40:0/654084917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d1c004d50 0x7f4d1c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d380cfc30 con 0x7f4d1c004d50 2024-03-24T00:35:05.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d380cfed0 con 0x7f4d1c003f40 2024-03-24T00:35:05.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c00b9d0 con 0x7f4d3806c270 2024-03-24T00:35:05.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d3cc00640 1 --2- 172.21.15.40:0/654084917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d1c004d50 0x7f4d1c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d37e00640 1 --2- 172.21.15.40:0/654084917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3806c270 0x7f4d1c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d2f400640 1 --2- 172.21.15.40:0/654084917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d1c003f40 0x7f4d1c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d3cc00640 1 -- 172.21.15.40:0/654084917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3806c270 msgr2=0x7f4d1c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:05.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d3cc00640 1 --2- 172.21.15.40:0/654084917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3806c270 0x7f4d1c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d3cc00640 1 -- 172.21.15.40:0/654084917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d1c003f40 msgr2=0x7f4d1c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:05.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d3cc00640 1 --2- 172.21.15.40:0/654084917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d1c003f40 0x7f4d1c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.403+0000 7f4d3cc00640 1 -- 172.21.15.40:0/654084917 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c003530 con 0x7f4d1c004d50 2024-03-24T00:35:05.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d3cc00640 1 --2- 172.21.15.40:0/654084917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d1c004d50 0x7f4d1c0051d0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f4d30009fc0 tx=0x7f4d3001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d26c1f882b1b6e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:05.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d300b2070 con 0x7f4d1c004d50 2024-03-24T00:35:05.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d300a5c90 con 0x7f4d1c004d50 2024-03-24T00:35:05.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d37400640 1 -- 172.21.15.40:0/654084917 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d30009c10 con 0x7f4d1c004d50 2024-03-24T00:35:05.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d3d600640 1 -- 172.21.15.40:0/654084917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d1c004d50 msgr2=0x7f4d1c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d3d600640 1 --2- 172.21.15.40:0/654084917 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d1c004d50 0x7f4d1c0051d0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f4d30009fc0 tx=0x7f4d3001cc20 comp rx=0 tx=0).stop 2024-03-24T00:35:05.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d3cc00640 1 -- 172.21.15.40:0/654084917 reap_dead start 2024-03-24T00:35:05.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d3d600640 1 -- 172.21.15.40:0/654084917 shutdown_connections 2024-03-24T00:35:05.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d3d600640 1 -- 172.21.15.40:0/654084917 >> 172.21.15.40:0/654084917 conn(0x7f4d380b34b0 msgr2=0x7f4d380b38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:05.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d3d600640 1 -- 172.21.15.40:0/654084917 shutdown_connections 2024-03-24T00:35:05.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d3d600640 1 -- 172.21.15.40:0/654084917 wait complete. 2024-03-24T00:35:05.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d3d600640 1 Processor -- start 2024-03-24T00:35:05.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.407+0000 7f4d3d600640 1 -- start start 2024-03-24T00:35:05.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d3d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d1c004d50 0x7f4d38154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d3d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d3809e010 0x7f4d380b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d3d600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3809f540 0x7f4d380af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d3d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d380d0a50 con 0x7f4d1c004d50 2024-03-24T00:35:05.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d3d600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d380d0770 con 0x7f4d3809e010 2024-03-24T00:35:05.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d3d600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3806b8a0 con 0x7f4d3809f540 2024-03-24T00:35:05.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d37e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d1c004d50 0x7f4d38154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d37e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d1c004d50 0x7f4d38154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46754/0 (socket says 172.21.15.40:46754) 2024-03-24T00:35:05.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d37e00640 1 -- 172.21.15.40:0/1877377583 learned_addr learned my addr 172.21.15.40:0/1877377583 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:05.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d2f400640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d3809e010 0x7f4d380b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d3cc00640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3809f540 0x7f4d380af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d37e00640 1 -- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3809f540 msgr2=0x7f4d380af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d37e00640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3809f540 0x7f4d380af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d37e00640 1 -- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d3809e010 msgr2=0x7f4d380b21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d37e00640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d3809e010 0x7f4d380b21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d37e00640 1 -- 172.21.15.40:0/1877377583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3809e4a0 con 0x7f4d1c004d50 2024-03-24T00:35:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d37e00640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d1c004d50 0x7f4d38154fb0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4d28011ad0 tx=0x7f4d28011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d34c00640 1 -- 172.21.15.40:0/1877377583 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2800faf0 con 0x7f4d1c004d50 2024-03-24T00:35:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d34c00640 1 -- 172.21.15.40:0/1877377583 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d2800fc90 con 0x7f4d1c004d50 2024-03-24T00:35:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d34c00640 1 -- 172.21.15.40:0/1877377583 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d28014440 con 0x7f4d1c004d50 2024-03-24T00:35:05.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d3d600640 1 -- 172.21.15.40:0/1877377583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d3809e780 con 0x7f4d1c004d50 2024-03-24T00:35:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d3d600640 1 -- 172.21.15.40:0/1877377583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d3809ed40 con 0x7f4d1c004d50 2024-03-24T00:35:05.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d3d600640 1 -- 172.21.15.40:0/1877377583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d3804f410 con 0x7f4d1c004d50 2024-03-24T00:35:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.411+0000 7f4d34c00640 1 -- 172.21.15.40:0/1877377583 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4d28012040 con 0x7f4d1c004d50 2024-03-24T00:35:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.415+0000 7f4d34c00640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4d14067660 0x7f4d14069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.415+0000 7f4d34c00640 1 -- 172.21.15.40:0/1877377583 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f4d28085ca0 con 0x7f4d1c004d50 2024-03-24T00:35:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.415+0000 7f4d2f400640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4d14067660 0x7f4d14069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.415+0000 7f4d2f400640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4d14067660 0x7f4d14069b20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4d380c53e0 tx=0x7f4d20007980 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:05.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.419+0000 7f4d34c00640 1 -- 172.21.15.40:0/1877377583 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4d28019030 con 0x7f4d1c004d50 2024-03-24T00:35:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.579+0000 7f4d3d600640 1 -- 172.21.15.40:0/1877377583 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"} v 0) v1 -- 0x7f4d38073610 con 0x7f4d1c004d50 2024-03-24T00:35:05.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.583+0000 7f4d34c00640 1 -- 172.21.15.40:0/1877377583 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"}]=0 v31) v1 ==== 110+0+0 (secure 0 0 0) 0x7f4d28056b60 con 0x7f4d1c004d50 2024-03-24T00:35:05.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.587+0000 7f4d3d600640 1 -- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4d14067660 msgr2=0x7f4d14069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.587+0000 7f4d3d600640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4d14067660 0x7f4d14069b20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4d380c53e0 tx=0x7f4d20007980 comp rx=0 tx=0).stop 2024-03-24T00:35:05.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.587+0000 7f4d3d600640 1 -- 172.21.15.40:0/1877377583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d1c004d50 msgr2=0x7f4d38154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.587+0000 7f4d3d600640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d1c004d50 0x7f4d38154fb0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f4d28011ad0 tx=0x7f4d28011ed0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.587+0000 7f4d3d600640 1 -- 172.21.15.40:0/1877377583 shutdown_connections 2024-03-24T00:35:05.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.587+0000 7f4d3d600640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4d14067660 0x7f4d14069b20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.587+0000 7f4d3d600640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d3809f540 0x7f4d380af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.587+0000 7f4d3d600640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d3809e010 0x7f4d380b21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.587+0000 7f4d3d600640 1 --2- 172.21.15.40:0/1877377583 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4d1c004d50 0x7f4d38154fb0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.587+0000 7f4d3d600640 1 -- 172.21.15.40:0/1877377583 >> 172.21.15.40:0/1877377583 conn(0x7f4d380b34b0 msgr2=0x7f4d38160c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:05.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.587+0000 7f4d3d600640 1 -- 172.21.15.40:0/1877377583 shutdown_connections 2024-03-24T00:35:05.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.587+0000 7f4d3d600640 1 -- 172.21.15.40:0/1877377583 wait complete. 2024-03-24T00:35:05.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_distributed 2024-03-24T00:35:05.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0cac00640 1 Processor -- start 2024-03-24T00:35:05.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0cac00640 1 -- start start 2024-03-24T00:35:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0cac00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb0c4145050 0x7fb0c40b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0cac00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c40b3ee0 con 0x7fb0c41454a0 2024-03-24T00:35:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0cac00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c40b4160 con 0x7fb0c40cd650 2024-03-24T00:35:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0cac00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c40b4440 con 0x7fb0c4145050 2024-03-24T00:35:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0c8e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb0c41454a0 0x7fb0c40ccf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:56526/0 (socket says 172.21.15.40:56526) 2024-03-24T00:35:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0c8e00640 1 -- 172.21.15.40:0/4137607220 learned_addr learned my addr 172.21.15.40:0/4137607220 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0c9800640 1 --2- 172.21.15.40:0/4137607220 >> v2:172.21.15.53:3300/0 conn(0x7fb0c4145050 0x7fb0c40b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0c3e00640 1 -- 172.21.15.40:0/4137607220 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912711153 0 0) 0x7fb0b0001120 con 0x7fb0c41454a0 2024-03-24T00:35:05.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0c3e00640 1 -- 172.21.15.40:0/4137607220 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0ac003530 con 0x7fb0c41454a0 2024-03-24T00:35:05.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0c3e00640 1 -- 172.21.15.40:0/4137607220 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367344081 0 0) 0x7fb0c40b4160 con 0x7fb0c40cd650 2024-03-24T00:35:05.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0c3e00640 1 -- 172.21.15.40:0/4137607220 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b0001120 con 0x7fb0c40cd650 2024-03-24T00:35:05.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0c9800640 1 -- 172.21.15.40:0/4137607220 >> v1:172.21.15.53:6789/0 conn(0x7fb0c40cd650 legacy=0x7fb0c406cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0c9800640 1 -- 172.21.15.40:0/4137607220 >> v1:172.21.15.40:6789/0 conn(0x7fb0c41454a0 legacy=0x7fb0c40ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.759+0000 7fb0c9800640 1 -- 172.21.15.40:0/4137607220 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c40b4720 con 0x7fb0c4145050 2024-03-24T00:35:05.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.763+0000 7fb0c9800640 1 --2- 172.21.15.40:0/4137607220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0c4145050 0x7fb0c40b39a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80097c0 tx=0x7fb0b801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bda884480e72195 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:05.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.763+0000 7fb0c3e00640 1 -- 172.21.15.40:0/4137607220 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80b0070 con 0x7fb0c4145050 2024-03-24T00:35:05.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.763+0000 7fb0c3e00640 1 -- 172.21.15.40:0/4137607220 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b80ab600 con 0x7fb0c4145050 2024-03-24T00:35:05.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.763+0000 7fb0c3e00640 1 -- 172.21.15.40:0/4137607220 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80ab9d0 con 0x7fb0c4145050 2024-03-24T00:35:05.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.763+0000 7fb0cac00640 1 -- 172.21.15.40:0/4137607220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0c4145050 msgr2=0x7fb0c40b39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.763+0000 7fb0cac00640 1 --2- 172.21.15.40:0/4137607220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0c4145050 0x7fb0c40b39a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80097c0 tx=0x7fb0b801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.763+0000 7fb0cac00640 1 -- 172.21.15.40:0/4137607220 shutdown_connections 2024-03-24T00:35:05.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.763+0000 7fb0cac00640 1 --2- 172.21.15.40:0/4137607220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0c4145050 0x7fb0c40b39a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.763+0000 7fb0cac00640 1 -- 172.21.15.40:0/4137607220 >> 172.21.15.40:0/4137607220 conn(0x7fb0c40743d0 msgr2=0x7fb0c4066e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:05.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.763+0000 7fb0cac00640 1 -- 172.21.15.40:0/4137607220 shutdown_connections 2024-03-24T00:35:05.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.763+0000 7fb0cac00640 1 -- 172.21.15.40:0/4137607220 wait complete. 2024-03-24T00:35:05.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0cac00640 1 Processor -- start 2024-03-24T00:35:05.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0cac00640 1 -- start start 2024-03-24T00:35:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0cac00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0c40cd650 0x7fb0c413c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0cac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c4145050 0x7fb0c4071b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0cac00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0c41454a0 0x7fb0c4160de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0cac00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c406dd90 con 0x7fb0c4145050 2024-03-24T00:35:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0cac00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c406dc10 con 0x7fb0c41454a0 2024-03-24T00:35:05.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0cac00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c4169970 con 0x7fb0c40cd650 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c4145050 0x7fb0c4071b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c4145050 0x7fb0c4071b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46770/0 (socket says 172.21.15.40:46770) 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c8e00640 1 -- 172.21.15.40:0/537978791 learned_addr learned my addr 172.21.15.40:0/537978791 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0ca200640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0c41454a0 0x7fb0c4160de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c9800640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0c40cd650 0x7fb0c413c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c8e00640 1 -- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0c40cd650 msgr2=0x7fb0c413c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c8e00640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0c40cd650 0x7fb0c413c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c8e00640 1 -- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0c41454a0 msgr2=0x7fb0c4160de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c8e00640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0c41454a0 0x7fb0c4160de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c8e00640 1 -- 172.21.15.40:0/537978791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c413c830 con 0x7fb0c4145050 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c8e00640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c4145050 0x7fb0c4071b40 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000c730 tx=0x7fb0b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c1600640 1 -- 172.21.15.40:0/537978791 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0017070 con 0x7fb0c4145050 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c1600640 1 -- 172.21.15.40:0/537978791 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b0005ae0 con 0x7fb0c4145050 2024-03-24T00:35:05.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0c1600640 1 -- 172.21.15.40:0/537978791 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0007070 con 0x7fb0c4145050 2024-03-24T00:35:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0cac00640 1 -- 172.21.15.40:0/537978791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c41424a0 con 0x7fb0c4145050 2024-03-24T00:35:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.767+0000 7fb0cac00640 1 -- 172.21.15.40:0/537978791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c4071620 con 0x7fb0c4145050 2024-03-24T00:35:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.771+0000 7fb0cac00640 1 -- 172.21.15.40:0/537978791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c4071820 con 0x7fb0c4145050 2024-03-24T00:35:05.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.771+0000 7fb0c1600640 1 -- 172.21.15.40:0/537978791 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb0b001a070 con 0x7fb0c4145050 2024-03-24T00:35:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.771+0000 7fb0c1600640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0a4067660 0x7fb0a4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.771+0000 7fb0c1600640 1 -- 172.21.15.40:0/537978791 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fb0b0084e50 con 0x7fb0c4145050 2024-03-24T00:35:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.771+0000 7fb0c9800640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0a4067660 0x7fb0a4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:05.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.771+0000 7fb0c9800640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0a4067660 0x7fb0a4069b20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80ad000 tx=0x7fb0b80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:05.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.775+0000 7fb0c1600640 1 -- 172.21.15.40:0/537978791 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb0c4071820 con 0x7fb0c4145050 2024-03-24T00:35:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.939+0000 7fb0cac00640 1 -- 172.21.15.40:0/537978791 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"} v 0) v1 -- 0x7fb0c404f410 con 0x7fb0c4145050 2024-03-24T00:35:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0c1600640 1 -- 172.21.15.40:0/537978791 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"}]=0 v32) v1 ==== 115+0+0 (secure 0 0 0) 0x7fb0c40b3ee0 con 0x7fb0c4145050 2024-03-24T00:35:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0cac00640 1 -- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0a4067660 msgr2=0x7fb0a4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0cac00640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0a4067660 0x7fb0a4069b20 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80ad000 tx=0x7fb0b80a6000 comp rx=0 tx=0).stop 2024-03-24T00:35:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0cac00640 1 -- 172.21.15.40:0/537978791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c4145050 msgr2=0x7fb0c4071b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0cac00640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c4145050 0x7fb0c4071b40 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000c730 tx=0x7fb0b000cbf0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0cac00640 1 -- 172.21.15.40:0/537978791 shutdown_connections 2024-03-24T00:35:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0cac00640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0a4067660 0x7fb0a4069b20 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0cac00640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0c41454a0 0x7fb0c4160de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0cac00640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb0c4145050 0x7fb0c4071b40 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0cac00640 1 --2- 172.21.15.40:0/537978791 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb0c40cd650 0x7fb0c413c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0cac00640 1 -- 172.21.15.40:0/537978791 >> 172.21.15.40:0/537978791 conn(0x7fb0c40743d0 msgr2=0x7fb0c40b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0cac00640 1 -- 172.21.15.40:0/537978791 shutdown_connections 2024-03-24T00:35:05.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:05.943+0000 7fb0cac00640 1 -- 172.21.15.40:0/537978791 wait complete. 2024-03-24T00:35:05.966 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random_max 2024-03-24T00:35:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7592600640 1 Processor -- start 2024-03-24T00:35:06.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7592600640 1 -- start start 2024-03-24T00:35:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7592600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f758c15bd00 0x7f758c0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7592600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f758c0728d0 0x7f758c0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7592600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f758c15f8c0 0x7f758c15fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7592600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f758c0cf790 con 0x7f758c0728d0 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7592600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f758c160250 con 0x7f758c15bd00 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7592600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758c072d60 con 0x7f758c15f8c0 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7582a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f758c0728d0 0x7f758c0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7582a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f758c0728d0 0x7f758c0cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46780/0 (socket says 172.21.15.40:46780) 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7582a00640 1 -- 172.21.15.40:0/4040848718 learned_addr learned my addr 172.21.15.40:0/4040848718 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f758be00640 1 --2- 172.21.15.40:0/4040848718 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f758c15f8c0 0x7f758c15fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f758b400640 1 --2- 172.21.15.40:0/4040848718 >> v2:172.21.15.53:3300/0 conn(0x7f758c15bd00 0x7f758c0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7582a00640 1 -- 172.21.15.40:0/4040848718 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f758c15f8c0 msgr2=0x7f758c15fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7582a00640 1 --2- 172.21.15.40:0/4040848718 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f758c15f8c0 0x7f758c15fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7582a00640 1 -- 172.21.15.40:0/4040848718 >> v2:172.21.15.53:3300/0 conn(0x7f758c15bd00 msgr2=0x7f758c0ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7582a00640 1 --2- 172.21.15.40:0/4040848718 >> v2:172.21.15.53:3300/0 conn(0x7f758c15bd00 0x7f758c0ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.107+0000 7f7582a00640 1 -- 172.21.15.40:0/4040848718 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c071de0 con 0x7f758c0728d0 2024-03-24T00:35:06.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f7582a00640 1 --2- 172.21.15.40:0/4040848718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c0728d0 0x7f758c0cf250 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f758c0733c0 tx=0x7f757401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2551f05db7e71f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:06.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f758aa00640 1 -- 172.21.15.40:0/4040848718 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75740aa610 con 0x7f758c0728d0 2024-03-24T00:35:06.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f758aa00640 1 -- 172.21.15.40:0/4040848718 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75740af070 con 0x7f758c0728d0 2024-03-24T00:35:06.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f758aa00640 1 -- 172.21.15.40:0/4040848718 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75740aa9d0 con 0x7f758c0728d0 2024-03-24T00:35:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f7592600640 1 -- 172.21.15.40:0/4040848718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c0728d0 msgr2=0x7f758c0cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f7592600640 1 --2- 172.21.15.40:0/4040848718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c0728d0 0x7f758c0cf250 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f758c0733c0 tx=0x7f757401c4f0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f7592600640 1 -- 172.21.15.40:0/4040848718 shutdown_connections 2024-03-24T00:35:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f7592600640 1 --2- 172.21.15.40:0/4040848718 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f758c15f8c0 0x7f758c15fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f7592600640 1 --2- 172.21.15.40:0/4040848718 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c0728d0 0x7f758c0cf250 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f7592600640 1 --2- 172.21.15.40:0/4040848718 >> v2:172.21.15.53:3300/0 conn(0x7f758c15bd00 0x7f758c0ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f7592600640 1 -- 172.21.15.40:0/4040848718 >> 172.21.15.40:0/4040848718 conn(0x7f758c0b1660 msgr2=0x7f758c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f7592600640 1 -- 172.21.15.40:0/4040848718 shutdown_connections 2024-03-24T00:35:06.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.111+0000 7f7592600640 1 -- 172.21.15.40:0/4040848718 wait complete. 2024-03-24T00:35:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7592600640 1 Processor -- start 2024-03-24T00:35:06.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7592600640 1 -- start start 2024-03-24T00:35:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7592600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f758c0728d0 0x7f758c1478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7592600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f758c15bd00 0x7f758c14fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7592600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c15f8c0 0x7f758c15ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7592600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c06d320 con 0x7f758c15f8c0 2024-03-24T00:35:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7592600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c15c0e0 con 0x7f758c0728d0 2024-03-24T00:35:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7592600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758c06d4a0 con 0x7f758c15bd00 2024-03-24T00:35:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f758be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c15f8c0 0x7f758c15ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f758be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c15f8c0 0x7f758c15ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46796/0 (socket says 172.21.15.40:46796) 2024-03-24T00:35:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f758be00640 1 -- 172.21.15.40:0/3137332716 learned_addr learned my addr 172.21.15.40:0/3137332716 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7582a00640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f758c15bd00 0x7f758c14fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f758b400640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f758c0728d0 0x7f758c1478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f758be00640 1 -- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f758c15bd00 msgr2=0x7f758c14fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f758be00640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f758c15bd00 0x7f758c14fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f758be00640 1 -- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f758c0728d0 msgr2=0x7f758c1478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f758be00640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f758c0728d0 0x7f758c1478a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f758be00640 1 -- 172.21.15.40:0/3137332716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c157160 con 0x7f758c15f8c0 2024-03-24T00:35:06.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f758be00640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c15f8c0 0x7f758c15ad80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f758400e7f0 tx=0x7f758400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:06.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7583e00640 1 -- 172.21.15.40:0/3137332716 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7584017070 con 0x7f758c15f8c0 2024-03-24T00:35:06.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7583e00640 1 -- 172.21.15.40:0/3137332716 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758400c940 con 0x7f758c15f8c0 2024-03-24T00:35:06.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7592600640 1 -- 172.21.15.40:0/3137332716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758c157440 con 0x7f758c15f8c0 2024-03-24T00:35:06.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7583e00640 1 -- 172.21.15.40:0/3137332716 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758400eea0 con 0x7f758c15f8c0 2024-03-24T00:35:06.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7592600640 1 -- 172.21.15.40:0/3137332716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758c157a00 con 0x7f758c15f8c0 2024-03-24T00:35:06.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.115+0000 7f7592600640 1 -- 172.21.15.40:0/3137332716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758c068170 con 0x7f758c15f8c0 2024-03-24T00:35:06.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.119+0000 7f7583e00640 1 -- 172.21.15.40:0/3137332716 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f758401a070 con 0x7f758c15f8c0 2024-03-24T00:35:06.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.119+0000 7f7583e00640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7568067660 0x7f7568069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.119+0000 7f7583e00640 1 -- 172.21.15.40:0/3137332716 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f7584085770 con 0x7f758c15f8c0 2024-03-24T00:35:06.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.119+0000 7f758b400640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7568067660 0x7f7568069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.119+0000 7f758b400640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7568067660 0x7f7568069b20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f757c0113e0 tx=0x7f757c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:06.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.123+0000 7f7583e00640 1 -- 172.21.15.40:0/3137332716 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7584055ff0 con 0x7f758c15f8c0 2024-03-24T00:35:06.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.287+0000 7f7592600640 1 -- 172.21.15.40:0/3137332716 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"} v 0) v1 -- 0x7f758c152130 con 0x7f758c15f8c0 2024-03-24T00:35:06.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.291+0000 7f7583e00640 1 -- 172.21.15.40:0/3137332716 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"}]=0 v33) v1 ==== 114+0+0 (secure 0 0 0) 0x7f758405a090 con 0x7f758c15f8c0 2024-03-24T00:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.295+0000 7f7592600640 1 -- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7568067660 msgr2=0x7f7568069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.295+0000 7f7592600640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7568067660 0x7f7568069b20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f757c0113e0 tx=0x7f757c00b040 comp rx=0 tx=0).stop 2024-03-24T00:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.295+0000 7f7592600640 1 -- 172.21.15.40:0/3137332716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c15f8c0 msgr2=0x7f758c15ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.295+0000 7f7592600640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c15f8c0 0x7f758c15ad80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f758400e7f0 tx=0x7f758400ecb0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.295+0000 7f7592600640 1 -- 172.21.15.40:0/3137332716 shutdown_connections 2024-03-24T00:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.295+0000 7f7592600640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7568067660 0x7f7568069b20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.295+0000 7f7592600640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f758c15f8c0 0x7f758c15ad80 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.295+0000 7f7592600640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f758c15bd00 0x7f758c14fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.295+0000 7f7592600640 1 --2- 172.21.15.40:0/3137332716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f758c0728d0 0x7f758c1478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.295+0000 7f7592600640 1 -- 172.21.15.40:0/3137332716 >> 172.21.15.40:0/3137332716 conn(0x7f758c0b1660 msgr2=0x7f758c152930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:06.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.295+0000 7f7592600640 1 -- 172.21.15.40:0/3137332716 shutdown_connections 2024-03-24T00:35:06.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.295+0000 7f7592600640 1 -- 172.21.15.40:0/3137332716 wait complete. 2024-03-24T00:35:06.316 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:35:06.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.455+0000 7f557ca00640 1 Processor -- start 2024-03-24T00:35:06.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.455+0000 7f557ca00640 1 -- start start 2024-03-24T00:35:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.455+0000 7f557ca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f557815bd00 0x7f55780ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.455+0000 7f557ca00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55780728d0 0x7f55780cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.455+0000 7f557ca00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5578072d60 con 0x7f557815f8c0 2024-03-24T00:35:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.455+0000 7f557ca00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55780cf790 con 0x7f55780728d0 2024-03-24T00:35:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.455+0000 7f557ca00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5578071de0 con 0x7f557815bd00 2024-03-24T00:35:06.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f556fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55780728d0 0x7f55780cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f556fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55780728d0 0x7f55780cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46800/0 (socket says 172.21.15.40:46800) 2024-03-24T00:35:06.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f556fe00640 1 -- 172.21.15.40:0/1534174606 learned_addr learned my addr 172.21.15.40:0/1534174606 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:06.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f5575e00640 1 --2- 172.21.15.40:0/1534174606 >> v2:172.21.15.53:3300/0 conn(0x7f557815bd00 0x7f55780ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f556fe00640 1 -- 172.21.15.40:0/1534174606 >> v2:172.21.15.53:3300/0 conn(0x7f557815bd00 msgr2=0x7f55780ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:06.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f556fe00640 1 --2- 172.21.15.40:0/1534174606 >> v2:172.21.15.53:3300/0 conn(0x7f557815bd00 0x7f55780ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f556fe00640 1 -- 172.21.15.40:0/1534174606 >> v1:172.21.15.53:6789/0 conn(0x7f557815f8c0 legacy=0x7f557815fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:06.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f556fe00640 1 -- 172.21.15.40:0/1534174606 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55780cea50 con 0x7f55780728d0 2024-03-24T00:35:06.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f5575e00640 1 --2- 172.21.15.40:0/1534174606 >> v2:172.21.15.53:3300/0 conn(0x7f557815bd00 0x7f55780ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:35:06.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f556fe00640 1 --2- 172.21.15.40:0/1534174606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780728d0 0x7f55780cf250 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f55780733c0 tx=0x7f556001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3853e54fb698f634 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:06.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f5575400640 1 -- 172.21.15.40:0/1534174606 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55600aa610 con 0x7f55780728d0 2024-03-24T00:35:06.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f5575400640 1 -- 172.21.15.40:0/1534174606 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55600af070 con 0x7f55780728d0 2024-03-24T00:35:06.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.459+0000 7f5575400640 1 -- 172.21.15.40:0/1534174606 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55600aa9d0 con 0x7f55780728d0 2024-03-24T00:35:06.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 -- 172.21.15.40:0/1534174606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780728d0 msgr2=0x7f55780cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 --2- 172.21.15.40:0/1534174606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780728d0 0x7f55780cf250 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f55780733c0 tx=0x7f556001c4b0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 -- 172.21.15.40:0/1534174606 shutdown_connections 2024-03-24T00:35:06.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 --2- 172.21.15.40:0/1534174606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780728d0 0x7f55780cf250 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 --2- 172.21.15.40:0/1534174606 >> v2:172.21.15.53:3300/0 conn(0x7f557815bd00 0x7f55780ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 -- 172.21.15.40:0/1534174606 >> 172.21.15.40:0/1534174606 conn(0x7f55780b1660 msgr2=0x7f55780b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:06.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 -- 172.21.15.40:0/1534174606 shutdown_connections 2024-03-24T00:35:06.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 -- 172.21.15.40:0/1534174606 wait complete. 2024-03-24T00:35:06.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 Processor -- start 2024-03-24T00:35:06.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 -- start start 2024-03-24T00:35:06.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780728d0 0x7f55780b3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f557815bd00 0x7f5578146370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f557815f8c0 0x7f5578154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557806d320 con 0x7f55780728d0 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557815c0e0 con 0x7f557815bd00 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f557ca00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557806d4a0 con 0x7f557815f8c0 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f5575e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780728d0 0x7f55780b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f5575e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780728d0 0x7f55780b3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46802/0 (socket says 172.21.15.40:46802) 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f5575e00640 1 -- 172.21.15.40:0/1962832136 learned_addr learned my addr 172.21.15.40:0/1962832136 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f556fe00640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f557815bd00 0x7f5578146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.463+0000 7f5576800640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f557815f8c0 0x7f5578154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f5575e00640 1 -- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f557815f8c0 msgr2=0x7f5578154fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f5575e00640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f557815f8c0 0x7f5578154fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f5575e00640 1 -- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f557815bd00 msgr2=0x7f5578146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f5575e00640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f557815bd00 0x7f5578146370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f5575e00640 1 -- 172.21.15.40:0/1962832136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55780b37e0 con 0x7f55780728d0 2024-03-24T00:35:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f5575e00640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780728d0 0x7f55780b3070 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f556800c7c0 tx=0x7f556800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f556e000640 1 -- 172.21.15.40:0/1962832136 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568007a30 con 0x7f55780728d0 2024-03-24T00:35:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f556e000640 1 -- 172.21.15.40:0/1962832136 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5568007bd0 con 0x7f55780728d0 2024-03-24T00:35:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f556e000640 1 -- 172.21.15.40:0/1962832136 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55680113f0 con 0x7f55780728d0 2024-03-24T00:35:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f557ca00640 1 -- 172.21.15.40:0/1962832136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55780b3ac0 con 0x7f55780728d0 2024-03-24T00:35:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f557ca00640 1 -- 172.21.15.40:0/1962832136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55780b4080 con 0x7f55780728d0 2024-03-24T00:35:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f557ca00640 1 -- 172.21.15.40:0/1962832136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5578068170 con 0x7f55780728d0 2024-03-24T00:35:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f556e000640 1 -- 172.21.15.40:0/1962832136 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5568011590 con 0x7f55780728d0 2024-03-24T00:35:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f556e000640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5554067660 0x7f5554069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f556e000640 1 -- 172.21.15.40:0/1962832136 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f5568085800 con 0x7f55780728d0 2024-03-24T00:35:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.467+0000 7f556fe00640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5554067660 0x7f5554069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.471+0000 7f556fe00640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5554067660 0x7f5554069b20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f55600ae000 tx=0x7f55600a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:06.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.471+0000 7f556e000640 1 -- 172.21.15.40:0/1962832136 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5568015020 con 0x7f55780728d0 2024-03-24T00:35:06.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.635+0000 7f557ca00640 1 -- 172.21.15.40:0/1962832136 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f557806e870 con 0x7f55780728d0 2024-03-24T00:35:06.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f556e000640 1 -- 172.21.15.40:0/1962832136 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v34) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5568056070 con 0x7f55780728d0 2024-03-24T00:35:06.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f557ca00640 1 -- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5554067660 msgr2=0x7f5554069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f557ca00640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5554067660 0x7f5554069b20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f55600ae000 tx=0x7f55600a7000 comp rx=0 tx=0).stop 2024-03-24T00:35:06.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f557ca00640 1 -- 172.21.15.40:0/1962832136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780728d0 msgr2=0x7f55780b3070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f557ca00640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780728d0 0x7f55780b3070 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f556800c7c0 tx=0x7f556800cc80 comp rx=0 tx=0).stop 2024-03-24T00:35:06.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f557ca00640 1 -- 172.21.15.40:0/1962832136 shutdown_connections 2024-03-24T00:35:06.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f557ca00640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5554067660 0x7f5554069b20 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f557ca00640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f557815f8c0 0x7f5578154fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f557ca00640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f557815bd00 0x7f5578146370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f557ca00640 1 --2- 172.21.15.40:0/1962832136 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55780728d0 0x7f55780b3070 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f557ca00640 1 -- 172.21.15.40:0/1962832136 >> 172.21.15.40:0/1962832136 conn(0x7f55780b1660 msgr2=0x7f557814fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:06.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f557ca00640 1 -- 172.21.15.40:0/1962832136 shutdown_connections 2024-03-24T00:35:06.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.643+0000 7f557ca00640 1 -- 172.21.15.40:0/1962832136 wait complete. 2024-03-24T00:35:06.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_off' 2024-03-24T00:35:06.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.803+0000 7fc0ef200640 1 Processor -- start 2024-03-24T00:35:06.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.803+0000 7fc0ef200640 1 -- start start 2024-03-24T00:35:06.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.803+0000 7fc0ef200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0e80afc50 0x7fc0e80cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.803+0000 7fc0ef200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0e806d5c0 0x7fc0e806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.803+0000 7fc0ef200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e80cde10 con 0x7fc0e80cce80 2024-03-24T00:35:06.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.803+0000 7fc0ef200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e80ce0f0 con 0x7fc0e806d5c0 2024-03-24T00:35:06.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.803+0000 7fc0ef200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80ce3d0 con 0x7fc0e80afc50 2024-03-24T00:35:06.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.807+0000 7fc0ed200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0e806d5c0 0x7fc0e806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.807+0000 7fc0ed200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0e806d5c0 0x7fc0e806d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46814/0 (socket says 172.21.15.40:46814) 2024-03-24T00:35:06.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.807+0000 7fc0ed200640 1 -- 172.21.15.40:0/1919642868 learned_addr learned my addr 172.21.15.40:0/1919642868 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:06.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.807+0000 7fc0dfe00640 1 --2- 172.21.15.40:0/1919642868 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0e80afc50 0x7fc0e80cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.807+0000 7fc0ed200640 1 -- 172.21.15.40:0/1919642868 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0e80afc50 msgr2=0x7fc0e80cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:06.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.807+0000 7fc0ed200640 1 --2- 172.21.15.40:0/1919642868 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0e80afc50 0x7fc0e80cfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.807+0000 7fc0ed200640 1 -- 172.21.15.40:0/1919642868 >> v1:172.21.15.53:6789/0 conn(0x7fc0e80cce80 legacy=0x7fc0e80cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.807+0000 7fc0ed200640 1 -- 172.21.15.40:0/1919642868 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e815c550 con 0x7fc0e806d5c0 2024-03-24T00:35:06.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.807+0000 7fc0ed200640 1 --2- 172.21.15.40:0/1919642868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0e806d5c0 0x7fc0e806d9f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009e50 tx=0x7fc0e001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ac26bc4996af47 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:06.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.807+0000 7fc0e7400640 1 -- 172.21.15.40:0/1919642868 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e00a5770 con 0x7fc0e806d5c0 2024-03-24T00:35:06.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.807+0000 7fc0e7400640 1 -- 172.21.15.40:0/1919642868 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e00a5910 con 0x7fc0e806d5c0 2024-03-24T00:35:06.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.807+0000 7fc0e7400640 1 -- 172.21.15.40:0/1919642868 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e00a5ce0 con 0x7fc0e806d5c0 2024-03-24T00:35:06.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 -- 172.21.15.40:0/1919642868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0e806d5c0 msgr2=0x7fc0e806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 --2- 172.21.15.40:0/1919642868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0e806d5c0 0x7fc0e806d9f0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009e50 tx=0x7fc0e001c9a0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 -- 172.21.15.40:0/1919642868 shutdown_connections 2024-03-24T00:35:06.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 --2- 172.21.15.40:0/1919642868 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0e80afc50 0x7fc0e80cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 --2- 172.21.15.40:0/1919642868 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0e806d5c0 0x7fc0e806d9f0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 -- 172.21.15.40:0/1919642868 >> 172.21.15.40:0/1919642868 conn(0x7fc0e80b21a0 msgr2=0x7fc0e80b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:06.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 -- 172.21.15.40:0/1919642868 shutdown_connections 2024-03-24T00:35:06.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 -- 172.21.15.40:0/1919642868 wait complete. 2024-03-24T00:35:06.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 Processor -- start 2024-03-24T00:35:06.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 -- start start 2024-03-24T00:35:06.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0e806d5c0 0x7fc0e81518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e80afc50 0x7fc0e815b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0e80cce80 0x7fc0e815ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e8168f50 con 0x7fc0e80cce80 2024-03-24T00:35:06.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e8168dd0 con 0x7fc0e80afc50 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ef200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e81690d0 con 0x7fc0e806d5c0 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ed200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0e80cce80 0x7fc0e815ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ed200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0e80cce80 0x7fc0e815ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46820/0 (socket says 172.21.15.40:46820) 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ed200640 1 -- 172.21.15.40:0/1874673255 learned_addr learned my addr 172.21.15.40:0/1874673255 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0dfe00640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e80afc50 0x7fc0e815b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0e7e00640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0e806d5c0 0x7fc0e81518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ed200640 1 -- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0e806d5c0 msgr2=0x7fc0e81518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ed200640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0e806d5c0 0x7fc0e81518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ed200640 1 -- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e80afc50 msgr2=0x7fc0e815b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ed200640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e80afc50 0x7fc0e815b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.811+0000 7fc0ed200640 1 -- 172.21.15.40:0/1874673255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e815c060 con 0x7fc0e80cce80 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.815+0000 7fc0ed200640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0e80cce80 0x7fc0e815ba30 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc0e001a5d0 tx=0x7fc0e0019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.815+0000 7fc0e4c00640 1 -- 172.21.15.40:0/1874673255 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e00bc020 con 0x7fc0e80cce80 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.815+0000 7fc0e4c00640 1 -- 172.21.15.40:0/1874673255 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e00b8670 con 0x7fc0e80cce80 2024-03-24T00:35:06.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.815+0000 7fc0e4c00640 1 -- 172.21.15.40:0/1874673255 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0009510 con 0x7fc0e80cce80 2024-03-24T00:35:06.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.815+0000 7fc0ef200640 1 -- 172.21.15.40:0/1874673255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e81570b0 con 0x7fc0e80cce80 2024-03-24T00:35:06.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.815+0000 7fc0ef200640 1 -- 172.21.15.40:0/1874673255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e8157670 con 0x7fc0e80cce80 2024-03-24T00:35:06.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.815+0000 7fc0ef200640 1 -- 172.21.15.40:0/1874673255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e8068170 con 0x7fc0e80cce80 2024-03-24T00:35:06.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.815+0000 7fc0e4c00640 1 -- 172.21.15.40:0/1874673255 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc0e00bf020 con 0x7fc0e80cce80 2024-03-24T00:35:06.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.815+0000 7fc0e4c00640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc0c4067660 0x7fc0c4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:06.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.815+0000 7fc0e4c00640 1 -- 172.21.15.40:0/1874673255 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fc0e01205b0 con 0x7fc0e80cce80 2024-03-24T00:35:06.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.815+0000 7fc0e7e00640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc0c4067660 0x7fc0c4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:06.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.819+0000 7fc0e7e00640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc0c4067660 0x7fc0c4069b20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc0e815ebd0 tx=0x7fc0d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:06.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:06.819+0000 7fc0e4c00640 1 -- 172.21.15.40:0/1874673255 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc0e00c6030 con 0x7fc0e80cce80 2024-03-24T00:35:07.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.039+0000 7fc0ef200640 1 -- 172.21.15.40:0/1874673255 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_off"]} v 0) v1 -- 0x7fc0e8074a00 con 0x7fc0e80cce80 2024-03-24T00:35:07.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0e4c00640 1 -- 172.21.15.40:0/1874673255 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_off"]}]=0 v218) v1 ==== 150+0+0 (secure 0 0 0) 0x7fc0e0016620 con 0x7fc0e80cce80 2024-03-24T00:35:07.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0ef200640 1 -- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc0c4067660 msgr2=0x7fc0c4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:07.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0ef200640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc0c4067660 0x7fc0c4069b20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc0e815ebd0 tx=0x7fc0d8009040 comp rx=0 tx=0).stop 2024-03-24T00:35:07.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0ef200640 1 -- 172.21.15.40:0/1874673255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0e80cce80 msgr2=0x7fc0e815ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:07.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0ef200640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0e80cce80 0x7fc0e815ba30 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fc0e001a5d0 tx=0x7fc0e0019740 comp rx=0 tx=0).stop 2024-03-24T00:35:07.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0ef200640 1 -- 172.21.15.40:0/1874673255 shutdown_connections 2024-03-24T00:35:07.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0ef200640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc0c4067660 0x7fc0c4069b20 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0ef200640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0e80cce80 0x7fc0e815ba30 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0ef200640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0e80afc50 0x7fc0e815b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0ef200640 1 --2- 172.21.15.40:0/1874673255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0e806d5c0 0x7fc0e81518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0ef200640 1 -- 172.21.15.40:0/1874673255 >> 172.21.15.40:0/1874673255 conn(0x7fc0e80b21a0 msgr2=0x7fc0e80b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:07.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0ef200640 1 -- 172.21.15.40:0/1874673255 shutdown_connections 2024-03-24T00:35:07.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.243+0000 7fc0ef200640 1 -- 172.21.15.40:0/1874673255 wait complete. 2024-03-24T00:35:07.266 INFO:tasks.cephfs_test_runner:That turning off ephemeral distributed pin merges subtrees. ... ok 2024-03-24T00:35:07.266 INFO:tasks.cephfs_test_runner:Starting test: test_ephemeral_pin_dist_override (tasks.cephfs.test_exports.TestEphemeralPins) That an ephemeral distributed pin overrides a normal export pin. 2024-03-24T00:35:07.266 INFO:tasks.cephfs_test_runner:test_ephemeral_pin_dist_override (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:35:07.267 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:07.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.403+0000 7f2df9600640 1 Processor -- start 2024-03-24T00:35:07.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.403+0000 7f2df9600640 1 -- start start 2024-03-24T00:35:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2df9600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2df413ec80 0x7f2df41414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2df9600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2df41419f0 0x7f2df40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2df9600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2df40ce630 0x7f2df415f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2df9600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df4141e50 con 0x7f2df41419f0 2024-03-24T00:35:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2df9600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df4070e00 con 0x7f2df40ce630 2024-03-24T00:35:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2df9600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df40cee20 con 0x7f2df413ec80 2024-03-24T00:35:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2debe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2df41419f0 0x7f2df40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2debe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2df41419f0 0x7f2df40ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46328/0 (socket says 172.21.15.40:46328) 2024-03-24T00:35:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2debe00640 1 -- 172.21.15.40:0/335545169 learned_addr learned my addr 172.21.15.40:0/335545169 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2df3400640 1 --2- 172.21.15.40:0/335545169 >> v2:172.21.15.53:3300/0 conn(0x7f2df40ce630 0x7f2df415f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2df2a00640 1 --2- 172.21.15.40:0/335545169 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2df413ec80 0x7f2df41414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2debe00640 1 -- 172.21.15.40:0/335545169 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2df413ec80 msgr2=0x7f2df41414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2debe00640 1 --2- 172.21.15.40:0/335545169 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2df413ec80 0x7f2df41414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2debe00640 1 -- 172.21.15.40:0/335545169 >> v2:172.21.15.53:3300/0 conn(0x7f2df40ce630 msgr2=0x7f2df415f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2debe00640 1 --2- 172.21.15.40:0/335545169 >> v2:172.21.15.53:3300/0 conn(0x7f2df40ce630 0x7f2df415f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2debe00640 1 -- 172.21.15.40:0/335545169 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df40cf100 con 0x7f2df41419f0 2024-03-24T00:35:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2df3400640 1 --2- 172.21.15.40:0/335545169 >> v2:172.21.15.53:3300/0 conn(0x7f2df40ce630 0x7f2df415f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:35:07.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.407+0000 7f2df2a00640 1 --2- 172.21.15.40:0/335545169 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2df413ec80 0x7f2df41414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:35:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2debe00640 1 --2- 172.21.15.40:0/335545169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2df41419f0 0x7f2df40ce0f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f2df40b2b60 tx=0x7f2ddc01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59bf01e0af2e10b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df2000640 1 -- 172.21.15.40:0/335545169 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0b0070 con 0x7f2df41419f0 2024-03-24T00:35:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df2000640 1 -- 172.21.15.40:0/335545169 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ddc0ab600 con 0x7f2df41419f0 2024-03-24T00:35:07.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df2000640 1 -- 172.21.15.40:0/335545169 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0ab9d0 con 0x7f2df41419f0 2024-03-24T00:35:07.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 -- 172.21.15.40:0/335545169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2df41419f0 msgr2=0x7f2df40ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:07.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 --2- 172.21.15.40:0/335545169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2df41419f0 0x7f2df40ce0f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f2df40b2b60 tx=0x7f2ddc01c260 comp rx=0 tx=0).stop 2024-03-24T00:35:07.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 -- 172.21.15.40:0/335545169 shutdown_connections 2024-03-24T00:35:07.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 --2- 172.21.15.40:0/335545169 >> v2:172.21.15.53:3300/0 conn(0x7f2df40ce630 0x7f2df415f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 --2- 172.21.15.40:0/335545169 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2df41419f0 0x7f2df40ce0f0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 --2- 172.21.15.40:0/335545169 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2df413ec80 0x7f2df41414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 -- 172.21.15.40:0/335545169 >> 172.21.15.40:0/335545169 conn(0x7f2df40b17b0 msgr2=0x7f2df40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:07.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 -- 172.21.15.40:0/335545169 shutdown_connections 2024-03-24T00:35:07.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 -- 172.21.15.40:0/335545169 wait complete. 2024-03-24T00:35:07.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 Processor -- start 2024-03-24T00:35:07.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 -- start start 2024-03-24T00:35:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2df40ce630 0x7f2df4137df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2df413ec80 0x7f2df4150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2df41419f0 0x7f2df4153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df4071460 con 0x7f2df413ec80 2024-03-24T00:35:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.411+0000 7f2df9600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df40712e0 con 0x7f2df41419f0 2024-03-24T00:35:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2df9600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df40715e0 con 0x7f2df40ce630 2024-03-24T00:35:07.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2debe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2df413ec80 0x7f2df4150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2debe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2df413ec80 0x7f2df4150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46330/0 (socket says 172.21.15.40:46330) 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2debe00640 1 -- 172.21.15.40:0/1669940313 learned_addr learned my addr 172.21.15.40:0/1669940313 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2df2a00640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2df40ce630 0x7f2df4137df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2df3400640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2df41419f0 0x7f2df4153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2debe00640 1 -- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2df40ce630 msgr2=0x7f2df4137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2debe00640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2df40ce630 0x7f2df4137df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2debe00640 1 -- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2df41419f0 msgr2=0x7f2df4153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2debe00640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2df41419f0 0x7f2df4153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2debe00640 1 -- 172.21.15.40:0/1669940313 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df4151870 con 0x7f2df413ec80 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2debe00640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2df413ec80 0x7f2df4150c80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc01a910 tx=0x7f2ddc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2deaa00640 1 -- 172.21.15.40:0/1669940313 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0b9070 con 0x7f2df413ec80 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2deaa00640 1 -- 172.21.15.40:0/1669940313 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ddc0b4600 con 0x7f2df413ec80 2024-03-24T00:35:07.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2df9600640 1 -- 172.21.15.40:0/1669940313 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df4151b50 con 0x7f2df413ec80 2024-03-24T00:35:07.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2deaa00640 1 -- 172.21.15.40:0/1669940313 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc0b0040 con 0x7f2df413ec80 2024-03-24T00:35:07.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2df9600640 1 -- 172.21.15.40:0/1669940313 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df4153a90 con 0x7f2df413ec80 2024-03-24T00:35:07.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2df9600640 1 -- 172.21.15.40:0/1669940313 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df4068170 con 0x7f2df413ec80 2024-03-24T00:35:07.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.415+0000 7f2deaa00640 1 -- 172.21.15.40:0/1669940313 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2ddc0bc070 con 0x7f2df413ec80 2024-03-24T00:35:07.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.419+0000 7f2deaa00640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2dd0067660 0x7f2dd0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.419+0000 7f2deaa00640 1 -- 172.21.15.40:0/1669940313 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f2ddc11e5a0 con 0x7f2df413ec80 2024-03-24T00:35:07.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.419+0000 7f2df2a00640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2dd0067660 0x7f2dd0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.419+0000 7f2df2a00640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2dd0067660 0x7f2dd0069b20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2de400aae0 tx=0x7f2de4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:07.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.423+0000 7f2deaa00640 1 -- 172.21.15.40:0/1669940313 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2ddc016110 con 0x7f2df413ec80 2024-03-24T00:35:07.458 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:35:07.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:35:07.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.591+0000 7f2df9600640 1 -- 172.21.15.40:0/1669940313 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2df4072b90 con 0x7f2df413ec80 2024-03-24T00:35:07.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.591+0000 7f2deaa00640 1 -- 172.21.15.40:0/1669940313 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+13218 (secure 0 0 0) 0x7f2ddc0eeea0 con 0x7f2df413ec80 2024-03-24T00:35:07.594 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:07.595 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":62,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:35:05.232438+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6835/1312292485":"2024-03-25T00:35:02.218817+0000","172.21.15.40:6833/1312292485":"2024-03-25T00:35:02.218817+0000","172.21.15.53:6835/501079401":"2024-03-25T00:35:02.218824+0000","172.21.15.53:6834/501079401":"2024-03-25T00:35:02.218824+0000","172.21.15.40:6834/1278515461":"2024-03-25T00:35:02.218801+0000","172.21.15.40:6832/1278515461":"2024-03-25T00:35:02.218801+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:07.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.595+0000 7f2df9600640 1 -- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2dd0067660 msgr2=0x7f2dd0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:07.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.595+0000 7f2df9600640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2dd0067660 0x7f2dd0069b20 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2de400aae0 tx=0x7f2de4008040 comp rx=0 tx=0).stop 2024-03-24T00:35:07.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.595+0000 7f2df9600640 1 -- 172.21.15.40:0/1669940313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2df413ec80 msgr2=0x7f2df4150c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:07.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.595+0000 7f2df9600640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2df413ec80 0x7f2df4150c80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc01a910 tx=0x7f2ddc019000 comp rx=0 tx=0).stop 2024-03-24T00:35:07.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.595+0000 7f2df9600640 1 -- 172.21.15.40:0/1669940313 shutdown_connections 2024-03-24T00:35:07.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.595+0000 7f2df9600640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2dd0067660 0x7f2dd0069b20 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.595+0000 7f2df9600640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2df41419f0 0x7f2df4153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.595+0000 7f2df9600640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2df413ec80 0x7f2df4150c80 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.595+0000 7f2df9600640 1 --2- 172.21.15.40:0/1669940313 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2df40ce630 0x7f2df4137df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.595+0000 7f2df9600640 1 -- 172.21.15.40:0/1669940313 >> 172.21.15.40:0/1669940313 conn(0x7f2df40b17b0 msgr2=0x7f2df41501c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:07.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.595+0000 7f2df9600640 1 -- 172.21.15.40:0/1669940313 shutdown_connections 2024-03-24T00:35:07.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.595+0000 7f2df9600640 1 -- 172.21.15.40:0/1669940313 wait complete. 2024-03-24T00:35:07.616 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:35:07.616 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:35:07.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.802+0000 7fce06200640 1 Processor -- start 2024-03-24T00:35:07.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.802+0000 7fce06200640 1 -- start start 2024-03-24T00:35:07.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.802+0000 7fce06200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce0015ac60 0x7fce000b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.802+0000 7fce06200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce0006cb80 0x7fce000b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.802+0000 7fce06200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce000b3260 con 0x7fce000b4840 2024-03-24T00:35:07.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fce06200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce000b3e70 con 0x7fce0015ac60 2024-03-24T00:35:07.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fce06200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce000b3540 con 0x7fce0006cb80 2024-03-24T00:35:07.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fcdff400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce0015ac60 0x7fce000b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fcdff400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce0015ac60 0x7fce000b2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46340/0 (socket says 172.21.15.40:46340) 2024-03-24T00:35:07.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fcdff400640 1 -- 172.21.15.40:0/699287877 learned_addr learned my addr 172.21.15.40:0/699287877 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:07.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fcdf7e00640 1 --2- 172.21.15.40:0/699287877 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce0006cb80 0x7fce000b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fcdff400640 1 -- 172.21.15.40:0/699287877 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce0006cb80 msgr2=0x7fce000b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:07.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fcdff400640 1 --2- 172.21.15.40:0/699287877 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce0006cb80 0x7fce000b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fcdff400640 1 -- 172.21.15.40:0/699287877 >> v1:172.21.15.53:6789/0 conn(0x7fce000b4840 legacy=0x7fce000ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:07.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fcdff400640 1 -- 172.21.15.40:0/699287877 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce000b3820 con 0x7fce0015ac60 2024-03-24T00:35:07.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fcdff400640 1 --2- 172.21.15.40:0/699287877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce0015ac60 0x7fce000b2960 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009870 tx=0x7fcdf001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649b6c9e1d83ef1d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:07.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fcdfea00640 1 -- 172.21.15.40:0/699287877 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf00b1070 con 0x7fce0015ac60 2024-03-24T00:35:07.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fcdfea00640 1 -- 172.21.15.40:0/699287877 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf00ac410 con 0x7fce0015ac60 2024-03-24T00:35:07.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.806+0000 7fcdfea00640 1 -- 172.21.15.40:0/699287877 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf00ac7e0 con 0x7fce0015ac60 2024-03-24T00:35:07.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 -- 172.21.15.40:0/699287877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce0015ac60 msgr2=0x7fce000b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:07.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 --2- 172.21.15.40:0/699287877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce0015ac60 0x7fce000b2960 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009870 tx=0x7fcdf001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 -- 172.21.15.40:0/699287877 shutdown_connections 2024-03-24T00:35:07.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 --2- 172.21.15.40:0/699287877 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce0006cb80 0x7fce000b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 --2- 172.21.15.40:0/699287877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce0015ac60 0x7fce000b2960 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 -- 172.21.15.40:0/699287877 >> 172.21.15.40:0/699287877 conn(0x7fce000b1660 msgr2=0x7fce000b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:07.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 -- 172.21.15.40:0/699287877 shutdown_connections 2024-03-24T00:35:07.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 -- 172.21.15.40:0/699287877 wait complete. 2024-03-24T00:35:07.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 Processor -- start 2024-03-24T00:35:07.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 -- start start 2024-03-24T00:35:07.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce0006cb80 0x7fce0014a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce000b4840 0x7fce0015dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce0015ac60 0x7fce000c6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce00167a40 con 0x7fce0006cb80 2024-03-24T00:35:07.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce001678c0 con 0x7fce000b4840 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fce06200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce00167bc0 con 0x7fce0015ac60 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fcdff400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce0006cb80 0x7fce0014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fcdff400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce0006cb80 0x7fce0014a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46346/0 (socket says 172.21.15.40:46346) 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fcdff400640 1 -- 172.21.15.40:0/2354619449 learned_addr learned my addr 172.21.15.40:0/2354619449 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fcdf7e00640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce000b4840 0x7fce0015dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fcdffe00640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce0015ac60 0x7fce000c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fcdff400640 1 -- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce0015ac60 msgr2=0x7fce000c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fcdff400640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce0015ac60 0x7fce000c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fcdff400640 1 -- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce000b4840 msgr2=0x7fce0015dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fcdff400640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce000b4840 0x7fce0015dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.810+0000 7fcdff400640 1 -- 172.21.15.40:0/2354619449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce00157a50 con 0x7fce0006cb80 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.814+0000 7fcdff400640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce0006cb80 0x7fce0014a360 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fcdf00af040 tx=0x7fcdf0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.814+0000 7fcdf7400640 1 -- 172.21.15.40:0/2354619449 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf00bc070 con 0x7fce0006cb80 2024-03-24T00:35:07.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.814+0000 7fcdf7400640 1 -- 172.21.15.40:0/2354619449 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf00acdc0 con 0x7fce0006cb80 2024-03-24T00:35:07.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.814+0000 7fcdf7400640 1 -- 172.21.15.40:0/2354619449 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf00b1040 con 0x7fce0006cb80 2024-03-24T00:35:07.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.814+0000 7fce06200640 1 -- 172.21.15.40:0/2354619449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce0014a8a0 con 0x7fce0006cb80 2024-03-24T00:35:07.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.814+0000 7fce06200640 1 -- 172.21.15.40:0/2354619449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce000c5910 con 0x7fce0006cb80 2024-03-24T00:35:07.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.814+0000 7fce06200640 1 -- 172.21.15.40:0/2354619449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce00068170 con 0x7fce0006cb80 2024-03-24T00:35:07.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.814+0000 7fcdf7400640 1 -- 172.21.15.40:0/2354619449 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcdf00bf070 con 0x7fce0006cb80 2024-03-24T00:35:07.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.814+0000 7fcdf7400640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcddc067660 0x7fcddc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:07.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.814+0000 7fcdf7400640 1 -- 172.21.15.40:0/2354619449 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fcdf011ed70 con 0x7fce0006cb80 2024-03-24T00:35:07.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.814+0000 7fcdf7e00640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcddc067660 0x7fcddc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:07.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.818+0000 7fcdf7e00640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcddc067660 0x7fcddc069b20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fce001f9fe0 tx=0x7fcde8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:07.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.818+0000 7fcdf7400640 1 -- 172.21.15.40:0/2354619449 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fce00157a50 con 0x7fce0006cb80 2024-03-24T00:35:07.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 -- 172.21.15.40:0/2354619449 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fce0006bad0 con 0x7fce0006cb80 2024-03-24T00:35:07.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fcdf7400640 1 -- 172.21.15.40:0/2354619449 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcdf0016150 con 0x7fce0006cb80 2024-03-24T00:35:07.989 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:35:07.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 -- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcddc067660 msgr2=0x7fcddc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:07.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcddc067660 0x7fcddc069b20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fce001f9fe0 tx=0x7fcde8008040 comp rx=0 tx=0).stop 2024-03-24T00:35:07.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 -- 172.21.15.40:0/2354619449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce0006cb80 msgr2=0x7fce0014a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:07.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce0006cb80 0x7fce0014a360 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fcdf00af040 tx=0x7fcdf0019240 comp rx=0 tx=0).stop 2024-03-24T00:35:07.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 -- 172.21.15.40:0/2354619449 shutdown_connections 2024-03-24T00:35:07.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcddc067660 0x7fcddc069b20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce0015ac60 0x7fce000c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce000b4840 0x7fce0015dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 --2- 172.21.15.40:0/2354619449 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce0006cb80 0x7fce0014a360 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:07.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 -- 172.21.15.40:0/2354619449 >> 172.21.15.40:0/2354619449 conn(0x7fce000b1660 msgr2=0x7fce000ae6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:07.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 -- 172.21.15.40:0/2354619449 shutdown_connections 2024-03-24T00:35:07.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:07.986+0000 7fce06200640 1 -- 172.21.15.40:0/2354619449 wait complete. 2024-03-24T00:35:08.011 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override' 2024-03-24T00:35:08.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fbae00640 1 Processor -- start 2024-03-24T00:35:08.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fbae00640 1 -- start start 2024-03-24T00:35:08.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fbae00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3fb40ce630 0x7f3fb40d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:08.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fbae00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fb40cea10 0x7f3fb406b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:08.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fbae00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb40cc190 con 0x7f3fb40cfdc0 2024-03-24T00:35:08.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fbae00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fb40d1080 con 0x7f3fb40ce630 2024-03-24T00:35:08.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fbae00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb40d0da0 con 0x7f3fb40cea10 2024-03-24T00:35:08.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fb9a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3fb40ce630 0x7f3fb40d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:08.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fb9a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3fb40ce630 0x7f3fb40d2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46360/0 (socket says 172.21.15.40:46360) 2024-03-24T00:35:08.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fb9a00640 1 -- 172.21.15.40:0/3359158159 learned_addr learned my addr 172.21.15.40:0/3359158159 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:08.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fb9000640 1 --2- 172.21.15.40:0/3359158159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fb40cea10 0x7f3fb406b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fb9a00640 1 -- 172.21.15.40:0/3359158159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fb40cea10 msgr2=0x7f3fb406b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fb9a00640 1 --2- 172.21.15.40:0/3359158159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fb40cea10 0x7f3fb406b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fb9a00640 1 -- 172.21.15.40:0/3359158159 >> v1:172.21.15.53:6789/0 conn(0x7f3fb40cfdc0 legacy=0x7f3fb406b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.150+0000 7f3fb9a00640 1 -- 172.21.15.40:0/3359158159 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb40cc430 con 0x7f3fb40ce630 2024-03-24T00:35:08.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fb9a00640 1 --2- 172.21.15.40:0/3359158159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fb40ce630 0x7f3fb40d2590 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8009870 tx=0x7f3fa801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b0578dffc8297 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:08.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fb3e00640 1 -- 172.21.15.40:0/3359158159 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa80b5070 con 0x7f3fb40ce630 2024-03-24T00:35:08.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fb3e00640 1 -- 172.21.15.40:0/3359158159 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa80a8a50 con 0x7f3fb40ce630 2024-03-24T00:35:08.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fb3e00640 1 -- 172.21.15.40:0/3359158159 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa80a8e20 con 0x7f3fb40ce630 2024-03-24T00:35:08.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fbae00640 1 -- 172.21.15.40:0/3359158159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fb40ce630 msgr2=0x7f3fb40d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:08.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fbae00640 1 --2- 172.21.15.40:0/3359158159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fb40ce630 0x7f3fb40d2590 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8009870 tx=0x7f3fa801ed50 comp rx=0 tx=0).stop 2024-03-24T00:35:08.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fbae00640 1 -- 172.21.15.40:0/3359158159 shutdown_connections 2024-03-24T00:35:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fbae00640 1 --2- 172.21.15.40:0/3359158159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fb40cea10 0x7f3fb406b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fbae00640 1 --2- 172.21.15.40:0/3359158159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fb40ce630 0x7f3fb40d2590 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fbae00640 1 -- 172.21.15.40:0/3359158159 >> 172.21.15.40:0/3359158159 conn(0x7f3fb40d1aa0 msgr2=0x7f3fb40d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fbae00640 1 -- 172.21.15.40:0/3359158159 shutdown_connections 2024-03-24T00:35:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fbae00640 1 -- 172.21.15.40:0/3359158159 wait complete. 2024-03-24T00:35:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fbae00640 1 Processor -- start 2024-03-24T00:35:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.154+0000 7f3fbae00640 1 -- start start 2024-03-24T00:35:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fbae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fb40ce630 0x7f3fb415cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fbae00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fb40cea10 0x7f3fb40c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fbae00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fb40cfdc0 0x7f3fb40caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fbae00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb406c640 con 0x7f3fb40cea10 2024-03-24T00:35:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fbae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb406c4c0 con 0x7f3fb40ce630 2024-03-24T00:35:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fbae00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb406bfb0 con 0x7f3fb40cfdc0 2024-03-24T00:35:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb9000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fb40cea10 0x7f3fb40c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb9000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fb40cea10 0x7f3fb40c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46364/0 (socket says 172.21.15.40:46364) 2024-03-24T00:35:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb9000640 1 -- 172.21.15.40:0/2078776606 learned_addr learned my addr 172.21.15.40:0/2078776606 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb9000640 1 -- 172.21.15.40:0/2078776606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fb40cfdc0 msgr2=0x7f3fb40caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb9000640 1 --2- 172.21.15.40:0/2078776606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fb40cfdc0 0x7f3fb40caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb9000640 1 -- 172.21.15.40:0/2078776606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fb40ce630 msgr2=0x7f3fb415cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:08.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb9000640 1 --2- 172.21.15.40:0/2078776606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fb40ce630 0x7f3fb415cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:08.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb9000640 1 -- 172.21.15.40:0/2078776606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb415e0b0 con 0x7f3fb40cea10 2024-03-24T00:35:08.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb9000640 1 --2- 172.21.15.40:0/2078776606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fb40cea10 0x7f3fb40c9a60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3fa000d5f0 tx=0x7f3fa000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:08.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb1600640 1 -- 172.21.15.40:0/2078776606 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa00125f0 con 0x7f3fb40cea10 2024-03-24T00:35:08.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb1600640 1 -- 172.21.15.40:0/2078776606 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa0019070 con 0x7f3fb40cea10 2024-03-24T00:35:08.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fbae00640 1 -- 172.21.15.40:0/2078776606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb40ca570 con 0x7f3fb40cea10 2024-03-24T00:35:08.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb1600640 1 -- 172.21.15.40:0/2078776606 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fa00129b0 con 0x7f3fb40cea10 2024-03-24T00:35:08.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fbae00640 1 -- 172.21.15.40:0/2078776606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb415e650 con 0x7f3fb40cea10 2024-03-24T00:35:08.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fbae00640 1 -- 172.21.15.40:0/2078776606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb409fb10 con 0x7f3fb40cea10 2024-03-24T00:35:08.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.158+0000 7f3fb1600640 1 -- 172.21.15.40:0/2078776606 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3fa0012b50 con 0x7f3fb40cea10 2024-03-24T00:35:08.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.162+0000 7f3fb1600640 1 --2- 172.21.15.40:0/2078776606 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f94067660 0x7f3f94069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:08.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.162+0000 7f3fb1600640 1 -- 172.21.15.40:0/2078776606 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3fa0085b60 con 0x7f3fb40cea10 2024-03-24T00:35:08.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.162+0000 7f3fb9a00640 1 --2- 172.21.15.40:0/2078776606 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f94067660 0x7f3f94069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:08.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.162+0000 7f3fb9a00640 1 --2- 172.21.15.40:0/2078776606 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f94067660 0x7f3f94069b20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3fa80b2000 tx=0x7f3fa8004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:08.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.166+0000 7f3fb1600640 1 -- 172.21.15.40:0/2078776606 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3fa0056990 con 0x7f3fb40cea10 2024-03-24T00:35:08.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:08.386+0000 7f3fbae00640 1 -- 172.21.15.40:0/2078776606 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override"]} v 0) v1 -- 0x7f3fb40c88a0 con 0x7f3fb40cea10 2024-03-24T00:35:09.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.250+0000 7f3fb1600640 1 -- 172.21.15.40:0/2078776606 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override"]}]=0 v220) v1 ==== 158+0+0 (secure 0 0 0) 0x7f3fa0056290 con 0x7f3fb40cea10 2024-03-24T00:35:09.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.254+0000 7f3fbae00640 1 -- 172.21.15.40:0/2078776606 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f94067660 msgr2=0x7f3f94069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:09.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.254+0000 7f3fbae00640 1 --2- 172.21.15.40:0/2078776606 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f94067660 0x7f3f94069b20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3fa80b2000 tx=0x7f3fa8004740 comp rx=0 tx=0).stop 2024-03-24T00:35:09.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.254+0000 7f3fbae00640 1 -- 172.21.15.40:0/2078776606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fb40cea10 msgr2=0x7f3fb40c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:09.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.254+0000 7f3fbae00640 1 --2- 172.21.15.40:0/2078776606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fb40cea10 0x7f3fb40c9a60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f3fa000d5f0 tx=0x7f3fa000dab0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.254+0000 7f3fbae00640 1 -- 172.21.15.40:0/2078776606 shutdown_connections 2024-03-24T00:35:09.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.254+0000 7f3fbae00640 1 --2- 172.21.15.40:0/2078776606 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f94067660 0x7f3f94069b20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.254+0000 7f3fbae00640 1 --2- 172.21.15.40:0/2078776606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fb40cfdc0 0x7f3fb40caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.254+0000 7f3fbae00640 1 --2- 172.21.15.40:0/2078776606 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fb40cea10 0x7f3fb40c9a60 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.254+0000 7f3fbae00640 1 --2- 172.21.15.40:0/2078776606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fb40ce630 0x7f3fb415cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.254+0000 7f3fbae00640 1 -- 172.21.15.40:0/2078776606 >> 172.21.15.40:0/2078776606 conn(0x7f3fb40d1aa0 msgr2=0x7f3fb406fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:09.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.254+0000 7f3fbae00640 1 -- 172.21.15.40:0/2078776606 shutdown_connections 2024-03-24T00:35:09.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.254+0000 7f3fbae00640 1 -- 172.21.15.40:0/2078776606 wait complete. 2024-03-24T00:35:09.281 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:35:09.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f9652200640 1 Processor -- start 2024-03-24T00:35:09.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f9652200640 1 -- start start 2024-03-24T00:35:09.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f9652200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c072b60 0x7f964c0b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:09.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f9652200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c15f2c0 con 0x7f964c0cf4e0 2024-03-24T00:35:09.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f9652200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c06d230 con 0x7f964c0724a0 2024-03-24T00:35:09.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f9652200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964c06cfe0 con 0x7f964c072b60 2024-03-24T00:35:09.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f964c0cf4e0 0x7f964c06dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48428/0 (socket says 172.21.15.40:48428) 2024-03-24T00:35:09.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964b400640 1 -- 172.21.15.40:0/1118596739 learned_addr learned my addr 172.21.15.40:0/1118596739 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964aa00640 1 --2- 172.21.15.40:0/1118596739 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c072b60 0x7f964c0b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976474353 0 0) 0x7f964c15f2c0 con 0x7f964c0cf4e0 2024-03-24T00:35:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9630003530 con 0x7f964c0cf4e0 2024-03-24T00:35:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1174795911 0 0) 0x7f9630003530 con 0x7f964c0cf4e0 2024-03-24T00:35:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c072b60 msgr2=0x7f964c0b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 --2- 172.21.15.40:0/1118596739 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c072b60 0x7f964c0b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 >> v1:172.21.15.53:6789/0 conn(0x7f964c0724a0 legacy=0x7f964c070ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c06d510 con 0x7f964c0cf4e0 2024-03-24T00:35:09.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f963c001680 con 0x7f964c0cf4e0 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 >> v1:172.21.15.40:6789/0 conn(0x7f964c0cf4e0 legacy=0x7f964c06dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 --2- 172.21.15.40:0/1118596739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96300039f0 0x7f9630003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 --2- 172.21.15.40:0/1118596739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9630004810 0x7f9630004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 --2- 172.21.15.40:0/1118596739 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c072b60 0x7f96300051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c06cfe0 con 0x7f9630004810 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963000bb10 con 0x7f96300039f0 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963000bd10 con 0x7f964c072b60 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964aa00640 1 --2- 172.21.15.40:0/1118596739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9630004810 0x7f9630004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964b400640 1 --2- 172.21.15.40:0/1118596739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96300039f0 0x7f9630003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964be00640 1 --2- 172.21.15.40:0/1118596739 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c072b60 0x7f96300051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964aa00640 1 -- 172.21.15.40:0/1118596739 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c072b60 msgr2=0x7f96300051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964aa00640 1 --2- 172.21.15.40:0/1118596739 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c072b60 0x7f96300051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964aa00640 1 -- 172.21.15.40:0/1118596739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96300039f0 msgr2=0x7f9630003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:09.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964aa00640 1 --2- 172.21.15.40:0/1118596739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96300039f0 0x7f9630003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.426+0000 7f964aa00640 1 -- 172.21.15.40:0/1118596739 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9630003530 con 0x7f9630004810 2024-03-24T00:35:09.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.430+0000 7f964aa00640 1 --2- 172.21.15.40:0/1118596739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9630004810 0x7f9630004c90 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f964c072fa0 tx=0x7f963801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff075268774d2370 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:09.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.430+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96380b2070 con 0x7f9630004810 2024-03-24T00:35:09.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.430+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96380a55e0 con 0x7f9630004810 2024-03-24T00:35:09.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.430+0000 7f964a000640 1 -- 172.21.15.40:0/1118596739 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96380096b0 con 0x7f9630004810 2024-03-24T00:35:09.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.430+0000 7f9652200640 1 -- 172.21.15.40:0/1118596739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9630004810 msgr2=0x7f9630004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:09.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.430+0000 7f9652200640 1 --2- 172.21.15.40:0/1118596739 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9630004810 0x7f9630004c90 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f964c072fa0 tx=0x7f963801c9f0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.430+0000 7f964be00640 1 -- 172.21.15.40:0/1118596739 reap_dead start 2024-03-24T00:35:09.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.430+0000 7f9652200640 1 -- 172.21.15.40:0/1118596739 shutdown_connections 2024-03-24T00:35:09.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.430+0000 7f9652200640 1 -- 172.21.15.40:0/1118596739 >> 172.21.15.40:0/1118596739 conn(0x7f964c0b1660 msgr2=0x7f964c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:09.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.430+0000 7f9652200640 1 -- 172.21.15.40:0/1118596739 shutdown_connections 2024-03-24T00:35:09.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.430+0000 7f9652200640 1 -- 172.21.15.40:0/1118596739 wait complete. 2024-03-24T00:35:09.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9652200640 1 Processor -- start 2024-03-24T00:35:09.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9652200640 1 -- start start 2024-03-24T00:35:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9652200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9630004810 0x7f964c15c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9652200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f964c1608c0 0x7f964c160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9652200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f964c149c80 0x7f964c14a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9652200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c0712b0 con 0x7f964c1608c0 2024-03-24T00:35:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9652200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c06d880 con 0x7f964c149c80 2024-03-24T00:35:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9652200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964c071430 con 0x7f9630004810 2024-03-24T00:35:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f964aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f964c1608c0 0x7f964c160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f964aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f964c1608c0 0x7f964c160d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46394/0 (socket says 172.21.15.40:46394) 2024-03-24T00:35:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f964aa00640 1 -- 172.21.15.40:0/3961627491 learned_addr learned my addr 172.21.15.40:0/3961627491 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:09.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f964b400640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9630004810 0x7f964c15c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f964be00640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f964c149c80 0x7f964c14a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f964aa00640 1 -- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9630004810 msgr2=0x7f964c15c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f964aa00640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9630004810 0x7f964c15c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f964aa00640 1 -- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f964c149c80 msgr2=0x7f964c14a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f964aa00640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f964c149c80 0x7f964c14a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f964aa00640 1 -- 172.21.15.40:0/3961627491 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c1f41e0 con 0x7f964c1608c0 2024-03-24T00:35:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f964aa00640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f964c1608c0 0x7f964c160d20 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f963801c820 tx=0x7f9630005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9643400640 1 -- 172.21.15.40:0/3961627491 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96380b2040 con 0x7f964c1608c0 2024-03-24T00:35:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9643400640 1 -- 172.21.15.40:0/3961627491 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9638004020 con 0x7f964c1608c0 2024-03-24T00:35:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9652200640 1 -- 172.21.15.40:0/3961627491 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964c1f44a0 con 0x7f964c1608c0 2024-03-24T00:35:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9643400640 1 -- 172.21.15.40:0/3961627491 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96380096b0 con 0x7f964c1608c0 2024-03-24T00:35:09.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.434+0000 7f9652200640 1 -- 172.21.15.40:0/3961627491 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964c1f4a60 con 0x7f964c1608c0 2024-03-24T00:35:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.438+0000 7f9652200640 1 -- 172.21.15.40:0/3961627491 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964c068170 con 0x7f964c1608c0 2024-03-24T00:35:09.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.438+0000 7f9643400640 1 -- 172.21.15.40:0/3961627491 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f96380c0060 con 0x7f964c1608c0 2024-03-24T00:35:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.438+0000 7f9643400640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f962c067660 0x7f962c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.438+0000 7f9643400640 1 -- 172.21.15.40:0/3961627491 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f963811f270 con 0x7f964c1608c0 2024-03-24T00:35:09.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.438+0000 7f964b400640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f962c067660 0x7f962c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.438+0000 7f964b400640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f962c067660 0x7f962c069b20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f963c0011f0 tx=0x7f963c001160 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:09.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.442+0000 7f9643400640 1 -- 172.21.15.40:0/3961627491 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9638016700 con 0x7f964c1608c0 2024-03-24T00:35:09.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.606+0000 7f9652200640 1 -- 172.21.15.40:0/3961627491 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f964c0b45e0 con 0x7f964c1608c0 2024-03-24T00:35:09.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9643400640 1 -- 172.21.15.40:0/3961627491 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v35)=0 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7f964c0b45e0 con 0x7f964c1608c0 2024-03-24T00:35:09.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9652200640 1 -- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f962c067660 msgr2=0x7f962c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:09.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9652200640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f962c067660 0x7f962c069b20 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f963c0011f0 tx=0x7f963c001160 comp rx=0 tx=0).stop 2024-03-24T00:35:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9652200640 1 -- 172.21.15.40:0/3961627491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f964c1608c0 msgr2=0x7f964c160d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9652200640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f964c1608c0 0x7f964c160d20 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f963801c820 tx=0x7f9630005840 comp rx=0 tx=0).stop 2024-03-24T00:35:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9652200640 1 -- 172.21.15.40:0/3961627491 shutdown_connections 2024-03-24T00:35:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9652200640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f962c067660 0x7f962c069b20 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9652200640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f964c149c80 0x7f964c14a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9652200640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f964c1608c0 0x7f964c160d20 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9652200640 1 --2- 172.21.15.40:0/3961627491 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9630004810 0x7f964c15c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:09.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9652200640 1 -- 172.21.15.40:0/3961627491 >> 172.21.15.40:0/3961627491 conn(0x7f964c0b1660 msgr2=0x7f964c0ae470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9652200640 1 -- 172.21.15.40:0/3961627491 shutdown_connections 2024-03-24T00:35:09.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:09.610+0000 7f9652200640 1 -- 172.21.15.40:0/3961627491 wait complete. 2024-03-24T00:35:09.633 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:35:09.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:35:09.726 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:35:09.811 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:35:09.811 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:35:09.811 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:35:09.820 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:35:09.820 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:35:09.820 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:35:09.830 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:35:09.830 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:35:09.830 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:35:09.871 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:35:09.871 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:35:09.871 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:35:09.880 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:35:09.880 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:35:09.881 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:10.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f2e200640 1 Processor -- start 2024-03-24T00:35:10.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f2e200640 1 -- start start 2024-03-24T00:35:10.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f2e200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f280cf330 0x7f8f280d0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f2e200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f280ccb50 con 0x7f8f2815c1d0 2024-03-24T00:35:10.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f2e200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f280cce30 con 0x7f8f2815bdf0 2024-03-24T00:35:10.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f2e200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f280cd110 con 0x7f8f280cf330 2024-03-24T00:35:10.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f1ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f280cf330 0x7f8f280d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f1ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8f280cf330 0x7f8f280d0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46396/0 (socket says 172.21.15.40:46396) 2024-03-24T00:35:10.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f1ea00640 1 -- 172.21.15.40:0/3903511070 learned_addr learned my addr 172.21.15.40:0/3903511070 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:10.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f27e00640 1 --1- 172.21.15.40:0/3903511070 >> v1:172.21.15.40:6789/0 conn(0x7f8f2815c1d0 0x7f8f2806ef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48430/0 (socket says 172.21.15.40:48430) 2024-03-24T00:35:10.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f1ea00640 1 -- 172.21.15.40:0/3903511070 >> v1:172.21.15.53:6789/0 conn(0x7f8f2815bdf0 legacy=0x7f8f280cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f1ea00640 1 -- 172.21.15.40:0/3903511070 >> v1:172.21.15.40:6789/0 conn(0x7f8f2815c1d0 legacy=0x7f8f2806ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f26a00640 1 -- 172.21.15.40:0/3903511070 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539992399 0 0) 0x7f8f280ccb50 con 0x7f8f2815c1d0 2024-03-24T00:35:10.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.054+0000 7f8f1ea00640 1 -- 172.21.15.40:0/3903511070 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f280cd3f0 con 0x7f8f280cf330 2024-03-24T00:35:10.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.058+0000 7f8f1ea00640 1 --2- 172.21.15.40:0/3903511070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f280cf330 0x7f8f280d0260 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009870 tx=0x7f8f1001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab3b3776ccbda56 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:10.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.058+0000 7f8f26a00640 1 -- 172.21.15.40:0/3903511070 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f100b2070 con 0x7f8f280cf330 2024-03-24T00:35:10.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.058+0000 7f8f26a00640 1 -- 172.21.15.40:0/3903511070 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f100a9070 con 0x7f8f280cf330 2024-03-24T00:35:10.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.058+0000 7f8f26a00640 1 -- 172.21.15.40:0/3903511070 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f100ad880 con 0x7f8f280cf330 2024-03-24T00:35:10.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.058+0000 7f8f2e200640 1 -- 172.21.15.40:0/3903511070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f280cf330 msgr2=0x7f8f280d0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.058+0000 7f8f2e200640 1 --2- 172.21.15.40:0/3903511070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f280cf330 0x7f8f280d0260 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009870 tx=0x7f8f1001c560 comp rx=0 tx=0).stop 2024-03-24T00:35:10.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.058+0000 7f8f2e200640 1 -- 172.21.15.40:0/3903511070 shutdown_connections 2024-03-24T00:35:10.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.058+0000 7f8f2e200640 1 --2- 172.21.15.40:0/3903511070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f280cf330 0x7f8f280d0260 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.058+0000 7f8f2e200640 1 -- 172.21.15.40:0/3903511070 >> 172.21.15.40:0/3903511070 conn(0x7f8f280b37f0 msgr2=0x7f8f280b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:10.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.058+0000 7f8f2e200640 1 -- 172.21.15.40:0/3903511070 shutdown_connections 2024-03-24T00:35:10.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.058+0000 7f8f2e200640 1 -- 172.21.15.40:0/3903511070 wait complete. 2024-03-24T00:35:10.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.058+0000 7f8f2e200640 1 Processor -- start 2024-03-24T00:35:10.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f2e200640 1 -- start start 2024-03-24T00:35:10.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f2e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f280cf330 0x7f8f280c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f2e200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f2815bdf0 0x7f8f280c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f2e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f2815c1d0 0x7f8f28154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f2e200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2816a260 con 0x7f8f2815c1d0 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f2e200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2816a0e0 con 0x7f8f280cf330 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f2e200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f2816a3e0 con 0x7f8f2815bdf0 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f27e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f2815c1d0 0x7f8f28154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f27e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f2815c1d0 0x7f8f28154230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46398/0 (socket says 172.21.15.40:46398) 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f27e00640 1 -- 172.21.15.40:0/3161982703 learned_addr learned my addr 172.21.15.40:0/3161982703 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f1ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f2815bdf0 0x7f8f280c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f27400640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f280cf330 0x7f8f280c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f27e00640 1 -- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f2815bdf0 msgr2=0x7f8f280c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f27e00640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f2815bdf0 0x7f8f280c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f27e00640 1 -- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f280cf330 msgr2=0x7f8f280c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f27e00640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f280cf330 0x7f8f280c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f27e00640 1 -- 172.21.15.40:0/3161982703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f280b5180 con 0x7f8f2815c1d0 2024-03-24T00:35:10.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f27e00640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f2815c1d0 0x7f8f28154230 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f8f2000c7a0 tx=0x7f8f2000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:10.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f1fe00640 1 -- 172.21.15.40:0/3161982703 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f20018070 con 0x7f8f2815c1d0 2024-03-24T00:35:10.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f1fe00640 1 -- 172.21.15.40:0/3161982703 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2000a640 con 0x7f8f2815c1d0 2024-03-24T00:35:10.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f2e200640 1 -- 172.21.15.40:0/3161982703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f280cb2c0 con 0x7f8f2815c1d0 2024-03-24T00:35:10.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f1fe00640 1 -- 172.21.15.40:0/3161982703 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2000aab0 con 0x7f8f2815c1d0 2024-03-24T00:35:10.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f2e200640 1 -- 172.21.15.40:0/3161982703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f280c6140 con 0x7f8f2815c1d0 2024-03-24T00:35:10.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.062+0000 7f8f2e200640 1 -- 172.21.15.40:0/3161982703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f28068170 con 0x7f8f2815c1d0 2024-03-24T00:35:10.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.066+0000 7f8f1fe00640 1 -- 172.21.15.40:0/3161982703 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8f2001b070 con 0x7f8f2815c1d0 2024-03-24T00:35:10.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.066+0000 7f8f1fe00640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8f04067660 0x7f8f04069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.066+0000 7f8f1fe00640 1 -- 172.21.15.40:0/3161982703 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f8f20084b80 con 0x7f8f2815c1d0 2024-03-24T00:35:10.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.066+0000 7f8f27400640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8f04067660 0x7f8f04069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.066+0000 7f8f27400640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8f04067660 0x7f8f04069b20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8f18007fd0 tx=0x7f8f18009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.070+0000 7f8f1fe00640 1 -- 172.21.15.40:0/3161982703 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8f280cb2c0 con 0x7f8f2815c1d0 2024-03-24T00:35:10.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.266+0000 7f8f2e200640 1 -- 172.21.15.40:0/3161982703 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f280b7c20 con 0x7f8f2815c1d0 2024-03-24T00:35:10.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.266+0000 7f8f1fe00640 1 -- 172.21.15.40:0/3161982703 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f8f280b5180 con 0x7f8f2815c1d0 2024-03-24T00:35:10.271 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:10.271 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":60,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6277,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1633817617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":1633817617},{"type":"v1","addr":"172.21.15.53:6837","nonce":1633817617}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":47},{"gid":6631,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2201251471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2201251471},{"type":"v1","addr":"172.21.15.53:6835","nonce":2201251471}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6852,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/2500457564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":2500457564},{"type":"v1","addr":"172.21.15.40:6833","nonce":2500457564}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6858,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1097408817","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":1097408817},{"type":"v1","addr":"172.21.15.40:6835","nonce":1097408817}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[]} 2024-03-24T00:35:10.271 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 60 2024-03-24T00:35:10.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.270+0000 7f8f2e200640 1 -- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8f04067660 msgr2=0x7f8f04069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.270+0000 7f8f2e200640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8f04067660 0x7f8f04069b20 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f8f18007fd0 tx=0x7f8f18009040 comp rx=0 tx=0).stop 2024-03-24T00:35:10.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.270+0000 7f8f2e200640 1 -- 172.21.15.40:0/3161982703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f2815c1d0 msgr2=0x7f8f28154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.270+0000 7f8f2e200640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f2815c1d0 0x7f8f28154230 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f8f2000c7a0 tx=0x7f8f2000cc60 comp rx=0 tx=0).stop 2024-03-24T00:35:10.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.270+0000 7f8f2e200640 1 -- 172.21.15.40:0/3161982703 shutdown_connections 2024-03-24T00:35:10.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.270+0000 7f8f2e200640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8f04067660 0x7f8f04069b20 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.270+0000 7f8f2e200640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8f2815c1d0 0x7f8f28154230 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.270+0000 7f8f2e200640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8f2815bdf0 0x7f8f280c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.270+0000 7f8f2e200640 1 --2- 172.21.15.40:0/3161982703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f280cf330 0x7f8f280c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.270+0000 7f8f2e200640 1 -- 172.21.15.40:0/3161982703 >> 172.21.15.40:0/3161982703 conn(0x7f8f280b37f0 msgr2=0x7f8f2806cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.270+0000 7f8f2e200640 1 -- 172.21.15.40:0/3161982703 shutdown_connections 2024-03-24T00:35:10.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.270+0000 7f8f2e200640 1 -- 172.21.15.40:0/3161982703 wait complete. 2024-03-24T00:35:10.292 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:35:10.292 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:35:10.293 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:35:10.293 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:35:10.293 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:35:10.293 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:35:10.293 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:35:10.293 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:35:10.293 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:35:10.294 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:35:10.294 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:35:10.294 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:35:10.294 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:35:10.294 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:35:10.294 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:35:10.294 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:35:10.305 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:35:10.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:35:10.306 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:35:10.306 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:35:10.313 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:35:10.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:35:10.314 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:35:10.315 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:35:10.354 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:35:10.354 DEBUG:teuthology.parallel:result is None 2024-03-24T00:35:10.356 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:35:10.356 DEBUG:teuthology.parallel:result is None 2024-03-24T00:35:10.359 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:35:10.359 DEBUG:teuthology.parallel:result is None 2024-03-24T00:35:10.361 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:35:10.361 DEBUG:teuthology.parallel:result is None 2024-03-24T00:35:10.361 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:35:10.441 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:35:10.442 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:35:10.454 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:35:10.454 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:35:10.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.502+0000 7f7cc3600640 1 Processor -- start 2024-03-24T00:35:10.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc3600640 1 -- start start 2024-03-24T00:35:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc3600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cbc0b46c0 0x7f7cbc0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc3600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cbc070f60 con 0x7f7cbc0b2ff0 2024-03-24T00:35:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc3600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cbc0b3620 con 0x7f7cbc06e620 2024-03-24T00:35:10.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc3600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0b3900 con 0x7f7cbc0b46c0 2024-03-24T00:35:10.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc1600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7cbc0b2ff0 0x7f7cbc0ccfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48460/0 (socket says 172.21.15.40:48460) 2024-03-24T00:35:10.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc1600640 1 -- 172.21.15.40:0/56981519 learned_addr learned my addr 172.21.15.40:0/56981519 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cb3e00640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cbc0b46c0 0x7f7cbc0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796513595 0 0) 0x7f7cbc070f60 con 0x7f7cbc0b2ff0 2024-03-24T00:35:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ca0003530 con 0x7f7cbc0b2ff0 2024-03-24T00:35:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443747539 0 0) 0x7f7cbc0b3620 con 0x7f7cbc06e620 2024-03-24T00:35:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cbc070f60 con 0x7f7cbc06e620 2024-03-24T00:35:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 434401990 0 0) 0x7f7ca0003530 con 0x7f7cbc0b2ff0 2024-03-24T00:35:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cbc0b46c0 msgr2=0x7f7cbc0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cbc0b46c0 0x7f7cbc0b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 >> v1:172.21.15.53:6789/0 conn(0x7f7cbc06e620 legacy=0x7f7cbc06ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc0b3be0 con 0x7f7cbc0b2ff0 2024-03-24T00:35:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f7cb4001a00 con 0x7f7cbc0b2ff0 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cb3e00640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cbc0b46c0 0x7f7cbc0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 >> v1:172.21.15.40:6789/0 conn(0x7f7cbc0b2ff0 legacy=0x7f7cbc0ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca0003b80 0x7f7ca0003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ca0004810 0x7f7ca0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cbc0b46c0 0x7f7ca00051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0b3900 con 0x7f7ca0003b80 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca000bb10 con 0x7f7ca0004810 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca000df20 con 0x7f7cbc0b46c0 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc0c00640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca0003b80 0x7f7ca0003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cb3e00640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ca0004810 0x7f7ca0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc1600640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cbc0b46c0 0x7f7ca00051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc0c00640 1 -- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cbc0b46c0 msgr2=0x7f7ca00051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc0c00640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cbc0b46c0 0x7f7ca00051f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc0c00640 1 -- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ca0004810 msgr2=0x7f7ca0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:10.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc0c00640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ca0004810 0x7f7ca0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc0c00640 1 -- 172.21.15.40:0/56981519 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca0003530 con 0x7f7ca0003b80 2024-03-24T00:35:10.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.506+0000 7f7cc1600640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cbc0b46c0 0x7f7ca00051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:10.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.510+0000 7f7cc0c00640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca0003b80 0x7f7ca0003f80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f7cac0018d0 tx=0x7f7cac01c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2226877196ae1e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:10.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.510+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac001520 con 0x7f7ca0003b80 2024-03-24T00:35:10.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.510+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cac0052f0 con 0x7f7ca0003b80 2024-03-24T00:35:10.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.510+0000 7f7cbbe00640 1 -- 172.21.15.40:0/56981519 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac003840 con 0x7f7ca0003b80 2024-03-24T00:35:10.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.510+0000 7f7cc3600640 1 -- 172.21.15.40:0/56981519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca0003b80 msgr2=0x7f7ca0003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.510+0000 7f7cc3600640 1 --2- 172.21.15.40:0/56981519 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ca0003b80 0x7f7ca0003f80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f7cac0018d0 tx=0x7f7cac01c580 comp rx=0 tx=0).stop 2024-03-24T00:35:10.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.510+0000 7f7cc1600640 1 -- 172.21.15.40:0/56981519 reap_dead start 2024-03-24T00:35:10.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.510+0000 7f7cc3600640 1 -- 172.21.15.40:0/56981519 shutdown_connections 2024-03-24T00:35:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.510+0000 7f7cc3600640 1 -- 172.21.15.40:0/56981519 >> 172.21.15.40:0/56981519 conn(0x7f7cbc0b17b0 msgr2=0x7f7cbc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.510+0000 7f7cc3600640 1 -- 172.21.15.40:0/56981519 shutdown_connections 2024-03-24T00:35:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.510+0000 7f7cc3600640 1 -- 172.21.15.40:0/56981519 wait complete. 2024-03-24T00:35:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc3600640 1 Processor -- start 2024-03-24T00:35:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc3600640 1 -- start start 2024-03-24T00:35:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc3600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7ca0003b80 0x7f7cbc13d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc3600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cbc160670 0x7f7cbc1f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc3600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cbc0ce860 0x7f7cbc0cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc3600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc144920 con 0x7f7cbc0ce860 2024-03-24T00:35:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc3600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0cdac0 con 0x7f7cbc160670 2024-03-24T00:35:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc3600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc159b60 con 0x7f7ca0003b80 2024-03-24T00:35:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cbc0ce860 0x7f7cbc0cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cbc0ce860 0x7f7cbc0cece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46448/0 (socket says 172.21.15.40:46448) 2024-03-24T00:35:10.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc1600640 1 -- 172.21.15.40:0/3296827517 learned_addr learned my addr 172.21.15.40:0/3296827517 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:10.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc0c00640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7ca0003b80 0x7f7cbc13d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cb3e00640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cbc160670 0x7f7cbc1f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc1600640 1 -- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7ca0003b80 msgr2=0x7f7cbc13d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc1600640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7ca0003b80 0x7f7cbc13d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc1600640 1 -- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cbc160670 msgr2=0x7f7cbc1f2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:10.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc1600640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cbc160670 0x7f7cbc1f2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:10.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc1600640 1 -- 172.21.15.40:0/3296827517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc13db50 con 0x7f7cbc0ce860 2024-03-24T00:35:10.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc1600640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cbc0ce860 0x7f7cbc0cece0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4001340 tx=0x7f7cb4007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:10.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cb9600640 1 -- 172.21.15.40:0/3296827517 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb400ea80 con 0x7f7cbc0ce860 2024-03-24T00:35:10.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cb9600640 1 -- 172.21.15.40:0/3296827517 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb4018070 con 0x7f7cbc0ce860 2024-03-24T00:35:10.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc3600640 1 -- 172.21.15.40:0/3296827517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cbc13de30 con 0x7f7cbc0ce860 2024-03-24T00:35:10.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cb9600640 1 -- 172.21.15.40:0/3296827517 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb400eea0 con 0x7f7cbc0ce860 2024-03-24T00:35:10.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.514+0000 7f7cc3600640 1 -- 172.21.15.40:0/3296827517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cbc13d860 con 0x7f7cbc0ce860 2024-03-24T00:35:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.518+0000 7f7cc3600640 1 -- 172.21.15.40:0/3296827517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cbc068170 con 0x7f7cbc0ce860 2024-03-24T00:35:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.518+0000 7f7cb9600640 1 -- 172.21.15.40:0/3296827517 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7cb401d460 con 0x7f7cbc0ce860 2024-03-24T00:35:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.518+0000 7f7cb9600640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c98067690 0x7f7c98069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.518+0000 7f7cb9600640 1 -- 172.21.15.40:0/3296827517 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f7cb4085720 con 0x7f7cbc0ce860 2024-03-24T00:35:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.518+0000 7f7cc0c00640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c98067690 0x7f7c98069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:10.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.518+0000 7f7cc0c00640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c98067690 0x7f7c98069b50 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7cac0ac000 tx=0x7f7cac0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:10.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.522+0000 7f7cb9600640 1 -- 172.21.15.40:0/3296827517 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7cb4055f90 con 0x7f7cbc0ce860 2024-03-24T00:35:10.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:10.694+0000 7f7cc3600640 1 -- 172.21.15.40:0/3296827517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7cbc072f50 con 0x7f7cbc0ce860 2024-03-24T00:35:11.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.262+0000 7f7cb9600640 1 -- 172.21.15.40:0/3296827517 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7cb405a030 con 0x7f7cbc0ce860 2024-03-24T00:35:11.268 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-03-24T00:35:11.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.266+0000 7f7cc3600640 1 -- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c98067690 msgr2=0x7f7c98069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:11.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.266+0000 7f7cc3600640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c98067690 0x7f7c98069b50 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f7cac0ac000 tx=0x7f7cac0a6000 comp rx=0 tx=0).stop 2024-03-24T00:35:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.266+0000 7f7cc3600640 1 -- 172.21.15.40:0/3296827517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cbc0ce860 msgr2=0x7f7cbc0cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.266+0000 7f7cc3600640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cbc0ce860 0x7f7cbc0cece0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4001340 tx=0x7f7cb4007bc0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.266+0000 7f7cc3600640 1 -- 172.21.15.40:0/3296827517 shutdown_connections 2024-03-24T00:35:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.266+0000 7f7cc3600640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c98067690 0x7f7c98069b50 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.266+0000 7f7cc3600640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cbc0ce860 0x7f7cbc0cece0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.266+0000 7f7cc3600640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cbc160670 0x7f7cbc1f2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.266+0000 7f7cc3600640 1 --2- 172.21.15.40:0/3296827517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7ca0003b80 0x7f7cbc13d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.266+0000 7f7cc3600640 1 -- 172.21.15.40:0/3296827517 >> 172.21.15.40:0/3296827517 conn(0x7f7cbc0b17b0 msgr2=0x7f7cbc06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.266+0000 7f7cc3600640 1 -- 172.21.15.40:0/3296827517 shutdown_connections 2024-03-24T00:35:11.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.266+0000 7f7cc3600640 1 -- 172.21.15.40:0/3296827517 wait complete. 2024-03-24T00:35:11.296 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:35:11.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.434+0000 7f62cf200640 1 Processor -- start 2024-03-24T00:35:11.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.434+0000 7f62cf200640 1 -- start start 2024-03-24T00:35:11.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cf200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f62c815dfc0 0x7f62c80cd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cf200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62c815e3a0 0x7f62c80cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cf200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f62c80d0b40 0x7f62c80ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cf200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c80cf870 con 0x7f62c80d0b40 2024-03-24T00:35:11.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cf200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c80cfaa0 con 0x7f62c815dfc0 2024-03-24T00:35:11.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cf200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c80cfd80 con 0x7f62c815e3a0 2024-03-24T00:35:11.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cd200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f62c80d0b40 0x7f62c80ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:11.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cd200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f62c80d0b40 0x7f62c80ce690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46456/0 (socket says 172.21.15.40:46456) 2024-03-24T00:35:11.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cd200640 1 -- 172.21.15.40:0/902838445 learned_addr learned my addr 172.21.15.40:0/902838445 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:11.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62c7e00640 1 --2- 172.21.15.40:0/902838445 >> v2:172.21.15.53:3300/0 conn(0x7f62c815dfc0 0x7f62c80cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:11.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cd200640 1 -- 172.21.15.40:0/902838445 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62c815e3a0 msgr2=0x7f62c80cd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:11.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cd200640 1 --2- 172.21.15.40:0/902838445 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62c815e3a0 0x7f62c80cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cd200640 1 -- 172.21.15.40:0/902838445 >> v2:172.21.15.53:3300/0 conn(0x7f62c815dfc0 msgr2=0x7f62c80cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:11.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cd200640 1 --2- 172.21.15.40:0/902838445 >> v2:172.21.15.53:3300/0 conn(0x7f62c815dfc0 0x7f62c80cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62cd200640 1 -- 172.21.15.40:0/902838445 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c80d0060 con 0x7f62c80d0b40 2024-03-24T00:35:11.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.438+0000 7f62c7e00640 1 --2- 172.21.15.40:0/902838445 >> v2:172.21.15.53:3300/0 conn(0x7f62c815dfc0 0x7f62c80cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:35:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cd200640 1 --2- 172.21.15.40:0/902838445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80d0b40 0x7f62c80ce690 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f62c80720f0 tx=0x7f62c001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ca77143377fa82 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62c6a00640 1 -- 172.21.15.40:0/902838445 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c00a5550 con 0x7f62c80d0b40 2024-03-24T00:35:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62c6a00640 1 -- 172.21.15.40:0/902838445 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c00a56f0 con 0x7f62c80d0b40 2024-03-24T00:35:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62c6a00640 1 -- 172.21.15.40:0/902838445 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c00a5ac0 con 0x7f62c80d0b40 2024-03-24T00:35:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 -- 172.21.15.40:0/902838445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80d0b40 msgr2=0x7f62c80ce690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 --2- 172.21.15.40:0/902838445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80d0b40 0x7f62c80ce690 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f62c80720f0 tx=0x7f62c001c8d0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 -- 172.21.15.40:0/902838445 shutdown_connections 2024-03-24T00:35:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 --2- 172.21.15.40:0/902838445 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80d0b40 0x7f62c80ce690 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 --2- 172.21.15.40:0/902838445 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62c815e3a0 0x7f62c80cd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 --2- 172.21.15.40:0/902838445 >> v2:172.21.15.53:3300/0 conn(0x7f62c815dfc0 0x7f62c80cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 -- 172.21.15.40:0/902838445 >> 172.21.15.40:0/902838445 conn(0x7f62c80b37f0 msgr2=0x7f62c80b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 -- 172.21.15.40:0/902838445 shutdown_connections 2024-03-24T00:35:11.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 -- 172.21.15.40:0/902838445 wait complete. 2024-03-24T00:35:11.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 Processor -- start 2024-03-24T00:35:11.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 -- start start 2024-03-24T00:35:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80d0b40 0x7f62c814ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62c815dfc0 0x7f62c80b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62c815e3a0 0x7f62c814d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c8162860 con 0x7f62c80d0b40 2024-03-24T00:35:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c81626e0 con 0x7f62c815e3a0 2024-03-24T00:35:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cf200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c81629e0 con 0x7f62c815dfc0 2024-03-24T00:35:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62c7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80d0b40 0x7f62c814ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62c7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80d0b40 0x7f62c814ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46466/0 (socket says 172.21.15.40:46466) 2024-03-24T00:35:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62c7e00640 1 -- 172.21.15.40:0/3679514301 learned_addr learned my addr 172.21.15.40:0/3679514301 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62c7400640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62c815dfc0 0x7f62c80b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.442+0000 7f62cd200640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62c815e3a0 0x7f62c814d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62c7e00640 1 -- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62c815dfc0 msgr2=0x7f62c80b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62c7e00640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62c815dfc0 0x7f62c80b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62c7e00640 1 -- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62c815e3a0 msgr2=0x7f62c814d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62c7e00640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62c815e3a0 0x7f62c814d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62c7e00640 1 -- 172.21.15.40:0/3679514301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c8152240 con 0x7f62c80d0b40 2024-03-24T00:35:11.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62c7e00640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80d0b40 0x7f62c814ca60 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f62b800c7f0 tx=0x7f62b800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62bfe00640 1 -- 172.21.15.40:0/3679514301 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8015070 con 0x7f62c80d0b40 2024-03-24T00:35:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62bfe00640 1 -- 172.21.15.40:0/3679514301 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b8007820 con 0x7f62c80d0b40 2024-03-24T00:35:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62bfe00640 1 -- 172.21.15.40:0/3679514301 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8007c10 con 0x7f62c80d0b40 2024-03-24T00:35:11.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62cf200640 1 -- 172.21.15.40:0/3679514301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c8151f60 con 0x7f62c80d0b40 2024-03-24T00:35:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62cf200640 1 -- 172.21.15.40:0/3679514301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c80af580 con 0x7f62c80d0b40 2024-03-24T00:35:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62cf200640 1 -- 172.21.15.40:0/3679514301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c8068170 con 0x7f62c80d0b40 2024-03-24T00:35:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62bfe00640 1 -- 172.21.15.40:0/3679514301 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f62b8018070 con 0x7f62c80d0b40 2024-03-24T00:35:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62bfe00640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f62a8067660 0x7f62a8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.446+0000 7f62bfe00640 1 -- 172.21.15.40:0/3679514301 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f62b8085690 con 0x7f62c80d0b40 2024-03-24T00:35:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.450+0000 7f62c7400640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f62a8067660 0x7f62a8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.450+0000 7f62c7400640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f62a8067660 0x7f62a8069b20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f62c80b0770 tx=0x7f62b4009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:11.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.450+0000 7f62bfe00640 1 -- 172.21.15.40:0/3679514301 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f62c8152240 con 0x7f62c80d0b40 2024-03-24T00:35:11.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 -- 172.21.15.40:0/3679514301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f62c80ce990 con 0x7f62c80d0b40 2024-03-24T00:35:11.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62bfe00640 1 -- 172.21.15.40:0/3679514301 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f62b8055f70 con 0x7f62c80d0b40 2024-03-24T00:35:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:11.673 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:35:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-03-24T00:35:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lCkU6EhAA0q21F90epsGPK5x17s9NLw==", 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llkfOHhAAqKKFyEJzG0UxbLJMFPET8A==", 2024-03-24T00:35:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lKMmwGRAAAKaOSnhTRbHIpPKXXkAe+Q==", 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:35:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l7quKJhAA8QsquiFFmj5H1G5yck0FpA==", 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAVdP9lVk3uBxAA9TC1OpeB5GIFlfJl5+ILBg==", 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYdP9lUc1IDhAAPwCV9eqL7bFibkfMFt9wPw==", 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAbdP9lwjyuFBAAJ1JAIAnLXAn03+lQPwwY9Q==", 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-03-24T00:35:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAedP9lZ1bgGRAAuLAay48N7jXjj8/yaYwSqg==", 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAidP9lvyk4JhAA8w4/XBYHDLoqJ1zop9sUiw==", 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAldP9lcb9qLBAAXts6VU/4kVqUWS/WORV7zQ==", 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAodP9l8ipTMhAAjpagojh+woHyGvlQlIkHDw==", 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:35:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:35:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-03-24T00:35:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQArdP9lmGfpORAAPvow5nF/O7RLmuXEqNWENg==", 2024-03-24T00:35:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:35:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:35:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw==", 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow r", 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lpIZrNBAAzpGWZuQ6ndjNMAOmAsebvQ==", 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l6gXMMBAAd6Oe6S73hj/WfLXTsYiEZw==", 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llFHCOhAATtv8gcL6BHPnxB1dHrg6vw==", 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAQdP9lmmd7KxAA4uk6aUKP6egYj6O1NgdadQ==", 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lebNrBxAAvbcCAJhDQn/5Uk1dJ01oHw==", 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-03-24T00:35:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lEnZyAxAAW5Lrm7LZPTVfLCOY6As1PQ==", 2024-03-24T00:35:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:35:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:35:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:35:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:35:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-24T00:35:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:35:11.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 -- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f62a8067660 msgr2=0x7f62a8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:11.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f62a8067660 0x7f62a8069b20 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f62c80b0770 tx=0x7f62b4009110 comp rx=0 tx=0).stop 2024-03-24T00:35:11.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 -- 172.21.15.40:0/3679514301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80d0b40 msgr2=0x7f62c814ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80d0b40 0x7f62c814ca60 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f62b800c7f0 tx=0x7f62b800ccb0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 -- 172.21.15.40:0/3679514301 shutdown_connections 2024-03-24T00:35:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f62a8067660 0x7f62a8069b20 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62c815e3a0 0x7f62c814d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62c815dfc0 0x7f62c80b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 --2- 172.21.15.40:0/3679514301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f62c80d0b40 0x7f62c814ca60 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 -- 172.21.15.40:0/3679514301 >> 172.21.15.40:0/3679514301 conn(0x7f62c80b37f0 msgr2=0x7f62c815d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 -- 172.21.15.40:0/3679514301 shutdown_connections 2024-03-24T00:35:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.670+0000 7f62cf200640 1 -- 172.21.15.40:0/3679514301 wait complete. 2024-03-24T00:35:11.694 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:11.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.830+0000 7f53e4000640 1 Processor -- start 2024-03-24T00:35:11.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.830+0000 7f53e4000640 1 -- start start 2024-03-24T00:35:11.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e4000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f53dc0b46c0 0x7f53dc0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e4000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc0b2ff0 0x7f53dc0ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e4000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53dc070f60 con 0x7f53dc06e620 2024-03-24T00:35:11.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e4000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53dc0b3620 con 0x7f53dc0b46c0 2024-03-24T00:35:11.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e4000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc0b3900 con 0x7f53dc0b2ff0 2024-03-24T00:35:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e2000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc0b2ff0 0x7f53dc0ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e0c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f53dc0b46c0 0x7f53dc0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e2000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc0b2ff0 0x7f53dc0ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:58266/0 (socket says 172.21.15.40:58266) 2024-03-24T00:35:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e2000640 1 -- 172.21.15.40:0/2333163248 learned_addr learned my addr 172.21.15.40:0/2333163248 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e2000640 1 -- 172.21.15.40:0/2333163248 >> v2:172.21.15.53:3300/0 conn(0x7f53dc0b46c0 msgr2=0x7f53dc0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e2000640 1 --2- 172.21.15.40:0/2333163248 >> v2:172.21.15.53:3300/0 conn(0x7f53dc0b46c0 0x7f53dc0b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e2000640 1 -- 172.21.15.40:0/2333163248 >> v1:172.21.15.53:6789/0 conn(0x7f53dc06e620 legacy=0x7f53dc06ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e2000640 1 -- 172.21.15.40:0/2333163248 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc0b3be0 con 0x7f53dc0b2ff0 2024-03-24T00:35:11.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.834+0000 7f53e0c00640 1 --2- 172.21.15.40:0/2333163248 >> v2:172.21.15.53:3300/0 conn(0x7f53dc0b46c0 0x7f53dc0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:11.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53e2000640 1 --2- 172.21.15.40:0/2333163248 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc0b2ff0 0x7f53dc0ccfa0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f53d4009e50 tx=0x7f53d401c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78aabaa8bee033df server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:11.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53dbe00640 1 -- 172.21.15.40:0/2333163248 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53dc0cdcf0 con 0x7f53dc0b2ff0 2024-03-24T00:35:11.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53dbe00640 1 -- 172.21.15.40:0/2333163248 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d40a5750 con 0x7f53dc0b2ff0 2024-03-24T00:35:11.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53dbe00640 1 -- 172.21.15.40:0/2333163248 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d40a5b20 con 0x7f53dc0b2ff0 2024-03-24T00:35:11.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53e4000640 1 -- 172.21.15.40:0/2333163248 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc0b2ff0 msgr2=0x7f53dc0ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:11.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53e4000640 1 --2- 172.21.15.40:0/2333163248 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc0b2ff0 0x7f53dc0ccfa0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f53d4009e50 tx=0x7f53d401c9a0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53e4000640 1 -- 172.21.15.40:0/2333163248 shutdown_connections 2024-03-24T00:35:11.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53e4000640 1 --2- 172.21.15.40:0/2333163248 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc0b2ff0 0x7f53dc0ccfa0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53e4000640 1 --2- 172.21.15.40:0/2333163248 >> v2:172.21.15.53:3300/0 conn(0x7f53dc0b46c0 0x7f53dc0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53e4000640 1 -- 172.21.15.40:0/2333163248 >> 172.21.15.40:0/2333163248 conn(0x7f53dc0b17b0 msgr2=0x7f53dc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:11.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53e4000640 1 -- 172.21.15.40:0/2333163248 shutdown_connections 2024-03-24T00:35:11.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53e4000640 1 -- 172.21.15.40:0/2333163248 wait complete. 2024-03-24T00:35:11.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53e4000640 1 Processor -- start 2024-03-24T00:35:11.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53e4000640 1 -- start start 2024-03-24T00:35:11.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.838+0000 7f53e4000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc06e620 0x7f53dc0cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e4000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc0b2ff0 0x7f53dc13c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e4000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53dc0b46c0 0x7f53dc13caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e4000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc144920 con 0x7f53dc0b46c0 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e4000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc165c70 con 0x7f53dc0b2ff0 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e4000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53dc168d10 con 0x7f53dc06e620 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53dc0b46c0 0x7f53dc13caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53dc0b46c0 0x7f53dc13caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46478/0 (socket says 172.21.15.40:46478) 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e2000640 1 -- 172.21.15.40:0/923365840 learned_addr learned my addr 172.21.15.40:0/923365840 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e0c00640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc0b2ff0 0x7f53dc13c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e1600640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc06e620 0x7f53dc0cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e2000640 1 -- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc06e620 msgr2=0x7f53dc0cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e2000640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc06e620 0x7f53dc0cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e2000640 1 -- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc0b2ff0 msgr2=0x7f53dc13c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:11.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e2000640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc0b2ff0 0x7f53dc13c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:11.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e2000640 1 -- 172.21.15.40:0/923365840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53dc167f90 con 0x7f53dc0b46c0 2024-03-24T00:35:11.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e1600640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc06e620 0x7f53dc0cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:11.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e2000640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53dc0b46c0 0x7f53dc13caf0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f53d40af040 tx=0x7f53d4019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:11.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53d9600640 1 -- 172.21.15.40:0/923365840 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d40bc020 con 0x7f53dc0b46c0 2024-03-24T00:35:11.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53d9600640 1 -- 172.21.15.40:0/923365840 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d40b8670 con 0x7f53dc0b46c0 2024-03-24T00:35:11.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e4000640 1 -- 172.21.15.40:0/923365840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53dc168230 con 0x7f53dc0b46c0 2024-03-24T00:35:11.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53d9600640 1 -- 172.21.15.40:0/923365840 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d40b8a40 con 0x7f53dc0b46c0 2024-03-24T00:35:11.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e4000640 1 -- 172.21.15.40:0/923365840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53dc1687f0 con 0x7f53dc0b46c0 2024-03-24T00:35:11.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53e4000640 1 -- 172.21.15.40:0/923365840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53dc068170 con 0x7f53dc0b46c0 2024-03-24T00:35:11.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.842+0000 7f53d9600640 1 -- 172.21.15.40:0/923365840 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f53d40bf020 con 0x7f53dc0b46c0 2024-03-24T00:35:11.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.846+0000 7f53d9600640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53bc067690 0x7f53bc069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:11.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.846+0000 7f53d9600640 1 -- 172.21.15.40:0/923365840 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f53d41200c0 con 0x7f53dc0b46c0 2024-03-24T00:35:11.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.846+0000 7f53e1600640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53bc067690 0x7f53bc069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:11.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.846+0000 7f53e1600640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53bc067690 0x7f53bc069b50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f53d0001710 tx=0x7f53d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:11.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:11.850+0000 7f53d9600640 1 -- 172.21.15.40:0/923365840 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f53d4016620 con 0x7f53dc0b46c0 2024-03-24T00:35:12.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.010+0000 7f53e4000640 1 -- 172.21.15.40:0/923365840 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53dc072f50 con 0x7f53dc0b46c0 2024-03-24T00:35:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53d9600640 1 -- 172.21.15.40:0/923365840 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+12831 (secure 0 0 0) 0x7f53d40f1110 con 0x7f53dc0b46c0 2024-03-24T00:35:12.016 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:12.016 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":66,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:35:11.275293+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:12.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53e4000640 1 -- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53bc067690 msgr2=0x7f53bc069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53e4000640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53bc067690 0x7f53bc069b50 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f53d0001710 tx=0x7f53d0009040 comp rx=0 tx=0).stop 2024-03-24T00:35:12.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53e4000640 1 -- 172.21.15.40:0/923365840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53dc0b46c0 msgr2=0x7f53dc13caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53e4000640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53dc0b46c0 0x7f53dc13caf0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f53d40af040 tx=0x7f53d4019740 comp rx=0 tx=0).stop 2024-03-24T00:35:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53e4000640 1 -- 172.21.15.40:0/923365840 shutdown_connections 2024-03-24T00:35:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53e4000640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53bc067690 0x7f53bc069b50 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53e4000640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53dc0b46c0 0x7f53dc13caf0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53e4000640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53dc0b2ff0 0x7f53dc13c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53e4000640 1 --2- 172.21.15.40:0/923365840 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53dc06e620 0x7f53dc0cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53e4000640 1 -- 172.21.15.40:0/923365840 >> 172.21.15.40:0/923365840 conn(0x7f53dc0b17b0 msgr2=0x7f53dc15ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53e4000640 1 -- 172.21.15.40:0/923365840 shutdown_connections 2024-03-24T00:35:12.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.014+0000 7f53e4000640 1 -- 172.21.15.40:0/923365840 wait complete. 2024-03-24T00:35:12.037 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:35:12.038 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:35:12.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.178+0000 7ffae8400640 1 Processor -- start 2024-03-24T00:35:12.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.178+0000 7ffae8400640 1 -- start start 2024-03-24T00:35:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.178+0000 7ffae8400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffae015e350 0x7ffae00cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.178+0000 7ffae8400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffae00c9580 0x7ffae00c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.178+0000 7ffae8400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffae00ad1e0 0x7ffae00ad630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.178+0000 7ffae8400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae00cd660 con 0x7ffae015e350 2024-03-24T00:35:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.178+0000 7ffae8400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae00c9f70 con 0x7ffae00ad1e0 2024-03-24T00:35:12.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.178+0000 7ffae8400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae00734e0 con 0x7ffae00c9580 2024-03-24T00:35:12.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.182+0000 7ffae5a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffae015e350 0x7ffae00cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.182+0000 7ffae5a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ffae015e350 0x7ffae00cfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46492/0 (socket says 172.21.15.40:46492) 2024-03-24T00:35:12.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.182+0000 7ffae5a00640 1 -- 172.21.15.40:0/2868085769 learned_addr learned my addr 172.21.15.40:0/2868085769 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:12.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.182+0000 7ffae5000640 1 --2- 172.21.15.40:0/2868085769 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffae00c9580 0x7ffae00c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.182+0000 7ffae6400640 1 --2- 172.21.15.40:0/2868085769 >> v2:172.21.15.53:3300/0 conn(0x7ffae00ad1e0 0x7ffae00ad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.182+0000 7ffae5a00640 1 -- 172.21.15.40:0/2868085769 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffae00c9580 msgr2=0x7ffae00c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:12.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.182+0000 7ffae5a00640 1 --2- 172.21.15.40:0/2868085769 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffae00c9580 0x7ffae00c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.182+0000 7ffae5a00640 1 -- 172.21.15.40:0/2868085769 >> v2:172.21.15.53:3300/0 conn(0x7ffae00ad1e0 msgr2=0x7ffae00ad630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:12.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.182+0000 7ffae5a00640 1 --2- 172.21.15.40:0/2868085769 >> v2:172.21.15.53:3300/0 conn(0x7ffae00ad1e0 0x7ffae00ad630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.182+0000 7ffae5a00640 1 -- 172.21.15.40:0/2868085769 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae00d0030 con 0x7ffae015e350 2024-03-24T00:35:12.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.182+0000 7ffae5a00640 1 --2- 172.21.15.40:0/2868085769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae015e350 0x7ffae00cfaf0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ffae0071410 tx=0x7ffad401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6b5c65db56fd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:12.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffadfe00640 1 -- 172.21.15.40:0/2868085769 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad40ab5f0 con 0x7ffae015e350 2024-03-24T00:35:12.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffadfe00640 1 -- 172.21.15.40:0/2868085769 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad40b0070 con 0x7ffae015e350 2024-03-24T00:35:12.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffadfe00640 1 -- 172.21.15.40:0/2868085769 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad40ab9b0 con 0x7ffae015e350 2024-03-24T00:35:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 -- 172.21.15.40:0/2868085769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae015e350 msgr2=0x7ffae00cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 --2- 172.21.15.40:0/2868085769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae015e350 0x7ffae00cfaf0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ffae0071410 tx=0x7ffad401c520 comp rx=0 tx=0).stop 2024-03-24T00:35:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 -- 172.21.15.40:0/2868085769 shutdown_connections 2024-03-24T00:35:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 --2- 172.21.15.40:0/2868085769 >> v2:172.21.15.53:3300/0 conn(0x7ffae00ad1e0 0x7ffae00ad630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 --2- 172.21.15.40:0/2868085769 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffae00c9580 0x7ffae00c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 --2- 172.21.15.40:0/2868085769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae015e350 0x7ffae00cfaf0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 -- 172.21.15.40:0/2868085769 >> 172.21.15.40:0/2868085769 conn(0x7ffae0072870 msgr2=0x7ffae0072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 -- 172.21.15.40:0/2868085769 shutdown_connections 2024-03-24T00:35:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 -- 172.21.15.40:0/2868085769 wait complete. 2024-03-24T00:35:12.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 Processor -- start 2024-03-24T00:35:12.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 -- start start 2024-03-24T00:35:12.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffae00ad1e0 0x7ffae01537d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae00c9580 0x7ffae00cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffae015e350 0x7ffae00cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae00cda30 con 0x7ffae00c9580 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae00716b0 con 0x7ffae015e350 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae8400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae00cdbb0 con 0x7ffae00ad1e0 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae00c9580 0x7ffae00cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae00c9580 0x7ffae00cb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46498/0 (socket says 172.21.15.40:46498) 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae5000640 1 -- 172.21.15.40:0/3175160846 learned_addr learned my addr 172.21.15.40:0/3175160846 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae5a00640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffae00ad1e0 0x7ffae01537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.186+0000 7ffae6400640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffae015e350 0x7ffae00cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffae5000640 1 -- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffae00ad1e0 msgr2=0x7ffae01537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffae5000640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffae00ad1e0 0x7ffae01537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffae5000640 1 -- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffae015e350 msgr2=0x7ffae00cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffae5000640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffae015e350 0x7ffae00cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffae5000640 1 -- 172.21.15.40:0/3175160846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae0153d10 con 0x7ffae00c9580 2024-03-24T00:35:12.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffae5000640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae00c9580 0x7ffae00cb900 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00d800 tx=0x7ffacc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffadd600640 1 -- 172.21.15.40:0/3175160846 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc00b6c0 con 0x7ffae00c9580 2024-03-24T00:35:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffadd600640 1 -- 172.21.15.40:0/3175160846 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffacc019070 con 0x7ffae00c9580 2024-03-24T00:35:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffadd600640 1 -- 172.21.15.40:0/3175160846 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffacc00ba60 con 0x7ffae00c9580 2024-03-24T00:35:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffae8400640 1 -- 172.21.15.40:0/3175160846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae015fc50 con 0x7ffae00c9580 2024-03-24T00:35:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffae8400640 1 -- 172.21.15.40:0/3175160846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae0160210 con 0x7ffae00c9580 2024-03-24T00:35:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffae8400640 1 -- 172.21.15.40:0/3175160846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae0068170 con 0x7ffae00c9580 2024-03-24T00:35:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffadd600640 1 -- 172.21.15.40:0/3175160846 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ffacc00bc00 con 0x7ffae00c9580 2024-03-24T00:35:12.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffadd600640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffac0067660 0x7ffac0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffadd600640 1 -- 172.21.15.40:0/3175160846 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ffacc0858e0 con 0x7ffae00c9580 2024-03-24T00:35:12.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.190+0000 7ffae5a00640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffac0067660 0x7ffac0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.194+0000 7ffae5a00640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffac0067660 0x7ffac0069b20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ffad40af000 tx=0x7ffad40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:12.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.194+0000 7ffadd600640 1 -- 172.21.15.40:0/3175160846 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ffacc056250 con 0x7ffae00c9580 2024-03-24T00:35:12.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 -- 172.21.15.40:0/3175160846 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffae00cfdf0 con 0x7ffae00c9580 2024-03-24T00:35:12.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffadd600640 1 -- 172.21.15.40:0/3175160846 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v66) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffacc05a2f0 con 0x7ffae00c9580 2024-03-24T00:35:12.365 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:35:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 -- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffac0067660 msgr2=0x7ffac0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffac0067660 0x7ffac0069b20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ffad40af000 tx=0x7ffad40a7000 comp rx=0 tx=0).stop 2024-03-24T00:35:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 -- 172.21.15.40:0/3175160846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae00c9580 msgr2=0x7ffae00cb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae00c9580 0x7ffae00cb900 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7ffacc00d800 tx=0x7ffacc00dcc0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 -- 172.21.15.40:0/3175160846 shutdown_connections 2024-03-24T00:35:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffac0067660 0x7ffac0069b20 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffae015e350 0x7ffae00cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffae00c9580 0x7ffae00cb900 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 --2- 172.21.15.40:0/3175160846 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffae00ad1e0 0x7ffae01537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 -- 172.21.15.40:0/3175160846 >> 172.21.15.40:0/3175160846 conn(0x7ffae0072870 msgr2=0x7ffae00afc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:12.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 -- 172.21.15.40:0/3175160846 shutdown_connections 2024-03-24T00:35:12.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.362+0000 7ffae8400640 1 -- 172.21.15.40:0/3175160846 wait complete. 2024-03-24T00:35:12.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:35:12.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.522+0000 7ff9be600640 1 Processor -- start 2024-03-24T00:35:12.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.522+0000 7ff9be600640 1 -- start start 2024-03-24T00:35:12.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.522+0000 7ff9be600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b813ec80 0x7ff9b81414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.522+0000 7ff9be600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff9b81419f0 0x7ff9b80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9be600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b8070e00 con 0x7ff9b80ce630 2024-03-24T00:35:12.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9be600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b8141e50 con 0x7ff9b81419f0 2024-03-24T00:35:12.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9be600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b80cee20 con 0x7ff9b813ec80 2024-03-24T00:35:12.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9aea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff9b81419f0 0x7ff9b80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9aea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff9b81419f0 0x7ff9b80ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46506/0 (socket says 172.21.15.40:46506) 2024-03-24T00:35:12.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9aea00640 1 -- 172.21.15.40:0/4151428963 learned_addr learned my addr 172.21.15.40:0/4151428963 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:12.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9b7400640 1 --2- 172.21.15.40:0/4151428963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b813ec80 0x7ff9b81414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9aea00640 1 -- 172.21.15.40:0/4151428963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b813ec80 msgr2=0x7ff9b81414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:12.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9aea00640 1 --2- 172.21.15.40:0/4151428963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b813ec80 0x7ff9b81414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9aea00640 1 -- 172.21.15.40:0/4151428963 >> v1:172.21.15.53:6789/0 conn(0x7ff9b80ce630 legacy=0x7ff9b815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9aea00640 1 -- 172.21.15.40:0/4151428963 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b80cf100 con 0x7ff9b81419f0 2024-03-24T00:35:12.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9aea00640 1 --2- 172.21.15.40:0/4151428963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b81419f0 0x7ff9b80ce0f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0009870 tx=0x7ff9a001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6795e504a587334c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:12.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9b6a00640 1 -- 172.21.15.40:0/4151428963 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a00a9640 con 0x7ff9b81419f0 2024-03-24T00:35:12.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9b6a00640 1 -- 172.21.15.40:0/4151428963 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a00a97e0 con 0x7ff9b81419f0 2024-03-24T00:35:12.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.526+0000 7ff9b6a00640 1 -- 172.21.15.40:0/4151428963 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a00a9bb0 con 0x7ff9b81419f0 2024-03-24T00:35:12.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 -- 172.21.15.40:0/4151428963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b81419f0 msgr2=0x7ff9b80ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 --2- 172.21.15.40:0/4151428963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b81419f0 0x7ff9b80ce0f0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0009870 tx=0x7ff9a001c560 comp rx=0 tx=0).stop 2024-03-24T00:35:12.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 -- 172.21.15.40:0/4151428963 shutdown_connections 2024-03-24T00:35:12.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 --2- 172.21.15.40:0/4151428963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b81419f0 0x7ff9b80ce0f0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 --2- 172.21.15.40:0/4151428963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b813ec80 0x7ff9b81414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 -- 172.21.15.40:0/4151428963 >> 172.21.15.40:0/4151428963 conn(0x7ff9b80b17b0 msgr2=0x7ff9b80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:12.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 -- 172.21.15.40:0/4151428963 shutdown_connections 2024-03-24T00:35:12.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 -- 172.21.15.40:0/4151428963 wait complete. 2024-03-24T00:35:12.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 Processor -- start 2024-03-24T00:35:12.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 -- start start 2024-03-24T00:35:12.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b80ce630 0x7ff9b813df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b813ec80 0x7ff9b81407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b81419f0 0x7ff9b8155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b8168ec0 con 0x7ff9b81419f0 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b8168d40 con 0x7ff9b80ce630 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9be600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b8169040 con 0x7ff9b813ec80 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9b7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b81419f0 0x7ff9b8155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9b7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b81419f0 0x7ff9b8155130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46524/0 (socket says 172.21.15.40:46524) 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9b7e00640 1 -- 172.21.15.40:0/3390733159 learned_addr learned my addr 172.21.15.40:0/3390733159 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9b7400640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b80ce630 0x7ff9b813df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9aea00640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b813ec80 0x7ff9b81407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9b7e00640 1 -- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b813ec80 msgr2=0x7ff9b81407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9b7e00640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b813ec80 0x7ff9b81407e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9b7e00640 1 -- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b80ce630 msgr2=0x7ff9b813df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9b7e00640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b80ce630 0x7ff9b813df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.530+0000 7ff9b7e00640 1 -- 172.21.15.40:0/3390733159 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b813f640 con 0x7ff9b81419f0 2024-03-24T00:35:12.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.534+0000 7ff9b7e00640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b81419f0 0x7ff9b8155130 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000c760 tx=0x7ff9b000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:12.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.534+0000 7ff9afe00640 1 -- 172.21.15.40:0/3390733159 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b0003b80 con 0x7ff9b81419f0 2024-03-24T00:35:12.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.534+0000 7ff9afe00640 1 -- 172.21.15.40:0/3390733159 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b0003d20 con 0x7ff9b81419f0 2024-03-24T00:35:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.534+0000 7ff9afe00640 1 -- 172.21.15.40:0/3390733159 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b000aa30 con 0x7ff9b81419f0 2024-03-24T00:35:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.534+0000 7ff9be600640 1 -- 172.21.15.40:0/3390733159 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b813f920 con 0x7ff9b81419f0 2024-03-24T00:35:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.534+0000 7ff9be600640 1 -- 172.21.15.40:0/3390733159 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b813fee0 con 0x7ff9b81419f0 2024-03-24T00:35:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.534+0000 7ff9be600640 1 -- 172.21.15.40:0/3390733159 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b8068170 con 0x7ff9b81419f0 2024-03-24T00:35:12.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.534+0000 7ff9afe00640 1 -- 172.21.15.40:0/3390733159 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff9b000abd0 con 0x7ff9b81419f0 2024-03-24T00:35:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.534+0000 7ff9afe00640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff994067660 0x7ff994069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.534+0000 7ff9afe00640 1 -- 172.21.15.40:0/3390733159 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff9b0085350 con 0x7ff9b81419f0 2024-03-24T00:35:12.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.538+0000 7ff9b7400640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff994067660 0x7ff994069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.538+0000 7ff9b7400640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff994067660 0x7ff994069b20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8006aa0 tx=0x7ff9a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:12.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.538+0000 7ff9afe00640 1 -- 172.21.15.40:0/3390733159 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff9b0055c30 con 0x7ff9b81419f0 2024-03-24T00:35:12.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.702+0000 7ff9be600640 1 -- 172.21.15.40:0/3390733159 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff9b8072b90 con 0x7ff9b81419f0 2024-03-24T00:35:12.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.702+0000 7ff9afe00640 1 -- 172.21.15.40:0/3390733159 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v66) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff9b0059cd0 con 0x7ff9b81419f0 2024-03-24T00:35:12.708 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:12.708 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-03-24T00:35:12.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:35:12.708 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-03-24T00:35:12.708 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-03-24T00:35:12.708 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:12.708 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-03-24T00:35:12.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.706+0000 7ff9be600640 1 -- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff994067660 msgr2=0x7ff994069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.706+0000 7ff9be600640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff994067660 0x7ff994069b20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8006aa0 tx=0x7ff9a8008040 comp rx=0 tx=0).stop 2024-03-24T00:35:12.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.706+0000 7ff9be600640 1 -- 172.21.15.40:0/3390733159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b81419f0 msgr2=0x7ff9b8155130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.706+0000 7ff9be600640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b81419f0 0x7ff9b8155130 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff9b000c760 tx=0x7ff9b000cc20 comp rx=0 tx=0).stop 2024-03-24T00:35:12.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.706+0000 7ff9be600640 1 -- 172.21.15.40:0/3390733159 shutdown_connections 2024-03-24T00:35:12.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.706+0000 7ff9be600640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff994067660 0x7ff994069b20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.706+0000 7ff9be600640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b81419f0 0x7ff9b8155130 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.706+0000 7ff9be600640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b813ec80 0x7ff9b81407e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.706+0000 7ff9be600640 1 --2- 172.21.15.40:0/3390733159 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b80ce630 0x7ff9b813df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.706+0000 7ff9be600640 1 -- 172.21.15.40:0/3390733159 >> 172.21.15.40:0/3390733159 conn(0x7ff9b80b17b0 msgr2=0x7ff9b806e520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:12.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.706+0000 7ff9be600640 1 -- 172.21.15.40:0/3390733159 shutdown_connections 2024-03-24T00:35:12.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.706+0000 7ff9be600640 1 -- 172.21.15.40:0/3390733159 wait complete. 2024-03-24T00:35:12.728 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:35:12.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:35:12.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f3572800640 1 Processor -- start 2024-03-24T00:35:12.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f3572800640 1 -- start start 2024-03-24T00:35:12.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f3572800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f356c15ac60 0x7f356c0b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f3572800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f356c06cb80 0x7f356c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f3572800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0b3260 con 0x7f356c0b4840 2024-03-24T00:35:12.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f3572800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c0b3e70 con 0x7f356c15ac60 2024-03-24T00:35:12.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f3572800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c0b3540 con 0x7f356c06cb80 2024-03-24T00:35:12.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f356b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f356c15ac60 0x7f356c0b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f356be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f356c0b4840 0x7f356c0ccd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48476/0 (socket says 172.21.15.40:48476) 2024-03-24T00:35:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f356be00640 1 -- 172.21.15.40:0/2193468779 learned_addr learned my addr 172.21.15.40:0/2193468779 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f3563e00640 1 --2- 172.21.15.40:0/2193468779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f356c06cb80 0x7f356c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f356b400640 1 -- 172.21.15.40:0/2193468779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f356c06cb80 msgr2=0x7f356c0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f356b400640 1 --2- 172.21.15.40:0/2193468779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f356c06cb80 0x7f356c0b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f356b400640 1 -- 172.21.15.40:0/2193468779 >> v1:172.21.15.40:6789/0 conn(0x7f356c0b4840 legacy=0x7f356c0ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f356b400640 1 -- 172.21.15.40:0/2193468779 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0b3820 con 0x7f356c15ac60 2024-03-24T00:35:12.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.870+0000 7f356aa00640 1 -- 172.21.15.40:0/2193468779 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816977548 0 0) 0x7f356c0b3260 con 0x7f356c0b4840 2024-03-24T00:35:12.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.874+0000 7f356b400640 1 --2- 172.21.15.40:0/2193468779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f356c15ac60 0x7f356c0b2960 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f355c0097c0 tx=0x7f355c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552d49be4bd52a89 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:12.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.874+0000 7f356aa00640 1 -- 172.21.15.40:0/2193468779 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c0b0070 con 0x7f356c15ac60 2024-03-24T00:35:12.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.874+0000 7f356aa00640 1 -- 172.21.15.40:0/2193468779 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c0ab600 con 0x7f356c15ac60 2024-03-24T00:35:12.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.874+0000 7f356aa00640 1 -- 172.21.15.40:0/2193468779 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c0ab9d0 con 0x7f356c15ac60 2024-03-24T00:35:12.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.874+0000 7f3572800640 1 -- 172.21.15.40:0/2193468779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f356c15ac60 msgr2=0x7f356c0b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.874+0000 7f3572800640 1 --2- 172.21.15.40:0/2193468779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f356c15ac60 0x7f356c0b2960 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f355c0097c0 tx=0x7f355c01c3e0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.874+0000 7f3572800640 1 -- 172.21.15.40:0/2193468779 shutdown_connections 2024-03-24T00:35:12.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.874+0000 7f3572800640 1 --2- 172.21.15.40:0/2193468779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f356c06cb80 0x7f356c0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.874+0000 7f3572800640 1 --2- 172.21.15.40:0/2193468779 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f356c15ac60 0x7f356c0b2960 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.874+0000 7f3572800640 1 -- 172.21.15.40:0/2193468779 >> 172.21.15.40:0/2193468779 conn(0x7f356c0b1660 msgr2=0x7f356c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:12.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.874+0000 7f3572800640 1 -- 172.21.15.40:0/2193468779 shutdown_connections 2024-03-24T00:35:12.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.874+0000 7f3572800640 1 -- 172.21.15.40:0/2193468779 wait complete. 2024-03-24T00:35:12.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3572800640 1 Processor -- start 2024-03-24T00:35:12.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3572800640 1 -- start start 2024-03-24T00:35:12.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3572800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f356c06cb80 0x7f356c1527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3572800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f356c0b4840 0x7f356c154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3572800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f356c15ac60 0x7f356c154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3572800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c167a40 con 0x7f356c06cb80 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3572800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c1678c0 con 0x7f356c15ac60 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3572800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c167bc0 con 0x7f356c0b4840 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f356b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f356c06cb80 0x7f356c1527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f356b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f356c06cb80 0x7f356c1527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46548/0 (socket says 172.21.15.40:46548) 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f356b400640 1 -- 172.21.15.40:0/4182029426 learned_addr learned my addr 172.21.15.40:0/4182029426 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3563e00640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f356c0b4840 0x7f356c154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f356be00640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f356c15ac60 0x7f356c154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f356b400640 1 -- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f356c0b4840 msgr2=0x7f356c154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f356b400640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f356c0b4840 0x7f356c154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f356b400640 1 -- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f356c15ac60 msgr2=0x7f356c154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:12.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f356b400640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f356c15ac60 0x7f356c154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f356b400640 1 -- 172.21.15.40:0/4182029426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0ce4a0 con 0x7f356c06cb80 2024-03-24T00:35:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f356b400640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f356c06cb80 0x7f356c1527c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f355c0097c0 tx=0x7f355c01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3563400640 1 -- 172.21.15.40:0/4182029426 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c0b0040 con 0x7f356c06cb80 2024-03-24T00:35:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3563400640 1 -- 172.21.15.40:0/4182029426 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c004030 con 0x7f356c06cb80 2024-03-24T00:35:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3563400640 1 -- 172.21.15.40:0/4182029426 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c006e20 con 0x7f356c06cb80 2024-03-24T00:35:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3572800640 1 -- 172.21.15.40:0/4182029426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c0ce740 con 0x7f356c06cb80 2024-03-24T00:35:12.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.878+0000 7f3572800640 1 -- 172.21.15.40:0/4182029426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c0ced00 con 0x7f356c06cb80 2024-03-24T00:35:12.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.882+0000 7f3572800640 1 -- 172.21.15.40:0/4182029426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c068170 con 0x7f356c06cb80 2024-03-24T00:35:12.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.882+0000 7f3563400640 1 -- 172.21.15.40:0/4182029426 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f355c003620 con 0x7f356c06cb80 2024-03-24T00:35:12.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.882+0000 7f3563400640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3548067660 0x7f3548069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:12.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.882+0000 7f3563400640 1 -- 172.21.15.40:0/4182029426 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f355c11faa0 con 0x7f356c06cb80 2024-03-24T00:35:12.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.882+0000 7f3563e00640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3548067660 0x7f3548069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:12.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.882+0000 7f3563e00640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3548067660 0x7f3548069b20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f356c156870 tx=0x7f3554008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:12.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:12.886+0000 7f3563400640 1 -- 172.21.15.40:0/4182029426 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f355c121e20 con 0x7f356c06cb80 2024-03-24T00:35:13.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.050+0000 7f3572800640 1 -- 172.21.15.40:0/4182029426 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f356c069250 con 0x7f356c06cb80 2024-03-24T00:35:13.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.270+0000 7f3563400640 1 -- 172.21.15.40:0/4182029426 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v67) v1 ==== 138+0+0 (secure 0 0 0) 0x7f355c016090 con 0x7f356c06cb80 2024-03-24T00:35:13.275 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-03-24T00:35:13.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.274+0000 7f3572800640 1 -- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3548067660 msgr2=0x7f3548069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:13.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.274+0000 7f3572800640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3548067660 0x7f3548069b20 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f356c156870 tx=0x7f3554008040 comp rx=0 tx=0).stop 2024-03-24T00:35:13.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.274+0000 7f3572800640 1 -- 172.21.15.40:0/4182029426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f356c06cb80 msgr2=0x7f356c1527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:13.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.274+0000 7f3572800640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f356c06cb80 0x7f356c1527c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f355c0097c0 tx=0x7f355c01ac20 comp rx=0 tx=0).stop 2024-03-24T00:35:13.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.274+0000 7f3572800640 1 -- 172.21.15.40:0/4182029426 shutdown_connections 2024-03-24T00:35:13.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.274+0000 7f3572800640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3548067660 0x7f3548069b20 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.274+0000 7f3572800640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f356c15ac60 0x7f356c154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.274+0000 7f3572800640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f356c0b4840 0x7f356c154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.274+0000 7f3572800640 1 --2- 172.21.15.40:0/4182029426 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f356c06cb80 0x7f356c1527c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.274+0000 7f3572800640 1 -- 172.21.15.40:0/4182029426 >> 172.21.15.40:0/4182029426 conn(0x7f356c0b1660 msgr2=0x7f356c0b2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:13.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.274+0000 7f3572800640 1 -- 172.21.15.40:0/4182029426 shutdown_connections 2024-03-24T00:35:13.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.274+0000 7f3572800640 1 -- 172.21.15.40:0/4182029426 wait complete. 2024-03-24T00:35:13.302 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:35:13.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43f4e00640 1 Processor -- start 2024-03-24T00:35:13.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43f4e00640 1 -- start start 2024-03-24T00:35:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43f4e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f43f015df80 0x7f43f0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43f4e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f43f006c950 0x7f43f00d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43f4e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43f006bd20 0x7f43f00d20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43f4e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f006b890 con 0x7f43f006c950 2024-03-24T00:35:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43f4e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f006d0f0 con 0x7f43f015df80 2024-03-24T00:35:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43f4e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f006b280 con 0x7f43f006bd20 2024-03-24T00:35:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43e7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f43f006c950 0x7f43f00d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43e7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f43f006c950 0x7f43f00d1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46554/0 (socket says 172.21.15.40:46554) 2024-03-24T00:35:13.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43e7e00640 1 -- 172.21.15.40:0/127976202 learned_addr learned my addr 172.21.15.40:0/127976202 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43ede00640 1 --2- 172.21.15.40:0/127976202 >> v2:172.21.15.53:3300/0 conn(0x7f43f015df80 0x7f43f0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43ee800640 1 --2- 172.21.15.40:0/127976202 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43f006bd20 0x7f43f00d20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43e7e00640 1 -- 172.21.15.40:0/127976202 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43f006bd20 msgr2=0x7f43f00d20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43e7e00640 1 --2- 172.21.15.40:0/127976202 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43f006bd20 0x7f43f00d20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43e7e00640 1 -- 172.21.15.40:0/127976202 >> v2:172.21.15.53:3300/0 conn(0x7f43f015df80 msgr2=0x7f43f0070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43e7e00640 1 --2- 172.21.15.40:0/127976202 >> v2:172.21.15.53:3300/0 conn(0x7f43f015df80 0x7f43f0070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.446+0000 7f43e7e00640 1 -- 172.21.15.40:0/127976202 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f006ce10 con 0x7f43f006c950 2024-03-24T00:35:13.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43e7e00640 1 --2- 172.21.15.40:0/127976202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f006c950 0x7f43f00d1b60 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f43f00d2c90 tx=0x7f43d801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0374856ba02f50a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43ed400640 1 -- 172.21.15.40:0/127976202 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d80a7070 con 0x7f43f006c950 2024-03-24T00:35:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43ed400640 1 -- 172.21.15.40:0/127976202 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d80b0070 con 0x7f43f006c950 2024-03-24T00:35:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43ed400640 1 -- 172.21.15.40:0/127976202 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d80ab890 con 0x7f43f006c950 2024-03-24T00:35:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43f4e00640 1 -- 172.21.15.40:0/127976202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f006c950 msgr2=0x7f43f00d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43f4e00640 1 --2- 172.21.15.40:0/127976202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f006c950 0x7f43f00d1b60 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f43f00d2c90 tx=0x7f43d801c4f0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43f4e00640 1 -- 172.21.15.40:0/127976202 shutdown_connections 2024-03-24T00:35:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43f4e00640 1 --2- 172.21.15.40:0/127976202 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43f006bd20 0x7f43f00d20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43f4e00640 1 --2- 172.21.15.40:0/127976202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f006c950 0x7f43f00d1b60 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43f4e00640 1 --2- 172.21.15.40:0/127976202 >> v2:172.21.15.53:3300/0 conn(0x7f43f015df80 0x7f43f0070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43f4e00640 1 -- 172.21.15.40:0/127976202 >> 172.21.15.40:0/127976202 conn(0x7f43f006ecf0 msgr2=0x7f43f006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43f4e00640 1 -- 172.21.15.40:0/127976202 shutdown_connections 2024-03-24T00:35:13.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.450+0000 7f43f4e00640 1 -- 172.21.15.40:0/127976202 wait complete. 2024-03-24T00:35:13.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43f4e00640 1 Processor -- start 2024-03-24T00:35:13.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43f4e00640 1 -- start start 2024-03-24T00:35:13.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43f4e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43f006bd20 0x7f43f00c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:13.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43f4e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f006c950 0x7f43f01fb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:13.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43f4e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43f015df80 0x7f43f015c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:13.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43f4e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f0162460 con 0x7f43f006c950 2024-03-24T00:35:13.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43f4e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01622e0 con 0x7f43f006bd20 2024-03-24T00:35:13.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43f4e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f01625e0 con 0x7f43f015df80 2024-03-24T00:35:13.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f006c950 0x7f43f01fb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:13.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43e7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f006c950 0x7f43f01fb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46566/0 (socket says 172.21.15.40:46566) 2024-03-24T00:35:13.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43e7e00640 1 -- 172.21.15.40:0/1266803477 learned_addr learned my addr 172.21.15.40:0/1266803477 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:13.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43ede00640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43f006bd20 0x7f43f00c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:13.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43ee800640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43f015df80 0x7f43f015c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43e7e00640 1 -- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43f015df80 msgr2=0x7f43f015c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43e7e00640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43f015df80 0x7f43f015c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43e7e00640 1 -- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43f006bd20 msgr2=0x7f43f00c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43e7e00640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43f006bd20 0x7f43f00c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43e7e00640 1 -- 172.21.15.40:0/1266803477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f01fb7a0 con 0x7f43f006c950 2024-03-24T00:35:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43ee800640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43f015df80 0x7f43f015c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43e7e00640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f006c950 0x7f43f01fb260 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f43d801aba0 tx=0x7f43d801ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43e6000640 1 -- 172.21.15.40:0/1266803477 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d80be5f0 con 0x7f43f006c950 2024-03-24T00:35:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43e6000640 1 -- 172.21.15.40:0/1266803477 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d80be790 con 0x7f43f006c950 2024-03-24T00:35:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43e6000640 1 -- 172.21.15.40:0/1266803477 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d80beb80 con 0x7f43f006c950 2024-03-24T00:35:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43f4e00640 1 -- 172.21.15.40:0/1266803477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f014f9a0 con 0x7f43f006c950 2024-03-24T00:35:13.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.454+0000 7f43f4e00640 1 -- 172.21.15.40:0/1266803477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f0151130 con 0x7f43f006c950 2024-03-24T00:35:13.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.458+0000 7f43f4e00640 1 -- 172.21.15.40:0/1266803477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f0068170 con 0x7f43f006c950 2024-03-24T00:35:13.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.458+0000 7f43e6000640 1 -- 172.21.15.40:0/1266803477 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f43d80b43f0 con 0x7f43f006c950 2024-03-24T00:35:13.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.458+0000 7f43e6000640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f43cc067660 0x7f43cc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:13.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.458+0000 7f43e6000640 1 -- 172.21.15.40:0/1266803477 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f43d8120580 con 0x7f43f006c950 2024-03-24T00:35:13.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.458+0000 7f43ede00640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f43cc067660 0x7f43cc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:13.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.458+0000 7f43ede00640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f43cc067660 0x7f43cc069b20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f43e000d8d0 tx=0x7f43e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:13.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.462+0000 7f43e6000640 1 -- 172.21.15.40:0/1266803477 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f43d80c4060 con 0x7f43f006c950 2024-03-24T00:35:13.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:13.626+0000 7f43f4e00640 1 -- 172.21.15.40:0/1266803477 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f43f00c8a10 con 0x7f43f006c950 2024-03-24T00:35:14.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.278+0000 7f43e6000640 1 -- 172.21.15.40:0/1266803477 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v68) v1 ==== 170+0+0 (secure 0 0 0) 0x7f43f01fb7a0 con 0x7f43f006c950 2024-03-24T00:35:14.284 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-03-24T00:35:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.282+0000 7f43f4e00640 1 -- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f43cc067660 msgr2=0x7f43cc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.282+0000 7f43f4e00640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f43cc067660 0x7f43cc069b20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f43e000d8d0 tx=0x7f43e0008040 comp rx=0 tx=0).stop 2024-03-24T00:35:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.282+0000 7f43f4e00640 1 -- 172.21.15.40:0/1266803477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f006c950 msgr2=0x7f43f01fb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.282+0000 7f43f4e00640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f006c950 0x7f43f01fb260 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f43d801aba0 tx=0x7f43d801ad30 comp rx=0 tx=0).stop 2024-03-24T00:35:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.282+0000 7f43f4e00640 1 -- 172.21.15.40:0/1266803477 shutdown_connections 2024-03-24T00:35:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.282+0000 7f43f4e00640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f43cc067660 0x7f43cc069b20 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.282+0000 7f43f4e00640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43f015df80 0x7f43f015c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.282+0000 7f43f4e00640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43f006c950 0x7f43f01fb260 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.282+0000 7f43f4e00640 1 --2- 172.21.15.40:0/1266803477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43f006bd20 0x7f43f00c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.282+0000 7f43f4e00640 1 -- 172.21.15.40:0/1266803477 >> 172.21.15.40:0/1266803477 conn(0x7f43f006ecf0 msgr2=0x7f43f015fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.282+0000 7f43f4e00640 1 -- 172.21.15.40:0/1266803477 shutdown_connections 2024-03-24T00:35:14.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.282+0000 7f43f4e00640 1 -- 172.21.15.40:0/1266803477 wait complete. 2024-03-24T00:35:14.323 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.466+0000 7f2fef400640 1 Processor -- start 2024-03-24T00:35:14.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fef400640 1 -- start start 2024-03-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fef400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fe815bd00 0x7f2fe80ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fef400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2fe80728d0 0x7f2fe80cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fef400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2fe815f8c0 0x7f2fe815fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fef400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe80cf790 con 0x7f2fe80728d0 2024-03-24T00:35:14.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fef400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe8160250 con 0x7f2fe815f8c0 2024-03-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fef400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe8072d30 con 0x7f2fe815bd00 2024-03-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fe7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2fe80728d0 0x7f2fe80cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fe7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2fe80728d0 0x7f2fe80cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46576/0 (socket says 172.21.15.40:46576) 2024-03-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fe7e00640 1 -- 172.21.15.40:0/3661862847 learned_addr learned my addr 172.21.15.40:0/3661862847 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fed400640 1 --2- 172.21.15.40:0/3661862847 >> v2:172.21.15.53:3300/0 conn(0x7f2fe815f8c0 0x7f2fe815fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2feca00640 1 --2- 172.21.15.40:0/3661862847 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fe815bd00 0x7f2fe80ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:14.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fe7e00640 1 -- 172.21.15.40:0/3661862847 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fe815bd00 msgr2=0x7f2fe80ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fe7e00640 1 --2- 172.21.15.40:0/3661862847 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fe815bd00 0x7f2fe80ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fe7e00640 1 -- 172.21.15.40:0/3661862847 >> v2:172.21.15.53:3300/0 conn(0x7f2fe815f8c0 msgr2=0x7f2fe815fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fe7e00640 1 --2- 172.21.15.40:0/3661862847 >> v2:172.21.15.53:3300/0 conn(0x7f2fe815f8c0 0x7f2fe815fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:14.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.470+0000 7f2fe7e00640 1 -- 172.21.15.40:0/3661862847 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe8071de0 con 0x7f2fe80728d0 2024-03-24T00:35:14.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fe7e00640 1 --2- 172.21.15.40:0/3661862847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fe80728d0 0x7f2fe80cf250 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8073320 tx=0x7f2fd401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9b5af3b07329353 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fe7400640 1 -- 172.21.15.40:0/3661862847 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd40aa610 con 0x7f2fe80728d0 2024-03-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fe7400640 1 -- 172.21.15.40:0/3661862847 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd40af070 con 0x7f2fe80728d0 2024-03-24T00:35:14.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fe7400640 1 -- 172.21.15.40:0/3661862847 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd40aa9d0 con 0x7f2fe80728d0 2024-03-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fef400640 1 -- 172.21.15.40:0/3661862847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fe80728d0 msgr2=0x7f2fe80cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fef400640 1 --2- 172.21.15.40:0/3661862847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fe80728d0 0x7f2fe80cf250 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f2fe8073320 tx=0x7f2fd401c4f0 comp rx=0 tx=0).stop 2024-03-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fef400640 1 -- 172.21.15.40:0/3661862847 shutdown_connections 2024-03-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fef400640 1 --2- 172.21.15.40:0/3661862847 >> v2:172.21.15.53:3300/0 conn(0x7f2fe815f8c0 0x7f2fe815fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fef400640 1 --2- 172.21.15.40:0/3661862847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fe80728d0 0x7f2fe80cf250 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fef400640 1 --2- 172.21.15.40:0/3661862847 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fe815bd00 0x7f2fe80ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fef400640 1 -- 172.21.15.40:0/3661862847 >> 172.21.15.40:0/3661862847 conn(0x7f2fe80b1660 msgr2=0x7f2fe80b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fef400640 1 -- 172.21.15.40:0/3661862847 shutdown_connections 2024-03-24T00:35:14.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fef400640 1 -- 172.21.15.40:0/3661862847 wait complete. 2024-03-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fef400640 1 Processor -- start 2024-03-24T00:35:14.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.474+0000 7f2fef400640 1 -- start start 2024-03-24T00:35:14.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fef400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fe80728d0 0x7f2fe8151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fef400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fe815bd00 0x7f2fe815ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fef400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fe815f8c0 0x7f2fe81421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fef400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe806d320 con 0x7f2fe815bd00 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fef400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe815c0e0 con 0x7f2fe80728d0 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fef400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe806d4a0 con 0x7f2fe815f8c0 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fe7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fe815bd00 0x7f2fe815ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2feca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fe80728d0 0x7f2fe8151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fe7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fe815bd00 0x7f2fe815ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46590/0 (socket says 172.21.15.40:46590) 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fe7e00640 1 -- 172.21.15.40:0/3837429442 learned_addr learned my addr 172.21.15.40:0/3837429442 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fed400640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fe815f8c0 0x7f2fe81421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fe7e00640 1 -- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fe815f8c0 msgr2=0x7f2fe81421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fe7e00640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fe815f8c0 0x7f2fe81421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fe7e00640 1 -- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fe80728d0 msgr2=0x7f2fe8151060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:14.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fe7e00640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fe80728d0 0x7f2fe8151060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fe7e00640 1 -- 172.21.15.40:0/3837429442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe815f370 con 0x7f2fe815bd00 2024-03-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fe7e00640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fe815bd00 0x7f2fe815ad00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2fd401aba0 tx=0x7f2fd401ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fe4c00640 1 -- 172.21.15.40:0/3837429442 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd40bd3f0 con 0x7f2fe815bd00 2024-03-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fe4c00640 1 -- 172.21.15.40:0/3837429442 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd40bd590 con 0x7f2fe815bd00 2024-03-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fe4c00640 1 -- 172.21.15.40:0/3837429442 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd40bd940 con 0x7f2fe815bd00 2024-03-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fef400640 1 -- 172.21.15.40:0/3837429442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe81515a0 con 0x7f2fe815bd00 2024-03-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fef400640 1 -- 172.21.15.40:0/3837429442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe815f090 con 0x7f2fe815bd00 2024-03-24T00:35:14.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.478+0000 7f2fef400640 1 -- 172.21.15.40:0/3837429442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe8068170 con 0x7f2fe815bd00 2024-03-24T00:35:14.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.482+0000 7f2fe4c00640 1 -- 172.21.15.40:0/3837429442 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2fd40c7420 con 0x7f2fe815bd00 2024-03-24T00:35:14.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.482+0000 7f2fe4c00640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2fc8067660 0x7f2fc8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.482+0000 7f2fe4c00640 1 -- 172.21.15.40:0/3837429442 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f2fd4120080 con 0x7f2fe815bd00 2024-03-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.482+0000 7f2feca00640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2fc8067660 0x7f2fc8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:14.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.482+0000 7f2feca00640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2fc8067660 0x7f2fc8069b20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009b60 tx=0x7f2fdc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:14.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.486+0000 7f2fe4c00640 1 -- 172.21.15.40:0/3837429442 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2fd40c3060 con 0x7f2fe815bd00 2024-03-24T00:35:14.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:14.686+0000 7f2fef400640 1 -- 172.21.15.40:0/3837429442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2fe80723d0 con 0x7f2fe815bd00 2024-03-24T00:35:15.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.294+0000 7f2fe4c00640 1 -- 172.21.15.40:0/3837429442 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v63) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2fd40a7070 con 0x7f2fe815bd00 2024-03-24T00:35:15.304 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 10 and data pool 11 2024-03-24T00:35:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.302+0000 7f2fef400640 1 -- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2fc8067660 msgr2=0x7f2fc8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.302+0000 7f2fef400640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2fc8067660 0x7f2fc8069b20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009b60 tx=0x7f2fdc009190 comp rx=0 tx=0).stop 2024-03-24T00:35:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.302+0000 7f2fef400640 1 -- 172.21.15.40:0/3837429442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fe815bd00 msgr2=0x7f2fe815ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.302+0000 7f2fef400640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fe815bd00 0x7f2fe815ad00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f2fd401aba0 tx=0x7f2fd401ad30 comp rx=0 tx=0).stop 2024-03-24T00:35:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.306+0000 7f2fef400640 1 -- 172.21.15.40:0/3837429442 shutdown_connections 2024-03-24T00:35:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.306+0000 7f2fef400640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2fc8067660 0x7f2fc8069b20 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.306+0000 7f2fef400640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fe815f8c0 0x7f2fe81421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.306+0000 7f2fef400640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fe815bd00 0x7f2fe815ad00 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.306+0000 7f2fef400640 1 --2- 172.21.15.40:0/3837429442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fe80728d0 0x7f2fe8151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.306+0000 7f2fef400640 1 -- 172.21.15.40:0/3837429442 >> 172.21.15.40:0/3837429442 conn(0x7f2fe80b1660 msgr2=0x7f2fe80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.310+0000 7f2fef400640 1 -- 172.21.15.40:0/3837429442 shutdown_connections 2024-03-24T00:35:15.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.310+0000 7f2fef400640 1 -- 172.21.15.40:0/3837429442 wait complete. 2024-03-24T00:35:15.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:15.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39ebe00640 1 Processor -- start 2024-03-24T00:35:15.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39ebe00640 1 -- start start 2024-03-24T00:35:15.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39ebe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39e406f100 0x7f39e4070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39ebe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f39e406f4e0 0x7f39e4070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39ebe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39e40d27b0 con 0x7f39e40aea60 2024-03-24T00:35:15.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39ebe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39e40b1490 con 0x7f39e406f4e0 2024-03-24T00:35:15.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39ebe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e406fa70 con 0x7f39e406f100 2024-03-24T00:35:15.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39e9400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39e406f100 0x7f39e4070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39e9400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39e406f100 0x7f39e4070be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:58360/0 (socket says 172.21.15.40:58360) 2024-03-24T00:35:15.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39e9400640 1 -- 172.21.15.40:0/3688606221 learned_addr learned my addr 172.21.15.40:0/3688606221 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:15.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39e8a00640 1 --2- 172.21.15.40:0/3688606221 >> v2:172.21.15.53:3300/0 conn(0x7f39e406f4e0 0x7f39e4070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39e9400640 1 -- 172.21.15.40:0/3688606221 >> v2:172.21.15.53:3300/0 conn(0x7f39e406f4e0 msgr2=0x7f39e4070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:15.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39e9400640 1 --2- 172.21.15.40:0/3688606221 >> v2:172.21.15.53:3300/0 conn(0x7f39e406f4e0 0x7f39e4070030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39e9400640 1 -- 172.21.15.40:0/3688606221 >> v1:172.21.15.53:6789/0 conn(0x7f39e40aea60 legacy=0x7f39e40aee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.482+0000 7f39e9400640 1 -- 172.21.15.40:0/3688606221 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e406fd50 con 0x7f39e406f100 2024-03-24T00:35:15.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.486+0000 7f39e9400640 1 --2- 172.21.15.40:0/3688606221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39e406f100 0x7f39e4070be0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f39d8009870 tx=0x7f39d801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5902258953d59825 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:15.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.486+0000 7f39e3e00640 1 -- 172.21.15.40:0/3688606221 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d80b2070 con 0x7f39e406f100 2024-03-24T00:35:15.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.486+0000 7f39e3e00640 1 -- 172.21.15.40:0/3688606221 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d80a9070 con 0x7f39e406f100 2024-03-24T00:35:15.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.486+0000 7f39e3e00640 1 -- 172.21.15.40:0/3688606221 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d80ad880 con 0x7f39e406f100 2024-03-24T00:35:15.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.486+0000 7f39ebe00640 1 -- 172.21.15.40:0/3688606221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39e406f100 msgr2=0x7f39e4070be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.486+0000 7f39ebe00640 1 --2- 172.21.15.40:0/3688606221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39e406f100 0x7f39e4070be0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f39d8009870 tx=0x7f39d801c560 comp rx=0 tx=0).stop 2024-03-24T00:35:15.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.486+0000 7f39ebe00640 1 -- 172.21.15.40:0/3688606221 shutdown_connections 2024-03-24T00:35:15.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.486+0000 7f39ebe00640 1 --2- 172.21.15.40:0/3688606221 >> v2:172.21.15.53:3300/0 conn(0x7f39e406f4e0 0x7f39e4070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.486+0000 7f39ebe00640 1 --2- 172.21.15.40:0/3688606221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39e406f100 0x7f39e4070be0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.486+0000 7f39ebe00640 1 -- 172.21.15.40:0/3688606221 >> 172.21.15.40:0/3688606221 conn(0x7f39e40d1c90 msgr2=0x7f39e40d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:15.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39ebe00640 1 -- 172.21.15.40:0/3688606221 shutdown_connections 2024-03-24T00:35:15.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39ebe00640 1 -- 172.21.15.40:0/3688606221 wait complete. 2024-03-24T00:35:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39ebe00640 1 Processor -- start 2024-03-24T00:35:15.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39ebe00640 1 -- start start 2024-03-24T00:35:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39ebe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39e406f100 0x7f39e4151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39ebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39e406f4e0 0x7f39e40c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39ebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39e40aea60 0x7f39e40c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39ebe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e40b0e10 con 0x7f39e406f4e0 2024-03-24T00:35:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39ebe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e40b0c90 con 0x7f39e40aea60 2024-03-24T00:35:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39ebe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39e40b0f90 con 0x7f39e406f100 2024-03-24T00:35:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39e8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39e406f4e0 0x7f39e40c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39e8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39e406f4e0 0x7f39e40c7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46600/0 (socket says 172.21.15.40:46600) 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39e8a00640 1 -- 172.21.15.40:0/3774525484 learned_addr learned my addr 172.21.15.40:0/3774525484 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39e9400640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39e406f100 0x7f39e4151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39e9e00640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39e40aea60 0x7f39e40c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39e8a00640 1 -- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39e406f100 msgr2=0x7f39e4151c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39e8a00640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39e406f100 0x7f39e4151c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39e8a00640 1 -- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39e40aea60 msgr2=0x7f39e40c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39e8a00640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39e40aea60 0x7f39e40c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39e8a00640 1 -- 172.21.15.40:0/3774525484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e40b47d0 con 0x7f39e406f4e0 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.490+0000 7f39e8a00640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39e406f4e0 0x7f39e40c7cd0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f39d000e960 tx=0x7f39d000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.494+0000 7f39e1600640 1 -- 172.21.15.40:0/3774525484 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0017070 con 0x7f39e406f4e0 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.494+0000 7f39e1600640 1 -- 172.21.15.40:0/3774525484 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39d000ca00 con 0x7f39e406f4e0 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.494+0000 7f39e1600640 1 -- 172.21.15.40:0/3774525484 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d000cdb0 con 0x7f39e406f4e0 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.494+0000 7f39ebe00640 1 -- 172.21.15.40:0/3774525484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e40b1610 con 0x7f39e406f4e0 2024-03-24T00:35:15.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.494+0000 7f39ebe00640 1 -- 172.21.15.40:0/3774525484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e40b3cf0 con 0x7f39e406f4e0 2024-03-24T00:35:15.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.494+0000 7f39ebe00640 1 -- 172.21.15.40:0/3774525484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e40b1a70 con 0x7f39e406f4e0 2024-03-24T00:35:15.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.494+0000 7f39e1600640 1 -- 172.21.15.40:0/3774525484 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f39d001a070 con 0x7f39e406f4e0 2024-03-24T00:35:15.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.494+0000 7f39e1600640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f39c4067660 0x7f39c4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.494+0000 7f39e1600640 1 -- 172.21.15.40:0/3774525484 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39d0085860 con 0x7f39e406f4e0 2024-03-24T00:35:15.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.494+0000 7f39e9400640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f39c4067660 0x7f39c4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.498+0000 7f39e9400640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f39c4067660 0x7f39c4069b20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f39d80af000 tx=0x7f39d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:15.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.498+0000 7f39e1600640 1 -- 172.21.15.40:0/3774525484 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f39d0055e80 con 0x7f39e406f4e0 2024-03-24T00:35:15.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.662+0000 7f39ebe00640 1 -- 172.21.15.40:0/3774525484 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39e4068170 con 0x7f39e406f4e0 2024-03-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.662+0000 7f39e1600640 1 -- 172.21.15.40:0/3774525484 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f39d0059f20 con 0x7f39e406f4e0 2024-03-24T00:35:15.667 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:15.668 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":69,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:35:14.689262+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:13.099776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:13.675426+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.666+0000 7f39ebe00640 1 -- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f39c4067660 msgr2=0x7f39c4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.666+0000 7f39ebe00640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f39c4067660 0x7f39c4069b20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f39d80af000 tx=0x7f39d8008040 comp rx=0 tx=0).stop 2024-03-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.666+0000 7f39ebe00640 1 -- 172.21.15.40:0/3774525484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39e406f4e0 msgr2=0x7f39e40c7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.666+0000 7f39ebe00640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39e406f4e0 0x7f39e40c7cd0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f39d000e960 tx=0x7f39d000ee20 comp rx=0 tx=0).stop 2024-03-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.666+0000 7f39ebe00640 1 -- 172.21.15.40:0/3774525484 shutdown_connections 2024-03-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.666+0000 7f39ebe00640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f39c4067660 0x7f39c4069b20 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.666+0000 7f39ebe00640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39e40aea60 0x7f39e40c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.666+0000 7f39ebe00640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39e406f4e0 0x7f39e40c7cd0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.666+0000 7f39ebe00640 1 --2- 172.21.15.40:0/3774525484 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39e406f100 0x7f39e4151c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.666+0000 7f39ebe00640 1 -- 172.21.15.40:0/3774525484 >> 172.21.15.40:0/3774525484 conn(0x7f39e40d1c90 msgr2=0x7f39e40aee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:15.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.666+0000 7f39ebe00640 1 -- 172.21.15.40:0/3774525484 shutdown_connections 2024-03-24T00:35:15.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.666+0000 7f39ebe00640 1 -- 172.21.15.40:0/3774525484 wait complete. 2024-03-24T00:35:15.688 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:15.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a1a800640 1 Processor -- start 2024-03-24T00:35:15.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a1a800640 1 -- start start 2024-03-24T00:35:15.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a1a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0a1415c080 0x7f0a140d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a1a800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a140d0780 con 0x7f0a1415bca0 2024-03-24T00:35:15.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a1a800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a140b4a90 con 0x7f0a140ceba0 2024-03-24T00:35:15.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a1a800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a140b4d70 con 0x7f0a1415c080 2024-03-24T00:35:15.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a19400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0a1415bca0 0x7f0a140ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48488/0 (socket says 172.21.15.40:48488) 2024-03-24T00:35:15.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a19400640 1 -- 172.21.15.40:0/2071720678 learned_addr learned my addr 172.21.15.40:0/2071720678 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:15.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a19e00640 1 --2- 172.21.15.40:0/2071720678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0a1415c080 0x7f0a140d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a13e00640 1 -- 172.21.15.40:0/2071720678 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417502398 0 0) 0x7f0a140d0780 con 0x7f0a1415bca0 2024-03-24T00:35:15.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a13e00640 1 -- 172.21.15.40:0/2071720678 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09fc003530 con 0x7f0a1415bca0 2024-03-24T00:35:15.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a13e00640 1 -- 172.21.15.40:0/2071720678 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380400983 0 0) 0x7f0a140b4a90 con 0x7f0a140ceba0 2024-03-24T00:35:15.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a19e00640 1 -- 172.21.15.40:0/2071720678 >> v1:172.21.15.53:6789/0 conn(0x7f0a140ceba0 legacy=0x7f0a140b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a19e00640 1 -- 172.21.15.40:0/2071720678 >> v1:172.21.15.40:6789/0 conn(0x7f0a1415bca0 legacy=0x7f0a140ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.826+0000 7f0a19e00640 1 -- 172.21.15.40:0/2071720678 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a140b5050 con 0x7f0a1415c080 2024-03-24T00:35:15.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.830+0000 7f0a19e00640 1 --2- 172.21.15.40:0/2071720678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0a1415c080 0x7f0a140d2590 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c009dc0 tx=0x7f0a0c01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de420d6c0bf9db56 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:15.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.830+0000 7f0a13e00640 1 -- 172.21.15.40:0/2071720678 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c0b2070 con 0x7f0a1415c080 2024-03-24T00:35:15.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.830+0000 7f0a13e00640 1 -- 172.21.15.40:0/2071720678 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0c0a5960 con 0x7f0a1415c080 2024-03-24T00:35:15.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.830+0000 7f0a13e00640 1 -- 172.21.15.40:0/2071720678 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c0a5d30 con 0x7f0a1415c080 2024-03-24T00:35:15.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.830+0000 7f0a1a800640 1 -- 172.21.15.40:0/2071720678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0a1415c080 msgr2=0x7f0a140d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.830+0000 7f0a1a800640 1 --2- 172.21.15.40:0/2071720678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0a1415c080 0x7f0a140d2590 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c009dc0 tx=0x7f0a0c01c990 comp rx=0 tx=0).stop 2024-03-24T00:35:15.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.830+0000 7f0a1a800640 1 -- 172.21.15.40:0/2071720678 shutdown_connections 2024-03-24T00:35:15.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.830+0000 7f0a1a800640 1 --2- 172.21.15.40:0/2071720678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0a1415c080 0x7f0a140d2590 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.830+0000 7f0a1a800640 1 -- 172.21.15.40:0/2071720678 >> 172.21.15.40:0/2071720678 conn(0x7f0a140d1aa0 msgr2=0x7f0a140d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:15.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.830+0000 7f0a1a800640 1 -- 172.21.15.40:0/2071720678 shutdown_connections 2024-03-24T00:35:15.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.830+0000 7f0a1a800640 1 -- 172.21.15.40:0/2071720678 wait complete. 2024-03-24T00:35:15.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a1a800640 1 Processor -- start 2024-03-24T00:35:15.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a1a800640 1 -- start start 2024-03-24T00:35:15.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a1a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a140ceba0 0x7f0a14150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a1a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0a1415bca0 0x7f0a141554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a1a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a1415c080 0x7f0a140c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a1a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a140b6820 con 0x7f0a1415c080 2024-03-24T00:35:15.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a1a800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a140b66a0 con 0x7f0a140ceba0 2024-03-24T00:35:15.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a1a800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a14169a50 con 0x7f0a1415bca0 2024-03-24T00:35:15.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a19e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a1415c080 0x7f0a140c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a19e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a1415c080 0x7f0a140c9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46606/0 (socket says 172.21.15.40:46606) 2024-03-24T00:35:15.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a19e00640 1 -- 172.21.15.40:0/1760567605 learned_addr learned my addr 172.21.15.40:0/1760567605 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:15.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a18a00640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0a1415bca0 0x7f0a141554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a19400640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a140ceba0 0x7f0a14150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a19e00640 1 -- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0a1415bca0 msgr2=0x7f0a141554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a19e00640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0a1415bca0 0x7f0a141554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a19e00640 1 -- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a140ceba0 msgr2=0x7f0a14150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:15.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a19e00640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a140ceba0 0x7f0a14150ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:15.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a19e00640 1 -- 172.21.15.40:0/1760567605 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a14155eb0 con 0x7f0a1415c080 2024-03-24T00:35:15.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a19e00640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a1415c080 0x7f0a140c9560 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c01ce70 tx=0x7f0a0c019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:15.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a11600640 1 -- 172.21.15.40:0/1760567605 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c0b2050 con 0x7f0a1415c080 2024-03-24T00:35:15.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a11600640 1 -- 172.21.15.40:0/1760567605 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0c0be070 con 0x7f0a1415c080 2024-03-24T00:35:15.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a11600640 1 -- 172.21.15.40:0/1760567605 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c009450 con 0x7f0a1415c080 2024-03-24T00:35:15.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a1a800640 1 -- 172.21.15.40:0/1760567605 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a14156190 con 0x7f0a1415c080 2024-03-24T00:35:15.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a1a800640 1 -- 172.21.15.40:0/1760567605 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a14156750 con 0x7f0a1415c080 2024-03-24T00:35:15.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.834+0000 7f0a1a800640 1 -- 172.21.15.40:0/1760567605 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1409fb10 con 0x7f0a1415c080 2024-03-24T00:35:15.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.838+0000 7f0a11600640 1 -- 172.21.15.40:0/1760567605 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f0a0c0c3460 con 0x7f0a1415c080 2024-03-24T00:35:15.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.838+0000 7f0a11600640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f09f4067660 0x7f09f4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:15.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.838+0000 7f0a11600640 1 -- 172.21.15.40:0/1760567605 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a0c120d10 con 0x7f0a1415c080 2024-03-24T00:35:15.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.838+0000 7f0a19400640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f09f4067660 0x7f09f4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:15.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.838+0000 7f0a19400640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f09f4067660 0x7f09f4069b20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0a08007fd0 tx=0x7f0a08007d00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:15.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:15.842+0000 7f0a11600640 1 -- 172.21.15.40:0/1760567605 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0a0c126020 con 0x7f0a1415c080 2024-03-24T00:35:16.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.002+0000 7f0a1a800640 1 -- 172.21.15.40:0/1760567605 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a141515a0 con 0x7f0a1415c080 2024-03-24T00:35:16.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a11600640 1 -- 172.21.15.40:0/1760567605 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f0a0c016580 con 0x7f0a1415c080 2024-03-24T00:35:16.008 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:16.008 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":69,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:35:14.689262+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:13.099776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:13.675426+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a1a800640 1 -- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f09f4067660 msgr2=0x7f09f4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a1a800640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f09f4067660 0x7f09f4069b20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0a08007fd0 tx=0x7f0a08007d00 comp rx=0 tx=0).stop 2024-03-24T00:35:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a1a800640 1 -- 172.21.15.40:0/1760567605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a1415c080 msgr2=0x7f0a140c9560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a1a800640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a1415c080 0x7f0a140c9560 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c01ce70 tx=0x7f0a0c019730 comp rx=0 tx=0).stop 2024-03-24T00:35:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a1a800640 1 -- 172.21.15.40:0/1760567605 shutdown_connections 2024-03-24T00:35:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a1a800640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f09f4067660 0x7f09f4069b20 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a1a800640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0a1415c080 0x7f0a140c9560 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:16.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a1a800640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0a1415bca0 0x7f0a141554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a1a800640 1 --2- 172.21.15.40:0/1760567605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a140ceba0 0x7f0a14150ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a1a800640 1 -- 172.21.15.40:0/1760567605 >> 172.21.15.40:0/1760567605 conn(0x7f0a140d1aa0 msgr2=0x7f0a14070490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a1a800640 1 -- 172.21.15.40:0/1760567605 shutdown_connections 2024-03-24T00:35:16.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.006+0000 7f0a1a800640 1 -- 172.21.15.40:0/1760567605 wait complete. 2024-03-24T00:35:16.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:35:16.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.170+0000 7f0526a00640 1 Processor -- start 2024-03-24T00:35:16.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.170+0000 7f0526a00640 1 -- start start 2024-03-24T00:35:16.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.170+0000 7f0526a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f05200b2360 0x7f05200b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:16.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.170+0000 7f0526a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05200cf090 con 0x7f05200b4330 2024-03-24T00:35:16.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.170+0000 7f0526a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05200cf330 con 0x7f05200cebd0 2024-03-24T00:35:16.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.170+0000 7f0526a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05200d0f60 con 0x7f05200b2360 2024-03-24T00:35:16.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.170+0000 7f0517e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f05200b4330 0x7f05200ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:48498/0 (socket says 172.21.15.40:48498) 2024-03-24T00:35:16.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.170+0000 7f0517e00640 1 -- 172.21.15.40:0/657355564 learned_addr learned my addr 172.21.15.40:0/657355564 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:16.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.170+0000 7f051fe00640 1 --2- 172.21.15.40:0/657355564 >> v2:172.21.15.53:3300/0 conn(0x7f05200b2360 0x7f05200b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:16.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.174+0000 7f051f400640 1 -- 172.21.15.40:0/657355564 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861520675 0 0) 0x7f05200cf090 con 0x7f05200b4330 2024-03-24T00:35:16.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.174+0000 7f051f400640 1 -- 172.21.15.40:0/657355564 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0504003530 con 0x7f05200b4330 2024-03-24T00:35:16.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.174+0000 7f051fe00640 1 -- 172.21.15.40:0/657355564 >> v1:172.21.15.53:6789/0 conn(0x7f05200cebd0 legacy=0x7f05200cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:16.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.174+0000 7f051fe00640 1 -- 172.21.15.40:0/657355564 >> v1:172.21.15.40:6789/0 conn(0x7f05200b4330 legacy=0x7f05200ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:16.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.174+0000 7f051fe00640 1 -- 172.21.15.40:0/657355564 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05200d0910 con 0x7f05200b2360 2024-03-24T00:35:16.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.174+0000 7f051fe00640 1 --2- 172.21.15.40:0/657355564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05200b2360 0x7f05200b2740 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f05100097c0 tx=0x7f051001c350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75a40a80f2cccd34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:16.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.174+0000 7f051f400640 1 -- 172.21.15.40:0/657355564 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05100b0070 con 0x7f05200b2360 2024-03-24T00:35:16.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.174+0000 7f051f400640 1 -- 172.21.15.40:0/657355564 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0510004230 con 0x7f05200b2360 2024-03-24T00:35:16.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.174+0000 7f051f400640 1 -- 172.21.15.40:0/657355564 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510003620 con 0x7f05200b2360 2024-03-24T00:35:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.174+0000 7f0526a00640 1 -- 172.21.15.40:0/657355564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05200b2360 msgr2=0x7f05200b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.174+0000 7f0526a00640 1 --2- 172.21.15.40:0/657355564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05200b2360 0x7f05200b2740 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f05100097c0 tx=0x7f051001c350 comp rx=0 tx=0).stop 2024-03-24T00:35:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 -- 172.21.15.40:0/657355564 shutdown_connections 2024-03-24T00:35:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 --2- 172.21.15.40:0/657355564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05200b2360 0x7f05200b2740 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 -- 172.21.15.40:0/657355564 >> 172.21.15.40:0/657355564 conn(0x7f05200b3810 msgr2=0x7f05200b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:16.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 -- 172.21.15.40:0/657355564 shutdown_connections 2024-03-24T00:35:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 -- 172.21.15.40:0/657355564 wait complete. 2024-03-24T00:35:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 Processor -- start 2024-03-24T00:35:16.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 -- start start 2024-03-24T00:35:16.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05200b2360 0x7f0520154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:16.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05200b4330 0x7f052015ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:16.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05200cebd0 0x7f05201fa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:16.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05200cd360 con 0x7f05200b2360 2024-03-24T00:35:16.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05200d19e0 con 0x7f05200b4330 2024-03-24T00:35:16.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0526a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05200cd4e0 con 0x7f05200cebd0 2024-03-24T00:35:16.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f051fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05200b2360 0x7f0520154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:16.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f051fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05200b2360 0x7f0520154de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:46618/0 (socket says 172.21.15.40:46618) 2024-03-24T00:35:16.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f051fe00640 1 -- 172.21.15.40:0/2363150573 learned_addr learned my addr 172.21.15.40:0/2363150573 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:16.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0524a00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05200cebd0 0x7f05201fa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:16.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0517e00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05200b4330 0x7f052015ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:16.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f051fe00640 1 -- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05200cebd0 msgr2=0x7f05201fa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:16.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f051fe00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05200cebd0 0x7f05201fa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:16.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f051fe00640 1 -- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05200b4330 msgr2=0x7f052015ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:16.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f051fe00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05200b4330 0x7f052015ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:16.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f051fe00640 1 -- 172.21.15.40:0/2363150573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05200c5e30 con 0x7f05200b2360 2024-03-24T00:35:16.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f0517e00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05200b4330 0x7f052015ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:16.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.178+0000 7f051fe00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05200b2360 0x7f0520154de0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f05100ae040 tx=0x7f05100190f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:16.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.182+0000 7f051cc00640 1 -- 172.21.15.40:0/2363150573 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05100b9070 con 0x7f05200b2360 2024-03-24T00:35:16.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.182+0000 7f051cc00640 1 -- 172.21.15.40:0/2363150573 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0510003e10 con 0x7f05200b2360 2024-03-24T00:35:16.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.182+0000 7f051cc00640 1 -- 172.21.15.40:0/2363150573 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05100b0040 con 0x7f05200b2360 2024-03-24T00:35:16.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.182+0000 7f0526a00640 1 -- 172.21.15.40:0/2363150573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f052015f7e0 con 0x7f05200b2360 2024-03-24T00:35:16.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.182+0000 7f0526a00640 1 -- 172.21.15.40:0/2363150573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052015e570 con 0x7f05200b2360 2024-03-24T00:35:16.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.182+0000 7f0526a00640 1 -- 172.21.15.40:0/2363150573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0520068170 con 0x7f05200b2360 2024-03-24T00:35:16.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.182+0000 7f051cc00640 1 -- 172.21.15.40:0/2363150573 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f05100bc070 con 0x7f05200b2360 2024-03-24T00:35:16.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.182+0000 7f051cc00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f04fc067660 0x7f04fc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:16.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.182+0000 7f051cc00640 1 -- 172.21.15.40:0/2363150573 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f051011f920 con 0x7f05200b2360 2024-03-24T00:35:16.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.182+0000 7f0517e00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f04fc067660 0x7f04fc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:16.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.186+0000 7f0517e00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f04fc067660 0x7f04fc069b20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f05200cc180 tx=0x7f0508009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:16.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.186+0000 7f051cc00640 1 -- 172.21.15.40:0/2363150573 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0510016090 con 0x7f05200b2360 2024-03-24T00:35:16.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:16.390+0000 7f0526a00640 1 -- 172.21.15.40:0/2363150573 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0520070c20 con 0x7f05200b2360 2024-03-24T00:35:17.275 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-03-24T00:35:17.270+0000 7f321f200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:35:17.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f051cc00640 1 -- 172.21.15.40:0/2363150573 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v66) v1 ==== 118+0+0 (secure 0 0 0) 0x7f05100f0060 con 0x7f05200b2360 2024-03-24T00:35:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f0526a00640 1 -- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f04fc067660 msgr2=0x7f04fc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f0526a00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f04fc067660 0x7f04fc069b20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f05200cc180 tx=0x7f0508009040 comp rx=0 tx=0).stop 2024-03-24T00:35:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f0526a00640 1 -- 172.21.15.40:0/2363150573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05200b2360 msgr2=0x7f0520154de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f0526a00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05200b2360 0x7f0520154de0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f05100ae040 tx=0x7f05100190f0 comp rx=0 tx=0).stop 2024-03-24T00:35:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f0526a00640 1 -- 172.21.15.40:0/2363150573 shutdown_connections 2024-03-24T00:35:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f0526a00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f04fc067660 0x7f04fc069b20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f0526a00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05200cebd0 0x7f05201fa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f0526a00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05200b4330 0x7f052015ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f0526a00640 1 --2- 172.21.15.40:0/2363150573 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05200b2360 0x7f0520154de0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f0526a00640 1 -- 172.21.15.40:0/2363150573 >> 172.21.15.40:0/2363150573 conn(0x7f05200b3810 msgr2=0x7f05200b3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:17.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f0526a00640 1 -- 172.21.15.40:0/2363150573 shutdown_connections 2024-03-24T00:35:17.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.310+0000 7f0526a00640 1 -- 172.21.15.40:0/2363150573 wait complete. 2024-03-24T00:35:17.334 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:35:17.335 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:35:17.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.482+0000 7fa6fca00640 1 Processor -- start 2024-03-24T00:35:17.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.482+0000 7fa6fca00640 1 -- start start 2024-03-24T00:35:17.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.486+0000 7fa6fca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa6f80d2270 0x7fa6f8070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:17.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.486+0000 7fa6fca00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f806d0f0 con 0x7fa6f806c950 2024-03-24T00:35:17.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.486+0000 7fa6fca00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f806b280 con 0x7fa6f815df80 2024-03-24T00:35:17.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.486+0000 7fa6fca00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f806b890 con 0x7fa6f80d2270 2024-03-24T00:35:17.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.486+0000 7fa6f6800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa6f806c950 0x7fa6f8161ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36828/0 (socket says 172.21.15.40:36828) 2024-03-24T00:35:17.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.486+0000 7fa6f6800640 1 -- 172.21.15.40:0/236381083 learned_addr learned my addr 172.21.15.40:0/236381083 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:17.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.486+0000 7fa6efe00640 1 --2- 172.21.15.40:0/236381083 >> v2:172.21.15.53:3300/0 conn(0x7fa6f80d2270 0x7fa6f8070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:17.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.486+0000 7fa6f5400640 1 -- 172.21.15.40:0/236381083 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971279972 0 0) 0x7fa6f806d0f0 con 0x7fa6f806c950 2024-03-24T00:35:17.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.486+0000 7fa6f5400640 1 -- 172.21.15.40:0/236381083 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6dc003530 con 0x7fa6f806c950 2024-03-24T00:35:17.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.486+0000 7fa6efe00640 1 -- 172.21.15.40:0/236381083 >> v1:172.21.15.53:6789/0 conn(0x7fa6f815df80 legacy=0x7fa6f80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:17.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.486+0000 7fa6efe00640 1 -- 172.21.15.40:0/236381083 >> v1:172.21.15.40:6789/0 conn(0x7fa6f806c950 legacy=0x7fa6f8161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:17.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.486+0000 7fa6efe00640 1 -- 172.21.15.40:0/236381083 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f806ce10 con 0x7fa6f80d2270 2024-03-24T00:35:17.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6efe00640 1 --2- 172.21.15.40:0/236381083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6f80d2270 0x7fa6f8070ff0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa6f806ba10 tx=0x7fa6e001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ec0f8dbc320f534 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:17.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6f5400640 1 -- 172.21.15.40:0/236381083 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e00ab430 con 0x7fa6f80d2270 2024-03-24T00:35:17.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6f5400640 1 -- 172.21.15.40:0/236381083 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e00b0070 con 0x7fa6f80d2270 2024-03-24T00:35:17.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6f5400640 1 -- 172.21.15.40:0/236381083 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e00ab7f0 con 0x7fa6f80d2270 2024-03-24T00:35:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6fca00640 1 -- 172.21.15.40:0/236381083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6f80d2270 msgr2=0x7fa6f8070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6fca00640 1 --2- 172.21.15.40:0/236381083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6f80d2270 0x7fa6f8070ff0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa6f806ba10 tx=0x7fa6e001c470 comp rx=0 tx=0).stop 2024-03-24T00:35:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6fca00640 1 -- 172.21.15.40:0/236381083 shutdown_connections 2024-03-24T00:35:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6fca00640 1 --2- 172.21.15.40:0/236381083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6f80d2270 0x7fa6f8070ff0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6fca00640 1 -- 172.21.15.40:0/236381083 >> 172.21.15.40:0/236381083 conn(0x7fa6f806ecf0 msgr2=0x7fa6f806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6fca00640 1 -- 172.21.15.40:0/236381083 shutdown_connections 2024-03-24T00:35:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6fca00640 1 -- 172.21.15.40:0/236381083 wait complete. 2024-03-24T00:35:17.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6fca00640 1 Processor -- start 2024-03-24T00:35:17.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.490+0000 7fa6fca00640 1 -- start start 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6fca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6f806c950 0x7fa6f8160c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6fca00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6f80d2270 0x7fa6f81529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6fca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6f815df80 0x7fa6f815c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6fca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f816bfa0 con 0x7fa6f815df80 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6fca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f816be20 con 0x7fa6f806c950 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6fca00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f816c120 con 0x7fa6f80d2270 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6f6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6f815df80 0x7fa6f815c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6f6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6f815df80 0x7fa6f815c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54924/0 (socket says 172.21.15.40:54924) 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6f6800640 1 -- 172.21.15.40:0/1343080450 learned_addr learned my addr 172.21.15.40:0/1343080450 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6f5e00640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6f806c950 0x7fa6f8160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6efe00640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6f80d2270 0x7fa6f81529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6f6800640 1 -- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6f80d2270 msgr2=0x7fa6f81529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:17.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6f6800640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6f80d2270 0x7fa6f81529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:17.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6f6800640 1 -- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6f806c950 msgr2=0x7fa6f8160c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:17.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6f6800640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6f806c950 0x7fa6f8160c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:17.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6f6800640 1 -- 172.21.15.40:0/1343080450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f815d300 con 0x7fa6f815df80 2024-03-24T00:35:17.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6f6800640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6f815df80 0x7fa6f815c380 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000cbb0 tx=0x7fa6f000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:17.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6ee000640 1 -- 172.21.15.40:0/1343080450 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0017070 con 0x7fa6f815df80 2024-03-24T00:35:17.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6fca00640 1 -- 172.21.15.40:0/1343080450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f815d5e0 con 0x7fa6f815df80 2024-03-24T00:35:17.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6ee000640 1 -- 172.21.15.40:0/1343080450 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f000ad40 con 0x7fa6f815df80 2024-03-24T00:35:17.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6ee000640 1 -- 172.21.15.40:0/1343080450 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0012600 con 0x7fa6f815df80 2024-03-24T00:35:17.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6fca00640 1 -- 172.21.15.40:0/1343080450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f8160440 con 0x7fa6f815df80 2024-03-24T00:35:17.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6fca00640 1 -- 172.21.15.40:0/1343080450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6f815d080 con 0x7fa6f815df80 2024-03-24T00:35:17.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.494+0000 7fa6ee000640 1 -- 172.21.15.40:0/1343080450 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa6f001a070 con 0x7fa6f815df80 2024-03-24T00:35:17.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.498+0000 7fa6ee000640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6d4067690 0x7fa6d4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:17.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.498+0000 7fa6ee000640 1 -- 172.21.15.40:0/1343080450 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6f00855d0 con 0x7fa6f815df80 2024-03-24T00:35:17.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.498+0000 7fa6f5e00640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6d4067690 0x7fa6d4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:17.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.498+0000 7fa6f5e00640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6d4067690 0x7fa6d4069b50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80c86b0 tx=0x7fa6e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:17.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.502+0000 7fa6ee000640 1 -- 172.21.15.40:0/1343080450 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa6f815d080 con 0x7fa6f815df80 2024-03-24T00:35:17.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:17.698+0000 7fa6fca00640 1 -- 172.21.15.40:0/1343080450 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa6f8068170 con 0x7fa6f815df80 2024-03-24T00:35:18.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.310+0000 7fa6ee000640 1 -- 172.21.15.40:0/1343080450 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v67) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa6f0055b70 con 0x7fa6f815df80 2024-03-24T00:35:18.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.314+0000 7fa6fca00640 1 -- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6d4067690 msgr2=0x7fa6d4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.314+0000 7fa6fca00640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6d4067690 0x7fa6d4069b50 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa6f80c86b0 tx=0x7fa6e8009040 comp rx=0 tx=0).stop 2024-03-24T00:35:18.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.314+0000 7fa6fca00640 1 -- 172.21.15.40:0/1343080450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6f815df80 msgr2=0x7fa6f815c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.314+0000 7fa6fca00640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6f815df80 0x7fa6f815c380 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000cbb0 tx=0x7fa6f000a430 comp rx=0 tx=0).stop 2024-03-24T00:35:18.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.314+0000 7fa6fca00640 1 -- 172.21.15.40:0/1343080450 shutdown_connections 2024-03-24T00:35:18.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.314+0000 7fa6fca00640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6d4067690 0x7fa6d4069b50 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.314+0000 7fa6fca00640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6f815df80 0x7fa6f815c380 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.314+0000 7fa6fca00640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6f80d2270 0x7fa6f81529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.314+0000 7fa6fca00640 1 --2- 172.21.15.40:0/1343080450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6f806c950 0x7fa6f8160c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.314+0000 7fa6fca00640 1 -- 172.21.15.40:0/1343080450 >> 172.21.15.40:0/1343080450 conn(0x7fa6f806ecf0 msgr2=0x7fa6f8158f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:18.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.314+0000 7fa6fca00640 1 -- 172.21.15.40:0/1343080450 shutdown_connections 2024-03-24T00:35:18.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.314+0000 7fa6fca00640 1 -- 172.21.15.40:0/1343080450 wait complete. 2024-03-24T00:35:18.336 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:35:18.336 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:18.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.514+0000 7fba2f000640 1 Processor -- start 2024-03-24T00:35:18.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.514+0000 7fba2f000640 1 -- start start 2024-03-24T00:35:18.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba2f000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba2813ec80 0x7fba281414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba2f000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fba280ce800 0x7fba28071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba2f000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba28070e00 con 0x7fba281419f0 2024-03-24T00:35:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba2f000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba28141e50 con 0x7fba280ce800 2024-03-24T00:35:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba2f000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba280cee20 con 0x7fba2813ec80 2024-03-24T00:35:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba1fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fba281419f0 0x7fba280ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36842/0 (socket says 172.21.15.40:36842) 2024-03-24T00:35:18.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba1fe00640 1 -- 172.21.15.40:0/3037001992 learned_addr learned my addr 172.21.15.40:0/3037001992 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba27e00640 1 --2- 172.21.15.40:0/3037001992 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba2813ec80 0x7fba281414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba2d000640 1 --2- 172.21.15.40:0/3037001992 >> v2:172.21.15.53:3300/0 conn(0x7fba280ce800 0x7fba28071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba27400640 1 -- 172.21.15.40:0/3037001992 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807001234 0 0) 0x7fba28070e00 con 0x7fba281419f0 2024-03-24T00:35:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba27400640 1 -- 172.21.15.40:0/3037001992 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba0c003530 con 0x7fba281419f0 2024-03-24T00:35:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba27400640 1 -- 172.21.15.40:0/3037001992 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2533135870 0 0) 0x7fba0c003530 con 0x7fba281419f0 2024-03-24T00:35:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba27e00640 1 -- 172.21.15.40:0/3037001992 >> v2:172.21.15.53:3300/0 conn(0x7fba280ce800 msgr2=0x7fba28071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba27e00640 1 --2- 172.21.15.40:0/3037001992 >> v2:172.21.15.53:3300/0 conn(0x7fba280ce800 0x7fba28071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba27e00640 1 -- 172.21.15.40:0/3037001992 >> v1:172.21.15.40:6789/0 conn(0x7fba281419f0 legacy=0x7fba280ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.518+0000 7fba27e00640 1 -- 172.21.15.40:0/3037001992 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba280cf100 con 0x7fba2813ec80 2024-03-24T00:35:18.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba27e00640 1 --2- 172.21.15.40:0/3037001992 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba2813ec80 0x7fba281414b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fba180097c0 tx=0x7fba1801c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f11e91da5781e32 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:18.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba27400640 1 -- 172.21.15.40:0/3037001992 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba180b0070 con 0x7fba2813ec80 2024-03-24T00:35:18.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba27400640 1 -- 172.21.15.40:0/3037001992 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba180ab400 con 0x7fba2813ec80 2024-03-24T00:35:18.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba27400640 1 -- 172.21.15.40:0/3037001992 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba10001680 con 0x7fba2813ec80 2024-03-24T00:35:18.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba2f000640 1 -- 172.21.15.40:0/3037001992 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba2813ec80 msgr2=0x7fba281414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba2f000640 1 --2- 172.21.15.40:0/3037001992 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba2813ec80 0x7fba281414b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fba180097c0 tx=0x7fba1801c3c0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba2f000640 1 -- 172.21.15.40:0/3037001992 shutdown_connections 2024-03-24T00:35:18.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba2f000640 1 --2- 172.21.15.40:0/3037001992 >> v2:172.21.15.53:3300/0 conn(0x7fba280ce800 0x7fba28071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba2f000640 1 --2- 172.21.15.40:0/3037001992 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba2813ec80 0x7fba281414b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba2f000640 1 -- 172.21.15.40:0/3037001992 >> 172.21.15.40:0/3037001992 conn(0x7fba280b17b0 msgr2=0x7fba280b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:18.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba2f000640 1 -- 172.21.15.40:0/3037001992 shutdown_connections 2024-03-24T00:35:18.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba2f000640 1 -- 172.21.15.40:0/3037001992 wait complete. 2024-03-24T00:35:18.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba2f000640 1 Processor -- start 2024-03-24T00:35:18.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.522+0000 7fba2f000640 1 -- start start 2024-03-24T00:35:18.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba2f000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba280ce800 0x7fba28158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:18.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba2f000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba2813ec80 0x7fba28140980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:18.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba2f000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba281419f0 0x7fba281557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:18.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba2f000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba280723e0 con 0x7fba2813ec80 2024-03-24T00:35:18.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba2f000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba28072260 con 0x7fba280ce800 2024-03-24T00:35:18.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba2f000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba28072560 con 0x7fba281419f0 2024-03-24T00:35:18.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba1fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba2813ec80 0x7fba28140980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:18.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba1fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba2813ec80 0x7fba28140980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54940/0 (socket says 172.21.15.40:54940) 2024-03-24T00:35:18.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba1fe00640 1 -- 172.21.15.40:0/725930845 learned_addr learned my addr 172.21.15.40:0/725930845 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:18.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba27e00640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba280ce800 0x7fba28158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:18.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba2d000640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba281419f0 0x7fba281557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:18.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba1fe00640 1 -- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba281419f0 msgr2=0x7fba281557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba1fe00640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba281419f0 0x7fba281557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba1fe00640 1 -- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba280ce800 msgr2=0x7fba28158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba1fe00640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba280ce800 0x7fba28158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba1fe00640 1 -- 172.21.15.40:0/725930845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2815e560 con 0x7fba2813ec80 2024-03-24T00:35:18.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba2d000640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba281419f0 0x7fba281557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:18.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba1fe00640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba2813ec80 0x7fba28140980 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fba1000c750 tx=0x7fba1000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:18.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba24c00640 1 -- 172.21.15.40:0/725930845 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba10005ae0 con 0x7fba2813ec80 2024-03-24T00:35:18.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba24c00640 1 -- 172.21.15.40:0/725930845 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba10005ca0 con 0x7fba2813ec80 2024-03-24T00:35:18.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba2f000640 1 -- 172.21.15.40:0/725930845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2815e840 con 0x7fba2813ec80 2024-03-24T00:35:18.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba24c00640 1 -- 172.21.15.40:0/725930845 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba100072b0 con 0x7fba2813ec80 2024-03-24T00:35:18.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba2f000640 1 -- 172.21.15.40:0/725930845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2815ee00 con 0x7fba2813ec80 2024-03-24T00:35:18.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.526+0000 7fba2f000640 1 -- 172.21.15.40:0/725930845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba28068170 con 0x7fba2813ec80 2024-03-24T00:35:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.530+0000 7fba24c00640 1 -- 172.21.15.40:0/725930845 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fba10007450 con 0x7fba2813ec80 2024-03-24T00:35:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.530+0000 7fba24c00640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fba04067660 0x7fba04069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:18.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.530+0000 7fba24c00640 1 -- 172.21.15.40:0/725930845 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba10085810 con 0x7fba2813ec80 2024-03-24T00:35:18.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.530+0000 7fba27e00640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fba04067660 0x7fba04069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:18.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.530+0000 7fba27e00640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fba04067660 0x7fba04069b20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fba180ad000 tx=0x7fba180a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:18.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.534+0000 7fba24c00640 1 -- 172.21.15.40:0/725930845 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fba100563f0 con 0x7fba2813ec80 2024-03-24T00:35:18.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.942+0000 7fba2f000640 1 -- 172.21.15.40:0/725930845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba28072b90 con 0x7fba2813ec80 2024-03-24T00:35:18.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba24c00640 1 -- 172.21.15.40:0/725930845 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4973 (secure 0 0 0) 0x7fba10055cf0 con 0x7fba2813ec80 2024-03-24T00:35:18.948 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:18.948 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":67,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6062,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/4182264423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":4182264423},{"type":"v1","addr":"172.21.15.53:6836","nonce":4182264423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6733,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/334375209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":334375209},{"type":"v1","addr":"172.21.15.53:6837","nonce":334375209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/221849812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":221849812},{"type":"v1","addr":"172.21.15.40:6834","nonce":221849812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:35:14.689580+0000","modified":"2024-03-24T00:35:17.703915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6954":{"gid":6954,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/2177098794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2177098794},{"type":"v1","addr":"172.21.15.40:6835","nonce":2177098794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6954,"qdb_cluster":[6954]},"id":5}]} 2024-03-24T00:35:18.949 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 67 2024-03-24T00:35:18.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba2f000640 1 -- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fba04067660 msgr2=0x7fba04069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba2f000640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fba04067660 0x7fba04069b20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fba180ad000 tx=0x7fba180a6000 comp rx=0 tx=0).stop 2024-03-24T00:35:18.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba2f000640 1 -- 172.21.15.40:0/725930845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba2813ec80 msgr2=0x7fba28140980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:18.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba2f000640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba2813ec80 0x7fba28140980 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fba1000c750 tx=0x7fba1000cc10 comp rx=0 tx=0).stop 2024-03-24T00:35:18.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba2f000640 1 -- 172.21.15.40:0/725930845 shutdown_connections 2024-03-24T00:35:18.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba2f000640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fba04067660 0x7fba04069b20 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba2f000640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba281419f0 0x7fba281557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba2f000640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fba2813ec80 0x7fba28140980 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba2f000640 1 --2- 172.21.15.40:0/725930845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba280ce800 0x7fba28158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:18.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba2f000640 1 -- 172.21.15.40:0/725930845 >> 172.21.15.40:0/725930845 conn(0x7fba280b17b0 msgr2=0x7fba28150b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:18.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba2f000640 1 -- 172.21.15.40:0/725930845 shutdown_connections 2024-03-24T00:35:18.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:18.946+0000 7fba2f000640 1 -- 172.21.15.40:0/725930845 wait complete. 2024-03-24T00:35:18.979 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:19.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5c00640 1 Processor -- start 2024-03-24T00:35:19.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5c00640 1 -- start start 2024-03-24T00:35:19.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e0159770 0x7fa5e00cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa5e0067590 0x7fa5e015fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e015d1c0 con 0x7fa5e006f280 2024-03-24T00:35:19.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e015c640 con 0x7fa5e0067590 2024-03-24T00:35:19.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e015c920 con 0x7fa5e0159770 2024-03-24T00:35:19.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa5e0067590 0x7fa5e015fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5dfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e0159770 0x7fa5e00cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa5e0067590 0x7fa5e015fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:33560/0 (socket says 172.21.15.40:33560) 2024-03-24T00:35:19.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5200640 1 -- 172.21.15.40:0/718126533 learned_addr learned my addr 172.21.15.40:0/718126533 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:19.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5200640 1 -- 172.21.15.40:0/718126533 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e0159770 msgr2=0x7fa5e00cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:19.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5200640 1 --2- 172.21.15.40:0/718126533 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e0159770 0x7fa5e00cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5200640 1 -- 172.21.15.40:0/718126533 >> v1:172.21.15.53:6789/0 conn(0x7fa5e006f280 legacy=0x7fa5e0066e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.118+0000 7fa5e5200640 1 -- 172.21.15.40:0/718126533 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e015cc00 con 0x7fa5e0067590 2024-03-24T00:35:19.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.122+0000 7fa5e5200640 1 --2- 172.21.15.40:0/718126533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e0067590 0x7fa5e015fb30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8009e50 tx=0x7fa5d801ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=344ca0a3545ad2b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:19.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.122+0000 7fa5dea00640 1 -- 172.21.15.40:0/718126533 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d80a5700 con 0x7fa5e0067590 2024-03-24T00:35:19.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.122+0000 7fa5dea00640 1 -- 172.21.15.40:0/718126533 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d80a58a0 con 0x7fa5e0067590 2024-03-24T00:35:19.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.122+0000 7fa5dea00640 1 -- 172.21.15.40:0/718126533 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d80a5c70 con 0x7fa5e0067590 2024-03-24T00:35:19.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.122+0000 7fa5e5c00640 1 -- 172.21.15.40:0/718126533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e0067590 msgr2=0x7fa5e015fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.122+0000 7fa5e5c00640 1 --2- 172.21.15.40:0/718126533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e0067590 0x7fa5e015fb30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8009e50 tx=0x7fa5d801ca30 comp rx=0 tx=0).stop 2024-03-24T00:35:19.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.122+0000 7fa5e5c00640 1 -- 172.21.15.40:0/718126533 shutdown_connections 2024-03-24T00:35:19.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.122+0000 7fa5e5c00640 1 --2- 172.21.15.40:0/718126533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e0067590 0x7fa5e015fb30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.122+0000 7fa5e5c00640 1 --2- 172.21.15.40:0/718126533 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e0159770 0x7fa5e00cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.122+0000 7fa5e5c00640 1 -- 172.21.15.40:0/718126533 >> 172.21.15.40:0/718126533 conn(0x7fa5e006b150 msgr2=0x7fa5e006b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:19.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.122+0000 7fa5e5c00640 1 -- 172.21.15.40:0/718126533 shutdown_connections 2024-03-24T00:35:19.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.122+0000 7fa5e5c00640 1 -- 172.21.15.40:0/718126533 wait complete. 2024-03-24T00:35:19.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5e5c00640 1 Processor -- start 2024-03-24T00:35:19.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5e5c00640 1 -- start start 2024-03-24T00:35:19.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5e5c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e0067590 0x7fa5e01fb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5e5c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e006f280 0x7fa5e0159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5e5c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e0159770 0x7fa5e01661b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5e5c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e015c370 con 0x7fa5e0067590 2024-03-24T00:35:19.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5e5c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e015c1f0 con 0x7fa5e0159770 2024-03-24T00:35:19.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5e5c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e0168b80 con 0x7fa5e006f280 2024-03-24T00:35:19.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5dfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e0067590 0x7fa5e01fb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5dfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e0067590 0x7fa5e01fb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54956/0 (socket says 172.21.15.40:54956) 2024-03-24T00:35:19.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5dfe00640 1 -- 172.21.15.40:0/71413503 learned_addr learned my addr 172.21.15.40:0/71413503 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:19.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5df400640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e006f280 0x7fa5e0159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5e5200640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e0159770 0x7fa5e01661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5dfe00640 1 -- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e006f280 msgr2=0x7fa5e0159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5dfe00640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e006f280 0x7fa5e0159df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5dfe00640 1 -- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e0159770 msgr2=0x7fa5e01661b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5dfe00640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e0159770 0x7fa5e01661b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5dfe00640 1 -- 172.21.15.40:0/71413503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e00b3990 con 0x7fa5e0067590 2024-03-24T00:35:19.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5dfe00640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e0067590 0x7fa5e01fb200 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fa5d000e870 tx=0x7fa5d000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:19.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5d7e00640 1 -- 172.21.15.40:0/71413503 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0017070 con 0x7fa5e0067590 2024-03-24T00:35:19.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5d7e00640 1 -- 172.21.15.40:0/71413503 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d000c940 con 0x7fa5e0067590 2024-03-24T00:35:19.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.126+0000 7fa5e5c00640 1 -- 172.21.15.40:0/71413503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e015a4f0 con 0x7fa5e0067590 2024-03-24T00:35:19.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.130+0000 7fa5d7e00640 1 -- 172.21.15.40:0/71413503 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d000ccf0 con 0x7fa5e0067590 2024-03-24T00:35:19.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.130+0000 7fa5e5c00640 1 -- 172.21.15.40:0/71413503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e015aab0 con 0x7fa5e0067590 2024-03-24T00:35:19.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.130+0000 7fa5e5c00640 1 -- 172.21.15.40:0/71413503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e015acf0 con 0x7fa5e0067590 2024-03-24T00:35:19.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.130+0000 7fa5d7e00640 1 -- 172.21.15.40:0/71413503 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa5d001a070 con 0x7fa5e0067590 2024-03-24T00:35:19.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.130+0000 7fa5d7e00640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5c0067660 0x7fa5c0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.130+0000 7fa5d7e00640 1 -- 172.21.15.40:0/71413503 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5d0085b10 con 0x7fa5e0067590 2024-03-24T00:35:19.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.130+0000 7fa5df400640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5c0067660 0x7fa5c0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.134+0000 7fa5df400640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5c0067660 0x7fa5c0069b20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa5e01fa750 tx=0x7fa5cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:19.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.134+0000 7fa5d7e00640 1 -- 172.21.15.40:0/71413503 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa5e00b3990 con 0x7fa5e0067590 2024-03-24T00:35:19.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.302+0000 7fa5e5c00640 1 -- 172.21.15.40:0/71413503 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5e00735b0 con 0x7fa5e0067590 2024-03-24T00:35:19.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.302+0000 7fa5d7e00640 1 -- 172.21.15.40:0/71413503 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16608 (secure 0 0 0) 0x7fa5d00560a0 con 0x7fa5e0067590 2024-03-24T00:35:19.307 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:19.307 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":70,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:35:16.298084+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:13.099776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:13.675426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:19.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.306+0000 7fa5e5c00640 1 -- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5c0067660 msgr2=0x7fa5c0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.306+0000 7fa5e5c00640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5c0067660 0x7fa5c0069b20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa5e01fa750 tx=0x7fa5cc009040 comp rx=0 tx=0).stop 2024-03-24T00:35:19.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.306+0000 7fa5e5c00640 1 -- 172.21.15.40:0/71413503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e0067590 msgr2=0x7fa5e01fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.306+0000 7fa5e5c00640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e0067590 0x7fa5e01fb200 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fa5d000e870 tx=0x7fa5d000ec70 comp rx=0 tx=0).stop 2024-03-24T00:35:19.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.306+0000 7fa5e5c00640 1 -- 172.21.15.40:0/71413503 shutdown_connections 2024-03-24T00:35:19.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.306+0000 7fa5e5c00640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5c0067660 0x7fa5c0069b20 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.306+0000 7fa5e5c00640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e0159770 0x7fa5e01661b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.306+0000 7fa5e5c00640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e006f280 0x7fa5e0159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.306+0000 7fa5e5c00640 1 --2- 172.21.15.40:0/71413503 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e0067590 0x7fa5e01fb200 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.306+0000 7fa5e5c00640 1 -- 172.21.15.40:0/71413503 >> 172.21.15.40:0/71413503 conn(0x7fa5e006b150 msgr2=0x7fa5e01550c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:19.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.306+0000 7fa5e5c00640 1 -- 172.21.15.40:0/71413503 shutdown_connections 2024-03-24T00:35:19.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.306+0000 7fa5e5c00640 1 -- 172.21.15.40:0/71413503 wait complete. 2024-03-24T00:35:19.332 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:35:19.332 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:35:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715f200640 1 Processor -- start 2024-03-24T00:35:19.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715f200640 1 -- start start 2024-03-24T00:35:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715f200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f715806e620 0x7f71580b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715f200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f71580ccfa0 0x7f71580cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715f200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7158070f60 con 0x7f715806ea00 2024-03-24T00:35:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715f200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71580b4c00 con 0x7f71580ccfa0 2024-03-24T00:35:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715f200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71580b3620 con 0x7f715806e620 2024-03-24T00:35:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715d200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f71580ccfa0 0x7f71580cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715d200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f71580ccfa0 0x7f71580cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54968/0 (socket says 172.21.15.40:54968) 2024-03-24T00:35:19.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715d200640 1 -- 172.21.15.40:0/739598661 learned_addr learned my addr 172.21.15.40:0/739598661 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f7157e00640 1 --2- 172.21.15.40:0/739598661 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f715806e620 0x7f71580b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715d200640 1 -- 172.21.15.40:0/739598661 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f715806e620 msgr2=0x7f71580b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715d200640 1 --2- 172.21.15.40:0/739598661 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f715806e620 0x7f71580b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715d200640 1 -- 172.21.15.40:0/739598661 >> v1:172.21.15.53:6789/0 conn(0x7f715806ea00 legacy=0x7f71580b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:19.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.470+0000 7f715d200640 1 -- 172.21.15.40:0/739598661 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71580b3900 con 0x7f71580ccfa0 2024-03-24T00:35:19.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.474+0000 7f715d200640 1 --2- 172.21.15.40:0/739598661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71580ccfa0 0x7f71580cd3f0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f7150009e50 tx=0x7f715001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55436d4a266de0b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.474+0000 7f7157400640 1 -- 172.21.15.40:0/739598661 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71500a5740 con 0x7f71580ccfa0 2024-03-24T00:35:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.474+0000 7f7157400640 1 -- 172.21.15.40:0/739598661 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71500a58e0 con 0x7f71580ccfa0 2024-03-24T00:35:19.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.474+0000 7f7157400640 1 -- 172.21.15.40:0/739598661 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71500a5cb0 con 0x7f71580ccfa0 2024-03-24T00:35:19.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.474+0000 7f715f200640 1 -- 172.21.15.40:0/739598661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71580ccfa0 msgr2=0x7f71580cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.474+0000 7f715f200640 1 --2- 172.21.15.40:0/739598661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71580ccfa0 0x7f71580cd3f0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f7150009e50 tx=0x7f715001c970 comp rx=0 tx=0).stop 2024-03-24T00:35:19.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.474+0000 7f715f200640 1 -- 172.21.15.40:0/739598661 shutdown_connections 2024-03-24T00:35:19.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.474+0000 7f715f200640 1 --2- 172.21.15.40:0/739598661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f71580ccfa0 0x7f71580cd3f0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.474+0000 7f715f200640 1 --2- 172.21.15.40:0/739598661 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f715806e620 0x7f71580b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.474+0000 7f715f200640 1 -- 172.21.15.40:0/739598661 >> 172.21.15.40:0/739598661 conn(0x7f71580b17b0 msgr2=0x7f71580b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:19.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.474+0000 7f715f200640 1 -- 172.21.15.40:0/739598661 shutdown_connections 2024-03-24T00:35:19.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.474+0000 7f715f200640 1 -- 172.21.15.40:0/739598661 wait complete. 2024-03-24T00:35:19.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f715f200640 1 Processor -- start 2024-03-24T00:35:19.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f715f200640 1 -- start start 2024-03-24T00:35:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f715f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f715806e620 0x7f7158158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f715f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f715806ea00 0x7f715815db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f715f200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71580ccfa0 0x7f715813adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f715f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7158144920 con 0x7f715806ea00 2024-03-24T00:35:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f715f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715806ede0 con 0x7f715806e620 2024-03-24T00:35:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f715f200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715813f6f0 con 0x7f71580ccfa0 2024-03-24T00:35:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f714fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f715806ea00 0x7f715815db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f714fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f715806ea00 0x7f715815db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:54980/0 (socket says 172.21.15.40:54980) 2024-03-24T00:35:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f714fe00640 1 -- 172.21.15.40:0/908817013 learned_addr learned my addr 172.21.15.40:0/908817013 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f715d200640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71580ccfa0 0x7f715813adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f7157e00640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f715806e620 0x7f7158158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f714fe00640 1 -- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71580ccfa0 msgr2=0x7f715813adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f714fe00640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71580ccfa0 0x7f715813adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f714fe00640 1 -- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f715806e620 msgr2=0x7f7158158fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f714fe00640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f715806e620 0x7f7158158fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f714fe00640 1 -- 172.21.15.40:0/908817013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715813f9d0 con 0x7f715806ea00 2024-03-24T00:35:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f714fe00640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f715806ea00 0x7f715815db70 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f714000c420 tx=0x7f714000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f7154c00640 1 -- 172.21.15.40:0/908817013 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71400052d0 con 0x7f715806ea00 2024-03-24T00:35:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f7154c00640 1 -- 172.21.15.40:0/908817013 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7140005470 con 0x7f715806ea00 2024-03-24T00:35:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f7154c00640 1 -- 172.21.15.40:0/908817013 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7140007260 con 0x7f715806ea00 2024-03-24T00:35:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f715f200640 1 -- 172.21.15.40:0/908817013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715813fcb0 con 0x7f715806ea00 2024-03-24T00:35:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.478+0000 7f715f200640 1 -- 172.21.15.40:0/908817013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71581594f0 con 0x7f715806ea00 2024-03-24T00:35:19.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.482+0000 7f715f200640 1 -- 172.21.15.40:0/908817013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7158068170 con 0x7f715806ea00 2024-03-24T00:35:19.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.482+0000 7f7154c00640 1 -- 172.21.15.40:0/908817013 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7140007400 con 0x7f715806ea00 2024-03-24T00:35:19.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.482+0000 7f7154c00640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7134067660 0x7f7134069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.482+0000 7f7154c00640 1 -- 172.21.15.40:0/908817013 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7140084d90 con 0x7f715806ea00 2024-03-24T00:35:19.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.482+0000 7f7157e00640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7134067660 0x7f7134069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.482+0000 7f7157e00640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7134067660 0x7f7134069b20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f71580af660 tx=0x7f7148008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:19.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.486+0000 7f7154c00640 1 -- 172.21.15.40:0/908817013 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7140055320 con 0x7f715806ea00 2024-03-24T00:35:19.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.646+0000 7f715f200640 1 -- 172.21.15.40:0/908817013 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7158072f50 con 0x7f7134067660 2024-03-24T00:35:19.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.654+0000 7f7154c00640 1 -- 172.21.15.40:0/908817013 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252069 (secure 0 0 0) 0x7f7158072f50 con 0x7f7134067660 2024-03-24T00:35:19.659 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:19.661 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:35:19.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.658+0000 7f715f200640 1 -- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7134067660 msgr2=0x7f7134069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.658+0000 7f715f200640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7134067660 0x7f7134069b20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f71580af660 tx=0x7f7148008040 comp rx=0 tx=0).stop 2024-03-24T00:35:19.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.658+0000 7f715f200640 1 -- 172.21.15.40:0/908817013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f715806ea00 msgr2=0x7f715815db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.658+0000 7f715f200640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f715806ea00 0x7f715815db70 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f714000c420 tx=0x7f714000c8e0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.658+0000 7f715f200640 1 -- 172.21.15.40:0/908817013 shutdown_connections 2024-03-24T00:35:19.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.658+0000 7f715f200640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7134067660 0x7f7134069b20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.658+0000 7f715f200640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71580ccfa0 0x7f715813adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.658+0000 7f715f200640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f715806ea00 0x7f715815db70 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.658+0000 7f715f200640 1 --2- 172.21.15.40:0/908817013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f715806e620 0x7f7158158fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.658+0000 7f715f200640 1 -- 172.21.15.40:0/908817013 >> 172.21.15.40:0/908817013 conn(0x7f71580b17b0 msgr2=0x7f715806dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:19.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.658+0000 7f715f200640 1 -- 172.21.15.40:0/908817013 shutdown_connections 2024-03-24T00:35:19.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.658+0000 7f715f200640 1 -- 172.21.15.40:0/908817013 wait complete. 2024-03-24T00:35:19.680 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-03-24T00:35:19.249611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":269816,"kb_used_data":7880,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754704904,"statfs":{"total":773094113280,"available":772817821696,"internally_reserved":0,"allocated":8069120,"data_stored":4098724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":268095026},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.971634"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309470+0000","last_change":"2024-03-24T00:35:15.307223+0000","last_active":"2024-03-24T00:35:16.309470+0000","last_peered":"2024-03-24T00:35:16.309470+0000","last_clean":"2024-03-24T00:35:16.309470+0000","last_became_active":"2024-03-24T00:35:15.306876+0000","last_became_peered":"2024-03-24T00:35:15.306876+0000","last_unstale":"2024-03-24T00:35:16.309470+0000","last_undegraded":"2024-03-24T00:35:16.309470+0000","last_fullsized":"2024-03-24T00:35:16.309470+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:03:28.550395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674867+0000","last_change":"2024-03-24T00:35:15.315880+0000","last_active":"2024-03-24T00:35:16.674867+0000","last_peered":"2024-03-24T00:35:16.674867+0000","last_clean":"2024-03-24T00:35:16.674867+0000","last_became_active":"2024-03-24T00:35:15.315578+0000","last_became_peered":"2024-03-24T00:35:15.315578+0000","last_unstale":"2024-03-24T00:35:16.674867+0000","last_undegraded":"2024-03-24T00:35:16.674867+0000","last_fullsized":"2024-03-24T00:35:16.674867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:19:15.121845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672485+0000","last_change":"2024-03-24T00:35:15.308742+0000","last_active":"2024-03-24T00:35:16.672485+0000","last_peered":"2024-03-24T00:35:16.672485+0000","last_clean":"2024-03-24T00:35:16.672485+0000","last_became_active":"2024-03-24T00:35:15.305351+0000","last_became_peered":"2024-03-24T00:35:15.305351+0000","last_unstale":"2024-03-24T00:35:16.672485+0000","last_undegraded":"2024-03-24T00:35:16.672485+0000","last_fullsized":"2024-03-24T00:35:16.672485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:01.930255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309276+0000","last_change":"2024-03-24T00:35:15.319557+0000","last_active":"2024-03-24T00:35:16.309276+0000","last_peered":"2024-03-24T00:35:16.309276+0000","last_clean":"2024-03-24T00:35:16.309276+0000","last_became_active":"2024-03-24T00:35:15.319234+0000","last_became_peered":"2024-03-24T00:35:15.319234+0000","last_unstale":"2024-03-24T00:35:16.309276+0000","last_undegraded":"2024-03-24T00:35:16.309276+0000","last_fullsized":"2024-03-24T00:35:16.309276+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:48.311977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672701+0000","last_change":"2024-03-24T00:35:15.303853+0000","last_active":"2024-03-24T00:35:16.672701+0000","last_peered":"2024-03-24T00:35:16.672701+0000","last_clean":"2024-03-24T00:35:16.672701+0000","last_became_active":"2024-03-24T00:35:15.303545+0000","last_became_peered":"2024-03-24T00:35:15.303545+0000","last_unstale":"2024-03-24T00:35:16.672701+0000","last_undegraded":"2024-03-24T00:35:16.672701+0000","last_fullsized":"2024-03-24T00:35:16.672701+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:50.684191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674802+0000","last_change":"2024-03-24T00:35:15.317700+0000","last_active":"2024-03-24T00:35:16.674802+0000","last_peered":"2024-03-24T00:35:16.674802+0000","last_clean":"2024-03-24T00:35:16.674802+0000","last_became_active":"2024-03-24T00:35:15.314044+0000","last_became_peered":"2024-03-24T00:35:15.314044+0000","last_unstale":"2024-03-24T00:35:16.674802+0000","last_undegraded":"2024-03-24T00:35:16.674802+0000","last_fullsized":"2024-03-24T00:35:16.674802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:06.464485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315291+0000","last_change":"2024-03-24T00:35:15.304952+0000","last_active":"2024-03-24T00:35:16.315291+0000","last_peered":"2024-03-24T00:35:16.315291+0000","last_clean":"2024-03-24T00:35:16.315291+0000","last_became_active":"2024-03-24T00:35:15.304668+0000","last_became_peered":"2024-03-24T00:35:15.304668+0000","last_unstale":"2024-03-24T00:35:16.315291+0000","last_undegraded":"2024-03-24T00:35:16.315291+0000","last_fullsized":"2024-03-24T00:35:16.315291+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:17.358722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674208+0000","last_change":"2024-03-24T00:35:15.316878+0000","last_active":"2024-03-24T00:35:16.674208+0000","last_peered":"2024-03-24T00:35:16.674208+0000","last_clean":"2024-03-24T00:35:16.674208+0000","last_became_active":"2024-03-24T00:35:15.314872+0000","last_became_peered":"2024-03-24T00:35:15.314872+0000","last_unstale":"2024-03-24T00:35:16.674208+0000","last_undegraded":"2024-03-24T00:35:16.674208+0000","last_fullsized":"2024-03-24T00:35:16.674208+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:17:26.079724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673297+0000","last_change":"2024-03-24T00:35:15.313509+0000","last_active":"2024-03-24T00:35:16.673297+0000","last_peered":"2024-03-24T00:35:16.673297+0000","last_clean":"2024-03-24T00:35:16.673297+0000","last_became_active":"2024-03-24T00:35:15.313255+0000","last_became_peered":"2024-03-24T00:35:15.313255+0000","last_unstale":"2024-03-24T00:35:16.673297+0000","last_undegraded":"2024-03-24T00:35:16.673297+0000","last_fullsized":"2024-03-24T00:35:16.673297+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:16.274846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310275+0000","last_change":"2024-03-24T00:35:15.305894+0000","last_active":"2024-03-24T00:35:16.310275+0000","last_peered":"2024-03-24T00:35:16.310275+0000","last_clean":"2024-03-24T00:35:16.310275+0000","last_became_active":"2024-03-24T00:35:15.305006+0000","last_became_peered":"2024-03-24T00:35:15.305006+0000","last_unstale":"2024-03-24T00:35:16.310275+0000","last_undegraded":"2024-03-24T00:35:16.310275+0000","last_fullsized":"2024-03-24T00:35:16.310275+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:50.245680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309336+0000","last_change":"2024-03-24T00:35:15.311331+0000","last_active":"2024-03-24T00:35:16.309336+0000","last_peered":"2024-03-24T00:35:16.309336+0000","last_clean":"2024-03-24T00:35:16.309336+0000","last_became_active":"2024-03-24T00:35:15.310929+0000","last_became_peered":"2024-03-24T00:35:15.310929+0000","last_unstale":"2024-03-24T00:35:16.309336+0000","last_undegraded":"2024-03-24T00:35:16.309336+0000","last_fullsized":"2024-03-24T00:35:16.309336+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:42.481496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309255+0000","last_change":"2024-03-24T00:35:15.301914+0000","last_active":"2024-03-24T00:35:16.309255+0000","last_peered":"2024-03-24T00:35:16.309255+0000","last_clean":"2024-03-24T00:35:16.309255+0000","last_became_active":"2024-03-24T00:35:15.301693+0000","last_became_peered":"2024-03-24T00:35:15.301693+0000","last_unstale":"2024-03-24T00:35:16.309255+0000","last_undegraded":"2024-03-24T00:35:16.309255+0000","last_fullsized":"2024-03-24T00:35:16.309255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:48:46.946260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317767+0000","last_change":"2024-03-24T00:35:15.321430+0000","last_active":"2024-03-24T00:35:16.317767+0000","last_peered":"2024-03-24T00:35:16.317767+0000","last_clean":"2024-03-24T00:35:16.317767+0000","last_became_active":"2024-03-24T00:35:15.321160+0000","last_became_peered":"2024-03-24T00:35:15.321160+0000","last_unstale":"2024-03-24T00:35:16.317767+0000","last_undegraded":"2024-03-24T00:35:16.317767+0000","last_fullsized":"2024-03-24T00:35:16.317767+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:29.341119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672864+0000","last_change":"2024-03-24T00:35:15.304930+0000","last_active":"2024-03-24T00:35:16.672864+0000","last_peered":"2024-03-24T00:35:16.672864+0000","last_clean":"2024-03-24T00:35:16.672864+0000","last_became_active":"2024-03-24T00:35:15.304110+0000","last_became_peered":"2024-03-24T00:35:15.304110+0000","last_unstale":"2024-03-24T00:35:16.672864+0000","last_undegraded":"2024-03-24T00:35:16.672864+0000","last_fullsized":"2024-03-24T00:35:16.672864+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:36.995688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317386+0000","last_change":"2024-03-24T00:35:15.310204+0000","last_active":"2024-03-24T00:35:16.317386+0000","last_peered":"2024-03-24T00:35:16.317386+0000","last_clean":"2024-03-24T00:35:16.317386+0000","last_became_active":"2024-03-24T00:35:15.309941+0000","last_became_peered":"2024-03-24T00:35:15.309941+0000","last_unstale":"2024-03-24T00:35:16.317386+0000","last_undegraded":"2024-03-24T00:35:16.317386+0000","last_fullsized":"2024-03-24T00:35:16.317386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:49:55.880879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.308884+0000","last_change":"2024-03-24T00:35:15.320547+0000","last_active":"2024-03-24T00:35:16.308884+0000","last_peered":"2024-03-24T00:35:16.308884+0000","last_clean":"2024-03-24T00:35:16.308884+0000","last_became_active":"2024-03-24T00:35:15.320252+0000","last_became_peered":"2024-03-24T00:35:15.320252+0000","last_unstale":"2024-03-24T00:35:16.308884+0000","last_undegraded":"2024-03-24T00:35:16.308884+0000","last_fullsized":"2024-03-24T00:35:16.308884+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:45.255626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673383+0000","last_change":"2024-03-24T00:35:15.302677+0000","last_active":"2024-03-24T00:35:16.673383+0000","last_peered":"2024-03-24T00:35:16.673383+0000","last_clean":"2024-03-24T00:35:16.673383+0000","last_became_active":"2024-03-24T00:35:15.302221+0000","last_became_peered":"2024-03-24T00:35:15.302221+0000","last_unstale":"2024-03-24T00:35:16.673383+0000","last_undegraded":"2024-03-24T00:35:16.673383+0000","last_fullsized":"2024-03-24T00:35:16.673383+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:35:44.727723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309134+0000","last_change":"2024-03-24T00:35:15.319921+0000","last_active":"2024-03-24T00:35:16.309134+0000","last_peered":"2024-03-24T00:35:16.309134+0000","last_clean":"2024-03-24T00:35:16.309134+0000","last_became_active":"2024-03-24T00:35:15.319449+0000","last_became_peered":"2024-03-24T00:35:15.319449+0000","last_unstale":"2024-03-24T00:35:16.309134+0000","last_undegraded":"2024-03-24T00:35:16.309134+0000","last_fullsized":"2024-03-24T00:35:16.309134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:39.876925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674520+0000","last_change":"2024-03-24T00:35:15.318292+0000","last_active":"2024-03-24T00:35:16.674520+0000","last_peered":"2024-03-24T00:35:16.674520+0000","last_clean":"2024-03-24T00:35:16.674520+0000","last_became_active":"2024-03-24T00:35:15.317893+0000","last_became_peered":"2024-03-24T00:35:15.317893+0000","last_unstale":"2024-03-24T00:35:16.674520+0000","last_undegraded":"2024-03-24T00:35:16.674520+0000","last_fullsized":"2024-03-24T00:35:16.674520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:30.057017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317448+0000","last_change":"2024-03-24T00:35:15.317651+0000","last_active":"2024-03-24T00:35:16.317448+0000","last_peered":"2024-03-24T00:35:16.317448+0000","last_clean":"2024-03-24T00:35:16.317448+0000","last_became_active":"2024-03-24T00:35:15.317310+0000","last_became_peered":"2024-03-24T00:35:15.317310+0000","last_unstale":"2024-03-24T00:35:16.317448+0000","last_undegraded":"2024-03-24T00:35:16.317448+0000","last_fullsized":"2024-03-24T00:35:16.317448+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:42.481496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310034+0000","last_change":"2024-03-24T00:35:15.306910+0000","last_active":"2024-03-24T00:35:16.310034+0000","last_peered":"2024-03-24T00:35:16.310034+0000","last_clean":"2024-03-24T00:35:16.310034+0000","last_became_active":"2024-03-24T00:35:15.306184+0000","last_became_peered":"2024-03-24T00:35:15.306184+0000","last_unstale":"2024-03-24T00:35:16.310034+0000","last_undegraded":"2024-03-24T00:35:16.310034+0000","last_fullsized":"2024-03-24T00:35:16.310034+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:39.295158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309925+0000","last_change":"2024-03-24T00:35:15.298126+0000","last_active":"2024-03-24T00:35:16.309925+0000","last_peered":"2024-03-24T00:35:16.309925+0000","last_clean":"2024-03-24T00:35:16.309925+0000","last_became_active":"2024-03-24T00:35:15.297811+0000","last_became_peered":"2024-03-24T00:35:15.297811+0000","last_unstale":"2024-03-24T00:35:16.309925+0000","last_undegraded":"2024-03-24T00:35:16.309925+0000","last_fullsized":"2024-03-24T00:35:16.309925+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:53.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309518+0000","last_change":"2024-03-24T00:35:15.312972+0000","last_active":"2024-03-24T00:35:16.309518+0000","last_peered":"2024-03-24T00:35:16.309518+0000","last_clean":"2024-03-24T00:35:16.309518+0000","last_became_active":"2024-03-24T00:35:15.312688+0000","last_became_peered":"2024-03-24T00:35:15.312688+0000","last_unstale":"2024-03-24T00:35:16.309518+0000","last_undegraded":"2024-03-24T00:35:16.309518+0000","last_fullsized":"2024-03-24T00:35:16.309518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:25:54.111013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309032+0000","last_change":"2024-03-24T00:35:15.319725+0000","last_active":"2024-03-24T00:35:16.309032+0000","last_peered":"2024-03-24T00:35:16.309032+0000","last_clean":"2024-03-24T00:35:16.309032+0000","last_became_active":"2024-03-24T00:35:15.319455+0000","last_became_peered":"2024-03-24T00:35:15.319455+0000","last_unstale":"2024-03-24T00:35:16.309032+0000","last_undegraded":"2024-03-24T00:35:16.309032+0000","last_fullsized":"2024-03-24T00:35:16.309032+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:49:04.935445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310794+0000","last_change":"2024-03-24T00:35:15.307844+0000","last_active":"2024-03-24T00:35:16.310794+0000","last_peered":"2024-03-24T00:35:16.310794+0000","last_clean":"2024-03-24T00:35:16.310794+0000","last_became_active":"2024-03-24T00:35:15.307562+0000","last_became_peered":"2024-03-24T00:35:15.307562+0000","last_unstale":"2024-03-24T00:35:16.310794+0000","last_undegraded":"2024-03-24T00:35:16.310794+0000","last_fullsized":"2024-03-24T00:35:16.310794+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:59.255013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315726+0000","last_change":"2024-03-24T00:35:15.321355+0000","last_active":"2024-03-24T00:35:16.315726+0000","last_peered":"2024-03-24T00:35:16.315726+0000","last_clean":"2024-03-24T00:35:16.315726+0000","last_became_active":"2024-03-24T00:35:15.320952+0000","last_became_peered":"2024-03-24T00:35:15.320952+0000","last_unstale":"2024-03-24T00:35:16.315726+0000","last_undegraded":"2024-03-24T00:35:16.315726+0000","last_fullsized":"2024-03-24T00:35:16.315726+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:37:58.309026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672623+0000","last_change":"2024-03-24T00:35:15.309026+0000","last_active":"2024-03-24T00:35:16.672623+0000","last_peered":"2024-03-24T00:35:16.672623+0000","last_clean":"2024-03-24T00:35:16.672623+0000","last_became_active":"2024-03-24T00:35:15.308451+0000","last_became_peered":"2024-03-24T00:35:15.308451+0000","last_unstale":"2024-03-24T00:35:16.672623+0000","last_undegraded":"2024-03-24T00:35:16.672623+0000","last_fullsized":"2024-03-24T00:35:16.672623+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:02:37.899211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310409+0000","last_change":"2024-03-24T00:35:15.299238+0000","last_active":"2024-03-24T00:35:16.310409+0000","last_peered":"2024-03-24T00:35:16.310409+0000","last_clean":"2024-03-24T00:35:16.310409+0000","last_became_active":"2024-03-24T00:35:15.298885+0000","last_became_peered":"2024-03-24T00:35:15.298885+0000","last_unstale":"2024-03-24T00:35:16.310409+0000","last_undegraded":"2024-03-24T00:35:16.310409+0000","last_fullsized":"2024-03-24T00:35:16.310409+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:17.358722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672447+0000","last_change":"2024-03-24T00:35:15.306518+0000","last_active":"2024-03-24T00:35:16.672447+0000","last_peered":"2024-03-24T00:35:16.672447+0000","last_clean":"2024-03-24T00:35:16.672447+0000","last_became_active":"2024-03-24T00:35:15.306250+0000","last_became_peered":"2024-03-24T00:35:15.306250+0000","last_unstale":"2024-03-24T00:35:16.672447+0000","last_undegraded":"2024-03-24T00:35:16.672447+0000","last_fullsized":"2024-03-24T00:35:16.672447+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:01:20.263015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315386+0000","last_change":"2024-03-24T00:35:15.322670+0000","last_active":"2024-03-24T00:35:16.315386+0000","last_peered":"2024-03-24T00:35:16.315386+0000","last_clean":"2024-03-24T00:35:16.315386+0000","last_became_active":"2024-03-24T00:35:15.322338+0000","last_became_peered":"2024-03-24T00:35:15.322338+0000","last_unstale":"2024-03-24T00:35:16.315386+0000","last_undegraded":"2024-03-24T00:35:16.315386+0000","last_fullsized":"2024-03-24T00:35:16.315386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:59.255013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315640+0000","last_change":"2024-03-24T00:35:15.304729+0000","last_active":"2024-03-24T00:35:16.315640+0000","last_peered":"2024-03-24T00:35:16.315640+0000","last_clean":"2024-03-24T00:35:16.315640+0000","last_became_active":"2024-03-24T00:35:15.304326+0000","last_became_peered":"2024-03-24T00:35:15.304326+0000","last_unstale":"2024-03-24T00:35:16.315640+0000","last_undegraded":"2024-03-24T00:35:16.315640+0000","last_fullsized":"2024-03-24T00:35:16.315640+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:13:46.464517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674279+0000","last_change":"2024-03-24T00:35:15.318362+0000","last_active":"2024-03-24T00:35:16.674279+0000","last_peered":"2024-03-24T00:35:16.674279+0000","last_clean":"2024-03-24T00:35:16.674279+0000","last_became_active":"2024-03-24T00:35:15.317329+0000","last_became_peered":"2024-03-24T00:35:15.317329+0000","last_unstale":"2024-03-24T00:35:16.674279+0000","last_undegraded":"2024-03-24T00:35:16.674279+0000","last_fullsized":"2024-03-24T00:35:16.674279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:29.049736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309906+0000","last_change":"2024-03-24T00:35:15.321846+0000","last_active":"2024-03-24T00:35:16.309906+0000","last_peered":"2024-03-24T00:35:16.309906+0000","last_clean":"2024-03-24T00:35:16.309906+0000","last_became_active":"2024-03-24T00:35:15.321483+0000","last_became_peered":"2024-03-24T00:35:15.321483+0000","last_unstale":"2024-03-24T00:35:16.309906+0000","last_undegraded":"2024-03-24T00:35:16.309906+0000","last_fullsized":"2024-03-24T00:35:16.309906+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:32:33.917825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"69'4","reported_seq":18,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309789+0000","last_change":"2024-03-24T00:35:14.306932+0000","last_active":"2024-03-24T00:35:16.309789+0000","last_peered":"2024-03-24T00:35:16.309789+0000","last_clean":"2024-03-24T00:35:16.309789+0000","last_became_active":"2024-03-24T00:35:14.306595+0000","last_became_peered":"2024-03-24T00:35:14.306595+0000","last_unstale":"2024-03-24T00:35:16.309789+0000","last_undegraded":"2024-03-24T00:35:16.309789+0000","last_fullsized":"2024-03-24T00:35:16.309789+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:38.289092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673438+0000","last_change":"2024-03-24T00:35:15.302471+0000","last_active":"2024-03-24T00:35:16.673438+0000","last_peered":"2024-03-24T00:35:16.673438+0000","last_clean":"2024-03-24T00:35:16.673438+0000","last_became_active":"2024-03-24T00:35:15.301859+0000","last_became_peered":"2024-03-24T00:35:15.301859+0000","last_unstale":"2024-03-24T00:35:16.673438+0000","last_undegraded":"2024-03-24T00:35:16.673438+0000","last_fullsized":"2024-03-24T00:35:16.673438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:11.400856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317620+0000","last_change":"2024-03-24T00:35:14.305894+0000","last_active":"2024-03-24T00:35:16.317620+0000","last_peered":"2024-03-24T00:35:16.317620+0000","last_clean":"2024-03-24T00:35:16.317620+0000","last_became_active":"2024-03-24T00:35:14.305623+0000","last_became_peered":"2024-03-24T00:35:14.305623+0000","last_unstale":"2024-03-24T00:35:16.317620+0000","last_undegraded":"2024-03-24T00:35:16.317620+0000","last_fullsized":"2024-03-24T00:35:16.317620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:16.352656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674842+0000","last_change":"2024-03-24T00:35:15.312109+0000","last_active":"2024-03-24T00:35:16.674842+0000","last_peered":"2024-03-24T00:35:16.674842+0000","last_clean":"2024-03-24T00:35:16.674842+0000","last_became_active":"2024-03-24T00:35:15.310930+0000","last_became_peered":"2024-03-24T00:35:15.310930+0000","last_unstale":"2024-03-24T00:35:16.674842+0000","last_undegraded":"2024-03-24T00:35:16.674842+0000","last_fullsized":"2024-03-24T00:35:16.674842+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:06:29.643031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310609+0000","last_change":"2024-03-24T00:35:14.307432+0000","last_active":"2024-03-24T00:35:16.310609+0000","last_peered":"2024-03-24T00:35:16.310609+0000","last_clean":"2024-03-24T00:35:16.310609+0000","last_became_active":"2024-03-24T00:35:14.307081+0000","last_became_peered":"2024-03-24T00:35:14.307081+0000","last_unstale":"2024-03-24T00:35:16.310609+0000","last_undegraded":"2024-03-24T00:35:16.310609+0000","last_fullsized":"2024-03-24T00:35:16.310609+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:17:25.073658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309502+0000","last_change":"2024-03-24T00:35:15.309412+0000","last_active":"2024-03-24T00:35:16.309502+0000","last_peered":"2024-03-24T00:35:16.309502+0000","last_clean":"2024-03-24T00:35:16.309502+0000","last_became_active":"2024-03-24T00:35:15.306563+0000","last_became_peered":"2024-03-24T00:35:15.306563+0000","last_unstale":"2024-03-24T00:35:16.309502+0000","last_undegraded":"2024-03-24T00:35:16.309502+0000","last_fullsized":"2024-03-24T00:35:16.309502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:29.341119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674762+0000","last_change":"2024-03-24T00:35:14.310422+0000","last_active":"2024-03-24T00:35:16.674762+0000","last_peered":"2024-03-24T00:35:16.674762+0000","last_clean":"2024-03-24T00:35:16.674762+0000","last_became_active":"2024-03-24T00:35:14.310121+0000","last_became_peered":"2024-03-24T00:35:14.310121+0000","last_unstale":"2024-03-24T00:35:16.674762+0000","last_undegraded":"2024-03-24T00:35:16.674762+0000","last_fullsized":"2024-03-24T00:35:16.674762+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:00.480161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315570+0000","last_change":"2024-03-24T00:35:15.315495+0000","last_active":"2024-03-24T00:35:16.315570+0000","last_peered":"2024-03-24T00:35:16.315570+0000","last_clean":"2024-03-24T00:35:16.315570+0000","last_became_active":"2024-03-24T00:35:15.315170+0000","last_became_peered":"2024-03-24T00:35:15.315170+0000","last_unstale":"2024-03-24T00:35:16.315570+0000","last_undegraded":"2024-03-24T00:35:16.315570+0000","last_fullsized":"2024-03-24T00:35:16.315570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:56:12.607356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672710+0000","last_change":"2024-03-24T00:35:14.302271+0000","last_active":"2024-03-24T00:35:16.672710+0000","last_peered":"2024-03-24T00:35:16.672710+0000","last_clean":"2024-03-24T00:35:16.672710+0000","last_became_active":"2024-03-24T00:35:14.301296+0000","last_became_peered":"2024-03-24T00:35:14.301296+0000","last_unstale":"2024-03-24T00:35:16.672710+0000","last_undegraded":"2024-03-24T00:35:16.672710+0000","last_fullsized":"2024-03-24T00:35:16.672710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:25:53.104947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310100+0000","last_change":"2024-03-24T00:35:15.308484+0000","last_active":"2024-03-24T00:35:16.310100+0000","last_peered":"2024-03-24T00:35:16.310100+0000","last_clean":"2024-03-24T00:35:16.310100+0000","last_became_active":"2024-03-24T00:35:15.308205+0000","last_became_peered":"2024-03-24T00:35:15.308205+0000","last_unstale":"2024-03-24T00:35:16.310100+0000","last_undegraded":"2024-03-24T00:35:16.310100+0000","last_fullsized":"2024-03-24T00:35:16.310100+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:15.706626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674290+0000","last_change":"2024-03-24T00:35:14.307471+0000","last_active":"2024-03-24T00:35:16.674290+0000","last_peered":"2024-03-24T00:35:16.674290+0000","last_clean":"2024-03-24T00:35:16.674290+0000","last_became_active":"2024-03-24T00:35:14.306751+0000","last_became_peered":"2024-03-24T00:35:14.306751+0000","last_unstale":"2024-03-24T00:35:16.674290+0000","last_undegraded":"2024-03-24T00:35:16.674290+0000","last_fullsized":"2024-03-24T00:35:16.674290+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:35:43.721657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672468+0000","last_change":"2024-03-24T00:35:15.296516+0000","last_active":"2024-03-24T00:35:16.672468+0000","last_peered":"2024-03-24T00:35:16.672468+0000","last_clean":"2024-03-24T00:35:16.672468+0000","last_became_active":"2024-03-24T00:35:15.296247+0000","last_became_peered":"2024-03-24T00:35:15.296247+0000","last_unstale":"2024-03-24T00:35:16.672468+0000","last_undegraded":"2024-03-24T00:35:16.672468+0000","last_fullsized":"2024-03-24T00:35:16.672468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:37.629888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.308764+0000","last_change":"2024-03-24T00:35:14.308598+0000","last_active":"2024-03-24T00:35:16.308764+0000","last_peered":"2024-03-24T00:35:16.308764+0000","last_clean":"2024-03-24T00:35:16.308764+0000","last_became_active":"2024-03-24T00:35:14.308273+0000","last_became_peered":"2024-03-24T00:35:14.308273+0000","last_unstale":"2024-03-24T00:35:16.308764+0000","last_undegraded":"2024-03-24T00:35:16.308764+0000","last_fullsized":"2024-03-24T00:35:16.308764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:58:36.065782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317443+0000","last_change":"2024-03-24T00:35:15.312374+0000","last_active":"2024-03-24T00:35:16.317443+0000","last_peered":"2024-03-24T00:35:16.317443+0000","last_clean":"2024-03-24T00:35:16.317443+0000","last_became_active":"2024-03-24T00:35:15.312014+0000","last_became_peered":"2024-03-24T00:35:15.312014+0000","last_unstale":"2024-03-24T00:35:16.317443+0000","last_undegraded":"2024-03-24T00:35:16.317443+0000","last_fullsized":"2024-03-24T00:35:16.317443+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:49:04.935445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672433+0000","last_change":"2024-03-24T00:35:14.303519+0000","last_active":"2024-03-24T00:35:16.672433+0000","last_peered":"2024-03-24T00:35:16.672433+0000","last_clean":"2024-03-24T00:35:16.672433+0000","last_became_active":"2024-03-24T00:35:14.303177+0000","last_became_peered":"2024-03-24T00:35:14.303177+0000","last_unstale":"2024-03-24T00:35:16.672433+0000","last_undegraded":"2024-03-24T00:35:16.672433+0000","last_fullsized":"2024-03-24T00:35:16.672433+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:01:59.262128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315743+0000","last_change":"2024-03-24T00:35:15.307721+0000","last_active":"2024-03-24T00:35:16.315743+0000","last_peered":"2024-03-24T00:35:16.315743+0000","last_clean":"2024-03-24T00:35:16.315743+0000","last_became_active":"2024-03-24T00:35:15.307434+0000","last_became_peered":"2024-03-24T00:35:15.307434+0000","last_unstale":"2024-03-24T00:35:16.315743+0000","last_undegraded":"2024-03-24T00:35:16.315743+0000","last_fullsized":"2024-03-24T00:35:16.315743+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:30.392706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310837+0000","last_change":"2024-03-24T00:35:14.302006+0000","last_active":"2024-03-24T00:35:16.310837+0000","last_peered":"2024-03-24T00:35:16.310837+0000","last_clean":"2024-03-24T00:35:16.310837+0000","last_became_active":"2024-03-24T00:35:14.301696+0000","last_became_peered":"2024-03-24T00:35:14.301696+0000","last_unstale":"2024-03-24T00:35:16.310837+0000","last_undegraded":"2024-03-24T00:35:16.310837+0000","last_fullsized":"2024-03-24T00:35:16.310837+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:57:38.831255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309638+0000","last_change":"2024-03-24T00:35:15.320670+0000","last_active":"2024-03-24T00:35:16.309638+0000","last_peered":"2024-03-24T00:35:16.309638+0000","last_clean":"2024-03-24T00:35:16.309638+0000","last_became_active":"2024-03-24T00:35:15.320389+0000","last_became_peered":"2024-03-24T00:35:15.320389+0000","last_unstale":"2024-03-24T00:35:16.309638+0000","last_undegraded":"2024-03-24T00:35:16.309638+0000","last_fullsized":"2024-03-24T00:35:16.309638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:10:50.773699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674915+0000","last_change":"2024-03-24T00:35:14.312360+0000","last_active":"2024-03-24T00:35:16.674915+0000","last_peered":"2024-03-24T00:35:16.674915+0000","last_clean":"2024-03-24T00:35:16.674915+0000","last_became_active":"2024-03-24T00:35:14.312011+0000","last_became_peered":"2024-03-24T00:35:14.312011+0000","last_unstale":"2024-03-24T00:35:16.674915+0000","last_undegraded":"2024-03-24T00:35:16.674915+0000","last_fullsized":"2024-03-24T00:35:16.674915+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:31.055987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674696+0000","last_change":"2024-03-24T00:35:15.313321+0000","last_active":"2024-03-24T00:35:16.674696+0000","last_peered":"2024-03-24T00:35:16.674696+0000","last_clean":"2024-03-24T00:35:16.674696+0000","last_became_active":"2024-03-24T00:35:15.310427+0000","last_became_peered":"2024-03-24T00:35:15.310427+0000","last_unstale":"2024-03-24T00:35:16.674696+0000","last_undegraded":"2024-03-24T00:35:16.674696+0000","last_fullsized":"2024-03-24T00:35:16.674696+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:52.402901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"69'3","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672729+0000","last_change":"2024-03-24T00:35:14.312312+0000","last_active":"2024-03-24T00:35:16.672729+0000","last_peered":"2024-03-24T00:35:16.672729+0000","last_clean":"2024-03-24T00:35:16.672729+0000","last_became_active":"2024-03-24T00:35:14.311575+0000","last_became_peered":"2024-03-24T00:35:14.311575+0000","last_unstale":"2024-03-24T00:35:16.672729+0000","last_undegraded":"2024-03-24T00:35:16.672729+0000","last_fullsized":"2024-03-24T00:35:16.672729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:10:49.767633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673122+0000","last_change":"2024-03-24T00:35:15.298413+0000","last_active":"2024-03-24T00:35:16.673122+0000","last_peered":"2024-03-24T00:35:16.673122+0000","last_clean":"2024-03-24T00:35:16.673122+0000","last_became_active":"2024-03-24T00:35:15.298069+0000","last_became_peered":"2024-03-24T00:35:15.298069+0000","last_unstale":"2024-03-24T00:35:16.673122+0000","last_undegraded":"2024-03-24T00:35:16.673122+0000","last_fullsized":"2024-03-24T00:35:16.673122+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:29.889339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"69'6","reported_seq":20,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309336+0000","last_change":"2024-03-24T00:35:14.307257+0000","last_active":"2024-03-24T00:35:16.309336+0000","last_peered":"2024-03-24T00:35:16.309336+0000","last_clean":"2024-03-24T00:35:16.309336+0000","last_became_active":"2024-03-24T00:35:14.306920+0000","last_became_peered":"2024-03-24T00:35:14.306920+0000","last_unstale":"2024-03-24T00:35:16.309336+0000","last_undegraded":"2024-03-24T00:35:16.309336+0000","last_fullsized":"2024-03-24T00:35:16.309336+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:00.480161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309007+0000","last_change":"2024-03-24T00:35:15.299865+0000","last_active":"2024-03-24T00:35:16.309007+0000","last_peered":"2024-03-24T00:35:16.309007+0000","last_clean":"2024-03-24T00:35:16.309007+0000","last_became_active":"2024-03-24T00:35:15.297759+0000","last_became_peered":"2024-03-24T00:35:15.297759+0000","last_unstale":"2024-03-24T00:35:16.309007+0000","last_undegraded":"2024-03-24T00:35:16.309007+0000","last_fullsized":"2024-03-24T00:35:16.309007+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:49:55.880879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309194+0000","last_change":"2024-03-24T00:35:14.302886+0000","last_active":"2024-03-24T00:35:16.309194+0000","last_peered":"2024-03-24T00:35:16.309194+0000","last_clean":"2024-03-24T00:35:16.309194+0000","last_became_active":"2024-03-24T00:35:14.300392+0000","last_became_peered":"2024-03-24T00:35:14.300392+0000","last_unstale":"2024-03-24T00:35:16.309194+0000","last_undegraded":"2024-03-24T00:35:16.309194+0000","last_fullsized":"2024-03-24T00:35:16.309194+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:22.279990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315477+0000","last_change":"2024-03-24T00:35:15.310176+0000","last_active":"2024-03-24T00:35:16.315477+0000","last_peered":"2024-03-24T00:35:16.315477+0000","last_clean":"2024-03-24T00:35:16.315477+0000","last_became_active":"2024-03-24T00:35:15.309857+0000","last_became_peered":"2024-03-24T00:35:15.309857+0000","last_unstale":"2024-03-24T00:35:16.315477+0000","last_undegraded":"2024-03-24T00:35:16.315477+0000","last_fullsized":"2024-03-24T00:35:16.315477+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:08.227783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674436+0000","last_change":"2024-03-24T00:35:14.304511+0000","last_active":"2024-03-24T00:35:16.674436+0000","last_peered":"2024-03-24T00:35:16.674436+0000","last_clean":"2024-03-24T00:35:16.674436+0000","last_became_active":"2024-03-24T00:35:14.304179+0000","last_became_peered":"2024-03-24T00:35:14.304179+0000","last_unstale":"2024-03-24T00:35:16.674436+0000","last_undegraded":"2024-03-24T00:35:16.674436+0000","last_fullsized":"2024-03-24T00:35:16.674436+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:10.394790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317539+0000","last_change":"2024-03-24T00:35:15.320538+0000","last_active":"2024-03-24T00:35:16.317539+0000","last_peered":"2024-03-24T00:35:16.317539+0000","last_clean":"2024-03-24T00:35:16.317539+0000","last_became_active":"2024-03-24T00:35:15.320266+0000","last_became_peered":"2024-03-24T00:35:15.320266+0000","last_unstale":"2024-03-24T00:35:16.317539+0000","last_undegraded":"2024-03-24T00:35:16.317539+0000","last_fullsized":"2024-03-24T00:35:16.317539+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:03:28.550395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315272+0000","last_change":"2024-03-24T00:35:14.316216+0000","last_active":"2024-03-24T00:35:16.315272+0000","last_peered":"2024-03-24T00:35:16.315272+0000","last_clean":"2024-03-24T00:35:16.315272+0000","last_became_active":"2024-03-24T00:35:14.314089+0000","last_became_peered":"2024-03-24T00:35:14.314089+0000","last_unstale":"2024-03-24T00:35:16.315272+0000","last_undegraded":"2024-03-24T00:35:16.315272+0000","last_fullsized":"2024-03-24T00:35:16.315272+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:05.458419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672844+0000","last_change":"2024-03-24T00:35:15.301366+0000","last_active":"2024-03-24T00:35:16.672844+0000","last_peered":"2024-03-24T00:35:16.672844+0000","last_clean":"2024-03-24T00:35:16.672844+0000","last_became_active":"2024-03-24T00:35:15.301022+0000","last_became_peered":"2024-03-24T00:35:15.301022+0000","last_unstale":"2024-03-24T00:35:16.672844+0000","last_undegraded":"2024-03-24T00:35:16.672844+0000","last_fullsized":"2024-03-24T00:35:16.672844+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:46.498885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"69'8","reported_seq":22,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.308950+0000","last_change":"2024-03-24T00:35:14.309496+0000","last_active":"2024-03-24T00:35:16.308950+0000","last_peered":"2024-03-24T00:35:16.308950+0000","last_clean":"2024-03-24T00:35:16.308950+0000","last_became_active":"2024-03-24T00:35:14.305359+0000","last_became_peered":"2024-03-24T00:35:14.305359+0000","last_unstale":"2024-03-24T00:35:16.308950+0000","last_undegraded":"2024-03-24T00:35:16.308950+0000","last_fullsized":"2024-03-24T00:35:16.308950+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:48:45.940194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310386+0000","last_change":"2024-03-24T00:35:15.303098+0000","last_active":"2024-03-24T00:35:16.310386+0000","last_peered":"2024-03-24T00:35:16.310386+0000","last_clean":"2024-03-24T00:35:16.310386+0000","last_became_active":"2024-03-24T00:35:15.302783+0000","last_became_peered":"2024-03-24T00:35:15.302783+0000","last_unstale":"2024-03-24T00:35:16.310386+0000","last_undegraded":"2024-03-24T00:35:16.310386+0000","last_fullsized":"2024-03-24T00:35:16.310386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:45.432659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674498+0000","last_change":"2024-03-24T00:35:14.305678+0000","last_active":"2024-03-24T00:35:16.674498+0000","last_peered":"2024-03-24T00:35:16.674498+0000","last_clean":"2024-03-24T00:35:16.674498+0000","last_became_active":"2024-03-24T00:35:14.305363+0000","last_became_peered":"2024-03-24T00:35:14.305363+0000","last_unstale":"2024-03-24T00:35:16.674498+0000","last_undegraded":"2024-03-24T00:35:16.674498+0000","last_fullsized":"2024-03-24T00:35:16.674498+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:46.471632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309740+0000","last_change":"2024-03-24T00:35:15.307318+0000","last_active":"2024-03-24T00:35:16.309740+0000","last_peered":"2024-03-24T00:35:16.309740+0000","last_clean":"2024-03-24T00:35:16.309740+0000","last_became_active":"2024-03-24T00:35:15.307044+0000","last_became_peered":"2024-03-24T00:35:15.307044+0000","last_unstale":"2024-03-24T00:35:16.309740+0000","last_undegraded":"2024-03-24T00:35:16.309740+0000","last_fullsized":"2024-03-24T00:35:16.309740+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:09.891306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310338+0000","last_change":"2024-03-24T00:35:14.301078+0000","last_active":"2024-03-24T00:35:16.310338+0000","last_peered":"2024-03-24T00:35:16.310338+0000","last_clean":"2024-03-24T00:35:16.310338+0000","last_became_active":"2024-03-24T00:35:14.300735+0000","last_became_peered":"2024-03-24T00:35:14.300735+0000","last_unstale":"2024-03-24T00:35:16.310338+0000","last_undegraded":"2024-03-24T00:35:16.310338+0000","last_fullsized":"2024-03-24T00:35:16.310338+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:00.130300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309338+0000","last_change":"2024-03-24T00:35:15.300688+0000","last_active":"2024-03-24T00:35:16.309338+0000","last_peered":"2024-03-24T00:35:16.309338+0000","last_clean":"2024-03-24T00:35:16.309338+0000","last_became_active":"2024-03-24T00:35:15.299409+0000","last_became_peered":"2024-03-24T00:35:15.299409+0000","last_unstale":"2024-03-24T00:35:16.309338+0000","last_undegraded":"2024-03-24T00:35:16.309338+0000","last_fullsized":"2024-03-24T00:35:16.309338+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:57:39.837321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674882+0000","last_change":"2024-03-24T00:35:14.315167+0000","last_active":"2024-03-24T00:35:16.674882+0000","last_peered":"2024-03-24T00:35:16.674882+0000","last_clean":"2024-03-24T00:35:16.674882+0000","last_became_active":"2024-03-24T00:35:14.314848+0000","last_became_peered":"2024-03-24T00:35:14.314848+0000","last_unstale":"2024-03-24T00:35:16.674882+0000","last_undegraded":"2024-03-24T00:35:16.674882+0000","last_fullsized":"2024-03-24T00:35:16.674882+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:03.351920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673262+0000","last_change":"2024-03-24T00:35:15.309311+0000","last_active":"2024-03-24T00:35:16.673262+0000","last_peered":"2024-03-24T00:35:16.673262+0000","last_clean":"2024-03-24T00:35:16.673262+0000","last_became_active":"2024-03-24T00:35:15.309005+0000","last_became_peered":"2024-03-24T00:35:15.309005+0000","last_unstale":"2024-03-24T00:35:16.673262+0000","last_undegraded":"2024-03-24T00:35:16.673262+0000","last_fullsized":"2024-03-24T00:35:16.673262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:54:52.276445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309548+0000","last_change":"2024-03-24T00:35:14.313483+0000","last_active":"2024-03-24T00:35:16.309548+0000","last_peered":"2024-03-24T00:35:16.309548+0000","last_clean":"2024-03-24T00:35:16.309548+0000","last_became_active":"2024-03-24T00:35:14.311301+0000","last_became_peered":"2024-03-24T00:35:14.311301+0000","last_unstale":"2024-03-24T00:35:16.309548+0000","last_undegraded":"2024-03-24T00:35:16.309548+0000","last_fullsized":"2024-03-24T00:35:16.309548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:52.115501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672604+0000","last_change":"2024-03-24T00:35:15.306801+0000","last_active":"2024-03-24T00:35:16.672604+0000","last_peered":"2024-03-24T00:35:16.672604+0000","last_clean":"2024-03-24T00:35:16.672604+0000","last_became_active":"2024-03-24T00:35:15.306493+0000","last_became_peered":"2024-03-24T00:35:15.306493+0000","last_unstale":"2024-03-24T00:35:16.672604+0000","last_undegraded":"2024-03-24T00:35:16.672604+0000","last_fullsized":"2024-03-24T00:35:16.672604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:59:21.692918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674588+0000","last_change":"2024-03-24T00:35:14.307612+0000","last_active":"2024-03-24T00:35:16.674588+0000","last_peered":"2024-03-24T00:35:16.674588+0000","last_clean":"2024-03-24T00:35:16.674588+0000","last_became_active":"2024-03-24T00:35:14.307103+0000","last_became_peered":"2024-03-24T00:35:14.307103+0000","last_unstale":"2024-03-24T00:35:16.674588+0000","last_undegraded":"2024-03-24T00:35:16.674588+0000","last_fullsized":"2024-03-24T00:35:16.674588+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:20:19.408158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672420+0000","last_change":"2024-03-24T00:35:15.314657+0000","last_active":"2024-03-24T00:35:16.672420+0000","last_peered":"2024-03-24T00:35:16.672420+0000","last_clean":"2024-03-24T00:35:16.672420+0000","last_became_active":"2024-03-24T00:35:15.314388+0000","last_became_peered":"2024-03-24T00:35:15.314388+0000","last_unstale":"2024-03-24T00:35:16.672420+0000","last_undegraded":"2024-03-24T00:35:16.672420+0000","last_fullsized":"2024-03-24T00:35:16.672420+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:25.099400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315552+0000","last_change":"2024-03-24T00:35:14.315703+0000","last_active":"2024-03-24T00:35:16.315552+0000","last_peered":"2024-03-24T00:35:16.315552+0000","last_clean":"2024-03-24T00:35:16.315552+0000","last_became_active":"2024-03-24T00:35:14.314862+0000","last_became_peered":"2024-03-24T00:35:14.314862+0000","last_unstale":"2024-03-24T00:35:16.315552+0000","last_undegraded":"2024-03-24T00:35:16.315552+0000","last_fullsized":"2024-03-24T00:35:16.315552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:06:28.636965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672843+0000","last_change":"2024-03-24T00:35:15.315799+0000","last_active":"2024-03-24T00:35:16.672843+0000","last_peered":"2024-03-24T00:35:16.672843+0000","last_clean":"2024-03-24T00:35:16.672843+0000","last_became_active":"2024-03-24T00:35:15.315512+0000","last_became_peered":"2024-03-24T00:35:15.315512+0000","last_unstale":"2024-03-24T00:35:16.672843+0000","last_undegraded":"2024-03-24T00:35:16.672843+0000","last_fullsized":"2024-03-24T00:35:16.672843+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:08:21.273790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317323+0000","last_change":"2024-03-24T00:35:14.306441+0000","last_active":"2024-03-24T00:35:16.317323+0000","last_peered":"2024-03-24T00:35:16.317323+0000","last_clean":"2024-03-24T00:35:16.317323+0000","last_became_active":"2024-03-24T00:35:14.306136+0000","last_became_peered":"2024-03-24T00:35:14.306136+0000","last_unstale":"2024-03-24T00:35:16.317323+0000","last_undegraded":"2024-03-24T00:35:16.317323+0000","last_fullsized":"2024-03-24T00:35:16.317323+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:07.221717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317637+0000","last_change":"2024-03-24T00:35:15.309020+0000","last_active":"2024-03-24T00:35:16.317637+0000","last_peered":"2024-03-24T00:35:16.317637+0000","last_clean":"2024-03-24T00:35:16.317637+0000","last_became_active":"2024-03-24T00:35:15.308668+0000","last_became_peered":"2024-03-24T00:35:15.308668+0000","last_unstale":"2024-03-24T00:35:16.317637+0000","last_undegraded":"2024-03-24T00:35:16.317637+0000","last_fullsized":"2024-03-24T00:35:16.317637+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:28:38.970698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674548+0000","last_change":"2024-03-24T00:35:14.304572+0000","last_active":"2024-03-24T00:35:16.674548+0000","last_peered":"2024-03-24T00:35:16.674548+0000","last_clean":"2024-03-24T00:35:16.674548+0000","last_became_active":"2024-03-24T00:35:14.304230+0000","last_became_peered":"2024-03-24T00:35:14.304230+0000","last_unstale":"2024-03-24T00:35:16.674548+0000","last_undegraded":"2024-03-24T00:35:16.674548+0000","last_fullsized":"2024-03-24T00:35:16.674548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:31.842925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315785+0000","last_change":"2024-03-24T00:35:15.308983+0000","last_active":"2024-03-24T00:35:16.315785+0000","last_peered":"2024-03-24T00:35:16.315785+0000","last_clean":"2024-03-24T00:35:16.315785+0000","last_became_active":"2024-03-24T00:35:15.308707+0000","last_became_peered":"2024-03-24T00:35:15.308707+0000","last_unstale":"2024-03-24T00:35:16.315785+0000","last_undegraded":"2024-03-24T00:35:16.315785+0000","last_fullsized":"2024-03-24T00:35:16.315785+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:19.665491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"70'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317724+0000","last_change":"2024-03-24T00:35:14.314282+0000","last_active":"2024-03-24T00:35:16.317724+0000","last_peered":"2024-03-24T00:35:16.317724+0000","last_clean":"2024-03-24T00:35:16.317724+0000","last_became_active":"2024-03-24T00:35:14.312772+0000","last_became_peered":"2024-03-24T00:35:14.312772+0000","last_unstale":"2024-03-24T00:35:16.317724+0000","last_undegraded":"2024-03-24T00:35:16.317724+0000","last_fullsized":"2024-03-24T00:35:16.317724+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:51.396835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309484+0000","last_change":"2024-03-24T00:35:15.320643+0000","last_active":"2024-03-24T00:35:16.309484+0000","last_peered":"2024-03-24T00:35:16.309484+0000","last_clean":"2024-03-24T00:35:16.309484+0000","last_became_active":"2024-03-24T00:35:15.320317+0000","last_became_peered":"2024-03-24T00:35:15.320317+0000","last_unstale":"2024-03-24T00:35:16.309484+0000","last_undegraded":"2024-03-24T00:35:16.309484+0000","last_fullsized":"2024-03-24T00:35:16.309484+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:28:38.970698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673353+0000","last_change":"2024-03-24T00:35:14.306286+0000","last_active":"2024-03-24T00:35:16.673353+0000","last_peered":"2024-03-24T00:35:16.673353+0000","last_clean":"2024-03-24T00:35:16.673353+0000","last_became_active":"2024-03-24T00:35:14.305944+0000","last_became_peered":"2024-03-24T00:35:14.305944+0000","last_unstale":"2024-03-24T00:35:16.673353+0000","last_undegraded":"2024-03-24T00:35:16.673353+0000","last_fullsized":"2024-03-24T00:35:16.673353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:20.973520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309665+0000","last_change":"2024-03-24T00:35:15.303121+0000","last_active":"2024-03-24T00:35:16.309665+0000","last_peered":"2024-03-24T00:35:16.309665+0000","last_clean":"2024-03-24T00:35:16.309665+0000","last_became_active":"2024-03-24T00:35:15.302847+0000","last_became_peered":"2024-03-24T00:35:15.302847+0000","last_unstale":"2024-03-24T00:35:16.309665+0000","last_undegraded":"2024-03-24T00:35:16.309665+0000","last_fullsized":"2024-03-24T00:35:16.309665+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:29:44.114132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317535+0000","last_change":"2024-03-24T00:35:14.305600+0000","last_active":"2024-03-24T00:35:16.317535+0000","last_peered":"2024-03-24T00:35:16.317535+0000","last_clean":"2024-03-24T00:35:16.317535+0000","last_became_active":"2024-03-24T00:35:14.305305+0000","last_became_peered":"2024-03-24T00:35:14.305305+0000","last_unstale":"2024-03-24T00:35:16.317535+0000","last_undegraded":"2024-03-24T00:35:16.317535+0000","last_fullsized":"2024-03-24T00:35:16.317535+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:58.248947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673012+0000","last_change":"2024-03-24T00:35:15.304530+0000","last_active":"2024-03-24T00:35:16.673012+0000","last_peered":"2024-03-24T00:35:16.673012+0000","last_clean":"2024-03-24T00:35:16.673012+0000","last_became_active":"2024-03-24T00:35:15.303763+0000","last_became_peered":"2024-03-24T00:35:15.303763+0000","last_unstale":"2024-03-24T00:35:16.673012+0000","last_undegraded":"2024-03-24T00:35:16.673012+0000","last_fullsized":"2024-03-24T00:35:16.673012+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:38:45.109294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672859+0000","last_change":"2024-03-24T00:35:14.311840+0000","last_active":"2024-03-24T00:35:16.672859+0000","last_peered":"2024-03-24T00:35:16.672859+0000","last_clean":"2024-03-24T00:35:16.672859+0000","last_became_active":"2024-03-24T00:35:14.311512+0000","last_became_peered":"2024-03-24T00:35:14.311512+0000","last_unstale":"2024-03-24T00:35:16.672859+0000","last_undegraded":"2024-03-24T00:35:16.672859+0000","last_fullsized":"2024-03-24T00:35:16.672859+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:41.761662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674641+0000","last_change":"2024-03-24T00:35:15.313450+0000","last_active":"2024-03-24T00:35:16.674641+0000","last_peered":"2024-03-24T00:35:16.674641+0000","last_clean":"2024-03-24T00:35:16.674641+0000","last_became_active":"2024-03-24T00:35:15.312976+0000","last_became_peered":"2024-03-24T00:35:15.312976+0000","last_unstale":"2024-03-24T00:35:16.674641+0000","last_undegraded":"2024-03-24T00:35:16.674641+0000","last_fullsized":"2024-03-24T00:35:16.674641+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:57.420849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.308763+0000","last_change":"2024-03-24T00:35:14.302928+0000","last_active":"2024-03-24T00:35:16.308763+0000","last_peered":"2024-03-24T00:35:16.308763+0000","last_clean":"2024-03-24T00:35:16.308763+0000","last_became_active":"2024-03-24T00:35:14.301283+0000","last_became_peered":"2024-03-24T00:35:14.301283+0000","last_unstale":"2024-03-24T00:35:16.308763+0000","last_undegraded":"2024-03-24T00:35:16.308763+0000","last_fullsized":"2024-03-24T00:35:16.308763+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:21:08.519311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309323+0000","last_change":"2024-03-24T00:35:15.307308+0000","last_active":"2024-03-24T00:35:16.309323+0000","last_peered":"2024-03-24T00:35:16.309323+0000","last_clean":"2024-03-24T00:35:16.309323+0000","last_became_active":"2024-03-24T00:35:15.307017+0000","last_became_peered":"2024-03-24T00:35:15.307017+0000","last_unstale":"2024-03-24T00:35:16.309323+0000","last_undegraded":"2024-03-24T00:35:16.309323+0000","last_fullsized":"2024-03-24T00:35:16.309323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:01.136366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.308816+0000","last_change":"2024-03-24T00:35:14.300683+0000","last_active":"2024-03-24T00:35:16.308816+0000","last_peered":"2024-03-24T00:35:16.308816+0000","last_clean":"2024-03-24T00:35:16.308816+0000","last_became_active":"2024-03-24T00:35:14.300308+0000","last_became_peered":"2024-03-24T00:35:14.300308+0000","last_unstale":"2024-03-24T00:35:16.308816+0000","last_undegraded":"2024-03-24T00:35:16.308816+0000","last_fullsized":"2024-03-24T00:35:16.308816+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:47.305911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674269+0000","last_change":"2024-03-24T00:35:15.316367+0000","last_active":"2024-03-24T00:35:16.674269+0000","last_peered":"2024-03-24T00:35:16.674269+0000","last_clean":"2024-03-24T00:35:16.674269+0000","last_became_active":"2024-03-24T00:35:15.316100+0000","last_became_peered":"2024-03-24T00:35:15.316100+0000","last_unstale":"2024-03-24T00:35:16.674269+0000","last_undegraded":"2024-03-24T00:35:16.674269+0000","last_fullsized":"2024-03-24T00:35:16.674269+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:57:39.837321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674349+0000","last_change":"2024-03-24T00:35:14.310352+0000","last_active":"2024-03-24T00:35:16.674349+0000","last_peered":"2024-03-24T00:35:16.674349+0000","last_clean":"2024-03-24T00:35:16.674349+0000","last_became_active":"2024-03-24T00:35:14.309943+0000","last_became_peered":"2024-03-24T00:35:14.309943+0000","last_unstale":"2024-03-24T00:35:16.674349+0000","last_undegraded":"2024-03-24T00:35:16.674349+0000","last_fullsized":"2024-03-24T00:35:16.674349+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:15.268780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672586+0000","last_change":"2024-03-24T00:35:15.318636+0000","last_active":"2024-03-24T00:35:16.672586+0000","last_peered":"2024-03-24T00:35:16.672586+0000","last_clean":"2024-03-24T00:35:16.672586+0000","last_became_active":"2024-03-24T00:35:15.318385+0000","last_became_peered":"2024-03-24T00:35:15.318385+0000","last_unstale":"2024-03-24T00:35:16.672586+0000","last_undegraded":"2024-03-24T00:35:16.672586+0000","last_fullsized":"2024-03-24T00:35:16.672586+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:50.117855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674448+0000","last_change":"2024-03-24T00:35:14.310608+0000","last_active":"2024-03-24T00:35:16.674448+0000","last_peered":"2024-03-24T00:35:16.674448+0000","last_clean":"2024-03-24T00:35:16.674448+0000","last_became_active":"2024-03-24T00:35:14.309037+0000","last_became_peered":"2024-03-24T00:35:14.309037+0000","last_unstale":"2024-03-24T00:35:16.674448+0000","last_undegraded":"2024-03-24T00:35:16.674448+0000","last_fullsized":"2024-03-24T00:35:16.674448+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:22.279990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":272,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309099+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:35:16.309099+0000","last_peered":"2024-03-24T00:35:16.309099+0000","last_clean":"2024-03-24T00:35:16.309099+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:35:16.309099+0000","last_undegraded":"2024-03-24T00:35:16.309099+0000","last_fullsized":"2024-03-24T00:35:16.309099+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32600,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339240,"statfs":{"total":96636764160,"available":96603381760,"internally_reserved":0,"allocated":1454080,"data_stored":955130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":364238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574904,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":856064,"data_stored":364236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574904,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40340,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331500,"statfs":{"total":96636764160,"available":96595456000,"internally_reserved":0,"allocated":856064,"data_stored":364216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":860160,"data_stored":364258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":864256,"data_stored":365758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":872448,"data_stored":366374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36756,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335084,"statfs":{"total":96636764160,"available":96599126016,"internally_reserved":0,"allocated":1445888,"data_stored":954514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:35:19.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:35:19.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.858+0000 7f623d200640 1 Processor -- start 2024-03-24T00:35:19.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.858+0000 7f623d200640 1 -- start start 2024-03-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f623d200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f623806f100 0x7f6238070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f623d200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f623806f4e0 0x7f6238070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f623d200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62380d27b0 con 0x7f62380aea60 2024-03-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f623d200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62380b1490 con 0x7f623806f4e0 2024-03-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f623d200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f623806fa70 con 0x7f623806f100 2024-03-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f622fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f623806f4e0 0x7f6238070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f6236800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f62380aea60 0x7f62380aee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36858/0 (socket says 172.21.15.40:36858) 2024-03-24T00:35:19.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f6236800640 1 -- 172.21.15.40:0/3014858716 learned_addr learned my addr 172.21.15.40:0/3014858716 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f6235e00640 1 --2- 172.21.15.40:0/3014858716 >> v2:172.21.15.53:3300/0 conn(0x7f623806f100 0x7f6238070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f622fe00640 1 -- 172.21.15.40:0/3014858716 >> v2:172.21.15.53:3300/0 conn(0x7f623806f100 msgr2=0x7f6238070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f622fe00640 1 --2- 172.21.15.40:0/3014858716 >> v2:172.21.15.53:3300/0 conn(0x7f623806f100 0x7f6238070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f622fe00640 1 -- 172.21.15.40:0/3014858716 >> v1:172.21.15.40:6789/0 conn(0x7f62380aea60 legacy=0x7f62380aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.862+0000 7f622fe00640 1 -- 172.21.15.40:0/3014858716 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623806fd50 con 0x7f623806f4e0 2024-03-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.866+0000 7f622fe00640 1 --2- 172.21.15.40:0/3014858716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623806f4e0 0x7f6238070030 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6238071870 tx=0x7f622001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a246c2ea1e34c949 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:19.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.866+0000 7f6235400640 1 -- 172.21.15.40:0/3014858716 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62200b0070 con 0x7f623806f4e0 2024-03-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.866+0000 7f6235400640 1 -- 172.21.15.40:0/3014858716 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62200ab600 con 0x7f623806f4e0 2024-03-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.866+0000 7f6235400640 1 -- 172.21.15.40:0/3014858716 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62200ab9d0 con 0x7f623806f4e0 2024-03-24T00:35:19.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.866+0000 7f623d200640 1 -- 172.21.15.40:0/3014858716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623806f4e0 msgr2=0x7f6238070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.866+0000 7f623d200640 1 --2- 172.21.15.40:0/3014858716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623806f4e0 0x7f6238070030 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6238071870 tx=0x7f622001c480 comp rx=0 tx=0).stop 2024-03-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.866+0000 7f623d200640 1 -- 172.21.15.40:0/3014858716 shutdown_connections 2024-03-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.866+0000 7f623d200640 1 --2- 172.21.15.40:0/3014858716 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623806f4e0 0x7f6238070030 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.866+0000 7f623d200640 1 --2- 172.21.15.40:0/3014858716 >> v2:172.21.15.53:3300/0 conn(0x7f623806f100 0x7f6238070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.866+0000 7f623d200640 1 -- 172.21.15.40:0/3014858716 >> 172.21.15.40:0/3014858716 conn(0x7f62380d1c90 msgr2=0x7f62380d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.866+0000 7f623d200640 1 -- 172.21.15.40:0/3014858716 shutdown_connections 2024-03-24T00:35:19.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.866+0000 7f623d200640 1 -- 172.21.15.40:0/3014858716 wait complete. 2024-03-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f623d200640 1 Processor -- start 2024-03-24T00:35:19.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f623d200640 1 -- start start 2024-03-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f623d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f623806f100 0x7f623815e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f623d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623806f4e0 0x7f623809f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f623d200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62380aea60 0x7f62381fa2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f623d200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62380b0e10 con 0x7f623806f4e0 2024-03-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f623d200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62380b0c90 con 0x7f623806f100 2024-03-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f623d200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62380b0f90 con 0x7f62380aea60 2024-03-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f622fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623806f4e0 0x7f623809f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f6235e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f623806f100 0x7f623815e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f622fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623806f4e0 0x7f623809f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55000/0 (socket says 172.21.15.40:55000) 2024-03-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f622fe00640 1 -- 172.21.15.40:0/387085457 learned_addr learned my addr 172.21.15.40:0/387085457 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f6236800640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62380aea60 0x7f62381fa2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f622fe00640 1 -- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62380aea60 msgr2=0x7f62381fa2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f622fe00640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62380aea60 0x7f62381fa2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f622fe00640 1 -- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f623806f100 msgr2=0x7f623815e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f622fe00640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f623806f100 0x7f623815e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:19.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f622fe00640 1 -- 172.21.15.40:0/387085457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62380c86e0 con 0x7f623806f4e0 2024-03-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f622fe00640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623806f4e0 0x7f623809f4c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f62200ae040 tx=0x7f6220019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.870+0000 7f622e000640 1 -- 172.21.15.40:0/387085457 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62200b0040 con 0x7f623806f4e0 2024-03-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.874+0000 7f622e000640 1 -- 172.21.15.40:0/387085457 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62200b9070 con 0x7f623806f4e0 2024-03-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.874+0000 7f622e000640 1 -- 172.21.15.40:0/387085457 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62200b4690 con 0x7f623806f4e0 2024-03-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.874+0000 7f623d200640 1 -- 172.21.15.40:0/387085457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62380c7e40 con 0x7f623806f4e0 2024-03-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.874+0000 7f623d200640 1 -- 172.21.15.40:0/387085457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62380c8400 con 0x7f623806f4e0 2024-03-24T00:35:19.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.874+0000 7f623d200640 1 -- 172.21.15.40:0/387085457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6238068170 con 0x7f623806f4e0 2024-03-24T00:35:19.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.874+0000 7f622e000640 1 -- 172.21.15.40:0/387085457 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f62200b4830 con 0x7f623806f4e0 2024-03-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.874+0000 7f622e000640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6214067690 0x7f6214069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.874+0000 7f622e000640 1 -- 172.21.15.40:0/387085457 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f622011f4c0 con 0x7f623806f4e0 2024-03-24T00:35:19.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.874+0000 7f6235e00640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6214067690 0x7f6214069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:19.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.878+0000 7f6235e00640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6214067690 0x7f6214069b50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6228007a60 tx=0x7f6228008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:19.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:19.878+0000 7f622e000640 1 -- 172.21.15.40:0/387085457 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6220016070 con 0x7f623806f4e0 2024-03-24T00:35:20.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.038+0000 7f623d200640 1 -- 172.21.15.40:0/387085457 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62380b4e80 con 0x7f6214067690 2024-03-24T00:35:20.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.046+0000 7f622e000640 1 -- 172.21.15.40:0/387085457 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252069 (secure 0 0 0) 0x7f62380b4e80 con 0x7f6214067690 2024-03-24T00:35:20.051 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:20.053 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:35:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.050+0000 7f623d200640 1 -- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6214067690 msgr2=0x7f6214069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.050+0000 7f623d200640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6214067690 0x7f6214069b50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f6228007a60 tx=0x7f6228008040 comp rx=0 tx=0).stop 2024-03-24T00:35:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.050+0000 7f623d200640 1 -- 172.21.15.40:0/387085457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623806f4e0 msgr2=0x7f623809f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.050+0000 7f623d200640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623806f4e0 0x7f623809f4c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f62200ae040 tx=0x7f6220019220 comp rx=0 tx=0).stop 2024-03-24T00:35:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.050+0000 7f623d200640 1 -- 172.21.15.40:0/387085457 shutdown_connections 2024-03-24T00:35:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.050+0000 7f623d200640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6214067690 0x7f6214069b50 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.050+0000 7f623d200640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62380aea60 0x7f62381fa2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.050+0000 7f623d200640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f623806f4e0 0x7f623809f4c0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.050+0000 7f623d200640 1 --2- 172.21.15.40:0/387085457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f623806f100 0x7f623815e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.050+0000 7f623d200640 1 -- 172.21.15.40:0/387085457 >> 172.21.15.40:0/387085457 conn(0x7f62380d1c90 msgr2=0x7f62380aee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:20.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.050+0000 7f623d200640 1 -- 172.21.15.40:0/387085457 shutdown_connections 2024-03-24T00:35:20.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.050+0000 7f623d200640 1 -- 172.21.15.40:0/387085457 wait complete. 2024-03-24T00:35:20.073 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-03-24T00:35:19.249611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":269816,"kb_used_data":7880,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754704904,"statfs":{"total":773094113280,"available":772817821696,"internally_reserved":0,"allocated":8069120,"data_stored":4098724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":268095026},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.971634"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309470+0000","last_change":"2024-03-24T00:35:15.307223+0000","last_active":"2024-03-24T00:35:16.309470+0000","last_peered":"2024-03-24T00:35:16.309470+0000","last_clean":"2024-03-24T00:35:16.309470+0000","last_became_active":"2024-03-24T00:35:15.306876+0000","last_became_peered":"2024-03-24T00:35:15.306876+0000","last_unstale":"2024-03-24T00:35:16.309470+0000","last_undegraded":"2024-03-24T00:35:16.309470+0000","last_fullsized":"2024-03-24T00:35:16.309470+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:03:28.550395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674867+0000","last_change":"2024-03-24T00:35:15.315880+0000","last_active":"2024-03-24T00:35:16.674867+0000","last_peered":"2024-03-24T00:35:16.674867+0000","last_clean":"2024-03-24T00:35:16.674867+0000","last_became_active":"2024-03-24T00:35:15.315578+0000","last_became_peered":"2024-03-24T00:35:15.315578+0000","last_unstale":"2024-03-24T00:35:16.674867+0000","last_undegraded":"2024-03-24T00:35:16.674867+0000","last_fullsized":"2024-03-24T00:35:16.674867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:19:15.121845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672485+0000","last_change":"2024-03-24T00:35:15.308742+0000","last_active":"2024-03-24T00:35:16.672485+0000","last_peered":"2024-03-24T00:35:16.672485+0000","last_clean":"2024-03-24T00:35:16.672485+0000","last_became_active":"2024-03-24T00:35:15.305351+0000","last_became_peered":"2024-03-24T00:35:15.305351+0000","last_unstale":"2024-03-24T00:35:16.672485+0000","last_undegraded":"2024-03-24T00:35:16.672485+0000","last_fullsized":"2024-03-24T00:35:16.672485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:01.930255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309276+0000","last_change":"2024-03-24T00:35:15.319557+0000","last_active":"2024-03-24T00:35:16.309276+0000","last_peered":"2024-03-24T00:35:16.309276+0000","last_clean":"2024-03-24T00:35:16.309276+0000","last_became_active":"2024-03-24T00:35:15.319234+0000","last_became_peered":"2024-03-24T00:35:15.319234+0000","last_unstale":"2024-03-24T00:35:16.309276+0000","last_undegraded":"2024-03-24T00:35:16.309276+0000","last_fullsized":"2024-03-24T00:35:16.309276+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:48.311977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672701+0000","last_change":"2024-03-24T00:35:15.303853+0000","last_active":"2024-03-24T00:35:16.672701+0000","last_peered":"2024-03-24T00:35:16.672701+0000","last_clean":"2024-03-24T00:35:16.672701+0000","last_became_active":"2024-03-24T00:35:15.303545+0000","last_became_peered":"2024-03-24T00:35:15.303545+0000","last_unstale":"2024-03-24T00:35:16.672701+0000","last_undegraded":"2024-03-24T00:35:16.672701+0000","last_fullsized":"2024-03-24T00:35:16.672701+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:50.684191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674802+0000","last_change":"2024-03-24T00:35:15.317700+0000","last_active":"2024-03-24T00:35:16.674802+0000","last_peered":"2024-03-24T00:35:16.674802+0000","last_clean":"2024-03-24T00:35:16.674802+0000","last_became_active":"2024-03-24T00:35:15.314044+0000","last_became_peered":"2024-03-24T00:35:15.314044+0000","last_unstale":"2024-03-24T00:35:16.674802+0000","last_undegraded":"2024-03-24T00:35:16.674802+0000","last_fullsized":"2024-03-24T00:35:16.674802+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:06.464485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315291+0000","last_change":"2024-03-24T00:35:15.304952+0000","last_active":"2024-03-24T00:35:16.315291+0000","last_peered":"2024-03-24T00:35:16.315291+0000","last_clean":"2024-03-24T00:35:16.315291+0000","last_became_active":"2024-03-24T00:35:15.304668+0000","last_became_peered":"2024-03-24T00:35:15.304668+0000","last_unstale":"2024-03-24T00:35:16.315291+0000","last_undegraded":"2024-03-24T00:35:16.315291+0000","last_fullsized":"2024-03-24T00:35:16.315291+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:17.358722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674208+0000","last_change":"2024-03-24T00:35:15.316878+0000","last_active":"2024-03-24T00:35:16.674208+0000","last_peered":"2024-03-24T00:35:16.674208+0000","last_clean":"2024-03-24T00:35:16.674208+0000","last_became_active":"2024-03-24T00:35:15.314872+0000","last_became_peered":"2024-03-24T00:35:15.314872+0000","last_unstale":"2024-03-24T00:35:16.674208+0000","last_undegraded":"2024-03-24T00:35:16.674208+0000","last_fullsized":"2024-03-24T00:35:16.674208+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:17:26.079724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673297+0000","last_change":"2024-03-24T00:35:15.313509+0000","last_active":"2024-03-24T00:35:16.673297+0000","last_peered":"2024-03-24T00:35:16.673297+0000","last_clean":"2024-03-24T00:35:16.673297+0000","last_became_active":"2024-03-24T00:35:15.313255+0000","last_became_peered":"2024-03-24T00:35:15.313255+0000","last_unstale":"2024-03-24T00:35:16.673297+0000","last_undegraded":"2024-03-24T00:35:16.673297+0000","last_fullsized":"2024-03-24T00:35:16.673297+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:16.274846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310275+0000","last_change":"2024-03-24T00:35:15.305894+0000","last_active":"2024-03-24T00:35:16.310275+0000","last_peered":"2024-03-24T00:35:16.310275+0000","last_clean":"2024-03-24T00:35:16.310275+0000","last_became_active":"2024-03-24T00:35:15.305006+0000","last_became_peered":"2024-03-24T00:35:15.305006+0000","last_unstale":"2024-03-24T00:35:16.310275+0000","last_undegraded":"2024-03-24T00:35:16.310275+0000","last_fullsized":"2024-03-24T00:35:16.310275+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:42:50.245680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309336+0000","last_change":"2024-03-24T00:35:15.311331+0000","last_active":"2024-03-24T00:35:16.309336+0000","last_peered":"2024-03-24T00:35:16.309336+0000","last_clean":"2024-03-24T00:35:16.309336+0000","last_became_active":"2024-03-24T00:35:15.310929+0000","last_became_peered":"2024-03-24T00:35:15.310929+0000","last_unstale":"2024-03-24T00:35:16.309336+0000","last_undegraded":"2024-03-24T00:35:16.309336+0000","last_fullsized":"2024-03-24T00:35:16.309336+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:42.481496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309255+0000","last_change":"2024-03-24T00:35:15.301914+0000","last_active":"2024-03-24T00:35:16.309255+0000","last_peered":"2024-03-24T00:35:16.309255+0000","last_clean":"2024-03-24T00:35:16.309255+0000","last_became_active":"2024-03-24T00:35:15.301693+0000","last_became_peered":"2024-03-24T00:35:15.301693+0000","last_unstale":"2024-03-24T00:35:16.309255+0000","last_undegraded":"2024-03-24T00:35:16.309255+0000","last_fullsized":"2024-03-24T00:35:16.309255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:48:46.946260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317767+0000","last_change":"2024-03-24T00:35:15.321430+0000","last_active":"2024-03-24T00:35:16.317767+0000","last_peered":"2024-03-24T00:35:16.317767+0000","last_clean":"2024-03-24T00:35:16.317767+0000","last_became_active":"2024-03-24T00:35:15.321160+0000","last_became_peered":"2024-03-24T00:35:15.321160+0000","last_unstale":"2024-03-24T00:35:16.317767+0000","last_undegraded":"2024-03-24T00:35:16.317767+0000","last_fullsized":"2024-03-24T00:35:16.317767+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:29.341119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672864+0000","last_change":"2024-03-24T00:35:15.304930+0000","last_active":"2024-03-24T00:35:16.672864+0000","last_peered":"2024-03-24T00:35:16.672864+0000","last_clean":"2024-03-24T00:35:16.672864+0000","last_became_active":"2024-03-24T00:35:15.304110+0000","last_became_peered":"2024-03-24T00:35:15.304110+0000","last_unstale":"2024-03-24T00:35:16.672864+0000","last_undegraded":"2024-03-24T00:35:16.672864+0000","last_fullsized":"2024-03-24T00:35:16.672864+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:36.995688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317386+0000","last_change":"2024-03-24T00:35:15.310204+0000","last_active":"2024-03-24T00:35:16.317386+0000","last_peered":"2024-03-24T00:35:16.317386+0000","last_clean":"2024-03-24T00:35:16.317386+0000","last_became_active":"2024-03-24T00:35:15.309941+0000","last_became_peered":"2024-03-24T00:35:15.309941+0000","last_unstale":"2024-03-24T00:35:16.317386+0000","last_undegraded":"2024-03-24T00:35:16.317386+0000","last_fullsized":"2024-03-24T00:35:16.317386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:49:55.880879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.308884+0000","last_change":"2024-03-24T00:35:15.320547+0000","last_active":"2024-03-24T00:35:16.308884+0000","last_peered":"2024-03-24T00:35:16.308884+0000","last_clean":"2024-03-24T00:35:16.308884+0000","last_became_active":"2024-03-24T00:35:15.320252+0000","last_became_peered":"2024-03-24T00:35:15.320252+0000","last_unstale":"2024-03-24T00:35:16.308884+0000","last_undegraded":"2024-03-24T00:35:16.308884+0000","last_fullsized":"2024-03-24T00:35:16.308884+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:45.255626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673383+0000","last_change":"2024-03-24T00:35:15.302677+0000","last_active":"2024-03-24T00:35:16.673383+0000","last_peered":"2024-03-24T00:35:16.673383+0000","last_clean":"2024-03-24T00:35:16.673383+0000","last_became_active":"2024-03-24T00:35:15.302221+0000","last_became_peered":"2024-03-24T00:35:15.302221+0000","last_unstale":"2024-03-24T00:35:16.673383+0000","last_undegraded":"2024-03-24T00:35:16.673383+0000","last_fullsized":"2024-03-24T00:35:16.673383+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:35:44.727723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309134+0000","last_change":"2024-03-24T00:35:15.319921+0000","last_active":"2024-03-24T00:35:16.309134+0000","last_peered":"2024-03-24T00:35:16.309134+0000","last_clean":"2024-03-24T00:35:16.309134+0000","last_became_active":"2024-03-24T00:35:15.319449+0000","last_became_peered":"2024-03-24T00:35:15.319449+0000","last_unstale":"2024-03-24T00:35:16.309134+0000","last_undegraded":"2024-03-24T00:35:16.309134+0000","last_fullsized":"2024-03-24T00:35:16.309134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:39.876925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674520+0000","last_change":"2024-03-24T00:35:15.318292+0000","last_active":"2024-03-24T00:35:16.674520+0000","last_peered":"2024-03-24T00:35:16.674520+0000","last_clean":"2024-03-24T00:35:16.674520+0000","last_became_active":"2024-03-24T00:35:15.317893+0000","last_became_peered":"2024-03-24T00:35:15.317893+0000","last_unstale":"2024-03-24T00:35:16.674520+0000","last_undegraded":"2024-03-24T00:35:16.674520+0000","last_fullsized":"2024-03-24T00:35:16.674520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:30.057017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317448+0000","last_change":"2024-03-24T00:35:15.317651+0000","last_active":"2024-03-24T00:35:16.317448+0000","last_peered":"2024-03-24T00:35:16.317448+0000","last_clean":"2024-03-24T00:35:16.317448+0000","last_became_active":"2024-03-24T00:35:15.317310+0000","last_became_peered":"2024-03-24T00:35:15.317310+0000","last_unstale":"2024-03-24T00:35:16.317448+0000","last_undegraded":"2024-03-24T00:35:16.317448+0000","last_fullsized":"2024-03-24T00:35:16.317448+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:40:42.481496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310034+0000","last_change":"2024-03-24T00:35:15.306910+0000","last_active":"2024-03-24T00:35:16.310034+0000","last_peered":"2024-03-24T00:35:16.310034+0000","last_clean":"2024-03-24T00:35:16.310034+0000","last_became_active":"2024-03-24T00:35:15.306184+0000","last_became_peered":"2024-03-24T00:35:15.306184+0000","last_unstale":"2024-03-24T00:35:16.310034+0000","last_undegraded":"2024-03-24T00:35:16.310034+0000","last_fullsized":"2024-03-24T00:35:16.310034+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:39.295158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309925+0000","last_change":"2024-03-24T00:35:15.298126+0000","last_active":"2024-03-24T00:35:16.309925+0000","last_peered":"2024-03-24T00:35:16.309925+0000","last_clean":"2024-03-24T00:35:16.309925+0000","last_became_active":"2024-03-24T00:35:15.297811+0000","last_became_peered":"2024-03-24T00:35:15.297811+0000","last_unstale":"2024-03-24T00:35:16.309925+0000","last_undegraded":"2024-03-24T00:35:16.309925+0000","last_fullsized":"2024-03-24T00:35:16.309925+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:53.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309518+0000","last_change":"2024-03-24T00:35:15.312972+0000","last_active":"2024-03-24T00:35:16.309518+0000","last_peered":"2024-03-24T00:35:16.309518+0000","last_clean":"2024-03-24T00:35:16.309518+0000","last_became_active":"2024-03-24T00:35:15.312688+0000","last_became_peered":"2024-03-24T00:35:15.312688+0000","last_unstale":"2024-03-24T00:35:16.309518+0000","last_undegraded":"2024-03-24T00:35:16.309518+0000","last_fullsized":"2024-03-24T00:35:16.309518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:25:54.111013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309032+0000","last_change":"2024-03-24T00:35:15.319725+0000","last_active":"2024-03-24T00:35:16.309032+0000","last_peered":"2024-03-24T00:35:16.309032+0000","last_clean":"2024-03-24T00:35:16.309032+0000","last_became_active":"2024-03-24T00:35:15.319455+0000","last_became_peered":"2024-03-24T00:35:15.319455+0000","last_unstale":"2024-03-24T00:35:16.309032+0000","last_undegraded":"2024-03-24T00:35:16.309032+0000","last_fullsized":"2024-03-24T00:35:16.309032+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:49:04.935445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310794+0000","last_change":"2024-03-24T00:35:15.307844+0000","last_active":"2024-03-24T00:35:16.310794+0000","last_peered":"2024-03-24T00:35:16.310794+0000","last_clean":"2024-03-24T00:35:16.310794+0000","last_became_active":"2024-03-24T00:35:15.307562+0000","last_became_peered":"2024-03-24T00:35:15.307562+0000","last_unstale":"2024-03-24T00:35:16.310794+0000","last_undegraded":"2024-03-24T00:35:16.310794+0000","last_fullsized":"2024-03-24T00:35:16.310794+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:59.255013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315726+0000","last_change":"2024-03-24T00:35:15.321355+0000","last_active":"2024-03-24T00:35:16.315726+0000","last_peered":"2024-03-24T00:35:16.315726+0000","last_clean":"2024-03-24T00:35:16.315726+0000","last_became_active":"2024-03-24T00:35:15.320952+0000","last_became_peered":"2024-03-24T00:35:15.320952+0000","last_unstale":"2024-03-24T00:35:16.315726+0000","last_undegraded":"2024-03-24T00:35:16.315726+0000","last_fullsized":"2024-03-24T00:35:16.315726+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:37:58.309026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672623+0000","last_change":"2024-03-24T00:35:15.309026+0000","last_active":"2024-03-24T00:35:16.672623+0000","last_peered":"2024-03-24T00:35:16.672623+0000","last_clean":"2024-03-24T00:35:16.672623+0000","last_became_active":"2024-03-24T00:35:15.308451+0000","last_became_peered":"2024-03-24T00:35:15.308451+0000","last_unstale":"2024-03-24T00:35:16.672623+0000","last_undegraded":"2024-03-24T00:35:16.672623+0000","last_fullsized":"2024-03-24T00:35:16.672623+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:02:37.899211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310409+0000","last_change":"2024-03-24T00:35:15.299238+0000","last_active":"2024-03-24T00:35:16.310409+0000","last_peered":"2024-03-24T00:35:16.310409+0000","last_clean":"2024-03-24T00:35:16.310409+0000","last_became_active":"2024-03-24T00:35:15.298885+0000","last_became_peered":"2024-03-24T00:35:15.298885+0000","last_unstale":"2024-03-24T00:35:16.310409+0000","last_undegraded":"2024-03-24T00:35:16.310409+0000","last_fullsized":"2024-03-24T00:35:16.310409+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:17.358722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672447+0000","last_change":"2024-03-24T00:35:15.306518+0000","last_active":"2024-03-24T00:35:16.672447+0000","last_peered":"2024-03-24T00:35:16.672447+0000","last_clean":"2024-03-24T00:35:16.672447+0000","last_became_active":"2024-03-24T00:35:15.306250+0000","last_became_peered":"2024-03-24T00:35:15.306250+0000","last_unstale":"2024-03-24T00:35:16.672447+0000","last_undegraded":"2024-03-24T00:35:16.672447+0000","last_fullsized":"2024-03-24T00:35:16.672447+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:01:20.263015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315386+0000","last_change":"2024-03-24T00:35:15.322670+0000","last_active":"2024-03-24T00:35:16.315386+0000","last_peered":"2024-03-24T00:35:16.315386+0000","last_clean":"2024-03-24T00:35:16.315386+0000","last_became_active":"2024-03-24T00:35:15.322338+0000","last_became_peered":"2024-03-24T00:35:15.322338+0000","last_unstale":"2024-03-24T00:35:16.315386+0000","last_undegraded":"2024-03-24T00:35:16.315386+0000","last_fullsized":"2024-03-24T00:35:16.315386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:59.255013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315640+0000","last_change":"2024-03-24T00:35:15.304729+0000","last_active":"2024-03-24T00:35:16.315640+0000","last_peered":"2024-03-24T00:35:16.315640+0000","last_clean":"2024-03-24T00:35:16.315640+0000","last_became_active":"2024-03-24T00:35:15.304326+0000","last_became_peered":"2024-03-24T00:35:15.304326+0000","last_unstale":"2024-03-24T00:35:16.315640+0000","last_undegraded":"2024-03-24T00:35:16.315640+0000","last_fullsized":"2024-03-24T00:35:16.315640+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:13:46.464517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674279+0000","last_change":"2024-03-24T00:35:15.318362+0000","last_active":"2024-03-24T00:35:16.674279+0000","last_peered":"2024-03-24T00:35:16.674279+0000","last_clean":"2024-03-24T00:35:16.674279+0000","last_became_active":"2024-03-24T00:35:15.317329+0000","last_became_peered":"2024-03-24T00:35:15.317329+0000","last_unstale":"2024-03-24T00:35:16.674279+0000","last_undegraded":"2024-03-24T00:35:16.674279+0000","last_fullsized":"2024-03-24T00:35:16.674279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:29.049736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309906+0000","last_change":"2024-03-24T00:35:15.321846+0000","last_active":"2024-03-24T00:35:16.309906+0000","last_peered":"2024-03-24T00:35:16.309906+0000","last_clean":"2024-03-24T00:35:16.309906+0000","last_became_active":"2024-03-24T00:35:15.321483+0000","last_became_peered":"2024-03-24T00:35:15.321483+0000","last_unstale":"2024-03-24T00:35:16.309906+0000","last_undegraded":"2024-03-24T00:35:16.309906+0000","last_fullsized":"2024-03-24T00:35:16.309906+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:32:33.917825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"69'4","reported_seq":18,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309789+0000","last_change":"2024-03-24T00:35:14.306932+0000","last_active":"2024-03-24T00:35:16.309789+0000","last_peered":"2024-03-24T00:35:16.309789+0000","last_clean":"2024-03-24T00:35:16.309789+0000","last_became_active":"2024-03-24T00:35:14.306595+0000","last_became_peered":"2024-03-24T00:35:14.306595+0000","last_unstale":"2024-03-24T00:35:16.309789+0000","last_undegraded":"2024-03-24T00:35:16.309789+0000","last_fullsized":"2024-03-24T00:35:16.309789+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:21:38.289092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673438+0000","last_change":"2024-03-24T00:35:15.302471+0000","last_active":"2024-03-24T00:35:16.673438+0000","last_peered":"2024-03-24T00:35:16.673438+0000","last_clean":"2024-03-24T00:35:16.673438+0000","last_became_active":"2024-03-24T00:35:15.301859+0000","last_became_peered":"2024-03-24T00:35:15.301859+0000","last_unstale":"2024-03-24T00:35:16.673438+0000","last_undegraded":"2024-03-24T00:35:16.673438+0000","last_fullsized":"2024-03-24T00:35:16.673438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:11.400856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317620+0000","last_change":"2024-03-24T00:35:14.305894+0000","last_active":"2024-03-24T00:35:16.317620+0000","last_peered":"2024-03-24T00:35:16.317620+0000","last_clean":"2024-03-24T00:35:16.317620+0000","last_became_active":"2024-03-24T00:35:14.305623+0000","last_became_peered":"2024-03-24T00:35:14.305623+0000","last_unstale":"2024-03-24T00:35:16.317620+0000","last_undegraded":"2024-03-24T00:35:16.317620+0000","last_fullsized":"2024-03-24T00:35:16.317620+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:16.352656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674842+0000","last_change":"2024-03-24T00:35:15.312109+0000","last_active":"2024-03-24T00:35:16.674842+0000","last_peered":"2024-03-24T00:35:16.674842+0000","last_clean":"2024-03-24T00:35:16.674842+0000","last_became_active":"2024-03-24T00:35:15.310930+0000","last_became_peered":"2024-03-24T00:35:15.310930+0000","last_unstale":"2024-03-24T00:35:16.674842+0000","last_undegraded":"2024-03-24T00:35:16.674842+0000","last_fullsized":"2024-03-24T00:35:16.674842+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:06:29.643031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310609+0000","last_change":"2024-03-24T00:35:14.307432+0000","last_active":"2024-03-24T00:35:16.310609+0000","last_peered":"2024-03-24T00:35:16.310609+0000","last_clean":"2024-03-24T00:35:16.310609+0000","last_became_active":"2024-03-24T00:35:14.307081+0000","last_became_peered":"2024-03-24T00:35:14.307081+0000","last_unstale":"2024-03-24T00:35:16.310609+0000","last_undegraded":"2024-03-24T00:35:16.310609+0000","last_fullsized":"2024-03-24T00:35:16.310609+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:17:25.073658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309502+0000","last_change":"2024-03-24T00:35:15.309412+0000","last_active":"2024-03-24T00:35:16.309502+0000","last_peered":"2024-03-24T00:35:16.309502+0000","last_clean":"2024-03-24T00:35:16.309502+0000","last_became_active":"2024-03-24T00:35:15.306563+0000","last_became_peered":"2024-03-24T00:35:15.306563+0000","last_unstale":"2024-03-24T00:35:16.309502+0000","last_undegraded":"2024-03-24T00:35:16.309502+0000","last_fullsized":"2024-03-24T00:35:16.309502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:07:29.341119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674762+0000","last_change":"2024-03-24T00:35:14.310422+0000","last_active":"2024-03-24T00:35:16.674762+0000","last_peered":"2024-03-24T00:35:16.674762+0000","last_clean":"2024-03-24T00:35:16.674762+0000","last_became_active":"2024-03-24T00:35:14.310121+0000","last_became_peered":"2024-03-24T00:35:14.310121+0000","last_unstale":"2024-03-24T00:35:16.674762+0000","last_undegraded":"2024-03-24T00:35:16.674762+0000","last_fullsized":"2024-03-24T00:35:16.674762+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:00.480161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315570+0000","last_change":"2024-03-24T00:35:15.315495+0000","last_active":"2024-03-24T00:35:16.315570+0000","last_peered":"2024-03-24T00:35:16.315570+0000","last_clean":"2024-03-24T00:35:16.315570+0000","last_became_active":"2024-03-24T00:35:15.315170+0000","last_became_peered":"2024-03-24T00:35:15.315170+0000","last_unstale":"2024-03-24T00:35:16.315570+0000","last_undegraded":"2024-03-24T00:35:16.315570+0000","last_fullsized":"2024-03-24T00:35:16.315570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:56:12.607356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672710+0000","last_change":"2024-03-24T00:35:14.302271+0000","last_active":"2024-03-24T00:35:16.672710+0000","last_peered":"2024-03-24T00:35:16.672710+0000","last_clean":"2024-03-24T00:35:16.672710+0000","last_became_active":"2024-03-24T00:35:14.301296+0000","last_became_peered":"2024-03-24T00:35:14.301296+0000","last_unstale":"2024-03-24T00:35:16.672710+0000","last_undegraded":"2024-03-24T00:35:16.672710+0000","last_fullsized":"2024-03-24T00:35:16.672710+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:25:53.104947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310100+0000","last_change":"2024-03-24T00:35:15.308484+0000","last_active":"2024-03-24T00:35:16.310100+0000","last_peered":"2024-03-24T00:35:16.310100+0000","last_clean":"2024-03-24T00:35:16.310100+0000","last_became_active":"2024-03-24T00:35:15.308205+0000","last_became_peered":"2024-03-24T00:35:15.308205+0000","last_unstale":"2024-03-24T00:35:16.310100+0000","last_undegraded":"2024-03-24T00:35:16.310100+0000","last_fullsized":"2024-03-24T00:35:16.310100+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:15.706626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674290+0000","last_change":"2024-03-24T00:35:14.307471+0000","last_active":"2024-03-24T00:35:16.674290+0000","last_peered":"2024-03-24T00:35:16.674290+0000","last_clean":"2024-03-24T00:35:16.674290+0000","last_became_active":"2024-03-24T00:35:14.306751+0000","last_became_peered":"2024-03-24T00:35:14.306751+0000","last_unstale":"2024-03-24T00:35:16.674290+0000","last_undegraded":"2024-03-24T00:35:16.674290+0000","last_fullsized":"2024-03-24T00:35:16.674290+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:35:43.721657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672468+0000","last_change":"2024-03-24T00:35:15.296516+0000","last_active":"2024-03-24T00:35:16.672468+0000","last_peered":"2024-03-24T00:35:16.672468+0000","last_clean":"2024-03-24T00:35:16.672468+0000","last_became_active":"2024-03-24T00:35:15.296247+0000","last_became_peered":"2024-03-24T00:35:15.296247+0000","last_unstale":"2024-03-24T00:35:16.672468+0000","last_undegraded":"2024-03-24T00:35:16.672468+0000","last_fullsized":"2024-03-24T00:35:16.672468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:37.629888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.308764+0000","last_change":"2024-03-24T00:35:14.308598+0000","last_active":"2024-03-24T00:35:16.308764+0000","last_peered":"2024-03-24T00:35:16.308764+0000","last_clean":"2024-03-24T00:35:16.308764+0000","last_became_active":"2024-03-24T00:35:14.308273+0000","last_became_peered":"2024-03-24T00:35:14.308273+0000","last_unstale":"2024-03-24T00:35:16.308764+0000","last_undegraded":"2024-03-24T00:35:16.308764+0000","last_fullsized":"2024-03-24T00:35:16.308764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:58:36.065782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317443+0000","last_change":"2024-03-24T00:35:15.312374+0000","last_active":"2024-03-24T00:35:16.317443+0000","last_peered":"2024-03-24T00:35:16.317443+0000","last_clean":"2024-03-24T00:35:16.317443+0000","last_became_active":"2024-03-24T00:35:15.312014+0000","last_became_peered":"2024-03-24T00:35:15.312014+0000","last_unstale":"2024-03-24T00:35:16.317443+0000","last_undegraded":"2024-03-24T00:35:16.317443+0000","last_fullsized":"2024-03-24T00:35:16.317443+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:49:04.935445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672433+0000","last_change":"2024-03-24T00:35:14.303519+0000","last_active":"2024-03-24T00:35:16.672433+0000","last_peered":"2024-03-24T00:35:16.672433+0000","last_clean":"2024-03-24T00:35:16.672433+0000","last_became_active":"2024-03-24T00:35:14.303177+0000","last_became_peered":"2024-03-24T00:35:14.303177+0000","last_unstale":"2024-03-24T00:35:16.672433+0000","last_undegraded":"2024-03-24T00:35:16.672433+0000","last_fullsized":"2024-03-24T00:35:16.672433+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:01:59.262128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315743+0000","last_change":"2024-03-24T00:35:15.307721+0000","last_active":"2024-03-24T00:35:16.315743+0000","last_peered":"2024-03-24T00:35:16.315743+0000","last_clean":"2024-03-24T00:35:16.315743+0000","last_became_active":"2024-03-24T00:35:15.307434+0000","last_became_peered":"2024-03-24T00:35:15.307434+0000","last_unstale":"2024-03-24T00:35:16.315743+0000","last_undegraded":"2024-03-24T00:35:16.315743+0000","last_fullsized":"2024-03-24T00:35:16.315743+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:52:30.392706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310837+0000","last_change":"2024-03-24T00:35:14.302006+0000","last_active":"2024-03-24T00:35:16.310837+0000","last_peered":"2024-03-24T00:35:16.310837+0000","last_clean":"2024-03-24T00:35:16.310837+0000","last_became_active":"2024-03-24T00:35:14.301696+0000","last_became_peered":"2024-03-24T00:35:14.301696+0000","last_unstale":"2024-03-24T00:35:16.310837+0000","last_undegraded":"2024-03-24T00:35:16.310837+0000","last_fullsized":"2024-03-24T00:35:16.310837+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:57:38.831255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309638+0000","last_change":"2024-03-24T00:35:15.320670+0000","last_active":"2024-03-24T00:35:16.309638+0000","last_peered":"2024-03-24T00:35:16.309638+0000","last_clean":"2024-03-24T00:35:16.309638+0000","last_became_active":"2024-03-24T00:35:15.320389+0000","last_became_peered":"2024-03-24T00:35:15.320389+0000","last_unstale":"2024-03-24T00:35:16.309638+0000","last_undegraded":"2024-03-24T00:35:16.309638+0000","last_fullsized":"2024-03-24T00:35:16.309638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:10:50.773699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674915+0000","last_change":"2024-03-24T00:35:14.312360+0000","last_active":"2024-03-24T00:35:16.674915+0000","last_peered":"2024-03-24T00:35:16.674915+0000","last_clean":"2024-03-24T00:35:16.674915+0000","last_became_active":"2024-03-24T00:35:14.312011+0000","last_became_peered":"2024-03-24T00:35:14.312011+0000","last_unstale":"2024-03-24T00:35:16.674915+0000","last_undegraded":"2024-03-24T00:35:16.674915+0000","last_fullsized":"2024-03-24T00:35:16.674915+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:31:31.055987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674696+0000","last_change":"2024-03-24T00:35:15.313321+0000","last_active":"2024-03-24T00:35:16.674696+0000","last_peered":"2024-03-24T00:35:16.674696+0000","last_clean":"2024-03-24T00:35:16.674696+0000","last_became_active":"2024-03-24T00:35:15.310427+0000","last_became_peered":"2024-03-24T00:35:15.310427+0000","last_unstale":"2024-03-24T00:35:16.674696+0000","last_undegraded":"2024-03-24T00:35:16.674696+0000","last_fullsized":"2024-03-24T00:35:16.674696+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:52.402901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"69'3","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672729+0000","last_change":"2024-03-24T00:35:14.312312+0000","last_active":"2024-03-24T00:35:16.672729+0000","last_peered":"2024-03-24T00:35:16.672729+0000","last_clean":"2024-03-24T00:35:16.672729+0000","last_became_active":"2024-03-24T00:35:14.311575+0000","last_became_peered":"2024-03-24T00:35:14.311575+0000","last_unstale":"2024-03-24T00:35:16.672729+0000","last_undegraded":"2024-03-24T00:35:16.672729+0000","last_fullsized":"2024-03-24T00:35:16.672729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:10:49.767633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673122+0000","last_change":"2024-03-24T00:35:15.298413+0000","last_active":"2024-03-24T00:35:16.673122+0000","last_peered":"2024-03-24T00:35:16.673122+0000","last_clean":"2024-03-24T00:35:16.673122+0000","last_became_active":"2024-03-24T00:35:15.298069+0000","last_became_peered":"2024-03-24T00:35:15.298069+0000","last_unstale":"2024-03-24T00:35:16.673122+0000","last_undegraded":"2024-03-24T00:35:16.673122+0000","last_fullsized":"2024-03-24T00:35:16.673122+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:01:29.889339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"69'6","reported_seq":20,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309336+0000","last_change":"2024-03-24T00:35:14.307257+0000","last_active":"2024-03-24T00:35:16.309336+0000","last_peered":"2024-03-24T00:35:16.309336+0000","last_clean":"2024-03-24T00:35:16.309336+0000","last_became_active":"2024-03-24T00:35:14.306920+0000","last_became_peered":"2024-03-24T00:35:14.306920+0000","last_unstale":"2024-03-24T00:35:16.309336+0000","last_undegraded":"2024-03-24T00:35:16.309336+0000","last_fullsized":"2024-03-24T00:35:16.309336+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:00.480161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309007+0000","last_change":"2024-03-24T00:35:15.299865+0000","last_active":"2024-03-24T00:35:16.309007+0000","last_peered":"2024-03-24T00:35:16.309007+0000","last_clean":"2024-03-24T00:35:16.309007+0000","last_became_active":"2024-03-24T00:35:15.297759+0000","last_became_peered":"2024-03-24T00:35:15.297759+0000","last_unstale":"2024-03-24T00:35:16.309007+0000","last_undegraded":"2024-03-24T00:35:16.309007+0000","last_fullsized":"2024-03-24T00:35:16.309007+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:49:55.880879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309194+0000","last_change":"2024-03-24T00:35:14.302886+0000","last_active":"2024-03-24T00:35:16.309194+0000","last_peered":"2024-03-24T00:35:16.309194+0000","last_clean":"2024-03-24T00:35:16.309194+0000","last_became_active":"2024-03-24T00:35:14.300392+0000","last_became_peered":"2024-03-24T00:35:14.300392+0000","last_unstale":"2024-03-24T00:35:16.309194+0000","last_undegraded":"2024-03-24T00:35:16.309194+0000","last_fullsized":"2024-03-24T00:35:16.309194+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:22.279990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315477+0000","last_change":"2024-03-24T00:35:15.310176+0000","last_active":"2024-03-24T00:35:16.315477+0000","last_peered":"2024-03-24T00:35:16.315477+0000","last_clean":"2024-03-24T00:35:16.315477+0000","last_became_active":"2024-03-24T00:35:15.309857+0000","last_became_peered":"2024-03-24T00:35:15.309857+0000","last_unstale":"2024-03-24T00:35:16.315477+0000","last_undegraded":"2024-03-24T00:35:16.315477+0000","last_fullsized":"2024-03-24T00:35:16.315477+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:08.227783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674436+0000","last_change":"2024-03-24T00:35:14.304511+0000","last_active":"2024-03-24T00:35:16.674436+0000","last_peered":"2024-03-24T00:35:16.674436+0000","last_clean":"2024-03-24T00:35:16.674436+0000","last_became_active":"2024-03-24T00:35:14.304179+0000","last_became_peered":"2024-03-24T00:35:14.304179+0000","last_unstale":"2024-03-24T00:35:16.674436+0000","last_undegraded":"2024-03-24T00:35:16.674436+0000","last_fullsized":"2024-03-24T00:35:16.674436+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:39:10.394790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317539+0000","last_change":"2024-03-24T00:35:15.320538+0000","last_active":"2024-03-24T00:35:16.317539+0000","last_peered":"2024-03-24T00:35:16.317539+0000","last_clean":"2024-03-24T00:35:16.317539+0000","last_became_active":"2024-03-24T00:35:15.320266+0000","last_became_peered":"2024-03-24T00:35:15.320266+0000","last_unstale":"2024-03-24T00:35:16.317539+0000","last_undegraded":"2024-03-24T00:35:16.317539+0000","last_fullsized":"2024-03-24T00:35:16.317539+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:03:28.550395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315272+0000","last_change":"2024-03-24T00:35:14.316216+0000","last_active":"2024-03-24T00:35:16.315272+0000","last_peered":"2024-03-24T00:35:16.315272+0000","last_clean":"2024-03-24T00:35:16.315272+0000","last_became_active":"2024-03-24T00:35:14.314089+0000","last_became_peered":"2024-03-24T00:35:14.314089+0000","last_unstale":"2024-03-24T00:35:16.315272+0000","last_undegraded":"2024-03-24T00:35:16.315272+0000","last_fullsized":"2024-03-24T00:35:16.315272+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:05.458419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672844+0000","last_change":"2024-03-24T00:35:15.301366+0000","last_active":"2024-03-24T00:35:16.672844+0000","last_peered":"2024-03-24T00:35:16.672844+0000","last_clean":"2024-03-24T00:35:16.672844+0000","last_became_active":"2024-03-24T00:35:15.301022+0000","last_became_peered":"2024-03-24T00:35:15.301022+0000","last_unstale":"2024-03-24T00:35:16.672844+0000","last_undegraded":"2024-03-24T00:35:16.672844+0000","last_fullsized":"2024-03-24T00:35:16.672844+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:46.498885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"69'8","reported_seq":22,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.308950+0000","last_change":"2024-03-24T00:35:14.309496+0000","last_active":"2024-03-24T00:35:16.308950+0000","last_peered":"2024-03-24T00:35:16.308950+0000","last_clean":"2024-03-24T00:35:16.308950+0000","last_became_active":"2024-03-24T00:35:14.305359+0000","last_became_peered":"2024-03-24T00:35:14.305359+0000","last_unstale":"2024-03-24T00:35:16.308950+0000","last_undegraded":"2024-03-24T00:35:16.308950+0000","last_fullsized":"2024-03-24T00:35:16.308950+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:48:45.940194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310386+0000","last_change":"2024-03-24T00:35:15.303098+0000","last_active":"2024-03-24T00:35:16.310386+0000","last_peered":"2024-03-24T00:35:16.310386+0000","last_clean":"2024-03-24T00:35:16.310386+0000","last_became_active":"2024-03-24T00:35:15.302783+0000","last_became_peered":"2024-03-24T00:35:15.302783+0000","last_unstale":"2024-03-24T00:35:16.310386+0000","last_undegraded":"2024-03-24T00:35:16.310386+0000","last_fullsized":"2024-03-24T00:35:16.310386+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:45.432659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674498+0000","last_change":"2024-03-24T00:35:14.305678+0000","last_active":"2024-03-24T00:35:16.674498+0000","last_peered":"2024-03-24T00:35:16.674498+0000","last_clean":"2024-03-24T00:35:16.674498+0000","last_became_active":"2024-03-24T00:35:14.305363+0000","last_became_peered":"2024-03-24T00:35:14.305363+0000","last_unstale":"2024-03-24T00:35:16.674498+0000","last_undegraded":"2024-03-24T00:35:16.674498+0000","last_fullsized":"2024-03-24T00:35:16.674498+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:46.471632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309740+0000","last_change":"2024-03-24T00:35:15.307318+0000","last_active":"2024-03-24T00:35:16.309740+0000","last_peered":"2024-03-24T00:35:16.309740+0000","last_clean":"2024-03-24T00:35:16.309740+0000","last_became_active":"2024-03-24T00:35:15.307044+0000","last_became_peered":"2024-03-24T00:35:15.307044+0000","last_unstale":"2024-03-24T00:35:16.309740+0000","last_undegraded":"2024-03-24T00:35:16.309740+0000","last_fullsized":"2024-03-24T00:35:16.309740+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:45:09.891306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.310338+0000","last_change":"2024-03-24T00:35:14.301078+0000","last_active":"2024-03-24T00:35:16.310338+0000","last_peered":"2024-03-24T00:35:16.310338+0000","last_clean":"2024-03-24T00:35:16.310338+0000","last_became_active":"2024-03-24T00:35:14.300735+0000","last_became_peered":"2024-03-24T00:35:14.300735+0000","last_unstale":"2024-03-24T00:35:16.310338+0000","last_undegraded":"2024-03-24T00:35:16.310338+0000","last_fullsized":"2024-03-24T00:35:16.310338+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:00.130300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309338+0000","last_change":"2024-03-24T00:35:15.300688+0000","last_active":"2024-03-24T00:35:16.309338+0000","last_peered":"2024-03-24T00:35:16.309338+0000","last_clean":"2024-03-24T00:35:16.309338+0000","last_became_active":"2024-03-24T00:35:15.299409+0000","last_became_peered":"2024-03-24T00:35:15.299409+0000","last_unstale":"2024-03-24T00:35:16.309338+0000","last_undegraded":"2024-03-24T00:35:16.309338+0000","last_fullsized":"2024-03-24T00:35:16.309338+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:57:39.837321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"69'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674882+0000","last_change":"2024-03-24T00:35:14.315167+0000","last_active":"2024-03-24T00:35:16.674882+0000","last_peered":"2024-03-24T00:35:16.674882+0000","last_clean":"2024-03-24T00:35:16.674882+0000","last_became_active":"2024-03-24T00:35:14.314848+0000","last_became_peered":"2024-03-24T00:35:14.314848+0000","last_unstale":"2024-03-24T00:35:16.674882+0000","last_undegraded":"2024-03-24T00:35:16.674882+0000","last_fullsized":"2024-03-24T00:35:16.674882+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:03.351920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673262+0000","last_change":"2024-03-24T00:35:15.309311+0000","last_active":"2024-03-24T00:35:16.673262+0000","last_peered":"2024-03-24T00:35:16.673262+0000","last_clean":"2024-03-24T00:35:16.673262+0000","last_became_active":"2024-03-24T00:35:15.309005+0000","last_became_peered":"2024-03-24T00:35:15.309005+0000","last_unstale":"2024-03-24T00:35:16.673262+0000","last_undegraded":"2024-03-24T00:35:16.673262+0000","last_fullsized":"2024-03-24T00:35:16.673262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:54:52.276445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309548+0000","last_change":"2024-03-24T00:35:14.313483+0000","last_active":"2024-03-24T00:35:16.309548+0000","last_peered":"2024-03-24T00:35:16.309548+0000","last_clean":"2024-03-24T00:35:16.309548+0000","last_became_active":"2024-03-24T00:35:14.311301+0000","last_became_peered":"2024-03-24T00:35:14.311301+0000","last_unstale":"2024-03-24T00:35:16.309548+0000","last_undegraded":"2024-03-24T00:35:16.309548+0000","last_fullsized":"2024-03-24T00:35:16.309548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:38:52.115501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672604+0000","last_change":"2024-03-24T00:35:15.306801+0000","last_active":"2024-03-24T00:35:16.672604+0000","last_peered":"2024-03-24T00:35:16.672604+0000","last_clean":"2024-03-24T00:35:16.672604+0000","last_became_active":"2024-03-24T00:35:15.306493+0000","last_became_peered":"2024-03-24T00:35:15.306493+0000","last_unstale":"2024-03-24T00:35:16.672604+0000","last_undegraded":"2024-03-24T00:35:16.672604+0000","last_fullsized":"2024-03-24T00:35:16.672604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:59:21.692918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674588+0000","last_change":"2024-03-24T00:35:14.307612+0000","last_active":"2024-03-24T00:35:16.674588+0000","last_peered":"2024-03-24T00:35:16.674588+0000","last_clean":"2024-03-24T00:35:16.674588+0000","last_became_active":"2024-03-24T00:35:14.307103+0000","last_became_peered":"2024-03-24T00:35:14.307103+0000","last_unstale":"2024-03-24T00:35:16.674588+0000","last_undegraded":"2024-03-24T00:35:16.674588+0000","last_fullsized":"2024-03-24T00:35:16.674588+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:20:19.408158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672420+0000","last_change":"2024-03-24T00:35:15.314657+0000","last_active":"2024-03-24T00:35:16.672420+0000","last_peered":"2024-03-24T00:35:16.672420+0000","last_clean":"2024-03-24T00:35:16.672420+0000","last_became_active":"2024-03-24T00:35:15.314388+0000","last_became_peered":"2024-03-24T00:35:15.314388+0000","last_unstale":"2024-03-24T00:35:16.672420+0000","last_undegraded":"2024-03-24T00:35:16.672420+0000","last_fullsized":"2024-03-24T00:35:16.672420+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:50:25.099400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315552+0000","last_change":"2024-03-24T00:35:14.315703+0000","last_active":"2024-03-24T00:35:16.315552+0000","last_peered":"2024-03-24T00:35:16.315552+0000","last_clean":"2024-03-24T00:35:16.315552+0000","last_became_active":"2024-03-24T00:35:14.314862+0000","last_became_peered":"2024-03-24T00:35:14.314862+0000","last_unstale":"2024-03-24T00:35:16.315552+0000","last_undegraded":"2024-03-24T00:35:16.315552+0000","last_fullsized":"2024-03-24T00:35:16.315552+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:06:28.636965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672843+0000","last_change":"2024-03-24T00:35:15.315799+0000","last_active":"2024-03-24T00:35:16.672843+0000","last_peered":"2024-03-24T00:35:16.672843+0000","last_clean":"2024-03-24T00:35:16.672843+0000","last_became_active":"2024-03-24T00:35:15.315512+0000","last_became_peered":"2024-03-24T00:35:15.315512+0000","last_unstale":"2024-03-24T00:35:16.672843+0000","last_undegraded":"2024-03-24T00:35:16.672843+0000","last_fullsized":"2024-03-24T00:35:16.672843+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:08:21.273790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317323+0000","last_change":"2024-03-24T00:35:14.306441+0000","last_active":"2024-03-24T00:35:16.317323+0000","last_peered":"2024-03-24T00:35:16.317323+0000","last_clean":"2024-03-24T00:35:16.317323+0000","last_became_active":"2024-03-24T00:35:14.306136+0000","last_became_peered":"2024-03-24T00:35:14.306136+0000","last_unstale":"2024-03-24T00:35:16.317323+0000","last_undegraded":"2024-03-24T00:35:16.317323+0000","last_fullsized":"2024-03-24T00:35:16.317323+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:30:07.221717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317637+0000","last_change":"2024-03-24T00:35:15.309020+0000","last_active":"2024-03-24T00:35:16.317637+0000","last_peered":"2024-03-24T00:35:16.317637+0000","last_clean":"2024-03-24T00:35:16.317637+0000","last_became_active":"2024-03-24T00:35:15.308668+0000","last_became_peered":"2024-03-24T00:35:15.308668+0000","last_unstale":"2024-03-24T00:35:16.317637+0000","last_undegraded":"2024-03-24T00:35:16.317637+0000","last_fullsized":"2024-03-24T00:35:16.317637+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:28:38.970698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674548+0000","last_change":"2024-03-24T00:35:14.304572+0000","last_active":"2024-03-24T00:35:16.674548+0000","last_peered":"2024-03-24T00:35:16.674548+0000","last_clean":"2024-03-24T00:35:16.674548+0000","last_became_active":"2024-03-24T00:35:14.304230+0000","last_became_peered":"2024-03-24T00:35:14.304230+0000","last_unstale":"2024-03-24T00:35:16.674548+0000","last_undegraded":"2024-03-24T00:35:16.674548+0000","last_fullsized":"2024-03-24T00:35:16.674548+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:56:31.842925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.315785+0000","last_change":"2024-03-24T00:35:15.308983+0000","last_active":"2024-03-24T00:35:16.315785+0000","last_peered":"2024-03-24T00:35:16.315785+0000","last_clean":"2024-03-24T00:35:16.315785+0000","last_became_active":"2024-03-24T00:35:15.308707+0000","last_became_peered":"2024-03-24T00:35:15.308707+0000","last_unstale":"2024-03-24T00:35:16.315785+0000","last_undegraded":"2024-03-24T00:35:16.315785+0000","last_fullsized":"2024-03-24T00:35:16.315785+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:50:19.665491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"70'1","reported_seq":15,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317724+0000","last_change":"2024-03-24T00:35:14.314282+0000","last_active":"2024-03-24T00:35:16.317724+0000","last_peered":"2024-03-24T00:35:16.317724+0000","last_clean":"2024-03-24T00:35:16.317724+0000","last_became_active":"2024-03-24T00:35:14.312772+0000","last_became_peered":"2024-03-24T00:35:14.312772+0000","last_unstale":"2024-03-24T00:35:16.317724+0000","last_undegraded":"2024-03-24T00:35:16.317724+0000","last_fullsized":"2024-03-24T00:35:16.317724+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:51.396835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309484+0000","last_change":"2024-03-24T00:35:15.320643+0000","last_active":"2024-03-24T00:35:16.309484+0000","last_peered":"2024-03-24T00:35:16.309484+0000","last_clean":"2024-03-24T00:35:16.309484+0000","last_became_active":"2024-03-24T00:35:15.320317+0000","last_became_peered":"2024-03-24T00:35:15.320317+0000","last_unstale":"2024-03-24T00:35:16.309484+0000","last_undegraded":"2024-03-24T00:35:16.309484+0000","last_fullsized":"2024-03-24T00:35:16.309484+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:28:38.970698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673353+0000","last_change":"2024-03-24T00:35:14.306286+0000","last_active":"2024-03-24T00:35:16.673353+0000","last_peered":"2024-03-24T00:35:16.673353+0000","last_clean":"2024-03-24T00:35:16.673353+0000","last_became_active":"2024-03-24T00:35:14.305944+0000","last_became_peered":"2024-03-24T00:35:14.305944+0000","last_unstale":"2024-03-24T00:35:16.673353+0000","last_undegraded":"2024-03-24T00:35:16.673353+0000","last_fullsized":"2024-03-24T00:35:16.673353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:20.973520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309665+0000","last_change":"2024-03-24T00:35:15.303121+0000","last_active":"2024-03-24T00:35:16.309665+0000","last_peered":"2024-03-24T00:35:16.309665+0000","last_clean":"2024-03-24T00:35:16.309665+0000","last_became_active":"2024-03-24T00:35:15.302847+0000","last_became_peered":"2024-03-24T00:35:15.302847+0000","last_unstale":"2024-03-24T00:35:16.309665+0000","last_undegraded":"2024-03-24T00:35:16.309665+0000","last_fullsized":"2024-03-24T00:35:16.309665+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:29:44.114132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.317535+0000","last_change":"2024-03-24T00:35:14.305600+0000","last_active":"2024-03-24T00:35:16.317535+0000","last_peered":"2024-03-24T00:35:16.317535+0000","last_clean":"2024-03-24T00:35:16.317535+0000","last_became_active":"2024-03-24T00:35:14.305305+0000","last_became_peered":"2024-03-24T00:35:14.305305+0000","last_unstale":"2024-03-24T00:35:16.317535+0000","last_undegraded":"2024-03-24T00:35:16.317535+0000","last_fullsized":"2024-03-24T00:35:16.317535+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:03:58.248947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.673012+0000","last_change":"2024-03-24T00:35:15.304530+0000","last_active":"2024-03-24T00:35:16.673012+0000","last_peered":"2024-03-24T00:35:16.673012+0000","last_clean":"2024-03-24T00:35:16.673012+0000","last_became_active":"2024-03-24T00:35:15.303763+0000","last_became_peered":"2024-03-24T00:35:15.303763+0000","last_unstale":"2024-03-24T00:35:16.673012+0000","last_undegraded":"2024-03-24T00:35:16.673012+0000","last_fullsized":"2024-03-24T00:35:16.673012+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:38:45.109294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672859+0000","last_change":"2024-03-24T00:35:14.311840+0000","last_active":"2024-03-24T00:35:16.672859+0000","last_peered":"2024-03-24T00:35:16.672859+0000","last_clean":"2024-03-24T00:35:16.672859+0000","last_became_active":"2024-03-24T00:35:14.311512+0000","last_became_peered":"2024-03-24T00:35:14.311512+0000","last_unstale":"2024-03-24T00:35:16.672859+0000","last_undegraded":"2024-03-24T00:35:16.672859+0000","last_fullsized":"2024-03-24T00:35:16.672859+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:41.761662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674641+0000","last_change":"2024-03-24T00:35:15.313450+0000","last_active":"2024-03-24T00:35:16.674641+0000","last_peered":"2024-03-24T00:35:16.674641+0000","last_clean":"2024-03-24T00:35:16.674641+0000","last_became_active":"2024-03-24T00:35:15.312976+0000","last_became_peered":"2024-03-24T00:35:15.312976+0000","last_unstale":"2024-03-24T00:35:16.674641+0000","last_undegraded":"2024-03-24T00:35:16.674641+0000","last_fullsized":"2024-03-24T00:35:16.674641+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:57.420849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.308763+0000","last_change":"2024-03-24T00:35:14.302928+0000","last_active":"2024-03-24T00:35:16.308763+0000","last_peered":"2024-03-24T00:35:16.308763+0000","last_clean":"2024-03-24T00:35:16.308763+0000","last_became_active":"2024-03-24T00:35:14.301283+0000","last_became_peered":"2024-03-24T00:35:14.301283+0000","last_unstale":"2024-03-24T00:35:16.308763+0000","last_undegraded":"2024-03-24T00:35:16.308763+0000","last_fullsized":"2024-03-24T00:35:16.308763+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:21:08.519311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309323+0000","last_change":"2024-03-24T00:35:15.307308+0000","last_active":"2024-03-24T00:35:16.309323+0000","last_peered":"2024-03-24T00:35:16.309323+0000","last_clean":"2024-03-24T00:35:16.309323+0000","last_became_active":"2024-03-24T00:35:15.307017+0000","last_became_peered":"2024-03-24T00:35:15.307017+0000","last_unstale":"2024-03-24T00:35:16.309323+0000","last_undegraded":"2024-03-24T00:35:16.309323+0000","last_fullsized":"2024-03-24T00:35:16.309323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:02:01.136366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.308816+0000","last_change":"2024-03-24T00:35:14.300683+0000","last_active":"2024-03-24T00:35:16.308816+0000","last_peered":"2024-03-24T00:35:16.308816+0000","last_clean":"2024-03-24T00:35:16.308816+0000","last_became_active":"2024-03-24T00:35:14.300308+0000","last_became_peered":"2024-03-24T00:35:14.300308+0000","last_unstale":"2024-03-24T00:35:16.308816+0000","last_undegraded":"2024-03-24T00:35:16.308816+0000","last_fullsized":"2024-03-24T00:35:16.308816+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:14:47.305911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674269+0000","last_change":"2024-03-24T00:35:15.316367+0000","last_active":"2024-03-24T00:35:16.674269+0000","last_peered":"2024-03-24T00:35:16.674269+0000","last_clean":"2024-03-24T00:35:16.674269+0000","last_became_active":"2024-03-24T00:35:15.316100+0000","last_became_peered":"2024-03-24T00:35:15.316100+0000","last_unstale":"2024-03-24T00:35:16.674269+0000","last_undegraded":"2024-03-24T00:35:16.674269+0000","last_fullsized":"2024-03-24T00:35:16.674269+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:57:39.837321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":14,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674349+0000","last_change":"2024-03-24T00:35:14.310352+0000","last_active":"2024-03-24T00:35:16.674349+0000","last_peered":"2024-03-24T00:35:16.674349+0000","last_clean":"2024-03-24T00:35:16.674349+0000","last_became_active":"2024-03-24T00:35:14.309943+0000","last_became_peered":"2024-03-24T00:35:14.309943+0000","last_unstale":"2024-03-24T00:35:16.674349+0000","last_undegraded":"2024-03-24T00:35:16.674349+0000","last_fullsized":"2024-03-24T00:35:16.674349+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:47:15.268780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":12,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.672586+0000","last_change":"2024-03-24T00:35:15.318636+0000","last_active":"2024-03-24T00:35:16.672586+0000","last_peered":"2024-03-24T00:35:16.672586+0000","last_clean":"2024-03-24T00:35:16.672586+0000","last_became_active":"2024-03-24T00:35:15.318385+0000","last_became_peered":"2024-03-24T00:35:15.318385+0000","last_unstale":"2024-03-24T00:35:16.672586+0000","last_undegraded":"2024-03-24T00:35:16.672586+0000","last_fullsized":"2024-03-24T00:35:16.672586+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:14.276723+0000","last_clean_scrub_stamp":"2024-03-24T00:35:14.276723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:36:50.117855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"69'2","reported_seq":16,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.674448+0000","last_change":"2024-03-24T00:35:14.310608+0000","last_active":"2024-03-24T00:35:16.674448+0000","last_peered":"2024-03-24T00:35:16.674448+0000","last_clean":"2024-03-24T00:35:16.674448+0000","last_became_active":"2024-03-24T00:35:14.309037+0000","last_became_peered":"2024-03-24T00:35:14.309037+0000","last_unstale":"2024-03-24T00:35:16.674448+0000","last_undegraded":"2024-03-24T00:35:16.674448+0000","last_fullsized":"2024-03-24T00:35:16.674448+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:35:13.270657+0000","last_clean_scrub_stamp":"2024-03-24T00:35:13.270657+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:56:22.279990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":272,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-24T00:35:16.309099+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:35:16.309099+0000","last_peered":"2024-03-24T00:35:16.309099+0000","last_clean":"2024-03-24T00:35:16.309099+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:35:16.309099+0000","last_undegraded":"2024-03-24T00:35:16.309099+0000","last_fullsized":"2024-03-24T00:35:16.309099+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32600,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339240,"statfs":{"total":96636764160,"available":96603381760,"internally_reserved":0,"allocated":1454080,"data_stored":955130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574904,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":364238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574904,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32016,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339824,"statfs":{"total":96636764160,"available":96603979776,"internally_reserved":0,"allocated":856064,"data_stored":364236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574904,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40340,"kb_used_data":836,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331500,"statfs":{"total":96636764160,"available":96595456000,"internally_reserved":0,"allocated":856064,"data_stored":364216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":860160,"data_stored":364258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":864256,"data_stored":365758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":872448,"data_stored":366374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36756,"kb_used_data":1412,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335084,"statfs":{"total":96636764160,"available":96599126016,"internally_reserved":0,"allocated":1445888,"data_stored":954514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:35:20.077 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:35:20.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:35:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.258+0000 7fc7f9800640 1 Processor -- start 2024-03-24T00:35:20.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.258+0000 7fc7f9800640 1 -- start start 2024-03-24T00:35:20.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7f9800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc7f41628d0 0x7fc7f40d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7f9800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc7f40b42d0 0x7fc7f406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7f9800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f415e3e0 con 0x7fc7f415e000 2024-03-24T00:35:20.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7f9800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f4161ba0 con 0x7fc7f41628d0 2024-03-24T00:35:20.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7f9800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7f40b5f10 con 0x7fc7f40b42d0 2024-03-24T00:35:20.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7ebe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc7f41628d0 0x7fc7f40d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7f2a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc7f415e000 0x7fc7f40b3bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:34274/0 (socket says 172.21.15.40:34274) 2024-03-24T00:35:20.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7ebe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc7f41628d0 0x7fc7f40d16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55002/0 (socket says 172.21.15.40:55002) 2024-03-24T00:35:20.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7f2a00640 1 -- 172.21.15.40:0/3681384752 learned_addr learned my addr 172.21.15.40:0/3681384752 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:20.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7f3400640 1 --2- 172.21.15.40:0/3681384752 >> v2:172.21.15.53:3300/0 conn(0x7fc7f40b42d0 0x7fc7f406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7ebe00640 1 -- 172.21.15.40:0/3681384752 >> v2:172.21.15.53:3300/0 conn(0x7fc7f40b42d0 msgr2=0x7fc7f406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:20.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7ebe00640 1 --2- 172.21.15.40:0/3681384752 >> v2:172.21.15.53:3300/0 conn(0x7fc7f40b42d0 0x7fc7f406dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7ebe00640 1 -- 172.21.15.40:0/3681384752 >> v1:172.21.15.53:6789/0 conn(0x7fc7f415e000 legacy=0x7fc7f40b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.262+0000 7fc7ebe00640 1 -- 172.21.15.40:0/3681384752 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f4161df0 con 0x7fc7f41628d0 2024-03-24T00:35:20.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.266+0000 7fc7ebe00640 1 --2- 172.21.15.40:0/3681384752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7f41628d0 0x7fc7f40d16d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc7f40d0b50 tx=0x7fc7dc01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a165b107b122dde server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:20.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.266+0000 7fc7f2000640 1 -- 172.21.15.40:0/3681384752 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc0b0070 con 0x7fc7f41628d0 2024-03-24T00:35:20.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.266+0000 7fc7f2000640 1 -- 172.21.15.40:0/3681384752 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7dc0ab600 con 0x7fc7f41628d0 2024-03-24T00:35:20.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.266+0000 7fc7f2000640 1 -- 172.21.15.40:0/3681384752 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc0ab9d0 con 0x7fc7f41628d0 2024-03-24T00:35:20.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.266+0000 7fc7f9800640 1 -- 172.21.15.40:0/3681384752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7f41628d0 msgr2=0x7fc7f40d16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.266+0000 7fc7f9800640 1 --2- 172.21.15.40:0/3681384752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7f41628d0 0x7fc7f40d16d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc7f40d0b50 tx=0x7fc7dc01c450 comp rx=0 tx=0).stop 2024-03-24T00:35:20.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.270+0000 7fc7f9800640 1 -- 172.21.15.40:0/3681384752 shutdown_connections 2024-03-24T00:35:20.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.270+0000 7fc7f9800640 1 --2- 172.21.15.40:0/3681384752 >> v2:172.21.15.53:3300/0 conn(0x7fc7f40b42d0 0x7fc7f406dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.270+0000 7fc7f9800640 1 --2- 172.21.15.40:0/3681384752 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7f41628d0 0x7fc7f40d16d0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.270+0000 7fc7f9800640 1 -- 172.21.15.40:0/3681384752 >> 172.21.15.40:0/3681384752 conn(0x7fc7f40d0bb0 msgr2=0x7fc7f40d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:20.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.270+0000 7fc7f9800640 1 -- 172.21.15.40:0/3681384752 shutdown_connections 2024-03-24T00:35:20.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.270+0000 7fc7f9800640 1 -- 172.21.15.40:0/3681384752 wait complete. 2024-03-24T00:35:20.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.270+0000 7fc7f9800640 1 Processor -- start 2024-03-24T00:35:20.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.270+0000 7fc7f9800640 1 -- start start 2024-03-24T00:35:20.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7f9800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7f40b42d0 0x7fc7f4156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7f9800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7f415e000 0x7fc7f41459c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7f9800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7f41628d0 0x7fc7f4145f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7f9800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f416b0c0 con 0x7fc7f415e000 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7f9800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f416af40 con 0x7fc7f40b42d0 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7f9800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f416b240 con 0x7fc7f41628d0 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7ebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7f415e000 0x7fc7f41459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7ebe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7f415e000 0x7fc7f41459c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55008/0 (socket says 172.21.15.40:55008) 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7ebe00640 1 -- 172.21.15.40:0/3543450909 learned_addr learned my addr 172.21.15.40:0/3543450909 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7f3400640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7f41628d0 0x7fc7f4145f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7f2a00640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7f40b42d0 0x7fc7f4156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7ebe00640 1 -- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7f41628d0 msgr2=0x7fc7f4145f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7ebe00640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7f41628d0 0x7fc7f4145f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7ebe00640 1 -- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7f40b42d0 msgr2=0x7fc7f4156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7ebe00640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7f40b42d0 0x7fc7f4156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7ebe00640 1 -- 172.21.15.40:0/3543450909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f41477a0 con 0x7fc7f415e000 2024-03-24T00:35:20.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7ebe00640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7f415e000 0x7fc7f41459c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc01a080 tx=0x7fc7dc0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:20.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7eaa00640 1 -- 172.21.15.40:0/3543450909 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc0b0040 con 0x7fc7f415e000 2024-03-24T00:35:20.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7eaa00640 1 -- 172.21.15.40:0/3543450909 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7dc0b9070 con 0x7fc7f415e000 2024-03-24T00:35:20.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7eaa00640 1 -- 172.21.15.40:0/3543450909 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc006e20 con 0x7fc7f415e000 2024-03-24T00:35:20.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7f9800640 1 -- 172.21.15.40:0/3543450909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f4147a80 con 0x7fc7f415e000 2024-03-24T00:35:20.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7f9800640 1 -- 172.21.15.40:0/3543450909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f4150610 con 0x7fc7f415e000 2024-03-24T00:35:20.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7f9800640 1 -- 172.21.15.40:0/3543450909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f4068170 con 0x7fc7f415e000 2024-03-24T00:35:20.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.274+0000 7fc7eaa00640 1 -- 172.21.15.40:0/3543450909 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc7dc0a4d10 con 0x7fc7f415e000 2024-03-24T00:35:20.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.278+0000 7fc7eaa00640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7d0067690 0x7fc7d0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.278+0000 7fc7eaa00640 1 -- 172.21.15.40:0/3543450909 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7dc11f1f0 con 0x7fc7f415e000 2024-03-24T00:35:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.278+0000 7fc7f2a00640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7d0067690 0x7fc7d0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.278+0000 7fc7f2a00640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7d0067690 0x7fc7d0069b50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4005890 tx=0x7fc7e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:20.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.282+0000 7fc7eaa00640 1 -- 172.21.15.40:0/3543450909 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc7dc016070 con 0x7fc7f415e000 2024-03-24T00:35:20.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.514+0000 7fc7f9800640 1 -- 172.21.15.40:0/3543450909 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc7f4156330 con 0x7fc7f415e000 2024-03-24T00:35:20.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7eaa00640 1 -- 172.21.15.40:0/3543450909 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc7dc0efe60 con 0x7fc7f415e000 2024-03-24T00:35:20.520 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-03-24T00:35:20.521 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw== 2024-03-24T00:35:20.521 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-03-24T00:35:20.521 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-03-24T00:35:20.521 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:35:20.521 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-03-24T00:35:20.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7f9800640 1 -- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7d0067690 msgr2=0x7fc7d0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7f9800640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7d0067690 0x7fc7d0069b50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4005890 tx=0x7fc7e4009040 comp rx=0 tx=0).stop 2024-03-24T00:35:20.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7f9800640 1 -- 172.21.15.40:0/3543450909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7f415e000 msgr2=0x7fc7f41459c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7f9800640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7f415e000 0x7fc7f41459c0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc01a080 tx=0x7fc7dc0191f0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7f9800640 1 -- 172.21.15.40:0/3543450909 shutdown_connections 2024-03-24T00:35:20.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7f9800640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc7d0067690 0x7fc7d0069b50 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7f9800640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc7f41628d0 0x7fc7f4145f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7f9800640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc7f415e000 0x7fc7f41459c0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7f9800640 1 --2- 172.21.15.40:0/3543450909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7f40b42d0 0x7fc7f4156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7f9800640 1 -- 172.21.15.40:0/3543450909 >> 172.21.15.40:0/3543450909 conn(0x7fc7f40d0bb0 msgr2=0x7fc7f4147df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:20.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7f9800640 1 -- 172.21.15.40:0/3543450909 shutdown_connections 2024-03-24T00:35:20.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.518+0000 7fc7f9800640 1 -- 172.21.15.40:0/3543450909 wait complete. 2024-03-24T00:35:20.543 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03baa00640 1 Processor -- start 2024-03-24T00:35:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03baa00640 1 -- start start 2024-03-24T00:35:20.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03baa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f03b415f600 0x7f03b40b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03baa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03b40af170 0x7f03b4165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03baa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b40b1e50 con 0x7f03b40d2830 2024-03-24T00:35:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03baa00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b40b3630 con 0x7f03b40af170 2024-03-24T00:35:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03baa00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b40d2ca0 con 0x7f03b415f600 2024-03-24T00:35:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03b8a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03b40af170 0x7f03b4165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03b8a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f03b40af170 0x7f03b4165e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55024/0 (socket says 172.21.15.40:55024) 2024-03-24T00:35:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03b8a00640 1 -- 172.21.15.40:0/1349158048 learned_addr learned my addr 172.21.15.40:0/1349158048 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03b3e00640 1 --2- 172.21.15.40:0/1349158048 >> v2:172.21.15.53:3300/0 conn(0x7f03b415f600 0x7f03b40b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03b8a00640 1 -- 172.21.15.40:0/1349158048 >> v2:172.21.15.53:3300/0 conn(0x7f03b415f600 msgr2=0x7f03b40b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03b8a00640 1 --2- 172.21.15.40:0/1349158048 >> v2:172.21.15.53:3300/0 conn(0x7f03b415f600 0x7f03b40b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03b8a00640 1 -- 172.21.15.40:0/1349158048 >> v1:172.21.15.53:6789/0 conn(0x7f03b40d2830 legacy=0x7f03b40aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.682+0000 7f03b8a00640 1 -- 172.21.15.40:0/1349158048 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b40b20f0 con 0x7f03b40af170 2024-03-24T00:35:20.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.686+0000 7f03b8a00640 1 --2- 172.21.15.40:0/1349158048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b40af170 0x7f03b4165e40 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f03b4071f30 tx=0x7f03ac01f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35bb749f74de0f05 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:20.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.686+0000 7f03b3400640 1 -- 172.21.15.40:0/1349158048 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac0b4070 con 0x7f03b40af170 2024-03-24T00:35:20.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.686+0000 7f03b3400640 1 -- 172.21.15.40:0/1349158048 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ac0af400 con 0x7f03b40af170 2024-03-24T00:35:20.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.686+0000 7f03b3400640 1 -- 172.21.15.40:0/1349158048 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac0af7d0 con 0x7f03b40af170 2024-03-24T00:35:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.686+0000 7f03baa00640 1 -- 172.21.15.40:0/1349158048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b40af170 msgr2=0x7f03b4165e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.686+0000 7f03baa00640 1 --2- 172.21.15.40:0/1349158048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b40af170 0x7f03b4165e40 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f03b4071f30 tx=0x7f03ac01f1a0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.686+0000 7f03baa00640 1 -- 172.21.15.40:0/1349158048 shutdown_connections 2024-03-24T00:35:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.686+0000 7f03baa00640 1 --2- 172.21.15.40:0/1349158048 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b40af170 0x7f03b4165e40 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.686+0000 7f03baa00640 1 --2- 172.21.15.40:0/1349158048 >> v2:172.21.15.53:3300/0 conn(0x7f03b415f600 0x7f03b40b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.686+0000 7f03baa00640 1 -- 172.21.15.40:0/1349158048 >> 172.21.15.40:0/1349158048 conn(0x7f03b40d1d10 msgr2=0x7f03b40d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.686+0000 7f03baa00640 1 -- 172.21.15.40:0/1349158048 shutdown_connections 2024-03-24T00:35:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.686+0000 7f03baa00640 1 -- 172.21.15.40:0/1349158048 wait complete. 2024-03-24T00:35:20.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03baa00640 1 Processor -- start 2024-03-24T00:35:20.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03baa00640 1 -- start start 2024-03-24T00:35:20.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03baa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03b40af170 0x7f03b40c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03baa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b40d2830 0x7f03b40c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03baa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f03b415f600 0x7f03b40ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03baa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40b2fa0 con 0x7f03b40d2830 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03baa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40b2740 con 0x7f03b40af170 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03baa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b415f9e0 con 0x7f03b415f600 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b40d2830 0x7f03b40c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b40d2830 0x7f03b40c55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55040/0 (socket says 172.21.15.40:55040) 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03abe00640 1 -- 172.21.15.40:0/3855714089 learned_addr learned my addr 172.21.15.40:0/3855714089 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03b3e00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03b40af170 0x7f03b40c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03b8a00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f03b415f600 0x7f03b40ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03abe00640 1 -- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f03b415f600 msgr2=0x7f03b40ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03abe00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f03b415f600 0x7f03b40ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03abe00640 1 -- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03b40af170 msgr2=0x7f03b40c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03abe00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03b40af170 0x7f03b40c8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03abe00640 1 -- 172.21.15.40:0/3855714089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b40c5d60 con 0x7f03b40d2830 2024-03-24T00:35:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03b8a00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f03b415f600 0x7f03b40ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03abe00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b40d2830 0x7f03b40c55c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f039c00c450 tx=0x7f039c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03b0c00640 1 -- 172.21.15.40:0/3855714089 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c012830 con 0x7f03b40d2830 2024-03-24T00:35:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03b0c00640 1 -- 172.21.15.40:0/3855714089 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039c017070 con 0x7f03b40d2830 2024-03-24T00:35:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03b0c00640 1 -- 172.21.15.40:0/3855714089 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c012c30 con 0x7f03b40d2830 2024-03-24T00:35:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03baa00640 1 -- 172.21.15.40:0/3855714089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b40c6040 con 0x7f03b40d2830 2024-03-24T00:35:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03baa00640 1 -- 172.21.15.40:0/3855714089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b40c6600 con 0x7f03b40d2830 2024-03-24T00:35:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.690+0000 7f03baa00640 1 -- 172.21.15.40:0/3855714089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b4068170 con 0x7f03b40d2830 2024-03-24T00:35:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.694+0000 7f03b0c00640 1 -- 172.21.15.40:0/3855714089 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f039c01c460 con 0x7f03b40d2830 2024-03-24T00:35:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.694+0000 7f03b0c00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0390067660 0x7f0390069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.694+0000 7f03b0c00640 1 -- 172.21.15.40:0/3855714089 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f039c085610 con 0x7f03b40d2830 2024-03-24T00:35:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.694+0000 7f03b3e00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0390067660 0x7f0390069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.694+0000 7f03b3e00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0390067660 0x7f0390069b20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f03a40079c0 tx=0x7f03a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:20.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.698+0000 7f03b0c00640 1 -- 172.21.15.40:0/3855714089 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f039c056280 con 0x7f03b40d2830 2024-03-24T00:35:20.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.894+0000 7f03baa00640 1 -- 172.21.15.40:0/3855714089 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03b40b1ad0 con 0x7f03b40d2830 2024-03-24T00:35:20.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03b0c00640 1 -- 172.21.15.40:0/3855714089 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+4973 (secure 0 0 0) 0x7f039c055b80 con 0x7f03b40d2830 2024-03-24T00:35:20.901 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:20.901 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":67,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6062,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/4182264423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":4182264423},{"type":"v1","addr":"172.21.15.53:6836","nonce":4182264423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6733,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/334375209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":334375209},{"type":"v1","addr":"172.21.15.53:6837","nonce":334375209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6951,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/221849812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":221849812},{"type":"v1","addr":"172.21.15.40:6834","nonce":221849812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:35:14.689580+0000","modified":"2024-03-24T00:35:17.703915+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6954":{"gid":6954,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/2177098794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2177098794},{"type":"v1","addr":"172.21.15.40:6835","nonce":2177098794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6954,"qdb_cluster":[6954]},"id":5}]} 2024-03-24T00:35:20.901 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 67 2024-03-24T00:35:20.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03baa00640 1 -- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0390067660 msgr2=0x7f0390069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03baa00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0390067660 0x7f0390069b20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f03a40079c0 tx=0x7f03a4008040 comp rx=0 tx=0).stop 2024-03-24T00:35:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03baa00640 1 -- 172.21.15.40:0/3855714089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b40d2830 msgr2=0x7f03b40c55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03baa00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b40d2830 0x7f03b40c55c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f039c00c450 tx=0x7f039c00c910 comp rx=0 tx=0).stop 2024-03-24T00:35:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03baa00640 1 -- 172.21.15.40:0/3855714089 shutdown_connections 2024-03-24T00:35:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03baa00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0390067660 0x7f0390069b20 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03baa00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f03b415f600 0x7f03b40ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03baa00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f03b40d2830 0x7f03b40c55c0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03baa00640 1 --2- 172.21.15.40:0/3855714089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03b40af170 0x7f03b40c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03baa00640 1 -- 172.21.15.40:0/3855714089 >> 172.21.15.40:0/3855714089 conn(0x7f03b40d1d10 msgr2=0x7f03b415d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03baa00640 1 -- 172.21.15.40:0/3855714089 shutdown_connections 2024-03-24T00:35:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:20.898+0000 7f03baa00640 1 -- 172.21.15.40:0/3855714089 wait complete. 2024-03-24T00:35:21.325 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:35:14.689580+0000', 'modified': '2024-03-24T00:35:17.703915+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6954': {'gid': 6954, 'name': 'a', 'rank': 0, 'incarnation': 63, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/2177098794', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 2177098794}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 2177098794}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6954, 'qdb_cluster': [6954]} 2024-03-24T00:35:21.325 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:35:21.325 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:35:21.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.462+0000 7f6a6fe00640 1 Processor -- start 2024-03-24T00:35:21.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a6fe00640 1 -- start start 2024-03-24T00:35:21.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a6fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6a6815f600 0x7f6a680b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a6fe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a680b1e50 con 0x7f6a680af170 2024-03-24T00:35:21.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a6fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a680b20f0 con 0x7f6a680d2830 2024-03-24T00:35:21.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a6fe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a680b3630 con 0x7f6a6815f600 2024-03-24T00:35:21.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a6de00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6a680af170 0x7f6a68165e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36872/0 (socket says 172.21.15.40:36872) 2024-03-24T00:35:21.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a6de00640 1 -- 172.21.15.40:0/1606851577 learned_addr learned my addr 172.21.15.40:0/1606851577 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:21.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a6d400640 1 --2- 172.21.15.40:0/1606851577 >> v2:172.21.15.53:3300/0 conn(0x7f6a6815f600 0x7f6a680b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a67e00640 1 -- 172.21.15.40:0/1606851577 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150906829 0 0) 0x7f6a680b1e50 con 0x7f6a680af170 2024-03-24T00:35:21.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a67e00640 1 -- 172.21.15.40:0/1606851577 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a50003530 con 0x7f6a680af170 2024-03-24T00:35:21.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a6d400640 1 -- 172.21.15.40:0/1606851577 >> v1:172.21.15.53:6789/0 conn(0x7f6a680d2830 legacy=0x7f6a680aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a6d400640 1 -- 172.21.15.40:0/1606851577 >> v1:172.21.15.40:6789/0 conn(0x7f6a680af170 legacy=0x7f6a68165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.466+0000 7f6a6d400640 1 -- 172.21.15.40:0/1606851577 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680b23d0 con 0x7f6a6815f600 2024-03-24T00:35:21.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.470+0000 7f6a6d400640 1 --2- 172.21.15.40:0/1606851577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a6815f600 0x7f6a680b2910 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0097c0 tx=0x7f6a5c01ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e83cd0d8c076e86f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.470+0000 7f6a67e00640 1 -- 172.21.15.40:0/1606851577 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c0b5070 con 0x7f6a6815f600 2024-03-24T00:35:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.470+0000 7f6a67e00640 1 -- 172.21.15.40:0/1606851577 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c0a8990 con 0x7f6a6815f600 2024-03-24T00:35:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.470+0000 7f6a67e00640 1 -- 172.21.15.40:0/1606851577 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c0a8d60 con 0x7f6a6815f600 2024-03-24T00:35:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.470+0000 7f6a6fe00640 1 -- 172.21.15.40:0/1606851577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a6815f600 msgr2=0x7f6a680b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.470+0000 7f6a6fe00640 1 --2- 172.21.15.40:0/1606851577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a6815f600 0x7f6a680b2910 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0097c0 tx=0x7f6a5c01ec90 comp rx=0 tx=0).stop 2024-03-24T00:35:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.470+0000 7f6a6fe00640 1 -- 172.21.15.40:0/1606851577 shutdown_connections 2024-03-24T00:35:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.470+0000 7f6a6fe00640 1 --2- 172.21.15.40:0/1606851577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a6815f600 0x7f6a680b2910 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.470+0000 7f6a6fe00640 1 -- 172.21.15.40:0/1606851577 >> 172.21.15.40:0/1606851577 conn(0x7f6a680d1d10 msgr2=0x7f6a680d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:21.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.470+0000 7f6a6fe00640 1 -- 172.21.15.40:0/1606851577 shutdown_connections 2024-03-24T00:35:21.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.470+0000 7f6a6fe00640 1 -- 172.21.15.40:0/1606851577 wait complete. 2024-03-24T00:35:21.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6fe00640 1 Processor -- start 2024-03-24T00:35:21.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6fe00640 1 -- start start 2024-03-24T00:35:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a680af170 0x7f6a6815b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a680d2830 0x7f6a6814d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815f600 0x7f6a680c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680b2740 con 0x7f6a680d2830 2024-03-24T00:35:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680d2ca0 con 0x7f6a680af170 2024-03-24T00:35:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6815f9e0 con 0x7f6a6815f600 2024-03-24T00:35:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a680d2830 0x7f6a6814d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a680d2830 0x7f6a6814d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55044/0 (socket says 172.21.15.40:55044) 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6ca00640 1 -- 172.21.15.40:0/530479354 learned_addr learned my addr 172.21.15.40:0/530479354 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6d400640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a680af170 0x7f6a6815b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6de00640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815f600 0x7f6a680c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6ca00640 1 -- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815f600 msgr2=0x7f6a680c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6ca00640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815f600 0x7f6a680c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6ca00640 1 -- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a680af170 msgr2=0x7f6a6815b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6ca00640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a680af170 0x7f6a6815b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6ca00640 1 -- 172.21.15.40:0/530479354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680c8de0 con 0x7f6a680d2830 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6d400640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a680af170 0x7f6a6815b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6ca00640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a680d2830 0x7f6a6814d0e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6a5400c500 tx=0x7f6a5400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a65600640 1 -- 172.21.15.40:0/530479354 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a54016020 con 0x7f6a680d2830 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a65600640 1 -- 172.21.15.40:0/530479354 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a54005220 con 0x7f6a680d2830 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a65600640 1 -- 172.21.15.40:0/530479354 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a54007030 con 0x7f6a680d2830 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6fe00640 1 -- 172.21.15.40:0/530479354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a680c90c0 con 0x7f6a680d2830 2024-03-24T00:35:21.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.474+0000 7f6a6fe00640 1 -- 172.21.15.40:0/530479354 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a680c9680 con 0x7f6a680d2830 2024-03-24T00:35:21.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a65600640 1 -- 172.21.15.40:0/530479354 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6a540071d0 con 0x7f6a680d2830 2024-03-24T00:35:21.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a65600640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a48067660 0x7f6a48069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a65600640 1 -- 172.21.15.40:0/530479354 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a540853f0 con 0x7f6a680d2830 2024-03-24T00:35:21.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a6d400640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a48067660 0x7f6a48069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a73207000 1 Processor -- start 2024-03-24T00:35:21.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a73207000 1 -- start start 2024-03-24T00:35:21.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a6d400640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a48067660 0x7f6a48069b20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0b2000 tx=0x7f6a5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a73207000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 0x562ad0e65d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a73207000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e66240 0x562ad0e1bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a73207000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e1bff0 0x562ad0e1c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a73207000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ad0c69680 con 0x562ad0e65920 2024-03-24T00:35:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a73207000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ad0bf0b00 con 0x562ad0e66240 2024-03-24T00:35:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a73207000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ad0c65720 con 0x562ad0e1bff0 2024-03-24T00:35:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a6d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 0x562ad0e65d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a6d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 0x562ad0e65d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55050/0 (socket says 172.21.15.40:55050) 2024-03-24T00:35:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.478+0000 7f6a6d400640 1 -- 172.21.15.40:0/2503527375 learned_addr learned my addr 172.21.15.40:0/2503527375 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a6de00640 1 --2- 172.21.15.40:0/2503527375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e1bff0 0x562ad0e1c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a6ca00640 1 --2- 172.21.15.40:0/2503527375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e66240 0x562ad0e1bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a6d400640 1 -- 172.21.15.40:0/2503527375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e1bff0 msgr2=0x562ad0e1c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a6d400640 1 --2- 172.21.15.40:0/2503527375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e1bff0 0x562ad0e1c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a6d400640 1 -- 172.21.15.40:0/2503527375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e66240 msgr2=0x562ad0e1bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a6d400640 1 --2- 172.21.15.40:0/2503527375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e66240 0x562ad0e1bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a6d400640 1 -- 172.21.15.40:0/2503527375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ad0c41940 con 0x562ad0e65920 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a6d400640 1 --2- 172.21.15.40:0/2503527375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 0x562ad0e65d00 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c008010 tx=0x7f6a5c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596df6ea1f24c19e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a5a000640 1 -- 172.21.15.40:0/2503527375 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c0cc070 con 0x562ad0e65920 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a5a000640 1 -- 172.21.15.40:0/2503527375 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c0b6040 con 0x562ad0e65920 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a5a000640 1 -- 172.21.15.40:0/2503527375 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c0b0780 con 0x562ad0e65920 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 -- 172.21.15.40:0/2503527375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 msgr2=0x562ad0e65d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 --2- 172.21.15.40:0/2503527375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 0x562ad0e65d00 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c008010 tx=0x7f6a5c004740 comp rx=0 tx=0).stop 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 -- 172.21.15.40:0/2503527375 shutdown_connections 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 --2- 172.21.15.40:0/2503527375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e1bff0 0x562ad0e1c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 --2- 172.21.15.40:0/2503527375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e66240 0x562ad0e1bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 --2- 172.21.15.40:0/2503527375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 0x562ad0e65d00 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 -- 172.21.15.40:0/2503527375 >> 172.21.15.40:0/2503527375 conn(0x562ad0e17fa0 msgr2=0x562ad0e1a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:21.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 -- 172.21.15.40:0/2503527375 shutdown_connections 2024-03-24T00:35:21.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 -- 172.21.15.40:0/2503527375 wait complete. 2024-03-24T00:35:21.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 -- start start 2024-03-24T00:35:21.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 Processor -- start 2024-03-24T00:35:21.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e1bff0 0x562ad0e4ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e65920 0x562ad0e4b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e66240 0x562ad0e4b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ad0c68990 con 0x562ad0e66240 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ad0c23ae0 con 0x562ad0e1bff0 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ad0c01750 con 0x562ad0e65920 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a73207000 10 client.0 _ll_register_callbacks cb 0x562ad0e06ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a6de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e66240 0x562ad0e4b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a6de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e66240 0x562ad0e4b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55056/0 (socket says 172.21.15.40:55056) 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a6de00640 1 -- 172.21.15.40:0/2791192149 learned_addr learned my addr 172.21.15.40:0/2791192149 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.482+0000 7f6a6d400640 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e1bff0 0x562ad0e4ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a6ca00640 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e65920 0x562ad0e4b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a6de00640 1 -- 172.21.15.40:0/2791192149 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e65920 msgr2=0x562ad0e4b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a6de00640 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e65920 0x562ad0e4b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a6de00640 1 -- 172.21.15.40:0/2791192149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e1bff0 msgr2=0x562ad0e4ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a6de00640 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e1bff0 0x562ad0e4ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a6de00640 1 -- 172.21.15.40:0/2791192149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ad0caed80 con 0x562ad0e66240 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a6de00640 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e66240 0x562ad0e4b7c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f6a60011940 tx=0x7f6a60011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8640387a2b791684 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a4ea00640 1 -- 172.21.15.40:0/2791192149 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6000f750 con 0x562ad0e66240 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a4ea00640 1 -- 172.21.15.40:0/2791192149 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a6000f8f0 con 0x562ad0e66240 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 -- 172.21.15.40:0/2791192149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a5c0b6040 con 0x562ad0e66240 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a4ea00640 1 -- 172.21.15.40:0/2791192149 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a60022050 con 0x562ad0e66240 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 10 client.7086 fetch_fsmap learned FSMap version 67 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 -- 172.21.15.40:0/2791192149 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x562ad0bf0d10 con 0x562ad0e66240 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a4ea00640 1 -- 172.21.15.40:0/2791192149 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c0b6040 con 0x562ad0e66240 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a4ea00640 1 -- 172.21.15.40:0/2791192149 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 67) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6a60025020 con 0x562ad0e66240 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 10 client.7086 fetch_fsmap finished waiting for FSMap version 67 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 10 client.7086 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 20 client.7086 populate_metadata read hostname 'smithi040' 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x562ad0e4c060 0x562ad0e77ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 4 client.7086 mds_command: new command op to 6954 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:35:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 -- 172.21.15.40:0/2791192149 --> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ad0c04b60 con 0x562ad0e4c060 2024-03-24T00:35:21.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a6d400640 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x562ad0e4c060 0x562ad0e77ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a6d400640 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x562ad0e4c060 0x562ad0e77ea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d4e25c8b5ed8949 server_cookie=f613d063c5613775 in_seq=0 out_seq=0 2024-03-24T00:35:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a4ea00640 10 client.7086 ms_handle_connect on v2:172.21.15.40:6833/2177098794 2024-03-24T00:35:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a4ea00640 1 -- 172.21.15.40:0/2791192149 <== mds.0 v2:172.21.15.40:6833/2177098794 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562ad0c04b60 con 0x562ad0e4c060 2024-03-24T00:35:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a4ea00640 10 client.7086 handle_command_reply: tid=0 2024-03-24T00:35:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 client.7086 shutdown 2024-03-24T00:35:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 -- 172.21.15.40:0/2791192149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e66240 msgr2=0x562ad0e4b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e66240 0x562ad0e4b7c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f6a60011940 tx=0x7f6a60011e00 comp rx=0 tx=0).stop 2024-03-24T00:35:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 -- 172.21.15.40:0/2791192149 shutdown_connections 2024-03-24T00:35:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x562ad0e4c060 0x562ad0e77ea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e66240 0x562ad0e4b7c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e65920 0x562ad0e4b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a4ea00640 0 client.7086 ms_handle_reset on v2:172.21.15.40:6833/2177098794 2024-03-24T00:35:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 --2- 172.21.15.40:0/2791192149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e1bff0 0x562ad0e4ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 -- 172.21.15.40:0/2791192149 >> 172.21.15.40:0/2791192149 conn(0x562ad0e17fa0 msgr2=0x562ad0e1ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 -- 172.21.15.40:0/2791192149 shutdown_connections 2024-03-24T00:35:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 1 -- 172.21.15.40:0/2791192149 wait complete. 2024-03-24T00:35:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.486+0000 7f6a73207000 20 client.7086 trim_cache size 0 max 16384 2024-03-24T00:35:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 Processor -- start 2024-03-24T00:35:21.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 -- start start 2024-03-24T00:35:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e1bff0 0x562ad0e64550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e65920 0x562ad0e64a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e66240 0x562ad0e485d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68200180 con 0x562ad0e1bff0 2024-03-24T00:35:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6809c8e0 con 0x562ad0e65920 2024-03-24T00:35:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6809bce0 con 0x562ad0e66240 2024-03-24T00:35:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e1bff0 0x562ad0e64550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e1bff0 0x562ad0e64550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55070/0 (socket says 172.21.15.40:55070) 2024-03-24T00:35:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6d400640 1 -- 172.21.15.40:0/1164326013 learned_addr learned my addr 172.21.15.40:0/1164326013 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6ca00640 1 --2- 172.21.15.40:0/1164326013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e65920 0x562ad0e64a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6d400640 1 -- 172.21.15.40:0/1164326013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e66240 msgr2=0x562ad0e485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6de00640 1 --2- 172.21.15.40:0/1164326013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e66240 0x562ad0e485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6d400640 1 --2- 172.21.15.40:0/1164326013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e66240 0x562ad0e485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6d400640 1 -- 172.21.15.40:0/1164326013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e65920 msgr2=0x562ad0e64a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6d400640 1 --2- 172.21.15.40:0/1164326013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e65920 0x562ad0e64a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6d400640 1 -- 172.21.15.40:0/1164326013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ad0c36540 con 0x562ad0e1bff0 2024-03-24T00:35:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6ca00640 1 --2- 172.21.15.40:0/1164326013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e65920 0x562ad0e64a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:35:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6de00640 1 --2- 172.21.15.40:0/1164326013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e66240 0x562ad0e485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:35:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a6d400640 1 --2- 172.21.15.40:0/1164326013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e1bff0 0x562ad0e64550 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c008010 tx=0x7f6a5c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3963ea8537e4d942 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a59600640 1 -- 172.21.15.40:0/1164326013 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c0cc040 con 0x562ad0e1bff0 2024-03-24T00:35:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a59600640 1 -- 172.21.15.40:0/1164326013 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c0b6040 con 0x562ad0e1bff0 2024-03-24T00:35:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a59600640 1 -- 172.21.15.40:0/1164326013 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5c0b09b0 con 0x562ad0e1bff0 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 -- 172.21.15.40:0/1164326013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e1bff0 msgr2=0x562ad0e64550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 --2- 172.21.15.40:0/1164326013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e1bff0 0x562ad0e64550 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c008010 tx=0x7f6a5c009190 comp rx=0 tx=0).stop 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 -- 172.21.15.40:0/1164326013 shutdown_connections 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 --2- 172.21.15.40:0/1164326013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e66240 0x562ad0e485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 --2- 172.21.15.40:0/1164326013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e65920 0x562ad0e64a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 --2- 172.21.15.40:0/1164326013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e1bff0 0x562ad0e64550 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 -- 172.21.15.40:0/1164326013 >> 172.21.15.40:0/1164326013 conn(0x562ad0e17fa0 msgr2=0x562ad0e72f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 -- 172.21.15.40:0/1164326013 shutdown_connections 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 -- 172.21.15.40:0/1164326013 wait complete. 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 -- start start 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.506+0000 7f6a73207000 1 Processor -- start 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e1bff0 0x562ad0e1b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 0x562ad0e78ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e66240 0x562ad0e79430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ad0c11a80 con 0x562ad0e65920 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6809c2e0 con 0x562ad0e1bff0 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ad0d26700 con 0x562ad0e66240 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 10 client.0 _ll_register_callbacks cb 0x7f6a600115a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 0x562ad0e78ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 0x562ad0e78ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55072/0 (socket says 172.21.15.40:55072) 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6ca00640 1 -- 172.21.15.40:0/842834818 learned_addr learned my addr 172.21.15.40:0/842834818 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6de00640 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e66240 0x562ad0e79430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6d400640 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e1bff0 0x562ad0e1b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6ca00640 1 -- 172.21.15.40:0/842834818 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e66240 msgr2=0x562ad0e79430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6ca00640 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e66240 0x562ad0e79430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6ca00640 1 -- 172.21.15.40:0/842834818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e1bff0 msgr2=0x562ad0e1b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6ca00640 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e1bff0 0x562ad0e1b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6ca00640 1 -- 172.21.15.40:0/842834818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ad0bfd4b0 con 0x562ad0e65920 2024-03-24T00:35:21.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6ca00640 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 0x562ad0e78ef0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f6a540012c0 tx=0x7f6a540559a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11aa11bfa23e73a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a4ea00640 1 -- 172.21.15.40:0/842834818 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a54051f50 con 0x562ad0e65920 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a4ea00640 1 -- 172.21.15.40:0/842834818 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a54085070 con 0x562ad0e65920 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 -- 172.21.15.40:0/842834818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a5c0b6040 con 0x562ad0e65920 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a4ea00640 1 -- 172.21.15.40:0/842834818 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=67) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a540520f0 con 0x562ad0e65920 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 10 client.7092 fetch_fsmap learned FSMap version 67 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 -- 172.21.15.40:0/842834818 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=67,monmap=2+}) v3 -- 0x562ad0c102f0 con 0x562ad0e65920 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a4ea00640 1 -- 172.21.15.40:0/842834818 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a5c0b6040 con 0x562ad0e65920 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a4ea00640 1 -- 172.21.15.40:0/842834818 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 67) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f6a5405db60 con 0x562ad0e65920 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 10 client.7092 fetch_fsmap finished waiting for FSMap version 67 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 10 client.7092 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 20 client.7092 populate_metadata read hostname 'smithi040' 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x562ad0eca5c0 0x562ad0eca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 4 client.7092 mds_command: new command op to 6954 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:35:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 -- 172.21.15.40:0/842834818 --> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562ad0c76bc0 con 0x562ad0eca5c0 2024-03-24T00:35:21.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6d400640 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x562ad0eca5c0 0x562ad0eca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a6d400640 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x562ad0eca5c0 0x562ad0eca9a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d7db024f7799240 server_cookie=6c2d045ace3f48dd in_seq=0 out_seq=0 2024-03-24T00:35:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a4ea00640 10 client.7092 ms_handle_connect on v2:172.21.15.40:6833/2177098794 2024-03-24T00:35:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a4ea00640 1 -- 172.21.15.40:0/842834818 <== mds.0 v2:172.21.15.40:6833/2177098794 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x562ad0c76bc0 con 0x562ad0eca5c0 2024-03-24T00:35:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a4ea00640 10 client.7092 handle_command_reply: tid=0 2024-03-24T00:35:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 client.7092 shutdown 2024-03-24T00:35:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 -- 172.21.15.40:0/842834818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 msgr2=0x562ad0e78ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.510+0000 7f6a73207000 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 0x562ad0e78ef0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f6a540012c0 tx=0x7f6a540559a0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a73207000 1 -- 172.21.15.40:0/842834818 shutdown_connections 2024-03-24T00:35:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a73207000 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x562ad0eca5c0 0x562ad0eca9a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a73207000 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x562ad0e66240 0x562ad0e79430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a73207000 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x562ad0e65920 0x562ad0e78ef0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a73207000 1 --2- 172.21.15.40:0/842834818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x562ad0e1bff0 0x562ad0e1b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a4ea00640 0 client.7092 ms_handle_reset on v2:172.21.15.40:6833/2177098794 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a73207000 1 -- 172.21.15.40:0/842834818 >> 172.21.15.40:0/842834818 conn(0x562ad0e17fa0 msgr2=0x562ad0e485d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a73207000 1 -- 172.21.15.40:0/842834818 shutdown_connections 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a73207000 1 -- 172.21.15.40:0/842834818 wait complete. 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a73207000 20 client.7092 trim_cache size 0 max 16384 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6954, 2024-03-24T00:35:21.520 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:35:21.521 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:35:21.521 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:35:21.521 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 6.2143074660000002, 2024-03-24T00:35:21.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 67, 2024-03-24T00:35:21.521 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 70, 2024-03-24T00:35:21.521 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 70, 2024-03-24T00:35:21.521 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 11.069395901 2024-03-24T00:35:21.521 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:35:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a6fe00640 1 -- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a48067660 msgr2=0x7f6a48069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a6fe00640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a48067660 0x7f6a48069b20 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c0b2000 tx=0x7f6a5c008040 comp rx=0 tx=0).stop 2024-03-24T00:35:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a6fe00640 1 -- 172.21.15.40:0/530479354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a680d2830 msgr2=0x7f6a6814d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a6fe00640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a680d2830 0x7f6a6814d0e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f6a5400c500 tx=0x7f6a5400c9c0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a6fe00640 1 -- 172.21.15.40:0/530479354 shutdown_connections 2024-03-24T00:35:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a6fe00640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6a48067660 0x7f6a48069b20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a6fe00640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6a6815f600 0x7f6a680c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a6fe00640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6a680d2830 0x7f6a6814d0e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a6fe00640 1 --2- 172.21.15.40:0/530479354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a680af170 0x7f6a6815b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a6fe00640 1 -- 172.21.15.40:0/530479354 >> 172.21.15.40:0/530479354 conn(0x7f6a680d1d10 msgr2=0x7f6a6806c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:21.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a6fe00640 1 -- 172.21.15.40:0/530479354 shutdown_connections 2024-03-24T00:35:21.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.514+0000 7f6a6fe00640 1 -- 172.21.15.40:0/530479354 wait complete. 2024-03-24T00:35:21.547 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:35:21.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.682+0000 7fcbb0a00640 1 Processor -- start 2024-03-24T00:35:21.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.682+0000 7fcbb0a00640 1 -- start start 2024-03-24T00:35:21.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcbb0a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcbac1628a0 0x7fcbac0d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcbb0a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcbac161c90 0x7fcbac1620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcbb0a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbac06c240 con 0x7fcbac15e100 2024-03-24T00:35:21.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcbb0a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbac06c4b0 con 0x7fcbac1628a0 2024-03-24T00:35:21.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcbb0a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac1625e0 con 0x7fcbac161c90 2024-03-24T00:35:21.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcba1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcbac1628a0 0x7fcbac0d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcba1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcbac1628a0 0x7fcbac0d0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55088/0 (socket says 172.21.15.40:55088) 2024-03-24T00:35:21.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcba1400640 1 -- 172.21.15.40:0/1543075149 learned_addr learned my addr 172.21.15.40:0/1543075149 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:21.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcbaa800640 1 --2- 172.21.15.40:0/1543075149 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcbac161c90 0x7fcbac1620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcba1400640 1 -- 172.21.15.40:0/1543075149 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcbac161c90 msgr2=0x7fcbac1620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:21.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcba1400640 1 --2- 172.21.15.40:0/1543075149 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcbac161c90 0x7fcbac1620a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcba1400640 1 -- 172.21.15.40:0/1543075149 >> v1:172.21.15.53:6789/0 conn(0x7fcbac15e100 legacy=0x7fcbac06bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.686+0000 7fcba1400640 1 -- 172.21.15.40:0/1543075149 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0cffe0 con 0x7fcbac1628a0 2024-03-24T00:35:21.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcba1400640 1 --2- 172.21.15.40:0/1543075149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac1628a0 0x7fcbac0d0da0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009e50 tx=0x7fcb9401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd3ae5a476b327b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:21.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcba9400640 1 -- 172.21.15.40:0/1543075149 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb940a5700 con 0x7fcbac1628a0 2024-03-24T00:35:21.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcba9400640 1 -- 172.21.15.40:0/1543075149 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb940a58a0 con 0x7fcbac1628a0 2024-03-24T00:35:21.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcba9400640 1 -- 172.21.15.40:0/1543075149 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb940a5c70 con 0x7fcbac1628a0 2024-03-24T00:35:21.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 -- 172.21.15.40:0/1543075149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac1628a0 msgr2=0x7fcbac0d0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 --2- 172.21.15.40:0/1543075149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac1628a0 0x7fcbac0d0da0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009e50 tx=0x7fcb9401ca30 comp rx=0 tx=0).stop 2024-03-24T00:35:21.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 -- 172.21.15.40:0/1543075149 shutdown_connections 2024-03-24T00:35:21.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 --2- 172.21.15.40:0/1543075149 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcbac161c90 0x7fcbac1620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 --2- 172.21.15.40:0/1543075149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac1628a0 0x7fcbac0d0da0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 -- 172.21.15.40:0/1543075149 >> 172.21.15.40:0/1543075149 conn(0x7fcbac0d1cb0 msgr2=0x7fcbac0d20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:21.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 -- 172.21.15.40:0/1543075149 shutdown_connections 2024-03-24T00:35:21.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 -- 172.21.15.40:0/1543075149 wait complete. 2024-03-24T00:35:21.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 Processor -- start 2024-03-24T00:35:21.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 -- start start 2024-03-24T00:35:21.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcbac15e100 0x7fcbac0c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac161c90 0x7fcbac154d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbac1628a0 0x7fcbac155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac06f670 con 0x7fcbac161c90 2024-03-24T00:35:21.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac15e4e0 con 0x7fcbac1628a0 2024-03-24T00:35:21.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.690+0000 7fcbb0a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac06f7f0 con 0x7fcbac15e100 2024-03-24T00:35:21.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac161c90 0x7fcbac154d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac161c90 0x7fcbac154d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55090/0 (socket says 172.21.15.40:55090) 2024-03-24T00:35:21.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba1400640 1 -- 172.21.15.40:0/3851211436 learned_addr learned my addr 172.21.15.40:0/3851211436 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:21.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba9e00640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcbac15e100 0x7fcbac0c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcbaa800640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbac1628a0 0x7fcbac155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba1400640 1 -- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcbac15e100 msgr2=0x7fcbac0c8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba1400640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcbac15e100 0x7fcbac0c8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba1400640 1 -- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbac1628a0 msgr2=0x7fcbac155280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba1400640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbac1628a0 0x7fcbac155280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba1400640 1 -- 172.21.15.40:0/3851211436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac15cbc0 con 0x7fcbac161c90 2024-03-24T00:35:21.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba1400640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac161c90 0x7fcbac154d40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fcb940af040 tx=0x7fcb940197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:21.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba2a00640 1 -- 172.21.15.40:0/3851211436 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb940b8470 con 0x7fcbac161c90 2024-03-24T00:35:21.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba2a00640 1 -- 172.21.15.40:0/3851211436 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb940b8650 con 0x7fcbac161c90 2024-03-24T00:35:21.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba2a00640 1 -- 172.21.15.40:0/3851211436 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb940b8a40 con 0x7fcbac161c90 2024-03-24T00:35:21.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcbb0a00640 1 -- 172.21.15.40:0/3851211436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbac15ce60 con 0x7fcbac161c90 2024-03-24T00:35:21.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcbb0a00640 1 -- 172.21.15.40:0/3851211436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbac15d420 con 0x7fcbac161c90 2024-03-24T00:35:21.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcbb0a00640 1 -- 172.21.15.40:0/3851211436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbac068170 con 0x7fcbac161c90 2024-03-24T00:35:21.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.694+0000 7fcba2a00640 1 -- 172.21.15.40:0/3851211436 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcb940c0020 con 0x7fcbac161c90 2024-03-24T00:35:21.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.698+0000 7fcba2a00640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcb88067660 0x7fcb88069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:21.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.698+0000 7fcba2a00640 1 -- 172.21.15.40:0/3851211436 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb94120d50 con 0x7fcbac161c90 2024-03-24T00:35:21.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.698+0000 7fcba9e00640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcb88067660 0x7fcb88069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:21.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.698+0000 7fcba9e00640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcb88067660 0x7fcb88069b20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c007fd0 tx=0x7fcb9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:21.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.698+0000 7fcba2a00640 1 -- 172.21.15.40:0/3851211436 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcb94016620 con 0x7fcbac161c90 2024-03-24T00:35:21.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.898+0000 7fcbb0a00640 1 -- 172.21.15.40:0/3851211436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcbac0d1760 con 0x7fcbac161c90 2024-03-24T00:35:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.898+0000 7fcba2a00640 1 -- 172.21.15.40:0/3851211436 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v67) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcb940f12d0 con 0x7fcbac161c90 2024-03-24T00:35:21.902 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:35:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.902+0000 7fcbb0a00640 1 -- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcb88067660 msgr2=0x7fcb88069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.902+0000 7fcbb0a00640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcb88067660 0x7fcb88069b20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c007fd0 tx=0x7fcb9c009040 comp rx=0 tx=0).stop 2024-03-24T00:35:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.902+0000 7fcbb0a00640 1 -- 172.21.15.40:0/3851211436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac161c90 msgr2=0x7fcbac154d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:21.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.902+0000 7fcbb0a00640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac161c90 0x7fcbac154d40 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fcb940af040 tx=0x7fcb940197d0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.902+0000 7fcbb0a00640 1 -- 172.21.15.40:0/3851211436 shutdown_connections 2024-03-24T00:35:21.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.902+0000 7fcbb0a00640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcb88067660 0x7fcb88069b20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.902+0000 7fcbb0a00640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbac1628a0 0x7fcbac155280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.902+0000 7fcbb0a00640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcbac161c90 0x7fcbac154d40 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.902+0000 7fcbb0a00640 1 --2- 172.21.15.40:0/3851211436 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcbac15e100 0x7fcbac0c8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:21.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.902+0000 7fcbb0a00640 1 -- 172.21.15.40:0/3851211436 >> 172.21.15.40:0/3851211436 conn(0x7fcbac0d1cb0 msgr2=0x7fcbac0c6030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:21.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.902+0000 7fcbb0a00640 1 -- 172.21.15.40:0/3851211436 shutdown_connections 2024-03-24T00:35:21.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:21.902+0000 7fcbb0a00640 1 -- 172.21.15.40:0/3851211436 wait complete. 2024-03-24T00:35:21.933 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:35:21.933 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:35:21.933 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-03-24T00:35:21.933 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:35:21.933 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:35:21.934 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:35:21.934 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:35:21.934 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:35:21.934 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:35:21.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:21.934 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-03-24T00:35:21.944 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:35:21.944 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:35:21.944 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:35:21.944 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:35:21.944 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-24T00:35:21.944 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:35:21.944 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42228sec preferred_lft 42228sec 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:35:21.945 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 02:42:9e:ab:82:f9 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-24T00:35:21.946 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:35:21.947 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:35:21.947 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:35:21.947 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-24T00:35:21.947 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:35:21.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:21.948 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:35:22.010 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-03-24T00:35:22.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:22.012 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-03-24T00:35:22.064 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-03-24T00:35:22.064 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:35:22.065 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-24T00:35:22.065 INFO:teuthology.orchestra.run.smithi040.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-24T00:35:22.065 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-24T00:35:22.065 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:35:22.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:22.065 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:35:22.065 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:35:22.065 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T00:35:22.065 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T00:35:22.065 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T00:35:22.066 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:35:22.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:22.158 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-24T00:35:22.166 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:35:22.166 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:35:22.167 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:35:22.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:22.167 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-03-24T00:35:22.232 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T00:35:22.232 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:35:22.232 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:35:22.288 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:35:22.289 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:35:22.289 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:35:22.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:35:22.405 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:35:22.457 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:35:22.458 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:35:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:35:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:35:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:35:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:35:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:35:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:35:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:35:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:35:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:35:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:35:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:35:22.459 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:35:22.460 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:35:22.460 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:35:22.460 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:35:22.460 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:35:22.460 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:35:22.461 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:35:22.461 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:35:22.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:22.462 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:35:22.511 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T00:35:22.511 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T00:35:22.558 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:35:22.581 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:35:22.637 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:35:22.637 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:35:22.637 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:35:22.637 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:35:22.637 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:35:22.637 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:35:22.637 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:35:22.638 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:35:22.639 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:35:22.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:22.640 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:35:22.655 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:2024-03-24T00:35:22.650+0000 7ffbac56c580 -1 init, newargv = 0x56409a40aac0 newargc=15 2024-03-24T00:35:22.655 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[44485]: starting ceph client 2024-03-24T00:35:22.664 INFO:teuthology.orchestra.run.smithi040.stdout:43 2024-03-24T00:35:22.665 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[44485]: starting fuse 2024-03-24T00:35:22.665 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-24T00:35:22.665 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:35:22.666 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-03-24T00:35:22.666 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-03-24T00:35:22.666 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-03-24T00:35:22.666 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-03-24T00:35:22.666 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:35:22.666 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-03-24T00:35:22.666 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T00:35:22.667 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-03-24T00:35:22.667 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T00:35:22.667 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:35:22.667 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-03-24T00:35:22.667 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-03-24T00:35:22.667 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-03-24T00:35:22.667 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-03-24T00:35:22.667 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:35:22.667 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-03-24T00:35:22.668 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:35:22.668 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:35:22.668 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:35:22.668 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-03-24T00:35:22.668 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-03-24T00:35:22.668 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-03-24T00:35:22.668 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:35:22.669 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:35:22.669 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.0")) 2024-03-24T00:35:22.669 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-03-24T00:35:22.793 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.0.44485.asok 2024-03-24T00:35:22.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:35:22.805 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44485.asok 2024-03-24T00:35:22.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:22.805 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44485.asok status 2024-03-24T00:35:22.978 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "0", 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "44485", 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6883, 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-03-24T00:35:22.979 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 6883 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1292826326 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1292826326 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.6883 192.168.0.1:0/1292826326", 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "192.168.0.1:0/1292826326", 2024-03-24T00:35:22.980 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-03-24T00:35:22.981 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 67, 2024-03-24T00:35:22.981 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 70, 2024-03-24T00:35:22.981 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:35:22.981 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-03-24T00:35:22.981 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-03-24T00:35:22.981 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:35:22.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:22.994 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-03-24T00:35:22.994 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T00:35:23.003 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-03-24T00:35:23.003 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T00:35:23.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:23.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:35:23.068 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random True 2024-03-24T00:35:23.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.254+0000 7f668d200640 1 Processor -- start 2024-03-24T00:35:23.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.254+0000 7f668d200640 1 -- start start 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f668d200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6688160080 0x7f6688168ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f668d200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66880cfcb0 con 0x7f668815e170 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f668d200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66880d1120 con 0x7f668806f7b0 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f668d200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668806fc00 con 0x7f6688160080 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f6685e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f668815e170 0x7f66880d0a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36874/0 (socket says 172.21.15.40:36874) 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f6685e00640 1 -- 172.21.15.40:0/2412318531 learned_addr learned my addr 172.21.15.40:0/2412318531 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f6686800640 1 --2- 172.21.15.40:0/2412318531 >> v2:172.21.15.40:3300/0 conn(0x7f6688160080 0x7f6688168ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f6685400640 1 -- 172.21.15.40:0/2412318531 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046581520 0 0) 0x7f6678001120 con 0x7f668815e170 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f6685400640 1 -- 172.21.15.40:0/2412318531 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f666c003530 con 0x7f668815e170 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f6686800640 1 -- 172.21.15.40:0/2412318531 >> v1:172.21.15.53:6789/0 conn(0x7f668806f7b0 legacy=0x7f668815f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f6686800640 1 -- 172.21.15.40:0/2412318531 >> v1:172.21.15.40:6789/0 conn(0x7f668815e170 legacy=0x7f66880d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f6685400640 1 -- 172.21.15.40:0/2412318531 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874876716 0 0) 0x7f66880d1120 con 0x7f668806f7b0 2024-03-24T00:35:23.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.258+0000 7f6686800640 1 -- 172.21.15.40:0/2412318531 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668806fe20 con 0x7f6688160080 2024-03-24T00:35:23.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.262+0000 7f6686800640 1 --2- 172.21.15.40:0/2412318531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6688160080 0x7f6688168ea0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f6680009d90 tx=0x7f668001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef6f0b070902f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:23.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.262+0000 7f6685400640 1 -- 172.21.15.40:0/2412318531 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800b4070 con 0x7f6688160080 2024-03-24T00:35:23.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.262+0000 7f6685400640 1 -- 172.21.15.40:0/2412318531 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66800af600 con 0x7f6688160080 2024-03-24T00:35:23.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.262+0000 7f6685400640 1 -- 172.21.15.40:0/2412318531 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800af9d0 con 0x7f6688160080 2024-03-24T00:35:23.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.262+0000 7f668d200640 1 -- 172.21.15.40:0/2412318531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6688160080 msgr2=0x7f6688168ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.262+0000 7f668d200640 1 --2- 172.21.15.40:0/2412318531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6688160080 0x7f6688168ea0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f6680009d90 tx=0x7f668001f150 comp rx=0 tx=0).stop 2024-03-24T00:35:23.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.262+0000 7f668d200640 1 -- 172.21.15.40:0/2412318531 shutdown_connections 2024-03-24T00:35:23.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.262+0000 7f668d200640 1 --2- 172.21.15.40:0/2412318531 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6688160080 0x7f6688168ea0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.262+0000 7f668d200640 1 -- 172.21.15.40:0/2412318531 >> 172.21.15.40:0/2412318531 conn(0x7f66880b36a0 msgr2=0x7f66880b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:23.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.262+0000 7f668d200640 1 -- 172.21.15.40:0/2412318531 shutdown_connections 2024-03-24T00:35:23.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.262+0000 7f668d200640 1 -- 172.21.15.40:0/2412318531 wait complete. 2024-03-24T00:35:23.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f668d200640 1 Processor -- start 2024-03-24T00:35:23.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f668d200640 1 -- start start 2024-03-24T00:35:23.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f668d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f668806f7b0 0x7f66880b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f668d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668815e170 0x7f66880c6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f668d200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6688160080 0x7f66880c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f668d200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66880d0210 con 0x7f668815e170 2024-03-24T00:35:23.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f668d200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6688070100 con 0x7f668806f7b0 2024-03-24T00:35:23.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f668d200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66880d04f0 con 0x7f6688160080 2024-03-24T00:35:23.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f667fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668815e170 0x7f66880c6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f667fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668815e170 0x7f66880c6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55106/0 (socket says 172.21.15.40:55106) 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f667fe00640 1 -- 172.21.15.40:0/2833974735 learned_addr learned my addr 172.21.15.40:0/2833974735 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f6685e00640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f668806f7b0 0x7f66880b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f6686800640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6688160080 0x7f66880c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f667fe00640 1 -- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6688160080 msgr2=0x7f66880c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f667fe00640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6688160080 0x7f66880c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f667fe00640 1 -- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f668806f7b0 msgr2=0x7f66880b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f667fe00640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f668806f7b0 0x7f66880b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f667fe00640 1 -- 172.21.15.40:0/2833974735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66880b7910 con 0x7f668815e170 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f6686800640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6688160080 0x7f66880c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f667fe00640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668815e170 0x7f66880c6a80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f667000c480 tx=0x7f667000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f667e000640 1 -- 172.21.15.40:0/2833974735 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6670005480 con 0x7f668815e170 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f667e000640 1 -- 172.21.15.40:0/2833974735 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6670003600 con 0x7f668815e170 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f668d200640 1 -- 172.21.15.40:0/2833974735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66880c7500 con 0x7f668815e170 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f667e000640 1 -- 172.21.15.40:0/2833974735 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66700039f0 con 0x7f668815e170 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f668d200640 1 -- 172.21.15.40:0/2833974735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66880b4b30 con 0x7f668815e170 2024-03-24T00:35:23.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.266+0000 7f668d200640 1 -- 172.21.15.40:0/2833974735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6688068170 con 0x7f668815e170 2024-03-24T00:35:23.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.270+0000 7f667e000640 1 -- 172.21.15.40:0/2833974735 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6670003b90 con 0x7f668815e170 2024-03-24T00:35:23.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.270+0000 7f667e000640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6664067660 0x7f6664069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.270+0000 7f667e000640 1 -- 172.21.15.40:0/2833974735 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6670085610 con 0x7f668815e170 2024-03-24T00:35:23.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.270+0000 7f6685e00640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6664067660 0x7f6664069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.270+0000 7f6685e00640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6664067660 0x7f6664069b20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6678005890 tx=0x7f6678009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:23.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.274+0000 7f667e000640 1 -- 172.21.15.40:0/2833974735 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f66880c7500 con 0x7f668815e170 2024-03-24T00:35:23.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.454+0000 7f668d200640 1 -- 172.21.15.40:0/2833974735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random}] v 0) v1 -- 0x7f66880b7b10 con 0x7f668815e170 2024-03-24T00:35:23.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f667e000640 1 -- 172.21.15.40:0/2833974735 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random}]=0 v36)=0 v36) v1 ==== 128+0+0 (secure 0 0 0) 0x7f66700561f0 con 0x7f668815e170 2024-03-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f668d200640 1 -- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6664067660 msgr2=0x7f6664069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f668d200640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6664067660 0x7f6664069b20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f6678005890 tx=0x7f6678009040 comp rx=0 tx=0).stop 2024-03-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f668d200640 1 -- 172.21.15.40:0/2833974735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668815e170 msgr2=0x7f66880c6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f668d200640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668815e170 0x7f66880c6a80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f667000c480 tx=0x7f667000c940 comp rx=0 tx=0).stop 2024-03-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f668d200640 1 -- 172.21.15.40:0/2833974735 shutdown_connections 2024-03-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f668d200640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6664067660 0x7f6664069b20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f668d200640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6688160080 0x7f66880c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f668d200640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f668815e170 0x7f66880c6a80 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f668d200640 1 --2- 172.21.15.40:0/2833974735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f668806f7b0 0x7f66880b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f668d200640 1 -- 172.21.15.40:0/2833974735 >> 172.21.15.40:0/2833974735 conn(0x7f66880b36a0 msgr2=0x7f66880d1860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f668d200640 1 -- 172.21.15.40:0/2833974735 shutdown_connections 2024-03-24T00:35:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.458+0000 7f668d200640 1 -- 172.21.15.40:0/2833974735 wait complete. 2024-03-24T00:35:23.483 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_distributed True 2024-03-24T00:35:23.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.622+0000 7f5638600640 1 Processor -- start 2024-03-24T00:35:23.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.622+0000 7f5638600640 1 -- start start 2024-03-24T00:35:23.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5638600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56300cf010 0x7f56300ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5638600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56300c5010 0x7f5630070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5638600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f56300d1b60 0x7f56300d1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5638600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f563006b150 con 0x7f56300cf010 2024-03-24T00:35:23.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5638600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56300d0f10 con 0x7f56300d1b60 2024-03-24T00:35:23.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5638600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56300d11b0 con 0x7f56300c5010 2024-03-24T00:35:23.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5635c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56300cf010 0x7f56300ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5635200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56300c5010 0x7f5630070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5635c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f56300cf010 0x7f56300ce1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55122/0 (socket says 172.21.15.40:55122) 2024-03-24T00:35:23.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5635c00640 1 -- 172.21.15.40:0/485601245 learned_addr learned my addr 172.21.15.40:0/485601245 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:23.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5636600640 1 --2- 172.21.15.40:0/485601245 >> v2:172.21.15.53:3300/0 conn(0x7f56300d1b60 0x7f56300d1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5635c00640 1 -- 172.21.15.40:0/485601245 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56300c5010 msgr2=0x7f5630070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:23.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5635c00640 1 --2- 172.21.15.40:0/485601245 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56300c5010 0x7f5630070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5635c00640 1 -- 172.21.15.40:0/485601245 >> v2:172.21.15.53:3300/0 conn(0x7f56300d1b60 msgr2=0x7f56300d1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:23.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5635c00640 1 --2- 172.21.15.40:0/485601245 >> v2:172.21.15.53:3300/0 conn(0x7f56300d1b60 0x7f56300d1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.626+0000 7f5635c00640 1 -- 172.21.15.40:0/485601245 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56300d1490 con 0x7f56300cf010 2024-03-24T00:35:23.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.630+0000 7f5635c00640 1 --2- 172.21.15.40:0/485601245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56300cf010 0x7f56300ce1b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f56300c4a50 tx=0x7f562401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aca065b3fa6b404b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:23.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.630+0000 7f562fe00640 1 -- 172.21.15.40:0/485601245 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56240b0070 con 0x7f56300cf010 2024-03-24T00:35:23.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.630+0000 7f562fe00640 1 -- 172.21.15.40:0/485601245 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56240ab600 con 0x7f56300cf010 2024-03-24T00:35:23.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.630+0000 7f562fe00640 1 -- 172.21.15.40:0/485601245 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56240ab9d0 con 0x7f56300cf010 2024-03-24T00:35:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.630+0000 7f5638600640 1 -- 172.21.15.40:0/485601245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56300cf010 msgr2=0x7f56300ce1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.630+0000 7f5638600640 1 --2- 172.21.15.40:0/485601245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56300cf010 0x7f56300ce1b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f56300c4a50 tx=0x7f562401c520 comp rx=0 tx=0).stop 2024-03-24T00:35:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.630+0000 7f5638600640 1 -- 172.21.15.40:0/485601245 shutdown_connections 2024-03-24T00:35:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.630+0000 7f5638600640 1 --2- 172.21.15.40:0/485601245 >> v2:172.21.15.53:3300/0 conn(0x7f56300d1b60 0x7f56300d1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.630+0000 7f5638600640 1 --2- 172.21.15.40:0/485601245 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56300c5010 0x7f5630070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.630+0000 7f5638600640 1 --2- 172.21.15.40:0/485601245 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56300cf010 0x7f56300ce1b0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.630+0000 7f5638600640 1 -- 172.21.15.40:0/485601245 >> 172.21.15.40:0/485601245 conn(0x7f563006ecf0 msgr2=0x7f563006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.630+0000 7f5638600640 1 -- 172.21.15.40:0/485601245 shutdown_connections 2024-03-24T00:35:23.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5638600640 1 -- 172.21.15.40:0/485601245 wait complete. 2024-03-24T00:35:23.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5638600640 1 Processor -- start 2024-03-24T00:35:23.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5638600640 1 -- start start 2024-03-24T00:35:23.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5638600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56300c5010 0x7f5630155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5638600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56300cf010 0x7f56300c5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5638600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56300d1b60 0x7f56300cb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5638600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563006c180 con 0x7f56300d1b60 2024-03-24T00:35:23.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5638600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563006c000 con 0x7f56300cf010 2024-03-24T00:35:23.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5638600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563006c300 con 0x7f56300c5010 2024-03-24T00:35:23.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5636600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56300d1b60 0x7f56300cb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5636600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56300d1b60 0x7f56300cb2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55136/0 (socket says 172.21.15.40:55136) 2024-03-24T00:35:23.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5636600640 1 -- 172.21.15.40:0/1402097093 learned_addr learned my addr 172.21.15.40:0/1402097093 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:23.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5635c00640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56300c5010 0x7f5630155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5635200640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56300cf010 0x7f56300c5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5636600640 1 -- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56300c5010 msgr2=0x7f5630155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5636600640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56300c5010 0x7f5630155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5636600640 1 -- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56300cf010 msgr2=0x7f56300c5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5636600640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56300cf010 0x7f56300c5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5636600640 1 -- 172.21.15.40:0/1402097093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56301566b0 con 0x7f56300d1b60 2024-03-24T00:35:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5635200640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56300cf010 0x7f56300c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.634+0000 7f5636600640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56300d1b60 0x7f56300cb2a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f562800efc0 tx=0x7f562800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.638+0000 7f562d600640 1 -- 172.21.15.40:0/1402097093 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5628018020 con 0x7f56300d1b60 2024-03-24T00:35:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.638+0000 7f562d600640 1 -- 172.21.15.40:0/1402097093 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5628014480 con 0x7f56300d1b60 2024-03-24T00:35:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.638+0000 7f562d600640 1 -- 172.21.15.40:0/1402097093 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5628014870 con 0x7f56300d1b60 2024-03-24T00:35:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.638+0000 7f5638600640 1 -- 172.21.15.40:0/1402097093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5630158bd0 con 0x7f56300d1b60 2024-03-24T00:35:23.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.638+0000 7f5638600640 1 -- 172.21.15.40:0/1402097093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56300c8000 con 0x7f56300d1b60 2024-03-24T00:35:23.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.638+0000 7f5638600640 1 -- 172.21.15.40:0/1402097093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5630068170 con 0x7f56300d1b60 2024-03-24T00:35:23.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.638+0000 7f562d600640 1 -- 172.21.15.40:0/1402097093 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f562801b020 con 0x7f56300d1b60 2024-03-24T00:35:23.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.642+0000 7f562d600640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5610067690 0x7f5610069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.642+0000 7f562d600640 1 -- 172.21.15.40:0/1402097093 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5628086150 con 0x7f56300d1b60 2024-03-24T00:35:23.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.642+0000 7f5635c00640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5610067690 0x7f5610069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.642+0000 7f5635c00640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5610067690 0x7f5610069b50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f56240ad000 tx=0x7f56240a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:23.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.646+0000 7f562d600640 1 -- 172.21.15.40:0/1402097093 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5628022070 con 0x7f56300d1b60 2024-03-24T00:35:23.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.806+0000 7f5638600640 1 -- 172.21.15.40:0/1402097093 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_distributed}] v 0) v1 -- 0x7f5630070910 con 0x7f56300d1b60 2024-03-24T00:35:23.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.810+0000 7f562d600640 1 -- 172.21.15.40:0/1402097093 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_distributed}]=0 v37)=0 v37) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5628010040 con 0x7f56300d1b60 2024-03-24T00:35:23.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.814+0000 7f5638600640 1 -- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5610067690 msgr2=0x7f5610069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.814+0000 7f5638600640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5610067690 0x7f5610069b50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f56240ad000 tx=0x7f56240a6000 comp rx=0 tx=0).stop 2024-03-24T00:35:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.814+0000 7f5638600640 1 -- 172.21.15.40:0/1402097093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56300d1b60 msgr2=0x7f56300cb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.814+0000 7f5638600640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56300d1b60 0x7f56300cb2a0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f562800efc0 tx=0x7f562800c490 comp rx=0 tx=0).stop 2024-03-24T00:35:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.814+0000 7f5638600640 1 -- 172.21.15.40:0/1402097093 shutdown_connections 2024-03-24T00:35:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.814+0000 7f5638600640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5610067690 0x7f5610069b50 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.814+0000 7f5638600640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f56300d1b60 0x7f56300cb2a0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.814+0000 7f5638600640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56300cf010 0x7f56300c5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.814+0000 7f5638600640 1 --2- 172.21.15.40:0/1402097093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56300c5010 0x7f5630155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.814+0000 7f5638600640 1 -- 172.21.15.40:0/1402097093 >> 172.21.15.40:0/1402097093 conn(0x7f563006ecf0 msgr2=0x7f56300b23c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.814+0000 7f5638600640 1 -- 172.21.15.40:0/1402097093 shutdown_connections 2024-03-24T00:35:23.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.814+0000 7f5638600640 1 -- 172.21.15.40:0/1402097093 wait complete. 2024-03-24T00:35:23.835 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random_max 1.0 2024-03-24T00:35:23.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.974+0000 7fd93d200640 1 Processor -- start 2024-03-24T00:35:23.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.974+0000 7fd93d200640 1 -- start start 2024-03-24T00:35:23.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.974+0000 7fd93d200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd93815ffd0 0x7fd9380ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.974+0000 7fd93d200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd9380728d0 0x7fd9380cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.974+0000 7fd93d200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd938071de0 con 0x7fd93815bd00 2024-03-24T00:35:23.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.974+0000 7fd93d200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9380cf790 con 0x7fd9380728d0 2024-03-24T00:35:23.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.974+0000 7fd93d200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9380cea50 con 0x7fd93815ffd0 2024-03-24T00:35:23.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.978+0000 7fd935e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd93815bd00 0x7fd93815f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36876/0 (socket says 172.21.15.40:36876) 2024-03-24T00:35:23.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.978+0000 7fd935e00640 1 -- 172.21.15.40:0/788446914 learned_addr learned my addr 172.21.15.40:0/788446914 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.978+0000 7fd936800640 1 --2- 172.21.15.40:0/788446914 >> v2:172.21.15.53:3300/0 conn(0x7fd9380728d0 0x7fd9380cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.978+0000 7fd92d400640 1 --2- 172.21.15.40:0/788446914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd93815ffd0 0x7fd9380ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.978+0000 7fd935400640 1 -- 172.21.15.40:0/788446914 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311088235 0 0) 0x7fd938071de0 con 0x7fd93815bd00 2024-03-24T00:35:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.978+0000 7fd935400640 1 -- 172.21.15.40:0/788446914 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd91c003530 con 0x7fd93815bd00 2024-03-24T00:35:23.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.978+0000 7fd935400640 1 -- 172.21.15.40:0/788446914 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1807204740 0 0) 0x7fd91c003530 con 0x7fd93815bd00 2024-03-24T00:35:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.978+0000 7fd936800640 1 -- 172.21.15.40:0/788446914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd93815ffd0 msgr2=0x7fd9380ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.978+0000 7fd936800640 1 --2- 172.21.15.40:0/788446914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd93815ffd0 0x7fd9380ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.978+0000 7fd936800640 1 -- 172.21.15.40:0/788446914 >> v1:172.21.15.40:6789/0 conn(0x7fd93815bd00 legacy=0x7fd93815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.978+0000 7fd936800640 1 -- 172.21.15.40:0/788446914 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9380ced30 con 0x7fd9380728d0 2024-03-24T00:35:23.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.978+0000 7fd92d400640 1 --2- 172.21.15.40:0/788446914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd93815ffd0 0x7fd9380ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd936800640 1 --2- 172.21.15.40:0/788446914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9380728d0 0x7fd9380cf250 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd930009d90 tx=0x7fd93001c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df145a1465fdfcbd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd935400640 1 -- 172.21.15.40:0/788446914 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9300b2070 con 0x7fd9380728d0 2024-03-24T00:35:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd935400640 1 -- 172.21.15.40:0/788446914 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9300a5640 con 0x7fd9380728d0 2024-03-24T00:35:23.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd935400640 1 -- 172.21.15.40:0/788446914 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd928001680 con 0x7fd9380728d0 2024-03-24T00:35:23.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 -- 172.21.15.40:0/788446914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9380728d0 msgr2=0x7fd9380cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 --2- 172.21.15.40:0/788446914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9380728d0 0x7fd9380cf250 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fd930009d90 tx=0x7fd93001c860 comp rx=0 tx=0).stop 2024-03-24T00:35:23.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 -- 172.21.15.40:0/788446914 shutdown_connections 2024-03-24T00:35:23.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 --2- 172.21.15.40:0/788446914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9380728d0 0x7fd9380cf250 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 --2- 172.21.15.40:0/788446914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd93815ffd0 0x7fd9380ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 -- 172.21.15.40:0/788446914 >> 172.21.15.40:0/788446914 conn(0x7fd9380b1660 msgr2=0x7fd9380b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:23.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 -- 172.21.15.40:0/788446914 shutdown_connections 2024-03-24T00:35:23.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 -- 172.21.15.40:0/788446914 wait complete. 2024-03-24T00:35:23.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 Processor -- start 2024-03-24T00:35:23.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 -- start start 2024-03-24T00:35:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9380728d0 0x7fd938146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd93815bd00 0x7fd9381455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd93815ffd0 0x7fd93814fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93806d320 con 0x7fd93815bd00 2024-03-24T00:35:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93815c0e0 con 0x7fd9380728d0 2024-03-24T00:35:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.982+0000 7fd93d200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93806d4a0 con 0x7fd93815ffd0 2024-03-24T00:35:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd93815bd00 0x7fd9381455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd93815bd00 0x7fd9381455c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55140/0 (socket says 172.21.15.40:55140) 2024-03-24T00:35:23.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92d400640 1 -- 172.21.15.40:0/2474750908 learned_addr learned my addr 172.21.15.40:0/2474750908 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:23.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd935e00640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9380728d0 0x7fd938146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd936800640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd93815ffd0 0x7fd93814fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92d400640 1 -- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd93815ffd0 msgr2=0x7fd93814fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92d400640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd93815ffd0 0x7fd93814fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92d400640 1 -- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9380728d0 msgr2=0x7fd938146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:23.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92d400640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9380728d0 0x7fd938146200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:23.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92d400640 1 -- 172.21.15.40:0/2474750908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9380b49b0 con 0x7fd93815bd00 2024-03-24T00:35:23.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92d400640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd93815bd00 0x7fd9381455c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fd92000e980 tx=0x7fd92000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:23.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92ea00640 1 -- 172.21.15.40:0/2474750908 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd920017070 con 0x7fd93815bd00 2024-03-24T00:35:23.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92ea00640 1 -- 172.21.15.40:0/2474750908 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92000ca00 con 0x7fd93815bd00 2024-03-24T00:35:23.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd93d200640 1 -- 172.21.15.40:0/2474750908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9380b0030 con 0x7fd93815bd00 2024-03-24T00:35:23.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92ea00640 1 -- 172.21.15.40:0/2474750908 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd92000cdd0 con 0x7fd93815bd00 2024-03-24T00:35:23.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd93d200640 1 -- 172.21.15.40:0/2474750908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9380b3ed0 con 0x7fd93815bd00 2024-03-24T00:35:23.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd93d200640 1 -- 172.21.15.40:0/2474750908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd938068170 con 0x7fd93815bd00 2024-03-24T00:35:23.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.986+0000 7fd92ea00640 1 -- 172.21.15.40:0/2474750908 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd92001a070 con 0x7fd93815bd00 2024-03-24T00:35:23.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.990+0000 7fd92ea00640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd914067660 0x7fd914069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:23.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.990+0000 7fd92ea00640 1 -- 172.21.15.40:0/2474750908 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd920085760 con 0x7fd93815bd00 2024-03-24T00:35:23.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.990+0000 7fd935e00640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd914067660 0x7fd914069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:23.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.990+0000 7fd935e00640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd914067660 0x7fd914069b20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd9280017f0 tx=0x7fd928001620 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:23.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:23.994+0000 7fd92ea00640 1 -- 172.21.15.40:0/2474750908 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd920055d00 con 0x7fd93815bd00 2024-03-24T00:35:24.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.154+0000 7fd93d200640 1 -- 172.21.15.40:0/2474750908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random_max}] v 0) v1 -- 0x7fd93806e870 con 0x7fd93815bd00 2024-03-24T00:35:24.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.158+0000 7fd92ea00640 1 -- 172.21.15.40:0/2474750908 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random_max}]=0 v38)=0 v38) v1 ==== 131+0+0 (secure 0 0 0) 0x7fd920059da0 con 0x7fd93815bd00 2024-03-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.162+0000 7fd93d200640 1 -- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd914067660 msgr2=0x7fd914069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.162+0000 7fd93d200640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd914067660 0x7fd914069b20 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd9280017f0 tx=0x7fd928001620 comp rx=0 tx=0).stop 2024-03-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.162+0000 7fd93d200640 1 -- 172.21.15.40:0/2474750908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd93815bd00 msgr2=0x7fd9381455c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:24.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.162+0000 7fd93d200640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd93815bd00 0x7fd9381455c0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fd92000e980 tx=0x7fd92000ee40 comp rx=0 tx=0).stop 2024-03-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.162+0000 7fd93d200640 1 -- 172.21.15.40:0/2474750908 shutdown_connections 2024-03-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.162+0000 7fd93d200640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd914067660 0x7fd914069b20 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.162+0000 7fd93d200640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd93815ffd0 0x7fd93814fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.162+0000 7fd93d200640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd93815bd00 0x7fd9381455c0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.162+0000 7fd93d200640 1 --2- 172.21.15.40:0/2474750908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9380728d0 0x7fd938146200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.162+0000 7fd93d200640 1 -- 172.21.15.40:0/2474750908 >> 172.21.15.40:0/2474750908 conn(0x7fd9380b1660 msgr2=0x7fd9380ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.162+0000 7fd93d200640 1 -- 172.21.15.40:0/2474750908 shutdown_connections 2024-03-24T00:35:24.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.162+0000 7fd93d200640 1 -- 172.21.15.40:0/2474750908 wait complete. 2024-03-24T00:35:24.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:24.190 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:35:24.190 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:35:24.190 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:35:24.190 DEBUG:teuthology.orchestra.run.smithi040:> # Use up a random number of inode numbers so the ephemeral pinning is not the same every test. 2024-03-24T00:35:24.191 DEBUG:teuthology.orchestra.run.smithi040:> mkdir .inode_number_thrash 2024-03-24T00:35:24.191 DEBUG:teuthology.orchestra.run.smithi040:> count=$((RANDOM % 1024)) 2024-03-24T00:35:24.191 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < count; i++)); do touch .inode_number_thrash/$i; done 2024-03-24T00:35:24.192 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf .inode_number_thrash 2024-03-24T00:35:24.192 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:35:24.768 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-24T00:35:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85b0200640 1 Processor -- start 2024-03-24T00:35:24.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85b0200640 1 -- start start 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85b0200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f85a815bd00 0x7f85a80ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85b0200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85a80728d0 0x7f85a80cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85b0200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a8072d60 con 0x7f85a815f8c0 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85b0200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a80cf790 con 0x7f85a80728d0 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85b0200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a8071de0 con 0x7f85a815bd00 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85ace00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85a80728d0 0x7f85a80cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85ace00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f85a80728d0 0x7f85a80cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55154/0 (socket says 172.21.15.40:55154) 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85ace00640 1 -- 172.21.15.40:0/4059107491 learned_addr learned my addr 172.21.15.40:0/4059107491 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85ad800640 1 --2- 172.21.15.40:0/4059107491 >> v2:172.21.15.53:3300/0 conn(0x7f85a815bd00 0x7f85a80ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85ace00640 1 -- 172.21.15.40:0/4059107491 >> v2:172.21.15.53:3300/0 conn(0x7f85a815bd00 msgr2=0x7f85a80ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85ace00640 1 --2- 172.21.15.40:0/4059107491 >> v2:172.21.15.53:3300/0 conn(0x7f85a815bd00 0x7f85a80ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85ace00640 1 -- 172.21.15.40:0/4059107491 >> v1:172.21.15.53:6789/0 conn(0x7f85a815f8c0 legacy=0x7f85a815fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:24.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.962+0000 7f85ace00640 1 -- 172.21.15.40:0/4059107491 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a80cea50 con 0x7f85a80728d0 2024-03-24T00:35:24.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.966+0000 7f85ace00640 1 --2- 172.21.15.40:0/4059107491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80728d0 0x7f85a80cf250 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f85a80733c0 tx=0x7f859401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88e19f2f05d9354 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:24.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.966+0000 7f85a7e00640 1 -- 172.21.15.40:0/4059107491 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85940aa650 con 0x7f85a80728d0 2024-03-24T00:35:24.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.966+0000 7f85a7e00640 1 -- 172.21.15.40:0/4059107491 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85940af070 con 0x7f85a80728d0 2024-03-24T00:35:24.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.966+0000 7f85a7e00640 1 -- 172.21.15.40:0/4059107491 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85940aaa10 con 0x7f85a80728d0 2024-03-24T00:35:24.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.966+0000 7f85b0200640 1 -- 172.21.15.40:0/4059107491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80728d0 msgr2=0x7f85a80cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:24.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.966+0000 7f85b0200640 1 --2- 172.21.15.40:0/4059107491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80728d0 0x7f85a80cf250 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f85a80733c0 tx=0x7f859401c4f0 comp rx=0 tx=0).stop 2024-03-24T00:35:24.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.966+0000 7f85b0200640 1 -- 172.21.15.40:0/4059107491 shutdown_connections 2024-03-24T00:35:24.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.966+0000 7f85b0200640 1 --2- 172.21.15.40:0/4059107491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a80728d0 0x7f85a80cf250 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:24.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.966+0000 7f85b0200640 1 --2- 172.21.15.40:0/4059107491 >> v2:172.21.15.53:3300/0 conn(0x7f85a815bd00 0x7f85a80ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:24.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.966+0000 7f85b0200640 1 -- 172.21.15.40:0/4059107491 >> 172.21.15.40:0/4059107491 conn(0x7f85a80b1660 msgr2=0x7f85a80b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:24.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.966+0000 7f85b0200640 1 -- 172.21.15.40:0/4059107491 shutdown_connections 2024-03-24T00:35:24.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.966+0000 7f85b0200640 1 -- 172.21.15.40:0/4059107491 wait complete. 2024-03-24T00:35:24.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85b0200640 1 Processor -- start 2024-03-24T00:35:24.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85b0200640 1 -- start start 2024-03-24T00:35:24.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85b0200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a80728d0 0x7f85a81548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:24.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85b0200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85a815bd00 0x7f85a814e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:24.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85b0200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a815f8c0 0x7f85a8151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:24.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85b0200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a806d320 con 0x7f85a815f8c0 2024-03-24T00:35:24.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85b0200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a815c0e0 con 0x7f85a80728d0 2024-03-24T00:35:24.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85b0200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a806d4a0 con 0x7f85a815bd00 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85ae200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a815f8c0 0x7f85a8151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85ae200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a815f8c0 0x7f85a8151020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55160/0 (socket says 172.21.15.40:55160) 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85ae200640 1 -- 172.21.15.40:0/1749846436 learned_addr learned my addr 172.21.15.40:0/1749846436 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85ad800640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a80728d0 0x7f85a81548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85ace00640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85a815bd00 0x7f85a814e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85ae200640 1 -- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85a815bd00 msgr2=0x7f85a814e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85ae200640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85a815bd00 0x7f85a814e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85ae200640 1 -- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a80728d0 msgr2=0x7f85a81548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85ae200640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a80728d0 0x7f85a81548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85ae200640 1 -- 172.21.15.40:0/1749846436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a814fa30 con 0x7f85a815f8c0 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85ace00640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85a815bd00 0x7f85a814e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85ae200640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a815f8c0 0x7f85a8151020 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f85a000c730 tx=0x7f85a000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85a5600640 1 -- 172.21.15.40:0/1749846436 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a0003b80 con 0x7f85a815f8c0 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85a5600640 1 -- 172.21.15.40:0/1749846436 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a0003d20 con 0x7f85a815f8c0 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85a5600640 1 -- 172.21.15.40:0/1749846436 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a000aa30 con 0x7f85a815f8c0 2024-03-24T00:35:24.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85b0200640 1 -- 172.21.15.40:0/1749846436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a814fcd0 con 0x7f85a815f8c0 2024-03-24T00:35:24.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.970+0000 7f85b0200640 1 -- 172.21.15.40:0/1749846436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a80aa6c0 con 0x7f85a815f8c0 2024-03-24T00:35:24.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.974+0000 7f85b0200640 1 -- 172.21.15.40:0/1749846436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a8068170 con 0x7f85a815f8c0 2024-03-24T00:35:24.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.974+0000 7f85a5600640 1 -- 172.21.15.40:0/1749846436 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f85a000abd0 con 0x7f85a815f8c0 2024-03-24T00:35:24.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.974+0000 7f85a5600640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8588067690 0x7f8588069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:24.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.974+0000 7f85a5600640 1 -- 172.21.15.40:0/1749846436 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85a00855c0 con 0x7f85a815f8c0 2024-03-24T00:35:24.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.974+0000 7f85ad800640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8588067690 0x7f8588069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:24.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.978+0000 7f85ad800640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8588067690 0x7f8588069b50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f859c007a60 tx=0x7f859c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:24.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:24.978+0000 7f85a5600640 1 -- 172.21.15.40:0/1749846436 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f85a00562d0 con 0x7f85a815f8c0 2024-03-24T00:35:25.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.178+0000 7f85b0200640 1 -- 172.21.15.40:0/1749846436 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f85a806e870 con 0x7f85a815f8c0 2024-03-24T00:35:25.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.462+0000 7f85a5600640 1 -- 172.21.15.40:0/1749846436 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v68) v1 ==== 105+0+0 (secure 0 0 0) 0x7f85a0055bd0 con 0x7f85a815f8c0 2024-03-24T00:35:25.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.466+0000 7f85b0200640 1 -- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8588067690 msgr2=0x7f8588069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.466+0000 7f85b0200640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8588067690 0x7f8588069b50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f859c007a60 tx=0x7f859c008040 comp rx=0 tx=0).stop 2024-03-24T00:35:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.466+0000 7f85b0200640 1 -- 172.21.15.40:0/1749846436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a815f8c0 msgr2=0x7f85a8151020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.466+0000 7f85b0200640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a815f8c0 0x7f85a8151020 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f85a000c730 tx=0x7f85a000cbf0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.466+0000 7f85b0200640 1 -- 172.21.15.40:0/1749846436 shutdown_connections 2024-03-24T00:35:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.466+0000 7f85b0200640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8588067690 0x7f8588069b50 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.466+0000 7f85b0200640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85a815f8c0 0x7f85a8151020 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.466+0000 7f85b0200640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85a815bd00 0x7f85a814e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.466+0000 7f85b0200640 1 --2- 172.21.15.40:0/1749846436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85a80728d0 0x7f85a81548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.466+0000 7f85b0200640 1 -- 172.21.15.40:0/1749846436 >> 172.21.15.40:0/1749846436 conn(0x7f85a80b1660 msgr2=0x7f85a80cf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.466+0000 7f85b0200640 1 -- 172.21.15.40:0/1749846436 shutdown_connections 2024-03-24T00:35:25.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.466+0000 7f85b0200640 1 -- 172.21.15.40:0/1749846436 wait complete. 2024-03-24T00:35:25.497 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:25.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f9585600640 1 Processor -- start 2024-03-24T00:35:25.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f9585600640 1 -- start start 2024-03-24T00:35:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f9585600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9580141bc0 0x7f95800ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f9585600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f95800ce630 0x7f9580071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f9585600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9580070e00 con 0x7f958013ec80 2024-03-24T00:35:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f9585600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95800cee20 con 0x7f9580141bc0 2024-03-24T00:35:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f9585600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95800cf100 con 0x7f95800ce630 2024-03-24T00:35:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f957e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9580141bc0 0x7f95800ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f957ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f958013ec80 0x7f95801414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:36884/0 (socket says 172.21.15.40:36884) 2024-03-24T00:35:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f957e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9580141bc0 0x7f95800ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55162/0 (socket says 172.21.15.40:55162) 2024-03-24T00:35:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f957ea00640 1 -- 172.21.15.40:0/69975735 learned_addr learned my addr 172.21.15.40:0/69975735 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f957f400640 1 --2- 172.21.15.40:0/69975735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f95800ce630 0x7f9580071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:25.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f957e000640 1 -- 172.21.15.40:0/69975735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f95800ce630 msgr2=0x7f9580071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f957e000640 1 --2- 172.21.15.40:0/69975735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f95800ce630 0x7f9580071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f957e000640 1 -- 172.21.15.40:0/69975735 >> v1:172.21.15.40:6789/0 conn(0x7f958013ec80 legacy=0x7f95801414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:25.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.634+0000 7f957e000640 1 -- 172.21.15.40:0/69975735 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95800cf3e0 con 0x7f9580141bc0 2024-03-24T00:35:25.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.638+0000 7f957e000640 1 --2- 172.21.15.40:0/69975735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9580141bc0 0x7f95800ce0f0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f956c0097c0 tx=0x7f956c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81cf993241692f26 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:25.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.638+0000 7f957d600640 1 -- 172.21.15.40:0/69975735 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c0aa620 con 0x7f9580141bc0 2024-03-24T00:35:25.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.638+0000 7f957d600640 1 -- 172.21.15.40:0/69975735 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956c0af070 con 0x7f9580141bc0 2024-03-24T00:35:25.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.638+0000 7f957d600640 1 -- 172.21.15.40:0/69975735 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c0aa9e0 con 0x7f9580141bc0 2024-03-24T00:35:25.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.638+0000 7f9585600640 1 -- 172.21.15.40:0/69975735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9580141bc0 msgr2=0x7f95800ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:25.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.638+0000 7f9585600640 1 --2- 172.21.15.40:0/69975735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9580141bc0 0x7f95800ce0f0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f956c0097c0 tx=0x7f956c01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.638+0000 7f9585600640 1 -- 172.21.15.40:0/69975735 shutdown_connections 2024-03-24T00:35:25.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.638+0000 7f9585600640 1 --2- 172.21.15.40:0/69975735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f95800ce630 0x7f9580071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.638+0000 7f9585600640 1 --2- 172.21.15.40:0/69975735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9580141bc0 0x7f95800ce0f0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.638+0000 7f9585600640 1 -- 172.21.15.40:0/69975735 >> 172.21.15.40:0/69975735 conn(0x7f95800b17b0 msgr2=0x7f95800b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:25.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.638+0000 7f9585600640 1 -- 172.21.15.40:0/69975735 shutdown_connections 2024-03-24T00:35:25.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.638+0000 7f9585600640 1 -- 172.21.15.40:0/69975735 wait complete. 2024-03-24T00:35:25.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.642+0000 7f9585600640 1 Processor -- start 2024-03-24T00:35:25.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.642+0000 7f9585600640 1 -- start start 2024-03-24T00:35:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.642+0000 7f9585600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f95800ce630 0x7f9580138be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.642+0000 7f9585600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958013ec80 0x7f95801594e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.642+0000 7f9585600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9580141bc0 0x7f958013a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.642+0000 7f9585600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9580072460 con 0x7f958013ec80 2024-03-24T00:35:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.642+0000 7f9585600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95800722e0 con 0x7f9580141bc0 2024-03-24T00:35:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.642+0000 7f9585600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95800725e0 con 0x7f95800ce630 2024-03-24T00:35:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.642+0000 7f957e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958013ec80 0x7f95801594e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.642+0000 7f957e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958013ec80 0x7f95801594e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55176/0 (socket says 172.21.15.40:55176) 2024-03-24T00:35:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.642+0000 7f957e000640 1 -- 172.21.15.40:0/26806294 learned_addr learned my addr 172.21.15.40:0/26806294 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f957f400640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9580141bc0 0x7f958013a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f957ea00640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f95800ce630 0x7f9580138be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:25.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f957e000640 1 -- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f95800ce630 msgr2=0x7f9580138be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f957e000640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f95800ce630 0x7f9580138be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f957e000640 1 -- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9580141bc0 msgr2=0x7f958013a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f957e000640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9580141bc0 0x7f958013a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f957e000640 1 -- 172.21.15.40:0/26806294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958006d970 con 0x7f958013ec80 2024-03-24T00:35:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f957ea00640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f95800ce630 0x7f9580138be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f957e000640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958013ec80 0x7f95801594e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f956c01a0d0 tx=0x7f956c019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f9576a00640 1 -- 172.21.15.40:0/26806294 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c0aada0 con 0x7f958013ec80 2024-03-24T00:35:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f9576a00640 1 -- 172.21.15.40:0/26806294 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956c0be450 con 0x7f958013ec80 2024-03-24T00:35:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f9585600640 1 -- 172.21.15.40:0/26806294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958006dc50 con 0x7f958013ec80 2024-03-24T00:35:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f9576a00640 1 -- 172.21.15.40:0/26806294 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c0be820 con 0x7f958013ec80 2024-03-24T00:35:25.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f9585600640 1 -- 172.21.15.40:0/26806294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958006e210 con 0x7f958013ec80 2024-03-24T00:35:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f9585600640 1 -- 172.21.15.40:0/26806294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9580068170 con 0x7f958013ec80 2024-03-24T00:35:25.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.646+0000 7f9576a00640 1 -- 172.21.15.40:0/26806294 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f956c0c73f0 con 0x7f958013ec80 2024-03-24T00:35:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.650+0000 7f9576a00640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9560067690 0x7f9560069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.650+0000 7f9576a00640 1 -- 172.21.15.40:0/26806294 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f956c11feb0 con 0x7f958013ec80 2024-03-24T00:35:25.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.650+0000 7f957ea00640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9560067690 0x7f9560069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:25.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.650+0000 7f957ea00640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9560067690 0x7f9560069b50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f95801589f0 tx=0x7f9570009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:25.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.650+0000 7f9576a00640 1 -- 172.21.15.40:0/26806294 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f956c016090 con 0x7f958013ec80 2024-03-24T00:35:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.878+0000 7f9585600640 1 -- 172.21.15.40:0/26806294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9580072b90 con 0x7f958013ec80 2024-03-24T00:35:25.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.878+0000 7f9576a00640 1 -- 172.21.15.40:0/26806294 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4989 (secure 0 0 0) 0x7f956c0f0bb0 con 0x7f958013ec80 2024-03-24T00:35:25.883 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:25.883 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":69,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6062,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/4182264423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":4182264423},{"type":"v1","addr":"172.21.15.53:6836","nonce":4182264423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6733,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/334375209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":334375209},{"type":"v1","addr":"172.21.15.53:6837","nonce":334375209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:35:14.689580+0000","modified":"2024-03-24T00:35:25.469360+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":6954,"mds_1":6951},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6951":{"gid":6951,"name":"c","rank":1,"incarnation":69,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6834/221849812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":221849812},{"type":"v1","addr":"172.21.15.40:6834","nonce":221849812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6954":{"gid":6954,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/2177098794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2177098794},{"type":"v1","addr":"172.21.15.40:6835","nonce":2177098794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6954,"qdb_cluster":[6954]},"id":5}]} 2024-03-24T00:35:25.883 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 69 2024-03-24T00:35:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.882+0000 7f9585600640 1 -- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9560067690 msgr2=0x7f9560069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.882+0000 7f9585600640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9560067690 0x7f9560069b50 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f95801589f0 tx=0x7f9570009040 comp rx=0 tx=0).stop 2024-03-24T00:35:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.882+0000 7f9585600640 1 -- 172.21.15.40:0/26806294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958013ec80 msgr2=0x7f95801594e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.882+0000 7f9585600640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958013ec80 0x7f95801594e0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f956c01a0d0 tx=0x7f956c019240 comp rx=0 tx=0).stop 2024-03-24T00:35:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.882+0000 7f9585600640 1 -- 172.21.15.40:0/26806294 shutdown_connections 2024-03-24T00:35:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.882+0000 7f9585600640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9560067690 0x7f9560069b50 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.882+0000 7f9585600640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9580141bc0 0x7f958013a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.882+0000 7f9585600640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f958013ec80 0x7f95801594e0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.882+0000 7f9585600640 1 --2- 172.21.15.40:0/26806294 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f95800ce630 0x7f9580138be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:25.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.882+0000 7f9585600640 1 -- 172.21.15.40:0/26806294 >> 172.21.15.40:0/26806294 conn(0x7f95800b17b0 msgr2=0x7f958015e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:25.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.882+0000 7f9585600640 1 -- 172.21.15.40:0/26806294 shutdown_connections 2024-03-24T00:35:25.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:25.882+0000 7f9585600640 1 -- 172.21.15.40:0/26806294 wait complete. 2024-03-24T00:35:26.307 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:35:14.689580+0000', 'modified': '2024-03-24T00:35:25.469360+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 6954, 'mds_1': 6951}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6951': {'gid': 6951, 'name': 'c', 'rank': 1, 'incarnation': 69, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6834/221849812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 221849812}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 221849812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_6954': {'gid': 6954, 'name': 'a', 'rank': 0, 'incarnation': 63, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/2177098794', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 2177098794}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 2177098794}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6954, 'qdb_cluster': [6954]} 2024-03-24T00:35:26.308 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6951:up:creating 2024-03-24T00:35:27.275 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-03-24T00:35:27.270+0000 7f321f200640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:35:27.309 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:27.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6cb600640 1 Processor -- start 2024-03-24T00:35:27.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6cb600640 1 -- start start 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6cb600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa6c4161880 0x7fa6c40b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6cb600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c40d0600 con 0x7fa6c40ce610 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6cb600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c40d08e0 con 0x7fa6c40cd960 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6cb600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c4161c60 con 0x7fa6c4161880 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6bbe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa6c40ce610 0x7fa6c406b8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47570/0 (socket says 172.21.15.40:47570) 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6bbe00640 1 -- 172.21.15.40:0/2773462038 learned_addr learned my addr 172.21.15.40:0/2773462038 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6c8c00640 1 --2- 172.21.15.40:0/2773462038 >> v2:172.21.15.53:3300/0 conn(0x7fa6c4161880 0x7fa6c40b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6c3e00640 1 -- 172.21.15.40:0/2773462038 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933776427 0 0) 0x7fa6c40d0600 con 0x7fa6c40ce610 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6c3e00640 1 -- 172.21.15.40:0/2773462038 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a8003530 con 0x7fa6c40ce610 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6c3e00640 1 -- 172.21.15.40:0/2773462038 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113857738 0 0) 0x7fa6c40d08e0 con 0x7fa6c40cd960 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6c3e00640 1 -- 172.21.15.40:0/2773462038 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6c40d0600 con 0x7fa6c40cd960 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6c3e00640 1 -- 172.21.15.40:0/2773462038 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3536397437 0 0) 0x7fa6a8003530 con 0x7fa6c40ce610 2024-03-24T00:35:27.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6c8c00640 1 -- 172.21.15.40:0/2773462038 >> v1:172.21.15.53:6789/0 conn(0x7fa6c40cd960 legacy=0x7fa6c406bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:27.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6c8c00640 1 -- 172.21.15.40:0/2773462038 >> v1:172.21.15.40:6789/0 conn(0x7fa6c40ce610 legacy=0x7fa6c406b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:27.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.454+0000 7fa6c8c00640 1 -- 172.21.15.40:0/2773462038 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c40d0bc0 con 0x7fa6c4161880 2024-03-24T00:35:27.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6c8c00640 1 --2- 172.21.15.40:0/2773462038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6c4161880 0x7fa6c40b2d50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40097c0 tx=0x7fa6b401c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4672142fbd4f7316 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:27.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6c3e00640 1 -- 172.21.15.40:0/2773462038 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b40b0070 con 0x7fa6c4161880 2024-03-24T00:35:27.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6c3e00640 1 -- 172.21.15.40:0/2773462038 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b40ab400 con 0x7fa6c4161880 2024-03-24T00:35:27.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6c3e00640 1 -- 172.21.15.40:0/2773462038 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac001680 con 0x7fa6c4161880 2024-03-24T00:35:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6cb600640 1 -- 172.21.15.40:0/2773462038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6c4161880 msgr2=0x7fa6c40b2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6cb600640 1 --2- 172.21.15.40:0/2773462038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6c4161880 0x7fa6c40b2d50 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40097c0 tx=0x7fa6b401c330 comp rx=0 tx=0).stop 2024-03-24T00:35:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6cb600640 1 -- 172.21.15.40:0/2773462038 shutdown_connections 2024-03-24T00:35:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6cb600640 1 --2- 172.21.15.40:0/2773462038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6c4161880 0x7fa6c40b2d50 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6cb600640 1 -- 172.21.15.40:0/2773462038 >> 172.21.15.40:0/2773462038 conn(0x7fa6c40b38f0 msgr2=0x7fa6c40b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6cb600640 1 -- 172.21.15.40:0/2773462038 shutdown_connections 2024-03-24T00:35:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6cb600640 1 -- 172.21.15.40:0/2773462038 wait complete. 2024-03-24T00:35:27.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6cb600640 1 Processor -- start 2024-03-24T00:35:27.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.462+0000 7fa6cb600640 1 -- start start 2024-03-24T00:35:27.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6cb600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6c40cd960 0x7fa6c414f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:27.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6cb600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6c40ce610 0x7fa6c41563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:27.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6cb600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6c4161880 0x7fa6c414cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:27.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6cb600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c406b430 con 0x7fa6c40ce610 2024-03-24T00:35:27.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6cb600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c406b150 con 0x7fa6c40cd960 2024-03-24T00:35:27.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6cb600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c416c6c0 con 0x7fa6c4161880 2024-03-24T00:35:27.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6bbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6c40ce610 0x7fa6c41563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:27.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6bbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6c40ce610 0x7fa6c41563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36744/0 (socket says 172.21.15.40:36744) 2024-03-24T00:35:27.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6bbe00640 1 -- 172.21.15.40:0/3332610135 learned_addr learned my addr 172.21.15.40:0/3332610135 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:27.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6c8c00640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6c40cd960 0x7fa6c414f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:27.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6c9600640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6c4161880 0x7fa6c414cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:27.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6bbe00640 1 -- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6c4161880 msgr2=0x7fa6c414cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:27.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6bbe00640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6c4161880 0x7fa6c414cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:27.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6bbe00640 1 -- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6c40cd960 msgr2=0x7fa6c414f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:27.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6bbe00640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6c40cd960 0x7fa6c414f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:27.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6bbe00640 1 -- 172.21.15.40:0/3332610135 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c4150060 con 0x7fa6c40ce610 2024-03-24T00:35:27.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6bbe00640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6c40ce610 0x7fa6c41563a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00c760 tx=0x7fa6ac00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:27.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6c1600640 1 -- 172.21.15.40:0/3332610135 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac017070 con 0x7fa6c40ce610 2024-03-24T00:35:27.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6c1600640 1 -- 172.21.15.40:0/3332610135 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac005b00 con 0x7fa6c40ce610 2024-03-24T00:35:27.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6c1600640 1 -- 172.21.15.40:0/3332610135 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac007070 con 0x7fa6c40ce610 2024-03-24T00:35:27.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6cb600640 1 -- 172.21.15.40:0/3332610135 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c40c9060 con 0x7fa6c40ce610 2024-03-24T00:35:27.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6cb600640 1 -- 172.21.15.40:0/3332610135 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c40c9620 con 0x7fa6c40ce610 2024-03-24T00:35:27.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.466+0000 7fa6cb600640 1 -- 172.21.15.40:0/3332610135 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c4068170 con 0x7fa6c40ce610 2024-03-24T00:35:27.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.470+0000 7fa6c1600640 1 -- 172.21.15.40:0/3332610135 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa6ac01a070 con 0x7fa6c40ce610 2024-03-24T00:35:27.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.470+0000 7fa6c1600640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6a0067660 0x7fa6a0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:27.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.470+0000 7fa6c1600640 1 -- 172.21.15.40:0/3332610135 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6ac084c20 con 0x7fa6c40ce610 2024-03-24T00:35:27.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.470+0000 7fa6c8c00640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6a0067660 0x7fa6a0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:27.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.470+0000 7fa6c8c00640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6a0067660 0x7fa6a0069b20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40ad000 tx=0x7fa6b40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:27.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.474+0000 7fa6c1600640 1 -- 172.21.15.40:0/3332610135 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa6ac0552e0 con 0x7fa6c40ce610 2024-03-24T00:35:27.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.674+0000 7fa6cb600640 1 -- 172.21.15.40:0/3332610135 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6c40d23a0 con 0x7fa6c40ce610 2024-03-24T00:35:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.674+0000 7fa6c1600640 1 -- 172.21.15.40:0/3332610135 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5011 (secure 0 0 0) 0x7fa6ac059380 con 0x7fa6c40ce610 2024-03-24T00:35:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:27.678 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":72,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6062,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/4182264423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":4182264423},{"type":"v1","addr":"172.21.15.53:6836","nonce":4182264423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:35:14.689580+0000","modified":"2024-03-24T00:35:27.479149+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6954,"mds_1":6951,"mds_2":6733},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6733":{"gid":6733,"name":"b","rank":2,"incarnation":71,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/334375209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":334375209},{"type":"v1","addr":"172.21.15.53:6837","nonce":334375209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6951":{"gid":6951,"name":"c","rank":1,"incarnation":69,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/221849812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":221849812},{"type":"v1","addr":"172.21.15.40:6834","nonce":221849812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6954":{"gid":6954,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/2177098794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2177098794},{"type":"v1","addr":"172.21.15.40:6835","nonce":2177098794}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6954,"qdb_cluster":[6733,6954,6951]},"id":5}]} 2024-03-24T00:35:27.678 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 72 2024-03-24T00:35:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.678+0000 7fa6cb600640 1 -- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6a0067660 msgr2=0x7fa6a0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.678+0000 7fa6cb600640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6a0067660 0x7fa6a0069b20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40ad000 tx=0x7fa6b40a6000 comp rx=0 tx=0).stop 2024-03-24T00:35:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.678+0000 7fa6cb600640 1 -- 172.21.15.40:0/3332610135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6c40ce610 msgr2=0x7fa6c41563a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.678+0000 7fa6cb600640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6c40ce610 0x7fa6c41563a0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00c760 tx=0x7fa6ac00cc20 comp rx=0 tx=0).stop 2024-03-24T00:35:27.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.678+0000 7fa6cb600640 1 -- 172.21.15.40:0/3332610135 shutdown_connections 2024-03-24T00:35:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.678+0000 7fa6cb600640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6a0067660 0x7fa6a0069b20 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.678+0000 7fa6cb600640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6c4161880 0x7fa6c414cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.678+0000 7fa6cb600640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6c40ce610 0x7fa6c41563a0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.678+0000 7fa6cb600640 1 --2- 172.21.15.40:0/3332610135 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6c40cd960 0x7fa6c414f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.678+0000 7fa6cb600640 1 -- 172.21.15.40:0/3332610135 >> 172.21.15.40:0/3332610135 conn(0x7fa6c40b38f0 msgr2=0x7fa6c41574c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.678+0000 7fa6cb600640 1 -- 172.21.15.40:0/3332610135 shutdown_connections 2024-03-24T00:35:27.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:27.678+0000 7fa6cb600640 1 -- 172.21.15.40:0/3332610135 wait complete. 2024-03-24T00:35:28.102 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:35:14.689580+0000', 'modified': '2024-03-24T00:35:27.479149+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 6954, 'mds_1': 6951, 'mds_2': 6733}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6733': {'gid': 6733, 'name': 'b', 'rank': 2, 'incarnation': 71, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.53:6837/334375209', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 334375209}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 334375209}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_6951': {'gid': 6951, 'name': 'c', 'rank': 1, 'incarnation': 69, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.40:6834/221849812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 221849812}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 221849812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_6954': {'gid': 6954, 'name': 'a', 'rank': 0, 'incarnation': 63, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/2177098794', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 2177098794}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 2177098794}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6954, 'qdb_cluster': [6733, 6954, 6951]} 2024-03-24T00:35:28.102 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T00:35:28.102 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:35:28.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e98a00640 1 Processor -- start 2024-03-24T00:35:28.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e98a00640 1 -- start start 2024-03-24T00:35:28.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e98a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9e9415bca0 0x7f9e940d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e98a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415c080 0x7f9e940d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e98a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e940d0780 con 0x7f9e940ce490 2024-03-24T00:35:28.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e98a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e940b4a90 con 0x7f9e9415bca0 2024-03-24T00:35:28.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e98a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e940b4d70 con 0x7f9e9415c080 2024-03-24T00:35:28.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e92a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415c080 0x7f9e940d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e93400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9e9415bca0 0x7f9e940d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e93400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9e9415bca0 0x7f9e940d2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:49392/0 (socket says 172.21.15.40:49392) 2024-03-24T00:35:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e93400640 1 -- 172.21.15.40:0/3282195127 learned_addr learned my addr 172.21.15.40:0/3282195127 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e92a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415c080 0x7f9e940d0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:54828/0 (socket says 172.21.15.40:54828) 2024-03-24T00:35:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e93e00640 1 --1- 172.21.15.40:0/3282195127 >> v1:172.21.15.53:6789/0 conn(0x7f9e940ce490 0x7f9e940cc5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58922/0 (socket says 172.21.15.40:58922) 2024-03-24T00:35:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e93400640 1 -- 172.21.15.40:0/3282195127 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415c080 msgr2=0x7f9e940d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e93400640 1 --2- 172.21.15.40:0/3282195127 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415c080 0x7f9e940d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e93400640 1 -- 172.21.15.40:0/3282195127 >> v1:172.21.15.53:6789/0 conn(0x7f9e940ce490 legacy=0x7f9e940cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e93400640 1 -- 172.21.15.40:0/3282195127 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e940b5050 con 0x7f9e9415bca0 2024-03-24T00:35:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e92a00640 1 --2- 172.21.15.40:0/3282195127 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415c080 0x7f9e940d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:28.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.242+0000 7f9e92000640 1 -- 172.21.15.40:0/3282195127 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267518837 0 0) 0x7f9e940d0780 con 0x7f9e940ce490 2024-03-24T00:35:28.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.246+0000 7f9e93400640 1 --2- 172.21.15.40:0/3282195127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e9415bca0 0x7f9e940d2590 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009870 tx=0x7f9e8401c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d01c769ab90fd2d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.246+0000 7f9e92000640 1 -- 172.21.15.40:0/3282195127 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840b2070 con 0x7f9e9415bca0 2024-03-24T00:35:28.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.246+0000 7f9e92000640 1 -- 172.21.15.40:0/3282195127 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e840a9070 con 0x7f9e9415bca0 2024-03-24T00:35:28.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.246+0000 7f9e92000640 1 -- 172.21.15.40:0/3282195127 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840ad880 con 0x7f9e9415bca0 2024-03-24T00:35:28.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.246+0000 7f9e98a00640 1 -- 172.21.15.40:0/3282195127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e9415bca0 msgr2=0x7f9e940d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.246+0000 7f9e98a00640 1 --2- 172.21.15.40:0/3282195127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e9415bca0 0x7f9e940d2590 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009870 tx=0x7f9e8401c560 comp rx=0 tx=0).stop 2024-03-24T00:35:28.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.246+0000 7f9e98a00640 1 -- 172.21.15.40:0/3282195127 shutdown_connections 2024-03-24T00:35:28.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.246+0000 7f9e98a00640 1 --2- 172.21.15.40:0/3282195127 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415c080 0x7f9e940d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.246+0000 7f9e98a00640 1 --2- 172.21.15.40:0/3282195127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e9415bca0 0x7f9e940d2590 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.246+0000 7f9e98a00640 1 -- 172.21.15.40:0/3282195127 >> 172.21.15.40:0/3282195127 conn(0x7f9e940d1aa0 msgr2=0x7f9e940d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.246+0000 7f9e98a00640 1 -- 172.21.15.40:0/3282195127 shutdown_connections 2024-03-24T00:35:28.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.246+0000 7f9e98a00640 1 -- 172.21.15.40:0/3282195127 wait complete. 2024-03-24T00:35:28.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e98a00640 1 Processor -- start 2024-03-24T00:35:28.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e98a00640 1 -- start start 2024-03-24T00:35:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e98a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e940ce490 0x7f9e94159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e98a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415bca0 0x7f9e940c76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e98a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e9415c080 0x7f9e94151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e98a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94168c50 con 0x7f9e9415c080 2024-03-24T00:35:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e98a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e940cd4d0 con 0x7f9e940ce490 2024-03-24T00:35:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e98a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94168dd0 con 0x7f9e9415bca0 2024-03-24T00:35:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e93400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e940ce490 0x7f9e94159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e92a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415bca0 0x7f9e940c76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e93400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e940ce490 0x7f9e94159c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:49396/0 (socket says 172.21.15.40:49396) 2024-03-24T00:35:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e93400640 1 -- 172.21.15.40:0/380076769 learned_addr learned my addr 172.21.15.40:0/380076769 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e93e00640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e9415c080 0x7f9e94151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e93400640 1 -- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415bca0 msgr2=0x7f9e940c76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e93400640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415bca0 0x7f9e940c76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e93400640 1 -- 172.21.15.40:0/380076769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e9415c080 msgr2=0x7f9e94151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e93400640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e9415c080 0x7f9e94151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e93400640 1 -- 172.21.15.40:0/380076769 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e941fa650 con 0x7f9e940ce490 2024-03-24T00:35:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e92a00640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415bca0 0x7f9e940c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e93400640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e940ce490 0x7f9e94159c00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9e840b0040 tx=0x7f9e8401ada0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e8b400640 1 -- 172.21.15.40:0/380076769 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840b2040 con 0x7f9e940ce490 2024-03-24T00:35:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e8b400640 1 -- 172.21.15.40:0/380076769 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e840b8420 con 0x7f9e940ce490 2024-03-24T00:35:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e98a00640 1 -- 172.21.15.40:0/380076769 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e940c7c30 con 0x7f9e940ce490 2024-03-24T00:35:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e8b400640 1 -- 172.21.15.40:0/380076769 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840b8830 con 0x7f9e940ce490 2024-03-24T00:35:28.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.250+0000 7f9e98a00640 1 -- 172.21.15.40:0/380076769 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e941fabd0 con 0x7f9e940ce490 2024-03-24T00:35:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.254+0000 7f9e8b400640 1 -- 172.21.15.40:0/380076769 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f9e840c0030 con 0x7f9e940ce490 2024-03-24T00:35:28.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.254+0000 7f9e8b400640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9e74067690 0x7f9e74069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.254+0000 7f9e8b400640 1 -- 172.21.15.40:0/380076769 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e8411f3f0 con 0x7f9e940ce490 2024-03-24T00:35:28.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.254+0000 7f9e92a00640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9e74067690 0x7f9e74069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.254+0000 7f9e9d436000 1 Processor -- start 2024-03-24T00:35:28.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.254+0000 7f9e9d436000 1 -- start start 2024-03-24T00:35:28.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.254+0000 7f9e92a00640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9e74067690 0x7f9e74069b50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9e941fa5b0 tx=0x7f9e80009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317ec920 0x55d8317ecd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ed240 0x55d8317a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317a2ff0 0x55d8317a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8315f0680 con 0x55d8317a2ff0 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d831577b00 con 0x55d8317ec920 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d8315ec720 con 0x55d8317ed240 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e93400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317ec920 0x55d8317ecd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e93e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317a2ff0 0x55d8317a33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e93400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317ec920 0x55d8317ecd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:49410/0 (socket says 172.21.15.40:49410) 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e93400640 1 -- 172.21.15.40:0/2640602397 learned_addr learned my addr 172.21.15.40:0/2640602397 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e92a00640 1 --2- 172.21.15.40:0/2640602397 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ed240 0x55d8317a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e93400640 1 -- 172.21.15.40:0/2640602397 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ed240 msgr2=0x55d8317a2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e93400640 1 --2- 172.21.15.40:0/2640602397 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ed240 0x55d8317a2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e93400640 1 -- 172.21.15.40:0/2640602397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317a2ff0 msgr2=0x55d8317a33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e93400640 1 --2- 172.21.15.40:0/2640602397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317a2ff0 0x55d8317a33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e93400640 1 -- 172.21.15.40:0/2640602397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8315c8940 con 0x55d8317ec920 2024-03-24T00:35:28.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e93e00640 1 --2- 172.21.15.40:0/2640602397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317a2ff0 0x55d8317a33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:28.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e93400640 1 --2- 172.21.15.40:0/2640602397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317ec920 0x55d8317ecd00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9e840f7b40 tx=0x7f9e840ef910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bde8b372791a4696 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e7fe00640 1 -- 172.21.15.40:0/2640602397 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840f3ab0 con 0x55d8317ec920 2024-03-24T00:35:28.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e7fe00640 1 -- 172.21.15.40:0/2640602397 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e840a9040 con 0x55d8317ec920 2024-03-24T00:35:28.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e7fe00640 1 -- 172.21.15.40:0/2640602397 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840f8070 con 0x55d8317ec920 2024-03-24T00:35:28.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 -- 172.21.15.40:0/2640602397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317ec920 msgr2=0x55d8317ecd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 --2- 172.21.15.40:0/2640602397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317ec920 0x55d8317ecd00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f9e840f7b40 tx=0x7f9e840ef910 comp rx=0 tx=0).stop 2024-03-24T00:35:28.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 -- 172.21.15.40:0/2640602397 shutdown_connections 2024-03-24T00:35:28.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 --2- 172.21.15.40:0/2640602397 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317a2ff0 0x55d8317a33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 --2- 172.21.15.40:0/2640602397 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ed240 0x55d8317a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 --2- 172.21.15.40:0/2640602397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317ec920 0x55d8317ecd00 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 -- 172.21.15.40:0/2640602397 >> 172.21.15.40:0/2640602397 conn(0x55d83179efa0 msgr2=0x55d8317a1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 -- 172.21.15.40:0/2640602397 shutdown_connections 2024-03-24T00:35:28.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 -- 172.21.15.40:0/2640602397 wait complete. 2024-03-24T00:35:28.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 -- start start 2024-03-24T00:35:28.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.258+0000 7f9e9d436000 1 Processor -- start 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317a2ff0 0x55d8317d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317ec920 0x55d8317d2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ed240 0x55d8317d27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8315ef990 con 0x55d8317ed240 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8315aaae0 con 0x55d8317ec920 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d831588750 con 0x55d8317a2ff0 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 10 client.0 _ll_register_callbacks cb 0x55d83178dff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e93400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317a2ff0 0x55d8317d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e93e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ed240 0x55d8317d27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e92a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317ec920 0x55d8317d2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e93400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317a2ff0 0x55d8317d1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:54858/0 (socket says 172.21.15.40:54858) 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e93400640 1 -- 172.21.15.40:0/1899399659 learned_addr learned my addr 172.21.15.40:0/1899399659 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e93400640 1 -- 172.21.15.40:0/1899399659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317ec920 msgr2=0x55d8317d2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e93400640 1 --2- 172.21.15.40:0/1899399659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317ec920 0x55d8317d2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e93400640 1 -- 172.21.15.40:0/1899399659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ed240 msgr2=0x55d8317d27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e93400640 1 --2- 172.21.15.40:0/1899399659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ed240 0x55d8317d27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e93400640 1 -- 172.21.15.40:0/1899399659 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d831635d80 con 0x55d8317a2ff0 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e93400640 1 --2- 172.21.15.40:0/1899399659 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317a2ff0 0x55d8317d1d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9e8401ca40 tx=0x7f9e840ba000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1afb3056d658db7a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e7cc00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e7cc00640 1 -- 172.21.15.40:0/1899399659 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840f8070 con 0x55d8317a2ff0 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e7cc00640 1 -- 172.21.15.40:0/1899399659 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e840eee60 con 0x55d8317a2ff0 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 1 -- 172.21.15.40:0/1899399659 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e840a9040 con 0x55d8317a2ff0 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e7cc00640 1 -- 172.21.15.40:0/1899399659 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e840ef060 con 0x55d8317a2ff0 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 10 client.6215 fetch_fsmap learned FSMap version 72 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 1 -- 172.21.15.40:0/1899399659 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x55d831577d10 con 0x55d8317a2ff0 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e7cc00640 1 -- 172.21.15.40:0/1899399659 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e840a9040 con 0x55d8317a2ff0 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e7cc00640 1 -- 172.21.15.40:0/1899399659 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 72) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f9e8411dce0 con 0x55d8317a2ff0 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 10 client.6215 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 10 client.6215 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:35:28.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 20 client.6215 populate_metadata read hostname 'smithi040' 2024-03-24T00:35:28.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 1 --2- 172.21.15.40:0/1899399659 >> [v2:172.21.15.53:6834/334375209,v1:172.21.15.53:6837/334375209] conn(0x55d8317d3090 0x55d8317feea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 4 client.6215 mds_command: new command op to 6733 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:35:28.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e9d436000 1 -- 172.21.15.40:0/1899399659 --> [v2:172.21.15.53:6834/334375209,v1:172.21.15.53:6837/334375209] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d83158bb60 con 0x55d8317d3090 2024-03-24T00:35:28.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e92a00640 1 --2- 172.21.15.40:0/1899399659 >> [v2:172.21.15.53:6834/334375209,v1:172.21.15.53:6837/334375209] conn(0x55d8317d3090 0x55d8317feea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e7cc00640 10 client.6215 ms_handle_connect on v2:172.21.15.53:6834/334375209 2024-03-24T00:35:28.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.262+0000 7f9e92a00640 1 --2- 172.21.15.40:0/1899399659 >> [v2:172.21.15.53:6834/334375209,v1:172.21.15.53:6837/334375209] conn(0x55d8317d3090 0x55d8317feea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=69e7b81d3033c442 server_cookie=553be1b947d28a26 in_seq=0 out_seq=0 2024-03-24T00:35:28.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e7cc00640 1 -- 172.21.15.40:0/1899399659 <== mds.2 v2:172.21.15.53:6834/334375209 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d83158bb60 con 0x55d8317d3090 2024-03-24T00:35:28.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e7cc00640 10 client.6215 handle_command_reply: tid=0 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e9d436000 1 client.6215 shutdown 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e9d436000 1 -- 172.21.15.40:0/1899399659 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317a2ff0 msgr2=0x55d8317d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e9d436000 1 --2- 172.21.15.40:0/1899399659 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317a2ff0 0x55d8317d1d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9e8401ca40 tx=0x7f9e840ba000 comp rx=0 tx=0).stop 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e9d436000 1 -- 172.21.15.40:0/1899399659 shutdown_connections 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e9d436000 1 --2- 172.21.15.40:0/1899399659 >> [v2:172.21.15.53:6834/334375209,v1:172.21.15.53:6837/334375209] conn(0x55d8317d3090 0x55d8317feea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e9d436000 1 --2- 172.21.15.40:0/1899399659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ed240 0x55d8317d27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e9d436000 1 --2- 172.21.15.40:0/1899399659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317ec920 0x55d8317d2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e7cc00640 0 client.6215 ms_handle_reset on v2:172.21.15.53:6834/334375209 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e9d436000 1 --2- 172.21.15.40:0/1899399659 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317a2ff0 0x55d8317d1d40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e9d436000 1 -- 172.21.15.40:0/1899399659 >> 172.21.15.40:0/1899399659 conn(0x55d83179efa0 msgr2=0x55d8317a1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e9d436000 1 -- 172.21.15.40:0/1899399659 shutdown_connections 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e9d436000 1 -- 172.21.15.40:0/1899399659 wait complete. 2024-03-24T00:35:28.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.266+0000 7f9e9d436000 20 client.6215 trim_cache size 0 max 16384 2024-03-24T00:35:28.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e9d436000 1 Processor -- start 2024-03-24T00:35:28.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e9d436000 1 -- start start 2024-03-24T00:35:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e9d436000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 0x55d8317eb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e9d436000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ec920 0x55d8317eba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e9d436000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ed240 0x55d8317a1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e9d436000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e94202bf0 con 0x55d8317ed240 2024-03-24T00:35:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e9d436000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e942017d0 con 0x55d8317a2ff0 2024-03-24T00:35:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e9d436000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d831598a80 con 0x55d8317ec920 2024-03-24T00:35:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e93400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 0x55d8317eb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e93e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ed240 0x55d8317a1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e92a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ec920 0x55d8317eba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e93400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 0x55d8317eb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:49418/0 (socket says 172.21.15.40:49418) 2024-03-24T00:35:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e93400640 1 -- 172.21.15.40:0/1587793342 learned_addr learned my addr 172.21.15.40:0/1587793342 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e93400640 1 -- 172.21.15.40:0/1587793342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ec920 msgr2=0x55d8317eba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e93400640 1 --2- 172.21.15.40:0/1587793342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ec920 0x55d8317eba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e93400640 1 -- 172.21.15.40:0/1587793342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ed240 msgr2=0x55d8317a1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e93400640 1 --2- 172.21.15.40:0/1587793342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ed240 0x55d8317a1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e93400640 1 -- 172.21.15.40:0/1587793342 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8315bd540 con 0x55d8317a2ff0 2024-03-24T00:35:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e93e00640 1 --2- 172.21.15.40:0/1587793342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ed240 0x55d8317a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:28.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e93400640 1 --2- 172.21.15.40:0/1587793342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 0x55d8317eb530 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9e8401ac10 tx=0x7f9e840f3000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f839648d1f22443 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e7f400640 1 -- 172.21.15.40:0/1587793342 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840ade50 con 0x55d8317a2ff0 2024-03-24T00:35:28.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e7f400640 1 -- 172.21.15.40:0/1587793342 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e840a9070 con 0x55d8317a2ff0 2024-03-24T00:35:28.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.282+0000 7f9e7f400640 1 -- 172.21.15.40:0/1587793342 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840f8070 con 0x55d8317a2ff0 2024-03-24T00:35:28.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 -- 172.21.15.40:0/1587793342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 msgr2=0x55d8317eb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 --2- 172.21.15.40:0/1587793342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 0x55d8317eb530 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9e8401ac10 tx=0x7f9e840f3000 comp rx=0 tx=0).stop 2024-03-24T00:35:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 -- 172.21.15.40:0/1587793342 shutdown_connections 2024-03-24T00:35:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 --2- 172.21.15.40:0/1587793342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ed240 0x55d8317a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 --2- 172.21.15.40:0/1587793342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ec920 0x55d8317eba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 --2- 172.21.15.40:0/1587793342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 0x55d8317eb530 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 -- 172.21.15.40:0/1587793342 >> 172.21.15.40:0/1587793342 conn(0x55d83179efa0 msgr2=0x55d8317d0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 -- 172.21.15.40:0/1587793342 shutdown_connections 2024-03-24T00:35:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 -- 172.21.15.40:0/1587793342 wait complete. 2024-03-24T00:35:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 -- start start 2024-03-24T00:35:28.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 Processor -- start 2024-03-24T00:35:28.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 0x55d8318002e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ec920 0x55d831800820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ed240 0x55d831800d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8315aaae0 con 0x55d8317ec920 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8316ad700 con 0x55d8317a2ff0 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d831588750 con 0x55d8317ed240 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 10 client.0 _ll_register_callbacks cb 0x7f9e8c004bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e92a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ec920 0x55d831800820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e93400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 0x55d8318002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e92a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ec920 0x55d831800820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36794/0 (socket says 172.21.15.40:36794) 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e92a00640 1 -- 172.21.15.40:0/4162187097 learned_addr learned my addr 172.21.15.40:0/4162187097 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e93e00640 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ed240 0x55d831800d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e93400640 1 -- 172.21.15.40:0/4162187097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ed240 msgr2=0x55d831800d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e93400640 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ed240 0x55d831800d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e93400640 1 -- 172.21.15.40:0/4162187097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ec920 msgr2=0x55d831800820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e93400640 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ec920 0x55d831800820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e93400640 1 -- 172.21.15.40:0/4162187097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d8315844b0 con 0x55d8317a2ff0 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e92a00640 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ec920 0x55d831800820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e93400640 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 0x55d8318002e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9e8401c390 tx=0x7f9e84000bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37f4538a3360c23b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e7cc00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e7cc00640 1 -- 172.21.15.40:0/4162187097 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e840ade50 con 0x55d8317a2ff0 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 -- 172.21.15.40:0/4162187097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9e840a9070 con 0x55d8317a2ff0 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e7cc00640 1 -- 172.21.15.40:0/4162187097 <== mon.1 v2:172.21.15.53:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e840eb760 con 0x55d8317a2ff0 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e7cc00640 1 -- 172.21.15.40:0/4162187097 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e840f8070 con 0x55d8317a2ff0 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 10 client.6943 fetch_fsmap learned FSMap version 72 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.286+0000 7f9e9d436000 1 -- 172.21.15.40:0/4162187097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x55d8315972f0 con 0x55d8317a2ff0 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e7cc00640 1 -- 172.21.15.40:0/4162187097 <== mon.1 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e840a9070 con 0x55d8317a2ff0 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e7cc00640 1 -- 172.21.15.40:0/4162187097 <== mon.1 v2:172.21.15.53:3300/0 5 ==== fsmap(e 72) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f9e8411dce0 con 0x55d8317a2ff0 2024-03-24T00:35:28.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 10 client.6943 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 10 client.6943 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 20 client.6943 populate_metadata read hostname 'smithi040' 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.53:6834/334375209,v1:172.21.15.53:6837/334375209] conn(0x55d8318581e0 0x55d83185a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 4 client.6943 mds_command: new command op to 6733 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 -- 172.21.15.40:0/4162187097 --> [v2:172.21.15.53:6834/334375209,v1:172.21.15.53:6837/334375209] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d8315fdbc0 con 0x55d8318581e0 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e92a00640 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.53:6834/334375209,v1:172.21.15.53:6837/334375209] conn(0x55d8318581e0 0x55d83185a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e92a00640 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.53:6834/334375209,v1:172.21.15.53:6837/334375209] conn(0x55d8318581e0 0x55d83185a5d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=254a5d7555528c5e server_cookie=28dc09f0e8b1dfe0 in_seq=0 out_seq=0 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e7cc00640 10 client.6943 ms_handle_connect on v2:172.21.15.53:6834/334375209 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e7cc00640 1 -- 172.21.15.40:0/4162187097 <== mds.2 v2:172.21.15.53:6834/334375209 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x55d8315fdbc0 con 0x55d8318581e0 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e7cc00640 10 client.6943 handle_command_reply: tid=0 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 client.6943 shutdown 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 -- 172.21.15.40:0/4162187097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 msgr2=0x55d8318002e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 0x55d8318002e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9e8401c390 tx=0x7f9e84000bd0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 -- 172.21.15.40:0/4162187097 shutdown_connections 2024-03-24T00:35:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.53:6834/334375209,v1:172.21.15.53:6837/334375209] conn(0x55d8318581e0 0x55d83185a5d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d8317ed240 0x55d831800d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55d8317ec920 0x55d831800820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e7cc00640 0 client.6943 ms_handle_reset on v2:172.21.15.53:6834/334375209 2024-03-24T00:35:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 --2- 172.21.15.40:0/4162187097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d8317a2ff0 0x55d8318002e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 -- 172.21.15.40:0/4162187097 >> 172.21.15.40:0/4162187097 conn(0x55d83179efa0 msgr2=0x55d8317d1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 -- 172.21.15.40:0/4162187097 shutdown_connections 2024-03-24T00:35:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 1 -- 172.21.15.40:0/4162187097 wait complete. 2024-03-24T00:35:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e9d436000 20 client.6943 trim_cache size 0 max 16384 2024-03-24T00:35:28.297 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:35:28.298 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:35:28.298 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-03-24T00:35:28.298 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6733, 2024-03-24T00:35:28.298 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:35:28.298 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:35:28.298 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:35:28.298 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 1.813762042, 2024-03-24T00:35:28.298 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 72, 2024-03-24T00:35:28.298 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 70, 2024-03-24T00:35:28.298 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 70, 2024-03-24T00:35:28.298 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 17.835255712999999 2024-03-24T00:35:28.298 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:35:28.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e98a00640 1 -- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9e74067690 msgr2=0x7f9e74069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e98a00640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9e74067690 0x7f9e74069b50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9e941fa5b0 tx=0x7f9e80009190 comp rx=0 tx=0).stop 2024-03-24T00:35:28.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e98a00640 1 -- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e940ce490 msgr2=0x7f9e94159c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e98a00640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e940ce490 0x7f9e94159c00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f9e840b0040 tx=0x7f9e8401ada0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e98a00640 1 -- 172.21.15.40:0/380076769 shutdown_connections 2024-03-24T00:35:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e98a00640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9e74067690 0x7f9e74069b50 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e98a00640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9e9415c080 0x7f9e94151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e98a00640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9415bca0 0x7f9e940c76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e98a00640 1 --2- 172.21.15.40:0/380076769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e940ce490 0x7f9e94159c00 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e98a00640 1 -- 172.21.15.40:0/380076769 >> 172.21.15.40:0/380076769 conn(0x7f9e940d1aa0 msgr2=0x7f9e94157fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e98a00640 1 -- 172.21.15.40:0/380076769 shutdown_connections 2024-03-24T00:35:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.290+0000 7f9e98a00640 1 -- 172.21.15.40:0/380076769 wait complete. 2024-03-24T00:35:28.313 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:35:28.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b6ea00640 1 Processor -- start 2024-03-24T00:35:28.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b6ea00640 1 -- start start 2024-03-24T00:35:28.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b6ea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b6813ec80 0x7f2b681414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b6ea00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b68070e00 con 0x7f2b680ce800 2024-03-24T00:35:28.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b6ea00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b680cee20 con 0x7f2b681419f0 2024-03-24T00:35:28.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b6ea00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b68141e50 con 0x7f2b6813ec80 2024-03-24T00:35:28.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b67e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2b6813ec80 0x7f2b681414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b5fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2b681419f0 0x7f2b680ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58928/0 (socket says 172.21.15.40:58928) 2024-03-24T00:35:28.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b5fe00640 1 -- 172.21.15.40:0/4287120858 learned_addr learned my addr 172.21.15.40:0/4287120858 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:28.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b6ca00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2b680ce800 0x7f2b68071300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:47576/0 (socket says 172.21.15.40:47576) 2024-03-24T00:35:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b67400640 1 -- 172.21.15.40:0/4287120858 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800987828 0 0) 0x7f2b680cee20 con 0x7f2b681419f0 2024-03-24T00:35:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b67400640 1 -- 172.21.15.40:0/4287120858 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b4c003530 con 0x7f2b681419f0 2024-03-24T00:35:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b67400640 1 -- 172.21.15.40:0/4287120858 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877383605 0 0) 0x7f2b68070e00 con 0x7f2b680ce800 2024-03-24T00:35:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b67400640 1 -- 172.21.15.40:0/4287120858 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b680cee20 con 0x7f2b680ce800 2024-03-24T00:35:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b67e00640 1 -- 172.21.15.40:0/4287120858 >> v1:172.21.15.53:6789/0 conn(0x7f2b681419f0 legacy=0x7f2b680ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b67e00640 1 -- 172.21.15.40:0/4287120858 >> v1:172.21.15.40:6789/0 conn(0x7f2b680ce800 legacy=0x7f2b68071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.454+0000 7f2b67e00640 1 -- 172.21.15.40:0/4287120858 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b680cf100 con 0x7f2b6813ec80 2024-03-24T00:35:28.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.458+0000 7f2b67e00640 1 --2- 172.21.15.40:0/4287120858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b6813ec80 0x7f2b681414b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2b580097c0 tx=0x7f2b5801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcde44b7556c9277 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.458+0000 7f2b67400640 1 -- 172.21.15.40:0/4287120858 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b580aa620 con 0x7f2b6813ec80 2024-03-24T00:35:28.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.458+0000 7f2b67400640 1 -- 172.21.15.40:0/4287120858 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b580af070 con 0x7f2b6813ec80 2024-03-24T00:35:28.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.458+0000 7f2b67400640 1 -- 172.21.15.40:0/4287120858 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b580aa9e0 con 0x7f2b6813ec80 2024-03-24T00:35:28.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 -- 172.21.15.40:0/4287120858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b6813ec80 msgr2=0x7f2b681414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 --2- 172.21.15.40:0/4287120858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b6813ec80 0x7f2b681414b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f2b580097c0 tx=0x7f2b5801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 -- 172.21.15.40:0/4287120858 shutdown_connections 2024-03-24T00:35:28.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 --2- 172.21.15.40:0/4287120858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b6813ec80 0x7f2b681414b0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 -- 172.21.15.40:0/4287120858 >> 172.21.15.40:0/4287120858 conn(0x7f2b680b17b0 msgr2=0x7f2b680b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 -- 172.21.15.40:0/4287120858 shutdown_connections 2024-03-24T00:35:28.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 -- 172.21.15.40:0/4287120858 wait complete. 2024-03-24T00:35:28.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 Processor -- start 2024-03-24T00:35:28.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 -- start start 2024-03-24T00:35:28.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b680ce800 0x7f2b681563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b6813ec80 0x7f2b681577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b681419f0 0x7f2b6806de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680725f0 con 0x7f2b681419f0 2024-03-24T00:35:28.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b68072470 con 0x7f2b6813ec80 2024-03-24T00:35:28.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ea00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b68169d50 con 0x7f2b680ce800 2024-03-24T00:35:28.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b681419f0 0x7f2b6806de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b681419f0 0x7f2b6806de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36824/0 (socket says 172.21.15.40:36824) 2024-03-24T00:35:28.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ca00640 1 -- 172.21.15.40:0/2380382845 learned_addr learned my addr 172.21.15.40:0/2380382845 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b5fe00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b6813ec80 0x7f2b681577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b67e00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b680ce800 0x7f2b681563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ca00640 1 -- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b680ce800 msgr2=0x7f2b681563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ca00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b680ce800 0x7f2b681563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ca00640 1 -- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b6813ec80 msgr2=0x7f2b681577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ca00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b6813ec80 0x7f2b681577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.462+0000 7f2b6ca00640 1 -- 172.21.15.40:0/2380382845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6813bd60 con 0x7f2b681419f0 2024-03-24T00:35:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.466+0000 7f2b6ca00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b681419f0 0x7f2b6806de00 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000c770 tx=0x7f2b6000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.466+0000 7f2b64c00640 1 -- 172.21.15.40:0/2380382845 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b60017070 con 0x7f2b681419f0 2024-03-24T00:35:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.466+0000 7f2b64c00640 1 -- 172.21.15.40:0/2380382845 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6000a640 con 0x7f2b681419f0 2024-03-24T00:35:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.466+0000 7f2b6ea00640 1 -- 172.21.15.40:0/2380382845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b6813c040 con 0x7f2b681419f0 2024-03-24T00:35:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.466+0000 7f2b64c00640 1 -- 172.21.15.40:0/2380382845 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6000aa30 con 0x7f2b681419f0 2024-03-24T00:35:28.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.466+0000 7f2b6ea00640 1 -- 172.21.15.40:0/2380382845 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6813c600 con 0x7f2b681419f0 2024-03-24T00:35:28.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.466+0000 7f2b64c00640 1 -- 172.21.15.40:0/2380382845 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2b6001a070 con 0x7f2b681419f0 2024-03-24T00:35:28.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.466+0000 7f2b71e68000 1 Processor -- start 2024-03-24T00:35:28.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.466+0000 7f2b71e68000 1 -- start start 2024-03-24T00:35:28.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b64c00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2b44067660 0x7f2b44069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b64c00640 1 -- 172.21.15.40:0/2380382845 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b60084f70 con 0x7f2b681419f0 2024-03-24T00:35:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b5fe00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2b44067660 0x7f2b44069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c4118d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c4119240 0x5628c40ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c40ceff0 0x5628c40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c3f1c680 con 0x5628c4119240 2024-03-24T00:35:28.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c3ea3b00 con 0x5628c4118920 2024-03-24T00:35:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5628c3f18720 con 0x5628c40ceff0 2024-03-24T00:35:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b67e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c4118d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b67e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c4118d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:49454/0 (socket says 172.21.15.40:49454) 2024-03-24T00:35:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b5fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c4119240 0x5628c40ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b67e00640 1 -- 172.21.15.40:0/3080196546 learned_addr learned my addr 172.21.15.40:0/3080196546 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b6ca00640 1 --2- 172.21.15.40:0/3080196546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c40ceff0 0x5628c40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b5fe00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2b44067660 0x7f2b44069b20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2b6813a1a0 tx=0x7f2b50007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b67e00640 1 -- 172.21.15.40:0/3080196546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c40ceff0 msgr2=0x5628c40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b67e00640 1 --2- 172.21.15.40:0/3080196546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c40ceff0 0x5628c40cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b67e00640 1 -- 172.21.15.40:0/3080196546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c4119240 msgr2=0x5628c40ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b67e00640 1 --2- 172.21.15.40:0/3080196546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c4119240 0x5628c40ceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b67e00640 1 -- 172.21.15.40:0/3080196546 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628c3ef4940 con 0x5628c4118920 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b67e00640 1 --2- 172.21.15.40:0/3080196546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c4118d00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2b580190b0 tx=0x7f2b58004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f811136cf1e43519 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b5cc00640 1 -- 172.21.15.40:0/3080196546 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b580aa6e0 con 0x5628c4118920 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b5cc00640 1 -- 172.21.15.40:0/3080196546 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b580af040 con 0x5628c4118920 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b5cc00640 1 -- 172.21.15.40:0/3080196546 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b580aaae0 con 0x5628c4118920 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 -- 172.21.15.40:0/3080196546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 msgr2=0x5628c4118d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 --2- 172.21.15.40:0/3080196546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c4118d00 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2b580190b0 tx=0x7f2b58004740 comp rx=0 tx=0).stop 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 -- 172.21.15.40:0/3080196546 shutdown_connections 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 --2- 172.21.15.40:0/3080196546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c40ceff0 0x5628c40cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 --2- 172.21.15.40:0/3080196546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c4119240 0x5628c40ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 --2- 172.21.15.40:0/3080196546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c4118d00 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 -- 172.21.15.40:0/3080196546 >> 172.21.15.40:0/3080196546 conn(0x5628c40cafa0 msgr2=0x5628c40cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 -- 172.21.15.40:0/3080196546 shutdown_connections 2024-03-24T00:35:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 -- 172.21.15.40:0/3080196546 wait complete. 2024-03-24T00:35:28.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.470+0000 7f2b71e68000 1 -- start start 2024-03-24T00:35:28.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 1 Processor -- start 2024-03-24T00:35:28.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c40ceff0 0x5628c40fdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c40fe280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c4119240 0x5628c40fe7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c3f1b990 con 0x5628c4119240 2024-03-24T00:35:28.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c3ed6ae0 con 0x5628c4118920 2024-03-24T00:35:28.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5628c3eb4750 con 0x5628c40ceff0 2024-03-24T00:35:28.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 10 client.0 _ll_register_callbacks cb 0x5628c40b9ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:35:28.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b5fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c40fe280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b67e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c40ceff0 0x5628c40fdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b6ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c4119240 0x5628c40fe7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b5fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c40fe280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:49460/0 (socket says 172.21.15.40:49460) 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b5fe00640 1 -- 172.21.15.40:0/590818342 learned_addr learned my addr 172.21.15.40:0/590818342 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b5fe00640 1 -- 172.21.15.40:0/590818342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c40ceff0 msgr2=0x5628c40fdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b5fe00640 1 --2- 172.21.15.40:0/590818342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c40ceff0 0x5628c40fdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b5fe00640 1 -- 172.21.15.40:0/590818342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c4119240 msgr2=0x5628c40fe7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b5fe00640 1 --2- 172.21.15.40:0/590818342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c4119240 0x5628c40fe7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b5fe00640 1 -- 172.21.15.40:0/590818342 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628c3f61d80 con 0x5628c4118920 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b5fe00640 1 --2- 172.21.15.40:0/590818342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c40fe280 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2b500156e0 tx=0x7f2b50001160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78f165f37cf4d730 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b55600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b55600640 1 -- 172.21.15.40:0/590818342 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50025020 con 0x5628c4118920 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 1 -- 172.21.15.40:0/590818342 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2b580af040 con 0x5628c4118920 2024-03-24T00:35:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b55600640 1 -- 172.21.15.40:0/590818342 <== mon.1 v2:172.21.15.53:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b50014790 con 0x5628c4118920 2024-03-24T00:35:28.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b55600640 1 -- 172.21.15.40:0/590818342 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b5002f050 con 0x5628c4118920 2024-03-24T00:35:28.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 10 client.6955 fetch_fsmap learned FSMap version 72 2024-03-24T00:35:28.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 1 -- 172.21.15.40:0/590818342 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x5628c3ea3d10 con 0x5628c4118920 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b55600640 1 -- 172.21.15.40:0/590818342 <== mon.1 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b580af040 con 0x5628c4118920 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b55600640 1 -- 172.21.15.40:0/590818342 <== mon.1 v2:172.21.15.53:3300/0 5 ==== fsmap(e 72) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f2b50032020 con 0x5628c4118920 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 10 client.6955 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 10 client.6955 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 20 client.6955 populate_metadata read hostname 'smithi040' 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 1 --2- 172.21.15.40:0/590818342 >> [v2:172.21.15.40:6832/221849812,v1:172.21.15.40:6834/221849812] conn(0x5628c40ff030 0x5628c412aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 4 client.6955 mds_command: new command op to 6951 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b71e68000 1 -- 172.21.15.40:0/590818342 --> [v2:172.21.15.40:6832/221849812,v1:172.21.15.40:6834/221849812] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5628c3eb7b60 con 0x5628c40ff030 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.474+0000 7f2b67e00640 1 --2- 172.21.15.40:0/590818342 >> [v2:172.21.15.40:6832/221849812,v1:172.21.15.40:6834/221849812] conn(0x5628c40ff030 0x5628c412aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b67e00640 1 --2- 172.21.15.40:0/590818342 >> [v2:172.21.15.40:6832/221849812,v1:172.21.15.40:6834/221849812] conn(0x5628c40ff030 0x5628c412aea0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=48e6b780a2991e05 server_cookie=8d377eb8b7e5673a in_seq=0 out_seq=0 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b55600640 10 client.6955 ms_handle_connect on v2:172.21.15.40:6832/221849812 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b55600640 1 -- 172.21.15.40:0/590818342 <== mds.1 v2:172.21.15.40:6832/221849812 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5628c3eb7b60 con 0x5628c40ff030 2024-03-24T00:35:28.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b55600640 10 client.6955 handle_command_reply: tid=0 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b71e68000 1 client.6955 shutdown 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b71e68000 1 -- 172.21.15.40:0/590818342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 msgr2=0x5628c40fe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b71e68000 1 --2- 172.21.15.40:0/590818342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c40fe280 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f2b500156e0 tx=0x7f2b50001160 comp rx=0 tx=0).stop 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b71e68000 1 -- 172.21.15.40:0/590818342 shutdown_connections 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b71e68000 1 --2- 172.21.15.40:0/590818342 >> [v2:172.21.15.40:6832/221849812,v1:172.21.15.40:6834/221849812] conn(0x5628c40ff030 0x5628c412aea0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b71e68000 1 --2- 172.21.15.40:0/590818342 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c4119240 0x5628c40fe7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b71e68000 1 --2- 172.21.15.40:0/590818342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c40fe280 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b55600640 0 client.6955 ms_handle_reset on v2:172.21.15.40:6832/221849812 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b71e68000 1 --2- 172.21.15.40:0/590818342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c40ceff0 0x5628c40fdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b71e68000 1 -- 172.21.15.40:0/590818342 >> 172.21.15.40:0/590818342 conn(0x5628c40cafa0 msgr2=0x5628c40cdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b71e68000 1 -- 172.21.15.40:0/590818342 shutdown_connections 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b71e68000 1 -- 172.21.15.40:0/590818342 wait complete. 2024-03-24T00:35:28.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.478+0000 7f2b71e68000 20 client.6955 trim_cache size 0 max 16384 2024-03-24T00:35:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.494+0000 7f2b71e68000 1 Processor -- start 2024-03-24T00:35:28.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.494+0000 7f2b71e68000 1 -- start start 2024-03-24T00:35:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 0x5628c40fc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c40fce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c4119240 0x5628c41174d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b68202ed0 con 0x5628c40ceff0 2024-03-24T00:35:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b682023b0 con 0x5628c4118920 2024-03-24T00:35:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b682028b0 con 0x5628c4119240 2024-03-24T00:35:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 0x5628c40fc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 0x5628c40fc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36852/0 (socket says 172.21.15.40:36852) 2024-03-24T00:35:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b67e00640 1 -- 172.21.15.40:0/4101779690 learned_addr learned my addr 172.21.15.40:0/4101779690 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b6ca00640 1 --2- 172.21.15.40:0/4101779690 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c4119240 0x5628c41174d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b5fe00640 1 --2- 172.21.15.40:0/4101779690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c40fce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b67e00640 1 -- 172.21.15.40:0/4101779690 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c4119240 msgr2=0x5628c41174d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b67e00640 1 --2- 172.21.15.40:0/4101779690 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c4119240 0x5628c41174d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b67e00640 1 -- 172.21.15.40:0/4101779690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 msgr2=0x5628c40fce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b67e00640 1 --2- 172.21.15.40:0/4101779690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c40fce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b67e00640 1 -- 172.21.15.40:0/4101779690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628c40fe240 con 0x5628c40ceff0 2024-03-24T00:35:28.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b67e00640 1 --2- 172.21.15.40:0/4101779690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 0x5628c40fc900 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2b5801c980 tx=0x7f2b58009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58124e42d38b0fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b57e00640 1 -- 172.21.15.40:0/4101779690 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b58006a30 con 0x5628c40ceff0 2024-03-24T00:35:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b57e00640 1 -- 172.21.15.40:0/4101779690 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b580af040 con 0x5628c40ceff0 2024-03-24T00:35:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b57e00640 1 -- 172.21.15.40:0/4101779690 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b580044f0 con 0x5628c40ceff0 2024-03-24T00:35:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 -- 172.21.15.40:0/4101779690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 msgr2=0x5628c40fc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 --2- 172.21.15.40:0/4101779690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 0x5628c40fc900 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f2b5801c980 tx=0x7f2b58009190 comp rx=0 tx=0).stop 2024-03-24T00:35:28.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 -- 172.21.15.40:0/4101779690 shutdown_connections 2024-03-24T00:35:28.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 --2- 172.21.15.40:0/4101779690 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c4119240 0x5628c41174d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 --2- 172.21.15.40:0/4101779690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c40fce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 --2- 172.21.15.40:0/4101779690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 0x5628c40fc900 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 -- 172.21.15.40:0/4101779690 >> 172.21.15.40:0/4101779690 conn(0x5628c40cafa0 msgr2=0x5628c4125f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 -- 172.21.15.40:0/4101779690 shutdown_connections 2024-03-24T00:35:28.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 -- 172.21.15.40:0/4101779690 wait complete. 2024-03-24T00:35:28.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 -- start start 2024-03-24T00:35:28.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.498+0000 7f2b71e68000 1 Processor -- start 2024-03-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 0x5628c417e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c4180700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c4119240 0x5628c4182c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628c3ec4a80 con 0x5628c40ceff0 2024-03-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680b3ca0 con 0x5628c4118920 2024-03-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5628c3fd9700 con 0x5628c4119240 2024-03-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 10 client.0 _ll_register_callbacks cb 0x7f2b50001d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 0x5628c417e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b67e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 0x5628c417e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36864/0 (socket says 172.21.15.40:36864) 2024-03-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b67e00640 1 -- 172.21.15.40:0/2605052768 learned_addr learned my addr 172.21.15.40:0/2605052768 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b6ca00640 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c4119240 0x5628c4182c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b5fe00640 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c4180700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b67e00640 1 -- 172.21.15.40:0/2605052768 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c4119240 msgr2=0x5628c4182c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b67e00640 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c4119240 0x5628c4182c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b67e00640 1 -- 172.21.15.40:0/2605052768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 msgr2=0x5628c4180700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b67e00640 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c4180700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b67e00640 1 -- 172.21.15.40:0/2605052768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628c3ee9540 con 0x5628c40ceff0 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b67e00640 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 0x5628c417e1b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f2b580097c0 tx=0x7f2b5801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31fa2697f40fa680 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b55600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b55600640 1 -- 172.21.15.40:0/2605052768 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b58004100 con 0x5628c40ceff0 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b55600640 1 -- 172.21.15.40:0/2605052768 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b580042a0 con 0x5628c40ceff0 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 -- 172.21.15.40:0/2605052768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2b580af040 con 0x5628c40ceff0 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b55600640 1 -- 172.21.15.40:0/2605052768 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b580044a0 con 0x5628c40ceff0 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 10 client.7179 fetch_fsmap learned FSMap version 72 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 -- 172.21.15.40:0/2605052768 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x5628c3eb04b0 con 0x5628c40ceff0 2024-03-24T00:35:28.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b55600640 1 -- 172.21.15.40:0/2605052768 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b580af040 con 0x5628c40ceff0 2024-03-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b55600640 1 -- 172.21.15.40:0/2605052768 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 72) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f2b580d8060 con 0x5628c40ceff0 2024-03-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 10 client.7179 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 10 client.7179 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 20 client.7179 populate_metadata read hostname 'smithi040' 2024-03-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.40:6832/221849812,v1:172.21.15.40:6834/221849812] conn(0x5628c41833a0 0x5628c4185790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 4 client.7179 mds_command: new command op to 6951 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 -- 172.21.15.40:0/2605052768 --> [v2:172.21.15.40:6832/221849812,v1:172.21.15.40:6834/221849812] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5628c3f29bc0 con 0x5628c41833a0 2024-03-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b5fe00640 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.40:6832/221849812,v1:172.21.15.40:6834/221849812] conn(0x5628c41833a0 0x5628c4185790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b55600640 10 client.7179 ms_handle_connect on v2:172.21.15.40:6832/221849812 2024-03-24T00:35:28.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b5fe00640 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.40:6832/221849812,v1:172.21.15.40:6834/221849812] conn(0x5628c41833a0 0x5628c4185790 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e54b4234caa5be02 server_cookie=490072f054fa8a in_seq=0 out_seq=0 2024-03-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b55600640 1 -- 172.21.15.40:0/2605052768 <== mds.1 v2:172.21.15.40:6832/221849812 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5628c3f29bc0 con 0x5628c41833a0 2024-03-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b55600640 10 client.7179 handle_command_reply: tid=0 2024-03-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 client.7179 shutdown 2024-03-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 -- 172.21.15.40:0/2605052768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 msgr2=0x5628c417e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 0x5628c417e1b0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f2b580097c0 tx=0x7f2b5801ace0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 -- 172.21.15.40:0/2605052768 shutdown_connections 2024-03-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.40:6832/221849812,v1:172.21.15.40:6834/221849812] conn(0x5628c41833a0 0x5628c4185790 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5628c4119240 0x5628c4182c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5628c4118920 0x5628c4180700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 --2- 172.21.15.40:0/2605052768 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5628c40ceff0 0x5628c417e1b0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b55600640 0 client.7179 ms_handle_reset on v2:172.21.15.40:6832/221849812 2024-03-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.502+0000 7f2b71e68000 1 -- 172.21.15.40:0/2605052768 >> 172.21.15.40:0/2605052768 conn(0x5628c40cafa0 msgr2=0x5628c40fb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b71e68000 1 -- 172.21.15.40:0/2605052768 shutdown_connections 2024-03-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b71e68000 1 -- 172.21.15.40:0/2605052768 wait complete. 2024-03-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b71e68000 20 client.7179 trim_cache size 0 max 16384 2024-03-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:35:28.511 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-03-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6951, 2024-03-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.036659271, 2024-03-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 72, 2024-03-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 70, 2024-03-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 70, 2024-03-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 18.061496808000001 2024-03-24T00:35:28.512 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b6ea00640 1 -- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2b44067660 msgr2=0x7f2b44069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b6ea00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2b44067660 0x7f2b44069b20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2b6813a1a0 tx=0x7f2b50007d40 comp rx=0 tx=0).stop 2024-03-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b6ea00640 1 -- 172.21.15.40:0/2380382845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b681419f0 msgr2=0x7f2b6806de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b6ea00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b681419f0 0x7f2b6806de00 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000c770 tx=0x7f2b6000cc30 comp rx=0 tx=0).stop 2024-03-24T00:35:28.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b6ea00640 1 -- 172.21.15.40:0/2380382845 shutdown_connections 2024-03-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b6ea00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2b44067660 0x7f2b44069b20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b6ea00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2b681419f0 0x7f2b6806de00 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b6ea00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b6813ec80 0x7f2b681577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b6ea00640 1 --2- 172.21.15.40:0/2380382845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b680ce800 0x7f2b681563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b6ea00640 1 -- 172.21.15.40:0/2380382845 >> 172.21.15.40:0/2380382845 conn(0x7f2b680b17b0 msgr2=0x7f2b68071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b6ea00640 1 -- 172.21.15.40:0/2380382845 shutdown_connections 2024-03-24T00:35:28.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.506+0000 7f2b6ea00640 1 -- 172.21.15.40:0/2380382845 wait complete. 2024-03-24T00:35:28.528 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:35:28.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e8400640 1 Processor -- start 2024-03-24T00:35:28.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e8400640 1 -- start start 2024-03-24T00:35:28.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e8400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14e0156320 0x7f14e0072920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e8400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14e006ee60 0x7f14e0067300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e8400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e00af600 con 0x7f14e0066e80 2024-03-24T00:35:28.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e8400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e00b0470 con 0x7f14e006ee60 2024-03-24T00:35:28.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e8400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00af8a0 con 0x7f14e0156320 2024-03-24T00:35:28.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e5a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14e0156320 0x7f14e0072920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e6400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f14e006ee60 0x7f14e0067300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e5000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f14e0066e80 0x7f14e015f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58940/0 (socket says 172.21.15.40:58940) 2024-03-24T00:35:28.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e5000640 1 -- 172.21.15.40:0/1591927211 learned_addr learned my addr 172.21.15.40:0/1591927211 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:28.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14dfe00640 1 -- 172.21.15.40:0/1591927211 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121702759 0 0) 0x7f14e00af600 con 0x7f14e0066e80 2024-03-24T00:35:28.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e5a00640 1 -- 172.21.15.40:0/1591927211 >> v2:172.21.15.40:3300/0 conn(0x7f14e006ee60 msgr2=0x7f14e0067300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e5a00640 1 --2- 172.21.15.40:0/1591927211 >> v2:172.21.15.40:3300/0 conn(0x7f14e006ee60 0x7f14e0067300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.714+0000 7f14e5a00640 1 -- 172.21.15.40:0/1591927211 >> v1:172.21.15.53:6789/0 conn(0x7f14e0066e80 legacy=0x7f14e015f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14e5a00640 1 -- 172.21.15.40:0/1591927211 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e00afb40 con 0x7f14e0156320 2024-03-24T00:35:28.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14e5a00640 1 --2- 172.21.15.40:0/1591927211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14e0156320 0x7f14e0072920 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f14d40097c0 tx=0x7f14d401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca1aebb9326b2e46 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14dfe00640 1 -- 172.21.15.40:0/1591927211 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d40b0070 con 0x7f14e0156320 2024-03-24T00:35:28.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14dfe00640 1 -- 172.21.15.40:0/1591927211 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d40ab600 con 0x7f14e0156320 2024-03-24T00:35:28.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14dfe00640 1 -- 172.21.15.40:0/1591927211 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d40ab9d0 con 0x7f14e0156320 2024-03-24T00:35:28.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14e8400640 1 -- 172.21.15.40:0/1591927211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14e0156320 msgr2=0x7f14e0072920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14e8400640 1 --2- 172.21.15.40:0/1591927211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14e0156320 0x7f14e0072920 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f14d40097c0 tx=0x7f14d401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14e8400640 1 -- 172.21.15.40:0/1591927211 shutdown_connections 2024-03-24T00:35:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14e8400640 1 --2- 172.21.15.40:0/1591927211 >> v2:172.21.15.40:3300/0 conn(0x7f14e006ee60 0x7f14e0067300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14e8400640 1 --2- 172.21.15.40:0/1591927211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14e0156320 0x7f14e0072920 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14e8400640 1 -- 172.21.15.40:0/1591927211 >> 172.21.15.40:0/1591927211 conn(0x7f14e006b150 msgr2=0x7f14e006b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14e8400640 1 -- 172.21.15.40:0/1591927211 shutdown_connections 2024-03-24T00:35:28.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.718+0000 7f14e8400640 1 -- 172.21.15.40:0/1591927211 wait complete. 2024-03-24T00:35:28.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e8400640 1 Processor -- start 2024-03-24T00:35:28.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e8400640 1 -- start start 2024-03-24T00:35:28.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e8400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14e0066e80 0x7f14e014b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e8400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14e006ee60 0x7f14e01f6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e8400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14e0156320 0x7f14e014d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e8400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e0072fe0 con 0x7f14e006ee60 2024-03-24T00:35:28.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e8400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e0072e60 con 0x7f14e0066e80 2024-03-24T00:35:28.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e8400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e00adf00 con 0x7f14e0156320 2024-03-24T00:35:28.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14e006ee60 0x7f14e01f6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14e006ee60 0x7f14e01f6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36896/0 (socket says 172.21.15.40:36896) 2024-03-24T00:35:28.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e5000640 1 -- 172.21.15.40:0/812280858 learned_addr learned my addr 172.21.15.40:0/812280858 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e5a00640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14e0066e80 0x7f14e014b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e6400640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14e0156320 0x7f14e014d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e5000640 1 -- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14e0156320 msgr2=0x7f14e014d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e5000640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14e0156320 0x7f14e014d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e5000640 1 -- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14e0066e80 msgr2=0x7f14e014b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e5000640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14e0066e80 0x7f14e014b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e5000640 1 -- 172.21.15.40:0/812280858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e015d440 con 0x7f14e006ee60 2024-03-24T00:35:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e5000640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14e006ee60 0x7f14e01f6af0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00c480 tx=0x7f14cc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14dd600640 1 -- 172.21.15.40:0/812280858 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc018070 con 0x7f14e006ee60 2024-03-24T00:35:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14dd600640 1 -- 172.21.15.40:0/812280858 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc007500 con 0x7f14e006ee60 2024-03-24T00:35:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14dd600640 1 -- 172.21.15.40:0/812280858 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc004a30 con 0x7f14e006ee60 2024-03-24T00:35:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e8400640 1 -- 172.21.15.40:0/812280858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e015d720 con 0x7f14e006ee60 2024-03-24T00:35:28.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.722+0000 7f14e8400640 1 -- 172.21.15.40:0/812280858 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e015dce0 con 0x7f14e006ee60 2024-03-24T00:35:28.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.726+0000 7f14dd600640 1 -- 172.21.15.40:0/812280858 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f14cc01b070 con 0x7f14e006ee60 2024-03-24T00:35:28.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.726+0000 7f14dd600640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f14c0067660 0x7f14c0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.726+0000 7f14dd600640 1 -- 172.21.15.40:0/812280858 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14cc084a80 con 0x7f14e006ee60 2024-03-24T00:35:28.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.726+0000 7f14e5a00640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f14c0067660 0x7f14c0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.726+0000 7f14eb7fb000 1 Processor -- start 2024-03-24T00:35:28.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 -- start start 2024-03-24T00:35:28.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14e5a00640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f14c0067660 0x7f14c0069b20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f14d40ad000 tx=0x7f14d40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd926920 0x5580fd926d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd927240 0x5580fd8dcab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8dcff0 0x5580fd8dd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580fd72a680 con 0x5580fd926920 2024-03-24T00:35:28.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580fd6b1b00 con 0x5580fd927240 2024-03-24T00:35:28.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580fd726720 con 0x5580fd8dcff0 2024-03-24T00:35:28.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14e5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd926920 0x5580fd926d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14e5a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd926920 0x5580fd926d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36912/0 (socket says 172.21.15.40:36912) 2024-03-24T00:35:28.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14e5000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd927240 0x5580fd8dcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14e5a00640 1 -- 172.21.15.40:0/571767919 learned_addr learned my addr 172.21.15.40:0/571767919 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14e6400640 1 --2- 172.21.15.40:0/571767919 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8dcff0 0x5580fd8dd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14e5000640 1 -- 172.21.15.40:0/571767919 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8dcff0 msgr2=0x5580fd8dd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14e5000640 1 --2- 172.21.15.40:0/571767919 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8dcff0 0x5580fd8dd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14e5000640 1 -- 172.21.15.40:0/571767919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd926920 msgr2=0x5580fd926d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14e5000640 1 --2- 172.21.15.40:0/571767919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd926920 0x5580fd926d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14e5000640 1 -- 172.21.15.40:0/571767919 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580fd702940 con 0x5580fd927240 2024-03-24T00:35:28.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14e5000640 1 --2- 172.21.15.40:0/571767919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd927240 0x5580fd8dcab0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f14cc083090 tx=0x7f14cc05d7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7a716ed970bf951 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14d2000640 1 -- 172.21.15.40:0/571767919 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc0517b0 con 0x5580fd927240 2024-03-24T00:35:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14d2000640 1 -- 172.21.15.40:0/571767919 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc05d310 con 0x5580fd927240 2024-03-24T00:35:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14d2000640 1 -- 172.21.15.40:0/571767919 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc054590 con 0x5580fd927240 2024-03-24T00:35:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 -- 172.21.15.40:0/571767919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd927240 msgr2=0x5580fd8dcab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/571767919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd927240 0x5580fd8dcab0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f14cc083090 tx=0x7f14cc05d7a0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 -- 172.21.15.40:0/571767919 shutdown_connections 2024-03-24T00:35:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/571767919 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8dcff0 0x5580fd8dd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/571767919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd927240 0x5580fd8dcab0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/571767919 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd926920 0x5580fd926d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 -- 172.21.15.40:0/571767919 >> 172.21.15.40:0/571767919 conn(0x5580fd8d8fa0 msgr2=0x5580fd8db390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 -- 172.21.15.40:0/571767919 shutdown_connections 2024-03-24T00:35:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.730+0000 7f14eb7fb000 1 -- 172.21.15.40:0/571767919 wait complete. 2024-03-24T00:35:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14eb7fb000 1 -- start start 2024-03-24T00:35:28.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14eb7fb000 1 Processor -- start 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14eb7fb000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8dcff0 0x5580fd90bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14eb7fb000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd926920 0x5580fd90c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14eb7fb000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd927240 0x5580fd90c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14eb7fb000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580fd729990 con 0x5580fd926920 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14eb7fb000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580fd6e4ae0 con 0x5580fd927240 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14eb7fb000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580fd6c2750 con 0x5580fd8dcff0 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14eb7fb000 10 client.0 _ll_register_callbacks cb 0x5580fd8c7ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14e5a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8dcff0 0x5580fd90bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14e5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd926920 0x5580fd90c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14e5a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8dcff0 0x5580fd90bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:54946/0 (socket says 172.21.15.40:54946) 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14e5a00640 1 -- 172.21.15.40:0/1838956307 learned_addr learned my addr 172.21.15.40:0/1838956307 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14e6400640 1 --2- 172.21.15.40:0/1838956307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd927240 0x5580fd90c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14e5000640 1 -- 172.21.15.40:0/1838956307 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8dcff0 msgr2=0x5580fd90bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14e5000640 1 --2- 172.21.15.40:0/1838956307 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8dcff0 0x5580fd90bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14e5000640 1 -- 172.21.15.40:0/1838956307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd927240 msgr2=0x5580fd90c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14e5000640 1 --2- 172.21.15.40:0/1838956307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd927240 0x5580fd90c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14e5000640 1 -- 172.21.15.40:0/1838956307 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5580fd76fd80 con 0x5580fd926920 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14e5000640 1 --2- 172.21.15.40:0/1838956307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd926920 0x5580fd90c280 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f14cc083010 tx=0x7f14cc000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6915b7c77d53ea34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14c6a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14c6a00640 1 -- 172.21.15.40:0/1838956307 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14cc0513e0 con 0x5580fd926920 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14eb7fb000 1 -- 172.21.15.40:0/1838956307 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14cc05d310 con 0x5580fd926920 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14c6a00640 1 -- 172.21.15.40:0/1838956307 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14cc054790 con 0x5580fd926920 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14c6a00640 1 -- 172.21.15.40:0/1838956307 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14cc07a050 con 0x5580fd926920 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14eb7fb000 10 client.7191 fetch_fsmap learned FSMap version 72 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14eb7fb000 1 -- 172.21.15.40:0/1838956307 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x5580fd6b1d10 con 0x5580fd926920 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14c6a00640 1 -- 172.21.15.40:0/1838956307 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14cc05d310 con 0x5580fd926920 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.734+0000 7f14c6a00640 1 -- 172.21.15.40:0/1838956307 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 72) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f14cc070020 con 0x5580fd926920 2024-03-24T00:35:28.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 10 client.7191 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:35:28.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 10 client.7191 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:35:28.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 20 client.7191 populate_metadata read hostname 'smithi040' 2024-03-24T00:35:28.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/1838956307 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x5580fd90d060 0x5580fd938ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 4 client.7191 mds_command: new command op to 6954 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:35:28.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 -- 172.21.15.40:0/1838956307 --> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5580fd6c5b60 con 0x5580fd90d060 2024-03-24T00:35:28.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14e5a00640 1 --2- 172.21.15.40:0/1838956307 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x5580fd90d060 0x5580fd938ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14c6a00640 10 client.7191 ms_handle_connect on v2:172.21.15.40:6833/2177098794 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14e5a00640 1 --2- 172.21.15.40:0/1838956307 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x5580fd90d060 0x5580fd938ea0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a69460fa9486eff8 server_cookie=b6c03e98b39d2be0 in_seq=0 out_seq=0 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14c6a00640 1 -- 172.21.15.40:0/1838956307 <== mds.0 v2:172.21.15.40:6833/2177098794 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5580fd6c5b60 con 0x5580fd90d060 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14c6a00640 10 client.7191 handle_command_reply: tid=0 2024-03-24T00:35:28.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 client.7191 shutdown 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 -- 172.21.15.40:0/1838956307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd926920 msgr2=0x5580fd90c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/1838956307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd926920 0x5580fd90c280 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f14cc083010 tx=0x7f14cc000ee0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 -- 172.21.15.40:0/1838956307 shutdown_connections 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/1838956307 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x5580fd90d060 0x5580fd938ea0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/1838956307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd927240 0x5580fd90c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/1838956307 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd926920 0x5580fd90c280 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14c6a00640 0 client.7191 ms_handle_reset on v2:172.21.15.40:6833/2177098794 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/1838956307 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8dcff0 0x5580fd90bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 -- 172.21.15.40:0/1838956307 >> 172.21.15.40:0/1838956307 conn(0x5580fd8d8fa0 msgr2=0x5580fd8dbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 -- 172.21.15.40:0/1838956307 shutdown_connections 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 1 -- 172.21.15.40:0/1838956307 wait complete. 2024-03-24T00:35:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.738+0000 7f14eb7fb000 20 client.7191 trim_cache size 0 max 16384 2024-03-24T00:35:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.754+0000 7f14eb7fb000 1 Processor -- start 2024-03-24T00:35:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.754+0000 7f14eb7fb000 1 -- start start 2024-03-24T00:35:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd90bed0 0x5580fd8dc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8d8fa0 0x5580fd9095d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd8dcff0 0x5580fd909b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e0202950 con 0x5580fd8dcff0 2024-03-24T00:35:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e015c580 con 0x5580fd90bed0 2024-03-24T00:35:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580fd6d2a80 con 0x5580fd8d8fa0 2024-03-24T00:35:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14e5a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd90bed0 0x5580fd8dc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14e5000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8d8fa0 0x5580fd9095d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14e5a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd90bed0 0x5580fd8dc7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:49518/0 (socket says 172.21.15.40:49518) 2024-03-24T00:35:28.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14e5a00640 1 -- 172.21.15.40:0/734681631 learned_addr learned my addr 172.21.15.40:0/734681631 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14e6400640 1 --2- 172.21.15.40:0/734681631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd8dcff0 0x5580fd909b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14e5a00640 1 -- 172.21.15.40:0/734681631 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8d8fa0 msgr2=0x5580fd9095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14e5a00640 1 --2- 172.21.15.40:0/734681631 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8d8fa0 0x5580fd9095d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14e5a00640 1 -- 172.21.15.40:0/734681631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd8dcff0 msgr2=0x5580fd909b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14e5a00640 1 --2- 172.21.15.40:0/734681631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd8dcff0 0x5580fd909b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14e5a00640 1 -- 172.21.15.40:0/734681631 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580fd8e11d0 con 0x5580fd90bed0 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14e5000640 1 --2- 172.21.15.40:0/734681631 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8d8fa0 0x5580fd9095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14e5a00640 1 --2- 172.21.15.40:0/734681631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd90bed0 0x5580fd8dc7c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f14d40ae040 tx=0x7f14d40a8000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fef4e0b201c7e11c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14d1600640 1 -- 172.21.15.40:0/734681631 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d40d6070 con 0x5580fd90bed0 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14d1600640 1 -- 172.21.15.40:0/734681631 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d40d0050 con 0x5580fd90bed0 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14d1600640 1 -- 172.21.15.40:0/734681631 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d40bb070 con 0x5580fd90bed0 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 -- 172.21.15.40:0/734681631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd90bed0 msgr2=0x5580fd8dc7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/734681631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd90bed0 0x5580fd8dc7c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f14d40ae040 tx=0x7f14d40a8000 comp rx=0 tx=0).stop 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 -- 172.21.15.40:0/734681631 shutdown_connections 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/734681631 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd8dcff0 0x5580fd909b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/734681631 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8d8fa0 0x5580fd9095d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/734681631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd90bed0 0x5580fd8dc7c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 -- 172.21.15.40:0/734681631 >> 172.21.15.40:0/734681631 conn(0x5580fd8df750 msgr2=0x5580fd925500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 -- 172.21.15.40:0/734681631 shutdown_connections 2024-03-24T00:35:28.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 -- 172.21.15.40:0/734681631 wait complete. 2024-03-24T00:35:28.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 -- start start 2024-03-24T00:35:28.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.758+0000 7f14eb7fb000 1 Processor -- start 2024-03-24T00:35:28.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8d8fa0 0x5580fd98c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd8dcff0 0x5580fd98ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd90bed0 0x5580fd93a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580fd6c2750 con 0x5580fd90bed0 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580fd7e7700 con 0x5580fd8dcff0 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580fd729990 con 0x5580fd8d8fa0 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 10 client.0 _ll_register_callbacks cb 0x7f14cc06d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14e6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd90bed0 0x5580fd93a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14e6400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd90bed0 0x5580fd93a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36930/0 (socket says 172.21.15.40:36930) 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14e5a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8d8fa0 0x5580fd98c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14e6400640 1 -- 172.21.15.40:0/506186690 learned_addr learned my addr 172.21.15.40:0/506186690 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14e5000640 1 --2- 172.21.15.40:0/506186690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd8dcff0 0x5580fd98ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14e6400640 1 -- 172.21.15.40:0/506186690 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8d8fa0 msgr2=0x5580fd98c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14e6400640 1 --2- 172.21.15.40:0/506186690 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8d8fa0 0x5580fd98c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14e6400640 1 -- 172.21.15.40:0/506186690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd8dcff0 msgr2=0x5580fd98ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14e6400640 1 --2- 172.21.15.40:0/506186690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd8dcff0 0x5580fd98ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14e6400640 1 -- 172.21.15.40:0/506186690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5580fd6f7540 con 0x5580fd90bed0 2024-03-24T00:35:28.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14e6400640 1 --2- 172.21.15.40:0/506186690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd90bed0 0x5580fd93a350 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f14d8015390 tx=0x7f14d8015850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d2973540428679 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:28.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14c6a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:35:28.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14c6a00640 1 -- 172.21.15.40:0/506186690 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d8025060 con 0x5580fd90bed0 2024-03-24T00:35:28.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14c6a00640 1 -- 172.21.15.40:0/506186690 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14d8017320 con 0x5580fd90bed0 2024-03-24T00:35:28.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 1 -- 172.21.15.40:0/506186690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14d40d0050 con 0x5580fd90bed0 2024-03-24T00:35:28.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14c6a00640 1 -- 172.21.15.40:0/506186690 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=72) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14d8020410 con 0x5580fd90bed0 2024-03-24T00:35:28.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 10 client.7197 fetch_fsmap learned FSMap version 72 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 1 -- 172.21.15.40:0/506186690 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=72,monmap=2+}) v3 -- 0x5580fd6be4b0 con 0x5580fd90bed0 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14c6a00640 1 -- 172.21.15.40:0/506186690 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d40d0050 con 0x5580fd90bed0 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14c6a00640 1 -- 172.21.15.40:0/506186690 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 72) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f14d8020670 con 0x5580fd90bed0 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 10 client.7197 fetch_fsmap finished waiting for FSMap version 72 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 10 client.7197 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 20 client.7197 populate_metadata read hostname 'smithi040' 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/506186690 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x5580fd926920 0x5580fd995620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 4 client.7197 mds_command: new command op to 6954 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14eb7fb000 1 -- 172.21.15.40:0/506186690 --> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5580fd737bc0 con 0x5580fd926920 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.762+0000 7f14e5a00640 1 --2- 172.21.15.40:0/506186690 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x5580fd926920 0x5580fd995620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e5a00640 1 --2- 172.21.15.40:0/506186690 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x5580fd926920 0x5580fd995620 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44cc0ac44c7eb5d2 server_cookie=7f57fba9d5388802 in_seq=0 out_seq=0 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14c6a00640 10 client.7197 ms_handle_connect on v2:172.21.15.40:6833/2177098794 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14c6a00640 1 -- 172.21.15.40:0/506186690 <== mds.0 v2:172.21.15.40:6833/2177098794 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5580fd737bc0 con 0x5580fd926920 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14c6a00640 10 client.7197 handle_command_reply: tid=0 2024-03-24T00:35:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14eb7fb000 1 client.7197 shutdown 2024-03-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14eb7fb000 1 -- 172.21.15.40:0/506186690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd90bed0 msgr2=0x5580fd93a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/506186690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd90bed0 0x5580fd93a350 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f14d8015390 tx=0x7f14d8015850 comp rx=0 tx=0).stop 2024-03-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14eb7fb000 1 -- 172.21.15.40:0/506186690 shutdown_connections 2024-03-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/506186690 >> [v2:172.21.15.40:6833/2177098794,v1:172.21.15.40:6835/2177098794] conn(0x5580fd926920 0x5580fd995620 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/506186690 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5580fd90bed0 0x5580fd93a350 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/506186690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5580fd8dcff0 0x5580fd98ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14eb7fb000 1 --2- 172.21.15.40:0/506186690 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5580fd8d8fa0 0x5580fd98c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14c6a00640 0 client.7197 ms_handle_reset on v2:172.21.15.40:6833/2177098794 2024-03-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14eb7fb000 1 -- 172.21.15.40:0/506186690 >> 172.21.15.40:0/506186690 conn(0x5580fd8df750 msgr2=0x5580fd90a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14eb7fb000 1 -- 172.21.15.40:0/506186690 shutdown_connections 2024-03-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14eb7fb000 1 -- 172.21.15.40:0/506186690 wait complete. 2024-03-24T00:35:28.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14eb7fb000 20 client.7197 trim_cache size 0 max 16384 2024-03-24T00:35:28.772 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:35:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:35:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:35:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6954, 2024-03-24T00:35:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:35:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:35:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:35:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 13.467000699, 2024-03-24T00:35:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 72, 2024-03-24T00:35:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 70, 2024-03-24T00:35:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 70, 2024-03-24T00:35:28.772 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 18.322089983000001 2024-03-24T00:35:28.773 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:35:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e8400640 1 -- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f14c0067660 msgr2=0x7f14c0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e8400640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f14c0067660 0x7f14c0069b20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f14d40ad000 tx=0x7f14d40a6000 comp rx=0 tx=0).stop 2024-03-24T00:35:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e8400640 1 -- 172.21.15.40:0/812280858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14e006ee60 msgr2=0x7f14e01f6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:28.773 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e8400640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14e006ee60 0x7f14e01f6af0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f14cc00c480 tx=0x7f14cc00c940 comp rx=0 tx=0).stop 2024-03-24T00:35:28.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e8400640 1 -- 172.21.15.40:0/812280858 shutdown_connections 2024-03-24T00:35:28.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e8400640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f14c0067660 0x7f14c0069b20 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e8400640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f14e0156320 0x7f14e014d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e8400640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f14e006ee60 0x7f14e01f6af0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e8400640 1 --2- 172.21.15.40:0/812280858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f14e0066e80 0x7f14e014b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:28.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e8400640 1 -- 172.21.15.40:0/812280858 >> 172.21.15.40:0/812280858 conn(0x7f14e006b150 msgr2=0x7f14e00b5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:28.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e8400640 1 -- 172.21.15.40:0/812280858 shutdown_connections 2024-03-24T00:35:28.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:28.766+0000 7f14e8400640 1 -- 172.21.15.40:0/812280858 wait complete. 2024-03-24T00:35:28.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:28.788 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:35:28.788 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:35:28.788 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p tree 2024-03-24T00:35:28.788 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:35:28.788 DEBUG:teuthology.orchestra.run.smithi040:> setfattr -n ceph.dir.pin.distributed -v 1 tree 2024-03-24T00:35:28.789 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:35:28.789 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < 100; i++)); do 2024-03-24T00:35:28.789 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p "tree/$i" 2024-03-24T00:35:28.789 DEBUG:teuthology.orchestra.run.smithi040:> echo file > "tree/$i/file" 2024-03-24T00:35:28.789 DEBUG:teuthology.orchestra.run.smithi040:> done 2024-03-24T00:35:28.789 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:35:28.801 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree 2024-03-24T00:35:28.805 INFO:teuthology.orchestra.run.smithi040.stderr:+ setfattr -n ceph.dir.pin.distributed -v 1 tree 2024-03-24T00:35:28.809 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i = 0 )) 2024-03-24T00:35:28.809 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.809 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/0 2024-03-24T00:35:28.814 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.820 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.820 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.820 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/1 2024-03-24T00:35:28.832 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.835 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.835 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.835 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/2 2024-03-24T00:35:28.842 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.845 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.845 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.845 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/3 2024-03-24T00:35:28.851 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.854 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.854 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.854 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/4 2024-03-24T00:35:28.861 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.863 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.863 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.863 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/5 2024-03-24T00:35:28.870 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.873 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.873 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.873 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/6 2024-03-24T00:35:28.880 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.884 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.884 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.884 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/7 2024-03-24T00:35:28.890 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/8 2024-03-24T00:35:28.900 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.903 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/9 2024-03-24T00:35:28.909 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.912 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/10 2024-03-24T00:35:28.919 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.921 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.921 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.921 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/11 2024-03-24T00:35:28.928 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/12 2024-03-24T00:35:28.936 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.939 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.939 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.939 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/13 2024-03-24T00:35:28.946 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.949 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.949 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.949 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/14 2024-03-24T00:35:28.956 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.960 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/15 2024-03-24T00:35:28.967 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.969 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.970 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.970 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/16 2024-03-24T00:35:28.976 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.979 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.979 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.979 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/17 2024-03-24T00:35:28.987 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.990 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.990 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.990 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/18 2024-03-24T00:35:28.997 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:28.999 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:28.999 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:28.999 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/19 2024-03-24T00:35:29.006 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.009 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.009 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.009 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/20 2024-03-24T00:35:29.017 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.019 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.019 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.019 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/21 2024-03-24T00:35:29.026 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.029 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.029 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.030 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/22 2024-03-24T00:35:29.037 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.040 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.040 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.040 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/23 2024-03-24T00:35:29.046 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.049 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.050 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/24 2024-03-24T00:35:29.056 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.060 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/25 2024-03-24T00:35:29.066 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.070 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/26 2024-03-24T00:35:29.076 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.079 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/27 2024-03-24T00:35:29.086 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.089 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/28 2024-03-24T00:35:29.096 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.100 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/29 2024-03-24T00:35:29.107 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.110 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.111 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/30 2024-03-24T00:35:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.120 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.121 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/31 2024-03-24T00:35:29.127 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.130 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.131 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/32 2024-03-24T00:35:29.138 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.141 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/33 2024-03-24T00:35:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.150 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.150 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.150 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/34 2024-03-24T00:35:29.157 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.160 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.160 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.160 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/35 2024-03-24T00:35:29.167 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.170 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.170 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.170 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/36 2024-03-24T00:35:29.177 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.180 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.180 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.180 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/37 2024-03-24T00:35:29.187 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.190 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.190 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.190 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/38 2024-03-24T00:35:29.196 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.199 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.199 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.199 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/39 2024-03-24T00:35:29.205 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.208 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.208 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.208 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/40 2024-03-24T00:35:29.215 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.217 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.217 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.217 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/41 2024-03-24T00:35:29.224 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.227 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/42 2024-03-24T00:35:29.234 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.237 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/43 2024-03-24T00:35:29.242 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.245 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.245 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.245 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/44 2024-03-24T00:35:29.252 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.254 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.254 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.254 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/45 2024-03-24T00:35:29.260 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.263 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.263 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.263 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/46 2024-03-24T00:35:29.269 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.272 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.272 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.272 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/47 2024-03-24T00:35:29.278 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.281 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.281 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.281 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/48 2024-03-24T00:35:29.288 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.291 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.291 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.291 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/49 2024-03-24T00:35:29.298 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.301 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.301 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.301 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/50 2024-03-24T00:35:29.307 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.310 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.310 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.310 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/51 2024-03-24T00:35:29.317 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.320 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.320 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.320 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/52 2024-03-24T00:35:29.328 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.330 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.331 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.331 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/53 2024-03-24T00:35:29.338 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.340 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.341 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.341 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/54 2024-03-24T00:35:29.348 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.351 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.351 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.351 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/55 2024-03-24T00:35:29.359 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.362 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.362 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.362 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/56 2024-03-24T00:35:29.368 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/57 2024-03-24T00:35:29.379 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.382 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.382 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.382 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/58 2024-03-24T00:35:29.389 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.391 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.391 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.391 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/59 2024-03-24T00:35:29.398 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.400 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/60 2024-03-24T00:35:29.408 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.411 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.411 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.411 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/61 2024-03-24T00:35:29.418 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.420 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.420 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.420 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/62 2024-03-24T00:35:29.426 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.429 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.429 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.429 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/63 2024-03-24T00:35:29.436 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.438 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/64 2024-03-24T00:35:29.445 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.447 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.447 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.447 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/65 2024-03-24T00:35:29.454 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.457 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.458 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.458 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/66 2024-03-24T00:35:29.465 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.468 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.468 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.468 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/67 2024-03-24T00:35:29.475 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.478 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.478 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.478 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/68 2024-03-24T00:35:29.485 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.487 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.487 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.488 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/69 2024-03-24T00:35:29.495 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.498 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.498 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.498 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/70 2024-03-24T00:35:29.505 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.509 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.509 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.509 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/71 2024-03-24T00:35:29.516 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.519 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.519 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.519 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/72 2024-03-24T00:35:29.526 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.529 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.529 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.529 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/73 2024-03-24T00:35:29.535 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.538 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.538 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.538 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/74 2024-03-24T00:35:29.545 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.549 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.549 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.549 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/75 2024-03-24T00:35:29.556 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.559 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.559 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.559 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/76 2024-03-24T00:35:29.566 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.569 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.569 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.569 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/77 2024-03-24T00:35:29.576 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.578 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.578 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.579 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/78 2024-03-24T00:35:29.585 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.588 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.588 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.588 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/79 2024-03-24T00:35:29.595 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.598 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.598 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.598 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/80 2024-03-24T00:35:29.605 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.608 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.608 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.608 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/81 2024-03-24T00:35:29.616 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.618 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.618 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.619 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/82 2024-03-24T00:35:29.625 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.628 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.628 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.628 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/83 2024-03-24T00:35:29.634 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.636 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.636 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.636 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/84 2024-03-24T00:35:29.643 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.645 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.645 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.645 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/85 2024-03-24T00:35:29.651 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.654 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.654 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.654 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/86 2024-03-24T00:35:29.661 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.665 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/87 2024-03-24T00:35:29.672 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.674 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/88 2024-03-24T00:35:29.682 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.685 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/89 2024-03-24T00:35:29.692 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.695 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/90 2024-03-24T00:35:29.702 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.705 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/91 2024-03-24T00:35:29.712 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.715 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/92 2024-03-24T00:35:29.722 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.725 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.725 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.725 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/93 2024-03-24T00:35:29.731 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.734 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.735 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.735 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/94 2024-03-24T00:35:29.741 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.743 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.744 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/95 2024-03-24T00:35:29.751 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.753 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.753 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.754 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/96 2024-03-24T00:35:29.761 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.763 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.764 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/97 2024-03-24T00:35:29.771 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.774 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.775 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/98 2024-03-24T00:35:29.782 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.784 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.785 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.785 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/99 2024-03-24T00:35:29.793 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:35:29.796 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:35:29.796 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:35:29.798 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:35:29.957 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:35:29.957 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:35:29.958 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-24T00:35:30.119 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:35:30.120 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:35:30.120 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:35:30.280 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"474","version":"472","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"91","version":"91","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-24T00:35:30.281 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "474", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "472" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "91", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "91" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:35:30.281 INFO:tasks.cephfs.cephfs_test_case:len=0 [] 2024-03-24T00:35:35.282 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:35:35.441 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:35:35.441 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:35:35.442 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-24T00:35:35.632 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"481","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":3},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000221.110*","snapid_first":2,"projected_version":"438","version":"438","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000221.101*","snapid_first":2,"projected_version":"420","version":"420","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000221.011*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-24T00:35:35.633 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 3 }, "snapid_first": 2, "states": [], "version": "481" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000221.110*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "438", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "438" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000221.101*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "420", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "420" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000221.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:35:35.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:35:35.793 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"483","version":"481","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":3,"2":3}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"93","version":"93","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x10000000221.000*","snapid_first":2,"projected_version":"424","version":"424","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x10000000221.001*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x10000000221.010*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000221.100*","snapid_first":2,"projected_version":"426","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":2,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"waiter":0,"authpin":1},"nref":4}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000221.110*","snapid_first":2,"projected_version":"438","version":"438","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000221.101*","snapid_first":2,"projected_version":"420","version":"420","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000221.011*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000221.111*","snapid_first":2,"projected_version":"416","version":"414","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":1},"nref":4}}] 2024-03-24T00:35:35.794 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "1": 3, "2": 3 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "483", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "481" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "93", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "93" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x10000000221.000*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "424", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "424" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x10000000221.001*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "422", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x10000000221.010*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "422", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 2, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000221.100*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 1, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000221.110*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "438", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "438" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000221.101*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "420", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "420" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000221.011*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000221.111*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 1, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "416", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "414" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:35:35.795 INFO:tasks.cephfs.cephfs_test_case:len=5 [{'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.110*', 'snapid_first': 2, 'projected_version': '438', 'version': '438', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.101*', 'snapid_first': 2, 'projected_version': '420', 'version': '420', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.011*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.100*', 'snapid_first': 2, 'projected_version': '426', 'version': '422', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 2, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'waiter': 0, 'authpin': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.111*', 'snapid_first': 2, 'projected_version': '416', 'version': '414', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 1, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 1}, 'nref': 4}}] 2024-03-24T00:35:37.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:35:37.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:35:40.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:35:40.956 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"481","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":3},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x10000000221.010*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x10000000221.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x10000000221.000*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}}] 2024-03-24T00:35:40.956 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 3 }, "snapid_first": 2, "states": [], "version": "481" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x10000000221.010*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x10000000221.001*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "428", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x10000000221.000*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:35:40.957 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-24T00:35:41.164 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"481","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":3},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000221.110*","snapid_first":2,"projected_version":"442","version":"442","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000221.101*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000221.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}}] 2024-03-24T00:35:41.165 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 3 }, "snapid_first": 2, "states": [], "version": "481" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000221.110*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "442", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "442" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000221.101*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "422", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000221.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "430", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:35:41.165 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:35:41.326 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"483","version":"483","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":3,"2":3}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"93","version":"93","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x10000000221.000*","snapid_first":2,"projected_version":"424","version":"424","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x10000000221.001*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x10000000221.010*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000221.100*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"waiter":0,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000221.110*","snapid_first":2,"projected_version":"438","version":"438","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000221.101*","snapid_first":2,"projected_version":"420","version":"420","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000221.011*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000221.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-24T00:35:41.327 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 3, "2": 3 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "483", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "483" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "93", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "93" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x10000000221.000*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "424", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "424" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x10000000221.001*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "422", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x10000000221.010*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "422", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000221.100*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000221.110*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "438", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "438" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000221.101*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "420", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "420" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000221.011*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000221.111*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "416", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:35:41.327 INFO:tasks.cephfs.cephfs_test_case:len=8 [{'is_auth': True, 'auth_first': 2, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 2, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.010*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '2', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [2, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1, 'authpin': 0}, 'nref': 4}}, {'is_auth': True, 'auth_first': 2, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 2, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.001*', 'snapid_first': 2, 'projected_version': '428', 'version': '428', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '2', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [2, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1, 'authpin': 0}, 'nref': 4}}, {'is_auth': True, 'auth_first': 2, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 2, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.000*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '2', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [2, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1, 'authpin': 0}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.110*', 'snapid_first': 2, 'projected_version': '442', 'version': '442', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1, 'authpin': 0}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.101*', 'snapid_first': 2, 'projected_version': '422', 'version': '422', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1, 'authpin': 0}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.011*', 'snapid_first': 2, 'projected_version': '430', 'version': '430', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1, 'authpin': 0}, 'nref': 4}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.100*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'waiter': 0, 'authpin': 0}, 'nref': 3}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x10000000221.111*', 'snapid_first': 2, 'projected_version': '416', 'version': '416', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}] 2024-03-24T00:35:41.327 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:35:41.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:35:41.403 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:35:41.443 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:35:41.443 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:35:41.443 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:35:41.452 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:41.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3ae000640 1 Processor -- start 2024-03-24T00:35:41.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3ae000640 1 -- start start 2024-03-24T00:35:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3ae000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc3a80d03f0 0x7fc3a80b2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3ae000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a80b2ee0 con 0x7fc3a815f7e0 2024-03-24T00:35:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3ae000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a80af570 con 0x7fc3a80d0010 2024-03-24T00:35:41.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3ae000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a80af850 con 0x7fc3a80d03f0 2024-03-24T00:35:41.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3a6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc3a80d03f0 0x7fc3a80b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3a7400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc3a80d0010 0x7fc3a815f0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:53318/0 (socket says 172.21.15.40:53318) 2024-03-24T00:35:41.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3a7400640 1 -- 172.21.15.40:0/3239513622 learned_addr learned my addr 172.21.15.40:0/3239513622 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:41.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3a6a00640 1 -- 172.21.15.40:0/3239513622 >> v1:172.21.15.53:6789/0 conn(0x7fc3a80d0010 legacy=0x7fc3a815f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3a6000640 1 -- 172.21.15.40:0/3239513622 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219140089 0 0) 0x7fc3a80af570 con 0x7fc3a80d0010 2024-03-24T00:35:41.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3a6a00640 1 -- 172.21.15.40:0/3239513622 >> v1:172.21.15.40:6789/0 conn(0x7fc3a815f7e0 legacy=0x7fc3a8165d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.634+0000 7fc3a6a00640 1 -- 172.21.15.40:0/3239513622 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a80afb30 con 0x7fc3a80d03f0 2024-03-24T00:35:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3a6a00640 1 --2- 172.21.15.40:0/3239513622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3a80d03f0 0x7fc3a80b2900 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc3940097c0 tx=0x7fc39401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567fd0d268cf73ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3a6000640 1 -- 172.21.15.40:0/3239513622 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3940b0070 con 0x7fc3a80d03f0 2024-03-24T00:35:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3a6000640 1 -- 172.21.15.40:0/3239513622 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3940ab600 con 0x7fc3a80d03f0 2024-03-24T00:35:41.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3a6000640 1 -- 172.21.15.40:0/3239513622 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3940ab9d0 con 0x7fc3a80d03f0 2024-03-24T00:35:41.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3ae000640 1 -- 172.21.15.40:0/3239513622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3a80d03f0 msgr2=0x7fc3a80b2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3ae000640 1 --2- 172.21.15.40:0/3239513622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3a80d03f0 0x7fc3a80b2900 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fc3940097c0 tx=0x7fc39401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3ae000640 1 -- 172.21.15.40:0/3239513622 shutdown_connections 2024-03-24T00:35:41.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3ae000640 1 --2- 172.21.15.40:0/3239513622 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3a80d03f0 0x7fc3a80b2900 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3ae000640 1 -- 172.21.15.40:0/3239513622 >> 172.21.15.40:0/3239513622 conn(0x7fc3a80b38f0 msgr2=0x7fc3a80b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:41.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3ae000640 1 -- 172.21.15.40:0/3239513622 shutdown_connections 2024-03-24T00:35:41.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3ae000640 1 -- 172.21.15.40:0/3239513622 wait complete. 2024-03-24T00:35:41.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3ae000640 1 Processor -- start 2024-03-24T00:35:41.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.638+0000 7fc3ae000640 1 -- start start 2024-03-24T00:35:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3ae000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3a80d0010 0x7fc3a814d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3ae000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3a80d03f0 0x7fc3a806b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3ae000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3a815f7e0 0x7fc3a80b5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3ae000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a816a060 con 0x7fc3a815f7e0 2024-03-24T00:35:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3ae000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a8169ee0 con 0x7fc3a80d0010 2024-03-24T00:35:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3ae000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a816a1e0 con 0x7fc3a80d03f0 2024-03-24T00:35:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3a815f7e0 0x7fc3a80b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3a7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3a80d0010 0x7fc3a814d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3a6a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3a80d03f0 0x7fc3a806b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3a815f7e0 0x7fc3a80b5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56072/0 (socket says 172.21.15.40:56072) 2024-03-24T00:35:41.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3a7e00640 1 -- 172.21.15.40:0/84278456 learned_addr learned my addr 172.21.15.40:0/84278456 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3a7e00640 1 -- 172.21.15.40:0/84278456 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3a80d03f0 msgr2=0x7fc3a806b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3a7e00640 1 --2- 172.21.15.40:0/84278456 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3a80d03f0 0x7fc3a806b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3a7e00640 1 -- 172.21.15.40:0/84278456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3a80d0010 msgr2=0x7fc3a814d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3a7e00640 1 --2- 172.21.15.40:0/84278456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3a80d0010 0x7fc3a814d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3a7e00640 1 -- 172.21.15.40:0/84278456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a806c290 con 0x7fc3a815f7e0 2024-03-24T00:35:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3a7e00640 1 --2- 172.21.15.40:0/84278456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3a815f7e0 0x7fc3a80b5140 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc3a000c730 tx=0x7fc3a000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc39f400640 1 -- 172.21.15.40:0/84278456 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0003b80 con 0x7fc3a815f7e0 2024-03-24T00:35:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc39f400640 1 -- 172.21.15.40:0/84278456 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a0003d40 con 0x7fc3a815f7e0 2024-03-24T00:35:41.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3ae000640 1 -- 172.21.15.40:0/84278456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a806c570 con 0x7fc3a815f7e0 2024-03-24T00:35:41.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc39f400640 1 -- 172.21.15.40:0/84278456 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0005310 con 0x7fc3a815f7e0 2024-03-24T00:35:41.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3ae000640 1 -- 172.21.15.40:0/84278456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a806cb30 con 0x7fc3a815f7e0 2024-03-24T00:35:41.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.642+0000 7fc3ae000640 1 -- 172.21.15.40:0/84278456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a8068170 con 0x7fc3a815f7e0 2024-03-24T00:35:41.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.646+0000 7fc39f400640 1 -- 172.21.15.40:0/84278456 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc3a00054b0 con 0x7fc3a815f7e0 2024-03-24T00:35:41.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.646+0000 7fc39f400640 1 --2- 172.21.15.40:0/84278456 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc388067660 0x7fc388069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:41.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.646+0000 7fc39f400640 1 -- 172.21.15.40:0/84278456 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3a0085730 con 0x7fc3a815f7e0 2024-03-24T00:35:41.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.646+0000 7fc3a7400640 1 --2- 172.21.15.40:0/84278456 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc388067660 0x7fc388069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:41.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.646+0000 7fc3a7400640 1 --2- 172.21.15.40:0/84278456 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc388067660 0x7fc388069b20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc398000f70 tx=0x7fc398007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:41.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.650+0000 7fc39f400640 1 -- 172.21.15.40:0/84278456 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc3a0055cc0 con 0x7fc3a815f7e0 2024-03-24T00:35:41.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.842+0000 7fc3ae000640 1 -- 172.21.15.40:0/84278456 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3a80cec80 con 0x7fc3a815f7e0 2024-03-24T00:35:41.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.842+0000 7fc39f400640 1 -- 172.21.15.40:0/84278456 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16608 (secure 0 0 0) 0x7fc3a0059d60 con 0x7fc3a815f7e0 2024-03-24T00:35:41.845 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:41.846 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":70,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:35:16.298084+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:13.099776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:13.675426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.846+0000 7fc3ae000640 1 -- 172.21.15.40:0/84278456 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc388067660 msgr2=0x7fc388069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.846+0000 7fc3ae000640 1 --2- 172.21.15.40:0/84278456 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc388067660 0x7fc388069b20 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc398000f70 tx=0x7fc398007240 comp rx=0 tx=0).stop 2024-03-24T00:35:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.846+0000 7fc3ae000640 1 -- 172.21.15.40:0/84278456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3a815f7e0 msgr2=0x7fc3a80b5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.846+0000 7fc3ae000640 1 --2- 172.21.15.40:0/84278456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3a815f7e0 0x7fc3a80b5140 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fc3a000c730 tx=0x7fc3a000cb30 comp rx=0 tx=0).stop 2024-03-24T00:35:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.846+0000 7fc3ae000640 1 -- 172.21.15.40:0/84278456 shutdown_connections 2024-03-24T00:35:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.846+0000 7fc3ae000640 1 --2- 172.21.15.40:0/84278456 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc388067660 0x7fc388069b20 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.846+0000 7fc3ae000640 1 --2- 172.21.15.40:0/84278456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3a815f7e0 0x7fc3a80b5140 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.846+0000 7fc3ae000640 1 --2- 172.21.15.40:0/84278456 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3a80d03f0 0x7fc3a806b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.846+0000 7fc3ae000640 1 --2- 172.21.15.40:0/84278456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3a80d0010 0x7fc3a814d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.846+0000 7fc3ae000640 1 -- 172.21.15.40:0/84278456 >> 172.21.15.40:0/84278456 conn(0x7fc3a80b38f0 msgr2=0x7fc3a815bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.846+0000 7fc3ae000640 1 -- 172.21.15.40:0/84278456 shutdown_connections 2024-03-24T00:35:41.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:41.846+0000 7fc3ae000640 1 -- 172.21.15.40:0/84278456 wait complete. 2024-03-24T00:35:41.878 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:35:13.099776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:35:13.675426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:35:41.879 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:35:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.022+0000 7f66ac400640 1 Processor -- start 2024-03-24T00:35:42.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.022+0000 7f66ac400640 1 -- start start 2024-03-24T00:35:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.022+0000 7f66ac400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f66a40d02c0 0x7f66a40b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.022+0000 7f66ac400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f66a40b2120 0x7f66a40ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.022+0000 7f66ac400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66a40b0f10 con 0x7f66a415f5c0 2024-03-24T00:35:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.022+0000 7f66ac400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a415f9a0 con 0x7f66a40d02c0 2024-03-24T00:35:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.022+0000 7f66ac400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66a40cd130 con 0x7f66a40b2120 2024-03-24T00:35:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.022+0000 7f66a9000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f66a40d02c0 0x7f66a40b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.022+0000 7f66a9a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f66a415f5c0 0x7f66a40cfbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:53320/0 (socket says 172.21.15.40:53320) 2024-03-24T00:35:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.022+0000 7f66a9a00640 1 -- 172.21.15.40:0/529946842 learned_addr learned my addr 172.21.15.40:0/529946842 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:42.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.026+0000 7f66aa400640 1 --2- 172.21.15.40:0/529946842 >> v2:172.21.15.53:3300/0 conn(0x7f66a40b2120 0x7f66a40ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.026+0000 7f66a3e00640 1 -- 172.21.15.40:0/529946842 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204003063 0 0) 0x7f6698001120 con 0x7f66a415f5c0 2024-03-24T00:35:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.026+0000 7f66a9000640 1 -- 172.21.15.40:0/529946842 >> v2:172.21.15.53:3300/0 conn(0x7f66a40b2120 msgr2=0x7f66a40ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.026+0000 7f66a9000640 1 --2- 172.21.15.40:0/529946842 >> v2:172.21.15.53:3300/0 conn(0x7f66a40b2120 0x7f66a40ce1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.026+0000 7f66a9000640 1 -- 172.21.15.40:0/529946842 >> v1:172.21.15.53:6789/0 conn(0x7f66a415f5c0 legacy=0x7f66a40cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.026+0000 7f66a9000640 1 -- 172.21.15.40:0/529946842 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a40cd410 con 0x7f66a40d02c0 2024-03-24T00:35:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.026+0000 7f66a9000640 1 --2- 172.21.15.40:0/529946842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66a40d02c0 0x7f66a40b3060 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f66a406ecf0 tx=0x7f669001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d1f7339831027d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:42.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.026+0000 7f66a3e00640 1 -- 172.21.15.40:0/529946842 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66900ab650 con 0x7f66a40d02c0 2024-03-24T00:35:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.026+0000 7f66a3e00640 1 -- 172.21.15.40:0/529946842 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66900b0070 con 0x7f66a40d02c0 2024-03-24T00:35:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.026+0000 7f66a3e00640 1 -- 172.21.15.40:0/529946842 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66900aba10 con 0x7f66a40d02c0 2024-03-24T00:35:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.030+0000 7f66ac400640 1 -- 172.21.15.40:0/529946842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66a40d02c0 msgr2=0x7f66a40b3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.030+0000 7f66ac400640 1 --2- 172.21.15.40:0/529946842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66a40d02c0 0x7f66a40b3060 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f66a406ecf0 tx=0x7f669001c450 comp rx=0 tx=0).stop 2024-03-24T00:35:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.030+0000 7f66ac400640 1 -- 172.21.15.40:0/529946842 shutdown_connections 2024-03-24T00:35:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.030+0000 7f66ac400640 1 --2- 172.21.15.40:0/529946842 >> v2:172.21.15.53:3300/0 conn(0x7f66a40b2120 0x7f66a40ce1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.030+0000 7f66ac400640 1 --2- 172.21.15.40:0/529946842 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66a40d02c0 0x7f66a40b3060 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.030+0000 7f66ac400640 1 -- 172.21.15.40:0/529946842 >> 172.21.15.40:0/529946842 conn(0x7f66a40b3810 msgr2=0x7f66a40b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.030+0000 7f66ac400640 1 -- 172.21.15.40:0/529946842 shutdown_connections 2024-03-24T00:35:42.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.030+0000 7f66ac400640 1 -- 172.21.15.40:0/529946842 wait complete. 2024-03-24T00:35:42.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.030+0000 7f66ac400640 1 Processor -- start 2024-03-24T00:35:42.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.030+0000 7f66ac400640 1 -- start start 2024-03-24T00:35:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66ac400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a40b2120 0x7f66a41f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66ac400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66a40d02c0 0x7f66a4157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66ac400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66a415f5c0 0x7f66a41588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66ac400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40cf0a0 con 0x7f66a40d02c0 2024-03-24T00:35:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66ac400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40cef20 con 0x7f66a40b2120 2024-03-24T00:35:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66ac400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66a40cf220 con 0x7f66a415f5c0 2024-03-24T00:35:42.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a40b2120 0x7f66a41f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a40b2120 0x7f66a41f5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:36204/0 (socket says 172.21.15.40:36204) 2024-03-24T00:35:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a9a00640 1 -- 172.21.15.40:0/836104772 learned_addr learned my addr 172.21.15.40:0/836104772 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a9000640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66a40d02c0 0x7f66a4157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66aa400640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66a415f5c0 0x7f66a41588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a9a00640 1 -- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66a415f5c0 msgr2=0x7f66a41588b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a9a00640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66a415f5c0 0x7f66a41588b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a9a00640 1 -- 172.21.15.40:0/836104772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66a40d02c0 msgr2=0x7f66a4157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a9a00640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66a40d02c0 0x7f66a4157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a9a00640 1 -- 172.21.15.40:0/836104772 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a406be30 con 0x7f66a40b2120 2024-03-24T00:35:42.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a9a00640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a40b2120 0x7f66a41f5f60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f669800c410 tx=0x7f669800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a1600640 1 -- 172.21.15.40:0/836104772 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66980074e0 con 0x7f66a40b2120 2024-03-24T00:35:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a1600640 1 -- 172.21.15.40:0/836104772 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66980047f0 con 0x7f66a40b2120 2024-03-24T00:35:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66ac400640 1 -- 172.21.15.40:0/836104772 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66a406c110 con 0x7f66a40b2120 2024-03-24T00:35:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66a1600640 1 -- 172.21.15.40:0/836104772 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6698007680 con 0x7f66a40b2120 2024-03-24T00:35:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66ac400640 1 -- 172.21.15.40:0/836104772 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66a406c6d0 con 0x7f66a40b2120 2024-03-24T00:35:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.034+0000 7f66ac400640 1 -- 172.21.15.40:0/836104772 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66a4068170 con 0x7f66a40b2120 2024-03-24T00:35:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.038+0000 7f66a1600640 1 -- 172.21.15.40:0/836104772 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6698011070 con 0x7f66a40b2120 2024-03-24T00:35:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.038+0000 7f66a1600640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6684067660 0x7f6684069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.038+0000 7f66a1600640 1 -- 172.21.15.40:0/836104772 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6698085450 con 0x7f66a40b2120 2024-03-24T00:35:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.038+0000 7f66a9000640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6684067660 0x7f6684069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.038+0000 7f66a9000640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6684067660 0x7f6684069b20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f66900af000 tx=0x7f66900a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:42.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.042+0000 7f66a1600640 1 -- 172.21.15.40:0/836104772 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f66a40b0f10 con 0x7f66a40b2120 2024-03-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.206+0000 7f66ac400640 1 -- 172.21.15.40:0/836104772 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f66a415d890 con 0x7f66a40b2120 2024-03-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.206+0000 7f66a1600640 1 -- 172.21.15.40:0/836104772 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f66980559e0 con 0x7f66a40b2120 2024-03-24T00:35:42.211 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.210+0000 7f66ac400640 1 -- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6684067660 msgr2=0x7f6684069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.210+0000 7f66ac400640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6684067660 0x7f6684069b20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f66900af000 tx=0x7f66900a7000 comp rx=0 tx=0).stop 2024-03-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.210+0000 7f66ac400640 1 -- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a40b2120 msgr2=0x7f66a41f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.210+0000 7f66ac400640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a40b2120 0x7f66a41f5f60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f669800c410 tx=0x7f669800c8d0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.210+0000 7f66ac400640 1 -- 172.21.15.40:0/836104772 shutdown_connections 2024-03-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.210+0000 7f66ac400640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6684067660 0x7f6684069b20 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.210+0000 7f66ac400640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66a415f5c0 0x7f66a41588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.210+0000 7f66ac400640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66a40d02c0 0x7f66a4157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.210+0000 7f66ac400640 1 --2- 172.21.15.40:0/836104772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66a40b2120 0x7f66a41f5f60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.210+0000 7f66ac400640 1 -- 172.21.15.40:0/836104772 >> 172.21.15.40:0/836104772 conn(0x7f66a40b3810 msgr2=0x7f66a40d0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.210+0000 7f66ac400640 1 -- 172.21.15.40:0/836104772 shutdown_connections 2024-03-24T00:35:42.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.210+0000 7f66ac400640 1 -- 172.21.15.40:0/836104772 wait complete. 2024-03-24T00:35:42.234 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:35:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.374+0000 7f6ee2c00640 1 Processor -- start 2024-03-24T00:35:42.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.374+0000 7f6ee2c00640 1 -- start start 2024-03-24T00:35:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.374+0000 7f6ee2c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6edc0b4330 0x7f6edc0ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.374+0000 7f6ee2c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6edc0cea00 0x7f6edc0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.374+0000 7f6ee2c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6edc0cf090 con 0x7f6edc0b2360 2024-03-24T00:35:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.374+0000 7f6ee2c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6edc0d0f60 con 0x7f6edc0cea00 2024-03-24T00:35:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.374+0000 7f6ee2c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0cee30 con 0x7f6edc0b4330 2024-03-24T00:35:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6ed3400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6edc0b4330 0x7f6edc0ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6edbe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6edc0b2360 0x7f6edc0b2740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45714/0 (socket says 172.21.15.40:45714) 2024-03-24T00:35:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6edbe00640 1 -- 172.21.15.40:0/4154092228 learned_addr learned my addr 172.21.15.40:0/4154092228 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:42.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6ee0c00640 1 --2- 172.21.15.40:0/4154092228 >> v2:172.21.15.53:3300/0 conn(0x7f6edc0cea00 0x7f6edc0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6ee0c00640 1 -- 172.21.15.40:0/4154092228 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6edc0b4330 msgr2=0x7f6edc0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6ee0c00640 1 --2- 172.21.15.40:0/4154092228 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6edc0b4330 0x7f6edc0ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6ee0c00640 1 -- 172.21.15.40:0/4154092228 >> v1:172.21.15.40:6789/0 conn(0x7f6edc0b2360 legacy=0x7f6edc0b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6ee0c00640 1 -- 172.21.15.40:0/4154092228 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc0cf330 con 0x7f6edc0cea00 2024-03-24T00:35:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6edb400640 1 -- 172.21.15.40:0/4154092228 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005811699 0 0) 0x7f6edc0cf090 con 0x7f6edc0b2360 2024-03-24T00:35:42.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6ed3400640 1 --2- 172.21.15.40:0/4154092228 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6edc0b4330 0x7f6edc0ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:42.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6ee0c00640 1 --2- 172.21.15.40:0/4154092228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6edc0cea00 0x7f6edc0cc440 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40097c0 tx=0x7f6ed401c350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=604b594fe47d2a7f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6edb400640 1 -- 172.21.15.40:0/4154092228 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed40b0070 con 0x7f6edc0cea00 2024-03-24T00:35:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6edb400640 1 -- 172.21.15.40:0/4154092228 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed4004230 con 0x7f6edc0cea00 2024-03-24T00:35:42.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.378+0000 7f6edb400640 1 -- 172.21.15.40:0/4154092228 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4003620 con 0x7f6edc0cea00 2024-03-24T00:35:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 -- 172.21.15.40:0/4154092228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6edc0cea00 msgr2=0x7f6edc0cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 --2- 172.21.15.40:0/4154092228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6edc0cea00 0x7f6edc0cc440 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40097c0 tx=0x7f6ed401c350 comp rx=0 tx=0).stop 2024-03-24T00:35:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 -- 172.21.15.40:0/4154092228 shutdown_connections 2024-03-24T00:35:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 --2- 172.21.15.40:0/4154092228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6edc0cea00 0x7f6edc0cc440 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 --2- 172.21.15.40:0/4154092228 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6edc0b4330 0x7f6edc0ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 -- 172.21.15.40:0/4154092228 >> 172.21.15.40:0/4154092228 conn(0x7f6edc0b3810 msgr2=0x7f6edc0b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 -- 172.21.15.40:0/4154092228 shutdown_connections 2024-03-24T00:35:42.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 -- 172.21.15.40:0/4154092228 wait complete. 2024-03-24T00:35:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 Processor -- start 2024-03-24T00:35:42.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 -- start start 2024-03-24T00:35:42.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc0b2360 0x7f6edc0c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6edc0b4330 0x7f6edc0c7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6edc0cea00 0x7f6edc159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0cd190 con 0x7f6edc0b2360 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0d19e0 con 0x7f6edc0b4330 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee2c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6edc0cd310 con 0x7f6edc0cea00 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6edbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc0b2360 0x7f6edc0c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ed3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6edc0b4330 0x7f6edc0c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6edbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc0b2360 0x7f6edc0c56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56094/0 (socket says 172.21.15.40:56094) 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6edbe00640 1 -- 172.21.15.40:0/593002429 learned_addr learned my addr 172.21.15.40:0/593002429 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ee0c00640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6edc0cea00 0x7f6edc159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ed3400640 1 -- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6edc0cea00 msgr2=0x7f6edc159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ed3400640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6edc0cea00 0x7f6edc159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ed3400640 1 -- 172.21.15.40:0/593002429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc0b2360 msgr2=0x7f6edc0c56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ed3400640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc0b2360 0x7f6edc0c56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6ed3400640 1 -- 172.21.15.40:0/593002429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6edc1f9f20 con 0x7f6edc0b4330 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.382+0000 7f6edbe00640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc0b2360 0x7f6edc0c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.386+0000 7f6ed3400640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6edc0b4330 0x7f6edc0c7790 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400ea10 tx=0x7f6ec400eed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:42.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.386+0000 7f6ed8c00640 1 -- 172.21.15.40:0/593002429 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4017070 con 0x7f6edc0b4330 2024-03-24T00:35:42.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.386+0000 7f6ed8c00640 1 -- 172.21.15.40:0/593002429 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec400ca00 con 0x7f6edc0b4330 2024-03-24T00:35:42.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.386+0000 7f6ee2c00640 1 -- 172.21.15.40:0/593002429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6edc1fa200 con 0x7f6edc0b4330 2024-03-24T00:35:42.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.386+0000 7f6ed8c00640 1 -- 172.21.15.40:0/593002429 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec400cdf0 con 0x7f6edc0b4330 2024-03-24T00:35:42.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.386+0000 7f6ee2c00640 1 -- 172.21.15.40:0/593002429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6edc1fa7c0 con 0x7f6edc0b4330 2024-03-24T00:35:42.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.386+0000 7f6ee2c00640 1 -- 172.21.15.40:0/593002429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6edc14b900 con 0x7f6edc0b4330 2024-03-24T00:35:42.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.386+0000 7f6ed8c00640 1 -- 172.21.15.40:0/593002429 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6ec401a070 con 0x7f6edc0b4330 2024-03-24T00:35:42.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.386+0000 7f6ed8c00640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6eb8067690 0x7f6eb8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.386+0000 7f6ed8c00640 1 -- 172.21.15.40:0/593002429 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ec4085ad0 con 0x7f6edc0b4330 2024-03-24T00:35:42.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.386+0000 7f6edbe00640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6eb8067690 0x7f6eb8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.390+0000 7f6edbe00640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6eb8067690 0x7f6eb8069b50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc0012f0 tx=0x7f6ecc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:42.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.390+0000 7f6ed8c00640 1 -- 172.21.15.40:0/593002429 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6edc14b900 con 0x7f6edc0b4330 2024-03-24T00:35:42.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.554+0000 7f6ee2c00640 1 -- 172.21.15.40:0/593002429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6edc068170 con 0x7f6edc0b4330 2024-03-24T00:35:42.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.554+0000 7f6ed8c00640 1 -- 172.21.15.40:0/593002429 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6ec4056060 con 0x7f6edc0b4330 2024-03-24T00:35:42.558 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:35:42.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.558+0000 7f6ee2c00640 1 -- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6eb8067690 msgr2=0x7f6eb8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.558+0000 7f6ee2c00640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6eb8067690 0x7f6eb8069b50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc0012f0 tx=0x7f6ecc009040 comp rx=0 tx=0).stop 2024-03-24T00:35:42.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.558+0000 7f6ee2c00640 1 -- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6edc0b4330 msgr2=0x7f6edc0c7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.558+0000 7f6ee2c00640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6edc0b4330 0x7f6edc0c7790 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400ea10 tx=0x7f6ec400eed0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.558+0000 7f6ee2c00640 1 -- 172.21.15.40:0/593002429 shutdown_connections 2024-03-24T00:35:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.558+0000 7f6ee2c00640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6eb8067690 0x7f6eb8069b50 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.558+0000 7f6ee2c00640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6edc0cea00 0x7f6edc159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.558+0000 7f6ee2c00640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6edc0b4330 0x7f6edc0c7790 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.558+0000 7f6ee2c00640 1 --2- 172.21.15.40:0/593002429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6edc0b2360 0x7f6edc0c56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.558+0000 7f6ee2c00640 1 -- 172.21.15.40:0/593002429 >> 172.21.15.40:0/593002429 conn(0x7f6edc0b3810 msgr2=0x7f6edc06bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.558+0000 7f6ee2c00640 1 -- 172.21.15.40:0/593002429 shutdown_connections 2024-03-24T00:35:42.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.558+0000 7f6ee2c00640 1 -- 172.21.15.40:0/593002429 wait complete. 2024-03-24T00:35:42.580 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:35:42.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.722+0000 7f346a200640 1 Processor -- start 2024-03-24T00:35:42.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.722+0000 7f346a200640 1 -- start start 2024-03-24T00:35:42.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f346a200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f346415f5b0 0x7f34640cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f346a200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f34640d0750 0x7f3464165b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f346a200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34640b33e0 con 0x7f34640cc4c0 2024-03-24T00:35:42.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f346a200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34640afa90 con 0x7f34640d0750 2024-03-24T00:35:42.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f346a200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34640b22f0 con 0x7f346415f5b0 2024-03-24T00:35:42.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f3468e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f346415f5b0 0x7f34640cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f3469800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f34640d0750 0x7f3464165b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f3463e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f34640cc4c0 0x7f34640d0040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45724/0 (socket says 172.21.15.40:45724) 2024-03-24T00:35:42.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f3463e00640 1 -- 172.21.15.40:0/1541444776 learned_addr learned my addr 172.21.15.40:0/1541444776 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:42.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f3468e00640 1 -- 172.21.15.40:0/1541444776 >> v2:172.21.15.53:3300/0 conn(0x7f34640d0750 msgr2=0x7f3464165b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:42.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f3468e00640 1 --2- 172.21.15.40:0/1541444776 >> v2:172.21.15.53:3300/0 conn(0x7f34640d0750 0x7f3464165b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f3468e00640 1 -- 172.21.15.40:0/1541444776 >> v1:172.21.15.40:6789/0 conn(0x7f34640cc4c0 legacy=0x7f34640d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f3468e00640 1 -- 172.21.15.40:0/1541444776 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34640af440 con 0x7f346415f5b0 2024-03-24T00:35:42.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.726+0000 7f3469800640 1 --2- 172.21.15.40:0/1541444776 >> v2:172.21.15.53:3300/0 conn(0x7f34640d0750 0x7f3464165b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:42.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.730+0000 7f3468e00640 1 --2- 172.21.15.40:0/1541444776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f346415f5b0 0x7f34640cfb00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f34580097c0 tx=0x7f345801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d51084ad0ad232e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:42.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.730+0000 7f3463400640 1 -- 172.21.15.40:0/1541444776 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34580b0070 con 0x7f346415f5b0 2024-03-24T00:35:42.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.730+0000 7f3463400640 1 -- 172.21.15.40:0/1541444776 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34580ab600 con 0x7f346415f5b0 2024-03-24T00:35:42.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.730+0000 7f3463400640 1 -- 172.21.15.40:0/1541444776 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34580ab9d0 con 0x7f346415f5b0 2024-03-24T00:35:42.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.730+0000 7f346a200640 1 -- 172.21.15.40:0/1541444776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f346415f5b0 msgr2=0x7f34640cfb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.730+0000 7f346a200640 1 --2- 172.21.15.40:0/1541444776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f346415f5b0 0x7f34640cfb00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f34580097c0 tx=0x7f345801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.730+0000 7f346a200640 1 -- 172.21.15.40:0/1541444776 shutdown_connections 2024-03-24T00:35:42.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.730+0000 7f346a200640 1 --2- 172.21.15.40:0/1541444776 >> v2:172.21.15.53:3300/0 conn(0x7f34640d0750 0x7f3464165b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.730+0000 7f346a200640 1 --2- 172.21.15.40:0/1541444776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f346415f5b0 0x7f34640cfb00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.730+0000 7f346a200640 1 -- 172.21.15.40:0/1541444776 >> 172.21.15.40:0/1541444776 conn(0x7f34640b3620 msgr2=0x7f34640b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.730+0000 7f346a200640 1 -- 172.21.15.40:0/1541444776 shutdown_connections 2024-03-24T00:35:42.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.730+0000 7f346a200640 1 -- 172.21.15.40:0/1541444776 wait complete. 2024-03-24T00:35:42.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f346a200640 1 Processor -- start 2024-03-24T00:35:42.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f346a200640 1 -- start start 2024-03-24T00:35:42.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f346a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34640cc4c0 0x7f3464158f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f346a200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34640d0750 0x7f3464163eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f346a200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f346415f5b0 0x7f34641643f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f346a200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3464168c40 con 0x7f34640cc4c0 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f346a200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3464168ac0 con 0x7f346415f5b0 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f346a200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3464168dc0 con 0x7f34640d0750 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3468e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34640cc4c0 0x7f3464158f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3468e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34640cc4c0 0x7f3464158f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56102/0 (socket says 172.21.15.40:56102) 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3468e00640 1 -- 172.21.15.40:0/1862739203 learned_addr learned my addr 172.21.15.40:0/1862739203 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3463e00640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34640d0750 0x7f3464163eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3469800640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f346415f5b0 0x7f34641643f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3468e00640 1 -- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34640d0750 msgr2=0x7f3464163eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3468e00640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34640d0750 0x7f3464163eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3468e00640 1 -- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f346415f5b0 msgr2=0x7f34641643f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3468e00640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f346415f5b0 0x7f34641643f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3468e00640 1 -- 172.21.15.40:0/1862739203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34640c88c0 con 0x7f34640cc4c0 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3469800640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f346415f5b0 0x7f34641643f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:42.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3468e00640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34640cc4c0 0x7f3464158f10 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f345801a0d0 tx=0x7f3458019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:42.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3460c00640 1 -- 172.21.15.40:0/1862739203 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34580ba070 con 0x7f34640cc4c0 2024-03-24T00:35:42.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3460c00640 1 -- 172.21.15.40:0/1862739203 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34580b5460 con 0x7f34640cc4c0 2024-03-24T00:35:42.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f346a200640 1 -- 172.21.15.40:0/1862739203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346406bed0 con 0x7f34640cc4c0 2024-03-24T00:35:42.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f3460c00640 1 -- 172.21.15.40:0/1862739203 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34580b0040 con 0x7f34640cc4c0 2024-03-24T00:35:42.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.734+0000 7f346a200640 1 -- 172.21.15.40:0/1862739203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346406c490 con 0x7f34640cc4c0 2024-03-24T00:35:42.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.738+0000 7f346a200640 1 -- 172.21.15.40:0/1862739203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346404f410 con 0x7f34640cc4c0 2024-03-24T00:35:42.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.738+0000 7f3460c00640 1 -- 172.21.15.40:0/1862739203 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f34580bd070 con 0x7f34640cc4c0 2024-03-24T00:35:42.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.738+0000 7f3460c00640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3444067660 0x7f3444069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:42.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.738+0000 7f3460c00640 1 -- 172.21.15.40:0/1862739203 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f345811fc70 con 0x7f34640cc4c0 2024-03-24T00:35:42.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.738+0000 7f3463e00640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3444067660 0x7f3444069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:42.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.738+0000 7f3463e00640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3444067660 0x7f3444069b20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3464165950 tx=0x7f3450007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:42.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.742+0000 7f3460c00640 1 -- 172.21.15.40:0/1862739203 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3458016090 con 0x7f34640cc4c0 2024-03-24T00:35:42.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 -- 172.21.15.40:0/1862739203 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f346415cc10 con 0x7f34640cc4c0 2024-03-24T00:35:42.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f3460c00640 1 -- 172.21.15.40:0/1862739203 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7f34580f03b0 con 0x7f34640cc4c0 2024-03-24T00:35:42.909 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:35:42.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 -- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3444067660 msgr2=0x7f3444069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3444067660 0x7f3444069b20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3464165950 tx=0x7f3450007e90 comp rx=0 tx=0).stop 2024-03-24T00:35:42.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 -- 172.21.15.40:0/1862739203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34640cc4c0 msgr2=0x7f3464158f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:42.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34640cc4c0 0x7f3464158f10 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f345801a0d0 tx=0x7f3458019240 comp rx=0 tx=0).stop 2024-03-24T00:35:42.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 -- 172.21.15.40:0/1862739203 shutdown_connections 2024-03-24T00:35:42.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3444067660 0x7f3444069b20 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f346415f5b0 0x7f34641643f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34640d0750 0x7f3464163eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 --2- 172.21.15.40:0/1862739203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f34640cc4c0 0x7f3464158f10 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:42.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 -- 172.21.15.40:0/1862739203 >> 172.21.15.40:0/1862739203 conn(0x7f34640b3620 msgr2=0x7f346406fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:42.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 -- 172.21.15.40:0/1862739203 shutdown_connections 2024-03-24T00:35:42.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:42.906+0000 7f346a200640 1 -- 172.21.15.40:0/1862739203 wait complete. 2024-03-24T00:35:42.935 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:35:43.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc03000640 1 Processor -- start 2024-03-24T00:35:43.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc03000640 1 -- start start 2024-03-24T00:35:43.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc03000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbbfc0ceba0 0x7fbbfc0d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:43.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc03000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbfc15c080 0x7fbbfc0d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:43.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc03000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbfc0d0780 con 0x7fbbfc15bca0 2024-03-24T00:35:43.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc03000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbfc0b4a90 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc03000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbfc0b4d70 con 0x7fbbfc15c080 2024-03-24T00:35:43.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc01200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbbfc0ceba0 0x7fbbfc0d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:43.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc01c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbbfc15bca0 0x7fbbfc0ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45732/0 (socket says 172.21.15.40:45732) 2024-03-24T00:35:43.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc01c00640 1 -- 172.21.15.40:0/2724182459 learned_addr learned my addr 172.21.15.40:0/2724182459 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:43.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc02600640 1 --2- 172.21.15.40:0/2724182459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbfc15c080 0x7fbbfc0d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:43.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc01200640 1 -- 172.21.15.40:0/2724182459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbfc15c080 msgr2=0x7fbbfc0d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:43.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc01200640 1 --2- 172.21.15.40:0/2724182459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbfc15c080 0x7fbbfc0d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc01200640 1 -- 172.21.15.40:0/2724182459 >> v1:172.21.15.40:6789/0 conn(0x7fbbfc15bca0 legacy=0x7fbbfc0ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:43.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc01200640 1 -- 172.21.15.40:0/2724182459 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbfc0b5050 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.074+0000 7fbc02600640 1 --2- 172.21.15.40:0/2724182459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbfc15c080 0x7fbbfc0d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:43.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.078+0000 7fbc01200640 1 --2- 172.21.15.40:0/2724182459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbfc0ceba0 0x7fbbfc0d2590 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80097c0 tx=0x7fbbe801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42fe85141bacff19 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:43.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.078+0000 7fbbfbe00640 1 -- 172.21.15.40:0/2724182459 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe80b0070 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.078+0000 7fbbfbe00640 1 -- 172.21.15.40:0/2724182459 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe80ab600 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.078+0000 7fbbfbe00640 1 -- 172.21.15.40:0/2724182459 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe80ab9d0 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.078+0000 7fbc03000640 1 -- 172.21.15.40:0/2724182459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbfc0ceba0 msgr2=0x7fbbfc0d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.078+0000 7fbc03000640 1 --2- 172.21.15.40:0/2724182459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbfc0ceba0 0x7fbbfc0d2590 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80097c0 tx=0x7fbbe801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.078+0000 7fbc03000640 1 -- 172.21.15.40:0/2724182459 shutdown_connections 2024-03-24T00:35:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.078+0000 7fbc03000640 1 --2- 172.21.15.40:0/2724182459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbfc15c080 0x7fbbfc0d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.078+0000 7fbc03000640 1 --2- 172.21.15.40:0/2724182459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbfc0ceba0 0x7fbbfc0d2590 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.078+0000 7fbc03000640 1 -- 172.21.15.40:0/2724182459 >> 172.21.15.40:0/2724182459 conn(0x7fbbfc0d1aa0 msgr2=0x7fbbfc0d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.078+0000 7fbc03000640 1 -- 172.21.15.40:0/2724182459 shutdown_connections 2024-03-24T00:35:43.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.078+0000 7fbc03000640 1 -- 172.21.15.40:0/2724182459 wait complete. 2024-03-24T00:35:43.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc03000640 1 Processor -- start 2024-03-24T00:35:43.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc03000640 1 -- start start 2024-03-24T00:35:43.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc03000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbfc0ceba0 0x7fbbfc156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc03000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbfc15bca0 0x7fbbfc15ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc03000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbfc15c080 0x7fbbfc15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc03000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbfc0b6150 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc03000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbfc0b5fd0 con 0x7fbbfc15c080 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc03000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbfc0b62d0 con 0x7fbbfc15bca0 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc01c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbfc0ceba0 0x7fbbfc156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc01c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbfc0ceba0 0x7fbbfc156d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56120/0 (socket says 172.21.15.40:56120) 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc01c00640 1 -- 172.21.15.40:0/3210301429 learned_addr learned my addr 172.21.15.40:0/3210301429 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc01200640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbfc15bca0 0x7fbbfc15ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.082+0000 7fbc02600640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbfc15c080 0x7fbbfc15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbc01c00640 1 -- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbfc15bca0 msgr2=0x7fbbfc15ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbc01c00640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbfc15bca0 0x7fbbfc15ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbc01c00640 1 -- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbfc15c080 msgr2=0x7fbbfc15c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbc01c00640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbfc15c080 0x7fbbfc15c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbc01c00640 1 -- 172.21.15.40:0/3210301429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbfc06cbf0 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbc01c00640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbfc0ceba0 0x7fbbfc156d80 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000c450 tx=0x7fbbf000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:43.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbbf9600640 1 -- 172.21.15.40:0/3210301429 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0017070 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbbf9600640 1 -- 172.21.15.40:0/3210301429 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf0005cf0 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbbf9600640 1 -- 172.21.15.40:0/3210301429 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00072f0 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbc03000640 1 -- 172.21.15.40:0/3210301429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbfc06ced0 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbc03000640 1 -- 172.21.15.40:0/3210301429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbfc15a6f0 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbc03000640 1 -- 172.21.15.40:0/3210301429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbfc09fb10 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.086+0000 7fbbf9600640 1 -- 172.21.15.40:0/3210301429 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbbf001a070 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.090+0000 7fbbf9600640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbbdc067660 0x7fbbdc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:43.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.090+0000 7fbbf9600640 1 -- 172.21.15.40:0/3210301429 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbf0084b80 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.090+0000 7fbc01200640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbbdc067660 0x7fbbdc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:43.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.090+0000 7fbc01200640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbbdc067660 0x7fbbdc069b20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80ad000 tx=0x7fbbe8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:43.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.090+0000 7fbbf9600640 1 -- 172.21.15.40:0/3210301429 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbbf00551a0 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 -- 172.21.15.40:0/3210301429 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbbfc0d1880 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbbf9600640 1 -- 172.21.15.40:0/3210301429 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v70) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbbf0059240 con 0x7fbbfc0ceba0 2024-03-24T00:35:43.273 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:35:43.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 -- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbbdc067660 msgr2=0x7fbbdc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:43.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbbdc067660 0x7fbbdc069b20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80ad000 tx=0x7fbbe8008040 comp rx=0 tx=0).stop 2024-03-24T00:35:43.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 -- 172.21.15.40:0/3210301429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbfc0ceba0 msgr2=0x7fbbfc156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:43.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbfc0ceba0 0x7fbbfc156d80 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000c450 tx=0x7fbbf000c850 comp rx=0 tx=0).stop 2024-03-24T00:35:43.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 -- 172.21.15.40:0/3210301429 shutdown_connections 2024-03-24T00:35:43.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbbdc067660 0x7fbbdc069b20 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbfc15c080 0x7fbbfc15c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbfc15bca0 0x7fbbfc15ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 --2- 172.21.15.40:0/3210301429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbfc0ceba0 0x7fbbfc156d80 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 -- 172.21.15.40:0/3210301429 >> 172.21.15.40:0/3210301429 conn(0x7fbbfc0d1aa0 msgr2=0x7fbbfc158680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:43.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 -- 172.21.15.40:0/3210301429 shutdown_connections 2024-03-24T00:35:43.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.270+0000 7fbc03000640 1 -- 172.21.15.40:0/3210301429 wait complete. 2024-03-24T00:35:43.293 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-03-24T00:35:43.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:43.294 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T00:35:43.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[44485]: fuse finished with error 0 and tester_r 0 2024-03-24T00:35:43.316 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:35:43.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:43.316 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:35:43.458 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:35:43.459 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:35:43.468 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:35:43.468 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:35:43.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:43.469 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T00:35:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:35:43.526 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T00:35:43.526 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:35:43.526 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:35:43.535 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:35:43.535 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:35:43.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:43.535 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T00:35:43.586 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T00:35:43.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:35:43.587 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:35:43.588 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:35:43.597 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:35:43.597 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:35:43.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:35:43.597 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T00:35:43.647 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T00:35:43.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:35:43.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:43.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e93200640 1 Processor -- start 2024-03-24T00:35:43.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e93200640 1 -- start start 2024-03-24T00:35:43.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e93200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7e940ceba0 0x7f7e940d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:43.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e93200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e9415c080 0x7f7e940d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:43.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e93200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e940d0780 con 0x7f7e9415bca0 2024-03-24T00:35:43.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e93200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e940b4a90 con 0x7f7e940ceba0 2024-03-24T00:35:43.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e93200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940b4d70 con 0x7f7e9415c080 2024-03-24T00:35:43.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e89400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7e940ceba0 0x7f7e940d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:43.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e89400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7e940ceba0 0x7f7e940d2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56126/0 (socket says 172.21.15.40:56126) 2024-03-24T00:35:43.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e89400640 1 -- 172.21.15.40:0/1139312062 learned_addr learned my addr 172.21.15.40:0/1139312062 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:43.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e91e00640 1 --1- 172.21.15.40:0/1139312062 >> v1:172.21.15.53:6789/0 conn(0x7f7e9415bca0 0x7f7e940ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:53326/0 (socket says 172.21.15.40:53326) 2024-03-24T00:35:43.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e92800640 1 --2- 172.21.15.40:0/1139312062 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e9415c080 0x7f7e940d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:43.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e89400640 1 -- 172.21.15.40:0/1139312062 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e9415c080 msgr2=0x7f7e940d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:43.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e89400640 1 --2- 172.21.15.40:0/1139312062 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e9415c080 0x7f7e940d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e89400640 1 -- 172.21.15.40:0/1139312062 >> v1:172.21.15.53:6789/0 conn(0x7f7e9415bca0 legacy=0x7f7e940ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:43.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e89400640 1 -- 172.21.15.40:0/1139312062 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e940b5050 con 0x7f7e940ceba0 2024-03-24T00:35:43.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.786+0000 7f7e91400640 1 -- 172.21.15.40:0/1139312062 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543145405 0 0) 0x7f7e940d0780 con 0x7f7e9415bca0 2024-03-24T00:35:43.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e89400640 1 --2- 172.21.15.40:0/1139312062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e940ceba0 0x7f7e940d2590 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c009870 tx=0x7f7e7c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d826b892cba801b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:43.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e91400640 1 -- 172.21.15.40:0/1139312062 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0b2070 con 0x7f7e940ceba0 2024-03-24T00:35:43.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e91400640 1 -- 172.21.15.40:0/1139312062 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c0a9070 con 0x7f7e940ceba0 2024-03-24T00:35:43.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e91400640 1 -- 172.21.15.40:0/1139312062 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0ad880 con 0x7f7e940ceba0 2024-03-24T00:35:43.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e93200640 1 -- 172.21.15.40:0/1139312062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e940ceba0 msgr2=0x7f7e940d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:43.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e93200640 1 --2- 172.21.15.40:0/1139312062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e940ceba0 0x7f7e940d2590 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c009870 tx=0x7f7e7c01c560 comp rx=0 tx=0).stop 2024-03-24T00:35:43.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e93200640 1 -- 172.21.15.40:0/1139312062 shutdown_connections 2024-03-24T00:35:43.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e93200640 1 --2- 172.21.15.40:0/1139312062 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e9415c080 0x7f7e940d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e93200640 1 --2- 172.21.15.40:0/1139312062 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e940ceba0 0x7f7e940d2590 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e93200640 1 -- 172.21.15.40:0/1139312062 >> 172.21.15.40:0/1139312062 conn(0x7f7e940d1aa0 msgr2=0x7f7e940d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:43.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e93200640 1 -- 172.21.15.40:0/1139312062 shutdown_connections 2024-03-24T00:35:43.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e93200640 1 -- 172.21.15.40:0/1139312062 wait complete. 2024-03-24T00:35:43.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e93200640 1 Processor -- start 2024-03-24T00:35:43.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.790+0000 7f7e93200640 1 -- start start 2024-03-24T00:35:43.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e93200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e940ceba0 0x7f7e94155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:43.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e93200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e9415bca0 0x7f7e940c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:43.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e93200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e9415c080 0x7f7e9409b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:43.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e93200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940b6150 con 0x7f7e9415bca0 2024-03-24T00:35:43.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e93200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940b5fd0 con 0x7f7e9415c080 2024-03-24T00:35:43.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e93200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940b62d0 con 0x7f7e940ceba0 2024-03-24T00:35:43.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e89400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e9415bca0 0x7f7e940c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:43.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e91e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e940ceba0 0x7f7e94155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:43.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e92800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e9415c080 0x7f7e9409b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:43.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e91e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e940ceba0 0x7f7e94155db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:36664/0 (socket says 172.21.15.40:36664) 2024-03-24T00:35:43.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e91e00640 1 -- 172.21.15.40:0/3931169604 learned_addr learned my addr 172.21.15.40:0/3931169604 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:43.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e89400640 1 -- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e940ceba0 msgr2=0x7f7e94155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:43.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e89400640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e940ceba0 0x7f7e94155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e89400640 1 -- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e9415c080 msgr2=0x7f7e9409b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:43.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e89400640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e9415c080 0x7f7e9409b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:43.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e89400640 1 -- 172.21.15.40:0/3931169604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e940ca7f0 con 0x7f7e9415bca0 2024-03-24T00:35:43.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e91e00640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e940ceba0 0x7f7e94155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:43.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e92800640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e9415c080 0x7f7e9409b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:43.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e89400640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e9415bca0 0x7f7e940c8310 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0ac000 tx=0x7f7e7c01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:43.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e8aa00640 1 -- 172.21.15.40:0/3931169604 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0b2040 con 0x7f7e9415bca0 2024-03-24T00:35:43.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e8aa00640 1 -- 172.21.15.40:0/3931169604 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c004250 con 0x7f7e9415bca0 2024-03-24T00:35:43.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e93200640 1 -- 172.21.15.40:0/3931169604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e940caad0 con 0x7f7e9415bca0 2024-03-24T00:35:43.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e8aa00640 1 -- 172.21.15.40:0/3931169604 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c003620 con 0x7f7e9415bca0 2024-03-24T00:35:43.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e93200640 1 -- 172.21.15.40:0/3931169604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e94156a00 con 0x7f7e9415bca0 2024-03-24T00:35:43.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.794+0000 7f7e93200640 1 -- 172.21.15.40:0/3931169604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9409fb10 con 0x7f7e9415bca0 2024-03-24T00:35:43.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.798+0000 7f7e8aa00640 1 -- 172.21.15.40:0/3931169604 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7e7c0b5070 con 0x7f7e9415bca0 2024-03-24T00:35:43.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.798+0000 7f7e8aa00640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e70067690 0x7f7e70069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:43.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.798+0000 7f7e8aa00640 1 -- 172.21.15.40:0/3931169604 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e7c11fb30 con 0x7f7e9415bca0 2024-03-24T00:35:43.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.798+0000 7f7e91e00640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e70067690 0x7f7e70069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:43.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.798+0000 7f7e91e00640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e70067690 0x7f7e70069b50 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7e84000f70 tx=0x7f7e84007e20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:43.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.802+0000 7f7e8aa00640 1 -- 172.21.15.40:0/3931169604 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7e7c0c5030 con 0x7f7e9415bca0 2024-03-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:43.998+0000 7f7e93200640 1 -- 172.21.15.40:0/3931169604 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e940d1880 con 0x7f7e9415bca0 2024-03-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e8aa00640 1 -- 172.21.15.40:0/3931169604 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f7e7c0f6020 con 0x7f7e9415bca0 2024-03-24T00:35:44.004 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":73,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6062,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/4182264423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":4182264423},{"type":"v1","addr":"172.21.15.53:6836","nonce":4182264423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:35:14.689580+0000","modified":"2024-03-24T00:35:35.452176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6954,"mds_1":6951,"mds_2":6733},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6733":{"gid":6733,"name":"b","rank":2,"incarnation":71,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/334375209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":334375209},{"type":"v1","addr":"172.21.15.53:6837","nonce":334375209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6951":{"gid":6951,"name":"c","rank":1,"incarnation":69,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/221849812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":221849812},{"type":"v1","addr":"172.21.15.40:6834","nonce":221849812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6954":{"gid":6954,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/2177098794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2177098794},{"type":"v1","addr":"172.21.15.40:6835","nonce":2177098794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6954,"qdb_cluster":[6951,6954,6733]},"id":5}]} 2024-03-24T00:35:44.005 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 73 2024-03-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e93200640 1 -- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e70067690 msgr2=0x7f7e70069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e93200640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e70067690 0x7f7e70069b50 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f7e84000f70 tx=0x7f7e84007e20 comp rx=0 tx=0).stop 2024-03-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e93200640 1 -- 172.21.15.40:0/3931169604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e9415bca0 msgr2=0x7f7e940c8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e93200640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e9415bca0 0x7f7e940c8310 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0ac000 tx=0x7f7e7c01ada0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e93200640 1 -- 172.21.15.40:0/3931169604 shutdown_connections 2024-03-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e93200640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e70067690 0x7f7e70069b50 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e93200640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e9415c080 0x7f7e9409b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e93200640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e9415bca0 0x7f7e940c8310 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e93200640 1 --2- 172.21.15.40:0/3931169604 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e940ceba0 0x7f7e94155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e93200640 1 -- 172.21.15.40:0/3931169604 >> 172.21.15.40:0/3931169604 conn(0x7f7e940d1aa0 msgr2=0x7f7e9406ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e93200640 1 -- 172.21.15.40:0/3931169604 shutdown_connections 2024-03-24T00:35:44.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.002+0000 7f7e93200640 1 -- 172.21.15.40:0/3931169604 wait complete. 2024-03-24T00:35:44.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:44.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.162+0000 7faccbc00640 1 Processor -- start 2024-03-24T00:35:44.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.162+0000 7faccbc00640 1 -- start start 2024-03-24T00:35:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.162+0000 7faccbc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facc415dfc0 0x7facc40cd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.162+0000 7faccbc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7facc415e3a0 0x7facc40cd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.162+0000 7faccbc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facc40cfaa0 con 0x7facc40d0b40 2024-03-24T00:35:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.162+0000 7faccbc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facc40cf870 con 0x7facc415e3a0 2024-03-24T00:35:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.162+0000 7faccbc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc40cfd80 con 0x7facc415dfc0 2024-03-24T00:35:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7facc415e3a0 0x7facc40cd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc9200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facc415dfc0 0x7facc40cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7facc415e3a0 0x7facc40cd8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56140/0 (socket says 172.21.15.40:56140) 2024-03-24T00:35:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc9200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facc415dfc0 0x7facc40cd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:36668/0 (socket says 172.21.15.40:36668) 2024-03-24T00:35:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc3e00640 1 -- 172.21.15.40:0/3307796806 learned_addr learned my addr 172.21.15.40:0/3307796806 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:44.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc9c00640 1 --1- 172.21.15.40:0/3307796806 >> v1:172.21.15.40:6789/0 conn(0x7facc40d0b40 0x7facc40ce690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45738/0 (socket says 172.21.15.40:45738) 2024-03-24T00:35:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc3e00640 1 -- 172.21.15.40:0/3307796806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facc415dfc0 msgr2=0x7facc40cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc3e00640 1 --2- 172.21.15.40:0/3307796806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facc415dfc0 0x7facc40cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc3e00640 1 -- 172.21.15.40:0/3307796806 >> v1:172.21.15.40:6789/0 conn(0x7facc40d0b40 legacy=0x7facc40ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc3e00640 1 -- 172.21.15.40:0/3307796806 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc40d0060 con 0x7facc415e3a0 2024-03-24T00:35:44.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc3400640 1 -- 172.21.15.40:0/3307796806 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963100442 0 0) 0x7facc40cfaa0 con 0x7facc40d0b40 2024-03-24T00:35:44.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc3e00640 1 --2- 172.21.15.40:0/3307796806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc415e3a0 0x7facc40cd8d0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7facb0009870 tx=0x7facb001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617013f8049f59e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:44.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc3400640 1 -- 172.21.15.40:0/3307796806 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00b0070 con 0x7facc415e3a0 2024-03-24T00:35:44.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc3400640 1 -- 172.21.15.40:0/3307796806 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb0004230 con 0x7facc415e3a0 2024-03-24T00:35:44.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.166+0000 7facc3400640 1 -- 172.21.15.40:0/3307796806 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb0003620 con 0x7facc415e3a0 2024-03-24T00:35:44.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 -- 172.21.15.40:0/3307796806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc415e3a0 msgr2=0x7facc40cd8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 --2- 172.21.15.40:0/3307796806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc415e3a0 0x7facc40cd8d0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7facb0009870 tx=0x7facb001c440 comp rx=0 tx=0).stop 2024-03-24T00:35:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 -- 172.21.15.40:0/3307796806 shutdown_connections 2024-03-24T00:35:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 --2- 172.21.15.40:0/3307796806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc415e3a0 0x7facc40cd8d0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 --2- 172.21.15.40:0/3307796806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facc415dfc0 0x7facc40cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 -- 172.21.15.40:0/3307796806 >> 172.21.15.40:0/3307796806 conn(0x7facc40b37f0 msgr2=0x7facc40b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 -- 172.21.15.40:0/3307796806 shutdown_connections 2024-03-24T00:35:44.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 -- 172.21.15.40:0/3307796806 wait complete. 2024-03-24T00:35:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 Processor -- start 2024-03-24T00:35:44.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 -- start start 2024-03-24T00:35:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facc40d0b40 0x7facc4155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc415dfc0 0x7facc40aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facc415e3a0 0x7facc415cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc416ab60 con 0x7facc415dfc0 2024-03-24T00:35:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc416a9e0 con 0x7facc415e3a0 2024-03-24T00:35:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7faccbc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facc416ace0 con 0x7facc40d0b40 2024-03-24T00:35:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7facc3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc415dfc0 0x7facc40aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7facc3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc415dfc0 0x7facc40aead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56144/0 (socket says 172.21.15.40:56144) 2024-03-24T00:35:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7facc9200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facc40d0b40 0x7facc4155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7facc3e00640 1 -- 172.21.15.40:0/2269489997 learned_addr learned my addr 172.21.15.40:0/2269489997 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7facc9c00640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facc415e3a0 0x7facc415cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7facc3e00640 1 -- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facc40d0b40 msgr2=0x7facc4155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7facc3e00640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facc40d0b40 0x7facc4155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7facc3e00640 1 -- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facc415e3a0 msgr2=0x7facc415cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7facc3e00640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facc415e3a0 0x7facc415cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.170+0000 7facc3e00640 1 -- 172.21.15.40:0/2269489997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc406b2f0 con 0x7facc415dfc0 2024-03-24T00:35:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.174+0000 7facc3e00640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc415dfc0 0x7facc40aead0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7facb001aaf0 tx=0x7facb00191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.174+0000 7facc0c00640 1 -- 172.21.15.40:0/2269489997 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00b0040 con 0x7facc415dfc0 2024-03-24T00:35:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.174+0000 7facc0c00640 1 -- 172.21.15.40:0/2269489997 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb0003d30 con 0x7facc415dfc0 2024-03-24T00:35:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.174+0000 7faccbc00640 1 -- 172.21.15.40:0/2269489997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc406b5d0 con 0x7facc415dfc0 2024-03-24T00:35:44.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.174+0000 7facc0c00640 1 -- 172.21.15.40:0/2269489997 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb00b6680 con 0x7facc415dfc0 2024-03-24T00:35:44.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.174+0000 7faccbc00640 1 -- 172.21.15.40:0/2269489997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc406bb90 con 0x7facc415dfc0 2024-03-24T00:35:44.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.174+0000 7faccbc00640 1 -- 172.21.15.40:0/2269489997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc4068170 con 0x7facc415dfc0 2024-03-24T00:35:44.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.174+0000 7facc0c00640 1 -- 172.21.15.40:0/2269489997 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7facb00bd060 con 0x7facc415dfc0 2024-03-24T00:35:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.178+0000 7facc0c00640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faca4067660 0x7faca4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.178+0000 7facc0c00640 1 -- 172.21.15.40:0/2269489997 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7facb011efe0 con 0x7facc415dfc0 2024-03-24T00:35:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.178+0000 7facc9200640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faca4067660 0x7faca4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.178+0000 7facc9200640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faca4067660 0x7faca4069b20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7facb8009ac0 tx=0x7facb8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:44.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.178+0000 7facc0c00640 1 -- 172.21.15.40:0/2269489997 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7facb0016150 con 0x7facc415dfc0 2024-03-24T00:35:44.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.342+0000 7faccbc00640 1 -- 172.21.15.40:0/2269489997 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facc41619f0 con 0x7facc415dfc0 2024-03-24T00:35:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7facc0c00640 1 -- 172.21.15.40:0/2269489997 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16608 (secure 0 0 0) 0x7facb00ef590 con 0x7facc415dfc0 2024-03-24T00:35:44.349 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:44.349 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":70,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:35:16.298084+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:13.099776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:13.675426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7faccbc00640 1 -- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faca4067660 msgr2=0x7faca4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7faccbc00640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faca4067660 0x7faca4069b20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7facb8009ac0 tx=0x7facb8009190 comp rx=0 tx=0).stop 2024-03-24T00:35:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7faccbc00640 1 -- 172.21.15.40:0/2269489997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc415dfc0 msgr2=0x7facc40aead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7faccbc00640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc415dfc0 0x7facc40aead0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7facb001aaf0 tx=0x7facb00191e0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7faccbc00640 1 -- 172.21.15.40:0/2269489997 shutdown_connections 2024-03-24T00:35:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7faccbc00640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faca4067660 0x7faca4069b20 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7faccbc00640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facc415e3a0 0x7facc415cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7faccbc00640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facc415dfc0 0x7facc40aead0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7faccbc00640 1 --2- 172.21.15.40:0/2269489997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facc40d0b40 0x7facc4155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7faccbc00640 1 -- 172.21.15.40:0/2269489997 >> 172.21.15.40:0/2269489997 conn(0x7facc40b37f0 msgr2=0x7facc40b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7faccbc00640 1 -- 172.21.15.40:0/2269489997 shutdown_connections 2024-03-24T00:35:44.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.346+0000 7faccbc00640 1 -- 172.21.15.40:0/2269489997 wait complete. 2024-03-24T00:35:44.371 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:35:13.099776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:35:13.675426+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:35:44.371 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:35:44.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.510+0000 7f05b1400640 1 Processor -- start 2024-03-24T00:35:44.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.510+0000 7f05b1400640 1 -- start start 2024-03-24T00:35:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.514+0000 7f05b1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f05ac071b00 0x7f05ac071f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.514+0000 7f05b1400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ac0b3260 con 0x7f05ac15ac60 2024-03-24T00:35:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.514+0000 7f05b1400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ac0b3540 con 0x7f05ac06cb80 2024-03-24T00:35:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.514+0000 7f05b1400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05ac072490 con 0x7f05ac071b00 2024-03-24T00:35:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.514+0000 7f05ab200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f05ac071b00 0x7f05ac071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.514+0000 7f05aa800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f05ac15ac60 0x7f05ac0b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45742/0 (socket says 172.21.15.40:45742) 2024-03-24T00:35:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.514+0000 7f05aa800640 1 -- 172.21.15.40:0/4057683620 learned_addr learned my addr 172.21.15.40:0/4057683620 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:44.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.514+0000 7f05a3e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f05ac06cb80 0x7f05ac0b4300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:53332/0 (socket says 172.21.15.40:53332) 2024-03-24T00:35:44.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.514+0000 7f05ab200640 1 -- 172.21.15.40:0/4057683620 >> v1:172.21.15.53:6789/0 conn(0x7f05ac06cb80 legacy=0x7f05ac0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.514+0000 7f05ab200640 1 -- 172.21.15.40:0/4057683620 >> v1:172.21.15.40:6789/0 conn(0x7f05ac15ac60 legacy=0x7f05ac0b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.514+0000 7f05ab200640 1 -- 172.21.15.40:0/4057683620 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac0b3820 con 0x7f05ac071b00 2024-03-24T00:35:44.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.518+0000 7f05ab200640 1 --2- 172.21.15.40:0/4057683620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05ac071b00 0x7f05ac071f50 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009d90 tx=0x7f05a401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edaafa91b1a8a87b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:44.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.518+0000 7f05a9e00640 1 -- 172.21.15.40:0/4057683620 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a40b2070 con 0x7f05ac071b00 2024-03-24T00:35:44.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.518+0000 7f05a9e00640 1 -- 172.21.15.40:0/4057683620 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a40a57a0 con 0x7f05ac071b00 2024-03-24T00:35:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.518+0000 7f05a9e00640 1 -- 172.21.15.40:0/4057683620 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a40a5b70 con 0x7f05ac071b00 2024-03-24T00:35:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.518+0000 7f05b1400640 1 -- 172.21.15.40:0/4057683620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05ac071b00 msgr2=0x7f05ac071f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.518+0000 7f05b1400640 1 --2- 172.21.15.40:0/4057683620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05ac071b00 0x7f05ac071f50 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009d90 tx=0x7f05a401c8e0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.518+0000 7f05b1400640 1 -- 172.21.15.40:0/4057683620 shutdown_connections 2024-03-24T00:35:44.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.518+0000 7f05b1400640 1 --2- 172.21.15.40:0/4057683620 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05ac071b00 0x7f05ac071f50 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.518+0000 7f05b1400640 1 -- 172.21.15.40:0/4057683620 >> 172.21.15.40:0/4057683620 conn(0x7f05ac0b1660 msgr2=0x7f05ac0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:44.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.518+0000 7f05b1400640 1 -- 172.21.15.40:0/4057683620 shutdown_connections 2024-03-24T00:35:44.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.518+0000 7f05b1400640 1 -- 172.21.15.40:0/4057683620 wait complete. 2024-03-24T00:35:44.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05b1400640 1 Processor -- start 2024-03-24T00:35:44.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05b1400640 1 -- start start 2024-03-24T00:35:44.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05b1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05ac06cb80 0x7f05ac14fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05b1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05ac071b00 0x7f05ac0b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05b1400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05ac15ac60 0x7f05ac0b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05b1400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac06d520 con 0x7f05ac071b00 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05b1400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac06d3a0 con 0x7f05ac06cb80 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05b1400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05ac06d6a0 con 0x7f05ac15ac60 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05aa800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05ac06cb80 0x7f05ac14fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05a3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05ac071b00 0x7f05ac0b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05aa800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05ac06cb80 0x7f05ac14fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:36252/0 (socket says 172.21.15.40:36252) 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05aa800640 1 -- 172.21.15.40:0/1120936737 learned_addr learned my addr 172.21.15.40:0/1120936737 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05ab200640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05ac15ac60 0x7f05ac0b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05aa800640 1 -- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05ac15ac60 msgr2=0x7f05ac0b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05aa800640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05ac15ac60 0x7f05ac0b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05aa800640 1 -- 172.21.15.40:0/1120936737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05ac071b00 msgr2=0x7f05ac0b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05aa800640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05ac071b00 0x7f05ac0b0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05aa800640 1 -- 172.21.15.40:0/1120936737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac15cf50 con 0x7f05ac06cb80 2024-03-24T00:35:44.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05a3e00640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05ac071b00 0x7f05ac0b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:44.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05aa800640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05ac06cb80 0x7f05ac14fcf0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f059c00c370 tx=0x7f059c00c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:44.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05a2a00640 1 -- 172.21.15.40:0/1120936737 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c017070 con 0x7f05ac06cb80 2024-03-24T00:35:44.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05a2a00640 1 -- 172.21.15.40:0/1120936737 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059c005cf0 con 0x7f05ac06cb80 2024-03-24T00:35:44.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05a2a00640 1 -- 172.21.15.40:0/1120936737 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059c0072f0 con 0x7f05ac06cb80 2024-03-24T00:35:44.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05b1400640 1 -- 172.21.15.40:0/1120936737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac1fa7f0 con 0x7f05ac06cb80 2024-03-24T00:35:44.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.522+0000 7f05b1400640 1 -- 172.21.15.40:0/1120936737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac1faa90 con 0x7f05ac06cb80 2024-03-24T00:35:44.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.526+0000 7f05b1400640 1 -- 172.21.15.40:0/1120936737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ac068170 con 0x7f05ac06cb80 2024-03-24T00:35:44.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.526+0000 7f05a2a00640 1 -- 172.21.15.40:0/1120936737 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f059c01a070 con 0x7f05ac06cb80 2024-03-24T00:35:44.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.526+0000 7f05a2a00640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0588067690 0x7f0588069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.526+0000 7f05a2a00640 1 -- 172.21.15.40:0/1120936737 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f059c084b30 con 0x7f05ac06cb80 2024-03-24T00:35:44.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.526+0000 7f05a3e00640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0588067690 0x7f0588069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.526+0000 7f05a3e00640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0588067690 0x7f0588069b50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f05ac0b2190 tx=0x7f0594007e50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:44.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.530+0000 7f05a2a00640 1 -- 172.21.15.40:0/1120936737 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f05ac15cf50 con 0x7f05ac06cb80 2024-03-24T00:35:44.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 -- 172.21.15.40:0/1120936737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05ac0ae290 con 0x7f05ac06cb80 2024-03-24T00:35:44.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05a2a00640 1 -- 172.21.15.40:0/1120936737 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f059c055160 con 0x7f05ac06cb80 2024-03-24T00:35:44.697 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:35:44.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 -- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0588067690 msgr2=0x7f0588069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0588067690 0x7f0588069b50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f05ac0b2190 tx=0x7f0594007e50 comp rx=0 tx=0).stop 2024-03-24T00:35:44.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 -- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05ac06cb80 msgr2=0x7f05ac14fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05ac06cb80 0x7f05ac14fcf0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f059c00c370 tx=0x7f059c00c830 comp rx=0 tx=0).stop 2024-03-24T00:35:44.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 -- 172.21.15.40:0/1120936737 shutdown_connections 2024-03-24T00:35:44.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0588067690 0x7f0588069b50 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05ac15ac60 0x7f05ac0b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f05ac071b00 0x7f05ac0b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 --2- 172.21.15.40:0/1120936737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05ac06cb80 0x7f05ac14fcf0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 -- 172.21.15.40:0/1120936737 >> 172.21.15.40:0/1120936737 conn(0x7f05ac0b1660 msgr2=0x7f05ac0ce220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:44.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 -- 172.21.15.40:0/1120936737 shutdown_connections 2024-03-24T00:35:44.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.694+0000 7f05b1400640 1 -- 172.21.15.40:0/1120936737 wait complete. 2024-03-24T00:35:44.719 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:35:44.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.858+0000 7ff10fa00640 1 Processor -- start 2024-03-24T00:35:44.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.858+0000 7ff10fa00640 1 -- start start 2024-03-24T00:35:44.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff10fa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff10815bd00 0x7ff1080ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff10fa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff1080728d0 0x7ff1080cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff10fa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff108072d30 con 0x7ff10815f8c0 2024-03-24T00:35:44.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff10fa00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1080cf790 con 0x7ff1080728d0 2024-03-24T00:35:44.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff10fa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff108071de0 con 0x7ff10815bd00 2024-03-24T00:35:44.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff107e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff1080728d0 0x7ff1080cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff10d000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff10815bd00 0x7ff1080ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff10da00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff10815f8c0 0x7ff10806cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:53334/0 (socket says 172.21.15.40:53334) 2024-03-24T00:35:44.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff10da00640 1 -- 172.21.15.40:0/476124985 learned_addr learned my addr 172.21.15.40:0/476124985 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:44.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff107e00640 1 -- 172.21.15.40:0/476124985 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff10815bd00 msgr2=0x7ff1080ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff107e00640 1 --2- 172.21.15.40:0/476124985 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff10815bd00 0x7ff1080ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff107e00640 1 -- 172.21.15.40:0/476124985 >> v1:172.21.15.53:6789/0 conn(0x7ff10815f8c0 legacy=0x7ff10806cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.862+0000 7ff107e00640 1 -- 172.21.15.40:0/476124985 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1080cea50 con 0x7ff1080728d0 2024-03-24T00:35:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.866+0000 7ff107e00640 1 --2- 172.21.15.40:0/476124985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1080728d0 0x7ff1080cf250 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff0f40097c0 tx=0x7ff0f401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f11b00f1834da3c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.866+0000 7ff107400640 1 -- 172.21.15.40:0/476124985 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f40a7070 con 0x7ff1080728d0 2024-03-24T00:35:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.866+0000 7ff107400640 1 -- 172.21.15.40:0/476124985 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f40b0070 con 0x7ff1080728d0 2024-03-24T00:35:44.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.866+0000 7ff107400640 1 -- 172.21.15.40:0/476124985 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f40ab890 con 0x7ff1080728d0 2024-03-24T00:35:44.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.866+0000 7ff10fa00640 1 -- 172.21.15.40:0/476124985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1080728d0 msgr2=0x7ff1080cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.866+0000 7ff10fa00640 1 --2- 172.21.15.40:0/476124985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1080728d0 0x7ff1080cf250 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff0f40097c0 tx=0x7ff0f401c470 comp rx=0 tx=0).stop 2024-03-24T00:35:44.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.866+0000 7ff10fa00640 1 -- 172.21.15.40:0/476124985 shutdown_connections 2024-03-24T00:35:44.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.866+0000 7ff10fa00640 1 --2- 172.21.15.40:0/476124985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1080728d0 0x7ff1080cf250 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.866+0000 7ff10fa00640 1 --2- 172.21.15.40:0/476124985 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff10815bd00 0x7ff1080ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.866+0000 7ff10fa00640 1 -- 172.21.15.40:0/476124985 >> 172.21.15.40:0/476124985 conn(0x7ff1080b1660 msgr2=0x7ff1080b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:44.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.866+0000 7ff10fa00640 1 -- 172.21.15.40:0/476124985 shutdown_connections 2024-03-24T00:35:44.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.866+0000 7ff10fa00640 1 -- 172.21.15.40:0/476124985 wait complete. 2024-03-24T00:35:44.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10fa00640 1 Processor -- start 2024-03-24T00:35:44.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10fa00640 1 -- start start 2024-03-24T00:35:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10fa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1080728d0 0x7ff1080b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10fa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff10815bd00 0x7ff1081486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10fa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10815f8c0 0x7ff10806dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10fa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10815c0e0 con 0x7ff1080728d0 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10fa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10806d630 con 0x7ff10815f8c0 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10fa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1081600e0 con 0x7ff10815bd00 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1080728d0 0x7ff1080b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff107e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff10815bd00 0x7ff1081486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff107e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff10815bd00 0x7ff1081486f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:36706/0 (socket says 172.21.15.40:36706) 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff107e00640 1 -- 172.21.15.40:0/396897328 learned_addr learned my addr 172.21.15.40:0/396897328 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1080728d0 0x7ff1080b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56176/0 (socket says 172.21.15.40:56176) 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10da00640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10815f8c0 0x7ff10806dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10d000640 1 -- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff10815bd00 msgr2=0x7ff1081486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10d000640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff10815bd00 0x7ff1081486f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10d000640 1 -- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10815f8c0 msgr2=0x7ff10806dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10d000640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10815f8c0 0x7ff10806dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:44.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10d000640 1 -- 172.21.15.40:0/396897328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10815b4c0 con 0x7ff1080728d0 2024-03-24T00:35:44.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff107e00640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff10815bd00 0x7ff1081486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:44.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10da00640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10815f8c0 0x7ff10806dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:44.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10d000640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1080728d0 0x7ff1080b3c90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00e8a0 tx=0x7ff0fc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff104c00640 1 -- 172.21.15.40:0/396897328 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc017070 con 0x7ff1080728d0 2024-03-24T00:35:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff104c00640 1 -- 172.21.15.40:0/396897328 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc00c940 con 0x7ff1080728d0 2024-03-24T00:35:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10fa00640 1 -- 172.21.15.40:0/396897328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1080b45e0 con 0x7ff1080728d0 2024-03-24T00:35:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff104c00640 1 -- 172.21.15.40:0/396897328 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc00ccf0 con 0x7ff1080728d0 2024-03-24T00:35:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.870+0000 7ff10fa00640 1 -- 172.21.15.40:0/396897328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff10815acc0 con 0x7ff1080728d0 2024-03-24T00:35:44.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.874+0000 7ff10fa00640 1 -- 172.21.15.40:0/396897328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff108068170 con 0x7ff1080728d0 2024-03-24T00:35:44.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.874+0000 7ff104c00640 1 -- 172.21.15.40:0/396897328 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff0fc01a070 con 0x7ff1080728d0 2024-03-24T00:35:44.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.874+0000 7ff104c00640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff0e8067660 0x7ff0e8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:44.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.874+0000 7ff104c00640 1 -- 172.21.15.40:0/396897328 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0fc0857e0 con 0x7ff1080728d0 2024-03-24T00:35:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.878+0000 7ff107e00640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff0e8067660 0x7ff0e8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:44.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.878+0000 7ff107e00640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff0e8067660 0x7ff0e8069b20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff0f40097c0 tx=0x7ff0f40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:44.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:44.886+0000 7ff104c00640 1 -- 172.21.15.40:0/396897328 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff0fc055e10 con 0x7ff1080728d0 2024-03-24T00:35:45.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.046+0000 7ff10fa00640 1 -- 172.21.15.40:0/396897328 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff1080b11a0 con 0x7ff1080728d0 2024-03-24T00:35:45.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.046+0000 7ff104c00640 1 -- 172.21.15.40:0/396897328 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff0fc059eb0 con 0x7ff1080728d0 2024-03-24T00:35:45.051 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:35:45.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.050+0000 7ff10fa00640 1 -- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff0e8067660 msgr2=0x7ff0e8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.050+0000 7ff10fa00640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff0e8067660 0x7ff0e8069b20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff0f40097c0 tx=0x7ff0f40a7000 comp rx=0 tx=0).stop 2024-03-24T00:35:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.050+0000 7ff10fa00640 1 -- 172.21.15.40:0/396897328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1080728d0 msgr2=0x7ff1080b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.050+0000 7ff10fa00640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1080728d0 0x7ff1080b3c90 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00e8a0 tx=0x7ff0fc00ed60 comp rx=0 tx=0).stop 2024-03-24T00:35:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.050+0000 7ff10fa00640 1 -- 172.21.15.40:0/396897328 shutdown_connections 2024-03-24T00:35:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.050+0000 7ff10fa00640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff0e8067660 0x7ff0e8069b20 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.050+0000 7ff10fa00640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10815f8c0 0x7ff10806dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.050+0000 7ff10fa00640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff10815bd00 0x7ff1081486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.050+0000 7ff10fa00640 1 --2- 172.21.15.40:0/396897328 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1080728d0 0x7ff1080b3c90 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.050+0000 7ff10fa00640 1 -- 172.21.15.40:0/396897328 >> 172.21.15.40:0/396897328 conn(0x7ff1080b1660 msgr2=0x7ff1080b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.050+0000 7ff10fa00640 1 -- 172.21.15.40:0/396897328 shutdown_connections 2024-03-24T00:35:45.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.050+0000 7ff10fa00640 1 -- 172.21.15.40:0/396897328 wait complete. 2024-03-24T00:35:45.072 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:35:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77cbc00640 1 Processor -- start 2024-03-24T00:35:45.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77cbc00640 1 -- start start 2024-03-24T00:35:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77cbc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f77c415f5c0 0x7f77c40b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77cbc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f77c40cf1e0 0x7f77c40d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77cbc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c40b0f10 con 0x7f77c40cfbb0 2024-03-24T00:35:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77cbc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c415f9a0 con 0x7f77c40cf1e0 2024-03-24T00:35:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77cbc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c40cf5f0 con 0x7f77c415f5c0 2024-03-24T00:35:45.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77c3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f77c40cfbb0 0x7f77c40b1110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45746/0 (socket says 172.21.15.40:45746) 2024-03-24T00:35:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77c3e00640 1 -- 172.21.15.40:0/1943380699 learned_addr learned my addr 172.21.15.40:0/1943380699 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77c9200640 1 --2- 172.21.15.40:0/1943380699 >> v2:172.21.15.53:3300/0 conn(0x7f77c415f5c0 0x7f77c40b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77c9c00640 1 --2- 172.21.15.40:0/1943380699 >> v2:172.21.15.40:3300/0 conn(0x7f77c40cf1e0 0x7f77c40d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77c9200640 1 -- 172.21.15.40:0/1943380699 >> v2:172.21.15.40:3300/0 conn(0x7f77c40cf1e0 msgr2=0x7f77c40d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77c9200640 1 --2- 172.21.15.40:0/1943380699 >> v2:172.21.15.40:3300/0 conn(0x7f77c40cf1e0 0x7f77c40d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77c3400640 1 -- 172.21.15.40:0/1943380699 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797901093 0 0) 0x7f77b0001120 con 0x7f77c40cfbb0 2024-03-24T00:35:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77c9200640 1 -- 172.21.15.40:0/1943380699 >> v1:172.21.15.40:6789/0 conn(0x7f77c40cfbb0 legacy=0x7f77c40b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.214+0000 7f77c9200640 1 -- 172.21.15.40:0/1943380699 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c40cd130 con 0x7f77c415f5c0 2024-03-24T00:35:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77c9200640 1 --2- 172.21.15.40:0/1943380699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77c415f5c0 0x7f77c40b3060 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f77c4071d50 tx=0x7f77b801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=511ffb3d50d07217 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77c3400640 1 -- 172.21.15.40:0/1943380699 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b80b0070 con 0x7f77c415f5c0 2024-03-24T00:35:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77c3400640 1 -- 172.21.15.40:0/1943380699 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b80ab600 con 0x7f77c415f5c0 2024-03-24T00:35:45.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77c3400640 1 -- 172.21.15.40:0/1943380699 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b80ab9d0 con 0x7f77c415f5c0 2024-03-24T00:35:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77cbc00640 1 -- 172.21.15.40:0/1943380699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77c415f5c0 msgr2=0x7f77c40b3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77cbc00640 1 --2- 172.21.15.40:0/1943380699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77c415f5c0 0x7f77c40b3060 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f77c4071d50 tx=0x7f77b801c480 comp rx=0 tx=0).stop 2024-03-24T00:35:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77cbc00640 1 -- 172.21.15.40:0/1943380699 shutdown_connections 2024-03-24T00:35:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77cbc00640 1 --2- 172.21.15.40:0/1943380699 >> v2:172.21.15.40:3300/0 conn(0x7f77c40cf1e0 0x7f77c40d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77cbc00640 1 --2- 172.21.15.40:0/1943380699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77c415f5c0 0x7f77c40b3060 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77cbc00640 1 -- 172.21.15.40:0/1943380699 >> 172.21.15.40:0/1943380699 conn(0x7f77c40b3810 msgr2=0x7f77c40b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77cbc00640 1 -- 172.21.15.40:0/1943380699 shutdown_connections 2024-03-24T00:35:45.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77cbc00640 1 -- 172.21.15.40:0/1943380699 wait complete. 2024-03-24T00:35:45.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77cbc00640 1 Processor -- start 2024-03-24T00:35:45.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.218+0000 7f77cbc00640 1 -- start start 2024-03-24T00:35:45.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77cbc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77c40cf1e0 0x7f77c406c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77cbc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77c40cfbb0 0x7f77c406b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77cbc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77c415f5c0 0x7f77c40af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77cbc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40b2030 con 0x7f77c40cfbb0 2024-03-24T00:35:45.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77cbc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40cdcb0 con 0x7f77c40cf1e0 2024-03-24T00:35:45.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77cbc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40b21b0 con 0x7f77c415f5c0 2024-03-24T00:35:45.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77c40cfbb0 0x7f77c406b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77c40cf1e0 0x7f77c406c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77c40cfbb0 0x7f77c406b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56196/0 (socket says 172.21.15.40:56196) 2024-03-24T00:35:45.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c3e00640 1 -- 172.21.15.40:0/3515493828 learned_addr learned my addr 172.21.15.40:0/3515493828 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:45.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77c40cf1e0 0x7f77c406c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:36292/0 (socket says 172.21.15.40:36292) 2024-03-24T00:35:45.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c9c00640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77c415f5c0 0x7f77c40af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c3e00640 1 -- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77c415f5c0 msgr2=0x7f77c40af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c3e00640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77c415f5c0 0x7f77c40af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c3e00640 1 -- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77c40cf1e0 msgr2=0x7f77c406c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c3e00640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77c40cf1e0 0x7f77c406c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c3e00640 1 -- 172.21.15.40:0/3515493828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c4152610 con 0x7f77c40cfbb0 2024-03-24T00:35:45.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c9200640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77c40cf1e0 0x7f77c406c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:45.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c3e00640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77c40cfbb0 0x7f77c406b950 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f77b000c7b0 tx=0x7f77b000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:45.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c0c00640 1 -- 172.21.15.40:0/3515493828 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b0018070 con 0x7f77c40cfbb0 2024-03-24T00:35:45.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c0c00640 1 -- 172.21.15.40:0/3515493828 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b00072f0 con 0x7f77c40cfbb0 2024-03-24T00:35:45.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77c0c00640 1 -- 172.21.15.40:0/3515493828 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b00047f0 con 0x7f77c40cfbb0 2024-03-24T00:35:45.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77cbc00640 1 -- 172.21.15.40:0/3515493828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c406ca70 con 0x7f77c40cfbb0 2024-03-24T00:35:45.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77cbc00640 1 -- 172.21.15.40:0/3515493828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c406b150 con 0x7f77c40cfbb0 2024-03-24T00:35:45.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.222+0000 7f77cbc00640 1 -- 172.21.15.40:0/3515493828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c406cc40 con 0x7f77c40cfbb0 2024-03-24T00:35:45.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.226+0000 7f77c0c00640 1 -- 172.21.15.40:0/3515493828 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f77b001b070 con 0x7f77c40cfbb0 2024-03-24T00:35:45.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.226+0000 7f77c0c00640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f77a4067660 0x7f77a4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.226+0000 7f77c0c00640 1 -- 172.21.15.40:0/3515493828 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77b0084c80 con 0x7f77c40cfbb0 2024-03-24T00:35:45.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.226+0000 7f77c9200640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f77a4067660 0x7f77a4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.226+0000 7f77c9200640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f77a4067660 0x7f77a4069b20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f77b80ad000 tx=0x7f77b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:45.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.230+0000 7f77c0c00640 1 -- 172.21.15.40:0/3515493828 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f77c4152610 con 0x7f77c40cfbb0 2024-03-24T00:35:45.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 -- 172.21.15.40:0/3515493828 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f77c40b3e20 con 0x7f77c40cfbb0 2024-03-24T00:35:45.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77c0c00640 1 -- 172.21.15.40:0/3515493828 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7f77c40b0f10 con 0x7f77c40cfbb0 2024-03-24T00:35:45.397 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:35:45.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 -- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f77a4067660 msgr2=0x7f77a4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f77a4067660 0x7f77a4069b20 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f77b80ad000 tx=0x7f77b8008040 comp rx=0 tx=0).stop 2024-03-24T00:35:45.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 -- 172.21.15.40:0/3515493828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77c40cfbb0 msgr2=0x7f77c406b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77c40cfbb0 0x7f77c406b950 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f77b000c7b0 tx=0x7f77b000cc70 comp rx=0 tx=0).stop 2024-03-24T00:35:45.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 -- 172.21.15.40:0/3515493828 shutdown_connections 2024-03-24T00:35:45.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f77a4067660 0x7f77a4069b20 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77c415f5c0 0x7f77c40af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f77c40cfbb0 0x7f77c406b950 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 --2- 172.21.15.40:0/3515493828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77c40cf1e0 0x7f77c406c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 -- 172.21.15.40:0/3515493828 >> 172.21.15.40:0/3515493828 conn(0x7f77c40b3810 msgr2=0x7f77c406f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:45.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 -- 172.21.15.40:0/3515493828 shutdown_connections 2024-03-24T00:35:45.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.394+0000 7f77cbc00640 1 -- 172.21.15.40:0/3515493828 wait complete. 2024-03-24T00:35:45.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:45.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa264000640 1 Processor -- start 2024-03-24T00:35:45.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa264000640 1 -- start start 2024-03-24T00:35:45.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa264000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa25c0b2490 0x7fa25c0cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa264000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa25c0b2aa0 0x7fa25c0d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa264000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa25c0d08b0 0x7fa25c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa264000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa25c06b6d0 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa264000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa25c0b2870 con 0x7fa25c0b2490 2024-03-24T00:35:45.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa264000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0b3190 con 0x7fa25c0d08b0 2024-03-24T00:35:45.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa260c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa25c0b2aa0 0x7fa25c0d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa260c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa25c0b2aa0 0x7fa25c0d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56206/0 (socket says 172.21.15.40:56206) 2024-03-24T00:35:45.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa260c00640 1 -- 172.21.15.40:0/228428507 learned_addr learned my addr 172.21.15.40:0/228428507 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:45.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa261600640 1 --2- 172.21.15.40:0/228428507 >> v2:172.21.15.53:3300/0 conn(0x7fa25c0b2490 0x7fa25c0cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.562+0000 7fa262000640 1 --2- 172.21.15.40:0/228428507 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa25c0d08b0 0x7fa25c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.566+0000 7fa260c00640 1 -- 172.21.15.40:0/228428507 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa25c0d08b0 msgr2=0x7fa25c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:45.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.566+0000 7fa260c00640 1 --2- 172.21.15.40:0/228428507 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa25c0d08b0 0x7fa25c0ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.566+0000 7fa260c00640 1 -- 172.21.15.40:0/228428507 >> v2:172.21.15.53:3300/0 conn(0x7fa25c0b2490 msgr2=0x7fa25c0cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:45.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.566+0000 7fa260c00640 1 --2- 172.21.15.40:0/228428507 >> v2:172.21.15.53:3300/0 conn(0x7fa25c0b2490 0x7fa25c0cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.566+0000 7fa260c00640 1 -- 172.21.15.40:0/228428507 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c0b3470 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.566+0000 7fa261600640 1 --2- 172.21.15.40:0/228428507 >> v2:172.21.15.53:3300/0 conn(0x7fa25c0b2490 0x7fa25c0cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:45.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.566+0000 7fa260c00640 1 --2- 172.21.15.40:0/228428507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa25c0b2aa0 0x7fa25c0d0370 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa25c071f30 tx=0x7fa24801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b475f15204977e00 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:45.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.566+0000 7fa25be00640 1 -- 172.21.15.40:0/228428507 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480b0070 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.566+0000 7fa25be00640 1 -- 172.21.15.40:0/228428507 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2480ab600 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.566+0000 7fa25be00640 1 -- 172.21.15.40:0/228428507 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480ab9d0 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.566+0000 7fa264000640 1 -- 172.21.15.40:0/228428507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa25c0b2aa0 msgr2=0x7fa25c0d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.566+0000 7fa264000640 1 --2- 172.21.15.40:0/228428507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa25c0b2aa0 0x7fa25c0d0370 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa25c071f30 tx=0x7fa24801c520 comp rx=0 tx=0).stop 2024-03-24T00:35:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 -- 172.21.15.40:0/228428507 shutdown_connections 2024-03-24T00:35:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 --2- 172.21.15.40:0/228428507 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa25c0d08b0 0x7fa25c0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 --2- 172.21.15.40:0/228428507 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa25c0b2aa0 0x7fa25c0d0370 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 --2- 172.21.15.40:0/228428507 >> v2:172.21.15.53:3300/0 conn(0x7fa25c0b2490 0x7fa25c0cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 -- 172.21.15.40:0/228428507 >> 172.21.15.40:0/228428507 conn(0x7fa25c0d1d10 msgr2=0x7fa25c0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 -- 172.21.15.40:0/228428507 shutdown_connections 2024-03-24T00:35:45.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 -- 172.21.15.40:0/228428507 wait complete. 2024-03-24T00:35:45.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 Processor -- start 2024-03-24T00:35:45.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 -- start start 2024-03-24T00:35:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa25c0b2490 0x7fa25c06f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa25c0b2aa0 0x7fa25c161110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa25c0d08b0 0x7fa25c14f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0cc810 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0cc690 con 0x7fa25c0b2490 2024-03-24T00:35:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa264000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25c0cc990 con 0x7fa25c0d08b0 2024-03-24T00:35:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa260c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa25c0b2aa0 0x7fa25c161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa261600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa25c0b2490 0x7fa25c06f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa260c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa25c0b2aa0 0x7fa25c161110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56220/0 (socket says 172.21.15.40:56220) 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa260c00640 1 -- 172.21.15.40:0/111220104 learned_addr learned my addr 172.21.15.40:0/111220104 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa262000640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa25c0d08b0 0x7fa25c14f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa260c00640 1 -- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa25c0d08b0 msgr2=0x7fa25c14f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa260c00640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa25c0d08b0 0x7fa25c14f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa260c00640 1 -- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa25c0b2490 msgr2=0x7fa25c06f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa260c00640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa25c0b2490 0x7fa25c06f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa260c00640 1 -- 172.21.15.40:0/111220104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25c155480 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa261600640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa25c0b2490 0x7fa25c06f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.570+0000 7fa260c00640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa25c0b2aa0 0x7fa25c161110 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fa24801abd0 tx=0x7fa2480192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.574+0000 7fa259600640 1 -- 172.21.15.40:0/111220104 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480ba070 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.574+0000 7fa259600640 1 -- 172.21.15.40:0/111220104 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2480b4420 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.574+0000 7fa259600640 1 -- 172.21.15.40:0/111220104 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2480b0040 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.574+0000 7fa264000640 1 -- 172.21.15.40:0/111220104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25c06fe90 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.574+0000 7fa264000640 1 -- 172.21.15.40:0/111220104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25c070450 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.574+0000 7fa264000640 1 -- 172.21.15.40:0/111220104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa25c068170 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.574+0000 7fa259600640 1 -- 172.21.15.40:0/111220104 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa2480b47f0 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.574+0000 7fa259600640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa23c067660 0x7fa23c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.574+0000 7fa259600640 1 -- 172.21.15.40:0/111220104 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa24811eb40 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.574+0000 7fa261600640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa23c067660 0x7fa23c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.578+0000 7fa261600640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa23c067660 0x7fa23c069b20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa250009580 tx=0x7fa250009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:45.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.578+0000 7fa259600640 1 -- 172.21.15.40:0/111220104 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa248016110 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.778+0000 7fa264000640 1 -- 172.21.15.40:0/111220104 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa25c15efd0 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.778+0000 7fa259600640 1 -- 172.21.15.40:0/111220104 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5014 (secure 0 0 0) 0x7fa2480ef0d0 con 0x7fa25c0b2aa0 2024-03-24T00:35:45.784 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:45.784 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":73,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6062,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/4182264423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":4182264423},{"type":"v1","addr":"172.21.15.53:6836","nonce":4182264423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:35:14.689580+0000","modified":"2024-03-24T00:35:35.452176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6954,"mds_1":6951,"mds_2":6733},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6733":{"gid":6733,"name":"b","rank":2,"incarnation":71,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/334375209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":334375209},{"type":"v1","addr":"172.21.15.53:6837","nonce":334375209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6951":{"gid":6951,"name":"c","rank":1,"incarnation":69,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/221849812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":221849812},{"type":"v1","addr":"172.21.15.40:6834","nonce":221849812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6954":{"gid":6954,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/2177098794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2177098794},{"type":"v1","addr":"172.21.15.40:6835","nonce":2177098794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6954,"qdb_cluster":[6951,6954,6733]},"id":5}]} 2024-03-24T00:35:45.784 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 73 2024-03-24T00:35:45.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.782+0000 7fa264000640 1 -- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa23c067660 msgr2=0x7fa23c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.782+0000 7fa264000640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa23c067660 0x7fa23c069b20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa250009580 tx=0x7fa250009190 comp rx=0 tx=0).stop 2024-03-24T00:35:45.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.782+0000 7fa264000640 1 -- 172.21.15.40:0/111220104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa25c0b2aa0 msgr2=0x7fa25c161110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.782+0000 7fa264000640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa25c0b2aa0 0x7fa25c161110 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fa24801abd0 tx=0x7fa2480192c0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.782+0000 7fa264000640 1 -- 172.21.15.40:0/111220104 shutdown_connections 2024-03-24T00:35:45.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.782+0000 7fa264000640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa23c067660 0x7fa23c069b20 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.782+0000 7fa264000640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa25c0d08b0 0x7fa25c14f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.782+0000 7fa264000640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa25c0b2aa0 0x7fa25c161110 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.782+0000 7fa264000640 1 --2- 172.21.15.40:0/111220104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa25c0b2490 0x7fa25c06f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.782+0000 7fa264000640 1 -- 172.21.15.40:0/111220104 >> 172.21.15.40:0/111220104 conn(0x7fa25c0d1d10 msgr2=0x7fa25c0cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:45.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.782+0000 7fa264000640 1 -- 172.21.15.40:0/111220104 shutdown_connections 2024-03-24T00:35:45.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.782+0000 7fa264000640 1 -- 172.21.15.40:0/111220104 wait complete. 2024-03-24T00:35:45.805 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:45.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.942+0000 7efe12c00640 1 Processor -- start 2024-03-24T00:35:45.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.942+0000 7efe12c00640 1 -- start start 2024-03-24T00:35:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.942+0000 7efe12c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe0c15e0b0 0x7efe0c0d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe12c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0c0d0e10 con 0x7efe0c06b150 2024-03-24T00:35:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe12c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0c06efd0 con 0x7efe0c0cd3d0 2024-03-24T00:35:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe12c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe0c0d10f0 con 0x7efe0c15e0b0 2024-03-24T00:35:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe0be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efe0c15e0b0 0x7efe0c0d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe03e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7efe0c0cd3d0 0x7efe0c0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:53346/0 (socket says 172.21.15.40:53346) 2024-03-24T00:35:45.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe03e00640 1 -- 172.21.15.40:0/2572967764 learned_addr learned my addr 172.21.15.40:0/2572967764 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:45.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe0be00640 1 -- 172.21.15.40:0/2572967764 >> v1:172.21.15.53:6789/0 conn(0x7efe0c0cd3d0 legacy=0x7efe0c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe0be00640 1 -- 172.21.15.40:0/2572967764 >> v1:172.21.15.40:6789/0 conn(0x7efe0c06b150 legacy=0x7efe0c06b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe0b400640 1 -- 172.21.15.40:0/2572967764 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134180054 0 0) 0x7efe0c06efd0 con 0x7efe0c0cd3d0 2024-03-24T00:35:45.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe0be00640 1 -- 172.21.15.40:0/2572967764 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c06f2b0 con 0x7efe0c15e0b0 2024-03-24T00:35:45.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe0be00640 1 --2- 172.21.15.40:0/2572967764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c15e0b0 0x7efe0c0d0830 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0097c0 tx=0x7efdfc01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d332b83000ed3700 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:45.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe0b400640 1 -- 172.21.15.40:0/2572967764 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc0b0070 con 0x7efe0c15e0b0 2024-03-24T00:35:45.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe0b400640 1 -- 172.21.15.40:0/2572967764 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdfc0ab600 con 0x7efe0c15e0b0 2024-03-24T00:35:45.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.946+0000 7efe0b400640 1 -- 172.21.15.40:0/2572967764 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdfc0ab9d0 con 0x7efe0c15e0b0 2024-03-24T00:35:45.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 -- 172.21.15.40:0/2572967764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c15e0b0 msgr2=0x7efe0c0d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 --2- 172.21.15.40:0/2572967764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c15e0b0 0x7efe0c0d0830 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0097c0 tx=0x7efdfc01c330 comp rx=0 tx=0).stop 2024-03-24T00:35:45.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 -- 172.21.15.40:0/2572967764 shutdown_connections 2024-03-24T00:35:45.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 --2- 172.21.15.40:0/2572967764 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c15e0b0 0x7efe0c0d0830 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 -- 172.21.15.40:0/2572967764 >> 172.21.15.40:0/2572967764 conn(0x7efe0c0d1c90 msgr2=0x7efe0c0d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:45.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 -- 172.21.15.40:0/2572967764 shutdown_connections 2024-03-24T00:35:45.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 -- 172.21.15.40:0/2572967764 wait complete. 2024-03-24T00:35:45.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 Processor -- start 2024-03-24T00:35:45.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 -- start start 2024-03-24T00:35:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe0c06b150 0x7efe0c15c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe0c0cd3d0 0x7efe0c0c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c15e0b0 0x7efe0c0ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c15e490 con 0x7efe0c15e0b0 2024-03-24T00:35:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c0d29d0 con 0x7efe0c06b150 2024-03-24T00:35:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe12c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c15e610 con 0x7efe0c0cd3d0 2024-03-24T00:35:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe10c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c15e0b0 0x7efe0c0ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe10c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c15e0b0 0x7efe0c0ca2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56226/0 (socket says 172.21.15.40:56226) 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe10c00640 1 -- 172.21.15.40:0/3776566889 learned_addr learned my addr 172.21.15.40:0/3776566889 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe0be00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe0c06b150 0x7efe0c15c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.950+0000 7efe03e00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe0c0cd3d0 0x7efe0c0c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe10c00640 1 -- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe0c0cd3d0 msgr2=0x7efe0c0c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe10c00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe0c0cd3d0 0x7efe0c0c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe10c00640 1 -- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe0c06b150 msgr2=0x7efe0c15c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe10c00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe0c06b150 0x7efe0c15c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe10c00640 1 -- 172.21.15.40:0/3776566889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c0b4e80 con 0x7efe0c15e0b0 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe0be00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe0c06b150 0x7efe0c15c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe10c00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c15e0b0 0x7efe0c0ca2f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7efe0400c1b0 tx=0x7efe0400c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe08c00640 1 -- 172.21.15.40:0/3776566889 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04019070 con 0x7efe0c15e0b0 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe08c00640 1 -- 172.21.15.40:0/3776566889 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe04009050 con 0x7efe0c15e0b0 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe08c00640 1 -- 172.21.15.40:0/3776566889 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04009440 con 0x7efe0c15e0b0 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe12c00640 1 -- 172.21.15.40:0/3776566889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0c0b16d0 con 0x7efe0c15e0b0 2024-03-24T00:35:45.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe12c00640 1 -- 172.21.15.40:0/3776566889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0c0b1c90 con 0x7efe0c15e0b0 2024-03-24T00:35:45.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe12c00640 1 -- 172.21.15.40:0/3776566889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0c0b1e90 con 0x7efe0c15e0b0 2024-03-24T00:35:45.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe08c00640 1 -- 172.21.15.40:0/3776566889 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7efe0401c070 con 0x7efe0c15e0b0 2024-03-24T00:35:45.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe08c00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efde8067660 0x7efde8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:45.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.954+0000 7efe08c00640 1 -- 172.21.15.40:0/3776566889 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe04087980 con 0x7efe0c15e0b0 2024-03-24T00:35:45.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.958+0000 7efe0be00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efde8067660 0x7efde8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:45.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.958+0000 7efe0be00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efde8067660 0x7efde8069b20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0ad000 tx=0x7efdfc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:45.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:45.958+0000 7efe08c00640 1 -- 172.21.15.40:0/3776566889 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7efe0c0b1e90 con 0x7efe0c15e0b0 2024-03-24T00:35:46.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.122+0000 7efe12c00640 1 -- 172.21.15.40:0/3776566889 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe0c068170 con 0x7efe0c15e0b0 2024-03-24T00:35:46.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe08c00640 1 -- 172.21.15.40:0/3776566889 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16608 (secure 0 0 0) 0x7efe04057fa0 con 0x7efe0c15e0b0 2024-03-24T00:35:46.129 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:46.129 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":70,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:35:16.298084+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:13.099776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:13.675426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:46.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe12c00640 1 -- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efde8067660 msgr2=0x7efde8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe12c00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efde8067660 0x7efde8069b20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7efdfc0ad000 tx=0x7efdfc0a6000 comp rx=0 tx=0).stop 2024-03-24T00:35:46.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe12c00640 1 -- 172.21.15.40:0/3776566889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c15e0b0 msgr2=0x7efe0c0ca2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe12c00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c15e0b0 0x7efe0c0ca2f0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7efe0400c1b0 tx=0x7efe0400c670 comp rx=0 tx=0).stop 2024-03-24T00:35:46.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe12c00640 1 -- 172.21.15.40:0/3776566889 shutdown_connections 2024-03-24T00:35:46.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe12c00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efde8067660 0x7efde8069b20 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe12c00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe0c15e0b0 0x7efe0c0ca2f0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe12c00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe0c0cd3d0 0x7efe0c0c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe12c00640 1 --2- 172.21.15.40:0/3776566889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe0c06b150 0x7efe0c15c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe12c00640 1 -- 172.21.15.40:0/3776566889 >> 172.21.15.40:0/3776566889 conn(0x7efe0c0d1c90 msgr2=0x7efe0c0aea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:46.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe12c00640 1 -- 172.21.15.40:0/3776566889 shutdown_connections 2024-03-24T00:35:46.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.126+0000 7efe12c00640 1 -- 172.21.15.40:0/3776566889 wait complete. 2024-03-24T00:35:46.150 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:35:46.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:46.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.286+0000 7f450fc00640 1 Processor -- start 2024-03-24T00:35:46.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.286+0000 7f450fc00640 1 -- start start 2024-03-24T00:35:46.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.290+0000 7f450fc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f450813ec80 0x7f45081414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.290+0000 7f450fc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4508070e00 con 0x7f45081419f0 2024-03-24T00:35:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.290+0000 7f450fc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45080cee20 con 0x7f45080ce800 2024-03-24T00:35:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.290+0000 7f450fc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4508141e50 con 0x7f450813ec80 2024-03-24T00:35:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.290+0000 7f450d200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f450813ec80 0x7f45081414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.290+0000 7f450d200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f450813ec80 0x7f45081414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:36768/0 (socket says 172.21.15.40:36768) 2024-03-24T00:35:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.290+0000 7f4507e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f45081419f0 0x7f45080ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45778/0 (socket says 172.21.15.40:45778) 2024-03-24T00:35:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.290+0000 7f450d200640 1 -- 172.21.15.40:0/2305389605 learned_addr learned my addr 172.21.15.40:0/2305389605 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:46.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.290+0000 7f450dc00640 1 --1- 172.21.15.40:0/2305389605 >> v1:172.21.15.53:6789/0 conn(0x7f45080ce800 0x7f4508071300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:53348/0 (socket says 172.21.15.40:53348) 2024-03-24T00:35:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.290+0000 7f450d200640 1 -- 172.21.15.40:0/2305389605 >> v1:172.21.15.53:6789/0 conn(0x7f45080ce800 legacy=0x7f4508071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.290+0000 7f450d200640 1 -- 172.21.15.40:0/2305389605 >> v1:172.21.15.40:6789/0 conn(0x7f45081419f0 legacy=0x7f45080ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.290+0000 7f450d200640 1 -- 172.21.15.40:0/2305389605 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45080cf100 con 0x7f450813ec80 2024-03-24T00:35:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.294+0000 7f450d200640 1 --2- 172.21.15.40:0/2305389605 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f450813ec80 0x7f45081414b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0098a0 tx=0x7f44fc01c580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b819840ac5f49636 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.294+0000 7f4507400640 1 -- 172.21.15.40:0/2305389605 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc0a9660 con 0x7f450813ec80 2024-03-24T00:35:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.294+0000 7f4507400640 1 -- 172.21.15.40:0/2305389605 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44fc0a9800 con 0x7f450813ec80 2024-03-24T00:35:46.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.294+0000 7f4507400640 1 -- 172.21.15.40:0/2305389605 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc0a9bd0 con 0x7f450813ec80 2024-03-24T00:35:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.294+0000 7f450fc00640 1 -- 172.21.15.40:0/2305389605 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f450813ec80 msgr2=0x7f45081414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.294+0000 7f450fc00640 1 --2- 172.21.15.40:0/2305389605 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f450813ec80 0x7f45081414b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0098a0 tx=0x7f44fc01c580 comp rx=0 tx=0).stop 2024-03-24T00:35:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.294+0000 7f450fc00640 1 -- 172.21.15.40:0/2305389605 shutdown_connections 2024-03-24T00:35:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.294+0000 7f450fc00640 1 --2- 172.21.15.40:0/2305389605 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f450813ec80 0x7f45081414b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.294+0000 7f450fc00640 1 -- 172.21.15.40:0/2305389605 >> 172.21.15.40:0/2305389605 conn(0x7f45080b17b0 msgr2=0x7f45080b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:46.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.294+0000 7f450fc00640 1 -- 172.21.15.40:0/2305389605 shutdown_connections 2024-03-24T00:35:46.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.294+0000 7f450fc00640 1 -- 172.21.15.40:0/2305389605 wait complete. 2024-03-24T00:35:46.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450fc00640 1 Processor -- start 2024-03-24T00:35:46.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450fc00640 1 -- start start 2024-03-24T00:35:46.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450fc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45080ce800 0x7f4508156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:46.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450fc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f450813ec80 0x7f4508159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:46.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450fc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f45081419f0 0x7f45081596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:46.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450fc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45080725f0 con 0x7f45080ce800 2024-03-24T00:35:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450fc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4508072470 con 0x7f450813ec80 2024-03-24T00:35:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450fc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45080b2bb0 con 0x7f45081419f0 2024-03-24T00:35:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45080ce800 0x7f4508156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f4507e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f450813ec80 0x7f4508159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450dc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f45081419f0 0x7f45081596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45080ce800 0x7f4508156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56228/0 (socket says 172.21.15.40:56228) 2024-03-24T00:35:46.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450d200640 1 -- 172.21.15.40:0/2478501958 learned_addr learned my addr 172.21.15.40:0/2478501958 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f4507e00640 1 -- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f45081419f0 msgr2=0x7f45081596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f4507e00640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f45081419f0 0x7f45081596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f4507e00640 1 -- 172.21.15.40:0/2478501958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45080ce800 msgr2=0x7f4508156a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f4507e00640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45080ce800 0x7f4508156a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f4507e00640 1 -- 172.21.15.40:0/2478501958 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45081574d0 con 0x7f450813ec80 2024-03-24T00:35:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450d200640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45080ce800 0x7f4508156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:46.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f4507e00640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f450813ec80 0x7f4508159160 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f44f400c390 tx=0x7f44f400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:46.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f4504c00640 1 -- 172.21.15.40:0/2478501958 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f40074e0 con 0x7f450813ec80 2024-03-24T00:35:46.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f4504c00640 1 -- 172.21.15.40:0/2478501958 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f4007680 con 0x7f450813ec80 2024-03-24T00:35:46.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f4504c00640 1 -- 172.21.15.40:0/2478501958 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f4004a30 con 0x7f450813ec80 2024-03-24T00:35:46.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450fc00640 1 -- 172.21.15.40:0/2478501958 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45081577b0 con 0x7f450813ec80 2024-03-24T00:35:46.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.298+0000 7f450fc00640 1 -- 172.21.15.40:0/2478501958 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4508157d70 con 0x7f450813ec80 2024-03-24T00:35:46.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.302+0000 7f450fc00640 1 -- 172.21.15.40:0/2478501958 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4508068170 con 0x7f450813ec80 2024-03-24T00:35:46.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.302+0000 7f4504c00640 1 -- 172.21.15.40:0/2478501958 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f44f401a020 con 0x7f450813ec80 2024-03-24T00:35:46.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.302+0000 7f4504c00640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44e80676b0 0x7f44e8069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:46.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.302+0000 7f4504c00640 1 -- 172.21.15.40:0/2478501958 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44f4084d00 con 0x7f450813ec80 2024-03-24T00:35:46.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.302+0000 7f450d200640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44e80676b0 0x7f44e8069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:46.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.306+0000 7f450d200640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44e80676b0 0x7f44e8069b70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f44fc01a1b0 tx=0x7f44fc0b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:46.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.306+0000 7f4504c00640 1 -- 172.21.15.40:0/2478501958 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f44f4088030 con 0x7f450813ec80 2024-03-24T00:35:46.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.506+0000 7f450fc00640 1 -- 172.21.15.40:0/2478501958 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45080b4650 con 0x7f450813ec80 2024-03-24T00:35:46.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.506+0000 7f4504c00640 1 -- 172.21.15.40:0/2478501958 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f44f40554a0 con 0x7f450813ec80 2024-03-24T00:35:46.510 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:46.510 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":73,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6062,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/4182264423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":4182264423},{"type":"v1","addr":"172.21.15.53:6836","nonce":4182264423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:35:14.689580+0000","modified":"2024-03-24T00:35:35.452176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6954,"mds_1":6951,"mds_2":6733},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6733":{"gid":6733,"name":"b","rank":2,"incarnation":71,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/334375209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":334375209},{"type":"v1","addr":"172.21.15.53:6837","nonce":334375209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6951":{"gid":6951,"name":"c","rank":1,"incarnation":69,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/221849812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":221849812},{"type":"v1","addr":"172.21.15.40:6834","nonce":221849812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6954":{"gid":6954,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/2177098794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2177098794},{"type":"v1","addr":"172.21.15.40:6835","nonce":2177098794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6954,"qdb_cluster":[6951,6954,6733]},"id":5}]} 2024-03-24T00:35:46.510 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 73 2024-03-24T00:35:46.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.510+0000 7f450fc00640 1 -- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44e80676b0 msgr2=0x7f44e8069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.510+0000 7f450fc00640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44e80676b0 0x7f44e8069b70 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f44fc01a1b0 tx=0x7f44fc0b3040 comp rx=0 tx=0).stop 2024-03-24T00:35:46.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.510+0000 7f450fc00640 1 -- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f450813ec80 msgr2=0x7f4508159160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.510+0000 7f450fc00640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f450813ec80 0x7f4508159160 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f44f400c390 tx=0x7f44f400c850 comp rx=0 tx=0).stop 2024-03-24T00:35:46.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.510+0000 7f450fc00640 1 -- 172.21.15.40:0/2478501958 shutdown_connections 2024-03-24T00:35:46.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.510+0000 7f450fc00640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44e80676b0 0x7f44e8069b70 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.510+0000 7f450fc00640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f45081419f0 0x7f45081596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.510+0000 7f450fc00640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f450813ec80 0x7f4508159160 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.510+0000 7f450fc00640 1 --2- 172.21.15.40:0/2478501958 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f45080ce800 0x7f4508156a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.510+0000 7f450fc00640 1 -- 172.21.15.40:0/2478501958 >> 172.21.15.40:0/2478501958 conn(0x7f45080b17b0 msgr2=0x7f45080b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:46.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.510+0000 7f450fc00640 1 -- 172.21.15.40:0/2478501958 shutdown_connections 2024-03-24T00:35:46.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.510+0000 7f450fc00640 1 -- 172.21.15.40:0/2478501958 wait complete. 2024-03-24T00:35:46.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:46.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.666+0000 7f2fbb200640 1 Processor -- start 2024-03-24T00:35:46.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fbb200640 1 -- start start 2024-03-24T00:35:46.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fbb200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2fbc145520 0x7f2fbc15f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:46.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fbb200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fbc15fe60 0x7f2fbc06cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:46.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fbb200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fbc0cec10 con 0x7f2fbc06d330 2024-03-24T00:35:46.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fbb200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fbc1602a0 con 0x7f2fbc145520 2024-03-24T00:35:46.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fbb200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc0ceeb0 con 0x7f2fbc15fe60 2024-03-24T00:35:46.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fb9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2fbc145520 0x7f2fbc15f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fb9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2fbc145520 0x7f2fbc15f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56238/0 (socket says 172.21.15.40:56238) 2024-03-24T00:35:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fb9400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fbc15fe60 0x7f2fbc06cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fb9e00640 1 -- 172.21.15.40:0/2582594610 learned_addr learned my addr 172.21.15.40:0/2582594610 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fb9e00640 1 -- 172.21.15.40:0/2582594610 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fbc15fe60 msgr2=0x7f2fbc06cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fb9e00640 1 --2- 172.21.15.40:0/2582594610 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fbc15fe60 0x7f2fbc06cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fb9e00640 1 -- 172.21.15.40:0/2582594610 >> v1:172.21.15.53:6789/0 conn(0x7f2fbc06d330 legacy=0x7f2fbc06d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.670+0000 7f2fb9e00640 1 -- 172.21.15.40:0/2582594610 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc0cf190 con 0x7f2fbc145520 2024-03-24T00:35:46.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fb9e00640 1 --2- 172.21.15.40:0/2582594610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fbc145520 0x7f2fbc15f920 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009870 tx=0x7f2fac01cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b24e878dba1cd88 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:46.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fb8a00640 1 -- 172.21.15.40:0/2582594610 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac0aa620 con 0x7f2fbc145520 2024-03-24T00:35:46.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fb8a00640 1 -- 172.21.15.40:0/2582594610 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac0af070 con 0x7f2fbc145520 2024-03-24T00:35:46.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fb8a00640 1 -- 172.21.15.40:0/2582594610 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac0aa9e0 con 0x7f2fbc145520 2024-03-24T00:35:46.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fbb200640 1 -- 172.21.15.40:0/2582594610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fbc145520 msgr2=0x7f2fbc15f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fbb200640 1 --2- 172.21.15.40:0/2582594610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fbc145520 0x7f2fbc15f920 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009870 tx=0x7f2fac01cf00 comp rx=0 tx=0).stop 2024-03-24T00:35:46.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fbb200640 1 -- 172.21.15.40:0/2582594610 shutdown_connections 2024-03-24T00:35:46.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fbb200640 1 --2- 172.21.15.40:0/2582594610 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fbc15fe60 0x7f2fbc06cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fbb200640 1 --2- 172.21.15.40:0/2582594610 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fbc145520 0x7f2fbc15f920 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fbb200640 1 -- 172.21.15.40:0/2582594610 >> 172.21.15.40:0/2582594610 conn(0x7f2fbc0b1740 msgr2=0x7f2fbc0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:46.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fbb200640 1 -- 172.21.15.40:0/2582594610 shutdown_connections 2024-03-24T00:35:46.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fbb200640 1 -- 172.21.15.40:0/2582594610 wait complete. 2024-03-24T00:35:46.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fbb200640 1 Processor -- start 2024-03-24T00:35:46.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.674+0000 7f2fbb200640 1 -- start start 2024-03-24T00:35:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fbb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fbc06d330 0x7f2fbc13b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fbb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fbc145520 0x7f2fbc154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fbb200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fbc15fe60 0x7f2fbc1548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fbb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc1605a0 con 0x7f2fbc06d330 2024-03-24T00:35:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fbb200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc145b10 con 0x7f2fbc145520 2024-03-24T00:35:46.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fbb200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fbc160720 con 0x7f2fbc15fe60 2024-03-24T00:35:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb9e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fbc06d330 0x7f2fbc13b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fbc145520 0x7f2fbc154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fbc145520 0x7f2fbc154380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:36354/0 (socket says 172.21.15.40:36354) 2024-03-24T00:35:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb9400640 1 -- 172.21.15.40:0/2138653023 learned_addr learned my addr 172.21.15.40:0/2138653023 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fba800640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fbc15fe60 0x7f2fbc1548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb9400640 1 -- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fbc15fe60 msgr2=0x7f2fbc1548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb9400640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fbc15fe60 0x7f2fbc1548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb9400640 1 -- 172.21.15.40:0/2138653023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fbc06d330 msgr2=0x7f2fbc13b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb9400640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fbc06d330 0x7f2fbc13b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb9400640 1 -- 172.21.15.40:0/2138653023 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc1f7740 con 0x7f2fbc145520 2024-03-24T00:35:46.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb9400640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fbc145520 0x7f2fbc154380 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800e9e0 tx=0x7f2fa800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb2000640 1 -- 172.21.15.40:0/2138653023 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8017070 con 0x7f2fbc145520 2024-03-24T00:35:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb2000640 1 -- 172.21.15.40:0/2138653023 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa800c9d0 con 0x7f2fbc145520 2024-03-24T00:35:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fbb200640 1 -- 172.21.15.40:0/2138653023 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fbc1f7a20 con 0x7f2fbc145520 2024-03-24T00:35:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fb2000640 1 -- 172.21.15.40:0/2138653023 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa800cd80 con 0x7f2fbc145520 2024-03-24T00:35:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fbb200640 1 -- 172.21.15.40:0/2138653023 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fbc1f7fe0 con 0x7f2fbc145520 2024-03-24T00:35:46.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.678+0000 7f2fbb200640 1 -- 172.21.15.40:0/2138653023 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fbc073540 con 0x7f2fbc145520 2024-03-24T00:35:46.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.682+0000 7f2fb2000640 1 -- 172.21.15.40:0/2138653023 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2fa801a070 con 0x7f2fbc145520 2024-03-24T00:35:46.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.682+0000 7f2fb2000640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f9c067660 0x7f2f9c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:46.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.682+0000 7f2fb2000640 1 -- 172.21.15.40:0/2138653023 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fa8085c70 con 0x7f2fbc145520 2024-03-24T00:35:46.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.682+0000 7f2fb9e00640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f9c067660 0x7f2f9c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:46.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.682+0000 7f2fb9e00640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f9c067660 0x7f2f9c069b20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0ae000 tx=0x7f2fac004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:46.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.686+0000 7f2fb2000640 1 -- 172.21.15.40:0/2138653023 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2fa8056290 con 0x7f2fbc145520 2024-03-24T00:35:46.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.882+0000 7f2fbb200640 1 -- 172.21.15.40:0/2138653023 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fbc0b00f0 con 0x7f2fbc145520 2024-03-24T00:35:46.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.882+0000 7f2fb2000640 1 -- 172.21.15.40:0/2138653023 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f2fa805a330 con 0x7f2fbc145520 2024-03-24T00:35:46.887 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:46.887 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":73,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6062,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/4182264423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":4182264423},{"type":"v1","addr":"172.21.15.53:6836","nonce":4182264423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:35:14.689580+0000","modified":"2024-03-24T00:35:35.452176+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":6954,"mds_1":6951,"mds_2":6733},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6733":{"gid":6733,"name":"b","rank":2,"incarnation":71,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/334375209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":334375209},{"type":"v1","addr":"172.21.15.53:6837","nonce":334375209}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6951":{"gid":6951,"name":"c","rank":1,"incarnation":69,"state":"up:active","state_seq":5,"addr":"172.21.15.40:6834/221849812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":221849812},{"type":"v1","addr":"172.21.15.40:6834","nonce":221849812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6954":{"gid":6954,"name":"a","rank":0,"incarnation":63,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/2177098794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2177098794},{"type":"v1","addr":"172.21.15.40:6835","nonce":2177098794}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6954,"qdb_cluster":[6951,6954,6733]},"id":5}]} 2024-03-24T00:35:46.887 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 73 2024-03-24T00:35:46.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.886+0000 7f2fbb200640 1 -- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f9c067660 msgr2=0x7f2f9c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.886+0000 7f2fbb200640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f9c067660 0x7f2f9c069b20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f2fac0ae000 tx=0x7f2fac004740 comp rx=0 tx=0).stop 2024-03-24T00:35:46.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.886+0000 7f2fbb200640 1 -- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fbc145520 msgr2=0x7f2fbc154380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:46.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.886+0000 7f2fbb200640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fbc145520 0x7f2fbc154380 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800e9e0 tx=0x7f2fa800eea0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.886+0000 7f2fbb200640 1 -- 172.21.15.40:0/2138653023 shutdown_connections 2024-03-24T00:35:46.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.886+0000 7f2fbb200640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f9c067660 0x7f2f9c069b20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.886+0000 7f2fbb200640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fbc15fe60 0x7f2fbc1548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.886+0000 7f2fbb200640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fbc145520 0x7f2fbc154380 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.886+0000 7f2fbb200640 1 --2- 172.21.15.40:0/2138653023 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fbc06d330 0x7f2fbc13b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:46.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.886+0000 7f2fbb200640 1 -- 172.21.15.40:0/2138653023 >> 172.21.15.40:0/2138653023 conn(0x7f2fbc0b1740 msgr2=0x7f2fbc0b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:46.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.886+0000 7f2fbb200640 1 -- 172.21.15.40:0/2138653023 shutdown_connections 2024-03-24T00:35:46.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:46.886+0000 7f2fbb200640 1 -- 172.21.15.40:0/2138653023 wait complete. 2024-03-24T00:35:46.907 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f209d000640 1 Processor -- start 2024-03-24T00:35:47.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f209d000640 1 -- start start 2024-03-24T00:35:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f209d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f209813ec80 0x7f20981414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f209d000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f20981419f0 0x7f20980ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f209d000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2098070e00 con 0x7f20980ce630 2024-03-24T00:35:47.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f209d000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2098141e50 con 0x7f20981419f0 2024-03-24T00:35:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f209d000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20980cee20 con 0x7f209813ec80 2024-03-24T00:35:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f2095400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f20981419f0 0x7f20980ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f2095e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f209813ec80 0x7f20981414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f2096800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f20980ce630 0x7f209815f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55798/0 (socket says 172.21.15.40:55798) 2024-03-24T00:35:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f2096800640 1 -- 172.21.15.40:0/1745041675 learned_addr learned my addr 172.21.15.40:0/1745041675 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f2095e00640 1 -- 172.21.15.40:0/1745041675 >> v2:172.21.15.40:3300/0 conn(0x7f20981419f0 msgr2=0x7f20980ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f2095e00640 1 --2- 172.21.15.40:0/1745041675 >> v2:172.21.15.40:3300/0 conn(0x7f20981419f0 0x7f20980ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f2095e00640 1 -- 172.21.15.40:0/1745041675 >> v1:172.21.15.40:6789/0 conn(0x7f20980ce630 legacy=0x7f209815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.106+0000 7f2095e00640 1 -- 172.21.15.40:0/1745041675 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20980cf100 con 0x7f209813ec80 2024-03-24T00:35:47.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f2095e00640 1 --2- 172.21.15.40:0/1745041675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f209813ec80 0x7f20981414b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f209806b170 tx=0x7f208801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bbb95f061f5c3e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:47.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f2094a00640 1 -- 172.21.15.40:0/1745041675 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20880aa600 con 0x7f209813ec80 2024-03-24T00:35:47.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f2094a00640 1 -- 172.21.15.40:0/1745041675 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20880af070 con 0x7f209813ec80 2024-03-24T00:35:47.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f2094a00640 1 -- 172.21.15.40:0/1745041675 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20880aa9c0 con 0x7f209813ec80 2024-03-24T00:35:47.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 -- 172.21.15.40:0/1745041675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f209813ec80 msgr2=0x7f20981414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 --2- 172.21.15.40:0/1745041675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f209813ec80 0x7f20981414b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f209806b170 tx=0x7f208801c480 comp rx=0 tx=0).stop 2024-03-24T00:35:47.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 -- 172.21.15.40:0/1745041675 shutdown_connections 2024-03-24T00:35:47.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 --2- 172.21.15.40:0/1745041675 >> v2:172.21.15.40:3300/0 conn(0x7f20981419f0 0x7f20980ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 --2- 172.21.15.40:0/1745041675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f209813ec80 0x7f20981414b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 -- 172.21.15.40:0/1745041675 >> 172.21.15.40:0/1745041675 conn(0x7f20980b17b0 msgr2=0x7f20980b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:47.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 -- 172.21.15.40:0/1745041675 shutdown_connections 2024-03-24T00:35:47.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 -- 172.21.15.40:0/1745041675 wait complete. 2024-03-24T00:35:47.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 Processor -- start 2024-03-24T00:35:47.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 -- start start 2024-03-24T00:35:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20980ce630 0x7f2098151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f209813ec80 0x7f209806e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20981419f0 0x7f209814c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2098168ec0 con 0x7f20981419f0 2024-03-24T00:35:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2098168d40 con 0x7f20980ce630 2024-03-24T00:35:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f209d000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2098169040 con 0x7f209813ec80 2024-03-24T00:35:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f2096800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20981419f0 0x7f209814c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f2096800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20981419f0 0x7f209814c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56910/0 (socket says 172.21.15.40:56910) 2024-03-24T00:35:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f2096800640 1 -- 172.21.15.40:0/3802070237 learned_addr learned my addr 172.21.15.40:0/3802070237 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f2095e00640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20980ce630 0x7f2098151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.114+0000 7f2095400640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f209813ec80 0x7f209806e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f2096800640 1 -- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f209813ec80 msgr2=0x7f209806e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f2096800640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f209813ec80 0x7f209806e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f2096800640 1 -- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20980ce630 msgr2=0x7f2098151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f2096800640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20980ce630 0x7f2098151070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f2096800640 1 -- 172.21.15.40:0/3802070237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209814ca20 con 0x7f20981419f0 2024-03-24T00:35:47.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f2096800640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20981419f0 0x7f209814c280 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f209000cbc0 tx=0x7f209000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:47.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f208e000640 1 -- 172.21.15.40:0/3802070237 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090017070 con 0x7f20981419f0 2024-03-24T00:35:47.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f208e000640 1 -- 172.21.15.40:0/3802070237 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209000ad40 con 0x7f20981419f0 2024-03-24T00:35:47.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f209d000640 1 -- 172.21.15.40:0/3802070237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20981502b0 con 0x7f20981419f0 2024-03-24T00:35:47.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f208e000640 1 -- 172.21.15.40:0/3802070237 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090012600 con 0x7f20981419f0 2024-03-24T00:35:47.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f209d000640 1 -- 172.21.15.40:0/3802070237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2098150870 con 0x7f20981419f0 2024-03-24T00:35:47.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f209d000640 1 -- 172.21.15.40:0/3802070237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2098068170 con 0x7f20981419f0 2024-03-24T00:35:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.118+0000 7f208e000640 1 -- 172.21.15.40:0/3802070237 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f209001a070 con 0x7f20981419f0 2024-03-24T00:35:47.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.122+0000 7f208e000640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2078067660 0x7f2078069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.122+0000 7f208e000640 1 -- 172.21.15.40:0/3802070237 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20900854d0 con 0x7f20981419f0 2024-03-24T00:35:47.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.122+0000 7f2095e00640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2078067660 0x7f2078069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.122+0000 7f2095e00640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2078067660 0x7f2078069b20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f20880ae000 tx=0x7f20880a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:47.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.126+0000 7f208e000640 1 -- 172.21.15.40:0/3802070237 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2090055af0 con 0x7f20981419f0 2024-03-24T00:35:47.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 -- 172.21.15.40:0/3802070237 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2098071fe0 con 0x7f20981419f0 2024-03-24T00:35:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f208e000640 1 -- 172.21.15.40:0/3802070237 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16608 (secure 0 0 0) 0x7f2090059b90 con 0x7f20981419f0 2024-03-24T00:35:47.293 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:47.293 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":70,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:35:16.298084+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:35:13.099776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-03-24T00:35:13.675426+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:47.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 -- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2078067660 msgr2=0x7f2078069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2078067660 0x7f2078069b20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f20880ae000 tx=0x7f20880a7000 comp rx=0 tx=0).stop 2024-03-24T00:35:47.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 -- 172.21.15.40:0/3802070237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20981419f0 msgr2=0x7f209814c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20981419f0 0x7f209814c280 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f209000cbc0 tx=0x7f209000a430 comp rx=0 tx=0).stop 2024-03-24T00:35:47.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 -- 172.21.15.40:0/3802070237 shutdown_connections 2024-03-24T00:35:47.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2078067660 0x7f2078069b20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f20981419f0 0x7f209814c280 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f209813ec80 0x7f209806e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 --2- 172.21.15.40:0/3802070237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20980ce630 0x7f2098151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 -- 172.21.15.40:0/3802070237 >> 172.21.15.40:0/3802070237 conn(0x7f20980b17b0 msgr2=0x7f209813dce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:47.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 -- 172.21.15.40:0/3802070237 shutdown_connections 2024-03-24T00:35:47.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.290+0000 7f209d000640 1 -- 172.21.15.40:0/3802070237 wait complete. 2024-03-24T00:35:47.316 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:35:47.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.454+0000 7f703f600640 1 Processor -- start 2024-03-24T00:35:47.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.454+0000 7f703f600640 1 -- start start 2024-03-24T00:35:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f703f600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703813ec80 0x7f70381414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f703f600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70381419f0 0x7f70380ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f703f600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7038070e00 con 0x7f70380ce630 2024-03-24T00:35:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f703f600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7038141e50 con 0x7f70381419f0 2024-03-24T00:35:47.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f703f600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70380cee20 con 0x7f703813ec80 2024-03-24T00:35:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f703d600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f70380ce630 0x7f703815f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55808/0 (socket says 172.21.15.40:55808) 2024-03-24T00:35:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f703d600640 1 -- 172.21.15.40:0/680269145 learned_addr learned my addr 172.21.15.40:0/680269145 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f703cc00640 1 --2- 172.21.15.40:0/680269145 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703813ec80 0x7f70381414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f702fe00640 1 --2- 172.21.15.40:0/680269145 >> v2:172.21.15.53:3300/0 conn(0x7f70381419f0 0x7f70380ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f703cc00640 1 -- 172.21.15.40:0/680269145 >> v2:172.21.15.53:3300/0 conn(0x7f70381419f0 msgr2=0x7f70380ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f703cc00640 1 --2- 172.21.15.40:0/680269145 >> v2:172.21.15.53:3300/0 conn(0x7f70381419f0 0x7f70380ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f703cc00640 1 -- 172.21.15.40:0/680269145 >> v1:172.21.15.40:6789/0 conn(0x7f70380ce630 legacy=0x7f703815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f703cc00640 1 -- 172.21.15.40:0/680269145 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380cf100 con 0x7f703813ec80 2024-03-24T00:35:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f7037e00640 1 -- 172.21.15.40:0/680269145 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150159384 0 0) 0x7f7038070e00 con 0x7f70380ce630 2024-03-24T00:35:47.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.458+0000 7f702fe00640 1 --2- 172.21.15.40:0/680269145 >> v2:172.21.15.53:3300/0 conn(0x7f70381419f0 0x7f70380ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:47.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f703cc00640 1 --2- 172.21.15.40:0/680269145 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703813ec80 0x7f70381414b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f70280097c0 tx=0x7f702801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a03e41602d75b74f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:47.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f7037e00640 1 -- 172.21.15.40:0/680269145 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70280aa620 con 0x7f703813ec80 2024-03-24T00:35:47.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f7037e00640 1 -- 172.21.15.40:0/680269145 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70280af070 con 0x7f703813ec80 2024-03-24T00:35:47.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f7037e00640 1 -- 172.21.15.40:0/680269145 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70280aa9e0 con 0x7f703813ec80 2024-03-24T00:35:47.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f703f600640 1 -- 172.21.15.40:0/680269145 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703813ec80 msgr2=0x7f70381414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f703f600640 1 --2- 172.21.15.40:0/680269145 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703813ec80 0x7f70381414b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f70280097c0 tx=0x7f702801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f703f600640 1 -- 172.21.15.40:0/680269145 shutdown_connections 2024-03-24T00:35:47.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f703f600640 1 --2- 172.21.15.40:0/680269145 >> v2:172.21.15.53:3300/0 conn(0x7f70381419f0 0x7f70380ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f703f600640 1 --2- 172.21.15.40:0/680269145 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703813ec80 0x7f70381414b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f703f600640 1 -- 172.21.15.40:0/680269145 >> 172.21.15.40:0/680269145 conn(0x7f70380b17b0 msgr2=0x7f70380b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:47.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f703f600640 1 -- 172.21.15.40:0/680269145 shutdown_connections 2024-03-24T00:35:47.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f703f600640 1 -- 172.21.15.40:0/680269145 wait complete. 2024-03-24T00:35:47.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f703f600640 1 Processor -- start 2024-03-24T00:35:47.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.462+0000 7f703f600640 1 -- start start 2024-03-24T00:35:47.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703f600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380ce630 0x7f7038158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703f600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f703813ec80 0x7f703815eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703f600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70381419f0 0x7f7038155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703f600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7038168ec0 con 0x7f70381419f0 2024-03-24T00:35:47.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703f600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7038168d40 con 0x7f703813ec80 2024-03-24T00:35:47.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703f600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7038169040 con 0x7f70380ce630 2024-03-24T00:35:47.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70381419f0 0x7f7038155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380ce630 0x7f7038158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f702fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f703813ec80 0x7f703815eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380ce630 0x7f7038158c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:54710/0 (socket says 172.21.15.40:54710) 2024-03-24T00:35:47.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703cc00640 1 -- 172.21.15.40:0/424831373 learned_addr learned my addr 172.21.15.40:0/424831373 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:47.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703cc00640 1 -- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f703813ec80 msgr2=0x7f703815eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703cc00640 1 --2- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f703813ec80 0x7f703815eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703cc00640 1 -- 172.21.15.40:0/424831373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70381419f0 msgr2=0x7f7038155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703cc00640 1 --2- 172.21.15.40:0/424831373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70381419f0 0x7f7038155b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703cc00640 1 -- 172.21.15.40:0/424831373 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7038159db0 con 0x7f70380ce630 2024-03-24T00:35:47.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703cc00640 1 --2- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380ce630 0x7f7038158c60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f70280ac040 tx=0x7f702801ace0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:47.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f7035600640 1 -- 172.21.15.40:0/424831373 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70280aabe0 con 0x7f70380ce630 2024-03-24T00:35:47.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f7035600640 1 -- 172.21.15.40:0/424831373 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70280aada0 con 0x7f70380ce630 2024-03-24T00:35:47.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f7035600640 1 -- 172.21.15.40:0/424831373 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70280be650 con 0x7f70380ce630 2024-03-24T00:35:47.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703f600640 1 -- 172.21.15.40:0/424831373 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70381560c0 con 0x7f70380ce630 2024-03-24T00:35:47.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703f600640 1 -- 172.21.15.40:0/424831373 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7038159240 con 0x7f70380ce630 2024-03-24T00:35:47.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f703f600640 1 -- 172.21.15.40:0/424831373 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7038068170 con 0x7f70380ce630 2024-03-24T00:35:47.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.466+0000 7f7035600640 1 -- 172.21.15.40:0/424831373 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f70280bc020 con 0x7f70380ce630 2024-03-24T00:35:47.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.470+0000 7f7035600640 1 --2- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7014067660 0x7f7014069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.470+0000 7f7035600640 1 -- 172.21.15.40:0/424831373 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f702811f450 con 0x7f70380ce630 2024-03-24T00:35:47.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.470+0000 7f702fe00640 1 --2- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7014067660 0x7f7014069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.470+0000 7f702fe00640 1 --2- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7014067660 0x7f7014069b20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f703806e8f0 tx=0x7f7020009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:47.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.474+0000 7f7035600640 1 -- 172.21.15.40:0/424831373 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f70280c5030 con 0x7f70380ce630 2024-03-24T00:35:47.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.670+0000 7f703f600640 1 -- 172.21.15.40:0/424831373 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7038072b90 con 0x7f70380ce630 2024-03-24T00:35:47.675 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-03-24T00:35:47.670+0000 7fa64e600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:35:47.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.682+0000 7f7035600640 1 -- 172.21.15.40:0/424831373 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7028016090 con 0x7f70380ce630 2024-03-24T00:35:47.687 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:35:47.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.686+0000 7f703f600640 1 -- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7014067660 msgr2=0x7f7014069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.686+0000 7f703f600640 1 --2- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7014067660 0x7f7014069b20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f703806e8f0 tx=0x7f7020009190 comp rx=0 tx=0).stop 2024-03-24T00:35:47.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.686+0000 7f703f600640 1 -- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380ce630 msgr2=0x7f7038158c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.686+0000 7f703f600640 1 --2- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380ce630 0x7f7038158c60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f70280ac040 tx=0x7f702801ace0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.686+0000 7f703f600640 1 -- 172.21.15.40:0/424831373 shutdown_connections 2024-03-24T00:35:47.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.686+0000 7f703f600640 1 --2- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7014067660 0x7f7014069b20 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.686+0000 7f703f600640 1 --2- 172.21.15.40:0/424831373 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70381419f0 0x7f7038155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.686+0000 7f703f600640 1 --2- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f703813ec80 0x7f703815eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.686+0000 7f703f600640 1 --2- 172.21.15.40:0/424831373 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380ce630 0x7f7038158c60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.686+0000 7f703f600640 1 -- 172.21.15.40:0/424831373 >> 172.21.15.40:0/424831373 conn(0x7f70380b17b0 msgr2=0x7f703813f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:47.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.686+0000 7f703f600640 1 -- 172.21.15.40:0/424831373 shutdown_connections 2024-03-24T00:35:47.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.686+0000 7f703f600640 1 -- 172.21.15.40:0/424831373 wait complete. 2024-03-24T00:35:47.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:35:47.743 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:35:47.756 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:35:47.766 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:35:47.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.866+0000 7f9c39c00640 1 Processor -- start 2024-03-24T00:35:47.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.866+0000 7f9c39c00640 1 -- start start 2024-03-24T00:35:47.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c39c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9c3415ffd0 0x7f9c340ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c39c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c34071de0 con 0x7f9c340728d0 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c39c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c340cea50 con 0x7f9c3415bd00 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c39c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c340ced30 con 0x7f9c3415ffd0 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c33400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9c340728d0 0x7f9c3406cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55816/0 (socket says 172.21.15.40:55816) 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c33400640 1 -- 172.21.15.40:0/3616597140 learned_addr learned my addr 172.21.15.40:0/3616597140 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c2be00640 1 --2- 172.21.15.40:0/3616597140 >> v2:172.21.15.53:3300/0 conn(0x7f9c3415ffd0 0x7f9c340ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c32000640 1 -- 172.21.15.40:0/3616597140 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246733452 0 0) 0x7f9c34071de0 con 0x7f9c340728d0 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c32000640 1 -- 172.21.15.40:0/3616597140 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c18003530 con 0x7f9c340728d0 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c32000640 1 -- 172.21.15.40:0/3616597140 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927574106 0 0) 0x7f9c340cea50 con 0x7f9c3415bd00 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c32000640 1 -- 172.21.15.40:0/3616597140 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c34071de0 con 0x7f9c3415bd00 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c2be00640 1 -- 172.21.15.40:0/3616597140 >> v1:172.21.15.53:6789/0 conn(0x7f9c3415bd00 legacy=0x7f9c3415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c2be00640 1 -- 172.21.15.40:0/3616597140 >> v1:172.21.15.40:6789/0 conn(0x7f9c340728d0 legacy=0x7f9c3406cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.870+0000 7f9c2be00640 1 -- 172.21.15.40:0/3616597140 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c34071b00 con 0x7f9c3415ffd0 2024-03-24T00:35:47.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c2be00640 1 --2- 172.21.15.40:0/3616597140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c3415ffd0 0x7f9c340ae8e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c0097c0 tx=0x7f9c1c01c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a14fb22c6938b75f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:47.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c32000640 1 -- 172.21.15.40:0/3616597140 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0ab430 con 0x7f9c3415ffd0 2024-03-24T00:35:47.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c32000640 1 -- 172.21.15.40:0/3616597140 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c0b0070 con 0x7f9c3415ffd0 2024-03-24T00:35:47.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c32000640 1 -- 172.21.15.40:0/3616597140 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0ab7f0 con 0x7f9c3415ffd0 2024-03-24T00:35:47.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 -- 172.21.15.40:0/3616597140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c3415ffd0 msgr2=0x7f9c340ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 --2- 172.21.15.40:0/3616597140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c3415ffd0 0x7f9c340ae8e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c0097c0 tx=0x7f9c1c01c460 comp rx=0 tx=0).stop 2024-03-24T00:35:47.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 -- 172.21.15.40:0/3616597140 shutdown_connections 2024-03-24T00:35:47.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 --2- 172.21.15.40:0/3616597140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c3415ffd0 0x7f9c340ae8e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 -- 172.21.15.40:0/3616597140 >> 172.21.15.40:0/3616597140 conn(0x7f9c340b1660 msgr2=0x7f9c340b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:47.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 -- 172.21.15.40:0/3616597140 shutdown_connections 2024-03-24T00:35:47.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 -- 172.21.15.40:0/3616597140 wait complete. 2024-03-24T00:35:47.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 Processor -- start 2024-03-24T00:35:47.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 -- start start 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c340728d0 0x7f9c3415c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c3415bd00 0x7f9c340b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c3415ffd0 0x7f9c3414fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3415c0e0 con 0x7f9c3415bd00 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c340cf660 con 0x7f9c340728d0 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c39c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3416a420 con 0x7f9c3415ffd0 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c2be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c3415bd00 0x7f9c340b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c2be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c3415bd00 0x7f9c340b2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56964/0 (socket says 172.21.15.40:56964) 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c2be00640 1 -- 172.21.15.40:0/1901072490 learned_addr learned my addr 172.21.15.40:0/1901072490 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c33400640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c3415ffd0 0x7f9c3414fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.874+0000 7f9c32a00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c340728d0 0x7f9c3415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c2be00640 1 -- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c3415ffd0 msgr2=0x7f9c3414fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c2be00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c3415ffd0 0x7f9c3414fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c2be00640 1 -- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c340728d0 msgr2=0x7f9c3415c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:47.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c2be00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c340728d0 0x7f9c3415c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:47.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c2be00640 1 -- 172.21.15.40:0/1901072490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c340b3400 con 0x7f9c3415bd00 2024-03-24T00:35:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c2be00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c3415bd00 0x7f9c340b2610 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c0ad040 tx=0x7f9c1c01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c2aa00640 1 -- 172.21.15.40:0/1901072490 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0bf610 con 0x7f9c3415bd00 2024-03-24T00:35:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c2aa00640 1 -- 172.21.15.40:0/1901072490 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c0bf7d0 con 0x7f9c3415bd00 2024-03-24T00:35:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c39c00640 1 -- 172.21.15.40:0/1901072490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c340b36e0 con 0x7f9c3415bd00 2024-03-24T00:35:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c2aa00640 1 -- 172.21.15.40:0/1901072490 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0bfbc0 con 0x7f9c3415bd00 2024-03-24T00:35:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c39c00640 1 -- 172.21.15.40:0/1901072490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3415a140 con 0x7f9c3415bd00 2024-03-24T00:35:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c39c00640 1 -- 172.21.15.40:0/1901072490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c34068170 con 0x7f9c3415bd00 2024-03-24T00:35:47.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c2aa00640 1 -- 172.21.15.40:0/1901072490 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f9c1c0bd020 con 0x7f9c3415bd00 2024-03-24T00:35:47.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c2aa00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c10067690 0x7f9c10069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:47.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.878+0000 7f9c2aa00640 1 -- 172.21.15.40:0/1901072490 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9c1c0a8030 con 0x7f9c3415bd00 2024-03-24T00:35:47.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.882+0000 7f9c32a00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c10067690 0x7f9c10069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:47.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.882+0000 7f9c32a00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c10067690 0x7f9c10069b50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9c240011f0 tx=0x7f9c24009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:47.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:47.886+0000 7f9c2aa00640 1 -- 172.21.15.40:0/1901072490 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9c1c0c4060 con 0x7f9c3415bd00 2024-03-24T00:35:48.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.086+0000 7f9c39c00640 1 -- 172.21.15.40:0/1901072490 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9c3406e870 con 0x7f9c3415bd00 2024-03-24T00:35:48.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c2aa00640 1 -- 172.21.15.40:0/1901072490 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9c1c016050 con 0x7f9c3415bd00 2024-03-24T00:35:48.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c39c00640 1 -- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c10067690 msgr2=0x7f9c10069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:48.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c39c00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c10067690 0x7f9c10069b50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9c240011f0 tx=0x7f9c24009040 comp rx=0 tx=0).stop 2024-03-24T00:35:48.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c39c00640 1 -- 172.21.15.40:0/1901072490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c3415bd00 msgr2=0x7f9c340b2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:48.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c39c00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c3415bd00 0x7f9c340b2610 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c0ad040 tx=0x7f9c1c01aca0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c39c00640 1 -- 172.21.15.40:0/1901072490 shutdown_connections 2024-03-24T00:35:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c39c00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9c10067690 0x7f9c10069b50 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c39c00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c3415ffd0 0x7f9c3414fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c39c00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9c3415bd00 0x7f9c340b2610 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c39c00640 1 --2- 172.21.15.40:0/1901072490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c340728d0 0x7f9c3415c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c39c00640 1 -- 172.21.15.40:0/1901072490 >> 172.21.15.40:0/1901072490 conn(0x7f9c340b1660 msgr2=0x7f9c34151490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c39c00640 1 -- 172.21.15.40:0/1901072490 shutdown_connections 2024-03-24T00:35:48.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.678+0000 7f9c39c00640 1 -- 172.21.15.40:0/1901072490 wait complete. 2024-03-24T00:35:48.708 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:35:48.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.890+0000 7f00c3c00640 1 Processor -- start 2024-03-24T00:35:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c3c00640 1 -- start start 2024-03-24T00:35:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c3c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc15bd00 0x7f00bc0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c3c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00bc0728d0 0x7f00bc0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c3c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f00bc15f8c0 0x7f00bc15fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c3c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00bc0cf790 con 0x7f00bc15f8c0 2024-03-24T00:35:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c3c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00bc160250 con 0x7f00bc0728d0 2024-03-24T00:35:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c3c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc072d30 con 0x7f00bc15bd00 2024-03-24T00:35:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c1c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f00bc15f8c0 0x7f00bc15fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c1c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f00bc15f8c0 0x7f00bc15fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56970/0 (socket says 172.21.15.40:56970) 2024-03-24T00:35:48.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c1c00640 1 -- 172.21.15.40:0/1942336914 learned_addr learned my addr 172.21.15.40:0/1942336914 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:48.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00b3e00640 1 --2- 172.21.15.40:0/1942336914 >> v2:172.21.15.53:3300/0 conn(0x7f00bc0728d0 0x7f00bc0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:48.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c1200640 1 --2- 172.21.15.40:0/1942336914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc15bd00 0x7f00bc0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:48.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c1c00640 1 -- 172.21.15.40:0/1942336914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc15bd00 msgr2=0x7f00bc0ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:48.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c1c00640 1 --2- 172.21.15.40:0/1942336914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc15bd00 0x7f00bc0ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c1c00640 1 -- 172.21.15.40:0/1942336914 >> v2:172.21.15.53:3300/0 conn(0x7f00bc0728d0 msgr2=0x7f00bc0cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:48.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c1c00640 1 --2- 172.21.15.40:0/1942336914 >> v2:172.21.15.53:3300/0 conn(0x7f00bc0728d0 0x7f00bc0cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.894+0000 7f00c1c00640 1 -- 172.21.15.40:0/1942336914 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00bc071de0 con 0x7f00bc15f8c0 2024-03-24T00:35:48.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00c1c00640 1 --2- 172.21.15.40:0/1942336914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00bc15f8c0 0x7f00bc15fd10 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f00bc06b7f0 tx=0x7f00b401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452c31691c1050e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:48.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00bbe00640 1 -- 172.21.15.40:0/1942336914 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40a5840 con 0x7f00bc15f8c0 2024-03-24T00:35:48.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00bbe00640 1 -- 172.21.15.40:0/1942336914 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00b40a59e0 con 0x7f00bc15f8c0 2024-03-24T00:35:48.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00bbe00640 1 -- 172.21.15.40:0/1942336914 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b40a5db0 con 0x7f00bc15f8c0 2024-03-24T00:35:48.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00c3c00640 1 -- 172.21.15.40:0/1942336914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00bc15f8c0 msgr2=0x7f00bc15fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:48.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00c3c00640 1 --2- 172.21.15.40:0/1942336914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00bc15f8c0 0x7f00bc15fd10 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f00bc06b7f0 tx=0x7f00b401c9c0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00c3c00640 1 -- 172.21.15.40:0/1942336914 shutdown_connections 2024-03-24T00:35:48.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00c3c00640 1 --2- 172.21.15.40:0/1942336914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00bc15f8c0 0x7f00bc15fd10 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00c3c00640 1 --2- 172.21.15.40:0/1942336914 >> v2:172.21.15.53:3300/0 conn(0x7f00bc0728d0 0x7f00bc0cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00c3c00640 1 --2- 172.21.15.40:0/1942336914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc15bd00 0x7f00bc0ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00c3c00640 1 -- 172.21.15.40:0/1942336914 >> 172.21.15.40:0/1942336914 conn(0x7f00bc0b1660 msgr2=0x7f00bc0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:48.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00c3c00640 1 -- 172.21.15.40:0/1942336914 shutdown_connections 2024-03-24T00:35:48.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00c3c00640 1 -- 172.21.15.40:0/1942336914 wait complete. 2024-03-24T00:35:48.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00c3c00640 1 Processor -- start 2024-03-24T00:35:48.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.898+0000 7f00c3c00640 1 -- start start 2024-03-24T00:35:48.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c3c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00bc0728d0 0x7f00bc1556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:48.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c3c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc15bd00 0x7f00bc151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:48.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c3c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00bc15f8c0 0x7f00bc1f9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:48.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c3c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc06d320 con 0x7f00bc0728d0 2024-03-24T00:35:48.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c3c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc15c0e0 con 0x7f00bc15f8c0 2024-03-24T00:35:48.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c3c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00bc06d4a0 con 0x7f00bc15bd00 2024-03-24T00:35:48.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00bc0728d0 0x7f00bc1556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:48.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00bc0728d0 0x7f00bc1556c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:56982/0 (socket says 172.21.15.40:56982) 2024-03-24T00:35:48.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c1200640 1 -- 172.21.15.40:0/4236319735 learned_addr learned my addr 172.21.15.40:0/4236319735 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:48.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00b3e00640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc15bd00 0x7f00bc151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:48.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c1c00640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00bc15f8c0 0x7f00bc1f9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:48.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c1200640 1 -- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc15bd00 msgr2=0x7f00bc151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:48.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c1200640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc15bd00 0x7f00bc151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c1200640 1 -- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00bc15f8c0 msgr2=0x7f00bc1f9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:48.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c1200640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00bc15f8c0 0x7f00bc1f9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:48.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c1200640 1 -- 172.21.15.40:0/4236319735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00bc0cf400 con 0x7f00bc0728d0 2024-03-24T00:35:48.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c1c00640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00bc15f8c0 0x7f00bc1f9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:48.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c1200640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00bc0728d0 0x7f00bc1556c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00e930 tx=0x7f00ac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:48.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00b9600640 1 -- 172.21.15.40:0/4236319735 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ac017070 con 0x7f00bc0728d0 2024-03-24T00:35:48.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00b9600640 1 -- 172.21.15.40:0/4236319735 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ac00ca00 con 0x7f00bc0728d0 2024-03-24T00:35:48.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c3c00640 1 -- 172.21.15.40:0/4236319735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00bc15ae50 con 0x7f00bc0728d0 2024-03-24T00:35:48.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00b9600640 1 -- 172.21.15.40:0/4236319735 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ac00cdb0 con 0x7f00bc0728d0 2024-03-24T00:35:48.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c3c00640 1 -- 172.21.15.40:0/4236319735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00bc15a090 con 0x7f00bc0728d0 2024-03-24T00:35:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00c3c00640 1 -- 172.21.15.40:0/4236319735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00bc154d70 con 0x7f00bc0728d0 2024-03-24T00:35:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.902+0000 7f00b9600640 1 -- 172.21.15.40:0/4236319735 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f00ac01a070 con 0x7f00bc0728d0 2024-03-24T00:35:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.906+0000 7f00b9600640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0098067660 0x7f0098069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.906+0000 7f00b9600640 1 -- 172.21.15.40:0/4236319735 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6025+0+0 (secure 0 0 0) 0x7f00ac085910 con 0x7f00bc0728d0 2024-03-24T00:35:48.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.906+0000 7f00b3e00640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0098067660 0x7f0098069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:48.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.906+0000 7f00b3e00640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0098067660 0x7f0098069b20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f00bc1f9ee0 tx=0x7f00a4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:48.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:48.910+0000 7f00b9600640 1 -- 172.21.15.40:0/4236319735 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f00ac055eb0 con 0x7f00bc0728d0 2024-03-24T00:35:49.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.078+0000 7f00c3c00640 1 -- 172.21.15.40:0/4236319735 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f00bc068170 con 0x7f00bc0728d0 2024-03-24T00:35:49.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.686+0000 7f00b9600640 1 -- 172.21.15.40:0/4236319735 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v73) v1 ==== 181+0+0 (secure 0 0 0) 0x7f00ac059f50 con 0x7f00bc0728d0 2024-03-24T00:35:49.690 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:35:49.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.690+0000 7f00c3c00640 1 -- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0098067660 msgr2=0x7f0098069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:49.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.690+0000 7f00c3c00640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0098067660 0x7f0098069b20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f00bc1f9ee0 tx=0x7f00a4009190 comp rx=0 tx=0).stop 2024-03-24T00:35:49.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.690+0000 7f00c3c00640 1 -- 172.21.15.40:0/4236319735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00bc0728d0 msgr2=0x7f00bc1556c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:49.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.690+0000 7f00c3c00640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00bc0728d0 0x7f00bc1556c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00e930 tx=0x7f00ac00edf0 comp rx=0 tx=0).stop 2024-03-24T00:35:49.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.690+0000 7f00c3c00640 1 -- 172.21.15.40:0/4236319735 shutdown_connections 2024-03-24T00:35:49.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.690+0000 7f00c3c00640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0098067660 0x7f0098069b20 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:49.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.690+0000 7f00c3c00640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00bc15f8c0 0x7f00bc1f9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:49.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.690+0000 7f00c3c00640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc15bd00 0x7f00bc151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:49.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.690+0000 7f00c3c00640 1 --2- 172.21.15.40:0/4236319735 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f00bc0728d0 0x7f00bc1556c0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:49.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.690+0000 7f00c3c00640 1 -- 172.21.15.40:0/4236319735 >> 172.21.15.40:0/4236319735 conn(0x7f00bc0b1660 msgr2=0x7f00bc0ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:49.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.690+0000 7f00c3c00640 1 -- 172.21.15.40:0/4236319735 shutdown_connections 2024-03-24T00:35:49.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.690+0000 7f00c3c00640 1 -- 172.21.15.40:0/4236319735 wait complete. 2024-03-24T00:35:49.718 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:35:49.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafe7000640 1 Processor -- start 2024-03-24T00:35:49.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafe7000640 1 -- start start 2024-03-24T00:35:49.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafe7000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafe00b2360 0x7fafe00b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:49.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafe7000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fafe00b4330 0x7fafe00ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:49.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafe7000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe00cf090 con 0x7fafe00cea00 2024-03-24T00:35:49.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafe7000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe00d0f60 con 0x7fafe00b4330 2024-03-24T00:35:49.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafe7000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe00cee30 con 0x7fafe00b2360 2024-03-24T00:35:49.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafe5000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fafe00cea00 0x7fafe006b150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55830/0 (socket says 172.21.15.40:55830) 2024-03-24T00:35:49.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafe5000640 1 -- 172.21.15.40:0/345535914 learned_addr learned my addr 172.21.15.40:0/345535914 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:49.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdfe00640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafe00b2360 0x7fafe00b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafd7e00640 1 --2- 172.21.15.40:0/345535914 >> v2:172.21.15.53:3300/0 conn(0x7fafe00b4330 0x7fafe00ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184031052 0 0) 0x7fafe00cf090 con 0x7fafe00cea00 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc4003530 con 0x7fafe00cea00 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 188464791 0 0) 0x7fafc4003530 con 0x7fafe00cea00 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafe00b2360 msgr2=0x7fafe00b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafe00b2360 0x7fafe00b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 >> v2:172.21.15.53:3300/0 conn(0x7fafe00b4330 msgr2=0x7fafe00ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 --2- 172.21.15.40:0/345535914 >> v2:172.21.15.53:3300/0 conn(0x7fafe00b4330 0x7fafe00ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe00cf330 con 0x7fafe00cea00 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fafd8001a20 con 0x7fafe00cea00 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 >> v1:172.21.15.40:6789/0 conn(0x7fafe00cea00 legacy=0x7fafe006b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafe00b2360 0x7fafc40039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc4003f90 0x7fafc4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:49.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc4004d50 0x7fafc40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe00d0f60 con 0x7fafc4003f90 2024-03-24T00:35:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe00cee30 con 0x7fafc4004d50 2024-03-24T00:35:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc400bb00 con 0x7fafe00b2360 2024-03-24T00:35:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafd7e00640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc4003f90 0x7fafc4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafdfe00640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafe00b2360 0x7fafc40039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.862+0000 7fafe5000640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc4004d50 0x7fafc40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafd7e00640 1 -- 172.21.15.40:0/345535914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafe00b2360 msgr2=0x7fafc40039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafd7e00640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafe00b2360 0x7fafc40039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafd7e00640 1 -- 172.21.15.40:0/345535914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc4004d50 msgr2=0x7fafc40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafd7e00640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc4004d50 0x7fafc40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafd7e00640 1 -- 172.21.15.40:0/345535914 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc4003530 con 0x7fafc4003f90 2024-03-24T00:35:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafe5000640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafc4004d50 0x7fafc40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:49.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafd7e00640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc4003f90 0x7fafc4004810 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009ae0 tx=0x7fafc801ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec05bc902929f5dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:49.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc80b2070 con 0x7fafc4003f90 2024-03-24T00:35:49.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc80a5650 con 0x7fafc4003f90 2024-03-24T00:35:49.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafdf400640 1 -- 172.21.15.40:0/345535914 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc80096b0 con 0x7fafc4003f90 2024-03-24T00:35:49.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafe7000640 1 -- 172.21.15.40:0/345535914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc4003f90 msgr2=0x7fafc4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:49.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafe7000640 1 --2- 172.21.15.40:0/345535914 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafc4003f90 0x7fafc4004810 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009ae0 tx=0x7fafc801ca80 comp rx=0 tx=0).stop 2024-03-24T00:35:49.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafe5000640 1 -- 172.21.15.40:0/345535914 reap_dead start 2024-03-24T00:35:49.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafe7000640 1 -- 172.21.15.40:0/345535914 shutdown_connections 2024-03-24T00:35:49.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafe7000640 1 -- 172.21.15.40:0/345535914 >> 172.21.15.40:0/345535914 conn(0x7fafe00b3810 msgr2=0x7fafe00b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:49.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafe7000640 1 -- 172.21.15.40:0/345535914 shutdown_connections 2024-03-24T00:35:49.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.866+0000 7fafe7000640 1 -- 172.21.15.40:0/345535914 wait complete. 2024-03-24T00:35:49.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe7000640 1 Processor -- start 2024-03-24T00:35:49.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe7000640 1 -- start start 2024-03-24T00:35:49.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe7000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc4003f90 0x7fafe015d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:49.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe7000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafe00c57a0 0x7fafe0165c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe7000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafe00b6c20 0x7fafe00b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe7000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe00d1950 con 0x7fafe00b6c20 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe7000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe00d17d0 con 0x7fafe00c57a0 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe7000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe0168e00 con 0x7fafc4003f90 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafe00b6c20 0x7fafe00b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafe00b6c20 0x7fafe00b70a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57000/0 (socket says 172.21.15.40:57000) 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe5000640 1 -- 172.21.15.40:0/2349977101 learned_addr learned my addr 172.21.15.40:0/2349977101 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafdfe00640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc4003f90 0x7fafe015d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafd7e00640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafe00c57a0 0x7fafe0165c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe5000640 1 -- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc4003f90 msgr2=0x7fafe015d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe5000640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc4003f90 0x7fafe015d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe5000640 1 -- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafe00c57a0 msgr2=0x7fafe0165c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe5000640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafe00c57a0 0x7fafe0165c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:49.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe5000640 1 -- 172.21.15.40:0/2349977101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe00c8100 con 0x7fafe00b6c20 2024-03-24T00:35:49.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe5000640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafe00b6c20 0x7fafe00b70a0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fafe00d1270 tx=0x7fafd8007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:49.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafdcc00640 1 -- 172.21.15.40:0/2349977101 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd800e9c0 con 0x7fafe00b6c20 2024-03-24T00:35:49.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafdcc00640 1 -- 172.21.15.40:0/2349977101 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd800eb80 con 0x7fafe00b6c20 2024-03-24T00:35:49.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafdcc00640 1 -- 172.21.15.40:0/2349977101 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd800ed70 con 0x7fafe00b6c20 2024-03-24T00:35:49.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe7000640 1 -- 172.21.15.40:0/2349977101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe00c83e0 con 0x7fafe00b6c20 2024-03-24T00:35:49.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.870+0000 7fafe7000640 1 -- 172.21.15.40:0/2349977101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe00c7db0 con 0x7fafe00b6c20 2024-03-24T00:35:49.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.874+0000 7fafe7000640 1 -- 172.21.15.40:0/2349977101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe0068170 con 0x7fafe00b6c20 2024-03-24T00:35:49.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.874+0000 7fafdcc00640 1 -- 172.21.15.40:0/2349977101 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fafd801a020 con 0x7fafe00b6c20 2024-03-24T00:35:49.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.874+0000 7fafdcc00640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafbc067660 0x7fafbc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:49.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.874+0000 7fafdcc00640 1 -- 172.21.15.40:0/2349977101 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5587+0+0 (secure 0 0 0) 0x7fafd8085210 con 0x7fafe00b6c20 2024-03-24T00:35:49.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.874+0000 7fafdfe00640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafbc067660 0x7fafbc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:49.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.874+0000 7fafdfe00640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafbc067660 0x7fafbc069b20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009b00 tx=0x7fafd0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:49.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:49.878+0000 7fafdcc00640 1 -- 172.21.15.40:0/2349977101 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fafd8021020 con 0x7fafe00b6c20 2024-03-24T00:35:50.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.042+0000 7fafe7000640 1 -- 172.21.15.40:0/2349977101 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fafe006c2e0 con 0x7fafe00b6c20 2024-03-24T00:35:50.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.690+0000 7fafdcc00640 1 -- 172.21.15.40:0/2349977101 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v74) v1 ==== 169+0+0 (secure 0 0 0) 0x7fafd8055dd0 con 0x7fafe00b6c20 2024-03-24T00:35:50.695 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-03-24T00:35:50.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.694+0000 7fafe7000640 1 -- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafbc067660 msgr2=0x7fafbc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:50.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.694+0000 7fafe7000640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafbc067660 0x7fafbc069b20 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009b00 tx=0x7fafd0009190 comp rx=0 tx=0).stop 2024-03-24T00:35:50.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.694+0000 7fafe7000640 1 -- 172.21.15.40:0/2349977101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafe00b6c20 msgr2=0x7fafe00b70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:50.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.694+0000 7fafe7000640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafe00b6c20 0x7fafe00b70a0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fafe00d1270 tx=0x7fafd8007b00 comp rx=0 tx=0).stop 2024-03-24T00:35:50.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.694+0000 7fafe7000640 1 -- 172.21.15.40:0/2349977101 shutdown_connections 2024-03-24T00:35:50.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.694+0000 7fafe7000640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fafbc067660 0x7fafbc069b20 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:50.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.694+0000 7fafe7000640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fafe00b6c20 0x7fafe00b70a0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:50.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.694+0000 7fafe7000640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafe00c57a0 0x7fafe0165c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:50.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.694+0000 7fafe7000640 1 --2- 172.21.15.40:0/2349977101 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafc4003f90 0x7fafe015d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:50.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.694+0000 7fafe7000640 1 -- 172.21.15.40:0/2349977101 >> 172.21.15.40:0/2349977101 conn(0x7fafe00b3810 msgr2=0x7fafe006b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:50.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.694+0000 7fafe7000640 1 -- 172.21.15.40:0/2349977101 shutdown_connections 2024-03-24T00:35:50.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.694+0000 7fafe7000640 1 -- 172.21.15.40:0/2349977101 wait complete. 2024-03-24T00:35:50.722 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random 2024-03-24T00:35:50.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.858+0000 7f2ce1000640 1 Processor -- start 2024-03-24T00:35:50.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.862+0000 7f2ce1000640 1 -- start start 2024-03-24T00:35:50.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.862+0000 7f2ce1000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2cdc0ae8e0 0x7f2cdc0728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:50.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.862+0000 7f2ce1000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc0aed00 con 0x7f2cdc15ffd0 2024-03-24T00:35:50.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.862+0000 7f2ce1000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc071de0 con 0x7f2cdc15bd00 2024-03-24T00:35:50.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.862+0000 7f2ce1000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cdc0cea50 con 0x7f2cdc0ae8e0 2024-03-24T00:35:50.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.862+0000 7f2cda800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2cdc0ae8e0 0x7f2cdc0728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:50.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.862+0000 7f2cd3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2cdc15ffd0 0x7f2cdc06cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55842/0 (socket says 172.21.15.40:55842) 2024-03-24T00:35:50.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.862+0000 7f2cd3e00640 1 -- 172.21.15.40:0/3278136742 learned_addr learned my addr 172.21.15.40:0/3278136742 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:50.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.862+0000 7f2cda800640 1 -- 172.21.15.40:0/3278136742 >> v1:172.21.15.53:6789/0 conn(0x7f2cdc15bd00 legacy=0x7f2cdc15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:50.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.862+0000 7f2cda800640 1 -- 172.21.15.40:0/3278136742 >> v1:172.21.15.40:6789/0 conn(0x7f2cdc15ffd0 legacy=0x7f2cdc06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:50.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.862+0000 7f2cda800640 1 -- 172.21.15.40:0/3278136742 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc0ced30 con 0x7f2cdc0ae8e0 2024-03-24T00:35:50.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.862+0000 7f2cd9400640 1 -- 172.21.15.40:0/3278136742 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282188252 0 0) 0x7f2cdc0aed00 con 0x7f2cdc15ffd0 2024-03-24T00:35:50.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2cda800640 1 --2- 172.21.15.40:0/3278136742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cdc0ae8e0 0x7f2cdc0728d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009d90 tx=0x7f2cd401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acfe9d874c389e4f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:50.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2cd9400640 1 -- 172.21.15.40:0/3278136742 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd40b2070 con 0x7f2cdc0ae8e0 2024-03-24T00:35:50.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2cd9400640 1 -- 172.21.15.40:0/3278136742 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd40a5900 con 0x7f2cdc0ae8e0 2024-03-24T00:35:50.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2cd9400640 1 -- 172.21.15.40:0/3278136742 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd40a5cd0 con 0x7f2cdc0ae8e0 2024-03-24T00:35:50.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2ce1000640 1 -- 172.21.15.40:0/3278136742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cdc0ae8e0 msgr2=0x7f2cdc0728d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:50.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2ce1000640 1 --2- 172.21.15.40:0/3278136742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cdc0ae8e0 0x7f2cdc0728d0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4009d90 tx=0x7f2cd401c970 comp rx=0 tx=0).stop 2024-03-24T00:35:50.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2ce1000640 1 -- 172.21.15.40:0/3278136742 shutdown_connections 2024-03-24T00:35:50.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2ce1000640 1 --2- 172.21.15.40:0/3278136742 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cdc0ae8e0 0x7f2cdc0728d0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:50.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2ce1000640 1 -- 172.21.15.40:0/3278136742 >> 172.21.15.40:0/3278136742 conn(0x7f2cdc0b1660 msgr2=0x7f2cdc0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:50.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2ce1000640 1 -- 172.21.15.40:0/3278136742 shutdown_connections 2024-03-24T00:35:50.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2ce1000640 1 -- 172.21.15.40:0/3278136742 wait complete. 2024-03-24T00:35:50.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2ce1000640 1 Processor -- start 2024-03-24T00:35:50.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.866+0000 7f2ce1000640 1 -- start start 2024-03-24T00:35:50.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2ce1000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cdc0ae8e0 0x7f2cdc15ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2ce1000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cdc15bd00 0x7f2cdc14b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2ce1000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2cdc15ffd0 0x7f2cdc14ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2ce1000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc15c0e0 con 0x7f2cdc15bd00 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2ce1000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc0cf660 con 0x7f2cdc0ae8e0 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2ce1000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc16a420 con 0x7f2cdc15ffd0 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cdc15bd00 0x7f2cdc14b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cdc15bd00 0x7f2cdc14b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57022/0 (socket says 172.21.15.40:57022) 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd3e00640 1 -- 172.21.15.40:0/1823319396 learned_addr learned my addr 172.21.15.40:0/1823319396 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd9e00640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cdc0ae8e0 0x7f2cdc15ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cda800640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2cdc15ffd0 0x7f2cdc14ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd3e00640 1 -- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2cdc15ffd0 msgr2=0x7f2cdc14ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd3e00640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2cdc15ffd0 0x7f2cdc14ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:50.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd3e00640 1 -- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cdc0ae8e0 msgr2=0x7f2cdc15ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:50.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd3e00640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cdc0ae8e0 0x7f2cdc15ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:50.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd3e00640 1 -- 172.21.15.40:0/1823319396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc154d70 con 0x7f2cdc15bd00 2024-03-24T00:35:50.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cda800640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2cdc15ffd0 0x7f2cdc14ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:50.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd3e00640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cdc15bd00 0x7f2cdc14b370 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400c440 tx=0x7f2cc400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:50.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd2000640 1 -- 172.21.15.40:0/1823319396 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40074e0 con 0x7f2cdc15bd00 2024-03-24T00:35:50.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd2000640 1 -- 172.21.15.40:0/1823319396 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc40047f0 con 0x7f2cdc15bd00 2024-03-24T00:35:50.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd2000640 1 -- 172.21.15.40:0/1823319396 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4007680 con 0x7f2cdc15bd00 2024-03-24T00:35:50.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2ce1000640 1 -- 172.21.15.40:0/1823319396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc155050 con 0x7f2cdc15bd00 2024-03-24T00:35:50.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2ce1000640 1 -- 172.21.15.40:0/1823319396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc15d0d0 con 0x7f2cdc15bd00 2024-03-24T00:35:50.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2ce1000640 1 -- 172.21.15.40:0/1823319396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cdc15d2d0 con 0x7f2cdc15bd00 2024-03-24T00:35:50.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.870+0000 7f2cd2000640 1 -- 172.21.15.40:0/1823319396 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2cc4011070 con 0x7f2cdc15bd00 2024-03-24T00:35:50.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.874+0000 7f2cd2000640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2cb8067660 0x7f2cb8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:50.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.874+0000 7f2cd2000640 1 -- 172.21.15.40:0/1823319396 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f2cc4085220 con 0x7f2cdc15bd00 2024-03-24T00:35:50.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.874+0000 7f2cd9e00640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2cb8067660 0x7f2cb8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:50.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.874+0000 7f2cd9e00640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2cb8067660 0x7f2cb8069b20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc150bc0 tx=0x7f2ccc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:50.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:50.878+0000 7f2cd2000640 1 -- 172.21.15.40:0/1823319396 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2cdc15d2d0 con 0x7f2cdc15bd00 2024-03-24T00:35:51.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.038+0000 7f2ce1000640 1 -- 172.21.15.40:0/1823319396 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"} v 0) v1 -- 0x7f2cdc068170 con 0x7f2cdc15bd00 2024-03-24T00:35:51.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.042+0000 7f2cd2000640 1 -- 172.21.15.40:0/1823319396 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"}]=0 v39) v1 ==== 110+0+0 (secure 0 0 0) 0x7f2cc4056050 con 0x7f2cdc15bd00 2024-03-24T00:35:51.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.046+0000 7f2ce1000640 1 -- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2cb8067660 msgr2=0x7f2cb8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.046+0000 7f2ce1000640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2cb8067660 0x7f2cb8069b20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc150bc0 tx=0x7f2ccc009040 comp rx=0 tx=0).stop 2024-03-24T00:35:51.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.046+0000 7f2ce1000640 1 -- 172.21.15.40:0/1823319396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cdc15bd00 msgr2=0x7f2cdc14b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.046+0000 7f2ce1000640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cdc15bd00 0x7f2cdc14b370 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400c440 tx=0x7f2cc400c900 comp rx=0 tx=0).stop 2024-03-24T00:35:51.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.046+0000 7f2ce1000640 1 -- 172.21.15.40:0/1823319396 shutdown_connections 2024-03-24T00:35:51.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.046+0000 7f2ce1000640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2cb8067660 0x7f2cb8069b20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.046+0000 7f2ce1000640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2cdc15ffd0 0x7f2cdc14ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.046+0000 7f2ce1000640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2cdc15bd00 0x7f2cdc14b370 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.046+0000 7f2ce1000640 1 --2- 172.21.15.40:0/1823319396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cdc0ae8e0 0x7f2cdc15ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.046+0000 7f2ce1000640 1 -- 172.21.15.40:0/1823319396 >> 172.21.15.40:0/1823319396 conn(0x7f2cdc0b1660 msgr2=0x7f2cdc0cf970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:51.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.046+0000 7f2ce1000640 1 -- 172.21.15.40:0/1823319396 shutdown_connections 2024-03-24T00:35:51.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.046+0000 7f2ce1000640 1 -- 172.21.15.40:0/1823319396 wait complete. 2024-03-24T00:35:51.069 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_distributed 2024-03-24T00:35:51.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.206+0000 7f4cd9000640 1 Processor -- start 2024-03-24T00:35:51.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.206+0000 7f4cd9000640 1 -- start start 2024-03-24T00:35:51.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.210+0000 7f4cd9000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4cd415ffd0 0x7f4cd40ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.210+0000 7f4cd9000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd4071de0 con 0x7f4cd40728d0 2024-03-24T00:35:51.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.210+0000 7f4cd9000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd40cea50 con 0x7f4cd415bd00 2024-03-24T00:35:51.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.210+0000 7f4cd9000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd40ced30 con 0x7f4cd415ffd0 2024-03-24T00:35:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.210+0000 7f4cc9400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f4cd415ffd0 0x7f4cd40ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.210+0000 7f4cd2800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4cd40728d0 0x7f4cd406cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55858/0 (socket says 172.21.15.40:55858) 2024-03-24T00:35:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.210+0000 7f4cd2800640 1 -- 172.21.15.40:0/109071666 learned_addr learned my addr 172.21.15.40:0/109071666 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.210+0000 7f4cc9400640 1 -- 172.21.15.40:0/109071666 >> v1:172.21.15.53:6789/0 conn(0x7f4cd415bd00 legacy=0x7f4cd415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.210+0000 7f4cc9400640 1 -- 172.21.15.40:0/109071666 >> v1:172.21.15.40:6789/0 conn(0x7f4cd40728d0 legacy=0x7f4cd406cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.210+0000 7f4cc9400640 1 -- 172.21.15.40:0/109071666 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd4071b00 con 0x7f4cd415ffd0 2024-03-24T00:35:51.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cc9400640 1 --2- 172.21.15.40:0/109071666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd415ffd0 0x7f4cd40ae8e0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0097c0 tx=0x7f4cbc01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7592b74c1036a0d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:51.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd1400640 1 -- 172.21.15.40:0/109071666 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc0ab430 con 0x7f4cd415ffd0 2024-03-24T00:35:51.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd1400640 1 -- 172.21.15.40:0/109071666 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cbc0b0070 con 0x7f4cd415ffd0 2024-03-24T00:35:51.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd1400640 1 -- 172.21.15.40:0/109071666 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc0ab7f0 con 0x7f4cd415ffd0 2024-03-24T00:35:51.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 -- 172.21.15.40:0/109071666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd415ffd0 msgr2=0x7f4cd40ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 --2- 172.21.15.40:0/109071666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd415ffd0 0x7f4cd40ae8e0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0097c0 tx=0x7f4cbc01c460 comp rx=0 tx=0).stop 2024-03-24T00:35:51.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 -- 172.21.15.40:0/109071666 shutdown_connections 2024-03-24T00:35:51.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 --2- 172.21.15.40:0/109071666 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd415ffd0 0x7f4cd40ae8e0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 -- 172.21.15.40:0/109071666 >> 172.21.15.40:0/109071666 conn(0x7f4cd40b1660 msgr2=0x7f4cd40b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:51.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 -- 172.21.15.40:0/109071666 shutdown_connections 2024-03-24T00:35:51.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 -- 172.21.15.40:0/109071666 wait complete. 2024-03-24T00:35:51.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 Processor -- start 2024-03-24T00:35:51.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 -- start start 2024-03-24T00:35:51.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd40728d0 0x7f4cd414ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd415bd00 0x7f4cd41503a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4cd415ffd0 0x7f4cd4146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd415c0e0 con 0x7f4cd40728d0 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd40cf660 con 0x7f4cd415bd00 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd9000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd416a420 con 0x7f4cd415ffd0 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd40728d0 0x7f4cd414ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd40728d0 0x7f4cd414ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57042/0 (socket says 172.21.15.40:57042) 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cd1e00640 1 -- 172.21.15.40:0/1185953086 learned_addr learned my addr 172.21.15.40:0/1185953086 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.214+0000 7f4cc9400640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd415bd00 0x7f4cd41503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4cd2800640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4cd415ffd0 0x7f4cd4146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4cd1e00640 1 -- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4cd415ffd0 msgr2=0x7f4cd4146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4cd1e00640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4cd415ffd0 0x7f4cd4146a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4cd1e00640 1 -- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd415bd00 msgr2=0x7f4cd41503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4cd1e00640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd415bd00 0x7f4cd41503a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4cd1e00640 1 -- 172.21.15.40:0/1185953086 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd414f120 con 0x7f4cd40728d0 2024-03-24T00:35:51.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4cc9400640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd415bd00 0x7f4cd41503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4cd1e00640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd40728d0 0x7f4cd414ea40 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400c420 tx=0x7f4cc400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4ccaa00640 1 -- 172.21.15.40:0/1185953086 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc40052d0 con 0x7f4cd40728d0 2024-03-24T00:35:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4ccaa00640 1 -- 172.21.15.40:0/1185953086 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc4005470 con 0x7f4cd40728d0 2024-03-24T00:35:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4ccaa00640 1 -- 172.21.15.40:0/1185953086 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc4007260 con 0x7f4cd40728d0 2024-03-24T00:35:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4cd9000640 1 -- 172.21.15.40:0/1185953086 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd414f400 con 0x7f4cd40728d0 2024-03-24T00:35:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4cd9000640 1 -- 172.21.15.40:0/1185953086 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd414f9c0 con 0x7f4cd40728d0 2024-03-24T00:35:51.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4cd9000640 1 -- 172.21.15.40:0/1185953086 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd4068170 con 0x7f4cd40728d0 2024-03-24T00:35:51.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4ccaa00640 1 -- 172.21.15.40:0/1185953086 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4cc4007400 con 0x7f4cd40728d0 2024-03-24T00:35:51.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4ccaa00640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4cb0067660 0x7f4cb0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.218+0000 7f4ccaa00640 1 -- 172.21.15.40:0/1185953086 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f4cc4084bd0 con 0x7f4cd40728d0 2024-03-24T00:35:51.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.222+0000 7f4cc9400640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4cb0067660 0x7f4cb0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.222+0000 7f4cc9400640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4cb0067660 0x7f4cb0069b20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0af000 tx=0x7f4cbc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:51.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.222+0000 7f4ccaa00640 1 -- 172.21.15.40:0/1185953086 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4cc4055a00 con 0x7f4cd40728d0 2024-03-24T00:35:51.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.390+0000 7f4cd9000640 1 -- 172.21.15.40:0/1185953086 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"} v 0) v1 -- 0x7f4cd40b2610 con 0x7f4cd40728d0 2024-03-24T00:35:51.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.394+0000 7f4ccaa00640 1 -- 172.21.15.40:0/1185953086 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"}]=0 v40) v1 ==== 115+0+0 (secure 0 0 0) 0x7f4cc4055300 con 0x7f4cd40728d0 2024-03-24T00:35:51.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.398+0000 7f4cd9000640 1 -- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4cb0067660 msgr2=0x7f4cb0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.398+0000 7f4cd9000640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4cb0067660 0x7f4cb0069b20 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0af000 tx=0x7f4cbc0a7000 comp rx=0 tx=0).stop 2024-03-24T00:35:51.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.398+0000 7f4cd9000640 1 -- 172.21.15.40:0/1185953086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd40728d0 msgr2=0x7f4cd414ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.398+0000 7f4cd9000640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd40728d0 0x7f4cd414ea40 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f4cc400c420 tx=0x7f4cc400c8e0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.398+0000 7f4cd9000640 1 -- 172.21.15.40:0/1185953086 shutdown_connections 2024-03-24T00:35:51.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.398+0000 7f4cd9000640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4cb0067660 0x7f4cb0069b20 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.398+0000 7f4cd9000640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4cd415ffd0 0x7f4cd4146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.398+0000 7f4cd9000640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd415bd00 0x7f4cd41503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.398+0000 7f4cd9000640 1 --2- 172.21.15.40:0/1185953086 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4cd40728d0 0x7f4cd414ea40 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.398+0000 7f4cd9000640 1 -- 172.21.15.40:0/1185953086 >> 172.21.15.40:0/1185953086 conn(0x7f4cd40b1660 msgr2=0x7f4cd40ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:51.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.398+0000 7f4cd9000640 1 -- 172.21.15.40:0/1185953086 shutdown_connections 2024-03-24T00:35:51.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.398+0000 7f4cd9000640 1 -- 172.21.15.40:0/1185953086 wait complete. 2024-03-24T00:35:51.419 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random_max 2024-03-24T00:35:51.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f3257200640 1 Processor -- start 2024-03-24T00:35:51.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f3257200640 1 -- start start 2024-03-24T00:35:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f3257200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f325015bb10 0x7f325006f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f3257200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f325006f570 0x7f3250066ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f3257200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3250067c60 con 0x7f3250067430 2024-03-24T00:35:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f3257200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325006f9d0 con 0x7f325015bb10 2024-03-24T00:35:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f3257200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325015ef60 con 0x7f325006f570 2024-03-24T00:35:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f324fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f325015bb10 0x7f325006f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f324fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f325015bb10 0x7f325006f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57058/0 (socket says 172.21.15.40:57058) 2024-03-24T00:35:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f324fe00640 1 -- 172.21.15.40:0/2474158543 learned_addr learned my addr 172.21.15.40:0/2474158543 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f324f400640 1 --2- 172.21.15.40:0/2474158543 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f325006f570 0x7f3250066ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f324fe00640 1 -- 172.21.15.40:0/2474158543 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f325006f570 msgr2=0x7f3250066ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:51.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f324fe00640 1 --2- 172.21.15.40:0/2474158543 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f325006f570 0x7f3250066ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f324fe00640 1 -- 172.21.15.40:0/2474158543 >> v1:172.21.15.53:6789/0 conn(0x7f3250067430 legacy=0x7f3250071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.558+0000 7f324fe00640 1 -- 172.21.15.40:0/2474158543 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325015e910 con 0x7f325015bb10 2024-03-24T00:35:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.562+0000 7f324fe00640 1 --2- 172.21.15.40:0/2474158543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f325015bb10 0x7f325006f030 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3240009870 tx=0x7f324001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec6cc55e0073ebd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.562+0000 7f324ea00640 1 -- 172.21.15.40:0/2474158543 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32400b2070 con 0x7f325015bb10 2024-03-24T00:35:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.562+0000 7f324ea00640 1 -- 172.21.15.40:0/2474158543 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32400ad640 con 0x7f325015bb10 2024-03-24T00:35:51.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.562+0000 7f324ea00640 1 -- 172.21.15.40:0/2474158543 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32400ada10 con 0x7f325015bb10 2024-03-24T00:35:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.562+0000 7f3257200640 1 -- 172.21.15.40:0/2474158543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f325015bb10 msgr2=0x7f325006f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.562+0000 7f3257200640 1 --2- 172.21.15.40:0/2474158543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f325015bb10 0x7f325006f030 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3240009870 tx=0x7f324001c560 comp rx=0 tx=0).stop 2024-03-24T00:35:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.562+0000 7f3257200640 1 -- 172.21.15.40:0/2474158543 shutdown_connections 2024-03-24T00:35:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.562+0000 7f3257200640 1 --2- 172.21.15.40:0/2474158543 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f325006f570 0x7f3250066ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.562+0000 7f3257200640 1 --2- 172.21.15.40:0/2474158543 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f325015bb10 0x7f325006f030 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.562+0000 7f3257200640 1 -- 172.21.15.40:0/2474158543 >> 172.21.15.40:0/2474158543 conn(0x7f32500b27c0 msgr2=0x7f32500b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.562+0000 7f3257200640 1 -- 172.21.15.40:0/2474158543 shutdown_connections 2024-03-24T00:35:51.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.562+0000 7f3257200640 1 -- 172.21.15.40:0/2474158543 wait complete. 2024-03-24T00:35:51.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3257200640 1 Processor -- start 2024-03-24T00:35:51.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3257200640 1 -- start start 2024-03-24T00:35:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3257200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3250067430 0x7f325015af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3257200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f325006f570 0x7f325013cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3257200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f325015bb10 0x7f325006dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3257200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325015f570 con 0x7f325006f570 2024-03-24T00:35:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3257200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325015f3f0 con 0x7f325015bb10 2024-03-24T00:35:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3257200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3250067840 con 0x7f3250067430 2024-03-24T00:35:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f324f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f325006f570 0x7f325013cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f324f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f325006f570 0x7f325013cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57072/0 (socket says 172.21.15.40:57072) 2024-03-24T00:35:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f324f400640 1 -- 172.21.15.40:0/1002089883 learned_addr learned my addr 172.21.15.40:0/1002089883 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3255200640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f325015bb10 0x7f325006dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f324fe00640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3250067430 0x7f325015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f324f400640 1 -- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3250067430 msgr2=0x7f325015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f324f400640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3250067430 0x7f325015af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f324f400640 1 -- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f325015bb10 msgr2=0x7f325006dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f324f400640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f325015bb10 0x7f325006dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f324f400640 1 -- 172.21.15.40:0/1002089883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325013da20 con 0x7f325006f570 2024-03-24T00:35:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3255200640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f325015bb10 0x7f325006dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:35:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f324f400640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f325006f570 0x7f325013cfd0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f323c00d800 tx=0x7f323c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3247e00640 1 -- 172.21.15.40:0/1002089883 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c00b6c0 con 0x7f325006f570 2024-03-24T00:35:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3247e00640 1 -- 172.21.15.40:0/1002089883 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323c019070 con 0x7f325006f570 2024-03-24T00:35:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3257200640 1 -- 172.21.15.40:0/1002089883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325013dd00 con 0x7f325006f570 2024-03-24T00:35:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3247e00640 1 -- 172.21.15.40:0/1002089883 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323c00ba60 con 0x7f325006f570 2024-03-24T00:35:51.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3257200640 1 -- 172.21.15.40:0/1002089883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3250150bb0 con 0x7f325006f570 2024-03-24T00:35:51.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.566+0000 7f3257200640 1 -- 172.21.15.40:0/1002089883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3250068170 con 0x7f325006f570 2024-03-24T00:35:51.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.570+0000 7f3247e00640 1 -- 172.21.15.40:0/1002089883 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f323c00bc00 con 0x7f325006f570 2024-03-24T00:35:51.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.570+0000 7f3247e00640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f32300676b0 0x7f3230069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.570+0000 7f3247e00640 1 -- 172.21.15.40:0/1002089883 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f323c085a90 con 0x7f325006f570 2024-03-24T00:35:51.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.570+0000 7f324fe00640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f32300676b0 0x7f3230069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.570+0000 7f324fe00640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f32300676b0 0x7f3230069b70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f32400af000 tx=0x7f3240008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:51.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.574+0000 7f3247e00640 1 -- 172.21.15.40:0/1002089883 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f323c0569d0 con 0x7f325006f570 2024-03-24T00:35:51.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.742+0000 7f3257200640 1 -- 172.21.15.40:0/1002089883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"} v 0) v1 -- 0x7f32500cfb40 con 0x7f325006f570 2024-03-24T00:35:51.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3247e00640 1 -- 172.21.15.40:0/1002089883 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"}]=0 v41) v1 ==== 114+0+0 (secure 0 0 0) 0x7f323c0562d0 con 0x7f325006f570 2024-03-24T00:35:51.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3257200640 1 -- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f32300676b0 msgr2=0x7f3230069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3257200640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f32300676b0 0x7f3230069b70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f32400af000 tx=0x7f3240008040 comp rx=0 tx=0).stop 2024-03-24T00:35:51.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3257200640 1 -- 172.21.15.40:0/1002089883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f325006f570 msgr2=0x7f325013cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3257200640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f325006f570 0x7f325013cfd0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f323c00d800 tx=0x7f323c00dcc0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3257200640 1 -- 172.21.15.40:0/1002089883 shutdown_connections 2024-03-24T00:35:51.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3257200640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f32300676b0 0x7f3230069b70 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3257200640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f325015bb10 0x7f325006dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3257200640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f325006f570 0x7f325013cfd0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3257200640 1 --2- 172.21.15.40:0/1002089883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3250067430 0x7f325015af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3257200640 1 -- 172.21.15.40:0/1002089883 >> 172.21.15.40:0/1002089883 conn(0x7f32500b27c0 msgr2=0x7f32500d0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:51.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3257200640 1 -- 172.21.15.40:0/1002089883 shutdown_connections 2024-03-24T00:35:51.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.746+0000 7f3257200640 1 -- 172.21.15.40:0/1002089883 wait complete. 2024-03-24T00:35:51.770 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:35:51.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc4800640 1 Processor -- start 2024-03-24T00:35:51.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc4800640 1 -- start start 2024-03-24T00:35:51.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc4800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbbc15bd00 0x7fbbbc0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc4800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbbbc0728d0 0x7fbbbc0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc4800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbbc072d30 con 0x7fbbbc15f8c0 2024-03-24T00:35:51.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc4800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbbc0cf790 con 0x7fbbbc0728d0 2024-03-24T00:35:51.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc4800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc071de0 con 0x7fbbbc15bd00 2024-03-24T00:35:51.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc2800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbbbc15f8c0 0x7fbbbc06cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55870/0 (socket says 172.21.15.40:55870) 2024-03-24T00:35:51.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc2800640 1 -- 172.21.15.40:0/70686440 learned_addr learned my addr 172.21.15.40:0/70686440 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:51.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc1e00640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbbc15bd00 0x7fbbbc0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc1400640 1 --2- 172.21.15.40:0/70686440 >> v2:172.21.15.53:3300/0 conn(0x7fbbbc0728d0 0x7fbbbc0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619279570 0 0) 0x7fbbbc072d30 con 0x7fbbbc15f8c0 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba4003530 con 0x7fbbbc15f8c0 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2520472803 0 0) 0x7fbba4003530 con 0x7fbbbc15f8c0 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbbc15bd00 msgr2=0x7fbbbc0ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbbc15bd00 0x7fbbbc0ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 >> v2:172.21.15.53:3300/0 conn(0x7fbbbc0728d0 msgr2=0x7fbbbc0cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 --2- 172.21.15.40:0/70686440 >> v2:172.21.15.53:3300/0 conn(0x7fbbbc0728d0 0x7fbbbc0cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc0cea50 con 0x7fbbbc15f8c0 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fbbb8001a00 con 0x7fbbbc15f8c0 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 >> v1:172.21.15.40:6789/0 conn(0x7fbbbc15f8c0 legacy=0x7fbbbc06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba40039c0 0x7fbba4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbba4004810 0x7fbba4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba40051d0 0x7fbba400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc0cf790 con 0x7fbba40051d0 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc071de0 con 0x7fbba40039c0 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba400bf10 con 0x7fbba4004810 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc1e00640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbbc15bd00 0x7fbbbc0ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc1400640 1 --2- 172.21.15.40:0/70686440 >> v2:172.21.15.53:3300/0 conn(0x7fbbbc0728d0 0x7fbbbc0cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.910+0000 7fbbc2800640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba40051d0 0x7fbba400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc1400640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbba4004810 0x7fbba4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc1e00640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba40039c0 0x7fbba4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc2800640 1 -- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbba4004810 msgr2=0x7fbba4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc2800640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbba4004810 0x7fbba4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc2800640 1 -- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba40039c0 msgr2=0x7fbba4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc2800640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba40039c0 0x7fbba4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc2800640 1 -- 172.21.15.40:0/70686440 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba4003530 con 0x7fbba40051d0 2024-03-24T00:35:51.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc1e00640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba40039c0 0x7fbba4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:51.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc2800640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba40051d0 0x7fbba400b9d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800a950 tx=0x7fbbb801d440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa371cd507b3275d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:51.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb800a4c0 con 0x7fbba40051d0 2024-03-24T00:35:51.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb80072f0 con 0x7fbba40051d0 2024-03-24T00:35:51.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc0a00640 1 -- 172.21.15.40:0/70686440 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb80076e0 con 0x7fbba40051d0 2024-03-24T00:35:51.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc4800640 1 -- 172.21.15.40:0/70686440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba40051d0 msgr2=0x7fbba400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc4800640 1 --2- 172.21.15.40:0/70686440 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbba40051d0 0x7fbba400b9d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800a950 tx=0x7fbbb801d440 comp rx=0 tx=0).stop 2024-03-24T00:35:51.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc2800640 1 -- 172.21.15.40:0/70686440 reap_dead start 2024-03-24T00:35:51.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc4800640 1 -- 172.21.15.40:0/70686440 shutdown_connections 2024-03-24T00:35:51.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc4800640 1 -- 172.21.15.40:0/70686440 >> 172.21.15.40:0/70686440 conn(0x7fbbbc0b1660 msgr2=0x7fbbbc0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:51.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc4800640 1 -- 172.21.15.40:0/70686440 shutdown_connections 2024-03-24T00:35:51.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.914+0000 7fbbc4800640 1 -- 172.21.15.40:0/70686440 wait complete. 2024-03-24T00:35:51.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc4800640 1 Processor -- start 2024-03-24T00:35:51.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc4800640 1 -- start start 2024-03-24T00:35:51.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc4800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba40051d0 0x7fbbbc15ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc4800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbbc150a40 0x7fbbbc1f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc4800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbbc1f4fd0 0x7fbbbc06d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc4800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc0720c0 con 0x7fbbbc150a40 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc4800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc071b00 con 0x7fbba40051d0 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc4800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbbc15c0e0 con 0x7fbbbc1f4fd0 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbbc150a40 0x7fbbbc1f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbbc150a40 0x7fbbbc1f2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57090/0 (socket says 172.21.15.40:57090) 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc1400640 1 -- 172.21.15.40:0/3206178648 learned_addr learned my addr 172.21.15.40:0/3206178648 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc1e00640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba40051d0 0x7fbbbc15ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc2800640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbbc1f4fd0 0x7fbbbc06d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc1400640 1 -- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbbc1f4fd0 msgr2=0x7fbbbc06d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc1400640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbbc1f4fd0 0x7fbbbc06d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc1400640 1 -- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba40051d0 msgr2=0x7fbbbc15ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc1400640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba40051d0 0x7fbbbc15ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc1400640 1 -- 172.21.15.40:0/3206178648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbbc15f380 con 0x7fbbbc150a40 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc2800640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbbc1f4fd0 0x7fbbbc06d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:35:51.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc1400640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbbc150a40 0x7fbbbc1f2380 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fbbac011930 tx=0x7fbbac011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:51.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbb6000640 1 -- 172.21.15.40:0/3206178648 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac00fb70 con 0x7fbbbc150a40 2024-03-24T00:35:51.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbb6000640 1 -- 172.21.15.40:0/3206178648 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbac00fd10 con 0x7fbbbc150a40 2024-03-24T00:35:51.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc4800640 1 -- 172.21.15.40:0/3206178648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbbc1f58d0 con 0x7fbbbc150a40 2024-03-24T00:35:51.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbb6000640 1 -- 172.21.15.40:0/3206178648 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac015660 con 0x7fbbbc150a40 2024-03-24T00:35:51.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.918+0000 7fbbc4800640 1 -- 172.21.15.40:0/3206178648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbbc1f1b80 con 0x7fbbbc150a40 2024-03-24T00:35:51.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.922+0000 7fbbc4800640 1 -- 172.21.15.40:0/3206178648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbbc068170 con 0x7fbbbc150a40 2024-03-24T00:35:51.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.922+0000 7fbbb6000640 1 -- 172.21.15.40:0/3206178648 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbbac015800 con 0x7fbbbc150a40 2024-03-24T00:35:51.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.922+0000 7fbbb6000640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbba0067690 0x7fbba0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:51.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.922+0000 7fbbb6000640 1 -- 172.21.15.40:0/3206178648 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fbbac0861e0 con 0x7fbbbc150a40 2024-03-24T00:35:51.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.922+0000 7fbbc1e00640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbba0067690 0x7fbba0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:51.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.922+0000 7fbbc1e00640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbba0067690 0x7fbba0069b50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc14e380 tx=0x7fbbb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:51.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:51.926+0000 7fbbb6000640 1 -- 172.21.15.40:0/3206178648 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbbac019030 con 0x7fbbbc150a40 2024-03-24T00:35:52.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.094+0000 7fbbc4800640 1 -- 172.21.15.40:0/3206178648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbbbc06e870 con 0x7fbbbc150a40 2024-03-24T00:35:52.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbb6000640 1 -- 172.21.15.40:0/3206178648 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v42) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbbac057010 con 0x7fbbbc150a40 2024-03-24T00:35:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbc4800640 1 -- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbba0067690 msgr2=0x7fbba0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbc4800640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbba0067690 0x7fbba0069b50 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc14e380 tx=0x7fbbb0008040 comp rx=0 tx=0).stop 2024-03-24T00:35:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbc4800640 1 -- 172.21.15.40:0/3206178648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbbc150a40 msgr2=0x7fbbbc1f2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbc4800640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbbc150a40 0x7fbbbc1f2380 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fbbac011930 tx=0x7fbbac011df0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbc4800640 1 -- 172.21.15.40:0/3206178648 shutdown_connections 2024-03-24T00:35:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbc4800640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbba0067690 0x7fbba0069b50 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbc4800640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbbc1f4fd0 0x7fbbbc06d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbc4800640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbbbc150a40 0x7fbbbc1f2380 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbc4800640 1 --2- 172.21.15.40:0/3206178648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba40051d0 0x7fbbbc15ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbc4800640 1 -- 172.21.15.40:0/3206178648 >> 172.21.15.40:0/3206178648 conn(0x7fbbbc0b1660 msgr2=0x7fbbbc156290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:52.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbc4800640 1 -- 172.21.15.40:0/3206178648 shutdown_connections 2024-03-24T00:35:52.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.098+0000 7fbbc4800640 1 -- 172.21.15.40:0/3206178648 wait complete. 2024-03-24T00:35:52.121 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override' 2024-03-24T00:35:52.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7f0de00640 1 Processor -- start 2024-03-24T00:35:52.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7f0de00640 1 -- start start 2024-03-24T00:35:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7f0de00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7f08141bc0 0x7f7f080ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7f0de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7f080ce630 0x7f7f08071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7f0de00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f08070e00 con 0x7f7f0813ec80 2024-03-24T00:35:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7f0de00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f080cee20 con 0x7f7f08141bc0 2024-03-24T00:35:52.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7f0de00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f080cf100 con 0x7f7f080ce630 2024-03-24T00:35:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7efe000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7f08141bc0 0x7f7f080ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7f06a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7f0813ec80 0x7f7f081414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55874/0 (socket says 172.21.15.40:55874) 2024-03-24T00:35:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7f06a00640 1 -- 172.21.15.40:0/3624144518 learned_addr learned my addr 172.21.15.40:0/3624144518 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7f07400640 1 --2- 172.21.15.40:0/3624144518 >> v2:172.21.15.53:3300/0 conn(0x7f7f080ce630 0x7f7f08071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7efe000640 1 -- 172.21.15.40:0/3624144518 >> v2:172.21.15.53:3300/0 conn(0x7f7f080ce630 msgr2=0x7f7f08071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7efe000640 1 --2- 172.21.15.40:0/3624144518 >> v2:172.21.15.53:3300/0 conn(0x7f7f080ce630 0x7f7f08071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7efe000640 1 -- 172.21.15.40:0/3624144518 >> v1:172.21.15.40:6789/0 conn(0x7f7f0813ec80 legacy=0x7f7f081414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7efe000640 1 -- 172.21.15.40:0/3624144518 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f080cf3e0 con 0x7f7f08141bc0 2024-03-24T00:35:52.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.258+0000 7f7f07400640 1 --2- 172.21.15.40:0/3624144518 >> v2:172.21.15.53:3300/0 conn(0x7f7f080ce630 0x7f7f08071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:35:52.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.262+0000 7f7efe000640 1 --2- 172.21.15.40:0/3624144518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f08141bc0 0x7f7f080ce0f0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f7f08067e60 tx=0x7f7ef001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25140679771ab4f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:52.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.262+0000 7f7f06000640 1 -- 172.21.15.40:0/3624144518 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef00aa600 con 0x7f7f08141bc0 2024-03-24T00:35:52.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.262+0000 7f7f06000640 1 -- 172.21.15.40:0/3624144518 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef00af070 con 0x7f7f08141bc0 2024-03-24T00:35:52.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.262+0000 7f7f06000640 1 -- 172.21.15.40:0/3624144518 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef00aa9c0 con 0x7f7f08141bc0 2024-03-24T00:35:52.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.262+0000 7f7f0de00640 1 -- 172.21.15.40:0/3624144518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f08141bc0 msgr2=0x7f7f080ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.262+0000 7f7f0de00640 1 --2- 172.21.15.40:0/3624144518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f08141bc0 0x7f7f080ce0f0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f7f08067e60 tx=0x7f7ef001c480 comp rx=0 tx=0).stop 2024-03-24T00:35:52.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.262+0000 7f7f0de00640 1 -- 172.21.15.40:0/3624144518 shutdown_connections 2024-03-24T00:35:52.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.262+0000 7f7f0de00640 1 --2- 172.21.15.40:0/3624144518 >> v2:172.21.15.53:3300/0 conn(0x7f7f080ce630 0x7f7f08071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.262+0000 7f7f0de00640 1 --2- 172.21.15.40:0/3624144518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f08141bc0 0x7f7f080ce0f0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.262+0000 7f7f0de00640 1 -- 172.21.15.40:0/3624144518 >> 172.21.15.40:0/3624144518 conn(0x7f7f080b17b0 msgr2=0x7f7f080b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:52.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.262+0000 7f7f0de00640 1 -- 172.21.15.40:0/3624144518 shutdown_connections 2024-03-24T00:35:52.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.262+0000 7f7f0de00640 1 -- 172.21.15.40:0/3624144518 wait complete. 2024-03-24T00:35:52.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7f0de00640 1 Processor -- start 2024-03-24T00:35:52.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7f0de00640 1 -- start start 2024-03-24T00:35:52.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7f0de00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7f080ce630 0x7f7f0813b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7f0de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0813ec80 0x7f7f080adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7f0de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f08141bc0 0x7f7f080b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7f0de00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f080723e0 con 0x7f7f0813ec80 2024-03-24T00:35:52.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7f0de00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f08072260 con 0x7f7f08141bc0 2024-03-24T00:35:52.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7f0de00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f08072560 con 0x7f7f080ce630 2024-03-24T00:35:52.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7efe000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0813ec80 0x7f7f080adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7efe000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0813ec80 0x7f7f080adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57110/0 (socket says 172.21.15.40:57110) 2024-03-24T00:35:52.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7efe000640 1 -- 172.21.15.40:0/2660766336 learned_addr learned my addr 172.21.15.40:0/2660766336 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:52.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7f06a00640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7f080ce630 0x7f7f0813b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7f07400640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f08141bc0 0x7f7f080b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7efe000640 1 -- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7f080ce630 msgr2=0x7f7f0813b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7efe000640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7f080ce630 0x7f7f0813b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7efe000640 1 -- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f08141bc0 msgr2=0x7f7f080b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7efe000640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f08141bc0 0x7f7f080b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7efe000640 1 -- 172.21.15.40:0/2660766336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f080ae570 con 0x7f7f0813ec80 2024-03-24T00:35:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7efe000640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0813ec80 0x7f7f080adb60 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f7ef001a0b0 tx=0x7f7ef0019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7effe00640 1 -- 172.21.15.40:0/2660766336 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef00aad90 con 0x7f7f0813ec80 2024-03-24T00:35:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7effe00640 1 -- 172.21.15.40:0/2660766336 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef00be450 con 0x7f7f0813ec80 2024-03-24T00:35:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7effe00640 1 -- 172.21.15.40:0/2660766336 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef00be840 con 0x7f7f0813ec80 2024-03-24T00:35:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7f0de00640 1 -- 172.21.15.40:0/2660766336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f080ae850 con 0x7f7f0813ec80 2024-03-24T00:35:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.266+0000 7f7f0de00640 1 -- 172.21.15.40:0/2660766336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f080aee10 con 0x7f7f0813ec80 2024-03-24T00:35:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.270+0000 7f7f0de00640 1 -- 172.21.15.40:0/2660766336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f08068170 con 0x7f7f0813ec80 2024-03-24T00:35:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.270+0000 7f7effe00640 1 -- 172.21.15.40:0/2660766336 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7ef00a9070 con 0x7f7f0813ec80 2024-03-24T00:35:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.270+0000 7f7effe00640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7ee4067660 0x7f7ee4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.270+0000 7f7effe00640 1 -- 172.21.15.40:0/2660766336 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f7ef011f8a0 con 0x7f7f0813ec80 2024-03-24T00:35:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.270+0000 7f7f06a00640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7ee4067660 0x7f7ee4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.270+0000 7f7f06a00640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7ee4067660 0x7f7ee4069b20 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8000f70 tx=0x7f7ef8007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:52.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.274+0000 7f7effe00640 1 -- 172.21.15.40:0/2660766336 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7ef0016070 con 0x7f7f0813ec80 2024-03-24T00:35:52.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.510+0000 7f7f0de00640 1 -- 172.21.15.40:0/2660766336 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override"]} v 0) v1 -- 0x7f7f080cd710 con 0x7f7f0813ec80 2024-03-24T00:35:52.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.694+0000 7f7effe00640 1 -- 172.21.15.40:0/2660766336 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override"]}]=0 v255) v1 ==== 155+0+0 (secure 0 0 0) 0x7f7ef00f0240 con 0x7f7f0813ec80 2024-03-24T00:35:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.698+0000 7f7f0de00640 1 -- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7ee4067660 msgr2=0x7f7ee4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.698+0000 7f7f0de00640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7ee4067660 0x7f7ee4069b20 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8000f70 tx=0x7f7ef8007e90 comp rx=0 tx=0).stop 2024-03-24T00:35:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.698+0000 7f7f0de00640 1 -- 172.21.15.40:0/2660766336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0813ec80 msgr2=0x7f7f080adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.698+0000 7f7f0de00640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0813ec80 0x7f7f080adb60 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f7ef001a0b0 tx=0x7f7ef0019220 comp rx=0 tx=0).stop 2024-03-24T00:35:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.698+0000 7f7f0de00640 1 -- 172.21.15.40:0/2660766336 shutdown_connections 2024-03-24T00:35:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.698+0000 7f7f0de00640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7ee4067660 0x7f7ee4069b20 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.698+0000 7f7f0de00640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f08141bc0 0x7f7f080b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.698+0000 7f7f0de00640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7f0813ec80 0x7f7f080adb60 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.698+0000 7f7f0de00640 1 --2- 172.21.15.40:0/2660766336 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7f080ce630 0x7f7f0813b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.698+0000 7f7f0de00640 1 -- 172.21.15.40:0/2660766336 >> 172.21.15.40:0/2660766336 conn(0x7f7f080b17b0 msgr2=0x7f7f080d01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.698+0000 7f7f0de00640 1 -- 172.21.15.40:0/2660766336 shutdown_connections 2024-03-24T00:35:52.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.698+0000 7f7f0de00640 1 -- 172.21.15.40:0/2660766336 wait complete. 2024-03-24T00:35:52.720 INFO:tasks.cephfs_test_runner:That an ephemeral distributed pin overrides a normal export pin. ... ok 2024-03-24T00:35:52.720 INFO:tasks.cephfs_test_runner:Starting test: test_ephemeral_pin_dist_override_after (tasks.cephfs.test_exports.TestEphemeralPins) That a conventional export pin overrides the distributed policy _after_ distributed policy is set. 2024-03-24T00:35:52.721 INFO:tasks.cephfs_test_runner:test_ephemeral_pin_dist_override_after (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:35:52.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:35:52.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.858+0000 7f78a5600640 1 Processor -- start 2024-03-24T00:35:52.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.858+0000 7f78a5600640 1 -- start start 2024-03-24T00:35:52.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.858+0000 7f78a5600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f78a0160030 0x7f78a0070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.858+0000 7f78a5600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a00cec10 con 0x7f78a0145520 2024-03-24T00:35:52.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.858+0000 7f78a5600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a00ceeb0 con 0x7f78a006de00 2024-03-24T00:35:52.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.858+0000 7f78a5600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a00cf190 con 0x7f78a0160030 2024-03-24T00:35:52.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.858+0000 7f789fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f78a0145520 0x7f78a015f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55890/0 (socket says 172.21.15.40:55890) 2024-03-24T00:35:52.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.858+0000 7f789fe00640 1 -- 172.21.15.40:0/3371557005 learned_addr learned my addr 172.21.15.40:0/3371557005 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:52.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.858+0000 7f7897400640 1 --2- 172.21.15.40:0/3371557005 >> v2:172.21.15.53:3300/0 conn(0x7f78a0160030 0x7f78a0070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.858+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300002329 0 0) 0x7f78a00cec10 con 0x7f78a0145520 2024-03-24T00:35:52.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.858+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7884003530 con 0x7f78a0145520 2024-03-24T00:35:52.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1051583694 0 0) 0x7f7884003530 con 0x7f78a0145520 2024-03-24T00:35:52.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 >> v2:172.21.15.53:3300/0 conn(0x7f78a0160030 msgr2=0x7f78a0070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:52.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 --2- 172.21.15.40:0/3371557005 >> v2:172.21.15.53:3300/0 conn(0x7f78a0160030 0x7f78a0070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 >> v1:172.21.15.53:6789/0 conn(0x7f78a006de00 legacy=0x7f78a0071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00cf470 con 0x7f78a0145520 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f7890001680 con 0x7f78a0145520 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 >> v1:172.21.15.40:6789/0 conn(0x7f78a0145520 legacy=0x7f78a015f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 --2- 172.21.15.40:0/3371557005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 0x7f7884003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 --2- 172.21.15.40:0/3371557005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884004810 0x7f7884004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 --2- 172.21.15.40:0/3371557005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78840051d0 0x7f788400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a00cf190 con 0x7f78840039c0 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788400c170 con 0x7f7884004810 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788400c370 con 0x7f78840051d0 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789fe00640 1 --2- 172.21.15.40:0/3371557005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 0x7f7884003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f7897400640 1 --2- 172.21.15.40:0/3371557005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884004810 0x7f7884004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f78a4c00640 1 --2- 172.21.15.40:0/3371557005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78840051d0 0x7f788400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789fe00640 1 -- 172.21.15.40:0/3371557005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78840051d0 msgr2=0x7f788400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789fe00640 1 --2- 172.21.15.40:0/3371557005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78840051d0 0x7f788400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789fe00640 1 -- 172.21.15.40:0/3371557005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884004810 msgr2=0x7f7884004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789fe00640 1 --2- 172.21.15.40:0/3371557005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884004810 0x7f7884004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789fe00640 1 -- 172.21.15.40:0/3371557005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7884003530 con 0x7f78840039c0 2024-03-24T00:35:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f7897400640 1 --2- 172.21.15.40:0/3371557005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7884004810 0x7f7884004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f78a4c00640 1 -- 172.21.15.40:0/3371557005 reap_dead start 2024-03-24T00:35:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789fe00640 1 --2- 172.21.15.40:0/3371557005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 0x7f7884003de0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f789000c240 tx=0x7f789001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84149516b63fef07 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.862+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890007d20 con 0x7f78840039c0 2024-03-24T00:35:52.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78900a9020 con 0x7f78840039c0 2024-03-24T00:35:52.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f789f400640 1 -- 172.21.15.40:0/3371557005 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78900a5b80 con 0x7f78840039c0 2024-03-24T00:35:52.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 -- 172.21.15.40:0/3371557005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 msgr2=0x7f7884003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 --2- 172.21.15.40:0/3371557005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 0x7f7884003de0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f789000c240 tx=0x7f789001cc80 comp rx=0 tx=0).stop 2024-03-24T00:35:52.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 -- 172.21.15.40:0/3371557005 shutdown_connections 2024-03-24T00:35:52.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 --2- 172.21.15.40:0/3371557005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 0x7f7884003de0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 -- 172.21.15.40:0/3371557005 >> 172.21.15.40:0/3371557005 conn(0x7f78a00b1740 msgr2=0x7f78a00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 -- 172.21.15.40:0/3371557005 shutdown_connections 2024-03-24T00:35:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 -- 172.21.15.40:0/3371557005 wait complete. 2024-03-24T00:35:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 Processor -- start 2024-03-24T00:35:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 -- start start 2024-03-24T00:35:52.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 0x7f78a00aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78a00c49b0 0x7f78a00b3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a00b2b40 0x7f78a01ea3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a01605a0 con 0x7f78840039c0 2024-03-24T00:35:52.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a0145b10 con 0x7f78a00c49b0 2024-03-24T00:35:52.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a5600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a0160720 con 0x7f78a00b2b40 2024-03-24T00:35:52.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f789fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 0x7f78a00aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f789fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 0x7f78a00aefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57126/0 (socket says 172.21.15.40:57126) 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f789fe00640 1 -- 172.21.15.40:0/3710564892 learned_addr learned my addr 172.21.15.40:0/3710564892 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f7897400640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78a00c49b0 0x7f78a00b3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f78a4c00640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a00b2b40 0x7f78a01ea3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f789fe00640 1 -- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a00b2b40 msgr2=0x7f78a01ea3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f789fe00640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a00b2b40 0x7f78a01ea3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f789fe00640 1 -- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78a00c49b0 msgr2=0x7f78a00b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f789fe00640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78a00c49b0 0x7f78a00b3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f789fe00640 1 -- 172.21.15.40:0/3710564892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a00c4f00 con 0x7f78840039c0 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.866+0000 7f7897400640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78a00c49b0 0x7f78a00b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.870+0000 7f789fe00640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 0x7f78a00aefc0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f789000c240 tx=0x7f7884005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.870+0000 7f789cc00640 1 -- 172.21.15.40:0/3710564892 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78900a5b80 con 0x7f78840039c0 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.870+0000 7f789cc00640 1 -- 172.21.15.40:0/3710564892 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78900a5d20 con 0x7f78840039c0 2024-03-24T00:35:52.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.870+0000 7f789cc00640 1 -- 172.21.15.40:0/3710564892 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7890004a30 con 0x7f78840039c0 2024-03-24T00:35:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.870+0000 7f78a5600640 1 -- 172.21.15.40:0/3710564892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a00c51e0 con 0x7f78840039c0 2024-03-24T00:35:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.870+0000 7f78a5600640 1 -- 172.21.15.40:0/3710564892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a00adac0 con 0x7f78840039c0 2024-03-24T00:35:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.870+0000 7f78a5600640 1 -- 172.21.15.40:0/3710564892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a00ad8e0 con 0x7f78840039c0 2024-03-24T00:35:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.870+0000 7f789cc00640 1 -- 172.21.15.40:0/3710564892 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7890004bd0 con 0x7f78840039c0 2024-03-24T00:35:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.870+0000 7f789cc00640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f787c067660 0x7f787c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.870+0000 7f789cc00640 1 -- 172.21.15.40:0/3710564892 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f789011f3a0 con 0x7f78840039c0 2024-03-24T00:35:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.870+0000 7f7897400640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f787c067660 0x7f787c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:52.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.874+0000 7f7897400640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f787c067660 0x7f787c069b20 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f78a00af500 tx=0x7f7888008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:52.874+0000 7f789cc00640 1 -- 172.21.15.40:0/3710564892 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f78a00ad8e0 con 0x7f78840039c0 2024-03-24T00:35:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.042+0000 7f78a5600640 1 -- 172.21.15.40:0/3710564892 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78a0073540 con 0x7f78840039c0 2024-03-24T00:35:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.042+0000 7f789cc00640 1 -- 172.21.15.40:0/3710564892 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) v1 ==== 74+0+13217 (secure 0 0 0) 0x7f7890016990 con 0x7f78840039c0 2024-03-24T00:35:53.046 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:53.047 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":74,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:35:50.691002+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6834/221849812":"2024-03-25T00:35:47.676258+0000","172.21.15.40:6832/221849812":"2024-03-25T00:35:47.676258+0000","172.21.15.40:6835/2177098794":"2024-03-25T00:35:47.676262+0000","172.21.15.40:6833/2177098794":"2024-03-25T00:35:47.676262+0000","172.21.15.53:6837/334375209":"2024-03-25T00:35:47.676247+0000","172.21.15.53:6834/334375209":"2024-03-25T00:35:47.676247+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:35:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.046+0000 7f78a5600640 1 -- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f787c067660 msgr2=0x7f787c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.046+0000 7f78a5600640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f787c067660 0x7f787c069b20 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f78a00af500 tx=0x7f7888008040 comp rx=0 tx=0).stop 2024-03-24T00:35:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.046+0000 7f78a5600640 1 -- 172.21.15.40:0/3710564892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 msgr2=0x7f78a00aefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.046+0000 7f78a5600640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 0x7f78a00aefc0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f789000c240 tx=0x7f7884005840 comp rx=0 tx=0).stop 2024-03-24T00:35:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.046+0000 7f78a5600640 1 -- 172.21.15.40:0/3710564892 shutdown_connections 2024-03-24T00:35:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.046+0000 7f78a5600640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f787c067660 0x7f787c069b20 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.046+0000 7f78a5600640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a00b2b40 0x7f78a01ea3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.046+0000 7f78a5600640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78a00c49b0 0x7f78a00b3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.046+0000 7f78a5600640 1 --2- 172.21.15.40:0/3710564892 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78840039c0 0x7f78a00aefc0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.046+0000 7f78a5600640 1 -- 172.21.15.40:0/3710564892 >> 172.21.15.40:0/3710564892 conn(0x7f78a00b1740 msgr2=0x7f78a00af5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:53.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.046+0000 7f78a5600640 1 -- 172.21.15.40:0/3710564892 shutdown_connections 2024-03-24T00:35:53.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.046+0000 7f78a5600640 1 -- 172.21.15.40:0/3710564892 wait complete. 2024-03-24T00:35:53.068 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:35:53.068 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:35:53.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48fa000640 1 Processor -- start 2024-03-24T00:35:53.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48fa000640 1 -- start start 2024-03-24T00:35:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48fa000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48f4160080 0x7f48f4168ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48fa000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f40cfcb0 con 0x7f48f406f7b0 2024-03-24T00:35:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48fa000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f40d1120 con 0x7f48f415e170 2024-03-24T00:35:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48fa000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f406fc00 con 0x7f48f4160080 2024-03-24T00:35:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48eaa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f48f406f7b0 0x7f48f415f970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55896/0 (socket says 172.21.15.40:55896) 2024-03-24T00:35:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48eaa00640 1 -- 172.21.15.40:0/842014649 learned_addr learned my addr 172.21.15.40:0/842014649 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48f3e00640 1 --2- 172.21.15.40:0/842014649 >> v2:172.21.15.40:3300/0 conn(0x7f48f4160080 0x7f48f4168ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48f2a00640 1 -- 172.21.15.40:0/842014649 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585773354 0 0) 0x7f48dc0014a0 con 0x7f48f406f7b0 2024-03-24T00:35:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48f2a00640 1 -- 172.21.15.40:0/842014649 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48d8003530 con 0x7f48f406f7b0 2024-03-24T00:35:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48f2a00640 1 -- 172.21.15.40:0/842014649 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134160784 0 0) 0x7f48f40d1120 con 0x7f48f415e170 2024-03-24T00:35:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48f3e00640 1 -- 172.21.15.40:0/842014649 >> v1:172.21.15.53:6789/0 conn(0x7f48f415e170 legacy=0x7f48f40d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48f3e00640 1 -- 172.21.15.40:0/842014649 >> v1:172.21.15.40:6789/0 conn(0x7f48f406f7b0 legacy=0x7f48f415f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.210+0000 7f48f3e00640 1 -- 172.21.15.40:0/842014649 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f406fe20 con 0x7f48f4160080 2024-03-24T00:35:53.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.214+0000 7f48f3e00640 1 --2- 172.21.15.40:0/842014649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48f4160080 0x7f48f4168ea0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f48ec0097c0 tx=0x7f48ec01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2174736a452032 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.214+0000 7f48f2a00640 1 -- 172.21.15.40:0/842014649 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec0b5070 con 0x7f48f4160080 2024-03-24T00:35:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.214+0000 7f48f2a00640 1 -- 172.21.15.40:0/842014649 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ec0a8990 con 0x7f48f4160080 2024-03-24T00:35:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.214+0000 7f48f2a00640 1 -- 172.21.15.40:0/842014649 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ec0a8d60 con 0x7f48f4160080 2024-03-24T00:35:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.214+0000 7f48fa000640 1 -- 172.21.15.40:0/842014649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48f4160080 msgr2=0x7f48f4168ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.214+0000 7f48fa000640 1 --2- 172.21.15.40:0/842014649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48f4160080 0x7f48f4168ea0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f48ec0097c0 tx=0x7f48ec01ec90 comp rx=0 tx=0).stop 2024-03-24T00:35:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.214+0000 7f48fa000640 1 -- 172.21.15.40:0/842014649 shutdown_connections 2024-03-24T00:35:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.214+0000 7f48fa000640 1 --2- 172.21.15.40:0/842014649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48f4160080 0x7f48f4168ea0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.214+0000 7f48fa000640 1 -- 172.21.15.40:0/842014649 >> 172.21.15.40:0/842014649 conn(0x7f48f40b36a0 msgr2=0x7f48f40b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.214+0000 7f48fa000640 1 -- 172.21.15.40:0/842014649 shutdown_connections 2024-03-24T00:35:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.214+0000 7f48fa000640 1 -- 172.21.15.40:0/842014649 wait complete. 2024-03-24T00:35:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48fa000640 1 Processor -- start 2024-03-24T00:35:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48fa000640 1 -- start start 2024-03-24T00:35:53.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48fa000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48f406f7b0 0x7f48f414a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:53.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48fa000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48f415e170 0x7f48f40c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48fa000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f4160080 0x7f48f4146b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48fa000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f40d0210 con 0x7f48f406f7b0 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48fa000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f4070100 con 0x7f48f4160080 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48fa000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f40d04f0 con 0x7f48f415e170 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48f3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48f406f7b0 0x7f48f414a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48f3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48f406f7b0 0x7f48f414a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57146/0 (socket says 172.21.15.40:57146) 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48f3400640 1 -- 172.21.15.40:0/3376076881 learned_addr learned my addr 172.21.15.40:0/3376076881 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48eaa00640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48f415e170 0x7f48f40c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48f3e00640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f4160080 0x7f48f4146b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48f3400640 1 -- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48f415e170 msgr2=0x7f48f40c5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48f3400640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48f415e170 0x7f48f40c5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48f3400640 1 -- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f4160080 msgr2=0x7f48f4146b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48f3400640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f4160080 0x7f48f4146b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48f3400640 1 -- 172.21.15.40:0/3376076881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f40cf420 con 0x7f48f406f7b0 2024-03-24T00:35:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48eaa00640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48f415e170 0x7f48f40c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48f3e00640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f4160080 0x7f48f4146b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48f3400640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48f406f7b0 0x7f48f414a060 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f48e400c4b0 tx=0x7f48e400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48ebe00640 1 -- 172.21.15.40:0/3376076881 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e4005480 con 0x7f48f406f7b0 2024-03-24T00:35:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48ebe00640 1 -- 172.21.15.40:0/3376076881 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e4003600 con 0x7f48f406f7b0 2024-03-24T00:35:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48fa000640 1 -- 172.21.15.40:0/3376076881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f406cac0 con 0x7f48f406f7b0 2024-03-24T00:35:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48ebe00640 1 -- 172.21.15.40:0/3376076881 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e40039f0 con 0x7f48f406f7b0 2024-03-24T00:35:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48fa000640 1 -- 172.21.15.40:0/3376076881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f406bfe0 con 0x7f48f406f7b0 2024-03-24T00:35:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.218+0000 7f48fa000640 1 -- 172.21.15.40:0/3376076881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f4068170 con 0x7f48f406f7b0 2024-03-24T00:35:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.222+0000 7f48ebe00640 1 -- 172.21.15.40:0/3376076881 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f48e4003b90 con 0x7f48f406f7b0 2024-03-24T00:35:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.222+0000 7f48ebe00640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f48d0067690 0x7f48d0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.222+0000 7f48ebe00640 1 -- 172.21.15.40:0/3376076881 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f48e4085250 con 0x7f48f406f7b0 2024-03-24T00:35:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.222+0000 7f48eaa00640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f48d0067690 0x7f48d0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:53.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.222+0000 7f48eaa00640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f48d0067690 0x7f48d0069b50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f48f414a5a0 tx=0x7f48dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:53.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.226+0000 7f48ebe00640 1 -- 172.21.15.40:0/3376076881 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f48e4055ad0 con 0x7f48f406f7b0 2024-03-24T00:35:53.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.394+0000 7f48fa000640 1 -- 172.21.15.40:0/3376076881 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f48f4072f30 con 0x7f48f406f7b0 2024-03-24T00:35:53.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.394+0000 7f48ebe00640 1 -- 172.21.15.40:0/3376076881 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) v1 ==== 93+0+10 (secure 0 0 0) 0x7f48e4059b70 con 0x7f48f406f7b0 2024-03-24T00:35:53.398 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:35:53.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.398+0000 7f48fa000640 1 -- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f48d0067690 msgr2=0x7f48d0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.398+0000 7f48fa000640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f48d0067690 0x7f48d0069b50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f48f414a5a0 tx=0x7f48dc009040 comp rx=0 tx=0).stop 2024-03-24T00:35:53.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.398+0000 7f48fa000640 1 -- 172.21.15.40:0/3376076881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48f406f7b0 msgr2=0x7f48f414a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.398+0000 7f48fa000640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48f406f7b0 0x7f48f414a060 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f48e400c4b0 tx=0x7f48e400c970 comp rx=0 tx=0).stop 2024-03-24T00:35:53.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.398+0000 7f48fa000640 1 -- 172.21.15.40:0/3376076881 shutdown_connections 2024-03-24T00:35:53.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.398+0000 7f48fa000640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f48d0067690 0x7f48d0069b50 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.398+0000 7f48fa000640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48f4160080 0x7f48f4146b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.398+0000 7f48fa000640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48f415e170 0x7f48f40c5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.398+0000 7f48fa000640 1 --2- 172.21.15.40:0/3376076881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48f406f7b0 0x7f48f414a060 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.398+0000 7f48fa000640 1 -- 172.21.15.40:0/3376076881 >> 172.21.15.40:0/3376076881 conn(0x7f48f40b36a0 msgr2=0x7f48f40aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:53.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.398+0000 7f48fa000640 1 -- 172.21.15.40:0/3376076881 shutdown_connections 2024-03-24T00:35:53.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.398+0000 7f48fa000640 1 -- 172.21.15.40:0/3376076881 wait complete. 2024-03-24T00:35:53.419 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_after' 2024-03-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.554+0000 7fe4b3e00640 1 Processor -- start 2024-03-24T00:35:53.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b3e00640 1 -- start start 2024-03-24T00:35:53.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe4ac06e620 0x7fe4ac0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4ac0ccfa0 0x7fe4ac0cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b3e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4ac070f60 con 0x7fe4ac06ea00 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b3e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4ac0b4c00 con 0x7fe4ac06e620 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b3e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac0b3620 con 0x7fe4ac0ccfa0 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe4ac06e620 0x7fe4ac0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe4ac06e620 0x7fe4ac0b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57158/0 (socket says 172.21.15.40:57158) 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b1400640 1 -- 172.21.15.40:0/4025646854 learned_addr learned my addr 172.21.15.40:0/4025646854 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b1e00640 1 --2- 172.21.15.40:0/4025646854 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4ac0ccfa0 0x7fe4ac0cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b1400640 1 -- 172.21.15.40:0/4025646854 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4ac0ccfa0 msgr2=0x7fe4ac0cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b1400640 1 --2- 172.21.15.40:0/4025646854 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4ac0ccfa0 0x7fe4ac0cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b1400640 1 -- 172.21.15.40:0/4025646854 >> v1:172.21.15.53:6789/0 conn(0x7fe4ac06ea00 legacy=0x7fe4ac0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b1400640 1 -- 172.21.15.40:0/4025646854 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac0b3900 con 0x7fe4ac06e620 2024-03-24T00:35:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.558+0000 7fe4b1e00640 1 --2- 172.21.15.40:0/4025646854 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4ac0ccfa0 0x7fe4ac0cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4b1400640 1 --2- 172.21.15.40:0/4025646854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4ac06e620 0x7fe4ac0b46c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009870 tx=0x7fe4a001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60aae02562196518 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4abe00640 1 -- 172.21.15.40:0/4025646854 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a00b1070 con 0x7fe4ac06e620 2024-03-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4abe00640 1 -- 172.21.15.40:0/4025646854 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a00ac410 con 0x7fe4ac06e620 2024-03-24T00:35:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4abe00640 1 -- 172.21.15.40:0/4025646854 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a00ac7e0 con 0x7fe4ac06e620 2024-03-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4b3e00640 1 -- 172.21.15.40:0/4025646854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4ac06e620 msgr2=0x7fe4ac0b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4b3e00640 1 --2- 172.21.15.40:0/4025646854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4ac06e620 0x7fe4ac0b46c0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0009870 tx=0x7fe4a001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4b3e00640 1 -- 172.21.15.40:0/4025646854 shutdown_connections 2024-03-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4b3e00640 1 --2- 172.21.15.40:0/4025646854 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4ac0ccfa0 0x7fe4ac0cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4b3e00640 1 --2- 172.21.15.40:0/4025646854 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4ac06e620 0x7fe4ac0b46c0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4b3e00640 1 -- 172.21.15.40:0/4025646854 >> 172.21.15.40:0/4025646854 conn(0x7fe4ac0b17b0 msgr2=0x7fe4ac0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4b3e00640 1 -- 172.21.15.40:0/4025646854 shutdown_connections 2024-03-24T00:35:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4b3e00640 1 -- 172.21.15.40:0/4025646854 wait complete. 2024-03-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4b3e00640 1 Processor -- start 2024-03-24T00:35:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.562+0000 7fe4b3e00640 1 -- start start 2024-03-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4ac06e620 0x7fe4ac1687a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4ac06ea00 0x7fe4ac13e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4ac0ccfa0 0x7fe4ac1394f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac144920 con 0x7fe4ac0ccfa0 2024-03-24T00:35:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac06ee10 con 0x7fe4ac06ea00 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b3e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac168ce0 con 0x7fe4ac06e620 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4ac0ccfa0 0x7fe4ac1394f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4ac0ccfa0 0x7fe4ac1394f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57160/0 (socket says 172.21.15.40:57160) 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b1e00640 1 -- 172.21.15.40:0/295645357 learned_addr learned my addr 172.21.15.40:0/295645357 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b1400640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4ac06e620 0x7fe4ac1687a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b0a00640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4ac06ea00 0x7fe4ac13e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b1e00640 1 -- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4ac06e620 msgr2=0x7fe4ac1687a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b1e00640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4ac06e620 0x7fe4ac1687a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b1e00640 1 -- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4ac06ea00 msgr2=0x7fe4ac13e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b1e00640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4ac06ea00 0x7fe4ac13e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b1e00640 1 -- 172.21.15.40:0/295645357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac13a390 con 0x7fe4ac0ccfa0 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b1e00640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4ac0ccfa0 0x7fe4ac1394f0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4007ef0 tx=0x7fe4a400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4a9600640 1 -- 172.21.15.40:0/295645357 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a401a070 con 0x7fe4ac0ccfa0 2024-03-24T00:35:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4a9600640 1 -- 172.21.15.40:0/295645357 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a4003ba0 con 0x7fe4ac0ccfa0 2024-03-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4a9600640 1 -- 172.21.15.40:0/295645357 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a4004030 con 0x7fe4ac0ccfa0 2024-03-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b3e00640 1 -- 172.21.15.40:0/295645357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ac1683a0 con 0x7fe4ac0ccfa0 2024-03-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b3e00640 1 -- 172.21.15.40:0/295645357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ac15d450 con 0x7fe4ac0ccfa0 2024-03-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4b3e00640 1 -- 172.21.15.40:0/295645357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ac15d260 con 0x7fe4ac0ccfa0 2024-03-24T00:35:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.566+0000 7fe4a9600640 1 -- 172.21.15.40:0/295645357 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe4a401d070 con 0x7fe4ac0ccfa0 2024-03-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.570+0000 7fe4a9600640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe48c067660 0x7fe48c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.570+0000 7fe4a9600640 1 -- 172.21.15.40:0/295645357 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fe4a40857d0 con 0x7fe4ac0ccfa0 2024-03-24T00:35:53.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.570+0000 7fe4b1400640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe48c067660 0x7fe48c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:53.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.570+0000 7fe4b1400640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe48c067660 0x7fe48c069b20 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00ae000 tx=0x7fe4a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.574+0000 7fe4a9600640 1 -- 172.21.15.40:0/295645357 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe4ac13a390 con 0x7fe4ac0ccfa0 2024-03-24T00:35:53.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:53.794+0000 7fe4b3e00640 1 -- 172.21.15.40:0/295645357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_after"]} v 0) v1 -- 0x7fe4ac068170 con 0x7fe4ac0ccfa0 2024-03-24T00:35:54.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.702+0000 7fe4a9600640 1 -- 172.21.15.40:0/295645357 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_after"]}]=0 v257) v1 ==== 164+0+0 (secure 0 0 0) 0x7fe4a4010070 con 0x7fe4ac0ccfa0 2024-03-24T00:35:54.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.706+0000 7fe4b3e00640 1 -- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe48c067660 msgr2=0x7fe48c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:54.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.706+0000 7fe4b3e00640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe48c067660 0x7fe48c069b20 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00ae000 tx=0x7fe4a0009040 comp rx=0 tx=0).stop 2024-03-24T00:35:54.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.706+0000 7fe4b3e00640 1 -- 172.21.15.40:0/295645357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4ac0ccfa0 msgr2=0x7fe4ac1394f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:54.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.706+0000 7fe4b3e00640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4ac0ccfa0 0x7fe4ac1394f0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4007ef0 tx=0x7fe4a400d4a0 comp rx=0 tx=0).stop 2024-03-24T00:35:54.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.706+0000 7fe4b3e00640 1 -- 172.21.15.40:0/295645357 shutdown_connections 2024-03-24T00:35:54.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.706+0000 7fe4b3e00640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe48c067660 0x7fe48c069b20 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:54.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.706+0000 7fe4b3e00640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe4ac0ccfa0 0x7fe4ac1394f0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:54.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.706+0000 7fe4b3e00640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4ac06ea00 0x7fe4ac13e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:54.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.706+0000 7fe4b3e00640 1 --2- 172.21.15.40:0/295645357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe4ac06e620 0x7fe4ac1687a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:54.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.706+0000 7fe4b3e00640 1 -- 172.21.15.40:0/295645357 >> 172.21.15.40:0/295645357 conn(0x7fe4ac0b17b0 msgr2=0x7fe4ac15a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:54.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.706+0000 7fe4b3e00640 1 -- 172.21.15.40:0/295645357 shutdown_connections 2024-03-24T00:35:54.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.706+0000 7fe4b3e00640 1 -- 172.21.15.40:0/295645357 wait complete. 2024-03-24T00:35:54.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:35:54.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90fda00640 1 Processor -- start 2024-03-24T00:35:54.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90fda00640 1 -- start start 2024-03-24T00:35:54.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90fda00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f90f815bd00 0x7f90f80ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:54.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90fda00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f8072d60 con 0x7f90f80728d0 2024-03-24T00:35:54.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90fda00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90f8071de0 con 0x7f90f815ffd0 2024-03-24T00:35:54.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90fda00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90f80cea50 con 0x7f90f815bd00 2024-03-24T00:35:54.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90ee000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f90f80728d0 0x7f90f815f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:55900/0 (socket says 172.21.15.40:55900) 2024-03-24T00:35:54.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90ee000640 1 -- 172.21.15.40:0/3134003492 learned_addr learned my addr 172.21.15.40:0/3134003492 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:35:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90f6a00640 1 --2- 172.21.15.40:0/3134003492 >> v2:172.21.15.53:3300/0 conn(0x7f90f815bd00 0x7f90f80ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90f6000640 1 -- 172.21.15.40:0/3134003492 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458382082 0 0) 0x7f90f8072d60 con 0x7f90f80728d0 2024-03-24T00:35:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90f6000640 1 -- 172.21.15.40:0/3134003492 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90dc003530 con 0x7f90f80728d0 2024-03-24T00:35:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90f6a00640 1 -- 172.21.15.40:0/3134003492 >> v1:172.21.15.53:6789/0 conn(0x7f90f815ffd0 legacy=0x7f90f806cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:54.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90f6a00640 1 -- 172.21.15.40:0/3134003492 >> v1:172.21.15.40:6789/0 conn(0x7f90f80728d0 legacy=0x7f90f815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:54.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.874+0000 7f90f6a00640 1 -- 172.21.15.40:0/3134003492 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f80ced30 con 0x7f90f815bd00 2024-03-24T00:35:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.878+0000 7f90f6a00640 1 --2- 172.21.15.40:0/3134003492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90f815bd00 0x7f90f80ae8e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f90e80097c0 tx=0x7f90e801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fdee5f04619f1f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.878+0000 7f90f6000640 1 -- 172.21.15.40:0/3134003492 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e80ab650 con 0x7f90f815bd00 2024-03-24T00:35:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.878+0000 7f90f6000640 1 -- 172.21.15.40:0/3134003492 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e80b0070 con 0x7f90f815bd00 2024-03-24T00:35:54.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.878+0000 7f90f6000640 1 -- 172.21.15.40:0/3134003492 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e80ab9f0 con 0x7f90f815bd00 2024-03-24T00:35:54.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.878+0000 7f90fda00640 1 -- 172.21.15.40:0/3134003492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90f815bd00 msgr2=0x7f90f80ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:54.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.878+0000 7f90fda00640 1 --2- 172.21.15.40:0/3134003492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90f815bd00 0x7f90f80ae8e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f90e80097c0 tx=0x7f90e801c470 comp rx=0 tx=0).stop 2024-03-24T00:35:54.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.878+0000 7f90fda00640 1 -- 172.21.15.40:0/3134003492 shutdown_connections 2024-03-24T00:35:54.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.878+0000 7f90fda00640 1 --2- 172.21.15.40:0/3134003492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90f815bd00 0x7f90f80ae8e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:54.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.878+0000 7f90fda00640 1 -- 172.21.15.40:0/3134003492 >> 172.21.15.40:0/3134003492 conn(0x7f90f80b1660 msgr2=0x7f90f80b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:54.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.878+0000 7f90fda00640 1 -- 172.21.15.40:0/3134003492 shutdown_connections 2024-03-24T00:35:54.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.878+0000 7f90fda00640 1 -- 172.21.15.40:0/3134003492 wait complete. 2024-03-24T00:35:54.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90fda00640 1 Processor -- start 2024-03-24T00:35:54.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90fda00640 1 -- start start 2024-03-24T00:35:54.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90fda00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f80728d0 0x7f90f81637b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:54.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90fda00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90f815bd00 0x7f90f814ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:54.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90fda00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90f815ffd0 0x7f90f80ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:54.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90fda00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f815c0e0 con 0x7f90f80728d0 2024-03-24T00:35:54.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90fda00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f80cf660 con 0x7f90f815bd00 2024-03-24T00:35:54.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90fda00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90f816a420 con 0x7f90f815ffd0 2024-03-24T00:35:54.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f80728d0 0x7f90f81637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:54.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90f6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f80728d0 0x7f90f81637b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57162/0 (socket says 172.21.15.40:57162) 2024-03-24T00:35:54.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90f6a00640 1 -- 172.21.15.40:0/2814487677 learned_addr learned my addr 172.21.15.40:0/2814487677 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90f6a00640 1 -- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90f815ffd0 msgr2=0x7f90f80ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90f6a00640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90f815ffd0 0x7f90f80ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90f6a00640 1 -- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90f815bd00 msgr2=0x7f90f814ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90ee000640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90f815bd00 0x7f90f814ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90f6a00640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90f815bd00 0x7f90f814ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90f6a00640 1 -- 172.21.15.40:0/2814487677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f8162010 con 0x7f90f80728d0 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90ee000640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90f815bd00 0x7f90f814ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90f6a00640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f80728d0 0x7f90f81637b0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f90e80097c0 tx=0x7f90e8019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90ef400640 1 -- 172.21.15.40:0/2814487677 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e80c1070 con 0x7f90f80728d0 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90ef400640 1 -- 172.21.15.40:0/2814487677 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e8003820 con 0x7f90f80728d0 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90ef400640 1 -- 172.21.15.40:0/2814487677 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8003bd0 con 0x7f90f80728d0 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90fda00640 1 -- 172.21.15.40:0/2814487677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f81622f0 con 0x7f90f80728d0 2024-03-24T00:35:54.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.882+0000 7f90fda00640 1 -- 172.21.15.40:0/2814487677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f80b3520 con 0x7f90f80728d0 2024-03-24T00:35:54.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.886+0000 7f90fda00640 1 -- 172.21.15.40:0/2814487677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f8068170 con 0x7f90f80728d0 2024-03-24T00:35:54.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.886+0000 7f90ef400640 1 -- 172.21.15.40:0/2814487677 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f90e80ab450 con 0x7f90f80728d0 2024-03-24T00:35:54.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.886+0000 7f90ef400640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f90d4067660 0x7f90d4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:54.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.886+0000 7f90ef400640 1 -- 172.21.15.40:0/2814487677 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f90e811f9b0 con 0x7f90f80728d0 2024-03-24T00:35:54.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.886+0000 7f90ee000640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f90d4067660 0x7f90d4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:54.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.886+0000 7f90ee000640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f90d4067660 0x7f90d4069b20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f90f80ae5c0 tx=0x7f90e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:54.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:54.890+0000 7f90ef400640 1 -- 172.21.15.40:0/2814487677 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f90e8016090 con 0x7f90f80728d0 2024-03-24T00:35:55.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.054+0000 7f90fda00640 1 -- 172.21.15.40:0/2814487677 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f90f806e870 con 0x7f90f80728d0 2024-03-24T00:35:55.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90ef400640 1 -- 172.21.15.40:0/2814487677 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v43)=0 v43) v1 ==== 122+0+0 (secure 0 0 0) 0x7f90e80f0870 con 0x7f90f80728d0 2024-03-24T00:35:55.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90fda00640 1 -- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f90d4067660 msgr2=0x7f90d4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:55.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90fda00640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f90d4067660 0x7f90d4069b20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f90f80ae5c0 tx=0x7f90e0009040 comp rx=0 tx=0).stop 2024-03-24T00:35:55.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90fda00640 1 -- 172.21.15.40:0/2814487677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f80728d0 msgr2=0x7f90f81637b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:55.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90fda00640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f80728d0 0x7f90f81637b0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f90e80097c0 tx=0x7f90e8019210 comp rx=0 tx=0).stop 2024-03-24T00:35:55.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90fda00640 1 -- 172.21.15.40:0/2814487677 shutdown_connections 2024-03-24T00:35:55.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90fda00640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f90d4067660 0x7f90d4069b20 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90fda00640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90f815ffd0 0x7f90f80ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90fda00640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90f815bd00 0x7f90f814ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90fda00640 1 --2- 172.21.15.40:0/2814487677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90f80728d0 0x7f90f81637b0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90fda00640 1 -- 172.21.15.40:0/2814487677 >> 172.21.15.40:0/2814487677 conn(0x7f90f80b1660 msgr2=0x7f90f8070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:55.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90fda00640 1 -- 172.21.15.40:0/2814487677 shutdown_connections 2024-03-24T00:35:55.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.058+0000 7f90fda00640 1 -- 172.21.15.40:0/2814487677 wait complete. 2024-03-24T00:35:55.082 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:35:55.082 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:35:55.166 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:35:55.211 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:35:55.211 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:35:55.211 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:35:55.220 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:35:55.220 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:35:55.220 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:35:55.265 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:35:55.265 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:35:55.265 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:35:55.274 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:35:55.274 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:35:55.274 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:35:55.320 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:35:55.320 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:35:55.321 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:35:55.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3ce7400640 1 Processor -- start 2024-03-24T00:35:55.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3ce7400640 1 -- start start 2024-03-24T00:35:55.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3ce7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3ce015e0b0 0x7f3ce00d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:55.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3ce7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ce00cd3d0 0x7f3ce00cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:55.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3ce7400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce00d0e10 con 0x7f3ce00cdd80 2024-03-24T00:35:55.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3ce7400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce00d10f0 con 0x7f3ce00cd3d0 2024-03-24T00:35:55.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3ce7400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce006efd0 con 0x7f3ce015e0b0 2024-03-24T00:35:55.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3cdfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ce00cd3d0 0x7f3ce00cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:55.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3cdfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3ce00cd3d0 0x7f3ce00cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57164/0 (socket says 172.21.15.40:57164) 2024-03-24T00:35:55.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3cdfe00640 1 -- 172.21.15.40:0/3069659514 learned_addr learned my addr 172.21.15.40:0/3069659514 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:55.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3ce4a00640 1 --2- 172.21.15.40:0/3069659514 >> v2:172.21.15.53:3300/0 conn(0x7f3ce015e0b0 0x7f3ce00d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:55.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3cdfe00640 1 -- 172.21.15.40:0/3069659514 >> v2:172.21.15.53:3300/0 conn(0x7f3ce015e0b0 msgr2=0x7f3ce00d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:55.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3cdfe00640 1 --2- 172.21.15.40:0/3069659514 >> v2:172.21.15.53:3300/0 conn(0x7f3ce015e0b0 0x7f3ce00d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3cdfe00640 1 -- 172.21.15.40:0/3069659514 >> v1:172.21.15.53:6789/0 conn(0x7f3ce00cdd80 legacy=0x7f3ce00ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:35:55.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.458+0000 7f3cdfe00640 1 -- 172.21.15.40:0/3069659514 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce006f2b0 con 0x7f3ce00cd3d0 2024-03-24T00:35:55.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.462+0000 7f3cdfe00640 1 --2- 172.21.15.40:0/3069659514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce00cd3d0 0x7f3ce00cd840 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3ce006ebb0 tx=0x7f3ccc01c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db06110f6a7f4e22 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:55.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.462+0000 7f3cdf400640 1 -- 172.21.15.40:0/3069659514 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc0b0070 con 0x7f3ce00cd3d0 2024-03-24T00:35:55.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.462+0000 7f3cdf400640 1 -- 172.21.15.40:0/3069659514 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ccc0ab600 con 0x7f3ce00cd3d0 2024-03-24T00:35:55.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.462+0000 7f3cdf400640 1 -- 172.21.15.40:0/3069659514 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc0ab9d0 con 0x7f3ce00cd3d0 2024-03-24T00:35:55.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.462+0000 7f3ce7400640 1 -- 172.21.15.40:0/3069659514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce00cd3d0 msgr2=0x7f3ce00cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:55.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.462+0000 7f3ce7400640 1 --2- 172.21.15.40:0/3069659514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce00cd3d0 0x7f3ce00cd840 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3ce006ebb0 tx=0x7f3ccc01c4c0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.462+0000 7f3ce7400640 1 -- 172.21.15.40:0/3069659514 shutdown_connections 2024-03-24T00:35:55.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.462+0000 7f3ce7400640 1 --2- 172.21.15.40:0/3069659514 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce00cd3d0 0x7f3ce00cd840 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.462+0000 7f3ce7400640 1 --2- 172.21.15.40:0/3069659514 >> v2:172.21.15.53:3300/0 conn(0x7f3ce015e0b0 0x7f3ce00d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.462+0000 7f3ce7400640 1 -- 172.21.15.40:0/3069659514 >> 172.21.15.40:0/3069659514 conn(0x7f3ce00d1c90 msgr2=0x7f3ce00d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:55.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.462+0000 7f3ce7400640 1 -- 172.21.15.40:0/3069659514 shutdown_connections 2024-03-24T00:35:55.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.462+0000 7f3ce7400640 1 -- 172.21.15.40:0/3069659514 wait complete. 2024-03-24T00:35:55.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3ce7400640 1 Processor -- start 2024-03-24T00:35:55.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3ce7400640 1 -- start start 2024-03-24T00:35:55.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3ce7400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ce00cd3d0 0x7f3ce0151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:55.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3ce7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce00cdd80 0x7f3ce014f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:55.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3ce7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ce015e0b0 0x7f3ce0154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:55.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3ce7400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce006fc30 con 0x7f3ce00cdd80 2024-03-24T00:35:55.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3ce7400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce006fab0 con 0x7f3ce015e0b0 2024-03-24T00:35:55.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3ce7400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce006fdb0 con 0x7f3ce00cd3d0 2024-03-24T00:35:55.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3cdfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce00cdd80 0x7f3ce014f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:55.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3cdfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce00cdd80 0x7f3ce014f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:57166/0 (socket says 172.21.15.40:57166) 2024-03-24T00:35:55.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3cdfe00640 1 -- 172.21.15.40:0/3278229572 learned_addr learned my addr 172.21.15.40:0/3278229572 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:35:55.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3ce5400640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ce015e0b0 0x7f3ce0154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:55.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3ce4a00640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ce00cd3d0 0x7f3ce0151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:55.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3cdfe00640 1 -- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ce00cd3d0 msgr2=0x7f3ce0151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:55.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3cdfe00640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ce00cd3d0 0x7f3ce0151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3cdfe00640 1 -- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ce015e0b0 msgr2=0x7f3ce0154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:55.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3cdfe00640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ce015e0b0 0x7f3ce0154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3cdfe00640 1 -- 172.21.15.40:0/3278229572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce015eba0 con 0x7f3ce00cdd80 2024-03-24T00:35:55.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3cdfe00640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce00cdd80 0x7f3ce014f410 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0ae040 tx=0x7f3ccc019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:55.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3cdcc00640 1 -- 172.21.15.40:0/3278229572 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc0b0040 con 0x7f3ce00cdd80 2024-03-24T00:35:55.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3cdcc00640 1 -- 172.21.15.40:0/3278229572 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ccc0b4420 con 0x7f3ce00cdd80 2024-03-24T00:35:55.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3ce7400640 1 -- 172.21.15.40:0/3278229572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce00b4c10 con 0x7f3ce00cdd80 2024-03-24T00:35:55.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3cdcc00640 1 -- 172.21.15.40:0/3278229572 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc0b4810 con 0x7f3ce00cdd80 2024-03-24T00:35:55.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.466+0000 7f3ce7400640 1 -- 172.21.15.40:0/3278229572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce00b51d0 con 0x7f3ce00cdd80 2024-03-24T00:35:55.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.470+0000 7f3ce7400640 1 -- 172.21.15.40:0/3278229572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce00b53d0 con 0x7f3ce00cdd80 2024-03-24T00:35:55.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.470+0000 7f3cdcc00640 1 -- 172.21.15.40:0/3278229572 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3ccc0b49b0 con 0x7f3ce00cdd80 2024-03-24T00:35:55.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.470+0000 7f3cdcc00640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3cc0067660 0x7f3cc0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:35:55.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.470+0000 7f3cdcc00640 1 -- 172.21.15.40:0/3278229572 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3ccc11f270 con 0x7f3ce00cdd80 2024-03-24T00:35:55.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.470+0000 7f3ce4a00640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3cc0067660 0x7f3cc0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:35:55.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.470+0000 7f3ce4a00640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3cc0067660 0x7f3cc0069b20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0156240 tx=0x7f3cd4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:35:55.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.474+0000 7f3cdcc00640 1 -- 172.21.15.40:0/3278229572 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3ce00b53d0 con 0x7f3ce00cdd80 2024-03-24T00:35:55.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.674+0000 7f3ce7400640 1 -- 172.21.15.40:0/3278229572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ce0068170 con 0x7f3ce00cdd80 2024-03-24T00:35:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.674+0000 7f3cdcc00640 1 -- 172.21.15.40:0/3278229572 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+3690 (secure 0 0 0) 0x7f3ccc0160b0 con 0x7f3ce00cdd80 2024-03-24T00:35:55.679 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:35:55.679 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":75,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6062,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/4182264423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":4182264423},{"type":"v1","addr":"172.21.15.53:6836","nonce":4182264423}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":62},{"gid":6341,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/138071649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":138071649},{"type":"v1","addr":"172.21.15.40:6833","nonce":138071649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7078,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/291725059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":291725059},{"type":"v1","addr":"172.21.15.53:6837","nonce":291725059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75},{"gid":7329,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/420320254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":420320254},{"type":"v1","addr":"172.21.15.40:6835","nonce":420320254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[]} 2024-03-24T00:35:55.679 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 75 2024-03-24T00:35:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.678+0000 7f3ce7400640 1 -- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3cc0067660 msgr2=0x7f3cc0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.678+0000 7f3ce7400640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3cc0067660 0x7f3cc0069b20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0156240 tx=0x7f3cd4008040 comp rx=0 tx=0).stop 2024-03-24T00:35:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.678+0000 7f3ce7400640 1 -- 172.21.15.40:0/3278229572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce00cdd80 msgr2=0x7f3ce014f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:35:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.678+0000 7f3ce7400640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce00cdd80 0x7f3ce014f410 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0ae040 tx=0x7f3ccc019260 comp rx=0 tx=0).stop 2024-03-24T00:35:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.678+0000 7f3ce7400640 1 -- 172.21.15.40:0/3278229572 shutdown_connections 2024-03-24T00:35:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.678+0000 7f3ce7400640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3cc0067660 0x7f3cc0069b20 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.678+0000 7f3ce7400640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ce015e0b0 0x7f3ce0154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.678+0000 7f3ce7400640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3ce00cdd80 0x7f3ce014f410 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.678+0000 7f3ce7400640 1 --2- 172.21.15.40:0/3278229572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3ce00cd3d0 0x7f3ce0151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:35:55.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.678+0000 7f3ce7400640 1 -- 172.21.15.40:0/3278229572 >> 172.21.15.40:0/3278229572 conn(0x7f3ce00d1c90 msgr2=0x7f3ce006b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:35:55.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.678+0000 7f3ce7400640 1 -- 172.21.15.40:0/3278229572 shutdown_connections 2024-03-24T00:35:55.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:35:55.678+0000 7f3ce7400640 1 -- 172.21.15.40:0/3278229572 wait complete. 2024-03-24T00:35:55.701 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:35:55.701 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:35:55.701 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:35:55.701 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:35:55.701 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:35:55.702 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:35:55.702 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:35:55.702 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:35:55.702 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:35:55.702 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:35:55.702 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:35:55.702 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:35:55.703 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:35:55.703 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:35:55.703 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:35:55.703 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:35:55.716 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:35:55.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:35:55.717 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:35:55.717 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:35:55.719 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:35:55.719 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:35:55.762 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:35:55.762 DEBUG:teuthology.parallel:result is None 2024-03-24T00:35:55.763 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:35:55.764 DEBUG:teuthology.parallel:result is None 2024-03-24T00:35:55.766 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:35:55.766 DEBUG:teuthology.parallel:result is None 2024-03-24T00:35:55.847 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:35:55.847 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:35:55.857 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:35:59.315 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-24T00:36:01.719 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:36:01.719 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:36:01.722 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:36:01.723 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:01.723 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:36:01.817 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:36:01.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.874+0000 7febab600640 1 Processor -- start 2024-03-24T00:36:01.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.874+0000 7febab600640 1 -- start start 2024-03-24T00:36:01.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.874+0000 7febab600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feba406e620 0x7feba40b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:01.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.874+0000 7febab600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7feba406ea00 0x7feba40b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:01.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.874+0000 7febab600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba4070f60 con 0x7feba40b2ff0 2024-03-24T00:36:01.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.874+0000 7febab600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba40b4c00 con 0x7feba406ea00 2024-03-24T00:36:01.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.874+0000 7febab600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba40b3620 con 0x7feba406e620 2024-03-24T00:36:01.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.874+0000 7feba8c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feba406e620 0x7feba40b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:01.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.874+0000 7feba8c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feba406e620 0x7feba40b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:47954/0 (socket says 172.21.15.40:47954) 2024-03-24T00:36:01.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.874+0000 7feba8c00640 1 -- 172.21.15.40:0/123686776 learned_addr learned my addr 172.21.15.40:0/123686776 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:01.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.874+0000 7feba9600640 1 --1- 172.21.15.40:0/123686776 >> v1:172.21.15.53:6789/0 conn(0x7feba40b2ff0 0x7feba40ccfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:56950/0 (socket says 172.21.15.40:56950) 2024-03-24T00:36:01.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.874+0000 7feb9be00640 1 --2- 172.21.15.40:0/123686776 >> v2:172.21.15.53:3300/0 conn(0x7feba406ea00 0x7feba40b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:01.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7feba8c00640 1 -- 172.21.15.40:0/123686776 >> v2:172.21.15.53:3300/0 conn(0x7feba406ea00 msgr2=0x7feba40b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:01.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7feba8c00640 1 --2- 172.21.15.40:0/123686776 >> v2:172.21.15.53:3300/0 conn(0x7feba406ea00 0x7feba40b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:01.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7feba8c00640 1 -- 172.21.15.40:0/123686776 >> v1:172.21.15.53:6789/0 conn(0x7feba40b2ff0 legacy=0x7feba40ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:01.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7feba8c00640 1 -- 172.21.15.40:0/123686776 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba40b3900 con 0x7feba406e620 2024-03-24T00:36:01.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7feba8c00640 1 --2- 172.21.15.40:0/123686776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feba406e620 0x7feba40b46c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feb94009870 tx=0x7feb9401c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b952e35b7c50563c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:01.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7feba3e00640 1 -- 172.21.15.40:0/123686776 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940b1070 con 0x7feba406e620 2024-03-24T00:36:01.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7feba3e00640 1 -- 172.21.15.40:0/123686776 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb940ac620 con 0x7feba406e620 2024-03-24T00:36:01.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7feba3e00640 1 -- 172.21.15.40:0/123686776 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940ac9f0 con 0x7feba406e620 2024-03-24T00:36:01.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7febab600640 1 -- 172.21.15.40:0/123686776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feba406e620 msgr2=0x7feba40b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:01.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7febab600640 1 --2- 172.21.15.40:0/123686776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feba406e620 0x7feba40b46c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feb94009870 tx=0x7feb9401c4d0 comp rx=0 tx=0).stop 2024-03-24T00:36:01.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7febab600640 1 -- 172.21.15.40:0/123686776 shutdown_connections 2024-03-24T00:36:01.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7febab600640 1 --2- 172.21.15.40:0/123686776 >> v2:172.21.15.53:3300/0 conn(0x7feba406ea00 0x7feba40b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:01.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7febab600640 1 --2- 172.21.15.40:0/123686776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feba406e620 0x7feba40b46c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:01.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7febab600640 1 -- 172.21.15.40:0/123686776 >> 172.21.15.40:0/123686776 conn(0x7feba40b17b0 msgr2=0x7feba40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:01.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7febab600640 1 -- 172.21.15.40:0/123686776 shutdown_connections 2024-03-24T00:36:01.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.878+0000 7febab600640 1 -- 172.21.15.40:0/123686776 wait complete. 2024-03-24T00:36:01.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7febab600640 1 Processor -- start 2024-03-24T00:36:01.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7febab600640 1 -- start start 2024-03-24T00:36:01.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7febab600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feba406e620 0x7feba40aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:01.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7febab600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba406ea00 0x7feba40b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7febab600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feba40b2ff0 0x7feba415b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7febab600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba4144920 con 0x7feba406ea00 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7febab600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba406ede0 con 0x7feba40b2ff0 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7febab600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba413c420 con 0x7feba406e620 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feb9be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba406ea00 0x7feba40b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feb9be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba406ea00 0x7feba40b0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49712/0 (socket says 172.21.15.40:49712) 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feb9be00640 1 -- 172.21.15.40:0/1400740011 learned_addr learned my addr 172.21.15.40:0/1400740011 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feba8c00640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feba406e620 0x7feba40aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feba9600640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feba40b2ff0 0x7feba415b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feb9be00640 1 -- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feba406e620 msgr2=0x7feba40aefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feb9be00640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feba406e620 0x7feba40aefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feb9be00640 1 -- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feba40b2ff0 msgr2=0x7feba415b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feb9be00640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feba40b2ff0 0x7feba415b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:01.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feb9be00640 1 -- 172.21.15.40:0/1400740011 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba4166ee0 con 0x7feba406ea00 2024-03-24T00:36:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feba9600640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feba40b2ff0 0x7feba415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:36:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feb9be00640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba406ea00 0x7feba40b0420 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7feb8c00e930 tx=0x7feb8c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feba1600640 1 -- 172.21.15.40:0/1400740011 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8c017070 con 0x7feba406ea00 2024-03-24T00:36:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feba1600640 1 -- 172.21.15.40:0/1400740011 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8c00ca00 con 0x7feba406ea00 2024-03-24T00:36:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7feba1600640 1 -- 172.21.15.40:0/1400740011 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8c00cdb0 con 0x7feba406ea00 2024-03-24T00:36:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.882+0000 7febab600640 1 -- 172.21.15.40:0/1400740011 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba41671c0 con 0x7feba406ea00 2024-03-24T00:36:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.886+0000 7febab600640 1 -- 172.21.15.40:0/1400740011 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba4167780 con 0x7feba406ea00 2024-03-24T00:36:01.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.886+0000 7febab600640 1 -- 172.21.15.40:0/1400740011 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba4068170 con 0x7feba406ea00 2024-03-24T00:36:01.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.886+0000 7feba1600640 1 -- 172.21.15.40:0/1400740011 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7feb8c01a070 con 0x7feba406ea00 2024-03-24T00:36:01.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.886+0000 7feba1600640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb80067660 0x7feb80069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:01.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.886+0000 7feba1600640 1 -- 172.21.15.40:0/1400740011 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5115+0+0 (secure 0 0 0) 0x7feb8c085a60 con 0x7feba406ea00 2024-03-24T00:36:01.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.886+0000 7feba8c00640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb80067660 0x7feb80069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:01.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.890+0000 7feba8c00640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb80067660 0x7feb80069b20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feb940ae000 tx=0x7feb940a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:01.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:01.890+0000 7feba1600640 1 -- 172.21.15.40:0/1400740011 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7feb8c0569e0 con 0x7feba406ea00 2024-03-24T00:36:02.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:02.066+0000 7febab600640 1 -- 172.21.15.40:0/1400740011 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7feba4072f50 con 0x7feba406ea00 2024-03-24T00:36:03.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.726+0000 7feba1600640 1 -- 172.21.15.40:0/1400740011 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v79) v1 ==== 97+0+0 (secure 0 0 0) 0x7feb8c0562e0 con 0x7feba406ea00 2024-03-24T00:36:03.732 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-03-24T00:36:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.730+0000 7febab600640 1 -- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb80067660 msgr2=0x7feb80069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.730+0000 7febab600640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb80067660 0x7feb80069b20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7feb940ae000 tx=0x7feb940a7000 comp rx=0 tx=0).stop 2024-03-24T00:36:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.730+0000 7febab600640 1 -- 172.21.15.40:0/1400740011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba406ea00 msgr2=0x7feba40b0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.730+0000 7febab600640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba406ea00 0x7feba40b0420 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7feb8c00e930 tx=0x7feb8c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:36:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.730+0000 7febab600640 1 -- 172.21.15.40:0/1400740011 shutdown_connections 2024-03-24T00:36:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.730+0000 7febab600640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb80067660 0x7feb80069b20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.730+0000 7febab600640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feba40b2ff0 0x7feba415b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:03.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.730+0000 7febab600640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feba406ea00 0x7feba40b0420 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:03.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.730+0000 7febab600640 1 --2- 172.21.15.40:0/1400740011 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feba406e620 0x7feba40aefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:03.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.730+0000 7febab600640 1 -- 172.21.15.40:0/1400740011 >> 172.21.15.40:0/1400740011 conn(0x7feba40b17b0 msgr2=0x7feba40af820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:03.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.730+0000 7febab600640 1 -- 172.21.15.40:0/1400740011 shutdown_connections 2024-03-24T00:36:03.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.730+0000 7febab600640 1 -- 172.21.15.40:0/1400740011 wait complete. 2024-03-24T00:36:03.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:36:03.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff65a800640 1 Processor -- start 2024-03-24T00:36:03.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff65a800640 1 -- start start 2024-03-24T00:36:03.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff65a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff654141bc0 0x7ff6540ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:03.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff65a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6540ce630 0x7ff654071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:03.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff65a800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff654070e00 con 0x7ff65413ec80 2024-03-24T00:36:03.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff65a800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6540cee20 con 0x7ff6540ce630 2024-03-24T00:36:03.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff65a800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6540cf100 con 0x7ff654141bc0 2024-03-24T00:36:03.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff653e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6540ce630 0x7ff654071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:03.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff653e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff6540ce630 0x7ff654071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49716/0 (socket says 172.21.15.40:49716) 2024-03-24T00:36:03.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff653e00640 1 -- 172.21.15.40:0/3771542659 learned_addr learned my addr 172.21.15.40:0/3771542659 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:03.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff64aa00640 1 --2- 172.21.15.40:0/3771542659 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff654141bc0 0x7ff6540ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:03.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff653e00640 1 -- 172.21.15.40:0/3771542659 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff654141bc0 msgr2=0x7ff6540ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:03.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff653e00640 1 --2- 172.21.15.40:0/3771542659 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff654141bc0 0x7ff6540ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:03.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff653e00640 1 -- 172.21.15.40:0/3771542659 >> v1:172.21.15.53:6789/0 conn(0x7ff65413ec80 legacy=0x7ff6541414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:03.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.902+0000 7ff653e00640 1 -- 172.21.15.40:0/3771542659 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6540cf3e0 con 0x7ff6540ce630 2024-03-24T00:36:03.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.906+0000 7ff653e00640 1 --2- 172.21.15.40:0/3771542659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6540ce630 0x7ff654071300 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009e50 tx=0x7ff64c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3215c2a59e7b9d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:03.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.906+0000 7ff652a00640 1 -- 172.21.15.40:0/3771542659 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c0a5920 con 0x7ff6540ce630 2024-03-24T00:36:03.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.906+0000 7ff652a00640 1 -- 172.21.15.40:0/3771542659 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64c0a5ac0 con 0x7ff6540ce630 2024-03-24T00:36:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.906+0000 7ff652a00640 1 -- 172.21.15.40:0/3771542659 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c0ad3f0 con 0x7ff6540ce630 2024-03-24T00:36:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.906+0000 7ff65a800640 1 -- 172.21.15.40:0/3771542659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6540ce630 msgr2=0x7ff654071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.906+0000 7ff65a800640 1 --2- 172.21.15.40:0/3771542659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6540ce630 0x7ff654071300 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009e50 tx=0x7ff64c01ca30 comp rx=0 tx=0).stop 2024-03-24T00:36:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.906+0000 7ff65a800640 1 -- 172.21.15.40:0/3771542659 shutdown_connections 2024-03-24T00:36:03.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.906+0000 7ff65a800640 1 --2- 172.21.15.40:0/3771542659 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff654141bc0 0x7ff6540ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:03.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.906+0000 7ff65a800640 1 --2- 172.21.15.40:0/3771542659 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff6540ce630 0x7ff654071300 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:03.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.906+0000 7ff65a800640 1 -- 172.21.15.40:0/3771542659 >> 172.21.15.40:0/3771542659 conn(0x7ff6540b17b0 msgr2=0x7ff6540b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:03.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.906+0000 7ff65a800640 1 -- 172.21.15.40:0/3771542659 shutdown_connections 2024-03-24T00:36:03.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.906+0000 7ff65a800640 1 -- 172.21.15.40:0/3771542659 wait complete. 2024-03-24T00:36:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff65a800640 1 Processor -- start 2024-03-24T00:36:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff65a800640 1 -- start start 2024-03-24T00:36:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff65a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6540ce630 0x7ff654155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff65a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65413ec80 0x7ff65413f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff65a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff654141bc0 0x7ff65406df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff65a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6540723e0 con 0x7ff65413ec80 2024-03-24T00:36:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff65a800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff654072260 con 0x7ff654141bc0 2024-03-24T00:36:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff65a800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff654072560 con 0x7ff6540ce630 2024-03-24T00:36:03.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff64aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65413ec80 0x7ff65413f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff64aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65413ec80 0x7ff65413f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49720/0 (socket says 172.21.15.40:49720) 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff64aa00640 1 -- 172.21.15.40:0/3633386644 learned_addr learned my addr 172.21.15.40:0/3633386644 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff653e00640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff654141bc0 0x7ff65406df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff64aa00640 1 -- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6540ce630 msgr2=0x7ff654155b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff64aa00640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6540ce630 0x7ff654155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff64aa00640 1 -- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff654141bc0 msgr2=0x7ff65406df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff64aa00640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff654141bc0 0x7ff65406df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff64aa00640 1 -- 172.21.15.40:0/3633386644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65413fc70 con 0x7ff65413ec80 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff653e00640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff654141bc0 0x7ff65406df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff64aa00640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65413ec80 0x7ff65413f300 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00e930 tx=0x7ff63c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff64be00640 1 -- 172.21.15.40:0/3633386644 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c017070 con 0x7ff65413ec80 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff64be00640 1 -- 172.21.15.40:0/3633386644 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63c00ca00 con 0x7ff65413ec80 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff65a800640 1 -- 172.21.15.40:0/3633386644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65413ff50 con 0x7ff65413ec80 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff64be00640 1 -- 172.21.15.40:0/3633386644 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c00cdb0 con 0x7ff65413ec80 2024-03-24T00:36:03.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.910+0000 7ff65a800640 1 -- 172.21.15.40:0/3633386644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6541560c0 con 0x7ff65413ec80 2024-03-24T00:36:03.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.914+0000 7ff65a800640 1 -- 172.21.15.40:0/3633386644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff654068170 con 0x7ff65413ec80 2024-03-24T00:36:03.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.914+0000 7ff64be00640 1 -- 172.21.15.40:0/3633386644 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff63c01a070 con 0x7ff65413ec80 2024-03-24T00:36:03.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.914+0000 7ff64be00640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff630067660 0x7ff630069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:03.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.914+0000 7ff64be00640 1 -- 172.21.15.40:0/3633386644 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff63c085920 con 0x7ff65413ec80 2024-03-24T00:36:03.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.914+0000 7ff653400640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff630067660 0x7ff630069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:03.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.914+0000 7ff653400640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff630067660 0x7ff630069b20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff644000f70 tx=0x7ff6440079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:03.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:03.918+0000 7ff64be00640 1 -- 172.21.15.40:0/3633386644 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff63c0569a0 con 0x7ff65413ec80 2024-03-24T00:36:04.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.138+0000 7ff65a800640 1 -- 172.21.15.40:0/3633386644 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff6540ce3f0 con 0x7ff65413ec80 2024-03-24T00:36:04.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.138+0000 7ff64be00640 1 -- 172.21.15.40:0/3633386644 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff63c0562a0 con 0x7ff65413ec80 2024-03-24T00:36:04.143 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:04.143 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:36:04.143 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-03-24T00:36:04.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lCkU6EhAA0q21F90epsGPK5x17s9NLw==", 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llkfOHhAAqKKFyEJzG0UxbLJMFPET8A==", 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:04.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lKMmwGRAAAKaOSnhTRbHIpPKXXkAe+Q==", 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l7quKJhAA8QsquiFFmj5H1G5yck0FpA==", 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAVdP9lVk3uBxAA9TC1OpeB5GIFlfJl5+ILBg==", 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:04.146 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYdP9lUc1IDhAAPwCV9eqL7bFibkfMFt9wPw==", 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAbdP9lwjyuFBAAJ1JAIAnLXAn03+lQPwwY9Q==", 2024-03-24T00:36:04.147 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAedP9lZ1bgGRAAuLAay48N7jXjj8/yaYwSqg==", 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAidP9lvyk4JhAA8w4/XBYHDLoqJ1zop9sUiw==", 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAldP9lcb9qLBAAXts6VU/4kVqUWS/WORV7zQ==", 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.149 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAodP9l8ipTMhAAjpagojh+woHyGvlQlIkHDw==", 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQArdP9lmGfpORAAPvow5nF/O7RLmuXEqNWENg==", 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.150 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw==", 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow r", 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-03-24T00:36:04.151 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lpIZrNBAAzpGWZuQ6ndjNMAOmAsebvQ==", 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l6gXMMBAAd6Oe6S73hj/WfLXTsYiEZw==", 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:36:04.152 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llFHCOhAATtv8gcL6BHPnxB1dHrg6vw==", 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAQdP9lmmd7KxAA4uk6aUKP6egYj6O1NgdadQ==", 2024-03-24T00:36:04.153 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lebNrBxAAvbcCAJhDQn/5Uk1dJ01oHw==", 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:04.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:04.155 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-03-24T00:36:04.155 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lEnZyAxAAW5Lrm7LZPTVfLCOY6As1PQ==", 2024-03-24T00:36:04.155 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:04.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:36:04.155 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:36:04.155 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:04.155 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.155 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:04.155 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-24T00:36:04.155 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:36:04.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.142+0000 7ff65a800640 1 -- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff630067660 msgr2=0x7ff630069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.142+0000 7ff65a800640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff630067660 0x7ff630069b20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff644000f70 tx=0x7ff6440079c0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.142+0000 7ff65a800640 1 -- 172.21.15.40:0/3633386644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65413ec80 msgr2=0x7ff65413f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.142+0000 7ff65a800640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65413ec80 0x7ff65413f300 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00e930 tx=0x7ff63c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.142+0000 7ff65a800640 1 -- 172.21.15.40:0/3633386644 shutdown_connections 2024-03-24T00:36:04.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.142+0000 7ff65a800640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff630067660 0x7ff630069b20 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.142+0000 7ff65a800640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff654141bc0 0x7ff65406df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.142+0000 7ff65a800640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff65413ec80 0x7ff65413f300 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.142+0000 7ff65a800640 1 --2- 172.21.15.40:0/3633386644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6540ce630 0x7ff654155b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.142+0000 7ff65a800640 1 -- 172.21.15.40:0/3633386644 >> 172.21.15.40:0/3633386644 conn(0x7ff6540b17b0 msgr2=0x7ff65415e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.142+0000 7ff65a800640 1 -- 172.21.15.40:0/3633386644 shutdown_connections 2024-03-24T00:36:04.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.142+0000 7ff65a800640 1 -- 172.21.15.40:0/3633386644 wait complete. 2024-03-24T00:36:04.165 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:36:04.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.302+0000 7f355ec00640 1 Processor -- start 2024-03-24T00:36:04.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.302+0000 7f355ec00640 1 -- start start 2024-03-24T00:36:04.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f355ec00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f355806e620 0x7f35580b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f355ec00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35580ccfa0 0x7f35580cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f355ec00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3558070f60 con 0x7f355806ea00 2024-03-24T00:36:04.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f355ec00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35580b4c00 con 0x7f355806e620 2024-03-24T00:36:04.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f355ec00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35580b3620 con 0x7f35580ccfa0 2024-03-24T00:36:04.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f3557e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f355806e620 0x7f35580b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f355cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35580ccfa0 0x7f35580cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f354fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f355806ea00 0x7f35580b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:56970/0 (socket says 172.21.15.40:56970) 2024-03-24T00:36:04.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f354fe00640 1 -- 172.21.15.40:0/2906200114 learned_addr learned my addr 172.21.15.40:0/2906200114 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:04.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f3557e00640 1 -- 172.21.15.40:0/2906200114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35580ccfa0 msgr2=0x7f35580cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:04.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f3557e00640 1 --2- 172.21.15.40:0/2906200114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35580ccfa0 0x7f35580cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f3557e00640 1 -- 172.21.15.40:0/2906200114 >> v1:172.21.15.53:6789/0 conn(0x7f355806ea00 legacy=0x7f35580b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.306+0000 7f3557e00640 1 -- 172.21.15.40:0/2906200114 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35580b3900 con 0x7f355806e620 2024-03-24T00:36:04.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f3557e00640 1 --2- 172.21.15.40:0/2906200114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f355806e620 0x7f35580b46c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f35480097c0 tx=0x7f354801c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2793ab2f0149347b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f3557400640 1 -- 172.21.15.40:0/2906200114 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35480b0070 con 0x7f355806e620 2024-03-24T00:36:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f3557400640 1 -- 172.21.15.40:0/2906200114 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35480ab600 con 0x7f355806e620 2024-03-24T00:36:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f3557400640 1 -- 172.21.15.40:0/2906200114 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35480ab9d0 con 0x7f355806e620 2024-03-24T00:36:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f355ec00640 1 -- 172.21.15.40:0/2906200114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f355806e620 msgr2=0x7f35580b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f355ec00640 1 --2- 172.21.15.40:0/2906200114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f355806e620 0x7f35580b46c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f35480097c0 tx=0x7f354801c3e0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f355ec00640 1 -- 172.21.15.40:0/2906200114 shutdown_connections 2024-03-24T00:36:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f355ec00640 1 --2- 172.21.15.40:0/2906200114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35580ccfa0 0x7f35580cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f355ec00640 1 --2- 172.21.15.40:0/2906200114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f355806e620 0x7f35580b46c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f355ec00640 1 -- 172.21.15.40:0/2906200114 >> 172.21.15.40:0/2906200114 conn(0x7f35580b17b0 msgr2=0x7f35580b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f355ec00640 1 -- 172.21.15.40:0/2906200114 shutdown_connections 2024-03-24T00:36:04.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f355ec00640 1 -- 172.21.15.40:0/2906200114 wait complete. 2024-03-24T00:36:04.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f355ec00640 1 Processor -- start 2024-03-24T00:36:04.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.310+0000 7f355ec00640 1 -- start start 2024-03-24T00:36:04.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f355ec00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f355806e620 0x7f355813de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f355ec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355806ea00 0x7f355815cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f355ec00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35580ccfa0 0x7f355815d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f355ec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3558144920 con 0x7f355806ea00 2024-03-24T00:36:04.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f355ec00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355806ee10 con 0x7f355806e620 2024-03-24T00:36:04.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f355ec00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3558165ac0 con 0x7f35580ccfa0 2024-03-24T00:36:04.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f3557e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f355806e620 0x7f355813de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f3557e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f355806e620 0x7f355813de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39454/0 (socket says 172.21.15.40:39454) 2024-03-24T00:36:04.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f3557e00640 1 -- 172.21.15.40:0/2652993416 learned_addr learned my addr 172.21.15.40:0/2652993416 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:04.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f354fe00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355806ea00 0x7f355815cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f355cc00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35580ccfa0 0x7f355815d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f3557e00640 1 -- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35580ccfa0 msgr2=0x7f355815d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f3557e00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35580ccfa0 0x7f355815d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f3557e00640 1 -- 172.21.15.40:0/2652993416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355806ea00 msgr2=0x7f355815cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f3557e00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355806ea00 0x7f355815cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f3557e00640 1 -- 172.21.15.40:0/2652993416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355815e230 con 0x7f355806e620 2024-03-24T00:36:04.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f3557e00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f355806e620 0x7f355813de20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f354801a010 tx=0x7f3548019180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:04.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f3554c00640 1 -- 172.21.15.40:0/2652993416 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35480b8020 con 0x7f355806e620 2024-03-24T00:36:04.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f3554c00640 1 -- 172.21.15.40:0/2652993416 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35480b4410 con 0x7f355806e620 2024-03-24T00:36:04.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f3554c00640 1 -- 172.21.15.40:0/2652993416 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35480b0040 con 0x7f355806e620 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f355ec00640 1 -- 172.21.15.40:0/2652993416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355815be30 con 0x7f355806e620 2024-03-24T00:36:04.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f355ec00640 1 -- 172.21.15.40:0/2652993416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355815c3b0 con 0x7f355806e620 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.314+0000 7f355ec00640 1 -- 172.21.15.40:0/2652993416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3558068170 con 0x7f355806e620 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.318+0000 7f3554c00640 1 -- 172.21.15.40:0/2652993416 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f35480bb020 con 0x7f355806e620 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.318+0000 7f3554c00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3534067660 0x7f3534069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.318+0000 7f3554c00640 1 -- 172.21.15.40:0/2652993416 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f354811ef80 con 0x7f355806e620 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.318+0000 7f354fe00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3534067660 0x7f3534069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.318+0000 7f354fe00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3534067660 0x7f3534069b20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f355813e5f0 tx=0x7f3540007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:04.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.322+0000 7f3554c00640 1 -- 172.21.15.40:0/2652993416 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f35480c2030 con 0x7f355806e620 2024-03-24T00:36:04.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.486+0000 7f355ec00640 1 -- 172.21.15.40:0/2652993416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3558072f50 con 0x7f355806e620 2024-03-24T00:36:04.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.486+0000 7f3554c00640 1 -- 172.21.15.40:0/2652993416 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+12832 (secure 0 0 0) 0x7f3548016090 con 0x7f355806e620 2024-03-24T00:36:04.490 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:04.490 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":79,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:36:03.730342+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:36:04.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.486+0000 7f355ec00640 1 -- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3534067660 msgr2=0x7f3534069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.486+0000 7f355ec00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3534067660 0x7f3534069b20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f355813e5f0 tx=0x7f3540007e90 comp rx=0 tx=0).stop 2024-03-24T00:36:04.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.486+0000 7f355ec00640 1 -- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f355806e620 msgr2=0x7f355813de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.486+0000 7f355ec00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f355806e620 0x7f355813de20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f354801a010 tx=0x7f3548019180 comp rx=0 tx=0).stop 2024-03-24T00:36:04.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.490+0000 7f355ec00640 1 -- 172.21.15.40:0/2652993416 shutdown_connections 2024-03-24T00:36:04.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.490+0000 7f355ec00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3534067660 0x7f3534069b20 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.490+0000 7f355ec00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35580ccfa0 0x7f355815d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.490+0000 7f355ec00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f355806ea00 0x7f355815cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.490+0000 7f355ec00640 1 --2- 172.21.15.40:0/2652993416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f355806e620 0x7f355813de20 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.490+0000 7f355ec00640 1 -- 172.21.15.40:0/2652993416 >> 172.21.15.40:0/2652993416 conn(0x7f35580b17b0 msgr2=0x7f35580b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:04.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.490+0000 7f355ec00640 1 -- 172.21.15.40:0/2652993416 shutdown_connections 2024-03-24T00:36:04.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.490+0000 7f355ec00640 1 -- 172.21.15.40:0/2652993416 wait complete. 2024-03-24T00:36:04.511 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:36:04.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:36:04.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e94400640 1 Processor -- start 2024-03-24T00:36:04.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e94400640 1 -- start start 2024-03-24T00:36:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e94400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3e8c15ac60 0x7f3e8c0b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e94400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e8c06cb80 0x7f3e8c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e94400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8c0b3260 con 0x7f3e8c0b4840 2024-03-24T00:36:04.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e94400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8c0b3e70 con 0x7f3e8c15ac60 2024-03-24T00:36:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e94400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c0b3540 con 0x7f3e8c06cb80 2024-03-24T00:36:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e92400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3e8c0b4840 0x7f3e8c0ccd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59892/0 (socket says 172.21.15.40:59892) 2024-03-24T00:36:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e92400640 1 -- 172.21.15.40:0/768008099 learned_addr learned my addr 172.21.15.40:0/768008099 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e91a00640 1 --2- 172.21.15.40:0/768008099 >> v2:172.21.15.53:3300/0 conn(0x7f3e8c15ac60 0x7f3e8c0b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e91000640 1 --2- 172.21.15.40:0/768008099 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e8c06cb80 0x7f3e8c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e8be00640 1 -- 172.21.15.40:0/768008099 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129893813 0 0) 0x7f3e8c0b3260 con 0x7f3e8c0b4840 2024-03-24T00:36:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e8be00640 1 -- 172.21.15.40:0/768008099 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e74003530 con 0x7f3e8c0b4840 2024-03-24T00:36:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e91a00640 1 -- 172.21.15.40:0/768008099 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e8c06cb80 msgr2=0x7f3e8c0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e91a00640 1 --2- 172.21.15.40:0/768008099 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e8c06cb80 0x7f3e8c0b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e91a00640 1 -- 172.21.15.40:0/768008099 >> v1:172.21.15.40:6789/0 conn(0x7f3e8c0b4840 legacy=0x7f3e8c0ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e91a00640 1 -- 172.21.15.40:0/768008099 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c0b3820 con 0x7f3e8c15ac60 2024-03-24T00:36:04.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.654+0000 7f3e91000640 1 --2- 172.21.15.40:0/768008099 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e8c06cb80 0x7f3e8c0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e91a00640 1 --2- 172.21.15.40:0/768008099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e8c15ac60 0x7f3e8c0b2960 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3e800097c0 tx=0x7f3e8001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cf21188bcf3928a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e8be00640 1 -- 172.21.15.40:0/768008099 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e800b0070 con 0x7f3e8c15ac60 2024-03-24T00:36:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e8be00640 1 -- 172.21.15.40:0/768008099 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e800ab600 con 0x7f3e8c15ac60 2024-03-24T00:36:04.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e8be00640 1 -- 172.21.15.40:0/768008099 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e800ab9d0 con 0x7f3e8c15ac60 2024-03-24T00:36:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e94400640 1 -- 172.21.15.40:0/768008099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e8c15ac60 msgr2=0x7f3e8c0b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e94400640 1 --2- 172.21.15.40:0/768008099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e8c15ac60 0x7f3e8c0b2960 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3e800097c0 tx=0x7f3e8001c3e0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e94400640 1 -- 172.21.15.40:0/768008099 shutdown_connections 2024-03-24T00:36:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e94400640 1 --2- 172.21.15.40:0/768008099 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e8c06cb80 0x7f3e8c0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e94400640 1 --2- 172.21.15.40:0/768008099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e8c15ac60 0x7f3e8c0b2960 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e94400640 1 -- 172.21.15.40:0/768008099 >> 172.21.15.40:0/768008099 conn(0x7f3e8c0b1660 msgr2=0x7f3e8c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:04.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e94400640 1 -- 172.21.15.40:0/768008099 shutdown_connections 2024-03-24T00:36:04.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e94400640 1 -- 172.21.15.40:0/768008099 wait complete. 2024-03-24T00:36:04.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.658+0000 7f3e94400640 1 Processor -- start 2024-03-24T00:36:04.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e94400640 1 -- start start 2024-03-24T00:36:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e94400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e8c06cb80 0x7f3e8c1515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e94400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e8c0b4840 0x7f3e8c147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e94400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e8c15ac60 0x7f3e8c0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e94400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c167a40 con 0x7f3e8c15ac60 2024-03-24T00:36:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e94400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c1678c0 con 0x7f3e8c06cb80 2024-03-24T00:36:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e94400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8c167bc0 con 0x7f3e8c0b4840 2024-03-24T00:36:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e92400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e8c15ac60 0x7f3e8c0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e92400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e8c15ac60 0x7f3e8c0ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49742/0 (socket says 172.21.15.40:49742) 2024-03-24T00:36:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e92400640 1 -- 172.21.15.40:0/3777875018 learned_addr learned my addr 172.21.15.40:0/3777875018 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e91000640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e8c0b4840 0x7f3e8c147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e91a00640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e8c06cb80 0x7f3e8c1515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e92400640 1 -- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e8c0b4840 msgr2=0x7f3e8c147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e92400640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e8c0b4840 0x7f3e8c147550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e92400640 1 -- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e8c06cb80 msgr2=0x7f3e8c1515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e92400640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e8c06cb80 0x7f3e8c1515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e92400640 1 -- 172.21.15.40:0/3777875018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c151dc0 con 0x7f3e8c15ac60 2024-03-24T00:36:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e92400640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e8c15ac60 0x7f3e8c0ad8e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400cb40 tx=0x7f3e8400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e89600640 1 -- 172.21.15.40:0/3777875018 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84017070 con 0x7f3e8c15ac60 2024-03-24T00:36:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e89600640 1 -- 172.21.15.40:0/3777875018 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8400aca0 con 0x7f3e8c15ac60 2024-03-24T00:36:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e94400640 1 -- 172.21.15.40:0/3777875018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8c158270 con 0x7f3e8c15ac60 2024-03-24T00:36:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e89600640 1 -- 172.21.15.40:0/3777875018 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e84012620 con 0x7f3e8c15ac60 2024-03-24T00:36:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e94400640 1 -- 172.21.15.40:0/3777875018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8c158510 con 0x7f3e8c15ac60 2024-03-24T00:36:04.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.662+0000 7f3e94400640 1 -- 172.21.15.40:0/3777875018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8c068170 con 0x7f3e8c15ac60 2024-03-24T00:36:04.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.666+0000 7f3e89600640 1 -- 172.21.15.40:0/3777875018 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3e8401a070 con 0x7f3e8c15ac60 2024-03-24T00:36:04.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.666+0000 7f3e89600640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e6c067660 0x7f3e6c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:04.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.666+0000 7f3e89600640 1 -- 172.21.15.40:0/3777875018 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3e84085410 con 0x7f3e8c15ac60 2024-03-24T00:36:04.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.666+0000 7f3e91a00640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e6c067660 0x7f3e6c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:04.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.666+0000 7f3e91a00640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e6c067660 0x7f3e6c069b20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3e800ad000 tx=0x7f3e800a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:04.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.670+0000 7f3e89600640 1 -- 172.21.15.40:0/3777875018 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3e84055c70 con 0x7f3e8c15ac60 2024-03-24T00:36:04.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 -- 172.21.15.40:0/3777875018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e8c0690c0 con 0x7f3e8c15ac60 2024-03-24T00:36:04.837 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e89600640 1 -- 172.21.15.40:0/3777875018 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3e84059d10 con 0x7f3e8c15ac60 2024-03-24T00:36:04.837 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:36:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 -- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e6c067660 msgr2=0x7f3e6c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e6c067660 0x7f3e6c069b20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3e800ad000 tx=0x7f3e800a6000 comp rx=0 tx=0).stop 2024-03-24T00:36:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 -- 172.21.15.40:0/3777875018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e8c15ac60 msgr2=0x7f3e8c0ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e8c15ac60 0x7f3e8c0ad8e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f3e8400cb40 tx=0x7f3e8400a060 comp rx=0 tx=0).stop 2024-03-24T00:36:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 -- 172.21.15.40:0/3777875018 shutdown_connections 2024-03-24T00:36:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e6c067660 0x7f3e6c069b20 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e8c15ac60 0x7f3e8c0ad8e0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e8c0b4840 0x7f3e8c147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 --2- 172.21.15.40:0/3777875018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e8c06cb80 0x7f3e8c1515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 -- 172.21.15.40:0/3777875018 >> 172.21.15.40:0/3777875018 conn(0x7f3e8c0b1660 msgr2=0x7f3e8c0b2260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 -- 172.21.15.40:0/3777875018 shutdown_connections 2024-03-24T00:36:04.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.834+0000 7f3e94400640 1 -- 172.21.15.40:0/3777875018 wait complete. 2024-03-24T00:36:04.860 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:36:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc82000640 1 Processor -- start 2024-03-24T00:36:05.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc82000640 1 -- start start 2024-03-24T00:36:05.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc82000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc7c15bf30 0x7fcc7c15fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc82000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcc7c0cfa30 0x7fcc7c0cc320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc82000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c0d1110 con 0x7fcc7c0d0ca0 2024-03-24T00:36:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc82000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7c0cbbd0 con 0x7fcc7c15bf30 2024-03-24T00:36:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc82000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7c1609f0 con 0x7fcc7c0cfa30 2024-03-24T00:36:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc7b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc7c15bf30 0x7fcc7c15fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc7b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc7c15bf30 0x7fcc7c15fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49754/0 (socket says 172.21.15.40:49754) 2024-03-24T00:36:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc7b400640 1 -- 172.21.15.40:0/2300216674 learned_addr learned my addr 172.21.15.40:0/2300216674 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc7be00640 1 --2- 172.21.15.40:0/2300216674 >> v2:172.21.15.53:3300/0 conn(0x7fcc7c0cfa30 0x7fcc7c0cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc7b400640 1 -- 172.21.15.40:0/2300216674 >> v2:172.21.15.53:3300/0 conn(0x7fcc7c0cfa30 msgr2=0x7fcc7c0cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc7b400640 1 --2- 172.21.15.40:0/2300216674 >> v2:172.21.15.53:3300/0 conn(0x7fcc7c0cfa30 0x7fcc7c0cc320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc7b400640 1 -- 172.21.15.40:0/2300216674 >> v1:172.21.15.53:6789/0 conn(0x7fcc7c0d0ca0 legacy=0x7fcc7c0cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:05.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:04.998+0000 7fcc7b400640 1 -- 172.21.15.40:0/2300216674 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c160150 con 0x7fcc7c15bf30 2024-03-24T00:36:05.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.002+0000 7fcc7b400640 1 --2- 172.21.15.40:0/2300216674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c15bf30 0x7fcc7c15fac0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c0c7360 tx=0x7fcc6c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8c1d4b1380d89 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.002+0000 7fcc7aa00640 1 -- 172.21.15.40:0/2300216674 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c0aa650 con 0x7fcc7c15bf30 2024-03-24T00:36:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.002+0000 7fcc7aa00640 1 -- 172.21.15.40:0/2300216674 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c0af070 con 0x7fcc7c15bf30 2024-03-24T00:36:05.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.002+0000 7fcc7aa00640 1 -- 172.21.15.40:0/2300216674 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c0aaa10 con 0x7fcc7c15bf30 2024-03-24T00:36:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.002+0000 7fcc82000640 1 -- 172.21.15.40:0/2300216674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c15bf30 msgr2=0x7fcc7c15fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.002+0000 7fcc82000640 1 --2- 172.21.15.40:0/2300216674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c15bf30 0x7fcc7c15fac0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c0c7360 tx=0x7fcc6c01c4f0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.002+0000 7fcc82000640 1 -- 172.21.15.40:0/2300216674 shutdown_connections 2024-03-24T00:36:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.002+0000 7fcc82000640 1 --2- 172.21.15.40:0/2300216674 >> v2:172.21.15.53:3300/0 conn(0x7fcc7c0cfa30 0x7fcc7c0cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.002+0000 7fcc82000640 1 --2- 172.21.15.40:0/2300216674 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c15bf30 0x7fcc7c15fac0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.002+0000 7fcc82000640 1 -- 172.21.15.40:0/2300216674 >> 172.21.15.40:0/2300216674 conn(0x7fcc7c072890 msgr2=0x7fcc7c072ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.002+0000 7fcc82000640 1 -- 172.21.15.40:0/2300216674 shutdown_connections 2024-03-24T00:36:05.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.002+0000 7fcc82000640 1 -- 172.21.15.40:0/2300216674 wait complete. 2024-03-24T00:36:05.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc82000640 1 Processor -- start 2024-03-24T00:36:05.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc82000640 1 -- start start 2024-03-24T00:36:05.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc82000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc7c0cfa30 0x7fcc7c0c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc82000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc7c0d0ca0 0x7fcc7c1f7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc82000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c15bf30 0x7fcc7c0c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc82000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0cd070 con 0x7fcc7c15bf30 2024-03-24T00:36:05.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc82000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c15c310 con 0x7fcc7c0cfa30 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc82000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0cd1f0 con 0x7fcc7c0d0ca0 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc7be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c15bf30 0x7fcc7c0c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc7be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c15bf30 0x7fcc7c0c8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49766/0 (socket says 172.21.15.40:49766) 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc7be00640 1 -- 172.21.15.40:0/3312548886 learned_addr learned my addr 172.21.15.40:0/3312548886 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc7b400640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc7c0cfa30 0x7fcc7c0c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc72a00640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc7c0d0ca0 0x7fcc7c1f7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc7be00640 1 -- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc7c0d0ca0 msgr2=0x7fcc7c1f7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc7be00640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc7c0d0ca0 0x7fcc7c1f7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc7be00640 1 -- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc7c0cfa30 msgr2=0x7fcc7c0c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc7be00640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc7c0cfa30 0x7fcc7c0c73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc7be00640 1 -- 172.21.15.40:0/3312548886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c1f78c0 con 0x7fcc7c15bf30 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc7be00640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c15bf30 0x7fcc7c0c8a50 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fcc7400d800 tx=0x7fcc7400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc73e00640 1 -- 172.21.15.40:0/3312548886 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7400b6c0 con 0x7fcc7c15bf30 2024-03-24T00:36:05.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc73e00640 1 -- 172.21.15.40:0/3312548886 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc74019070 con 0x7fcc7c15bf30 2024-03-24T00:36:05.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc73e00640 1 -- 172.21.15.40:0/3312548886 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7400baa0 con 0x7fcc7c15bf30 2024-03-24T00:36:05.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc82000640 1 -- 172.21.15.40:0/3312548886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7c0c79b0 con 0x7fcc7c15bf30 2024-03-24T00:36:05.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc82000640 1 -- 172.21.15.40:0/3312548886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c0c7f70 con 0x7fcc7c15bf30 2024-03-24T00:36:05.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.006+0000 7fcc82000640 1 -- 172.21.15.40:0/3312548886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7c068170 con 0x7fcc7c15bf30 2024-03-24T00:36:05.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.010+0000 7fcc73e00640 1 -- 172.21.15.40:0/3312548886 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcc7400bc40 con 0x7fcc7c15bf30 2024-03-24T00:36:05.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.010+0000 7fcc73e00640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc58067660 0x7fcc58069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.010+0000 7fcc73e00640 1 -- 172.21.15.40:0/3312548886 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcc74085930 con 0x7fcc7c15bf30 2024-03-24T00:36:05.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.010+0000 7fcc7b400640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc58067660 0x7fcc58069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.010+0000 7fcc7b400640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc58067660 0x7fcc58069b20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0ae000 tx=0x7fcc6c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.014+0000 7fcc73e00640 1 -- 172.21.15.40:0/3312548886 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcc7c1f78c0 con 0x7fcc7c15bf30 2024-03-24T00:36:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 -- 172.21.15.40:0/3312548886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcc7c0cc430 con 0x7fcc7c15bf30 2024-03-24T00:36:05.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc73e00640 1 -- 172.21.15.40:0/3312548886 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcc740568f0 con 0x7fcc7c15bf30 2024-03-24T00:36:05.181 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:05.181 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-03-24T00:36:05.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:05.182 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-03-24T00:36:05.182 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-03-24T00:36:05.182 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:05.182 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-03-24T00:36:05.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 -- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc58067660 msgr2=0x7fcc58069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc58067660 0x7fcc58069b20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c0ae000 tx=0x7fcc6c0a7000 comp rx=0 tx=0).stop 2024-03-24T00:36:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 -- 172.21.15.40:0/3312548886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c15bf30 msgr2=0x7fcc7c0c8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c15bf30 0x7fcc7c0c8a50 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fcc7400d800 tx=0x7fcc7400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 -- 172.21.15.40:0/3312548886 shutdown_connections 2024-03-24T00:36:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc58067660 0x7fcc58069b20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc7c15bf30 0x7fcc7c0c8a50 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc7c0d0ca0 0x7fcc7c1f7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 --2- 172.21.15.40:0/3312548886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc7c0cfa30 0x7fcc7c0c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 -- 172.21.15.40:0/3312548886 >> 172.21.15.40:0/3312548886 conn(0x7fcc7c072890 msgr2=0x7fcc7c1666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 -- 172.21.15.40:0/3312548886 shutdown_connections 2024-03-24T00:36:05.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.178+0000 7fcc82000640 1 -- 172.21.15.40:0/3312548886 wait complete. 2024-03-24T00:36:05.202 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:36:05.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.342+0000 7f5e29e00640 1 Processor -- start 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.342+0000 7f5e29e00640 1 -- start start 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e29e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e240b42d0 0x7f5e240d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e29e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2415e3e0 con 0x7f5e241628d0 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e29e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e240b5f10 con 0x7f5e2415e000 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e29e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e24161ba0 con 0x7f5e240b42d0 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e1be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5e241628d0 0x7f5e2406dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59896/0 (socket says 172.21.15.40:59896) 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e1be00640 1 -- 172.21.15.40:0/911725175 learned_addr learned my addr 172.21.15.40:0/911725175 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e23400640 1 --2- 172.21.15.40:0/911725175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e240b42d0 0x7f5e240d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260519932 0 0) 0x7f5e2415e3e0 con 0x7f5e241628d0 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e08003530 con 0x7f5e241628d0 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589644110 0 0) 0x7f5e240b5f10 con 0x7f5e2415e000 2024-03-24T00:36:05.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e2415e3e0 con 0x7f5e2415e000 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2497270931 0 0) 0x7f5e08003530 con 0x7f5e241628d0 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e240b42d0 msgr2=0x7f5e240d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 --2- 172.21.15.40:0/911725175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e240b42d0 0x7f5e240d16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 >> v1:172.21.15.53:6789/0 conn(0x7f5e2415e000 legacy=0x7f5e240b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e24161df0 con 0x7f5e241628d0 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f5e0c001600 con 0x7f5e241628d0 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 >> v1:172.21.15.40:6789/0 conn(0x7f5e241628d0 legacy=0x7f5e2406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 --2- 172.21.15.40:0/911725175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e08003b80 0x7f5e08003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 --2- 172.21.15.40:0/911725175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e240b42d0 0x7f5e08004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 --2- 172.21.15.40:0/911725175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e08004d50 0x7f5e080051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e24161ba0 con 0x7f5e08004d50 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0800bab0 con 0x7f5e08003b80 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0800bcb0 con 0x7f5e240b42d0 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e23400640 1 --2- 172.21.15.40:0/911725175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e08004d50 0x7f5e080051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e22a00640 1 --2- 172.21.15.40:0/911725175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e08003b80 0x7f5e08003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e1be00640 1 --2- 172.21.15.40:0/911725175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e240b42d0 0x7f5e08004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e23400640 1 -- 172.21.15.40:0/911725175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e240b42d0 msgr2=0x7f5e08004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e23400640 1 --2- 172.21.15.40:0/911725175 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e240b42d0 0x7f5e08004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e23400640 1 -- 172.21.15.40:0/911725175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e08003b80 msgr2=0x7f5e08003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e23400640 1 --2- 172.21.15.40:0/911725175 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e08003b80 0x7f5e08003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.346+0000 7f5e23400640 1 -- 172.21.15.40:0/911725175 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e08003530 con 0x7f5e08004d50 2024-03-24T00:36:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.350+0000 7f5e23400640 1 --2- 172.21.15.40:0/911725175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e08004d50 0x7f5e080051d0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c009fc0 tx=0x7f5e1c01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d5569efb00cc32 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.350+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0b2070 con 0x7f5e08004d50 2024-03-24T00:36:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.350+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c0a5c90 con 0x7f5e08004d50 2024-03-24T00:36:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.350+0000 7f5e22000640 1 -- 172.21.15.40:0/911725175 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c009c10 con 0x7f5e08004d50 2024-03-24T00:36:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.350+0000 7f5e29e00640 1 -- 172.21.15.40:0/911725175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e08004d50 msgr2=0x7f5e080051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.350+0000 7f5e29e00640 1 --2- 172.21.15.40:0/911725175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e08004d50 0x7f5e080051d0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c009fc0 tx=0x7f5e1c01cc80 comp rx=0 tx=0).stop 2024-03-24T00:36:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.350+0000 7f5e23400640 1 -- 172.21.15.40:0/911725175 reap_dead start 2024-03-24T00:36:05.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.350+0000 7f5e29e00640 1 -- 172.21.15.40:0/911725175 shutdown_connections 2024-03-24T00:36:05.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.350+0000 7f5e29e00640 1 -- 172.21.15.40:0/911725175 >> 172.21.15.40:0/911725175 conn(0x7f5e240d0bb0 msgr2=0x7f5e240d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:05.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.350+0000 7f5e29e00640 1 -- 172.21.15.40:0/911725175 shutdown_connections 2024-03-24T00:36:05.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.350+0000 7f5e29e00640 1 -- 172.21.15.40:0/911725175 wait complete. 2024-03-24T00:36:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e29e00640 1 Processor -- start 2024-03-24T00:36:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e29e00640 1 -- start start 2024-03-24T00:36:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e29e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e08004d50 0x7f5e2414f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e29e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e24150470 0x7f5e24160590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e29e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e24151d30 0x7f5e241521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e29e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e240b5b30 con 0x7f5e24151d30 2024-03-24T00:36:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e29e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e241623b0 con 0x7f5e08004d50 2024-03-24T00:36:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e29e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e240b5cb0 con 0x7f5e24150470 2024-03-24T00:36:05.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e23400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e24151d30 0x7f5e241521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e23400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e24151d30 0x7f5e241521e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49774/0 (socket says 172.21.15.40:49774) 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e23400640 1 -- 172.21.15.40:0/1218209727 learned_addr learned my addr 172.21.15.40:0/1218209727 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e22a00640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e08004d50 0x7f5e2414f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e1be00640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e24150470 0x7f5e24160590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e23400640 1 -- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e24150470 msgr2=0x7f5e24160590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e23400640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e24150470 0x7f5e24160590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e23400640 1 -- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e08004d50 msgr2=0x7f5e2414f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e23400640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e08004d50 0x7f5e2414f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e23400640 1 -- 172.21.15.40:0/1218209727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e24150a40 con 0x7f5e24151d30 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e22a00640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e08004d50 0x7f5e2414f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e23400640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e24151d30 0x7f5e241521e0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c002240 tx=0x7f5e1c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e1b400640 1 -- 172.21.15.40:0/1218209727 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0bd070 con 0x7f5e24151d30 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e1b400640 1 -- 172.21.15.40:0/1218209727 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1c0a5a90 con 0x7f5e24151d30 2024-03-24T00:36:05.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e1b400640 1 -- 172.21.15.40:0/1218209727 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1c0b2040 con 0x7f5e24151d30 2024-03-24T00:36:05.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e29e00640 1 -- 172.21.15.40:0/1218209727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e24150d20 con 0x7f5e24151d30 2024-03-24T00:36:05.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e29e00640 1 -- 172.21.15.40:0/1218209727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e241512e0 con 0x7f5e24151d30 2024-03-24T00:36:05.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.354+0000 7f5e29e00640 1 -- 172.21.15.40:0/1218209727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e24068170 con 0x7f5e24151d30 2024-03-24T00:36:05.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.358+0000 7f5e1b400640 1 -- 172.21.15.40:0/1218209727 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5e1c0c0070 con 0x7f5e24151d30 2024-03-24T00:36:05.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.358+0000 7f5e1b400640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e00067660 0x7f5e00069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.358+0000 7f5e1b400640 1 -- 172.21.15.40:0/1218209727 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5e1c11f180 con 0x7f5e24151d30 2024-03-24T00:36:05.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.358+0000 7f5e22a00640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e00067660 0x7f5e00069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.358+0000 7f5e22a00640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e00067660 0x7f5e00069b20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5e240b4d40 tx=0x7f5e14009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.362+0000 7f5e1b400640 1 -- 172.21.15.40:0/1218209727 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5e1c016990 con 0x7f5e24151d30 2024-03-24T00:36:05.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.522+0000 7f5e29e00640 1 -- 172.21.15.40:0/1218209727 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5e24067b90 con 0x7f5e24151d30 2024-03-24T00:36:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.746+0000 7f5e1b400640 1 -- 172.21.15.40:0/1218209727 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5e1c0a5c60 con 0x7f5e24151d30 2024-03-24T00:36:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-03-24T00:36:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.750+0000 7f5e29e00640 1 -- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e00067660 msgr2=0x7f5e00069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.750+0000 7f5e29e00640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e00067660 0x7f5e00069b20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5e240b4d40 tx=0x7f5e14009040 comp rx=0 tx=0).stop 2024-03-24T00:36:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.750+0000 7f5e29e00640 1 -- 172.21.15.40:0/1218209727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e24151d30 msgr2=0x7f5e241521e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.750+0000 7f5e29e00640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e24151d30 0x7f5e241521e0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f5e1c002240 tx=0x7f5e1c009750 comp rx=0 tx=0).stop 2024-03-24T00:36:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.750+0000 7f5e29e00640 1 -- 172.21.15.40:0/1218209727 shutdown_connections 2024-03-24T00:36:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.750+0000 7f5e29e00640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e00067660 0x7f5e00069b20 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.750+0000 7f5e29e00640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e24151d30 0x7f5e241521e0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.750+0000 7f5e29e00640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e24150470 0x7f5e24160590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.750+0000 7f5e29e00640 1 --2- 172.21.15.40:0/1218209727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e08004d50 0x7f5e2414f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.750+0000 7f5e29e00640 1 -- 172.21.15.40:0/1218209727 >> 172.21.15.40:0/1218209727 conn(0x7f5e240d0bb0 msgr2=0x7f5e2406e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.750+0000 7f5e29e00640 1 -- 172.21.15.40:0/1218209727 shutdown_connections 2024-03-24T00:36:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.750+0000 7f5e29e00640 1 -- 172.21.15.40:0/1218209727 wait complete. 2024-03-24T00:36:05.777 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:36:05.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6f6600640 1 Processor -- start 2024-03-24T00:36:05.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6f6600640 1 -- start start 2024-03-24T00:36:05.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6f6600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6f00ae6b0 0x7fe6f015a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6f6600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe6f00b3fc0 0x7fe6f00b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6f6600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f0070ed0 con 0x7fe6f00aec70 2024-03-24T00:36:05.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6f6600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f00cf240 con 0x7fe6f00ae6b0 2024-03-24T00:36:05.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6f6600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f00cf520 con 0x7fe6f00b3fc0 2024-03-24T00:36:05.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6e7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe6f00ae6b0 0x7fe6f015a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6ef400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe6f00aec70 0x7fe6f00cc4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:59910/0 (socket says 172.21.15.40:59910) 2024-03-24T00:36:05.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6ef400640 1 -- 172.21.15.40:0/1286600746 learned_addr learned my addr 172.21.15.40:0/1286600746 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:05.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6efe00640 1 --2- 172.21.15.40:0/1286600746 >> v2:172.21.15.53:3300/0 conn(0x7fe6f00b3fc0 0x7fe6f00b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6e7e00640 1 -- 172.21.15.40:0/1286600746 >> v2:172.21.15.53:3300/0 conn(0x7fe6f00b3fc0 msgr2=0x7fe6f00b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:05.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6e7e00640 1 --2- 172.21.15.40:0/1286600746 >> v2:172.21.15.53:3300/0 conn(0x7fe6f00b3fc0 0x7fe6f00b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6e7e00640 1 -- 172.21.15.40:0/1286600746 >> v1:172.21.15.40:6789/0 conn(0x7fe6f00aec70 legacy=0x7fe6f00cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6e7e00640 1 -- 172.21.15.40:0/1286600746 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f00b3aa0 con 0x7fe6f00ae6b0 2024-03-24T00:36:05.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.922+0000 7fe6eea00640 1 -- 172.21.15.40:0/1286600746 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144303600 0 0) 0x7fe6f0070ed0 con 0x7fe6f00aec70 2024-03-24T00:36:05.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.926+0000 7fe6e7e00640 1 --2- 172.21.15.40:0/1286600746 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6f00ae6b0 0x7fe6f015a560 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00ae640 tx=0x7fe6d801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a888df8ec21ce9f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.926+0000 7fe6eea00640 1 -- 172.21.15.40:0/1286600746 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d80b2070 con 0x7fe6f00ae6b0 2024-03-24T00:36:05.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.926+0000 7fe6eea00640 1 -- 172.21.15.40:0/1286600746 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d80ad680 con 0x7fe6f00ae6b0 2024-03-24T00:36:05.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.926+0000 7fe6eea00640 1 -- 172.21.15.40:0/1286600746 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d80ada50 con 0x7fe6f00ae6b0 2024-03-24T00:36:05.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.926+0000 7fe6f6600640 1 -- 172.21.15.40:0/1286600746 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6f00ae6b0 msgr2=0x7fe6f015a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.926+0000 7fe6f6600640 1 --2- 172.21.15.40:0/1286600746 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6f00ae6b0 0x7fe6f015a560 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00ae640 tx=0x7fe6d801c360 comp rx=0 tx=0).stop 2024-03-24T00:36:05.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.926+0000 7fe6f6600640 1 -- 172.21.15.40:0/1286600746 shutdown_connections 2024-03-24T00:36:05.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.926+0000 7fe6f6600640 1 --2- 172.21.15.40:0/1286600746 >> v2:172.21.15.53:3300/0 conn(0x7fe6f00b3fc0 0x7fe6f00b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.926+0000 7fe6f6600640 1 --2- 172.21.15.40:0/1286600746 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6f00ae6b0 0x7fe6f015a560 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.926+0000 7fe6f6600640 1 -- 172.21.15.40:0/1286600746 >> 172.21.15.40:0/1286600746 conn(0x7fe6f00b1660 msgr2=0x7fe6f00b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:05.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.926+0000 7fe6f6600640 1 -- 172.21.15.40:0/1286600746 shutdown_connections 2024-03-24T00:36:05.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.926+0000 7fe6f6600640 1 -- 172.21.15.40:0/1286600746 wait complete. 2024-03-24T00:36:05.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6f6600640 1 Processor -- start 2024-03-24T00:36:05.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6f6600640 1 -- start start 2024-03-24T00:36:05.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6f6600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00ae6b0 0x7fe6f015d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6f6600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe6f00aec70 0x7fe6f00af810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6f6600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6f00b3fc0 0x7fe6f014ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6f6600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f006d6b0 con 0x7fe6f00b3fc0 2024-03-24T00:36:05.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6f6600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f006d530 con 0x7fe6f00ae6b0 2024-03-24T00:36:05.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6f6600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f006d830 con 0x7fe6f00aec70 2024-03-24T00:36:05.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6efe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6f00b3fc0 0x7fe6f014ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6efe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6f00b3fc0 0x7fe6f014ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49792/0 (socket says 172.21.15.40:49792) 2024-03-24T00:36:05.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6efe00640 1 -- 172.21.15.40:0/1517174578 learned_addr learned my addr 172.21.15.40:0/1517174578 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:05.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6ef400640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00ae6b0 0x7fe6f015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6e7e00640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe6f00aec70 0x7fe6f00af810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6efe00640 1 -- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe6f00aec70 msgr2=0x7fe6f00af810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6efe00640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe6f00aec70 0x7fe6f00af810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6efe00640 1 -- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00ae6b0 msgr2=0x7fe6f015d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:05.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6efe00640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00ae6b0 0x7fe6f015d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:05.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6efe00640 1 -- 172.21.15.40:0/1517174578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f00b03e0 con 0x7fe6f00b3fc0 2024-03-24T00:36:05.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6efe00640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6f00b3fc0 0x7fe6f014ffc0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009dd0 tx=0x7fe6e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6e7400640 1 -- 172.21.15.40:0/1517174578 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8018020 con 0x7fe6f00b3fc0 2024-03-24T00:36:05.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6e7400640 1 -- 172.21.15.40:0/1517174578 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e8014480 con 0x7fe6f00b3fc0 2024-03-24T00:36:05.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6e7400640 1 -- 172.21.15.40:0/1517174578 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e8014870 con 0x7fe6f00b3fc0 2024-03-24T00:36:05.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6f6600640 1 -- 172.21.15.40:0/1517174578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f015def0 con 0x7fe6f00b3fc0 2024-03-24T00:36:05.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6f6600640 1 -- 172.21.15.40:0/1517174578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f01515b0 con 0x7fe6f00b3fc0 2024-03-24T00:36:05.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.930+0000 7fe6f6600640 1 -- 172.21.15.40:0/1517174578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f0068170 con 0x7fe6f00b3fc0 2024-03-24T00:36:05.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.934+0000 7fe6e7400640 1 -- 172.21.15.40:0/1517174578 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe6e801b020 con 0x7fe6f00b3fc0 2024-03-24T00:36:05.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.934+0000 7fe6e7400640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe6cc067660 0x7fe6cc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:05.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.934+0000 7fe6e7400640 1 -- 172.21.15.40:0/1517174578 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fe6e8086c10 con 0x7fe6f00b3fc0 2024-03-24T00:36:05.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.934+0000 7fe6ef400640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe6cc067660 0x7fe6cc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:05.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.934+0000 7fe6ef400640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe6cc067660 0x7fe6cc069b20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0000f70 tx=0x7fe6e0007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:05.938+0000 7fe6e7400640 1 -- 172.21.15.40:0/1517174578 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe6e8022070 con 0x7fe6f00b3fc0 2024-03-24T00:36:06.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-24T00:36:06.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.102+0000 7fe6f6600640 1 -- 172.21.15.40:0/1517174578 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe6f0071ba0 con 0x7fe6f00b3fc0 2024-03-24T00:36:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.750+0000 7fe6e7400640 1 -- 172.21.15.40:0/1517174578 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v81) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe6f015def0 con 0x7fe6f00b3fc0 2024-03-24T00:36:06.754 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-03-24T00:36:06.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.754+0000 7fe6f6600640 1 -- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe6cc067660 msgr2=0x7fe6cc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:06.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.754+0000 7fe6f6600640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe6cc067660 0x7fe6cc069b20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0000f70 tx=0x7fe6e0007df0 comp rx=0 tx=0).stop 2024-03-24T00:36:06.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.754+0000 7fe6f6600640 1 -- 172.21.15.40:0/1517174578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6f00b3fc0 msgr2=0x7fe6f014ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.754+0000 7fe6f6600640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6f00b3fc0 0x7fe6f014ffc0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009dd0 tx=0x7fe6e800c490 comp rx=0 tx=0).stop 2024-03-24T00:36:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.754+0000 7fe6f6600640 1 -- 172.21.15.40:0/1517174578 shutdown_connections 2024-03-24T00:36:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.754+0000 7fe6f6600640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe6cc067660 0x7fe6cc069b20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.754+0000 7fe6f6600640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe6f00b3fc0 0x7fe6f014ffc0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.754+0000 7fe6f6600640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe6f00aec70 0x7fe6f00af810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.754+0000 7fe6f6600640 1 --2- 172.21.15.40:0/1517174578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00ae6b0 0x7fe6f015d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.754+0000 7fe6f6600640 1 -- 172.21.15.40:0/1517174578 >> 172.21.15.40:0/1517174578 conn(0x7fe6f00b1660 msgr2=0x7fe6f00ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.754+0000 7fe6f6600640 1 -- 172.21.15.40:0/1517174578 shutdown_connections 2024-03-24T00:36:06.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.754+0000 7fe6f6600640 1 -- 172.21.15.40:0/1517174578 wait complete. 2024-03-24T00:36:06.797 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:36:06.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f4721000640 1 Processor -- start 2024-03-24T00:36:06.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f4721000640 1 -- start start 2024-03-24T00:36:06.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f4721000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c0b2940 0x7f471c0b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:06.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f4721000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f471c0b1b10 0x7f471c06d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f4721000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c0b2ea0 con 0x7f471c146e20 2024-03-24T00:36:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f4721000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471c1622f0 con 0x7f471c0b1b10 2024-03-24T00:36:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f4721000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c1625d0 con 0x7f471c0b2940 2024-03-24T00:36:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f471be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f471c0b1b10 0x7f471c06d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f471be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f471c0b1b10 0x7f471c06d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49796/0 (socket says 172.21.15.40:49796) 2024-03-24T00:36:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f471be00640 1 -- 172.21.15.40:0/1109629825 learned_addr learned my addr 172.21.15.40:0/1109629825 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:06.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f4712a00640 1 --2- 172.21.15.40:0/1109629825 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c0b2940 0x7f471c0b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:06.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f471be00640 1 -- 172.21.15.40:0/1109629825 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c0b2940 msgr2=0x7f471c0b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:06.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f471be00640 1 --2- 172.21.15.40:0/1109629825 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c0b2940 0x7f471c0b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:06.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f471be00640 1 -- 172.21.15.40:0/1109629825 >> v1:172.21.15.53:6789/0 conn(0x7f471c146e20 legacy=0x7f471c1618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:06.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.942+0000 7f471be00640 1 -- 172.21.15.40:0/1109629825 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c06cc40 con 0x7f471c0b1b10 2024-03-24T00:36:06.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f471be00640 1 --2- 172.21.15.40:0/1109629825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c0b1b10 0x7f471c06d440 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f4714009870 tx=0x7f471401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7dbbfed78bfc31e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:06.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f471aa00640 1 -- 172.21.15.40:0/1109629825 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47140b0070 con 0x7f471c0b1b10 2024-03-24T00:36:06.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f471aa00640 1 -- 172.21.15.40:0/1109629825 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47140ab600 con 0x7f471c0b1b10 2024-03-24T00:36:06.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f471aa00640 1 -- 172.21.15.40:0/1109629825 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47140ab9d0 con 0x7f471c0b1b10 2024-03-24T00:36:06.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f4721000640 1 -- 172.21.15.40:0/1109629825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c0b1b10 msgr2=0x7f471c06d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:06.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f4721000640 1 --2- 172.21.15.40:0/1109629825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c0b1b10 0x7f471c06d440 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f4714009870 tx=0x7f471401c500 comp rx=0 tx=0).stop 2024-03-24T00:36:06.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f4721000640 1 -- 172.21.15.40:0/1109629825 shutdown_connections 2024-03-24T00:36:06.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f4721000640 1 --2- 172.21.15.40:0/1109629825 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c0b2940 0x7f471c0b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:06.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f4721000640 1 --2- 172.21.15.40:0/1109629825 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c0b1b10 0x7f471c06d440 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:06.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f4721000640 1 -- 172.21.15.40:0/1109629825 >> 172.21.15.40:0/1109629825 conn(0x7f471c066e80 msgr2=0x7f471c067290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:06.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f4721000640 1 -- 172.21.15.40:0/1109629825 shutdown_connections 2024-03-24T00:36:06.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f4721000640 1 -- 172.21.15.40:0/1109629825 wait complete. 2024-03-24T00:36:06.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f4721000640 1 Processor -- start 2024-03-24T00:36:06.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.946+0000 7f4721000640 1 -- start start 2024-03-24T00:36:06.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4721000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c0b1b10 0x7f471c15d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:06.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4721000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c0b2940 0x7f471c0ade70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:06.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4721000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f471c146e20 0x7f471c0ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4721000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c147380 con 0x7f471c0b2940 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4721000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c147200 con 0x7f471c146e20 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4721000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c16ac40 con 0x7f471c0b1b10 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4712a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c0b2940 0x7f471c0ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f471b400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c0b1b10 0x7f471c15d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4712a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c0b2940 0x7f471c0ade70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49810/0 (socket says 172.21.15.40:49810) 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4712a00640 1 -- 172.21.15.40:0/3561208059 learned_addr learned my addr 172.21.15.40:0/3561208059 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f471be00640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f471c146e20 0x7f471c0ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4712a00640 1 -- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c0b1b10 msgr2=0x7f471c15d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4712a00640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c0b1b10 0x7f471c15d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4712a00640 1 -- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f471c146e20 msgr2=0x7f471c0ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4712a00640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f471c146e20 0x7f471c0ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4712a00640 1 -- 172.21.15.40:0/3561208059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c15f460 con 0x7f471c0b2940 2024-03-24T00:36:06.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4712a00640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c0b2940 0x7f471c0ade70 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f4704007e60 tx=0x7f470400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4713e00640 1 -- 172.21.15.40:0/3561208059 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4704014620 con 0x7f471c0b2940 2024-03-24T00:36:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4713e00640 1 -- 172.21.15.40:0/3561208059 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4704019070 con 0x7f471c0b2940 2024-03-24T00:36:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4713e00640 1 -- 172.21.15.40:0/3561208059 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4704014a20 con 0x7f471c0b2940 2024-03-24T00:36:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4721000640 1 -- 172.21.15.40:0/3561208059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c15f740 con 0x7f471c0b2940 2024-03-24T00:36:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4721000640 1 -- 172.21.15.40:0/3561208059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c15fd00 con 0x7f471c0b2940 2024-03-24T00:36:06.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.950+0000 7f4721000640 1 -- 172.21.15.40:0/3561208059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c04f410 con 0x7f471c0b2940 2024-03-24T00:36:06.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.954+0000 7f4713e00640 1 -- 172.21.15.40:0/3561208059 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4704014bc0 con 0x7f471c0b2940 2024-03-24T00:36:06.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.954+0000 7f4713e00640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46f80676e0 0x7f46f8069ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:06.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.954+0000 7f4713e00640 1 -- 172.21.15.40:0/3561208059 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f4704086e70 con 0x7f471c0b2940 2024-03-24T00:36:06.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.954+0000 7f471b400640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46f80676e0 0x7f46f8069ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:06.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.954+0000 7f471b400640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46f80676e0 0x7f46f8069ba0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f470c001770 tx=0x7f470c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:06.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:06.958+0000 7f4713e00640 1 -- 172.21.15.40:0/3561208059 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f47040574e0 con 0x7f471c0b2940 2024-03-24T00:36:07.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.158+0000 7f4721000640 1 -- 172.21.15.40:0/3561208059 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f471c1430f0 con 0x7f471c0b2940 2024-03-24T00:36:07.630 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:36:07.634 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:36:07.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.782+0000 7f4713e00640 1 -- 172.21.15.40:0/3561208059 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v78) v1 ==== 174+0+0 (secure 0 0 0) 0x7f470405b580 con 0x7f471c0b2940 2024-03-24T00:36:07.791 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 12 and data pool 13 2024-03-24T00:36:07.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.790+0000 7f4721000640 1 -- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46f80676e0 msgr2=0x7f46f8069ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:07.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.790+0000 7f4721000640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46f80676e0 0x7f46f8069ba0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f470c001770 tx=0x7f470c009040 comp rx=0 tx=0).stop 2024-03-24T00:36:07.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.790+0000 7f4721000640 1 -- 172.21.15.40:0/3561208059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c0b2940 msgr2=0x7f471c0ade70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:07.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.790+0000 7f4721000640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c0b2940 0x7f471c0ade70 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f4704007e60 tx=0x7f470400e4b0 comp rx=0 tx=0).stop 2024-03-24T00:36:07.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.794+0000 7f4721000640 1 -- 172.21.15.40:0/3561208059 shutdown_connections 2024-03-24T00:36:07.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.794+0000 7f4721000640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46f80676e0 0x7f46f8069ba0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:07.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.794+0000 7f4721000640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f471c146e20 0x7f471c0ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:07.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.794+0000 7f4721000640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f471c0b2940 0x7f471c0ade70 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:07.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.794+0000 7f4721000640 1 --2- 172.21.15.40:0/3561208059 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c0b1b10 0x7f471c15d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:07.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.794+0000 7f4721000640 1 -- 172.21.15.40:0/3561208059 >> 172.21.15.40:0/3561208059 conn(0x7f471c066e80 msgr2=0x7f471c0af7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:07.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.794+0000 7f4721000640 1 -- 172.21.15.40:0/3561208059 shutdown_connections 2024-03-24T00:36:07.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.794+0000 7f4721000640 1 -- 172.21.15.40:0/3561208059 wait complete. 2024-03-24T00:36:07.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:36:07.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.966+0000 7fd4fe800640 1 Processor -- start 2024-03-24T00:36:07.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.966+0000 7fd4fe800640 1 -- start start 2024-03-24T00:36:07.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4fe800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd4f813ec80 0x7fd4f81414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4fe800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4f81419f0 0x7fd4f80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4fe800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f8070e00 con 0x7fd4f80ce630 2024-03-24T00:36:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4fe800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f8141e50 con 0x7fd4f81419f0 2024-03-24T00:36:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4fe800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f80cee20 con 0x7fd4f813ec80 2024-03-24T00:36:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4efe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4f81419f0 0x7fd4f80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4efe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd4f81419f0 0x7fd4f80ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39684/0 (socket says 172.21.15.40:39684) 2024-03-24T00:36:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4efe00640 1 -- 172.21.15.40:0/1589920202 learned_addr learned my addr 172.21.15.40:0/1589920202 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4f7400640 1 --2- 172.21.15.40:0/1589920202 >> v2:172.21.15.53:3300/0 conn(0x7fd4f813ec80 0x7fd4f81414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4efe00640 1 -- 172.21.15.40:0/1589920202 >> v2:172.21.15.53:3300/0 conn(0x7fd4f813ec80 msgr2=0x7fd4f81414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4efe00640 1 --2- 172.21.15.40:0/1589920202 >> v2:172.21.15.53:3300/0 conn(0x7fd4f813ec80 0x7fd4f81414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4efe00640 1 -- 172.21.15.40:0/1589920202 >> v1:172.21.15.53:6789/0 conn(0x7fd4f80ce630 legacy=0x7fd4f815f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:07.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4efe00640 1 -- 172.21.15.40:0/1589920202 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f80cf100 con 0x7fd4f81419f0 2024-03-24T00:36:07.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.970+0000 7fd4efe00640 1 --2- 172.21.15.40:0/1589920202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f81419f0 0x7fd4f80ce0f0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fd4f806b170 tx=0x7fd4e001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b24e32d18ec0c47 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4f6a00640 1 -- 172.21.15.40:0/1589920202 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e00aa660 con 0x7fd4f81419f0 2024-03-24T00:36:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4f6a00640 1 -- 172.21.15.40:0/1589920202 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e00af070 con 0x7fd4f81419f0 2024-03-24T00:36:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4f6a00640 1 -- 172.21.15.40:0/1589920202 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e00aaa20 con 0x7fd4f81419f0 2024-03-24T00:36:07.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 -- 172.21.15.40:0/1589920202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f81419f0 msgr2=0x7fd4f80ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:07.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 --2- 172.21.15.40:0/1589920202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f81419f0 0x7fd4f80ce0f0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fd4f806b170 tx=0x7fd4e001c4e0 comp rx=0 tx=0).stop 2024-03-24T00:36:07.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 -- 172.21.15.40:0/1589920202 shutdown_connections 2024-03-24T00:36:07.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 --2- 172.21.15.40:0/1589920202 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f81419f0 0x7fd4f80ce0f0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:07.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 --2- 172.21.15.40:0/1589920202 >> v2:172.21.15.53:3300/0 conn(0x7fd4f813ec80 0x7fd4f81414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:07.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 -- 172.21.15.40:0/1589920202 >> 172.21.15.40:0/1589920202 conn(0x7fd4f80b17b0 msgr2=0x7fd4f80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:07.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 -- 172.21.15.40:0/1589920202 shutdown_connections 2024-03-24T00:36:07.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 -- 172.21.15.40:0/1589920202 wait complete. 2024-03-24T00:36:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 Processor -- start 2024-03-24T00:36:07.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 -- start start 2024-03-24T00:36:07.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f80ce630 0x7fd4f8139610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd4f813ec80 0x7fd4f8166470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f81419f0 0x7fd4f815b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f8168ec0 con 0x7fd4f80ce630 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f8168d40 con 0x7fd4f81419f0 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4fe800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f8169040 con 0x7fd4f813ec80 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4f7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f80ce630 0x7fd4f8139610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4f7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f80ce630 0x7fd4f8139610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39686/0 (socket says 172.21.15.40:39686) 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.974+0000 7fd4f7400640 1 -- 172.21.15.40:0/4119148096 learned_addr learned my addr 172.21.15.40:0/4119148096 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4efe00640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd4f813ec80 0x7fd4f8166470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4f7e00640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f81419f0 0x7fd4f815b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4f7400640 1 -- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd4f813ec80 msgr2=0x7fd4f8166470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4f7400640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd4f813ec80 0x7fd4f8166470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:07.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4f7400640 1 -- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f81419f0 msgr2=0x7fd4f815b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4f7400640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f81419f0 0x7fd4f815b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4f7400640 1 -- 172.21.15.40:0/4119148096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f8139bf0 con 0x7fd4f80ce630 2024-03-24T00:36:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4f7400640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f80ce630 0x7fd4f8139610 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd4e800d830 tx=0x7fd4e800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4ef400640 1 -- 172.21.15.40:0/4119148096 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e800b6c0 con 0x7fd4f80ce630 2024-03-24T00:36:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4ef400640 1 -- 172.21.15.40:0/4119148096 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e8019070 con 0x7fd4f80ce630 2024-03-24T00:36:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4fe800640 1 -- 172.21.15.40:0/4119148096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f8139ed0 con 0x7fd4f80ce630 2024-03-24T00:36:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4ef400640 1 -- 172.21.15.40:0/4119148096 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e800ba60 con 0x7fd4f80ce630 2024-03-24T00:36:07.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4fe800640 1 -- 172.21.15.40:0/4119148096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f81f5150 con 0x7fd4f80ce630 2024-03-24T00:36:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4fe800640 1 -- 172.21.15.40:0/4119148096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f8068170 con 0x7fd4f80ce630 2024-03-24T00:36:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4ef400640 1 -- 172.21.15.40:0/4119148096 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd4e801e460 con 0x7fd4f80ce630 2024-03-24T00:36:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4ef400640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd4d40676b0 0x7fd4d4069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.978+0000 7fd4ef400640 1 -- 172.21.15.40:0/4119148096 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4e8085f90 con 0x7fd4f80ce630 2024-03-24T00:36:07.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.982+0000 7fd4efe00640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd4d40676b0 0x7fd4d4069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:07.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.982+0000 7fd4efe00640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd4d40676b0 0x7fd4d4069b70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00ae000 tx=0x7fd4e00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:07.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:07.982+0000 7fd4ef400640 1 -- 172.21.15.40:0/4119148096 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd4e8056bf0 con 0x7fd4f80ce630 2024-03-24T00:36:08.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.146+0000 7fd4fe800640 1 -- 172.21.15.40:0/4119148096 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4f8072b90 con 0x7fd4f80ce630 2024-03-24T00:36:08.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4ef400640 1 -- 172.21.15.40:0/4119148096 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16613 (secure 0 0 0) 0x7fd4e80564f0 con 0x7fd4f80ce630 2024-03-24T00:36:08.153 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:08.153 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:36:07.163559+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:05.573740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-24T00:36:06.150226+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:36:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4fe800640 1 -- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd4d40676b0 msgr2=0x7fd4d4069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4fe800640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd4d40676b0 0x7fd4d4069b70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00ae000 tx=0x7fd4e00a7000 comp rx=0 tx=0).stop 2024-03-24T00:36:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4fe800640 1 -- 172.21.15.40:0/4119148096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f80ce630 msgr2=0x7fd4f8139610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4fe800640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f80ce630 0x7fd4f8139610 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fd4e800d830 tx=0x7fd4e800dcf0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4fe800640 1 -- 172.21.15.40:0/4119148096 shutdown_connections 2024-03-24T00:36:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4fe800640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd4d40676b0 0x7fd4d4069b70 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4fe800640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4f81419f0 0x7fd4f815b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4fe800640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd4f813ec80 0x7fd4f8166470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4fe800640 1 --2- 172.21.15.40:0/4119148096 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd4f80ce630 0x7fd4f8139610 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4fe800640 1 -- 172.21.15.40:0/4119148096 >> 172.21.15.40:0/4119148096 conn(0x7fd4f80b17b0 msgr2=0x7fd4f813fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4fe800640 1 -- 172.21.15.40:0/4119148096 shutdown_connections 2024-03-24T00:36:08.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.150+0000 7fd4fe800640 1 -- 172.21.15.40:0/4119148096 wait complete. 2024-03-24T00:36:08.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:36:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f5074e00640 1 Processor -- start 2024-03-24T00:36:08.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f5074e00640 1 -- start start 2024-03-24T00:36:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f5074e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f507013ec80 0x7f50701414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f5074e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5070070e00 con 0x7f50701419f0 2024-03-24T00:36:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f5074e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50700cee20 con 0x7f50700ce800 2024-03-24T00:36:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f5074e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5070141e50 con 0x7f507013ec80 2024-03-24T00:36:08.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f506d400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f50701419f0 0x7f50700ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42936/0 (socket says 172.21.15.40:42936) 2024-03-24T00:36:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f506d400640 1 -- 172.21.15.40:0/2452968103 learned_addr learned my addr 172.21.15.40:0/2452968103 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f506de00640 1 --2- 172.21.15.40:0/2452968103 >> v2:172.21.15.40:3300/0 conn(0x7f507013ec80 0x7f50701414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f506de00640 1 -- 172.21.15.40:0/2452968103 >> v1:172.21.15.53:6789/0 conn(0x7f50700ce800 legacy=0x7f5070071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f506de00640 1 -- 172.21.15.40:0/2452968103 >> v1:172.21.15.40:6789/0 conn(0x7f50701419f0 legacy=0x7f50700ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f506de00640 1 -- 172.21.15.40:0/2452968103 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50700cf100 con 0x7f507013ec80 2024-03-24T00:36:08.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.314+0000 7f506ca00640 1 -- 172.21.15.40:0/2452968103 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857376270 0 0) 0x7f5070070e00 con 0x7f50701419f0 2024-03-24T00:36:08.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f506de00640 1 --2- 172.21.15.40:0/2452968103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f507013ec80 0x7f50701414b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f50600097c0 tx=0x7f506001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6545ab47eda753 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:08.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f506ca00640 1 -- 172.21.15.40:0/2452968103 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50600aa620 con 0x7f507013ec80 2024-03-24T00:36:08.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f506ca00640 1 -- 172.21.15.40:0/2452968103 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50600af070 con 0x7f507013ec80 2024-03-24T00:36:08.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f506ca00640 1 -- 172.21.15.40:0/2452968103 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50600aa9e0 con 0x7f507013ec80 2024-03-24T00:36:08.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f5074e00640 1 -- 172.21.15.40:0/2452968103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f507013ec80 msgr2=0x7f50701414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f5074e00640 1 --2- 172.21.15.40:0/2452968103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f507013ec80 0x7f50701414b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f50600097c0 tx=0x7f506001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f5074e00640 1 -- 172.21.15.40:0/2452968103 shutdown_connections 2024-03-24T00:36:08.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f5074e00640 1 --2- 172.21.15.40:0/2452968103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f507013ec80 0x7f50701414b0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f5074e00640 1 -- 172.21.15.40:0/2452968103 >> 172.21.15.40:0/2452968103 conn(0x7f50700b17b0 msgr2=0x7f50700b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:08.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f5074e00640 1 -- 172.21.15.40:0/2452968103 shutdown_connections 2024-03-24T00:36:08.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f5074e00640 1 -- 172.21.15.40:0/2452968103 wait complete. 2024-03-24T00:36:08.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f5074e00640 1 Processor -- start 2024-03-24T00:36:08.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.318+0000 7f5074e00640 1 -- start start 2024-03-24T00:36:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f5074e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50700ce800 0x7f5070152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:08.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f5074e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f507013ec80 0x7f507013d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f5074e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50701419f0 0x7f5070132d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f5074e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50700725b0 con 0x7f507013ec80 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f5074e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5070072430 con 0x7f50700ce800 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f5074e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5070169d50 con 0x7f50701419f0 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f506d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f507013ec80 0x7f507013d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f506d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f507013ec80 0x7f507013d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39708/0 (socket says 172.21.15.40:39708) 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f506d400640 1 -- 172.21.15.40:0/3199312358 learned_addr learned my addr 172.21.15.40:0/3199312358 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f506de00640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50700ce800 0x7f5070152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f506e800640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50701419f0 0x7f5070132d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f506d400640 1 -- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50701419f0 msgr2=0x7f5070132d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f506d400640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50701419f0 0x7f5070132d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f506d400640 1 -- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50700ce800 msgr2=0x7f5070152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f506d400640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50700ce800 0x7f5070152470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f506d400640 1 -- 172.21.15.40:0/3199312358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50701529b0 con 0x7f507013ec80 2024-03-24T00:36:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f506de00640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50700ce800 0x7f5070152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f506d400640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f507013ec80 0x7f507013d480 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f505c00c4d0 tx=0x7f505c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f5066000640 1 -- 172.21.15.40:0/3199312358 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c018070 con 0x7f507013ec80 2024-03-24T00:36:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f5066000640 1 -- 172.21.15.40:0/3199312358 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505c0074e0 con 0x7f507013ec80 2024-03-24T00:36:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f5066000640 1 -- 172.21.15.40:0/3199312358 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505c004a30 con 0x7f507013ec80 2024-03-24T00:36:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f5074e00640 1 -- 172.21.15.40:0/3199312358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5070159920 con 0x7f507013ec80 2024-03-24T00:36:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f5074e00640 1 -- 172.21.15.40:0/3199312358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5070138b10 con 0x7f507013ec80 2024-03-24T00:36:08.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.322+0000 7f5074e00640 1 -- 172.21.15.40:0/3199312358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5070068170 con 0x7f507013ec80 2024-03-24T00:36:08.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.326+0000 7f5066000640 1 -- 172.21.15.40:0/3199312358 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f505c004bd0 con 0x7f507013ec80 2024-03-24T00:36:08.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.326+0000 7f5066000640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5050067690 0x7f5050069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:08.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.326+0000 7f5066000640 1 -- 172.21.15.40:0/3199312358 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f505c084b80 con 0x7f507013ec80 2024-03-24T00:36:08.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.326+0000 7f506de00640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5050067690 0x7f5050069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:08.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.326+0000 7f506de00640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5050067690 0x7f5050069b50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f50600ae000 tx=0x7f50600a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:08.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.330+0000 7f5066000640 1 -- 172.21.15.40:0/3199312358 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f505c055760 con 0x7f507013ec80 2024-03-24T00:36:08.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.498+0000 7f5074e00640 1 -- 172.21.15.40:0/3199312358 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5070072b90 con 0x7f507013ec80 2024-03-24T00:36:08.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.498+0000 7f5066000640 1 -- 172.21.15.40:0/3199312358 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16613 (secure 0 0 0) 0x7f505c055060 con 0x7f507013ec80 2024-03-24T00:36:08.503 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:08.503 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:36:07.163559+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:05.573740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-24T00:36:06.150226+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:36:08.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.502+0000 7f5074e00640 1 -- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5050067690 msgr2=0x7f5050069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.502+0000 7f5074e00640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5050067690 0x7f5050069b50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f50600ae000 tx=0x7f50600a7000 comp rx=0 tx=0).stop 2024-03-24T00:36:08.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.502+0000 7f5074e00640 1 -- 172.21.15.40:0/3199312358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f507013ec80 msgr2=0x7f507013d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.502+0000 7f5074e00640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f507013ec80 0x7f507013d480 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f505c00c4d0 tx=0x7f505c00c990 comp rx=0 tx=0).stop 2024-03-24T00:36:08.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.502+0000 7f5074e00640 1 -- 172.21.15.40:0/3199312358 shutdown_connections 2024-03-24T00:36:08.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.502+0000 7f5074e00640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5050067690 0x7f5050069b50 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.502+0000 7f5074e00640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50701419f0 0x7f5070132d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.502+0000 7f5074e00640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f507013ec80 0x7f507013d480 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.502+0000 7f5074e00640 1 --2- 172.21.15.40:0/3199312358 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50700ce800 0x7f5070152470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.502+0000 7f5074e00640 1 -- 172.21.15.40:0/3199312358 >> 172.21.15.40:0/3199312358 conn(0x7f50700b17b0 msgr2=0x7f5070071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:08.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.502+0000 7f5074e00640 1 -- 172.21.15.40:0/3199312358 shutdown_connections 2024-03-24T00:36:08.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.502+0000 7f5074e00640 1 -- 172.21.15.40:0/3199312358 wait complete. 2024-03-24T00:36:08.524 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:36:08.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d3e00640 1 Processor -- start 2024-03-24T00:36:08.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d3e00640 1 -- start start 2024-03-24T00:36:08.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f86cc15ac60 0x7f86cc0b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:08.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86cc071b00 0x7f86cc071f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:08.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d3e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86cc0b3260 con 0x7f86cc06cb80 2024-03-24T00:36:08.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d3e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86cc072490 con 0x7f86cc15ac60 2024-03-24T00:36:08.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d3e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc0b3e70 con 0x7f86cc071b00 2024-03-24T00:36:08.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d0a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f86cc06cb80 0x7f86cc0b4300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42938/0 (socket says 172.21.15.40:42938) 2024-03-24T00:36:08.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d0a00640 1 -- 172.21.15.40:0/1049232260 learned_addr learned my addr 172.21.15.40:0/1049232260 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:08.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d1e00640 1 --2- 172.21.15.40:0/1049232260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86cc071b00 0x7f86cc071f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:08.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d1400640 1 --2- 172.21.15.40:0/1049232260 >> v2:172.21.15.53:3300/0 conn(0x7f86cc15ac60 0x7f86cc0b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:08.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86cbe00640 1 -- 172.21.15.40:0/1049232260 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060079455 0 0) 0x7f86cc0b3260 con 0x7f86cc06cb80 2024-03-24T00:36:08.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86cbe00640 1 -- 172.21.15.40:0/1049232260 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b4003530 con 0x7f86cc06cb80 2024-03-24T00:36:08.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d1e00640 1 -- 172.21.15.40:0/1049232260 >> v2:172.21.15.53:3300/0 conn(0x7f86cc15ac60 msgr2=0x7f86cc0b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:08.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d1e00640 1 --2- 172.21.15.40:0/1049232260 >> v2:172.21.15.53:3300/0 conn(0x7f86cc15ac60 0x7f86cc0b2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d1e00640 1 -- 172.21.15.40:0/1049232260 >> v1:172.21.15.40:6789/0 conn(0x7f86cc06cb80 legacy=0x7f86cc0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86cbe00640 1 -- 172.21.15.40:0/1049232260 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257170358 0 0) 0x7f86b4003530 con 0x7f86cc06cb80 2024-03-24T00:36:08.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.666+0000 7f86d1e00640 1 -- 172.21.15.40:0/1049232260 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86cc0b3540 con 0x7f86cc071b00 2024-03-24T00:36:08.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.670+0000 7f86d1e00640 1 --2- 172.21.15.40:0/1049232260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86cc071b00 0x7f86cc071f50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009d90 tx=0x7f86c401c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d90cb7203f660006 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:08.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.670+0000 7f86cbe00640 1 -- 172.21.15.40:0/1049232260 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c40b2070 con 0x7f86cc071b00 2024-03-24T00:36:08.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.670+0000 7f86cbe00640 1 -- 172.21.15.40:0/1049232260 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c40a55d0 con 0x7f86cc071b00 2024-03-24T00:36:08.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.670+0000 7f86cbe00640 1 -- 172.21.15.40:0/1049232260 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8001680 con 0x7f86cc071b00 2024-03-24T00:36:08.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.670+0000 7f86d3e00640 1 -- 172.21.15.40:0/1049232260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86cc071b00 msgr2=0x7f86cc071f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.670+0000 7f86d3e00640 1 --2- 172.21.15.40:0/1049232260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86cc071b00 0x7f86cc071f50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009d90 tx=0x7f86c401c7d0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.670+0000 7f86d3e00640 1 -- 172.21.15.40:0/1049232260 shutdown_connections 2024-03-24T00:36:08.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.670+0000 7f86d3e00640 1 --2- 172.21.15.40:0/1049232260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86cc071b00 0x7f86cc071f50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.670+0000 7f86d3e00640 1 --2- 172.21.15.40:0/1049232260 >> v2:172.21.15.53:3300/0 conn(0x7f86cc15ac60 0x7f86cc0b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.670+0000 7f86d3e00640 1 -- 172.21.15.40:0/1049232260 >> 172.21.15.40:0/1049232260 conn(0x7f86cc0b1660 msgr2=0x7f86cc0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:08.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.670+0000 7f86d3e00640 1 -- 172.21.15.40:0/1049232260 shutdown_connections 2024-03-24T00:36:08.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.670+0000 7f86d3e00640 1 -- 172.21.15.40:0/1049232260 wait complete. 2024-03-24T00:36:08.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d3e00640 1 Processor -- start 2024-03-24T00:36:08.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d3e00640 1 -- start start 2024-03-24T00:36:08.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86cc06cb80 0x7f86cc15dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:08.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86cc071b00 0x7f86cc14eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86cc15ac60 0x7f86cc0cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc0b3020 con 0x7f86cc06cb80 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc0b2ea0 con 0x7f86cc15ac60 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d3e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86cc15b040 con 0x7f86cc071b00 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86cc06cb80 0x7f86cc15dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86cc06cb80 0x7f86cc15dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39714/0 (socket says 172.21.15.40:39714) 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d1400640 1 -- 172.21.15.40:0/684337873 learned_addr learned my addr 172.21.15.40:0/684337873 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d1e00640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86cc15ac60 0x7f86cc0cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d0a00640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86cc071b00 0x7f86cc14eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d1400640 1 -- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86cc071b00 msgr2=0x7f86cc14eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d1400640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86cc071b00 0x7f86cc14eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d1400640 1 -- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86cc15ac60 msgr2=0x7f86cc0cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d1400640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86cc15ac60 0x7f86cc0cfb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d1400640 1 -- 172.21.15.40:0/684337873 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86cc15e250 con 0x7f86cc06cb80 2024-03-24T00:36:08.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d1400640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86cc06cb80 0x7f86cc15dd10 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f86c000e930 tx=0x7f86c000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:08.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86c9600640 1 -- 172.21.15.40:0/684337873 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c0017070 con 0x7f86cc06cb80 2024-03-24T00:36:08.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86c9600640 1 -- 172.21.15.40:0/684337873 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c000ca00 con 0x7f86cc06cb80 2024-03-24T00:36:08.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86c9600640 1 -- 172.21.15.40:0/684337873 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c000cdb0 con 0x7f86cc06cb80 2024-03-24T00:36:08.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d3e00640 1 -- 172.21.15.40:0/684337873 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86cc06dc00 con 0x7f86cc06cb80 2024-03-24T00:36:08.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d3e00640 1 -- 172.21.15.40:0/684337873 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86cc06e1c0 con 0x7f86cc06cb80 2024-03-24T00:36:08.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.674+0000 7f86d3e00640 1 -- 172.21.15.40:0/684337873 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86cc068170 con 0x7f86cc06cb80 2024-03-24T00:36:08.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.678+0000 7f86c9600640 1 -- 172.21.15.40:0/684337873 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f86c001a070 con 0x7f86cc06cb80 2024-03-24T00:36:08.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.678+0000 7f86c9600640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f86ac067660 0x7f86ac069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:08.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.678+0000 7f86c9600640 1 -- 172.21.15.40:0/684337873 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86c0085c90 con 0x7f86cc06cb80 2024-03-24T00:36:08.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.678+0000 7f86d0a00640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f86ac067660 0x7f86ac069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:08.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.678+0000 7f86d0a00640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f86ac067660 0x7f86ac069b20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f86b80011f0 tx=0x7f86b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:08.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.682+0000 7f86c9600640 1 -- 172.21.15.40:0/684337873 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f86cc15e250 con 0x7f86cc06cb80 2024-03-24T00:36:08.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:08.886+0000 7f86d3e00640 1 -- 172.21.15.40:0/684337873 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f86cc153190 con 0x7f86cc06cb80 2024-03-24T00:36:09.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.794+0000 7f86c9600640 1 -- 172.21.15.40:0/684337873 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v81) v1 ==== 118+0+0 (secure 0 0 0) 0x7f86c00562e0 con 0x7f86cc06cb80 2024-03-24T00:36:09.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.798+0000 7f86d3e00640 1 -- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f86ac067660 msgr2=0x7f86ac069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:09.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.798+0000 7f86d3e00640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f86ac067660 0x7f86ac069b20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f86b80011f0 tx=0x7f86b8009040 comp rx=0 tx=0).stop 2024-03-24T00:36:09.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.798+0000 7f86d3e00640 1 -- 172.21.15.40:0/684337873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86cc06cb80 msgr2=0x7f86cc15dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:09.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.798+0000 7f86d3e00640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86cc06cb80 0x7f86cc15dd10 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f86c000e930 tx=0x7f86c000edf0 comp rx=0 tx=0).stop 2024-03-24T00:36:09.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.798+0000 7f86d3e00640 1 -- 172.21.15.40:0/684337873 shutdown_connections 2024-03-24T00:36:09.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.798+0000 7f86d3e00640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f86ac067660 0x7f86ac069b20 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:09.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.798+0000 7f86d3e00640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86cc15ac60 0x7f86cc0cfb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:09.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.798+0000 7f86d3e00640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86cc071b00 0x7f86cc14eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:09.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.798+0000 7f86d3e00640 1 --2- 172.21.15.40:0/684337873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f86cc06cb80 0x7f86cc15dd10 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:09.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.798+0000 7f86d3e00640 1 -- 172.21.15.40:0/684337873 >> 172.21.15.40:0/684337873 conn(0x7f86cc0b1660 msgr2=0x7f86cc0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:09.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.798+0000 7f86d3e00640 1 -- 172.21.15.40:0/684337873 shutdown_connections 2024-03-24T00:36:09.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.798+0000 7f86d3e00640 1 -- 172.21.15.40:0/684337873 wait complete. 2024-03-24T00:36:09.823 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:36:09.823 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:36:09.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f9431e00640 1 Processor -- start 2024-03-24T00:36:09.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f9431e00640 1 -- start start 2024-03-24T00:36:09.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f9431e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f942c15bd00 0x7f942c0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:09.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f9431e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942c072d60 con 0x7f942c15ffd0 2024-03-24T00:36:09.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f9431e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f942c071de0 con 0x7f942c0728d0 2024-03-24T00:36:09.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f9431e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f942c0cea50 con 0x7f942c15bd00 2024-03-24T00:36:09.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f942b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f942c15ffd0 0x7f942c06cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42942/0 (socket says 172.21.15.40:42942) 2024-03-24T00:36:09.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f942b400640 1 -- 172.21.15.40:0/778230522 learned_addr learned my addr 172.21.15.40:0/778230522 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:09.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f942aa00640 1 --2- 172.21.15.40:0/778230522 >> v2:172.21.15.53:3300/0 conn(0x7f942c15bd00 0x7f942c0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:09.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f9429600640 1 -- 172.21.15.40:0/778230522 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110149367 0 0) 0x7f942c072d60 con 0x7f942c15ffd0 2024-03-24T00:36:09.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f9429600640 1 -- 172.21.15.40:0/778230522 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9410003530 con 0x7f942c15ffd0 2024-03-24T00:36:09.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f9429600640 1 -- 172.21.15.40:0/778230522 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414890408 0 0) 0x7f942c071de0 con 0x7f942c0728d0 2024-03-24T00:36:09.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f9429600640 1 -- 172.21.15.40:0/778230522 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f942c072d60 con 0x7f942c0728d0 2024-03-24T00:36:09.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f942aa00640 1 -- 172.21.15.40:0/778230522 >> v1:172.21.15.53:6789/0 conn(0x7f942c0728d0 legacy=0x7f942c15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:09.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f942aa00640 1 -- 172.21.15.40:0/778230522 >> v1:172.21.15.40:6789/0 conn(0x7f942c15ffd0 legacy=0x7f942c06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:09.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.970+0000 7f942aa00640 1 -- 172.21.15.40:0/778230522 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942c0ced30 con 0x7f942c15bd00 2024-03-24T00:36:09.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.974+0000 7f942aa00640 1 --2- 172.21.15.40:0/778230522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f942c15bd00 0x7f942c0ae8e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f941c0097c0 tx=0x7f941c01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35c9ea47df414b49 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:09.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.974+0000 7f9429600640 1 -- 172.21.15.40:0/778230522 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c0ab650 con 0x7f942c15bd00 2024-03-24T00:36:09.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.974+0000 7f9429600640 1 -- 172.21.15.40:0/778230522 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941c0b0070 con 0x7f942c15bd00 2024-03-24T00:36:09.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.974+0000 7f9429600640 1 -- 172.21.15.40:0/778230522 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c0aba10 con 0x7f942c15bd00 2024-03-24T00:36:09.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.974+0000 7f9431e00640 1 -- 172.21.15.40:0/778230522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f942c15bd00 msgr2=0x7f942c0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:09.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.974+0000 7f9431e00640 1 --2- 172.21.15.40:0/778230522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f942c15bd00 0x7f942c0ae8e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f941c0097c0 tx=0x7f941c01c470 comp rx=0 tx=0).stop 2024-03-24T00:36:09.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.974+0000 7f9431e00640 1 -- 172.21.15.40:0/778230522 shutdown_connections 2024-03-24T00:36:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.974+0000 7f9431e00640 1 --2- 172.21.15.40:0/778230522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f942c15bd00 0x7f942c0ae8e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.974+0000 7f9431e00640 1 -- 172.21.15.40:0/778230522 >> 172.21.15.40:0/778230522 conn(0x7f942c0b1660 msgr2=0x7f942c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.974+0000 7f9431e00640 1 -- 172.21.15.40:0/778230522 shutdown_connections 2024-03-24T00:36:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.974+0000 7f9431e00640 1 -- 172.21.15.40:0/778230522 wait complete. 2024-03-24T00:36:09.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9431e00640 1 Processor -- start 2024-03-24T00:36:09.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9431e00640 1 -- start start 2024-03-24T00:36:09.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9431e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f942c0728d0 0x7f942c14eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:09.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9431e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f942c15bd00 0x7f942c1502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:09.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9431e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f942c15ffd0 0x7f942c13ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:09.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9431e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942c15c0e0 con 0x7f942c0728d0 2024-03-24T00:36:09.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9431e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942c0cf660 con 0x7f942c15ffd0 2024-03-24T00:36:09.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9431e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942c16a420 con 0x7f942c15bd00 2024-03-24T00:36:09.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f942aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f942c0728d0 0x7f942c14eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f942aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f942c0728d0 0x7f942c14eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39716/0 (socket says 172.21.15.40:39716) 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f942aa00640 1 -- 172.21.15.40:0/109527955 learned_addr learned my addr 172.21.15.40:0/109527955 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f942a000640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f942c15bd00 0x7f942c1502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f942b400640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f942c15ffd0 0x7f942c13ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f942aa00640 1 -- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f942c15bd00 msgr2=0x7f942c1502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f942aa00640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f942c15bd00 0x7f942c1502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f942aa00640 1 -- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f942c15ffd0 msgr2=0x7f942c13ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f942aa00640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f942c15ffd0 0x7f942c13ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f942aa00640 1 -- 172.21.15.40:0/109527955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942c13b2c0 con 0x7f942c0728d0 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f942b400640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f942c15ffd0 0x7f942c13ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f942aa00640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f942c0728d0 0x7f942c14eb60 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f941c0ad040 tx=0x7f941c01acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9423400640 1 -- 172.21.15.40:0/109527955 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c0bf610 con 0x7f942c0728d0 2024-03-24T00:36:09.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9423400640 1 -- 172.21.15.40:0/109527955 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941c0bf7d0 con 0x7f942c0728d0 2024-03-24T00:36:09.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9423400640 1 -- 172.21.15.40:0/109527955 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f941c0bfbc0 con 0x7f942c0728d0 2024-03-24T00:36:09.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9431e00640 1 -- 172.21.15.40:0/109527955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942c14f200 con 0x7f942c0728d0 2024-03-24T00:36:09.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9431e00640 1 -- 172.21.15.40:0/109527955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942c14f7c0 con 0x7f942c0728d0 2024-03-24T00:36:09.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.978+0000 7f9431e00640 1 -- 172.21.15.40:0/109527955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f942c068170 con 0x7f942c0728d0 2024-03-24T00:36:09.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.982+0000 7f9423400640 1 -- 172.21.15.40:0/109527955 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f941c0bd020 con 0x7f942c0728d0 2024-03-24T00:36:09.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.982+0000 7f9423400640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f940c067660 0x7f940c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:09.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.982+0000 7f9423400640 1 -- 172.21.15.40:0/109527955 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f941c11fc60 con 0x7f942c0728d0 2024-03-24T00:36:09.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.982+0000 7f942a000640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f940c067660 0x7f940c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:09.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.982+0000 7f942a000640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f940c067660 0x7f940c069b20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f942c0b1370 tx=0x7f9418009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:09.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:09.986+0000 7f9423400640 1 -- 172.21.15.40:0/109527955 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f941c0c6030 con 0x7f942c0728d0 2024-03-24T00:36:10.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.186+0000 7f9431e00640 1 -- 172.21.15.40:0/109527955 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f942c06e870 con 0x7f942c0728d0 2024-03-24T00:36:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.794+0000 7f9423400640 1 -- 172.21.15.40:0/109527955 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7f941c016090 con 0x7f942c0728d0 2024-03-24T00:36:10.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.798+0000 7f9431e00640 1 -- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f940c067660 msgr2=0x7f940c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:10.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.798+0000 7f9431e00640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f940c067660 0x7f940c069b20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f942c0b1370 tx=0x7f9418009040 comp rx=0 tx=0).stop 2024-03-24T00:36:10.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.798+0000 7f9431e00640 1 -- 172.21.15.40:0/109527955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f942c0728d0 msgr2=0x7f942c14eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:10.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.798+0000 7f9431e00640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f942c0728d0 0x7f942c14eb60 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f941c0ad040 tx=0x7f941c01acb0 comp rx=0 tx=0).stop 2024-03-24T00:36:10.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.798+0000 7f9431e00640 1 -- 172.21.15.40:0/109527955 shutdown_connections 2024-03-24T00:36:10.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.798+0000 7f9431e00640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f940c067660 0x7f940c069b20 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:10.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.798+0000 7f9431e00640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f942c15ffd0 0x7f942c13ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:10.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.798+0000 7f9431e00640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f942c15bd00 0x7f942c1502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:10.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.798+0000 7f9431e00640 1 --2- 172.21.15.40:0/109527955 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f942c0728d0 0x7f942c14eb60 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:10.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.798+0000 7f9431e00640 1 -- 172.21.15.40:0/109527955 >> 172.21.15.40:0/109527955 conn(0x7f942c0b1660 msgr2=0x7f942c0b32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:10.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.798+0000 7f9431e00640 1 -- 172.21.15.40:0/109527955 shutdown_connections 2024-03-24T00:36:10.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.798+0000 7f9431e00640 1 -- 172.21.15.40:0/109527955 wait complete. 2024-03-24T00:36:10.825 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:36:10.825 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:36:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f373e200640 1 Processor -- start 2024-03-24T00:36:10.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f373e200640 1 -- start start 2024-03-24T00:36:10.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f373e200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f37380b46c0 0x7f37380b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:10.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f373e200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f37380b2ff0 0x7f37380ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:10.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f373e200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3738070f60 con 0x7f373806e620 2024-03-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f373e200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37380b3400 con 0x7f37380b2ff0 2024-03-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f373e200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37380b3620 con 0x7f37380b46c0 2024-03-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f3737e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f37380b2ff0 0x7f37380ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f3737400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f373806e620 0x7f373806ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42956/0 (socket says 172.21.15.40:42956) 2024-03-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f3737400640 1 -- 172.21.15.40:0/2772359436 learned_addr learned my addr 172.21.15.40:0/2772359436 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f372fe00640 1 --2- 172.21.15.40:0/2772359436 >> v2:172.21.15.53:3300/0 conn(0x7f37380b46c0 0x7f37380b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f3737e00640 1 -- 172.21.15.40:0/2772359436 >> v2:172.21.15.53:3300/0 conn(0x7f37380b46c0 msgr2=0x7f37380b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f3737e00640 1 --2- 172.21.15.40:0/2772359436 >> v2:172.21.15.53:3300/0 conn(0x7f37380b46c0 0x7f37380b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:10.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f3737e00640 1 -- 172.21.15.40:0/2772359436 >> v1:172.21.15.40:6789/0 conn(0x7f373806e620 legacy=0x7f373806ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:10.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f3737e00640 1 -- 172.21.15.40:0/2772359436 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37380b3900 con 0x7f37380b2ff0 2024-03-24T00:36:10.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.966+0000 7f3736a00640 1 -- 172.21.15.40:0/2772359436 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069735885 0 0) 0x7f3738070f60 con 0x7f373806e620 2024-03-24T00:36:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.970+0000 7f3737e00640 1 --2- 172.21.15.40:0/2772359436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37380b2ff0 0x7f37380ccfa0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f373806cea0 tx=0x7f373001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd62cf14ce9fb36 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.970+0000 7f3736a00640 1 -- 172.21.15.40:0/2772359436 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37300b2070 con 0x7f37380b2ff0 2024-03-24T00:36:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.970+0000 7f3736a00640 1 -- 172.21.15.40:0/2772359436 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37300a5930 con 0x7f37380b2ff0 2024-03-24T00:36:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.970+0000 7f3736a00640 1 -- 172.21.15.40:0/2772359436 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37380cdcf0 con 0x7f37380b2ff0 2024-03-24T00:36:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.970+0000 7f373e200640 1 -- 172.21.15.40:0/2772359436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37380b2ff0 msgr2=0x7f37380ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.970+0000 7f373e200640 1 --2- 172.21.15.40:0/2772359436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37380b2ff0 0x7f37380ccfa0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f373806cea0 tx=0x7f373001c870 comp rx=0 tx=0).stop 2024-03-24T00:36:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.970+0000 7f373e200640 1 -- 172.21.15.40:0/2772359436 shutdown_connections 2024-03-24T00:36:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.970+0000 7f373e200640 1 --2- 172.21.15.40:0/2772359436 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37380b2ff0 0x7f37380ccfa0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.970+0000 7f373e200640 1 --2- 172.21.15.40:0/2772359436 >> v2:172.21.15.53:3300/0 conn(0x7f37380b46c0 0x7f37380b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.970+0000 7f373e200640 1 -- 172.21.15.40:0/2772359436 >> 172.21.15.40:0/2772359436 conn(0x7f37380b17b0 msgr2=0x7f37380b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:10.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.970+0000 7f373e200640 1 -- 172.21.15.40:0/2772359436 shutdown_connections 2024-03-24T00:36:10.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.970+0000 7f373e200640 1 -- 172.21.15.40:0/2772359436 wait complete. 2024-03-24T00:36:10.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f373e200640 1 Processor -- start 2024-03-24T00:36:10.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f373e200640 1 -- start start 2024-03-24T00:36:10.978 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f373e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f373806e620 0x7f373813caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f373e200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37380b2ff0 0x7f373813dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f373e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37380b46c0 0x7f37381f7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f373e200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3738144920 con 0x7f37380b46c0 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f373e200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37380b4ad0 con 0x7f373806e620 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f373e200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373815dbb0 con 0x7f37380b2ff0 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f3737e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37380b46c0 0x7f37381f7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f3737400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f373806e620 0x7f373813caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f3737e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37380b46c0 0x7f37381f7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39740/0 (socket says 172.21.15.40:39740) 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f3737e00640 1 -- 172.21.15.40:0/2366331345 learned_addr learned my addr 172.21.15.40:0/2366331345 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f372fe00640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37380b2ff0 0x7f373813dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f3737e00640 1 -- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37380b2ff0 msgr2=0x7f373813dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f3737e00640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37380b2ff0 0x7f373813dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:10.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f3737e00640 1 -- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f373806e620 msgr2=0x7f373813caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f3737e00640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f373806e620 0x7f373813caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f3737e00640 1 -- 172.21.15.40:0/2366331345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373813e110 con 0x7f37380b46c0 2024-03-24T00:36:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f3737e00640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37380b46c0 0x7f37381f7a90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f3730098450 tx=0x7f3730019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f372f400640 1 -- 172.21.15.40:0/2366331345 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37300bd030 con 0x7f37380b46c0 2024-03-24T00:36:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f372f400640 1 -- 172.21.15.40:0/2366331345 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37300039a0 con 0x7f37380b46c0 2024-03-24T00:36:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f372f400640 1 -- 172.21.15.40:0/2366331345 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37300b2040 con 0x7f37380b46c0 2024-03-24T00:36:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f373e200640 1 -- 172.21.15.40:0/2366331345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f373813d080 con 0x7f37380b46c0 2024-03-24T00:36:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.974+0000 7f373e200640 1 -- 172.21.15.40:0/2366331345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37381415f0 con 0x7f37380b46c0 2024-03-24T00:36:10.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.978+0000 7f373e200640 1 -- 172.21.15.40:0/2366331345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3738068170 con 0x7f37380b46c0 2024-03-24T00:36:10.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.978+0000 7f372f400640 1 -- 172.21.15.40:0/2366331345 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f37300b8470 con 0x7f37380b46c0 2024-03-24T00:36:10.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.978+0000 7f372f400640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3714067660 0x7f3714069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:10.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.978+0000 7f372f400640 1 -- 172.21.15.40:0/2366331345 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37301204a0 con 0x7f37380b46c0 2024-03-24T00:36:10.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.978+0000 7f3737400640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3714067660 0x7f3714069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:10.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.978+0000 7f3737400640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3714067660 0x7f3714069b20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3728000f70 tx=0x7f3728009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:10.982+0000 7f372f400640 1 -- 172.21.15.40:0/2366331345 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3730016520 con 0x7f37380b46c0 2024-03-24T00:36:11.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.178+0000 7f373e200640 1 -- 172.21.15.40:0/2366331345 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3738072f50 con 0x7f37380b46c0 2024-03-24T00:36:11.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.178+0000 7f372f400640 1 -- 172.21.15.40:0/2366331345 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4976 (secure 0 0 0) 0x7f37300f0b60 con 0x7f37380b46c0 2024-03-24T00:36:11.182 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:11.182 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1097448819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1097448819},{"type":"v1","addr":"172.21.15.40:6834","nonce":1097448819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":6404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/983819364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":983819364},{"type":"v1","addr":"172.21.15.53:6837","nonce":983819364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2302548193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2302548193},{"type":"v1","addr":"172.21.15.40:6835","nonce":2302548193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:36:07.163876+0000","modified":"2024-03-24T00:36:10.190591+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7186":{"gid":7186,"name":"d","rank":0,"incarnation":78,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/1878888046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1878888046},{"type":"v1","addr":"172.21.15.53:6835","nonce":1878888046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7186,"qdb_cluster":[7186]},"id":6}]} 2024-03-24T00:36:11.183 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 82 2024-03-24T00:36:11.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.182+0000 7f373e200640 1 -- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3714067660 msgr2=0x7f3714069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.182+0000 7f373e200640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3714067660 0x7f3714069b20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f3728000f70 tx=0x7f3728009040 comp rx=0 tx=0).stop 2024-03-24T00:36:11.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.182+0000 7f373e200640 1 -- 172.21.15.40:0/2366331345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37380b46c0 msgr2=0x7f37381f7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.182+0000 7f373e200640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37380b46c0 0x7f37381f7a90 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f3730098450 tx=0x7f3730019610 comp rx=0 tx=0).stop 2024-03-24T00:36:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.182+0000 7f373e200640 1 -- 172.21.15.40:0/2366331345 shutdown_connections 2024-03-24T00:36:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.182+0000 7f373e200640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3714067660 0x7f3714069b20 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.182+0000 7f373e200640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f37380b46c0 0x7f37381f7a90 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.182+0000 7f373e200640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f37380b2ff0 0x7f373813dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.182+0000 7f373e200640 1 --2- 172.21.15.40:0/2366331345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f373806e620 0x7f373813caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.182+0000 7f373e200640 1 -- 172.21.15.40:0/2366331345 >> 172.21.15.40:0/2366331345 conn(0x7f37380b17b0 msgr2=0x7f3738071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.182+0000 7f373e200640 1 -- 172.21.15.40:0/2366331345 shutdown_connections 2024-03-24T00:36:11.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.182+0000 7f373e200640 1 -- 172.21.15.40:0/2366331345 wait complete. 2024-03-24T00:36:11.204 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:36:11.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3eda00640 1 Processor -- start 2024-03-24T00:36:11.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3eda00640 1 -- start start 2024-03-24T00:36:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3eda00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc3e80cfcc0 0x7fc3e80b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3eda00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3e80b4460 0x7fc3e8071560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3eda00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc3e8071aa0 0x7fc3e8161be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3eda00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e815e9b0 con 0x7fc3e8071aa0 2024-03-24T00:36:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3eda00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e80b4ae0 con 0x7fc3e80cfcc0 2024-03-24T00:36:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3eda00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80b4dc0 con 0x7fc3e80b4460 2024-03-24T00:36:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3e7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc3e8071aa0 0x7fc3e8161be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3de000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3e80b4460 0x7fc3e8071560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3e7400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc3e8071aa0 0x7fc3e8161be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39750/0 (socket says 172.21.15.40:39750) 2024-03-24T00:36:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3e6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc3e80cfcc0 0x7fc3e80b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3e7400640 1 -- 172.21.15.40:0/1762203393 learned_addr learned my addr 172.21.15.40:0/1762203393 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:11.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3e7400640 1 -- 172.21.15.40:0/1762203393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3e80b4460 msgr2=0x7fc3e8071560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:11.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3e7400640 1 --2- 172.21.15.40:0/1762203393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3e80b4460 0x7fc3e8071560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3e7400640 1 -- 172.21.15.40:0/1762203393 >> v2:172.21.15.53:3300/0 conn(0x7fc3e80cfcc0 msgr2=0x7fc3e80b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:11.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3e7400640 1 --2- 172.21.15.40:0/1762203393 >> v2:172.21.15.53:3300/0 conn(0x7fc3e80cfcc0 0x7fc3e80b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3e7400640 1 -- 172.21.15.40:0/1762203393 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e80ae740 con 0x7fc3e8071aa0 2024-03-24T00:36:11.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.342+0000 7fc3e6a00640 1 --2- 172.21.15.40:0/1762203393 >> v2:172.21.15.53:3300/0 conn(0x7fc3e80cfcc0 0x7fc3e80b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:11.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3e7400640 1 --2- 172.21.15.40:0/1762203393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e8071aa0 0x7fc3e8161be0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80b31e0 tx=0x7fc3e001c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38aaa92635e4560e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:11.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3e6000640 1 -- 172.21.15.40:0/1762203393 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e00a64d0 con 0x7fc3e8071aa0 2024-03-24T00:36:11.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3e6000640 1 -- 172.21.15.40:0/1762203393 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e00a6670 con 0x7fc3e8071aa0 2024-03-24T00:36:11.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3e6000640 1 -- 172.21.15.40:0/1762203393 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e00a6a40 con 0x7fc3e8071aa0 2024-03-24T00:36:11.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3eda00640 1 -- 172.21.15.40:0/1762203393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e8071aa0 msgr2=0x7fc3e8161be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3eda00640 1 --2- 172.21.15.40:0/1762203393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e8071aa0 0x7fc3e8161be0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80b31e0 tx=0x7fc3e001c730 comp rx=0 tx=0).stop 2024-03-24T00:36:11.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3eda00640 1 -- 172.21.15.40:0/1762203393 shutdown_connections 2024-03-24T00:36:11.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3eda00640 1 --2- 172.21.15.40:0/1762203393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e8071aa0 0x7fc3e8161be0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3eda00640 1 --2- 172.21.15.40:0/1762203393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3e80b4460 0x7fc3e8071560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3eda00640 1 --2- 172.21.15.40:0/1762203393 >> v2:172.21.15.53:3300/0 conn(0x7fc3e80cfcc0 0x7fc3e80b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3eda00640 1 -- 172.21.15.40:0/1762203393 >> 172.21.15.40:0/1762203393 conn(0x7fc3e80b21a0 msgr2=0x7fc3e80b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:11.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3eda00640 1 -- 172.21.15.40:0/1762203393 shutdown_connections 2024-03-24T00:36:11.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.346+0000 7fc3eda00640 1 -- 172.21.15.40:0/1762203393 wait complete. 2024-03-24T00:36:11.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3eda00640 1 Processor -- start 2024-03-24T00:36:11.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3eda00640 1 -- start start 2024-03-24T00:36:11.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3eda00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3e8071aa0 0x7fc3e813dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3eda00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3e80b4460 0x7fc3e8151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3eda00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e80cfcc0 0x7fc3e8151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3eda00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80d00a0 con 0x7fc3e80cfcc0 2024-03-24T00:36:11.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3eda00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80b4840 con 0x7fc3e80b4460 2024-03-24T00:36:11.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3eda00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80d0220 con 0x7fc3e8071aa0 2024-03-24T00:36:11.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3e7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e80cfcc0 0x7fc3e8151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3de000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3e80b4460 0x7fc3e8151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3e6a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3e8071aa0 0x7fc3e813dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3de000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3e80b4460 0x7fc3e8151700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:53834/0 (socket says 172.21.15.40:53834) 2024-03-24T00:36:11.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3de000640 1 -- 172.21.15.40:0/2066891273 learned_addr learned my addr 172.21.15.40:0/2066891273 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3de000640 1 -- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3e8071aa0 msgr2=0x7fc3e813dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3de000640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3e8071aa0 0x7fc3e813dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3de000640 1 -- 172.21.15.40:0/2066891273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e80cfcc0 msgr2=0x7fc3e8151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3de000640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e80cfcc0 0x7fc3e8151c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3de000640 1 -- 172.21.15.40:0/2066891273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e8151000 con 0x7fc3e80b4460 2024-03-24T00:36:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3e7400640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e80cfcc0 0x7fc3e8151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3e6a00640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3e8071aa0 0x7fc3e813dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:11.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3de000640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3e80b4460 0x7fc3e8151700 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000e870 tx=0x7fc3d000ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:11.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3df400640 1 -- 172.21.15.40:0/2066891273 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d0017070 con 0x7fc3e80b4460 2024-03-24T00:36:11.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3df400640 1 -- 172.21.15.40:0/2066891273 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d000c940 con 0x7fc3e80b4460 2024-03-24T00:36:11.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3df400640 1 -- 172.21.15.40:0/2066891273 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d000ccf0 con 0x7fc3e80b4460 2024-03-24T00:36:11.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3eda00640 1 -- 172.21.15.40:0/2066891273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e80b0b40 con 0x7fc3e80b4460 2024-03-24T00:36:11.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.350+0000 7fc3eda00640 1 -- 172.21.15.40:0/2066891273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e80b1100 con 0x7fc3e80b4460 2024-03-24T00:36:11.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.354+0000 7fc3eda00640 1 -- 172.21.15.40:0/2066891273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e8068170 con 0x7fc3e80b4460 2024-03-24T00:36:11.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.354+0000 7fc3df400640 1 -- 172.21.15.40:0/2066891273 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc3d001a070 con 0x7fc3e80b4460 2024-03-24T00:36:11.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.354+0000 7fc3df400640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc3c4067660 0x7fc3c4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.354+0000 7fc3df400640 1 -- 172.21.15.40:0/2066891273 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3d0085b10 con 0x7fc3e80b4460 2024-03-24T00:36:11.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.354+0000 7fc3e6a00640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc3c4067660 0x7fc3c4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.354+0000 7fc3e6a00640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc3c4067660 0x7fc3c4069b20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009580 tx=0x7fc3d8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:11.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.366+0000 7fc3df400640 1 -- 172.21.15.40:0/2066891273 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc3d00560a0 con 0x7fc3e80b4460 2024-03-24T00:36:11.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.530+0000 7fc3eda00640 1 -- 172.21.15.40:0/2066891273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3e8168b20 con 0x7fc3e80b4460 2024-03-24T00:36:11.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.530+0000 7fc3df400640 1 -- 172.21.15.40:0/2066891273 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16600 (secure 0 0 0) 0x7fc3d005a140 con 0x7fc3e80b4460 2024-03-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:11.535 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":83,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:36:08.785471+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:05.573740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-24T00:36:06.150226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.534+0000 7fc3eda00640 1 -- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc3c4067660 msgr2=0x7fc3c4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.534+0000 7fc3eda00640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc3c4067660 0x7fc3c4069b20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009580 tx=0x7fc3d8009190 comp rx=0 tx=0).stop 2024-03-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.534+0000 7fc3eda00640 1 -- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3e80b4460 msgr2=0x7fc3e8151700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.534+0000 7fc3eda00640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3e80b4460 0x7fc3e8151700 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc3d000e870 tx=0x7fc3d000ec70 comp rx=0 tx=0).stop 2024-03-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.534+0000 7fc3eda00640 1 -- 172.21.15.40:0/2066891273 shutdown_connections 2024-03-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.534+0000 7fc3eda00640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc3c4067660 0x7fc3c4069b20 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.534+0000 7fc3eda00640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc3e80cfcc0 0x7fc3e8151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.534+0000 7fc3eda00640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc3e80b4460 0x7fc3e8151700 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.534+0000 7fc3eda00640 1 --2- 172.21.15.40:0/2066891273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc3e8071aa0 0x7fc3e813dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.534+0000 7fc3eda00640 1 -- 172.21.15.40:0/2066891273 >> 172.21.15.40:0/2066891273 conn(0x7fc3e80b21a0 msgr2=0x7fc3e815d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:11.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.534+0000 7fc3eda00640 1 -- 172.21.15.40:0/2066891273 shutdown_connections 2024-03-24T00:36:11.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.534+0000 7fc3eda00640 1 -- 172.21.15.40:0/2066891273 wait complete. 2024-03-24T00:36:11.557 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:36:11.557 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:36:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.694+0000 7fe3d4000640 1 Processor -- start 2024-03-24T00:36:11.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.694+0000 7fe3d4000640 1 -- start start 2024-03-24T00:36:11.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.694+0000 7fe3d4000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3cc141bc0 0x7fe3cc0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.694+0000 7fe3d4000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe3cc0ce630 0x7fe3cc071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.694+0000 7fe3d4000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3cc070e00 con 0x7fe3cc13ec80 2024-03-24T00:36:11.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.694+0000 7fe3d4000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3cc0cee20 con 0x7fe3cc0ce630 2024-03-24T00:36:11.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.694+0000 7fe3d4000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0cf100 con 0x7fe3cc141bc0 2024-03-24T00:36:11.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.694+0000 7fe3d2000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe3cc0ce630 0x7fe3cc071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.694+0000 7fe3d1600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe3cc13ec80 0x7fe3cc1414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42964/0 (socket says 172.21.15.40:42964) 2024-03-24T00:36:11.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.694+0000 7fe3d1600640 1 -- 172.21.15.40:0/1217989411 learned_addr learned my addr 172.21.15.40:0/1217989411 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:11.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.694+0000 7fe3d0c00640 1 --2- 172.21.15.40:0/1217989411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3cc141bc0 0x7fe3cc0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3d0c00640 1 -- 172.21.15.40:0/1217989411 >> v2:172.21.15.40:3300/0 conn(0x7fe3cc0ce630 msgr2=0x7fe3cc071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:11.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3d0c00640 1 --2- 172.21.15.40:0/1217989411 >> v2:172.21.15.40:3300/0 conn(0x7fe3cc0ce630 0x7fe3cc071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3d0c00640 1 -- 172.21.15.40:0/1217989411 >> v1:172.21.15.40:6789/0 conn(0x7fe3cc13ec80 legacy=0x7fe3cc1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3d0c00640 1 -- 172.21.15.40:0/1217989411 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc0cf3e0 con 0x7fe3cc141bc0 2024-03-24T00:36:11.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3d2000640 1 --2- 172.21.15.40:0/1217989411 >> v2:172.21.15.40:3300/0 conn(0x7fe3cc0ce630 0x7fe3cc071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:11.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3d0c00640 1 --2- 172.21.15.40:0/1217989411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3cc141bc0 0x7fe3cc0ce0f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80097c0 tx=0x7fe3b801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4653861d11fed18 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:11.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3cbe00640 1 -- 172.21.15.40:0/1217989411 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80aa620 con 0x7fe3cc141bc0 2024-03-24T00:36:11.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3cbe00640 1 -- 172.21.15.40:0/1217989411 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b80af070 con 0x7fe3cc141bc0 2024-03-24T00:36:11.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3cbe00640 1 -- 172.21.15.40:0/1217989411 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80aa9e0 con 0x7fe3cc141bc0 2024-03-24T00:36:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3d4000640 1 -- 172.21.15.40:0/1217989411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3cc141bc0 msgr2=0x7fe3cc0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3d4000640 1 --2- 172.21.15.40:0/1217989411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3cc141bc0 0x7fe3cc0ce0f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80097c0 tx=0x7fe3b801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3d4000640 1 -- 172.21.15.40:0/1217989411 shutdown_connections 2024-03-24T00:36:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3d4000640 1 --2- 172.21.15.40:0/1217989411 >> v2:172.21.15.40:3300/0 conn(0x7fe3cc0ce630 0x7fe3cc071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3d4000640 1 --2- 172.21.15.40:0/1217989411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3cc141bc0 0x7fe3cc0ce0f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.698+0000 7fe3d4000640 1 -- 172.21.15.40:0/1217989411 >> 172.21.15.40:0/1217989411 conn(0x7fe3cc0b17b0 msgr2=0x7fe3cc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d4000640 1 -- 172.21.15.40:0/1217989411 shutdown_connections 2024-03-24T00:36:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d4000640 1 -- 172.21.15.40:0/1217989411 wait complete. 2024-03-24T00:36:11.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d4000640 1 Processor -- start 2024-03-24T00:36:11.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d4000640 1 -- start start 2024-03-24T00:36:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d4000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3cc0ce630 0x7fe3cc0b3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d4000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3cc13ec80 0x7fe3cc148cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d4000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3cc141bc0 0x7fe3cc140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d4000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0723e0 con 0x7fe3cc13ec80 2024-03-24T00:36:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d4000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc072260 con 0x7fe3cc141bc0 2024-03-24T00:36:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d4000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc072560 con 0x7fe3cc0ce630 2024-03-24T00:36:11.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3cc13ec80 0x7fe3cc148cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d1600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3cc0ce630 0x7fe3cc0b3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d2000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3cc141bc0 0x7fe3cc140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d1600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3cc0ce630 0x7fe3cc0b3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:46632/0 (socket says 172.21.15.40:46632) 2024-03-24T00:36:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d1600640 1 -- 172.21.15.40:0/3233948374 learned_addr learned my addr 172.21.15.40:0/3233948374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d0c00640 1 -- 172.21.15.40:0/3233948374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3cc0ce630 msgr2=0x7fe3cc0b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d0c00640 1 --2- 172.21.15.40:0/3233948374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3cc0ce630 0x7fe3cc0b3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d0c00640 1 -- 172.21.15.40:0/3233948374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3cc141bc0 msgr2=0x7fe3cc140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d0c00640 1 --2- 172.21.15.40:0/3233948374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3cc141bc0 0x7fe3cc140b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d0c00640 1 -- 172.21.15.40:0/3233948374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc0b2f30 con 0x7fe3cc13ec80 2024-03-24T00:36:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.702+0000 7fe3d0c00640 1 --2- 172.21.15.40:0/3233948374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3cc13ec80 0x7fe3cc148cd0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80ab000 tx=0x7fe3b801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.706+0000 7fe3c9600640 1 -- 172.21.15.40:0/3233948374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80aac10 con 0x7fe3cc13ec80 2024-03-24T00:36:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.706+0000 7fe3c9600640 1 -- 172.21.15.40:0/3233948374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b80aadb0 con 0x7fe3cc13ec80 2024-03-24T00:36:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.706+0000 7fe3c9600640 1 -- 172.21.15.40:0/3233948374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b80b4640 con 0x7fe3cc13ec80 2024-03-24T00:36:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.706+0000 7fe3d4000640 1 -- 172.21.15.40:0/3233948374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3cc0b3210 con 0x7fe3cc13ec80 2024-03-24T00:36:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.706+0000 7fe3d4000640 1 -- 172.21.15.40:0/3233948374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3cc0b37d0 con 0x7fe3cc13ec80 2024-03-24T00:36:11.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.706+0000 7fe3d4000640 1 -- 172.21.15.40:0/3233948374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3cc068170 con 0x7fe3cc13ec80 2024-03-24T00:36:11.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.706+0000 7fe3c9600640 1 -- 172.21.15.40:0/3233948374 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe3b80bc070 con 0x7fe3cc13ec80 2024-03-24T00:36:11.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.706+0000 7fe3c9600640 1 --2- 172.21.15.40:0/3233948374 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe3ac067660 0x7fe3ac069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:11.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.706+0000 7fe3c9600640 1 -- 172.21.15.40:0/3233948374 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3b811eff0 con 0x7fe3cc13ec80 2024-03-24T00:36:11.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.706+0000 7fe3d1600640 1 --2- 172.21.15.40:0/3233948374 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe3ac067660 0x7fe3ac069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:11.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.710+0000 7fe3d1600640 1 --2- 172.21.15.40:0/3233948374 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe3ac067660 0x7fe3ac069b20 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0007fd0 tx=0x7fe3c0007f20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.710+0000 7fe3c9600640 1 -- 172.21.15.40:0/3233948374 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe3b8016090 con 0x7fe3cc13ec80 2024-03-24T00:36:11.722 INFO:tasks.ceph.mds.d.smithi053.stderr:2024-03-24T00:36:11.720+0000 7fc819600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:36:11.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.874+0000 7fe3d4000640 1 -- 172.21.15.40:0/3233948374 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3cc072b90 con 0x7fe3ac067660 2024-03-24T00:36:11.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.882+0000 7fe3c9600640 1 -- 172.21.15.40:0/3233948374 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252007 (secure 0 0 0) 0x7fe3cc072b90 con 0x7fe3ac067660 2024-03-24T00:36:11.890 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:36:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.886+0000 7fe3d4000640 1 -- 172.21.15.40:0/3233948374 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe3ac067660 msgr2=0x7fe3ac069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.886+0000 7fe3d4000640 1 --2- 172.21.15.40:0/3233948374 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe3ac067660 0x7fe3ac069b20 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe3c0007fd0 tx=0x7fe3c0007f20 comp rx=0 tx=0).stop 2024-03-24T00:36:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.886+0000 7fe3d4000640 1 -- 172.21.15.40:0/3233948374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3cc13ec80 msgr2=0x7fe3cc148cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.886+0000 7fe3d4000640 1 --2- 172.21.15.40:0/3233948374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3cc13ec80 0x7fe3cc148cd0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fe3b80ab000 tx=0x7fe3b801ace0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.886+0000 7fe3d4000640 1 -- 172.21.15.40:0/3233948374 shutdown_connections 2024-03-24T00:36:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.886+0000 7fe3d4000640 1 --2- 172.21.15.40:0/3233948374 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe3ac067660 0x7fe3ac069b20 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.886+0000 7fe3d4000640 1 --2- 172.21.15.40:0/3233948374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3cc141bc0 0x7fe3cc140b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.886+0000 7fe3d4000640 1 --2- 172.21.15.40:0/3233948374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe3cc13ec80 0x7fe3cc148cd0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.886+0000 7fe3d4000640 1 --2- 172.21.15.40:0/3233948374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3cc0ce630 0x7fe3cc0b3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:11.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.886+0000 7fe3d4000640 1 -- 172.21.15.40:0/3233948374 >> 172.21.15.40:0/3233948374 conn(0x7fe3cc0b17b0 msgr2=0x7fe3cc15e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:11.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.886+0000 7fe3d4000640 1 -- 172.21.15.40:0/3233948374 shutdown_connections 2024-03-24T00:36:11.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:11.886+0000 7fe3d4000640 1 -- 172.21.15.40:0/3233948374 wait complete. 2024-03-24T00:36:11.907 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-03-24T00:36:11.260349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":287608,"kb_used_data":9040,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":754687112,"statfs":{"total":773094113280,"available":772799602688,"internally_reserved":0,"allocated":9256960,"data_stored":4637147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":285134386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.507175"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.816677+0000","last_change":"2024-03-24T00:36:07.809594+0000","last_active":"2024-03-24T00:36:07.816677+0000","last_peered":"2024-03-24T00:36:07.816677+0000","last_clean":"2024-03-24T00:36:07.816677+0000","last_became_active":"2024-03-24T00:36:07.809057+0000","last_became_peered":"2024-03-24T00:36:07.809057+0000","last_unstale":"2024-03-24T00:36:07.816677+0000","last_undegraded":"2024-03-24T00:36:07.816677+0000","last_fullsized":"2024-03-24T00:36:07.816677+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:07.699038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.807251+0000","last_change":"2024-03-24T00:36:07.806701+0000","last_active":"2024-03-24T00:36:07.807251+0000","last_peered":"2024-03-24T00:36:07.807251+0000","last_clean":"2024-03-24T00:36:07.807251+0000","last_became_active":"2024-03-24T00:36:07.806310+0000","last_became_peered":"2024-03-24T00:36:07.806310+0000","last_unstale":"2024-03-24T00:36:07.807251+0000","last_undegraded":"2024-03-24T00:36:07.807251+0000","last_fullsized":"2024-03-24T00:36:07.807251+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:26.392012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.810302+0000","last_change":"2024-03-24T00:36:07.806772+0000","last_active":"2024-03-24T00:36:07.810302+0000","last_peered":"2024-03-24T00:36:07.810302+0000","last_clean":"2024-03-24T00:36:07.810302+0000","last_became_active":"2024-03-24T00:36:07.805929+0000","last_became_peered":"2024-03-24T00:36:07.805929+0000","last_unstale":"2024-03-24T00:36:07.810302+0000","last_undegraded":"2024-03-24T00:36:07.810302+0000","last_fullsized":"2024-03-24T00:36:07.810302+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:44.777901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.802651+0000","last_change":"2024-03-24T00:36:07.799542+0000","last_active":"2024-03-24T00:36:07.802651+0000","last_peered":"2024-03-24T00:36:07.802651+0000","last_clean":"2024-03-24T00:36:07.802651+0000","last_became_active":"2024-03-24T00:36:07.799254+0000","last_became_peered":"2024-03-24T00:36:07.799254+0000","last_unstale":"2024-03-24T00:36:07.802651+0000","last_undegraded":"2024-03-24T00:36:07.802651+0000","last_fullsized":"2024-03-24T00:36:07.802651+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:07:22.117218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.808247+0000","last_change":"2024-03-24T00:36:07.800263+0000","last_active":"2024-03-24T00:36:07.808247+0000","last_peered":"2024-03-24T00:36:07.808247+0000","last_clean":"2024-03-24T00:36:07.808247+0000","last_became_active":"2024-03-24T00:36:07.796248+0000","last_became_peered":"2024-03-24T00:36:07.796248+0000","last_unstale":"2024-03-24T00:36:07.808247+0000","last_undegraded":"2024-03-24T00:36:07.808247+0000","last_fullsized":"2024-03-24T00:36:07.808247+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:47.539461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.802160+0000","last_change":"2024-03-24T00:36:07.800419+0000","last_active":"2024-03-24T00:36:07.802160+0000","last_peered":"2024-03-24T00:36:07.802160+0000","last_clean":"2024-03-24T00:36:07.802160+0000","last_became_active":"2024-03-24T00:36:07.800132+0000","last_became_peered":"2024-03-24T00:36:07.800132+0000","last_unstale":"2024-03-24T00:36:07.802160+0000","last_undegraded":"2024-03-24T00:36:07.802160+0000","last_fullsized":"2024-03-24T00:36:07.802160+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:07.699038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.804144+0000","last_change":"2024-03-24T00:36:07.802548+0000","last_active":"2024-03-24T00:36:07.804144+0000","last_peered":"2024-03-24T00:36:07.804144+0000","last_clean":"2024-03-24T00:36:07.804144+0000","last_became_active":"2024-03-24T00:36:07.802270+0000","last_became_peered":"2024-03-24T00:36:07.802270+0000","last_unstale":"2024-03-24T00:36:07.804144+0000","last_undegraded":"2024-03-24T00:36:07.804144+0000","last_fullsized":"2024-03-24T00:36:07.804144+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:33:37.729814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786870+0000","last_change":"2024-03-24T00:36:07.782456+0000","last_active":"2024-03-24T00:36:07.786870+0000","last_peered":"2024-03-24T00:36:07.786870+0000","last_clean":"2024-03-24T00:36:07.786870+0000","last_became_active":"2024-03-24T00:36:07.782156+0000","last_became_peered":"2024-03-24T00:36:07.782156+0000","last_unstale":"2024-03-24T00:36:07.786870+0000","last_undegraded":"2024-03-24T00:36:07.786870+0000","last_fullsized":"2024-03-24T00:36:07.786870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:00:14.167105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.796509+0000","last_change":"2024-03-24T00:36:07.791775+0000","last_active":"2024-03-24T00:36:07.796509+0000","last_peered":"2024-03-24T00:36:07.796509+0000","last_clean":"2024-03-24T00:36:07.796509+0000","last_became_active":"2024-03-24T00:36:07.791437+0000","last_became_peered":"2024-03-24T00:36:07.791437+0000","last_unstale":"2024-03-24T00:36:07.796509+0000","last_undegraded":"2024-03-24T00:36:07.796509+0000","last_fullsized":"2024-03-24T00:36:07.796509+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:35:17.249084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.809107+0000","last_change":"2024-03-24T00:36:07.797117+0000","last_active":"2024-03-24T00:36:07.809107+0000","last_peered":"2024-03-24T00:36:07.809107+0000","last_clean":"2024-03-24T00:36:07.809107+0000","last_became_active":"2024-03-24T00:36:07.793748+0000","last_became_peered":"2024-03-24T00:36:07.793748+0000","last_unstale":"2024-03-24T00:36:07.809107+0000","last_undegraded":"2024-03-24T00:36:07.809107+0000","last_fullsized":"2024-03-24T00:36:07.809107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:43.247886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.793599+0000","last_change":"2024-03-24T00:36:07.782891+0000","last_active":"2024-03-24T00:36:07.793599+0000","last_peered":"2024-03-24T00:36:07.793599+0000","last_clean":"2024-03-24T00:36:07.793599+0000","last_became_active":"2024-03-24T00:36:07.782573+0000","last_became_peered":"2024-03-24T00:36:07.782573+0000","last_unstale":"2024-03-24T00:36:07.793599+0000","last_undegraded":"2024-03-24T00:36:07.793599+0000","last_fullsized":"2024-03-24T00:36:07.793599+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:41:34.955684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.817169+0000","last_change":"2024-03-24T00:36:07.806975+0000","last_active":"2024-03-24T00:36:07.817169+0000","last_peered":"2024-03-24T00:36:07.817169+0000","last_clean":"2024-03-24T00:36:07.817169+0000","last_became_active":"2024-03-24T00:36:07.806633+0000","last_became_peered":"2024-03-24T00:36:07.806633+0000","last_unstale":"2024-03-24T00:36:07.817169+0000","last_undegraded":"2024-03-24T00:36:07.817169+0000","last_fullsized":"2024-03-24T00:36:07.817169+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:09:13.747978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.803061+0000","last_change":"2024-03-24T00:36:07.793547+0000","last_active":"2024-03-24T00:36:07.803061+0000","last_peered":"2024-03-24T00:36:07.803061+0000","last_clean":"2024-03-24T00:36:07.803061+0000","last_became_active":"2024-03-24T00:36:07.793254+0000","last_became_peered":"2024-03-24T00:36:07.793254+0000","last_unstale":"2024-03-24T00:36:07.803061+0000","last_undegraded":"2024-03-24T00:36:07.803061+0000","last_fullsized":"2024-03-24T00:36:07.803061+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:27.573566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.802799+0000","last_change":"2024-03-24T00:36:07.800373+0000","last_active":"2024-03-24T00:36:07.802799+0000","last_peered":"2024-03-24T00:36:07.802799+0000","last_clean":"2024-03-24T00:36:07.802799+0000","last_became_active":"2024-03-24T00:36:07.796861+0000","last_became_peered":"2024-03-24T00:36:07.796861+0000","last_unstale":"2024-03-24T00:36:07.802799+0000","last_undegraded":"2024-03-24T00:36:07.802799+0000","last_fullsized":"2024-03-24T00:36:07.802799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:07.596033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786362+0000","last_change":"2024-03-24T00:36:07.784028+0000","last_active":"2024-03-24T00:36:07.786362+0000","last_peered":"2024-03-24T00:36:07.786362+0000","last_clean":"2024-03-24T00:36:07.786362+0000","last_became_active":"2024-03-24T00:36:07.783398+0000","last_became_peered":"2024-03-24T00:36:07.783398+0000","last_unstale":"2024-03-24T00:36:07.786362+0000","last_undegraded":"2024-03-24T00:36:07.786362+0000","last_fullsized":"2024-03-24T00:36:07.786362+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:51:24.214744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.815908+0000","last_change":"2024-03-24T00:36:07.813152+0000","last_active":"2024-03-24T00:36:07.815908+0000","last_peered":"2024-03-24T00:36:07.815908+0000","last_clean":"2024-03-24T00:36:07.815908+0000","last_became_active":"2024-03-24T00:36:07.812863+0000","last_became_peered":"2024-03-24T00:36:07.812863+0000","last_unstale":"2024-03-24T00:36:07.815908+0000","last_undegraded":"2024-03-24T00:36:07.815908+0000","last_fullsized":"2024-03-24T00:36:07.815908+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:25:52.928229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.809522+0000","last_change":"2024-03-24T00:36:07.799799+0000","last_active":"2024-03-24T00:36:07.809522+0000","last_peered":"2024-03-24T00:36:07.809522+0000","last_clean":"2024-03-24T00:36:07.809522+0000","last_became_active":"2024-03-24T00:36:07.799529+0000","last_became_peered":"2024-03-24T00:36:07.799529+0000","last_unstale":"2024-03-24T00:36:07.809522+0000","last_undegraded":"2024-03-24T00:36:07.809522+0000","last_fullsized":"2024-03-24T00:36:07.809522+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:15.579736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.810663+0000","last_change":"2024-03-24T00:36:07.809376+0000","last_active":"2024-03-24T00:36:07.810663+0000","last_peered":"2024-03-24T00:36:07.810663+0000","last_clean":"2024-03-24T00:36:07.810663+0000","last_became_active":"2024-03-24T00:36:07.808991+0000","last_became_peered":"2024-03-24T00:36:07.808991+0000","last_unstale":"2024-03-24T00:36:07.810663+0000","last_undegraded":"2024-03-24T00:36:07.810663+0000","last_fullsized":"2024-03-24T00:36:07.810663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:45.595754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.791111+0000","last_change":"2024-03-24T00:36:07.789129+0000","last_active":"2024-03-24T00:36:07.791111+0000","last_peered":"2024-03-24T00:36:07.791111+0000","last_clean":"2024-03-24T00:36:07.791111+0000","last_became_active":"2024-03-24T00:36:07.788771+0000","last_became_peered":"2024-03-24T00:36:07.788771+0000","last_unstale":"2024-03-24T00:36:07.791111+0000","last_undegraded":"2024-03-24T00:36:07.791111+0000","last_fullsized":"2024-03-24T00:36:07.791111+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:03:07.442469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.799436+0000","last_change":"2024-03-24T00:36:07.794991+0000","last_active":"2024-03-24T00:36:07.799436+0000","last_peered":"2024-03-24T00:36:07.799436+0000","last_clean":"2024-03-24T00:36:07.799436+0000","last_became_active":"2024-03-24T00:36:07.794723+0000","last_became_peered":"2024-03-24T00:36:07.794723+0000","last_unstale":"2024-03-24T00:36:07.799436+0000","last_undegraded":"2024-03-24T00:36:07.799436+0000","last_fullsized":"2024-03-24T00:36:07.799436+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:49.895037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.803567+0000","last_change":"2024-03-24T00:36:07.802043+0000","last_active":"2024-03-24T00:36:07.803567+0000","last_peered":"2024-03-24T00:36:07.803567+0000","last_clean":"2024-03-24T00:36:07.803567+0000","last_became_active":"2024-03-24T00:36:07.801739+0000","last_became_peered":"2024-03-24T00:36:07.801739+0000","last_unstale":"2024-03-24T00:36:07.803567+0000","last_undegraded":"2024-03-24T00:36:07.803567+0000","last_fullsized":"2024-03-24T00:36:07.803567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:29.771243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.798748+0000","last_change":"2024-03-24T00:36:07.795150+0000","last_active":"2024-03-24T00:36:07.798748+0000","last_peered":"2024-03-24T00:36:07.798748+0000","last_clean":"2024-03-24T00:36:07.798748+0000","last_became_active":"2024-03-24T00:36:07.794859+0000","last_became_peered":"2024-03-24T00:36:07.794859+0000","last_unstale":"2024-03-24T00:36:07.798748+0000","last_undegraded":"2024-03-24T00:36:07.798748+0000","last_fullsized":"2024-03-24T00:36:07.798748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:26:46.585200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.810197+0000","last_change":"2024-03-24T00:36:07.803372+0000","last_active":"2024-03-24T00:36:07.810197+0000","last_peered":"2024-03-24T00:36:07.810197+0000","last_clean":"2024-03-24T00:36:07.810197+0000","last_became_active":"2024-03-24T00:36:07.803025+0000","last_became_peered":"2024-03-24T00:36:07.803025+0000","last_unstale":"2024-03-24T00:36:07.810197+0000","last_undegraded":"2024-03-24T00:36:07.810197+0000","last_fullsized":"2024-03-24T00:36:07.810197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:00:14.167105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.800225+0000","last_change":"2024-03-24T00:36:07.797993+0000","last_active":"2024-03-24T00:36:07.800225+0000","last_peered":"2024-03-24T00:36:07.800225+0000","last_clean":"2024-03-24T00:36:07.800225+0000","last_became_active":"2024-03-24T00:36:07.794434+0000","last_became_peered":"2024-03-24T00:36:07.794434+0000","last_unstale":"2024-03-24T00:36:07.800225+0000","last_undegraded":"2024-03-24T00:36:07.800225+0000","last_fullsized":"2024-03-24T00:36:07.800225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:44.877088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.812895+0000","last_change":"2024-03-24T00:36:07.810464+0000","last_active":"2024-03-24T00:36:07.812895+0000","last_peered":"2024-03-24T00:36:07.812895+0000","last_clean":"2024-03-24T00:36:07.812895+0000","last_became_active":"2024-03-24T00:36:07.809970+0000","last_became_peered":"2024-03-24T00:36:07.809970+0000","last_unstale":"2024-03-24T00:36:07.812895+0000","last_undegraded":"2024-03-24T00:36:07.812895+0000","last_fullsized":"2024-03-24T00:36:07.812895+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:14.714477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.801693+0000","last_change":"2024-03-24T00:36:07.789583+0000","last_active":"2024-03-24T00:36:07.801693+0000","last_peered":"2024-03-24T00:36:07.801693+0000","last_clean":"2024-03-24T00:36:07.801693+0000","last_became_active":"2024-03-24T00:36:07.784901+0000","last_became_peered":"2024-03-24T00:36:07.784901+0000","last_unstale":"2024-03-24T00:36:07.801693+0000","last_undegraded":"2024-03-24T00:36:07.801693+0000","last_fullsized":"2024-03-24T00:36:07.801693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:36:22.540685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.804759+0000","last_change":"2024-03-24T00:36:07.797595+0000","last_active":"2024-03-24T00:36:07.804759+0000","last_peered":"2024-03-24T00:36:07.804759+0000","last_clean":"2024-03-24T00:36:07.804759+0000","last_became_active":"2024-03-24T00:36:07.797216+0000","last_became_peered":"2024-03-24T00:36:07.797216+0000","last_unstale":"2024-03-24T00:36:07.804759+0000","last_undegraded":"2024-03-24T00:36:07.804759+0000","last_fullsized":"2024-03-24T00:36:07.804759+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:32:15.351688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.817192+0000","last_change":"2024-03-24T00:36:07.810139+0000","last_active":"2024-03-24T00:36:07.817192+0000","last_peered":"2024-03-24T00:36:07.817192+0000","last_clean":"2024-03-24T00:36:07.817192+0000","last_became_active":"2024-03-24T00:36:07.806399+0000","last_became_peered":"2024-03-24T00:36:07.806399+0000","last_unstale":"2024-03-24T00:36:07.817192+0000","last_undegraded":"2024-03-24T00:36:07.817192+0000","last_fullsized":"2024-03-24T00:36:07.817192+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:15.579736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.794542+0000","last_change":"2024-03-24T00:36:07.784265+0000","last_active":"2024-03-24T00:36:07.794542+0000","last_peered":"2024-03-24T00:36:07.794542+0000","last_clean":"2024-03-24T00:36:07.794542+0000","last_became_active":"2024-03-24T00:36:07.783997+0000","last_became_peered":"2024-03-24T00:36:07.783997+0000","last_unstale":"2024-03-24T00:36:07.794542+0000","last_undegraded":"2024-03-24T00:36:07.794542+0000","last_fullsized":"2024-03-24T00:36:07.794542+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:04:21.024582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.801339+0000","last_change":"2024-03-24T00:36:07.799677+0000","last_active":"2024-03-24T00:36:07.801339+0000","last_peered":"2024-03-24T00:36:07.801339+0000","last_clean":"2024-03-24T00:36:07.801339+0000","last_became_active":"2024-03-24T00:36:07.799366+0000","last_became_peered":"2024-03-24T00:36:07.799366+0000","last_unstale":"2024-03-24T00:36:07.801339+0000","last_undegraded":"2024-03-24T00:36:07.801339+0000","last_fullsized":"2024-03-24T00:36:07.801339+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:36.588320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.809924+0000","last_change":"2024-03-24T00:36:07.801007+0000","last_active":"2024-03-24T00:36:07.809924+0000","last_peered":"2024-03-24T00:36:07.809924+0000","last_clean":"2024-03-24T00:36:07.809924+0000","last_became_active":"2024-03-24T00:36:07.796203+0000","last_became_peered":"2024-03-24T00:36:07.796203+0000","last_unstale":"2024-03-24T00:36:07.809924+0000","last_undegraded":"2024-03-24T00:36:07.809924+0000","last_fullsized":"2024-03-24T00:36:07.809924+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:51.731361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.809268+0000","last_change":"2024-03-24T00:36:07.797764+0000","last_active":"2024-03-24T00:36:07.809268+0000","last_peered":"2024-03-24T00:36:07.809268+0000","last_clean":"2024-03-24T00:36:07.809268+0000","last_became_active":"2024-03-24T00:36:07.797501+0000","last_became_peered":"2024-03-24T00:36:07.797501+0000","last_unstale":"2024-03-24T00:36:07.809268+0000","last_undegraded":"2024-03-24T00:36:07.809268+0000","last_fullsized":"2024-03-24T00:36:07.809268+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:36.588320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.809576+0000","last_change":"2024-03-24T00:36:07.799673+0000","last_active":"2024-03-24T00:36:07.809576+0000","last_peered":"2024-03-24T00:36:07.809576+0000","last_clean":"2024-03-24T00:36:07.809576+0000","last_became_active":"2024-03-24T00:36:07.799407+0000","last_became_peered":"2024-03-24T00:36:07.799407+0000","last_unstale":"2024-03-24T00:36:07.809576+0000","last_undegraded":"2024-03-24T00:36:07.809576+0000","last_fullsized":"2024-03-24T00:36:07.809576+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:07.699038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"82'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.822139+0000","last_change":"2024-03-24T00:36:06.779007+0000","last_active":"2024-03-24T00:36:07.822139+0000","last_peered":"2024-03-24T00:36:07.822139+0000","last_clean":"2024-03-24T00:36:07.822139+0000","last_became_active":"2024-03-24T00:36:06.778716+0000","last_became_peered":"2024-03-24T00:36:06.778716+0000","last_unstale":"2024-03-24T00:36:07.822139+0000","last_undegraded":"2024-03-24T00:36:07.822139+0000","last_fullsized":"2024-03-24T00:36:07.822139+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:02:11.729651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.816632+0000","last_change":"2024-03-24T00:36:07.809262+0000","last_active":"2024-03-24T00:36:07.816632+0000","last_peered":"2024-03-24T00:36:07.816632+0000","last_clean":"2024-03-24T00:36:07.816632+0000","last_became_active":"2024-03-24T00:36:07.808496+0000","last_became_peered":"2024-03-24T00:36:07.808496+0000","last_unstale":"2024-03-24T00:36:07.816632+0000","last_undegraded":"2024-03-24T00:36:07.816632+0000","last_fullsized":"2024-03-24T00:36:07.816632+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:35:17.249084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"82'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.823154+0000","last_change":"2024-03-24T00:36:06.788811+0000","last_active":"2024-03-24T00:36:07.823154+0000","last_peered":"2024-03-24T00:36:07.823154+0000","last_clean":"2024-03-24T00:36:07.823154+0000","last_became_active":"2024-03-24T00:36:06.788484+0000","last_became_peered":"2024-03-24T00:36:06.788484+0000","last_unstale":"2024-03-24T00:36:07.823154+0000","last_undegraded":"2024-03-24T00:36:07.823154+0000","last_fullsized":"2024-03-24T00:36:07.823154+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:00:13.159554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.809975+0000","last_change":"2024-03-24T00:36:07.798671+0000","last_active":"2024-03-24T00:36:07.809975+0000","last_peered":"2024-03-24T00:36:07.809975+0000","last_clean":"2024-03-24T00:36:07.809975+0000","last_became_active":"2024-03-24T00:36:07.798336+0000","last_became_peered":"2024-03-24T00:36:07.798336+0000","last_unstale":"2024-03-24T00:36:07.809975+0000","last_undegraded":"2024-03-24T00:36:07.809975+0000","last_fullsized":"2024-03-24T00:36:07.809975+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:02.365494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.818858+0000","last_change":"2024-03-24T00:36:06.776010+0000","last_active":"2024-03-24T00:36:07.818858+0000","last_peered":"2024-03-24T00:36:07.818858+0000","last_clean":"2024-03-24T00:36:07.818858+0000","last_became_active":"2024-03-24T00:36:06.775739+0000","last_became_peered":"2024-03-24T00:36:06.775739+0000","last_unstale":"2024-03-24T00:36:07.818858+0000","last_undegraded":"2024-03-24T00:36:07.818858+0000","last_fullsized":"2024-03-24T00:36:07.818858+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:07.273374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.805103+0000","last_change":"2024-03-24T00:36:07.803836+0000","last_active":"2024-03-24T00:36:07.805103+0000","last_peered":"2024-03-24T00:36:07.805103+0000","last_clean":"2024-03-24T00:36:07.805103+0000","last_became_active":"2024-03-24T00:36:07.802301+0000","last_became_peered":"2024-03-24T00:36:07.802301+0000","last_unstale":"2024-03-24T00:36:07.805103+0000","last_undegraded":"2024-03-24T00:36:07.805103+0000","last_fullsized":"2024-03-24T00:36:07.805103+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:07.407835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.819201+0000","last_change":"2024-03-24T00:36:06.775287+0000","last_active":"2024-03-24T00:36:07.819201+0000","last_peered":"2024-03-24T00:36:07.819201+0000","last_clean":"2024-03-24T00:36:07.819201+0000","last_became_active":"2024-03-24T00:36:06.774998+0000","last_became_peered":"2024-03-24T00:36:06.774998+0000","last_unstale":"2024-03-24T00:36:07.819201+0000","last_undegraded":"2024-03-24T00:36:07.819201+0000","last_fullsized":"2024-03-24T00:36:07.819201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:26.566015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.801010+0000","last_change":"2024-03-24T00:36:07.799119+0000","last_active":"2024-03-24T00:36:07.801010+0000","last_peered":"2024-03-24T00:36:07.801010+0000","last_clean":"2024-03-24T00:36:07.801010+0000","last_became_active":"2024-03-24T00:36:07.798807+0000","last_became_peered":"2024-03-24T00:36:07.798807+0000","last_unstale":"2024-03-24T00:36:07.801010+0000","last_undegraded":"2024-03-24T00:36:07.801010+0000","last_fullsized":"2024-03-24T00:36:07.801010+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:43.247886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"82'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.819579+0000","last_change":"2024-03-24T00:36:06.766374+0000","last_active":"2024-03-24T00:36:07.819579+0000","last_peered":"2024-03-24T00:36:07.819579+0000","last_clean":"2024-03-24T00:36:07.819579+0000","last_became_active":"2024-03-24T00:36:06.765492+0000","last_became_peered":"2024-03-24T00:36:06.765492+0000","last_unstale":"2024-03-24T00:36:07.819579+0000","last_undegraded":"2024-03-24T00:36:07.819579+0000","last_fullsized":"2024-03-24T00:36:07.819579+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:41.584491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.808272+0000","last_change":"2024-03-24T00:36:07.803639+0000","last_active":"2024-03-24T00:36:07.808272+0000","last_peered":"2024-03-24T00:36:07.808272+0000","last_clean":"2024-03-24T00:36:07.808272+0000","last_became_active":"2024-03-24T00:36:07.803357+0000","last_became_peered":"2024-03-24T00:36:07.803357+0000","last_unstale":"2024-03-24T00:36:07.808272+0000","last_undegraded":"2024-03-24T00:36:07.808272+0000","last_fullsized":"2024-03-24T00:36:07.808272+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:31.695719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"82'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.820717+0000","last_change":"2024-03-24T00:36:06.788690+0000","last_active":"2024-03-24T00:36:07.820717+0000","last_peered":"2024-03-24T00:36:07.820717+0000","last_clean":"2024-03-24T00:36:07.820717+0000","last_became_active":"2024-03-24T00:36:06.788392+0000","last_became_peered":"2024-03-24T00:36:06.788392+0000","last_unstale":"2024-03-24T00:36:07.820717+0000","last_undegraded":"2024-03-24T00:36:07.820717+0000","last_fullsized":"2024-03-24T00:36:07.820717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:02:11.729651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.804466+0000","last_change":"2024-03-24T00:36:07.795173+0000","last_active":"2024-03-24T00:36:07.804466+0000","last_peered":"2024-03-24T00:36:07.804466+0000","last_clean":"2024-03-24T00:36:07.804466+0000","last_became_active":"2024-03-24T00:36:07.794862+0000","last_became_peered":"2024-03-24T00:36:07.794862+0000","last_unstale":"2024-03-24T00:36:07.804466+0000","last_undegraded":"2024-03-24T00:36:07.804466+0000","last_fullsized":"2024-03-24T00:36:07.804466+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:08.280925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.785051+0000","last_change":"2024-03-24T00:36:06.776923+0000","last_active":"2024-03-24T00:36:07.785051+0000","last_peered":"2024-03-24T00:36:07.785051+0000","last_clean":"2024-03-24T00:36:07.785051+0000","last_became_active":"2024-03-24T00:36:06.776619+0000","last_became_peered":"2024-03-24T00:36:06.776619+0000","last_unstale":"2024-03-24T00:36:07.785051+0000","last_undegraded":"2024-03-24T00:36:07.785051+0000","last_fullsized":"2024-03-24T00:36:07.785051+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:06.400284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786383+0000","last_change":"2024-03-24T00:36:07.784117+0000","last_active":"2024-03-24T00:36:07.786383+0000","last_peered":"2024-03-24T00:36:07.786383+0000","last_clean":"2024-03-24T00:36:07.786383+0000","last_became_active":"2024-03-24T00:36:07.783795+0000","last_became_peered":"2024-03-24T00:36:07.783795+0000","last_unstale":"2024-03-24T00:36:07.786383+0000","last_undegraded":"2024-03-24T00:36:07.786383+0000","last_fullsized":"2024-03-24T00:36:07.786383+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:01.647460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.775493+0000","last_change":"2024-03-24T00:36:06.770035+0000","last_active":"2024-03-24T00:36:07.775493+0000","last_peered":"2024-03-24T00:36:07.775493+0000","last_clean":"2024-03-24T00:36:07.775493+0000","last_became_active":"2024-03-24T00:36:06.766581+0000","last_became_peered":"2024-03-24T00:36:06.766581+0000","last_unstale":"2024-03-24T00:36:07.775493+0000","last_undegraded":"2024-03-24T00:36:07.775493+0000","last_fullsized":"2024-03-24T00:36:07.775493+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:51:11.132128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786045+0000","last_change":"2024-03-24T00:36:07.782705+0000","last_active":"2024-03-24T00:36:07.786045+0000","last_peered":"2024-03-24T00:36:07.786045+0000","last_clean":"2024-03-24T00:36:07.786045+0000","last_became_active":"2024-03-24T00:36:07.782334+0000","last_became_peered":"2024-03-24T00:36:07.782334+0000","last_unstale":"2024-03-24T00:36:07.786045+0000","last_undegraded":"2024-03-24T00:36:07.786045+0000","last_fullsized":"2024-03-24T00:36:07.786045+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:34:39.426391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.801581+0000","last_change":"2024-03-24T00:36:06.790394+0000","last_active":"2024-03-24T00:36:07.801581+0000","last_peered":"2024-03-24T00:36:07.801581+0000","last_clean":"2024-03-24T00:36:07.801581+0000","last_became_active":"2024-03-24T00:36:06.790201+0000","last_became_peered":"2024-03-24T00:36:06.790201+0000","last_unstale":"2024-03-24T00:36:07.801581+0000","last_undegraded":"2024-03-24T00:36:07.801581+0000","last_fullsized":"2024-03-24T00:36:07.801581+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:53.396891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.806180+0000","last_change":"2024-03-24T00:36:07.801340+0000","last_active":"2024-03-24T00:36:07.806180+0000","last_peered":"2024-03-24T00:36:07.806180+0000","last_clean":"2024-03-24T00:36:07.806180+0000","last_became_active":"2024-03-24T00:36:07.801053+0000","last_became_peered":"2024-03-24T00:36:07.801053+0000","last_unstale":"2024-03-24T00:36:07.806180+0000","last_undegraded":"2024-03-24T00:36:07.806180+0000","last_fullsized":"2024-03-24T00:36:07.806180+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:15.579736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.800220+0000","last_change":"2024-03-24T00:36:06.775340+0000","last_active":"2024-03-24T00:36:07.800220+0000","last_peered":"2024-03-24T00:36:07.800220+0000","last_clean":"2024-03-24T00:36:07.800220+0000","last_became_active":"2024-03-24T00:36:06.775080+0000","last_became_peered":"2024-03-24T00:36:06.775080+0000","last_unstale":"2024-03-24T00:36:07.800220+0000","last_undegraded":"2024-03-24T00:36:07.800220+0000","last_fullsized":"2024-03-24T00:36:07.800220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:14.552068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.814879+0000","last_change":"2024-03-24T00:36:07.811537+0000","last_active":"2024-03-24T00:36:07.814879+0000","last_peered":"2024-03-24T00:36:07.814879+0000","last_clean":"2024-03-24T00:36:07.814879+0000","last_became_active":"2024-03-24T00:36:07.811241+0000","last_became_peered":"2024-03-24T00:36:07.811241+0000","last_unstale":"2024-03-24T00:36:07.814879+0000","last_undegraded":"2024-03-24T00:36:07.814879+0000","last_fullsized":"2024-03-24T00:36:07.814879+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:35:15.406036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"82'3","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.823176+0000","last_change":"2024-03-24T00:36:06.781402+0000","last_active":"2024-03-24T00:36:07.823176+0000","last_peered":"2024-03-24T00:36:07.823176+0000","last_clean":"2024-03-24T00:36:07.823176+0000","last_became_active":"2024-03-24T00:36:06.781117+0000","last_became_peered":"2024-03-24T00:36:06.781117+0000","last_unstale":"2024-03-24T00:36:07.823176+0000","last_undegraded":"2024-03-24T00:36:07.823176+0000","last_fullsized":"2024-03-24T00:36:07.823176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:13.706926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.804676+0000","last_change":"2024-03-24T00:36:07.796829+0000","last_active":"2024-03-24T00:36:07.804676+0000","last_peered":"2024-03-24T00:36:07.804676+0000","last_clean":"2024-03-24T00:36:07.804676+0000","last_became_active":"2024-03-24T00:36:07.796152+0000","last_became_peered":"2024-03-24T00:36:07.796152+0000","last_unstale":"2024-03-24T00:36:07.804676+0000","last_undegraded":"2024-03-24T00:36:07.804676+0000","last_fullsized":"2024-03-24T00:36:07.804676+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:35:17.249084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.823425+0000","last_change":"2024-03-24T00:36:06.787486+0000","last_active":"2024-03-24T00:36:07.823425+0000","last_peered":"2024-03-24T00:36:07.823425+0000","last_clean":"2024-03-24T00:36:07.823425+0000","last_became_active":"2024-03-24T00:36:06.787119+0000","last_became_peered":"2024-03-24T00:36:06.787119+0000","last_unstale":"2024-03-24T00:36:07.823425+0000","last_undegraded":"2024-03-24T00:36:07.823425+0000","last_fullsized":"2024-03-24T00:36:07.823425+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:41.584491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.806883+0000","last_change":"2024-03-24T00:36:07.805898+0000","last_active":"2024-03-24T00:36:07.806883+0000","last_peered":"2024-03-24T00:36:07.806883+0000","last_clean":"2024-03-24T00:36:07.806883+0000","last_became_active":"2024-03-24T00:36:07.805647+0000","last_became_peered":"2024-03-24T00:36:07.805647+0000","last_unstale":"2024-03-24T00:36:07.806883+0000","last_undegraded":"2024-03-24T00:36:07.806883+0000","last_fullsized":"2024-03-24T00:36:07.806883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:04.419002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.820428+0000","last_change":"2024-03-24T00:36:06.770844+0000","last_active":"2024-03-24T00:36:07.820428+0000","last_peered":"2024-03-24T00:36:07.820428+0000","last_clean":"2024-03-24T00:36:07.820428+0000","last_became_active":"2024-03-24T00:36:06.770606+0000","last_became_peered":"2024-03-24T00:36:06.770606+0000","last_unstale":"2024-03-24T00:36:07.820428+0000","last_undegraded":"2024-03-24T00:36:07.820428+0000","last_fullsized":"2024-03-24T00:36:07.820428+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:08.347951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.804564+0000","last_change":"2024-03-24T00:36:07.801022+0000","last_active":"2024-03-24T00:36:07.804564+0000","last_peered":"2024-03-24T00:36:07.804564+0000","last_clean":"2024-03-24T00:36:07.804564+0000","last_became_active":"2024-03-24T00:36:07.800676+0000","last_became_peered":"2024-03-24T00:36:07.800676+0000","last_unstale":"2024-03-24T00:36:07.804564+0000","last_undegraded":"2024-03-24T00:36:07.804564+0000","last_fullsized":"2024-03-24T00:36:07.804564+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:37.605081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.780048+0000","last_change":"2024-03-24T00:36:06.787472+0000","last_active":"2024-03-24T00:36:07.780048+0000","last_peered":"2024-03-24T00:36:07.780048+0000","last_clean":"2024-03-24T00:36:07.780048+0000","last_became_active":"2024-03-24T00:36:06.787163+0000","last_became_peered":"2024-03-24T00:36:06.787163+0000","last_unstale":"2024-03-24T00:36:07.780048+0000","last_undegraded":"2024-03-24T00:36:07.780048+0000","last_fullsized":"2024-03-24T00:36:07.780048+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:35:14.398485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.799073+0000","last_change":"2024-03-24T00:36:07.795051+0000","last_active":"2024-03-24T00:36:07.799073+0000","last_peered":"2024-03-24T00:36:07.799073+0000","last_clean":"2024-03-24T00:36:07.799073+0000","last_became_active":"2024-03-24T00:36:07.794681+0000","last_became_peered":"2024-03-24T00:36:07.794681+0000","last_unstale":"2024-03-24T00:36:07.799073+0000","last_undegraded":"2024-03-24T00:36:07.799073+0000","last_fullsized":"2024-03-24T00:36:07.799073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:21.815307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.780203+0000","last_change":"2024-03-24T00:36:06.776589+0000","last_active":"2024-03-24T00:36:07.780203+0000","last_peered":"2024-03-24T00:36:07.780203+0000","last_clean":"2024-03-24T00:36:07.780203+0000","last_became_active":"2024-03-24T00:36:06.775939+0000","last_became_peered":"2024-03-24T00:36:06.775939+0000","last_unstale":"2024-03-24T00:36:07.780203+0000","last_undegraded":"2024-03-24T00:36:07.780203+0000","last_fullsized":"2024-03-24T00:36:07.780203+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:53.396891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.806204+0000","last_change":"2024-03-24T00:36:07.805320+0000","last_active":"2024-03-24T00:36:07.806204+0000","last_peered":"2024-03-24T00:36:07.806204+0000","last_clean":"2024-03-24T00:36:07.806204+0000","last_became_active":"2024-03-24T00:36:07.805042+0000","last_became_peered":"2024-03-24T00:36:07.805042+0000","last_unstale":"2024-03-24T00:36:07.806204+0000","last_undegraded":"2024-03-24T00:36:07.806204+0000","last_fullsized":"2024-03-24T00:36:07.806204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:02.365494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.822096+0000","last_change":"2024-03-24T00:36:06.769747+0000","last_active":"2024-03-24T00:36:07.822096+0000","last_peered":"2024-03-24T00:36:07.822096+0000","last_clean":"2024-03-24T00:36:07.822096+0000","last_became_active":"2024-03-24T00:36:06.769455+0000","last_became_peered":"2024-03-24T00:36:06.769455+0000","last_unstale":"2024-03-24T00:36:07.822096+0000","last_undegraded":"2024-03-24T00:36:07.822096+0000","last_fullsized":"2024-03-24T00:36:07.822096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:42.579400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.796608+0000","last_change":"2024-03-24T00:36:07.787309+0000","last_active":"2024-03-24T00:36:07.796608+0000","last_peered":"2024-03-24T00:36:07.796608+0000","last_clean":"2024-03-24T00:36:07.796608+0000","last_became_active":"2024-03-24T00:36:07.785330+0000","last_became_peered":"2024-03-24T00:36:07.785330+0000","last_unstale":"2024-03-24T00:36:07.796608+0000","last_undegraded":"2024-03-24T00:36:07.796608+0000","last_fullsized":"2024-03-24T00:36:07.796608+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:43.586951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"82'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.821008+0000","last_change":"2024-03-24T00:36:06.779681+0000","last_active":"2024-03-24T00:36:07.821008+0000","last_peered":"2024-03-24T00:36:07.821008+0000","last_clean":"2024-03-24T00:36:07.821008+0000","last_became_active":"2024-03-24T00:36:06.779412+0000","last_became_peered":"2024-03-24T00:36:06.779412+0000","last_unstale":"2024-03-24T00:36:07.821008+0000","last_undegraded":"2024-03-24T00:36:07.821008+0000","last_fullsized":"2024-03-24T00:36:07.821008+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:41.584491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.816656+0000","last_change":"2024-03-24T00:36:07.809378+0000","last_active":"2024-03-24T00:36:07.816656+0000","last_peered":"2024-03-24T00:36:07.816656+0000","last_clean":"2024-03-24T00:36:07.816656+0000","last_became_active":"2024-03-24T00:36:07.808599+0000","last_became_peered":"2024-03-24T00:36:07.808599+0000","last_unstale":"2024-03-24T00:36:07.816656+0000","last_undegraded":"2024-03-24T00:36:07.816656+0000","last_fullsized":"2024-03-24T00:36:07.816656+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:32:15.351688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.819745+0000","last_change":"2024-03-24T00:36:06.773088+0000","last_active":"2024-03-24T00:36:07.819745+0000","last_peered":"2024-03-24T00:36:07.819745+0000","last_clean":"2024-03-24T00:36:07.819745+0000","last_became_active":"2024-03-24T00:36:06.772820+0000","last_became_peered":"2024-03-24T00:36:06.772820+0000","last_unstale":"2024-03-24T00:36:07.819745+0000","last_undegraded":"2024-03-24T00:36:07.819745+0000","last_fullsized":"2024-03-24T00:36:07.819745+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:36:21.533134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.804652+0000","last_change":"2024-03-24T00:36:07.796802+0000","last_active":"2024-03-24T00:36:07.804652+0000","last_peered":"2024-03-24T00:36:07.804652+0000","last_clean":"2024-03-24T00:36:07.804652+0000","last_became_active":"2024-03-24T00:36:07.796050+0000","last_became_peered":"2024-03-24T00:36:07.796050+0000","last_unstale":"2024-03-24T00:36:07.804652+0000","last_undegraded":"2024-03-24T00:36:07.804652+0000","last_fullsized":"2024-03-24T00:36:07.804652+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:09:13.747978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.784830+0000","last_change":"2024-03-24T00:36:06.778230+0000","last_active":"2024-03-24T00:36:07.784830+0000","last_peered":"2024-03-24T00:36:07.784830+0000","last_clean":"2024-03-24T00:36:07.784830+0000","last_became_active":"2024-03-24T00:36:06.777922+0000","last_became_peered":"2024-03-24T00:36:06.777922+0000","last_unstale":"2024-03-24T00:36:07.784830+0000","last_undegraded":"2024-03-24T00:36:07.784830+0000","last_fullsized":"2024-03-24T00:36:07.784830+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:36.597530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.800972+0000","last_change":"2024-03-24T00:36:07.799062+0000","last_active":"2024-03-24T00:36:07.800972+0000","last_peered":"2024-03-24T00:36:07.800972+0000","last_clean":"2024-03-24T00:36:07.800972+0000","last_became_active":"2024-03-24T00:36:07.797704+0000","last_became_peered":"2024-03-24T00:36:07.797704+0000","last_unstale":"2024-03-24T00:36:07.800972+0000","last_undegraded":"2024-03-24T00:36:07.800972+0000","last_fullsized":"2024-03-24T00:36:07.800972+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:58.938672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.818768+0000","last_change":"2024-03-24T00:36:06.785932+0000","last_active":"2024-03-24T00:36:07.818768+0000","last_peered":"2024-03-24T00:36:07.818768+0000","last_clean":"2024-03-24T00:36:07.818768+0000","last_became_active":"2024-03-24T00:36:06.785320+0000","last_became_peered":"2024-03-24T00:36:06.785320+0000","last_unstale":"2024-03-24T00:36:07.818768+0000","last_undegraded":"2024-03-24T00:36:07.818768+0000","last_fullsized":"2024-03-24T00:36:07.818768+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:37:49.817212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786510+0000","last_change":"2024-03-24T00:36:07.784193+0000","last_active":"2024-03-24T00:36:07.786510+0000","last_peered":"2024-03-24T00:36:07.786510+0000","last_clean":"2024-03-24T00:36:07.786510+0000","last_became_active":"2024-03-24T00:36:07.783836+0000","last_became_peered":"2024-03-24T00:36:07.783836+0000","last_unstale":"2024-03-24T00:36:07.786510+0000","last_undegraded":"2024-03-24T00:36:07.786510+0000","last_fullsized":"2024-03-24T00:36:07.786510+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:03:30.373399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.785723+0000","last_change":"2024-03-24T00:36:06.784639+0000","last_active":"2024-03-24T00:36:07.785723+0000","last_peered":"2024-03-24T00:36:07.785723+0000","last_clean":"2024-03-24T00:36:07.785723+0000","last_became_active":"2024-03-24T00:36:06.784248+0000","last_became_peered":"2024-03-24T00:36:06.784248+0000","last_unstale":"2024-03-24T00:36:07.785723+0000","last_undegraded":"2024-03-24T00:36:07.785723+0000","last_fullsized":"2024-03-24T00:36:07.785723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:17.546360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.800203+0000","last_change":"2024-03-24T00:36:07.793988+0000","last_active":"2024-03-24T00:36:07.800203+0000","last_peered":"2024-03-24T00:36:07.800203+0000","last_clean":"2024-03-24T00:36:07.800203+0000","last_became_active":"2024-03-24T00:36:07.793088+0000","last_became_peered":"2024-03-24T00:36:07.793088+0000","last_unstale":"2024-03-24T00:36:07.800203+0000","last_undegraded":"2024-03-24T00:36:07.800203+0000","last_fullsized":"2024-03-24T00:36:07.800203+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:22.531204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.787096+0000","last_change":"2024-03-24T00:36:06.777058+0000","last_active":"2024-03-24T00:36:07.787096+0000","last_peered":"2024-03-24T00:36:07.787096+0000","last_clean":"2024-03-24T00:36:07.787096+0000","last_became_active":"2024-03-24T00:36:06.775965+0000","last_became_peered":"2024-03-24T00:36:06.775965+0000","last_unstale":"2024-03-24T00:36:07.787096+0000","last_undegraded":"2024-03-24T00:36:07.787096+0000","last_fullsized":"2024-03-24T00:36:07.787096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:58:20.516373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.800951+0000","last_change":"2024-03-24T00:36:07.795087+0000","last_active":"2024-03-24T00:36:07.800951+0000","last_peered":"2024-03-24T00:36:07.800951+0000","last_clean":"2024-03-24T00:36:07.800951+0000","last_became_active":"2024-03-24T00:36:07.794779+0000","last_became_peered":"2024-03-24T00:36:07.794779+0000","last_unstale":"2024-03-24T00:36:07.800951+0000","last_undegraded":"2024-03-24T00:36:07.800951+0000","last_fullsized":"2024-03-24T00:36:07.800951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:32:15.351688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.779693+0000","last_change":"2024-03-24T00:36:06.785925+0000","last_active":"2024-03-24T00:36:07.779693+0000","last_peered":"2024-03-24T00:36:07.779693+0000","last_clean":"2024-03-24T00:36:07.779693+0000","last_became_active":"2024-03-24T00:36:06.784803+0000","last_became_peered":"2024-03-24T00:36:06.784803+0000","last_unstale":"2024-03-24T00:36:07.779693+0000","last_undegraded":"2024-03-24T00:36:07.779693+0000","last_fullsized":"2024-03-24T00:36:07.779693+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:03.411451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.803798+0000","last_change":"2024-03-24T00:36:07.794130+0000","last_active":"2024-03-24T00:36:07.803798+0000","last_peered":"2024-03-24T00:36:07.803798+0000","last_clean":"2024-03-24T00:36:07.803798+0000","last_became_active":"2024-03-24T00:36:07.793038+0000","last_became_peered":"2024-03-24T00:36:07.793038+0000","last_unstale":"2024-03-24T00:36:07.803798+0000","last_undegraded":"2024-03-24T00:36:07.803798+0000","last_fullsized":"2024-03-24T00:36:07.803798+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:03:30.373399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.817346+0000","last_change":"2024-03-24T00:36:06.768410+0000","last_active":"2024-03-24T00:36:07.817346+0000","last_peered":"2024-03-24T00:36:07.817346+0000","last_clean":"2024-03-24T00:36:07.817346+0000","last_became_active":"2024-03-24T00:36:06.767675+0000","last_became_peered":"2024-03-24T00:36:06.767675+0000","last_unstale":"2024-03-24T00:36:07.817346+0000","last_undegraded":"2024-03-24T00:36:07.817346+0000","last_fullsized":"2024-03-24T00:36:07.817346+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:50:47.347515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.813913+0000","last_change":"2024-03-24T00:36:07.811569+0000","last_active":"2024-03-24T00:36:07.813913+0000","last_peered":"2024-03-24T00:36:07.813913+0000","last_clean":"2024-03-24T00:36:07.813913+0000","last_became_active":"2024-03-24T00:36:07.811224+0000","last_became_peered":"2024-03-24T00:36:07.811224+0000","last_unstale":"2024-03-24T00:36:07.813913+0000","last_undegraded":"2024-03-24T00:36:07.813913+0000","last_fullsized":"2024-03-24T00:36:07.813913+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:03:07.442469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.776578+0000","last_change":"2024-03-24T00:36:06.771523+0000","last_active":"2024-03-24T00:36:07.776578+0000","last_peered":"2024-03-24T00:36:07.776578+0000","last_clean":"2024-03-24T00:36:07.776578+0000","last_became_active":"2024-03-24T00:36:06.771247+0000","last_became_peered":"2024-03-24T00:36:06.771247+0000","last_unstale":"2024-03-24T00:36:07.776578+0000","last_undegraded":"2024-03-24T00:36:07.776578+0000","last_fullsized":"2024-03-24T00:36:07.776578+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:49:56.402081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.817146+0000","last_change":"2024-03-24T00:36:07.806055+0000","last_active":"2024-03-24T00:36:07.817146+0000","last_peered":"2024-03-24T00:36:07.817146+0000","last_clean":"2024-03-24T00:36:07.817146+0000","last_became_active":"2024-03-24T00:36:07.805638+0000","last_became_peered":"2024-03-24T00:36:07.805638+0000","last_unstale":"2024-03-24T00:36:07.817146+0000","last_undegraded":"2024-03-24T00:36:07.817146+0000","last_fullsized":"2024-03-24T00:36:07.817146+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:03:30.373399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.799749+0000","last_change":"2024-03-24T00:36:06.779343+0000","last_active":"2024-03-24T00:36:07.799749+0000","last_peered":"2024-03-24T00:36:07.799749+0000","last_clean":"2024-03-24T00:36:07.799749+0000","last_became_active":"2024-03-24T00:36:06.778003+0000","last_became_peered":"2024-03-24T00:36:06.778003+0000","last_unstale":"2024-03-24T00:36:07.799749+0000","last_undegraded":"2024-03-24T00:36:07.799749+0000","last_fullsized":"2024-03-24T00:36:07.799749+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:53.545725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.797068+0000","last_change":"2024-03-24T00:36:07.791525+0000","last_active":"2024-03-24T00:36:07.797068+0000","last_peered":"2024-03-24T00:36:07.797068+0000","last_clean":"2024-03-24T00:36:07.797068+0000","last_became_active":"2024-03-24T00:36:07.790720+0000","last_became_peered":"2024-03-24T00:36:07.790720+0000","last_unstale":"2024-03-24T00:36:07.797068+0000","last_undegraded":"2024-03-24T00:36:07.797068+0000","last_fullsized":"2024-03-24T00:36:07.797068+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:15.559619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.799272+0000","last_change":"2024-03-24T00:36:06.787365+0000","last_active":"2024-03-24T00:36:07.799272+0000","last_peered":"2024-03-24T00:36:07.799272+0000","last_clean":"2024-03-24T00:36:07.799272+0000","last_became_active":"2024-03-24T00:36:06.787075+0000","last_became_peered":"2024-03-24T00:36:06.787075+0000","last_unstale":"2024-03-24T00:36:07.799272+0000","last_undegraded":"2024-03-24T00:36:07.799272+0000","last_fullsized":"2024-03-24T00:36:07.799272+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:25.384461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786537+0000","last_change":"2024-03-24T00:36:07.784223+0000","last_active":"2024-03-24T00:36:07.786537+0000","last_peered":"2024-03-24T00:36:07.786537+0000","last_clean":"2024-03-24T00:36:07.786537+0000","last_became_active":"2024-03-24T00:36:07.783854+0000","last_became_peered":"2024-03-24T00:36:07.783854+0000","last_unstale":"2024-03-24T00:36:07.786537+0000","last_undegraded":"2024-03-24T00:36:07.786537+0000","last_fullsized":"2024-03-24T00:36:07.786537+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:09:13.747978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786842+0000","last_change":"2024-03-24T00:36:06.768563+0000","last_active":"2024-03-24T00:36:07.786842+0000","last_peered":"2024-03-24T00:36:07.786842+0000","last_clean":"2024-03-24T00:36:07.786842+0000","last_became_active":"2024-03-24T00:36:06.768279+0000","last_became_peered":"2024-03-24T00:36:06.768279+0000","last_unstale":"2024-03-24T00:36:07.786842+0000","last_undegraded":"2024-03-24T00:36:07.786842+0000","last_fullsized":"2024-03-24T00:36:07.786842+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:43.770350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.800465+0000","last_change":"2024-03-24T00:36:07.797402+0000","last_active":"2024-03-24T00:36:07.800465+0000","last_peered":"2024-03-24T00:36:07.800465+0000","last_clean":"2024-03-24T00:36:07.800465+0000","last_became_active":"2024-03-24T00:36:07.797045+0000","last_became_peered":"2024-03-24T00:36:07.797045+0000","last_unstale":"2024-03-24T00:36:07.800465+0000","last_undegraded":"2024-03-24T00:36:07.800465+0000","last_fullsized":"2024-03-24T00:36:07.800465+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:03.022418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.792373+0000","last_change":"2024-03-24T00:36:06.788740+0000","last_active":"2024-03-24T00:36:07.792373+0000","last_peered":"2024-03-24T00:36:07.792373+0000","last_clean":"2024-03-24T00:36:07.792373+0000","last_became_active":"2024-03-24T00:36:06.788406+0000","last_became_peered":"2024-03-24T00:36:06.788406+0000","last_unstale":"2024-03-24T00:36:07.792373+0000","last_undegraded":"2024-03-24T00:36:07.792373+0000","last_fullsized":"2024-03-24T00:36:07.792373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:53.396891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.789427+0000","last_change":"2024-03-24T00:36:07.784571+0000","last_active":"2024-03-24T00:36:07.789427+0000","last_peered":"2024-03-24T00:36:07.789427+0000","last_clean":"2024-03-24T00:36:07.789427+0000","last_became_active":"2024-03-24T00:36:07.784254+0000","last_became_peered":"2024-03-24T00:36:07.784254+0000","last_unstale":"2024-03-24T00:36:07.789427+0000","last_undegraded":"2024-03-24T00:36:07.789427+0000","last_fullsized":"2024-03-24T00:36:07.789427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:13.299874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.799932+0000","last_change":"2024-03-24T00:36:06.786048+0000","last_active":"2024-03-24T00:36:07.799932+0000","last_peered":"2024-03-24T00:36:07.799932+0000","last_clean":"2024-03-24T00:36:07.799932+0000","last_became_active":"2024-03-24T00:36:06.785829+0000","last_became_peered":"2024-03-24T00:36:06.785829+0000","last_unstale":"2024-03-24T00:36:07.799932+0000","last_undegraded":"2024-03-24T00:36:07.799932+0000","last_fullsized":"2024-03-24T00:36:07.799932+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:04:40.750748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.810175+0000","last_change":"2024-03-24T00:36:07.800801+0000","last_active":"2024-03-24T00:36:07.810175+0000","last_peered":"2024-03-24T00:36:07.810175+0000","last_clean":"2024-03-24T00:36:07.810175+0000","last_became_active":"2024-03-24T00:36:07.800481+0000","last_became_peered":"2024-03-24T00:36:07.800481+0000","last_unstale":"2024-03-24T00:36:07.810175+0000","last_undegraded":"2024-03-24T00:36:07.810175+0000","last_fullsized":"2024-03-24T00:36:07.810175+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:51:17.573588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.791834+0000","last_change":"2024-03-24T00:36:06.773919+0000","last_active":"2024-03-24T00:36:07.791834+0000","last_peered":"2024-03-24T00:36:07.791834+0000","last_clean":"2024-03-24T00:36:07.791834+0000","last_became_active":"2024-03-24T00:36:06.773572+0000","last_became_peered":"2024-03-24T00:36:06.773572+0000","last_unstale":"2024-03-24T00:36:07.791834+0000","last_undegraded":"2024-03-24T00:36:07.791834+0000","last_fullsized":"2024-03-24T00:36:07.791834+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:51:16.566037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.794164+0000","last_change":"2024-03-24T00:36:07.787225+0000","last_active":"2024-03-24T00:36:07.794164+0000","last_peered":"2024-03-24T00:36:07.794164+0000","last_clean":"2024-03-24T00:36:07.794164+0000","last_became_active":"2024-03-24T00:36:07.786635+0000","last_became_peered":"2024-03-24T00:36:07.786635+0000","last_unstale":"2024-03-24T00:36:07.794164+0000","last_undegraded":"2024-03-24T00:36:07.794164+0000","last_fullsized":"2024-03-24T00:36:07.794164+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:29:31.444886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"82'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.819182+0000","last_change":"2024-03-24T00:36:06.776469+0000","last_active":"2024-03-24T00:36:07.819182+0000","last_peered":"2024-03-24T00:36:07.819182+0000","last_clean":"2024-03-24T00:36:07.819182+0000","last_became_active":"2024-03-24T00:36:06.776177+0000","last_became_peered":"2024-03-24T00:36:06.776177+0000","last_unstale":"2024-03-24T00:36:07.819182+0000","last_undegraded":"2024-03-24T00:36:07.819182+0000","last_fullsized":"2024-03-24T00:36:07.819182+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:53.328381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":295,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.779328+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:36:07.779328+0000","last_peered":"2024-03-24T00:36:07.779328+0000","last_clean":"2024-03-24T00:36:07.779328+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:36:07.779328+0000","last_undegraded":"2024-03-24T00:36:07.779328+0000","last_fullsized":"2024-03-24T00:36:07.779328+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36908,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334932,"statfs":{"total":96636764160,"available":96598970368,"internally_reserved":0,"allocated":1605632,"data_stored":1024706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":434962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1024000,"data_stored":435902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40488,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331352,"statfs":{"total":96636764160,"available":96595304448,"internally_reserved":0,"allocated":1007616,"data_stored":434268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36312,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335528,"statfs":{"total":96636764160,"available":96599580672,"internally_reserved":0,"allocated":995328,"data_stored":427493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36312,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335528,"statfs":{"total":96636764160,"available":96599580672,"internally_reserved":0,"allocated":995328,"data_stored":427493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335500,"statfs":{"total":96636764160,"available":96599552000,"internally_reserved":0,"allocated":1019904,"data_stored":434462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36892,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334948,"statfs":{"total":96636764160,"available":96598986752,"internally_reserved":0,"allocated":1589248,"data_stored":1017861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:36:11.912 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.086+0000 7f1366800640 1 Processor -- start 2024-03-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.090+0000 7f1366800640 1 -- start start 2024-03-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.090+0000 7f1366800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f136006e620 0x7f13600b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.090+0000 7f1366800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1360070f60 con 0x7f136006ea00 2024-03-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.090+0000 7f1366800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13600b3620 con 0x7f13600ccfa0 2024-03-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.090+0000 7f1366800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13600b4c00 con 0x7f136006e620 2024-03-24T00:36:12.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.090+0000 7f1357e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f136006ea00 0x7f13600b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42980/0 (socket says 172.21.15.40:42980) 2024-03-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.090+0000 7f1357e00640 1 -- 172.21.15.40:0/2662827630 learned_addr learned my addr 172.21.15.40:0/2662827630 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.090+0000 7f135f400640 1 --2- 172.21.15.40:0/2662827630 >> v2:172.21.15.40:3300/0 conn(0x7f136006e620 0x7f13600b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.090+0000 7f135f400640 1 -- 172.21.15.40:0/2662827630 >> v1:172.21.15.53:6789/0 conn(0x7f13600ccfa0 legacy=0x7f13600cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.090+0000 7f135f400640 1 -- 172.21.15.40:0/2662827630 >> v1:172.21.15.40:6789/0 conn(0x7f136006ea00 legacy=0x7f13600b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.090+0000 7f135f400640 1 -- 172.21.15.40:0/2662827630 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13600b3900 con 0x7f136006e620 2024-03-24T00:36:12.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f135f400640 1 --2- 172.21.15.40:0/2662827630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136006e620 0x7f13600b46c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f13500097c0 tx=0x7f135001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbf9d345ad81024 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f135ea00640 1 -- 172.21.15.40:0/2662827630 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13500b1070 con 0x7f136006e620 2024-03-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f135ea00640 1 -- 172.21.15.40:0/2662827630 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13500a8050 con 0x7f136006e620 2024-03-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f135ea00640 1 -- 172.21.15.40:0/2662827630 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13500ac860 con 0x7f136006e620 2024-03-24T00:36:12.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f1366800640 1 -- 172.21.15.40:0/2662827630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136006e620 msgr2=0x7f13600b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f1366800640 1 --2- 172.21.15.40:0/2662827630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136006e620 0x7f13600b46c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f13500097c0 tx=0x7f135001c3e0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f1366800640 1 -- 172.21.15.40:0/2662827630 shutdown_connections 2024-03-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f1366800640 1 --2- 172.21.15.40:0/2662827630 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136006e620 0x7f13600b46c0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f1366800640 1 -- 172.21.15.40:0/2662827630 >> 172.21.15.40:0/2662827630 conn(0x7f13600b17b0 msgr2=0x7f13600b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f1366800640 1 -- 172.21.15.40:0/2662827630 shutdown_connections 2024-03-24T00:36:12.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f1366800640 1 -- 172.21.15.40:0/2662827630 wait complete. 2024-03-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f1366800640 1 Processor -- start 2024-03-24T00:36:12.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.094+0000 7f1366800640 1 -- start start 2024-03-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1366800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f136006e620 0x7f136015a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1366800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136006ea00 0x7f136015d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1366800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13600ccfa0 0x7f136013b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1366800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1360144920 con 0x7f136006ea00 2024-03-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1366800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136006ee10 con 0x7f13600ccfa0 2024-03-24T00:36:12.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1366800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136015c580 con 0x7f136006e620 2024-03-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1357e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136006ea00 0x7f136015d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f135f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f136006e620 0x7f136015a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f135fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13600ccfa0 0x7f136013b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1357e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136006ea00 0x7f136015d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39792/0 (socket says 172.21.15.40:39792) 2024-03-24T00:36:12.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1357e00640 1 -- 172.21.15.40:0/3029075589 learned_addr learned my addr 172.21.15.40:0/3029075589 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f135f400640 1 -- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13600ccfa0 msgr2=0x7f136013b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f135f400640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13600ccfa0 0x7f136013b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f135f400640 1 -- 172.21.15.40:0/3029075589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136006ea00 msgr2=0x7f136015d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f135f400640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136006ea00 0x7f136015d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f135f400640 1 -- 172.21.15.40:0/3029075589 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f136015c860 con 0x7f136006e620 2024-03-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1357e00640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136006ea00 0x7f136015d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f135f400640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f136006e620 0x7f136015a0d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f13500ac000 tx=0x7f1350019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1357400640 1 -- 172.21.15.40:0/3029075589 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13500b1040 con 0x7f136006e620 2024-03-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1357400640 1 -- 172.21.15.40:0/3029075589 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13500bc070 con 0x7f136006e620 2024-03-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1366800640 1 -- 172.21.15.40:0/3029075589 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136015cb40 con 0x7f136006e620 2024-03-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1357400640 1 -- 172.21.15.40:0/3029075589 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13500b5650 con 0x7f136006e620 2024-03-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1366800640 1 -- 172.21.15.40:0/3029075589 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136015a610 con 0x7f136006e620 2024-03-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1366800640 1 -- 172.21.15.40:0/3029075589 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136015cf30 con 0x7f136006e620 2024-03-24T00:36:12.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.098+0000 7f1357400640 1 -- 172.21.15.40:0/3029075589 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f13500b57f0 con 0x7f136006e620 2024-03-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.102+0000 7f1357400640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f133c067660 0x7f133c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.102+0000 7f1357400640 1 -- 172.21.15.40:0/3029075589 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f135011f5e0 con 0x7f136006e620 2024-03-24T00:36:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.102+0000 7f1357e00640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f133c067660 0x7f133c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.102+0000 7f1357e00640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f133c067660 0x7f133c069b20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f13600aeca0 tx=0x7f1348009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:12.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.106+0000 7f1357400640 1 -- 172.21.15.40:0/3029075589 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f13500c6020 con 0x7f136006e620 2024-03-24T00:36:12.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.262+0000 7f1366800640 1 -- 172.21.15.40:0/3029075589 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1360068170 con 0x7f133c067660 2024-03-24T00:36:12.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.270+0000 7f1357400640 1 -- 172.21.15.40:0/3029075589 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252007 (secure 0 0 0) 0x7f1360068170 con 0x7f133c067660 2024-03-24T00:36:12.275 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:12.277 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:36:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.274+0000 7f1366800640 1 -- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f133c067660 msgr2=0x7f133c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.274+0000 7f1366800640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f133c067660 0x7f133c069b20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f13600aeca0 tx=0x7f1348009040 comp rx=0 tx=0).stop 2024-03-24T00:36:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.274+0000 7f1366800640 1 -- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f136006e620 msgr2=0x7f136015a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.274+0000 7f1366800640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f136006e620 0x7f136015a0d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f13500ac000 tx=0x7f1350019180 comp rx=0 tx=0).stop 2024-03-24T00:36:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.274+0000 7f1366800640 1 -- 172.21.15.40:0/3029075589 shutdown_connections 2024-03-24T00:36:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.274+0000 7f1366800640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f133c067660 0x7f133c069b20 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.274+0000 7f1366800640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13600ccfa0 0x7f136013b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.274+0000 7f1366800640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f136006ea00 0x7f136015d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.274+0000 7f1366800640 1 --2- 172.21.15.40:0/3029075589 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f136006e620 0x7f136015a0d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.274+0000 7f1366800640 1 -- 172.21.15.40:0/3029075589 >> 172.21.15.40:0/3029075589 conn(0x7f13600b17b0 msgr2=0x7f13600aef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:12.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.274+0000 7f1366800640 1 -- 172.21.15.40:0/3029075589 shutdown_connections 2024-03-24T00:36:12.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.274+0000 7f1366800640 1 -- 172.21.15.40:0/3029075589 wait complete. 2024-03-24T00:36:12.296 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2024-03-24T00:36:11.260349+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":29,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":287608,"kb_used_data":9040,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":754687112,"statfs":{"total":773094113280,"available":772799602688,"internally_reserved":0,"allocated":9256960,"data_stored":4637147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":285134386},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.507175"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.816677+0000","last_change":"2024-03-24T00:36:07.809594+0000","last_active":"2024-03-24T00:36:07.816677+0000","last_peered":"2024-03-24T00:36:07.816677+0000","last_clean":"2024-03-24T00:36:07.816677+0000","last_became_active":"2024-03-24T00:36:07.809057+0000","last_became_peered":"2024-03-24T00:36:07.809057+0000","last_unstale":"2024-03-24T00:36:07.816677+0000","last_undegraded":"2024-03-24T00:36:07.816677+0000","last_fullsized":"2024-03-24T00:36:07.816677+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:07.699038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.807251+0000","last_change":"2024-03-24T00:36:07.806701+0000","last_active":"2024-03-24T00:36:07.807251+0000","last_peered":"2024-03-24T00:36:07.807251+0000","last_clean":"2024-03-24T00:36:07.807251+0000","last_became_active":"2024-03-24T00:36:07.806310+0000","last_became_peered":"2024-03-24T00:36:07.806310+0000","last_unstale":"2024-03-24T00:36:07.807251+0000","last_undegraded":"2024-03-24T00:36:07.807251+0000","last_fullsized":"2024-03-24T00:36:07.807251+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:26.392012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.810302+0000","last_change":"2024-03-24T00:36:07.806772+0000","last_active":"2024-03-24T00:36:07.810302+0000","last_peered":"2024-03-24T00:36:07.810302+0000","last_clean":"2024-03-24T00:36:07.810302+0000","last_became_active":"2024-03-24T00:36:07.805929+0000","last_became_peered":"2024-03-24T00:36:07.805929+0000","last_unstale":"2024-03-24T00:36:07.810302+0000","last_undegraded":"2024-03-24T00:36:07.810302+0000","last_fullsized":"2024-03-24T00:36:07.810302+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:44.777901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.802651+0000","last_change":"2024-03-24T00:36:07.799542+0000","last_active":"2024-03-24T00:36:07.802651+0000","last_peered":"2024-03-24T00:36:07.802651+0000","last_clean":"2024-03-24T00:36:07.802651+0000","last_became_active":"2024-03-24T00:36:07.799254+0000","last_became_peered":"2024-03-24T00:36:07.799254+0000","last_unstale":"2024-03-24T00:36:07.802651+0000","last_undegraded":"2024-03-24T00:36:07.802651+0000","last_fullsized":"2024-03-24T00:36:07.802651+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:07:22.117218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.808247+0000","last_change":"2024-03-24T00:36:07.800263+0000","last_active":"2024-03-24T00:36:07.808247+0000","last_peered":"2024-03-24T00:36:07.808247+0000","last_clean":"2024-03-24T00:36:07.808247+0000","last_became_active":"2024-03-24T00:36:07.796248+0000","last_became_peered":"2024-03-24T00:36:07.796248+0000","last_unstale":"2024-03-24T00:36:07.808247+0000","last_undegraded":"2024-03-24T00:36:07.808247+0000","last_fullsized":"2024-03-24T00:36:07.808247+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:10:47.539461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.802160+0000","last_change":"2024-03-24T00:36:07.800419+0000","last_active":"2024-03-24T00:36:07.802160+0000","last_peered":"2024-03-24T00:36:07.802160+0000","last_clean":"2024-03-24T00:36:07.802160+0000","last_became_active":"2024-03-24T00:36:07.800132+0000","last_became_peered":"2024-03-24T00:36:07.800132+0000","last_unstale":"2024-03-24T00:36:07.802160+0000","last_undegraded":"2024-03-24T00:36:07.802160+0000","last_fullsized":"2024-03-24T00:36:07.802160+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:07.699038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.804144+0000","last_change":"2024-03-24T00:36:07.802548+0000","last_active":"2024-03-24T00:36:07.804144+0000","last_peered":"2024-03-24T00:36:07.804144+0000","last_clean":"2024-03-24T00:36:07.804144+0000","last_became_active":"2024-03-24T00:36:07.802270+0000","last_became_peered":"2024-03-24T00:36:07.802270+0000","last_unstale":"2024-03-24T00:36:07.804144+0000","last_undegraded":"2024-03-24T00:36:07.804144+0000","last_fullsized":"2024-03-24T00:36:07.804144+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:33:37.729814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786870+0000","last_change":"2024-03-24T00:36:07.782456+0000","last_active":"2024-03-24T00:36:07.786870+0000","last_peered":"2024-03-24T00:36:07.786870+0000","last_clean":"2024-03-24T00:36:07.786870+0000","last_became_active":"2024-03-24T00:36:07.782156+0000","last_became_peered":"2024-03-24T00:36:07.782156+0000","last_unstale":"2024-03-24T00:36:07.786870+0000","last_undegraded":"2024-03-24T00:36:07.786870+0000","last_fullsized":"2024-03-24T00:36:07.786870+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:00:14.167105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.796509+0000","last_change":"2024-03-24T00:36:07.791775+0000","last_active":"2024-03-24T00:36:07.796509+0000","last_peered":"2024-03-24T00:36:07.796509+0000","last_clean":"2024-03-24T00:36:07.796509+0000","last_became_active":"2024-03-24T00:36:07.791437+0000","last_became_peered":"2024-03-24T00:36:07.791437+0000","last_unstale":"2024-03-24T00:36:07.796509+0000","last_undegraded":"2024-03-24T00:36:07.796509+0000","last_fullsized":"2024-03-24T00:36:07.796509+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:35:17.249084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.809107+0000","last_change":"2024-03-24T00:36:07.797117+0000","last_active":"2024-03-24T00:36:07.809107+0000","last_peered":"2024-03-24T00:36:07.809107+0000","last_clean":"2024-03-24T00:36:07.809107+0000","last_became_active":"2024-03-24T00:36:07.793748+0000","last_became_peered":"2024-03-24T00:36:07.793748+0000","last_unstale":"2024-03-24T00:36:07.809107+0000","last_undegraded":"2024-03-24T00:36:07.809107+0000","last_fullsized":"2024-03-24T00:36:07.809107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:43.247886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.793599+0000","last_change":"2024-03-24T00:36:07.782891+0000","last_active":"2024-03-24T00:36:07.793599+0000","last_peered":"2024-03-24T00:36:07.793599+0000","last_clean":"2024-03-24T00:36:07.793599+0000","last_became_active":"2024-03-24T00:36:07.782573+0000","last_became_peered":"2024-03-24T00:36:07.782573+0000","last_unstale":"2024-03-24T00:36:07.793599+0000","last_undegraded":"2024-03-24T00:36:07.793599+0000","last_fullsized":"2024-03-24T00:36:07.793599+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:41:34.955684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.817169+0000","last_change":"2024-03-24T00:36:07.806975+0000","last_active":"2024-03-24T00:36:07.817169+0000","last_peered":"2024-03-24T00:36:07.817169+0000","last_clean":"2024-03-24T00:36:07.817169+0000","last_became_active":"2024-03-24T00:36:07.806633+0000","last_became_peered":"2024-03-24T00:36:07.806633+0000","last_unstale":"2024-03-24T00:36:07.817169+0000","last_undegraded":"2024-03-24T00:36:07.817169+0000","last_fullsized":"2024-03-24T00:36:07.817169+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:09:13.747978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.803061+0000","last_change":"2024-03-24T00:36:07.793547+0000","last_active":"2024-03-24T00:36:07.803061+0000","last_peered":"2024-03-24T00:36:07.803061+0000","last_clean":"2024-03-24T00:36:07.803061+0000","last_became_active":"2024-03-24T00:36:07.793254+0000","last_became_peered":"2024-03-24T00:36:07.793254+0000","last_unstale":"2024-03-24T00:36:07.803061+0000","last_undegraded":"2024-03-24T00:36:07.803061+0000","last_fullsized":"2024-03-24T00:36:07.803061+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:27.573566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.802799+0000","last_change":"2024-03-24T00:36:07.800373+0000","last_active":"2024-03-24T00:36:07.802799+0000","last_peered":"2024-03-24T00:36:07.802799+0000","last_clean":"2024-03-24T00:36:07.802799+0000","last_became_active":"2024-03-24T00:36:07.796861+0000","last_became_peered":"2024-03-24T00:36:07.796861+0000","last_unstale":"2024-03-24T00:36:07.802799+0000","last_undegraded":"2024-03-24T00:36:07.802799+0000","last_fullsized":"2024-03-24T00:36:07.802799+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:20:07.596033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786362+0000","last_change":"2024-03-24T00:36:07.784028+0000","last_active":"2024-03-24T00:36:07.786362+0000","last_peered":"2024-03-24T00:36:07.786362+0000","last_clean":"2024-03-24T00:36:07.786362+0000","last_became_active":"2024-03-24T00:36:07.783398+0000","last_became_peered":"2024-03-24T00:36:07.783398+0000","last_unstale":"2024-03-24T00:36:07.786362+0000","last_undegraded":"2024-03-24T00:36:07.786362+0000","last_fullsized":"2024-03-24T00:36:07.786362+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:51:24.214744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.815908+0000","last_change":"2024-03-24T00:36:07.813152+0000","last_active":"2024-03-24T00:36:07.815908+0000","last_peered":"2024-03-24T00:36:07.815908+0000","last_clean":"2024-03-24T00:36:07.815908+0000","last_became_active":"2024-03-24T00:36:07.812863+0000","last_became_peered":"2024-03-24T00:36:07.812863+0000","last_unstale":"2024-03-24T00:36:07.815908+0000","last_undegraded":"2024-03-24T00:36:07.815908+0000","last_fullsized":"2024-03-24T00:36:07.815908+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:25:52.928229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.809522+0000","last_change":"2024-03-24T00:36:07.799799+0000","last_active":"2024-03-24T00:36:07.809522+0000","last_peered":"2024-03-24T00:36:07.809522+0000","last_clean":"2024-03-24T00:36:07.809522+0000","last_became_active":"2024-03-24T00:36:07.799529+0000","last_became_peered":"2024-03-24T00:36:07.799529+0000","last_unstale":"2024-03-24T00:36:07.809522+0000","last_undegraded":"2024-03-24T00:36:07.809522+0000","last_fullsized":"2024-03-24T00:36:07.809522+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:15.579736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.810663+0000","last_change":"2024-03-24T00:36:07.809376+0000","last_active":"2024-03-24T00:36:07.810663+0000","last_peered":"2024-03-24T00:36:07.810663+0000","last_clean":"2024-03-24T00:36:07.810663+0000","last_became_active":"2024-03-24T00:36:07.808991+0000","last_became_peered":"2024-03-24T00:36:07.808991+0000","last_unstale":"2024-03-24T00:36:07.810663+0000","last_undegraded":"2024-03-24T00:36:07.810663+0000","last_fullsized":"2024-03-24T00:36:07.810663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:45.595754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.791111+0000","last_change":"2024-03-24T00:36:07.789129+0000","last_active":"2024-03-24T00:36:07.791111+0000","last_peered":"2024-03-24T00:36:07.791111+0000","last_clean":"2024-03-24T00:36:07.791111+0000","last_became_active":"2024-03-24T00:36:07.788771+0000","last_became_peered":"2024-03-24T00:36:07.788771+0000","last_unstale":"2024-03-24T00:36:07.791111+0000","last_undegraded":"2024-03-24T00:36:07.791111+0000","last_fullsized":"2024-03-24T00:36:07.791111+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:03:07.442469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.799436+0000","last_change":"2024-03-24T00:36:07.794991+0000","last_active":"2024-03-24T00:36:07.799436+0000","last_peered":"2024-03-24T00:36:07.799436+0000","last_clean":"2024-03-24T00:36:07.799436+0000","last_became_active":"2024-03-24T00:36:07.794723+0000","last_became_peered":"2024-03-24T00:36:07.794723+0000","last_unstale":"2024-03-24T00:36:07.799436+0000","last_undegraded":"2024-03-24T00:36:07.799436+0000","last_fullsized":"2024-03-24T00:36:07.799436+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:49.895037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.803567+0000","last_change":"2024-03-24T00:36:07.802043+0000","last_active":"2024-03-24T00:36:07.803567+0000","last_peered":"2024-03-24T00:36:07.803567+0000","last_clean":"2024-03-24T00:36:07.803567+0000","last_became_active":"2024-03-24T00:36:07.801739+0000","last_became_peered":"2024-03-24T00:36:07.801739+0000","last_unstale":"2024-03-24T00:36:07.803567+0000","last_undegraded":"2024-03-24T00:36:07.803567+0000","last_fullsized":"2024-03-24T00:36:07.803567+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:29.771243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.798748+0000","last_change":"2024-03-24T00:36:07.795150+0000","last_active":"2024-03-24T00:36:07.798748+0000","last_peered":"2024-03-24T00:36:07.798748+0000","last_clean":"2024-03-24T00:36:07.798748+0000","last_became_active":"2024-03-24T00:36:07.794859+0000","last_became_peered":"2024-03-24T00:36:07.794859+0000","last_unstale":"2024-03-24T00:36:07.798748+0000","last_undegraded":"2024-03-24T00:36:07.798748+0000","last_fullsized":"2024-03-24T00:36:07.798748+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:26:46.585200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.810197+0000","last_change":"2024-03-24T00:36:07.803372+0000","last_active":"2024-03-24T00:36:07.810197+0000","last_peered":"2024-03-24T00:36:07.810197+0000","last_clean":"2024-03-24T00:36:07.810197+0000","last_became_active":"2024-03-24T00:36:07.803025+0000","last_became_peered":"2024-03-24T00:36:07.803025+0000","last_unstale":"2024-03-24T00:36:07.810197+0000","last_undegraded":"2024-03-24T00:36:07.810197+0000","last_fullsized":"2024-03-24T00:36:07.810197+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:00:14.167105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.800225+0000","last_change":"2024-03-24T00:36:07.797993+0000","last_active":"2024-03-24T00:36:07.800225+0000","last_peered":"2024-03-24T00:36:07.800225+0000","last_clean":"2024-03-24T00:36:07.800225+0000","last_became_active":"2024-03-24T00:36:07.794434+0000","last_became_peered":"2024-03-24T00:36:07.794434+0000","last_unstale":"2024-03-24T00:36:07.800225+0000","last_undegraded":"2024-03-24T00:36:07.800225+0000","last_fullsized":"2024-03-24T00:36:07.800225+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:44.877088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.812895+0000","last_change":"2024-03-24T00:36:07.810464+0000","last_active":"2024-03-24T00:36:07.812895+0000","last_peered":"2024-03-24T00:36:07.812895+0000","last_clean":"2024-03-24T00:36:07.812895+0000","last_became_active":"2024-03-24T00:36:07.809970+0000","last_became_peered":"2024-03-24T00:36:07.809970+0000","last_unstale":"2024-03-24T00:36:07.812895+0000","last_undegraded":"2024-03-24T00:36:07.812895+0000","last_fullsized":"2024-03-24T00:36:07.812895+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:14.714477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.801693+0000","last_change":"2024-03-24T00:36:07.789583+0000","last_active":"2024-03-24T00:36:07.801693+0000","last_peered":"2024-03-24T00:36:07.801693+0000","last_clean":"2024-03-24T00:36:07.801693+0000","last_became_active":"2024-03-24T00:36:07.784901+0000","last_became_peered":"2024-03-24T00:36:07.784901+0000","last_unstale":"2024-03-24T00:36:07.801693+0000","last_undegraded":"2024-03-24T00:36:07.801693+0000","last_fullsized":"2024-03-24T00:36:07.801693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:36:22.540685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.804759+0000","last_change":"2024-03-24T00:36:07.797595+0000","last_active":"2024-03-24T00:36:07.804759+0000","last_peered":"2024-03-24T00:36:07.804759+0000","last_clean":"2024-03-24T00:36:07.804759+0000","last_became_active":"2024-03-24T00:36:07.797216+0000","last_became_peered":"2024-03-24T00:36:07.797216+0000","last_unstale":"2024-03-24T00:36:07.804759+0000","last_undegraded":"2024-03-24T00:36:07.804759+0000","last_fullsized":"2024-03-24T00:36:07.804759+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:32:15.351688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.817192+0000","last_change":"2024-03-24T00:36:07.810139+0000","last_active":"2024-03-24T00:36:07.817192+0000","last_peered":"2024-03-24T00:36:07.817192+0000","last_clean":"2024-03-24T00:36:07.817192+0000","last_became_active":"2024-03-24T00:36:07.806399+0000","last_became_peered":"2024-03-24T00:36:07.806399+0000","last_unstale":"2024-03-24T00:36:07.817192+0000","last_undegraded":"2024-03-24T00:36:07.817192+0000","last_fullsized":"2024-03-24T00:36:07.817192+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:15.579736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.794542+0000","last_change":"2024-03-24T00:36:07.784265+0000","last_active":"2024-03-24T00:36:07.794542+0000","last_peered":"2024-03-24T00:36:07.794542+0000","last_clean":"2024-03-24T00:36:07.794542+0000","last_became_active":"2024-03-24T00:36:07.783997+0000","last_became_peered":"2024-03-24T00:36:07.783997+0000","last_unstale":"2024-03-24T00:36:07.794542+0000","last_undegraded":"2024-03-24T00:36:07.794542+0000","last_fullsized":"2024-03-24T00:36:07.794542+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:04:21.024582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.801339+0000","last_change":"2024-03-24T00:36:07.799677+0000","last_active":"2024-03-24T00:36:07.801339+0000","last_peered":"2024-03-24T00:36:07.801339+0000","last_clean":"2024-03-24T00:36:07.801339+0000","last_became_active":"2024-03-24T00:36:07.799366+0000","last_became_peered":"2024-03-24T00:36:07.799366+0000","last_unstale":"2024-03-24T00:36:07.801339+0000","last_undegraded":"2024-03-24T00:36:07.801339+0000","last_fullsized":"2024-03-24T00:36:07.801339+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:36.588320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.809924+0000","last_change":"2024-03-24T00:36:07.801007+0000","last_active":"2024-03-24T00:36:07.809924+0000","last_peered":"2024-03-24T00:36:07.809924+0000","last_clean":"2024-03-24T00:36:07.809924+0000","last_became_active":"2024-03-24T00:36:07.796203+0000","last_became_peered":"2024-03-24T00:36:07.796203+0000","last_unstale":"2024-03-24T00:36:07.809924+0000","last_undegraded":"2024-03-24T00:36:07.809924+0000","last_fullsized":"2024-03-24T00:36:07.809924+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:51.731361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.809268+0000","last_change":"2024-03-24T00:36:07.797764+0000","last_active":"2024-03-24T00:36:07.809268+0000","last_peered":"2024-03-24T00:36:07.809268+0000","last_clean":"2024-03-24T00:36:07.809268+0000","last_became_active":"2024-03-24T00:36:07.797501+0000","last_became_peered":"2024-03-24T00:36:07.797501+0000","last_unstale":"2024-03-24T00:36:07.809268+0000","last_undegraded":"2024-03-24T00:36:07.809268+0000","last_fullsized":"2024-03-24T00:36:07.809268+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:30:36.588320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.809576+0000","last_change":"2024-03-24T00:36:07.799673+0000","last_active":"2024-03-24T00:36:07.809576+0000","last_peered":"2024-03-24T00:36:07.809576+0000","last_clean":"2024-03-24T00:36:07.809576+0000","last_became_active":"2024-03-24T00:36:07.799407+0000","last_became_peered":"2024-03-24T00:36:07.799407+0000","last_unstale":"2024-03-24T00:36:07.809576+0000","last_undegraded":"2024-03-24T00:36:07.809576+0000","last_fullsized":"2024-03-24T00:36:07.809576+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:07.699038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"82'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.822139+0000","last_change":"2024-03-24T00:36:06.779007+0000","last_active":"2024-03-24T00:36:07.822139+0000","last_peered":"2024-03-24T00:36:07.822139+0000","last_clean":"2024-03-24T00:36:07.822139+0000","last_became_active":"2024-03-24T00:36:06.778716+0000","last_became_peered":"2024-03-24T00:36:06.778716+0000","last_unstale":"2024-03-24T00:36:07.822139+0000","last_undegraded":"2024-03-24T00:36:07.822139+0000","last_fullsized":"2024-03-24T00:36:07.822139+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:02:11.729651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.816632+0000","last_change":"2024-03-24T00:36:07.809262+0000","last_active":"2024-03-24T00:36:07.816632+0000","last_peered":"2024-03-24T00:36:07.816632+0000","last_clean":"2024-03-24T00:36:07.816632+0000","last_became_active":"2024-03-24T00:36:07.808496+0000","last_became_peered":"2024-03-24T00:36:07.808496+0000","last_unstale":"2024-03-24T00:36:07.816632+0000","last_undegraded":"2024-03-24T00:36:07.816632+0000","last_fullsized":"2024-03-24T00:36:07.816632+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:35:17.249084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"82'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.823154+0000","last_change":"2024-03-24T00:36:06.788811+0000","last_active":"2024-03-24T00:36:07.823154+0000","last_peered":"2024-03-24T00:36:07.823154+0000","last_clean":"2024-03-24T00:36:07.823154+0000","last_became_active":"2024-03-24T00:36:06.788484+0000","last_became_peered":"2024-03-24T00:36:06.788484+0000","last_unstale":"2024-03-24T00:36:07.823154+0000","last_undegraded":"2024-03-24T00:36:07.823154+0000","last_fullsized":"2024-03-24T00:36:07.823154+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:00:13.159554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.809975+0000","last_change":"2024-03-24T00:36:07.798671+0000","last_active":"2024-03-24T00:36:07.809975+0000","last_peered":"2024-03-24T00:36:07.809975+0000","last_clean":"2024-03-24T00:36:07.809975+0000","last_became_active":"2024-03-24T00:36:07.798336+0000","last_became_peered":"2024-03-24T00:36:07.798336+0000","last_unstale":"2024-03-24T00:36:07.809975+0000","last_undegraded":"2024-03-24T00:36:07.809975+0000","last_fullsized":"2024-03-24T00:36:07.809975+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:02.365494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.818858+0000","last_change":"2024-03-24T00:36:06.776010+0000","last_active":"2024-03-24T00:36:07.818858+0000","last_peered":"2024-03-24T00:36:07.818858+0000","last_clean":"2024-03-24T00:36:07.818858+0000","last_became_active":"2024-03-24T00:36:06.775739+0000","last_became_peered":"2024-03-24T00:36:06.775739+0000","last_unstale":"2024-03-24T00:36:07.818858+0000","last_undegraded":"2024-03-24T00:36:07.818858+0000","last_fullsized":"2024-03-24T00:36:07.818858+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:07.273374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.805103+0000","last_change":"2024-03-24T00:36:07.803836+0000","last_active":"2024-03-24T00:36:07.805103+0000","last_peered":"2024-03-24T00:36:07.805103+0000","last_clean":"2024-03-24T00:36:07.805103+0000","last_became_active":"2024-03-24T00:36:07.802301+0000","last_became_peered":"2024-03-24T00:36:07.802301+0000","last_unstale":"2024-03-24T00:36:07.805103+0000","last_undegraded":"2024-03-24T00:36:07.805103+0000","last_fullsized":"2024-03-24T00:36:07.805103+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:07.407835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.819201+0000","last_change":"2024-03-24T00:36:06.775287+0000","last_active":"2024-03-24T00:36:07.819201+0000","last_peered":"2024-03-24T00:36:07.819201+0000","last_clean":"2024-03-24T00:36:07.819201+0000","last_became_active":"2024-03-24T00:36:06.774998+0000","last_became_peered":"2024-03-24T00:36:06.774998+0000","last_unstale":"2024-03-24T00:36:07.819201+0000","last_undegraded":"2024-03-24T00:36:07.819201+0000","last_fullsized":"2024-03-24T00:36:07.819201+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:26.566015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.801010+0000","last_change":"2024-03-24T00:36:07.799119+0000","last_active":"2024-03-24T00:36:07.801010+0000","last_peered":"2024-03-24T00:36:07.801010+0000","last_clean":"2024-03-24T00:36:07.801010+0000","last_became_active":"2024-03-24T00:36:07.798807+0000","last_became_peered":"2024-03-24T00:36:07.798807+0000","last_unstale":"2024-03-24T00:36:07.801010+0000","last_undegraded":"2024-03-24T00:36:07.801010+0000","last_fullsized":"2024-03-24T00:36:07.801010+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:11:43.247886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"82'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.819579+0000","last_change":"2024-03-24T00:36:06.766374+0000","last_active":"2024-03-24T00:36:07.819579+0000","last_peered":"2024-03-24T00:36:07.819579+0000","last_clean":"2024-03-24T00:36:07.819579+0000","last_became_active":"2024-03-24T00:36:06.765492+0000","last_became_peered":"2024-03-24T00:36:06.765492+0000","last_unstale":"2024-03-24T00:36:07.819579+0000","last_undegraded":"2024-03-24T00:36:07.819579+0000","last_fullsized":"2024-03-24T00:36:07.819579+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:41.584491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.808272+0000","last_change":"2024-03-24T00:36:07.803639+0000","last_active":"2024-03-24T00:36:07.808272+0000","last_peered":"2024-03-24T00:36:07.808272+0000","last_clean":"2024-03-24T00:36:07.808272+0000","last_became_active":"2024-03-24T00:36:07.803357+0000","last_became_peered":"2024-03-24T00:36:07.803357+0000","last_unstale":"2024-03-24T00:36:07.808272+0000","last_undegraded":"2024-03-24T00:36:07.808272+0000","last_fullsized":"2024-03-24T00:36:07.808272+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:13:31.695719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"82'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.820717+0000","last_change":"2024-03-24T00:36:06.788690+0000","last_active":"2024-03-24T00:36:07.820717+0000","last_peered":"2024-03-24T00:36:07.820717+0000","last_clean":"2024-03-24T00:36:07.820717+0000","last_became_active":"2024-03-24T00:36:06.788392+0000","last_became_peered":"2024-03-24T00:36:06.788392+0000","last_unstale":"2024-03-24T00:36:07.820717+0000","last_undegraded":"2024-03-24T00:36:07.820717+0000","last_fullsized":"2024-03-24T00:36:07.820717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:02:11.729651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.804466+0000","last_change":"2024-03-24T00:36:07.795173+0000","last_active":"2024-03-24T00:36:07.804466+0000","last_peered":"2024-03-24T00:36:07.804466+0000","last_clean":"2024-03-24T00:36:07.804466+0000","last_became_active":"2024-03-24T00:36:07.794862+0000","last_became_peered":"2024-03-24T00:36:07.794862+0000","last_unstale":"2024-03-24T00:36:07.804466+0000","last_undegraded":"2024-03-24T00:36:07.804466+0000","last_fullsized":"2024-03-24T00:36:07.804466+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:08.280925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.785051+0000","last_change":"2024-03-24T00:36:06.776923+0000","last_active":"2024-03-24T00:36:07.785051+0000","last_peered":"2024-03-24T00:36:07.785051+0000","last_clean":"2024-03-24T00:36:07.785051+0000","last_became_active":"2024-03-24T00:36:06.776619+0000","last_became_peered":"2024-03-24T00:36:06.776619+0000","last_unstale":"2024-03-24T00:36:07.785051+0000","last_undegraded":"2024-03-24T00:36:07.785051+0000","last_fullsized":"2024-03-24T00:36:07.785051+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:06.400284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786383+0000","last_change":"2024-03-24T00:36:07.784117+0000","last_active":"2024-03-24T00:36:07.786383+0000","last_peered":"2024-03-24T00:36:07.786383+0000","last_clean":"2024-03-24T00:36:07.786383+0000","last_became_active":"2024-03-24T00:36:07.783795+0000","last_became_peered":"2024-03-24T00:36:07.783795+0000","last_unstale":"2024-03-24T00:36:07.786383+0000","last_undegraded":"2024-03-24T00:36:07.786383+0000","last_fullsized":"2024-03-24T00:36:07.786383+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:01.647460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.775493+0000","last_change":"2024-03-24T00:36:06.770035+0000","last_active":"2024-03-24T00:36:07.775493+0000","last_peered":"2024-03-24T00:36:07.775493+0000","last_clean":"2024-03-24T00:36:07.775493+0000","last_became_active":"2024-03-24T00:36:06.766581+0000","last_became_peered":"2024-03-24T00:36:06.766581+0000","last_unstale":"2024-03-24T00:36:07.775493+0000","last_undegraded":"2024-03-24T00:36:07.775493+0000","last_fullsized":"2024-03-24T00:36:07.775493+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:51:11.132128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786045+0000","last_change":"2024-03-24T00:36:07.782705+0000","last_active":"2024-03-24T00:36:07.786045+0000","last_peered":"2024-03-24T00:36:07.786045+0000","last_clean":"2024-03-24T00:36:07.786045+0000","last_became_active":"2024-03-24T00:36:07.782334+0000","last_became_peered":"2024-03-24T00:36:07.782334+0000","last_unstale":"2024-03-24T00:36:07.786045+0000","last_undegraded":"2024-03-24T00:36:07.786045+0000","last_fullsized":"2024-03-24T00:36:07.786045+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:34:39.426391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.801581+0000","last_change":"2024-03-24T00:36:06.790394+0000","last_active":"2024-03-24T00:36:07.801581+0000","last_peered":"2024-03-24T00:36:07.801581+0000","last_clean":"2024-03-24T00:36:07.801581+0000","last_became_active":"2024-03-24T00:36:06.790201+0000","last_became_peered":"2024-03-24T00:36:06.790201+0000","last_unstale":"2024-03-24T00:36:07.801581+0000","last_undegraded":"2024-03-24T00:36:07.801581+0000","last_fullsized":"2024-03-24T00:36:07.801581+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:53.396891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.806180+0000","last_change":"2024-03-24T00:36:07.801340+0000","last_active":"2024-03-24T00:36:07.806180+0000","last_peered":"2024-03-24T00:36:07.806180+0000","last_clean":"2024-03-24T00:36:07.806180+0000","last_became_active":"2024-03-24T00:36:07.801053+0000","last_became_peered":"2024-03-24T00:36:07.801053+0000","last_unstale":"2024-03-24T00:36:07.806180+0000","last_undegraded":"2024-03-24T00:36:07.806180+0000","last_fullsized":"2024-03-24T00:36:07.806180+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:15.579736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.800220+0000","last_change":"2024-03-24T00:36:06.775340+0000","last_active":"2024-03-24T00:36:07.800220+0000","last_peered":"2024-03-24T00:36:07.800220+0000","last_clean":"2024-03-24T00:36:07.800220+0000","last_became_active":"2024-03-24T00:36:06.775080+0000","last_became_peered":"2024-03-24T00:36:06.775080+0000","last_unstale":"2024-03-24T00:36:07.800220+0000","last_undegraded":"2024-03-24T00:36:07.800220+0000","last_fullsized":"2024-03-24T00:36:07.800220+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:14.552068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.814879+0000","last_change":"2024-03-24T00:36:07.811537+0000","last_active":"2024-03-24T00:36:07.814879+0000","last_peered":"2024-03-24T00:36:07.814879+0000","last_clean":"2024-03-24T00:36:07.814879+0000","last_became_active":"2024-03-24T00:36:07.811241+0000","last_became_peered":"2024-03-24T00:36:07.811241+0000","last_unstale":"2024-03-24T00:36:07.814879+0000","last_undegraded":"2024-03-24T00:36:07.814879+0000","last_fullsized":"2024-03-24T00:36:07.814879+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:35:15.406036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"82'3","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.823176+0000","last_change":"2024-03-24T00:36:06.781402+0000","last_active":"2024-03-24T00:36:07.823176+0000","last_peered":"2024-03-24T00:36:07.823176+0000","last_clean":"2024-03-24T00:36:07.823176+0000","last_became_active":"2024-03-24T00:36:06.781117+0000","last_became_peered":"2024-03-24T00:36:06.781117+0000","last_unstale":"2024-03-24T00:36:07.823176+0000","last_undegraded":"2024-03-24T00:36:07.823176+0000","last_fullsized":"2024-03-24T00:36:07.823176+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:13.706926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.804676+0000","last_change":"2024-03-24T00:36:07.796829+0000","last_active":"2024-03-24T00:36:07.804676+0000","last_peered":"2024-03-24T00:36:07.804676+0000","last_clean":"2024-03-24T00:36:07.804676+0000","last_became_active":"2024-03-24T00:36:07.796152+0000","last_became_peered":"2024-03-24T00:36:07.796152+0000","last_unstale":"2024-03-24T00:36:07.804676+0000","last_undegraded":"2024-03-24T00:36:07.804676+0000","last_fullsized":"2024-03-24T00:36:07.804676+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:35:17.249084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"82'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.823425+0000","last_change":"2024-03-24T00:36:06.787486+0000","last_active":"2024-03-24T00:36:07.823425+0000","last_peered":"2024-03-24T00:36:07.823425+0000","last_clean":"2024-03-24T00:36:07.823425+0000","last_became_active":"2024-03-24T00:36:06.787119+0000","last_became_peered":"2024-03-24T00:36:06.787119+0000","last_unstale":"2024-03-24T00:36:07.823425+0000","last_undegraded":"2024-03-24T00:36:07.823425+0000","last_fullsized":"2024-03-24T00:36:07.823425+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:41.584491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.806883+0000","last_change":"2024-03-24T00:36:07.805898+0000","last_active":"2024-03-24T00:36:07.806883+0000","last_peered":"2024-03-24T00:36:07.806883+0000","last_clean":"2024-03-24T00:36:07.806883+0000","last_became_active":"2024-03-24T00:36:07.805647+0000","last_became_peered":"2024-03-24T00:36:07.805647+0000","last_unstale":"2024-03-24T00:36:07.806883+0000","last_undegraded":"2024-03-24T00:36:07.806883+0000","last_fullsized":"2024-03-24T00:36:07.806883+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:04.419002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.820428+0000","last_change":"2024-03-24T00:36:06.770844+0000","last_active":"2024-03-24T00:36:07.820428+0000","last_peered":"2024-03-24T00:36:07.820428+0000","last_clean":"2024-03-24T00:36:07.820428+0000","last_became_active":"2024-03-24T00:36:06.770606+0000","last_became_peered":"2024-03-24T00:36:06.770606+0000","last_unstale":"2024-03-24T00:36:07.820428+0000","last_undegraded":"2024-03-24T00:36:07.820428+0000","last_fullsized":"2024-03-24T00:36:07.820428+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:08.347951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.804564+0000","last_change":"2024-03-24T00:36:07.801022+0000","last_active":"2024-03-24T00:36:07.804564+0000","last_peered":"2024-03-24T00:36:07.804564+0000","last_clean":"2024-03-24T00:36:07.804564+0000","last_became_active":"2024-03-24T00:36:07.800676+0000","last_became_peered":"2024-03-24T00:36:07.800676+0000","last_unstale":"2024-03-24T00:36:07.804564+0000","last_undegraded":"2024-03-24T00:36:07.804564+0000","last_fullsized":"2024-03-24T00:36:07.804564+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:37.605081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.780048+0000","last_change":"2024-03-24T00:36:06.787472+0000","last_active":"2024-03-24T00:36:07.780048+0000","last_peered":"2024-03-24T00:36:07.780048+0000","last_clean":"2024-03-24T00:36:07.780048+0000","last_became_active":"2024-03-24T00:36:06.787163+0000","last_became_peered":"2024-03-24T00:36:06.787163+0000","last_unstale":"2024-03-24T00:36:07.780048+0000","last_undegraded":"2024-03-24T00:36:07.780048+0000","last_fullsized":"2024-03-24T00:36:07.780048+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:35:14.398485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.799073+0000","last_change":"2024-03-24T00:36:07.795051+0000","last_active":"2024-03-24T00:36:07.799073+0000","last_peered":"2024-03-24T00:36:07.799073+0000","last_clean":"2024-03-24T00:36:07.799073+0000","last_became_active":"2024-03-24T00:36:07.794681+0000","last_became_peered":"2024-03-24T00:36:07.794681+0000","last_unstale":"2024-03-24T00:36:07.799073+0000","last_undegraded":"2024-03-24T00:36:07.799073+0000","last_fullsized":"2024-03-24T00:36:07.799073+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:21.815307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.780203+0000","last_change":"2024-03-24T00:36:06.776589+0000","last_active":"2024-03-24T00:36:07.780203+0000","last_peered":"2024-03-24T00:36:07.780203+0000","last_clean":"2024-03-24T00:36:07.780203+0000","last_became_active":"2024-03-24T00:36:06.775939+0000","last_became_peered":"2024-03-24T00:36:06.775939+0000","last_unstale":"2024-03-24T00:36:07.780203+0000","last_undegraded":"2024-03-24T00:36:07.780203+0000","last_fullsized":"2024-03-24T00:36:07.780203+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:53.396891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.806204+0000","last_change":"2024-03-24T00:36:07.805320+0000","last_active":"2024-03-24T00:36:07.806204+0000","last_peered":"2024-03-24T00:36:07.806204+0000","last_clean":"2024-03-24T00:36:07.806204+0000","last_became_active":"2024-03-24T00:36:07.805042+0000","last_became_peered":"2024-03-24T00:36:07.805042+0000","last_unstale":"2024-03-24T00:36:07.806204+0000","last_undegraded":"2024-03-24T00:36:07.806204+0000","last_fullsized":"2024-03-24T00:36:07.806204+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:46:02.365494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"82'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.822096+0000","last_change":"2024-03-24T00:36:06.769747+0000","last_active":"2024-03-24T00:36:07.822096+0000","last_peered":"2024-03-24T00:36:07.822096+0000","last_clean":"2024-03-24T00:36:07.822096+0000","last_became_active":"2024-03-24T00:36:06.769455+0000","last_became_peered":"2024-03-24T00:36:06.769455+0000","last_unstale":"2024-03-24T00:36:07.822096+0000","last_undegraded":"2024-03-24T00:36:07.822096+0000","last_fullsized":"2024-03-24T00:36:07.822096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:42.579400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.796608+0000","last_change":"2024-03-24T00:36:07.787309+0000","last_active":"2024-03-24T00:36:07.796608+0000","last_peered":"2024-03-24T00:36:07.796608+0000","last_clean":"2024-03-24T00:36:07.796608+0000","last_became_active":"2024-03-24T00:36:07.785330+0000","last_became_peered":"2024-03-24T00:36:07.785330+0000","last_unstale":"2024-03-24T00:36:07.796608+0000","last_undegraded":"2024-03-24T00:36:07.796608+0000","last_fullsized":"2024-03-24T00:36:07.796608+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:39:43.586951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"82'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.821008+0000","last_change":"2024-03-24T00:36:06.779681+0000","last_active":"2024-03-24T00:36:07.821008+0000","last_peered":"2024-03-24T00:36:07.821008+0000","last_clean":"2024-03-24T00:36:07.821008+0000","last_became_active":"2024-03-24T00:36:06.779412+0000","last_became_peered":"2024-03-24T00:36:06.779412+0000","last_unstale":"2024-03-24T00:36:07.821008+0000","last_undegraded":"2024-03-24T00:36:07.821008+0000","last_fullsized":"2024-03-24T00:36:07.821008+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:41.584491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.816656+0000","last_change":"2024-03-24T00:36:07.809378+0000","last_active":"2024-03-24T00:36:07.816656+0000","last_peered":"2024-03-24T00:36:07.816656+0000","last_clean":"2024-03-24T00:36:07.816656+0000","last_became_active":"2024-03-24T00:36:07.808599+0000","last_became_peered":"2024-03-24T00:36:07.808599+0000","last_unstale":"2024-03-24T00:36:07.816656+0000","last_undegraded":"2024-03-24T00:36:07.816656+0000","last_fullsized":"2024-03-24T00:36:07.816656+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:32:15.351688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.819745+0000","last_change":"2024-03-24T00:36:06.773088+0000","last_active":"2024-03-24T00:36:07.819745+0000","last_peered":"2024-03-24T00:36:07.819745+0000","last_clean":"2024-03-24T00:36:07.819745+0000","last_became_active":"2024-03-24T00:36:06.772820+0000","last_became_peered":"2024-03-24T00:36:06.772820+0000","last_unstale":"2024-03-24T00:36:07.819745+0000","last_undegraded":"2024-03-24T00:36:07.819745+0000","last_fullsized":"2024-03-24T00:36:07.819745+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:36:21.533134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.804652+0000","last_change":"2024-03-24T00:36:07.796802+0000","last_active":"2024-03-24T00:36:07.804652+0000","last_peered":"2024-03-24T00:36:07.804652+0000","last_clean":"2024-03-24T00:36:07.804652+0000","last_became_active":"2024-03-24T00:36:07.796050+0000","last_became_peered":"2024-03-24T00:36:07.796050+0000","last_unstale":"2024-03-24T00:36:07.804652+0000","last_undegraded":"2024-03-24T00:36:07.804652+0000","last_fullsized":"2024-03-24T00:36:07.804652+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:09:13.747978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.784830+0000","last_change":"2024-03-24T00:36:06.778230+0000","last_active":"2024-03-24T00:36:07.784830+0000","last_peered":"2024-03-24T00:36:07.784830+0000","last_clean":"2024-03-24T00:36:07.784830+0000","last_became_active":"2024-03-24T00:36:06.777922+0000","last_became_peered":"2024-03-24T00:36:06.777922+0000","last_unstale":"2024-03-24T00:36:07.784830+0000","last_undegraded":"2024-03-24T00:36:07.784830+0000","last_fullsized":"2024-03-24T00:36:07.784830+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:08:36.597530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.800972+0000","last_change":"2024-03-24T00:36:07.799062+0000","last_active":"2024-03-24T00:36:07.800972+0000","last_peered":"2024-03-24T00:36:07.800972+0000","last_clean":"2024-03-24T00:36:07.800972+0000","last_became_active":"2024-03-24T00:36:07.797704+0000","last_became_peered":"2024-03-24T00:36:07.797704+0000","last_unstale":"2024-03-24T00:36:07.800972+0000","last_undegraded":"2024-03-24T00:36:07.800972+0000","last_fullsized":"2024-03-24T00:36:07.800972+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:58.938672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.818768+0000","last_change":"2024-03-24T00:36:06.785932+0000","last_active":"2024-03-24T00:36:07.818768+0000","last_peered":"2024-03-24T00:36:07.818768+0000","last_clean":"2024-03-24T00:36:07.818768+0000","last_became_active":"2024-03-24T00:36:06.785320+0000","last_became_peered":"2024-03-24T00:36:06.785320+0000","last_unstale":"2024-03-24T00:36:07.818768+0000","last_undegraded":"2024-03-24T00:36:07.818768+0000","last_fullsized":"2024-03-24T00:36:07.818768+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:37:49.817212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786510+0000","last_change":"2024-03-24T00:36:07.784193+0000","last_active":"2024-03-24T00:36:07.786510+0000","last_peered":"2024-03-24T00:36:07.786510+0000","last_clean":"2024-03-24T00:36:07.786510+0000","last_became_active":"2024-03-24T00:36:07.783836+0000","last_became_peered":"2024-03-24T00:36:07.783836+0000","last_unstale":"2024-03-24T00:36:07.786510+0000","last_undegraded":"2024-03-24T00:36:07.786510+0000","last_fullsized":"2024-03-24T00:36:07.786510+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:03:30.373399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.785723+0000","last_change":"2024-03-24T00:36:06.784639+0000","last_active":"2024-03-24T00:36:07.785723+0000","last_peered":"2024-03-24T00:36:07.785723+0000","last_clean":"2024-03-24T00:36:07.785723+0000","last_became_active":"2024-03-24T00:36:06.784248+0000","last_became_peered":"2024-03-24T00:36:06.784248+0000","last_unstale":"2024-03-24T00:36:07.785723+0000","last_undegraded":"2024-03-24T00:36:07.785723+0000","last_fullsized":"2024-03-24T00:36:07.785723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:17.546360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.800203+0000","last_change":"2024-03-24T00:36:07.793988+0000","last_active":"2024-03-24T00:36:07.800203+0000","last_peered":"2024-03-24T00:36:07.800203+0000","last_clean":"2024-03-24T00:36:07.800203+0000","last_became_active":"2024-03-24T00:36:07.793088+0000","last_became_peered":"2024-03-24T00:36:07.793088+0000","last_unstale":"2024-03-24T00:36:07.800203+0000","last_undegraded":"2024-03-24T00:36:07.800203+0000","last_fullsized":"2024-03-24T00:36:07.800203+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:55:22.531204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.787096+0000","last_change":"2024-03-24T00:36:06.777058+0000","last_active":"2024-03-24T00:36:07.787096+0000","last_peered":"2024-03-24T00:36:07.787096+0000","last_clean":"2024-03-24T00:36:07.787096+0000","last_became_active":"2024-03-24T00:36:06.775965+0000","last_became_peered":"2024-03-24T00:36:06.775965+0000","last_unstale":"2024-03-24T00:36:07.787096+0000","last_undegraded":"2024-03-24T00:36:07.787096+0000","last_fullsized":"2024-03-24T00:36:07.787096+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:58:20.516373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.800951+0000","last_change":"2024-03-24T00:36:07.795087+0000","last_active":"2024-03-24T00:36:07.800951+0000","last_peered":"2024-03-24T00:36:07.800951+0000","last_clean":"2024-03-24T00:36:07.800951+0000","last_became_active":"2024-03-24T00:36:07.794779+0000","last_became_peered":"2024-03-24T00:36:07.794779+0000","last_unstale":"2024-03-24T00:36:07.800951+0000","last_undegraded":"2024-03-24T00:36:07.800951+0000","last_fullsized":"2024-03-24T00:36:07.800951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:32:15.351688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.779693+0000","last_change":"2024-03-24T00:36:06.785925+0000","last_active":"2024-03-24T00:36:07.779693+0000","last_peered":"2024-03-24T00:36:07.779693+0000","last_clean":"2024-03-24T00:36:07.779693+0000","last_became_active":"2024-03-24T00:36:06.784803+0000","last_became_peered":"2024-03-24T00:36:06.784803+0000","last_unstale":"2024-03-24T00:36:07.779693+0000","last_undegraded":"2024-03-24T00:36:07.779693+0000","last_fullsized":"2024-03-24T00:36:07.779693+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:03.411451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.803798+0000","last_change":"2024-03-24T00:36:07.794130+0000","last_active":"2024-03-24T00:36:07.803798+0000","last_peered":"2024-03-24T00:36:07.803798+0000","last_clean":"2024-03-24T00:36:07.803798+0000","last_became_active":"2024-03-24T00:36:07.793038+0000","last_became_peered":"2024-03-24T00:36:07.793038+0000","last_unstale":"2024-03-24T00:36:07.803798+0000","last_undegraded":"2024-03-24T00:36:07.803798+0000","last_fullsized":"2024-03-24T00:36:07.803798+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:03:30.373399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"82'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.817346+0000","last_change":"2024-03-24T00:36:06.768410+0000","last_active":"2024-03-24T00:36:07.817346+0000","last_peered":"2024-03-24T00:36:07.817346+0000","last_clean":"2024-03-24T00:36:07.817346+0000","last_became_active":"2024-03-24T00:36:06.767675+0000","last_became_peered":"2024-03-24T00:36:06.767675+0000","last_unstale":"2024-03-24T00:36:07.817346+0000","last_undegraded":"2024-03-24T00:36:07.817346+0000","last_fullsized":"2024-03-24T00:36:07.817346+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:50:47.347515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.813913+0000","last_change":"2024-03-24T00:36:07.811569+0000","last_active":"2024-03-24T00:36:07.813913+0000","last_peered":"2024-03-24T00:36:07.813913+0000","last_clean":"2024-03-24T00:36:07.813913+0000","last_became_active":"2024-03-24T00:36:07.811224+0000","last_became_peered":"2024-03-24T00:36:07.811224+0000","last_unstale":"2024-03-24T00:36:07.813913+0000","last_undegraded":"2024-03-24T00:36:07.813913+0000","last_fullsized":"2024-03-24T00:36:07.813913+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:03:07.442469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.776578+0000","last_change":"2024-03-24T00:36:06.771523+0000","last_active":"2024-03-24T00:36:07.776578+0000","last_peered":"2024-03-24T00:36:07.776578+0000","last_clean":"2024-03-24T00:36:07.776578+0000","last_became_active":"2024-03-24T00:36:06.771247+0000","last_became_peered":"2024-03-24T00:36:06.771247+0000","last_unstale":"2024-03-24T00:36:07.776578+0000","last_undegraded":"2024-03-24T00:36:07.776578+0000","last_fullsized":"2024-03-24T00:36:07.776578+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:49:56.402081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.817146+0000","last_change":"2024-03-24T00:36:07.806055+0000","last_active":"2024-03-24T00:36:07.817146+0000","last_peered":"2024-03-24T00:36:07.817146+0000","last_clean":"2024-03-24T00:36:07.817146+0000","last_became_active":"2024-03-24T00:36:07.805638+0000","last_became_peered":"2024-03-24T00:36:07.805638+0000","last_unstale":"2024-03-24T00:36:07.817146+0000","last_undegraded":"2024-03-24T00:36:07.817146+0000","last_fullsized":"2024-03-24T00:36:07.817146+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:03:30.373399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.799749+0000","last_change":"2024-03-24T00:36:06.779343+0000","last_active":"2024-03-24T00:36:07.799749+0000","last_peered":"2024-03-24T00:36:07.799749+0000","last_clean":"2024-03-24T00:36:07.799749+0000","last_became_active":"2024-03-24T00:36:06.778003+0000","last_became_peered":"2024-03-24T00:36:06.778003+0000","last_unstale":"2024-03-24T00:36:07.799749+0000","last_undegraded":"2024-03-24T00:36:07.799749+0000","last_fullsized":"2024-03-24T00:36:07.799749+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:16:53.545725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.797068+0000","last_change":"2024-03-24T00:36:07.791525+0000","last_active":"2024-03-24T00:36:07.797068+0000","last_peered":"2024-03-24T00:36:07.797068+0000","last_clean":"2024-03-24T00:36:07.797068+0000","last_became_active":"2024-03-24T00:36:07.790720+0000","last_became_peered":"2024-03-24T00:36:07.790720+0000","last_unstale":"2024-03-24T00:36:07.797068+0000","last_undegraded":"2024-03-24T00:36:07.797068+0000","last_fullsized":"2024-03-24T00:36:07.797068+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:15.559619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.799272+0000","last_change":"2024-03-24T00:36:06.787365+0000","last_active":"2024-03-24T00:36:07.799272+0000","last_peered":"2024-03-24T00:36:07.799272+0000","last_clean":"2024-03-24T00:36:07.799272+0000","last_became_active":"2024-03-24T00:36:06.787075+0000","last_became_peered":"2024-03-24T00:36:06.787075+0000","last_unstale":"2024-03-24T00:36:07.799272+0000","last_undegraded":"2024-03-24T00:36:07.799272+0000","last_fullsized":"2024-03-24T00:36:07.799272+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:25.384461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786537+0000","last_change":"2024-03-24T00:36:07.784223+0000","last_active":"2024-03-24T00:36:07.786537+0000","last_peered":"2024-03-24T00:36:07.786537+0000","last_clean":"2024-03-24T00:36:07.786537+0000","last_became_active":"2024-03-24T00:36:07.783854+0000","last_became_peered":"2024-03-24T00:36:07.783854+0000","last_unstale":"2024-03-24T00:36:07.786537+0000","last_undegraded":"2024-03-24T00:36:07.786537+0000","last_fullsized":"2024-03-24T00:36:07.786537+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:09:13.747978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.786842+0000","last_change":"2024-03-24T00:36:06.768563+0000","last_active":"2024-03-24T00:36:07.786842+0000","last_peered":"2024-03-24T00:36:07.786842+0000","last_clean":"2024-03-24T00:36:07.786842+0000","last_became_active":"2024-03-24T00:36:06.768279+0000","last_became_peered":"2024-03-24T00:36:06.768279+0000","last_unstale":"2024-03-24T00:36:07.786842+0000","last_undegraded":"2024-03-24T00:36:07.786842+0000","last_fullsized":"2024-03-24T00:36:07.786842+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:43.770350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.800465+0000","last_change":"2024-03-24T00:36:07.797402+0000","last_active":"2024-03-24T00:36:07.800465+0000","last_peered":"2024-03-24T00:36:07.800465+0000","last_clean":"2024-03-24T00:36:07.800465+0000","last_became_active":"2024-03-24T00:36:07.797045+0000","last_became_peered":"2024-03-24T00:36:07.797045+0000","last_unstale":"2024-03-24T00:36:07.800465+0000","last_undegraded":"2024-03-24T00:36:07.800465+0000","last_fullsized":"2024-03-24T00:36:07.800465+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:03.022418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.792373+0000","last_change":"2024-03-24T00:36:06.788740+0000","last_active":"2024-03-24T00:36:07.792373+0000","last_peered":"2024-03-24T00:36:07.792373+0000","last_clean":"2024-03-24T00:36:07.792373+0000","last_became_active":"2024-03-24T00:36:06.788406+0000","last_became_peered":"2024-03-24T00:36:06.788406+0000","last_unstale":"2024-03-24T00:36:07.792373+0000","last_undegraded":"2024-03-24T00:36:07.792373+0000","last_fullsized":"2024-03-24T00:36:07.792373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:53.396891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.789427+0000","last_change":"2024-03-24T00:36:07.784571+0000","last_active":"2024-03-24T00:36:07.789427+0000","last_peered":"2024-03-24T00:36:07.789427+0000","last_clean":"2024-03-24T00:36:07.789427+0000","last_became_active":"2024-03-24T00:36:07.784254+0000","last_became_peered":"2024-03-24T00:36:07.784254+0000","last_unstale":"2024-03-24T00:36:07.789427+0000","last_undegraded":"2024-03-24T00:36:07.789427+0000","last_fullsized":"2024-03-24T00:36:07.789427+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:13.299874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.799932+0000","last_change":"2024-03-24T00:36:06.786048+0000","last_active":"2024-03-24T00:36:07.799932+0000","last_peered":"2024-03-24T00:36:07.799932+0000","last_clean":"2024-03-24T00:36:07.799932+0000","last_became_active":"2024-03-24T00:36:06.785829+0000","last_became_peered":"2024-03-24T00:36:06.785829+0000","last_unstale":"2024-03-24T00:36:07.799932+0000","last_undegraded":"2024-03-24T00:36:07.799932+0000","last_fullsized":"2024-03-24T00:36:07.799932+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:04:40.750748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.810175+0000","last_change":"2024-03-24T00:36:07.800801+0000","last_active":"2024-03-24T00:36:07.810175+0000","last_peered":"2024-03-24T00:36:07.810175+0000","last_clean":"2024-03-24T00:36:07.810175+0000","last_became_active":"2024-03-24T00:36:07.800481+0000","last_became_peered":"2024-03-24T00:36:07.800481+0000","last_unstale":"2024-03-24T00:36:07.810175+0000","last_undegraded":"2024-03-24T00:36:07.810175+0000","last_fullsized":"2024-03-24T00:36:07.810175+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:51:17.573588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.791834+0000","last_change":"2024-03-24T00:36:06.773919+0000","last_active":"2024-03-24T00:36:07.791834+0000","last_peered":"2024-03-24T00:36:07.791834+0000","last_clean":"2024-03-24T00:36:07.791834+0000","last_became_active":"2024-03-24T00:36:06.773572+0000","last_became_peered":"2024-03-24T00:36:06.773572+0000","last_unstale":"2024-03-24T00:36:07.791834+0000","last_undegraded":"2024-03-24T00:36:07.791834+0000","last_fullsized":"2024-03-24T00:36:07.791834+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:51:16.566037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.794164+0000","last_change":"2024-03-24T00:36:07.787225+0000","last_active":"2024-03-24T00:36:07.794164+0000","last_peered":"2024-03-24T00:36:07.794164+0000","last_clean":"2024-03-24T00:36:07.794164+0000","last_became_active":"2024-03-24T00:36:07.786635+0000","last_became_peered":"2024-03-24T00:36:07.786635+0000","last_unstale":"2024-03-24T00:36:07.794164+0000","last_undegraded":"2024-03-24T00:36:07.794164+0000","last_fullsized":"2024-03-24T00:36:07.794164+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:06.750911+0000","last_clean_scrub_stamp":"2024-03-24T00:36:06.750911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:29:31.444886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"82'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.819182+0000","last_change":"2024-03-24T00:36:06.776469+0000","last_active":"2024-03-24T00:36:07.819182+0000","last_peered":"2024-03-24T00:36:07.819182+0000","last_clean":"2024-03-24T00:36:07.819182+0000","last_became_active":"2024-03-24T00:36:06.776177+0000","last_became_peered":"2024-03-24T00:36:06.776177+0000","last_unstale":"2024-03-24T00:36:07.819182+0000","last_undegraded":"2024-03-24T00:36:07.819182+0000","last_fullsized":"2024-03-24T00:36:07.819182+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:36:05.743360+0000","last_clean_scrub_stamp":"2024-03-24T00:36:05.743360+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:30:53.328381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":295,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-03-24T00:36:07.779328+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:36:07.779328+0000","last_peered":"2024-03-24T00:36:07.779328+0000","last_clean":"2024-03-24T00:36:07.779328+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:36:07.779328+0000","last_undegraded":"2024-03-24T00:36:07.779328+0000","last_fullsized":"2024-03-24T00:36:07.779328+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36908,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334932,"statfs":{"total":96636764160,"available":96598970368,"internally_reserved":0,"allocated":1605632,"data_stored":1024706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339664,"statfs":{"total":96636764160,"available":96603815936,"internally_reserved":0,"allocated":1019904,"data_stored":434962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1024000,"data_stored":435902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40488,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331352,"statfs":{"total":96636764160,"available":96595304448,"internally_reserved":0,"allocated":1007616,"data_stored":434268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36312,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335528,"statfs":{"total":96636764160,"available":96599580672,"internally_reserved":0,"allocated":995328,"data_stored":427493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36312,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335528,"statfs":{"total":96636764160,"available":96599580672,"internally_reserved":0,"allocated":995328,"data_stored":427493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335500,"statfs":{"total":96636764160,"available":96599552000,"internally_reserved":0,"allocated":1019904,"data_stored":434462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36892,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334948,"statfs":{"total":96636764160,"available":96598986752,"internally_reserved":0,"allocated":1589248,"data_stored":1017861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:36:12.300 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:36:12.300 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:36:12.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.482+0000 7f480f000640 1 Processor -- start 2024-03-24T00:36:12.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.482+0000 7f480f000640 1 -- start start 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f480f000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f480806cb80 0x7f48080b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f480f000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48080b4840 0x7f4808071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f480f000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48080b3260 con 0x7f480815ac60 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f480f000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48080b3e70 con 0x7f48080b4840 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f480f000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48080b3540 con 0x7f480806cb80 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f480d000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f48080b4840 0x7f4808071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f4807e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f480815ac60 0x7f48080b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:35840/0 (socket says 172.21.15.40:35840) 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f4807e00640 1 -- 172.21.15.40:0/3718531154 learned_addr learned my addr 172.21.15.40:0/3718531154 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f47ffe00640 1 --2- 172.21.15.40:0/3718531154 >> v2:172.21.15.53:3300/0 conn(0x7f480806cb80 0x7f48080b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f480d000640 1 -- 172.21.15.40:0/3718531154 >> v2:172.21.15.53:3300/0 conn(0x7f480806cb80 msgr2=0x7f48080b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f480d000640 1 --2- 172.21.15.40:0/3718531154 >> v2:172.21.15.53:3300/0 conn(0x7f480806cb80 0x7f48080b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f480d000640 1 -- 172.21.15.40:0/3718531154 >> v1:172.21.15.53:6789/0 conn(0x7f480815ac60 legacy=0x7f48080b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.486+0000 7f480d000640 1 -- 172.21.15.40:0/3718531154 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48080b3820 con 0x7f48080b4840 2024-03-24T00:36:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480d000640 1 --2- 172.21.15.40:0/3718531154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48080b4840 0x7f4808071b00 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f480815e7d0 tx=0x7f480001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b794e8ce443292e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f4807400640 1 -- 172.21.15.40:0/3718531154 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000b2070 con 0x7f48080b4840 2024-03-24T00:36:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f4807400640 1 -- 172.21.15.40:0/3718531154 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48000a5760 con 0x7f48080b4840 2024-03-24T00:36:12.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f4807400640 1 -- 172.21.15.40:0/3718531154 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000a5b30 con 0x7f48080b4840 2024-03-24T00:36:12.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 -- 172.21.15.40:0/3718531154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48080b4840 msgr2=0x7f4808071b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 --2- 172.21.15.40:0/3718531154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48080b4840 0x7f4808071b00 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f480815e7d0 tx=0x7f480001c8a0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 -- 172.21.15.40:0/3718531154 shutdown_connections 2024-03-24T00:36:12.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 --2- 172.21.15.40:0/3718531154 >> v2:172.21.15.53:3300/0 conn(0x7f480806cb80 0x7f48080b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 --2- 172.21.15.40:0/3718531154 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f48080b4840 0x7f4808071b00 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 -- 172.21.15.40:0/3718531154 >> 172.21.15.40:0/3718531154 conn(0x7f48080b1660 msgr2=0x7f48080b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:12.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 -- 172.21.15.40:0/3718531154 shutdown_connections 2024-03-24T00:36:12.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 -- 172.21.15.40:0/3718531154 wait complete. 2024-03-24T00:36:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 Processor -- start 2024-03-24T00:36:12.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 -- start start 2024-03-24T00:36:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f480806cb80 0x7f4808152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48080b4840 0x7f48080ce350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480815ac60 0x7f48080ce890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4808167c50 con 0x7f480815ac60 2024-03-24T00:36:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4808167ad0 con 0x7f48080b4840 2024-03-24T00:36:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.490+0000 7f480f000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4808167dd0 con 0x7f480806cb80 2024-03-24T00:36:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f480d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480815ac60 0x7f48080ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f4807e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f480806cb80 0x7f4808152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f47ffe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48080b4840 0x7f48080ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f4807e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f480806cb80 0x7f4808152170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:46642/0 (socket says 172.21.15.40:46642) 2024-03-24T00:36:12.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f4807e00640 1 -- 172.21.15.40:0/2955708216 learned_addr learned my addr 172.21.15.40:0/2955708216 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f480d000640 1 -- 172.21.15.40:0/2955708216 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f480806cb80 msgr2=0x7f4808152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f480d000640 1 --2- 172.21.15.40:0/2955708216 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f480806cb80 0x7f4808152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f480d000640 1 -- 172.21.15.40:0/2955708216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48080b4840 msgr2=0x7f48080ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f480d000640 1 --2- 172.21.15.40:0/2955708216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48080b4840 0x7f48080ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f480d000640 1 -- 172.21.15.40:0/2955708216 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480815b9c0 con 0x7f480815ac60 2024-03-24T00:36:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f480d000640 1 --2- 172.21.15.40:0/2955708216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480815ac60 0x7f48080ce890 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f48000ac000 tx=0x7f4800019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f4804c00640 1 -- 172.21.15.40:0/2955708216 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000b2050 con 0x7f480815ac60 2024-03-24T00:36:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f4804c00640 1 -- 172.21.15.40:0/2955708216 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48000b3040 con 0x7f480815ac60 2024-03-24T00:36:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f480f000640 1 -- 172.21.15.40:0/2955708216 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48081fb5a0 con 0x7f480815ac60 2024-03-24T00:36:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f4804c00640 1 -- 172.21.15.40:0/2955708216 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48000ad650 con 0x7f480815ac60 2024-03-24T00:36:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f480f000640 1 -- 172.21.15.40:0/2955708216 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48080cf8e0 con 0x7f480815ac60 2024-03-24T00:36:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f480f000640 1 -- 172.21.15.40:0/2955708216 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4808068170 con 0x7f480815ac60 2024-03-24T00:36:12.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.494+0000 7f4804c00640 1 -- 172.21.15.40:0/2955708216 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f48000c0020 con 0x7f480815ac60 2024-03-24T00:36:12.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.498+0000 7f4804c00640 1 --2- 172.21.15.40:0/2955708216 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f47e4067660 0x7f47e4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.498+0000 7f4804c00640 1 -- 172.21.15.40:0/2955708216 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f480011fe00 con 0x7f480815ac60 2024-03-24T00:36:12.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.498+0000 7f4807e00640 1 --2- 172.21.15.40:0/2955708216 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f47e4067660 0x7f47e4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.498+0000 7f4807e00640 1 --2- 172.21.15.40:0/2955708216 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f47e4067660 0x7f47e4069b20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f47f8007fd0 tx=0x7f47f8007f20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:12.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.502+0000 7f4804c00640 1 -- 172.21.15.40:0/2955708216 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f48000c7020 con 0x7f480815ac60 2024-03-24T00:36:12.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.722+0000 7f480f000640 1 -- 172.21.15.40:0/2955708216 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f48080b0930 con 0x7f480815ac60 2024-03-24T00:36:12.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f4804c00640 1 -- 172.21.15.40:0/2955708216 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7f4800016520 con 0x7f480815ac60 2024-03-24T00:36:12.729 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-03-24T00:36:12.729 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw== 2024-03-24T00:36:12.729 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-03-24T00:36:12.729 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-03-24T00:36:12.730 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:36:12.730 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-03-24T00:36:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f480f000640 1 -- 172.21.15.40:0/2955708216 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f47e4067660 msgr2=0x7f47e4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f480f000640 1 --2- 172.21.15.40:0/2955708216 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f47e4067660 0x7f47e4069b20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f47f8007fd0 tx=0x7f47f8007f20 comp rx=0 tx=0).stop 2024-03-24T00:36:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f480f000640 1 -- 172.21.15.40:0/2955708216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480815ac60 msgr2=0x7f48080ce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f480f000640 1 --2- 172.21.15.40:0/2955708216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480815ac60 0x7f48080ce890 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f48000ac000 tx=0x7f4800019640 comp rx=0 tx=0).stop 2024-03-24T00:36:12.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f480f000640 1 -- 172.21.15.40:0/2955708216 shutdown_connections 2024-03-24T00:36:12.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f480f000640 1 --2- 172.21.15.40:0/2955708216 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f47e4067660 0x7f47e4069b20 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f480f000640 1 --2- 172.21.15.40:0/2955708216 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f480815ac60 0x7f48080ce890 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f480f000640 1 --2- 172.21.15.40:0/2955708216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48080b4840 0x7f48080ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f480f000640 1 --2- 172.21.15.40:0/2955708216 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f480806cb80 0x7f4808152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f480f000640 1 -- 172.21.15.40:0/2955708216 >> 172.21.15.40:0/2955708216 conn(0x7f48080b1660 msgr2=0x7f48080ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:12.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f480f000640 1 -- 172.21.15.40:0/2955708216 shutdown_connections 2024-03-24T00:36:12.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.726+0000 7f480f000640 1 -- 172.21.15.40:0/2955708216 wait complete. 2024-03-24T00:36:12.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:36:12.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.886+0000 7facd8a00640 1 Processor -- start 2024-03-24T00:36:12.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.886+0000 7facd8a00640 1 -- start start 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd8a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facd4141bc0 0x7facd40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd8a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7facd40ce630 0x7facd4071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd8a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd4070e00 con 0x7facd413ec80 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd8a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd40cee20 con 0x7facd40ce630 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd8a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd40cf100 con 0x7facd4141bc0 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd2800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7facd40ce630 0x7facd4071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd2800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7facd40ce630 0x7facd4071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39812/0 (socket says 172.21.15.40:39812) 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd2800640 1 -- 172.21.15.40:0/2819886872 learned_addr learned my addr 172.21.15.40:0/2819886872 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facc9400640 1 --2- 172.21.15.40:0/2819886872 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facd4141bc0 0x7facd40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd2800640 1 -- 172.21.15.40:0/2819886872 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facd4141bc0 msgr2=0x7facd40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd2800640 1 --2- 172.21.15.40:0/2819886872 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facd4141bc0 0x7facd40ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd2800640 1 -- 172.21.15.40:0/2819886872 >> v1:172.21.15.53:6789/0 conn(0x7facd413ec80 legacy=0x7facd41414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd2800640 1 -- 172.21.15.40:0/2819886872 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40cf3e0 con 0x7facd40ce630 2024-03-24T00:36:12.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facc9400640 1 --2- 172.21.15.40:0/2819886872 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facd4141bc0 0x7facd40ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:12.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.890+0000 7facd2800640 1 --2- 172.21.15.40:0/2819886872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facd40ce630 0x7facd4071300 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7faccc009e50 tx=0x7faccc01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7219b357a2355b13 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:12.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd1400640 1 -- 172.21.15.40:0/2819886872 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc0a5920 con 0x7facd40ce630 2024-03-24T00:36:12.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd1400640 1 -- 172.21.15.40:0/2819886872 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc0a5ac0 con 0x7facd40ce630 2024-03-24T00:36:12.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd1400640 1 -- 172.21.15.40:0/2819886872 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc0ad3f0 con 0x7facd40ce630 2024-03-24T00:36:12.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 -- 172.21.15.40:0/2819886872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facd40ce630 msgr2=0x7facd4071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 --2- 172.21.15.40:0/2819886872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facd40ce630 0x7facd4071300 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7faccc009e50 tx=0x7faccc01ca30 comp rx=0 tx=0).stop 2024-03-24T00:36:12.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 -- 172.21.15.40:0/2819886872 shutdown_connections 2024-03-24T00:36:12.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 --2- 172.21.15.40:0/2819886872 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facd4141bc0 0x7facd40ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 --2- 172.21.15.40:0/2819886872 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facd40ce630 0x7facd4071300 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 -- 172.21.15.40:0/2819886872 >> 172.21.15.40:0/2819886872 conn(0x7facd40b17b0 msgr2=0x7facd40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:12.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 -- 172.21.15.40:0/2819886872 shutdown_connections 2024-03-24T00:36:12.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 -- 172.21.15.40:0/2819886872 wait complete. 2024-03-24T00:36:12.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 Processor -- start 2024-03-24T00:36:12.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 -- start start 2024-03-24T00:36:12.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facd40ce630 0x7facd4156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facd413ec80 0x7facd4155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facd4141bc0 0x7facd4155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd40723e0 con 0x7facd40ce630 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd4072260 con 0x7facd413ec80 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd8a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd4072560 con 0x7facd4141bc0 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facd40ce630 0x7facd4156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facd40ce630 0x7facd4156a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39820/0 (socket says 172.21.15.40:39820) 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd1e00640 1 -- 172.21.15.40:0/294258063 learned_addr learned my addr 172.21.15.40:0/294258063 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facc9400640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facd413ec80 0x7facd4155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.894+0000 7facd2800640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facd4141bc0 0x7facd4155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7facd1e00640 1 -- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facd4141bc0 msgr2=0x7facd4155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7facd1e00640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facd4141bc0 0x7facd4155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7facd1e00640 1 -- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facd413ec80 msgr2=0x7facd4155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:12.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7facd1e00640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facd413ec80 0x7facd4155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:12.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7facd1e00640 1 -- 172.21.15.40:0/294258063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd413fcc0 con 0x7facd40ce630 2024-03-24T00:36:12.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7facd1e00640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facd40ce630 0x7facd4156a70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7facc400c420 tx=0x7facc400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:12.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7faccaa00640 1 -- 172.21.15.40:0/294258063 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc40052d0 con 0x7facd40ce630 2024-03-24T00:36:12.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7faccaa00640 1 -- 172.21.15.40:0/294258063 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc4005470 con 0x7facd40ce630 2024-03-24T00:36:12.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7faccaa00640 1 -- 172.21.15.40:0/294258063 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc4007260 con 0x7facd40ce630 2024-03-24T00:36:12.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7facd8a00640 1 -- 172.21.15.40:0/294258063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd413ffa0 con 0x7facd40ce630 2024-03-24T00:36:12.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7facd8a00640 1 -- 172.21.15.40:0/294258063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd406d930 con 0x7facd40ce630 2024-03-24T00:36:12.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7facd8a00640 1 -- 172.21.15.40:0/294258063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd4068170 con 0x7facd40ce630 2024-03-24T00:36:12.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7faccaa00640 1 -- 172.21.15.40:0/294258063 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7facc4007400 con 0x7facd40ce630 2024-03-24T00:36:12.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7faccaa00640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7facb0067660 0x7facb0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:12.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7faccaa00640 1 -- 172.21.15.40:0/294258063 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7facc4084d70 con 0x7facd40ce630 2024-03-24T00:36:12.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.898+0000 7facc9400640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7facb0067660 0x7facb0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:12.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.902+0000 7facc9400640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7facb0067660 0x7facb0069b20 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7facd4068be0 tx=0x7facbc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:12.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:12.902+0000 7faccaa00640 1 -- 172.21.15.40:0/294258063 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7facc4055300 con 0x7facd40ce630 2024-03-24T00:36:13.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.098+0000 7facd8a00640 1 -- 172.21.15.40:0/294258063 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7facd4072b90 con 0x7facd40ce630 2024-03-24T00:36:13.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.098+0000 7faccaa00640 1 -- 172.21.15.40:0/294258063 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4976 (secure 0 0 0) 0x7facc40593a0 con 0x7facd40ce630 2024-03-24T00:36:13.103 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:13.103 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":82,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1097448819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1097448819},{"type":"v1","addr":"172.21.15.40:6834","nonce":1097448819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":6404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/983819364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":983819364},{"type":"v1","addr":"172.21.15.53:6837","nonce":983819364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7180,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/2302548193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2302548193},{"type":"v1","addr":"172.21.15.40:6835","nonce":2302548193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:36:07.163876+0000","modified":"2024-03-24T00:36:10.190591+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7186},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7186":{"gid":7186,"name":"d","rank":0,"incarnation":78,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/1878888046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1878888046},{"type":"v1","addr":"172.21.15.53:6835","nonce":1878888046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7186,"qdb_cluster":[7186]},"id":6}]} 2024-03-24T00:36:13.104 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 82 2024-03-24T00:36:13.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.102+0000 7facd8a00640 1 -- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7facb0067660 msgr2=0x7facb0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.102+0000 7facd8a00640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7facb0067660 0x7facb0069b20 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7facd4068be0 tx=0x7facbc008040 comp rx=0 tx=0).stop 2024-03-24T00:36:13.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.102+0000 7facd8a00640 1 -- 172.21.15.40:0/294258063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facd40ce630 msgr2=0x7facd4156a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.102+0000 7facd8a00640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facd40ce630 0x7facd4156a70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7facc400c420 tx=0x7facc400c8e0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.102+0000 7facd8a00640 1 -- 172.21.15.40:0/294258063 shutdown_connections 2024-03-24T00:36:13.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.102+0000 7facd8a00640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7facb0067660 0x7facb0069b20 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.102+0000 7facd8a00640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facd4141bc0 0x7facd4155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.102+0000 7facd8a00640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facd413ec80 0x7facd4155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.102+0000 7facd8a00640 1 --2- 172.21.15.40:0/294258063 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facd40ce630 0x7facd4156a70 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.102+0000 7facd8a00640 1 -- 172.21.15.40:0/294258063 >> 172.21.15.40:0/294258063 conn(0x7facd40b17b0 msgr2=0x7facd415ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:13.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.102+0000 7facd8a00640 1 -- 172.21.15.40:0/294258063 shutdown_connections 2024-03-24T00:36:13.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.102+0000 7facd8a00640 1 -- 172.21.15.40:0/294258063 wait complete. 2024-03-24T00:36:13.526 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:36:07.163876+0000', 'modified': '2024-03-24T00:36:10.190591+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7186}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7186': {'gid': 7186, 'name': 'd', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.53:6835/1878888046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 1878888046}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 1878888046}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7186, 'qdb_cluster': [7186]} 2024-03-24T00:36:13.526 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:36:13.527 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T00:36:13.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff119600640 1 Processor -- start 2024-03-24T00:36:13.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff119600640 1 -- start start 2024-03-24T00:36:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff119600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 0x7ff1140b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff119600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff1140ccfa0 0x7ff1140cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff119600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff114070f60 con 0x7ff11406ea00 2024-03-24T00:36:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff119600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1140b4c00 con 0x7ff1140ccfa0 2024-03-24T00:36:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff119600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1140b3620 con 0x7ff11406e620 2024-03-24T00:36:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff10be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff11406ea00 0x7ff1140b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42992/0 (socket says 172.21.15.40:42992) 2024-03-24T00:36:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff10be00640 1 -- 172.21.15.40:0/3685313467 learned_addr learned my addr 172.21.15.40:0/3685313467 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff112a00640 1 --2- 172.21.15.40:0/3685313467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 0x7ff1140b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff113400640 1 --2- 172.21.15.40:0/3685313467 >> v2:172.21.15.53:3300/0 conn(0x7ff1140ccfa0 0x7ff1140cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff112000640 1 -- 172.21.15.40:0/3685313467 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753661238 0 0) 0x7ff114070f60 con 0x7ff11406ea00 2024-03-24T00:36:13.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff112a00640 1 -- 172.21.15.40:0/3685313467 >> v2:172.21.15.53:3300/0 conn(0x7ff1140ccfa0 msgr2=0x7ff1140cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:13.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff112a00640 1 --2- 172.21.15.40:0/3685313467 >> v2:172.21.15.53:3300/0 conn(0x7ff1140ccfa0 0x7ff1140cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff112a00640 1 -- 172.21.15.40:0/3685313467 >> v1:172.21.15.40:6789/0 conn(0x7ff11406ea00 legacy=0x7ff1140b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.666+0000 7ff112a00640 1 -- 172.21.15.40:0/3685313467 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1140b3900 con 0x7ff11406e620 2024-03-24T00:36:13.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.670+0000 7ff112a00640 1 --2- 172.21.15.40:0/3685313467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 0x7ff1140b46c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff1040097c0 tx=0x7ff10401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5039b857ba0568ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:13.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.670+0000 7ff112000640 1 -- 172.21.15.40:0/3685313467 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040b0070 con 0x7ff11406e620 2024-03-24T00:36:13.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.670+0000 7ff112000640 1 -- 172.21.15.40:0/3685313467 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1040ab600 con 0x7ff11406e620 2024-03-24T00:36:13.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.670+0000 7ff112000640 1 -- 172.21.15.40:0/3685313467 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040ab9d0 con 0x7ff11406e620 2024-03-24T00:36:13.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.670+0000 7ff119600640 1 -- 172.21.15.40:0/3685313467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 msgr2=0x7ff1140b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.670+0000 7ff119600640 1 --2- 172.21.15.40:0/3685313467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 0x7ff1140b46c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff1040097c0 tx=0x7ff10401c3e0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.670+0000 7ff119600640 1 -- 172.21.15.40:0/3685313467 shutdown_connections 2024-03-24T00:36:13.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.670+0000 7ff119600640 1 --2- 172.21.15.40:0/3685313467 >> v2:172.21.15.53:3300/0 conn(0x7ff1140ccfa0 0x7ff1140cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.670+0000 7ff119600640 1 --2- 172.21.15.40:0/3685313467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 0x7ff1140b46c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.670+0000 7ff119600640 1 -- 172.21.15.40:0/3685313467 >> 172.21.15.40:0/3685313467 conn(0x7ff1140b17b0 msgr2=0x7ff1140b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:13.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.670+0000 7ff119600640 1 -- 172.21.15.40:0/3685313467 shutdown_connections 2024-03-24T00:36:13.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.670+0000 7ff119600640 1 -- 172.21.15.40:0/3685313467 wait complete. 2024-03-24T00:36:13.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff119600640 1 Processor -- start 2024-03-24T00:36:13.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff119600640 1 -- start start 2024-03-24T00:36:13.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff119600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 0x7ff11413a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff119600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff11406ea00 0x7ff11413f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff119600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1140ccfa0 0x7ff114165080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff119600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff114144920 con 0x7ff1140ccfa0 2024-03-24T00:36:13.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff119600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11406ee20 con 0x7ff11406ea00 2024-03-24T00:36:13.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff119600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1141f7dc0 con 0x7ff11406e620 2024-03-24T00:36:13.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff112a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 0x7ff11413a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff113400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1140ccfa0 0x7ff114165080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff112a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 0x7ff11413a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:46676/0 (socket says 172.21.15.40:46676) 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff112a00640 1 -- 172.21.15.40:0/666947036 learned_addr learned my addr 172.21.15.40:0/666947036 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff10be00640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff11406ea00 0x7ff11413f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff112a00640 1 -- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff11406ea00 msgr2=0x7ff11413f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff112a00640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff11406ea00 0x7ff11413f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff112a00640 1 -- 172.21.15.40:0/666947036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1140ccfa0 msgr2=0x7ff114165080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff112a00640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1140ccfa0 0x7ff114165080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff112a00640 1 -- 172.21.15.40:0/666947036 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1141f80a0 con 0x7ff11406e620 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff113400640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1140ccfa0 0x7ff114165080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff112a00640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 0x7ff11413a780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff1040ae040 tx=0x7ff10401ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff10aa00640 1 -- 172.21.15.40:0/666947036 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040b0050 con 0x7ff11406e620 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff10aa00640 1 -- 172.21.15.40:0/666947036 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1040bc070 con 0x7ff11406e620 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff10aa00640 1 -- 172.21.15.40:0/666947036 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040b5870 con 0x7ff11406e620 2024-03-24T00:36:13.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff119600640 1 -- 172.21.15.40:0/666947036 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1141f8380 con 0x7ff11406e620 2024-03-24T00:36:13.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.674+0000 7ff119600640 1 -- 172.21.15.40:0/666947036 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1141f7ac0 con 0x7ff11406e620 2024-03-24T00:36:13.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.678+0000 7ff10aa00640 1 -- 172.21.15.40:0/666947036 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff1040b5a10 con 0x7ff11406e620 2024-03-24T00:36:13.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.678+0000 7ff10aa00640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff0f0067660 0x7ff0f0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.678+0000 7ff10aa00640 1 -- 172.21.15.40:0/666947036 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1041200f0 con 0x7ff11406e620 2024-03-24T00:36:13.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.678+0000 7ff10be00640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff0f0067660 0x7ff0f0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.678+0000 7ff11cb8f000 1 Processor -- start 2024-03-24T00:36:13.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.678+0000 7ff11cb8f000 1 -- start start 2024-03-24T00:36:13.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.678+0000 7ff10be00640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff0f0067660 0x7ff0f0069b20 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff11414f0f0 tx=0x7ff0fc007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:13.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d18920 0x55a3a0d18d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d19240 0x55a3a0cceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0cceff0 0x55a3a0ccf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3a0b1c680 con 0x55a3a0d19240 2024-03-24T00:36:13.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3a0aa3b00 con 0x55a3a0d18920 2024-03-24T00:36:13.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a3a0b18720 con 0x55a3a0cceff0 2024-03-24T00:36:13.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff10be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d19240 0x55a3a0cceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff112a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d18920 0x55a3a0d18d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff112a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d18920 0x55a3a0d18d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:53918/0 (socket says 172.21.15.40:53918) 2024-03-24T00:36:13.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff112a00640 1 -- 172.21.15.40:0/2209735883 learned_addr learned my addr 172.21.15.40:0/2209735883 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:13.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff113400640 1 --2- 172.21.15.40:0/2209735883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0cceff0 0x55a3a0ccf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff112a00640 1 -- 172.21.15.40:0/2209735883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0cceff0 msgr2=0x55a3a0ccf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:13.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff112a00640 1 --2- 172.21.15.40:0/2209735883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0cceff0 0x55a3a0ccf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff112a00640 1 -- 172.21.15.40:0/2209735883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d19240 msgr2=0x55a3a0cceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:13.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff112a00640 1 --2- 172.21.15.40:0/2209735883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d19240 0x55a3a0cceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff112a00640 1 -- 172.21.15.40:0/2209735883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3a0af4940 con 0x55a3a0d18920 2024-03-24T00:36:13.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff113400640 1 --2- 172.21.15.40:0/2209735883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0cceff0 0x55a3a0ccf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:13.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff112a00640 1 --2- 172.21.15.40:0/2209735883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d18920 0x55a3a0d18d00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff1040f06a0 tx=0x7ff1040b3000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fafdb44660a33ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:13.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff103400640 1 -- 172.21.15.40:0/2209735883 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040f47b0 con 0x55a3a0d18920 2024-03-24T00:36:13.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff103400640 1 -- 172.21.15.40:0/2209735883 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1040b1040 con 0x55a3a0d18920 2024-03-24T00:36:13.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff103400640 1 -- 172.21.15.40:0/2209735883 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040f8e00 con 0x55a3a0d18920 2024-03-24T00:36:13.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 -- 172.21.15.40:0/2209735883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d18920 msgr2=0x55a3a0d18d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/2209735883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d18920 0x55a3a0d18d00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff1040f06a0 tx=0x7ff1040b3000 comp rx=0 tx=0).stop 2024-03-24T00:36:13.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 -- 172.21.15.40:0/2209735883 shutdown_connections 2024-03-24T00:36:13.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/2209735883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0cceff0 0x55a3a0ccf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/2209735883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d19240 0x55a3a0cceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/2209735883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d18920 0x55a3a0d18d00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 -- 172.21.15.40:0/2209735883 >> 172.21.15.40:0/2209735883 conn(0x55a3a0ccafa0 msgr2=0x55a3a0ccd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:13.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 -- 172.21.15.40:0/2209735883 shutdown_connections 2024-03-24T00:36:13.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 -- 172.21.15.40:0/2209735883 wait complete. 2024-03-24T00:36:13.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 -- start start 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 Processor -- start 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0cceff0 0x55a3a0cfdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d18920 0x55a3a0cfe280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d19240 0x55a3a0cfe7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3a0b1b990 con 0x55a3a0d18920 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3a0ad6ae0 con 0x55a3a0d19240 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a3a0ab4750 con 0x55a3a0cceff0 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.682+0000 7ff11cb8f000 10 client.0 _ll_register_callbacks cb 0x55a3a0cb9ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff10be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d18920 0x55a3a0cfe280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff112a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0cceff0 0x55a3a0cfdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff10be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d18920 0x55a3a0cfe280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39848/0 (socket says 172.21.15.40:39848) 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff10be00640 1 -- 172.21.15.40:0/3903523891 learned_addr learned my addr 172.21.15.40:0/3903523891 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:13.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff113400640 1 --2- 172.21.15.40:0/3903523891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d19240 0x55a3a0cfe7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff10be00640 1 -- 172.21.15.40:0/3903523891 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0cceff0 msgr2=0x55a3a0cfdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff10be00640 1 --2- 172.21.15.40:0/3903523891 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0cceff0 0x55a3a0cfdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff10be00640 1 -- 172.21.15.40:0/3903523891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d19240 msgr2=0x55a3a0cfe7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff10be00640 1 --2- 172.21.15.40:0/3903523891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d19240 0x55a3a0cfe7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff10be00640 1 -- 172.21.15.40:0/3903523891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a3a0b61d80 con 0x55a3a0d18920 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff10be00640 1 --2- 172.21.15.40:0/3903523891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d18920 0x55a3a0cfe280 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc0136a0 tx=0x7ff0fc000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b803e110ad4d8bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff0f7e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff0f7e00640 1 -- 172.21.15.40:0/3903523891 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc020700 con 0x55a3a0d18920 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff0f7e00640 1 -- 172.21.15.40:0/3903523891 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0fc0208a0 con 0x55a3a0d18920 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 -- 172.21.15.40:0/3903523891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1040b1040 con 0x55a3a0d18920 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff0f7e00640 1 -- 172.21.15.40:0/3903523891 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0fc020ae0 con 0x55a3a0d18920 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 10 client.7572 fetch_fsmap learned FSMap version 82 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 -- 172.21.15.40:0/3903523891 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55a3a0aa3d10 con 0x55a3a0d18920 2024-03-24T00:36:13.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff0f7e00640 1 -- 172.21.15.40:0/3903523891 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1040b1040 con 0x55a3a0d18920 2024-03-24T00:36:13.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff0f7e00640 1 -- 172.21.15.40:0/3903523891 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 82) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff0fc031020 con 0x55a3a0d18920 2024-03-24T00:36:13.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 10 client.7572 fetch_fsmap finished waiting for FSMap version 82 2024-03-24T00:36:13.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 10 client.7572 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:36:13.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 20 client.7572 populate_metadata read hostname 'smithi040' 2024-03-24T00:36:13.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/3903523891 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55a3a0cff090 0x55a3a0d2aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 4 client.7572 mds_command: new command op to 7186 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:36:13.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 -- 172.21.15.40:0/3903523891 --> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a3a0ab7b60 con 0x55a3a0cff090 2024-03-24T00:36:13.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff112a00640 1 --2- 172.21.15.40:0/3903523891 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55a3a0cff090 0x55a3a0d2aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff112a00640 1 --2- 172.21.15.40:0/3903523891 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55a3a0cff090 0x55a3a0d2aea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9df233249857de3 server_cookie=38174b6fa256db36 in_seq=0 out_seq=0 2024-03-24T00:36:13.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff0f7e00640 10 client.7572 ms_handle_connect on v2:172.21.15.53:6834/1878888046 2024-03-24T00:36:13.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff0f7e00640 1 -- 172.21.15.40:0/3903523891 <== mds.0 v2:172.21.15.53:6834/1878888046 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a3a0ab7b60 con 0x55a3a0cff090 2024-03-24T00:36:13.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff0f7e00640 10 client.7572 handle_command_reply: tid=0 2024-03-24T00:36:13.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 client.7572 shutdown 2024-03-24T00:36:13.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 -- 172.21.15.40:0/3903523891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d18920 msgr2=0x55a3a0cfe280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/3903523891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d18920 0x55a3a0cfe280 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc0136a0 tx=0x7ff0fc000ee0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 -- 172.21.15.40:0/3903523891 shutdown_connections 2024-03-24T00:36:13.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/3903523891 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55a3a0cff090 0x55a3a0d2aea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/3903523891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d19240 0x55a3a0cfe7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/3903523891 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d18920 0x55a3a0cfe280 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/3903523891 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0cceff0 0x55a3a0cfdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff0f7e00640 0 client.7572 ms_handle_reset on v2:172.21.15.53:6834/1878888046 2024-03-24T00:36:13.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 -- 172.21.15.40:0/3903523891 >> 172.21.15.40:0/3903523891 conn(0x55a3a0ccafa0 msgr2=0x55a3a0ccdc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:13.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 -- 172.21.15.40:0/3903523891 shutdown_connections 2024-03-24T00:36:13.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.686+0000 7ff11cb8f000 1 -- 172.21.15.40:0/3903523891 wait complete. 2024-03-24T00:36:13.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.690+0000 7ff11cb8f000 20 client.7572 trim_cache size 0 max 16384 2024-03-24T00:36:13.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 Processor -- start 2024-03-24T00:36:13.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 -- start start 2024-03-24T00:36:13.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0cceff0 0x55a3a0cfbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0d18920 0x55a3a0d17530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d19240 0x55a3a0d17a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff114203190 con 0x55a3a0d19240 2024-03-24T00:36:13.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff114202380 con 0x55a3a0cceff0 2024-03-24T00:36:13.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1140009b0 con 0x55a3a0d18920 2024-03-24T00:36:13.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff112a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0cceff0 0x55a3a0cfbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff112a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0cceff0 0x55a3a0cfbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:53938/0 (socket says 172.21.15.40:53938) 2024-03-24T00:36:13.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff112a00640 1 -- 172.21.15.40:0/141363511 learned_addr learned my addr 172.21.15.40:0/141363511 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:13.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff10be00640 1 --2- 172.21.15.40:0/141363511 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0d18920 0x55a3a0d17530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff113400640 1 --2- 172.21.15.40:0/141363511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d19240 0x55a3a0d17a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff112a00640 1 -- 172.21.15.40:0/141363511 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0d18920 msgr2=0x55a3a0d17530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:13.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff112a00640 1 --2- 172.21.15.40:0/141363511 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0d18920 0x55a3a0d17530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff112a00640 1 -- 172.21.15.40:0/141363511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d19240 msgr2=0x55a3a0d17a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:13.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff112a00640 1 --2- 172.21.15.40:0/141363511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d19240 0x55a3a0d17a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff112a00640 1 -- 172.21.15.40:0/141363511 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3a0ae9540 con 0x55a3a0cceff0 2024-03-24T00:36:13.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff112a00640 1 --2- 172.21.15.40:0/141363511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0cceff0 0x55a3a0cfbef0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff104000c10 tx=0x7ff104105000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a41b7b17800b375 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:13.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff102a00640 1 -- 172.21.15.40:0/141363511 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040f0070 con 0x55a3a0cceff0 2024-03-24T00:36:13.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff102a00640 1 -- 172.21.15.40:0/141363511 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1040f47b0 con 0x55a3a0cceff0 2024-03-24T00:36:13.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff102a00640 1 -- 172.21.15.40:0/141363511 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040eea10 con 0x55a3a0cceff0 2024-03-24T00:36:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 -- 172.21.15.40:0/141363511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0cceff0 msgr2=0x55a3a0cfbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/141363511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0cceff0 0x55a3a0cfbef0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff104000c10 tx=0x7ff104105000 comp rx=0 tx=0).stop 2024-03-24T00:36:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 -- 172.21.15.40:0/141363511 shutdown_connections 2024-03-24T00:36:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/141363511 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0d19240 0x55a3a0d17a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/141363511 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0d18920 0x55a3a0d17530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/141363511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0cceff0 0x55a3a0cfbef0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 -- 172.21.15.40:0/141363511 >> 172.21.15.40:0/141363511 conn(0x55a3a0ccafa0 msgr2=0x55a3a0cfb5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 -- 172.21.15.40:0/141363511 shutdown_connections 2024-03-24T00:36:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 -- 172.21.15.40:0/141363511 wait complete. 2024-03-24T00:36:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.706+0000 7ff11cb8f000 1 -- start start 2024-03-24T00:36:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 Processor -- start 2024-03-24T00:36:13.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0cceff0 0x55a3a0d17e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d18920 0x55a3a0d2bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0d19240 0x55a3a0d2c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3a0ac4a80 con 0x55a3a0cceff0 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff114164b80 con 0x55a3a0d18920 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a3a0bd9700 con 0x55a3a0d19240 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 10 client.0 _ll_register_callbacks cb 0x7ff0fc004c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff112a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0cceff0 0x55a3a0d17e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff112a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0cceff0 0x55a3a0d17e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39874/0 (socket says 172.21.15.40:39874) 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff112a00640 1 -- 172.21.15.40:0/645062661 learned_addr learned my addr 172.21.15.40:0/645062661 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff10be00640 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d18920 0x55a3a0d2bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff113400640 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0d19240 0x55a3a0d2c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff112a00640 1 -- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0d19240 msgr2=0x55a3a0d2c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff112a00640 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0d19240 0x55a3a0d2c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff112a00640 1 -- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d18920 msgr2=0x55a3a0d2bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff112a00640 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d18920 0x55a3a0d2bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff112a00640 1 -- 172.21.15.40:0/645062661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a3a0ab04b0 con 0x55a3a0cceff0 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff113400640 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0d19240 0x55a3a0d2c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff112a00640 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0cceff0 0x55a3a0d17e30 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7ff10401aa90 tx=0x7ff104019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c3b8da942b938f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff0f7e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff0f7e00640 1 -- 172.21.15.40:0/645062661 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1040f0070 con 0x55a3a0cceff0 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff0f7e00640 1 -- 172.21.15.40:0/645062661 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1040ee7b0 con 0x55a3a0cceff0 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 -- 172.21.15.40:0/645062661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1040f47b0 con 0x55a3a0cceff0 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff0f7e00640 1 -- 172.21.15.40:0/645062661 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1040eea10 con 0x55a3a0cceff0 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 10 client.7578 fetch_fsmap learned FSMap version 82 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 -- 172.21.15.40:0/645062661 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55a3a0ac32f0 con 0x55a3a0cceff0 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff0f7e00640 1 -- 172.21.15.40:0/645062661 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1040f47b0 con 0x55a3a0cceff0 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff0f7e00640 1 -- 172.21.15.40:0/645062661 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 82) v1 ==== 3314+0+0 (secure 0 0 0) 0x7ff1040ecd80 con 0x55a3a0cceff0 2024-03-24T00:36:13.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 10 client.7578 fetch_fsmap finished waiting for FSMap version 82 2024-03-24T00:36:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 10 client.7578 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:36:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 20 client.7578 populate_metadata read hostname 'smithi040' 2024-03-24T00:36:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55a3a0d7d7d0 0x55a3a0d7dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 4 client.7578 mds_command: new command op to 7186 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:36:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 -- 172.21.15.40:0/645062661 --> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a3a0b29bc0 con 0x55a3a0d7d7d0 2024-03-24T00:36:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff10be00640 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55a3a0d7d7d0 0x55a3a0d7dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff10be00640 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55a3a0d7d7d0 0x55a3a0d7dbb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5145fa064d2cf358 server_cookie=75453fc7d5a98859 in_seq=0 out_seq=0 2024-03-24T00:36:13.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff0f7e00640 10 client.7578 ms_handle_connect on v2:172.21.15.53:6834/1878888046 2024-03-24T00:36:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff0f7e00640 1 -- 172.21.15.40:0/645062661 <== mds.0 v2:172.21.15.53:6834/1878888046 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55a3a0b29bc0 con 0x55a3a0d7d7d0 2024-03-24T00:36:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff0f7e00640 10 client.7578 handle_command_reply: tid=0 2024-03-24T00:36:13.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 client.7578 shutdown 2024-03-24T00:36:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 -- 172.21.15.40:0/645062661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0cceff0 msgr2=0x55a3a0d17e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.710+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0cceff0 0x55a3a0d17e30 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7ff10401aa90 tx=0x7ff104019180 comp rx=0 tx=0).stop 2024-03-24T00:36:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff11cb8f000 1 -- 172.21.15.40:0/645062661 shutdown_connections 2024-03-24T00:36:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55a3a0d7d7d0 0x55a3a0d7dbb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3a0d19240 0x55a3a0d2c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3a0d18920 0x55a3a0d2bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff0f7e00640 0 client.7578 ms_handle_reset on v2:172.21.15.53:6834/1878888046 2024-03-24T00:36:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff11cb8f000 1 --2- 172.21.15.40:0/645062661 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3a0cceff0 0x55a3a0d17e30 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff11cb8f000 1 -- 172.21.15.40:0/645062661 >> 172.21.15.40:0/645062661 conn(0x55a3a0ccafa0 msgr2=0x55a3a0cfbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff11cb8f000 1 -- 172.21.15.40:0/645062661 shutdown_connections 2024-03-24T00:36:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff11cb8f000 1 -- 172.21.15.40:0/645062661 wait complete. 2024-03-24T00:36:13.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff11cb8f000 20 client.7578 trim_cache size 0 max 16384 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 7186, 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 5.9199243360000002, 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 82, 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 83, 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 17.854758681 2024-03-24T00:36:13.721 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:36:13.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff119600640 1 -- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff0f0067660 msgr2=0x7ff0f0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff119600640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff0f0067660 0x7ff0f0069b20 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff11414f0f0 tx=0x7ff0fc007d60 comp rx=0 tx=0).stop 2024-03-24T00:36:13.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff119600640 1 -- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 msgr2=0x7ff11413a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff119600640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 0x7ff11413a780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff1040ae040 tx=0x7ff10401ac20 comp rx=0 tx=0).stop 2024-03-24T00:36:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff119600640 1 -- 172.21.15.40:0/666947036 shutdown_connections 2024-03-24T00:36:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff119600640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff0f0067660 0x7ff0f0069b20 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff119600640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1140ccfa0 0x7ff114165080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff119600640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff11406ea00 0x7ff11413f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff119600640 1 --2- 172.21.15.40:0/666947036 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff11406e620 0x7ff11413a780 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff119600640 1 -- 172.21.15.40:0/666947036 >> 172.21.15.40:0/666947036 conn(0x7ff1140b17b0 msgr2=0x7ff1140cde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff119600640 1 -- 172.21.15.40:0/666947036 shutdown_connections 2024-03-24T00:36:13.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.714+0000 7ff119600640 1 -- 172.21.15.40:0/666947036 wait complete. 2024-03-24T00:36:13.735 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:36:13.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7f2a00640 1 Processor -- start 2024-03-24T00:36:13.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7f2a00640 1 -- start start 2024-03-24T00:36:13.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7f2a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff7ec13ec80 0x7ff7ec1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7f2a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff7ec1419f0 0x7ff7ec0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7f2a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7ec070e00 con 0x7ff7ec0ce630 2024-03-24T00:36:13.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7f2a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7ec141e50 con 0x7ff7ec1419f0 2024-03-24T00:36:13.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7f2a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7ec0cee20 con 0x7ff7ec13ec80 2024-03-24T00:36:13.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7f0a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff7ec0ce630 0x7ff7ec15f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:43008/0 (socket says 172.21.15.40:43008) 2024-03-24T00:36:13.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7f0a00640 1 -- 172.21.15.40:0/247203694 learned_addr learned my addr 172.21.15.40:0/247203694 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:13.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7ebe00640 1 --2- 172.21.15.40:0/247203694 >> v2:172.21.15.53:3300/0 conn(0x7ff7ec13ec80 0x7ff7ec1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7e3e00640 1 --2- 172.21.15.40:0/247203694 >> v2:172.21.15.40:3300/0 conn(0x7ff7ec1419f0 0x7ff7ec0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7eb400640 1 -- 172.21.15.40:0/247203694 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442682001 0 0) 0x7ff7ec070e00 con 0x7ff7ec0ce630 2024-03-24T00:36:13.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7eb400640 1 -- 172.21.15.40:0/247203694 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7d0003530 con 0x7ff7ec0ce630 2024-03-24T00:36:13.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7e3e00640 1 -- 172.21.15.40:0/247203694 >> v2:172.21.15.53:3300/0 conn(0x7ff7ec13ec80 msgr2=0x7ff7ec1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:13.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7e3e00640 1 --2- 172.21.15.40:0/247203694 >> v2:172.21.15.53:3300/0 conn(0x7ff7ec13ec80 0x7ff7ec1414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7e3e00640 1 -- 172.21.15.40:0/247203694 >> v1:172.21.15.40:6789/0 conn(0x7ff7ec0ce630 legacy=0x7ff7ec15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7e3e00640 1 -- 172.21.15.40:0/247203694 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ec0cf100 con 0x7ff7ec1419f0 2024-03-24T00:36:13.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.874+0000 7ff7ebe00640 1 --2- 172.21.15.40:0/247203694 >> v2:172.21.15.53:3300/0 conn(0x7ff7ec13ec80 0x7ff7ec1414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:13.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.878+0000 7ff7e3e00640 1 --2- 172.21.15.40:0/247203694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7ec1419f0 0x7ff7ec0ce0f0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec06b170 tx=0x7ff7d401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1058a61017e594 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:13.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.878+0000 7ff7eb400640 1 -- 172.21.15.40:0/247203694 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d40aa600 con 0x7ff7ec1419f0 2024-03-24T00:36:13.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.878+0000 7ff7eb400640 1 -- 172.21.15.40:0/247203694 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d40af070 con 0x7ff7ec1419f0 2024-03-24T00:36:13.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.878+0000 7ff7eb400640 1 -- 172.21.15.40:0/247203694 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d40aa9c0 con 0x7ff7ec1419f0 2024-03-24T00:36:13.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.878+0000 7ff7f2a00640 1 -- 172.21.15.40:0/247203694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7ec1419f0 msgr2=0x7ff7ec0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.878+0000 7ff7f2a00640 1 --2- 172.21.15.40:0/247203694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7ec1419f0 0x7ff7ec0ce0f0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec06b170 tx=0x7ff7d401c480 comp rx=0 tx=0).stop 2024-03-24T00:36:13.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.878+0000 7ff7f2a00640 1 -- 172.21.15.40:0/247203694 shutdown_connections 2024-03-24T00:36:13.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.878+0000 7ff7f2a00640 1 --2- 172.21.15.40:0/247203694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7ec1419f0 0x7ff7ec0ce0f0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.878+0000 7ff7f2a00640 1 --2- 172.21.15.40:0/247203694 >> v2:172.21.15.53:3300/0 conn(0x7ff7ec13ec80 0x7ff7ec1414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.878+0000 7ff7f2a00640 1 -- 172.21.15.40:0/247203694 >> 172.21.15.40:0/247203694 conn(0x7ff7ec0b17b0 msgr2=0x7ff7ec0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:13.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.878+0000 7ff7f2a00640 1 -- 172.21.15.40:0/247203694 shutdown_connections 2024-03-24T00:36:13.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.878+0000 7ff7f2a00640 1 -- 172.21.15.40:0/247203694 wait complete. 2024-03-24T00:36:13.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7f2a00640 1 Processor -- start 2024-03-24T00:36:13.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7f2a00640 1 -- start start 2024-03-24T00:36:13.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7f2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7ec0ce630 0x7ff7ec06df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7f2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7ec13ec80 0x7ff7ec15c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7f2a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff7ec1419f0 0x7ff7ec15c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7f2a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec168ec0 con 0x7ff7ec13ec80 2024-03-24T00:36:13.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7f2a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec168d40 con 0x7ff7ec0ce630 2024-03-24T00:36:13.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7f2a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7ec169040 con 0x7ff7ec1419f0 2024-03-24T00:36:13.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7ec13ec80 0x7ff7ec15c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7e3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7ec13ec80 0x7ff7ec15c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39894/0 (socket says 172.21.15.40:39894) 2024-03-24T00:36:13.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7e3e00640 1 -- 172.21.15.40:0/2211485572 learned_addr learned my addr 172.21.15.40:0/2211485572 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:13.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7f0a00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff7ec1419f0 0x7ff7ec15c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7ebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7ec0ce630 0x7ff7ec06df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7e3e00640 1 -- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff7ec1419f0 msgr2=0x7ff7ec15c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7e3e00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff7ec1419f0 0x7ff7ec15c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7e3e00640 1 -- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7ec0ce630 msgr2=0x7ff7ec06df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:13.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7e3e00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7ec0ce630 0x7ff7ec06df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:13.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7e3e00640 1 -- 172.21.15.40:0/2211485572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ec13a730 con 0x7ff7ec13ec80 2024-03-24T00:36:13.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7e3e00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7ec13ec80 0x7ff7ec15c450 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4019090 tx=0x7ff7d4019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:13.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2211485572 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d40aabc0 con 0x7ff7ec13ec80 2024-03-24T00:36:13.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2211485572 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7d40aad60 con 0x7ff7ec13ec80 2024-03-24T00:36:13.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7f2a00640 1 -- 172.21.15.40:0/2211485572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7ec13aa10 con 0x7ff7ec13ec80 2024-03-24T00:36:13.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2211485572 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7d40be650 con 0x7ff7ec13ec80 2024-03-24T00:36:13.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7f2a00640 1 -- 172.21.15.40:0/2211485572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7ec13afd0 con 0x7ff7ec13ec80 2024-03-24T00:36:13.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.882+0000 7ff7f2a00640 1 -- 172.21.15.40:0/2211485572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7ec068170 con 0x7ff7ec13ec80 2024-03-24T00:36:13.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.886+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2211485572 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff7d40a9070 con 0x7ff7ec13ec80 2024-03-24T00:36:13.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.886+0000 7ff7e8c00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff7c8067660 0x7ff7c8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:13.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.886+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2211485572 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7d411f460 con 0x7ff7ec13ec80 2024-03-24T00:36:13.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.886+0000 7ff7ebe00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff7c8067660 0x7ff7c8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:13.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.886+0000 7ff7ebe00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff7c8067660 0x7ff7c8069b20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec06eab0 tx=0x7ff7dc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:13.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:13.890+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2211485572 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff7d40c5070 con 0x7ff7ec13ec80 2024-03-24T00:36:14.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.090+0000 7ff7f2a00640 1 -- 172.21.15.40:0/2211485572 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff7ec072b90 con 0x7ff7ec13ec80 2024-03-24T00:36:14.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.090+0000 7ff7e8c00640 1 -- 172.21.15.40:0/2211485572 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v82) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff7d4016070 con 0x7ff7ec13ec80 2024-03-24T00:36:14.094 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:36:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.094+0000 7ff7f2a00640 1 -- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff7c8067660 msgr2=0x7ff7c8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.094+0000 7ff7f2a00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff7c8067660 0x7ff7c8069b20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec06eab0 tx=0x7ff7dc009190 comp rx=0 tx=0).stop 2024-03-24T00:36:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.094+0000 7ff7f2a00640 1 -- 172.21.15.40:0/2211485572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7ec13ec80 msgr2=0x7ff7ec15c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.094+0000 7ff7f2a00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7ec13ec80 0x7ff7ec15c450 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4019090 tx=0x7ff7d4019220 comp rx=0 tx=0).stop 2024-03-24T00:36:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.094+0000 7ff7f2a00640 1 -- 172.21.15.40:0/2211485572 shutdown_connections 2024-03-24T00:36:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.094+0000 7ff7f2a00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff7c8067660 0x7ff7c8069b20 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.094+0000 7ff7f2a00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff7ec1419f0 0x7ff7ec15c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.094+0000 7ff7f2a00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff7ec13ec80 0x7ff7ec15c450 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.094+0000 7ff7f2a00640 1 --2- 172.21.15.40:0/2211485572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7ec0ce630 0x7ff7ec06df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.094+0000 7ff7f2a00640 1 -- 172.21.15.40:0/2211485572 >> 172.21.15.40:0/2211485572 conn(0x7ff7ec0b17b0 msgr2=0x7ff7ec166620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.094+0000 7ff7f2a00640 1 -- 172.21.15.40:0/2211485572 shutdown_connections 2024-03-24T00:36:14.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14.094+0000 7ff7f2a00640 1 -- 172.21.15.40:0/2211485572 wait complete. 2024-03-24T00:36:14.115 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:36:14.115 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:36:14.115 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-03-24T00:36:14.115 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:36:14.115 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:36:14.116 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:36:14.116 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:36:14.116 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:36:14.116 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:36:14.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:14.116 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:36:14.126 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42176sec preferred_lft 42176sec 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-03-24T00:36:14.127 INFO:teuthology.orchestra.run.smithi040.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 02:42:9e:ab:82:f9 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-24T00:36:14.128 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:36:14.129 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:36:14.129 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:36:14.129 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-24T00:36:14.129 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:36:14.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:14.129 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:36:14.189 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-03-24T00:36:14.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:14.191 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-03-24T00:36:14.245 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-03-24T00:36:14.245 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:36:14.245 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-24T00:36:14.246 INFO:teuthology.orchestra.run.smithi040.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-24T00:36:14.246 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-24T00:36:14.246 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:36:14.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:14.246 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:36:14.246 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:36:14.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T00:36:14.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T00:36:14.247 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T00:36:14.247 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:36:14.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:14.337 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-24T00:36:14.345 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:36:14.345 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:36:14.346 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:36:14.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:14.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-03-24T00:36:14.407 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T00:36:14.408 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:36:14.408 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:36:14.460 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:36:14.460 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:36:14.460 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:36:14.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:36:14.576 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:36:14.629 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:36:14.629 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:36:14.629 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:36:14.629 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:36:14.629 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:36:14.629 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:36:14.629 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:36:14.629 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:36:14.629 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:36:14.629 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:36:14.629 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:36:14.629 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:36:14.630 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:36:14.631 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:36:14.631 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:36:14.631 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:36:14.631 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:36:14.631 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:36:14.631 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:36:14.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:14.632 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:36:14.683 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T00:36:14.683 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T00:36:14.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:36:14.753 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:36:14.804 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:36:14.805 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:36:14.806 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:36:14.807 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:36:14.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:14.808 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:36:14.828 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:2024-03-24T00:36:14.826+0000 7f6c0e847580 -1 init, newargv = 0x555f9274e650 newargc=15 2024-03-24T00:36:14.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[46469]: starting ceph client 2024-03-24T00:36:14.835 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[46469]: starting fuse 2024-03-24T00:36:14.837 INFO:teuthology.orchestra.run.smithi040.stdout:43 2024-03-24T00:36:14.838 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-24T00:36:14.838 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:36:14.838 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-03-24T00:36:14.838 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-03-24T00:36:14.838 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-03-24T00:36:14.838 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-03-24T00:36:14.838 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:36:14.838 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-03-24T00:36:14.838 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T00:36:14.838 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-03-24T00:36:14.838 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T00:36:14.838 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:36:14.838 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:36:14.839 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:36:14.840 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.0")) 2024-03-24T00:36:14.840 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-03-24T00:36:14.962 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.0.46469.asok 2024-03-24T00:36:14.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:36:14.973 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.46469.asok 2024-03-24T00:36:14.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:14.974 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.46469.asok status 2024-03-24T00:36:15.147 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:36:15.147 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-03-24T00:36:15.147 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:36:15.147 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:36:15.147 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "0", 2024-03-24T00:36:15.147 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-03-24T00:36:15.147 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T00:36:15.147 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "46469", 2024-03-24T00:36:15.147 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-03-24T00:36:15.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 7593, 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 7593 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1850312894 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:15.148 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 1850312894 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.7593 192.168.0.1:0/1850312894", 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "192.168.0.1:0/1850312894", 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 82, 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 83, 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-03-24T00:36:15.149 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:36:15.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:15.162 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-03-24T00:36:15.162 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T00:36:15.172 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-03-24T00:36:15.172 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T00:36:15.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:15.172 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:36:15.239 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random True 2024-03-24T00:36:15.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47ae000640 1 Processor -- start 2024-03-24T00:36:15.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47ae000640 1 -- start start 2024-03-24T00:36:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47ae000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47a806cb80 0x7f47a80b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47ae000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f47a80b4840 0x7f47a8071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47ae000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47a80b3260 con 0x7f47a815ac60 2024-03-24T00:36:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47ae000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47a80b3e70 con 0x7f47a80b4840 2024-03-24T00:36:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47ae000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a80b3540 con 0x7f47a806cb80 2024-03-24T00:36:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47a7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f47a80b4840 0x7f47a8071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47a7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f47a80b4840 0x7f47a8071b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39908/0 (socket says 172.21.15.40:39908) 2024-03-24T00:36:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47a7e00640 1 -- 172.21.15.40:0/4262634619 learned_addr learned my addr 172.21.15.40:0/4262634619 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:15.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f479fe00640 1 --2- 172.21.15.40:0/4262634619 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47a806cb80 0x7f47a80b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:15.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47a7e00640 1 -- 172.21.15.40:0/4262634619 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47a806cb80 msgr2=0x7f47a80b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:15.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47a7e00640 1 --2- 172.21.15.40:0/4262634619 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47a806cb80 0x7f47a80b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47a7e00640 1 -- 172.21.15.40:0/4262634619 >> v1:172.21.15.40:6789/0 conn(0x7f47a815ac60 legacy=0x7f47a80b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:15.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47a7e00640 1 -- 172.21.15.40:0/4262634619 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a80b3820 con 0x7f47a80b4840 2024-03-24T00:36:15.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.422+0000 7f47a6a00640 1 -- 172.21.15.40:0/4262634619 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674713823 0 0) 0x7f47a80b3260 con 0x7f47a815ac60 2024-03-24T00:36:15.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.426+0000 7f47a7e00640 1 --2- 172.21.15.40:0/4262634619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a80b4840 0x7f47a8071b00 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f47a000d150 tx=0x7f47a001eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f552b37a3bc35eb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:15.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.426+0000 7f47a6a00640 1 -- 172.21.15.40:0/4262634619 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a000b2d0 con 0x7f47a80b4840 2024-03-24T00:36:15.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.426+0000 7f47a6a00640 1 -- 172.21.15.40:0/4262634619 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a000b470 con 0x7f47a80b4840 2024-03-24T00:36:15.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.426+0000 7f47a6a00640 1 -- 172.21.15.40:0/4262634619 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a00a7a90 con 0x7f47a80b4840 2024-03-24T00:36:15.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.426+0000 7f47ae000640 1 -- 172.21.15.40:0/4262634619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a80b4840 msgr2=0x7f47a8071b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:15.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.426+0000 7f47ae000640 1 --2- 172.21.15.40:0/4262634619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a80b4840 0x7f47a8071b00 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f47a000d150 tx=0x7f47a001eba0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.426+0000 7f47ae000640 1 -- 172.21.15.40:0/4262634619 shutdown_connections 2024-03-24T00:36:15.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.426+0000 7f47ae000640 1 --2- 172.21.15.40:0/4262634619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a80b4840 0x7f47a8071b00 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.426+0000 7f47ae000640 1 --2- 172.21.15.40:0/4262634619 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47a806cb80 0x7f47a80b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.426+0000 7f47ae000640 1 -- 172.21.15.40:0/4262634619 >> 172.21.15.40:0/4262634619 conn(0x7f47a80b1660 msgr2=0x7f47a80b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:15.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.426+0000 7f47ae000640 1 -- 172.21.15.40:0/4262634619 shutdown_connections 2024-03-24T00:36:15.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.426+0000 7f47ae000640 1 -- 172.21.15.40:0/4262634619 wait complete. 2024-03-24T00:36:15.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47ae000640 1 Processor -- start 2024-03-24T00:36:15.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47ae000640 1 -- start start 2024-03-24T00:36:15.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47ae000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47a806cb80 0x7f47a8146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:15.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47ae000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47a80b4840 0x7f47a815b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47ae000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a815ac60 0x7f47a80ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47ae000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a806d730 con 0x7f47a815ac60 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47ae000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a806d5b0 con 0x7f47a80b4840 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47ae000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47a806d8b0 con 0x7f47a806cb80 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a815ac60 0x7f47a80ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47a7400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47a806cb80 0x7f47a8146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f479fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47a80b4840 0x7f47a815b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47a7400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47a806cb80 0x7f47a8146f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:46730/0 (socket says 172.21.15.40:46730) 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f479fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47a80b4840 0x7f47a815b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:53982/0 (socket says 172.21.15.40:53982) 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47a7400640 1 -- 172.21.15.40:0/810759181 learned_addr learned my addr 172.21.15.40:0/810759181 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a815ac60 0x7f47a80ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39912/0 (socket says 172.21.15.40:39912) 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47a7e00640 1 -- 172.21.15.40:0/810759181 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47a806cb80 msgr2=0x7f47a8146f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47a7e00640 1 --2- 172.21.15.40:0/810759181 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47a806cb80 0x7f47a8146f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47a7e00640 1 -- 172.21.15.40:0/810759181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47a80b4840 msgr2=0x7f47a815b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:15.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47a7e00640 1 --2- 172.21.15.40:0/810759181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47a80b4840 0x7f47a815b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47a7e00640 1 -- 172.21.15.40:0/810759181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a815bbc0 con 0x7f47a815ac60 2024-03-24T00:36:15.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47a7e00640 1 --2- 172.21.15.40:0/810759181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a815ac60 0x7f47a80ad8e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f47a00ad000 tx=0x7f47a001b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:15.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f479f400640 1 -- 172.21.15.40:0/810759181 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a000b720 con 0x7f47a815ac60 2024-03-24T00:36:15.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f479f400640 1 -- 172.21.15.40:0/810759181 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a0003ba0 con 0x7f47a815ac60 2024-03-24T00:36:15.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47ae000640 1 -- 172.21.15.40:0/810759181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a815bea0 con 0x7f47a815ac60 2024-03-24T00:36:15.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f479f400640 1 -- 172.21.15.40:0/810759181 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a0004030 con 0x7f47a815ac60 2024-03-24T00:36:15.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.430+0000 7f47ae000640 1 -- 172.21.15.40:0/810759181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a815c460 con 0x7f47a815ac60 2024-03-24T00:36:15.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.434+0000 7f47ae000640 1 -- 172.21.15.40:0/810759181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a8068170 con 0x7f47a815ac60 2024-03-24T00:36:15.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.434+0000 7f479f400640 1 -- 172.21.15.40:0/810759181 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f47a00b6070 con 0x7f47a815ac60 2024-03-24T00:36:15.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.434+0000 7f479f400640 1 --2- 172.21.15.40:0/810759181 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4784067660 0x7f4784069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:15.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.434+0000 7f479f400640 1 -- 172.21.15.40:0/810759181 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47a01203f0 con 0x7f47a815ac60 2024-03-24T00:36:15.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.434+0000 7f47a7400640 1 --2- 172.21.15.40:0/810759181 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4784067660 0x7f4784069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:15.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.434+0000 7f47a7400640 1 --2- 172.21.15.40:0/810759181 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4784067660 0x7f4784069b20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4798000f70 tx=0x7f4798009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:15.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.438+0000 7f479f400640 1 -- 172.21.15.40:0/810759181 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f47a0018820 con 0x7f47a815ac60 2024-03-24T00:36:15.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.602+0000 7f47ae000640 1 -- 172.21.15.40:0/810759181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random}] v 0) v1 -- 0x7f47a81474b0 con 0x7f47a815ac60 2024-03-24T00:36:15.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.606+0000 7f479f400640 1 -- 172.21.15.40:0/810759181 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random}]=0 v44)=0 v44) v1 ==== 128+0+0 (secure 0 0 0) 0x7f47a00f0b30 con 0x7f47a815ac60 2024-03-24T00:36:15.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.610+0000 7f47ae000640 1 -- 172.21.15.40:0/810759181 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4784067660 msgr2=0x7f4784069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:15.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.610+0000 7f47ae000640 1 --2- 172.21.15.40:0/810759181 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4784067660 0x7f4784069b20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4798000f70 tx=0x7f4798009040 comp rx=0 tx=0).stop 2024-03-24T00:36:15.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.610+0000 7f47ae000640 1 -- 172.21.15.40:0/810759181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a815ac60 msgr2=0x7f47a80ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:15.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.610+0000 7f47ae000640 1 --2- 172.21.15.40:0/810759181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a815ac60 0x7f47a80ad8e0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f47a00ad000 tx=0x7f47a001b940 comp rx=0 tx=0).stop 2024-03-24T00:36:15.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.610+0000 7f47ae000640 1 -- 172.21.15.40:0/810759181 shutdown_connections 2024-03-24T00:36:15.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.610+0000 7f47ae000640 1 --2- 172.21.15.40:0/810759181 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4784067660 0x7f4784069b20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.610+0000 7f47ae000640 1 --2- 172.21.15.40:0/810759181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47a815ac60 0x7f47a80ad8e0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.610+0000 7f47ae000640 1 --2- 172.21.15.40:0/810759181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47a80b4840 0x7f47a815b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.610+0000 7f47ae000640 1 --2- 172.21.15.40:0/810759181 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47a806cb80 0x7f47a8146f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.610+0000 7f47ae000640 1 -- 172.21.15.40:0/810759181 >> 172.21.15.40:0/810759181 conn(0x7f47a80b1660 msgr2=0x7f47a81581d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:15.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.610+0000 7f47ae000640 1 -- 172.21.15.40:0/810759181 shutdown_connections 2024-03-24T00:36:15.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.610+0000 7f47ae000640 1 -- 172.21.15.40:0/810759181 wait complete. 2024-03-24T00:36:15.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_distributed True 2024-03-24T00:36:15.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a8d000640 1 Processor -- start 2024-03-24T00:36:15.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a8d000640 1 -- start start 2024-03-24T00:36:15.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a8d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9a8815ffd0 0x7f9a880ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a8d000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a880728d0 0x7f9a880cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a8d000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a88072d40 con 0x7f9a8815bd00 2024-03-24T00:36:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a8d000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a880cf790 con 0x7f9a8815ffd0 2024-03-24T00:36:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a8d000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a88071de0 con 0x7f9a880728d0 2024-03-24T00:36:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a85400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9a8815ffd0 0x7f9a880ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a86800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a880728d0 0x7f9a880cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a85e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9a8815bd00 0x7f9a8815f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:35854/0 (socket says 172.21.15.40:35854) 2024-03-24T00:36:15.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a85e00640 1 -- 172.21.15.40:0/4204138216 learned_addr learned my addr 172.21.15.40:0/4204138216 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:15.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a86800640 1 -- 172.21.15.40:0/4204138216 >> v2:172.21.15.53:3300/0 conn(0x7f9a8815ffd0 msgr2=0x7f9a880ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:15.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a86800640 1 --2- 172.21.15.40:0/4204138216 >> v2:172.21.15.53:3300/0 conn(0x7f9a8815ffd0 0x7f9a880ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a86800640 1 -- 172.21.15.40:0/4204138216 >> v1:172.21.15.53:6789/0 conn(0x7f9a8815bd00 legacy=0x7f9a8815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:15.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a86800640 1 -- 172.21.15.40:0/4204138216 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a880cea50 con 0x7f9a880728d0 2024-03-24T00:36:15.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.774+0000 7f9a85400640 1 --2- 172.21.15.40:0/4204138216 >> v2:172.21.15.53:3300/0 conn(0x7f9a8815ffd0 0x7f9a880ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:15.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.778+0000 7f9a86800640 1 --2- 172.21.15.40:0/4204138216 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a880728d0 0x7f9a880cf250 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009d90 tx=0x7f9a8001c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9111cd008b0ee3fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:15.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.778+0000 7f9a84a00640 1 -- 172.21.15.40:0/4204138216 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a800b2070 con 0x7f9a880728d0 2024-03-24T00:36:15.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.778+0000 7f9a84a00640 1 -- 172.21.15.40:0/4204138216 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a800a5a00 con 0x7f9a880728d0 2024-03-24T00:36:15.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.778+0000 7f9a84a00640 1 -- 172.21.15.40:0/4204138216 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a800a5dd0 con 0x7f9a880728d0 2024-03-24T00:36:15.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.778+0000 7f9a8d000640 1 -- 172.21.15.40:0/4204138216 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a880728d0 msgr2=0x7f9a880cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:15.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.778+0000 7f9a8d000640 1 --2- 172.21.15.40:0/4204138216 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a880728d0 0x7f9a880cf250 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9a80009d90 tx=0x7f9a8001c940 comp rx=0 tx=0).stop 2024-03-24T00:36:15.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.778+0000 7f9a8d000640 1 -- 172.21.15.40:0/4204138216 shutdown_connections 2024-03-24T00:36:15.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.778+0000 7f9a8d000640 1 --2- 172.21.15.40:0/4204138216 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a880728d0 0x7f9a880cf250 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.778+0000 7f9a8d000640 1 --2- 172.21.15.40:0/4204138216 >> v2:172.21.15.53:3300/0 conn(0x7f9a8815ffd0 0x7f9a880ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.778+0000 7f9a8d000640 1 -- 172.21.15.40:0/4204138216 >> 172.21.15.40:0/4204138216 conn(0x7f9a880b1660 msgr2=0x7f9a880b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:15.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.778+0000 7f9a8d000640 1 -- 172.21.15.40:0/4204138216 shutdown_connections 2024-03-24T00:36:15.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.778+0000 7f9a8d000640 1 -- 172.21.15.40:0/4204138216 wait complete. 2024-03-24T00:36:15.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a8d000640 1 Processor -- start 2024-03-24T00:36:15.784 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a8d000640 1 -- start start 2024-03-24T00:36:15.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a8d000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a880728d0 0x7f9a8814bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:15.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a8d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a8815bd00 0x7f9a880b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:15.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a8d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8815ffd0 0x7f9a880adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:15.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a8d000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8806d320 con 0x7f9a8815ffd0 2024-03-24T00:36:15.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a8d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8815c0e0 con 0x7f9a8815bd00 2024-03-24T00:36:15.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a8d000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8806d4a0 con 0x7f9a880728d0 2024-03-24T00:36:15.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a86800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8815ffd0 0x7f9a880adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:15.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a86800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8815ffd0 0x7f9a880adb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39916/0 (socket says 172.21.15.40:39916) 2024-03-24T00:36:15.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a86800640 1 -- 172.21.15.40:0/3812014697 learned_addr learned my addr 172.21.15.40:0/3812014697 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:15.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a85400640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a8815bd00 0x7f9a880b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:15.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a85e00640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a880728d0 0x7f9a8814bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:15.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a86800640 1 -- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a880728d0 msgr2=0x7f9a8814bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:15.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a86800640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a880728d0 0x7f9a8814bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a86800640 1 -- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a8815bd00 msgr2=0x7f9a880b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:15.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a86800640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a8815bd00 0x7f9a880b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a86800640 1 -- 172.21.15.40:0/3812014697 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a880b3be0 con 0x7f9a8815ffd0 2024-03-24T00:36:15.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a86800640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8815ffd0 0x7f9a880adb40 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f9a800b0040 tx=0x7f9a800196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:15.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a7e000640 1 -- 172.21.15.40:0/3812014697 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a800b2050 con 0x7f9a8815ffd0 2024-03-24T00:36:15.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a7e000640 1 -- 172.21.15.40:0/3812014697 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a800be070 con 0x7f9a8815ffd0 2024-03-24T00:36:15.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a8d000640 1 -- 172.21.15.40:0/3812014697 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a880b3ec0 con 0x7f9a8815ffd0 2024-03-24T00:36:15.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a7e000640 1 -- 172.21.15.40:0/3812014697 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a800adb10 con 0x7f9a8815ffd0 2024-03-24T00:36:15.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a8d000640 1 -- 172.21.15.40:0/3812014697 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a880b4480 con 0x7f9a8815ffd0 2024-03-24T00:36:15.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.782+0000 7f9a8d000640 1 -- 172.21.15.40:0/3812014697 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a88068170 con 0x7f9a8815ffd0 2024-03-24T00:36:15.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.786+0000 7f9a7e000640 1 -- 172.21.15.40:0/3812014697 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f9a800adcb0 con 0x7f9a8815ffd0 2024-03-24T00:36:15.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.786+0000 7f9a7e000640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9a68067660 0x7f9a68069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:15.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.786+0000 7f9a7e000640 1 -- 172.21.15.40:0/3812014697 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a80120b40 con 0x7f9a8815ffd0 2024-03-24T00:36:15.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.786+0000 7f9a85e00640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9a68067660 0x7f9a68069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:15.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.786+0000 7f9a85e00640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9a68067660 0x7f9a68069b20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9a78000f70 tx=0x7f9a78007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:15.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.790+0000 7f9a7e000640 1 -- 172.21.15.40:0/3812014697 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9a80016560 con 0x7f9a8815ffd0 2024-03-24T00:36:15.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.958+0000 7f9a8d000640 1 -- 172.21.15.40:0/3812014697 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_distributed}] v 0) v1 -- 0x7f9a8806e870 con 0x7f9a8815ffd0 2024-03-24T00:36:15.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.962+0000 7f9a7e000640 1 -- 172.21.15.40:0/3812014697 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_distributed}]=0 v45)=0 v45) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9a800f1160 con 0x7f9a8815ffd0 2024-03-24T00:36:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.962+0000 7f9a8d000640 1 -- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9a68067660 msgr2=0x7f9a68069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.962+0000 7f9a8d000640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9a68067660 0x7f9a68069b20 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f9a78000f70 tx=0x7f9a78007e90 comp rx=0 tx=0).stop 2024-03-24T00:36:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.962+0000 7f9a8d000640 1 -- 172.21.15.40:0/3812014697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8815ffd0 msgr2=0x7f9a880adb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.962+0000 7f9a8d000640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8815ffd0 0x7f9a880adb40 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f9a800b0040 tx=0x7f9a800196e0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.962+0000 7f9a8d000640 1 -- 172.21.15.40:0/3812014697 shutdown_connections 2024-03-24T00:36:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.962+0000 7f9a8d000640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9a68067660 0x7f9a68069b20 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.962+0000 7f9a8d000640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9a8815ffd0 0x7f9a880adb40 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.966+0000 7f9a8d000640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a8815bd00 0x7f9a880b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.966+0000 7f9a8d000640 1 --2- 172.21.15.40:0/3812014697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9a880728d0 0x7f9a8814bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.966+0000 7f9a8d000640 1 -- 172.21.15.40:0/3812014697 >> 172.21.15.40:0/3812014697 conn(0x7f9a880b1660 msgr2=0x7f9a880cf930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.966+0000 7f9a8d000640 1 -- 172.21.15.40:0/3812014697 shutdown_connections 2024-03-24T00:36:15.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:15.966+0000 7f9a8d000640 1 -- 172.21.15.40:0/3812014697 wait complete. 2024-03-24T00:36:15.987 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random_max 1.0 2024-03-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.126+0000 7f883b200640 1 Processor -- start 2024-03-24T00:36:16.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.126+0000 7f883b200640 1 -- start start 2024-03-24T00:36:16.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f883b200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f883c0b4b60 0x7f883c0b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f883b200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f883c0b1c30 0x7f883c0cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f883b200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883c0b20b0 con 0x7f883c1595d0 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f883b200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f883c0b2940 con 0x7f883c0b1c30 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f883b200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883c15c110 con 0x7f883c0b4b60 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f883a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f883c0b1c30 0x7f883c0cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f8839e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f883c1595d0 0x7f883c0b4450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:35860/0 (socket says 172.21.15.40:35860) 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f8839e00640 1 -- 172.21.15.40:0/1219209149 learned_addr learned my addr 172.21.15.40:0/1219209149 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f8833e00640 1 --2- 172.21.15.40:0/1219209149 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f883c0b4b60 0x7f883c0b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f883a800640 1 -- 172.21.15.40:0/1219209149 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f883c0b4b60 msgr2=0x7f883c0b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f883a800640 1 --2- 172.21.15.40:0/1219209149 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f883c0b4b60 0x7f883c0b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f883a800640 1 -- 172.21.15.40:0/1219209149 >> v1:172.21.15.53:6789/0 conn(0x7f883c1595d0 legacy=0x7f883c0b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f8839400640 1 -- 172.21.15.40:0/1219209149 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493436860 0 0) 0x7f883c0b20b0 con 0x7f883c1595d0 2024-03-24T00:36:16.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.130+0000 7f883a800640 1 -- 172.21.15.40:0/1219209149 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883c15c350 con 0x7f883c0b1c30 2024-03-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883a800640 1 --2- 172.21.15.40:0/1219209149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f883c0b1c30 0x7f883c0cdc70 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f8834009dc0 tx=0x7f883401c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73b2c5a3ed0f165 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f8839400640 1 -- 172.21.15.40:0/1219209149 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88340b2070 con 0x7f883c0b1c30 2024-03-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f8839400640 1 -- 172.21.15.40:0/1219209149 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88340a5950 con 0x7f883c0b1c30 2024-03-24T00:36:16.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f8839400640 1 -- 172.21.15.40:0/1219209149 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88340a5d20 con 0x7f883c0b1c30 2024-03-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 -- 172.21.15.40:0/1219209149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f883c0b1c30 msgr2=0x7f883c0cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 --2- 172.21.15.40:0/1219209149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f883c0b1c30 0x7f883c0cdc70 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f8834009dc0 tx=0x7f883401c890 comp rx=0 tx=0).stop 2024-03-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 -- 172.21.15.40:0/1219209149 shutdown_connections 2024-03-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 --2- 172.21.15.40:0/1219209149 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f883c0b4b60 0x7f883c0b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 --2- 172.21.15.40:0/1219209149 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f883c0b1c30 0x7f883c0cdc70 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 -- 172.21.15.40:0/1219209149 >> 172.21.15.40:0/1219209149 conn(0x7f883c066e80 msgr2=0x7f883c067290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 -- 172.21.15.40:0/1219209149 shutdown_connections 2024-03-24T00:36:16.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 -- 172.21.15.40:0/1219209149 wait complete. 2024-03-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 Processor -- start 2024-03-24T00:36:16.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 -- start start 2024-03-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f883c0b1c30 0x7f883c14f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f883c0b4b60 0x7f883c0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f883c1595d0 0x7f883c14b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c1599b0 con 0x7f883c0b4b60 2024-03-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883c15ced0 con 0x7f883c1595d0 2024-03-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883b200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883c159b30 con 0x7f883c0b1c30 2024-03-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f8833e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f883c0b4b60 0x7f883c0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f8833e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f883c0b4b60 0x7f883c0b3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39940/0 (socket says 172.21.15.40:39940) 2024-03-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f8833e00640 1 -- 172.21.15.40:0/686433141 learned_addr learned my addr 172.21.15.40:0/686433141 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f8839e00640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f883c0b1c30 0x7f883c14f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.134+0000 7f883a800640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f883c1595d0 0x7f883c14b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f8833e00640 1 -- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f883c0b1c30 msgr2=0x7f883c14f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f8833e00640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f883c0b1c30 0x7f883c14f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f8833e00640 1 -- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f883c1595d0 msgr2=0x7f883c14b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f8833e00640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f883c1595d0 0x7f883c14b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f8833e00640 1 -- 172.21.15.40:0/686433141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883c0b3cf0 con 0x7f883c0b4b60 2024-03-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f883a800640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f883c1595d0 0x7f883c14b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:16.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f8833e00640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f883c0b4b60 0x7f883c0b3550 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f882400e930 tx=0x7f882400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f8832000640 1 -- 172.21.15.40:0/686433141 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824017070 con 0x7f883c0b4b60 2024-03-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f8832000640 1 -- 172.21.15.40:0/686433141 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882400ca00 con 0x7f883c0b4b60 2024-03-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f8832000640 1 -- 172.21.15.40:0/686433141 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882400cdb0 con 0x7f883c0b4b60 2024-03-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f883b200640 1 -- 172.21.15.40:0/686433141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883c156d70 con 0x7f883c0b4b60 2024-03-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f883b200640 1 -- 172.21.15.40:0/686433141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883c157330 con 0x7f883c0b4b60 2024-03-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f883b200640 1 -- 172.21.15.40:0/686433141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883c04f410 con 0x7f883c0b4b60 2024-03-24T00:36:16.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f8832000640 1 -- 172.21.15.40:0/686433141 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f882401a070 con 0x7f883c0b4b60 2024-03-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f8832000640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8818067690 0x7f8818069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.138+0000 7f8832000640 1 -- 172.21.15.40:0/686433141 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8824085c10 con 0x7f883c0b4b60 2024-03-24T00:36:16.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.142+0000 7f8839e00640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8818067690 0x7f8818069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:16.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.142+0000 7f8839e00640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8818067690 0x7f8818069b50 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f882c000f70 tx=0x7f882c007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:16.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.142+0000 7f8832000640 1 -- 172.21.15.40:0/686433141 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8824056260 con 0x7f883c0b4b60 2024-03-24T00:36:16.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.310+0000 7f883b200640 1 -- 172.21.15.40:0/686433141 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random_max}] v 0) v1 -- 0x7f883c06dd50 con 0x7f883c0b4b60 2024-03-24T00:36:16.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.310+0000 7f8832000640 1 -- 172.21.15.40:0/686433141 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random_max}]=0 v46)=0 v46) v1 ==== 131+0+0 (secure 0 0 0) 0x7f882405a300 con 0x7f883c0b4b60 2024-03-24T00:36:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.314+0000 7f883b200640 1 -- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8818067690 msgr2=0x7f8818069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.314+0000 7f883b200640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8818067690 0x7f8818069b50 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f882c000f70 tx=0x7f882c007df0 comp rx=0 tx=0).stop 2024-03-24T00:36:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.314+0000 7f883b200640 1 -- 172.21.15.40:0/686433141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f883c0b4b60 msgr2=0x7f883c0b3550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.314+0000 7f883b200640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f883c0b4b60 0x7f883c0b3550 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f882400e930 tx=0x7f882400edf0 comp rx=0 tx=0).stop 2024-03-24T00:36:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.314+0000 7f883b200640 1 -- 172.21.15.40:0/686433141 shutdown_connections 2024-03-24T00:36:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.314+0000 7f883b200640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8818067690 0x7f8818069b50 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.314+0000 7f883b200640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f883c1595d0 0x7f883c14b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.314+0000 7f883b200640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f883c0b4b60 0x7f883c0b3550 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.314+0000 7f883b200640 1 --2- 172.21.15.40:0/686433141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f883c0b1c30 0x7f883c14f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.314+0000 7f883b200640 1 -- 172.21.15.40:0/686433141 >> 172.21.15.40:0/686433141 conn(0x7f883c066e80 msgr2=0x7f883c0b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.314+0000 7f883b200640 1 -- 172.21.15.40:0/686433141 shutdown_connections 2024-03-24T00:36:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:16.314+0000 7f883b200640 1 -- 172.21.15.40:0/686433141 wait complete. 2024-03-24T00:36:16.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:16.337 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:36:16.337 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:36:16.337 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:36:16.337 DEBUG:teuthology.orchestra.run.smithi040:> # Use up a random number of inode numbers so the ephemeral pinning is not the same every test. 2024-03-24T00:36:16.337 DEBUG:teuthology.orchestra.run.smithi040:> mkdir .inode_number_thrash 2024-03-24T00:36:16.337 DEBUG:teuthology.orchestra.run.smithi040:> count=$((RANDOM % 1024)) 2024-03-24T00:36:16.337 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < count; i++)); do touch .inode_number_thrash/$i; done 2024-03-24T00:36:16.338 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf .inode_number_thrash 2024-03-24T00:36:16.338 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:36:17.709 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-24T00:36:17.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.850+0000 7f7cba400640 1 Processor -- start 2024-03-24T00:36:17.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.850+0000 7f7cba400640 1 -- start start 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cba400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7cb415bd00 0x7f7cb40ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cba400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7cb40728d0 0x7f7cb40cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cba400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cb415f8c0 0x7f7cb415fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cba400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb40cf790 con 0x7f7cb415bd00 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cba400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb4160250 con 0x7f7cb40728d0 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cba400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb4072d60 con 0x7f7cb415f8c0 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cb3400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7cb415bd00 0x7f7cb40ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cb3400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7cb415bd00 0x7f7cb40ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60508/0 (socket says 172.21.15.40:60508) 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cb3400640 1 -- 172.21.15.40:0/1557818441 learned_addr learned my addr 172.21.15.40:0/1557818441 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cb3e00640 1 --2- 172.21.15.40:0/1557818441 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cb415f8c0 0x7f7cb415fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cb2a00640 1 --2- 172.21.15.40:0/1557818441 >> v2:172.21.15.53:3300/0 conn(0x7f7cb40728d0 0x7f7cb40cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cb3400640 1 -- 172.21.15.40:0/1557818441 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cb415f8c0 msgr2=0x7f7cb415fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cb3400640 1 --2- 172.21.15.40:0/1557818441 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cb415f8c0 0x7f7cb415fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cb3400640 1 -- 172.21.15.40:0/1557818441 >> v2:172.21.15.53:3300/0 conn(0x7f7cb40728d0 msgr2=0x7f7cb40cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cb3400640 1 --2- 172.21.15.40:0/1557818441 >> v2:172.21.15.53:3300/0 conn(0x7f7cb40728d0 0x7f7cb40cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.854+0000 7f7cb3400640 1 -- 172.21.15.40:0/1557818441 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb4071de0 con 0x7f7cb415bd00 2024-03-24T00:36:17.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cb3400640 1 --2- 172.21.15.40:0/1557818441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb415bd00 0x7f7cb40ae8e0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f7cb406ff40 tx=0x7f7ca401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc3bad657333b85 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:17.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cb2000640 1 -- 172.21.15.40:0/1557818441 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40aa610 con 0x7f7cb415bd00 2024-03-24T00:36:17.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cb2000640 1 -- 172.21.15.40:0/1557818441 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca40af070 con 0x7f7cb415bd00 2024-03-24T00:36:17.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cb2000640 1 -- 172.21.15.40:0/1557818441 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40aa9d0 con 0x7f7cb415bd00 2024-03-24T00:36:17.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 -- 172.21.15.40:0/1557818441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb415bd00 msgr2=0x7f7cb40ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:17.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 --2- 172.21.15.40:0/1557818441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb415bd00 0x7f7cb40ae8e0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f7cb406ff40 tx=0x7f7ca401c4f0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 -- 172.21.15.40:0/1557818441 shutdown_connections 2024-03-24T00:36:17.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 --2- 172.21.15.40:0/1557818441 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cb415f8c0 0x7f7cb415fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 --2- 172.21.15.40:0/1557818441 >> v2:172.21.15.53:3300/0 conn(0x7f7cb40728d0 0x7f7cb40cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 --2- 172.21.15.40:0/1557818441 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb415bd00 0x7f7cb40ae8e0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 -- 172.21.15.40:0/1557818441 >> 172.21.15.40:0/1557818441 conn(0x7f7cb40b1660 msgr2=0x7f7cb40b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:17.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 -- 172.21.15.40:0/1557818441 shutdown_connections 2024-03-24T00:36:17.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 -- 172.21.15.40:0/1557818441 wait complete. 2024-03-24T00:36:17.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 Processor -- start 2024-03-24T00:36:17.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 -- start start 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cb40728d0 0x7f7cb414e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb415bd00 0x7f7cb4147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cb415f8c0 0x7f7cb40aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb406d320 con 0x7f7cb415bd00 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb415c0e0 con 0x7f7cb40728d0 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cba400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb406d4a0 con 0x7f7cb415f8c0 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.858+0000 7f7cb2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb415bd00 0x7f7cb4147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cb2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb415bd00 0x7f7cb4147f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60510/0 (socket says 172.21.15.40:60510) 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cb3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cb40728d0 0x7f7cb414e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cb2a00640 1 -- 172.21.15.40:0/2647249191 learned_addr learned my addr 172.21.15.40:0/2647249191 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cb3e00640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cb415f8c0 0x7f7cb40aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cb3400640 1 -- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cb415f8c0 msgr2=0x7f7cb40aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:17.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cb3400640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cb415f8c0 0x7f7cb40aef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cb3400640 1 -- 172.21.15.40:0/2647249191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb415bd00 msgr2=0x7f7cb4147f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:17.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cb3400640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb415bd00 0x7f7cb4147f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:17.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cb3400640 1 -- 172.21.15.40:0/2647249191 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb414ccf0 con 0x7f7cb40728d0 2024-03-24T00:36:17.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cb2a00640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb415bd00 0x7f7cb4147f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:17.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cb3400640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cb40728d0 0x7f7cb414e120 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40ab000 tx=0x7f7ca401ad30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:17.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cab400640 1 -- 172.21.15.40:0/2647249191 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40bd5f0 con 0x7f7cb40728d0 2024-03-24T00:36:17.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cab400640 1 -- 172.21.15.40:0/2647249191 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca40bd7b0 con 0x7f7cb40728d0 2024-03-24T00:36:17.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cba400640 1 -- 172.21.15.40:0/2647249191 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb414cfd0 con 0x7f7cb40728d0 2024-03-24T00:36:17.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cab400640 1 -- 172.21.15.40:0/2647249191 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca40bdba0 con 0x7f7cb40728d0 2024-03-24T00:36:17.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cba400640 1 -- 172.21.15.40:0/2647249191 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb414d590 con 0x7f7cb40728d0 2024-03-24T00:36:17.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cba400640 1 -- 172.21.15.40:0/2647249191 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb4068170 con 0x7f7cb40728d0 2024-03-24T00:36:17.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cab400640 1 -- 172.21.15.40:0/2647249191 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7ca40059a0 con 0x7f7cb40728d0 2024-03-24T00:36:17.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.862+0000 7f7cab400640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c94067660 0x7f7c94069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:17.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.866+0000 7f7cab400640 1 -- 172.21.15.40:0/2647249191 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ca4120630 con 0x7f7cb40728d0 2024-03-24T00:36:17.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.866+0000 7f7cb2a00640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c94067660 0x7f7c94069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:17.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.866+0000 7f7cb2a00640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c94067660 0x7f7c94069b20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0011310 tx=0x7f7ca0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:17.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:17.866+0000 7f7cab400640 1 -- 172.21.15.40:0/2647249191 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7ca40c3060 con 0x7f7cb40728d0 2024-03-24T00:36:18.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.066+0000 7f7cba400640 1 -- 172.21.15.40:0/2647249191 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f7cb4151ca0 con 0x7f7cb40728d0 2024-03-24T00:36:18.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.610+0000 7f7cab400640 1 -- 172.21.15.40:0/2647249191 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v83) v1 ==== 105+0+0 (secure 0 0 0) 0x7f7ca40a7070 con 0x7f7cb40728d0 2024-03-24T00:36:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.614+0000 7f7cba400640 1 -- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c94067660 msgr2=0x7f7c94069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.614+0000 7f7cba400640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c94067660 0x7f7c94069b20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f7ca0011310 tx=0x7f7ca0009190 comp rx=0 tx=0).stop 2024-03-24T00:36:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.614+0000 7f7cba400640 1 -- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cb40728d0 msgr2=0x7f7cb414e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.614+0000 7f7cba400640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cb40728d0 0x7f7cb414e120 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f7ca40ab000 tx=0x7f7ca401ad30 comp rx=0 tx=0).stop 2024-03-24T00:36:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.614+0000 7f7cba400640 1 -- 172.21.15.40:0/2647249191 shutdown_connections 2024-03-24T00:36:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.614+0000 7f7cba400640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c94067660 0x7f7c94069b20 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.614+0000 7f7cba400640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cb415f8c0 0x7f7cb40aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.614+0000 7f7cba400640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7cb415bd00 0x7f7cb4147f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.614+0000 7f7cba400640 1 --2- 172.21.15.40:0/2647249191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cb40728d0 0x7f7cb414e120 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.614+0000 7f7cba400640 1 -- 172.21.15.40:0/2647249191 >> 172.21.15.40:0/2647249191 conn(0x7f7cb40b1660 msgr2=0x7f7cb4070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.614+0000 7f7cba400640 1 -- 172.21.15.40:0/2647249191 shutdown_connections 2024-03-24T00:36:18.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.614+0000 7f7cba400640 1 -- 172.21.15.40:0/2647249191 wait complete. 2024-03-24T00:36:18.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:36:18.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb61a00640 1 Processor -- start 2024-03-24T00:36:18.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb61a00640 1 -- start start 2024-03-24T00:36:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb61a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdb5c15e410 0x7fdb5c06df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb61a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb5c06f310 0x7fdb5c0ce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb61a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb5c0ce8d0 0x7fdb5c0cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb61a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5c06e8c0 con 0x7fdb5c0ce8d0 2024-03-24T00:36:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb61a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5c0cacd0 con 0x7fdb5c15e410 2024-03-24T00:36:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb61a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0cafb0 con 0x7fdb5c06f310 2024-03-24T00:36:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb5b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb5c0ce8d0 0x7fdb5c0cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb5aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdb5c15e410 0x7fdb5c06df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb5a000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb5c06f310 0x7fdb5c0ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb5b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb5c0ce8d0 0x7fdb5c0cccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60524/0 (socket says 172.21.15.40:60524) 2024-03-24T00:36:18.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb5b400640 1 -- 172.21.15.40:0/4172369575 learned_addr learned my addr 172.21.15.40:0/4172369575 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:18.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb5b400640 1 -- 172.21.15.40:0/4172369575 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb5c06f310 msgr2=0x7fdb5c0ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:18.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb5b400640 1 --2- 172.21.15.40:0/4172369575 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb5c06f310 0x7fdb5c0ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb5b400640 1 -- 172.21.15.40:0/4172369575 >> v2:172.21.15.53:3300/0 conn(0x7fdb5c15e410 msgr2=0x7fdb5c06df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:18.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb5b400640 1 --2- 172.21.15.40:0/4172369575 >> v2:172.21.15.53:3300/0 conn(0x7fdb5c15e410 0x7fdb5c06df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.786+0000 7fdb5b400640 1 -- 172.21.15.40:0/4172369575 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0cb290 con 0x7fdb5c0ce8d0 2024-03-24T00:36:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb5b400640 1 --2- 172.21.15.40:0/4172369575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb5c0ce8d0 0x7fdb5c0cccc0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c072820 tx=0x7fdb5401c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3579b85ac155a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb59600640 1 -- 172.21.15.40:0/4172369575 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb540a64d0 con 0x7fdb5c0ce8d0 2024-03-24T00:36:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb59600640 1 -- 172.21.15.40:0/4172369575 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb540a6670 con 0x7fdb5c0ce8d0 2024-03-24T00:36:18.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb59600640 1 -- 172.21.15.40:0/4172369575 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb540a6a40 con 0x7fdb5c0ce8d0 2024-03-24T00:36:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb61a00640 1 -- 172.21.15.40:0/4172369575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb5c0ce8d0 msgr2=0x7fdb5c0cccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb61a00640 1 --2- 172.21.15.40:0/4172369575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb5c0ce8d0 0x7fdb5c0cccc0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c072820 tx=0x7fdb5401c730 comp rx=0 tx=0).stop 2024-03-24T00:36:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb61a00640 1 -- 172.21.15.40:0/4172369575 shutdown_connections 2024-03-24T00:36:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb61a00640 1 --2- 172.21.15.40:0/4172369575 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb5c0ce8d0 0x7fdb5c0cccc0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb61a00640 1 --2- 172.21.15.40:0/4172369575 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb5c06f310 0x7fdb5c0ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb61a00640 1 --2- 172.21.15.40:0/4172369575 >> v2:172.21.15.53:3300/0 conn(0x7fdb5c15e410 0x7fdb5c06df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb61a00640 1 -- 172.21.15.40:0/4172369575 >> 172.21.15.40:0/4172369575 conn(0x7fdb5c072890 msgr2=0x7fdb5c072ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb61a00640 1 -- 172.21.15.40:0/4172369575 shutdown_connections 2024-03-24T00:36:18.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb61a00640 1 -- 172.21.15.40:0/4172369575 wait complete. 2024-03-24T00:36:18.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb61a00640 1 Processor -- start 2024-03-24T00:36:18.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.790+0000 7fdb61a00640 1 -- start start 2024-03-24T00:36:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb61a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb5c06f310 0x7fdb5c0c26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb61a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb5c0ce8d0 0x7fdb5c1589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb61a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb5c15e410 0x7fdb5c0adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb61a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0d0300 con 0x7fdb5c15e410 2024-03-24T00:36:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb61a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c06e6a0 con 0x7fdb5c0ce8d0 2024-03-24T00:36:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb61a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0d0480 con 0x7fdb5c06f310 2024-03-24T00:36:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb5b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb5c15e410 0x7fdb5c0adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb5b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb5c15e410 0x7fdb5c0adad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60534/0 (socket says 172.21.15.40:60534) 2024-03-24T00:36:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb5b400640 1 -- 172.21.15.40:0/4224884151 learned_addr learned my addr 172.21.15.40:0/4224884151 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb5a000640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb5c0ce8d0 0x7fdb5c1589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb5aa00640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb5c06f310 0x7fdb5c0c26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb5b400640 1 -- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb5c06f310 msgr2=0x7fdb5c0c26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb5b400640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb5c06f310 0x7fdb5c0c26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb5b400640 1 -- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb5c0ce8d0 msgr2=0x7fdb5c1589f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:18.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb5b400640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb5c0ce8d0 0x7fdb5c1589f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:18.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb5b400640 1 -- 172.21.15.40:0/4224884151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0ae590 con 0x7fdb5c15e410 2024-03-24T00:36:18.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb5b400640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb5c15e410 0x7fdb5c0adad0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fdb5401ade0 tx=0x7fdb540194d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:18.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb52a00640 1 -- 172.21.15.40:0/4224884151 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb540bf070 con 0x7fdb5c15e410 2024-03-24T00:36:18.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb52a00640 1 -- 172.21.15.40:0/4224884151 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb540090d0 con 0x7fdb5c15e410 2024-03-24T00:36:18.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb52a00640 1 -- 172.21.15.40:0/4224884151 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb540a6a40 con 0x7fdb5c15e410 2024-03-24T00:36:18.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb61a00640 1 -- 172.21.15.40:0/4224884151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c0ae870 con 0x7fdb5c15e410 2024-03-24T00:36:18.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb61a00640 1 -- 172.21.15.40:0/4224884151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c1581f0 con 0x7fdb5c15e410 2024-03-24T00:36:18.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.794+0000 7fdb61a00640 1 -- 172.21.15.40:0/4224884151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c0cde30 con 0x7fdb5c15e410 2024-03-24T00:36:18.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.798+0000 7fdb52a00640 1 -- 172.21.15.40:0/4224884151 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fdb540a6be0 con 0x7fdb5c15e410 2024-03-24T00:36:18.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.798+0000 7fdb52a00640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb3c067660 0x7fdb3c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:18.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.798+0000 7fdb52a00640 1 -- 172.21.15.40:0/4224884151 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb5411f3b0 con 0x7fdb5c15e410 2024-03-24T00:36:18.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.798+0000 7fdb5aa00640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb3c067660 0x7fdb3c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:18.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.798+0000 7fdb5aa00640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb3c067660 0x7fdb3c069b20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009ac0 tx=0x7fdb4c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:18.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.802+0000 7fdb52a00640 1 -- 172.21.15.40:0/4224884151 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdb540165e0 con 0x7fdb5c15e410 2024-03-24T00:36:19.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:18.998+0000 7fdb61a00640 1 -- 172.21.15.40:0/4224884151 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb5c068170 con 0x7fdb5c15e410 2024-03-24T00:36:19.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb52a00640 1 -- 172.21.15.40:0/4224884151 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4992 (secure 0 0 0) 0x7fdb540a6640 con 0x7fdb5c15e410 2024-03-24T00:36:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:19.004 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":84,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1097448819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1097448819},{"type":"v1","addr":"172.21.15.40:6834","nonce":1097448819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":6404,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/983819364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":983819364},{"type":"v1","addr":"172.21.15.53:6837","nonce":983819364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:36:07.163876+0000","modified":"2024-03-24T00:36:18.615921+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":7186,"mds_1":7180},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7180":{"gid":7180,"name":"a","rank":1,"incarnation":84,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6835/2302548193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2302548193},{"type":"v1","addr":"172.21.15.40:6835","nonce":2302548193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7186":{"gid":7186,"name":"d","rank":0,"incarnation":78,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/1878888046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1878888046},{"type":"v1","addr":"172.21.15.53:6835","nonce":1878888046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7186,"qdb_cluster":[7186]},"id":6}]} 2024-03-24T00:36:19.004 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 84 2024-03-24T00:36:19.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb61a00640 1 -- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb3c067660 msgr2=0x7fdb3c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:19.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb61a00640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb3c067660 0x7fdb3c069b20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009ac0 tx=0x7fdb4c009190 comp rx=0 tx=0).stop 2024-03-24T00:36:19.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb61a00640 1 -- 172.21.15.40:0/4224884151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb5c15e410 msgr2=0x7fdb5c0adad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:19.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb61a00640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb5c15e410 0x7fdb5c0adad0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fdb5401ade0 tx=0x7fdb540194d0 comp rx=0 tx=0).stop 2024-03-24T00:36:19.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb61a00640 1 -- 172.21.15.40:0/4224884151 shutdown_connections 2024-03-24T00:36:19.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb61a00640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb3c067660 0x7fdb3c069b20 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:19.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb61a00640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb5c15e410 0x7fdb5c0adad0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:19.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb61a00640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb5c0ce8d0 0x7fdb5c1589f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:19.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb61a00640 1 --2- 172.21.15.40:0/4224884151 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb5c06f310 0x7fdb5c0c26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:19.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb61a00640 1 -- 172.21.15.40:0/4224884151 >> 172.21.15.40:0/4224884151 conn(0x7fdb5c072890 msgr2=0x7fdb5c0b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:19.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb61a00640 1 -- 172.21.15.40:0/4224884151 shutdown_connections 2024-03-24T00:36:19.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:19.002+0000 7fdb61a00640 1 -- 172.21.15.40:0/4224884151 wait complete. 2024-03-24T00:36:19.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 84, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:36:07.163876+0000', 'modified': '2024-03-24T00:36:18.615921+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 7186, 'mds_1': 7180}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7180': {'gid': 7180, 'name': 'a', 'rank': 1, 'incarnation': 84, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6835/2302548193', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 2302548193}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 2302548193}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7186': {'gid': 7186, 'name': 'd', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.53:6835/1878888046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 1878888046}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 1878888046}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7186, 'qdb_cluster': [7186]} 2024-03-24T00:36:19.429 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7180:up:creating 2024-03-24T00:36:20.429 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:36:20.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.566+0000 7f514de00640 1 Processor -- start 2024-03-24T00:36:20.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.566+0000 7f514de00640 1 -- start start 2024-03-24T00:36:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f514de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f514815bdf0 0x7f51480cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f514de00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f51480cf330 0x7f51480d0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f514de00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51480ccb50 con 0x7f514815c1d0 2024-03-24T00:36:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f514de00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51480cce30 con 0x7f51480cf330 2024-03-24T00:36:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f514de00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51480cd110 con 0x7f514815bdf0 2024-03-24T00:36:20.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f513e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f51480cf330 0x7f51480d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f5147400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f514815c1d0 0x7f514806ef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52622/0 (socket says 172.21.15.40:52622) 2024-03-24T00:36:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f5147400640 1 -- 172.21.15.40:0/4015155708 learned_addr learned my addr 172.21.15.40:0/4015155708 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f513e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f51480cf330 0x7f51480d0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60540/0 (socket says 172.21.15.40:60540) 2024-03-24T00:36:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f5146a00640 1 --2- 172.21.15.40:0/4015155708 >> v2:172.21.15.53:3300/0 conn(0x7f514815bdf0 0x7f51480cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f513e000640 1 -- 172.21.15.40:0/4015155708 >> v2:172.21.15.53:3300/0 conn(0x7f514815bdf0 msgr2=0x7f51480cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f513e000640 1 --2- 172.21.15.40:0/4015155708 >> v2:172.21.15.53:3300/0 conn(0x7f514815bdf0 0x7f51480cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f513e000640 1 -- 172.21.15.40:0/4015155708 >> v1:172.21.15.40:6789/0 conn(0x7f514815c1d0 legacy=0x7f514806ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.570+0000 7f513e000640 1 -- 172.21.15.40:0/4015155708 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51480cd3f0 con 0x7f51480cf330 2024-03-24T00:36:20.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f513e000640 1 --2- 172.21.15.40:0/4015155708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51480cf330 0x7f51480d0260 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5148073170 tx=0x7f513001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d7a8d709bf3b78 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:20.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f5146000640 1 -- 172.21.15.40:0/4015155708 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51300b0070 con 0x7f51480cf330 2024-03-24T00:36:20.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f5146000640 1 -- 172.21.15.40:0/4015155708 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51300ab600 con 0x7f51480cf330 2024-03-24T00:36:20.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f5146000640 1 -- 172.21.15.40:0/4015155708 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51300ab9d0 con 0x7f51480cf330 2024-03-24T00:36:20.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f514de00640 1 -- 172.21.15.40:0/4015155708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51480cf330 msgr2=0x7f51480d0260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f514de00640 1 --2- 172.21.15.40:0/4015155708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51480cf330 0x7f51480d0260 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f5148073170 tx=0x7f513001c420 comp rx=0 tx=0).stop 2024-03-24T00:36:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f514de00640 1 -- 172.21.15.40:0/4015155708 shutdown_connections 2024-03-24T00:36:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f514de00640 1 --2- 172.21.15.40:0/4015155708 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51480cf330 0x7f51480d0260 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f514de00640 1 --2- 172.21.15.40:0/4015155708 >> v2:172.21.15.53:3300/0 conn(0x7f514815bdf0 0x7f51480cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f514de00640 1 -- 172.21.15.40:0/4015155708 >> 172.21.15.40:0/4015155708 conn(0x7f51480b37f0 msgr2=0x7f51480b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f514de00640 1 -- 172.21.15.40:0/4015155708 shutdown_connections 2024-03-24T00:36:20.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f514de00640 1 -- 172.21.15.40:0/4015155708 wait complete. 2024-03-24T00:36:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f514de00640 1 Processor -- start 2024-03-24T00:36:20.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f514de00640 1 -- start start 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f514de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51480cf330 0x7f5148151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.574+0000 7f514de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f514815bdf0 0x7f51480c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f514de00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f514815c1d0 0x7f51480b5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f514de00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51481690d0 con 0x7f51480cf330 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f514de00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51480d0bb0 con 0x7f514815bdf0 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f514de00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5148169250 con 0x7f514815c1d0 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f5146a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51480cf330 0x7f5148151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f5146a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51480cf330 0x7f5148151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60550/0 (socket says 172.21.15.40:60550) 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f5146a00640 1 -- 172.21.15.40:0/155664478 learned_addr learned my addr 172.21.15.40:0/155664478 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f513e000640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f514815bdf0 0x7f51480c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f5147400640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f514815c1d0 0x7f51480b5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f5146a00640 1 -- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f514815c1d0 msgr2=0x7f51480b5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f5146a00640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f514815c1d0 0x7f51480b5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f5146a00640 1 -- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f514815bdf0 msgr2=0x7f51480c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f5146a00640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f514815bdf0 0x7f51480c9830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f5146a00640 1 -- 172.21.15.40:0/155664478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51480c8d70 con 0x7f51480cf330 2024-03-24T00:36:20.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f5146a00640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51480cf330 0x7f5148151770 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f513800efc0 tx=0x7f513800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:20.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f513fe00640 1 -- 172.21.15.40:0/155664478 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5138018020 con 0x7f51480cf330 2024-03-24T00:36:20.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f513fe00640 1 -- 172.21.15.40:0/155664478 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5138014480 con 0x7f51480cf330 2024-03-24T00:36:20.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f513fe00640 1 -- 172.21.15.40:0/155664478 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5138014850 con 0x7f51480cf330 2024-03-24T00:36:20.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f514de00640 1 -- 172.21.15.40:0/155664478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51480ca860 con 0x7f51480cf330 2024-03-24T00:36:20.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f514de00640 1 -- 172.21.15.40:0/155664478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51480c9030 con 0x7f51480cf330 2024-03-24T00:36:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f514de00640 1 -- 172.21.15.40:0/155664478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5148068170 con 0x7f51480cf330 2024-03-24T00:36:20.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.578+0000 7f513fe00640 1 -- 172.21.15.40:0/155664478 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f513801b020 con 0x7f51480cf330 2024-03-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.582+0000 7f513fe00640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5124067660 0x7f5124069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.582+0000 7f513fe00640 1 -- 172.21.15.40:0/155664478 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5138086450 con 0x7f51480cf330 2024-03-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.582+0000 7f513e000640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5124067660 0x7f5124069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:20.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.582+0000 7f513e000640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5124067660 0x7f5124069b20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f51300ad000 tx=0x7f5130008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.586+0000 7f513fe00640 1 -- 172.21.15.40:0/155664478 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5138022030 con 0x7f51480cf330 2024-03-24T00:36:20.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.782+0000 7f514de00640 1 -- 172.21.15.40:0/155664478 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51480b3440 con 0x7f51480cf330 2024-03-24T00:36:20.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f513fe00640 1 -- 172.21.15.40:0/155664478 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5014 (secure 0 0 0) 0x7f5138010040 con 0x7f51480cf330 2024-03-24T00:36:20.788 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:20.788 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":87,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1097448819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1097448819},{"type":"v1","addr":"172.21.15.40:6834","nonce":1097448819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:36:07.163876+0000","modified":"2024-03-24T00:36:20.627770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7186,"mds_1":7180,"mds_2":6404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6404":{"gid":6404,"name":"b","rank":2,"incarnation":86,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/983819364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":983819364},{"type":"v1","addr":"172.21.15.53:6837","nonce":983819364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7180":{"gid":7180,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6835/2302548193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2302548193},{"type":"v1","addr":"172.21.15.40:6835","nonce":2302548193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7186":{"gid":7186,"name":"d","rank":0,"incarnation":78,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/1878888046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1878888046},{"type":"v1","addr":"172.21.15.53:6835","nonce":1878888046}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7186,"qdb_cluster":[6404,7180,7186]},"id":6}]} 2024-03-24T00:36:20.788 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 87 2024-03-24T00:36:20.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f514de00640 1 -- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5124067660 msgr2=0x7f5124069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f514de00640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5124067660 0x7f5124069b20 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f51300ad000 tx=0x7f5130008040 comp rx=0 tx=0).stop 2024-03-24T00:36:20.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f514de00640 1 -- 172.21.15.40:0/155664478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51480cf330 msgr2=0x7f5148151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:20.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f514de00640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51480cf330 0x7f5148151770 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f513800efc0 tx=0x7f513800c490 comp rx=0 tx=0).stop 2024-03-24T00:36:20.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f514de00640 1 -- 172.21.15.40:0/155664478 shutdown_connections 2024-03-24T00:36:20.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f514de00640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5124067660 0x7f5124069b20 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f514de00640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f514815c1d0 0x7f51480b5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f514de00640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f514815bdf0 0x7f51480c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f514de00640 1 --2- 172.21.15.40:0/155664478 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f51480cf330 0x7f5148151770 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:20.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f514de00640 1 -- 172.21.15.40:0/155664478 >> 172.21.15.40:0/155664478 conn(0x7f51480b37f0 msgr2=0x7f51480b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:20.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f514de00640 1 -- 172.21.15.40:0/155664478 shutdown_connections 2024-03-24T00:36:20.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:20.786+0000 7f514de00640 1 -- 172.21.15.40:0/155664478 wait complete. 2024-03-24T00:36:21.211 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 87, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:36:07.163876+0000', 'modified': '2024-03-24T00:36:20.627770+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 7186, 'mds_1': 7180, 'mds_2': 6404}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6404': {'gid': 6404, 'name': 'b', 'rank': 2, 'incarnation': 86, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.53:6837/983819364', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6836', 'nonce': 983819364}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 983819364}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7180': {'gid': 7180, 'name': 'a', 'rank': 1, 'incarnation': 84, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.40:6835/2302548193', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 2302548193}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 2302548193}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7186': {'gid': 7186, 'name': 'd', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.53:6835/1878888046', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 1878888046}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 1878888046}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7186, 'qdb_cluster': [6404, 7180, 7186]} 2024-03-24T00:36:21.211 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T00:36:21.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:36:21.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.350+0000 7f660ba00640 1 Processor -- start 2024-03-24T00:36:21.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.350+0000 7f660ba00640 1 -- start start 2024-03-24T00:36:21.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.350+0000 7f660ba00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f660415f600 0x7f66040b2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.350+0000 7f660ba00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66040d2830 0x7f66040aea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.350+0000 7f660ba00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66040b1e50 con 0x7f66040aefa0 2024-03-24T00:36:21.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.350+0000 7f660ba00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66040b3630 con 0x7f660415f600 2024-03-24T00:36:21.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.350+0000 7f660ba00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66040d2ca0 con 0x7f66040d2830 2024-03-24T00:36:21.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.350+0000 7f65fbe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66040d2830 0x7f66040aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.354+0000 7f65fbe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66040d2830 0x7f66040aea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:59500/0 (socket says 172.21.15.40:59500) 2024-03-24T00:36:21.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.354+0000 7f65fbe00640 1 -- 172.21.15.40:0/905670213 learned_addr learned my addr 172.21.15.40:0/905670213 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.354+0000 7f6609000640 1 --2- 172.21.15.40:0/905670213 >> v2:172.21.15.53:3300/0 conn(0x7f660415f600 0x7f66040b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.354+0000 7f65fbe00640 1 -- 172.21.15.40:0/905670213 >> v2:172.21.15.53:3300/0 conn(0x7f660415f600 msgr2=0x7f66040b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.354+0000 7f65fbe00640 1 --2- 172.21.15.40:0/905670213 >> v2:172.21.15.53:3300/0 conn(0x7f660415f600 0x7f66040b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.354+0000 7f65fbe00640 1 -- 172.21.15.40:0/905670213 >> v1:172.21.15.53:6789/0 conn(0x7f66040aefa0 legacy=0x7f6604164e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.354+0000 7f65fbe00640 1 -- 172.21.15.40:0/905670213 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66040b20f0 con 0x7f66040d2830 2024-03-24T00:36:21.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.354+0000 7f6609000640 1 --2- 172.21.15.40:0/905670213 >> v2:172.21.15.53:3300/0 conn(0x7f660415f600 0x7f66040b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:21.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.354+0000 7f65fbe00640 1 --2- 172.21.15.40:0/905670213 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66040d2830 0x7f66040aea60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009870 tx=0x7f65ec01ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a3236c4c48b0078 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.354+0000 7f6603e00640 1 -- 172.21.15.40:0/905670213 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0b5070 con 0x7f66040d2830 2024-03-24T00:36:21.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.354+0000 7f6603e00640 1 -- 172.21.15.40:0/905670213 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ec0a8a50 con 0x7f66040d2830 2024-03-24T00:36:21.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.354+0000 7f6603e00640 1 -- 172.21.15.40:0/905670213 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0a8e20 con 0x7f66040d2830 2024-03-24T00:36:21.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 -- 172.21.15.40:0/905670213 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66040d2830 msgr2=0x7f66040aea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 --2- 172.21.15.40:0/905670213 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66040d2830 0x7f66040aea60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009870 tx=0x7f65ec01ed50 comp rx=0 tx=0).stop 2024-03-24T00:36:21.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 -- 172.21.15.40:0/905670213 shutdown_connections 2024-03-24T00:36:21.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 --2- 172.21.15.40:0/905670213 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66040d2830 0x7f66040aea60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 --2- 172.21.15.40:0/905670213 >> v2:172.21.15.53:3300/0 conn(0x7f660415f600 0x7f66040b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 -- 172.21.15.40:0/905670213 >> 172.21.15.40:0/905670213 conn(0x7f66040d1d10 msgr2=0x7f66040d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 -- 172.21.15.40:0/905670213 shutdown_connections 2024-03-24T00:36:21.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 -- 172.21.15.40:0/905670213 wait complete. 2024-03-24T00:36:21.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 Processor -- start 2024-03-24T00:36:21.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 -- start start 2024-03-24T00:36:21.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66040aefa0 0x7f66041595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66040d2830 0x7f660415e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f660415f600 0x7f660415e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66040b2fa0 con 0x7f66040d2830 2024-03-24T00:36:21.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66040b2740 con 0x7f66040aefa0 2024-03-24T00:36:21.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f660ba00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f660415f9e0 con 0x7f660415f600 2024-03-24T00:36:21.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f65fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66040d2830 0x7f660415e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f6609000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66040aefa0 0x7f66041595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f6609000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66040aefa0 0x7f66041595c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:48326/0 (socket says 172.21.15.40:48326) 2024-03-24T00:36:21.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f65fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66040d2830 0x7f660415e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60564/0 (socket says 172.21.15.40:60564) 2024-03-24T00:36:21.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f6609000640 1 -- 172.21.15.40:0/3784359125 learned_addr learned my addr 172.21.15.40:0/3784359125 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.358+0000 7f6609a00640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f660415f600 0x7f660415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f65fbe00640 1 -- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f660415f600 msgr2=0x7f660415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f65fbe00640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f660415f600 0x7f660415e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f65fbe00640 1 -- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66040aefa0 msgr2=0x7f66041595c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f65fbe00640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66040aefa0 0x7f66041595c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f65fbe00640 1 -- 172.21.15.40:0/3784359125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6604159b00 con 0x7f66040d2830 2024-03-24T00:36:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f65fbe00640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66040d2830 0x7f660415e310 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0af000 tx=0x7f65ec01baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f6601600640 1 -- 172.21.15.40:0/3784359125 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0c0070 con 0x7f66040d2830 2024-03-24T00:36:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f6601600640 1 -- 172.21.15.40:0/3784359125 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ec0ae550 con 0x7f66040d2830 2024-03-24T00:36:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f6601600640 1 -- 172.21.15.40:0/3784359125 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0b5070 con 0x7f66040d2830 2024-03-24T00:36:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f660ba00640 1 -- 172.21.15.40:0/3784359125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66041f1610 con 0x7f66040d2830 2024-03-24T00:36:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f660ba00640 1 -- 172.21.15.40:0/3784359125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66041f18d0 con 0x7f66040d2830 2024-03-24T00:36:21.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f6601600640 1 -- 172.21.15.40:0/3784359125 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f65ec0b9070 con 0x7f66040d2830 2024-03-24T00:36:21.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f6601600640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f65e0067660 0x7f65e0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.362+0000 7f6601600640 1 -- 172.21.15.40:0/3784359125 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65ec122f90 con 0x7f66040d2830 2024-03-24T00:36:21.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f6609000640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f65e0067660 0x7f65e0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f6609000640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f65e0067660 0x7f65e0069b20 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f65f400bb70 tx=0x7f65f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f660eddd000 1 Processor -- start 2024-03-24T00:36:21.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f660eddd000 1 -- start start 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f660eddd000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e71920 0x55a3c2e71d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f660eddd000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e72240 0x55a3c2e27ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f660eddd000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e283d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f660eddd000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3c2c75680 con 0x55a3c2e72240 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f660eddd000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3c2bfcb00 con 0x55a3c2e27ff0 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f660eddd000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a3c2c71720 con 0x55a3c2e71920 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f65fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e72240 0x55a3c2e27ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f65fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e72240 0x55a3c2e27ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60578/0 (socket says 172.21.15.40:60578) 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f6609000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e71920 0x55a3c2e71d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f65fbe00640 1 -- 172.21.15.40:0/3926773301 learned_addr learned my addr 172.21.15.40:0/3926773301 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f6609a00640 1 --2- 172.21.15.40:0/3926773301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e283d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f65fbe00640 1 -- 172.21.15.40:0/3926773301 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e71920 msgr2=0x55a3c2e71d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f65fbe00640 1 --2- 172.21.15.40:0/3926773301 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e71920 0x55a3c2e71d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f65fbe00640 1 -- 172.21.15.40:0/3926773301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e27ff0 msgr2=0x55a3c2e283d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f65fbe00640 1 --2- 172.21.15.40:0/3926773301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e283d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.366+0000 7f65fbe00640 1 -- 172.21.15.40:0/3926773301 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3c2c4d940 con 0x55a3c2e72240 2024-03-24T00:36:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f65fbe00640 1 --2- 172.21.15.40:0/3926773301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e72240 0x55a3c2e27ab0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f65ec01e7c0 tx=0x7f65ec0f2df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7699f5e115372f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f65f9600640 1 -- 172.21.15.40:0/3926773301 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0f7530 con 0x55a3c2e72240 2024-03-24T00:36:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f65f9600640 1 -- 172.21.15.40:0/3926773301 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ec0fbb80 con 0x55a3c2e72240 2024-03-24T00:36:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f65f9600640 1 -- 172.21.15.40:0/3926773301 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0efb00 con 0x55a3c2e72240 2024-03-24T00:36:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f660eddd000 1 -- 172.21.15.40:0/3926773301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e72240 msgr2=0x55a3c2e27ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f660eddd000 1 --2- 172.21.15.40:0/3926773301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e72240 0x55a3c2e27ab0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f65ec01e7c0 tx=0x7f65ec0f2df0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f660eddd000 1 -- 172.21.15.40:0/3926773301 shutdown_connections 2024-03-24T00:36:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f660eddd000 1 --2- 172.21.15.40:0/3926773301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e283d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f660eddd000 1 --2- 172.21.15.40:0/3926773301 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e72240 0x55a3c2e27ab0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f660eddd000 1 --2- 172.21.15.40:0/3926773301 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e71920 0x55a3c2e71d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f660eddd000 1 -- 172.21.15.40:0/3926773301 >> 172.21.15.40:0/3926773301 conn(0x55a3c2e23fa0 msgr2=0x55a3c2e26390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f660eddd000 1 -- 172.21.15.40:0/3926773301 shutdown_connections 2024-03-24T00:36:21.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f660eddd000 1 -- 172.21.15.40:0/3926773301 wait complete. 2024-03-24T00:36:21.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f660eddd000 1 -- start start 2024-03-24T00:36:21.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.370+0000 7f660eddd000 1 Processor -- start 2024-03-24T00:36:21.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e56b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 0x55a3c2e570a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e72240 0x55a3c2e575e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3c2c74990 con 0x55a3c2e71920 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3c2c2fae0 con 0x55a3c2e27ff0 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a3c2c0d750 con 0x55a3c2e72240 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 10 client.0 _ll_register_callbacks cb 0x55a3c2e12ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 0x55a3c2e570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f6609a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e72240 0x55a3c2e575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 0x55a3c2e570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60588/0 (socket says 172.21.15.40:60588) 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65fbe00640 1 -- 172.21.15.40:0/2623570744 learned_addr learned my addr 172.21.15.40:0/2623570744 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f6609000640 1 --2- 172.21.15.40:0/2623570744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e56b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65fbe00640 1 -- 172.21.15.40:0/2623570744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e72240 msgr2=0x55a3c2e575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65fbe00640 1 --2- 172.21.15.40:0/2623570744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e72240 0x55a3c2e575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65fbe00640 1 -- 172.21.15.40:0/2623570744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e27ff0 msgr2=0x55a3c2e56b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65fbe00640 1 --2- 172.21.15.40:0/2623570744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e56b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65fbe00640 1 -- 172.21.15.40:0/2623570744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a3c2cbad80 con 0x55a3c2e71920 2024-03-24T00:36:21.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65fbe00640 1 --2- 172.21.15.40:0/2623570744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 0x55a3c2e570a0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009870 tx=0x7f65ec0c6040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bce0892d7f4d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65f2000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:36:21.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65f2000640 1 -- 172.21.15.40:0/2623570744 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec110060 con 0x55a3c2e71920 2024-03-24T00:36:21.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65f2000640 1 -- 172.21.15.40:0/2623570744 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65ec10b320 con 0x55a3c2e71920 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 1 -- 172.21.15.40:0/2623570744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65ec0fbb80 con 0x55a3c2e71920 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65f2000640 1 -- 172.21.15.40:0/2623570744 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65ec116050 con 0x55a3c2e71920 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 10 client.7641 fetch_fsmap learned FSMap version 87 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 1 -- 172.21.15.40:0/2623570744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x55a3c2bfcd10 con 0x55a3c2e71920 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65f2000640 1 -- 172.21.15.40:0/2623570744 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ec0fbb80 con 0x55a3c2e71920 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65f2000640 1 -- 172.21.15.40:0/2623570744 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 87) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f65ec119020 con 0x55a3c2e71920 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 10 client.7641 fetch_fsmap finished waiting for FSMap version 87 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 10 client.7641 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 20 client.7641 populate_metadata read hostname 'smithi040' 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 1 --2- 172.21.15.40:0/2623570744 >> [v2:172.21.15.53:6836/983819364,v1:172.21.15.53:6837/983819364] conn(0x55a3c2e819c0 0x55a3c2e83db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 4 client.7641 mds_command: new command op to 6404 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f660eddd000 1 -- 172.21.15.40:0/2623570744 --> [v2:172.21.15.53:6836/983819364,v1:172.21.15.53:6837/983819364] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a3c2c10b60 con 0x55a3c2e819c0 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f6609000640 1 --2- 172.21.15.40:0/2623570744 >> [v2:172.21.15.53:6836/983819364,v1:172.21.15.53:6837/983819364] conn(0x55a3c2e819c0 0x55a3c2e83db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f65f2000640 10 client.7641 ms_handle_connect on v2:172.21.15.53:6836/983819364 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.374+0000 7f6609000640 1 --2- 172.21.15.40:0/2623570744 >> [v2:172.21.15.53:6836/983819364,v1:172.21.15.53:6837/983819364] conn(0x55a3c2e819c0 0x55a3c2e83db0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2fd91ccfee817881 server_cookie=16ec374f809604a5 in_seq=0 out_seq=0 2024-03-24T00:36:21.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f65f2000640 1 -- 172.21.15.40:0/2623570744 <== mds.2 v2:172.21.15.53:6836/983819364 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a3c2c10b60 con 0x55a3c2e819c0 2024-03-24T00:36:21.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f65f2000640 10 client.7641 handle_command_reply: tid=0 2024-03-24T00:36:21.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f660eddd000 1 client.7641 shutdown 2024-03-24T00:36:21.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f660eddd000 1 -- 172.21.15.40:0/2623570744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 msgr2=0x55a3c2e570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f660eddd000 1 --2- 172.21.15.40:0/2623570744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 0x55a3c2e570a0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009870 tx=0x7f65ec0c6040 comp rx=0 tx=0).stop 2024-03-24T00:36:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f660eddd000 1 -- 172.21.15.40:0/2623570744 shutdown_connections 2024-03-24T00:36:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f660eddd000 1 --2- 172.21.15.40:0/2623570744 >> [v2:172.21.15.53:6836/983819364,v1:172.21.15.53:6837/983819364] conn(0x55a3c2e819c0 0x55a3c2e83db0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f660eddd000 1 --2- 172.21.15.40:0/2623570744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e72240 0x55a3c2e575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f65f2000640 0 client.7641 ms_handle_reset on v2:172.21.15.53:6836/983819364 2024-03-24T00:36:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f660eddd000 1 --2- 172.21.15.40:0/2623570744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 0x55a3c2e570a0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f660eddd000 1 --2- 172.21.15.40:0/2623570744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e56b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f660eddd000 1 -- 172.21.15.40:0/2623570744 >> 172.21.15.40:0/2623570744 conn(0x55a3c2e23fa0 msgr2=0x55a3c2e26c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f660eddd000 1 -- 172.21.15.40:0/2623570744 shutdown_connections 2024-03-24T00:36:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f660eddd000 1 -- 172.21.15.40:0/2623570744 wait complete. 2024-03-24T00:36:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.378+0000 7f660eddd000 20 client.7641 trim_cache size 0 max 16384 2024-03-24T00:36:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f660eddd000 1 Processor -- start 2024-03-24T00:36:21.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f660eddd000 1 -- start start 2024-03-24T00:36:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f660eddd000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e545d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f660eddd000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e71920 0x55a3c2e54b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f660eddd000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e72240 0x55a3c2e70410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f660eddd000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660406cd00 con 0x55a3c2e27ff0 2024-03-24T00:36:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f660eddd000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66040c9ac0 con 0x55a3c2e71920 2024-03-24T00:36:21.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f660eddd000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a3c2c1da80 con 0x55a3c2e72240 2024-03-24T00:36:21.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f6609000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f6609000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e545d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60594/0 (socket says 172.21.15.40:60594) 2024-03-24T00:36:21.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f6609000640 1 -- 172.21.15.40:0/546146271 learned_addr learned my addr 172.21.15.40:0/546146271 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f6609a00640 1 --2- 172.21.15.40:0/546146271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e72240 0x55a3c2e70410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f65fbe00640 1 --2- 172.21.15.40:0/546146271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e71920 0x55a3c2e54b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f6609000640 1 -- 172.21.15.40:0/546146271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e72240 msgr2=0x55a3c2e70410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f6609000640 1 --2- 172.21.15.40:0/546146271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e72240 0x55a3c2e70410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f6609000640 1 -- 172.21.15.40:0/546146271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e71920 msgr2=0x55a3c2e54b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f6609000640 1 --2- 172.21.15.40:0/546146271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e71920 0x55a3c2e54b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f6609000640 1 -- 172.21.15.40:0/546146271 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a3c2e2c180 con 0x55a3c2e27ff0 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f6609000640 1 --2- 172.21.15.40:0/546146271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e545d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x55a3c2c71b00 tx=0x7f65f4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97888b86ace0d101 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f65f8c00640 1 -- 172.21.15.40:0/546146271 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f401ce50 con 0x55a3c2e27ff0 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f65f8c00640 1 -- 172.21.15.40:0/546146271 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f4010db0 con 0x55a3c2e27ff0 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.394+0000 7f65f8c00640 1 -- 172.21.15.40:0/546146271 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f40059a0 con 0x55a3c2e27ff0 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 -- 172.21.15.40:0/546146271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e27ff0 msgr2=0x55a3c2e545d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 --2- 172.21.15.40:0/546146271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e545d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x55a3c2c71b00 tx=0x7f65f4016000 comp rx=0 tx=0).stop 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 -- 172.21.15.40:0/546146271 shutdown_connections 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 --2- 172.21.15.40:0/546146271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e72240 0x55a3c2e70410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 --2- 172.21.15.40:0/546146271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e71920 0x55a3c2e54b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 --2- 172.21.15.40:0/546146271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e27ff0 0x55a3c2e545d0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 -- 172.21.15.40:0/546146271 >> 172.21.15.40:0/546146271 conn(0x55a3c2e23fa0 msgr2=0x55a3c2e7ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 -- 172.21.15.40:0/546146271 shutdown_connections 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 -- 172.21.15.40:0/546146271 wait complete. 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 -- start start 2024-03-24T00:36:21.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 Processor -- start 2024-03-24T00:36:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e27ff0 0x55a3c2e27690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 0x55a3c2e84e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e72240 0x55a3c2e85340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3c2c0d750 con 0x55a3c2e71920 2024-03-24T00:36:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3c2d32700 con 0x55a3c2e72240 2024-03-24T00:36:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a3c2c74990 con 0x55a3c2e27ff0 2024-03-24T00:36:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 10 client.0 _ll_register_callbacks cb 0x7f65fc001940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:36:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 0x55a3c2e84e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 0x55a3c2e84e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60608/0 (socket says 172.21.15.40:60608) 2024-03-24T00:36:21.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f6609a00640 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e72240 0x55a3c2e85340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65fbe00640 1 -- 172.21.15.40:0/1616867054 learned_addr learned my addr 172.21.15.40:0/1616867054 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f6609000640 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e27ff0 0x55a3c2e27690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65fbe00640 1 -- 172.21.15.40:0/1616867054 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e27ff0 msgr2=0x55a3c2e27690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65fbe00640 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e27ff0 0x55a3c2e27690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65fbe00640 1 -- 172.21.15.40:0/1616867054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e72240 msgr2=0x55a3c2e85340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65fbe00640 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e72240 0x55a3c2e85340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65fbe00640 1 -- 172.21.15.40:0/1616867054 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a3c2c42540 con 0x55a3c2e71920 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65fbe00640 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 0x55a3c2e84e00 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0b0cd0 tx=0x7f65ec009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b621ce74726e1330 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65f2000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65f2000640 1 -- 172.21.15.40:0/1616867054 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec0fbb80 con 0x55a3c2e71920 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65f2000640 1 -- 172.21.15.40:0/1616867054 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65ec0f7250 con 0x55a3c2e71920 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 -- 172.21.15.40:0/1616867054 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65f4010db0 con 0x55a3c2e71920 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65f2000640 1 -- 172.21.15.40:0/1616867054 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65ec110020 con 0x55a3c2e71920 2024-03-24T00:36:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 10 client.7647 fetch_fsmap learned FSMap version 87 2024-03-24T00:36:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 1 -- 172.21.15.40:0/1616867054 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x55a3c2c094b0 con 0x55a3c2e71920 2024-03-24T00:36:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65f2000640 1 -- 172.21.15.40:0/1616867054 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f4010db0 con 0x55a3c2e71920 2024-03-24T00:36:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f65f2000640 1 -- 172.21.15.40:0/1616867054 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 87) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f65ec119040 con 0x55a3c2e71920 2024-03-24T00:36:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 10 client.7647 fetch_fsmap finished waiting for FSMap version 87 2024-03-24T00:36:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 10 client.7647 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:36:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.398+0000 7f660eddd000 20 client.7647 populate_metadata read hostname 'smithi040' 2024-03-24T00:36:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.53:6836/983819364,v1:172.21.15.53:6837/983819364] conn(0x55a3c2ed62c0 0x55a3c2ed66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 4 client.7647 mds_command: new command op to 6404 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:36:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 -- 172.21.15.40:0/1616867054 --> [v2:172.21.15.53:6836/983819364,v1:172.21.15.53:6837/983819364] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a3c2c82bc0 con 0x55a3c2ed62c0 2024-03-24T00:36:21.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f6609000640 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.53:6836/983819364,v1:172.21.15.53:6837/983819364] conn(0x55a3c2ed62c0 0x55a3c2ed66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f6609000640 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.53:6836/983819364,v1:172.21.15.53:6837/983819364] conn(0x55a3c2ed62c0 0x55a3c2ed66a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6725b5c79ced5437 server_cookie=7ee6b0f3bb3dccf2 in_seq=0 out_seq=0 2024-03-24T00:36:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f65f2000640 10 client.7647 ms_handle_connect on v2:172.21.15.53:6836/983819364 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f65f2000640 1 -- 172.21.15.40:0/1616867054 <== mds.2 v2:172.21.15.53:6836/983819364 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55a3c2c82bc0 con 0x55a3c2ed62c0 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f65f2000640 10 client.7647 handle_command_reply: tid=0 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 client.7647 shutdown 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 -- 172.21.15.40:0/1616867054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 msgr2=0x55a3c2e84e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 0x55a3c2e84e00 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0b0cd0 tx=0x7f65ec009240 comp rx=0 tx=0).stop 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 -- 172.21.15.40:0/1616867054 shutdown_connections 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.53:6836/983819364,v1:172.21.15.53:6837/983819364] conn(0x55a3c2ed62c0 0x55a3c2ed66a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a3c2e72240 0x55a3c2e85340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a3c2e71920 0x55a3c2e84e00 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f65f2000640 0 client.7647 ms_handle_reset on v2:172.21.15.53:6836/983819364 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 --2- 172.21.15.40:0/1616867054 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a3c2e27ff0 0x55a3c2e27690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 -- 172.21.15.40:0/1616867054 >> 172.21.15.40:0/1616867054 conn(0x55a3c2e23fa0 msgr2=0x55a3c2e55900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 -- 172.21.15.40:0/1616867054 shutdown_connections 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 1 -- 172.21.15.40:0/1616867054 wait complete. 2024-03-24T00:36:21.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660eddd000 20 client.7647 trim_cache size 0 max 16384 2024-03-24T00:36:21.409 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:36:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:36:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-03-24T00:36:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 6404, 2024-03-24T00:36:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:36:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:36:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:36:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 1.7768752219999999, 2024-03-24T00:36:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 87, 2024-03-24T00:36:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2024-03-24T00:36:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 83, 2024-03-24T00:36:21.409 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 19.584146316000002 2024-03-24T00:36:21.410 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:36:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660ba00640 1 -- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f65e0067660 msgr2=0x7f65e0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660ba00640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f65e0067660 0x7f65e0069b20 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f65f400bb70 tx=0x7f65f4008040 comp rx=0 tx=0).stop 2024-03-24T00:36:21.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660ba00640 1 -- 172.21.15.40:0/3784359125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66040d2830 msgr2=0x7f660415e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660ba00640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66040d2830 0x7f660415e310 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f65ec0af000 tx=0x7f65ec01baf0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660ba00640 1 -- 172.21.15.40:0/3784359125 shutdown_connections 2024-03-24T00:36:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660ba00640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f65e0067660 0x7f65e0069b20 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660ba00640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f660415f600 0x7f660415e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660ba00640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66040d2830 0x7f660415e310 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660ba00640 1 --2- 172.21.15.40:0/3784359125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66040aefa0 0x7f66041595c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660ba00640 1 -- 172.21.15.40:0/3784359125 >> 172.21.15.40:0/3784359125 conn(0x7f66040d1d10 msgr2=0x7f660415d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660ba00640 1 -- 172.21.15.40:0/3784359125 shutdown_connections 2024-03-24T00:36:21.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.402+0000 7f660ba00640 1 -- 172.21.15.40:0/3784359125 wait complete. 2024-03-24T00:36:21.425 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:36:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.562+0000 7f7e55a00640 1 Processor -- start 2024-03-24T00:36:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.562+0000 7f7e55a00640 1 -- start start 2024-03-24T00:36:21.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.562+0000 7f7e55a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e500d0400 0x7f7e500aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.562+0000 7f7e55a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7e50070970 0x7f7e500af210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.562+0000 7f7e55a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e50070650 con 0x7f7e5015f350 2024-03-24T00:36:21.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.562+0000 7f7e55a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e50160500 con 0x7f7e50070970 2024-03-24T00:36:21.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.562+0000 7f7e55a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50160a30 con 0x7f7e500d0400 2024-03-24T00:36:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.562+0000 7f7e4f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7e50070970 0x7f7e500af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.562+0000 7f7e47e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e500d0400 0x7f7e500aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.562+0000 7f7e4ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7e5015f350 0x7f7e500cfcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52638/0 (socket says 172.21.15.40:52638) 2024-03-24T00:36:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.562+0000 7f7e4ea00640 1 -- 172.21.15.40:0/2812209131 learned_addr learned my addr 172.21.15.40:0/2812209131 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e47e00640 1 -- 172.21.15.40:0/2812209131 >> v2:172.21.15.40:3300/0 conn(0x7f7e50070970 msgr2=0x7f7e500af210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e47e00640 1 --2- 172.21.15.40:0/2812209131 >> v2:172.21.15.40:3300/0 conn(0x7f7e50070970 0x7f7e500af210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e47e00640 1 -- 172.21.15.40:0/2812209131 >> v1:172.21.15.40:6789/0 conn(0x7f7e5015f350 legacy=0x7f7e500cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e47e00640 1 -- 172.21.15.40:0/2812209131 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e500d0a50 con 0x7f7e500d0400 2024-03-24T00:36:21.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e47e00640 1 --2- 172.21.15.40:0/2812209131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e500d0400 0x7f7e500aecd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7e380097c0 tx=0x7f7e3801c350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eee5ce24a6f11f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e4e000640 1 -- 172.21.15.40:0/2812209131 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e380b0070 con 0x7f7e500d0400 2024-03-24T00:36:21.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e4e000640 1 -- 172.21.15.40:0/2812209131 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e38004230 con 0x7f7e500d0400 2024-03-24T00:36:21.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e4e000640 1 -- 172.21.15.40:0/2812209131 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e38003620 con 0x7f7e500d0400 2024-03-24T00:36:21.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e55a00640 1 -- 172.21.15.40:0/2812209131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e500d0400 msgr2=0x7f7e500aecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e55a00640 1 --2- 172.21.15.40:0/2812209131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e500d0400 0x7f7e500aecd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7e380097c0 tx=0x7f7e3801c350 comp rx=0 tx=0).stop 2024-03-24T00:36:21.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e55a00640 1 -- 172.21.15.40:0/2812209131 shutdown_connections 2024-03-24T00:36:21.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e55a00640 1 --2- 172.21.15.40:0/2812209131 >> v2:172.21.15.40:3300/0 conn(0x7f7e50070970 0x7f7e500af210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e55a00640 1 --2- 172.21.15.40:0/2812209131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e500d0400 0x7f7e500aecd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e55a00640 1 -- 172.21.15.40:0/2812209131 >> 172.21.15.40:0/2812209131 conn(0x7f7e500d1c80 msgr2=0x7f7e500d2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e55a00640 1 -- 172.21.15.40:0/2812209131 shutdown_connections 2024-03-24T00:36:21.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.566+0000 7f7e55a00640 1 -- 172.21.15.40:0/2812209131 wait complete. 2024-03-24T00:36:21.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e55a00640 1 Processor -- start 2024-03-24T00:36:21.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e55a00640 1 -- start start 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e55a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e50070970 0x7f7e50148070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e55a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e500d0400 0x7f7e500c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e55a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e5015f350 0x7f7e500cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e55a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5015f8b0 con 0x7f7e500d0400 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e55a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5015f730 con 0x7f7e5015f350 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e55a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50168e30 con 0x7f7e50070970 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e47e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e500d0400 0x7f7e500c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e4ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e50070970 0x7f7e50148070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e47e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e500d0400 0x7f7e500c5410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60636/0 (socket says 172.21.15.40:60636) 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e47e00640 1 -- 172.21.15.40:0/4194070318 learned_addr learned my addr 172.21.15.40:0/4194070318 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e4f400640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e5015f350 0x7f7e500cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e47e00640 1 -- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e50070970 msgr2=0x7f7e50148070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e47e00640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e50070970 0x7f7e50148070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e47e00640 1 -- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e5015f350 msgr2=0x7f7e500cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e47e00640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e5015f350 0x7f7e500cb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e47e00640 1 -- 172.21.15.40:0/4194070318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e501485b0 con 0x7f7e500d0400 2024-03-24T00:36:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e47e00640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e500d0400 0x7f7e500c5410 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7e380ae040 tx=0x7f7e380190f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e46a00640 1 -- 172.21.15.40:0/4194070318 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e380b9070 con 0x7f7e500d0400 2024-03-24T00:36:21.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e46a00640 1 -- 172.21.15.40:0/4194070318 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e38003df0 con 0x7f7e500d0400 2024-03-24T00:36:21.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e55a00640 1 -- 172.21.15.40:0/4194070318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e500ca520 con 0x7f7e500d0400 2024-03-24T00:36:21.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e46a00640 1 -- 172.21.15.40:0/4194070318 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e380b0040 con 0x7f7e500d0400 2024-03-24T00:36:21.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.570+0000 7f7e55a00640 1 -- 172.21.15.40:0/4194070318 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e500caae0 con 0x7f7e500d0400 2024-03-24T00:36:21.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.574+0000 7f7e46a00640 1 -- 172.21.15.40:0/4194070318 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7e380bc070 con 0x7f7e500d0400 2024-03-24T00:36:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.574+0000 7f7e46a00640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e2c067690 0x7f7e2c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.574+0000 7f7e46a00640 1 -- 172.21.15.40:0/4194070318 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e3811faa0 con 0x7f7e500d0400 2024-03-24T00:36:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.574+0000 7f7e4ea00640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e2c067690 0x7f7e2c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.574+0000 7f7e58e3c000 1 Processor -- start 2024-03-24T00:36:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 -- start start 2024-03-24T00:36:21.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e4ea00640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e2c067690 0x7f7e2c069b50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7e40000f70 tx=0x7f7e40007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4ede1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4ed97ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4ed983d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed4ebe5680 con 0x55ed4ed97ff0 2024-03-24T00:36:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed4eb6cb00 con 0x55ed4ede2240 2024-03-24T00:36:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed4ebe1720 con 0x55ed4ede1920 2024-03-24T00:36:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e4f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4ed983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e4ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4ede1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e4f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4ed983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60638/0 (socket says 172.21.15.40:60638) 2024-03-24T00:36:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e4f400640 1 -- 172.21.15.40:0/1382651332 learned_addr learned my addr 172.21.15.40:0/1382651332 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e47e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4ed97ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e4f400640 1 -- 172.21.15.40:0/1382651332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 msgr2=0x55ed4ede1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e4f400640 1 --2- 172.21.15.40:0/1382651332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4ede1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e4f400640 1 -- 172.21.15.40:0/1382651332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 msgr2=0x55ed4ed97ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e4f400640 1 --2- 172.21.15.40:0/1382651332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4ed97ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e4f400640 1 -- 172.21.15.40:0/1382651332 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed4ebbd940 con 0x55ed4ed97ff0 2024-03-24T00:36:21.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e47e00640 1 --2- 172.21.15.40:0/1382651332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4ed97ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e4f400640 1 --2- 172.21.15.40:0/1382651332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4ed983d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f7e480113f0 tx=0x7f7e48009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1f838041672325 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e3f400640 1 -- 172.21.15.40:0/1382651332 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48015900 con 0x55ed4ed97ff0 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e3f400640 1 -- 172.21.15.40:0/1382651332 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e48015aa0 con 0x55ed4ed97ff0 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e3f400640 1 -- 172.21.15.40:0/1382651332 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e48015e50 con 0x55ed4ed97ff0 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1382651332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 msgr2=0x55ed4ed983d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/1382651332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4ed983d0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f7e480113f0 tx=0x7f7e48009750 comp rx=0 tx=0).stop 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1382651332 shutdown_connections 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/1382651332 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4ed983d0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/1382651332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4ed97ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/1382651332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4ede1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1382651332 >> 172.21.15.40:0/1382651332 conn(0x55ed4ed93fa0 msgr2=0x55ed4ed96390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1382651332 shutdown_connections 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1382651332 wait complete. 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 -- start start 2024-03-24T00:36:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.578+0000 7f7e58e3c000 1 Processor -- start 2024-03-24T00:36:21.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4edc6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4edc7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4edc77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed4ebe4990 con 0x55ed4ed97ff0 2024-03-24T00:36:21.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed4eb9fae0 con 0x55ed4ede2240 2024-03-24T00:36:21.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed4eb7d750 con 0x55ed4ede1920 2024-03-24T00:36:21.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 10 client.0 _ll_register_callbacks cb 0x55ed4ed82ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:36:21.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e4ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4edc6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e47e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4edc7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e47e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4edc7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:59574/0 (socket says 172.21.15.40:59574) 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e4ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4edc6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60652/0 (socket says 172.21.15.40:60652) 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e47e00640 1 -- 172.21.15.40:0/1857942781 learned_addr learned my addr 172.21.15.40:0/1857942781 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e4f400640 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4edc77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e47e00640 1 -- 172.21.15.40:0/1857942781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 msgr2=0x55ed4edc77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e47e00640 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4edc77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e47e00640 1 -- 172.21.15.40:0/1857942781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 msgr2=0x55ed4edc6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e47e00640 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4edc6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e47e00640 1 -- 172.21.15.40:0/1857942781 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed4ec2ad80 con 0x55ed4ede1920 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e4ea00640 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4edc6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e47e00640 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4edc7280 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7e380097c0 tx=0x7f7e380efa30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e9f7134f1063ffa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e33e00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e33e00640 1 -- 172.21.15.40:0/1857942781 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e380f84a0 con 0x55ed4ede1920 2024-03-24T00:36:21.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e33e00640 1 -- 172.21.15.40:0/1857942781 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e380ef520 con 0x55ed4ede1920 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1857942781 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7e48015aa0 con 0x55ed4ede1920 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e33e00640 1 -- 172.21.15.40:0/1857942781 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e380f4170 con 0x55ed4ede1920 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 10 client.6575 fetch_fsmap learned FSMap version 87 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1857942781 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x55ed4eb6cd10 con 0x55ed4ede1920 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e33e00640 1 -- 172.21.15.40:0/1857942781 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e48015aa0 con 0x55ed4ede1920 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e33e00640 1 -- 172.21.15.40:0/1857942781 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 87) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f7e380f6100 con 0x55ed4ede1920 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 10 client.6575 fetch_fsmap finished waiting for FSMap version 87 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 10 client.6575 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 20 client.6575 populate_metadata read hostname 'smithi040' 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.40:6833/2302548193,v1:172.21.15.40:6835/2302548193] conn(0x55ed4edc8060 0x55ed4edf3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 4 client.6575 mds_command: new command op to 7180 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1857942781 --> [v2:172.21.15.40:6833/2302548193,v1:172.21.15.40:6835/2302548193] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed4eb80b60 con 0x55ed4edc8060 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e4ea00640 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.40:6833/2302548193,v1:172.21.15.40:6835/2302548193] conn(0x55ed4edc8060 0x55ed4edf3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e33e00640 10 client.6575 ms_handle_connect on v2:172.21.15.40:6833/2302548193 2024-03-24T00:36:21.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.582+0000 7f7e4ea00640 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.40:6833/2302548193,v1:172.21.15.40:6835/2302548193] conn(0x55ed4edc8060 0x55ed4edf3ea0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7a45d26012d7b56a server_cookie=32853120c2a4cda9 in_seq=0 out_seq=0 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e33e00640 1 -- 172.21.15.40:0/1857942781 <== mds.1 v2:172.21.15.40:6833/2302548193 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ed4eb80b60 con 0x55ed4edc8060 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e33e00640 10 client.6575 handle_command_reply: tid=0 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e58e3c000 1 client.6575 shutdown 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1857942781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 msgr2=0x55ed4edc7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4edc7280 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7e380097c0 tx=0x7f7e380efa30 comp rx=0 tx=0).stop 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1857942781 shutdown_connections 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.40:6833/2302548193,v1:172.21.15.40:6835/2302548193] conn(0x55ed4edc8060 0x55ed4edf3ea0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4edc77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4edc7280 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/1857942781 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4edc6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e33e00640 0 client.6575 ms_handle_reset on v2:172.21.15.40:6833/2302548193 2024-03-24T00:36:21.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1857942781 >> 172.21.15.40:0/1857942781 conn(0x55ed4ed93fa0 msgr2=0x55ed4ed96c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1857942781 shutdown_connections 2024-03-24T00:36:21.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e58e3c000 1 -- 172.21.15.40:0/1857942781 wait complete. 2024-03-24T00:36:21.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.586+0000 7f7e58e3c000 20 client.6575 trim_cache size 0 max 16384 2024-03-24T00:36:21.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e58e3c000 1 Processor -- start 2024-03-24T00:36:21.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e58e3c000 1 -- start start 2024-03-24T00:36:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e58e3c000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4ede0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e58e3c000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4edc5900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e58e3c000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4edc5e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e58e3c000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50203670 con 0x55ed4ed97ff0 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e58e3c000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e50203010 con 0x55ed4ede2240 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e58e3c000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e500c8d30 con 0x55ed4ede1920 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e4ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4ede0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e47e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4edc5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e4f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4edc5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e4ea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4ede0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60664/0 (socket says 172.21.15.40:60664) 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e4ea00640 1 -- 172.21.15.40:0/3957411459 learned_addr learned my addr 172.21.15.40:0/3957411459 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e4ea00640 1 -- 172.21.15.40:0/3957411459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 msgr2=0x55ed4edc5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e4ea00640 1 --2- 172.21.15.40:0/3957411459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4edc5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e4ea00640 1 -- 172.21.15.40:0/3957411459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 msgr2=0x55ed4edc5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e4ea00640 1 --2- 172.21.15.40:0/3957411459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4edc5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e4ea00640 1 -- 172.21.15.40:0/3957411459 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed4ebb2540 con 0x55ed4ed97ff0 2024-03-24T00:36:21.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e4ea00640 1 --2- 172.21.15.40:0/3957411459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4ede0e20 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f7e40001270 tx=0x7f7e40013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac91037016e3565 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e3ea00640 1 -- 172.21.15.40:0/3957411459 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4002f070 con 0x55ed4ed97ff0 2024-03-24T00:36:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e3ea00640 1 -- 172.21.15.40:0/3957411459 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e40029e30 con 0x55ed4ed97ff0 2024-03-24T00:36:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.602+0000 7f7e3ea00640 1 -- 172.21.15.40:0/3957411459 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e40026030 con 0x55ed4ed97ff0 2024-03-24T00:36:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3957411459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 msgr2=0x55ed4ede0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/3957411459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4ede0e20 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f7e40001270 tx=0x7f7e40013000 comp rx=0 tx=0).stop 2024-03-24T00:36:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3957411459 shutdown_connections 2024-03-24T00:36:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/3957411459 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ede2240 0x55ed4edc5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/3957411459 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4edc5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/3957411459 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ed97ff0 0x55ed4ede0e20 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3957411459 >> 172.21.15.40:0/3957411459 conn(0x55ed4ed93fa0 msgr2=0x55ed4ede0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3957411459 shutdown_connections 2024-03-24T00:36:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3957411459 wait complete. 2024-03-24T00:36:21.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 -- start start 2024-03-24T00:36:21.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 Processor -- start 2024-03-24T00:36:21.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ed97ff0 0x55ed4ee471b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4ee49700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ede2240 0x55ed4ee4bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed4eca2700 con 0x55ed4ede2240 2024-03-24T00:36:21.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed4eb8da80 con 0x55ed4ed97ff0 2024-03-24T00:36:21.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed4eb7d750 con 0x55ed4ede1920 2024-03-24T00:36:21.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 10 client.0 _ll_register_callbacks cb 0x7f7e38009190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:36:21.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e4f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ede2240 0x55ed4ee4bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e4f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ede2240 0x55ed4ee4bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60672/0 (socket says 172.21.15.40:60672) 2024-03-24T00:36:21.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e4f400640 1 -- 172.21.15.40:0/3250410297 learned_addr learned my addr 172.21.15.40:0/3250410297 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e4ea00640 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ed97ff0 0x55ed4ee471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e47e00640 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4ee49700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e4f400640 1 -- 172.21.15.40:0/3250410297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 msgr2=0x55ed4ee49700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e4f400640 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4ee49700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e4f400640 1 -- 172.21.15.40:0/3250410297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ed97ff0 msgr2=0x55ed4ee471b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e4f400640 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ed97ff0 0x55ed4ee471b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e4f400640 1 -- 172.21.15.40:0/3250410297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed4eb794b0 con 0x55ed4ede2240 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e4f400640 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ede2240 0x55ed4ee4bc50 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f7e48012010 tx=0x7f7e48004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc607164cdeb4b35 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e33e00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e33e00640 1 -- 172.21.15.40:0/3250410297 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4802e060 con 0x55ed4ede2240 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e33e00640 1 -- 172.21.15.40:0/3250410297 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e48015c30 con 0x55ed4ede2240 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3250410297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7e40029e30 con 0x55ed4ede2240 2024-03-24T00:36:21.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e33e00640 1 -- 172.21.15.40:0/3250410297 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e48034050 con 0x55ed4ede2240 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 10 client.7668 fetch_fsmap learned FSMap version 87 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3250410297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x55ed4eb8c2f0 con 0x55ed4ede2240 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e33e00640 1 -- 172.21.15.40:0/3250410297 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e40029e30 con 0x55ed4ede2240 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e33e00640 1 -- 172.21.15.40:0/3250410297 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 87) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f7e48037020 con 0x55ed4ede2240 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.606+0000 7f7e58e3c000 10 client.7668 fetch_fsmap finished waiting for FSMap version 87 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 10 client.7668 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 20 client.7668 populate_metadata read hostname 'smithi040' 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.40:6833/2302548193,v1:172.21.15.40:6835/2302548193] conn(0x55ed4ee4c3a0 0x55ed4ee4e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 4 client.7668 mds_command: new command op to 7180 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3250410297 --> [v2:172.21.15.40:6833/2302548193,v1:172.21.15.40:6835/2302548193] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ed4ebf2bc0 con 0x55ed4ee4c3a0 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e4ea00640 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.40:6833/2302548193,v1:172.21.15.40:6835/2302548193] conn(0x55ed4ee4c3a0 0x55ed4ee4e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e4ea00640 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.40:6833/2302548193,v1:172.21.15.40:6835/2302548193] conn(0x55ed4ee4c3a0 0x55ed4ee4e790 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c2b69e6f3bdff14d server_cookie=682ea2c57ac7d0b3 in_seq=0 out_seq=0 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e33e00640 10 client.7668 ms_handle_connect on v2:172.21.15.40:6833/2302548193 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e33e00640 1 -- 172.21.15.40:0/3250410297 <== mds.1 v2:172.21.15.40:6833/2302548193 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55ed4ebf2bc0 con 0x55ed4ee4c3a0 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e33e00640 10 client.7668 handle_command_reply: tid=0 2024-03-24T00:36:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 client.7668 shutdown 2024-03-24T00:36:21.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3250410297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ede2240 msgr2=0x55ed4ee4bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ede2240 0x55ed4ee4bc50 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f7e48012010 tx=0x7f7e48004ac0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3250410297 shutdown_connections 2024-03-24T00:36:21.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.40:6833/2302548193,v1:172.21.15.40:6835/2302548193] conn(0x55ed4ee4c3a0 0x55ed4ee4e790 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55ed4ede2240 0x55ed4ee4bc50 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55ed4ede1920 0x55ed4ee49700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e33e00640 0 client.7668 ms_handle_reset on v2:172.21.15.40:6833/2302548193 2024-03-24T00:36:21.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 --2- 172.21.15.40:0/3250410297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ed4ed97ff0 0x55ed4ee471b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3250410297 >> 172.21.15.40:0/3250410297 conn(0x55ed4ed93fa0 msgr2=0x55ed4ede0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3250410297 shutdown_connections 2024-03-24T00:36:21.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 1 -- 172.21.15.40:0/3250410297 wait complete. 2024-03-24T00:36:21.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e58e3c000 20 client.7668 trim_cache size 0 max 16384 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 7180, 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 2.9945933990000002, 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 87, 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 83, 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 25.761650170999999 2024-03-24T00:36:21.617 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:36:21.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e55a00640 1 -- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e2c067690 msgr2=0x7f7e2c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e55a00640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e2c067690 0x7f7e2c069b50 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f7e40000f70 tx=0x7f7e40007e90 comp rx=0 tx=0).stop 2024-03-24T00:36:21.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e55a00640 1 -- 172.21.15.40:0/4194070318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e500d0400 msgr2=0x7f7e500c5410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e55a00640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e500d0400 0x7f7e500c5410 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7e380ae040 tx=0x7f7e380190f0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e55a00640 1 -- 172.21.15.40:0/4194070318 shutdown_connections 2024-03-24T00:36:21.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e55a00640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e2c067690 0x7f7e2c069b50 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e55a00640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e5015f350 0x7f7e500cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e55a00640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7e500d0400 0x7f7e500c5410 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e55a00640 1 --2- 172.21.15.40:0/4194070318 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e50070970 0x7f7e50148070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e55a00640 1 -- 172.21.15.40:0/4194070318 >> 172.21.15.40:0/4194070318 conn(0x7f7e500d1c80 msgr2=0x7f7e500b6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e55a00640 1 -- 172.21.15.40:0/4194070318 shutdown_connections 2024-03-24T00:36:21.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.610+0000 7f7e55a00640 1 -- 172.21.15.40:0/4194070318 wait complete. 2024-03-24T00:36:21.634 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T00:36:21.722 INFO:tasks.ceph.mds.d.smithi053.stderr:2024-03-24T00:36:21.720+0000 7fc819600640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:36:21.722 INFO:tasks.ceph.mds.d.smithi053.stderr:2024-03-24T00:36:21.720+0000 7fc819600640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-24T00:36:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.770+0000 7ffb1aa00640 1 Processor -- start 2024-03-24T00:36:21.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.770+0000 7ffb1aa00640 1 -- start start 2024-03-24T00:36:21.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb1aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb14070be0 0x7ffb14070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb1aa00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb140af880 con 0x7ffb1406f4e0 2024-03-24T00:36:21.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb1aa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb140d27b0 con 0x7ffb1406f100 2024-03-24T00:36:21.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb1aa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb140b1490 con 0x7ffb14070be0 2024-03-24T00:36:21.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb0be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ffb1406f4e0 0x7ffb140af170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52644/0 (socket says 172.21.15.40:52644) 2024-03-24T00:36:21.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb0be00640 1 -- 172.21.15.40:0/4279259401 learned_addr learned my addr 172.21.15.40:0/4279259401 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:21.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb18a00640 1 --2- 172.21.15.40:0/4279259401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb14070be0 0x7ffb14070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb13400640 1 -- 172.21.15.40:0/4279259401 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405293652 0 0) 0x7ffb140af880 con 0x7ffb1406f4e0 2024-03-24T00:36:21.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb13400640 1 -- 172.21.15.40:0/4279259401 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf8003530 con 0x7ffb1406f4e0 2024-03-24T00:36:21.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb13400640 1 -- 172.21.15.40:0/4279259401 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780351030 0 0) 0x7ffb140d27b0 con 0x7ffb1406f100 2024-03-24T00:36:21.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb18a00640 1 -- 172.21.15.40:0/4279259401 >> v1:172.21.15.53:6789/0 conn(0x7ffb1406f100 legacy=0x7ffb140aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb18a00640 1 -- 172.21.15.40:0/4279259401 >> v1:172.21.15.40:6789/0 conn(0x7ffb1406f4e0 legacy=0x7ffb140af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.774+0000 7ffb18a00640 1 -- 172.21.15.40:0/4279259401 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1406fa70 con 0x7ffb14070be0 2024-03-24T00:36:21.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb18a00640 1 --2- 172.21.15.40:0/4279259401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb14070be0 0x7ffb14070030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009dc0 tx=0x7ffb0c01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=903b4ae48878f30a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb13400640 1 -- 172.21.15.40:0/4279259401 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c0b2070 con 0x7ffb14070be0 2024-03-24T00:36:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb13400640 1 -- 172.21.15.40:0/4279259401 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb0c0a5850 con 0x7ffb14070be0 2024-03-24T00:36:21.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb13400640 1 -- 172.21.15.40:0/4279259401 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb0c0a5c20 con 0x7ffb14070be0 2024-03-24T00:36:21.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb1aa00640 1 -- 172.21.15.40:0/4279259401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb14070be0 msgr2=0x7ffb14070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb1aa00640 1 --2- 172.21.15.40:0/4279259401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb14070be0 0x7ffb14070030 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ffb0c009dc0 tx=0x7ffb0c01c990 comp rx=0 tx=0).stop 2024-03-24T00:36:21.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb1aa00640 1 -- 172.21.15.40:0/4279259401 shutdown_connections 2024-03-24T00:36:21.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb1aa00640 1 --2- 172.21.15.40:0/4279259401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb14070be0 0x7ffb14070030 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb1aa00640 1 -- 172.21.15.40:0/4279259401 >> 172.21.15.40:0/4279259401 conn(0x7ffb140d1c90 msgr2=0x7ffb140d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb1aa00640 1 -- 172.21.15.40:0/4279259401 shutdown_connections 2024-03-24T00:36:21.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb1aa00640 1 -- 172.21.15.40:0/4279259401 wait complete. 2024-03-24T00:36:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb1aa00640 1 Processor -- start 2024-03-24T00:36:21.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb1aa00640 1 -- start start 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb1aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1406f100 0x7ffb140b3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb1aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1406f4e0 0x7ffb141534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.778+0000 7ffb1aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb14070be0 0x7ffb140b1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb1aa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb140d0a70 con 0x7ffb1406f100 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb1aa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb140d08f0 con 0x7ffb14070be0 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb1aa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb14169c30 con 0x7ffb1406f4e0 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1406f100 0x7ffb140b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb0be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1406f4e0 0x7ffb141534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1406f100 0x7ffb140b3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60682/0 (socket says 172.21.15.40:60682) 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb13e00640 1 -- 172.21.15.40:0/2441842137 learned_addr learned my addr 172.21.15.40:0/2441842137 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb18a00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb14070be0 0x7ffb140b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb13e00640 1 -- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1406f4e0 msgr2=0x7ffb141534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb13e00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1406f4e0 0x7ffb141534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb13e00640 1 -- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb14070be0 msgr2=0x7ffb140b1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb13e00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb14070be0 0x7ffb140b1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb13e00640 1 -- 172.21.15.40:0/2441842137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb140b2340 con 0x7ffb1406f100 2024-03-24T00:36:21.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb13e00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1406f100 0x7ffb140b3100 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400c410 tx=0x7ffb0400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb10c00640 1 -- 172.21.15.40:0/2441842137 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04017070 con 0x7ffb1406f100 2024-03-24T00:36:21.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb10c00640 1 -- 172.21.15.40:0/2441842137 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb04003600 con 0x7ffb1406f100 2024-03-24T00:36:21.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb1aa00640 1 -- 172.21.15.40:0/2441842137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb140b2620 con 0x7ffb1406f100 2024-03-24T00:36:21.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb10c00640 1 -- 172.21.15.40:0/2441842137 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb040039f0 con 0x7ffb1406f100 2024-03-24T00:36:21.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb1aa00640 1 -- 172.21.15.40:0/2441842137 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb140b2be0 con 0x7ffb1406f100 2024-03-24T00:36:21.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.782+0000 7ffb10c00640 1 -- 172.21.15.40:0/2441842137 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ffb0401a070 con 0x7ffb1406f100 2024-03-24T00:36:21.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.786+0000 7ffb10c00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffaf0067660 0x7ffaf0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.786+0000 7ffb10c00640 1 -- 172.21.15.40:0/2441842137 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb04084c90 con 0x7ffb1406f100 2024-03-24T00:36:21.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.786+0000 7ffb0be00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffaf0067660 0x7ffaf0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.786+0000 7ffb0be00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffaf0067660 0x7ffaf0069b20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ffafc007fd0 tx=0x7ffafc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.786+0000 7ffb1de94000 1 Processor -- start 2024-03-24T00:36:21.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.786+0000 7ffb1de94000 1 -- start start 2024-03-24T00:36:21.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.786+0000 7ffb1de94000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 0x55c6b2907d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.786+0000 7ffb1de94000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28bdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.786+0000 7ffb1de94000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b28bdff0 0x55c6b28be3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.786+0000 7ffb1de94000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b270b680 con 0x55c6b2907920 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.786+0000 7ffb1de94000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b2692b00 con 0x55c6b28bdff0 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.786+0000 7ffb1de94000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6b2707720 con 0x55c6b2908240 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 0x55c6b2907d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 0x55c6b2907d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60698/0 (socket says 172.21.15.40:60698) 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb13e00640 1 -- 172.21.15.40:0/3127531124 learned_addr learned my addr 172.21.15.40:0/3127531124 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb0be00640 1 --2- 172.21.15.40:0/3127531124 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28bdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb18a00640 1 --2- 172.21.15.40:0/3127531124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b28bdff0 0x55c6b28be3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb13e00640 1 -- 172.21.15.40:0/3127531124 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 msgr2=0x55c6b28bdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb13e00640 1 --2- 172.21.15.40:0/3127531124 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28bdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb13e00640 1 -- 172.21.15.40:0/3127531124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b28bdff0 msgr2=0x55c6b28be3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb13e00640 1 --2- 172.21.15.40:0/3127531124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b28bdff0 0x55c6b28be3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb13e00640 1 -- 172.21.15.40:0/3127531124 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6b26e3940 con 0x55c6b2907920 2024-03-24T00:36:21.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb13e00640 1 --2- 172.21.15.40:0/3127531124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 0x55c6b2907d00 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7ffb04051630 tx=0x7ffb040552e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5831f4f58ecbd087 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb08c00640 1 -- 172.21.15.40:0/3127531124 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb040532c0 con 0x55c6b2907920 2024-03-24T00:36:21.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb08c00640 1 -- 172.21.15.40:0/3127531124 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb04053520 con 0x55c6b2907920 2024-03-24T00:36:21.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb08c00640 1 -- 172.21.15.40:0/3127531124 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04003b90 con 0x55c6b2907920 2024-03-24T00:36:21.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 -- 172.21.15.40:0/3127531124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 msgr2=0x55c6b2907d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 --2- 172.21.15.40:0/3127531124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 0x55c6b2907d00 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7ffb04051630 tx=0x7ffb040552e0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 -- 172.21.15.40:0/3127531124 shutdown_connections 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 --2- 172.21.15.40:0/3127531124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b28bdff0 0x55c6b28be3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 --2- 172.21.15.40:0/3127531124 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28bdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 --2- 172.21.15.40:0/3127531124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 0x55c6b2907d00 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 -- 172.21.15.40:0/3127531124 >> 172.21.15.40:0/3127531124 conn(0x55c6b28b9fa0 msgr2=0x55c6b28bc390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 -- 172.21.15.40:0/3127531124 shutdown_connections 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 -- 172.21.15.40:0/3127531124 wait complete. 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 -- start start 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 Processor -- start 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 0x55c6b28ecd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b2907920 0x55c6b28ed280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28ed7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b270a990 con 0x55c6b28bdff0 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b26c5ae0 con 0x55c6b2907920 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6b26a3750 con 0x55c6b2908240 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.790+0000 7ffb1de94000 10 client.0 _ll_register_callbacks cb 0x55c6b28a8ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 0x55c6b28ecd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 0x55c6b28ecd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60706/0 (socket says 172.21.15.40:60706) 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb13e00640 1 -- 172.21.15.40:0/2295576694 learned_addr learned my addr 172.21.15.40:0/2295576694 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb0be00640 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b2907920 0x55c6b28ed280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb18a00640 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28ed7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb13e00640 1 -- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 msgr2=0x55c6b28ed7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb13e00640 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28ed7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb13e00640 1 -- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b2907920 msgr2=0x55c6b28ed280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb13e00640 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b2907920 0x55c6b28ed280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb13e00640 1 -- 172.21.15.40:0/2295576694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6b2750d80 con 0x55c6b28bdff0 2024-03-24T00:36:21.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb18a00640 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28ed7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb13e00640 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 0x55c6b28ecd40 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400cdd0 tx=0x7ffb04000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54becb06210c1f42 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb01600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb01600640 1 -- 172.21.15.40:0/2295576694 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04003d90 con 0x55c6b28bdff0 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb01600640 1 -- 172.21.15.40:0/2295576694 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb04054670 con 0x55c6b28bdff0 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 -- 172.21.15.40:0/2295576694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb04053520 con 0x55c6b28bdff0 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb01600640 1 -- 172.21.15.40:0/2295576694 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb04054870 con 0x55c6b28bdff0 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 10 client.7680 fetch_fsmap learned FSMap version 87 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 -- 172.21.15.40:0/2295576694 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x55c6b2692d10 con 0x55c6b28bdff0 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb01600640 1 -- 172.21.15.40:0/2295576694 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb04053520 con 0x55c6b28bdff0 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb01600640 1 -- 172.21.15.40:0/2295576694 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 87) v1 ==== 3354+0+0 (secure 0 0 0) 0x7ffb0405b260 con 0x55c6b28bdff0 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 10 client.7680 fetch_fsmap finished waiting for FSMap version 87 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 10 client.7680 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 20 client.7680 populate_metadata read hostname 'smithi040' 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55c6b28ee030 0x55c6b2919ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 4 client.7680 mds_command: new command op to 7186 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:36:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 -- 172.21.15.40:0/2295576694 --> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6b26a6b60 con 0x55c6b28ee030 2024-03-24T00:36:21.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb0be00640 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55c6b28ee030 0x55c6b2919ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb0be00640 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55c6b28ee030 0x55c6b2919ea0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6e40225cb057de6 server_cookie=cbb6b31960793352 in_seq=0 out_seq=0 2024-03-24T00:36:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb01600640 10 client.7680 ms_handle_connect on v2:172.21.15.53:6834/1878888046 2024-03-24T00:36:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb01600640 1 -- 172.21.15.40:0/2295576694 <== mds.0 v2:172.21.15.53:6834/1878888046 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c6b26a6b60 con 0x55c6b28ee030 2024-03-24T00:36:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb01600640 10 client.7680 handle_command_reply: tid=0 2024-03-24T00:36:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 client.7680 shutdown 2024-03-24T00:36:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 -- 172.21.15.40:0/2295576694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 msgr2=0x55c6b28ecd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 0x55c6b28ecd40 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400cdd0 tx=0x7ffb04000ee0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 -- 172.21.15.40:0/2295576694 shutdown_connections 2024-03-24T00:36:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55c6b28ee030 0x55c6b2919ea0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28ed7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b2907920 0x55c6b28ed280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 --2- 172.21.15.40:0/2295576694 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 0x55c6b28ecd40 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb01600640 0 client.7680 ms_handle_reset on v2:172.21.15.53:6834/1878888046 2024-03-24T00:36:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 -- 172.21.15.40:0/2295576694 >> 172.21.15.40:0/2295576694 conn(0x55c6b28b9fa0 msgr2=0x55c6b28bcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 -- 172.21.15.40:0/2295576694 shutdown_connections 2024-03-24T00:36:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.794+0000 7ffb1de94000 1 -- 172.21.15.40:0/2295576694 wait complete. 2024-03-24T00:36:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.798+0000 7ffb1de94000 20 client.7680 trim_cache size 0 max 16384 2024-03-24T00:36:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 Processor -- start 2024-03-24T00:36:21.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 -- start start 2024-03-24T00:36:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 0x55c6b28bd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b2907920 0x55c6b28eb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28ebe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb14201420 con 0x55c6b28bdff0 2024-03-24T00:36:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b26b3a80 con 0x55c6b2907920 2024-03-24T00:36:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6b27c8700 con 0x55c6b2908240 2024-03-24T00:36:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 0x55c6b28bd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb13e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 0x55c6b28bd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60710/0 (socket says 172.21.15.40:60710) 2024-03-24T00:36:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb0be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b2907920 0x55c6b28eb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb13e00640 1 -- 172.21.15.40:0/2012677462 learned_addr learned my addr 172.21.15.40:0/2012677462 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb18a00640 1 --2- 172.21.15.40:0/2012677462 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28ebe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb13e00640 1 -- 172.21.15.40:0/2012677462 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 msgr2=0x55c6b28ebe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb13e00640 1 --2- 172.21.15.40:0/2012677462 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28ebe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb13e00640 1 -- 172.21.15.40:0/2012677462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b2907920 msgr2=0x55c6b28eb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb13e00640 1 --2- 172.21.15.40:0/2012677462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b2907920 0x55c6b28eb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb13e00640 1 -- 172.21.15.40:0/2012677462 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6b26d8540 con 0x55c6b28bdff0 2024-03-24T00:36:21.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb13e00640 1 --2- 172.21.15.40:0/2012677462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 0x55c6b28bd7c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7ffb0405d350 tx=0x7ffb0407d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b1b47176b5eaca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb03e00640 1 -- 172.21.15.40:0/2012677462 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04053520 con 0x55c6b28bdff0 2024-03-24T00:36:21.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb03e00640 1 -- 172.21.15.40:0/2012677462 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb04082070 con 0x55c6b28bdff0 2024-03-24T00:36:21.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb03e00640 1 -- 172.21.15.40:0/2012677462 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb04003b90 con 0x55c6b28bdff0 2024-03-24T00:36:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 -- 172.21.15.40:0/2012677462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 msgr2=0x55c6b28bd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 --2- 172.21.15.40:0/2012677462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 0x55c6b28bd7c0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7ffb0405d350 tx=0x7ffb0407d000 comp rx=0 tx=0).stop 2024-03-24T00:36:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 -- 172.21.15.40:0/2012677462 shutdown_connections 2024-03-24T00:36:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 --2- 172.21.15.40:0/2012677462 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b28ebe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 --2- 172.21.15.40:0/2012677462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b2907920 0x55c6b28eb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 --2- 172.21.15.40:0/2012677462 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b28bdff0 0x55c6b28bd7c0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.814+0000 7ffb1de94000 1 -- 172.21.15.40:0/2012677462 >> 172.21.15.40:0/2012677462 conn(0x55c6b28b9fa0 msgr2=0x55c6b2914680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 -- 172.21.15.40:0/2012677462 shutdown_connections 2024-03-24T00:36:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 -- 172.21.15.40:0/2012677462 wait complete. 2024-03-24T00:36:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 -- start start 2024-03-24T00:36:21.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 Processor -- start 2024-03-24T00:36:21.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b28bdff0 0x55c6b296c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 0x55c6b296cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b2919ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b270a990 con 0x55c6b2907920 2024-03-24T00:36:21.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6b26b1e20 con 0x55c6b28bdff0 2024-03-24T00:36:21.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c6b26c5ae0 con 0x55c6b2908240 2024-03-24T00:36:21.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 10 client.0 _ll_register_callbacks cb 0x7ffaf8003700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb0be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 0x55c6b296cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb0be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 0x55c6b296cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:60724/0 (socket says 172.21.15.40:60724) 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb0be00640 1 -- 172.21.15.40:0/1894887569 learned_addr learned my addr 172.21.15.40:0/1894887569 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb13e00640 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b28bdff0 0x55c6b296c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb18a00640 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b2919ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb0be00640 1 -- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 msgr2=0x55c6b2919ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb0be00640 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b2919ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb0be00640 1 -- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b28bdff0 msgr2=0x55c6b296c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb0be00640 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b28bdff0 0x55c6b296c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb0be00640 1 -- 172.21.15.40:0/1894887569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c6b269f4b0 con 0x55c6b2907920 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb13e00640 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b28bdff0 0x55c6b296c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb0be00640 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 0x55c6b296cad0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00fb90 tx=0x7ffafc000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecdcebe40495a101 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:21.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb01600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb01600640 1 -- 172.21.15.40:0/1894887569 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc0280c0 con 0x55c6b2907920 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb01600640 1 -- 172.21.15.40:0/1894887569 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffafc010040 con 0x55c6b2907920 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 -- 172.21.15.40:0/1894887569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb04082070 con 0x55c6b2907920 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb01600640 1 -- 172.21.15.40:0/1894887569 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffafc037050 con 0x55c6b2907920 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 10 client.7686 fetch_fsmap learned FSMap version 87 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 -- 172.21.15.40:0/1894887569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=87,monmap=2+}) v3 -- 0x55c6b26b22f0 con 0x55c6b2907920 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb01600640 1 -- 172.21.15.40:0/1894887569 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb04082070 con 0x55c6b2907920 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb01600640 1 -- 172.21.15.40:0/1894887569 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 87) v1 ==== 3354+0+0 (secure 0 0 0) 0x7ffafc03a020 con 0x55c6b2907920 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 10 client.7686 fetch_fsmap finished waiting for FSMap version 87 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 10 client.7686 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 20 client.7686 populate_metadata read hostname 'smithi040' 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55c6b29733f0 0x55c6b29757e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 4 client.7686 mds_command: new command op to 7186 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:36:21.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.818+0000 7ffb1de94000 1 -- 172.21.15.40:0/1894887569 --> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c6b2718bc0 con 0x55c6b29733f0 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb13e00640 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55c6b29733f0 0x55c6b29757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb13e00640 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55c6b29733f0 0x55c6b29757e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=926d1df329e062fb server_cookie=c2f12b7742f7cdd9 in_seq=0 out_seq=0 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb01600640 10 client.7686 ms_handle_connect on v2:172.21.15.53:6834/1878888046 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb01600640 1 -- 172.21.15.40:0/1894887569 <== mds.0 v2:172.21.15.53:6834/1878888046 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x55c6b2718bc0 con 0x55c6b29733f0 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb01600640 10 client.7686 handle_command_reply: tid=0 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1de94000 1 client.7686 shutdown 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1de94000 1 -- 172.21.15.40:0/1894887569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 msgr2=0x55c6b296cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1de94000 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 0x55c6b296cad0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7ffafc00fb90 tx=0x7ffafc000bd0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1de94000 1 -- 172.21.15.40:0/1894887569 shutdown_connections 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1de94000 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:6834/1878888046,v1:172.21.15.53:6835/1878888046] conn(0x55c6b29733f0 0x55c6b29757e0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1de94000 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6b2908240 0x55c6b2919ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1de94000 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c6b2907920 0x55c6b296cad0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1de94000 1 --2- 172.21.15.40:0/1894887569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6b28bdff0 0x55c6b296c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1de94000 1 -- 172.21.15.40:0/1894887569 >> 172.21.15.40:0/1894887569 conn(0x55c6b28b9fa0 msgr2=0x55c6b28eae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb01600640 0 client.7686 ms_handle_reset on v2:172.21.15.53:6834/1878888046 2024-03-24T00:36:21.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1de94000 1 -- 172.21.15.40:0/1894887569 shutdown_connections 2024-03-24T00:36:21.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1de94000 1 -- 172.21.15.40:0/1894887569 wait complete. 2024-03-24T00:36:21.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1de94000 20 client.7686 trim_cache size 0 max 16384 2024-03-24T00:36:21.829 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:36:21.829 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:36:21.829 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:36:21.829 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 7186, 2024-03-24T00:36:21.829 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:36:21.829 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:36:21.829 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:36:21.830 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 14.029085487, 2024-03-24T00:36:21.830 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 87, 2024-03-24T00:36:21.830 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 83, 2024-03-24T00:36:21.830 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 83, 2024-03-24T00:36:21.830 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 25.963915254 2024-03-24T00:36:21.830 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1aa00640 1 -- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffaf0067660 msgr2=0x7ffaf0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1aa00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffaf0067660 0x7ffaf0069b20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ffafc007fd0 tx=0x7ffafc009040 comp rx=0 tx=0).stop 2024-03-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1aa00640 1 -- 172.21.15.40:0/2441842137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1406f100 msgr2=0x7ffb140b3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1aa00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1406f100 0x7ffb140b3100 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7ffb0400c410 tx=0x7ffb0400c8d0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1aa00640 1 -- 172.21.15.40:0/2441842137 shutdown_connections 2024-03-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1aa00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffaf0067660 0x7ffaf0069b20 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1aa00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb14070be0 0x7ffb140b1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1aa00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffb1406f4e0 0x7ffb141534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1aa00640 1 --2- 172.21.15.40:0/2441842137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffb1406f100 0x7ffb140b3100 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1aa00640 1 -- 172.21.15.40:0/2441842137 >> 172.21.15.40:0/2441842137 conn(0x7ffb140d1c90 msgr2=0x7ffb1415f770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1aa00640 1 -- 172.21.15.40:0/2441842137 shutdown_connections 2024-03-24T00:36:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:21.822+0000 7ffb1aa00640 1 -- 172.21.15.40:0/2441842137 wait complete. 2024-03-24T00:36:21.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:21.845 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:36:21.845 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:36:21.845 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p tree 2024-03-24T00:36:21.845 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:36:21.845 DEBUG:teuthology.orchestra.run.smithi040:> setfattr -n ceph.dir.pin.distributed -v 1 tree 2024-03-24T00:36:21.845 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:36:21.845 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < 100; i++)); do 2024-03-24T00:36:21.845 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p "tree/$i" 2024-03-24T00:36:21.846 DEBUG:teuthology.orchestra.run.smithi040:> echo file > "tree/$i/file" 2024-03-24T00:36:21.846 DEBUG:teuthology.orchestra.run.smithi040:> done 2024-03-24T00:36:21.846 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:36:21.856 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree 2024-03-24T00:36:21.860 INFO:teuthology.orchestra.run.smithi040.stderr:+ setfattr -n ceph.dir.pin.distributed -v 1 tree 2024-03-24T00:36:21.864 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i = 0 )) 2024-03-24T00:36:21.864 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.864 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/0 2024-03-24T00:36:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/1 2024-03-24T00:36:21.883 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.886 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.886 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/2 2024-03-24T00:36:21.892 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/3 2024-03-24T00:36:21.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.904 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.905 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/4 2024-03-24T00:36:21.911 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.913 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.913 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.913 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/5 2024-03-24T00:36:21.919 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.922 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.922 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.922 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/6 2024-03-24T00:36:21.928 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.930 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/7 2024-03-24T00:36:21.936 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.938 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.939 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.939 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/8 2024-03-24T00:36:21.945 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.947 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.947 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.948 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/9 2024-03-24T00:36:21.954 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.956 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.956 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.956 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/10 2024-03-24T00:36:21.962 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.964 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.964 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.964 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/11 2024-03-24T00:36:21.969 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.972 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.972 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.972 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/12 2024-03-24T00:36:21.978 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.981 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/13 2024-03-24T00:36:21.987 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/14 2024-03-24T00:36:21.995 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:21.997 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:21.998 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/15 2024-03-24T00:36:22.003 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.006 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.006 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.006 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/16 2024-03-24T00:36:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.015 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.015 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.015 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/17 2024-03-24T00:36:22.021 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.024 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.024 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.024 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/18 2024-03-24T00:36:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.032 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/19 2024-03-24T00:36:22.039 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.041 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.041 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.041 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/20 2024-03-24T00:36:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.051 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/21 2024-03-24T00:36:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.060 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.060 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.060 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/22 2024-03-24T00:36:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.068 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.069 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.069 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/23 2024-03-24T00:36:22.074 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.077 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.077 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.077 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/24 2024-03-24T00:36:22.083 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.086 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.086 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.086 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/25 2024-03-24T00:36:22.092 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.094 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.094 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.094 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/26 2024-03-24T00:36:22.100 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.103 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.103 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.103 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/27 2024-03-24T00:36:22.109 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.112 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.112 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.112 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/28 2024-03-24T00:36:22.118 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.120 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.120 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.121 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/29 2024-03-24T00:36:22.127 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.129 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.129 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.129 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/30 2024-03-24T00:36:22.135 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.138 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.138 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.138 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/31 2024-03-24T00:36:22.144 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.146 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.147 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.147 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/32 2024-03-24T00:36:22.153 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.155 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/33 2024-03-24T00:36:22.161 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.164 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/34 2024-03-24T00:36:22.169 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.172 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.172 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.172 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/35 2024-03-24T00:36:22.178 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.180 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.180 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.180 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/36 2024-03-24T00:36:22.186 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.188 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.189 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/37 2024-03-24T00:36:22.195 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.197 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.197 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.197 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/38 2024-03-24T00:36:22.203 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.206 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.206 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.206 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/39 2024-03-24T00:36:22.212 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.214 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.214 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.214 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/40 2024-03-24T00:36:22.220 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.222 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.222 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.222 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/41 2024-03-24T00:36:22.228 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.231 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.231 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.231 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/42 2024-03-24T00:36:22.237 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.239 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.239 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.239 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/43 2024-03-24T00:36:22.246 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.248 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.248 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.248 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/44 2024-03-24T00:36:22.254 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.257 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.257 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.257 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/45 2024-03-24T00:36:22.262 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.265 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.265 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.265 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/46 2024-03-24T00:36:22.271 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.274 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.274 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.274 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/47 2024-03-24T00:36:22.280 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.283 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.283 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.283 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/48 2024-03-24T00:36:22.288 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.291 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.291 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.291 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/49 2024-03-24T00:36:22.297 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.300 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.300 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.300 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/50 2024-03-24T00:36:22.305 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.308 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.308 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.308 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/51 2024-03-24T00:36:22.314 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.316 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.316 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.316 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/52 2024-03-24T00:36:22.322 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.324 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.324 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.324 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/53 2024-03-24T00:36:22.330 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.332 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.333 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.333 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/54 2024-03-24T00:36:22.338 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.341 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.341 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.341 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/55 2024-03-24T00:36:22.347 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.349 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.349 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.349 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/56 2024-03-24T00:36:22.355 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.358 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/57 2024-03-24T00:36:22.364 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.367 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.367 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.367 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/58 2024-03-24T00:36:22.373 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.376 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.376 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.376 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/59 2024-03-24T00:36:22.382 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.385 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.385 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.385 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/60 2024-03-24T00:36:22.390 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.393 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.393 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.393 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/61 2024-03-24T00:36:22.399 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/62 2024-03-24T00:36:22.409 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.411 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.411 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.411 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/63 2024-03-24T00:36:22.418 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.421 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.421 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.421 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/64 2024-03-24T00:36:22.428 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.431 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.431 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.431 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/65 2024-03-24T00:36:22.438 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.441 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.441 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.441 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/66 2024-03-24T00:36:22.448 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.451 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/67 2024-03-24T00:36:22.457 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.461 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.461 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.461 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/68 2024-03-24T00:36:22.468 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.471 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/69 2024-03-24T00:36:22.478 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.481 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.481 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.481 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/70 2024-03-24T00:36:22.488 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.491 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.492 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.492 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/71 2024-03-24T00:36:22.499 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.502 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.502 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.502 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/72 2024-03-24T00:36:22.509 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.511 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.511 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.511 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/73 2024-03-24T00:36:22.517 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.520 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.520 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.520 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/74 2024-03-24T00:36:22.526 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.528 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.529 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.529 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/75 2024-03-24T00:36:22.535 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.538 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.538 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.538 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/76 2024-03-24T00:36:22.545 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.547 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.547 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.547 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/77 2024-03-24T00:36:22.554 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.556 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.557 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.557 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/78 2024-03-24T00:36:22.563 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.566 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.566 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.566 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/79 2024-03-24T00:36:22.572 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.575 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.575 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.575 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/80 2024-03-24T00:36:22.582 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.584 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.584 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.584 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/81 2024-03-24T00:36:22.591 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.594 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.594 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.594 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/82 2024-03-24T00:36:22.601 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.603 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.603 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.603 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/83 2024-03-24T00:36:22.610 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.612 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.612 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.612 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/84 2024-03-24T00:36:22.618 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.621 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.621 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.621 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/85 2024-03-24T00:36:22.627 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.630 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.630 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.630 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/86 2024-03-24T00:36:22.637 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.639 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.639 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.639 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/87 2024-03-24T00:36:22.646 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.648 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.648 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.648 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/88 2024-03-24T00:36:22.655 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.657 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.658 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/89 2024-03-24T00:36:22.664 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.667 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.667 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.667 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/90 2024-03-24T00:36:22.673 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.676 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.676 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.676 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/91 2024-03-24T00:36:22.682 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.685 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.685 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.685 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/92 2024-03-24T00:36:22.691 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.694 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.694 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.694 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/93 2024-03-24T00:36:22.701 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.703 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.703 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.704 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/94 2024-03-24T00:36:22.710 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.713 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.713 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.713 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/95 2024-03-24T00:36:22.719 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.723 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.723 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.723 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/96 2024-03-24T00:36:22.728 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.731 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.731 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.732 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/97 2024-03-24T00:36:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.741 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.741 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.741 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/98 2024-03-24T00:36:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.750 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.750 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.750 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/99 2024-03-24T00:36:22.756 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:36:22.759 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:36:22.759 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:36:22.760 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:36:22.945 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:22.946 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:22.946 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:36:23.105 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:23.106 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:23.106 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:36:23.265 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"875","version":"873","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"225","version":"225","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-24T00:36:23.266 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "875", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "873" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "225", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "225" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:23.266 INFO:tasks.cephfs.cephfs_test_case:len=0 [] 2024-03-24T00:36:28.267 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:36:28.434 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:28.434 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:28.435 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:36:28.595 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:28.596 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:28.596 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:36:28.756 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"880","version":"878","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"227","version":"227","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-24T00:36:28.757 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "880", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "878" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "227", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "227" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:28.757 INFO:tasks.cephfs.cephfs_test_case:len=0 [] 2024-03-24T00:36:33.759 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:36:33.920 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:33.920 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:33.920 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:36:34.080 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"882","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":4},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.000*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:34.081 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 4 }, "snapid_first": 2, "states": [], "version": "882" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.111*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "416", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "430", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.001*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "428", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.000*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:34.081 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:36:34.241 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"882","version":"882","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":4}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"227","version":"227","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.110*","snapid_first":2,"projected_version":"442","version":"442","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.100*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"waiter":0,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.101*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.010*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.000*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-03-24T00:36:34.243 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 4 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "882", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "882" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "227", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "227" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.111*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "416", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.110*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "442", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "442" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.100*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.011*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "430", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.101*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "422", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.010*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.000*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.001*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "428", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-24T00:36:34.243 INFO:tasks.cephfs.cephfs_test_case:len=8 [{'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x10000000264.111*', 'snapid_first': 2, 'projected_version': '416', 'version': '416', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x10000000264.011*', 'snapid_first': 2, 'projected_version': '430', 'version': '430', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x10000000264.001*', 'snapid_first': 2, 'projected_version': '428', 'version': '428', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x10000000264.000*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x10000000264.110*', 'snapid_first': 2, 'projected_version': '442', 'version': '442', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x10000000264.100*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'waiter': 0, 'authpin': 0}, 'nref': 3}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x10000000264.101*', 'snapid_first': 2, 'projected_version': '422', 'version': '422', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x10000000264.010*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}] 2024-03-24T00:36:34.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:34.244 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/0) 2024-03-24T00:36:34.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:34.260 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/1) 2024-03-24T00:36:34.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:34.316 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/2) 2024-03-24T00:36:34.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:34.372 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/3) 2024-03-24T00:36:34.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:34.426 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/4) 2024-03-24T00:36:34.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:34.478 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/5) 2024-03-24T00:36:34.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:34.530 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/6) 2024-03-24T00:36:34.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:34.582 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/7) 2024-03-24T00:36:34.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:34.634 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/8) 2024-03-24T00:36:34.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:34.686 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/9) 2024-03-24T00:36:34.739 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:36:34.898 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:34.899 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:34.899 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:36:35.060 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"882","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":4},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.111*","snapid_first":2,"projected_version":"419","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":2,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.011*","snapid_first":2,"projected_version":"437","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":4,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.001*","snapid_first":2,"projected_version":"429","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.000*","snapid_first":2,"projected_version":"433","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":4,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:35.061 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 4 }, "snapid_first": 2, "states": [], "version": "882" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 2, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.111*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 5, "path": "/tree", "pins": { "authpin": 1, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "419", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 4, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 5, "path": "/tree", "pins": { "authpin": 1, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "437", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.001*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 5, "path": "/tree", "pins": { "authpin": 1, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "429", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 4, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.000*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 5, "path": "/tree", "pins": { "authpin": 1, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "433", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:35.061 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:36:35.221 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"890","version":"884","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":4}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":3,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":1},"nref":5}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"227","version":"227","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.110*","snapid_first":2,"projected_version":"447","version":"443","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":2,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":1},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.100*","snapid_first":2,"projected_version":"429","version":"427","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"waiter":0,"authpin":1},"nref":4}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.101*","snapid_first":2,"projected_version":"423","version":"423","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.010*","snapid_first":2,"projected_version":"427","version":"427","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.000*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-03-24T00:36:35.222 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 3, "auth_state": { "replicas": { "1": 4 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 5, "path": "", "pins": { "authpin": 1, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "890", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "884" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "227", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "227" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.111*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "416", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 2, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.110*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 1, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "447", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "443" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 1, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.100*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 1, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1, "waiter": 0 }, "projected_version": "429", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "427" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.011*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "430", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.101*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "423", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "423" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.010*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "427", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "427" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.000*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.001*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "428", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-24T00:36:35.222 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/tree/0', 1), ('/tree/1', 1), ('/tree/2', 1), ('/tree/3', 1), ('/tree/4', 1), ('/tree/5', 1), ('/tree/6', 1), ('/tree/7', 1), ('/tree/8', 1), ('/tree/9', 1)] 2024-03-24T00:36:37.224 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:36:37.387 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:37.388 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:37.388 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:36:37.547 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"882","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":4},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.111*","snapid_first":2,"projected_version":"419","version":"419","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.011*","snapid_first":2,"projected_version":"437","version":"437","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.001*","snapid_first":2,"projected_version":"429","version":"429","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.000*","snapid_first":2,"projected_version":"433","version":"433","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:37.547 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 4 }, "snapid_first": 2, "states": [], "version": "882" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.111*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "419", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "419" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "437", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "437" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.001*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "429", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "429" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.000*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "433", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "433" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:37.548 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:36:37.687 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:36:37.690 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:36:37.710 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"890","version":"890","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":4}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"227","version":"227","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.110*","snapid_first":2,"projected_version":"447","version":"447","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.100*","snapid_first":2,"projected_version":"429","version":"429","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"waiter":0,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.101*","snapid_first":2,"projected_version":"423","version":"423","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.010*","snapid_first":2,"projected_version":"427","version":"427","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.000*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-03-24T00:36:37.711 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 4 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "890", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "890" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "227", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "227" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.111*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "416", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.110*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "447", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "447" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.100*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1, "waiter": 0 }, "projected_version": "429", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "429" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.011*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "430", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.101*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "423", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "423" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.010*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "427", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "427" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.000*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.001*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "428", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-24T00:36:37.711 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/tree/0', 1), ('/tree/1', 1), ('/tree/2', 1), ('/tree/3', 1), ('/tree/4', 1), ('/tree/5', 1), ('/tree/6', 1), ('/tree/7', 1), ('/tree/8', 1), ('/tree/9', 1)] 2024-03-24T00:36:39.712 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:36:39.881 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:39.881 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:39.882 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:36:40.042 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"882","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":4},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.111*","snapid_first":2,"projected_version":"419","version":"419","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.011*","snapid_first":2,"projected_version":"437","version":"437","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.001*","snapid_first":2,"projected_version":"429","version":"429","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.000*","snapid_first":2,"projected_version":"433","version":"433","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:40.043 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 4 }, "snapid_first": 2, "states": [], "version": "882" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.111*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "419", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "419" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "437", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "437" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.001*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "429", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "429" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.000*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "433", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "433" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:40.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:36:40.203 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"890","version":"890","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":4}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"227","version":"227","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.110*","snapid_first":2,"projected_version":"447","version":"447","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.100*","snapid_first":2,"projected_version":"429","version":"429","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"waiter":0,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.101*","snapid_first":2,"projected_version":"423","version":"423","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.010*","snapid_first":2,"projected_version":"427","version":"427","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.000*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-03-24T00:36:40.205 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 4 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "890", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "890" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "227", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "227" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.111*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "416", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.110*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "447", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "447" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.100*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1, "waiter": 0 }, "projected_version": "429", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "429" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.011*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "430", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.101*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "423", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "423" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.010*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "427", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "427" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.000*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.001*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "428", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-24T00:36:40.205 INFO:tasks.cephfs.cephfs_test_case:[] =?= [('/tree/0', 1), ('/tree/1', 1), ('/tree/2', 1), ('/tree/3', 1), ('/tree/4', 1), ('/tree/5', 1), ('/tree/6', 1), ('/tree/7', 1), ('/tree/8', 1), ('/tree/9', 1)] 2024-03-24T00:36:42.206 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:36:42.366 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:36:42.367 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:42.367 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:36:42.526 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"882","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":4},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.111*","snapid_first":2,"projected_version":"420","version":"420","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.011*","snapid_first":2,"projected_version":"438","version":"438","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.001*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.000*","snapid_first":2,"projected_version":"434","version":"434","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/8","dirfrag":"0x10000000275","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1},"nref":3}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/3","dirfrag":"0x1000000026b","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1},"nref":3}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/9","dirfrag":"0x10000000277","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1},"nref":3}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/2","dirfrag":"0x10000000269","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1},"nref":3}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/4","dirfrag":"0x1000000026d","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1},"nref":3}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/1","dirfrag":"0x10000000267","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1},"nref":3}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/6","dirfrag":"0x10000000271","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1},"nref":3}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/0","dirfrag":"0x10000000265","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/5","dirfrag":"0x1000000026f","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/7","dirfrag":"0x10000000273","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-24T00:36:42.528 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 4 }, "snapid_first": 2, "states": [], "version": "882" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.111*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "420", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "420" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "438", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "438" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.001*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "430", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.000*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "434", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "434" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000275", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree/8", "pins": { "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x1000000026b", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree/3", "pins": { "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000277", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree/9", "pins": { "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000269", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree/2", "pins": { "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x1000000026d", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree/4", "pins": { "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000267", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree/1", "pins": { "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000271", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree/6", "pins": { "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000265", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree/0", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x1000000026f", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree/5", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000273", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree/7", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:36:42.528 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:36:42.690 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"893","version":"893","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":4}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"227","version":"227","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/7","dirfrag":"0x10000000273","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.110*","snapid_first":2,"projected_version":"448","version":"448","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":2}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.100*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"replicated":1,"dirty":1,"waiter":0,"authpin":0},"nref":4}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/0","dirfrag":"0x10000000265","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.101*","snapid_first":2,"projected_version":"424","version":"424","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x10000000264.010*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.000*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x10000000264.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/5","dirfrag":"0x1000000026f","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-03-24T00:36:42.691 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 4 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "893", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "893" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "227", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "227" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000273", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree/7", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.111*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "416", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 2 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.110*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "replicated": 1, "subtree": 1 }, "projected_version": "448", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "448" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.100*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "replicated": 1, "subtree": 1, "waiter": 0 }, "projected_version": "430", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x10000000265", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree/0", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.011*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "430", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.101*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "424", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "424" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x10000000264.010*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "428", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.000*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x10000000264.001*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "428", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x1000000026f", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree/5", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-24T00:36:42.691 INFO:tasks.cephfs.cephfs_test_case:[('/tree/0', 1), ('/tree/1', 1), ('/tree/2', 1), ('/tree/3', 1), ('/tree/4', 1), ('/tree/5', 1), ('/tree/6', 1), ('/tree/7', 1), ('/tree/8', 1), ('/tree/9', 1)] =?= [('/tree/0', 1), ('/tree/1', 1), ('/tree/2', 1), ('/tree/3', 1), ('/tree/4', 1), ('/tree/5', 1), ('/tree/6', 1), ('/tree/7', 1), ('/tree/8', 1), ('/tree/9', 1)] 2024-03-24T00:36:42.692 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:36:42.692 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:36:42.766 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:36:42.847 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:36:42.847 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:36:42.847 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:36:42.856 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:36:43.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.034+0000 7f0802400640 1 Processor -- start 2024-03-24T00:36:43.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.034+0000 7f0802400640 1 -- start start 2024-03-24T00:36:43.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f0802400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07fc13ec80 0x7f07fc1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f0802400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f07fc1419f0 0x7f07fc0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f0802400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07fc070e00 con 0x7f07fc0ce630 2024-03-24T00:36:43.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f0802400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07fc141e50 con 0x7f07fc1419f0 2024-03-24T00:36:43.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f0802400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc0cee20 con 0x7f07fc13ec80 2024-03-24T00:36:43.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f07f3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f07fc1419f0 0x7f07fc0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f07fbe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f07fc0ce630 0x7f07fc15f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44584/0 (socket says 172.21.15.40:44584) 2024-03-24T00:36:43.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f07fbe00640 1 -- 172.21.15.40:0/720901579 learned_addr learned my addr 172.21.15.40:0/720901579 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:43.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f07fb400640 1 --2- 172.21.15.40:0/720901579 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07fc13ec80 0x7f07fc1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f07f3e00640 1 -- 172.21.15.40:0/720901579 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07fc13ec80 msgr2=0x7f07fc1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:43.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f07f3e00640 1 --2- 172.21.15.40:0/720901579 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07fc13ec80 0x7f07fc1414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f07f3e00640 1 -- 172.21.15.40:0/720901579 >> v1:172.21.15.40:6789/0 conn(0x7f07fc0ce630 legacy=0x7f07fc15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f07f3e00640 1 -- 172.21.15.40:0/720901579 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc0cf100 con 0x7f07fc1419f0 2024-03-24T00:36:43.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.038+0000 7f07faa00640 1 -- 172.21.15.40:0/720901579 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665083286 0 0) 0x7f07fc070e00 con 0x7f07fc0ce630 2024-03-24T00:36:43.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f07f3e00640 1 --2- 172.21.15.40:0/720901579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07fc1419f0 0x7f07fc0ce0f0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f07e40097c0 tx=0x7f07e401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699c6f24941982a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:43.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f07faa00640 1 -- 172.21.15.40:0/720901579 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e40aa620 con 0x7f07fc1419f0 2024-03-24T00:36:43.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f07faa00640 1 -- 172.21.15.40:0/720901579 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e40af070 con 0x7f07fc1419f0 2024-03-24T00:36:43.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f07faa00640 1 -- 172.21.15.40:0/720901579 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e40aa9e0 con 0x7f07fc1419f0 2024-03-24T00:36:43.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 -- 172.21.15.40:0/720901579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07fc1419f0 msgr2=0x7f07fc0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 --2- 172.21.15.40:0/720901579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07fc1419f0 0x7f07fc0ce0f0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f07e40097c0 tx=0x7f07e401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 -- 172.21.15.40:0/720901579 shutdown_connections 2024-03-24T00:36:43.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 --2- 172.21.15.40:0/720901579 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07fc1419f0 0x7f07fc0ce0f0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 --2- 172.21.15.40:0/720901579 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07fc13ec80 0x7f07fc1414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 -- 172.21.15.40:0/720901579 >> 172.21.15.40:0/720901579 conn(0x7f07fc0b17b0 msgr2=0x7f07fc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:43.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 -- 172.21.15.40:0/720901579 shutdown_connections 2024-03-24T00:36:43.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 -- 172.21.15.40:0/720901579 wait complete. 2024-03-24T00:36:43.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 Processor -- start 2024-03-24T00:36:43.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 -- start start 2024-03-24T00:36:43.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07fc0ce630 0x7f07fc15bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07fc13ec80 0x7f07fc139550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07fc1419f0 0x7f07fc159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc168ec0 con 0x7f07fc1419f0 2024-03-24T00:36:43.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc168d40 con 0x7f07fc0ce630 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.042+0000 7f0802400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc169040 con 0x7f07fc13ec80 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07fc1419f0 0x7f07fc159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07fb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07fc0ce630 0x7f07fc15bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07fc1419f0 0x7f07fc159fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40132/0 (socket says 172.21.15.40:40132) 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07f3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07fc13ec80 0x7f07fc139550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07fbe00640 1 -- 172.21.15.40:0/2776113727 learned_addr learned my addr 172.21.15.40:0/2776113727 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07fb400640 1 -- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07fc13ec80 msgr2=0x7f07fc139550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07fb400640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07fc13ec80 0x7f07fc139550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07fb400640 1 -- 172.21.15.40:0/2776113727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07fc1419f0 msgr2=0x7f07fc159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07fb400640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07fc1419f0 0x7f07fc159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07fb400640 1 -- 172.21.15.40:0/2776113727 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc139a90 con 0x7f07fc0ce630 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07fbe00640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07fc1419f0 0x7f07fc159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:43.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07fb400640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07fc0ce630 0x7f07fc15bb30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f07ec00e870 tx=0x7f07ec00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:43.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07f3400640 1 -- 172.21.15.40:0/2776113727 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ec017070 con 0x7f07fc0ce630 2024-03-24T00:36:43.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07f3400640 1 -- 172.21.15.40:0/2776113727 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07ec00c940 con 0x7f07fc0ce630 2024-03-24T00:36:43.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07f3400640 1 -- 172.21.15.40:0/2776113727 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07ec00ccf0 con 0x7f07fc0ce630 2024-03-24T00:36:43.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f0802400640 1 -- 172.21.15.40:0/2776113727 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07fc15aa90 con 0x7f07fc0ce630 2024-03-24T00:36:43.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f0802400640 1 -- 172.21.15.40:0/2776113727 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07fc15b050 con 0x7f07fc0ce630 2024-03-24T00:36:43.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f0802400640 1 -- 172.21.15.40:0/2776113727 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07fc068170 con 0x7f07fc0ce630 2024-03-24T00:36:43.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.046+0000 7f07f3400640 1 -- 172.21.15.40:0/2776113727 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f07ec01a070 con 0x7f07fc0ce630 2024-03-24T00:36:43.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.050+0000 7f07f3400640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07d8067660 0x7f07d8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.050+0000 7f07f3400640 1 -- 172.21.15.40:0/2776113727 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07ec0856f0 con 0x7f07fc0ce630 2024-03-24T00:36:43.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.050+0000 7f07f3e00640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07d8067660 0x7f07d8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.050+0000 7f07f3e00640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07d8067660 0x7f07d8069b20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f07e40ae000 tx=0x7f07e40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:43.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.054+0000 7f07f3400640 1 -- 172.21.15.40:0/2776113727 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f07ec055db0 con 0x7f07fc0ce630 2024-03-24T00:36:43.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.214+0000 7f0802400640 1 -- 172.21.15.40:0/2776113727 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07fc072b90 con 0x7f07fc0ce630 2024-03-24T00:36:43.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.214+0000 7f07f3400640 1 -- 172.21.15.40:0/2776113727 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16600 (secure 0 0 0) 0x7f07ec059e50 con 0x7f07fc0ce630 2024-03-24T00:36:43.219 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:43.219 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":83,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:36:08.785471+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:05.573740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-24T00:36:06.150226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.218+0000 7f0802400640 1 -- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07d8067660 msgr2=0x7f07d8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.218+0000 7f0802400640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07d8067660 0x7f07d8069b20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f07e40ae000 tx=0x7f07e40a7000 comp rx=0 tx=0).stop 2024-03-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.218+0000 7f0802400640 1 -- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07fc0ce630 msgr2=0x7f07fc15bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.218+0000 7f0802400640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07fc0ce630 0x7f07fc15bb30 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f07ec00e870 tx=0x7f07ec00ed30 comp rx=0 tx=0).stop 2024-03-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.218+0000 7f0802400640 1 -- 172.21.15.40:0/2776113727 shutdown_connections 2024-03-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.218+0000 7f0802400640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07d8067660 0x7f07d8069b20 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.218+0000 7f0802400640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07fc1419f0 0x7f07fc159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.218+0000 7f0802400640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07fc13ec80 0x7f07fc139550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.218+0000 7f0802400640 1 --2- 172.21.15.40:0/2776113727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07fc0ce630 0x7f07fc15bb30 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.218+0000 7f0802400640 1 -- 172.21.15.40:0/2776113727 >> 172.21.15.40:0/2776113727 conn(0x7f07fc0b17b0 msgr2=0x7f07fc15d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.218+0000 7f0802400640 1 -- 172.21.15.40:0/2776113727 shutdown_connections 2024-03-24T00:36:43.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.218+0000 7f0802400640 1 -- 172.21.15.40:0/2776113727 wait complete. 2024-03-24T00:36:43.240 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:36:05.573740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:36:06.150226+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:36:43.241 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:36:43.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.382+0000 7fcaf4c00640 1 Processor -- start 2024-03-24T00:36:43.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.382+0000 7fcaf4c00640 1 -- start start 2024-03-24T00:36:43.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.382+0000 7fcaf4c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcaf0145700 0x7fcaf00725c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.382+0000 7fcaf4c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcaf00cd320 0x7fcaf00cfd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.382+0000 7fcaf4c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf00ceff0 con 0x7fcaf0066e80 2024-03-24T00:36:43.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.382+0000 7fcaf4c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf00cf8e0 con 0x7fcaf00cd320 2024-03-24T00:36:43.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.382+0000 7fcaf4c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf00cdb70 con 0x7fcaf0145700 2024-03-24T00:36:43.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.382+0000 7fcae5400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcaf00cd320 0x7fcaf00cfd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.382+0000 7fcaede00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcaf0145700 0x7fcaf00725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.382+0000 7fcae5400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcaf00cd320 0x7fcaf00cfd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40136/0 (socket says 172.21.15.40:40136) 2024-03-24T00:36:43.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.382+0000 7fcae5400640 1 -- 172.21.15.40:0/1613991853 learned_addr learned my addr 172.21.15.40:0/1613991853 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:43.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.382+0000 7fcaee800640 1 --1- 172.21.15.40:0/1613991853 >> v1:172.21.15.53:6789/0 conn(0x7fcaf0066e80 0x7fcaf006dbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:55090/0 (socket says 172.21.15.40:55090) 2024-03-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.386+0000 7fcaede00640 1 -- 172.21.15.40:0/1613991853 >> v2:172.21.15.40:3300/0 conn(0x7fcaf00cd320 msgr2=0x7fcaf00cfd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.386+0000 7fcaede00640 1 --2- 172.21.15.40:0/1613991853 >> v2:172.21.15.40:3300/0 conn(0x7fcaf00cd320 0x7fcaf00cfd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.386+0000 7fcaede00640 1 -- 172.21.15.40:0/1613991853 >> v1:172.21.15.53:6789/0 conn(0x7fcaf0066e80 legacy=0x7fcaf006dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.386+0000 7fcaede00640 1 -- 172.21.15.40:0/1613991853 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf00cd890 con 0x7fcaf0145700 2024-03-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.386+0000 7fcae5400640 1 --2- 172.21.15.40:0/1613991853 >> v2:172.21.15.40:3300/0 conn(0x7fcaf00cd320 0x7fcaf00cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:43.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.386+0000 7fcaed400640 1 -- 172.21.15.40:0/1613991853 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085488435 0 0) 0x7fcaf00ceff0 con 0x7fcaf0066e80 2024-03-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaede00640 1 --2- 172.21.15.40:0/1613991853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf0145700 0x7fcaf00725c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0071700 tx=0x7fcae001c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a784d10d636a8c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaed400640 1 -- 172.21.15.40:0/1613991853 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00b2070 con 0x7fcaf0145700 2024-03-24T00:36:43.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaed400640 1 -- 172.21.15.40:0/1613991853 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae00a58d0 con 0x7fcaf0145700 2024-03-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaed400640 1 -- 172.21.15.40:0/1613991853 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00a5ca0 con 0x7fcaf0145700 2024-03-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1613991853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf0145700 msgr2=0x7fcaf00725c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaf4c00640 1 --2- 172.21.15.40:0/1613991853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf0145700 0x7fcaf00725c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0071700 tx=0x7fcae001c900 comp rx=0 tx=0).stop 2024-03-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1613991853 shutdown_connections 2024-03-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaf4c00640 1 --2- 172.21.15.40:0/1613991853 >> v2:172.21.15.40:3300/0 conn(0x7fcaf00cd320 0x7fcaf00cfd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaf4c00640 1 --2- 172.21.15.40:0/1613991853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf0145700 0x7fcaf00725c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1613991853 >> 172.21.15.40:0/1613991853 conn(0x7fcaf00b27c0 msgr2=0x7fcaf00b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1613991853 shutdown_connections 2024-03-24T00:36:43.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1613991853 wait complete. 2024-03-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaf4c00640 1 Processor -- start 2024-03-24T00:36:43.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.390+0000 7fcaf4c00640 1 -- start start 2024-03-24T00:36:43.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaf4c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf0066e80 0x7fcaf01414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaf4c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf00cd320 0x7fcaf0143800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaf4c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf0145700 0x7fcaf0155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaf4c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf00d02b0 con 0x7fcaf0066e80 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaf4c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf00676e0 con 0x7fcaf00cd320 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaf4c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf0144030 con 0x7fcaf0145700 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaede00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf0066e80 0x7fcaf01414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaede00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf0066e80 0x7fcaf01414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40138/0 (socket says 172.21.15.40:40138) 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaede00640 1 -- 172.21.15.40:0/1754465045 learned_addr learned my addr 172.21.15.40:0/1754465045 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcae5400640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf00cd320 0x7fcaf0143800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaee800640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf0145700 0x7fcaf0155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaede00640 1 -- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf0145700 msgr2=0x7fcaf0155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaede00640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf0145700 0x7fcaf0155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaede00640 1 -- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf00cd320 msgr2=0x7fcaf0143800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaede00640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf00cd320 0x7fcaf0143800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaede00640 1 -- 172.21.15.40:0/1754465045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf0144310 con 0x7fcaf0066e80 2024-03-24T00:36:43.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaede00640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf0066e80 0x7fcaf01414b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fcae001a530 tx=0x7fcae00196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcae6a00640 1 -- 172.21.15.40:0/1754465045 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00b2070 con 0x7fcaf0066e80 2024-03-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcae6a00640 1 -- 172.21.15.40:0/1754465045 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae00bf070 con 0x7fcaf0066e80 2024-03-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1754465045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf01445f0 con 0x7fcaf0066e80 2024-03-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcae6a00640 1 -- 172.21.15.40:0/1754465045 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00b9ab0 con 0x7fcaf0066e80 2024-03-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1754465045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf015c770 con 0x7fcaf0066e80 2024-03-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1754465045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf0068170 con 0x7fcaf0066e80 2024-03-24T00:36:43.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.394+0000 7fcae6a00640 1 -- 172.21.15.40:0/1754465045 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcae00b9c50 con 0x7fcaf0066e80 2024-03-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.398+0000 7fcae6a00640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcacc067660 0x7fcacc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.398+0000 7fcae6a00640 1 -- 172.21.15.40:0/1754465045 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcae01206f0 con 0x7fcaf0066e80 2024-03-24T00:36:43.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.398+0000 7fcae5400640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcacc067660 0x7fcacc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.398+0000 7fcae5400640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcacc067660 0x7fcacc069b20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0142a10 tx=0x7fcad8009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:43.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.402+0000 7fcae6a00640 1 -- 172.21.15.40:0/1754465045 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcae0016520 con 0x7fcaf0066e80 2024-03-24T00:36:43.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1754465045 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcaf0072de0 con 0x7fcaf0066e80 2024-03-24T00:36:43.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcae6a00640 1 -- 172.21.15.40:0/1754465045 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcae00f0da0 con 0x7fcaf0066e80 2024-03-24T00:36:43.569 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:36:43.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcacc067660 msgr2=0x7fcacc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcacc067660 0x7fcacc069b20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0142a10 tx=0x7fcad8009240 comp rx=0 tx=0).stop 2024-03-24T00:36:43.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1754465045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf0066e80 msgr2=0x7fcaf01414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf0066e80 0x7fcaf01414b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fcae001a530 tx=0x7fcae00196a0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1754465045 shutdown_connections 2024-03-24T00:36:43.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcacc067660 0x7fcacc069b20 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf0145700 0x7fcaf0155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf00cd320 0x7fcaf0143800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 --2- 172.21.15.40:0/1754465045 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaf0066e80 0x7fcaf01414b0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1754465045 >> 172.21.15.40:0/1754465045 conn(0x7fcaf00b27c0 msgr2=0x7fcaf006e820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:43.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1754465045 shutdown_connections 2024-03-24T00:36:43.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.566+0000 7fcaf4c00640 1 -- 172.21.15.40:0/1754465045 wait complete. 2024-03-24T00:36:43.590 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:36:43.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.730+0000 7f7deb000640 1 Processor -- start 2024-03-24T00:36:43.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.730+0000 7f7deb000640 1 -- start start 2024-03-24T00:36:43.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7deb000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7de415e170 0x7f7de406f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7deb000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7de4160080 0x7f7de415e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7deb000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de40cfcb0 con 0x7f7de40d0a10 2024-03-24T00:36:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7deb000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de4070100 con 0x7f7de4160080 2024-03-24T00:36:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7deb000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de406fe20 con 0x7f7de415e170 2024-03-24T00:36:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7de3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7de415e170 0x7f7de406f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7ddb400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7de40d0a10 0x7f7de415f970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:55096/0 (socket says 172.21.15.40:55096) 2024-03-24T00:36:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7ddb400640 1 -- 172.21.15.40:0/4200682141 learned_addr learned my addr 172.21.15.40:0/4200682141 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:43.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7de9000640 1 --2- 172.21.15.40:0/4200682141 >> v2:172.21.15.53:3300/0 conn(0x7f7de4160080 0x7f7de415e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7de3e00640 1 -- 172.21.15.40:0/4200682141 >> v2:172.21.15.53:3300/0 conn(0x7f7de4160080 msgr2=0x7f7de415e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7de3e00640 1 --2- 172.21.15.40:0/4200682141 >> v2:172.21.15.53:3300/0 conn(0x7f7de4160080 0x7f7de415e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7de3e00640 1 -- 172.21.15.40:0/4200682141 >> v1:172.21.15.53:6789/0 conn(0x7f7de40d0a10 legacy=0x7f7de415f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7de3e00640 1 -- 172.21.15.40:0/4200682141 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de40cff30 con 0x7f7de415e170 2024-03-24T00:36:43.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.734+0000 7f7de3400640 1 -- 172.21.15.40:0/4200682141 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557702560 0 0) 0x7f7dcc001120 con 0x7f7de40d0a10 2024-03-24T00:36:43.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7de3e00640 1 --2- 172.21.15.40:0/4200682141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7de415e170 0x7f7de406f7b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40097c0 tx=0x7f7dd401c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edda698d8529b184 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:43.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7de3400640 1 -- 172.21.15.40:0/4200682141 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd40b2070 con 0x7f7de415e170 2024-03-24T00:36:43.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7de3400640 1 -- 172.21.15.40:0/4200682141 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd40ad680 con 0x7f7de415e170 2024-03-24T00:36:43.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7de3400640 1 -- 172.21.15.40:0/4200682141 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd40ada50 con 0x7f7de415e170 2024-03-24T00:36:43.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7deb000640 1 -- 172.21.15.40:0/4200682141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7de415e170 msgr2=0x7f7de406f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7deb000640 1 --2- 172.21.15.40:0/4200682141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7de415e170 0x7f7de406f7b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40097c0 tx=0x7f7dd401c470 comp rx=0 tx=0).stop 2024-03-24T00:36:43.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7deb000640 1 -- 172.21.15.40:0/4200682141 shutdown_connections 2024-03-24T00:36:43.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7deb000640 1 --2- 172.21.15.40:0/4200682141 >> v2:172.21.15.53:3300/0 conn(0x7f7de4160080 0x7f7de415e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7deb000640 1 --2- 172.21.15.40:0/4200682141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7de415e170 0x7f7de406f7b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7deb000640 1 -- 172.21.15.40:0/4200682141 >> 172.21.15.40:0/4200682141 conn(0x7f7de40b36a0 msgr2=0x7f7de40b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:43.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7deb000640 1 -- 172.21.15.40:0/4200682141 shutdown_connections 2024-03-24T00:36:43.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7deb000640 1 -- 172.21.15.40:0/4200682141 wait complete. 2024-03-24T00:36:43.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7deb000640 1 Processor -- start 2024-03-24T00:36:43.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7deb000640 1 -- start start 2024-03-24T00:36:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.738+0000 7f7deb000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7de40d0a10 0x7f7de41532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7deb000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7de415e170 0x7f7de406c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7deb000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4160080 0x7f7de4152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7deb000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de416b030 con 0x7f7de40d0a10 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7deb000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de416aeb0 con 0x7f7de4160080 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7deb000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de416b1b0 con 0x7f7de415e170 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7de40d0a10 0x7f7de41532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7de40d0a10 0x7f7de41532e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40150/0 (socket says 172.21.15.40:40150) 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de3e00640 1 -- 172.21.15.40:0/3268390940 learned_addr learned my addr 172.21.15.40:0/3268390940 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de9000640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4160080 0x7f7de4152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7ddb400640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7de415e170 0x7f7de406c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de3e00640 1 -- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7de415e170 msgr2=0x7f7de406c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de3e00640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7de415e170 0x7f7de406c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de3e00640 1 -- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4160080 msgr2=0x7f7de4152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de3e00640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4160080 0x7f7de4152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de3e00640 1 -- 172.21.15.40:0/3268390940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de41539e0 con 0x7f7de40d0a10 2024-03-24T00:36:43.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de3e00640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7de40d0a10 0x7f7de41532e0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40097c0 tx=0x7f7dd401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de0c00640 1 -- 172.21.15.40:0/3268390940 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd40be070 con 0x7f7de40d0a10 2024-03-24T00:36:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de0c00640 1 -- 172.21.15.40:0/3268390940 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd40b3040 con 0x7f7de40d0a10 2024-03-24T00:36:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de0c00640 1 -- 172.21.15.40:0/3268390940 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd40b2050 con 0x7f7de40d0a10 2024-03-24T00:36:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7deb000640 1 -- 172.21.15.40:0/3268390940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de4153cc0 con 0x7f7de40d0a10 2024-03-24T00:36:43.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7deb000640 1 -- 172.21.15.40:0/3268390940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de4154280 con 0x7f7de40d0a10 2024-03-24T00:36:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7deb000640 1 -- 172.21.15.40:0/3268390940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de4068170 con 0x7f7de40d0a10 2024-03-24T00:36:43.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.742+0000 7f7de0c00640 1 -- 172.21.15.40:0/3268390940 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7dd4004030 con 0x7f7de40d0a10 2024-03-24T00:36:43.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.746+0000 7f7de0c00640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7dc0067660 0x7f7dc0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:43.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.746+0000 7f7de0c00640 1 -- 172.21.15.40:0/3268390940 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7dd411f2d0 con 0x7f7de40d0a10 2024-03-24T00:36:43.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.746+0000 7f7ddb400640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7dc0067660 0x7f7dc0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:43.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.746+0000 7f7ddb400640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7dc0067660 0x7f7dc0069b20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7de4156620 tx=0x7f7dcc007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:43.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.750+0000 7f7de0c00640 1 -- 172.21.15.40:0/3268390940 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7dd4016090 con 0x7f7de40d0a10 2024-03-24T00:36:43.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.910+0000 7f7deb000640 1 -- 172.21.15.40:0/3268390940 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7de4154800 con 0x7f7de40d0a10 2024-03-24T00:36:43.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7de0c00640 1 -- 172.21.15.40:0/3268390940 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7dd40ef860 con 0x7f7de40d0a10 2024-03-24T00:36:43.916 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:36:43.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7deb000640 1 -- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7dc0067660 msgr2=0x7f7dc0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7deb000640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7dc0067660 0x7f7dc0069b20 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7de4156620 tx=0x7f7dcc007df0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7deb000640 1 -- 172.21.15.40:0/3268390940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7de40d0a10 msgr2=0x7f7de41532e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7deb000640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7de40d0a10 0x7f7de41532e0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40097c0 tx=0x7f7dd401acb0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7deb000640 1 -- 172.21.15.40:0/3268390940 shutdown_connections 2024-03-24T00:36:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7deb000640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7dc0067660 0x7f7dc0069b20 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7deb000640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7de4160080 0x7f7de4152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7deb000640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7de415e170 0x7f7de406c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7deb000640 1 --2- 172.21.15.40:0/3268390940 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7de40d0a10 0x7f7de41532e0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7deb000640 1 -- 172.21.15.40:0/3268390940 >> 172.21.15.40:0/3268390940 conn(0x7f7de40b36a0 msgr2=0x7f7de415b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7deb000640 1 -- 172.21.15.40:0/3268390940 shutdown_connections 2024-03-24T00:36:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:43.914+0000 7f7deb000640 1 -- 172.21.15.40:0/3268390940 wait complete. 2024-03-24T00:36:43.937 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:36:44.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf2bc00640 1 Processor -- start 2024-03-24T00:36:44.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf2bc00640 1 -- start start 2024-03-24T00:36:44.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf2bc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcf240b2360 0x7fcf240b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf2bc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf240b4330 0x7fcf240ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf2bc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf240cea00 0x7fcf2406b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf2bc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf240cee70 con 0x7fcf240b4330 2024-03-24T00:36:44.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf2bc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf240d0f60 con 0x7fcf240b2360 2024-03-24T00:36:44.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf2bc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240cf090 con 0x7fcf240cea00 2024-03-24T00:36:44.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf1be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf240b4330 0x7fcf240ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf1be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf240b4330 0x7fcf240ce4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40166/0 (socket says 172.21.15.40:40166) 2024-03-24T00:36:44.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf1be00640 1 -- 172.21.15.40:0/3320095460 learned_addr learned my addr 172.21.15.40:0/3320095460 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:44.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf29200640 1 --2- 172.21.15.40:0/3320095460 >> v2:172.21.15.53:3300/0 conn(0x7fcf240b2360 0x7fcf240b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf29c00640 1 --2- 172.21.15.40:0/3320095460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf240cea00 0x7fcf2406b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf1be00640 1 -- 172.21.15.40:0/3320095460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf240cea00 msgr2=0x7fcf2406b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:44.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf1be00640 1 --2- 172.21.15.40:0/3320095460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf240cea00 0x7fcf2406b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf1be00640 1 -- 172.21.15.40:0/3320095460 >> v2:172.21.15.53:3300/0 conn(0x7fcf240b2360 msgr2=0x7fcf240b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:44.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf1be00640 1 --2- 172.21.15.40:0/3320095460 >> v2:172.21.15.53:3300/0 conn(0x7fcf240b2360 0x7fcf240b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf1be00640 1 -- 172.21.15.40:0/3320095460 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf240cf330 con 0x7fcf240b4330 2024-03-24T00:36:44.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.078+0000 7fcf29c00640 1 --2- 172.21.15.40:0/3320095460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf240cea00 0x7fcf2406b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:44.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf1be00640 1 --2- 172.21.15.40:0/3320095460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf240b4330 0x7fcf240ce4c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fcf24073240 tx=0x7fcf0c01c110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a311d273c1daa458 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:44.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf23e00640 1 -- 172.21.15.40:0/3320095460 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0b0070 con 0x7fcf240b4330 2024-03-24T00:36:44.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf23e00640 1 -- 172.21.15.40:0/3320095460 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c0ab600 con 0x7fcf240b4330 2024-03-24T00:36:44.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf23e00640 1 -- 172.21.15.40:0/3320095460 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0ab9d0 con 0x7fcf240b4330 2024-03-24T00:36:44.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf2bc00640 1 -- 172.21.15.40:0/3320095460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf240b4330 msgr2=0x7fcf240ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf2bc00640 1 --2- 172.21.15.40:0/3320095460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf240b4330 0x7fcf240ce4c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fcf24073240 tx=0x7fcf0c01c110 comp rx=0 tx=0).stop 2024-03-24T00:36:44.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf2bc00640 1 -- 172.21.15.40:0/3320095460 shutdown_connections 2024-03-24T00:36:44.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf2bc00640 1 --2- 172.21.15.40:0/3320095460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf240cea00 0x7fcf2406b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf2bc00640 1 --2- 172.21.15.40:0/3320095460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf240b4330 0x7fcf240ce4c0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf2bc00640 1 --2- 172.21.15.40:0/3320095460 >> v2:172.21.15.53:3300/0 conn(0x7fcf240b2360 0x7fcf240b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf2bc00640 1 -- 172.21.15.40:0/3320095460 >> 172.21.15.40:0/3320095460 conn(0x7fcf240b3810 msgr2=0x7fcf240b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:44.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf2bc00640 1 -- 172.21.15.40:0/3320095460 shutdown_connections 2024-03-24T00:36:44.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.082+0000 7fcf2bc00640 1 -- 172.21.15.40:0/3320095460 wait complete. 2024-03-24T00:36:44.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf2bc00640 1 Processor -- start 2024-03-24T00:36:44.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf2bc00640 1 -- start start 2024-03-24T00:36:44.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf2bc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf240b2360 0x7fcf2415fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf2bc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf240b4330 0x7fcf24156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf2bc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf240cea00 0x7fcf24156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf2bc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240d1950 con 0x7fcf240b4330 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf2bc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240d17d0 con 0x7fcf240b2360 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf2bc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf240cc420 con 0x7fcf240cea00 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf240b4330 0x7fcf24156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf29200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf240b2360 0x7fcf2415fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf1be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf240b4330 0x7fcf24156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40182/0 (socket says 172.21.15.40:40182) 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf29200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf240b2360 0x7fcf2415fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:60464/0 (socket says 172.21.15.40:60464) 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf1be00640 1 -- 172.21.15.40:0/1512479577 learned_addr learned my addr 172.21.15.40:0/1512479577 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf29c00640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf240cea00 0x7fcf24156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf1be00640 1 -- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf240cea00 msgr2=0x7fcf24156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf1be00640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf240cea00 0x7fcf24156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf1be00640 1 -- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf240b2360 msgr2=0x7fcf2415fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf1be00640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf240b2360 0x7fcf2415fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf1be00640 1 -- 172.21.15.40:0/1512479577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf240743a0 con 0x7fcf240b4330 2024-03-24T00:36:44.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf1be00640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf240b4330 0x7fcf24156680 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0ae040 tx=0x7fcf0c018eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:44.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf21600640 1 -- 172.21.15.40:0/1512479577 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0b0040 con 0x7fcf240b4330 2024-03-24T00:36:44.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf21600640 1 -- 172.21.15.40:0/1512479577 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c0b4420 con 0x7fcf240b4330 2024-03-24T00:36:44.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf2bc00640 1 -- 172.21.15.40:0/1512479577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2415d3e0 con 0x7fcf240b4330 2024-03-24T00:36:44.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf21600640 1 -- 172.21.15.40:0/1512479577 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c0b4810 con 0x7fcf240b4330 2024-03-24T00:36:44.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf2bc00640 1 -- 172.21.15.40:0/1512479577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2415d9a0 con 0x7fcf240b4330 2024-03-24T00:36:44.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.086+0000 7fcf2bc00640 1 -- 172.21.15.40:0/1512479577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2415e060 con 0x7fcf240b4330 2024-03-24T00:36:44.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.090+0000 7fcf21600640 1 -- 172.21.15.40:0/1512479577 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcf0c0b49b0 con 0x7fcf240b4330 2024-03-24T00:36:44.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.090+0000 7fcf21600640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcf00067660 0x7fcf00069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.090+0000 7fcf21600640 1 -- 172.21.15.40:0/1512479577 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf0c11f4a0 con 0x7fcf240b4330 2024-03-24T00:36:44.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.090+0000 7fcf29200640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcf00067660 0x7fcf00069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.090+0000 7fcf29200640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcf00067660 0x7fcf00069b20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcf24156000 tx=0x7fcf14009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:44.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.094+0000 7fcf21600640 1 -- 172.21.15.40:0/1512479577 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcf2415e060 con 0x7fcf240b4330 2024-03-24T00:36:44.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 -- 172.21.15.40:0/1512479577 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcf24068170 con 0x7fcf240b4330 2024-03-24T00:36:44.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf21600640 1 -- 172.21.15.40:0/1512479577 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcf0c016110 con 0x7fcf240b4330 2024-03-24T00:36:44.261 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:36:44.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 -- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcf00067660 msgr2=0x7fcf00069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcf00067660 0x7fcf00069b20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcf24156000 tx=0x7fcf14009190 comp rx=0 tx=0).stop 2024-03-24T00:36:44.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 -- 172.21.15.40:0/1512479577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf240b4330 msgr2=0x7fcf24156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf240b4330 0x7fcf24156680 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0ae040 tx=0x7fcf0c018eb0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 -- 172.21.15.40:0/1512479577 shutdown_connections 2024-03-24T00:36:44.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcf00067660 0x7fcf00069b20 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf240cea00 0x7fcf24156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf240b4330 0x7fcf24156680 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 --2- 172.21.15.40:0/1512479577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf240b2360 0x7fcf2415fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 -- 172.21.15.40:0/1512479577 >> 172.21.15.40:0/1512479577 conn(0x7fcf240b3810 msgr2=0x7fcf24070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:44.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 -- 172.21.15.40:0/1512479577 shutdown_connections 2024-03-24T00:36:44.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.258+0000 7fcf2bc00640 1 -- 172.21.15.40:0/1512479577 wait complete. 2024-03-24T00:36:44.283 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:36:44.424 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe61800640 1 Processor -- start 2024-03-24T00:36:44.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe61800640 1 -- start start 2024-03-24T00:36:44.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe61800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 0x7fbe5c0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.425 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe61800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c070f60 con 0x7fbe5c06ea00 2024-03-24T00:36:44.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe61800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c0b3620 con 0x7fbe5c0ccfa0 2024-03-24T00:36:44.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe61800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5c0b4c00 con 0x7fbe5c06e620 2024-03-24T00:36:44.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe5aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 0x7fbe5c0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe53e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbe5c06ea00 0x7fbe5c0b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44588/0 (socket says 172.21.15.40:44588) 2024-03-24T00:36:44.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe5b400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbe5c0ccfa0 0x7fbe5c0cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:55110/0 (socket says 172.21.15.40:55110) 2024-03-24T00:36:44.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe53e00640 1 -- 172.21.15.40:0/301878342 learned_addr learned my addr 172.21.15.40:0/301878342 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:44.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe5aa00640 1 -- 172.21.15.40:0/301878342 >> v1:172.21.15.53:6789/0 conn(0x7fbe5c0ccfa0 legacy=0x7fbe5c0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe5aa00640 1 -- 172.21.15.40:0/301878342 >> v1:172.21.15.40:6789/0 conn(0x7fbe5c06ea00 legacy=0x7fbe5c0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe5a000640 1 -- 172.21.15.40:0/301878342 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139514259 0 0) 0x7fbe5c0b3620 con 0x7fbe5c0ccfa0 2024-03-24T00:36:44.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.422+0000 7fbe5aa00640 1 -- 172.21.15.40:0/301878342 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c0b3900 con 0x7fbe5c06e620 2024-03-24T00:36:44.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.426+0000 7fbe5aa00640 1 --2- 172.21.15.40:0/301878342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 0x7fbe5c0b46c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c0097f0 tx=0x7fbe4c01c2b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a176d85c7d1f99f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:44.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.426+0000 7fbe5a000640 1 -- 172.21.15.40:0/301878342 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c0b0070 con 0x7fbe5c06e620 2024-03-24T00:36:44.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.426+0000 7fbe5a000640 1 -- 172.21.15.40:0/301878342 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4c0ab600 con 0x7fbe5c06e620 2024-03-24T00:36:44.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.426+0000 7fbe5a000640 1 -- 172.21.15.40:0/301878342 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c0ab9d0 con 0x7fbe5c06e620 2024-03-24T00:36:44.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.426+0000 7fbe61800640 1 -- 172.21.15.40:0/301878342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 msgr2=0x7fbe5c0b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.426+0000 7fbe61800640 1 --2- 172.21.15.40:0/301878342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 0x7fbe5c0b46c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c0097f0 tx=0x7fbe4c01c2b0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.426+0000 7fbe61800640 1 -- 172.21.15.40:0/301878342 shutdown_connections 2024-03-24T00:36:44.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.426+0000 7fbe61800640 1 --2- 172.21.15.40:0/301878342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 0x7fbe5c0b46c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.426+0000 7fbe61800640 1 -- 172.21.15.40:0/301878342 >> 172.21.15.40:0/301878342 conn(0x7fbe5c0b17b0 msgr2=0x7fbe5c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:44.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.426+0000 7fbe61800640 1 -- 172.21.15.40:0/301878342 shutdown_connections 2024-03-24T00:36:44.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.426+0000 7fbe61800640 1 -- 172.21.15.40:0/301878342 wait complete. 2024-03-24T00:36:44.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe61800640 1 Processor -- start 2024-03-24T00:36:44.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe61800640 1 -- start start 2024-03-24T00:36:44.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe61800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 0x7fbe5c0aee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe61800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe5c06ea00 0x7fbe5c143920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe61800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe5c0ccfa0 0x7fbe5c133b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe61800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5c144920 con 0x7fbe5c0ccfa0 2024-03-24T00:36:44.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe61800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5c06ee20 con 0x7fbe5c06ea00 2024-03-24T00:36:44.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe61800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5c0add70 con 0x7fbe5c06e620 2024-03-24T00:36:44.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe5b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe5c0ccfa0 0x7fbe5c133b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe5aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 0x7fbe5c0aee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe53e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe5c06ea00 0x7fbe5c143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe5aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 0x7fbe5c0aee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:57712/0 (socket says 172.21.15.40:57712) 2024-03-24T00:36:44.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe5aa00640 1 -- 172.21.15.40:0/226135596 learned_addr learned my addr 172.21.15.40:0/226135596 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:44.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe5aa00640 1 -- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe5c06ea00 msgr2=0x7fbe5c143920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe5aa00640 1 --2- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe5c06ea00 0x7fbe5c143920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe5aa00640 1 -- 172.21.15.40:0/226135596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe5c0ccfa0 msgr2=0x7fbe5c133b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe5aa00640 1 --2- 172.21.15.40:0/226135596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe5c0ccfa0 0x7fbe5c133b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe5aa00640 1 -- 172.21.15.40:0/226135596 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c0ae050 con 0x7fbe5c06e620 2024-03-24T00:36:44.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe5aa00640 1 --2- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 0x7fbe5c0aee10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c019ee0 tx=0x7fbe4c01aaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:44.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe52a00640 1 -- 172.21.15.40:0/226135596 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c0b0050 con 0x7fbe5c06e620 2024-03-24T00:36:44.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe52a00640 1 -- 172.21.15.40:0/226135596 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe4c0bc070 con 0x7fbe5c06e620 2024-03-24T00:36:44.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe52a00640 1 -- 172.21.15.40:0/226135596 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c0b5a70 con 0x7fbe5c06e620 2024-03-24T00:36:44.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe61800640 1 -- 172.21.15.40:0/226135596 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5c0ae330 con 0x7fbe5c06e620 2024-03-24T00:36:44.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.430+0000 7fbe61800640 1 -- 172.21.15.40:0/226135596 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5c0ae8f0 con 0x7fbe5c06e620 2024-03-24T00:36:44.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.434+0000 7fbe61800640 1 -- 172.21.15.40:0/226135596 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe5c068170 con 0x7fbe5c06e620 2024-03-24T00:36:44.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.434+0000 7fbe52a00640 1 -- 172.21.15.40:0/226135596 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbe4c0ab480 con 0x7fbe5c06e620 2024-03-24T00:36:44.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.434+0000 7fbe52a00640 1 --2- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbe38067660 0x7fbe38069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:44.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.434+0000 7fbe52a00640 1 -- 172.21.15.40:0/226135596 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe4c1201d0 con 0x7fbe5c06e620 2024-03-24T00:36:44.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.434+0000 7fbe53e00640 1 --2- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbe38067660 0x7fbe38069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:44.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.434+0000 7fbe53e00640 1 --2- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbe38067660 0x7fbe38069b20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c135b50 tx=0x7fbe44007e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:44.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.438+0000 7fbe52a00640 1 -- 172.21.15.40:0/226135596 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbe4c015ff0 con 0x7fbe5c06e620 2024-03-24T00:36:44.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.622+0000 7fbe61800640 1 -- 172.21.15.40:0/226135596 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbe5c072f50 con 0x7fbe5c06e620 2024-03-24T00:36:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.622+0000 7fbe52a00640 1 -- 172.21.15.40:0/226135596 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v83) v1 ==== 80+0+0 (secure 0 0 0) 0x7fbe4c0f0920 con 0x7fbe5c06e620 2024-03-24T00:36:44.627 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.626+0000 7fbe61800640 1 -- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbe38067660 msgr2=0x7fbe38069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.626+0000 7fbe61800640 1 --2- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbe38067660 0x7fbe38069b20 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c135b50 tx=0x7fbe44007e10 comp rx=0 tx=0).stop 2024-03-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.626+0000 7fbe61800640 1 -- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 msgr2=0x7fbe5c0aee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.626+0000 7fbe61800640 1 --2- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 0x7fbe5c0aee10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c019ee0 tx=0x7fbe4c01aaf0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.626+0000 7fbe61800640 1 -- 172.21.15.40:0/226135596 shutdown_connections 2024-03-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.626+0000 7fbe61800640 1 --2- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbe38067660 0x7fbe38069b20 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.626+0000 7fbe61800640 1 --2- 172.21.15.40:0/226135596 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbe5c0ccfa0 0x7fbe5c133b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.626+0000 7fbe61800640 1 --2- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe5c06ea00 0x7fbe5c143920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.626+0000 7fbe61800640 1 --2- 172.21.15.40:0/226135596 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe5c06e620 0x7fbe5c0aee10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.626+0000 7fbe61800640 1 -- 172.21.15.40:0/226135596 >> 172.21.15.40:0/226135596 conn(0x7fbe5c0b17b0 msgr2=0x7fbe5c0b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.626+0000 7fbe61800640 1 -- 172.21.15.40:0/226135596 shutdown_connections 2024-03-24T00:36:44.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:44.626+0000 7fbe61800640 1 -- 172.21.15.40:0/226135596 wait complete. 2024-03-24T00:36:44.648 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-03-24T00:36:44.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:44.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T00:36:44.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[46469]: fuse finished with error 0 and tester_r 0 2024-03-24T00:36:44.671 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:36:44.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:44.671 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:36:44.845 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:36:44.845 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:36:44.854 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:36:44.854 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:36:44.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:44.854 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T00:36:44.903 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T00:36:44.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:36:44.904 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:36:44.904 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:36:44.912 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:36:44.913 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:36:44.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:44.913 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T00:36:44.962 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T00:36:44.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:36:44.964 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:36:44.964 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:36:44.973 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:36:45.094 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:36:45.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:36:45.094 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T00:36:45.102 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T00:36:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:36:45.103 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:36:45.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.238+0000 7f2fb2e00640 1 Processor -- start 2024-03-24T00:36:45.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.238+0000 7f2fb2e00640 1 -- start start 2024-03-24T00:36:45.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.242+0000 7f2fb2e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2fac0cfdf0 0x7f2fac0d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.242+0000 7f2fb2e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fac0af710 con 0x7f2fac15f7a0 2024-03-24T00:36:45.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.242+0000 7f2fb2e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fac0b34f0 con 0x7f2fac0b43a0 2024-03-24T00:36:45.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.242+0000 7f2fb2e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fac0b2bb0 con 0x7f2fac0cfdf0 2024-03-24T00:36:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.242+0000 7f2fabe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2fac0cfdf0 0x7f2fac0d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.242+0000 7f2fa3400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2fac0b43a0 0x7f2fac15f090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:55118/0 (socket says 172.21.15.40:55118) 2024-03-24T00:36:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.242+0000 7f2fb0e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f2fac15f7a0 0x7f2fac0afbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44602/0 (socket says 172.21.15.40:44602) 2024-03-24T00:36:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.242+0000 7f2fa3400640 1 -- 172.21.15.40:0/3141086710 learned_addr learned my addr 172.21.15.40:0/3141086710 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.242+0000 7f2fabe00640 1 -- 172.21.15.40:0/3141086710 >> v1:172.21.15.53:6789/0 conn(0x7f2fac0b43a0 legacy=0x7f2fac15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.242+0000 7f2fabe00640 1 -- 172.21.15.40:0/3141086710 >> v1:172.21.15.40:6789/0 conn(0x7f2fac15f7a0 legacy=0x7f2fac0afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.242+0000 7f2fabe00640 1 -- 172.21.15.40:0/3141086710 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac0b28d0 con 0x7f2fac0cfdf0 2024-03-24T00:36:45.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fabe00640 1 --2- 172.21.15.40:0/3141086710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fac0cfdf0 0x7f2fac0d01d0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0097c0 tx=0x7f2f9c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b90233559dc1e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:45.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fab400640 1 -- 172.21.15.40:0/3141086710 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c0b0070 con 0x7f2fac0cfdf0 2024-03-24T00:36:45.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fab400640 1 -- 172.21.15.40:0/3141086710 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac0d0b20 con 0x7f2fac0cfdf0 2024-03-24T00:36:45.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fab400640 1 -- 172.21.15.40:0/3141086710 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c0ab820 con 0x7f2fac0cfdf0 2024-03-24T00:36:45.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fb2e00640 1 -- 172.21.15.40:0/3141086710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fac0cfdf0 msgr2=0x7f2fac0d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fb2e00640 1 --2- 172.21.15.40:0/3141086710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fac0cfdf0 0x7f2fac0d01d0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0097c0 tx=0x7f2f9c01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fb2e00640 1 -- 172.21.15.40:0/3141086710 shutdown_connections 2024-03-24T00:36:45.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fb2e00640 1 --2- 172.21.15.40:0/3141086710 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fac0cfdf0 0x7f2fac0d01d0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fb2e00640 1 -- 172.21.15.40:0/3141086710 >> 172.21.15.40:0/3141086710 conn(0x7f2fac0b3880 msgr2=0x7f2fac0b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:45.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fb2e00640 1 -- 172.21.15.40:0/3141086710 shutdown_connections 2024-03-24T00:36:45.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fb2e00640 1 -- 172.21.15.40:0/3141086710 wait complete. 2024-03-24T00:36:45.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fb2e00640 1 Processor -- start 2024-03-24T00:36:45.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fb2e00640 1 -- start start 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fb2e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fac0b43a0 0x7f2fac1534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.246+0000 7f2fb2e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fac0cfdf0 0x7f2fac0b7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb2e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fac15f7a0 0x7f2fac0d1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb2e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0b0ee0 con 0x7f2fac15f7a0 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb2e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0b0d60 con 0x7f2fac0cfdf0 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb2e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0b32a0 con 0x7f2fac0b43a0 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb0e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fac15f7a0 0x7f2fac0d1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fabe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fac0b43a0 0x7f2fac1534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb0e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fac15f7a0 0x7f2fac0d1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40212/0 (socket says 172.21.15.40:40212) 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb0e00640 1 -- 172.21.15.40:0/2760098314 learned_addr learned my addr 172.21.15.40:0/2760098314 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fa3400640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fac0cfdf0 0x7f2fac0b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb0e00640 1 -- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fac0b43a0 msgr2=0x7f2fac1534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb0e00640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fac0b43a0 0x7f2fac1534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb0e00640 1 -- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fac0cfdf0 msgr2=0x7f2fac0b7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb0e00640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fac0cfdf0 0x7f2fac0b7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb0e00640 1 -- 172.21.15.40:0/2760098314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac0c92b0 con 0x7f2fac15f7a0 2024-03-24T00:36:45.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb0e00640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fac15f7a0 0x7f2fac0d1c90 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400c740 tx=0x7f2fa400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:45.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fa8c00640 1 -- 172.21.15.40:0/2760098314 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa4017070 con 0x7f2fac15f7a0 2024-03-24T00:36:45.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fa8c00640 1 -- 172.21.15.40:0/2760098314 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa400ce50 con 0x7f2fac15f7a0 2024-03-24T00:36:45.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fa8c00640 1 -- 172.21.15.40:0/2760098314 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400a880 con 0x7f2fac15f7a0 2024-03-24T00:36:45.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb2e00640 1 -- 172.21.15.40:0/2760098314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fac0c9590 con 0x7f2fac15f7a0 2024-03-24T00:36:45.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb2e00640 1 -- 172.21.15.40:0/2760098314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fac0c9b50 con 0x7f2fac15f7a0 2024-03-24T00:36:45.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fb2e00640 1 -- 172.21.15.40:0/2760098314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fac068170 con 0x7f2fac15f7a0 2024-03-24T00:36:45.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.250+0000 7f2fa8c00640 1 -- 172.21.15.40:0/2760098314 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2fa401a070 con 0x7f2fac15f7a0 2024-03-24T00:36:45.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.254+0000 7f2fa8c00640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f88067660 0x7f2f88069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.254+0000 7f2fa8c00640 1 -- 172.21.15.40:0/2760098314 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2fa4084f10 con 0x7f2fac15f7a0 2024-03-24T00:36:45.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.254+0000 7f2fabe00640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f88067660 0x7f2f88069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.254+0000 7f2fabe00640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f88067660 0x7f2f88069b20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0ad000 tx=0x7f2f9c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:45.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.258+0000 7f2fa8c00640 1 -- 172.21.15.40:0/2760098314 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2fa40554a0 con 0x7f2fac15f7a0 2024-03-24T00:36:45.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 -- 172.21.15.40:0/2760098314 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2fac0cdc00 con 0x7f2fac15f7a0 2024-03-24T00:36:45.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fa8c00640 1 -- 172.21.15.40:0/2760098314 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5015 (secure 0 0 0) 0x7f2fa4059540 con 0x7f2fac15f7a0 2024-03-24T00:36:45.468 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:45.469 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":88,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1097448819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1097448819},{"type":"v1","addr":"172.21.15.40:6834","nonce":1097448819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:36:07.163876+0000","modified":"2024-03-24T00:36:30.866122+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7186,"mds_1":7180,"mds_2":6404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6404":{"gid":6404,"name":"b","rank":2,"incarnation":86,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/983819364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":983819364},{"type":"v1","addr":"172.21.15.53:6837","nonce":983819364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7180":{"gid":7180,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6835/2302548193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2302548193},{"type":"v1","addr":"172.21.15.40:6835","nonce":2302548193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7186":{"gid":7186,"name":"d","rank":0,"incarnation":78,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/1878888046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1878888046},{"type":"v1","addr":"172.21.15.53:6835","nonce":1878888046}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7186,"qdb_cluster":[7186,7180,6404]},"id":6}]} 2024-03-24T00:36:45.469 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 88 2024-03-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 -- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f88067660 msgr2=0x7f2f88069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f88067660 0x7f2f88069b20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0ad000 tx=0x7f2f9c0a6000 comp rx=0 tx=0).stop 2024-03-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 -- 172.21.15.40:0/2760098314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fac15f7a0 msgr2=0x7f2fac0d1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fac15f7a0 0x7f2fac0d1c90 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400c740 tx=0x7f2fa400cc00 comp rx=0 tx=0).stop 2024-03-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 -- 172.21.15.40:0/2760098314 shutdown_connections 2024-03-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2f88067660 0x7f2f88069b20 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2fac15f7a0 0x7f2fac0d1c90 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fac0cfdf0 0x7f2fac0b7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 --2- 172.21.15.40:0/2760098314 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fac0b43a0 0x7f2fac1534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 -- 172.21.15.40:0/2760098314 >> 172.21.15.40:0/2760098314 conn(0x7f2fac0b3880 msgr2=0x7f2fac0ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 -- 172.21.15.40:0/2760098314 shutdown_connections 2024-03-24T00:36:45.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.466+0000 7f2fb2e00640 1 -- 172.21.15.40:0/2760098314 wait complete. 2024-03-24T00:36:45.489 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:36:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.626+0000 7f9d52800640 1 Processor -- start 2024-03-24T00:36:45.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.626+0000 7f9d52800640 1 -- start start 2024-03-24T00:36:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d52800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9d4c0b2aa0 0x7f9d4c0d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d52800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9d4c0d0a80 0x7f9d4c0cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d52800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c0b2870 con 0x7f9d4c0b2490 2024-03-24T00:36:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d52800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c06b6d0 con 0x7f9d4c0b2aa0 2024-03-24T00:36:45.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d52800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c0b3190 con 0x7f9d4c0d0a80 2024-03-24T00:36:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d43e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f9d4c0b2aa0 0x7f9d4c0d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d4b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9d4c0b2490 0x7f9d4c0cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44616/0 (socket says 172.21.15.40:44616) 2024-03-24T00:36:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d4b400640 1 -- 172.21.15.40:0/308415571 learned_addr learned my addr 172.21.15.40:0/308415571 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d4be00640 1 --2- 172.21.15.40:0/308415571 >> v2:172.21.15.53:3300/0 conn(0x7f9d4c0d0a80 0x7f9d4c0cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d43e00640 1 -- 172.21.15.40:0/308415571 >> v2:172.21.15.53:3300/0 conn(0x7f9d4c0d0a80 msgr2=0x7f9d4c0cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d43e00640 1 --2- 172.21.15.40:0/308415571 >> v2:172.21.15.53:3300/0 conn(0x7f9d4c0d0a80 0x7f9d4c0cc460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d43e00640 1 -- 172.21.15.40:0/308415571 >> v1:172.21.15.40:6789/0 conn(0x7f9d4c0b2490 legacy=0x7f9d4c0cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.630+0000 7f9d43e00640 1 -- 172.21.15.40:0/308415571 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c0b3470 con 0x7f9d4c0b2aa0 2024-03-24T00:36:45.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d43e00640 1 --2- 172.21.15.40:0/308415571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d4c0b2aa0 0x7f9d4c0d0540 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c072990 tx=0x7f9d3401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97fe5f091a5547c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d4aa00640 1 -- 172.21.15.40:0/308415571 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340b1070 con 0x7f9d4c0b2aa0 2024-03-24T00:36:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d4aa00640 1 -- 172.21.15.40:0/308415571 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d340ac5f0 con 0x7f9d4c0b2aa0 2024-03-24T00:36:45.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d4aa00640 1 -- 172.21.15.40:0/308415571 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340ac9c0 con 0x7f9d4c0b2aa0 2024-03-24T00:36:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 -- 172.21.15.40:0/308415571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d4c0b2aa0 msgr2=0x7f9d4c0d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 --2- 172.21.15.40:0/308415571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d4c0b2aa0 0x7f9d4c0d0540 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c072990 tx=0x7f9d3401c480 comp rx=0 tx=0).stop 2024-03-24T00:36:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 -- 172.21.15.40:0/308415571 shutdown_connections 2024-03-24T00:36:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 --2- 172.21.15.40:0/308415571 >> v2:172.21.15.53:3300/0 conn(0x7f9d4c0d0a80 0x7f9d4c0cc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 --2- 172.21.15.40:0/308415571 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d4c0b2aa0 0x7f9d4c0d0540 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 -- 172.21.15.40:0/308415571 >> 172.21.15.40:0/308415571 conn(0x7f9d4c0d1d10 msgr2=0x7f9d4c0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 -- 172.21.15.40:0/308415571 shutdown_connections 2024-03-24T00:36:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 -- 172.21.15.40:0/308415571 wait complete. 2024-03-24T00:36:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 Processor -- start 2024-03-24T00:36:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 -- start start 2024-03-24T00:36:45.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c0b2490 0x7f9d4c1514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d4c0b2aa0 0x7f9d4c15cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d4c0d0a80 0x7f9d4c15d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0cd580 con 0x7f9d4c0b2aa0 2024-03-24T00:36:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0cd400 con 0x7f9d4c0b2490 2024-03-24T00:36:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.634+0000 7f9d52800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c0cd700 con 0x7f9d4c0d0a80 2024-03-24T00:36:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d4b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c0b2490 0x7f9d4c1514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d43e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d4c0b2aa0 0x7f9d4c15cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d4be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d4c0d0a80 0x7f9d4c15d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d4b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c0b2490 0x7f9d4c1514b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:60484/0 (socket says 172.21.15.40:60484) 2024-03-24T00:36:45.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d4b400640 1 -- 172.21.15.40:0/825083633 learned_addr learned my addr 172.21.15.40:0/825083633 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d4b400640 1 -- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d4c0d0a80 msgr2=0x7f9d4c15d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d4b400640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d4c0d0a80 0x7f9d4c15d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d4b400640 1 -- 172.21.15.40:0/825083633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d4c0b2aa0 msgr2=0x7f9d4c15cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d4b400640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d4c0b2aa0 0x7f9d4c15cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d4b400640 1 -- 172.21.15.40:0/825083633 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c0af640 con 0x7f9d4c0b2490 2024-03-24T00:36:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d43e00640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d4c0b2aa0 0x7f9d4c15cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d4b400640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c0b2490 0x7f9d4c1514b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00c370 tx=0x7f9d3c00c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d43400640 1 -- 172.21.15.40:0/825083633 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c017070 con 0x7f9d4c0b2490 2024-03-24T00:36:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d43400640 1 -- 172.21.15.40:0/825083633 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3c005cf0 con 0x7f9d4c0b2490 2024-03-24T00:36:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d43400640 1 -- 172.21.15.40:0/825083633 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c0072f0 con 0x7f9d4c0b2490 2024-03-24T00:36:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d52800640 1 -- 172.21.15.40:0/825083633 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4c1600e0 con 0x7f9d4c0b2490 2024-03-24T00:36:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d52800640 1 -- 172.21.15.40:0/825083633 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4c1589b0 con 0x7f9d4c0b2490 2024-03-24T00:36:45.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.638+0000 7f9d52800640 1 -- 172.21.15.40:0/825083633 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4c068170 con 0x7f9d4c0b2490 2024-03-24T00:36:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.642+0000 7f9d43400640 1 -- 172.21.15.40:0/825083633 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f9d3c01a070 con 0x7f9d4c0b2490 2024-03-24T00:36:45.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.642+0000 7f9d43400640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9d28067660 0x7f9d28069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.642+0000 7f9d43400640 1 -- 172.21.15.40:0/825083633 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9d3c084b60 con 0x7f9d4c0b2490 2024-03-24T00:36:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.642+0000 7f9d43e00640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9d28067660 0x7f9d28069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.642+0000 7f9d43e00640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9d28067660 0x7f9d28069b20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9d340ae000 tx=0x7f9d34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:45.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.646+0000 7f9d43400640 1 -- 172.21.15.40:0/825083633 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9d3c055200 con 0x7f9d4c0b2490 2024-03-24T00:36:45.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.818+0000 7f9d52800640 1 -- 172.21.15.40:0/825083633 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d4c0c8ef0 con 0x7f9d4c0b2490 2024-03-24T00:36:45.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.818+0000 7f9d43400640 1 -- 172.21.15.40:0/825083633 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16600 (secure 0 0 0) 0x7f9d3c0592a0 con 0x7f9d4c0b2490 2024-03-24T00:36:45.822 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:45.823 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":83,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:36:08.785471+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:05.573740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-24T00:36:06.150226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:36:45.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.822+0000 7f9d52800640 1 -- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9d28067660 msgr2=0x7f9d28069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.822+0000 7f9d52800640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9d28067660 0x7f9d28069b20 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9d340ae000 tx=0x7f9d34008040 comp rx=0 tx=0).stop 2024-03-24T00:36:45.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.822+0000 7f9d52800640 1 -- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c0b2490 msgr2=0x7f9d4c1514b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.822+0000 7f9d52800640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c0b2490 0x7f9d4c1514b0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c00c370 tx=0x7f9d3c00c830 comp rx=0 tx=0).stop 2024-03-24T00:36:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.822+0000 7f9d52800640 1 -- 172.21.15.40:0/825083633 shutdown_connections 2024-03-24T00:36:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.822+0000 7f9d52800640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9d28067660 0x7f9d28069b20 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.822+0000 7f9d52800640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d4c0d0a80 0x7f9d4c15d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.822+0000 7f9d52800640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9d4c0b2aa0 0x7f9d4c15cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.822+0000 7f9d52800640 1 --2- 172.21.15.40:0/825083633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c0b2490 0x7f9d4c1514b0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.822+0000 7f9d52800640 1 -- 172.21.15.40:0/825083633 >> 172.21.15.40:0/825083633 conn(0x7f9d4c0d1d10 msgr2=0x7f9d4c0c9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.822+0000 7f9d52800640 1 -- 172.21.15.40:0/825083633 shutdown_connections 2024-03-24T00:36:45.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.822+0000 7f9d52800640 1 -- 172.21.15.40:0/825083633 wait complete. 2024-03-24T00:36:45.842 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:36:05.573740+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:36:06.150226+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:36:45.843 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:36:45.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91b1c00640 1 Processor -- start 2024-03-24T00:36:45.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91b1c00640 1 -- start start 2024-03-24T00:36:45.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91b1c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91ac0d2270 0x7f91ac070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91b1c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac06d0f0 con 0x7f91ac15df80 2024-03-24T00:36:45.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91b1c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ac06b280 con 0x7f91ac06c950 2024-03-24T00:36:45.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91b1c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ac06b890 con 0x7f91ac0d2270 2024-03-24T00:36:45.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91aaa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f91ac15df80 0x7f91ac0d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44620/0 (socket says 172.21.15.40:44620) 2024-03-24T00:36:45.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91aa000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f91ac0d2270 0x7f91ac070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.989 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91aaa00640 1 -- 172.21.15.40:0/4165558018 learned_addr learned my addr 172.21.15.40:0/4165558018 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:45.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91a9600640 1 -- 172.21.15.40:0/4165558018 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798160273 0 0) 0x7f91ac06d0f0 con 0x7f91ac15df80 2024-03-24T00:36:45.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91aa000640 1 -- 172.21.15.40:0/4165558018 >> v1:172.21.15.53:6789/0 conn(0x7f91ac06c950 legacy=0x7f91ac161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91aa000640 1 -- 172.21.15.40:0/4165558018 >> v1:172.21.15.40:6789/0 conn(0x7f91ac15df80 legacy=0x7f91ac0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.986+0000 7f91aa000640 1 -- 172.21.15.40:0/4165558018 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac06ce10 con 0x7f91ac0d2270 2024-03-24T00:36:45.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.990+0000 7f91aa000640 1 --2- 172.21.15.40:0/4165558018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91ac0d2270 0x7f91ac070ff0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f91ac06ba10 tx=0x7f919801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c974165aa37650 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:45.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.990+0000 7f91a9600640 1 -- 172.21.15.40:0/4165558018 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91980ab430 con 0x7f91ac0d2270 2024-03-24T00:36:45.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.990+0000 7f91a9600640 1 -- 172.21.15.40:0/4165558018 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91980b0070 con 0x7f91ac0d2270 2024-03-24T00:36:45.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.990+0000 7f91a9600640 1 -- 172.21.15.40:0/4165558018 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91980ab7f0 con 0x7f91ac0d2270 2024-03-24T00:36:45.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.990+0000 7f91b1c00640 1 -- 172.21.15.40:0/4165558018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91ac0d2270 msgr2=0x7f91ac070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:45.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.990+0000 7f91b1c00640 1 --2- 172.21.15.40:0/4165558018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91ac0d2270 0x7f91ac070ff0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f91ac06ba10 tx=0x7f919801c470 comp rx=0 tx=0).stop 2024-03-24T00:36:45.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.990+0000 7f91b1c00640 1 -- 172.21.15.40:0/4165558018 shutdown_connections 2024-03-24T00:36:45.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.990+0000 7f91b1c00640 1 --2- 172.21.15.40:0/4165558018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91ac0d2270 0x7f91ac070ff0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:45.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.990+0000 7f91b1c00640 1 -- 172.21.15.40:0/4165558018 >> 172.21.15.40:0/4165558018 conn(0x7f91ac06ecf0 msgr2=0x7f91ac06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:45.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.990+0000 7f91b1c00640 1 -- 172.21.15.40:0/4165558018 shutdown_connections 2024-03-24T00:36:45.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.990+0000 7f91b1c00640 1 -- 172.21.15.40:0/4165558018 wait complete. 2024-03-24T00:36:45.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91b1c00640 1 Processor -- start 2024-03-24T00:36:45.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91b1c00640 1 -- start start 2024-03-24T00:36:45.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91b1c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91ac06c950 0x7f91ac15ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91b1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91ac0d2270 0x7f91ac152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91b1c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ac15df80 0x7f91ac15a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:45.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91b1c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac16bfa0 con 0x7f91ac0d2270 2024-03-24T00:36:45.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91b1c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac16be20 con 0x7f91ac06c950 2024-03-24T00:36:45.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91b1c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ac16c120 con 0x7f91ac15df80 2024-03-24T00:36:45.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91aa000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91ac0d2270 0x7f91ac152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91aaa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91ac06c950 0x7f91ac15ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:45.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91aa000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91ac0d2270 0x7f91ac152940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40256/0 (socket says 172.21.15.40:40256) 2024-03-24T00:36:45.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91aa000640 1 -- 172.21.15.40:0/51017271 learned_addr learned my addr 172.21.15.40:0/51017271 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:45.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91ab400640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ac15df80 0x7f91ac15a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91aa000640 1 -- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ac15df80 msgr2=0x7f91ac15a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91aa000640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ac15df80 0x7f91ac15a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91aa000640 1 -- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91ac06c950 msgr2=0x7f91ac15ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91aa000640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91ac06c950 0x7f91ac15ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91aa000640 1 -- 172.21.15.40:0/51017271 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ac155180 con 0x7f91ac0d2270 2024-03-24T00:36:46.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91aa000640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91ac0d2270 0x7f91ac152940 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f919801a0a0 tx=0x7f919801acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:46.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91a2a00640 1 -- 172.21.15.40:0/51017271 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198004220 con 0x7f91ac0d2270 2024-03-24T00:36:46.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91a2a00640 1 -- 172.21.15.40:0/51017271 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91980043c0 con 0x7f91ac0d2270 2024-03-24T00:36:46.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91a2a00640 1 -- 172.21.15.40:0/51017271 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198003820 con 0x7f91ac0d2270 2024-03-24T00:36:46.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91b1c00640 1 -- 172.21.15.40:0/51017271 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ac15cfb0 con 0x7f91ac0d2270 2024-03-24T00:36:46.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.994+0000 7f91b1c00640 1 -- 172.21.15.40:0/51017271 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ac154b30 con 0x7f91ac0d2270 2024-03-24T00:36:46.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.998+0000 7f91b1c00640 1 -- 172.21.15.40:0/51017271 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ac068170 con 0x7f91ac0d2270 2024-03-24T00:36:46.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.998+0000 7f91a2a00640 1 -- 172.21.15.40:0/51017271 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f91980039c0 con 0x7f91ac0d2270 2024-03-24T00:36:46.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.998+0000 7f91a2a00640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f918c067660 0x7f918c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.998+0000 7f91a2a00640 1 -- 172.21.15.40:0/51017271 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f91980a8020 con 0x7f91ac0d2270 2024-03-24T00:36:46.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.998+0000 7f91aaa00640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f918c067660 0x7f918c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:45.998+0000 7f91aaa00640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f918c067660 0x7f918c069b20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f919c000f70 tx=0x7f919c007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:46.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.002+0000 7f91a2a00640 1 -- 172.21.15.40:0/51017271 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f91980c4030 con 0x7f91ac0d2270 2024-03-24T00:36:46.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.186+0000 7f91b1c00640 1 -- 172.21.15.40:0/51017271 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91ac15a2a0 con 0x7f91ac0d2270 2024-03-24T00:36:46.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.186+0000 7f91a2a00640 1 -- 172.21.15.40:0/51017271 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9198016090 con 0x7f91ac0d2270 2024-03-24T00:36:46.191 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:36:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.190+0000 7f91b1c00640 1 -- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f918c067660 msgr2=0x7f918c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.190+0000 7f91b1c00640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f918c067660 0x7f918c069b20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f919c000f70 tx=0x7f919c007240 comp rx=0 tx=0).stop 2024-03-24T00:36:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.190+0000 7f91b1c00640 1 -- 172.21.15.40:0/51017271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91ac0d2270 msgr2=0x7f91ac152940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.190+0000 7f91b1c00640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91ac0d2270 0x7f91ac152940 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f919801a0a0 tx=0x7f919801acb0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.190+0000 7f91b1c00640 1 -- 172.21.15.40:0/51017271 shutdown_connections 2024-03-24T00:36:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.190+0000 7f91b1c00640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f918c067660 0x7f918c069b20 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.190+0000 7f91b1c00640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ac15df80 0x7f91ac15a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.190+0000 7f91b1c00640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f91ac0d2270 0x7f91ac152940 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.190+0000 7f91b1c00640 1 --2- 172.21.15.40:0/51017271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91ac06c950 0x7f91ac15ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.190+0000 7f91b1c00640 1 -- 172.21.15.40:0/51017271 >> 172.21.15.40:0/51017271 conn(0x7f91ac06ecf0 msgr2=0x7f91ac158ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.190+0000 7f91b1c00640 1 -- 172.21.15.40:0/51017271 shutdown_connections 2024-03-24T00:36:46.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.190+0000 7f91b1c00640 1 -- 172.21.15.40:0/51017271 wait complete. 2024-03-24T00:36:46.212 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:36:46.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcf05000640 1 Processor -- start 2024-03-24T00:36:46.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcf05000640 1 -- start start 2024-03-24T00:36:46.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcf05000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcf000b32e0 0x7fcf000b6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcf05000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf000b7460 0x7fcf000d2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcf05000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf000b97a0 0x7fcf000d32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcf05000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf000cf560 con 0x7fcf000b7460 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcf05000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf000d37f0 con 0x7fcf000b32e0 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcf05000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf000cecc0 con 0x7fcf000b97a0 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcef5400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf000b7460 0x7fcf000d2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcef5400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcf000b7460 0x7fcf000d2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40264/0 (socket says 172.21.15.40:40264) 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcef5400640 1 -- 172.21.15.40:0/3460001932 learned_addr learned my addr 172.21.15.40:0/3460001932 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcefe800640 1 --2- 172.21.15.40:0/3460001932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf000b97a0 0x7fcf000d32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcefde00640 1 --2- 172.21.15.40:0/3460001932 >> v2:172.21.15.53:3300/0 conn(0x7fcf000b32e0 0x7fcf000b6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcef5400640 1 -- 172.21.15.40:0/3460001932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf000b97a0 msgr2=0x7fcf000d32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcef5400640 1 --2- 172.21.15.40:0/3460001932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf000b97a0 0x7fcf000d32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcef5400640 1 -- 172.21.15.40:0/3460001932 >> v2:172.21.15.53:3300/0 conn(0x7fcf000b32e0 msgr2=0x7fcf000b6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcef5400640 1 --2- 172.21.15.40:0/3460001932 >> v2:172.21.15.53:3300/0 conn(0x7fcf000b32e0 0x7fcf000b6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.354+0000 7fcef5400640 1 -- 172.21.15.40:0/3460001932 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf000cefa0 con 0x7fcf000b7460 2024-03-24T00:36:46.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcef5400640 1 --2- 172.21.15.40:0/3460001932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf000b7460 0x7fcf000d2d70 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fcf000b69b0 tx=0x7fcee801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514b44a103e4241 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:46.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcefd400640 1 -- 172.21.15.40:0/3460001932 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee80b0070 con 0x7fcf000b7460 2024-03-24T00:36:46.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcefd400640 1 -- 172.21.15.40:0/3460001932 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee80ab600 con 0x7fcf000b7460 2024-03-24T00:36:46.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcefd400640 1 -- 172.21.15.40:0/3460001932 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee80ab9d0 con 0x7fcf000b7460 2024-03-24T00:36:46.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcf05000640 1 -- 172.21.15.40:0/3460001932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf000b7460 msgr2=0x7fcf000d2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcf05000640 1 --2- 172.21.15.40:0/3460001932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf000b7460 0x7fcf000d2d70 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fcf000b69b0 tx=0x7fcee801c520 comp rx=0 tx=0).stop 2024-03-24T00:36:46.362 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcf05000640 1 -- 172.21.15.40:0/3460001932 shutdown_connections 2024-03-24T00:36:46.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcf05000640 1 --2- 172.21.15.40:0/3460001932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf000b97a0 0x7fcf000d32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcf05000640 1 --2- 172.21.15.40:0/3460001932 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf000b7460 0x7fcf000d2d70 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcf05000640 1 --2- 172.21.15.40:0/3460001932 >> v2:172.21.15.53:3300/0 conn(0x7fcf000b32e0 0x7fcf000b6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcf05000640 1 -- 172.21.15.40:0/3460001932 >> 172.21.15.40:0/3460001932 conn(0x7fcf000d3df0 msgr2=0x7fcf000d4200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:46.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcf05000640 1 -- 172.21.15.40:0/3460001932 shutdown_connections 2024-03-24T00:36:46.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.358+0000 7fcf05000640 1 -- 172.21.15.40:0/3460001932 wait complete. 2024-03-24T00:36:46.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcf05000640 1 Processor -- start 2024-03-24T00:36:46.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcf05000640 1 -- start start 2024-03-24T00:36:46.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcf05000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf000b7460 0x7fcf001556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcf05000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf000b97a0 0x7fcf00154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcf05000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf000cbab0 0x7fcf00154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcf05000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf00072100 con 0x7fcf000cbab0 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcf05000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf00071f80 con 0x7fcf000b7460 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcf05000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf00072280 con 0x7fcf000b97a0 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcefe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf000cbab0 0x7fcf00154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcefde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf000b7460 0x7fcf001556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcefe800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf000cbab0 0x7fcf00154750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40274/0 (socket says 172.21.15.40:40274) 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcefe800640 1 -- 172.21.15.40:0/1196220512 learned_addr learned my addr 172.21.15.40:0/1196220512 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcef5400640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf000b97a0 0x7fcf00154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcefe800640 1 -- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf000b97a0 msgr2=0x7fcf00154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcefe800640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf000b97a0 0x7fcf00154210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcefe800640 1 -- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf000b7460 msgr2=0x7fcf001556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcefe800640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf000b7460 0x7fcf001556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.362+0000 7fcefe800640 1 -- 172.21.15.40:0/1196220512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf000cbf70 con 0x7fcf000cbab0 2024-03-24T00:36:46.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.366+0000 7fcefe800640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf000cbab0 0x7fcf00154750 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fcef800efc0 tx=0x7fcef800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:46.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.366+0000 7fcef6a00640 1 -- 172.21.15.40:0/1196220512 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef8018020 con 0x7fcf000cbab0 2024-03-24T00:36:46.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.366+0000 7fcef6a00640 1 -- 172.21.15.40:0/1196220512 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef8014480 con 0x7fcf000cbab0 2024-03-24T00:36:46.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.366+0000 7fcf05000640 1 -- 172.21.15.40:0/1196220512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf000cc250 con 0x7fcf000cbab0 2024-03-24T00:36:46.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.366+0000 7fcef6a00640 1 -- 172.21.15.40:0/1196220512 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef8014870 con 0x7fcf000cbab0 2024-03-24T00:36:46.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.366+0000 7fcf05000640 1 -- 172.21.15.40:0/1196220512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf00155bf0 con 0x7fcf000cbab0 2024-03-24T00:36:46.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.366+0000 7fcf05000640 1 -- 172.21.15.40:0/1196220512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf00068170 con 0x7fcf000cbab0 2024-03-24T00:36:46.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.366+0000 7fcef6a00640 1 -- 172.21.15.40:0/1196220512 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcef801b020 con 0x7fcf000cbab0 2024-03-24T00:36:46.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.366+0000 7fcef6a00640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcedc067660 0x7fcedc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.366+0000 7fcef6a00640 1 -- 172.21.15.40:0/1196220512 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcef8086290 con 0x7fcf000cbab0 2024-03-24T00:36:46.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.370+0000 7fcefde00640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcedc067660 0x7fcedc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.370+0000 7fcefde00640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcedc067660 0x7fcedc069b20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcef0009ae0 tx=0x7fcef0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:46.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.370+0000 7fcef6a00640 1 -- 172.21.15.40:0/1196220512 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcef8022070 con 0x7fcf000cbab0 2024-03-24T00:36:46.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.534+0000 7fcf05000640 1 -- 172.21.15.40:0/1196220512 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcf000d0590 con 0x7fcf000cbab0 2024-03-24T00:36:46.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.534+0000 7fcef6a00640 1 -- 172.21.15.40:0/1196220512 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcef8010040 con 0x7fcf000cbab0 2024-03-24T00:36:46.539 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:36:46.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.538+0000 7fcf05000640 1 -- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcedc067660 msgr2=0x7fcedc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.538+0000 7fcf05000640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcedc067660 0x7fcedc069b20 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fcef0009ae0 tx=0x7fcef0009190 comp rx=0 tx=0).stop 2024-03-24T00:36:46.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.538+0000 7fcf05000640 1 -- 172.21.15.40:0/1196220512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf000cbab0 msgr2=0x7fcf00154750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.538+0000 7fcf05000640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf000cbab0 0x7fcf00154750 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fcef800efc0 tx=0x7fcef800c490 comp rx=0 tx=0).stop 2024-03-24T00:36:46.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.538+0000 7fcf05000640 1 -- 172.21.15.40:0/1196220512 shutdown_connections 2024-03-24T00:36:46.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.538+0000 7fcf05000640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcedc067660 0x7fcedc069b20 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.538+0000 7fcf05000640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcf000cbab0 0x7fcf00154750 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.538+0000 7fcf05000640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf000b97a0 0x7fcf00154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.538+0000 7fcf05000640 1 --2- 172.21.15.40:0/1196220512 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf000b7460 0x7fcf001556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.538+0000 7fcf05000640 1 -- 172.21.15.40:0/1196220512 >> 172.21.15.40:0/1196220512 conn(0x7fcf000d3df0 msgr2=0x7fcf001622d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:46.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.538+0000 7fcf05000640 1 -- 172.21.15.40:0/1196220512 shutdown_connections 2024-03-24T00:36:46.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.538+0000 7fcf05000640 1 -- 172.21.15.40:0/1196220512 wait complete. 2024-03-24T00:36:46.559 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:36:46.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d4600640 1 Processor -- start 2024-03-24T00:36:46.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d4600640 1 -- start start 2024-03-24T00:36:46.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d4600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f64cc15b960 0x7f64cc070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d4600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f64cc0d16f0 0x7f64cc0712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d4600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64cc15f970 con 0x7f64cc15bd40 2024-03-24T00:36:46.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d4600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64cc160580 con 0x7f64cc0d16f0 2024-03-24T00:36:46.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d4600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64cc15fc50 con 0x7f64cc15b960 2024-03-24T00:36:46.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d2600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f64cc0d16f0 0x7f64cc0712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d1200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f64cc15bd40 0x7f64cc0d0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44636/0 (socket says 172.21.15.40:44636) 2024-03-24T00:36:46.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d1200640 1 -- 172.21.15.40:0/3882653899 learned_addr learned my addr 172.21.15.40:0/3882653899 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:46.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d1c00640 1 --2- 172.21.15.40:0/3882653899 >> v2:172.21.15.53:3300/0 conn(0x7f64cc15b960 0x7f64cc070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64cbe00640 1 -- 172.21.15.40:0/3882653899 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898583875 0 0) 0x7f64cc15f970 con 0x7f64cc15bd40 2024-03-24T00:36:46.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64cbe00640 1 -- 172.21.15.40:0/3882653899 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b4003530 con 0x7f64cc15bd40 2024-03-24T00:36:46.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d2600640 1 -- 172.21.15.40:0/3882653899 >> v2:172.21.15.53:3300/0 conn(0x7f64cc15b960 msgr2=0x7f64cc070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:46.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d2600640 1 --2- 172.21.15.40:0/3882653899 >> v2:172.21.15.53:3300/0 conn(0x7f64cc15b960 0x7f64cc070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d2600640 1 -- 172.21.15.40:0/3882653899 >> v1:172.21.15.40:6789/0 conn(0x7f64cc15bd40 legacy=0x7f64cc0d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d2600640 1 -- 172.21.15.40:0/3882653899 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc15ff30 con 0x7f64cc0d16f0 2024-03-24T00:36:46.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.702+0000 7f64d1c00640 1 --2- 172.21.15.40:0/3882653899 >> v2:172.21.15.53:3300/0 conn(0x7f64cc15b960 0x7f64cc070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:46.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.706+0000 7f64d2600640 1 --2- 172.21.15.40:0/3882653899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64cc0d16f0 0x7f64cc0712a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f64cc0d1b30 tx=0x7f64c401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40fb75a3761679d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:46.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.706+0000 7f64cbe00640 1 -- 172.21.15.40:0/3882653899 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c40b2070 con 0x7f64cc0d16f0 2024-03-24T00:36:46.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.706+0000 7f64cbe00640 1 -- 172.21.15.40:0/3882653899 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c40a5900 con 0x7f64cc0d16f0 2024-03-24T00:36:46.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.706+0000 7f64cbe00640 1 -- 172.21.15.40:0/3882653899 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c40a5cd0 con 0x7f64cc0d16f0 2024-03-24T00:36:46.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.706+0000 7f64d4600640 1 -- 172.21.15.40:0/3882653899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64cc0d16f0 msgr2=0x7f64cc0712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.706+0000 7f64d4600640 1 --2- 172.21.15.40:0/3882653899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64cc0d16f0 0x7f64cc0712a0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f64cc0d1b30 tx=0x7f64c401c930 comp rx=0 tx=0).stop 2024-03-24T00:36:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 -- 172.21.15.40:0/3882653899 shutdown_connections 2024-03-24T00:36:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 --2- 172.21.15.40:0/3882653899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64cc0d16f0 0x7f64cc0712a0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 --2- 172.21.15.40:0/3882653899 >> v2:172.21.15.53:3300/0 conn(0x7f64cc15b960 0x7f64cc070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 -- 172.21.15.40:0/3882653899 >> 172.21.15.40:0/3882653899 conn(0x7f64cc06ecf0 msgr2=0x7f64cc06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 -- 172.21.15.40:0/3882653899 shutdown_connections 2024-03-24T00:36:46.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 -- 172.21.15.40:0/3882653899 wait complete. 2024-03-24T00:36:46.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 Processor -- start 2024-03-24T00:36:46.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 -- start start 2024-03-24T00:36:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64cc0d16f0 0x7f64cc156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64cc15b960 0x7f64cc153ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64cc15bd40 0x7f64cc15a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0ce600 con 0x7f64cc15b960 2024-03-24T00:36:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0ce480 con 0x7f64cc0d16f0 2024-03-24T00:36:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d4600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0ce780 con 0x7f64cc15bd40 2024-03-24T00:36:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64cc15b960 0x7f64cc153ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d1c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64cc0d16f0 0x7f64cc156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d1200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64cc15b960 0x7f64cc153ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40296/0 (socket says 172.21.15.40:40296) 2024-03-24T00:36:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d1200640 1 -- 172.21.15.40:0/126446066 learned_addr learned my addr 172.21.15.40:0/126446066 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:46.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d2600640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64cc15bd40 0x7f64cc15a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d1200640 1 -- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64cc15bd40 msgr2=0x7f64cc15a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d1200640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64cc15bd40 0x7f64cc15a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d1200640 1 -- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64cc0d16f0 msgr2=0x7f64cc156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d1200640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64cc0d16f0 0x7f64cc156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d1200640 1 -- 172.21.15.40:0/126446066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc155cc0 con 0x7f64cc15b960 2024-03-24T00:36:46.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.710+0000 7f64d1c00640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64cc0d16f0 0x7f64cc156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:46.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.714+0000 7f64d1200640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64cc15b960 0x7f64cc153ea0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f64b800c3f0 tx=0x7f64b800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:46.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.714+0000 7f64c9600640 1 -- 172.21.15.40:0/126446066 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8017070 con 0x7f64cc15b960 2024-03-24T00:36:46.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.714+0000 7f64c9600640 1 -- 172.21.15.40:0/126446066 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b8005cf0 con 0x7f64cc15b960 2024-03-24T00:36:46.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.714+0000 7f64d4600640 1 -- 172.21.15.40:0/126446066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc154920 con 0x7f64cc15b960 2024-03-24T00:36:46.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.714+0000 7f64c9600640 1 -- 172.21.15.40:0/126446066 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b80072f0 con 0x7f64cc15b960 2024-03-24T00:36:46.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.714+0000 7f64d4600640 1 -- 172.21.15.40:0/126446066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64cc156210 con 0x7f64cc15b960 2024-03-24T00:36:46.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.714+0000 7f64d4600640 1 -- 172.21.15.40:0/126446066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64cc068170 con 0x7f64cc15b960 2024-03-24T00:36:46.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.714+0000 7f64c9600640 1 -- 172.21.15.40:0/126446066 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f64b801a070 con 0x7f64cc15b960 2024-03-24T00:36:46.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.718+0000 7f64c9600640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f64ac067660 0x7f64ac069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:46.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.718+0000 7f64c9600640 1 -- 172.21.15.40:0/126446066 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64b8084cb0 con 0x7f64cc15b960 2024-03-24T00:36:46.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.718+0000 7f64d1c00640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f64ac067660 0x7f64ac069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:46.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.718+0000 7f64d1c00640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f64ac067660 0x7f64ac069b20 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f64c0009580 tx=0x7f64c0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:46.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.722+0000 7f64c9600640 1 -- 172.21.15.40:0/126446066 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f64b80552c0 con 0x7f64cc15b960 2024-03-24T00:36:46.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.886+0000 7f64d4600640 1 -- 172.21.15.40:0/126446066 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f64cc0b1e30 con 0x7f64cc15b960 2024-03-24T00:36:46.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.886+0000 7f64c9600640 1 -- 172.21.15.40:0/126446066 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f64b8059360 con 0x7f64cc15b960 2024-03-24T00:36:46.890 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:36:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.886+0000 7f64d4600640 1 -- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f64ac067660 msgr2=0x7f64ac069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.886+0000 7f64d4600640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f64ac067660 0x7f64ac069b20 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f64c0009580 tx=0x7f64c0009190 comp rx=0 tx=0).stop 2024-03-24T00:36:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.886+0000 7f64d4600640 1 -- 172.21.15.40:0/126446066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64cc15b960 msgr2=0x7f64cc153ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.886+0000 7f64d4600640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64cc15b960 0x7f64cc153ea0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f64b800c3f0 tx=0x7f64b800c8b0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.890+0000 7f64d4600640 1 -- 172.21.15.40:0/126446066 shutdown_connections 2024-03-24T00:36:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.890+0000 7f64d4600640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f64ac067660 0x7f64ac069b20 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.890+0000 7f64d4600640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64cc15bd40 0x7f64cc15a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.890+0000 7f64d4600640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f64cc15b960 0x7f64cc153ea0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.890+0000 7f64d4600640 1 --2- 172.21.15.40:0/126446066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64cc0d16f0 0x7f64cc156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.890+0000 7f64d4600640 1 -- 172.21.15.40:0/126446066 >> 172.21.15.40:0/126446066 conn(0x7f64cc06ecf0 msgr2=0x7f64cc166300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.890+0000 7f64d4600640 1 -- 172.21.15.40:0/126446066 shutdown_connections 2024-03-24T00:36:46.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:46.890+0000 7f64d4600640 1 -- 172.21.15.40:0/126446066 wait complete. 2024-03-24T00:36:46.911 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:36:47.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.046+0000 7f5255a00640 1 Processor -- start 2024-03-24T00:36:47.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.046+0000 7f5255a00640 1 -- start start 2024-03-24T00:36:47.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f5255a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52500d3a90 0x7f52500ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f5255a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52500d10f0 0x7f52500cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f5255a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52500d29f0 con 0x7f525015e020 2024-03-24T00:36:47.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f5255a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52500b7ff0 con 0x7f52500d10f0 2024-03-24T00:36:47.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f5255a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52500d3600 con 0x7f52500d3a90 2024-03-24T00:36:47.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f524f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52500d10f0 0x7f52500cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f524ea00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f525015e020 0x7f52500d2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57200/0 (socket says 172.21.15.40:57200) 2024-03-24T00:36:47.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f524ea00640 1 -- 172.21.15.40:0/2997147137 learned_addr learned my addr 172.21.15.40:0/2997147137 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:47.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f5247e00640 1 --2- 172.21.15.40:0/2997147137 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52500d3a90 0x7f52500ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f524f400640 1 -- 172.21.15.40:0/2997147137 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52500d3a90 msgr2=0x7f52500ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:47.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f524f400640 1 --2- 172.21.15.40:0/2997147137 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52500d3a90 0x7f52500ceb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f524f400640 1 -- 172.21.15.40:0/2997147137 >> v1:172.21.15.40:6789/0 conn(0x7f525015e020 legacy=0x7f52500d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f524f400640 1 -- 172.21.15.40:0/2997147137 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52500d2cd0 con 0x7f52500d10f0 2024-03-24T00:36:47.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f524f400640 1 --2- 172.21.15.40:0/2997147137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52500d10f0 0x7f52500cf0c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f5248009d90 tx=0x7f524801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddbdee1cc9c31f53 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:47.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.050+0000 7f524e000640 1 -- 172.21.15.40:0/2997147137 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52480b2070 con 0x7f52500d10f0 2024-03-24T00:36:47.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.054+0000 7f524e000640 1 -- 172.21.15.40:0/2997147137 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52480a5a20 con 0x7f52500d10f0 2024-03-24T00:36:47.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.054+0000 7f524e000640 1 -- 172.21.15.40:0/2997147137 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52480a5df0 con 0x7f52500d10f0 2024-03-24T00:36:47.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.054+0000 7f5255a00640 1 -- 172.21.15.40:0/2997147137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52500d10f0 msgr2=0x7f52500cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.054+0000 7f5255a00640 1 --2- 172.21.15.40:0/2997147137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52500d10f0 0x7f52500cf0c0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f5248009d90 tx=0x7f524801c940 comp rx=0 tx=0).stop 2024-03-24T00:36:47.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.054+0000 7f5255a00640 1 -- 172.21.15.40:0/2997147137 shutdown_connections 2024-03-24T00:36:47.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.054+0000 7f5255a00640 1 --2- 172.21.15.40:0/2997147137 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52500d10f0 0x7f52500cf0c0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.054+0000 7f5255a00640 1 --2- 172.21.15.40:0/2997147137 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52500d3a90 0x7f52500ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.054+0000 7f5255a00640 1 -- 172.21.15.40:0/2997147137 >> 172.21.15.40:0/2997147137 conn(0x7f5250066e80 msgr2=0x7f5250067290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:47.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.054+0000 7f5255a00640 1 -- 172.21.15.40:0/2997147137 shutdown_connections 2024-03-24T00:36:47.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.054+0000 7f5255a00640 1 -- 172.21.15.40:0/2997147137 wait complete. 2024-03-24T00:36:47.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.054+0000 7f5255a00640 1 Processor -- start 2024-03-24T00:36:47.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.054+0000 7f5255a00640 1 -- start start 2024-03-24T00:36:47.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5255a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52500d10f0 0x7f5250158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5255a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52500d3a90 0x7f5250154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5255a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f525015e020 0x7f525015e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5255a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52500b7970 con 0x7f52500d3a90 2024-03-24T00:36:47.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5255a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52500b77f0 con 0x7f52500d10f0 2024-03-24T00:36:47.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5255a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52500b7af0 con 0x7f525015e020 2024-03-24T00:36:47.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5247e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52500d3a90 0x7f5250154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5247e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52500d3a90 0x7f5250154e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36604/0 (socket says 172.21.15.40:36604) 2024-03-24T00:36:47.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5247e00640 1 -- 172.21.15.40:0/3648532972 learned_addr learned my addr 172.21.15.40:0/3648532972 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:47.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f524ea00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52500d10f0 0x7f5250158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f524f400640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f525015e020 0x7f525015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5247e00640 1 -- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f525015e020 msgr2=0x7f525015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5247e00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f525015e020 0x7f525015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5247e00640 1 -- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52500d10f0 msgr2=0x7f5250158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5247e00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52500d10f0 0x7f5250158b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5247e00640 1 -- 172.21.15.40:0/3648532972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52501590b0 con 0x7f52500d3a90 2024-03-24T00:36:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5247e00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52500d3a90 0x7f5250154e50 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f523800e930 tx=0x7f523800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5246a00640 1 -- 172.21.15.40:0/3648532972 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5238017070 con 0x7f52500d3a90 2024-03-24T00:36:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5246a00640 1 -- 172.21.15.40:0/3648532972 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523800ca00 con 0x7f52500d3a90 2024-03-24T00:36:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5246a00640 1 -- 172.21.15.40:0/3648532972 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523800cdb0 con 0x7f52500d3a90 2024-03-24T00:36:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5255a00640 1 -- 172.21.15.40:0/3648532972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5250156100 con 0x7f52500d3a90 2024-03-24T00:36:47.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5255a00640 1 -- 172.21.15.40:0/3648532972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52501566c0 con 0x7f52500d3a90 2024-03-24T00:36:47.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.058+0000 7f5255a00640 1 -- 172.21.15.40:0/3648532972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5250068170 con 0x7f52500d3a90 2024-03-24T00:36:47.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.062+0000 7f5246a00640 1 -- 172.21.15.40:0/3648532972 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f523801a070 con 0x7f52500d3a90 2024-03-24T00:36:47.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.062+0000 7f5246a00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f522c067660 0x7f522c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.062+0000 7f5246a00640 1 -- 172.21.15.40:0/3648532972 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5238085980 con 0x7f52500d3a90 2024-03-24T00:36:47.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.062+0000 7f524ea00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f522c067660 0x7f522c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.062+0000 7f524ea00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f522c067660 0x7f522c069b20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5240000f70 tx=0x7f5240007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:47.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.066+0000 7f5246a00640 1 -- 172.21.15.40:0/3648532972 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5238055f40 con 0x7f52500d3a90 2024-03-24T00:36:47.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 -- 172.21.15.40:0/3648532972 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52500c7a80 con 0x7f52500d3a90 2024-03-24T00:36:47.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5246a00640 1 -- 172.21.15.40:0/3648532972 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5015 (secure 0 0 0) 0x7f5238059fe0 con 0x7f52500d3a90 2024-03-24T00:36:47.265 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:47.265 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":88,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1097448819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1097448819},{"type":"v1","addr":"172.21.15.40:6834","nonce":1097448819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:36:07.163876+0000","modified":"2024-03-24T00:36:30.866122+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7186,"mds_1":7180,"mds_2":6404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6404":{"gid":6404,"name":"b","rank":2,"incarnation":86,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/983819364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":983819364},{"type":"v1","addr":"172.21.15.53:6837","nonce":983819364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7180":{"gid":7180,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6835/2302548193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2302548193},{"type":"v1","addr":"172.21.15.40:6835","nonce":2302548193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7186":{"gid":7186,"name":"d","rank":0,"incarnation":78,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/1878888046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1878888046},{"type":"v1","addr":"172.21.15.53:6835","nonce":1878888046}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7186,"qdb_cluster":[7186,7180,6404]},"id":6}]} 2024-03-24T00:36:47.265 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 88 2024-03-24T00:36:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 -- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f522c067660 msgr2=0x7f522c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f522c067660 0x7f522c069b20 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5240000f70 tx=0x7f5240007e90 comp rx=0 tx=0).stop 2024-03-24T00:36:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 -- 172.21.15.40:0/3648532972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52500d3a90 msgr2=0x7f5250154e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52500d3a90 0x7f5250154e50 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f523800e930 tx=0x7f523800edf0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 -- 172.21.15.40:0/3648532972 shutdown_connections 2024-03-24T00:36:47.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f522c067660 0x7f522c069b20 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f525015e020 0x7f525015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52500d3a90 0x7f5250154e50 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 --2- 172.21.15.40:0/3648532972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52500d10f0 0x7f5250158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 -- 172.21.15.40:0/3648532972 >> 172.21.15.40:0/3648532972 conn(0x7f5250066e80 msgr2=0x7f52500c8480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:47.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 -- 172.21.15.40:0/3648532972 shutdown_connections 2024-03-24T00:36:47.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.262+0000 7f5255a00640 1 -- 172.21.15.40:0/3648532972 wait complete. 2024-03-24T00:36:47.287 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:36:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b8c00640 1 Processor -- start 2024-03-24T00:36:47.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b8c00640 1 -- start start 2024-03-24T00:36:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b8c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f60b413ec80 0x7f60b41414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b8c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b4070e00 con 0x7f60b40ce800 2024-03-24T00:36:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b8c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b40cee20 con 0x7f60b41419f0 2024-03-24T00:36:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b8c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b4141e50 con 0x7f60b413ec80 2024-03-24T00:36:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b2800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f60b40ce800 0x7f60b4071300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57208/0 (socket says 172.21.15.40:57208) 2024-03-24T00:36:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b2800640 1 -- 172.21.15.40:0/712512538 learned_addr learned my addr 172.21.15.40:0/712512538 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:47.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b1e00640 1 --2- 172.21.15.40:0/712512538 >> v2:172.21.15.53:3300/0 conn(0x7f60b413ec80 0x7f60b41414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b1400640 1 -- 172.21.15.40:0/712512538 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474086453 0 0) 0x7f60b40cee20 con 0x7f60b41419f0 2024-03-24T00:36:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b1400640 1 -- 172.21.15.40:0/712512538 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6098003530 con 0x7f60b41419f0 2024-03-24T00:36:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b1400640 1 -- 172.21.15.40:0/712512538 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745892437 0 0) 0x7f60b4070e00 con 0x7f60b40ce800 2024-03-24T00:36:47.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b1400640 1 -- 172.21.15.40:0/712512538 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60b40cee20 con 0x7f60b40ce800 2024-03-24T00:36:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b1e00640 1 -- 172.21.15.40:0/712512538 >> v1:172.21.15.53:6789/0 conn(0x7f60b41419f0 legacy=0x7f60b40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b1e00640 1 -- 172.21.15.40:0/712512538 >> v1:172.21.15.40:6789/0 conn(0x7f60b40ce800 legacy=0x7f60b4071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.426+0000 7f60b1e00640 1 -- 172.21.15.40:0/712512538 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b40cf100 con 0x7f60b413ec80 2024-03-24T00:36:47.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.430+0000 7f60b1e00640 1 --2- 172.21.15.40:0/712512538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b413ec80 0x7f60b41414b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f60a40097c0 tx=0x7f60a401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5048edcb06c4f6a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:47.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.430+0000 7f60b1400640 1 -- 172.21.15.40:0/712512538 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a40aa620 con 0x7f60b413ec80 2024-03-24T00:36:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.430+0000 7f60b1400640 1 -- 172.21.15.40:0/712512538 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a40af070 con 0x7f60b413ec80 2024-03-24T00:36:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.430+0000 7f60b1400640 1 -- 172.21.15.40:0/712512538 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a40aa9e0 con 0x7f60b413ec80 2024-03-24T00:36:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.430+0000 7f60b8c00640 1 -- 172.21.15.40:0/712512538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b413ec80 msgr2=0x7f60b41414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.430+0000 7f60b8c00640 1 --2- 172.21.15.40:0/712512538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b413ec80 0x7f60b41414b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f60a40097c0 tx=0x7f60a401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.430+0000 7f60b8c00640 1 -- 172.21.15.40:0/712512538 shutdown_connections 2024-03-24T00:36:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.430+0000 7f60b8c00640 1 --2- 172.21.15.40:0/712512538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b413ec80 0x7f60b41414b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.430+0000 7f60b8c00640 1 -- 172.21.15.40:0/712512538 >> 172.21.15.40:0/712512538 conn(0x7f60b40b17b0 msgr2=0x7f60b40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.430+0000 7f60b8c00640 1 -- 172.21.15.40:0/712512538 shutdown_connections 2024-03-24T00:36:47.435 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.430+0000 7f60b8c00640 1 -- 172.21.15.40:0/712512538 wait complete. 2024-03-24T00:36:47.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b8c00640 1 Processor -- start 2024-03-24T00:36:47.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b8c00640 1 -- start start 2024-03-24T00:36:47.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b40ce800 0x7f60b413bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b8c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60b413ec80 0x7f60b41fa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b8c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b41419f0 0x7f60b41fa710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b8c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b40725f0 con 0x7f60b413ec80 2024-03-24T00:36:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b8c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4072470 con 0x7f60b40ce800 2024-03-24T00:36:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b8c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b4169d50 con 0x7f60b41419f0 2024-03-24T00:36:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60a9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60b413ec80 0x7f60b41fa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60a9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60b413ec80 0x7f60b41fa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36618/0 (socket says 172.21.15.40:36618) 2024-03-24T00:36:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60a9400640 1 -- 172.21.15.40:0/615553545 learned_addr learned my addr 172.21.15.40:0/615553545 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b1e00640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b40ce800 0x7f60b413bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b2800640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b41419f0 0x7f60b41fa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60a9400640 1 -- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b41419f0 msgr2=0x7f60b41fa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60a9400640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b41419f0 0x7f60b41fa710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60a9400640 1 -- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b40ce800 msgr2=0x7f60b413bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60a9400640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b40ce800 0x7f60b413bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60a9400640 1 -- 172.21.15.40:0/615553545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b4075470 con 0x7f60b413ec80 2024-03-24T00:36:47.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b2800640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b41419f0 0x7f60b41fa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:47.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60a9400640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60b413ec80 0x7f60b41fa1d0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f609c0010f0 tx=0x7f609c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:47.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60aaa00640 1 -- 172.21.15.40:0/615553545 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c017070 con 0x7f60b413ec80 2024-03-24T00:36:47.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60aaa00640 1 -- 172.21.15.40:0/615553545 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c0052d0 con 0x7f60b413ec80 2024-03-24T00:36:47.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b8c00640 1 -- 172.21.15.40:0/615553545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b41596c0 con 0x7f60b413ec80 2024-03-24T00:36:47.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60aaa00640 1 -- 172.21.15.40:0/615553545 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c012830 con 0x7f60b413ec80 2024-03-24T00:36:47.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.434+0000 7f60b8c00640 1 -- 172.21.15.40:0/615553545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b413ca80 con 0x7f60b413ec80 2024-03-24T00:36:47.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.438+0000 7f60b8c00640 1 -- 172.21.15.40:0/615553545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b4068170 con 0x7f60b413ec80 2024-03-24T00:36:47.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.438+0000 7f60aaa00640 1 -- 172.21.15.40:0/615553545 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f609c01a070 con 0x7f60b413ec80 2024-03-24T00:36:47.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.438+0000 7f60aaa00640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6090067660 0x7f6090069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.438+0000 7f60aaa00640 1 -- 172.21.15.40:0/615553545 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f609c084b80 con 0x7f60b413ec80 2024-03-24T00:36:47.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.438+0000 7f60b1e00640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6090067660 0x7f6090069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.438+0000 7f60b1e00640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6090067660 0x7f6090069b20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f60a40ae000 tx=0x7f60a40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:47.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.442+0000 7f60aaa00640 1 -- 172.21.15.40:0/615553545 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f609c005480 con 0x7f60b413ec80 2024-03-24T00:36:47.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 -- 172.21.15.40:0/615553545 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60b415f0a0 con 0x7f60b413ec80 2024-03-24T00:36:47.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60aaa00640 1 -- 172.21.15.40:0/615553545 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16600 (secure 0 0 0) 0x7f609c055110 con 0x7f60b413ec80 2024-03-24T00:36:47.613 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:47.613 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":83,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:36:08.785471+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:05.573740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-24T00:36:06.150226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:36:47.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 -- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6090067660 msgr2=0x7f6090069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6090067660 0x7f6090069b20 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f60a40ae000 tx=0x7f60a40a7000 comp rx=0 tx=0).stop 2024-03-24T00:36:47.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 -- 172.21.15.40:0/615553545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60b413ec80 msgr2=0x7f60b41fa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60b413ec80 0x7f60b41fa1d0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f609c0010f0 tx=0x7f609c00c960 comp rx=0 tx=0).stop 2024-03-24T00:36:47.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 -- 172.21.15.40:0/615553545 shutdown_connections 2024-03-24T00:36:47.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6090067660 0x7f6090069b20 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b41419f0 0x7f60b41fa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60b413ec80 0x7f60b41fa1d0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 --2- 172.21.15.40:0/615553545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b40ce800 0x7f60b413bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 -- 172.21.15.40:0/615553545 >> 172.21.15.40:0/615553545 conn(0x7f60b40b17b0 msgr2=0x7f60b4150860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:47.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 -- 172.21.15.40:0/615553545 shutdown_connections 2024-03-24T00:36:47.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.610+0000 7f60b8c00640 1 -- 172.21.15.40:0/615553545 wait complete. 2024-03-24T00:36:47.634 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:36:47.638 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:36:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.774+0000 7f5944600640 1 Processor -- start 2024-03-24T00:36:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.774+0000 7f5944600640 1 -- start start 2024-03-24T00:36:47.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.778+0000 7f5944600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f593c0cfdf0 0x7f593c0d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.778+0000 7f5944600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593c0af710 con 0x7f593c15f7a0 2024-03-24T00:36:47.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.778+0000 7f5944600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f593c0b34f0 con 0x7f593c0b43a0 2024-03-24T00:36:47.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.778+0000 7f5944600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f593c0b2bb0 con 0x7f593c0cfdf0 2024-03-24T00:36:47.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.778+0000 7f5942600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f593c15f7a0 0x7f593c0afbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57216/0 (socket says 172.21.15.40:57216) 2024-03-24T00:36:47.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.778+0000 7f5942600640 1 -- 172.21.15.40:0/2506453753 learned_addr learned my addr 172.21.15.40:0/2506453753 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:47.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.778+0000 7f5941c00640 1 --2- 172.21.15.40:0/2506453753 >> v2:172.21.15.40:3300/0 conn(0x7f593c0cfdf0 0x7f593c0d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.778+0000 7f5941c00640 1 -- 172.21.15.40:0/2506453753 >> v1:172.21.15.53:6789/0 conn(0x7f593c0b43a0 legacy=0x7f593c15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.778+0000 7f5941c00640 1 -- 172.21.15.40:0/2506453753 >> v1:172.21.15.40:6789/0 conn(0x7f593c15f7a0 legacy=0x7f593c0afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.778+0000 7f593be00640 1 -- 172.21.15.40:0/2506453753 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058079371 0 0) 0x7f593c0b34f0 con 0x7f593c0b43a0 2024-03-24T00:36:47.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.778+0000 7f5941c00640 1 -- 172.21.15.40:0/2506453753 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c0b28d0 con 0x7f593c0cfdf0 2024-03-24T00:36:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.782+0000 7f5941c00640 1 --2- 172.21.15.40:0/2506453753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f593c0cfdf0 0x7f593c0d01d0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f59300097c0 tx=0x7f593001c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44466225c092dbf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.782+0000 7f593be00640 1 -- 172.21.15.40:0/2506453753 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300b0070 con 0x7f593c0cfdf0 2024-03-24T00:36:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.782+0000 7f593be00640 1 -- 172.21.15.40:0/2506453753 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59300ab600 con 0x7f593c0cfdf0 2024-03-24T00:36:47.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.782+0000 7f593be00640 1 -- 172.21.15.40:0/2506453753 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300ab9d0 con 0x7f593c0cfdf0 2024-03-24T00:36:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.782+0000 7f5944600640 1 -- 172.21.15.40:0/2506453753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f593c0cfdf0 msgr2=0x7f593c0d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.782+0000 7f5944600640 1 --2- 172.21.15.40:0/2506453753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f593c0cfdf0 0x7f593c0d01d0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f59300097c0 tx=0x7f593001c330 comp rx=0 tx=0).stop 2024-03-24T00:36:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.782+0000 7f5944600640 1 -- 172.21.15.40:0/2506453753 shutdown_connections 2024-03-24T00:36:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.782+0000 7f5944600640 1 --2- 172.21.15.40:0/2506453753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f593c0cfdf0 0x7f593c0d01d0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.782+0000 7f5944600640 1 -- 172.21.15.40:0/2506453753 >> 172.21.15.40:0/2506453753 conn(0x7f593c0b3880 msgr2=0x7f593c0b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.782+0000 7f5944600640 1 -- 172.21.15.40:0/2506453753 shutdown_connections 2024-03-24T00:36:47.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.782+0000 7f5944600640 1 -- 172.21.15.40:0/2506453753 wait complete. 2024-03-24T00:36:47.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5944600640 1 Processor -- start 2024-03-24T00:36:47.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5944600640 1 -- start start 2024-03-24T00:36:47.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5944600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f593c0b43a0 0x7f593c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5944600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f593c0cfdf0 0x7f593c15cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5944600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f593c15f7a0 0x7f593c15d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5944600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0d0ae0 con 0x7f593c0b43a0 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5944600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593c0d0960 con 0x7f593c0cfdf0 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5944600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593c1509c0 con 0x7f593c15f7a0 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5941c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f593c0b43a0 0x7f593c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5941c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f593c0b43a0 0x7f593c0cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36626/0 (socket says 172.21.15.40:36626) 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5941c00640 1 -- 172.21.15.40:0/576460637 learned_addr learned my addr 172.21.15.40:0/576460637 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5941200640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f593c0cfdf0 0x7f593c15cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5942600640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f593c15f7a0 0x7f593c15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5941c00640 1 -- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f593c15f7a0 msgr2=0x7f593c15d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5941c00640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f593c15f7a0 0x7f593c15d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5941c00640 1 -- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f593c0cfdf0 msgr2=0x7f593c15cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5941c00640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f593c0cfdf0 0x7f593c15cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:47.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5941c00640 1 -- 172.21.15.40:0/576460637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c0b68f0 con 0x7f593c0b43a0 2024-03-24T00:36:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5941c00640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f593c0b43a0 0x7f593c0cb6f0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f59300097c0 tx=0x7f593001ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5939600640 1 -- 172.21.15.40:0/576460637 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300b0050 con 0x7f593c0b43a0 2024-03-24T00:36:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5939600640 1 -- 172.21.15.40:0/576460637 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59300bc070 con 0x7f593c0b43a0 2024-03-24T00:36:47.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5939600640 1 -- 172.21.15.40:0/576460637 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59300aae40 con 0x7f593c0b43a0 2024-03-24T00:36:47.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5944600640 1 -- 172.21.15.40:0/576460637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593c0b6bd0 con 0x7f593c0b43a0 2024-03-24T00:36:47.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.786+0000 7f5944600640 1 -- 172.21.15.40:0/576460637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f593c0b76b0 con 0x7f593c0b43a0 2024-03-24T00:36:47.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.790+0000 7f5944600640 1 -- 172.21.15.40:0/576460637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f593c068170 con 0x7f593c0b43a0 2024-03-24T00:36:47.794 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.790+0000 7f5939600640 1 -- 172.21.15.40:0/576460637 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f59300ab420 con 0x7f593c0b43a0 2024-03-24T00:36:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.794+0000 7f5939600640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f591c067660 0x7f591c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.794+0000 7f5939600640 1 -- 172.21.15.40:0/576460637 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f59301204d0 con 0x7f593c0b43a0 2024-03-24T00:36:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.794+0000 7f5941200640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f591c067660 0x7f591c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:47.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.794+0000 7f5941200640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f591c067660 0x7f591c069b20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f593c15b9d0 tx=0x7f5928009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:47.800 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.798+0000 7f5939600640 1 -- 172.21.15.40:0/576460637 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5930015fb0 con 0x7f593c0b43a0 2024-03-24T00:36:48.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 -- 172.21.15.40:0/576460637 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f593c06dd80 con 0x7f593c0b43a0 2024-03-24T00:36:48.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5939600640 1 -- 172.21.15.40:0/576460637 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5015 (secure 0 0 0) 0x7f59300f0af0 con 0x7f593c0b43a0 2024-03-24T00:36:48.001 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:48.001 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":88,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1097448819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1097448819},{"type":"v1","addr":"172.21.15.40:6834","nonce":1097448819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:36:07.163876+0000","modified":"2024-03-24T00:36:30.866122+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7186,"mds_1":7180,"mds_2":6404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6404":{"gid":6404,"name":"b","rank":2,"incarnation":86,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/983819364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":983819364},{"type":"v1","addr":"172.21.15.53:6837","nonce":983819364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7180":{"gid":7180,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6835/2302548193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2302548193},{"type":"v1","addr":"172.21.15.40:6835","nonce":2302548193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7186":{"gid":7186,"name":"d","rank":0,"incarnation":78,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/1878888046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1878888046},{"type":"v1","addr":"172.21.15.53:6835","nonce":1878888046}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7186,"qdb_cluster":[7186,7180,6404]},"id":6}]} 2024-03-24T00:36:48.001 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 88 2024-03-24T00:36:48.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 -- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f591c067660 msgr2=0x7f591c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f591c067660 0x7f591c069b20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f593c15b9d0 tx=0x7f5928009040 comp rx=0 tx=0).stop 2024-03-24T00:36:48.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 -- 172.21.15.40:0/576460637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f593c0b43a0 msgr2=0x7f593c0cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f593c0b43a0 0x7f593c0cb6f0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f59300097c0 tx=0x7f593001ab70 comp rx=0 tx=0).stop 2024-03-24T00:36:48.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 -- 172.21.15.40:0/576460637 shutdown_connections 2024-03-24T00:36:48.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f591c067660 0x7f591c069b20 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f593c15f7a0 0x7f593c15d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f593c0cfdf0 0x7f593c15cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 --2- 172.21.15.40:0/576460637 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f593c0b43a0 0x7f593c0cb6f0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 -- 172.21.15.40:0/576460637 >> 172.21.15.40:0/576460637 conn(0x7f593c0b3880 msgr2=0x7f593c15ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:48.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 -- 172.21.15.40:0/576460637 shutdown_connections 2024-03-24T00:36:48.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:47.998+0000 7f5944600640 1 -- 172.21.15.40:0/576460637 wait complete. 2024-03-24T00:36:48.022 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:36:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f4325800640 1 Processor -- start 2024-03-24T00:36:48.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f4325800640 1 -- start start 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f4325800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f432013ec80 0x7f43201414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f4325800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f43201419f0 0x7f43200ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f4325800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43200ce630 0x7f432015f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f4325800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4320141e50 con 0x7f43201419f0 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f4325800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4320070e00 con 0x7f432013ec80 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f4325800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43200cee20 con 0x7f43200ce630 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f431e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f43201419f0 0x7f43200ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f431ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f432013ec80 0x7f43201414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f431e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f43201419f0 0x7f43200ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36638/0 (socket says 172.21.15.40:36638) 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f431e000640 1 -- 172.21.15.40:0/4192965475 learned_addr learned my addr 172.21.15.40:0/4192965475 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f431f400640 1 --2- 172.21.15.40:0/4192965475 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43200ce630 0x7f432015f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f431e000640 1 -- 172.21.15.40:0/4192965475 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43200ce630 msgr2=0x7f432015f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f431e000640 1 --2- 172.21.15.40:0/4192965475 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43200ce630 0x7f432015f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f431e000640 1 -- 172.21.15.40:0/4192965475 >> v2:172.21.15.53:3300/0 conn(0x7f432013ec80 msgr2=0x7f43201414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f431e000640 1 --2- 172.21.15.40:0/4192965475 >> v2:172.21.15.53:3300/0 conn(0x7f432013ec80 0x7f43201414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f431e000640 1 -- 172.21.15.40:0/4192965475 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43200cf100 con 0x7f43201419f0 2024-03-24T00:36:48.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.162+0000 7f431ea00640 1 --2- 172.21.15.40:0/4192965475 >> v2:172.21.15.53:3300/0 conn(0x7f432013ec80 0x7f43201414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:48.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f431e000640 1 --2- 172.21.15.40:0/4192965475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43201419f0 0x7f43200ce0f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f432006b170 tx=0x7f430c01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=438cc59f8fd30a7e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:48.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f431d600640 1 -- 172.21.15.40:0/4192965475 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430c0b0070 con 0x7f43201419f0 2024-03-24T00:36:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f431d600640 1 -- 172.21.15.40:0/4192965475 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430c0ab600 con 0x7f43201419f0 2024-03-24T00:36:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f431d600640 1 -- 172.21.15.40:0/4192965475 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430c0ab9d0 con 0x7f43201419f0 2024-03-24T00:36:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f4325800640 1 -- 172.21.15.40:0/4192965475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43201419f0 msgr2=0x7f43200ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f4325800640 1 --2- 172.21.15.40:0/4192965475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43201419f0 0x7f43200ce0f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f432006b170 tx=0x7f430c01c260 comp rx=0 tx=0).stop 2024-03-24T00:36:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f4325800640 1 -- 172.21.15.40:0/4192965475 shutdown_connections 2024-03-24T00:36:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f4325800640 1 --2- 172.21.15.40:0/4192965475 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f43200ce630 0x7f432015f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f4325800640 1 --2- 172.21.15.40:0/4192965475 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43201419f0 0x7f43200ce0f0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f4325800640 1 --2- 172.21.15.40:0/4192965475 >> v2:172.21.15.53:3300/0 conn(0x7f432013ec80 0x7f43201414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f4325800640 1 -- 172.21.15.40:0/4192965475 >> 172.21.15.40:0/4192965475 conn(0x7f43200b17b0 msgr2=0x7f43200b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f4325800640 1 -- 172.21.15.40:0/4192965475 shutdown_connections 2024-03-24T00:36:48.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f4325800640 1 -- 172.21.15.40:0/4192965475 wait complete. 2024-03-24T00:36:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f4325800640 1 Processor -- start 2024-03-24T00:36:48.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.166+0000 7f4325800640 1 -- start start 2024-03-24T00:36:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f4325800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43200ce630 0x7f432013d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f4325800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f432013ec80 0x7f4320154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f4325800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43201419f0 0x7f432013cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f4325800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4320071460 con 0x7f43201419f0 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f4325800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43200712e0 con 0x7f43200ce630 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f4325800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43200715e0 con 0x7f432013ec80 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f431f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43201419f0 0x7f432013cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f431f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43201419f0 0x7f432013cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36650/0 (socket says 172.21.15.40:36650) 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f431ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43200ce630 0x7f432013d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f431f400640 1 -- 172.21.15.40:0/3484854165 learned_addr learned my addr 172.21.15.40:0/3484854165 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f431e000640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f432013ec80 0x7f4320154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f431f400640 1 -- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f432013ec80 msgr2=0x7f4320154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f431f400640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f432013ec80 0x7f4320154760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f431f400640 1 -- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43200ce630 msgr2=0x7f432013d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f431f400640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43200ce630 0x7f432013d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f431f400640 1 -- 172.21.15.40:0/3484854165 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432013f4d0 con 0x7f43201419f0 2024-03-24T00:36:48.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f431f400640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43201419f0 0x7f432013cc80 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f431800efc0 tx=0x7f431800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f4316a00640 1 -- 172.21.15.40:0/3484854165 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318018020 con 0x7f43201419f0 2024-03-24T00:36:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f4316a00640 1 -- 172.21.15.40:0/3484854165 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4318014480 con 0x7f43201419f0 2024-03-24T00:36:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f4316a00640 1 -- 172.21.15.40:0/3484854165 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318014870 con 0x7f43201419f0 2024-03-24T00:36:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f4325800640 1 -- 172.21.15.40:0/3484854165 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432013f740 con 0x7f43201419f0 2024-03-24T00:36:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f4325800640 1 -- 172.21.15.40:0/3484854165 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f432013fd00 con 0x7f43201419f0 2024-03-24T00:36:48.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.170+0000 7f4325800640 1 -- 172.21.15.40:0/3484854165 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4320068170 con 0x7f43201419f0 2024-03-24T00:36:48.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.174+0000 7f4316a00640 1 -- 172.21.15.40:0/3484854165 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f431801b020 con 0x7f43201419f0 2024-03-24T00:36:48.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.174+0000 7f4316a00640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f43000676b0 0x7f4300069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.174+0000 7f4316a00640 1 -- 172.21.15.40:0/3484854165 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43180869b0 con 0x7f43201419f0 2024-03-24T00:36:48.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.174+0000 7f431ea00640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f43000676b0 0x7f4300069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.174+0000 7f431ea00640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f43000676b0 0x7f4300069b70 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4310009580 tx=0x7f4310009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:48.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.178+0000 7f4316a00640 1 -- 172.21.15.40:0/3484854165 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4318022030 con 0x7f43201419f0 2024-03-24T00:36:48.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.374+0000 7f4325800640 1 -- 172.21.15.40:0/3484854165 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43200b48f0 con 0x7f43201419f0 2024-03-24T00:36:48.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.374+0000 7f4316a00640 1 -- 172.21.15.40:0/3484854165 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5015 (secure 0 0 0) 0x7f4318010040 con 0x7f43201419f0 2024-03-24T00:36:48.378 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:48.378 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":88,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1097448819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1097448819},{"type":"v1","addr":"172.21.15.40:6834","nonce":1097448819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:36:07.163876+0000","modified":"2024-03-24T00:36:30.866122+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7186,"mds_1":7180,"mds_2":6404},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6404":{"gid":6404,"name":"b","rank":2,"incarnation":86,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6837/983819364","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":983819364},{"type":"v1","addr":"172.21.15.53:6837","nonce":983819364}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7180":{"gid":7180,"name":"a","rank":1,"incarnation":84,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6835/2302548193","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":2302548193},{"type":"v1","addr":"172.21.15.40:6835","nonce":2302548193}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7186":{"gid":7186,"name":"d","rank":0,"incarnation":78,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/1878888046","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":1878888046},{"type":"v1","addr":"172.21.15.53:6835","nonce":1878888046}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7186,"qdb_cluster":[7186,7180,6404]},"id":6}]} 2024-03-24T00:36:48.378 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 88 2024-03-24T00:36:48.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.374+0000 7f4325800640 1 -- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f43000676b0 msgr2=0x7f4300069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.374+0000 7f4325800640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f43000676b0 0x7f4300069b70 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f4310009580 tx=0x7f4310009190 comp rx=0 tx=0).stop 2024-03-24T00:36:48.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.374+0000 7f4325800640 1 -- 172.21.15.40:0/3484854165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43201419f0 msgr2=0x7f432013cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.378+0000 7f4325800640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43201419f0 0x7f432013cc80 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f431800efc0 tx=0x7f431800c490 comp rx=0 tx=0).stop 2024-03-24T00:36:48.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.378+0000 7f4325800640 1 -- 172.21.15.40:0/3484854165 shutdown_connections 2024-03-24T00:36:48.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.378+0000 7f4325800640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f43000676b0 0x7f4300069b70 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.378+0000 7f4325800640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f43201419f0 0x7f432013cc80 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.378+0000 7f4325800640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f432013ec80 0x7f4320154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.378+0000 7f4325800640 1 --2- 172.21.15.40:0/3484854165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f43200ce630 0x7f432013d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.378+0000 7f4325800640 1 -- 172.21.15.40:0/3484854165 >> 172.21.15.40:0/3484854165 conn(0x7f43200b17b0 msgr2=0x7f4320151e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:48.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.378+0000 7f4325800640 1 -- 172.21.15.40:0/3484854165 shutdown_connections 2024-03-24T00:36:48.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.378+0000 7f4325800640 1 -- 172.21.15.40:0/3484854165 wait complete. 2024-03-24T00:36:48.398 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:36:48.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.534+0000 7f6e3da00640 1 Processor -- start 2024-03-24T00:36:48.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.534+0000 7f6e3da00640 1 -- start start 2024-03-24T00:36:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.534+0000 7f6e3da00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6e38145760 0x7f6e3806f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.534+0000 7f6e3da00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e380cf200 0x7f6e380cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.534+0000 7f6e3da00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e38072bc0 con 0x7f6e38072750 2024-03-24T00:36:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.534+0000 7f6e3da00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e380cd630 con 0x7f6e380cf200 2024-03-24T00:36:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.534+0000 7f6e3da00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e380cd910 con 0x7f6e38145760 2024-03-24T00:36:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.534+0000 7f6e37400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e380cf200 0x7f6e380cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.534+0000 7f6e36a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6e38145760 0x7f6e3806f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.534+0000 7f6e2fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6e38072750 0x7f6e3806e510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:36112/0 (socket says 172.21.15.40:36112) 2024-03-24T00:36:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.534+0000 7f6e37400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6e380cf200 0x7f6e380cf670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36652/0 (socket says 172.21.15.40:36652) 2024-03-24T00:36:48.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.534+0000 7f6e2fe00640 1 -- 172.21.15.40:0/4234434203 learned_addr learned my addr 172.21.15.40:0/4234434203 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e36000640 1 -- 172.21.15.40:0/4234434203 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101599778 0 0) 0x7f6e38072bc0 con 0x7f6e38072750 2024-03-24T00:36:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e36000640 1 -- 172.21.15.40:0/4234434203 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e1c003530 con 0x7f6e38072750 2024-03-24T00:36:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e37400640 1 -- 172.21.15.40:0/4234434203 >> v2:172.21.15.53:3300/0 conn(0x7f6e38145760 msgr2=0x7f6e3806f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e37400640 1 --2- 172.21.15.40:0/4234434203 >> v2:172.21.15.53:3300/0 conn(0x7f6e38145760 0x7f6e3806f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e37400640 1 -- 172.21.15.40:0/4234434203 >> v1:172.21.15.53:6789/0 conn(0x7f6e38072750 legacy=0x7f6e3806e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e37400640 1 -- 172.21.15.40:0/4234434203 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e380cdbf0 con 0x7f6e380cf200 2024-03-24T00:36:48.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e36a00640 1 --2- 172.21.15.40:0/4234434203 >> v2:172.21.15.53:3300/0 conn(0x7f6e38145760 0x7f6e3806f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:48.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e37400640 1 --2- 172.21.15.40:0/4234434203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e380cf200 0x7f6e380cf670 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6e38072de0 tx=0x7f6e3001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f40c570adc34df8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:48.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e36000640 1 -- 172.21.15.40:0/4234434203 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e300b2070 con 0x7f6e380cf200 2024-03-24T00:36:48.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e36000640 1 -- 172.21.15.40:0/4234434203 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e300a57f0 con 0x7f6e380cf200 2024-03-24T00:36:48.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e36000640 1 -- 172.21.15.40:0/4234434203 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e300a5ba0 con 0x7f6e380cf200 2024-03-24T00:36:48.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e3da00640 1 -- 172.21.15.40:0/4234434203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e380cf200 msgr2=0x7f6e380cf670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e3da00640 1 --2- 172.21.15.40:0/4234434203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e380cf200 0x7f6e380cf670 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6e38072de0 tx=0x7f6e3001c930 comp rx=0 tx=0).stop 2024-03-24T00:36:48.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e3da00640 1 -- 172.21.15.40:0/4234434203 shutdown_connections 2024-03-24T00:36:48.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e3da00640 1 --2- 172.21.15.40:0/4234434203 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e380cf200 0x7f6e380cf670 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e3da00640 1 --2- 172.21.15.40:0/4234434203 >> v2:172.21.15.53:3300/0 conn(0x7f6e38145760 0x7f6e3806f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e3da00640 1 -- 172.21.15.40:0/4234434203 >> 172.21.15.40:0/4234434203 conn(0x7f6e380d0c20 msgr2=0x7f6e380d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:48.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e3da00640 1 -- 172.21.15.40:0/4234434203 shutdown_connections 2024-03-24T00:36:48.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.538+0000 7f6e3da00640 1 -- 172.21.15.40:0/4234434203 wait complete. 2024-03-24T00:36:48.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e3da00640 1 Processor -- start 2024-03-24T00:36:48.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e3da00640 1 -- start start 2024-03-24T00:36:48.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e3da00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6e38072750 0x7f6e3813d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e3da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e380cf200 0x7f6e3815a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e3da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e38145760 0x7f6e38141130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e3da00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3806eda0 con 0x7f6e38145760 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e3da00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3806ec20 con 0x7f6e380cf200 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e3da00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e38168a60 con 0x7f6e38072750 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e37400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e38145760 0x7f6e38141130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e36a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6e38072750 0x7f6e3813d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e2fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e380cf200 0x7f6e3815a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e37400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e38145760 0x7f6e38141130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36664/0 (socket says 172.21.15.40:36664) 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e37400640 1 -- 172.21.15.40:0/4256841905 learned_addr learned my addr 172.21.15.40:0/4256841905 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e37400640 1 -- 172.21.15.40:0/4256841905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6e38072750 msgr2=0x7f6e3813d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e37400640 1 --2- 172.21.15.40:0/4256841905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6e38072750 0x7f6e3813d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e37400640 1 -- 172.21.15.40:0/4256841905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e380cf200 msgr2=0x7f6e3815a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e37400640 1 --2- 172.21.15.40:0/4256841905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e380cf200 0x7f6e3815a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e37400640 1 -- 172.21.15.40:0/4256841905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3813d540 con 0x7f6e38145760 2024-03-24T00:36:48.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e37400640 1 --2- 172.21.15.40:0/4256841905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e38145760 0x7f6e38141130 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6e30019540 tx=0x7f6e300196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e2ea00640 1 -- 172.21.15.40:0/4256841905 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e300b2050 con 0x7f6e38145760 2024-03-24T00:36:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e2ea00640 1 -- 172.21.15.40:0/4256841905 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e300be070 con 0x7f6e38145760 2024-03-24T00:36:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e2ea00640 1 -- 172.21.15.40:0/4256841905 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e300ad870 con 0x7f6e38145760 2024-03-24T00:36:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.542+0000 7f6e3da00640 1 -- 172.21.15.40:0/4256841905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3815b980 con 0x7f6e38145760 2024-03-24T00:36:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.546+0000 7f6e3da00640 1 -- 172.21.15.40:0/4256841905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3815b3c0 con 0x7f6e38145760 2024-03-24T00:36:48.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.546+0000 7f6e3da00640 1 -- 172.21.15.40:0/4256841905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e38068170 con 0x7f6e38145760 2024-03-24T00:36:48.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.546+0000 7f6e2ea00640 1 -- 172.21.15.40:0/4256841905 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6e300ada10 con 0x7f6e38145760 2024-03-24T00:36:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.546+0000 7f6e2ea00640 1 --2- 172.21.15.40:0/4256841905 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6e14067660 0x7f6e14069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.546+0000 7f6e2ea00640 1 -- 172.21.15.40:0/4256841905 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e30120690 con 0x7f6e38145760 2024-03-24T00:36:48.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.550+0000 7f6e36a00640 1 --2- 172.21.15.40:0/4256841905 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6e14067660 0x7f6e14069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.550+0000 7f6e36a00640 1 --2- 172.21.15.40:0/4256841905 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6e14067660 0x7f6e14069b20 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6e28009580 tx=0x7f6e28012040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:48.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.550+0000 7f6e2ea00640 1 -- 172.21.15.40:0/4256841905 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6e30016520 con 0x7f6e38145760 2024-03-24T00:36:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.718+0000 7f6e3da00640 1 -- 172.21.15.40:0/4256841905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e38067d70 con 0x7f6e38145760 2024-03-24T00:36:48.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.718+0000 7f6e2ea00640 1 -- 172.21.15.40:0/4256841905 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16600 (secure 0 0 0) 0x7f6e300f0e70 con 0x7f6e38145760 2024-03-24T00:36:48.722 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:48.722 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":83,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:36:08.785471+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:05.573740+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-03-24T00:36:06.150226+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:36:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.718+0000 7f6e3da00640 1 -- 172.21.15.40:0/4256841905 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6e14067660 msgr2=0x7f6e14069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.722+0000 7f6e3da00640 1 --2- 172.21.15.40:0/4256841905 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6e14067660 0x7f6e14069b20 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6e28009580 tx=0x7f6e28012040 comp rx=0 tx=0).stop 2024-03-24T00:36:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.722+0000 7f6e3da00640 1 -- 172.21.15.40:0/4256841905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e38145760 msgr2=0x7f6e38141130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.722+0000 7f6e3da00640 1 --2- 172.21.15.40:0/4256841905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e38145760 0x7f6e38141130 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f6e30019540 tx=0x7f6e300196d0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.722+0000 7f6e3da00640 1 -- 172.21.15.40:0/4256841905 shutdown_connections 2024-03-24T00:36:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.722+0000 7f6e3da00640 1 --2- 172.21.15.40:0/4256841905 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6e14067660 0x7f6e14069b20 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.722+0000 7f6e3da00640 1 --2- 172.21.15.40:0/4256841905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6e38145760 0x7f6e38141130 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.722+0000 7f6e3da00640 1 --2- 172.21.15.40:0/4256841905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e380cf200 0x7f6e3815a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.722+0000 7f6e3da00640 1 --2- 172.21.15.40:0/4256841905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6e38072750 0x7f6e3813d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.722+0000 7f6e3da00640 1 -- 172.21.15.40:0/4256841905 >> 172.21.15.40:0/4256841905 conn(0x7f6e380d0c20 msgr2=0x7f6e3815df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:48.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.722+0000 7f6e3da00640 1 -- 172.21.15.40:0/4256841905 shutdown_connections 2024-03-24T00:36:48.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.722+0000 7f6e3da00640 1 -- 172.21.15.40:0/4256841905 wait complete. 2024-03-24T00:36:48.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.882+0000 7f0c2ee00640 1 Processor -- start 2024-03-24T00:36:48.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.882+0000 7f0c2ee00640 1 -- start start 2024-03-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.882+0000 7f0c2ee00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0c280ced20 0x7f0c280b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.882+0000 7f0c2ee00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0c2806c8d0 0x7f0c280cc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.882+0000 7f0c2ee00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c280d0600 con 0x7f0c28161880 2024-03-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.882+0000 7f0c2ee00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c28161c60 con 0x7f0c2806c8d0 2024-03-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.882+0000 7f0c2ee00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c2806b150 con 0x7f0c280ced20 2024-03-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.882+0000 7f0c27e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0c28161880 0x7f0c280ce610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57228/0 (socket says 172.21.15.40:57228) 2024-03-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.882+0000 7f0c27e00640 1 -- 172.21.15.40:0/1666762563 learned_addr learned my addr 172.21.15.40:0/1666762563 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.882+0000 7f0c2ce00640 1 --2- 172.21.15.40:0/1666762563 >> v2:172.21.15.40:3300/0 conn(0x7f0c2806c8d0 0x7f0c280cc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.886+0000 7f0c1f400640 1 --2- 172.21.15.40:0/1666762563 >> v2:172.21.15.53:3300/0 conn(0x7f0c280ced20 0x7f0c280b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.886+0000 7f0c27400640 1 -- 172.21.15.40:0/1666762563 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765593513 0 0) 0x7f0c280d0600 con 0x7f0c28161880 2024-03-24T00:36:48.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.886+0000 7f0c27400640 1 -- 172.21.15.40:0/1666762563 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c0c003530 con 0x7f0c28161880 2024-03-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.886+0000 7f0c2ce00640 1 -- 172.21.15.40:0/1666762563 >> v2:172.21.15.53:3300/0 conn(0x7f0c280ced20 msgr2=0x7f0c280b2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.886+0000 7f0c2ce00640 1 --2- 172.21.15.40:0/1666762563 >> v2:172.21.15.53:3300/0 conn(0x7f0c280ced20 0x7f0c280b2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.886+0000 7f0c2ce00640 1 -- 172.21.15.40:0/1666762563 >> v1:172.21.15.40:6789/0 conn(0x7f0c28161880 legacy=0x7f0c280ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.886+0000 7f0c2ce00640 1 -- 172.21.15.40:0/1666762563 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c280d08e0 con 0x7f0c2806c8d0 2024-03-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.886+0000 7f0c2ce00640 1 --2- 172.21.15.40:0/1666762563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c2806c8d0 0x7f0c280cc950 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f0c28071dd0 tx=0x7f0c2001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c012a1df0f5346 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.886+0000 7f0c27400640 1 -- 172.21.15.40:0/1666762563 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c200a66f0 con 0x7f0c2806c8d0 2024-03-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.886+0000 7f0c27400640 1 -- 172.21.15.40:0/1666762563 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c200a6890 con 0x7f0c2806c8d0 2024-03-24T00:36:48.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.886+0000 7f0c27400640 1 -- 172.21.15.40:0/1666762563 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c200a6c60 con 0x7f0c2806c8d0 2024-03-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 -- 172.21.15.40:0/1666762563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c2806c8d0 msgr2=0x7f0c280cc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 --2- 172.21.15.40:0/1666762563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c2806c8d0 0x7f0c280cc950 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f0c28071dd0 tx=0x7f0c2001c8a0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 -- 172.21.15.40:0/1666762563 shutdown_connections 2024-03-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 --2- 172.21.15.40:0/1666762563 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c2806c8d0 0x7f0c280cc950 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 --2- 172.21.15.40:0/1666762563 >> v2:172.21.15.53:3300/0 conn(0x7f0c280ced20 0x7f0c280b2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 -- 172.21.15.40:0/1666762563 >> 172.21.15.40:0/1666762563 conn(0x7f0c280b38f0 msgr2=0x7f0c280b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 -- 172.21.15.40:0/1666762563 shutdown_connections 2024-03-24T00:36:48.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 -- 172.21.15.40:0/1666762563 wait complete. 2024-03-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 Processor -- start 2024-03-24T00:36:48.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 -- start start 2024-03-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c2806c8d0 0x7f0c281575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c280ced20 0x7f0c280b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c28161880 0x7f0c28148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280cda30 con 0x7f0c280ced20 2024-03-24T00:36:48.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280cd8b0 con 0x7f0c2806c8d0 2024-03-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ee00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280cdbb0 con 0x7f0c28161880 2024-03-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c1f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c280ced20 0x7f0c280b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c1f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c280ced20 0x7f0c280b6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36674/0 (socket says 172.21.15.40:36674) 2024-03-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c1f400640 1 -- 172.21.15.40:0/223670165 learned_addr learned my addr 172.21.15.40:0/223670165 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c27e00640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c2806c8d0 0x7f0c281575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c2ce00640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c28161880 0x7f0c28148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c1f400640 1 -- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c28161880 msgr2=0x7f0c28148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c1f400640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c28161880 0x7f0c28148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c1f400640 1 -- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c2806c8d0 msgr2=0x7f0c281575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c1f400640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c2806c8d0 0x7f0c281575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c1f400640 1 -- 172.21.15.40:0/223670165 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c280b7840 con 0x7f0c280ced20 2024-03-24T00:36:48.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.890+0000 7f0c1f400640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c280ced20 0x7f0c280b6970 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f0c1000e8a0 tx=0x7f0c1000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.894+0000 7f0c24c00640 1 -- 172.21.15.40:0/223670165 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c10017070 con 0x7f0c280ced20 2024-03-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.894+0000 7f0c24c00640 1 -- 172.21.15.40:0/223670165 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1000c940 con 0x7f0c280ced20 2024-03-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.894+0000 7f0c24c00640 1 -- 172.21.15.40:0/223670165 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1000ccf0 con 0x7f0c280ced20 2024-03-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.894+0000 7f0c2ee00640 1 -- 172.21.15.40:0/223670165 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c280b6fa0 con 0x7f0c280ced20 2024-03-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.894+0000 7f0c2ee00640 1 -- 172.21.15.40:0/223670165 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c280b7560 con 0x7f0c280ced20 2024-03-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.894+0000 7f0c2ee00640 1 -- 172.21.15.40:0/223670165 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c28068170 con 0x7f0c280ced20 2024-03-24T00:36:48.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.894+0000 7f0c24c00640 1 -- 172.21.15.40:0/223670165 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f0c1001a070 con 0x7f0c280ced20 2024-03-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.894+0000 7f0c24c00640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c04067660 0x7f0c04069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.894+0000 7f0c24c00640 1 -- 172.21.15.40:0/223670165 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c100857e0 con 0x7f0c280ced20 2024-03-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.894+0000 7f0c27e00640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c04067660 0x7f0c04069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:48.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.894+0000 7f0c27e00640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c04067660 0x7f0c04069b20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0c18007fd0 tx=0x7f0c18009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:48.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:48.898+0000 7f0c24c00640 1 -- 172.21.15.40:0/223670165 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0c10055d70 con 0x7f0c280ced20 2024-03-24T00:36:49.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.094+0000 7f0c2ee00640 1 -- 172.21.15.40:0/223670165 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0c280ce960 con 0x7f0c280ced20 2024-03-24T00:36:49.099 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-03-24T00:36:49.094+0000 7fa64e600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:36:49.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.098+0000 7f0c24c00640 1 -- 172.21.15.40:0/223670165 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v89) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0c1005c0c0 con 0x7f0c280ced20 2024-03-24T00:36:49.100 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:36:49.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.098+0000 7f0c2ee00640 1 -- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c04067660 msgr2=0x7f0c04069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:49.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.098+0000 7f0c2ee00640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c04067660 0x7f0c04069b20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0c18007fd0 tx=0x7f0c18009040 comp rx=0 tx=0).stop 2024-03-24T00:36:49.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.098+0000 7f0c2ee00640 1 -- 172.21.15.40:0/223670165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c280ced20 msgr2=0x7f0c280b6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:49.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.098+0000 7f0c2ee00640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c280ced20 0x7f0c280b6970 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f0c1000e8a0 tx=0x7f0c1000eca0 comp rx=0 tx=0).stop 2024-03-24T00:36:49.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.102+0000 7f0c2ee00640 1 -- 172.21.15.40:0/223670165 shutdown_connections 2024-03-24T00:36:49.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.102+0000 7f0c2ee00640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0c04067660 0x7f0c04069b20 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:49.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.102+0000 7f0c2ee00640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c28161880 0x7f0c28148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:49.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.102+0000 7f0c2ee00640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0c280ced20 0x7f0c280b6970 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:49.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.102+0000 7f0c2ee00640 1 --2- 172.21.15.40:0/223670165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c2806c8d0 0x7f0c281575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:49.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.102+0000 7f0c2ee00640 1 -- 172.21.15.40:0/223670165 >> 172.21.15.40:0/223670165 conn(0x7f0c280b38f0 msgr2=0x7f0c2815e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:49.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.102+0000 7f0c2ee00640 1 -- 172.21.15.40:0/223670165 shutdown_connections 2024-03-24T00:36:49.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.102+0000 7f0c2ee00640 1 -- 172.21.15.40:0/223670165 wait complete. 2024-03-24T00:36:49.127 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:36:49.152 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:36:49.171 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:36:49.174 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:36:49.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421f400640 1 Processor -- start 2024-03-24T00:36:49.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421f400640 1 -- start start 2024-03-24T00:36:49.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421f400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f421815b960 0x7f4218070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42180d16f0 0x7f42180712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421f400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421815f970 con 0x7f421815bd40 2024-03-24T00:36:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421f400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4218160580 con 0x7f421815b960 2024-03-24T00:36:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421f400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f421815fc50 con 0x7f42180d16f0 2024-03-24T00:36:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f420fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f421815bd40 0x7f42180d0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57252/0 (socket says 172.21.15.40:57252) 2024-03-24T00:36:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f420fe00640 1 -- 172.21.15.40:0/781761805 learned_addr learned my addr 172.21.15.40:0/781761805 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421ca00640 1 --2- 172.21.15.40:0/781761805 >> v2:172.21.15.53:3300/0 conn(0x7f421815b960 0x7f4218070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421d400640 1 --2- 172.21.15.40:0/781761805 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42180d16f0 0x7f42180712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f4217e00640 1 -- 172.21.15.40:0/781761805 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962770166 0 0) 0x7f421815f970 con 0x7f421815bd40 2024-03-24T00:36:49.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f4217e00640 1 -- 172.21.15.40:0/781761805 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41fc003530 con 0x7f421815bd40 2024-03-24T00:36:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421ca00640 1 -- 172.21.15.40:0/781761805 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42180d16f0 msgr2=0x7f42180712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f4217e00640 1 -- 172.21.15.40:0/781761805 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 729355313 0 0) 0x7f41fc003530 con 0x7f421815bd40 2024-03-24T00:36:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421ca00640 1 --2- 172.21.15.40:0/781761805 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42180d16f0 0x7f42180712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421ca00640 1 -- 172.21.15.40:0/781761805 >> v1:172.21.15.40:6789/0 conn(0x7f421815bd40 legacy=0x7f42180d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421ca00640 1 -- 172.21.15.40:0/781761805 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421815ff30 con 0x7f421815b960 2024-03-24T00:36:49.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.270+0000 7f421d400640 1 --2- 172.21.15.40:0/781761805 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42180d16f0 0x7f42180712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:49.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.274+0000 7f421ca00640 1 --2- 172.21.15.40:0/781761805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f421815b960 0x7f4218070d60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f42080097c0 tx=0x7f420801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11f06697b3fd8258 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:49.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.274+0000 7f4217e00640 1 -- 172.21.15.40:0/781761805 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42080b0070 con 0x7f421815b960 2024-03-24T00:36:49.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.274+0000 7f4217e00640 1 -- 172.21.15.40:0/781761805 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42080ab400 con 0x7f421815b960 2024-03-24T00:36:49.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.274+0000 7f4217e00640 1 -- 172.21.15.40:0/781761805 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4200001680 con 0x7f421815b960 2024-03-24T00:36:49.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.274+0000 7f421f400640 1 -- 172.21.15.40:0/781761805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f421815b960 msgr2=0x7f4218070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:49.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.274+0000 7f421f400640 1 --2- 172.21.15.40:0/781761805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f421815b960 0x7f4218070d60 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f42080097c0 tx=0x7f420801c3c0 comp rx=0 tx=0).stop 2024-03-24T00:36:49.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.274+0000 7f421f400640 1 -- 172.21.15.40:0/781761805 shutdown_connections 2024-03-24T00:36:49.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.274+0000 7f421f400640 1 --2- 172.21.15.40:0/781761805 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42180d16f0 0x7f42180712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:49.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.274+0000 7f421f400640 1 --2- 172.21.15.40:0/781761805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f421815b960 0x7f4218070d60 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:49.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.274+0000 7f421f400640 1 -- 172.21.15.40:0/781761805 >> 172.21.15.40:0/781761805 conn(0x7f421806ecf0 msgr2=0x7f421806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:49.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.274+0000 7f421f400640 1 -- 172.21.15.40:0/781761805 shutdown_connections 2024-03-24T00:36:49.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.274+0000 7f421f400640 1 -- 172.21.15.40:0/781761805 wait complete. 2024-03-24T00:36:49.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421f400640 1 Processor -- start 2024-03-24T00:36:49.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421f400640 1 -- start start 2024-03-24T00:36:49.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42180d16f0 0x7f42180c45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f421815b960 0x7f42180c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f421815bd40 0x7f42180c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421f400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42180ce5a0 con 0x7f421815bd40 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421f400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42180ce420 con 0x7f42180d16f0 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421f400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42180ce720 con 0x7f421815b960 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f421815bd40 0x7f42180c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f421815bd40 0x7f42180c9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36698/0 (socket says 172.21.15.40:36698) 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421d400640 1 -- 172.21.15.40:0/2940540350 learned_addr learned my addr 172.21.15.40:0/2940540350 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421ca00640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42180d16f0 0x7f42180c45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f420fe00640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f421815b960 0x7f42180c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421d400640 1 -- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f421815b960 msgr2=0x7f42180c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421d400640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f421815b960 0x7f42180c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:49.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421d400640 1 -- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42180d16f0 msgr2=0x7f42180c45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:49.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421d400640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42180d16f0 0x7f42180c45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:49.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421d400640 1 -- 172.21.15.40:0/2940540350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42180c75a0 con 0x7f421815bd40 2024-03-24T00:36:49.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421d400640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f421815bd40 0x7f42180c9b40 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f421000efc0 tx=0x7f421000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:49.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f4215600640 1 -- 172.21.15.40:0/2940540350 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210018020 con 0x7f421815bd40 2024-03-24T00:36:49.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f4215600640 1 -- 172.21.15.40:0/2940540350 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4210014480 con 0x7f421815bd40 2024-03-24T00:36:49.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421f400640 1 -- 172.21.15.40:0/2940540350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421815eb80 con 0x7f421815bd40 2024-03-24T00:36:49.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f4215600640 1 -- 172.21.15.40:0/2940540350 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4210014850 con 0x7f421815bd40 2024-03-24T00:36:49.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.278+0000 7f421f400640 1 -- 172.21.15.40:0/2940540350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42180c7b20 con 0x7f421815bd40 2024-03-24T00:36:49.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.282+0000 7f421f400640 1 -- 172.21.15.40:0/2940540350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4218068170 con 0x7f421815bd40 2024-03-24T00:36:49.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.282+0000 7f4215600640 1 -- 172.21.15.40:0/2940540350 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f421001b020 con 0x7f421815bd40 2024-03-24T00:36:49.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.282+0000 7f4215600640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f41f4067660 0x7f41f4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:49.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.282+0000 7f4215600640 1 -- 172.21.15.40:0/2940540350 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5996+0+0 (secure 0 0 0) 0x7f4210086620 con 0x7f421815bd40 2024-03-24T00:36:49.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.282+0000 7f421ca00640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f41f4067660 0x7f41f4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:49.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.282+0000 7f421ca00640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f41f4067660 0x7f41f4069b20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f42080ad000 tx=0x7f42080a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:49.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.286+0000 7f4215600640 1 -- 172.21.15.40:0/2940540350 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4210022030 con 0x7f421815bd40 2024-03-24T00:36:49.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:49.482+0000 7f421f400640 1 -- 172.21.15.40:0/2940540350 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f42180b1f90 con 0x7f421815bd40 2024-03-24T00:36:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.102+0000 7f4215600640 1 -- 172.21.15.40:0/2940540350 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v90) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4210010040 con 0x7f421815bd40 2024-03-24T00:36:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.106+0000 7f421f400640 1 -- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f41f4067660 msgr2=0x7f41f4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.106+0000 7f421f400640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f41f4067660 0x7f41f4069b20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f42080ad000 tx=0x7f42080a6000 comp rx=0 tx=0).stop 2024-03-24T00:36:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.106+0000 7f421f400640 1 -- 172.21.15.40:0/2940540350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f421815bd40 msgr2=0x7f42180c9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.106+0000 7f421f400640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f421815bd40 0x7f42180c9b40 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f421000efc0 tx=0x7f421000c490 comp rx=0 tx=0).stop 2024-03-24T00:36:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.106+0000 7f421f400640 1 -- 172.21.15.40:0/2940540350 shutdown_connections 2024-03-24T00:36:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.106+0000 7f421f400640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f41f4067660 0x7f41f4069b20 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.106+0000 7f421f400640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f421815bd40 0x7f42180c9b40 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.106+0000 7f421f400640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f421815b960 0x7f42180c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.106+0000 7f421f400640 1 --2- 172.21.15.40:0/2940540350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42180d16f0 0x7f42180c45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.106+0000 7f421f400640 1 -- 172.21.15.40:0/2940540350 >> 172.21.15.40:0/2940540350 conn(0x7f421806ecf0 msgr2=0x7f4218070b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.106+0000 7f421f400640 1 -- 172.21.15.40:0/2940540350 shutdown_connections 2024-03-24T00:36:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.106+0000 7f421f400640 1 -- 172.21.15.40:0/2940540350 wait complete. 2024-03-24T00:36:50.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:36:50.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f397f200640 1 Processor -- start 2024-03-24T00:36:50.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f397f200640 1 -- start start 2024-03-24T00:36:50.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f397f200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f39800b2c10 0x7f39800b3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:50.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f397f200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398006cfa0 con 0x7f3980071300 2024-03-24T00:36:50.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f397f200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f398006d240 con 0x7f398015a8f0 2024-03-24T00:36:50.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f397f200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f398006d520 con 0x7f39800b2c10 2024-03-24T00:36:50.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f397e800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3980071300 0x7f398006e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57266/0 (socket says 172.21.15.40:57266) 2024-03-24T00:36:50.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f3975400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f39800b2c10 0x7f39800b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:50.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f397e800640 1 -- 172.21.15.40:0/3930843106 learned_addr learned my addr 172.21.15.40:0/3930843106 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:50.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f3975400640 1 -- 172.21.15.40:0/3930843106 >> v1:172.21.15.53:6789/0 conn(0x7f398015a8f0 legacy=0x7f398015ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:50.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f3975400640 1 -- 172.21.15.40:0/3930843106 >> v1:172.21.15.40:6789/0 conn(0x7f3980071300 legacy=0x7f398006e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:50.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f3975400640 1 -- 172.21.15.40:0/3930843106 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f398006d800 con 0x7f39800b2c10 2024-03-24T00:36:50.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.318+0000 7f397d400640 1 -- 172.21.15.40:0/3930843106 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993509367 0 0) 0x7f398006cfa0 con 0x7f3980071300 2024-03-24T00:36:50.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f3975400640 1 --2- 172.21.15.40:0/3930843106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39800b2c10 0x7f39800b3050 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f39680097c0 tx=0x7f396801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15aa017ec1e8c396 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:50.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f397d400640 1 -- 172.21.15.40:0/3930843106 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39680a7070 con 0x7f39800b2c10 2024-03-24T00:36:50.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f397d400640 1 -- 172.21.15.40:0/3930843106 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39680b0070 con 0x7f39800b2c10 2024-03-24T00:36:50.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f397d400640 1 -- 172.21.15.40:0/3930843106 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39680ab890 con 0x7f39800b2c10 2024-03-24T00:36:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f397f200640 1 -- 172.21.15.40:0/3930843106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39800b2c10 msgr2=0x7f39800b3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f397f200640 1 --2- 172.21.15.40:0/3930843106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39800b2c10 0x7f39800b3050 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f39680097c0 tx=0x7f396801c470 comp rx=0 tx=0).stop 2024-03-24T00:36:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f397f200640 1 -- 172.21.15.40:0/3930843106 shutdown_connections 2024-03-24T00:36:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f397f200640 1 --2- 172.21.15.40:0/3930843106 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39800b2c10 0x7f39800b3050 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f397f200640 1 -- 172.21.15.40:0/3930843106 >> 172.21.15.40:0/3930843106 conn(0x7f39800b1480 msgr2=0x7f39800b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f397f200640 1 -- 172.21.15.40:0/3930843106 shutdown_connections 2024-03-24T00:36:50.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f397f200640 1 -- 172.21.15.40:0/3930843106 wait complete. 2024-03-24T00:36:50.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f397f200640 1 Processor -- start 2024-03-24T00:36:50.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.322+0000 7f397f200640 1 -- start start 2024-03-24T00:36:50.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3980071300 0x7f3980158630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:50.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397f200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39800b2c10 0x7f398013e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:50.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397f200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398015a8f0 0x7f398015ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:50.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397f200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398015ae50 con 0x7f398015a8f0 2024-03-24T00:36:50.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f398015acd0 con 0x7f3980071300 2024-03-24T00:36:50.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397f200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39800aed30 con 0x7f39800b2c10 2024-03-24T00:36:50.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398015a8f0 0x7f398015ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:50.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398015a8f0 0x7f398015ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36728/0 (socket says 172.21.15.40:36728) 2024-03-24T00:36:50.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397e800640 1 -- 172.21.15.40:0/1191681633 learned_addr learned my addr 172.21.15.40:0/1191681633 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:50.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397de00640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3980071300 0x7f3980158630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:50.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f3975400640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39800b2c10 0x7f398013e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:50.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397e800640 1 -- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39800b2c10 msgr2=0x7f398013e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:50.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397e800640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39800b2c10 0x7f398013e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:50.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397e800640 1 -- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3980071300 msgr2=0x7f3980158630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:50.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397e800640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3980071300 0x7f3980158630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:50.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397e800640 1 -- 172.21.15.40:0/1191681633 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3980158f30 con 0x7f398015a8f0 2024-03-24T00:36:50.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397e800640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398015a8f0 0x7f398015ba00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f397800c480 tx=0x7f397800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f3976a00640 1 -- 172.21.15.40:0/1191681633 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3978005cf0 con 0x7f398015a8f0 2024-03-24T00:36:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f3976a00640 1 -- 172.21.15.40:0/1191681633 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3978007070 con 0x7f398015a8f0 2024-03-24T00:36:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f3976a00640 1 -- 172.21.15.40:0/1191681633 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39780074a0 con 0x7f398015a8f0 2024-03-24T00:36:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397f200640 1 -- 172.21.15.40:0/1191681633 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3980159210 con 0x7f398015a8f0 2024-03-24T00:36:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397f200640 1 -- 172.21.15.40:0/1191681633 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39801597d0 con 0x7f398015a8f0 2024-03-24T00:36:50.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.326+0000 7f397f200640 1 -- 172.21.15.40:0/1191681633 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f398004f410 con 0x7f398015a8f0 2024-03-24T00:36:50.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.330+0000 7f3976a00640 1 -- 172.21.15.40:0/1191681633 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f397801b400 con 0x7f398015a8f0 2024-03-24T00:36:50.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.330+0000 7f3976a00640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f395c067660 0x7f395c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:50.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.330+0000 7f3976a00640 1 -- 172.21.15.40:0/1191681633 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5996+0+0 (secure 0 0 0) 0x7f3978085490 con 0x7f398015a8f0 2024-03-24T00:36:50.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.330+0000 7f397de00640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f395c067660 0x7f395c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:50.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.330+0000 7f397de00640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f395c067660 0x7f395c069b20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f39700012f0 tx=0x7f3970009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:50.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.334+0000 7f3976a00640 1 -- 172.21.15.40:0/1191681633 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3978055a30 con 0x7f398015a8f0 2024-03-24T00:36:50.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:50.498+0000 7f397f200640 1 -- 172.21.15.40:0/1191681633 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f39800cca30 con 0x7f398015a8f0 2024-03-24T00:36:51.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.102+0000 7f3976a00640 1 -- 172.21.15.40:0/1191681633 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v86) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3978059ad0 con 0x7f398015a8f0 2024-03-24T00:36:51.108 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:36:51.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.106+0000 7f397f200640 1 -- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f395c067660 msgr2=0x7f395c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:51.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.106+0000 7f397f200640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f395c067660 0x7f395c069b20 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f39700012f0 tx=0x7f3970009040 comp rx=0 tx=0).stop 2024-03-24T00:36:51.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.106+0000 7f397f200640 1 -- 172.21.15.40:0/1191681633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398015a8f0 msgr2=0x7f398015ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:51.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.106+0000 7f397f200640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398015a8f0 0x7f398015ba00 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f397800c480 tx=0x7f397800c940 comp rx=0 tx=0).stop 2024-03-24T00:36:51.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.106+0000 7f397f200640 1 -- 172.21.15.40:0/1191681633 shutdown_connections 2024-03-24T00:36:51.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.106+0000 7f397f200640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f395c067660 0x7f395c069b20 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:51.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.106+0000 7f397f200640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f398015a8f0 0x7f398015ba00 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:51.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.106+0000 7f397f200640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39800b2c10 0x7f398013e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:51.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.106+0000 7f397f200640 1 --2- 172.21.15.40:0/1191681633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3980071300 0x7f3980158630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:51.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.106+0000 7f397f200640 1 -- 172.21.15.40:0/1191681633 >> 172.21.15.40:0/1191681633 conn(0x7f39800b1480 msgr2=0x7f398015ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:51.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.106+0000 7f397f200640 1 -- 172.21.15.40:0/1191681633 shutdown_connections 2024-03-24T00:36:51.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.110+0000 7f397f200640 1 -- 172.21.15.40:0/1191681633 wait complete. 2024-03-24T00:36:51.134 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:36:51.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.274+0000 7fa00fe00640 1 Processor -- start 2024-03-24T00:36:51.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00fe00640 1 -- start start 2024-03-24T00:36:51.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa01015ba00 0x7fa010072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:51.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa0100cda30 0x7fa0100cc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:51.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00fe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0100ce2a0 con 0x7fa0100cc050 2024-03-24T00:36:51.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00fe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0100cc9d0 con 0x7fa0100cda30 2024-03-24T00:36:51.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01015bde0 con 0x7fa01015ba00 2024-03-24T00:36:51.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa006000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa0100cc050 0x7fa0100cd320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57274/0 (socket says 172.21.15.40:57274) 2024-03-24T00:36:51.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa006000640 1 -- 172.21.15.40:0/1318233757 learned_addr learned my addr 172.21.15.40:0/1318233757 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00f400640 1 --2- 172.21.15.40:0/1318233757 >> v2:172.21.15.53:3300/0 conn(0x7fa0100cda30 0x7fa0100cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00ea00640 1 --2- 172.21.15.40:0/1318233757 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa01015ba00 0x7fa010072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00e000640 1 -- 172.21.15.40:0/1318233757 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888488892 0 0) 0x7fa0100ce2a0 con 0x7fa0100cc050 2024-03-24T00:36:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00e000640 1 -- 172.21.15.40:0/1318233757 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ff4003530 con 0x7fa0100cc050 2024-03-24T00:36:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00ea00640 1 -- 172.21.15.40:0/1318233757 >> v2:172.21.15.53:3300/0 conn(0x7fa0100cda30 msgr2=0x7fa0100cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00e000640 1 -- 172.21.15.40:0/1318233757 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1541342792 0 0) 0x7f9ff4003530 con 0x7fa0100cc050 2024-03-24T00:36:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00ea00640 1 --2- 172.21.15.40:0/1318233757 >> v2:172.21.15.53:3300/0 conn(0x7fa0100cda30 0x7fa0100cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:51.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00ea00640 1 -- 172.21.15.40:0/1318233757 >> v1:172.21.15.40:6789/0 conn(0x7fa0100cc050 legacy=0x7fa0100cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:51.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.278+0000 7fa00ea00640 1 -- 172.21.15.40:0/1318233757 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0100cdfc0 con 0x7fa01015ba00 2024-03-24T00:36:51.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.282+0000 7fa00ea00640 1 --2- 172.21.15.40:0/1318233757 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa01015ba00 0x7fa010072720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa0000097c0 tx=0x7fa00001c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecb6a0b26109900b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:51.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.282+0000 7fa00e000640 1 -- 172.21.15.40:0/1318233757 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0000b0070 con 0x7fa01015ba00 2024-03-24T00:36:51.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.282+0000 7fa00e000640 1 -- 172.21.15.40:0/1318233757 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0000ab400 con 0x7fa01015ba00 2024-03-24T00:36:51.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.282+0000 7fa00e000640 1 -- 172.21.15.40:0/1318233757 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8001680 con 0x7fa01015ba00 2024-03-24T00:36:51.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.282+0000 7fa00fe00640 1 -- 172.21.15.40:0/1318233757 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa01015ba00 msgr2=0x7fa010072720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:51.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.282+0000 7fa00fe00640 1 --2- 172.21.15.40:0/1318233757 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa01015ba00 0x7fa010072720 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa0000097c0 tx=0x7fa00001c390 comp rx=0 tx=0).stop 2024-03-24T00:36:51.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.282+0000 7fa00fe00640 1 -- 172.21.15.40:0/1318233757 shutdown_connections 2024-03-24T00:36:51.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.282+0000 7fa00fe00640 1 --2- 172.21.15.40:0/1318233757 >> v2:172.21.15.53:3300/0 conn(0x7fa0100cda30 0x7fa0100cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:51.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.282+0000 7fa00fe00640 1 --2- 172.21.15.40:0/1318233757 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa01015ba00 0x7fa010072720 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:51.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.282+0000 7fa00fe00640 1 -- 172.21.15.40:0/1318233757 >> 172.21.15.40:0/1318233757 conn(0x7fa0100b1480 msgr2=0x7fa0100b1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:51.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.282+0000 7fa00fe00640 1 -- 172.21.15.40:0/1318233757 shutdown_connections 2024-03-24T00:36:51.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.282+0000 7fa00fe00640 1 -- 172.21.15.40:0/1318233757 wait complete. 2024-03-24T00:36:51.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00fe00640 1 Processor -- start 2024-03-24T00:36:51.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00fe00640 1 -- start start 2024-03-24T00:36:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0100cc050 0x7fa010150ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa0100cda30 0x7fa0100ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01015ba00 0x7fa0100ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100b4630 con 0x7fa01015ba00 2024-03-24T00:36:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100b44b0 con 0x7fa0100cc050 2024-03-24T00:36:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100b47b0 con 0x7fa0100cda30 2024-03-24T00:36:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01015ba00 0x7fa0100ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01015ba00 0x7fa0100ade20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36740/0 (socket says 172.21.15.40:36740) 2024-03-24T00:36:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00f400640 1 -- 172.21.15.40:0/1998767876 learned_addr learned my addr 172.21.15.40:0/1998767876 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa006000640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa0100cda30 0x7fa0100ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00ea00640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0100cc050 0x7fa010150ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:51.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00f400640 1 -- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa0100cda30 msgr2=0x7fa0100ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:51.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00f400640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa0100cda30 0x7fa0100ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:51.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00f400640 1 -- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0100cc050 msgr2=0x7fa010150ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:51.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00f400640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0100cc050 0x7fa010150ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:51.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00f400640 1 -- 172.21.15.40:0/1998767876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa010153e90 con 0x7fa01015ba00 2024-03-24T00:36:51.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa006000640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa0100cda30 0x7fa0100ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:36:51.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00ea00640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0100cc050 0x7fa010150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:36:51.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00f400640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01015ba00 0x7fa0100ade20 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa00800efc0 tx=0x7fa00800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:51.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa007400640 1 -- 172.21.15.40:0/1998767876 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008018020 con 0x7fa01015ba00 2024-03-24T00:36:51.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa007400640 1 -- 172.21.15.40:0/1998767876 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa008014480 con 0x7fa01015ba00 2024-03-24T00:36:51.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa007400640 1 -- 172.21.15.40:0/1998767876 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008014870 con 0x7fa01015ba00 2024-03-24T00:36:51.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00fe00640 1 -- 172.21.15.40:0/1998767876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa010154170 con 0x7fa01015ba00 2024-03-24T00:36:51.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00fe00640 1 -- 172.21.15.40:0/1998767876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa010154730 con 0x7fa01015ba00 2024-03-24T00:36:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.286+0000 7fa00fe00640 1 -- 172.21.15.40:0/1998767876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa01004f410 con 0x7fa01015ba00 2024-03-24T00:36:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.290+0000 7fa007400640 1 -- 172.21.15.40:0/1998767876 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa00801b020 con 0x7fa01015ba00 2024-03-24T00:36:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.290+0000 7fa007400640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9fec067690 0x7f9fec069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.290+0000 7fa007400640 1 -- 172.21.15.40:0/1998767876 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5558+0+0 (secure 0 0 0) 0x7fa0080866f0 con 0x7fa01015ba00 2024-03-24T00:36:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.290+0000 7fa00ea00640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9fec067690 0x7f9fec069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:51.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.290+0000 7fa00ea00640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9fec067690 0x7f9fec069b50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa0000ad000 tx=0x7fa0000a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:51.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.294+0000 7fa007400640 1 -- 172.21.15.40:0/1998767876 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa008022020 con 0x7fa01015ba00 2024-03-24T00:36:51.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:51.458+0000 7fa00fe00640 1 -- 172.21.15.40:0/1998767876 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa010155ec0 con 0x7fa01015ba00 2024-03-24T00:36:52.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.110+0000 7fa007400640 1 -- 172.21.15.40:0/1998767876 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v87) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa008010040 con 0x7fa01015ba00 2024-03-24T00:36:52.114 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-03-24T00:36:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.114+0000 7fa00fe00640 1 -- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9fec067690 msgr2=0x7f9fec069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.114+0000 7fa00fe00640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9fec067690 0x7f9fec069b50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa0000ad000 tx=0x7fa0000a6000 comp rx=0 tx=0).stop 2024-03-24T00:36:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.114+0000 7fa00fe00640 1 -- 172.21.15.40:0/1998767876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01015ba00 msgr2=0x7fa0100ade20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.114+0000 7fa00fe00640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01015ba00 0x7fa0100ade20 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fa00800efc0 tx=0x7fa00800c490 comp rx=0 tx=0).stop 2024-03-24T00:36:52.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.114+0000 7fa00fe00640 1 -- 172.21.15.40:0/1998767876 shutdown_connections 2024-03-24T00:36:52.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.114+0000 7fa00fe00640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9fec067690 0x7f9fec069b50 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.114+0000 7fa00fe00640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa01015ba00 0x7fa0100ade20 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.114+0000 7fa00fe00640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa0100cda30 0x7fa0100ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.114+0000 7fa00fe00640 1 --2- 172.21.15.40:0/1998767876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0100cc050 0x7fa010150ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.114+0000 7fa00fe00640 1 -- 172.21.15.40:0/1998767876 >> 172.21.15.40:0/1998767876 conn(0x7fa0100b1480 msgr2=0x7fa0101593c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.114+0000 7fa00fe00640 1 -- 172.21.15.40:0/1998767876 shutdown_connections 2024-03-24T00:36:52.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.114+0000 7fa00fe00640 1 -- 172.21.15.40:0/1998767876 wait complete. 2024-03-24T00:36:52.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random 2024-03-24T00:36:52.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa305e00640 1 Processor -- start 2024-03-24T00:36:52.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa305e00640 1 -- start start 2024-03-24T00:36:52.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa305e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa30015bd00 0x7fa3000ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa305e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa3000728d0 0x7fa3000cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa305e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa300072d60 con 0x7fa30015f8c0 2024-03-24T00:36:52.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa305e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3000cf790 con 0x7fa30015bd00 2024-03-24T00:36:52.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa305e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa300071de0 con 0x7fa3000728d0 2024-03-24T00:36:52.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa2fea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa30015bd00 0x7fa3000ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa2fea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa30015bd00 0x7fa3000ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36756/0 (socket says 172.21.15.40:36756) 2024-03-24T00:36:52.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa2fea00640 1 -- 172.21.15.40:0/747656084 learned_addr learned my addr 172.21.15.40:0/747656084 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:52.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa2f7e00640 1 --2- 172.21.15.40:0/747656084 >> v2:172.21.15.53:3300/0 conn(0x7fa3000728d0 0x7fa3000cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa2fea00640 1 -- 172.21.15.40:0/747656084 >> v2:172.21.15.53:3300/0 conn(0x7fa3000728d0 msgr2=0x7fa3000cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa2fea00640 1 --2- 172.21.15.40:0/747656084 >> v2:172.21.15.53:3300/0 conn(0x7fa3000728d0 0x7fa3000cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa2fea00640 1 -- 172.21.15.40:0/747656084 >> v1:172.21.15.53:6789/0 conn(0x7fa30015f8c0 legacy=0x7fa30015fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.294+0000 7fa2fea00640 1 -- 172.21.15.40:0/747656084 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3000cea50 con 0x7fa30015bd00 2024-03-24T00:36:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.298+0000 7fa2fea00640 1 --2- 172.21.15.40:0/747656084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30015bd00 0x7fa3000ae8e0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa30006ff40 tx=0x7fa2f001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75256a8302539346 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.298+0000 7fa2fe000640 1 -- 172.21.15.40:0/747656084 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f00aa650 con 0x7fa30015bd00 2024-03-24T00:36:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.298+0000 7fa2fe000640 1 -- 172.21.15.40:0/747656084 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f00af070 con 0x7fa30015bd00 2024-03-24T00:36:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.298+0000 7fa2fe000640 1 -- 172.21.15.40:0/747656084 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f00aaa10 con 0x7fa30015bd00 2024-03-24T00:36:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.298+0000 7fa305e00640 1 -- 172.21.15.40:0/747656084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30015bd00 msgr2=0x7fa3000ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.298+0000 7fa305e00640 1 --2- 172.21.15.40:0/747656084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30015bd00 0x7fa3000ae8e0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fa30006ff40 tx=0x7fa2f001c4f0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.298+0000 7fa305e00640 1 -- 172.21.15.40:0/747656084 shutdown_connections 2024-03-24T00:36:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.298+0000 7fa305e00640 1 --2- 172.21.15.40:0/747656084 >> v2:172.21.15.53:3300/0 conn(0x7fa3000728d0 0x7fa3000cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.298+0000 7fa305e00640 1 --2- 172.21.15.40:0/747656084 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa30015bd00 0x7fa3000ae8e0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.298+0000 7fa305e00640 1 -- 172.21.15.40:0/747656084 >> 172.21.15.40:0/747656084 conn(0x7fa3000b1660 msgr2=0x7fa3000b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:52.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.298+0000 7fa305e00640 1 -- 172.21.15.40:0/747656084 shutdown_connections 2024-03-24T00:36:52.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.298+0000 7fa305e00640 1 -- 172.21.15.40:0/747656084 wait complete. 2024-03-24T00:36:52.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa305e00640 1 Processor -- start 2024-03-24T00:36:52.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa305e00640 1 -- start start 2024-03-24T00:36:52.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa305e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000728d0 0x7fa30015eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa305e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa30015bd00 0x7fa3001f85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa305e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa30015f8c0 0x7fa30014b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa305e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30006d320 con 0x7fa3000728d0 2024-03-24T00:36:52.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa305e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30015c0e0 con 0x7fa30015f8c0 2024-03-24T00:36:52.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa305e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30006d4a0 con 0x7fa30015bd00 2024-03-24T00:36:52.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2fea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000728d0 0x7fa30015eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2fea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000728d0 0x7fa30015eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36768/0 (socket says 172.21.15.40:36768) 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2fea00640 1 -- 172.21.15.40:0/3572172224 learned_addr learned my addr 172.21.15.40:0/3572172224 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2f7e00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa30015bd00 0x7fa3001f85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2ff400640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa30015f8c0 0x7fa30014b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2fea00640 1 -- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa30015bd00 msgr2=0x7fa3001f85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2fea00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa30015bd00 0x7fa3001f85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2fea00640 1 -- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa30015f8c0 msgr2=0x7fa30014b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2fea00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa30015f8c0 0x7fa30014b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2fea00640 1 -- 172.21.15.40:0/3572172224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30015cb30 con 0x7fa3000728d0 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2f7e00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa30015bd00 0x7fa3001f85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2ff400640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa30015f8c0 0x7fa30014b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2fea00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000728d0 0x7fa30015eeb0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0019100 tx=0x7fa2f001ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2f7400640 1 -- 172.21.15.40:0/3572172224 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f00bd5f0 con 0x7fa3000728d0 2024-03-24T00:36:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2f7400640 1 -- 172.21.15.40:0/3572172224 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f00a6040 con 0x7fa3000728d0 2024-03-24T00:36:52.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa305e00640 1 -- 172.21.15.40:0/3572172224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3001f8b10 con 0x7fa3000728d0 2024-03-24T00:36:52.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa2f7400640 1 -- 172.21.15.40:0/3572172224 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f00bd990 con 0x7fa3000728d0 2024-03-24T00:36:52.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa305e00640 1 -- 172.21.15.40:0/3572172224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa30014abd0 con 0x7fa3000728d0 2024-03-24T00:36:52.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.302+0000 7fa305e00640 1 -- 172.21.15.40:0/3572172224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa300068170 con 0x7fa3000728d0 2024-03-24T00:36:52.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.306+0000 7fa2f7400640 1 -- 172.21.15.40:0/3572172224 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa2f00c6440 con 0x7fa3000728d0 2024-03-24T00:36:52.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.306+0000 7fa2f7400640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa2dc067660 0x7fa2dc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.306+0000 7fa2f7400640 1 -- 172.21.15.40:0/3572172224 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fa2f011fed0 con 0x7fa3000728d0 2024-03-24T00:36:52.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.306+0000 7fa2f7e00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa2dc067660 0x7fa2dc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.306+0000 7fa2f7e00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa2dc067660 0x7fa2dc069b20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa300068a10 tx=0x7fa2e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:52.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.310+0000 7fa2f7400640 1 -- 172.21.15.40:0/3572172224 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa2f0016110 con 0x7fa3000728d0 2024-03-24T00:36:52.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.474+0000 7fa305e00640 1 -- 172.21.15.40:0/3572172224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"} v 0) v1 -- 0x7fa3000affe0 con 0x7fa3000728d0 2024-03-24T00:36:52.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.478+0000 7fa2f7400640 1 -- 172.21.15.40:0/3572172224 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"}]=0 v47) v1 ==== 110+0+0 (secure 0 0 0) 0x7fa2f00f0d90 con 0x7fa3000728d0 2024-03-24T00:36:52.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.482+0000 7fa305e00640 1 -- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa2dc067660 msgr2=0x7fa2dc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.482+0000 7fa305e00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa2dc067660 0x7fa2dc069b20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa300068a10 tx=0x7fa2e8008040 comp rx=0 tx=0).stop 2024-03-24T00:36:52.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.482+0000 7fa305e00640 1 -- 172.21.15.40:0/3572172224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000728d0 msgr2=0x7fa30015eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.482+0000 7fa305e00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000728d0 0x7fa30015eeb0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0019100 tx=0x7fa2f001ad30 comp rx=0 tx=0).stop 2024-03-24T00:36:52.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.482+0000 7fa305e00640 1 -- 172.21.15.40:0/3572172224 shutdown_connections 2024-03-24T00:36:52.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.482+0000 7fa305e00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa2dc067660 0x7fa2dc069b20 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.482+0000 7fa305e00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa30015f8c0 0x7fa30014b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.482+0000 7fa305e00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa30015bd00 0x7fa3001f85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.482+0000 7fa305e00640 1 --2- 172.21.15.40:0/3572172224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa3000728d0 0x7fa30015eeb0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.485 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.482+0000 7fa305e00640 1 -- 172.21.15.40:0/3572172224 >> 172.21.15.40:0/3572172224 conn(0x7fa3000b1660 msgr2=0x7fa3000b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:52.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.482+0000 7fa305e00640 1 -- 172.21.15.40:0/3572172224 shutdown_connections 2024-03-24T00:36:52.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.482+0000 7fa305e00640 1 -- 172.21.15.40:0/3572172224 wait complete. 2024-03-24T00:36:52.504 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_distributed 2024-03-24T00:36:52.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.642+0000 7feaca000640 1 Processor -- start 2024-03-24T00:36:52.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.642+0000 7feaca000640 1 -- start start 2024-03-24T00:36:52.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feaca000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feac415ffd0 0x7feac40ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feaca000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac4071de0 con 0x7feac415bd00 2024-03-24T00:36:52.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feaca000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac40cea50 con 0x7feac40728d0 2024-03-24T00:36:52.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feaca000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac40ced30 con 0x7feac415ffd0 2024-03-24T00:36:52.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac3400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7feac415bd00 0x7feac415f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57284/0 (socket says 172.21.15.40:57284) 2024-03-24T00:36:52.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac3400640 1 -- 172.21.15.40:0/25149515 learned_addr learned my addr 172.21.15.40:0/25149515 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feabaa00640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feac415ffd0 0x7feac40ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034522417 0 0) 0x7feac4071de0 con 0x7feac415bd00 2024-03-24T00:36:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaa8003530 con 0x7feac415bd00 2024-03-24T00:36:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3079631903 0 0) 0x7feaa8003530 con 0x7feac415bd00 2024-03-24T00:36:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feac415ffd0 msgr2=0x7feac40ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feac415ffd0 0x7feac40ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 >> v1:172.21.15.53:6789/0 conn(0x7feac40728d0 legacy=0x7feac406cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac4071b00 con 0x7feac415bd00 2024-03-24T00:36:52.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7feab4001680 con 0x7feac415bd00 2024-03-24T00:36:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 >> v1:172.21.15.40:6789/0 conn(0x7feac415bd00 legacy=0x7feac415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa80039f0 0x7feaa8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feaa8004810 0x7feaa8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feac415ffd0 0x7feaa80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac40ced30 con 0x7feaa80039f0 2024-03-24T00:36:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa800bab0 con 0x7feaa8004810 2024-03-24T00:36:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaa800bcb0 con 0x7feac415ffd0 2024-03-24T00:36:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac3400640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa80039f0 0x7feaa8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feabaa00640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feaa8004810 0x7feaa8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac3e00640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feac415ffd0 0x7feaa80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac3400640 1 -- 172.21.15.40:0/25149515 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feac415ffd0 msgr2=0x7feaa80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:52.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac3400640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feac415ffd0 0x7feaa80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac3400640 1 -- 172.21.15.40:0/25149515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feaa8004810 msgr2=0x7feaa8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:52.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac3400640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feaa8004810 0x7feaa8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feac3400640 1 -- 172.21.15.40:0/25149515 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa8003530 con 0x7feaa80039f0 2024-03-24T00:36:52.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.646+0000 7feabaa00640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feaa8004810 0x7feaa8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:52.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.650+0000 7feac3400640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa80039f0 0x7feaa8003e10 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7feab400c0e0 tx=0x7feab401cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13aa9f5c7edebc10 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:52.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.650+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab4007c30 con 0x7feaa80039f0 2024-03-24T00:36:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.650+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab40a9020 con 0x7feaa80039f0 2024-03-24T00:36:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.650+0000 7feac2a00640 1 -- 172.21.15.40:0/25149515 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab4005ae0 con 0x7feaa80039f0 2024-03-24T00:36:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.650+0000 7feaca000640 1 -- 172.21.15.40:0/25149515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa80039f0 msgr2=0x7feaa8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.650+0000 7feaca000640 1 --2- 172.21.15.40:0/25149515 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feaa80039f0 0x7feaa8003e10 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7feab400c0e0 tx=0x7feab401cb00 comp rx=0 tx=0).stop 2024-03-24T00:36:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.650+0000 7feac3e00640 1 -- 172.21.15.40:0/25149515 reap_dead start 2024-03-24T00:36:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.650+0000 7feaca000640 1 -- 172.21.15.40:0/25149515 shutdown_connections 2024-03-24T00:36:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.650+0000 7feaca000640 1 -- 172.21.15.40:0/25149515 >> 172.21.15.40:0/25149515 conn(0x7feac40b1660 msgr2=0x7feac40b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.650+0000 7feaca000640 1 -- 172.21.15.40:0/25149515 shutdown_connections 2024-03-24T00:36:52.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.650+0000 7feaca000640 1 -- 172.21.15.40:0/25149515 wait complete. 2024-03-24T00:36:52.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feaca000640 1 Processor -- start 2024-03-24T00:36:52.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feaca000640 1 -- start start 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feaca000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feaa80039f0 0x7feac4151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feaca000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac40b05d0 0x7feac413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feaca000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac40aef70 0x7feac415d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feaca000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac40cf250 con 0x7feac40aef70 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feaca000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac415c0e0 con 0x7feac40b05d0 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feaca000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac40cf3d0 con 0x7feaa80039f0 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feac3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac40aef70 0x7feac415d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feac3400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feaa80039f0 0x7feac4151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feac3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac40aef70 0x7feac415d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36790/0 (socket says 172.21.15.40:36790) 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feac3e00640 1 -- 172.21.15.40:0/1779341822 learned_addr learned my addr 172.21.15.40:0/1779341822 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feabaa00640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac40b05d0 0x7feac413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feac3e00640 1 -- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feaa80039f0 msgr2=0x7feac4151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feac3e00640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feaa80039f0 0x7feac4151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feac3e00640 1 -- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac40b05d0 msgr2=0x7feac413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feac3e00640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac40b05d0 0x7feac413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feac3e00640 1 -- 172.21.15.40:0/1779341822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac40af530 con 0x7feac40aef70 2024-03-24T00:36:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feac3e00640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac40aef70 0x7feac415d2e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7feac40b4e30 tx=0x7feabc00f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feabbe00640 1 -- 172.21.15.40:0/1779341822 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc0053a0 con 0x7feac40aef70 2024-03-24T00:36:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feabbe00640 1 -- 172.21.15.40:0/1779341822 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feabc018070 con 0x7feac40aef70 2024-03-24T00:36:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feabbe00640 1 -- 172.21.15.40:0/1779341822 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc003840 con 0x7feac40aef70 2024-03-24T00:36:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feaca000640 1 -- 172.21.15.40:0/1779341822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac40af810 con 0x7feac40aef70 2024-03-24T00:36:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feaca000640 1 -- 172.21.15.40:0/1779341822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac40afdd0 con 0x7feac40aef70 2024-03-24T00:36:52.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.654+0000 7feaca000640 1 -- 172.21.15.40:0/1779341822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac4068170 con 0x7feac40aef70 2024-03-24T00:36:52.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.658+0000 7feabbe00640 1 -- 172.21.15.40:0/1779341822 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7feabc0039e0 con 0x7feac40aef70 2024-03-24T00:36:52.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.658+0000 7feabbe00640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feaa0067690 0x7feaa0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.658+0000 7feabbe00640 1 -- 172.21.15.40:0/1779341822 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5115+0+0 (secure 0 0 0) 0x7feabc085160 con 0x7feac40aef70 2024-03-24T00:36:52.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.658+0000 7feac3400640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feaa0067690 0x7feaa0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.658+0000 7feac3400640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feaa0067690 0x7feaa0069b50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7feab40aa000 tx=0x7feab40b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:52.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.662+0000 7feabbe00640 1 -- 172.21.15.40:0/1779341822 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7feabc055940 con 0x7feac40aef70 2024-03-24T00:36:52.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.822+0000 7feaca000640 1 -- 172.21.15.40:0/1779341822 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"} v 0) v1 -- 0x7feac406e870 con 0x7feac40aef70 2024-03-24T00:36:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.826+0000 7feabbe00640 1 -- 172.21.15.40:0/1779341822 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"}]=0 v48) v1 ==== 115+0+0 (secure 0 0 0) 0x7feabc0599e0 con 0x7feac40aef70 2024-03-24T00:36:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.830+0000 7feaca000640 1 -- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feaa0067690 msgr2=0x7feaa0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.830+0000 7feaca000640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feaa0067690 0x7feaa0069b50 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7feab40aa000 tx=0x7feab40b1040 comp rx=0 tx=0).stop 2024-03-24T00:36:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.830+0000 7feaca000640 1 -- 172.21.15.40:0/1779341822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac40aef70 msgr2=0x7feac415d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.830+0000 7feaca000640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac40aef70 0x7feac415d2e0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7feac40b4e30 tx=0x7feabc00f960 comp rx=0 tx=0).stop 2024-03-24T00:36:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.830+0000 7feaca000640 1 -- 172.21.15.40:0/1779341822 shutdown_connections 2024-03-24T00:36:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.830+0000 7feaca000640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feaa0067690 0x7feaa0069b50 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.830+0000 7feaca000640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7feac40aef70 0x7feac415d2e0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.830+0000 7feaca000640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac40b05d0 0x7feac413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.830+0000 7feaca000640 1 --2- 172.21.15.40:0/1779341822 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7feaa80039f0 0x7feac4151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.830+0000 7feaca000640 1 -- 172.21.15.40:0/1779341822 >> 172.21.15.40:0/1779341822 conn(0x7feac40b1660 msgr2=0x7feac40b3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.830+0000 7feaca000640 1 -- 172.21.15.40:0/1779341822 shutdown_connections 2024-03-24T00:36:52.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.830+0000 7feaca000640 1 -- 172.21.15.40:0/1779341822 wait complete. 2024-03-24T00:36:52.851 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random_max 2024-03-24T00:36:52.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.990+0000 7f005cc00640 1 Processor -- start 2024-03-24T00:36:52.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.990+0000 7f005cc00640 1 -- start start 2024-03-24T00:36:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f005cc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f005806f2c0 0x7f005806ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f005cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00580b3ac0 0x7f005815da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f005cc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005815c9d0 con 0x7f005815b110 2024-03-24T00:36:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f005cc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f005815ccb0 con 0x7f005806f2c0 2024-03-24T00:36:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f005cc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f005815cf90 con 0x7f00580b3ac0 2024-03-24T00:36:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f0055400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f005815b110 0x7f005806f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57290/0 (socket says 172.21.15.40:57290) 2024-03-24T00:36:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f0055400640 1 -- 172.21.15.40:0/2568937815 learned_addr learned my addr 172.21.15.40:0/2568937815 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f0055e00640 1 --2- 172.21.15.40:0/2568937815 >> v2:172.21.15.40:3300/0 conn(0x7f005806f2c0 0x7f005806ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f0056800640 1 --2- 172.21.15.40:0/2568937815 >> v2:172.21.15.53:3300/0 conn(0x7f00580b3ac0 0x7f005815da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:52.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f0054a00640 1 -- 172.21.15.40:0/2568937815 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3117159422 0 0) 0x7f005815c9d0 con 0x7f005815b110 2024-03-24T00:36:52.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f0054a00640 1 -- 172.21.15.40:0/2568937815 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f003c005740 con 0x7f005815b110 2024-03-24T00:36:52.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f0055e00640 1 -- 172.21.15.40:0/2568937815 >> v2:172.21.15.53:3300/0 conn(0x7f00580b3ac0 msgr2=0x7f005815da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:52.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f0055e00640 1 --2- 172.21.15.40:0/2568937815 >> v2:172.21.15.53:3300/0 conn(0x7f00580b3ac0 0x7f005815da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:52.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f0055e00640 1 -- 172.21.15.40:0/2568937815 >> v1:172.21.15.40:6789/0 conn(0x7f005815b110 legacy=0x7f005806f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:52.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f0055e00640 1 -- 172.21.15.40:0/2568937815 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005815d270 con 0x7f005806f2c0 2024-03-24T00:36:52.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.994+0000 7f0056800640 1 --2- 172.21.15.40:0/2568937815 >> v2:172.21.15.53:3300/0 conn(0x7f00580b3ac0 0x7f005815da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f0055e00640 1 --2- 172.21.15.40:0/2568937815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f005806f2c0 0x7f005806ecf0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f00580b3660 tx=0x7f004801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c3a7e3e1789831 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f0054a00640 1 -- 172.21.15.40:0/2568937815 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00480b0070 con 0x7f005806f2c0 2024-03-24T00:36:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f0054a00640 1 -- 172.21.15.40:0/2568937815 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00480ab600 con 0x7f005806f2c0 2024-03-24T00:36:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f0054a00640 1 -- 172.21.15.40:0/2568937815 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00480ab9b0 con 0x7f005806f2c0 2024-03-24T00:36:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 -- 172.21.15.40:0/2568937815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f005806f2c0 msgr2=0x7f005806ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 --2- 172.21.15.40:0/2568937815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f005806f2c0 0x7f005806ecf0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f00580b3660 tx=0x7f004801c480 comp rx=0 tx=0).stop 2024-03-24T00:36:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 -- 172.21.15.40:0/2568937815 shutdown_connections 2024-03-24T00:36:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 --2- 172.21.15.40:0/2568937815 >> v2:172.21.15.53:3300/0 conn(0x7f00580b3ac0 0x7f005815da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 --2- 172.21.15.40:0/2568937815 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f005806f2c0 0x7f005806ecf0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 -- 172.21.15.40:0/2568937815 >> 172.21.15.40:0/2568937815 conn(0x7f005806b150 msgr2=0x7f005806b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 -- 172.21.15.40:0/2568937815 shutdown_connections 2024-03-24T00:36:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 -- 172.21.15.40:0/2568937815 wait complete. 2024-03-24T00:36:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 Processor -- start 2024-03-24T00:36:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 -- start start 2024-03-24T00:36:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f005806f2c0 0x7f00580ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00580b3ac0 0x7f00581342d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f005815b110 0x7f00580b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0058067810 con 0x7f005815b110 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0058067690 con 0x7f00580b3ac0 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:52.998+0000 7f005cc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0058067990 con 0x7f005806f2c0 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f0056800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f005815b110 0x7f00580b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f0056800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f005815b110 0x7f00580b4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36814/0 (socket says 172.21.15.40:36814) 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f0056800640 1 -- 172.21.15.40:0/242327238 learned_addr learned my addr 172.21.15.40:0/242327238 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f0055400640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00580b3ac0 0x7f00581342d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f0055e00640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f005806f2c0 0x7f00580ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f0056800640 1 -- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f005806f2c0 msgr2=0x7f00580ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f0056800640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f005806f2c0 0x7f00580ceda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f0056800640 1 -- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00580b3ac0 msgr2=0x7f00581342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f0056800640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00580b3ac0 0x7f00581342d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f0056800640 1 -- 172.21.15.40:0/242327238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00580cf2e0 con 0x7f005815b110 2024-03-24T00:36:53.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f0055e00640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f005806f2c0 0x7f00580ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:53.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f0056800640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f005815b110 0x7f00580b4310 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f0050009fa0 tx=0x7f005000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:53.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f004e000640 1 -- 172.21.15.40:0/242327238 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005001a070 con 0x7f005815b110 2024-03-24T00:36:53.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f004e000640 1 -- 172.21.15.40:0/242327238 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0050003bc0 con 0x7f005815b110 2024-03-24T00:36:53.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f004e000640 1 -- 172.21.15.40:0/242327238 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050004030 con 0x7f005815b110 2024-03-24T00:36:53.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f005cc00640 1 -- 172.21.15.40:0/242327238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0058164c70 con 0x7f005815b110 2024-03-24T00:36:53.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f005cc00640 1 -- 172.21.15.40:0/242327238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0058165230 con 0x7f005815b110 2024-03-24T00:36:53.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f005cc00640 1 -- 172.21.15.40:0/242327238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0058068170 con 0x7f005815b110 2024-03-24T00:36:53.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.002+0000 7f004e000640 1 -- 172.21.15.40:0/242327238 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f005001d070 con 0x7f005815b110 2024-03-24T00:36:53.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.006+0000 7f004e000640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0038067660 0x7f0038069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.006+0000 7f004e000640 1 -- 172.21.15.40:0/242327238 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f00500858c0 con 0x7f005815b110 2024-03-24T00:36:53.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.006+0000 7f0055e00640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0038067660 0x7f0038069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.006+0000 7f0055e00640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0038067660 0x7f0038069b20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f00480ad000 tx=0x7f00480a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:53.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.006+0000 7f004e000640 1 -- 172.21.15.40:0/242327238 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0050010070 con 0x7f005815b110 2024-03-24T00:36:53.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.174+0000 7f005cc00640 1 -- 172.21.15.40:0/242327238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"} v 0) v1 -- 0x7f005806bb90 con 0x7f005815b110 2024-03-24T00:36:53.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f004e000640 1 -- 172.21.15.40:0/242327238 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"}]=0 v49) v1 ==== 114+0+0 (secure 0 0 0) 0x7f0050056140 con 0x7f005815b110 2024-03-24T00:36:53.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f005cc00640 1 -- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0038067660 msgr2=0x7f0038069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f005cc00640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0038067660 0x7f0038069b20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f00480ad000 tx=0x7f00480a6000 comp rx=0 tx=0).stop 2024-03-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f005cc00640 1 -- 172.21.15.40:0/242327238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f005815b110 msgr2=0x7f00580b4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f005cc00640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f005815b110 0x7f00580b4310 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f0050009fa0 tx=0x7f005000d560 comp rx=0 tx=0).stop 2024-03-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f005cc00640 1 -- 172.21.15.40:0/242327238 shutdown_connections 2024-03-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f005cc00640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0038067660 0x7f0038069b20 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f005cc00640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f005815b110 0x7f00580b4310 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f005cc00640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00580b3ac0 0x7f00581342d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f005cc00640 1 --2- 172.21.15.40:0/242327238 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f005806f2c0 0x7f00580ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f005cc00640 1 -- 172.21.15.40:0/242327238 >> 172.21.15.40:0/242327238 conn(0x7f005806b150 msgr2=0x7f00580cf730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f005cc00640 1 -- 172.21.15.40:0/242327238 shutdown_connections 2024-03-24T00:36:53.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.178+0000 7f005cc00640 1 -- 172.21.15.40:0/242327238 wait complete. 2024-03-24T00:36:53.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:36:53.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29d5000640 1 Processor -- start 2024-03-24T00:36:53.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29d5000640 1 -- start start 2024-03-24T00:36:53.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29d5000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29d015bd00 0x7f29d00ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29d5000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f29d00728d0 0x7f29d00cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29d5000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d0072d30 con 0x7f29d015f8c0 2024-03-24T00:36:53.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29d5000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d00cf790 con 0x7f29d00728d0 2024-03-24T00:36:53.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29d5000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d0071de0 con 0x7f29d015bd00 2024-03-24T00:36:53.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29c7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f29d00728d0 0x7f29d00cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29c7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f29d00728d0 0x7f29d00cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36822/0 (socket says 172.21.15.40:36822) 2024-03-24T00:36:53.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29c7e00640 1 -- 172.21.15.40:0/3014292627 learned_addr learned my addr 172.21.15.40:0/3014292627 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:53.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29cde00640 1 --2- 172.21.15.40:0/3014292627 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29d015bd00 0x7f29d00ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29c7e00640 1 -- 172.21.15.40:0/3014292627 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29d015bd00 msgr2=0x7f29d00ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:53.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29c7e00640 1 --2- 172.21.15.40:0/3014292627 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29d015bd00 0x7f29d00ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29c7e00640 1 -- 172.21.15.40:0/3014292627 >> v1:172.21.15.53:6789/0 conn(0x7f29d015f8c0 legacy=0x7f29d006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:53.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.342+0000 7f29c7e00640 1 -- 172.21.15.40:0/3014292627 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d00cea50 con 0x7f29d00728d0 2024-03-24T00:36:53.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.346+0000 7f29c7e00640 1 --2- 172.21.15.40:0/3014292627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29d00728d0 0x7f29d00cf250 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009870 tx=0x7f29b801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf6ad118fdebc5d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:53.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.346+0000 7f29cd400640 1 -- 172.21.15.40:0/3014292627 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b80a95f0 con 0x7f29d00728d0 2024-03-24T00:36:53.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.346+0000 7f29cd400640 1 -- 172.21.15.40:0/3014292627 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b80a9790 con 0x7f29d00728d0 2024-03-24T00:36:53.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.346+0000 7f29cd400640 1 -- 172.21.15.40:0/3014292627 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b80a9b60 con 0x7f29d00728d0 2024-03-24T00:36:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.346+0000 7f29d5000640 1 -- 172.21.15.40:0/3014292627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29d00728d0 msgr2=0x7f29d00cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.346+0000 7f29d5000640 1 --2- 172.21.15.40:0/3014292627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29d00728d0 0x7f29d00cf250 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009870 tx=0x7f29b801c530 comp rx=0 tx=0).stop 2024-03-24T00:36:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.346+0000 7f29d5000640 1 -- 172.21.15.40:0/3014292627 shutdown_connections 2024-03-24T00:36:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.346+0000 7f29d5000640 1 --2- 172.21.15.40:0/3014292627 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29d00728d0 0x7f29d00cf250 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.346+0000 7f29d5000640 1 --2- 172.21.15.40:0/3014292627 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29d015bd00 0x7f29d00ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.346+0000 7f29d5000640 1 -- 172.21.15.40:0/3014292627 >> 172.21.15.40:0/3014292627 conn(0x7f29d00b1660 msgr2=0x7f29d00b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.346+0000 7f29d5000640 1 -- 172.21.15.40:0/3014292627 shutdown_connections 2024-03-24T00:36:53.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.346+0000 7f29d5000640 1 -- 172.21.15.40:0/3014292627 wait complete. 2024-03-24T00:36:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29d5000640 1 Processor -- start 2024-03-24T00:36:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29d5000640 1 -- start start 2024-03-24T00:36:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29d5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29d00728d0 0x7f29d0154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29d5000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29d015bd00 0x7f29d0145f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29d5000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29d015f8c0 0x7f29d01fb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29d5000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d015c0e0 con 0x7f29d00728d0 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29d5000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d006d630 con 0x7f29d015f8c0 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29d5000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d01600e0 con 0x7f29d015bd00 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29cde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29d00728d0 0x7f29d0154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29cde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29d00728d0 0x7f29d0154a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36826/0 (socket says 172.21.15.40:36826) 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29cde00640 1 -- 172.21.15.40:0/667841375 learned_addr learned my addr 172.21.15.40:0/667841375 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29c7e00640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29d015bd00 0x7f29d0145f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29ce800640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29d015f8c0 0x7f29d01fb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29cde00640 1 -- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29d015bd00 msgr2=0x7f29d0145f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29cde00640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29d015bd00 0x7f29d0145f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29cde00640 1 -- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29d015f8c0 msgr2=0x7f29d01fb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29cde00640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29d015f8c0 0x7f29d01fb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29cde00640 1 -- 172.21.15.40:0/667841375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d01464a0 con 0x7f29d00728d0 2024-03-24T00:36:53.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29cde00640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29d00728d0 0x7f29d0154a40 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f29c000d800 tx=0x7f29c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29c6000640 1 -- 172.21.15.40:0/667841375 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c000b6c0 con 0x7f29d00728d0 2024-03-24T00:36:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29c6000640 1 -- 172.21.15.40:0/667841375 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c0019070 con 0x7f29d00728d0 2024-03-24T00:36:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29d5000640 1 -- 172.21.15.40:0/667841375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d01fbad0 con 0x7f29d00728d0 2024-03-24T00:36:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29c6000640 1 -- 172.21.15.40:0/667841375 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c000ba60 con 0x7f29d00728d0 2024-03-24T00:36:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29d5000640 1 -- 172.21.15.40:0/667841375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d0155450 con 0x7f29d00728d0 2024-03-24T00:36:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.350+0000 7f29d5000640 1 -- 172.21.15.40:0/667841375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d0068170 con 0x7f29d00728d0 2024-03-24T00:36:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.354+0000 7f29c6000640 1 -- 172.21.15.40:0/667841375 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f29c000bc00 con 0x7f29d00728d0 2024-03-24T00:36:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.354+0000 7f29c6000640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f29ac067690 0x7f29ac069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.354+0000 7f29c6000640 1 -- 172.21.15.40:0/667841375 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f29c0085a10 con 0x7f29d00728d0 2024-03-24T00:36:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.354+0000 7f29c7e00640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f29ac067690 0x7f29ac069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.354+0000 7f29c7e00640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f29ac067690 0x7f29ac069b50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f29b80ae000 tx=0x7f29b80b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:53.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.358+0000 7f29c6000640 1 -- 172.21.15.40:0/667841375 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f29c00561f0 con 0x7f29d00728d0 2024-03-24T00:36:53.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.522+0000 7f29d5000640 1 -- 172.21.15.40:0/667841375 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f29d006e870 con 0x7f29d00728d0 2024-03-24T00:36:53.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.526+0000 7f29c6000640 1 -- 172.21.15.40:0/667841375 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7f29c005c020 con 0x7f29d00728d0 2024-03-24T00:36:53.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.530+0000 7f29d5000640 1 -- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f29ac067690 msgr2=0x7f29ac069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.530+0000 7f29d5000640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f29ac067690 0x7f29ac069b50 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f29b80ae000 tx=0x7f29b80b3040 comp rx=0 tx=0).stop 2024-03-24T00:36:53.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.530+0000 7f29d5000640 1 -- 172.21.15.40:0/667841375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29d00728d0 msgr2=0x7f29d0154a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.530+0000 7f29d5000640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29d00728d0 0x7f29d0154a40 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f29c000d800 tx=0x7f29c000dcc0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.530+0000 7f29d5000640 1 -- 172.21.15.40:0/667841375 shutdown_connections 2024-03-24T00:36:53.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.530+0000 7f29d5000640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f29ac067690 0x7f29ac069b50 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.530+0000 7f29d5000640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29d015f8c0 0x7f29d01fb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.530+0000 7f29d5000640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29d015bd00 0x7f29d0145f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.530+0000 7f29d5000640 1 --2- 172.21.15.40:0/667841375 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f29d00728d0 0x7f29d0154a40 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.530+0000 7f29d5000640 1 -- 172.21.15.40:0/667841375 >> 172.21.15.40:0/667841375 conn(0x7f29d00b1660 msgr2=0x7f29d006d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:53.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.530+0000 7f29d5000640 1 -- 172.21.15.40:0/667841375 shutdown_connections 2024-03-24T00:36:53.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.530+0000 7f29d5000640 1 -- 172.21.15.40:0/667841375 wait complete. 2024-03-24T00:36:53.551 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_after' 2024-03-24T00:36:53.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7f5600640 1 Processor -- start 2024-03-24T00:36:53.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7f5600640 1 -- start start 2024-03-24T00:36:53.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7f5600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7f015aa50 0x7fe7f00ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7f5600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f00b3420 con 0x7fe7f00722e0 2024-03-24T00:36:53.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7f5600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f0070d40 con 0x7fe7f015ae30 2024-03-24T00:36:53.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7f5600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f0070af0 con 0x7fe7f015aa50 2024-03-24T00:36:53.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7ef400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe7f00722e0 0x7fe7f006cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57300/0 (socket says 172.21.15.40:57300) 2024-03-24T00:36:53.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7ef400640 1 -- 172.21.15.40:0/160752513 learned_addr learned my addr 172.21.15.40:0/160752513 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:53.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7efe00640 1 --2- 172.21.15.40:0/160752513 >> v2:172.21.15.53:3300/0 conn(0x7fe7f015aa50 0x7fe7f00ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7eea00640 1 -- 172.21.15.40:0/160752513 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270435937 0 0) 0x7fe7dc004b40 con 0x7fe7f00722e0 2024-03-24T00:36:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7eea00640 1 -- 172.21.15.40:0/160752513 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7d4003530 con 0x7fe7f00722e0 2024-03-24T00:36:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7eea00640 1 -- 172.21.15.40:0/160752513 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1779266959 0 0) 0x7fe7f0070d40 con 0x7fe7f015ae30 2024-03-24T00:36:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7eea00640 1 -- 172.21.15.40:0/160752513 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7dc004b40 con 0x7fe7f015ae30 2024-03-24T00:36:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7efe00640 1 -- 172.21.15.40:0/160752513 >> v1:172.21.15.53:6789/0 conn(0x7fe7f015ae30 legacy=0x7fe7f0071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7efe00640 1 -- 172.21.15.40:0/160752513 >> v1:172.21.15.40:6789/0 conn(0x7fe7f00722e0 legacy=0x7fe7f006cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.690+0000 7fe7efe00640 1 -- 172.21.15.40:0/160752513 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f0071020 con 0x7fe7f015aa50 2024-03-24T00:36:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.694+0000 7fe7efe00640 1 --2- 172.21.15.40:0/160752513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f015aa50 0x7fe7f00ccde0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00097c0 tx=0x7fe7e001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6075cbe0299648a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.694+0000 7fe7eea00640 1 -- 172.21.15.40:0/160752513 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e00b0070 con 0x7fe7f015aa50 2024-03-24T00:36:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.694+0000 7fe7eea00640 1 -- 172.21.15.40:0/160752513 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e00ab600 con 0x7fe7f015aa50 2024-03-24T00:36:53.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.694+0000 7fe7eea00640 1 -- 172.21.15.40:0/160752513 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e00ab9d0 con 0x7fe7f015aa50 2024-03-24T00:36:53.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.694+0000 7fe7f5600640 1 -- 172.21.15.40:0/160752513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f015aa50 msgr2=0x7fe7f00ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.694+0000 7fe7f5600640 1 --2- 172.21.15.40:0/160752513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f015aa50 0x7fe7f00ccde0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00097c0 tx=0x7fe7e001c410 comp rx=0 tx=0).stop 2024-03-24T00:36:53.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.694+0000 7fe7f5600640 1 -- 172.21.15.40:0/160752513 shutdown_connections 2024-03-24T00:36:53.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.694+0000 7fe7f5600640 1 --2- 172.21.15.40:0/160752513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f015aa50 0x7fe7f00ccde0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.694+0000 7fe7f5600640 1 -- 172.21.15.40:0/160752513 >> 172.21.15.40:0/160752513 conn(0x7fe7f00b1740 msgr2=0x7fe7f00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:53.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.694+0000 7fe7f5600640 1 -- 172.21.15.40:0/160752513 shutdown_connections 2024-03-24T00:36:53.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.694+0000 7fe7f5600640 1 -- 172.21.15.40:0/160752513 wait complete. 2024-03-24T00:36:53.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f5600640 1 Processor -- start 2024-03-24T00:36:53.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f5600640 1 -- start start 2024-03-24T00:36:53.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f00722e0 0x7fe7f014edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f015aa50 0x7fe7f015d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f5600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe7f015ae30 0x7fe7f015d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f5600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f00b46b0 con 0x7fe7f00722e0 2024-03-24T00:36:53.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f5600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f0071f20 con 0x7fe7f015aa50 2024-03-24T00:36:53.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f5600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f00b4830 con 0x7fe7f015ae30 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7efe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f00722e0 0x7fe7f014edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7efe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f00722e0 0x7fe7f014edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36836/0 (socket says 172.21.15.40:36836) 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7efe00640 1 -- 172.21.15.40:0/4145505549 learned_addr learned my addr 172.21.15.40:0/4145505549 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7ef400640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f015aa50 0x7fe7f015d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f4c00640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe7f015ae30 0x7fe7f015d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7efe00640 1 -- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe7f015ae30 msgr2=0x7fe7f015d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7efe00640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe7f015ae30 0x7fe7f015d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7efe00640 1 -- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f015aa50 msgr2=0x7fe7f015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7efe00640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f015aa50 0x7fe7f015d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7efe00640 1 -- 172.21.15.40:0/4145505549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f015a160 con 0x7fe7f00722e0 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f4c00640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe7f015ae30 0x7fe7f015d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7efe00640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f00722e0 0x7fe7f014edb0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00ae040 tx=0x7fe7e001ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7e7e00640 1 -- 172.21.15.40:0/4145505549 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e00b0050 con 0x7fe7f00722e0 2024-03-24T00:36:53.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7e7e00640 1 -- 172.21.15.40:0/4145505549 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e00bc070 con 0x7fe7f00722e0 2024-03-24T00:36:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f5600640 1 -- 172.21.15.40:0/4145505549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f014f620 con 0x7fe7f00722e0 2024-03-24T00:36:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7e7e00640 1 -- 172.21.15.40:0/4145505549 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e00b5a30 con 0x7fe7f00722e0 2024-03-24T00:36:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f5600640 1 -- 172.21.15.40:0/4145505549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f014fbe0 con 0x7fe7f00722e0 2024-03-24T00:36:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.698+0000 7fe7f5600640 1 -- 172.21.15.40:0/4145505549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f0073540 con 0x7fe7f00722e0 2024-03-24T00:36:53.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.702+0000 7fe7e7e00640 1 -- 172.21.15.40:0/4145505549 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe7e00ab480 con 0x7fe7f00722e0 2024-03-24T00:36:53.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.702+0000 7fe7e7e00640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe7d0067660 0x7fe7d0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:53.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.702+0000 7fe7e7e00640 1 -- 172.21.15.40:0/4145505549 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fe7e011ff70 con 0x7fe7f00722e0 2024-03-24T00:36:53.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.702+0000 7fe7ef400640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe7d0067660 0x7fe7d0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:53.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.702+0000 7fe7ef400640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe7d0067660 0x7fe7d0069b20 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00adcf0 tx=0x7fe7dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:53.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.706+0000 7fe7e7e00640 1 -- 172.21.15.40:0/4145505549 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe7e0016090 con 0x7fe7f00722e0 2024-03-24T00:36:53.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:53.930+0000 7fe7f5600640 1 -- 172.21.15.40:0/4145505549 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_after"]} v 0) v1 -- 0x7fe7f004f4f0 con 0x7fe7f00722e0 2024-03-24T00:36:54.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.114+0000 7fe7e7e00640 1 -- 172.21.15.40:0/4145505549 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_after"]}]=0 v302) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe7e00f0e30 con 0x7fe7f00722e0 2024-03-24T00:36:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.118+0000 7fe7f5600640 1 -- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe7d0067660 msgr2=0x7fe7d0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.118+0000 7fe7f5600640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe7d0067660 0x7fe7d0069b20 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00adcf0 tx=0x7fe7dc009040 comp rx=0 tx=0).stop 2024-03-24T00:36:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.118+0000 7fe7f5600640 1 -- 172.21.15.40:0/4145505549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f00722e0 msgr2=0x7fe7f014edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.118+0000 7fe7f5600640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f00722e0 0x7fe7f014edb0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00ae040 tx=0x7fe7e001ac50 comp rx=0 tx=0).stop 2024-03-24T00:36:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.118+0000 7fe7f5600640 1 -- 172.21.15.40:0/4145505549 shutdown_connections 2024-03-24T00:36:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.118+0000 7fe7f5600640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe7d0067660 0x7fe7d0069b20 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.118+0000 7fe7f5600640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe7f015ae30 0x7fe7f015d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.118+0000 7fe7f5600640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f015aa50 0x7fe7f015d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.118+0000 7fe7f5600640 1 --2- 172.21.15.40:0/4145505549 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe7f00722e0 0x7fe7f014edb0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.118+0000 7fe7f5600640 1 -- 172.21.15.40:0/4145505549 >> 172.21.15.40:0/4145505549 conn(0x7fe7f00b1740 msgr2=0x7fe7f00ae240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.118+0000 7fe7f5600640 1 -- 172.21.15.40:0/4145505549 shutdown_connections 2024-03-24T00:36:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.118+0000 7fe7f5600640 1 -- 172.21.15.40:0/4145505549 wait complete. 2024-03-24T00:36:54.141 INFO:tasks.cephfs_test_runner:That a conventional export pin overrides the distributed policy _after_ distributed policy is set. ... ok 2024-03-24T00:36:54.141 INFO:tasks.cephfs_test_runner:Starting test: test_ephemeral_pin_dist_override_before (tasks.cephfs.test_exports.TestEphemeralPins) That a conventional export pin overrides the distributed policy _before_ distributed policy is set. 2024-03-24T00:36:54.141 INFO:tasks.cephfs_test_runner:test_ephemeral_pin_dist_override_before (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:36:54.142 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:36:54.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.278+0000 7f87d8000640 1 Processor -- start 2024-03-24T00:36:54.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.278+0000 7f87d8000640 1 -- start start 2024-03-24T00:36:54.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d8000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87d006e620 0x7f87d00b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d8000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f87d006ea00 0x7f87d00b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d8000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d0070f60 con 0x7f87d00b2ff0 2024-03-24T00:36:54.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d8000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d00b4c00 con 0x7f87d006ea00 2024-03-24T00:36:54.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d8000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d00b3620 con 0x7f87d006e620 2024-03-24T00:36:54.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d4c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f87d006ea00 0x7f87d00b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d4c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f87d006ea00 0x7f87d00b2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36844/0 (socket says 172.21.15.40:36844) 2024-03-24T00:36:54.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d4c00640 1 -- 172.21.15.40:0/1267613090 learned_addr learned my addr 172.21.15.40:0/1267613090 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:54.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d5600640 1 --2- 172.21.15.40:0/1267613090 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87d006e620 0x7f87d00b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d4c00640 1 -- 172.21.15.40:0/1267613090 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87d006e620 msgr2=0x7f87d00b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:54.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d4c00640 1 --2- 172.21.15.40:0/1267613090 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87d006e620 0x7f87d00b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d4c00640 1 -- 172.21.15.40:0/1267613090 >> v1:172.21.15.53:6789/0 conn(0x7f87d00b2ff0 legacy=0x7f87d00ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.282+0000 7f87d4c00640 1 -- 172.21.15.40:0/1267613090 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d00b3900 con 0x7f87d006ea00 2024-03-24T00:36:54.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87d4c00640 1 --2- 172.21.15.40:0/1267613090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87d006ea00 0x7f87d00b2ab0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009870 tx=0x7f87bc01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf187588c14bd94 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:54.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87cfe00640 1 -- 172.21.15.40:0/1267613090 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc0b1070 con 0x7f87d006ea00 2024-03-24T00:36:54.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87cfe00640 1 -- 172.21.15.40:0/1267613090 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87bc0ac620 con 0x7f87d006ea00 2024-03-24T00:36:54.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87cfe00640 1 -- 172.21.15.40:0/1267613090 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87bc0ac9f0 con 0x7f87d006ea00 2024-03-24T00:36:54.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87d8000640 1 -- 172.21.15.40:0/1267613090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87d006ea00 msgr2=0x7f87d00b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87d8000640 1 --2- 172.21.15.40:0/1267613090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87d006ea00 0x7f87d00b2ab0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f87bc009870 tx=0x7f87bc01c4d0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87d8000640 1 -- 172.21.15.40:0/1267613090 shutdown_connections 2024-03-24T00:36:54.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87d8000640 1 --2- 172.21.15.40:0/1267613090 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87d006ea00 0x7f87d00b2ab0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87d8000640 1 --2- 172.21.15.40:0/1267613090 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87d006e620 0x7f87d00b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87d8000640 1 -- 172.21.15.40:0/1267613090 >> 172.21.15.40:0/1267613090 conn(0x7f87d00b17b0 msgr2=0x7f87d00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:54.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87d8000640 1 -- 172.21.15.40:0/1267613090 shutdown_connections 2024-03-24T00:36:54.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87d8000640 1 -- 172.21.15.40:0/1267613090 wait complete. 2024-03-24T00:36:54.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87d8000640 1 Processor -- start 2024-03-24T00:36:54.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.286+0000 7f87d8000640 1 -- start start 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d8000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d006e620 0x7f87d0154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d8000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87d006ea00 0x7f87d0128a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d8000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87d00b2ff0 0x7f87d0155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d8000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d0144920 con 0x7f87d00b2ff0 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d8000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d006ee60 con 0x7f87d006e620 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d8000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d0141f20 con 0x7f87d006ea00 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d006e620 0x7f87d0154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87d00b2ff0 0x7f87d0155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d006e620 0x7f87d0154800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:58446/0 (socket says 172.21.15.40:58446) 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d5600640 1 -- 172.21.15.40:0/2872115000 learned_addr learned my addr 172.21.15.40:0/2872115000 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d4c00640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87d006ea00 0x7f87d0128a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d6000640 1 -- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87d006ea00 msgr2=0x7f87d0128a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d6000640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87d006ea00 0x7f87d0128a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d6000640 1 -- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d006e620 msgr2=0x7f87d0154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d6000640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d006e620 0x7f87d0154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d6000640 1 -- 172.21.15.40:0/2872115000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d0154d40 con 0x7f87d00b2ff0 2024-03-24T00:36:54.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d6000640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87d00b2ff0 0x7f87d0155060 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f87c800c700 tx=0x7f87c800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:54.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87cd600640 1 -- 172.21.15.40:0/2872115000 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8003b80 con 0x7f87d00b2ff0 2024-03-24T00:36:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87cd600640 1 -- 172.21.15.40:0/2872115000 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c8003d20 con 0x7f87d00b2ff0 2024-03-24T00:36:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d8000640 1 -- 172.21.15.40:0/2872115000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d01593f0 con 0x7f87d00b2ff0 2024-03-24T00:36:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87cd600640 1 -- 172.21.15.40:0/2872115000 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8005310 con 0x7f87d00b2ff0 2024-03-24T00:36:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d8000640 1 -- 172.21.15.40:0/2872115000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d01424c0 con 0x7f87d00b2ff0 2024-03-24T00:36:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87d8000640 1 -- 172.21.15.40:0/2872115000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d0068170 con 0x7f87d00b2ff0 2024-03-24T00:36:54.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.290+0000 7f87cd600640 1 -- 172.21.15.40:0/2872115000 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f87c80054b0 con 0x7f87d00b2ff0 2024-03-24T00:36:54.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.294+0000 7f87cd600640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f87b0067660 0x7f87b0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.294+0000 7f87cd600640 1 -- 172.21.15.40:0/2872115000 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f87c8085330 con 0x7f87d00b2ff0 2024-03-24T00:36:54.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.294+0000 7f87d5600640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f87b0067660 0x7f87b0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.294+0000 7f87d5600640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f87b0067660 0x7f87b0069b20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009c20 tx=0x7f87c4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:54.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.298+0000 7f87cd600640 1 -- 172.21.15.40:0/2872115000 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f87c8055b10 con 0x7f87d00b2ff0 2024-03-24T00:36:54.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.462+0000 7f87d8000640 1 -- 172.21.15.40:0/2872115000 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87d0072f50 con 0x7f87d00b2ff0 2024-03-24T00:36:54.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87cd600640 1 -- 172.21.15.40:0/2872115000 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+13219 (secure 0 0 0) 0x7f87c8059bb0 con 0x7f87d00b2ff0 2024-03-24T00:36:54.468 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:54.468 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":87,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:36:52.109001+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6833/2302548193":"2024-03-25T00:36:49.099856+0000","172.21.15.40:6835/2302548193":"2024-03-25T00:36:49.099856+0000","172.21.15.53:6834/1878888046":"2024-03-25T00:36:49.099861+0000","172.21.15.53:6837/983819364":"2024-03-25T00:36:49.099843+0000","172.21.15.53:6835/1878888046":"2024-03-25T00:36:49.099861+0000","172.21.15.53:6836/983819364":"2024-03-25T00:36:49.099843+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:36:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87d8000640 1 -- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f87b0067660 msgr2=0x7f87b0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87d8000640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f87b0067660 0x7f87b0069b20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f87c4009c20 tx=0x7f87c4009190 comp rx=0 tx=0).stop 2024-03-24T00:36:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87d8000640 1 -- 172.21.15.40:0/2872115000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87d00b2ff0 msgr2=0x7f87d0155060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87d8000640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87d00b2ff0 0x7f87d0155060 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f87c800c700 tx=0x7f87c800cbc0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87d8000640 1 -- 172.21.15.40:0/2872115000 shutdown_connections 2024-03-24T00:36:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87d8000640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f87b0067660 0x7f87b0069b20 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87d8000640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f87d00b2ff0 0x7f87d0155060 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87d8000640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87d006ea00 0x7f87d0128a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87d8000640 1 --2- 172.21.15.40:0/2872115000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d006e620 0x7f87d0154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87d8000640 1 -- 172.21.15.40:0/2872115000 >> 172.21.15.40:0/2872115000 conn(0x7f87d00b17b0 msgr2=0x7f87d00cdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:54.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87d8000640 1 -- 172.21.15.40:0/2872115000 shutdown_connections 2024-03-24T00:36:54.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.466+0000 7f87d8000640 1 -- 172.21.15.40:0/2872115000 wait complete. 2024-03-24T00:36:54.490 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:36:54.490 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:36:54.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.630+0000 7f5f99200640 1 Processor -- start 2024-03-24T00:36:54.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.630+0000 7f5f99200640 1 -- start start 2024-03-24T00:36:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.630+0000 7f5f99200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9406cb80 0x7f5f940b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.630+0000 7f5f99200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5f940b4840 0x7f5f94071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.630+0000 7f5f99200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f940b3260 con 0x7f5f9415ac60 2024-03-24T00:36:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.630+0000 7f5f99200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f940b3e70 con 0x7f5f940b4840 2024-03-24T00:36:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.630+0000 7f5f99200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f940b3540 con 0x7f5f9406cb80 2024-03-24T00:36:54.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.630+0000 7f5f91e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5f9415ac60 0x7f5f940b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57302/0 (socket says 172.21.15.40:57302) 2024-03-24T00:36:54.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.630+0000 7f5f91e00640 1 -- 172.21.15.40:0/2079285908 learned_addr learned my addr 172.21.15.40:0/2079285908 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:54.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.630+0000 7f5f92800640 1 --2- 172.21.15.40:0/2079285908 >> v2:172.21.15.40:3300/0 conn(0x7f5f940b4840 0x7f5f94071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.634+0000 7f5f8be00640 1 --2- 172.21.15.40:0/2079285908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9406cb80 0x7f5f940b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.634+0000 7f5f91400640 1 -- 172.21.15.40:0/2079285908 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521370266 0 0) 0x7f5f940b3260 con 0x7f5f9415ac60 2024-03-24T00:36:54.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.634+0000 7f5f91400640 1 -- 172.21.15.40:0/2079285908 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f78005740 con 0x7f5f9415ac60 2024-03-24T00:36:54.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.634+0000 7f5f92800640 1 -- 172.21.15.40:0/2079285908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9406cb80 msgr2=0x7f5f940b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:54.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.634+0000 7f5f92800640 1 --2- 172.21.15.40:0/2079285908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9406cb80 0x7f5f940b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.634+0000 7f5f92800640 1 -- 172.21.15.40:0/2079285908 >> v1:172.21.15.40:6789/0 conn(0x7f5f9415ac60 legacy=0x7f5f940b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.634+0000 7f5f92800640 1 -- 172.21.15.40:0/2079285908 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f940b3820 con 0x7f5f940b4840 2024-03-24T00:36:54.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.634+0000 7f5f92800640 1 --2- 172.21.15.40:0/2079285908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f940b4840 0x7f5f94071b00 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c009d90 tx=0x7f5f8c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb26a3df7f3d261 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:54.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.634+0000 7f5f91400640 1 -- 172.21.15.40:0/2079285908 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c0b2070 con 0x7f5f940b4840 2024-03-24T00:36:54.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.634+0000 7f5f91400640 1 -- 172.21.15.40:0/2079285908 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8c0a57a0 con 0x7f5f940b4840 2024-03-24T00:36:54.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.634+0000 7f5f91400640 1 -- 172.21.15.40:0/2079285908 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c0a5b70 con 0x7f5f940b4840 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 -- 172.21.15.40:0/2079285908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f940b4840 msgr2=0x7f5f94071b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 --2- 172.21.15.40:0/2079285908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f940b4840 0x7f5f94071b00 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c009d90 tx=0x7f5f8c01c8e0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 -- 172.21.15.40:0/2079285908 shutdown_connections 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 --2- 172.21.15.40:0/2079285908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f940b4840 0x7f5f94071b00 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 --2- 172.21.15.40:0/2079285908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9406cb80 0x7f5f940b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 -- 172.21.15.40:0/2079285908 >> 172.21.15.40:0/2079285908 conn(0x7f5f940b1660 msgr2=0x7f5f940b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 -- 172.21.15.40:0/2079285908 shutdown_connections 2024-03-24T00:36:54.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 -- 172.21.15.40:0/2079285908 wait complete. 2024-03-24T00:36:54.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 Processor -- start 2024-03-24T00:36:54.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 -- start start 2024-03-24T00:36:54.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f9406cb80 0x7f5f94146b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940b4840 0x7f5f94154d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9415ac60 0x7f5f940ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9406d730 con 0x7f5f9406cb80 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9406d5b0 con 0x7f5f940b4840 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f99200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9406d8b0 con 0x7f5f9415ac60 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f91e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f9406cb80 0x7f5f94146b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f91e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f9406cb80 0x7f5f94146b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36874/0 (socket says 172.21.15.40:36874) 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f91e00640 1 -- 172.21.15.40:0/4115223530 learned_addr learned my addr 172.21.15.40:0/4115223530 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f8be00640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940b4840 0x7f5f94154d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f92800640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9415ac60 0x7f5f940ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f91e00640 1 -- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9415ac60 msgr2=0x7f5f940ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f91e00640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9415ac60 0x7f5f940ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f91e00640 1 -- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940b4840 msgr2=0x7f5f94154d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f91e00640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940b4840 0x7f5f94154d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f91e00640 1 -- 172.21.15.40:0/4115223530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9415b2f0 con 0x7f5f9406cb80 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f92800640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9415ac60 0x7f5f940ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:54.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f8be00640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940b4840 0x7f5f94154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:54.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f91e00640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f9406cb80 0x7f5f94146b90 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f5f8400c3c0 tx=0x7f5f8400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:54.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.638+0000 7f5f8a000640 1 -- 172.21.15.40:0/4115223530 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84017070 con 0x7f5f9406cb80 2024-03-24T00:36:54.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.642+0000 7f5f8a000640 1 -- 172.21.15.40:0/4115223530 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f84005cf0 con 0x7f5f9406cb80 2024-03-24T00:36:54.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.642+0000 7f5f8a000640 1 -- 172.21.15.40:0/4115223530 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f840072f0 con 0x7f5f9406cb80 2024-03-24T00:36:54.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.642+0000 7f5f99200640 1 -- 172.21.15.40:0/4115223530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9415b590 con 0x7f5f9406cb80 2024-03-24T00:36:54.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.642+0000 7f5f99200640 1 -- 172.21.15.40:0/4115223530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9415bb50 con 0x7f5f9406cb80 2024-03-24T00:36:54.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.642+0000 7f5f99200640 1 -- 172.21.15.40:0/4115223530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f94068170 con 0x7f5f9406cb80 2024-03-24T00:36:54.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.642+0000 7f5f8a000640 1 -- 172.21.15.40:0/4115223530 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5f8401a070 con 0x7f5f9406cb80 2024-03-24T00:36:54.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.642+0000 7f5f8a000640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f70067690 0x7f5f70069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.642+0000 7f5f8a000640 1 -- 172.21.15.40:0/4115223530 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f5f84084af0 con 0x7f5f9406cb80 2024-03-24T00:36:54.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.642+0000 7f5f8be00640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f70067690 0x7f5f70069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.646+0000 7f5f8be00640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f70067690 0x7f5f70069b50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5f9414e130 tx=0x7f5f7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:54.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.646+0000 7f5f8a000640 1 -- 172.21.15.40:0/4115223530 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5f84055920 con 0x7f5f9406cb80 2024-03-24T00:36:54.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 -- 172.21.15.40:0/4115223530 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5f940b2d00 con 0x7f5f9406cb80 2024-03-24T00:36:54.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f8a000640 1 -- 172.21.15.40:0/4115223530 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5f84055220 con 0x7f5f9406cb80 2024-03-24T00:36:54.817 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:36:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 -- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f70067690 msgr2=0x7f5f70069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f70067690 0x7f5f70069b50 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f5f9414e130 tx=0x7f5f7c008040 comp rx=0 tx=0).stop 2024-03-24T00:36:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 -- 172.21.15.40:0/4115223530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f9406cb80 msgr2=0x7f5f94146b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f9406cb80 0x7f5f94146b90 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f5f8400c3c0 tx=0x7f5f8400c880 comp rx=0 tx=0).stop 2024-03-24T00:36:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 -- 172.21.15.40:0/4115223530 shutdown_connections 2024-03-24T00:36:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5f70067690 0x7f5f70069b50 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5f9415ac60 0x7f5f940ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940b4840 0x7f5f94154d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 --2- 172.21.15.40:0/4115223530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5f9406cb80 0x7f5f94146b90 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 -- 172.21.15.40:0/4115223530 >> 172.21.15.40:0/4115223530 conn(0x7f5f940b1660 msgr2=0x7f5f94154510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:54.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 -- 172.21.15.40:0/4115223530 shutdown_connections 2024-03-24T00:36:54.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.814+0000 7f5f99200640 1 -- 172.21.15.40:0/4115223530 wait complete. 2024-03-24T00:36:54.838 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_before' 2024-03-24T00:36:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe6be00640 1 Processor -- start 2024-03-24T00:36:54.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe6be00640 1 -- start start 2024-03-24T00:36:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe6be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe6c072510 0x7efe6c072910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe6be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c06d900 con 0x7efe6c15d890 2024-03-24T00:36:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe6be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6c06db70 con 0x7efe6c0cd1a0 2024-03-24T00:36:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe6be00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c0cd580 con 0x7efe6c072510 2024-03-24T00:36:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe6b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7efe6c15d890 0x7efe6c15dd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57316/0 (socket says 172.21.15.40:57316) 2024-03-24T00:36:54.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe6b400640 1 -- 172.21.15.40:0/1619663493 learned_addr learned my addr 172.21.15.40:0/1619663493 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe6a000640 1 --2- 172.21.15.40:0/1619663493 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe6c072510 0x7efe6c072910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe69600640 1 -- 172.21.15.40:0/1619663493 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040271616 0 0) 0x7efe6c06d900 con 0x7efe6c15d890 2024-03-24T00:36:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe69600640 1 -- 172.21.15.40:0/1619663493 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe50003530 con 0x7efe6c15d890 2024-03-24T00:36:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe69600640 1 -- 172.21.15.40:0/1619663493 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308881152 0 0) 0x7efe6c06db70 con 0x7efe6c0cd1a0 2024-03-24T00:36:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe69600640 1 -- 172.21.15.40:0/1619663493 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe6c06d900 con 0x7efe6c0cd1a0 2024-03-24T00:36:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe6a000640 1 -- 172.21.15.40:0/1619663493 >> v1:172.21.15.53:6789/0 conn(0x7efe6c0cd1a0 legacy=0x7efe6c06d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.987 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe6a000640 1 -- 172.21.15.40:0/1619663493 >> v1:172.21.15.40:6789/0 conn(0x7efe6c15d890 legacy=0x7efe6c15dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe69600640 1 -- 172.21.15.40:0/1619663493 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1118265081 0 0) 0x7efe50003530 con 0x7efe6c15d890 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.978+0000 7efe6a000640 1 -- 172.21.15.40:0/1619663493 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c06de30 con 0x7efe6c072510 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.982+0000 7efe6a000640 1 --2- 172.21.15.40:0/1619663493 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe6c072510 0x7efe6c072910 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efe580097f0 tx=0x7efe5801c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4e4b198b0ff15e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.982+0000 7efe69600640 1 -- 172.21.15.40:0/1619663493 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe580b0070 con 0x7efe6c072510 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.982+0000 7efe69600640 1 -- 172.21.15.40:0/1619663493 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe580ab400 con 0x7efe6c072510 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.982+0000 7efe69600640 1 -- 172.21.15.40:0/1619663493 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe64001a00 con 0x7efe6c072510 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.982+0000 7efe6be00640 1 -- 172.21.15.40:0/1619663493 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe6c072510 msgr2=0x7efe6c072910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.982+0000 7efe6be00640 1 --2- 172.21.15.40:0/1619663493 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe6c072510 0x7efe6c072910 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7efe580097f0 tx=0x7efe5801c400 comp rx=0 tx=0).stop 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.982+0000 7efe6be00640 1 -- 172.21.15.40:0/1619663493 shutdown_connections 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.982+0000 7efe6be00640 1 --2- 172.21.15.40:0/1619663493 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe6c072510 0x7efe6c072910 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.982+0000 7efe6be00640 1 -- 172.21.15.40:0/1619663493 >> 172.21.15.40:0/1619663493 conn(0x7efe6c0b1c80 msgr2=0x7efe6c0b2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.982+0000 7efe6be00640 1 -- 172.21.15.40:0/1619663493 shutdown_connections 2024-03-24T00:36:54.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.982+0000 7efe6be00640 1 -- 172.21.15.40:0/1619663493 wait complete. 2024-03-24T00:36:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6be00640 1 Processor -- start 2024-03-24T00:36:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6be00640 1 -- start start 2024-03-24T00:36:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe6c072510 0x7efe6c14e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe6c0cd1a0 0x7efe6c156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe6c15d890 0x7efe6c150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c09e0a0 con 0x7efe6c15d890 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c09df20 con 0x7efe6c0cd1a0 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6be00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6c169890 con 0x7efe6c072510 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe6c15d890 0x7efe6c150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe6c15d890 0x7efe6c150930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36882/0 (socket says 172.21.15.40:36882) 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6b400640 1 -- 172.21.15.40:0/4122620400 learned_addr learned my addr 172.21.15.40:0/4122620400 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6aa00640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe6c072510 0x7efe6c14e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6a000640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe6c0cd1a0 0x7efe6c156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6b400640 1 -- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe6c072510 msgr2=0x7efe6c14e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6b400640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe6c072510 0x7efe6c14e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6b400640 1 -- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe6c0cd1a0 msgr2=0x7efe6c156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6b400640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe6c0cd1a0 0x7efe6c156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6b400640 1 -- 172.21.15.40:0/4122620400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6c157490 con 0x7efe6c15d890 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6b400640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe6c15d890 0x7efe6c150930 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7efe6400cc20 tx=0x7efe6400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe62a00640 1 -- 172.21.15.40:0/4122620400 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6400ce30 con 0x7efe6c15d890 2024-03-24T00:36:54.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe62a00640 1 -- 172.21.15.40:0/4122620400 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe64012400 con 0x7efe6c15d890 2024-03-24T00:36:54.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6be00640 1 -- 172.21.15.40:0/4122620400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe6c156bf0 con 0x7efe6c15d890 2024-03-24T00:36:54.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe62a00640 1 -- 172.21.15.40:0/4122620400 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe64012830 con 0x7efe6c15d890 2024-03-24T00:36:54.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6be00640 1 -- 172.21.15.40:0/4122620400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe6c1571b0 con 0x7efe6c15d890 2024-03-24T00:36:54.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.986+0000 7efe6be00640 1 -- 172.21.15.40:0/4122620400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6c09e2e0 con 0x7efe6c15d890 2024-03-24T00:36:54.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.990+0000 7efe62a00640 1 -- 172.21.15.40:0/4122620400 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7efe640129d0 con 0x7efe6c15d890 2024-03-24T00:36:54.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.990+0000 7efe62a00640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe4c067660 0x7efe4c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:54.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.990+0000 7efe62a00640 1 -- 172.21.15.40:0/4122620400 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5115+0+0 (secure 0 0 0) 0x7efe64085dc0 con 0x7efe6c15d890 2024-03-24T00:36:54.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.990+0000 7efe6aa00640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe4c067660 0x7efe4c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:54.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.990+0000 7efe6aa00640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe4c067660 0x7efe4c069b20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7efe5c0011f0 tx=0x7efe5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:54.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:54.994+0000 7efe62a00640 1 -- 172.21.15.40:0/4122620400 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7efe640565a0 con 0x7efe6c15d890 2024-03-24T00:36:55.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:55.214+0000 7efe6be00640 1 -- 172.21.15.40:0/4122620400 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_before"]} v 0) v1 -- 0x7efe6c0b19f0 con 0x7efe6c15d890 2024-03-24T00:36:56.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.122+0000 7efe62a00640 1 -- 172.21.15.40:0/4122620400 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_before"]}]=0 v304) v1 ==== 165+0+0 (secure 0 0 0) 0x7efe6405a640 con 0x7efe6c15d890 2024-03-24T00:36:56.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.126+0000 7efe6be00640 1 -- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe4c067660 msgr2=0x7efe4c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:56.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.126+0000 7efe6be00640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe4c067660 0x7efe4c069b20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7efe5c0011f0 tx=0x7efe5c009040 comp rx=0 tx=0).stop 2024-03-24T00:36:56.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.126+0000 7efe6be00640 1 -- 172.21.15.40:0/4122620400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe6c15d890 msgr2=0x7efe6c150930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:56.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.126+0000 7efe6be00640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe6c15d890 0x7efe6c150930 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7efe6400cc20 tx=0x7efe6400a430 comp rx=0 tx=0).stop 2024-03-24T00:36:56.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.126+0000 7efe6be00640 1 -- 172.21.15.40:0/4122620400 shutdown_connections 2024-03-24T00:36:56.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.126+0000 7efe6be00640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efe4c067660 0x7efe4c069b20 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.126+0000 7efe6be00640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efe6c15d890 0x7efe6c150930 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.126+0000 7efe6be00640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efe6c0cd1a0 0x7efe6c156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.126+0000 7efe6be00640 1 --2- 172.21.15.40:0/4122620400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efe6c072510 0x7efe6c14e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.126+0000 7efe6be00640 1 -- 172.21.15.40:0/4122620400 >> 172.21.15.40:0/4122620400 conn(0x7efe6c0b1c80 msgr2=0x7efe6c15bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:56.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.126+0000 7efe6be00640 1 -- 172.21.15.40:0/4122620400 shutdown_connections 2024-03-24T00:36:56.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.126+0000 7efe6be00640 1 -- 172.21.15.40:0/4122620400 wait complete. 2024-03-24T00:36:56.154 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:36:56.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.298+0000 7f70a5200640 1 Processor -- start 2024-03-24T00:36:56.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.298+0000 7f70a5200640 1 -- start start 2024-03-24T00:36:56.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f70a5200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70a00adaf0 0x7f70a00d13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:56.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f70a5200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70a006eaf0 0x7f70a006ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:56.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f70a5200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a00b4010 con 0x7f70a00aded0 2024-03-24T00:36:56.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f70a5200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a00b0210 con 0x7f70a006eaf0 2024-03-24T00:36:56.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f70a5200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a006fcd0 con 0x7f70a00adaf0 2024-03-24T00:36:56.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f709f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70a00adaf0 0x7f70a00d13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:56.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f709fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70a006eaf0 0x7f70a006ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:56.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f709f400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70a00adaf0 0x7f70a00d13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:45642/0 (socket says 172.21.15.40:45642) 2024-03-24T00:36:56.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f7096a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f70a00aded0 0x7f70a00b3900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:36184/0 (socket says 172.21.15.40:36184) 2024-03-24T00:36:56.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f709f400640 1 -- 172.21.15.40:0/2414344113 learned_addr learned my addr 172.21.15.40:0/2414344113 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:56.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f709fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70a006eaf0 0x7f70a006ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:58464/0 (socket says 172.21.15.40:58464) 2024-03-24T00:36:56.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f709fe00640 1 -- 172.21.15.40:0/2414344113 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70a00adaf0 msgr2=0x7f70a00d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:56.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f709fe00640 1 --2- 172.21.15.40:0/2414344113 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70a00adaf0 0x7f70a00d13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f709fe00640 1 -- 172.21.15.40:0/2414344113 >> v1:172.21.15.53:6789/0 conn(0x7f70a00aded0 legacy=0x7f70a00b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:56.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.302+0000 7f709fe00640 1 -- 172.21.15.40:0/2414344113 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a00aff30 con 0x7f70a006eaf0 2024-03-24T00:36:56.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f709fe00640 1 --2- 172.21.15.40:0/2414344113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70a006eaf0 0x7f70a006ef80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f70980097c0 tx=0x7f709801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f203de006fbc3807 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:56.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f709ea00640 1 -- 172.21.15.40:0/2414344113 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70980ab430 con 0x7f70a006eaf0 2024-03-24T00:36:56.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f709ea00640 1 -- 172.21.15.40:0/2414344113 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70980b0070 con 0x7f70a006eaf0 2024-03-24T00:36:56.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f709ea00640 1 -- 172.21.15.40:0/2414344113 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70980ab7f0 con 0x7f70a006eaf0 2024-03-24T00:36:56.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f70a5200640 1 -- 172.21.15.40:0/2414344113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70a006eaf0 msgr2=0x7f70a006ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:56.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f70a5200640 1 --2- 172.21.15.40:0/2414344113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70a006eaf0 0x7f70a006ef80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f70980097c0 tx=0x7f709801c410 comp rx=0 tx=0).stop 2024-03-24T00:36:56.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f70a5200640 1 -- 172.21.15.40:0/2414344113 shutdown_connections 2024-03-24T00:36:56.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f70a5200640 1 --2- 172.21.15.40:0/2414344113 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70a006eaf0 0x7f70a006ef80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f70a5200640 1 --2- 172.21.15.40:0/2414344113 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70a00adaf0 0x7f70a00d13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f70a5200640 1 -- 172.21.15.40:0/2414344113 >> 172.21.15.40:0/2414344113 conn(0x7f70a00d08f0 msgr2=0x7f70a00d0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:56.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f70a5200640 1 -- 172.21.15.40:0/2414344113 shutdown_connections 2024-03-24T00:36:56.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f70a5200640 1 -- 172.21.15.40:0/2414344113 wait complete. 2024-03-24T00:36:56.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f70a5200640 1 Processor -- start 2024-03-24T00:36:56.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.306+0000 7f70a5200640 1 -- start start 2024-03-24T00:36:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f70a5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70a006eaf0 0x7f70a014ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f70a5200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70a00adaf0 0x7f70a014e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f70a5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70a00aded0 0x7f70a014bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f70a5200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a006fab0 con 0x7f70a006eaf0 2024-03-24T00:36:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f70a5200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a006f930 con 0x7f70a00aded0 2024-03-24T00:36:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f70a5200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a00afb50 con 0x7f70a00adaf0 2024-03-24T00:36:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f709f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70a006eaf0 0x7f70a014ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f709f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70a006eaf0 0x7f70a014ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36898/0 (socket says 172.21.15.40:36898) 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f709f400640 1 -- 172.21.15.40:0/3836700288 learned_addr learned my addr 172.21.15.40:0/3836700288 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f709fe00640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70a00aded0 0x7f70a014bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f7096a00640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70a00adaf0 0x7f70a014e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f709f400640 1 -- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70a00adaf0 msgr2=0x7f70a014e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f709f400640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70a00adaf0 0x7f70a014e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f709f400640 1 -- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70a00aded0 msgr2=0x7f70a014bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f709f400640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70a00aded0 0x7f70a014bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f709f400640 1 -- 172.21.15.40:0/3836700288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a015f8d0 con 0x7f70a006eaf0 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f7096a00640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70a00adaf0 0x7f70a014e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f709f400640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70a006eaf0 0x7f70a014ddd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f7090009cf0 tx=0x7f709000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f7097e00640 1 -- 172.21.15.40:0/3836700288 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709000cda0 con 0x7f70a006eaf0 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f7097e00640 1 -- 172.21.15.40:0/3836700288 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7090019070 con 0x7f70a006eaf0 2024-03-24T00:36:56.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f7097e00640 1 -- 172.21.15.40:0/3836700288 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70900146a0 con 0x7f70a006eaf0 2024-03-24T00:36:56.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f70a5200640 1 -- 172.21.15.40:0/3836700288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a015fbb0 con 0x7f70a006eaf0 2024-03-24T00:36:56.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f70a5200640 1 -- 172.21.15.40:0/3836700288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a01537a0 con 0x7f70a006eaf0 2024-03-24T00:36:56.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f70a5200640 1 -- 172.21.15.40:0/3836700288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a004f410 con 0x7f70a006eaf0 2024-03-24T00:36:56.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.310+0000 7f7097e00640 1 -- 172.21.15.40:0/3836700288 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7090012050 con 0x7f70a006eaf0 2024-03-24T00:36:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.314+0000 7f7097e00640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f707c067690 0x7f707c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.314+0000 7f7097e00640 1 -- 172.21.15.40:0/3836700288 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f7090085ba0 con 0x7f70a006eaf0 2024-03-24T00:36:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.314+0000 7f7096a00640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f707c067690 0x7f707c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.314+0000 7f7096a00640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f707c067690 0x7f707c069b50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f70a00b0b30 tx=0x7f7088009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.318+0000 7f7097e00640 1 -- 172.21.15.40:0/3836700288 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7090010040 con 0x7f70a006eaf0 2024-03-24T00:36:56.482 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.478+0000 7f70a5200640 1 -- 172.21.15.40:0/3836700288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f70a0072aa0 con 0x7f70a006eaf0 2024-03-24T00:36:56.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.482+0000 7f7097e00640 1 -- 172.21.15.40:0/3836700288 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v51)=0 v51) v1 ==== 122+0+0 (secure 0 0 0) 0x7f70900569d0 con 0x7f70a006eaf0 2024-03-24T00:36:56.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.486+0000 7f70a5200640 1 -- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f707c067690 msgr2=0x7f707c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:56.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.486+0000 7f70a5200640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f707c067690 0x7f707c069b50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f70a00b0b30 tx=0x7f7088009040 comp rx=0 tx=0).stop 2024-03-24T00:36:56.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.486+0000 7f70a5200640 1 -- 172.21.15.40:0/3836700288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70a006eaf0 msgr2=0x7f70a014ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:56.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.486+0000 7f70a5200640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70a006eaf0 0x7f70a014ddd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f7090009cf0 tx=0x7f709000c490 comp rx=0 tx=0).stop 2024-03-24T00:36:56.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.486+0000 7f70a5200640 1 -- 172.21.15.40:0/3836700288 shutdown_connections 2024-03-24T00:36:56.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.486+0000 7f70a5200640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f707c067690 0x7f707c069b50 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.486+0000 7f70a5200640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70a00aded0 0x7f70a014bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.486+0000 7f70a5200640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70a00adaf0 0x7f70a014e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.486+0000 7f70a5200640 1 --2- 172.21.15.40:0/3836700288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70a006eaf0 0x7f70a014ddd0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.486+0000 7f70a5200640 1 -- 172.21.15.40:0/3836700288 >> 172.21.15.40:0/3836700288 conn(0x7f70a00d08f0 msgr2=0x7f70a0066e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:56.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.486+0000 7f70a5200640 1 -- 172.21.15.40:0/3836700288 shutdown_connections 2024-03-24T00:36:56.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.486+0000 7f70a5200640 1 -- 172.21.15.40:0/3836700288 wait complete. 2024-03-24T00:36:56.508 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:36:56.508 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:36:56.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:36:56.702 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:36:56.702 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:36:56.703 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:36:56.711 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:36:56.711 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:36:56.711 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:36:56.720 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:36:56.721 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:36:56.721 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:36:56.763 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:36:56.763 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:36:56.763 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:36:56.775 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:36:56.775 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:36:56.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:36:56.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.942+0000 7f8357200640 1 Processor -- start 2024-03-24T00:36:56.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.942+0000 7f8357200640 1 -- start start 2024-03-24T00:36:56.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f8357200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f835006ec20 0x7f83500b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:56.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f8357200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f835006d510 0x7f835006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f8357200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8350070d20 con 0x7f835006e840 2024-03-24T00:36:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f8357200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8350071f50 con 0x7f835006ec20 2024-03-24T00:36:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f8357200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f835015b820 con 0x7f835006d510 2024-03-24T00:36:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f834fe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f835006e840 0x7f8350071840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:57328/0 (socket says 172.21.15.40:57328) 2024-03-24T00:36:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f834fe00640 1 -- 172.21.15.40:0/4161655088 learned_addr learned my addr 172.21.15.40:0/4161655088 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:56.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f8347e00640 1 --2- 172.21.15.40:0/4161655088 >> v2:172.21.15.40:3300/0 conn(0x7f835006ec20 0x7f83500b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f8355200640 1 --2- 172.21.15.40:0/4161655088 >> v2:172.21.15.53:3300/0 conn(0x7f835006d510 0x7f835006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f834f400640 1 -- 172.21.15.40:0/4161655088 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050738669 0 0) 0x7f8350070d20 con 0x7f835006e840 2024-03-24T00:36:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f834f400640 1 -- 172.21.15.40:0/4161655088 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8334003530 con 0x7f835006e840 2024-03-24T00:36:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f8347e00640 1 -- 172.21.15.40:0/4161655088 >> v2:172.21.15.53:3300/0 conn(0x7f835006d510 msgr2=0x7f835006d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f8347e00640 1 --2- 172.21.15.40:0/4161655088 >> v2:172.21.15.53:3300/0 conn(0x7f835006d510 0x7f835006d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f8347e00640 1 -- 172.21.15.40:0/4161655088 >> v1:172.21.15.40:6789/0 conn(0x7f835006e840 legacy=0x7f8350071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:56.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.946+0000 7f8347e00640 1 -- 172.21.15.40:0/4161655088 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8350070fc0 con 0x7f835006ec20 2024-03-24T00:36:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f8347e00640 1 --2- 172.21.15.40:0/4161655088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835006ec20 0x7f83500b4a00 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f83500b4f40 tx=0x7f833801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf229e689714a8aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f834f400640 1 -- 172.21.15.40:0/4161655088 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83380b0070 con 0x7f835006ec20 2024-03-24T00:36:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f834f400640 1 -- 172.21.15.40:0/4161655088 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83380ab400 con 0x7f835006ec20 2024-03-24T00:36:56.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f834f400640 1 -- 172.21.15.40:0/4161655088 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83380ab7d0 con 0x7f835006ec20 2024-03-24T00:36:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f8357200640 1 -- 172.21.15.40:0/4161655088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835006ec20 msgr2=0x7f83500b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f8357200640 1 --2- 172.21.15.40:0/4161655088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835006ec20 0x7f83500b4a00 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f83500b4f40 tx=0x7f833801c360 comp rx=0 tx=0).stop 2024-03-24T00:36:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f8357200640 1 -- 172.21.15.40:0/4161655088 shutdown_connections 2024-03-24T00:36:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f8357200640 1 --2- 172.21.15.40:0/4161655088 >> v2:172.21.15.53:3300/0 conn(0x7f835006d510 0x7f835006d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f8357200640 1 --2- 172.21.15.40:0/4161655088 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835006ec20 0x7f83500b4a00 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f8357200640 1 -- 172.21.15.40:0/4161655088 >> 172.21.15.40:0/4161655088 conn(0x7f83500b17b0 msgr2=0x7f83500b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f8357200640 1 -- 172.21.15.40:0/4161655088 shutdown_connections 2024-03-24T00:36:56.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f8357200640 1 -- 172.21.15.40:0/4161655088 wait complete. 2024-03-24T00:36:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f8357200640 1 Processor -- start 2024-03-24T00:36:56.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.950+0000 7f8357200640 1 -- start start 2024-03-24T00:36:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8357200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f835006d510 0x7f835013aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8357200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835006e840 0x7f835013a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8357200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835006ec20 0x7f835014bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8357200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835015e3e0 con 0x7f835006ec20 2024-03-24T00:36:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8357200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835015e260 con 0x7f835006d510 2024-03-24T00:36:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8357200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835015e560 con 0x7f835006e840 2024-03-24T00:36:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8355200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835006ec20 0x7f835014bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8355200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835006ec20 0x7f835014bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:36918/0 (socket says 172.21.15.40:36918) 2024-03-24T00:36:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8355200640 1 -- 172.21.15.40:0/3804873297 learned_addr learned my addr 172.21.15.40:0/3804873297 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8347e00640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835006e840 0x7f835013a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:56.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f834fe00640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f835006d510 0x7f835013aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8355200640 1 -- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835006e840 msgr2=0x7f835013a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8355200640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835006e840 0x7f835013a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8355200640 1 -- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f835006d510 msgr2=0x7f835013aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8355200640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f835006d510 0x7f835013aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8355200640 1 -- 172.21.15.40:0/3804873297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835014c6b0 con 0x7f835006ec20 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8347e00640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835006e840 0x7f835013a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f834fe00640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f835006d510 0x7f835013aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8355200640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835006ec20 0x7f835014bd40 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f8348007e70 tx=0x7f834800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f834cc00640 1 -- 172.21.15.40:0/3804873297 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348014620 con 0x7f835006ec20 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f834cc00640 1 -- 172.21.15.40:0/3804873297 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8348019070 con 0x7f835006ec20 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8357200640 1 -- 172.21.15.40:0/3804873297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f835014eda0 con 0x7f835006ec20 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f834cc00640 1 -- 172.21.15.40:0/3804873297 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8348014a20 con 0x7f835006ec20 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8357200640 1 -- 172.21.15.40:0/3804873297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835014f360 con 0x7f835006ec20 2024-03-24T00:36:56.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f8357200640 1 -- 172.21.15.40:0/3804873297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8350068170 con 0x7f835006ec20 2024-03-24T00:36:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.954+0000 7f834cc00640 1 -- 172.21.15.40:0/3804873297 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8348014bc0 con 0x7f835006ec20 2024-03-24T00:36:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.958+0000 7f834cc00640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f832c067660 0x7f832c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.958+0000 7f834cc00640 1 -- 172.21.15.40:0/3804873297 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f8348085f70 con 0x7f835006ec20 2024-03-24T00:36:56.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.958+0000 7f834fe00640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f832c067660 0x7f832c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.958+0000 7f834fe00640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f832c067660 0x7f832c069b20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8340005890 tx=0x7f8340009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:56.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:56.962+0000 7f834cc00640 1 -- 172.21.15.40:0/3804873297 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8348056e30 con 0x7f835006ec20 2024-03-24T00:36:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.154+0000 7f8357200640 1 -- 172.21.15.40:0/3804873297 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83500b2fe0 con 0x7f835006ec20 2024-03-24T00:36:57.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f834cc00640 1 -- 172.21.15.40:0/3804873297 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f8348056730 con 0x7f835006ec20 2024-03-24T00:36:57.160 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:57.160 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":90,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6395,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1097448819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1097448819},{"type":"v1","addr":"172.21.15.40:6834","nonce":1097448819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76},{"gid":6683,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1461851569","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1461851569},{"type":"v1","addr":"172.21.15.40:6835","nonce":1461851569}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":6686,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2012638914","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2012638914},{"type":"v1","addr":"172.21.15.53:6837","nonce":2012638914}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90},{"gid":7534,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/132099954","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":132099954},{"type":"v1","addr":"172.21.15.53:6835","nonce":132099954}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":90}],"filesystems":[]} 2024-03-24T00:36:57.160 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 90 2024-03-24T00:36:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f8357200640 1 -- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f832c067660 msgr2=0x7f832c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f8357200640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f832c067660 0x7f832c069b20 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8340005890 tx=0x7f8340009040 comp rx=0 tx=0).stop 2024-03-24T00:36:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f8357200640 1 -- 172.21.15.40:0/3804873297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835006ec20 msgr2=0x7f835014bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f8357200640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835006ec20 0x7f835014bd40 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f8348007e70 tx=0x7f834800e4b0 comp rx=0 tx=0).stop 2024-03-24T00:36:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f8357200640 1 -- 172.21.15.40:0/3804873297 shutdown_connections 2024-03-24T00:36:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f8357200640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f832c067660 0x7f832c069b20 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f8357200640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f835006ec20 0x7f835014bd40 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f8357200640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f835006e840 0x7f835013a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:57.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f8357200640 1 --2- 172.21.15.40:0/3804873297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f835006d510 0x7f835013aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f8357200640 1 -- 172.21.15.40:0/3804873297 >> 172.21.15.40:0/3804873297 conn(0x7f83500b17b0 msgr2=0x7f835015c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f8357200640 1 -- 172.21.15.40:0/3804873297 shutdown_connections 2024-03-24T00:36:57.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.158+0000 7f8357200640 1 -- 172.21.15.40:0/3804873297 wait complete. 2024-03-24T00:36:57.181 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:36:57.181 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:36:57.181 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:36:57.181 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:36:57.182 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:36:57.182 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:36:57.182 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:36:57.182 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:36:57.182 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:36:57.182 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:36:57.182 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:36:57.183 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:36:57.183 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:36:57.183 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:36:57.183 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:36:57.183 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:36:57.193 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:36:57.194 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:36:57.195 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:36:57.195 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:36:57.197 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:36:57.197 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:36:57.198 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:36:57.198 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:36:57.241 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:36:57.242 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:57.243 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:36:57.244 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:57.246 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:36:57.246 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:57.248 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:36:57.248 DEBUG:teuthology.parallel:result is None 2024-03-24T00:36:57.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:36:57.328 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:36:57.329 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:36:57.340 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:36:57.341 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:36:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.390+0000 7f5741c00640 1 Processor -- start 2024-03-24T00:36:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.390+0000 7f5741c00640 1 -- start start 2024-03-24T00:36:57.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f5741c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f573c141bc0 0x7f573c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f5741c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c070e00 con 0x7f573c0ce630 2024-03-24T00:36:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f5741c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573c0cee20 con 0x7f573c13ec80 2024-03-24T00:36:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f5741c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573c0cf100 con 0x7f573c141bc0 2024-03-24T00:36:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f573b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f573c0ce630 0x7f573c071300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44760/0 (socket says 172.21.15.40:44760) 2024-03-24T00:36:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f573b400640 1 -- 172.21.15.40:0/1635857883 learned_addr learned my addr 172.21.15.40:0/1635857883 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:57.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f5732000640 1 --2- 172.21.15.40:0/1635857883 >> v2:172.21.15.53:3300/0 conn(0x7f573c141bc0 0x7f573c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f573a000640 1 -- 172.21.15.40:0/1635857883 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578105111 0 0) 0x7f573c070e00 con 0x7f573c0ce630 2024-03-24T00:36:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f573a000640 1 -- 172.21.15.40:0/1635857883 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5720003530 con 0x7f573c0ce630 2024-03-24T00:36:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f5732000640 1 -- 172.21.15.40:0/1635857883 >> v1:172.21.15.53:6789/0 conn(0x7f573c13ec80 legacy=0x7f573c1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f5732000640 1 -- 172.21.15.40:0/1635857883 >> v1:172.21.15.40:6789/0 conn(0x7f573c0ce630 legacy=0x7f573c071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f5732000640 1 -- 172.21.15.40:0/1635857883 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c0cf3e0 con 0x7f573c141bc0 2024-03-24T00:36:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.394+0000 7f5732000640 1 --2- 172.21.15.40:0/1635857883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f573c141bc0 0x7f573c0ce0f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f57240097c0 tx=0x7f572401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d7d3ec53d53a5e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f573a000640 1 -- 172.21.15.40:0/1635857883 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57240aa620 con 0x7f573c141bc0 2024-03-24T00:36:57.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f573a000640 1 -- 172.21.15.40:0/1635857883 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57240af070 con 0x7f573c141bc0 2024-03-24T00:36:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f573a000640 1 -- 172.21.15.40:0/1635857883 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57240aa9e0 con 0x7f573c141bc0 2024-03-24T00:36:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 -- 172.21.15.40:0/1635857883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f573c141bc0 msgr2=0x7f573c0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 --2- 172.21.15.40:0/1635857883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f573c141bc0 0x7f573c0ce0f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f57240097c0 tx=0x7f572401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:36:57.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 -- 172.21.15.40:0/1635857883 shutdown_connections 2024-03-24T00:36:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 --2- 172.21.15.40:0/1635857883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f573c141bc0 0x7f573c0ce0f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 -- 172.21.15.40:0/1635857883 >> 172.21.15.40:0/1635857883 conn(0x7f573c0b17b0 msgr2=0x7f573c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 -- 172.21.15.40:0/1635857883 shutdown_connections 2024-03-24T00:36:57.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 -- 172.21.15.40:0/1635857883 wait complete. 2024-03-24T00:36:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 Processor -- start 2024-03-24T00:36:57.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 -- start start 2024-03-24T00:36:57.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f573c0ce630 0x7f573c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f573c13ec80 0x7f573c154ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f573c141bc0 0x7f573c155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c0725f0 con 0x7f573c13ec80 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c072470 con 0x7f573c141bc0 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5741c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c169d50 con 0x7f573c0ce630 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5732000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f573c13ec80 0x7f573c154ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5732000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f573c13ec80 0x7f573c154ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39872/0 (socket says 172.21.15.40:39872) 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f5732000640 1 -- 172.21.15.40:0/1036556014 learned_addr learned my addr 172.21.15.40:0/1036556014 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f573aa00640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f573c0ce630 0x7f573c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.398+0000 7f573b400640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f573c141bc0 0x7f573c155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5732000640 1 -- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f573c0ce630 msgr2=0x7f573c06e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5732000640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f573c0ce630 0x7f573c06e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5732000640 1 -- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f573c141bc0 msgr2=0x7f573c155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5732000640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f573c141bc0 0x7f573c155430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:57.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5732000640 1 -- 172.21.15.40:0/1036556014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c0ae5c0 con 0x7f573c13ec80 2024-03-24T00:36:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f573aa00640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f573c0ce630 0x7f573c06e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:36:57.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5732000640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f573c13ec80 0x7f573c154ef0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f57240190b0 tx=0x7f5724019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5733400640 1 -- 172.21.15.40:0/1036556014 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57240aac10 con 0x7f573c13ec80 2024-03-24T00:36:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5733400640 1 -- 172.21.15.40:0/1036556014 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57240aadd0 con 0x7f573c13ec80 2024-03-24T00:36:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5733400640 1 -- 172.21.15.40:0/1036556014 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57240be690 con 0x7f573c13ec80 2024-03-24T00:36:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5741c00640 1 -- 172.21.15.40:0/1036556014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c0ae8a0 con 0x7f573c13ec80 2024-03-24T00:36:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5741c00640 1 -- 172.21.15.40:0/1036556014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c06eda0 con 0x7f573c13ec80 2024-03-24T00:36:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5741c00640 1 -- 172.21.15.40:0/1036556014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c068170 con 0x7f573c13ec80 2024-03-24T00:36:57.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5733400640 1 -- 172.21.15.40:0/1036556014 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f57240b33f0 con 0x7f573c13ec80 2024-03-24T00:36:57.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5733400640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5718067660 0x7f5718069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:57.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.402+0000 7f5733400640 1 -- 172.21.15.40:0/1036556014 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f572411f970 con 0x7f573c13ec80 2024-03-24T00:36:57.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.406+0000 7f573aa00640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5718067660 0x7f5718069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:57.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.406+0000 7f573aa00640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5718067660 0x7f5718069b20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f573c1fa050 tx=0x7f572c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:57.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.406+0000 7f5733400640 1 -- 172.21.15.40:0/1036556014 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5724016090 con 0x7f573c13ec80 2024-03-24T00:36:57.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:57.582+0000 7f5741c00640 1 -- 172.21.15.40:0/1036556014 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f573c136d10 con 0x7f573c13ec80 2024-03-24T00:36:58.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5733400640 1 -- 172.21.15.40:0/1036556014 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v89) v1 ==== 97+0+0 (secure 0 0 0) 0x7f57240f0300 con 0x7f573c13ec80 2024-03-24T00:36:58.140 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-03-24T00:36:58.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5741c00640 1 -- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5718067660 msgr2=0x7f5718069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5741c00640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5718067660 0x7f5718069b20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f573c1fa050 tx=0x7f572c009040 comp rx=0 tx=0).stop 2024-03-24T00:36:58.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5741c00640 1 -- 172.21.15.40:0/1036556014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f573c13ec80 msgr2=0x7f573c154ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5741c00640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f573c13ec80 0x7f573c154ef0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f57240190b0 tx=0x7f5724019240 comp rx=0 tx=0).stop 2024-03-24T00:36:58.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5741c00640 1 -- 172.21.15.40:0/1036556014 shutdown_connections 2024-03-24T00:36:58.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5741c00640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5718067660 0x7f5718069b20 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5741c00640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f573c141bc0 0x7f573c155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5741c00640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f573c13ec80 0x7f573c154ef0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5741c00640 1 --2- 172.21.15.40:0/1036556014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f573c0ce630 0x7f573c06e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5741c00640 1 -- 172.21.15.40:0/1036556014 >> 172.21.15.40:0/1036556014 conn(0x7f573c0b17b0 msgr2=0x7f573c0b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:58.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5741c00640 1 -- 172.21.15.40:0/1036556014 shutdown_connections 2024-03-24T00:36:58.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.138+0000 7f5741c00640 1 -- 172.21.15.40:0/1036556014 wait complete. 2024-03-24T00:36:58.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:36:58.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.306+0000 7f1e10c00640 1 Processor -- start 2024-03-24T00:36:58.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.306+0000 7f1e10c00640 1 -- start start 2024-03-24T00:36:58.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e10c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e0c13ec80 0x7f1e0c1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e10c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0c0ce800 0x7f1e0c071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e10c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0c070e00 con 0x7f1e0c1419f0 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e10c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e0c141e50 con 0x7f1e0c13ec80 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e10c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c0cee20 con 0x7f1e0c0ce800 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e09e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e0c13ec80 0x7f1e0c1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e09e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1e0c13ec80 0x7f1e0c1414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39884/0 (socket says 172.21.15.40:39884) 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e01400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f1e0c1419f0 0x7f1e0c0ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44770/0 (socket says 172.21.15.40:44770) 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e09e00640 1 -- 172.21.15.40:0/75626429 learned_addr learned my addr 172.21.15.40:0/75626429 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e0a800640 1 --2- 172.21.15.40:0/75626429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0c0ce800 0x7f1e0c071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e09e00640 1 -- 172.21.15.40:0/75626429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0c0ce800 msgr2=0x7f1e0c071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e09e00640 1 --2- 172.21.15.40:0/75626429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0c0ce800 0x7f1e0c071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e09e00640 1 -- 172.21.15.40:0/75626429 >> v1:172.21.15.40:6789/0 conn(0x7f1e0c1419f0 legacy=0x7f1e0c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.310+0000 7f1e09e00640 1 -- 172.21.15.40:0/75626429 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c0cf100 con 0x7f1e0c13ec80 2024-03-24T00:36:58.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e09e00640 1 --2- 172.21.15.40:0/75626429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c13ec80 0x7f1e0c1414b0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009870 tx=0x7f1dfc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e25c26f8969e4f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:58.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e09400640 1 -- 172.21.15.40:0/75626429 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0a9640 con 0x7f1e0c13ec80 2024-03-24T00:36:58.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e09400640 1 -- 172.21.15.40:0/75626429 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc0a97e0 con 0x7f1e0c13ec80 2024-03-24T00:36:58.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e09400640 1 -- 172.21.15.40:0/75626429 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0a9bb0 con 0x7f1e0c13ec80 2024-03-24T00:36:58.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 -- 172.21.15.40:0/75626429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c13ec80 msgr2=0x7f1e0c1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 --2- 172.21.15.40:0/75626429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c13ec80 0x7f1e0c1414b0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009870 tx=0x7f1dfc01c560 comp rx=0 tx=0).stop 2024-03-24T00:36:58.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 -- 172.21.15.40:0/75626429 shutdown_connections 2024-03-24T00:36:58.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 --2- 172.21.15.40:0/75626429 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0c0ce800 0x7f1e0c071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 --2- 172.21.15.40:0/75626429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c13ec80 0x7f1e0c1414b0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 -- 172.21.15.40:0/75626429 >> 172.21.15.40:0/75626429 conn(0x7f1e0c0b17b0 msgr2=0x7f1e0c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:58.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 -- 172.21.15.40:0/75626429 shutdown_connections 2024-03-24T00:36:58.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 -- 172.21.15.40:0/75626429 wait complete. 2024-03-24T00:36:58.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 Processor -- start 2024-03-24T00:36:58.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 -- start start 2024-03-24T00:36:58.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c0ce800 0x7f1e0c15ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0c13ec80 0x7f1e0c14e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e0c1419f0 0x7f1e0c1580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c072460 con 0x7f1e0c0ce800 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c0722e0 con 0x7f1e0c1419f0 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.314+0000 7f1e10c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c0725e0 con 0x7f1e0c13ec80 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e09e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c0ce800 0x7f1e0c15ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e09e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c0ce800 0x7f1e0c15ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39892/0 (socket says 172.21.15.40:39892) 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e09e00640 1 -- 172.21.15.40:0/1588695870 learned_addr learned my addr 172.21.15.40:0/1588695870 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e0a800640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e0c1419f0 0x7f1e0c1580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e09e00640 1 -- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0c13ec80 msgr2=0x7f1e0c14e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e01400640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0c13ec80 0x7f1e0c14e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e09e00640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0c13ec80 0x7f1e0c14e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e09e00640 1 -- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e0c1419f0 msgr2=0x7f1e0c1580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e09e00640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e0c1419f0 0x7f1e0c1580d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e09e00640 1 -- 172.21.15.40:0/1588695870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c15f280 con 0x7f1e0c0ce800 2024-03-24T00:36:58.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e0a800640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e0c1419f0 0x7f1e0c1580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:36:58.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e01400640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0c13ec80 0x7f1e0c14e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:36:58.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e09e00640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c0ce800 0x7f1e0c15ed40 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc01a190 tx=0x7f1dfc019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:58.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e02a00640 1 -- 172.21.15.40:0/1588695870 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0b1800 con 0x7f1e0c0ce800 2024-03-24T00:36:58.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e02a00640 1 -- 172.21.15.40:0/1588695870 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc0b19a0 con 0x7f1e0c0ce800 2024-03-24T00:36:58.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e10c00640 1 -- 172.21.15.40:0/1588695870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0c156040 con 0x7f1e0c0ce800 2024-03-24T00:36:58.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e02a00640 1 -- 172.21.15.40:0/1588695870 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0b1d90 con 0x7f1e0c0ce800 2024-03-24T00:36:58.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e10c00640 1 -- 172.21.15.40:0/1588695870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0c1557a0 con 0x7f1e0c0ce800 2024-03-24T00:36:58.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e10c00640 1 -- 172.21.15.40:0/1588695870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e0c068170 con 0x7f1e0c0ce800 2024-03-24T00:36:58.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e02a00640 1 -- 172.21.15.40:0/1588695870 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f1dfc0ba480 con 0x7f1e0c0ce800 2024-03-24T00:36:58.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.318+0000 7f1e02a00640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1de8067660 0x7f1de8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.322+0000 7f1e02a00640 1 -- 172.21.15.40:0/1588695870 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1dfc11fe10 con 0x7f1e0c0ce800 2024-03-24T00:36:58.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.322+0000 7f1e01400640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1de8067660 0x7f1de8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.322+0000 7f1e01400640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1de8067660 0x7f1de8069b20 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1df4007fd0 tx=0x7f1df4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:58.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.326+0000 7f1e02a00640 1 -- 172.21.15.40:0/1588695870 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f1dfc123030 con 0x7f1e0c0ce800 2024-03-24T00:36:58.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.546+0000 7f1e10c00640 1 -- 172.21.15.40:0/1588695870 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1e0c072b90 con 0x7f1e0c0ce800 2024-03-24T00:36:58.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.546+0000 7f1e02a00640 1 -- 172.21.15.40:0/1588695870 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f1dfc0ad020 con 0x7f1e0c0ce800 2024-03-24T00:36:58.550 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:58.550 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:36:58.550 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-03-24T00:36:58.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lCkU6EhAA0q21F90epsGPK5x17s9NLw==", 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llkfOHhAAqKKFyEJzG0UxbLJMFPET8A==", 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.551 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lKMmwGRAAAKaOSnhTRbHIpPKXXkAe+Q==", 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l7quKJhAA8QsquiFFmj5H1G5yck0FpA==", 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAVdP9lVk3uBxAA9TC1OpeB5GIFlfJl5+ILBg==", 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.553 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYdP9lUc1IDhAAPwCV9eqL7bFibkfMFt9wPw==", 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAbdP9lwjyuFBAAJ1JAIAnLXAn03+lQPwwY9Q==", 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAedP9lZ1bgGRAAuLAay48N7jXjj8/yaYwSqg==", 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.555 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAidP9lvyk4JhAA8w4/XBYHDLoqJ1zop9sUiw==", 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAldP9lcb9qLBAAXts6VU/4kVqUWS/WORV7zQ==", 2024-03-24T00:36:58.556 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAodP9l8ipTMhAAjpagojh+woHyGvlQlIkHDw==", 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQArdP9lmGfpORAAPvow5nF/O7RLmuXEqNWENg==", 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw==", 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.558 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:58.559 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow r", 2024-03-24T00:36:58.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:36:58.559 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.559 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-03-24T00:36:58.559 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lpIZrNBAAzpGWZuQ6ndjNMAOmAsebvQ==", 2024-03-24T00:36:58.559 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.559 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:36:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:36:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:36:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-03-24T00:36:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l6gXMMBAAd6Oe6S73hj/WfLXTsYiEZw==", 2024-03-24T00:36:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:36:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:36:58.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:36:58.561 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.561 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-03-24T00:36:58.561 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llFHCOhAATtv8gcL6BHPnxB1dHrg6vw==", 2024-03-24T00:36:58.561 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.561 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:36:58.561 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:36:58.561 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:36:58.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAQdP9lmmd7KxAA4uk6aUKP6egYj6O1NgdadQ==", 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.562 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lebNrBxAAvbcCAJhDQn/5Uk1dJ01oHw==", 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lEnZyAxAAW5Lrm7LZPTVfLCOY6As1PQ==", 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:36:58.563 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:36:58.564 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.564 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:58.564 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-24T00:36:58.564 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:36:58.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.550+0000 7f1e10c00640 1 -- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1de8067660 msgr2=0x7f1de8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.550+0000 7f1e10c00640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1de8067660 0x7f1de8069b20 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1df4007fd0 tx=0x7f1df4009040 comp rx=0 tx=0).stop 2024-03-24T00:36:58.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.550+0000 7f1e10c00640 1 -- 172.21.15.40:0/1588695870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c0ce800 msgr2=0x7f1e0c15ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.550+0000 7f1e10c00640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c0ce800 0x7f1e0c15ed40 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc01a190 tx=0x7f1dfc019300 comp rx=0 tx=0).stop 2024-03-24T00:36:58.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.550+0000 7f1e10c00640 1 -- 172.21.15.40:0/1588695870 shutdown_connections 2024-03-24T00:36:58.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.550+0000 7f1e10c00640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1de8067660 0x7f1de8069b20 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.550+0000 7f1e10c00640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e0c1419f0 0x7f1e0c1580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.550+0000 7f1e10c00640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1e0c13ec80 0x7f1e0c14e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.550+0000 7f1e10c00640 1 --2- 172.21.15.40:0/1588695870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1e0c0ce800 0x7f1e0c15ed40 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.550+0000 7f1e10c00640 1 -- 172.21.15.40:0/1588695870 >> 172.21.15.40:0/1588695870 conn(0x7f1e0c0b17b0 msgr2=0x7f1e0c159fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:58.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.550+0000 7f1e10c00640 1 -- 172.21.15.40:0/1588695870 shutdown_connections 2024-03-24T00:36:58.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.550+0000 7f1e10c00640 1 -- 172.21.15.40:0/1588695870 wait complete. 2024-03-24T00:36:58.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcfe400640 1 Processor -- start 2024-03-24T00:36:58.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcfe400640 1 -- start start 2024-03-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcfe400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbcf80b5be0 0x7fbcf815f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcfe400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf8072c10 con 0x7fbcf80725a0 2024-03-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcfe400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf8072ef0 con 0x7fbcf81477a0 2024-03-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcfe400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf815ebc0 con 0x7fbcf80b5be0 2024-03-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcf6a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbcf80725a0 0x7fbcf80b54d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44774/0 (socket says 172.21.15.40:44774) 2024-03-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcf6a00640 1 -- 172.21.15.40:0/1463614103 learned_addr learned my addr 172.21.15.40:0/1463614103 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcf7e00640 1 --2- 172.21.15.40:0/1463614103 >> v2:172.21.15.40:3300/0 conn(0x7fbcf80b5be0 0x7fbcf815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcf6000640 1 -- 172.21.15.40:0/1463614103 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375002389 0 0) 0x7fbcf8072c10 con 0x7fbcf80725a0 2024-03-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcf6000640 1 -- 172.21.15.40:0/1463614103 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcdc003530 con 0x7fbcf80725a0 2024-03-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcf6000640 1 -- 172.21.15.40:0/1463614103 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172613253 0 0) 0x7fbcf8072ef0 con 0x7fbcf81477a0 2024-03-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcf7e00640 1 -- 172.21.15.40:0/1463614103 >> v1:172.21.15.53:6789/0 conn(0x7fbcf81477a0 legacy=0x7fbcf806f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcf7e00640 1 -- 172.21.15.40:0/1463614103 >> v1:172.21.15.40:6789/0 conn(0x7fbcf80725a0 legacy=0x7fbcf80b54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.710+0000 7fbcf7e00640 1 -- 172.21.15.40:0/1463614103 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf815e850 con 0x7fbcf80b5be0 2024-03-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.714+0000 7fbcf7e00640 1 --2- 172.21.15.40:0/1463614103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcf80b5be0 0x7fbcf815f050 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0009d90 tx=0x7fbcf001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d22d28b1dd7046 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.714+0000 7fbcf6000640 1 -- 172.21.15.40:0/1463614103 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf00b2070 con 0x7fbcf80b5be0 2024-03-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.714+0000 7fbcf6000640 1 -- 172.21.15.40:0/1463614103 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf00a57a0 con 0x7fbcf80b5be0 2024-03-24T00:36:58.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.714+0000 7fbcf6000640 1 -- 172.21.15.40:0/1463614103 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf00a5b70 con 0x7fbcf80b5be0 2024-03-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.714+0000 7fbcfe400640 1 -- 172.21.15.40:0/1463614103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcf80b5be0 msgr2=0x7fbcf815f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.714+0000 7fbcfe400640 1 --2- 172.21.15.40:0/1463614103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcf80b5be0 0x7fbcf815f050 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0009d90 tx=0x7fbcf001c8e0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.714+0000 7fbcfe400640 1 -- 172.21.15.40:0/1463614103 shutdown_connections 2024-03-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.714+0000 7fbcfe400640 1 --2- 172.21.15.40:0/1463614103 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcf80b5be0 0x7fbcf815f050 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.714+0000 7fbcfe400640 1 -- 172.21.15.40:0/1463614103 >> 172.21.15.40:0/1463614103 conn(0x7fbcf806b150 msgr2=0x7fbcf806b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.714+0000 7fbcfe400640 1 -- 172.21.15.40:0/1463614103 shutdown_connections 2024-03-24T00:36:58.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.714+0000 7fbcfe400640 1 -- 172.21.15.40:0/1463614103 wait complete. 2024-03-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcfe400640 1 Processor -- start 2024-03-24T00:36:58.719 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcfe400640 1 -- start start 2024-03-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcfe400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcf80725a0 0x7fbcf80aeff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcfe400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcf80b5be0 0x7fbcf813e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcfe400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf81477a0 0x7fbcf80b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcfe400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf80d1490 con 0x7fbcf80b5be0 2024-03-24T00:36:58.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcfe400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf80d1310 con 0x7fbcf80725a0 2024-03-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcfe400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf815dac0 con 0x7fbcf81477a0 2024-03-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcf6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcf80b5be0 0x7fbcf813e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcf6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcf80b5be0 0x7fbcf813e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39906/0 (socket says 172.21.15.40:39906) 2024-03-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcf6a00640 1 -- 172.21.15.40:0/666699893 learned_addr learned my addr 172.21.15.40:0/666699893 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcf7400640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcf80725a0 0x7fbcf80aeff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcf7e00640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf81477a0 0x7fbcf80b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcf6a00640 1 -- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf81477a0 msgr2=0x7fbcf80b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcf6a00640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf81477a0 0x7fbcf80b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcf6a00640 1 -- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcf80725a0 msgr2=0x7fbcf80aeff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcf6a00640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcf80725a0 0x7fbcf80aeff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcf6a00640 1 -- 172.21.15.40:0/666699893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf813eba0 con 0x7fbcf80b5be0 2024-03-24T00:36:58.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcf6a00640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcf80b5be0 0x7fbcf813e660 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fbce400c3f0 tx=0x7fbce400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcef400640 1 -- 172.21.15.40:0/666699893 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce4017070 con 0x7fbcf80b5be0 2024-03-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcef400640 1 -- 172.21.15.40:0/666699893 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce4005cf0 con 0x7fbcf80b5be0 2024-03-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcef400640 1 -- 172.21.15.40:0/666699893 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40072f0 con 0x7fbcf80b5be0 2024-03-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcfe400640 1 -- 172.21.15.40:0/666699893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf80adf50 con 0x7fbcf80b5be0 2024-03-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcfe400640 1 -- 172.21.15.40:0/666699893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf80ae510 con 0x7fbcf80b5be0 2024-03-24T00:36:58.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.718+0000 7fbcfe400640 1 -- 172.21.15.40:0/666699893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf8068170 con 0x7fbcf80b5be0 2024-03-24T00:36:58.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.722+0000 7fbcef400640 1 -- 172.21.15.40:0/666699893 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbce401a070 con 0x7fbcf80b5be0 2024-03-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.722+0000 7fbcef400640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbcd8067690 0x7fbcd8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.722+0000 7fbcef400640 1 -- 172.21.15.40:0/666699893 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbce40849f0 con 0x7fbcf80b5be0 2024-03-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.722+0000 7fbcf7400640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbcd8067690 0x7fbcd8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:58.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.722+0000 7fbcf7400640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbcd8067690 0x7fbcd8069b50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbce8000f70 tx=0x7fbce8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:58.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.726+0000 7fbcef400640 1 -- 172.21.15.40:0/666699893 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbce4055920 con 0x7fbcf80b5be0 2024-03-24T00:36:58.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.886+0000 7fbcfe400640 1 -- 172.21.15.40:0/666699893 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcf8146fd0 con 0x7fbcf80b5be0 2024-03-24T00:36:58.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcef400640 1 -- 172.21.15.40:0/666699893 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+12832 (secure 0 0 0) 0x7fbce4055220 con 0x7fbcf80b5be0 2024-03-24T00:36:58.892 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:58.892 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":91,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:36:58.148443+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:36:58.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcfe400640 1 -- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbcd8067690 msgr2=0x7fbcd8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcfe400640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbcd8067690 0x7fbcd8069b50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbce8000f70 tx=0x7fbce8009040 comp rx=0 tx=0).stop 2024-03-24T00:36:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcfe400640 1 -- 172.21.15.40:0/666699893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcf80b5be0 msgr2=0x7fbcf813e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcfe400640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcf80b5be0 0x7fbcf813e660 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fbce400c3f0 tx=0x7fbce400c8b0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcfe400640 1 -- 172.21.15.40:0/666699893 shutdown_connections 2024-03-24T00:36:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcfe400640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbcd8067690 0x7fbcd8069b50 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcfe400640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf81477a0 0x7fbcf80b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcfe400640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbcf80b5be0 0x7fbcf813e660 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcfe400640 1 --2- 172.21.15.40:0/666699893 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcf80725a0 0x7fbcf80aeff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcfe400640 1 -- 172.21.15.40:0/666699893 >> 172.21.15.40:0/666699893 conn(0x7fbcf806b150 msgr2=0x7fbcf80cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcfe400640 1 -- 172.21.15.40:0/666699893 shutdown_connections 2024-03-24T00:36:58.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:58.890+0000 7fbcfe400640 1 -- 172.21.15.40:0/666699893 wait complete. 2024-03-24T00:36:58.913 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:36:58.913 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:36:59.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170f000640 1 Processor -- start 2024-03-24T00:36:59.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170f000640 1 -- start start 2024-03-24T00:36:59.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170f000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17081547a0 0x7f1708070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170f000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f17080b2940 0x7f17080b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170f000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170806cc20 con 0x7f170806d9c0 2024-03-24T00:36:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170f000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f170806cee0 con 0x7f17080b2940 2024-03-24T00:36:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170f000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170806d1c0 con 0x7f17081547a0 2024-03-24T00:36:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170d200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f17080b2940 0x7f17080b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170d200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f17080b2940 0x7f17080b2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39910/0 (socket says 172.21.15.40:39910) 2024-03-24T00:36:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170d200640 1 -- 172.21.15.40:0/3923144930 learned_addr learned my addr 172.21.15.40:0/3923144930 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170dc00640 1 --2- 172.21.15.40:0/3923144930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17081547a0 0x7f1708070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170d200640 1 -- 172.21.15.40:0/3923144930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17081547a0 msgr2=0x7f1708070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170d200640 1 --2- 172.21.15.40:0/3923144930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17081547a0 0x7f1708070e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170d200640 1 -- 172.21.15.40:0/3923144930 >> v1:172.21.15.40:6789/0 conn(0x7f170806d9c0 legacy=0x7f170806de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.054+0000 7f170d200640 1 -- 172.21.15.40:0/3923144930 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1708154b80 con 0x7f17080b2940 2024-03-24T00:36:59.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f170d200640 1 --2- 172.21.15.40:0/3923144930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17080b2940 0x7f17080b2dc0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009870 tx=0x7f16f401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92163d499bccb523 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:59.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f1707e00640 1 -- 172.21.15.40:0/3923144930 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f40b0070 con 0x7f17080b2940 2024-03-24T00:36:59.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f1707e00640 1 -- 172.21.15.40:0/3923144930 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f40ab600 con 0x7f17080b2940 2024-03-24T00:36:59.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f1707e00640 1 -- 172.21.15.40:0/3923144930 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f40ab9d0 con 0x7f17080b2940 2024-03-24T00:36:59.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f170f000640 1 -- 172.21.15.40:0/3923144930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17080b2940 msgr2=0x7f17080b2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f170f000640 1 --2- 172.21.15.40:0/3923144930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17080b2940 0x7f17080b2dc0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009870 tx=0x7f16f401c560 comp rx=0 tx=0).stop 2024-03-24T00:36:59.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f170f000640 1 -- 172.21.15.40:0/3923144930 shutdown_connections 2024-03-24T00:36:59.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f170f000640 1 --2- 172.21.15.40:0/3923144930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17080b2940 0x7f17080b2dc0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f170f000640 1 --2- 172.21.15.40:0/3923144930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17081547a0 0x7f1708070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f170f000640 1 -- 172.21.15.40:0/3923144930 >> 172.21.15.40:0/3923144930 conn(0x7f1708066e80 msgr2=0x7f1708067290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:59.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f170f000640 1 -- 172.21.15.40:0/3923144930 shutdown_connections 2024-03-24T00:36:59.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f170f000640 1 -- 172.21.15.40:0/3923144930 wait complete. 2024-03-24T00:36:59.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.058+0000 7f170f000640 1 Processor -- start 2024-03-24T00:36:59.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170f000640 1 -- start start 2024-03-24T00:36:59.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170f000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f170806d9c0 0x7f170814a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170f000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f17080b2940 0x7f1708149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170f000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17081547a0 0x7f1708152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170f000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17080715a0 con 0x7f170806d9c0 2024-03-24T00:36:59.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170f000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170815b4e0 con 0x7f17080b2940 2024-03-24T00:36:59.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170f000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17081690d0 con 0x7f17081547a0 2024-03-24T00:36:59.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f170806d9c0 0x7f170814a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f170806d9c0 0x7f170814a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39918/0 (socket says 172.21.15.40:39918) 2024-03-24T00:36:59.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170dc00640 1 -- 172.21.15.40:0/3781237501 learned_addr learned my addr 172.21.15.40:0/3781237501 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:59.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170d200640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f17080b2940 0x7f1708149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170e600640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17081547a0 0x7f1708152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170dc00640 1 -- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17081547a0 msgr2=0x7f1708152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170dc00640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17081547a0 0x7f1708152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170dc00640 1 -- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f17080b2940 msgr2=0x7f1708149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170dc00640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f17080b2940 0x7f1708149ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170dc00640 1 -- 172.21.15.40:0/3781237501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17080af910 con 0x7f170806d9c0 2024-03-24T00:36:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170d200640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f17080b2940 0x7f1708149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170e600640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17081547a0 0x7f1708152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:36:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170dc00640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f170806d9c0 0x7f170814a2f0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00d5f0 tx=0x7f16fc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:59.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f1705600640 1 -- 172.21.15.40:0/3781237501 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc0125f0 con 0x7f170806d9c0 2024-03-24T00:36:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f1705600640 1 -- 172.21.15.40:0/3781237501 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc019070 con 0x7f170806d9c0 2024-03-24T00:36:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f1705600640 1 -- 172.21.15.40:0/3781237501 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc0129b0 con 0x7f170806d9c0 2024-03-24T00:36:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170f000640 1 -- 172.21.15.40:0/3781237501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17080afbf0 con 0x7f170806d9c0 2024-03-24T00:36:59.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170f000640 1 -- 172.21.15.40:0/3781237501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17080b01b0 con 0x7f170806d9c0 2024-03-24T00:36:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.062+0000 7f170f000640 1 -- 172.21.15.40:0/3781237501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170804f410 con 0x7f170806d9c0 2024-03-24T00:36:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.066+0000 7f1705600640 1 -- 172.21.15.40:0/3781237501 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f16fc012b50 con 0x7f170806d9c0 2024-03-24T00:36:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.066+0000 7f1705600640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f16e8067690 0x7f16e8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.066+0000 7f1705600640 1 -- 172.21.15.40:0/3781237501 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f16fc085940 con 0x7f170806d9c0 2024-03-24T00:36:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.066+0000 7f170d200640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f16e8067690 0x7f16e8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.066+0000 7f170d200640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f16e8067690 0x7f16e8069b50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f16f40ad000 tx=0x7f16f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:59.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.070+0000 7f1705600640 1 -- 172.21.15.40:0/3781237501 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f16fc056900 con 0x7f170806d9c0 2024-03-24T00:36:59.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.234+0000 7f170f000640 1 -- 172.21.15.40:0/3781237501 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f170806e150 con 0x7f170806d9c0 2024-03-24T00:36:59.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.234+0000 7f1705600640 1 -- 172.21.15.40:0/3781237501 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f16fc056200 con 0x7f170806d9c0 2024-03-24T00:36:59.239 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:36:59.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.238+0000 7f170f000640 1 -- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f16e8067690 msgr2=0x7f16e8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.238+0000 7f170f000640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f16e8067690 0x7f16e8069b50 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f16f40ad000 tx=0x7f16f4008040 comp rx=0 tx=0).stop 2024-03-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.238+0000 7f170f000640 1 -- 172.21.15.40:0/3781237501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f170806d9c0 msgr2=0x7f170814a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.238+0000 7f170f000640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f170806d9c0 0x7f170814a2f0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00d5f0 tx=0x7f16fc00dab0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.238+0000 7f170f000640 1 -- 172.21.15.40:0/3781237501 shutdown_connections 2024-03-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.238+0000 7f170f000640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f16e8067690 0x7f16e8069b50 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.238+0000 7f170f000640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17081547a0 0x7f1708152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.238+0000 7f170f000640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f17080b2940 0x7f1708149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.238+0000 7f170f000640 1 --2- 172.21.15.40:0/3781237501 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f170806d9c0 0x7f170814a2f0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.238+0000 7f170f000640 1 -- 172.21.15.40:0/3781237501 >> 172.21.15.40:0/3781237501 conn(0x7f1708066e80 msgr2=0x7f170815c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.238+0000 7f170f000640 1 -- 172.21.15.40:0/3781237501 shutdown_connections 2024-03-24T00:36:59.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.238+0000 7f170f000640 1 -- 172.21.15.40:0/3781237501 wait complete. 2024-03-24T00:36:59.260 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:36:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b9600640 1 Processor -- start 2024-03-24T00:36:59.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b9600640 1 -- start start 2024-03-24T00:36:59.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b9600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55b415f300 0x7f55b40cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b9600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f55b40cd200 0x7f55b415b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b9600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55b415b750 0x7f55b4071c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b9600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b40cc220 con 0x7f55b415b750 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b9600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b415bc60 con 0x7f55b40cd200 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b9600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b415bf40 con 0x7f55b415f300 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b3400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55b415b750 0x7f55b4071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b3400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f55b415b750 0x7f55b4071c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39924/0 (socket says 172.21.15.40:39924) 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b3400640 1 -- 172.21.15.40:0/319460530 learned_addr learned my addr 172.21.15.40:0/319460530 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b2a00640 1 --2- 172.21.15.40:0/319460530 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55b415f300 0x7f55b40cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55aa000640 1 --2- 172.21.15.40:0/319460530 >> v2:172.21.15.53:3300/0 conn(0x7f55b40cd200 0x7f55b415b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b3400640 1 -- 172.21.15.40:0/319460530 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55b415f300 msgr2=0x7f55b40cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b3400640 1 --2- 172.21.15.40:0/319460530 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55b415f300 0x7f55b40cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b3400640 1 -- 172.21.15.40:0/319460530 >> v2:172.21.15.53:3300/0 conn(0x7f55b40cd200 msgr2=0x7f55b415b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:36:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b3400640 1 --2- 172.21.15.40:0/319460530 >> v2:172.21.15.53:3300/0 conn(0x7f55b40cd200 0x7f55b415b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.398+0000 7f55b3400640 1 -- 172.21.15.40:0/319460530 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b40cc4c0 con 0x7f55b415b750 2024-03-24T00:36:59.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b3400640 1 --2- 172.21.15.40:0/319460530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55b415b750 0x7f55b4071c70 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f55b4071860 tx=0x7f55ac01ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ce7d5905d8b0c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:59.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b2000640 1 -- 172.21.15.40:0/319460530 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac0b4070 con 0x7f55b415b750 2024-03-24T00:36:59.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b2000640 1 -- 172.21.15.40:0/319460530 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac0a7a80 con 0x7f55b415b750 2024-03-24T00:36:59.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b2000640 1 -- 172.21.15.40:0/319460530 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac0a7e50 con 0x7f55b415b750 2024-03-24T00:36:59.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b9600640 1 -- 172.21.15.40:0/319460530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55b415b750 msgr2=0x7f55b4071c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b9600640 1 --2- 172.21.15.40:0/319460530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55b415b750 0x7f55b4071c70 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f55b4071860 tx=0x7f55ac01ebf0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b9600640 1 -- 172.21.15.40:0/319460530 shutdown_connections 2024-03-24T00:36:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b9600640 1 --2- 172.21.15.40:0/319460530 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55b415b750 0x7f55b4071c70 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b9600640 1 --2- 172.21.15.40:0/319460530 >> v2:172.21.15.53:3300/0 conn(0x7f55b40cd200 0x7f55b415b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b9600640 1 --2- 172.21.15.40:0/319460530 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55b415f300 0x7f55b40cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b9600640 1 -- 172.21.15.40:0/319460530 >> 172.21.15.40:0/319460530 conn(0x7f55b40cd610 msgr2=0x7f55b40cda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b9600640 1 -- 172.21.15.40:0/319460530 shutdown_connections 2024-03-24T00:36:59.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.402+0000 7f55b9600640 1 -- 172.21.15.40:0/319460530 wait complete. 2024-03-24T00:36:59.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b9600640 1 Processor -- start 2024-03-24T00:36:59.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b9600640 1 -- start start 2024-03-24T00:36:59.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55b40cd200 0x7f55b40b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b415b750 0x7f55b414e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b9600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55b415f300 0x7f55b41fa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b9600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40c5a30 con 0x7f55b40cd200 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b9600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b415eb60 con 0x7f55b415b750 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b9600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40c5bb0 con 0x7f55b415f300 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55b40cd200 0x7f55b40b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55b40cd200 0x7f55b40b4270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39934/0 (socket says 172.21.15.40:39934) 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b2a00640 1 -- 172.21.15.40:0/2371580713 learned_addr learned my addr 172.21.15.40:0/2371580713 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55aa000640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b415b750 0x7f55b414e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b3400640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55b415f300 0x7f55b41fa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b2a00640 1 -- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55b415f300 msgr2=0x7f55b41fa8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b2a00640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55b415f300 0x7f55b41fa8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b2a00640 1 -- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b415b750 msgr2=0x7f55b414e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b2a00640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b415b750 0x7f55b414e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b2a00640 1 -- 172.21.15.40:0/2371580713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b414ecc0 con 0x7f55b40cd200 2024-03-24T00:36:59.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b2a00640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55b40cd200 0x7f55b40b4270 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f55a400efc0 tx=0x7f55a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:59.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55ab400640 1 -- 172.21.15.40:0/2371580713 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4018020 con 0x7f55b40cd200 2024-03-24T00:36:59.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55ab400640 1 -- 172.21.15.40:0/2371580713 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a4014480 con 0x7f55b40cd200 2024-03-24T00:36:59.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55ab400640 1 -- 172.21.15.40:0/2371580713 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4014850 con 0x7f55b40cd200 2024-03-24T00:36:59.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b9600640 1 -- 172.21.15.40:0/2371580713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b40b49f0 con 0x7f55b40cd200 2024-03-24T00:36:59.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.406+0000 7f55b9600640 1 -- 172.21.15.40:0/2371580713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b406e0d0 con 0x7f55b40cd200 2024-03-24T00:36:59.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.410+0000 7f55b9600640 1 -- 172.21.15.40:0/2371580713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b4068170 con 0x7f55b40cd200 2024-03-24T00:36:59.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.410+0000 7f55ab400640 1 -- 172.21.15.40:0/2371580713 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f55a401b020 con 0x7f55b40cd200 2024-03-24T00:36:59.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.410+0000 7f55ab400640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5590067660 0x7f5590069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.410+0000 7f55ab400640 1 -- 172.21.15.40:0/2371580713 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f55a4085e10 con 0x7f55b40cd200 2024-03-24T00:36:59.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.410+0000 7f55aa000640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5590067660 0x7f5590069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.410+0000 7f55aa000640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5590067660 0x7f5590069b20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f55b41522e0 tx=0x7f559c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:59.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.414+0000 7f55ab400640 1 -- 172.21.15.40:0/2371580713 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f55a4089030 con 0x7f55b40cd200 2024-03-24T00:36:59.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 -- 172.21.15.40:0/2371580713 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f55b40b1340 con 0x7f55b40cd200 2024-03-24T00:36:59.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55ab400640 1 -- 172.21.15.40:0/2371580713 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v91) v1 ==== 84+0+65 (secure 0 0 0) 0x7f55a4022070 con 0x7f55b40cd200 2024-03-24T00:36:59.581 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:36:59.581 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-03-24T00:36:59.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:36:59.582 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-03-24T00:36:59.582 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-03-24T00:36:59.582 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:36:59.582 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-03-24T00:36:59.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 -- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5590067660 msgr2=0x7f5590069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5590067660 0x7f5590069b20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f55b41522e0 tx=0x7f559c009190 comp rx=0 tx=0).stop 2024-03-24T00:36:59.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 -- 172.21.15.40:0/2371580713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55b40cd200 msgr2=0x7f55b40b4270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55b40cd200 0x7f55b40b4270 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f55a400efc0 tx=0x7f55a400c490 comp rx=0 tx=0).stop 2024-03-24T00:36:59.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 -- 172.21.15.40:0/2371580713 shutdown_connections 2024-03-24T00:36:59.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5590067660 0x7f5590069b20 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55b415f300 0x7f55b41fa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b415b750 0x7f55b414e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 --2- 172.21.15.40:0/2371580713 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55b40cd200 0x7f55b40b4270 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 -- 172.21.15.40:0/2371580713 >> 172.21.15.40:0/2371580713 conn(0x7f55b40cd610 msgr2=0x7f55b4066e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:59.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 -- 172.21.15.40:0/2371580713 shutdown_connections 2024-03-24T00:36:59.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.578+0000 7f55b9600640 1 -- 172.21.15.40:0/2371580713 wait complete. 2024-03-24T00:36:59.603 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:36:59.603 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:36:59.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe18d000640 1 Processor -- start 2024-03-24T00:36:59.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe18d000640 1 -- start start 2024-03-24T00:36:59.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe18d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe1880b2730 0x7fe188166b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe18d000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1880b33e0 con 0x7fe18815f5b0 2024-03-24T00:36:59.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe18d000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1880b22f0 con 0x7fe1880cc4c0 2024-03-24T00:36:59.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe18d000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1880afa90 con 0x7fe1880b2730 2024-03-24T00:36:59.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe187400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe18815f5b0 0x7fe1880cfb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44790/0 (socket says 172.21.15.40:44790) 2024-03-24T00:36:59.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe187400640 1 -- 172.21.15.40:0/3814736158 learned_addr learned my addr 172.21.15.40:0/3814736158 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:36:59.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe187e00640 1 --2- 172.21.15.40:0/3814736158 >> v2:172.21.15.53:3300/0 conn(0x7fe1880b2730 0x7fe188166b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe186a00640 1 -- 172.21.15.40:0/3814736158 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622045163 0 0) 0x7fe1880b33e0 con 0x7fe18815f5b0 2024-03-24T00:36:59.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe186a00640 1 -- 172.21.15.40:0/3814736158 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe16c003530 con 0x7fe18815f5b0 2024-03-24T00:36:59.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe187e00640 1 -- 172.21.15.40:0/3814736158 >> v1:172.21.15.53:6789/0 conn(0x7fe1880cc4c0 legacy=0x7fe1880d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe187e00640 1 -- 172.21.15.40:0/3814736158 >> v1:172.21.15.40:6789/0 conn(0x7fe18815f5b0 legacy=0x7fe1880cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe186a00640 1 -- 172.21.15.40:0/3814736158 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1623491239 0 0) 0x7fe16c003530 con 0x7fe18815f5b0 2024-03-24T00:36:59.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.746+0000 7fe187e00640 1 -- 172.21.15.40:0/3814736158 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1880af440 con 0x7fe1880b2730 2024-03-24T00:36:59.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.750+0000 7fe187e00640 1 --2- 172.21.15.40:0/3814736158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880b2730 0x7fe188166b60 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe180009d90 tx=0x7fe18001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2ba28736c859068 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:59.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.750+0000 7fe186a00640 1 -- 172.21.15.40:0/3814736158 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178001680 con 0x7fe1880b2730 2024-03-24T00:36:59.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.750+0000 7fe186a00640 1 -- 172.21.15.40:0/3814736158 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1800a64e0 con 0x7fe1880b2730 2024-03-24T00:36:59.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.750+0000 7fe186a00640 1 -- 172.21.15.40:0/3814736158 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1800a6890 con 0x7fe1880b2730 2024-03-24T00:36:59.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.750+0000 7fe18d000640 1 -- 172.21.15.40:0/3814736158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880b2730 msgr2=0x7fe188166b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.750+0000 7fe18d000640 1 --2- 172.21.15.40:0/3814736158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880b2730 0x7fe188166b60 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe180009d90 tx=0x7fe18001c890 comp rx=0 tx=0).stop 2024-03-24T00:36:59.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.750+0000 7fe18d000640 1 -- 172.21.15.40:0/3814736158 shutdown_connections 2024-03-24T00:36:59.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.750+0000 7fe18d000640 1 --2- 172.21.15.40:0/3814736158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880b2730 0x7fe188166b60 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.750+0000 7fe18d000640 1 -- 172.21.15.40:0/3814736158 >> 172.21.15.40:0/3814736158 conn(0x7fe1880b3620 msgr2=0x7fe1880b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:36:59.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.750+0000 7fe18d000640 1 -- 172.21.15.40:0/3814736158 shutdown_connections 2024-03-24T00:36:59.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.750+0000 7fe18d000640 1 -- 172.21.15.40:0/3814736158 wait complete. 2024-03-24T00:36:59.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe18d000640 1 Processor -- start 2024-03-24T00:36:59.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe18d000640 1 -- start start 2024-03-24T00:36:59.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe18d000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1880b2730 0x7fe18806cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe18d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880cc4c0 0x7fe1880c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe18d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18815f5b0 0x7fe18806b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe18d000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe188169a40 con 0x7fe18815f5b0 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe18d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1881698c0 con 0x7fe1880cc4c0 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe18d000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe188169bc0 con 0x7fe1880b2730 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe187e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18815f5b0 0x7fe18806b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe187e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18815f5b0 0x7fe18806b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39948/0 (socket says 172.21.15.40:39948) 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe187e00640 1 -- 172.21.15.40:0/3623598017 learned_addr learned my addr 172.21.15.40:0/3623598017 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe187400640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1880b2730 0x7fe18806cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe17ea00640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880cc4c0 0x7fe1880c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe187e00640 1 -- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1880b2730 msgr2=0x7fe18806cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe187e00640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1880b2730 0x7fe18806cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe187e00640 1 -- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880cc4c0 msgr2=0x7fe1880c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe187e00640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880cc4c0 0x7fe1880c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe187e00640 1 -- 172.21.15.40:0/3623598017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18806bbf0 con 0x7fe18815f5b0 2024-03-24T00:36:59.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe187e00640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18815f5b0 0x7fe18806b150 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fe1800b0040 tx=0x7fe180019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:59.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe17fe00640 1 -- 172.21.15.40:0/3623598017 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe180009450 con 0x7fe18815f5b0 2024-03-24T00:36:59.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe17fe00640 1 -- 172.21.15.40:0/3623598017 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1800c1070 con 0x7fe18815f5b0 2024-03-24T00:36:59.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe17fe00640 1 -- 172.21.15.40:0/3623598017 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1800b8aa0 con 0x7fe18815f5b0 2024-03-24T00:36:59.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe18d000640 1 -- 172.21.15.40:0/3623598017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18806bed0 con 0x7fe18815f5b0 2024-03-24T00:36:59.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe18d000640 1 -- 172.21.15.40:0/3623598017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18806c490 con 0x7fe18815f5b0 2024-03-24T00:36:59.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.754+0000 7fe18d000640 1 -- 172.21.15.40:0/3623598017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18804f410 con 0x7fe18815f5b0 2024-03-24T00:36:59.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.758+0000 7fe17fe00640 1 -- 172.21.15.40:0/3623598017 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe1800b8c40 con 0x7fe18815f5b0 2024-03-24T00:36:59.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.758+0000 7fe17fe00640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe164067660 0x7fe164069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:36:59.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.758+0000 7fe17fe00640 1 -- 172.21.15.40:0/3623598017 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe180120800 con 0x7fe18815f5b0 2024-03-24T00:36:59.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.758+0000 7fe187400640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe164067660 0x7fe164069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:36:59.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.758+0000 7fe187400640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe164067660 0x7fe164069b20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe178005890 tx=0x7fe178009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:36:59.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.762+0000 7fe17fe00640 1 -- 172.21.15.40:0/3623598017 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe180016480 con 0x7fe18815f5b0 2024-03-24T00:36:59.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:36:59.926+0000 7fe18d000640 1 -- 172.21.15.40:0/3623598017 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe188158a50 con 0x7fe18815f5b0 2024-03-24T00:37:00.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe17fe00640 1 -- 172.21.15.40:0/3623598017 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v92) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe1800c4360 con 0x7fe18815f5b0 2024-03-24T00:37:00.152 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-03-24T00:37:00.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe18d000640 1 -- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe164067660 msgr2=0x7fe164069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:00.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe18d000640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe164067660 0x7fe164069b20 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fe178005890 tx=0x7fe178009040 comp rx=0 tx=0).stop 2024-03-24T00:37:00.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe18d000640 1 -- 172.21.15.40:0/3623598017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18815f5b0 msgr2=0x7fe18806b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:00.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe18d000640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18815f5b0 0x7fe18806b150 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fe1800b0040 tx=0x7fe180019630 comp rx=0 tx=0).stop 2024-03-24T00:37:00.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe18d000640 1 -- 172.21.15.40:0/3623598017 shutdown_connections 2024-03-24T00:37:00.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe18d000640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe164067660 0x7fe164069b20 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:00.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe18d000640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe18815f5b0 0x7fe18806b150 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:00.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe18d000640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880cc4c0 0x7fe1880c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:00.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe18d000640 1 --2- 172.21.15.40:0/3623598017 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1880b2730 0x7fe18806cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:00.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe18d000640 1 -- 172.21.15.40:0/3623598017 >> 172.21.15.40:0/3623598017 conn(0x7fe1880b3620 msgr2=0x7fe18815cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:00.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe18d000640 1 -- 172.21.15.40:0/3623598017 shutdown_connections 2024-03-24T00:37:00.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.150+0000 7fe18d000640 1 -- 172.21.15.40:0/3623598017 wait complete. 2024-03-24T00:37:00.181 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:37:00.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde3400640 1 Processor -- start 2024-03-24T00:37:00.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde3400640 1 -- start start 2024-03-24T00:37:00.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde3400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdddc15bd00 0x7fdddc0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:00.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde3400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdddc15ffd0 0x7fdddc0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:00.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde3400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc072d60 con 0x7fdddc0728d0 2024-03-24T00:37:00.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde3400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdddc0cf790 con 0x7fdddc15bd00 2024-03-24T00:37:00.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde3400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdddc071de0 con 0x7fdddc15ffd0 2024-03-24T00:37:00.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde0a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdddc15bd00 0x7fdddc0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:00.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde0a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdddc15bd00 0x7fdddc0ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39952/0 (socket says 172.21.15.40:39952) 2024-03-24T00:37:00.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde0a00640 1 -- 172.21.15.40:0/1150584300 learned_addr learned my addr 172.21.15.40:0/1150584300 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:00.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdddbe00640 1 --1- 172.21.15.40:0/1150584300 >> v1:172.21.15.40:6789/0 conn(0x7fdddc0728d0 0x7fdddc15f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44794/0 (socket says 172.21.15.40:44794) 2024-03-24T00:37:00.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde1400640 1 --2- 172.21.15.40:0/1150584300 >> v2:172.21.15.53:3300/0 conn(0x7fdddc15ffd0 0x7fdddc0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:00.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde0a00640 1 -- 172.21.15.40:0/1150584300 >> v2:172.21.15.53:3300/0 conn(0x7fdddc15ffd0 msgr2=0x7fdddc0cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:00.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde0a00640 1 --2- 172.21.15.40:0/1150584300 >> v2:172.21.15.53:3300/0 conn(0x7fdddc15ffd0 0x7fdddc0cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:00.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde0a00640 1 -- 172.21.15.40:0/1150584300 >> v1:172.21.15.40:6789/0 conn(0x7fdddc0728d0 legacy=0x7fdddc15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:00.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.326+0000 7fdde0a00640 1 -- 172.21.15.40:0/1150584300 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc0cea50 con 0x7fdddc15bd00 2024-03-24T00:37:00.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.330+0000 7fdde0a00640 1 --2- 172.21.15.40:0/1150584300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdddc15bd00 0x7fdddc0ae8e0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fdddc070a80 tx=0x7fddd001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76f35976e94cda4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:00.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.330+0000 7fdddb400640 1 -- 172.21.15.40:0/1150584300 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd00aa650 con 0x7fdddc15bd00 2024-03-24T00:37:00.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.330+0000 7fdddb400640 1 -- 172.21.15.40:0/1150584300 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd00af070 con 0x7fdddc15bd00 2024-03-24T00:37:00.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.330+0000 7fdddb400640 1 -- 172.21.15.40:0/1150584300 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd00aaa10 con 0x7fdddc15bd00 2024-03-24T00:37:00.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.330+0000 7fdde3400640 1 -- 172.21.15.40:0/1150584300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdddc15bd00 msgr2=0x7fdddc0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:00.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.330+0000 7fdde3400640 1 --2- 172.21.15.40:0/1150584300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdddc15bd00 0x7fdddc0ae8e0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fdddc070a80 tx=0x7fddd001c4f0 comp rx=0 tx=0).stop 2024-03-24T00:37:00.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.330+0000 7fdde3400640 1 -- 172.21.15.40:0/1150584300 shutdown_connections 2024-03-24T00:37:00.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.330+0000 7fdde3400640 1 --2- 172.21.15.40:0/1150584300 >> v2:172.21.15.53:3300/0 conn(0x7fdddc15ffd0 0x7fdddc0cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:00.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.330+0000 7fdde3400640 1 --2- 172.21.15.40:0/1150584300 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdddc15bd00 0x7fdddc0ae8e0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:00.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.330+0000 7fdde3400640 1 -- 172.21.15.40:0/1150584300 >> 172.21.15.40:0/1150584300 conn(0x7fdddc0b1660 msgr2=0x7fdddc0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:00.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.330+0000 7fdde3400640 1 -- 172.21.15.40:0/1150584300 shutdown_connections 2024-03-24T00:37:00.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.330+0000 7fdde3400640 1 -- 172.21.15.40:0/1150584300 wait complete. 2024-03-24T00:37:00.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde3400640 1 Processor -- start 2024-03-24T00:37:00.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde3400640 1 -- start start 2024-03-24T00:37:00.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdddc0728d0 0x7fdddc153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:00.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdddc15bd00 0x7fdddc13c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:00.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde3400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdddc15ffd0 0x7fdddc14f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:00.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde3400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc06d320 con 0x7fdddc0728d0 2024-03-24T00:37:00.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde3400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc15c0e0 con 0x7fdddc15bd00 2024-03-24T00:37:00.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde3400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc06d4a0 con 0x7fdddc15ffd0 2024-03-24T00:37:00.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdddc0728d0 0x7fdddc153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:00.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdddc0728d0 0x7fdddc153380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39958/0 (socket says 172.21.15.40:39958) 2024-03-24T00:37:00.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde0a00640 1 -- 172.21.15.40:0/411122598 learned_addr learned my addr 172.21.15.40:0/411122598 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:00.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde0a00640 1 -- 172.21.15.40:0/411122598 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdddc15ffd0 msgr2=0x7fdddc14f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:00.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde0a00640 1 --2- 172.21.15.40:0/411122598 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdddc15ffd0 0x7fdddc14f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:00.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde0a00640 1 -- 172.21.15.40:0/411122598 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdddc15bd00 msgr2=0x7fdddc13c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:00.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde0a00640 1 --2- 172.21.15.40:0/411122598 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdddc15bd00 0x7fdddc13c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:00.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde0a00640 1 -- 172.21.15.40:0/411122598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc152000 con 0x7fdddc0728d0 2024-03-24T00:37:00.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde0a00640 1 --2- 172.21.15.40:0/411122598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdddc0728d0 0x7fdddc153380 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fddd00ac040 tx=0x7fddd001ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:00.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fddd8c00640 1 -- 172.21.15.40:0/411122598 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd00bd5f0 con 0x7fdddc0728d0 2024-03-24T00:37:00.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fddd8c00640 1 -- 172.21.15.40:0/411122598 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd00a6040 con 0x7fdddc0728d0 2024-03-24T00:37:00.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fddd8c00640 1 -- 172.21.15.40:0/411122598 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd00bd990 con 0x7fdddc0728d0 2024-03-24T00:37:00.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde3400640 1 -- 172.21.15.40:0/411122598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc1522e0 con 0x7fdddc0728d0 2024-03-24T00:37:00.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde3400640 1 -- 172.21.15.40:0/411122598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc1528a0 con 0x7fdddc0728d0 2024-03-24T00:37:00.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.334+0000 7fdde3400640 1 -- 172.21.15.40:0/411122598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc068170 con 0x7fdddc0728d0 2024-03-24T00:37:00.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.338+0000 7fddd8c00640 1 -- 172.21.15.40:0/411122598 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fddd00b33f0 con 0x7fdddc0728d0 2024-03-24T00:37:00.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.338+0000 7fddd8c00640 1 --2- 172.21.15.40:0/411122598 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fddbc067660 0x7fddbc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:00.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.338+0000 7fddd8c00640 1 -- 172.21.15.40:0/411122598 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fddd011f690 con 0x7fdddc0728d0 2024-03-24T00:37:00.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.338+0000 7fdddbe00640 1 --2- 172.21.15.40:0/411122598 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fddbc067660 0x7fddbc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:00.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.338+0000 7fdddbe00640 1 --2- 172.21.15.40:0/411122598 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fddbc067660 0x7fddbc069b20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fddc8000f70 tx=0x7fddc8007a10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:00.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.342+0000 7fddd8c00640 1 -- 172.21.15.40:0/411122598 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fddd0016110 con 0x7fdddc0728d0 2024-03-24T00:37:00.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:00.506+0000 7fdde3400640 1 -- 172.21.15.40:0/411122598 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fdddc06e870 con 0x7fdddc0728d0 2024-03-24T00:37:01.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.150+0000 7fddd8c00640 1 -- 172.21.15.40:0/411122598 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v93) v1 ==== 170+0+0 (secure 0 0 0) 0x7fddd00efe00 con 0x7fdddc0728d0 2024-03-24T00:37:01.155 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-03-24T00:37:01.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.154+0000 7fdde3400640 1 -- 172.21.15.40:0/411122598 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fddbc067660 msgr2=0x7fddbc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:01.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.154+0000 7fdde3400640 1 --2- 172.21.15.40:0/411122598 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fddbc067660 0x7fddbc069b20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fddc8000f70 tx=0x7fddc8007a10 comp rx=0 tx=0).stop 2024-03-24T00:37:01.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.154+0000 7fdde3400640 1 -- 172.21.15.40:0/411122598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdddc0728d0 msgr2=0x7fdddc153380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:01.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.154+0000 7fdde3400640 1 --2- 172.21.15.40:0/411122598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdddc0728d0 0x7fdddc153380 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fddd00ac040 tx=0x7fddd001ad30 comp rx=0 tx=0).stop 2024-03-24T00:37:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.158+0000 7fdde3400640 1 -- 172.21.15.40:0/411122598 shutdown_connections 2024-03-24T00:37:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.158+0000 7fdde3400640 1 --2- 172.21.15.40:0/411122598 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fddbc067660 0x7fddbc069b20 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.158+0000 7fdde3400640 1 --2- 172.21.15.40:0/411122598 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdddc15ffd0 0x7fdddc14f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.158+0000 7fdde3400640 1 --2- 172.21.15.40:0/411122598 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdddc15bd00 0x7fdddc13c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.158+0000 7fdde3400640 1 --2- 172.21.15.40:0/411122598 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdddc0728d0 0x7fdddc153380 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:01.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.158+0000 7fdde3400640 1 -- 172.21.15.40:0/411122598 >> 172.21.15.40:0/411122598 conn(0x7fdddc0b1660 msgr2=0x7fdddc0cdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:01.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.158+0000 7fdde3400640 1 -- 172.21.15.40:0/411122598 shutdown_connections 2024-03-24T00:37:01.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.158+0000 7fdde3400640 1 -- 172.21.15.40:0/411122598 wait complete. 2024-03-24T00:37:01.201 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:37:01.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.346+0000 7f232ba00640 1 Processor -- start 2024-03-24T00:37:01.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.346+0000 7f232ba00640 1 -- start start 2024-03-24T00:37:01.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.346+0000 7f232ba00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2324160080 0x7f2324168ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:01.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.346+0000 7f232ba00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23240cfcb0 con 0x7f232406f7b0 2024-03-24T00:37:01.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.346+0000 7f232ba00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23240d1120 con 0x7f232415e170 2024-03-24T00:37:01.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.346+0000 7f232ba00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f232406fc00 con 0x7f2324160080 2024-03-24T00:37:01.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.346+0000 7f2329a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2324160080 0x7f2324168ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:01.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.346+0000 7f231be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f232406f7b0 0x7f232415f970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44808/0 (socket says 172.21.15.40:44808) 2024-03-24T00:37:01.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.346+0000 7f231be00640 1 -- 172.21.15.40:0/32919175 learned_addr learned my addr 172.21.15.40:0/32919175 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:01.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.346+0000 7f2329a00640 1 -- 172.21.15.40:0/32919175 >> v1:172.21.15.53:6789/0 conn(0x7f232415e170 legacy=0x7f23240d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:01.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.346+0000 7f2329a00640 1 -- 172.21.15.40:0/32919175 >> v1:172.21.15.40:6789/0 conn(0x7f232406f7b0 legacy=0x7f232415f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:01.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.346+0000 7f2329a00640 1 -- 172.21.15.40:0/32919175 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232406fe20 con 0x7f2324160080 2024-03-24T00:37:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.350+0000 7f2329a00640 1 --2- 172.21.15.40:0/32919175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2324160080 0x7f2324168ea0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f231c009d90 tx=0x7f231c01f120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc67f679e9311fd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.350+0000 7f2323e00640 1 -- 172.21.15.40:0/32919175 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c0b4070 con 0x7f2324160080 2024-03-24T00:37:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.350+0000 7f2323e00640 1 -- 172.21.15.40:0/32919175 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c0af600 con 0x7f2324160080 2024-03-24T00:37:01.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.350+0000 7f2323e00640 1 -- 172.21.15.40:0/32919175 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c0af9d0 con 0x7f2324160080 2024-03-24T00:37:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.350+0000 7f232ba00640 1 -- 172.21.15.40:0/32919175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2324160080 msgr2=0x7f2324168ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.350+0000 7f232ba00640 1 --2- 172.21.15.40:0/32919175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2324160080 0x7f2324168ea0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f231c009d90 tx=0x7f231c01f120 comp rx=0 tx=0).stop 2024-03-24T00:37:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.350+0000 7f232ba00640 1 -- 172.21.15.40:0/32919175 shutdown_connections 2024-03-24T00:37:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.350+0000 7f232ba00640 1 --2- 172.21.15.40:0/32919175 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2324160080 0x7f2324168ea0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:01.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.350+0000 7f232ba00640 1 -- 172.21.15.40:0/32919175 >> 172.21.15.40:0/32919175 conn(0x7f23240b36a0 msgr2=0x7f23240b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.350+0000 7f232ba00640 1 -- 172.21.15.40:0/32919175 shutdown_connections 2024-03-24T00:37:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.350+0000 7f232ba00640 1 -- 172.21.15.40:0/32919175 wait complete. 2024-03-24T00:37:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f232ba00640 1 Processor -- start 2024-03-24T00:37:01.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f232ba00640 1 -- start start 2024-03-24T00:37:01.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f232ba00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f232406f7b0 0x7f23241530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:01.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f232ba00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f232415e170 0x7f23240b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:01.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f232ba00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2324160080 0x7f23240b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:01.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f232ba00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23240d0210 con 0x7f232415e170 2024-03-24T00:37:01.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f232ba00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324070100 con 0x7f232406f7b0 2024-03-24T00:37:01.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f232ba00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23240d04f0 con 0x7f2324160080 2024-03-24T00:37:01.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f231be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f232415e170 0x7f23240b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:01.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f231be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f232415e170 0x7f23240b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39982/0 (socket says 172.21.15.40:39982) 2024-03-24T00:37:01.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f231be00640 1 -- 172.21.15.40:0/2263969771 learned_addr learned my addr 172.21.15.40:0/2263969771 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:01.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f2329a00640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2324160080 0x7f23240b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f2329000640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f232406f7b0 0x7f23241530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f231be00640 1 -- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2324160080 msgr2=0x7f23240b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f231be00640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2324160080 0x7f23240b4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f231be00640 1 -- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f232406f7b0 msgr2=0x7f23241530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f231be00640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f232406f7b0 0x7f23241530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f231be00640 1 -- 172.21.15.40:0/2263969771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23240b2090 con 0x7f232415e170 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f2329a00640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2324160080 0x7f23240b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f2329000640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f232406f7b0 0x7f23241530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f231be00640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f232415e170 0x7f23240b46d0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f230c00c420 tx=0x7f230c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f2321600640 1 -- 172.21.15.40:0/2263969771 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c017070 con 0x7f232415e170 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f2321600640 1 -- 172.21.15.40:0/2263969771 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f230c005cf0 con 0x7f232415e170 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f2321600640 1 -- 172.21.15.40:0/2263969771 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230c0072f0 con 0x7f232415e170 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f232ba00640 1 -- 172.21.15.40:0/2263969771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23240b2370 con 0x7f232415e170 2024-03-24T00:37:01.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f232ba00640 1 -- 172.21.15.40:0/2263969771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23240b2930 con 0x7f232415e170 2024-03-24T00:37:01.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.354+0000 7f232ba00640 1 -- 172.21.15.40:0/2263969771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2324068170 con 0x7f232415e170 2024-03-24T00:37:01.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.358+0000 7f2321600640 1 -- 172.21.15.40:0/2263969771 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f230c01a070 con 0x7f232415e170 2024-03-24T00:37:01.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.358+0000 7f2321600640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2300067660 0x7f2300069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:01.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.358+0000 7f2321600640 1 -- 172.21.15.40:0/2263969771 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f230c084b60 con 0x7f232415e170 2024-03-24T00:37:01.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.358+0000 7f2329000640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2300067660 0x7f2300069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:01.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.358+0000 7f2329000640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2300067660 0x7f2300069b20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2314003ec0 tx=0x7f2314009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:01.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.362+0000 7f2321600640 1 -- 172.21.15.40:0/2263969771 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f23240cfcb0 con 0x7f232415e170 2024-03-24T00:37:01.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:01.562+0000 7f232ba00640 1 -- 172.21.15.40:0/2263969771 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2324144140 con 0x7f232415e170 2024-03-24T00:37:02.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.185+0000 7f2321600640 1 -- 172.21.15.40:0/2263969771 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v93) v1 ==== 174+0+0 (secure 0 0 0) 0x7f230c0558d0 con 0x7f232415e170 2024-03-24T00:37:02.199 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 14 and data pool 15 2024-03-24T00:37:02.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.197+0000 7f232ba00640 1 -- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2300067660 msgr2=0x7f2300069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.197+0000 7f232ba00640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2300067660 0x7f2300069b20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2314003ec0 tx=0x7f2314009040 comp rx=0 tx=0).stop 2024-03-24T00:37:02.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.197+0000 7f232ba00640 1 -- 172.21.15.40:0/2263969771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f232415e170 msgr2=0x7f23240b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.197+0000 7f232ba00640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f232415e170 0x7f23240b46d0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f230c00c420 tx=0x7f230c00c8e0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.197+0000 7f232ba00640 1 -- 172.21.15.40:0/2263969771 shutdown_connections 2024-03-24T00:37:02.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.197+0000 7f232ba00640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2300067660 0x7f2300069b20 secure :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f2314003ec0 tx=0x7f2314009040 comp rx=0 tx=0).stop 2024-03-24T00:37:02.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.197+0000 7f232ba00640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2324160080 0x7f23240b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.197+0000 7f232ba00640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f232415e170 0x7f23240b46d0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.197+0000 7f232ba00640 1 --2- 172.21.15.40:0/2263969771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f232406f7b0 0x7f23241530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.197+0000 7f232ba00640 1 -- 172.21.15.40:0/2263969771 >> 172.21.15.40:0/2263969771 conn(0x7f23240b36a0 msgr2=0x7f23240d20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:02.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.197+0000 7f232ba00640 1 -- 172.21.15.40:0/2263969771 shutdown_connections 2024-03-24T00:37:02.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.201+0000 7f232ba00640 1 -- 172.21.15.40:0/2263969771 wait complete. 2024-03-24T00:37:02.239 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:02.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5fb200640 1 Processor -- start 2024-03-24T00:37:02.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5fb200640 1 -- start start 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5fb200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb5f406f100 0x7fb5f4070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5fb200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5f4070030 0x7fb5f40af170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5fb200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f4070440 con 0x7fb5f406f4e0 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5fb200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f40d27b0 con 0x7fb5f4070030 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5fb200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f40b1490 con 0x7fb5f406f100 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5f9200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5f4070030 0x7fb5f40af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5f9200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb5f4070030 0x7fb5f40af170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39988/0 (socket says 172.21.15.40:39988) 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5f9200640 1 -- 172.21.15.40:0/3133036565 learned_addr learned my addr 172.21.15.40:0/3133036565 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5eb400640 1 --1- 172.21.15.40:0/3133036565 >> v1:172.21.15.40:6789/0 conn(0x7fb5f406f4e0 0x7fb5f40aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44822/0 (socket says 172.21.15.40:44822) 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5f3e00640 1 --2- 172.21.15.40:0/3133036565 >> v2:172.21.15.53:3300/0 conn(0x7fb5f406f100 0x7fb5f4070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5f9200640 1 -- 172.21.15.40:0/3133036565 >> v2:172.21.15.53:3300/0 conn(0x7fb5f406f100 msgr2=0x7fb5f4070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5f9200640 1 --2- 172.21.15.40:0/3133036565 >> v2:172.21.15.53:3300/0 conn(0x7fb5f406f100 0x7fb5f4070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5f9200640 1 -- 172.21.15.40:0/3133036565 >> v1:172.21.15.40:6789/0 conn(0x7fb5f406f4e0 legacy=0x7fb5f40aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.377+0000 7fb5f9200640 1 -- 172.21.15.40:0/3133036565 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f406fa70 con 0x7fb5f4070030 2024-03-24T00:37:02.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.381+0000 7fb5f9200640 1 --2- 172.21.15.40:0/3133036565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f4070030 0x7fb5f40af170 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4071870 tx=0x7fb5ec01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77834a61bb05692a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:02.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.381+0000 7fb5f3400640 1 -- 172.21.15.40:0/3133036565 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec0a57a0 con 0x7fb5f4070030 2024-03-24T00:37:02.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.381+0000 7fb5f3400640 1 -- 172.21.15.40:0/3133036565 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ec0a5940 con 0x7fb5f4070030 2024-03-24T00:37:02.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.381+0000 7fb5f3400640 1 -- 172.21.15.40:0/3133036565 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec0a5d10 con 0x7fb5f4070030 2024-03-24T00:37:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.381+0000 7fb5fb200640 1 -- 172.21.15.40:0/3133036565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f4070030 msgr2=0x7fb5f40af170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.381+0000 7fb5fb200640 1 --2- 172.21.15.40:0/3133036565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f4070030 0x7fb5f40af170 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4071870 tx=0x7fb5ec01c9f0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.381+0000 7fb5fb200640 1 -- 172.21.15.40:0/3133036565 shutdown_connections 2024-03-24T00:37:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.381+0000 7fb5fb200640 1 --2- 172.21.15.40:0/3133036565 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f4070030 0x7fb5f40af170 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.381+0000 7fb5fb200640 1 --2- 172.21.15.40:0/3133036565 >> v2:172.21.15.53:3300/0 conn(0x7fb5f406f100 0x7fb5f4070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.381+0000 7fb5fb200640 1 -- 172.21.15.40:0/3133036565 >> 172.21.15.40:0/3133036565 conn(0x7fb5f40d1c90 msgr2=0x7fb5f40d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.381+0000 7fb5fb200640 1 -- 172.21.15.40:0/3133036565 shutdown_connections 2024-03-24T00:37:02.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.381+0000 7fb5fb200640 1 -- 172.21.15.40:0/3133036565 wait complete. 2024-03-24T00:37:02.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5fb200640 1 Processor -- start 2024-03-24T00:37:02.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5fb200640 1 -- start start 2024-03-24T00:37:02.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5fb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f406f100 0x7fb5f40cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5fb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5f406f4e0 0x7fb5f4153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5fb200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5f4070030 0x7fb5f4158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5fb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f40d0a70 con 0x7fb5f406f100 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5fb200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f40d08f0 con 0x7fb5f406f4e0 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5fb200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f40af6b0 con 0x7fb5f4070030 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f406f100 0x7fb5f40cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f406f100 0x7fb5f40cb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39990/0 (socket says 172.21.15.40:39990) 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5f3e00640 1 -- 172.21.15.40:0/1980165363 learned_addr learned my addr 172.21.15.40:0/1980165363 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5eb400640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5f406f4e0 0x7fb5f4153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5f3e00640 1 -- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5f4070030 msgr2=0x7fb5f4158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5f3e00640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5f4070030 0x7fb5f4158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5f3e00640 1 -- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5f406f4e0 msgr2=0x7fb5f4153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5f3e00640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5f406f4e0 0x7fb5f4153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5f3e00640 1 -- 172.21.15.40:0/1980165363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f40c9d00 con 0x7fb5f406f100 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5eb400640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5f406f4e0 0x7fb5f4153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5f3e00640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f406f100 0x7fb5f40cb750 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fb5e400d800 tx=0x7fb5e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5f0c00640 1 -- 172.21.15.40:0/1980165363 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e400b6c0 con 0x7fb5f406f100 2024-03-24T00:37:02.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5f0c00640 1 -- 172.21.15.40:0/1980165363 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e4019070 con 0x7fb5f406f100 2024-03-24T00:37:02.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5f0c00640 1 -- 172.21.15.40:0/1980165363 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e400ba60 con 0x7fb5f406f100 2024-03-24T00:37:02.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5fb200640 1 -- 172.21.15.40:0/1980165363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f40ca990 con 0x7fb5f406f100 2024-03-24T00:37:02.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.385+0000 7fb5fb200640 1 -- 172.21.15.40:0/1980165363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f40caf50 con 0x7fb5f406f100 2024-03-24T00:37:02.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.389+0000 7fb5fb200640 1 -- 172.21.15.40:0/1980165363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f4068170 con 0x7fb5f406f100 2024-03-24T00:37:02.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.389+0000 7fb5f0c00640 1 -- 172.21.15.40:0/1980165363 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb5e400bc00 con 0x7fb5f406f100 2024-03-24T00:37:02.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.389+0000 7fb5f0c00640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb5d0067660 0x7fb5d0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.389+0000 7fb5f0c00640 1 -- 172.21.15.40:0/1980165363 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5e4085d80 con 0x7fb5f406f100 2024-03-24T00:37:02.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.389+0000 7fb5eb400640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb5d0067660 0x7fb5d0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.389+0000 7fb5eb400640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb5d0067660 0x7fb5d0069b20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb5f41569d0 tx=0x7fb5dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:02.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.393+0000 7fb5f0c00640 1 -- 172.21.15.40:0/1980165363 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb5e40569f0 con 0x7fb5f406f100 2024-03-24T00:37:02.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.557+0000 7fb5fb200640 1 -- 172.21.15.40:0/1980165363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5f40b4e60 con 0x7fb5f406f100 2024-03-24T00:37:02.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.557+0000 7fb5f0c00640 1 -- 172.21.15.40:0/1980165363 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16561 (secure 0 0 0) 0x7fb5e40562f0 con 0x7fb5f406f100 2024-03-24T00:37:02.563 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:02.563 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":94,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:37:01.566639+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:59.977310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-24T00:37:00.557420+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.561+0000 7fb5fb200640 1 -- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb5d0067660 msgr2=0x7fb5d0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.561+0000 7fb5fb200640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb5d0067660 0x7fb5d0069b20 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fb5f41569d0 tx=0x7fb5dc009040 comp rx=0 tx=0).stop 2024-03-24T00:37:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.561+0000 7fb5fb200640 1 -- 172.21.15.40:0/1980165363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f406f100 msgr2=0x7fb5f40cb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.561+0000 7fb5fb200640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f406f100 0x7fb5f40cb750 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fb5e400d800 tx=0x7fb5e400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.561+0000 7fb5fb200640 1 -- 172.21.15.40:0/1980165363 shutdown_connections 2024-03-24T00:37:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.561+0000 7fb5fb200640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb5d0067660 0x7fb5d0069b20 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.561+0000 7fb5fb200640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb5f4070030 0x7fb5f4158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.561+0000 7fb5fb200640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5f406f4e0 0x7fb5f4153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.561+0000 7fb5fb200640 1 --2- 172.21.15.40:0/1980165363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb5f406f100 0x7fb5f40cb750 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.561+0000 7fb5fb200640 1 -- 172.21.15.40:0/1980165363 >> 172.21.15.40:0/1980165363 conn(0x7fb5f40d1c90 msgr2=0x7fb5f40aea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.561+0000 7fb5fb200640 1 -- 172.21.15.40:0/1980165363 shutdown_connections 2024-03-24T00:37:02.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.561+0000 7fb5fb200640 1 -- 172.21.15.40:0/1980165363 wait complete. 2024-03-24T00:37:02.585 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3bec00640 1 Processor -- start 2024-03-24T00:37:02.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3bec00640 1 -- start start 2024-03-24T00:37:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3bec00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8072310 0x7fd3b80ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3bec00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b8070e00 con 0x7fd3b813ec80 2024-03-24T00:37:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3bec00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3b80cee20 con 0x7fd3b8141bc0 2024-03-24T00:37:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3bec00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b80cf100 con 0x7fd3b8072310 2024-03-24T00:37:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd3b813ec80 0x7fd3b81414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44826/0 (socket says 172.21.15.40:44826) 2024-03-24T00:37:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7e00640 1 -- 172.21.15.40:0/3522062092 learned_addr learned my addr 172.21.15.40:0/3522062092 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3bcc00640 1 --2- 172.21.15.40:0/3522062092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8072310 0x7fd3b80ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707093750 0 0) 0x7fd3b8070e00 con 0x7fd3b813ec80 2024-03-24T00:37:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd39c003530 con 0x7fd3b813ec80 2024-03-24T00:37:02.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701601419 0 0) 0x7fd3b80cee20 con 0x7fd3b8141bc0 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3b8070e00 con 0x7fd3b8141bc0 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4205972459 0 0) 0x7fd39c003530 con 0x7fd3b813ec80 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8072310 msgr2=0x7fd3b80ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 --2- 172.21.15.40:0/3522062092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8072310 0x7fd3b80ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 >> v1:172.21.15.53:6789/0 conn(0x7fd3b8141bc0 legacy=0x7fd3b80ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b80cf3e0 con 0x7fd3b813ec80 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fd3a8001680 con 0x7fd3b813ec80 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 >> v1:172.21.15.40:6789/0 conn(0x7fd3b813ec80 legacy=0x7fd3b81414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 --2- 172.21.15.40:0/3522062092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c004be0 0x7fd39c004fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 --2- 172.21.15.40:0/3522062092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8072310 0x7fd39c0037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 --2- 172.21.15.40:0/3522062092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd39c003cf0 0x7fd39c004170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b80cf100 con 0x7fd39c004be0 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c00ba80 con 0x7fd39c003cf0 2024-03-24T00:37:02.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.721+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c00bc80 con 0x7fd3b8072310 2024-03-24T00:37:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3b7e00640 1 --2- 172.21.15.40:0/3522062092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c004be0 0x7fd39c004fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3bcc00640 1 --2- 172.21.15.40:0/3522062092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd39c003cf0 0x7fd39c004170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3af400640 1 --2- 172.21.15.40:0/3522062092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8072310 0x7fd39c0037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3b7e00640 1 -- 172.21.15.40:0/3522062092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8072310 msgr2=0x7fd39c0037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3b7e00640 1 --2- 172.21.15.40:0/3522062092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8072310 0x7fd39c0037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3b7e00640 1 -- 172.21.15.40:0/3522062092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd39c003cf0 msgr2=0x7fd39c004170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3b7e00640 1 --2- 172.21.15.40:0/3522062092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd39c003cf0 0x7fd39c004170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3b7e00640 1 -- 172.21.15.40:0/3522062092 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c003530 con 0x7fd39c004be0 2024-03-24T00:37:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3b7e00640 1 --2- 172.21.15.40:0/3522062092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c004be0 0x7fd39c004fe0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fd3a800c1c0 tx=0x7fd3a801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3f87c696da91c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8007cf0 con 0x7fd39c004be0 2024-03-24T00:37:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a80a9020 con 0x7fd39c004be0 2024-03-24T00:37:02.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3b7400640 1 -- 172.21.15.40:0/3522062092 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a80a5b40 con 0x7fd39c004be0 2024-03-24T00:37:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3bec00640 1 -- 172.21.15.40:0/3522062092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c004be0 msgr2=0x7fd39c004fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3bec00640 1 --2- 172.21.15.40:0/3522062092 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c004be0 0x7fd39c004fe0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fd3a800c1c0 tx=0x7fd3a801cc20 comp rx=0 tx=0).stop 2024-03-24T00:37:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.725+0000 7fd3bcc00640 1 -- 172.21.15.40:0/3522062092 reap_dead start 2024-03-24T00:37:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bec00640 1 -- 172.21.15.40:0/3522062092 shutdown_connections 2024-03-24T00:37:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bec00640 1 -- 172.21.15.40:0/3522062092 >> 172.21.15.40:0/3522062092 conn(0x7fd3b80b17b0 msgr2=0x7fd3b80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bec00640 1 -- 172.21.15.40:0/3522062092 shutdown_connections 2024-03-24T00:37:02.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bec00640 1 -- 172.21.15.40:0/3522062092 wait complete. 2024-03-24T00:37:02.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bec00640 1 Processor -- start 2024-03-24T00:37:02.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bec00640 1 -- start start 2024-03-24T00:37:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c004be0 0x7fd3b81645b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bec00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3b8150a40 0x7fd3b813bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bec00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8153fc0 0x7fd3b8154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b81678d0 con 0x7fd39c004be0 2024-03-24T00:37:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bec00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b8167750 con 0x7fd3b8150a40 2024-03-24T00:37:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bec00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b8167a50 con 0x7fd3b8153fc0 2024-03-24T00:37:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3b7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c004be0 0x7fd3b81645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3b7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c004be0 0x7fd3b81645b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40000/0 (socket says 172.21.15.40:40000) 2024-03-24T00:37:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3b7e00640 1 -- 172.21.15.40:0/2803243807 learned_addr learned my addr 172.21.15.40:0/2803243807 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3af400640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3b8150a40 0x7fd3b813bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3bcc00640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8153fc0 0x7fd3b8154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3b7e00640 1 -- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8153fc0 msgr2=0x7fd3b8154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3b7e00640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8153fc0 0x7fd3b8154440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3b7e00640 1 -- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3b8150a40 msgr2=0x7fd3b813bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3b7e00640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3b8150a40 0x7fd3b813bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3b7e00640 1 -- 172.21.15.40:0/2803243807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b813d590 con 0x7fd39c004be0 2024-03-24T00:37:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.729+0000 7fd3b7e00640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c004be0 0x7fd3b81645b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8007040 tx=0x7fd39c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:02.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.733+0000 7fd3b4c00640 1 -- 172.21.15.40:0/2803243807 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8007cf0 con 0x7fd39c004be0 2024-03-24T00:37:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.733+0000 7fd3b4c00640 1 -- 172.21.15.40:0/2803243807 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a80049f0 con 0x7fd39c004be0 2024-03-24T00:37:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.733+0000 7fd3b4c00640 1 -- 172.21.15.40:0/2803243807 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8001c20 con 0x7fd39c004be0 2024-03-24T00:37:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.733+0000 7fd3bec00640 1 -- 172.21.15.40:0/2803243807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b8150fb0 con 0x7fd39c004be0 2024-03-24T00:37:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.733+0000 7fd3bec00640 1 -- 172.21.15.40:0/2803243807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b8151570 con 0x7fd39c004be0 2024-03-24T00:37:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.733+0000 7fd3bec00640 1 -- 172.21.15.40:0/2803243807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b8068170 con 0x7fd39c004be0 2024-03-24T00:37:02.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.733+0000 7fd3b4c00640 1 -- 172.21.15.40:0/2803243807 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd3a8007110 con 0x7fd39c004be0 2024-03-24T00:37:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.733+0000 7fd3b4c00640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd394067660 0x7fd394069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.733+0000 7fd3b4c00640 1 -- 172.21.15.40:0/2803243807 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3a811f8a0 con 0x7fd39c004be0 2024-03-24T00:37:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.733+0000 7fd3af400640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd394067660 0x7fd394069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.737+0000 7fd3af400640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd394067660 0x7fd394069b20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0007fd0 tx=0x7fd3a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:02.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.737+0000 7fd3b4c00640 1 -- 172.21.15.40:0/2803243807 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd3a8016930 con 0x7fd39c004be0 2024-03-24T00:37:02.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.905+0000 7fd3bec00640 1 -- 172.21.15.40:0/2803243807 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3b8072b90 con 0x7fd39c004be0 2024-03-24T00:37:02.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.905+0000 7fd3b4c00640 1 -- 172.21.15.40:0/2803243807 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16561 (secure 0 0 0) 0x7fd3a80eff50 con 0x7fd39c004be0 2024-03-24T00:37:02.909 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:02.910 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":94,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:37:01.566639+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:59.977310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-24T00:37:00.557420+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.905+0000 7fd3bec00640 1 -- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd394067660 msgr2=0x7fd394069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.905+0000 7fd3bec00640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd394067660 0x7fd394069b20 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0007fd0 tx=0x7fd3a0009040 comp rx=0 tx=0).stop 2024-03-24T00:37:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.905+0000 7fd3bec00640 1 -- 172.21.15.40:0/2803243807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c004be0 msgr2=0x7fd3b81645b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.905+0000 7fd3bec00640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c004be0 0x7fd3b81645b0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8007040 tx=0x7fd39c005840 comp rx=0 tx=0).stop 2024-03-24T00:37:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.905+0000 7fd3bec00640 1 -- 172.21.15.40:0/2803243807 shutdown_connections 2024-03-24T00:37:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.905+0000 7fd3bec00640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd394067660 0x7fd394069b20 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.905+0000 7fd3bec00640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3b8153fc0 0x7fd3b8154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.905+0000 7fd3bec00640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3b8150a40 0x7fd3b813bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.905+0000 7fd3bec00640 1 --2- 172.21.15.40:0/2803243807 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd39c004be0 0x7fd3b81645b0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.905+0000 7fd3bec00640 1 -- 172.21.15.40:0/2803243807 >> 172.21.15.40:0/2803243807 conn(0x7fd3b80b17b0 msgr2=0x7fd3b813f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.909+0000 7fd3bec00640 1 -- 172.21.15.40:0/2803243807 shutdown_connections 2024-03-24T00:37:02.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:02.909+0000 7fd3bec00640 1 -- 172.21.15.40:0/2803243807 wait complete. 2024-03-24T00:37:02.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce78800640 1 Processor -- start 2024-03-24T00:37:03.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce78800640 1 -- start start 2024-03-24T00:37:03.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce78800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fce7015e350 0x7fce700cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce78800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce700c9580 0x7fce700c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce78800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce700ad1e0 0x7fce700ad630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce78800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce700cd660 con 0x7fce700ad1e0 2024-03-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce78800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce700c9f70 con 0x7fce7015e350 2024-03-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce78800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce700734e0 con 0x7fce700c9580 2024-03-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce76800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce700ad1e0 0x7fce700ad630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce76800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fce700ad1e0 0x7fce700ad630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40012/0 (socket says 172.21.15.40:40012) 2024-03-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce76800640 1 -- 172.21.15.40:0/3134855437 learned_addr learned my addr 172.21.15.40:0/3134855437 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce75400640 1 --2- 172.21.15.40:0/3134855437 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce700c9580 0x7fce700c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:03.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce75e00640 1 --2- 172.21.15.40:0/3134855437 >> v2:172.21.15.53:3300/0 conn(0x7fce7015e350 0x7fce700cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce76800640 1 -- 172.21.15.40:0/3134855437 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce700c9580 msgr2=0x7fce700c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce76800640 1 --2- 172.21.15.40:0/3134855437 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce700c9580 0x7fce700c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce76800640 1 -- 172.21.15.40:0/3134855437 >> v2:172.21.15.53:3300/0 conn(0x7fce7015e350 msgr2=0x7fce700cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce76800640 1 --2- 172.21.15.40:0/3134855437 >> v2:172.21.15.53:3300/0 conn(0x7fce7015e350 0x7fce700cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:03.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.073+0000 7fce76800640 1 -- 172.21.15.40:0/3134855437 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce700d0030 con 0x7fce700ad1e0 2024-03-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce76800640 1 --2- 172.21.15.40:0/3134855437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce700ad1e0 0x7fce700ad630 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fce70070d30 tx=0x7fce6c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88922413c2b6bfc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce74a00640 1 -- 172.21.15.40:0/3134855437 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c0a5780 con 0x7fce700ad1e0 2024-03-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce74a00640 1 -- 172.21.15.40:0/3134855437 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6c0a5920 con 0x7fce700ad1e0 2024-03-24T00:37:03.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce74a00640 1 -- 172.21.15.40:0/3134855437 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c0a5cf0 con 0x7fce700ad1e0 2024-03-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce78800640 1 -- 172.21.15.40:0/3134855437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce700ad1e0 msgr2=0x7fce700ad630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce78800640 1 --2- 172.21.15.40:0/3134855437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce700ad1e0 0x7fce700ad630 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fce70070d30 tx=0x7fce6c01c9f0 comp rx=0 tx=0).stop 2024-03-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce78800640 1 -- 172.21.15.40:0/3134855437 shutdown_connections 2024-03-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce78800640 1 --2- 172.21.15.40:0/3134855437 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce700ad1e0 0x7fce700ad630 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce78800640 1 --2- 172.21.15.40:0/3134855437 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce700c9580 0x7fce700c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce78800640 1 --2- 172.21.15.40:0/3134855437 >> v2:172.21.15.53:3300/0 conn(0x7fce7015e350 0x7fce700cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce78800640 1 -- 172.21.15.40:0/3134855437 >> 172.21.15.40:0/3134855437 conn(0x7fce70072870 msgr2=0x7fce70072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce78800640 1 -- 172.21.15.40:0/3134855437 shutdown_connections 2024-03-24T00:37:03.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce78800640 1 -- 172.21.15.40:0/3134855437 wait complete. 2024-03-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce78800640 1 Processor -- start 2024-03-24T00:37:03.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.077+0000 7fce78800640 1 -- start start 2024-03-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce78800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce700ad1e0 0x7fce70156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:03.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce78800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce700c9580 0x7fce70152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce78800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce7015e350 0x7fce701597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce78800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce700cda30 con 0x7fce700c9580 2024-03-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce78800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce700716b0 con 0x7fce700ad1e0 2024-03-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce78800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce700cdbb0 con 0x7fce7015e350 2024-03-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce75400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce700c9580 0x7fce70152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce75400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce700c9580 0x7fce70152770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40024/0 (socket says 172.21.15.40:40024) 2024-03-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce75400640 1 -- 172.21.15.40:0/949129407 learned_addr learned my addr 172.21.15.40:0/949129407 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce75e00640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce700ad1e0 0x7fce70156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce76800640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce7015e350 0x7fce701597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce75400640 1 -- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce7015e350 msgr2=0x7fce701597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce75400640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce7015e350 0x7fce701597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce75400640 1 -- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce700ad1e0 msgr2=0x7fce70156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce75400640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce700ad1e0 0x7fce70156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce75400640 1 -- 172.21.15.40:0/949129407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce701572b0 con 0x7fce700c9580 2024-03-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce75400640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce700c9580 0x7fce70152770 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fce6000e8b0 tx=0x7fce6000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce6a000640 1 -- 172.21.15.40:0/949129407 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce60017070 con 0x7fce700c9580 2024-03-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce6a000640 1 -- 172.21.15.40:0/949129407 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6000c940 con 0x7fce700c9580 2024-03-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce78800640 1 -- 172.21.15.40:0/949129407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7015a230 con 0x7fce700c9580 2024-03-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce6a000640 1 -- 172.21.15.40:0/949129407 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6000cd10 con 0x7fce700c9580 2024-03-24T00:37:03.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce78800640 1 -- 172.21.15.40:0/949129407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce70156570 con 0x7fce700c9580 2024-03-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.081+0000 7fce78800640 1 -- 172.21.15.40:0/949129407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce70068170 con 0x7fce700c9580 2024-03-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.085+0000 7fce6a000640 1 -- 172.21.15.40:0/949129407 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fce6001a070 con 0x7fce700c9580 2024-03-24T00:37:03.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.085+0000 7fce6a000640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce54067660 0x7fce54069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.085+0000 7fce6a000640 1 -- 172.21.15.40:0/949129407 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce60085cd0 con 0x7fce700c9580 2024-03-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.085+0000 7fce75e00640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce54067660 0x7fce54069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:03.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.085+0000 7fce75e00640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce54067660 0x7fce54069b20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fce64009af0 tx=0x7fce64008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:03.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.089+0000 7fce6a000640 1 -- 172.21.15.40:0/949129407 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fce600568b0 con 0x7fce700c9580 2024-03-24T00:37:03.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:03.293+0000 7fce78800640 1 -- 172.21.15.40:0/949129407 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fce7006d520 con 0x7fce700c9580 2024-03-24T00:37:04.149 INFO:tasks.ceph.mds.d.smithi053.stderr:2024-03-24T00:37:04.147+0000 7fcea3600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:37:04.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce6a000640 1 -- 172.21.15.40:0/949129407 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v96) v1 ==== 118+0+0 (secure 0 0 0) 0x7fce600561b0 con 0x7fce700c9580 2024-03-24T00:37:04.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce78800640 1 -- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce54067660 msgr2=0x7fce54069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:04.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce78800640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce54067660 0x7fce54069b20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fce64009af0 tx=0x7fce64008040 comp rx=0 tx=0).stop 2024-03-24T00:37:04.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce78800640 1 -- 172.21.15.40:0/949129407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce700c9580 msgr2=0x7fce70152770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:04.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce78800640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce700c9580 0x7fce70152770 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fce6000e8b0 tx=0x7fce6000ed70 comp rx=0 tx=0).stop 2024-03-24T00:37:04.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce78800640 1 -- 172.21.15.40:0/949129407 shutdown_connections 2024-03-24T00:37:04.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce78800640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce54067660 0x7fce54069b20 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:04.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce78800640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fce7015e350 0x7fce701597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:04.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce78800640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fce700c9580 0x7fce70152770 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:04.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce78800640 1 --2- 172.21.15.40:0/949129407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce700ad1e0 0x7fce70156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:04.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce78800640 1 -- 172.21.15.40:0/949129407 >> 172.21.15.40:0/949129407 conn(0x7fce70072870 msgr2=0x7fce7006eb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:04.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce78800640 1 -- 172.21.15.40:0/949129407 shutdown_connections 2024-03-24T00:37:04.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.197+0000 7fce78800640 1 -- 172.21.15.40:0/949129407 wait complete. 2024-03-24T00:37:04.228 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:37:04.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:37:04.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f701ca00640 1 Processor -- start 2024-03-24T00:37:04.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f701ca00640 1 -- start start 2024-03-24T00:37:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f701ca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f701815ffd0 0x7f70180ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f701ca00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70180728d0 0x7f70180cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f701ca00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7018071de0 con 0x7f701815bd00 2024-03-24T00:37:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f701ca00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70180cf790 con 0x7f70180728d0 2024-03-24T00:37:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f701ca00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70180cea50 con 0x7f701815ffd0 2024-03-24T00:37:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f7016800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70180728d0 0x7f70180cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f7016800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70180728d0 0x7f70180cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40036/0 (socket says 172.21.15.40:40036) 2024-03-24T00:37:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f7016800640 1 -- 172.21.15.40:0/3686453174 learned_addr learned my addr 172.21.15.40:0/3686453174 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f700fe00640 1 --2- 172.21.15.40:0/3686453174 >> v2:172.21.15.53:3300/0 conn(0x7f701815ffd0 0x7f70180ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f7016800640 1 -- 172.21.15.40:0/3686453174 >> v2:172.21.15.53:3300/0 conn(0x7f701815ffd0 msgr2=0x7f70180ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f7016800640 1 --2- 172.21.15.40:0/3686453174 >> v2:172.21.15.53:3300/0 conn(0x7f701815ffd0 0x7f70180ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:04.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f7016800640 1 -- 172.21.15.40:0/3686453174 >> v1:172.21.15.53:6789/0 conn(0x7f701815bd00 legacy=0x7f701815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:04.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.373+0000 7f7016800640 1 -- 172.21.15.40:0/3686453174 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70180ced30 con 0x7f70180728d0 2024-03-24T00:37:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.377+0000 7f7016800640 1 --2- 172.21.15.40:0/3686453174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70180728d0 0x7f70180cf250 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f701806ff40 tx=0x7f701001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4d948a4172eff1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.377+0000 7f7015400640 1 -- 172.21.15.40:0/3686453174 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70100a5860 con 0x7f70180728d0 2024-03-24T00:37:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.377+0000 7f7015400640 1 -- 172.21.15.40:0/3686453174 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70100a5a00 con 0x7f70180728d0 2024-03-24T00:37:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.377+0000 7f7015400640 1 -- 172.21.15.40:0/3686453174 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70100a5dd0 con 0x7f70180728d0 2024-03-24T00:37:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.377+0000 7f701ca00640 1 -- 172.21.15.40:0/3686453174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70180728d0 msgr2=0x7f70180cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.377+0000 7f701ca00640 1 --2- 172.21.15.40:0/3686453174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70180728d0 0x7f70180cf250 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f701806ff40 tx=0x7f701001c9c0 comp rx=0 tx=0).stop 2024-03-24T00:37:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.377+0000 7f701ca00640 1 -- 172.21.15.40:0/3686453174 shutdown_connections 2024-03-24T00:37:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.377+0000 7f701ca00640 1 --2- 172.21.15.40:0/3686453174 >> v2:172.21.15.53:3300/0 conn(0x7f701815ffd0 0x7f70180ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.377+0000 7f701ca00640 1 --2- 172.21.15.40:0/3686453174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f70180728d0 0x7f70180cf250 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.377+0000 7f701ca00640 1 -- 172.21.15.40:0/3686453174 >> 172.21.15.40:0/3686453174 conn(0x7f70180b1660 msgr2=0x7f70180b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.377+0000 7f701ca00640 1 -- 172.21.15.40:0/3686453174 shutdown_connections 2024-03-24T00:37:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.377+0000 7f701ca00640 1 -- 172.21.15.40:0/3686453174 wait complete. 2024-03-24T00:37:04.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f701ca00640 1 Processor -- start 2024-03-24T00:37:04.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f701ca00640 1 -- start start 2024-03-24T00:37:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f701ca00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70180728d0 0x7f7018150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f701ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701815bd00 0x7f70180b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f701ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f701815ffd0 0x7f701815a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f701ca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701806d320 con 0x7f701815ffd0 2024-03-24T00:37:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f701ca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701815c0e0 con 0x7f701815bd00 2024-03-24T00:37:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f701ca00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701806d4a0 con 0x7f70180728d0 2024-03-24T00:37:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f7016800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f701815ffd0 0x7f701815a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f7016800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f701815ffd0 0x7f701815a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40038/0 (socket says 172.21.15.40:40038) 2024-03-24T00:37:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f7016800640 1 -- 172.21.15.40:0/3690876665 learned_addr learned my addr 172.21.15.40:0/3690876665 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f700fe00640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701815bd00 0x7f70180b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f7015e00640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70180728d0 0x7f7018150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:04.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f7016800640 1 -- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70180728d0 msgr2=0x7f7018150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f7016800640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70180728d0 0x7f7018150590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f7016800640 1 -- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701815bd00 msgr2=0x7f70180b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f7016800640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701815bd00 0x7f70180b0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f7016800640 1 -- 172.21.15.40:0/3690876665 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701815f3b0 con 0x7f701815ffd0 2024-03-24T00:37:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f700fe00640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701815bd00 0x7f70180b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f7016800640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f701815ffd0 0x7f701815a840 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f70100195d0 tx=0x7f7010019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f700e000640 1 -- 172.21.15.40:0/3690876665 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70100b8670 con 0x7f701815ffd0 2024-03-24T00:37:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f700e000640 1 -- 172.21.15.40:0/3690876665 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70100be070 con 0x7f701815ffd0 2024-03-24T00:37:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f700e000640 1 -- 172.21.15.40:0/3690876665 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70100b8a70 con 0x7f701815ffd0 2024-03-24T00:37:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f701ca00640 1 -- 172.21.15.40:0/3690876665 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701814e050 con 0x7f701815ffd0 2024-03-24T00:37:04.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f701ca00640 1 -- 172.21.15.40:0/3690876665 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7018150e50 con 0x7f701815ffd0 2024-03-24T00:37:04.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.381+0000 7f701ca00640 1 -- 172.21.15.40:0/3690876665 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7018068170 con 0x7f701815ffd0 2024-03-24T00:37:04.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.385+0000 7f700e000640 1 -- 172.21.15.40:0/3690876665 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f70100b8c10 con 0x7f701815ffd0 2024-03-24T00:37:04.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.385+0000 7f700e000640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6ff4067660 0x7f6ff4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:04.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.385+0000 7f700e000640 1 -- 172.21.15.40:0/3690876665 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f70101209d0 con 0x7f701815ffd0 2024-03-24T00:37:04.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.385+0000 7f7015e00640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6ff4067660 0x7f6ff4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:04.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.385+0000 7f7015e00640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6ff4067660 0x7f6ff4069b20 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7008007fd0 tx=0x7f7008009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:04.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.389+0000 7f700e000640 1 -- 172.21.15.40:0/3690876665 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f70100165e0 con 0x7f701815ffd0 2024-03-24T00:37:04.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:04.585+0000 7f701ca00640 1 -- 172.21.15.40:0/3690876665 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f70181556c0 con 0x7f701815ffd0 2024-03-24T00:37:05.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.197+0000 7f700e000640 1 -- 172.21.15.40:0/3690876665 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v97) v1 ==== 122+0+0 (secure 0 0 0) 0x7f70100f1120 con 0x7f701815ffd0 2024-03-24T00:37:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.201+0000 7f701ca00640 1 -- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6ff4067660 msgr2=0x7f6ff4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.201+0000 7f701ca00640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6ff4067660 0x7f6ff4069b20 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7008007fd0 tx=0x7f7008009040 comp rx=0 tx=0).stop 2024-03-24T00:37:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.201+0000 7f701ca00640 1 -- 172.21.15.40:0/3690876665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f701815ffd0 msgr2=0x7f701815a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.201+0000 7f701ca00640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f701815ffd0 0x7f701815a840 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f70100195d0 tx=0x7f7010019760 comp rx=0 tx=0).stop 2024-03-24T00:37:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.201+0000 7f701ca00640 1 -- 172.21.15.40:0/3690876665 shutdown_connections 2024-03-24T00:37:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.201+0000 7f701ca00640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6ff4067660 0x7f6ff4069b20 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.201+0000 7f701ca00640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f701815ffd0 0x7f701815a840 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.201+0000 7f701ca00640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701815bd00 0x7f70180b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.201+0000 7f701ca00640 1 --2- 172.21.15.40:0/3690876665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70180728d0 0x7f7018150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.201+0000 7f701ca00640 1 -- 172.21.15.40:0/3690876665 >> 172.21.15.40:0/3690876665 conn(0x7f70180b1660 msgr2=0x7f7018155ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.201+0000 7f701ca00640 1 -- 172.21.15.40:0/3690876665 shutdown_connections 2024-03-24T00:37:05.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.201+0000 7f701ca00640 1 -- 172.21.15.40:0/3690876665 wait complete. 2024-03-24T00:37:05.226 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:37:05.226 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90afa00640 1 Processor -- start 2024-03-24T00:37:05.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90afa00640 1 -- start start 2024-03-24T00:37:05.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90afa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f90a806e5a0 0x7f90a806e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:05.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90afa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f90a8070c80 0x7f90a806db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:05.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90afa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90a8071690 0x7f90a80b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:05.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90afa00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a806cdc0 con 0x7f90a806e5a0 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90afa00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a806eec0 con 0x7f90a8070c80 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90afa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a8071160 con 0x7f90a8071690 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90ada00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90a8071690 0x7f90a80b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f909fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f90a8070c80 0x7f90a806db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90ada00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90a8071690 0x7f90a80b33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:40780/0 (socket says 172.21.15.40:40780) 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90ada00640 1 -- 172.21.15.40:0/2446181364 learned_addr learned my addr 172.21.15.40:0/2446181364 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90ad000640 1 --2- 172.21.15.40:0/2446181364 >> v2:172.21.15.40:3300/0 conn(0x7f90a806e5a0 0x7f90a806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90ada00640 1 -- 172.21.15.40:0/2446181364 >> v2:172.21.15.53:3300/0 conn(0x7f90a8070c80 msgr2=0x7f90a806db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90ada00640 1 --2- 172.21.15.40:0/2446181364 >> v2:172.21.15.53:3300/0 conn(0x7f90a8070c80 0x7f90a806db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90ada00640 1 -- 172.21.15.40:0/2446181364 >> v2:172.21.15.40:3300/0 conn(0x7f90a806e5a0 msgr2=0x7f90a806e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90ada00640 1 --2- 172.21.15.40:0/2446181364 >> v2:172.21.15.40:3300/0 conn(0x7f90a806e5a0 0x7f90a806e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.365+0000 7f90ada00640 1 -- 172.21.15.40:0/2446181364 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a806d040 con 0x7f90a8071690 2024-03-24T00:37:05.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f909fe00640 1 --2- 172.21.15.40:0/2446181364 >> v2:172.21.15.53:3300/0 conn(0x7f90a8070c80 0x7f90a806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:05.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90ada00640 1 --2- 172.21.15.40:0/2446181364 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90a8071690 0x7f90a80b33c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f90a80708e0 tx=0x7f90a001c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b251f521fc2405ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:05.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90a7e00640 1 -- 172.21.15.40:0/2446181364 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a00a5670 con 0x7f90a8071690 2024-03-24T00:37:05.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90a7e00640 1 -- 172.21.15.40:0/2446181364 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a00a5810 con 0x7f90a8071690 2024-03-24T00:37:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90a7e00640 1 -- 172.21.15.40:0/2446181364 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90a00a5be0 con 0x7f90a8071690 2024-03-24T00:37:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90afa00640 1 -- 172.21.15.40:0/2446181364 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90a8071690 msgr2=0x7f90a80b33c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90afa00640 1 --2- 172.21.15.40:0/2446181364 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90a8071690 0x7f90a80b33c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f90a80708e0 tx=0x7f90a001c9f0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90afa00640 1 -- 172.21.15.40:0/2446181364 shutdown_connections 2024-03-24T00:37:05.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90afa00640 1 --2- 172.21.15.40:0/2446181364 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90a8071690 0x7f90a80b33c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90afa00640 1 --2- 172.21.15.40:0/2446181364 >> v2:172.21.15.53:3300/0 conn(0x7f90a8070c80 0x7f90a806db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90afa00640 1 --2- 172.21.15.40:0/2446181364 >> v2:172.21.15.40:3300/0 conn(0x7f90a806e5a0 0x7f90a806e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90afa00640 1 -- 172.21.15.40:0/2446181364 >> 172.21.15.40:0/2446181364 conn(0x7f90a80b17b0 msgr2=0x7f90a80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:05.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90afa00640 1 -- 172.21.15.40:0/2446181364 shutdown_connections 2024-03-24T00:37:05.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.369+0000 7f90afa00640 1 -- 172.21.15.40:0/2446181364 wait complete. 2024-03-24T00:37:05.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90afa00640 1 Processor -- start 2024-03-24T00:37:05.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90afa00640 1 -- start start 2024-03-24T00:37:05.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90afa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a806e5a0 0x7f90a814f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:05.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90afa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a8070c80 0x7f90a8153aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:05.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90afa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90a8071690 0x7f90a8153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:05.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90afa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a8155460 con 0x7f90a8070c80 2024-03-24T00:37:05.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90afa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a81552e0 con 0x7f90a806e5a0 2024-03-24T00:37:05.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90afa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a81623a0 con 0x7f90a8071690 2024-03-24T00:37:05.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f909fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a8070c80 0x7f90a8153aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:05.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f909fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a8070c80 0x7f90a8153aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40054/0 (socket says 172.21.15.40:40054) 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f909fe00640 1 -- 172.21.15.40:0/866442223 learned_addr learned my addr 172.21.15.40:0/866442223 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90ad000640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a806e5a0 0x7f90a814f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90ada00640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90a8071690 0x7f90a8153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f909fe00640 1 -- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90a8071690 msgr2=0x7f90a8153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f909fe00640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90a8071690 0x7f90a8153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f909fe00640 1 -- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a806e5a0 msgr2=0x7f90a814f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f909fe00640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a806e5a0 0x7f90a814f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f909fe00640 1 -- 172.21.15.40:0/866442223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a8161230 con 0x7f90a8070c80 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90ad000640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a806e5a0 0x7f90a814f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f909fe00640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a8070c80 0x7f90a8153aa0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f909000e9d0 tx=0x7f909000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90a5600640 1 -- 172.21.15.40:0/866442223 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090017070 con 0x7f90a8070c80 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90a5600640 1 -- 172.21.15.40:0/866442223 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909000ca60 con 0x7f90a8070c80 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90a5600640 1 -- 172.21.15.40:0/866442223 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909000ce50 con 0x7f90a8070c80 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90afa00640 1 -- 172.21.15.40:0/866442223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a80cd090 con 0x7f90a8070c80 2024-03-24T00:37:05.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.373+0000 7f90afa00640 1 -- 172.21.15.40:0/866442223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a80cea00 con 0x7f90a8070c80 2024-03-24T00:37:05.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.377+0000 7f90afa00640 1 -- 172.21.15.40:0/866442223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a8068170 con 0x7f90a8070c80 2024-03-24T00:37:05.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.377+0000 7f90a5600640 1 -- 172.21.15.40:0/866442223 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f909001a070 con 0x7f90a8070c80 2024-03-24T00:37:05.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.377+0000 7f90a5600640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9084067800 0x7f9084069cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:05.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.377+0000 7f90a5600640 1 -- 172.21.15.40:0/866442223 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90900853e0 con 0x7f90a8070c80 2024-03-24T00:37:05.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.377+0000 7f90ad000640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9084067800 0x7f9084069cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:05.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.377+0000 7f90ad000640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9084067800 0x7f9084069cc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9098009ed0 tx=0x7f9098009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:05.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.381+0000 7f90a5600640 1 -- 172.21.15.40:0/866442223 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f90a8161230 con 0x7f90a8070c80 2024-03-24T00:37:05.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.577+0000 7f90afa00640 1 -- 172.21.15.40:0/866442223 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90a80b2fa0 con 0x7f90a8070c80 2024-03-24T00:37:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.577+0000 7f90a5600640 1 -- 172.21.15.40:0/866442223 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f9090055a10 con 0x7f90a8070c80 2024-03-24T00:37:05.583 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:05.583 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":97,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6755,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2936601899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2936601899},{"type":"v1","addr":"172.21.15.53:6836","nonce":2936601899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":7633,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1997904254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1997904254},{"type":"v1","addr":"172.21.15.40:6835","nonce":1997904254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":7917,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1019660586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1019660586},{"type":"v1","addr":"172.21.15.40:6834","nonce":1019660586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:37:01.566949+0000","modified":"2024-03-24T00:37:04.591207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"d","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/3095501397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3095501397},{"type":"v1","addr":"172.21.15.53:6837","nonce":3095501397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":7}]} 2024-03-24T00:37:05.583 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 97 2024-03-24T00:37:05.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.581+0000 7f90afa00640 1 -- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9084067800 msgr2=0x7f9084069cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.581+0000 7f90afa00640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9084067800 0x7f9084069cc0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f9098009ed0 tx=0x7f9098009190 comp rx=0 tx=0).stop 2024-03-24T00:37:05.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.581+0000 7f90afa00640 1 -- 172.21.15.40:0/866442223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a8070c80 msgr2=0x7f90a8153aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.581+0000 7f90afa00640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a8070c80 0x7f90a8153aa0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f909000e9d0 tx=0x7f909000ee90 comp rx=0 tx=0).stop 2024-03-24T00:37:05.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.581+0000 7f90afa00640 1 -- 172.21.15.40:0/866442223 shutdown_connections 2024-03-24T00:37:05.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.581+0000 7f90afa00640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9084067800 0x7f9084069cc0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.581+0000 7f90afa00640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90a8071690 0x7f90a8153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.581+0000 7f90afa00640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f90a8070c80 0x7f90a8153aa0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.581+0000 7f90afa00640 1 --2- 172.21.15.40:0/866442223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90a806e5a0 0x7f90a814f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.581+0000 7f90afa00640 1 -- 172.21.15.40:0/866442223 >> 172.21.15.40:0/866442223 conn(0x7f90a80b17b0 msgr2=0x7f90a81546c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:05.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.581+0000 7f90afa00640 1 -- 172.21.15.40:0/866442223 shutdown_connections 2024-03-24T00:37:05.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.581+0000 7f90afa00640 1 -- 172.21.15.40:0/866442223 wait complete. 2024-03-24T00:37:05.604 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:05.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda6b000640 1 Processor -- start 2024-03-24T00:37:05.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda6b000640 1 -- start start 2024-03-24T00:37:05.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda6b000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda640d2830 0x7fda64166e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:05.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda6b000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda640af880 con 0x7fda6415f600 2024-03-24T00:37:05.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda6b000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda640b1e50 con 0x7fda640b2910 2024-03-24T00:37:05.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda6b000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda640b3630 con 0x7fda640d2830 2024-03-24T00:37:05.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda63e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fda6415f600 0x7fda640aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44828/0 (socket says 172.21.15.40:44828) 2024-03-24T00:37:05.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda63e00640 1 -- 172.21.15.40:0/1676491526 learned_addr learned my addr 172.21.15.40:0/1676491526 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:05.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda69000640 1 --2- 172.21.15.40:0/1676491526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda640d2830 0x7fda64166e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:05.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda63400640 1 -- 172.21.15.40:0/1676491526 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414511999 0 0) 0x7fda640af880 con 0x7fda6415f600 2024-03-24T00:37:05.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda63400640 1 -- 172.21.15.40:0/1676491526 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda48003530 con 0x7fda6415f600 2024-03-24T00:37:05.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda63400640 1 -- 172.21.15.40:0/1676491526 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358711332 0 0) 0x7fda640b1e50 con 0x7fda640b2910 2024-03-24T00:37:05.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda63400640 1 -- 172.21.15.40:0/1676491526 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda640af880 con 0x7fda640b2910 2024-03-24T00:37:05.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda69000640 1 -- 172.21.15.40:0/1676491526 >> v1:172.21.15.53:6789/0 conn(0x7fda640b2910 legacy=0x7fda640af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda69000640 1 -- 172.21.15.40:0/1676491526 >> v1:172.21.15.40:6789/0 conn(0x7fda6415f600 legacy=0x7fda640aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.741+0000 7fda69000640 1 -- 172.21.15.40:0/1676491526 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda640d2c40 con 0x7fda640d2830 2024-03-24T00:37:05.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.745+0000 7fda69000640 1 --2- 172.21.15.40:0/1676491526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda640d2830 0x7fda64166e50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fda5c009dc0 tx=0x7fda5c01f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=342c831c5bf6348e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.745+0000 7fda63400640 1 -- 172.21.15.40:0/1676491526 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c0b4070 con 0x7fda640d2830 2024-03-24T00:37:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.745+0000 7fda63400640 1 -- 172.21.15.40:0/1676491526 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5c0af600 con 0x7fda640d2830 2024-03-24T00:37:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.745+0000 7fda63400640 1 -- 172.21.15.40:0/1676491526 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c0af9d0 con 0x7fda640d2830 2024-03-24T00:37:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.745+0000 7fda6b000640 1 -- 172.21.15.40:0/1676491526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda640d2830 msgr2=0x7fda64166e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.745+0000 7fda6b000640 1 --2- 172.21.15.40:0/1676491526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda640d2830 0x7fda64166e50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fda5c009dc0 tx=0x7fda5c01f170 comp rx=0 tx=0).stop 2024-03-24T00:37:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.745+0000 7fda6b000640 1 -- 172.21.15.40:0/1676491526 shutdown_connections 2024-03-24T00:37:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.745+0000 7fda6b000640 1 --2- 172.21.15.40:0/1676491526 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda640d2830 0x7fda64166e50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.745+0000 7fda6b000640 1 -- 172.21.15.40:0/1676491526 >> 172.21.15.40:0/1676491526 conn(0x7fda640d1d10 msgr2=0x7fda640d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.745+0000 7fda6b000640 1 -- 172.21.15.40:0/1676491526 shutdown_connections 2024-03-24T00:37:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.745+0000 7fda6b000640 1 -- 172.21.15.40:0/1676491526 wait complete. 2024-03-24T00:37:05.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda6b000640 1 Processor -- start 2024-03-24T00:37:05.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda6b000640 1 -- start start 2024-03-24T00:37:05.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda6b000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda640b2910 0x7fda640cb110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:05.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda6b000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda640d2830 0x7fda640c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:05.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda6b000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda6415f600 0x7fda6415a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda6b000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda640b2740 con 0x7fda640b2910 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda6b000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda640b2d80 con 0x7fda640d2830 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda6b000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda640b23d0 con 0x7fda6415f600 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda640b2910 0x7fda640cb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda640b2910 0x7fda640cb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40070/0 (socket says 172.21.15.40:40070) 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda63e00640 1 -- 172.21.15.40:0/176090263 learned_addr learned my addr 172.21.15.40:0/176090263 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda69000640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda6415f600 0x7fda6415a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda5be00640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda640d2830 0x7fda640c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda69000640 1 -- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda640d2830 msgr2=0x7fda640c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda69000640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda640d2830 0x7fda640c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda69000640 1 -- 172.21.15.40:0/176090263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda640b2910 msgr2=0x7fda640cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda69000640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda640b2910 0x7fda640cb110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda69000640 1 -- 172.21.15.40:0/176090263 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6406b680 con 0x7fda6415f600 2024-03-24T00:37:05.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda63e00640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda640b2910 0x7fda640cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda69000640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda6415f600 0x7fda6415a8e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fda5c0af000 tx=0x7fda5c01bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda60c00640 1 -- 172.21.15.40:0/176090263 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c0b4040 con 0x7fda6415f600 2024-03-24T00:37:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda60c00640 1 -- 172.21.15.40:0/176090263 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5c0c1070 con 0x7fda6415f600 2024-03-24T00:37:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda60c00640 1 -- 172.21.15.40:0/176090263 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c009450 con 0x7fda6415f600 2024-03-24T00:37:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda6b000640 1 -- 172.21.15.40:0/176090263 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6406b960 con 0x7fda6415f600 2024-03-24T00:37:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda6b000640 1 -- 172.21.15.40:0/176090263 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6406bf20 con 0x7fda6415f600 2024-03-24T00:37:05.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.749+0000 7fda6b000640 1 -- 172.21.15.40:0/176090263 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda64068170 con 0x7fda6415f600 2024-03-24T00:37:05.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.753+0000 7fda60c00640 1 -- 172.21.15.40:0/176090263 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fda5c0ba6b0 con 0x7fda6415f600 2024-03-24T00:37:05.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.753+0000 7fda60c00640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fda40067690 0x7fda40069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:05.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.753+0000 7fda60c00640 1 -- 172.21.15.40:0/176090263 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda5c123f50 con 0x7fda6415f600 2024-03-24T00:37:05.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.753+0000 7fda63e00640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fda40067690 0x7fda40069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:05.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.753+0000 7fda63e00640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fda40067690 0x7fda40069b50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fda54000f70 tx=0x7fda54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:05.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.757+0000 7fda60c00640 1 -- 172.21.15.40:0/176090263 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fda5c018cd0 con 0x7fda6415f600 2024-03-24T00:37:05.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.921+0000 7fda6b000640 1 -- 172.21.15.40:0/176090263 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda640b00d0 con 0x7fda6415f600 2024-03-24T00:37:05.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.921+0000 7fda60c00640 1 -- 172.21.15.40:0/176090263 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16548 (secure 0 0 0) 0x7fda5c0f4690 con 0x7fda6415f600 2024-03-24T00:37:05.926 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:05.926 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":95,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:37:03.189247+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:59.977310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-24T00:37:00.557420+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:05.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.925+0000 7fda6b000640 1 -- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fda40067690 msgr2=0x7fda40069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.925+0000 7fda6b000640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fda40067690 0x7fda40069b50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fda54000f70 tx=0x7fda54009040 comp rx=0 tx=0).stop 2024-03-24T00:37:05.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.925+0000 7fda6b000640 1 -- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda6415f600 msgr2=0x7fda6415a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:05.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.925+0000 7fda6b000640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda6415f600 0x7fda6415a8e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fda5c0af000 tx=0x7fda5c01bf10 comp rx=0 tx=0).stop 2024-03-24T00:37:05.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.925+0000 7fda6b000640 1 -- 172.21.15.40:0/176090263 shutdown_connections 2024-03-24T00:37:05.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.925+0000 7fda6b000640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fda40067690 0x7fda40069b50 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.925+0000 7fda6b000640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda6415f600 0x7fda6415a8e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.925+0000 7fda6b000640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda640d2830 0x7fda640c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.925+0000 7fda6b000640 1 --2- 172.21.15.40:0/176090263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda640b2910 0x7fda640cb110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:05.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.925+0000 7fda6b000640 1 -- 172.21.15.40:0/176090263 >> 172.21.15.40:0/176090263 conn(0x7fda640d1d10 msgr2=0x7fda640af140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:05.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.925+0000 7fda6b000640 1 -- 172.21.15.40:0/176090263 shutdown_connections 2024-03-24T00:37:05.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:05.925+0000 7fda6b000640 1 -- 172.21.15.40:0/176090263 wait complete. 2024-03-24T00:37:05.946 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:37:05.947 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:37:06.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.081+0000 7f6932000640 1 Processor -- start 2024-03-24T00:37:06.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.081+0000 7f6932000640 1 -- start start 2024-03-24T00:37:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f6932000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f692c06e620 0x7f692c0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f6932000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f692c0ccfa0 0x7f692c0cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f6932000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f692c070f60 con 0x7f692c06ea00 2024-03-24T00:37:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f6932000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f692c0b4c00 con 0x7f692c06e620 2024-03-24T00:37:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f6932000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f692c0b3620 con 0x7f692c0ccfa0 2024-03-24T00:37:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f692b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f692c06e620 0x7f692c0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f6922a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f692c06ea00 0x7f692c0b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44836/0 (socket says 172.21.15.40:44836) 2024-03-24T00:37:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f6922a00640 1 -- 172.21.15.40:0/543676198 learned_addr learned my addr 172.21.15.40:0/543676198 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:06.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f692be00640 1 --2- 172.21.15.40:0/543676198 >> v2:172.21.15.53:3300/0 conn(0x7f692c0ccfa0 0x7f692c0cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f692b400640 1 -- 172.21.15.40:0/543676198 >> v2:172.21.15.53:3300/0 conn(0x7f692c0ccfa0 msgr2=0x7f692c0cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f692b400640 1 --2- 172.21.15.40:0/543676198 >> v2:172.21.15.53:3300/0 conn(0x7f692c0ccfa0 0x7f692c0cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f692b400640 1 -- 172.21.15.40:0/543676198 >> v1:172.21.15.40:6789/0 conn(0x7f692c06ea00 legacy=0x7f692c0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f692aa00640 1 -- 172.21.15.40:0/543676198 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355846474 0 0) 0x7f692c070f60 con 0x7f692c06ea00 2024-03-24T00:37:06.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.085+0000 7f692b400640 1 -- 172.21.15.40:0/543676198 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c0b3900 con 0x7f692c06e620 2024-03-24T00:37:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f692b400640 1 --2- 172.21.15.40:0/543676198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f692c06e620 0x7f692c0b46c0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f692c067e30 tx=0x7f691c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60fdaddfd984a3c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f692aa00640 1 -- 172.21.15.40:0/543676198 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c0b0070 con 0x7f692c06e620 2024-03-24T00:37:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f692aa00640 1 -- 172.21.15.40:0/543676198 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c0ab600 con 0x7f692c06e620 2024-03-24T00:37:06.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f692aa00640 1 -- 172.21.15.40:0/543676198 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c0ab9d0 con 0x7f692c06e620 2024-03-24T00:37:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f6932000640 1 -- 172.21.15.40:0/543676198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f692c06e620 msgr2=0x7f692c0b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f6932000640 1 --2- 172.21.15.40:0/543676198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f692c06e620 0x7f692c0b46c0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f692c067e30 tx=0x7f691c01c3c0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f6932000640 1 -- 172.21.15.40:0/543676198 shutdown_connections 2024-03-24T00:37:06.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f6932000640 1 --2- 172.21.15.40:0/543676198 >> v2:172.21.15.53:3300/0 conn(0x7f692c0ccfa0 0x7f692c0cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f6932000640 1 --2- 172.21.15.40:0/543676198 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f692c06e620 0x7f692c0b46c0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f6932000640 1 -- 172.21.15.40:0/543676198 >> 172.21.15.40:0/543676198 conn(0x7f692c0b17b0 msgr2=0x7f692c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:06.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f6932000640 1 -- 172.21.15.40:0/543676198 shutdown_connections 2024-03-24T00:37:06.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f6932000640 1 -- 172.21.15.40:0/543676198 wait complete. 2024-03-24T00:37:06.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f6932000640 1 Processor -- start 2024-03-24T00:37:06.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.089+0000 7f6932000640 1 -- start start 2024-03-24T00:37:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6932000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f692c06e620 0x7f692c15a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6932000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f692c06ea00 0x7f692c15d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6932000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f692c0ccfa0 0x7f692c15de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6932000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c06efd0 con 0x7f692c06e620 2024-03-24T00:37:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6932000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c06ee50 con 0x7f692c06ea00 2024-03-24T00:37:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6932000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f692c144920 con 0x7f692c0ccfa0 2024-03-24T00:37:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f692b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f692c06e620 0x7f692c15a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f692b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f692c06e620 0x7f692c15a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40088/0 (socket says 172.21.15.40:40088) 2024-03-24T00:37:06.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f692b400640 1 -- 172.21.15.40:0/2059679025 learned_addr learned my addr 172.21.15.40:0/2059679025 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6922a00640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f692c06ea00 0x7f692c15d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f692be00640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f692c0ccfa0 0x7f692c15de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f692b400640 1 -- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f692c0ccfa0 msgr2=0x7f692c15de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f692b400640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f692c0ccfa0 0x7f692c15de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f692b400640 1 -- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f692c06ea00 msgr2=0x7f692c15d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f692b400640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f692c06ea00 0x7f692c15d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f692b400640 1 -- 172.21.15.40:0/2059679025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c143200 con 0x7f692c06e620 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f692be00640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f692c0ccfa0 0x7f692c15de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f692b400640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f692c06e620 0x7f692c15a550 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f691c018fd0 tx=0x7f691c01ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6923e00640 1 -- 172.21.15.40:0/2059679025 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c0b0050 con 0x7f692c06e620 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6923e00640 1 -- 172.21.15.40:0/2059679025 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f691c0b1040 con 0x7f692c06e620 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6932000640 1 -- 172.21.15.40:0/2059679025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692c142160 con 0x7f692c06e620 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6923e00640 1 -- 172.21.15.40:0/2059679025 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c006e20 con 0x7f692c06e620 2024-03-24T00:37:06.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6932000640 1 -- 172.21.15.40:0/2059679025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c142720 con 0x7f692c06e620 2024-03-24T00:37:06.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6932000640 1 -- 172.21.15.40:0/2059679025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692c068170 con 0x7f692c06e620 2024-03-24T00:37:06.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.093+0000 7f6923e00640 1 -- 172.21.15.40:0/2059679025 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f691c0be060 con 0x7f692c06e620 2024-03-24T00:37:06.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.097+0000 7f6923e00640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6908067660 0x7f6908069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.097+0000 7f6923e00640 1 -- 172.21.15.40:0/2059679025 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f691c11f0b0 con 0x7f692c06e620 2024-03-24T00:37:06.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.097+0000 7f6922a00640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6908067660 0x7f6908069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.097+0000 7f6922a00640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6908067660 0x7f6908069b20 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f692c0af0e0 tx=0x7f6914009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:06.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.101+0000 7f6923e00640 1 -- 172.21.15.40:0/2059679025 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f691c016070 con 0x7f692c06e620 2024-03-24T00:37:06.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.261+0000 7f6932000640 1 -- 172.21.15.40:0/2059679025 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f692c072f50 con 0x7f6908067660 2024-03-24T00:37:06.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6923e00640 1 -- 172.21.15.40:0/2059679025 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252058 (secure 0 0 0) 0x7f692c072f50 con 0x7f6908067660 2024-03-24T00:37:06.282 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:06.284 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:37:06.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6932000640 1 -- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6908067660 msgr2=0x7f6908069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6932000640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6908067660 0x7f6908069b20 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f692c0af0e0 tx=0x7f6914009040 comp rx=0 tx=0).stop 2024-03-24T00:37:06.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6932000640 1 -- 172.21.15.40:0/2059679025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f692c06e620 msgr2=0x7f692c15a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6932000640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f692c06e620 0x7f692c15a550 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f691c018fd0 tx=0x7f691c01ac00 comp rx=0 tx=0).stop 2024-03-24T00:37:06.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6932000640 1 -- 172.21.15.40:0/2059679025 shutdown_connections 2024-03-24T00:37:06.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6932000640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6908067660 0x7f6908069b20 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6932000640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f692c0ccfa0 0x7f692c15de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6932000640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f692c06ea00 0x7f692c15d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6932000640 1 --2- 172.21.15.40:0/2059679025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f692c06e620 0x7f692c15a550 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6932000640 1 -- 172.21.15.40:0/2059679025 >> 172.21.15.40:0/2059679025 conn(0x7f692c0b17b0 msgr2=0x7f692c06d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:06.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6932000640 1 -- 172.21.15.40:0/2059679025 shutdown_connections 2024-03-24T00:37:06.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.277+0000 7f6932000640 1 -- 172.21.15.40:0/2059679025 wait complete. 2024-03-24T00:37:06.302 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-03-24T00:37:05.270291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":297248,"kb_used_data":10324,"kb_used_omap":12,"kb_used_meta":286771,"kb_avail":754677472,"statfs":{"total":773094113280,"available":772789731328,"internally_reserved":0,"allocated":10571776,"data_stored":5230064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":293654066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.114368"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.204513+0000","last_change":"2024-03-24T00:37:02.200895+0000","last_active":"2024-03-24T00:37:02.204513+0000","last_peered":"2024-03-24T00:37:02.204513+0000","last_clean":"2024-03-24T00:37:02.204513+0000","last_became_active":"2024-03-24T00:37:02.199999+0000","last_became_peered":"2024-03-24T00:37:02.199999+0000","last_unstale":"2024-03-24T00:37:02.204513+0000","last_undegraded":"2024-03-24T00:37:02.204513+0000","last_fullsized":"2024-03-24T00:37:02.204513+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:28:38.887578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205134+0000","last_change":"2024-03-24T00:37:02.215943+0000","last_active":"2024-03-24T00:37:03.205134+0000","last_peered":"2024-03-24T00:37:03.205134+0000","last_clean":"2024-03-24T00:37:03.205134+0000","last_became_active":"2024-03-24T00:37:02.215643+0000","last_became_peered":"2024-03-24T00:37:02.215643+0000","last_unstale":"2024-03-24T00:37:03.205134+0000","last_undegraded":"2024-03-24T00:37:03.205134+0000","last_fullsized":"2024-03-24T00:37:03.205134+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:34:21.769977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.204221+0000","last_change":"2024-03-24T00:37:02.200559+0000","last_active":"2024-03-24T00:37:02.204221+0000","last_peered":"2024-03-24T00:37:02.204221+0000","last_clean":"2024-03-24T00:37:02.204221+0000","last_became_active":"2024-03-24T00:37:02.200151+0000","last_became_peered":"2024-03-24T00:37:02.200151+0000","last_unstale":"2024-03-24T00:37:02.204221+0000","last_undegraded":"2024-03-24T00:37:02.204221+0000","last_fullsized":"2024-03-24T00:37:02.204221+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:29.892702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.216647+0000","last_change":"2024-03-24T00:37:02.205873+0000","last_active":"2024-03-24T00:37:02.216647+0000","last_peered":"2024-03-24T00:37:02.216647+0000","last_clean":"2024-03-24T00:37:02.216647+0000","last_became_active":"2024-03-24T00:37:02.205579+0000","last_became_peered":"2024-03-24T00:37:02.205579+0000","last_unstale":"2024-03-24T00:37:02.216647+0000","last_undegraded":"2024-03-24T00:37:02.216647+0000","last_fullsized":"2024-03-24T00:37:02.216647+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:52.195925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215352+0000","last_change":"2024-03-24T00:37:02.208098+0000","last_active":"2024-03-24T00:37:02.215352+0000","last_peered":"2024-03-24T00:37:02.215352+0000","last_clean":"2024-03-24T00:37:02.215352+0000","last_became_active":"2024-03-24T00:37:02.207798+0000","last_became_peered":"2024-03-24T00:37:02.207798+0000","last_unstale":"2024-03-24T00:37:02.215352+0000","last_undegraded":"2024-03-24T00:37:02.215352+0000","last_fullsized":"2024-03-24T00:37:02.215352+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:21:08.391723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201057+0000","last_change":"2024-03-24T00:37:02.203597+0000","last_active":"2024-03-24T00:37:03.201057+0000","last_peered":"2024-03-24T00:37:03.201057+0000","last_clean":"2024-03-24T00:37:03.201057+0000","last_became_active":"2024-03-24T00:37:02.203302+0000","last_became_peered":"2024-03-24T00:37:02.203302+0000","last_unstale":"2024-03-24T00:37:03.201057+0000","last_undegraded":"2024-03-24T00:37:03.201057+0000","last_fullsized":"2024-03-24T00:37:03.201057+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:45:55.182522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200665+0000","last_change":"2024-03-24T00:37:02.200991+0000","last_active":"2024-03-24T00:37:03.200665+0000","last_peered":"2024-03-24T00:37:03.200665+0000","last_clean":"2024-03-24T00:37:03.200665+0000","last_became_active":"2024-03-24T00:37:02.200617+0000","last_became_peered":"2024-03-24T00:37:02.200617+0000","last_unstale":"2024-03-24T00:37:03.200665+0000","last_undegraded":"2024-03-24T00:37:03.200665+0000","last_fullsized":"2024-03-24T00:37:03.200665+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:11:18.916706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204691+0000","last_change":"2024-03-24T00:37:02.217747+0000","last_active":"2024-03-24T00:37:03.204691+0000","last_peered":"2024-03-24T00:37:03.204691+0000","last_clean":"2024-03-24T00:37:03.204691+0000","last_became_active":"2024-03-24T00:37:02.217536+0000","last_became_peered":"2024-03-24T00:37:02.217536+0000","last_unstale":"2024-03-24T00:37:03.204691+0000","last_undegraded":"2024-03-24T00:37:03.204691+0000","last_fullsized":"2024-03-24T00:37:03.204691+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:40.631539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201068+0000","last_change":"2024-03-24T00:37:02.203121+0000","last_active":"2024-03-24T00:37:03.201068+0000","last_peered":"2024-03-24T00:37:03.201068+0000","last_clean":"2024-03-24T00:37:03.201068+0000","last_became_active":"2024-03-24T00:37:02.202411+0000","last_became_peered":"2024-03-24T00:37:02.202411+0000","last_unstale":"2024-03-24T00:37:03.201068+0000","last_undegraded":"2024-03-24T00:37:03.201068+0000","last_fullsized":"2024-03-24T00:37:03.201068+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:25.855556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.214651+0000","last_change":"2024-03-24T00:37:02.202560+0000","last_active":"2024-03-24T00:37:02.214651+0000","last_peered":"2024-03-24T00:37:02.214651+0000","last_clean":"2024-03-24T00:37:02.214651+0000","last_became_active":"2024-03-24T00:37:02.199273+0000","last_became_peered":"2024-03-24T00:37:02.199273+0000","last_unstale":"2024-03-24T00:37:02.214651+0000","last_undegraded":"2024-03-24T00:37:02.214651+0000","last_fullsized":"2024-03-24T00:37:02.214651+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:59.531126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201575+0000","last_change":"2024-03-24T00:37:02.195658+0000","last_active":"2024-03-24T00:37:03.201575+0000","last_peered":"2024-03-24T00:37:03.201575+0000","last_clean":"2024-03-24T00:37:03.201575+0000","last_became_active":"2024-03-24T00:37:02.194784+0000","last_became_peered":"2024-03-24T00:37:02.194784+0000","last_unstale":"2024-03-24T00:37:03.201575+0000","last_undegraded":"2024-03-24T00:37:03.201575+0000","last_fullsized":"2024-03-24T00:37:03.201575+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:49.266439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201315+0000","last_change":"2024-03-24T00:37:02.203871+0000","last_active":"2024-03-24T00:37:03.201315+0000","last_peered":"2024-03-24T00:37:03.201315+0000","last_clean":"2024-03-24T00:37:03.201315+0000","last_became_active":"2024-03-24T00:37:02.203584+0000","last_became_peered":"2024-03-24T00:37:02.203584+0000","last_unstale":"2024-03-24T00:37:03.201315+0000","last_undegraded":"2024-03-24T00:37:03.201315+0000","last_fullsized":"2024-03-24T00:37:03.201315+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:30.857697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215649+0000","last_change":"2024-03-24T00:37:02.206089+0000","last_active":"2024-03-24T00:37:02.215649+0000","last_peered":"2024-03-24T00:37:02.215649+0000","last_clean":"2024-03-24T00:37:02.215649+0000","last_became_active":"2024-03-24T00:37:02.205522+0000","last_became_peered":"2024-03-24T00:37:02.205522+0000","last_unstale":"2024-03-24T00:37:02.215649+0000","last_undegraded":"2024-03-24T00:37:02.215649+0000","last_fullsized":"2024-03-24T00:37:02.215649+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:25.855556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198028+0000","last_change":"2024-03-24T00:37:02.212573+0000","last_active":"2024-03-24T00:37:03.198028+0000","last_peered":"2024-03-24T00:37:03.198028+0000","last_clean":"2024-03-24T00:37:03.198028+0000","last_became_active":"2024-03-24T00:37:02.212278+0000","last_became_peered":"2024-03-24T00:37:02.212278+0000","last_unstale":"2024-03-24T00:37:03.198028+0000","last_undegraded":"2024-03-24T00:37:03.198028+0000","last_fullsized":"2024-03-24T00:37:03.198028+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:01.809774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.210582+0000","last_change":"2024-03-24T00:37:02.202335+0000","last_active":"2024-03-24T00:37:02.210582+0000","last_peered":"2024-03-24T00:37:02.210582+0000","last_clean":"2024-03-24T00:37:02.210582+0000","last_became_active":"2024-03-24T00:37:02.202007+0000","last_became_peered":"2024-03-24T00:37:02.202007+0000","last_unstale":"2024-03-24T00:37:02.210582+0000","last_undegraded":"2024-03-24T00:37:02.210582+0000","last_fullsized":"2024-03-24T00:37:02.210582+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:21:08.391723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200345+0000","last_change":"2024-03-24T00:37:02.194567+0000","last_active":"2024-03-24T00:37:03.200345+0000","last_peered":"2024-03-24T00:37:03.200345+0000","last_clean":"2024-03-24T00:37:03.200345+0000","last_became_active":"2024-03-24T00:37:02.194267+0000","last_became_peered":"2024-03-24T00:37:02.194267+0000","last_unstale":"2024-03-24T00:37:03.200345+0000","last_undegraded":"2024-03-24T00:37:03.200345+0000","last_fullsized":"2024-03-24T00:37:03.200345+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:37.053363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205224+0000","last_change":"2024-03-24T00:37:02.202700+0000","last_active":"2024-03-24T00:37:03.205224+0000","last_peered":"2024-03-24T00:37:03.205224+0000","last_clean":"2024-03-24T00:37:03.205224+0000","last_became_active":"2024-03-24T00:37:02.202405+0000","last_became_peered":"2024-03-24T00:37:02.202405+0000","last_unstale":"2024-03-24T00:37:03.205224+0000","last_undegraded":"2024-03-24T00:37:03.205224+0000","last_fullsized":"2024-03-24T00:37:03.205224+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:44.003595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215802+0000","last_change":"2024-03-24T00:37:02.206690+0000","last_active":"2024-03-24T00:37:02.215802+0000","last_peered":"2024-03-24T00:37:02.215802+0000","last_clean":"2024-03-24T00:37:02.215802+0000","last_became_active":"2024-03-24T00:37:02.206024+0000","last_became_peered":"2024-03-24T00:37:02.206024+0000","last_unstale":"2024-03-24T00:37:02.215802+0000","last_undegraded":"2024-03-24T00:37:02.215802+0000","last_fullsized":"2024-03-24T00:37:02.215802+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:34.773035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198084+0000","last_change":"2024-03-24T00:37:02.196609+0000","last_active":"2024-03-24T00:37:03.198084+0000","last_peered":"2024-03-24T00:37:03.198084+0000","last_clean":"2024-03-24T00:37:03.198084+0000","last_became_active":"2024-03-24T00:37:02.196104+0000","last_became_peered":"2024-03-24T00:37:02.196104+0000","last_unstale":"2024-03-24T00:37:03.198084+0000","last_undegraded":"2024-03-24T00:37:03.198084+0000","last_fullsized":"2024-03-24T00:37:03.198084+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:39.179840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201463+0000","last_change":"2024-03-24T00:37:02.198834+0000","last_active":"2024-03-24T00:37:03.201463+0000","last_peered":"2024-03-24T00:37:03.201463+0000","last_clean":"2024-03-24T00:37:03.201463+0000","last_became_active":"2024-03-24T00:37:02.197500+0000","last_became_peered":"2024-03-24T00:37:02.197500+0000","last_unstale":"2024-03-24T00:37:03.201463+0000","last_undegraded":"2024-03-24T00:37:03.201463+0000","last_fullsized":"2024-03-24T00:37:03.201463+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:30.972900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198226+0000","last_change":"2024-03-24T00:37:02.200810+0000","last_active":"2024-03-24T00:37:03.198226+0000","last_peered":"2024-03-24T00:37:03.198226+0000","last_clean":"2024-03-24T00:37:03.198226+0000","last_became_active":"2024-03-24T00:37:02.200482+0000","last_became_peered":"2024-03-24T00:37:02.200482+0000","last_unstale":"2024-03-24T00:37:03.198226+0000","last_undegraded":"2024-03-24T00:37:03.198226+0000","last_fullsized":"2024-03-24T00:37:03.198226+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:58.820942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204695+0000","last_change":"2024-03-24T00:37:02.205543+0000","last_active":"2024-03-24T00:37:03.204695+0000","last_peered":"2024-03-24T00:37:03.204695+0000","last_clean":"2024-03-24T00:37:03.204695+0000","last_became_active":"2024-03-24T00:37:02.205262+0000","last_became_peered":"2024-03-24T00:37:02.205262+0000","last_unstale":"2024-03-24T00:37:03.204695+0000","last_undegraded":"2024-03-24T00:37:03.204695+0000","last_fullsized":"2024-03-24T00:37:03.204695+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:16:23.877808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204792+0000","last_change":"2024-03-24T00:37:02.202790+0000","last_active":"2024-03-24T00:37:03.204792+0000","last_peered":"2024-03-24T00:37:03.204792+0000","last_clean":"2024-03-24T00:37:03.204792+0000","last_became_active":"2024-03-24T00:37:02.202513+0000","last_became_peered":"2024-03-24T00:37:02.202513+0000","last_unstale":"2024-03-24T00:37:03.204792+0000","last_undegraded":"2024-03-24T00:37:03.204792+0000","last_fullsized":"2024-03-24T00:37:03.204792+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:32:23.886507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200376+0000","last_change":"2024-03-24T00:37:02.192248+0000","last_active":"2024-03-24T00:37:03.200376+0000","last_peered":"2024-03-24T00:37:03.200376+0000","last_clean":"2024-03-24T00:37:03.200376+0000","last_became_active":"2024-03-24T00:37:02.191365+0000","last_became_peered":"2024-03-24T00:37:02.191365+0000","last_unstale":"2024-03-24T00:37:03.200376+0000","last_undegraded":"2024-03-24T00:37:03.200376+0000","last_fullsized":"2024-03-24T00:37:03.200376+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:21.928238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.214671+0000","last_change":"2024-03-24T00:37:02.202749+0000","last_active":"2024-03-24T00:37:02.214671+0000","last_peered":"2024-03-24T00:37:02.214671+0000","last_clean":"2024-03-24T00:37:02.214671+0000","last_became_active":"2024-03-24T00:37:02.202007+0000","last_became_peered":"2024-03-24T00:37:02.202007+0000","last_unstale":"2024-03-24T00:37:02.214671+0000","last_undegraded":"2024-03-24T00:37:02.214671+0000","last_fullsized":"2024-03-24T00:37:02.214671+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:21.852373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205142+0000","last_change":"2024-03-24T00:37:02.200452+0000","last_active":"2024-03-24T00:37:03.205142+0000","last_peered":"2024-03-24T00:37:03.205142+0000","last_clean":"2024-03-24T00:37:03.205142+0000","last_became_active":"2024-03-24T00:37:02.200140+0000","last_became_peered":"2024-03-24T00:37:02.200140+0000","last_unstale":"2024-03-24T00:37:03.205142+0000","last_undegraded":"2024-03-24T00:37:03.205142+0000","last_fullsized":"2024-03-24T00:37:03.205142+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:22.264783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201822+0000","last_change":"2024-03-24T00:37:02.195418+0000","last_active":"2024-03-24T00:37:03.201822+0000","last_peered":"2024-03-24T00:37:03.201822+0000","last_clean":"2024-03-24T00:37:03.201822+0000","last_became_active":"2024-03-24T00:37:02.194433+0000","last_became_peered":"2024-03-24T00:37:02.194433+0000","last_unstale":"2024-03-24T00:37:03.201822+0000","last_undegraded":"2024-03-24T00:37:03.201822+0000","last_fullsized":"2024-03-24T00:37:03.201822+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:08:59.489021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215292+0000","last_change":"2024-03-24T00:37:02.207008+0000","last_active":"2024-03-24T00:37:02.215292+0000","last_peered":"2024-03-24T00:37:02.215292+0000","last_clean":"2024-03-24T00:37:02.215292+0000","last_became_active":"2024-03-24T00:37:02.206710+0000","last_became_peered":"2024-03-24T00:37:02.206710+0000","last_unstale":"2024-03-24T00:37:02.215292+0000","last_undegraded":"2024-03-24T00:37:02.215292+0000","last_fullsized":"2024-03-24T00:37:02.215292+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:03:19.520445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204812+0000","last_change":"2024-03-24T00:37:02.202368+0000","last_active":"2024-03-24T00:37:03.204812+0000","last_peered":"2024-03-24T00:37:03.204812+0000","last_clean":"2024-03-24T00:37:03.204812+0000","last_became_active":"2024-03-24T00:37:02.202074+0000","last_became_peered":"2024-03-24T00:37:02.202074+0000","last_unstale":"2024-03-24T00:37:03.204812+0000","last_undegraded":"2024-03-24T00:37:03.204812+0000","last_fullsized":"2024-03-24T00:37:03.204812+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:59:23.304072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.197881+0000","last_change":"2024-03-24T00:37:02.197028+0000","last_active":"2024-03-24T00:37:03.197881+0000","last_peered":"2024-03-24T00:37:03.197881+0000","last_clean":"2024-03-24T00:37:03.197881+0000","last_became_active":"2024-03-24T00:37:02.196596+0000","last_became_peered":"2024-03-24T00:37:02.196596+0000","last_unstale":"2024-03-24T00:37:03.197881+0000","last_undegraded":"2024-03-24T00:37:03.197881+0000","last_fullsized":"2024-03-24T00:37:03.197881+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:34:02.682865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.216371+0000","last_change":"2024-03-24T00:37:02.209550+0000","last_active":"2024-03-24T00:37:02.216371+0000","last_peered":"2024-03-24T00:37:02.216371+0000","last_clean":"2024-03-24T00:37:02.216371+0000","last_became_active":"2024-03-24T00:37:02.209231+0000","last_became_peered":"2024-03-24T00:37:02.209231+0000","last_unstale":"2024-03-24T00:37:02.216371+0000","last_undegraded":"2024-03-24T00:37:02.216371+0000","last_fullsized":"2024-03-24T00:37:02.216371+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:30.857697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.199869+0000","last_change":"2024-03-24T00:37:02.193735+0000","last_active":"2024-03-24T00:37:02.199869+0000","last_peered":"2024-03-24T00:37:02.199869+0000","last_clean":"2024-03-24T00:37:02.199869+0000","last_became_active":"2024-03-24T00:37:02.192927+0000","last_became_peered":"2024-03-24T00:37:02.192927+0000","last_unstale":"2024-03-24T00:37:02.199869+0000","last_undegraded":"2024-03-24T00:37:02.199869+0000","last_fullsized":"2024-03-24T00:37:02.199869+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:12:06.270215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198442+0000","last_change":"2024-03-24T00:37:02.211783+0000","last_active":"2024-03-24T00:37:03.198442+0000","last_peered":"2024-03-24T00:37:03.198442+0000","last_clean":"2024-03-24T00:37:03.198442+0000","last_became_active":"2024-03-24T00:37:02.211523+0000","last_became_peered":"2024-03-24T00:37:02.211523+0000","last_unstale":"2024-03-24T00:37:03.198442+0000","last_undegraded":"2024-03-24T00:37:03.198442+0000","last_fullsized":"2024-03-24T00:37:03.198442+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:33.828330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"94'4","reported_seq":18,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205257+0000","last_change":"2024-03-24T00:37:01.178257+0000","last_active":"2024-03-24T00:37:03.205257+0000","last_peered":"2024-03-24T00:37:03.205257+0000","last_clean":"2024-03-24T00:37:03.205257+0000","last_became_active":"2024-03-24T00:37:01.171859+0000","last_became_peered":"2024-03-24T00:37:01.171859+0000","last_unstale":"2024-03-24T00:37:03.205257+0000","last_undegraded":"2024-03-24T00:37:03.205257+0000","last_fullsized":"2024-03-24T00:37:03.205257+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:13:58.704360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.216405+0000","last_change":"2024-03-24T00:37:02.212348+0000","last_active":"2024-03-24T00:37:02.216405+0000","last_peered":"2024-03-24T00:37:02.216405+0000","last_clean":"2024-03-24T00:37:02.216405+0000","last_became_active":"2024-03-24T00:37:02.212061+0000","last_became_peered":"2024-03-24T00:37:02.212061+0000","last_unstale":"2024-03-24T00:37:02.216405+0000","last_undegraded":"2024-03-24T00:37:02.216405+0000","last_fullsized":"2024-03-24T00:37:02.216405+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:05.510696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.202024+0000","last_change":"2024-03-24T00:37:01.186120+0000","last_active":"2024-03-24T00:37:03.202024+0000","last_peered":"2024-03-24T00:37:03.202024+0000","last_clean":"2024-03-24T00:37:03.202024+0000","last_became_active":"2024-03-24T00:37:01.185711+0000","last_became_peered":"2024-03-24T00:37:01.185711+0000","last_unstale":"2024-03-24T00:37:03.202024+0000","last_undegraded":"2024-03-24T00:37:03.202024+0000","last_fullsized":"2024-03-24T00:37:03.202024+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:21:07.383330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.201266+0000","last_change":"2024-03-24T00:37:02.196249+0000","last_active":"2024-03-24T00:37:02.201266+0000","last_peered":"2024-03-24T00:37:02.201266+0000","last_clean":"2024-03-24T00:37:02.201266+0000","last_became_active":"2024-03-24T00:37:02.195987+0000","last_became_peered":"2024-03-24T00:37:02.195987+0000","last_unstale":"2024-03-24T00:37:02.201266+0000","last_undegraded":"2024-03-24T00:37:02.201266+0000","last_fullsized":"2024-03-24T00:37:02.201266+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:31.890531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204973+0000","last_change":"2024-03-24T00:37:01.182997+0000","last_active":"2024-03-24T00:37:03.204973+0000","last_peered":"2024-03-24T00:37:03.204973+0000","last_clean":"2024-03-24T00:37:03.204973+0000","last_became_active":"2024-03-24T00:37:01.181663+0000","last_became_peered":"2024-03-24T00:37:01.181663+0000","last_unstale":"2024-03-24T00:37:03.204973+0000","last_undegraded":"2024-03-24T00:37:03.204973+0000","last_fullsized":"2024-03-24T00:37:03.204973+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:35:42.237011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.202025+0000","last_change":"2024-03-24T00:37:02.201155+0000","last_active":"2024-03-24T00:37:03.202025+0000","last_peered":"2024-03-24T00:37:03.202025+0000","last_clean":"2024-03-24T00:37:03.202025+0000","last_became_active":"2024-03-24T00:37:02.200808+0000","last_became_peered":"2024-03-24T00:37:02.200808+0000","last_unstale":"2024-03-24T00:37:03.202025+0000","last_undegraded":"2024-03-24T00:37:03.202025+0000","last_fullsized":"2024-03-24T00:37:03.202025+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:05.510696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204902+0000","last_change":"2024-03-24T00:37:01.176718+0000","last_active":"2024-03-24T00:37:03.204902+0000","last_peered":"2024-03-24T00:37:03.204902+0000","last_clean":"2024-03-24T00:37:03.204902+0000","last_became_active":"2024-03-24T00:37:01.176354+0000","last_became_peered":"2024-03-24T00:37:01.176354+0000","last_unstale":"2024-03-24T00:37:03.204902+0000","last_undegraded":"2024-03-24T00:37:03.204902+0000","last_fullsized":"2024-03-24T00:37:03.204902+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:29.964508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.205779+0000","last_change":"2024-03-24T00:37:02.197368+0000","last_active":"2024-03-24T00:37:02.205779+0000","last_peered":"2024-03-24T00:37:02.205779+0000","last_clean":"2024-03-24T00:37:02.205779+0000","last_became_active":"2024-03-24T00:37:02.197017+0000","last_became_peered":"2024-03-24T00:37:02.197017+0000","last_unstale":"2024-03-24T00:37:02.205779+0000","last_undegraded":"2024-03-24T00:37:02.205779+0000","last_fullsized":"2024-03-24T00:37:02.205779+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:27:24.816115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"94'1","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215857+0000","last_change":"2024-03-24T00:37:01.173373+0000","last_active":"2024-03-24T00:37:02.215857+0000","last_peered":"2024-03-24T00:37:02.215857+0000","last_clean":"2024-03-24T00:37:02.215857+0000","last_became_active":"2024-03-24T00:37:01.171443+0000","last_became_peered":"2024-03-24T00:37:01.171443+0000","last_unstale":"2024-03-24T00:37:02.215857+0000","last_undegraded":"2024-03-24T00:37:02.215857+0000","last_fullsized":"2024-03-24T00:37:02.215857+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:44.218310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201395+0000","last_change":"2024-03-24T00:37:02.211801+0000","last_active":"2024-03-24T00:37:03.201395+0000","last_peered":"2024-03-24T00:37:03.201395+0000","last_clean":"2024-03-24T00:37:03.201395+0000","last_became_active":"2024-03-24T00:37:02.211199+0000","last_became_peered":"2024-03-24T00:37:02.211199+0000","last_unstale":"2024-03-24T00:37:03.201395+0000","last_undegraded":"2024-03-24T00:37:03.201395+0000","last_fullsized":"2024-03-24T00:37:03.201395+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:45:45.591503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204698+0000","last_change":"2024-03-24T00:37:01.166426+0000","last_active":"2024-03-24T00:37:03.204698+0000","last_peered":"2024-03-24T00:37:03.204698+0000","last_clean":"2024-03-24T00:37:03.204698+0000","last_became_active":"2024-03-24T00:37:01.166149+0000","last_became_peered":"2024-03-24T00:37:01.166149+0000","last_unstale":"2024-03-24T00:37:03.204698+0000","last_undegraded":"2024-03-24T00:37:03.204698+0000","last_fullsized":"2024-03-24T00:37:03.204698+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:24.847163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.211301+0000","last_change":"2024-03-24T00:37:02.203024+0000","last_active":"2024-03-24T00:37:02.211301+0000","last_peered":"2024-03-24T00:37:02.211301+0000","last_clean":"2024-03-24T00:37:02.211301+0000","last_became_active":"2024-03-24T00:37:02.201986+0000","last_became_peered":"2024-03-24T00:37:02.201986+0000","last_unstale":"2024-03-24T00:37:02.211301+0000","last_undegraded":"2024-03-24T00:37:02.211301+0000","last_fullsized":"2024-03-24T00:37:02.211301+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:39.876486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.190357+0000","last_change":"2024-03-24T00:37:01.184761+0000","last_active":"2024-03-24T00:37:02.190357+0000","last_peered":"2024-03-24T00:37:02.190357+0000","last_clean":"2024-03-24T00:37:02.190357+0000","last_became_active":"2024-03-24T00:37:01.184391+0000","last_became_peered":"2024-03-24T00:37:01.184391+0000","last_unstale":"2024-03-24T00:37:02.190357+0000","last_undegraded":"2024-03-24T00:37:02.190357+0000","last_fullsized":"2024-03-24T00:37:02.190357+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:41.286975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200954+0000","last_change":"2024-03-24T00:37:02.201265+0000","last_active":"2024-03-24T00:37:03.200954+0000","last_peered":"2024-03-24T00:37:03.200954+0000","last_clean":"2024-03-24T00:37:03.200954+0000","last_became_active":"2024-03-24T00:37:02.200932+0000","last_became_peered":"2024-03-24T00:37:02.200932+0000","last_unstale":"2024-03-24T00:37:03.200954+0000","last_undegraded":"2024-03-24T00:37:03.200954+0000","last_fullsized":"2024-03-24T00:37:03.200954+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:52.195925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.180322+0000","last_change":"2024-03-24T00:37:01.180472+0000","last_active":"2024-03-24T00:37:02.180322+0000","last_peered":"2024-03-24T00:37:02.180322+0000","last_clean":"2024-03-24T00:37:02.180322+0000","last_became_active":"2024-03-24T00:37:01.180162+0000","last_became_peered":"2024-03-24T00:37:01.180162+0000","last_unstale":"2024-03-24T00:37:02.180322+0000","last_undegraded":"2024-03-24T00:37:02.180322+0000","last_fullsized":"2024-03-24T00:37:02.180322+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.819938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205286+0000","last_change":"2024-03-24T00:37:02.203852+0000","last_active":"2024-03-24T00:37:03.205286+0000","last_peered":"2024-03-24T00:37:03.205286+0000","last_clean":"2024-03-24T00:37:03.205286+0000","last_became_active":"2024-03-24T00:37:02.203149+0000","last_became_peered":"2024-03-24T00:37:02.203149+0000","last_unstale":"2024-03-24T00:37:03.205286+0000","last_undegraded":"2024-03-24T00:37:03.205286+0000","last_fullsized":"2024-03-24T00:37:03.205286+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:20.353725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.180885+0000","last_change":"2024-03-24T00:37:01.172991+0000","last_active":"2024-03-24T00:37:02.180885+0000","last_peered":"2024-03-24T00:37:02.180885+0000","last_clean":"2024-03-24T00:37:02.180885+0000","last_became_active":"2024-03-24T00:37:01.172687+0000","last_became_peered":"2024-03-24T00:37:01.172687+0000","last_unstale":"2024-03-24T00:37:02.180885+0000","last_undegraded":"2024-03-24T00:37:02.180885+0000","last_fullsized":"2024-03-24T00:37:02.180885+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:55.041007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.202368+0000","last_change":"2024-03-24T00:37:02.208179+0000","last_active":"2024-03-24T00:37:03.202368+0000","last_peered":"2024-03-24T00:37:03.202368+0000","last_clean":"2024-03-24T00:37:03.202368+0000","last_became_active":"2024-03-24T00:37:02.207854+0000","last_became_peered":"2024-03-24T00:37:02.207854+0000","last_unstale":"2024-03-24T00:37:03.202368+0000","last_undegraded":"2024-03-24T00:37:03.202368+0000","last_fullsized":"2024-03-24T00:37:03.202368+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:15:16.935643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.191682+0000","last_change":"2024-03-24T00:37:01.187103+0000","last_active":"2024-03-24T00:37:02.191682+0000","last_peered":"2024-03-24T00:37:02.191682+0000","last_clean":"2024-03-24T00:37:02.191682+0000","last_became_active":"2024-03-24T00:37:01.186828+0000","last_became_peered":"2024-03-24T00:37:01.186828+0000","last_unstale":"2024-03-24T00:37:02.191682+0000","last_undegraded":"2024-03-24T00:37:02.191682+0000","last_fullsized":"2024-03-24T00:37:02.191682+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:03:18.512052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.212458+0000","last_change":"2024-03-24T00:37:02.198498+0000","last_active":"2024-03-24T00:37:02.212458+0000","last_peered":"2024-03-24T00:37:02.212458+0000","last_clean":"2024-03-24T00:37:02.212458+0000","last_became_active":"2024-03-24T00:37:02.198194+0000","last_became_peered":"2024-03-24T00:37:02.198194+0000","last_unstale":"2024-03-24T00:37:02.212458+0000","last_undegraded":"2024-03-24T00:37:02.212458+0000","last_fullsized":"2024-03-24T00:37:02.212458+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:43.825403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"94'3","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201984+0000","last_change":"2024-03-24T00:37:01.180675+0000","last_active":"2024-03-24T00:37:03.201984+0000","last_peered":"2024-03-24T00:37:03.201984+0000","last_clean":"2024-03-24T00:37:03.201984+0000","last_became_active":"2024-03-24T00:37:01.180346+0000","last_became_peered":"2024-03-24T00:37:01.180346+0000","last_unstale":"2024-03-24T00:37:03.201984+0000","last_undegraded":"2024-03-24T00:37:03.201984+0000","last_fullsized":"2024-03-24T00:37:03.201984+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:38.868093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201860+0000","last_change":"2024-03-24T00:37:02.198375+0000","last_active":"2024-03-24T00:37:03.201860+0000","last_peered":"2024-03-24T00:37:03.201860+0000","last_clean":"2024-03-24T00:37:03.201860+0000","last_became_active":"2024-03-24T00:37:02.198080+0000","last_became_peered":"2024-03-24T00:37:02.198080+0000","last_unstale":"2024-03-24T00:37:03.201860+0000","last_undegraded":"2024-03-24T00:37:03.201860+0000","last_fullsized":"2024-03-24T00:37:03.201860+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:30:56.839768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"94'6","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201563+0000","last_change":"2024-03-24T00:37:01.183010+0000","last_active":"2024-03-24T00:37:03.201563+0000","last_peered":"2024-03-24T00:37:03.201563+0000","last_clean":"2024-03-24T00:37:03.201563+0000","last_became_active":"2024-03-24T00:37:01.182663+0000","last_became_peered":"2024-03-24T00:37:01.182663+0000","last_unstale":"2024-03-24T00:37:03.201563+0000","last_undegraded":"2024-03-24T00:37:03.201563+0000","last_fullsized":"2024-03-24T00:37:03.201563+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:22:27.182497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.216407+0000","last_change":"2024-03-24T00:37:02.210695+0000","last_active":"2024-03-24T00:37:02.216407+0000","last_peered":"2024-03-24T00:37:02.216407+0000","last_clean":"2024-03-24T00:37:02.216407+0000","last_became_active":"2024-03-24T00:37:02.210387+0000","last_became_peered":"2024-03-24T00:37:02.210387+0000","last_unstale":"2024-03-24T00:37:02.216407+0000","last_undegraded":"2024-03-24T00:37:02.216407+0000","last_fullsized":"2024-03-24T00:37:02.216407+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:30.972900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198584+0000","last_change":"2024-03-24T00:37:01.175110+0000","last_active":"2024-03-24T00:37:03.198584+0000","last_peered":"2024-03-24T00:37:03.198584+0000","last_clean":"2024-03-24T00:37:03.198584+0000","last_became_active":"2024-03-24T00:37:01.174831+0000","last_became_peered":"2024-03-24T00:37:01.174831+0000","last_unstale":"2024-03-24T00:37:03.198584+0000","last_undegraded":"2024-03-24T00:37:03.198584+0000","last_fullsized":"2024-03-24T00:37:03.198584+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:38:35.985589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.197916+0000","last_change":"2024-03-24T00:37:02.192522+0000","last_active":"2024-03-24T00:37:02.197916+0000","last_peered":"2024-03-24T00:37:02.197916+0000","last_clean":"2024-03-24T00:37:02.197916+0000","last_became_active":"2024-03-24T00:37:02.192133+0000","last_became_peered":"2024-03-24T00:37:02.192133+0000","last_unstale":"2024-03-24T00:37:02.197916+0000","last_undegraded":"2024-03-24T00:37:02.197916+0000","last_fullsized":"2024-03-24T00:37:02.197916+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:39.814622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198283+0000","last_change":"2024-03-24T00:37:01.183666+0000","last_active":"2024-03-24T00:37:03.198283+0000","last_peered":"2024-03-24T00:37:03.198283+0000","last_clean":"2024-03-24T00:37:03.198283+0000","last_became_active":"2024-03-24T00:37:01.182569+0000","last_became_peered":"2024-03-24T00:37:01.182569+0000","last_unstale":"2024-03-24T00:37:03.198283+0000","last_undegraded":"2024-03-24T00:37:03.198283+0000","last_fullsized":"2024-03-24T00:37:03.198283+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:07.560652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200342+0000","last_change":"2024-03-24T00:37:02.208886+0000","last_active":"2024-03-24T00:37:03.200342+0000","last_peered":"2024-03-24T00:37:03.200342+0000","last_clean":"2024-03-24T00:37:03.200342+0000","last_became_active":"2024-03-24T00:37:02.208613+0000","last_became_peered":"2024-03-24T00:37:02.208613+0000","last_unstale":"2024-03-24T00:37:03.200342+0000","last_undegraded":"2024-03-24T00:37:03.200342+0000","last_fullsized":"2024-03-24T00:37:03.200342+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:39.876486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200918+0000","last_change":"2024-03-24T00:37:01.174927+0000","last_active":"2024-03-24T00:37:03.200918+0000","last_peered":"2024-03-24T00:37:03.200918+0000","last_clean":"2024-03-24T00:37:03.200918+0000","last_became_active":"2024-03-24T00:37:01.174315+0000","last_became_peered":"2024-03-24T00:37:01.174315+0000","last_unstale":"2024-03-24T00:37:03.200918+0000","last_undegraded":"2024-03-24T00:37:03.200918+0000","last_fullsized":"2024-03-24T00:37:03.200918+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:41.286975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.197890+0000","last_change":"2024-03-24T00:37:02.207182+0000","last_active":"2024-03-24T00:37:03.197890+0000","last_peered":"2024-03-24T00:37:03.197890+0000","last_clean":"2024-03-24T00:37:03.197890+0000","last_became_active":"2024-03-24T00:37:02.206278+0000","last_became_peered":"2024-03-24T00:37:02.206278+0000","last_unstale":"2024-03-24T00:37:03.197890+0000","last_undegraded":"2024-03-24T00:37:03.197890+0000","last_fullsized":"2024-03-24T00:37:03.197890+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:09:32.007020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"94'8","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.220974+0000","last_change":"2024-03-24T00:37:01.179650+0000","last_active":"2024-03-24T00:37:02.220974+0000","last_peered":"2024-03-24T00:37:02.220974+0000","last_clean":"2024-03-24T00:37:02.220974+0000","last_became_active":"2024-03-24T00:37:01.177991+0000","last_became_peered":"2024-03-24T00:37:01.177991+0000","last_unstale":"2024-03-24T00:37:02.220974+0000","last_undegraded":"2024-03-24T00:37:02.220974+0000","last_fullsized":"2024-03-24T00:37:02.220974+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:52:17.608291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.214697+0000","last_change":"2024-03-24T00:37:02.202987+0000","last_active":"2024-03-24T00:37:02.214697+0000","last_peered":"2024-03-24T00:37:02.214697+0000","last_clean":"2024-03-24T00:37:02.214697+0000","last_became_active":"2024-03-24T00:37:02.202454+0000","last_became_peered":"2024-03-24T00:37:02.202454+0000","last_unstale":"2024-03-24T00:37:02.214697+0000","last_undegraded":"2024-03-24T00:37:02.214697+0000","last_fullsized":"2024-03-24T00:37:02.214697+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:59.736598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.202286+0000","last_change":"2024-03-24T00:37:01.182857+0000","last_active":"2024-03-24T00:37:03.202286+0000","last_peered":"2024-03-24T00:37:03.202286+0000","last_clean":"2024-03-24T00:37:03.202286+0000","last_became_active":"2024-03-24T00:37:01.182545+0000","last_became_peered":"2024-03-24T00:37:01.182545+0000","last_unstale":"2024-03-24T00:37:03.202286+0000","last_undegraded":"2024-03-24T00:37:03.202286+0000","last_fullsized":"2024-03-24T00:37:03.202286+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:34:57.865809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.212940+0000","last_change":"2024-03-24T00:37:02.209335+0000","last_active":"2024-03-24T00:37:02.212940+0000","last_peered":"2024-03-24T00:37:02.212940+0000","last_clean":"2024-03-24T00:37:02.212940+0000","last_became_active":"2024-03-24T00:37:02.209039+0000","last_became_peered":"2024-03-24T00:37:02.209039+0000","last_unstale":"2024-03-24T00:37:02.212940+0000","last_undegraded":"2024-03-24T00:37:02.212940+0000","last_fullsized":"2024-03-24T00:37:02.212940+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:22:31.502603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"94'1","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215770+0000","last_change":"2024-03-24T00:37:01.173919+0000","last_active":"2024-03-24T00:37:02.215770+0000","last_peered":"2024-03-24T00:37:02.215770+0000","last_clean":"2024-03-24T00:37:02.215770+0000","last_became_active":"2024-03-24T00:37:01.172497+0000","last_became_peered":"2024-03-24T00:37:01.172497+0000","last_unstale":"2024-03-24T00:37:02.215770+0000","last_undegraded":"2024-03-24T00:37:02.215770+0000","last_fullsized":"2024-03-24T00:37:02.215770+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:22:27.182497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201995+0000","last_change":"2024-03-24T00:37:02.195888+0000","last_active":"2024-03-24T00:37:03.201995+0000","last_peered":"2024-03-24T00:37:03.201995+0000","last_clean":"2024-03-24T00:37:03.201995+0000","last_became_active":"2024-03-24T00:37:02.195113+0000","last_became_peered":"2024-03-24T00:37:02.195113+0000","last_unstale":"2024-03-24T00:37:03.201995+0000","last_undegraded":"2024-03-24T00:37:03.201995+0000","last_fullsized":"2024-03-24T00:37:03.201995+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:34:26.165155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204971+0000","last_change":"2024-03-24T00:37:01.165422+0000","last_active":"2024-03-24T00:37:03.204971+0000","last_peered":"2024-03-24T00:37:03.204971+0000","last_clean":"2024-03-24T00:37:03.204971+0000","last_became_active":"2024-03-24T00:37:01.165128+0000","last_became_peered":"2024-03-24T00:37:01.165128+0000","last_unstale":"2024-03-24T00:37:03.204971+0000","last_undegraded":"2024-03-24T00:37:03.204971+0000","last_fullsized":"2024-03-24T00:37:03.204971+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:04.502303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198203+0000","last_change":"2024-03-24T00:37:02.205039+0000","last_active":"2024-03-24T00:37:03.198203+0000","last_peered":"2024-03-24T00:37:03.198203+0000","last_clean":"2024-03-24T00:37:03.198203+0000","last_became_active":"2024-03-24T00:37:02.204651+0000","last_became_peered":"2024-03-24T00:37:02.204651+0000","last_unstale":"2024-03-24T00:37:03.198203+0000","last_undegraded":"2024-03-24T00:37:03.198203+0000","last_fullsized":"2024-03-24T00:37:03.198203+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:57.424357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204923+0000","last_change":"2024-03-24T00:37:01.176582+0000","last_active":"2024-03-24T00:37:03.204923+0000","last_peered":"2024-03-24T00:37:03.204923+0000","last_clean":"2024-03-24T00:37:03.204923+0000","last_became_active":"2024-03-24T00:37:01.176281+0000","last_became_peered":"2024-03-24T00:37:01.176281+0000","last_unstale":"2024-03-24T00:37:03.204923+0000","last_undegraded":"2024-03-24T00:37:03.204923+0000","last_fullsized":"2024-03-24T00:37:03.204923+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:33.764642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198396+0000","last_change":"2024-03-24T00:37:02.198104+0000","last_active":"2024-03-24T00:37:03.198396+0000","last_peered":"2024-03-24T00:37:03.198396+0000","last_clean":"2024-03-24T00:37:03.198396+0000","last_became_active":"2024-03-24T00:37:02.197784+0000","last_became_peered":"2024-03-24T00:37:02.197784+0000","last_unstale":"2024-03-24T00:37:03.198396+0000","last_undegraded":"2024-03-24T00:37:03.198396+0000","last_fullsized":"2024-03-24T00:37:03.198396+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:45.226702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.186026+0000","last_change":"2024-03-24T00:37:01.175712+0000","last_active":"2024-03-24T00:37:02.186026+0000","last_peered":"2024-03-24T00:37:02.186026+0000","last_clean":"2024-03-24T00:37:02.186026+0000","last_became_active":"2024-03-24T00:37:01.173276+0000","last_became_peered":"2024-03-24T00:37:01.173276+0000","last_unstale":"2024-03-24T00:37:02.186026+0000","last_undegraded":"2024-03-24T00:37:02.186026+0000","last_fullsized":"2024-03-24T00:37:02.186026+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:10:18.273637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.197861+0000","last_change":"2024-03-24T00:37:02.195524+0000","last_active":"2024-03-24T00:37:03.197861+0000","last_peered":"2024-03-24T00:37:03.197861+0000","last_clean":"2024-03-24T00:37:03.197861+0000","last_became_active":"2024-03-24T00:37:02.195222+0000","last_became_peered":"2024-03-24T00:37:02.195222+0000","last_unstale":"2024-03-24T00:37:03.197861+0000","last_undegraded":"2024-03-24T00:37:03.197861+0000","last_fullsized":"2024-03-24T00:37:03.197861+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:16.942624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.185124+0000","last_change":"2024-03-24T00:37:01.184432+0000","last_active":"2024-03-24T00:37:02.185124+0000","last_peered":"2024-03-24T00:37:02.185124+0000","last_clean":"2024-03-24T00:37:02.185124+0000","last_became_active":"2024-03-24T00:37:01.184079+0000","last_became_peered":"2024-03-24T00:37:01.184079+0000","last_unstale":"2024-03-24T00:37:02.185124+0000","last_undegraded":"2024-03-24T00:37:02.185124+0000","last_fullsized":"2024-03-24T00:37:02.185124+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:38.827292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.214609+0000","last_change":"2024-03-24T00:37:02.199703+0000","last_active":"2024-03-24T00:37:02.214609+0000","last_peered":"2024-03-24T00:37:02.214609+0000","last_clean":"2024-03-24T00:37:02.214609+0000","last_became_active":"2024-03-24T00:37:02.197761+0000","last_became_peered":"2024-03-24T00:37:02.197761+0000","last_unstale":"2024-03-24T00:37:02.214609+0000","last_undegraded":"2024-03-24T00:37:02.214609+0000","last_fullsized":"2024-03-24T00:37:02.214609+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:54:59.045039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204648+0000","last_change":"2024-03-24T00:37:01.166659+0000","last_active":"2024-03-24T00:37:03.204648+0000","last_peered":"2024-03-24T00:37:03.204648+0000","last_clean":"2024-03-24T00:37:03.204648+0000","last_became_active":"2024-03-24T00:37:01.165603+0000","last_became_peered":"2024-03-24T00:37:01.165603+0000","last_unstale":"2024-03-24T00:37:03.204648+0000","last_undegraded":"2024-03-24T00:37:03.204648+0000","last_fullsized":"2024-03-24T00:37:03.204648+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:51.187532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205303+0000","last_change":"2024-03-24T00:37:02.204072+0000","last_active":"2024-03-24T00:37:03.205303+0000","last_peered":"2024-03-24T00:37:03.205303+0000","last_clean":"2024-03-24T00:37:03.205303+0000","last_became_active":"2024-03-24T00:37:02.203597+0000","last_became_peered":"2024-03-24T00:37:02.203597+0000","last_unstale":"2024-03-24T00:37:03.205303+0000","last_undegraded":"2024-03-24T00:37:03.205303+0000","last_fullsized":"2024-03-24T00:37:03.205303+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:16.914566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"94'2","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.222059+0000","last_change":"2024-03-24T00:37:01.175623+0000","last_active":"2024-03-24T00:37:02.222059+0000","last_peered":"2024-03-24T00:37:02.222059+0000","last_clean":"2024-03-24T00:37:02.222059+0000","last_became_active":"2024-03-24T00:37:01.175183+0000","last_became_peered":"2024-03-24T00:37:01.175183+0000","last_unstale":"2024-03-24T00:37:02.222059+0000","last_undegraded":"2024-03-24T00:37:02.222059+0000","last_fullsized":"2024-03-24T00:37:02.222059+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:05.549402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.201391+0000","last_change":"2024-03-24T00:37:02.197487+0000","last_active":"2024-03-24T00:37:02.201391+0000","last_peered":"2024-03-24T00:37:02.201391+0000","last_clean":"2024-03-24T00:37:02.201391+0000","last_became_active":"2024-03-24T00:37:02.193759+0000","last_became_peered":"2024-03-24T00:37:02.193759+0000","last_unstale":"2024-03-24T00:37:02.201391+0000","last_undegraded":"2024-03-24T00:37:02.201391+0000","last_fullsized":"2024-03-24T00:37:02.201391+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:25:47.230816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"95'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.206584+0000","last_change":"2024-03-24T00:37:01.183457+0000","last_active":"2024-03-24T00:37:03.206584+0000","last_peered":"2024-03-24T00:37:03.206584+0000","last_clean":"2024-03-24T00:37:03.206584+0000","last_became_active":"2024-03-24T00:37:01.181903+0000","last_became_peered":"2024-03-24T00:37:01.181903+0000","last_unstale":"2024-03-24T00:37:03.206584+0000","last_undegraded":"2024-03-24T00:37:03.206584+0000","last_fullsized":"2024-03-24T00:37:03.206584+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:03:12.503349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.207972+0000","last_change":"2024-03-24T00:37:02.207184+0000","last_active":"2024-03-24T00:37:02.207972+0000","last_peered":"2024-03-24T00:37:02.207972+0000","last_clean":"2024-03-24T00:37:02.207972+0000","last_became_active":"2024-03-24T00:37:02.206753+0000","last_became_peered":"2024-03-24T00:37:02.206753+0000","last_unstale":"2024-03-24T00:37:02.207972+0000","last_undegraded":"2024-03-24T00:37:02.207972+0000","last_fullsized":"2024-03-24T00:37:02.207972+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:57.728969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205064+0000","last_change":"2024-03-24T00:37:01.174263+0000","last_active":"2024-03-24T00:37:03.205064+0000","last_peered":"2024-03-24T00:37:03.205064+0000","last_clean":"2024-03-24T00:37:03.205064+0000","last_became_active":"2024-03-24T00:37:01.173928+0000","last_became_peered":"2024-03-24T00:37:01.173928+0000","last_unstale":"2024-03-24T00:37:03.205064+0000","last_undegraded":"2024-03-24T00:37:03.205064+0000","last_fullsized":"2024-03-24T00:37:03.205064+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:15:15.927250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215636+0000","last_change":"2024-03-24T00:37:02.199921+0000","last_active":"2024-03-24T00:37:02.215636+0000","last_peered":"2024-03-24T00:37:02.215636+0000","last_clean":"2024-03-24T00:37:02.215636+0000","last_became_active":"2024-03-24T00:37:02.199538+0000","last_became_peered":"2024-03-24T00:37:02.199538+0000","last_unstale":"2024-03-24T00:37:02.215636+0000","last_undegraded":"2024-03-24T00:37:02.215636+0000","last_fullsized":"2024-03-24T00:37:02.215636+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:39.876486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.191939+0000","last_change":"2024-03-24T00:37:01.175025+0000","last_active":"2024-03-24T00:37:02.191939+0000","last_peered":"2024-03-24T00:37:02.191939+0000","last_clean":"2024-03-24T00:37:02.191939+0000","last_became_active":"2024-03-24T00:37:01.174249+0000","last_became_peered":"2024-03-24T00:37:01.174249+0000","last_unstale":"2024-03-24T00:37:02.191939+0000","last_undegraded":"2024-03-24T00:37:02.191939+0000","last_fullsized":"2024-03-24T00:37:02.191939+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:34:57.865809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198184+0000","last_change":"2024-03-24T00:37:02.201036+0000","last_active":"2024-03-24T00:37:03.198184+0000","last_peered":"2024-03-24T00:37:03.198184+0000","last_clean":"2024-03-24T00:37:03.198184+0000","last_became_active":"2024-03-24T00:37:02.200733+0000","last_became_peered":"2024-03-24T00:37:02.200733+0000","last_unstale":"2024-03-24T00:37:03.198184+0000","last_undegraded":"2024-03-24T00:37:03.198184+0000","last_fullsized":"2024-03-24T00:37:03.198184+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:04:01.844409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204945+0000","last_change":"2024-03-24T00:37:01.181936+0000","last_active":"2024-03-24T00:37:03.204945+0000","last_peered":"2024-03-24T00:37:03.204945+0000","last_clean":"2024-03-24T00:37:03.204945+0000","last_became_active":"2024-03-24T00:37:01.181622+0000","last_became_peered":"2024-03-24T00:37:01.181622+0000","last_unstale":"2024-03-24T00:37:03.204945+0000","last_undegraded":"2024-03-24T00:37:03.204945+0000","last_fullsized":"2024-03-24T00:37:03.204945+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:29.849304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201143+0000","last_change":"2024-03-24T00:37:02.212383+0000","last_active":"2024-03-24T00:37:03.201143+0000","last_peered":"2024-03-24T00:37:03.201143+0000","last_clean":"2024-03-24T00:37:03.201143+0000","last_became_active":"2024-03-24T00:37:02.212077+0000","last_became_peered":"2024-03-24T00:37:02.212077+0000","last_unstale":"2024-03-24T00:37:03.201143+0000","last_undegraded":"2024-03-24T00:37:03.201143+0000","last_fullsized":"2024-03-24T00:37:03.201143+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:34.773035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.190293+0000","last_change":"2024-03-24T00:37:01.179834+0000","last_active":"2024-03-24T00:37:02.190293+0000","last_peered":"2024-03-24T00:37:02.190293+0000","last_clean":"2024-03-24T00:37:02.190293+0000","last_became_active":"2024-03-24T00:37:01.179519+0000","last_became_peered":"2024-03-24T00:37:01.179519+0000","last_unstale":"2024-03-24T00:37:02.190293+0000","last_undegraded":"2024-03-24T00:37:02.190293+0000","last_fullsized":"2024-03-24T00:37:02.190293+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:14:37.043174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.197861+0000","last_change":"2024-03-24T00:37:02.194387+0000","last_active":"2024-03-24T00:37:03.197861+0000","last_peered":"2024-03-24T00:37:03.197861+0000","last_clean":"2024-03-24T00:37:03.197861+0000","last_became_active":"2024-03-24T00:37:02.194122+0000","last_became_peered":"2024-03-24T00:37:02.194122+0000","last_unstale":"2024-03-24T00:37:03.197861+0000","last_undegraded":"2024-03-24T00:37:03.197861+0000","last_fullsized":"2024-03-24T00:37:03.197861+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:21.975505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.191038+0000","last_change":"2024-03-24T00:37:01.181575+0000","last_active":"2024-03-24T00:37:02.191038+0000","last_peered":"2024-03-24T00:37:02.191038+0000","last_clean":"2024-03-24T00:37:02.191038+0000","last_became_active":"2024-03-24T00:37:01.181148+0000","last_became_peered":"2024-03-24T00:37:01.181148+0000","last_unstale":"2024-03-24T00:37:02.191038+0000","last_undegraded":"2024-03-24T00:37:02.191038+0000","last_fullsized":"2024-03-24T00:37:02.191038+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:54.794241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.214997+0000","last_change":"2024-03-24T00:37:02.203117+0000","last_active":"2024-03-24T00:37:02.214997+0000","last_peered":"2024-03-24T00:37:02.214997+0000","last_clean":"2024-03-24T00:37:02.214997+0000","last_became_active":"2024-03-24T00:37:02.200867+0000","last_became_peered":"2024-03-24T00:37:02.200867+0000","last_unstale":"2024-03-24T00:37:02.214997+0000","last_undegraded":"2024-03-24T00:37:02.214997+0000","last_fullsized":"2024-03-24T00:37:02.214997+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:55.802633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.197987+0000","last_change":"2024-03-24T00:37:01.176391+0000","last_active":"2024-03-24T00:37:03.197987+0000","last_peered":"2024-03-24T00:37:03.197987+0000","last_clean":"2024-03-24T00:37:03.197987+0000","last_became_active":"2024-03-24T00:37:01.176117+0000","last_became_peered":"2024-03-24T00:37:01.176117+0000","last_unstale":"2024-03-24T00:37:03.197987+0000","last_undegraded":"2024-03-24T00:37:03.197987+0000","last_fullsized":"2024-03-24T00:37:03.197987+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:56:47.797989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198330+0000","last_change":"2024-03-24T00:37:02.209726+0000","last_active":"2024-03-24T00:37:03.198330+0000","last_peered":"2024-03-24T00:37:03.198330+0000","last_clean":"2024-03-24T00:37:03.198330+0000","last_became_active":"2024-03-24T00:37:02.207157+0000","last_became_peered":"2024-03-24T00:37:02.207157+0000","last_unstale":"2024-03-24T00:37:03.198330+0000","last_undegraded":"2024-03-24T00:37:03.198330+0000","last_fullsized":"2024-03-24T00:37:03.198330+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:09.807975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.202195+0000","last_change":"2024-03-24T00:37:01.185186+0000","last_active":"2024-03-24T00:37:03.202195+0000","last_peered":"2024-03-24T00:37:03.202195+0000","last_clean":"2024-03-24T00:37:03.202195+0000","last_became_active":"2024-03-24T00:37:01.184837+0000","last_became_peered":"2024-03-24T00:37:01.184837+0000","last_unstale":"2024-03-24T00:37:03.202195+0000","last_undegraded":"2024-03-24T00:37:03.202195+0000","last_fullsized":"2024-03-24T00:37:03.202195+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:03:18.512052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":322,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200439+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:37:03.200439+0000","last_peered":"2024-03-24T00:37:03.200439+0000","last_clean":"2024-03-24T00:37:03.200439+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:37:03.200439+0000","last_undegraded":"2024-03-24T00:37:03.200439+0000","last_fullsized":"2024-03-24T00:37:03.200439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574924,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1765376,"data_stored":1095256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36488,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335352,"statfs":{"total":96636764160,"available":96599400448,"internally_reserved":0,"allocated":1171456,"data_stored":505428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574925,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36480,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335360,"statfs":{"total":96636764160,"available":96599408640,"internally_reserved":0,"allocated":1167360,"data_stored":504846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40668,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331172,"statfs":{"total":96636764160,"available":96595120128,"internally_reserved":0,"allocated":1196032,"data_stored":512812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":1179648,"data_stored":506410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36484,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335356,"statfs":{"total":96636764160,"available":96599404544,"internally_reserved":0,"allocated":1167360,"data_stored":504810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":1163264,"data_stored":504776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574924,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37076,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334764,"statfs":{"total":96636764160,"available":96598798336,"internally_reserved":0,"allocated":1761280,"data_stored":1095726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:37:06.306 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:37:06.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d3200640 1 Processor -- start 2024-03-24T00:37:06.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d3200640 1 -- start start 2024-03-24T00:37:06.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d3200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd9d415bca0 0x7fd9d40d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d3200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9d415c080 0x7fd9d40d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d3200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d40d0780 con 0x7fd9d40ce490 2024-03-24T00:37:06.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d3200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d40b4a90 con 0x7fd9d415bca0 2024-03-24T00:37:06.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d3200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40b4d70 con 0x7fd9d415c080 2024-03-24T00:37:06.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d2800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd9d40ce490 0x7fd9d40cc5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44842/0 (socket says 172.21.15.40:44842) 2024-03-24T00:37:06.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d2800640 1 -- 172.21.15.40:0/2838114059 learned_addr learned my addr 172.21.15.40:0/2838114059 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:06.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d1e00640 1 --2- 172.21.15.40:0/2838114059 >> v2:172.21.15.53:3300/0 conn(0x7fd9d415bca0 0x7fd9d40d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9cbe00640 1 --2- 172.21.15.40:0/2838114059 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9d415c080 0x7fd9d40d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d1400640 1 -- 172.21.15.40:0/2838114059 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352585798 0 0) 0x7fd9d40d0780 con 0x7fd9d40ce490 2024-03-24T00:37:06.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d1400640 1 -- 172.21.15.40:0/2838114059 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9b8003530 con 0x7fd9d40ce490 2024-03-24T00:37:06.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d1e00640 1 -- 172.21.15.40:0/2838114059 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9d415c080 msgr2=0x7fd9d40d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:06.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d1e00640 1 --2- 172.21.15.40:0/2838114059 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9d415c080 0x7fd9d40d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d1e00640 1 -- 172.21.15.40:0/2838114059 >> v1:172.21.15.40:6789/0 conn(0x7fd9d40ce490 legacy=0x7fd9d40cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.485+0000 7fd9d1e00640 1 -- 172.21.15.40:0/2838114059 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d40b5050 con 0x7fd9d415bca0 2024-03-24T00:37:06.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d1e00640 1 --2- 172.21.15.40:0/2838114059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9d415bca0 0x7fd9d40d2590 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40097c0 tx=0x7fd9c401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d6ee22e5fcbded8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:06.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d1400640 1 -- 172.21.15.40:0/2838114059 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c40b0070 con 0x7fd9d415bca0 2024-03-24T00:37:06.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d1400640 1 -- 172.21.15.40:0/2838114059 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c40ab600 con 0x7fd9d415bca0 2024-03-24T00:37:06.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d1400640 1 -- 172.21.15.40:0/2838114059 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c40ab9d0 con 0x7fd9d415bca0 2024-03-24T00:37:06.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d3200640 1 -- 172.21.15.40:0/2838114059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9d415bca0 msgr2=0x7fd9d40d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d3200640 1 --2- 172.21.15.40:0/2838114059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9d415bca0 0x7fd9d40d2590 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40097c0 tx=0x7fd9c401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d3200640 1 -- 172.21.15.40:0/2838114059 shutdown_connections 2024-03-24T00:37:06.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d3200640 1 --2- 172.21.15.40:0/2838114059 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9d415c080 0x7fd9d40d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d3200640 1 --2- 172.21.15.40:0/2838114059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9d415bca0 0x7fd9d40d2590 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d3200640 1 -- 172.21.15.40:0/2838114059 >> 172.21.15.40:0/2838114059 conn(0x7fd9d40d1aa0 msgr2=0x7fd9d40d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:06.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d3200640 1 -- 172.21.15.40:0/2838114059 shutdown_connections 2024-03-24T00:37:06.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d3200640 1 -- 172.21.15.40:0/2838114059 wait complete. 2024-03-24T00:37:06.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d3200640 1 Processor -- start 2024-03-24T00:37:06.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.489+0000 7fd9d3200640 1 -- start start 2024-03-24T00:37:06.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9d3200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9d40ce490 0x7fd9d40c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9d3200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9d415bca0 0x7fd9d40cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9d3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9d415c080 0x7fd9d40cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9d3200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d4168c50 con 0x7fd9d415bca0 2024-03-24T00:37:06.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9d3200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40cd4d0 con 0x7fd9d415c080 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9d3200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d4168dd0 con 0x7fd9d40ce490 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9d415bca0 0x7fd9d40cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9cbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9d415bca0 0x7fd9d40cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40104/0 (socket says 172.21.15.40:40104) 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9cbe00640 1 -- 172.21.15.40:0/3099498830 learned_addr learned my addr 172.21.15.40:0/3099498830 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9d1e00640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9d40ce490 0x7fd9d40c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9d2800640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9d415c080 0x7fd9d40cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9cbe00640 1 -- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9d40ce490 msgr2=0x7fd9d40c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9cbe00640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9d40ce490 0x7fd9d40c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9cbe00640 1 -- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9d415c080 msgr2=0x7fd9d40cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9cbe00640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9d415c080 0x7fd9d40cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9cbe00640 1 -- 172.21.15.40:0/3099498830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d41faae0 con 0x7fd9d415bca0 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9cbe00640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9d415bca0 0x7fd9d40cab80 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00e900 tx=0x7fd9bc00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:06.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9ca000640 1 -- 172.21.15.40:0/3099498830 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc017070 con 0x7fd9d415bca0 2024-03-24T00:37:06.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9ca000640 1 -- 172.21.15.40:0/3099498830 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9bc00ca00 con 0x7fd9d415bca0 2024-03-24T00:37:06.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9ca000640 1 -- 172.21.15.40:0/3099498830 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc00cdf0 con 0x7fd9d415bca0 2024-03-24T00:37:06.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9d3200640 1 -- 172.21.15.40:0/3099498830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d41fadc0 con 0x7fd9d415bca0 2024-03-24T00:37:06.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9d3200640 1 -- 172.21.15.40:0/3099498830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d41fb380 con 0x7fd9d415bca0 2024-03-24T00:37:06.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9d3200640 1 -- 172.21.15.40:0/3099498830 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d409fb10 con 0x7fd9d415bca0 2024-03-24T00:37:06.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.493+0000 7fd9ca000640 1 -- 172.21.15.40:0/3099498830 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd9bc01a070 con 0x7fd9d415bca0 2024-03-24T00:37:06.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.497+0000 7fd9ca000640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd9b0067660 0x7fd9b0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.497+0000 7fd9ca000640 1 -- 172.21.15.40:0/3099498830 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd9bc085b80 con 0x7fd9d415bca0 2024-03-24T00:37:06.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.497+0000 7fd9d1e00640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd9b0067660 0x7fd9b0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.497+0000 7fd9d1e00640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd9b0067660 0x7fd9b0069b20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40ad000 tx=0x7fd9c40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:06.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.501+0000 7fd9ca000640 1 -- 172.21.15.40:0/3099498830 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd9bc0561b0 con 0x7fd9d415bca0 2024-03-24T00:37:06.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.657+0000 7fd9d3200640 1 -- 172.21.15.40:0/3099498830 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9d40c8900 con 0x7fd9b0067660 2024-03-24T00:37:06.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.665+0000 7fd9ca000640 1 -- 172.21.15.40:0/3099498830 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252058 (secure 0 0 0) 0x7fd9d40c8900 con 0x7fd9b0067660 2024-03-24T00:37:06.672 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:06.673 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:37:06.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.669+0000 7fd9d3200640 1 -- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd9b0067660 msgr2=0x7fd9b0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.669+0000 7fd9d3200640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd9b0067660 0x7fd9b0069b20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40ad000 tx=0x7fd9c40a6000 comp rx=0 tx=0).stop 2024-03-24T00:37:06.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.669+0000 7fd9d3200640 1 -- 172.21.15.40:0/3099498830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9d415bca0 msgr2=0x7fd9d40cab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.669+0000 7fd9d3200640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9d415bca0 0x7fd9d40cab80 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00e900 tx=0x7fd9bc00edc0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.669+0000 7fd9d3200640 1 -- 172.21.15.40:0/3099498830 shutdown_connections 2024-03-24T00:37:06.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.669+0000 7fd9d3200640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd9b0067660 0x7fd9b0069b20 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.669+0000 7fd9d3200640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd9d415c080 0x7fd9d40cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.669+0000 7fd9d3200640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd9d415bca0 0x7fd9d40cab80 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.669+0000 7fd9d3200640 1 --2- 172.21.15.40:0/3099498830 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd9d40ce490 0x7fd9d40c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.669+0000 7fd9d3200640 1 -- 172.21.15.40:0/3099498830 >> 172.21.15.40:0/3099498830 conn(0x7fd9d40d1aa0 msgr2=0x7fd9d406c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:06.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.669+0000 7fd9d3200640 1 -- 172.21.15.40:0/3099498830 shutdown_connections 2024-03-24T00:37:06.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.669+0000 7fd9d3200640 1 -- 172.21.15.40:0/3099498830 wait complete. 2024-03-24T00:37:06.693 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-03-24T00:37:05.270291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":297248,"kb_used_data":10324,"kb_used_omap":12,"kb_used_meta":286771,"kb_avail":754677472,"statfs":{"total":773094113280,"available":772789731328,"internally_reserved":0,"allocated":10571776,"data_stored":5230064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":293654066},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.114368"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.204513+0000","last_change":"2024-03-24T00:37:02.200895+0000","last_active":"2024-03-24T00:37:02.204513+0000","last_peered":"2024-03-24T00:37:02.204513+0000","last_clean":"2024-03-24T00:37:02.204513+0000","last_became_active":"2024-03-24T00:37:02.199999+0000","last_became_peered":"2024-03-24T00:37:02.199999+0000","last_unstale":"2024-03-24T00:37:02.204513+0000","last_undegraded":"2024-03-24T00:37:02.204513+0000","last_fullsized":"2024-03-24T00:37:02.204513+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:28:38.887578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205134+0000","last_change":"2024-03-24T00:37:02.215943+0000","last_active":"2024-03-24T00:37:03.205134+0000","last_peered":"2024-03-24T00:37:03.205134+0000","last_clean":"2024-03-24T00:37:03.205134+0000","last_became_active":"2024-03-24T00:37:02.215643+0000","last_became_peered":"2024-03-24T00:37:02.215643+0000","last_unstale":"2024-03-24T00:37:03.205134+0000","last_undegraded":"2024-03-24T00:37:03.205134+0000","last_fullsized":"2024-03-24T00:37:03.205134+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:34:21.769977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.204221+0000","last_change":"2024-03-24T00:37:02.200559+0000","last_active":"2024-03-24T00:37:02.204221+0000","last_peered":"2024-03-24T00:37:02.204221+0000","last_clean":"2024-03-24T00:37:02.204221+0000","last_became_active":"2024-03-24T00:37:02.200151+0000","last_became_peered":"2024-03-24T00:37:02.200151+0000","last_unstale":"2024-03-24T00:37:02.204221+0000","last_undegraded":"2024-03-24T00:37:02.204221+0000","last_fullsized":"2024-03-24T00:37:02.204221+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:39:29.892702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.216647+0000","last_change":"2024-03-24T00:37:02.205873+0000","last_active":"2024-03-24T00:37:02.216647+0000","last_peered":"2024-03-24T00:37:02.216647+0000","last_clean":"2024-03-24T00:37:02.216647+0000","last_became_active":"2024-03-24T00:37:02.205579+0000","last_became_peered":"2024-03-24T00:37:02.205579+0000","last_unstale":"2024-03-24T00:37:02.216647+0000","last_undegraded":"2024-03-24T00:37:02.216647+0000","last_fullsized":"2024-03-24T00:37:02.216647+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:52.195925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215352+0000","last_change":"2024-03-24T00:37:02.208098+0000","last_active":"2024-03-24T00:37:02.215352+0000","last_peered":"2024-03-24T00:37:02.215352+0000","last_clean":"2024-03-24T00:37:02.215352+0000","last_became_active":"2024-03-24T00:37:02.207798+0000","last_became_peered":"2024-03-24T00:37:02.207798+0000","last_unstale":"2024-03-24T00:37:02.215352+0000","last_undegraded":"2024-03-24T00:37:02.215352+0000","last_fullsized":"2024-03-24T00:37:02.215352+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:21:08.391723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201057+0000","last_change":"2024-03-24T00:37:02.203597+0000","last_active":"2024-03-24T00:37:03.201057+0000","last_peered":"2024-03-24T00:37:03.201057+0000","last_clean":"2024-03-24T00:37:03.201057+0000","last_became_active":"2024-03-24T00:37:02.203302+0000","last_became_peered":"2024-03-24T00:37:02.203302+0000","last_unstale":"2024-03-24T00:37:03.201057+0000","last_undegraded":"2024-03-24T00:37:03.201057+0000","last_fullsized":"2024-03-24T00:37:03.201057+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:45:55.182522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200665+0000","last_change":"2024-03-24T00:37:02.200991+0000","last_active":"2024-03-24T00:37:03.200665+0000","last_peered":"2024-03-24T00:37:03.200665+0000","last_clean":"2024-03-24T00:37:03.200665+0000","last_became_active":"2024-03-24T00:37:02.200617+0000","last_became_peered":"2024-03-24T00:37:02.200617+0000","last_unstale":"2024-03-24T00:37:03.200665+0000","last_undegraded":"2024-03-24T00:37:03.200665+0000","last_fullsized":"2024-03-24T00:37:03.200665+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:11:18.916706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204691+0000","last_change":"2024-03-24T00:37:02.217747+0000","last_active":"2024-03-24T00:37:03.204691+0000","last_peered":"2024-03-24T00:37:03.204691+0000","last_clean":"2024-03-24T00:37:03.204691+0000","last_became_active":"2024-03-24T00:37:02.217536+0000","last_became_peered":"2024-03-24T00:37:02.217536+0000","last_unstale":"2024-03-24T00:37:03.204691+0000","last_undegraded":"2024-03-24T00:37:03.204691+0000","last_fullsized":"2024-03-24T00:37:03.204691+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:35:40.631539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201068+0000","last_change":"2024-03-24T00:37:02.203121+0000","last_active":"2024-03-24T00:37:03.201068+0000","last_peered":"2024-03-24T00:37:03.201068+0000","last_clean":"2024-03-24T00:37:03.201068+0000","last_became_active":"2024-03-24T00:37:02.202411+0000","last_became_peered":"2024-03-24T00:37:02.202411+0000","last_unstale":"2024-03-24T00:37:03.201068+0000","last_undegraded":"2024-03-24T00:37:03.201068+0000","last_fullsized":"2024-03-24T00:37:03.201068+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:25.855556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.214651+0000","last_change":"2024-03-24T00:37:02.202560+0000","last_active":"2024-03-24T00:37:02.214651+0000","last_peered":"2024-03-24T00:37:02.214651+0000","last_clean":"2024-03-24T00:37:02.214651+0000","last_became_active":"2024-03-24T00:37:02.199273+0000","last_became_peered":"2024-03-24T00:37:02.199273+0000","last_unstale":"2024-03-24T00:37:02.214651+0000","last_undegraded":"2024-03-24T00:37:02.214651+0000","last_fullsized":"2024-03-24T00:37:02.214651+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:59.531126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201575+0000","last_change":"2024-03-24T00:37:02.195658+0000","last_active":"2024-03-24T00:37:03.201575+0000","last_peered":"2024-03-24T00:37:03.201575+0000","last_clean":"2024-03-24T00:37:03.201575+0000","last_became_active":"2024-03-24T00:37:02.194784+0000","last_became_peered":"2024-03-24T00:37:02.194784+0000","last_unstale":"2024-03-24T00:37:03.201575+0000","last_undegraded":"2024-03-24T00:37:03.201575+0000","last_fullsized":"2024-03-24T00:37:03.201575+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:49.266439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201315+0000","last_change":"2024-03-24T00:37:02.203871+0000","last_active":"2024-03-24T00:37:03.201315+0000","last_peered":"2024-03-24T00:37:03.201315+0000","last_clean":"2024-03-24T00:37:03.201315+0000","last_became_active":"2024-03-24T00:37:02.203584+0000","last_became_peered":"2024-03-24T00:37:02.203584+0000","last_unstale":"2024-03-24T00:37:03.201315+0000","last_undegraded":"2024-03-24T00:37:03.201315+0000","last_fullsized":"2024-03-24T00:37:03.201315+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:30.857697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215649+0000","last_change":"2024-03-24T00:37:02.206089+0000","last_active":"2024-03-24T00:37:02.215649+0000","last_peered":"2024-03-24T00:37:02.215649+0000","last_clean":"2024-03-24T00:37:02.215649+0000","last_became_active":"2024-03-24T00:37:02.205522+0000","last_became_peered":"2024-03-24T00:37:02.205522+0000","last_unstale":"2024-03-24T00:37:02.215649+0000","last_undegraded":"2024-03-24T00:37:02.215649+0000","last_fullsized":"2024-03-24T00:37:02.215649+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:25.855556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198028+0000","last_change":"2024-03-24T00:37:02.212573+0000","last_active":"2024-03-24T00:37:03.198028+0000","last_peered":"2024-03-24T00:37:03.198028+0000","last_clean":"2024-03-24T00:37:03.198028+0000","last_became_active":"2024-03-24T00:37:02.212278+0000","last_became_peered":"2024-03-24T00:37:02.212278+0000","last_unstale":"2024-03-24T00:37:03.198028+0000","last_undegraded":"2024-03-24T00:37:03.198028+0000","last_fullsized":"2024-03-24T00:37:03.198028+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:48:01.809774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.210582+0000","last_change":"2024-03-24T00:37:02.202335+0000","last_active":"2024-03-24T00:37:02.210582+0000","last_peered":"2024-03-24T00:37:02.210582+0000","last_clean":"2024-03-24T00:37:02.210582+0000","last_became_active":"2024-03-24T00:37:02.202007+0000","last_became_peered":"2024-03-24T00:37:02.202007+0000","last_unstale":"2024-03-24T00:37:02.210582+0000","last_undegraded":"2024-03-24T00:37:02.210582+0000","last_fullsized":"2024-03-24T00:37:02.210582+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:21:08.391723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200345+0000","last_change":"2024-03-24T00:37:02.194567+0000","last_active":"2024-03-24T00:37:03.200345+0000","last_peered":"2024-03-24T00:37:03.200345+0000","last_clean":"2024-03-24T00:37:03.200345+0000","last_became_active":"2024-03-24T00:37:02.194267+0000","last_became_peered":"2024-03-24T00:37:02.194267+0000","last_unstale":"2024-03-24T00:37:03.200345+0000","last_undegraded":"2024-03-24T00:37:03.200345+0000","last_fullsized":"2024-03-24T00:37:03.200345+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:37.053363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205224+0000","last_change":"2024-03-24T00:37:02.202700+0000","last_active":"2024-03-24T00:37:03.205224+0000","last_peered":"2024-03-24T00:37:03.205224+0000","last_clean":"2024-03-24T00:37:03.205224+0000","last_became_active":"2024-03-24T00:37:02.202405+0000","last_became_peered":"2024-03-24T00:37:02.202405+0000","last_unstale":"2024-03-24T00:37:03.205224+0000","last_undegraded":"2024-03-24T00:37:03.205224+0000","last_fullsized":"2024-03-24T00:37:03.205224+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:44.003595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215802+0000","last_change":"2024-03-24T00:37:02.206690+0000","last_active":"2024-03-24T00:37:02.215802+0000","last_peered":"2024-03-24T00:37:02.215802+0000","last_clean":"2024-03-24T00:37:02.215802+0000","last_became_active":"2024-03-24T00:37:02.206024+0000","last_became_peered":"2024-03-24T00:37:02.206024+0000","last_unstale":"2024-03-24T00:37:02.215802+0000","last_undegraded":"2024-03-24T00:37:02.215802+0000","last_fullsized":"2024-03-24T00:37:02.215802+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:34.773035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198084+0000","last_change":"2024-03-24T00:37:02.196609+0000","last_active":"2024-03-24T00:37:03.198084+0000","last_peered":"2024-03-24T00:37:03.198084+0000","last_clean":"2024-03-24T00:37:03.198084+0000","last_became_active":"2024-03-24T00:37:02.196104+0000","last_became_peered":"2024-03-24T00:37:02.196104+0000","last_unstale":"2024-03-24T00:37:03.198084+0000","last_undegraded":"2024-03-24T00:37:03.198084+0000","last_fullsized":"2024-03-24T00:37:03.198084+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:09:39.179840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201463+0000","last_change":"2024-03-24T00:37:02.198834+0000","last_active":"2024-03-24T00:37:03.201463+0000","last_peered":"2024-03-24T00:37:03.201463+0000","last_clean":"2024-03-24T00:37:03.201463+0000","last_became_active":"2024-03-24T00:37:02.197500+0000","last_became_peered":"2024-03-24T00:37:02.197500+0000","last_unstale":"2024-03-24T00:37:03.201463+0000","last_undegraded":"2024-03-24T00:37:03.201463+0000","last_fullsized":"2024-03-24T00:37:03.201463+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:30.972900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198226+0000","last_change":"2024-03-24T00:37:02.200810+0000","last_active":"2024-03-24T00:37:03.198226+0000","last_peered":"2024-03-24T00:37:03.198226+0000","last_clean":"2024-03-24T00:37:03.198226+0000","last_became_active":"2024-03-24T00:37:02.200482+0000","last_became_peered":"2024-03-24T00:37:02.200482+0000","last_unstale":"2024-03-24T00:37:03.198226+0000","last_undegraded":"2024-03-24T00:37:03.198226+0000","last_fullsized":"2024-03-24T00:37:03.198226+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:58.820942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204695+0000","last_change":"2024-03-24T00:37:02.205543+0000","last_active":"2024-03-24T00:37:03.204695+0000","last_peered":"2024-03-24T00:37:03.204695+0000","last_clean":"2024-03-24T00:37:03.204695+0000","last_became_active":"2024-03-24T00:37:02.205262+0000","last_became_peered":"2024-03-24T00:37:02.205262+0000","last_unstale":"2024-03-24T00:37:03.204695+0000","last_undegraded":"2024-03-24T00:37:03.204695+0000","last_fullsized":"2024-03-24T00:37:03.204695+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:16:23.877808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204792+0000","last_change":"2024-03-24T00:37:02.202790+0000","last_active":"2024-03-24T00:37:03.204792+0000","last_peered":"2024-03-24T00:37:03.204792+0000","last_clean":"2024-03-24T00:37:03.204792+0000","last_became_active":"2024-03-24T00:37:02.202513+0000","last_became_peered":"2024-03-24T00:37:02.202513+0000","last_unstale":"2024-03-24T00:37:03.204792+0000","last_undegraded":"2024-03-24T00:37:03.204792+0000","last_fullsized":"2024-03-24T00:37:03.204792+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:32:23.886507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200376+0000","last_change":"2024-03-24T00:37:02.192248+0000","last_active":"2024-03-24T00:37:03.200376+0000","last_peered":"2024-03-24T00:37:03.200376+0000","last_clean":"2024-03-24T00:37:03.200376+0000","last_became_active":"2024-03-24T00:37:02.191365+0000","last_became_peered":"2024-03-24T00:37:02.191365+0000","last_unstale":"2024-03-24T00:37:03.200376+0000","last_undegraded":"2024-03-24T00:37:03.200376+0000","last_fullsized":"2024-03-24T00:37:03.200376+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:21.928238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.214671+0000","last_change":"2024-03-24T00:37:02.202749+0000","last_active":"2024-03-24T00:37:02.214671+0000","last_peered":"2024-03-24T00:37:02.214671+0000","last_clean":"2024-03-24T00:37:02.214671+0000","last_became_active":"2024-03-24T00:37:02.202007+0000","last_became_peered":"2024-03-24T00:37:02.202007+0000","last_unstale":"2024-03-24T00:37:02.214671+0000","last_undegraded":"2024-03-24T00:37:02.214671+0000","last_fullsized":"2024-03-24T00:37:02.214671+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:21.852373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205142+0000","last_change":"2024-03-24T00:37:02.200452+0000","last_active":"2024-03-24T00:37:03.205142+0000","last_peered":"2024-03-24T00:37:03.205142+0000","last_clean":"2024-03-24T00:37:03.205142+0000","last_became_active":"2024-03-24T00:37:02.200140+0000","last_became_peered":"2024-03-24T00:37:02.200140+0000","last_unstale":"2024-03-24T00:37:03.205142+0000","last_undegraded":"2024-03-24T00:37:03.205142+0000","last_fullsized":"2024-03-24T00:37:03.205142+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:22.264783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201822+0000","last_change":"2024-03-24T00:37:02.195418+0000","last_active":"2024-03-24T00:37:03.201822+0000","last_peered":"2024-03-24T00:37:03.201822+0000","last_clean":"2024-03-24T00:37:03.201822+0000","last_became_active":"2024-03-24T00:37:02.194433+0000","last_became_peered":"2024-03-24T00:37:02.194433+0000","last_unstale":"2024-03-24T00:37:03.201822+0000","last_undegraded":"2024-03-24T00:37:03.201822+0000","last_fullsized":"2024-03-24T00:37:03.201822+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:08:59.489021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215292+0000","last_change":"2024-03-24T00:37:02.207008+0000","last_active":"2024-03-24T00:37:02.215292+0000","last_peered":"2024-03-24T00:37:02.215292+0000","last_clean":"2024-03-24T00:37:02.215292+0000","last_became_active":"2024-03-24T00:37:02.206710+0000","last_became_peered":"2024-03-24T00:37:02.206710+0000","last_unstale":"2024-03-24T00:37:02.215292+0000","last_undegraded":"2024-03-24T00:37:02.215292+0000","last_fullsized":"2024-03-24T00:37:02.215292+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:03:19.520445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204812+0000","last_change":"2024-03-24T00:37:02.202368+0000","last_active":"2024-03-24T00:37:03.204812+0000","last_peered":"2024-03-24T00:37:03.204812+0000","last_clean":"2024-03-24T00:37:03.204812+0000","last_became_active":"2024-03-24T00:37:02.202074+0000","last_became_peered":"2024-03-24T00:37:02.202074+0000","last_unstale":"2024-03-24T00:37:03.204812+0000","last_undegraded":"2024-03-24T00:37:03.204812+0000","last_fullsized":"2024-03-24T00:37:03.204812+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:59:23.304072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.197881+0000","last_change":"2024-03-24T00:37:02.197028+0000","last_active":"2024-03-24T00:37:03.197881+0000","last_peered":"2024-03-24T00:37:03.197881+0000","last_clean":"2024-03-24T00:37:03.197881+0000","last_became_active":"2024-03-24T00:37:02.196596+0000","last_became_peered":"2024-03-24T00:37:02.196596+0000","last_unstale":"2024-03-24T00:37:03.197881+0000","last_undegraded":"2024-03-24T00:37:03.197881+0000","last_fullsized":"2024-03-24T00:37:03.197881+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:34:02.682865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.216371+0000","last_change":"2024-03-24T00:37:02.209550+0000","last_active":"2024-03-24T00:37:02.216371+0000","last_peered":"2024-03-24T00:37:02.216371+0000","last_clean":"2024-03-24T00:37:02.216371+0000","last_became_active":"2024-03-24T00:37:02.209231+0000","last_became_peered":"2024-03-24T00:37:02.209231+0000","last_unstale":"2024-03-24T00:37:02.216371+0000","last_undegraded":"2024-03-24T00:37:02.216371+0000","last_fullsized":"2024-03-24T00:37:02.216371+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:30.857697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.199869+0000","last_change":"2024-03-24T00:37:02.193735+0000","last_active":"2024-03-24T00:37:02.199869+0000","last_peered":"2024-03-24T00:37:02.199869+0000","last_clean":"2024-03-24T00:37:02.199869+0000","last_became_active":"2024-03-24T00:37:02.192927+0000","last_became_peered":"2024-03-24T00:37:02.192927+0000","last_unstale":"2024-03-24T00:37:02.199869+0000","last_undegraded":"2024-03-24T00:37:02.199869+0000","last_fullsized":"2024-03-24T00:37:02.199869+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:12:06.270215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198442+0000","last_change":"2024-03-24T00:37:02.211783+0000","last_active":"2024-03-24T00:37:03.198442+0000","last_peered":"2024-03-24T00:37:03.198442+0000","last_clean":"2024-03-24T00:37:03.198442+0000","last_became_active":"2024-03-24T00:37:02.211523+0000","last_became_peered":"2024-03-24T00:37:02.211523+0000","last_unstale":"2024-03-24T00:37:03.198442+0000","last_undegraded":"2024-03-24T00:37:03.198442+0000","last_fullsized":"2024-03-24T00:37:03.198442+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:33.828330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"94'4","reported_seq":18,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205257+0000","last_change":"2024-03-24T00:37:01.178257+0000","last_active":"2024-03-24T00:37:03.205257+0000","last_peered":"2024-03-24T00:37:03.205257+0000","last_clean":"2024-03-24T00:37:03.205257+0000","last_became_active":"2024-03-24T00:37:01.171859+0000","last_became_peered":"2024-03-24T00:37:01.171859+0000","last_unstale":"2024-03-24T00:37:03.205257+0000","last_undegraded":"2024-03-24T00:37:03.205257+0000","last_fullsized":"2024-03-24T00:37:03.205257+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:13:58.704360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.216405+0000","last_change":"2024-03-24T00:37:02.212348+0000","last_active":"2024-03-24T00:37:02.216405+0000","last_peered":"2024-03-24T00:37:02.216405+0000","last_clean":"2024-03-24T00:37:02.216405+0000","last_became_active":"2024-03-24T00:37:02.212061+0000","last_became_peered":"2024-03-24T00:37:02.212061+0000","last_unstale":"2024-03-24T00:37:02.216405+0000","last_undegraded":"2024-03-24T00:37:02.216405+0000","last_fullsized":"2024-03-24T00:37:02.216405+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:05.510696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.202024+0000","last_change":"2024-03-24T00:37:01.186120+0000","last_active":"2024-03-24T00:37:03.202024+0000","last_peered":"2024-03-24T00:37:03.202024+0000","last_clean":"2024-03-24T00:37:03.202024+0000","last_became_active":"2024-03-24T00:37:01.185711+0000","last_became_peered":"2024-03-24T00:37:01.185711+0000","last_unstale":"2024-03-24T00:37:03.202024+0000","last_undegraded":"2024-03-24T00:37:03.202024+0000","last_fullsized":"2024-03-24T00:37:03.202024+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:21:07.383330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.201266+0000","last_change":"2024-03-24T00:37:02.196249+0000","last_active":"2024-03-24T00:37:02.201266+0000","last_peered":"2024-03-24T00:37:02.201266+0000","last_clean":"2024-03-24T00:37:02.201266+0000","last_became_active":"2024-03-24T00:37:02.195987+0000","last_became_peered":"2024-03-24T00:37:02.195987+0000","last_unstale":"2024-03-24T00:37:02.201266+0000","last_undegraded":"2024-03-24T00:37:02.201266+0000","last_fullsized":"2024-03-24T00:37:02.201266+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:31.890531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204973+0000","last_change":"2024-03-24T00:37:01.182997+0000","last_active":"2024-03-24T00:37:03.204973+0000","last_peered":"2024-03-24T00:37:03.204973+0000","last_clean":"2024-03-24T00:37:03.204973+0000","last_became_active":"2024-03-24T00:37:01.181663+0000","last_became_peered":"2024-03-24T00:37:01.181663+0000","last_unstale":"2024-03-24T00:37:03.204973+0000","last_undegraded":"2024-03-24T00:37:03.204973+0000","last_fullsized":"2024-03-24T00:37:03.204973+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:35:42.237011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.202025+0000","last_change":"2024-03-24T00:37:02.201155+0000","last_active":"2024-03-24T00:37:03.202025+0000","last_peered":"2024-03-24T00:37:03.202025+0000","last_clean":"2024-03-24T00:37:03.202025+0000","last_became_active":"2024-03-24T00:37:02.200808+0000","last_became_peered":"2024-03-24T00:37:02.200808+0000","last_unstale":"2024-03-24T00:37:03.202025+0000","last_undegraded":"2024-03-24T00:37:03.202025+0000","last_fullsized":"2024-03-24T00:37:03.202025+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:05.510696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204902+0000","last_change":"2024-03-24T00:37:01.176718+0000","last_active":"2024-03-24T00:37:03.204902+0000","last_peered":"2024-03-24T00:37:03.204902+0000","last_clean":"2024-03-24T00:37:03.204902+0000","last_became_active":"2024-03-24T00:37:01.176354+0000","last_became_peered":"2024-03-24T00:37:01.176354+0000","last_unstale":"2024-03-24T00:37:03.204902+0000","last_undegraded":"2024-03-24T00:37:03.204902+0000","last_fullsized":"2024-03-24T00:37:03.204902+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:29.964508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.205779+0000","last_change":"2024-03-24T00:37:02.197368+0000","last_active":"2024-03-24T00:37:02.205779+0000","last_peered":"2024-03-24T00:37:02.205779+0000","last_clean":"2024-03-24T00:37:02.205779+0000","last_became_active":"2024-03-24T00:37:02.197017+0000","last_became_peered":"2024-03-24T00:37:02.197017+0000","last_unstale":"2024-03-24T00:37:02.205779+0000","last_undegraded":"2024-03-24T00:37:02.205779+0000","last_fullsized":"2024-03-24T00:37:02.205779+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:27:24.816115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"94'1","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215857+0000","last_change":"2024-03-24T00:37:01.173373+0000","last_active":"2024-03-24T00:37:02.215857+0000","last_peered":"2024-03-24T00:37:02.215857+0000","last_clean":"2024-03-24T00:37:02.215857+0000","last_became_active":"2024-03-24T00:37:01.171443+0000","last_became_peered":"2024-03-24T00:37:01.171443+0000","last_unstale":"2024-03-24T00:37:02.215857+0000","last_undegraded":"2024-03-24T00:37:02.215857+0000","last_fullsized":"2024-03-24T00:37:02.215857+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:44.218310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201395+0000","last_change":"2024-03-24T00:37:02.211801+0000","last_active":"2024-03-24T00:37:03.201395+0000","last_peered":"2024-03-24T00:37:03.201395+0000","last_clean":"2024-03-24T00:37:03.201395+0000","last_became_active":"2024-03-24T00:37:02.211199+0000","last_became_peered":"2024-03-24T00:37:02.211199+0000","last_unstale":"2024-03-24T00:37:03.201395+0000","last_undegraded":"2024-03-24T00:37:03.201395+0000","last_fullsized":"2024-03-24T00:37:03.201395+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:45:45.591503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204698+0000","last_change":"2024-03-24T00:37:01.166426+0000","last_active":"2024-03-24T00:37:03.204698+0000","last_peered":"2024-03-24T00:37:03.204698+0000","last_clean":"2024-03-24T00:37:03.204698+0000","last_became_active":"2024-03-24T00:37:01.166149+0000","last_became_peered":"2024-03-24T00:37:01.166149+0000","last_unstale":"2024-03-24T00:37:03.204698+0000","last_undegraded":"2024-03-24T00:37:03.204698+0000","last_fullsized":"2024-03-24T00:37:03.204698+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:36:24.847163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.211301+0000","last_change":"2024-03-24T00:37:02.203024+0000","last_active":"2024-03-24T00:37:02.211301+0000","last_peered":"2024-03-24T00:37:02.211301+0000","last_clean":"2024-03-24T00:37:02.211301+0000","last_became_active":"2024-03-24T00:37:02.201986+0000","last_became_peered":"2024-03-24T00:37:02.201986+0000","last_unstale":"2024-03-24T00:37:02.211301+0000","last_undegraded":"2024-03-24T00:37:02.211301+0000","last_fullsized":"2024-03-24T00:37:02.211301+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:39.876486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.190357+0000","last_change":"2024-03-24T00:37:01.184761+0000","last_active":"2024-03-24T00:37:02.190357+0000","last_peered":"2024-03-24T00:37:02.190357+0000","last_clean":"2024-03-24T00:37:02.190357+0000","last_became_active":"2024-03-24T00:37:01.184391+0000","last_became_peered":"2024-03-24T00:37:01.184391+0000","last_unstale":"2024-03-24T00:37:02.190357+0000","last_undegraded":"2024-03-24T00:37:02.190357+0000","last_fullsized":"2024-03-24T00:37:02.190357+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:41.286975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200954+0000","last_change":"2024-03-24T00:37:02.201265+0000","last_active":"2024-03-24T00:37:03.200954+0000","last_peered":"2024-03-24T00:37:03.200954+0000","last_clean":"2024-03-24T00:37:03.200954+0000","last_became_active":"2024-03-24T00:37:02.200932+0000","last_became_peered":"2024-03-24T00:37:02.200932+0000","last_unstale":"2024-03-24T00:37:03.200954+0000","last_undegraded":"2024-03-24T00:37:03.200954+0000","last_fullsized":"2024-03-24T00:37:03.200954+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:52.195925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.180322+0000","last_change":"2024-03-24T00:37:01.180472+0000","last_active":"2024-03-24T00:37:02.180322+0000","last_peered":"2024-03-24T00:37:02.180322+0000","last_clean":"2024-03-24T00:37:02.180322+0000","last_became_active":"2024-03-24T00:37:01.180162+0000","last_became_peered":"2024-03-24T00:37:01.180162+0000","last_unstale":"2024-03-24T00:37:02.180322+0000","last_undegraded":"2024-03-24T00:37:02.180322+0000","last_fullsized":"2024-03-24T00:37:02.180322+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:35:32.819938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205286+0000","last_change":"2024-03-24T00:37:02.203852+0000","last_active":"2024-03-24T00:37:03.205286+0000","last_peered":"2024-03-24T00:37:03.205286+0000","last_clean":"2024-03-24T00:37:03.205286+0000","last_became_active":"2024-03-24T00:37:02.203149+0000","last_became_peered":"2024-03-24T00:37:02.203149+0000","last_unstale":"2024-03-24T00:37:03.205286+0000","last_undegraded":"2024-03-24T00:37:03.205286+0000","last_fullsized":"2024-03-24T00:37:03.205286+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:20.353725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.180885+0000","last_change":"2024-03-24T00:37:01.172991+0000","last_active":"2024-03-24T00:37:02.180885+0000","last_peered":"2024-03-24T00:37:02.180885+0000","last_clean":"2024-03-24T00:37:02.180885+0000","last_became_active":"2024-03-24T00:37:01.172687+0000","last_became_peered":"2024-03-24T00:37:01.172687+0000","last_unstale":"2024-03-24T00:37:02.180885+0000","last_undegraded":"2024-03-24T00:37:02.180885+0000","last_fullsized":"2024-03-24T00:37:02.180885+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:53:55.041007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.202368+0000","last_change":"2024-03-24T00:37:02.208179+0000","last_active":"2024-03-24T00:37:03.202368+0000","last_peered":"2024-03-24T00:37:03.202368+0000","last_clean":"2024-03-24T00:37:03.202368+0000","last_became_active":"2024-03-24T00:37:02.207854+0000","last_became_peered":"2024-03-24T00:37:02.207854+0000","last_unstale":"2024-03-24T00:37:03.202368+0000","last_undegraded":"2024-03-24T00:37:03.202368+0000","last_fullsized":"2024-03-24T00:37:03.202368+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:15:16.935643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.191682+0000","last_change":"2024-03-24T00:37:01.187103+0000","last_active":"2024-03-24T00:37:02.191682+0000","last_peered":"2024-03-24T00:37:02.191682+0000","last_clean":"2024-03-24T00:37:02.191682+0000","last_became_active":"2024-03-24T00:37:01.186828+0000","last_became_peered":"2024-03-24T00:37:01.186828+0000","last_unstale":"2024-03-24T00:37:02.191682+0000","last_undegraded":"2024-03-24T00:37:02.191682+0000","last_fullsized":"2024-03-24T00:37:02.191682+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:03:18.512052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.212458+0000","last_change":"2024-03-24T00:37:02.198498+0000","last_active":"2024-03-24T00:37:02.212458+0000","last_peered":"2024-03-24T00:37:02.212458+0000","last_clean":"2024-03-24T00:37:02.212458+0000","last_became_active":"2024-03-24T00:37:02.198194+0000","last_became_peered":"2024-03-24T00:37:02.198194+0000","last_unstale":"2024-03-24T00:37:02.212458+0000","last_undegraded":"2024-03-24T00:37:02.212458+0000","last_fullsized":"2024-03-24T00:37:02.212458+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:00:43.825403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"94'3","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201984+0000","last_change":"2024-03-24T00:37:01.180675+0000","last_active":"2024-03-24T00:37:03.201984+0000","last_peered":"2024-03-24T00:37:03.201984+0000","last_clean":"2024-03-24T00:37:03.201984+0000","last_became_active":"2024-03-24T00:37:01.180346+0000","last_became_peered":"2024-03-24T00:37:01.180346+0000","last_unstale":"2024-03-24T00:37:03.201984+0000","last_undegraded":"2024-03-24T00:37:03.201984+0000","last_fullsized":"2024-03-24T00:37:03.201984+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:38.868093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201860+0000","last_change":"2024-03-24T00:37:02.198375+0000","last_active":"2024-03-24T00:37:03.201860+0000","last_peered":"2024-03-24T00:37:03.201860+0000","last_clean":"2024-03-24T00:37:03.201860+0000","last_became_active":"2024-03-24T00:37:02.198080+0000","last_became_peered":"2024-03-24T00:37:02.198080+0000","last_unstale":"2024-03-24T00:37:03.201860+0000","last_undegraded":"2024-03-24T00:37:03.201860+0000","last_fullsized":"2024-03-24T00:37:03.201860+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:30:56.839768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"94'6","reported_seq":20,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201563+0000","last_change":"2024-03-24T00:37:01.183010+0000","last_active":"2024-03-24T00:37:03.201563+0000","last_peered":"2024-03-24T00:37:03.201563+0000","last_clean":"2024-03-24T00:37:03.201563+0000","last_became_active":"2024-03-24T00:37:01.182663+0000","last_became_peered":"2024-03-24T00:37:01.182663+0000","last_unstale":"2024-03-24T00:37:03.201563+0000","last_undegraded":"2024-03-24T00:37:03.201563+0000","last_fullsized":"2024-03-24T00:37:03.201563+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:22:27.182497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.216407+0000","last_change":"2024-03-24T00:37:02.210695+0000","last_active":"2024-03-24T00:37:02.216407+0000","last_peered":"2024-03-24T00:37:02.216407+0000","last_clean":"2024-03-24T00:37:02.216407+0000","last_became_active":"2024-03-24T00:37:02.210387+0000","last_became_peered":"2024-03-24T00:37:02.210387+0000","last_unstale":"2024-03-24T00:37:02.216407+0000","last_undegraded":"2024-03-24T00:37:02.216407+0000","last_fullsized":"2024-03-24T00:37:02.216407+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:11:30.972900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198584+0000","last_change":"2024-03-24T00:37:01.175110+0000","last_active":"2024-03-24T00:37:03.198584+0000","last_peered":"2024-03-24T00:37:03.198584+0000","last_clean":"2024-03-24T00:37:03.198584+0000","last_became_active":"2024-03-24T00:37:01.174831+0000","last_became_peered":"2024-03-24T00:37:01.174831+0000","last_unstale":"2024-03-24T00:37:03.198584+0000","last_undegraded":"2024-03-24T00:37:03.198584+0000","last_fullsized":"2024-03-24T00:37:03.198584+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:38:35.985589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.197916+0000","last_change":"2024-03-24T00:37:02.192522+0000","last_active":"2024-03-24T00:37:02.197916+0000","last_peered":"2024-03-24T00:37:02.197916+0000","last_clean":"2024-03-24T00:37:02.197916+0000","last_became_active":"2024-03-24T00:37:02.192133+0000","last_became_peered":"2024-03-24T00:37:02.192133+0000","last_unstale":"2024-03-24T00:37:02.197916+0000","last_undegraded":"2024-03-24T00:37:02.197916+0000","last_fullsized":"2024-03-24T00:37:02.197916+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:01:39.814622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198283+0000","last_change":"2024-03-24T00:37:01.183666+0000","last_active":"2024-03-24T00:37:03.198283+0000","last_peered":"2024-03-24T00:37:03.198283+0000","last_clean":"2024-03-24T00:37:03.198283+0000","last_became_active":"2024-03-24T00:37:01.182569+0000","last_became_peered":"2024-03-24T00:37:01.182569+0000","last_unstale":"2024-03-24T00:37:03.198283+0000","last_undegraded":"2024-03-24T00:37:03.198283+0000","last_fullsized":"2024-03-24T00:37:03.198283+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:07.560652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200342+0000","last_change":"2024-03-24T00:37:02.208886+0000","last_active":"2024-03-24T00:37:03.200342+0000","last_peered":"2024-03-24T00:37:03.200342+0000","last_clean":"2024-03-24T00:37:03.200342+0000","last_became_active":"2024-03-24T00:37:02.208613+0000","last_became_peered":"2024-03-24T00:37:02.208613+0000","last_unstale":"2024-03-24T00:37:03.200342+0000","last_undegraded":"2024-03-24T00:37:03.200342+0000","last_fullsized":"2024-03-24T00:37:03.200342+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:39.876486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200918+0000","last_change":"2024-03-24T00:37:01.174927+0000","last_active":"2024-03-24T00:37:03.200918+0000","last_peered":"2024-03-24T00:37:03.200918+0000","last_clean":"2024-03-24T00:37:03.200918+0000","last_became_active":"2024-03-24T00:37:01.174315+0000","last_became_peered":"2024-03-24T00:37:01.174315+0000","last_unstale":"2024-03-24T00:37:03.200918+0000","last_undegraded":"2024-03-24T00:37:03.200918+0000","last_fullsized":"2024-03-24T00:37:03.200918+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:03:41.286975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.197890+0000","last_change":"2024-03-24T00:37:02.207182+0000","last_active":"2024-03-24T00:37:03.197890+0000","last_peered":"2024-03-24T00:37:03.197890+0000","last_clean":"2024-03-24T00:37:03.197890+0000","last_became_active":"2024-03-24T00:37:02.206278+0000","last_became_peered":"2024-03-24T00:37:02.206278+0000","last_unstale":"2024-03-24T00:37:03.197890+0000","last_undegraded":"2024-03-24T00:37:03.197890+0000","last_fullsized":"2024-03-24T00:37:03.197890+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:09:32.007020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"94'8","reported_seq":20,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.220974+0000","last_change":"2024-03-24T00:37:01.179650+0000","last_active":"2024-03-24T00:37:02.220974+0000","last_peered":"2024-03-24T00:37:02.220974+0000","last_clean":"2024-03-24T00:37:02.220974+0000","last_became_active":"2024-03-24T00:37:01.177991+0000","last_became_peered":"2024-03-24T00:37:01.177991+0000","last_unstale":"2024-03-24T00:37:02.220974+0000","last_undegraded":"2024-03-24T00:37:02.220974+0000","last_fullsized":"2024-03-24T00:37:02.220974+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:52:17.608291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.214697+0000","last_change":"2024-03-24T00:37:02.202987+0000","last_active":"2024-03-24T00:37:02.214697+0000","last_peered":"2024-03-24T00:37:02.214697+0000","last_clean":"2024-03-24T00:37:02.214697+0000","last_became_active":"2024-03-24T00:37:02.202454+0000","last_became_peered":"2024-03-24T00:37:02.202454+0000","last_unstale":"2024-03-24T00:37:02.214697+0000","last_undegraded":"2024-03-24T00:37:02.214697+0000","last_fullsized":"2024-03-24T00:37:02.214697+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:59.736598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.202286+0000","last_change":"2024-03-24T00:37:01.182857+0000","last_active":"2024-03-24T00:37:03.202286+0000","last_peered":"2024-03-24T00:37:03.202286+0000","last_clean":"2024-03-24T00:37:03.202286+0000","last_became_active":"2024-03-24T00:37:01.182545+0000","last_became_peered":"2024-03-24T00:37:01.182545+0000","last_unstale":"2024-03-24T00:37:03.202286+0000","last_undegraded":"2024-03-24T00:37:03.202286+0000","last_fullsized":"2024-03-24T00:37:03.202286+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:34:57.865809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.212940+0000","last_change":"2024-03-24T00:37:02.209335+0000","last_active":"2024-03-24T00:37:02.212940+0000","last_peered":"2024-03-24T00:37:02.212940+0000","last_clean":"2024-03-24T00:37:02.212940+0000","last_became_active":"2024-03-24T00:37:02.209039+0000","last_became_peered":"2024-03-24T00:37:02.209039+0000","last_unstale":"2024-03-24T00:37:02.212940+0000","last_undegraded":"2024-03-24T00:37:02.212940+0000","last_fullsized":"2024-03-24T00:37:02.212940+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:22:31.502603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"94'1","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215770+0000","last_change":"2024-03-24T00:37:01.173919+0000","last_active":"2024-03-24T00:37:02.215770+0000","last_peered":"2024-03-24T00:37:02.215770+0000","last_clean":"2024-03-24T00:37:02.215770+0000","last_became_active":"2024-03-24T00:37:01.172497+0000","last_became_peered":"2024-03-24T00:37:01.172497+0000","last_unstale":"2024-03-24T00:37:02.215770+0000","last_undegraded":"2024-03-24T00:37:02.215770+0000","last_fullsized":"2024-03-24T00:37:02.215770+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:22:27.182497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201995+0000","last_change":"2024-03-24T00:37:02.195888+0000","last_active":"2024-03-24T00:37:03.201995+0000","last_peered":"2024-03-24T00:37:03.201995+0000","last_clean":"2024-03-24T00:37:03.201995+0000","last_became_active":"2024-03-24T00:37:02.195113+0000","last_became_peered":"2024-03-24T00:37:02.195113+0000","last_unstale":"2024-03-24T00:37:03.201995+0000","last_undegraded":"2024-03-24T00:37:03.201995+0000","last_fullsized":"2024-03-24T00:37:03.201995+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:34:26.165155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204971+0000","last_change":"2024-03-24T00:37:01.165422+0000","last_active":"2024-03-24T00:37:03.204971+0000","last_peered":"2024-03-24T00:37:03.204971+0000","last_clean":"2024-03-24T00:37:03.204971+0000","last_became_active":"2024-03-24T00:37:01.165128+0000","last_became_peered":"2024-03-24T00:37:01.165128+0000","last_unstale":"2024-03-24T00:37:03.204971+0000","last_undegraded":"2024-03-24T00:37:03.204971+0000","last_fullsized":"2024-03-24T00:37:03.204971+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:44:04.502303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198203+0000","last_change":"2024-03-24T00:37:02.205039+0000","last_active":"2024-03-24T00:37:03.198203+0000","last_peered":"2024-03-24T00:37:03.198203+0000","last_clean":"2024-03-24T00:37:03.198203+0000","last_became_active":"2024-03-24T00:37:02.204651+0000","last_became_peered":"2024-03-24T00:37:02.204651+0000","last_unstale":"2024-03-24T00:37:03.198203+0000","last_undegraded":"2024-03-24T00:37:03.198203+0000","last_fullsized":"2024-03-24T00:37:03.198203+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:57.424357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"94'2","reported_seq":16,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204923+0000","last_change":"2024-03-24T00:37:01.176582+0000","last_active":"2024-03-24T00:37:03.204923+0000","last_peered":"2024-03-24T00:37:03.204923+0000","last_clean":"2024-03-24T00:37:03.204923+0000","last_became_active":"2024-03-24T00:37:01.176281+0000","last_became_peered":"2024-03-24T00:37:01.176281+0000","last_unstale":"2024-03-24T00:37:03.204923+0000","last_undegraded":"2024-03-24T00:37:03.204923+0000","last_fullsized":"2024-03-24T00:37:03.204923+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:33.764642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198396+0000","last_change":"2024-03-24T00:37:02.198104+0000","last_active":"2024-03-24T00:37:03.198396+0000","last_peered":"2024-03-24T00:37:03.198396+0000","last_clean":"2024-03-24T00:37:03.198396+0000","last_became_active":"2024-03-24T00:37:02.197784+0000","last_became_peered":"2024-03-24T00:37:02.197784+0000","last_unstale":"2024-03-24T00:37:03.198396+0000","last_undegraded":"2024-03-24T00:37:03.198396+0000","last_fullsized":"2024-03-24T00:37:03.198396+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:45.226702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.186026+0000","last_change":"2024-03-24T00:37:01.175712+0000","last_active":"2024-03-24T00:37:02.186026+0000","last_peered":"2024-03-24T00:37:02.186026+0000","last_clean":"2024-03-24T00:37:02.186026+0000","last_became_active":"2024-03-24T00:37:01.173276+0000","last_became_peered":"2024-03-24T00:37:01.173276+0000","last_unstale":"2024-03-24T00:37:02.186026+0000","last_undegraded":"2024-03-24T00:37:02.186026+0000","last_fullsized":"2024-03-24T00:37:02.186026+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:10:18.273637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.197861+0000","last_change":"2024-03-24T00:37:02.195524+0000","last_active":"2024-03-24T00:37:03.197861+0000","last_peered":"2024-03-24T00:37:03.197861+0000","last_clean":"2024-03-24T00:37:03.197861+0000","last_became_active":"2024-03-24T00:37:02.195222+0000","last_became_peered":"2024-03-24T00:37:02.195222+0000","last_unstale":"2024-03-24T00:37:03.197861+0000","last_undegraded":"2024-03-24T00:37:03.197861+0000","last_fullsized":"2024-03-24T00:37:03.197861+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:37:16.942624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.185124+0000","last_change":"2024-03-24T00:37:01.184432+0000","last_active":"2024-03-24T00:37:02.185124+0000","last_peered":"2024-03-24T00:37:02.185124+0000","last_clean":"2024-03-24T00:37:02.185124+0000","last_became_active":"2024-03-24T00:37:01.184079+0000","last_became_peered":"2024-03-24T00:37:01.184079+0000","last_unstale":"2024-03-24T00:37:02.185124+0000","last_undegraded":"2024-03-24T00:37:02.185124+0000","last_fullsized":"2024-03-24T00:37:02.185124+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:56:38.827292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.214609+0000","last_change":"2024-03-24T00:37:02.199703+0000","last_active":"2024-03-24T00:37:02.214609+0000","last_peered":"2024-03-24T00:37:02.214609+0000","last_clean":"2024-03-24T00:37:02.214609+0000","last_became_active":"2024-03-24T00:37:02.197761+0000","last_became_peered":"2024-03-24T00:37:02.197761+0000","last_unstale":"2024-03-24T00:37:02.214609+0000","last_undegraded":"2024-03-24T00:37:02.214609+0000","last_fullsized":"2024-03-24T00:37:02.214609+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:54:59.045039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204648+0000","last_change":"2024-03-24T00:37:01.166659+0000","last_active":"2024-03-24T00:37:03.204648+0000","last_peered":"2024-03-24T00:37:03.204648+0000","last_clean":"2024-03-24T00:37:03.204648+0000","last_became_active":"2024-03-24T00:37:01.165603+0000","last_became_peered":"2024-03-24T00:37:01.165603+0000","last_unstale":"2024-03-24T00:37:03.204648+0000","last_undegraded":"2024-03-24T00:37:03.204648+0000","last_fullsized":"2024-03-24T00:37:03.204648+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:01:51.187532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205303+0000","last_change":"2024-03-24T00:37:02.204072+0000","last_active":"2024-03-24T00:37:03.205303+0000","last_peered":"2024-03-24T00:37:03.205303+0000","last_clean":"2024-03-24T00:37:03.205303+0000","last_became_active":"2024-03-24T00:37:02.203597+0000","last_became_peered":"2024-03-24T00:37:02.203597+0000","last_unstale":"2024-03-24T00:37:03.205303+0000","last_undegraded":"2024-03-24T00:37:03.205303+0000","last_fullsized":"2024-03-24T00:37:03.205303+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:11:16.914566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"94'2","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.222059+0000","last_change":"2024-03-24T00:37:01.175623+0000","last_active":"2024-03-24T00:37:02.222059+0000","last_peered":"2024-03-24T00:37:02.222059+0000","last_clean":"2024-03-24T00:37:02.222059+0000","last_became_active":"2024-03-24T00:37:01.175183+0000","last_became_peered":"2024-03-24T00:37:01.175183+0000","last_unstale":"2024-03-24T00:37:02.222059+0000","last_undegraded":"2024-03-24T00:37:02.222059+0000","last_fullsized":"2024-03-24T00:37:02.222059+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:45:05.549402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.201391+0000","last_change":"2024-03-24T00:37:02.197487+0000","last_active":"2024-03-24T00:37:02.201391+0000","last_peered":"2024-03-24T00:37:02.201391+0000","last_clean":"2024-03-24T00:37:02.201391+0000","last_became_active":"2024-03-24T00:37:02.193759+0000","last_became_peered":"2024-03-24T00:37:02.193759+0000","last_unstale":"2024-03-24T00:37:02.201391+0000","last_undegraded":"2024-03-24T00:37:02.201391+0000","last_fullsized":"2024-03-24T00:37:02.201391+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:25:47.230816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"95'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.206584+0000","last_change":"2024-03-24T00:37:01.183457+0000","last_active":"2024-03-24T00:37:03.206584+0000","last_peered":"2024-03-24T00:37:03.206584+0000","last_clean":"2024-03-24T00:37:03.206584+0000","last_became_active":"2024-03-24T00:37:01.181903+0000","last_became_peered":"2024-03-24T00:37:01.181903+0000","last_unstale":"2024-03-24T00:37:03.206584+0000","last_undegraded":"2024-03-24T00:37:03.206584+0000","last_fullsized":"2024-03-24T00:37:03.206584+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:03:12.503349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.207972+0000","last_change":"2024-03-24T00:37:02.207184+0000","last_active":"2024-03-24T00:37:02.207972+0000","last_peered":"2024-03-24T00:37:02.207972+0000","last_clean":"2024-03-24T00:37:02.207972+0000","last_became_active":"2024-03-24T00:37:02.206753+0000","last_became_peered":"2024-03-24T00:37:02.206753+0000","last_unstale":"2024-03-24T00:37:02.207972+0000","last_undegraded":"2024-03-24T00:37:02.207972+0000","last_fullsized":"2024-03-24T00:37:02.207972+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:57.728969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.205064+0000","last_change":"2024-03-24T00:37:01.174263+0000","last_active":"2024-03-24T00:37:03.205064+0000","last_peered":"2024-03-24T00:37:03.205064+0000","last_clean":"2024-03-24T00:37:03.205064+0000","last_became_active":"2024-03-24T00:37:01.173928+0000","last_became_peered":"2024-03-24T00:37:01.173928+0000","last_unstale":"2024-03-24T00:37:03.205064+0000","last_undegraded":"2024-03-24T00:37:03.205064+0000","last_fullsized":"2024-03-24T00:37:03.205064+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:15:15.927250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.215636+0000","last_change":"2024-03-24T00:37:02.199921+0000","last_active":"2024-03-24T00:37:02.215636+0000","last_peered":"2024-03-24T00:37:02.215636+0000","last_clean":"2024-03-24T00:37:02.215636+0000","last_became_active":"2024-03-24T00:37:02.199538+0000","last_became_peered":"2024-03-24T00:37:02.199538+0000","last_unstale":"2024-03-24T00:37:02.215636+0000","last_undegraded":"2024-03-24T00:37:02.215636+0000","last_fullsized":"2024-03-24T00:37:02.215636+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:08:39.876486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.191939+0000","last_change":"2024-03-24T00:37:01.175025+0000","last_active":"2024-03-24T00:37:02.191939+0000","last_peered":"2024-03-24T00:37:02.191939+0000","last_clean":"2024-03-24T00:37:02.191939+0000","last_became_active":"2024-03-24T00:37:01.174249+0000","last_became_peered":"2024-03-24T00:37:01.174249+0000","last_unstale":"2024-03-24T00:37:02.191939+0000","last_undegraded":"2024-03-24T00:37:02.191939+0000","last_fullsized":"2024-03-24T00:37:02.191939+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:34:57.865809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198184+0000","last_change":"2024-03-24T00:37:02.201036+0000","last_active":"2024-03-24T00:37:03.198184+0000","last_peered":"2024-03-24T00:37:03.198184+0000","last_clean":"2024-03-24T00:37:03.198184+0000","last_became_active":"2024-03-24T00:37:02.200733+0000","last_became_peered":"2024-03-24T00:37:02.200733+0000","last_unstale":"2024-03-24T00:37:03.198184+0000","last_undegraded":"2024-03-24T00:37:03.198184+0000","last_fullsized":"2024-03-24T00:37:03.198184+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:04:01.844409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.204945+0000","last_change":"2024-03-24T00:37:01.181936+0000","last_active":"2024-03-24T00:37:03.204945+0000","last_peered":"2024-03-24T00:37:03.204945+0000","last_clean":"2024-03-24T00:37:03.204945+0000","last_became_active":"2024-03-24T00:37:01.181622+0000","last_became_peered":"2024-03-24T00:37:01.181622+0000","last_unstale":"2024-03-24T00:37:03.204945+0000","last_undegraded":"2024-03-24T00:37:03.204945+0000","last_fullsized":"2024-03-24T00:37:03.204945+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:26:29.849304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.201143+0000","last_change":"2024-03-24T00:37:02.212383+0000","last_active":"2024-03-24T00:37:03.201143+0000","last_peered":"2024-03-24T00:37:03.201143+0000","last_clean":"2024-03-24T00:37:03.201143+0000","last_became_active":"2024-03-24T00:37:02.212077+0000","last_became_peered":"2024-03-24T00:37:02.212077+0000","last_unstale":"2024-03-24T00:37:03.201143+0000","last_undegraded":"2024-03-24T00:37:03.201143+0000","last_fullsized":"2024-03-24T00:37:03.201143+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:25:34.773035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.190293+0000","last_change":"2024-03-24T00:37:01.179834+0000","last_active":"2024-03-24T00:37:02.190293+0000","last_peered":"2024-03-24T00:37:02.190293+0000","last_clean":"2024-03-24T00:37:02.190293+0000","last_became_active":"2024-03-24T00:37:01.179519+0000","last_became_peered":"2024-03-24T00:37:01.179519+0000","last_unstale":"2024-03-24T00:37:02.190293+0000","last_undegraded":"2024-03-24T00:37:02.190293+0000","last_fullsized":"2024-03-24T00:37:02.190293+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:14:37.043174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.197861+0000","last_change":"2024-03-24T00:37:02.194387+0000","last_active":"2024-03-24T00:37:03.197861+0000","last_peered":"2024-03-24T00:37:03.197861+0000","last_clean":"2024-03-24T00:37:03.197861+0000","last_became_active":"2024-03-24T00:37:02.194122+0000","last_became_peered":"2024-03-24T00:37:02.194122+0000","last_unstale":"2024-03-24T00:37:03.197861+0000","last_undegraded":"2024-03-24T00:37:03.197861+0000","last_fullsized":"2024-03-24T00:37:03.197861+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:57:21.975505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.191038+0000","last_change":"2024-03-24T00:37:01.181575+0000","last_active":"2024-03-24T00:37:02.191038+0000","last_peered":"2024-03-24T00:37:02.191038+0000","last_clean":"2024-03-24T00:37:02.191038+0000","last_became_active":"2024-03-24T00:37:01.181148+0000","last_became_peered":"2024-03-24T00:37:01.181148+0000","last_unstale":"2024-03-24T00:37:02.191038+0000","last_undegraded":"2024-03-24T00:37:02.191038+0000","last_fullsized":"2024-03-24T00:37:02.191038+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:54.794241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-24T00:37:02.214997+0000","last_change":"2024-03-24T00:37:02.203117+0000","last_active":"2024-03-24T00:37:02.214997+0000","last_peered":"2024-03-24T00:37:02.214997+0000","last_clean":"2024-03-24T00:37:02.214997+0000","last_became_active":"2024-03-24T00:37:02.200867+0000","last_became_peered":"2024-03-24T00:37:02.200867+0000","last_unstale":"2024-03-24T00:37:02.214997+0000","last_undegraded":"2024-03-24T00:37:02.214997+0000","last_fullsized":"2024-03-24T00:37:02.214997+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:06:55.802633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":14,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.197987+0000","last_change":"2024-03-24T00:37:01.176391+0000","last_active":"2024-03-24T00:37:03.197987+0000","last_peered":"2024-03-24T00:37:03.197987+0000","last_clean":"2024-03-24T00:37:03.197987+0000","last_became_active":"2024-03-24T00:37:01.176117+0000","last_became_peered":"2024-03-24T00:37:01.176117+0000","last_unstale":"2024-03-24T00:37:03.197987+0000","last_undegraded":"2024-03-24T00:37:03.197987+0000","last_fullsized":"2024-03-24T00:37:03.197987+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:56:47.797989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.198330+0000","last_change":"2024-03-24T00:37:02.209726+0000","last_active":"2024-03-24T00:37:03.198330+0000","last_peered":"2024-03-24T00:37:03.198330+0000","last_clean":"2024-03-24T00:37:03.198330+0000","last_became_active":"2024-03-24T00:37:02.207157+0000","last_became_peered":"2024-03-24T00:37:02.207157+0000","last_unstale":"2024-03-24T00:37:03.198330+0000","last_undegraded":"2024-03-24T00:37:03.198330+0000","last_fullsized":"2024-03-24T00:37:03.198330+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:01.152850+0000","last_clean_scrub_stamp":"2024-03-24T00:37:01.152850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:36:09.807975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"94'1","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.202195+0000","last_change":"2024-03-24T00:37:01.185186+0000","last_active":"2024-03-24T00:37:03.202195+0000","last_peered":"2024-03-24T00:37:03.202195+0000","last_clean":"2024-03-24T00:37:03.202195+0000","last_became_active":"2024-03-24T00:37:01.184837+0000","last_became_peered":"2024-03-24T00:37:01.184837+0000","last_unstale":"2024-03-24T00:37:03.202195+0000","last_undegraded":"2024-03-24T00:37:03.202195+0000","last_fullsized":"2024-03-24T00:37:03.202195+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:37:00.144457+0000","last_clean_scrub_stamp":"2024-03-24T00:37:00.144457+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:03:18.512052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":322,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-03-24T00:37:03.200439+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:37:03.200439+0000","last_peered":"2024-03-24T00:37:03.200439+0000","last_clean":"2024-03-24T00:37:03.200439+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:37:03.200439+0000","last_undegraded":"2024-03-24T00:37:03.200439+0000","last_fullsized":"2024-03-24T00:37:03.200439+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574924,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37068,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334772,"statfs":{"total":96636764160,"available":96598806528,"internally_reserved":0,"allocated":1765376,"data_stored":1095256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574925,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36488,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335352,"statfs":{"total":96636764160,"available":96599400448,"internally_reserved":0,"allocated":1171456,"data_stored":505428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574925,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36480,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335360,"statfs":{"total":96636764160,"available":96599408640,"internally_reserved":0,"allocated":1167360,"data_stored":504846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40668,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331172,"statfs":{"total":96636764160,"available":96595120128,"internally_reserved":0,"allocated":1196032,"data_stored":512812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":1179648,"data_stored":506410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36484,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335356,"statfs":{"total":96636764160,"available":96599404544,"internally_reserved":0,"allocated":1167360,"data_stored":504810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36492,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335348,"statfs":{"total":96636764160,"available":96599396352,"internally_reserved":0,"allocated":1163264,"data_stored":504776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574924,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37076,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334764,"statfs":{"total":96636764160,"available":96598798336,"internally_reserved":0,"allocated":1761280,"data_stored":1095726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:37:06.697 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:37:06.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:37:06.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.877+0000 7ff139c00640 1 Processor -- start 2024-03-24T00:37:06.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.877+0000 7ff139c00640 1 -- start start 2024-03-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.881+0000 7ff139c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff13415ffd0 0x7ff1340ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.881+0000 7ff139c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff134071de0 con 0x7ff1340728d0 2024-03-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.881+0000 7ff139c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1340cea50 con 0x7ff13415bd00 2024-03-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.881+0000 7ff139c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1340ced30 con 0x7ff13415ffd0 2024-03-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.881+0000 7ff133400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff1340728d0 0x7ff13406cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:44856/0 (socket says 172.21.15.40:44856) 2024-03-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.881+0000 7ff133400640 1 -- 172.21.15.40:0/3941309560 learned_addr learned my addr 172.21.15.40:0/3941309560 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:06.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.881+0000 7ff12a000640 1 --2- 172.21.15.40:0/3941309560 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff13415ffd0 0x7ff1340ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.881+0000 7ff12a000640 1 -- 172.21.15.40:0/3941309560 >> v1:172.21.15.53:6789/0 conn(0x7ff13415bd00 legacy=0x7ff13415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.881+0000 7ff12a000640 1 -- 172.21.15.40:0/3941309560 >> v1:172.21.15.40:6789/0 conn(0x7ff1340728d0 legacy=0x7ff13406cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.881+0000 7ff132000640 1 -- 172.21.15.40:0/3941309560 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162894392 0 0) 0x7ff134071de0 con 0x7ff1340728d0 2024-03-24T00:37:06.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.881+0000 7ff12a000640 1 -- 172.21.15.40:0/3941309560 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff134071b00 con 0x7ff13415ffd0 2024-03-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff12a000640 1 --2- 172.21.15.40:0/3941309560 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff13415ffd0 0x7ff1340ae8e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009770 tx=0x7ff11c01c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ed4740f00cf9763 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff132000640 1 -- 172.21.15.40:0/3941309560 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0b0070 con 0x7ff13415ffd0 2024-03-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff132000640 1 -- 172.21.15.40:0/3941309560 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11c0ab600 con 0x7ff13415ffd0 2024-03-24T00:37:06.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff132000640 1 -- 172.21.15.40:0/3941309560 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0ab9d0 con 0x7ff13415ffd0 2024-03-24T00:37:06.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff139c00640 1 -- 172.21.15.40:0/3941309560 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff13415ffd0 msgr2=0x7ff1340ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff139c00640 1 --2- 172.21.15.40:0/3941309560 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff13415ffd0 0x7ff1340ae8e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009770 tx=0x7ff11c01c2d0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff139c00640 1 -- 172.21.15.40:0/3941309560 shutdown_connections 2024-03-24T00:37:06.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff139c00640 1 --2- 172.21.15.40:0/3941309560 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff13415ffd0 0x7ff1340ae8e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff139c00640 1 -- 172.21.15.40:0/3941309560 >> 172.21.15.40:0/3941309560 conn(0x7ff1340b1660 msgr2=0x7ff1340b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:06.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff139c00640 1 -- 172.21.15.40:0/3941309560 shutdown_connections 2024-03-24T00:37:06.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff139c00640 1 -- 172.21.15.40:0/3941309560 wait complete. 2024-03-24T00:37:06.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff139c00640 1 Processor -- start 2024-03-24T00:37:06.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.885+0000 7ff139c00640 1 -- start start 2024-03-24T00:37:06.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff139c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1340728d0 0x7ff134145f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff139c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff13415bd00 0x7ff1341456c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff139c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13415ffd0 0x7ff1340aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff139c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13415c0e0 con 0x7ff13415ffd0 2024-03-24T00:37:06.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff139c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1340cf660 con 0x7ff13415bd00 2024-03-24T00:37:06.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff139c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13416a420 con 0x7ff1340728d0 2024-03-24T00:37:06.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff133400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13415ffd0 0x7ff1340aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff132a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1340728d0 0x7ff134145f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff133400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13415ffd0 0x7ff1340aef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:40108/0 (socket says 172.21.15.40:40108) 2024-03-24T00:37:06.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff133400640 1 -- 172.21.15.40:0/4100237915 learned_addr learned my addr 172.21.15.40:0/4100237915 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:06.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff12a000640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff13415bd00 0x7ff1341456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff133400640 1 -- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1340728d0 msgr2=0x7ff134145f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff133400640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1340728d0 0x7ff134145f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff133400640 1 -- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff13415bd00 msgr2=0x7ff1341456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:06.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff133400640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff13415bd00 0x7ff1341456c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:06.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff133400640 1 -- 172.21.15.40:0/4100237915 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1340b0910 con 0x7ff13415ffd0 2024-03-24T00:37:06.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff12a000640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff13415bd00 0x7ff1341456c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:06.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff133400640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13415ffd0 0x7ff1340aef70 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00c8c0 tx=0x7ff12c00cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:06.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff12b400640 1 -- 172.21.15.40:0/4100237915 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c018070 con 0x7ff13415ffd0 2024-03-24T00:37:06.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff12b400640 1 -- 172.21.15.40:0/4100237915 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12c00a640 con 0x7ff13415ffd0 2024-03-24T00:37:06.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff12b400640 1 -- 172.21.15.40:0/4100237915 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c00aa30 con 0x7ff13415ffd0 2024-03-24T00:37:06.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff139c00640 1 -- 172.21.15.40:0/4100237915 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1340b0bf0 con 0x7ff13415ffd0 2024-03-24T00:37:06.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff139c00640 1 -- 172.21.15.40:0/4100237915 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13414dc80 con 0x7ff13415ffd0 2024-03-24T00:37:06.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff139c00640 1 -- 172.21.15.40:0/4100237915 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff134068170 con 0x7ff13415ffd0 2024-03-24T00:37:06.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.889+0000 7ff12b400640 1 -- 172.21.15.40:0/4100237915 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff12c00abd0 con 0x7ff13415ffd0 2024-03-24T00:37:06.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.893+0000 7ff12b400640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff110067660 0x7ff110069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:06.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.893+0000 7ff12b400640 1 -- 172.21.15.40:0/4100237915 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff12c084e30 con 0x7ff13415ffd0 2024-03-24T00:37:06.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.893+0000 7ff132a00640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff110067660 0x7ff110069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:06.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.893+0000 7ff132a00640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff110067660 0x7ff110069b20 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff124007fd0 tx=0x7ff124009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:06.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:06.897+0000 7ff12b400640 1 -- 172.21.15.40:0/4100237915 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff12c055450 con 0x7ff13415ffd0 2024-03-24T00:37:07.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.117+0000 7ff139c00640 1 -- 172.21.15.40:0/4100237915 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff13406e870 con 0x7ff13415ffd0 2024-03-24T00:37:07.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.121+0000 7ff12b400640 1 -- 172.21.15.40:0/4100237915 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff12c0594f0 con 0x7ff13415ffd0 2024-03-24T00:37:07.125 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-03-24T00:37:07.125 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw== 2024-03-24T00:37:07.125 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-03-24T00:37:07.125 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-03-24T00:37:07.125 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:37:07.126 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-03-24T00:37:07.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.121+0000 7ff139c00640 1 -- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff110067660 msgr2=0x7ff110069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.121+0000 7ff139c00640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff110067660 0x7ff110069b20 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff124007fd0 tx=0x7ff124009040 comp rx=0 tx=0).stop 2024-03-24T00:37:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.121+0000 7ff139c00640 1 -- 172.21.15.40:0/4100237915 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13415ffd0 msgr2=0x7ff1340aef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.121+0000 7ff139c00640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13415ffd0 0x7ff1340aef70 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00c8c0 tx=0x7ff12c00cd80 comp rx=0 tx=0).stop 2024-03-24T00:37:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.121+0000 7ff139c00640 1 -- 172.21.15.40:0/4100237915 shutdown_connections 2024-03-24T00:37:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.121+0000 7ff139c00640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff110067660 0x7ff110069b20 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.121+0000 7ff139c00640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff13415ffd0 0x7ff1340aef70 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.121+0000 7ff139c00640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff13415bd00 0x7ff1341456c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.121+0000 7ff139c00640 1 --2- 172.21.15.40:0/4100237915 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1340728d0 0x7ff134145f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.121+0000 7ff139c00640 1 -- 172.21.15.40:0/4100237915 >> 172.21.15.40:0/4100237915 conn(0x7ff1340b1660 msgr2=0x7ff134158b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.125+0000 7ff139c00640 1 -- 172.21.15.40:0/4100237915 shutdown_connections 2024-03-24T00:37:07.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.125+0000 7ff139c00640 1 -- 172.21.15.40:0/4100237915 wait complete. 2024-03-24T00:37:07.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f757a800640 1 Processor -- start 2024-03-24T00:37:07.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f757a800640 1 -- start start 2024-03-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f757a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75740b2490 0x7f75740cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f757a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75740b2aa0 0x7f75740d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f757a800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75740b2870 con 0x7f75740d08b0 2024-03-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f757a800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757406b6d0 con 0x7f75740b2490 2024-03-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f757a800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75740b3190 con 0x7f75740b2aa0 2024-03-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f7573e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f75740d08b0 0x7f75740ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60146/0 (socket says 172.21.15.40:60146) 2024-03-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f7573400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f75740b2490 0x7f75740cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f7573e00640 1 -- 172.21.15.40:0/147492898 learned_addr learned my addr 172.21.15.40:0/147492898 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f7572a00640 1 --2- 172.21.15.40:0/147492898 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75740b2aa0 0x7f75740d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f7572000640 1 -- 172.21.15.40:0/147492898 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645530749 0 0) 0x7f75740b2870 con 0x7f75740d08b0 2024-03-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f7572000640 1 -- 172.21.15.40:0/147492898 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7558003530 con 0x7f75740d08b0 2024-03-24T00:37:07.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f7572a00640 1 -- 172.21.15.40:0/147492898 >> v2:172.21.15.40:3300/0 conn(0x7f75740b2490 msgr2=0x7f75740cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f7572a00640 1 --2- 172.21.15.40:0/147492898 >> v2:172.21.15.40:3300/0 conn(0x7f75740b2490 0x7f75740cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f7572a00640 1 -- 172.21.15.40:0/147492898 >> v1:172.21.15.40:6789/0 conn(0x7f75740d08b0 legacy=0x7f75740ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f7572a00640 1 -- 172.21.15.40:0/147492898 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75740b3470 con 0x7f75740b2aa0 2024-03-24T00:37:07.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.285+0000 7f7573400640 1 --2- 172.21.15.40:0/147492898 >> v2:172.21.15.40:3300/0 conn(0x7f75740b2490 0x7f75740cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.289+0000 7f7572a00640 1 --2- 172.21.15.40:0/147492898 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75740b2aa0 0x7f75740d0370 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f75600097c0 tx=0x7f756001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9d984ef87eace26 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.289+0000 7f7572000640 1 -- 172.21.15.40:0/147492898 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75600b0070 con 0x7f75740b2aa0 2024-03-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.289+0000 7f7572000640 1 -- 172.21.15.40:0/147492898 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75600ab600 con 0x7f75740b2aa0 2024-03-24T00:37:07.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.289+0000 7f7572000640 1 -- 172.21.15.40:0/147492898 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75600ab9d0 con 0x7f75740b2aa0 2024-03-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.289+0000 7f757a800640 1 -- 172.21.15.40:0/147492898 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75740b2aa0 msgr2=0x7f75740d0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.289+0000 7f757a800640 1 --2- 172.21.15.40:0/147492898 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75740b2aa0 0x7f75740d0370 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f75600097c0 tx=0x7f756001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.289+0000 7f757a800640 1 -- 172.21.15.40:0/147492898 shutdown_connections 2024-03-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.289+0000 7f757a800640 1 --2- 172.21.15.40:0/147492898 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75740b2aa0 0x7f75740d0370 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.289+0000 7f757a800640 1 --2- 172.21.15.40:0/147492898 >> v2:172.21.15.40:3300/0 conn(0x7f75740b2490 0x7f75740cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.289+0000 7f757a800640 1 -- 172.21.15.40:0/147492898 >> 172.21.15.40:0/147492898 conn(0x7f75740d1d10 msgr2=0x7f75740d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.289+0000 7f757a800640 1 -- 172.21.15.40:0/147492898 shutdown_connections 2024-03-24T00:37:07.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.289+0000 7f757a800640 1 -- 172.21.15.40:0/147492898 wait complete. 2024-03-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f757a800640 1 Processor -- start 2024-03-24T00:37:07.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f757a800640 1 -- start start 2024-03-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f757a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75740b2490 0x7f757415d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f757a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75740b2aa0 0x7f757415a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f757a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75740d08b0 0x7f75740b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f757a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757416b350 con 0x7f75740b2aa0 2024-03-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f757a800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757416b1d0 con 0x7f75740b2490 2024-03-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f757a800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757416b4d0 con 0x7f75740d08b0 2024-03-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f7572a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75740b2aa0 0x7f757415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f7572a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75740b2aa0 0x7f757415a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49316/0 (socket says 172.21.15.40:49316) 2024-03-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f7572a00640 1 -- 172.21.15.40:0/4159122993 learned_addr learned my addr 172.21.15.40:0/4159122993 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f7573400640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75740b2490 0x7f757415d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f7573e00640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75740d08b0 0x7f75740b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f7572a00640 1 -- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75740d08b0 msgr2=0x7f75740b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f7572a00640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75740d08b0 0x7f75740b0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f7572a00640 1 -- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75740b2490 msgr2=0x7f757415d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f7572a00640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75740b2490 0x7f757415d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f7572a00640 1 -- 172.21.15.40:0/4159122993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757415db60 con 0x7f75740b2aa0 2024-03-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f7573e00640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75740d08b0 0x7f75740b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f7572a00640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75740b2aa0 0x7f757415a350 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f75600ae040 tx=0x7f7560019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f756b400640 1 -- 172.21.15.40:0/4159122993 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75600b0040 con 0x7f75740b2aa0 2024-03-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f756b400640 1 -- 172.21.15.40:0/4159122993 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75600b9070 con 0x7f75740b2aa0 2024-03-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f757a800640 1 -- 172.21.15.40:0/4159122993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f757415de20 con 0x7f75740b2aa0 2024-03-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f756b400640 1 -- 172.21.15.40:0/4159122993 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75600b4690 con 0x7f75740b2aa0 2024-03-24T00:37:07.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.293+0000 7f757a800640 1 -- 172.21.15.40:0/4159122993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7574159b50 con 0x7f75740b2aa0 2024-03-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.297+0000 7f757a800640 1 -- 172.21.15.40:0/4159122993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757415e340 con 0x7f75740b2aa0 2024-03-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.297+0000 7f756b400640 1 -- 172.21.15.40:0/4159122993 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f75600b4830 con 0x7f75740b2aa0 2024-03-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.297+0000 7f756b400640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7554067660 0x7f7554069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:07.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.297+0000 7f756b400640 1 -- 172.21.15.40:0/4159122993 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f756011f610 con 0x7f75740b2aa0 2024-03-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.297+0000 7f7573400640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7554067660 0x7f7554069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:07.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.297+0000 7f7573400640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7554067660 0x7f7554069b20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f75740cb830 tx=0x7f7564009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:07.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.301+0000 7f756b400640 1 -- 172.21.15.40:0/4159122993 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7560016090 con 0x7f75740b2aa0 2024-03-24T00:37:07.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.497+0000 7f757a800640 1 -- 172.21.15.40:0/4159122993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7574068170 con 0x7f75740b2aa0 2024-03-24T00:37:07.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.497+0000 7f756b400640 1 -- 172.21.15.40:0/4159122993 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4979 (secure 0 0 0) 0x7f75600f0e20 con 0x7f75740b2aa0 2024-03-24T00:37:07.503 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:07.503 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":97,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6755,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2936601899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2936601899},{"type":"v1","addr":"172.21.15.53:6836","nonce":2936601899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":7633,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1997904254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1997904254},{"type":"v1","addr":"172.21.15.40:6835","nonce":1997904254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":7917,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/1019660586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1019660586},{"type":"v1","addr":"172.21.15.40:6834","nonce":1019660586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:37:01.566949+0000","modified":"2024-03-24T00:37:04.591207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7932},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7932":{"gid":7932,"name":"d","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/3095501397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3095501397},{"type":"v1","addr":"172.21.15.53:6837","nonce":3095501397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":7}]} 2024-03-24T00:37:07.503 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 97 2024-03-24T00:37:07.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.501+0000 7f757a800640 1 -- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7554067660 msgr2=0x7f7554069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.501+0000 7f757a800640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7554067660 0x7f7554069b20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f75740cb830 tx=0x7f7564009190 comp rx=0 tx=0).stop 2024-03-24T00:37:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.501+0000 7f757a800640 1 -- 172.21.15.40:0/4159122993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75740b2aa0 msgr2=0x7f757415a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.501+0000 7f757a800640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75740b2aa0 0x7f757415a350 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f75600ae040 tx=0x7f7560019240 comp rx=0 tx=0).stop 2024-03-24T00:37:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.501+0000 7f757a800640 1 -- 172.21.15.40:0/4159122993 shutdown_connections 2024-03-24T00:37:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.501+0000 7f757a800640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7554067660 0x7f7554069b20 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.501+0000 7f757a800640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75740d08b0 0x7f75740b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.501+0000 7f757a800640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f75740b2aa0 0x7f757415a350 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.501+0000 7f757a800640 1 --2- 172.21.15.40:0/4159122993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75740b2490 0x7f757415d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.501+0000 7f757a800640 1 -- 172.21.15.40:0/4159122993 >> 172.21.15.40:0/4159122993 conn(0x7f75740d1d10 msgr2=0x7f75740aef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.501+0000 7f757a800640 1 -- 172.21.15.40:0/4159122993 shutdown_connections 2024-03-24T00:37:07.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:07.501+0000 7f757a800640 1 -- 172.21.15.40:0/4159122993 wait complete. 2024-03-24T00:37:07.763 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:37:07.767 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:37:07.925 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:37:01.566949+0000', 'modified': '2024-03-24T00:37:04.591207+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7932}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7932': {'gid': 7932, 'name': 'd', 'rank': 0, 'incarnation': 93, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6837/3095501397', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6835', 'nonce': 3095501397}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 3095501397}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7932, 'qdb_cluster': [7932]} 2024-03-24T00:37:07.925 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:37:07.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T00:37:08.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.065+0000 7f3fc7200640 1 Processor -- start 2024-03-24T00:37:08.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.065+0000 7f3fc7200640 1 -- start start 2024-03-24T00:37:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fc7200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3fc006e620 0x7f3fc00b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fc7200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fc00ccfa0 0x7f3fc00cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fc7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fc0070f60 con 0x7f3fc006ea00 2024-03-24T00:37:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fc7200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fc00b4c00 con 0x7f3fc006e620 2024-03-24T00:37:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fc7200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc00b3620 con 0x7f3fc00ccfa0 2024-03-24T00:37:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fc5200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fc00ccfa0 0x7f3fc00cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fbfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3fc006e620 0x7f3fc00b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fbfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3fc006e620 0x7f3fc00b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:53686/0 (socket says 172.21.15.40:53686) 2024-03-24T00:37:08.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fc5200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fc00ccfa0 0x7f3fc00cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:48776/0 (socket says 172.21.15.40:48776) 2024-03-24T00:37:08.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fbfe00640 1 -- 172.21.15.40:0/2571531387 learned_addr learned my addr 172.21.15.40:0/2571531387 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:08.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fc5200640 1 -- 172.21.15.40:0/2571531387 >> v2:172.21.15.53:3300/0 conn(0x7f3fc006e620 msgr2=0x7f3fc00b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:08.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fc5200640 1 --2- 172.21.15.40:0/2571531387 >> v2:172.21.15.53:3300/0 conn(0x7f3fc006e620 0x7f3fc00b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fc5200640 1 -- 172.21.15.40:0/2571531387 >> v1:172.21.15.53:6789/0 conn(0x7f3fc006ea00 legacy=0x7f3fc00b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fc5200640 1 -- 172.21.15.40:0/2571531387 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc00b3900 con 0x7f3fc00ccfa0 2024-03-24T00:37:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.069+0000 7f3fc5200640 1 --2- 172.21.15.40:0/2571531387 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fc00ccfa0 0x7f3fc00cd3d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009d90 tx=0x7f3fb801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b20912b01b56218 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fbea00640 1 -- 172.21.15.40:0/2571531387 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80b2070 con 0x7f3fc00ccfa0 2024-03-24T00:37:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fbea00640 1 -- 172.21.15.40:0/2571531387 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb80a5880 con 0x7f3fc00ccfa0 2024-03-24T00:37:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fbea00640 1 -- 172.21.15.40:0/2571531387 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb80a5c50 con 0x7f3fc00ccfa0 2024-03-24T00:37:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 -- 172.21.15.40:0/2571531387 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fc00ccfa0 msgr2=0x7f3fc00cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 --2- 172.21.15.40:0/2571531387 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fc00ccfa0 0x7f3fc00cd3d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3fb8009d90 tx=0x7f3fb801c8b0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 -- 172.21.15.40:0/2571531387 shutdown_connections 2024-03-24T00:37:08.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 --2- 172.21.15.40:0/2571531387 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fc00ccfa0 0x7f3fc00cd3d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 --2- 172.21.15.40:0/2571531387 >> v2:172.21.15.53:3300/0 conn(0x7f3fc006e620 0x7f3fc00b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 -- 172.21.15.40:0/2571531387 >> 172.21.15.40:0/2571531387 conn(0x7f3fc00b17b0 msgr2=0x7f3fc00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 -- 172.21.15.40:0/2571531387 shutdown_connections 2024-03-24T00:37:08.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 -- 172.21.15.40:0/2571531387 wait complete. 2024-03-24T00:37:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 Processor -- start 2024-03-24T00:37:08.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 -- start start 2024-03-24T00:37:08.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc006e620 0x7f3fc0159fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fc006ea00 0x7f3fc013c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fc00ccfa0 0x7f3fc0071300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc0144920 con 0x7f3fc006e620 2024-03-24T00:37:08.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc006ee10 con 0x7f3fc00ccfa0 2024-03-24T00:37:08.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc7200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc013d2d0 con 0x7f3fc006ea00 2024-03-24T00:37:08.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc006e620 0x7f3fc0159fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc006e620 0x7f3fc0159fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49320/0 (socket says 172.21.15.40:49320) 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fbfe00640 1 -- 172.21.15.40:0/3501045111 learned_addr learned my addr 172.21.15.40:0/3501045111 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fc5200640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fc00ccfa0 0x7f3fc0071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.073+0000 7f3fbf400640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fc006ea00 0x7f3fc013c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.077+0000 7f3fbfe00640 1 -- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fc006ea00 msgr2=0x7f3fc013c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.077+0000 7f3fbfe00640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fc006ea00 0x7f3fc013c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.077+0000 7f3fbfe00640 1 -- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fc00ccfa0 msgr2=0x7f3fc0071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.077+0000 7f3fbfe00640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fc00ccfa0 0x7f3fc0071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.077+0000 7f3fbfe00640 1 -- 172.21.15.40:0/3501045111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fc013d5b0 con 0x7f3fc006e620 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.077+0000 7f3fbfe00640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc006e620 0x7f3fc0159fc0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000e820 tx=0x7f3fb000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.077+0000 7f3fb7e00640 1 -- 172.21.15.40:0/3501045111 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0017070 con 0x7f3fc006e620 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.077+0000 7f3fb7e00640 1 -- 172.21.15.40:0/3501045111 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb000c940 con 0x7f3fc006e620 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.077+0000 7f3fb7e00640 1 -- 172.21.15.40:0/3501045111 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb000cd30 con 0x7f3fc006e620 2024-03-24T00:37:08.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.077+0000 7f3fc7200640 1 -- 172.21.15.40:0/3501045111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fc013d890 con 0x7f3fc006e620 2024-03-24T00:37:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.077+0000 7f3fc7200640 1 -- 172.21.15.40:0/3501045111 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fc013de50 con 0x7f3fc006e620 2024-03-24T00:37:08.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.077+0000 7f3fb7e00640 1 -- 172.21.15.40:0/3501045111 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3fb001a070 con 0x7f3fc006e620 2024-03-24T00:37:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fb7e00640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3fa0067660 0x7f3fa0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fb7e00640 1 -- 172.21.15.40:0/3501045111 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fb0085bd0 con 0x7f3fc006e620 2024-03-24T00:37:08.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fbf400640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3fa0067660 0x7f3fa0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fca76b000 1 Processor -- start 2024-03-24T00:37:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fca76b000 1 -- start start 2024-03-24T00:37:08.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fbf400640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3fa0067660 0x7f3fa0069b20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3fc013e6a0 tx=0x7f3fac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fca76b000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd205920 0x561cbd205d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fca76b000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 0x561cbd1bbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fca76b000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd1bbff0 0x561cbd1bc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fca76b000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cbd009680 con 0x561cbd206240 2024-03-24T00:37:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fca76b000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cbcf90b00 con 0x561cbd1bbff0 2024-03-24T00:37:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fca76b000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561cbd005720 con 0x561cbd205920 2024-03-24T00:37:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fbf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 0x561cbd1bbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fbfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd205920 0x561cbd205d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fc5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd1bbff0 0x561cbd1bc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fbf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 0x561cbd1bbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49326/0 (socket says 172.21.15.40:49326) 2024-03-24T00:37:08.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fbf400640 1 -- 172.21.15.40:0/1541800913 learned_addr learned my addr 172.21.15.40:0/1541800913 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fbf400640 1 -- 172.21.15.40:0/1541800913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd205920 msgr2=0x561cbd205d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fbf400640 1 --2- 172.21.15.40:0/1541800913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd205920 0x561cbd205d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fbf400640 1 -- 172.21.15.40:0/1541800913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd1bbff0 msgr2=0x561cbd1bc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fbf400640 1 --2- 172.21.15.40:0/1541800913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd1bbff0 0x561cbd1bc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fbf400640 1 -- 172.21.15.40:0/1541800913 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561cbcfe1940 con 0x561cbd206240 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fbf400640 1 --2- 172.21.15.40:0/1541800913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 0x561cbd1bbab0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f3fac001640 tx=0x7f3fac012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6814a978aa1ae1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fb4c00640 1 -- 172.21.15.40:0/1541800913 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac025070 con 0x561cbd206240 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fb4c00640 1 -- 172.21.15.40:0/1541800913 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fac01b050 con 0x561cbd206240 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.081+0000 7f3fb4c00640 1 -- 172.21.15.40:0/1541800913 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac014de0 con 0x561cbd206240 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 -- 172.21.15.40:0/1541800913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 msgr2=0x561cbd1bbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 --2- 172.21.15.40:0/1541800913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 0x561cbd1bbab0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f3fac001640 tx=0x7f3fac012000 comp rx=0 tx=0).stop 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 -- 172.21.15.40:0/1541800913 shutdown_connections 2024-03-24T00:37:08.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 --2- 172.21.15.40:0/1541800913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd1bbff0 0x561cbd1bc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 --2- 172.21.15.40:0/1541800913 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 0x561cbd1bbab0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 --2- 172.21.15.40:0/1541800913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd205920 0x561cbd205d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 -- 172.21.15.40:0/1541800913 >> 172.21.15.40:0/1541800913 conn(0x561cbd1b7fa0 msgr2=0x561cbd1ba390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 -- 172.21.15.40:0/1541800913 shutdown_connections 2024-03-24T00:37:08.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 -- 172.21.15.40:0/1541800913 wait complete. 2024-03-24T00:37:08.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 -- start start 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 Processor -- start 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd1bbff0 0x561cbd1ead40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd205920 0x561cbd1eb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 0x561cbd1eb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cbd008990 con 0x561cbd206240 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cbcfc3ae0 con 0x561cbd1bbff0 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561cbcfa1750 con 0x561cbd205920 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 10 client.0 _ll_register_callbacks cb 0x561cbd1a6ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fc5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 0x561cbd1eb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fbf400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd205920 0x561cbd1eb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fbfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd1bbff0 0x561cbd1ead40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fc5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 0x561cbd1eb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49330/0 (socket says 172.21.15.40:49330) 2024-03-24T00:37:08.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fc5200640 1 -- 172.21.15.40:0/775553909 learned_addr learned my addr 172.21.15.40:0/775553909 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fbf400640 1 -- 172.21.15.40:0/775553909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd1bbff0 msgr2=0x561cbd1ead40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fbf400640 1 --2- 172.21.15.40:0/775553909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd1bbff0 0x561cbd1ead40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fbf400640 1 -- 172.21.15.40:0/775553909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 msgr2=0x561cbd1eb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fbf400640 1 --2- 172.21.15.40:0/775553909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 0x561cbd1eb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fbf400640 1 -- 172.21.15.40:0/775553909 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561cbd04ed80 con 0x561cbd205920 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fbfe00640 1 --2- 172.21.15.40:0/775553909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd1bbff0 0x561cbd1ead40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fbf400640 1 --2- 172.21.15.40:0/775553909 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd205920 0x561cbd1eb280 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0067e60 tx=0x7f3fac000ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=252ae853334a4a6a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fa9600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fa9600640 1 -- 172.21.15.40:0/775553909 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac02f020 con 0x561cbd205920 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fa9600640 1 -- 172.21.15.40:0/775553909 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fac020a40 con 0x561cbd205920 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 -- 172.21.15.40:0/775553909 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3fac01b050 con 0x561cbd205920 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fa9600640 1 -- 172.21.15.40:0/775553909 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fac039050 con 0x561cbd205920 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 10 client.6845 fetch_fsmap learned FSMap version 97 2024-03-24T00:37:08.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fca76b000 1 -- 172.21.15.40:0/775553909 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x561cbcf90d10 con 0x561cbd205920 2024-03-24T00:37:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.085+0000 7f3fa9600640 1 -- 172.21.15.40:0/775553909 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fac01b050 con 0x561cbd205920 2024-03-24T00:37:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fa9600640 1 -- 172.21.15.40:0/775553909 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 97) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3fac03c020 con 0x561cbd205920 2024-03-24T00:37:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 10 client.6845 fetch_fsmap finished waiting for FSMap version 97 2024-03-24T00:37:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 10 client.6845 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:37:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 20 client.6845 populate_metadata read hostname 'smithi040' 2024-03-24T00:37:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 --2- 172.21.15.40:0/775553909 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x561cbd1ec090 0x561cbd217ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 4 client.6845 mds_command: new command op to 7932 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:37:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 -- 172.21.15.40:0/775553909 --> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561cbcfa4b60 con 0x561cbd1ec090 2024-03-24T00:37:08.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fbfe00640 1 --2- 172.21.15.40:0/775553909 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x561cbd1ec090 0x561cbd217ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fbfe00640 1 --2- 172.21.15.40:0/775553909 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x561cbd1ec090 0x561cbd217ea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c628b5281c7cbb4 server_cookie=579d811fec9c13b8 in_seq=0 out_seq=0 2024-03-24T00:37:08.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fa9600640 10 client.6845 ms_handle_connect on v2:172.21.15.53:6835/3095501397 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fa9600640 1 -- 172.21.15.40:0/775553909 <== mds.0 v2:172.21.15.53:6835/3095501397 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561cbcfa4b60 con 0x561cbd1ec090 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fa9600640 10 client.6845 handle_command_reply: tid=0 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 client.6845 shutdown 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 -- 172.21.15.40:0/775553909 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd205920 msgr2=0x561cbd1eb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 --2- 172.21.15.40:0/775553909 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd205920 0x561cbd1eb280 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3fc0067e60 tx=0x7f3fac000ee0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 -- 172.21.15.40:0/775553909 shutdown_connections 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 --2- 172.21.15.40:0/775553909 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x561cbd1ec090 0x561cbd217ea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 --2- 172.21.15.40:0/775553909 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd206240 0x561cbd1eb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 --2- 172.21.15.40:0/775553909 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd205920 0x561cbd1eb280 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fa9600640 0 client.6845 ms_handle_reset on v2:172.21.15.53:6835/3095501397 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 --2- 172.21.15.40:0/775553909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd1bbff0 0x561cbd1ead40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 -- 172.21.15.40:0/775553909 >> 172.21.15.40:0/775553909 conn(0x561cbd1b7fa0 msgr2=0x561cbd1bac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 -- 172.21.15.40:0/775553909 shutdown_connections 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 1 -- 172.21.15.40:0/775553909 wait complete. 2024-03-24T00:37:08.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.089+0000 7f3fca76b000 20 client.6845 trim_cache size 0 max 16384 2024-03-24T00:37:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.105+0000 7f3fca76b000 1 Processor -- start 2024-03-24T00:37:08.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.105+0000 7f3fca76b000 1 -- start start 2024-03-24T00:37:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 0x561cbd1e9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd205920 0x561cbd1e9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd206240 0x561cbd267170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc0203410 con 0x561cbd1bbff0 2024-03-24T00:37:08.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc02025d0 con 0x561cbd205920 2024-03-24T00:37:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc0201b70 con 0x561cbd206240 2024-03-24T00:37:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 0x561cbd1e9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 0x561cbd1e9900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49338/0 (socket says 172.21.15.40:49338) 2024-03-24T00:37:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fbfe00640 1 -- 172.21.15.40:0/1575199142 learned_addr learned my addr 172.21.15.40:0/1575199142 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fbf400640 1 --2- 172.21.15.40:0/1575199142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd205920 0x561cbd1e9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fc5200640 1 --2- 172.21.15.40:0/1575199142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd206240 0x561cbd267170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fbfe00640 1 -- 172.21.15.40:0/1575199142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd206240 msgr2=0x561cbd267170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fbfe00640 1 --2- 172.21.15.40:0/1575199142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd206240 0x561cbd267170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fbfe00640 1 -- 172.21.15.40:0/1575199142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd205920 msgr2=0x561cbd1e9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fbfe00640 1 --2- 172.21.15.40:0/1575199142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd205920 0x561cbd1e9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fbfe00640 1 -- 172.21.15.40:0/1575199142 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561cbcfd6540 con 0x561cbd1bbff0 2024-03-24T00:37:08.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fbfe00640 1 --2- 172.21.15.40:0/1575199142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 0x561cbd1e9900 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0084060 tx=0x7f3fb001e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c53863a4dd4c90f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fabe00640 1 -- 172.21.15.40:0/1575199142 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0084170 con 0x561cbd1bbff0 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fabe00640 1 -- 172.21.15.40:0/1575199142 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb00543e0 con 0x561cbd1bbff0 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fabe00640 1 -- 172.21.15.40:0/1575199142 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb0055730 con 0x561cbd1bbff0 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 -- 172.21.15.40:0/1575199142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 msgr2=0x561cbd1e9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 --2- 172.21.15.40:0/1575199142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 0x561cbd1e9900 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0084060 tx=0x7f3fb001e040 comp rx=0 tx=0).stop 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 -- 172.21.15.40:0/1575199142 shutdown_connections 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 --2- 172.21.15.40:0/1575199142 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd206240 0x561cbd267170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 --2- 172.21.15.40:0/1575199142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd205920 0x561cbd1e9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 --2- 172.21.15.40:0/1575199142 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 0x561cbd1e9900 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 -- 172.21.15.40:0/1575199142 >> 172.21.15.40:0/1575199142 conn(0x561cbd1b7fa0 msgr2=0x561cbd212f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 -- 172.21.15.40:0/1575199142 shutdown_connections 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 -- 172.21.15.40:0/1575199142 wait complete. 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 -- start start 2024-03-24T00:37:08.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 Processor -- start 2024-03-24T00:37:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 0x561cbd218ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd205920 0x561cbd219430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd206240 0x561cbd219970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561cbcfb1a80 con 0x561cbd1bbff0 2024-03-24T00:37:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fc0201fd0 con 0x561cbd205920 2024-03-24T00:37:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.109+0000 7f3fca76b000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561cbd0c6700 con 0x561cbd206240 2024-03-24T00:37:08.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 10 client.0 _ll_register_callbacks cb 0x7f3fac023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 0x561cbd218ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbfe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 0x561cbd218ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49342/0 (socket says 172.21.15.40:49342) 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd205920 0x561cbd219430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbfe00640 1 -- 172.21.15.40:0/3604804636 learned_addr learned my addr 172.21.15.40:0/3604804636 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fc5200640 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd206240 0x561cbd219970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbfe00640 1 -- 172.21.15.40:0/3604804636 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd206240 msgr2=0x561cbd219970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbfe00640 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd206240 0x561cbd219970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbfe00640 1 -- 172.21.15.40:0/3604804636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd205920 msgr2=0x561cbd219430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbfe00640 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd205920 0x561cbd219430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbfe00640 1 -- 172.21.15.40:0/3604804636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561cbcf9d4b0 con 0x561cbd1bbff0 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbf400640 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd205920 0x561cbd219430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbfe00640 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 0x561cbd218ef0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0055de0 tx=0x7f3fb0083f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb7bbc95c383b22 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fa9600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fa9600640 1 -- 172.21.15.40:0/3604804636 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb007a070 con 0x561cbd1bbff0 2024-03-24T00:37:08.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fa9600640 1 -- 172.21.15.40:0/3604804636 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3fb0055730 con 0x561cbd1bbff0 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 -- 172.21.15.40:0/3604804636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3fb00543e0 con 0x561cbd1bbff0 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fa9600640 1 -- 172.21.15.40:0/3604804636 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=97) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3fb005c110 con 0x561cbd1bbff0 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 10 client.8079 fetch_fsmap learned FSMap version 97 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 -- 172.21.15.40:0/3604804636 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=97,monmap=2+}) v3 -- 0x561cbcfb02f0 con 0x561cbd1bbff0 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fa9600640 1 -- 172.21.15.40:0/3604804636 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb00543e0 con 0x561cbd1bbff0 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fa9600640 1 -- 172.21.15.40:0/3604804636 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 97) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f3fb005c2b0 con 0x561cbd1bbff0 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 10 client.8079 fetch_fsmap finished waiting for FSMap version 97 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 10 client.8079 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 20 client.8079 populate_metadata read hostname 'smithi040' 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x561cbd21a2a0 0x561cbd2731f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 4 client.8079 mds_command: new command op to 7932 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 -- 172.21.15.40:0/3604804636 --> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561cbd016bc0 con 0x561cbd21a2a0 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbf400640 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x561cbd21a2a0 0x561cbd2731f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fbf400640 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x561cbd21a2a0 0x561cbd2731f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33ddab416cb6bdef server_cookie=28c72829b31f859f in_seq=0 out_seq=0 2024-03-24T00:37:08.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fa9600640 10 client.8079 ms_handle_connect on v2:172.21.15.53:6835/3095501397 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fa9600640 1 -- 172.21.15.40:0/3604804636 <== mds.0 v2:172.21.15.53:6835/3095501397 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x561cbd016bc0 con 0x561cbd21a2a0 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fa9600640 10 client.8079 handle_command_reply: tid=0 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 client.8079 shutdown 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 -- 172.21.15.40:0/3604804636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 msgr2=0x561cbd218ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 0x561cbd218ef0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f3fb0055de0 tx=0x7f3fb0083f30 comp rx=0 tx=0).stop 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 -- 172.21.15.40:0/3604804636 shutdown_connections 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x561cbd21a2a0 0x561cbd2731f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561cbd206240 0x561cbd219970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561cbd205920 0x561cbd219430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fa9600640 0 client.8079 ms_handle_reset on v2:172.21.15.53:6835/3095501397 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 --2- 172.21.15.40:0/3604804636 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561cbd1bbff0 0x561cbd218ef0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 -- 172.21.15.40:0/3604804636 >> 172.21.15.40:0/3604804636 conn(0x561cbd1b7fa0 msgr2=0x561cbd212680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 -- 172.21.15.40:0/3604804636 shutdown_connections 2024-03-24T00:37:08.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 1 -- 172.21.15.40:0/3604804636 wait complete. 2024-03-24T00:37:08.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fca76b000 20 client.8079 trim_cache size 0 max 16384 2024-03-24T00:37:08.123 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:37:08.123 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:37:08.123 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:37:08.123 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 7932, 2024-03-24T00:37:08.123 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:37:08.123 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:37:08.123 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:37:08.123 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 5.9257988599999996, 2024-03-24T00:37:08.123 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 97, 2024-03-24T00:37:08.123 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 95, 2024-03-24T00:37:08.124 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 95, 2024-03-24T00:37:08.124 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 10.774046796 2024-03-24T00:37:08.124 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:37:08.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fc7200640 1 -- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3fa0067660 msgr2=0x7f3fa0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fc7200640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3fa0067660 0x7f3fa0069b20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f3fc013e6a0 tx=0x7f3fac009040 comp rx=0 tx=0).stop 2024-03-24T00:37:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fc7200640 1 -- 172.21.15.40:0/3501045111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc006e620 msgr2=0x7f3fc0159fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fc7200640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc006e620 0x7f3fc0159fc0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000e820 tx=0x7f3fb000ece0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fc7200640 1 -- 172.21.15.40:0/3501045111 shutdown_connections 2024-03-24T00:37:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fc7200640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3fa0067660 0x7f3fa0069b20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fc7200640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3fc00ccfa0 0x7f3fc0071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fc7200640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3fc006ea00 0x7f3fc013c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fc7200640 1 --2- 172.21.15.40:0/3501045111 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3fc006e620 0x7f3fc0159fc0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.113+0000 7f3fc7200640 1 -- 172.21.15.40:0/3501045111 >> 172.21.15.40:0/3501045111 conn(0x7f3fc00b17b0 msgr2=0x7f3fc015f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.117+0000 7f3fc7200640 1 -- 172.21.15.40:0/3501045111 shutdown_connections 2024-03-24T00:37:08.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.117+0000 7f3fc7200640 1 -- 172.21.15.40:0/3501045111 wait complete. 2024-03-24T00:37:08.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:37:08.280 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6512e00640 1 Processor -- start 2024-03-24T00:37:08.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6512e00640 1 -- start start 2024-03-24T00:37:08.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6512e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f650c0b46c0 0x7f650c0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6512e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f650c0b2ff0 0x7f650c0ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6512e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f650c070f60 con 0x7f650c06e620 2024-03-24T00:37:08.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6512e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f650c0b3620 con 0x7f650c0b46c0 2024-03-24T00:37:08.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6512e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f650c0b3900 con 0x7f650c0b2ff0 2024-03-24T00:37:08.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f650be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f650c06e620 0x7f650c06ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60158/0 (socket says 172.21.15.40:60158) 2024-03-24T00:37:08.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f650be00640 1 -- 172.21.15.40:0/3019116683 learned_addr learned my addr 172.21.15.40:0/3019116683 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:08.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6503400640 1 --2- 172.21.15.40:0/3019116683 >> v2:172.21.15.53:3300/0 conn(0x7f650c0b46c0 0x7f650c0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6510e00640 1 --2- 172.21.15.40:0/3019116683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f650c0b2ff0 0x7f650c0ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f650b400640 1 -- 172.21.15.40:0/3019116683 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181058929 0 0) 0x7f650c070f60 con 0x7f650c06e620 2024-03-24T00:37:08.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f650b400640 1 -- 172.21.15.40:0/3019116683 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f0003530 con 0x7f650c06e620 2024-03-24T00:37:08.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6503400640 1 -- 172.21.15.40:0/3019116683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f650c0b2ff0 msgr2=0x7f650c0ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:08.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6503400640 1 --2- 172.21.15.40:0/3019116683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f650c0b2ff0 0x7f650c0ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6503400640 1 -- 172.21.15.40:0/3019116683 >> v1:172.21.15.40:6789/0 conn(0x7f650c06e620 legacy=0x7f650c06ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6503400640 1 -- 172.21.15.40:0/3019116683 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650c0b3be0 con 0x7f650c0b46c0 2024-03-24T00:37:08.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.277+0000 7f6510e00640 1 --2- 172.21.15.40:0/3019116683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f650c0b2ff0 0x7f650c0ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:08.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.281+0000 7f6503400640 1 --2- 172.21.15.40:0/3019116683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f650c0b46c0 0x7f650c0b2ab0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f64f40097c0 tx=0x7f64f401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70287dd5241a5b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.281+0000 7f650b400640 1 -- 172.21.15.40:0/3019116683 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f40b0070 con 0x7f650c0b46c0 2024-03-24T00:37:08.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.281+0000 7f650b400640 1 -- 172.21.15.40:0/3019116683 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f40ab600 con 0x7f650c0b46c0 2024-03-24T00:37:08.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.281+0000 7f650b400640 1 -- 172.21.15.40:0/3019116683 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c0cdcf0 con 0x7f650c0b46c0 2024-03-24T00:37:08.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.281+0000 7f6512e00640 1 -- 172.21.15.40:0/3019116683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f650c0b46c0 msgr2=0x7f650c0b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.281+0000 7f6512e00640 1 --2- 172.21.15.40:0/3019116683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f650c0b46c0 0x7f650c0b2ab0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f64f40097c0 tx=0x7f64f401c410 comp rx=0 tx=0).stop 2024-03-24T00:37:08.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.281+0000 7f6512e00640 1 -- 172.21.15.40:0/3019116683 shutdown_connections 2024-03-24T00:37:08.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.281+0000 7f6512e00640 1 --2- 172.21.15.40:0/3019116683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f650c0b2ff0 0x7f650c0ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.281+0000 7f6512e00640 1 --2- 172.21.15.40:0/3019116683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f650c0b46c0 0x7f650c0b2ab0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.281+0000 7f6512e00640 1 -- 172.21.15.40:0/3019116683 >> 172.21.15.40:0/3019116683 conn(0x7f650c0b17b0 msgr2=0x7f650c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:08.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.281+0000 7f6512e00640 1 -- 172.21.15.40:0/3019116683 shutdown_connections 2024-03-24T00:37:08.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.281+0000 7f6512e00640 1 -- 172.21.15.40:0/3019116683 wait complete. 2024-03-24T00:37:08.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6512e00640 1 Processor -- start 2024-03-24T00:37:08.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6512e00640 1 -- start start 2024-03-24T00:37:08.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6512e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f650c06e620 0x7f650c13b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6512e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f650c0b2ff0 0x7f650c1585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6512e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f650c0b46c0 0x7f650c13e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6512e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c144920 con 0x7f650c0b2ff0 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6512e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650c167c80 con 0x7f650c06e620 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6512e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f650c158af0 con 0x7f650c0b46c0 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6503400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f650c0b2ff0 0x7f650c1585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6503400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f650c0b2ff0 0x7f650c1585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49356/0 (socket says 172.21.15.40:49356) 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6503400640 1 -- 172.21.15.40:0/3727600038 learned_addr learned my addr 172.21.15.40:0/3727600038 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f650be00640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f650c06e620 0x7f650c13b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6510e00640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f650c0b46c0 0x7f650c13e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6503400640 1 -- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f650c0b46c0 msgr2=0x7f650c13e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6503400640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f650c0b46c0 0x7f650c13e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6503400640 1 -- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f650c06e620 msgr2=0x7f650c13b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6503400640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f650c06e620 0x7f650c13b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6503400640 1 -- 172.21.15.40:0/3727600038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650c15c100 con 0x7f650c0b2ff0 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f650be00640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f650c06e620 0x7f650c13b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:08.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6503400640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f650c0b2ff0 0x7f650c1585b0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f64f401aac0 tx=0x7f64f40191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6508c00640 1 -- 172.21.15.40:0/3727600038 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f40ba020 con 0x7f650c0b2ff0 2024-03-24T00:37:08.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6508c00640 1 -- 172.21.15.40:0/3727600038 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f40b5460 con 0x7f650c0b2ff0 2024-03-24T00:37:08.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6512e00640 1 -- 172.21.15.40:0/3727600038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650c15c3e0 con 0x7f650c0b2ff0 2024-03-24T00:37:08.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6508c00640 1 -- 172.21.15.40:0/3727600038 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f40b0040 con 0x7f650c0b2ff0 2024-03-24T00:37:08.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.285+0000 7f6512e00640 1 -- 172.21.15.40:0/3727600038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650c13f450 con 0x7f650c0b2ff0 2024-03-24T00:37:08.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.289+0000 7f6512e00640 1 -- 172.21.15.40:0/3727600038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650c068170 con 0x7f650c0b2ff0 2024-03-24T00:37:08.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.289+0000 7f6508c00640 1 -- 172.21.15.40:0/3727600038 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f64f40bd020 con 0x7f650c0b2ff0 2024-03-24T00:37:08.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.289+0000 7f6508c00640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f64e8067690 0x7f64e8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:08.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.289+0000 7f6508c00640 1 -- 172.21.15.40:0/3727600038 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f64f411fad0 con 0x7f650c0b2ff0 2024-03-24T00:37:08.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.289+0000 7f650be00640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f64e8067690 0x7f64e8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:08.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.289+0000 7f650be00640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f64e8067690 0x7f64e8069b50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f64fc005890 tx=0x7f64fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:08.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.293+0000 7f6508c00640 1 -- 172.21.15.40:0/3727600038 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f64f40c4030 con 0x7f650c0b2ff0 2024-03-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.493+0000 7f6512e00640 1 -- 172.21.15.40:0/3727600038 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f650c072f50 con 0x7f650c0b2ff0 2024-03-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.493+0000 7f6508c00640 1 -- 172.21.15.40:0/3727600038 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v97) v1 ==== 53+0+73 (secure 0 0 0) 0x7f64f4016090 con 0x7f650c0b2ff0 2024-03-24T00:37:08.498 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.497+0000 7f6512e00640 1 -- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f64e8067690 msgr2=0x7f64e8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.497+0000 7f6512e00640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f64e8067690 0x7f64e8069b50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f64fc005890 tx=0x7f64fc009040 comp rx=0 tx=0).stop 2024-03-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.497+0000 7f6512e00640 1 -- 172.21.15.40:0/3727600038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f650c0b2ff0 msgr2=0x7f650c1585b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.497+0000 7f6512e00640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f650c0b2ff0 0x7f650c1585b0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f64f401aac0 tx=0x7f64f40191b0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.497+0000 7f6512e00640 1 -- 172.21.15.40:0/3727600038 shutdown_connections 2024-03-24T00:37:08.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.497+0000 7f6512e00640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f64e8067690 0x7f64e8069b50 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.497+0000 7f6512e00640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f650c0b46c0 0x7f650c13e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.497+0000 7f6512e00640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f650c0b2ff0 0x7f650c1585b0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.497+0000 7f6512e00640 1 --2- 172.21.15.40:0/3727600038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f650c06e620 0x7f650c13b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.497+0000 7f6512e00640 1 -- 172.21.15.40:0/3727600038 >> 172.21.15.40:0/3727600038 conn(0x7f650c0b17b0 msgr2=0x7f650c15e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.497+0000 7f6512e00640 1 -- 172.21.15.40:0/3727600038 shutdown_connections 2024-03-24T00:37:08.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:08.497+0000 7f6512e00640 1 -- 172.21.15.40:0/3727600038 wait complete. 2024-03-24T00:37:08.519 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:37:08.519 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:37:08.519 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-03-24T00:37:08.520 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:37:08.520 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:08.520 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:37:08.520 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:37:08.520 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:37:08.520 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:37:08.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:08.520 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-03-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-03-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:37:08.530 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42121sec preferred_lft 42121sec 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 02:42:9e:ab:82:f9 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:37:08.531 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-24T00:37:08.532 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:37:08.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:08.533 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:37:08.589 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-03-24T00:37:08.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:08.591 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-03-24T00:37:08.645 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-03-24T00:37:08.645 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:37:08.645 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-24T00:37:08.645 INFO:teuthology.orchestra.run.smithi040.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-24T00:37:08.645 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-24T00:37:08.645 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:37:08.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:08.646 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:37:08.646 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:37:08.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T00:37:08.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T00:37:08.646 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T00:37:08.646 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:37:08.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:08.737 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-24T00:37:08.745 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:37:08.745 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:37:08.746 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:37:08.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:08.746 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-03-24T00:37:08.811 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T00:37:08.811 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:37:08.811 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:08.863 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:37:08.863 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:37:08.864 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:08.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:37:08.975 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:37:09.033 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:37:09.034 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:37:09.035 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:37:09.035 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:37:09.035 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:37:09.035 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:37:09.035 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:37:09.035 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:37:09.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:09.036 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:37:09.088 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T00:37:09.088 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T00:37:09.134 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:37:09.155 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:37:09.205 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:37:09.206 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:37:09.207 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:37:09.207 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:37:09.207 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:37:09.207 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:37:09.207 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:37:09.207 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:37:09.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:09.208 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:37:09.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:2024-03-24T00:37:09.229+0000 7f02c2b83580 -1 init, newargv = 0x560f4517c650 newargc=15 2024-03-24T00:37:09.232 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[48527]: starting ceph client 2024-03-24T00:37:09.238 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[48527]: starting fuse 2024-03-24T00:37:09.242 INFO:teuthology.orchestra.run.smithi040.stdout:43 2024-03-24T00:37:09.242 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-24T00:37:09.242 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:37:09.242 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-03-24T00:37:09.242 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-03-24T00:37:09.242 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-03-24T00:37:09.242 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-03-24T00:37:09.242 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:37:09.242 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-03-24T00:37:09.242 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T00:37:09.242 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:37:09.243 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:37:09.244 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.0")) 2024-03-24T00:37:09.244 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-03-24T00:37:09.369 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.0.48527.asok 2024-03-24T00:37:09.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:37:09.381 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.48527.asok 2024-03-24T00:37:09.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:09.381 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.48527.asok status 2024-03-24T00:37:09.560 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:37:09.560 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-03-24T00:37:09.560 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "0", 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "48527", 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8091, 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 8091 2024-03-24T00:37:09.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 153107016 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 153107016 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.8091 192.168.0.1:0/153107016", 2024-03-24T00:37:09.562 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "192.168.0.1:0/153107016", 2024-03-24T00:37:09.563 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-03-24T00:37:09.563 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 97, 2024-03-24T00:37:09.563 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 95, 2024-03-24T00:37:09.563 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:37:09.563 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-03-24T00:37:09.563 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-03-24T00:37:09.563 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:37:09.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:09.576 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-03-24T00:37:09.576 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:09.586 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-03-24T00:37:09.586 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:09.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:09.586 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:09.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random True 2024-03-24T00:37:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.829+0000 7f666c000640 1 Processor -- start 2024-03-24T00:37:09.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.829+0000 7f666c000640 1 -- start start 2024-03-24T00:37:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.829+0000 7f666c000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f666415bd00 0x7f66640ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.829+0000 7f666c000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f66640728d0 0x7f66640cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.829+0000 7f666c000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6664072d30 con 0x7f666415f8c0 2024-03-24T00:37:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.829+0000 7f666c000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66640cf790 con 0x7f66640728d0 2024-03-24T00:37:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.829+0000 7f666c000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6664071de0 con 0x7f666415bd00 2024-03-24T00:37:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.829+0000 7f6669600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f666415bd00 0x7f66640ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.829+0000 7f6669600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f666415bd00 0x7f66640ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:48848/0 (socket says 172.21.15.40:48848) 2024-03-24T00:37:09.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.829+0000 7f6669600640 1 -- 172.21.15.40:0/3523626841 learned_addr learned my addr 172.21.15.40:0/3523626841 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.829+0000 7f6668c00640 1 --2- 172.21.15.40:0/3523626841 >> v2:172.21.15.53:3300/0 conn(0x7f66640728d0 0x7f66640cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.833+0000 7f6669600640 1 -- 172.21.15.40:0/3523626841 >> v2:172.21.15.53:3300/0 conn(0x7f66640728d0 msgr2=0x7f66640cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.833+0000 7f6669600640 1 --2- 172.21.15.40:0/3523626841 >> v2:172.21.15.53:3300/0 conn(0x7f66640728d0 0x7f66640cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.833+0000 7f6669600640 1 -- 172.21.15.40:0/3523626841 >> v1:172.21.15.53:6789/0 conn(0x7f666415f8c0 legacy=0x7f666406cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.833+0000 7f6669600640 1 -- 172.21.15.40:0/3523626841 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66640cea50 con 0x7f666415bd00 2024-03-24T00:37:09.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.833+0000 7f6669600640 1 --2- 172.21.15.40:0/3523626841 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f666415bd00 0x7f66640ae8e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6658009870 tx=0x7f665801c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfba16fbc590fe69 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:09.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.833+0000 7f6663e00640 1 -- 172.21.15.40:0/3523626841 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66580a95f0 con 0x7f666415bd00 2024-03-24T00:37:09.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.833+0000 7f6663e00640 1 -- 172.21.15.40:0/3523626841 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66580a9790 con 0x7f666415bd00 2024-03-24T00:37:09.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.833+0000 7f6663e00640 1 -- 172.21.15.40:0/3523626841 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66580a9b60 con 0x7f666415bd00 2024-03-24T00:37:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 -- 172.21.15.40:0/3523626841 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f666415bd00 msgr2=0x7f66640ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 --2- 172.21.15.40:0/3523626841 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f666415bd00 0x7f66640ae8e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6658009870 tx=0x7f665801c530 comp rx=0 tx=0).stop 2024-03-24T00:37:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 -- 172.21.15.40:0/3523626841 shutdown_connections 2024-03-24T00:37:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 --2- 172.21.15.40:0/3523626841 >> v2:172.21.15.53:3300/0 conn(0x7f66640728d0 0x7f66640cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 --2- 172.21.15.40:0/3523626841 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f666415bd00 0x7f66640ae8e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 -- 172.21.15.40:0/3523626841 >> 172.21.15.40:0/3523626841 conn(0x7f66640b1660 msgr2=0x7f66640b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 -- 172.21.15.40:0/3523626841 shutdown_connections 2024-03-24T00:37:09.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 -- 172.21.15.40:0/3523626841 wait complete. 2024-03-24T00:37:09.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 Processor -- start 2024-03-24T00:37:09.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 -- start start 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66640728d0 0x7f666415cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f666415bd00 0x7f666414e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f666415f8c0 0x7f666414e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666415c0e0 con 0x7f666415bd00 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666406d630 con 0x7f666415f8c0 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666c000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66641600e0 con 0x7f66640728d0 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f6668c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f666415bd00 0x7f666414e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f6669600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66640728d0 0x7f666415cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f6668c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f666415bd00 0x7f666414e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49362/0 (socket says 172.21.15.40:49362) 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f6669600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66640728d0 0x7f666415cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:48858/0 (socket says 172.21.15.40:48858) 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f6668c00640 1 -- 172.21.15.40:0/2741849738 learned_addr learned my addr 172.21.15.40:0/2741849738 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f666a000640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f666415f8c0 0x7f666414e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f6668c00640 1 -- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66640728d0 msgr2=0x7f666415cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f6668c00640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66640728d0 0x7f666415cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f6668c00640 1 -- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f666415f8c0 msgr2=0x7f666414e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f6668c00640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f666415f8c0 0x7f666414e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.837+0000 7f6668c00640 1 -- 172.21.15.40:0/2741849738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66640b2ff0 con 0x7f666415bd00 2024-03-24T00:37:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.841+0000 7f6668c00640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f666415bd00 0x7f666414e190 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f665000e930 tx=0x7f665000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.841+0000 7f6661600640 1 -- 172.21.15.40:0/2741849738 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6650017070 con 0x7f666415bd00 2024-03-24T00:37:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.841+0000 7f6661600640 1 -- 172.21.15.40:0/2741849738 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665000ca00 con 0x7f666415bd00 2024-03-24T00:37:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.841+0000 7f666c000640 1 -- 172.21.15.40:0/2741849738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66640b32d0 con 0x7f666415bd00 2024-03-24T00:37:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.841+0000 7f6661600640 1 -- 172.21.15.40:0/2741849738 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665000cdb0 con 0x7f666415bd00 2024-03-24T00:37:09.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.841+0000 7f666c000640 1 -- 172.21.15.40:0/2741849738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66640b3890 con 0x7f666415bd00 2024-03-24T00:37:09.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.841+0000 7f666c000640 1 -- 172.21.15.40:0/2741849738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6664068170 con 0x7f666415bd00 2024-03-24T00:37:09.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.841+0000 7f6661600640 1 -- 172.21.15.40:0/2741849738 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f665001a070 con 0x7f666415bd00 2024-03-24T00:37:09.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.841+0000 7f6661600640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6644067660 0x7f6644069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:09.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.841+0000 7f6661600640 1 -- 172.21.15.40:0/2741849738 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6650085c90 con 0x7f666415bd00 2024-03-24T00:37:09.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.845+0000 7f6669600640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6644067660 0x7f6644069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:09.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.845+0000 7f6669600640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6644067660 0x7f6644069b20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f66580ae000 tx=0x7f66580b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:09.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:09.849+0000 7f6661600640 1 -- 172.21.15.40:0/2741849738 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f66500562e0 con 0x7f666415bd00 2024-03-24T00:37:10.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.013+0000 7f666c000640 1 -- 172.21.15.40:0/2741849738 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random}] v 0) v1 -- 0x7f666406e870 con 0x7f666415bd00 2024-03-24T00:37:10.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f6661600640 1 -- 172.21.15.40:0/2741849738 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random}]=0 v52)=0 v52) v1 ==== 128+0+0 (secure 0 0 0) 0x7f665005a380 con 0x7f666415bd00 2024-03-24T00:37:10.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f666c000640 1 -- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6644067660 msgr2=0x7f6644069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f666c000640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6644067660 0x7f6644069b20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f66580ae000 tx=0x7f66580b3040 comp rx=0 tx=0).stop 2024-03-24T00:37:10.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f666c000640 1 -- 172.21.15.40:0/2741849738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f666415bd00 msgr2=0x7f666414e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f666c000640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f666415bd00 0x7f666414e190 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f665000e930 tx=0x7f665000edf0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f666c000640 1 -- 172.21.15.40:0/2741849738 shutdown_connections 2024-03-24T00:37:10.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f666c000640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6644067660 0x7f6644069b20 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f666c000640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f666415f8c0 0x7f666414e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f666c000640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f666415bd00 0x7f666414e190 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f666c000640 1 --2- 172.21.15.40:0/2741849738 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66640728d0 0x7f666415cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f666c000640 1 -- 172.21.15.40:0/2741849738 >> 172.21.15.40:0/2741849738 conn(0x7f66640b1660 msgr2=0x7f66641546c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:10.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f666c000640 1 -- 172.21.15.40:0/2741849738 shutdown_connections 2024-03-24T00:37:10.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.017+0000 7f666c000640 1 -- 172.21.15.40:0/2741849738 wait complete. 2024-03-24T00:37:10.043 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_distributed True 2024-03-24T00:37:10.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.181+0000 7f390dc00640 1 Processor -- start 2024-03-24T00:37:10.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.181+0000 7f390dc00640 1 -- start start 2024-03-24T00:37:10.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.181+0000 7f390dc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f390815e350 0x7f39080cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.181+0000 7f390dc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f39080c9580 0x7f39080c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.181+0000 7f390dc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39080734e0 con 0x7f39080ad1e0 2024-03-24T00:37:10.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.181+0000 7f390dc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39080cd660 con 0x7f390815e350 2024-03-24T00:37:10.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.181+0000 7f390dc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39080c9f70 con 0x7f39080c9580 2024-03-24T00:37:10.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.185+0000 7f3906a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f390815e350 0x7f39080cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.185+0000 7f3907400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f39080ad1e0 0x7f39080ad5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60166/0 (socket says 172.21.15.40:60166) 2024-03-24T00:37:10.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.185+0000 7f3907400640 1 -- 172.21.15.40:0/1949393751 learned_addr learned my addr 172.21.15.40:0/1949393751 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:10.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.185+0000 7f38fe000640 1 --2- 172.21.15.40:0/1949393751 >> v2:172.21.15.53:3300/0 conn(0x7f39080c9580 0x7f39080c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.185+0000 7f3906000640 1 -- 172.21.15.40:0/1949393751 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1174640463 0 0) 0x7f39080734e0 con 0x7f39080ad1e0 2024-03-24T00:37:10.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.185+0000 7f3906000640 1 -- 172.21.15.40:0/1949393751 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38ec003530 con 0x7f39080ad1e0 2024-03-24T00:37:10.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.185+0000 7f3906a00640 1 -- 172.21.15.40:0/1949393751 >> v2:172.21.15.53:3300/0 conn(0x7f39080c9580 msgr2=0x7f39080c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:10.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.185+0000 7f3906a00640 1 --2- 172.21.15.40:0/1949393751 >> v2:172.21.15.53:3300/0 conn(0x7f39080c9580 0x7f39080c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.185+0000 7f3906a00640 1 -- 172.21.15.40:0/1949393751 >> v1:172.21.15.40:6789/0 conn(0x7f39080ad1e0 legacy=0x7f39080ad5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.185+0000 7f3906a00640 1 -- 172.21.15.40:0/1949393751 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39080d0030 con 0x7f390815e350 2024-03-24T00:37:10.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.185+0000 7f3906a00640 1 --2- 172.21.15.40:0/1949393751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390815e350 0x7f39080cfaf0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3908067b10 tx=0x7f38f801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53f501daa4ffe50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:10.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.185+0000 7f3906000640 1 -- 172.21.15.40:0/1949393751 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f80b1070 con 0x7f390815e350 2024-03-24T00:37:10.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f3906000640 1 -- 172.21.15.40:0/1949393751 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f80ac5f0 con 0x7f390815e350 2024-03-24T00:37:10.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f3906000640 1 -- 172.21.15.40:0/1949393751 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f80ac9c0 con 0x7f390815e350 2024-03-24T00:37:10.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 -- 172.21.15.40:0/1949393751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390815e350 msgr2=0x7f39080cfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 --2- 172.21.15.40:0/1949393751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390815e350 0x7f39080cfaf0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f3908067b10 tx=0x7f38f801c480 comp rx=0 tx=0).stop 2024-03-24T00:37:10.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 -- 172.21.15.40:0/1949393751 shutdown_connections 2024-03-24T00:37:10.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 --2- 172.21.15.40:0/1949393751 >> v2:172.21.15.53:3300/0 conn(0x7f39080c9580 0x7f39080c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 --2- 172.21.15.40:0/1949393751 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f390815e350 0x7f39080cfaf0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 -- 172.21.15.40:0/1949393751 >> 172.21.15.40:0/1949393751 conn(0x7f3908072870 msgr2=0x7f3908072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:10.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 -- 172.21.15.40:0/1949393751 shutdown_connections 2024-03-24T00:37:10.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 -- 172.21.15.40:0/1949393751 wait complete. 2024-03-24T00:37:10.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 Processor -- start 2024-03-24T00:37:10.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 -- start start 2024-03-24T00:37:10.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39080ad1e0 0x7f39080cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39080c9580 0x7f3908151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f390815e350 0x7f390814e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39080716b0 con 0x7f39080ad1e0 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3908071fe0 con 0x7f390815e350 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f390dc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39080cda30 con 0x7f39080c9580 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f3906a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39080ad1e0 0x7f39080cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f3906a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39080ad1e0 0x7f39080cb750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49378/0 (socket says 172.21.15.40:49378) 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.189+0000 7f3906a00640 1 -- 172.21.15.40:0/1617669704 learned_addr learned my addr 172.21.15.40:0/1617669704 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f38fe000640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39080c9580 0x7f3908151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f3907400640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f390815e350 0x7f390814e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f3906a00640 1 -- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39080c9580 msgr2=0x7f3908151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f3906a00640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39080c9580 0x7f3908151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f3906a00640 1 -- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f390815e350 msgr2=0x7f390814e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f3906a00640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f390815e350 0x7f390814e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f3906a00640 1 -- 172.21.15.40:0/1617669704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39081533d0 con 0x7f39080ad1e0 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f3906a00640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39080ad1e0 0x7f39080cb750 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f38f801ab30 tx=0x7f38f8019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:10.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f38ff400640 1 -- 172.21.15.40:0/1617669704 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f80b9070 con 0x7f39080ad1e0 2024-03-24T00:37:10.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f38ff400640 1 -- 172.21.15.40:0/1617669704 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38f80b5630 con 0x7f39080ad1e0 2024-03-24T00:37:10.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f38ff400640 1 -- 172.21.15.40:0/1617669704 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f80b1040 con 0x7f39080ad1e0 2024-03-24T00:37:10.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f390dc00640 1 -- 172.21.15.40:0/1617669704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39080ca990 con 0x7f39080ad1e0 2024-03-24T00:37:10.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f390dc00640 1 -- 172.21.15.40:0/1617669704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39080caf50 con 0x7f39080ad1e0 2024-03-24T00:37:10.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f390dc00640 1 -- 172.21.15.40:0/1617669704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3908068170 con 0x7f39080ad1e0 2024-03-24T00:37:10.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.193+0000 7f38ff400640 1 -- 172.21.15.40:0/1617669704 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f38f80bc070 con 0x7f39080ad1e0 2024-03-24T00:37:10.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.197+0000 7f38ff400640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f38e4067660 0x7f38e4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.197+0000 7f38ff400640 1 -- 172.21.15.40:0/1617669704 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38f811f4e0 con 0x7f39080ad1e0 2024-03-24T00:37:10.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.197+0000 7f38fe000640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f38e4067660 0x7f38e4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.197+0000 7f38fe000640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f38e4067660 0x7f38e4069b20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3908154080 tx=0x7f38f0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:10.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.201+0000 7f38ff400640 1 -- 172.21.15.40:0/1617669704 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f38f80a9070 con 0x7f39080ad1e0 2024-03-24T00:37:10.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.365+0000 7f390dc00640 1 -- 172.21.15.40:0/1617669704 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_distributed}] v 0) v1 -- 0x7f39080ad660 con 0x7f39080ad1e0 2024-03-24T00:37:10.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.365+0000 7f38ff400640 1 -- 172.21.15.40:0/1617669704 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_distributed}]=0 v53)=0 v53) v1 ==== 133+0+0 (secure 0 0 0) 0x7f38f8016070 con 0x7f39080ad1e0 2024-03-24T00:37:10.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.369+0000 7f390dc00640 1 -- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f38e4067660 msgr2=0x7f38e4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.369+0000 7f390dc00640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f38e4067660 0x7f38e4069b20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f3908154080 tx=0x7f38f0009190 comp rx=0 tx=0).stop 2024-03-24T00:37:10.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.369+0000 7f390dc00640 1 -- 172.21.15.40:0/1617669704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39080ad1e0 msgr2=0x7f39080cb750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.369+0000 7f390dc00640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39080ad1e0 0x7f39080cb750 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f38f801ab30 tx=0x7f38f8019220 comp rx=0 tx=0).stop 2024-03-24T00:37:10.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.369+0000 7f390dc00640 1 -- 172.21.15.40:0/1617669704 shutdown_connections 2024-03-24T00:37:10.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.369+0000 7f390dc00640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f38e4067660 0x7f38e4069b20 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.369+0000 7f390dc00640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f390815e350 0x7f390814e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.369+0000 7f390dc00640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f39080c9580 0x7f3908151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.369+0000 7f390dc00640 1 --2- 172.21.15.40:0/1617669704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f39080ad1e0 0x7f39080cb750 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.369+0000 7f390dc00640 1 -- 172.21.15.40:0/1617669704 >> 172.21.15.40:0/1617669704 conn(0x7f3908072870 msgr2=0x7f3908159880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:10.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.369+0000 7f390dc00640 1 -- 172.21.15.40:0/1617669704 shutdown_connections 2024-03-24T00:37:10.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.369+0000 7f390dc00640 1 -- 172.21.15.40:0/1617669704 wait complete. 2024-03-24T00:37:10.393 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random_max 1.0 2024-03-24T00:37:10.537 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.533+0000 7f8a9d200640 1 Processor -- start 2024-03-24T00:37:10.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.533+0000 7f8a9d200640 1 -- start start 2024-03-24T00:37:10.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.533+0000 7f8a9d200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a9806cb80 0x7f8a980b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.533+0000 7f8a9d200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8a980b4840 0x7f8a98071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.533+0000 7f8a9d200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a980b3260 con 0x7f8a9815ac60 2024-03-24T00:37:10.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.533+0000 7f8a9d200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a980b3e70 con 0x7f8a9806cb80 2024-03-24T00:37:10.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.533+0000 7f8a9d200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a980b3540 con 0x7f8a980b4840 2024-03-24T00:37:10.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.533+0000 7f8a8fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a9806cb80 0x7f8a980b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.533+0000 7f8a95e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8a9815ac60 0x7f8a980b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:60176/0 (socket says 172.21.15.40:60176) 2024-03-24T00:37:10.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.533+0000 7f8a95e00640 1 -- 172.21.15.40:0/277786719 learned_addr learned my addr 172.21.15.40:0/277786719 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:10.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.533+0000 7f8a96800640 1 --2- 172.21.15.40:0/277786719 >> v2:172.21.15.53:3300/0 conn(0x7f8a980b4840 0x7f8a98071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a8fe00640 1 -- 172.21.15.40:0/277786719 >> v2:172.21.15.53:3300/0 conn(0x7f8a980b4840 msgr2=0x7f8a98071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:10.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a8fe00640 1 --2- 172.21.15.40:0/277786719 >> v2:172.21.15.53:3300/0 conn(0x7f8a980b4840 0x7f8a98071b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a8fe00640 1 -- 172.21.15.40:0/277786719 >> v1:172.21.15.40:6789/0 conn(0x7f8a9815ac60 legacy=0x7f8a980b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a8fe00640 1 -- 172.21.15.40:0/277786719 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a980b3820 con 0x7f8a9806cb80 2024-03-24T00:37:10.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a95400640 1 -- 172.21.15.40:0/277786719 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477978265 0 0) 0x7f8a980b3260 con 0x7f8a9815ac60 2024-03-24T00:37:10.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a8fe00640 1 --2- 172.21.15.40:0/277786719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a9806cb80 0x7f8a980b4300 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f8a9806d370 tx=0x7f8a8001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e19bfd680d3894 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:10.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a95400640 1 -- 172.21.15.40:0/277786719 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a800b0070 con 0x7f8a9806cb80 2024-03-24T00:37:10.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a95400640 1 -- 172.21.15.40:0/277786719 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a800ab600 con 0x7f8a9806cb80 2024-03-24T00:37:10.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a95400640 1 -- 172.21.15.40:0/277786719 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a800ab9d0 con 0x7f8a9806cb80 2024-03-24T00:37:10.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a9d200640 1 -- 172.21.15.40:0/277786719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a9806cb80 msgr2=0x7f8a980b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a9d200640 1 --2- 172.21.15.40:0/277786719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a9806cb80 0x7f8a980b4300 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f8a9806d370 tx=0x7f8a8001c3f0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a9d200640 1 -- 172.21.15.40:0/277786719 shutdown_connections 2024-03-24T00:37:10.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a9d200640 1 --2- 172.21.15.40:0/277786719 >> v2:172.21.15.53:3300/0 conn(0x7f8a980b4840 0x7f8a98071b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a9d200640 1 --2- 172.21.15.40:0/277786719 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a9806cb80 0x7f8a980b4300 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a9d200640 1 -- 172.21.15.40:0/277786719 >> 172.21.15.40:0/277786719 conn(0x7f8a980b1660 msgr2=0x7f8a980b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:10.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a9d200640 1 -- 172.21.15.40:0/277786719 shutdown_connections 2024-03-24T00:37:10.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.537+0000 7f8a9d200640 1 -- 172.21.15.40:0/277786719 wait complete. 2024-03-24T00:37:10.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a9d200640 1 Processor -- start 2024-03-24T00:37:10.545 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a9d200640 1 -- start start 2024-03-24T00:37:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a9d200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a9806cb80 0x7f8a98155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a9d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a980b4840 0x7f8a9815bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a9d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a9815ac60 0x7f8a981f7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a9d200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a98167c50 con 0x7f8a9815ac60 2024-03-24T00:37:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a9d200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a98167ad0 con 0x7f8a980b4840 2024-03-24T00:37:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a9d200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a98167dd0 con 0x7f8a9806cb80 2024-03-24T00:37:10.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a95e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a9806cb80 0x7f8a98155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a9815ac60 0x7f8a981f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a95e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a9806cb80 0x7f8a98155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:48868/0 (socket says 172.21.15.40:48868) 2024-03-24T00:37:10.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a96800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a9815ac60 0x7f8a981f7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49392/0 (socket says 172.21.15.40:49392) 2024-03-24T00:37:10.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a95e00640 1 -- 172.21.15.40:0/2030330798 learned_addr learned my addr 172.21.15.40:0/2030330798 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:10.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a8fe00640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a980b4840 0x7f8a9815bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a95e00640 1 -- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a980b4840 msgr2=0x7f8a9815bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a95e00640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a980b4840 0x7f8a9815bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a95e00640 1 -- 172.21.15.40:0/2030330798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a9815ac60 msgr2=0x7f8a981f7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a95e00640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a9815ac60 0x7f8a981f7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a95e00640 1 -- 172.21.15.40:0/2030330798 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a98155880 con 0x7f8a9806cb80 2024-03-24T00:37:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a8fe00640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a980b4840 0x7f8a9815bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a96800640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a9815ac60 0x7f8a981f7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a95e00640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a9806cb80 0x7f8a98155210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800c480 tx=0x7f8a8800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:10.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a8e000640 1 -- 172.21.15.40:0/2030330798 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88018070 con 0x7f8a9806cb80 2024-03-24T00:37:10.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a8e000640 1 -- 172.21.15.40:0/2030330798 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a88007500 con 0x7f8a9806cb80 2024-03-24T00:37:10.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a9d200640 1 -- 172.21.15.40:0/2030330798 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a98155b60 con 0x7f8a9806cb80 2024-03-24T00:37:10.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.541+0000 7f8a8e000640 1 -- 172.21.15.40:0/2030330798 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88004a30 con 0x7f8a9806cb80 2024-03-24T00:37:10.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.545+0000 7f8a9d200640 1 -- 172.21.15.40:0/2030330798 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a98156400 con 0x7f8a9806cb80 2024-03-24T00:37:10.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.545+0000 7f8a9d200640 1 -- 172.21.15.40:0/2030330798 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a98068170 con 0x7f8a9806cb80 2024-03-24T00:37:10.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.545+0000 7f8a8e000640 1 -- 172.21.15.40:0/2030330798 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8a88004bd0 con 0x7f8a9806cb80 2024-03-24T00:37:10.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.545+0000 7f8a8e000640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a74067660 0x7f8a74069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:10.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.545+0000 7f8a8e000640 1 -- 172.21.15.40:0/2030330798 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a88084ae0 con 0x7f8a9806cb80 2024-03-24T00:37:10.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.545+0000 7f8a8fe00640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a74067660 0x7f8a74069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:10.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.545+0000 7f8a8fe00640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a74067660 0x7f8a74069b20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8a800ad000 tx=0x7f8a80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:10.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.557+0000 7f8a8e000640 1 -- 172.21.15.40:0/2030330798 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8a88055120 con 0x7f8a9806cb80 2024-03-24T00:37:10.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.721+0000 7f8a9d200640 1 -- 172.21.15.40:0/2030330798 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random_max}] v 0) v1 -- 0x7f8a980aea30 con 0x7f8a9806cb80 2024-03-24T00:37:10.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.729+0000 7f8a8e000640 1 -- 172.21.15.40:0/2030330798 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random_max}]=0 v54)=0 v54) v1 ==== 131+0+0 (secure 0 0 0) 0x7f8a880591c0 con 0x7f8a9806cb80 2024-03-24T00:37:10.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.729+0000 7f8a9d200640 1 -- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a74067660 msgr2=0x7f8a74069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.729+0000 7f8a9d200640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a74067660 0x7f8a74069b20 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f8a800ad000 tx=0x7f8a80008040 comp rx=0 tx=0).stop 2024-03-24T00:37:10.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.729+0000 7f8a9d200640 1 -- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a9806cb80 msgr2=0x7f8a98155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:10.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.729+0000 7f8a9d200640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a9806cb80 0x7f8a98155210 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800c480 tx=0x7f8a8800c940 comp rx=0 tx=0).stop 2024-03-24T00:37:10.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.729+0000 7f8a9d200640 1 -- 172.21.15.40:0/2030330798 shutdown_connections 2024-03-24T00:37:10.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.729+0000 7f8a9d200640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a74067660 0x7f8a74069b20 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.729+0000 7f8a9d200640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a9815ac60 0x7f8a981f7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.729+0000 7f8a9d200640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a980b4840 0x7f8a9815bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.729+0000 7f8a9d200640 1 --2- 172.21.15.40:0/2030330798 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a9806cb80 0x7f8a98155210 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:10.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.729+0000 7f8a9d200640 1 -- 172.21.15.40:0/2030330798 >> 172.21.15.40:0/2030330798 conn(0x7f8a980b1660 msgr2=0x7f8a980cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:10.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.729+0000 7f8a9d200640 1 -- 172.21.15.40:0/2030330798 shutdown_connections 2024-03-24T00:37:10.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:10.733+0000 7f8a9d200640 1 -- 172.21.15.40:0/2030330798 wait complete. 2024-03-24T00:37:10.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:10.755 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:37:10.755 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:37:10.755 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:37:10.755 DEBUG:teuthology.orchestra.run.smithi040:> # Use up a random number of inode numbers so the ephemeral pinning is not the same every test. 2024-03-24T00:37:10.755 DEBUG:teuthology.orchestra.run.smithi040:> mkdir .inode_number_thrash 2024-03-24T00:37:10.755 DEBUG:teuthology.orchestra.run.smithi040:> count=$((RANDOM % 1024)) 2024-03-24T00:37:10.755 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < count; i++)); do touch .inode_number_thrash/$i; done 2024-03-24T00:37:10.755 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf .inode_number_thrash 2024-03-24T00:37:10.755 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:37:22.592 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-24T00:37:22.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5657400640 1 Processor -- start 2024-03-24T00:37:22.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5657400640 1 -- start start 2024-03-24T00:37:22.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5657400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f565015ac60 0x7f56500b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5657400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f565006cb80 0x7f56500b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5657400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f56500b4840 0x7f56500ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5657400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56500b3e70 con 0x7f565015ac60 2024-03-24T00:37:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5657400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56500b3260 con 0x7f56500b4840 2024-03-24T00:37:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5657400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56500b3540 con 0x7f565006cb80 2024-03-24T00:37:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5654a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f565015ac60 0x7f56500b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5654a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f565015ac60 0x7f56500b2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55768/0 (socket says 172.21.15.40:55768) 2024-03-24T00:37:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5654a00640 1 -- 172.21.15.40:0/2150954618 learned_addr learned my addr 172.21.15.40:0/2150954618 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5647e00640 1 --2- 172.21.15.40:0/2150954618 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f565006cb80 0x7f56500b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:22.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5655400640 1 --2- 172.21.15.40:0/2150954618 >> v2:172.21.15.53:3300/0 conn(0x7f56500b4840 0x7f56500ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:22.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5654a00640 1 -- 172.21.15.40:0/2150954618 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f565006cb80 msgr2=0x7f56500b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:22.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5654a00640 1 --2- 172.21.15.40:0/2150954618 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f565006cb80 0x7f56500b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:22.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5654a00640 1 -- 172.21.15.40:0/2150954618 >> v2:172.21.15.53:3300/0 conn(0x7f56500b4840 msgr2=0x7f56500ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:22.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5654a00640 1 --2- 172.21.15.40:0/2150954618 >> v2:172.21.15.53:3300/0 conn(0x7f56500b4840 0x7f56500ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:22.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.733+0000 7f5654a00640 1 -- 172.21.15.40:0/2150954618 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56500b3820 con 0x7f565015ac60 2024-03-24T00:37:22.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f5654a00640 1 --2- 172.21.15.40:0/2150954618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f565015ac60 0x7f56500b2960 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f56500b2210 tx=0x7f564001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97832bb7eb85e51 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:22.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f564fe00640 1 -- 172.21.15.40:0/2150954618 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56400ab640 con 0x7f565015ac60 2024-03-24T00:37:22.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f564fe00640 1 -- 172.21.15.40:0/2150954618 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56400b0070 con 0x7f565015ac60 2024-03-24T00:37:22.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f564fe00640 1 -- 172.21.15.40:0/2150954618 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56400aba00 con 0x7f565015ac60 2024-03-24T00:37:22.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f5657400640 1 -- 172.21.15.40:0/2150954618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f565015ac60 msgr2=0x7f56500b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:22.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f5657400640 1 --2- 172.21.15.40:0/2150954618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f565015ac60 0x7f56500b2960 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f56500b2210 tx=0x7f564001c460 comp rx=0 tx=0).stop 2024-03-24T00:37:22.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f5657400640 1 -- 172.21.15.40:0/2150954618 shutdown_connections 2024-03-24T00:37:22.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f5657400640 1 --2- 172.21.15.40:0/2150954618 >> v2:172.21.15.53:3300/0 conn(0x7f56500b4840 0x7f56500ccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:22.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f5657400640 1 --2- 172.21.15.40:0/2150954618 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f565006cb80 0x7f56500b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:22.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f5657400640 1 --2- 172.21.15.40:0/2150954618 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f565015ac60 0x7f56500b2960 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:22.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f5657400640 1 -- 172.21.15.40:0/2150954618 >> 172.21.15.40:0/2150954618 conn(0x7f56500b1660 msgr2=0x7f56500b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:22.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f5657400640 1 -- 172.21.15.40:0/2150954618 shutdown_connections 2024-03-24T00:37:22.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.737+0000 7f5657400640 1 -- 172.21.15.40:0/2150954618 wait complete. 2024-03-24T00:37:22.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5657400640 1 Processor -- start 2024-03-24T00:37:22.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5657400640 1 -- start start 2024-03-24T00:37:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5657400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f565006cb80 0x7f5650156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5657400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56500b4840 0x7f565006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5657400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f565015ac60 0x7f56501f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5657400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565015b1c0 con 0x7f565006cb80 2024-03-24T00:37:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5657400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565015b040 con 0x7f565015ac60 2024-03-24T00:37:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5657400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5650071300 con 0x7f56500b4840 2024-03-24T00:37:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5654a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f565006cb80 0x7f5650156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5654a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f565006cb80 0x7f5650156180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55784/0 (socket says 172.21.15.40:55784) 2024-03-24T00:37:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5654a00640 1 -- 172.21.15.40:0/3970968013 learned_addr learned my addr 172.21.15.40:0/3970968013 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:22.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5647e00640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56500b4840 0x7f565006d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5655400640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f565015ac60 0x7f56501f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5654a00640 1 -- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56500b4840 msgr2=0x7f565006d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5654a00640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56500b4840 0x7f565006d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5654a00640 1 -- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f565015ac60 msgr2=0x7f56501f69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5654a00640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f565015ac60 0x7f56501f69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5654a00640 1 -- 172.21.15.40:0/3970968013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565015c8a0 con 0x7f565006cb80 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5655400640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f565015ac60 0x7f56501f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5654a00640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f565006cb80 0x7f5650156180 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f56400ad040 tx=0x7f564001aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f564d600640 1 -- 172.21.15.40:0/3970968013 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56400abdf0 con 0x7f565006cb80 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f564d600640 1 -- 172.21.15.40:0/3970968013 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56400be3f0 con 0x7f565006cb80 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f564d600640 1 -- 172.21.15.40:0/3970968013 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56400be7c0 con 0x7f565006cb80 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5657400640 1 -- 172.21.15.40:0/3970968013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565015cb80 con 0x7f565006cb80 2024-03-24T00:37:22.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5657400640 1 -- 172.21.15.40:0/3970968013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565015d140 con 0x7f565006cb80 2024-03-24T00:37:22.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.741+0000 7f5657400640 1 -- 172.21.15.40:0/3970968013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5650068170 con 0x7f565006cb80 2024-03-24T00:37:22.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.745+0000 7f564d600640 1 -- 172.21.15.40:0/3970968013 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f56400b43f0 con 0x7f565006cb80 2024-03-24T00:37:22.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.745+0000 7f564d600640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f562c067660 0x7f562c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:22.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.745+0000 7f564d600640 1 -- 172.21.15.40:0/3970968013 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f564011fb00 con 0x7f565006cb80 2024-03-24T00:37:22.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.745+0000 7f5647e00640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f562c067660 0x7f562c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:22.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.745+0000 7f5647e00640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f562c067660 0x7f562c069b20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5650165880 tx=0x7f5638009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:22.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.749+0000 7f564d600640 1 -- 172.21.15.40:0/3970968013 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5640016110 con 0x7f565006cb80 2024-03-24T00:37:22.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:22.949+0000 7f5657400640 1 -- 172.21.15.40:0/3970968013 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f56501566c0 con 0x7f565006cb80 2024-03-24T00:37:23.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f564d600640 1 -- 172.21.15.40:0/3970968013 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v98) v1 ==== 105+0+0 (secure 0 0 0) 0x7f56400f0120 con 0x7f565006cb80 2024-03-24T00:37:23.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f5657400640 1 -- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f562c067660 msgr2=0x7f562c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f5657400640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f562c067660 0x7f562c069b20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f5650165880 tx=0x7f5638009190 comp rx=0 tx=0).stop 2024-03-24T00:37:23.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f5657400640 1 -- 172.21.15.40:0/3970968013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f565006cb80 msgr2=0x7f5650156180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f5657400640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f565006cb80 0x7f5650156180 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f56400ad040 tx=0x7f564001aca0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f5657400640 1 -- 172.21.15.40:0/3970968013 shutdown_connections 2024-03-24T00:37:23.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f5657400640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f562c067660 0x7f562c069b20 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f5657400640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f565015ac60 0x7f56501f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f5657400640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f56500b4840 0x7f565006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f5657400640 1 --2- 172.21.15.40:0/3970968013 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f565006cb80 0x7f5650156180 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f5657400640 1 -- 172.21.15.40:0/3970968013 >> 172.21.15.40:0/3970968013 conn(0x7f56500b1660 msgr2=0x7f56500b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:23.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f5657400640 1 -- 172.21.15.40:0/3970968013 shutdown_connections 2024-03-24T00:37:23.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.037+0000 7f5657400640 1 -- 172.21.15.40:0/3970968013 wait complete. 2024-03-24T00:37:23.070 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:23.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e3a00640 1 Processor -- start 2024-03-24T00:37:23.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e3a00640 1 -- start start 2024-03-24T00:37:23.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e3a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42dc06e620 0x7f42dc0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:23.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e3a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f42dc06ea00 0x7f42dc0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:23.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e3a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42dc070f60 con 0x7f42dc0b2ff0 2024-03-24T00:37:23.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e3a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42dc0b4c00 con 0x7f42dc06ea00 2024-03-24T00:37:23.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e3a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42dc0b33d0 con 0x7f42dc06e620 2024-03-24T00:37:23.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e1a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f42dc0b2ff0 0x7f42dc0ccfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37358/0 (socket says 172.21.15.40:37358) 2024-03-24T00:37:23.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e1a00640 1 -- 172.21.15.40:0/1421961658 learned_addr learned my addr 172.21.15.40:0/1421961658 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:23.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42d3e00640 1 --2- 172.21.15.40:0/1421961658 >> v2:172.21.15.40:3300/0 conn(0x7f42dc06ea00 0x7f42dc0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:23.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e1000640 1 --2- 172.21.15.40:0/1421961658 >> v2:172.21.15.53:3300/0 conn(0x7f42dc06e620 0x7f42dc0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:23.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42dbe00640 1 -- 172.21.15.40:0/1421961658 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313139774 0 0) 0x7f42dc070f60 con 0x7f42dc0b2ff0 2024-03-24T00:37:23.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42dbe00640 1 -- 172.21.15.40:0/1421961658 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42c0005740 con 0x7f42dc0b2ff0 2024-03-24T00:37:23.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e1000640 1 -- 172.21.15.40:0/1421961658 >> v2:172.21.15.40:3300/0 conn(0x7f42dc06ea00 msgr2=0x7f42dc0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:23.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e1000640 1 --2- 172.21.15.40:0/1421961658 >> v2:172.21.15.40:3300/0 conn(0x7f42dc06ea00 0x7f42dc0b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e1000640 1 -- 172.21.15.40:0/1421961658 >> v1:172.21.15.40:6789/0 conn(0x7f42dc0b2ff0 legacy=0x7f42dc0ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42e1000640 1 -- 172.21.15.40:0/1421961658 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc0b3620 con 0x7f42dc06e620 2024-03-24T00:37:23.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.209+0000 7f42d3e00640 1 --2- 172.21.15.40:0/1421961658 >> v2:172.21.15.40:3300/0 conn(0x7f42dc06ea00 0x7f42dc0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:23.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42e1000640 1 --2- 172.21.15.40:0/1421961658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42dc06e620 0x7f42dc0b46c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f42dc06c5f0 tx=0x7f42cc01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a20124b26a1e64d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:23.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42dbe00640 1 -- 172.21.15.40:0/1421961658 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0b0070 con 0x7f42dc06e620 2024-03-24T00:37:23.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42dbe00640 1 -- 172.21.15.40:0/1421961658 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc0ab600 con 0x7f42dc06e620 2024-03-24T00:37:23.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42dbe00640 1 -- 172.21.15.40:0/1421961658 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0ab9b0 con 0x7f42dc06e620 2024-03-24T00:37:23.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42e3a00640 1 -- 172.21.15.40:0/1421961658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42dc06e620 msgr2=0x7f42dc0b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42e3a00640 1 --2- 172.21.15.40:0/1421961658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42dc06e620 0x7f42dc0b46c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f42dc06c5f0 tx=0x7f42cc01c3c0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42e3a00640 1 -- 172.21.15.40:0/1421961658 shutdown_connections 2024-03-24T00:37:23.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42e3a00640 1 --2- 172.21.15.40:0/1421961658 >> v2:172.21.15.40:3300/0 conn(0x7f42dc06ea00 0x7f42dc0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42e3a00640 1 --2- 172.21.15.40:0/1421961658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42dc06e620 0x7f42dc0b46c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42e3a00640 1 -- 172.21.15.40:0/1421961658 >> 172.21.15.40:0/1421961658 conn(0x7f42dc0b17b0 msgr2=0x7f42dc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:23.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42e3a00640 1 -- 172.21.15.40:0/1421961658 shutdown_connections 2024-03-24T00:37:23.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42e3a00640 1 -- 172.21.15.40:0/1421961658 wait complete. 2024-03-24T00:37:23.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.213+0000 7f42e3a00640 1 Processor -- start 2024-03-24T00:37:23.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e3a00640 1 -- start start 2024-03-24T00:37:23.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e3a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42dc06e620 0x7f42dc141550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:23.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e3a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42dc06ea00 0x7f42dc0c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e3a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42dc0b2ff0 0x7f42dc1f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e3a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc144920 con 0x7f42dc0b2ff0 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e3a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc0b41a0 con 0x7f42dc06e620 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e3a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42dc0c4a20 con 0x7f42dc06ea00 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42dc0b2ff0 0x7f42dc1f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e1000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42dc06e620 0x7f42dc141550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e1000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42dc06e620 0x7f42dc141550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:47502/0 (socket says 172.21.15.40:47502) 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42dc0b2ff0 0x7f42dc1f6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55816/0 (socket says 172.21.15.40:55816) 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e1000640 1 -- 172.21.15.40:0/1956390677 learned_addr learned my addr 172.21.15.40:0/1956390677 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42d3e00640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42dc06ea00 0x7f42dc0c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e1000640 1 -- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42dc06ea00 msgr2=0x7f42dc0c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e1000640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42dc06ea00 0x7f42dc0c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e1000640 1 -- 172.21.15.40:0/1956390677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42dc0b2ff0 msgr2=0x7f42dc1f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e1000640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42dc0b2ff0 0x7f42dc1f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e1000640 1 -- 172.21.15.40:0/1956390677 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42dc141ea0 con 0x7f42dc06e620 2024-03-24T00:37:23.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e1000640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42dc06e620 0x7f42dc141550 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0ae040 tx=0x7f42cc019160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:23.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42d9600640 1 -- 172.21.15.40:0/1956390677 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0b0040 con 0x7f42dc06e620 2024-03-24T00:37:23.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42d9600640 1 -- 172.21.15.40:0/1956390677 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc0b9070 con 0x7f42dc06e620 2024-03-24T00:37:23.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e3a00640 1 -- 172.21.15.40:0/1956390677 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42dc142180 con 0x7f42dc06e620 2024-03-24T00:37:23.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42d9600640 1 -- 172.21.15.40:0/1956390677 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0b4630 con 0x7f42dc06e620 2024-03-24T00:37:23.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e3a00640 1 -- 172.21.15.40:0/1956390677 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42dc0c4c70 con 0x7f42dc06e620 2024-03-24T00:37:23.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.217+0000 7f42e3a00640 1 -- 172.21.15.40:0/1956390677 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42dc068170 con 0x7f42dc06e620 2024-03-24T00:37:23.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.221+0000 7f42d9600640 1 -- 172.21.15.40:0/1956390677 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f42cc0b47d0 con 0x7f42dc06e620 2024-03-24T00:37:23.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.221+0000 7f42d9600640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f42b8067660 0x7f42b8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:23.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.221+0000 7f42d9600640 1 -- 172.21.15.40:0/1956390677 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42cc1200b0 con 0x7f42dc06e620 2024-03-24T00:37:23.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.221+0000 7f42d3e00640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f42b8067660 0x7f42b8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:23.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.221+0000 7f42d3e00640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f42b8067660 0x7f42b8069b20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009580 tx=0x7f42c4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:23.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.225+0000 7f42d9600640 1 -- 172.21.15.40:0/1956390677 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f42cc016070 con 0x7f42dc06e620 2024-03-24T00:37:23.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.445+0000 7f42e3a00640 1 -- 172.21.15.40:0/1956390677 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42dc072f50 con 0x7f42dc06e620 2024-03-24T00:37:23.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.445+0000 7f42d9600640 1 -- 172.21.15.40:0/1956390677 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+4995 (secure 0 0 0) 0x7f42cc0f0870 con 0x7f42dc06e620 2024-03-24T00:37:23.451 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:23.451 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":99,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6755,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2936601899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2936601899},{"type":"v1","addr":"172.21.15.53:6836","nonce":2936601899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":7633,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/1997904254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1997904254},{"type":"v1","addr":"172.21.15.40:6835","nonce":1997904254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:37:01.566949+0000","modified":"2024-03-24T00:37:23.044140+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":7932,"mds_1":7917},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7917":{"gid":7917,"name":"c","rank":1,"incarnation":99,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6834/1019660586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1019660586},{"type":"v1","addr":"172.21.15.40:6834","nonce":1019660586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7932":{"gid":7932,"name":"d","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/3095501397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3095501397},{"type":"v1","addr":"172.21.15.53:6837","nonce":3095501397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7932]},"id":7}]} 2024-03-24T00:37:23.451 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 99 2024-03-24T00:37:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.449+0000 7f42e3a00640 1 -- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f42b8067660 msgr2=0x7f42b8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.449+0000 7f42e3a00640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f42b8067660 0x7f42b8069b20 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f42c4009580 tx=0x7f42c4009190 comp rx=0 tx=0).stop 2024-03-24T00:37:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.449+0000 7f42e3a00640 1 -- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42dc06e620 msgr2=0x7f42dc141550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.449+0000 7f42e3a00640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42dc06e620 0x7f42dc141550 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0ae040 tx=0x7f42cc019160 comp rx=0 tx=0).stop 2024-03-24T00:37:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.449+0000 7f42e3a00640 1 -- 172.21.15.40:0/1956390677 shutdown_connections 2024-03-24T00:37:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.449+0000 7f42e3a00640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f42b8067660 0x7f42b8069b20 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.449+0000 7f42e3a00640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42dc0b2ff0 0x7f42dc1f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.449+0000 7f42e3a00640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42dc06ea00 0x7f42dc0c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.449+0000 7f42e3a00640 1 --2- 172.21.15.40:0/1956390677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42dc06e620 0x7f42dc141550 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.449+0000 7f42e3a00640 1 -- 172.21.15.40:0/1956390677 >> 172.21.15.40:0/1956390677 conn(0x7f42dc0b17b0 msgr2=0x7f42dc0b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.449+0000 7f42e3a00640 1 -- 172.21.15.40:0/1956390677 shutdown_connections 2024-03-24T00:37:23.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:23.449+0000 7f42e3a00640 1 -- 172.21.15.40:0/1956390677 wait complete. 2024-03-24T00:37:23.874 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 99, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:37:01.566949+0000', 'modified': '2024-03-24T00:37:23.044140+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 7932, 'mds_1': 7917}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7917': {'gid': 7917, 'name': 'c', 'rank': 1, 'incarnation': 99, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6834/1019660586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 1019660586}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 1019660586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7932': {'gid': 7932, 'name': 'd', 'rank': 0, 'incarnation': 93, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6837/3095501397', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6835', 'nonce': 3095501397}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 3095501397}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7932, 'qdb_cluster': [7932]} 2024-03-24T00:37:23.874 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7917:up:creating 2024-03-24T00:37:24.150 INFO:tasks.ceph.mds.d.smithi053.stderr:2024-03-24T00:37:24.147+0000 7fcea3600640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:37:24.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:25.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f36a4c00640 1 Processor -- start 2024-03-24T00:37:25.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f36a4c00640 1 -- start start 2024-03-24T00:37:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f36a4c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f36a00d2d20 0x7f36a00d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f36a4c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a00d0e30 0x7f36a00b6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f36a4c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a00cf5b0 con 0x7f36a015e180 2024-03-24T00:37:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f36a4c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a00d12a0 con 0x7f36a00d2d20 2024-03-24T00:37:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f36a4c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a015e560 con 0x7f36a00d0e30 2024-03-24T00:37:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f369de00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f36a015e180 0x7f36a00d34e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37368/0 (socket says 172.21.15.40:37368) 2024-03-24T00:37:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f369de00640 1 -- 172.21.15.40:0/3553558563 learned_addr learned my addr 172.21.15.40:0/3553558563 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f3697e00640 1 --2- 172.21.15.40:0/3553558563 >> v2:172.21.15.53:3300/0 conn(0x7f36a00d2d20 0x7f36a00d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.018 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f369e800640 1 --2- 172.21.15.40:0/3553558563 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a00d0e30 0x7f36a00b6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f369d400640 1 -- 172.21.15.40:0/3553558563 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808300804 0 0) 0x7f36a00cf5b0 con 0x7f36a015e180 2024-03-24T00:37:25.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f369d400640 1 -- 172.21.15.40:0/3553558563 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3684003530 con 0x7f36a015e180 2024-03-24T00:37:25.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f369e800640 1 -- 172.21.15.40:0/3553558563 >> v2:172.21.15.53:3300/0 conn(0x7f36a00d2d20 msgr2=0x7f36a00d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:25.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f369e800640 1 --2- 172.21.15.40:0/3553558563 >> v2:172.21.15.53:3300/0 conn(0x7f36a00d2d20 0x7f36a00d08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f369e800640 1 -- 172.21.15.40:0/3553558563 >> v1:172.21.15.40:6789/0 conn(0x7f36a015e180 legacy=0x7f36a00d34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f369e800640 1 -- 172.21.15.40:0/3553558563 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a00d1c80 con 0x7f36a00d0e30 2024-03-24T00:37:25.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.013+0000 7f3697e00640 1 --2- 172.21.15.40:0/3553558563 >> v2:172.21.15.53:3300/0 conn(0x7f36a00d2d20 0x7f36a00d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:25.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f369e800640 1 --2- 172.21.15.40:0/3553558563 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a00d0e30 0x7f36a00b6fc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3698009d90 tx=0x7f369801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=800af4921e53b716 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:25.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f369d400640 1 -- 172.21.15.40:0/3553558563 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36980b2070 con 0x7f36a00d0e30 2024-03-24T00:37:25.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f369d400640 1 -- 172.21.15.40:0/3553558563 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36980a5920 con 0x7f36a00d0e30 2024-03-24T00:37:25.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f369d400640 1 -- 172.21.15.40:0/3553558563 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36980a5cf0 con 0x7f36a00d0e30 2024-03-24T00:37:25.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 -- 172.21.15.40:0/3553558563 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a00d0e30 msgr2=0x7f36a00b6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 --2- 172.21.15.40:0/3553558563 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a00d0e30 0x7f36a00b6fc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3698009d90 tx=0x7f369801c970 comp rx=0 tx=0).stop 2024-03-24T00:37:25.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 -- 172.21.15.40:0/3553558563 shutdown_connections 2024-03-24T00:37:25.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 --2- 172.21.15.40:0/3553558563 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a00d0e30 0x7f36a00b6fc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 --2- 172.21.15.40:0/3553558563 >> v2:172.21.15.53:3300/0 conn(0x7f36a00d2d20 0x7f36a00d08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 -- 172.21.15.40:0/3553558563 >> 172.21.15.40:0/3553558563 conn(0x7f36a0066e80 msgr2=0x7f36a0067290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 -- 172.21.15.40:0/3553558563 shutdown_connections 2024-03-24T00:37:25.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 -- 172.21.15.40:0/3553558563 wait complete. 2024-03-24T00:37:25.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 Processor -- start 2024-03-24T00:37:25.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 -- start start 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a00d0e30 0x7f36a00c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a00d2d20 0x7f36a00cca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a015e180 0x7f36a014c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a00d3bf0 con 0x7f36a00d2d20 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a00d2800 con 0x7f36a015e180 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f36a4c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a00d3d70 con 0x7f36a00d0e30 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f3697e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a00d2d20 0x7f36a00cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f369de00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a00d0e30 0x7f36a00c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.021+0000 7f3697e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a00d2d20 0x7f36a00cca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55818/0 (socket says 172.21.15.40:55818) 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f3697e00640 1 -- 172.21.15.40:0/1337615460 learned_addr learned my addr 172.21.15.40:0/1337615460 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f369e800640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a015e180 0x7f36a014c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f3697e00640 1 -- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a00d0e30 msgr2=0x7f36a00c6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f3697e00640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a00d0e30 0x7f36a00c6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f3697e00640 1 -- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a015e180 msgr2=0x7f36a014c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f3697e00640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a015e180 0x7f36a014c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f3697e00640 1 -- 172.21.15.40:0/1337615460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a00cd1b0 con 0x7f36a00d2d20 2024-03-24T00:37:25.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f3697e00640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a00d2d20 0x7f36a00cca60 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f368800ec10 tx=0x7f368800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:25.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f3696000640 1 -- 172.21.15.40:0/1337615460 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688017070 con 0x7f36a00d2d20 2024-03-24T00:37:25.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f3696000640 1 -- 172.21.15.40:0/1337615460 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368800cc30 con 0x7f36a00d2d20 2024-03-24T00:37:25.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f3696000640 1 -- 172.21.15.40:0/1337615460 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688013470 con 0x7f36a00d2d20 2024-03-24T00:37:25.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f36a4c00640 1 -- 172.21.15.40:0/1337615460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a00cd490 con 0x7f36a00d2d20 2024-03-24T00:37:25.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f36a4c00640 1 -- 172.21.15.40:0/1337615460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a00cb480 con 0x7f36a00d2d20 2024-03-24T00:37:25.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f36a4c00640 1 -- 172.21.15.40:0/1337615460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a0068170 con 0x7f36a00d2d20 2024-03-24T00:37:25.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f3696000640 1 -- 172.21.15.40:0/1337615460 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f368801a070 con 0x7f36a00d2d20 2024-03-24T00:37:25.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.025+0000 7f3696000640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f367c067660 0x7f367c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.029+0000 7f3696000640 1 -- 172.21.15.40:0/1337615460 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3688085b50 con 0x7f36a00d2d20 2024-03-24T00:37:25.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.029+0000 7f369de00640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f367c067660 0x7f367c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.029+0000 7f369de00640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f367c067660 0x7f367c069b20 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3690007fd0 tx=0x7f3690009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:25.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.033+0000 7f3696000640 1 -- 172.21.15.40:0/1337615460 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3688056170 con 0x7f36a00d2d20 2024-03-24T00:37:25.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 -- 172.21.15.40:0/1337615460 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36a015be40 con 0x7f36a00d2d20 2024-03-24T00:37:25.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f3696000640 1 -- 172.21.15.40:0/1337615460 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5020 (secure 0 0 0) 0x7f368805a210 con 0x7f36a00d2d20 2024-03-24T00:37:25.245 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:25.245 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6755,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2936601899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2936601899},{"type":"v1","addr":"172.21.15.53:6836","nonce":2936601899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:37:01.566949+0000","modified":"2024-03-24T00:37:25.054299+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7932,"mds_1":7917,"mds_2":7633},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7633":{"gid":7633,"name":"a","rank":2,"incarnation":101,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6835/1997904254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1997904254},{"type":"v1","addr":"172.21.15.40:6835","nonce":1997904254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7917":{"gid":7917,"name":"c","rank":1,"incarnation":99,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6834/1019660586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1019660586},{"type":"v1","addr":"172.21.15.40:6834","nonce":1019660586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7932":{"gid":7932,"name":"d","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/3095501397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3095501397},{"type":"v1","addr":"172.21.15.53:6837","nonce":3095501397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7633,7932,7917]},"id":7}]} 2024-03-24T00:37:25.245 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 102 2024-03-24T00:37:25.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 -- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f367c067660 msgr2=0x7f367c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f367c067660 0x7f367c069b20 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f3690007fd0 tx=0x7f3690009040 comp rx=0 tx=0).stop 2024-03-24T00:37:25.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 -- 172.21.15.40:0/1337615460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a00d2d20 msgr2=0x7f36a00cca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a00d2d20 0x7f36a00cca60 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f368800ec10 tx=0x7f368800c520 comp rx=0 tx=0).stop 2024-03-24T00:37:25.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 -- 172.21.15.40:0/1337615460 shutdown_connections 2024-03-24T00:37:25.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f367c067660 0x7f367c069b20 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a015e180 0x7f36a014c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f36a00d2d20 0x7f36a00cca60 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 --2- 172.21.15.40:0/1337615460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f36a00d0e30 0x7f36a00c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 -- 172.21.15.40:0/1337615460 >> 172.21.15.40:0/1337615460 conn(0x7f36a0066e80 msgr2=0x7f36a00b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:25.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 -- 172.21.15.40:0/1337615460 shutdown_connections 2024-03-24T00:37:25.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.241+0000 7f36a4c00640 1 -- 172.21.15.40:0/1337615460 wait complete. 2024-03-24T00:37:25.677 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:37:01.566949+0000', 'modified': '2024-03-24T00:37:25.054299+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 7932, 'mds_1': 7917, 'mds_2': 7633}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7633': {'gid': 7633, 'name': 'a', 'rank': 2, 'incarnation': 101, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.40:6835/1997904254', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 1997904254}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 1997904254}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7917': {'gid': 7917, 'name': 'c', 'rank': 1, 'incarnation': 99, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.40:6834/1019660586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 1019660586}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 1019660586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_7932': {'gid': 7932, 'name': 'd', 'rank': 0, 'incarnation': 93, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6837/3095501397', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6835', 'nonce': 3095501397}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 3095501397}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 7932, 'qdb_cluster': [7633, 7932, 7917]} 2024-03-24T00:37:25.677 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T00:37:25.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:37:25.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.813+0000 7fa2a4600640 1 Processor -- start 2024-03-24T00:37:25.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.813+0000 7fa2a4600640 1 -- start start 2024-03-24T00:37:25.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a4600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa29c13ec80 0x7fa29c1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a4600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa29c1419f0 0x7fa29c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a4600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa29c0ce630 0x7fa29c15f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a4600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa29c141e50 con 0x7fa29c13ec80 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a4600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa29c070e00 con 0x7fa29c1419f0 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a4600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c0cee20 con 0x7fa29c0ce630 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a1c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa29c13ec80 0x7fa29c1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a1c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa29c13ec80 0x7fa29c1414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55834/0 (socket says 172.21.15.40:55834) 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a1c00640 1 -- 172.21.15.40:0/1761855997 learned_addr learned my addr 172.21.15.40:0/1761855997 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a1200640 1 --2- 172.21.15.40:0/1761855997 >> v2:172.21.15.53:3300/0 conn(0x7fa29c1419f0 0x7fa29c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a2600640 1 --2- 172.21.15.40:0/1761855997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa29c0ce630 0x7fa29c15f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a1c00640 1 -- 172.21.15.40:0/1761855997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa29c0ce630 msgr2=0x7fa29c15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/1761855997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa29c0ce630 0x7fa29c15f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a1c00640 1 -- 172.21.15.40:0/1761855997 >> v2:172.21.15.53:3300/0 conn(0x7fa29c1419f0 msgr2=0x7fa29c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:25.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/1761855997 >> v2:172.21.15.53:3300/0 conn(0x7fa29c1419f0 0x7fa29c0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.817+0000 7fa2a1c00640 1 -- 172.21.15.40:0/1761855997 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c0cf100 con 0x7fa29c13ec80 2024-03-24T00:37:25.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/1761855997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa29c13ec80 0x7fa29c1414b0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fa29c04fa60 tx=0x7fa29001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3223cbcd4fa3c8a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:25.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa29be00640 1 -- 172.21.15.40:0/1761855997 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900b0070 con 0x7fa29c13ec80 2024-03-24T00:37:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa29be00640 1 -- 172.21.15.40:0/1761855997 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2900ab600 con 0x7fa29c13ec80 2024-03-24T00:37:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa29be00640 1 -- 172.21.15.40:0/1761855997 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900ab9d0 con 0x7fa29c13ec80 2024-03-24T00:37:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 -- 172.21.15.40:0/1761855997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa29c13ec80 msgr2=0x7fa29c1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 --2- 172.21.15.40:0/1761855997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa29c13ec80 0x7fa29c1414b0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fa29c04fa60 tx=0x7fa29001c260 comp rx=0 tx=0).stop 2024-03-24T00:37:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 -- 172.21.15.40:0/1761855997 shutdown_connections 2024-03-24T00:37:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 --2- 172.21.15.40:0/1761855997 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa29c0ce630 0x7fa29c15f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 --2- 172.21.15.40:0/1761855997 >> v2:172.21.15.53:3300/0 conn(0x7fa29c1419f0 0x7fa29c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 --2- 172.21.15.40:0/1761855997 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa29c13ec80 0x7fa29c1414b0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 -- 172.21.15.40:0/1761855997 >> 172.21.15.40:0/1761855997 conn(0x7fa29c0b17b0 msgr2=0x7fa29c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 -- 172.21.15.40:0/1761855997 shutdown_connections 2024-03-24T00:37:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 -- 172.21.15.40:0/1761855997 wait complete. 2024-03-24T00:37:25.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 Processor -- start 2024-03-24T00:37:25.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 -- start start 2024-03-24T00:37:25.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa29c0ce630 0x7fa29c1580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa29c13ec80 0x7fa29c141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.821+0000 7fa2a4600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa29c1419f0 0x7fa29c15b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a4600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c071480 con 0x7fa29c0ce630 2024-03-24T00:37:25.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a4600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c071300 con 0x7fa29c1419f0 2024-03-24T00:37:25.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a4600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c071600 con 0x7fa29c13ec80 2024-03-24T00:37:25.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa29c0ce630 0x7fa29c1580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa29c0ce630 0x7fa29c1580b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55846/0 (socket says 172.21.15.40:55846) 2024-03-24T00:37:25.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a1c00640 1 -- 172.21.15.40:0/16364969 learned_addr learned my addr 172.21.15.40:0/16364969 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:25.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a2600640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa29c1419f0 0x7fa29c15b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a1200640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa29c13ec80 0x7fa29c141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a1c00640 1 -- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa29c13ec80 msgr2=0x7fa29c141010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa29c13ec80 0x7fa29c141010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a1c00640 1 -- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa29c1419f0 msgr2=0x7fa29c15b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa29c1419f0 0x7fa29c15b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a1c00640 1 -- 172.21.15.40:0/16364969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c156ba0 con 0x7fa29c0ce630 2024-03-24T00:37:25.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a1200640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa29c13ec80 0x7fa29c141010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:25.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa29c0ce630 0x7fa29c1580b0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fa29001cb00 tx=0x7fa29001aaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:25.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa299600640 1 -- 172.21.15.40:0/16364969 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900ba020 con 0x7fa29c0ce630 2024-03-24T00:37:25.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa299600640 1 -- 172.21.15.40:0/16364969 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2900b6420 con 0x7fa29c0ce630 2024-03-24T00:37:25.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa299600640 1 -- 172.21.15.40:0/16364969 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900b0040 con 0x7fa29c0ce630 2024-03-24T00:37:25.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a4600640 1 -- 172.21.15.40:0/16364969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29c156e80 con 0x7fa29c0ce630 2024-03-24T00:37:25.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.825+0000 7fa2a4600640 1 -- 172.21.15.40:0/16364969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29c13d410 con 0x7fa29c0ce630 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa299600640 1 -- 172.21.15.40:0/16364969 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa2900bd020 con 0x7fa29c0ce630 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 Processor -- start 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 -- start start 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa299600640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa27c067660 0x7fa27c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa299600640 1 -- 172.21.15.40:0/16364969 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa29011f800 con 0x7fa29c0ce630 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d7c920 0x560255d7cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d7d240 0x560255d32ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d32ff0 0x560255d333d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560255b80680 con 0x560255d32ff0 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x560255b07b00 con 0x560255d7c920 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x560255b7c720 con 0x560255d7d240 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a2600640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa27c067660 0x7fa27c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a1c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d7c920 0x560255d7cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a1c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d7c920 0x560255d7cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:47560/0 (socket says 172.21.15.40:47560) 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a1c00640 1 -- 172.21.15.40:0/3350411968 learned_addr learned my addr 172.21.15.40:0/3350411968 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a2600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d32ff0 0x560255d333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a1200640 1 --2- 172.21.15.40:0/3350411968 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d7d240 0x560255d32ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a1c00640 1 -- 172.21.15.40:0/3350411968 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d7d240 msgr2=0x560255d32ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/3350411968 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d7d240 0x560255d32ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a1c00640 1 -- 172.21.15.40:0/3350411968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d32ff0 msgr2=0x560255d333d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/3350411968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d32ff0 0x560255d333d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a1c00640 1 -- 172.21.15.40:0/3350411968 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560255b58940 con 0x560255d7c920 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/3350411968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d7c920 0x560255d7cd00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa2900ec390 tx=0x7fa290009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67dc0fefa1c5dc82 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a2600640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa27c067660 0x7fa27c069b20 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa294008d70 tx=0x7fa2940083f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa28e000640 1 -- 172.21.15.40:0/3350411968 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29011d070 con 0x560255d7c920 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa28e000640 1 -- 172.21.15.40:0/3350411968 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2900eff30 con 0x560255d7c920 2024-03-24T00:37:25.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa28e000640 1 -- 172.21.15.40:0/3350411968 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900f4080 con 0x560255d7c920 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 -- 172.21.15.40:0/3350411968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d7c920 msgr2=0x560255d7cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/3350411968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d7c920 0x560255d7cd00 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa2900ec390 tx=0x7fa290009240 comp rx=0 tx=0).stop 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 -- 172.21.15.40:0/3350411968 shutdown_connections 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/3350411968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d32ff0 0x560255d333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/3350411968 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d7d240 0x560255d32ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/3350411968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d7c920 0x560255d7cd00 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 -- 172.21.15.40:0/3350411968 >> 172.21.15.40:0/3350411968 conn(0x560255d2efa0 msgr2=0x560255d31390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 -- 172.21.15.40:0/3350411968 shutdown_connections 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.829+0000 7fa2a7a17000 1 -- 172.21.15.40:0/3350411968 wait complete. 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 1 -- start start 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 1 Processor -- start 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d32ff0 0x560255d61d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d7c920 0x560255d62280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d7d240 0x560255d627c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560255b7f990 con 0x560255d32ff0 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x560255b3aae0 con 0x560255d7c920 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x560255b18750 con 0x560255d7d240 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 10 client.0 _ll_register_callbacks cb 0x560255d1dff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d32ff0 0x560255d61d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d32ff0 0x560255d61d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55872/0 (socket says 172.21.15.40:55872) 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a1c00640 1 -- 172.21.15.40:0/1168905569 learned_addr learned my addr 172.21.15.40:0/1168905569 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a1200640 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d7c920 0x560255d62280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a2600640 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d7d240 0x560255d627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a1c00640 1 -- 172.21.15.40:0/1168905569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d7d240 msgr2=0x560255d627c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d7d240 0x560255d627c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a1c00640 1 -- 172.21.15.40:0/1168905569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d7c920 msgr2=0x560255d62280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:25.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d7c920 0x560255d62280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a1c00640 1 -- 172.21.15.40:0/1168905569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560255bc5d80 con 0x560255d32ff0 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d32ff0 0x560255d61d40 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fa2900efec0 tx=0x7fa290019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ea3c5d869c155e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa282a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa282a00640 1 -- 172.21.15.40:0/1168905569 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29011d070 con 0x560255d32ff0 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa282a00640 1 -- 172.21.15.40:0/1168905569 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2900f83b0 con 0x560255d32ff0 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1168905569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2900eff30 con 0x560255d32ff0 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa282a00640 1 -- 172.21.15.40:0/1168905569 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2900f4080 con 0x560255d32ff0 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 10 client.8145 fetch_fsmap learned FSMap version 102 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1168905569 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x560255b07d10 con 0x560255d32ff0 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa282a00640 1 -- 172.21.15.40:0/1168905569 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2900eff30 con 0x560255d32ff0 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa282a00640 1 -- 172.21.15.40:0/1168905569 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 102) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fa2900b3030 con 0x560255d32ff0 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 10 client.8145 fetch_fsmap finished waiting for FSMap version 102 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 10 client.8145 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:37:25.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 20 client.8145 populate_metadata read hostname 'smithi040' 2024-03-24T00:37:25.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.40:6833/1997904254,v1:172.21.15.40:6835/1997904254] conn(0x560255d63060 0x560255d8eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 4 client.8145 mds_command: new command op to 7633 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:37:25.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1168905569 --> [v2:172.21.15.40:6833/1997904254,v1:172.21.15.40:6835/1997904254] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560255b1bb60 con 0x560255d63060 2024-03-24T00:37:25.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.833+0000 7fa2a1200640 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.40:6833/1997904254,v1:172.21.15.40:6835/1997904254] conn(0x560255d63060 0x560255d8eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a1200640 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.40:6833/1997904254,v1:172.21.15.40:6835/1997904254] conn(0x560255d63060 0x560255d8eea0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c1e1545f595257d2 server_cookie=da9ab7f45cddf60 in_seq=0 out_seq=0 2024-03-24T00:37:25.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa282a00640 10 client.8145 ms_handle_connect on v2:172.21.15.40:6833/1997904254 2024-03-24T00:37:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa282a00640 1 -- 172.21.15.40:0/1168905569 <== mds.2 v2:172.21.15.40:6833/1997904254 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560255b1bb60 con 0x560255d63060 2024-03-24T00:37:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa282a00640 10 client.8145 handle_command_reply: tid=0 2024-03-24T00:37:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a7a17000 1 client.8145 shutdown 2024-03-24T00:37:25.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1168905569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d32ff0 msgr2=0x560255d61d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d32ff0 0x560255d61d40 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fa2900efec0 tx=0x7fa290019000 comp rx=0 tx=0).stop 2024-03-24T00:37:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1168905569 shutdown_connections 2024-03-24T00:37:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.40:6833/1997904254,v1:172.21.15.40:6835/1997904254] conn(0x560255d63060 0x560255d8eea0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d7d240 0x560255d627c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d7c920 0x560255d62280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/1168905569 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d32ff0 0x560255d61d40 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1168905569 >> 172.21.15.40:0/1168905569 conn(0x560255d2efa0 msgr2=0x560255d31c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa282a00640 0 client.8145 ms_handle_reset on v2:172.21.15.40:6833/1997904254 2024-03-24T00:37:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1168905569 shutdown_connections 2024-03-24T00:37:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1168905569 wait complete. 2024-03-24T00:37:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.837+0000 7fa2a7a17000 20 client.8145 trim_cache size 0 max 16384 2024-03-24T00:37:25.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a7a17000 1 Processor -- start 2024-03-24T00:37:25.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a7a17000 1 -- start start 2024-03-24T00:37:25.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a7a17000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d61ed0 0x560255d60100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a7a17000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d2efa0 0x560255d7b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a7a17000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d32ff0 0x560255d7b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a7a17000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c202bd0 con 0x560255d61ed0 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a7a17000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c15cd40 con 0x560255d32ff0 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a7a17000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x560255b28a80 con 0x560255d2efa0 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d61ed0 0x560255d60100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d61ed0 0x560255d60100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55884/0 (socket says 172.21.15.40:55884) 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a1c00640 1 -- 172.21.15.40:0/2862785603 learned_addr learned my addr 172.21.15.40:0/2862785603 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a1200640 1 --2- 172.21.15.40:0/2862785603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d2efa0 0x560255d7b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a2600640 1 --2- 172.21.15.40:0/2862785603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d32ff0 0x560255d7b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a1c00640 1 -- 172.21.15.40:0/2862785603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d2efa0 msgr2=0x560255d7b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/2862785603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d2efa0 0x560255d7b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.853+0000 7fa2a1c00640 1 -- 172.21.15.40:0/2862785603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d32ff0 msgr2=0x560255d7b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:25.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/2862785603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d32ff0 0x560255d7b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a1c00640 1 -- 172.21.15.40:0/2862785603 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560255d37210 con 0x560255d61ed0 2024-03-24T00:37:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/2862785603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d61ed0 0x560255d60100 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fa2900ec8b0 tx=0x7fa29011d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f849afa97dd62e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa28d600640 1 -- 172.21.15.40:0/2862785603 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900eff30 con 0x560255d61ed0 2024-03-24T00:37:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa28d600640 1 -- 172.21.15.40:0/2862785603 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2900ff070 con 0x560255d61ed0 2024-03-24T00:37:25.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa28d600640 1 -- 172.21.15.40:0/2862785603 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2900f83b0 con 0x560255d61ed0 2024-03-24T00:37:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 -- 172.21.15.40:0/2862785603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d61ed0 msgr2=0x560255d60100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/2862785603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d61ed0 0x560255d60100 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fa2900ec8b0 tx=0x7fa29011d000 comp rx=0 tx=0).stop 2024-03-24T00:37:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 -- 172.21.15.40:0/2862785603 shutdown_connections 2024-03-24T00:37:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/2862785603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d32ff0 0x560255d7b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/2862785603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d2efa0 0x560255d7b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/2862785603 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d61ed0 0x560255d60100 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 -- 172.21.15.40:0/2862785603 >> 172.21.15.40:0/2862785603 conn(0x560255d35750 msgr2=0x560255d5f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 -- 172.21.15.40:0/2862785603 shutdown_connections 2024-03-24T00:37:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 -- 172.21.15.40:0/2862785603 wait complete. 2024-03-24T00:37:25.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 -- start start 2024-03-24T00:37:25.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 Processor -- start 2024-03-24T00:37:25.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d2efa0 0x560255de21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d32ff0 0x560255de4700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d61ed0 0x560255de6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x560255b18750 con 0x560255d61ed0 2024-03-24T00:37:25.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x560255c3d700 con 0x560255d2efa0 2024-03-24T00:37:25.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x560255b7f990 con 0x560255d32ff0 2024-03-24T00:37:25.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a7a17000 10 client.0 _ll_register_callbacks cb 0x7fa29011b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:25.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a2600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d61ed0 0x560255de6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a1200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d32ff0 0x560255de4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a1c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d2efa0 0x560255de21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a1200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d32ff0 0x560255de4700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:59096/0 (socket says 172.21.15.40:59096) 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a1200640 1 -- 172.21.15.40:0/1376279814 learned_addr learned my addr 172.21.15.40:0/1376279814 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a1200640 1 -- 172.21.15.40:0/1376279814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d2efa0 msgr2=0x560255de21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a1200640 1 --2- 172.21.15.40:0/1376279814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d2efa0 0x560255de21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a1200640 1 -- 172.21.15.40:0/1376279814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d61ed0 msgr2=0x560255de6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a1200640 1 --2- 172.21.15.40:0/1376279814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d61ed0 0x560255de6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.857+0000 7fa2a1200640 1 -- 172.21.15.40:0/1376279814 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560255b4d540 con 0x560255d32ff0 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a1200640 1 --2- 172.21.15.40:0/1376279814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d32ff0 0x560255de4700 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa288009fa0 tx=0x7fa28801f960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56d7f7889e19c6a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa282a00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa282a00640 1 -- 172.21.15.40:0/1376279814 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa288028440 con 0x560255d32ff0 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa282a00640 1 -- 172.21.15.40:0/1376279814 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa288029c90 con 0x560255d32ff0 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1376279814 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2900ff070 con 0x560255d32ff0 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa282a00640 1 -- 172.21.15.40:0/1376279814 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa288016430 con 0x560255d32ff0 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 10 client.6908 fetch_fsmap learned FSMap version 102 2024-03-24T00:37:25.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1376279814 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x560255b144b0 con 0x560255d32ff0 2024-03-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa282a00640 1 -- 172.21.15.40:0/1376279814 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2900ff070 con 0x560255d32ff0 2024-03-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa282a00640 1 -- 172.21.15.40:0/1376279814 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 102) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fa288030050 con 0x560255d32ff0 2024-03-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 10 client.6908 fetch_fsmap finished waiting for FSMap version 102 2024-03-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 10 client.6908 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 20 client.6908 populate_metadata read hostname 'smithi040' 2024-03-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/1376279814 >> [v2:172.21.15.40:6833/1997904254,v1:172.21.15.40:6835/1997904254] conn(0x560255d7c920 0x560255de93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 4 client.6908 mds_command: new command op to 7633 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1376279814 --> [v2:172.21.15.40:6833/1997904254,v1:172.21.15.40:6835/1997904254] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560255b8dbc0 con 0x560255d7c920 2024-03-24T00:37:25.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/1376279814 >> [v2:172.21.15.40:6833/1997904254,v1:172.21.15.40:6835/1997904254] conn(0x560255d7c920 0x560255de93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa282a00640 10 client.6908 ms_handle_connect on v2:172.21.15.40:6833/1997904254 2024-03-24T00:37:25.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a1c00640 1 --2- 172.21.15.40:0/1376279814 >> [v2:172.21.15.40:6833/1997904254,v1:172.21.15.40:6835/1997904254] conn(0x560255d7c920 0x560255de93b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=94eea9b2add60b0e server_cookie=868364b51e8610d7 in_seq=0 out_seq=0 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa282a00640 1 -- 172.21.15.40:0/1376279814 <== mds.2 v2:172.21.15.40:6833/1997904254 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x560255b8dbc0 con 0x560255d7c920 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa282a00640 10 client.6908 handle_command_reply: tid=0 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 client.6908 shutdown 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1376279814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d32ff0 msgr2=0x560255de4700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/1376279814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d32ff0 0x560255de4700 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa288009fa0 tx=0x7fa28801f960 comp rx=0 tx=0).stop 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1376279814 shutdown_connections 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/1376279814 >> [v2:172.21.15.40:6833/1997904254,v1:172.21.15.40:6835/1997904254] conn(0x560255d7c920 0x560255de93b0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/1376279814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x560255d61ed0 0x560255de6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/1376279814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560255d32ff0 0x560255de4700 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa282a00640 0 client.6908 ms_handle_reset on v2:172.21.15.40:6833/1997904254 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 --2- 172.21.15.40:0/1376279814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560255d2efa0 0x560255de21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1376279814 >> 172.21.15.40:0/1376279814 conn(0x560255d35750 msgr2=0x560255d61190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1376279814 shutdown_connections 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 1 -- 172.21.15.40:0/1376279814 wait complete. 2024-03-24T00:37:25.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a7a17000 20 client.6908 trim_cache size 0 max 16384 2024-03-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-03-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 7633, 2024-03-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 1.812192875, 2024-03-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 102, 2024-03-24T00:37:25.870 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 95, 2024-03-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 95, 2024-03-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 28.533649452999999 2024-03-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a4600640 1 -- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa27c067660 msgr2=0x7fa27c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a4600640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa27c067660 0x7fa27c069b20 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa294008d70 tx=0x7fa2940083f0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a4600640 1 -- 172.21.15.40:0/16364969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa29c0ce630 msgr2=0x7fa29c1580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a4600640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa29c0ce630 0x7fa29c1580b0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fa29001cb00 tx=0x7fa29001aaa0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a4600640 1 -- 172.21.15.40:0/16364969 shutdown_connections 2024-03-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a4600640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa27c067660 0x7fa27c069b20 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a4600640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa29c1419f0 0x7fa29c15b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a4600640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa29c13ec80 0x7fa29c141010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a4600640 1 --2- 172.21.15.40:0/16364969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa29c0ce630 0x7fa29c1580b0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a4600640 1 -- 172.21.15.40:0/16364969 >> 172.21.15.40:0/16364969 conn(0x7fa29c0b17b0 msgr2=0x7fa29c0719e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.861+0000 7fa2a4600640 1 -- 172.21.15.40:0/16364969 shutdown_connections 2024-03-24T00:37:25.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:25.865+0000 7fa2a4600640 1 -- 172.21.15.40:0/16364969 wait complete. 2024-03-24T00:37:25.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:37:26.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.029+0000 7f98cea00640 1 Processor -- start 2024-03-24T00:37:26.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.029+0000 7f98cea00640 1 -- start start 2024-03-24T00:37:26.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.029+0000 7f98cea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f98c813ec80 0x7f98c81414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.029+0000 7f98cea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f98c80ce800 0x7f98c8071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.029+0000 7f98cea00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98c8070e00 con 0x7f98c81419f0 2024-03-24T00:37:26.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.029+0000 7f98cea00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c8141e50 con 0x7f98c813ec80 2024-03-24T00:37:26.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.029+0000 7f98cea00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98c80cee20 con 0x7f98c80ce800 2024-03-24T00:37:26.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.033+0000 7f98c7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f98c813ec80 0x7f98c81414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.033+0000 7f98cca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f98c80ce800 0x7f98c8071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.033+0000 7f98c7400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f98c81419f0 0x7f98c80ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37376/0 (socket says 172.21.15.40:37376) 2024-03-24T00:37:26.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.033+0000 7f98c7400640 1 -- 172.21.15.40:0/1978366822 learned_addr learned my addr 172.21.15.40:0/1978366822 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:26.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.033+0000 7f98cca00640 1 -- 172.21.15.40:0/1978366822 >> v2:172.21.15.40:3300/0 conn(0x7f98c813ec80 msgr2=0x7f98c81414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.033+0000 7f98cca00640 1 --2- 172.21.15.40:0/1978366822 >> v2:172.21.15.40:3300/0 conn(0x7f98c813ec80 0x7f98c81414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.033+0000 7f98cca00640 1 -- 172.21.15.40:0/1978366822 >> v1:172.21.15.40:6789/0 conn(0x7f98c81419f0 legacy=0x7f98c80ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.033+0000 7f98cca00640 1 -- 172.21.15.40:0/1978366822 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c80cf100 con 0x7f98c80ce800 2024-03-24T00:37:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.033+0000 7f98cca00640 1 --2- 172.21.15.40:0/1978366822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c80ce800 0x7f98c8071300 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f98c8067e60 tx=0x7f98c001c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e320799345ad408 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.033+0000 7f98c6a00640 1 -- 172.21.15.40:0/1978366822 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c00b2070 con 0x7f98c80ce800 2024-03-24T00:37:26.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.033+0000 7f98c6a00640 1 -- 172.21.15.40:0/1978366822 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c00a5a10 con 0x7f98c80ce800 2024-03-24T00:37:26.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.033+0000 7f98c6a00640 1 -- 172.21.15.40:0/1978366822 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c00a5de0 con 0x7f98c80ce800 2024-03-24T00:37:26.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 -- 172.21.15.40:0/1978366822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c80ce800 msgr2=0x7f98c8071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 --2- 172.21.15.40:0/1978366822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c80ce800 0x7f98c8071300 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f98c8067e60 tx=0x7f98c001c930 comp rx=0 tx=0).stop 2024-03-24T00:37:26.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 -- 172.21.15.40:0/1978366822 shutdown_connections 2024-03-24T00:37:26.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 --2- 172.21.15.40:0/1978366822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c80ce800 0x7f98c8071300 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 --2- 172.21.15.40:0/1978366822 >> v2:172.21.15.40:3300/0 conn(0x7f98c813ec80 0x7f98c81414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 -- 172.21.15.40:0/1978366822 >> 172.21.15.40:0/1978366822 conn(0x7f98c80b17b0 msgr2=0x7f98c80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:26.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 -- 172.21.15.40:0/1978366822 shutdown_connections 2024-03-24T00:37:26.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 -- 172.21.15.40:0/1978366822 wait complete. 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 Processor -- start 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 -- start start 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98c80ce800 0x7f98c813dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98c813ec80 0x7f98c806d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c81419f0 0x7f98c815abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c80723e0 con 0x7f98c813ec80 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c8072260 con 0x7f98c81419f0 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cea00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98c8072560 con 0x7f98c80ce800 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98c7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98c813ec80 0x7f98c806d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98c7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98c80ce800 0x7f98c813dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98c7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98c80ce800 0x7f98c813dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:59100/0 (socket says 172.21.15.40:59100) 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98c7e00640 1 -- 172.21.15.40:0/957913657 learned_addr learned my addr 172.21.15.40:0/957913657 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98cca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c81419f0 0x7f98c815abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98c7e00640 1 -- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c81419f0 msgr2=0x7f98c815abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98c7e00640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c81419f0 0x7f98c815abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98c7e00640 1 -- 172.21.15.40:0/957913657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98c813ec80 msgr2=0x7f98c806d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98c7e00640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98c813ec80 0x7f98c806d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98c7e00640 1 -- 172.21.15.40:0/957913657 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c813e020 con 0x7f98c80ce800 2024-03-24T00:37:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.037+0000 7f98c7400640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98c813ec80 0x7f98c806d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.041+0000 7f98c7e00640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98c80ce800 0x7f98c813dae0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f98b800e930 tx=0x7f98b800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.041+0000 7f98bfe00640 1 -- 172.21.15.40:0/957913657 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8017070 con 0x7f98c80ce800 2024-03-24T00:37:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.041+0000 7f98bfe00640 1 -- 172.21.15.40:0/957913657 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b800ca00 con 0x7f98c80ce800 2024-03-24T00:37:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.041+0000 7f98bfe00640 1 -- 172.21.15.40:0/957913657 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b800cdb0 con 0x7f98c80ce800 2024-03-24T00:37:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.041+0000 7f98cea00640 1 -- 172.21.15.40:0/957913657 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98c81550f0 con 0x7f98c80ce800 2024-03-24T00:37:26.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.041+0000 7f98cea00640 1 -- 172.21.15.40:0/957913657 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98c81556b0 con 0x7f98c80ce800 2024-03-24T00:37:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.041+0000 7f98bfe00640 1 -- 172.21.15.40:0/957913657 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f98b801a070 con 0x7f98c80ce800 2024-03-24T00:37:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.041+0000 7f98d1ee3000 1 Processor -- start 2024-03-24T00:37:26.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 -- start start 2024-03-24T00:37:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98bfe00640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f98a8067660 0x7f98a8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98bfe00640 1 -- 172.21.15.40:0/957913657 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98b8085860 con 0x7f98c80ce800 2024-03-24T00:37:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98cca00640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f98a8067660 0x7f98a8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 0x558bb7bb6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 0x558bb7b6cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b6d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bb79ba680 con 0x558bb7bb6920 2024-03-24T00:37:26.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bb7941b00 con 0x558bb7bb7240 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x558bb79b6720 con 0x558bb7b6cff0 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98c7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 0x558bb7bb6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98c7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 0x558bb7bb6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55904/0 (socket says 172.21.15.40:55904) 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98c7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 0x558bb7b6cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98c7e00640 1 -- 172.21.15.40:0/2998846074 learned_addr learned my addr 172.21.15.40:0/2998846074 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98cca00640 1 --2- 172.21.15.40:0/2998846074 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b6d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98c7e00640 1 -- 172.21.15.40:0/2998846074 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 msgr2=0x558bb7b6d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98c7e00640 1 --2- 172.21.15.40:0/2998846074 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b6d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98c7e00640 1 -- 172.21.15.40:0/2998846074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 msgr2=0x558bb7b6cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98c7e00640 1 --2- 172.21.15.40:0/2998846074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 0x558bb7b6cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98c7e00640 1 -- 172.21.15.40:0/2998846074 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558bb7992940 con 0x558bb7bb6920 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98cca00640 1 --2- 172.21.15.40:0/2998846074 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b6d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98cca00640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f98a8067660 0x7f98a8069b20 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f98c00af000 tx=0x7f98c00a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98c7e00640 1 --2- 172.21.15.40:0/2998846074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 0x558bb7bb6d00 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009310 tx=0x7f98b80560c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810c38890e78c284 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98bcc00640 1 -- 172.21.15.40:0/2998846074 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8021040 con 0x558bb7bb6920 2024-03-24T00:37:26.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98bcc00640 1 -- 172.21.15.40:0/2998846074 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b80525b0 con 0x558bb7bb6920 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98bcc00640 1 -- 172.21.15.40:0/2998846074 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8055390 con 0x558bb7bb6920 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2998846074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 msgr2=0x558bb7bb6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/2998846074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 0x558bb7bb6d00 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009310 tx=0x7f98b80560c0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2998846074 shutdown_connections 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/2998846074 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b6d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/2998846074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 0x558bb7b6cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/2998846074 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 0x558bb7bb6d00 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2998846074 >> 172.21.15.40:0/2998846074 conn(0x558bb7b68fa0 msgr2=0x558bb7b6b390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2998846074 shutdown_connections 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2998846074 wait complete. 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.045+0000 7f98d1ee3000 1 -- start start 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 1 Processor -- start 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b9bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb6920 0x558bb7b9c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb7240 0x558bb7b9c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bb79b9990 con 0x558bb7bb7240 2024-03-24T00:37:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bb7974ae0 con 0x558bb7bb6920 2024-03-24T00:37:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x558bb7952750 con 0x558bb7b6cff0 2024-03-24T00:37:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 10 client.0 _ll_register_callbacks cb 0x558bb7b57ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b9bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98cca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb7240 0x558bb7b9c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb6920 0x558bb7b9c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b9bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:59128/0 (socket says 172.21.15.40:59128) 2024-03-24T00:37:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7e00640 1 -- 172.21.15.40:0/3535664134 learned_addr learned my addr 172.21.15.40:0/3535664134 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:26.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb6920 0x558bb7b9c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:47642/0 (socket says 172.21.15.40:47642) 2024-03-24T00:37:26.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7e00640 1 -- 172.21.15.40:0/3535664134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb6920 msgr2=0x558bb7b9c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7e00640 1 --2- 172.21.15.40:0/3535664134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb6920 0x558bb7b9c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7e00640 1 -- 172.21.15.40:0/3535664134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb7240 msgr2=0x558bb7b9c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7e00640 1 --2- 172.21.15.40:0/3535664134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb7240 0x558bb7b9c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7e00640 1 -- 172.21.15.40:0/3535664134 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558bb79ffd80 con 0x558bb7b6cff0 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98cca00640 1 --2- 172.21.15.40:0/3535664134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb7240 0x558bb7b9c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7e00640 1 --2- 172.21.15.40:0/3535664134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b9bd40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f98b80541f0 tx=0x7f98b80092a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c74068418dfb044e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98b1600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98b1600640 1 -- 172.21.15.40:0/3535664134 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8071070 con 0x558bb7b6cff0 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98b1600640 1 -- 172.21.15.40:0/3535664134 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98b8075c50 con 0x558bb7b6cff0 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 1 -- 172.21.15.40:0/3535664134 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f98b80525b0 con 0x558bb7b6cff0 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98b1600640 1 -- 172.21.15.40:0/3535664134 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98b8021070 con 0x558bb7b6cff0 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 10 client.6914 fetch_fsmap learned FSMap version 102 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 1 -- 172.21.15.40:0/3535664134 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x558bb7941d10 con 0x558bb7b6cff0 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98b1600640 1 -- 172.21.15.40:0/3535664134 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b80525b0 con 0x558bb7b6cff0 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98b1600640 1 -- 172.21.15.40:0/3535664134 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 102) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f98b8024060 con 0x558bb7b6cff0 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 10 client.6914 fetch_fsmap finished waiting for FSMap version 102 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 10 client.6914 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 20 client.6914 populate_metadata read hostname 'smithi040' 2024-03-24T00:37:26.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/3535664134 >> [v2:172.21.15.40:6832/1019660586,v1:172.21.15.40:6834/1019660586] conn(0x558bb7b9d090 0x558bb7bc8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 4 client.6914 mds_command: new command op to 7917 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:37:26.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98d1ee3000 1 -- 172.21.15.40:0/3535664134 --> [v2:172.21.15.40:6832/1019660586,v1:172.21.15.40:6834/1019660586] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558bb7955b60 con 0x558bb7b9d090 2024-03-24T00:37:26.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7400640 1 --2- 172.21.15.40:0/3535664134 >> [v2:172.21.15.40:6832/1019660586,v1:172.21.15.40:6834/1019660586] conn(0x558bb7b9d090 0x558bb7bc8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98c7400640 1 --2- 172.21.15.40:0/3535664134 >> [v2:172.21.15.40:6832/1019660586,v1:172.21.15.40:6834/1019660586] conn(0x558bb7b9d090 0x558bb7bc8ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d74100e3fc425b31 server_cookie=9376609233a544a7 in_seq=0 out_seq=0 2024-03-24T00:37:26.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.049+0000 7f98b1600640 10 client.6914 ms_handle_connect on v2:172.21.15.40:6832/1019660586 2024-03-24T00:37:26.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98b1600640 1 -- 172.21.15.40:0/3535664134 <== mds.1 v2:172.21.15.40:6832/1019660586 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x558bb7955b60 con 0x558bb7b9d090 2024-03-24T00:37:26.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98b1600640 10 client.6914 handle_command_reply: tid=0 2024-03-24T00:37:26.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98d1ee3000 1 client.6914 shutdown 2024-03-24T00:37:26.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98d1ee3000 1 -- 172.21.15.40:0/3535664134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 msgr2=0x558bb7b9bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/3535664134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b9bd40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f98b80541f0 tx=0x7f98b80092a0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98d1ee3000 1 -- 172.21.15.40:0/3535664134 shutdown_connections 2024-03-24T00:37:26.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/3535664134 >> [v2:172.21.15.40:6832/1019660586,v1:172.21.15.40:6834/1019660586] conn(0x558bb7b9d090 0x558bb7bc8ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/3535664134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb7240 0x558bb7b9c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98b1600640 0 client.6914 ms_handle_reset on v2:172.21.15.40:6832/1019660586 2024-03-24T00:37:26.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/3535664134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb6920 0x558bb7b9c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/3535664134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b9bd40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98d1ee3000 1 -- 172.21.15.40:0/3535664134 >> 172.21.15.40:0/3535664134 conn(0x558bb7b68fa0 msgr2=0x558bb7b6bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:26.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98d1ee3000 1 -- 172.21.15.40:0/3535664134 shutdown_connections 2024-03-24T00:37:26.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98d1ee3000 1 -- 172.21.15.40:0/3535664134 wait complete. 2024-03-24T00:37:26.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.053+0000 7f98d1ee3000 20 client.6914 trim_cache size 0 max 16384 2024-03-24T00:37:26.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.069+0000 7f98d1ee3000 1 Processor -- start 2024-03-24T00:37:26.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.069+0000 7f98d1ee3000 1 -- start start 2024-03-24T00:37:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b99ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 0x558bb7bb5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 0x558bb7bb5a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c8202aa0 con 0x558bb7bb6920 2024-03-24T00:37:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98c8201630 con 0x558bb7bb7240 2024-03-24T00:37:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x558bb7962a80 con 0x558bb7b6cff0 2024-03-24T00:37:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98c7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 0x558bb7bb5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98c7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 0x558bb7bb5530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55922/0 (socket says 172.21.15.40:55922) 2024-03-24T00:37:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98c7400640 1 -- 172.21.15.40:0/4032281020 learned_addr learned my addr 172.21.15.40:0/4032281020 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98cca00640 1 --2- 172.21.15.40:0/4032281020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 0x558bb7bb5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98c7e00640 1 --2- 172.21.15.40:0/4032281020 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b99ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98c7400640 1 -- 172.21.15.40:0/4032281020 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 msgr2=0x558bb7b99ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98c7400640 1 --2- 172.21.15.40:0/4032281020 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b99ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98c7400640 1 -- 172.21.15.40:0/4032281020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 msgr2=0x558bb7bb5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98c7400640 1 --2- 172.21.15.40:0/4032281020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 0x558bb7bb5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98c7400640 1 -- 172.21.15.40:0/4032281020 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558bb7987540 con 0x558bb7bb6920 2024-03-24T00:37:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98c7400640 1 --2- 172.21.15.40:0/4032281020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 0x558bb7bb5530 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x558bb79c87b0 tx=0x7f98b4006040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3626df310b735202 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98b3e00640 1 -- 172.21.15.40:0/4032281020 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b401d1d0 con 0x558bb7bb6920 2024-03-24T00:37:26.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98b3e00640 1 -- 172.21.15.40:0/4032281020 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b4023070 con 0x558bb7bb6920 2024-03-24T00:37:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98b3e00640 1 -- 172.21.15.40:0/4032281020 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b4001c30 con 0x558bb7bb6920 2024-03-24T00:37:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 -- 172.21.15.40:0/4032281020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 msgr2=0x558bb7bb5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/4032281020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 0x558bb7bb5530 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x558bb79c87b0 tx=0x7f98b4006040 comp rx=0 tx=0).stop 2024-03-24T00:37:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 -- 172.21.15.40:0/4032281020 shutdown_connections 2024-03-24T00:37:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/4032281020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 0x558bb7bb5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/4032281020 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7bb6920 0x558bb7bb5530 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/4032281020 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7b6cff0 0x558bb7b99ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 -- 172.21.15.40:0/4032281020 >> 172.21.15.40:0/4032281020 conn(0x558bb7b68fa0 msgr2=0x558bb7b995d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 -- 172.21.15.40:0/4032281020 shutdown_connections 2024-03-24T00:37:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 -- 172.21.15.40:0/4032281020 wait complete. 2024-03-24T00:37:26.079 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 -- start start 2024-03-24T00:37:26.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.073+0000 7f98d1ee3000 1 Processor -- start 2024-03-24T00:37:26.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7b6cff0 0x558bb7bb5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7bb6920 0x558bb7bc9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 0x558bb7bca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bb7974ae0 con 0x558bb7b6cff0 2024-03-24T00:37:26.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bb7a77700 con 0x558bb7bb7240 2024-03-24T00:37:26.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x558bb7952750 con 0x558bb7bb6920 2024-03-24T00:37:26.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 10 client.0 _ll_register_callbacks cb 0x7f98b40048f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:26.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98c7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7b6cff0 0x558bb7bb5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98c7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7b6cff0 0x558bb7bb5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55926/0 (socket says 172.21.15.40:55926) 2024-03-24T00:37:26.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98cca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 0x558bb7bca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98c7e00640 1 -- 172.21.15.40:0/2691172055 learned_addr learned my addr 172.21.15.40:0/2691172055 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:26.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98c7400640 1 --2- 172.21.15.40:0/2691172055 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7bb6920 0x558bb7bc9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98c7e00640 1 -- 172.21.15.40:0/2691172055 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7bb6920 msgr2=0x558bb7bc9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98c7e00640 1 --2- 172.21.15.40:0/2691172055 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7bb6920 0x558bb7bc9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98c7e00640 1 -- 172.21.15.40:0/2691172055 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 msgr2=0x558bb7bca430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98c7e00640 1 --2- 172.21.15.40:0/2691172055 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 0x558bb7bca430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98c7e00640 1 -- 172.21.15.40:0/2691172055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558bb794e4b0 con 0x558bb7b6cff0 2024-03-24T00:37:26.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98c7e00640 1 --2- 172.21.15.40:0/2691172055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7b6cff0 0x558bb7bb5c50 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f98b805c440 tx=0x7f98b8052260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea458fd4d0b9135d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98b1600640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:37:26.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98b1600640 1 -- 172.21.15.40:0/2691172055 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b8071030 con 0x558bb7b6cff0 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98b1600640 1 -- 172.21.15.40:0/2691172055 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98b80525b0 con 0x558bb7b6cff0 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2691172055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f98b4023070 con 0x558bb7b6cff0 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98b1600640 1 -- 172.21.15.40:0/2691172055 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98b8051050 con 0x558bb7b6cff0 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 10 client.8172 fetch_fsmap learned FSMap version 102 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2691172055 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x558bb79612f0 con 0x558bb7b6cff0 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98b1600640 1 -- 172.21.15.40:0/2691172055 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98b4023070 con 0x558bb7b6cff0 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98b1600640 1 -- 172.21.15.40:0/2691172055 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 102) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f98b8054e00 con 0x558bb7b6cff0 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 10 client.8172 fetch_fsmap finished waiting for FSMap version 102 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 10 client.8172 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 20 client.8172 populate_metadata read hostname 'smithi040' 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/2691172055 >> [v2:172.21.15.40:6832/1019660586,v1:172.21.15.40:6834/1019660586] conn(0x558bb7bca970 0x558bb7c1b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 4 client.8172 mds_command: new command op to 7917 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2691172055 --> [v2:172.21.15.40:6832/1019660586,v1:172.21.15.40:6834/1019660586] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558bb79c7bc0 con 0x558bb7bca970 2024-03-24T00:37:26.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.077+0000 7f98c7400640 1 --2- 172.21.15.40:0/2691172055 >> [v2:172.21.15.40:6832/1019660586,v1:172.21.15.40:6834/1019660586] conn(0x558bb7bca970 0x558bb7c1b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98c7400640 1 --2- 172.21.15.40:0/2691172055 >> [v2:172.21.15.40:6832/1019660586,v1:172.21.15.40:6834/1019660586] conn(0x558bb7bca970 0x558bb7c1b7d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e7765a3cf8670c31 server_cookie=42e309f403ba07bd in_seq=0 out_seq=0 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98b1600640 10 client.8172 ms_handle_connect on v2:172.21.15.40:6832/1019660586 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98b1600640 1 -- 172.21.15.40:0/2691172055 <== mds.1 v2:172.21.15.40:6832/1019660586 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x558bb79c7bc0 con 0x558bb7bca970 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98b1600640 10 client.8172 handle_command_reply: tid=0 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98d1ee3000 1 client.8172 shutdown 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2691172055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7b6cff0 msgr2=0x558bb7bb5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/2691172055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7b6cff0 0x558bb7bb5c50 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f98b805c440 tx=0x7f98b8052260 comp rx=0 tx=0).stop 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2691172055 shutdown_connections 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/2691172055 >> [v2:172.21.15.40:6832/1019660586,v1:172.21.15.40:6834/1019660586] conn(0x558bb7bca970 0x558bb7c1b7d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/2691172055 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558bb7bb7240 0x558bb7bca430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/2691172055 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x558bb7bb6920 0x558bb7bc9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98d1ee3000 1 --2- 172.21.15.40:0/2691172055 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x558bb7b6cff0 0x558bb7bb5c50 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2691172055 >> 172.21.15.40:0/2691172055 conn(0x558bb7b68fa0 msgr2=0x558bb7b99e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98b1600640 0 client.8172 ms_handle_reset on v2:172.21.15.40:6832/1019660586 2024-03-24T00:37:26.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2691172055 shutdown_connections 2024-03-24T00:37:26.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98d1ee3000 1 -- 172.21.15.40:0/2691172055 wait complete. 2024-03-24T00:37:26.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98d1ee3000 20 client.8172 trim_cache size 0 max 16384 2024-03-24T00:37:26.087 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:37:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:37:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-03-24T00:37:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 7917, 2024-03-24T00:37:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:37:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:37:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:37:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.0388251479999999, 2024-03-24T00:37:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 102, 2024-03-24T00:37:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 95, 2024-03-24T00:37:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 95, 2024-03-24T00:37:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 28.751672356 2024-03-24T00:37:26.088 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:37:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98cea00640 1 -- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f98a8067660 msgr2=0x7f98a8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98cea00640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f98a8067660 0x7f98a8069b20 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f98c00af000 tx=0x7f98c00a7070 comp rx=0 tx=0).stop 2024-03-24T00:37:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98cea00640 1 -- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98c80ce800 msgr2=0x7f98c813dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98cea00640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98c80ce800 0x7f98c813dae0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f98b800e930 tx=0x7f98b800edf0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98cea00640 1 -- 172.21.15.40:0/957913657 shutdown_connections 2024-03-24T00:37:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98cea00640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f98a8067660 0x7f98a8069b20 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98cea00640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98c81419f0 0x7f98c815abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98cea00640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f98c813ec80 0x7f98c806d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98cea00640 1 --2- 172.21.15.40:0/957913657 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98c80ce800 0x7f98c813dae0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98cea00640 1 -- 172.21.15.40:0/957913657 >> 172.21.15.40:0/957913657 conn(0x7f98c80b17b0 msgr2=0x7f98c813fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98cea00640 1 -- 172.21.15.40:0/957913657 shutdown_connections 2024-03-24T00:37:26.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.081+0000 7f98cea00640 1 -- 172.21.15.40:0/957913657 wait complete. 2024-03-24T00:37:26.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T00:37:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.245+0000 7f68d8800640 1 Processor -- start 2024-03-24T00:37:26.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.245+0000 7f68d8800640 1 -- start start 2024-03-24T00:37:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.249+0000 7f68d8800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f68d0072310 0x7f68d00ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.249+0000 7f68d8800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d0070e00 con 0x7f68d013ec80 2024-03-24T00:37:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.249+0000 7f68d8800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d00cee20 con 0x7f68d0141bc0 2024-03-24T00:37:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.249+0000 7f68d8800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d00cf100 con 0x7f68d0072310 2024-03-24T00:37:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.249+0000 7f68d6800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f68d0072310 0x7f68d00ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.249+0000 7f68d5e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f68d013ec80 0x7f68d01414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:37378/0 (socket says 172.21.15.40:37378) 2024-03-24T00:37:26.252 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.249+0000 7f68d5e00640 1 -- 172.21.15.40:0/2259558870 learned_addr learned my addr 172.21.15.40:0/2259558870 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.249+0000 7f68d4a00640 1 -- 172.21.15.40:0/2259558870 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391384298 0 0) 0x7f68d0070e00 con 0x7f68d013ec80 2024-03-24T00:37:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.249+0000 7f68d6800640 1 -- 172.21.15.40:0/2259558870 >> v1:172.21.15.53:6789/0 conn(0x7f68d0141bc0 legacy=0x7f68d00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.249+0000 7f68d6800640 1 -- 172.21.15.40:0/2259558870 >> v1:172.21.15.40:6789/0 conn(0x7f68d013ec80 legacy=0x7f68d01414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.249+0000 7f68d6800640 1 -- 172.21.15.40:0/2259558870 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d00cf3e0 con 0x7f68d0072310 2024-03-24T00:37:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.253+0000 7f68d6800640 1 --2- 172.21.15.40:0/2259558870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d0072310 0x7f68d00ce800 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009d90 tx=0x7f68cc01c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3682ee16c1e5b06 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.253+0000 7f68d4a00640 1 -- 172.21.15.40:0/2259558870 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0b2070 con 0x7f68d0072310 2024-03-24T00:37:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.253+0000 7f68d4a00640 1 -- 172.21.15.40:0/2259558870 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68cc0a5880 con 0x7f68d0072310 2024-03-24T00:37:26.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.253+0000 7f68d4a00640 1 -- 172.21.15.40:0/2259558870 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68cc0a5c50 con 0x7f68d0072310 2024-03-24T00:37:26.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.253+0000 7f68d8800640 1 -- 172.21.15.40:0/2259558870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d0072310 msgr2=0x7f68d00ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.253+0000 7f68d8800640 1 --2- 172.21.15.40:0/2259558870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d0072310 0x7f68d00ce800 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f68cc009d90 tx=0x7f68cc01c820 comp rx=0 tx=0).stop 2024-03-24T00:37:26.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.253+0000 7f68d8800640 1 -- 172.21.15.40:0/2259558870 shutdown_connections 2024-03-24T00:37:26.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.253+0000 7f68d8800640 1 --2- 172.21.15.40:0/2259558870 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d0072310 0x7f68d00ce800 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.253+0000 7f68d8800640 1 -- 172.21.15.40:0/2259558870 >> 172.21.15.40:0/2259558870 conn(0x7f68d00b17b0 msgr2=0x7f68d00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:26.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.253+0000 7f68d8800640 1 -- 172.21.15.40:0/2259558870 shutdown_connections 2024-03-24T00:37:26.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.253+0000 7f68d8800640 1 -- 172.21.15.40:0/2259558870 wait complete. 2024-03-24T00:37:26.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d8800640 1 Processor -- start 2024-03-24T00:37:26.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d8800640 1 -- start start 2024-03-24T00:37:26.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d8800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68d0072310 0x7f68d00ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d8800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d013ec80 0x7f68d0139de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d8800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68d0141bc0 0x7f68d00b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d8800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d0169f00 con 0x7f68d013ec80 2024-03-24T00:37:26.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d8800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d0169d80 con 0x7f68d0072310 2024-03-24T00:37:26.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d8800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d016a080 con 0x7f68d0141bc0 2024-03-24T00:37:26.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d013ec80 0x7f68d0139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68d0072310 0x7f68d00ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d6800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68d0141bc0 0x7f68d00b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d013ec80 0x7f68d0139de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55942/0 (socket says 172.21.15.40:55942) 2024-03-24T00:37:26.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d5400640 1 -- 172.21.15.40:0/4025518946 learned_addr learned my addr 172.21.15.40:0/4025518946 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:26.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d5400640 1 -- 172.21.15.40:0/4025518946 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68d0141bc0 msgr2=0x7f68d00b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d5400640 1 --2- 172.21.15.40:0/4025518946 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68d0141bc0 0x7f68d00b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d5400640 1 -- 172.21.15.40:0/4025518946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68d0072310 msgr2=0x7f68d00ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d5400640 1 --2- 172.21.15.40:0/4025518946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68d0072310 0x7f68d00ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d5400640 1 -- 172.21.15.40:0/4025518946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d00ae250 con 0x7f68d013ec80 2024-03-24T00:37:26.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d5400640 1 --2- 172.21.15.40:0/4025518946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d013ec80 0x7f68d0139de0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f68c000c750 tx=0x7f68c000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68ca000640 1 -- 172.21.15.40:0/4025518946 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0017070 con 0x7f68d013ec80 2024-03-24T00:37:26.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68ca000640 1 -- 172.21.15.40:0/4025518946 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c0005270 con 0x7f68d013ec80 2024-03-24T00:37:26.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68ca000640 1 -- 172.21.15.40:0/4025518946 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c000a640 con 0x7f68d013ec80 2024-03-24T00:37:26.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d8800640 1 -- 172.21.15.40:0/4025518946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d00ae530 con 0x7f68d013ec80 2024-03-24T00:37:26.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.257+0000 7f68d8800640 1 -- 172.21.15.40:0/4025518946 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d00aeaf0 con 0x7f68d013ec80 2024-03-24T00:37:26.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.261+0000 7f68ca000640 1 -- 172.21.15.40:0/4025518946 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f68c001a070 con 0x7f68d013ec80 2024-03-24T00:37:26.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.261+0000 7f68dbda8000 1 Processor -- start 2024-03-24T00:37:26.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.261+0000 7f68ca000640 1 --2- 172.21.15.40:0/4025518946 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68b4067660 0x7f68b4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.261+0000 7f68dbda8000 1 -- start start 2024-03-24T00:37:26.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.261+0000 7f68ca000640 1 -- 172.21.15.40:0/4025518946 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68c0084cf0 con 0x7f68d013ec80 2024-03-24T00:37:26.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.261+0000 7f68d6800640 1 --2- 172.21.15.40:0/4025518946 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68b4067660 0x7f68b4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 0x55804feb0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d6800640 1 --2- 172.21.15.40:0/4025518946 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68b4067660 0x7f68b4069b20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0af000 tx=0x7f68cc0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 0x55804fe66ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804fe673d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55804fcb4680 con 0x55804feb1240 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55804fc3bb00 con 0x55804feb0920 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55804fcb0720 con 0x55804fe66ff0 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 0x55804fe66ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 0x55804feb0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 0x55804fe66ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55948/0 (socket says 172.21.15.40:55948) 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d5400640 1 -- 172.21.15.40:0/346281970 learned_addr learned my addr 172.21.15.40:0/346281970 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d6800640 1 --2- 172.21.15.40:0/346281970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804fe673d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d5400640 1 -- 172.21.15.40:0/346281970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 msgr2=0x55804fe673d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d5400640 1 --2- 172.21.15.40:0/346281970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804fe673d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d5400640 1 -- 172.21.15.40:0/346281970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 msgr2=0x55804feb0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d5400640 1 --2- 172.21.15.40:0/346281970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 0x55804feb0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d5400640 1 -- 172.21.15.40:0/346281970 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55804fc8c940 con 0x55804feb1240 2024-03-24T00:37:26.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d6800640 1 --2- 172.21.15.40:0/346281970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804fe673d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:26.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68d5400640 1 --2- 172.21.15.40:0/346281970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 0x55804fe66ab0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f68c0051770 tx=0x7f68c00593b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768d8c73b3cbb762 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68bea00640 1 -- 172.21.15.40:0/346281970 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c006d070 con 0x55804feb1240 2024-03-24T00:37:26.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68bea00640 1 -- 172.21.15.40:0/346281970 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c0059170 con 0x55804feb1240 2024-03-24T00:37:26.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68bea00640 1 -- 172.21.15.40:0/346281970 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0051980 con 0x55804feb1240 2024-03-24T00:37:26.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 -- 172.21.15.40:0/346281970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 msgr2=0x55804fe66ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 --2- 172.21.15.40:0/346281970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 0x55804fe66ab0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f68c0051770 tx=0x7f68c00593b0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 -- 172.21.15.40:0/346281970 shutdown_connections 2024-03-24T00:37:26.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 --2- 172.21.15.40:0/346281970 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804fe673d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 --2- 172.21.15.40:0/346281970 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 0x55804fe66ab0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 --2- 172.21.15.40:0/346281970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 0x55804feb0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 -- 172.21.15.40:0/346281970 >> 172.21.15.40:0/346281970 conn(0x55804fe62fa0 msgr2=0x55804fe65390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:26.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 -- 172.21.15.40:0/346281970 shutdown_connections 2024-03-24T00:37:26.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 -- 172.21.15.40:0/346281970 wait complete. 2024-03-24T00:37:26.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 -- start start 2024-03-24T00:37:26.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.265+0000 7f68dbda8000 1 Processor -- start 2024-03-24T00:37:26.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804fe95d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb0920 0x55804fe96280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb1240 0x55804fe967c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55804fcb3990 con 0x55804feb0920 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55804fc6eae0 con 0x55804feb1240 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55804fc4c750 con 0x55804fe66ff0 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 10 client.0 _ll_register_callbacks cb 0x55804fe51ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb0920 0x55804fe96280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d5e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804fe95d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb0920 0x55804fe96280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55962/0 (socket says 172.21.15.40:55962) 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d5400640 1 -- 172.21.15.40:0/1480285491 learned_addr learned my addr 172.21.15.40:0/1480285491 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d6800640 1 --2- 172.21.15.40:0/1480285491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb1240 0x55804fe967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d5400640 1 -- 172.21.15.40:0/1480285491 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 msgr2=0x55804fe95d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d5400640 1 --2- 172.21.15.40:0/1480285491 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804fe95d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d5400640 1 -- 172.21.15.40:0/1480285491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb1240 msgr2=0x55804fe967c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d5400640 1 --2- 172.21.15.40:0/1480285491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb1240 0x55804fe967c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d5400640 1 -- 172.21.15.40:0/1480285491 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55804fcf9d80 con 0x55804feb0920 2024-03-24T00:37:26.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d5400640 1 --2- 172.21.15.40:0/1480285491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb0920 0x55804fe96280 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f68c0054fc0 tx=0x7f68c0001220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdd44eef03da304 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68b3400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68b3400640 1 -- 172.21.15.40:0/1480285491 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0080070 con 0x55804feb0920 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68b3400640 1 -- 172.21.15.40:0/1480285491 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68c005d6f0 con 0x55804feb0920 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 1 -- 172.21.15.40:0/1480285491 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f68c0059170 con 0x55804feb0920 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68b3400640 1 -- 172.21.15.40:0/1480285491 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68c006d040 con 0x55804feb0920 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 10 client.8187 fetch_fsmap learned FSMap version 102 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 1 -- 172.21.15.40:0/1480285491 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x55804fc3bd10 con 0x55804feb0920 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68b3400640 1 -- 172.21.15.40:0/1480285491 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c0059170 con 0x55804feb0920 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68b3400640 1 -- 172.21.15.40:0/1480285491 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 102) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f68c007cb70 con 0x55804feb0920 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 10 client.8187 fetch_fsmap finished waiting for FSMap version 102 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 10 client.8187 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 20 client.8187 populate_metadata read hostname 'smithi040' 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1480285491 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x55804fe97030 0x55804fec2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 4 client.8187 mds_command: new command op to 7932 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68dbda8000 1 -- 172.21.15.40:0/1480285491 --> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55804fc4fb60 con 0x55804fe97030 2024-03-24T00:37:26.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d5e00640 1 --2- 172.21.15.40:0/1480285491 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x55804fe97030 0x55804fec2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68d5e00640 1 --2- 172.21.15.40:0/1480285491 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x55804fe97030 0x55804fec2ea0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f32caf8628e33c3 server_cookie=d2643f2f9d52fa87 in_seq=0 out_seq=0 2024-03-24T00:37:26.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.269+0000 7f68b3400640 10 client.8187 ms_handle_connect on v2:172.21.15.53:6835/3095501397 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68b3400640 1 -- 172.21.15.40:0/1480285491 <== mds.0 v2:172.21.15.53:6835/3095501397 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55804fc4fb60 con 0x55804fe97030 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68b3400640 10 client.8187 handle_command_reply: tid=0 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68dbda8000 1 client.8187 shutdown 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68dbda8000 1 -- 172.21.15.40:0/1480285491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb0920 msgr2=0x55804fe96280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1480285491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb0920 0x55804fe96280 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f68c0054fc0 tx=0x7f68c0001220 comp rx=0 tx=0).stop 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68dbda8000 1 -- 172.21.15.40:0/1480285491 shutdown_connections 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1480285491 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x55804fe97030 0x55804fec2ea0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1480285491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb1240 0x55804fe967c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1480285491 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb0920 0x55804fe96280 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1480285491 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804fe95d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68dbda8000 1 -- 172.21.15.40:0/1480285491 >> 172.21.15.40:0/1480285491 conn(0x55804fe62fa0 msgr2=0x55804fe65c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68b3400640 0 client.8187 ms_handle_reset on v2:172.21.15.53:6835/3095501397 2024-03-24T00:37:26.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68dbda8000 1 -- 172.21.15.40:0/1480285491 shutdown_connections 2024-03-24T00:37:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68dbda8000 1 -- 172.21.15.40:0/1480285491 wait complete. 2024-03-24T00:37:26.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.273+0000 7f68dbda8000 20 client.8187 trim_cache size 0 max 16384 2024-03-24T00:37:26.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 Processor -- start 2024-03-24T00:37:26.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 -- start start 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804feaf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 0x55804feafa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 0x55804fe94900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d0202ef0 con 0x55804feb1240 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d013caf0 con 0x55804feb0920 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55804fc5ca80 con 0x55804fe66ff0 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68d5e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804feaf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68d6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 0x55804fe94900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68d5e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804feaf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:59178/0 (socket says 172.21.15.40:59178) 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68d5e00640 1 -- 172.21.15.40:0/1169989402 learned_addr learned my addr 172.21.15.40:0/1169989402 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68d5400640 1 --2- 172.21.15.40:0/1169989402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 0x55804feafa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68d5e00640 1 -- 172.21.15.40:0/1169989402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 msgr2=0x55804feafa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68d5e00640 1 --2- 172.21.15.40:0/1169989402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 0x55804feafa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68d5e00640 1 -- 172.21.15.40:0/1169989402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 msgr2=0x55804fe94900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68d5e00640 1 --2- 172.21.15.40:0/1169989402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 0x55804fe94900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68d5e00640 1 -- 172.21.15.40:0/1169989402 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55804fe6b210 con 0x55804fe66ff0 2024-03-24T00:37:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68d6800640 1 --2- 172.21.15.40:0/1169989402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 0x55804fe94900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:26.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68d5e00640 1 --2- 172.21.15.40:0/1169989402 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804feaf4d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f68c4000fc0 tx=0x7f68c4006040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91cbf46683077a39 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68be000640 1 -- 172.21.15.40:0/1169989402 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c4022070 con 0x55804fe66ff0 2024-03-24T00:37:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68be000640 1 -- 172.21.15.40:0/1169989402 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c401d1d0 con 0x55804fe66ff0 2024-03-24T00:37:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68be000640 1 -- 172.21.15.40:0/1169989402 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c40188f0 con 0x55804fe66ff0 2024-03-24T00:37:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 -- 172.21.15.40:0/1169989402 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 msgr2=0x55804feaf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1169989402 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804feaf4d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f68c4000fc0 tx=0x7f68c4006040 comp rx=0 tx=0).stop 2024-03-24T00:37:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 -- 172.21.15.40:0/1169989402 shutdown_connections 2024-03-24T00:37:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1169989402 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804feb1240 0x55804fe94900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1169989402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 0x55804feafa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1169989402 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804fe66ff0 0x55804feaf4d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 -- 172.21.15.40:0/1169989402 >> 172.21.15.40:0/1169989402 conn(0x55804fe62fa0 msgr2=0x55804febdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:26.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 -- 172.21.15.40:0/1169989402 shutdown_connections 2024-03-24T00:37:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.305+0000 7f68dbda8000 1 -- 172.21.15.40:0/1169989402 wait complete. 2024-03-24T00:37:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68dbda8000 1 -- start start 2024-03-24T00:37:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68dbda8000 1 Processor -- start 2024-03-24T00:37:26.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68dbda8000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804fe66ff0 0x55804fe66690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68dbda8000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 0x55804fec3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68dbda8000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804feb1240 0x55804fec4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68dbda8000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55804fc4c750 con 0x55804fe66ff0 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68dbda8000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55804fd71700 con 0x55804feb0920 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68dbda8000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55804fcb3990 con 0x55804feb1240 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68dbda8000 10 client.0 _ll_register_callbacks cb 0x7f68c40070e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68d5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804fe66ff0 0x55804fe66690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68d5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804fe66ff0 0x55804fe66690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:55974/0 (socket says 172.21.15.40:55974) 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68d5e00640 1 -- 172.21.15.40:0/1709758376 learned_addr learned my addr 172.21.15.40:0/1709758376 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68d5400640 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 0x55804fec3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68d6800640 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804feb1240 0x55804fec4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68d5e00640 1 -- 172.21.15.40:0/1709758376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804feb1240 msgr2=0x55804fec4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68d5e00640 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804feb1240 0x55804fec4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68d5e00640 1 -- 172.21.15.40:0/1709758376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 msgr2=0x55804fec3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68d5e00640 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 0x55804fec3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68d5e00640 1 -- 172.21.15.40:0/1709758376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55804fc81540 con 0x55804fe66ff0 2024-03-24T00:37:26.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68d5e00640 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804fe66ff0 0x55804fe66690 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f68c401a040 tx=0x7f68c401d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c025a15cd05366 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:26.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.309+0000 7f68b3400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:37:26.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68b3400640 1 -- 172.21.15.40:0/1709758376 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c402b060 con 0x55804fe66ff0 2024-03-24T00:37:26.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68b3400640 1 -- 172.21.15.40:0/1709758376 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68c4018d00 con 0x55804fe66ff0 2024-03-24T00:37:26.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 -- 172.21.15.40:0/1709758376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f68c401d1d0 con 0x55804fe66ff0 2024-03-24T00:37:26.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68b3400640 1 -- 172.21.15.40:0/1709758376 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=102) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68c4031050 con 0x55804fe66ff0 2024-03-24T00:37:26.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 10 client.8193 fetch_fsmap learned FSMap version 102 2024-03-24T00:37:26.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 -- 172.21.15.40:0/1709758376 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=102,monmap=2+}) v3 -- 0x55804fc484b0 con 0x55804fe66ff0 2024-03-24T00:37:26.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68b3400640 1 -- 172.21.15.40:0/1709758376 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c401d1d0 con 0x55804fe66ff0 2024-03-24T00:37:26.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68b3400640 1 -- 172.21.15.40:0/1709758376 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 102) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f68c4034020 con 0x55804fe66ff0 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 10 client.8193 fetch_fsmap finished waiting for FSMap version 102 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 10 client.8193 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 20 client.8193 populate_metadata read hostname 'smithi040' 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x55804ff155c0 0x55804ff159a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 4 client.8193 mds_command: new command op to 7932 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 -- 172.21.15.40:0/1709758376 --> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55804fcc1bc0 con 0x55804ff155c0 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d5400640 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x55804ff155c0 0x55804ff159a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d5400640 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x55804ff155c0 0x55804ff159a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e7315dab173c331 server_cookie=2ef170559e46ecaa in_seq=0 out_seq=0 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68b3400640 10 client.8193 ms_handle_connect on v2:172.21.15.53:6835/3095501397 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68b3400640 1 -- 172.21.15.40:0/1709758376 <== mds.0 v2:172.21.15.53:6835/3095501397 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55804fcc1bc0 con 0x55804ff155c0 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68b3400640 10 client.8193 handle_command_reply: tid=0 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 client.8193 shutdown 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 -- 172.21.15.40:0/1709758376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804fe66ff0 msgr2=0x55804fe66690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804fe66ff0 0x55804fe66690 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f68c401a040 tx=0x7f68c401d4b0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 -- 172.21.15.40:0/1709758376 shutdown_connections 2024-03-24T00:37:26.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.53:6835/3095501397,v1:172.21.15.53:6837/3095501397] conn(0x55804ff155c0 0x55804ff159a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55804feb1240 0x55804fec4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55804feb0920 0x55804fec3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 --2- 172.21.15.40:0/1709758376 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55804fe66ff0 0x55804fe66690 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 -- 172.21.15.40:0/1709758376 >> 172.21.15.40:0/1709758376 conn(0x55804fe62fa0 msgr2=0x55804febd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:26.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68b3400640 0 client.8193 ms_handle_reset on v2:172.21.15.53:6835/3095501397 2024-03-24T00:37:26.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 -- 172.21.15.40:0/1709758376 shutdown_connections 2024-03-24T00:37:26.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 1 -- 172.21.15.40:0/1709758376 wait complete. 2024-03-24T00:37:26.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68dbda8000 20 client.8193 trim_cache size 0 max 16384 2024-03-24T00:37:26.320 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:37:26.321 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:37:26.321 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:37:26.321 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 7932, 2024-03-24T00:37:26.321 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:37:26.321 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:37:26.321 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:37:26.321 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 24.125258956, 2024-03-24T00:37:26.321 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 102, 2024-03-24T00:37:26.321 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 95, 2024-03-24T00:37:26.321 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 95, 2024-03-24T00:37:26.321 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 28.973503426000001 2024-03-24T00:37:26.321 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:37:26.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d8800640 1 -- 172.21.15.40:0/4025518946 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68b4067660 msgr2=0x7f68b4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d8800640 1 --2- 172.21.15.40:0/4025518946 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68b4067660 0x7f68b4069b20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f68cc0af000 tx=0x7f68cc0a7070 comp rx=0 tx=0).stop 2024-03-24T00:37:26.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d8800640 1 -- 172.21.15.40:0/4025518946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d013ec80 msgr2=0x7f68d0139de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:26.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d8800640 1 --2- 172.21.15.40:0/4025518946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d013ec80 0x7f68d0139de0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f68c000c750 tx=0x7f68c000cc10 comp rx=0 tx=0).stop 2024-03-24T00:37:26.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d8800640 1 -- 172.21.15.40:0/4025518946 shutdown_connections 2024-03-24T00:37:26.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d8800640 1 --2- 172.21.15.40:0/4025518946 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f68b4067660 0x7f68b4069b20 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d8800640 1 --2- 172.21.15.40:0/4025518946 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f68d0141bc0 0x7f68d00b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d8800640 1 --2- 172.21.15.40:0/4025518946 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f68d013ec80 0x7f68d0139de0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d8800640 1 --2- 172.21.15.40:0/4025518946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68d0072310 0x7f68d00ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:26.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d8800640 1 -- 172.21.15.40:0/4025518946 >> 172.21.15.40:0/4025518946 conn(0x7f68d00b17b0 msgr2=0x7f68d015d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:26.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d8800640 1 -- 172.21.15.40:0/4025518946 shutdown_connections 2024-03-24T00:37:26.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:26.313+0000 7f68d8800640 1 -- 172.21.15.40:0/4025518946 wait complete. 2024-03-24T00:37:26.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:26.337 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:37:26.337 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:37:26.337 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p tree 2024-03-24T00:37:26.337 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:37:26.337 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:37:26.338 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:37:26.338 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < 10; i++)); do 2024-03-24T00:37:26.338 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p "tree/$i" 2024-03-24T00:37:26.338 DEBUG:teuthology.orchestra.run.smithi040:> echo file > "tree/$i/file" 2024-03-24T00:37:26.338 DEBUG:teuthology.orchestra.run.smithi040:> done 2024-03-24T00:37:26.338 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:37:26.353 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree 2024-03-24T00:37:26.364 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i = 0 )) 2024-03-24T00:37:26.364 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 10 )) 2024-03-24T00:37:26.364 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/0 2024-03-24T00:37:26.370 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:37:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:37:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 10 )) 2024-03-24T00:37:26.377 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/1 2024-03-24T00:37:26.393 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:37:26.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:37:26.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 10 )) 2024-03-24T00:37:26.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/2 2024-03-24T00:37:26.403 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:37:26.406 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:37:26.406 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 10 )) 2024-03-24T00:37:26.406 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/3 2024-03-24T00:37:26.412 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:37:26.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:37:26.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 10 )) 2024-03-24T00:37:26.415 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/4 2024-03-24T00:37:26.421 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:37:26.424 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:37:26.424 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 10 )) 2024-03-24T00:37:26.424 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/5 2024-03-24T00:37:26.430 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:37:26.433 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:37:26.433 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 10 )) 2024-03-24T00:37:26.433 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/6 2024-03-24T00:37:26.439 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:37:26.441 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 10 )) 2024-03-24T00:37:26.442 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/7 2024-03-24T00:37:26.447 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 10 )) 2024-03-24T00:37:26.450 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/8 2024-03-24T00:37:26.457 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:37:26.459 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 10 )) 2024-03-24T00:37:26.460 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/9 2024-03-24T00:37:26.466 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:37:26.469 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 10 )) 2024-03-24T00:37:26.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:26.470 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/0) 2024-03-24T00:37:26.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:26.523 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/1) 2024-03-24T00:37:26.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:26.583 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/2) 2024-03-24T00:37:26.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:26.639 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/3) 2024-03-24T00:37:26.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:26.699 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/4) 2024-03-24T00:37:26.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:26.755 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/5) 2024-03-24T00:37:26.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:26.811 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/6) 2024-03-24T00:37:26.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:26.871 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/7) 2024-03-24T00:37:26.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:26.932 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/8) 2024-03-24T00:37:26.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:26.990 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 1 tree/9) 2024-03-24T00:37:27.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:27.047 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin.distributed -v 1 tree) 2024-03-24T00:37:37.817 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:37:37.821 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:37:42.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json get subtrees 2024-03-24T00:37:42.269 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:37:42.270 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:37:42.270 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-24T00:37:42.434 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"6112","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/0","dirfrag":"0x100000005e8","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"importbound":0,"sticky":0,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/8","dirfrag":"0x100000005f8","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"importbound":0,"sticky":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/7","dirfrag":"0x100000005f6","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/6","dirfrag":"0x100000005f4","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"importbound":0,"sticky":0,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/5","dirfrag":"0x100000005f2","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/4","dirfrag":"0x100000005f0","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"importbound":0,"sticky":0,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/3","dirfrag":"0x100000005ee","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"importbound":0,"sticky":0,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/2","dirfrag":"0x100000005ec","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"importbound":0,"sticky":0,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/1","dirfrag":"0x100000005ea","snapid_first":2,"projected_version":"6","version":"6","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"importbound":0,"sticky":0,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/9","dirfrag":"0x100000005fa","snapid_first":2,"projected_version":"5","version":"5","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"importbound":0,"sticky":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000005e7.000*","snapid_first":2,"projected_version":"68","version":"68","committing_version":"64","committed_version":"64","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000005e7.011*","snapid_first":2,"projected_version":"70","version":"70","committing_version":"64","committed_version":"64","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000005e7.100*","snapid_first":2,"projected_version":"66","version":"66","committing_version":"64","committed_version":"64","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000005e7.111*","snapid_first":2,"projected_version":"66","version":"66","committing_version":"64","committed_version":"64","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}}] 2024-03-24T00:37:42.436 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "6112" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x100000005e8", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree/0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importbound": 0, "importing": 0, "ptrwaiter": 0, "replicated": 0, "sticky": 0, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x100000005f8", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree/8", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importbound": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "sticky": 0, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x100000005f6", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree/7", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x100000005f4", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree/6", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importbound": 0, "importing": 0, "ptrwaiter": 0, "replicated": 0, "sticky": 0, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x100000005f2", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree/5", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x100000005f0", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree/4", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importbound": 0, "importing": 0, "ptrwaiter": 0, "replicated": 0, "sticky": 0, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x100000005ee", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree/3", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importbound": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "sticky": 0, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x100000005ec", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree/2", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importbound": 0, "importing": 0, "ptrwaiter": 0, "replicated": 0, "sticky": 0, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x100000005ea", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree/1", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importbound": 0, "importing": 0, "ptrwaiter": 0, "replicated": 0, "sticky": 0, "subtree": 1 }, "projected_version": "6", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x100000005fa", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree/9", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importbound": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "sticky": 0, "subtree": 1 }, "projected_version": "5", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "5" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "64", "committing_version": "64", "dir_auth": "1", "dirfrag": "0x100000005e7.000*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "68", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "68" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "64", "committing_version": "64", "dir_auth": "1", "dirfrag": "0x100000005e7.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "70", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "70" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "64", "committing_version": "64", "dir_auth": "1", "dirfrag": "0x100000005e7.100*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "66", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "66" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "64", "committing_version": "64", "dir_auth": "1", "dirfrag": "0x100000005e7.111*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "66", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "66" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:37:42.436 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:37:42.598 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"6137","version":"6137","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"2032","version":"2032","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000005e7.100*","snapid_first":2,"projected_version":"66","version":"66","committing_version":"64","committed_version":"64","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/5","dirfrag":"0x100000005f2","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree/7","dirfrag":"0x100000005f6","snapid_first":2,"projected_version":"3","version":"3","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x100000005e7.101*","snapid_first":2,"projected_version":"64","version":"64","committing_version":"64","committed_version":"64","is_rep":false,"dir_auth":"0","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{"1":2}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"frozen":0,"subtree":1,"replicated":1,"dirty":0,"authpin":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000005e7.000*","snapid_first":2,"projected_version":"68","version":"68","committing_version":"64","committed_version":"64","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":2}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000005e7.011*","snapid_first":2,"projected_version":"70","version":"70","committing_version":"64","committed_version":"64","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x100000005e7.110*","snapid_first":2,"projected_version":"68","version":"68","committing_version":"64","committed_version":"64","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":2}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x100000005e7.111*","snapid_first":2,"projected_version":"66","version":"66","committing_version":"64","committed_version":"64","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-03-24T00:37:42.599 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "6137", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "6137" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "2032", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "2032" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "64", "committing_version": "64", "dir_auth": "1", "dirfrag": "0x100000005e7.100*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "66", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "66" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x100000005f2", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree/5", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "3", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "1", "dirfrag": "0x100000005f6", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree/7", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "3", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "3" }, "distributed_ephemeral_pin": false, "export_pin": 1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 2 } }, "committed_version": "64", "committing_version": "64", "dir_auth": "0", "dirfrag": "0x100000005e7.101*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/tree", "pins": { "authpin": 0, "dirty": 0, "frozen": 0, "replicated": 1, "subtree": 1 }, "projected_version": "64", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "64" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "64", "committing_version": "64", "dir_auth": "1", "dirfrag": "0x100000005e7.000*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "68", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "68" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "64", "committing_version": "64", "dir_auth": "1", "dirfrag": "0x100000005e7.011*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "70", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "70" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 2 } }, "committed_version": "64", "committing_version": "64", "dir_auth": "0", "dirfrag": "0x100000005e7.110*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "replicated": 1, "subtree": 1 }, "projected_version": "68", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "68" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "64", "committing_version": "64", "dir_auth": "1", "dirfrag": "0x100000005e7.111*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "66", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "66" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-24T00:37:42.599 INFO:tasks.cephfs.cephfs_test_case:[('/tree/0', 1), ('/tree/1', 1), ('/tree/2', 1), ('/tree/3', 1), ('/tree/4', 1), ('/tree/5', 1), ('/tree/6', 1), ('/tree/7', 1), ('/tree/8', 1), ('/tree/9', 1)] =?= [('/tree/0', 1), ('/tree/1', 1), ('/tree/2', 1), ('/tree/3', 1), ('/tree/4', 1), ('/tree/5', 1), ('/tree/6', 1), ('/tree/7', 1), ('/tree/8', 1), ('/tree/9', 1)] 2024-03-24T00:37:42.599 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:37:42.599 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:37:42.650 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:37:42.718 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:37:42.718 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:37:42.718 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:37:42.727 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:42.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.909+0000 7f73a8e00640 1 Processor -- start 2024-03-24T00:37:42.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.909+0000 7f73a8e00640 1 -- start start 2024-03-24T00:37:42.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.913+0000 7f73a8e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f73a413ec80 0x7f73a41414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.913+0000 7f73a8e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a4070e00 con 0x7f73a41419f0 2024-03-24T00:37:42.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.913+0000 7f73a8e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a40cee20 con 0x7f73a40ce800 2024-03-24T00:37:42.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.913+0000 7f73a8e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73a4141e50 con 0x7f73a413ec80 2024-03-24T00:37:42.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.913+0000 7f739be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f73a41419f0 0x7f73a40ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53588/0 (socket says 172.21.15.40:53588) 2024-03-24T00:37:42.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.913+0000 7f739be00640 1 -- 172.21.15.40:0/2055790650 learned_addr learned my addr 172.21.15.40:0/2055790650 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:42.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.913+0000 7f73a1e00640 1 --2- 172.21.15.40:0/2055790650 >> v2:172.21.15.53:3300/0 conn(0x7f73a413ec80 0x7f73a41414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.913+0000 7f73a1400640 1 -- 172.21.15.40:0/2055790650 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539497874 0 0) 0x7f73a4070e00 con 0x7f73a41419f0 2024-03-24T00:37:42.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.913+0000 7f73a1400640 1 -- 172.21.15.40:0/2055790650 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7388003530 con 0x7f73a41419f0 2024-03-24T00:37:42.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.913+0000 7f73a1e00640 1 -- 172.21.15.40:0/2055790650 >> v1:172.21.15.53:6789/0 conn(0x7f73a40ce800 legacy=0x7f73a4071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.913+0000 7f73a1e00640 1 -- 172.21.15.40:0/2055790650 >> v1:172.21.15.40:6789/0 conn(0x7f73a41419f0 legacy=0x7f73a40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.913+0000 7f73a1e00640 1 -- 172.21.15.40:0/2055790650 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a40cf100 con 0x7f73a413ec80 2024-03-24T00:37:42.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a1e00640 1 --2- 172.21.15.40:0/2055790650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a413ec80 0x7f73a41414b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f73940097c0 tx=0x7f739401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3270c59a49b7e533 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:42.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a1400640 1 -- 172.21.15.40:0/2055790650 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73940aa620 con 0x7f73a413ec80 2024-03-24T00:37:42.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a1400640 1 -- 172.21.15.40:0/2055790650 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73940af070 con 0x7f73a413ec80 2024-03-24T00:37:42.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a1400640 1 -- 172.21.15.40:0/2055790650 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73940aa9e0 con 0x7f73a413ec80 2024-03-24T00:37:42.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a8e00640 1 -- 172.21.15.40:0/2055790650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a413ec80 msgr2=0x7f73a41414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a8e00640 1 --2- 172.21.15.40:0/2055790650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a413ec80 0x7f73a41414b0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f73940097c0 tx=0x7f739401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a8e00640 1 -- 172.21.15.40:0/2055790650 shutdown_connections 2024-03-24T00:37:42.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a8e00640 1 --2- 172.21.15.40:0/2055790650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a413ec80 0x7f73a41414b0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a8e00640 1 -- 172.21.15.40:0/2055790650 >> 172.21.15.40:0/2055790650 conn(0x7f73a40b17b0 msgr2=0x7f73a40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:42.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a8e00640 1 -- 172.21.15.40:0/2055790650 shutdown_connections 2024-03-24T00:37:42.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a8e00640 1 -- 172.21.15.40:0/2055790650 wait complete. 2024-03-24T00:37:42.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a8e00640 1 Processor -- start 2024-03-24T00:37:42.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.917+0000 7f73a8e00640 1 -- start start 2024-03-24T00:37:42.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a8e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a40ce800 0x7f73a4157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a8e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a413ec80 0x7f73a413a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a8e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73a41419f0 0x7f73a415e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a8e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40725f0 con 0x7f73a40ce800 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a8e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a4072470 con 0x7f73a413ec80 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a8e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a4169d50 con 0x7f73a41419f0 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a40ce800 0x7f73a4157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a40ce800 0x7f73a4157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51880/0 (socket says 172.21.15.40:51880) 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a1e00640 1 -- 172.21.15.40:0/1986754930 learned_addr learned my addr 172.21.15.40:0/1986754930 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f739be00640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a413ec80 0x7f73a413a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a2800640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73a41419f0 0x7f73a415e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a1e00640 1 -- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73a41419f0 msgr2=0x7f73a415e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a1e00640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73a41419f0 0x7f73a415e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a1e00640 1 -- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a413ec80 msgr2=0x7f73a413a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a1e00640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a413ec80 0x7f73a413a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:42.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a1e00640 1 -- 172.21.15.40:0/1986754930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a40b39e0 con 0x7f73a40ce800 2024-03-24T00:37:42.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a1e00640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a40ce800 0x7f73a4157560 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f73940190b0 tx=0x7f739401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:42.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f739a000640 1 -- 172.21.15.40:0/1986754930 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73940aac70 con 0x7f73a40ce800 2024-03-24T00:37:42.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f739a000640 1 -- 172.21.15.40:0/1986754930 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73940aae10 con 0x7f73a40ce800 2024-03-24T00:37:42.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a8e00640 1 -- 172.21.15.40:0/1986754930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a413bf00 con 0x7f73a40ce800 2024-03-24T00:37:42.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f739a000640 1 -- 172.21.15.40:0/1986754930 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73940be650 con 0x7f73a40ce800 2024-03-24T00:37:42.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a8e00640 1 -- 172.21.15.40:0/1986754930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a413ab40 con 0x7f73a40ce800 2024-03-24T00:37:42.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.921+0000 7f73a8e00640 1 -- 172.21.15.40:0/1986754930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a4068170 con 0x7f73a40ce800 2024-03-24T00:37:42.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.925+0000 7f739a000640 1 -- 172.21.15.40:0/1986754930 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f73940bc020 con 0x7f73a40ce800 2024-03-24T00:37:42.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.925+0000 7f739a000640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7380067660 0x7f7380069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:42.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.925+0000 7f739a000640 1 -- 172.21.15.40:0/1986754930 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f739411fab0 con 0x7f73a40ce800 2024-03-24T00:37:42.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.925+0000 7f739be00640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7380067660 0x7f7380069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:42.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.925+0000 7f739be00640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7380067660 0x7f7380069b20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f73a4151960 tx=0x7f738c007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:42.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:42.933+0000 7f739a000640 1 -- 172.21.15.40:0/1986754930 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7394016090 con 0x7f73a40ce800 2024-03-24T00:37:43.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.129+0000 7f73a8e00640 1 -- 172.21.15.40:0/1986754930 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73a4072b90 con 0x7f73a40ce800 2024-03-24T00:37:43.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.129+0000 7f739a000640 1 -- 172.21.15.40:0/1986754930 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16548 (secure 0 0 0) 0x7f73940f0200 con 0x7f73a40ce800 2024-03-24T00:37:43.135 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:43.135 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":95,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:37:03.189247+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:59.977310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-24T00:37:00.557420+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:43.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.133+0000 7f73a8e00640 1 -- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7380067660 msgr2=0x7f7380069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.133+0000 7f73a8e00640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7380067660 0x7f7380069b20 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f73a4151960 tx=0x7f738c007d40 comp rx=0 tx=0).stop 2024-03-24T00:37:43.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.133+0000 7f73a8e00640 1 -- 172.21.15.40:0/1986754930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a40ce800 msgr2=0x7f73a4157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.133+0000 7f73a8e00640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a40ce800 0x7f73a4157560 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f73940190b0 tx=0x7f739401ace0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.133+0000 7f73a8e00640 1 -- 172.21.15.40:0/1986754930 shutdown_connections 2024-03-24T00:37:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.133+0000 7f73a8e00640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7380067660 0x7f7380069b20 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.133+0000 7f73a8e00640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73a41419f0 0x7f73a415e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.133+0000 7f73a8e00640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73a413ec80 0x7f73a413a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.133+0000 7f73a8e00640 1 --2- 172.21.15.40:0/1986754930 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f73a40ce800 0x7f73a4157560 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.133+0000 7f73a8e00640 1 -- 172.21.15.40:0/1986754930 >> 172.21.15.40:0/1986754930 conn(0x7f73a40b17b0 msgr2=0x7f73a4071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.133+0000 7f73a8e00640 1 -- 172.21.15.40:0/1986754930 shutdown_connections 2024-03-24T00:37:43.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.133+0000 7f73a8e00640 1 -- 172.21.15.40:0/1986754930 wait complete. 2024-03-24T00:37:43.156 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:36:59.977310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:37:00.557420+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:37:43.157 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:37:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.297+0000 7f44f3e00640 1 Processor -- start 2024-03-24T00:37:43.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.297+0000 7f44f3e00640 1 -- start start 2024-03-24T00:37:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.297+0000 7f44f3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f44f40d0780 0x7f44f40cd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.297+0000 7f44f3e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f40cf2e0 con 0x7f44f40b0bd0 2024-03-24T00:37:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.297+0000 7f44f3e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f40cc7b0 con 0x7f44f415e0e0 2024-03-24T00:37:43.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.297+0000 7f44f3e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f40cd0e0 con 0x7f44f40d0780 2024-03-24T00:37:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.301+0000 7f44ebe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f44f40b0bd0 0x7f44f40d0070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53596/0 (socket says 172.21.15.40:53596) 2024-03-24T00:37:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.301+0000 7f44ebe00640 1 -- 172.21.15.40:0/2736086047 learned_addr learned my addr 172.21.15.40:0/2736086047 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.301+0000 7f44f3400640 1 --2- 172.21.15.40:0/2736086047 >> v2:172.21.15.53:3300/0 conn(0x7f44f40d0780 0x7f44f40cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.301+0000 7f44f2000640 1 -- 172.21.15.40:0/2736086047 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000999585 0 0) 0x7f44f40cf2e0 con 0x7f44f40b0bd0 2024-03-24T00:37:43.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.301+0000 7f44f2000640 1 -- 172.21.15.40:0/2736086047 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d8003530 con 0x7f44f40b0bd0 2024-03-24T00:37:43.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.301+0000 7f44f2000640 1 -- 172.21.15.40:0/2736086047 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354607848 0 0) 0x7f44f40cc7b0 con 0x7f44f415e0e0 2024-03-24T00:37:43.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.301+0000 7f44f2000640 1 -- 172.21.15.40:0/2736086047 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f40cf2e0 con 0x7f44f415e0e0 2024-03-24T00:37:43.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.301+0000 7f44f3400640 1 -- 172.21.15.40:0/2736086047 >> v1:172.21.15.53:6789/0 conn(0x7f44f415e0e0 legacy=0x7f44f40ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.301+0000 7f44f3400640 1 -- 172.21.15.40:0/2736086047 >> v1:172.21.15.40:6789/0 conn(0x7f44f40b0bd0 legacy=0x7f44f40d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.301+0000 7f44f3400640 1 -- 172.21.15.40:0/2736086047 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f40cca90 con 0x7f44f40d0780 2024-03-24T00:37:43.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.301+0000 7f44f3400640 1 --2- 172.21.15.40:0/2736086047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f40d0780 0x7f44f40cd570 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009d90 tx=0x7f44ec01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e205854628e7f22 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:43.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f2000640 1 -- 172.21.15.40:0/2736086047 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec0b2070 con 0x7f44f40d0780 2024-03-24T00:37:43.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f2000640 1 -- 172.21.15.40:0/2736086047 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ec0a5940 con 0x7f44f40d0780 2024-03-24T00:37:43.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f2000640 1 -- 172.21.15.40:0/2736086047 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec0a5d10 con 0x7f44f40d0780 2024-03-24T00:37:43.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 -- 172.21.15.40:0/2736086047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f40d0780 msgr2=0x7f44f40cd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 --2- 172.21.15.40:0/2736086047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f40d0780 0x7f44f40cd570 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009d90 tx=0x7f44ec01c970 comp rx=0 tx=0).stop 2024-03-24T00:37:43.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 -- 172.21.15.40:0/2736086047 shutdown_connections 2024-03-24T00:37:43.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 --2- 172.21.15.40:0/2736086047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f40d0780 0x7f44f40cd570 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 -- 172.21.15.40:0/2736086047 >> 172.21.15.40:0/2736086047 conn(0x7f44f40b3620 msgr2=0x7f44f40b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:43.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 -- 172.21.15.40:0/2736086047 shutdown_connections 2024-03-24T00:37:43.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 -- 172.21.15.40:0/2736086047 wait complete. 2024-03-24T00:37:43.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 Processor -- start 2024-03-24T00:37:43.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 -- start start 2024-03-24T00:37:43.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f40b0bd0 0x7f44f41555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f40d0780 0x7f44f40b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f415e0e0 0x7f44f40b6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f406bd30 con 0x7f44f40b0bd0 2024-03-24T00:37:43.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f406bbb0 con 0x7f44f40d0780 2024-03-24T00:37:43.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f416be30 con 0x7f44f415e0e0 2024-03-24T00:37:43.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f40b0bd0 0x7f44f41555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f40b0bd0 0x7f44f41555c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51892/0 (socket says 172.21.15.40:51892) 2024-03-24T00:37:43.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f2a00640 1 -- 172.21.15.40:0/1686013648 learned_addr learned my addr 172.21.15.40:0/1686013648 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:43.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44ebe00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f40d0780 0x7f44f40b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.305+0000 7f44f3400640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f415e0e0 0x7f44f40b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44f2a00640 1 -- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f415e0e0 msgr2=0x7f44f40b6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44f2a00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f415e0e0 0x7f44f40b6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44f2a00640 1 -- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f40d0780 msgr2=0x7f44f40b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44f2a00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f40d0780 0x7f44f40b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44f2a00640 1 -- 172.21.15.40:0/1686013648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f4155b00 con 0x7f44f40b0bd0 2024-03-24T00:37:43.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44f2a00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f40b0bd0 0x7f44f41555c0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f44e400c490 tx=0x7f44e400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:43.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44eaa00640 1 -- 172.21.15.40:0/1686013648 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e40052d0 con 0x7f44f40b0bd0 2024-03-24T00:37:43.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44eaa00640 1 -- 172.21.15.40:0/1686013648 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e4005470 con 0x7f44f40b0bd0 2024-03-24T00:37:43.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44f3e00640 1 -- 172.21.15.40:0/1686013648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f40c7770 con 0x7f44f40b0bd0 2024-03-24T00:37:43.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44eaa00640 1 -- 172.21.15.40:0/1686013648 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4012870 con 0x7f44f40b0bd0 2024-03-24T00:37:43.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44f3e00640 1 -- 172.21.15.40:0/1686013648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f40c7d30 con 0x7f44f40b0bd0 2024-03-24T00:37:43.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44f3e00640 1 -- 172.21.15.40:0/1686013648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f40c7f00 con 0x7f44f40b0bd0 2024-03-24T00:37:43.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44eaa00640 1 -- 172.21.15.40:0/1686013648 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f44e4012a10 con 0x7f44f40b0bd0 2024-03-24T00:37:43.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44eaa00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44d0067660 0x7f44d0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.309+0000 7f44eaa00640 1 -- 172.21.15.40:0/1686013648 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44e4085460 con 0x7f44f40b0bd0 2024-03-24T00:37:43.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.313+0000 7f44ebe00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44d0067660 0x7f44d0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.313+0000 7f44ebe00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44d0067660 0x7f44d0069b20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f44f40c85b0 tx=0x7f44dc007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:43.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.317+0000 7f44eaa00640 1 -- 172.21.15.40:0/1686013648 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f44e40559f0 con 0x7f44f40b0bd0 2024-03-24T00:37:43.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 -- 172.21.15.40:0/1686013648 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44f40b4140 con 0x7f44f40b0bd0 2024-03-24T00:37:43.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44eaa00640 1 -- 172.21.15.40:0/1686013648 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f44e4059a90 con 0x7f44f40b0bd0 2024-03-24T00:37:43.489 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:37:43.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 -- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44d0067660 msgr2=0x7f44d0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44d0067660 0x7f44d0069b20 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f44f40c85b0 tx=0x7f44dc007d40 comp rx=0 tx=0).stop 2024-03-24T00:37:43.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 -- 172.21.15.40:0/1686013648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f40b0bd0 msgr2=0x7f44f41555c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f40b0bd0 0x7f44f41555c0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f44e400c490 tx=0x7f44e400c950 comp rx=0 tx=0).stop 2024-03-24T00:37:43.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 -- 172.21.15.40:0/1686013648 shutdown_connections 2024-03-24T00:37:43.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f44d0067660 0x7f44d0069b20 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44f415e0e0 0x7f44f40b6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44f40d0780 0x7f44f40b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 --2- 172.21.15.40:0/1686013648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f44f40b0bd0 0x7f44f41555c0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 -- 172.21.15.40:0/1686013648 >> 172.21.15.40:0/1686013648 conn(0x7f44f40b3620 msgr2=0x7f44f40cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:43.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 -- 172.21.15.40:0/1686013648 shutdown_connections 2024-03-24T00:37:43.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.485+0000 7f44f3e00640 1 -- 172.21.15.40:0/1686013648 wait complete. 2024-03-24T00:37:43.509 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:37:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.649+0000 7fee46e00640 1 Processor -- start 2024-03-24T00:37:43.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.649+0000 7fee46e00640 1 -- start start 2024-03-24T00:37:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.649+0000 7fee46e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fee4015bd00 0x7fee400ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.649+0000 7fee46e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee40072d60 con 0x7fee4015ffd0 2024-03-24T00:37:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.649+0000 7fee46e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee40071de0 con 0x7fee400728d0 2024-03-24T00:37:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.653+0000 7fee46e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee400cea50 con 0x7fee4015bd00 2024-03-24T00:37:43.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.653+0000 7fee44e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fee4015ffd0 0x7fee4006cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53600/0 (socket says 172.21.15.40:53600) 2024-03-24T00:37:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.653+0000 7fee44e00640 1 -- 172.21.15.40:0/1953370833 learned_addr learned my addr 172.21.15.40:0/1953370833 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.653+0000 7fee3fe00640 1 --2- 172.21.15.40:0/1953370833 >> v2:172.21.15.53:3300/0 conn(0x7fee4015bd00 0x7fee400ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.653+0000 7fee3f400640 1 -- 172.21.15.40:0/1953370833 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697939288 0 0) 0x7fee40072d60 con 0x7fee4015ffd0 2024-03-24T00:37:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.653+0000 7fee3f400640 1 -- 172.21.15.40:0/1953370833 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee24003530 con 0x7fee4015ffd0 2024-03-24T00:37:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.653+0000 7fee3fe00640 1 -- 172.21.15.40:0/1953370833 >> v1:172.21.15.53:6789/0 conn(0x7fee400728d0 legacy=0x7fee4015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.653+0000 7fee3f400640 1 -- 172.21.15.40:0/1953370833 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980609839 0 0) 0x7fee40071de0 con 0x7fee400728d0 2024-03-24T00:37:43.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.653+0000 7fee3fe00640 1 -- 172.21.15.40:0/1953370833 >> v1:172.21.15.40:6789/0 conn(0x7fee4015ffd0 legacy=0x7fee4006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.653+0000 7fee3fe00640 1 -- 172.21.15.40:0/1953370833 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee400ced30 con 0x7fee4015bd00 2024-03-24T00:37:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.653+0000 7fee3fe00640 1 --2- 172.21.15.40:0/1953370833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4015bd00 0x7fee400ae8e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fee300097c0 tx=0x7fee3001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48ea2db3691561c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.653+0000 7fee3f400640 1 -- 172.21.15.40:0/1953370833 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300ab650 con 0x7fee4015bd00 2024-03-24T00:37:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee3f400640 1 -- 172.21.15.40:0/1953370833 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee300b0070 con 0x7fee4015bd00 2024-03-24T00:37:43.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee3f400640 1 -- 172.21.15.40:0/1953370833 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300aba10 con 0x7fee4015bd00 2024-03-24T00:37:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 -- 172.21.15.40:0/1953370833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4015bd00 msgr2=0x7fee400ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 --2- 172.21.15.40:0/1953370833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4015bd00 0x7fee400ae8e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fee300097c0 tx=0x7fee3001c470 comp rx=0 tx=0).stop 2024-03-24T00:37:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 -- 172.21.15.40:0/1953370833 shutdown_connections 2024-03-24T00:37:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 --2- 172.21.15.40:0/1953370833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4015bd00 0x7fee400ae8e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 -- 172.21.15.40:0/1953370833 >> 172.21.15.40:0/1953370833 conn(0x7fee400b1660 msgr2=0x7fee400b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 -- 172.21.15.40:0/1953370833 shutdown_connections 2024-03-24T00:37:43.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 -- 172.21.15.40:0/1953370833 wait complete. 2024-03-24T00:37:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 Processor -- start 2024-03-24T00:37:43.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 -- start start 2024-03-24T00:37:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fee400728d0 0x7fee401482b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee4015bd00 0x7fee400af210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4015ffd0 0x7fee4015c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4015c0e0 con 0x7fee4015bd00 2024-03-24T00:37:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee400cf660 con 0x7fee4015ffd0 2024-03-24T00:37:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee46e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee4016a420 con 0x7fee400728d0 2024-03-24T00:37:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee37e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee4015bd00 0x7fee400af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee3fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fee400728d0 0x7fee401482b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee37e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee4015bd00 0x7fee400af210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51898/0 (socket says 172.21.15.40:51898) 2024-03-24T00:37:43.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee37e00640 1 -- 172.21.15.40:0/791411034 learned_addr learned my addr 172.21.15.40:0/791411034 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:43.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.657+0000 7fee44e00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4015ffd0 0x7fee4015c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee3fe00640 1 -- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4015ffd0 msgr2=0x7fee4015c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee3fe00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4015ffd0 0x7fee4015c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee3fe00640 1 -- 172.21.15.40:0/791411034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee4015bd00 msgr2=0x7fee400af210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee3fe00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee4015bd00 0x7fee400af210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee3fe00640 1 -- 172.21.15.40:0/791411034 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee400afcd0 con 0x7fee400728d0 2024-03-24T00:37:43.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee37e00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee4015bd00 0x7fee400af210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:43.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee3fe00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fee400728d0 0x7fee401482b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fee300ac000 tx=0x7fee3001acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:43.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee3cc00640 1 -- 172.21.15.40:0/791411034 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300bf610 con 0x7fee400728d0 2024-03-24T00:37:43.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee3cc00640 1 -- 172.21.15.40:0/791411034 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee300bf7b0 con 0x7fee400728d0 2024-03-24T00:37:43.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee3cc00640 1 -- 172.21.15.40:0/791411034 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee300bfb80 con 0x7fee400728d0 2024-03-24T00:37:43.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee46e00640 1 -- 172.21.15.40:0/791411034 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee400affb0 con 0x7fee400728d0 2024-03-24T00:37:43.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee46e00640 1 -- 172.21.15.40:0/791411034 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee400b0570 con 0x7fee400728d0 2024-03-24T00:37:43.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee46e00640 1 -- 172.21.15.40:0/791411034 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee40068170 con 0x7fee400728d0 2024-03-24T00:37:43.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee3cc00640 1 -- 172.21.15.40:0/791411034 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fee300bd020 con 0x7fee400728d0 2024-03-24T00:37:43.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee3cc00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fee1c067660 0x7fee1c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:43.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee3cc00640 1 -- 172.21.15.40:0/791411034 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee30120040 con 0x7fee400728d0 2024-03-24T00:37:43.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.661+0000 7fee37e00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fee1c067660 0x7fee1c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:43.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.665+0000 7fee37e00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fee1c067660 0x7fee1c069b20 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fee40151360 tx=0x7fee28009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:43.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.669+0000 7fee3cc00640 1 -- 172.21.15.40:0/791411034 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fee30016090 con 0x7fee400728d0 2024-03-24T00:37:43.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 -- 172.21.15.40:0/791411034 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fee4006e870 con 0x7fee400728d0 2024-03-24T00:37:43.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee3cc00640 1 -- 172.21.15.40:0/791411034 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7fee300f0660 con 0x7fee400728d0 2024-03-24T00:37:43.872 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:37:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 -- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fee1c067660 msgr2=0x7fee1c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fee1c067660 0x7fee1c069b20 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fee40151360 tx=0x7fee28009040 comp rx=0 tx=0).stop 2024-03-24T00:37:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 -- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fee400728d0 msgr2=0x7fee401482b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fee400728d0 0x7fee401482b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fee300ac000 tx=0x7fee3001acb0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 -- 172.21.15.40:0/791411034 shutdown_connections 2024-03-24T00:37:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fee1c067660 0x7fee1c069b20 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4015ffd0 0x7fee4015c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fee4015bd00 0x7fee400af210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 --2- 172.21.15.40:0/791411034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fee400728d0 0x7fee401482b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:43.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 -- 172.21.15.40:0/791411034 >> 172.21.15.40:0/791411034 conn(0x7fee400b1660 msgr2=0x7fee400b2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:43.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 -- 172.21.15.40:0/791411034 shutdown_connections 2024-03-24T00:37:43.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:43.869+0000 7fee46e00640 1 -- 172.21.15.40:0/791411034 wait complete. 2024-03-24T00:37:43.892 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:37:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1757400640 1 Processor -- start 2024-03-24T00:37:44.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1757400640 1 -- start start 2024-03-24T00:37:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1757400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f175015c0e0 0x7f1750074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1757400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17500afa60 0x7f17500ae520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1757400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1750073360 0x7f17500aea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1757400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17500cdb80 con 0x7f1750073360 2024-03-24T00:37:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1757400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17500b2df0 con 0x7f175015c0e0 2024-03-24T00:37:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1757400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17500cd8a0 con 0x7f17500afa60 2024-03-24T00:37:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1755400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1750073360 0x7f17500aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1755400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f1750073360 0x7f17500aea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51914/0 (socket says 172.21.15.40:51914) 2024-03-24T00:37:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1755400640 1 -- 172.21.15.40:0/1460145396 learned_addr learned my addr 172.21.15.40:0/1460145396 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:44.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f174fe00640 1 --2- 172.21.15.40:0/1460145396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17500afa60 0x7f17500ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1754a00640 1 --2- 172.21.15.40:0/1460145396 >> v2:172.21.15.53:3300/0 conn(0x7f175015c0e0 0x7f1750074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1755400640 1 -- 172.21.15.40:0/1460145396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17500afa60 msgr2=0x7f17500ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1755400640 1 --2- 172.21.15.40:0/1460145396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17500afa60 0x7f17500ae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1755400640 1 -- 172.21.15.40:0/1460145396 >> v2:172.21.15.53:3300/0 conn(0x7f175015c0e0 msgr2=0x7f1750074420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1755400640 1 --2- 172.21.15.40:0/1460145396 >> v2:172.21.15.53:3300/0 conn(0x7f175015c0e0 0x7f1750074420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.033+0000 7f1755400640 1 -- 172.21.15.40:0/1460145396 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17500add20 con 0x7f1750073360 2024-03-24T00:37:44.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f1755400640 1 --2- 172.21.15.40:0/1460145396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1750073360 0x7f17500aea60 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f1750070b00 tx=0x7f174801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a33ba47cb61d9a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:44.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f174f400640 1 -- 172.21.15.40:0/1460145396 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17480a5630 con 0x7f1750073360 2024-03-24T00:37:44.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f174f400640 1 -- 172.21.15.40:0/1460145396 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17480a57d0 con 0x7f1750073360 2024-03-24T00:37:44.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f174f400640 1 -- 172.21.15.40:0/1460145396 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17480a5ba0 con 0x7f1750073360 2024-03-24T00:37:44.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f1757400640 1 -- 172.21.15.40:0/1460145396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1750073360 msgr2=0x7f17500aea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f1757400640 1 --2- 172.21.15.40:0/1460145396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1750073360 0x7f17500aea60 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f1750070b00 tx=0x7f174801c8a0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f1757400640 1 -- 172.21.15.40:0/1460145396 shutdown_connections 2024-03-24T00:37:44.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f1757400640 1 --2- 172.21.15.40:0/1460145396 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1750073360 0x7f17500aea60 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f1757400640 1 --2- 172.21.15.40:0/1460145396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f17500afa60 0x7f17500ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f1757400640 1 --2- 172.21.15.40:0/1460145396 >> v2:172.21.15.53:3300/0 conn(0x7f175015c0e0 0x7f1750074420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f1757400640 1 -- 172.21.15.40:0/1460145396 >> 172.21.15.40:0/1460145396 conn(0x7f1750072870 msgr2=0x7f1750072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f1757400640 1 -- 172.21.15.40:0/1460145396 shutdown_connections 2024-03-24T00:37:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f1757400640 1 -- 172.21.15.40:0/1460145396 wait complete. 2024-03-24T00:37:44.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.037+0000 7f1757400640 1 Processor -- start 2024-03-24T00:37:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1757400640 1 -- start start 2024-03-24T00:37:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1757400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1750073360 0x7f17501552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1757400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17500afa60 0x7f175015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1757400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f175015c0e0 0x7f175014b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1757400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17500cc540 con 0x7f17500afa60 2024-03-24T00:37:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1757400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17500cc3c0 con 0x7f1750073360 2024-03-24T00:37:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1757400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17500cc6c0 con 0x7f175015c0e0 2024-03-24T00:37:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f174fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17500afa60 0x7f175015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1754a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1750073360 0x7f17501552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1754a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1750073360 0x7f17501552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:50432/0 (socket says 172.21.15.40:50432) 2024-03-24T00:37:44.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1754a00640 1 -- 172.21.15.40:0/133347028 learned_addr learned my addr 172.21.15.40:0/133347028 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1755400640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f175015c0e0 0x7f175014b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f174fe00640 1 -- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f175015c0e0 msgr2=0x7f175014b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f174fe00640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f175015c0e0 0x7f175014b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f174fe00640 1 -- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1750073360 msgr2=0x7f17501552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f174fe00640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1750073360 0x7f17501552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f174fe00640 1 -- 172.21.15.40:0/133347028 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17500b40f0 con 0x7f17500afa60 2024-03-24T00:37:44.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1754a00640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1750073360 0x7f17501552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:44.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f174fe00640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17500afa60 0x7f175015f0e0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f173c00e7f0 tx=0x7f173c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:44.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f174cc00640 1 -- 172.21.15.40:0/133347028 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c017070 con 0x7f17500afa60 2024-03-24T00:37:44.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f174cc00640 1 -- 172.21.15.40:0/133347028 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173c00c940 con 0x7f17500afa60 2024-03-24T00:37:44.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1757400640 1 -- 172.21.15.40:0/133347028 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17500b43d0 con 0x7f17500afa60 2024-03-24T00:37:44.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f174cc00640 1 -- 172.21.15.40:0/133347028 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c00ccf0 con 0x7f17500afa60 2024-03-24T00:37:44.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1757400640 1 -- 172.21.15.40:0/133347028 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17500b4990 con 0x7f17500afa60 2024-03-24T00:37:44.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.041+0000 7f1757400640 1 -- 172.21.15.40:0/133347028 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1750068170 con 0x7f17500afa60 2024-03-24T00:37:44.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.045+0000 7f174cc00640 1 -- 172.21.15.40:0/133347028 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f173c01a070 con 0x7f17500afa60 2024-03-24T00:37:44.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.045+0000 7f174cc00640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1730067660 0x7f1730069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.045+0000 7f174cc00640 1 -- 172.21.15.40:0/133347028 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f173c085a00 con 0x7f17500afa60 2024-03-24T00:37:44.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.045+0000 7f1754a00640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1730067660 0x7f1730069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.045+0000 7f1754a00640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1730067660 0x7f1730069b20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1744009f70 tx=0x7f1744009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:44.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.049+0000 7f174cc00640 1 -- 172.21.15.40:0/133347028 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f173c0560a0 con 0x7f17500afa60 2024-03-24T00:37:44.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.213+0000 7f1757400640 1 -- 172.21.15.40:0/133347028 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f175015b090 con 0x7f17500afa60 2024-03-24T00:37:44.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.213+0000 7f174cc00640 1 -- 172.21.15.40:0/133347028 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7f173c05a140 con 0x7f17500afa60 2024-03-24T00:37:44.218 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:37:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.217+0000 7f1757400640 1 -- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1730067660 msgr2=0x7f1730069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.217+0000 7f1757400640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1730067660 0x7f1730069b20 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1744009f70 tx=0x7f1744009190 comp rx=0 tx=0).stop 2024-03-24T00:37:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.217+0000 7f1757400640 1 -- 172.21.15.40:0/133347028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17500afa60 msgr2=0x7f175015f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.217+0000 7f1757400640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17500afa60 0x7f175015f0e0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f173c00e7f0 tx=0x7f173c00ecb0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.217+0000 7f1757400640 1 -- 172.21.15.40:0/133347028 shutdown_connections 2024-03-24T00:37:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.217+0000 7f1757400640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1730067660 0x7f1730069b20 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.217+0000 7f1757400640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f175015c0e0 0x7f175014b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.217+0000 7f1757400640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f17500afa60 0x7f175015f0e0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.217+0000 7f1757400640 1 --2- 172.21.15.40:0/133347028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1750073360 0x7f17501552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.217+0000 7f1757400640 1 -- 172.21.15.40:0/133347028 >> 172.21.15.40:0/133347028 conn(0x7f1750072870 msgr2=0x7f17500ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.217+0000 7f1757400640 1 -- 172.21.15.40:0/133347028 shutdown_connections 2024-03-24T00:37:44.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.217+0000 7f1757400640 1 -- 172.21.15.40:0/133347028 wait complete. 2024-03-24T00:37:44.239 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f6017c00640 1 Processor -- start 2024-03-24T00:37:44.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f6017c00640 1 -- start start 2024-03-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f6017c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60100d0830 0x7f60100cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f6017c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f601006b150 0x7f601006b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f6017c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60100d0e10 con 0x7f601015e0b0 2024-03-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f6017c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60100d10f0 con 0x7f601006b150 2024-03-24T00:37:44.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f6017c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601006efd0 con 0x7f60100d0830 2024-03-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f600fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60100d0830 0x7f60100cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f6015200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f601015e0b0 0x7f60100cd3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:46116/0 (socket says 172.21.15.40:46116) 2024-03-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f6015200640 1 -- 172.21.15.40:0/1620746158 learned_addr learned my addr 172.21.15.40:0/1620746158 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:44.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f6015c00640 1 --2- 172.21.15.40:0/1620746158 >> v2:172.21.15.40:3300/0 conn(0x7f601006b150 0x7f601006b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f600f400640 1 -- 172.21.15.40:0/1620746158 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041700924 0 0) 0x7f60100d0e10 con 0x7f601015e0b0 2024-03-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f600f400640 1 -- 172.21.15.40:0/1620746158 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff8003530 con 0x7f601015e0b0 2024-03-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f600fe00640 1 -- 172.21.15.40:0/1620746158 >> v2:172.21.15.40:3300/0 conn(0x7f601006b150 msgr2=0x7f601006b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f600fe00640 1 --2- 172.21.15.40:0/1620746158 >> v2:172.21.15.40:3300/0 conn(0x7f601006b150 0x7f601006b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f600fe00640 1 -- 172.21.15.40:0/1620746158 >> v1:172.21.15.53:6789/0 conn(0x7f601015e0b0 legacy=0x7f60100cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f600fe00640 1 -- 172.21.15.40:0/1620746158 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601006f2b0 con 0x7f60100d0830 2024-03-24T00:37:44.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.377+0000 7f6015c00640 1 --2- 172.21.15.40:0/1620746158 >> v2:172.21.15.40:3300/0 conn(0x7f601006b150 0x7f601006b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:44.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.385+0000 7f600fe00640 1 --2- 172.21.15.40:0/1620746158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60100d0830 0x7f60100cdae0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0097c0 tx=0x7f5ffc01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b080b220e0753dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.385+0000 7f600f400640 1 -- 172.21.15.40:0/1620746158 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0b0070 con 0x7f60100d0830 2024-03-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.385+0000 7f600f400640 1 -- 172.21.15.40:0/1620746158 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc0ab600 con 0x7f60100d0830 2024-03-24T00:37:44.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.385+0000 7f600f400640 1 -- 172.21.15.40:0/1620746158 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0ab9d0 con 0x7f60100d0830 2024-03-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.385+0000 7f6017c00640 1 -- 172.21.15.40:0/1620746158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60100d0830 msgr2=0x7f60100cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.385+0000 7f6017c00640 1 --2- 172.21.15.40:0/1620746158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60100d0830 0x7f60100cdae0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0097c0 tx=0x7f5ffc01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.385+0000 7f6017c00640 1 -- 172.21.15.40:0/1620746158 shutdown_connections 2024-03-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.385+0000 7f6017c00640 1 --2- 172.21.15.40:0/1620746158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60100d0830 0x7f60100cdae0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.385+0000 7f6017c00640 1 --2- 172.21.15.40:0/1620746158 >> v2:172.21.15.40:3300/0 conn(0x7f601006b150 0x7f601006b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.385+0000 7f6017c00640 1 -- 172.21.15.40:0/1620746158 >> 172.21.15.40:0/1620746158 conn(0x7f60100d1c90 msgr2=0x7f60100d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.385+0000 7f6017c00640 1 -- 172.21.15.40:0/1620746158 shutdown_connections 2024-03-24T00:37:44.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.385+0000 7f6017c00640 1 -- 172.21.15.40:0/1620746158 wait complete. 2024-03-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f6017c00640 1 Processor -- start 2024-03-24T00:37:44.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f6017c00640 1 -- start start 2024-03-24T00:37:44.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f6017c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f601006b150 0x7f60100c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f6017c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60100d0830 0x7f6010161200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f6017c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f601015e0b0 0x7f60100c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f6017c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601015e490 con 0x7f60100d0830 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f6017c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60100ce020 con 0x7f601015e0b0 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f6017c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601015e610 con 0x7f601006b150 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f600fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60100d0830 0x7f6010161200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f600fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60100d0830 0x7f6010161200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51942/0 (socket says 172.21.15.40:51942) 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f600fe00640 1 -- 172.21.15.40:0/2435086730 learned_addr learned my addr 172.21.15.40:0/2435086730 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f6015c00640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f601015e0b0 0x7f60100c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f6015200640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f601006b150 0x7f60100c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f600fe00640 1 -- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f601006b150 msgr2=0x7f60100c57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f600fe00640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f601006b150 0x7f60100c57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f600fe00640 1 -- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f601015e0b0 msgr2=0x7f60100c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f600fe00640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f601015e0b0 0x7f60100c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f600fe00640 1 -- 172.21.15.40:0/2435086730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60100c7eb0 con 0x7f60100d0830 2024-03-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f600fe00640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60100d0830 0x7f6010161200 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0ae040 tx=0x7f5ffc019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f600cc00640 1 -- 172.21.15.40:0/2435086730 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0b0040 con 0x7f60100d0830 2024-03-24T00:37:44.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f600cc00640 1 -- 172.21.15.40:0/2435086730 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ffc0ba070 con 0x7f60100d0830 2024-03-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f6017c00640 1 -- 172.21.15.40:0/2435086730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60100c5da0 con 0x7f60100d0830 2024-03-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f600cc00640 1 -- 172.21.15.40:0/2435086730 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc006e20 con 0x7f60100d0830 2024-03-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.389+0000 7f6017c00640 1 -- 172.21.15.40:0/2435086730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60100c6360 con 0x7f60100d0830 2024-03-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.393+0000 7f6017c00640 1 -- 172.21.15.40:0/2435086730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6010068170 con 0x7f60100d0830 2024-03-24T00:37:44.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.393+0000 7f600cc00640 1 -- 172.21.15.40:0/2435086730 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5ffc0b4670 con 0x7f60100d0830 2024-03-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.393+0000 7f600cc00640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5ff0067690 0x7f5ff0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.393+0000 7f600cc00640 1 -- 172.21.15.40:0/2435086730 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ffc11f2e0 con 0x7f60100d0830 2024-03-24T00:37:44.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.393+0000 7f6015200640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5ff0067690 0x7f5ff0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:44.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.393+0000 7f6015200640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5ff0067690 0x7f5ff0069b50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6004007fd0 tx=0x7f6004009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:44.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.397+0000 7f600cc00640 1 -- 172.21.15.40:0/2435086730 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f60100c7eb0 con 0x7f60100d0830 2024-03-24T00:37:44.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.569+0000 7f6017c00640 1 -- 172.21.15.40:0/2435086730 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f60100d0390 con 0x7f60100d0830 2024-03-24T00:37:44.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f600cc00640 1 -- 172.21.15.40:0/2435086730 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v95) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5ffc016090 con 0x7f60100d0830 2024-03-24T00:37:44.575 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:37:44.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f6017c00640 1 -- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5ff0067690 msgr2=0x7f5ff0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f6017c00640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5ff0067690 0x7f5ff0069b50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6004007fd0 tx=0x7f6004009040 comp rx=0 tx=0).stop 2024-03-24T00:37:44.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f6017c00640 1 -- 172.21.15.40:0/2435086730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60100d0830 msgr2=0x7f6010161200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:44.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f6017c00640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60100d0830 0x7f6010161200 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc0ae040 tx=0x7f5ffc019240 comp rx=0 tx=0).stop 2024-03-24T00:37:44.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f6017c00640 1 -- 172.21.15.40:0/2435086730 shutdown_connections 2024-03-24T00:37:44.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f6017c00640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5ff0067690 0x7f5ff0069b50 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f6017c00640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f601015e0b0 0x7f60100c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f6017c00640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f60100d0830 0x7f6010161200 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f6017c00640 1 --2- 172.21.15.40:0/2435086730 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f601006b150 0x7f60100c57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:44.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f6017c00640 1 -- 172.21.15.40:0/2435086730 >> 172.21.15.40:0/2435086730 conn(0x7f60100d1c90 msgr2=0x7f60100d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:44.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f6017c00640 1 -- 172.21.15.40:0/2435086730 shutdown_connections 2024-03-24T00:37:44.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:44.573+0000 7f6017c00640 1 -- 172.21.15.40:0/2435086730 wait complete. 2024-03-24T00:37:44.596 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-03-24T00:37:44.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:44.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T00:37:44.614 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[48527]: fuse finished with error 0 and tester_r 0 2024-03-24T00:37:44.618 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:37:44.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:44.618 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:37:44.793 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:37:44.793 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:37:44.801 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:37:44.802 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:37:44.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:44.802 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T00:37:44.850 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T00:37:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:37:44.851 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:37:44.851 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:37:44.859 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:37:44.859 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:37:44.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:44.860 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T00:37:44.914 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T00:37:44.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:37:44.915 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:37:44.915 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:37:44.923 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:37:44.923 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:37:44.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:37:44.923 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T00:37:44.974 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T00:37:44.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:37:44.975 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:45.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3ffa00640 1 Processor -- start 2024-03-24T00:37:45.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3ffa00640 1 -- start start 2024-03-24T00:37:45.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3ffa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb3f80cfdf0 0x7fb3f80d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3ffa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 0x7fb3f80d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3ffa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3f815f090 0x7fb3f815f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3ffa00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f80b2bb0 con 0x7fb3f815f090 2024-03-24T00:37:45.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3ffa00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f80b2fe0 con 0x7fb3f80cfdf0 2024-03-24T00:37:45.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3ffa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f80b32a0 con 0x7fb3f80b43a0 2024-03-24T00:37:45.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3fda00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3f815f090 0x7fb3f815f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3fd000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb3f80cfdf0 0x7fb3f80d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3f7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 0x7fb3f80d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3fd000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb3f80cfdf0 0x7fb3f80d01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:50444/0 (socket says 172.21.15.40:50444) 2024-03-24T00:37:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3fda00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb3f815f090 0x7fb3f815f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51944/0 (socket says 172.21.15.40:51944) 2024-03-24T00:37:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3f7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 0x7fb3f80d0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:39002/0 (socket says 172.21.15.40:39002) 2024-03-24T00:37:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3fd000640 1 -- 172.21.15.40:0/181253995 learned_addr learned my addr 172.21.15.40:0/181253995 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3fd000640 1 -- 172.21.15.40:0/181253995 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 msgr2=0x7fb3f80d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3fd000640 1 --2- 172.21.15.40:0/181253995 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 0x7fb3f80d0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3fd000640 1 -- 172.21.15.40:0/181253995 >> v2:172.21.15.40:3300/0 conn(0x7fb3f815f090 msgr2=0x7fb3f815f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3fd000640 1 --2- 172.21.15.40:0/181253995 >> v2:172.21.15.40:3300/0 conn(0x7fb3f815f090 0x7fb3f815f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3fd000640 1 -- 172.21.15.40:0/181253995 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f80af710 con 0x7fb3f80cfdf0 2024-03-24T00:37:45.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.113+0000 7fb3f7e00640 1 --2- 172.21.15.40:0/181253995 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 0x7fb3f80d0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:45.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.117+0000 7fb3fd000640 1 --2- 172.21.15.40:0/181253995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f80cfdf0 0x7fb3f80d01d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8071950 tx=0x7fb3ec01c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51a5f83107fc2b83 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:45.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.117+0000 7fb3f7400640 1 -- 172.21.15.40:0/181253995 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec0a6040 con 0x7fb3f80cfdf0 2024-03-24T00:37:45.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.117+0000 7fb3f7400640 1 -- 172.21.15.40:0/181253995 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ec0af070 con 0x7fb3f80cfdf0 2024-03-24T00:37:45.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.117+0000 7fb3f7400640 1 -- 172.21.15.40:0/181253995 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec0aa860 con 0x7fb3f80cfdf0 2024-03-24T00:37:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.117+0000 7fb3ffa00640 1 -- 172.21.15.40:0/181253995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f80cfdf0 msgr2=0x7fb3f80d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.117+0000 7fb3ffa00640 1 --2- 172.21.15.40:0/181253995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f80cfdf0 0x7fb3f80d01d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8071950 tx=0x7fb3ec01c520 comp rx=0 tx=0).stop 2024-03-24T00:37:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.117+0000 7fb3ffa00640 1 -- 172.21.15.40:0/181253995 shutdown_connections 2024-03-24T00:37:45.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.117+0000 7fb3ffa00640 1 --2- 172.21.15.40:0/181253995 >> v2:172.21.15.40:3300/0 conn(0x7fb3f815f090 0x7fb3f815f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.117+0000 7fb3ffa00640 1 --2- 172.21.15.40:0/181253995 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 0x7fb3f80d0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.117+0000 7fb3ffa00640 1 --2- 172.21.15.40:0/181253995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f80cfdf0 0x7fb3f80d01d0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.117+0000 7fb3ffa00640 1 -- 172.21.15.40:0/181253995 >> 172.21.15.40:0/181253995 conn(0x7fb3f80b3880 msgr2=0x7fb3f80b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:45.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3ffa00640 1 -- 172.21.15.40:0/181253995 shutdown_connections 2024-03-24T00:37:45.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3ffa00640 1 -- 172.21.15.40:0/181253995 wait complete. 2024-03-24T00:37:45.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3ffa00640 1 Processor -- start 2024-03-24T00:37:45.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3ffa00640 1 -- start start 2024-03-24T00:37:45.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3ffa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 0x7fb3f8158340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3ffa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f80cfdf0 0x7fb3f815c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3ffa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f815f090 0x7fb3f8157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3ffa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f80b2020 con 0x7fb3f80cfdf0 2024-03-24T00:37:45.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3ffa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f80b1ea0 con 0x7fb3f815f090 2024-03-24T00:37:45.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3ffa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f80b21a0 con 0x7fb3f80b43a0 2024-03-24T00:37:45.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f80cfdf0 0x7fb3f815c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3fd000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 0x7fb3f8158340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3fd000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 0x7fb3f8158340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:39010/0 (socket says 172.21.15.40:39010) 2024-03-24T00:37:45.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3fd000640 1 -- 172.21.15.40:0/3268970372 learned_addr learned my addr 172.21.15.40:0/3268970372 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:45.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3fda00640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f815f090 0x7fb3f8157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3fda00640 1 -- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 msgr2=0x7fb3f8158340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3fda00640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 0x7fb3f8158340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3fda00640 1 -- 172.21.15.40:0/3268970372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f80cfdf0 msgr2=0x7fb3f815c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3fda00640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f80cfdf0 0x7fb3f815c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3fda00640 1 -- 172.21.15.40:0/3268970372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f80d1880 con 0x7fb3f815f090 2024-03-24T00:37:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3f7e00640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f80cfdf0 0x7fb3f815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:45.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3fd000640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 0x7fb3f8158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3fda00640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f815f090 0x7fb3f8157ad0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000eba0 tx=0x7fb3f000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.121+0000 7fb3f4c00640 1 -- 172.21.15.40:0/3268970372 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0018020 con 0x7fb3f815f090 2024-03-24T00:37:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.125+0000 7fb3f4c00640 1 -- 172.21.15.40:0/3268970372 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f000cd20 con 0x7fb3f815f090 2024-03-24T00:37:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.125+0000 7fb3f4c00640 1 -- 172.21.15.40:0/3268970372 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f00146c0 con 0x7fb3f815f090 2024-03-24T00:37:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.125+0000 7fb3ffa00640 1 -- 172.21.15.40:0/3268970372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f815c830 con 0x7fb3f815f090 2024-03-24T00:37:45.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.125+0000 7fb3ffa00640 1 -- 172.21.15.40:0/3268970372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f815cdb0 con 0x7fb3f815f090 2024-03-24T00:37:45.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.125+0000 7fb3ffa00640 1 -- 172.21.15.40:0/3268970372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f8068170 con 0x7fb3f815f090 2024-03-24T00:37:45.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.125+0000 7fb3f4c00640 1 -- 172.21.15.40:0/3268970372 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb3f001b020 con 0x7fb3f815f090 2024-03-24T00:37:45.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.125+0000 7fb3f4c00640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3d8067660 0x7fb3d8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.125+0000 7fb3f4c00640 1 -- 172.21.15.40:0/3268970372 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3f00863f0 con 0x7fb3f815f090 2024-03-24T00:37:45.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.125+0000 7fb3fd000640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3d8067660 0x7fb3d8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.125+0000 7fb3fd000640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3d8067660 0x7fb3d8069b20 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb3f806d110 tx=0x7fb3ec0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:45.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.129+0000 7fb3f4c00640 1 -- 172.21.15.40:0/3268970372 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb3f0022030 con 0x7fb3f815f090 2024-03-24T00:37:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.333+0000 7fb3ffa00640 1 -- 172.21.15.40:0/3268970372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3f80d0f80 con 0x7fb3f815f090 2024-03-24T00:37:45.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.333+0000 7fb3f4c00640 1 -- 172.21.15.40:0/3268970372 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5021 (secure 0 0 0) 0x7fb3f0010040 con 0x7fb3f815f090 2024-03-24T00:37:45.338 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:45.338 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":103,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6755,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2936601899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2936601899},{"type":"v1","addr":"172.21.15.53:6836","nonce":2936601899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:37:01.566949+0000","modified":"2024-03-24T00:37:27.349073+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7932,"mds_1":7917,"mds_2":7633},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7633":{"gid":7633,"name":"a","rank":2,"incarnation":101,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6835/1997904254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1997904254},{"type":"v1","addr":"172.21.15.40:6835","nonce":1997904254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7917":{"gid":7917,"name":"c","rank":1,"incarnation":99,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6834/1019660586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1019660586},{"type":"v1","addr":"172.21.15.40:6834","nonce":1019660586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7932":{"gid":7932,"name":"d","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/3095501397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3095501397},{"type":"v1","addr":"172.21.15.53:6837","nonce":3095501397}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7917,7932,7633]},"id":7}]} 2024-03-24T00:37:45.338 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 103 2024-03-24T00:37:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.337+0000 7fb3ffa00640 1 -- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3d8067660 msgr2=0x7fb3d8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.337+0000 7fb3ffa00640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3d8067660 0x7fb3d8069b20 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb3f806d110 tx=0x7fb3ec0a7000 comp rx=0 tx=0).stop 2024-03-24T00:37:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.337+0000 7fb3ffa00640 1 -- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f815f090 msgr2=0x7fb3f8157ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.337+0000 7fb3ffa00640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f815f090 0x7fb3f8157ad0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000eba0 tx=0x7fb3f000c490 comp rx=0 tx=0).stop 2024-03-24T00:37:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.337+0000 7fb3ffa00640 1 -- 172.21.15.40:0/3268970372 shutdown_connections 2024-03-24T00:37:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.337+0000 7fb3ffa00640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb3d8067660 0x7fb3d8069b20 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.337+0000 7fb3ffa00640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3f815f090 0x7fb3f8157ad0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.337+0000 7fb3ffa00640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb3f80cfdf0 0x7fb3f815c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.337+0000 7fb3ffa00640 1 --2- 172.21.15.40:0/3268970372 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb3f80b43a0 0x7fb3f8158340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.337+0000 7fb3ffa00640 1 -- 172.21.15.40:0/3268970372 >> 172.21.15.40:0/3268970372 conn(0x7fb3f80b3880 msgr2=0x7fb3f80afbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.337+0000 7fb3ffa00640 1 -- 172.21.15.40:0/3268970372 shutdown_connections 2024-03-24T00:37:45.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.337+0000 7fb3ffa00640 1 -- 172.21.15.40:0/3268970372 wait complete. 2024-03-24T00:37:45.362 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:45.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.497+0000 7f4c1fa00640 1 Processor -- start 2024-03-24T00:37:45.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.497+0000 7f4c1fa00640 1 -- start start 2024-03-24T00:37:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.501+0000 7f4c1fa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4c180b6740 0x7f4c180ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.501+0000 7f4c1fa00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1806e860 con 0x7f4c18159850 2024-03-24T00:37:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.501+0000 7f4c1fa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1806eb40 con 0x7f4c18159c30 2024-03-24T00:37:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.501+0000 7f4c1fa00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c1806ee20 con 0x7f4c180b6740 2024-03-24T00:37:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.501+0000 7f4c1d000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4c18159850 0x7f4c1815fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53612/0 (socket says 172.21.15.40:53612) 2024-03-24T00:37:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.501+0000 7f4c1d000640 1 -- 172.21.15.40:0/1827762010 learned_addr learned my addr 172.21.15.40:0/1827762010 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:45.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.501+0000 7f4c1da00640 1 --2- 172.21.15.40:0/1827762010 >> v2:172.21.15.53:3300/0 conn(0x7f4c180b6740 0x7f4c180ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.501+0000 7f4c17400640 1 -- 172.21.15.40:0/1827762010 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193142843 0 0) 0x7f4c1806e860 con 0x7f4c18159850 2024-03-24T00:37:45.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.501+0000 7f4c17400640 1 -- 172.21.15.40:0/1827762010 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c00003530 con 0x7f4c18159850 2024-03-24T00:37:45.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.501+0000 7f4c1da00640 1 -- 172.21.15.40:0/1827762010 >> v1:172.21.15.53:6789/0 conn(0x7f4c18159c30 legacy=0x7f4c180b6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.501+0000 7f4c1da00640 1 -- 172.21.15.40:0/1827762010 >> v1:172.21.15.40:6789/0 conn(0x7f4c18159850 legacy=0x7f4c1815fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.501+0000 7f4c1da00640 1 -- 172.21.15.40:0/1827762010 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1806f100 con 0x7f4c180b6740 2024-03-24T00:37:45.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.505+0000 7f4c1da00640 1 --2- 172.21.15.40:0/1827762010 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c180b6740 0x7f4c180ceaf0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009d90 tx=0x7f4c1001c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2884b19d9d8102 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:45.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.505+0000 7f4c17400640 1 -- 172.21.15.40:0/1827762010 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c100b2070 con 0x7f4c180b6740 2024-03-24T00:37:45.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.505+0000 7f4c17400640 1 -- 172.21.15.40:0/1827762010 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c100a5880 con 0x7f4c180b6740 2024-03-24T00:37:45.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.505+0000 7f4c17400640 1 -- 172.21.15.40:0/1827762010 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c100a5c30 con 0x7f4c180b6740 2024-03-24T00:37:45.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.505+0000 7f4c1fa00640 1 -- 172.21.15.40:0/1827762010 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c180b6740 msgr2=0x7f4c180ceaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.505+0000 7f4c1fa00640 1 --2- 172.21.15.40:0/1827762010 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c180b6740 0x7f4c180ceaf0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009d90 tx=0x7f4c1001c8b0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.505+0000 7f4c1fa00640 1 -- 172.21.15.40:0/1827762010 shutdown_connections 2024-03-24T00:37:45.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.505+0000 7f4c1fa00640 1 --2- 172.21.15.40:0/1827762010 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c180b6740 0x7f4c180ceaf0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.505+0000 7f4c1fa00640 1 -- 172.21.15.40:0/1827762010 >> 172.21.15.40:0/1827762010 conn(0x7f4c180b2a20 msgr2=0x7f4c180b2e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.505+0000 7f4c1fa00640 1 -- 172.21.15.40:0/1827762010 shutdown_connections 2024-03-24T00:37:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.505+0000 7f4c1fa00640 1 -- 172.21.15.40:0/1827762010 wait complete. 2024-03-24T00:37:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1fa00640 1 Processor -- start 2024-03-24T00:37:45.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1fa00640 1 -- start start 2024-03-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1fa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c180b6740 0x7f4c181508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1fa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c18159850 0x7f4c1815e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1fa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c18159c30 0x7f4c1815ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1fa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1815d3e0 con 0x7f4c18159c30 2024-03-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1fa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1815d260 con 0x7f4c18159850 2024-03-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1fa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c18169c10 con 0x7f4c180b6740 2024-03-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c18159c30 0x7f4c1815ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c17e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c18159850 0x7f4c1815e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1da00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c18159c30 0x7f4c1815ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51956/0 (socket says 172.21.15.40:51956) 2024-03-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1da00640 1 -- 172.21.15.40:0/274473572 learned_addr learned my addr 172.21.15.40:0/274473572 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:45.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1d000640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c180b6740 0x7f4c181508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c17e00640 1 -- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c180b6740 msgr2=0x7f4c181508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c17e00640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c180b6740 0x7f4c181508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c17e00640 1 -- 172.21.15.40:0/274473572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c18159c30 msgr2=0x7f4c1815ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c17e00640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c18159c30 0x7f4c1815ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c17e00640 1 -- 172.21.15.40:0/274473572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c180b1de0 con 0x7f4c18159850 2024-03-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1da00640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c18159c30 0x7f4c1815ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c17e00640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c18159850 0x7f4c1815e710 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f4c04000f70 tx=0x7f4c0400c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c14c00640 1 -- 172.21.15.40:0/274473572 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04017070 con 0x7f4c18159850 2024-03-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c14c00640 1 -- 172.21.15.40:0/274473572 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c040052d0 con 0x7f4c18159850 2024-03-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c1fa00640 1 -- 172.21.15.40:0/274473572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c18156d30 con 0x7f4c18159850 2024-03-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.509+0000 7f4c14c00640 1 -- 172.21.15.40:0/274473572 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04003840 con 0x7f4c18159850 2024-03-24T00:37:45.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.513+0000 7f4c1fa00640 1 -- 172.21.15.40:0/274473572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c181572f0 con 0x7f4c18159850 2024-03-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.513+0000 7f4c1fa00640 1 -- 172.21.15.40:0/274473572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c18068170 con 0x7f4c18159850 2024-03-24T00:37:45.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.513+0000 7f4c14c00640 1 -- 172.21.15.40:0/274473572 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4c0401a070 con 0x7f4c18159850 2024-03-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.513+0000 7f4c14c00640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4bf8067690 0x7f4bf8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.513+0000 7f4c14c00640 1 -- 172.21.15.40:0/274473572 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c04084b00 con 0x7f4c18159850 2024-03-24T00:37:45.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.513+0000 7f4c1d000640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4bf8067690 0x7f4bf8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.513+0000 7f4c1d000640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4bf8067690 0x7f4bf8069b50 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c0011f0 tx=0x7f4c0c007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:45.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.521+0000 7f4c14c00640 1 -- 172.21.15.40:0/274473572 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4c04005480 con 0x7f4c18159850 2024-03-24T00:37:45.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.685+0000 7f4c1fa00640 1 -- 172.21.15.40:0/274473572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c181574c0 con 0x7f4c18159850 2024-03-24T00:37:45.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c14c00640 1 -- 172.21.15.40:0/274473572 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16548 (secure 0 0 0) 0x7f4c040550a0 con 0x7f4c18159850 2024-03-24T00:37:45.693 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:45.693 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":95,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:37:03.189247+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:59.977310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-24T00:37:00.557420+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:45.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c1fa00640 1 -- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4bf8067690 msgr2=0x7f4bf8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c1fa00640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4bf8067690 0x7f4bf8069b50 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c0011f0 tx=0x7f4c0c007d40 comp rx=0 tx=0).stop 2024-03-24T00:37:45.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c1fa00640 1 -- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c18159850 msgr2=0x7f4c1815e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c1fa00640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c18159850 0x7f4c1815e710 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f4c04000f70 tx=0x7f4c0400c820 comp rx=0 tx=0).stop 2024-03-24T00:37:45.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c1fa00640 1 -- 172.21.15.40:0/274473572 shutdown_connections 2024-03-24T00:37:45.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c1fa00640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4bf8067690 0x7f4bf8069b50 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c1fa00640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c18159c30 0x7f4c1815ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c1fa00640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c18159850 0x7f4c1815e710 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c1fa00640 1 --2- 172.21.15.40:0/274473572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c180b6740 0x7f4c181508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c1fa00640 1 -- 172.21.15.40:0/274473572 >> 172.21.15.40:0/274473572 conn(0x7f4c180b2a20 msgr2=0x7f4c1815a030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:45.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c1fa00640 1 -- 172.21.15.40:0/274473572 shutdown_connections 2024-03-24T00:37:45.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.689+0000 7f4c1fa00640 1 -- 172.21.15.40:0/274473572 wait complete. 2024-03-24T00:37:45.715 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:36:59.977310+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:37:00.557420+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:37:45.715 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:37:45.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6eaca00640 1 Processor -- start 2024-03-24T00:37:45.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6eaca00640 1 -- start start 2024-03-24T00:37:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6eaca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6ea815e0e0 0x7f6ea80cd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6eaca00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ea80d0070 0x7f6ea80d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6eaca00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea80cf2e0 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6eaca00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea80d0a20 con 0x7f6ea80d0070 2024-03-24T00:37:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6eaca00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea80cd0e0 con 0x7f6ea815e0e0 2024-03-24T00:37:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6ea5e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f6ea80b0bd0 0x7f6ea80ceab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53628/0 (socket says 172.21.15.40:53628) 2024-03-24T00:37:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6ea5e00640 1 -- 172.21.15.40:0/3861731206 learned_addr learned my addr 172.21.15.40:0/3861731206 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6ea7200640 1 --2- 172.21.15.40:0/3861731206 >> v2:172.21.15.40:3300/0 conn(0x7f6ea80d0070 0x7f6ea80d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6ea6800640 1 --2- 172.21.15.40:0/3861731206 >> v2:172.21.15.53:3300/0 conn(0x7f6ea815e0e0 0x7f6ea80cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6ea5400640 1 -- 172.21.15.40:0/3861731206 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013196858 0 0) 0x7f6ea80cf2e0 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6ea5400640 1 -- 172.21.15.40:0/3861731206 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e8c003530 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6ea7200640 1 -- 172.21.15.40:0/3861731206 >> v2:172.21.15.53:3300/0 conn(0x7f6ea815e0e0 msgr2=0x7f6ea80cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6ea7200640 1 --2- 172.21.15.40:0/3861731206 >> v2:172.21.15.53:3300/0 conn(0x7f6ea815e0e0 0x7f6ea80cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6ea7200640 1 -- 172.21.15.40:0/3861731206 >> v1:172.21.15.40:6789/0 conn(0x7f6ea80b0bd0 legacy=0x7f6ea80ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.857+0000 7f6ea7200640 1 -- 172.21.15.40:0/3861731206 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea80cc7b0 con 0x7f6ea80d0070 2024-03-24T00:37:45.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.861+0000 7f6ea7200640 1 --2- 172.21.15.40:0/3861731206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea80d0070 0x7f6ea80d04e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8072280 tx=0x7f6ea001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fb1a1bfaf23537 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:45.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.861+0000 7f6ea5400640 1 -- 172.21.15.40:0/3861731206 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00b2070 con 0x7f6ea80d0070 2024-03-24T00:37:45.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.861+0000 7f6ea5400640 1 -- 172.21.15.40:0/3861731206 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea00a5900 con 0x7f6ea80d0070 2024-03-24T00:37:45.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.861+0000 7f6ea5400640 1 -- 172.21.15.40:0/3861731206 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00a5cd0 con 0x7f6ea80d0070 2024-03-24T00:37:45.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.861+0000 7f6eaca00640 1 -- 172.21.15.40:0/3861731206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea80d0070 msgr2=0x7f6ea80d04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.861+0000 7f6eaca00640 1 --2- 172.21.15.40:0/3861731206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea80d0070 0x7f6ea80d04e0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8072280 tx=0x7f6ea001c930 comp rx=0 tx=0).stop 2024-03-24T00:37:45.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.861+0000 7f6eaca00640 1 -- 172.21.15.40:0/3861731206 shutdown_connections 2024-03-24T00:37:45.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.861+0000 7f6eaca00640 1 --2- 172.21.15.40:0/3861731206 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea80d0070 0x7f6ea80d04e0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.861+0000 7f6eaca00640 1 --2- 172.21.15.40:0/3861731206 >> v2:172.21.15.53:3300/0 conn(0x7f6ea815e0e0 0x7f6ea80cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.861+0000 7f6eaca00640 1 -- 172.21.15.40:0/3861731206 >> 172.21.15.40:0/3861731206 conn(0x7f6ea80b3620 msgr2=0x7f6ea80b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:45.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.861+0000 7f6eaca00640 1 -- 172.21.15.40:0/3861731206 shutdown_connections 2024-03-24T00:37:45.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.861+0000 7f6eaca00640 1 -- 172.21.15.40:0/3861731206 wait complete. 2024-03-24T00:37:45.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6eaca00640 1 Processor -- start 2024-03-24T00:37:45.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6eaca00640 1 -- start start 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6eaca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea80b0bd0 0x7f6ea80ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6eaca00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ea80d0070 0x7f6ea81576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6eaca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ea815e0e0 0x7f6ea8157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6eaca00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea806bd30 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6eaca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea806bbb0 con 0x7f6ea815e0e0 2024-03-24T00:37:45.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6eaca00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea816ae20 con 0x7f6ea80d0070 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6ea6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea80b0bd0 0x7f6ea80ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6ea6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea80b0bd0 0x7f6ea80ce190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:51984/0 (socket says 172.21.15.40:51984) 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6ea6800640 1 -- 172.21.15.40:0/2981877592 learned_addr learned my addr 172.21.15.40:0/2981877592 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6ea5e00640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ea80d0070 0x7f6ea81576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6ea7200640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ea815e0e0 0x7f6ea8157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6ea6800640 1 -- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ea80d0070 msgr2=0x7f6ea81576c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6ea6800640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ea80d0070 0x7f6ea81576c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6ea6800640 1 -- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ea815e0e0 msgr2=0x7f6ea8157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6ea6800640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ea815e0e0 0x7f6ea8157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6ea6800640 1 -- 172.21.15.40:0/2981877592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea815f100 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6ea6800640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea80b0bd0 0x7f6ea80ce190 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800e930 tx=0x7f6e9800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6e9ea00640 1 -- 172.21.15.40:0/2981877592 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98017070 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6e9ea00640 1 -- 172.21.15.40:0/2981877592 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9800ca00 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.865+0000 7f6eaca00640 1 -- 172.21.15.40:0/2981877592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea815f680 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.869+0000 7f6e9ea00640 1 -- 172.21.15.40:0/2981877592 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9800cdb0 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.869+0000 7f6eaca00640 1 -- 172.21.15.40:0/2981877592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea8160d90 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.869+0000 7f6eaca00640 1 -- 172.21.15.40:0/2981877592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea804f410 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.869+0000 7f6e9ea00640 1 -- 172.21.15.40:0/2981877592 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6e9801a070 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.873+0000 7f6e9ea00640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6e88067660 0x7f6e88069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.873+0000 7f6e9ea00640 1 -- 172.21.15.40:0/2981877592 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e98085fe0 con 0x7f6ea80b0bd0 2024-03-24T00:37:45.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.873+0000 7f6ea5e00640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6e88067660 0x7f6e88069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:45.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.873+0000 7f6ea5e00640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6e88067660 0x7f6e88069b20 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6e94007fd0 tx=0x7f6e94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:45.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:45.873+0000 7f6e9ea00640 1 -- 172.21.15.40:0/2981877592 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6e980567a0 con 0x7f6ea80b0bd0 2024-03-24T00:37:46.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 -- 172.21.15.40:0/2981877592 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ea815a740 con 0x7f6ea80b0bd0 2024-03-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6e9ea00640 1 -- 172.21.15.40:0/2981877592 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e9805a840 con 0x7f6ea80b0bd0 2024-03-24T00:37:46.044 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 -- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6e88067660 msgr2=0x7f6e88069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6e88067660 0x7f6e88069b20 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6e94007fd0 tx=0x7f6e94009040 comp rx=0 tx=0).stop 2024-03-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 -- 172.21.15.40:0/2981877592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea80b0bd0 msgr2=0x7f6ea80ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea80b0bd0 0x7f6ea80ce190 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800e930 tx=0x7f6e9800edf0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 -- 172.21.15.40:0/2981877592 shutdown_connections 2024-03-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6e88067660 0x7f6e88069b20 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ea815e0e0 0x7f6ea8157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ea80d0070 0x7f6ea81576c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 --2- 172.21.15.40:0/2981877592 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ea80b0bd0 0x7f6ea80ce190 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 -- 172.21.15.40:0/2981877592 >> 172.21.15.40:0/2981877592 conn(0x7f6ea80b3620 msgr2=0x7f6ea806f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 -- 172.21.15.40:0/2981877592 shutdown_connections 2024-03-24T00:37:46.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.041+0000 7f6eaca00640 1 -- 172.21.15.40:0/2981877592 wait complete. 2024-03-24T00:37:46.066 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:37:46.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.205+0000 7f2140000640 1 Processor -- start 2024-03-24T00:37:46.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.205+0000 7f2140000640 1 -- start start 2024-03-24T00:37:46.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f2140000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f21380aec70 0x7f21380ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f2140000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f21380b3fc0 0x7f21380b4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f2140000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2138070ed0 con 0x7f21380cc4f0 2024-03-24T00:37:46.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f2140000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21380cf240 con 0x7f21380aec70 2024-03-24T00:37:46.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f2140000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21380cf520 con 0x7f21380b3fc0 2024-03-24T00:37:46.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f213d600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f21380aec70 0x7f21380ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f213e000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f21380b3fc0 0x7f21380b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f213d600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f21380aec70 0x7f21380ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52000/0 (socket says 172.21.15.40:52000) 2024-03-24T00:37:46.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f213d600640 1 -- 172.21.15.40:0/1202978033 learned_addr learned my addr 172.21.15.40:0/1202978033 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:46.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f213e000640 1 -- 172.21.15.40:0/1202978033 >> v2:172.21.15.40:3300/0 conn(0x7f21380aec70 msgr2=0x7f21380ae6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:46.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f213e000640 1 --2- 172.21.15.40:0/1202978033 >> v2:172.21.15.40:3300/0 conn(0x7f21380aec70 0x7f21380ae6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f213e000640 1 -- 172.21.15.40:0/1202978033 >> v1:172.21.15.40:6789/0 conn(0x7f21380cc4f0 legacy=0x7f213815a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.209+0000 7f213e000640 1 -- 172.21.15.40:0/1202978033 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21380b3aa0 con 0x7f21380b3fc0 2024-03-24T00:37:46.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.213+0000 7f213e000640 1 --2- 172.21.15.40:0/1202978033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21380b3fc0 0x7f21380b4450 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f21380ae640 tx=0x7f213001c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ad3435e1c8b9475 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:46.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.213+0000 7f2137e00640 1 -- 172.21.15.40:0/1202978033 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21300b2070 con 0x7f21380b3fc0 2024-03-24T00:37:46.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.213+0000 7f2137e00640 1 -- 172.21.15.40:0/1202978033 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21300a5770 con 0x7f21380b3fc0 2024-03-24T00:37:46.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.213+0000 7f2137e00640 1 -- 172.21.15.40:0/1202978033 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21300a5b40 con 0x7f21380b3fc0 2024-03-24T00:37:46.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.213+0000 7f2140000640 1 -- 172.21.15.40:0/1202978033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21380b3fc0 msgr2=0x7f21380b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.213+0000 7f2140000640 1 --2- 172.21.15.40:0/1202978033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21380b3fc0 0x7f21380b4450 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f21380ae640 tx=0x7f213001c810 comp rx=0 tx=0).stop 2024-03-24T00:37:46.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.213+0000 7f2140000640 1 -- 172.21.15.40:0/1202978033 shutdown_connections 2024-03-24T00:37:46.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.213+0000 7f2140000640 1 --2- 172.21.15.40:0/1202978033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21380b3fc0 0x7f21380b4450 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.213+0000 7f2140000640 1 --2- 172.21.15.40:0/1202978033 >> v2:172.21.15.40:3300/0 conn(0x7f21380aec70 0x7f21380ae6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.213+0000 7f2140000640 1 -- 172.21.15.40:0/1202978033 >> 172.21.15.40:0/1202978033 conn(0x7f21380b1660 msgr2=0x7f21380b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:46.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.213+0000 7f2140000640 1 -- 172.21.15.40:0/1202978033 shutdown_connections 2024-03-24T00:37:46.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.213+0000 7f2140000640 1 -- 172.21.15.40:0/1202978033 wait complete. 2024-03-24T00:37:46.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f2140000640 1 Processor -- start 2024-03-24T00:37:46.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f2140000640 1 -- start start 2024-03-24T00:37:46.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f2140000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21380aec70 0x7f2138155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f2140000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21380b3fc0 0x7f2138151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f2140000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21380cc4f0 0x7f213806cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f2140000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213815fdb0 con 0x7f21380cc4f0 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f2140000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213815fc30 con 0x7f21380aec70 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f2140000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213815ff30 con 0x7f21380b3fc0 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21380cc4f0 0x7f213806cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21380aec70 0x7f2138155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21380b3fc0 0x7f2138151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213e000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21380cc4f0 0x7f213806cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52002/0 (socket says 172.21.15.40:52002) 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213e000640 1 -- 172.21.15.40:0/1244605682 learned_addr learned my addr 172.21.15.40:0/1244605682 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21380aec70 0x7f2138155d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:50512/0 (socket says 172.21.15.40:50512) 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213e000640 1 -- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21380b3fc0 msgr2=0x7f2138151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213e000640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21380b3fc0 0x7f2138151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213e000640 1 -- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21380aec70 msgr2=0x7f2138155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213e000640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21380aec70 0x7f2138155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213e000640 1 -- 172.21.15.40:0/1244605682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2138154450 con 0x7f21380cc4f0 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213d600640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21380aec70 0x7f2138155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:46.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f213e000640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21380cc4f0 0x7f213806cb10 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f213001a440 tx=0x7f21300195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:46.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f2135600640 1 -- 172.21.15.40:0/1244605682 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21300bd020 con 0x7f21380cc4f0 2024-03-24T00:37:46.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f2135600640 1 -- 172.21.15.40:0/1244605682 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21300b8470 con 0x7f21380cc4f0 2024-03-24T00:37:46.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.217+0000 7f2135600640 1 -- 172.21.15.40:0/1244605682 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21300b2040 con 0x7f21380cc4f0 2024-03-24T00:37:46.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.221+0000 7f2140000640 1 -- 172.21.15.40:0/1244605682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2138154730 con 0x7f21380cc4f0 2024-03-24T00:37:46.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.221+0000 7f2140000640 1 -- 172.21.15.40:0/1244605682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f213815c650 con 0x7f21380cc4f0 2024-03-24T00:37:46.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.221+0000 7f2140000640 1 -- 172.21.15.40:0/1244605682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2138068170 con 0x7f21380cc4f0 2024-03-24T00:37:46.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.221+0000 7f2135600640 1 -- 172.21.15.40:0/1244605682 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f21300c0020 con 0x7f21380cc4f0 2024-03-24T00:37:46.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.221+0000 7f2135600640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2118067660 0x7f2118069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.221+0000 7f2135600640 1 -- 172.21.15.40:0/1244605682 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21301205c0 con 0x7f21380cc4f0 2024-03-24T00:37:46.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.221+0000 7f213d600640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2118067660 0x7f2118069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.225+0000 7f213d600640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2118067660 0x7f2118069b20 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f21380cc120 tx=0x7f212c014040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:46.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.225+0000 7f2135600640 1 -- 172.21.15.40:0/1244605682 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f21300c7030 con 0x7f21380cc4f0 2024-03-24T00:37:46.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 -- 172.21.15.40:0/1244605682 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2138071ba0 con 0x7f21380cc4f0 2024-03-24T00:37:46.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2135600640 1 -- 172.21.15.40:0/1244605682 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2130016520 con 0x7f21380cc4f0 2024-03-24T00:37:46.392 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:37:46.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 -- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2118067660 msgr2=0x7f2118069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2118067660 0x7f2118069b20 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f21380cc120 tx=0x7f212c014040 comp rx=0 tx=0).stop 2024-03-24T00:37:46.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 -- 172.21.15.40:0/1244605682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21380cc4f0 msgr2=0x7f213806cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21380cc4f0 0x7f213806cb10 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f213001a440 tx=0x7f21300195b0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 -- 172.21.15.40:0/1244605682 shutdown_connections 2024-03-24T00:37:46.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2118067660 0x7f2118069b20 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21380cc4f0 0x7f213806cb10 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21380b3fc0 0x7f2138151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 --2- 172.21.15.40:0/1244605682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21380aec70 0x7f2138155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 -- 172.21.15.40:0/1244605682 >> 172.21.15.40:0/1244605682 conn(0x7f21380b1660 msgr2=0x7f21380ce120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:46.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 -- 172.21.15.40:0/1244605682 shutdown_connections 2024-03-24T00:37:46.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.389+0000 7f2140000640 1 -- 172.21.15.40:0/1244605682 wait complete. 2024-03-24T00:37:46.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:37:46.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c4e200640 1 Processor -- start 2024-03-24T00:37:46.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c4e200640 1 -- start start 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c4e200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7c480d02c0 0x7f7c480b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c4e200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c480b0f10 con 0x7f7c4815f5c0 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c4e200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c480cd130 con 0x7f7c480b2120 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c4e200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4815f9a0 con 0x7f7c480d02c0 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c47400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7c4815f5c0 0x7f7c480cfbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53644/0 (socket says 172.21.15.40:53644) 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c47400640 1 -- 172.21.15.40:0/2660153644 learned_addr learned my addr 172.21.15.40:0/2660153644 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c3ea00640 1 --2- 172.21.15.40:0/2660153644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7c480d02c0 0x7f7c480b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c46a00640 1 -- 172.21.15.40:0/2660153644 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721568292 0 0) 0x7f7c38001120 con 0x7f7c4815f5c0 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c46a00640 1 -- 172.21.15.40:0/2660153644 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c2c003530 con 0x7f7c4815f5c0 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c46a00640 1 -- 172.21.15.40:0/2660153644 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486750403 0 0) 0x7f7c480cd130 con 0x7f7c480b2120 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c46a00640 1 -- 172.21.15.40:0/2660153644 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c38001120 con 0x7f7c480b2120 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c3ea00640 1 -- 172.21.15.40:0/2660153644 >> v1:172.21.15.53:6789/0 conn(0x7f7c480b2120 legacy=0x7f7c480ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c3ea00640 1 -- 172.21.15.40:0/2660153644 >> v1:172.21.15.40:6789/0 conn(0x7f7c4815f5c0 legacy=0x7f7c480cfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.553+0000 7f7c3ea00640 1 -- 172.21.15.40:0/2660153644 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c480cd410 con 0x7f7c480d02c0 2024-03-24T00:37:46.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.557+0000 7f7c3ea00640 1 --2- 172.21.15.40:0/2660153644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7c480d02c0 0x7f7c480b3060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7c300097f0 tx=0x7f7c3001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c58bbb0b9b1e4ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:46.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.557+0000 7f7c46a00640 1 -- 172.21.15.40:0/2660153644 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c300ab620 con 0x7f7c480d02c0 2024-03-24T00:37:46.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.557+0000 7f7c46a00640 1 -- 172.21.15.40:0/2660153644 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c300b0070 con 0x7f7c480d02c0 2024-03-24T00:37:46.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.557+0000 7f7c46a00640 1 -- 172.21.15.40:0/2660153644 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c300ab9e0 con 0x7f7c480d02c0 2024-03-24T00:37:46.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.557+0000 7f7c4e200640 1 -- 172.21.15.40:0/2660153644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7c480d02c0 msgr2=0x7f7c480b3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.557+0000 7f7c4e200640 1 --2- 172.21.15.40:0/2660153644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7c480d02c0 0x7f7c480b3060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7c300097f0 tx=0x7f7c3001c420 comp rx=0 tx=0).stop 2024-03-24T00:37:46.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.557+0000 7f7c4e200640 1 -- 172.21.15.40:0/2660153644 shutdown_connections 2024-03-24T00:37:46.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.557+0000 7f7c4e200640 1 --2- 172.21.15.40:0/2660153644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7c480d02c0 0x7f7c480b3060 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.557+0000 7f7c4e200640 1 -- 172.21.15.40:0/2660153644 >> 172.21.15.40:0/2660153644 conn(0x7f7c480b3810 msgr2=0x7f7c480b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:46.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.557+0000 7f7c4e200640 1 -- 172.21.15.40:0/2660153644 shutdown_connections 2024-03-24T00:37:46.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.557+0000 7f7c4e200640 1 -- 172.21.15.40:0/2660153644 wait complete. 2024-03-24T00:37:46.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c4e200640 1 Processor -- start 2024-03-24T00:37:46.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c4e200640 1 -- start start 2024-03-24T00:37:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c4e200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c480b2120 0x7f7c48163f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c4e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c480d02c0 0x7f7c48156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c4e200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7c4815f5c0 0x7f7c480cb520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c4e200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c480cf270 con 0x7f7c480b2120 2024-03-24T00:37:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c4e200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c480cf0f0 con 0x7f7c480d02c0 2024-03-24T00:37:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c4e200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c480cf3f0 con 0x7f7c4815f5c0 2024-03-24T00:37:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c47400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c480b2120 0x7f7c48163f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c47400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c480b2120 0x7f7c48163f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52012/0 (socket says 172.21.15.40:52012) 2024-03-24T00:37:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c47400640 1 -- 172.21.15.40:0/4190284294 learned_addr learned my addr 172.21.15.40:0/4190284294 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:46.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c3ea00640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c480d02c0 0x7f7c48156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c47e00640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7c4815f5c0 0x7f7c480cb520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c47400640 1 -- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7c4815f5c0 msgr2=0x7f7c480cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c47400640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7c4815f5c0 0x7f7c480cb520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c47400640 1 -- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c480d02c0 msgr2=0x7f7c48156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c47400640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c480d02c0 0x7f7c48156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c47400640 1 -- 172.21.15.40:0/4190284294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c480c94f0 con 0x7f7c480b2120 2024-03-24T00:37:46.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c47400640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c480b2120 0x7f7c48163f60 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800c690 tx=0x7f7c3800cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:46.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c3fe00640 1 -- 172.21.15.40:0/4190284294 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38017070 con 0x7f7c480b2120 2024-03-24T00:37:46.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c3fe00640 1 -- 172.21.15.40:0/4190284294 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c38005ae0 con 0x7f7c480b2120 2024-03-24T00:37:46.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c4e200640 1 -- 172.21.15.40:0/4190284294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c480afe50 con 0x7f7c480b2120 2024-03-24T00:37:46.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c3fe00640 1 -- 172.21.15.40:0/4190284294 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38007070 con 0x7f7c480b2120 2024-03-24T00:37:46.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.561+0000 7f7c4e200640 1 -- 172.21.15.40:0/4190284294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c480b00f0 con 0x7f7c480b2120 2024-03-24T00:37:46.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.565+0000 7f7c4e200640 1 -- 172.21.15.40:0/4190284294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c48068170 con 0x7f7c480b2120 2024-03-24T00:37:46.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.565+0000 7f7c3fe00640 1 -- 172.21.15.40:0/4190284294 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7c3801a070 con 0x7f7c480b2120 2024-03-24T00:37:46.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.565+0000 7f7c3fe00640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c24067660 0x7f7c24069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.565+0000 7f7c3fe00640 1 -- 172.21.15.40:0/4190284294 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c38084e90 con 0x7f7c480b2120 2024-03-24T00:37:46.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.565+0000 7f7c3ea00640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c24067660 0x7f7c24069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.565+0000 7f7c3ea00640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c24067660 0x7f7c24069b20 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f7c300af000 tx=0x7f7c300a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:46.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.569+0000 7f7c3fe00640 1 -- 172.21.15.40:0/4190284294 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7c480b0f10 con 0x7f7c480b2120 2024-03-24T00:37:46.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 -- 172.21.15.40:0/4190284294 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7c4815df30 con 0x7f7c480b2120 2024-03-24T00:37:46.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c3fe00640 1 -- 172.21.15.40:0/4190284294 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7c38086020 con 0x7f7c480b2120 2024-03-24T00:37:46.736 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:37:46.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 -- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c24067660 msgr2=0x7f7c24069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c24067660 0x7f7c24069b20 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f7c300af000 tx=0x7f7c300a7000 comp rx=0 tx=0).stop 2024-03-24T00:37:46.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 -- 172.21.15.40:0/4190284294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c480b2120 msgr2=0x7f7c48163f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c480b2120 0x7f7c48163f60 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800c690 tx=0x7f7c3800cb50 comp rx=0 tx=0).stop 2024-03-24T00:37:46.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 -- 172.21.15.40:0/4190284294 shutdown_connections 2024-03-24T00:37:46.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7c24067660 0x7f7c24069b20 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7c4815f5c0 0x7f7c480cb520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7c480d02c0 0x7f7c48156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 --2- 172.21.15.40:0/4190284294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7c480b2120 0x7f7c48163f60 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 -- 172.21.15.40:0/4190284294 >> 172.21.15.40:0/4190284294 conn(0x7f7c480b3810 msgr2=0x7f7c4815c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:46.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 -- 172.21.15.40:0/4190284294 shutdown_connections 2024-03-24T00:37:46.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.733+0000 7f7c4e200640 1 -- 172.21.15.40:0/4190284294 wait complete. 2024-03-24T00:37:46.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:46.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.893+0000 7f9148800640 1 Processor -- start 2024-03-24T00:37:46.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.893+0000 7f9148800640 1 -- start start 2024-03-24T00:37:46.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9148800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f914006e5a0 0x7f914006e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9148800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914006cdc0 con 0x7f9140070c80 2024-03-24T00:37:46.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9148800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914006eec0 con 0x7f9140071690 2024-03-24T00:37:46.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9148800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9140071160 con 0x7f914006e5a0 2024-03-24T00:37:46.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9145400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9140070c80 0x7f914006db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53654/0 (socket says 172.21.15.40:53654) 2024-03-24T00:37:46.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9145400640 1 -- 172.21.15.40:0/2061730564 learned_addr learned my addr 172.21.15.40:0/2061730564 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9145e00640 1 --2- 172.21.15.40:0/2061730564 >> v2:172.21.15.53:3300/0 conn(0x7f914006e5a0 0x7f914006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9144a00640 1 -- 172.21.15.40:0/2061730564 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000447231 0 0) 0x7f9130001120 con 0x7f9140070c80 2024-03-24T00:37:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9144a00640 1 -- 172.21.15.40:0/2061730564 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9128003530 con 0x7f9140070c80 2024-03-24T00:37:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9145e00640 1 -- 172.21.15.40:0/2061730564 >> v1:172.21.15.53:6789/0 conn(0x7f9140071690 legacy=0x7f91400b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9144a00640 1 -- 172.21.15.40:0/2061730564 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947550468 0 0) 0x7f914006eec0 con 0x7f9140071690 2024-03-24T00:37:46.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9145e00640 1 -- 172.21.15.40:0/2061730564 >> v1:172.21.15.40:6789/0 conn(0x7f9140070c80 legacy=0x7f914006db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.897+0000 7f9145e00640 1 -- 172.21.15.40:0/2061730564 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914006d040 con 0x7f914006e5a0 2024-03-24T00:37:46.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9145e00640 1 --2- 172.21.15.40:0/2061730564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914006e5a0 0x7f914006e980 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f91340097c0 tx=0x7f913401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=661688fb0a9dba84 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:46.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9144a00640 1 -- 172.21.15.40:0/2061730564 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340b0070 con 0x7f914006e5a0 2024-03-24T00:37:46.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9144a00640 1 -- 172.21.15.40:0/2061730564 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91340ab600 con 0x7f914006e5a0 2024-03-24T00:37:46.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9144a00640 1 -- 172.21.15.40:0/2061730564 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340ab9d0 con 0x7f914006e5a0 2024-03-24T00:37:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9148800640 1 -- 172.21.15.40:0/2061730564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914006e5a0 msgr2=0x7f914006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9148800640 1 --2- 172.21.15.40:0/2061730564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914006e5a0 0x7f914006e980 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f91340097c0 tx=0x7f913401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9148800640 1 -- 172.21.15.40:0/2061730564 shutdown_connections 2024-03-24T00:37:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9148800640 1 --2- 172.21.15.40:0/2061730564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f914006e5a0 0x7f914006e980 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9148800640 1 -- 172.21.15.40:0/2061730564 >> 172.21.15.40:0/2061730564 conn(0x7f91400b17b0 msgr2=0x7f91400b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9148800640 1 -- 172.21.15.40:0/2061730564 shutdown_connections 2024-03-24T00:37:46.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9148800640 1 -- 172.21.15.40:0/2061730564 wait complete. 2024-03-24T00:37:46.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9148800640 1 Processor -- start 2024-03-24T00:37:46.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.901+0000 7f9148800640 1 -- start start 2024-03-24T00:37:46.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9148800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f914006e5a0 0x7f91400ae350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9148800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9140070c80 0x7f9140156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9148800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9140071690 0x7f91400cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9148800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914016cae0 con 0x7f914006e5a0 2024-03-24T00:37:46.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9148800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914016c960 con 0x7f9140070c80 2024-03-24T00:37:46.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9148800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914016cc60 con 0x7f9140071690 2024-03-24T00:37:46.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9145e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f914006e5a0 0x7f91400ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9145e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f914006e5a0 0x7f91400ae350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:52026/0 (socket says 172.21.15.40:52026) 2024-03-24T00:37:46.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9145e00640 1 -- 172.21.15.40:0/3544657700 learned_addr learned my addr 172.21.15.40:0/3544657700 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:46.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9145400640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9140070c80 0x7f9140156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9146800640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9140071690 0x7f91400cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9145e00640 1 -- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9140071690 msgr2=0x7f91400cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9145e00640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9140071690 0x7f91400cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9145e00640 1 -- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9140070c80 msgr2=0x7f9140156950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:46.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9145e00640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9140070c80 0x7f9140156950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:46.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9145e00640 1 -- 172.21.15.40:0/3544657700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91400cf160 con 0x7f914006e5a0 2024-03-24T00:37:46.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9145e00640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f914006e5a0 0x7f91400ae350 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f91340097c0 tx=0x7f913401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:46.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f913a000640 1 -- 172.21.15.40:0/3544657700 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340bb020 con 0x7f914006e5a0 2024-03-24T00:37:46.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f913a000640 1 -- 172.21.15.40:0/3544657700 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91340b1040 con 0x7f914006e5a0 2024-03-24T00:37:46.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f913a000640 1 -- 172.21.15.40:0/3544657700 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340b0050 con 0x7f914006e5a0 2024-03-24T00:37:46.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9148800640 1 -- 172.21.15.40:0/3544657700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91400cc550 con 0x7f914006e5a0 2024-03-24T00:37:46.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9148800640 1 -- 172.21.15.40:0/3544657700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91400cf440 con 0x7f914006e5a0 2024-03-24T00:37:46.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.905+0000 7f9148800640 1 -- 172.21.15.40:0/3544657700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9140068170 con 0x7f914006e5a0 2024-03-24T00:37:46.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.909+0000 7f913a000640 1 -- 172.21.15.40:0/3544657700 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f91340be070 con 0x7f914006e5a0 2024-03-24T00:37:46.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.909+0000 7f913a000640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9124067660 0x7f9124069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:46.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.909+0000 7f913a000640 1 -- 172.21.15.40:0/3544657700 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f913411eee0 con 0x7f914006e5a0 2024-03-24T00:37:46.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.909+0000 7f9145400640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9124067660 0x7f9124069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:46.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.909+0000 7f9145400640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9124067660 0x7f9124069b20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9130005890 tx=0x7f9130009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:46.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:46.917+0000 7f913a000640 1 -- 172.21.15.40:0/3544657700 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9134016090 con 0x7f914006e5a0 2024-03-24T00:37:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 -- 172.21.15.40:0/3544657700 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91400b2fa0 con 0x7f914006e5a0 2024-03-24T00:37:47.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f913a000640 1 -- 172.21.15.40:0/3544657700 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5021 (secure 0 0 0) 0x7f91340efac0 con 0x7f914006e5a0 2024-03-24T00:37:47.121 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:47.121 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":103,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6755,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2936601899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2936601899},{"type":"v1","addr":"172.21.15.53:6836","nonce":2936601899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:37:01.566949+0000","modified":"2024-03-24T00:37:27.349073+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7932,"mds_1":7917,"mds_2":7633},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7633":{"gid":7633,"name":"a","rank":2,"incarnation":101,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6835/1997904254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1997904254},{"type":"v1","addr":"172.21.15.40:6835","nonce":1997904254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7917":{"gid":7917,"name":"c","rank":1,"incarnation":99,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6834/1019660586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1019660586},{"type":"v1","addr":"172.21.15.40:6834","nonce":1019660586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7932":{"gid":7932,"name":"d","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/3095501397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3095501397},{"type":"v1","addr":"172.21.15.53:6837","nonce":3095501397}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7917,7932,7633]},"id":7}]} 2024-03-24T00:37:47.121 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 103 2024-03-24T00:37:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 -- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9124067660 msgr2=0x7f9124069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9124067660 0x7f9124069b20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9130005890 tx=0x7f9130009040 comp rx=0 tx=0).stop 2024-03-24T00:37:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 -- 172.21.15.40:0/3544657700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f914006e5a0 msgr2=0x7f91400ae350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f914006e5a0 0x7f91400ae350 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f91340097c0 tx=0x7f913401ace0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 -- 172.21.15.40:0/3544657700 shutdown_connections 2024-03-24T00:37:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9124067660 0x7f9124069b20 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9140071690 0x7f91400cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9140070c80 0x7f9140156950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 --2- 172.21.15.40:0/3544657700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f914006e5a0 0x7f91400ae350 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 -- 172.21.15.40:0/3544657700 >> 172.21.15.40:0/3544657700 conn(0x7f91400b17b0 msgr2=0x7f91401546b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 -- 172.21.15.40:0/3544657700 shutdown_connections 2024-03-24T00:37:47.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.117+0000 7f9148800640 1 -- 172.21.15.40:0/3544657700 wait complete. 2024-03-24T00:37:47.143 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb63800640 1 Processor -- start 2024-03-24T00:37:47.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb63800640 1 -- start start 2024-03-24T00:37:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb63800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbb5c141bc0 0x7fbb5c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb63800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbb5c0ce630 0x7fbb5c071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb63800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c070e00 con 0x7fbb5c13ec80 2024-03-24T00:37:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb63800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb5c0cee20 con 0x7fbb5c0ce630 2024-03-24T00:37:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb63800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb5c0cf100 con 0x7fbb5c141bc0 2024-03-24T00:37:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb61800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbb5c0ce630 0x7fbb5c071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb61800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbb5c0ce630 0x7fbb5c071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42132/0 (socket says 172.21.15.40:42132) 2024-03-24T00:37:47.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb5be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbb5c141bc0 0x7fbb5c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:47.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb61800640 1 -- 172.21.15.40:0/1806270923 learned_addr learned my addr 172.21.15.40:0/1806270923 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:47.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb61800640 1 -- 172.21.15.40:0/1806270923 >> v2:172.21.15.53:3300/0 conn(0x7fbb5c141bc0 msgr2=0x7fbb5c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:47.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb61800640 1 --2- 172.21.15.40:0/1806270923 >> v2:172.21.15.53:3300/0 conn(0x7fbb5c141bc0 0x7fbb5c0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb61800640 1 -- 172.21.15.40:0/1806270923 >> v1:172.21.15.53:6789/0 conn(0x7fbb5c13ec80 legacy=0x7fbb5c1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.281+0000 7fbb61800640 1 -- 172.21.15.40:0/1806270923 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c0cf3e0 con 0x7fbb5c0ce630 2024-03-24T00:37:47.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.289+0000 7fbb61800640 1 --2- 172.21.15.40:0/1806270923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5c0ce630 0x7fbb5c071300 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c04fa60 tx=0x7fbb5401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a80cf2b134046dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.289+0000 7fbb5b400640 1 -- 172.21.15.40:0/1806270923 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb540a58e0 con 0x7fbb5c0ce630 2024-03-24T00:37:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.289+0000 7fbb5b400640 1 -- 172.21.15.40:0/1806270923 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb540a5a80 con 0x7fbb5c0ce630 2024-03-24T00:37:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.289+0000 7fbb5b400640 1 -- 172.21.15.40:0/1806270923 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb540a5e50 con 0x7fbb5c0ce630 2024-03-24T00:37:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.289+0000 7fbb63800640 1 -- 172.21.15.40:0/1806270923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5c0ce630 msgr2=0x7fbb5c071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.289+0000 7fbb63800640 1 --2- 172.21.15.40:0/1806270923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5c0ce630 0x7fbb5c071300 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c04fa60 tx=0x7fbb5401c9f0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.289+0000 7fbb63800640 1 -- 172.21.15.40:0/1806270923 shutdown_connections 2024-03-24T00:37:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.289+0000 7fbb63800640 1 --2- 172.21.15.40:0/1806270923 >> v2:172.21.15.53:3300/0 conn(0x7fbb5c141bc0 0x7fbb5c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.289+0000 7fbb63800640 1 --2- 172.21.15.40:0/1806270923 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5c0ce630 0x7fbb5c071300 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.289+0000 7fbb63800640 1 -- 172.21.15.40:0/1806270923 >> 172.21.15.40:0/1806270923 conn(0x7fbb5c0b17b0 msgr2=0x7fbb5c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.289+0000 7fbb63800640 1 -- 172.21.15.40:0/1806270923 shutdown_connections 2024-03-24T00:37:47.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.289+0000 7fbb63800640 1 -- 172.21.15.40:0/1806270923 wait complete. 2024-03-24T00:37:47.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb63800640 1 Processor -- start 2024-03-24T00:37:47.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb63800640 1 -- start start 2024-03-24T00:37:47.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb63800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb5c0ce630 0x7fbb5c157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:47.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb63800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5c13ec80 0x7fbb5c145060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:47.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb63800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb5c141bc0 0x7fbb5c158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:47.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb63800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5c0723e0 con 0x7fbb5c13ec80 2024-03-24T00:37:47.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb63800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5c072260 con 0x7fbb5c0ce630 2024-03-24T00:37:47.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb63800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5c072560 con 0x7fbb5c141bc0 2024-03-24T00:37:47.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb5be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5c13ec80 0x7fbb5c145060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:47.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb60e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb5c0ce630 0x7fbb5c157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:47.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb5be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5c13ec80 0x7fbb5c145060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42146/0 (socket says 172.21.15.40:42146) 2024-03-24T00:37:47.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb5be00640 1 -- 172.21.15.40:0/405620650 learned_addr learned my addr 172.21.15.40:0/405620650 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:47.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb61800640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb5c141bc0 0x7fbb5c158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:47.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb5be00640 1 -- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb5c141bc0 msgr2=0x7fbb5c158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb5be00640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb5c141bc0 0x7fbb5c158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb5be00640 1 -- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb5c0ce630 msgr2=0x7fbb5c157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb5be00640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb5c0ce630 0x7fbb5c157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb5be00640 1 -- 172.21.15.40:0/405620650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c159440 con 0x7fbb5c13ec80 2024-03-24T00:37:47.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb5be00640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5c13ec80 0x7fbb5c145060 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800e930 tx=0x7fbb4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:47.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb58c00640 1 -- 172.21.15.40:0/405620650 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb48017070 con 0x7fbb5c13ec80 2024-03-24T00:37:47.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb58c00640 1 -- 172.21.15.40:0/405620650 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4800ca00 con 0x7fbb5c13ec80 2024-03-24T00:37:47.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb58c00640 1 -- 172.21.15.40:0/405620650 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4800cdb0 con 0x7fbb5c13ec80 2024-03-24T00:37:47.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb63800640 1 -- 172.21.15.40:0/405620650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb5c159720 con 0x7fbb5c13ec80 2024-03-24T00:37:47.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.293+0000 7fbb63800640 1 -- 172.21.15.40:0/405620650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5c1455a0 con 0x7fbb5c13ec80 2024-03-24T00:37:47.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.297+0000 7fbb63800640 1 -- 172.21.15.40:0/405620650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5c068170 con 0x7fbb5c13ec80 2024-03-24T00:37:47.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.297+0000 7fbb58c00640 1 -- 172.21.15.40:0/405620650 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbb4801a070 con 0x7fbb5c13ec80 2024-03-24T00:37:47.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.297+0000 7fbb58c00640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbb3c067660 0x7fbb3c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:47.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.297+0000 7fbb58c00640 1 -- 172.21.15.40:0/405620650 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb48085bd0 con 0x7fbb5c13ec80 2024-03-24T00:37:47.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.301+0000 7fbb60e00640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbb3c067660 0x7fbb3c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:47.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.301+0000 7fbb60e00640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbb3c067660 0x7fbb3c069b20 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fbb50007fd0 tx=0x7fbb50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:47.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.305+0000 7fbb58c00640 1 -- 172.21.15.40:0/405620650 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbb480562b0 con 0x7fbb5c13ec80 2024-03-24T00:37:47.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 -- 172.21.15.40:0/405620650 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb5c0ce250 con 0x7fbb5c13ec80 2024-03-24T00:37:47.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb58c00640 1 -- 172.21.15.40:0/405620650 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16548 (secure 0 0 0) 0x7fbb4805a350 con 0x7fbb5c13ec80 2024-03-24T00:37:47.477 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:47.477 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":95,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:37:03.189247+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:59.977310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-24T00:37:00.557420+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:47.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 -- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbb3c067660 msgr2=0x7fbb3c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbb3c067660 0x7fbb3c069b20 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fbb50007fd0 tx=0x7fbb50009040 comp rx=0 tx=0).stop 2024-03-24T00:37:47.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 -- 172.21.15.40:0/405620650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5c13ec80 msgr2=0x7fbb5c145060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5c13ec80 0x7fbb5c145060 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800e930 tx=0x7fbb4800edf0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 -- 172.21.15.40:0/405620650 shutdown_connections 2024-03-24T00:37:47.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbb3c067660 0x7fbb3c069b20 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb5c141bc0 0x7fbb5c158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbb5c13ec80 0x7fbb5c145060 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 --2- 172.21.15.40:0/405620650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb5c0ce630 0x7fbb5c157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 -- 172.21.15.40:0/405620650 >> 172.21.15.40:0/405620650 conn(0x7fbb5c0b17b0 msgr2=0x7fbb5c0b2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:47.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 -- 172.21.15.40:0/405620650 shutdown_connections 2024-03-24T00:37:47.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.473+0000 7fbb63800640 1 -- 172.21.15.40:0/405620650 wait complete. 2024-03-24T00:37:47.498 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:37:47.499 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:47.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd97e00640 1 Processor -- start 2024-03-24T00:37:47.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd97e00640 1 -- start start 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd97e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efd98144980 0x7efd98070fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd97e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efd98144d60 0x7efd980d1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd97e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9815f8e0 con 0x7efd98145220 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd97e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd981604f0 con 0x7efd98144d60 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd97e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd9815fbc0 con 0x7efd98144980 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd8fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efd98144d60 0x7efd980d1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd8fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efd98144d60 0x7efd980d1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42150/0 (socket says 172.21.15.40:42150) 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd8fe00640 1 -- 172.21.15.40:0/2396479133 learned_addr learned my addr 172.21.15.40:0/2396479133 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:47.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd96a00640 1 --2- 172.21.15.40:0/2396479133 >> v2:172.21.15.53:3300/0 conn(0x7efd98144980 0x7efd98070fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd8fe00640 1 -- 172.21.15.40:0/2396479133 >> v2:172.21.15.53:3300/0 conn(0x7efd98144980 msgr2=0x7efd98070fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd8fe00640 1 --2- 172.21.15.40:0/2396479133 >> v2:172.21.15.53:3300/0 conn(0x7efd98144980 0x7efd98070fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd8fe00640 1 -- 172.21.15.40:0/2396479133 >> v1:172.21.15.53:6789/0 conn(0x7efd98145220 legacy=0x7efd980ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.637+0000 7efd8fe00640 1 -- 172.21.15.40:0/2396479133 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9815fea0 con 0x7efd98144d60 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.641+0000 7efd8fe00640 1 --2- 172.21.15.40:0/2396479133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd98144d60 0x7efd980d1b20 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7efd980d1af0 tx=0x7efd8001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7bcce421a4a1f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.641+0000 7efd96000640 1 -- 172.21.15.40:0/2396479133 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd800ab680 con 0x7efd98144d60 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.641+0000 7efd96000640 1 -- 172.21.15.40:0/2396479133 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd800b0070 con 0x7efd98144d60 2024-03-24T00:37:47.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.641+0000 7efd96000640 1 -- 172.21.15.40:0/2396479133 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd800aba40 con 0x7efd98144d60 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.641+0000 7efd97e00640 1 -- 172.21.15.40:0/2396479133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd98144d60 msgr2=0x7efd980d1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.641+0000 7efd97e00640 1 --2- 172.21.15.40:0/2396479133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd98144d60 0x7efd980d1b20 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7efd980d1af0 tx=0x7efd8001c460 comp rx=0 tx=0).stop 2024-03-24T00:37:47.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.641+0000 7efd97e00640 1 -- 172.21.15.40:0/2396479133 shutdown_connections 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.641+0000 7efd97e00640 1 --2- 172.21.15.40:0/2396479133 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd98144d60 0x7efd980d1b20 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.641+0000 7efd97e00640 1 --2- 172.21.15.40:0/2396479133 >> v2:172.21.15.53:3300/0 conn(0x7efd98144980 0x7efd98070fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.641+0000 7efd97e00640 1 -- 172.21.15.40:0/2396479133 >> 172.21.15.40:0/2396479133 conn(0x7efd9806ed30 msgr2=0x7efd9806f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.641+0000 7efd97e00640 1 -- 172.21.15.40:0/2396479133 shutdown_connections 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.641+0000 7efd97e00640 1 -- 172.21.15.40:0/2396479133 wait complete. 2024-03-24T00:37:47.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd97e00640 1 Processor -- start 2024-03-24T00:37:47.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd97e00640 1 -- start start 2024-03-24T00:37:47.649 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd97e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd98144980 0x7efd98158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd97e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd98144d60 0x7efd9815ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd97e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd98145220 0x7efd9815b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd97e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd98168df0 con 0x7efd98144980 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd97e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd980d2470 con 0x7efd98144d60 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd97e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd98168f70 con 0x7efd98145220 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd96a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd98144980 0x7efd98158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd96a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd98144980 0x7efd98158580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42160/0 (socket says 172.21.15.40:42160) 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd96a00640 1 -- 172.21.15.40:0/2437182697 learned_addr learned my addr 172.21.15.40:0/2437182697 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd8fe00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd98144d60 0x7efd9815ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd97400640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd98145220 0x7efd9815b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:47.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd96a00640 1 -- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd98145220 msgr2=0x7efd9815b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd96a00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd98145220 0x7efd9815b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd96a00640 1 -- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd98144d60 msgr2=0x7efd9815ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd96a00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd98144d60 0x7efd9815ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd96a00640 1 -- 172.21.15.40:0/2437182697 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd980b5060 con 0x7efd98144980 2024-03-24T00:37:47.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd96a00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd98144980 0x7efd98158580 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7efd8800e930 tx=0x7efd8800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd8ea00640 1 -- 172.21.15.40:0/2437182697 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88017070 con 0x7efd98144980 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd8ea00640 1 -- 172.21.15.40:0/2437182697 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd8800ca00 con 0x7efd98144980 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd8ea00640 1 -- 172.21.15.40:0/2437182697 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8800cdb0 con 0x7efd98144980 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd97e00640 1 -- 172.21.15.40:0/2437182697 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd980b5340 con 0x7efd98144980 2024-03-24T00:37:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.645+0000 7efd97e00640 1 -- 172.21.15.40:0/2437182697 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd980b5900 con 0x7efd98144980 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.649+0000 7efd97e00640 1 -- 172.21.15.40:0/2437182697 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd98070260 con 0x7efd98144980 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.649+0000 7efd8ea00640 1 -- 172.21.15.40:0/2437182697 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7efd8801a070 con 0x7efd98144980 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.649+0000 7efd8ea00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efd74067660 0x7efd74069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:47.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.649+0000 7efd8ea00640 1 -- 172.21.15.40:0/2437182697 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd88085c00 con 0x7efd98144980 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.649+0000 7efd8fe00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efd74067660 0x7efd74069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:47.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.649+0000 7efd8fe00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efd74067660 0x7efd74069b20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7efd800af000 tx=0x7efd800a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:47.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.653+0000 7efd8ea00640 1 -- 172.21.15.40:0/2437182697 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7efd880562e0 con 0x7efd98144980 2024-03-24T00:37:47.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.849+0000 7efd97e00640 1 -- 172.21.15.40:0/2437182697 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd9804f4f0 con 0x7efd98144980 2024-03-24T00:37:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd8ea00640 1 -- 172.21.15.40:0/2437182697 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5021 (secure 0 0 0) 0x7efd8805a380 con 0x7efd98144980 2024-03-24T00:37:47.856 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:47.856 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":103,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6755,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2936601899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2936601899},{"type":"v1","addr":"172.21.15.53:6836","nonce":2936601899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:37:01.566949+0000","modified":"2024-03-24T00:37:27.349073+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7932,"mds_1":7917,"mds_2":7633},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7633":{"gid":7633,"name":"a","rank":2,"incarnation":101,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6835/1997904254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1997904254},{"type":"v1","addr":"172.21.15.40:6835","nonce":1997904254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7917":{"gid":7917,"name":"c","rank":1,"incarnation":99,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6834/1019660586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1019660586},{"type":"v1","addr":"172.21.15.40:6834","nonce":1019660586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7932":{"gid":7932,"name":"d","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/3095501397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3095501397},{"type":"v1","addr":"172.21.15.53:6837","nonce":3095501397}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7917,7932,7633]},"id":7}]} 2024-03-24T00:37:47.856 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 103 2024-03-24T00:37:47.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd97e00640 1 -- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efd74067660 msgr2=0x7efd74069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd97e00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efd74067660 0x7efd74069b20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7efd800af000 tx=0x7efd800a7000 comp rx=0 tx=0).stop 2024-03-24T00:37:47.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd97e00640 1 -- 172.21.15.40:0/2437182697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd98144980 msgr2=0x7efd98158580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:47.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd97e00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd98144980 0x7efd98158580 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7efd8800e930 tx=0x7efd8800edf0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd97e00640 1 -- 172.21.15.40:0/2437182697 shutdown_connections 2024-03-24T00:37:47.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd97e00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efd74067660 0x7efd74069b20 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd97e00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd98145220 0x7efd9815b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd97e00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd98144d60 0x7efd9815ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd97e00640 1 --2- 172.21.15.40:0/2437182697 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efd98144980 0x7efd98158580 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:47.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd97e00640 1 -- 172.21.15.40:0/2437182697 >> 172.21.15.40:0/2437182697 conn(0x7efd9806ed30 msgr2=0x7efd9806f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:47.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd97e00640 1 -- 172.21.15.40:0/2437182697 shutdown_connections 2024-03-24T00:37:47.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:47.853+0000 7efd97e00640 1 -- 172.21.15.40:0/2437182697 wait complete. 2024-03-24T00:37:47.877 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:48.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.013+0000 7faf75000640 1 Processor -- start 2024-03-24T00:37:48.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.013+0000 7faf75000640 1 -- start start 2024-03-24T00:37:48.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.013+0000 7faf75000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faf70141bc0 0x7faf700ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.013+0000 7faf75000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf700ce630 0x7faf70071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.013+0000 7faf75000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf70070e00 con 0x7faf7013ec80 2024-03-24T00:37:48.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.013+0000 7faf75000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf700cee20 con 0x7faf70141bc0 2024-03-24T00:37:48.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.013+0000 7faf75000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf700cf100 con 0x7faf700ce630 2024-03-24T00:37:48.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.013+0000 7faf6e800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf700ce630 0x7faf70071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.013+0000 7faf6e800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf700ce630 0x7faf70071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:60306/0 (socket says 172.21.15.40:60306) 2024-03-24T00:37:48.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.013+0000 7faf6e800640 1 -- 172.21.15.40:0/1779876302 learned_addr learned my addr 172.21.15.40:0/1779876302 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:48.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.013+0000 7faf65400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faf70141bc0 0x7faf700ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.017+0000 7faf6e800640 1 -- 172.21.15.40:0/1779876302 >> v2:172.21.15.53:3300/0 conn(0x7faf70141bc0 msgr2=0x7faf700ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:48.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.017+0000 7faf6e800640 1 --2- 172.21.15.40:0/1779876302 >> v2:172.21.15.53:3300/0 conn(0x7faf70141bc0 0x7faf700ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.017+0000 7faf6e800640 1 -- 172.21.15.40:0/1779876302 >> v1:172.21.15.53:6789/0 conn(0x7faf7013ec80 legacy=0x7faf701414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.017+0000 7faf6e800640 1 -- 172.21.15.40:0/1779876302 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf700cf3e0 con 0x7faf700ce630 2024-03-24T00:37:48.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.017+0000 7faf65400640 1 --2- 172.21.15.40:0/1779876302 >> v2:172.21.15.53:3300/0 conn(0x7faf70141bc0 0x7faf700ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:48.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf6e800640 1 --2- 172.21.15.40:0/1779876302 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf700ce630 0x7faf70071300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7faf68009870 tx=0x7faf6801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cf7ccf98d8748c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:48.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf6d400640 1 -- 172.21.15.40:0/1779876302 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf680a9640 con 0x7faf700ce630 2024-03-24T00:37:48.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf6d400640 1 -- 172.21.15.40:0/1779876302 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf680a97e0 con 0x7faf700ce630 2024-03-24T00:37:48.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf6d400640 1 -- 172.21.15.40:0/1779876302 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf680a9bb0 con 0x7faf700ce630 2024-03-24T00:37:48.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf75000640 1 -- 172.21.15.40:0/1779876302 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf700ce630 msgr2=0x7faf70071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf75000640 1 --2- 172.21.15.40:0/1779876302 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf700ce630 0x7faf70071300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7faf68009870 tx=0x7faf6801c560 comp rx=0 tx=0).stop 2024-03-24T00:37:48.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf75000640 1 -- 172.21.15.40:0/1779876302 shutdown_connections 2024-03-24T00:37:48.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf75000640 1 --2- 172.21.15.40:0/1779876302 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf700ce630 0x7faf70071300 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf75000640 1 --2- 172.21.15.40:0/1779876302 >> v2:172.21.15.53:3300/0 conn(0x7faf70141bc0 0x7faf700ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf75000640 1 -- 172.21.15.40:0/1779876302 >> 172.21.15.40:0/1779876302 conn(0x7faf700b17b0 msgr2=0x7faf700b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:48.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf75000640 1 -- 172.21.15.40:0/1779876302 shutdown_connections 2024-03-24T00:37:48.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf75000640 1 -- 172.21.15.40:0/1779876302 wait complete. 2024-03-24T00:37:48.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf75000640 1 Processor -- start 2024-03-24T00:37:48.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.021+0000 7faf75000640 1 -- start start 2024-03-24T00:37:48.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf75000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf700ce630 0x7faf7013bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf75000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf7013ec80 0x7faf70156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf75000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf70141bc0 0x7faf700b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf75000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf700723e0 con 0x7faf700ce630 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf75000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf70072260 con 0x7faf70141bc0 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf75000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf70072560 con 0x7faf7013ec80 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf6de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf700ce630 0x7faf7013bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf6de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf700ce630 0x7faf7013bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42170/0 (socket says 172.21.15.40:42170) 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf65400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf7013ec80 0x7faf70156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf6de00640 1 -- 172.21.15.40:0/1336931281 learned_addr learned my addr 172.21.15.40:0/1336931281 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf6e800640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf70141bc0 0x7faf700b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf65400640 1 -- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf70141bc0 msgr2=0x7faf700b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf65400640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf70141bc0 0x7faf700b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf65400640 1 -- 172.21.15.40:0/1336931281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf700ce630 msgr2=0x7faf7013bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf65400640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf700ce630 0x7faf7013bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf65400640 1 -- 172.21.15.40:0/1336931281 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7013af80 con 0x7faf7013ec80 2024-03-24T00:37:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf6de00640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf700ce630 0x7faf7013bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf65400640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf7013ec80 0x7faf70156260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7faf5800efc0 tx=0x7faf5800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:48.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf66a00640 1 -- 172.21.15.40:0/1336931281 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58018020 con 0x7faf7013ec80 2024-03-24T00:37:48.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf66a00640 1 -- 172.21.15.40:0/1336931281 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf58014680 con 0x7faf7013ec80 2024-03-24T00:37:48.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf66a00640 1 -- 172.21.15.40:0/1336931281 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf58014a70 con 0x7faf7013ec80 2024-03-24T00:37:48.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf75000640 1 -- 172.21.15.40:0/1336931281 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7013b260 con 0x7faf7013ec80 2024-03-24T00:37:48.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf75000640 1 -- 172.21.15.40:0/1336931281 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7013b540 con 0x7faf7013ec80 2024-03-24T00:37:48.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.025+0000 7faf75000640 1 -- 172.21.15.40:0/1336931281 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf70068170 con 0x7faf7013ec80 2024-03-24T00:37:48.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.029+0000 7faf66a00640 1 -- 172.21.15.40:0/1336931281 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7faf5801b020 con 0x7faf7013ec80 2024-03-24T00:37:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.029+0000 7faf66a00640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faf4c067660 0x7faf4c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.029+0000 7faf66a00640 1 -- 172.21.15.40:0/1336931281 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf58086470 con 0x7faf7013ec80 2024-03-24T00:37:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.029+0000 7faf6de00640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faf4c067660 0x7faf4c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.029+0000 7faf6de00640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faf4c067660 0x7faf4c069b20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7faf7015e6d0 tx=0x7faf60009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:48.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.033+0000 7faf66a00640 1 -- 172.21.15.40:0/1336931281 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7faf58022030 con 0x7faf7013ec80 2024-03-24T00:37:48.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.233+0000 7faf75000640 1 -- 172.21.15.40:0/1336931281 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf70072b90 con 0x7faf7013ec80 2024-03-24T00:37:48.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.233+0000 7faf66a00640 1 -- 172.21.15.40:0/1336931281 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5021 (secure 0 0 0) 0x7faf58010040 con 0x7faf7013ec80 2024-03-24T00:37:48.237 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:48.238 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":103,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6755,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2936601899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2936601899},{"type":"v1","addr":"172.21.15.53:6836","nonce":2936601899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:37:01.566949+0000","modified":"2024-03-24T00:37:27.349073+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":7932,"mds_1":7917,"mds_2":7633},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7633":{"gid":7633,"name":"a","rank":2,"incarnation":101,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6835/1997904254","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":1997904254},{"type":"v1","addr":"172.21.15.40:6835","nonce":1997904254}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7917":{"gid":7917,"name":"c","rank":1,"incarnation":99,"state":"up:active","state_seq":8,"addr":"172.21.15.40:6834/1019660586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":1019660586},{"type":"v1","addr":"172.21.15.40:6834","nonce":1019660586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_7932":{"gid":7932,"name":"d","rank":0,"incarnation":93,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/3095501397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":3095501397},{"type":"v1","addr":"172.21.15.53:6837","nonce":3095501397}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7932,"qdb_cluster":[7917,7932,7633]},"id":7}]} 2024-03-24T00:37:48.238 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 103 2024-03-24T00:37:48.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.237+0000 7faf75000640 1 -- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faf4c067660 msgr2=0x7faf4c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.237+0000 7faf75000640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faf4c067660 0x7faf4c069b20 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7faf7015e6d0 tx=0x7faf60009040 comp rx=0 tx=0).stop 2024-03-24T00:37:48.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.237+0000 7faf75000640 1 -- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf7013ec80 msgr2=0x7faf70156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.237+0000 7faf75000640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf7013ec80 0x7faf70156260 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7faf5800efc0 tx=0x7faf5800c490 comp rx=0 tx=0).stop 2024-03-24T00:37:48.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.237+0000 7faf75000640 1 -- 172.21.15.40:0/1336931281 shutdown_connections 2024-03-24T00:37:48.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.237+0000 7faf75000640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faf4c067660 0x7faf4c069b20 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.237+0000 7faf75000640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf70141bc0 0x7faf700b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.237+0000 7faf75000640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf7013ec80 0x7faf70156260 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.237+0000 7faf75000640 1 --2- 172.21.15.40:0/1336931281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf700ce630 0x7faf7013bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.237+0000 7faf75000640 1 -- 172.21.15.40:0/1336931281 >> 172.21.15.40:0/1336931281 conn(0x7faf700b17b0 msgr2=0x7faf7013c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:48.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.237+0000 7faf75000640 1 -- 172.21.15.40:0/1336931281 shutdown_connections 2024-03-24T00:37:48.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.237+0000 7faf75000640 1 -- 172.21.15.40:0/1336931281 wait complete. 2024-03-24T00:37:48.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53c6a00640 1 Processor -- start 2024-03-24T00:37:48.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53c6a00640 1 -- start start 2024-03-24T00:37:48.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53c6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53c0072310 0x7f53c00ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53c6a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c0070e00 con 0x7f53c0141bc0 2024-03-24T00:37:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53c6a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c00cee20 con 0x7f53c013ec80 2024-03-24T00:37:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53c6a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c00cf100 con 0x7f53c0072310 2024-03-24T00:37:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53c4a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f53c0072310 0x7f53c00ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53b7e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f53c0141bc0 0x7f53c00ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42822/0 (socket says 172.21.15.40:42822) 2024-03-24T00:37:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53b7e00640 1 -- 172.21.15.40:0/2154787667 learned_addr learned my addr 172.21.15.40:0/2154787667 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53bfe00640 1 --1- 172.21.15.40:0/2154787667 >> v1:172.21.15.53:6789/0 conn(0x7f53c013ec80 0x7f53c01414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:60494/0 (socket says 172.21.15.40:60494) 2024-03-24T00:37:48.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53c4a00640 1 -- 172.21.15.40:0/2154787667 >> v1:172.21.15.53:6789/0 conn(0x7f53c013ec80 legacy=0x7f53c01414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53c4a00640 1 -- 172.21.15.40:0/2154787667 >> v1:172.21.15.40:6789/0 conn(0x7f53c0141bc0 legacy=0x7f53c00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.397+0000 7f53c4a00640 1 -- 172.21.15.40:0/2154787667 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c00cf3e0 con 0x7f53c0072310 2024-03-24T00:37:48.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.401+0000 7f53c4a00640 1 --2- 172.21.15.40:0/2154787667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53c0072310 0x7f53c00ce800 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009d90 tx=0x7f53b801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805a8e4870fe4803 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:48.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.401+0000 7f53bf400640 1 -- 172.21.15.40:0/2154787667 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b80b2070 con 0x7f53c0072310 2024-03-24T00:37:48.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.401+0000 7f53bf400640 1 -- 172.21.15.40:0/2154787667 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b80a5a50 con 0x7f53c0072310 2024-03-24T00:37:48.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.401+0000 7f53bf400640 1 -- 172.21.15.40:0/2154787667 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b80a5e20 con 0x7f53c0072310 2024-03-24T00:37:48.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.401+0000 7f53c6a00640 1 -- 172.21.15.40:0/2154787667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53c0072310 msgr2=0x7f53c00ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.401+0000 7f53c6a00640 1 --2- 172.21.15.40:0/2154787667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53c0072310 0x7f53c00ce800 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009d90 tx=0x7f53b801c970 comp rx=0 tx=0).stop 2024-03-24T00:37:48.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.401+0000 7f53c6a00640 1 -- 172.21.15.40:0/2154787667 shutdown_connections 2024-03-24T00:37:48.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.401+0000 7f53c6a00640 1 --2- 172.21.15.40:0/2154787667 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53c0072310 0x7f53c00ce800 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.401+0000 7f53c6a00640 1 -- 172.21.15.40:0/2154787667 >> 172.21.15.40:0/2154787667 conn(0x7f53c00b17b0 msgr2=0x7f53c00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:48.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.401+0000 7f53c6a00640 1 -- 172.21.15.40:0/2154787667 shutdown_connections 2024-03-24T00:37:48.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.401+0000 7f53c6a00640 1 -- 172.21.15.40:0/2154787667 wait complete. 2024-03-24T00:37:48.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c6a00640 1 Processor -- start 2024-03-24T00:37:48.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c6a00640 1 -- start start 2024-03-24T00:37:48.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c6a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c0072310 0x7f53c015b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c013ec80 0x7f53c0156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53c0141bc0 0x7f53c00b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c6a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c0169f00 con 0x7f53c0141bc0 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c6a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c0169d80 con 0x7f53c013ec80 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c6a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c016a080 con 0x7f53c0072310 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53c0141bc0 0x7f53c00b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53bfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c0072310 0x7f53c015b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53bfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c0072310 0x7f53c015b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:60330/0 (socket says 172.21.15.40:60330) 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c4a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53c0141bc0 0x7f53c00b3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42182/0 (socket says 172.21.15.40:42182) 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53bfe00640 1 -- 172.21.15.40:0/110249960 learned_addr learned my addr 172.21.15.40:0/110249960 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53b7e00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c013ec80 0x7f53c0156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c4a00640 1 -- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c0072310 msgr2=0x7f53c015b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c4a00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c0072310 0x7f53c015b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c4a00640 1 -- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c013ec80 msgr2=0x7f53c0156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c4a00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c013ec80 0x7f53c0156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c4a00640 1 -- 172.21.15.40:0/110249960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c00b2ef0 con 0x7f53c0141bc0 2024-03-24T00:37:48.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53b7e00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c013ec80 0x7f53c0156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:48.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c4a00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53c0141bc0 0x7f53c00b3f90 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f53b80ac000 tx=0x7f53b8019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:48.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53bcc00640 1 -- 172.21.15.40:0/110249960 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b80b2050 con 0x7f53c0141bc0 2024-03-24T00:37:48.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53bcc00640 1 -- 172.21.15.40:0/110249960 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b80b3040 con 0x7f53c0141bc0 2024-03-24T00:37:48.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c6a00640 1 -- 172.21.15.40:0/110249960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c00b31d0 con 0x7f53c0141bc0 2024-03-24T00:37:48.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53bcc00640 1 -- 172.21.15.40:0/110249960 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b80b9450 con 0x7f53c0141bc0 2024-03-24T00:37:48.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c6a00640 1 -- 172.21.15.40:0/110249960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c00b3a70 con 0x7f53c0141bc0 2024-03-24T00:37:48.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.405+0000 7f53c6a00640 1 -- 172.21.15.40:0/110249960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c00b33a0 con 0x7f53c0141bc0 2024-03-24T00:37:48.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.409+0000 7f53bcc00640 1 -- 172.21.15.40:0/110249960 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f53b80c0020 con 0x7f53c0141bc0 2024-03-24T00:37:48.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.409+0000 7f53bcc00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f539c067660 0x7f539c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.409+0000 7f53bcc00640 1 -- 172.21.15.40:0/110249960 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53b8120ec0 con 0x7f53c0141bc0 2024-03-24T00:37:48.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.409+0000 7f53bfe00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f539c067660 0x7f539c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.413+0000 7f53bfe00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f539c067660 0x7f539c069b20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f53b0000f70 tx=0x7f53b0007f10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:48.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.413+0000 7f53bcc00640 1 -- 172.21.15.40:0/110249960 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f53b8016560 con 0x7f53c0141bc0 2024-03-24T00:37:48.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.573+0000 7f53c6a00640 1 -- 172.21.15.40:0/110249960 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53c0068170 con 0x7f53c0141bc0 2024-03-24T00:37:48.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53bcc00640 1 -- 172.21.15.40:0/110249960 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16548 (secure 0 0 0) 0x7f53b80f1e00 con 0x7f53c0141bc0 2024-03-24T00:37:48.580 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:48.580 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":95,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:37:03.189247+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:36:59.977310+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-03-24T00:37:00.557420+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:48.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53c6a00640 1 -- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f539c067660 msgr2=0x7f539c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53c6a00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f539c067660 0x7f539c069b20 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f53b0000f70 tx=0x7f53b0007f10 comp rx=0 tx=0).stop 2024-03-24T00:37:48.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53c6a00640 1 -- 172.21.15.40:0/110249960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53c0141bc0 msgr2=0x7f53c00b3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53c6a00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53c0141bc0 0x7f53c00b3f90 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f53b80ac000 tx=0x7f53b8019710 comp rx=0 tx=0).stop 2024-03-24T00:37:48.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53c6a00640 1 -- 172.21.15.40:0/110249960 shutdown_connections 2024-03-24T00:37:48.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53c6a00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f539c067660 0x7f539c069b20 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53c6a00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f53c0141bc0 0x7f53c00b3f90 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53c6a00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c013ec80 0x7f53c0156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53c6a00640 1 --2- 172.21.15.40:0/110249960 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c0072310 0x7f53c015b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53c6a00640 1 -- 172.21.15.40:0/110249960 >> 172.21.15.40:0/110249960 conn(0x7f53c00b17b0 msgr2=0x7f53c015c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:48.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53c6a00640 1 -- 172.21.15.40:0/110249960 shutdown_connections 2024-03-24T00:37:48.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.577+0000 7f53c6a00640 1 -- 172.21.15.40:0/110249960 wait complete. 2024-03-24T00:37:48.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:37:48.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c7e00640 1 Processor -- start 2024-03-24T00:37:48.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c7e00640 1 -- start start 2024-03-24T00:37:48.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa1c815bca0 0x7fa1c80d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c815c080 0x7fa1c80d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa1c80ce490 0x7fa1c80ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c7e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1c80d0780 con 0x7fa1c815bca0 2024-03-24T00:37:48.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c7e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1c80b4a90 con 0x7fa1c80ce490 2024-03-24T00:37:48.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c7e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c80b4d70 con 0x7fa1c815c080 2024-03-24T00:37:48.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c6a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa1c815bca0 0x7fa1c80d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c815c080 0x7fa1c80d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa1c80ce490 0x7fa1c80ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c815c080 0x7fa1c80d0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:60336/0 (socket says 172.21.15.40:60336) 2024-03-24T00:37:48.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c6000640 1 -- 172.21.15.40:0/3458783294 learned_addr learned my addr 172.21.15.40:0/3458783294 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:48.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c6a00640 1 -- 172.21.15.40:0/3458783294 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c815c080 msgr2=0x7fa1c80d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:48.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c6a00640 1 --2- 172.21.15.40:0/3458783294 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c815c080 0x7fa1c80d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c6a00640 1 -- 172.21.15.40:0/3458783294 >> v2:172.21.15.53:3300/0 conn(0x7fa1c80ce490 msgr2=0x7fa1c80ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:48.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c6a00640 1 --2- 172.21.15.40:0/3458783294 >> v2:172.21.15.53:3300/0 conn(0x7fa1c80ce490 0x7fa1c80ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c6a00640 1 -- 172.21.15.40:0/3458783294 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c80b5050 con 0x7fa1c815bca0 2024-03-24T00:37:48.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.741+0000 7fa1c7400640 1 --2- 172.21.15.40:0/3458783294 >> v2:172.21.15.53:3300/0 conn(0x7fa1c80ce490 0x7fa1c80ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:48.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c6a00640 1 --2- 172.21.15.40:0/3458783294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c815bca0 0x7fa1c80d2590 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8067e30 tx=0x7fa1b801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5596a6a254d2b0dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:48.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c5600640 1 -- 172.21.15.40:0/3458783294 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b80b0070 con 0x7fa1c815bca0 2024-03-24T00:37:48.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c5600640 1 -- 172.21.15.40:0/3458783294 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b80ab600 con 0x7fa1c815bca0 2024-03-24T00:37:48.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c5600640 1 -- 172.21.15.40:0/3458783294 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b80ab9d0 con 0x7fa1c815bca0 2024-03-24T00:37:48.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c7e00640 1 -- 172.21.15.40:0/3458783294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c815bca0 msgr2=0x7fa1c80d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c7e00640 1 --2- 172.21.15.40:0/3458783294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c815bca0 0x7fa1c80d2590 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8067e30 tx=0x7fa1b801c480 comp rx=0 tx=0).stop 2024-03-24T00:37:48.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c7e00640 1 -- 172.21.15.40:0/3458783294 shutdown_connections 2024-03-24T00:37:48.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c7e00640 1 --2- 172.21.15.40:0/3458783294 >> v2:172.21.15.53:3300/0 conn(0x7fa1c80ce490 0x7fa1c80ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c7e00640 1 --2- 172.21.15.40:0/3458783294 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c815c080 0x7fa1c80d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c7e00640 1 --2- 172.21.15.40:0/3458783294 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c815bca0 0x7fa1c80d2590 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c7e00640 1 -- 172.21.15.40:0/3458783294 >> 172.21.15.40:0/3458783294 conn(0x7fa1c80d1aa0 msgr2=0x7fa1c80d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:48.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c7e00640 1 -- 172.21.15.40:0/3458783294 shutdown_connections 2024-03-24T00:37:48.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.745+0000 7fa1c7e00640 1 -- 172.21.15.40:0/3458783294 wait complete. 2024-03-24T00:37:48.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c7e00640 1 Processor -- start 2024-03-24T00:37:48.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c7e00640 1 -- start start 2024-03-24T00:37:48.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80ce490 0x7fa1c814b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c815bca0 0x7fa1c8163bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c815c080 0x7fa1c806c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c7e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c80cd150 con 0x7fa1c815bca0 2024-03-24T00:37:48.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c80ccfd0 con 0x7fa1c80ce490 2024-03-24T00:37:48.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c7e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c80cd2d0 con 0x7fa1c815c080 2024-03-24T00:37:48.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c815bca0 0x7fa1c8163bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c6000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c815bca0 0x7fa1c8163bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42190/0 (socket says 172.21.15.40:42190) 2024-03-24T00:37:48.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c6000640 1 -- 172.21.15.40:0/2096869793 learned_addr learned my addr 172.21.15.40:0/2096869793 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:48.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c6a00640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80ce490 0x7fa1c814b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c7400640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c815c080 0x7fa1c806c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c6000640 1 -- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c815c080 msgr2=0x7fa1c806c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c6000640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c815c080 0x7fa1c806c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c6000640 1 -- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80ce490 msgr2=0x7fa1c814b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c6000640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80ce490 0x7fa1c814b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c6000640 1 -- 172.21.15.40:0/2096869793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c80712a0 con 0x7fa1c815bca0 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c6a00640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80ce490 0x7fa1c814b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c6000640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c815bca0 0x7fa1c8163bc0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400e900 tx=0x7fa1b400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1bea00640 1 -- 172.21.15.40:0/2096869793 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b4017070 con 0x7fa1c815bca0 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1bea00640 1 -- 172.21.15.40:0/2096869793 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b400c940 con 0x7fa1c815bca0 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1bea00640 1 -- 172.21.15.40:0/2096869793 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b400ccf0 con 0x7fa1c815bca0 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c7e00640 1 -- 172.21.15.40:0/2096869793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c8165910 con 0x7fa1c815bca0 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c7e00640 1 -- 172.21.15.40:0/2096869793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c8165ed0 con 0x7fa1c815bca0 2024-03-24T00:37:48.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.749+0000 7fa1c7e00640 1 -- 172.21.15.40:0/2096869793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c809fb10 con 0x7fa1c815bca0 2024-03-24T00:37:48.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.753+0000 7fa1bea00640 1 -- 172.21.15.40:0/2096869793 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa1b401a070 con 0x7fa1c815bca0 2024-03-24T00:37:48.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.753+0000 7fa1bea00640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1a8067660 0x7fa1a8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:48.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.753+0000 7fa1bea00640 1 -- 172.21.15.40:0/2096869793 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1b4085eb0 con 0x7fa1c815bca0 2024-03-24T00:37:48.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.753+0000 7fa1c6a00640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1a8067660 0x7fa1a8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:48.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.753+0000 7fa1c6a00640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1a8067660 0x7fa1a8069b20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa1b80ad000 tx=0x7fa1b80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:48.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.757+0000 7fa1bea00640 1 -- 172.21.15.40:0/2096869793 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa1b4056530 con 0x7fa1c815bca0 2024-03-24T00:37:48.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.957+0000 7fa1c7e00640 1 -- 172.21.15.40:0/2096869793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa1c80d1880 con 0x7fa1c815bca0 2024-03-24T00:37:48.960 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-03-24T00:37:48.957+0000 7fa64e600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:37:48.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.957+0000 7fa1bea00640 1 -- 172.21.15.40:0/2096869793 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v104) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa1b405a5d0 con 0x7fa1c815bca0 2024-03-24T00:37:48.962 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:37:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.961+0000 7fa1c7e00640 1 -- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1a8067660 msgr2=0x7fa1a8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.961+0000 7fa1c7e00640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1a8067660 0x7fa1a8069b20 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa1b80ad000 tx=0x7fa1b80a6000 comp rx=0 tx=0).stop 2024-03-24T00:37:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.961+0000 7fa1c7e00640 1 -- 172.21.15.40:0/2096869793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c815bca0 msgr2=0x7fa1c8163bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.961+0000 7fa1c7e00640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c815bca0 0x7fa1c8163bc0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400e900 tx=0x7fa1b400ed00 comp rx=0 tx=0).stop 2024-03-24T00:37:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.965+0000 7fa1c7e00640 1 -- 172.21.15.40:0/2096869793 shutdown_connections 2024-03-24T00:37:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.965+0000 7fa1c7e00640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1a8067660 0x7fa1a8069b20 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.965+0000 7fa1c7e00640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c815c080 0x7fa1c806c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.965+0000 7fa1c7e00640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c815bca0 0x7fa1c8163bc0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.965+0000 7fa1c7e00640 1 --2- 172.21.15.40:0/2096869793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c80ce490 0x7fa1c814b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.965+0000 7fa1c7e00640 1 -- 172.21.15.40:0/2096869793 >> 172.21.15.40:0/2096869793 conn(0x7fa1c80d1aa0 msgr2=0x7fa1c80b5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.965+0000 7fa1c7e00640 1 -- 172.21.15.40:0/2096869793 shutdown_connections 2024-03-24T00:37:48.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:48.965+0000 7fa1c7e00640 1 -- 172.21.15.40:0/2096869793 wait complete. 2024-03-24T00:37:48.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:37:49.026 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:37:49.032 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:37:49.032 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.133+0000 7f78ad000640 1 Processor -- start 2024-03-24T00:37:49.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.133+0000 7f78ad000640 1 -- start start 2024-03-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78ad000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a815bd00 0x7f78a80ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78ad000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f78a80728d0 0x7f78a80cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78ad000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78a8072d30 con 0x7f78a815f8c0 2024-03-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78ad000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78a80cf790 con 0x7f78a80728d0 2024-03-24T00:37:49.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78ad000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a8071de0 con 0x7f78a815bd00 2024-03-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78a6800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f78a815f8c0 0x7f78a806cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42828/0 (socket says 172.21.15.40:42828) 2024-03-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78a6800640 1 -- 172.21.15.40:0/2822631350 learned_addr learned my addr 172.21.15.40:0/2822631350 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f789d400640 1 --2- 172.21.15.40:0/2822631350 >> v2:172.21.15.53:3300/0 conn(0x7f78a80728d0 0x7f78a80cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78a5e00640 1 --2- 172.21.15.40:0/2822631350 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a815bd00 0x7f78a80ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:49.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78a5400640 1 -- 172.21.15.40:0/2822631350 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711251109 0 0) 0x7f78a8072d30 con 0x7f78a815f8c0 2024-03-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78a5400640 1 -- 172.21.15.40:0/2822631350 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f788c003530 con 0x7f78a815f8c0 2024-03-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78a5e00640 1 -- 172.21.15.40:0/2822631350 >> v2:172.21.15.53:3300/0 conn(0x7f78a80728d0 msgr2=0x7f78a80cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78a5e00640 1 --2- 172.21.15.40:0/2822631350 >> v2:172.21.15.53:3300/0 conn(0x7f78a80728d0 0x7f78a80cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78a5e00640 1 -- 172.21.15.40:0/2822631350 >> v1:172.21.15.40:6789/0 conn(0x7f78a815f8c0 legacy=0x7f78a806cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:49.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.137+0000 7f78a5e00640 1 -- 172.21.15.40:0/2822631350 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a80cea50 con 0x7f78a815bd00 2024-03-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78a5e00640 1 --2- 172.21.15.40:0/2822631350 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a815bd00 0x7f78a80ae8e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f78980097c0 tx=0x7f789801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b64226b289059eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78a5400640 1 -- 172.21.15.40:0/2822631350 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78980a7070 con 0x7f78a815bd00 2024-03-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78a5400640 1 -- 172.21.15.40:0/2822631350 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78980b0070 con 0x7f78a815bd00 2024-03-24T00:37:49.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78a5400640 1 -- 172.21.15.40:0/2822631350 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78980ab890 con 0x7f78a815bd00 2024-03-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78ad000640 1 -- 172.21.15.40:0/2822631350 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a815bd00 msgr2=0x7f78a80ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78ad000640 1 --2- 172.21.15.40:0/2822631350 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a815bd00 0x7f78a80ae8e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f78980097c0 tx=0x7f789801c470 comp rx=0 tx=0).stop 2024-03-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78ad000640 1 -- 172.21.15.40:0/2822631350 shutdown_connections 2024-03-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78ad000640 1 --2- 172.21.15.40:0/2822631350 >> v2:172.21.15.53:3300/0 conn(0x7f78a80728d0 0x7f78a80cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:49.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78ad000640 1 --2- 172.21.15.40:0/2822631350 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a815bd00 0x7f78a80ae8e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78ad000640 1 -- 172.21.15.40:0/2822631350 >> 172.21.15.40:0/2822631350 conn(0x7f78a80b1660 msgr2=0x7f78a80b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78ad000640 1 -- 172.21.15.40:0/2822631350 shutdown_connections 2024-03-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78ad000640 1 -- 172.21.15.40:0/2822631350 wait complete. 2024-03-24T00:37:49.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78ad000640 1 Processor -- start 2024-03-24T00:37:49.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.141+0000 7f78ad000640 1 -- start start 2024-03-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78ad000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78a80728d0 0x7f78a80b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78ad000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78a815bd00 0x7f78a80a9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78ad000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a815f8c0 0x7f78a80aa180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78ad000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a815c0e0 con 0x7f78a80728d0 2024-03-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78ad000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78a806d630 con 0x7f78a815bd00 2024-03-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78ad000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78a81600e0 con 0x7f78a815f8c0 2024-03-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78a5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78a80728d0 0x7f78a80b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:49.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78a5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78a80728d0 0x7f78a80b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42224/0 (socket says 172.21.15.40:42224) 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78a5e00640 1 -- 172.21.15.40:0/307822363 learned_addr learned my addr 172.21.15.40:0/307822363 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f789d400640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78a815bd00 0x7f78a80a9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78a6800640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a815f8c0 0x7f78a80aa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78a5e00640 1 -- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a815f8c0 msgr2=0x7f78a80aa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78a5e00640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a815f8c0 0x7f78a80aa180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78a5e00640 1 -- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78a815bd00 msgr2=0x7f78a80a9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78a5e00640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78a815bd00 0x7f78a80a9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78a5e00640 1 -- 172.21.15.40:0/307822363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78a80ccf40 con 0x7f78a80728d0 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78a5e00640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78a80728d0 0x7f78a80b3470 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f789801a0a0 tx=0x7f789801acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f789ea00640 1 -- 172.21.15.40:0/307822363 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78980bf610 con 0x7f78a80728d0 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f789ea00640 1 -- 172.21.15.40:0/307822363 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78980abe50 con 0x7f78a80728d0 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f789ea00640 1 -- 172.21.15.40:0/307822363 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78980bfa10 con 0x7f78a80728d0 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78ad000640 1 -- 172.21.15.40:0/307822363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78a80b3ac0 con 0x7f78a80728d0 2024-03-24T00:37:49.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78ad000640 1 -- 172.21.15.40:0/307822363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78a80b4080 con 0x7f78a80728d0 2024-03-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.145+0000 7f78ad000640 1 -- 172.21.15.40:0/307822363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78a8068170 con 0x7f78a80728d0 2024-03-24T00:37:49.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.149+0000 7f789ea00640 1 -- 172.21.15.40:0/307822363 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f78980bd020 con 0x7f78a80728d0 2024-03-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.149+0000 7f789ea00640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7884067660 0x7f7884069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.149+0000 7f789ea00640 1 -- 172.21.15.40:0/307822363 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f789811fbf0 con 0x7f78a80728d0 2024-03-24T00:37:49.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.149+0000 7f789d400640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7884067660 0x7f7884069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:49.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.149+0000 7f789d400640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7884067660 0x7f7884069b20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7890009b60 tx=0x7f7890009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:49.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.153+0000 7f789ea00640 1 -- 172.21.15.40:0/307822363 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f78980c6030 con 0x7f78a80728d0 2024-03-24T00:37:49.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.353+0000 7f78ad000640 1 -- 172.21.15.40:0/307822363 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f78a806e870 con 0x7f78a80728d0 2024-03-24T00:37:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.965+0000 7f789ea00640 1 -- 172.21.15.40:0/307822363 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v105) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7898016090 con 0x7f78a80728d0 2024-03-24T00:37:49.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.969+0000 7f78ad000640 1 -- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7884067660 msgr2=0x7f7884069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.969+0000 7f78ad000640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7884067660 0x7f7884069b20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f7890009b60 tx=0x7f7890009190 comp rx=0 tx=0).stop 2024-03-24T00:37:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.969+0000 7f78ad000640 1 -- 172.21.15.40:0/307822363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78a80728d0 msgr2=0x7f78a80b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.969+0000 7f78ad000640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78a80728d0 0x7f78a80b3470 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f789801a0a0 tx=0x7f789801acb0 comp rx=0 tx=0).stop 2024-03-24T00:37:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.969+0000 7f78ad000640 1 -- 172.21.15.40:0/307822363 shutdown_connections 2024-03-24T00:37:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.969+0000 7f78ad000640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7884067660 0x7f7884069b20 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.969+0000 7f78ad000640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78a815f8c0 0x7f78a80aa180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.969+0000 7f78ad000640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78a815bd00 0x7f78a80a9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.969+0000 7f78ad000640 1 --2- 172.21.15.40:0/307822363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78a80728d0 0x7f78a80b3470 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.969+0000 7f78ad000640 1 -- 172.21.15.40:0/307822363 >> 172.21.15.40:0/307822363 conn(0x7f78a80b1660 msgr2=0x7f78a80aef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.969+0000 7f78ad000640 1 -- 172.21.15.40:0/307822363 shutdown_connections 2024-03-24T00:37:49.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:49.969+0000 7f78ad000640 1 -- 172.21.15.40:0/307822363 wait complete. 2024-03-24T00:37:49.996 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:37:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5ad5600640 1 Processor -- start 2024-03-24T00:37:50.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5ad5600640 1 -- start start 2024-03-24T00:37:50.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5ad5600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5ad015bd00 0x7f5ad00ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5ad5600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ad015ffd0 0x7f5ad00cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5ad5600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad0072d60 con 0x7f5ad00728d0 2024-03-24T00:37:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5ad5600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad00cf790 con 0x7f5ad015ffd0 2024-03-24T00:37:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5ad5600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad0071de0 con 0x7f5ad015bd00 2024-03-24T00:37:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5acf400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ad015ffd0 0x7f5ad00cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5acf400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5ad015ffd0 0x7f5ad00cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42234/0 (socket says 172.21.15.40:42234) 2024-03-24T00:37:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5acf400640 1 -- 172.21.15.40:0/2437925479 learned_addr learned my addr 172.21.15.40:0/2437925479 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5acea00640 1 --2- 172.21.15.40:0/2437925479 >> v2:172.21.15.53:3300/0 conn(0x7f5ad015bd00 0x7f5ad00ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5acf400640 1 -- 172.21.15.40:0/2437925479 >> v2:172.21.15.53:3300/0 conn(0x7f5ad015bd00 msgr2=0x7f5ad00ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5acf400640 1 --2- 172.21.15.40:0/2437925479 >> v2:172.21.15.53:3300/0 conn(0x7f5ad015bd00 0x7f5ad00ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5acf400640 1 -- 172.21.15.40:0/2437925479 >> v1:172.21.15.53:6789/0 conn(0x7f5ad00728d0 legacy=0x7f5ad015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:50.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.181+0000 7f5acf400640 1 -- 172.21.15.40:0/2437925479 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad00cea50 con 0x7f5ad015ffd0 2024-03-24T00:37:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.185+0000 7f5acf400640 1 --2- 172.21.15.40:0/2437925479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad015ffd0 0x7f5ad00cf250 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00733c0 tx=0x7f5ac801c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20d53de91c52c554 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.185+0000 7f5acd600640 1 -- 172.21.15.40:0/2437925479 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac80a5840 con 0x7f5ad015ffd0 2024-03-24T00:37:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.185+0000 7f5acd600640 1 -- 172.21.15.40:0/2437925479 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac80a59e0 con 0x7f5ad015ffd0 2024-03-24T00:37:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.185+0000 7f5acd600640 1 -- 172.21.15.40:0/2437925479 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac80a5db0 con 0x7f5ad015ffd0 2024-03-24T00:37:50.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.185+0000 7f5ad5600640 1 -- 172.21.15.40:0/2437925479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad015ffd0 msgr2=0x7f5ad00cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.185+0000 7f5ad5600640 1 --2- 172.21.15.40:0/2437925479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad015ffd0 0x7f5ad00cf250 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00733c0 tx=0x7f5ac801c980 comp rx=0 tx=0).stop 2024-03-24T00:37:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.185+0000 7f5ad5600640 1 -- 172.21.15.40:0/2437925479 shutdown_connections 2024-03-24T00:37:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.185+0000 7f5ad5600640 1 --2- 172.21.15.40:0/2437925479 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad015ffd0 0x7f5ad00cf250 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.185+0000 7f5ad5600640 1 --2- 172.21.15.40:0/2437925479 >> v2:172.21.15.53:3300/0 conn(0x7f5ad015bd00 0x7f5ad00ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.185+0000 7f5ad5600640 1 -- 172.21.15.40:0/2437925479 >> 172.21.15.40:0/2437925479 conn(0x7f5ad00b1660 msgr2=0x7f5ad00b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.185+0000 7f5ad5600640 1 -- 172.21.15.40:0/2437925479 shutdown_connections 2024-03-24T00:37:50.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.185+0000 7f5ad5600640 1 -- 172.21.15.40:0/2437925479 wait complete. 2024-03-24T00:37:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ad5600640 1 Processor -- start 2024-03-24T00:37:50.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ad5600640 1 -- start start 2024-03-24T00:37:50.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ad5600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00728d0 0x7f5ad013b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:50.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ad5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad015bd00 0x7f5ad0159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:50.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ad5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad015ffd0 0x7f5ad0152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:50.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ad5600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad006d320 con 0x7f5ad015bd00 2024-03-24T00:37:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ad5600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad015c0e0 con 0x7f5ad015ffd0 2024-03-24T00:37:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ad5600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad006d4a0 con 0x7f5ad00728d0 2024-03-24T00:37:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ace000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad015bd00 0x7f5ad0159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ace000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad015bd00 0x7f5ad0159d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42240/0 (socket says 172.21.15.40:42240) 2024-03-24T00:37:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ace000640 1 -- 172.21.15.40:0/508533925 learned_addr learned my addr 172.21.15.40:0/508533925 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5acea00640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00728d0 0x7f5ad013b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5acf400640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad015ffd0 0x7f5ad0152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ace000640 1 -- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00728d0 msgr2=0x7f5ad013b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ace000640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00728d0 0x7f5ad013b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ace000640 1 -- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad015ffd0 msgr2=0x7f5ad0152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ace000640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad015ffd0 0x7f5ad0152660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:50.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ace000640 1 -- 172.21.15.40:0/508533925 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad013b840 con 0x7f5ad015bd00 2024-03-24T00:37:50.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ace000640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad015bd00 0x7f5ad0159d10 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00c450 tx=0x7f5abc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:50.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ac6a00640 1 -- 172.21.15.40:0/508533925 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc012830 con 0x7f5ad015bd00 2024-03-24T00:37:50.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ac6a00640 1 -- 172.21.15.40:0/508533925 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc017070 con 0x7f5ad015bd00 2024-03-24T00:37:50.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ac6a00640 1 -- 172.21.15.40:0/508533925 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc012c30 con 0x7f5ad015bd00 2024-03-24T00:37:50.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ad5600640 1 -- 172.21.15.40:0/508533925 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad014fd10 con 0x7f5ad015bd00 2024-03-24T00:37:50.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.189+0000 7f5ad5600640 1 -- 172.21.15.40:0/508533925 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad01502d0 con 0x7f5ad015bd00 2024-03-24T00:37:50.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.193+0000 7f5ad5600640 1 -- 172.21.15.40:0/508533925 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad0068170 con 0x7f5ad015bd00 2024-03-24T00:37:50.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.193+0000 7f5ac6a00640 1 -- 172.21.15.40:0/508533925 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5abc01c460 con 0x7f5ad015bd00 2024-03-24T00:37:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.193+0000 7f5ac6a00640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5ab0067660 0x7f5ab0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.193+0000 7f5ac6a00640 1 -- 172.21.15.40:0/508533925 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5abc085630 con 0x7f5ad015bd00 2024-03-24T00:37:50.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.193+0000 7f5acea00640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5ab0067660 0x7f5ab0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:50.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.197+0000 7f5acea00640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5ab0067660 0x7f5ab0069b20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0007a60 tx=0x7f5ac0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:50.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.197+0000 7f5ac6a00640 1 -- 172.21.15.40:0/508533925 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5abc055b60 con 0x7f5ad015bd00 2024-03-24T00:37:50.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.365+0000 7f5ad5600640 1 -- 172.21.15.40:0/508533925 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5ad00d0700 con 0x7f5ad015bd00 2024-03-24T00:37:50.972 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.969+0000 7f5ac6a00640 1 -- 172.21.15.40:0/508533925 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v97) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5abc059c00 con 0x7f5ad015bd00 2024-03-24T00:37:50.972 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:37:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.973+0000 7f5ad5600640 1 -- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5ab0067660 msgr2=0x7f5ab0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.973+0000 7f5ad5600640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5ab0067660 0x7f5ab0069b20 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0007a60 tx=0x7f5ac0008040 comp rx=0 tx=0).stop 2024-03-24T00:37:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.973+0000 7f5ad5600640 1 -- 172.21.15.40:0/508533925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad015bd00 msgr2=0x7f5ad0159d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.973+0000 7f5ad5600640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad015bd00 0x7f5ad0159d10 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f5abc00c450 tx=0x7f5abc00c910 comp rx=0 tx=0).stop 2024-03-24T00:37:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.973+0000 7f5ad5600640 1 -- 172.21.15.40:0/508533925 shutdown_connections 2024-03-24T00:37:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.973+0000 7f5ad5600640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5ab0067660 0x7f5ab0069b20 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.973+0000 7f5ad5600640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad015ffd0 0x7f5ad0152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.973+0000 7f5ad5600640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5ad015bd00 0x7f5ad0159d10 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.973+0000 7f5ad5600640 1 --2- 172.21.15.40:0/508533925 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ad00728d0 0x7f5ad013b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.973+0000 7f5ad5600640 1 -- 172.21.15.40:0/508533925 >> 172.21.15.40:0/508533925 conn(0x7f5ad00b1660 msgr2=0x7f5ad00cfff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.973+0000 7f5ad5600640 1 -- 172.21.15.40:0/508533925 shutdown_connections 2024-03-24T00:37:50.977 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:50.973+0000 7f5ad5600640 1 -- 172.21.15.40:0/508533925 wait complete. 2024-03-24T00:37:51.005 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:37:51.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.149+0000 7fe8ace00640 1 Processor -- start 2024-03-24T00:37:51.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.149+0000 7fe8ace00640 1 -- start start 2024-03-24T00:37:51.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8ace00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe8a806cb80 0x7fe8a80b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:51.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8ace00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8a80b4840 0x7fe8a8071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:51.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8ace00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a80b3260 con 0x7fe8a815ac60 2024-03-24T00:37:51.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8ace00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a80b3e70 con 0x7fe8a80b4840 2024-03-24T00:37:51.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8ace00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a80b3540 con 0x7fe8a806cb80 2024-03-24T00:37:51.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8a6800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8a80b4840 0x7fe8a8071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:51.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8a5e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fe8a815ac60 0x7fe8a80b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42830/0 (socket says 172.21.15.40:42830) 2024-03-24T00:37:51.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8a5e00640 1 -- 172.21.15.40:0/2449759700 learned_addr learned my addr 172.21.15.40:0/2449759700 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:51.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8a6800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8a80b4840 0x7fe8a8071b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42242/0 (socket says 172.21.15.40:42242) 2024-03-24T00:37:51.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe89d400640 1 --2- 172.21.15.40:0/2449759700 >> v2:172.21.15.53:3300/0 conn(0x7fe8a806cb80 0x7fe8a80b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:51.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8a6800640 1 -- 172.21.15.40:0/2449759700 >> v2:172.21.15.53:3300/0 conn(0x7fe8a806cb80 msgr2=0x7fe8a80b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:51.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8a6800640 1 --2- 172.21.15.40:0/2449759700 >> v2:172.21.15.53:3300/0 conn(0x7fe8a806cb80 0x7fe8a80b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8a6800640 1 -- 172.21.15.40:0/2449759700 >> v1:172.21.15.40:6789/0 conn(0x7fe8a815ac60 legacy=0x7fe8a80b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:51.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8a6800640 1 -- 172.21.15.40:0/2449759700 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a80b3820 con 0x7fe8a80b4840 2024-03-24T00:37:51.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.153+0000 7fe8a5400640 1 -- 172.21.15.40:0/2449759700 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065169348 0 0) 0x7fe8a80b3260 con 0x7fe8a815ac60 2024-03-24T00:37:51.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8a6800640 1 --2- 172.21.15.40:0/2449759700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a80b4840 0x7fe8a8071b00 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fe8a815e7d0 tx=0x7fe8a001eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e736ce44e1686c9a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:51.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8a5400640 1 -- 172.21.15.40:0/2449759700 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a000b2d0 con 0x7fe8a80b4840 2024-03-24T00:37:51.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8a5400640 1 -- 172.21.15.40:0/2449759700 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8a00a7770 con 0x7fe8a80b4840 2024-03-24T00:37:51.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8a5400640 1 -- 172.21.15.40:0/2449759700 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8a00a7b40 con 0x7fe8a80b4840 2024-03-24T00:37:51.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8ace00640 1 -- 172.21.15.40:0/2449759700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a80b4840 msgr2=0x7fe8a8071b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:51.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8ace00640 1 --2- 172.21.15.40:0/2449759700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a80b4840 0x7fe8a8071b00 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7fe8a815e7d0 tx=0x7fe8a001eaa0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8ace00640 1 -- 172.21.15.40:0/2449759700 shutdown_connections 2024-03-24T00:37:51.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8ace00640 1 --2- 172.21.15.40:0/2449759700 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a80b4840 0x7fe8a8071b00 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8ace00640 1 --2- 172.21.15.40:0/2449759700 >> v2:172.21.15.53:3300/0 conn(0x7fe8a806cb80 0x7fe8a80b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8ace00640 1 -- 172.21.15.40:0/2449759700 >> 172.21.15.40:0/2449759700 conn(0x7fe8a80b1660 msgr2=0x7fe8a80b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:51.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8ace00640 1 -- 172.21.15.40:0/2449759700 shutdown_connections 2024-03-24T00:37:51.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8ace00640 1 -- 172.21.15.40:0/2449759700 wait complete. 2024-03-24T00:37:51.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8ace00640 1 Processor -- start 2024-03-24T00:37:51.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.157+0000 7fe8ace00640 1 -- start start 2024-03-24T00:37:51.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe8ace00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8a806cb80 0x7fe8a814c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe8ace00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a80b4840 0x7fe8a80cfc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe8ace00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8a815ac60 0x7fe8a814e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe8ace00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a806d770 con 0x7fe8a80b4840 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe8ace00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a806d5f0 con 0x7fe8a806cb80 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe8ace00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a806d8f0 con 0x7fe8a815ac60 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a80b4840 0x7fe8a80cfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a80b4840 0x7fe8a80cfc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42258/0 (socket says 172.21.15.40:42258) 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89d400640 1 -- 172.21.15.40:0/2950782912 learned_addr learned my addr 172.21.15.40:0/2950782912 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89d400640 1 -- 172.21.15.40:0/2950782912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8a815ac60 msgr2=0x7fe8a814e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89d400640 1 --2- 172.21.15.40:0/2950782912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8a815ac60 0x7fe8a814e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89d400640 1 -- 172.21.15.40:0/2950782912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8a806cb80 msgr2=0x7fe8a814c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89d400640 1 --2- 172.21.15.40:0/2950782912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8a806cb80 0x7fe8a814c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89d400640 1 -- 172.21.15.40:0/2950782912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a8155e00 con 0x7fe8a80b4840 2024-03-24T00:37:51.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89d400640 1 --2- 172.21.15.40:0/2950782912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a80b4840 0x7fe8a80cfc00 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe89000e900 tx=0x7fe89000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:51.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89ea00640 1 -- 172.21.15.40:0/2950782912 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890017070 con 0x7fe8a80b4840 2024-03-24T00:37:51.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89ea00640 1 -- 172.21.15.40:0/2950782912 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89000ca00 con 0x7fe8a80b4840 2024-03-24T00:37:51.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe8ace00640 1 -- 172.21.15.40:0/2950782912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a81560e0 con 0x7fe8a80b4840 2024-03-24T00:37:51.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89ea00640 1 -- 172.21.15.40:0/2950782912 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89000cdf0 con 0x7fe8a80b4840 2024-03-24T00:37:51.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe8ace00640 1 -- 172.21.15.40:0/2950782912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a81566a0 con 0x7fe8a80b4840 2024-03-24T00:37:51.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe8ace00640 1 -- 172.21.15.40:0/2950782912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a8068170 con 0x7fe8a80b4840 2024-03-24T00:37:51.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.161+0000 7fe89ea00640 1 -- 172.21.15.40:0/2950782912 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe89001a070 con 0x7fe8a80b4840 2024-03-24T00:37:51.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.165+0000 7fe89ea00640 1 --2- 172.21.15.40:0/2950782912 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe884067660 0x7fe884069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:51.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.165+0000 7fe89ea00640 1 -- 172.21.15.40:0/2950782912 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5529+0+0 (secure 0 0 0) 0x7fe890085770 con 0x7fe8a80b4840 2024-03-24T00:37:51.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.165+0000 7fe8a5e00640 1 --2- 172.21.15.40:0/2950782912 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe884067660 0x7fe884069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:51.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.165+0000 7fe8a5e00640 1 --2- 172.21.15.40:0/2950782912 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe884067660 0x7fe884069b20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe8980015a0 tx=0x7fe898007970 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:51.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.169+0000 7fe89ea00640 1 -- 172.21.15.40:0/2950782912 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe890056530 con 0x7fe8a80b4840 2024-03-24T00:37:51.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.333+0000 7fe8ace00640 1 -- 172.21.15.40:0/2950782912 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe8a80cf6e0 con 0x7fe8a80b4840 2024-03-24T00:37:51.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.973+0000 7fe89ea00640 1 -- 172.21.15.40:0/2950782912 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v98) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe890055e30 con 0x7fe8a80b4840 2024-03-24T00:37:51.979 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-03-24T00:37:51.981 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.977+0000 7fe8ace00640 1 -- 172.21.15.40:0/2950782912 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe884067660 msgr2=0x7fe884069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.977+0000 7fe8ace00640 1 --2- 172.21.15.40:0/2950782912 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe884067660 0x7fe884069b20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe8980015a0 tx=0x7fe898007970 comp rx=0 tx=0).stop 2024-03-24T00:37:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.977+0000 7fe8ace00640 1 -- 172.21.15.40:0/2950782912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a80b4840 msgr2=0x7fe8a80cfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.977+0000 7fe8ace00640 1 --2- 172.21.15.40:0/2950782912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a80b4840 0x7fe8a80cfc00 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fe89000e900 tx=0x7fe89000edc0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.977+0000 7fe8ace00640 1 -- 172.21.15.40:0/2950782912 shutdown_connections 2024-03-24T00:37:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.977+0000 7fe8ace00640 1 --2- 172.21.15.40:0/2950782912 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe884067660 0x7fe884069b20 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.977+0000 7fe8ace00640 1 --2- 172.21.15.40:0/2950782912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8a815ac60 0x7fe8a814e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.977+0000 7fe8ace00640 1 --2- 172.21.15.40:0/2950782912 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8a80b4840 0x7fe8a80cfc00 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.977+0000 7fe8ace00640 1 --2- 172.21.15.40:0/2950782912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8a806cb80 0x7fe8a814c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.977+0000 7fe8ace00640 1 -- 172.21.15.40:0/2950782912 >> 172.21.15.40:0/2950782912 conn(0x7fe8a80b1660 msgr2=0x7fe8a80b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.977+0000 7fe8ace00640 1 -- 172.21.15.40:0/2950782912 shutdown_connections 2024-03-24T00:37:51.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:51.977+0000 7fe8ace00640 1 -- 172.21.15.40:0/2950782912 wait complete. 2024-03-24T00:37:52.008 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random 2024-03-24T00:37:52.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff333800640 1 Processor -- start 2024-03-24T00:37:52.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff333800640 1 -- start start 2024-03-24T00:37:52.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff333800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff32c15ac60 0x7ff32c0b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff333800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff32c06cb80 0x7ff32c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff333800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32c0b3260 con 0x7ff32c0b4840 2024-03-24T00:37:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff333800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff32c0b3e70 con 0x7ff32c06cb80 2024-03-24T00:37:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff333800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c0b3540 con 0x7ff32c15ac60 2024-03-24T00:37:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff32be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff32c06cb80 0x7ff32c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff330e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff32c15ac60 0x7ff32c0b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff32be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff32c06cb80 0x7ff32c0b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:54982/0 (socket says 172.21.15.40:54982) 2024-03-24T00:37:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff32be00640 1 -- 172.21.15.40:0/2020798922 learned_addr learned my addr 172.21.15.40:0/2020798922 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff331800640 1 --1- 172.21.15.40:0/2020798922 >> v1:172.21.15.53:6789/0 conn(0x7ff32c0b4840 0x7ff32c0ccd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:60520/0 (socket says 172.21.15.40:60520) 2024-03-24T00:37:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff32be00640 1 -- 172.21.15.40:0/2020798922 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff32c15ac60 msgr2=0x7ff32c0b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff32be00640 1 --2- 172.21.15.40:0/2020798922 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff32c15ac60 0x7ff32c0b2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff32be00640 1 -- 172.21.15.40:0/2020798922 >> v1:172.21.15.53:6789/0 conn(0x7ff32c0b4840 legacy=0x7ff32c0ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.153+0000 7ff32be00640 1 -- 172.21.15.40:0/2020798922 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c0b3820 con 0x7ff32c06cb80 2024-03-24T00:37:52.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.157+0000 7ff32be00640 1 --2- 172.21.15.40:0/2020798922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff32c06cb80 0x7ff32c0b4300 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff318009870 tx=0x7ff31801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17adfe5e61763cc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.157+0000 7ff32b400640 1 -- 172.21.15.40:0/2020798922 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3180b1070 con 0x7ff32c06cb80 2024-03-24T00:37:52.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.157+0000 7ff32b400640 1 -- 172.21.15.40:0/2020798922 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3180ac410 con 0x7ff32c06cb80 2024-03-24T00:37:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.157+0000 7ff32b400640 1 -- 172.21.15.40:0/2020798922 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3180ac7e0 con 0x7ff32c06cb80 2024-03-24T00:37:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.157+0000 7ff333800640 1 -- 172.21.15.40:0/2020798922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff32c06cb80 msgr2=0x7ff32c0b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.157+0000 7ff333800640 1 --2- 172.21.15.40:0/2020798922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff32c06cb80 0x7ff32c0b4300 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff318009870 tx=0x7ff31801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.157+0000 7ff333800640 1 -- 172.21.15.40:0/2020798922 shutdown_connections 2024-03-24T00:37:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.157+0000 7ff333800640 1 --2- 172.21.15.40:0/2020798922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff32c06cb80 0x7ff32c0b4300 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.157+0000 7ff333800640 1 --2- 172.21.15.40:0/2020798922 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff32c15ac60 0x7ff32c0b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.157+0000 7ff333800640 1 -- 172.21.15.40:0/2020798922 >> 172.21.15.40:0/2020798922 conn(0x7ff32c0b1660 msgr2=0x7ff32c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.157+0000 7ff333800640 1 -- 172.21.15.40:0/2020798922 shutdown_connections 2024-03-24T00:37:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.157+0000 7ff333800640 1 -- 172.21.15.40:0/2020798922 wait complete. 2024-03-24T00:37:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff333800640 1 Processor -- start 2024-03-24T00:37:52.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff333800640 1 -- start start 2024-03-24T00:37:52.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff333800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff32c06cb80 0x7ff32c13d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff333800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff32c0b4840 0x7ff32c1f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff333800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff32c15ac60 0x7ff32c15da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff333800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c167a40 con 0x7ff32c0b4840 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff333800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c1678c0 con 0x7ff32c06cb80 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff333800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c167bc0 con 0x7ff32c15ac60 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff32be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff32c0b4840 0x7ff32c1f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff32be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff32c0b4840 0x7ff32c1f4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42272/0 (socket says 172.21.15.40:42272) 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff32be00640 1 -- 172.21.15.40:0/2609420238 learned_addr learned my addr 172.21.15.40:0/2609420238 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff330e00640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff32c06cb80 0x7ff32c13d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff331800640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff32c15ac60 0x7ff32c15da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff32be00640 1 -- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff32c15ac60 msgr2=0x7ff32c15da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff32be00640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff32c15ac60 0x7ff32c15da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff32be00640 1 -- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff32c06cb80 msgr2=0x7ff32c13d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff32be00640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff32c06cb80 0x7ff32c13d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff32be00640 1 -- 172.21.15.40:0/2609420238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c152de0 con 0x7ff32c0b4840 2024-03-24T00:37:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff331800640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff32c15ac60 0x7ff32c15da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff32be00640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff32c0b4840 0x7ff32c1f4f00 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7ff31801ab50 tx=0x7ff318019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff328c00640 1 -- 172.21.15.40:0/2609420238 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3180bc070 con 0x7ff32c0b4840 2024-03-24T00:37:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff328c00640 1 -- 172.21.15.40:0/2609420238 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3180ab5c0 con 0x7ff32c0b4840 2024-03-24T00:37:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff333800640 1 -- 172.21.15.40:0/2609420238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32c1530c0 con 0x7ff32c0b4840 2024-03-24T00:37:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff328c00640 1 -- 172.21.15.40:0/2609420238 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3180b1070 con 0x7ff32c0b4840 2024-03-24T00:37:52.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff333800640 1 -- 172.21.15.40:0/2609420238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32c14b340 con 0x7ff32c0b4840 2024-03-24T00:37:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.161+0000 7ff333800640 1 -- 172.21.15.40:0/2609420238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32c153570 con 0x7ff32c0b4840 2024-03-24T00:37:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.165+0000 7ff328c00640 1 -- 172.21.15.40:0/2609420238 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff3180bf070 con 0x7ff32c0b4840 2024-03-24T00:37:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.165+0000 7ff328c00640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff30c067660 0x7ff30c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.165+0000 7ff328c00640 1 -- 172.21.15.40:0/2609420238 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5115+0+0 (secure 0 0 0) 0x7ff31811f090 con 0x7ff32c0b4840 2024-03-24T00:37:52.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.165+0000 7ff330e00640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff30c067660 0x7ff30c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.165+0000 7ff330e00640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff30c067660 0x7ff30c069b20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff320009b60 tx=0x7ff320009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.169+0000 7ff328c00640 1 -- 172.21.15.40:0/2609420238 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff32c153570 con 0x7ff32c0b4840 2024-03-24T00:37:52.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.337+0000 7ff333800640 1 -- 172.21.15.40:0/2609420238 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"} v 0) v1 -- 0x7ff32c068170 con 0x7ff32c0b4840 2024-03-24T00:37:52.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff328c00640 1 -- 172.21.15.40:0/2609420238 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"}]=0 v55) v1 ==== 110+0+0 (secure 0 0 0) 0x7ff318016150 con 0x7ff32c0b4840 2024-03-24T00:37:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff333800640 1 -- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff30c067660 msgr2=0x7ff30c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff333800640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff30c067660 0x7ff30c069b20 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff320009b60 tx=0x7ff320009190 comp rx=0 tx=0).stop 2024-03-24T00:37:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff333800640 1 -- 172.21.15.40:0/2609420238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff32c0b4840 msgr2=0x7ff32c1f4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff333800640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff32c0b4840 0x7ff32c1f4f00 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7ff31801ab50 tx=0x7ff318019240 comp rx=0 tx=0).stop 2024-03-24T00:37:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff333800640 1 -- 172.21.15.40:0/2609420238 shutdown_connections 2024-03-24T00:37:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff333800640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff30c067660 0x7ff30c069b20 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff333800640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff32c15ac60 0x7ff32c15da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff333800640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff32c0b4840 0x7ff32c1f4f00 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff333800640 1 --2- 172.21.15.40:0/2609420238 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff32c06cb80 0x7ff32c13d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff333800640 1 -- 172.21.15.40:0/2609420238 >> 172.21.15.40:0/2609420238 conn(0x7ff32c0b1660 msgr2=0x7ff32c0b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:52.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff333800640 1 -- 172.21.15.40:0/2609420238 shutdown_connections 2024-03-24T00:37:52.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.341+0000 7ff333800640 1 -- 172.21.15.40:0/2609420238 wait complete. 2024-03-24T00:37:52.367 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_distributed 2024-03-24T00:37:52.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.509+0000 7f67a4800640 1 Processor -- start 2024-03-24T00:37:52.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.509+0000 7f67a4800640 1 -- start start 2024-03-24T00:37:52.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a4800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f679c06cb80 0x7f679c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a4800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f679c0b4840 0x7f679c071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a4800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679c0b3260 con 0x7f679c15ac60 2024-03-24T00:37:52.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a4800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679c0b3e70 con 0x7f679c0b4840 2024-03-24T00:37:52.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a4800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679c0b3540 con 0x7f679c06cb80 2024-03-24T00:37:52.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a1e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f679c15ac60 0x7f679c0b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42838/0 (socket says 172.21.15.40:42838) 2024-03-24T00:37:52.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a1e00640 1 -- 172.21.15.40:0/1133103382 learned_addr learned my addr 172.21.15.40:0/1133103382 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:52.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662503144 0 0) 0x7f679c0b3260 con 0x7f679c15ac60 2024-03-24T00:37:52.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a1400640 1 --2- 172.21.15.40:0/1133103382 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f679c06cb80 0x7f679c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6784005740 con 0x7f679c15ac60 2024-03-24T00:37:52.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a2800640 1 --2- 172.21.15.40:0/1133103382 >> v2:172.21.15.53:3300/0 conn(0x7f679c0b4840 0x7f679c071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2062897030 0 0) 0x7f6784005740 con 0x7f679c15ac60 2024-03-24T00:37:52.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f679c06cb80 msgr2=0x7f679c0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:52.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 --2- 172.21.15.40:0/1133103382 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f679c06cb80 0x7f679c0b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 >> v2:172.21.15.53:3300/0 conn(0x7f679c0b4840 msgr2=0x7f679c071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:52.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 --2- 172.21.15.40:0/1133103382 >> v2:172.21.15.53:3300/0 conn(0x7f679c0b4840 0x7f679c071b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c0b3820 con 0x7f679c15ac60 2024-03-24T00:37:52.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f6790001680 con 0x7f679c15ac60 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 >> v1:172.21.15.40:6789/0 conn(0x7f679c15ac60 legacy=0x7f679c0b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 --2- 172.21.15.40:0/1133103382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6784005bd0 0x7f6784005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 --2- 172.21.15.40:0/1133103382 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f679c06cb80 0x7f6784006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 --2- 172.21.15.40:0/1133103382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6784006f60 0x7f67840073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c0b3e70 con 0x7f6784005bd0 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c0b3540 con 0x7f6784006f60 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678400dbe0 con 0x7f679c06cb80 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a1e00640 1 --2- 172.21.15.40:0/1133103382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6784005bd0 0x7f6784005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a1400640 1 --2- 172.21.15.40:0/1133103382 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f679c06cb80 0x7f6784006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a2800640 1 --2- 172.21.15.40:0/1133103382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6784006f60 0x7f67840073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a1e00640 1 -- 172.21.15.40:0/1133103382 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f679c06cb80 msgr2=0x7f6784006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a1e00640 1 --2- 172.21.15.40:0/1133103382 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f679c06cb80 0x7f6784006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a1e00640 1 -- 172.21.15.40:0/1133103382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6784006f60 msgr2=0x7f67840073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a1e00640 1 --2- 172.21.15.40:0/1133103382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6784006f60 0x7f67840073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.513+0000 7f67a1e00640 1 -- 172.21.15.40:0/1133103382 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6784005740 con 0x7f6784005bd0 2024-03-24T00:37:52.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.517+0000 7f67a1e00640 1 --2- 172.21.15.40:0/1133103382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6784005bd0 0x7f6784005ff0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f679000c2c0 tx=0x7f679001cd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287774e0d3575733 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.517+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6790007d60 con 0x7f6784005bd0 2024-03-24T00:37:52.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.517+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67900a9020 con 0x7f6784005bd0 2024-03-24T00:37:52.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.517+0000 7f67a0a00640 1 -- 172.21.15.40:0/1133103382 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6790005ae0 con 0x7f6784005bd0 2024-03-24T00:37:52.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.517+0000 7f67a4800640 1 -- 172.21.15.40:0/1133103382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6784005bd0 msgr2=0x7f6784005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.517+0000 7f67a4800640 1 --2- 172.21.15.40:0/1133103382 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6784005bd0 0x7f6784005ff0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f679000c2c0 tx=0x7f679001cd40 comp rx=0 tx=0).stop 2024-03-24T00:37:52.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.517+0000 7f67a2800640 1 -- 172.21.15.40:0/1133103382 reap_dead start 2024-03-24T00:37:52.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.517+0000 7f67a4800640 1 -- 172.21.15.40:0/1133103382 shutdown_connections 2024-03-24T00:37:52.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.517+0000 7f67a4800640 1 -- 172.21.15.40:0/1133103382 >> 172.21.15.40:0/1133103382 conn(0x7f679c0b1660 msgr2=0x7f679c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:52.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.517+0000 7f67a4800640 1 -- 172.21.15.40:0/1133103382 shutdown_connections 2024-03-24T00:37:52.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.517+0000 7f67a4800640 1 -- 172.21.15.40:0/1133103382 wait complete. 2024-03-24T00:37:52.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a4800640 1 Processor -- start 2024-03-24T00:37:52.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a4800640 1 -- start start 2024-03-24T00:37:52.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a4800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6784005bd0 0x7f679c150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a4800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f679c14e070 0x7f679c150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a4800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f679c0cec70 0x7f679c0b0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a4800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c072450 con 0x7f679c0cec70 2024-03-24T00:37:52.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a4800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c0cd7b0 con 0x7f679c14e070 2024-03-24T00:37:52.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a4800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679c06d3a0 con 0x7f6784005bd0 2024-03-24T00:37:52.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f679c0cec70 0x7f679c0b0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a2800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f679c0cec70 0x7f679c0b0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42296/0 (socket says 172.21.15.40:42296) 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a2800640 1 -- 172.21.15.40:0/3727217158 learned_addr learned my addr 172.21.15.40:0/3727217158 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a1400640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f679c14e070 0x7f679c150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a1e00640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6784005bd0 0x7f679c150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a2800640 1 -- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6784005bd0 msgr2=0x7f679c150810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a2800640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6784005bd0 0x7f679c150810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a2800640 1 -- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f679c14e070 msgr2=0x7f679c150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a2800640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f679c14e070 0x7f679c150fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a2800640 1 -- 172.21.15.40:0/3727217158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c14e4d0 con 0x7f679c0cec70 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a1e00640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6784005bd0 0x7f679c150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a2800640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f679c0cec70 0x7f679c0b0620 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f679c072ba0 tx=0x7f679800f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f6796000640 1 -- 172.21.15.40:0/3727217158 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679800fce0 con 0x7f679c0cec70 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f6796000640 1 -- 172.21.15.40:0/3727217158 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679801d070 con 0x7f679c0cec70 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f6796000640 1 -- 172.21.15.40:0/3727217158 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6798015670 con 0x7f679c0cec70 2024-03-24T00:37:52.527 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a4800640 1 -- 172.21.15.40:0/3727217158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c151500 con 0x7f679c0cec70 2024-03-24T00:37:52.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a4800640 1 -- 172.21.15.40:0/3727217158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c0ce190 con 0x7f679c0cec70 2024-03-24T00:37:52.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.521+0000 7f67a4800640 1 -- 172.21.15.40:0/3727217158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679c068170 con 0x7f679c0cec70 2024-03-24T00:37:52.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.525+0000 7f6796000640 1 -- 172.21.15.40:0/3727217158 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6798015810 con 0x7f679c0cec70 2024-03-24T00:37:52.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.525+0000 7f6796000640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6780067660 0x7f6780069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.525+0000 7f6796000640 1 -- 172.21.15.40:0/3727217158 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f6798085f80 con 0x7f679c0cec70 2024-03-24T00:37:52.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.525+0000 7f67a1e00640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6780067660 0x7f6780069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.525+0000 7f67a1e00640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6780067660 0x7f6780069b20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f67900aa000 tx=0x7f6790001a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.529+0000 7f6796000640 1 -- 172.21.15.40:0/3727217158 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6798010040 con 0x7f679c0cec70 2024-03-24T00:37:52.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.693+0000 7f67a4800640 1 -- 172.21.15.40:0/3727217158 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"} v 0) v1 -- 0x7f679c161670 con 0x7f679c0cec70 2024-03-24T00:37:52.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f6796000640 1 -- 172.21.15.40:0/3727217158 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"}]=0 v56) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6798056db0 con 0x7f679c0cec70 2024-03-24T00:37:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f67a4800640 1 -- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6780067660 msgr2=0x7f6780069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f67a4800640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6780067660 0x7f6780069b20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f67900aa000 tx=0x7f6790001a50 comp rx=0 tx=0).stop 2024-03-24T00:37:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f67a4800640 1 -- 172.21.15.40:0/3727217158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f679c0cec70 msgr2=0x7f679c0b0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f67a4800640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f679c0cec70 0x7f679c0b0620 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f679c072ba0 tx=0x7f679800f1b0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f67a4800640 1 -- 172.21.15.40:0/3727217158 shutdown_connections 2024-03-24T00:37:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f67a4800640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6780067660 0x7f6780069b20 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f67a4800640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f679c0cec70 0x7f679c0b0620 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f67a4800640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f679c14e070 0x7f679c150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f67a4800640 1 --2- 172.21.15.40:0/3727217158 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6784005bd0 0x7f679c150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f67a4800640 1 -- 172.21.15.40:0/3727217158 >> 172.21.15.40:0/3727217158 conn(0x7f679c0b1660 msgr2=0x7f679c14f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:52.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f67a4800640 1 -- 172.21.15.40:0/3727217158 shutdown_connections 2024-03-24T00:37:52.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.697+0000 7f67a4800640 1 -- 172.21.15.40:0/3727217158 wait complete. 2024-03-24T00:37:52.722 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random_max 2024-03-24T00:37:52.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4008c00640 1 Processor -- start 2024-03-24T00:37:52.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4008c00640 1 -- start start 2024-03-24T00:37:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4008c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f400415bb10 0x7f400406f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4008c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f400406f570 0x7f4004066ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4008c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4004067c60 con 0x7f4004067430 2024-03-24T00:37:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4008c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f400406f9d0 con 0x7f400415bb10 2024-03-24T00:37:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4008c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400415ef60 con 0x7f400406f570 2024-03-24T00:37:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4001e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f400415bb10 0x7f400406f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4001e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f400415bb10 0x7f400406f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42300/0 (socket says 172.21.15.40:42300) 2024-03-24T00:37:52.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4001e00640 1 -- 172.21.15.40:0/2181276856 learned_addr learned my addr 172.21.15.40:0/2181276856 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4001e00640 1 -- 172.21.15.40:0/2181276856 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f400406f570 msgr2=0x7f4004066ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4001e00640 1 --2- 172.21.15.40:0/2181276856 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f400406f570 0x7f4004066ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4001e00640 1 -- 172.21.15.40:0/2181276856 >> v1:172.21.15.40:6789/0 conn(0x7f4004067430 legacy=0x7f4004071b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.861+0000 7f4001e00640 1 -- 172.21.15.40:0/2181276856 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400415e910 con 0x7f400415bb10 2024-03-24T00:37:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.865+0000 7f4001e00640 1 --2- 172.21.15.40:0/2181276856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f400415bb10 0x7f400406f030 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009e50 tx=0x7f3ff401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71fb4bff0a8931df server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.865+0000 7f4001400640 1 -- 172.21.15.40:0/2181276856 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff40a5720 con 0x7f400415bb10 2024-03-24T00:37:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.865+0000 7f4001400640 1 -- 172.21.15.40:0/2181276856 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff40a58c0 con 0x7f400415bb10 2024-03-24T00:37:52.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.865+0000 7f4001400640 1 -- 172.21.15.40:0/2181276856 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff40a5c90 con 0x7f400415bb10 2024-03-24T00:37:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.865+0000 7f4008c00640 1 -- 172.21.15.40:0/2181276856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f400415bb10 msgr2=0x7f400406f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.865+0000 7f4008c00640 1 --2- 172.21.15.40:0/2181276856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f400415bb10 0x7f400406f030 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009e50 tx=0x7f3ff401ca30 comp rx=0 tx=0).stop 2024-03-24T00:37:52.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.865+0000 7f4008c00640 1 -- 172.21.15.40:0/2181276856 shutdown_connections 2024-03-24T00:37:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.865+0000 7f4008c00640 1 --2- 172.21.15.40:0/2181276856 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f400406f570 0x7f4004066ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.865+0000 7f4008c00640 1 --2- 172.21.15.40:0/2181276856 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f400415bb10 0x7f400406f030 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.865+0000 7f4008c00640 1 -- 172.21.15.40:0/2181276856 >> 172.21.15.40:0/2181276856 conn(0x7f40040b27c0 msgr2=0x7f40040b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.865+0000 7f4008c00640 1 -- 172.21.15.40:0/2181276856 shutdown_connections 2024-03-24T00:37:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.865+0000 7f4008c00640 1 -- 172.21.15.40:0/2181276856 wait complete. 2024-03-24T00:37:52.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4008c00640 1 Processor -- start 2024-03-24T00:37:52.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4008c00640 1 -- start start 2024-03-24T00:37:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4008c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4004067430 0x7f400413d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4008c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f400406f570 0x7f400413a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4008c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f400415bb10 0x7f4004152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4008c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400415f570 con 0x7f400406f570 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4008c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400415f3f0 con 0x7f400415bb10 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4008c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4004067840 con 0x7f4004067430 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f3ff9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f400406f570 0x7f400413a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f3ff9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f400406f570 0x7f400413a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42312/0 (socket says 172.21.15.40:42312) 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f3ff9400640 1 -- 172.21.15.40:0/612487776 learned_addr learned my addr 172.21.15.40:0/612487776 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4001e00640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4004067430 0x7f400413d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f3ff9400640 1 -- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4004067430 msgr2=0x7f400413d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4002800640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f400415bb10 0x7f4004152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f3ff9400640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4004067430 0x7f400413d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f3ff9400640 1 -- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f400415bb10 msgr2=0x7f4004152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f3ff9400640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f400415bb10 0x7f4004152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:52.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f3ff9400640 1 -- 172.21.15.40:0/612487776 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4004152fb0 con 0x7f400406f570 2024-03-24T00:37:52.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4001e00640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4004067430 0x7f400413d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:37:52.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4002800640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f400415bb10 0x7f4004152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:37:52.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f3ff9400640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f400406f570 0x7f400413a3b0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00b550 tx=0x7f3fec00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f3ffaa00640 1 -- 172.21.15.40:0/612487776 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec014070 con 0x7f400406f570 2024-03-24T00:37:52.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f3ffaa00640 1 -- 172.21.15.40:0/612487776 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fec010890 con 0x7f400406f570 2024-03-24T00:37:52.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4008c00640 1 -- 172.21.15.40:0/612487776 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4004152cd0 con 0x7f400406f570 2024-03-24T00:37:52.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f3ffaa00640 1 -- 172.21.15.40:0/612487776 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fec010c80 con 0x7f400406f570 2024-03-24T00:37:52.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.869+0000 7f4008c00640 1 -- 172.21.15.40:0/612487776 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f400413d120 con 0x7f400406f570 2024-03-24T00:37:52.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.873+0000 7f4008c00640 1 -- 172.21.15.40:0/612487776 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4004068170 con 0x7f400406f570 2024-03-24T00:37:52.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.873+0000 7f3ffaa00640 1 -- 172.21.15.40:0/612487776 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3fec017070 con 0x7f400406f570 2024-03-24T00:37:52.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.873+0000 7f3ffaa00640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3fe0067660 0x7f3fe0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:52.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.873+0000 7f3ffaa00640 1 -- 172.21.15.40:0/612487776 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3fec085410 con 0x7f400406f570 2024-03-24T00:37:52.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.873+0000 7f4001e00640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3fe0067660 0x7f3fe0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:52.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.873+0000 7f4001e00640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3fe0067660 0x7f3fe0069b20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40ae000 tx=0x7f3ff40a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:52.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:52.877+0000 7f3ffaa00640 1 -- 172.21.15.40:0/612487776 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3fec0562d0 con 0x7f400406f570 2024-03-24T00:37:53.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.041+0000 7f4008c00640 1 -- 172.21.15.40:0/612487776 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"} v 0) v1 -- 0x7f4004150640 con 0x7f400406f570 2024-03-24T00:37:53.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f3ffaa00640 1 -- 172.21.15.40:0/612487776 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"}]=0 v57) v1 ==== 114+0+0 (secure 0 0 0) 0x7f3fec010e20 con 0x7f400406f570 2024-03-24T00:37:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f4008c00640 1 -- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3fe0067660 msgr2=0x7f3fe0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f4008c00640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3fe0067660 0x7f3fe0069b20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40ae000 tx=0x7f3ff40a7070 comp rx=0 tx=0).stop 2024-03-24T00:37:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f4008c00640 1 -- 172.21.15.40:0/612487776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f400406f570 msgr2=0x7f400413a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f4008c00640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f400406f570 0x7f400413a3b0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00b550 tx=0x7f3fec00ba10 comp rx=0 tx=0).stop 2024-03-24T00:37:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f4008c00640 1 -- 172.21.15.40:0/612487776 shutdown_connections 2024-03-24T00:37:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f4008c00640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3fe0067660 0x7f3fe0069b20 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f4008c00640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f400415bb10 0x7f4004152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f4008c00640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f400406f570 0x7f400413a3b0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f4008c00640 1 --2- 172.21.15.40:0/612487776 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4004067430 0x7f400413d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f4008c00640 1 -- 172.21.15.40:0/612487776 >> 172.21.15.40:0/612487776 conn(0x7f40040b27c0 msgr2=0x7f400406ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:53.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f4008c00640 1 -- 172.21.15.40:0/612487776 shutdown_connections 2024-03-24T00:37:53.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.045+0000 7f4008c00640 1 -- 172.21.15.40:0/612487776 wait complete. 2024-03-24T00:37:53.070 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:37:53.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f570e600640 1 Processor -- start 2024-03-24T00:37:53.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f570e600640 1 -- start start 2024-03-24T00:37:53.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f570e600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f570815ac60 0x7f57080b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f570e600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f570806cb80 0x7f57080b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f570e600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57080b3260 con 0x7f57080b4840 2024-03-24T00:37:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f570e600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57080b3e70 con 0x7f570806cb80 2024-03-24T00:37:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f570e600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57080b3540 con 0x7f570815ac60 2024-03-24T00:37:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f56fea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f570806cb80 0x7f57080b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f56fea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f570806cb80 0x7f57080b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42322/0 (socket says 172.21.15.40:42322) 2024-03-24T00:37:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f56fea00640 1 -- 172.21.15.40:0/1779127428 learned_addr learned my addr 172.21.15.40:0/1779127428 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f56fea00640 1 -- 172.21.15.40:0/1779127428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f570815ac60 msgr2=0x7f57080b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:53.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f56fea00640 1 --2- 172.21.15.40:0/1779127428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f570815ac60 0x7f57080b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f56fea00640 1 -- 172.21.15.40:0/1779127428 >> v1:172.21.15.53:6789/0 conn(0x7f57080b4840 legacy=0x7f57080ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:53.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.209+0000 7f56fea00640 1 -- 172.21.15.40:0/1779127428 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57080b3820 con 0x7f570806cb80 2024-03-24T00:37:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.213+0000 7f56fea00640 1 --2- 172.21.15.40:0/1779127428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f570806cb80 0x7f57080b4300 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f56f0009870 tx=0x7f56f001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcae34feef80556 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.213+0000 7f5706a00640 1 -- 172.21.15.40:0/1779127428 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00b0070 con 0x7f570806cb80 2024-03-24T00:37:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.213+0000 7f5706a00640 1 -- 172.21.15.40:0/1779127428 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f00ab600 con 0x7f570806cb80 2024-03-24T00:37:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.213+0000 7f5706a00640 1 -- 172.21.15.40:0/1779127428 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00ab9d0 con 0x7f570806cb80 2024-03-24T00:37:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.213+0000 7f570e600640 1 -- 172.21.15.40:0/1779127428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f570806cb80 msgr2=0x7f57080b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.213+0000 7f570e600640 1 --2- 172.21.15.40:0/1779127428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f570806cb80 0x7f57080b4300 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f56f0009870 tx=0x7f56f001c460 comp rx=0 tx=0).stop 2024-03-24T00:37:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.213+0000 7f570e600640 1 -- 172.21.15.40:0/1779127428 shutdown_connections 2024-03-24T00:37:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.213+0000 7f570e600640 1 --2- 172.21.15.40:0/1779127428 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f570806cb80 0x7f57080b4300 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.213+0000 7f570e600640 1 --2- 172.21.15.40:0/1779127428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f570815ac60 0x7f57080b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.213+0000 7f570e600640 1 -- 172.21.15.40:0/1779127428 >> 172.21.15.40:0/1779127428 conn(0x7f57080b1660 msgr2=0x7f57080b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.213+0000 7f570e600640 1 -- 172.21.15.40:0/1779127428 shutdown_connections 2024-03-24T00:37:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.213+0000 7f570e600640 1 -- 172.21.15.40:0/1779127428 wait complete. 2024-03-24T00:37:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f570e600640 1 Processor -- start 2024-03-24T00:37:53.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f570e600640 1 -- start start 2024-03-24T00:37:53.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f570e600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f570806cb80 0x7f570814f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f570e600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f57080b4840 0x7f5708150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f570e600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f570815ac60 0x7f5708154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f570e600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5708167a40 con 0x7f570806cb80 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f570e600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57081678c0 con 0x7f57080b4840 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f570e600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5708167bc0 con 0x7f570815ac60 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f5707400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f570806cb80 0x7f570814f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f5707400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f570806cb80 0x7f570814f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42334/0 (socket says 172.21.15.40:42334) 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f5707400640 1 -- 172.21.15.40:0/2922804018 learned_addr learned my addr 172.21.15.40:0/2922804018 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f56fea00640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f57080b4840 0x7f5708150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f5707e00640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f570815ac60 0x7f5708154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f5707400640 1 -- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f570815ac60 msgr2=0x7f5708154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f5707400640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f570815ac60 0x7f5708154e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f5707400640 1 -- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f57080b4840 msgr2=0x7f5708150cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f5707400640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f57080b4840 0x7f5708150cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f5707400640 1 -- 172.21.15.40:0/2922804018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570814fc30 con 0x7f570806cb80 2024-03-24T00:37:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f5707400640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f570806cb80 0x7f570814f650 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f56f800b410 tx=0x7f56f800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f56ffe00640 1 -- 172.21.15.40:0/2922804018 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8015020 con 0x7f570806cb80 2024-03-24T00:37:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f56ffe00640 1 -- 172.21.15.40:0/2922804018 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f8011830 con 0x7f570806cb80 2024-03-24T00:37:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f56ffe00640 1 -- 172.21.15.40:0/2922804018 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8011c20 con 0x7f570806cb80 2024-03-24T00:37:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f570e600640 1 -- 172.21.15.40:0/2922804018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570814ff10 con 0x7f570806cb80 2024-03-24T00:37:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.217+0000 7f570e600640 1 -- 172.21.15.40:0/2922804018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57081504d0 con 0x7f570806cb80 2024-03-24T00:37:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.221+0000 7f570e600640 1 -- 172.21.15.40:0/2922804018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5708068170 con 0x7f570806cb80 2024-03-24T00:37:53.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.221+0000 7f56ffe00640 1 -- 172.21.15.40:0/2922804018 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f56f8018020 con 0x7f570806cb80 2024-03-24T00:37:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.221+0000 7f56ffe00640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56e4067660 0x7f56e4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.221+0000 7f56ffe00640 1 -- 172.21.15.40:0/2922804018 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f56f80855f0 con 0x7f570806cb80 2024-03-24T00:37:53.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.221+0000 7f56fea00640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56e4067660 0x7f56e4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.221+0000 7f56fea00640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56e4067660 0x7f56e4069b20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f56f00ad000 tx=0x7f56f00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:53.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.225+0000 7f56ffe00640 1 -- 172.21.15.40:0/2922804018 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f56f801f020 con 0x7f570806cb80 2024-03-24T00:37:53.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.393+0000 7f570e600640 1 -- 172.21.15.40:0/2922804018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5708155df0 con 0x7f570806cb80 2024-03-24T00:37:53.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f56ffe00640 1 -- 172.21.15.40:0/2922804018 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7f56f80564c0 con 0x7f570806cb80 2024-03-24T00:37:53.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f570e600640 1 -- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56e4067660 msgr2=0x7f56e4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f570e600640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56e4067660 0x7f56e4069b20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f56f00ad000 tx=0x7f56f00a6000 comp rx=0 tx=0).stop 2024-03-24T00:37:53.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f570e600640 1 -- 172.21.15.40:0/2922804018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f570806cb80 msgr2=0x7f570814f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f570e600640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f570806cb80 0x7f570814f650 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f56f800b410 tx=0x7f56f800b8d0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f570e600640 1 -- 172.21.15.40:0/2922804018 shutdown_connections 2024-03-24T00:37:53.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f570e600640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56e4067660 0x7f56e4069b20 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f570e600640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f570815ac60 0x7f5708154e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f570e600640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f57080b4840 0x7f5708150cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f570e600640 1 --2- 172.21.15.40:0/2922804018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f570806cb80 0x7f570814f650 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f570e600640 1 -- 172.21.15.40:0/2922804018 >> 172.21.15.40:0/2922804018 conn(0x7f57080b1660 msgr2=0x7f570806d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:53.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f570e600640 1 -- 172.21.15.40:0/2922804018 shutdown_connections 2024-03-24T00:37:53.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.397+0000 7f570e600640 1 -- 172.21.15.40:0/2922804018 wait complete. 2024-03-24T00:37:53.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_before' 2024-03-24T00:37:53.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.557+0000 7fd383c00640 1 Processor -- start 2024-03-24T00:37:53.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.557+0000 7fd383c00640 1 -- start start 2024-03-24T00:37:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd383c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37c1587d0 0x7fd37c071560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd383c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37c0aef30 con 0x7fd37c071aa0 2024-03-24T00:37:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd383c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd37c0af210 con 0x7fd37c0ae9d0 2024-03-24T00:37:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd383c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c0af4f0 con 0x7fd37c1587d0 2024-03-24T00:37:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd37c071aa0 0x7fd37c0ae2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42862/0 (socket says 172.21.15.40:42862) 2024-03-24T00:37:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37be00640 1 -- 172.21.15.40:0/1401305442 learned_addr learned my addr 172.21.15.40:0/1401305442 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd381200640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37c1587d0 0x7fd37c071560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394187099 0 0) 0x7fd37c0aef30 con 0x7fd37c071aa0 2024-03-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd364003530 con 0x7fd37c071aa0 2024-03-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2841501643 0 0) 0x7fd364003530 con 0x7fd37c071aa0 2024-03-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37c1587d0 msgr2=0x7fd37c071560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37c1587d0 0x7fd37c071560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 >> v1:172.21.15.53:6789/0 conn(0x7fd37c0ae9d0 legacy=0x7fd37c06d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37c0af7d0 con 0x7fd37c071aa0 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fd368001680 con 0x7fd37c071aa0 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 >> v1:172.21.15.40:6789/0 conn(0x7fd37c071aa0 legacy=0x7fd37c0ae2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3640039f0 0x7fd364003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37c1587d0 0x7fd364004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd364004d50 0x7fd3640051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c0af4f0 con 0x7fd3640039f0 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36400bb00 con 0x7fd364004d50 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36400bd00 con 0x7fd37c1587d0 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd381200640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3640039f0 0x7fd364003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd37be00640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37c1587d0 0x7fd364004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd381c00640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd364004d50 0x7fd3640051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd381200640 1 -- 172.21.15.40:0/1401305442 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37c1587d0 msgr2=0x7fd364004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd381200640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37c1587d0 0x7fd364004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd381200640 1 -- 172.21.15.40:0/1401305442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd364004d50 msgr2=0x7fd3640051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:53.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd381200640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd364004d50 0x7fd3640051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd381200640 1 -- 172.21.15.40:0/1401305442 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd364003530 con 0x7fd3640039f0 2024-03-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.561+0000 7fd381c00640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd364004d50 0x7fd3640051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd381200640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3640039f0 0x7fd364003e10 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fd370009ae0 tx=0x7fd37001c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e9f131dc0c54854 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3700096b0 con 0x7fd3640039f0 2024-03-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3700b0400 con 0x7fd3640039f0 2024-03-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd37b400640 1 -- 172.21.15.40:0/1401305442 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3700b07f0 con 0x7fd3640039f0 2024-03-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd383c00640 1 -- 172.21.15.40:0/1401305442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3640039f0 msgr2=0x7fd364003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd383c00640 1 --2- 172.21.15.40:0/1401305442 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd3640039f0 0x7fd364003e10 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fd370009ae0 tx=0x7fd37001c9b0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd381c00640 1 -- 172.21.15.40:0/1401305442 reap_dead start 2024-03-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd383c00640 1 -- 172.21.15.40:0/1401305442 shutdown_connections 2024-03-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd383c00640 1 -- 172.21.15.40:0/1401305442 >> 172.21.15.40:0/1401305442 conn(0x7fd37c0b21a0 msgr2=0x7fd37c0b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd383c00640 1 -- 172.21.15.40:0/1401305442 shutdown_connections 2024-03-24T00:37:53.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd383c00640 1 -- 172.21.15.40:0/1401305442 wait complete. 2024-03-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd383c00640 1 Processor -- start 2024-03-24T00:37:53.571 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.565+0000 7fd383c00640 1 -- start start 2024-03-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd383c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3640039f0 0x7fd37c14a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd383c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37c15b3c0 0x7fd37c159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd383c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd37c1533a0 0x7fd37c153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd383c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c071ea0 con 0x7fd37c15b3c0 2024-03-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd383c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c06e6f0 con 0x7fd37c1533a0 2024-03-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd383c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37c169c60 con 0x7fd3640039f0 2024-03-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd37be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37c15b3c0 0x7fd37c159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd37be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37c15b3c0 0x7fd37c159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42364/0 (socket says 172.21.15.40:42364) 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd37be00640 1 -- 172.21.15.40:0/371931847 learned_addr learned my addr 172.21.15.40:0/371931847 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd381200640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3640039f0 0x7fd37c14a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd381c00640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd37c1533a0 0x7fd37c153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd37be00640 1 -- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3640039f0 msgr2=0x7fd37c14a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd37be00640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3640039f0 0x7fd37c14a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd37be00640 1 -- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd37c1533a0 msgr2=0x7fd37c153830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd37be00640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd37c1533a0 0x7fd37c153830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd37be00640 1 -- 172.21.15.40:0/371931847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37c14a8f0 con 0x7fd37c15b3c0 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd37be00640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37c15b3c0 0x7fd37c159b20 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fd36800e820 tx=0x7fd36800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd378c00640 1 -- 172.21.15.40:0/371931847 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36800c8d0 con 0x7fd37c15b3c0 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd378c00640 1 -- 172.21.15.40:0/371931847 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd368018070 con 0x7fd37c15b3c0 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd378c00640 1 -- 172.21.15.40:0/371931847 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36800ccb0 con 0x7fd37c15b3c0 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd383c00640 1 -- 172.21.15.40:0/371931847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37c15a060 con 0x7fd37c15b3c0 2024-03-24T00:37:53.575 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd383c00640 1 -- 172.21.15.40:0/371931847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37c15a600 con 0x7fd37c15b3c0 2024-03-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.569+0000 7fd383c00640 1 -- 172.21.15.40:0/371931847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37c068170 con 0x7fd37c15b3c0 2024-03-24T00:37:53.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.573+0000 7fd378c00640 1 -- 172.21.15.40:0/371931847 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd36801d460 con 0x7fd37c15b3c0 2024-03-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.573+0000 7fd378c00640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd35c067660 0x7fd35c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.573+0000 7fd378c00640 1 -- 172.21.15.40:0/371931847 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fd3680854d0 con 0x7fd37c15b3c0 2024-03-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.573+0000 7fd381200640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd35c067660 0x7fd35c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:53.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.573+0000 7fd381200640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd35c067660 0x7fd35c069b20 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd3700af000 tx=0x7fd3700b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:53.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.577+0000 7fd378c00640 1 -- 172.21.15.40:0/371931847 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd368056300 con 0x7fd37c15b3c0 2024-03-24T00:37:53.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.797+0000 7fd383c00640 1 -- 172.21.15.40:0/371931847 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_before"]} v 0) v1 -- 0x7fd37c06d7b0 con 0x7fd37c15b3c0 2024-03-24T00:37:53.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.977+0000 7fd378c00640 1 -- 172.21.15.40:0/371931847 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_before"]}]=0 v347) v1 ==== 162+0+0 (secure 0 0 0) 0x7fd368055c00 con 0x7fd37c15b3c0 2024-03-24T00:37:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.981+0000 7fd383c00640 1 -- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd35c067660 msgr2=0x7fd35c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.981+0000 7fd383c00640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd35c067660 0x7fd35c069b20 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7fd3700af000 tx=0x7fd3700b3040 comp rx=0 tx=0).stop 2024-03-24T00:37:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.981+0000 7fd383c00640 1 -- 172.21.15.40:0/371931847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37c15b3c0 msgr2=0x7fd37c159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:53.985 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.981+0000 7fd383c00640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37c15b3c0 0x7fd37c159b20 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fd36800e820 tx=0x7fd36800ece0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.981+0000 7fd383c00640 1 -- 172.21.15.40:0/371931847 shutdown_connections 2024-03-24T00:37:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.981+0000 7fd383c00640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd35c067660 0x7fd35c069b20 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.981+0000 7fd383c00640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd37c1533a0 0x7fd37c153830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.981+0000 7fd383c00640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37c15b3c0 0x7fd37c159b20 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.981+0000 7fd383c00640 1 --2- 172.21.15.40:0/371931847 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3640039f0 0x7fd37c14a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.981+0000 7fd383c00640 1 -- 172.21.15.40:0/371931847 >> 172.21.15.40:0/371931847 conn(0x7fd37c0b21a0 msgr2=0x7fd37c0b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.981+0000 7fd383c00640 1 -- 172.21.15.40:0/371931847 shutdown_connections 2024-03-24T00:37:53.986 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:53.981+0000 7fd383c00640 1 -- 172.21.15.40:0/371931847 wait complete. 2024-03-24T00:37:54.006 INFO:tasks.cephfs_test_runner:That a conventional export pin overrides the distributed policy _before_ distributed policy is set. ... ok 2024-03-24T00:37:54.006 INFO:tasks.cephfs_test_runner:Starting test: test_ephemeral_pin_dist_override_pin (tasks.cephfs.test_exports.TestEphemeralPins) That an export pin overrides an ephemerally pinned directory. 2024-03-24T00:37:54.006 INFO:tasks.cephfs_test_runner:test_ephemeral_pin_dist_override_pin (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:37:54.007 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:54.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa20ec00640 1 Processor -- start 2024-03-24T00:37:54.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa20ec00640 1 -- start start 2024-03-24T00:37:54.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa20ec00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa20813ec80 0x7fa2081414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa20ec00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa208070e00 con 0x7fa2081419f0 2024-03-24T00:37:54.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa20ec00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2080cee20 con 0x7fa2080ce800 2024-03-24T00:37:54.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa20ec00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa208141e50 con 0x7fa20813ec80 2024-03-24T00:37:54.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa1ff400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa2081419f0 0x7fa2080ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42874/0 (socket says 172.21.15.40:42874) 2024-03-24T00:37:54.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa1ff400640 1 -- 172.21.15.40:0/636170726 learned_addr learned my addr 172.21.15.40:0/636170726 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:54.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207e00640 1 --2- 172.21.15.40:0/636170726 >> v2:172.21.15.53:3300/0 conn(0x7fa20813ec80 0x7fa2081414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266934910 0 0) 0x7fa208070e00 con 0x7fa2081419f0 2024-03-24T00:37:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1ec003530 con 0x7fa2081419f0 2024-03-24T00:37:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310257393 0 0) 0x7fa2080cee20 con 0x7fa2080ce800 2024-03-24T00:37:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa208070e00 con 0x7fa2080ce800 2024-03-24T00:37:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4074998460 0 0) 0x7fa1ec003530 con 0x7fa2081419f0 2024-03-24T00:37:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 >> v2:172.21.15.53:3300/0 conn(0x7fa20813ec80 msgr2=0x7fa2081414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 --2- 172.21.15.40:0/636170726 >> v2:172.21.15.53:3300/0 conn(0x7fa20813ec80 0x7fa2081414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 >> v1:172.21.15.53:6789/0 conn(0x7fa2080ce800 legacy=0x7fa208071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2080cf100 con 0x7fa2081419f0 2024-03-24T00:37:54.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fa1f0001680 con 0x7fa2081419f0 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 >> v1:172.21.15.40:6789/0 conn(0x7fa2081419f0 legacy=0x7fa2080ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 --2- 172.21.15.40:0/636170726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec003ba0 0x7fa1ec003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 --2- 172.21.15.40:0/636170726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1ec004810 0x7fa1ec004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 --2- 172.21.15.40:0/636170726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1ec0051d0 0x7fa1ec00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec00c0a0 con 0x7fa1ec004810 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec00c220 con 0x7fa1ec003ba0 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ec00c420 con 0x7fa1ec0051d0 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa1ff400640 1 --2- 172.21.15.40:0/636170726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1ec004810 0x7fa1ec004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa207e00640 1 --2- 172.21.15.40:0/636170726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec003ba0 0x7fa1ec003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.145+0000 7fa20cc00640 1 --2- 172.21.15.40:0/636170726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1ec0051d0 0x7fa1ec00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa207e00640 1 -- 172.21.15.40:0/636170726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1ec0051d0 msgr2=0x7fa1ec00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa207e00640 1 --2- 172.21.15.40:0/636170726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1ec0051d0 0x7fa1ec00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa207e00640 1 -- 172.21.15.40:0/636170726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1ec004810 msgr2=0x7fa1ec004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:54.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa207e00640 1 --2- 172.21.15.40:0/636170726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1ec004810 0x7fa1ec004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa207e00640 1 -- 172.21.15.40:0/636170726 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ec003530 con 0x7fa1ec003ba0 2024-03-24T00:37:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa1ff400640 1 --2- 172.21.15.40:0/636170726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1ec004810 0x7fa1ec004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:37:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa20cc00640 1 -- 172.21.15.40:0/636170726 reap_dead start 2024-03-24T00:37:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa207e00640 1 --2- 172.21.15.40:0/636170726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec003ba0 0x7fa1ec003fa0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009ae0 tx=0x7fa1f801ca70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f25542b9adf479 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80b2070 con 0x7fa1ec003ba0 2024-03-24T00:37:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f80a5550 con 0x7fa1ec003ba0 2024-03-24T00:37:54.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa207400640 1 -- 172.21.15.40:0/636170726 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f80096b0 con 0x7fa1ec003ba0 2024-03-24T00:37:54.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa20ec00640 1 -- 172.21.15.40:0/636170726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec003ba0 msgr2=0x7fa1ec003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa20ec00640 1 --2- 172.21.15.40:0/636170726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec003ba0 0x7fa1ec003fa0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009ae0 tx=0x7fa1f801ca70 comp rx=0 tx=0).stop 2024-03-24T00:37:54.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa20ec00640 1 -- 172.21.15.40:0/636170726 shutdown_connections 2024-03-24T00:37:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa20ec00640 1 --2- 172.21.15.40:0/636170726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec003ba0 0x7fa1ec003fa0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.149+0000 7fa20ec00640 1 -- 172.21.15.40:0/636170726 >> 172.21.15.40:0/636170726 conn(0x7fa2080b17b0 msgr2=0x7fa2080b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa20ec00640 1 -- 172.21.15.40:0/636170726 shutdown_connections 2024-03-24T00:37:54.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa20ec00640 1 -- 172.21.15.40:0/636170726 wait complete. 2024-03-24T00:37:54.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa20ec00640 1 Processor -- start 2024-03-24T00:37:54.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa20ec00640 1 -- start start 2024-03-24T00:37:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa20ec00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec003ba0 0x7fa2080b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa20ec00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa20815e470 0x7fa2081f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa20ec00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa20815d3c0 0x7fa20815d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa20ec00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2081414b0 con 0x7fa20815e470 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa20ec00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa20813f060 con 0x7fa1ec003ba0 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa20ec00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa208141630 con 0x7fa20815d3c0 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa1ff400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa20815e470 0x7fa2081f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa1ff400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa20815e470 0x7fa2081f3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42372/0 (socket says 172.21.15.40:42372) 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa1ff400640 1 -- 172.21.15.40:0/3950113487 learned_addr learned my addr 172.21.15.40:0/3950113487 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa207e00640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec003ba0 0x7fa2080b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa20cc00640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa20815d3c0 0x7fa20815d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa1ff400640 1 -- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa20815d3c0 msgr2=0x7fa20815d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa1ff400640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa20815d3c0 0x7fa20815d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa1ff400640 1 -- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec003ba0 msgr2=0x7fa2080b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa1ff400640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec003ba0 0x7fa2080b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa1ff400640 1 -- 172.21.15.40:0/3950113487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2081f4080 con 0x7fa20815e470 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa1ff400640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa20815e470 0x7fa2081f3b40 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00017f0 tx=0x7fa1f0007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.153+0000 7fa204c00640 1 -- 172.21.15.40:0/3950113487 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0007bf0 con 0x7fa20815e470 2024-03-24T00:37:54.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.157+0000 7fa204c00640 1 -- 172.21.15.40:0/3950113487 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1f0018070 con 0x7fa20815e470 2024-03-24T00:37:54.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.157+0000 7fa204c00640 1 -- 172.21.15.40:0/3950113487 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f000fc00 con 0x7fa20815e470 2024-03-24T00:37:54.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.157+0000 7fa20ec00640 1 -- 172.21.15.40:0/3950113487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2081f4340 con 0x7fa20815e470 2024-03-24T00:37:54.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.157+0000 7fa20ec00640 1 -- 172.21.15.40:0/3950113487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2081f4900 con 0x7fa20815e470 2024-03-24T00:37:54.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.157+0000 7fa20ec00640 1 -- 172.21.15.40:0/3950113487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa208068170 con 0x7fa20815e470 2024-03-24T00:37:54.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.157+0000 7fa204c00640 1 -- 172.21.15.40:0/3950113487 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa1f001c410 con 0x7fa20815e470 2024-03-24T00:37:54.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.157+0000 7fa204c00640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1e4067690 0x7fa1e4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.157+0000 7fa204c00640 1 -- 172.21.15.40:0/3950113487 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fa1f0085210 con 0x7fa20815e470 2024-03-24T00:37:54.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.157+0000 7fa207e00640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1e4067690 0x7fa1e4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.157+0000 7fa207e00640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1e4067690 0x7fa1e4069b50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80af000 tx=0x7fa1f80046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.161+0000 7fa204c00640 1 -- 172.21.15.40:0/3950113487 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa1f0056040 con 0x7fa20815e470 2024-03-24T00:37:54.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.329+0000 7fa20ec00640 1 -- 172.21.15.40:0/3950113487 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa208072b90 con 0x7fa20815e470 2024-03-24T00:37:54.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.329+0000 7fa204c00640 1 -- 172.21.15.40:0/3950113487 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+13221 (secure 0 0 0) 0x7fa1f0055940 con 0x7fa20815e470 2024-03-24T00:37:54.336 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:54.337 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":98,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:37:51.973591+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.40:6835/1997904254":"2024-03-25T00:37:48.961741+0000","172.21.15.53:6837/3095501397":"2024-03-25T00:37:48.961757+0000","172.21.15.53:6835/3095501397":"2024-03-25T00:37:48.961757+0000","172.21.15.40:6834/1019660586":"2024-03-25T00:37:48.961752+0000","172.21.15.40:6833/1997904254":"2024-03-25T00:37:48.961741+0000","172.21.15.40:6832/1019660586":"2024-03-25T00:37:48.961752+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:54.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.333+0000 7fa20ec00640 1 -- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1e4067690 msgr2=0x7fa1e4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.333+0000 7fa20ec00640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1e4067690 0x7fa1e4069b50 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa1f80af000 tx=0x7fa1f80046d0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.333+0000 7fa20ec00640 1 -- 172.21.15.40:0/3950113487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa20815e470 msgr2=0x7fa2081f3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.333+0000 7fa20ec00640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa20815e470 0x7fa2081f3b40 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00017f0 tx=0x7fa1f0007720 comp rx=0 tx=0).stop 2024-03-24T00:37:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.333+0000 7fa20ec00640 1 -- 172.21.15.40:0/3950113487 shutdown_connections 2024-03-24T00:37:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.333+0000 7fa20ec00640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1e4067690 0x7fa1e4069b50 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.333+0000 7fa20ec00640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa20815d3c0 0x7fa20815d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.333+0000 7fa20ec00640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa20815e470 0x7fa2081f3b40 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.333+0000 7fa20ec00640 1 --2- 172.21.15.40:0/3950113487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1ec003ba0 0x7fa2080b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.333+0000 7fa20ec00640 1 -- 172.21.15.40:0/3950113487 >> 172.21.15.40:0/3950113487 conn(0x7fa2080b17b0 msgr2=0x7fa208140850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.333+0000 7fa20ec00640 1 -- 172.21.15.40:0/3950113487 shutdown_connections 2024-03-24T00:37:54.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.333+0000 7fa20ec00640 1 -- 172.21.15.40:0/3950113487 wait complete. 2024-03-24T00:37:54.356 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:37:54.357 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:37:54.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21dde00640 1 Processor -- start 2024-03-24T00:37:54.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21dde00640 1 -- start start 2024-03-24T00:37:54.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21dde00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21d806cb80 0x7f21d80b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21dde00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f21d80b4840 0x7f21d8071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21dde00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d80b3260 con 0x7f21d815ac60 2024-03-24T00:37:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21dde00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d80b3e70 con 0x7f21d80b4840 2024-03-24T00:37:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21dde00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80b3540 con 0x7f21d806cb80 2024-03-24T00:37:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21d6a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f21d815ac60 0x7f21d80b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:60558/0 (socket says 172.21.15.40:60558) 2024-03-24T00:37:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21d6a00640 1 -- 172.21.15.40:0/2300868930 learned_addr learned my addr 172.21.15.40:0/2300868930 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21ce000640 1 --2- 172.21.15.40:0/2300868930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21d806cb80 0x7f21d80b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21d7400640 1 --2- 172.21.15.40:0/2300868930 >> v2:172.21.15.53:3300/0 conn(0x7f21d80b4840 0x7f21d8071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21d6000640 1 -- 172.21.15.40:0/2300868930 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501659823 0 0) 0x7f21d80b3260 con 0x7f21d815ac60 2024-03-24T00:37:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21ce000640 1 -- 172.21.15.40:0/2300868930 >> v2:172.21.15.53:3300/0 conn(0x7f21d80b4840 msgr2=0x7f21d8071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21ce000640 1 --2- 172.21.15.40:0/2300868930 >> v2:172.21.15.53:3300/0 conn(0x7f21d80b4840 0x7f21d8071b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21ce000640 1 -- 172.21.15.40:0/2300868930 >> v1:172.21.15.53:6789/0 conn(0x7f21d815ac60 legacy=0x7f21d80b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.503 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.497+0000 7f21ce000640 1 -- 172.21.15.40:0/2300868930 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d80b3820 con 0x7f21d806cb80 2024-03-24T00:37:54.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.501+0000 7f21ce000640 1 --2- 172.21.15.40:0/2300868930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21d806cb80 0x7f21d80b4300 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f21c00097c0 tx=0x7f21c001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10873fb07cb00d20 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.501+0000 7f21d6000640 1 -- 172.21.15.40:0/2300868930 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c00b0070 con 0x7f21d806cb80 2024-03-24T00:37:54.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.501+0000 7f21d6000640 1 -- 172.21.15.40:0/2300868930 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c00ab600 con 0x7f21d806cb80 2024-03-24T00:37:54.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.501+0000 7f21d6000640 1 -- 172.21.15.40:0/2300868930 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c00ab9d0 con 0x7f21d806cb80 2024-03-24T00:37:54.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.501+0000 7f21dde00640 1 -- 172.21.15.40:0/2300868930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21d806cb80 msgr2=0x7f21d80b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.501+0000 7f21dde00640 1 --2- 172.21.15.40:0/2300868930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21d806cb80 0x7f21d80b4300 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f21c00097c0 tx=0x7f21c001c410 comp rx=0 tx=0).stop 2024-03-24T00:37:54.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.501+0000 7f21dde00640 1 -- 172.21.15.40:0/2300868930 shutdown_connections 2024-03-24T00:37:54.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.501+0000 7f21dde00640 1 --2- 172.21.15.40:0/2300868930 >> v2:172.21.15.53:3300/0 conn(0x7f21d80b4840 0x7f21d8071b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.501+0000 7f21dde00640 1 --2- 172.21.15.40:0/2300868930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21d806cb80 0x7f21d80b4300 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.501+0000 7f21dde00640 1 -- 172.21.15.40:0/2300868930 >> 172.21.15.40:0/2300868930 conn(0x7f21d80b1660 msgr2=0x7f21d80b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:54.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.501+0000 7f21dde00640 1 -- 172.21.15.40:0/2300868930 shutdown_connections 2024-03-24T00:37:54.507 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.501+0000 7f21dde00640 1 -- 172.21.15.40:0/2300868930 wait complete. 2024-03-24T00:37:54.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21dde00640 1 Processor -- start 2024-03-24T00:37:54.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21dde00640 1 -- start start 2024-03-24T00:37:54.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21dde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d806cb80 0x7f21d814a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21dde00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21d80b4840 0x7f21d80adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21dde00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d815ac60 0x7f21d8167580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21dde00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d8165c40 con 0x7f21d815ac60 2024-03-24T00:37:54.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21dde00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d8165ac0 con 0x7f21d806cb80 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21dde00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d8165dc0 con 0x7f21d80b4840 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21d7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d815ac60 0x7f21d8167580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21d7400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d815ac60 0x7f21d8167580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42384/0 (socket says 172.21.15.40:42384) 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21d7400640 1 -- 172.21.15.40:0/4062940500 learned_addr learned my addr 172.21.15.40:0/4062940500 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21ce000640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21d80b4840 0x7f21d80adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21d6a00640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d806cb80 0x7f21d814a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21d7400640 1 -- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21d80b4840 msgr2=0x7f21d80adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21d7400640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21d80b4840 0x7f21d80adb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21d7400640 1 -- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d806cb80 msgr2=0x7f21d814a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21d7400640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d806cb80 0x7f21d814a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21d7400640 1 -- 172.21.15.40:0/4062940500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d814c8f0 con 0x7f21d815ac60 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21ce000640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21d80b4840 0x7f21d80adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21d7400640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d815ac60 0x7f21d8167580 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f21d000efc0 tx=0x7f21d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21cfe00640 1 -- 172.21.15.40:0/4062940500 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0018020 con 0x7f21d815ac60 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21cfe00640 1 -- 172.21.15.40:0/4062940500 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d0014480 con 0x7f21d815ac60 2024-03-24T00:37:54.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21cfe00640 1 -- 172.21.15.40:0/4062940500 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0014870 con 0x7f21d815ac60 2024-03-24T00:37:54.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21dde00640 1 -- 172.21.15.40:0/4062940500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d814cbd0 con 0x7f21d815ac60 2024-03-24T00:37:54.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.505+0000 7f21dde00640 1 -- 172.21.15.40:0/4062940500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d814d190 con 0x7f21d815ac60 2024-03-24T00:37:54.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.509+0000 7f21dde00640 1 -- 172.21.15.40:0/4062940500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21d8068170 con 0x7f21d815ac60 2024-03-24T00:37:54.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.509+0000 7f21cfe00640 1 -- 172.21.15.40:0/4062940500 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f21d001b020 con 0x7f21d815ac60 2024-03-24T00:37:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.509+0000 7f21cfe00640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f21b4067690 0x7f21b4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.509+0000 7f21cfe00640 1 -- 172.21.15.40:0/4062940500 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f21d00862a0 con 0x7f21d815ac60 2024-03-24T00:37:54.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.509+0000 7f21d6a00640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f21b4067690 0x7f21b4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.509+0000 7f21d6a00640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f21b4067690 0x7f21b4069b50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f21c8000f70 tx=0x7f21c8007d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.513+0000 7f21cfe00640 1 -- 172.21.15.40:0/4062940500 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f21d0022030 con 0x7f21d815ac60 2024-03-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.677+0000 7f21dde00640 1 -- 172.21.15.40:0/4062940500 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f21d814d500 con 0x7f21d815ac60 2024-03-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.677+0000 7f21cfe00640 1 -- 172.21.15.40:0/4062940500 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f21d0010040 con 0x7f21d815ac60 2024-03-24T00:37:54.682 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.681+0000 7f21dde00640 1 -- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f21b4067690 msgr2=0x7f21b4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.681+0000 7f21dde00640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f21b4067690 0x7f21b4069b50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f21c8000f70 tx=0x7f21c8007d10 comp rx=0 tx=0).stop 2024-03-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.681+0000 7f21dde00640 1 -- 172.21.15.40:0/4062940500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d815ac60 msgr2=0x7f21d8167580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.681+0000 7f21dde00640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d815ac60 0x7f21d8167580 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f21d000efc0 tx=0x7f21d000c490 comp rx=0 tx=0).stop 2024-03-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.681+0000 7f21dde00640 1 -- 172.21.15.40:0/4062940500 shutdown_connections 2024-03-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.681+0000 7f21dde00640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f21b4067690 0x7f21b4069b50 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.681+0000 7f21dde00640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f21d815ac60 0x7f21d8167580 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.681+0000 7f21dde00640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f21d80b4840 0x7f21d80adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.681+0000 7f21dde00640 1 --2- 172.21.15.40:0/4062940500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21d806cb80 0x7f21d814a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.681+0000 7f21dde00640 1 -- 172.21.15.40:0/4062940500 >> 172.21.15.40:0/4062940500 conn(0x7f21d80b1660 msgr2=0x7f21d80af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.681+0000 7f21dde00640 1 -- 172.21.15.40:0/4062940500 shutdown_connections 2024-03-24T00:37:54.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.681+0000 7f21dde00640 1 -- 172.21.15.40:0/4062940500 wait complete. 2024-03-24T00:37:54.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_pin' 2024-03-24T00:37:54.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc252600640 1 Processor -- start 2024-03-24T00:37:54.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc252600640 1 -- start start 2024-03-24T00:37:54.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc252600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc24c06e620 0x7fc24c0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc252600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc24c0ccfa0 0x7fc24c0cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc252600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc24c070f60 con 0x7fc24c06ea00 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc252600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc24c0b4c00 con 0x7fc24c0ccfa0 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc252600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c0b3620 con 0x7fc24c06e620 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc24be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc24c0ccfa0 0x7fc24c0cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc24be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc24c0ccfa0 0x7fc24c0cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42386/0 (socket says 172.21.15.40:42386) 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc24be00640 1 -- 172.21.15.40:0/2754495559 learned_addr learned my addr 172.21.15.40:0/2754495559 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc24b400640 1 --2- 172.21.15.40:0/2754495559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc24c06e620 0x7fc24c0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc24be00640 1 -- 172.21.15.40:0/2754495559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc24c06e620 msgr2=0x7fc24c0b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc24be00640 1 --2- 172.21.15.40:0/2754495559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc24c06e620 0x7fc24c0b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc24be00640 1 -- 172.21.15.40:0/2754495559 >> v1:172.21.15.53:6789/0 conn(0x7fc24c06ea00 legacy=0x7fc24c0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc24be00640 1 -- 172.21.15.40:0/2754495559 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c0b3900 con 0x7fc24c0ccfa0 2024-03-24T00:37:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.841+0000 7fc24b400640 1 --2- 172.21.15.40:0/2754495559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc24c06e620 0x7fc24c0b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:37:54.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.845+0000 7fc24be00640 1 --2- 172.21.15.40:0/2754495559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc24c0ccfa0 0x7fc24c0cd3f0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fc24400d150 tx=0x7fc24401eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fd667d34b44975 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.845+0000 7fc24aa00640 1 -- 172.21.15.40:0/2754495559 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24400b2d0 con 0x7fc24c0ccfa0 2024-03-24T00:37:54.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.845+0000 7fc24aa00640 1 -- 172.21.15.40:0/2754495559 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24400b470 con 0x7fc24c0ccfa0 2024-03-24T00:37:54.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.845+0000 7fc24aa00640 1 -- 172.21.15.40:0/2754495559 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2440a79f0 con 0x7fc24c0ccfa0 2024-03-24T00:37:54.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.845+0000 7fc252600640 1 -- 172.21.15.40:0/2754495559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc24c0ccfa0 msgr2=0x7fc24c0cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.845+0000 7fc252600640 1 --2- 172.21.15.40:0/2754495559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc24c0ccfa0 0x7fc24c0cd3f0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fc24400d150 tx=0x7fc24401eb30 comp rx=0 tx=0).stop 2024-03-24T00:37:54.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.845+0000 7fc252600640 1 -- 172.21.15.40:0/2754495559 shutdown_connections 2024-03-24T00:37:54.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.845+0000 7fc252600640 1 --2- 172.21.15.40:0/2754495559 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc24c0ccfa0 0x7fc24c0cd3f0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.845+0000 7fc252600640 1 --2- 172.21.15.40:0/2754495559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc24c06e620 0x7fc24c0b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.845+0000 7fc252600640 1 -- 172.21.15.40:0/2754495559 >> 172.21.15.40:0/2754495559 conn(0x7fc24c0b17b0 msgr2=0x7fc24c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:54.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.845+0000 7fc252600640 1 -- 172.21.15.40:0/2754495559 shutdown_connections 2024-03-24T00:37:54.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.845+0000 7fc252600640 1 -- 172.21.15.40:0/2754495559 wait complete. 2024-03-24T00:37:54.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc252600640 1 Processor -- start 2024-03-24T00:37:54.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc252600640 1 -- start start 2024-03-24T00:37:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc252600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc24c06e620 0x7fc24c0af0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc252600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc24c06ea00 0x7fc24c0ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc252600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc24c0ccfa0 0x7fc24c0ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc252600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c144920 con 0x7fc24c06e620 2024-03-24T00:37:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc252600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c06ee60 con 0x7fc24c0ccfa0 2024-03-24T00:37:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc252600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c13e200 con 0x7fc24c06ea00 2024-03-24T00:37:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc24b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc24c06e620 0x7fc24c0af0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc24b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc24c06e620 0x7fc24c0af0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42392/0 (socket says 172.21.15.40:42392) 2024-03-24T00:37:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc24b400640 1 -- 172.21.15.40:0/2311051452 learned_addr learned my addr 172.21.15.40:0/2311051452 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc24be00640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc24c0ccfa0 0x7fc24c0ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc242a00640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc24c06ea00 0x7fc24c0ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc24b400640 1 -- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc24c06ea00 msgr2=0x7fc24c0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc24b400640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc24c06ea00 0x7fc24c0ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc24b400640 1 -- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc24c0ccfa0 msgr2=0x7fc24c0ce650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc24b400640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc24c0ccfa0 0x7fc24c0ce650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc24b400640 1 -- 172.21.15.40:0/2311051452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c0ae030 con 0x7fc24c06e620 2024-03-24T00:37:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc24b400640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc24c06e620 0x7fc24c0af0e0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00c7c0 tx=0x7fc23c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc243e00640 1 -- 172.21.15.40:0/2311051452 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23c0079d0 con 0x7fc24c06e620 2024-03-24T00:37:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc243e00640 1 -- 172.21.15.40:0/2311051452 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23c007b70 con 0x7fc24c06e620 2024-03-24T00:37:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc243e00640 1 -- 172.21.15.40:0/2311051452 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23c0113f0 con 0x7fc24c06e620 2024-03-24T00:37:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc252600640 1 -- 172.21.15.40:0/2311051452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc24c0ae310 con 0x7fc24c06e620 2024-03-24T00:37:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.849+0000 7fc252600640 1 -- 172.21.15.40:0/2311051452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc24c13e480 con 0x7fc24c06e620 2024-03-24T00:37:54.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.853+0000 7fc252600640 1 -- 172.21.15.40:0/2311051452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc24c068170 con 0x7fc24c06e620 2024-03-24T00:37:54.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.853+0000 7fc243e00640 1 -- 172.21.15.40:0/2311051452 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc23c011590 con 0x7fc24c06e620 2024-03-24T00:37:54.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.853+0000 7fc243e00640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc228067660 0x7fc228069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:54.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.853+0000 7fc243e00640 1 -- 172.21.15.40:0/2311051452 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fc23c085840 con 0x7fc24c06e620 2024-03-24T00:37:54.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.853+0000 7fc242a00640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc228067660 0x7fc228069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:54.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.853+0000 7fc242a00640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc228067660 0x7fc228069b20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0af620 tx=0x7fc234009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:54.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:54.857+0000 7fc243e00640 1 -- 172.21.15.40:0/2311051452 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc23c015020 con 0x7fc24c06e620 2024-03-24T00:37:55.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.077+0000 7fc252600640 1 -- 172.21.15.40:0/2311051452 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_pin"]} v 0) v1 -- 0x7fc24c072f50 con 0x7fc24c06e620 2024-03-24T00:37:55.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.985+0000 7fc243e00640 1 -- 172.21.15.40:0/2311051452 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_pin"]}]=0 v349) v1 ==== 162+0+0 (secure 0 0 0) 0x7fc23c056020 con 0x7fc24c06e620 2024-03-24T00:37:55.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.989+0000 7fc252600640 1 -- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc228067660 msgr2=0x7fc228069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.989+0000 7fc252600640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc228067660 0x7fc228069b20 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fc24c0af620 tx=0x7fc234009040 comp rx=0 tx=0).stop 2024-03-24T00:37:55.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.989+0000 7fc252600640 1 -- 172.21.15.40:0/2311051452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc24c06e620 msgr2=0x7fc24c0af0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:55.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.989+0000 7fc252600640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc24c06e620 0x7fc24c0af0e0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fc23c00c7c0 tx=0x7fc23c00cc80 comp rx=0 tx=0).stop 2024-03-24T00:37:55.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.989+0000 7fc252600640 1 -- 172.21.15.40:0/2311051452 shutdown_connections 2024-03-24T00:37:55.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.989+0000 7fc252600640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc228067660 0x7fc228069b20 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.989+0000 7fc252600640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc24c0ccfa0 0x7fc24c0ce650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.989+0000 7fc252600640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc24c06ea00 0x7fc24c0ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.989+0000 7fc252600640 1 --2- 172.21.15.40:0/2311051452 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc24c06e620 0x7fc24c0af0e0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:55.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.989+0000 7fc252600640 1 -- 172.21.15.40:0/2311051452 >> 172.21.15.40:0/2311051452 conn(0x7fc24c0b17b0 msgr2=0x7fc24c0cbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:55.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.989+0000 7fc252600640 1 -- 172.21.15.40:0/2311051452 shutdown_connections 2024-03-24T00:37:55.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:55.989+0000 7fc252600640 1 -- 172.21.15.40:0/2311051452 wait complete. 2024-03-24T00:37:56.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:37:56.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.161+0000 7fc627e00640 1 Processor -- start 2024-03-24T00:37:56.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.161+0000 7fc627e00640 1 -- start start 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.161+0000 7fc627e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc62015bd00 0x7fc6200ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.161+0000 7fc627e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc6200728d0 0x7fc6200cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.161+0000 7fc627e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc62015f8c0 0x7fc62015fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.161+0000 7fc627e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6200cf790 con 0x7fc6200728d0 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.161+0000 7fc627e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc620160250 con 0x7fc62015f8c0 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.161+0000 7fc627e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc620072d30 con 0x7fc62015bd00 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc624a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc6200728d0 0x7fc6200cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc624a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc6200728d0 0x7fc6200cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42396/0 (socket says 172.21.15.40:42396) 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc624a00640 1 -- 172.21.15.40:0/4286297199 learned_addr learned my addr 172.21.15.40:0/4286297199 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc625400640 1 --2- 172.21.15.40:0/4286297199 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc62015bd00 0x7fc6200ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc625e00640 1 --2- 172.21.15.40:0/4286297199 >> v2:172.21.15.53:3300/0 conn(0x7fc62015f8c0 0x7fc62015fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc624a00640 1 -- 172.21.15.40:0/4286297199 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc62015bd00 msgr2=0x7fc6200ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:56.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc624a00640 1 --2- 172.21.15.40:0/4286297199 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc62015bd00 0x7fc6200ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc624a00640 1 -- 172.21.15.40:0/4286297199 >> v2:172.21.15.53:3300/0 conn(0x7fc62015f8c0 msgr2=0x7fc62015fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:56.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc624a00640 1 --2- 172.21.15.40:0/4286297199 >> v2:172.21.15.53:3300/0 conn(0x7fc62015f8c0 0x7fc62015fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc624a00640 1 -- 172.21.15.40:0/4286297199 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc620071de0 con 0x7fc6200728d0 2024-03-24T00:37:56.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc624a00640 1 --2- 172.21.15.40:0/4286297199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6200728d0 0x7fc6200cf250 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fc620073320 tx=0x7fc60c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df25507699447f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:56.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc61fe00640 1 -- 172.21.15.40:0/4286297199 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0aa610 con 0x7fc6200728d0 2024-03-24T00:37:56.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc61fe00640 1 -- 172.21.15.40:0/4286297199 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc60c0af070 con 0x7fc6200728d0 2024-03-24T00:37:56.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.165+0000 7fc61fe00640 1 -- 172.21.15.40:0/4286297199 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc60c0aa9d0 con 0x7fc6200728d0 2024-03-24T00:37:56.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 -- 172.21.15.40:0/4286297199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6200728d0 msgr2=0x7fc6200cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 --2- 172.21.15.40:0/4286297199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6200728d0 0x7fc6200cf250 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fc620073320 tx=0x7fc60c01c4f0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 -- 172.21.15.40:0/4286297199 shutdown_connections 2024-03-24T00:37:56.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 --2- 172.21.15.40:0/4286297199 >> v2:172.21.15.53:3300/0 conn(0x7fc62015f8c0 0x7fc62015fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 --2- 172.21.15.40:0/4286297199 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6200728d0 0x7fc6200cf250 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 --2- 172.21.15.40:0/4286297199 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc62015bd00 0x7fc6200ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 -- 172.21.15.40:0/4286297199 >> 172.21.15.40:0/4286297199 conn(0x7fc6200b1660 msgr2=0x7fc6200b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:56.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 -- 172.21.15.40:0/4286297199 shutdown_connections 2024-03-24T00:37:56.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 -- 172.21.15.40:0/4286297199 wait complete. 2024-03-24T00:37:56.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 Processor -- start 2024-03-24T00:37:56.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 -- start start 2024-03-24T00:37:56.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6200728d0 0x7fc6201545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc62015bd00 0x7fc62014f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc62015f8c0 0x7fc620155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62006d320 con 0x7fc6200728d0 2024-03-24T00:37:56.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62015c0e0 con 0x7fc62015f8c0 2024-03-24T00:37:56.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc627e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc62006d4a0 con 0x7fc62015bd00 2024-03-24T00:37:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc625400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6200728d0 0x7fc6201545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc625400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6200728d0 0x7fc6201545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42406/0 (socket says 172.21.15.40:42406) 2024-03-24T00:37:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc625400640 1 -- 172.21.15.40:0/3881034438 learned_addr learned my addr 172.21.15.40:0/3881034438 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc624a00640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc62015bd00 0x7fc62014f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc625e00640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc62015f8c0 0x7fc620155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc625400640 1 -- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc62015bd00 msgr2=0x7fc62014f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc625400640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc62015bd00 0x7fc62014f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc625400640 1 -- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc62015f8c0 msgr2=0x7fc620155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc625400640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc62015f8c0 0x7fc620155f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.169+0000 7fc625400640 1 -- 172.21.15.40:0/3881034438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6200b3d20 con 0x7fc6200728d0 2024-03-24T00:37:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.173+0000 7fc625400640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6200728d0 0x7fc6201545e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc61400e930 tx=0x7fc61400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.173+0000 7fc61d600640 1 -- 172.21.15.40:0/3881034438 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc614017070 con 0x7fc6200728d0 2024-03-24T00:37:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.173+0000 7fc61d600640 1 -- 172.21.15.40:0/3881034438 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61400ca00 con 0x7fc6200728d0 2024-03-24T00:37:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.173+0000 7fc61d600640 1 -- 172.21.15.40:0/3881034438 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61400cdb0 con 0x7fc6200728d0 2024-03-24T00:37:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.173+0000 7fc627e00640 1 -- 172.21.15.40:0/3881034438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6200b4000 con 0x7fc6200728d0 2024-03-24T00:37:56.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.173+0000 7fc627e00640 1 -- 172.21.15.40:0/3881034438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6200b45c0 con 0x7fc6200728d0 2024-03-24T00:37:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.173+0000 7fc627e00640 1 -- 172.21.15.40:0/3881034438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc620068170 con 0x7fc6200728d0 2024-03-24T00:37:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.173+0000 7fc61d600640 1 -- 172.21.15.40:0/3881034438 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc61401a070 con 0x7fc6200728d0 2024-03-24T00:37:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.173+0000 7fc61d600640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6000676b0 0x7fc600069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.173+0000 7fc61d600640 1 -- 172.21.15.40:0/3881034438 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fc614085aa0 con 0x7fc6200728d0 2024-03-24T00:37:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.173+0000 7fc624a00640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6000676b0 0x7fc600069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.177+0000 7fc624a00640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6000676b0 0x7fc600069b70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc60c0ae000 tx=0x7fc60c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:56.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.177+0000 7fc61d600640 1 -- 172.21.15.40:0/3881034438 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc614056a10 con 0x7fc6200728d0 2024-03-24T00:37:56.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.341+0000 7fc627e00640 1 -- 172.21.15.40:0/3881034438 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc62006e870 con 0x7fc6200728d0 2024-03-24T00:37:56.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.345+0000 7fc61d600640 1 -- 172.21.15.40:0/3881034438 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v59)=0 v59) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc614056310 con 0x7fc6200728d0 2024-03-24T00:37:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.349+0000 7fc627e00640 1 -- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6000676b0 msgr2=0x7fc600069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.349+0000 7fc627e00640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6000676b0 0x7fc600069b70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc60c0ae000 tx=0x7fc60c0a7000 comp rx=0 tx=0).stop 2024-03-24T00:37:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.349+0000 7fc627e00640 1 -- 172.21.15.40:0/3881034438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6200728d0 msgr2=0x7fc6201545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.349+0000 7fc627e00640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6200728d0 0x7fc6201545e0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc61400e930 tx=0x7fc61400edf0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.349+0000 7fc627e00640 1 -- 172.21.15.40:0/3881034438 shutdown_connections 2024-03-24T00:37:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.349+0000 7fc627e00640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc6000676b0 0x7fc600069b70 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.349+0000 7fc627e00640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc62015f8c0 0x7fc620155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.349+0000 7fc627e00640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc62015bd00 0x7fc62014f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.349+0000 7fc627e00640 1 --2- 172.21.15.40:0/3881034438 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc6200728d0 0x7fc6201545e0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.349+0000 7fc627e00640 1 -- 172.21.15.40:0/3881034438 >> 172.21.15.40:0/3881034438 conn(0x7fc6200b1660 msgr2=0x7fc6200b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.349+0000 7fc627e00640 1 -- 172.21.15.40:0/3881034438 shutdown_connections 2024-03-24T00:37:56.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.349+0000 7fc627e00640 1 -- 172.21.15.40:0/3881034438 wait complete. 2024-03-24T00:37:56.372 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:37:56.372 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:37:56.469 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:37:56.549 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:37:56.550 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:37:56.550 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:37:56.559 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:37:56.559 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:37:56.559 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:37:56.568 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:37:56.568 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:37:56.568 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:37:56.610 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:37:56.611 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:37:56.611 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:37:56.620 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:37:56.620 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:37:56.620 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:37:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.789+0000 7f082b600640 1 Processor -- start 2024-03-24T00:37:56.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.789+0000 7f082b600640 1 -- start start 2024-03-24T00:37:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f082b600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f08241477a0 0x7f08240725a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f082b600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0824072c10 con 0x7f082415f050 2024-03-24T00:37:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f082b600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0824072ef0 con 0x7f08240b54d0 2024-03-24T00:37:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f082b600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f082415ebc0 con 0x7f08241477a0 2024-03-24T00:37:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f081be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f082415f050 0x7f082406f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42880/0 (socket says 172.21.15.40:42880) 2024-03-24T00:37:56.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f081be00640 1 -- 172.21.15.40:0/524097579 learned_addr learned my addr 172.21.15.40:0/524097579 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f0828c00640 1 --2- 172.21.15.40:0/524097579 >> v2:172.21.15.53:3300/0 conn(0x7f08241477a0 0x7f08240725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f0823e00640 1 -- 172.21.15.40:0/524097579 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531830167 0 0) 0x7f0824072c10 con 0x7f082415f050 2024-03-24T00:37:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f0823e00640 1 -- 172.21.15.40:0/524097579 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0808003530 con 0x7f082415f050 2024-03-24T00:37:56.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f0823e00640 1 -- 172.21.15.40:0/524097579 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833011729 0 0) 0x7f0824072ef0 con 0x7f08240b54d0 2024-03-24T00:37:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f0823e00640 1 -- 172.21.15.40:0/524097579 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0824072c10 con 0x7f08240b54d0 2024-03-24T00:37:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f0823e00640 1 -- 172.21.15.40:0/524097579 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3706749686 0 0) 0x7f0808003530 con 0x7f082415f050 2024-03-24T00:37:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f0828c00640 1 -- 172.21.15.40:0/524097579 >> v1:172.21.15.53:6789/0 conn(0x7f08240b54d0 legacy=0x7f08240b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f0828c00640 1 -- 172.21.15.40:0/524097579 >> v1:172.21.15.40:6789/0 conn(0x7f082415f050 legacy=0x7f082406f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.793+0000 7f0828c00640 1 -- 172.21.15.40:0/524097579 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082415e850 con 0x7f08241477a0 2024-03-24T00:37:56.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.797+0000 7f0828c00640 1 --2- 172.21.15.40:0/524097579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08241477a0 0x7f08240725a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f08140097c0 tx=0x7f081401c2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b7360eef9f6bd15 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:56.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.797+0000 7f0823e00640 1 -- 172.21.15.40:0/524097579 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08140b0070 con 0x7f08241477a0 2024-03-24T00:37:56.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.797+0000 7f0823e00640 1 -- 172.21.15.40:0/524097579 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08140ab400 con 0x7f08241477a0 2024-03-24T00:37:56.801 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.797+0000 7f0823e00640 1 -- 172.21.15.40:0/524097579 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080c001680 con 0x7f08241477a0 2024-03-24T00:37:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.797+0000 7f082b600640 1 -- 172.21.15.40:0/524097579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08241477a0 msgr2=0x7f08240725a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.797+0000 7f082b600640 1 --2- 172.21.15.40:0/524097579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08241477a0 0x7f08240725a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f08140097c0 tx=0x7f081401c2a0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.797+0000 7f082b600640 1 -- 172.21.15.40:0/524097579 shutdown_connections 2024-03-24T00:37:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.797+0000 7f082b600640 1 --2- 172.21.15.40:0/524097579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08241477a0 0x7f08240725a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.797+0000 7f082b600640 1 -- 172.21.15.40:0/524097579 >> 172.21.15.40:0/524097579 conn(0x7f082406b150 msgr2=0x7f082406b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.797+0000 7f082b600640 1 -- 172.21.15.40:0/524097579 shutdown_connections 2024-03-24T00:37:56.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.797+0000 7f082b600640 1 -- 172.21.15.40:0/524097579 wait complete. 2024-03-24T00:37:56.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f082b600640 1 Processor -- start 2024-03-24T00:37:56.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f082b600640 1 -- start start 2024-03-24T00:37:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f082b600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08240b54d0 0x7f082413ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f082b600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08241477a0 0x7f082413c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f082b600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f082415f050 0x7f0824157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f082b600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08240d14d0 con 0x7f08241477a0 2024-03-24T00:37:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f082b600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08240d1350 con 0x7f08240b54d0 2024-03-24T00:37:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f082b600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082416bc90 con 0x7f082415f050 2024-03-24T00:37:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f081be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08241477a0 0x7f082413c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f081be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08241477a0 0x7f082413c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42412/0 (socket says 172.21.15.40:42412) 2024-03-24T00:37:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f081be00640 1 -- 172.21.15.40:0/3287198832 learned_addr learned my addr 172.21.15.40:0/3287198832 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:56.806 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f0828c00640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08240b54d0 0x7f082413ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f0829600640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f082415f050 0x7f0824157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f081be00640 1 -- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f082415f050 msgr2=0x7f0824157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f081be00640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f082415f050 0x7f0824157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f081be00640 1 -- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08240b54d0 msgr2=0x7f082413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f081be00640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08240b54d0 0x7f082413ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f081be00640 1 -- 172.21.15.40:0/3287198832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082413d4b0 con 0x7f08241477a0 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f0829600640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f082415f050 0x7f0824157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f081be00640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08241477a0 0x7f082413c6f0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f080c00c780 tx=0x7f080c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f0821600640 1 -- 172.21.15.40:0/3287198832 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080c005ae0 con 0x7f08241477a0 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f0821600640 1 -- 172.21.15.40:0/3287198832 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f080c005c80 con 0x7f08241477a0 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f0821600640 1 -- 172.21.15.40:0/3287198832 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f080c0072b0 con 0x7f08241477a0 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f082b600640 1 -- 172.21.15.40:0/3287198832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082413d790 con 0x7f08241477a0 2024-03-24T00:37:56.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f082b600640 1 -- 172.21.15.40:0/3287198832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082413dd50 con 0x7f08241477a0 2024-03-24T00:37:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.801+0000 7f082b600640 1 -- 172.21.15.40:0/3287198832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f082406eec0 con 0x7f08241477a0 2024-03-24T00:37:56.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.805+0000 7f0821600640 1 -- 172.21.15.40:0/3287198832 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f080c007450 con 0x7f08241477a0 2024-03-24T00:37:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.805+0000 7f0821600640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0800067660 0x7f0800069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.805+0000 7f0821600640 1 -- 172.21.15.40:0/3287198832 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f080c085450 con 0x7f08241477a0 2024-03-24T00:37:56.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.805+0000 7f0828c00640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0800067660 0x7f0800069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:56.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.805+0000 7f0828c00640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0800067660 0x7f0800069b20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f08140ad000 tx=0x7f08140a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:56.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:56.809+0000 7f0821600640 1 -- 172.21.15.40:0/3287198832 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f080c056340 con 0x7f08241477a0 2024-03-24T00:37:57.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 -- 172.21.15.40:0/3287198832 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0824071df0 con 0x7f08241477a0 2024-03-24T00:37:57.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f0821600640 1 -- 172.21.15.40:0/3287198832 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+3700 (secure 0 0 0) 0x7f080c055c40 con 0x7f08241477a0 2024-03-24T00:37:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:57.013 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":105,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6755,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2936601899","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2936601899},{"type":"v1","addr":"172.21.15.53:6836","nonce":2936601899}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":92},{"gid":7016,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/4190528390","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":4190528390},{"type":"v1","addr":"172.21.15.53:6837","nonce":4190528390}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105},{"gid":8008,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/593833998","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":593833998},{"type":"v1","addr":"172.21.15.40:6835","nonce":593833998}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105},{"gid":8307,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4209120557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4209120557},{"type":"v1","addr":"172.21.15.40:6833","nonce":4209120557}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[]} 2024-03-24T00:37:57.013 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 105 2024-03-24T00:37:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 -- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0800067660 msgr2=0x7f0800069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0800067660 0x7f0800069b20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f08140ad000 tx=0x7f08140a6000 comp rx=0 tx=0).stop 2024-03-24T00:37:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 -- 172.21.15.40:0/3287198832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08241477a0 msgr2=0x7f082413c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08241477a0 0x7f082413c6f0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f080c00c780 tx=0x7f080c00cc40 comp rx=0 tx=0).stop 2024-03-24T00:37:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 -- 172.21.15.40:0/3287198832 shutdown_connections 2024-03-24T00:37:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0800067660 0x7f0800069b20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f082415f050 0x7f0824157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f08241477a0 0x7f082413c6f0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 --2- 172.21.15.40:0/3287198832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08240b54d0 0x7f082413ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 -- 172.21.15.40:0/3287198832 >> 172.21.15.40:0/3287198832 conn(0x7f082406b150 msgr2=0x7f08240b19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:57.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 -- 172.21.15.40:0/3287198832 shutdown_connections 2024-03-24T00:37:57.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.009+0000 7f082b600640 1 -- 172.21.15.40:0/3287198832 wait complete. 2024-03-24T00:37:57.034 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:37:57.034 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:37:57.035 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:37:57.035 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:37:57.035 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:37:57.035 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:37:57.035 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:37:57.035 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:37:57.036 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:37:57.036 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:37:57.036 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:37:57.036 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:37:57.036 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:37:57.036 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:37:57.036 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:37:57.036 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:37:57.047 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:37:57.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:37:57.049 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:37:57.049 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:37:57.052 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:37:57.052 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:37:57.053 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:37:57.053 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:37:57.093 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:37:57.093 DEBUG:teuthology.parallel:result is None 2024-03-24T00:37:57.095 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:37:57.096 DEBUG:teuthology.parallel:result is None 2024-03-24T00:37:57.102 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:37:57.102 DEBUG:teuthology.parallel:result is None 2024-03-24T00:37:57.103 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:37:57.104 DEBUG:teuthology.parallel:result is None 2024-03-24T00:37:57.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:37:57.180 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:37:57.181 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:37:57.198 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:37:57.198 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:37:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.245+0000 7f129a400640 1 Processor -- start 2024-03-24T00:37:57.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f129a400640 1 -- start start 2024-03-24T00:37:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f129a400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1294072310 0x7f12940ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f129a400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1294070e00 con 0x7f129413ec80 2024-03-24T00:37:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f129a400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12940cee20 con 0x7f1294141bc0 2024-03-24T00:37:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f129a400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12940cf100 con 0x7f1294072310 2024-03-24T00:37:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f1293400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f129413ec80 0x7f12941414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50468/0 (socket says 172.21.15.40:50468) 2024-03-24T00:37:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f1293400640 1 -- 172.21.15.40:0/2493700670 learned_addr learned my addr 172.21.15.40:0/2493700670 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f1293e00640 1 --2- 172.21.15.40:0/2493700670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1294072310 0x7f12940ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f1292a00640 1 -- 172.21.15.40:0/2493700670 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407285191 0 0) 0x7f1294070e00 con 0x7f129413ec80 2024-03-24T00:37:57.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f1292a00640 1 -- 172.21.15.40:0/2493700670 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1278003530 con 0x7f129413ec80 2024-03-24T00:37:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f1292a00640 1 -- 172.21.15.40:0/2493700670 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989188526 0 0) 0x7f12940cee20 con 0x7f1294141bc0 2024-03-24T00:37:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f1292a00640 1 -- 172.21.15.40:0/2493700670 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1294070e00 con 0x7f1294141bc0 2024-03-24T00:37:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f1293e00640 1 -- 172.21.15.40:0/2493700670 >> v1:172.21.15.53:6789/0 conn(0x7f1294141bc0 legacy=0x7f12940ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f1293e00640 1 -- 172.21.15.40:0/2493700670 >> v1:172.21.15.40:6789/0 conn(0x7f129413ec80 legacy=0x7f12941414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.249+0000 7f1293e00640 1 -- 172.21.15.40:0/2493700670 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12940cf3e0 con 0x7f1294072310 2024-03-24T00:37:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f1293e00640 1 --2- 172.21.15.40:0/2493700670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1294072310 0x7f12940ce800 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f128c009dc0 tx=0x7f128c01c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d431fceb6ae308b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f1292a00640 1 -- 172.21.15.40:0/2493700670 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c0b2070 con 0x7f1294072310 2024-03-24T00:37:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f1292a00640 1 -- 172.21.15.40:0/2493700670 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c0a5a70 con 0x7f1294072310 2024-03-24T00:37:57.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f1292a00640 1 -- 172.21.15.40:0/2493700670 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c0a5e40 con 0x7f1294072310 2024-03-24T00:37:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f129a400640 1 -- 172.21.15.40:0/2493700670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1294072310 msgr2=0x7f12940ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f129a400640 1 --2- 172.21.15.40:0/2493700670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1294072310 0x7f12940ce800 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f128c009dc0 tx=0x7f128c01c990 comp rx=0 tx=0).stop 2024-03-24T00:37:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f129a400640 1 -- 172.21.15.40:0/2493700670 shutdown_connections 2024-03-24T00:37:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f129a400640 1 --2- 172.21.15.40:0/2493700670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1294072310 0x7f12940ce800 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f129a400640 1 -- 172.21.15.40:0/2493700670 >> 172.21.15.40:0/2493700670 conn(0x7f12940b17b0 msgr2=0x7f12940b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:57.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f129a400640 1 -- 172.21.15.40:0/2493700670 shutdown_connections 2024-03-24T00:37:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f129a400640 1 -- 172.21.15.40:0/2493700670 wait complete. 2024-03-24T00:37:57.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f129a400640 1 Processor -- start 2024-03-24T00:37:57.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.253+0000 7f129a400640 1 -- start start 2024-03-24T00:37:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f129a400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1294072310 0x7f129413d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f129a400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f129413ec80 0x7f129414e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f129a400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1294141bc0 0x7f129414ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f129a400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1294169f00 con 0x7f1294141bc0 2024-03-24T00:37:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f129a400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1294169d80 con 0x7f1294072310 2024-03-24T00:37:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f129a400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129416a080 con 0x7f129413ec80 2024-03-24T00:37:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f1293e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1294141bc0 0x7f129414ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f1293e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1294141bc0 0x7f129414ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47846/0 (socket says 172.21.15.40:47846) 2024-03-24T00:37:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f1293e00640 1 -- 172.21.15.40:0/3607103150 learned_addr learned my addr 172.21.15.40:0/3607103150 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f1293400640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1294072310 0x7f129413d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f128aa00640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f129413ec80 0x7f129414e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f1293e00640 1 -- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f129413ec80 msgr2=0x7f129414e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f1293e00640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f129413ec80 0x7f129414e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f1293e00640 1 -- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1294072310 msgr2=0x7f129413d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f1293e00640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1294072310 0x7f129413d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f1293e00640 1 -- 172.21.15.40:0/3607103150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129406e420 con 0x7f1294141bc0 2024-03-24T00:37:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f1293400640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1294072310 0x7f129413d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f1293e00640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1294141bc0 0x7f129414ece0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f128c01a5c0 tx=0x7f128c019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f128be00640 1 -- 172.21.15.40:0/3607103150 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c0bd020 con 0x7f1294141bc0 2024-03-24T00:37:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f128be00640 1 -- 172.21.15.40:0/3607103150 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c0ada60 con 0x7f1294141bc0 2024-03-24T00:37:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f129a400640 1 -- 172.21.15.40:0/3607103150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129413d940 con 0x7f1294141bc0 2024-03-24T00:37:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f128be00640 1 -- 172.21.15.40:0/3607103150 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c0b2040 con 0x7f1294141bc0 2024-03-24T00:37:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f129a400640 1 -- 172.21.15.40:0/3607103150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129413fe10 con 0x7f1294141bc0 2024-03-24T00:37:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.257+0000 7f129a400640 1 -- 172.21.15.40:0/3607103150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1294068170 con 0x7f1294141bc0 2024-03-24T00:37:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.261+0000 7f128be00640 1 -- 172.21.15.40:0/3607103150 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f128c0c0020 con 0x7f1294141bc0 2024-03-24T00:37:57.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.261+0000 7f128be00640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1270067660 0x7f1270069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:57.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.261+0000 7f128be00640 1 -- 172.21.15.40:0/3607103150 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f128c120760 con 0x7f1294141bc0 2024-03-24T00:37:57.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.261+0000 7f1293400640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1270067660 0x7f1270069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:57.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.261+0000 7f1293400640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1270067660 0x7f1270069b20 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1284000f70 tx=0x7f1284009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.265+0000 7f128be00640 1 -- 172.21.15.40:0/3607103150 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f128c0c7030 con 0x7f1294141bc0 2024-03-24T00:37:57.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:57.437+0000 7f129a400640 1 -- 172.21.15.40:0/3607103150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1294072b90 con 0x7f1294141bc0 2024-03-24T00:37:58.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.001+0000 7f128be00640 1 -- 172.21.15.40:0/3607103150 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v100) v1 ==== 97+0+0 (secure 0 0 0) 0x7f129413d940 con 0x7f1294141bc0 2024-03-24T00:37:58.005 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-03-24T00:37:58.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.005+0000 7f129a400640 1 -- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1270067660 msgr2=0x7f1270069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.005+0000 7f129a400640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1270067660 0x7f1270069b20 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1284000f70 tx=0x7f1284009040 comp rx=0 tx=0).stop 2024-03-24T00:37:58.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.005+0000 7f129a400640 1 -- 172.21.15.40:0/3607103150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1294141bc0 msgr2=0x7f129414ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.005+0000 7f129a400640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1294141bc0 0x7f129414ece0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f128c01a5c0 tx=0x7f128c019730 comp rx=0 tx=0).stop 2024-03-24T00:37:58.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.005+0000 7f129a400640 1 -- 172.21.15.40:0/3607103150 shutdown_connections 2024-03-24T00:37:58.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.005+0000 7f129a400640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1270067660 0x7f1270069b20 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.005+0000 7f129a400640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f1294141bc0 0x7f129414ece0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.005+0000 7f129a400640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f129413ec80 0x7f129414e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.005+0000 7f129a400640 1 --2- 172.21.15.40:0/3607103150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1294072310 0x7f129413d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.005+0000 7f129a400640 1 -- 172.21.15.40:0/3607103150 >> 172.21.15.40:0/3607103150 conn(0x7f12940b17b0 msgr2=0x7f12940af920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:58.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.005+0000 7f129a400640 1 -- 172.21.15.40:0/3607103150 shutdown_connections 2024-03-24T00:37:58.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.005+0000 7f129a400640 1 -- 172.21.15.40:0/3607103150 wait complete. 2024-03-24T00:37:58.034 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:37:58.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.173+0000 7fa6d0c00640 1 Processor -- start 2024-03-24T00:37:58.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.173+0000 7fa6d0c00640 1 -- start start 2024-03-24T00:37:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6d0c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6cc13ec80 0x7fa6cc1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6d0c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6cc070e00 con 0x7fa6cc1419f0 2024-03-24T00:37:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6d0c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6cc0cee20 con 0x7fa6cc0ce800 2024-03-24T00:37:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6d0c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc141e50 con 0x7fa6cc13ec80 2024-03-24T00:37:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c3e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa6cc1419f0 0x7fa6cc0ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50470/0 (socket says 172.21.15.40:50470) 2024-03-24T00:37:58.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c3e00640 1 -- 172.21.15.40:0/2730188021 learned_addr learned my addr 172.21.15.40:0/2730188021 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:58.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9e00640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6cc13ec80 0x7fa6cc1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734255117 0 0) 0x7fa6cc070e00 con 0x7fa6cc1419f0 2024-03-24T00:37:58.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6b0003530 con 0x7fa6cc1419f0 2024-03-24T00:37:58.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970265636 0 0) 0x7fa6cc0cee20 con 0x7fa6cc0ce800 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6cc070e00 con 0x7fa6cc0ce800 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 350981004 0 0) 0x7fa6b0003530 con 0x7fa6cc1419f0 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6cc13ec80 msgr2=0x7fa6cc1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6cc13ec80 0x7fa6cc1414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 >> v1:172.21.15.53:6789/0 conn(0x7fa6cc0ce800 legacy=0x7fa6cc071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc0cf100 con 0x7fa6cc1419f0 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fa6b4001680 con 0x7fa6cc1419f0 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 >> v1:172.21.15.40:6789/0 conn(0x7fa6cc1419f0 legacy=0x7fa6cc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6cc13ec80 0x7fa6b0003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6b00040e0 0x7fa6b0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6b0004d50 0x7fa6b00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b000ba80 con 0x7fa6b00040e0 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b000bc30 con 0x7fa6b0004d50 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b000be30 con 0x7fa6cc13ec80 2024-03-24T00:37:58.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c3e00640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6b00040e0 0x7fa6b0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c9e00640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6cc13ec80 0x7fa6b0003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6ca800640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6b0004d50 0x7fa6b00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c3e00640 1 -- 172.21.15.40:0/2730188021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6cc13ec80 msgr2=0x7fa6b0003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:58.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c3e00640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6cc13ec80 0x7fa6b0003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c3e00640 1 -- 172.21.15.40:0/2730188021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6b0004d50 msgr2=0x7fa6b00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:58.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c3e00640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6b0004d50 0x7fa6b00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6c3e00640 1 -- 172.21.15.40:0/2730188021 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b0003530 con 0x7fa6b00040e0 2024-03-24T00:37:58.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.177+0000 7fa6ca800640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6b0004d50 0x7fa6b00051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:58.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.181+0000 7fa6c3e00640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6b00040e0 0x7fa6b0004810 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fa6b400c1c0 tx=0x7fa6b401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36612e8f7326dbe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.181+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4007cf0 con 0x7fa6b00040e0 2024-03-24T00:37:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.181+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b40a9020 con 0x7fa6b00040e0 2024-03-24T00:37:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.181+0000 7fa6c9400640 1 -- 172.21.15.40:0/2730188021 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b40a5b00 con 0x7fa6b00040e0 2024-03-24T00:37:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.181+0000 7fa6d0c00640 1 -- 172.21.15.40:0/2730188021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6b00040e0 msgr2=0x7fa6b0004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.181+0000 7fa6d0c00640 1 --2- 172.21.15.40:0/2730188021 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6b00040e0 0x7fa6b0004810 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7fa6b400c1c0 tx=0x7fa6b401cc00 comp rx=0 tx=0).stop 2024-03-24T00:37:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.181+0000 7fa6ca800640 1 -- 172.21.15.40:0/2730188021 reap_dead start 2024-03-24T00:37:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.181+0000 7fa6d0c00640 1 -- 172.21.15.40:0/2730188021 shutdown_connections 2024-03-24T00:37:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.181+0000 7fa6d0c00640 1 -- 172.21.15.40:0/2730188021 >> 172.21.15.40:0/2730188021 conn(0x7fa6cc0b17b0 msgr2=0x7fa6cc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.181+0000 7fa6d0c00640 1 -- 172.21.15.40:0/2730188021 shutdown_connections 2024-03-24T00:37:58.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.181+0000 7fa6d0c00640 1 -- 172.21.15.40:0/2730188021 wait complete. 2024-03-24T00:37:58.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6d0c00640 1 Processor -- start 2024-03-24T00:37:58.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6d0c00640 1 -- start start 2024-03-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6d0c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6b00040e0 0x7fa6cc15ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6d0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6cc13a4d0 0x7fa6cc0b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6d0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6cc06ded0 0x7fa6cc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6d0c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc1414b0 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6d0c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc13f060 con 0x7fa6cc06ded0 2024-03-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6d0c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc141630 con 0x7fa6b00040e0 2024-03-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6cc13a4d0 0x7fa6cc0b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6cc13a4d0 0x7fa6cc0b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47856/0 (socket says 172.21.15.40:47856) 2024-03-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c3e00640 1 -- 172.21.15.40:0/4233738005 learned_addr learned my addr 172.21.15.40:0/4233738005 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c9e00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6b00040e0 0x7fa6cc15ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6ca800640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6cc06ded0 0x7fa6cc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c3e00640 1 -- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6b00040e0 msgr2=0x7fa6cc15ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c3e00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6b00040e0 0x7fa6cc15ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c3e00640 1 -- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6cc06ded0 msgr2=0x7fa6cc06e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c3e00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6cc06ded0 0x7fa6cc06e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c3e00640 1 -- 172.21.15.40:0/4233738005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc0b3320 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c9e00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6b00040e0 0x7fa6cc15ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c3e00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6cc13a4d0 0x7fa6cc0b2bb0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40ab040 tx=0x7fa6b0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c2000640 1 -- 172.21.15.40:0/4233738005 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4001c20 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c2000640 1 -- 172.21.15.40:0/4233738005 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b40b3030 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6d0c00640 1 -- 172.21.15.40:0/4233738005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6cc0b3600 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6c2000640 1 -- 172.21.15.40:0/4233738005 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4004bf0 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6d0c00640 1 -- 172.21.15.40:0/4233738005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6cc0b3bc0 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.185+0000 7fa6d0c00640 1 -- 172.21.15.40:0/4233738005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6cc068170 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.189+0000 7fa6c2000640 1 -- 172.21.15.40:0/4233738005 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa6b4007070 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.189+0000 7fa6c2000640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6a8067690 0x7fa6a8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.189+0000 7fa6c2000640 1 -- 172.21.15.40:0/4233738005 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa6b411f090 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.189+0000 7fa6c9e00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6a8067690 0x7fa6a8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.189+0000 7fa6c9e00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6a8067690 0x7fa6a8069b50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009b00 tx=0x7fa6bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.193+0000 7fa6c2000640 1 -- 172.21.15.40:0/4233738005 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa6b4016910 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 -- 172.21.15.40:0/4233738005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa6cc13e5f0 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6c2000640 1 -- 172.21.15.40:0/4233738005 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa6b40effc0 con 0x7fa6cc13a4d0 2024-03-24T00:37:58.417 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:58.417 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:37:58.417 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-03-24T00:37:58.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.417 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-03-24T00:37:58.418 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lCkU6EhAA0q21F90epsGPK5x17s9NLw==", 2024-03-24T00:37:58.418 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.418 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:37:58.418 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:37:58.418 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:37:58.418 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.418 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.418 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-03-24T00:37:58.418 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llkfOHhAAqKKFyEJzG0UxbLJMFPET8A==", 2024-03-24T00:37:58.418 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lKMmwGRAAAKaOSnhTRbHIpPKXXkAe+Q==", 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.419 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l7quKJhAA8QsquiFFmj5H1G5yck0FpA==", 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAVdP9lVk3uBxAA9TC1OpeB5GIFlfJl5+ILBg==", 2024-03-24T00:37:58.420 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYdP9lUc1IDhAAPwCV9eqL7bFibkfMFt9wPw==", 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.421 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAbdP9lwjyuFBAAJ1JAIAnLXAn03+lQPwwY9Q==", 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAedP9lZ1bgGRAAuLAay48N7jXjj8/yaYwSqg==", 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.422 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAidP9lvyk4JhAA8w4/XBYHDLoqJ1zop9sUiw==", 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAldP9lcb9qLBAAXts6VU/4kVqUWS/WORV7zQ==", 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.423 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAodP9l8ipTMhAAjpagojh+woHyGvlQlIkHDw==", 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-03-24T00:37:58.424 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQArdP9lmGfpORAAPvow5nF/O7RLmuXEqNWENg==", 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw==", 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow r", 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:37:58.425 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.426 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-03-24T00:37:58.426 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lpIZrNBAAzpGWZuQ6ndjNMAOmAsebvQ==", 2024-03-24T00:37:58.426 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.426 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:37:58.426 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:37:58.426 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:37:58.426 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:37:58.426 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.427 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-03-24T00:37:58.427 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l6gXMMBAAd6Oe6S73hj/WfLXTsYiEZw==", 2024-03-24T00:37:58.427 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.427 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:37:58.427 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:37:58.427 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:37:58.427 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:37:58.427 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.427 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-03-24T00:37:58.427 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llFHCOhAATtv8gcL6BHPnxB1dHrg6vw==", 2024-03-24T00:37:58.428 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.428 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:37:58.428 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:37:58.428 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:37:58.428 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:37:58.428 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.428 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-03-24T00:37:58.428 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAQdP9lmmd7KxAA4uk6aUKP6egYj6O1NgdadQ==", 2024-03-24T00:37:58.429 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.429 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:37:58.429 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:37:58.429 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:37:58.429 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.429 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.430 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-03-24T00:37:58.430 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lebNrBxAAvbcCAJhDQn/5Uk1dJ01oHw==", 2024-03-24T00:37:58.430 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.430 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:37:58.431 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:37:58.431 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.431 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:37:58.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:58.431 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-03-24T00:37:58.431 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lEnZyAxAAW5Lrm7LZPTVfLCOY6As1PQ==", 2024-03-24T00:37:58.431 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:37:58.431 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:37:58.432 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:37:58.432 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:37:58.432 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.432 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:58.432 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-24T00:37:58.432 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:37:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 -- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6a8067690 msgr2=0x7fa6a8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6a8067690 0x7fa6a8069b50 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009b00 tx=0x7fa6bc008040 comp rx=0 tx=0).stop 2024-03-24T00:37:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 -- 172.21.15.40:0/4233738005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6cc13a4d0 msgr2=0x7fa6cc0b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6cc13a4d0 0x7fa6cc0b2bb0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40ab040 tx=0x7fa6b0005840 comp rx=0 tx=0).stop 2024-03-24T00:37:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 -- 172.21.15.40:0/4233738005 shutdown_connections 2024-03-24T00:37:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa6a8067690 0x7fa6a8069b50 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6cc06ded0 0x7fa6cc06e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa6cc13a4d0 0x7fa6cc0b2bb0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 --2- 172.21.15.40:0/4233738005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa6b00040e0 0x7fa6cc15ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 -- 172.21.15.40:0/4233738005 >> 172.21.15.40:0/4233738005 conn(0x7fa6cc0b17b0 msgr2=0x7fa6cc140d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:58.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 -- 172.21.15.40:0/4233738005 shutdown_connections 2024-03-24T00:37:58.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.413+0000 7fa6d0c00640 1 -- 172.21.15.40:0/4233738005 wait complete. 2024-03-24T00:37:58.439 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efef6c00640 1 Processor -- start 2024-03-24T00:37:58.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efef6c00640 1 -- start start 2024-03-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efef6c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efef00ce630 0x7efef00d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efef6c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef006c180 con 0x7efef00cea10 2024-03-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efef6c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef00cc190 con 0x7efef00cfdc0 2024-03-24T00:37:58.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efef6c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef00d1080 con 0x7efef00ce630 2024-03-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efef5800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efef00ce630 0x7efef00d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efef5800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efef00ce630 0x7efef00d2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47872/0 (socket says 172.21.15.40:47872) 2024-03-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efef5800640 1 -- 172.21.15.40:0/3592076269 learned_addr learned my addr 172.21.15.40:0/3592076269 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efef5800640 1 -- 172.21.15.40:0/3592076269 >> v1:172.21.15.53:6789/0 conn(0x7efef00cfdc0 legacy=0x7efef006ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efef5800640 1 -- 172.21.15.40:0/3592076269 >> v1:172.21.15.40:6789/0 conn(0x7efef00cea10 legacy=0x7efef006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efeefe00640 1 -- 172.21.15.40:0/3592076269 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266541365 0 0) 0x7efef006c180 con 0x7efef00cea10 2024-03-24T00:37:58.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.577+0000 7efef5800640 1 -- 172.21.15.40:0/3592076269 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef00d0da0 con 0x7efef00ce630 2024-03-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efef5800640 1 --2- 172.21.15.40:0/3592076269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef00ce630 0x7efef00d2590 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7efee4009870 tx=0x7efee401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bb1ee1e4e4930d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efeefe00640 1 -- 172.21.15.40:0/3592076269 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee40b5070 con 0x7efef00ce630 2024-03-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efeefe00640 1 -- 172.21.15.40:0/3592076269 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee40a8a10 con 0x7efef00ce630 2024-03-24T00:37:58.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efeefe00640 1 -- 172.21.15.40:0/3592076269 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee40a8de0 con 0x7efef00ce630 2024-03-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efef6c00640 1 -- 172.21.15.40:0/3592076269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef00ce630 msgr2=0x7efef00d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efef6c00640 1 --2- 172.21.15.40:0/3592076269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef00ce630 0x7efef00d2590 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7efee4009870 tx=0x7efee401ed10 comp rx=0 tx=0).stop 2024-03-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efef6c00640 1 -- 172.21.15.40:0/3592076269 shutdown_connections 2024-03-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efef6c00640 1 --2- 172.21.15.40:0/3592076269 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef00ce630 0x7efef00d2590 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efef6c00640 1 -- 172.21.15.40:0/3592076269 >> 172.21.15.40:0/3592076269 conn(0x7efef00d1aa0 msgr2=0x7efef00d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efef6c00640 1 -- 172.21.15.40:0/3592076269 shutdown_connections 2024-03-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efef6c00640 1 -- 172.21.15.40:0/3592076269 wait complete. 2024-03-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efef6c00640 1 Processor -- start 2024-03-24T00:37:58.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.581+0000 7efef6c00640 1 -- start start 2024-03-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efef00ce630 0x7efef01f32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efef00cea10 0x7efef009b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef00cfdc0 0x7efef00ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef006c4c0 con 0x7efef00cfdc0 2024-03-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef00cc710 con 0x7efef00ce630 2024-03-24T00:37:58.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef006c640 con 0x7efef00cea10 2024-03-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef00cfdc0 0x7efef00ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef00cfdc0 0x7efef00ca530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47878/0 (socket says 172.21.15.40:47878) 2024-03-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6200640 1 -- 172.21.15.40:0/1899806273 learned_addr learned my addr 172.21.15.40:0/1899806273 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef5800640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efef00ce630 0x7efef01f32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef4e00640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efef00cea10 0x7efef009b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6200640 1 -- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efef00cea10 msgr2=0x7efef009b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6200640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efef00cea10 0x7efef009b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6200640 1 -- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efef00ce630 msgr2=0x7efef01f32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6200640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efef00ce630 0x7efef01f32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6200640 1 -- 172.21.15.40:0/1899806273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef00c5670 con 0x7efef00cfdc0 2024-03-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6200640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef00cfdc0 0x7efef00ca530 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7efee800b970 tx=0x7efee800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efeed600640 1 -- 172.21.15.40:0/1899806273 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8019070 con 0x7efef00cfdc0 2024-03-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efeed600640 1 -- 172.21.15.40:0/1899806273 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee800cb40 con 0x7efef00cfdc0 2024-03-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efeed600640 1 -- 172.21.15.40:0/1899806273 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8014440 con 0x7efef00cfdc0 2024-03-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6c00640 1 -- 172.21.15.40:0/1899806273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef00c5950 con 0x7efef00cfdc0 2024-03-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6c00640 1 -- 172.21.15.40:0/1899806273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef00c61f0 con 0x7efef00cfdc0 2024-03-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efef6c00640 1 -- 172.21.15.40:0/1899806273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef00c5b20 con 0x7efef00cfdc0 2024-03-24T00:37:58.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.585+0000 7efeed600640 1 -- 172.21.15.40:0/1899806273 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7efee801c070 con 0x7efef00cfdc0 2024-03-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.589+0000 7efeed600640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efed0067660 0x7efed0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.589+0000 7efeed600640 1 -- 172.21.15.40:0/1899806273 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efee8085260 con 0x7efef00cfdc0 2024-03-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.589+0000 7efef5800640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efed0067660 0x7efed0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.589+0000 7efef5800640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efed0067660 0x7efed0069b20 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7efee40b2000 tx=0x7efee4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.593+0000 7efeed600640 1 -- 172.21.15.40:0/1899806273 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7efee8056190 con 0x7efef00cfdc0 2024-03-24T00:37:58.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.753+0000 7efef6c00640 1 -- 172.21.15.40:0/1899806273 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efef00c63f0 con 0x7efef00cfdc0 2024-03-24T00:37:58.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.753+0000 7efeed600640 1 -- 172.21.15.40:0/1899806273 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+12833 (secure 0 0 0) 0x7efef00c63f0 con 0x7efef00cfdc0 2024-03-24T00:37:58.759 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:58.759 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":102,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:37:58.013749+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:37:58.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.757+0000 7efef6c00640 1 -- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efed0067660 msgr2=0x7efed0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.757+0000 7efef6c00640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efed0067660 0x7efed0069b20 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7efee40b2000 tx=0x7efee4008040 comp rx=0 tx=0).stop 2024-03-24T00:37:58.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.757+0000 7efef6c00640 1 -- 172.21.15.40:0/1899806273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef00cfdc0 msgr2=0x7efef00ca530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.757+0000 7efef6c00640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef00cfdc0 0x7efef00ca530 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7efee800b970 tx=0x7efee800be30 comp rx=0 tx=0).stop 2024-03-24T00:37:58.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.757+0000 7efef6c00640 1 -- 172.21.15.40:0/1899806273 shutdown_connections 2024-03-24T00:37:58.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.757+0000 7efef6c00640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efed0067660 0x7efed0069b20 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.757+0000 7efef6c00640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efef00cfdc0 0x7efef00ca530 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.757+0000 7efef6c00640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efef00cea10 0x7efef009b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.757+0000 7efef6c00640 1 --2- 172.21.15.40:0/1899806273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efef00ce630 0x7efef01f32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.757+0000 7efef6c00640 1 -- 172.21.15.40:0/1899806273 >> 172.21.15.40:0/1899806273 conn(0x7efef00d1aa0 msgr2=0x7efef006fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:58.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.757+0000 7efef6c00640 1 -- 172.21.15.40:0/1899806273 shutdown_connections 2024-03-24T00:37:58.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.757+0000 7efef6c00640 1 -- 172.21.15.40:0/1899806273 wait complete. 2024-03-24T00:37:58.780 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:37:58.781 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:37:58.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb120c00640 1 Processor -- start 2024-03-24T00:37:58.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb120c00640 1 -- start start 2024-03-24T00:37:58.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb120c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb11c15ac60 0x7fb11c0b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb120c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c0b3260 con 0x7fb11c06cb80 2024-03-24T00:37:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb120c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c0b3540 con 0x7fb11c071b00 2024-03-24T00:37:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb120c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb11c0b3e70 con 0x7fb11c15ac60 2024-03-24T00:37:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb113e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fb11c06cb80 0x7fb11c0b4300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50488/0 (socket says 172.21.15.40:50488) 2024-03-24T00:37:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb113e00640 1 -- 172.21.15.40:0/2848444990 learned_addr learned my addr 172.21.15.40:0/2848444990 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:37:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb119e00640 1 --2- 172.21.15.40:0/2848444990 >> v2:172.21.15.53:3300/0 conn(0x7fb11c15ac60 0x7fb11c0b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb119400640 1 -- 172.21.15.40:0/2848444990 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166763482 0 0) 0x7fb11c0b3260 con 0x7fb11c06cb80 2024-03-24T00:37:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb119400640 1 -- 172.21.15.40:0/2848444990 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb100003530 con 0x7fb11c06cb80 2024-03-24T00:37:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb119400640 1 -- 172.21.15.40:0/2848444990 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276806609 0 0) 0x7fb11c0b3540 con 0x7fb11c071b00 2024-03-24T00:37:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb119e00640 1 -- 172.21.15.40:0/2848444990 >> v1:172.21.15.53:6789/0 conn(0x7fb11c071b00 legacy=0x7fb11c0ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb119e00640 1 -- 172.21.15.40:0/2848444990 >> v1:172.21.15.40:6789/0 conn(0x7fb11c06cb80 legacy=0x7fb11c0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.921+0000 7fb119e00640 1 -- 172.21.15.40:0/2848444990 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c0b3820 con 0x7fb11c15ac60 2024-03-24T00:37:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.925+0000 7fb119e00640 1 --2- 172.21.15.40:0/2848444990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15ac60 0x7fb11c0b2960 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0097c0 tx=0x7fb10c01c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2693f46add0c632 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.925+0000 7fb119400640 1 -- 172.21.15.40:0/2848444990 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c0ab600 con 0x7fb11c15ac60 2024-03-24T00:37:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.925+0000 7fb119400640 1 -- 172.21.15.40:0/2848444990 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c0b0070 con 0x7fb11c15ac60 2024-03-24T00:37:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.925+0000 7fb119400640 1 -- 172.21.15.40:0/2848444990 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c0ab9c0 con 0x7fb11c15ac60 2024-03-24T00:37:58.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.925+0000 7fb120c00640 1 -- 172.21.15.40:0/2848444990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15ac60 msgr2=0x7fb11c0b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.925+0000 7fb120c00640 1 --2- 172.21.15.40:0/2848444990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15ac60 0x7fb11c0b2960 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0097c0 tx=0x7fb10c01c3e0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.925+0000 7fb120c00640 1 -- 172.21.15.40:0/2848444990 shutdown_connections 2024-03-24T00:37:58.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.925+0000 7fb120c00640 1 --2- 172.21.15.40:0/2848444990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15ac60 0x7fb11c0b2960 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.925+0000 7fb120c00640 1 -- 172.21.15.40:0/2848444990 >> 172.21.15.40:0/2848444990 conn(0x7fb11c0b1660 msgr2=0x7fb11c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:58.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.925+0000 7fb120c00640 1 -- 172.21.15.40:0/2848444990 shutdown_connections 2024-03-24T00:37:58.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.925+0000 7fb120c00640 1 -- 172.21.15.40:0/2848444990 wait complete. 2024-03-24T00:37:58.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb120c00640 1 Processor -- start 2024-03-24T00:37:58.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb120c00640 1 -- start start 2024-03-24T00:37:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb120c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb11c06cb80 0x7fb11c1576f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb120c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c071b00 0x7fb11c0b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb120c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15ac60 0x7fb11c144f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb120c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c0b3020 con 0x7fb11c06cb80 2024-03-24T00:37:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb120c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c0b2ea0 con 0x7fb11c071b00 2024-03-24T00:37:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb120c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c15b040 con 0x7fb11c15ac60 2024-03-24T00:37:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb119e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb11c06cb80 0x7fb11c1576f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb119e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb11c06cb80 0x7fb11c1576f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47892/0 (socket says 172.21.15.40:47892) 2024-03-24T00:37:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb119e00640 1 -- 172.21.15.40:0/3962397226 learned_addr learned my addr 172.21.15.40:0/3962397226 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb113e00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c071b00 0x7fb11c0b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb11a800640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15ac60 0x7fb11c144f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb119e00640 1 -- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15ac60 msgr2=0x7fb11c144f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb119e00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15ac60 0x7fb11c144f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb119e00640 1 -- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c071b00 msgr2=0x7fb11c0b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb119e00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c071b00 0x7fb11c0b0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb119e00640 1 -- 172.21.15.40:0/3962397226 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c157e60 con 0x7fb11c06cb80 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb11a800640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15ac60 0x7fb11c144f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb113e00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c071b00 0x7fb11c0b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb119e00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb11c06cb80 0x7fb11c1576f0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0097c0 tx=0x7fb10c01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb112000640 1 -- 172.21.15.40:0/3962397226 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c0bf410 con 0x7fb11c06cb80 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb112000640 1 -- 172.21.15.40:0/3962397226 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10c0bf5b0 con 0x7fb11c06cb80 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb112000640 1 -- 172.21.15.40:0/3962397226 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10c0bf9a0 con 0x7fb11c06cb80 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb120c00640 1 -- 172.21.15.40:0/3962397226 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11c158140 con 0x7fb11c06cb80 2024-03-24T00:37:58.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb120c00640 1 -- 172.21.15.40:0/3962397226 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11c158700 con 0x7fb11c06cb80 2024-03-24T00:37:58.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.929+0000 7fb120c00640 1 -- 172.21.15.40:0/3962397226 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11c068170 con 0x7fb11c06cb80 2024-03-24T00:37:58.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.933+0000 7fb112000640 1 -- 172.21.15.40:0/3962397226 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb10c0bd020 con 0x7fb11c06cb80 2024-03-24T00:37:58.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.933+0000 7fb112000640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0f8067660 0x7fb0f8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:58.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.933+0000 7fb112000640 1 -- 172.21.15.40:0/3962397226 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb10c11f7f0 con 0x7fb11c06cb80 2024-03-24T00:37:58.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.933+0000 7fb113e00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0f8067660 0x7fb0f8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:58.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.933+0000 7fb113e00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0f8067660 0x7fb0f8069b20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb11c159650 tx=0x7fb104009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:58.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:58.937+0000 7fb112000640 1 -- 172.21.15.40:0/3962397226 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb10c0c6030 con 0x7fb11c06cb80 2024-03-24T00:37:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.105+0000 7fb120c00640 1 -- 172.21.15.40:0/3962397226 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb11c0cc6e0 con 0x7fb11c06cb80 2024-03-24T00:37:59.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.105+0000 7fb112000640 1 -- 172.21.15.40:0/3962397226 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb10c016090 con 0x7fb11c06cb80 2024-03-24T00:37:59.110 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:37:59.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.105+0000 7fb120c00640 1 -- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0f8067660 msgr2=0x7fb0f8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.105+0000 7fb120c00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0f8067660 0x7fb0f8069b20 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fb11c159650 tx=0x7fb104009040 comp rx=0 tx=0).stop 2024-03-24T00:37:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.105+0000 7fb120c00640 1 -- 172.21.15.40:0/3962397226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb11c06cb80 msgr2=0x7fb11c1576f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.105+0000 7fb120c00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb11c06cb80 0x7fb11c1576f0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0097c0 tx=0x7fb10c01ac20 comp rx=0 tx=0).stop 2024-03-24T00:37:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.109+0000 7fb120c00640 1 -- 172.21.15.40:0/3962397226 shutdown_connections 2024-03-24T00:37:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.109+0000 7fb120c00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb0f8067660 0x7fb0f8069b20 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.109+0000 7fb120c00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb11c15ac60 0x7fb11c144f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.109+0000 7fb120c00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c071b00 0x7fb11c0b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.109+0000 7fb120c00640 1 --2- 172.21.15.40:0/3962397226 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb11c06cb80 0x7fb11c1576f0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.109+0000 7fb120c00640 1 -- 172.21.15.40:0/3962397226 >> 172.21.15.40:0/3962397226 conn(0x7fb11c0b1660 msgr2=0x7fb11c0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.109+0000 7fb120c00640 1 -- 172.21.15.40:0/3962397226 shutdown_connections 2024-03-24T00:37:59.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.109+0000 7fb120c00640 1 -- 172.21.15.40:0/3962397226 wait complete. 2024-03-24T00:37:59.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:37:59.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8bb600640 1 Processor -- start 2024-03-24T00:37:59.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8bb600640 1 -- start start 2024-03-24T00:37:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8bb600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8b4145760 0x7fe8b406f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8bb600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe8b4072750 0x7fe8b406e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8bb600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8b406ea50 0x7fe8b40cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8bb600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b40cd630 con 0x7fe8b406ea50 2024-03-24T00:37:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8bb600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8b40cd910 con 0x7fe8b4072750 2024-03-24T00:37:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8bb600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b40cdbf0 con 0x7fe8b4145760 2024-03-24T00:37:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8b9600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8b406ea50 0x7fe8b40cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8b9600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe8b406ea50 0x7fe8b40cf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47894/0 (socket says 172.21.15.40:47894) 2024-03-24T00:37:59.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8b9600640 1 -- 172.21.15.40:0/1022803174 learned_addr learned my addr 172.21.15.40:0/1022803174 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:59.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8b8c00640 1 --2- 172.21.15.40:0/1022803174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8b4145760 0x7fe8b406f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8b3e00640 1 --2- 172.21.15.40:0/1022803174 >> v2:172.21.15.53:3300/0 conn(0x7fe8b4072750 0x7fe8b406e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8b9600640 1 -- 172.21.15.40:0/1022803174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8b4145760 msgr2=0x7fe8b406f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:59.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8b9600640 1 --2- 172.21.15.40:0/1022803174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8b4145760 0x7fe8b406f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8b9600640 1 -- 172.21.15.40:0/1022803174 >> v2:172.21.15.53:3300/0 conn(0x7fe8b4072750 msgr2=0x7fe8b406e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:37:59.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8b9600640 1 --2- 172.21.15.40:0/1022803174 >> v2:172.21.15.53:3300/0 conn(0x7fe8b4072750 0x7fe8b406e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.269+0000 7fe8b9600640 1 -- 172.21.15.40:0/1022803174 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b406dd10 con 0x7fe8b406ea50 2024-03-24T00:37:59.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8b9600640 1 --2- 172.21.15.40:0/1022803174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b406ea50 0x7fe8b40cf200 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fe8b406c2d0 tx=0x7fe8ac01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f28e3af8e5552e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8b3400640 1 -- 172.21.15.40:0/1022803174 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac0a5670 con 0x7fe8b406ea50 2024-03-24T00:37:59.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8b3400640 1 -- 172.21.15.40:0/1022803174 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ac0a5810 con 0x7fe8b406ea50 2024-03-24T00:37:59.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8b3400640 1 -- 172.21.15.40:0/1022803174 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac0a5be0 con 0x7fe8b406ea50 2024-03-24T00:37:59.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8bb600640 1 -- 172.21.15.40:0/1022803174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b406ea50 msgr2=0x7fe8b40cf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8bb600640 1 --2- 172.21.15.40:0/1022803174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b406ea50 0x7fe8b40cf200 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fe8b406c2d0 tx=0x7fe8ac01c9f0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8bb600640 1 -- 172.21.15.40:0/1022803174 shutdown_connections 2024-03-24T00:37:59.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8bb600640 1 --2- 172.21.15.40:0/1022803174 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b406ea50 0x7fe8b40cf200 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8bb600640 1 --2- 172.21.15.40:0/1022803174 >> v2:172.21.15.53:3300/0 conn(0x7fe8b4072750 0x7fe8b406e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8bb600640 1 --2- 172.21.15.40:0/1022803174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8b4145760 0x7fe8b406f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8bb600640 1 -- 172.21.15.40:0/1022803174 >> 172.21.15.40:0/1022803174 conn(0x7fe8b40d0c20 msgr2=0x7fe8b40d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:59.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8bb600640 1 -- 172.21.15.40:0/1022803174 shutdown_connections 2024-03-24T00:37:59.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8bb600640 1 -- 172.21.15.40:0/1022803174 wait complete. 2024-03-24T00:37:59.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8bb600640 1 Processor -- start 2024-03-24T00:37:59.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.273+0000 7fe8bb600640 1 -- start start 2024-03-24T00:37:59.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8bb600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8b406ea50 0x7fe8b413c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8bb600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b4072750 0x7fe8b4164f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8bb600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b4145760 0x7fe8b4165440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8bb600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b4167bd0 con 0x7fe8b4145760 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8bb600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b4167a50 con 0x7fe8b4072750 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8bb600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b4167d50 con 0x7fe8b406ea50 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8b9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b4145760 0x7fe8b4165440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8b9600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b4145760 0x7fe8b4165440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47908/0 (socket says 172.21.15.40:47908) 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8b9600640 1 -- 172.21.15.40:0/545242070 learned_addr learned my addr 172.21.15.40:0/545242070 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8b9600640 1 -- 172.21.15.40:0/545242070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8b406ea50 msgr2=0x7fe8b413c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8b9600640 1 --2- 172.21.15.40:0/545242070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8b406ea50 0x7fe8b413c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8b9600640 1 -- 172.21.15.40:0/545242070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b4072750 msgr2=0x7fe8b4164f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8b9600640 1 --2- 172.21.15.40:0/545242070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b4072750 0x7fe8b4164f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8b9600640 1 -- 172.21.15.40:0/545242070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b4160410 con 0x7fe8b4145760 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8b9600640 1 --2- 172.21.15.40:0/545242070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b4145760 0x7fe8b4165440 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac01d290 tx=0x7fe8ac019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8b0c00640 1 -- 172.21.15.40:0/545242070 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac009510 con 0x7fe8b4145760 2024-03-24T00:37:59.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8b0c00640 1 -- 172.21.15.40:0/545242070 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ac0be070 con 0x7fe8b4145760 2024-03-24T00:37:59.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8b0c00640 1 -- 172.21.15.40:0/545242070 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac0b88b0 con 0x7fe8b4145760 2024-03-24T00:37:59.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8bb600640 1 -- 172.21.15.40:0/545242070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b41606f0 con 0x7fe8b4145760 2024-03-24T00:37:59.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8bb600640 1 -- 172.21.15.40:0/545242070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b4160cb0 con 0x7fe8b4145760 2024-03-24T00:37:59.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.277+0000 7fe8bb600640 1 -- 172.21.15.40:0/545242070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b4068170 con 0x7fe8b4145760 2024-03-24T00:37:59.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.281+0000 7fe8b0c00640 1 -- 172.21.15.40:0/545242070 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe8ac0b8a50 con 0x7fe8b4145760 2024-03-24T00:37:59.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.281+0000 7fe8b0c00640 1 --2- 172.21.15.40:0/545242070 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe894067660 0x7fe894069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.281+0000 7fe8b0c00640 1 -- 172.21.15.40:0/545242070 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe8ac120750 con 0x7fe8b4145760 2024-03-24T00:37:59.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.281+0000 7fe8b8c00640 1 --2- 172.21.15.40:0/545242070 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe894067660 0x7fe894069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.281+0000 7fe8b8c00640 1 --2- 172.21.15.40:0/545242070 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe894067660 0x7fe894069b20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80098f0 tx=0x7fe8a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.285+0000 7fe8b0c00640 1 -- 172.21.15.40:0/545242070 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe8ac0165e0 con 0x7fe8b4145760 2024-03-24T00:37:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.449+0000 7fe8bb600640 1 -- 172.21.15.40:0/545242070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe8b4067d70 con 0x7fe8b4145760 2024-03-24T00:37:59.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.449+0000 7fe8b0c00640 1 -- 172.21.15.40:0/545242070 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v102) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe8ac0f1710 con 0x7fe8b4145760 2024-03-24T00:37:59.455 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:37:59.455 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-03-24T00:37:59.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:37:59.455 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-03-24T00:37:59.455 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-03-24T00:37:59.455 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:37:59.455 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-03-24T00:37:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.453+0000 7fe8bb600640 1 -- 172.21.15.40:0/545242070 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe894067660 msgr2=0x7fe894069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.453+0000 7fe8bb600640 1 --2- 172.21.15.40:0/545242070 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe894067660 0x7fe894069b20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fe8a80098f0 tx=0x7fe8a8008040 comp rx=0 tx=0).stop 2024-03-24T00:37:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.453+0000 7fe8bb600640 1 -- 172.21.15.40:0/545242070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b4145760 msgr2=0x7fe8b4165440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.453+0000 7fe8bb600640 1 --2- 172.21.15.40:0/545242070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b4145760 0x7fe8b4165440 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac01d290 tx=0x7fe8ac019790 comp rx=0 tx=0).stop 2024-03-24T00:37:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.453+0000 7fe8bb600640 1 -- 172.21.15.40:0/545242070 shutdown_connections 2024-03-24T00:37:59.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.453+0000 7fe8bb600640 1 --2- 172.21.15.40:0/545242070 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe894067660 0x7fe894069b20 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.453+0000 7fe8bb600640 1 --2- 172.21.15.40:0/545242070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe8b4145760 0x7fe8b4165440 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.453+0000 7fe8bb600640 1 --2- 172.21.15.40:0/545242070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8b4072750 0x7fe8b4164f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.453+0000 7fe8bb600640 1 --2- 172.21.15.40:0/545242070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8b406ea50 0x7fe8b413c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.453+0000 7fe8bb600640 1 -- 172.21.15.40:0/545242070 >> 172.21.15.40:0/545242070 conn(0x7fe8b40d0c20 msgr2=0x7fe8b415c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.453+0000 7fe8bb600640 1 -- 172.21.15.40:0/545242070 shutdown_connections 2024-03-24T00:37:59.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.453+0000 7fe8bb600640 1 -- 172.21.15.40:0/545242070 wait complete. 2024-03-24T00:37:59.476 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:37:59.477 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:37:59.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.617+0000 7ff18ba00640 1 Processor -- start 2024-03-24T00:37:59.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.617+0000 7ff18ba00640 1 -- start start 2024-03-24T00:37:59.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.617+0000 7ff18ba00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff18415ffd0 0x7ff1840ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.617+0000 7ff18ba00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff184071de0 con 0x7ff1840728d0 2024-03-24T00:37:59.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.617+0000 7ff18ba00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1840cea50 con 0x7ff18415bd00 2024-03-24T00:37:59.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.617+0000 7ff18ba00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1840ced30 con 0x7ff18415ffd0 2024-03-24T00:37:59.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.617+0000 7ff17be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff18415ffd0 0x7ff1840ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.617+0000 7ff17be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff18415ffd0 0x7ff1840ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47920/0 (socket says 172.21.15.40:47920) 2024-03-24T00:37:59.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.617+0000 7ff17be00640 1 -- 172.21.15.40:0/760791386 learned_addr learned my addr 172.21.15.40:0/760791386 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:59.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.617+0000 7ff189a00640 1 --1- 172.21.15.40:0/760791386 >> v1:172.21.15.40:6789/0 conn(0x7ff1840728d0 0x7ff18406cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50490/0 (socket says 172.21.15.40:50490) 2024-03-24T00:37:59.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff17be00640 1 -- 172.21.15.40:0/760791386 >> v1:172.21.15.53:6789/0 conn(0x7ff18415bd00 legacy=0x7ff18415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff17be00640 1 -- 172.21.15.40:0/760791386 >> v1:172.21.15.40:6789/0 conn(0x7ff1840728d0 legacy=0x7ff18406cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff17be00640 1 -- 172.21.15.40:0/760791386 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff184071b00 con 0x7ff18415ffd0 2024-03-24T00:37:59.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff183e00640 1 -- 172.21.15.40:0/760791386 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260376514 0 0) 0x7ff184071de0 con 0x7ff1840728d0 2024-03-24T00:37:59.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff17be00640 1 --2- 172.21.15.40:0/760791386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18415ffd0 0x7ff1840ae8e0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009870 tx=0x7ff16c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40024d032e14141f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff183e00640 1 -- 172.21.15.40:0/760791386 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c0aa650 con 0x7ff18415ffd0 2024-03-24T00:37:59.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff183e00640 1 -- 172.21.15.40:0/760791386 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16c0af070 con 0x7ff18415ffd0 2024-03-24T00:37:59.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff183e00640 1 -- 172.21.15.40:0/760791386 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c0aaa10 con 0x7ff18415ffd0 2024-03-24T00:37:59.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff18ba00640 1 -- 172.21.15.40:0/760791386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18415ffd0 msgr2=0x7ff1840ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff18ba00640 1 --2- 172.21.15.40:0/760791386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18415ffd0 0x7ff1840ae8e0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7ff16c009870 tx=0x7ff16c01c520 comp rx=0 tx=0).stop 2024-03-24T00:37:59.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff18ba00640 1 -- 172.21.15.40:0/760791386 shutdown_connections 2024-03-24T00:37:59.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff18ba00640 1 --2- 172.21.15.40:0/760791386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18415ffd0 0x7ff1840ae8e0 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff18ba00640 1 -- 172.21.15.40:0/760791386 >> 172.21.15.40:0/760791386 conn(0x7ff1840b1660 msgr2=0x7ff1840b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:37:59.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff18ba00640 1 -- 172.21.15.40:0/760791386 shutdown_connections 2024-03-24T00:37:59.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.621+0000 7ff18ba00640 1 -- 172.21.15.40:0/760791386 wait complete. 2024-03-24T00:37:59.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff18ba00640 1 Processor -- start 2024-03-24T00:37:59.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff18ba00640 1 -- start start 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff18ba00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1840728d0 0x7ff184155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff18ba00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18415bd00 0x7ff1841584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff18ba00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff18415ffd0 0x7ff184158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff18ba00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18415c0e0 con 0x7ff18415bd00 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff18ba00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1840cf660 con 0x7ff18415ffd0 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff18ba00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18416a420 con 0x7ff1840728d0 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff17be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18415bd00 0x7ff1841584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff17be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18415bd00 0x7ff1841584e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47924/0 (socket says 172.21.15.40:47924) 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff17be00640 1 -- 172.21.15.40:0/3882873685 learned_addr learned my addr 172.21.15.40:0/3882873685 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff189a00640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff18415ffd0 0x7ff184158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff17be00640 1 -- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1840728d0 msgr2=0x7ff184155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff189000640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1840728d0 0x7ff184155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff17be00640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1840728d0 0x7ff184155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff17be00640 1 -- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff18415ffd0 msgr2=0x7ff184158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:37:59.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff17be00640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff18415ffd0 0x7ff184158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:37:59.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff17be00640 1 -- 172.21.15.40:0/3882873685 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff184155d80 con 0x7ff18415bd00 2024-03-24T00:37:59.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff189a00640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff18415ffd0 0x7ff184158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:37:59.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff189000640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1840728d0 0x7ff184155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:37:59.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff17be00640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18415bd00 0x7ff1841584e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7ff16c0ac040 tx=0x7ff16c01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff181600640 1 -- 172.21.15.40:0/3882873685 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c0aae60 con 0x7ff18415bd00 2024-03-24T00:37:59.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff181600640 1 -- 172.21.15.40:0/3882873685 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff16c0a6040 con 0x7ff18415bd00 2024-03-24T00:37:59.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff18ba00640 1 -- 172.21.15.40:0/3882873685 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff184156060 con 0x7ff18415bd00 2024-03-24T00:37:59.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff181600640 1 -- 172.21.15.40:0/3882873685 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff16c0b3a40 con 0x7ff18415bd00 2024-03-24T00:37:59.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.625+0000 7ff18ba00640 1 -- 172.21.15.40:0/3882873685 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff184151230 con 0x7ff18415bd00 2024-03-24T00:37:59.632 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.629+0000 7ff18ba00640 1 -- 172.21.15.40:0/3882873685 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff184068170 con 0x7ff18415bd00 2024-03-24T00:37:59.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.629+0000 7ff181600640 1 -- 172.21.15.40:0/3882873685 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff16c0b3be0 con 0x7ff18415bd00 2024-03-24T00:37:59.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.629+0000 7ff181600640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff160067660 0x7ff160069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:37:59.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.629+0000 7ff181600640 1 -- 172.21.15.40:0/3882873685 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff16c11fc90 con 0x7ff18415bd00 2024-03-24T00:37:59.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.629+0000 7ff189000640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff160067660 0x7ff160069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:37:59.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.629+0000 7ff189000640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff160067660 0x7ff160069b20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff174000f70 tx=0x7ff174009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:37:59.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.633+0000 7ff181600640 1 -- 172.21.15.40:0/3882873685 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff16c0c3060 con 0x7ff18415bd00 2024-03-24T00:37:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:37:59.797+0000 7ff18ba00640 1 -- 172.21.15.40:0/3882873685 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff18406e870 con 0x7ff18415bd00 2024-03-24T00:38:00.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.009+0000 7ff181600640 1 -- 172.21.15.40:0/3882873685 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v103) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff16c016110 con 0x7ff18415bd00 2024-03-24T00:38:00.014 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-03-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.013+0000 7ff18ba00640 1 -- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff160067660 msgr2=0x7ff160069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.013+0000 7ff18ba00640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff160067660 0x7ff160069b20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff174000f70 tx=0x7ff174009040 comp rx=0 tx=0).stop 2024-03-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.013+0000 7ff18ba00640 1 -- 172.21.15.40:0/3882873685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18415bd00 msgr2=0x7ff1841584e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.013+0000 7ff18ba00640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18415bd00 0x7ff1841584e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7ff16c0ac040 tx=0x7ff16c01ad60 comp rx=0 tx=0).stop 2024-03-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.013+0000 7ff18ba00640 1 -- 172.21.15.40:0/3882873685 shutdown_connections 2024-03-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.013+0000 7ff18ba00640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff160067660 0x7ff160069b20 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.013+0000 7ff18ba00640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff18415ffd0 0x7ff184158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.013+0000 7ff18ba00640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff18415bd00 0x7ff1841584e0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.013+0000 7ff18ba00640 1 --2- 172.21.15.40:0/3882873685 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1840728d0 0x7ff184155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.013+0000 7ff18ba00640 1 -- 172.21.15.40:0/3882873685 >> 172.21.15.40:0/3882873685 conn(0x7ff1840b1660 msgr2=0x7ff1840cfb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.013+0000 7ff18ba00640 1 -- 172.21.15.40:0/3882873685 shutdown_connections 2024-03-24T00:38:00.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.013+0000 7ff18ba00640 1 -- 172.21.15.40:0/3882873685 wait complete. 2024-03-24T00:38:00.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:38:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.185+0000 7f3c3ac00640 1 Processor -- start 2024-03-24T00:38:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.185+0000 7f3c3ac00640 1 -- start start 2024-03-24T00:38:00.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.185+0000 7f3c3ac00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3c340b2360 0x7f3c340b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.185+0000 7f3c3ac00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3c340b4330 0x7f3c340ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.185+0000 7f3c3ac00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c340cf090 con 0x7f3c340cea00 2024-03-24T00:38:00.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.185+0000 7f3c3ac00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c340d0f60 con 0x7f3c340b2360 2024-03-24T00:38:00.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.185+0000 7f3c3ac00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c340cee30 con 0x7f3c340b4330 2024-03-24T00:38:00.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.185+0000 7f3c33e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3c340b2360 0x7f3c340b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.185+0000 7f3c2be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3c340b4330 0x7f3c340ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.185+0000 7f3c33e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3c340b2360 0x7f3c340b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39294/0 (socket says 172.21.15.40:39294) 2024-03-24T00:38:00.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.185+0000 7f3c33e00640 1 -- 172.21.15.40:0/2282977927 learned_addr learned my addr 172.21.15.40:0/2282977927 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:00.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.189+0000 7f3c33e00640 1 -- 172.21.15.40:0/2282977927 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3c340b4330 msgr2=0x7f3c340ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:00.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.189+0000 7f3c33e00640 1 --2- 172.21.15.40:0/2282977927 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3c340b4330 0x7f3c340ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.189+0000 7f3c33e00640 1 -- 172.21.15.40:0/2282977927 >> v1:172.21.15.53:6789/0 conn(0x7f3c340cea00 legacy=0x7f3c3406b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.189+0000 7f3c33e00640 1 -- 172.21.15.40:0/2282977927 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c340cf330 con 0x7f3c340b2360 2024-03-24T00:38:00.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.189+0000 7f3c2be00640 1 --2- 172.21.15.40:0/2282977927 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3c340b4330 0x7f3c340ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:00.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.189+0000 7f3c33e00640 1 --2- 172.21.15.40:0/2282977927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c340b2360 0x7f3c340b2740 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009870 tx=0x7f3c2401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0917636798dd827 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:00.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.189+0000 7f3c33400640 1 -- 172.21.15.40:0/2282977927 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c240ab610 con 0x7f3c340b2360 2024-03-24T00:38:00.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.189+0000 7f3c33400640 1 -- 172.21.15.40:0/2282977927 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c240b0070 con 0x7f3c340b2360 2024-03-24T00:38:00.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.189+0000 7f3c33400640 1 -- 172.21.15.40:0/2282977927 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c240ab9d0 con 0x7f3c340b2360 2024-03-24T00:38:00.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 -- 172.21.15.40:0/2282977927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c340b2360 msgr2=0x7f3c340b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 --2- 172.21.15.40:0/2282977927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c340b2360 0x7f3c340b2740 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3c24009870 tx=0x7f3c2401c410 comp rx=0 tx=0).stop 2024-03-24T00:38:00.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 -- 172.21.15.40:0/2282977927 shutdown_connections 2024-03-24T00:38:00.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 --2- 172.21.15.40:0/2282977927 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3c340b4330 0x7f3c340ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 --2- 172.21.15.40:0/2282977927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c340b2360 0x7f3c340b2740 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 -- 172.21.15.40:0/2282977927 >> 172.21.15.40:0/2282977927 conn(0x7f3c340b3810 msgr2=0x7f3c340b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:00.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 -- 172.21.15.40:0/2282977927 shutdown_connections 2024-03-24T00:38:00.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 -- 172.21.15.40:0/2282977927 wait complete. 2024-03-24T00:38:00.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 Processor -- start 2024-03-24T00:38:00.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 -- start start 2024-03-24T00:38:00.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c340b2360 0x7f3c3415a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3c340b4330 0x7f3c3415f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c340cea00 0x7f3c340d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c340d1950 con 0x7f3c340cea00 2024-03-24T00:38:00.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c340d17d0 con 0x7f3c340b2360 2024-03-24T00:38:00.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c3ac00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c3416ae10 con 0x7f3c340b4330 2024-03-24T00:38:00.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c38c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c340cea00 0x7f3c340d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c33e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c340b2360 0x7f3c3415a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c2be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3c340b4330 0x7f3c3415f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c38c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c340cea00 0x7f3c340d1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47938/0 (socket says 172.21.15.40:47938) 2024-03-24T00:38:00.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c38c00640 1 -- 172.21.15.40:0/3401127494 learned_addr learned my addr 172.21.15.40:0/3401127494 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:00.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c33e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c340b2360 0x7f3c3415a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39304/0 (socket says 172.21.15.40:39304) 2024-03-24T00:38:00.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c38c00640 1 -- 172.21.15.40:0/3401127494 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3c340b4330 msgr2=0x7f3c3415f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c38c00640 1 --2- 172.21.15.40:0/3401127494 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3c340b4330 0x7f3c3415f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c38c00640 1 -- 172.21.15.40:0/3401127494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c340b2360 msgr2=0x7f3c3415a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:00.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c38c00640 1 --2- 172.21.15.40:0/3401127494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c340b2360 0x7f3c3415a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:00.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.193+0000 7f3c38c00640 1 -- 172.21.15.40:0/3401127494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c340b51e0 con 0x7f3c340cea00 2024-03-24T00:38:00.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.197+0000 7f3c38c00640 1 --2- 172.21.15.40:0/3401127494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c340cea00 0x7f3c340d1b60 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00c7a0 tx=0x7f3c2c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:00.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.197+0000 7f3c30c00640 1 -- 172.21.15.40:0/3401127494 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c003ba0 con 0x7f3c340cea00 2024-03-24T00:38:00.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.197+0000 7f3c30c00640 1 -- 172.21.15.40:0/3401127494 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2c003d40 con 0x7f3c340cea00 2024-03-24T00:38:00.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.197+0000 7f3c30c00640 1 -- 172.21.15.40:0/3401127494 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2c00aa30 con 0x7f3c340cea00 2024-03-24T00:38:00.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.197+0000 7f3c3ac00640 1 -- 172.21.15.40:0/3401127494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c3415f960 con 0x7f3c340cea00 2024-03-24T00:38:00.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.197+0000 7f3c3ac00640 1 -- 172.21.15.40:0/3401127494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c3415ff20 con 0x7f3c340cea00 2024-03-24T00:38:00.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.197+0000 7f3c3ac00640 1 -- 172.21.15.40:0/3401127494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c34160120 con 0x7f3c340cea00 2024-03-24T00:38:00.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.197+0000 7f3c30c00640 1 -- 172.21.15.40:0/3401127494 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3c2c00abd0 con 0x7f3c340cea00 2024-03-24T00:38:00.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.197+0000 7f3c30c00640 1 --2- 172.21.15.40:0/3401127494 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3c10067690 0x7f3c10069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:00.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.197+0000 7f3c30c00640 1 -- 172.21.15.40:0/3401127494 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f3c2c085f00 con 0x7f3c340cea00 2024-03-24T00:38:00.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.197+0000 7f3c33e00640 1 --2- 172.21.15.40:0/3401127494 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3c10067690 0x7f3c10069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:00.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.201+0000 7f3c33e00640 1 --2- 172.21.15.40:0/3401127494 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3c10067690 0x7f3c10069b50 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3c240af000 tx=0x7f3c240a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:00.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.201+0000 7f3c30c00640 1 -- 172.21.15.40:0/3401127494 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3c34160120 con 0x7f3c340cea00 2024-03-24T00:38:00.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:00.365+0000 7f3c3ac00640 1 -- 172.21.15.40:0/3401127494 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3c3406b430 con 0x7f3c340cea00 2024-03-24T00:38:01.022 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.017+0000 7f3c30c00640 1 -- 172.21.15.40:0/3401127494 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v104) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3c2c056780 con 0x7f3c340cea00 2024-03-24T00:38:01.022 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-03-24T00:38:01.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.021+0000 7f3c3ac00640 1 -- 172.21.15.40:0/3401127494 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3c10067690 msgr2=0x7f3c10069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:01.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.021+0000 7f3c3ac00640 1 --2- 172.21.15.40:0/3401127494 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3c10067690 0x7f3c10069b50 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f3c240af000 tx=0x7f3c240a7000 comp rx=0 tx=0).stop 2024-03-24T00:38:01.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.021+0000 7f3c3ac00640 1 -- 172.21.15.40:0/3401127494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c340cea00 msgr2=0x7f3c340d1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:01.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.021+0000 7f3c3ac00640 1 --2- 172.21.15.40:0/3401127494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c340cea00 0x7f3c340d1b60 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c00c7a0 tx=0x7f3c2c00cc60 comp rx=0 tx=0).stop 2024-03-24T00:38:01.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.021+0000 7f3c3ac00640 1 -- 172.21.15.40:0/3401127494 shutdown_connections 2024-03-24T00:38:01.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.021+0000 7f3c3ac00640 1 --2- 172.21.15.40:0/3401127494 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3c10067690 0x7f3c10069b50 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:01.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.021+0000 7f3c3ac00640 1 --2- 172.21.15.40:0/3401127494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3c340cea00 0x7f3c340d1b60 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:01.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.021+0000 7f3c3ac00640 1 --2- 172.21.15.40:0/3401127494 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3c340b4330 0x7f3c3415f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:01.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.021+0000 7f3c3ac00640 1 --2- 172.21.15.40:0/3401127494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c340b2360 0x7f3c3415a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:01.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.021+0000 7f3c3ac00640 1 -- 172.21.15.40:0/3401127494 >> 172.21.15.40:0/3401127494 conn(0x7f3c340b3810 msgr2=0x7f3c340b3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:01.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.021+0000 7f3c3ac00640 1 -- 172.21.15.40:0/3401127494 shutdown_connections 2024-03-24T00:38:01.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.021+0000 7f3c3ac00640 1 -- 172.21.15.40:0/3401127494 wait complete. 2024-03-24T00:38:01.062 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:38:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febc0e00640 1 Processor -- start 2024-03-24T00:38:01.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febc0e00640 1 -- start start 2024-03-24T00:38:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febc0e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7febbc0d2270 0x7febbc070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febc0e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7febbc06c950 0x7febbc161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febc0e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc06d0f0 con 0x7febbc15df80 2024-03-24T00:38:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febc0e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febbc06b890 con 0x7febbc0d2270 2024-03-24T00:38:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febc0e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febbc06b280 con 0x7febbc06c950 2024-03-24T00:38:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febb3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7febbc0d2270 0x7febbc070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febb3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7febbc0d2270 0x7febbc070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47950/0 (socket says 172.21.15.40:47950) 2024-03-24T00:38:01.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febb3e00640 1 -- 172.21.15.40:0/1707933860 learned_addr learned my addr 172.21.15.40:0/1707933860 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:01.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febba800640 1 --2- 172.21.15.40:0/1707933860 >> v2:172.21.15.53:3300/0 conn(0x7febbc06c950 0x7febbc161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:01.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febb3e00640 1 -- 172.21.15.40:0/1707933860 >> v2:172.21.15.53:3300/0 conn(0x7febbc06c950 msgr2=0x7febbc161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:01.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febb3e00640 1 --2- 172.21.15.40:0/1707933860 >> v2:172.21.15.53:3300/0 conn(0x7febbc06c950 0x7febbc161ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:01.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febb3e00640 1 -- 172.21.15.40:0/1707933860 >> v1:172.21.15.53:6789/0 conn(0x7febbc15df80 legacy=0x7febbc0d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:01.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.205+0000 7febb3e00640 1 -- 172.21.15.40:0/1707933860 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc06ce10 con 0x7febbc0d2270 2024-03-24T00:38:01.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.209+0000 7febb3e00640 1 --2- 172.21.15.40:0/1707933860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febbc0d2270 0x7febbc070ff0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7febbc0ce480 tx=0x7feba401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ffca6eaa87babe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.209+0000 7febb9400640 1 -- 172.21.15.40:0/1707933860 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba40a7070 con 0x7febbc0d2270 2024-03-24T00:38:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.209+0000 7febb9400640 1 -- 172.21.15.40:0/1707933860 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba40b0070 con 0x7febbc0d2270 2024-03-24T00:38:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.209+0000 7febb9400640 1 -- 172.21.15.40:0/1707933860 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba40ab890 con 0x7febbc0d2270 2024-03-24T00:38:01.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.209+0000 7febc0e00640 1 -- 172.21.15.40:0/1707933860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febbc0d2270 msgr2=0x7febbc070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.209+0000 7febc0e00640 1 --2- 172.21.15.40:0/1707933860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febbc0d2270 0x7febbc070ff0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7febbc0ce480 tx=0x7feba401c4b0 comp rx=0 tx=0).stop 2024-03-24T00:38:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.209+0000 7febc0e00640 1 -- 172.21.15.40:0/1707933860 shutdown_connections 2024-03-24T00:38:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.209+0000 7febc0e00640 1 --2- 172.21.15.40:0/1707933860 >> v2:172.21.15.53:3300/0 conn(0x7febbc06c950 0x7febbc161ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.209+0000 7febc0e00640 1 --2- 172.21.15.40:0/1707933860 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febbc0d2270 0x7febbc070ff0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.209+0000 7febc0e00640 1 -- 172.21.15.40:0/1707933860 >> 172.21.15.40:0/1707933860 conn(0x7febbc06ecf0 msgr2=0x7febbc06f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.209+0000 7febc0e00640 1 -- 172.21.15.40:0/1707933860 shutdown_connections 2024-03-24T00:38:01.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.209+0000 7febc0e00640 1 -- 172.21.15.40:0/1707933860 wait complete. 2024-03-24T00:38:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febc0e00640 1 Processor -- start 2024-03-24T00:38:01.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febc0e00640 1 -- start start 2024-03-24T00:38:01.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febc0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febbc06c950 0x7febbc156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:01.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febc0e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febbc0d2270 0x7febbc159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febc0e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7febbc15df80 0x7febbc0b4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febc0e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febbc16af90 con 0x7febbc0d2270 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febc0e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febbc16ae10 con 0x7febbc06c950 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febc0e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febbc16b110 con 0x7febbc15df80 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febb3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febbc0d2270 0x7febbc159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febb3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febbc0d2270 0x7febbc159f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47958/0 (socket says 172.21.15.40:47958) 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febb3e00640 1 -- 172.21.15.40:0/2184105875 learned_addr learned my addr 172.21.15.40:0/2184105875 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febb3e00640 1 -- 172.21.15.40:0/2184105875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7febbc15df80 msgr2=0x7febbc0b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febb3e00640 1 --2- 172.21.15.40:0/2184105875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7febbc15df80 0x7febbc0b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febb3e00640 1 -- 172.21.15.40:0/2184105875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febbc06c950 msgr2=0x7febbc156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febb3e00640 1 --2- 172.21.15.40:0/2184105875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febbc06c950 0x7febbc156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febb3e00640 1 -- 172.21.15.40:0/2184105875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc156650 con 0x7febbc0d2270 2024-03-24T00:38:01.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febb3e00640 1 --2- 172.21.15.40:0/2184105875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febbc0d2270 0x7febbc159f20 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7feba40ad040 tx=0x7feba401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:01.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febb2000640 1 -- 172.21.15.40:0/2184105875 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba40be5f0 con 0x7febbc0d2270 2024-03-24T00:38:01.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febb2000640 1 -- 172.21.15.40:0/2184105875 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba40be790 con 0x7febbc0d2270 2024-03-24T00:38:01.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febb2000640 1 -- 172.21.15.40:0/2184105875 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba40beb60 con 0x7febbc0d2270 2024-03-24T00:38:01.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febc0e00640 1 -- 172.21.15.40:0/2184105875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febbc0b4b30 con 0x7febbc0d2270 2024-03-24T00:38:01.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febc0e00640 1 -- 172.21.15.40:0/2184105875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febbc1568e0 con 0x7febbc0d2270 2024-03-24T00:38:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.213+0000 7febc0e00640 1 -- 172.21.15.40:0/2184105875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febbc068170 con 0x7febbc0d2270 2024-03-24T00:38:01.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.217+0000 7febb2000640 1 -- 172.21.15.40:0/2184105875 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7feba40b43f0 con 0x7febbc0d2270 2024-03-24T00:38:01.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.217+0000 7febb2000640 1 --2- 172.21.15.40:0/2184105875 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb98067660 0x7feb98069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:01.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.217+0000 7febb2000640 1 -- 172.21.15.40:0/2184105875 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5609+0+0 (secure 0 0 0) 0x7feba41204b0 con 0x7febbc0d2270 2024-03-24T00:38:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.217+0000 7febb9e00640 1 --2- 172.21.15.40:0/2184105875 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb98067660 0x7feb98069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:01.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.217+0000 7febb9e00640 1 --2- 172.21.15.40:0/2184105875 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb98067660 0x7feb98069b20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7febac009670 tx=0x7febac006bd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:01.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.221+0000 7febb2000640 1 -- 172.21.15.40:0/2184105875 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7feba40160d0 con 0x7febbc0d2270 2024-03-24T00:38:01.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:01.417+0000 7febc0e00640 1 -- 172.21.15.40:0/2184105875 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7febbc0cf9d0 con 0x7febbc0d2270 2024-03-24T00:38:02.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.033+0000 7febb2000640 1 -- 172.21.15.40:0/2184105875 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v108) v1 ==== 174+0+0 (secure 0 0 0) 0x7feba40f0b20 con 0x7febbc0d2270 2024-03-24T00:38:02.039 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 16 and data pool 17 2024-03-24T00:38:02.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.045+0000 7febc0e00640 1 -- 172.21.15.40:0/2184105875 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb98067660 msgr2=0x7feb98069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.045+0000 7febc0e00640 1 --2- 172.21.15.40:0/2184105875 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb98067660 0x7feb98069b20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7febac009670 tx=0x7febac006bd0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.045+0000 7febc0e00640 1 -- 172.21.15.40:0/2184105875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febbc0d2270 msgr2=0x7febbc159f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.045+0000 7febc0e00640 1 --2- 172.21.15.40:0/2184105875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febbc0d2270 0x7febbc159f20 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7feba40ad040 tx=0x7feba401acf0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.045+0000 7febc0e00640 1 -- 172.21.15.40:0/2184105875 shutdown_connections 2024-03-24T00:38:02.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.045+0000 7febc0e00640 1 --2- 172.21.15.40:0/2184105875 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7feb98067660 0x7feb98069b20 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.045+0000 7febc0e00640 1 --2- 172.21.15.40:0/2184105875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7febbc15df80 0x7febbc0b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.045+0000 7febc0e00640 1 --2- 172.21.15.40:0/2184105875 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7febbc0d2270 0x7febbc159f20 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.045+0000 7febc0e00640 1 --2- 172.21.15.40:0/2184105875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febbc06c950 0x7febbc156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.045+0000 7febc0e00640 1 -- 172.21.15.40:0/2184105875 >> 172.21.15.40:0/2184105875 conn(0x7febbc06ecf0 msgr2=0x7febbc0b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:02.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.049+0000 7febc0e00640 1 -- 172.21.15.40:0/2184105875 shutdown_connections 2024-03-24T00:38:02.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.049+0000 7febc0e00640 1 -- 172.21.15.40:0/2184105875 wait complete. 2024-03-24T00:38:02.085 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:02.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf91a00640 1 Processor -- start 2024-03-24T00:38:02.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf91a00640 1 -- start start 2024-03-24T00:38:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf91a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf8c0b2910 0x7fbf8c0d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf91a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf8c0af170 0x7fbf8c165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf91a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf8c0b1e50 con 0x7fbf8c15f600 2024-03-24T00:38:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf91a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf8c0b3630 con 0x7fbf8c0b2910 2024-03-24T00:38:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf91a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c0b2d80 con 0x7fbf8c0af170 2024-03-24T00:38:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf8a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf8c0b2910 0x7fbf8c0d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf8a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf8c0b2910 0x7fbf8c0d2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47970/0 (socket says 172.21.15.40:47970) 2024-03-24T00:38:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf8a000640 1 -- 172.21.15.40:0/1245807649 learned_addr learned my addr 172.21.15.40:0/1245807649 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf8b400640 1 --2- 172.21.15.40:0/1245807649 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf8c0af170 0x7fbf8c165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf8a000640 1 -- 172.21.15.40:0/1245807649 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf8c0af170 msgr2=0x7fbf8c165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf8a000640 1 --2- 172.21.15.40:0/1245807649 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf8c0af170 0x7fbf8c165e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf8a000640 1 -- 172.21.15.40:0/1245807649 >> v1:172.21.15.53:6789/0 conn(0x7fbf8c15f600 legacy=0x7fbf8c0aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.225+0000 7fbf8a000640 1 -- 172.21.15.40:0/1245807649 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c0b20f0 con 0x7fbf8c0b2910 2024-03-24T00:38:02.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.229+0000 7fbf8a000640 1 --2- 172.21.15.40:0/1245807649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf8c0b2910 0x7fbf8c0d2830 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fbf78009870 tx=0x7fbf7801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=691ef00437c5213a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.229+0000 7fbf89600640 1 -- 172.21.15.40:0/1245807649 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf780b5070 con 0x7fbf8c0b2910 2024-03-24T00:38:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.229+0000 7fbf89600640 1 -- 172.21.15.40:0/1245807649 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf780a8a50 con 0x7fbf8c0b2910 2024-03-24T00:38:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.229+0000 7fbf89600640 1 -- 172.21.15.40:0/1245807649 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf780a8e20 con 0x7fbf8c0b2910 2024-03-24T00:38:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.229+0000 7fbf91a00640 1 -- 172.21.15.40:0/1245807649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf8c0b2910 msgr2=0x7fbf8c0d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.229+0000 7fbf91a00640 1 --2- 172.21.15.40:0/1245807649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf8c0b2910 0x7fbf8c0d2830 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fbf78009870 tx=0x7fbf7801ed50 comp rx=0 tx=0).stop 2024-03-24T00:38:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.229+0000 7fbf91a00640 1 -- 172.21.15.40:0/1245807649 shutdown_connections 2024-03-24T00:38:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.229+0000 7fbf91a00640 1 --2- 172.21.15.40:0/1245807649 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf8c0af170 0x7fbf8c165e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.229+0000 7fbf91a00640 1 --2- 172.21.15.40:0/1245807649 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf8c0b2910 0x7fbf8c0d2830 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.229+0000 7fbf91a00640 1 -- 172.21.15.40:0/1245807649 >> 172.21.15.40:0/1245807649 conn(0x7fbf8c0d1d10 msgr2=0x7fbf8c0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:02.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.229+0000 7fbf91a00640 1 -- 172.21.15.40:0/1245807649 shutdown_connections 2024-03-24T00:38:02.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.229+0000 7fbf91a00640 1 -- 172.21.15.40:0/1245807649 wait complete. 2024-03-24T00:38:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf91a00640 1 Processor -- start 2024-03-24T00:38:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf91a00640 1 -- start start 2024-03-24T00:38:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf91a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf8c0af170 0x7fbf8c1f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf91a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf8c0b2910 0x7fbf8c1f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf91a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf8c15f600 0x7fbf8c1ebd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf91a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c0b2fa0 con 0x7fbf8c0af170 2024-03-24T00:38:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf91a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c0b2740 con 0x7fbf8c15f600 2024-03-24T00:38:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf91a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8c15f9e0 con 0x7fbf8c0b2910 2024-03-24T00:38:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf8aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf8c0af170 0x7fbf8c1f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf8aa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf8c0af170 0x7fbf8c1f5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47980/0 (socket says 172.21.15.40:47980) 2024-03-24T00:38:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf8aa00640 1 -- 172.21.15.40:0/938895645 learned_addr learned my addr 172.21.15.40:0/938895645 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf8a000640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf8c0b2910 0x7fbf8c1f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf8b400640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf8c15f600 0x7fbf8c1ebd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf8aa00640 1 -- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf8c0b2910 msgr2=0x7fbf8c1f1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf8aa00640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf8c0b2910 0x7fbf8c1f1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf8aa00640 1 -- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf8c15f600 msgr2=0x7fbf8c1ebd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf8aa00640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf8c15f600 0x7fbf8c1ebd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf8aa00640 1 -- 172.21.15.40:0/938895645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c1f2430 con 0x7fbf8c0af170 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf8b400640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf8c15f600 0x7fbf8c1ebd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf8aa00640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf8c0af170 0x7fbf8c1f5f60 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00c420 tx=0x7fbf7c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf82a00640 1 -- 172.21.15.40:0/938895645 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c0052d0 con 0x7fbf8c0af170 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf82a00640 1 -- 172.21.15.40:0/938895645 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7c005470 con 0x7fbf8c0af170 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf82a00640 1 -- 172.21.15.40:0/938895645 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c007260 con 0x7fbf8c0af170 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf91a00640 1 -- 172.21.15.40:0/938895645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8c1f26f0 con 0x7fbf8c0af170 2024-03-24T00:38:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf91a00640 1 -- 172.21.15.40:0/938895645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf8c1f2cb0 con 0x7fbf8c0af170 2024-03-24T00:38:02.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.233+0000 7fbf91a00640 1 -- 172.21.15.40:0/938895645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8c068170 con 0x7fbf8c0af170 2024-03-24T00:38:02.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.237+0000 7fbf82a00640 1 -- 172.21.15.40:0/938895645 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbf7c007400 con 0x7fbf8c0af170 2024-03-24T00:38:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.237+0000 7fbf82a00640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf6c067660 0x7fbf6c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.237+0000 7fbf82a00640 1 -- 172.21.15.40:0/938895645 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf7c084d50 con 0x7fbf8c0af170 2024-03-24T00:38:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.237+0000 7fbf8a000640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf6c067660 0x7fbf6c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.237+0000 7fbf8a000640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf6c067660 0x7fbf6c069b20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbf780b2000 tx=0x7fbf78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:02.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.241+0000 7fbf82a00640 1 -- 172.21.15.40:0/938895645 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbf7c0552e0 con 0x7fbf8c0af170 2024-03-24T00:38:02.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.405+0000 7fbf91a00640 1 -- 172.21.15.40:0/938895645 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf8c15ee30 con 0x7fbf8c0af170 2024-03-24T00:38:02.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.405+0000 7fbf82a00640 1 -- 172.21.15.40:0/938895645 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16580 (secure 0 0 0) 0x7fbf7c059380 con 0x7fbf8c0af170 2024-03-24T00:38:02.411 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:02.411 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":105,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:38:01.424529+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:37:59.849300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:00.417342+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.409+0000 7fbf91a00640 1 -- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf6c067660 msgr2=0x7fbf6c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.409+0000 7fbf91a00640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf6c067660 0x7fbf6c069b20 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fbf780b2000 tx=0x7fbf78009040 comp rx=0 tx=0).stop 2024-03-24T00:38:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.409+0000 7fbf91a00640 1 -- 172.21.15.40:0/938895645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf8c0af170 msgr2=0x7fbf8c1f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.409+0000 7fbf91a00640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf8c0af170 0x7fbf8c1f5f60 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00c420 tx=0x7fbf7c00c8e0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.409+0000 7fbf91a00640 1 -- 172.21.15.40:0/938895645 shutdown_connections 2024-03-24T00:38:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.409+0000 7fbf91a00640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf6c067660 0x7fbf6c069b20 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.409+0000 7fbf91a00640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf8c15f600 0x7fbf8c1ebd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.409+0000 7fbf91a00640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf8c0b2910 0x7fbf8c1f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.409+0000 7fbf91a00640 1 --2- 172.21.15.40:0/938895645 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf8c0af170 0x7fbf8c1f5f60 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.409+0000 7fbf91a00640 1 -- 172.21.15.40:0/938895645 >> 172.21.15.40:0/938895645 conn(0x7fbf8c0d1d10 msgr2=0x7fbf8c0d20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:02.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.409+0000 7fbf91a00640 1 -- 172.21.15.40:0/938895645 shutdown_connections 2024-03-24T00:38:02.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.409+0000 7fbf91a00640 1 -- 172.21.15.40:0/938895645 wait complete. 2024-03-24T00:38:02.434 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:02.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.569+0000 7f3a9c200640 1 Processor -- start 2024-03-24T00:38:02.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.569+0000 7f3a9c200640 1 -- start start 2024-03-24T00:38:02.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a9c200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a9406e620 0x7f3a940b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a9c200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3a9406ea00 0x7f3a940b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a9c200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3a940b2ff0 0x7f3a940ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a9c200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a940b4c00 con 0x7f3a9406ea00 2024-03-24T00:38:02.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a9c200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a94070f60 con 0x7f3a940b2ff0 2024-03-24T00:38:02.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a9c200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a940b3620 con 0x7f3a9406e620 2024-03-24T00:38:02.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a98e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3a9406ea00 0x7f3a940b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a98e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3a9406ea00 0x7f3a940b2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47986/0 (socket says 172.21.15.40:47986) 2024-03-24T00:38:02.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a98e00640 1 -- 172.21.15.40:0/1516010991 learned_addr learned my addr 172.21.15.40:0/1516010991 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:02.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a99800640 1 --2- 172.21.15.40:0/1516010991 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a9406e620 0x7f3a940b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a9a200640 1 --2- 172.21.15.40:0/1516010991 >> v2:172.21.15.53:3300/0 conn(0x7f3a940b2ff0 0x7f3a940ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a98e00640 1 -- 172.21.15.40:0/1516010991 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a9406e620 msgr2=0x7f3a940b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:02.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a98e00640 1 --2- 172.21.15.40:0/1516010991 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a9406e620 0x7f3a940b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a98e00640 1 -- 172.21.15.40:0/1516010991 >> v2:172.21.15.53:3300/0 conn(0x7f3a940b2ff0 msgr2=0x7f3a940ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:02.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a98e00640 1 --2- 172.21.15.40:0/1516010991 >> v2:172.21.15.53:3300/0 conn(0x7f3a940b2ff0 0x7f3a940ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.577 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a98e00640 1 -- 172.21.15.40:0/1516010991 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a940b3900 con 0x7f3a9406ea00 2024-03-24T00:38:02.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.573+0000 7f3a98e00640 1 --2- 172.21.15.40:0/1516010991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a9406ea00 0x7f3a940b2ab0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f3a9406cea0 tx=0x7f3a8001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459c814cb568d39d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a93e00640 1 -- 172.21.15.40:0/1516010991 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a800b0070 con 0x7f3a9406ea00 2024-03-24T00:38:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a93e00640 1 -- 172.21.15.40:0/1516010991 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a800ab600 con 0x7f3a9406ea00 2024-03-24T00:38:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a93e00640 1 -- 172.21.15.40:0/1516010991 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a940cdcf0 con 0x7f3a9406ea00 2024-03-24T00:38:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 -- 172.21.15.40:0/1516010991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a9406ea00 msgr2=0x7f3a940b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 --2- 172.21.15.40:0/1516010991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a9406ea00 0x7f3a940b2ab0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f3a9406cea0 tx=0x7f3a8001c490 comp rx=0 tx=0).stop 2024-03-24T00:38:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 -- 172.21.15.40:0/1516010991 shutdown_connections 2024-03-24T00:38:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 --2- 172.21.15.40:0/1516010991 >> v2:172.21.15.53:3300/0 conn(0x7f3a940b2ff0 0x7f3a940ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 --2- 172.21.15.40:0/1516010991 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a9406ea00 0x7f3a940b2ab0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 --2- 172.21.15.40:0/1516010991 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a9406e620 0x7f3a940b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 -- 172.21.15.40:0/1516010991 >> 172.21.15.40:0/1516010991 conn(0x7f3a940b17b0 msgr2=0x7f3a940b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 -- 172.21.15.40:0/1516010991 shutdown_connections 2024-03-24T00:38:02.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 -- 172.21.15.40:0/1516010991 wait complete. 2024-03-24T00:38:02.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 Processor -- start 2024-03-24T00:38:02.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 -- start start 2024-03-24T00:38:02.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9406e620 0x7f3a9413add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a9406ea00 0x7f3a94137e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a940b2ff0 0x7f3a941383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a94144920 con 0x7f3a940b2ff0 2024-03-24T00:38:02.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9406ee60 con 0x7f3a9406e620 2024-03-24T00:38:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9c200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9415cf50 con 0x7f3a9406ea00 2024-03-24T00:38:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a940b2ff0 0x7f3a941383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9a200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a940b2ff0 0x7f3a941383c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:47996/0 (socket says 172.21.15.40:47996) 2024-03-24T00:38:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a9a200640 1 -- 172.21.15.40:0/35496546 learned_addr learned my addr 172.21.15.40:0/35496546 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a99800640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9406e620 0x7f3a9413add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.577+0000 7f3a98e00640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a9406ea00 0x7f3a94137e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a9a200640 1 -- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a9406ea00 msgr2=0x7f3a94137e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a9a200640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a9406ea00 0x7f3a94137e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a9a200640 1 -- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9406e620 msgr2=0x7f3a9413add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a9a200640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9406e620 0x7f3a9413add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a9a200640 1 -- 172.21.15.40:0/35496546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9415d230 con 0x7f3a940b2ff0 2024-03-24T00:38:02.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a9a200640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a940b2ff0 0x7f3a941383c0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00d800 tx=0x7f3a8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a91600640 1 -- 172.21.15.40:0/35496546 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c00b6c0 con 0x7f3a940b2ff0 2024-03-24T00:38:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a91600640 1 -- 172.21.15.40:0/35496546 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8c019070 con 0x7f3a940b2ff0 2024-03-24T00:38:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a91600640 1 -- 172.21.15.40:0/35496546 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c00baa0 con 0x7f3a940b2ff0 2024-03-24T00:38:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a9c200640 1 -- 172.21.15.40:0/35496546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9415d510 con 0x7f3a940b2ff0 2024-03-24T00:38:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a9c200640 1 -- 172.21.15.40:0/35496546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9415c340 con 0x7f3a940b2ff0 2024-03-24T00:38:02.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a9c200640 1 -- 172.21.15.40:0/35496546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a94068170 con 0x7f3a940b2ff0 2024-03-24T00:38:02.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a91600640 1 -- 172.21.15.40:0/35496546 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3a8c00bc40 con 0x7f3a940b2ff0 2024-03-24T00:38:02.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a91600640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a74067660 0x7f3a74069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a91600640 1 -- 172.21.15.40:0/35496546 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a8c085bd0 con 0x7f3a940b2ff0 2024-03-24T00:38:02.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.581+0000 7f3a99800640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a74067660 0x7f3a74069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.585+0000 7f3a99800640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a74067660 0x7f3a74069b20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f3a880039f0 tx=0x7f3a88009790 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:02.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.585+0000 7f3a91600640 1 -- 172.21.15.40:0/35496546 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3a8c0561f0 con 0x7f3a940b2ff0 2024-03-24T00:38:02.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.749+0000 7f3a9c200640 1 -- 172.21.15.40:0/35496546 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a94072f50 con 0x7f3a940b2ff0 2024-03-24T00:38:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.749+0000 7f3a91600640 1 -- 172.21.15.40:0/35496546 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16580 (secure 0 0 0) 0x7f3a8c05c030 con 0x7f3a940b2ff0 2024-03-24T00:38:02.754 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:02.754 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":105,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:38:01.424529+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:37:59.849300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:00.417342+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:02.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.753+0000 7f3a9c200640 1 -- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a74067660 msgr2=0x7f3a74069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.753+0000 7f3a9c200640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a74067660 0x7f3a74069b20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f3a880039f0 tx=0x7f3a88009790 comp rx=0 tx=0).stop 2024-03-24T00:38:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.753+0000 7f3a9c200640 1 -- 172.21.15.40:0/35496546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a940b2ff0 msgr2=0x7f3a941383c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.753+0000 7f3a9c200640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a940b2ff0 0x7f3a941383c0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00d800 tx=0x7f3a8c00dcc0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.753+0000 7f3a9c200640 1 -- 172.21.15.40:0/35496546 shutdown_connections 2024-03-24T00:38:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.753+0000 7f3a9c200640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3a74067660 0x7f3a74069b20 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.753+0000 7f3a9c200640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3a940b2ff0 0x7f3a941383c0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.753+0000 7f3a9c200640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3a9406ea00 0x7f3a94137e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.753+0000 7f3a9c200640 1 --2- 172.21.15.40:0/35496546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9406e620 0x7f3a9413add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.753+0000 7f3a9c200640 1 -- 172.21.15.40:0/35496546 >> 172.21.15.40:0/35496546 conn(0x7f3a940b17b0 msgr2=0x7f3a940b0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.753+0000 7f3a9c200640 1 -- 172.21.15.40:0/35496546 shutdown_connections 2024-03-24T00:38:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.753+0000 7f3a9c200640 1 -- 172.21.15.40:0/35496546 wait complete. 2024-03-24T00:38:02.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:38:02.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55d1c00640 1 Processor -- start 2024-03-24T00:38:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55d1c00640 1 -- start start 2024-03-24T00:38:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55d1c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f55cc0afa60 0x7f55cc074420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55d1c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55cc073360 0x7f55cc0aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55d1c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55cc0b2df0 con 0x7f55cc15c0e0 2024-03-24T00:38:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55d1c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55cc0cdb80 con 0x7f55cc0afa60 2024-03-24T00:38:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55d1c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0cd8a0 con 0x7f55cc073360 2024-03-24T00:38:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55ca000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f55cc0afa60 0x7f55cc074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55ca000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f55cc0afa60 0x7f55cc074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39352/0 (socket says 172.21.15.40:39352) 2024-03-24T00:38:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55ca000640 1 -- 172.21.15.40:0/3376765927 learned_addr learned my addr 172.21.15.40:0/3376765927 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55cb400640 1 --2- 172.21.15.40:0/3376765927 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55cc073360 0x7f55cc0aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55ca000640 1 -- 172.21.15.40:0/3376765927 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55cc073360 msgr2=0x7f55cc0aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55ca000640 1 --2- 172.21.15.40:0/3376765927 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55cc073360 0x7f55cc0aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55ca000640 1 -- 172.21.15.40:0/3376765927 >> v1:172.21.15.53:6789/0 conn(0x7f55cc15c0e0 legacy=0x7f55cc0ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.917+0000 7f55ca000640 1 -- 172.21.15.40:0/3376765927 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc0add20 con 0x7f55cc0afa60 2024-03-24T00:38:02.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.921+0000 7f55ca000640 1 --2- 172.21.15.40:0/3376765927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55cc0afa60 0x7f55cc074420 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009870 tx=0x7f55b801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a87a96126e3d92db server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:02.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.921+0000 7f55c9600640 1 -- 172.21.15.40:0/3376765927 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b80ab430 con 0x7f55cc0afa60 2024-03-24T00:38:02.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.921+0000 7f55c9600640 1 -- 172.21.15.40:0/3376765927 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55b80b0070 con 0x7f55cc0afa60 2024-03-24T00:38:02.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.921+0000 7f55c9600640 1 -- 172.21.15.40:0/3376765927 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b80ab7f0 con 0x7f55cc0afa60 2024-03-24T00:38:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.921+0000 7f55d1c00640 1 -- 172.21.15.40:0/3376765927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55cc0afa60 msgr2=0x7f55cc074420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.921+0000 7f55d1c00640 1 --2- 172.21.15.40:0/3376765927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55cc0afa60 0x7f55cc074420 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f55b8009870 tx=0x7f55b801c410 comp rx=0 tx=0).stop 2024-03-24T00:38:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.921+0000 7f55d1c00640 1 -- 172.21.15.40:0/3376765927 shutdown_connections 2024-03-24T00:38:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.921+0000 7f55d1c00640 1 --2- 172.21.15.40:0/3376765927 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55cc073360 0x7f55cc0aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.921+0000 7f55d1c00640 1 --2- 172.21.15.40:0/3376765927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55cc0afa60 0x7f55cc074420 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.921+0000 7f55d1c00640 1 -- 172.21.15.40:0/3376765927 >> 172.21.15.40:0/3376765927 conn(0x7f55cc072870 msgr2=0x7f55cc072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.921+0000 7f55d1c00640 1 -- 172.21.15.40:0/3376765927 shutdown_connections 2024-03-24T00:38:02.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.921+0000 7f55d1c00640 1 -- 172.21.15.40:0/3376765927 wait complete. 2024-03-24T00:38:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55d1c00640 1 Processor -- start 2024-03-24T00:38:02.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55d1c00640 1 -- start start 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55d1c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55cc073360 0x7f55cc0cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55d1c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55cc0afa60 0x7f55cc1466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55d1c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55cc15c0e0 0x7f55cc0c9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55d1c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0cd550 con 0x7f55cc0afa60 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55d1c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc0cd3d0 con 0x7f55cc073360 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55d1c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55cc168aa0 con 0x7f55cc15c0e0 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55ca000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55cc0afa60 0x7f55cc1466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55ca000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55cc0afa60 0x7f55cc1466b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48006/0 (socket says 172.21.15.40:48006) 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55ca000640 1 -- 172.21.15.40:0/2261718582 learned_addr learned my addr 172.21.15.40:0/2261718582 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55cb400640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55cc15c0e0 0x7f55cc0c9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55caa00640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55cc073360 0x7f55cc0cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55ca000640 1 -- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55cc15c0e0 msgr2=0x7f55cc0c9580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55ca000640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55cc15c0e0 0x7f55cc0c9580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55ca000640 1 -- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55cc073360 msgr2=0x7f55cc0cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55ca000640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55cc073360 0x7f55cc0cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55ca000640 1 -- 172.21.15.40:0/2261718582 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc0ce5e0 con 0x7f55cc0afa60 2024-03-24T00:38:02.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55ca000640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55cc0afa60 0x7f55cc1466b0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f55b801aac0 tx=0x7f55b80191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55c2a00640 1 -- 172.21.15.40:0/2261718582 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b80b9020 con 0x7f55cc0afa60 2024-03-24T00:38:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55c2a00640 1 -- 172.21.15.40:0/2261718582 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc0af380 con 0x7f55cc0afa60 2024-03-24T00:38:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55c2a00640 1 -- 172.21.15.40:0/2261718582 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55b80b4420 con 0x7f55cc0afa60 2024-03-24T00:38:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55d1c00640 1 -- 172.21.15.40:0/2261718582 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55cc15a460 con 0x7f55cc0afa60 2024-03-24T00:38:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55d1c00640 1 -- 172.21.15.40:0/2261718582 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55cc0ce340 con 0x7f55cc0afa60 2024-03-24T00:38:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.925+0000 7f55d1c00640 1 -- 172.21.15.40:0/2261718582 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55cc068170 con 0x7f55cc0afa60 2024-03-24T00:38:02.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.929+0000 7f55c2a00640 1 -- 172.21.15.40:0/2261718582 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f55b80bc020 con 0x7f55cc0afa60 2024-03-24T00:38:02.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.929+0000 7f55c2a00640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f55ac067690 0x7f55ac069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:02.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.929+0000 7f55c2a00640 1 -- 172.21.15.40:0/2261718582 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55b811f4e0 con 0x7f55cc0afa60 2024-03-24T00:38:02.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.929+0000 7f55caa00640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f55ac067690 0x7f55ac069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:02.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.929+0000 7f55caa00640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f55ac067690 0x7f55ac069b50 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f55bc0015b0 tx=0x7f55bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:02.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:02.933+0000 7f55c2a00640 1 -- 172.21.15.40:0/2261718582 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f55b80c3030 con 0x7f55cc0afa60 2024-03-24T00:38:03.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:03.133+0000 7f55d1c00640 1 -- 172.21.15.40:0/2261718582 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f55cc153c50 con 0x7f55cc0afa60 2024-03-24T00:38:04.014 INFO:tasks.ceph.mds.b.smithi053.stderr:2024-03-24T00:38:04.011+0000 7f852ec00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:38:04.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.041+0000 7f55c2a00640 1 -- 172.21.15.40:0/2261718582 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v111) v1 ==== 118+0+0 (secure 0 0 0) 0x7f55b8016150 con 0x7f55cc0afa60 2024-03-24T00:38:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.045+0000 7f55d1c00640 1 -- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f55ac067690 msgr2=0x7f55ac069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.045+0000 7f55d1c00640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f55ac067690 0x7f55ac069b50 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f55bc0015b0 tx=0x7f55bc009040 comp rx=0 tx=0).stop 2024-03-24T00:38:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.045+0000 7f55d1c00640 1 -- 172.21.15.40:0/2261718582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55cc0afa60 msgr2=0x7f55cc1466b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.045+0000 7f55d1c00640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55cc0afa60 0x7f55cc1466b0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f55b801aac0 tx=0x7f55b80191b0 comp rx=0 tx=0).stop 2024-03-24T00:38:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.045+0000 7f55d1c00640 1 -- 172.21.15.40:0/2261718582 shutdown_connections 2024-03-24T00:38:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.045+0000 7f55d1c00640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f55ac067690 0x7f55ac069b50 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.045+0000 7f55d1c00640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f55cc15c0e0 0x7f55cc0c9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.045+0000 7f55d1c00640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f55cc0afa60 0x7f55cc1466b0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.045+0000 7f55d1c00640 1 --2- 172.21.15.40:0/2261718582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55cc073360 0x7f55cc0cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.045+0000 7f55d1c00640 1 -- 172.21.15.40:0/2261718582 >> 172.21.15.40:0/2261718582 conn(0x7f55cc072870 msgr2=0x7f55cc0ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.045+0000 7f55d1c00640 1 -- 172.21.15.40:0/2261718582 shutdown_connections 2024-03-24T00:38:04.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.045+0000 7f55d1c00640 1 -- 172.21.15.40:0/2261718582 wait complete. 2024-03-24T00:38:04.072 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:38:04.073 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:38:04.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5833200640 1 Processor -- start 2024-03-24T00:38:04.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5833200640 1 -- start start 2024-03-24T00:38:04.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5833200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f58341443e0 0x7f58340b64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:04.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5833200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f583406eb00 0x7f583406ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:04.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5833200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f583415e600 0x7f583415ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:04.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5833200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58340cd790 con 0x7f58341443e0 2024-03-24T00:38:04.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5833200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583406f470 con 0x7f583406eb00 2024-03-24T00:38:04.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5833200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58341447c0 con 0x7f583415e600 2024-03-24T00:38:04.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5831e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f58341443e0 0x7f58340b64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:04.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5831400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f583406eb00 0x7f583406ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:04.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5832800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f583415e600 0x7f583415ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5831400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f583406eb00 0x7f583406ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39380/0 (socket says 172.21.15.40:39380) 2024-03-24T00:38:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5831e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f58341443e0 0x7f58340b64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48018/0 (socket says 172.21.15.40:48018) 2024-03-24T00:38:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.217+0000 7f5831400640 1 -- 172.21.15.40:0/362400797 learned_addr learned my addr 172.21.15.40:0/362400797 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.221+0000 7f5831e00640 1 -- 172.21.15.40:0/362400797 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f583415e600 msgr2=0x7f583415ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.221+0000 7f5831e00640 1 --2- 172.21.15.40:0/362400797 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f583415e600 0x7f583415ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:04.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.221+0000 7f5831e00640 1 -- 172.21.15.40:0/362400797 >> v2:172.21.15.53:3300/0 conn(0x7f583406eb00 msgr2=0x7f583406ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:04.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.221+0000 7f5831e00640 1 --2- 172.21.15.40:0/362400797 >> v2:172.21.15.53:3300/0 conn(0x7f583406eb00 0x7f583406ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:04.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.221+0000 7f5831e00640 1 -- 172.21.15.40:0/362400797 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58340cf200 con 0x7f58341443e0 2024-03-24T00:38:04.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.221+0000 7f5831e00640 1 --2- 172.21.15.40:0/362400797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58341443e0 0x7f58340b64d0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f5834072610 tx=0x7f582401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c5dc80b509d46e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.221+0000 7f5830a00640 1 -- 172.21.15.40:0/362400797 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58240b1070 con 0x7f58341443e0 2024-03-24T00:38:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.221+0000 7f5830a00640 1 -- 172.21.15.40:0/362400797 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58240ac640 con 0x7f58341443e0 2024-03-24T00:38:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.221+0000 7f5830a00640 1 -- 172.21.15.40:0/362400797 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58240aca10 con 0x7f58341443e0 2024-03-24T00:38:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.221+0000 7f5833200640 1 -- 172.21.15.40:0/362400797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58341443e0 msgr2=0x7f58340b64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.221+0000 7f5833200640 1 --2- 172.21.15.40:0/362400797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58341443e0 0x7f58340b64d0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f5834072610 tx=0x7f582401c460 comp rx=0 tx=0).stop 2024-03-24T00:38:04.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 -- 172.21.15.40:0/362400797 shutdown_connections 2024-03-24T00:38:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 --2- 172.21.15.40:0/362400797 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f583415e600 0x7f583415ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 --2- 172.21.15.40:0/362400797 >> v2:172.21.15.53:3300/0 conn(0x7f583406eb00 0x7f583406ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 --2- 172.21.15.40:0/362400797 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f58341443e0 0x7f58340b64d0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 -- 172.21.15.40:0/362400797 >> 172.21.15.40:0/362400797 conn(0x7f5834071a40 msgr2=0x7f5834071e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 -- 172.21.15.40:0/362400797 shutdown_connections 2024-03-24T00:38:04.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 -- 172.21.15.40:0/362400797 wait complete. 2024-03-24T00:38:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 Processor -- start 2024-03-24T00:38:04.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 -- start start 2024-03-24T00:38:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f583406eb00 0x7f583413fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f58341443e0 0x7f58341f7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f583415e600 0x7f58341434e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583415f3c0 con 0x7f583406eb00 2024-03-24T00:38:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58340cfd80 con 0x7f583415e600 2024-03-24T00:38:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58340b6a10 con 0x7f58341443e0 2024-03-24T00:38:04.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5831e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f583406eb00 0x7f583413fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5831e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f583406eb00 0x7f583413fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48020/0 (socket says 172.21.15.40:48020) 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5831e00640 1 -- 172.21.15.40:0/960547744 learned_addr learned my addr 172.21.15.40:0/960547744 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5831400640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f58341443e0 0x7f58341f7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5832800640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f583415e600 0x7f58341434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5831e00640 1 -- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f58341443e0 msgr2=0x7f58341f7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5831e00640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f58341443e0 0x7f58341f7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5831e00640 1 -- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f583415e600 msgr2=0x7f58341434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5831e00640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f583415e600 0x7f58341434e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5831e00640 1 -- 172.21.15.40:0/960547744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5834140140 con 0x7f583406eb00 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5831e00640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f583406eb00 0x7f583413fa40 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f582401ab10 tx=0x7f5824019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f582a000640 1 -- 172.21.15.40:0/960547744 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58240bb070 con 0x7f583406eb00 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f582a000640 1 -- 172.21.15.40:0/960547744 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58240acb50 con 0x7f583406eb00 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.225+0000 7f5833200640 1 -- 172.21.15.40:0/960547744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5834140420 con 0x7f583406eb00 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.229+0000 7f582a000640 1 -- 172.21.15.40:0/960547744 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58240b1040 con 0x7f583406eb00 2024-03-24T00:38:04.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.229+0000 7f5833200640 1 -- 172.21.15.40:0/960547744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58341409e0 con 0x7f583406eb00 2024-03-24T00:38:04.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.229+0000 7f5833200640 1 -- 172.21.15.40:0/960547744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583404f410 con 0x7f583406eb00 2024-03-24T00:38:04.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.229+0000 7f582a000640 1 -- 172.21.15.40:0/960547744 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f58240be070 con 0x7f583406eb00 2024-03-24T00:38:04.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.229+0000 7f582a000640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5814067660 0x7f5814069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:04.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.229+0000 7f582a000640 1 -- 172.21.15.40:0/960547744 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f582411e420 con 0x7f583406eb00 2024-03-24T00:38:04.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.229+0000 7f5831400640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5814067660 0x7f5814069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:04.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.229+0000 7f5831400640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5814067660 0x7f5814069b20 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5834160010 tx=0x7f5820009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:04.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.233+0000 7f582a000640 1 -- 172.21.15.40:0/960547744 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5824016050 con 0x7f583406eb00 2024-03-24T00:38:04.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:04.429+0000 7f5833200640 1 -- 172.21.15.40:0/960547744 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f58340b6570 con 0x7f583406eb00 2024-03-24T00:38:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.041+0000 7f582a000640 1 -- 172.21.15.40:0/960547744 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v112) v1 ==== 122+0+0 (secure 0 0 0) 0x7f58240ee9b0 con 0x7f583406eb00 2024-03-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.045+0000 7f5833200640 1 -- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5814067660 msgr2=0x7f5814069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.045+0000 7f5833200640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5814067660 0x7f5814069b20 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f5834160010 tx=0x7f5820009240 comp rx=0 tx=0).stop 2024-03-24T00:38:05.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.045+0000 7f5833200640 1 -- 172.21.15.40:0/960547744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f583406eb00 msgr2=0x7f583413fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.045+0000 7f5833200640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f583406eb00 0x7f583413fa40 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f582401ab10 tx=0x7f5824019200 comp rx=0 tx=0).stop 2024-03-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.045+0000 7f5833200640 1 -- 172.21.15.40:0/960547744 shutdown_connections 2024-03-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.045+0000 7f5833200640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5814067660 0x7f5814069b20 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.045+0000 7f5833200640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f583415e600 0x7f58341434e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.045+0000 7f5833200640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f58341443e0 0x7f58341f7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.045+0000 7f5833200640 1 --2- 172.21.15.40:0/960547744 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f583406eb00 0x7f583413fa40 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.045+0000 7f5833200640 1 -- 172.21.15.40:0/960547744 >> 172.21.15.40:0/960547744 conn(0x7f5834071a40 msgr2=0x7f583415a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.045+0000 7f5833200640 1 -- 172.21.15.40:0/960547744 shutdown_connections 2024-03-24T00:38:05.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.045+0000 7f5833200640 1 -- 172.21.15.40:0/960547744 wait complete. 2024-03-24T00:38:05.074 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:38:05.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:05.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.213+0000 7fb982c00640 1 Processor -- start 2024-03-24T00:38:05.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.213+0000 7fb982c00640 1 -- start start 2024-03-24T00:38:05.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.213+0000 7fb982c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb97c1477a0 0x7fb97c0725a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.213+0000 7fb982c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb97c15f050 0x7fb97c06f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.213+0000 7fb982c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb97c0b54d0 0x7fb97c0b5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.213+0000 7fb982c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97c15ebc0 con 0x7fb97c1477a0 2024-03-24T00:38:05.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.213+0000 7fb982c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97c15f4b0 con 0x7fb97c0b54d0 2024-03-24T00:38:05.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.213+0000 7fb982c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c072c10 con 0x7fb97c15f050 2024-03-24T00:38:05.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb97be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb97c1477a0 0x7fb97c0725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb973e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb97c15f050 0x7fb97c06f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb980c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb97c0b54d0 0x7fb97c0b5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb97be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fb97c1477a0 0x7fb97c0725a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48022/0 (socket says 172.21.15.40:48022) 2024-03-24T00:38:05.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb973e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb97c15f050 0x7fb97c06f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:33096/0 (socket says 172.21.15.40:33096) 2024-03-24T00:38:05.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb973e00640 1 -- 172.21.15.40:0/3901328088 learned_addr learned my addr 172.21.15.40:0/3901328088 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:05.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb980c00640 1 -- 172.21.15.40:0/3901328088 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb97c15f050 msgr2=0x7fb97c06f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:05.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb980c00640 1 --2- 172.21.15.40:0/3901328088 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb97c15f050 0x7fb97c06f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb980c00640 1 -- 172.21.15.40:0/3901328088 >> v2:172.21.15.40:3300/0 conn(0x7fb97c1477a0 msgr2=0x7fb97c0725a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:05.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb980c00640 1 --2- 172.21.15.40:0/3901328088 >> v2:172.21.15.40:3300/0 conn(0x7fb97c1477a0 0x7fb97c0725a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb980c00640 1 -- 172.21.15.40:0/3901328088 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c072ef0 con 0x7fb97c0b54d0 2024-03-24T00:38:05.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb97be00640 1 --2- 172.21.15.40:0/3901328088 >> v2:172.21.15.40:3300/0 conn(0x7fb97c1477a0 0x7fb97c0725a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:05.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb980c00640 1 --2- 172.21.15.40:0/3901328088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb97c0b54d0 0x7fb97c0b5920 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fb97c073200 tx=0x7fb97401c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31c7c2329c1eb075 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:05.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb97b400640 1 -- 172.21.15.40:0/3901328088 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9740b2070 con 0x7fb97c0b54d0 2024-03-24T00:38:05.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb97b400640 1 -- 172.21.15.40:0/3901328088 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9740a58f0 con 0x7fb97c0b54d0 2024-03-24T00:38:05.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.217+0000 7fb97b400640 1 -- 172.21.15.40:0/3901328088 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9740a5cc0 con 0x7fb97c0b54d0 2024-03-24T00:38:05.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 -- 172.21.15.40:0/3901328088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb97c0b54d0 msgr2=0x7fb97c0b5920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 --2- 172.21.15.40:0/3901328088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb97c0b54d0 0x7fb97c0b5920 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fb97c073200 tx=0x7fb97401c870 comp rx=0 tx=0).stop 2024-03-24T00:38:05.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 -- 172.21.15.40:0/3901328088 shutdown_connections 2024-03-24T00:38:05.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 --2- 172.21.15.40:0/3901328088 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb97c0b54d0 0x7fb97c0b5920 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 --2- 172.21.15.40:0/3901328088 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb97c15f050 0x7fb97c06f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 --2- 172.21.15.40:0/3901328088 >> v2:172.21.15.40:3300/0 conn(0x7fb97c1477a0 0x7fb97c0725a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 -- 172.21.15.40:0/3901328088 >> 172.21.15.40:0/3901328088 conn(0x7fb97c06b150 msgr2=0x7fb97c06b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:05.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 -- 172.21.15.40:0/3901328088 shutdown_connections 2024-03-24T00:38:05.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 -- 172.21.15.40:0/3901328088 wait complete. 2024-03-24T00:38:05.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 Processor -- start 2024-03-24T00:38:05.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 -- start start 2024-03-24T00:38:05.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb97c0b54d0 0x7fb97c146a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb97c1477a0 0x7fb97c13d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb97c15f050 0x7fb97c15bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c15e500 con 0x7fb97c15f050 2024-03-24T00:38:05.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c0b5e60 con 0x7fb97c1477a0 2024-03-24T00:38:05.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb982c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c0d1100 con 0x7fb97c0b54d0 2024-03-24T00:38:05.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb980c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb97c15f050 0x7fb97c15bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb97be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb97c0b54d0 0x7fb97c146a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb980c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb97c15f050 0x7fb97c15bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48038/0 (socket says 172.21.15.40:48038) 2024-03-24T00:38:05.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb97be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb97c0b54d0 0x7fb97c146a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:33104/0 (socket says 172.21.15.40:33104) 2024-03-24T00:38:05.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb980c00640 1 -- 172.21.15.40:0/1931342981 learned_addr learned my addr 172.21.15.40:0/1931342981 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:05.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb973e00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb97c1477a0 0x7fb97c13d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb980c00640 1 -- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb97c0b54d0 msgr2=0x7fb97c146a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb980c00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb97c0b54d0 0x7fb97c146a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb980c00640 1 -- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb97c1477a0 msgr2=0x7fb97c13d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb980c00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb97c1477a0 0x7fb97c13d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.221+0000 7fb980c00640 1 -- 172.21.15.40:0/1931342981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c146fc0 con 0x7fb97c15f050 2024-03-24T00:38:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.225+0000 7fb980c00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb97c15f050 0x7fb97c15bac0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fb97401af20 tx=0x7fb974019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.225+0000 7fb978c00640 1 -- 172.21.15.40:0/1931342981 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9740b2050 con 0x7fb97c15f050 2024-03-24T00:38:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.225+0000 7fb978c00640 1 -- 172.21.15.40:0/1931342981 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9740be070 con 0x7fb97c15f050 2024-03-24T00:38:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.225+0000 7fb978c00640 1 -- 172.21.15.40:0/1931342981 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9740ada70 con 0x7fb97c15f050 2024-03-24T00:38:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.225+0000 7fb982c00640 1 -- 172.21.15.40:0/1931342981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb97c145510 con 0x7fb97c15f050 2024-03-24T00:38:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.225+0000 7fb982c00640 1 -- 172.21.15.40:0/1931342981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97c145230 con 0x7fb97c15f050 2024-03-24T00:38:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.225+0000 7fb982c00640 1 -- 172.21.15.40:0/1931342981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97c13d5c0 con 0x7fb97c15f050 2024-03-24T00:38:05.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.225+0000 7fb978c00640 1 -- 172.21.15.40:0/1931342981 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fb9740a5770 con 0x7fb97c15f050 2024-03-24T00:38:05.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.225+0000 7fb978c00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb958067660 0x7fb958069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.225+0000 7fb978c00640 1 -- 172.21.15.40:0/1931342981 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9741209c0 con 0x7fb97c15f050 2024-03-24T00:38:05.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.225+0000 7fb97be00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb958067660 0x7fb958069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.229+0000 7fb97be00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb958067660 0x7fb958069b20 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009580 tx=0x7fb96c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:05.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.229+0000 7fb978c00640 1 -- 172.21.15.40:0/1931342981 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fb97c146fc0 con 0x7fb97c15f050 2024-03-24T00:38:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.429+0000 7fb982c00640 1 -- 172.21.15.40:0/1931342981 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb97c143c80 con 0x7fb97c15f050 2024-03-24T00:38:05.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.429+0000 7fb978c00640 1 -- 172.21.15.40:0/1931342981 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4973 (secure 0 0 0) 0x7fb974016520 con 0x7fb97c15f050 2024-03-24T00:38:05.434 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:05.435 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7079,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/755981654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":755981654},{"type":"v1","addr":"172.21.15.40:6835","nonce":755981654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":8101,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/3877615536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3877615536},{"type":"v1","addr":"172.21.15.53:6836","nonce":3877615536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":8400,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/3883043397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":3883043397},{"type":"v1","addr":"172.21.15.40:6834","nonce":3883043397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:01.424871+0000","modified":"2024-03-24T00:38:04.435669+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/8812937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":8812937},{"type":"v1","addr":"172.21.15.53:6837","nonce":8812937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8418,"qdb_cluster":[8418]},"id":8}]} 2024-03-24T00:38:05.435 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 112 2024-03-24T00:38:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.433+0000 7fb982c00640 1 -- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb958067660 msgr2=0x7fb958069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.433+0000 7fb982c00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb958067660 0x7fb958069b20 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb96c009580 tx=0x7fb96c009190 comp rx=0 tx=0).stop 2024-03-24T00:38:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.433+0000 7fb982c00640 1 -- 172.21.15.40:0/1931342981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb97c15f050 msgr2=0x7fb97c15bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.433+0000 7fb982c00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb97c15f050 0x7fb97c15bac0 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fb97401af20 tx=0x7fb974019610 comp rx=0 tx=0).stop 2024-03-24T00:38:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.433+0000 7fb982c00640 1 -- 172.21.15.40:0/1931342981 shutdown_connections 2024-03-24T00:38:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.433+0000 7fb982c00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fb958067660 0x7fb958069b20 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.433+0000 7fb982c00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fb97c15f050 0x7fb97c15bac0 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.433+0000 7fb982c00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb97c1477a0 0x7fb97c13d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.433+0000 7fb982c00640 1 --2- 172.21.15.40:0/1931342981 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb97c0b54d0 0x7fb97c146a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.433+0000 7fb982c00640 1 -- 172.21.15.40:0/1931342981 >> 172.21.15.40:0/1931342981 conn(0x7fb97c06b150 msgr2=0x7fb97c0b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.433+0000 7fb982c00640 1 -- 172.21.15.40:0/1931342981 shutdown_connections 2024-03-24T00:38:05.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.433+0000 7fb982c00640 1 -- 172.21.15.40:0/1931342981 wait complete. 2024-03-24T00:38:05.456 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:05.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff07c600640 1 Processor -- start 2024-03-24T00:38:05.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff07c600640 1 -- start start 2024-03-24T00:38:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff07c600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff07415efd0 0x7ff0740af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff07c600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff07415f3b0 0x7ff07406d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff07c600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff07406da60 0x7ff0740ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff07c600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0740ad630 con 0x7ff07406da60 2024-03-24T00:38:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff07c600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0740ad910 con 0x7ff07415efd0 2024-03-24T00:38:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff07c600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0740adbf0 con 0x7ff07415f3b0 2024-03-24T00:38:05.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff07a600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff07406da60 0x7ff0740ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff079200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff07415f3b0 0x7ff07406d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff07a600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff07406da60 0x7ff0740ae6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48050/0 (socket says 172.21.15.40:48050) 2024-03-24T00:38:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff07a600640 1 -- 172.21.15.40:0/239465788 learned_addr learned my addr 172.21.15.40:0/239465788 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.593+0000 7ff079c00640 1 --2- 172.21.15.40:0/239465788 >> v2:172.21.15.53:3300/0 conn(0x7ff07415efd0 0x7ff0740af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07a600640 1 -- 172.21.15.40:0/239465788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff07415f3b0 msgr2=0x7ff07406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07a600640 1 --2- 172.21.15.40:0/239465788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff07415f3b0 0x7ff07406d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07a600640 1 -- 172.21.15.40:0/239465788 >> v2:172.21.15.53:3300/0 conn(0x7ff07415efd0 msgr2=0x7ff0740af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07a600640 1 --2- 172.21.15.40:0/239465788 >> v2:172.21.15.53:3300/0 conn(0x7ff07415efd0 0x7ff0740af4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07a600640 1 -- 172.21.15.40:0/239465788 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0740aded0 con 0x7ff07406da60 2024-03-24T00:38:05.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07a600640 1 --2- 172.21.15.40:0/239465788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff07406da60 0x7ff0740ae6d0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff074072780 tx=0x7ff06c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac92035819edbda4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff073e00640 1 -- 172.21.15.40:0/239465788 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c0a7070 con 0x7ff07406da60 2024-03-24T00:38:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff073e00640 1 -- 172.21.15.40:0/239465788 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06c0b0070 con 0x7ff07406da60 2024-03-24T00:38:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff073e00640 1 -- 172.21.15.40:0/239465788 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c0ab890 con 0x7ff07406da60 2024-03-24T00:38:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07c600640 1 -- 172.21.15.40:0/239465788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff07406da60 msgr2=0x7ff0740ae6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07c600640 1 --2- 172.21.15.40:0/239465788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff07406da60 0x7ff0740ae6d0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff074072780 tx=0x7ff06c01c4f0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07c600640 1 -- 172.21.15.40:0/239465788 shutdown_connections 2024-03-24T00:38:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07c600640 1 --2- 172.21.15.40:0/239465788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff07406da60 0x7ff0740ae6d0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07c600640 1 --2- 172.21.15.40:0/239465788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff07415f3b0 0x7ff07406d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07c600640 1 --2- 172.21.15.40:0/239465788 >> v2:172.21.15.53:3300/0 conn(0x7ff07415efd0 0x7ff0740af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07c600640 1 -- 172.21.15.40:0/239465788 >> 172.21.15.40:0/239465788 conn(0x7ff0740b1f00 msgr2=0x7ff0740b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07c600640 1 -- 172.21.15.40:0/239465788 shutdown_connections 2024-03-24T00:38:05.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.597+0000 7ff07c600640 1 -- 172.21.15.40:0/239465788 wait complete. 2024-03-24T00:38:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff07c600640 1 Processor -- start 2024-03-24T00:38:05.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff07c600640 1 -- start start 2024-03-24T00:38:05.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff07c600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff07406da60 0x7ff074156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff07c600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff07415efd0 0x7ff0740c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff07c600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff07415f3b0 0x7ff07415a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff07c600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0740ad360 con 0x7ff07406da60 2024-03-24T00:38:05.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff07c600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0740ad1e0 con 0x7ff07415f3b0 2024-03-24T00:38:05.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff07c600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0740d0990 con 0x7ff07415efd0 2024-03-24T00:38:05.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff079c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff07406da60 0x7ff074156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff079c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff07406da60 0x7ff074156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48064/0 (socket says 172.21.15.40:48064) 2024-03-24T00:38:05.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff079c00640 1 -- 172.21.15.40:0/254264025 learned_addr learned my addr 172.21.15.40:0/254264025 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:05.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff079200640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff07415efd0 0x7ff0740c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff07a600640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff07415f3b0 0x7ff07415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff079c00640 1 -- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff07415efd0 msgr2=0x7ff0740c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff079c00640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff07415efd0 0x7ff0740c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff079c00640 1 -- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff07415f3b0 msgr2=0x7ff07415a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff079c00640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff07415f3b0 0x7ff07415a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff079c00640 1 -- 172.21.15.40:0/254264025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0740ceb90 con 0x7ff07406da60 2024-03-24T00:38:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff079c00640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff07406da60 0x7ff074156490 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff06800eed0 tx=0x7ff06800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff071600640 1 -- 172.21.15.40:0/254264025 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068018020 con 0x7ff07406da60 2024-03-24T00:38:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff071600640 1 -- 172.21.15.40:0/254264025 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff068014400 con 0x7ff07406da60 2024-03-24T00:38:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff071600640 1 -- 172.21.15.40:0/254264025 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0680147d0 con 0x7ff07406da60 2024-03-24T00:38:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff07c600640 1 -- 172.21.15.40:0/254264025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0740b1810 con 0x7ff07406da60 2024-03-24T00:38:05.608 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.601+0000 7ff07c600640 1 -- 172.21.15.40:0/254264025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0740c5fd0 con 0x7ff07406da60 2024-03-24T00:38:05.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.605+0000 7ff07c600640 1 -- 172.21.15.40:0/254264025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff074068170 con 0x7ff07406da60 2024-03-24T00:38:05.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.605+0000 7ff071600640 1 -- 172.21.15.40:0/254264025 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff06801b020 con 0x7ff07406da60 2024-03-24T00:38:05.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.605+0000 7ff071600640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff054067660 0x7ff054069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.605+0000 7ff071600640 1 -- 172.21.15.40:0/254264025 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff068085e20 con 0x7ff07406da60 2024-03-24T00:38:05.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.605+0000 7ff079200640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff054067660 0x7ff054069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.605+0000 7ff079200640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff054067660 0x7ff054069b20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff0740ae740 tx=0x7ff060009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:05.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.609+0000 7ff071600640 1 -- 172.21.15.40:0/254264025 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff068022070 con 0x7ff07406da60 2024-03-24T00:38:05.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 -- 172.21.15.40:0/254264025 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff074168ec0 con 0x7ff07406da60 2024-03-24T00:38:05.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff071600640 1 -- 172.21.15.40:0/254264025 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16567 (secure 0 0 0) 0x7ff068056440 con 0x7ff07406da60 2024-03-24T00:38:05.777 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:05.777 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":106,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:38:03.036173+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:37:59.849300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:00.417342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 -- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff054067660 msgr2=0x7ff054069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff054067660 0x7ff054069b20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff0740ae740 tx=0x7ff060009190 comp rx=0 tx=0).stop 2024-03-24T00:38:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 -- 172.21.15.40:0/254264025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff07406da60 msgr2=0x7ff074156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff07406da60 0x7ff074156490 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff06800eed0 tx=0x7ff06800b480 comp rx=0 tx=0).stop 2024-03-24T00:38:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 -- 172.21.15.40:0/254264025 shutdown_connections 2024-03-24T00:38:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff054067660 0x7ff054069b20 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff07415f3b0 0x7ff07415a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff07415efd0 0x7ff0740c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 --2- 172.21.15.40:0/254264025 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff07406da60 0x7ff074156490 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 -- 172.21.15.40:0/254264025 >> 172.21.15.40:0/254264025 conn(0x7ff0740b1f00 msgr2=0x7ff07415f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:05.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 -- 172.21.15.40:0/254264025 shutdown_connections 2024-03-24T00:38:05.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.773+0000 7ff07c600640 1 -- 172.21.15.40:0/254264025 wait complete. 2024-03-24T00:38:05.798 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:38:05.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:38:05.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d74a00640 1 Processor -- start 2024-03-24T00:38:05.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d74a00640 1 -- start start 2024-03-24T00:38:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d74a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d701619c0 0x7f5d70161e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d74a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d70066e80 0x7f5d70067310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d74a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d700adff0 con 0x7f5d7015dd70 2024-03-24T00:38:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d74a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d70162480 con 0x7f5d701619c0 2024-03-24T00:38:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d74a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d70162760 con 0x7f5d70066e80 2024-03-24T00:38:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d67e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f5d701619c0 0x7f5d70161e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d6e800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5d7015dd70 0x7f5d700ad8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:50492/0 (socket says 172.21.15.40:50492) 2024-03-24T00:38:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d6e800640 1 -- 172.21.15.40:0/3713197881 learned_addr learned my addr 172.21.15.40:0/3713197881 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:05.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d6f200640 1 --2- 172.21.15.40:0/3713197881 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d70066e80 0x7f5d70067310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d6de00640 1 -- 172.21.15.40:0/3713197881 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876015530 0 0) 0x7f5d700adff0 con 0x7f5d7015dd70 2024-03-24T00:38:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d6de00640 1 -- 172.21.15.40:0/3713197881 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d54003530 con 0x7f5d7015dd70 2024-03-24T00:38:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d67e00640 1 -- 172.21.15.40:0/3713197881 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d70066e80 msgr2=0x7f5d70067310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d67e00640 1 --2- 172.21.15.40:0/3713197881 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d70066e80 0x7f5d70067310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d67e00640 1 -- 172.21.15.40:0/3713197881 >> v1:172.21.15.40:6789/0 conn(0x7f5d7015dd70 legacy=0x7f5d700ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.937+0000 7f5d67e00640 1 -- 172.21.15.40:0/3713197881 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d700b4cf0 con 0x7f5d701619c0 2024-03-24T00:38:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d67e00640 1 --2- 172.21.15.40:0/3713197881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d701619c0 0x7f5d70161e30 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f5d580097c0 tx=0x7f5d5801cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69de8a217b2ed56 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d6de00640 1 -- 172.21.15.40:0/3713197881 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d580b0070 con 0x7f5d701619c0 2024-03-24T00:38:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d6de00640 1 -- 172.21.15.40:0/3713197881 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d580ab600 con 0x7f5d701619c0 2024-03-24T00:38:05.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d6de00640 1 -- 172.21.15.40:0/3713197881 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d580ab9b0 con 0x7f5d701619c0 2024-03-24T00:38:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d74a00640 1 -- 172.21.15.40:0/3713197881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d701619c0 msgr2=0x7f5d70161e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d74a00640 1 --2- 172.21.15.40:0/3713197881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d701619c0 0x7f5d70161e30 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f5d580097c0 tx=0x7f5d5801cf00 comp rx=0 tx=0).stop 2024-03-24T00:38:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d74a00640 1 -- 172.21.15.40:0/3713197881 shutdown_connections 2024-03-24T00:38:05.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d74a00640 1 --2- 172.21.15.40:0/3713197881 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d70066e80 0x7f5d70067310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d74a00640 1 --2- 172.21.15.40:0/3713197881 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d701619c0 0x7f5d70161e30 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d74a00640 1 -- 172.21.15.40:0/3713197881 >> 172.21.15.40:0/3713197881 conn(0x7f5d70071a40 msgr2=0x7f5d70071e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d74a00640 1 -- 172.21.15.40:0/3713197881 shutdown_connections 2024-03-24T00:38:05.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d74a00640 1 -- 172.21.15.40:0/3713197881 wait complete. 2024-03-24T00:38:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d74a00640 1 Processor -- start 2024-03-24T00:38:05.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.941+0000 7f5d74a00640 1 -- start start 2024-03-24T00:38:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d74a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d70066e80 0x7f5d70149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d74a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d7015dd70 0x7f5d7014e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d74a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d701619c0 0x7f5d70159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d74a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7015e2d0 con 0x7f5d70066e80 2024-03-24T00:38:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d74a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7015e150 con 0x7f5d7015dd70 2024-03-24T00:38:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d74a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7016ad60 con 0x7f5d701619c0 2024-03-24T00:38:05.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d6e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d70066e80 0x7f5d70149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d6e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d70066e80 0x7f5d70149ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48070/0 (socket says 172.21.15.40:48070) 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d6e800640 1 -- 172.21.15.40:0/2485142156 learned_addr learned my addr 172.21.15.40:0/2485142156 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d67e00640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d7015dd70 0x7f5d7014e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d6f200640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d701619c0 0x7f5d70159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d6e800640 1 -- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d701619c0 msgr2=0x7f5d70159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d6e800640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d701619c0 0x7f5d70159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d6e800640 1 -- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d7015dd70 msgr2=0x7f5d7014e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d6e800640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d7015dd70 0x7f5d7014e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d6e800640 1 -- 172.21.15.40:0/2485142156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7014ee30 con 0x7f5d70066e80 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d6e800640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d70066e80 0x7f5d70149ca0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000c730 tx=0x7f5d6000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d66a00640 1 -- 172.21.15.40:0/2485142156 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d60017070 con 0x7f5d70066e80 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d66a00640 1 -- 172.21.15.40:0/2485142156 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d60007270 con 0x7f5d70066e80 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d66a00640 1 -- 172.21.15.40:0/2485142156 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d60001d50 con 0x7f5d70066e80 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d74a00640 1 -- 172.21.15.40:0/2485142156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7014bc20 con 0x7f5d70066e80 2024-03-24T00:38:05.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d74a00640 1 -- 172.21.15.40:0/2485142156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7014c700 con 0x7f5d70066e80 2024-03-24T00:38:05.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.945+0000 7f5d74a00640 1 -- 172.21.15.40:0/2485142156 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d700733f0 con 0x7f5d70066e80 2024-03-24T00:38:05.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.949+0000 7f5d66a00640 1 -- 172.21.15.40:0/2485142156 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5d6001a070 con 0x7f5d70066e80 2024-03-24T00:38:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.949+0000 7f5d66a00640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5d4c067660 0x7f5d4c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.949+0000 7f5d66a00640 1 -- 172.21.15.40:0/2485142156 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d60084b90 con 0x7f5d70066e80 2024-03-24T00:38:05.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.949+0000 7f5d67e00640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5d4c067660 0x7f5d4c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:05.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.949+0000 7f5d67e00640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5d4c067660 0x7f5d4c069b20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5d580ad000 tx=0x7f5d58004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:05.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:05.953+0000 7f5d66a00640 1 -- 172.21.15.40:0/2485142156 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5d600551b0 con 0x7f5d70066e80 2024-03-24T00:38:06.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.113+0000 7f5d74a00640 1 -- 172.21.15.40:0/2485142156 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d700b4220 con 0x7f5d4c067660 2024-03-24T00:38:06.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d66a00640 1 -- 172.21.15.40:0/2485142156 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252453 (secure 0 0 0) 0x7f5d700b4220 con 0x7f5d4c067660 2024-03-24T00:38:06.128 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:06.129 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:38:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d74a00640 1 -- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5d4c067660 msgr2=0x7f5d4c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d74a00640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5d4c067660 0x7f5d4c069b20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f5d580ad000 tx=0x7f5d58004740 comp rx=0 tx=0).stop 2024-03-24T00:38:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d74a00640 1 -- 172.21.15.40:0/2485142156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d70066e80 msgr2=0x7f5d70149ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d74a00640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d70066e80 0x7f5d70149ca0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000c730 tx=0x7f5d6000cbf0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d74a00640 1 -- 172.21.15.40:0/2485142156 shutdown_connections 2024-03-24T00:38:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d74a00640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5d4c067660 0x7f5d4c069b20 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d74a00640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d701619c0 0x7f5d70159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d74a00640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d7015dd70 0x7f5d7014e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d74a00640 1 --2- 172.21.15.40:0/2485142156 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5d70066e80 0x7f5d70149ca0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d74a00640 1 -- 172.21.15.40:0/2485142156 >> 172.21.15.40:0/2485142156 conn(0x7f5d70071a40 msgr2=0x7f5d7006ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:06.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d74a00640 1 -- 172.21.15.40:0/2485142156 shutdown_connections 2024-03-24T00:38:06.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.125+0000 7f5d74a00640 1 -- 172.21.15.40:0/2485142156 wait complete. 2024-03-24T00:38:06.148 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-03-24T00:38:05.281916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":326156,"kb_used_data":11468,"kb_used_omap":12,"kb_used_meta":314547,"kb_avail":754648564,"statfs":{"total":773094113280,"available":772760129536,"internally_reserved":0,"allocated":11743232,"data_stored":5756264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":322096690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.264204"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.055220+0000","last_change":"2024-03-24T00:38:02.053053+0000","last_active":"2024-03-24T00:38:02.055220+0000","last_peered":"2024-03-24T00:38:02.055220+0000","last_clean":"2024-03-24T00:38:02.055220+0000","last_became_active":"2024-03-24T00:38:02.052642+0000","last_became_peered":"2024-03-24T00:38:02.052642+0000","last_unstale":"2024-03-24T00:38:02.055220+0000","last_undegraded":"2024-03-24T00:38:02.055220+0000","last_fullsized":"2024-03-24T00:38:02.055220+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:23:20.445022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047344+0000","last_change":"2024-03-24T00:38:02.039253+0000","last_active":"2024-03-24T00:38:03.047344+0000","last_peered":"2024-03-24T00:38:03.047344+0000","last_clean":"2024-03-24T00:38:03.047344+0000","last_became_active":"2024-03-24T00:38:02.038694+0000","last_became_peered":"2024-03-24T00:38:02.038694+0000","last_unstale":"2024-03-24T00:38:03.047344+0000","last_undegraded":"2024-03-24T00:38:03.047344+0000","last_fullsized":"2024-03-24T00:38:03.047344+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:05:20.785596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047758+0000","last_change":"2024-03-24T00:38:02.056053+0000","last_active":"2024-03-24T00:38:03.047758+0000","last_peered":"2024-03-24T00:38:03.047758+0000","last_clean":"2024-03-24T00:38:03.047758+0000","last_became_active":"2024-03-24T00:38:02.051992+0000","last_became_peered":"2024-03-24T00:38:02.051992+0000","last_unstale":"2024-03-24T00:38:03.047758+0000","last_undegraded":"2024-03-24T00:38:03.047758+0000","last_fullsized":"2024-03-24T00:38:03.047758+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:05.858925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.041135+0000","last_change":"2024-03-24T00:38:02.033932+0000","last_active":"2024-03-24T00:38:02.041135+0000","last_peered":"2024-03-24T00:38:02.041135+0000","last_clean":"2024-03-24T00:38:02.041135+0000","last_became_active":"2024-03-24T00:38:02.033651+0000","last_became_peered":"2024-03-24T00:38:02.033651+0000","last_unstale":"2024-03-24T00:38:02.041135+0000","last_undegraded":"2024-03-24T00:38:02.041135+0000","last_fullsized":"2024-03-24T00:38:02.041135+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:57.560121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.052166+0000","last_change":"2024-03-24T00:38:02.047238+0000","last_active":"2024-03-24T00:38:02.052166+0000","last_peered":"2024-03-24T00:38:02.052166+0000","last_clean":"2024-03-24T00:38:02.052166+0000","last_became_active":"2024-03-24T00:38:02.046942+0000","last_became_peered":"2024-03-24T00:38:02.046942+0000","last_unstale":"2024-03-24T00:38:02.052166+0000","last_undegraded":"2024-03-24T00:38:02.052166+0000","last_fullsized":"2024-03-24T00:38:02.052166+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:30.720526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.053030+0000","last_change":"2024-03-24T00:38:02.049961+0000","last_active":"2024-03-24T00:38:02.053030+0000","last_peered":"2024-03-24T00:38:02.053030+0000","last_clean":"2024-03-24T00:38:02.053030+0000","last_became_active":"2024-03-24T00:38:02.049623+0000","last_became_peered":"2024-03-24T00:38:02.049623+0000","last_unstale":"2024-03-24T00:38:02.053030+0000","last_undegraded":"2024-03-24T00:38:02.053030+0000","last_fullsized":"2024-03-24T00:38:02.053030+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:46:45.454332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061753+0000","last_change":"2024-03-24T00:38:02.056781+0000","last_active":"2024-03-24T00:38:03.061753+0000","last_peered":"2024-03-24T00:38:03.061753+0000","last_clean":"2024-03-24T00:38:03.061753+0000","last_became_active":"2024-03-24T00:38:02.053927+0000","last_became_peered":"2024-03-24T00:38:02.053927+0000","last_unstale":"2024-03-24T00:38:03.061753+0000","last_undegraded":"2024-03-24T00:38:03.061753+0000","last_fullsized":"2024-03-24T00:38:03.061753+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:35:21.632806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046615+0000","last_change":"2024-03-24T00:38:02.051319+0000","last_active":"2024-03-24T00:38:03.046615+0000","last_peered":"2024-03-24T00:38:03.046615+0000","last_clean":"2024-03-24T00:38:03.046615+0000","last_became_active":"2024-03-24T00:38:02.050929+0000","last_became_peered":"2024-03-24T00:38:02.050929+0000","last_unstale":"2024-03-24T00:38:03.046615+0000","last_undegraded":"2024-03-24T00:38:03.046615+0000","last_fullsized":"2024-03-24T00:38:03.046615+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:44.746992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.048328+0000","last_change":"2024-03-24T00:38:02.042223+0000","last_active":"2024-03-24T00:38:02.048328+0000","last_peered":"2024-03-24T00:38:02.048328+0000","last_clean":"2024-03-24T00:38:02.048328+0000","last_became_active":"2024-03-24T00:38:02.041336+0000","last_became_peered":"2024-03-24T00:38:02.041336+0000","last_unstale":"2024-03-24T00:38:02.048328+0000","last_undegraded":"2024-03-24T00:38:02.048328+0000","last_fullsized":"2024-03-24T00:38:02.048328+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:17.082488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.058940+0000","last_change":"2024-03-24T00:38:02.057633+0000","last_active":"2024-03-24T00:38:02.058940+0000","last_peered":"2024-03-24T00:38:02.058940+0000","last_clean":"2024-03-24T00:38:02.058940+0000","last_became_active":"2024-03-24T00:38:02.057309+0000","last_became_peered":"2024-03-24T00:38:02.057309+0000","last_unstale":"2024-03-24T00:38:02.058940+0000","last_undegraded":"2024-03-24T00:38:02.058940+0000","last_fullsized":"2024-03-24T00:38:02.058940+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:16:49.746603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046792+0000","last_change":"2024-03-24T00:38:02.057278+0000","last_active":"2024-03-24T00:38:03.046792+0000","last_peered":"2024-03-24T00:38:03.046792+0000","last_clean":"2024-03-24T00:38:03.046792+0000","last_became_active":"2024-03-24T00:38:02.056867+0000","last_became_peered":"2024-03-24T00:38:02.056867+0000","last_unstale":"2024-03-24T00:38:03.046792+0000","last_undegraded":"2024-03-24T00:38:03.046792+0000","last_fullsized":"2024-03-24T00:38:03.046792+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:56.438539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.054211+0000","last_change":"2024-03-24T00:38:02.049818+0000","last_active":"2024-03-24T00:38:02.054211+0000","last_peered":"2024-03-24T00:38:02.054211+0000","last_clean":"2024-03-24T00:38:02.054211+0000","last_became_active":"2024-03-24T00:38:02.049549+0000","last_became_peered":"2024-03-24T00:38:02.049549+0000","last_unstale":"2024-03-24T00:38:02.054211+0000","last_undegraded":"2024-03-24T00:38:02.054211+0000","last_fullsized":"2024-03-24T00:38:02.054211+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:01.957880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048015+0000","last_change":"2024-03-24T00:38:02.063775+0000","last_active":"2024-03-24T00:38:03.048015+0000","last_peered":"2024-03-24T00:38:03.048015+0000","last_clean":"2024-03-24T00:38:03.048015+0000","last_became_active":"2024-03-24T00:38:02.063415+0000","last_became_peered":"2024-03-24T00:38:02.063415+0000","last_unstale":"2024-03-24T00:38:03.048015+0000","last_undegraded":"2024-03-24T00:38:03.048015+0000","last_fullsized":"2024-03-24T00:38:03.048015+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:33.285058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048098+0000","last_change":"2024-03-24T00:38:02.058600+0000","last_active":"2024-03-24T00:38:03.048098+0000","last_peered":"2024-03-24T00:38:03.048098+0000","last_clean":"2024-03-24T00:38:03.048098+0000","last_became_active":"2024-03-24T00:38:02.056388+0000","last_became_peered":"2024-03-24T00:38:02.056388+0000","last_unstale":"2024-03-24T00:38:03.048098+0000","last_undegraded":"2024-03-24T00:38:03.048098+0000","last_fullsized":"2024-03-24T00:38:03.048098+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:19.015495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.053051+0000","last_change":"2024-03-24T00:38:02.049842+0000","last_active":"2024-03-24T00:38:02.053051+0000","last_peered":"2024-03-24T00:38:02.053051+0000","last_clean":"2024-03-24T00:38:02.053051+0000","last_became_active":"2024-03-24T00:38:02.049488+0000","last_became_peered":"2024-03-24T00:38:02.049488+0000","last_unstale":"2024-03-24T00:38:02.053051+0000","last_undegraded":"2024-03-24T00:38:02.053051+0000","last_fullsized":"2024-03-24T00:38:02.053051+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:55.045351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.050117+0000","last_change":"2024-03-24T00:38:02.048083+0000","last_active":"2024-03-24T00:38:02.050117+0000","last_peered":"2024-03-24T00:38:02.050117+0000","last_clean":"2024-03-24T00:38:02.050117+0000","last_became_active":"2024-03-24T00:38:02.047746+0000","last_became_peered":"2024-03-24T00:38:02.047746+0000","last_unstale":"2024-03-24T00:38:02.050117+0000","last_undegraded":"2024-03-24T00:38:02.050117+0000","last_fullsized":"2024-03-24T00:38:02.050117+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:25:24.459692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047270+0000","last_change":"2024-03-24T00:38:02.048239+0000","last_active":"2024-03-24T00:38:03.047270+0000","last_peered":"2024-03-24T00:38:03.047270+0000","last_clean":"2024-03-24T00:38:03.047270+0000","last_became_active":"2024-03-24T00:38:02.047884+0000","last_became_peered":"2024-03-24T00:38:02.047884+0000","last_unstale":"2024-03-24T00:38:03.047270+0000","last_undegraded":"2024-03-24T00:38:03.047270+0000","last_fullsized":"2024-03-24T00:38:03.047270+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:59:50.468231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059344+0000","last_change":"2024-03-24T00:38:02.046623+0000","last_active":"2024-03-24T00:38:03.059344+0000","last_peered":"2024-03-24T00:38:03.059344+0000","last_clean":"2024-03-24T00:38:03.059344+0000","last_became_active":"2024-03-24T00:38:02.046146+0000","last_became_peered":"2024-03-24T00:38:02.046146+0000","last_unstale":"2024-03-24T00:38:03.059344+0000","last_undegraded":"2024-03-24T00:38:03.059344+0000","last_fullsized":"2024-03-24T00:38:03.059344+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:38.764692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048219+0000","last_change":"2024-03-24T00:38:02.066388+0000","last_active":"2024-03-24T00:38:03.048219+0000","last_peered":"2024-03-24T00:38:03.048219+0000","last_clean":"2024-03-24T00:38:03.048219+0000","last_became_active":"2024-03-24T00:38:02.065984+0000","last_became_peered":"2024-03-24T00:38:02.065984+0000","last_unstale":"2024-03-24T00:38:03.048219+0000","last_undegraded":"2024-03-24T00:38:03.048219+0000","last_fullsized":"2024-03-24T00:38:03.048219+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:56:34.503377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047695+0000","last_change":"2024-03-24T00:38:02.050469+0000","last_active":"2024-03-24T00:38:03.047695+0000","last_peered":"2024-03-24T00:38:03.047695+0000","last_clean":"2024-03-24T00:38:03.047695+0000","last_became_active":"2024-03-24T00:38:02.050090+0000","last_became_peered":"2024-03-24T00:38:02.050090+0000","last_unstale":"2024-03-24T00:38:03.047695+0000","last_undegraded":"2024-03-24T00:38:03.047695+0000","last_fullsized":"2024-03-24T00:38:03.047695+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:59:16.472535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047039+0000","last_change":"2024-03-24T00:38:02.050263+0000","last_active":"2024-03-24T00:38:03.047039+0000","last_peered":"2024-03-24T00:38:03.047039+0000","last_clean":"2024-03-24T00:38:03.047039+0000","last_became_active":"2024-03-24T00:38:02.047937+0000","last_became_peered":"2024-03-24T00:38:02.047937+0000","last_unstale":"2024-03-24T00:38:03.047039+0000","last_undegraded":"2024-03-24T00:38:03.047039+0000","last_fullsized":"2024-03-24T00:38:03.047039+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:16:08.887440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043591+0000","last_change":"2024-03-24T00:38:02.052851+0000","last_active":"2024-03-24T00:38:03.043591+0000","last_peered":"2024-03-24T00:38:03.043591+0000","last_clean":"2024-03-24T00:38:03.043591+0000","last_became_active":"2024-03-24T00:38:02.052500+0000","last_became_peered":"2024-03-24T00:38:02.052500+0000","last_unstale":"2024-03-24T00:38:03.043591+0000","last_undegraded":"2024-03-24T00:38:03.043591+0000","last_fullsized":"2024-03-24T00:38:03.043591+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:05.858925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047592+0000","last_change":"2024-03-24T00:38:02.051064+0000","last_active":"2024-03-24T00:38:03.047592+0000","last_peered":"2024-03-24T00:38:03.047592+0000","last_clean":"2024-03-24T00:38:03.047592+0000","last_became_active":"2024-03-24T00:38:02.050775+0000","last_became_peered":"2024-03-24T00:38:02.050775+0000","last_unstale":"2024-03-24T00:38:03.047592+0000","last_undegraded":"2024-03-24T00:38:03.047592+0000","last_fullsized":"2024-03-24T00:38:03.047592+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:31:48.804402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043398+0000","last_change":"2024-03-24T00:38:02.056440+0000","last_active":"2024-03-24T00:38:03.043398+0000","last_peered":"2024-03-24T00:38:03.043398+0000","last_clean":"2024-03-24T00:38:03.043398+0000","last_became_active":"2024-03-24T00:38:02.056140+0000","last_became_peered":"2024-03-24T00:38:02.056140+0000","last_unstale":"2024-03-24T00:38:03.043398+0000","last_undegraded":"2024-03-24T00:38:03.043398+0000","last_fullsized":"2024-03-24T00:38:03.043398+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:35:59.814600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047454+0000","last_change":"2024-03-24T00:38:02.047976+0000","last_active":"2024-03-24T00:38:03.047454+0000","last_peered":"2024-03-24T00:38:03.047454+0000","last_clean":"2024-03-24T00:38:03.047454+0000","last_became_active":"2024-03-24T00:38:02.047501+0000","last_became_peered":"2024-03-24T00:38:02.047501+0000","last_unstale":"2024-03-24T00:38:03.047454+0000","last_undegraded":"2024-03-24T00:38:03.047454+0000","last_fullsized":"2024-03-24T00:38:03.047454+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:57.001029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.048194+0000","last_change":"2024-03-24T00:38:02.044475+0000","last_active":"2024-03-24T00:38:02.048194+0000","last_peered":"2024-03-24T00:38:02.048194+0000","last_clean":"2024-03-24T00:38:02.048194+0000","last_became_active":"2024-03-24T00:38:02.044213+0000","last_became_peered":"2024-03-24T00:38:02.044213+0000","last_unstale":"2024-03-24T00:38:02.048194+0000","last_undegraded":"2024-03-24T00:38:02.048194+0000","last_fullsized":"2024-03-24T00:38:02.048194+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:47:16.587931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.053208+0000","last_change":"2024-03-24T00:38:02.050311+0000","last_active":"2024-03-24T00:38:02.053208+0000","last_peered":"2024-03-24T00:38:02.053208+0000","last_clean":"2024-03-24T00:38:02.053208+0000","last_became_active":"2024-03-24T00:38:02.050004+0000","last_became_peered":"2024-03-24T00:38:02.050004+0000","last_unstale":"2024-03-24T00:38:02.053208+0000","last_undegraded":"2024-03-24T00:38:02.053208+0000","last_fullsized":"2024-03-24T00:38:02.053208+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:49.129268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043872+0000","last_change":"2024-03-24T00:38:02.049353+0000","last_active":"2024-03-24T00:38:03.043872+0000","last_peered":"2024-03-24T00:38:03.043872+0000","last_clean":"2024-03-24T00:38:03.043872+0000","last_became_active":"2024-03-24T00:38:02.049043+0000","last_became_peered":"2024-03-24T00:38:02.049043+0000","last_unstale":"2024-03-24T00:38:03.043872+0000","last_undegraded":"2024-03-24T00:38:03.043872+0000","last_fullsized":"2024-03-24T00:38:03.043872+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:21.620621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046553+0000","last_change":"2024-03-24T00:38:02.051647+0000","last_active":"2024-03-24T00:38:03.046553+0000","last_peered":"2024-03-24T00:38:03.046553+0000","last_clean":"2024-03-24T00:38:03.046553+0000","last_became_active":"2024-03-24T00:38:02.051097+0000","last_became_peered":"2024-03-24T00:38:02.051097+0000","last_unstale":"2024-03-24T00:38:03.046553+0000","last_undegraded":"2024-03-24T00:38:03.046553+0000","last_fullsized":"2024-03-24T00:38:03.046553+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:38:15.571575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061910+0000","last_change":"2024-03-24T00:38:02.046530+0000","last_active":"2024-03-24T00:38:03.061910+0000","last_peered":"2024-03-24T00:38:03.061910+0000","last_clean":"2024-03-24T00:38:03.061910+0000","last_became_active":"2024-03-24T00:38:02.046231+0000","last_became_peered":"2024-03-24T00:38:02.046231+0000","last_unstale":"2024-03-24T00:38:03.061910+0000","last_undegraded":"2024-03-24T00:38:03.061910+0000","last_fullsized":"2024-03-24T00:38:03.061910+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:22.127612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.048807+0000","last_change":"2024-03-24T00:38:02.043530+0000","last_active":"2024-03-24T00:38:02.048807+0000","last_peered":"2024-03-24T00:38:02.048807+0000","last_clean":"2024-03-24T00:38:02.048807+0000","last_became_active":"2024-03-24T00:38:02.043253+0000","last_became_peered":"2024-03-24T00:38:02.043253+0000","last_unstale":"2024-03-24T00:38:02.048807+0000","last_undegraded":"2024-03-24T00:38:02.048807+0000","last_fullsized":"2024-03-24T00:38:02.048807+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:09:31.230667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.051999+0000","last_change":"2024-03-24T00:38:02.045993+0000","last_active":"2024-03-24T00:38:02.051999+0000","last_peered":"2024-03-24T00:38:02.051999+0000","last_clean":"2024-03-24T00:38:02.051999+0000","last_became_active":"2024-03-24T00:38:02.045660+0000","last_became_peered":"2024-03-24T00:38:02.045660+0000","last_unstale":"2024-03-24T00:38:02.051999+0000","last_undegraded":"2024-03-24T00:38:02.051999+0000","last_fullsized":"2024-03-24T00:38:02.051999+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:21.791067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.057912+0000","last_change":"2024-03-24T00:38:02.055181+0000","last_active":"2024-03-24T00:38:02.057912+0000","last_peered":"2024-03-24T00:38:02.057912+0000","last_clean":"2024-03-24T00:38:02.057912+0000","last_became_active":"2024-03-24T00:38:02.054844+0000","last_became_peered":"2024-03-24T00:38:02.054844+0000","last_unstale":"2024-03-24T00:38:02.057912+0000","last_undegraded":"2024-03-24T00:38:02.057912+0000","last_fullsized":"2024-03-24T00:38:02.057912+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:56.702597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"105'4","reported_seq":18,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.062248+0000","last_change":"2024-03-24T00:38:01.034398+0000","last_active":"2024-03-24T00:38:03.062248+0000","last_peered":"2024-03-24T00:38:03.062248+0000","last_clean":"2024-03-24T00:38:03.062248+0000","last_became_active":"2024-03-24T00:38:01.034070+0000","last_became_peered":"2024-03-24T00:38:01.034070+0000","last_unstale":"2024-03-24T00:38:03.062248+0000","last_undegraded":"2024-03-24T00:38:03.062248+0000","last_fullsized":"2024-03-24T00:38:03.062248+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:09:58.345257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059562+0000","last_change":"2024-03-24T00:38:02.048706+0000","last_active":"2024-03-24T00:38:03.059562+0000","last_peered":"2024-03-24T00:38:03.059562+0000","last_clean":"2024-03-24T00:38:03.059562+0000","last_became_active":"2024-03-24T00:38:02.048392+0000","last_became_peered":"2024-03-24T00:38:02.048392+0000","last_unstale":"2024-03-24T00:38:03.059562+0000","last_undegraded":"2024-03-24T00:38:03.059562+0000","last_fullsized":"2024-03-24T00:38:03.059562+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:41:51.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047917+0000","last_change":"2024-03-24T00:38:01.040348+0000","last_active":"2024-03-24T00:38:03.047917+0000","last_peered":"2024-03-24T00:38:03.047917+0000","last_clean":"2024-03-24T00:38:03.047917+0000","last_became_active":"2024-03-24T00:38:01.039973+0000","last_became_peered":"2024-03-24T00:38:01.039973+0000","last_unstale":"2024-03-24T00:38:03.047917+0000","last_undegraded":"2024-03-24T00:38:03.047917+0000","last_fullsized":"2024-03-24T00:38:03.047917+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:38:14.564982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.053695+0000","last_change":"2024-03-24T00:38:02.052123+0000","last_active":"2024-03-24T00:38:02.053695+0000","last_peered":"2024-03-24T00:38:02.053695+0000","last_clean":"2024-03-24T00:38:02.053695+0000","last_became_active":"2024-03-24T00:38:02.051835+0000","last_became_peered":"2024-03-24T00:38:02.051835+0000","last_unstale":"2024-03-24T00:38:02.053695+0000","last_undegraded":"2024-03-24T00:38:02.053695+0000","last_fullsized":"2024-03-24T00:38:02.053695+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:09:59.351850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"105'2","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.072108+0000","last_change":"2024-03-24T00:38:01.051044+0000","last_active":"2024-03-24T00:38:02.072108+0000","last_peered":"2024-03-24T00:38:02.072108+0000","last_clean":"2024-03-24T00:38:02.072108+0000","last_became_active":"2024-03-24T00:38:01.050781+0000","last_became_peered":"2024-03-24T00:38:01.050781+0000","last_unstale":"2024-03-24T00:38:02.072108+0000","last_undegraded":"2024-03-24T00:38:02.072108+0000","last_fullsized":"2024-03-24T00:38:02.072108+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:09:38.732722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048326+0000","last_change":"2024-03-24T00:38:02.065641+0000","last_active":"2024-03-24T00:38:03.048326+0000","last_peered":"2024-03-24T00:38:03.048326+0000","last_clean":"2024-03-24T00:38:03.048326+0000","last_became_active":"2024-03-24T00:38:02.065310+0000","last_became_peered":"2024-03-24T00:38:02.065310+0000","last_unstale":"2024-03-24T00:38:03.048326+0000","last_undegraded":"2024-03-24T00:38:03.048326+0000","last_fullsized":"2024-03-24T00:38:03.048326+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:13:16.105953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"105'2","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.073461+0000","last_change":"2024-03-24T00:38:01.049067+0000","last_active":"2024-03-24T00:38:02.073461+0000","last_peered":"2024-03-24T00:38:02.073461+0000","last_clean":"2024-03-24T00:38:02.073461+0000","last_became_active":"2024-03-24T00:38:01.048807+0000","last_became_peered":"2024-03-24T00:38:01.048807+0000","last_unstale":"2024-03-24T00:38:02.073461+0000","last_undegraded":"2024-03-24T00:38:02.073461+0000","last_fullsized":"2024-03-24T00:38:02.073461+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:04:18.376681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.048756+0000","last_change":"2024-03-24T00:38:02.044640+0000","last_active":"2024-03-24T00:38:02.048756+0000","last_peered":"2024-03-24T00:38:02.048756+0000","last_clean":"2024-03-24T00:38:02.048756+0000","last_became_active":"2024-03-24T00:38:02.044353+0000","last_became_peered":"2024-03-24T00:38:02.044353+0000","last_unstale":"2024-03-24T00:38:02.048756+0000","last_undegraded":"2024-03-24T00:38:02.048756+0000","last_fullsized":"2024-03-24T00:38:02.048756+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:01:12.505348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061821+0000","last_change":"2024-03-24T00:38:01.041670+0000","last_active":"2024-03-24T00:38:03.061821+0000","last_peered":"2024-03-24T00:38:03.061821+0000","last_clean":"2024-03-24T00:38:03.061821+0000","last_became_active":"2024-03-24T00:38:01.041353+0000","last_became_peered":"2024-03-24T00:38:01.041353+0000","last_unstale":"2024-03-24T00:38:03.061821+0000","last_undegraded":"2024-03-24T00:38:03.061821+0000","last_fullsized":"2024-03-24T00:38:03.061821+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:54.038759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046722+0000","last_change":"2024-03-24T00:38:02.039321+0000","last_active":"2024-03-24T00:38:03.046722+0000","last_peered":"2024-03-24T00:38:03.046722+0000","last_clean":"2024-03-24T00:38:03.046722+0000","last_became_active":"2024-03-24T00:38:02.038982+0000","last_became_peered":"2024-03-24T00:38:02.038982+0000","last_unstale":"2024-03-24T00:38:03.046722+0000","last_undegraded":"2024-03-24T00:38:03.046722+0000","last_fullsized":"2024-03-24T00:38:03.046722+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:41:51.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043706+0000","last_change":"2024-03-24T00:38:01.044911+0000","last_active":"2024-03-24T00:38:03.043706+0000","last_peered":"2024-03-24T00:38:03.043706+0000","last_clean":"2024-03-24T00:38:03.043706+0000","last_became_active":"2024-03-24T00:38:01.042733+0000","last_became_peered":"2024-03-24T00:38:01.042733+0000","last_unstale":"2024-03-24T00:38:03.043706+0000","last_undegraded":"2024-03-24T00:38:03.043706+0000","last_fullsized":"2024-03-24T00:38:03.043706+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:41:50.116992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.047567+0000","last_change":"2024-03-24T00:38:02.041952+0000","last_active":"2024-03-24T00:38:02.047567+0000","last_peered":"2024-03-24T00:38:02.047567+0000","last_clean":"2024-03-24T00:38:02.047567+0000","last_became_active":"2024-03-24T00:38:02.041677+0000","last_became_peered":"2024-03-24T00:38:02.041677+0000","last_unstale":"2024-03-24T00:38:02.047567+0000","last_undegraded":"2024-03-24T00:38:02.047567+0000","last_fullsized":"2024-03-24T00:38:02.047567+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:43:26.480026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047276+0000","last_change":"2024-03-24T00:38:01.054368+0000","last_active":"2024-03-24T00:38:03.047276+0000","last_peered":"2024-03-24T00:38:03.047276+0000","last_clean":"2024-03-24T00:38:03.047276+0000","last_became_active":"2024-03-24T00:38:01.054053+0000","last_became_peered":"2024-03-24T00:38:01.054053+0000","last_unstale":"2024-03-24T00:38:03.047276+0000","last_undegraded":"2024-03-24T00:38:03.047276+0000","last_fullsized":"2024-03-24T00:38:03.047276+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:12:15.770803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061714+0000","last_change":"2024-03-24T00:38:02.057092+0000","last_active":"2024-03-24T00:38:03.061714+0000","last_peered":"2024-03-24T00:38:03.061714+0000","last_clean":"2024-03-24T00:38:03.061714+0000","last_became_active":"2024-03-24T00:38:02.056786+0000","last_became_peered":"2024-03-24T00:38:02.056786+0000","last_unstale":"2024-03-24T00:38:03.061714+0000","last_undegraded":"2024-03-24T00:38:03.061714+0000","last_fullsized":"2024-03-24T00:38:03.061714+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:00:23.166901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.030439+0000","last_change":"2024-03-24T00:38:01.056676+0000","last_active":"2024-03-24T00:38:02.030439+0000","last_peered":"2024-03-24T00:38:02.030439+0000","last_clean":"2024-03-24T00:38:02.030439+0000","last_became_active":"2024-03-24T00:38:01.056378+0000","last_became_peered":"2024-03-24T00:38:01.056378+0000","last_unstale":"2024-03-24T00:38:02.030439+0000","last_undegraded":"2024-03-24T00:38:02.030439+0000","last_fullsized":"2024-03-24T00:38:02.030439+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:56:33.496784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.062280+0000","last_change":"2024-03-24T00:38:02.057986+0000","last_active":"2024-03-24T00:38:03.062280+0000","last_peered":"2024-03-24T00:38:03.062280+0000","last_clean":"2024-03-24T00:38:03.062280+0000","last_became_active":"2024-03-24T00:38:02.057471+0000","last_became_peered":"2024-03-24T00:38:02.057471+0000","last_unstale":"2024-03-24T00:38:03.062280+0000","last_undegraded":"2024-03-24T00:38:03.062280+0000","last_fullsized":"2024-03-24T00:38:03.062280+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:20.216554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.042066+0000","last_change":"2024-03-24T00:38:01.048253+0000","last_active":"2024-03-24T00:38:02.042066+0000","last_peered":"2024-03-24T00:38:02.042066+0000","last_clean":"2024-03-24T00:38:02.042066+0000","last_became_active":"2024-03-24T00:38:01.047961+0000","last_became_peered":"2024-03-24T00:38:01.047961+0000","last_unstale":"2024-03-24T00:38:02.042066+0000","last_undegraded":"2024-03-24T00:38:02.042066+0000","last_fullsized":"2024-03-24T00:38:02.042066+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:33.629271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059592+0000","last_change":"2024-03-24T00:38:02.051972+0000","last_active":"2024-03-24T00:38:03.059592+0000","last_peered":"2024-03-24T00:38:03.059592+0000","last_clean":"2024-03-24T00:38:03.059592+0000","last_became_active":"2024-03-24T00:38:02.051647+0000","last_became_peered":"2024-03-24T00:38:02.051647+0000","last_unstale":"2024-03-24T00:38:03.059592+0000","last_undegraded":"2024-03-24T00:38:03.059592+0000","last_fullsized":"2024-03-24T00:38:03.059592+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:56.438539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043992+0000","last_change":"2024-03-24T00:38:01.043751+0000","last_active":"2024-03-24T00:38:03.043992+0000","last_peered":"2024-03-24T00:38:03.043992+0000","last_clean":"2024-03-24T00:38:03.043992+0000","last_became_active":"2024-03-24T00:38:01.042469+0000","last_became_peered":"2024-03-24T00:38:01.042469+0000","last_unstale":"2024-03-24T00:38:03.043992+0000","last_undegraded":"2024-03-24T00:38:03.043992+0000","last_fullsized":"2024-03-24T00:38:03.043992+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:15:19.531454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043919+0000","last_change":"2024-03-24T00:38:02.044985+0000","last_active":"2024-03-24T00:38:03.043919+0000","last_peered":"2024-03-24T00:38:03.043919+0000","last_clean":"2024-03-24T00:38:03.043919+0000","last_became_active":"2024-03-24T00:38:02.044645+0000","last_became_peered":"2024-03-24T00:38:02.044645+0000","last_unstale":"2024-03-24T00:38:03.043919+0000","last_undegraded":"2024-03-24T00:38:03.043919+0000","last_fullsized":"2024-03-24T00:38:03.043919+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:56:34.503377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"105'3","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.068329+0000","last_change":"2024-03-24T00:38:01.054223+0000","last_active":"2024-03-24T00:38:02.068329+0000","last_peered":"2024-03-24T00:38:02.068329+0000","last_clean":"2024-03-24T00:38:02.068329+0000","last_became_active":"2024-03-24T00:38:01.052988+0000","last_became_peered":"2024-03-24T00:38:01.052988+0000","last_unstale":"2024-03-24T00:38:02.068329+0000","last_undegraded":"2024-03-24T00:38:02.068329+0000","last_fullsized":"2024-03-24T00:38:02.068329+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:20.614028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.062158+0000","last_change":"2024-03-24T00:38:02.055777+0000","last_active":"2024-03-24T00:38:03.062158+0000","last_peered":"2024-03-24T00:38:03.062158+0000","last_clean":"2024-03-24T00:38:03.062158+0000","last_became_active":"2024-03-24T00:38:02.055434+0000","last_became_peered":"2024-03-24T00:38:02.055434+0000","last_unstale":"2024-03-24T00:38:03.062158+0000","last_undegraded":"2024-03-24T00:38:03.062158+0000","last_fullsized":"2024-03-24T00:38:03.062158+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:56:43.866424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"105'6","reported_seq":18,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.073685+0000","last_change":"2024-03-24T00:38:01.055839+0000","last_active":"2024-03-24T00:38:02.073685+0000","last_peered":"2024-03-24T00:38:02.073685+0000","last_clean":"2024-03-24T00:38:02.073685+0000","last_became_active":"2024-03-24T00:38:01.055497+0000","last_became_peered":"2024-03-24T00:38:01.055497+0000","last_unstale":"2024-03-24T00:38:02.073685+0000","last_undegraded":"2024-03-24T00:38:02.073685+0000","last_fullsized":"2024-03-24T00:38:02.073685+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:35.591064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043599+0000","last_change":"2024-03-24T00:38:02.046914+0000","last_active":"2024-03-24T00:38:03.043599+0000","last_peered":"2024-03-24T00:38:03.043599+0000","last_clean":"2024-03-24T00:38:03.043599+0000","last_became_active":"2024-03-24T00:38:02.046442+0000","last_became_peered":"2024-03-24T00:38:02.046442+0000","last_unstale":"2024-03-24T00:38:03.043599+0000","last_undegraded":"2024-03-24T00:38:03.043599+0000","last_fullsized":"2024-03-24T00:38:03.043599+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:33.285058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"105'2","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.070187+0000","last_change":"2024-03-24T00:38:01.033089+0000","last_active":"2024-03-24T00:38:02.070187+0000","last_peered":"2024-03-24T00:38:02.070187+0000","last_clean":"2024-03-24T00:38:02.070187+0000","last_became_active":"2024-03-24T00:38:01.032815+0000","last_became_peered":"2024-03-24T00:38:01.032815+0000","last_unstale":"2024-03-24T00:38:02.070187+0000","last_undegraded":"2024-03-24T00:38:02.070187+0000","last_fullsized":"2024-03-24T00:38:02.070187+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:13:35.909599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043852+0000","last_change":"2024-03-24T00:38:02.056007+0000","last_active":"2024-03-24T00:38:03.043852+0000","last_peered":"2024-03-24T00:38:03.043852+0000","last_clean":"2024-03-24T00:38:03.043852+0000","last_became_active":"2024-03-24T00:38:02.055708+0000","last_became_peered":"2024-03-24T00:38:02.055708+0000","last_unstale":"2024-03-24T00:38:03.043852+0000","last_undegraded":"2024-03-24T00:38:03.043852+0000","last_fullsized":"2024-03-24T00:38:03.043852+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:19.015495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059147+0000","last_change":"2024-03-24T00:38:01.052388+0000","last_active":"2024-03-24T00:38:03.059147+0000","last_peered":"2024-03-24T00:38:03.059147+0000","last_clean":"2024-03-24T00:38:03.059147+0000","last_became_active":"2024-03-24T00:38:01.052037+0000","last_became_peered":"2024-03-24T00:38:01.052037+0000","last_unstale":"2024-03-24T00:38:03.059147+0000","last_undegraded":"2024-03-24T00:38:03.059147+0000","last_fullsized":"2024-03-24T00:38:03.059147+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:26.448834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043396+0000","last_change":"2024-03-24T00:38:02.048249+0000","last_active":"2024-03-24T00:38:03.043396+0000","last_peered":"2024-03-24T00:38:03.043396+0000","last_clean":"2024-03-24T00:38:03.043396+0000","last_became_active":"2024-03-24T00:38:02.047969+0000","last_became_peered":"2024-03-24T00:38:02.047969+0000","last_unstale":"2024-03-24T00:38:03.043396+0000","last_undegraded":"2024-03-24T00:38:03.043396+0000","last_fullsized":"2024-03-24T00:38:03.043396+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:44.746992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"105'8","reported_seq":20,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.073470+0000","last_change":"2024-03-24T00:38:01.056110+0000","last_active":"2024-03-24T00:38:02.073470+0000","last_peered":"2024-03-24T00:38:02.073470+0000","last_clean":"2024-03-24T00:38:02.073470+0000","last_became_active":"2024-03-24T00:38:01.055840+0000","last_became_peered":"2024-03-24T00:38:01.055840+0000","last_unstale":"2024-03-24T00:38:02.073470+0000","last_undegraded":"2024-03-24T00:38:02.073470+0000","last_fullsized":"2024-03-24T00:38:02.073470+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:02:51.052161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.053560+0000","last_change":"2024-03-24T00:38:02.051368+0000","last_active":"2024-03-24T00:38:02.053560+0000","last_peered":"2024-03-24T00:38:02.053560+0000","last_clean":"2024-03-24T00:38:02.053560+0000","last_became_active":"2024-03-24T00:38:02.051020+0000","last_became_peered":"2024-03-24T00:38:02.051020+0000","last_unstale":"2024-03-24T00:38:02.053560+0000","last_undegraded":"2024-03-24T00:38:02.053560+0000","last_fullsized":"2024-03-24T00:38:02.053560+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:12:30.835729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"105'1","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.067397+0000","last_change":"2024-03-24T00:38:01.046814+0000","last_active":"2024-03-24T00:38:02.067397+0000","last_peered":"2024-03-24T00:38:02.067397+0000","last_clean":"2024-03-24T00:38:02.067397+0000","last_became_active":"2024-03-24T00:38:01.046520+0000","last_became_peered":"2024-03-24T00:38:01.046520+0000","last_unstale":"2024-03-24T00:38:02.067397+0000","last_undegraded":"2024-03-24T00:38:02.067397+0000","last_fullsized":"2024-03-24T00:38:02.067397+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:43.204857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.052918+0000","last_change":"2024-03-24T00:38:02.047450+0000","last_active":"2024-03-24T00:38:02.052918+0000","last_peered":"2024-03-24T00:38:02.052918+0000","last_clean":"2024-03-24T00:38:02.052918+0000","last_became_active":"2024-03-24T00:38:02.046202+0000","last_became_peered":"2024-03-24T00:38:02.046202+0000","last_unstale":"2024-03-24T00:38:02.052918+0000","last_undegraded":"2024-03-24T00:38:02.052918+0000","last_fullsized":"2024-03-24T00:38:02.052918+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:18.779535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"105'2","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.070953+0000","last_change":"2024-03-24T00:38:01.044025+0000","last_active":"2024-03-24T00:38:02.070953+0000","last_peered":"2024-03-24T00:38:02.070953+0000","last_clean":"2024-03-24T00:38:02.070953+0000","last_became_active":"2024-03-24T00:38:01.042969+0000","last_became_peered":"2024-03-24T00:38:01.042969+0000","last_unstale":"2024-03-24T00:38:02.070953+0000","last_undegraded":"2024-03-24T00:38:02.070953+0000","last_fullsized":"2024-03-24T00:38:02.070953+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:24.711793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048162+0000","last_change":"2024-03-24T00:38:02.062532+0000","last_active":"2024-03-24T00:38:03.048162+0000","last_peered":"2024-03-24T00:38:03.048162+0000","last_clean":"2024-03-24T00:38:03.048162+0000","last_became_active":"2024-03-24T00:38:02.062185+0000","last_became_peered":"2024-03-24T00:38:02.062185+0000","last_unstale":"2024-03-24T00:38:03.048162+0000","last_undegraded":"2024-03-24T00:38:03.048162+0000","last_fullsized":"2024-03-24T00:38:03.048162+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:21.620621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059378+0000","last_change":"2024-03-24T00:38:01.053941+0000","last_active":"2024-03-24T00:38:03.059378+0000","last_peered":"2024-03-24T00:38:03.059378+0000","last_clean":"2024-03-24T00:38:03.059378+0000","last_became_active":"2024-03-24T00:38:01.053530+0000","last_became_peered":"2024-03-24T00:38:01.053530+0000","last_unstale":"2024-03-24T00:38:03.059378+0000","last_undegraded":"2024-03-24T00:38:03.059378+0000","last_fullsized":"2024-03-24T00:38:03.059378+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:52.270501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048402+0000","last_change":"2024-03-24T00:38:02.052952+0000","last_active":"2024-03-24T00:38:03.048402+0000","last_peered":"2024-03-24T00:38:03.048402+0000","last_clean":"2024-03-24T00:38:03.048402+0000","last_became_active":"2024-03-24T00:38:02.052539+0000","last_became_peered":"2024-03-24T00:38:02.052539+0000","last_unstale":"2024-03-24T00:38:03.048402+0000","last_undegraded":"2024-03-24T00:38:03.048402+0000","last_fullsized":"2024-03-24T00:38:03.048402+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:26.764791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061996+0000","last_change":"2024-03-24T00:38:01.041124+0000","last_active":"2024-03-24T00:38:03.061996+0000","last_peered":"2024-03-24T00:38:03.061996+0000","last_clean":"2024-03-24T00:38:03.061996+0000","last_became_active":"2024-03-24T00:38:01.040822+0000","last_became_peered":"2024-03-24T00:38:01.040822+0000","last_unstale":"2024-03-24T00:38:03.061996+0000","last_undegraded":"2024-03-24T00:38:03.061996+0000","last_fullsized":"2024-03-24T00:38:03.061996+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:17.772943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048240+0000","last_change":"2024-03-24T00:38:02.052965+0000","last_active":"2024-03-24T00:38:03.048240+0000","last_peered":"2024-03-24T00:38:03.048240+0000","last_clean":"2024-03-24T00:38:03.048240+0000","last_became_active":"2024-03-24T00:38:02.050907+0000","last_became_peered":"2024-03-24T00:38:02.050907+0000","last_unstale":"2024-03-24T00:38:03.048240+0000","last_undegraded":"2024-03-24T00:38:03.048240+0000","last_fullsized":"2024-03-24T00:38:03.048240+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:44:45.455222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"105'2","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.073167+0000","last_change":"2024-03-24T00:38:01.047408+0000","last_active":"2024-03-24T00:38:02.073167+0000","last_peered":"2024-03-24T00:38:02.073167+0000","last_clean":"2024-03-24T00:38:02.073167+0000","last_became_active":"2024-03-24T00:38:01.046612+0000","last_became_peered":"2024-03-24T00:38:01.046612+0000","last_unstale":"2024-03-24T00:38:02.073167+0000","last_undegraded":"2024-03-24T00:38:02.073167+0000","last_fullsized":"2024-03-24T00:38:02.073167+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:12.367978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.054276+0000","last_change":"2024-03-24T00:38:02.050285+0000","last_active":"2024-03-24T00:38:02.054276+0000","last_peered":"2024-03-24T00:38:02.054276+0000","last_clean":"2024-03-24T00:38:02.054276+0000","last_became_active":"2024-03-24T00:38:02.049387+0000","last_became_peered":"2024-03-24T00:38:02.049387+0000","last_unstale":"2024-03-24T00:38:02.054276+0000","last_undegraded":"2024-03-24T00:38:02.054276+0000","last_fullsized":"2024-03-24T00:38:02.054276+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:26.331868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.038938+0000","last_change":"2024-03-24T00:38:01.050049+0000","last_active":"2024-03-24T00:38:02.038938+0000","last_peered":"2024-03-24T00:38:02.038938+0000","last_clean":"2024-03-24T00:38:02.038938+0000","last_became_active":"2024-03-24T00:38:01.049789+0000","last_became_peered":"2024-03-24T00:38:01.049789+0000","last_unstale":"2024-03-24T00:38:02.038938+0000","last_undegraded":"2024-03-24T00:38:02.038938+0000","last_fullsized":"2024-03-24T00:38:02.038938+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:54.658870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046779+0000","last_change":"2024-03-24T00:38:02.064234+0000","last_active":"2024-03-24T00:38:03.046779+0000","last_peered":"2024-03-24T00:38:03.046779+0000","last_clean":"2024-03-24T00:38:03.046779+0000","last_became_active":"2024-03-24T00:38:02.063928+0000","last_became_peered":"2024-03-24T00:38:02.063928+0000","last_unstale":"2024-03-24T00:38:03.046779+0000","last_undegraded":"2024-03-24T00:38:03.046779+0000","last_fullsized":"2024-03-24T00:38:03.046779+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:50.434313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.036097+0000","last_change":"2024-03-24T00:38:01.046058+0000","last_active":"2024-03-24T00:38:02.036097+0000","last_peered":"2024-03-24T00:38:02.036097+0000","last_clean":"2024-03-24T00:38:02.036097+0000","last_became_active":"2024-03-24T00:38:01.045759+0000","last_became_peered":"2024-03-24T00:38:01.045759+0000","last_unstale":"2024-03-24T00:38:02.036097+0000","last_undegraded":"2024-03-24T00:38:02.036097+0000","last_fullsized":"2024-03-24T00:38:02.036097+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:42.101640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043383+0000","last_change":"2024-03-24T00:38:02.049311+0000","last_active":"2024-03-24T00:38:03.043383+0000","last_peered":"2024-03-24T00:38:03.043383+0000","last_clean":"2024-03-24T00:38:03.043383+0000","last_became_active":"2024-03-24T00:38:02.048183+0000","last_became_peered":"2024-03-24T00:38:02.048183+0000","last_unstale":"2024-03-24T00:38:03.043383+0000","last_undegraded":"2024-03-24T00:38:03.043383+0000","last_fullsized":"2024-03-24T00:38:03.043383+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:39.197472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.030568+0000","last_change":"2024-03-24T00:38:01.051165+0000","last_active":"2024-03-24T00:38:02.030568+0000","last_peered":"2024-03-24T00:38:02.030568+0000","last_clean":"2024-03-24T00:38:02.030568+0000","last_became_active":"2024-03-24T00:38:01.050766+0000","last_became_peered":"2024-03-24T00:38:01.050766+0000","last_unstale":"2024-03-24T00:38:02.030568+0000","last_undegraded":"2024-03-24T00:38:02.030568+0000","last_fullsized":"2024-03-24T00:38:02.030568+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:13:15.099360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059390+0000","last_change":"2024-03-24T00:38:02.052352+0000","last_active":"2024-03-24T00:38:03.059390+0000","last_peered":"2024-03-24T00:38:03.059390+0000","last_clean":"2024-03-24T00:38:03.059390+0000","last_became_active":"2024-03-24T00:38:02.051268+0000","last_became_peered":"2024-03-24T00:38:02.051268+0000","last_unstale":"2024-03-24T00:38:03.059390+0000","last_undegraded":"2024-03-24T00:38:03.059390+0000","last_fullsized":"2024-03-24T00:38:03.059390+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:38:15.571575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043889+0000","last_change":"2024-03-24T00:38:01.047402+0000","last_active":"2024-03-24T00:38:03.043889+0000","last_peered":"2024-03-24T00:38:03.043889+0000","last_clean":"2024-03-24T00:38:03.043889+0000","last_became_active":"2024-03-24T00:38:01.047123+0000","last_became_peered":"2024-03-24T00:38:01.047123+0000","last_unstale":"2024-03-24T00:38:03.043889+0000","last_undegraded":"2024-03-24T00:38:03.043889+0000","last_fullsized":"2024-03-24T00:38:03.043889+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:38:14.564982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.040371+0000","last_change":"2024-03-24T00:38:02.033827+0000","last_active":"2024-03-24T00:38:02.040371+0000","last_peered":"2024-03-24T00:38:02.040371+0000","last_clean":"2024-03-24T00:38:02.040371+0000","last_became_active":"2024-03-24T00:38:02.033541+0000","last_became_peered":"2024-03-24T00:38:02.033541+0000","last_unstale":"2024-03-24T00:38:02.040371+0000","last_undegraded":"2024-03-24T00:38:02.040371+0000","last_fullsized":"2024-03-24T00:38:02.040371+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:05:48.731012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"106'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.060943+0000","last_change":"2024-03-24T00:38:01.044230+0000","last_active":"2024-03-24T00:38:03.060943+0000","last_peered":"2024-03-24T00:38:03.060943+0000","last_clean":"2024-03-24T00:38:03.060943+0000","last_became_active":"2024-03-24T00:38:01.043934+0000","last_became_peered":"2024-03-24T00:38:01.043934+0000","last_unstale":"2024-03-24T00:38:03.060943+0000","last_undegraded":"2024-03-24T00:38:03.060943+0000","last_fullsized":"2024-03-24T00:38:03.060943+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:43:54.494418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.046816+0000","last_change":"2024-03-24T00:38:02.041941+0000","last_active":"2024-03-24T00:38:02.046816+0000","last_peered":"2024-03-24T00:38:02.046816+0000","last_clean":"2024-03-24T00:38:02.046816+0000","last_became_active":"2024-03-24T00:38:02.041656+0000","last_became_peered":"2024-03-24T00:38:02.041656+0000","last_unstale":"2024-03-24T00:38:02.046816+0000","last_undegraded":"2024-03-24T00:38:02.046816+0000","last_fullsized":"2024-03-24T00:38:02.046816+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:16.674791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.030558+0000","last_change":"2024-03-24T00:38:01.048064+0000","last_active":"2024-03-24T00:38:02.030558+0000","last_peered":"2024-03-24T00:38:02.030558+0000","last_clean":"2024-03-24T00:38:02.030558+0000","last_became_active":"2024-03-24T00:38:01.047758+0000","last_became_peered":"2024-03-24T00:38:01.047758+0000","last_unstale":"2024-03-24T00:38:02.030558+0000","last_undegraded":"2024-03-24T00:38:02.030558+0000","last_fullsized":"2024-03-24T00:38:02.030558+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:44:44.448630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059356+0000","last_change":"2024-03-24T00:38:02.052725+0000","last_active":"2024-03-24T00:38:03.059356+0000","last_peered":"2024-03-24T00:38:03.059356+0000","last_clean":"2024-03-24T00:38:03.059356+0000","last_became_active":"2024-03-24T00:38:02.052340+0000","last_became_peered":"2024-03-24T00:38:02.052340+0000","last_unstale":"2024-03-24T00:38:03.059356+0000","last_undegraded":"2024-03-24T00:38:03.059356+0000","last_fullsized":"2024-03-24T00:38:03.059356+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:15:20.538046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043956+0000","last_change":"2024-03-24T00:38:01.046446+0000","last_active":"2024-03-24T00:38:03.043956+0000","last_peered":"2024-03-24T00:38:03.043956+0000","last_clean":"2024-03-24T00:38:03.043956+0000","last_became_active":"2024-03-24T00:38:01.044068+0000","last_became_peered":"2024-03-24T00:38:01.044068+0000","last_unstale":"2024-03-24T00:38:03.043956+0000","last_undegraded":"2024-03-24T00:38:03.043956+0000","last_fullsized":"2024-03-24T00:38:03.043956+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:37.758100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048024+0000","last_change":"2024-03-24T00:38:02.050141+0000","last_active":"2024-03-24T00:38:03.048024+0000","last_peered":"2024-03-24T00:38:03.048024+0000","last_clean":"2024-03-24T00:38:03.048024+0000","last_became_active":"2024-03-24T00:38:02.049816+0000","last_became_peered":"2024-03-24T00:38:02.049816+0000","last_unstale":"2024-03-24T00:38:03.048024+0000","last_undegraded":"2024-03-24T00:38:03.048024+0000","last_fullsized":"2024-03-24T00:38:03.048024+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:22.013335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061854+0000","last_change":"2024-03-24T00:38:01.045695+0000","last_active":"2024-03-24T00:38:03.061854+0000","last_peered":"2024-03-24T00:38:03.061854+0000","last_clean":"2024-03-24T00:38:03.061854+0000","last_became_active":"2024-03-24T00:38:01.045361+0000","last_became_peered":"2024-03-24T00:38:01.045361+0000","last_unstale":"2024-03-24T00:38:03.061854+0000","last_undegraded":"2024-03-24T00:38:03.061854+0000","last_fullsized":"2024-03-24T00:38:03.061854+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:46:44.447740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046898+0000","last_change":"2024-03-24T00:38:02.051378+0000","last_active":"2024-03-24T00:38:03.046898+0000","last_peered":"2024-03-24T00:38:03.046898+0000","last_clean":"2024-03-24T00:38:03.046898+0000","last_became_active":"2024-03-24T00:38:02.050709+0000","last_became_peered":"2024-03-24T00:38:02.050709+0000","last_unstale":"2024-03-24T00:38:03.046898+0000","last_undegraded":"2024-03-24T00:38:03.046898+0000","last_fullsized":"2024-03-24T00:38:03.046898+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:36.346943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046522+0000","last_change":"2024-03-24T00:38:01.049603+0000","last_active":"2024-03-24T00:38:03.046522+0000","last_peered":"2024-03-24T00:38:03.046522+0000","last_clean":"2024-03-24T00:38:03.046522+0000","last_became_active":"2024-03-24T00:38:01.049258+0000","last_became_peered":"2024-03-24T00:38:01.049258+0000","last_unstale":"2024-03-24T00:38:03.046522+0000","last_undegraded":"2024-03-24T00:38:03.046522+0000","last_fullsized":"2024-03-24T00:38:03.046522+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:19.209962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046941+0000","last_change":"2024-03-24T00:38:02.055424+0000","last_active":"2024-03-24T00:38:03.046941+0000","last_peered":"2024-03-24T00:38:03.046941+0000","last_clean":"2024-03-24T00:38:03.046941+0000","last_became_active":"2024-03-24T00:38:02.055133+0000","last_became_peered":"2024-03-24T00:38:02.055133+0000","last_unstale":"2024-03-24T00:38:03.046941+0000","last_undegraded":"2024-03-24T00:38:03.046941+0000","last_fullsized":"2024-03-24T00:38:03.046941+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:35:59.814600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.039843+0000","last_change":"2024-03-24T00:38:01.050832+0000","last_active":"2024-03-24T00:38:02.039843+0000","last_peered":"2024-03-24T00:38:02.039843+0000","last_clean":"2024-03-24T00:38:02.039843+0000","last_became_active":"2024-03-24T00:38:01.050554+0000","last_became_peered":"2024-03-24T00:38:01.050554+0000","last_unstale":"2024-03-24T00:38:02.039843+0000","last_undegraded":"2024-03-24T00:38:02.039843+0000","last_fullsized":"2024-03-24T00:38:02.039843+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:07.247959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.048851+0000","last_change":"2024-03-24T00:38:02.045232+0000","last_active":"2024-03-24T00:38:02.048851+0000","last_peered":"2024-03-24T00:38:02.048851+0000","last_clean":"2024-03-24T00:38:02.048851+0000","last_became_active":"2024-03-24T00:38:02.044967+0000","last_became_peered":"2024-03-24T00:38:02.044967+0000","last_unstale":"2024-03-24T00:38:02.048851+0000","last_undegraded":"2024-03-24T00:38:02.048851+0000","last_fullsized":"2024-03-24T00:38:02.048851+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:28:06.561931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.030617+0000","last_change":"2024-03-24T00:38:01.047818+0000","last_active":"2024-03-24T00:38:02.030617+0000","last_peered":"2024-03-24T00:38:02.030617+0000","last_clean":"2024-03-24T00:38:02.030617+0000","last_became_active":"2024-03-24T00:38:01.047272+0000","last_became_peered":"2024-03-24T00:38:01.047272+0000","last_unstale":"2024-03-24T00:38:02.030617+0000","last_undegraded":"2024-03-24T00:38:02.030617+0000","last_fullsized":"2024-03-24T00:38:02.030617+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:25.758199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043383+0000","last_change":"2024-03-24T00:38:02.056298+0000","last_active":"2024-03-24T00:38:03.043383+0000","last_peered":"2024-03-24T00:38:03.043383+0000","last_clean":"2024-03-24T00:38:03.043383+0000","last_became_active":"2024-03-24T00:38:02.055729+0000","last_became_peered":"2024-03-24T00:38:02.055729+0000","last_unstale":"2024-03-24T00:38:03.043383+0000","last_undegraded":"2024-03-24T00:38:03.043383+0000","last_fullsized":"2024-03-24T00:38:03.043383+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:50.434313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061843+0000","last_change":"2024-03-24T00:38:01.049259+0000","last_active":"2024-03-24T00:38:03.061843+0000","last_peered":"2024-03-24T00:38:03.061843+0000","last_clean":"2024-03-24T00:38:03.061843+0000","last_became_active":"2024-03-24T00:38:01.048839+0000","last_became_peered":"2024-03-24T00:38:01.048839+0000","last_unstale":"2024-03-24T00:38:03.061843+0000","last_undegraded":"2024-03-24T00:38:03.061843+0000","last_fullsized":"2024-03-24T00:38:03.061843+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:48.122676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":344,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047638+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:38:03.047638+0000","last_peered":"2024-03-24T00:38:03.047638+0000","last_clean":"2024-03-24T00:38:03.047638+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:38:03.047638+0000","last_undegraded":"2024-03-24T00:38:03.047638+0000","last_fullsized":"2024-03-24T00:38:03.047638+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574936,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1908736,"data_stored":1159960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574937,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40788,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331052,"statfs":{"total":96636764160,"available":96594997248,"internally_reserved":0,"allocated":1310720,"data_stored":569046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36636,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1327104,"data_stored":575880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574937,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50520,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321320,"statfs":{"total":96636764160,"available":96585031680,"internally_reserved":0,"allocated":1318912,"data_stored":575276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574936,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46376,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325464,"statfs":{"total":96636764160,"available":96589275136,"internally_reserved":0,"allocated":1335296,"data_stored":576958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574937,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1306624,"data_stored":568976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40792,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331048,"statfs":{"total":96636764160,"available":96594993152,"internally_reserved":0,"allocated":1314816,"data_stored":569100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574936,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1921024,"data_stored":1161068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:38:06.153 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:38:06.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f6461800640 1 Processor -- start 2024-03-24T00:38:06.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f6461800640 1 -- start start 2024-03-24T00:38:06.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f6461800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f645c1477b0 0x7f645c06e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:06.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f6461800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f645c147b90 0x7f645c06ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:06.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f6461800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c0addc0 con 0x7f645c0ad8e0 2024-03-24T00:38:06.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f6461800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f645c0ae9d0 con 0x7f645c147b90 2024-03-24T00:38:06.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f6461800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f645c0ae0a0 con 0x7f645c1477b0 2024-03-24T00:38:06.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f645b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f645c147b90 0x7f645c06ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:06.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f645b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f645c147b90 0x7f645c06ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48082/0 (socket says 172.21.15.40:48082) 2024-03-24T00:38:06.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f645aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f645c1477b0 0x7f645c06e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:06.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f645b400640 1 -- 172.21.15.40:0/366553954 learned_addr learned my addr 172.21.15.40:0/366553954 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:06.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f645aa00640 1 -- 172.21.15.40:0/366553954 >> v2:172.21.15.40:3300/0 conn(0x7f645c147b90 msgr2=0x7f645c06ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:06.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f645aa00640 1 --2- 172.21.15.40:0/366553954 >> v2:172.21.15.40:3300/0 conn(0x7f645c147b90 0x7f645c06ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f645aa00640 1 -- 172.21.15.40:0/366553954 >> v1:172.21.15.53:6789/0 conn(0x7f645c0ad8e0 legacy=0x7f645c15e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f645aa00640 1 -- 172.21.15.40:0/366553954 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645c0ae380 con 0x7f645c1477b0 2024-03-24T00:38:06.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f645a000640 1 -- 172.21.15.40:0/366553954 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713120941 0 0) 0x7f645c0addc0 con 0x7f645c0ad8e0 2024-03-24T00:38:06.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.329+0000 7f645b400640 1 --2- 172.21.15.40:0/366553954 >> v2:172.21.15.40:3300/0 conn(0x7f645c147b90 0x7f645c06ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:06.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.333+0000 7f645aa00640 1 --2- 172.21.15.40:0/366553954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645c1477b0 0x7f645c06e520 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f645c073ff0 tx=0x7f644c01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56db8b1c73a72b6e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:06.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.333+0000 7f645a000640 1 -- 172.21.15.40:0/366553954 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c0b1070 con 0x7f645c1477b0 2024-03-24T00:38:06.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.333+0000 7f645a000640 1 -- 172.21.15.40:0/366553954 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644c0a8050 con 0x7f645c1477b0 2024-03-24T00:38:06.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.333+0000 7f645a000640 1 -- 172.21.15.40:0/366553954 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c0ac860 con 0x7f645c1477b0 2024-03-24T00:38:06.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.333+0000 7f6461800640 1 -- 172.21.15.40:0/366553954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645c1477b0 msgr2=0x7f645c06e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.333+0000 7f6461800640 1 --2- 172.21.15.40:0/366553954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645c1477b0 0x7f645c06e520 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f645c073ff0 tx=0x7f644c01c3c0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.333+0000 7f6461800640 1 -- 172.21.15.40:0/366553954 shutdown_connections 2024-03-24T00:38:06.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.333+0000 7f6461800640 1 --2- 172.21.15.40:0/366553954 >> v2:172.21.15.40:3300/0 conn(0x7f645c147b90 0x7f645c06ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.333+0000 7f6461800640 1 --2- 172.21.15.40:0/366553954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645c1477b0 0x7f645c06e520 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.333+0000 7f6461800640 1 -- 172.21.15.40:0/366553954 >> 172.21.15.40:0/366553954 conn(0x7f645c0b1d40 msgr2=0x7f645c0b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:06.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.333+0000 7f6461800640 1 -- 172.21.15.40:0/366553954 shutdown_connections 2024-03-24T00:38:06.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.333+0000 7f6461800640 1 -- 172.21.15.40:0/366553954 wait complete. 2024-03-24T00:38:06.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6461800640 1 Processor -- start 2024-03-24T00:38:06.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6461800640 1 -- start start 2024-03-24T00:38:06.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6461800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645c0ad8e0 0x7f645c13e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:06.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6461800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f645c1477b0 0x7f645c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6461800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f645c147b90 0x7f645c145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6461800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c16b1d0 con 0x7f645c1477b0 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6461800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645c15f540 con 0x7f645c0ad8e0 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6461800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645c16b350 con 0x7f645c147b90 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6453e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f645c1477b0 0x7f645c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f645aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645c0ad8e0 0x7f645c13e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f645aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645c0ad8e0 0x7f645c13e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:39450/0 (socket says 172.21.15.40:39450) 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f645aa00640 1 -- 172.21.15.40:0/793608783 learned_addr learned my addr 172.21.15.40:0/793608783 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f645b400640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f645c147b90 0x7f645c145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f645aa00640 1 -- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f645c147b90 msgr2=0x7f645c145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f645aa00640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f645c147b90 0x7f645c145f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f645aa00640 1 -- 172.21.15.40:0/793608783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f645c1477b0 msgr2=0x7f645c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f645aa00640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f645c1477b0 0x7f645c0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f645aa00640 1 -- 172.21.15.40:0/793608783 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645c0ccfc0 con 0x7f645c0ad8e0 2024-03-24T00:38:06.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6453e00640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f645c1477b0 0x7f645c0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:06.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f645aa00640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645c0ad8e0 0x7f645c13e140 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f644c0af040 tx=0x7f644c01ac00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:06.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6452a00640 1 -- 172.21.15.40:0/793608783 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c0b1050 con 0x7f645c0ad8e0 2024-03-24T00:38:06.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6452a00640 1 -- 172.21.15.40:0/793608783 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644c0bd070 con 0x7f645c0ad8e0 2024-03-24T00:38:06.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6452a00640 1 -- 172.21.15.40:0/793608783 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c0b8690 con 0x7f645c0ad8e0 2024-03-24T00:38:06.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6461800640 1 -- 172.21.15.40:0/793608783 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645c13f1a0 con 0x7f645c0ad8e0 2024-03-24T00:38:06.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6461800640 1 -- 172.21.15.40:0/793608783 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645c13f760 con 0x7f645c0ad8e0 2024-03-24T00:38:06.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.337+0000 7f6461800640 1 -- 172.21.15.40:0/793608783 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645c068170 con 0x7f645c0ad8e0 2024-03-24T00:38:06.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.341+0000 7f6452a00640 1 -- 172.21.15.40:0/793608783 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f644c0b8830 con 0x7f645c0ad8e0 2024-03-24T00:38:06.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.341+0000 7f6452a00640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6438067660 0x7f6438069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:06.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.341+0000 7f6452a00640 1 -- 172.21.15.40:0/793608783 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f644c120100 con 0x7f645c0ad8e0 2024-03-24T00:38:06.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.341+0000 7f6453e00640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6438067660 0x7f6438069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:06.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.341+0000 7f6453e00640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6438067660 0x7f6438069b20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f645c072ea0 tx=0x7f6444009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:06.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.345+0000 7f6452a00640 1 -- 172.21.15.40:0/793608783 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f644c016070 con 0x7f645c0ad8e0 2024-03-24T00:38:06.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.505+0000 7f6461800640 1 -- 172.21.15.40:0/793608783 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f645c13e680 con 0x7f6438067660 2024-03-24T00:38:06.515 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.509+0000 7f6452a00640 1 -- 172.21.15.40:0/793608783 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252453 (secure 0 0 0) 0x7f645c13e680 con 0x7f6438067660 2024-03-24T00:38:06.517 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:06.519 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:38:06.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.513+0000 7f6461800640 1 -- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6438067660 msgr2=0x7f6438069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.513+0000 7f6461800640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6438067660 0x7f6438069b20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f645c072ea0 tx=0x7f6444009040 comp rx=0 tx=0).stop 2024-03-24T00:38:06.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.513+0000 7f6461800640 1 -- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645c0ad8e0 msgr2=0x7f645c13e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.513+0000 7f6461800640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645c0ad8e0 0x7f645c13e140 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f644c0af040 tx=0x7f644c01ac00 comp rx=0 tx=0).stop 2024-03-24T00:38:06.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.513+0000 7f6461800640 1 -- 172.21.15.40:0/793608783 shutdown_connections 2024-03-24T00:38:06.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.513+0000 7f6461800640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6438067660 0x7f6438069b20 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.513+0000 7f6461800640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f645c147b90 0x7f645c145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.513+0000 7f6461800640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f645c1477b0 0x7f645c0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.513+0000 7f6461800640 1 --2- 172.21.15.40:0/793608783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f645c0ad8e0 0x7f645c13e140 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.513+0000 7f6461800640 1 -- 172.21.15.40:0/793608783 >> 172.21.15.40:0/793608783 conn(0x7f645c0b1d40 msgr2=0x7f645c160ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:06.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.513+0000 7f6461800640 1 -- 172.21.15.40:0/793608783 shutdown_connections 2024-03-24T00:38:06.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.513+0000 7f6461800640 1 -- 172.21.15.40:0/793608783 wait complete. 2024-03-24T00:38:06.537 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-03-24T00:38:05.281916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":71,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":326156,"kb_used_data":11468,"kb_used_omap":12,"kb_used_meta":314547,"kb_avail":754648564,"statfs":{"total":773094113280,"available":772760129536,"internally_reserved":0,"allocated":11743232,"data_stored":5756264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":322096690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.264204"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.055220+0000","last_change":"2024-03-24T00:38:02.053053+0000","last_active":"2024-03-24T00:38:02.055220+0000","last_peered":"2024-03-24T00:38:02.055220+0000","last_clean":"2024-03-24T00:38:02.055220+0000","last_became_active":"2024-03-24T00:38:02.052642+0000","last_became_peered":"2024-03-24T00:38:02.052642+0000","last_unstale":"2024-03-24T00:38:02.055220+0000","last_undegraded":"2024-03-24T00:38:02.055220+0000","last_fullsized":"2024-03-24T00:38:02.055220+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:23:20.445022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047344+0000","last_change":"2024-03-24T00:38:02.039253+0000","last_active":"2024-03-24T00:38:03.047344+0000","last_peered":"2024-03-24T00:38:03.047344+0000","last_clean":"2024-03-24T00:38:03.047344+0000","last_became_active":"2024-03-24T00:38:02.038694+0000","last_became_peered":"2024-03-24T00:38:02.038694+0000","last_unstale":"2024-03-24T00:38:03.047344+0000","last_undegraded":"2024-03-24T00:38:03.047344+0000","last_fullsized":"2024-03-24T00:38:03.047344+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:05:20.785596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047758+0000","last_change":"2024-03-24T00:38:02.056053+0000","last_active":"2024-03-24T00:38:03.047758+0000","last_peered":"2024-03-24T00:38:03.047758+0000","last_clean":"2024-03-24T00:38:03.047758+0000","last_became_active":"2024-03-24T00:38:02.051992+0000","last_became_peered":"2024-03-24T00:38:02.051992+0000","last_unstale":"2024-03-24T00:38:03.047758+0000","last_undegraded":"2024-03-24T00:38:03.047758+0000","last_fullsized":"2024-03-24T00:38:03.047758+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:05.858925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.041135+0000","last_change":"2024-03-24T00:38:02.033932+0000","last_active":"2024-03-24T00:38:02.041135+0000","last_peered":"2024-03-24T00:38:02.041135+0000","last_clean":"2024-03-24T00:38:02.041135+0000","last_became_active":"2024-03-24T00:38:02.033651+0000","last_became_peered":"2024-03-24T00:38:02.033651+0000","last_unstale":"2024-03-24T00:38:02.041135+0000","last_undegraded":"2024-03-24T00:38:02.041135+0000","last_fullsized":"2024-03-24T00:38:02.041135+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:57.560121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.052166+0000","last_change":"2024-03-24T00:38:02.047238+0000","last_active":"2024-03-24T00:38:02.052166+0000","last_peered":"2024-03-24T00:38:02.052166+0000","last_clean":"2024-03-24T00:38:02.052166+0000","last_became_active":"2024-03-24T00:38:02.046942+0000","last_became_peered":"2024-03-24T00:38:02.046942+0000","last_unstale":"2024-03-24T00:38:02.052166+0000","last_undegraded":"2024-03-24T00:38:02.052166+0000","last_fullsized":"2024-03-24T00:38:02.052166+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:27:30.720526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.053030+0000","last_change":"2024-03-24T00:38:02.049961+0000","last_active":"2024-03-24T00:38:02.053030+0000","last_peered":"2024-03-24T00:38:02.053030+0000","last_clean":"2024-03-24T00:38:02.053030+0000","last_became_active":"2024-03-24T00:38:02.049623+0000","last_became_peered":"2024-03-24T00:38:02.049623+0000","last_unstale":"2024-03-24T00:38:02.053030+0000","last_undegraded":"2024-03-24T00:38:02.053030+0000","last_fullsized":"2024-03-24T00:38:02.053030+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:46:45.454332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061753+0000","last_change":"2024-03-24T00:38:02.056781+0000","last_active":"2024-03-24T00:38:03.061753+0000","last_peered":"2024-03-24T00:38:03.061753+0000","last_clean":"2024-03-24T00:38:03.061753+0000","last_became_active":"2024-03-24T00:38:02.053927+0000","last_became_peered":"2024-03-24T00:38:02.053927+0000","last_unstale":"2024-03-24T00:38:03.061753+0000","last_undegraded":"2024-03-24T00:38:03.061753+0000","last_fullsized":"2024-03-24T00:38:03.061753+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:35:21.632806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046615+0000","last_change":"2024-03-24T00:38:02.051319+0000","last_active":"2024-03-24T00:38:03.046615+0000","last_peered":"2024-03-24T00:38:03.046615+0000","last_clean":"2024-03-24T00:38:03.046615+0000","last_became_active":"2024-03-24T00:38:02.050929+0000","last_became_peered":"2024-03-24T00:38:02.050929+0000","last_unstale":"2024-03-24T00:38:03.046615+0000","last_undegraded":"2024-03-24T00:38:03.046615+0000","last_fullsized":"2024-03-24T00:38:03.046615+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:44.746992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.048328+0000","last_change":"2024-03-24T00:38:02.042223+0000","last_active":"2024-03-24T00:38:02.048328+0000","last_peered":"2024-03-24T00:38:02.048328+0000","last_clean":"2024-03-24T00:38:02.048328+0000","last_became_active":"2024-03-24T00:38:02.041336+0000","last_became_peered":"2024-03-24T00:38:02.041336+0000","last_unstale":"2024-03-24T00:38:02.048328+0000","last_undegraded":"2024-03-24T00:38:02.048328+0000","last_fullsized":"2024-03-24T00:38:02.048328+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:48:17.082488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.058940+0000","last_change":"2024-03-24T00:38:02.057633+0000","last_active":"2024-03-24T00:38:02.058940+0000","last_peered":"2024-03-24T00:38:02.058940+0000","last_clean":"2024-03-24T00:38:02.058940+0000","last_became_active":"2024-03-24T00:38:02.057309+0000","last_became_peered":"2024-03-24T00:38:02.057309+0000","last_unstale":"2024-03-24T00:38:02.058940+0000","last_undegraded":"2024-03-24T00:38:02.058940+0000","last_fullsized":"2024-03-24T00:38:02.058940+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:16:49.746603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046792+0000","last_change":"2024-03-24T00:38:02.057278+0000","last_active":"2024-03-24T00:38:03.046792+0000","last_peered":"2024-03-24T00:38:03.046792+0000","last_clean":"2024-03-24T00:38:03.046792+0000","last_became_active":"2024-03-24T00:38:02.056867+0000","last_became_peered":"2024-03-24T00:38:02.056867+0000","last_unstale":"2024-03-24T00:38:03.046792+0000","last_undegraded":"2024-03-24T00:38:03.046792+0000","last_fullsized":"2024-03-24T00:38:03.046792+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:56.438539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.054211+0000","last_change":"2024-03-24T00:38:02.049818+0000","last_active":"2024-03-24T00:38:02.054211+0000","last_peered":"2024-03-24T00:38:02.054211+0000","last_clean":"2024-03-24T00:38:02.054211+0000","last_became_active":"2024-03-24T00:38:02.049549+0000","last_became_peered":"2024-03-24T00:38:02.049549+0000","last_unstale":"2024-03-24T00:38:02.054211+0000","last_undegraded":"2024-03-24T00:38:02.054211+0000","last_fullsized":"2024-03-24T00:38:02.054211+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:01.957880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048015+0000","last_change":"2024-03-24T00:38:02.063775+0000","last_active":"2024-03-24T00:38:03.048015+0000","last_peered":"2024-03-24T00:38:03.048015+0000","last_clean":"2024-03-24T00:38:03.048015+0000","last_became_active":"2024-03-24T00:38:02.063415+0000","last_became_peered":"2024-03-24T00:38:02.063415+0000","last_unstale":"2024-03-24T00:38:03.048015+0000","last_undegraded":"2024-03-24T00:38:03.048015+0000","last_fullsized":"2024-03-24T00:38:03.048015+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:33.285058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048098+0000","last_change":"2024-03-24T00:38:02.058600+0000","last_active":"2024-03-24T00:38:03.048098+0000","last_peered":"2024-03-24T00:38:03.048098+0000","last_clean":"2024-03-24T00:38:03.048098+0000","last_became_active":"2024-03-24T00:38:02.056388+0000","last_became_peered":"2024-03-24T00:38:02.056388+0000","last_unstale":"2024-03-24T00:38:03.048098+0000","last_undegraded":"2024-03-24T00:38:03.048098+0000","last_fullsized":"2024-03-24T00:38:03.048098+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:19.015495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.053051+0000","last_change":"2024-03-24T00:38:02.049842+0000","last_active":"2024-03-24T00:38:02.053051+0000","last_peered":"2024-03-24T00:38:02.053051+0000","last_clean":"2024-03-24T00:38:02.053051+0000","last_became_active":"2024-03-24T00:38:02.049488+0000","last_became_peered":"2024-03-24T00:38:02.049488+0000","last_unstale":"2024-03-24T00:38:02.053051+0000","last_undegraded":"2024-03-24T00:38:02.053051+0000","last_fullsized":"2024-03-24T00:38:02.053051+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:55.045351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.050117+0000","last_change":"2024-03-24T00:38:02.048083+0000","last_active":"2024-03-24T00:38:02.050117+0000","last_peered":"2024-03-24T00:38:02.050117+0000","last_clean":"2024-03-24T00:38:02.050117+0000","last_became_active":"2024-03-24T00:38:02.047746+0000","last_became_peered":"2024-03-24T00:38:02.047746+0000","last_unstale":"2024-03-24T00:38:02.050117+0000","last_undegraded":"2024-03-24T00:38:02.050117+0000","last_fullsized":"2024-03-24T00:38:02.050117+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:25:24.459692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047270+0000","last_change":"2024-03-24T00:38:02.048239+0000","last_active":"2024-03-24T00:38:03.047270+0000","last_peered":"2024-03-24T00:38:03.047270+0000","last_clean":"2024-03-24T00:38:03.047270+0000","last_became_active":"2024-03-24T00:38:02.047884+0000","last_became_peered":"2024-03-24T00:38:02.047884+0000","last_unstale":"2024-03-24T00:38:03.047270+0000","last_undegraded":"2024-03-24T00:38:03.047270+0000","last_fullsized":"2024-03-24T00:38:03.047270+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:59:50.468231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059344+0000","last_change":"2024-03-24T00:38:02.046623+0000","last_active":"2024-03-24T00:38:03.059344+0000","last_peered":"2024-03-24T00:38:03.059344+0000","last_clean":"2024-03-24T00:38:03.059344+0000","last_became_active":"2024-03-24T00:38:02.046146+0000","last_became_peered":"2024-03-24T00:38:02.046146+0000","last_unstale":"2024-03-24T00:38:03.059344+0000","last_undegraded":"2024-03-24T00:38:03.059344+0000","last_fullsized":"2024-03-24T00:38:03.059344+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:38.764692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048219+0000","last_change":"2024-03-24T00:38:02.066388+0000","last_active":"2024-03-24T00:38:03.048219+0000","last_peered":"2024-03-24T00:38:03.048219+0000","last_clean":"2024-03-24T00:38:03.048219+0000","last_became_active":"2024-03-24T00:38:02.065984+0000","last_became_peered":"2024-03-24T00:38:02.065984+0000","last_unstale":"2024-03-24T00:38:03.048219+0000","last_undegraded":"2024-03-24T00:38:03.048219+0000","last_fullsized":"2024-03-24T00:38:03.048219+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:56:34.503377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047695+0000","last_change":"2024-03-24T00:38:02.050469+0000","last_active":"2024-03-24T00:38:03.047695+0000","last_peered":"2024-03-24T00:38:03.047695+0000","last_clean":"2024-03-24T00:38:03.047695+0000","last_became_active":"2024-03-24T00:38:02.050090+0000","last_became_peered":"2024-03-24T00:38:02.050090+0000","last_unstale":"2024-03-24T00:38:03.047695+0000","last_undegraded":"2024-03-24T00:38:03.047695+0000","last_fullsized":"2024-03-24T00:38:03.047695+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:59:16.472535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047039+0000","last_change":"2024-03-24T00:38:02.050263+0000","last_active":"2024-03-24T00:38:03.047039+0000","last_peered":"2024-03-24T00:38:03.047039+0000","last_clean":"2024-03-24T00:38:03.047039+0000","last_became_active":"2024-03-24T00:38:02.047937+0000","last_became_peered":"2024-03-24T00:38:02.047937+0000","last_unstale":"2024-03-24T00:38:03.047039+0000","last_undegraded":"2024-03-24T00:38:03.047039+0000","last_fullsized":"2024-03-24T00:38:03.047039+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:16:08.887440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043591+0000","last_change":"2024-03-24T00:38:02.052851+0000","last_active":"2024-03-24T00:38:03.043591+0000","last_peered":"2024-03-24T00:38:03.043591+0000","last_clean":"2024-03-24T00:38:03.043591+0000","last_became_active":"2024-03-24T00:38:02.052500+0000","last_became_peered":"2024-03-24T00:38:02.052500+0000","last_unstale":"2024-03-24T00:38:03.043591+0000","last_undegraded":"2024-03-24T00:38:03.043591+0000","last_fullsized":"2024-03-24T00:38:03.043591+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:33:05.858925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047592+0000","last_change":"2024-03-24T00:38:02.051064+0000","last_active":"2024-03-24T00:38:03.047592+0000","last_peered":"2024-03-24T00:38:03.047592+0000","last_clean":"2024-03-24T00:38:03.047592+0000","last_became_active":"2024-03-24T00:38:02.050775+0000","last_became_peered":"2024-03-24T00:38:02.050775+0000","last_unstale":"2024-03-24T00:38:03.047592+0000","last_undegraded":"2024-03-24T00:38:03.047592+0000","last_fullsized":"2024-03-24T00:38:03.047592+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:31:48.804402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043398+0000","last_change":"2024-03-24T00:38:02.056440+0000","last_active":"2024-03-24T00:38:03.043398+0000","last_peered":"2024-03-24T00:38:03.043398+0000","last_clean":"2024-03-24T00:38:03.043398+0000","last_became_active":"2024-03-24T00:38:02.056140+0000","last_became_peered":"2024-03-24T00:38:02.056140+0000","last_unstale":"2024-03-24T00:38:03.043398+0000","last_undegraded":"2024-03-24T00:38:03.043398+0000","last_fullsized":"2024-03-24T00:38:03.043398+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:35:59.814600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047454+0000","last_change":"2024-03-24T00:38:02.047976+0000","last_active":"2024-03-24T00:38:03.047454+0000","last_peered":"2024-03-24T00:38:03.047454+0000","last_clean":"2024-03-24T00:38:03.047454+0000","last_became_active":"2024-03-24T00:38:02.047501+0000","last_became_peered":"2024-03-24T00:38:02.047501+0000","last_unstale":"2024-03-24T00:38:03.047454+0000","last_undegraded":"2024-03-24T00:38:03.047454+0000","last_fullsized":"2024-03-24T00:38:03.047454+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:35:57.001029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.048194+0000","last_change":"2024-03-24T00:38:02.044475+0000","last_active":"2024-03-24T00:38:02.048194+0000","last_peered":"2024-03-24T00:38:02.048194+0000","last_clean":"2024-03-24T00:38:02.048194+0000","last_became_active":"2024-03-24T00:38:02.044213+0000","last_became_peered":"2024-03-24T00:38:02.044213+0000","last_unstale":"2024-03-24T00:38:02.048194+0000","last_undegraded":"2024-03-24T00:38:02.048194+0000","last_fullsized":"2024-03-24T00:38:02.048194+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:47:16.587931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.053208+0000","last_change":"2024-03-24T00:38:02.050311+0000","last_active":"2024-03-24T00:38:02.053208+0000","last_peered":"2024-03-24T00:38:02.053208+0000","last_clean":"2024-03-24T00:38:02.053208+0000","last_became_active":"2024-03-24T00:38:02.050004+0000","last_became_peered":"2024-03-24T00:38:02.050004+0000","last_unstale":"2024-03-24T00:38:02.053208+0000","last_undegraded":"2024-03-24T00:38:02.053208+0000","last_fullsized":"2024-03-24T00:38:02.053208+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:49.129268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043872+0000","last_change":"2024-03-24T00:38:02.049353+0000","last_active":"2024-03-24T00:38:03.043872+0000","last_peered":"2024-03-24T00:38:03.043872+0000","last_clean":"2024-03-24T00:38:03.043872+0000","last_became_active":"2024-03-24T00:38:02.049043+0000","last_became_peered":"2024-03-24T00:38:02.049043+0000","last_unstale":"2024-03-24T00:38:03.043872+0000","last_undegraded":"2024-03-24T00:38:03.043872+0000","last_fullsized":"2024-03-24T00:38:03.043872+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:21.620621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046553+0000","last_change":"2024-03-24T00:38:02.051647+0000","last_active":"2024-03-24T00:38:03.046553+0000","last_peered":"2024-03-24T00:38:03.046553+0000","last_clean":"2024-03-24T00:38:03.046553+0000","last_became_active":"2024-03-24T00:38:02.051097+0000","last_became_peered":"2024-03-24T00:38:02.051097+0000","last_unstale":"2024-03-24T00:38:03.046553+0000","last_undegraded":"2024-03-24T00:38:03.046553+0000","last_fullsized":"2024-03-24T00:38:03.046553+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:38:15.571575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061910+0000","last_change":"2024-03-24T00:38:02.046530+0000","last_active":"2024-03-24T00:38:03.061910+0000","last_peered":"2024-03-24T00:38:03.061910+0000","last_clean":"2024-03-24T00:38:03.061910+0000","last_became_active":"2024-03-24T00:38:02.046231+0000","last_became_peered":"2024-03-24T00:38:02.046231+0000","last_unstale":"2024-03-24T00:38:03.061910+0000","last_undegraded":"2024-03-24T00:38:03.061910+0000","last_fullsized":"2024-03-24T00:38:03.061910+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:42:22.127612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.048807+0000","last_change":"2024-03-24T00:38:02.043530+0000","last_active":"2024-03-24T00:38:02.048807+0000","last_peered":"2024-03-24T00:38:02.048807+0000","last_clean":"2024-03-24T00:38:02.048807+0000","last_became_active":"2024-03-24T00:38:02.043253+0000","last_became_peered":"2024-03-24T00:38:02.043253+0000","last_unstale":"2024-03-24T00:38:02.048807+0000","last_undegraded":"2024-03-24T00:38:02.048807+0000","last_fullsized":"2024-03-24T00:38:02.048807+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:09:31.230667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.051999+0000","last_change":"2024-03-24T00:38:02.045993+0000","last_active":"2024-03-24T00:38:02.051999+0000","last_peered":"2024-03-24T00:38:02.051999+0000","last_clean":"2024-03-24T00:38:02.051999+0000","last_became_active":"2024-03-24T00:38:02.045660+0000","last_became_peered":"2024-03-24T00:38:02.045660+0000","last_unstale":"2024-03-24T00:38:02.051999+0000","last_undegraded":"2024-03-24T00:38:02.051999+0000","last_fullsized":"2024-03-24T00:38:02.051999+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:56:21.791067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.057912+0000","last_change":"2024-03-24T00:38:02.055181+0000","last_active":"2024-03-24T00:38:02.057912+0000","last_peered":"2024-03-24T00:38:02.057912+0000","last_clean":"2024-03-24T00:38:02.057912+0000","last_became_active":"2024-03-24T00:38:02.054844+0000","last_became_peered":"2024-03-24T00:38:02.054844+0000","last_unstale":"2024-03-24T00:38:02.057912+0000","last_undegraded":"2024-03-24T00:38:02.057912+0000","last_fullsized":"2024-03-24T00:38:02.057912+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:31:56.702597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"105'4","reported_seq":18,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.062248+0000","last_change":"2024-03-24T00:38:01.034398+0000","last_active":"2024-03-24T00:38:03.062248+0000","last_peered":"2024-03-24T00:38:03.062248+0000","last_clean":"2024-03-24T00:38:03.062248+0000","last_became_active":"2024-03-24T00:38:01.034070+0000","last_became_peered":"2024-03-24T00:38:01.034070+0000","last_unstale":"2024-03-24T00:38:03.062248+0000","last_undegraded":"2024-03-24T00:38:03.062248+0000","last_fullsized":"2024-03-24T00:38:03.062248+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:09:58.345257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059562+0000","last_change":"2024-03-24T00:38:02.048706+0000","last_active":"2024-03-24T00:38:03.059562+0000","last_peered":"2024-03-24T00:38:03.059562+0000","last_clean":"2024-03-24T00:38:03.059562+0000","last_became_active":"2024-03-24T00:38:02.048392+0000","last_became_peered":"2024-03-24T00:38:02.048392+0000","last_unstale":"2024-03-24T00:38:03.059562+0000","last_undegraded":"2024-03-24T00:38:03.059562+0000","last_fullsized":"2024-03-24T00:38:03.059562+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:41:51.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047917+0000","last_change":"2024-03-24T00:38:01.040348+0000","last_active":"2024-03-24T00:38:03.047917+0000","last_peered":"2024-03-24T00:38:03.047917+0000","last_clean":"2024-03-24T00:38:03.047917+0000","last_became_active":"2024-03-24T00:38:01.039973+0000","last_became_peered":"2024-03-24T00:38:01.039973+0000","last_unstale":"2024-03-24T00:38:03.047917+0000","last_undegraded":"2024-03-24T00:38:03.047917+0000","last_fullsized":"2024-03-24T00:38:03.047917+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:38:14.564982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.053695+0000","last_change":"2024-03-24T00:38:02.052123+0000","last_active":"2024-03-24T00:38:02.053695+0000","last_peered":"2024-03-24T00:38:02.053695+0000","last_clean":"2024-03-24T00:38:02.053695+0000","last_became_active":"2024-03-24T00:38:02.051835+0000","last_became_peered":"2024-03-24T00:38:02.051835+0000","last_unstale":"2024-03-24T00:38:02.053695+0000","last_undegraded":"2024-03-24T00:38:02.053695+0000","last_fullsized":"2024-03-24T00:38:02.053695+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:09:59.351850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"105'2","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.072108+0000","last_change":"2024-03-24T00:38:01.051044+0000","last_active":"2024-03-24T00:38:02.072108+0000","last_peered":"2024-03-24T00:38:02.072108+0000","last_clean":"2024-03-24T00:38:02.072108+0000","last_became_active":"2024-03-24T00:38:01.050781+0000","last_became_peered":"2024-03-24T00:38:01.050781+0000","last_unstale":"2024-03-24T00:38:02.072108+0000","last_undegraded":"2024-03-24T00:38:02.072108+0000","last_fullsized":"2024-03-24T00:38:02.072108+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:09:38.732722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048326+0000","last_change":"2024-03-24T00:38:02.065641+0000","last_active":"2024-03-24T00:38:03.048326+0000","last_peered":"2024-03-24T00:38:03.048326+0000","last_clean":"2024-03-24T00:38:03.048326+0000","last_became_active":"2024-03-24T00:38:02.065310+0000","last_became_peered":"2024-03-24T00:38:02.065310+0000","last_unstale":"2024-03-24T00:38:03.048326+0000","last_undegraded":"2024-03-24T00:38:03.048326+0000","last_fullsized":"2024-03-24T00:38:03.048326+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:13:16.105953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"105'2","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.073461+0000","last_change":"2024-03-24T00:38:01.049067+0000","last_active":"2024-03-24T00:38:02.073461+0000","last_peered":"2024-03-24T00:38:02.073461+0000","last_clean":"2024-03-24T00:38:02.073461+0000","last_became_active":"2024-03-24T00:38:01.048807+0000","last_became_peered":"2024-03-24T00:38:01.048807+0000","last_unstale":"2024-03-24T00:38:02.073461+0000","last_undegraded":"2024-03-24T00:38:02.073461+0000","last_fullsized":"2024-03-24T00:38:02.073461+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:04:18.376681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.048756+0000","last_change":"2024-03-24T00:38:02.044640+0000","last_active":"2024-03-24T00:38:02.048756+0000","last_peered":"2024-03-24T00:38:02.048756+0000","last_clean":"2024-03-24T00:38:02.048756+0000","last_became_active":"2024-03-24T00:38:02.044353+0000","last_became_peered":"2024-03-24T00:38:02.044353+0000","last_unstale":"2024-03-24T00:38:02.048756+0000","last_undegraded":"2024-03-24T00:38:02.048756+0000","last_fullsized":"2024-03-24T00:38:02.048756+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:01:12.505348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061821+0000","last_change":"2024-03-24T00:38:01.041670+0000","last_active":"2024-03-24T00:38:03.061821+0000","last_peered":"2024-03-24T00:38:03.061821+0000","last_clean":"2024-03-24T00:38:03.061821+0000","last_became_active":"2024-03-24T00:38:01.041353+0000","last_became_peered":"2024-03-24T00:38:01.041353+0000","last_unstale":"2024-03-24T00:38:03.061821+0000","last_undegraded":"2024-03-24T00:38:03.061821+0000","last_fullsized":"2024-03-24T00:38:03.061821+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:46:54.038759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046722+0000","last_change":"2024-03-24T00:38:02.039321+0000","last_active":"2024-03-24T00:38:03.046722+0000","last_peered":"2024-03-24T00:38:03.046722+0000","last_clean":"2024-03-24T00:38:03.046722+0000","last_became_active":"2024-03-24T00:38:02.038982+0000","last_became_peered":"2024-03-24T00:38:02.038982+0000","last_unstale":"2024-03-24T00:38:03.046722+0000","last_undegraded":"2024-03-24T00:38:03.046722+0000","last_fullsized":"2024-03-24T00:38:03.046722+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:41:51.123584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043706+0000","last_change":"2024-03-24T00:38:01.044911+0000","last_active":"2024-03-24T00:38:03.043706+0000","last_peered":"2024-03-24T00:38:03.043706+0000","last_clean":"2024-03-24T00:38:03.043706+0000","last_became_active":"2024-03-24T00:38:01.042733+0000","last_became_peered":"2024-03-24T00:38:01.042733+0000","last_unstale":"2024-03-24T00:38:03.043706+0000","last_undegraded":"2024-03-24T00:38:03.043706+0000","last_fullsized":"2024-03-24T00:38:03.043706+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:41:50.116992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.047567+0000","last_change":"2024-03-24T00:38:02.041952+0000","last_active":"2024-03-24T00:38:02.047567+0000","last_peered":"2024-03-24T00:38:02.047567+0000","last_clean":"2024-03-24T00:38:02.047567+0000","last_became_active":"2024-03-24T00:38:02.041677+0000","last_became_peered":"2024-03-24T00:38:02.041677+0000","last_unstale":"2024-03-24T00:38:02.047567+0000","last_undegraded":"2024-03-24T00:38:02.047567+0000","last_fullsized":"2024-03-24T00:38:02.047567+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:43:26.480026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047276+0000","last_change":"2024-03-24T00:38:01.054368+0000","last_active":"2024-03-24T00:38:03.047276+0000","last_peered":"2024-03-24T00:38:03.047276+0000","last_clean":"2024-03-24T00:38:03.047276+0000","last_became_active":"2024-03-24T00:38:01.054053+0000","last_became_peered":"2024-03-24T00:38:01.054053+0000","last_unstale":"2024-03-24T00:38:03.047276+0000","last_undegraded":"2024-03-24T00:38:03.047276+0000","last_fullsized":"2024-03-24T00:38:03.047276+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:12:15.770803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061714+0000","last_change":"2024-03-24T00:38:02.057092+0000","last_active":"2024-03-24T00:38:03.061714+0000","last_peered":"2024-03-24T00:38:03.061714+0000","last_clean":"2024-03-24T00:38:03.061714+0000","last_became_active":"2024-03-24T00:38:02.056786+0000","last_became_peered":"2024-03-24T00:38:02.056786+0000","last_unstale":"2024-03-24T00:38:03.061714+0000","last_undegraded":"2024-03-24T00:38:03.061714+0000","last_fullsized":"2024-03-24T00:38:03.061714+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:00:23.166901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.030439+0000","last_change":"2024-03-24T00:38:01.056676+0000","last_active":"2024-03-24T00:38:02.030439+0000","last_peered":"2024-03-24T00:38:02.030439+0000","last_clean":"2024-03-24T00:38:02.030439+0000","last_became_active":"2024-03-24T00:38:01.056378+0000","last_became_peered":"2024-03-24T00:38:01.056378+0000","last_unstale":"2024-03-24T00:38:02.030439+0000","last_undegraded":"2024-03-24T00:38:02.030439+0000","last_fullsized":"2024-03-24T00:38:02.030439+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:56:33.496784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.062280+0000","last_change":"2024-03-24T00:38:02.057986+0000","last_active":"2024-03-24T00:38:03.062280+0000","last_peered":"2024-03-24T00:38:03.062280+0000","last_clean":"2024-03-24T00:38:03.062280+0000","last_became_active":"2024-03-24T00:38:02.057471+0000","last_became_peered":"2024-03-24T00:38:02.057471+0000","last_unstale":"2024-03-24T00:38:03.062280+0000","last_undegraded":"2024-03-24T00:38:03.062280+0000","last_fullsized":"2024-03-24T00:38:03.062280+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:20.216554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.042066+0000","last_change":"2024-03-24T00:38:01.048253+0000","last_active":"2024-03-24T00:38:02.042066+0000","last_peered":"2024-03-24T00:38:02.042066+0000","last_clean":"2024-03-24T00:38:02.042066+0000","last_became_active":"2024-03-24T00:38:01.047961+0000","last_became_peered":"2024-03-24T00:38:01.047961+0000","last_unstale":"2024-03-24T00:38:02.042066+0000","last_undegraded":"2024-03-24T00:38:02.042066+0000","last_fullsized":"2024-03-24T00:38:02.042066+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:26:33.629271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059592+0000","last_change":"2024-03-24T00:38:02.051972+0000","last_active":"2024-03-24T00:38:03.059592+0000","last_peered":"2024-03-24T00:38:03.059592+0000","last_clean":"2024-03-24T00:38:03.059592+0000","last_became_active":"2024-03-24T00:38:02.051647+0000","last_became_peered":"2024-03-24T00:38:02.051647+0000","last_unstale":"2024-03-24T00:38:03.059592+0000","last_undegraded":"2024-03-24T00:38:03.059592+0000","last_fullsized":"2024-03-24T00:38:03.059592+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:50:56.438539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043992+0000","last_change":"2024-03-24T00:38:01.043751+0000","last_active":"2024-03-24T00:38:03.043992+0000","last_peered":"2024-03-24T00:38:03.043992+0000","last_clean":"2024-03-24T00:38:03.043992+0000","last_became_active":"2024-03-24T00:38:01.042469+0000","last_became_peered":"2024-03-24T00:38:01.042469+0000","last_unstale":"2024-03-24T00:38:03.043992+0000","last_undegraded":"2024-03-24T00:38:03.043992+0000","last_fullsized":"2024-03-24T00:38:03.043992+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:15:19.531454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043919+0000","last_change":"2024-03-24T00:38:02.044985+0000","last_active":"2024-03-24T00:38:03.043919+0000","last_peered":"2024-03-24T00:38:03.043919+0000","last_clean":"2024-03-24T00:38:03.043919+0000","last_became_active":"2024-03-24T00:38:02.044645+0000","last_became_peered":"2024-03-24T00:38:02.044645+0000","last_unstale":"2024-03-24T00:38:03.043919+0000","last_undegraded":"2024-03-24T00:38:03.043919+0000","last_fullsized":"2024-03-24T00:38:03.043919+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:56:34.503377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"105'3","reported_seq":15,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.068329+0000","last_change":"2024-03-24T00:38:01.054223+0000","last_active":"2024-03-24T00:38:02.068329+0000","last_peered":"2024-03-24T00:38:02.068329+0000","last_clean":"2024-03-24T00:38:02.068329+0000","last_became_active":"2024-03-24T00:38:01.052988+0000","last_became_peered":"2024-03-24T00:38:01.052988+0000","last_unstale":"2024-03-24T00:38:02.068329+0000","last_undegraded":"2024-03-24T00:38:02.068329+0000","last_fullsized":"2024-03-24T00:38:02.068329+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:20.614028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.062158+0000","last_change":"2024-03-24T00:38:02.055777+0000","last_active":"2024-03-24T00:38:03.062158+0000","last_peered":"2024-03-24T00:38:03.062158+0000","last_clean":"2024-03-24T00:38:03.062158+0000","last_became_active":"2024-03-24T00:38:02.055434+0000","last_became_peered":"2024-03-24T00:38:02.055434+0000","last_unstale":"2024-03-24T00:38:03.062158+0000","last_undegraded":"2024-03-24T00:38:03.062158+0000","last_fullsized":"2024-03-24T00:38:03.062158+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:56:43.866424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"105'6","reported_seq":18,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.073685+0000","last_change":"2024-03-24T00:38:01.055839+0000","last_active":"2024-03-24T00:38:02.073685+0000","last_peered":"2024-03-24T00:38:02.073685+0000","last_clean":"2024-03-24T00:38:02.073685+0000","last_became_active":"2024-03-24T00:38:01.055497+0000","last_became_peered":"2024-03-24T00:38:01.055497+0000","last_unstale":"2024-03-24T00:38:02.073685+0000","last_undegraded":"2024-03-24T00:38:02.073685+0000","last_fullsized":"2024-03-24T00:38:02.073685+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:35.591064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043599+0000","last_change":"2024-03-24T00:38:02.046914+0000","last_active":"2024-03-24T00:38:03.043599+0000","last_peered":"2024-03-24T00:38:03.043599+0000","last_clean":"2024-03-24T00:38:03.043599+0000","last_became_active":"2024-03-24T00:38:02.046442+0000","last_became_peered":"2024-03-24T00:38:02.046442+0000","last_unstale":"2024-03-24T00:38:03.043599+0000","last_undegraded":"2024-03-24T00:38:03.043599+0000","last_fullsized":"2024-03-24T00:38:03.043599+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:33.285058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"105'2","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.070187+0000","last_change":"2024-03-24T00:38:01.033089+0000","last_active":"2024-03-24T00:38:02.070187+0000","last_peered":"2024-03-24T00:38:02.070187+0000","last_clean":"2024-03-24T00:38:02.070187+0000","last_became_active":"2024-03-24T00:38:01.032815+0000","last_became_peered":"2024-03-24T00:38:01.032815+0000","last_unstale":"2024-03-24T00:38:02.070187+0000","last_undegraded":"2024-03-24T00:38:02.070187+0000","last_fullsized":"2024-03-24T00:38:02.070187+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:13:35.909599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043852+0000","last_change":"2024-03-24T00:38:02.056007+0000","last_active":"2024-03-24T00:38:03.043852+0000","last_peered":"2024-03-24T00:38:03.043852+0000","last_clean":"2024-03-24T00:38:03.043852+0000","last_became_active":"2024-03-24T00:38:02.055708+0000","last_became_peered":"2024-03-24T00:38:02.055708+0000","last_unstale":"2024-03-24T00:38:03.043852+0000","last_undegraded":"2024-03-24T00:38:03.043852+0000","last_fullsized":"2024-03-24T00:38:03.043852+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:18:19.015495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059147+0000","last_change":"2024-03-24T00:38:01.052388+0000","last_active":"2024-03-24T00:38:03.059147+0000","last_peered":"2024-03-24T00:38:03.059147+0000","last_clean":"2024-03-24T00:38:03.059147+0000","last_became_active":"2024-03-24T00:38:01.052037+0000","last_became_peered":"2024-03-24T00:38:01.052037+0000","last_unstale":"2024-03-24T00:38:03.059147+0000","last_undegraded":"2024-03-24T00:38:03.059147+0000","last_fullsized":"2024-03-24T00:38:03.059147+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:26.448834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043396+0000","last_change":"2024-03-24T00:38:02.048249+0000","last_active":"2024-03-24T00:38:03.043396+0000","last_peered":"2024-03-24T00:38:03.043396+0000","last_clean":"2024-03-24T00:38:03.043396+0000","last_became_active":"2024-03-24T00:38:02.047969+0000","last_became_peered":"2024-03-24T00:38:02.047969+0000","last_unstale":"2024-03-24T00:38:03.043396+0000","last_undegraded":"2024-03-24T00:38:03.043396+0000","last_fullsized":"2024-03-24T00:38:03.043396+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:28:44.746992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"105'8","reported_seq":20,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.073470+0000","last_change":"2024-03-24T00:38:01.056110+0000","last_active":"2024-03-24T00:38:02.073470+0000","last_peered":"2024-03-24T00:38:02.073470+0000","last_clean":"2024-03-24T00:38:02.073470+0000","last_became_active":"2024-03-24T00:38:01.055840+0000","last_became_peered":"2024-03-24T00:38:01.055840+0000","last_unstale":"2024-03-24T00:38:02.073470+0000","last_undegraded":"2024-03-24T00:38:02.073470+0000","last_fullsized":"2024-03-24T00:38:02.073470+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:02:51.052161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.053560+0000","last_change":"2024-03-24T00:38:02.051368+0000","last_active":"2024-03-24T00:38:02.053560+0000","last_peered":"2024-03-24T00:38:02.053560+0000","last_clean":"2024-03-24T00:38:02.053560+0000","last_became_active":"2024-03-24T00:38:02.051020+0000","last_became_peered":"2024-03-24T00:38:02.051020+0000","last_unstale":"2024-03-24T00:38:02.053560+0000","last_undegraded":"2024-03-24T00:38:02.053560+0000","last_fullsized":"2024-03-24T00:38:02.053560+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:12:30.835729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"105'1","reported_seq":13,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.067397+0000","last_change":"2024-03-24T00:38:01.046814+0000","last_active":"2024-03-24T00:38:02.067397+0000","last_peered":"2024-03-24T00:38:02.067397+0000","last_clean":"2024-03-24T00:38:02.067397+0000","last_became_active":"2024-03-24T00:38:01.046520+0000","last_became_peered":"2024-03-24T00:38:01.046520+0000","last_unstale":"2024-03-24T00:38:02.067397+0000","last_undegraded":"2024-03-24T00:38:02.067397+0000","last_fullsized":"2024-03-24T00:38:02.067397+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:07:43.204857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.052918+0000","last_change":"2024-03-24T00:38:02.047450+0000","last_active":"2024-03-24T00:38:02.052918+0000","last_peered":"2024-03-24T00:38:02.052918+0000","last_clean":"2024-03-24T00:38:02.052918+0000","last_became_active":"2024-03-24T00:38:02.046202+0000","last_became_peered":"2024-03-24T00:38:02.046202+0000","last_unstale":"2024-03-24T00:38:02.052918+0000","last_undegraded":"2024-03-24T00:38:02.052918+0000","last_fullsized":"2024-03-24T00:38:02.052918+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:18.779535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"105'2","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.070953+0000","last_change":"2024-03-24T00:38:01.044025+0000","last_active":"2024-03-24T00:38:02.070953+0000","last_peered":"2024-03-24T00:38:02.070953+0000","last_clean":"2024-03-24T00:38:02.070953+0000","last_became_active":"2024-03-24T00:38:01.042969+0000","last_became_peered":"2024-03-24T00:38:01.042969+0000","last_unstale":"2024-03-24T00:38:02.070953+0000","last_undegraded":"2024-03-24T00:38:02.070953+0000","last_fullsized":"2024-03-24T00:38:02.070953+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:37:24.711793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048162+0000","last_change":"2024-03-24T00:38:02.062532+0000","last_active":"2024-03-24T00:38:03.048162+0000","last_peered":"2024-03-24T00:38:03.048162+0000","last_clean":"2024-03-24T00:38:03.048162+0000","last_became_active":"2024-03-24T00:38:02.062185+0000","last_became_peered":"2024-03-24T00:38:02.062185+0000","last_unstale":"2024-03-24T00:38:03.048162+0000","last_undegraded":"2024-03-24T00:38:03.048162+0000","last_fullsized":"2024-03-24T00:38:03.048162+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:07:21.620621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"105'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059378+0000","last_change":"2024-03-24T00:38:01.053941+0000","last_active":"2024-03-24T00:38:03.059378+0000","last_peered":"2024-03-24T00:38:03.059378+0000","last_clean":"2024-03-24T00:38:03.059378+0000","last_became_active":"2024-03-24T00:38:01.053530+0000","last_became_peered":"2024-03-24T00:38:01.053530+0000","last_unstale":"2024-03-24T00:38:03.059378+0000","last_undegraded":"2024-03-24T00:38:03.059378+0000","last_fullsized":"2024-03-24T00:38:03.059378+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:47:52.270501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048402+0000","last_change":"2024-03-24T00:38:02.052952+0000","last_active":"2024-03-24T00:38:03.048402+0000","last_peered":"2024-03-24T00:38:03.048402+0000","last_clean":"2024-03-24T00:38:03.048402+0000","last_became_active":"2024-03-24T00:38:02.052539+0000","last_became_peered":"2024-03-24T00:38:02.052539+0000","last_unstale":"2024-03-24T00:38:03.048402+0000","last_undegraded":"2024-03-24T00:38:03.048402+0000","last_fullsized":"2024-03-24T00:38:03.048402+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:26.764791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061996+0000","last_change":"2024-03-24T00:38:01.041124+0000","last_active":"2024-03-24T00:38:03.061996+0000","last_peered":"2024-03-24T00:38:03.061996+0000","last_clean":"2024-03-24T00:38:03.061996+0000","last_became_active":"2024-03-24T00:38:01.040822+0000","last_became_peered":"2024-03-24T00:38:01.040822+0000","last_unstale":"2024-03-24T00:38:03.061996+0000","last_undegraded":"2024-03-24T00:38:03.061996+0000","last_fullsized":"2024-03-24T00:38:03.061996+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:17.772943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048240+0000","last_change":"2024-03-24T00:38:02.052965+0000","last_active":"2024-03-24T00:38:03.048240+0000","last_peered":"2024-03-24T00:38:03.048240+0000","last_clean":"2024-03-24T00:38:03.048240+0000","last_became_active":"2024-03-24T00:38:02.050907+0000","last_became_peered":"2024-03-24T00:38:02.050907+0000","last_unstale":"2024-03-24T00:38:03.048240+0000","last_undegraded":"2024-03-24T00:38:03.048240+0000","last_fullsized":"2024-03-24T00:38:03.048240+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:44:45.455222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"105'2","reported_seq":14,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.073167+0000","last_change":"2024-03-24T00:38:01.047408+0000","last_active":"2024-03-24T00:38:02.073167+0000","last_peered":"2024-03-24T00:38:02.073167+0000","last_clean":"2024-03-24T00:38:02.073167+0000","last_became_active":"2024-03-24T00:38:01.046612+0000","last_became_peered":"2024-03-24T00:38:01.046612+0000","last_unstale":"2024-03-24T00:38:02.073167+0000","last_undegraded":"2024-03-24T00:38:02.073167+0000","last_fullsized":"2024-03-24T00:38:02.073167+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:04:12.367978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.054276+0000","last_change":"2024-03-24T00:38:02.050285+0000","last_active":"2024-03-24T00:38:02.054276+0000","last_peered":"2024-03-24T00:38:02.054276+0000","last_clean":"2024-03-24T00:38:02.054276+0000","last_became_active":"2024-03-24T00:38:02.049387+0000","last_became_peered":"2024-03-24T00:38:02.049387+0000","last_unstale":"2024-03-24T00:38:02.054276+0000","last_undegraded":"2024-03-24T00:38:02.054276+0000","last_fullsized":"2024-03-24T00:38:02.054276+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:55:26.331868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.038938+0000","last_change":"2024-03-24T00:38:01.050049+0000","last_active":"2024-03-24T00:38:02.038938+0000","last_peered":"2024-03-24T00:38:02.038938+0000","last_clean":"2024-03-24T00:38:02.038938+0000","last_became_active":"2024-03-24T00:38:01.049789+0000","last_became_peered":"2024-03-24T00:38:01.049789+0000","last_unstale":"2024-03-24T00:38:02.038938+0000","last_undegraded":"2024-03-24T00:38:02.038938+0000","last_fullsized":"2024-03-24T00:38:02.038938+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:07:54.658870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046779+0000","last_change":"2024-03-24T00:38:02.064234+0000","last_active":"2024-03-24T00:38:03.046779+0000","last_peered":"2024-03-24T00:38:03.046779+0000","last_clean":"2024-03-24T00:38:03.046779+0000","last_became_active":"2024-03-24T00:38:02.063928+0000","last_became_peered":"2024-03-24T00:38:02.063928+0000","last_unstale":"2024-03-24T00:38:03.046779+0000","last_undegraded":"2024-03-24T00:38:03.046779+0000","last_fullsized":"2024-03-24T00:38:03.046779+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:50.434313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.036097+0000","last_change":"2024-03-24T00:38:01.046058+0000","last_active":"2024-03-24T00:38:02.036097+0000","last_peered":"2024-03-24T00:38:02.036097+0000","last_clean":"2024-03-24T00:38:02.036097+0000","last_became_active":"2024-03-24T00:38:01.045759+0000","last_became_peered":"2024-03-24T00:38:01.045759+0000","last_unstale":"2024-03-24T00:38:02.036097+0000","last_undegraded":"2024-03-24T00:38:02.036097+0000","last_fullsized":"2024-03-24T00:38:02.036097+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:36:42.101640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043383+0000","last_change":"2024-03-24T00:38:02.049311+0000","last_active":"2024-03-24T00:38:03.043383+0000","last_peered":"2024-03-24T00:38:03.043383+0000","last_clean":"2024-03-24T00:38:03.043383+0000","last_became_active":"2024-03-24T00:38:02.048183+0000","last_became_peered":"2024-03-24T00:38:02.048183+0000","last_unstale":"2024-03-24T00:38:03.043383+0000","last_undegraded":"2024-03-24T00:38:03.043383+0000","last_fullsized":"2024-03-24T00:38:03.043383+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:38:39.197472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.030568+0000","last_change":"2024-03-24T00:38:01.051165+0000","last_active":"2024-03-24T00:38:02.030568+0000","last_peered":"2024-03-24T00:38:02.030568+0000","last_clean":"2024-03-24T00:38:02.030568+0000","last_became_active":"2024-03-24T00:38:01.050766+0000","last_became_peered":"2024-03-24T00:38:01.050766+0000","last_unstale":"2024-03-24T00:38:02.030568+0000","last_undegraded":"2024-03-24T00:38:02.030568+0000","last_fullsized":"2024-03-24T00:38:02.030568+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:13:15.099360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059390+0000","last_change":"2024-03-24T00:38:02.052352+0000","last_active":"2024-03-24T00:38:03.059390+0000","last_peered":"2024-03-24T00:38:03.059390+0000","last_clean":"2024-03-24T00:38:03.059390+0000","last_became_active":"2024-03-24T00:38:02.051268+0000","last_became_peered":"2024-03-24T00:38:02.051268+0000","last_unstale":"2024-03-24T00:38:03.059390+0000","last_undegraded":"2024-03-24T00:38:03.059390+0000","last_fullsized":"2024-03-24T00:38:03.059390+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:38:15.571575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"105'2","reported_seq":16,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043889+0000","last_change":"2024-03-24T00:38:01.047402+0000","last_active":"2024-03-24T00:38:03.043889+0000","last_peered":"2024-03-24T00:38:03.043889+0000","last_clean":"2024-03-24T00:38:03.043889+0000","last_became_active":"2024-03-24T00:38:01.047123+0000","last_became_peered":"2024-03-24T00:38:01.047123+0000","last_unstale":"2024-03-24T00:38:03.043889+0000","last_undegraded":"2024-03-24T00:38:03.043889+0000","last_fullsized":"2024-03-24T00:38:03.043889+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:38:14.564982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.040371+0000","last_change":"2024-03-24T00:38:02.033827+0000","last_active":"2024-03-24T00:38:02.040371+0000","last_peered":"2024-03-24T00:38:02.040371+0000","last_clean":"2024-03-24T00:38:02.040371+0000","last_became_active":"2024-03-24T00:38:02.033541+0000","last_became_peered":"2024-03-24T00:38:02.033541+0000","last_unstale":"2024-03-24T00:38:02.040371+0000","last_undegraded":"2024-03-24T00:38:02.040371+0000","last_fullsized":"2024-03-24T00:38:02.040371+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:05:48.731012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"106'1","reported_seq":15,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.060943+0000","last_change":"2024-03-24T00:38:01.044230+0000","last_active":"2024-03-24T00:38:03.060943+0000","last_peered":"2024-03-24T00:38:03.060943+0000","last_clean":"2024-03-24T00:38:03.060943+0000","last_became_active":"2024-03-24T00:38:01.043934+0000","last_became_peered":"2024-03-24T00:38:01.043934+0000","last_unstale":"2024-03-24T00:38:03.060943+0000","last_undegraded":"2024-03-24T00:38:03.060943+0000","last_fullsized":"2024-03-24T00:38:03.060943+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:43:54.494418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.046816+0000","last_change":"2024-03-24T00:38:02.041941+0000","last_active":"2024-03-24T00:38:02.046816+0000","last_peered":"2024-03-24T00:38:02.046816+0000","last_clean":"2024-03-24T00:38:02.046816+0000","last_became_active":"2024-03-24T00:38:02.041656+0000","last_became_peered":"2024-03-24T00:38:02.041656+0000","last_unstale":"2024-03-24T00:38:02.046816+0000","last_undegraded":"2024-03-24T00:38:02.046816+0000","last_fullsized":"2024-03-24T00:38:02.046816+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:09:16.674791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.030558+0000","last_change":"2024-03-24T00:38:01.048064+0000","last_active":"2024-03-24T00:38:02.030558+0000","last_peered":"2024-03-24T00:38:02.030558+0000","last_clean":"2024-03-24T00:38:02.030558+0000","last_became_active":"2024-03-24T00:38:01.047758+0000","last_became_peered":"2024-03-24T00:38:01.047758+0000","last_unstale":"2024-03-24T00:38:02.030558+0000","last_undegraded":"2024-03-24T00:38:02.030558+0000","last_fullsized":"2024-03-24T00:38:02.030558+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:44:44.448630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.059356+0000","last_change":"2024-03-24T00:38:02.052725+0000","last_active":"2024-03-24T00:38:03.059356+0000","last_peered":"2024-03-24T00:38:03.059356+0000","last_clean":"2024-03-24T00:38:03.059356+0000","last_became_active":"2024-03-24T00:38:02.052340+0000","last_became_peered":"2024-03-24T00:38:02.052340+0000","last_unstale":"2024-03-24T00:38:03.059356+0000","last_undegraded":"2024-03-24T00:38:03.059356+0000","last_fullsized":"2024-03-24T00:38:03.059356+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:15:20.538046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043956+0000","last_change":"2024-03-24T00:38:01.046446+0000","last_active":"2024-03-24T00:38:03.043956+0000","last_peered":"2024-03-24T00:38:03.043956+0000","last_clean":"2024-03-24T00:38:03.043956+0000","last_became_active":"2024-03-24T00:38:01.044068+0000","last_became_peered":"2024-03-24T00:38:01.044068+0000","last_unstale":"2024-03-24T00:38:03.043956+0000","last_undegraded":"2024-03-24T00:38:03.043956+0000","last_fullsized":"2024-03-24T00:38:03.043956+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:15:37.758100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.048024+0000","last_change":"2024-03-24T00:38:02.050141+0000","last_active":"2024-03-24T00:38:03.048024+0000","last_peered":"2024-03-24T00:38:03.048024+0000","last_clean":"2024-03-24T00:38:03.048024+0000","last_became_active":"2024-03-24T00:38:02.049816+0000","last_became_peered":"2024-03-24T00:38:02.049816+0000","last_unstale":"2024-03-24T00:38:03.048024+0000","last_undegraded":"2024-03-24T00:38:03.048024+0000","last_fullsized":"2024-03-24T00:38:03.048024+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:33:22.013335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061854+0000","last_change":"2024-03-24T00:38:01.045695+0000","last_active":"2024-03-24T00:38:03.061854+0000","last_peered":"2024-03-24T00:38:03.061854+0000","last_clean":"2024-03-24T00:38:03.061854+0000","last_became_active":"2024-03-24T00:38:01.045361+0000","last_became_peered":"2024-03-24T00:38:01.045361+0000","last_unstale":"2024-03-24T00:38:03.061854+0000","last_undegraded":"2024-03-24T00:38:03.061854+0000","last_fullsized":"2024-03-24T00:38:03.061854+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:46:44.447740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046898+0000","last_change":"2024-03-24T00:38:02.051378+0000","last_active":"2024-03-24T00:38:03.046898+0000","last_peered":"2024-03-24T00:38:03.046898+0000","last_clean":"2024-03-24T00:38:03.046898+0000","last_became_active":"2024-03-24T00:38:02.050709+0000","last_became_peered":"2024-03-24T00:38:02.050709+0000","last_unstale":"2024-03-24T00:38:03.046898+0000","last_undegraded":"2024-03-24T00:38:03.046898+0000","last_fullsized":"2024-03-24T00:38:03.046898+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:36.346943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046522+0000","last_change":"2024-03-24T00:38:01.049603+0000","last_active":"2024-03-24T00:38:03.046522+0000","last_peered":"2024-03-24T00:38:03.046522+0000","last_clean":"2024-03-24T00:38:03.046522+0000","last_became_active":"2024-03-24T00:38:01.049258+0000","last_became_peered":"2024-03-24T00:38:01.049258+0000","last_unstale":"2024-03-24T00:38:03.046522+0000","last_undegraded":"2024-03-24T00:38:03.046522+0000","last_fullsized":"2024-03-24T00:38:03.046522+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:20:19.209962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.046941+0000","last_change":"2024-03-24T00:38:02.055424+0000","last_active":"2024-03-24T00:38:03.046941+0000","last_peered":"2024-03-24T00:38:03.046941+0000","last_clean":"2024-03-24T00:38:03.046941+0000","last_became_active":"2024-03-24T00:38:02.055133+0000","last_became_peered":"2024-03-24T00:38:02.055133+0000","last_unstale":"2024-03-24T00:38:03.046941+0000","last_undegraded":"2024-03-24T00:38:03.046941+0000","last_fullsized":"2024-03-24T00:38:03.046941+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:35:59.814600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.039843+0000","last_change":"2024-03-24T00:38:01.050832+0000","last_active":"2024-03-24T00:38:02.039843+0000","last_peered":"2024-03-24T00:38:02.039843+0000","last_clean":"2024-03-24T00:38:02.039843+0000","last_became_active":"2024-03-24T00:38:01.050554+0000","last_became_peered":"2024-03-24T00:38:01.050554+0000","last_unstale":"2024-03-24T00:38:02.039843+0000","last_undegraded":"2024-03-24T00:38:02.039843+0000","last_fullsized":"2024-03-24T00:38:02.039843+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:07.247959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.048851+0000","last_change":"2024-03-24T00:38:02.045232+0000","last_active":"2024-03-24T00:38:02.048851+0000","last_peered":"2024-03-24T00:38:02.048851+0000","last_clean":"2024-03-24T00:38:02.048851+0000","last_became_active":"2024-03-24T00:38:02.044967+0000","last_became_peered":"2024-03-24T00:38:02.044967+0000","last_unstale":"2024-03-24T00:38:02.048851+0000","last_undegraded":"2024-03-24T00:38:02.048851+0000","last_fullsized":"2024-03-24T00:38:02.048851+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:28:06.561931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-24T00:38:02.030617+0000","last_change":"2024-03-24T00:38:01.047818+0000","last_active":"2024-03-24T00:38:02.030617+0000","last_peered":"2024-03-24T00:38:02.030617+0000","last_clean":"2024-03-24T00:38:02.030617+0000","last_became_active":"2024-03-24T00:38:01.047272+0000","last_became_peered":"2024-03-24T00:38:01.047272+0000","last_unstale":"2024-03-24T00:38:02.030617+0000","last_undegraded":"2024-03-24T00:38:02.030617+0000","last_fullsized":"2024-03-24T00:38:02.030617+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:06:25.758199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":12,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.043383+0000","last_change":"2024-03-24T00:38:02.056298+0000","last_active":"2024-03-24T00:38:03.043383+0000","last_peered":"2024-03-24T00:38:03.043383+0000","last_clean":"2024-03-24T00:38:03.043383+0000","last_became_active":"2024-03-24T00:38:02.055729+0000","last_became_peered":"2024-03-24T00:38:02.055729+0000","last_unstale":"2024-03-24T00:38:03.043383+0000","last_undegraded":"2024-03-24T00:38:03.043383+0000","last_fullsized":"2024-03-24T00:38:03.043383+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:01.015679+0000","last_clean_scrub_stamp":"2024-03-24T00:38:01.015679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:52:50.434313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":14,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.061843+0000","last_change":"2024-03-24T00:38:01.049259+0000","last_active":"2024-03-24T00:38:03.061843+0000","last_peered":"2024-03-24T00:38:03.061843+0000","last_clean":"2024-03-24T00:38:03.061843+0000","last_became_active":"2024-03-24T00:38:01.048839+0000","last_became_peered":"2024-03-24T00:38:01.048839+0000","last_unstale":"2024-03-24T00:38:03.061843+0000","last_undegraded":"2024-03-24T00:38:03.061843+0000","last_fullsized":"2024-03-24T00:38:03.061843+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:38:00.009086+0000","last_clean_scrub_stamp":"2024-03-24T00:38:00.009086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:48.122676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":344,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-24T00:38:03.047638+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:38:03.047638+0000","last_peered":"2024-03-24T00:38:03.047638+0000","last_clean":"2024-03-24T00:38:03.047638+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:38:03.047638+0000","last_undegraded":"2024-03-24T00:38:03.047638+0000","last_fullsized":"2024-03-24T00:38:03.047638+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574936,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37204,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334636,"statfs":{"total":96636764160,"available":96598667264,"internally_reserved":0,"allocated":1908736,"data_stored":1159960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574937,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40788,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331052,"statfs":{"total":96636764160,"available":96594997248,"internally_reserved":0,"allocated":1310720,"data_stored":569046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36636,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335204,"statfs":{"total":96636764160,"available":96599248896,"internally_reserved":0,"allocated":1327104,"data_stored":575880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574937,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50520,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321320,"statfs":{"total":96636764160,"available":96585031680,"internally_reserved":0,"allocated":1318912,"data_stored":575276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574936,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46376,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325464,"statfs":{"total":96636764160,"available":96589275136,"internally_reserved":0,"allocated":1335296,"data_stored":576958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574937,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36620,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335220,"statfs":{"total":96636764160,"available":96599265280,"internally_reserved":0,"allocated":1306624,"data_stored":568976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40792,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331048,"statfs":{"total":96636764160,"available":96594993152,"internally_reserved":0,"allocated":1314816,"data_stored":569100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574936,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37220,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334620,"statfs":{"total":96636764160,"available":96598650880,"internally_reserved":0,"allocated":1921024,"data_stored":1161068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:38:06.541 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:38:06.541 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:38:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.725+0000 7fd064200640 1 Processor -- start 2024-03-24T00:38:06.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.725+0000 7fd064200640 1 -- start start 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd064200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd05c15bd00 0x7fd05c0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd064200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd05c0728d0 0x7fd05c0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd064200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c072d30 con 0x7fd05c15f8c0 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd064200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05c0cf790 con 0x7fd05c15bd00 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd064200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c071de0 con 0x7fd05c0728d0 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd061800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd05c15bd00 0x7fd05c0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd061800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd05c15bd00 0x7fd05c0ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48106/0 (socket says 172.21.15.40:48106) 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd061800640 1 -- 172.21.15.40:0/4137476052 learned_addr learned my addr 172.21.15.40:0/4137476052 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd060e00640 1 --2- 172.21.15.40:0/4137476052 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd05c0728d0 0x7fd05c0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd061800640 1 -- 172.21.15.40:0/4137476052 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd05c0728d0 msgr2=0x7fd05c0cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd061800640 1 --2- 172.21.15.40:0/4137476052 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd05c0728d0 0x7fd05c0cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd061800640 1 -- 172.21.15.40:0/4137476052 >> v1:172.21.15.53:6789/0 conn(0x7fd05c15f8c0 legacy=0x7fd05c06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd061800640 1 -- 172.21.15.40:0/4137476052 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c0cea50 con 0x7fd05c15bd00 2024-03-24T00:38:06.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.729+0000 7fd060e00640 1 --2- 172.21.15.40:0/4137476052 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd05c0728d0 0x7fd05c0cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd061800640 1 --2- 172.21.15.40:0/4137476052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c15bd00 0x7fd05c0ae8e0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fd050009870 tx=0x7fd05001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fbf23ad1c828b48 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd05be00640 1 -- 172.21.15.40:0/4137476052 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0500a95f0 con 0x7fd05c15bd00 2024-03-24T00:38:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd05be00640 1 -- 172.21.15.40:0/4137476052 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0500a9790 con 0x7fd05c15bd00 2024-03-24T00:38:06.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd05be00640 1 -- 172.21.15.40:0/4137476052 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0500a9b60 con 0x7fd05c15bd00 2024-03-24T00:38:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd064200640 1 -- 172.21.15.40:0/4137476052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c15bd00 msgr2=0x7fd05c0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd064200640 1 --2- 172.21.15.40:0/4137476052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c15bd00 0x7fd05c0ae8e0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fd050009870 tx=0x7fd05001c530 comp rx=0 tx=0).stop 2024-03-24T00:38:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd064200640 1 -- 172.21.15.40:0/4137476052 shutdown_connections 2024-03-24T00:38:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd064200640 1 --2- 172.21.15.40:0/4137476052 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd05c0728d0 0x7fd05c0cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd064200640 1 --2- 172.21.15.40:0/4137476052 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c15bd00 0x7fd05c0ae8e0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd064200640 1 -- 172.21.15.40:0/4137476052 >> 172.21.15.40:0/4137476052 conn(0x7fd05c0b1660 msgr2=0x7fd05c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd064200640 1 -- 172.21.15.40:0/4137476052 shutdown_connections 2024-03-24T00:38:06.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd064200640 1 -- 172.21.15.40:0/4137476052 wait complete. 2024-03-24T00:38:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd064200640 1 Processor -- start 2024-03-24T00:38:06.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.733+0000 7fd064200640 1 -- start start 2024-03-24T00:38:06.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd064200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd05c0728d0 0x7fd05c15ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd064200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c15bd00 0x7fd05c14da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd064200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd05c15f8c0 0x7fd05c1574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd064200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c15c0e0 con 0x7fd05c15bd00 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd064200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c06d630 con 0x7fd05c0728d0 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd064200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c1600e0 con 0x7fd05c15f8c0 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd060e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c15bd00 0x7fd05c14da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd060e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c15bd00 0x7fd05c14da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48110/0 (socket says 172.21.15.40:48110) 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd060e00640 1 -- 172.21.15.40:0/694222983 learned_addr learned my addr 172.21.15.40:0/694222983 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd061800640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd05c0728d0 0x7fd05c15ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd062200640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd05c15f8c0 0x7fd05c1574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd060e00640 1 -- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd05c15f8c0 msgr2=0x7fd05c1574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd060e00640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd05c15f8c0 0x7fd05c1574d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd060e00640 1 -- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd05c0728d0 msgr2=0x7fd05c15ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd060e00640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd05c0728d0 0x7fd05c15ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd060e00640 1 -- 172.21.15.40:0/694222983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c15d290 con 0x7fd05c15bd00 2024-03-24T00:38:06.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd060e00640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c15bd00 0x7fd05c14da00 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fd04800e810 tx=0x7fd04800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:06.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd059600640 1 -- 172.21.15.40:0/694222983 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048017070 con 0x7fd05c15bd00 2024-03-24T00:38:06.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd059600640 1 -- 172.21.15.40:0/694222983 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04800c8d0 con 0x7fd05c15bd00 2024-03-24T00:38:06.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd059600640 1 -- 172.21.15.40:0/694222983 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04800f040 con 0x7fd05c15bd00 2024-03-24T00:38:06.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd064200640 1 -- 172.21.15.40:0/694222983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05c15d570 con 0x7fd05c15bd00 2024-03-24T00:38:06.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd064200640 1 -- 172.21.15.40:0/694222983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05c156fb0 con 0x7fd05c15bd00 2024-03-24T00:38:06.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd064200640 1 -- 172.21.15.40:0/694222983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05c15d740 con 0x7fd05c15bd00 2024-03-24T00:38:06.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.737+0000 7fd059600640 1 -- 172.21.15.40:0/694222983 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd04801a070 con 0x7fd05c15bd00 2024-03-24T00:38:06.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.741+0000 7fd059600640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd03c067660 0x7fd03c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:06.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.741+0000 7fd059600640 1 -- 172.21.15.40:0/694222983 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0480853e0 con 0x7fd05c15bd00 2024-03-24T00:38:06.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.741+0000 7fd061800640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd03c067660 0x7fd03c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:06.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.741+0000 7fd061800640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd03c067660 0x7fd03c069b20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd0500ae000 tx=0x7fd0500b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:06.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.745+0000 7fd059600640 1 -- 172.21.15.40:0/694222983 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd048055a00 con 0x7fd05c15bd00 2024-03-24T00:38:06.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.965+0000 7fd064200640 1 -- 172.21.15.40:0/694222983 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd05c068170 con 0x7fd05c15bd00 2024-03-24T00:38:06.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.965+0000 7fd059600640 1 -- 172.21.15.40:0/694222983 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd048059aa0 con 0x7fd05c15bd00 2024-03-24T00:38:06.970 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-03-24T00:38:06.970 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw== 2024-03-24T00:38:06.970 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-03-24T00:38:06.970 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-03-24T00:38:06.970 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:38:06.970 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-03-24T00:38:06.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.969+0000 7fd064200640 1 -- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd03c067660 msgr2=0x7fd03c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.969+0000 7fd064200640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd03c067660 0x7fd03c069b20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fd0500ae000 tx=0x7fd0500b3040 comp rx=0 tx=0).stop 2024-03-24T00:38:06.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.969+0000 7fd064200640 1 -- 172.21.15.40:0/694222983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c15bd00 msgr2=0x7fd05c14da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:06.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.969+0000 7fd064200640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c15bd00 0x7fd05c14da00 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fd04800e810 tx=0x7fd04800ecd0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.969+0000 7fd064200640 1 -- 172.21.15.40:0/694222983 shutdown_connections 2024-03-24T00:38:06.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.969+0000 7fd064200640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd03c067660 0x7fd03c069b20 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.969+0000 7fd064200640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd05c15f8c0 0x7fd05c1574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.969+0000 7fd064200640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd05c15bd00 0x7fd05c14da00 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.969+0000 7fd064200640 1 --2- 172.21.15.40:0/694222983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd05c0728d0 0x7fd05c15ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:06.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.969+0000 7fd064200640 1 -- 172.21.15.40:0/694222983 >> 172.21.15.40:0/694222983 conn(0x7fd05c0b1660 msgr2=0x7fd05c1545e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:06.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.969+0000 7fd064200640 1 -- 172.21.15.40:0/694222983 shutdown_connections 2024-03-24T00:38:06.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:06.969+0000 7fd064200640 1 -- 172.21.15.40:0/694222983 wait complete. 2024-03-24T00:38:06.992 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:07.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.129+0000 7faf96c00640 1 Processor -- start 2024-03-24T00:38:07.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.129+0000 7faf96c00640 1 -- start start 2024-03-24T00:38:07.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.129+0000 7faf96c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faf9006e620 0x7faf900b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.129+0000 7faf96c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf90070f60 con 0x7faf9006ea00 2024-03-24T00:38:07.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.129+0000 7faf96c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf900b3620 con 0x7faf900ccfa0 2024-03-24T00:38:07.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.129+0000 7faf96c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf900b4c00 con 0x7faf9006e620 2024-03-24T00:38:07.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.129+0000 7faf8fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faf9006e620 0x7faf900b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.133+0000 7faf87400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7faf9006ea00 0x7faf900b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53334/0 (socket says 172.21.15.40:53334) 2024-03-24T00:38:07.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.133+0000 7faf8fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7faf9006e620 0x7faf900b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39440/0 (socket says 172.21.15.40:39440) 2024-03-24T00:38:07.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.133+0000 7faf87400640 1 -- 172.21.15.40:0/1773041429 learned_addr learned my addr 172.21.15.40:0/1773041429 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:07.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.133+0000 7faf8fe00640 1 -- 172.21.15.40:0/1773041429 >> v1:172.21.15.53:6789/0 conn(0x7faf900ccfa0 legacy=0x7faf900cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.133+0000 7faf8fe00640 1 -- 172.21.15.40:0/1773041429 >> v1:172.21.15.40:6789/0 conn(0x7faf9006ea00 legacy=0x7faf900b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.133+0000 7faf8fe00640 1 -- 172.21.15.40:0/1773041429 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf900b3900 con 0x7faf9006e620 2024-03-24T00:38:07.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.133+0000 7faf8fe00640 1 --2- 172.21.15.40:0/1773041429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9006e620 0x7faf900b46c0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7faf800097c0 tx=0x7faf8001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0deabebf259e6c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:07.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.133+0000 7faf8f400640 1 -- 172.21.15.40:0/1773041429 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf800b1070 con 0x7faf9006e620 2024-03-24T00:38:07.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.133+0000 7faf8f400640 1 -- 172.21.15.40:0/1773041429 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf800a8050 con 0x7faf9006e620 2024-03-24T00:38:07.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.133+0000 7faf8f400640 1 -- 172.21.15.40:0/1773041429 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf800ac860 con 0x7faf9006e620 2024-03-24T00:38:07.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.133+0000 7faf96c00640 1 -- 172.21.15.40:0/1773041429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9006e620 msgr2=0x7faf900b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.133+0000 7faf96c00640 1 --2- 172.21.15.40:0/1773041429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9006e620 0x7faf900b46c0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7faf800097c0 tx=0x7faf8001c3e0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 -- 172.21.15.40:0/1773041429 shutdown_connections 2024-03-24T00:38:07.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 --2- 172.21.15.40:0/1773041429 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9006e620 0x7faf900b46c0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 -- 172.21.15.40:0/1773041429 >> 172.21.15.40:0/1773041429 conn(0x7faf900b17b0 msgr2=0x7faf900b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:07.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 -- 172.21.15.40:0/1773041429 shutdown_connections 2024-03-24T00:38:07.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 -- 172.21.15.40:0/1773041429 wait complete. 2024-03-24T00:38:07.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 Processor -- start 2024-03-24T00:38:07.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 -- start start 2024-03-24T00:38:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf9006e620 0x7faf9015a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9006ea00 0x7faf9015d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf900ccfa0 0x7faf9013b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf90144920 con 0x7faf9006ea00 2024-03-24T00:38:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9006ee10 con 0x7faf9006e620 2024-03-24T00:38:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf9015c580 con 0x7faf900ccfa0 2024-03-24T00:38:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf87400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9006ea00 0x7faf9015d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf87400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9006ea00 0x7faf9015d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39444/0 (socket says 172.21.15.40:39444) 2024-03-24T00:38:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf87400640 1 -- 172.21.15.40:0/1327107115 learned_addr learned my addr 172.21.15.40:0/1327107115 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf8fe00640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf9006e620 0x7faf9015a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf94c00640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf900ccfa0 0x7faf9013b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf87400640 1 -- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf900ccfa0 msgr2=0x7faf9013b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf87400640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf900ccfa0 0x7faf9013b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf87400640 1 -- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf9006e620 msgr2=0x7faf9015a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf87400640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf9006e620 0x7faf9015a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf87400640 1 -- 172.21.15.40:0/1327107115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9015c860 con 0x7faf9006ea00 2024-03-24T00:38:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf87400640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9006ea00 0x7faf9015d620 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7faf78005890 tx=0x7faf7800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf8cc00640 1 -- 172.21.15.40:0/1327107115 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf78017070 con 0x7faf9006ea00 2024-03-24T00:38:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf8cc00640 1 -- 172.21.15.40:0/1327107115 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf78005cf0 con 0x7faf9006ea00 2024-03-24T00:38:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf8cc00640 1 -- 172.21.15.40:0/1327107115 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf780072b0 con 0x7faf9006ea00 2024-03-24T00:38:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 -- 172.21.15.40:0/1327107115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9015cb40 con 0x7faf9006ea00 2024-03-24T00:38:07.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.137+0000 7faf96c00640 1 -- 172.21.15.40:0/1327107115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf9015d100 con 0x7faf9006ea00 2024-03-24T00:38:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.141+0000 7faf96c00640 1 -- 172.21.15.40:0/1327107115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf90068170 con 0x7faf9006ea00 2024-03-24T00:38:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.141+0000 7faf8cc00640 1 -- 172.21.15.40:0/1327107115 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7faf7801a070 con 0x7faf9006ea00 2024-03-24T00:38:07.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.141+0000 7faf8cc00640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faf6c067660 0x7faf6c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.141+0000 7faf8cc00640 1 -- 172.21.15.40:0/1327107115 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf78084bb0 con 0x7faf9006ea00 2024-03-24T00:38:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.141+0000 7faf8fe00640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faf6c067660 0x7faf6c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.141+0000 7faf8fe00640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faf6c067660 0x7faf6c069b20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7faf800ae000 tx=0x7faf80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:07.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.145+0000 7faf8cc00640 1 -- 172.21.15.40:0/1327107115 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7faf78055140 con 0x7faf9006ea00 2024-03-24T00:38:07.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.341+0000 7faf96c00640 1 -- 172.21.15.40:0/1327107115 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf90072f50 con 0x7faf9006ea00 2024-03-24T00:38:07.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf8cc00640 1 -- 172.21.15.40:0/1327107115 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+4973 (secure 0 0 0) 0x7faf780591e0 con 0x7faf9006ea00 2024-03-24T00:38:07.347 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:07.347 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":112,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7079,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/755981654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":755981654},{"type":"v1","addr":"172.21.15.40:6835","nonce":755981654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":8101,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/3877615536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3877615536},{"type":"v1","addr":"172.21.15.53:6836","nonce":3877615536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":8400,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/3883043397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":3883043397},{"type":"v1","addr":"172.21.15.40:6834","nonce":3883043397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:01.424871+0000","modified":"2024-03-24T00:38:04.435669+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8418},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8418":{"gid":8418,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/8812937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":8812937},{"type":"v1","addr":"172.21.15.53:6837","nonce":8812937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8418,"qdb_cluster":[8418]},"id":8}]} 2024-03-24T00:38:07.348 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 112 2024-03-24T00:38:07.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf96c00640 1 -- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faf6c067660 msgr2=0x7faf6c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf96c00640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faf6c067660 0x7faf6c069b20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7faf800ae000 tx=0x7faf80008040 comp rx=0 tx=0).stop 2024-03-24T00:38:07.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf96c00640 1 -- 172.21.15.40:0/1327107115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9006ea00 msgr2=0x7faf9015d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf96c00640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9006ea00 0x7faf9015d620 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7faf78005890 tx=0x7faf7800c960 comp rx=0 tx=0).stop 2024-03-24T00:38:07.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf96c00640 1 -- 172.21.15.40:0/1327107115 shutdown_connections 2024-03-24T00:38:07.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf96c00640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faf6c067660 0x7faf6c069b20 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf96c00640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7faf900ccfa0 0x7faf9013b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf96c00640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7faf9006ea00 0x7faf9015d620 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf96c00640 1 --2- 172.21.15.40:0/1327107115 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf9006e620 0x7faf9015a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf96c00640 1 -- 172.21.15.40:0/1327107115 >> 172.21.15.40:0/1327107115 conn(0x7faf900b17b0 msgr2=0x7faf900aef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:07.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf96c00640 1 -- 172.21.15.40:0/1327107115 shutdown_connections 2024-03-24T00:38:07.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.345+0000 7faf96c00640 1 -- 172.21.15.40:0/1327107115 wait complete. 2024-03-24T00:38:07.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 112, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:38:01.424871+0000', 'modified': '2024-03-24T00:38:04.435669+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8418}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8418': {'gid': 8418, 'name': 'b', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6837/8812937', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6835', 'nonce': 8812937}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 8812937}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8418, 'qdb_cluster': [8418]} 2024-03-24T00:38:07.770 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:38:07.770 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:38:07.887 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:38:07.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:38:07.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.909+0000 7f2858000640 1 Processor -- start 2024-03-24T00:38:07.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.909+0000 7f2858000640 1 -- start start 2024-03-24T00:38:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.909+0000 7f2858000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28500d08e0 0x7f285015f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.909+0000 7f2858000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28500af710 con 0x7f28500b43a0 2024-03-24T00:38:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.909+0000 7f2858000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28500b34f0 con 0x7f28500cfdf0 2024-03-24T00:38:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.909+0000 7f2858000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28500b2bb0 con 0x7f28500d08e0 2024-03-24T00:38:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f2856000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f28500b43a0 0x7f28500afbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53344/0 (socket says 172.21.15.40:53344) 2024-03-24T00:38:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f2856000640 1 -- 172.21.15.40:0/2527084453 learned_addr learned my addr 172.21.15.40:0/2527084453 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f2855600640 1 --1- 172.21.15.40:0/2527084453 >> v1:172.21.15.53:6789/0 conn(0x7f28500cfdf0 0x7f28500d01d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:48378/0 (socket says 172.21.15.40:48378) 2024-03-24T00:38:07.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f2854c00640 1 --2- 172.21.15.40:0/2527084453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28500d08e0 0x7f285015f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f284fe00640 1 -- 172.21.15.40:0/2527084453 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110223696 0 0) 0x7f28500af710 con 0x7f28500b43a0 2024-03-24T00:38:07.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f284fe00640 1 -- 172.21.15.40:0/2527084453 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2838003530 con 0x7f28500b43a0 2024-03-24T00:38:07.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f284fe00640 1 -- 172.21.15.40:0/2527084453 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950713899 0 0) 0x7f28500b34f0 con 0x7f28500cfdf0 2024-03-24T00:38:07.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f284fe00640 1 -- 172.21.15.40:0/2527084453 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28500af710 con 0x7f28500cfdf0 2024-03-24T00:38:07.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f2854c00640 1 -- 172.21.15.40:0/2527084453 >> v1:172.21.15.53:6789/0 conn(0x7f28500cfdf0 legacy=0x7f28500d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f2854c00640 1 -- 172.21.15.40:0/2527084453 >> v1:172.21.15.40:6789/0 conn(0x7f28500b43a0 legacy=0x7f28500afbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f2854c00640 1 -- 172.21.15.40:0/2527084453 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28500b28d0 con 0x7f28500d08e0 2024-03-24T00:38:07.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f2854c00640 1 --2- 172.21.15.40:0/2527084453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28500d08e0 0x7f285015f090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f283c0097f0 tx=0x7f283c01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7a3be69dfc918cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:07.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.913+0000 7f284fe00640 1 -- 172.21.15.40:0/2527084453 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c0b0070 con 0x7f28500d08e0 2024-03-24T00:38:07.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f284fe00640 1 -- 172.21.15.40:0/2527084453 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283c0ab600 con 0x7f28500d08e0 2024-03-24T00:38:07.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f284fe00640 1 -- 172.21.15.40:0/2527084453 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c0ab9d0 con 0x7f28500d08e0 2024-03-24T00:38:07.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 -- 172.21.15.40:0/2527084453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28500d08e0 msgr2=0x7f285015f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 --2- 172.21.15.40:0/2527084453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28500d08e0 0x7f285015f090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f283c0097f0 tx=0x7f283c01c4e0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 -- 172.21.15.40:0/2527084453 shutdown_connections 2024-03-24T00:38:07.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 --2- 172.21.15.40:0/2527084453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28500d08e0 0x7f285015f090 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 -- 172.21.15.40:0/2527084453 >> 172.21.15.40:0/2527084453 conn(0x7f28500b3880 msgr2=0x7f28500b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:07.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 -- 172.21.15.40:0/2527084453 shutdown_connections 2024-03-24T00:38:07.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 -- 172.21.15.40:0/2527084453 wait complete. 2024-03-24T00:38:07.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 Processor -- start 2024-03-24T00:38:07.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 -- start start 2024-03-24T00:38:07.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28500b43a0 0x7f285015c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28500cfdf0 0x7f2850155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2850155eb0 0x7f2850156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28500b32a0 con 0x7f28500b43a0 2024-03-24T00:38:07.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28500b0de0 con 0x7f2850155eb0 2024-03-24T00:38:07.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2858000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2850169d00 con 0x7f28500cfdf0 2024-03-24T00:38:07.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2855600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28500b43a0 0x7f285015c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2855600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28500b43a0 0x7f285015c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39450/0 (socket says 172.21.15.40:39450) 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2855600640 1 -- 172.21.15.40:0/3737790279 learned_addr learned my addr 172.21.15.40:0/3737790279 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2854c00640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28500cfdf0 0x7f2850155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.917+0000 7f2856000640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2850155eb0 0x7f2850156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f2855600640 1 -- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28500cfdf0 msgr2=0x7f2850155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f2855600640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28500cfdf0 0x7f2850155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f2855600640 1 -- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2850155eb0 msgr2=0x7f2850156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f2855600640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2850155eb0 0x7f2850156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f2855600640 1 -- 172.21.15.40:0/3737790279 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28500b5d80 con 0x7f28500b43a0 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f2855600640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28500b43a0 0x7f285015c9e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f284400c4d0 tx=0x7f284400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f284d600640 1 -- 172.21.15.40:0/3737790279 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2844018070 con 0x7f28500b43a0 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f284d600640 1 -- 172.21.15.40:0/3737790279 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2844007480 con 0x7f28500b43a0 2024-03-24T00:38:07.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f284d600640 1 -- 172.21.15.40:0/3737790279 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28440047f0 con 0x7f28500b43a0 2024-03-24T00:38:07.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f2858000640 1 -- 172.21.15.40:0/3737790279 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28500b6060 con 0x7f28500b43a0 2024-03-24T00:38:07.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f2858000640 1 -- 172.21.15.40:0/3737790279 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28500b6620 con 0x7f28500b43a0 2024-03-24T00:38:07.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f284d600640 1 -- 172.21.15.40:0/3737790279 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2844004990 con 0x7f28500b43a0 2024-03-24T00:38:07.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f284d600640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2830067690 0x7f2830069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f284d600640 1 -- 172.21.15.40:0/3737790279 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2844084a60 con 0x7f28500b43a0 2024-03-24T00:38:07.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.921+0000 7f2854c00640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2830067690 0x7f2830069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 Processor -- start 2024-03-24T00:38:07.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 -- start start 2024-03-24T00:38:07.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2854c00640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2830067690 0x7f2830069b50 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f283c0ad000 tx=0x7f283c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:07.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd335920 0x557cbd335d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd336240 0x557cbd2ebab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd2ebff0 0x557cbd2ec3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cbd139680 con 0x557cbd2ebff0 2024-03-24T00:38:07.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cbd0c0b00 con 0x557cbd336240 2024-03-24T00:38:07.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x557cbd135720 con 0x557cbd335920 2024-03-24T00:38:07.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2856000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd2ebff0 0x557cbd2ec3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2854c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd336240 0x557cbd2ebab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2856000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd2ebff0 0x557cbd2ec3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39462/0 (socket says 172.21.15.40:39462) 2024-03-24T00:38:07.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2856000640 1 -- 172.21.15.40:0/2645441388 learned_addr learned my addr 172.21.15.40:0/2645441388 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:07.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2855600640 1 --2- 172.21.15.40:0/2645441388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd335920 0x557cbd335d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2854c00640 1 -- 172.21.15.40:0/2645441388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd335920 msgr2=0x557cbd335d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:07.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2854c00640 1 --2- 172.21.15.40:0/2645441388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd335920 0x557cbd335d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2854c00640 1 -- 172.21.15.40:0/2645441388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd2ebff0 msgr2=0x557cbd2ec3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:07.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2854c00640 1 --2- 172.21.15.40:0/2645441388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd2ebff0 0x557cbd2ec3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2854c00640 1 -- 172.21.15.40:0/2645441388 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cbd111940 con 0x557cbd336240 2024-03-24T00:38:07.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2856000640 1 --2- 172.21.15.40:0/2645441388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd2ebff0 0x557cbd2ec3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:07.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2854c00640 1 --2- 172.21.15.40:0/2645441388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd336240 0x557cbd2ebab0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f283c0ab000 tx=0x7f283c004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0e04fe789bde5c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:07.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2842000640 1 -- 172.21.15.40:0/2645441388 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c0b0040 con 0x557cbd336240 2024-03-24T00:38:07.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2842000640 1 -- 172.21.15.40:0/2645441388 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283c0abd20 con 0x557cbd336240 2024-03-24T00:38:07.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f2842000640 1 -- 172.21.15.40:0/2645441388 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c0b6680 con 0x557cbd336240 2024-03-24T00:38:07.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 -- 172.21.15.40:0/2645441388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd336240 msgr2=0x557cbd2ebab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 --2- 172.21.15.40:0/2645441388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd336240 0x557cbd2ebab0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f283c0ab000 tx=0x7f283c004740 comp rx=0 tx=0).stop 2024-03-24T00:38:07.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 -- 172.21.15.40:0/2645441388 shutdown_connections 2024-03-24T00:38:07.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 --2- 172.21.15.40:0/2645441388 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd2ebff0 0x557cbd2ec3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 --2- 172.21.15.40:0/2645441388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd336240 0x557cbd2ebab0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 --2- 172.21.15.40:0/2645441388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd335920 0x557cbd335d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 -- 172.21.15.40:0/2645441388 >> 172.21.15.40:0/2645441388 conn(0x557cbd2e7fa0 msgr2=0x557cbd2ea390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:07.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 -- 172.21.15.40:0/2645441388 shutdown_connections 2024-03-24T00:38:07.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 -- 172.21.15.40:0/2645441388 wait complete. 2024-03-24T00:38:07.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.925+0000 7f285b551000 1 -- start start 2024-03-24T00:38:07.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 1 Processor -- start 2024-03-24T00:38:07.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd2ebff0 0x557cbd31ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd335920 0x557cbd31b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 0x557cbd31b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cbd138990 con 0x557cbd336240 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cbd0f3ae0 con 0x557cbd2ebff0 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x557cbd0d1750 con 0x557cbd335920 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 10 client.0 _ll_register_callbacks cb 0x557cbd2d6ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2856000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 0x557cbd31b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2856000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 0x557cbd31b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39470/0 (socket says 172.21.15.40:39470) 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2856000640 1 -- 172.21.15.40:0/2707014901 learned_addr learned my addr 172.21.15.40:0/2707014901 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2854c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd335920 0x557cbd31b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2855600640 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd2ebff0 0x557cbd31ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2856000640 1 -- 172.21.15.40:0/2707014901 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd335920 msgr2=0x557cbd31b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2856000640 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd335920 0x557cbd31b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2856000640 1 -- 172.21.15.40:0/2707014901 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd2ebff0 msgr2=0x557cbd31ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:07.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2856000640 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd2ebff0 0x557cbd31ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2856000640 1 -- 172.21.15.40:0/2707014901 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557cbd17ed80 con 0x557cbd336240 2024-03-24T00:38:07.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2855600640 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd2ebff0 0x557cbd31ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:07.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2856000640 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 0x557cbd31b7c0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f2848012ce0 tx=0x7f2848010610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2cb0c40d62758a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:07.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2836a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:38:07.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2836a00640 1 -- 172.21.15.40:0/2707014901 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848019020 con 0x557cbd336240 2024-03-24T00:38:07.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2836a00640 1 -- 172.21.15.40:0/2707014901 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28480070d0 con 0x557cbd336240 2024-03-24T00:38:07.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 1 -- 172.21.15.40:0/2707014901 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f283c0abd20 con 0x557cbd336240 2024-03-24T00:38:07.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2836a00640 1 -- 172.21.15.40:0/2707014901 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2848023050 con 0x557cbd336240 2024-03-24T00:38:07.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 10 client.8544 fetch_fsmap learned FSMap version 112 2024-03-24T00:38:07.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 1 -- 172.21.15.40:0/2707014901 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) v3 -- 0x557cbd0c0d10 con 0x557cbd336240 2024-03-24T00:38:07.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2836a00640 1 -- 172.21.15.40:0/2707014901 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283c0abd20 con 0x557cbd336240 2024-03-24T00:38:07.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2836a00640 1 -- 172.21.15.40:0/2707014901 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 112) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f2848026020 con 0x557cbd336240 2024-03-24T00:38:07.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 10 client.8544 fetch_fsmap finished waiting for FSMap version 112 2024-03-24T00:38:07.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 10 client.8544 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:38:07.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 20 client.8544 populate_metadata read hostname 'smithi040' 2024-03-24T00:38:07.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x557cbd31c060 0x557cbd347ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 4 client.8544 mds_command: new command op to 8418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:07.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f285b551000 1 -- 172.21.15.40:0/2707014901 --> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557cbd0d4b60 con 0x557cbd31c060 2024-03-24T00:38:07.940 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2855600640 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x557cbd31c060 0x557cbd347ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2855600640 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x557cbd31c060 0x557cbd347ea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63debddd598ca791 server_cookie=b93a174344b50f33 in_seq=0 out_seq=0 2024-03-24T00:38:07.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.929+0000 7f2836a00640 10 client.8544 ms_handle_connect on v2:172.21.15.53:6835/8812937 2024-03-24T00:38:07.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f2836a00640 1 -- 172.21.15.40:0/2707014901 <== mds.0 v2:172.21.15.53:6835/8812937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557cbd0d4b60 con 0x557cbd31c060 2024-03-24T00:38:07.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f2836a00640 10 client.8544 handle_command_reply: tid=0 2024-03-24T00:38:07.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f285b551000 1 client.8544 shutdown 2024-03-24T00:38:07.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f285b551000 1 -- 172.21.15.40:0/2707014901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 msgr2=0x557cbd31b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f285b551000 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 0x557cbd31b7c0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f2848012ce0 tx=0x7f2848010610 comp rx=0 tx=0).stop 2024-03-24T00:38:07.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f285b551000 1 -- 172.21.15.40:0/2707014901 shutdown_connections 2024-03-24T00:38:07.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f285b551000 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x557cbd31c060 0x557cbd347ea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f285b551000 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 0x557cbd31b7c0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f285b551000 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd335920 0x557cbd31b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f285b551000 1 --2- 172.21.15.40:0/2707014901 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd2ebff0 0x557cbd31ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f2836a00640 0 client.8544 ms_handle_reset on v2:172.21.15.53:6835/8812937 2024-03-24T00:38:07.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f285b551000 1 -- 172.21.15.40:0/2707014901 >> 172.21.15.40:0/2707014901 conn(0x557cbd2e7fa0 msgr2=0x557cbd2eac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:07.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f285b551000 1 -- 172.21.15.40:0/2707014901 shutdown_connections 2024-03-24T00:38:07.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f285b551000 1 -- 172.21.15.40:0/2707014901 wait complete. 2024-03-24T00:38:07.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.933+0000 7f285b551000 20 client.8544 trim_cache size 0 max 16384 2024-03-24T00:38:07.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f285b551000 1 Processor -- start 2024-03-24T00:38:07.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f285b551000 1 -- start start 2024-03-24T00:38:07.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f285b551000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd2ebff0 0x557cbd319900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f285b551000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd335920 0x557cbd319e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f285b551000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd336240 0x557cbd3185d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f285b551000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2850203080 con 0x557cbd335920 2024-03-24T00:38:07.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f285b551000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2850165ac0 con 0x557cbd336240 2024-03-24T00:38:07.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f285b551000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28500b5330 con 0x557cbd2ebff0 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f2854c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd335920 0x557cbd319e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f2854c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd335920 0x557cbd319e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39486/0 (socket says 172.21.15.40:39486) 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f2854c00640 1 -- 172.21.15.40:0/359969208 learned_addr learned my addr 172.21.15.40:0/359969208 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f2856000640 1 --2- 172.21.15.40:0/359969208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd336240 0x557cbd3185d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f2855600640 1 --2- 172.21.15.40:0/359969208 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd2ebff0 0x557cbd319900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f2854c00640 1 -- 172.21.15.40:0/359969208 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd2ebff0 msgr2=0x557cbd319900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f2854c00640 1 --2- 172.21.15.40:0/359969208 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd2ebff0 0x557cbd319900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f2854c00640 1 -- 172.21.15.40:0/359969208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd336240 msgr2=0x557cbd3185d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f2854c00640 1 --2- 172.21.15.40:0/359969208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd336240 0x557cbd3185d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f2854c00640 1 -- 172.21.15.40:0/359969208 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557cbd106540 con 0x557cbd335920 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f2856000640 1 --2- 172.21.15.40:0/359969208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd336240 0x557cbd3185d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.949+0000 7f2854c00640 1 --2- 172.21.15.40:0/359969208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd335920 0x557cbd319e40 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f283c0ae040 tx=0x7f283c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a16ecb7d5c2a7be9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2841600640 1 -- 172.21.15.40:0/359969208 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c0b0040 con 0x557cbd335920 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2841600640 1 -- 172.21.15.40:0/359969208 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283c0ca470 con 0x557cbd335920 2024-03-24T00:38:07.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2841600640 1 -- 172.21.15.40:0/359969208 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c0ca640 con 0x557cbd335920 2024-03-24T00:38:07.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 -- 172.21.15.40:0/359969208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd335920 msgr2=0x557cbd319e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 --2- 172.21.15.40:0/359969208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd335920 0x557cbd319e40 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f283c0ae040 tx=0x7f283c009190 comp rx=0 tx=0).stop 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 -- 172.21.15.40:0/359969208 shutdown_connections 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 --2- 172.21.15.40:0/359969208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd336240 0x557cbd3185d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 --2- 172.21.15.40:0/359969208 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd335920 0x557cbd319e40 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 --2- 172.21.15.40:0/359969208 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd2ebff0 0x557cbd319900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 -- 172.21.15.40:0/359969208 >> 172.21.15.40:0/359969208 conn(0x557cbd2e7fa0 msgr2=0x557cbd342f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 -- 172.21.15.40:0/359969208 shutdown_connections 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 -- 172.21.15.40:0/359969208 wait complete. 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 -- start start 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 Processor -- start 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd2ebff0 0x557cbd348ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 0x557cbd349430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd349970 0x557cbd349d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x557cbd0e1a80 con 0x557cbd336240 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28501654c0 con 0x557cbd349970 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x557cbd1f6700 con 0x557cbd2ebff0 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 10 client.0 _ll_register_callbacks cb 0x7f2848012940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2854c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 0x557cbd349430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2854c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 0x557cbd349430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39490/0 (socket says 172.21.15.40:39490) 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2854c00640 1 -- 172.21.15.40:0/1231866290 learned_addr learned my addr 172.21.15.40:0/1231866290 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2856000640 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd349970 0x557cbd349d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2855600640 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd2ebff0 0x557cbd348ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2854c00640 1 -- 172.21.15.40:0/1231866290 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd2ebff0 msgr2=0x557cbd348ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2854c00640 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd2ebff0 0x557cbd348ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2854c00640 1 -- 172.21.15.40:0/1231866290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd349970 msgr2=0x557cbd349d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2854c00640 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd349970 0x557cbd349d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2854c00640 1 -- 172.21.15.40:0/1231866290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557cbd0cd4b0 con 0x557cbd336240 2024-03-24T00:38:07.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2854c00640 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 0x557cbd349430 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f283c009740 tx=0x7f283c01ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df11c08490a92ac0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2836a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2836a00640 1 -- 172.21.15.40:0/1231866290 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f283c0d7060 con 0x557cbd336240 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2836a00640 1 -- 172.21.15.40:0/1231866290 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f283c0cf070 con 0x557cbd336240 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 -- 172.21.15.40:0/1231866290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f283c0ca470 con 0x557cbd336240 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f2836a00640 1 -- 172.21.15.40:0/1231866290 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7f283c0dd050 con 0x557cbd336240 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 10 client.8550 fetch_fsmap learned FSMap version 112 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.953+0000 7f285b551000 1 -- 172.21.15.40:0/1231866290 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=112,monmap=2+}) v3 -- 0x557cbd0e02f0 con 0x557cbd336240 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2836a00640 1 -- 172.21.15.40:0/1231866290 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f283c0ca470 con 0x557cbd336240 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2836a00640 1 -- 172.21.15.40:0/1231866290 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 112) v1 ==== 3314+0+0 (secure 0 0 0) 0x7f283c0caa50 con 0x557cbd336240 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 10 client.8550 fetch_fsmap finished waiting for FSMap version 112 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 10 client.8550 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 20 client.8550 populate_metadata read hostname 'smithi040' 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x557cbd3a11e0 0x557cbd3a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 4 client.8550 mds_command: new command op to 8418 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:38:07.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 -- 172.21.15.40:0/1231866290 --> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557cbd146bc0 con 0x557cbd3a11e0 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2855600640 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x557cbd3a11e0 0x557cbd3a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2855600640 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x557cbd3a11e0 0x557cbd3a35d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbff56b52131110f server_cookie=49f3a23d0c736f16 in_seq=0 out_seq=0 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2836a00640 10 client.8550 ms_handle_connect on v2:172.21.15.53:6835/8812937 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2836a00640 1 -- 172.21.15.40:0/1231866290 <== mds.0 v2:172.21.15.53:6835/8812937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x557cbd146bc0 con 0x557cbd3a11e0 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2836a00640 10 client.8550 handle_command_reply: tid=0 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 client.8550 shutdown 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 -- 172.21.15.40:0/1231866290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 msgr2=0x557cbd349430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 0x557cbd349430 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f283c009740 tx=0x7f283c01ad20 comp rx=0 tx=0).stop 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 -- 172.21.15.40:0/1231866290 shutdown_connections 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x557cbd3a11e0 0x557cbd3a35d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557cbd349970 0x557cbd349d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2836a00640 0 client.8550 ms_handle_reset on v2:172.21.15.53:6835/8812937 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x557cbd336240 0x557cbd349430 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 --2- 172.21.15.40:0/1231866290 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557cbd2ebff0 0x557cbd348ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 -- 172.21.15.40:0/1231866290 >> 172.21.15.40:0/1231866290 conn(0x557cbd2e7fa0 msgr2=0x557cbd342680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:07.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 -- 172.21.15.40:0/1231866290 shutdown_connections 2024-03-24T00:38:07.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 1 -- 172.21.15.40:0/1231866290 wait complete. 2024-03-24T00:38:07.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f285b551000 20 client.8550 trim_cache size 0 max 16384 2024-03-24T00:38:07.966 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:38:07.966 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:38:07.966 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:38:07.966 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8418, 2024-03-24T00:38:07.966 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:38:07.966 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:38:07.966 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:38:07.966 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 5.9208704880000003, 2024-03-24T00:38:07.967 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 112, 2024-03-24T00:38:07.967 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 106, 2024-03-24T00:38:07.967 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 106, 2024-03-24T00:38:07.967 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 10.758199226 2024-03-24T00:38:07.967 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:38:07.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2858000640 1 -- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2830067690 msgr2=0x7f2830069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2858000640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2830067690 0x7f2830069b50 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f283c0ad000 tx=0x7f283c008040 comp rx=0 tx=0).stop 2024-03-24T00:38:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2858000640 1 -- 172.21.15.40:0/3737790279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28500b43a0 msgr2=0x7f285015c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2858000640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28500b43a0 0x7f285015c9e0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f284400c4d0 tx=0x7f284400c990 comp rx=0 tx=0).stop 2024-03-24T00:38:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2858000640 1 -- 172.21.15.40:0/3737790279 shutdown_connections 2024-03-24T00:38:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2858000640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2830067690 0x7f2830069b50 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2858000640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2850155eb0 0x7f2850156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2858000640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f28500cfdf0 0x7f2850155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2858000640 1 --2- 172.21.15.40:0/3737790279 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f28500b43a0 0x7f285015c9e0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2858000640 1 -- 172.21.15.40:0/3737790279 >> 172.21.15.40:0/3737790279 conn(0x7f28500b3880 msgr2=0x7f28500b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2858000640 1 -- 172.21.15.40:0/3737790279 shutdown_connections 2024-03-24T00:38:07.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:07.957+0000 7f2858000640 1 -- 172.21.15.40:0/3737790279 wait complete. 2024-03-24T00:38:07.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb86400640 1 Processor -- start 2024-03-24T00:38:08.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb86400640 1 -- start start 2024-03-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb86400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb8006e620 0x7fdb800b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb86400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb800ccfa0 0x7fdb800cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:08.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb86400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb80070f60 con 0x7fdb8006ea00 2024-03-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb86400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb800b4c00 con 0x7fdb800ccfa0 2024-03-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb86400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800b3620 con 0x7fdb8006e620 2024-03-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb76a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fdb8006ea00 0x7fdb800b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53354/0 (socket says 172.21.15.40:53354) 2024-03-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb76a00640 1 -- 172.21.15.40:0/4165482513 learned_addr learned my addr 172.21.15.40:0/4165482513 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb7f400640 1 --2- 172.21.15.40:0/4165482513 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb8006e620 0x7fdb800b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb7fe00640 1 --2- 172.21.15.40:0/4165482513 >> v2:172.21.15.40:3300/0 conn(0x7fdb800ccfa0 0x7fdb800cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:08.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb7f400640 1 -- 172.21.15.40:0/4165482513 >> v2:172.21.15.40:3300/0 conn(0x7fdb800ccfa0 msgr2=0x7fdb800cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb7f400640 1 --2- 172.21.15.40:0/4165482513 >> v2:172.21.15.40:3300/0 conn(0x7fdb800ccfa0 0x7fdb800cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb7f400640 1 -- 172.21.15.40:0/4165482513 >> v1:172.21.15.40:6789/0 conn(0x7fdb8006ea00 legacy=0x7fdb800b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb7ea00640 1 -- 172.21.15.40:0/4165482513 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587130952 0 0) 0x7fdb80070f60 con 0x7fdb8006ea00 2024-03-24T00:38:08.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.161+0000 7fdb7f400640 1 -- 172.21.15.40:0/4165482513 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb800b3900 con 0x7fdb8006e620 2024-03-24T00:38:08.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.165+0000 7fdb7f400640 1 --2- 172.21.15.40:0/4165482513 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb8006e620 0x7fdb800b46c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdb700097c0 tx=0x7fdb7001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbe8bccc09e6589e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.165+0000 7fdb7ea00640 1 -- 172.21.15.40:0/4165482513 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb700b0070 con 0x7fdb8006e620 2024-03-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.165+0000 7fdb7ea00640 1 -- 172.21.15.40:0/4165482513 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb700ab600 con 0x7fdb8006e620 2024-03-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.165+0000 7fdb7ea00640 1 -- 172.21.15.40:0/4165482513 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb700ab9d0 con 0x7fdb8006e620 2024-03-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.165+0000 7fdb86400640 1 -- 172.21.15.40:0/4165482513 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb8006e620 msgr2=0x7fdb800b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.165+0000 7fdb86400640 1 --2- 172.21.15.40:0/4165482513 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb8006e620 0x7fdb800b46c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdb700097c0 tx=0x7fdb7001c3e0 comp rx=0 tx=0).stop 2024-03-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.165+0000 7fdb86400640 1 -- 172.21.15.40:0/4165482513 shutdown_connections 2024-03-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.165+0000 7fdb86400640 1 --2- 172.21.15.40:0/4165482513 >> v2:172.21.15.40:3300/0 conn(0x7fdb800ccfa0 0x7fdb800cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.165+0000 7fdb86400640 1 --2- 172.21.15.40:0/4165482513 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb8006e620 0x7fdb800b46c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.165+0000 7fdb86400640 1 -- 172.21.15.40:0/4165482513 >> 172.21.15.40:0/4165482513 conn(0x7fdb800b17b0 msgr2=0x7fdb800b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.165+0000 7fdb86400640 1 -- 172.21.15.40:0/4165482513 shutdown_connections 2024-03-24T00:38:08.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.165+0000 7fdb86400640 1 -- 172.21.15.40:0/4165482513 wait complete. 2024-03-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb86400640 1 Processor -- start 2024-03-24T00:38:08.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb86400640 1 -- start start 2024-03-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb86400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb8006e620 0x7fdb80143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb86400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb8006ea00 0x7fdb8013a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb86400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb800ccfa0 0x7fdb801f97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb86400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb80144920 con 0x7fdb800ccfa0 2024-03-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb86400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8006ee20 con 0x7fdb8006e620 2024-03-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb86400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb801f5800 con 0x7fdb8006ea00 2024-03-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb7fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb800ccfa0 0x7fdb801f97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb7fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb800ccfa0 0x7fdb801f97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39516/0 (socket says 172.21.15.40:39516) 2024-03-24T00:38:08.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb7fe00640 1 -- 172.21.15.40:0/2193618351 learned_addr learned my addr 172.21.15.40:0/2193618351 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb76a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb8006ea00 0x7fdb8013a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb7f400640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb8006e620 0x7fdb80143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb7fe00640 1 -- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb8006ea00 msgr2=0x7fdb8013a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb7fe00640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb8006ea00 0x7fdb8013a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb7fe00640 1 -- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb8006e620 msgr2=0x7fdb80143080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb7fe00640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb8006e620 0x7fdb80143080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb7fe00640 1 -- 172.21.15.40:0/2193618351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb801408e0 con 0x7fdb800ccfa0 2024-03-24T00:38:08.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb7f400640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb8006e620 0x7fdb80143080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb7fe00640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb800ccfa0 0x7fdb801f97e0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800efc0 tx=0x7fdb7800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb77e00640 1 -- 172.21.15.40:0/2193618351 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb78018020 con 0x7fdb800ccfa0 2024-03-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb77e00640 1 -- 172.21.15.40:0/2193618351 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb78014480 con 0x7fdb800ccfa0 2024-03-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb77e00640 1 -- 172.21.15.40:0/2193618351 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb78014870 con 0x7fdb800ccfa0 2024-03-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb86400640 1 -- 172.21.15.40:0/2193618351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb80140bc0 con 0x7fdb800ccfa0 2024-03-24T00:38:08.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb86400640 1 -- 172.21.15.40:0/2193618351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb80141180 con 0x7fdb800ccfa0 2024-03-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.169+0000 7fdb86400640 1 -- 172.21.15.40:0/2193618351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb80141380 con 0x7fdb800ccfa0 2024-03-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.173+0000 7fdb77e00640 1 -- 172.21.15.40:0/2193618351 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fdb7801b020 con 0x7fdb800ccfa0 2024-03-24T00:38:08.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.173+0000 7fdb77e00640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb5c067660 0x7fdb5c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.173+0000 7fdb77e00640 1 -- 172.21.15.40:0/2193618351 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb78086330 con 0x7fdb800ccfa0 2024-03-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.173+0000 7fdb7f400640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb5c067660 0x7fdb5c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:08.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.173+0000 7fdb7f400640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb5c067660 0x7fdb5c069b20 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdb700ad000 tx=0x7fdb700a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:08.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.177+0000 7fdb77e00640 1 -- 172.21.15.40:0/2193618351 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdb78022030 con 0x7fdb800ccfa0 2024-03-24T00:38:08.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 -- 172.21.15.40:0/2193618351 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdb80068170 con 0x7fdb800ccfa0 2024-03-24T00:38:08.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb77e00640 1 -- 172.21.15.40:0/2193618351 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v112) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdb78010040 con 0x7fdb800ccfa0 2024-03-24T00:38:08.385 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:38:08.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 -- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb5c067660 msgr2=0x7fdb5c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:08.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb5c067660 0x7fdb5c069b20 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fdb700ad000 tx=0x7fdb700a6000 comp rx=0 tx=0).stop 2024-03-24T00:38:08.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 -- 172.21.15.40:0/2193618351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb800ccfa0 msgr2=0x7fdb801f97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:08.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb800ccfa0 0x7fdb801f97e0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fdb7800efc0 tx=0x7fdb7800c490 comp rx=0 tx=0).stop 2024-03-24T00:38:08.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 -- 172.21.15.40:0/2193618351 shutdown_connections 2024-03-24T00:38:08.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb5c067660 0x7fdb5c069b20 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:08.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb800ccfa0 0x7fdb801f97e0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:08.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb8006ea00 0x7fdb8013a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:08.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 --2- 172.21.15.40:0/2193618351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb8006e620 0x7fdb80143080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:08.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 -- 172.21.15.40:0/2193618351 >> 172.21.15.40:0/2193618351 conn(0x7fdb800b17b0 msgr2=0x7fdb800cde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:08.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 -- 172.21.15.40:0/2193618351 shutdown_connections 2024-03-24T00:38:08.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:08.381+0000 7fdb86400640 1 -- 172.21.15.40:0/2193618351 wait complete. 2024-03-24T00:38:08.405 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:38:08.406 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:38:08.406 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-03-24T00:38:08.406 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:38:08.406 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:38:08.406 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:38:08.406 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:38:08.406 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:38:08.406 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:38:08.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:08.406 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-03-24T00:38:08.416 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:38:08.416 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:38:08.416 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:38:08.416 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:38:08.416 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-24T00:38:08.416 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:38:08.416 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:38:08.416 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 42061sec preferred_lft 42061sec 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-03-24T00:38:08.417 INFO:teuthology.orchestra.run.smithi040.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 02:42:9e:ab:82:f9 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:38:08.418 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:38:08.419 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-24T00:38:08.419 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:38:08.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:08.419 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:38:08.480 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-03-24T00:38:08.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:08.483 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-03-24T00:38:08.540 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-03-24T00:38:08.540 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:38:08.540 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-24T00:38:08.540 INFO:teuthology.orchestra.run.smithi040.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-24T00:38:08.540 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-24T00:38:08.540 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:38:08.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:08.541 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:38:08.541 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:38:08.541 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T00:38:08.541 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T00:38:08.541 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T00:38:08.541 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:38:08.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:08.632 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-24T00:38:08.641 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:38:08.641 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:38:08.642 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:38:08.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:08.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-03-24T00:38:08.702 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T00:38:08.702 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:38:08.702 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:38:08.751 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:38:08.751 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:38:08.751 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:38:08.802 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:38:08.863 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:38:08.920 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:38:08.920 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:38:08.921 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:38:08.922 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:38:08.923 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:38:08.923 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:38:08.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:08.924 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:38:08.974 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T00:38:08.974 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T00:38:09.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:38:09.040 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:38:09.096 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:38:09.096 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:38:09.096 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:38:09.096 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:38:09.096 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:38:09.096 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:38:09.097 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:38:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:38:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:38:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:38:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:38:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:38:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:38:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:38:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:38:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:38:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:38:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:38:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:38:09.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:09.099 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:38:09.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:2024-03-24T00:38:09.109+0000 7f3495538580 -1 init, newargv = 0x559461712810 newargc=15 2024-03-24T00:38:09.115 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[51406]: starting ceph client 2024-03-24T00:38:09.122 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[51406]: starting fuse 2024-03-24T00:38:09.124 INFO:teuthology.orchestra.run.smithi040.stdout:43 2024-03-24T00:38:09.124 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-24T00:38:09.124 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:38:09.124 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-03-24T00:38:09.124 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-03-24T00:38:09.124 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-03-24T00:38:09.124 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-03-24T00:38:09.124 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:38:09.124 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:38:09.125 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:38:09.126 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.0")) 2024-03-24T00:38:09.126 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-03-24T00:38:09.249 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.0.51406.asok 2024-03-24T00:38:09.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:38:09.261 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51406.asok 2024-03-24T00:38:09.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:09.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51406.asok status 2024-03-24T00:38:09.435 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:38:09.435 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-03-24T00:38:09.435 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:38:09.435 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:38:09.435 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "0", 2024-03-24T00:38:09.435 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-03-24T00:38:09.435 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "51406", 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8565, 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 8565 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:38:09.436 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 172290 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 172290 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.8565 192.168.0.1:0/172290", 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "192.168.0.1:0/172290", 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 112, 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 106, 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-03-24T00:38:09.437 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-03-24T00:38:09.438 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:38:09.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:09.451 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-03-24T00:38:09.451 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T00:38:09.460 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-03-24T00:38:09.460 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T00:38:09.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:09.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:38:09.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random True 2024-03-24T00:38:09.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.705+0000 7fbf97000640 1 Processor -- start 2024-03-24T00:38:09.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.705+0000 7fbf97000640 1 -- start start 2024-03-24T00:38:09.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.705+0000 7fbf97000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbf9015ac60 0x7fbf900b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:09.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf97000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900b3260 con 0x7fbf90071b00 2024-03-24T00:38:09.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf97000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900b3540 con 0x7fbf9006cb80 2024-03-24T00:38:09.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf97000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf900b3e70 con 0x7fbf9015ac60 2024-03-24T00:38:09.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf95000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbf90071b00 0x7fbf900ccd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53362/0 (socket says 172.21.15.40:53362) 2024-03-24T00:38:09.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf95000640 1 -- 172.21.15.40:0/1868116224 learned_addr learned my addr 172.21.15.40:0/1868116224 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:09.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf8fe00640 1 --2- 172.21.15.40:0/1868116224 >> v2:172.21.15.53:3300/0 conn(0x7fbf9015ac60 0x7fbf900b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:09.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf8f400640 1 -- 172.21.15.40:0/1868116224 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021818001 0 0) 0x7fbf900b3260 con 0x7fbf90071b00 2024-03-24T00:38:09.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf8f400640 1 -- 172.21.15.40:0/1868116224 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf74005740 con 0x7fbf90071b00 2024-03-24T00:38:09.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf8f400640 1 -- 172.21.15.40:0/1868116224 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12127883 0 0) 0x7fbf900b3540 con 0x7fbf9006cb80 2024-03-24T00:38:09.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf8f400640 1 -- 172.21.15.40:0/1868116224 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf900b3260 con 0x7fbf9006cb80 2024-03-24T00:38:09.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf8fe00640 1 -- 172.21.15.40:0/1868116224 >> v1:172.21.15.53:6789/0 conn(0x7fbf9006cb80 legacy=0x7fbf900b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:09.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf8fe00640 1 -- 172.21.15.40:0/1868116224 >> v1:172.21.15.40:6789/0 conn(0x7fbf90071b00 legacy=0x7fbf900ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:09.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf8fe00640 1 -- 172.21.15.40:0/1868116224 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf900b3820 con 0x7fbf9015ac60 2024-03-24T00:38:09.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf8fe00640 1 --2- 172.21.15.40:0/1868116224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf9015ac60 0x7fbf900b2960 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009d90 tx=0x7fbf8001c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bfbcf6ef1f12809 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:09.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf8f400640 1 -- 172.21.15.40:0/1868116224 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf800b2070 con 0x7fbf9015ac60 2024-03-24T00:38:09.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf8f400640 1 -- 172.21.15.40:0/1868116224 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf800a5890 con 0x7fbf9015ac60 2024-03-24T00:38:09.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.709+0000 7fbf8f400640 1 -- 172.21.15.40:0/1868116224 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf800a5c60 con 0x7fbf9015ac60 2024-03-24T00:38:09.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 -- 172.21.15.40:0/1868116224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf9015ac60 msgr2=0x7fbf900b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:09.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 --2- 172.21.15.40:0/1868116224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf9015ac60 0x7fbf900b2960 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009d90 tx=0x7fbf8001c8b0 comp rx=0 tx=0).stop 2024-03-24T00:38:09.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 -- 172.21.15.40:0/1868116224 shutdown_connections 2024-03-24T00:38:09.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 --2- 172.21.15.40:0/1868116224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf9015ac60 0x7fbf900b2960 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:09.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 -- 172.21.15.40:0/1868116224 >> 172.21.15.40:0/1868116224 conn(0x7fbf900b1660 msgr2=0x7fbf900b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:09.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 -- 172.21.15.40:0/1868116224 shutdown_connections 2024-03-24T00:38:09.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 -- 172.21.15.40:0/1868116224 wait complete. 2024-03-24T00:38:09.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 Processor -- start 2024-03-24T00:38:09.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 -- start start 2024-03-24T00:38:09.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf9006cb80 0x7fbf90151600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:09.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf90071b00 0x7fbf901473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:09.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf9015ac60 0x7fbf9015bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:09.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900b3020 con 0x7fbf90071b00 2024-03-24T00:38:09.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900b2ea0 con 0x7fbf9015ac60 2024-03-24T00:38:09.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf97000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9015b040 con 0x7fbf9006cb80 2024-03-24T00:38:09.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf87400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf90071b00 0x7fbf901473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:09.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf8fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf9006cb80 0x7fbf90151600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:09.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf87400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf90071b00 0x7fbf901473c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39530/0 (socket says 172.21.15.40:39530) 2024-03-24T00:38:09.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf87400640 1 -- 172.21.15.40:0/3668000360 learned_addr learned my addr 172.21.15.40:0/3668000360 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:09.720 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf8fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf9006cb80 0x7fbf90151600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:38776/0 (socket says 172.21.15.40:38776) 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf95000640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf9015ac60 0x7fbf9015bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf8fe00640 1 -- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf9015ac60 msgr2=0x7fbf9015bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf8fe00640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf9015ac60 0x7fbf9015bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf8fe00640 1 -- 172.21.15.40:0/3668000360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf90071b00 msgr2=0x7fbf901473c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf8fe00640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf90071b00 0x7fbf901473c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf8fe00640 1 -- 172.21.15.40:0/3668000360 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf90151be0 con 0x7fbf9006cb80 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.713+0000 7fbf87400640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf90071b00 0x7fbf901473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.717+0000 7fbf8fe00640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf9006cb80 0x7fbf90151600 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf800ac000 tx=0x7fbf80019650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.717+0000 7fbf8cc00640 1 -- 172.21.15.40:0/3668000360 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf800bd020 con 0x7fbf9006cb80 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.717+0000 7fbf8cc00640 1 -- 172.21.15.40:0/3668000360 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf800b8670 con 0x7fbf9006cb80 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.717+0000 7fbf8cc00640 1 -- 172.21.15.40:0/3668000360 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf800b2040 con 0x7fbf9006cb80 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.717+0000 7fbf97000640 1 -- 172.21.15.40:0/3668000360 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf90151ec0 con 0x7fbf9006cb80 2024-03-24T00:38:09.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.717+0000 7fbf97000640 1 -- 172.21.15.40:0/3668000360 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf90152480 con 0x7fbf9006cb80 2024-03-24T00:38:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.717+0000 7fbf97000640 1 -- 172.21.15.40:0/3668000360 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf90068170 con 0x7fbf9006cb80 2024-03-24T00:38:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.717+0000 7fbf8cc00640 1 -- 172.21.15.40:0/3668000360 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbf800c0020 con 0x7fbf9006cb80 2024-03-24T00:38:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.717+0000 7fbf8cc00640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf6c067660 0x7fbf6c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:09.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.717+0000 7fbf8cc00640 1 -- 172.21.15.40:0/3668000360 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf80120680 con 0x7fbf9006cb80 2024-03-24T00:38:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.717+0000 7fbf87400640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf6c067660 0x7fbf6c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:09.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.721+0000 7fbf87400640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf6c067660 0x7fbf6c069b20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbf900b1130 tx=0x7fbf78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:09.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.725+0000 7fbf8cc00640 1 -- 172.21.15.40:0/3668000360 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbf800c7030 con 0x7fbf9006cb80 2024-03-24T00:38:09.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.885+0000 7fbf97000640 1 -- 172.21.15.40:0/3668000360 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random}] v 0) v1 -- 0x7fbf90147900 con 0x7fbf9006cb80 2024-03-24T00:38:09.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf8cc00640 1 -- 172.21.15.40:0/3668000360 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random}]=0 v60)=0 v60) v1 ==== 128+0+0 (secure 0 0 0) 0x7fbf80016560 con 0x7fbf9006cb80 2024-03-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf97000640 1 -- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf6c067660 msgr2=0x7fbf6c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf97000640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf6c067660 0x7fbf6c069b20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbf900b1130 tx=0x7fbf78009040 comp rx=0 tx=0).stop 2024-03-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf97000640 1 -- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf9006cb80 msgr2=0x7fbf90151600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf97000640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf9006cb80 0x7fbf90151600 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf800ac000 tx=0x7fbf80019650 comp rx=0 tx=0).stop 2024-03-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf97000640 1 -- 172.21.15.40:0/3668000360 shutdown_connections 2024-03-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf97000640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf6c067660 0x7fbf6c069b20 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf97000640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf9015ac60 0x7fbf9015bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf97000640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf90071b00 0x7fbf901473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf97000640 1 --2- 172.21.15.40:0/3668000360 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf9006cb80 0x7fbf90151600 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:09.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf97000640 1 -- 172.21.15.40:0/3668000360 >> 172.21.15.40:0/3668000360 conn(0x7fbf900b1660 msgr2=0x7fbf9014a310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf97000640 1 -- 172.21.15.40:0/3668000360 shutdown_connections 2024-03-24T00:38:09.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:09.893+0000 7fbf97000640 1 -- 172.21.15.40:0/3668000360 wait complete. 2024-03-24T00:38:09.917 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_distributed True 2024-03-24T00:38:10.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f8205400640 1 Processor -- start 2024-03-24T00:38:10.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f8205400640 1 -- start start 2024-03-24T00:38:10.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f8205400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f82000cefb0 0x7f82000ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:10.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f8205400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82000cd510 0x7f82000cd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:10.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f8205400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82000cc550 con 0x7f820015f970 2024-03-24T00:38:10.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f8205400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82000ccb40 con 0x7f82000cefb0 2024-03-24T00:38:10.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f8205400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82000d0a10 con 0x7f82000cd510 2024-03-24T00:38:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f81f7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f82000cefb0 0x7f82000ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f81ff200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82000cd510 0x7f82000cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f81fe800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f820015f970 0x7f820015fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:48400/0 (socket says 172.21.15.40:48400) 2024-03-24T00:38:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f81fe800640 1 -- 172.21.15.40:0/1782801281 learned_addr learned my addr 172.21.15.40:0/1782801281 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f81fde00640 1 -- 172.21.15.40:0/1782801281 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312780780 0 0) 0x7f81f0001120 con 0x7f820015f970 2024-03-24T00:38:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f81fde00640 1 -- 172.21.15.40:0/1782801281 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e4003530 con 0x7f820015f970 2024-03-24T00:38:10.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f81f7e00640 1 -- 172.21.15.40:0/1782801281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82000cd510 msgr2=0x7f82000cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:10.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f81f7e00640 1 --2- 172.21.15.40:0/1782801281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82000cd510 0x7f82000cd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f81f7e00640 1 -- 172.21.15.40:0/1782801281 >> v1:172.21.15.53:6789/0 conn(0x7f820015f970 legacy=0x7f820015fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:10.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f81f7e00640 1 -- 172.21.15.40:0/1782801281 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82000d0170 con 0x7f82000cefb0 2024-03-24T00:38:10.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.061+0000 7f81ff200640 1 --2- 172.21.15.40:0/1782801281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82000cd510 0x7f82000cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:10.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.065+0000 7f81f7e00640 1 --2- 172.21.15.40:0/1782801281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82000cefb0 0x7f82000ccfd0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f81e80097c0 tx=0x7f81e801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c24e789855b3ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:10.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.065+0000 7f81fde00640 1 -- 172.21.15.40:0/1782801281 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e80b0070 con 0x7f82000cefb0 2024-03-24T00:38:10.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.065+0000 7f81fde00640 1 -- 172.21.15.40:0/1782801281 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e80ab600 con 0x7f82000cefb0 2024-03-24T00:38:10.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.065+0000 7f81fde00640 1 -- 172.21.15.40:0/1782801281 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e80ab9d0 con 0x7f82000cefb0 2024-03-24T00:38:10.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.065+0000 7f8205400640 1 -- 172.21.15.40:0/1782801281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82000cefb0 msgr2=0x7f82000ccfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:10.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.065+0000 7f8205400640 1 --2- 172.21.15.40:0/1782801281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82000cefb0 0x7f82000ccfd0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f81e80097c0 tx=0x7f81e801c3e0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.065+0000 7f8205400640 1 -- 172.21.15.40:0/1782801281 shutdown_connections 2024-03-24T00:38:10.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.065+0000 7f8205400640 1 --2- 172.21.15.40:0/1782801281 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82000cd510 0x7f82000cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.065+0000 7f8205400640 1 --2- 172.21.15.40:0/1782801281 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82000cefb0 0x7f82000ccfd0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.065+0000 7f8205400640 1 -- 172.21.15.40:0/1782801281 >> 172.21.15.40:0/1782801281 conn(0x7f82000b36a0 msgr2=0x7f82000b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:10.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.065+0000 7f8205400640 1 -- 172.21.15.40:0/1782801281 shutdown_connections 2024-03-24T00:38:10.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.065+0000 7f8205400640 1 -- 172.21.15.40:0/1782801281 wait complete. 2024-03-24T00:38:10.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f8205400640 1 Processor -- start 2024-03-24T00:38:10.072 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f8205400640 1 -- start start 2024-03-24T00:38:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f8205400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82000cd510 0x7f8200159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f8205400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82000cefb0 0x7f82001f6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f8205400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f820015f970 0x7f82001f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f8205400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82000cf440 con 0x7f82000cefb0 2024-03-24T00:38:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f8205400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82001606c0 con 0x7f82000cd510 2024-03-24T00:38:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f8205400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82000cf5c0 con 0x7f820015f970 2024-03-24T00:38:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82000cefb0 0x7f82001f6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81f7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82000cefb0 0x7f82001f6fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39546/0 (socket says 172.21.15.40:39546) 2024-03-24T00:38:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81f7e00640 1 -- 172.21.15.40:0/3926700331 learned_addr learned my addr 172.21.15.40:0/3926700331 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81ff200640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f820015f970 0x7f82001f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:10.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81fe800640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82000cd510 0x7f8200159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:10.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81f7e00640 1 -- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f820015f970 msgr2=0x7f82001f74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:10.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81f7e00640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f820015f970 0x7f82001f74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81f7e00640 1 -- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82000cd510 msgr2=0x7f8200159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:10.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81f7e00640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82000cd510 0x7f8200159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81f7e00640 1 -- 172.21.15.40:0/3926700331 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820015a990 con 0x7f82000cefb0 2024-03-24T00:38:10.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81f7e00640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82000cefb0 0x7f82001f6fa0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f81e801a010 tx=0x7f81e8019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:10.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81f6a00640 1 -- 172.21.15.40:0/3926700331 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e80b0040 con 0x7f82000cefb0 2024-03-24T00:38:10.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81f6a00640 1 -- 172.21.15.40:0/3926700331 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e80b9070 con 0x7f82000cefb0 2024-03-24T00:38:10.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f81f6a00640 1 -- 172.21.15.40:0/3926700331 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8006e20 con 0x7f82000cefb0 2024-03-24T00:38:10.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f8205400640 1 -- 172.21.15.40:0/3926700331 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f820015ac70 con 0x7f82000cefb0 2024-03-24T00:38:10.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.069+0000 7f8205400640 1 -- 172.21.15.40:0/3926700331 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820015b230 con 0x7f82000cefb0 2024-03-24T00:38:10.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.073+0000 7f8205400640 1 -- 172.21.15.40:0/3926700331 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8200068170 con 0x7f82000cefb0 2024-03-24T00:38:10.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.073+0000 7f81f6a00640 1 -- 172.21.15.40:0/3926700331 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f81e80b4650 con 0x7f82000cefb0 2024-03-24T00:38:10.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.073+0000 7f81f6a00640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81dc067660 0x7f81dc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:10.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.073+0000 7f81f6a00640 1 -- 172.21.15.40:0/3926700331 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81e811f430 con 0x7f82000cefb0 2024-03-24T00:38:10.077 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.073+0000 7f81fe800640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81dc067660 0x7f81dc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:10.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.073+0000 7f81fe800640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81dc067660 0x7f81dc069b20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f81f0005890 tx=0x7f81f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:10.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.077+0000 7f81f6a00640 1 -- 172.21.15.40:0/3926700331 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f81e8016090 con 0x7f82000cefb0 2024-03-24T00:38:10.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.241+0000 7f8205400640 1 -- 172.21.15.40:0/3926700331 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_distributed}] v 0) v1 -- 0x7f8200072f30 con 0x7f82000cefb0 2024-03-24T00:38:10.251 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f81f6a00640 1 -- 172.21.15.40:0/3926700331 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_distributed}]=0 v61)=0 v61) v1 ==== 133+0+0 (secure 0 0 0) 0x7f81e80f00a0 con 0x7f82000cefb0 2024-03-24T00:38:10.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f8205400640 1 -- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81dc067660 msgr2=0x7f81dc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:10.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f8205400640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81dc067660 0x7f81dc069b20 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f81f0005890 tx=0x7f81f0009040 comp rx=0 tx=0).stop 2024-03-24T00:38:10.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f8205400640 1 -- 172.21.15.40:0/3926700331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82000cefb0 msgr2=0x7f82001f6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:10.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f8205400640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82000cefb0 0x7f82001f6fa0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f81e801a010 tx=0x7f81e8019180 comp rx=0 tx=0).stop 2024-03-24T00:38:10.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f8205400640 1 -- 172.21.15.40:0/3926700331 shutdown_connections 2024-03-24T00:38:10.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f8205400640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81dc067660 0x7f81dc069b20 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f8205400640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f820015f970 0x7f82001f74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f8205400640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f82000cefb0 0x7f82001f6fa0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f8205400640 1 --2- 172.21.15.40:0/3926700331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82000cd510 0x7f8200159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f8205400640 1 -- 172.21.15.40:0/3926700331 >> 172.21.15.40:0/3926700331 conn(0x7f82000b36a0 msgr2=0x7f82000ce5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:10.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f8205400640 1 -- 172.21.15.40:0/3926700331 shutdown_connections 2024-03-24T00:38:10.255 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.249+0000 7f8205400640 1 -- 172.21.15.40:0/3926700331 wait complete. 2024-03-24T00:38:10.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random_max 1.0 2024-03-24T00:38:10.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c96e00640 1 Processor -- start 2024-03-24T00:38:10.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c96e00640 1 -- start start 2024-03-24T00:38:10.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c96e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c9015bd00 0x7f4c900ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:10.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c96e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4c9015ffd0 0x7f4c900cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:10.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c96e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c90072d30 con 0x7f4c900728d0 2024-03-24T00:38:10.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c96e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c900cf790 con 0x7f4c9015ffd0 2024-03-24T00:38:10.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c96e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c90071de0 con 0x7f4c9015bd00 2024-03-24T00:38:10.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c87400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4c900728d0 0x7f4c9015f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53374/0 (socket says 172.21.15.40:53374) 2024-03-24T00:38:10.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c87400640 1 -- 172.21.15.40:0/3935064449 learned_addr learned my addr 172.21.15.40:0/3935064449 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:10.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c8fe00640 1 --2- 172.21.15.40:0/3935064449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c9015bd00 0x7f4c900ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:10.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c94e00640 1 --2- 172.21.15.40:0/3935064449 >> v2:172.21.15.53:3300/0 conn(0x7f4c9015ffd0 0x7f4c900cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:10.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c8f400640 1 -- 172.21.15.40:0/3935064449 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788018162 0 0) 0x7f4c90072d30 con 0x7f4c900728d0 2024-03-24T00:38:10.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c8f400640 1 -- 172.21.15.40:0/3935064449 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c74003530 con 0x7f4c900728d0 2024-03-24T00:38:10.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c8fe00640 1 -- 172.21.15.40:0/3935064449 >> v2:172.21.15.53:3300/0 conn(0x7f4c9015ffd0 msgr2=0x7f4c900cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:10.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c8fe00640 1 --2- 172.21.15.40:0/3935064449 >> v2:172.21.15.53:3300/0 conn(0x7f4c9015ffd0 0x7f4c900cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c8fe00640 1 -- 172.21.15.40:0/3935064449 >> v1:172.21.15.40:6789/0 conn(0x7f4c900728d0 legacy=0x7f4c9015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:10.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.413+0000 7f4c8fe00640 1 -- 172.21.15.40:0/3935064449 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c900cea50 con 0x7f4c9015bd00 2024-03-24T00:38:10.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.417+0000 7f4c8fe00640 1 --2- 172.21.15.40:0/3935064449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c9015bd00 0x7f4c900ae8e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4c800097c0 tx=0x7f4c8001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0c0587004dff1c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:10.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.417+0000 7f4c8f400640 1 -- 172.21.15.40:0/3935064449 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c800a7070 con 0x7f4c9015bd00 2024-03-24T00:38:10.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.417+0000 7f4c8f400640 1 -- 172.21.15.40:0/3935064449 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c800b0070 con 0x7f4c9015bd00 2024-03-24T00:38:10.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.417+0000 7f4c8f400640 1 -- 172.21.15.40:0/3935064449 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c800ab890 con 0x7f4c9015bd00 2024-03-24T00:38:10.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.417+0000 7f4c96e00640 1 -- 172.21.15.40:0/3935064449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c9015bd00 msgr2=0x7f4c900ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:10.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.417+0000 7f4c96e00640 1 --2- 172.21.15.40:0/3935064449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c9015bd00 0x7f4c900ae8e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4c800097c0 tx=0x7f4c8001c470 comp rx=0 tx=0).stop 2024-03-24T00:38:10.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.417+0000 7f4c96e00640 1 -- 172.21.15.40:0/3935064449 shutdown_connections 2024-03-24T00:38:10.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.417+0000 7f4c96e00640 1 --2- 172.21.15.40:0/3935064449 >> v2:172.21.15.53:3300/0 conn(0x7f4c9015ffd0 0x7f4c900cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.417+0000 7f4c96e00640 1 --2- 172.21.15.40:0/3935064449 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c9015bd00 0x7f4c900ae8e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.417+0000 7f4c96e00640 1 -- 172.21.15.40:0/3935064449 >> 172.21.15.40:0/3935064449 conn(0x7f4c900b1660 msgr2=0x7f4c900b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:10.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.417+0000 7f4c96e00640 1 -- 172.21.15.40:0/3935064449 shutdown_connections 2024-03-24T00:38:10.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.417+0000 7f4c96e00640 1 -- 172.21.15.40:0/3935064449 wait complete. 2024-03-24T00:38:10.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c96e00640 1 Processor -- start 2024-03-24T00:38:10.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c96e00640 1 -- start start 2024-03-24T00:38:10.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c96e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c900728d0 0x7f4c900b1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:10.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c96e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c9015bd00 0x7f4c9013e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:10.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c96e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c9015ffd0 0x7f4c900b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:10.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c96e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9006d320 con 0x7f4c900728d0 2024-03-24T00:38:10.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c96e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9015c0e0 con 0x7f4c9015ffd0 2024-03-24T00:38:10.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c96e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9006d4a0 con 0x7f4c9015bd00 2024-03-24T00:38:10.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c8fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c900728d0 0x7f4c900b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:10.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c8fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c900728d0 0x7f4c900b1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:39562/0 (socket says 172.21.15.40:39562) 2024-03-24T00:38:10.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c8fe00640 1 -- 172.21.15.40:0/3388836435 learned_addr learned my addr 172.21.15.40:0/3388836435 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:10.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c87400640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c9015bd00 0x7f4c9013e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c94e00640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c9015ffd0 0x7f4c900b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c8fe00640 1 -- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c9015bd00 msgr2=0x7f4c9013e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c8fe00640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c9015bd00 0x7f4c9013e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c8fe00640 1 -- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c9015ffd0 msgr2=0x7f4c900b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c8fe00640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c9015ffd0 0x7f4c900b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c8fe00640 1 -- 172.21.15.40:0/3388836435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9015e5f0 con 0x7f4c900728d0 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c94e00640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c9015ffd0 0x7f4c900b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c8fe00640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c900728d0 0x7f4c900b1080 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f4c8001a0a0 tx=0x7f4c8001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c8cc00640 1 -- 172.21.15.40:0/3388836435 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c800bf610 con 0x7f4c900728d0 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c8cc00640 1 -- 172.21.15.40:0/3388836435 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c800abe50 con 0x7f4c900728d0 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c8cc00640 1 -- 172.21.15.40:0/3388836435 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c800bfa10 con 0x7f4c900728d0 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c96e00640 1 -- 172.21.15.40:0/3388836435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c9015e8d0 con 0x7f4c900728d0 2024-03-24T00:38:10.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c96e00640 1 -- 172.21.15.40:0/3388836435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c9015ee90 con 0x7f4c900728d0 2024-03-24T00:38:10.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.421+0000 7f4c96e00640 1 -- 172.21.15.40:0/3388836435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c90068170 con 0x7f4c900728d0 2024-03-24T00:38:10.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.425+0000 7f4c8cc00640 1 -- 172.21.15.40:0/3388836435 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4c800bd020 con 0x7f4c900728d0 2024-03-24T00:38:10.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.425+0000 7f4c8cc00640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4c6c067660 0x7f4c6c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:10.428 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.425+0000 7f4c8cc00640 1 -- 172.21.15.40:0/3388836435 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c8011fbc0 con 0x7f4c900728d0 2024-03-24T00:38:10.429 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.425+0000 7f4c87400640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4c6c067660 0x7f4c6c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:10.430 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.425+0000 7f4c87400640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4c6c067660 0x7f4c6c069b20 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4c78007fd0 tx=0x7f4c78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:10.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.429+0000 7f4c8cc00640 1 -- 172.21.15.40:0/3388836435 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4c800c6030 con 0x7f4c900728d0 2024-03-24T00:38:10.595 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.589+0000 7f4c96e00640 1 -- 172.21.15.40:0/3388836435 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random_max}] v 0) v1 -- 0x7f4c9006e870 con 0x7f4c900728d0 2024-03-24T00:38:10.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c8cc00640 1 -- 172.21.15.40:0/3388836435 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random_max}]=0 v62)=0 v62) v1 ==== 131+0+0 (secure 0 0 0) 0x7f4c80016090 con 0x7f4c900728d0 2024-03-24T00:38:10.602 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c96e00640 1 -- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4c6c067660 msgr2=0x7f4c6c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c96e00640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4c6c067660 0x7f4c6c069b20 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f4c78007fd0 tx=0x7f4c78009040 comp rx=0 tx=0).stop 2024-03-24T00:38:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c96e00640 1 -- 172.21.15.40:0/3388836435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c900728d0 msgr2=0x7f4c900b1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c96e00640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c900728d0 0x7f4c900b1080 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f4c8001a0a0 tx=0x7f4c8001acb0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c96e00640 1 -- 172.21.15.40:0/3388836435 shutdown_connections 2024-03-24T00:38:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c96e00640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4c6c067660 0x7f4c6c069b20 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c96e00640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c9015ffd0 0x7f4c900b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c96e00640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c9015bd00 0x7f4c9013e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c96e00640 1 --2- 172.21.15.40:0/3388836435 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4c900728d0 0x7f4c900b1080 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c96e00640 1 -- 172.21.15.40:0/3388836435 >> 172.21.15.40:0/3388836435 conn(0x7f4c900b1660 msgr2=0x7f4c900b40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c96e00640 1 -- 172.21.15.40:0/3388836435 shutdown_connections 2024-03-24T00:38:10.603 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:10.597+0000 7f4c96e00640 1 -- 172.21.15.40:0/3388836435 wait complete. 2024-03-24T00:38:10.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:10.621 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:38:10.621 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:38:10.621 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:38:10.621 DEBUG:teuthology.orchestra.run.smithi040:> # Use up a random number of inode numbers so the ephemeral pinning is not the same every test. 2024-03-24T00:38:10.621 DEBUG:teuthology.orchestra.run.smithi040:> mkdir .inode_number_thrash 2024-03-24T00:38:10.621 DEBUG:teuthology.orchestra.run.smithi040:> count=$((RANDOM % 1024)) 2024-03-24T00:38:10.621 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < count; i++)); do touch .inode_number_thrash/$i; done 2024-03-24T00:38:10.621 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf .inode_number_thrash 2024-03-24T00:38:10.621 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:38:16.958 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa50a00640 1 Processor -- start 2024-03-24T00:38:17.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa50a00640 1 -- start start 2024-03-24T00:38:17.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa50a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa4c0b2360 0x7ffa4c0b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa50a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0cf090 con 0x7ffa4c0cebd0 2024-03-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa50a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa4c0cf330 con 0x7ffa4c0b4330 2024-03-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa50a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c0d0f60 con 0x7ffa4c0b2360 2024-03-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa4a800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ffa4c0cebd0 0x7ffa4c0cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42084/0 (socket says 172.21.15.40:42084) 2024-03-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa4a800640 1 -- 172.21.15.40:0/787629200 learned_addr learned my addr 172.21.15.40:0/787629200 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa49e00640 1 --2- 172.21.15.40:0/787629200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa4c0b2360 0x7ffa4c0b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa49400640 1 -- 172.21.15.40:0/787629200 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619096581 0 0) 0x7ffa4c0cf330 con 0x7ffa4c0b4330 2024-03-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa49400640 1 -- 172.21.15.40:0/787629200 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa30003530 con 0x7ffa4c0b4330 2024-03-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa49400640 1 -- 172.21.15.40:0/787629200 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004939258 0 0) 0x7ffa4c0cf090 con 0x7ffa4c0cebd0 2024-03-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa49e00640 1 -- 172.21.15.40:0/787629200 >> v1:172.21.15.53:6789/0 conn(0x7ffa4c0b4330 legacy=0x7ffa4c0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa49e00640 1 -- 172.21.15.40:0/787629200 >> v1:172.21.15.40:6789/0 conn(0x7ffa4c0cebd0 legacy=0x7ffa4c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:17.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.117+0000 7ffa49e00640 1 -- 172.21.15.40:0/787629200 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c0d0910 con 0x7ffa4c0b2360 2024-03-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.121+0000 7ffa49e00640 1 --2- 172.21.15.40:0/787629200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa4c0b2360 0x7ffa4c0b2740 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0097f0 tx=0x7ffa3c01c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80a276e1d1ce112c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.121+0000 7ffa49400640 1 -- 172.21.15.40:0/787629200 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c0b1070 con 0x7ffa4c0b2360 2024-03-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.121+0000 7ffa49400640 1 -- 172.21.15.40:0/787629200 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3c0ac5f0 con 0x7ffa4c0b2360 2024-03-24T00:38:17.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.121+0000 7ffa49400640 1 -- 172.21.15.40:0/787629200 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c0ac9a0 con 0x7ffa4c0b2360 2024-03-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.121+0000 7ffa50a00640 1 -- 172.21.15.40:0/787629200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa4c0b2360 msgr2=0x7ffa4c0b2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.121+0000 7ffa50a00640 1 --2- 172.21.15.40:0/787629200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa4c0b2360 0x7ffa4c0b2740 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0097f0 tx=0x7ffa3c01c390 comp rx=0 tx=0).stop 2024-03-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.121+0000 7ffa50a00640 1 -- 172.21.15.40:0/787629200 shutdown_connections 2024-03-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.121+0000 7ffa50a00640 1 --2- 172.21.15.40:0/787629200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa4c0b2360 0x7ffa4c0b2740 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.121+0000 7ffa50a00640 1 -- 172.21.15.40:0/787629200 >> 172.21.15.40:0/787629200 conn(0x7ffa4c0b3810 msgr2=0x7ffa4c0b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.121+0000 7ffa50a00640 1 -- 172.21.15.40:0/787629200 shutdown_connections 2024-03-24T00:38:17.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.121+0000 7ffa50a00640 1 -- 172.21.15.40:0/787629200 wait complete. 2024-03-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa50a00640 1 Processor -- start 2024-03-24T00:38:17.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa50a00640 1 -- start start 2024-03-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa50a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0b2360 0x7ffa4c160530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa50a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa4c0b4330 0x7ffa4c150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa50a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa4c0cebd0 0x7ffa4c15f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa50a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c0cd4e0 con 0x7ffa4c0b2360 2024-03-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa50a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c0cd360 con 0x7ffa4c0b4330 2024-03-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa50a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c0cd660 con 0x7ffa4c0cebd0 2024-03-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa49e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0b2360 0x7ffa4c160530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa49e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0b2360 0x7ffa4c160530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45768/0 (socket says 172.21.15.40:45768) 2024-03-24T00:38:17.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa49e00640 1 -- 172.21.15.40:0/3609084522 learned_addr learned my addr 172.21.15.40:0/3609084522 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa43e00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa4c0b4330 0x7ffa4c150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa4a800640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa4c0cebd0 0x7ffa4c15f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa49e00640 1 -- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa4c0cebd0 msgr2=0x7ffa4c15f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa49e00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa4c0cebd0 0x7ffa4c15f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa49e00640 1 -- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa4c0b4330 msgr2=0x7ffa4c150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa49e00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa4c0b4330 0x7ffa4c150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa49e00640 1 -- 172.21.15.40:0/3609084522 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c15ec30 con 0x7ffa4c0b2360 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa43e00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa4c0b4330 0x7ffa4c150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa49e00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0b2360 0x7ffa4c160530 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0af040 tx=0x7ffa3c019130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa42000640 1 -- 172.21.15.40:0/3609084522 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c0b9020 con 0x7ffa4c0b2360 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa42000640 1 -- 172.21.15.40:0/3609084522 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3c0ac9a0 con 0x7ffa4c0b2360 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa42000640 1 -- 172.21.15.40:0/3609084522 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3c0b1040 con 0x7ffa4c0b2360 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa50a00640 1 -- 172.21.15.40:0/3609084522 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c0c8e90 con 0x7ffa4c0b2360 2024-03-24T00:38:17.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa50a00640 1 -- 172.21.15.40:0/3609084522 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c1506d0 con 0x7ffa4c0b2360 2024-03-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.125+0000 7ffa50a00640 1 -- 172.21.15.40:0/3609084522 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c068170 con 0x7ffa4c0b2360 2024-03-24T00:38:17.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.129+0000 7ffa42000640 1 -- 172.21.15.40:0/3609084522 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ffa3c0bc020 con 0x7ffa4c0b2360 2024-03-24T00:38:17.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.129+0000 7ffa42000640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffa28067690 0x7ffa28069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:17.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.129+0000 7ffa42000640 1 -- 172.21.15.40:0/3609084522 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa3c11f260 con 0x7ffa4c0b2360 2024-03-24T00:38:17.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.129+0000 7ffa43e00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffa28067690 0x7ffa28069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:17.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.129+0000 7ffa43e00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffa28067690 0x7ffa28069b50 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0c9510 tx=0x7ffa34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:17.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.133+0000 7ffa42000640 1 -- 172.21.15.40:0/3609084522 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ffa3c0c3030 con 0x7ffa4c0b2360 2024-03-24T00:38:17.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.333+0000 7ffa50a00640 1 -- 172.21.15.40:0/3609084522 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7ffa4c0cbe40 con 0x7ffa4c0b2360 2024-03-24T00:38:17.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.905+0000 7ffa42000640 1 -- 172.21.15.40:0/3609084522 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v113) v1 ==== 105+0+0 (secure 0 0 0) 0x7ffa3c0a9070 con 0x7ffa4c0b2360 2024-03-24T00:38:17.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.905+0000 7ffa50a00640 1 -- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffa28067690 msgr2=0x7ffa28069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:17.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.905+0000 7ffa50a00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffa28067690 0x7ffa28069b50 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0c9510 tx=0x7ffa34009040 comp rx=0 tx=0).stop 2024-03-24T00:38:17.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.905+0000 7ffa50a00640 1 -- 172.21.15.40:0/3609084522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0b2360 msgr2=0x7ffa4c160530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:17.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.905+0000 7ffa50a00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0b2360 0x7ffa4c160530 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c0af040 tx=0x7ffa3c019130 comp rx=0 tx=0).stop 2024-03-24T00:38:17.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.905+0000 7ffa50a00640 1 -- 172.21.15.40:0/3609084522 shutdown_connections 2024-03-24T00:38:17.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.905+0000 7ffa50a00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffa28067690 0x7ffa28069b50 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:17.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.905+0000 7ffa50a00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa4c0cebd0 0x7ffa4c15f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:17.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.905+0000 7ffa50a00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa4c0b4330 0x7ffa4c150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:17.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.905+0000 7ffa50a00640 1 --2- 172.21.15.40:0/3609084522 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa4c0b2360 0x7ffa4c160530 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:17.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.905+0000 7ffa50a00640 1 -- 172.21.15.40:0/3609084522 >> 172.21.15.40:0/3609084522 conn(0x7ffa4c0b3810 msgr2=0x7ffa4c0cfc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:17.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.909+0000 7ffa50a00640 1 -- 172.21.15.40:0/3609084522 shutdown_connections 2024-03-24T00:38:17.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:17.909+0000 7ffa50a00640 1 -- 172.21.15.40:0/3609084522 wait complete. 2024-03-24T00:38:17.939 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.081+0000 7f9118000640 1 Processor -- start 2024-03-24T00:38:18.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.081+0000 7f9118000640 1 -- start start 2024-03-24T00:38:18.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9118000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f911013ec80 0x7f91101414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:18.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9118000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f91100ce800 0x7f9110071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:18.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9118000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9110070e00 con 0x7f91101419f0 2024-03-24T00:38:18.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9118000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9110141e50 con 0x7f91100ce800 2024-03-24T00:38:18.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9118000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91100cee20 con 0x7f911013ec80 2024-03-24T00:38:18.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9114c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f91101419f0 0x7f91100ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:48288/0 (socket says 172.21.15.40:48288) 2024-03-24T00:38:18.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9114c00640 1 -- 172.21.15.40:0/463361813 learned_addr learned my addr 172.21.15.40:0/463361813 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:18.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9115600640 1 --2- 172.21.15.40:0/463361813 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f911013ec80 0x7f91101414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:18.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9116000640 1 --2- 172.21.15.40:0/463361813 >> v2:172.21.15.53:3300/0 conn(0x7f91100ce800 0x7f9110071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:18.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9115600640 1 -- 172.21.15.40:0/463361813 >> v2:172.21.15.53:3300/0 conn(0x7f91100ce800 msgr2=0x7f9110071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:18.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9115600640 1 --2- 172.21.15.40:0/463361813 >> v2:172.21.15.53:3300/0 conn(0x7f91100ce800 0x7f9110071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:18.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9115600640 1 -- 172.21.15.40:0/463361813 >> v1:172.21.15.53:6789/0 conn(0x7f91101419f0 legacy=0x7f91100ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:18.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f910fe00640 1 -- 172.21.15.40:0/463361813 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832583039 0 0) 0x7f9110070e00 con 0x7f91101419f0 2024-03-24T00:38:18.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.085+0000 7f9115600640 1 -- 172.21.15.40:0/463361813 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91100cf100 con 0x7f911013ec80 2024-03-24T00:38:18.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.089+0000 7f9115600640 1 --2- 172.21.15.40:0/463361813 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f911013ec80 0x7f91101414b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f91040097c0 tx=0x7f910401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68020c1d1ccc0a50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:18.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.093+0000 7f910fe00640 1 -- 172.21.15.40:0/463361813 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91040aa620 con 0x7f911013ec80 2024-03-24T00:38:18.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.093+0000 7f910fe00640 1 -- 172.21.15.40:0/463361813 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91040af070 con 0x7f911013ec80 2024-03-24T00:38:18.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.093+0000 7f910fe00640 1 -- 172.21.15.40:0/463361813 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91040aa9e0 con 0x7f911013ec80 2024-03-24T00:38:18.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.093+0000 7f9118000640 1 -- 172.21.15.40:0/463361813 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f911013ec80 msgr2=0x7f91101414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:18.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.093+0000 7f9118000640 1 --2- 172.21.15.40:0/463361813 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f911013ec80 0x7f91101414b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f91040097c0 tx=0x7f910401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:38:18.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.093+0000 7f9118000640 1 -- 172.21.15.40:0/463361813 shutdown_connections 2024-03-24T00:38:18.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.093+0000 7f9118000640 1 --2- 172.21.15.40:0/463361813 >> v2:172.21.15.53:3300/0 conn(0x7f91100ce800 0x7f9110071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:18.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.093+0000 7f9118000640 1 --2- 172.21.15.40:0/463361813 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f911013ec80 0x7f91101414b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:18.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.093+0000 7f9118000640 1 -- 172.21.15.40:0/463361813 >> 172.21.15.40:0/463361813 conn(0x7f91100b17b0 msgr2=0x7f91100b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:18.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.093+0000 7f9118000640 1 -- 172.21.15.40:0/463361813 shutdown_connections 2024-03-24T00:38:18.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.093+0000 7f9118000640 1 -- 172.21.15.40:0/463361813 wait complete. 2024-03-24T00:38:18.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9118000640 1 Processor -- start 2024-03-24T00:38:18.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9118000640 1 -- start start 2024-03-24T00:38:18.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9118000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91100ce800 0x7f911006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:18.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9118000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f911013ec80 0x7f91101408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:18.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9118000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91101419f0 0x7f9110140e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:18.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9118000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91100723e0 con 0x7f911013ec80 2024-03-24T00:38:18.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9118000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9110072260 con 0x7f91100ce800 2024-03-24T00:38:18.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9118000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9110072560 con 0x7f91101419f0 2024-03-24T00:38:18.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9114c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f911013ec80 0x7f91101408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:18.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9114c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f911013ec80 0x7f91101408f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45780/0 (socket says 172.21.15.40:45780) 2024-03-24T00:38:18.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9114c00640 1 -- 172.21.15.40:0/3207329311 learned_addr learned my addr 172.21.15.40:0/3207329311 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:18.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9114c00640 1 -- 172.21.15.40:0/3207329311 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91101419f0 msgr2=0x7f9110140e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9114c00640 1 --2- 172.21.15.40:0/3207329311 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91101419f0 0x7f9110140e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9114c00640 1 -- 172.21.15.40:0/3207329311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91100ce800 msgr2=0x7f911006d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9114c00640 1 --2- 172.21.15.40:0/3207329311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91100ce800 0x7f911006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:18.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9114c00640 1 -- 172.21.15.40:0/3207329311 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9110159900 con 0x7f911013ec80 2024-03-24T00:38:18.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f9114c00640 1 --2- 172.21.15.40:0/3207329311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f911013ec80 0x7f91101408f0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0011f0 tx=0x7f90fc00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:18.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.097+0000 7f910d600640 1 -- 172.21.15.40:0/3207329311 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc018070 con 0x7f911013ec80 2024-03-24T00:38:18.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.101+0000 7f910d600640 1 -- 172.21.15.40:0/3207329311 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90fc0074e0 con 0x7f911013ec80 2024-03-24T00:38:18.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.101+0000 7f9118000640 1 -- 172.21.15.40:0/3207329311 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9110156ba0 con 0x7f911013ec80 2024-03-24T00:38:18.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.101+0000 7f910d600640 1 -- 172.21.15.40:0/3207329311 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc004a10 con 0x7f911013ec80 2024-03-24T00:38:18.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.101+0000 7f9118000640 1 -- 172.21.15.40:0/3207329311 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9110157160 con 0x7f911013ec80 2024-03-24T00:38:18.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.101+0000 7f9118000640 1 -- 172.21.15.40:0/3207329311 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9110068170 con 0x7f911013ec80 2024-03-24T00:38:18.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.101+0000 7f910d600640 1 -- 172.21.15.40:0/3207329311 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f90fc01b070 con 0x7f911013ec80 2024-03-24T00:38:18.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.105+0000 7f910d600640 1 --2- 172.21.15.40:0/3207329311 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f90f0067660 0x7f90f0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:18.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.105+0000 7f910d600640 1 -- 172.21.15.40:0/3207329311 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f90fc084b60 con 0x7f911013ec80 2024-03-24T00:38:18.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.105+0000 7f9115600640 1 --2- 172.21.15.40:0/3207329311 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f90f0067660 0x7f90f0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:18.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.105+0000 7f9115600640 1 --2- 172.21.15.40:0/3207329311 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f90f0067660 0x7f90f0069b20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f91040ae000 tx=0x7f9104004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:18.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.113+0000 7f910d600640 1 -- 172.21.15.40:0/3207329311 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f90fc0557d0 con 0x7f911013ec80 2024-03-24T00:38:18.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.313+0000 7f9118000640 1 -- 172.21.15.40:0/3207329311 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9110072b90 con 0x7f911013ec80 2024-03-24T00:38:18.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f910d600640 1 -- 172.21.15.40:0/3207329311 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+4989 (secure 0 0 0) 0x7f90fc0550d0 con 0x7f911013ec80 2024-03-24T00:38:18.320 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:18.320 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":114,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7079,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/755981654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":755981654},{"type":"v1","addr":"172.21.15.40:6835","nonce":755981654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":8101,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/3877615536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3877615536},{"type":"v1","addr":"172.21.15.53:6836","nonce":3877615536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:01.424871+0000","modified":"2024-03-24T00:38:17.911899+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":8418,"mds_1":8400},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8400":{"gid":8400,"name":"c","rank":1,"incarnation":114,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6834/3883043397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":3883043397},{"type":"v1","addr":"172.21.15.40:6834","nonce":3883043397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8418":{"gid":8418,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/8812937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":8812937},{"type":"v1","addr":"172.21.15.53:6837","nonce":8812937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8418,"qdb_cluster":[8418]},"id":8}]} 2024-03-24T00:38:18.320 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 114 2024-03-24T00:38:18.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f9118000640 1 -- 172.21.15.40:0/3207329311 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f90f0067660 msgr2=0x7f90f0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:18.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f9118000640 1 --2- 172.21.15.40:0/3207329311 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f90f0067660 0x7f90f0069b20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f91040ae000 tx=0x7f9104004740 comp rx=0 tx=0).stop 2024-03-24T00:38:18.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f9118000640 1 -- 172.21.15.40:0/3207329311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f911013ec80 msgr2=0x7f91101408f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:18.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f9118000640 1 --2- 172.21.15.40:0/3207329311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f911013ec80 0x7f91101408f0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0011f0 tx=0x7f90fc00c9b0 comp rx=0 tx=0).stop 2024-03-24T00:38:18.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f9118000640 1 -- 172.21.15.40:0/3207329311 shutdown_connections 2024-03-24T00:38:18.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f9118000640 1 --2- 172.21.15.40:0/3207329311 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f90f0067660 0x7f90f0069b20 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:18.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f9118000640 1 --2- 172.21.15.40:0/3207329311 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91101419f0 0x7f9110140e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:18.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f9118000640 1 --2- 172.21.15.40:0/3207329311 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f911013ec80 0x7f91101408f0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:18.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f9118000640 1 --2- 172.21.15.40:0/3207329311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91100ce800 0x7f911006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:18.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f9118000640 1 -- 172.21.15.40:0/3207329311 >> 172.21.15.40:0/3207329311 conn(0x7f91100b17b0 msgr2=0x7f91100af710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:18.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f9118000640 1 -- 172.21.15.40:0/3207329311 shutdown_connections 2024-03-24T00:38:18.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:18.317+0000 7f9118000640 1 -- 172.21.15.40:0/3207329311 wait complete. 2024-03-24T00:38:18.743 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 114, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:38:01.424871+0000', 'modified': '2024-03-24T00:38:17.911899+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 8418, 'mds_1': 8400}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8400': {'gid': 8400, 'name': 'c', 'rank': 1, 'incarnation': 114, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6834/3883043397', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 3883043397}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 3883043397}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8418': {'gid': 8418, 'name': 'b', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6837/8812937', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6835', 'nonce': 8812937}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 8812937}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8418, 'qdb_cluster': [8418]} 2024-03-24T00:38:18.744 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8400:up:creating 2024-03-24T00:38:19.014 INFO:tasks.ceph.mds.b.smithi053.stderr:2024-03-24T00:38:19.011+0000 7f852ec00640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:38:19.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:19.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.881+0000 7fe092600640 1 Processor -- start 2024-03-24T00:38:19.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.881+0000 7fe092600640 1 -- start start 2024-03-24T00:38:19.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.881+0000 7fe092600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe08c06f4e0 0x7fe08c070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:19.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.881+0000 7fe092600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe08c070030 0x7fe08c0af170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:19.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.881+0000 7fe092600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe08c0d27b0 con 0x7fe08c06f100 2024-03-24T00:38:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.881+0000 7fe092600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe08c070460 con 0x7fe08c070030 2024-03-24T00:38:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.881+0000 7fe092600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c0b1490 con 0x7fe08c06f4e0 2024-03-24T00:38:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fe08c070030 0x7fe08c0af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe08c06f4e0 0x7fe08c070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08b400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe08c06f100 0x7fe08c0aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:48300/0 (socket says 172.21.15.40:48300) 2024-03-24T00:38:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08b400640 1 -- 172.21.15.40:0/2189948264 learned_addr learned my addr 172.21.15.40:0/2189948264 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08be00640 1 -- 172.21.15.40:0/2189948264 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe08c06f4e0 msgr2=0x7fe08c070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:19.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08be00640 1 --2- 172.21.15.40:0/2189948264 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe08c06f4e0 0x7fe08c070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08be00640 1 -- 172.21.15.40:0/2189948264 >> v1:172.21.15.53:6789/0 conn(0x7fe08c06f100 legacy=0x7fe08c0aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08be00640 1 -- 172.21.15.40:0/2189948264 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c06fa70 con 0x7fe08c070030 2024-03-24T00:38:19.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08a000640 1 -- 172.21.15.40:0/2189948264 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127335873 0 0) 0x7fe08c0d27b0 con 0x7fe08c06f100 2024-03-24T00:38:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08be00640 1 --2- 172.21.15.40:0/2189948264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe08c070030 0x7fe08c0af170 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fe084009d90 tx=0x7fe08401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81bda4870c04f4a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08a000640 1 -- 172.21.15.40:0/2189948264 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0840b2070 con 0x7fe08c070030 2024-03-24T00:38:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08a000640 1 -- 172.21.15.40:0/2189948264 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0840a5830 con 0x7fe08c070030 2024-03-24T00:38:19.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.885+0000 7fe08a000640 1 -- 172.21.15.40:0/2189948264 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0840a5c00 con 0x7fe08c070030 2024-03-24T00:38:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 -- 172.21.15.40:0/2189948264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe08c070030 msgr2=0x7fe08c0af170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 --2- 172.21.15.40:0/2189948264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe08c070030 0x7fe08c0af170 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fe084009d90 tx=0x7fe08401c970 comp rx=0 tx=0).stop 2024-03-24T00:38:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 -- 172.21.15.40:0/2189948264 shutdown_connections 2024-03-24T00:38:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 --2- 172.21.15.40:0/2189948264 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe08c06f4e0 0x7fe08c070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 --2- 172.21.15.40:0/2189948264 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe08c070030 0x7fe08c0af170 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 -- 172.21.15.40:0/2189948264 >> 172.21.15.40:0/2189948264 conn(0x7fe08c0d1c90 msgr2=0x7fe08c0d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 -- 172.21.15.40:0/2189948264 shutdown_connections 2024-03-24T00:38:19.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 -- 172.21.15.40:0/2189948264 wait complete. 2024-03-24T00:38:19.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 Processor -- start 2024-03-24T00:38:19.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 -- start start 2024-03-24T00:38:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe08c06f100 0x7fe08c15a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:19.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe08c06f4e0 0x7fe08c0b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:19.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe08c070030 0x7fe08c0b1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:19.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c0d0a70 con 0x7fe08c070030 2024-03-24T00:38:19.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c0d08f0 con 0x7fe08c06f100 2024-03-24T00:38:19.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe092600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08c0af6b0 con 0x7fe08c06f4e0 2024-03-24T00:38:19.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe08be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe08c070030 0x7fe08c0b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:19.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe08be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe08c070030 0x7fe08c0b1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45790/0 (socket says 172.21.15.40:45790) 2024-03-24T00:38:19.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe08b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe08c06f100 0x7fe08c15a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:19.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe08be00640 1 -- 172.21.15.40:0/212521517 learned_addr learned my addr 172.21.15.40:0/212521517 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:19.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe08aa00640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe08c06f4e0 0x7fe08c0b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:19.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe08be00640 1 -- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe08c06f4e0 msgr2=0x7fe08c0b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:19.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe08be00640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe08c06f4e0 0x7fe08c0b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:19.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe08be00640 1 -- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe08c06f100 msgr2=0x7fe08c15a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:19.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe08be00640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe08c06f100 0x7fe08c15a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:19.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.889+0000 7fe08be00640 1 -- 172.21.15.40:0/212521517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08c0b5690 con 0x7fe08c070030 2024-03-24T00:38:19.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.893+0000 7fe08be00640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe08c070030 0x7fe08c0b1610 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fe084009d90 tx=0x7fe084019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:19.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.893+0000 7fe083400640 1 -- 172.21.15.40:0/212521517 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0840b2040 con 0x7fe08c070030 2024-03-24T00:38:19.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.893+0000 7fe083400640 1 -- 172.21.15.40:0/212521517 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0840a5630 con 0x7fe08c070030 2024-03-24T00:38:19.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.893+0000 7fe083400640 1 -- 172.21.15.40:0/212521517 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0840b86b0 con 0x7fe08c070030 2024-03-24T00:38:19.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.893+0000 7fe092600640 1 -- 172.21.15.40:0/212521517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08c0b5970 con 0x7fe08c070030 2024-03-24T00:38:19.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.893+0000 7fe092600640 1 -- 172.21.15.40:0/212521517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08c0b5f30 con 0x7fe08c070030 2024-03-24T00:38:19.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.893+0000 7fe092600640 1 -- 172.21.15.40:0/212521517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08c068170 con 0x7fe08c070030 2024-03-24T00:38:19.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.893+0000 7fe083400640 1 -- 172.21.15.40:0/212521517 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fe0840c1020 con 0x7fe08c070030 2024-03-24T00:38:19.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.897+0000 7fe083400640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe06c067660 0x7fe06c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:19.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.897+0000 7fe083400640 1 -- 172.21.15.40:0/212521517 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe08411fff0 con 0x7fe08c070030 2024-03-24T00:38:19.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.897+0000 7fe08b400640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe06c067660 0x7fe06c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:19.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.897+0000 7fe08b400640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe06c067660 0x7fe06c069b20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe07c007fd0 tx=0x7fe07c007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:19.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:19.901+0000 7fe083400640 1 -- 172.21.15.40:0/212521517 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fe0840c8020 con 0x7fe08c070030 2024-03-24T00:38:20.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 -- 172.21.15.40:0/212521517 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe08c06ddc0 con 0x7fe08c070030 2024-03-24T00:38:20.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe083400640 1 -- 172.21.15.40:0/212521517 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+5011 (secure 0 0 0) 0x7fe084016560 con 0x7fe08c070030 2024-03-24T00:38:20.104 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:20.104 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":117,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7079,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/755981654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":755981654},{"type":"v1","addr":"172.21.15.40:6835","nonce":755981654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":117,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:01.424871+0000","modified":"2024-03-24T00:38:19.922972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8418,"mds_1":8400,"mds_2":8101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8101":{"gid":8101,"name":"d","rank":2,"incarnation":116,"state":"up:active","state_seq":7,"addr":"172.21.15.53:6836/3877615536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3877615536},{"type":"v1","addr":"172.21.15.53:6836","nonce":3877615536}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8400":{"gid":8400,"name":"c","rank":1,"incarnation":114,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6834/3883043397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":3883043397},{"type":"v1","addr":"172.21.15.40:6834","nonce":3883043397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8418":{"gid":8418,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/8812937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":8812937},{"type":"v1","addr":"172.21.15.53:6837","nonce":8812937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8418,"qdb_cluster":[8101,8400,8418]},"id":8}]} 2024-03-24T00:38:20.104 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 117 2024-03-24T00:38:20.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 -- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe06c067660 msgr2=0x7fe06c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe06c067660 0x7fe06c069b20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe07c007fd0 tx=0x7fe07c007d60 comp rx=0 tx=0).stop 2024-03-24T00:38:20.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 -- 172.21.15.40:0/212521517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe08c070030 msgr2=0x7fe08c0b1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe08c070030 0x7fe08c0b1610 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fe084009d90 tx=0x7fe084019710 comp rx=0 tx=0).stop 2024-03-24T00:38:20.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 -- 172.21.15.40:0/212521517 shutdown_connections 2024-03-24T00:38:20.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fe06c067660 0x7fe06c069b20 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fe08c070030 0x7fe08c0b1610 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe08c06f4e0 0x7fe08c0b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 --2- 172.21.15.40:0/212521517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe08c06f100 0x7fe08c15a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 -- 172.21.15.40:0/212521517 >> 172.21.15.40:0/212521517 conn(0x7fe08c0d1c90 msgr2=0x7fe08c1648b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 -- 172.21.15.40:0/212521517 shutdown_connections 2024-03-24T00:38:20.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.101+0000 7fe092600640 1 -- 172.21.15.40:0/212521517 wait complete. 2024-03-24T00:38:20.528 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 117, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:38:01.424871+0000', 'modified': '2024-03-24T00:38:19.922972+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 8418, 'mds_1': 8400, 'mds_2': 8101}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8101': {'gid': 8101, 'name': 'd', 'rank': 2, 'incarnation': 116, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.53:6836/3877615536', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 3877615536}, {'type': 'v1', 'addr': '172.21.15.53:6836', 'nonce': 3877615536}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8400': {'gid': 8400, 'name': 'c', 'rank': 1, 'incarnation': 114, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.40:6834/3883043397', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6833', 'nonce': 3883043397}, {'type': 'v1', 'addr': '172.21.15.40:6834', 'nonce': 3883043397}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8418': {'gid': 8418, 'name': 'b', 'rank': 0, 'incarnation': 108, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6837/8812937', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6835', 'nonce': 8812937}, {'type': 'v1', 'addr': '172.21.15.53:6837', 'nonce': 8812937}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8418, 'qdb_cluster': [8101, 8400, 8418]} 2024-03-24T00:38:20.528 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T00:38:20.529 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-24T00:38:20.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66ca200640 1 Processor -- start 2024-03-24T00:38:20.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66ca200640 1 -- start start 2024-03-24T00:38:20.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66ca200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f66c40b2910 0x7f66c40d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66ca200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f66c40af170 0x7f66c4165e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66ca200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66c40b1e50 con 0x7f66c415f600 2024-03-24T00:38:20.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66ca200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c40b3630 con 0x7f66c40b2910 2024-03-24T00:38:20.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66ca200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66c40b2d80 con 0x7f66c40af170 2024-03-24T00:38:20.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66c2a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f66c40b2910 0x7f66c40d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66c3400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f66c415f600 0x7f66c40aea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:48312/0 (socket says 172.21.15.40:48312) 2024-03-24T00:38:20.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66c3400640 1 -- 172.21.15.40:0/1168780007 learned_addr learned my addr 172.21.15.40:0/1168780007 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:20.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66c3e00640 1 --2- 172.21.15.40:0/1168780007 >> v2:172.21.15.53:3300/0 conn(0x7f66c40af170 0x7f66c4165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66c2a00640 1 -- 172.21.15.40:0/1168780007 >> v2:172.21.15.53:3300/0 conn(0x7f66c40af170 msgr2=0x7f66c4165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66c2a00640 1 --2- 172.21.15.40:0/1168780007 >> v2:172.21.15.53:3300/0 conn(0x7f66c40af170 0x7f66c4165e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66c2a00640 1 -- 172.21.15.40:0/1168780007 >> v1:172.21.15.53:6789/0 conn(0x7f66c415f600 legacy=0x7f66c40aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.669+0000 7f66c2a00640 1 -- 172.21.15.40:0/1168780007 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c40b20f0 con 0x7f66c40b2910 2024-03-24T00:38:20.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66c2a00640 1 --2- 172.21.15.40:0/1168780007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c40b2910 0x7f66c40d2830 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f66c40b1d60 tx=0x7f66b001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8501a03a47a0959 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66c2000640 1 -- 172.21.15.40:0/1168780007 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b00b5070 con 0x7f66c40b2910 2024-03-24T00:38:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66c2000640 1 -- 172.21.15.40:0/1168780007 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b00a8970 con 0x7f66c40b2910 2024-03-24T00:38:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66c2000640 1 -- 172.21.15.40:0/1168780007 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b00a8d40 con 0x7f66c40b2910 2024-03-24T00:38:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66ca200640 1 -- 172.21.15.40:0/1168780007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c40b2910 msgr2=0x7f66c40d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66ca200640 1 --2- 172.21.15.40:0/1168780007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c40b2910 0x7f66c40d2830 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f66c40b1d60 tx=0x7f66b001ec70 comp rx=0 tx=0).stop 2024-03-24T00:38:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66ca200640 1 -- 172.21.15.40:0/1168780007 shutdown_connections 2024-03-24T00:38:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66ca200640 1 --2- 172.21.15.40:0/1168780007 >> v2:172.21.15.53:3300/0 conn(0x7f66c40af170 0x7f66c4165e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66ca200640 1 --2- 172.21.15.40:0/1168780007 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c40b2910 0x7f66c40d2830 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66ca200640 1 -- 172.21.15.40:0/1168780007 >> 172.21.15.40:0/1168780007 conn(0x7f66c40d1d10 msgr2=0x7f66c40d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66ca200640 1 -- 172.21.15.40:0/1168780007 shutdown_connections 2024-03-24T00:38:20.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66ca200640 1 -- 172.21.15.40:0/1168780007 wait complete. 2024-03-24T00:38:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66ca200640 1 Processor -- start 2024-03-24T00:38:20.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.673+0000 7f66ca200640 1 -- start start 2024-03-24T00:38:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66ca200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66c40af170 0x7f66c40cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66ca200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c40b2910 0x7f66c4152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66ca200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66c415f600 0x7f66c40cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66ca200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c40b2fa0 con 0x7f66c40b2910 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66ca200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c40b2740 con 0x7f66c415f600 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66ca200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c415f9e0 con 0x7f66c40af170 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c40b2910 0x7f66c4152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66c2a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c40b2910 0x7f66c4152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45816/0 (socket says 172.21.15.40:45816) 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66c2a00640 1 -- 172.21.15.40:0/4169294411 learned_addr learned my addr 172.21.15.40:0/4169294411 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66c3400640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66c40af170 0x7f66c40cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66c3e00640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66c415f600 0x7f66c40cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66c2a00640 1 -- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66c40af170 msgr2=0x7f66c40cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66c2a00640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66c40af170 0x7f66c40cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66c2a00640 1 -- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66c415f600 msgr2=0x7f66c40cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66c2a00640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66c415f600 0x7f66c40cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66c2a00640 1 -- 172.21.15.40:0/4169294411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c40cc670 con 0x7f66c40b2910 2024-03-24T00:38:20.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66c2a00640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c40b2910 0x7f66c4152130 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f66b00b3040 tx=0x7f66b001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66bb400640 1 -- 172.21.15.40:0/4169294411 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b00bf070 con 0x7f66c40b2910 2024-03-24T00:38:20.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66bb400640 1 -- 172.21.15.40:0/4169294411 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b00b0640 con 0x7f66c40b2910 2024-03-24T00:38:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66bb400640 1 -- 172.21.15.40:0/4169294411 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b00b5040 con 0x7f66c40b2910 2024-03-24T00:38:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66ca200640 1 -- 172.21.15.40:0/4169294411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66c40cc950 con 0x7f66c40b2910 2024-03-24T00:38:20.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.677+0000 7f66ca200640 1 -- 172.21.15.40:0/4169294411 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66c40ccf10 con 0x7f66c40b2910 2024-03-24T00:38:20.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66bb400640 1 -- 172.21.15.40:0/4169294411 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f66b00c2070 con 0x7f66c40b2910 2024-03-24T00:38:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66cd6d5000 1 Processor -- start 2024-03-24T00:38:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66cd6d5000 1 -- start start 2024-03-24T00:38:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66bb400640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f66a4067660 0x7f66a4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66bb400640 1 -- 172.21.15.40:0/4169294411 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66b0121fa0 con 0x7f66c40b2910 2024-03-24T00:38:20.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c3e00640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f66a4067660 0x7f66a4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66cd6d5000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e337920 0x55847e337d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66cd6d5000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e338240 0x55847e2edab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66cd6d5000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e2edff0 0x55847e2ee3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66cd6d5000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55847e13b680 con 0x55847e337920 2024-03-24T00:38:20.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66cd6d5000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55847e0c2b00 con 0x55847e2edff0 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66cd6d5000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55847e137720 con 0x55847e338240 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e337920 0x55847e337d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c2a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e338240 0x55847e2edab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e337920 0x55847e337d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45830/0 (socket says 172.21.15.40:45830) 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e2edff0 0x55847e2ee3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c2a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e338240 0x55847e2edab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:48982/0 (socket says 172.21.15.40:48982) 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c2a00640 1 -- 172.21.15.40:0/2798878518 learned_addr learned my addr 172.21.15.40:0/2798878518 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c3e00640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f66a4067660 0x7f66a4069b20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f66c415a6a0 tx=0x7f66bc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c2a00640 1 -- 172.21.15.40:0/2798878518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e2edff0 msgr2=0x55847e2ee3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c2a00640 1 --2- 172.21.15.40:0/2798878518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e2edff0 0x55847e2ee3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c2a00640 1 -- 172.21.15.40:0/2798878518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e337920 msgr2=0x55847e337d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c2a00640 1 --2- 172.21.15.40:0/2798878518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e337920 0x55847e337d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.681+0000 7f66c2a00640 1 -- 172.21.15.40:0/2798878518 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55847e113940 con 0x55847e338240 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66c2a00640 1 --2- 172.21.15.40:0/2798878518 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e338240 0x55847e2edab0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f66b00eea30 tx=0x7f66b00f24d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e32482815770cd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66afe00640 1 -- 172.21.15.40:0/2798878518 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b00fac30 con 0x55847e338240 2024-03-24T00:38:20.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66afe00640 1 -- 172.21.15.40:0/2798878518 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b00f08d0 con 0x55847e338240 2024-03-24T00:38:20.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66afe00640 1 -- 172.21.15.40:0/2798878518 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b00fa7a0 con 0x55847e338240 2024-03-24T00:38:20.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2798878518 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e338240 msgr2=0x55847e2edab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/2798878518 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e338240 0x55847e2edab0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f66b00eea30 tx=0x7f66b00f24d0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2798878518 shutdown_connections 2024-03-24T00:38:20.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/2798878518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e2edff0 0x55847e2ee3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/2798878518 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e338240 0x55847e2edab0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/2798878518 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e337920 0x55847e337d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2798878518 >> 172.21.15.40:0/2798878518 conn(0x55847e2e9fa0 msgr2=0x55847e2ec390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2798878518 shutdown_connections 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2798878518 wait complete. 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 -- start start 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 Processor -- start 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e2edff0 0x55847e31cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e337920 0x55847e31d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e338240 0x55847e31d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55847e13a990 con 0x55847e338240 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55847e0f5ae0 con 0x55847e337920 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55847e0d3750 con 0x55847e2edff0 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66cd6d5000 10 client.0 _ll_register_callbacks cb 0x55847e2d8ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:20.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e338240 0x55847e31d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66c3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e338240 0x55847e31d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45842/0 (socket says 172.21.15.40:45842) 2024-03-24T00:38:20.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66c3e00640 1 -- 172.21.15.40:0/1813582537 learned_addr learned my addr 172.21.15.40:0/1813582537 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:20.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66c3400640 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e2edff0 0x55847e31cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66c2a00640 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e337920 0x55847e31d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66c3e00640 1 -- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e2edff0 msgr2=0x55847e31cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66c3e00640 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e2edff0 0x55847e31cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66c3e00640 1 -- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e337920 msgr2=0x55847e31d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66c3e00640 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e337920 0x55847e31d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66c3e00640 1 -- 172.21.15.40:0/1813582537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55847e180d80 con 0x55847e338240 2024-03-24T00:38:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.685+0000 7f66c3400640 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e2edff0 0x55847e31cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66c3e00640 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e338240 0x55847e31d7c0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f66bc017fc0 tx=0x7f66bc004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d747bee97172194f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66acc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:38:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66acc00640 1 -- 172.21.15.40:0/1813582537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66bc027030 con 0x55847e338240 2024-03-24T00:38:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1813582537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f66b00f08d0 con 0x55847e338240 2024-03-24T00:38:20.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66acc00640 1 -- 172.21.15.40:0/1813582537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66bc023690 con 0x55847e338240 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66acc00640 1 -- 172.21.15.40:0/1813582537 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66bc023890 con 0x55847e338240 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 10 client.8610 fetch_fsmap learned FSMap version 117 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1813582537 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x55847e0c2d10 con 0x55847e338240 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66acc00640 1 -- 172.21.15.40:0/1813582537 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b00f08d0 con 0x55847e338240 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66acc00640 1 -- 172.21.15.40:0/1813582537 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 117) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f66bc036080 con 0x55847e338240 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 10 client.8610 fetch_fsmap finished waiting for FSMap version 117 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 10 client.8610 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 20 client.8610 populate_metadata read hostname 'smithi040' 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:6834/3877615536,v1:172.21.15.53:6836/3877615536] conn(0x55847e31e090 0x55847e349ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 4 client.8610 mds_command: new command op to 8101 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1813582537 --> [v2:172.21.15.53:6834/3877615536,v1:172.21.15.53:6836/3877615536] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55847e0d6b60 con 0x55847e31e090 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66c3400640 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:6834/3877615536,v1:172.21.15.53:6836/3877615536] conn(0x55847e31e090 0x55847e349ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66c3400640 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:6834/3877615536,v1:172.21.15.53:6836/3877615536] conn(0x55847e31e090 0x55847e349ea0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b010ba4af42cfc78 server_cookie=5cc5afb77a11174d in_seq=0 out_seq=0 2024-03-24T00:38:20.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66acc00640 10 client.8610 ms_handle_connect on v2:172.21.15.53:6834/3877615536 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66acc00640 1 -- 172.21.15.40:0/1813582537 <== mds.2 v2:172.21.15.53:6834/3877615536 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55847e0d6b60 con 0x55847e31e090 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66acc00640 10 client.8610 handle_command_reply: tid=0 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 client.8610 shutdown 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1813582537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e338240 msgr2=0x55847e31d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e338240 0x55847e31d7c0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f66bc017fc0 tx=0x7f66bc004a50 comp rx=0 tx=0).stop 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1813582537 shutdown_connections 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e338240 0x55847e31d7c0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:6834/3877615536,v1:172.21.15.53:6836/3877615536] conn(0x55847e31e090 0x55847e349ea0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e337920 0x55847e31d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/1813582537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e2edff0 0x55847e31cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1813582537 >> 172.21.15.40:0/1813582537 conn(0x55847e2e9fa0 msgr2=0x55847e2ecc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66acc00640 0 client.8610 ms_handle_reset on v2:172.21.15.53:6834/3877615536 2024-03-24T00:38:20.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1813582537 shutdown_connections 2024-03-24T00:38:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1813582537 wait complete. 2024-03-24T00:38:20.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.689+0000 7f66cd6d5000 20 client.8610 trim_cache size 0 max 16384 2024-03-24T00:38:20.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 Processor -- start 2024-03-24T00:38:20.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 -- start start 2024-03-24T00:38:20.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 0x55847e336530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e337920 0x55847e336a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e338240 0x55847e31a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c42000e0 con 0x55847e2edff0 2024-03-24T00:38:20.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c415a1a0 con 0x55847e338240 2024-03-24T00:38:20.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c41ffa80 con 0x55847e337920 2024-03-24T00:38:20.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 0x55847e336530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 0x55847e336530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45844/0 (socket says 172.21.15.40:45844) 2024-03-24T00:38:20.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66c3400640 1 -- 172.21.15.40:0/1328640108 learned_addr learned my addr 172.21.15.40:0/1328640108 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:20.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66c3e00640 1 --2- 172.21.15.40:0/1328640108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e338240 0x55847e31a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66c2a00640 1 --2- 172.21.15.40:0/1328640108 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e337920 0x55847e336a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66c3400640 1 -- 172.21.15.40:0/1328640108 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e337920 msgr2=0x55847e336a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66c3400640 1 --2- 172.21.15.40:0/1328640108 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e337920 0x55847e336a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66c3400640 1 -- 172.21.15.40:0/1328640108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e338240 msgr2=0x55847e31a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66c3400640 1 --2- 172.21.15.40:0/1328640108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e338240 0x55847e31a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66c3400640 1 -- 172.21.15.40:0/1328640108 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55847e108540 con 0x55847e2edff0 2024-03-24T00:38:20.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66c3400640 1 --2- 172.21.15.40:0/1328640108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 0x55847e336530 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f66b4000f70 tx=0x7f66b4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41fea2c66113c51 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66af400640 1 -- 172.21.15.40:0/1328640108 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b4024070 con 0x55847e2edff0 2024-03-24T00:38:20.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66af400640 1 -- 172.21.15.40:0/1328640108 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b401e2d0 con 0x55847e2edff0 2024-03-24T00:38:20.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66af400640 1 -- 172.21.15.40:0/1328640108 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b4005ac0 con 0x55847e2edff0 2024-03-24T00:38:20.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1328640108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 msgr2=0x55847e336530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/1328640108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 0x55847e336530 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f66b4000f70 tx=0x7f66b4009040 comp rx=0 tx=0).stop 2024-03-24T00:38:20.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1328640108 shutdown_connections 2024-03-24T00:38:20.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/1328640108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e338240 0x55847e31a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/1328640108 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e337920 0x55847e336a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/1328640108 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 0x55847e336530 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1328640108 >> 172.21.15.40:0/1328640108 conn(0x55847e2e9fa0 msgr2=0x55847e344f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1328640108 shutdown_connections 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 -- 172.21.15.40:0/1328640108 wait complete. 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 -- start start 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.709+0000 7f66cd6d5000 1 Processor -- start 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 0x55847e34aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e337920 0x55847e34b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e338240 0x55847e34b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55847e0e3a80 con 0x55847e2edff0 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c41f48e0 con 0x55847e338240 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55847e1f8700 con 0x55847e337920 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 10 client.0 _ll_register_callbacks cb 0x7f66bc009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 0x55847e34aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 0x55847e34aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45848/0 (socket says 172.21.15.40:45848) 2024-03-24T00:38:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c3400640 1 -- 172.21.15.40:0/2134011502 learned_addr learned my addr 172.21.15.40:0/2134011502 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c3e00640 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e338240 0x55847e34b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c2a00640 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e337920 0x55847e34b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c3400640 1 -- 172.21.15.40:0/2134011502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e337920 msgr2=0x55847e34b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c3400640 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e337920 0x55847e34b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c3400640 1 -- 172.21.15.40:0/2134011502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e338240 msgr2=0x55847e34b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c3400640 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e338240 0x55847e34b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c3400640 1 -- 172.21.15.40:0/2134011502 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55847e0cf4b0 con 0x55847e2edff0 2024-03-24T00:38:20.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c3400640 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 0x55847e34aef0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f66b4022040 tx=0x7f66bc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98efe378f6c11465 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66acc00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66acc00640 1 -- 172.21.15.40:0/2134011502 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b4024040 con 0x55847e2edff0 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66acc00640 1 -- 172.21.15.40:0/2134011502 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66b402bad0 con 0x55847e2edff0 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2134011502 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f66b401e2d0 con 0x55847e2edff0 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66acc00640 1 -- 172.21.15.40:0/2134011502 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66b4007c70 con 0x55847e2edff0 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 10 client.8616 fetch_fsmap learned FSMap version 117 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2134011502 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x55847e0e22f0 con 0x55847e2edff0 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66acc00640 1 -- 172.21.15.40:0/2134011502 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b401e2d0 con 0x55847e2edff0 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66acc00640 1 -- 172.21.15.40:0/2134011502 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 117) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f66b402bd10 con 0x55847e2edff0 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 10 client.8616 fetch_fsmap finished waiting for FSMap version 117 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 10 client.8616 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 20 client.8616 populate_metadata read hostname 'smithi040' 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.53:6834/3877615536,v1:172.21.15.53:6836/3877615536] conn(0x55847e34beb0 0x55847e34c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 4 client.8616 mds_command: new command op to 8101 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2134011502 --> [v2:172.21.15.53:6834/3877615536,v1:172.21.15.53:6836/3877615536] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55847e148bc0 con 0x55847e34beb0 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c2a00640 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.53:6834/3877615536,v1:172.21.15.53:6836/3877615536] conn(0x55847e34beb0 0x55847e34c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66c2a00640 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.53:6834/3877615536,v1:172.21.15.53:6836/3877615536] conn(0x55847e34beb0 0x55847e34c290 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=415aa32c45b3f85a server_cookie=e95679ea3365bd1f in_seq=0 out_seq=0 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66acc00640 10 client.8616 ms_handle_connect on v2:172.21.15.53:6834/3877615536 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66acc00640 1 -- 172.21.15.40:0/2134011502 <== mds.2 v2:172.21.15.53:6834/3877615536 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55847e148bc0 con 0x55847e34beb0 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66acc00640 10 client.8616 handle_command_reply: tid=0 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 client.8616 shutdown 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2134011502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 msgr2=0x55847e34aef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 0x55847e34aef0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f66b4022040 tx=0x7f66bc014000 comp rx=0 tx=0).stop 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2134011502 shutdown_connections 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.53:6834/3877615536,v1:172.21.15.53:6836/3877615536] conn(0x55847e34beb0 0x55847e34c290 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55847e338240 0x55847e34b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55847e337920 0x55847e34b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66acc00640 0 client.8616 ms_handle_reset on v2:172.21.15.53:6834/3877615536 2024-03-24T00:38:20.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 --2- 172.21.15.40:0/2134011502 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55847e2edff0 0x55847e34aef0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2134011502 >> 172.21.15.40:0/2134011502 conn(0x55847e2e9fa0 msgr2=0x55847e344680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2134011502 shutdown_connections 2024-03-24T00:38:20.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.713+0000 7f66cd6d5000 1 -- 172.21.15.40:0/2134011502 wait complete. 2024-03-24T00:38:20.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66cd6d5000 20 client.8616 trim_cache size 0 max 16384 2024-03-24T00:38:20.724 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:38:20.724 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:38:20.724 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-03-24T00:38:20.724 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8101, 2024-03-24T00:38:20.724 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:38:20.724 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:38:20.724 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:38:20.725 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 1.7964011, 2024-03-24T00:38:20.725 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 117, 2024-03-24T00:38:20.725 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 106, 2024-03-24T00:38:20.725 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 106, 2024-03-24T00:38:20.725 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 23.517271811000001 2024-03-24T00:38:20.725 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:38:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66ca200640 1 -- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f66a4067660 msgr2=0x7f66a4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66ca200640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f66a4067660 0x7f66a4069b20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f66c415a6a0 tx=0x7f66bc00b040 comp rx=0 tx=0).stop 2024-03-24T00:38:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66ca200640 1 -- 172.21.15.40:0/4169294411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c40b2910 msgr2=0x7f66c4152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66ca200640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c40b2910 0x7f66c4152130 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f66b00b3040 tx=0x7f66b001ba10 comp rx=0 tx=0).stop 2024-03-24T00:38:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66ca200640 1 -- 172.21.15.40:0/4169294411 shutdown_connections 2024-03-24T00:38:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66ca200640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f66a4067660 0x7f66a4069b20 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66ca200640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66c415f600 0x7f66c40cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66ca200640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f66c40b2910 0x7f66c4152130 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66ca200640 1 --2- 172.21.15.40:0/4169294411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f66c40af170 0x7f66c40cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66ca200640 1 -- 172.21.15.40:0/4169294411 >> 172.21.15.40:0/4169294411 conn(0x7f66c40d1d10 msgr2=0x7f66c406c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66ca200640 1 -- 172.21.15.40:0/4169294411 shutdown_connections 2024-03-24T00:38:20.726 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.717+0000 7f66ca200640 1 -- 172.21.15.40:0/4169294411 wait complete. 2024-03-24T00:38:20.739 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:38:20.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78b3400640 1 Processor -- start 2024-03-24T00:38:20.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78b3400640 1 -- start start 2024-03-24T00:38:20.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78b3400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 0x7f78ac06e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78b3400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f78ac070c80 0x7f78ac06db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78b3400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78ac06cdc0 con 0x7f78ac071690 2024-03-24T00:38:20.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78b3400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78ac06eec0 con 0x7f78ac070c80 2024-03-24T00:38:20.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78b3400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac071160 con 0x7f78ac06e5a0 2024-03-24T00:38:20.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78abe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f78ac070c80 0x7f78ac06db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78b0a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 0x7f78ac06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78abe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f78ac070c80 0x7f78ac06db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:34450/0 (socket says 172.21.15.40:34450) 2024-03-24T00:38:20.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78abe00640 1 -- 172.21.15.40:0/3930817421 learned_addr learned my addr 172.21.15.40:0/3930817421 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:20.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78b0a00640 1 -- 172.21.15.40:0/3930817421 >> v2:172.21.15.53:3300/0 conn(0x7f78ac070c80 msgr2=0x7f78ac06db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78b0a00640 1 --2- 172.21.15.40:0/3930817421 >> v2:172.21.15.53:3300/0 conn(0x7f78ac070c80 0x7f78ac06db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78b0a00640 1 -- 172.21.15.40:0/3930817421 >> v1:172.21.15.53:6789/0 conn(0x7f78ac071690 legacy=0x7f78ac0b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.877+0000 7f78b0a00640 1 -- 172.21.15.40:0/3930817421 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac06d040 con 0x7f78ac06e5a0 2024-03-24T00:38:20.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.881+0000 7f78b0a00640 1 --2- 172.21.15.40:0/3930817421 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 0x7f78ac06e980 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f78a00097c0 tx=0x7f78a001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a8b3c696c2ad678 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.881+0000 7f78ab400640 1 -- 172.21.15.40:0/3930817421 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a00b0070 con 0x7f78ac06e5a0 2024-03-24T00:38:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.881+0000 7f78ab400640 1 -- 172.21.15.40:0/3930817421 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a00ab600 con 0x7f78ac06e5a0 2024-03-24T00:38:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.881+0000 7f78ab400640 1 -- 172.21.15.40:0/3930817421 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a00ab9d0 con 0x7f78ac06e5a0 2024-03-24T00:38:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.881+0000 7f78b3400640 1 -- 172.21.15.40:0/3930817421 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 msgr2=0x7f78ac06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.881+0000 7f78b3400640 1 --2- 172.21.15.40:0/3930817421 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 0x7f78ac06e980 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f78a00097c0 tx=0x7f78a001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.881+0000 7f78b3400640 1 -- 172.21.15.40:0/3930817421 shutdown_connections 2024-03-24T00:38:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.881+0000 7f78b3400640 1 --2- 172.21.15.40:0/3930817421 >> v2:172.21.15.53:3300/0 conn(0x7f78ac070c80 0x7f78ac06db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.881+0000 7f78b3400640 1 --2- 172.21.15.40:0/3930817421 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 0x7f78ac06e980 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.881+0000 7f78b3400640 1 -- 172.21.15.40:0/3930817421 >> 172.21.15.40:0/3930817421 conn(0x7f78ac0b17b0 msgr2=0x7f78ac0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.881+0000 7f78b3400640 1 -- 172.21.15.40:0/3930817421 shutdown_connections 2024-03-24T00:38:20.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.881+0000 7f78b3400640 1 -- 172.21.15.40:0/3930817421 wait complete. 2024-03-24T00:38:20.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b3400640 1 Processor -- start 2024-03-24T00:38:20.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b3400640 1 -- start start 2024-03-24T00:38:20.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b3400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 0x7f78ac146500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b3400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ac070c80 0x7f78ac152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78ac071690 0x7f78ac0afe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b3400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac16b8c0 con 0x7f78ac070c80 2024-03-24T00:38:20.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b3400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac16b740 con 0x7f78ac071690 2024-03-24T00:38:20.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b3400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac16ba40 con 0x7f78ac06e5a0 2024-03-24T00:38:20.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ac070c80 0x7f78ac152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b0a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 0x7f78ac146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ac070c80 0x7f78ac152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45850/0 (socket says 172.21.15.40:45850) 2024-03-24T00:38:20.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b0a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 0x7f78ac146500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:49016/0 (socket says 172.21.15.40:49016) 2024-03-24T00:38:20.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78abe00640 1 -- 172.21.15.40:0/1851417124 learned_addr learned my addr 172.21.15.40:0/1851417124 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:20.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b1400640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78ac071690 0x7f78ac0afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b0a00640 1 -- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78ac071690 msgr2=0x7f78ac0afe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b0a00640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78ac071690 0x7f78ac0afe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b0a00640 1 -- 172.21.15.40:0/1851417124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ac070c80 msgr2=0x7f78ac152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b0a00640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ac070c80 0x7f78ac152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b0a00640 1 -- 172.21.15.40:0/1851417124 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac15e1a0 con 0x7f78ac06e5a0 2024-03-24T00:38:20.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b0a00640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 0x7f78ac146500 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f78a00ab000 tx=0x7f78a0019240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78a8c00640 1 -- 172.21.15.40:0/1851417124 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a00b9020 con 0x7f78ac06e5a0 2024-03-24T00:38:20.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78a8c00640 1 -- 172.21.15.40:0/1851417124 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a00b5480 con 0x7f78ac06e5a0 2024-03-24T00:38:20.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78a8c00640 1 -- 172.21.15.40:0/1851417124 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a00b0040 con 0x7f78ac06e5a0 2024-03-24T00:38:20.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b3400640 1 -- 172.21.15.40:0/1851417124 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78ac0b09f0 con 0x7f78ac06e5a0 2024-03-24T00:38:20.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.885+0000 7f78b3400640 1 -- 172.21.15.40:0/1851417124 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78ac14f060 con 0x7f78ac06e5a0 2024-03-24T00:38:20.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.889+0000 7f78a8c00640 1 -- 172.21.15.40:0/1851417124 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f78a00bc020 con 0x7f78ac06e5a0 2024-03-24T00:38:20.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.889+0000 7f78a8c00640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f788c067660 0x7f788c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.889+0000 7f78a8c00640 1 -- 172.21.15.40:0/1851417124 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78a011f340 con 0x7f78ac06e5a0 2024-03-24T00:38:20.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.889+0000 7f78abe00640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f788c067660 0x7f788c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.889+0000 7f78abe00640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f788c067660 0x7f788c069b20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0b2550 tx=0x7f7898009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 Processor -- start 2024-03-24T00:38:20.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 -- start start 2024-03-24T00:38:20.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbad920 0x5654fcbadd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcbae240 0x5654fcb63ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 0x5654fcb643d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654fc9b1680 con 0x5654fcb63ff0 2024-03-24T00:38:20.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654fc938b00 con 0x5654fcbad920 2024-03-24T00:38:20.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654fc9ad720 con 0x5654fcbae240 2024-03-24T00:38:20.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 0x5654fcb643d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b0a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbad920 0x5654fcbadd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b0a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbad920 0x5654fcbadd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:34464/0 (socket says 172.21.15.40:34464) 2024-03-24T00:38:20.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b0a00640 1 -- 172.21.15.40:0/2044148969 learned_addr learned my addr 172.21.15.40:0/2044148969 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:20.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 0x5654fcb643d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45856/0 (socket says 172.21.15.40:45856) 2024-03-24T00:38:20.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78abe00640 1 --2- 172.21.15.40:0/2044148969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcbae240 0x5654fcb63ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b1400640 1 -- 172.21.15.40:0/2044148969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcbae240 msgr2=0x5654fcb63ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b1400640 1 --2- 172.21.15.40:0/2044148969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcbae240 0x5654fcb63ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b1400640 1 -- 172.21.15.40:0/2044148969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbad920 msgr2=0x5654fcbadd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b1400640 1 --2- 172.21.15.40:0/2044148969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbad920 0x5654fcbadd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b1400640 1 -- 172.21.15.40:0/2044148969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654fc989940 con 0x5654fcb63ff0 2024-03-24T00:38:20.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b0a00640 1 --2- 172.21.15.40:0/2044148969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbad920 0x5654fcbadd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:20.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b1400640 1 --2- 172.21.15.40:0/2044148969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 0x5654fcb643d0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f78a4001e80 tx=0x7f78a40016b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a292319bcee5dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f789d600640 1 -- 172.21.15.40:0/2044148969 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4002090 con 0x5654fcb63ff0 2024-03-24T00:38:20.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f789d600640 1 -- 172.21.15.40:0/2044148969 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a4018070 con 0x5654fcb63ff0 2024-03-24T00:38:20.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f789d600640 1 -- 172.21.15.40:0/2044148969 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4003ba0 con 0x5654fcb63ff0 2024-03-24T00:38:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 -- 172.21.15.40:0/2044148969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 msgr2=0x5654fcb643d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 --2- 172.21.15.40:0/2044148969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 0x5654fcb643d0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f78a4001e80 tx=0x7f78a40016b0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 -- 172.21.15.40:0/2044148969 shutdown_connections 2024-03-24T00:38:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 --2- 172.21.15.40:0/2044148969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 0x5654fcb643d0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 --2- 172.21.15.40:0/2044148969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcbae240 0x5654fcb63ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 --2- 172.21.15.40:0/2044148969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbad920 0x5654fcbadd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 -- 172.21.15.40:0/2044148969 >> 172.21.15.40:0/2044148969 conn(0x5654fcb5ffa0 msgr2=0x5654fcb62390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 -- 172.21.15.40:0/2044148969 shutdown_connections 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.893+0000 7f78b68f0000 1 -- 172.21.15.40:0/2044148969 wait complete. 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 1 -- start start 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 1 Processor -- start 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 0x5654fcb92d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcbad920 0x5654fcb93280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbae240 0x5654fcb937c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654fc9b0990 con 0x5654fcb63ff0 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654fc96bae0 con 0x5654fcbae240 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654fc949750 con 0x5654fcbad920 2024-03-24T00:38:20.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 10 client.0 _ll_register_callbacks cb 0x5654fcb4eff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b0a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 0x5654fcb92d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbae240 0x5654fcb937c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78abe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcbad920 0x5654fcb93280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbae240 0x5654fcb937c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:34466/0 (socket says 172.21.15.40:34466) 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b1400640 1 -- 172.21.15.40:0/100185995 learned_addr learned my addr 172.21.15.40:0/100185995 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b1400640 1 -- 172.21.15.40:0/100185995 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcbad920 msgr2=0x5654fcb93280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b1400640 1 --2- 172.21.15.40:0/100185995 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcbad920 0x5654fcb93280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b1400640 1 -- 172.21.15.40:0/100185995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 msgr2=0x5654fcb92d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b1400640 1 --2- 172.21.15.40:0/100185995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 0x5654fcb92d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b1400640 1 -- 172.21.15.40:0/100185995 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654fc9f6d80 con 0x5654fcbae240 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b0a00640 1 --2- 172.21.15.40:0/100185995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 0x5654fcb92d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b1400640 1 --2- 172.21.15.40:0/100185995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbae240 0x5654fcb937c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f78a400fa30 tx=0x7f78a4001260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be2e6b8bba768abd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f7892000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f7892000640 1 -- 172.21.15.40:0/100185995 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a40054e0 con 0x5654fcbae240 2024-03-24T00:38:20.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f7892000640 1 -- 172.21.15.40:0/100185995 <== mon.1 v2:172.21.15.53:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78a4005680 con 0x5654fcbae240 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 1 -- 172.21.15.40:0/100185995 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78a4018070 con 0x5654fcbae240 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f7892000640 1 -- 172.21.15.40:0/100185995 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78a402f050 con 0x5654fcbae240 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 10 client.8308 fetch_fsmap learned FSMap version 117 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 1 -- 172.21.15.40:0/100185995 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x5654fc938d10 con 0x5654fcbae240 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f7892000640 1 -- 172.21.15.40:0/100185995 <== mon.1 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a4018070 con 0x5654fcbae240 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f7892000640 1 -- 172.21.15.40:0/100185995 <== mon.1 v2:172.21.15.53:3300/0 5 ==== fsmap(e 117) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f78a4006b60 con 0x5654fcbae240 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 10 client.8308 fetch_fsmap finished waiting for FSMap version 117 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 10 client.8308 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 20 client.8308 populate_metadata read hostname 'smithi040' 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 1 --2- 172.21.15.40:0/100185995 >> [v2:172.21.15.40:6833/3883043397,v1:172.21.15.40:6834/3883043397] conn(0x5654fcb94060 0x5654fcbbfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 4 client.8308 mds_command: new command op to 8400 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b68f0000 1 -- 172.21.15.40:0/100185995 --> [v2:172.21.15.40:6833/3883043397,v1:172.21.15.40:6834/3883043397] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654fc94cb60 con 0x5654fcb94060 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.897+0000 7f78b0a00640 1 --2- 172.21.15.40:0/100185995 >> [v2:172.21.15.40:6833/3883043397,v1:172.21.15.40:6834/3883043397] conn(0x5654fcb94060 0x5654fcbbfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b0a00640 1 --2- 172.21.15.40:0/100185995 >> [v2:172.21.15.40:6833/3883043397,v1:172.21.15.40:6834/3883043397] conn(0x5654fcb94060 0x5654fcbbfea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5c7dee662d66c29a server_cookie=f89a1b3db8806b10 in_seq=0 out_seq=0 2024-03-24T00:38:20.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f7892000640 10 client.8308 ms_handle_connect on v2:172.21.15.40:6833/3883043397 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f7892000640 1 -- 172.21.15.40:0/100185995 <== mds.1 v2:172.21.15.40:6833/3883043397 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5654fc94cb60 con 0x5654fcb94060 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f7892000640 10 client.8308 handle_command_reply: tid=0 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b68f0000 1 client.8308 shutdown 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b68f0000 1 -- 172.21.15.40:0/100185995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbae240 msgr2=0x5654fcb937c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b68f0000 1 --2- 172.21.15.40:0/100185995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbae240 0x5654fcb937c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f78a400fa30 tx=0x7f78a4001260 comp rx=0 tx=0).stop 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b68f0000 1 -- 172.21.15.40:0/100185995 shutdown_connections 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b68f0000 1 --2- 172.21.15.40:0/100185995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbae240 0x5654fcb937c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b68f0000 1 --2- 172.21.15.40:0/100185995 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcbad920 0x5654fcb93280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b68f0000 1 --2- 172.21.15.40:0/100185995 >> [v2:172.21.15.40:6833/3883043397,v1:172.21.15.40:6834/3883043397] conn(0x5654fcb94060 0x5654fcbbfea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b68f0000 1 --2- 172.21.15.40:0/100185995 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcb63ff0 0x5654fcb92d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b68f0000 1 -- 172.21.15.40:0/100185995 >> 172.21.15.40:0/100185995 conn(0x5654fcb5ffa0 msgr2=0x5654fcb62c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f7892000640 0 client.8308 ms_handle_reset on v2:172.21.15.40:6833/3883043397 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b68f0000 1 -- 172.21.15.40:0/100185995 shutdown_connections 2024-03-24T00:38:20.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b68f0000 1 -- 172.21.15.40:0/100185995 wait complete. 2024-03-24T00:38:20.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.901+0000 7f78b68f0000 20 client.8308 trim_cache size 0 max 16384 2024-03-24T00:38:20.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78b68f0000 1 Processor -- start 2024-03-24T00:38:20.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78b68f0000 1 -- start start 2024-03-24T00:38:20.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78b68f0000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcb63ff0 0x5654fcb905d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78b68f0000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbad920 0x5654fcb90b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78b68f0000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbae240 0x5654fcb633a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78b68f0000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac202bf0 con 0x5654fcbad920 2024-03-24T00:38:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78b68f0000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac2031f0 con 0x5654fcbae240 2024-03-24T00:38:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78b68f0000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac1502d0 con 0x5654fcb63ff0 2024-03-24T00:38:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbad920 0x5654fcb90b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78b0a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcb63ff0 0x5654fcb905d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78abe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbad920 0x5654fcb90b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45864/0 (socket says 172.21.15.40:45864) 2024-03-24T00:38:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78abe00640 1 -- 172.21.15.40:0/972660800 learned_addr learned my addr 172.21.15.40:0/972660800 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.917+0000 7f78b1400640 1 --2- 172.21.15.40:0/972660800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbae240 0x5654fcb633a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b0a00640 1 -- 172.21.15.40:0/972660800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbae240 msgr2=0x5654fcb633a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b0a00640 1 --2- 172.21.15.40:0/972660800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbae240 0x5654fcb633a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b0a00640 1 -- 172.21.15.40:0/972660800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbad920 msgr2=0x5654fcb90b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b0a00640 1 --2- 172.21.15.40:0/972660800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbad920 0x5654fcb90b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b0a00640 1 -- 172.21.15.40:0/972660800 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654fc97e540 con 0x5654fcb63ff0 2024-03-24T00:38:20.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b0a00640 1 --2- 172.21.15.40:0/972660800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcb63ff0 0x5654fcb905d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f78a00ef8f0 tx=0x7f78a00b7040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55595b3565821c1c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f789cc00640 1 -- 172.21.15.40:0/972660800 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a011d8e0 con 0x5654fcb63ff0 2024-03-24T00:38:20.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f789cc00640 1 -- 172.21.15.40:0/972660800 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a0000e50 con 0x5654fcb63ff0 2024-03-24T00:38:20.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f789cc00640 1 -- 172.21.15.40:0/972660800 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a00edc60 con 0x5654fcb63ff0 2024-03-24T00:38:20.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 -- 172.21.15.40:0/972660800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcb63ff0 msgr2=0x5654fcb905d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 --2- 172.21.15.40:0/972660800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcb63ff0 0x5654fcb905d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f78a00ef8f0 tx=0x7f78a00b7040 comp rx=0 tx=0).stop 2024-03-24T00:38:20.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 -- 172.21.15.40:0/972660800 shutdown_connections 2024-03-24T00:38:20.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 --2- 172.21.15.40:0/972660800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbae240 0x5654fcb633a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 --2- 172.21.15.40:0/972660800 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbad920 0x5654fcb90b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 --2- 172.21.15.40:0/972660800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcb63ff0 0x5654fcb905d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 -- 172.21.15.40:0/972660800 >> 172.21.15.40:0/972660800 conn(0x5654fcb5ffa0 msgr2=0x5654fcbac500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 -- 172.21.15.40:0/972660800 shutdown_connections 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 -- 172.21.15.40:0/972660800 wait complete. 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 -- start start 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 Processor -- start 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcb63ff0 0x5654fcbbfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbad920 0x5654fcbc03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbae240 0x5654fcbc0920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654fca6e700 con 0x5654fcbae240 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654fc959a80 con 0x5654fcbad920 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654fcb6cc80 con 0x5654fcb63ff0 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b68f0000 10 client.0 _ll_register_callbacks cb 0x7f78a0009420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbae240 0x5654fcbc0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbae240 0x5654fcbc0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45868/0 (socket says 172.21.15.40:45868) 2024-03-24T00:38:20.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b1400640 1 -- 172.21.15.40:0/1892451754 learned_addr learned my addr 172.21.15.40:0/1892451754 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:20.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78abe00640 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbad920 0x5654fcbc03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.921+0000 7f78b0a00640 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcb63ff0 0x5654fcbbfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b1400640 1 -- 172.21.15.40:0/1892451754 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcb63ff0 msgr2=0x5654fcbbfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b1400640 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcb63ff0 0x5654fcbbfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b1400640 1 -- 172.21.15.40:0/1892451754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbad920 msgr2=0x5654fcbc03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:20.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b1400640 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbad920 0x5654fcbc03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b1400640 1 -- 172.21.15.40:0/1892451754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654fc9454b0 con 0x5654fcbae240 2024-03-24T00:38:20.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b1400640 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbae240 0x5654fcbc0920 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f78a400fa30 tx=0x7f78a4009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a7742d46db9a0a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:20.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f7892000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:38:20.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f7892000640 1 -- 172.21.15.40:0/1892451754 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a40056f0 con 0x5654fcbae240 2024-03-24T00:38:20.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f7892000640 1 -- 172.21.15.40:0/1892451754 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78a40296f0 con 0x5654fcbae240 2024-03-24T00:38:20.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b68f0000 1 -- 172.21.15.40:0/1892451754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78a0000e50 con 0x5654fcbae240 2024-03-24T00:38:20.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f7892000640 1 -- 172.21.15.40:0/1892451754 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78a40298f0 con 0x5654fcbae240 2024-03-24T00:38:20.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b68f0000 10 client.8631 fetch_fsmap learned FSMap version 117 2024-03-24T00:38:20.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b68f0000 1 -- 172.21.15.40:0/1892451754 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x5654fc9582f0 con 0x5654fcbae240 2024-03-24T00:38:20.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f7892000640 1 -- 172.21.15.40:0/1892451754 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a0000e50 con 0x5654fcbae240 2024-03-24T00:38:20.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f7892000640 1 -- 172.21.15.40:0/1892451754 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 117) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f78a4029a90 con 0x5654fcbae240 2024-03-24T00:38:20.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b68f0000 10 client.8631 fetch_fsmap finished waiting for FSMap version 117 2024-03-24T00:38:20.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b68f0000 10 client.8631 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:38:20.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b68f0000 20 client.8631 populate_metadata read hostname 'smithi040' 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b68f0000 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.40:6833/3883043397,v1:172.21.15.40:6834/3883043397] conn(0x5654fcbc12e0 0x5654fcc1a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b68f0000 4 client.8631 mds_command: new command op to 8400 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b68f0000 1 -- 172.21.15.40:0/1892451754 --> [v2:172.21.15.40:6833/3883043397,v1:172.21.15.40:6834/3883043397] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5654fc9bebc0 con 0x5654fcbc12e0 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.925+0000 7f78b0a00640 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.40:6833/3883043397,v1:172.21.15.40:6834/3883043397] conn(0x5654fcbc12e0 0x5654fcc1a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b0a00640 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.40:6833/3883043397,v1:172.21.15.40:6834/3883043397] conn(0x5654fcbc12e0 0x5654fcc1a1e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d79a45fa447b82c server_cookie=4e0b46b05470172c in_seq=0 out_seq=0 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f7892000640 10 client.8631 ms_handle_connect on v2:172.21.15.40:6833/3883043397 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f7892000640 1 -- 172.21.15.40:0/1892451754 <== mds.1 v2:172.21.15.40:6833/3883043397 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x5654fc9bebc0 con 0x5654fcbc12e0 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f7892000640 10 client.8631 handle_command_reply: tid=0 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b68f0000 1 client.8631 shutdown 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b68f0000 1 -- 172.21.15.40:0/1892451754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbae240 msgr2=0x5654fcbc0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b68f0000 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbae240 0x5654fcbc0920 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f78a400fa30 tx=0x7f78a4009040 comp rx=0 tx=0).stop 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b68f0000 1 -- 172.21.15.40:0/1892451754 shutdown_connections 2024-03-24T00:38:20.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b68f0000 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.40:6833/3883043397,v1:172.21.15.40:6834/3883043397] conn(0x5654fcbc12e0 0x5654fcc1a1e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b68f0000 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x5654fcbae240 0x5654fcbc0920 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b68f0000 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5654fcbad920 0x5654fcbc03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f7892000640 0 client.8631 ms_handle_reset on v2:172.21.15.40:6833/3883043397 2024-03-24T00:38:20.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b68f0000 1 --2- 172.21.15.40:0/1892451754 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5654fcb63ff0 0x5654fcbbfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b68f0000 1 -- 172.21.15.40:0/1892451754 >> 172.21.15.40:0/1892451754 conn(0x5654fcb5ffa0 msgr2=0x5654fcbacd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b68f0000 1 -- 172.21.15.40:0/1892451754 shutdown_connections 2024-03-24T00:38:20.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b68f0000 1 -- 172.21.15.40:0/1892451754 wait complete. 2024-03-24T00:38:20.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b68f0000 20 client.8631 trim_cache size 0 max 16384 2024-03-24T00:38:20.935 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:38:20.935 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:38:20.935 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-03-24T00:38:20.936 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8400, 2024-03-24T00:38:20.936 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:38:20.936 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:38:20.936 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:38:20.936 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.01893484, 2024-03-24T00:38:20.936 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 117, 2024-03-24T00:38:20.936 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 106, 2024-03-24T00:38:20.936 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 106, 2024-03-24T00:38:20.936 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 23.747766529 2024-03-24T00:38:20.936 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:38:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b3400640 1 -- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f788c067660 msgr2=0x7f788c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b3400640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f788c067660 0x7f788c069b20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f78ac0b2550 tx=0x7f7898009240 comp rx=0 tx=0).stop 2024-03-24T00:38:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b3400640 1 -- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 msgr2=0x7f78ac146500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b3400640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 0x7f78ac146500 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f78a00ab000 tx=0x7f78a0019240 comp rx=0 tx=0).stop 2024-03-24T00:38:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b3400640 1 -- 172.21.15.40:0/1851417124 shutdown_connections 2024-03-24T00:38:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b3400640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f788c067660 0x7f788c069b20 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b3400640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78ac071690 0x7f78ac0afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b3400640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78ac070c80 0x7f78ac152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b3400640 1 --2- 172.21.15.40:0/1851417124 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78ac06e5a0 0x7f78ac146500 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:20.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b3400640 1 -- 172.21.15.40:0/1851417124 >> 172.21.15.40:0/1851417124 conn(0x7f78ac0b17b0 msgr2=0x7f78ac150fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:20.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b3400640 1 -- 172.21.15.40:0/1851417124 shutdown_connections 2024-03-24T00:38:20.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:20.929+0000 7f78b3400640 1 -- 172.21.15.40:0/1851417124 wait complete. 2024-03-24T00:38:20.951 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:38:21.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec4000640 1 Processor -- start 2024-03-24T00:38:21.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec4000640 1 -- start start 2024-03-24T00:38:21.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec4000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcebc13ec80 0x7fcebc1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec4000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcebc1419f0 0x7fcebc0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec4000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcebc070e00 con 0x7fcebc0ce630 2024-03-24T00:38:21.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec4000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc141e50 con 0x7fcebc1419f0 2024-03-24T00:38:21.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec4000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcebc0cee20 con 0x7fcebc13ec80 2024-03-24T00:38:21.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec0c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcebc1419f0 0x7fcebc0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec0c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcebc1419f0 0x7fcebc0ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45880/0 (socket says 172.21.15.40:45880) 2024-03-24T00:38:21.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec1600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcebc13ec80 0x7fcebc1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec0c00640 1 -- 172.21.15.40:0/3127966892 learned_addr learned my addr 172.21.15.40:0/3127966892 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:21.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcebbe00640 1 -- 172.21.15.40:0/3127966892 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667973245 0 0) 0x7fcebc070e00 con 0x7fcebc0ce630 2024-03-24T00:38:21.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec1600640 1 -- 172.21.15.40:0/3127966892 >> v2:172.21.15.40:3300/0 conn(0x7fcebc1419f0 msgr2=0x7fcebc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:21.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec1600640 1 --2- 172.21.15.40:0/3127966892 >> v2:172.21.15.40:3300/0 conn(0x7fcebc1419f0 0x7fcebc0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec1600640 1 -- 172.21.15.40:0/3127966892 >> v1:172.21.15.53:6789/0 conn(0x7fcebc0ce630 legacy=0x7fcebc15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:21.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.089+0000 7fcec1600640 1 -- 172.21.15.40:0/3127966892 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc0cf100 con 0x7fcebc13ec80 2024-03-24T00:38:21.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.093+0000 7fcec1600640 1 --2- 172.21.15.40:0/3127966892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc13ec80 0x7fcebc1414b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fcebc06b170 tx=0x7fceb0020890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66f51ed6b1a45f5b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:21.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.093+0000 7fcebbe00640 1 -- 172.21.15.40:0/3127966892 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb00b6070 con 0x7fcebc13ec80 2024-03-24T00:38:21.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.093+0000 7fcebbe00640 1 -- 172.21.15.40:0/3127966892 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb00a9670 con 0x7fcebc13ec80 2024-03-24T00:38:21.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.093+0000 7fcebbe00640 1 -- 172.21.15.40:0/3127966892 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb00a9a40 con 0x7fcebc13ec80 2024-03-24T00:38:21.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.093+0000 7fcec4000640 1 -- 172.21.15.40:0/3127966892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc13ec80 msgr2=0x7fcebc1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:21.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.093+0000 7fcec4000640 1 --2- 172.21.15.40:0/3127966892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc13ec80 0x7fcebc1414b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fcebc06b170 tx=0x7fceb0020890 comp rx=0 tx=0).stop 2024-03-24T00:38:21.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.093+0000 7fcec4000640 1 -- 172.21.15.40:0/3127966892 shutdown_connections 2024-03-24T00:38:21.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.093+0000 7fcec4000640 1 --2- 172.21.15.40:0/3127966892 >> v2:172.21.15.40:3300/0 conn(0x7fcebc1419f0 0x7fcebc0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.093+0000 7fcec4000640 1 --2- 172.21.15.40:0/3127966892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc13ec80 0x7fcebc1414b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.093+0000 7fcec4000640 1 -- 172.21.15.40:0/3127966892 >> 172.21.15.40:0/3127966892 conn(0x7fcebc0b17b0 msgr2=0x7fcebc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:21.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.093+0000 7fcec4000640 1 -- 172.21.15.40:0/3127966892 shutdown_connections 2024-03-24T00:38:21.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.093+0000 7fcec4000640 1 -- 172.21.15.40:0/3127966892 wait complete. 2024-03-24T00:38:21.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec4000640 1 Processor -- start 2024-03-24T00:38:21.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec4000640 1 -- start start 2024-03-24T00:38:21.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec4000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcebc0ce630 0x7fcebc13d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec4000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcebc13ec80 0x7fcebc14d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec4000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc1419f0 0x7fcebc151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec4000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc166eb0 con 0x7fcebc13ec80 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec4000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc166d30 con 0x7fcebc1419f0 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec4000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc167030 con 0x7fcebc0ce630 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcebc13ec80 0x7fcebc14d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcebc13ec80 0x7fcebc14d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45882/0 (socket says 172.21.15.40:45882) 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec0c00640 1 -- 172.21.15.40:0/1431474369 learned_addr learned my addr 172.21.15.40:0/1431474369 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec1600640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcebc0ce630 0x7fcebc13d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec2000640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc1419f0 0x7fcebc151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec0c00640 1 -- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcebc0ce630 msgr2=0x7fcebc13d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec0c00640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcebc0ce630 0x7fcebc13d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec0c00640 1 -- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc1419f0 msgr2=0x7fcebc151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:21.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec0c00640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc1419f0 0x7fcebc151370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec0c00640 1 -- 172.21.15.40:0/1431474369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc14d8d0 con 0x7fcebc13ec80 2024-03-24T00:38:21.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec0c00640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcebc13ec80 0x7fcebc14d390 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fcea800e7f0 tx=0x7fcea800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:21.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fceb9600640 1 -- 172.21.15.40:0/1431474369 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8017070 con 0x7fcebc13ec80 2024-03-24T00:38:21.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fceb9600640 1 -- 172.21.15.40:0/1431474369 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea800c940 con 0x7fcebc13ec80 2024-03-24T00:38:21.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec4000640 1 -- 172.21.15.40:0/1431474369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcebc156b10 con 0x7fcebc13ec80 2024-03-24T00:38:21.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fceb9600640 1 -- 172.21.15.40:0/1431474369 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea800cd30 con 0x7fcebc13ec80 2024-03-24T00:38:21.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.097+0000 7fcec4000640 1 -- 172.21.15.40:0/1431474369 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcebc1570d0 con 0x7fcebc13ec80 2024-03-24T00:38:21.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.101+0000 7fceb9600640 1 -- 172.21.15.40:0/1431474369 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcea801a070 con 0x7fcebc13ec80 2024-03-24T00:38:21.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.101+0000 7fceb9600640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce9c067660 0x7fce9c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.101+0000 7fceb9600640 1 -- 172.21.15.40:0/1431474369 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcea8085b10 con 0x7fcebc13ec80 2024-03-24T00:38:21.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.101+0000 7fcec1600640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce9c067660 0x7fce9c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.101+0000 7fcec73fb000 1 Processor -- start 2024-03-24T00:38:21.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.101+0000 7fcec73fb000 1 -- start start 2024-03-24T00:38:21.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.101+0000 7fcec1600640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce9c067660 0x7fce9c069b20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fceb00b3000 tx=0x7fceb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:21.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce3dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfce3e240 0x55abfcdf3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfcdf3ff0 0x55abfcdf43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abfcc41680 con 0x55abfce3d920 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abfcbc8b00 con 0x55abfce3e240 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55abfcc3d720 con 0x55abfcdf3ff0 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce3dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce3dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45886/0 (socket says 172.21.15.40:45886) 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec1600640 1 -- 172.21.15.40:0/1811669227 learned_addr learned my addr 172.21.15.40:0/1811669227 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec0c00640 1 --2- 172.21.15.40:0/1811669227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfce3e240 0x55abfcdf3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec2000640 1 --2- 172.21.15.40:0/1811669227 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfcdf3ff0 0x55abfcdf43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec1600640 1 -- 172.21.15.40:0/1811669227 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfcdf3ff0 msgr2=0x55abfcdf43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec1600640 1 --2- 172.21.15.40:0/1811669227 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfcdf3ff0 0x55abfcdf43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec1600640 1 -- 172.21.15.40:0/1811669227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfce3e240 msgr2=0x55abfcdf3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec1600640 1 --2- 172.21.15.40:0/1811669227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfce3e240 0x55abfcdf3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec1600640 1 -- 172.21.15.40:0/1811669227 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55abfcc19940 con 0x55abfce3d920 2024-03-24T00:38:21.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec1600640 1 --2- 172.21.15.40:0/1811669227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce3dd00 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fceb00b4040 tx=0x7fceb0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26bc35e081fa1f6d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:21.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fceae000640 1 -- 172.21.15.40:0/1811669227 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb00cd070 con 0x55abfce3d920 2024-03-24T00:38:21.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fceae000640 1 -- 172.21.15.40:0/1811669227 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb00a9e50 con 0x55abfce3d920 2024-03-24T00:38:21.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fceae000640 1 -- 172.21.15.40:0/1811669227 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb00b6040 con 0x55abfce3d920 2024-03-24T00:38:21.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 -- 172.21.15.40:0/1811669227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 msgr2=0x55abfce3dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:21.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1811669227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce3dd00 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7fceb00b4040 tx=0x7fceb0004740 comp rx=0 tx=0).stop 2024-03-24T00:38:21.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 -- 172.21.15.40:0/1811669227 shutdown_connections 2024-03-24T00:38:21.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1811669227 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfcdf3ff0 0x55abfcdf43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1811669227 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfce3e240 0x55abfcdf3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1811669227 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce3dd00 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 -- 172.21.15.40:0/1811669227 >> 172.21.15.40:0/1811669227 conn(0x55abfcdeffa0 msgr2=0x55abfcdf2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:21.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 -- 172.21.15.40:0/1811669227 shutdown_connections 2024-03-24T00:38:21.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 -- 172.21.15.40:0/1811669227 wait complete. 2024-03-24T00:38:21.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 -- start start 2024-03-24T00:38:21.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.105+0000 7fcec73fb000 1 Processor -- start 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfcdf3ff0 0x55abfce22d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce23280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfce3e240 0x55abfce237c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abfcc40990 con 0x55abfce3d920 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abfcbfbae0 con 0x55abfce3e240 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55abfcbd9750 con 0x55abfcdf3ff0 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 10 client.0 _ll_register_callbacks cb 0x55abfcddeff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec1600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfcdf3ff0 0x55abfce22d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce23280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec2000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfce3e240 0x55abfce237c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec1600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfcdf3ff0 0x55abfce22d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:49074/0 (socket says 172.21.15.40:49074) 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec1600640 1 -- 172.21.15.40:0/3053954905 learned_addr learned my addr 172.21.15.40:0/3053954905 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec0c00640 1 -- 172.21.15.40:0/3053954905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfcdf3ff0 msgr2=0x55abfce22d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec0c00640 1 --2- 172.21.15.40:0/3053954905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfcdf3ff0 0x55abfce22d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec0c00640 1 -- 172.21.15.40:0/3053954905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfce3e240 msgr2=0x55abfce237c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec0c00640 1 --2- 172.21.15.40:0/3053954905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfce3e240 0x55abfce237c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec0c00640 1 -- 172.21.15.40:0/3053954905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55abfcc86d80 con 0x55abfce3d920 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec1600640 1 --2- 172.21.15.40:0/3053954905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfcdf3ff0 0x55abfce22d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec0c00640 1 --2- 172.21.15.40:0/3053954905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce23280 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fcea8083f70 tx=0x7fcea8055ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdeff309b0bd2ee9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcea2a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcea2a00640 1 -- 172.21.15.40:0/3053954905 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8083060 con 0x55abfce3d920 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcea2a00640 1 -- 172.21.15.40:0/3053954905 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcea80556f0 con 0x55abfce3d920 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 -- 172.21.15.40:0/3053954905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fceb00a9e50 con 0x55abfce3d920 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcea2a00640 1 -- 172.21.15.40:0/3053954905 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcea8074050 con 0x55abfce3d920 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 10 client.8643 fetch_fsmap learned FSMap version 117 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 -- 172.21.15.40:0/3053954905 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x55abfcbc8d10 con 0x55abfce3d920 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcea2a00640 1 -- 172.21.15.40:0/3053954905 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb00a9e50 con 0x55abfce3d920 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcea2a00640 1 -- 172.21.15.40:0/3053954905 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 117) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fcea80741f0 con 0x55abfce3d920 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 10 client.8643 fetch_fsmap finished waiting for FSMap version 117 2024-03-24T00:38:21.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 10 client.8643 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:38:21.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 20 client.8643 populate_metadata read hostname 'smithi040' 2024-03-24T00:38:21.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 --2- 172.21.15.40:0/3053954905 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x55abfce24060 0x55abfce4fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 4 client.8643 mds_command: new command op to 8418 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:38:21.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 -- 172.21.15.40:0/3053954905 --> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55abfcbdcb60 con 0x55abfce24060 2024-03-24T00:38:21.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec1600640 1 --2- 172.21.15.40:0/3053954905 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x55abfce24060 0x55abfce4fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec1600640 1 --2- 172.21.15.40:0/3053954905 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x55abfce24060 0x55abfce4fea0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3eb05dfb5f13a0cd server_cookie=7f8fdae7fc9bca27 in_seq=0 out_seq=0 2024-03-24T00:38:21.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcea2a00640 10 client.8643 ms_handle_connect on v2:172.21.15.53:6835/8812937 2024-03-24T00:38:21.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcea2a00640 1 -- 172.21.15.40:0/3053954905 <== mds.0 v2:172.21.15.53:6835/8812937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55abfcbdcb60 con 0x55abfce24060 2024-03-24T00:38:21.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcea2a00640 10 client.8643 handle_command_reply: tid=0 2024-03-24T00:38:21.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 client.8643 shutdown 2024-03-24T00:38:21.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 -- 172.21.15.40:0/3053954905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 msgr2=0x55abfce23280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:21.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.109+0000 7fcec73fb000 1 --2- 172.21.15.40:0/3053954905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce23280 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fcea8083f70 tx=0x7fcea8055ac0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.113+0000 7fcec73fb000 1 -- 172.21.15.40:0/3053954905 shutdown_connections 2024-03-24T00:38:21.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.113+0000 7fcec73fb000 1 --2- 172.21.15.40:0/3053954905 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x55abfce24060 0x55abfce4fea0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.113+0000 7fcec73fb000 1 --2- 172.21.15.40:0/3053954905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfce3e240 0x55abfce237c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.113+0000 7fcec73fb000 1 --2- 172.21.15.40:0/3053954905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce23280 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.113+0000 7fcec73fb000 1 --2- 172.21.15.40:0/3053954905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfcdf3ff0 0x55abfce22d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.113+0000 7fcec73fb000 1 -- 172.21.15.40:0/3053954905 >> 172.21.15.40:0/3053954905 conn(0x55abfcdeffa0 msgr2=0x55abfcdf2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:21.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.113+0000 7fcea2a00640 0 client.8643 ms_handle_reset on v2:172.21.15.53:6835/8812937 2024-03-24T00:38:21.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.113+0000 7fcec73fb000 1 -- 172.21.15.40:0/3053954905 shutdown_connections 2024-03-24T00:38:21.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.113+0000 7fcec73fb000 1 -- 172.21.15.40:0/3053954905 wait complete. 2024-03-24T00:38:21.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.113+0000 7fcec73fb000 20 client.8643 trim_cache size 0 max 16384 2024-03-24T00:38:21.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 Processor -- start 2024-03-24T00:38:21.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 -- start start 2024-03-24T00:38:21.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfcdf3ff0 0x55abfce20ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce3c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfce3e240 0x55abfce3c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc1ff000 con 0x55abfce3d920 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc1fe4a0 con 0x55abfcdf3ff0 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc1fe9a0 con 0x55abfce3e240 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce3c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec1600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfcdf3ff0 0x55abfce20ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec0c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce3c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45894/0 (socket says 172.21.15.40:45894) 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec0c00640 1 -- 172.21.15.40:0/1395796831 learned_addr learned my addr 172.21.15.40:0/1395796831 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec2000640 1 --2- 172.21.15.40:0/1395796831 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfce3e240 0x55abfce3c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec0c00640 1 -- 172.21.15.40:0/1395796831 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfce3e240 msgr2=0x55abfce3c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec0c00640 1 --2- 172.21.15.40:0/1395796831 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfce3e240 0x55abfce3c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec0c00640 1 -- 172.21.15.40:0/1395796831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfcdf3ff0 msgr2=0x55abfce20ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:21.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec0c00640 1 --2- 172.21.15.40:0/1395796831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfcdf3ff0 0x55abfce20ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec0c00640 1 -- 172.21.15.40:0/1395796831 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55abfcc0e540 con 0x55abfce3d920 2024-03-24T00:38:21.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec1600640 1 --2- 172.21.15.40:0/1395796831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfcdf3ff0 0x55abfce20ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:21.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec0c00640 1 --2- 172.21.15.40:0/1395796831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce3c120 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fcea8000c10 tx=0x7fcea806c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5a77fb7d8ecd1f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcead600640 1 -- 172.21.15.40:0/1395796831 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8021040 con 0x55abfce3d920 2024-03-24T00:38:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcead600640 1 -- 172.21.15.40:0/1395796831 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea80741f0 con 0x55abfce3d920 2024-03-24T00:38:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcead600640 1 -- 172.21.15.40:0/1395796831 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8055710 con 0x55abfce3d920 2024-03-24T00:38:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 -- 172.21.15.40:0/1395796831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 msgr2=0x55abfce3c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1395796831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce3c120 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fcea8000c10 tx=0x7fcea806c000 comp rx=0 tx=0).stop 2024-03-24T00:38:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 -- 172.21.15.40:0/1395796831 shutdown_connections 2024-03-24T00:38:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1395796831 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfce3e240 0x55abfce3c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1395796831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3d920 0x55abfce3c120 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1395796831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfcdf3ff0 0x55abfce20ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 -- 172.21.15.40:0/1395796831 >> 172.21.15.40:0/1395796831 conn(0x55abfcdeffa0 msgr2=0x55abfce205d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 -- 172.21.15.40:0/1395796831 shutdown_connections 2024-03-24T00:38:21.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.129+0000 7fcec73fb000 1 -- 172.21.15.40:0/1395796831 wait complete. 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 1 -- start start 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 1 Processor -- start 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfcdf3ff0 0x55abfcdf3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfce3d920 0x55abfce50ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3e240 0x55abfce51430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abfcbe9a80 con 0x55abfce3e240 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcebc146770 con 0x55abfcdf3ff0 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55abfccfe700 con 0x55abfce3d920 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 10 client.0 _ll_register_callbacks cb 0x7fcea80046d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3e240 0x55abfce51430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec0c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfce3d920 0x55abfce50ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec1600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfcdf3ff0 0x55abfcdf3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3e240 0x55abfce51430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:45902/0 (socket says 172.21.15.40:45902) 2024-03-24T00:38:21.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec2000640 1 -- 172.21.15.40:0/1445104908 learned_addr learned my addr 172.21.15.40:0/1445104908 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:21.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec2000640 1 -- 172.21.15.40:0/1445104908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfce3d920 msgr2=0x55abfce50ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:21.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec2000640 1 --2- 172.21.15.40:0/1445104908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfce3d920 0x55abfce50ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec2000640 1 -- 172.21.15.40:0/1445104908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfcdf3ff0 msgr2=0x55abfcdf3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:21.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec2000640 1 --2- 172.21.15.40:0/1445104908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfcdf3ff0 0x55abfcdf3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec2000640 1 -- 172.21.15.40:0/1445104908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55abfcbd54b0 con 0x55abfce3e240 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec2000640 1 --2- 172.21.15.40:0/1445104908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3e240 0x55abfce51430 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fceb4016af0 tx=0x7fceb40153b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eafd1b537228e09 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcea2a00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcea2a00640 1 -- 172.21.15.40:0/1445104908 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4006d50 con 0x55abfce3e240 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcea2a00640 1 -- 172.21.15.40:0/1445104908 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fceb4015b60 con 0x55abfce3e240 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 1 -- 172.21.15.40:0/1445104908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcea80741f0 con 0x55abfce3e240 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcea2a00640 1 -- 172.21.15.40:0/1445104908 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=117) v2 ==== 24+0+0 (secure 0 0 0) 0x7fceb4015d60 con 0x55abfce3e240 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 10 client.8649 fetch_fsmap learned FSMap version 117 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 1 -- 172.21.15.40:0/1445104908 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=117,monmap=2+}) v3 -- 0x55abfcbe82f0 con 0x55abfce3e240 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcea2a00640 1 -- 172.21.15.40:0/1445104908 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea80741f0 con 0x55abfce3e240 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcea2a00640 1 -- 172.21.15.40:0/1445104908 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 117) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fceb4029070 con 0x55abfce3e240 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 10 client.8649 fetch_fsmap finished waiting for FSMap version 117 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 10 client.8649 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:38:21.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 20 client.8649 populate_metadata read hostname 'smithi040' 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1445104908 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x55abfcea21a0 0x55abfcea2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 4 client.8649 mds_command: new command op to 8418 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec73fb000 1 -- 172.21.15.40:0/1445104908 --> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55abfcc4ebc0 con 0x55abfcea21a0 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.133+0000 7fcec1600640 1 --2- 172.21.15.40:0/1445104908 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x55abfcea21a0 0x55abfcea2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec1600640 1 --2- 172.21.15.40:0/1445104908 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x55abfcea21a0 0x55abfcea2580 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0c34eeb89db1bac server_cookie=d60a8edcc62bceed in_seq=0 out_seq=0 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcea2a00640 10 client.8649 ms_handle_connect on v2:172.21.15.53:6835/8812937 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcea2a00640 1 -- 172.21.15.40:0/1445104908 <== mds.0 v2:172.21.15.53:6835/8812937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x55abfcc4ebc0 con 0x55abfcea21a0 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcea2a00640 10 client.8649 handle_command_reply: tid=0 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec73fb000 1 client.8649 shutdown 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec73fb000 1 -- 172.21.15.40:0/1445104908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3e240 msgr2=0x55abfce51430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1445104908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3e240 0x55abfce51430 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7fceb4016af0 tx=0x7fceb40153b0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec73fb000 1 -- 172.21.15.40:0/1445104908 shutdown_connections 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1445104908 >> [v2:172.21.15.53:6835/8812937,v1:172.21.15.53:6837/8812937] conn(0x55abfcea21a0 0x55abfcea2580 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1445104908 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55abfce3e240 0x55abfce51430 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1445104908 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55abfce3d920 0x55abfce50ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcea2a00640 0 client.8649 ms_handle_reset on v2:172.21.15.53:6835/8812937 2024-03-24T00:38:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec73fb000 1 --2- 172.21.15.40:0/1445104908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55abfcdf3ff0 0x55abfcdf3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec73fb000 1 -- 172.21.15.40:0/1445104908 >> 172.21.15.40:0/1445104908 conn(0x55abfcdeffa0 msgr2=0x55abfce22190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec73fb000 1 -- 172.21.15.40:0/1445104908 shutdown_connections 2024-03-24T00:38:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec73fb000 1 -- 172.21.15.40:0/1445104908 wait complete. 2024-03-24T00:38:21.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec73fb000 20 client.8649 trim_cache size 0 max 16384 2024-03-24T00:38:21.144 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:38:21.144 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:38:21.144 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:38:21.144 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8418, 2024-03-24T00:38:21.144 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:38:21.145 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:38:21.145 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:38:21.145 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 19.100118774999999, 2024-03-24T00:38:21.145 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 117, 2024-03-24T00:38:21.145 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 106, 2024-03-24T00:38:21.145 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 106, 2024-03-24T00:38:21.145 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 23.937446684000001 2024-03-24T00:38:21.145 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:38:21.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec4000640 1 -- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce9c067660 msgr2=0x7fce9c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:21.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec4000640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce9c067660 0x7fce9c069b20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fceb00b3000 tx=0x7fceb0008040 comp rx=0 tx=0).stop 2024-03-24T00:38:21.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec4000640 1 -- 172.21.15.40:0/1431474369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcebc13ec80 msgr2=0x7fcebc14d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:21.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec4000640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcebc13ec80 0x7fcebc14d390 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7fcea800e7f0 tx=0x7fcea800ecb0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec4000640 1 -- 172.21.15.40:0/1431474369 shutdown_connections 2024-03-24T00:38:21.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec4000640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fce9c067660 0x7fce9c069b20 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec4000640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcebc1419f0 0x7fcebc151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec4000640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcebc13ec80 0x7fcebc14d390 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec4000640 1 --2- 172.21.15.40:0/1431474369 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcebc0ce630 0x7fcebc13d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:21.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec4000640 1 -- 172.21.15.40:0/1431474369 >> 172.21.15.40:0/1431474369 conn(0x7fcebc0b17b0 msgr2=0x7fcebc156060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:21.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec4000640 1 -- 172.21.15.40:0/1431474369 shutdown_connections 2024-03-24T00:38:21.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:21.137+0000 7fcec4000640 1 -- 172.21.15.40:0/1431474369 wait complete. 2024-03-24T00:38:21.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:21.160 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:38:21.160 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:38:21.160 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p tree 2024-03-24T00:38:21.160 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:38:21.160 DEBUG:teuthology.orchestra.run.smithi040:> setfattr -n ceph.dir.pin.distributed -v 1 tree 2024-03-24T00:38:21.160 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:38:21.160 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < 100; i++)); do 2024-03-24T00:38:21.160 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p "tree/$i" 2024-03-24T00:38:21.160 DEBUG:teuthology.orchestra.run.smithi040:> echo file > "tree/$i/file" 2024-03-24T00:38:21.160 DEBUG:teuthology.orchestra.run.smithi040:> done 2024-03-24T00:38:21.160 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:38:21.171 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree 2024-03-24T00:38:21.175 INFO:teuthology.orchestra.run.smithi040.stderr:+ setfattr -n ceph.dir.pin.distributed -v 1 tree 2024-03-24T00:38:21.179 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i = 0 )) 2024-03-24T00:38:21.179 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.179 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/0 2024-03-24T00:38:21.183 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.189 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.189 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.189 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/1 2024-03-24T00:38:21.207 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.211 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.211 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.211 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/2 2024-03-24T00:38:21.221 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.224 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.224 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.224 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/3 2024-03-24T00:38:21.231 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.234 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.234 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.234 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/4 2024-03-24T00:38:21.240 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.243 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/5 2024-03-24T00:38:21.250 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.253 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.253 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.253 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/6 2024-03-24T00:38:21.259 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.262 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/7 2024-03-24T00:38:21.268 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.271 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.271 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.271 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/8 2024-03-24T00:38:21.278 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.281 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/9 2024-03-24T00:38:21.287 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.290 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/10 2024-03-24T00:38:21.296 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.300 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/11 2024-03-24T00:38:21.305 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.308 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.308 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.308 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/12 2024-03-24T00:38:21.314 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.316 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.316 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.316 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/13 2024-03-24T00:38:21.322 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.325 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.325 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.325 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/14 2024-03-24T00:38:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.333 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.333 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.333 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/15 2024-03-24T00:38:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.341 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.341 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.341 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/16 2024-03-24T00:38:21.347 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.349 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.349 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.349 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/17 2024-03-24T00:38:21.354 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.357 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.357 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.357 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/18 2024-03-24T00:38:21.363 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.365 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/19 2024-03-24T00:38:21.371 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.373 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/20 2024-03-24T00:38:21.379 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.381 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.381 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.381 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/21 2024-03-24T00:38:21.387 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.390 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.390 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.390 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/22 2024-03-24T00:38:21.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.398 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.399 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/23 2024-03-24T00:38:21.405 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.407 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/24 2024-03-24T00:38:21.413 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.416 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.416 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.416 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/25 2024-03-24T00:38:21.422 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.425 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.425 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.425 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/26 2024-03-24T00:38:21.431 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.434 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.434 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.434 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/27 2024-03-24T00:38:21.440 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.442 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.442 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.442 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/28 2024-03-24T00:38:21.448 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.451 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.451 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.451 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/29 2024-03-24T00:38:21.457 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.460 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.460 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.460 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/30 2024-03-24T00:38:21.466 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.468 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.468 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.468 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/31 2024-03-24T00:38:21.475 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.477 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.477 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.477 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/32 2024-03-24T00:38:21.483 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.486 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/33 2024-03-24T00:38:21.491 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.494 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/34 2024-03-24T00:38:21.500 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.502 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.503 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.503 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/35 2024-03-24T00:38:21.509 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.511 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/36 2024-03-24T00:38:21.517 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.519 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/37 2024-03-24T00:38:21.525 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.527 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/38 2024-03-24T00:38:21.533 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.535 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.535 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.537 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/39 2024-03-24T00:38:21.541 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.543 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.543 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.544 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/40 2024-03-24T00:38:21.550 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.552 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.552 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.552 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/41 2024-03-24T00:38:21.558 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.560 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.560 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.560 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/42 2024-03-24T00:38:21.566 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.568 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/43 2024-03-24T00:38:21.574 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.576 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/44 2024-03-24T00:38:21.582 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.584 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.585 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/45 2024-03-24T00:38:21.590 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.592 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.592 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.592 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/46 2024-03-24T00:38:21.598 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.600 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.601 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.601 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/47 2024-03-24T00:38:21.607 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.609 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/48 2024-03-24T00:38:21.615 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.617 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.617 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.618 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/49 2024-03-24T00:38:21.623 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/50 2024-03-24T00:38:21.632 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.634 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.634 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.634 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/51 2024-03-24T00:38:21.640 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.642 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.642 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.642 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/52 2024-03-24T00:38:21.648 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.651 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.651 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.651 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/53 2024-03-24T00:38:21.656 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.659 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.659 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.659 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/54 2024-03-24T00:38:21.665 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.667 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.667 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.668 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/55 2024-03-24T00:38:21.673 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.675 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.676 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.676 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/56 2024-03-24T00:38:21.681 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.684 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.684 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.684 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/57 2024-03-24T00:38:21.690 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.692 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.692 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.693 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/58 2024-03-24T00:38:21.698 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.701 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.701 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.701 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/59 2024-03-24T00:38:21.707 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.710 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.710 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.710 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/60 2024-03-24T00:38:21.716 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.719 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.719 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.719 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/61 2024-03-24T00:38:21.725 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.728 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.728 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.728 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/62 2024-03-24T00:38:21.734 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.736 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.737 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.737 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/63 2024-03-24T00:38:21.743 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.745 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.745 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.745 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/64 2024-03-24T00:38:21.752 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.755 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.755 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.755 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/65 2024-03-24T00:38:21.761 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.764 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.764 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.764 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/66 2024-03-24T00:38:21.771 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.774 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.774 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.774 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/67 2024-03-24T00:38:21.781 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.784 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.784 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.784 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/68 2024-03-24T00:38:21.790 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.794 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.795 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.796 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/69 2024-03-24T00:38:21.800 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.803 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/70 2024-03-24T00:38:21.809 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.812 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/71 2024-03-24T00:38:21.819 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.822 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/72 2024-03-24T00:38:21.828 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.831 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/73 2024-03-24T00:38:21.837 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.840 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.840 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.840 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/74 2024-03-24T00:38:21.846 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.849 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.849 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.850 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/75 2024-03-24T00:38:21.856 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.859 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.859 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.859 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/76 2024-03-24T00:38:21.865 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.868 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.868 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.868 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/77 2024-03-24T00:38:21.875 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.878 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/78 2024-03-24T00:38:21.884 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.887 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/79 2024-03-24T00:38:21.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.895 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/80 2024-03-24T00:38:21.900 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.903 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/81 2024-03-24T00:38:21.909 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.911 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.911 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.911 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/82 2024-03-24T00:38:21.917 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.919 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.919 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.919 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/83 2024-03-24T00:38:21.925 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.928 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.928 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.928 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/84 2024-03-24T00:38:21.934 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.937 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.937 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.937 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/85 2024-03-24T00:38:21.945 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.947 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.947 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.947 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/86 2024-03-24T00:38:21.953 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.955 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.955 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.955 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/87 2024-03-24T00:38:21.961 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.963 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.963 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.963 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/88 2024-03-24T00:38:21.970 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.973 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.973 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.973 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/89 2024-03-24T00:38:21.980 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.983 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.983 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.983 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/90 2024-03-24T00:38:21.989 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:21.992 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:21.993 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/91 2024-03-24T00:38:22.000 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:22.003 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:22.003 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:22.003 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/92 2024-03-24T00:38:22.010 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:22.012 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/93 2024-03-24T00:38:22.018 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:22.020 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/94 2024-03-24T00:38:22.027 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:22.029 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/95 2024-03-24T00:38:22.036 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:22.039 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:22.039 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:22.039 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/96 2024-03-24T00:38:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/97 2024-03-24T00:38:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/98 2024-03-24T00:38:22.063 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:22.066 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/99 2024-03-24T00:38:22.072 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:38:22.074 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:38:22.075 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 100 )) 2024-03-24T00:38:22.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:38:22.243 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:38:22.244 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:38:22.244 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-24T00:38:22.404 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:38:22.404 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:38:22.405 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:38:22.564 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"3506","version":"3506","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":0,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1104","version":"1104","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}}] 2024-03-24T00:38:22.565 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 3, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 0, "subtree": 1 }, "projected_version": "3506", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3506" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "1104", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "1104" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:38:22.565 INFO:tasks.cephfs.cephfs_test_case:len=0 [] 2024-03-24T00:38:27.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:38:27.726 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:38:27.726 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:38:27.726 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-24T00:38:27.885 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:38:27.886 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:38:27.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:38:28.064 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"3511","version":"3511","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":2,"2":3}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1104","version":"1104","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x1000000041a.101*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x1000000041a.100*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x1000000041a.110*","snapid_first":2,"projected_version":"442","version":"442","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.000*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x1000000041a.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x1000000041a.010*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-03-24T00:38:28.065 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 2, "2": 3 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "3511", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3511" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "1104", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "1104" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.001*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "428", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x1000000041a.101*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "422", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x1000000041a.100*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "430", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x1000000041a.110*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "442", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "442" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.000*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x1000000041a.111*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "416", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x1000000041a.010*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-24T00:38:28.066 INFO:tasks.cephfs.cephfs_test_case:len=3 [{'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x1000000041a.001*', 'snapid_first': 2, 'projected_version': '428', 'version': '428', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x1000000041a.011*', 'snapid_first': 2, 'projected_version': '430', 'version': '430', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x1000000041a.000*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}] 2024-03-24T00:38:33.067 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json get subtrees 2024-03-24T00:38:33.230 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"3511","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":3},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x1000000041a.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x1000000041a.100*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x1000000041a.010*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds2","dirfrag":"0x102","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"2","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:38:33.230 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 3 }, "snapid_first": 2, "states": [], "version": "3511" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x1000000041a.111*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "416", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x1000000041a.100*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x1000000041a.010*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "2", "dirfrag": "0x102", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds2", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:38:33.231 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json get subtrees 2024-03-24T00:38:33.390 INFO:teuthology.orchestra.run.smithi040.stdout:[{"is_auth":false,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"0","version":"3511","committing_version":"0","committed_version":"0","is_rep":true,"dir_auth":"0","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":2},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1},"nref":2}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x1000000041a.101*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x1000000041a.110*","snapid_first":2,"projected_version":"442","version":"442","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"0":1}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1},"nref":4}},{"is_auth":true,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds1","dirfrag":"0x101","snapid_first":2,"projected_version":"1","version":"1","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"1","states":["auth","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":0,"authpin":0},"nref":2}}] 2024-03-24T00:38:33.391 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dir_auth": "0", "dirfrag": "0x1", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 2, "path": "", "pins": { "child": 1, "subtree": 1 }, "projected_version": "0", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 2 }, "snapid_first": 2, "states": [], "version": "3511" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x1000000041a.101*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "422", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "0": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x1000000041a.110*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "subtree": 1 }, "projected_version": "442", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "442" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "1", "dirfrag": "0x101", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "~mds1", "pins": { "authpin": 0, "child": 1, "dirty": 0, "subtree": 1 }, "projected_version": "1", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "complete" ], "version": "1" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:38:33.391 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:38:33.551 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"3511","version":"3511","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":2,"2":3}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1104","version":"1104","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x1000000041a.101*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x1000000041a.100*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":1,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":1,"dir":{"path":"/tree","dirfrag":"0x1000000041a.110*","snapid_first":2,"projected_version":"442","version":"442","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"1","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[1,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.000*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x1000000041a.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}},{"is_auth":false,"auth_first":2,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":true,"random_ephemeral_pin":false,"export_pin_target":2,"dir":{"path":"/tree","dirfrag":"0x1000000041a.010*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"2","states":[],"is_auth":false,"auth_state":{"replicas":{}},"replica_state":{"authority":[2,-2],"replica_nonce":1},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":0,"frozen":0,"subtree":1,"replicated":0,"dirty":0,"waiter":0,"authpin":0,"tempexporting":0},"nref":1}}] 2024-03-24T00:38:33.552 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 2, "2": 3 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "3511", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3511" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "1104", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "1104" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.001*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "428", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "428" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x1000000041a.101*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "422", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "422" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x1000000041a.100*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "430", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "430" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 1, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "1", "dirfrag": "0x1000000041a.110*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "442", "replica_state": { "authority": [ 1, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "442" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 1, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.000*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x1000000041a.111*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "416", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "416" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false }, { "auth_first": 2, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "2", "dirfrag": "0x1000000041a.010*", "is_auth": false, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 1, "path": "/tree", "pins": { "authpin": 0, "child": 0, "dirty": 0, "frozen": 0, "ptrwaiter": 0, "replicated": 0, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 2, -2 ], "replica_nonce": 1 }, "snapid_first": 2, "states": [], "version": "426" }, "distributed_ephemeral_pin": true, "export_pin": -1, "export_pin_target": 2, "is_auth": false, "random_ephemeral_pin": false } ] 2024-03-24T00:38:33.552 INFO:tasks.cephfs.cephfs_test_case:len=8 [{'is_auth': True, 'auth_first': 2, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 2, 'dir': {'path': '/tree', 'dirfrag': '0x1000000041a.111*', 'snapid_first': 2, 'projected_version': '416', 'version': '416', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '2', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [2, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 2, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 2, 'dir': {'path': '/tree', 'dirfrag': '0x1000000041a.100*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '2', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [2, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 2, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 2, 'dir': {'path': '/tree', 'dirfrag': '0x1000000041a.010*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '2', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [2, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x1000000041a.101*', 'snapid_first': 2, 'projected_version': '422', 'version': '422', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 1, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 1, 'dir': {'path': '/tree', 'dirfrag': '0x1000000041a.110*', 'snapid_first': 2, 'projected_version': '442', 'version': '442', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '1', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {'0': 1}}, 'replica_state': {'authority': [1, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'ptrwaiter': 0, 'child': 1, 'frozen': 0, 'subtree': 1, 'importing': 0, 'replicated': 1, 'dirty': 1}, 'nref': 4}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x1000000041a.001*', 'snapid_first': 2, 'projected_version': '428', 'version': '428', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x1000000041a.011*', 'snapid_first': 2, 'projected_version': '430', 'version': '430', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}, {'is_auth': True, 'auth_first': 0, 'auth_second': -2, 'export_pin': -1, 'distributed_ephemeral_pin': True, 'random_ephemeral_pin': False, 'export_pin_target': 0, 'dir': {'path': '/tree', 'dirfrag': '0x1000000041a.000*', 'snapid_first': 2, 'projected_version': '426', 'version': '426', 'committing_version': '401', 'committed_version': '401', 'is_rep': False, 'dir_auth': '0', 'states': ['auth', 'dirty', 'complete'], 'is_auth': True, 'auth_state': {'replicas': {}}, 'replica_state': {'authority': [0, -2], 'replica_nonce': 0}, 'auth_pins': 0, 'is_frozen': False, 'is_freezing': False, 'pins': {'child': 1, 'frozen': 0, 'subtree': 1, 'dirty': 1, 'authpin': 0}, 'nref': 3}}] 2024-03-24T00:38:33.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:33.552 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.pin -v 0 tree) 2024-03-24T00:38:37.945 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:38:37.949 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:38:48.566 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json get subtrees 2024-03-24T00:38:48.727 INFO:teuthology.orchestra.run.smithi053.stdout:[{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"","dirfrag":"0x1","snapid_first":2,"projected_version":"3514","version":"3514","committing_version":"1","committed_version":"1","is_rep":true,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":2,"2":3}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"replicated":1,"dirty":1,"authpin":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":-1,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":-1,"dir":{"path":"~mds0","dirfrag":"0x100","snapid_first":2,"projected_version":"1104","version":"1104","committing_version":"1","committed_version":"1","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.001*","snapid_first":2,"projected_version":"428","version":"428","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.101*","snapid_first":2,"projected_version":"422","version":"422","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"waiter":0,"authpin":0,"tempexporting":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.100*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"2":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"waiter":0,"authpin":0,"tempexporting":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.011*","snapid_first":2,"projected_version":"430","version":"430","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.110*","snapid_first":2,"projected_version":"442","version":"442","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"1":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"waiter":0,"authpin":0,"tempexporting":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.000*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"frozen":0,"subtree":1,"dirty":1,"authpin":0},"nref":3}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.111*","snapid_first":2,"projected_version":"416","version":"416","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"2":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"waiter":0,"authpin":0,"tempexporting":0},"nref":4}},{"is_auth":true,"auth_first":0,"auth_second":-2,"export_pin":0,"distributed_ephemeral_pin":false,"random_ephemeral_pin":false,"export_pin_target":0,"dir":{"path":"/tree","dirfrag":"0x1000000041a.010*","snapid_first":2,"projected_version":"426","version":"426","committing_version":"401","committed_version":"401","is_rep":false,"dir_auth":"0","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{"2":1}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"ptrwaiter":0,"request":0,"child":1,"frozen":0,"subtree":1,"importing":0,"replicated":1,"dirty":1,"waiter":0,"authpin":0,"tempexporting":0},"nref":4}}] 2024-03-24T00:38:48.729 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 2, "2": 3 } }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x1", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": true, "nref": 4, "path": "", "pins": { "authpin": 0, "child": 1, "dirty": 1, "replicated": 1, "subtree": 1 }, "projected_version": "3514", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "3514" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "1", "committing_version": "1", "dir_auth": "0", "dirfrag": "0x100", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "~mds0", "pins": { "authpin": 0, "child": 1, "dirty": 1, "subtree": 1 }, "projected_version": "1104", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "1104" }, "distributed_ephemeral_pin": false, "export_pin": -1, "export_pin_target": -1, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.001*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "428", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "428" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.101*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "422", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "422" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "2": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.100*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.011*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "430", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "430" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "1": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.110*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "442", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "442" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.000*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 3, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "subtree": 1 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "2": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.111*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "416", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "416" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false }, { "auth_first": 0, "auth_second": -2, "dir": { "auth_pins": 0, "auth_state": { "replicas": { "2": 1 } }, "committed_version": "401", "committing_version": "401", "dir_auth": "0", "dirfrag": "0x1000000041a.010*", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 4, "path": "/tree", "pins": { "authpin": 0, "child": 1, "dirty": 1, "frozen": 0, "importing": 0, "ptrwaiter": 0, "replicated": 1, "request": 0, "subtree": 1, "tempexporting": 0, "waiter": 0 }, "projected_version": "426", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "426" }, "distributed_ephemeral_pin": false, "export_pin": 0, "export_pin_target": 0, "is_auth": true, "random_ephemeral_pin": false } ] 2024-03-24T00:38:48.729 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:38:48.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:38:48.782 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:38:48.854 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:38:48.854 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:38:48.854 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:38:48.863 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:49.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdbe00640 1 Processor -- start 2024-03-24T00:38:49.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdbe00640 1 -- start start 2024-03-24T00:38:49.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6bdc145520 0x7f6bdc15f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdbe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6bdc071b00 0x7f6bdc06d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdbe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bdc0cec10 con 0x7f6bdc15fe60 2024-03-24T00:38:49.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdbe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bdc1602d0 con 0x7f6bdc071b00 2024-03-24T00:38:49.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdbe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bdc0ceeb0 con 0x7f6bdc145520 2024-03-24T00:38:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6bdc071b00 0x7f6bdc06d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdaa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6bdc145520 0x7f6bdc15f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bda000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6bdc15fe60 0x7f6bdc06cdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:60260/0 (socket says 172.21.15.40:60260) 2024-03-24T00:38:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6bdc071b00 0x7f6bdc06d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42336/0 (socket says 172.21.15.40:42336) 2024-03-24T00:38:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bda000640 1 -- 172.21.15.40:0/3142252582 learned_addr learned my addr 172.21.15.40:0/3142252582 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdb400640 1 -- 172.21.15.40:0/3142252582 >> v2:172.21.15.53:3300/0 conn(0x7f6bdc145520 msgr2=0x7f6bdc15f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:49.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdb400640 1 --2- 172.21.15.40:0/3142252582 >> v2:172.21.15.53:3300/0 conn(0x7f6bdc145520 0x7f6bdc15f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bd9600640 1 -- 172.21.15.40:0/3142252582 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610090703 0 0) 0x7f6bdc0cec10 con 0x7f6bdc15fe60 2024-03-24T00:38:49.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdb400640 1 -- 172.21.15.40:0/3142252582 >> v1:172.21.15.53:6789/0 conn(0x7f6bdc15fe60 legacy=0x7f6bdc06cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.045+0000 7f6bdb400640 1 -- 172.21.15.40:0/3142252582 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc0cf190 con 0x7f6bdc071b00 2024-03-24T00:38:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.049+0000 7f6bdb400640 1 --2- 172.21.15.40:0/3142252582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc071b00 0x7f6bdc06d500 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0b2800 tx=0x7f6bd401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94fd6d20c01d9e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.049+0000 7f6bd9600640 1 -- 172.21.15.40:0/3142252582 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40b2070 con 0x7f6bdc071b00 2024-03-24T00:38:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.049+0000 7f6bd9600640 1 -- 172.21.15.40:0/3142252582 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd40a5950 con 0x7f6bdc071b00 2024-03-24T00:38:49.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.049+0000 7f6bd9600640 1 -- 172.21.15.40:0/3142252582 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40a5d20 con 0x7f6bdc071b00 2024-03-24T00:38:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.049+0000 7f6bdbe00640 1 -- 172.21.15.40:0/3142252582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc071b00 msgr2=0x7f6bdc06d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.049+0000 7f6bdbe00640 1 --2- 172.21.15.40:0/3142252582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc071b00 0x7f6bdc06d500 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0b2800 tx=0x7f6bd401c870 comp rx=0 tx=0).stop 2024-03-24T00:38:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.049+0000 7f6bdbe00640 1 -- 172.21.15.40:0/3142252582 shutdown_connections 2024-03-24T00:38:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.049+0000 7f6bdbe00640 1 --2- 172.21.15.40:0/3142252582 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc071b00 0x7f6bdc06d500 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.049+0000 7f6bdbe00640 1 --2- 172.21.15.40:0/3142252582 >> v2:172.21.15.53:3300/0 conn(0x7f6bdc145520 0x7f6bdc15f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.049+0000 7f6bdbe00640 1 -- 172.21.15.40:0/3142252582 >> 172.21.15.40:0/3142252582 conn(0x7f6bdc0b1740 msgr2=0x7f6bdc0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:49.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.049+0000 7f6bdbe00640 1 -- 172.21.15.40:0/3142252582 shutdown_connections 2024-03-24T00:38:49.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.049+0000 7f6bdbe00640 1 -- 172.21.15.40:0/3142252582 wait complete. 2024-03-24T00:38:49.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdbe00640 1 Processor -- start 2024-03-24T00:38:49.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdbe00640 1 -- start start 2024-03-24T00:38:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc071b00 0x7f6bdc15c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bdc145520 0x7f6bdc1398d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdbe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6bdc15fe60 0x7f6bdc154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdbe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc1605a0 con 0x7f6bdc071b00 2024-03-24T00:38:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdbe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc145b10 con 0x7f6bdc145520 2024-03-24T00:38:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdbe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc160720 con 0x7f6bdc15fe60 2024-03-24T00:38:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc071b00 0x7f6bdc15c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc071b00 0x7f6bdc15c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42340/0 (socket says 172.21.15.40:42340) 2024-03-24T00:38:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdb400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6bdc15fe60 0x7f6bdc154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bda000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bdc145520 0x7f6bdc1398d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdaa00640 1 -- 172.21.15.40:0/2328370164 learned_addr learned my addr 172.21.15.40:0/2328370164 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdb400640 1 -- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bdc145520 msgr2=0x7f6bdc1398d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdb400640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bdc145520 0x7f6bdc1398d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdb400640 1 -- 172.21.15.40:0/2328370164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc071b00 msgr2=0x7f6bdc15c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdb400640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc071b00 0x7f6bdc15c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdb400640 1 -- 172.21.15.40:0/2328370164 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc1575d0 con 0x7f6bdc15fe60 2024-03-24T00:38:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdaa00640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc071b00 0x7f6bdc15c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:49.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdb400640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6bdc15fe60 0x7f6bdc154e20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40ac000 tx=0x7f6bdc06e250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bd2a00640 1 -- 172.21.15.40:0/2328370164 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40bc030 con 0x7f6bdc15fe60 2024-03-24T00:38:49.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdbe00640 1 -- 172.21.15.40:0/2328370164 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc144760 con 0x7f6bdc15fe60 2024-03-24T00:38:49.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bd2a00640 1 -- 172.21.15.40:0/2328370164 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd40ad440 con 0x7f6bdc15fe60 2024-03-24T00:38:49.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bd2a00640 1 -- 172.21.15.40:0/2328370164 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd40b2040 con 0x7f6bdc15fe60 2024-03-24T00:38:49.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.053+0000 7f6bdbe00640 1 -- 172.21.15.40:0/2328370164 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc139e10 con 0x7f6bdc15fe60 2024-03-24T00:38:49.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.057+0000 7f6bdbe00640 1 -- 172.21.15.40:0/2328370164 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bdc073540 con 0x7f6bdc15fe60 2024-03-24T00:38:49.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.057+0000 7f6bd2a00640 1 -- 172.21.15.40:0/2328370164 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6bd40bf070 con 0x7f6bdc15fe60 2024-03-24T00:38:49.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.057+0000 7f6bd2a00640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6bbc067690 0x7f6bbc069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.057+0000 7f6bd2a00640 1 -- 172.21.15.40:0/2328370164 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6bd4120170 con 0x7f6bdc15fe60 2024-03-24T00:38:49.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.057+0000 7f6bdaa00640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6bbc067690 0x7f6bbc069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.057+0000 7f6bdaa00640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6bbc067690 0x7f6bbc069b50 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc011310 tx=0x7f6bcc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.065+0000 7f6bd2a00640 1 -- 172.21.15.40:0/2328370164 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6bd4016520 con 0x7f6bdc15fe60 2024-03-24T00:38:49.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.229+0000 7f6bdbe00640 1 -- 172.21.15.40:0/2328370164 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bdc04f4f0 con 0x7f6bdc15fe60 2024-03-24T00:38:49.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bd2a00640 1 -- 172.21.15.40:0/2328370164 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16567 (secure 0 0 0) 0x7f6bd40f0710 con 0x7f6bdc15fe60 2024-03-24T00:38:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:49.236 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":106,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:38:03.036173+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:37:59.849300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:00.417342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bdbe00640 1 -- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6bbc067690 msgr2=0x7f6bbc069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bdbe00640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6bbc067690 0x7f6bbc069b50 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6bcc011310 tx=0x7f6bcc009190 comp rx=0 tx=0).stop 2024-03-24T00:38:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bdbe00640 1 -- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6bdc15fe60 msgr2=0x7f6bdc154e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bdbe00640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6bdc15fe60 0x7f6bdc154e20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40ac000 tx=0x7f6bdc06e250 comp rx=0 tx=0).stop 2024-03-24T00:38:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bdbe00640 1 -- 172.21.15.40:0/2328370164 shutdown_connections 2024-03-24T00:38:49.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bdbe00640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6bbc067690 0x7f6bbc069b50 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bdbe00640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6bdc15fe60 0x7f6bdc154e20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bdbe00640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6bdc145520 0x7f6bdc1398d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bdbe00640 1 --2- 172.21.15.40:0/2328370164 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6bdc071b00 0x7f6bdc15c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bdbe00640 1 -- 172.21.15.40:0/2328370164 >> 172.21.15.40:0/2328370164 conn(0x7f6bdc0b1740 msgr2=0x7f6bdc0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bdbe00640 1 -- 172.21.15.40:0/2328370164 shutdown_connections 2024-03-24T00:38:49.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.233+0000 7f6bdbe00640 1 -- 172.21.15.40:0/2328370164 wait complete. 2024-03-24T00:38:49.258 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:37:59.849300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:00.417342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:38:49.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:38:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5e5e00640 1 Processor -- start 2024-03-24T00:38:49.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5e5e00640 1 -- start start 2024-03-24T00:38:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5e5e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e015e170 0x7fa5e006f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5e5e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa5e00d0a10 0x7fa5e015f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5e5e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5e00cfcb0 con 0x7fa5e00d0e30 2024-03-24T00:38:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5e5e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5e0070100 con 0x7fa5e00d0a10 2024-03-24T00:38:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5e5e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e006fe20 con 0x7fa5e015e170 2024-03-24T00:38:49.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5dea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e015e170 0x7fa5e006f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5d7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa5e00d0a10 0x7fa5e015f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5d7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa5e00d0a10 0x7fa5e015f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:50670/0 (socket says 172.21.15.40:50670) 2024-03-24T00:38:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5d7e00640 1 -- 172.21.15.40:0/3638721291 learned_addr learned my addr 172.21.15.40:0/3638721291 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5dea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e015e170 0x7fa5e006f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:56154/0 (socket says 172.21.15.40:56154) 2024-03-24T00:38:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5df400640 1 --1- 172.21.15.40:0/3638721291 >> v1:172.21.15.53:6789/0 conn(0x7fa5e00d0e30 0x7fa5e0161b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:60262/0 (socket says 172.21.15.40:60262) 2024-03-24T00:38:49.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5dea00640 1 -- 172.21.15.40:0/3638721291 >> v2:172.21.15.53:3300/0 conn(0x7fa5e00d0a10 msgr2=0x7fa5e015f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5dea00640 1 --2- 172.21.15.40:0/3638721291 >> v2:172.21.15.53:3300/0 conn(0x7fa5e00d0a10 0x7fa5e015f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5dea00640 1 -- 172.21.15.40:0/3638721291 >> v1:172.21.15.53:6789/0 conn(0x7fa5e00d0e30 legacy=0x7fa5e0161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5dea00640 1 -- 172.21.15.40:0/3638721291 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e00cff30 con 0x7fa5e015e170 2024-03-24T00:38:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5d7e00640 1 --2- 172.21.15.40:0/3638721291 >> v2:172.21.15.53:3300/0 conn(0x7fa5e00d0a10 0x7fa5e015f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:49.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.401+0000 7fa5de000640 1 -- 172.21.15.40:0/3638721291 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919429494 0 0) 0x7fa5d80014a0 con 0x7fa5e00d0e30 2024-03-24T00:38:49.408 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.405+0000 7fa5dea00640 1 --2- 172.21.15.40:0/3638721291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e015e170 0x7fa5e006f7b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00097c0 tx=0x7fa5d001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa8d33fc8e6df89f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.405+0000 7fa5de000640 1 -- 172.21.15.40:0/3638721291 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d00aa600 con 0x7fa5e015e170 2024-03-24T00:38:49.410 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.405+0000 7fa5de000640 1 -- 172.21.15.40:0/3638721291 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d00af070 con 0x7fa5e015e170 2024-03-24T00:38:49.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.405+0000 7fa5de000640 1 -- 172.21.15.40:0/3638721291 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d00aa9c0 con 0x7fa5e015e170 2024-03-24T00:38:49.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.405+0000 7fa5e5e00640 1 -- 172.21.15.40:0/3638721291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e015e170 msgr2=0x7fa5e006f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.411 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.405+0000 7fa5e5e00640 1 --2- 172.21.15.40:0/3638721291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e015e170 0x7fa5e006f7b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00097c0 tx=0x7fa5d001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.405+0000 7fa5e5e00640 1 -- 172.21.15.40:0/3638721291 shutdown_connections 2024-03-24T00:38:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.405+0000 7fa5e5e00640 1 --2- 172.21.15.40:0/3638721291 >> v2:172.21.15.53:3300/0 conn(0x7fa5e00d0a10 0x7fa5e015f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.405+0000 7fa5e5e00640 1 --2- 172.21.15.40:0/3638721291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e015e170 0x7fa5e006f7b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.405+0000 7fa5e5e00640 1 -- 172.21.15.40:0/3638721291 >> 172.21.15.40:0/3638721291 conn(0x7fa5e00b36a0 msgr2=0x7fa5e00b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:49.412 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.405+0000 7fa5e5e00640 1 -- 172.21.15.40:0/3638721291 shutdown_connections 2024-03-24T00:38:49.413 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.405+0000 7fa5e5e00640 1 -- 172.21.15.40:0/3638721291 wait complete. 2024-03-24T00:38:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5e5e00640 1 Processor -- start 2024-03-24T00:38:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5e5e00640 1 -- start start 2024-03-24T00:38:49.416 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5e5e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e00d0a10 0x7fa5e00c56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5e5e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e00d0e30 0x7fa5e00c5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5e5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e015e170 0x7fa5e00c7800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5e5e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e016aeb0 con 0x7fa5e00d0a10 2024-03-24T00:38:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5e5e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e01602c0 con 0x7fa5e015e170 2024-03-24T00:38:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5e5e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e016b030 con 0x7fa5e00d0e30 2024-03-24T00:38:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5dea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e00d0a10 0x7fa5e00c56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5dea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e00d0a10 0x7fa5e00c56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42348/0 (socket says 172.21.15.40:42348) 2024-03-24T00:38:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5dea00640 1 -- 172.21.15.40:0/4049097070 learned_addr learned my addr 172.21.15.40:0/4049097070 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5df400640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e015e170 0x7fa5e00c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5d7e00640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e00d0e30 0x7fa5e00c5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5dea00640 1 -- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e00d0e30 msgr2=0x7fa5e00c5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.417 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5dea00640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e00d0e30 0x7fa5e00c5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5dea00640 1 -- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e015e170 msgr2=0x7fa5e00c7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5dea00640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e015e170 0x7fa5e00c7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5dea00640 1 -- 172.21.15.40:0/4049097070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e00b27c0 con 0x7fa5e00d0a10 2024-03-24T00:38:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5dea00640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e00d0a10 0x7fa5e00c56c0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fa5d001a0d0 tx=0x7fa5d0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5d7400640 1 -- 172.21.15.40:0/4049097070 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0004220 con 0x7fa5e00d0a10 2024-03-24T00:38:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5d7400640 1 -- 172.21.15.40:0/4049097070 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d00043c0 con 0x7fa5e00d0a10 2024-03-24T00:38:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5e5e00640 1 -- 172.21.15.40:0/4049097070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e00b50a0 con 0x7fa5e00d0a10 2024-03-24T00:38:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5d7400640 1 -- 172.21.15.40:0/4049097070 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d00aa870 con 0x7fa5e00d0a10 2024-03-24T00:38:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5e5e00640 1 -- 172.21.15.40:0/4049097070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e00b5660 con 0x7fa5e00d0a10 2024-03-24T00:38:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.409+0000 7fa5e5e00640 1 -- 172.21.15.40:0/4049097070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e0068170 con 0x7fa5e00d0a10 2024-03-24T00:38:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.413+0000 7fa5d7400640 1 -- 172.21.15.40:0/4049097070 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa5d00a9070 con 0x7fa5e00d0a10 2024-03-24T00:38:49.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.413+0000 7fa5d7400640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5bc067660 0x7fa5bc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.413+0000 7fa5d7400640 1 -- 172.21.15.40:0/4049097070 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5d011f690 con 0x7fa5e00d0a10 2024-03-24T00:38:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.413+0000 7fa5d7e00640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5bc067660 0x7fa5bc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.413+0000 7fa5d7e00640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5bc067660 0x7fa5bc069b20 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0168e70 tx=0x7fa5c8009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.420 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.417+0000 7fa5d7400640 1 -- 172.21.15.40:0/4049097070 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa5d00b8070 con 0x7fa5e00d0a10 2024-03-24T00:38:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 -- 172.21.15.40:0/4049097070 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5e0072f30 con 0x7fa5e00d0a10 2024-03-24T00:38:49.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5d7400640 1 -- 172.21.15.40:0/4049097070 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5d0016090 con 0x7fa5e00d0a10 2024-03-24T00:38:49.584 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:38:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 -- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5bc067660 msgr2=0x7fa5bc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5bc067660 0x7fa5bc069b20 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0168e70 tx=0x7fa5c8009240 comp rx=0 tx=0).stop 2024-03-24T00:38:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 -- 172.21.15.40:0/4049097070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e00d0a10 msgr2=0x7fa5e00c56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e00d0a10 0x7fa5e00c56c0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7fa5d001a0d0 tx=0x7fa5d0019240 comp rx=0 tx=0).stop 2024-03-24T00:38:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 -- 172.21.15.40:0/4049097070 shutdown_connections 2024-03-24T00:38:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa5bc067660 0x7fa5bc069b20 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5e015e170 0x7fa5e00c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5e00d0e30 0x7fa5e00c5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 --2- 172.21.15.40:0/4049097070 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa5e00d0a10 0x7fa5e00c56c0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 -- 172.21.15.40:0/4049097070 >> 172.21.15.40:0/4049097070 conn(0x7fa5e00b36a0 msgr2=0x7fa5e015afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 -- 172.21.15.40:0/4049097070 shutdown_connections 2024-03-24T00:38:49.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.581+0000 7fa5e5e00640 1 -- 172.21.15.40:0/4049097070 wait complete. 2024-03-24T00:38:49.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:38:49.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81b6e00640 1 Processor -- start 2024-03-24T00:38:49.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81b6e00640 1 -- start start 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81b6e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81b015f970 0x7f81b015fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81b6e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81b00cefb0 0x7f81b015c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81b6e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f81b015c690 0x7f81b015cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81b6e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b00ccb40 con 0x7f81b015f970 2024-03-24T00:38:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81b6e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81b00cc550 con 0x7f81b015c690 2024-03-24T00:38:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81b6e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00d0a10 con 0x7f81b00cefb0 2024-03-24T00:38:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81afe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81b015f970 0x7f81b015fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81a7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81b00cefb0 0x7f81b015c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81a7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81b00cefb0 0x7f81b015c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:56172/0 (socket says 172.21.15.40:56172) 2024-03-24T00:38:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81afe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f81b015f970 0x7f81b015fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42362/0 (socket says 172.21.15.40:42362) 2024-03-24T00:38:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81a7e00640 1 -- 172.21.15.40:0/2390116672 learned_addr learned my addr 172.21.15.40:0/2390116672 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:49.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81b4e00640 1 --2- 172.21.15.40:0/2390116672 >> v2:172.21.15.53:3300/0 conn(0x7f81b015c690 0x7f81b015cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81a7e00640 1 -- 172.21.15.40:0/2390116672 >> v2:172.21.15.53:3300/0 conn(0x7f81b015c690 msgr2=0x7f81b015cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81a7e00640 1 --2- 172.21.15.40:0/2390116672 >> v2:172.21.15.53:3300/0 conn(0x7f81b015c690 0x7f81b015cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81a7e00640 1 -- 172.21.15.40:0/2390116672 >> v2:172.21.15.40:3300/0 conn(0x7f81b015f970 msgr2=0x7f81b015fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81a7e00640 1 --2- 172.21.15.40:0/2390116672 >> v2:172.21.15.40:3300/0 conn(0x7f81b015f970 0x7f81b015fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81a7e00640 1 -- 172.21.15.40:0/2390116672 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b0160290 con 0x7f81b00cefb0 2024-03-24T00:38:49.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.745+0000 7f81afe00640 1 --2- 172.21.15.40:0/2390116672 >> v2:172.21.15.40:3300/0 conn(0x7f81b015f970 0x7f81b015fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:49.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81a7e00640 1 --2- 172.21.15.40:0/2390116672 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81b00cefb0 0x7f81b015c150 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f81b00b4270 tx=0x7f819801c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=408062f857c2cc72 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81af400640 1 -- 172.21.15.40:0/2390116672 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81980ab620 con 0x7f81b00cefb0 2024-03-24T00:38:49.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81af400640 1 -- 172.21.15.40:0/2390116672 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81980b0070 con 0x7f81b00cefb0 2024-03-24T00:38:49.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81af400640 1 -- 172.21.15.40:0/2390116672 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81980ab9e0 con 0x7f81b00cefb0 2024-03-24T00:38:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81b6e00640 1 -- 172.21.15.40:0/2390116672 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81b00cefb0 msgr2=0x7f81b015c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81b6e00640 1 --2- 172.21.15.40:0/2390116672 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81b00cefb0 0x7f81b015c150 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f81b00b4270 tx=0x7f819801c460 comp rx=0 tx=0).stop 2024-03-24T00:38:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81b6e00640 1 -- 172.21.15.40:0/2390116672 shutdown_connections 2024-03-24T00:38:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81b6e00640 1 --2- 172.21.15.40:0/2390116672 >> v2:172.21.15.53:3300/0 conn(0x7f81b015c690 0x7f81b015cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81b6e00640 1 --2- 172.21.15.40:0/2390116672 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81b00cefb0 0x7f81b015c150 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81b6e00640 1 --2- 172.21.15.40:0/2390116672 >> v2:172.21.15.40:3300/0 conn(0x7f81b015f970 0x7f81b015fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81b6e00640 1 -- 172.21.15.40:0/2390116672 >> 172.21.15.40:0/2390116672 conn(0x7f81b00b36a0 msgr2=0x7f81b00b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81b6e00640 1 -- 172.21.15.40:0/2390116672 shutdown_connections 2024-03-24T00:38:49.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.749+0000 7f81b6e00640 1 -- 172.21.15.40:0/2390116672 wait complete. 2024-03-24T00:38:49.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81b6e00640 1 Processor -- start 2024-03-24T00:38:49.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81b6e00640 1 -- start start 2024-03-24T00:38:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81b6e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81b00cefb0 0x7f81b00c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81b6e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81b015c690 0x7f81b006c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81b6e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81b015f970 0x7f81b0159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81b6e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b01606c0 con 0x7f81b015c690 2024-03-24T00:38:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81b6e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00cc7d0 con 0x7f81b015f970 2024-03-24T00:38:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81b6e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b00cf410 con 0x7f81b00cefb0 2024-03-24T00:38:49.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81b015c690 0x7f81b006c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81afe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81b00cefb0 0x7f81b00c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81a7e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81b015c690 0x7f81b006c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42374/0 (socket says 172.21.15.40:42374) 2024-03-24T00:38:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81a7e00640 1 -- 172.21.15.40:0/1543229016 learned_addr learned my addr 172.21.15.40:0/1543229016 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81b4e00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81b015f970 0x7f81b0159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81a7e00640 1 -- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81b00cefb0 msgr2=0x7f81b00c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81a7e00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81b00cefb0 0x7f81b00c82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81a7e00640 1 -- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81b015f970 msgr2=0x7f81b0159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81a7e00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81b015f970 0x7f81b0159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81a7e00640 1 -- 172.21.15.40:0/1543229016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b00b4d20 con 0x7f81b015c690 2024-03-24T00:38:49.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81a7e00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81b015c690 0x7f81b006c400 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f819801ab10 tx=0x7f819801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81acc00640 1 -- 172.21.15.40:0/1543229016 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81980abbe0 con 0x7f81b015c690 2024-03-24T00:38:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81acc00640 1 -- 172.21.15.40:0/1543229016 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81980abda0 con 0x7f81b015c690 2024-03-24T00:38:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81b6e00640 1 -- 172.21.15.40:0/1543229016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81b00b5000 con 0x7f81b015c690 2024-03-24T00:38:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81acc00640 1 -- 172.21.15.40:0/1543229016 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81980be630 con 0x7f81b015c690 2024-03-24T00:38:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81b6e00640 1 -- 172.21.15.40:0/1543229016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b00b55c0 con 0x7f81b015c690 2024-03-24T00:38:49.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.753+0000 7f81b6e00640 1 -- 172.21.15.40:0/1543229016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81b00b5790 con 0x7f81b015c690 2024-03-24T00:38:49.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.757+0000 7f81acc00640 1 -- 172.21.15.40:0/1543229016 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f81980b43f0 con 0x7f81b015c690 2024-03-24T00:38:49.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.757+0000 7f81acc00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f818c067660 0x7f818c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:49.760 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.757+0000 7f81acc00640 1 -- 172.21.15.40:0/1543229016 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f819811fb00 con 0x7f81b015c690 2024-03-24T00:38:49.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.757+0000 7f81afe00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f818c067660 0x7f818c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:49.761 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.757+0000 7f81afe00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f818c067660 0x7f818c069b20 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f81a0009580 tx=0x7f81a0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:49.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.761+0000 7f81acc00640 1 -- 172.21.15.40:0/1543229016 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f81980c4060 con 0x7f81b015c690 2024-03-24T00:38:49.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 -- 172.21.15.40:0/1543229016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f81b01604a0 con 0x7f81b015c690 2024-03-24T00:38:49.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81acc00640 1 -- 172.21.15.40:0/1543229016 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8198016110 con 0x7f81b015c690 2024-03-24T00:38:49.928 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:38:49.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 -- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f818c067660 msgr2=0x7f818c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f818c067660 0x7f818c069b20 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f81a0009580 tx=0x7f81a0009190 comp rx=0 tx=0).stop 2024-03-24T00:38:49.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 -- 172.21.15.40:0/1543229016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81b015c690 msgr2=0x7f81b006c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:49.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81b015c690 0x7f81b006c400 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f819801ab10 tx=0x7f819801aca0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 -- 172.21.15.40:0/1543229016 shutdown_connections 2024-03-24T00:38:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f818c067660 0x7f818c069b20 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81b015f970 0x7f81b0159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f81b015c690 0x7f81b006c400 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 --2- 172.21.15.40:0/1543229016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81b00cefb0 0x7f81b00c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 -- 172.21.15.40:0/1543229016 >> 172.21.15.40:0/1543229016 conn(0x7f81b00b36a0 msgr2=0x7f81b006fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 -- 172.21.15.40:0/1543229016 shutdown_connections 2024-03-24T00:38:49.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:49.925+0000 7f81b6e00640 1 -- 172.21.15.40:0/1543229016 wait complete. 2024-03-24T00:38:49.948 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:38:50.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.097+0000 7fa160e00640 1 Processor -- start 2024-03-24T00:38:50.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa160e00640 1 -- start start 2024-03-24T00:38:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa160e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa15c0b4b60 0x7fa15c0b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa160e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa15c0b1c30 0x7fa15c0cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa160e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa15c0b20e0 con 0x7fa15c1595d0 2024-03-24T00:38:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa160e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa15c0b2940 con 0x7fa15c0b4b60 2024-03-24T00:38:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa160e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa15c15c110 con 0x7fa15c0b1c30 2024-03-24T00:38:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa15b400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa15c1595d0 0x7fa15c0b4450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45262/0 (socket says 172.21.15.40:45262) 2024-03-24T00:38:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa15b400640 1 -- 172.21.15.40:0/846532935 learned_addr learned my addr 172.21.15.40:0/846532935 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:50.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa15aa00640 1 --2- 172.21.15.40:0/846532935 >> v2:172.21.15.40:3300/0 conn(0x7fa15c0b4b60 0x7fa15c0b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa15be00640 1 --2- 172.21.15.40:0/846532935 >> v2:172.21.15.53:3300/0 conn(0x7fa15c0b1c30 0x7fa15c0cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa15a000640 1 -- 172.21.15.40:0/846532935 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3940014358 0 0) 0x7fa14c001120 con 0x7fa15c1595d0 2024-03-24T00:38:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa15a000640 1 -- 172.21.15.40:0/846532935 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa140003530 con 0x7fa15c1595d0 2024-03-24T00:38:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa15aa00640 1 -- 172.21.15.40:0/846532935 >> v2:172.21.15.53:3300/0 conn(0x7fa15c0b1c30 msgr2=0x7fa15c0cdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa15aa00640 1 --2- 172.21.15.40:0/846532935 >> v2:172.21.15.53:3300/0 conn(0x7fa15c0b1c30 0x7fa15c0cdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa15aa00640 1 -- 172.21.15.40:0/846532935 >> v1:172.21.15.40:6789/0 conn(0x7fa15c1595d0 legacy=0x7fa15c0b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.101+0000 7fa15aa00640 1 -- 172.21.15.40:0/846532935 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c15c350 con 0x7fa15c0b4b60 2024-03-24T00:38:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa15aa00640 1 --2- 172.21.15.40:0/846532935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15c0b4b60 0x7fa15c0b16f0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0cd270 tx=0x7fa14801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5683cc6973b47256 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa15a000640 1 -- 172.21.15.40:0/846532935 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1480b1070 con 0x7fa15c0b4b60 2024-03-24T00:38:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa15a000640 1 -- 172.21.15.40:0/846532935 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1480ac620 con 0x7fa15c0b4b60 2024-03-24T00:38:50.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa15a000640 1 -- 172.21.15.40:0/846532935 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1480ac9f0 con 0x7fa15c0b4b60 2024-03-24T00:38:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa160e00640 1 -- 172.21.15.40:0/846532935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15c0b4b60 msgr2=0x7fa15c0b16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa160e00640 1 --2- 172.21.15.40:0/846532935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15c0b4b60 0x7fa15c0b16f0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fa15c0cd270 tx=0x7fa14801c3c0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa160e00640 1 -- 172.21.15.40:0/846532935 shutdown_connections 2024-03-24T00:38:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa160e00640 1 --2- 172.21.15.40:0/846532935 >> v2:172.21.15.53:3300/0 conn(0x7fa15c0b1c30 0x7fa15c0cdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa160e00640 1 --2- 172.21.15.40:0/846532935 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15c0b4b60 0x7fa15c0b16f0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa160e00640 1 -- 172.21.15.40:0/846532935 >> 172.21.15.40:0/846532935 conn(0x7fa15c066e80 msgr2=0x7fa15c067290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa160e00640 1 -- 172.21.15.40:0/846532935 shutdown_connections 2024-03-24T00:38:50.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa160e00640 1 -- 172.21.15.40:0/846532935 wait complete. 2024-03-24T00:38:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa160e00640 1 Processor -- start 2024-03-24T00:38:50.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.105+0000 7fa160e00640 1 -- start start 2024-03-24T00:38:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa160e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15c0b1c30 0x7fa15c1483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa160e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa15c0b4b60 0x7fa15c144630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa160e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa15c1595d0 0x7fa15c071710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa160e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c1599b0 con 0x7fa15c0b1c30 2024-03-24T00:38:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa160e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c15ced0 con 0x7fa15c1595d0 2024-03-24T00:38:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa160e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa15c159b30 con 0x7fa15c0b4b60 2024-03-24T00:38:50.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa15b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15c0b1c30 0x7fa15c1483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa15b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15c0b1c30 0x7fa15c1483e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42394/0 (socket says 172.21.15.40:42394) 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa15b400640 1 -- 172.21.15.40:0/593807005 learned_addr learned my addr 172.21.15.40:0/593807005 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa15be00640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa15c1595d0 0x7fa15c071710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa15aa00640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa15c0b4b60 0x7fa15c144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa15b400640 1 -- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa15c0b4b60 msgr2=0x7fa15c144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa15b400640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa15c0b4b60 0x7fa15c144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa15b400640 1 -- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa15c1595d0 msgr2=0x7fa15c071710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa15b400640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa15c1595d0 0x7fa15c071710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa15b400640 1 -- 172.21.15.40:0/593807005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa15c0721f0 con 0x7fa15c0b1c30 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa15aa00640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa15c0b4b60 0x7fa15c144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa15b400640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15c0b1c30 0x7fa15c1483e0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00c3f0 tx=0x7fa14c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa153400640 1 -- 172.21.15.40:0/593807005 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c017070 con 0x7fa15c0b1c30 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa153400640 1 -- 172.21.15.40:0/593807005 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa14c005cf0 con 0x7fa15c0b1c30 2024-03-24T00:38:50.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa153400640 1 -- 172.21.15.40:0/593807005 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa14c007310 con 0x7fa15c0b1c30 2024-03-24T00:38:50.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa160e00640 1 -- 172.21.15.40:0/593807005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa15c0724d0 con 0x7fa15c0b1c30 2024-03-24T00:38:50.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa160e00640 1 -- 172.21.15.40:0/593807005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa15c157350 con 0x7fa15c0b1c30 2024-03-24T00:38:50.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.109+0000 7fa160e00640 1 -- 172.21.15.40:0/593807005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa15c157160 con 0x7fa15c0b1c30 2024-03-24T00:38:50.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.113+0000 7fa153400640 1 -- 172.21.15.40:0/593807005 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa14c01a070 con 0x7fa15c0b1c30 2024-03-24T00:38:50.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.113+0000 7fa153400640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa13c067690 0x7fa13c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.113+0000 7fa153400640 1 -- 172.21.15.40:0/593807005 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa14c084ba0 con 0x7fa15c0b1c30 2024-03-24T00:38:50.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.113+0000 7fa15aa00640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa13c067690 0x7fa13c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.113+0000 7fa15aa00640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa13c067690 0x7fa13c069b50 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa1480ae000 tx=0x7fa148008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:50.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.117+0000 7fa153400640 1 -- 172.21.15.40:0/593807005 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa15c0b20e0 con 0x7fa15c0b1c30 2024-03-24T00:38:50.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.281+0000 7fa160e00640 1 -- 172.21.15.40:0/593807005 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa15c04f410 con 0x7fa15c0b1c30 2024-03-24T00:38:50.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.281+0000 7fa153400640 1 -- 172.21.15.40:0/593807005 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa14c055220 con 0x7fa15c0b1c30 2024-03-24T00:38:50.286 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:38:50.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.285+0000 7fa160e00640 1 -- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa13c067690 msgr2=0x7fa13c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.285+0000 7fa160e00640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa13c067690 0x7fa13c069b50 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa1480ae000 tx=0x7fa148008040 comp rx=0 tx=0).stop 2024-03-24T00:38:50.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.285+0000 7fa160e00640 1 -- 172.21.15.40:0/593807005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15c0b1c30 msgr2=0x7fa15c1483e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.285+0000 7fa160e00640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15c0b1c30 0x7fa15c1483e0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fa14c00c3f0 tx=0x7fa14c00c8b0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.285+0000 7fa160e00640 1 -- 172.21.15.40:0/593807005 shutdown_connections 2024-03-24T00:38:50.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.285+0000 7fa160e00640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa13c067690 0x7fa13c069b50 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.285+0000 7fa160e00640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa15c1595d0 0x7fa15c071710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.285+0000 7fa160e00640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa15c0b4b60 0x7fa15c144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.285+0000 7fa160e00640 1 --2- 172.21.15.40:0/593807005 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa15c0b1c30 0x7fa15c1483e0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.285+0000 7fa160e00640 1 -- 172.21.15.40:0/593807005 >> 172.21.15.40:0/593807005 conn(0x7fa15c066e80 msgr2=0x7fa15c15e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:50.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.285+0000 7fa160e00640 1 -- 172.21.15.40:0/593807005 shutdown_connections 2024-03-24T00:38:50.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.285+0000 7fa160e00640 1 -- 172.21.15.40:0/593807005 wait complete. 2024-03-24T00:38:50.309 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:38:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.445+0000 7fd7ef800640 1 Processor -- start 2024-03-24T00:38:50.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.445+0000 7fd7ef800640 1 -- start start 2024-03-24T00:38:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7ef800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7e80cdaa0 0x7fd7e8168fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7ef800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e80cfaa0 con 0x7fd7e815dfc0 2024-03-24T00:38:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7ef800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e80cfd80 con 0x7fd7e815e3a0 2024-03-24T00:38:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7ef800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e80cf870 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7ece00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd7e815dfc0 0x7fd7e80cd390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45276/0 (socket says 172.21.15.40:45276) 2024-03-24T00:38:50.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7ece00640 1 -- 172.21.15.40:0/2247761733 learned_addr learned my addr 172.21.15.40:0/2247761733 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:50.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7ed800640 1 --2- 172.21.15.40:0/2247761733 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7e80cdaa0 0x7fd7e8168fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7e7e00640 1 -- 172.21.15.40:0/2247761733 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467675976 0 0) 0x7fd7e80cfaa0 con 0x7fd7e815dfc0 2024-03-24T00:38:50.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7e7e00640 1 -- 172.21.15.40:0/2247761733 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7cc003530 con 0x7fd7e815dfc0 2024-03-24T00:38:50.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7e7e00640 1 -- 172.21.15.40:0/2247761733 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183375161 0 0) 0x7fd7e80cfd80 con 0x7fd7e815e3a0 2024-03-24T00:38:50.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7e7e00640 1 -- 172.21.15.40:0/2247761733 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7e80cfaa0 con 0x7fd7e815e3a0 2024-03-24T00:38:50.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7ed800640 1 -- 172.21.15.40:0/2247761733 >> v1:172.21.15.53:6789/0 conn(0x7fd7e815e3a0 legacy=0x7fd7e80ce690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7ed800640 1 -- 172.21.15.40:0/2247761733 >> v1:172.21.15.40:6789/0 conn(0x7fd7e815dfc0 legacy=0x7fd7e80cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7e7e00640 1 -- 172.21.15.40:0/2247761733 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4075479372 0 0) 0x7fd7cc003530 con 0x7fd7e815dfc0 2024-03-24T00:38:50.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.449+0000 7fd7ed800640 1 -- 172.21.15.40:0/2247761733 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e80d0060 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.453+0000 7fd7ed800640 1 --2- 172.21.15.40:0/2247761733 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7e80cdaa0 0x7fd7e8168fc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0009dc0 tx=0x7fd7e001c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba8582c8bc87759e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:50.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.453+0000 7fd7e7e00640 1 -- 172.21.15.40:0/2247761733 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8001680 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.453+0000 7fd7e7e00640 1 -- 172.21.15.40:0/2247761733 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e00a64f0 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.453+0000 7fd7e7e00640 1 -- 172.21.15.40:0/2247761733 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7e00a68c0 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.453+0000 7fd7ef800640 1 -- 172.21.15.40:0/2247761733 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7e80cdaa0 msgr2=0x7fd7e8168fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.453+0000 7fd7ef800640 1 --2- 172.21.15.40:0/2247761733 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7e80cdaa0 0x7fd7e8168fc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd7e0009dc0 tx=0x7fd7e001c790 comp rx=0 tx=0).stop 2024-03-24T00:38:50.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.453+0000 7fd7ef800640 1 -- 172.21.15.40:0/2247761733 shutdown_connections 2024-03-24T00:38:50.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.453+0000 7fd7ef800640 1 --2- 172.21.15.40:0/2247761733 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7e80cdaa0 0x7fd7e8168fc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.453+0000 7fd7ef800640 1 -- 172.21.15.40:0/2247761733 >> 172.21.15.40:0/2247761733 conn(0x7fd7e80b37f0 msgr2=0x7fd7e80b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:50.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.453+0000 7fd7ef800640 1 -- 172.21.15.40:0/2247761733 shutdown_connections 2024-03-24T00:38:50.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.453+0000 7fd7ef800640 1 -- 172.21.15.40:0/2247761733 wait complete. 2024-03-24T00:38:50.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ef800640 1 Processor -- start 2024-03-24T00:38:50.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ef800640 1 -- start start 2024-03-24T00:38:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ef800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80cdaa0 0x7fd7e80b54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ef800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7e815dfc0 0x7fd7e80d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ef800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7e815e3a0 0x7fd7e80d2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ef800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e816bd20 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ef800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e816bea0 con 0x7fd7e815dfc0 2024-03-24T00:38:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ef800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e816c020 con 0x7fd7e815e3a0 2024-03-24T00:38:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ece00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80cdaa0 0x7fd7e80b54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7dfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7e815dfc0 0x7fd7e80d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ece00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80cdaa0 0x7fd7e80b54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42408/0 (socket says 172.21.15.40:42408) 2024-03-24T00:38:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ece00640 1 -- 172.21.15.40:0/2990282077 learned_addr learned my addr 172.21.15.40:0/2990282077 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ed800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7e815e3a0 0x7fd7e80d2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ece00640 1 -- 172.21.15.40:0/2990282077 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7e815e3a0 msgr2=0x7fd7e80d2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ece00640 1 --2- 172.21.15.40:0/2990282077 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7e815e3a0 0x7fd7e80d2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ece00640 1 -- 172.21.15.40:0/2990282077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7e815dfc0 msgr2=0x7fd7e80d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ece00640 1 --2- 172.21.15.40:0/2990282077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7e815dfc0 0x7fd7e80d1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ece00640 1 -- 172.21.15.40:0/2990282077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e8153930 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ece00640 1 --2- 172.21.15.40:0/2990282077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80cdaa0 0x7fd7e80b54d0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8001ec0 tx=0x7fd7d800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:50.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7e5600640 1 -- 172.21.15.40:0/2990282077 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8005ae0 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7e5600640 1 -- 172.21.15.40:0/2990282077 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d8005c80 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7e5600640 1 -- 172.21.15.40:0/2990282077 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d80072b0 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.457+0000 7fd7ef800640 1 -- 172.21.15.40:0/2990282077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e8153c10 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.461+0000 7fd7ef800640 1 -- 172.21.15.40:0/2990282077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e80b5a10 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.461+0000 7fd7ef800640 1 -- 172.21.15.40:0/2990282077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e8068170 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.461+0000 7fd7e5600640 1 -- 172.21.15.40:0/2990282077 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd7d8007450 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.461+0000 7fd7e5600640 1 --2- 172.21.15.40:0/2990282077 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7c4067660 0x7fd7c4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:50.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.461+0000 7fd7e5600640 1 -- 172.21.15.40:0/2990282077 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd7d8085610 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.465+0000 7fd7dfe00640 1 --2- 172.21.15.40:0/2990282077 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7c4067660 0x7fd7c4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:50.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.465+0000 7fd7dfe00640 1 --2- 172.21.15.40:0/2990282077 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7c4067660 0x7fd7c4069b20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd7e806b320 tx=0x7fd7d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:50.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.469+0000 7fd7e5600640 1 -- 172.21.15.40:0/2990282077 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd7d8055c20 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 -- 172.21.15.40:0/2990282077 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd7e806ef30 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7e5600640 1 -- 172.21.15.40:0/2990282077 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v107) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd7d8059cc0 con 0x7fd7e80cdaa0 2024-03-24T00:38:50.648 INFO:teuthology.orchestra.run.smithi040.stderr:listed 0 entries 2024-03-24T00:38:50.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 -- 172.21.15.40:0/2990282077 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7c4067660 msgr2=0x7fd7c4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 --2- 172.21.15.40:0/2990282077 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7c4067660 0x7fd7c4069b20 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd7e806b320 tx=0x7fd7d0009040 comp rx=0 tx=0).stop 2024-03-24T00:38:50.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 -- 172.21.15.40:0/2990282077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80cdaa0 msgr2=0x7fd7e80b54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:50.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 --2- 172.21.15.40:0/2990282077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80cdaa0 0x7fd7e80b54d0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8001ec0 tx=0x7fd7d800cbe0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 -- 172.21.15.40:0/2990282077 shutdown_connections 2024-03-24T00:38:50.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 --2- 172.21.15.40:0/2990282077 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd7c4067660 0x7fd7c4069b20 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 --2- 172.21.15.40:0/2990282077 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd7e815e3a0 0x7fd7e80d2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 --2- 172.21.15.40:0/2990282077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7e815dfc0 0x7fd7e80d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 --2- 172.21.15.40:0/2990282077 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd7e80cdaa0 0x7fd7e80b54d0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:50.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 -- 172.21.15.40:0/2990282077 >> 172.21.15.40:0/2990282077 conn(0x7fd7e80b37f0 msgr2=0x7fd7e80705e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:50.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 -- 172.21.15.40:0/2990282077 shutdown_connections 2024-03-24T00:38:50.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:50.645+0000 7fd7ef800640 1 -- 172.21.15.40:0/2990282077 wait complete. 2024-03-24T00:38:50.669 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-03-24T00:38:50.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:50.669 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T00:38:50.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[51406]: fuse finished with error 0 and tester_r 0 2024-03-24T00:38:50.691 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:38:50.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:50.691 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:38:50.869 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:38:50.869 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:38:50.878 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:38:50.878 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:38:50.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:50.879 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T00:38:50.929 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T00:38:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:38:50.931 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:38:50.931 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:38:50.939 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:38:50.939 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:38:50.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:50.939 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T00:38:50.993 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T00:38:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:38:50.994 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:38:50.994 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:38:51.008 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:38:51.008 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:38:51.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:38:51.008 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T00:38:51.058 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T00:38:51.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:38:51.059 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:51.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.197+0000 7f85e5600640 1 Processor -- start 2024-03-24T00:38:51.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.197+0000 7f85e5600640 1 -- start start 2024-03-24T00:38:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.201+0000 7f85e5600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f85e00b2490 0x7f85e00cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.201+0000 7f85e5600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e00b2870 con 0x7f85e00b2aa0 2024-03-24T00:38:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.201+0000 7f85e5600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e00b3190 con 0x7f85e00d0a80 2024-03-24T00:38:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.201+0000 7f85e5600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85e006b6d0 con 0x7f85e00b2490 2024-03-24T00:38:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.201+0000 7f85d7e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f85e00b2aa0 0x7f85e00d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45284/0 (socket says 172.21.15.40:45284) 2024-03-24T00:38:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.201+0000 7f85d7e00640 1 -- 172.21.15.40:0/2740239886 learned_addr learned my addr 172.21.15.40:0/2740239886 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:51.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.201+0000 7f85dea00640 1 --2- 172.21.15.40:0/2740239886 >> v2:172.21.15.53:3300/0 conn(0x7f85e00b2490 0x7f85e00cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.201+0000 7f85de000640 1 -- 172.21.15.40:0/2740239886 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198209918 0 0) 0x7f85e00b2870 con 0x7f85e00b2aa0 2024-03-24T00:38:51.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.201+0000 7f85de000640 1 -- 172.21.15.40:0/2740239886 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85c4003530 con 0x7f85e00b2aa0 2024-03-24T00:38:51.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.201+0000 7f85dea00640 1 -- 172.21.15.40:0/2740239886 >> v1:172.21.15.53:6789/0 conn(0x7f85e00d0a80 legacy=0x7f85e00cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.201+0000 7f85dea00640 1 -- 172.21.15.40:0/2740239886 >> v1:172.21.15.40:6789/0 conn(0x7f85e00b2aa0 legacy=0x7f85e00d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.201+0000 7f85dea00640 1 -- 172.21.15.40:0/2740239886 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e00b3470 con 0x7f85e00b2490 2024-03-24T00:38:51.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85dea00640 1 --2- 172.21.15.40:0/2740239886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85e00b2490 0x7f85e00cfe30 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f85d00097c0 tx=0x7f85d001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83926c6875cf2236 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:51.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85de000640 1 -- 172.21.15.40:0/2740239886 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d00aa620 con 0x7f85e00b2490 2024-03-24T00:38:51.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85de000640 1 -- 172.21.15.40:0/2740239886 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d00af070 con 0x7f85e00b2490 2024-03-24T00:38:51.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85de000640 1 -- 172.21.15.40:0/2740239886 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d00aa9c0 con 0x7f85e00b2490 2024-03-24T00:38:51.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85e5600640 1 -- 172.21.15.40:0/2740239886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85e00b2490 msgr2=0x7f85e00cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85e5600640 1 --2- 172.21.15.40:0/2740239886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85e00b2490 0x7f85e00cfe30 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f85d00097c0 tx=0x7f85d001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85e5600640 1 -- 172.21.15.40:0/2740239886 shutdown_connections 2024-03-24T00:38:51.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85e5600640 1 --2- 172.21.15.40:0/2740239886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85e00b2490 0x7f85e00cfe30 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85e5600640 1 -- 172.21.15.40:0/2740239886 >> 172.21.15.40:0/2740239886 conn(0x7f85e00d1d10 msgr2=0x7f85e00d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:51.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85e5600640 1 -- 172.21.15.40:0/2740239886 shutdown_connections 2024-03-24T00:38:51.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85e5600640 1 -- 172.21.15.40:0/2740239886 wait complete. 2024-03-24T00:38:51.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85e5600640 1 Processor -- start 2024-03-24T00:38:51.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.205+0000 7f85e5600640 1 -- start start 2024-03-24T00:38:51.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85e5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85e00b2490 0x7f85e01560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85e5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85e00b2aa0 0x7f85e00cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85e5600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85e00d0a80 0x7f85e0154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85e5600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e016bfd0 con 0x7f85e00b2490 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85e5600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85e00cd650 con 0x7f85e00b2aa0 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85e5600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85e016c150 con 0x7f85e00d0a80 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85dea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85e00b2490 0x7f85e01560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85dea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85e00b2490 0x7f85e01560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42422/0 (socket says 172.21.15.40:42422) 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85dea00640 1 -- 172.21.15.40:0/1328500588 learned_addr learned my addr 172.21.15.40:0/1328500588 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85d7e00640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85e00b2aa0 0x7f85e00cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85df400640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85e00d0a80 0x7f85e0154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85dea00640 1 -- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85e00d0a80 msgr2=0x7f85e0154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85dea00640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85e00d0a80 0x7f85e0154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85dea00640 1 -- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85e00b2aa0 msgr2=0x7f85e00cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85dea00640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85e00b2aa0 0x7f85e00cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85dea00640 1 -- 172.21.15.40:0/1328500588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e015a330 con 0x7f85e00b2490 2024-03-24T00:38:51.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85df400640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85e00d0a80 0x7f85e0154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:51.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85dea00640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85e00b2490 0x7f85e01560c0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f85d00097c0 tx=0x7f85d001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:51.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85d6a00640 1 -- 172.21.15.40:0/1328500588 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d00aacc0 con 0x7f85e00b2490 2024-03-24T00:38:51.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85d6a00640 1 -- 172.21.15.40:0/1328500588 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d0004020 con 0x7f85e00b2490 2024-03-24T00:38:51.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85e5600640 1 -- 172.21.15.40:0/1328500588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e015a610 con 0x7f85e00b2490 2024-03-24T00:38:51.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85d6a00640 1 -- 172.21.15.40:0/1328500588 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d00aae60 con 0x7f85e00b2490 2024-03-24T00:38:51.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85e5600640 1 -- 172.21.15.40:0/1328500588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e0141af0 con 0x7f85e00b2490 2024-03-24T00:38:51.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85e5600640 1 -- 172.21.15.40:0/1328500588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e0068170 con 0x7f85e00b2490 2024-03-24T00:38:51.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.209+0000 7f85d6a00640 1 -- 172.21.15.40:0/1328500588 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f85d0003620 con 0x7f85e00b2490 2024-03-24T00:38:51.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.213+0000 7f85d6a00640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f85bc067690 0x7f85bc069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.213+0000 7f85d6a00640 1 -- 172.21.15.40:0/1328500588 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f85d011faa0 con 0x7f85e00b2490 2024-03-24T00:38:51.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.213+0000 7f85d7e00640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f85bc067690 0x7f85bc069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.213+0000 7f85d7e00640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f85bc067690 0x7f85bc069b50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f85e0158f70 tx=0x7f85c8007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:51.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.217+0000 7f85d6a00640 1 -- 172.21.15.40:0/1328500588 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f85d0016090 con 0x7f85e00b2490 2024-03-24T00:38:51.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.413+0000 7f85e5600640 1 -- 172.21.15.40:0/1328500588 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85e0069530 con 0x7f85e00b2490 2024-03-24T00:38:51.418 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.413+0000 7f85d6a00640 1 -- 172.21.15.40:0/1328500588 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f85d00f0240 con 0x7f85e00b2490 2024-03-24T00:38:51.419 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:51.419 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":119,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7079,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/755981654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":755981654},{"type":"v1","addr":"172.21.15.40:6835","nonce":755981654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:01.424871+0000","modified":"2024-03-24T00:38:42.207808+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8418,"mds_1":8400,"mds_2":8101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8101":{"gid":8101,"name":"d","rank":2,"incarnation":116,"state":"up:active","state_seq":7,"addr":"172.21.15.53:6836/3877615536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3877615536},{"type":"v1","addr":"172.21.15.53:6836","nonce":3877615536}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8400":{"gid":8400,"name":"c","rank":1,"incarnation":114,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6834/3883043397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":3883043397},{"type":"v1","addr":"172.21.15.40:6834","nonce":3883043397}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8418":{"gid":8418,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/8812937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":8812937},{"type":"v1","addr":"172.21.15.53:6837","nonce":8812937}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8418,"qdb_cluster":[8101,8400,8418]},"id":8}]} 2024-03-24T00:38:51.419 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 119 2024-03-24T00:38:51.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.417+0000 7f85e5600640 1 -- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f85bc067690 msgr2=0x7f85bc069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.417+0000 7f85e5600640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f85bc067690 0x7f85bc069b50 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f85e0158f70 tx=0x7f85c8007d40 comp rx=0 tx=0).stop 2024-03-24T00:38:51.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.417+0000 7f85e5600640 1 -- 172.21.15.40:0/1328500588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85e00b2490 msgr2=0x7f85e01560c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.417+0000 7f85e5600640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85e00b2490 0x7f85e01560c0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f85d00097c0 tx=0x7f85d001ace0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.417+0000 7f85e5600640 1 -- 172.21.15.40:0/1328500588 shutdown_connections 2024-03-24T00:38:51.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.417+0000 7f85e5600640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f85bc067690 0x7f85bc069b50 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.417+0000 7f85e5600640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85e00d0a80 0x7f85e0154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.417+0000 7f85e5600640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85e00b2aa0 0x7f85e00cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.417+0000 7f85e5600640 1 --2- 172.21.15.40:0/1328500588 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f85e00b2490 0x7f85e01560c0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.417+0000 7f85e5600640 1 -- 172.21.15.40:0/1328500588 >> 172.21.15.40:0/1328500588 conn(0x7f85e00d1d10 msgr2=0x7f85e00cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:51.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.417+0000 7f85e5600640 1 -- 172.21.15.40:0/1328500588 shutdown_connections 2024-03-24T00:38:51.421 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.417+0000 7f85e5600640 1 -- 172.21.15.40:0/1328500588 wait complete. 2024-03-24T00:38:51.440 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:51.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.577+0000 7f46dc600640 1 Processor -- start 2024-03-24T00:38:51.581 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.577+0000 7f46dc600640 1 -- start start 2024-03-24T00:38:51.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.577+0000 7f46dc600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f46d40b2490 0x7f46d40cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.577+0000 7f46dc600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46d40b2aa0 0x7f46d40d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.577+0000 7f46dc600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d40b2870 con 0x7f46d40d08b0 2024-03-24T00:38:51.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.577+0000 7f46dc600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d406b6d0 con 0x7f46d40b2490 2024-03-24T00:38:51.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.577+0000 7f46dc600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d40b3190 con 0x7f46d40b2aa0 2024-03-24T00:38:51.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.577+0000 7f46d9c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f46d40b2490 0x7f46d40cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.577+0000 7f46da600640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f46d40d08b0 0x7f46d40ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:60292/0 (socket says 172.21.15.40:60292) 2024-03-24T00:38:51.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.577+0000 7f46d9200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46d40b2aa0 0x7f46d40d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.577+0000 7f46da600640 1 -- 172.21.15.40:0/2924111977 learned_addr learned my addr 172.21.15.40:0/2924111977 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:51.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46d9c00640 1 -- 172.21.15.40:0/2924111977 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46d40b2aa0 msgr2=0x7f46d40d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:51.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46d9c00640 1 --2- 172.21.15.40:0/2924111977 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46d40b2aa0 0x7f46d40d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46d9c00640 1 -- 172.21.15.40:0/2924111977 >> v1:172.21.15.53:6789/0 conn(0x7f46d40d08b0 legacy=0x7f46d40ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46d9c00640 1 -- 172.21.15.40:0/2924111977 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d40b3470 con 0x7f46d40b2490 2024-03-24T00:38:51.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46d3e00640 1 -- 172.21.15.40:0/2924111977 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106553994 0 0) 0x7f46d40b2870 con 0x7f46d40d08b0 2024-03-24T00:38:51.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46d9c00640 1 --2- 172.21.15.40:0/2924111977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40b2490 0x7f46d40cfe30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f46c80097c0 tx=0x7f46c801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b7c347200bfe9fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:51.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46d3e00640 1 -- 172.21.15.40:0/2924111977 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c80b0070 con 0x7f46d40b2490 2024-03-24T00:38:51.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46d3e00640 1 -- 172.21.15.40:0/2924111977 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c80ab600 con 0x7f46d40b2490 2024-03-24T00:38:51.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46d3e00640 1 -- 172.21.15.40:0/2924111977 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c80ab9d0 con 0x7f46d40b2490 2024-03-24T00:38:51.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46dc600640 1 -- 172.21.15.40:0/2924111977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40b2490 msgr2=0x7f46d40cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46dc600640 1 --2- 172.21.15.40:0/2924111977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40b2490 0x7f46d40cfe30 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f46c80097c0 tx=0x7f46c801c4a0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46dc600640 1 -- 172.21.15.40:0/2924111977 shutdown_connections 2024-03-24T00:38:51.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46dc600640 1 --2- 172.21.15.40:0/2924111977 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46d40b2aa0 0x7f46d40d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.587 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46dc600640 1 --2- 172.21.15.40:0/2924111977 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40b2490 0x7f46d40cfe30 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.581+0000 7f46dc600640 1 -- 172.21.15.40:0/2924111977 >> 172.21.15.40:0/2924111977 conn(0x7f46d40d1d10 msgr2=0x7f46d40d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:51.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46dc600640 1 -- 172.21.15.40:0/2924111977 shutdown_connections 2024-03-24T00:38:51.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46dc600640 1 -- 172.21.15.40:0/2924111977 wait complete. 2024-03-24T00:38:51.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46dc600640 1 Processor -- start 2024-03-24T00:38:51.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46dc600640 1 -- start start 2024-03-24T00:38:51.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46dc600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40b2490 0x7f46d40cb210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46dc600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46d40b2aa0 0x7f46d40b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46dc600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46d40d08b0 0x7f46d414e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46dc600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d416b350 con 0x7f46d40d08b0 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46dc600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d416b1d0 con 0x7f46d40b2490 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46dc600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d416b4d0 con 0x7f46d40b2aa0 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46da600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46d40d08b0 0x7f46d414e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46d9200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46d40b2aa0 0x7f46d40b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46d9c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40b2490 0x7f46d40cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46d9200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46d40b2aa0 0x7f46d40b0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:56232/0 (socket says 172.21.15.40:56232) 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46da600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46d40d08b0 0x7f46d414e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42434/0 (socket says 172.21.15.40:42434) 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46d9200640 1 -- 172.21.15.40:0/2108418049 learned_addr learned my addr 172.21.15.40:0/2108418049 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46d9200640 1 -- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40b2490 msgr2=0x7f46d40cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46d9200640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40b2490 0x7f46d40cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46d9200640 1 -- 172.21.15.40:0/2108418049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46d40d08b0 msgr2=0x7f46d414e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46d9200640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46d40d08b0 0x7f46d414e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46d9200640 1 -- 172.21.15.40:0/2108418049 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d40b1a30 con 0x7f46d40b2aa0 2024-03-24T00:38:51.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46d9c00640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40b2490 0x7f46d40cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:51.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46d9200640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46d40b2aa0 0x7f46d40b0a60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f46c000e930 tx=0x7f46c000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:51.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.585+0000 7f46d1600640 1 -- 172.21.15.40:0/2108418049 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0017070 con 0x7f46d40b2aa0 2024-03-24T00:38:51.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.589+0000 7f46d1600640 1 -- 172.21.15.40:0/2108418049 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c000ca00 con 0x7f46d40b2aa0 2024-03-24T00:38:51.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.589+0000 7f46d1600640 1 -- 172.21.15.40:0/2108418049 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c000cdb0 con 0x7f46d40b2aa0 2024-03-24T00:38:51.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.589+0000 7f46dc600640 1 -- 172.21.15.40:0/2108418049 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d40b1190 con 0x7f46d40b2aa0 2024-03-24T00:38:51.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.589+0000 7f46dc600640 1 -- 172.21.15.40:0/2108418049 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d40b1750 con 0x7f46d40b2aa0 2024-03-24T00:38:51.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.589+0000 7f46dc600640 1 -- 172.21.15.40:0/2108418049 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d4068170 con 0x7f46d40b2aa0 2024-03-24T00:38:51.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.589+0000 7f46d1600640 1 -- 172.21.15.40:0/2108418049 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f46c001a070 con 0x7f46d40b2aa0 2024-03-24T00:38:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.589+0000 7f46d1600640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46b4067660 0x7f46b4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.589+0000 7f46d1600640 1 -- 172.21.15.40:0/2108418049 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f46c0085900 con 0x7f46d40b2aa0 2024-03-24T00:38:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.589+0000 7f46d9c00640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46b4067660 0x7f46b4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.589+0000 7f46d9c00640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46b4067660 0x7f46b4069b20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f46c80ad000 tx=0x7f46c80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:51.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.593+0000 7f46d1600640 1 -- 172.21.15.40:0/2108418049 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f46c0055f20 con 0x7f46d40b2aa0 2024-03-24T00:38:51.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.773+0000 7f46dc600640 1 -- 172.21.15.40:0/2108418049 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46d406ddc0 con 0x7f46d40b2aa0 2024-03-24T00:38:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.773+0000 7f46d1600640 1 -- 172.21.15.40:0/2108418049 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16720 (secure 0 0 0) 0x7f46c0059fc0 con 0x7f46d40b2aa0 2024-03-24T00:38:51.778 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:51.778 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":108,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:38:50.991073+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:37:59.849300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:00.417342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.1a","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:51.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.777+0000 7f46dc600640 1 -- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46b4067660 msgr2=0x7f46b4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.777+0000 7f46dc600640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46b4067660 0x7f46b4069b20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f46c80ad000 tx=0x7f46c80a6000 comp rx=0 tx=0).stop 2024-03-24T00:38:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.777+0000 7f46dc600640 1 -- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46d40b2aa0 msgr2=0x7f46d40b0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.777+0000 7f46dc600640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46d40b2aa0 0x7f46d40b0a60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f46c000e930 tx=0x7f46c000edf0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.777+0000 7f46dc600640 1 -- 172.21.15.40:0/2108418049 shutdown_connections 2024-03-24T00:38:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.777+0000 7f46dc600640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f46b4067660 0x7f46b4069b20 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.777+0000 7f46dc600640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f46d40d08b0 0x7f46d414e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.777+0000 7f46dc600640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46d40b2aa0 0x7f46d40b0a60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.777+0000 7f46dc600640 1 --2- 172.21.15.40:0/2108418049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d40b2490 0x7f46d40cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.777+0000 7f46dc600640 1 -- 172.21.15.40:0/2108418049 >> 172.21.15.40:0/2108418049 conn(0x7f46d40d1d10 msgr2=0x7f46d40cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.777+0000 7f46dc600640 1 -- 172.21.15.40:0/2108418049 shutdown_connections 2024-03-24T00:38:51.781 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.777+0000 7f46dc600640 1 -- 172.21.15.40:0/2108418049 wait complete. 2024-03-24T00:38:51.799 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:37:59.849300+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:38:00.417342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:38:51.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:38:51.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e5cc00640 1 Processor -- start 2024-03-24T00:38:51.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e5cc00640 1 -- start start 2024-03-24T00:38:51.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e5cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8e5815b960 0x7f8e58070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e5cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5815bd40 0x7f8e580712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e5cc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e580d0fe0 0x7f8e580d1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e5cc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e58160580 con 0x7f8e580d0fe0 2024-03-24T00:38:51.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e5cc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e5815f970 con 0x7f8e5815b960 2024-03-24T00:38:51.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e5cc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5815fc50 con 0x7f8e5815bd40 2024-03-24T00:38:51.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e56800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e580d0fe0 0x7f8e580d1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e56800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8e580d0fe0 0x7f8e580d1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42436/0 (socket says 172.21.15.40:42436) 2024-03-24T00:38:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e56800640 1 -- 172.21.15.40:0/1641340916 learned_addr learned my addr 172.21.15.40:0/1641340916 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e55e00640 1 --2- 172.21.15.40:0/1641340916 >> v2:172.21.15.53:3300/0 conn(0x7f8e5815b960 0x7f8e58070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e55400640 1 --2- 172.21.15.40:0/1641340916 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5815bd40 0x7f8e580712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e56800640 1 -- 172.21.15.40:0/1641340916 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5815bd40 msgr2=0x7f8e580712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e56800640 1 --2- 172.21.15.40:0/1641340916 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5815bd40 0x7f8e580712a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e56800640 1 -- 172.21.15.40:0/1641340916 >> v2:172.21.15.53:3300/0 conn(0x7f8e5815b960 msgr2=0x7f8e58070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:51.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e56800640 1 --2- 172.21.15.40:0/1641340916 >> v2:172.21.15.53:3300/0 conn(0x7f8e5815b960 0x7f8e58070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.941+0000 7f8e56800640 1 -- 172.21.15.40:0/1641340916 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5815ff30 con 0x7f8e580d0fe0 2024-03-24T00:38:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e56800640 1 --2- 172.21.15.40:0/1641340916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e580d0fe0 0x7f8e580d1470 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f8e5815fe60 tx=0x7f8e5001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5168f5f43e5c4fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e54a00640 1 -- 172.21.15.40:0/1641340916 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e500a5670 con 0x7f8e580d0fe0 2024-03-24T00:38:51.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e54a00640 1 -- 172.21.15.40:0/1641340916 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e500a5810 con 0x7f8e580d0fe0 2024-03-24T00:38:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e54a00640 1 -- 172.21.15.40:0/1641340916 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e500a5be0 con 0x7f8e580d0fe0 2024-03-24T00:38:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e5cc00640 1 -- 172.21.15.40:0/1641340916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e580d0fe0 msgr2=0x7f8e580d1470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e5cc00640 1 --2- 172.21.15.40:0/1641340916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e580d0fe0 0x7f8e580d1470 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f8e5815fe60 tx=0x7f8e5001c9f0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e5cc00640 1 -- 172.21.15.40:0/1641340916 shutdown_connections 2024-03-24T00:38:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e5cc00640 1 --2- 172.21.15.40:0/1641340916 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e580d0fe0 0x7f8e580d1470 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e5cc00640 1 --2- 172.21.15.40:0/1641340916 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5815bd40 0x7f8e580712a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e5cc00640 1 --2- 172.21.15.40:0/1641340916 >> v2:172.21.15.53:3300/0 conn(0x7f8e5815b960 0x7f8e58070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e5cc00640 1 -- 172.21.15.40:0/1641340916 >> 172.21.15.40:0/1641340916 conn(0x7f8e5806ecf0 msgr2=0x7f8e5806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:51.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e5cc00640 1 -- 172.21.15.40:0/1641340916 shutdown_connections 2024-03-24T00:38:51.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e5cc00640 1 -- 172.21.15.40:0/1641340916 wait complete. 2024-03-24T00:38:51.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e5cc00640 1 Processor -- start 2024-03-24T00:38:51.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.945+0000 7f8e5cc00640 1 -- start start 2024-03-24T00:38:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e5cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e580d0fe0 0x7f8e581491f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e5cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5815b960 0x7f8e581558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e5cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e5815bd40 0x7f8e5806ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e5cc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580d0340 con 0x7f8e580d0fe0 2024-03-24T00:38:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e5cc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580d01c0 con 0x7f8e5815bd40 2024-03-24T00:38:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e5cc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580d04c0 con 0x7f8e5815b960 2024-03-24T00:38:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e55e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e580d0fe0 0x7f8e581491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e55e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e580d0fe0 0x7f8e581491f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42438/0 (socket says 172.21.15.40:42438) 2024-03-24T00:38:51.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e55e00640 1 -- 172.21.15.40:0/2431894319 learned_addr learned my addr 172.21.15.40:0/2431894319 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:51.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e55400640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5815b960 0x7f8e581558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e56800640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e5815bd40 0x7f8e5806ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e55e00640 1 -- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5815b960 msgr2=0x7f8e581558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e55e00640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5815b960 0x7f8e581558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e55e00640 1 -- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e5815bd40 msgr2=0x7f8e5806ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:51.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e55e00640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e5815bd40 0x7f8e5806ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:51.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e55e00640 1 -- 172.21.15.40:0/2431894319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e580c7080 con 0x7f8e580d0fe0 2024-03-24T00:38:51.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e55e00640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e580d0fe0 0x7f8e581491f0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800e930 tx=0x7f8e4800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:51.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e4e000640 1 -- 172.21.15.40:0/2431894319 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48017070 con 0x7f8e580d0fe0 2024-03-24T00:38:51.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e4e000640 1 -- 172.21.15.40:0/2431894319 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4800c9a0 con 0x7f8e580d0fe0 2024-03-24T00:38:51.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e4e000640 1 -- 172.21.15.40:0/2431894319 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4800cd50 con 0x7f8e580d0fe0 2024-03-24T00:38:51.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e5cc00640 1 -- 172.21.15.40:0/2431894319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e580c7360 con 0x7f8e580d0fe0 2024-03-24T00:38:51.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e5cc00640 1 -- 172.21.15.40:0/2431894319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e580c7920 con 0x7f8e580d0fe0 2024-03-24T00:38:51.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.949+0000 7f8e5cc00640 1 -- 172.21.15.40:0/2431894319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e58068170 con 0x7f8e580d0fe0 2024-03-24T00:38:51.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.953+0000 7f8e4e000640 1 -- 172.21.15.40:0/2431894319 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8e4801a070 con 0x7f8e580d0fe0 2024-03-24T00:38:51.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.953+0000 7f8e4e000640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8e38067660 0x7f8e38069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:51.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.953+0000 7f8e4e000640 1 -- 172.21.15.40:0/2431894319 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8e48085c50 con 0x7f8e580d0fe0 2024-03-24T00:38:51.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.953+0000 7f8e55400640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8e38067660 0x7f8e38069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:51.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.953+0000 7f8e55400640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8e38067660 0x7f8e38069b20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f8e5806d170 tx=0x7f8e44009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:51.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:51.957+0000 7f8e4e000640 1 -- 172.21.15.40:0/2431894319 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8e48056260 con 0x7f8e580d0fe0 2024-03-24T00:38:52.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.121+0000 7f8e5cc00640 1 -- 172.21.15.40:0/2431894319 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8e5815b040 con 0x7f8e580d0fe0 2024-03-24T00:38:52.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.121+0000 7f8e4e000640 1 -- 172.21.15.40:0/2431894319 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v108) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e4805a300 con 0x7f8e580d0fe0 2024-03-24T00:38:52.125 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:38:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.121+0000 7f8e5cc00640 1 -- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8e38067660 msgr2=0x7f8e38069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.121+0000 7f8e5cc00640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8e38067660 0x7f8e38069b20 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f8e5806d170 tx=0x7f8e44009190 comp rx=0 tx=0).stop 2024-03-24T00:38:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.121+0000 7f8e5cc00640 1 -- 172.21.15.40:0/2431894319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e580d0fe0 msgr2=0x7f8e581491f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.121+0000 7f8e5cc00640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e580d0fe0 0x7f8e581491f0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800e930 tx=0x7f8e4800edf0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.121+0000 7f8e5cc00640 1 -- 172.21.15.40:0/2431894319 shutdown_connections 2024-03-24T00:38:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.121+0000 7f8e5cc00640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8e38067660 0x7f8e38069b20 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.121+0000 7f8e5cc00640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e5815bd40 0x7f8e5806ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.121+0000 7f8e5cc00640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8e5815b960 0x7f8e581558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.121+0000 7f8e5cc00640 1 --2- 172.21.15.40:0/2431894319 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8e580d0fe0 0x7f8e581491f0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.121+0000 7f8e5cc00640 1 -- 172.21.15.40:0/2431894319 >> 172.21.15.40:0/2431894319 conn(0x7f8e5806ecf0 msgr2=0x7f8e58152d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.125+0000 7f8e5cc00640 1 -- 172.21.15.40:0/2431894319 shutdown_connections 2024-03-24T00:38:52.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.125+0000 7f8e5cc00640 1 -- 172.21.15.40:0/2431894319 wait complete. 2024-03-24T00:38:52.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:38:52.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.285+0000 7f8a2ee00640 1 Processor -- start 2024-03-24T00:38:52.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.285+0000 7f8a2ee00640 1 -- start start 2024-03-24T00:38:52.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2ee00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a2815f5b0 0x7f8a280cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2ee00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8a280cc4c0 0x7f8a280d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2ee00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a280b33e0 con 0x7f8a280b2730 2024-03-24T00:38:52.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2ee00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a280afa90 con 0x7f8a280cc4c0 2024-03-24T00:38:52.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2ee00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280b22f0 con 0x7f8a2815f5b0 2024-03-24T00:38:52.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2e400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f8a280b2730 0x7f8a28164b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45298/0 (socket says 172.21.15.40:45298) 2024-03-24T00:38:52.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2e400640 1 -- 172.21.15.40:0/4216610435 learned_addr learned my addr 172.21.15.40:0/4216610435 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:52.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2d000640 1 --2- 172.21.15.40:0/4216610435 >> v2:172.21.15.53:3300/0 conn(0x7f8a280cc4c0 0x7f8a280d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2da00640 1 --2- 172.21.15.40:0/4216610435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a2815f5b0 0x7f8a280cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a27e00640 1 -- 172.21.15.40:0/4216610435 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714295345 0 0) 0x7f8a280b33e0 con 0x7f8a280b2730 2024-03-24T00:38:52.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a27e00640 1 -- 172.21.15.40:0/4216610435 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a10003530 con 0x7f8a280b2730 2024-03-24T00:38:52.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2d000640 1 -- 172.21.15.40:0/4216610435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a2815f5b0 msgr2=0x7f8a280cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:52.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2d000640 1 --2- 172.21.15.40:0/4216610435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a2815f5b0 0x7f8a280cfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2d000640 1 -- 172.21.15.40:0/4216610435 >> v1:172.21.15.40:6789/0 conn(0x7f8a280b2730 legacy=0x7f8a28164b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.289+0000 7f8a2d000640 1 -- 172.21.15.40:0/4216610435 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a280af440 con 0x7f8a280cc4c0 2024-03-24T00:38:52.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2d000640 1 --2- 172.21.15.40:0/4216610435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a280cc4c0 0x7f8a280d0040 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8a140097c0 tx=0x7f8a1401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c51297977b0f162 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a27e00640 1 -- 172.21.15.40:0/4216610435 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a140b0070 con 0x7f8a280cc4c0 2024-03-24T00:38:52.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a27e00640 1 -- 172.21.15.40:0/4216610435 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a140ab600 con 0x7f8a280cc4c0 2024-03-24T00:38:52.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a27e00640 1 -- 172.21.15.40:0/4216610435 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a140ab9d0 con 0x7f8a280cc4c0 2024-03-24T00:38:52.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 -- 172.21.15.40:0/4216610435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a280cc4c0 msgr2=0x7f8a280d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 --2- 172.21.15.40:0/4216610435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a280cc4c0 0x7f8a280d0040 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8a140097c0 tx=0x7f8a1401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 -- 172.21.15.40:0/4216610435 shutdown_connections 2024-03-24T00:38:52.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 --2- 172.21.15.40:0/4216610435 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a280cc4c0 0x7f8a280d0040 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 --2- 172.21.15.40:0/4216610435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a2815f5b0 0x7f8a280cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 -- 172.21.15.40:0/4216610435 >> 172.21.15.40:0/4216610435 conn(0x7f8a280b3620 msgr2=0x7f8a280b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 -- 172.21.15.40:0/4216610435 shutdown_connections 2024-03-24T00:38:52.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 -- 172.21.15.40:0/4216610435 wait complete. 2024-03-24T00:38:52.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 Processor -- start 2024-03-24T00:38:52.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 -- start start 2024-03-24T00:38:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a280b2730 0x7f8a2815c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a280cc4c0 0x7f8a28154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2815f5b0 0x7f8a281fa690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a28168c90 con 0x7f8a2815f5b0 2024-03-24T00:38:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280d0990 con 0x7f8a280cc4c0 2024-03-24T00:38:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2ee00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a28168e10 con 0x7f8a280b2730 2024-03-24T00:38:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.293+0000 7f8a2e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2815f5b0 0x7f8a281fa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2da00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a280b2730 0x7f8a2815c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2da00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a280b2730 0x7f8a2815c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:56282/0 (socket says 172.21.15.40:56282) 2024-03-24T00:38:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a280cc4c0 0x7f8a28154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2e400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2815f5b0 0x7f8a281fa690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42450/0 (socket says 172.21.15.40:42450) 2024-03-24T00:38:52.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2da00640 1 -- 172.21.15.40:0/567398401 learned_addr learned my addr 172.21.15.40:0/567398401 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:52.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2e400640 1 -- 172.21.15.40:0/567398401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a280b2730 msgr2=0x7f8a2815c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2e400640 1 --2- 172.21.15.40:0/567398401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a280b2730 0x7f8a2815c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2e400640 1 -- 172.21.15.40:0/567398401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a280cc4c0 msgr2=0x7f8a28154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2e400640 1 --2- 172.21.15.40:0/567398401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a280cc4c0 0x7f8a28154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2e400640 1 -- 172.21.15.40:0/567398401 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a28155560 con 0x7f8a2815f5b0 2024-03-24T00:38:52.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2e400640 1 --2- 172.21.15.40:0/567398401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2815f5b0 0x7f8a281fa690 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f8a2000cab0 tx=0x7f8a2000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a25600640 1 -- 172.21.15.40:0/567398401 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20017070 con 0x7f8a2815f5b0 2024-03-24T00:38:52.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a25600640 1 -- 172.21.15.40:0/567398401 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a2000ac30 con 0x7f8a2815f5b0 2024-03-24T00:38:52.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a25600640 1 -- 172.21.15.40:0/567398401 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20012600 con 0x7f8a2815f5b0 2024-03-24T00:38:52.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2ee00640 1 -- 172.21.15.40:0/567398401 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2815c9a0 con 0x7f8a2815f5b0 2024-03-24T00:38:52.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2ee00640 1 -- 172.21.15.40:0/567398401 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2815cf60 con 0x7f8a2815f5b0 2024-03-24T00:38:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a2ee00640 1 -- 172.21.15.40:0/567398401 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2804f410 con 0x7f8a2815f5b0 2024-03-24T00:38:52.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a25600640 1 -- 172.21.15.40:0/567398401 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8a2001a070 con 0x7f8a2815f5b0 2024-03-24T00:38:52.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.297+0000 7f8a25600640 1 --2- 172.21.15.40:0/567398401 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a08067690 0x7f8a08069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.301+0000 7f8a25600640 1 -- 172.21.15.40:0/567398401 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8a20085560 con 0x7f8a2815f5b0 2024-03-24T00:38:52.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.301+0000 7f8a2da00640 1 --2- 172.21.15.40:0/567398401 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a08067690 0x7f8a08069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.301+0000 7f8a2da00640 1 --2- 172.21.15.40:0/567398401 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a08067690 0x7f8a08069b50 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009b80 tx=0x7f8a1c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.305+0000 7f8a25600640 1 -- 172.21.15.40:0/567398401 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8a2000ae20 con 0x7f8a2815f5b0 2024-03-24T00:38:52.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.465+0000 7f8a2ee00640 1 -- 172.21.15.40:0/567398401 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a280c88d0 con 0x7f8a2815f5b0 2024-03-24T00:38:52.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.465+0000 7f8a25600640 1 -- 172.21.15.40:0/567398401 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v108) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a20055ab0 con 0x7f8a2815f5b0 2024-03-24T00:38:52.470 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:38:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.469+0000 7f8a2ee00640 1 -- 172.21.15.40:0/567398401 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a08067690 msgr2=0x7f8a08069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.469+0000 7f8a2ee00640 1 --2- 172.21.15.40:0/567398401 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a08067690 0x7f8a08069b50 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009b80 tx=0x7f8a1c009190 comp rx=0 tx=0).stop 2024-03-24T00:38:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.469+0000 7f8a2ee00640 1 -- 172.21.15.40:0/567398401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2815f5b0 msgr2=0x7f8a281fa690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.469+0000 7f8a2ee00640 1 --2- 172.21.15.40:0/567398401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2815f5b0 0x7f8a281fa690 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f8a2000cab0 tx=0x7f8a2000cf70 comp rx=0 tx=0).stop 2024-03-24T00:38:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.469+0000 7f8a2ee00640 1 -- 172.21.15.40:0/567398401 shutdown_connections 2024-03-24T00:38:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.469+0000 7f8a2ee00640 1 --2- 172.21.15.40:0/567398401 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a08067690 0x7f8a08069b50 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.469+0000 7f8a2ee00640 1 --2- 172.21.15.40:0/567398401 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a2815f5b0 0x7f8a281fa690 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.469+0000 7f8a2ee00640 1 --2- 172.21.15.40:0/567398401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a280cc4c0 0x7f8a28154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.469+0000 7f8a2ee00640 1 --2- 172.21.15.40:0/567398401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a280b2730 0x7f8a2815c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.469+0000 7f8a2ee00640 1 -- 172.21.15.40:0/567398401 >> 172.21.15.40:0/567398401 conn(0x7f8a280b3620 msgr2=0x7f8a2806b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.469+0000 7f8a2ee00640 1 -- 172.21.15.40:0/567398401 shutdown_connections 2024-03-24T00:38:52.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.469+0000 7f8a2ee00640 1 -- 172.21.15.40:0/567398401 wait complete. 2024-03-24T00:38:52.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:38:52.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.629+0000 7f9afb200640 1 Processor -- start 2024-03-24T00:38:52.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.629+0000 7f9afb200640 1 -- start start 2024-03-24T00:38:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.629+0000 7f9afb200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9af406b150 0x7f9af40d09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.629+0000 7f9afb200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9af406bc40 0x7f9af4167c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.629+0000 7f9afb200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9af40b14b0 con 0x7f9af40d0f00 2024-03-24T00:38:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.629+0000 7f9afb200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9af40b20c0 con 0x7f9af406bc40 2024-03-24T00:38:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.629+0000 7f9afb200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af40b1790 con 0x7f9af406b150 2024-03-24T00:38:52.635 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.633+0000 7f9aebe00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f9af40d0f00 0x7f9af406b530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45310/0 (socket says 172.21.15.40:45310) 2024-03-24T00:38:52.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.633+0000 7f9aebe00640 1 -- 172.21.15.40:0/2258231724 learned_addr learned my addr 172.21.15.40:0/2258231724 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:52.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.633+0000 7f9af3e00640 1 --2- 172.21.15.40:0/2258231724 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9af406b150 0x7f9af40d09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.633+0000 7f9af9200640 1 --2- 172.21.15.40:0/2258231724 >> v2:172.21.15.53:3300/0 conn(0x7f9af406bc40 0x7f9af4167c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.633+0000 7f9af3400640 1 -- 172.21.15.40:0/2258231724 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522475160 0 0) 0x7f9af40b14b0 con 0x7f9af40d0f00 2024-03-24T00:38:52.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.633+0000 7f9af3400640 1 -- 172.21.15.40:0/2258231724 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ad8003530 con 0x7f9af40d0f00 2024-03-24T00:38:52.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.633+0000 7f9af9200640 1 -- 172.21.15.40:0/2258231724 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9af406b150 msgr2=0x7f9af40d09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:52.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.633+0000 7f9af9200640 1 --2- 172.21.15.40:0/2258231724 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9af406b150 0x7f9af40d09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.633+0000 7f9af9200640 1 -- 172.21.15.40:0/2258231724 >> v1:172.21.15.40:6789/0 conn(0x7f9af40d0f00 legacy=0x7f9af406b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.633+0000 7f9af9200640 1 -- 172.21.15.40:0/2258231724 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af40b1a70 con 0x7f9af406bc40 2024-03-24T00:38:52.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.633+0000 7f9af3e00640 1 --2- 172.21.15.40:0/2258231724 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9af406b150 0x7f9af40d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:52.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.633+0000 7f9af9200640 1 --2- 172.21.15.40:0/2258231724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9af406bc40 0x7f9af4167c10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009d90 tx=0x7f9aec01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=213318a0be5fe53f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9af3400640 1 -- 172.21.15.40:0/2258231724 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec0b2070 con 0x7f9af406bc40 2024-03-24T00:38:52.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9af3400640 1 -- 172.21.15.40:0/2258231724 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec0a5830 con 0x7f9af406bc40 2024-03-24T00:38:52.639 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9af3400640 1 -- 172.21.15.40:0/2258231724 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec0a5c00 con 0x7f9af406bc40 2024-03-24T00:38:52.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 -- 172.21.15.40:0/2258231724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9af406bc40 msgr2=0x7f9af4167c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 --2- 172.21.15.40:0/2258231724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9af406bc40 0x7f9af4167c10 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009d90 tx=0x7f9aec01c970 comp rx=0 tx=0).stop 2024-03-24T00:38:52.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 -- 172.21.15.40:0/2258231724 shutdown_connections 2024-03-24T00:38:52.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 --2- 172.21.15.40:0/2258231724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9af406bc40 0x7f9af4167c10 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 --2- 172.21.15.40:0/2258231724 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9af406b150 0x7f9af40d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 -- 172.21.15.40:0/2258231724 >> 172.21.15.40:0/2258231724 conn(0x7f9af40b36a0 msgr2=0x7f9af40b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 -- 172.21.15.40:0/2258231724 shutdown_connections 2024-03-24T00:38:52.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 -- 172.21.15.40:0/2258231724 wait complete. 2024-03-24T00:38:52.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 Processor -- start 2024-03-24T00:38:52.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 -- start start 2024-03-24T00:38:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9af406b150 0x7f9af40c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9af406bc40 0x7f9af415a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d0f00 0x7f9af40ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af416a970 con 0x7f9af40d0f00 2024-03-24T00:38:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af416aaf0 con 0x7f9af406b150 2024-03-24T00:38:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9afb200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af416ac70 con 0x7f9af406bc40 2024-03-24T00:38:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9af9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d0f00 0x7f9af40ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9af9200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d0f00 0x7f9af40ca920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42452/0 (socket says 172.21.15.40:42452) 2024-03-24T00:38:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9af9200640 1 -- 172.21.15.40:0/150292765 learned_addr learned my addr 172.21.15.40:0/150292765 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9af3e00640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9af406b150 0x7f9af40c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9aebe00640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9af406bc40 0x7f9af415a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9af9200640 1 -- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9af406bc40 msgr2=0x7f9af415a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9af9200640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9af406bc40 0x7f9af415a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9af9200640 1 -- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9af406b150 msgr2=0x7f9af40c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.637+0000 7f9af9200640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9af406b150 0x7f9af40c72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.641+0000 7f9af9200640 1 -- 172.21.15.40:0/150292765 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af40c77e0 con 0x7f9af40d0f00 2024-03-24T00:38:52.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.641+0000 7f9af9200640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d0f00 0x7f9af40ca920 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009d90 tx=0x7f9aec019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.641+0000 7f9af0c00640 1 -- 172.21.15.40:0/150292765 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec0b2050 con 0x7f9af40d0f00 2024-03-24T00:38:52.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.641+0000 7f9af0c00640 1 -- 172.21.15.40:0/150292765 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aec0be070 con 0x7f9af40d0f00 2024-03-24T00:38:52.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.641+0000 7f9afb200640 1 -- 172.21.15.40:0/150292765 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af40c6200 con 0x7f9af40d0f00 2024-03-24T00:38:52.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.641+0000 7f9af0c00640 1 -- 172.21.15.40:0/150292765 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aec0ad890 con 0x7f9af40d0f00 2024-03-24T00:38:52.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.641+0000 7f9afb200640 1 -- 172.21.15.40:0/150292765 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af40c67c0 con 0x7f9af40d0f00 2024-03-24T00:38:52.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.641+0000 7f9afb200640 1 -- 172.21.15.40:0/150292765 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af4068170 con 0x7f9af40d0f00 2024-03-24T00:38:52.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.641+0000 7f9af0c00640 1 -- 172.21.15.40:0/150292765 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f9aec0ada30 con 0x7f9af40d0f00 2024-03-24T00:38:52.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.641+0000 7f9af0c00640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ad0067660 0x7f9ad0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.641+0000 7f9af0c00640 1 -- 172.21.15.40:0/150292765 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9aec1207f0 con 0x7f9af40d0f00 2024-03-24T00:38:52.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.641+0000 7f9af3e00640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ad0067660 0x7f9ad0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.645+0000 7f9af3e00640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ad0067660 0x7f9ad0069b20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4009580 tx=0x7f9ae4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.645+0000 7f9af0c00640 1 -- 172.21.15.40:0/150292765 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f9af40c77e0 con 0x7f9af40d0f00 2024-03-24T00:38:52.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 -- 172.21.15.40:0/150292765 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9af415ad50 con 0x7f9af40d0f00 2024-03-24T00:38:52.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9af0c00640 1 -- 172.21.15.40:0/150292765 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v108) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9aec016560 con 0x7f9af40d0f00 2024-03-24T00:38:52.828 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:38:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 -- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ad0067660 msgr2=0x7f9ad0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ad0067660 0x7f9ad0069b20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f9ae4009580 tx=0x7f9ae4009190 comp rx=0 tx=0).stop 2024-03-24T00:38:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 -- 172.21.15.40:0/150292765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d0f00 msgr2=0x7f9af40ca920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d0f00 0x7f9af40ca920 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f9aec009d90 tx=0x7f9aec019710 comp rx=0 tx=0).stop 2024-03-24T00:38:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 -- 172.21.15.40:0/150292765 shutdown_connections 2024-03-24T00:38:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9ad0067660 0x7f9ad0069b20 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f9af40d0f00 0x7f9af40ca920 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9af406bc40 0x7f9af415a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 --2- 172.21.15.40:0/150292765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9af406b150 0x7f9af40c72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 -- 172.21.15.40:0/150292765 >> 172.21.15.40:0/150292765 conn(0x7f9af40b36a0 msgr2=0x7f9af406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 -- 172.21.15.40:0/150292765 shutdown_connections 2024-03-24T00:38:52.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.825+0000 7f9afb200640 1 -- 172.21.15.40:0/150292765 wait complete. 2024-03-24T00:38:52.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:52.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.985+0000 7f6c38600640 1 Processor -- start 2024-03-24T00:38:52.990 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.985+0000 7f6c38600640 1 -- start start 2024-03-24T00:38:52.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.985+0000 7f6c38600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c300b2490 0x7f6c300cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.985+0000 7f6c38600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c300b2aa0 0x7f6c300d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c38600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c300b2870 con 0x7f6c300d08b0 2024-03-24T00:38:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c38600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3006b6d0 con 0x7f6c300b2490 2024-03-24T00:38:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c38600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c300b3190 con 0x7f6c300b2aa0 2024-03-24T00:38:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c35c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c300b2490 0x7f6c300cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c35c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6c300b2490 0x7f6c300cfe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42468/0 (socket says 172.21.15.40:42468) 2024-03-24T00:38:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c35200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c300b2aa0 0x7f6c300d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c35c00640 1 -- 172.21.15.40:0/1402995350 learned_addr learned my addr 172.21.15.40:0/1402995350 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c36600640 1 --1- 172.21.15.40:0/1402995350 >> v1:172.21.15.40:6789/0 conn(0x7f6c300d08b0 0x7f6c300ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45320/0 (socket says 172.21.15.40:45320) 2024-03-24T00:38:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c35c00640 1 -- 172.21.15.40:0/1402995350 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c300b2aa0 msgr2=0x7f6c300d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c35c00640 1 --2- 172.21.15.40:0/1402995350 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c300b2aa0 0x7f6c300d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c35c00640 1 -- 172.21.15.40:0/1402995350 >> v1:172.21.15.40:6789/0 conn(0x7f6c300d08b0 legacy=0x7f6c300ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c35c00640 1 -- 172.21.15.40:0/1402995350 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c300b3470 con 0x7f6c300b2490 2024-03-24T00:38:52.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c35c00640 1 --2- 172.21.15.40:0/1402995350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c300b2490 0x7f6c300cfe30 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009870 tx=0x7f6c2401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829ea5b970abe400 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:52.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c2fe00640 1 -- 172.21.15.40:0/1402995350 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c240b2070 con 0x7f6c300b2490 2024-03-24T00:38:52.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c2fe00640 1 -- 172.21.15.40:0/1402995350 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c240a9070 con 0x7f6c300b2490 2024-03-24T00:38:52.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.989+0000 7f6c2fe00640 1 -- 172.21.15.40:0/1402995350 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c240ad880 con 0x7f6c300b2490 2024-03-24T00:38:52.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 -- 172.21.15.40:0/1402995350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c300b2490 msgr2=0x7f6c300cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 --2- 172.21.15.40:0/1402995350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c300b2490 0x7f6c300cfe30 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f6c24009870 tx=0x7f6c2401c560 comp rx=0 tx=0).stop 2024-03-24T00:38:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 -- 172.21.15.40:0/1402995350 shutdown_connections 2024-03-24T00:38:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 --2- 172.21.15.40:0/1402995350 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c300b2aa0 0x7f6c300d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 --2- 172.21.15.40:0/1402995350 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c300b2490 0x7f6c300cfe30 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 -- 172.21.15.40:0/1402995350 >> 172.21.15.40:0/1402995350 conn(0x7f6c300d1d10 msgr2=0x7f6c300d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 -- 172.21.15.40:0/1402995350 shutdown_connections 2024-03-24T00:38:52.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 -- 172.21.15.40:0/1402995350 wait complete. 2024-03-24T00:38:52.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 Processor -- start 2024-03-24T00:38:52.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 -- start start 2024-03-24T00:38:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c300b2490 0x7f6c30154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:52.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c300b2aa0 0x7f6c30150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c300d08b0 0x7f6c30150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3016b350 con 0x7f6c300b2490 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3016b1d0 con 0x7f6c300d08b0 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c38600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3016b4d0 con 0x7f6c300b2aa0 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c35c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c300b2490 0x7f6c30154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c35c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c300b2490 0x7f6c30154950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42476/0 (socket says 172.21.15.40:42476) 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c35c00640 1 -- 172.21.15.40:0/4168805814 learned_addr learned my addr 172.21.15.40:0/4168805814 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c35200640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c300b2aa0 0x7f6c30150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c36600640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c300d08b0 0x7f6c30150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c35c00640 1 -- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c300b2aa0 msgr2=0x7f6c30150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c35c00640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c300b2aa0 0x7f6c30150740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c35c00640 1 -- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c300d08b0 msgr2=0x7f6c30150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c35c00640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c300d08b0 0x7f6c30150f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.993+0000 7f6c35c00640 1 -- 172.21.15.40:0/4168805814 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c30155570 con 0x7f6c300b2490 2024-03-24T00:38:53.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.997+0000 7f6c35c00640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c300b2490 0x7f6c30154950 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f6c240b0040 tx=0x7f6c2401ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.997+0000 7f6c2d600640 1 -- 172.21.15.40:0/4168805814 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c240bc020 con 0x7f6c300b2490 2024-03-24T00:38:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.997+0000 7f6c2d600640 1 -- 172.21.15.40:0/4168805814 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c240add60 con 0x7f6c300b2490 2024-03-24T00:38:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.997+0000 7f6c2d600640 1 -- 172.21.15.40:0/4168805814 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c240b2040 con 0x7f6c300b2490 2024-03-24T00:38:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.997+0000 7f6c38600640 1 -- 172.21.15.40:0/4168805814 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c30155850 con 0x7f6c300b2490 2024-03-24T00:38:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.997+0000 7f6c38600640 1 -- 172.21.15.40:0/4168805814 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c30152ea0 con 0x7f6c300b2490 2024-03-24T00:38:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.997+0000 7f6c38600640 1 -- 172.21.15.40:0/4168805814 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c30068170 con 0x7f6c300b2490 2024-03-24T00:38:53.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:52.997+0000 7f6c2d600640 1 -- 172.21.15.40:0/4168805814 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6c240bf020 con 0x7f6c300b2490 2024-03-24T00:38:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.001+0000 7f6c2d600640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6c10067660 0x7f6c10069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.001+0000 7f6c2d600640 1 -- 172.21.15.40:0/4168805814 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6c2411f940 con 0x7f6c300b2490 2024-03-24T00:38:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.001+0000 7f6c35200640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6c10067660 0x7f6c10069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.001+0000 7f6c35200640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6c10067660 0x7f6c10069b20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6c3015e880 tx=0x7f6c1c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:53.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.009+0000 7f6c2d600640 1 -- 172.21.15.40:0/4168805814 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f6c240c6030 con 0x7f6c300b2490 2024-03-24T00:38:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.213+0000 7f6c38600640 1 -- 172.21.15.40:0/4168805814 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c30153070 con 0x7f6c300b2490 2024-03-24T00:38:53.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.213+0000 7f6c2d600640 1 -- 172.21.15.40:0/4168805814 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f6c2411f020 con 0x7f6c300b2490 2024-03-24T00:38:53.219 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:53.219 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":119,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7079,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/755981654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":755981654},{"type":"v1","addr":"172.21.15.40:6835","nonce":755981654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:01.424871+0000","modified":"2024-03-24T00:38:42.207808+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8418,"mds_1":8400,"mds_2":8101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8101":{"gid":8101,"name":"d","rank":2,"incarnation":116,"state":"up:active","state_seq":7,"addr":"172.21.15.53:6836/3877615536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3877615536},{"type":"v1","addr":"172.21.15.53:6836","nonce":3877615536}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8400":{"gid":8400,"name":"c","rank":1,"incarnation":114,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6834/3883043397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":3883043397},{"type":"v1","addr":"172.21.15.40:6834","nonce":3883043397}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8418":{"gid":8418,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/8812937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":8812937},{"type":"v1","addr":"172.21.15.53:6837","nonce":8812937}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8418,"qdb_cluster":[8101,8400,8418]},"id":8}]} 2024-03-24T00:38:53.219 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 119 2024-03-24T00:38:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.217+0000 7f6c38600640 1 -- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6c10067660 msgr2=0x7f6c10069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.217+0000 7f6c38600640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6c10067660 0x7f6c10069b20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f6c3015e880 tx=0x7f6c1c009190 comp rx=0 tx=0).stop 2024-03-24T00:38:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.217+0000 7f6c38600640 1 -- 172.21.15.40:0/4168805814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c300b2490 msgr2=0x7f6c30154950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.217+0000 7f6c38600640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c300b2490 0x7f6c30154950 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f6c240b0040 tx=0x7f6c2401ada0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.217+0000 7f6c38600640 1 -- 172.21.15.40:0/4168805814 shutdown_connections 2024-03-24T00:38:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.217+0000 7f6c38600640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6c10067660 0x7f6c10069b20 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.217+0000 7f6c38600640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c300d08b0 0x7f6c30150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.217+0000 7f6c38600640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c300b2aa0 0x7f6c30150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.217+0000 7f6c38600640 1 --2- 172.21.15.40:0/4168805814 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6c300b2490 0x7f6c30154950 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.217+0000 7f6c38600640 1 -- 172.21.15.40:0/4168805814 >> 172.21.15.40:0/4168805814 conn(0x7f6c300d1d10 msgr2=0x7f6c30070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:53.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.217+0000 7f6c38600640 1 -- 172.21.15.40:0/4168805814 shutdown_connections 2024-03-24T00:38:53.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.217+0000 7f6c38600640 1 -- 172.21.15.40:0/4168805814 wait complete. 2024-03-24T00:38:53.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.377+0000 7fdb5c400640 1 Processor -- start 2024-03-24T00:38:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.377+0000 7fdb5c400640 1 -- start start 2024-03-24T00:38:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.377+0000 7fdb5c400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb540b2490 0x7fdb540cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.377+0000 7fdb5c400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb540d0a80 0x7fdb540cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.377+0000 7fdb5c400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb540b2870 con 0x7fdb540b2aa0 2024-03-24T00:38:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.377+0000 7fdb5c400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5406b6d0 con 0x7fdb540d0a80 2024-03-24T00:38:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.377+0000 7fdb5c400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb540b3190 con 0x7fdb540b2490 2024-03-24T00:38:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb5a400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb540d0a80 0x7fdb540cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb5a400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fdb540d0a80 0x7fdb540cc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42492/0 (socket says 172.21.15.40:42492) 2024-03-24T00:38:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb5a400640 1 -- 172.21.15.40:0/3176651570 learned_addr learned my addr 172.21.15.40:0/3176651570 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:53.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb59a00640 1 --2- 172.21.15.40:0/3176651570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb540b2490 0x7fdb540cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb5a400640 1 -- 172.21.15.40:0/3176651570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb540b2490 msgr2=0x7fdb540cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb5a400640 1 --2- 172.21.15.40:0/3176651570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb540b2490 0x7fdb540cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb5a400640 1 -- 172.21.15.40:0/3176651570 >> v1:172.21.15.53:6789/0 conn(0x7fdb540b2aa0 legacy=0x7fdb540d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb5a400640 1 -- 172.21.15.40:0/3176651570 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb540b3470 con 0x7fdb540d0a80 2024-03-24T00:38:53.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb53e00640 1 -- 172.21.15.40:0/3176651570 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25597681 0 0) 0x7fdb540b2870 con 0x7fdb540b2aa0 2024-03-24T00:38:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb5a400640 1 --2- 172.21.15.40:0/3176651570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb540d0a80 0x7fdb540cc460 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009e50 tx=0x7fdb4c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf4144ea4f46651 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb53e00640 1 -- 172.21.15.40:0/3176651570 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0a5810 con 0x7fdb540d0a80 2024-03-24T00:38:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb53e00640 1 -- 172.21.15.40:0/3176651570 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c0a59b0 con 0x7fdb540d0a80 2024-03-24T00:38:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.381+0000 7fdb53e00640 1 -- 172.21.15.40:0/3176651570 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0a5d80 con 0x7fdb540d0a80 2024-03-24T00:38:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 -- 172.21.15.40:0/3176651570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb540d0a80 msgr2=0x7fdb540cc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 --2- 172.21.15.40:0/3176651570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb540d0a80 0x7fdb540cc460 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009e50 tx=0x7fdb4c01ca30 comp rx=0 tx=0).stop 2024-03-24T00:38:53.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 -- 172.21.15.40:0/3176651570 shutdown_connections 2024-03-24T00:38:53.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 --2- 172.21.15.40:0/3176651570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb540d0a80 0x7fdb540cc460 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 --2- 172.21.15.40:0/3176651570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb540b2490 0x7fdb540cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 -- 172.21.15.40:0/3176651570 >> 172.21.15.40:0/3176651570 conn(0x7fdb540d1d10 msgr2=0x7fdb540d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:53.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 -- 172.21.15.40:0/3176651570 shutdown_connections 2024-03-24T00:38:53.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 -- 172.21.15.40:0/3176651570 wait complete. 2024-03-24T00:38:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 Processor -- start 2024-03-24T00:38:53.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 -- start start 2024-03-24T00:38:53.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb540b2490 0x7fdb54153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb540b2aa0 0x7fdb540c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb540d0a80 0x7fdb54159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb540cd580 con 0x7fdb540b2490 2024-03-24T00:38:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb540cd400 con 0x7fdb540d0a80 2024-03-24T00:38:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5c400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb540cd700 con 0x7fdb540b2aa0 2024-03-24T00:38:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb59a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb540b2490 0x7fdb54153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb59a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb540b2490 0x7fdb54153cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42494/0 (socket says 172.21.15.40:42494) 2024-03-24T00:38:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb59a00640 1 -- 172.21.15.40:0/3815484120 learned_addr learned my addr 172.21.15.40:0/3815484120 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb5a400640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb540d0a80 0x7fdb54159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb59000640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb540b2aa0 0x7fdb540c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb59a00640 1 -- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb540b2aa0 msgr2=0x7fdb540c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb59a00640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb540b2aa0 0x7fdb540c8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb59a00640 1 -- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb540d0a80 msgr2=0x7fdb54159410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb59a00640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb540d0a80 0x7fdb54159410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb59a00640 1 -- 172.21.15.40:0/3815484120 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb540cbda0 con 0x7fdb540b2490 2024-03-24T00:38:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.385+0000 7fdb59a00640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb540b2490 0x7fdb54153cc0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800e870 tx=0x7fdb4800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.389+0000 7fdb51600640 1 -- 172.21.15.40:0/3815484120 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb48017070 con 0x7fdb540b2490 2024-03-24T00:38:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.389+0000 7fdb51600640 1 -- 172.21.15.40:0/3815484120 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4800c940 con 0x7fdb540b2490 2024-03-24T00:38:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.389+0000 7fdb51600640 1 -- 172.21.15.40:0/3815484120 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4800ccf0 con 0x7fdb540b2490 2024-03-24T00:38:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.389+0000 7fdb5c400640 1 -- 172.21.15.40:0/3815484120 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb540cb300 con 0x7fdb540b2490 2024-03-24T00:38:53.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.389+0000 7fdb5c400640 1 -- 172.21.15.40:0/3815484120 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb541545a0 con 0x7fdb540b2490 2024-03-24T00:38:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.389+0000 7fdb5c400640 1 -- 172.21.15.40:0/3815484120 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb54068170 con 0x7fdb540b2490 2024-03-24T00:38:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.389+0000 7fdb51600640 1 -- 172.21.15.40:0/3815484120 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fdb4801a070 con 0x7fdb540b2490 2024-03-24T00:38:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.389+0000 7fdb51600640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb340676b0 0x7fdb34069b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.389+0000 7fdb51600640 1 -- 172.21.15.40:0/3815484120 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdb48085760 con 0x7fdb540b2490 2024-03-24T00:38:53.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.389+0000 7fdb59000640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb340676b0 0x7fdb34069b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.389+0000 7fdb59000640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb340676b0 0x7fdb34069b70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdb541605e0 tx=0x7fdb40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:53.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.397+0000 7fdb51600640 1 -- 172.21.15.40:0/3815484120 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fdb48055d30 con 0x7fdb540b2490 2024-03-24T00:38:53.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.561+0000 7fdb5c400640 1 -- 172.21.15.40:0/3815484120 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb54161040 con 0x7fdb540b2490 2024-03-24T00:38:53.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.561+0000 7fdb51600640 1 -- 172.21.15.40:0/3815484120 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16720 (secure 0 0 0) 0x7fdb48059dd0 con 0x7fdb540b2490 2024-03-24T00:38:53.566 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:53.566 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":108,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:38:50.991073+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:37:59.849300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:00.417342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.1a","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.565+0000 7fdb5c400640 1 -- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb340676b0 msgr2=0x7fdb34069b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.565+0000 7fdb5c400640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb340676b0 0x7fdb34069b70 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdb541605e0 tx=0x7fdb40009040 comp rx=0 tx=0).stop 2024-03-24T00:38:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.565+0000 7fdb5c400640 1 -- 172.21.15.40:0/3815484120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb540b2490 msgr2=0x7fdb54153cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.565+0000 7fdb5c400640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb540b2490 0x7fdb54153cc0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fdb4800e870 tx=0x7fdb4800ec70 comp rx=0 tx=0).stop 2024-03-24T00:38:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.565+0000 7fdb5c400640 1 -- 172.21.15.40:0/3815484120 shutdown_connections 2024-03-24T00:38:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.565+0000 7fdb5c400640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fdb340676b0 0x7fdb34069b70 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.565+0000 7fdb5c400640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdb540d0a80 0x7fdb54159410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.565+0000 7fdb5c400640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdb540b2aa0 0x7fdb540c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.565+0000 7fdb5c400640 1 --2- 172.21.15.40:0/3815484120 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fdb540b2490 0x7fdb54153cc0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.565+0000 7fdb5c400640 1 -- 172.21.15.40:0/3815484120 >> 172.21.15.40:0/3815484120 conn(0x7fdb540d1d10 msgr2=0x7fdb540cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:53.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.565+0000 7fdb5c400640 1 -- 172.21.15.40:0/3815484120 shutdown_connections 2024-03-24T00:38:53.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.565+0000 7fdb5c400640 1 -- 172.21.15.40:0/3815484120 wait complete. 2024-03-24T00:38:53.587 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:38:53.587 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab32e00640 1 Processor -- start 2024-03-24T00:38:53.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab32e00640 1 -- start start 2024-03-24T00:38:53.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab32e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fab2c145380 0x7fab2c070070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab32e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2c06ed30 con 0x7fab2c0d0040 2024-03-24T00:38:53.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab32e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2c06efd0 con 0x7fab2c0b6530 2024-03-24T00:38:53.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab32e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2c06fbe0 con 0x7fab2c145380 2024-03-24T00:38:53.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab31000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fab2c0d0040 0x7fab2c0b5e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45324/0 (socket says 172.21.15.40:45324) 2024-03-24T00:38:53.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab31000640 1 -- 172.21.15.40:0/444213477 learned_addr learned my addr 172.21.15.40:0/444213477 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:53.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab31a00640 1 --2- 172.21.15.40:0/444213477 >> v2:172.21.15.53:3300/0 conn(0x7fab2c145380 0x7fab2c070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab2be00640 1 -- 172.21.15.40:0/444213477 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466717095 0 0) 0x7fab2c06efd0 con 0x7fab2c0b6530 2024-03-24T00:38:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab2be00640 1 -- 172.21.15.40:0/444213477 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab14003530 con 0x7fab2c0b6530 2024-03-24T00:38:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab2be00640 1 -- 172.21.15.40:0/444213477 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984831284 0 0) 0x7fab2c06ed30 con 0x7fab2c0d0040 2024-03-24T00:38:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab2be00640 1 -- 172.21.15.40:0/444213477 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab2c06efd0 con 0x7fab2c0d0040 2024-03-24T00:38:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab31a00640 1 -- 172.21.15.40:0/444213477 >> v1:172.21.15.53:6789/0 conn(0x7fab2c0b6530 legacy=0x7fab2c15c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab31a00640 1 -- 172.21.15.40:0/444213477 >> v1:172.21.15.40:6789/0 conn(0x7fab2c0d0040 legacy=0x7fab2c0b5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.725+0000 7fab31a00640 1 -- 172.21.15.40:0/444213477 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c06f2b0 con 0x7fab2c145380 2024-03-24T00:38:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab31a00640 1 --2- 172.21.15.40:0/444213477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab2c145380 0x7fab2c070070 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fab200097a0 tx=0x7fab2001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b537610d38aebd7a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab2be00640 1 -- 172.21.15.40:0/444213477 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab200b0070 con 0x7fab2c145380 2024-03-24T00:38:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab2be00640 1 -- 172.21.15.40:0/444213477 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab200ab600 con 0x7fab2c145380 2024-03-24T00:38:53.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab2be00640 1 -- 172.21.15.40:0/444213477 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab200ab9b0 con 0x7fab2c145380 2024-03-24T00:38:53.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab32e00640 1 -- 172.21.15.40:0/444213477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab2c145380 msgr2=0x7fab2c070070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab32e00640 1 --2- 172.21.15.40:0/444213477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab2c145380 0x7fab2c070070 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fab200097a0 tx=0x7fab2001c3c0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab32e00640 1 -- 172.21.15.40:0/444213477 shutdown_connections 2024-03-24T00:38:53.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab32e00640 1 --2- 172.21.15.40:0/444213477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab2c145380 0x7fab2c070070 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.733 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab32e00640 1 -- 172.21.15.40:0/444213477 >> 172.21.15.40:0/444213477 conn(0x7fab2c071a40 msgr2=0x7fab2c071e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:53.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab32e00640 1 -- 172.21.15.40:0/444213477 shutdown_connections 2024-03-24T00:38:53.734 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab32e00640 1 -- 172.21.15.40:0/444213477 wait complete. 2024-03-24T00:38:53.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab32e00640 1 Processor -- start 2024-03-24T00:38:53.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.729+0000 7fab32e00640 1 -- start start 2024-03-24T00:38:53.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab32e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab2c0b6530 0x7fab2c0ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab32e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab2c0d0040 0x7fab2c143eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab32e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab2c145380 0x7fab2c1f6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab32e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c169850 con 0x7fab2c0b6530 2024-03-24T00:38:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab32e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c145760 con 0x7fab2c145380 2024-03-24T00:38:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab32e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c1699d0 con 0x7fab2c0d0040 2024-03-24T00:38:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab31a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab2c0b6530 0x7fab2c0ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab31000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab2c0d0040 0x7fab2c143eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab32400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab2c145380 0x7fab2c1f6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab31a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab2c0b6530 0x7fab2c0ae6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42500/0 (socket says 172.21.15.40:42500) 2024-03-24T00:38:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab31a00640 1 -- 172.21.15.40:0/1789898570 learned_addr learned my addr 172.21.15.40:0/1789898570 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab31a00640 1 -- 172.21.15.40:0/1789898570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab2c0d0040 msgr2=0x7fab2c143eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab31a00640 1 --2- 172.21.15.40:0/1789898570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab2c0d0040 0x7fab2c143eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab31a00640 1 -- 172.21.15.40:0/1789898570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab2c145380 msgr2=0x7fab2c1f6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab31a00640 1 --2- 172.21.15.40:0/1789898570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab2c145380 0x7fab2c1f6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab31a00640 1 -- 172.21.15.40:0/1789898570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c0b4140 con 0x7fab2c0b6530 2024-03-24T00:38:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab31a00640 1 --2- 172.21.15.40:0/1789898570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab2c0b6530 0x7fab2c0ae6c0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fab20000c10 tx=0x7fab20019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab29600640 1 -- 172.21.15.40:0/1789898570 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab200b0040 con 0x7fab2c0b6530 2024-03-24T00:38:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab29600640 1 -- 172.21.15.40:0/1789898570 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab200059a0 con 0x7fab2c0b6530 2024-03-24T00:38:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab29600640 1 -- 172.21.15.40:0/1789898570 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab20005d50 con 0x7fab2c0b6530 2024-03-24T00:38:53.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab32e00640 1 -- 172.21.15.40:0/1789898570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2c0b4420 con 0x7fab2c0b6530 2024-03-24T00:38:53.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab32e00640 1 -- 172.21.15.40:0/1789898570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2c0b49e0 con 0x7fab2c0b6530 2024-03-24T00:38:53.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.733+0000 7fab32e00640 1 -- 172.21.15.40:0/1789898570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab2c0733f0 con 0x7fab2c0b6530 2024-03-24T00:38:53.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.737+0000 7fab29600640 1 -- 172.21.15.40:0/1789898570 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fab200bc060 con 0x7fab2c0b6530 2024-03-24T00:38:53.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.737+0000 7fab29600640 1 --2- 172.21.15.40:0/1789898570 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab0c067660 0x7fab0c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:53.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.737+0000 7fab29600640 1 -- 172.21.15.40:0/1789898570 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fab2011e7d0 con 0x7fab2c0b6530 2024-03-24T00:38:53.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.737+0000 7fab31000640 1 --2- 172.21.15.40:0/1789898570 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab0c067660 0x7fab0c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:53.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.737+0000 7fab31000640 1 --2- 172.21.15.40:0/1789898570 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab0c067660 0x7fab0c069b20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0cc9a0 tx=0x7fab18009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:53.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.741+0000 7fab29600640 1 -- 172.21.15.40:0/1789898570 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fab20016070 con 0x7fab2c0b6530 2024-03-24T00:38:53.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.937+0000 7fab32e00640 1 -- 172.21.15.40:0/1789898570 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab2c04f4f0 con 0x7fab2c0b6530 2024-03-24T00:38:53.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab29600640 1 -- 172.21.15.40:0/1789898570 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+5016 (secure 0 0 0) 0x7fab200eec90 con 0x7fab2c0b6530 2024-03-24T00:38:53.943 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:53.943 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":119,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7079,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/755981654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":755981654},{"type":"v1","addr":"172.21.15.40:6835","nonce":755981654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:01.424871+0000","modified":"2024-03-24T00:38:42.207808+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8418,"mds_1":8400,"mds_2":8101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8101":{"gid":8101,"name":"d","rank":2,"incarnation":116,"state":"up:active","state_seq":7,"addr":"172.21.15.53:6836/3877615536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3877615536},{"type":"v1","addr":"172.21.15.53:6836","nonce":3877615536}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8400":{"gid":8400,"name":"c","rank":1,"incarnation":114,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6834/3883043397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":3883043397},{"type":"v1","addr":"172.21.15.40:6834","nonce":3883043397}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8418":{"gid":8418,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/8812937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":8812937},{"type":"v1","addr":"172.21.15.53:6837","nonce":8812937}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8418,"qdb_cluster":[8101,8400,8418]},"id":8}]} 2024-03-24T00:38:53.944 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 119 2024-03-24T00:38:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab32e00640 1 -- 172.21.15.40:0/1789898570 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab0c067660 msgr2=0x7fab0c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab32e00640 1 --2- 172.21.15.40:0/1789898570 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab0c067660 0x7fab0c069b20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0cc9a0 tx=0x7fab18009040 comp rx=0 tx=0).stop 2024-03-24T00:38:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab32e00640 1 -- 172.21.15.40:0/1789898570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab2c0b6530 msgr2=0x7fab2c0ae6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab32e00640 1 --2- 172.21.15.40:0/1789898570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab2c0b6530 0x7fab2c0ae6c0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fab20000c10 tx=0x7fab20019160 comp rx=0 tx=0).stop 2024-03-24T00:38:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab32e00640 1 -- 172.21.15.40:0/1789898570 shutdown_connections 2024-03-24T00:38:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab32e00640 1 --2- 172.21.15.40:0/1789898570 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab0c067660 0x7fab0c069b20 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab32e00640 1 --2- 172.21.15.40:0/1789898570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab2c145380 0x7fab2c1f6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab32e00640 1 --2- 172.21.15.40:0/1789898570 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab2c0d0040 0x7fab2c143eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab32e00640 1 --2- 172.21.15.40:0/1789898570 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab2c0b6530 0x7fab2c0ae6c0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab32e00640 1 -- 172.21.15.40:0/1789898570 >> 172.21.15.40:0/1789898570 conn(0x7fab2c071a40 msgr2=0x7fab2c15e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab32e00640 1 -- 172.21.15.40:0/1789898570 shutdown_connections 2024-03-24T00:38:53.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:53.941+0000 7fab32e00640 1 -- 172.21.15.40:0/1789898570 wait complete. 2024-03-24T00:38:53.965 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:38:54.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.105+0000 7f24c8a00640 1 Processor -- start 2024-03-24T00:38:54.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.105+0000 7f24c8a00640 1 -- start start 2024-03-24T00:38:54.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.105+0000 7f24c8a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24c406f4e0 0x7f24c4070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.105+0000 7f24c8a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f24c4070030 0x7f24c40af170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.105+0000 7f24c8a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c40d27b0 con 0x7f24c406f100 2024-03-24T00:38:54.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.105+0000 7f24c8a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24c4070460 con 0x7f24c4070030 2024-03-24T00:38:54.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.105+0000 7f24c8a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c40b1490 con 0x7f24c406f4e0 2024-03-24T00:38:54.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.109+0000 7f24c2800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f24c4070030 0x7f24c40af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.109+0000 7f24c2800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f24c4070030 0x7f24c40af170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42514/0 (socket says 172.21.15.40:42514) 2024-03-24T00:38:54.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.109+0000 7f24bbe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24c406f4e0 0x7f24c4070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.109+0000 7f24c2800640 1 -- 172.21.15.40:0/3081098269 learned_addr learned my addr 172.21.15.40:0/3081098269 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:54.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.109+0000 7f24bbe00640 1 -- 172.21.15.40:0/3081098269 >> v2:172.21.15.40:3300/0 conn(0x7f24c4070030 msgr2=0x7f24c40af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:54.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.109+0000 7f24bbe00640 1 --2- 172.21.15.40:0/3081098269 >> v2:172.21.15.40:3300/0 conn(0x7f24c4070030 0x7f24c40af170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.109+0000 7f24bbe00640 1 -- 172.21.15.40:0/3081098269 >> v1:172.21.15.53:6789/0 conn(0x7f24c406f100 legacy=0x7f24c40aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.112 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.109+0000 7f24bbe00640 1 -- 172.21.15.40:0/3081098269 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c406fa70 con 0x7f24c406f4e0 2024-03-24T00:38:54.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.109+0000 7f24bbe00640 1 --2- 172.21.15.40:0/3081098269 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24c406f4e0 0x7f24c4070be0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0097c0 tx=0x7f24ac01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=308f4de45e4926a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.109+0000 7f24c1400640 1 -- 172.21.15.40:0/3081098269 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0b2070 con 0x7f24c406f4e0 2024-03-24T00:38:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.109+0000 7f24c1400640 1 -- 172.21.15.40:0/3081098269 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac0ad640 con 0x7f24c406f4e0 2024-03-24T00:38:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.109+0000 7f24c1400640 1 -- 172.21.15.40:0/3081098269 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0ada10 con 0x7f24c406f4e0 2024-03-24T00:38:54.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 -- 172.21.15.40:0/3081098269 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24c406f4e0 msgr2=0x7f24c4070be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 --2- 172.21.15.40:0/3081098269 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24c406f4e0 0x7f24c4070be0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0097c0 tx=0x7f24ac01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 -- 172.21.15.40:0/3081098269 shutdown_connections 2024-03-24T00:38:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 --2- 172.21.15.40:0/3081098269 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24c406f4e0 0x7f24c4070be0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 --2- 172.21.15.40:0/3081098269 >> v2:172.21.15.40:3300/0 conn(0x7f24c4070030 0x7f24c40af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 -- 172.21.15.40:0/3081098269 >> 172.21.15.40:0/3081098269 conn(0x7f24c40d1c90 msgr2=0x7f24c40d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 -- 172.21.15.40:0/3081098269 shutdown_connections 2024-03-24T00:38:54.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 -- 172.21.15.40:0/3081098269 wait complete. 2024-03-24T00:38:54.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 Processor -- start 2024-03-24T00:38:54.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 -- start start 2024-03-24T00:38:54.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c406f100 0x7f24c40c7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24c406f4e0 0x7f24c40c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24c4070030 0x7f24c4160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c40d0a70 con 0x7f24c406f100 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c40d08f0 con 0x7f24c4070030 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c8a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c40af6b0 con 0x7f24c406f4e0 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c406f100 0x7f24c40c7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c1e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c406f100 0x7f24c40c7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42520/0 (socket says 172.21.15.40:42520) 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c1e00640 1 -- 172.21.15.40:0/4249951883 learned_addr learned my addr 172.21.15.40:0/4249951883 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c2800640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24c4070030 0x7f24c4160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24bbe00640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24c406f4e0 0x7f24c40c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c1e00640 1 -- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24c406f4e0 msgr2=0x7f24c40c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c1e00640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24c406f4e0 0x7f24c40c4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c1e00640 1 -- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24c4070030 msgr2=0x7f24c4160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c1e00640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24c4070030 0x7f24c4160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.113+0000 7f24c1e00640 1 -- 172.21.15.40:0/4249951883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c406cad0 con 0x7f24c406f100 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.117+0000 7f24c1e00640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c406f100 0x7f24c40c7ac0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f24b400c370 tx=0x7f24b400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:54.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.117+0000 7f24ba000640 1 -- 172.21.15.40:0/4249951883 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b40052d0 con 0x7f24c406f100 2024-03-24T00:38:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.117+0000 7f24ba000640 1 -- 172.21.15.40:0/4249951883 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b4005470 con 0x7f24c406f100 2024-03-24T00:38:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.117+0000 7f24c8a00640 1 -- 172.21.15.40:0/4249951883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c40c54e0 con 0x7f24c406f100 2024-03-24T00:38:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.117+0000 7f24ba000640 1 -- 172.21.15.40:0/4249951883 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b4007260 con 0x7f24c406f100 2024-03-24T00:38:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.117+0000 7f24c8a00640 1 -- 172.21.15.40:0/4249951883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c40c8000 con 0x7f24c406f100 2024-03-24T00:38:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.117+0000 7f24c8a00640 1 -- 172.21.15.40:0/4249951883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c4068170 con 0x7f24c406f100 2024-03-24T00:38:54.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.117+0000 7f24ba000640 1 -- 172.21.15.40:0/4249951883 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f24b4007400 con 0x7f24c406f100 2024-03-24T00:38:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.117+0000 7f24ba000640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f24a0067690 0x7f24a0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.117+0000 7f24ba000640 1 -- 172.21.15.40:0/4249951883 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f24b4084c90 con 0x7f24c406f100 2024-03-24T00:38:54.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.117+0000 7f24bbe00640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f24a0067690 0x7f24a0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.121+0000 7f24bbe00640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f24a0067690 0x7f24a0069b50 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0af000 tx=0x7f24ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:54.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.121+0000 7f24ba000640 1 -- 172.21.15.40:0/4249951883 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f24b4055270 con 0x7f24c406f100 2024-03-24T00:38:54.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.321+0000 7f24c8a00640 1 -- 172.21.15.40:0/4249951883 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24c40b5880 con 0x7f24c406f100 2024-03-24T00:38:54.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.321+0000 7f24ba000640 1 -- 172.21.15.40:0/4249951883 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+5016 (secure 0 0 0) 0x7f24b4059310 con 0x7f24c406f100 2024-03-24T00:38:54.326 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:54.326 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":119,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7079,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/755981654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":755981654},{"type":"v1","addr":"172.21.15.40:6835","nonce":755981654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:38:01.424871+0000","modified":"2024-03-24T00:38:42.207808+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8418,"mds_1":8400,"mds_2":8101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8101":{"gid":8101,"name":"d","rank":2,"incarnation":116,"state":"up:active","state_seq":7,"addr":"172.21.15.53:6836/3877615536","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3877615536},{"type":"v1","addr":"172.21.15.53:6836","nonce":3877615536}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8400":{"gid":8400,"name":"c","rank":1,"incarnation":114,"state":"up:active","state_seq":7,"addr":"172.21.15.40:6834/3883043397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":3883043397},{"type":"v1","addr":"172.21.15.40:6834","nonce":3883043397}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8418":{"gid":8418,"name":"b","rank":0,"incarnation":108,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6837/8812937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":8812937},{"type":"v1","addr":"172.21.15.53:6837","nonce":8812937}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8418,"qdb_cluster":[8101,8400,8418]},"id":8}]} 2024-03-24T00:38:54.326 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 119 2024-03-24T00:38:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.325+0000 7f24c8a00640 1 -- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f24a0067690 msgr2=0x7f24a0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.325+0000 7f24c8a00640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f24a0067690 0x7f24a0069b50 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0af000 tx=0x7f24ac008040 comp rx=0 tx=0).stop 2024-03-24T00:38:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.325+0000 7f24c8a00640 1 -- 172.21.15.40:0/4249951883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c406f100 msgr2=0x7f24c40c7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.325+0000 7f24c8a00640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c406f100 0x7f24c40c7ac0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f24b400c370 tx=0x7f24b400c830 comp rx=0 tx=0).stop 2024-03-24T00:38:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.325+0000 7f24c8a00640 1 -- 172.21.15.40:0/4249951883 shutdown_connections 2024-03-24T00:38:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.325+0000 7f24c8a00640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f24a0067690 0x7f24a0069b50 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.325+0000 7f24c8a00640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24c4070030 0x7f24c4160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.325+0000 7f24c8a00640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24c406f4e0 0x7f24c40c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.325+0000 7f24c8a00640 1 --2- 172.21.15.40:0/4249951883 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f24c406f100 0x7f24c40c7ac0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.325+0000 7f24c8a00640 1 -- 172.21.15.40:0/4249951883 >> 172.21.15.40:0/4249951883 conn(0x7f24c40d1c90 msgr2=0x7f24c40afa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.325+0000 7f24c8a00640 1 -- 172.21.15.40:0/4249951883 shutdown_connections 2024-03-24T00:38:54.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.325+0000 7f24c8a00640 1 -- 172.21.15.40:0/4249951883 wait complete. 2024-03-24T00:38:54.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:38:54.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc22600640 1 Processor -- start 2024-03-24T00:38:54.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc22600640 1 -- start start 2024-03-24T00:38:54.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc22600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcc1c0cfdf0 0x7fcc1c0d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc22600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc1c15f7a0 0x7fcc1c0d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc22600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1c0af710 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc22600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1c0b2bb0 con 0x7fcc1c15f7a0 2024-03-24T00:38:54.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc22600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1c0b26a0 con 0x7fcc1c0cfdf0 2024-03-24T00:38:54.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc1be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc1c15f7a0 0x7fcc1c0d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc1be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcc1c15f7a0 0x7fcc1c0d0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42530/0 (socket says 172.21.15.40:42530) 2024-03-24T00:38:54.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc1be00640 1 -- 172.21.15.40:0/1319598073 learned_addr learned my addr 172.21.15.40:0/1319598073 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:54.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc1b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcc1c0cfdf0 0x7fcc1c0d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc1be00640 1 -- 172.21.15.40:0/1319598073 >> v2:172.21.15.53:3300/0 conn(0x7fcc1c0cfdf0 msgr2=0x7fcc1c0d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:54.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc1be00640 1 --2- 172.21.15.40:0/1319598073 >> v2:172.21.15.53:3300/0 conn(0x7fcc1c0cfdf0 0x7fcc1c0d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc1be00640 1 -- 172.21.15.40:0/1319598073 >> v1:172.21.15.53:6789/0 conn(0x7fcc1c0b43a0 legacy=0x7fcc1c15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.485+0000 7fcc1be00640 1 -- 172.21.15.40:0/1319598073 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c0b34f0 con 0x7fcc1c15f7a0 2024-03-24T00:38:54.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.489+0000 7fcc1be00640 1 --2- 172.21.15.40:0/1319598073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc1c15f7a0 0x7fcc1c0d0710 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c071950 tx=0x7fcc1401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6062c7b3e8c03f7f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:54.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.489+0000 7fcc1aa00640 1 -- 172.21.15.40:0/1319598073 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc140a56c0 con 0x7fcc1c15f7a0 2024-03-24T00:38:54.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.489+0000 7fcc1aa00640 1 -- 172.21.15.40:0/1319598073 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc140a5860 con 0x7fcc1c15f7a0 2024-03-24T00:38:54.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.489+0000 7fcc1aa00640 1 -- 172.21.15.40:0/1319598073 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc140a5c30 con 0x7fcc1c15f7a0 2024-03-24T00:38:54.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.489+0000 7fcc22600640 1 -- 172.21.15.40:0/1319598073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc1c15f7a0 msgr2=0x7fcc1c0d0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.489+0000 7fcc22600640 1 --2- 172.21.15.40:0/1319598073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc1c15f7a0 0x7fcc1c0d0710 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c071950 tx=0x7fcc1401c9f0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.489+0000 7fcc22600640 1 -- 172.21.15.40:0/1319598073 shutdown_connections 2024-03-24T00:38:54.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.489+0000 7fcc22600640 1 --2- 172.21.15.40:0/1319598073 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc1c15f7a0 0x7fcc1c0d0710 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.489+0000 7fcc22600640 1 --2- 172.21.15.40:0/1319598073 >> v2:172.21.15.53:3300/0 conn(0x7fcc1c0cfdf0 0x7fcc1c0d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.489+0000 7fcc22600640 1 -- 172.21.15.40:0/1319598073 >> 172.21.15.40:0/1319598073 conn(0x7fcc1c0b3880 msgr2=0x7fcc1c0b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:54.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.489+0000 7fcc22600640 1 -- 172.21.15.40:0/1319598073 shutdown_connections 2024-03-24T00:38:54.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.489+0000 7fcc22600640 1 -- 172.21.15.40:0/1319598073 wait complete. 2024-03-24T00:38:54.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc22600640 1 Processor -- start 2024-03-24T00:38:54.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc22600640 1 -- start start 2024-03-24T00:38:54.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc22600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc1c0b43a0 0x7fcc1c0c5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc22600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc1c0cfdf0 0x7fcc1c0c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc22600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc1c15f7a0 0x7fcc1c164ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc22600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0b07d0 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc22600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0b0650 con 0x7fcc1c15f7a0 2024-03-24T00:38:54.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc22600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c0b0950 con 0x7fcc1c0cfdf0 2024-03-24T00:38:54.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc1b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc1c0b43a0 0x7fcc1c0c5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc1b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc1c0b43a0 0x7fcc1c0c5350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42538/0 (socket says 172.21.15.40:42538) 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc1b400640 1 -- 172.21.15.40:0/3925326788 learned_addr learned my addr 172.21.15.40:0/3925326788 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc13e00640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc1c0cfdf0 0x7fcc1c0c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc1be00640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc1c15f7a0 0x7fcc1c164ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc1b400640 1 -- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc1c0cfdf0 msgr2=0x7fcc1c0c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc1b400640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc1c0cfdf0 0x7fcc1c0c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc1b400640 1 -- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc1c15f7a0 msgr2=0x7fcc1c164ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc1b400640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc1c15f7a0 0x7fcc1c164ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc1b400640 1 -- 172.21.15.40:0/3925326788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c06b150 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc1b400640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc1c0b43a0 0x7fcc1c0c5350 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00e930 tx=0x7fcc0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc13400640 1 -- 172.21.15.40:0/3925326788 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c017070 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc13400640 1 -- 172.21.15.40:0/3925326788 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0c00ca00 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc13400640 1 -- 172.21.15.40:0/3925326788 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00cdb0 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc22600640 1 -- 172.21.15.40:0/3925326788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1c06b430 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.493+0000 7fcc22600640 1 -- 172.21.15.40:0/3925326788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc1c06b9f0 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.497+0000 7fcc22600640 1 -- 172.21.15.40:0/3925326788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1c068170 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.497+0000 7fcc13400640 1 -- 172.21.15.40:0/3925326788 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcc0c01a070 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.497+0000 7fcc13400640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcbf8067660 0x7fcbf8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.497+0000 7fcc13400640 1 -- 172.21.15.40:0/3925326788 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcc0c085cf0 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.501 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.497+0000 7fcc13e00640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcbf8067660 0x7fcbf8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.497+0000 7fcc13e00640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcbf8067660 0x7fcbf8069b20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0ce870 tx=0x7fcc04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:54.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.501+0000 7fcc13400640 1 -- 172.21.15.40:0/3925326788 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcc0c056270 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.665+0000 7fcc22600640 1 -- 172.21.15.40:0/3925326788 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc1c0cc0c0 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.665+0000 7fcc13400640 1 -- 172.21.15.40:0/3925326788 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16720 (secure 0 0 0) 0x7fcc0c05a310 con 0x7fcc1c0b43a0 2024-03-24T00:38:54.670 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:38:54.670 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":108,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:38:50.991073+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:37:59.849300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-03-24T00:38:00.417342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.9","mappings":[{"from":0,"to":5}]},{"pgid":"17.1a","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:38:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.669+0000 7fcc22600640 1 -- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcbf8067660 msgr2=0x7fcbf8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.669+0000 7fcc22600640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcbf8067660 0x7fcbf8069b20 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0ce870 tx=0x7fcc04009040 comp rx=0 tx=0).stop 2024-03-24T00:38:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.669+0000 7fcc22600640 1 -- 172.21.15.40:0/3925326788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc1c0b43a0 msgr2=0x7fcc1c0c5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.669+0000 7fcc22600640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc1c0b43a0 0x7fcc1c0c5350 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00e930 tx=0x7fcc0c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.669+0000 7fcc22600640 1 -- 172.21.15.40:0/3925326788 shutdown_connections 2024-03-24T00:38:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.669+0000 7fcc22600640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcbf8067660 0x7fcbf8069b20 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.669+0000 7fcc22600640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc1c15f7a0 0x7fcc1c164ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.669+0000 7fcc22600640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc1c0cfdf0 0x7fcc1c0c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.669+0000 7fcc22600640 1 --2- 172.21.15.40:0/3925326788 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc1c0b43a0 0x7fcc1c0c5350 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.669+0000 7fcc22600640 1 -- 172.21.15.40:0/3925326788 >> 172.21.15.40:0/3925326788 conn(0x7fcc1c0b3880 msgr2=0x7fcc1c0b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.669+0000 7fcc22600640 1 -- 172.21.15.40:0/3925326788 shutdown_connections 2024-03-24T00:38:54.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.669+0000 7fcc22600640 1 -- 172.21.15.40:0/3925326788 wait complete. 2024-03-24T00:38:54.691 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T00:38:54.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.829+0000 7ff1e8c00640 1 Processor -- start 2024-03-24T00:38:54.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.829+0000 7ff1e8c00640 1 -- start start 2024-03-24T00:38:54.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.829+0000 7ff1e8c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff1e40b2aa0 0x7ff1e40d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.829+0000 7ff1e8c00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e40b2870 con 0x7ff1e40b2490 2024-03-24T00:38:54.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.829+0000 7ff1e8c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1e40b3190 con 0x7ff1e40d0a80 2024-03-24T00:38:54.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.829+0000 7ff1e8c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1e406b6d0 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.829+0000 7ff1e1e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff1e40b2490 0x7ff1e40cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45332/0 (socket says 172.21.15.40:45332) 2024-03-24T00:38:54.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.829+0000 7ff1e1e00640 1 -- 172.21.15.40:0/766577882 learned_addr learned my addr 172.21.15.40:0/766577882 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:54.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.829+0000 7ff1d9400640 1 --2- 172.21.15.40:0/766577882 >> v2:172.21.15.53:3300/0 conn(0x7ff1e40b2aa0 0x7ff1e40d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.833+0000 7ff1e1400640 1 -- 172.21.15.40:0/766577882 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732227395 0 0) 0x7ff1e40b2870 con 0x7ff1e40b2490 2024-03-24T00:38:54.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.833+0000 7ff1e1400640 1 -- 172.21.15.40:0/766577882 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1c8003530 con 0x7ff1e40b2490 2024-03-24T00:38:54.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.833+0000 7ff1e1400640 1 -- 172.21.15.40:0/766577882 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826044175 0 0) 0x7ff1e40b3190 con 0x7ff1e40d0a80 2024-03-24T00:38:54.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.833+0000 7ff1d9400640 1 -- 172.21.15.40:0/766577882 >> v1:172.21.15.53:6789/0 conn(0x7ff1e40d0a80 legacy=0x7ff1e40cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.833+0000 7ff1d9400640 1 -- 172.21.15.40:0/766577882 >> v1:172.21.15.40:6789/0 conn(0x7ff1e40b2490 legacy=0x7ff1e40cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.833+0000 7ff1d9400640 1 -- 172.21.15.40:0/766577882 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e40b3470 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.833+0000 7ff1d9400640 1 --2- 172.21.15.40:0/766577882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1e40b2aa0 0x7ff1e40d0540 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0097c0 tx=0x7ff1cc01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48308a2db4becc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:54.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.833+0000 7ff1e1400640 1 -- 172.21.15.40:0/766577882 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc0b1070 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.833+0000 7ff1e1400640 1 -- 172.21.15.40:0/766577882 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1cc0ac410 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.838 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.833+0000 7ff1e1400640 1 -- 172.21.15.40:0/766577882 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc0ac7e0 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.839 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 -- 172.21.15.40:0/766577882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1e40b2aa0 msgr2=0x7ff1e40d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 --2- 172.21.15.40:0/766577882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1e40b2aa0 0x7ff1e40d0540 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0097c0 tx=0x7ff1cc01c4a0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 -- 172.21.15.40:0/766577882 shutdown_connections 2024-03-24T00:38:54.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 --2- 172.21.15.40:0/766577882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1e40b2aa0 0x7ff1e40d0540 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 -- 172.21.15.40:0/766577882 >> 172.21.15.40:0/766577882 conn(0x7ff1e40d1d10 msgr2=0x7ff1e40d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:54.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 -- 172.21.15.40:0/766577882 shutdown_connections 2024-03-24T00:38:54.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 -- 172.21.15.40:0/766577882 wait complete. 2024-03-24T00:38:54.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 Processor -- start 2024-03-24T00:38:54.841 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 -- start start 2024-03-24T00:38:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1e40b2490 0x7ff1e414f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1e40b2aa0 0x7ff1e41511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1e40d0a80 0x7ff1e4157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e416bfd0 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e40cd650 con 0x7ff1e40b2490 2024-03-24T00:38:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e8c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e416c150 con 0x7ff1e40d0a80 2024-03-24T00:38:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1d9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1e40b2aa0 0x7ff1e41511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.842 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1d9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1e40b2aa0 0x7ff1e41511a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42546/0 (socket says 172.21.15.40:42546) 2024-03-24T00:38:54.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1d9400640 1 -- 172.21.15.40:0/2110040616 learned_addr learned my addr 172.21.15.40:0/2110040616 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:54.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e1e00640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1e40b2490 0x7ff1e414f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.843 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1e2800640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1e40d0a80 0x7ff1e4157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1d9400640 1 -- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1e40d0a80 msgr2=0x7ff1e4157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1d9400640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1e40d0a80 0x7ff1e4157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1d9400640 1 -- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1e40b2490 msgr2=0x7ff1e414f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1d9400640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1e40b2490 0x7ff1e414f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1d9400640 1 -- 172.21.15.40:0/2110040616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e4152cb0 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.837+0000 7ff1d9400640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1e40b2aa0 0x7ff1e41511a0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0097c0 tx=0x7ff1cc019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.841+0000 7ff1daa00640 1 -- 172.21.15.40:0/2110040616 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc0b1040 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.841+0000 7ff1daa00640 1 -- 172.21.15.40:0/2110040616 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1cc004020 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.841+0000 7ff1daa00640 1 -- 172.21.15.40:0/2110040616 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc004410 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.841+0000 7ff1e8c00640 1 -- 172.21.15.40:0/2110040616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e4151c10 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.841+0000 7ff1e8c00640 1 -- 172.21.15.40:0/2110040616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e41521d0 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.841+0000 7ff1e8c00640 1 -- 172.21.15.40:0/2110040616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e4068170 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.845 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.841+0000 7ff1daa00640 1 -- 172.21.15.40:0/2110040616 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff1cc0be060 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.841+0000 7ff1daa00640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff1c0067690 0x7ff1c0069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:54.846 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.841+0000 7ff1daa00640 1 -- 172.21.15.40:0/2110040616 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff1cc11e680 con 0x7ff1e40b2aa0 2024-03-24T00:38:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.841+0000 7ff1e1e00640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff1c0067690 0x7ff1c0069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:54.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.845+0000 7ff1e1e00640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff1c0067690 0x7ff1c0069b50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4007fd0 tx=0x7ff1d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:54.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:54.845+0000 7ff1daa00640 1 -- 172.21.15.40:0/2110040616 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff1cc016090 con 0x7ff1e40b2aa0 2024-03-24T00:38:55.080 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.077+0000 7ff1e8c00640 1 -- 172.21.15.40:0/2110040616 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff1e40cb6e0 con 0x7ff1e40b2aa0 2024-03-24T00:38:55.080 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-03-24T00:38:55.077+0000 7fa64e600640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T00:38:55.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.077+0000 7ff1daa00640 1 -- 172.21.15.40:0/2110040616 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v120) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff1cc0eebd0 con 0x7ff1e40b2aa0 2024-03-24T00:38:55.082 INFO:teuthology.orchestra.run.smithi040.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T00:38:55.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.081+0000 7ff1e8c00640 1 -- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff1c0067690 msgr2=0x7ff1c0069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.081+0000 7ff1e8c00640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff1c0067690 0x7ff1c0069b50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4007fd0 tx=0x7ff1d4009040 comp rx=0 tx=0).stop 2024-03-24T00:38:55.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.081+0000 7ff1e8c00640 1 -- 172.21.15.40:0/2110040616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1e40b2aa0 msgr2=0x7ff1e41511a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.081+0000 7ff1e8c00640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1e40b2aa0 0x7ff1e41511a0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0097c0 tx=0x7ff1cc019240 comp rx=0 tx=0).stop 2024-03-24T00:38:55.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.081+0000 7ff1e8c00640 1 -- 172.21.15.40:0/2110040616 shutdown_connections 2024-03-24T00:38:55.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.081+0000 7ff1e8c00640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff1c0067690 0x7ff1c0069b50 secure :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4007fd0 tx=0x7ff1d4009040 comp rx=0 tx=0).stop 2024-03-24T00:38:55.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.081+0000 7ff1e8c00640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff1e40d0a80 0x7ff1e4157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.081+0000 7ff1e8c00640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff1e40b2aa0 0x7ff1e41511a0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.081+0000 7ff1e8c00640 1 --2- 172.21.15.40:0/2110040616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1e40b2490 0x7ff1e414f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.081+0000 7ff1e8c00640 1 -- 172.21.15.40:0/2110040616 >> 172.21.15.40:0/2110040616 conn(0x7ff1e40d1d10 msgr2=0x7ff1e40c9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:55.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.081+0000 7ff1e8c00640 1 -- 172.21.15.40:0/2110040616 shutdown_connections 2024-03-24T00:38:55.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.081+0000 7ff1e8c00640 1 -- 172.21.15.40:0/2110040616 wait complete. 2024-03-24T00:38:55.106 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T00:38:55.147 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:38:55.150 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:38:55.164 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:38:55.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee6800640 1 Processor -- start 2024-03-24T00:38:55.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee6800640 1 -- start start 2024-03-24T00:38:55.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee6800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbee00d0640 0x7fbee006b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:55.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee6800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbee006be20 0x7fbee006f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:55.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee6800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbee00d0ad0 con 0x7fbee0161360 2024-03-24T00:38:55.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee6800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee0161740 con 0x7fbee006be20 2024-03-24T00:38:55.258 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee6800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee006eba0 con 0x7fbee00d0640 2024-03-24T00:38:55.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee5e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbee006be20 0x7fbee006f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:55.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee5400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbee0161360 0x7fbee006b710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:45344/0 (socket says 172.21.15.40:45344) 2024-03-24T00:38:55.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee5400640 1 -- 172.21.15.40:0/3855737152 learned_addr learned my addr 172.21.15.40:0/3855737152 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:55.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee4a00640 1 --2- 172.21.15.40:0/3855737152 >> v2:172.21.15.53:3300/0 conn(0x7fbee00d0640 0x7fbee006b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:55.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee5e00640 1 -- 172.21.15.40:0/3855737152 >> v2:172.21.15.53:3300/0 conn(0x7fbee00d0640 msgr2=0x7fbee006b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:55.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee5e00640 1 --2- 172.21.15.40:0/3855737152 >> v2:172.21.15.53:3300/0 conn(0x7fbee00d0640 0x7fbee006b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee5e00640 1 -- 172.21.15.40:0/3855737152 >> v1:172.21.15.40:6789/0 conn(0x7fbee0161360 legacy=0x7fbee006b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbee5e00640 1 -- 172.21.15.40:0/3855737152 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee00d0db0 con 0x7fbee006be20 2024-03-24T00:38:55.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.253+0000 7fbedfe00640 1 -- 172.21.15.40:0/3855737152 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346424864 0 0) 0x7fbee00d0ad0 con 0x7fbee0161360 2024-03-24T00:38:55.261 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.257+0000 7fbee5e00640 1 --2- 172.21.15.40:0/3855737152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbee006be20 0x7fbee006f5f0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fbee0072e90 tx=0x7fbed801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5966741fc50b538d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:55.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.257+0000 7fbedfe00640 1 -- 172.21.15.40:0/3855737152 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed80b2070 con 0x7fbee006be20 2024-03-24T00:38:55.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.257+0000 7fbedfe00640 1 -- 172.21.15.40:0/3855737152 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed80a5900 con 0x7fbee006be20 2024-03-24T00:38:55.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.257+0000 7fbedfe00640 1 -- 172.21.15.40:0/3855737152 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee00b1180 con 0x7fbee006be20 2024-03-24T00:38:55.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.257+0000 7fbee6800640 1 -- 172.21.15.40:0/3855737152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbee006be20 msgr2=0x7fbee006f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.257+0000 7fbee6800640 1 --2- 172.21.15.40:0/3855737152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbee006be20 0x7fbee006f5f0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fbee0072e90 tx=0x7fbed801c930 comp rx=0 tx=0).stop 2024-03-24T00:38:55.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.257+0000 7fbee6800640 1 -- 172.21.15.40:0/3855737152 shutdown_connections 2024-03-24T00:38:55.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.257+0000 7fbee6800640 1 --2- 172.21.15.40:0/3855737152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbee006be20 0x7fbee006f5f0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.257+0000 7fbee6800640 1 --2- 172.21.15.40:0/3855737152 >> v2:172.21.15.53:3300/0 conn(0x7fbee00d0640 0x7fbee006b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.257+0000 7fbee6800640 1 -- 172.21.15.40:0/3855737152 >> 172.21.15.40:0/3855737152 conn(0x7fbee00d19c0 msgr2=0x7fbee00d1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:55.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.257+0000 7fbee6800640 1 -- 172.21.15.40:0/3855737152 shutdown_connections 2024-03-24T00:38:55.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.257+0000 7fbee6800640 1 -- 172.21.15.40:0/3855737152 wait complete. 2024-03-24T00:38:55.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee6800640 1 Processor -- start 2024-03-24T00:38:55.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee6800640 1 -- start start 2024-03-24T00:38:55.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee6800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbee006be20 0x7fbee0155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:55.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbee0161360 0x7fbee00b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:55.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee6800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbee014f120 0x7fbee00c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:55.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee6800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee00706d0 con 0x7fbee006be20 2024-03-24T00:38:55.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee6800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0070550 con 0x7fbee0161360 2024-03-24T00:38:55.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee6800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0070850 con 0x7fbee014f120 2024-03-24T00:38:55.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbee006be20 0x7fbee0155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:55.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbee006be20 0x7fbee0155890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42586/0 (socket says 172.21.15.40:42586) 2024-03-24T00:38:55.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee5400640 1 -- 172.21.15.40:0/4276664793 learned_addr learned my addr 172.21.15.40:0/4276664793 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:55.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee4a00640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbee0161360 0x7fbee00b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:55.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee5e00640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbee014f120 0x7fbee00c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:55.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee5400640 1 -- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbee014f120 msgr2=0x7fbee00c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee5400640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbee014f120 0x7fbee00c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee5400640 1 -- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbee0161360 msgr2=0x7fbee00b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:55.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee5400640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbee0161360 0x7fbee00b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:55.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee5400640 1 -- 172.21.15.40:0/4276664793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee00c5600 con 0x7fbee006be20 2024-03-24T00:38:55.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee5400640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbee006be20 0x7fbee0155890 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fbed400c480 tx=0x7fbed400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:55.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbedd600640 1 -- 172.21.15.40:0/4276664793 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed4018070 con 0x7fbee006be20 2024-03-24T00:38:55.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbedd600640 1 -- 172.21.15.40:0/4276664793 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed4007500 con 0x7fbee006be20 2024-03-24T00:38:55.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbedd600640 1 -- 172.21.15.40:0/4276664793 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed4004a30 con 0x7fbee006be20 2024-03-24T00:38:55.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee6800640 1 -- 172.21.15.40:0/4276664793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee00c58e0 con 0x7fbee006be20 2024-03-24T00:38:55.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee6800640 1 -- 172.21.15.40:0/4276664793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee014e920 con 0x7fbee006be20 2024-03-24T00:38:55.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.261+0000 7fbee6800640 1 -- 172.21.15.40:0/4276664793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee004f410 con 0x7fbee006be20 2024-03-24T00:38:55.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.265+0000 7fbedd600640 1 -- 172.21.15.40:0/4276664793 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbed4004bd0 con 0x7fbee006be20 2024-03-24T00:38:55.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.265+0000 7fbedd600640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbec0067660 0x7fbec0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:55.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.265+0000 7fbedd600640 1 -- 172.21.15.40:0/4276664793 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fbed4084c40 con 0x7fbee006be20 2024-03-24T00:38:55.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.265+0000 7fbee4a00640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbec0067660 0x7fbec0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:55.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.265+0000 7fbee4a00640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbec0067660 0x7fbec0069b20 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fbee015ff10 tx=0x7fbecc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:55.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.269+0000 7fbedd600640 1 -- 172.21.15.40:0/4276664793 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbed4055120 con 0x7fbee006be20 2024-03-24T00:38:55.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:55.469+0000 7fbee6800640 1 -- 172.21.15.40:0/4276664793 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbee0157010 con 0x7fbee006be20 2024-03-24T00:38:56.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.081+0000 7fbedd600640 1 -- 172.21.15.40:0/4276664793 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v121) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbed40591c0 con 0x7fbee006be20 2024-03-24T00:38:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.085+0000 7fbee6800640 1 -- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbec0067660 msgr2=0x7fbec0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.085+0000 7fbee6800640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbec0067660 0x7fbec0069b20 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fbee015ff10 tx=0x7fbecc008040 comp rx=0 tx=0).stop 2024-03-24T00:38:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.085+0000 7fbee6800640 1 -- 172.21.15.40:0/4276664793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbee006be20 msgr2=0x7fbee0155890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.085+0000 7fbee6800640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbee006be20 0x7fbee0155890 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fbed400c480 tx=0x7fbed400c940 comp rx=0 tx=0).stop 2024-03-24T00:38:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.085+0000 7fbee6800640 1 -- 172.21.15.40:0/4276664793 shutdown_connections 2024-03-24T00:38:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.085+0000 7fbee6800640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbec0067660 0x7fbec0069b20 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.085+0000 7fbee6800640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbee014f120 0x7fbee00c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.085+0000 7fbee6800640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbee0161360 0x7fbee00b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.085+0000 7fbee6800640 1 --2- 172.21.15.40:0/4276664793 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbee006be20 0x7fbee0155890 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.085+0000 7fbee6800640 1 -- 172.21.15.40:0/4276664793 >> 172.21.15.40:0/4276664793 conn(0x7fbee00d19c0 msgr2=0x7fbee00af830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:56.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.089+0000 7fbee6800640 1 -- 172.21.15.40:0/4276664793 shutdown_connections 2024-03-24T00:38:56.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.089+0000 7fbee6800640 1 -- 172.21.15.40:0/4276664793 wait complete. 2024-03-24T00:38:56.115 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T00:38:56.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.305+0000 7fca65e00640 1 Processor -- start 2024-03-24T00:38:56.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.305+0000 7fca65e00640 1 -- start start 2024-03-24T00:38:56.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca65e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca6015e350 0x7fca600cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca65e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca600c9580 0x7fca600c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca65e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fca600ad1e0 0x7fca600ad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca65e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca600cd660 con 0x7fca600c9580 2024-03-24T00:38:56.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca65e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca600734e0 con 0x7fca600ad1e0 2024-03-24T00:38:56.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca65e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca600c9f40 con 0x7fca6015e350 2024-03-24T00:38:56.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca56000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca600c9580 0x7fca600c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca56000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fca600c9580 0x7fca600c9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42596/0 (socket says 172.21.15.40:42596) 2024-03-24T00:38:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca56000640 1 -- 172.21.15.40:0/547712411 learned_addr learned my addr 172.21.15.40:0/547712411 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca5ea00640 1 --2- 172.21.15.40:0/547712411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca6015e350 0x7fca600cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca5f400640 1 --2- 172.21.15.40:0/547712411 >> v2:172.21.15.53:3300/0 conn(0x7fca600ad1e0 0x7fca600ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca56000640 1 -- 172.21.15.40:0/547712411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca6015e350 msgr2=0x7fca600cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca56000640 1 --2- 172.21.15.40:0/547712411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca6015e350 0x7fca600cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca56000640 1 -- 172.21.15.40:0/547712411 >> v2:172.21.15.53:3300/0 conn(0x7fca600ad1e0 msgr2=0x7fca600ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca56000640 1 --2- 172.21.15.40:0/547712411 >> v2:172.21.15.53:3300/0 conn(0x7fca600ad1e0 0x7fca600ad610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.309+0000 7fca56000640 1 -- 172.21.15.40:0/547712411 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca600d0030 con 0x7fca600c9580 2024-03-24T00:38:56.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca56000640 1 --2- 172.21.15.40:0/547712411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca600c9580 0x7fca600c9a00 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fca600727e0 tx=0x7fca4801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e895a1dbd03d99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca5e000640 1 -- 172.21.15.40:0/547712411 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca480ab5f0 con 0x7fca600c9580 2024-03-24T00:38:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca5e000640 1 -- 172.21.15.40:0/547712411 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca480b0070 con 0x7fca600c9580 2024-03-24T00:38:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca5e000640 1 -- 172.21.15.40:0/547712411 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca480ab9b0 con 0x7fca600c9580 2024-03-24T00:38:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 -- 172.21.15.40:0/547712411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca600c9580 msgr2=0x7fca600c9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 --2- 172.21.15.40:0/547712411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca600c9580 0x7fca600c9a00 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fca600727e0 tx=0x7fca4801c520 comp rx=0 tx=0).stop 2024-03-24T00:38:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 -- 172.21.15.40:0/547712411 shutdown_connections 2024-03-24T00:38:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 --2- 172.21.15.40:0/547712411 >> v2:172.21.15.53:3300/0 conn(0x7fca600ad1e0 0x7fca600ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 --2- 172.21.15.40:0/547712411 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca600c9580 0x7fca600c9a00 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 --2- 172.21.15.40:0/547712411 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca6015e350 0x7fca600cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 -- 172.21.15.40:0/547712411 >> 172.21.15.40:0/547712411 conn(0x7fca60072870 msgr2=0x7fca60072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:56.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 -- 172.21.15.40:0/547712411 shutdown_connections 2024-03-24T00:38:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 -- 172.21.15.40:0/547712411 wait complete. 2024-03-24T00:38:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 Processor -- start 2024-03-24T00:38:56.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 -- start start 2024-03-24T00:38:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca600ad1e0 0x7fca60152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca600c9580 0x7fca60150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca6015e350 0x7fca601faeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca600cda30 con 0x7fca6015e350 2024-03-24T00:38:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca600716b0 con 0x7fca600c9580 2024-03-24T00:38:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca65e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca600cdbb0 con 0x7fca600ad1e0 2024-03-24T00:38:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca5f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca6015e350 0x7fca601faeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca5ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca600ad1e0 0x7fca60152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca56000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca600c9580 0x7fca60150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca5f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca6015e350 0x7fca601faeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:42604/0 (socket says 172.21.15.40:42604) 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.313+0000 7fca5f400640 1 -- 172.21.15.40:0/2636723899 learned_addr learned my addr 172.21.15.40:0/2636723899 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca5f400640 1 -- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca600ad1e0 msgr2=0x7fca60152ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca5f400640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca600ad1e0 0x7fca60152ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca5f400640 1 -- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca600c9580 msgr2=0x7fca60150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca5f400640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca600c9580 0x7fca60150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca5f400640 1 -- 172.21.15.40:0/2636723899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca601fb3f0 con 0x7fca6015e350 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca56000640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca600c9580 0x7fca60150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca5f400640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca6015e350 0x7fca601faeb0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fca5800eed0 tx=0x7fca5800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca57e00640 1 -- 172.21.15.40:0/2636723899 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca58018020 con 0x7fca6015e350 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca57e00640 1 -- 172.21.15.40:0/2636723899 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca58014400 con 0x7fca6015e350 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca57e00640 1 -- 172.21.15.40:0/2636723899 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca580147f0 con 0x7fca6015e350 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca65e00640 1 -- 172.21.15.40:0/2636723899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca60150fb0 con 0x7fca6015e350 2024-03-24T00:38:56.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca65e00640 1 -- 172.21.15.40:0/2636723899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca600af9b0 con 0x7fca6015e350 2024-03-24T00:38:56.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca65e00640 1 -- 172.21.15.40:0/2636723899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca60068170 con 0x7fca6015e350 2024-03-24T00:38:56.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.317+0000 7fca57e00640 1 -- 172.21.15.40:0/2636723899 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fca5801b020 con 0x7fca6015e350 2024-03-24T00:38:56.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.321+0000 7fca57e00640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca3c067660 0x7fca3c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:56.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.321+0000 7fca57e00640 1 -- 172.21.15.40:0/2636723899 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 6025+0+0 (secure 0 0 0) 0x7fca58085fa0 con 0x7fca6015e350 2024-03-24T00:38:56.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.321+0000 7fca5ea00640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca3c067660 0x7fca3c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:56.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.321+0000 7fca5ea00640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca3c067660 0x7fca3c069b20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fca600aeaa0 tx=0x7fca50009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:56.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.325+0000 7fca57e00640 1 -- 172.21.15.40:0/2636723899 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fca58022070 con 0x7fca6015e350 2024-03-24T00:38:56.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:56.485+0000 7fca65e00640 1 -- 172.21.15.40:0/2636723899 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fca600af4a0 con 0x7fca6015e350 2024-03-24T00:38:57.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.085+0000 7fca57e00640 1 -- 172.21.15.40:0/2636723899 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v110) v1 ==== 181+0+0 (secure 0 0 0) 0x7fca601fb3f0 con 0x7fca6015e350 2024-03-24T00:38:57.089 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' removed 2024-03-24T00:38:57.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.089+0000 7fca65e00640 1 -- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca3c067660 msgr2=0x7fca3c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:57.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.089+0000 7fca65e00640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca3c067660 0x7fca3c069b20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fca600aeaa0 tx=0x7fca50009190 comp rx=0 tx=0).stop 2024-03-24T00:38:57.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.089+0000 7fca65e00640 1 -- 172.21.15.40:0/2636723899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca6015e350 msgr2=0x7fca601faeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:57.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.089+0000 7fca65e00640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca6015e350 0x7fca601faeb0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fca5800eed0 tx=0x7fca5800b480 comp rx=0 tx=0).stop 2024-03-24T00:38:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.089+0000 7fca65e00640 1 -- 172.21.15.40:0/2636723899 shutdown_connections 2024-03-24T00:38:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.089+0000 7fca65e00640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca3c067660 0x7fca3c069b20 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.089+0000 7fca65e00640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fca6015e350 0x7fca601faeb0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.089+0000 7fca65e00640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca600c9580 0x7fca60150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.089+0000 7fca65e00640 1 --2- 172.21.15.40:0/2636723899 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca600ad1e0 0x7fca60152ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.089+0000 7fca65e00640 1 -- 172.21.15.40:0/2636723899 >> 172.21.15.40:0/2636723899 conn(0x7fca60072870 msgr2=0x7fca601574a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.089+0000 7fca65e00640 1 -- 172.21.15.40:0/2636723899 shutdown_connections 2024-03-24T00:38:57.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.089+0000 7fca65e00640 1 -- 172.21.15.40:0/2636723899 wait complete. 2024-03-24T00:38:57.115 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T00:38:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.257+0000 7f12afc00640 1 Processor -- start 2024-03-24T00:38:57.262 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.257+0000 7f12afc00640 1 -- start start 2024-03-24T00:38:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.257+0000 7f12afc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f12a815e170 0x7f12a806f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.257+0000 7f12afc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f12a80d0e60 0x7f12a815e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.257+0000 7f12afc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a80cfcb0 con 0x7f12a80d0a10 2024-03-24T00:38:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.257+0000 7f12afc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a8070100 con 0x7f12a80d0e60 2024-03-24T00:38:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.257+0000 7f12afc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a806fe20 con 0x7f12a815e170 2024-03-24T00:38:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.261+0000 7f12adc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f12a80d0e60 0x7f12a815e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.261+0000 7f12adc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f12a80d0e60 0x7f12a815e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34294/0 (socket says 172.21.15.40:34294) 2024-03-24T00:38:57.263 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.261+0000 7f12adc00640 1 -- 172.21.15.40:0/605281976 learned_addr learned my addr 172.21.15.40:0/605281976 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:57.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.261+0000 7f12ad200640 1 --2- 172.21.15.40:0/605281976 >> v2:172.21.15.53:3300/0 conn(0x7f12a815e170 0x7f12a806f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:57.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.261+0000 7f12adc00640 1 -- 172.21.15.40:0/605281976 >> v2:172.21.15.53:3300/0 conn(0x7f12a815e170 msgr2=0x7f12a806f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:57.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.261+0000 7f12adc00640 1 --2- 172.21.15.40:0/605281976 >> v2:172.21.15.53:3300/0 conn(0x7f12a815e170 0x7f12a806f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.261+0000 7f12adc00640 1 -- 172.21.15.40:0/605281976 >> v1:172.21.15.53:6789/0 conn(0x7f12a80d0a10 legacy=0x7f12a815f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:57.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.261+0000 7f12adc00640 1 -- 172.21.15.40:0/605281976 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a80cff30 con 0x7f12a80d0e60 2024-03-24T00:38:57.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.261+0000 7f12adc00640 1 --2- 172.21.15.40:0/605281976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a80d0e60 0x7f12a815e550 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f12a806ebb0 tx=0x7f12a001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=397136cf89c6c4b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:57.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.261+0000 7f12a7400640 1 -- 172.21.15.40:0/605281976 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a00a5880 con 0x7f12a80d0e60 2024-03-24T00:38:57.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.261+0000 7f12a7400640 1 -- 172.21.15.40:0/605281976 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a00a5a20 con 0x7f12a80d0e60 2024-03-24T00:38:57.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.261+0000 7f12a7400640 1 -- 172.21.15.40:0/605281976 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a00a5df0 con 0x7f12a80d0e60 2024-03-24T00:38:57.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 -- 172.21.15.40:0/605281976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a80d0e60 msgr2=0x7f12a815e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:57.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 --2- 172.21.15.40:0/605281976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a80d0e60 0x7f12a815e550 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f12a806ebb0 tx=0x7f12a001c9c0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 -- 172.21.15.40:0/605281976 shutdown_connections 2024-03-24T00:38:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 --2- 172.21.15.40:0/605281976 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a80d0e60 0x7f12a815e550 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 --2- 172.21.15.40:0/605281976 >> v2:172.21.15.53:3300/0 conn(0x7f12a815e170 0x7f12a806f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 -- 172.21.15.40:0/605281976 >> 172.21.15.40:0/605281976 conn(0x7f12a80b36a0 msgr2=0x7f12a80b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 -- 172.21.15.40:0/605281976 shutdown_connections 2024-03-24T00:38:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 -- 172.21.15.40:0/605281976 wait complete. 2024-03-24T00:38:57.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 Processor -- start 2024-03-24T00:38:57.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 -- start start 2024-03-24T00:38:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a80d0a10 0x7f12a8153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12a80d0e60 0x7f12a80c6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a815e170 0x7f12a80b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a816b030 con 0x7f12a80d0a10 2024-03-24T00:38:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a816aeb0 con 0x7f12a80d0e60 2024-03-24T00:38:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12afc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a816b1b0 con 0x7f12a815e170 2024-03-24T00:38:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12ad200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a80d0a10 0x7f12a8153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12ad200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a80d0a10 0x7f12a8153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34298/0 (socket says 172.21.15.40:34298) 2024-03-24T00:38:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12ad200640 1 -- 172.21.15.40:0/3733875018 learned_addr learned my addr 172.21.15.40:0/3733875018 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12a7e00640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12a80d0e60 0x7f12a80c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:57.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12adc00640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a815e170 0x7f12a80b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12ad200640 1 -- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a815e170 msgr2=0x7f12a80b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12ad200640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a815e170 0x7f12a80b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12ad200640 1 -- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12a80d0e60 msgr2=0x7f12a80c6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12ad200640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12a80d0e60 0x7f12a80c6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12ad200640 1 -- 172.21.15.40:0/3733875018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a80b5160 con 0x7f12a80d0a10 2024-03-24T00:38:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.265+0000 7f12adc00640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a815e170 0x7f12a80b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.269+0000 7f12ad200640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a80d0a10 0x7f12a8153e30 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f129c00d620 tx=0x7f129c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.269+0000 7f12a4c00640 1 -- 172.21.15.40:0/3733875018 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c00b620 con 0x7f12a80d0a10 2024-03-24T00:38:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.269+0000 7f12a4c00640 1 -- 172.21.15.40:0/3733875018 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129c019070 con 0x7f12a80d0a10 2024-03-24T00:38:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.269+0000 7f12a4c00640 1 -- 172.21.15.40:0/3733875018 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129c00b9e0 con 0x7f12a80d0a10 2024-03-24T00:38:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.269+0000 7f12afc00640 1 -- 172.21.15.40:0/3733875018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a80b5440 con 0x7f12a80d0a10 2024-03-24T00:38:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.269+0000 7f12afc00640 1 -- 172.21.15.40:0/3733875018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a80b7660 con 0x7f12a80d0a10 2024-03-24T00:38:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.269+0000 7f12afc00640 1 -- 172.21.15.40:0/3733875018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a8068170 con 0x7f12a80d0a10 2024-03-24T00:38:57.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.269+0000 7f12a4c00640 1 -- 172.21.15.40:0/3733875018 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f129c00bb80 con 0x7f12a80d0a10 2024-03-24T00:38:57.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.269+0000 7f12a4c00640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1288067690 0x7f1288069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:57.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.269+0000 7f12a4c00640 1 -- 172.21.15.40:0/3733875018 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5587+0+0 (secure 0 0 0) 0x7f129c085bc0 con 0x7f12a80d0a10 2024-03-24T00:38:57.274 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.269+0000 7f12a7e00640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1288067690 0x7f1288069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:57.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.273+0000 7f12a7e00640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1288067690 0x7f1288069b50 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f12a806bc10 tx=0x7f1294009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:57.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.277+0000 7f12a4c00640 1 -- 172.21.15.40:0/3733875018 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f129c056810 con 0x7f12a80d0a10 2024-03-24T00:38:57.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:57.437+0000 7f12afc00640 1 -- 172.21.15.40:0/3733875018 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f12a8072f30 con 0x7f12a80d0a10 2024-03-24T00:38:58.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.089+0000 7f12a4c00640 1 -- 172.21.15.40:0/3733875018 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v111) v1 ==== 169+0+0 (secure 0 0 0) 0x7f129c056110 con 0x7f12a80d0a10 2024-03-24T00:38:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' removed 2024-03-24T00:38:58.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.093+0000 7f12afc00640 1 -- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1288067690 msgr2=0x7f1288069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.093+0000 7f12afc00640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1288067690 0x7f1288069b50 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f12a806bc10 tx=0x7f1294009040 comp rx=0 tx=0).stop 2024-03-24T00:38:58.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.093+0000 7f12afc00640 1 -- 172.21.15.40:0/3733875018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a80d0a10 msgr2=0x7f12a8153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.093+0000 7f12afc00640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a80d0a10 0x7f12a8153e30 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f129c00d620 tx=0x7f129c00dae0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.093+0000 7f12afc00640 1 -- 172.21.15.40:0/3733875018 shutdown_connections 2024-03-24T00:38:58.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.093+0000 7f12afc00640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f1288067690 0x7f1288069b50 secure :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f12a806bc10 tx=0x7f1294009040 comp rx=0 tx=0).stop 2024-03-24T00:38:58.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.093+0000 7f12afc00640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f12a815e170 0x7f12a80b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.093+0000 7f12afc00640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12a80d0e60 0x7f12a80c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.093+0000 7f12afc00640 1 --2- 172.21.15.40:0/3733875018 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f12a80d0a10 0x7f12a8153e30 secure :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f129c00d620 tx=0x7f129c00dae0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.093+0000 7f12afc00640 1 -- 172.21.15.40:0/3733875018 >> 172.21.15.40:0/3733875018 conn(0x7f12a80b36a0 msgr2=0x7f12a806b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:58.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.097+0000 7f12afc00640 1 -- 172.21.15.40:0/3733875018 shutdown_connections 2024-03-24T00:38:58.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.097+0000 7f12afc00640 1 -- 172.21.15.40:0/3733875018 wait complete. 2024-03-24T00:38:58.129 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random 2024-03-24T00:38:58.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7ec1600640 1 Processor -- start 2024-03-24T00:38:58.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7ec1600640 1 -- start start 2024-03-24T00:38:58.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7ec1600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7ebc0ae8e0 0x7f7ebc0728d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7ec1600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ebc0aed00 con 0x7f7ebc15ffd0 2024-03-24T00:38:58.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7ec1600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ebc071de0 con 0x7f7ebc15bd00 2024-03-24T00:38:58.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7ec1600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ebc0cea50 con 0x7f7ebc0ae8e0 2024-03-24T00:38:58.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7ebb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7ebc0ae8e0 0x7f7ebc0728d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7ebb400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7ebc0ae8e0 0x7f7ebc0728d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34312/0 (socket says 172.21.15.40:34312) 2024-03-24T00:38:58.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7ebb400640 1 -- 172.21.15.40:0/1418803729 learned_addr learned my addr 172.21.15.40:0/1418803729 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:58.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7eb2000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f7ebc15ffd0 0x7f7ebc06cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39708/0 (socket says 172.21.15.40:39708) 2024-03-24T00:38:58.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7ebb400640 1 -- 172.21.15.40:0/1418803729 >> v1:172.21.15.53:6789/0 conn(0x7f7ebc15bd00 legacy=0x7f7ebc15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7ebb400640 1 -- 172.21.15.40:0/1418803729 >> v1:172.21.15.40:6789/0 conn(0x7f7ebc15ffd0 legacy=0x7f7ebc06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.273+0000 7f7ebb400640 1 -- 172.21.15.40:0/1418803729 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc0ced30 con 0x7f7ebc0ae8e0 2024-03-24T00:38:58.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.277+0000 7f7ebb400640 1 --2- 172.21.15.40:0/1418803729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ebc0ae8e0 0x7f7ebc0728d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4009e50 tx=0x7f7eb401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc410e5b552a5af4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.277+0000 7f7eba000640 1 -- 172.21.15.40:0/1418803729 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb40a57d0 con 0x7f7ebc0ae8e0 2024-03-24T00:38:58.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.277+0000 7f7eba000640 1 -- 172.21.15.40:0/1418803729 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb40a5970 con 0x7f7ebc0ae8e0 2024-03-24T00:38:58.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.277+0000 7f7eba000640 1 -- 172.21.15.40:0/1418803729 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb40a5d40 con 0x7f7ebc0ae8e0 2024-03-24T00:38:58.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.277+0000 7f7ec1600640 1 -- 172.21.15.40:0/1418803729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ebc0ae8e0 msgr2=0x7f7ebc0728d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.277+0000 7f7ec1600640 1 --2- 172.21.15.40:0/1418803729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ebc0ae8e0 0x7f7ebc0728d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4009e50 tx=0x7f7eb401ca30 comp rx=0 tx=0).stop 2024-03-24T00:38:58.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.277+0000 7f7ec1600640 1 -- 172.21.15.40:0/1418803729 shutdown_connections 2024-03-24T00:38:58.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.277+0000 7f7ec1600640 1 --2- 172.21.15.40:0/1418803729 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ebc0ae8e0 0x7f7ebc0728d0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.277+0000 7f7ec1600640 1 -- 172.21.15.40:0/1418803729 >> 172.21.15.40:0/1418803729 conn(0x7f7ebc0b1660 msgr2=0x7f7ebc0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:58.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.277+0000 7f7ec1600640 1 -- 172.21.15.40:0/1418803729 shutdown_connections 2024-03-24T00:38:58.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.277+0000 7f7ec1600640 1 -- 172.21.15.40:0/1418803729 wait complete. 2024-03-24T00:38:58.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ec1600640 1 Processor -- start 2024-03-24T00:38:58.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ec1600640 1 -- start start 2024-03-24T00:38:58.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ec1600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc0ae8e0 0x7f7ebc14e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ec1600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ebc15bd00 0x7f7ebc151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ec1600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7ebc15ffd0 0x7f7ebc1535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ec1600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc15c0e0 con 0x7f7ebc15bd00 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ec1600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc0cf660 con 0x7f7ebc0ae8e0 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ec1600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc16a420 con 0x7f7ebc15ffd0 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7eb2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ebc15bd00 0x7f7ebc151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7eb2000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ebc15bd00 0x7f7ebc151320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34326/0 (socket says 172.21.15.40:34326) 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7eb2000640 1 -- 172.21.15.40:0/2730201116 learned_addr learned my addr 172.21.15.40:0/2730201116 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ebb400640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7ebc15ffd0 0x7f7ebc1535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ebaa00640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc0ae8e0 0x7f7ebc14e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7eb2000640 1 -- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7ebc15ffd0 msgr2=0x7f7ebc1535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7eb2000640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7ebc15ffd0 0x7f7ebc1535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7eb2000640 1 -- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc0ae8e0 msgr2=0x7f7ebc14e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7eb2000640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc0ae8e0 0x7f7ebc14e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7eb2000640 1 -- 172.21.15.40:0/2730201116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc150280 con 0x7f7ebc15bd00 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7eb2000640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ebc15bd00 0x7f7ebc151320 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400c430 tx=0x7f7ea400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7eb3400640 1 -- 172.21.15.40:0/2730201116 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea40074e0 con 0x7f7ebc15bd00 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7eb3400640 1 -- 172.21.15.40:0/2730201116 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea4007680 con 0x7f7ebc15bd00 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7eb3400640 1 -- 172.21.15.40:0/2730201116 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4004a10 con 0x7f7ebc15bd00 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ec1600640 1 -- 172.21.15.40:0/2730201116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ebc150560 con 0x7f7ebc15bd00 2024-03-24T00:38:58.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ec1600640 1 -- 172.21.15.40:0/2730201116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ebc150b20 con 0x7f7ebc15bd00 2024-03-24T00:38:58.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.281+0000 7f7ec1600640 1 -- 172.21.15.40:0/2730201116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ebc068170 con 0x7f7ebc15bd00 2024-03-24T00:38:58.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.285+0000 7f7eb3400640 1 -- 172.21.15.40:0/2730201116 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7ea401a020 con 0x7f7ebc15bd00 2024-03-24T00:38:58.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.285+0000 7f7eb3400640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e98067660 0x7f7e98069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.285+0000 7f7eb3400640 1 -- 172.21.15.40:0/2730201116 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f7ea4084dd0 con 0x7f7ebc15bd00 2024-03-24T00:38:58.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.285+0000 7f7ebaa00640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e98067660 0x7f7e98069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.289+0000 7f7ebaa00640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e98067660 0x7f7e98069b20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7eac001650 tx=0x7f7eac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.293+0000 7f7eb3400640 1 -- 172.21.15.40:0/2730201116 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7ea4055d30 con 0x7f7ebc15bd00 2024-03-24T00:38:58.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.469+0000 7f7ec1600640 1 -- 172.21.15.40:0/2730201116 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"} v 0) v1 -- 0x7f7ebc06e870 con 0x7f7ebc15bd00 2024-03-24T00:38:58.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7eb3400640 1 -- 172.21.15.40:0/2730201116 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random"}]=0 v63) v1 ==== 110+0+0 (secure 0 0 0) 0x7f7ea40860b0 con 0x7f7ebc15bd00 2024-03-24T00:38:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7ec1600640 1 -- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e98067660 msgr2=0x7f7e98069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7ec1600640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e98067660 0x7f7e98069b20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f7eac001650 tx=0x7f7eac009040 comp rx=0 tx=0).stop 2024-03-24T00:38:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7ec1600640 1 -- 172.21.15.40:0/2730201116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ebc15bd00 msgr2=0x7f7ebc151320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7ec1600640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ebc15bd00 0x7f7ebc151320 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f7ea400c430 tx=0x7f7ea400c8f0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7ec1600640 1 -- 172.21.15.40:0/2730201116 shutdown_connections 2024-03-24T00:38:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7ec1600640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7e98067660 0x7f7e98069b20 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7ec1600640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7ebc15ffd0 0x7f7ebc1535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7ec1600640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7ebc15bd00 0x7f7ebc151320 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7ec1600640 1 --2- 172.21.15.40:0/2730201116 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc0ae8e0 0x7f7ebc14e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7ec1600640 1 -- 172.21.15.40:0/2730201116 >> 172.21.15.40:0/2730201116 conn(0x7f7ebc0b1660 msgr2=0x7f7ebc158b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:58.479 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7ec1600640 1 -- 172.21.15.40:0/2730201116 shutdown_connections 2024-03-24T00:38:58.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.473+0000 7f7ec1600640 1 -- 172.21.15.40:0/2730201116 wait complete. 2024-03-24T00:38:58.498 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_distributed 2024-03-24T00:38:58.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f840fc00640 1 Processor -- start 2024-03-24T00:38:58.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f840fc00640 1 -- start start 2024-03-24T00:38:58.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f840fc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f840815b960 0x7f8408070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f840fc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f840815bd40 0x7f84080712a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f840fc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840815f970 con 0x7f84080d0fe0 2024-03-24T00:38:58.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f840fc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8408160580 con 0x7f840815b960 2024-03-24T00:38:58.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f840fc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f840815fc50 con 0x7f840815bd40 2024-03-24T00:38:58.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f840dc00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f84080d0fe0 0x7f84080cf7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39722/0 (socket says 172.21.15.40:39722) 2024-03-24T00:38:58.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f840dc00640 1 -- 172.21.15.40:0/1242516485 learned_addr learned my addr 172.21.15.40:0/1242516485 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:58.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f8407e00640 1 --2- 172.21.15.40:0/1242516485 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f840815bd40 0x7f84080712a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f840d200640 1 --2- 172.21.15.40:0/1242516485 >> v2:172.21.15.53:3300/0 conn(0x7f840815b960 0x7f8408070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f8407400640 1 -- 172.21.15.40:0/1242516485 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468476669 0 0) 0x7f840815f970 con 0x7f84080d0fe0 2024-03-24T00:38:58.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f8407400640 1 -- 172.21.15.40:0/1242516485 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f0003530 con 0x7f84080d0fe0 2024-03-24T00:38:58.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f8407e00640 1 -- 172.21.15.40:0/1242516485 >> v2:172.21.15.53:3300/0 conn(0x7f840815b960 msgr2=0x7f8408070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:58.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f8407e00640 1 --2- 172.21.15.40:0/1242516485 >> v2:172.21.15.53:3300/0 conn(0x7f840815b960 0x7f8408070d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f8407e00640 1 -- 172.21.15.40:0/1242516485 >> v1:172.21.15.40:6789/0 conn(0x7f84080d0fe0 legacy=0x7f84080cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.644 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.637+0000 7f8407e00640 1 -- 172.21.15.40:0/1242516485 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840815ff30 con 0x7f840815bd40 2024-03-24T00:38:58.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.641+0000 7f8407e00640 1 --2- 172.21.15.40:0/1242516485 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f840815bd40 0x7f84080712a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f83f40097c0 tx=0x7f83f401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=690747f615ef6316 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.641+0000 7f8407400640 1 -- 172.21.15.40:0/1242516485 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f40b1070 con 0x7f840815bd40 2024-03-24T00:38:58.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.641+0000 7f8407400640 1 -- 172.21.15.40:0/1242516485 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f40ac410 con 0x7f840815bd40 2024-03-24T00:38:58.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.641+0000 7f8407400640 1 -- 172.21.15.40:0/1242516485 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f40ac7c0 con 0x7f840815bd40 2024-03-24T00:38:58.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.641+0000 7f840fc00640 1 -- 172.21.15.40:0/1242516485 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f840815bd40 msgr2=0x7f84080712a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.641+0000 7f840fc00640 1 --2- 172.21.15.40:0/1242516485 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f840815bd40 0x7f84080712a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f83f40097c0 tx=0x7f83f401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.641+0000 7f840fc00640 1 -- 172.21.15.40:0/1242516485 shutdown_connections 2024-03-24T00:38:58.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.641+0000 7f840fc00640 1 --2- 172.21.15.40:0/1242516485 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f840815bd40 0x7f84080712a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.641+0000 7f840fc00640 1 --2- 172.21.15.40:0/1242516485 >> v2:172.21.15.53:3300/0 conn(0x7f840815b960 0x7f8408070d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.641+0000 7f840fc00640 1 -- 172.21.15.40:0/1242516485 >> 172.21.15.40:0/1242516485 conn(0x7f840806ecf0 msgr2=0x7f840806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:58.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.641+0000 7f840fc00640 1 -- 172.21.15.40:0/1242516485 shutdown_connections 2024-03-24T00:38:58.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.641+0000 7f840fc00640 1 -- 172.21.15.40:0/1242516485 wait complete. 2024-03-24T00:38:58.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840fc00640 1 Processor -- start 2024-03-24T00:38:58.648 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840fc00640 1 -- start start 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840fc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84080d0fe0 0x7f8408147770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840fc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f840815b960 0x7f840814dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840fc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f840815bd40 0x7f840814f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840fc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8408168f00 con 0x7f840815bd40 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840fc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8408168d80 con 0x7f840815b960 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840fc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8408169080 con 0x7f84080d0fe0 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f840815bd40 0x7f840814f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f840815bd40 0x7f840814f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34332/0 (socket says 172.21.15.40:34332) 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840dc00640 1 -- 172.21.15.40:0/960855561 learned_addr learned my addr 172.21.15.40:0/960855561 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840d200640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84080d0fe0 0x7f8408147770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f8407e00640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f840815b960 0x7f840814dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840dc00640 1 -- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84080d0fe0 msgr2=0x7f8408147770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840dc00640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84080d0fe0 0x7f8408147770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.651 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840dc00640 1 -- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f840815b960 msgr2=0x7f840814dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840dc00640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f840815b960 0x7f840814dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840dc00640 1 -- 172.21.15.40:0/960855561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84080c73b0 con 0x7f840815bd40 2024-03-24T00:38:58.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840d200640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84080d0fe0 0x7f8408147770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:38:58.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f8407e00640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f840815b960 0x7f840814dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:38:58.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840dc00640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f840815bd40 0x7f840814f880 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f840000cb40 tx=0x7f840000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f8404c00640 1 -- 172.21.15.40:0/960855561 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400017070 con 0x7f840815bd40 2024-03-24T00:38:58.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f8404c00640 1 -- 172.21.15.40:0/960855561 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840000aca0 con 0x7f840815bd40 2024-03-24T00:38:58.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f8404c00640 1 -- 172.21.15.40:0/960855561 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400012620 con 0x7f840815bd40 2024-03-24T00:38:58.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840fc00640 1 -- 172.21.15.40:0/960855561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84080c7690 con 0x7f840815bd40 2024-03-24T00:38:58.652 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.645+0000 7f840fc00640 1 -- 172.21.15.40:0/960855561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84080c7c50 con 0x7f840815bd40 2024-03-24T00:38:58.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.649+0000 7f840fc00640 1 -- 172.21.15.40:0/960855561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8408068170 con 0x7f840815bd40 2024-03-24T00:38:58.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.649+0000 7f8404c00640 1 -- 172.21.15.40:0/960855561 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f840001a070 con 0x7f840815bd40 2024-03-24T00:38:58.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.649+0000 7f8404c00640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f83e8067660 0x7f83e8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.649+0000 7f8404c00640 1 -- 172.21.15.40:0/960855561 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f84000855a0 con 0x7f840815bd40 2024-03-24T00:38:58.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.649+0000 7f840d200640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f83e8067660 0x7f83e8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.649+0000 7f840d200640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f83e8067660 0x7f83e8069b20 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f840815c5c0 tx=0x7f83fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.653+0000 7f8404c00640 1 -- 172.21.15.40:0/960855561 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f84000563d0 con 0x7f840815bd40 2024-03-24T00:38:58.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.817+0000 7f840fc00640 1 -- 172.21.15.40:0/960855561 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"} v 0) v1 -- 0x7f8408070910 con 0x7f840815bd40 2024-03-24T00:38:58.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.821+0000 7f8404c00640 1 -- 172.21.15.40:0/960855561 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_distributed"}]=0 v64) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8400055cd0 con 0x7f840815bd40 2024-03-24T00:38:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.825+0000 7f840fc00640 1 -- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f83e8067660 msgr2=0x7f83e8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.825+0000 7f840fc00640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f83e8067660 0x7f83e8069b20 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f840815c5c0 tx=0x7f83fc008040 comp rx=0 tx=0).stop 2024-03-24T00:38:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.825+0000 7f840fc00640 1 -- 172.21.15.40:0/960855561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f840815bd40 msgr2=0x7f840814f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.825+0000 7f840fc00640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f840815bd40 0x7f840814f880 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f840000cb40 tx=0x7f840000a060 comp rx=0 tx=0).stop 2024-03-24T00:38:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.825+0000 7f840fc00640 1 -- 172.21.15.40:0/960855561 shutdown_connections 2024-03-24T00:38:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.825+0000 7f840fc00640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f83e8067660 0x7f83e8069b20 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.825+0000 7f840fc00640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f840815bd40 0x7f840814f880 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.825+0000 7f840fc00640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f840815b960 0x7f840814dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.825+0000 7f840fc00640 1 --2- 172.21.15.40:0/960855561 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84080d0fe0 0x7f8408147770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.825+0000 7f840fc00640 1 -- 172.21.15.40:0/960855561 >> 172.21.15.40:0/960855561 conn(0x7f840806ecf0 msgr2=0x7f840815d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.825+0000 7f840fc00640 1 -- 172.21.15.40:0/960855561 shutdown_connections 2024-03-24T00:38:58.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.825+0000 7f840fc00640 1 -- 172.21.15.40:0/960855561 wait complete. 2024-03-24T00:38:58.847 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_export_ephemeral_random_max 2024-03-24T00:38:58.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.985+0000 7f3f40c00640 1 Processor -- start 2024-03-24T00:38:58.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f40c00640 1 -- start start 2024-03-24T00:38:58.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f40c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f3c15bd00 0x7f3f3c0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f40c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f3c0728d0 0x7f3f3c0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f40c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3f3c15f8c0 0x7f3f3c15fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:58.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f40c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3c0cf790 con 0x7f3f3c15bd00 2024-03-24T00:38:58.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f40c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3c160250 con 0x7f3f3c15f8c0 2024-03-24T00:38:58.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f40c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c072d30 con 0x7f3f3c0728d0 2024-03-24T00:38:58.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f39e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f3c15bd00 0x7f3f3c0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f39e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f3f3c15bd00 0x7f3f3c0ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34334/0 (socket says 172.21.15.40:34334) 2024-03-24T00:38:58.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f39e00640 1 -- 172.21.15.40:0/3987411178 learned_addr learned my addr 172.21.15.40:0/3987411178 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:58.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f3a800640 1 --2- 172.21.15.40:0/3987411178 >> v2:172.21.15.53:3300/0 conn(0x7f3f3c15f8c0 0x7f3f3c15fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.993 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f33e00640 1 --2- 172.21.15.40:0/3987411178 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f3c0728d0 0x7f3f3c0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:58.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f39e00640 1 -- 172.21.15.40:0/3987411178 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f3c0728d0 msgr2=0x7f3f3c0cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:58.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f39e00640 1 --2- 172.21.15.40:0/3987411178 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f3c0728d0 0x7f3f3c0cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f39e00640 1 -- 172.21.15.40:0/3987411178 >> v2:172.21.15.53:3300/0 conn(0x7f3f3c15f8c0 msgr2=0x7f3f3c15fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:58.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f39e00640 1 --2- 172.21.15.40:0/3987411178 >> v2:172.21.15.53:3300/0 conn(0x7f3f3c15f8c0 0x7f3f3c15fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f39e00640 1 -- 172.21.15.40:0/3987411178 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c071de0 con 0x7f3f3c15bd00 2024-03-24T00:38:58.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.989+0000 7f3f33e00640 1 --2- 172.21.15.40:0/3987411178 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f3c0728d0 0x7f3f3c0cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:38:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f39e00640 1 --2- 172.21.15.40:0/3987411178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f3c15bd00 0x7f3f3c0ae8e0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c073320 tx=0x7f3f2c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8855eceb614715d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f39400640 1 -- 172.21.15.40:0/3987411178 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c0aa610 con 0x7f3f3c15bd00 2024-03-24T00:38:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f39400640 1 -- 172.21.15.40:0/3987411178 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c0af070 con 0x7f3f3c15bd00 2024-03-24T00:38:58.996 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f39400640 1 -- 172.21.15.40:0/3987411178 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c0aa9d0 con 0x7f3f3c15bd00 2024-03-24T00:38:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f40c00640 1 -- 172.21.15.40:0/3987411178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f3c15bd00 msgr2=0x7f3f3c0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f40c00640 1 --2- 172.21.15.40:0/3987411178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f3c15bd00 0x7f3f3c0ae8e0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c073320 tx=0x7f3f2c01c4f0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f40c00640 1 -- 172.21.15.40:0/3987411178 shutdown_connections 2024-03-24T00:38:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f40c00640 1 --2- 172.21.15.40:0/3987411178 >> v2:172.21.15.53:3300/0 conn(0x7f3f3c15f8c0 0x7f3f3c15fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f40c00640 1 --2- 172.21.15.40:0/3987411178 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f3c0728d0 0x7f3f3c0cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f40c00640 1 --2- 172.21.15.40:0/3987411178 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f3c15bd00 0x7f3f3c0ae8e0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f40c00640 1 -- 172.21.15.40:0/3987411178 >> 172.21.15.40:0/3987411178 conn(0x7f3f3c0b1660 msgr2=0x7f3f3c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f40c00640 1 -- 172.21.15.40:0/3987411178 shutdown_connections 2024-03-24T00:38:58.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f40c00640 1 -- 172.21.15.40:0/3987411178 wait complete. 2024-03-24T00:38:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f40c00640 1 Processor -- start 2024-03-24T00:38:58.998 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.993+0000 7f3f40c00640 1 -- start start 2024-03-24T00:38:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f40c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f3c0728d0 0x7f3f3c1551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.000 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f40c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f3c15bd00 0x7f3f3c148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f40c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f3c15f8c0 0x7f3f3c1fa100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f40c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c06d320 con 0x7f3f3c15bd00 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f40c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c15c0e0 con 0x7f3f3c0728d0 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f40c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3c06d4a0 con 0x7f3f3c15f8c0 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f33e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f3c15bd00 0x7f3f3c148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f39e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f3c0728d0 0x7f3f3c1551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f39e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f3c0728d0 0x7f3f3c1551c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:58426/0 (socket says 172.21.15.40:58426) 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f39e00640 1 -- 172.21.15.40:0/3409194139 learned_addr learned my addr 172.21.15.40:0/3409194139 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f3a800640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f3c15f8c0 0x7f3f3c1fa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f39e00640 1 -- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f3c15f8c0 msgr2=0x7f3f3c1fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f39e00640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f3c15f8c0 0x7f3f3c1fa100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f39e00640 1 -- 172.21.15.40:0/3409194139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f3c15bd00 msgr2=0x7f3f3c148d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f39e00640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f3c15bd00 0x7f3f3c148d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f39e00640 1 -- 172.21.15.40:0/3409194139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c155860 con 0x7f3f3c0728d0 2024-03-24T00:38:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f39e00640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f3c0728d0 0x7f3f3c1551c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c01aba0 tx=0x7f3f2c01ad30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f32000640 1 -- 172.21.15.40:0/3409194139 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c0bd5f0 con 0x7f3f3c0728d0 2024-03-24T00:38:59.002 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f32000640 1 -- 172.21.15.40:0/3409194139 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2c0bd790 con 0x7f3f3c0728d0 2024-03-24T00:38:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f32000640 1 -- 172.21.15.40:0/3409194139 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c0bdb40 con 0x7f3f3c0728d0 2024-03-24T00:38:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f40c00640 1 -- 172.21.15.40:0/3409194139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3c155b40 con 0x7f3f3c0728d0 2024-03-24T00:38:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f40c00640 1 -- 172.21.15.40:0/3409194139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3c1f9900 con 0x7f3f3c0728d0 2024-03-24T00:38:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:58.997+0000 7f3f40c00640 1 -- 172.21.15.40:0/3409194139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f3c068170 con 0x7f3f3c0728d0 2024-03-24T00:38:59.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.001+0000 7f3f32000640 1 -- 172.21.15.40:0/3409194139 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3f2c0059a0 con 0x7f3f3c0728d0 2024-03-24T00:38:59.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.001+0000 7f3f32000640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f18067660 0x7f3f18069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.001+0000 7f3f32000640 1 -- 172.21.15.40:0/3409194139 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3f2c11fa40 con 0x7f3f3c0728d0 2024-03-24T00:38:59.004 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.001+0000 7f3f33e00640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f18067660 0x7f3f18069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.001+0000 7f3f33e00640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f18067660 0x7f3f18069b20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0b0720 tx=0x7f3f24008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.007 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.005+0000 7f3f32000640 1 -- 172.21.15.40:0/3409194139 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3f2c0a7070 con 0x7f3f3c0728d0 2024-03-24T00:38:59.175 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.173+0000 7f3f40c00640 1 -- 172.21.15.40:0/3409194139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"} v 0) v1 -- 0x7f3f3c06e870 con 0x7f3f3c0728d0 2024-03-24T00:38:59.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.177+0000 7f3f32000640 1 -- 172.21.15.40:0/3409194139 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_export_ephemeral_random_max"}]=0 v65) v1 ==== 114+0+0 (secure 0 0 0) 0x7f3f2c016110 con 0x7f3f3c0728d0 2024-03-24T00:38:59.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.181+0000 7f3f40c00640 1 -- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f18067660 msgr2=0x7f3f18069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.181+0000 7f3f40c00640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f18067660 0x7f3f18069b20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c0b0720 tx=0x7f3f24008040 comp rx=0 tx=0).stop 2024-03-24T00:38:59.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.181+0000 7f3f40c00640 1 -- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f3c0728d0 msgr2=0x7f3f3c1551c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.181+0000 7f3f40c00640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f3c0728d0 0x7f3f3c1551c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c01aba0 tx=0x7f3f2c01ad30 comp rx=0 tx=0).stop 2024-03-24T00:38:59.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.181+0000 7f3f40c00640 1 -- 172.21.15.40:0/3409194139 shutdown_connections 2024-03-24T00:38:59.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.181+0000 7f3f40c00640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3f18067660 0x7f3f18069b20 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.181+0000 7f3f40c00640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3f3c15f8c0 0x7f3f3c1fa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.181+0000 7f3f40c00640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3f3c15bd00 0x7f3f3c148d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.181+0000 7f3f40c00640 1 --2- 172.21.15.40:0/3409194139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3f3c0728d0 0x7f3f3c1551c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.181+0000 7f3f40c00640 1 -- 172.21.15.40:0/3409194139 >> 172.21.15.40:0/3409194139 conn(0x7f3f3c0b1660 msgr2=0x7f3f3c0cf910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:59.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.181+0000 7f3f40c00640 1 -- 172.21.15.40:0/3409194139 shutdown_connections 2024-03-24T00:38:59.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.181+0000 7f3f40c00640 1 -- 172.21.15.40:0/3409194139 wait complete. 2024-03-24T00:38:59.203 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-24T00:38:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f963a600640 1 Processor -- start 2024-03-24T00:38:59.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f963a600640 1 -- start start 2024-03-24T00:38:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f963a600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f96340b4330 0x7f96340ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f963a600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96340d1760 0x7f96340cea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f963a600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96340cf2d0 con 0x7f96340b22d0 2024-03-24T00:38:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f963a600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96340d0880 con 0x7f96340b4330 2024-03-24T00:38:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f963a600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96340d0b60 con 0x7f96340d1760 2024-03-24T00:38:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f962aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f96340b4330 0x7f96340ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f962aa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f96340b4330 0x7f96340ce4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34348/0 (socket says 172.21.15.40:34348) 2024-03-24T00:38:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f962aa00640 1 -- 172.21.15.40:0/2566877683 learned_addr learned my addr 172.21.15.40:0/2566877683 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:59.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f9633e00640 1 --2- 172.21.15.40:0/2566877683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96340d1760 0x7f96340cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f962aa00640 1 -- 172.21.15.40:0/2566877683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96340d1760 msgr2=0x7f96340cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:38:59.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f962aa00640 1 --2- 172.21.15.40:0/2566877683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96340d1760 0x7f96340cea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f962aa00640 1 -- 172.21.15.40:0/2566877683 >> v1:172.21.15.53:6789/0 conn(0x7f96340b22d0 legacy=0x7f96340b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f962aa00640 1 -- 172.21.15.40:0/2566877683 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96340cef40 con 0x7f96340b4330 2024-03-24T00:38:59.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.345+0000 7f9632a00640 1 -- 172.21.15.40:0/2566877683 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834925225 0 0) 0x7f96340cf2d0 con 0x7f96340b22d0 2024-03-24T00:38:59.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f962aa00640 1 --2- 172.21.15.40:0/2566877683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96340b4330 0x7f96340ce4c0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f9624009870 tx=0x7f962401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ef3aa73a2d1c91 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f9632a00640 1 -- 172.21.15.40:0/2566877683 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96240a7070 con 0x7f96340b4330 2024-03-24T00:38:59.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f9632a00640 1 -- 172.21.15.40:0/2566877683 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96240b0070 con 0x7f96340b4330 2024-03-24T00:38:59.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f9632a00640 1 -- 172.21.15.40:0/2566877683 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96240ab890 con 0x7f96340b4330 2024-03-24T00:38:59.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f963a600640 1 -- 172.21.15.40:0/2566877683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96340b4330 msgr2=0x7f96340ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f963a600640 1 --2- 172.21.15.40:0/2566877683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96340b4330 0x7f96340ce4c0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f9624009870 tx=0x7f962401c560 comp rx=0 tx=0).stop 2024-03-24T00:38:59.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f963a600640 1 -- 172.21.15.40:0/2566877683 shutdown_connections 2024-03-24T00:38:59.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f963a600640 1 --2- 172.21.15.40:0/2566877683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96340d1760 0x7f96340cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f963a600640 1 --2- 172.21.15.40:0/2566877683 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96340b4330 0x7f96340ce4c0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f963a600640 1 -- 172.21.15.40:0/2566877683 >> 172.21.15.40:0/2566877683 conn(0x7f96340b3810 msgr2=0x7f96340b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:59.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f963a600640 1 -- 172.21.15.40:0/2566877683 shutdown_connections 2024-03-24T00:38:59.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f963a600640 1 -- 172.21.15.40:0/2566877683 wait complete. 2024-03-24T00:38:59.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f963a600640 1 Processor -- start 2024-03-24T00:38:59.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.349+0000 7f963a600640 1 -- start start 2024-03-24T00:38:59.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f963a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96340b22d0 0x7f963415be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f963a600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96340b4330 0x7f963415e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f963a600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96340d1760 0x7f963415ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f963a600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96340ccfb0 con 0x7f96340b22d0 2024-03-24T00:38:59.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f963a600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96340cce30 con 0x7f96340b4330 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f963a600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96340cd130 con 0x7f96340d1760 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f9633400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96340b22d0 0x7f963415be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f9633400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96340b22d0 0x7f963415be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34360/0 (socket says 172.21.15.40:34360) 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f9633400640 1 -- 172.21.15.40:0/926938550 learned_addr learned my addr 172.21.15.40:0/926938550 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f9633e00640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96340d1760 0x7f963415ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f962aa00640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96340b4330 0x7f963415e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f9633400640 1 -- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96340d1760 msgr2=0x7f963415ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f9633400640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96340d1760 0x7f963415ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f9633400640 1 -- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96340b4330 msgr2=0x7f963415e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f9633400640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96340b4330 0x7f963415e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f9633400640 1 -- 172.21.15.40:0/926938550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963415ffc0 con 0x7f96340b22d0 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f9633400640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96340b22d0 0x7f963415be50 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f961c00c550 tx=0x7f961c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f962be00640 1 -- 172.21.15.40:0/926938550 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c0052d0 con 0x7f96340b22d0 2024-03-24T00:38:59.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f962be00640 1 -- 172.21.15.40:0/926938550 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961c005470 con 0x7f96340b22d0 2024-03-24T00:38:59.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f963a600640 1 -- 172.21.15.40:0/926938550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96341602a0 con 0x7f96340b22d0 2024-03-24T00:38:59.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f962be00640 1 -- 172.21.15.40:0/926938550 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961c012870 con 0x7f96340b22d0 2024-03-24T00:38:59.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f963a600640 1 -- 172.21.15.40:0/926938550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9634160860 con 0x7f96340b22d0 2024-03-24T00:38:59.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f963a600640 1 -- 172.21.15.40:0/926938550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9634160b90 con 0x7f96340b22d0 2024-03-24T00:38:59.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.353+0000 7f962be00640 1 -- 172.21.15.40:0/926938550 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f961c012a10 con 0x7f96340b22d0 2024-03-24T00:38:59.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.357+0000 7f962be00640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9610067660 0x7f9610069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.357+0000 7f962be00640 1 -- 172.21.15.40:0/926938550 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f961c0853e0 con 0x7f96340b22d0 2024-03-24T00:38:59.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.357+0000 7f962aa00640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9610067660 0x7f9610069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.357+0000 7f962aa00640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9610067660 0x7f9610069b20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f96240af000 tx=0x7f96240a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.361+0000 7f962be00640 1 -- 172.21.15.40:0/926938550 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f961c055b40 con 0x7f96340b22d0 2024-03-24T00:38:59.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.521+0000 7f963a600640 1 -- 172.21.15.40:0/926938550 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9634068170 con 0x7f96340b22d0 2024-03-24T00:38:59.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.525+0000 7f962be00640 1 -- 172.21.15.40:0/926938550 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v66) v1 ==== 104+0+0 (secure 0 0 0) 0x7f961c059be0 con 0x7f96340b22d0 2024-03-24T00:38:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.529+0000 7f963a600640 1 -- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9610067660 msgr2=0x7f9610069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.529+0000 7f963a600640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9610067660 0x7f9610069b20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f96240af000 tx=0x7f96240a7000 comp rx=0 tx=0).stop 2024-03-24T00:38:59.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.529+0000 7f963a600640 1 -- 172.21.15.40:0/926938550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96340b22d0 msgr2=0x7f963415be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.529+0000 7f963a600640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96340b22d0 0x7f963415be50 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f961c00c550 tx=0x7f961c00ca10 comp rx=0 tx=0).stop 2024-03-24T00:38:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.529+0000 7f963a600640 1 -- 172.21.15.40:0/926938550 shutdown_connections 2024-03-24T00:38:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.529+0000 7f963a600640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f9610067660 0x7f9610069b20 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.529+0000 7f963a600640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f96340d1760 0x7f963415ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.529+0000 7f963a600640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96340b4330 0x7f963415e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.529+0000 7f963a600640 1 --2- 172.21.15.40:0/926938550 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f96340b22d0 0x7f963415be50 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.529+0000 7f963a600640 1 -- 172.21.15.40:0/926938550 >> 172.21.15.40:0/926938550 conn(0x7f96340b3810 msgr2=0x7f963415dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.529+0000 7f963a600640 1 -- 172.21.15.40:0/926938550 shutdown_connections 2024-03-24T00:38:59.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.529+0000 7f963a600640 1 -- 172.21.15.40:0/926938550 wait complete. 2024-03-24T00:38:59.551 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_pin' 2024-03-24T00:38:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3d02600640 1 Processor -- start 2024-03-24T00:38:59.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3d02600640 1 -- start start 2024-03-24T00:38:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3d02600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3cfc0b2aa0 0x7f3cfc0d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3d02600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc0b2870 con 0x7f3cfc0b2490 2024-03-24T00:38:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3d02600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc0b3190 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3d02600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cfc06b6d0 con 0x7f3cfc0b2aa0 2024-03-24T00:38:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3cfb400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3cfc0b2490 0x7f3cfc0cfe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39728/0 (socket says 172.21.15.40:39728) 2024-03-24T00:38:59.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3cfb400640 1 -- 172.21.15.40:0/2865938771 learned_addr learned my addr 172.21.15.40:0/2865938771 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:38:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3cf3e00640 1 --2- 172.21.15.40:0/2865938771 >> v2:172.21.15.53:3300/0 conn(0x7f3cfc0b2aa0 0x7f3cfc0d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3cfaa00640 1 -- 172.21.15.40:0/2865938771 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508216258 0 0) 0x7f3cfc0b2870 con 0x7f3cfc0b2490 2024-03-24T00:38:59.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3cfaa00640 1 -- 172.21.15.40:0/2865938771 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ce0003530 con 0x7f3cfc0b2490 2024-03-24T00:38:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3cfaa00640 1 -- 172.21.15.40:0/2865938771 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487692942 0 0) 0x7f3cfc0b3190 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3cfaa00640 1 -- 172.21.15.40:0/2865938771 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cfc0b2870 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3cfaa00640 1 -- 172.21.15.40:0/2865938771 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3348067040 0 0) 0x7f3ce0003530 con 0x7f3cfc0b2490 2024-03-24T00:38:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3cf3e00640 1 -- 172.21.15.40:0/2865938771 >> v1:172.21.15.53:6789/0 conn(0x7f3cfc0d0a80 legacy=0x7f3cfc0cc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3cf3e00640 1 -- 172.21.15.40:0/2865938771 >> v1:172.21.15.40:6789/0 conn(0x7f3cfc0b2490 legacy=0x7f3cfc0cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.689+0000 7f3cf3e00640 1 -- 172.21.15.40:0/2865938771 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc0b3470 con 0x7f3cfc0b2aa0 2024-03-24T00:38:59.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.693+0000 7f3cf3e00640 1 --2- 172.21.15.40:0/2865938771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cfc0b2aa0 0x7f3cfc0d0540 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40097c0 tx=0x7f3ce401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19c6e510bab64602 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.693+0000 7f3cfaa00640 1 -- 172.21.15.40:0/2865938771 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce40b0070 con 0x7f3cfc0b2aa0 2024-03-24T00:38:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.693+0000 7f3cfaa00640 1 -- 172.21.15.40:0/2865938771 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce40ab400 con 0x7f3cfc0b2aa0 2024-03-24T00:38:59.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.693+0000 7f3cfaa00640 1 -- 172.21.15.40:0/2865938771 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cec001680 con 0x7f3cfc0b2aa0 2024-03-24T00:38:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.693+0000 7f3d02600640 1 -- 172.21.15.40:0/2865938771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cfc0b2aa0 msgr2=0x7f3cfc0d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.693+0000 7f3d02600640 1 --2- 172.21.15.40:0/2865938771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cfc0b2aa0 0x7f3cfc0d0540 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40097c0 tx=0x7f3ce401c3c0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.693+0000 7f3d02600640 1 -- 172.21.15.40:0/2865938771 shutdown_connections 2024-03-24T00:38:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.693+0000 7f3d02600640 1 --2- 172.21.15.40:0/2865938771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cfc0b2aa0 0x7f3cfc0d0540 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.693+0000 7f3d02600640 1 -- 172.21.15.40:0/2865938771 >> 172.21.15.40:0/2865938771 conn(0x7f3cfc0d1d10 msgr2=0x7f3cfc0d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:38:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.693+0000 7f3d02600640 1 -- 172.21.15.40:0/2865938771 shutdown_connections 2024-03-24T00:38:59.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.693+0000 7f3d02600640 1 -- 172.21.15.40:0/2865938771 wait complete. 2024-03-24T00:38:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3d02600640 1 Processor -- start 2024-03-24T00:38:59.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3d02600640 1 -- start start 2024-03-24T00:38:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3d02600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cfc0b2490 0x7f3cfc15f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3d02600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3cfc0b2aa0 0x7f3cfc15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3d02600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0d0a80 0x7f3cfc0b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3d02600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc16bfd0 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3d02600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0cd650 con 0x7f3cfc0b2490 2024-03-24T00:38:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3d02600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc16c150 con 0x7f3cfc0b2aa0 2024-03-24T00:38:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cfbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0d0a80 0x7f3cfc0b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cfbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0d0a80 0x7f3cfc0b11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34368/0 (socket says 172.21.15.40:34368) 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cfbe00640 1 -- 172.21.15.40:0/764417384 learned_addr learned my addr 172.21.15.40:0/764417384 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cfb400640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cfc0b2490 0x7f3cfc15f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cf3e00640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3cfc0b2aa0 0x7f3cfc15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cfbe00640 1 -- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3cfc0b2aa0 msgr2=0x7f3cfc15a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cfbe00640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3cfc0b2aa0 0x7f3cfc15a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cfbe00640 1 -- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cfc0b2490 msgr2=0x7f3cfc15f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cfbe00640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cfc0b2490 0x7f3cfc15f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cfbe00640 1 -- 172.21.15.40:0/764417384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc15fa40 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cfb400640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cfc0b2490 0x7f3cfc15f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cf3e00640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3cfc0b2aa0 0x7f3cfc15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cfbe00640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0d0a80 0x7f3cfc0b11a0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400c790 tx=0x7f3cf400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cf3400640 1 -- 172.21.15.40:0/764417384 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf4003b80 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cf3400640 1 -- 172.21.15.40:0/764417384 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf4003d20 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3d02600640 1 -- 172.21.15.40:0/764417384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cfc15f7a0 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3cf3400640 1 -- 172.21.15.40:0/764417384 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf400aa50 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3d02600640 1 -- 172.21.15.40:0/764417384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc0b06c0 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.697+0000 7f3d02600640 1 -- 172.21.15.40:0/764417384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc068170 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.701+0000 7f3cf3400640 1 -- 172.21.15.40:0/764417384 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3cf400abf0 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.701+0000 7f3cf3400640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3cd8067660 0x7f3cd8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:38:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.701+0000 7f3cf3400640 1 -- 172.21.15.40:0/764417384 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3cf4085700 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.701+0000 7f3cfb400640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3cd8067660 0x7f3cd8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:38:59.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.701+0000 7f3cfb400640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3cd8067660 0x7f3cd8069b20 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3cec001390 tx=0x7f3cec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:38:59.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.705+0000 7f3cf3400640 1 -- 172.21.15.40:0/764417384 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3cf4056530 con 0x7f3cfc0d0a80 2024-03-24T00:38:59.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:38:59.924+0000 7f3d02600640 1 -- 172.21.15.40:0/764417384 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_pin"]} v 0) v1 -- 0x7f3cfc0cdc50 con 0x7f3cfc0d0a80 2024-03-24T00:39:00.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3cf3400640 1 -- 172.21.15.40:0/764417384 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_dist_override_pin"]}]=0 v396) v1 ==== 159+0+0 (secure 0 0 0) 0x7f3cf4055e30 con 0x7f3cfc0d0a80 2024-03-24T00:39:00.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3d02600640 1 -- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3cd8067660 msgr2=0x7f3cd8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3d02600640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3cd8067660 0x7f3cd8069b20 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3cec001390 tx=0x7f3cec009040 comp rx=0 tx=0).stop 2024-03-24T00:39:00.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3d02600640 1 -- 172.21.15.40:0/764417384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0d0a80 msgr2=0x7f3cfc0b11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3d02600640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0d0a80 0x7f3cfc0b11a0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f3cf400c790 tx=0x7f3cf400cc50 comp rx=0 tx=0).stop 2024-03-24T00:39:00.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3d02600640 1 -- 172.21.15.40:0/764417384 shutdown_connections 2024-03-24T00:39:00.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3d02600640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3cd8067660 0x7f3cd8069b20 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3d02600640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3cfc0d0a80 0x7f3cfc0b11a0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3d02600640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3cfc0b2aa0 0x7f3cfc15a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3d02600640 1 --2- 172.21.15.40:0/764417384 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cfc0b2490 0x7f3cfc15f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3d02600640 1 -- 172.21.15.40:0/764417384 >> 172.21.15.40:0/764417384 conn(0x7f3cfc0d1d10 msgr2=0x7f3cfc0cae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:00.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3d02600640 1 -- 172.21.15.40:0/764417384 shutdown_connections 2024-03-24T00:39:00.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.096+0000 7f3d02600640 1 -- 172.21.15.40:0/764417384 wait complete. 2024-03-24T00:39:00.122 INFO:tasks.cephfs_test_runner:That an export pin overrides an ephemerally pinned directory. ... ok 2024-03-24T00:39:00.122 INFO:tasks.cephfs_test_runner:Starting test: test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins) That ephemerally pinned subtrees are somewhat evenly distributed. 2024-03-24T00:39:00.122 INFO:tasks.cephfs_test_runner:test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:39:00.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:00.264 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4de2400640 1 Processor -- start 2024-03-24T00:39:00.265 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4de2400640 1 -- start start 2024-03-24T00:39:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4de2400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc13ec80 0x7f4ddc1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4de2400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4ddc0ce800 0x7f4ddc071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4de2400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ddc070e00 con 0x7f4ddc1419f0 2024-03-24T00:39:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4de2400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ddc141e50 con 0x7f4ddc0ce800 2024-03-24T00:39:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4de2400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc0cee20 con 0x7f4ddc13ec80 2024-03-24T00:39:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4dd2a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f4ddc1419f0 0x7f4ddc0ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39742/0 (socket says 172.21.15.40:39742) 2024-03-24T00:39:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4dd2a00640 1 -- 172.21.15.40:0/763521393 learned_addr learned my addr 172.21.15.40:0/763521393 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddbe00640 1 --2- 172.21.15.40:0/763521393 >> v2:172.21.15.53:3300/0 conn(0x7f4ddc0ce800 0x7f4ddc071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddb400640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc13ec80 0x7f4ddc1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686951915 0 0) 0x7f4ddc070e00 con 0x7f4ddc1419f0 2024-03-24T00:39:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dc0003530 con 0x7f4ddc1419f0 2024-03-24T00:39:00.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2814128841 0 0) 0x7f4dc0003530 con 0x7f4ddc1419f0 2024-03-24T00:39:00.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc13ec80 msgr2=0x7f4ddc1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:00.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddaa00640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc13ec80 0x7f4ddc1414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 >> v2:172.21.15.53:3300/0 conn(0x7f4ddc0ce800 msgr2=0x7f4ddc071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:00.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddaa00640 1 --2- 172.21.15.40:0/763521393 >> v2:172.21.15.53:3300/0 conn(0x7f4ddc0ce800 0x7f4ddc071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc0cf100 con 0x7f4ddc1419f0 2024-03-24T00:39:00.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f4dc4001680 con 0x7f4ddc1419f0 2024-03-24T00:39:00.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddb400640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc13ec80 0x7f4ddc1414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:00.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.260+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 >> v1:172.21.15.40:6789/0 conn(0x7f4ddc1419f0 legacy=0x7f4ddc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.269 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddaa00640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc00039c0 0x7f4dc0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddaa00640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc13ec80 0x7f4dc0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddaa00640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4dc0004d50 0x7f4dc00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc0cee20 con 0x7f4dc0004d50 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc000b9d0 con 0x7f4dc00039c0 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc000bb50 con 0x7f4ddc13ec80 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddbe00640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4dc0004d50 0x7f4dc00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4dd2a00640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc13ec80 0x7f4dc0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddb400640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc00039c0 0x7f4dc0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddbe00640 1 -- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc13ec80 msgr2=0x7f4dc0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddbe00640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc13ec80 0x7f4dc0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddbe00640 1 -- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc00039c0 msgr2=0x7f4dc0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddbe00640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc00039c0 0x7f4dc0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddbe00640 1 -- 172.21.15.40:0/763521393 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc0003530 con 0x7f4dc0004d50 2024-03-24T00:39:00.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddb400640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc00039c0 0x7f4dc0003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddbe00640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4dc0004d50 0x7f4dc00051d0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4009fc0 tx=0x7f4dd401ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a25a1ae609d22c35 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd40b2070 con 0x7f4dc0004d50 2024-03-24T00:39:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd40a5cd0 con 0x7f4dc0004d50 2024-03-24T00:39:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.264+0000 7f4ddaa00640 1 -- 172.21.15.40:0/763521393 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4009c10 con 0x7f4dc0004d50 2024-03-24T00:39:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 -- 172.21.15.40:0/763521393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4dc0004d50 msgr2=0x7f4dc00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 --2- 172.21.15.40:0/763521393 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4dc0004d50 0x7f4dc00051d0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4009fc0 tx=0x7f4dd401ccc0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4ddbe00640 1 -- 172.21.15.40:0/763521393 reap_dead start 2024-03-24T00:39:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 -- 172.21.15.40:0/763521393 shutdown_connections 2024-03-24T00:39:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 -- 172.21.15.40:0/763521393 >> 172.21.15.40:0/763521393 conn(0x7f4ddc0b17b0 msgr2=0x7f4ddc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 -- 172.21.15.40:0/763521393 shutdown_connections 2024-03-24T00:39:00.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 -- 172.21.15.40:0/763521393 wait complete. 2024-03-24T00:39:00.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 Processor -- start 2024-03-24T00:39:00.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 -- start start 2024-03-24T00:39:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ddc13a4d0 0x7f4ddc0ad4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc0ac640 0x7f4ddc06cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0ab0a0 0x7f4ddc06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc1414b0 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc13f060 con 0x7f4ddc0ab0a0 2024-03-24T00:39:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4de2400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc141630 con 0x7f4ddc0ac640 2024-03-24T00:39:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4ddb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ddc13a4d0 0x7f4ddc0ad4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4ddb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ddc13a4d0 0x7f4ddc0ad4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34384/0 (socket says 172.21.15.40:34384) 2024-03-24T00:39:00.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4ddb400640 1 -- 172.21.15.40:0/3017129201 learned_addr learned my addr 172.21.15.40:0/3017129201 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:00.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4ddbe00640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0ab0a0 0x7f4ddc06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.268+0000 7f4dd2a00640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc0ac640 0x7f4ddc06cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4ddb400640 1 -- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc0ac640 msgr2=0x7f4ddc06cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4ddb400640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc0ac640 0x7f4ddc06cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4ddb400640 1 -- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0ab0a0 msgr2=0x7f4ddc06d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4ddb400640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0ab0a0 0x7f4ddc06d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4ddb400640 1 -- 172.21.15.40:0/3017129201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc0ab5a0 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4dd2a00640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc0ac640 0x7f4ddc06cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:00.276 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4ddb400640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ddc13a4d0 0x7f4ddc0ad4a0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0115f0 tx=0x7f4dcc011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4dd3e00640 1 -- 172.21.15.40:0/3017129201 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc00f8f0 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4dd3e00640 1 -- 172.21.15.40:0/3017129201 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc00fa90 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4dd3e00640 1 -- 172.21.15.40:0/3017129201 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc00fe40 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4de2400640 1 -- 172.21.15.40:0/3017129201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ddc0ab880 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4de2400640 1 -- 172.21.15.40:0/3017129201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ddc0abe40 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4de2400640 1 -- 172.21.15.40:0/3017129201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ddc068170 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4dd3e00640 1 -- 172.21.15.40:0/3017129201 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f4dcc015460 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4dd3e00640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4db8067660 0x7f4db8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.272+0000 7f4dd3e00640 1 -- 172.21.15.40:0/3017129201 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f4dcc086020 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.276+0000 7f4dd2a00640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4db8067660 0x7f4db8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.279 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.276+0000 7f4dd2a00640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4db8067660 0x7f4db8069b20 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc071ab0 tx=0x7f4dc4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.276+0000 7f4dd3e00640 1 -- 172.21.15.40:0/3017129201 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4dcc019030 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.446 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.440+0000 7f4de2400640 1 -- 172.21.15.40:0/3017129201 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ddc072b90 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4dd3e00640 1 -- 172.21.15.40:0/3017129201 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+13224 (secure 0 0 0) 0x7f4dcc056e50 con 0x7f4ddc13a4d0 2024-03-24T00:39:00.447 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:00.447 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":111,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:38:58.090948+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6834/3877615536":"2024-03-25T00:38:55.082027+0000","172.21.15.53:6837/8812937":"2024-03-25T00:38:55.082052+0000","172.21.15.53:6835/8812937":"2024-03-25T00:38:55.082052+0000","172.21.15.53:6836/3877615536":"2024-03-25T00:38:55.082027+0000","172.21.15.40:6834/3883043397":"2024-03-25T00:38:55.082045+0000","172.21.15.40:6833/3883043397":"2024-03-25T00:38:55.082045+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4de2400640 1 -- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4db8067660 msgr2=0x7f4db8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4de2400640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4db8067660 0x7f4db8069b20 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc071ab0 tx=0x7f4dc4009040 comp rx=0 tx=0).stop 2024-03-24T00:39:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4de2400640 1 -- 172.21.15.40:0/3017129201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ddc13a4d0 msgr2=0x7f4ddc0ad4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4de2400640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ddc13a4d0 0x7f4ddc0ad4a0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc0115f0 tx=0x7f4dcc011ab0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4de2400640 1 -- 172.21.15.40:0/3017129201 shutdown_connections 2024-03-24T00:39:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4de2400640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4db8067660 0x7f4db8069b20 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4de2400640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ddc0ab0a0 0x7f4ddc06d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4de2400640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ddc0ac640 0x7f4ddc06cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4de2400640 1 --2- 172.21.15.40:0/3017129201 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f4ddc13a4d0 0x7f4ddc0ad4a0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4de2400640 1 -- 172.21.15.40:0/3017129201 >> 172.21.15.40:0/3017129201 conn(0x7f4ddc0b17b0 msgr2=0x7f4ddc0b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4de2400640 1 -- 172.21.15.40:0/3017129201 shutdown_connections 2024-03-24T00:39:00.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.444+0000 7f4de2400640 1 -- 172.21.15.40:0/3017129201 wait complete. 2024-03-24T00:39:00.468 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:39:00.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:00.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.608+0000 7fa9c0e00640 1 Processor -- start 2024-03-24T00:39:00.613 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.608+0000 7fa9c0e00640 1 -- start start 2024-03-24T00:39:00.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.608+0000 7fa9c0e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa9bc15ac60 0x7fa9bc0b2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.608+0000 7fa9c0e00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc0b3260 con 0x7fa9bc071b00 2024-03-24T00:39:00.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.608+0000 7fa9c0e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9bc0b3540 con 0x7fa9bc06cb80 2024-03-24T00:39:00.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.608+0000 7fa9c0e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9bc0b3e70 con 0x7fa9bc15ac60 2024-03-24T00:39:00.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.608+0000 7fa9b9e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa9bc15ac60 0x7fa9bc0b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.608+0000 7fa9ba800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa9bc071b00 0x7fa9bc0ccd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39750/0 (socket says 172.21.15.40:39750) 2024-03-24T00:39:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.608+0000 7fa9ba800640 1 -- 172.21.15.40:0/3811523456 learned_addr learned my addr 172.21.15.40:0/3811523456 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.608+0000 7fa9b9e00640 1 -- 172.21.15.40:0/3811523456 >> v1:172.21.15.53:6789/0 conn(0x7fa9bc06cb80 legacy=0x7fa9bc0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.608+0000 7fa9b9e00640 1 -- 172.21.15.40:0/3811523456 >> v1:172.21.15.40:6789/0 conn(0x7fa9bc071b00 legacy=0x7fa9bc0ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.608+0000 7fa9b8a00640 1 -- 172.21.15.40:0/3811523456 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528181271 0 0) 0x7fa9bc0b3260 con 0x7fa9bc071b00 2024-03-24T00:39:00.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.612+0000 7fa9b9e00640 1 -- 172.21.15.40:0/3811523456 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9bc0b3820 con 0x7fa9bc15ac60 2024-03-24T00:39:00.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.612+0000 7fa9b9e00640 1 --2- 172.21.15.40:0/3811523456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc15ac60 0x7fa9bc0b2960 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0097c0 tx=0x7fa9ac01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b8454a70ba88a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.612+0000 7fa9b8a00640 1 -- 172.21.15.40:0/3811523456 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac0b0070 con 0x7fa9bc15ac60 2024-03-24T00:39:00.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.612+0000 7fa9b8a00640 1 -- 172.21.15.40:0/3811523456 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9ac0ab600 con 0x7fa9bc15ac60 2024-03-24T00:39:00.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.612+0000 7fa9b8a00640 1 -- 172.21.15.40:0/3811523456 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac0ab9d0 con 0x7fa9bc15ac60 2024-03-24T00:39:00.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.612+0000 7fa9c0e00640 1 -- 172.21.15.40:0/3811523456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc15ac60 msgr2=0x7fa9bc0b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.612+0000 7fa9c0e00640 1 --2- 172.21.15.40:0/3811523456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc15ac60 0x7fa9bc0b2960 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0097c0 tx=0x7fa9ac01c3e0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.612+0000 7fa9c0e00640 1 -- 172.21.15.40:0/3811523456 shutdown_connections 2024-03-24T00:39:00.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.612+0000 7fa9c0e00640 1 --2- 172.21.15.40:0/3811523456 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc15ac60 0x7fa9bc0b2960 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.612+0000 7fa9c0e00640 1 -- 172.21.15.40:0/3811523456 >> 172.21.15.40:0/3811523456 conn(0x7fa9bc0b1660 msgr2=0x7fa9bc0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:00.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.612+0000 7fa9c0e00640 1 -- 172.21.15.40:0/3811523456 shutdown_connections 2024-03-24T00:39:00.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.612+0000 7fa9c0e00640 1 -- 172.21.15.40:0/3811523456 wait complete. 2024-03-24T00:39:00.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9c0e00640 1 Processor -- start 2024-03-24T00:39:00.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9c0e00640 1 -- start start 2024-03-24T00:39:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9c0e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9bc06cb80 0x7fa9bc15cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9c0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9bc071b00 0x7fa9bc145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9c0e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc15ac60 0x7fa9bc146010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9c0e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc0b3020 con 0x7fa9bc15ac60 2024-03-24T00:39:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9c0e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc0b2ea0 con 0x7fa9bc071b00 2024-03-24T00:39:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9c0e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9bc15b040 con 0x7fa9bc06cb80 2024-03-24T00:39:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9ba800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc15ac60 0x7fa9bc146010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9ba800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc15ac60 0x7fa9bc146010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34410/0 (socket says 172.21.15.40:34410) 2024-03-24T00:39:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9ba800640 1 -- 172.21.15.40:0/1054789468 learned_addr learned my addr 172.21.15.40:0/1054789468 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9b9400640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9bc071b00 0x7fa9bc145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9b9e00640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9bc06cb80 0x7fa9bc15cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9ba800640 1 -- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9bc06cb80 msgr2=0x7fa9bc15cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9ba800640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9bc06cb80 0x7fa9bc15cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9ba800640 1 -- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9bc071b00 msgr2=0x7fa9bc145ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9ba800640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9bc071b00 0x7fa9bc145ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9ba800640 1 -- 172.21.15.40:0/1054789468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9bc15d8e0 con 0x7fa9bc15ac60 2024-03-24T00:39:00.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9ba800640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc15ac60 0x7fa9bc146010 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400c7a0 tx=0x7fa9b400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9b2000640 1 -- 172.21.15.40:0/1054789468 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b4018070 con 0x7fa9bc15ac60 2024-03-24T00:39:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9b2000640 1 -- 172.21.15.40:0/1054789468 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b400a640 con 0x7fa9bc15ac60 2024-03-24T00:39:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9b2000640 1 -- 172.21.15.40:0/1054789468 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9b400aab0 con 0x7fa9bc15ac60 2024-03-24T00:39:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9c0e00640 1 -- 172.21.15.40:0/1054789468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9bc15de70 con 0x7fa9bc15ac60 2024-03-24T00:39:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.616+0000 7fa9c0e00640 1 -- 172.21.15.40:0/1054789468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9bc15b350 con 0x7fa9bc15ac60 2024-03-24T00:39:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.620+0000 7fa9c0e00640 1 -- 172.21.15.40:0/1054789468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9bc068170 con 0x7fa9bc15ac60 2024-03-24T00:39:00.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.620+0000 7fa9b2000640 1 -- 172.21.15.40:0/1054789468 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa9b401b070 con 0x7fa9bc15ac60 2024-03-24T00:39:00.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.620+0000 7fa9b2000640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa99c067660 0x7fa99c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.620+0000 7fa9b2000640 1 -- 172.21.15.40:0/1054789468 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fa9b4084b90 con 0x7fa9bc15ac60 2024-03-24T00:39:00.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.620+0000 7fa9b9e00640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa99c067660 0x7fa99c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.620+0000 7fa9b9e00640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa99c067660 0x7fa99c069b20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0ad000 tx=0x7fa9ac0a4f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.624+0000 7fa9b2000640 1 -- 172.21.15.40:0/1054789468 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa9b4055400 con 0x7fa9bc15ac60 2024-03-24T00:39:00.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.788+0000 7fa9c0e00640 1 -- 172.21.15.40:0/1054789468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9bc15e200 con 0x7fa9bc15ac60 2024-03-24T00:39:00.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.788+0000 7fa9b2000640 1 -- 172.21.15.40:0/1054789468 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9b40594a0 con 0x7fa9bc15ac60 2024-03-24T00:39:00.794 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:39:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.792+0000 7fa9c0e00640 1 -- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa99c067660 msgr2=0x7fa99c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.792+0000 7fa9c0e00640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa99c067660 0x7fa99c069b20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa9ac0ad000 tx=0x7fa9ac0a4f90 comp rx=0 tx=0).stop 2024-03-24T00:39:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.792+0000 7fa9c0e00640 1 -- 172.21.15.40:0/1054789468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc15ac60 msgr2=0x7fa9bc146010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.792+0000 7fa9c0e00640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc15ac60 0x7fa9bc146010 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400c7a0 tx=0x7fa9b400cc60 comp rx=0 tx=0).stop 2024-03-24T00:39:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.792+0000 7fa9c0e00640 1 -- 172.21.15.40:0/1054789468 shutdown_connections 2024-03-24T00:39:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.792+0000 7fa9c0e00640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa99c067660 0x7fa99c069b20 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.792+0000 7fa9c0e00640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa9bc15ac60 0x7fa9bc146010 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.792+0000 7fa9c0e00640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9bc071b00 0x7fa9bc145ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.792+0000 7fa9c0e00640 1 --2- 172.21.15.40:0/1054789468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9bc06cb80 0x7fa9bc15cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.792+0000 7fa9c0e00640 1 -- 172.21.15.40:0/1054789468 >> 172.21.15.40:0/1054789468 conn(0x7fa9bc0b1660 msgr2=0x7fa9bc153420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.792+0000 7fa9c0e00640 1 -- 172.21.15.40:0/1054789468 shutdown_connections 2024-03-24T00:39:00.796 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.792+0000 7fa9c0e00640 1 -- 172.21.15.40:0/1054789468 wait complete. 2024-03-24T00:39:00.814 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_distribution' 2024-03-24T00:39:00.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.948+0000 7f8053400640 1 Processor -- start 2024-03-24T00:39:00.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.948+0000 7f8053400640 1 -- start start 2024-03-24T00:39:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8053400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f804c13ec80 0x7f804c1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8053400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f804c1419f0 0x7f804c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8053400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804c070e00 con 0x7f804c0ce630 2024-03-24T00:39:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8053400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f804c141e50 con 0x7f804c1419f0 2024-03-24T00:39:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8053400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804c0cee20 con 0x7f804c13ec80 2024-03-24T00:39:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8051400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f804c0ce630 0x7f804c15f9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39762/0 (socket says 172.21.15.40:39762) 2024-03-24T00:39:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8051400640 1 -- 172.21.15.40:0/3417501012 learned_addr learned my addr 172.21.15.40:0/3417501012 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8043e00640 1 --2- 172.21.15.40:0/3417501012 >> v2:172.21.15.53:3300/0 conn(0x7f804c1419f0 0x7f804c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8050a00640 1 --2- 172.21.15.40:0/3417501012 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f804c13ec80 0x7f804c1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f804be00640 1 -- 172.21.15.40:0/3417501012 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845271641 0 0) 0x7f804c070e00 con 0x7f804c0ce630 2024-03-24T00:39:00.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f804be00640 1 -- 172.21.15.40:0/3417501012 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8030003530 con 0x7f804c0ce630 2024-03-24T00:39:00.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8043e00640 1 -- 172.21.15.40:0/3417501012 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f804c13ec80 msgr2=0x7f804c1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:00.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8043e00640 1 --2- 172.21.15.40:0/3417501012 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f804c13ec80 0x7f804c1414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8043e00640 1 -- 172.21.15.40:0/3417501012 >> v1:172.21.15.40:6789/0 conn(0x7f804c0ce630 legacy=0x7f804c15f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.958 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.952+0000 7f8043e00640 1 -- 172.21.15.40:0/3417501012 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c0cf100 con 0x7f804c1419f0 2024-03-24T00:39:00.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f8043e00640 1 --2- 172.21.15.40:0/3417501012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804c1419f0 0x7f804c0ce0f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f80340097c0 tx=0x7f803401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad993e49992e5d01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f804be00640 1 -- 172.21.15.40:0/3417501012 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80340aa620 con 0x7f804c1419f0 2024-03-24T00:39:00.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f804be00640 1 -- 172.21.15.40:0/3417501012 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80340af070 con 0x7f804c1419f0 2024-03-24T00:39:00.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f804be00640 1 -- 172.21.15.40:0/3417501012 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80340aa9e0 con 0x7f804c1419f0 2024-03-24T00:39:00.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f8053400640 1 -- 172.21.15.40:0/3417501012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804c1419f0 msgr2=0x7f804c0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f8053400640 1 --2- 172.21.15.40:0/3417501012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804c1419f0 0x7f804c0ce0f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f80340097c0 tx=0x7f803401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f8053400640 1 -- 172.21.15.40:0/3417501012 shutdown_connections 2024-03-24T00:39:00.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f8053400640 1 --2- 172.21.15.40:0/3417501012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804c1419f0 0x7f804c0ce0f0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f8053400640 1 --2- 172.21.15.40:0/3417501012 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f804c13ec80 0x7f804c1414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f8053400640 1 -- 172.21.15.40:0/3417501012 >> 172.21.15.40:0/3417501012 conn(0x7f804c0b17b0 msgr2=0x7f804c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:00.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f8053400640 1 -- 172.21.15.40:0/3417501012 shutdown_connections 2024-03-24T00:39:00.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f8053400640 1 -- 172.21.15.40:0/3417501012 wait complete. 2024-03-24T00:39:00.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f8053400640 1 Processor -- start 2024-03-24T00:39:00.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.956+0000 7f8053400640 1 -- start start 2024-03-24T00:39:00.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8053400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f804c0ce630 0x7f804c15b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8053400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f804c13ec80 0x7f804c139fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8053400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804c1419f0 0x7f804c0b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8053400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804c168ec0 con 0x7f804c0ce630 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8053400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f804c168d40 con 0x7f804c1419f0 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8053400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f804c169040 con 0x7f804c13ec80 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8050a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f804c0ce630 0x7f804c15b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8050a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f804c0ce630 0x7f804c15b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34424/0 (socket says 172.21.15.40:34424) 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8050a00640 1 -- 172.21.15.40:0/2018703968 learned_addr learned my addr 172.21.15.40:0/2018703968 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8051400640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804c1419f0 0x7f804c0b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8043e00640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f804c13ec80 0x7f804c139fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8050a00640 1 -- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f804c13ec80 msgr2=0x7f804c139fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8050a00640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f804c13ec80 0x7f804c139fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8050a00640 1 -- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804c1419f0 msgr2=0x7f804c0b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8050a00640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804c1419f0 0x7f804c0b2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:00.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8050a00640 1 -- 172.21.15.40:0/2018703968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804c1508a0 con 0x7f804c0ce630 2024-03-24T00:39:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8051400640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804c1419f0 0x7f804c0b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8050a00640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f804c0ce630 0x7f804c15b280 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f803c00e960 tx=0x7f803c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8049600640 1 -- 172.21.15.40:0/2018703968 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c017070 con 0x7f804c0ce630 2024-03-24T00:39:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8049600640 1 -- 172.21.15.40:0/2018703968 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803c00ca00 con 0x7f804c0ce630 2024-03-24T00:39:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8053400640 1 -- 172.21.15.40:0/2018703968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f804c150620 con 0x7f804c0ce630 2024-03-24T00:39:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8049600640 1 -- 172.21.15.40:0/2018703968 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c00cdb0 con 0x7f804c0ce630 2024-03-24T00:39:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8053400640 1 -- 172.21.15.40:0/2018703968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f804c0b4400 con 0x7f804c0ce630 2024-03-24T00:39:00.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.960+0000 7f8053400640 1 -- 172.21.15.40:0/2018703968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f804c068170 con 0x7f804c0ce630 2024-03-24T00:39:00.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.964+0000 7f8049600640 1 -- 172.21.15.40:0/2018703968 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f803c01a070 con 0x7f804c0ce630 2024-03-24T00:39:00.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.964+0000 7f8049600640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8028067690 0x7f8028069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:00.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.964+0000 7f8049600640 1 -- 172.21.15.40:0/2018703968 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f803c085de0 con 0x7f804c0ce630 2024-03-24T00:39:00.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.964+0000 7f8043e00640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8028067690 0x7f8028069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:00.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.964+0000 7f8043e00640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8028067690 0x7f8028069b50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f80340ae000 tx=0x7f80340a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:00.971 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:00.968+0000 7f8049600640 1 -- 172.21.15.40:0/2018703968 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f803c056d20 con 0x7f804c0ce630 2024-03-24T00:39:01.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:01.188+0000 7f8053400640 1 -- 172.21.15.40:0/2018703968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_distribution"]} v 0) v1 -- 0x7f804c1539e0 con 0x7f804c0ce630 2024-03-24T00:39:02.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.100+0000 7f8049600640 1 -- 172.21.15.40:0/2018703968 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_exports.TestEphemeralPins.test_ephemeral_pin_distribution"]}]=0 v398) v1 ==== 157+0+0 (secure 0 0 0) 0x7f803c056620 con 0x7f804c0ce630 2024-03-24T00:39:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.104+0000 7f8053400640 1 -- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8028067690 msgr2=0x7f8028069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.104+0000 7f8053400640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8028067690 0x7f8028069b50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f80340ae000 tx=0x7f80340a7000 comp rx=0 tx=0).stop 2024-03-24T00:39:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.104+0000 7f8053400640 1 -- 172.21.15.40:0/2018703968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f804c0ce630 msgr2=0x7f804c15b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.104+0000 7f8053400640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f804c0ce630 0x7f804c15b280 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f803c00e960 tx=0x7f803c00ee20 comp rx=0 tx=0).stop 2024-03-24T00:39:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.104+0000 7f8053400640 1 -- 172.21.15.40:0/2018703968 shutdown_connections 2024-03-24T00:39:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.104+0000 7f8053400640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8028067690 0x7f8028069b50 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.104+0000 7f8053400640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804c1419f0 0x7f804c0b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.104+0000 7f8053400640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f804c13ec80 0x7f804c139fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.104+0000 7f8053400640 1 --2- 172.21.15.40:0/2018703968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f804c0ce630 0x7f804c15b280 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.104+0000 7f8053400640 1 -- 172.21.15.40:0/2018703968 >> 172.21.15.40:0/2018703968 conn(0x7f804c0b17b0 msgr2=0x7f804c15e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:02.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.104+0000 7f8053400640 1 -- 172.21.15.40:0/2018703968 shutdown_connections 2024-03-24T00:39:02.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.104+0000 7f8053400640 1 -- 172.21.15.40:0/2018703968 wait complete. 2024-03-24T00:39:02.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-24T00:39:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fcaa5200640 1 Processor -- start 2024-03-24T00:39:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fcaa5200640 1 -- start start 2024-03-24T00:39:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fcaa5200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcaa015e170 0x7fcaa006f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fcaa5200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa00cfcb0 con 0x7fcaa00d0e60 2024-03-24T00:39:02.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fcaa5200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa006fe20 con 0x7fcaa00d0a10 2024-03-24T00:39:02.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fcaa5200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaa0070100 con 0x7fcaa015e170 2024-03-24T00:39:02.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fca9de00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fcaa015e170 0x7fcaa006f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.282 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fca9e800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcaa00d0e60 0x7fcaa015e550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39776/0 (socket says 172.21.15.40:39776) 2024-03-24T00:39:02.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fca9e800640 1 -- 172.21.15.40:0/75473978 learned_addr learned my addr 172.21.15.40:0/75473978 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:02.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fca9de00640 1 -- 172.21.15.40:0/75473978 >> v1:172.21.15.53:6789/0 conn(0x7fcaa00d0a10 legacy=0x7fcaa015f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:02.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fca9de00640 1 -- 172.21.15.40:0/75473978 >> v1:172.21.15.40:6789/0 conn(0x7fcaa00d0e60 legacy=0x7fcaa015e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fca9de00640 1 -- 172.21.15.40:0/75473978 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa00cff30 con 0x7fcaa015e170 2024-03-24T00:39:02.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.276+0000 7fca9d400640 1 -- 172.21.15.40:0/75473978 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963377021 0 0) 0x7fca98001120 con 0x7fcaa00d0e60 2024-03-24T00:39:02.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.280+0000 7fca9de00640 1 --2- 172.21.15.40:0/75473978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa015e170 0x7fcaa006f7b0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fca90009d90 tx=0x7fca9001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da4f960b54ccb23f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.280+0000 7fca9d400640 1 -- 172.21.15.40:0/75473978 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca900b2070 con 0x7fcaa015e170 2024-03-24T00:39:02.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.280+0000 7fca9d400640 1 -- 172.21.15.40:0/75473978 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca900a59c0 con 0x7fcaa015e170 2024-03-24T00:39:02.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.280+0000 7fca9d400640 1 -- 172.21.15.40:0/75473978 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca900a5d90 con 0x7fcaa015e170 2024-03-24T00:39:02.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.280+0000 7fcaa5200640 1 -- 172.21.15.40:0/75473978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa015e170 msgr2=0x7fcaa006f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.280+0000 7fcaa5200640 1 --2- 172.21.15.40:0/75473978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa015e170 0x7fcaa006f7b0 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fca90009d90 tx=0x7fca9001c900 comp rx=0 tx=0).stop 2024-03-24T00:39:02.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.280+0000 7fcaa5200640 1 -- 172.21.15.40:0/75473978 shutdown_connections 2024-03-24T00:39:02.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.280+0000 7fcaa5200640 1 --2- 172.21.15.40:0/75473978 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa015e170 0x7fcaa006f7b0 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.280+0000 7fcaa5200640 1 -- 172.21.15.40:0/75473978 >> 172.21.15.40:0/75473978 conn(0x7fcaa00b36a0 msgr2=0x7fcaa00b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:02.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.280+0000 7fcaa5200640 1 -- 172.21.15.40:0/75473978 shutdown_connections 2024-03-24T00:39:02.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.280+0000 7fcaa5200640 1 -- 172.21.15.40:0/75473978 wait complete. 2024-03-24T00:39:02.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fcaa5200640 1 Processor -- start 2024-03-24T00:39:02.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fcaa5200640 1 -- start start 2024-03-24T00:39:02.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fcaa5200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaa00d0a10 0x7fcaa00aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fcaa5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaa00d0e60 0x7fcaa014cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fcaa5200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa015e170 0x7fcaa00c7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fcaa5200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa016c040 con 0x7fcaa015e170 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fcaa5200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa016bec0 con 0x7fcaa00d0e60 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fcaa5200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa016c1c0 con 0x7fcaa00d0a10 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca9e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa015e170 0x7fcaa00c7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca9e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa015e170 0x7fcaa00c7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34446/0 (socket says 172.21.15.40:34446) 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca9e800640 1 -- 172.21.15.40:0/1376884196 learned_addr learned my addr 172.21.15.40:0/1376884196 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca95400640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaa00d0e60 0x7fcaa014cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca9de00640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaa00d0a10 0x7fcaa00aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca9e800640 1 -- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaa00d0a10 msgr2=0x7fcaa00aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca9e800640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaa00d0a10 0x7fcaa00aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca9e800640 1 -- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaa00d0e60 msgr2=0x7fcaa014cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca9e800640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaa00d0e60 0x7fcaa014cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca9e800640 1 -- 172.21.15.40:0/1376884196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa00b04f0 con 0x7fcaa015e170 2024-03-24T00:39:02.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca9e800640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa015e170 0x7fcaa00c7770 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fca9800c410 tx=0x7fca9800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca96a00640 1 -- 172.21.15.40:0/1376884196 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca980074e0 con 0x7fcaa015e170 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca96a00640 1 -- 172.21.15.40:0/1376884196 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca980047f0 con 0x7fcaa015e170 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fca96a00640 1 -- 172.21.15.40:0/1376884196 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca98007680 con 0x7fcaa015e170 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fcaa5200640 1 -- 172.21.15.40:0/1376884196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa00b07d0 con 0x7fcaa015e170 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.284+0000 7fcaa5200640 1 -- 172.21.15.40:0/1376884196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa014d500 con 0x7fcaa015e170 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.288+0000 7fcaa5200640 1 -- 172.21.15.40:0/1376884196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa0068170 con 0x7fcaa015e170 2024-03-24T00:39:02.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.288+0000 7fca96a00640 1 -- 172.21.15.40:0/1376884196 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fca98011070 con 0x7fcaa015e170 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.288+0000 7fca96a00640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca7c067690 0x7fca7c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.288+0000 7fca96a00640 1 -- 172.21.15.40:0/1376884196 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fca980851a0 con 0x7fcaa015e170 2024-03-24T00:39:02.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.288+0000 7fca9de00640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca7c067690 0x7fca7c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.288+0000 7fca9de00640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca7c067690 0x7fca7c069b50 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fca900af000 tx=0x7fca900a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.292+0000 7fca96a00640 1 -- 172.21.15.40:0/1376884196 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fcaa00cfcb0 con 0x7fcaa015e170 2024-03-24T00:39:02.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.456+0000 7fcaa5200640 1 -- 172.21.15.40:0/1376884196 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcaa0072f30 con 0x7fcaa015e170 2024-03-24T00:39:02.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fca96a00640 1 -- 172.21.15.40:0/1376884196 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v67)=0 v67) v1 ==== 122+0+0 (secure 0 0 0) 0x7fca980558f0 con 0x7fcaa015e170 2024-03-24T00:39:02.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fcaa5200640 1 -- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca7c067690 msgr2=0x7fca7c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fcaa5200640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca7c067690 0x7fca7c069b50 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fca900af000 tx=0x7fca900a7070 comp rx=0 tx=0).stop 2024-03-24T00:39:02.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fcaa5200640 1 -- 172.21.15.40:0/1376884196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa015e170 msgr2=0x7fcaa00c7770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fcaa5200640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa015e170 0x7fcaa00c7770 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fca9800c410 tx=0x7fca9800c8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fcaa5200640 1 -- 172.21.15.40:0/1376884196 shutdown_connections 2024-03-24T00:39:02.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fcaa5200640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fca7c067690 0x7fca7c069b50 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fcaa5200640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcaa015e170 0x7fcaa00c7770 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fcaa5200640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaa00d0e60 0x7fcaa014cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fcaa5200640 1 --2- 172.21.15.40:0/1376884196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaa00d0a10 0x7fcaa00aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fcaa5200640 1 -- 172.21.15.40:0/1376884196 >> 172.21.15.40:0/1376884196 conn(0x7fcaa00b36a0 msgr2=0x7fcaa006b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:02.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fcaa5200640 1 -- 172.21.15.40:0/1376884196 shutdown_connections 2024-03-24T00:39:02.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.460+0000 7fcaa5200640 1 -- 172.21.15.40:0/1376884196 wait complete. 2024-03-24T00:39:02.486 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:39:02.486 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:39:02.589 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:39:02.639 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:39:02.639 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:39:02.640 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:02.650 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:39:02.650 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:39:02.650 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:02.695 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:39:02.695 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:39:02.695 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:02.703 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:39:02.703 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:39:02.703 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:39:02.747 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:39:02.748 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-24T00:39:02.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:02.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60fe00640 1 Processor -- start 2024-03-24T00:39:02.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60fe00640 1 -- start start 2024-03-24T00:39:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60fe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd60806e620 0x7fd6080b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd60806ea00 0x7fd6080b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd608070f60 con 0x7fd6080b2ff0 2024-03-24T00:39:02.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60fe00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6080b4c00 con 0x7fd60806e620 2024-03-24T00:39:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60fe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6080b33d0 con 0x7fd60806ea00 2024-03-24T00:39:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd60806e620 0x7fd6080b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60d400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd60806e620 0x7fd6080b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34458/0 (socket says 172.21.15.40:34458) 2024-03-24T00:39:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60d400640 1 -- 172.21.15.40:0/2557252043 learned_addr learned my addr 172.21.15.40:0/2557252043 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60ca00640 1 --2- 172.21.15.40:0/2557252043 >> v2:172.21.15.53:3300/0 conn(0x7fd60806ea00 0x7fd6080b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60d400640 1 -- 172.21.15.40:0/2557252043 >> v2:172.21.15.53:3300/0 conn(0x7fd60806ea00 msgr2=0x7fd6080b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60d400640 1 --2- 172.21.15.40:0/2557252043 >> v2:172.21.15.53:3300/0 conn(0x7fd60806ea00 0x7fd6080b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60d400640 1 -- 172.21.15.40:0/2557252043 >> v1:172.21.15.53:6789/0 conn(0x7fd6080b2ff0 legacy=0x7fd6080ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.884+0000 7fd60d400640 1 -- 172.21.15.40:0/2557252043 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6080b3620 con 0x7fd60806e620 2024-03-24T00:39:02.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.888+0000 7fd60d400640 1 --2- 172.21.15.40:0/2557252043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60806e620 0x7fd6080b46c0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fd60806caa0 tx=0x7fd5fc01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ddca337c9899ab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.888+0000 7fd607e00640 1 -- 172.21.15.40:0/2557252043 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0ab660 con 0x7fd60806e620 2024-03-24T00:39:02.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.888+0000 7fd607e00640 1 -- 172.21.15.40:0/2557252043 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5fc0b0070 con 0x7fd60806e620 2024-03-24T00:39:02.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.888+0000 7fd607e00640 1 -- 172.21.15.40:0/2557252043 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0aba20 con 0x7fd60806e620 2024-03-24T00:39:02.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.888+0000 7fd60fe00640 1 -- 172.21.15.40:0/2557252043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60806e620 msgr2=0x7fd6080b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.888+0000 7fd60fe00640 1 --2- 172.21.15.40:0/2557252043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60806e620 0x7fd6080b46c0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7fd60806caa0 tx=0x7fd5fc01c460 comp rx=0 tx=0).stop 2024-03-24T00:39:02.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.888+0000 7fd60fe00640 1 -- 172.21.15.40:0/2557252043 shutdown_connections 2024-03-24T00:39:02.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.888+0000 7fd60fe00640 1 --2- 172.21.15.40:0/2557252043 >> v2:172.21.15.53:3300/0 conn(0x7fd60806ea00 0x7fd6080b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.888+0000 7fd60fe00640 1 --2- 172.21.15.40:0/2557252043 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60806e620 0x7fd6080b46c0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.888+0000 7fd60fe00640 1 -- 172.21.15.40:0/2557252043 >> 172.21.15.40:0/2557252043 conn(0x7fd6080b17b0 msgr2=0x7fd6080b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:02.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.888+0000 7fd60fe00640 1 -- 172.21.15.40:0/2557252043 shutdown_connections 2024-03-24T00:39:02.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.888+0000 7fd60fe00640 1 -- 172.21.15.40:0/2557252043 wait complete. 2024-03-24T00:39:02.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60fe00640 1 Processor -- start 2024-03-24T00:39:02.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60fe00640 1 -- start start 2024-03-24T00:39:02.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd60806e620 0x7fd6080b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60806ea00 0x7fd608159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6080b2ff0 0x7fd6081419f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd608144920 con 0x7fd60806ea00 2024-03-24T00:39:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6080b41a0 con 0x7fd60806e620 2024-03-24T00:39:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60813c8f0 con 0x7fd6080b2ff0 2024-03-24T00:39:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60806ea00 0x7fd608159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60ca00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60806ea00 0x7fd608159fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34472/0 (socket says 172.21.15.40:34472) 2024-03-24T00:39:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60ca00640 1 -- 172.21.15.40:0/3378557483 learned_addr learned my addr 172.21.15.40:0/3378557483 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60d400640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd60806e620 0x7fd6080b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60de00640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6080b2ff0 0x7fd6081419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60ca00640 1 -- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6080b2ff0 msgr2=0x7fd6081419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60ca00640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6080b2ff0 0x7fd6081419f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60ca00640 1 -- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd60806e620 msgr2=0x7fd6080b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:02.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60ca00640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd60806e620 0x7fd6080b1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:02.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60ca00640 1 -- 172.21.15.40:0/3378557483 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60815d4a0 con 0x7fd60806ea00 2024-03-24T00:39:02.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60ca00640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60806ea00 0x7fd608159fe0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400d800 tx=0x7fd5f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd605600640 1 -- 172.21.15.40:0/3378557483 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f400b6c0 con 0x7fd60806ea00 2024-03-24T00:39:02.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd605600640 1 -- 172.21.15.40:0/3378557483 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5f4019070 con 0x7fd60806ea00 2024-03-24T00:39:02.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60fe00640 1 -- 172.21.15.40:0/3378557483 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6081429e0 con 0x7fd60806ea00 2024-03-24T00:39:02.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd605600640 1 -- 172.21.15.40:0/3378557483 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5f400ba60 con 0x7fd60806ea00 2024-03-24T00:39:02.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.892+0000 7fd60fe00640 1 -- 172.21.15.40:0/3378557483 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60815d760 con 0x7fd60806ea00 2024-03-24T00:39:02.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.896+0000 7fd60fe00640 1 -- 172.21.15.40:0/3378557483 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd608068170 con 0x7fd60806ea00 2024-03-24T00:39:02.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.896+0000 7fd605600640 1 -- 172.21.15.40:0/3378557483 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd5f400bc00 con 0x7fd60806ea00 2024-03-24T00:39:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.896+0000 7fd605600640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd5e8067660 0x7fd5e8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.896+0000 7fd605600640 1 -- 172.21.15.40:0/3378557483 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fd5f4086430 con 0x7fd60806ea00 2024-03-24T00:39:02.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.896+0000 7fd60d400640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd5e8067660 0x7fd5e8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:02.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.900+0000 7fd60d400640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd5e8067660 0x7fd5e8069b20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0af000 tx=0x7fd5fc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:02.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:02.900+0000 7fd605600640 1 -- 172.21.15.40:0/3378557483 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd5f4057470 con 0x7fd60806ea00 2024-03-24T00:39:03.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.096+0000 7fd60fe00640 1 -- 172.21.15.40:0/3378557483 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd608072f50 con 0x7fd60806ea00 2024-03-24T00:39:03.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd605600640 1 -- 172.21.15.40:0/3378557483 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+3698 (secure 0 0 0) 0x7fd5f4056d70 con 0x7fd60806ea00 2024-03-24T00:39:03.104 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:03.104 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":121,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7079,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6835/755981654","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":755981654},{"type":"v1","addr":"172.21.15.40:6835","nonce":755981654}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":107},{"gid":8461,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3392284788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3392284788},{"type":"v1","addr":"172.21.15.53:6835","nonce":3392284788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":8467,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/2737127931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6836","nonce":2737127931},{"type":"v1","addr":"172.21.15.53:6837","nonce":2737127931}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121},{"gid":8757,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6834/649559011","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6833","nonce":649559011},{"type":"v1","addr":"172.21.15.40:6834","nonce":649559011}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":121}],"filesystems":[]} 2024-03-24T00:39:03.104 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 121 2024-03-24T00:39:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd60fe00640 1 -- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd5e8067660 msgr2=0x7fd5e8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd60fe00640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd5e8067660 0x7fd5e8069b20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc0af000 tx=0x7fd5fc0a7000 comp rx=0 tx=0).stop 2024-03-24T00:39:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd60fe00640 1 -- 172.21.15.40:0/3378557483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60806ea00 msgr2=0x7fd608159fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd60fe00640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60806ea00 0x7fd608159fe0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400d800 tx=0x7fd5f400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd60fe00640 1 -- 172.21.15.40:0/3378557483 shutdown_connections 2024-03-24T00:39:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd60fe00640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd5e8067660 0x7fd5e8069b20 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd60fe00640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6080b2ff0 0x7fd6081419f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd60fe00640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd60806ea00 0x7fd608159fe0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd60fe00640 1 --2- 172.21.15.40:0/3378557483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd60806e620 0x7fd6080b1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd60fe00640 1 -- 172.21.15.40:0/3378557483 >> 172.21.15.40:0/3378557483 conn(0x7fd6080b17b0 msgr2=0x7fd6080ae4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd60fe00640 1 -- 172.21.15.40:0/3378557483 shutdown_connections 2024-03-24T00:39:03.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.100+0000 7fd60fe00640 1 -- 172.21.15.40:0/3378557483 wait complete. 2024-03-24T00:39:03.125 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-24T00:39:03.125 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-24T00:39:03.126 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:39:03.126 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-24T00:39:03.126 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-24T00:39:03.126 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:39:03.126 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:39:03.126 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-24T00:39:03.127 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-24T00:39:03.127 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:39:03.127 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:39:03.127 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-24T00:39:03.127 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-24T00:39:03.127 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:39:03.127 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:39:03.137 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:39:03.137 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-24T00:39:03.138 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:39:03.139 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-24T00:39:03.142 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:39:03.142 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-24T00:39:03.143 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:39:03.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-24T00:39:03.185 INFO:tasks.ceph.mds.c:Started 2024-03-24T00:39:03.186 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:03.187 INFO:tasks.ceph.mds.a:Started 2024-03-24T00:39:03.187 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:03.189 INFO:tasks.ceph.mds.b:Started 2024-03-24T00:39:03.189 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:03.190 INFO:tasks.ceph.mds.d:Started 2024-03-24T00:39:03.191 DEBUG:teuthology.parallel:result is None 2024-03-24T00:39:03.191 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-24T00:39:03.273 INFO:tasks.ceph.mds.c.smithi040.stdout:starting mds.c at 2024-03-24T00:39:03.275 INFO:tasks.ceph.mds.a.smithi040.stdout:starting mds.a at 2024-03-24T00:39:03.281 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-03-24T00:39:03.282 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-03-24T00:39:03.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f0552400640 1 Processor -- start 2024-03-24T00:39:03.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f0552400640 1 -- start start 2024-03-24T00:39:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f0552400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f054c141bc0 0x7f054c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f0552400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c070e00 con 0x7f054c0ce630 2024-03-24T00:39:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f0552400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c0cee20 con 0x7f054c13ec80 2024-03-24T00:39:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f0552400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054c0cf100 con 0x7f054c141bc0 2024-03-24T00:39:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f0542a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f054c141bc0 0x7f054c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f0542a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f054c141bc0 0x7f054c0ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34506/0 (socket says 172.21.15.40:34506) 2024-03-24T00:39:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f0542a00640 1 -- 172.21.15.40:0/3788156170 learned_addr learned my addr 172.21.15.40:0/3788156170 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:03.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f054b400640 1 --1- 172.21.15.40:0/3788156170 >> v1:172.21.15.53:6789/0 conn(0x7f054c13ec80 0x7f054c1414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:48574/0 (socket says 172.21.15.40:48574) 2024-03-24T00:39:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f0542a00640 1 -- 172.21.15.40:0/3788156170 >> v1:172.21.15.53:6789/0 conn(0x7f054c13ec80 legacy=0x7f054c1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f0542a00640 1 -- 172.21.15.40:0/3788156170 >> v1:172.21.15.40:6789/0 conn(0x7f054c0ce630 legacy=0x7f054c071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f0542a00640 1 -- 172.21.15.40:0/3788156170 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c0cf3e0 con 0x7f054c141bc0 2024-03-24T00:39:03.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.336+0000 7f054aa00640 1 -- 172.21.15.40:0/3788156170 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530174199 0 0) 0x7f054c0cee20 con 0x7f054c13ec80 2024-03-24T00:39:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.340+0000 7f0542a00640 1 --2- 172.21.15.40:0/3788156170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f054c141bc0 0x7f054c0ce0f0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f0534009870 tx=0x7f053401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abb15efc64e367cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.340+0000 7f054aa00640 1 -- 172.21.15.40:0/3788156170 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05340a9640 con 0x7f054c141bc0 2024-03-24T00:39:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.340+0000 7f054aa00640 1 -- 172.21.15.40:0/3788156170 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05340a97e0 con 0x7f054c141bc0 2024-03-24T00:39:03.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.340+0000 7f054aa00640 1 -- 172.21.15.40:0/3788156170 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05340a9bb0 con 0x7f054c141bc0 2024-03-24T00:39:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.340+0000 7f0552400640 1 -- 172.21.15.40:0/3788156170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f054c141bc0 msgr2=0x7f054c0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.340+0000 7f0552400640 1 --2- 172.21.15.40:0/3788156170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f054c141bc0 0x7f054c0ce0f0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f0534009870 tx=0x7f053401c560 comp rx=0 tx=0).stop 2024-03-24T00:39:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.340+0000 7f0552400640 1 -- 172.21.15.40:0/3788156170 shutdown_connections 2024-03-24T00:39:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.340+0000 7f0552400640 1 --2- 172.21.15.40:0/3788156170 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f054c141bc0 0x7f054c0ce0f0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.340+0000 7f0552400640 1 -- 172.21.15.40:0/3788156170 >> 172.21.15.40:0/3788156170 conn(0x7f054c0b17b0 msgr2=0x7f054c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.340+0000 7f0552400640 1 -- 172.21.15.40:0/3788156170 shutdown_connections 2024-03-24T00:39:03.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.340+0000 7f0552400640 1 -- 172.21.15.40:0/3788156170 wait complete. 2024-03-24T00:39:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.340+0000 7f0552400640 1 Processor -- start 2024-03-24T00:39:03.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0552400640 1 -- start start 2024-03-24T00:39:03.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0552400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f054c0ce630 0x7f054c139de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0552400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f054c13ec80 0x7f054c154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0552400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f054c141bc0 0x7f054c13f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0552400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0725f0 con 0x7f054c141bc0 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0552400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c072470 con 0x7f054c0ce630 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0552400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c169d50 con 0x7f054c13ec80 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f054be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f054c141bc0 0x7f054c13f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f054b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f054c0ce630 0x7f054c139de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0542a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f054c13ec80 0x7f054c154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f054be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f054c141bc0 0x7f054c13f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34508/0 (socket says 172.21.15.40:34508) 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f054be00640 1 -- 172.21.15.40:0/2295049726 learned_addr learned my addr 172.21.15.40:0/2295049726 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0542a00640 1 -- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f054c0ce630 msgr2=0x7f054c139de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0542a00640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f054c0ce630 0x7f054c139de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0542a00640 1 -- 172.21.15.40:0/2295049726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f054c141bc0 msgr2=0x7f054c13f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0542a00640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f054c141bc0 0x7f054c13f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0542a00640 1 -- 172.21.15.40:0/2295049726 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c0aed50 con 0x7f054c13ec80 2024-03-24T00:39:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f054be00640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f054c141bc0 0x7f054c13f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0542a00640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f054c13ec80 0x7f054c154100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f05340aa000 tx=0x7f0534019300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0543e00640 1 -- 172.21.15.40:0/2295049726 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05340b16c0 con 0x7f054c13ec80 2024-03-24T00:39:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0543e00640 1 -- 172.21.15.40:0/2295049726 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05340b1880 con 0x7f054c13ec80 2024-03-24T00:39:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0543e00640 1 -- 172.21.15.40:0/2295049726 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05340b1c70 con 0x7f054c13ec80 2024-03-24T00:39:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0552400640 1 -- 172.21.15.40:0/2295049726 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c0af030 con 0x7f054c13ec80 2024-03-24T00:39:03.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0552400640 1 -- 172.21.15.40:0/2295049726 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c13a320 con 0x7f054c13ec80 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.344+0000 7f0552400640 1 -- 172.21.15.40:0/2295049726 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054c068170 con 0x7f054c13ec80 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.348+0000 7f0543e00640 1 -- 172.21.15.40:0/2295049726 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f05340a83a0 con 0x7f054c13ec80 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.348+0000 7f0543e00640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0528067660 0x7f0528069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.348+0000 7f0543e00640 1 -- 172.21.15.40:0/2295049726 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f053411f700 con 0x7f054c13ec80 2024-03-24T00:39:03.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.348+0000 7f054b400640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0528067660 0x7f0528069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:03.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.348+0000 7f054b400640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0528067660 0x7f0528069b20 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f053c007fd0 tx=0x7f053c007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:03.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.352+0000 7f0543e00640 1 -- 172.21.15.40:0/2295049726 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f05340c0060 con 0x7f054c13ec80 2024-03-24T00:39:03.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:03.528+0000 7f0552400640 1 -- 172.21.15.40:0/2295049726 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f054c1418c0 con 0x7f054c13ec80 2024-03-24T00:39:04.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.116+0000 7f0543e00640 1 -- 172.21.15.40:0/2295049726 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v113) v1 ==== 97+0+0 (secure 0 0 0) 0x7f05340ad020 con 0x7f054c13ec80 2024-03-24T00:39:04.121 INFO:teuthology.orchestra.run.smithi040.stderr: removed all blocklist entries 2024-03-24T00:39:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.120+0000 7f0552400640 1 -- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0528067660 msgr2=0x7f0528069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.120+0000 7f0552400640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0528067660 0x7f0528069b20 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f053c007fd0 tx=0x7f053c007d60 comp rx=0 tx=0).stop 2024-03-24T00:39:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.120+0000 7f0552400640 1 -- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f054c13ec80 msgr2=0x7f054c154100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.120+0000 7f0552400640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f054c13ec80 0x7f054c154100 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f05340aa000 tx=0x7f0534019300 comp rx=0 tx=0).stop 2024-03-24T00:39:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.120+0000 7f0552400640 1 -- 172.21.15.40:0/2295049726 shutdown_connections 2024-03-24T00:39:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.120+0000 7f0552400640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0528067660 0x7f0528069b20 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.120+0000 7f0552400640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f054c141bc0 0x7f054c13f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.120+0000 7f0552400640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f054c13ec80 0x7f054c154100 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.120+0000 7f0552400640 1 --2- 172.21.15.40:0/2295049726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f054c0ce630 0x7f054c139de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.120+0000 7f0552400640 1 -- 172.21.15.40:0/2295049726 >> 172.21.15.40:0/2295049726 conn(0x7f054c0b17b0 msgr2=0x7f054c06d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.120+0000 7f0552400640 1 -- 172.21.15.40:0/2295049726 shutdown_connections 2024-03-24T00:39:04.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.120+0000 7f0552400640 1 -- 172.21.15.40:0/2295049726 wait complete. 2024-03-24T00:39:04.147 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-24T00:39:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.288+0000 7fda35a00640 1 Processor -- start 2024-03-24T00:39:04.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.288+0000 7fda35a00640 1 -- start start 2024-03-24T00:39:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.288+0000 7fda35a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fda300ccfa0 0x7fda300cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.288+0000 7fda35a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda30070f60 con 0x7fda300b46c0 2024-03-24T00:39:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.288+0000 7fda35a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda300b3620 con 0x7fda3006e620 2024-03-24T00:39:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.288+0000 7fda35a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda300b3900 con 0x7fda300ccfa0 2024-03-24T00:39:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.288+0000 7fda27e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fda300b46c0 0x7fda300b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39798/0 (socket says 172.21.15.40:39798) 2024-03-24T00:39:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.288+0000 7fda27e00640 1 -- 172.21.15.40:0/2883992714 learned_addr learned my addr 172.21.15.40:0/2883992714 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.288+0000 7fda2f400640 1 --2- 172.21.15.40:0/2883992714 >> v2:172.21.15.53:3300/0 conn(0x7fda300ccfa0 0x7fda300cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.288+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318091641 0 0) 0x7fda30070f60 con 0x7fda300b46c0 2024-03-24T00:39:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda14003530 con 0x7fda300b46c0 2024-03-24T00:39:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4139186719 0 0) 0x7fda14003530 con 0x7fda300b46c0 2024-03-24T00:39:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 >> v2:172.21.15.53:3300/0 conn(0x7fda300ccfa0 msgr2=0x7fda300cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 --2- 172.21.15.40:0/2883992714 >> v2:172.21.15.53:3300/0 conn(0x7fda300ccfa0 0x7fda300cd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 >> v1:172.21.15.53:6789/0 conn(0x7fda3006e620 legacy=0x7fda3006ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda300b3be0 con 0x7fda300b46c0 2024-03-24T00:39:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7fda180016a0 con 0x7fda300b46c0 2024-03-24T00:39:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 >> v1:172.21.15.40:6789/0 conn(0x7fda300b46c0 legacy=0x7fda300b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 --2- 172.21.15.40:0/2883992714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda140039c0 0x7fda14003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 --2- 172.21.15.40:0/2883992714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda14004810 0x7fda14004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 --2- 172.21.15.40:0/2883992714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda140051d0 0x7fda1400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda300b3900 con 0x7fda14004810 2024-03-24T00:39:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1400c170 con 0x7fda140051d0 2024-03-24T00:39:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda1400c370 con 0x7fda140039c0 2024-03-24T00:39:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda27e00640 1 --2- 172.21.15.40:0/2883992714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda14004810 0x7fda14004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2ea00640 1 --2- 172.21.15.40:0/2883992714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda140039c0 0x7fda14003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2f400640 1 --2- 172.21.15.40:0/2883992714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda140051d0 0x7fda1400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda27e00640 1 -- 172.21.15.40:0/2883992714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda140039c0 msgr2=0x7fda14003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda27e00640 1 --2- 172.21.15.40:0/2883992714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda140039c0 0x7fda14003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda27e00640 1 -- 172.21.15.40:0/2883992714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda140051d0 msgr2=0x7fda1400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda27e00640 1 --2- 172.21.15.40:0/2883992714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda140051d0 0x7fda1400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda2f400640 1 -- 172.21.15.40:0/2883992714 reap_dead start 2024-03-24T00:39:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda27e00640 1 -- 172.21.15.40:0/2883992714 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda14003530 con 0x7fda14004810 2024-03-24T00:39:04.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.292+0000 7fda27e00640 1 --2- 172.21.15.40:0/2883992714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda14004810 0x7fda14004c90 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fda1800c210 tx=0x7fda1801cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d4aefa4d04d2195 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18007d20 con 0x7fda14004810 2024-03-24T00:39:04.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda180a9020 con 0x7fda14004810 2024-03-24T00:39:04.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda2e000640 1 -- 172.21.15.40:0/2883992714 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18005ae0 con 0x7fda14004810 2024-03-24T00:39:04.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 -- 172.21.15.40:0/2883992714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda14004810 msgr2=0x7fda14004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 --2- 172.21.15.40:0/2883992714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda14004810 0x7fda14004c90 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fda1800c210 tx=0x7fda1801cb90 comp rx=0 tx=0).stop 2024-03-24T00:39:04.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 -- 172.21.15.40:0/2883992714 shutdown_connections 2024-03-24T00:39:04.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 --2- 172.21.15.40:0/2883992714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda14004810 0x7fda14004c90 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 -- 172.21.15.40:0/2883992714 >> 172.21.15.40:0/2883992714 conn(0x7fda300b17b0 msgr2=0x7fda300b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:04.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 -- 172.21.15.40:0/2883992714 shutdown_connections 2024-03-24T00:39:04.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 -- 172.21.15.40:0/2883992714 wait complete. 2024-03-24T00:39:04.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 Processor -- start 2024-03-24T00:39:04.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 -- start start 2024-03-24T00:39:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda14004810 0x7fda3015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda300a94d0 0x7fda300aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda301436b0 0x7fda30143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda30144920 con 0x7fda301436b0 2024-03-24T00:39:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3006ee10 con 0x7fda300a94d0 2024-03-24T00:39:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda35a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda300ad1b0 con 0x7fda14004810 2024-03-24T00:39:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda2f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda301436b0 0x7fda30143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda2ea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda14004810 0x7fda3015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda27e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda300a94d0 0x7fda300aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda2f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda301436b0 0x7fda30143b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34518/0 (socket says 172.21.15.40:34518) 2024-03-24T00:39:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.296+0000 7fda2f400640 1 -- 172.21.15.40:0/3394873286 learned_addr learned my addr 172.21.15.40:0/3394873286 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda2f400640 1 -- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda14004810 msgr2=0x7fda3015a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda2f400640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda14004810 0x7fda3015a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda2f400640 1 -- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda300a94d0 msgr2=0x7fda300aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda2f400640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda300a94d0 0x7fda300aedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda2f400640 1 -- 172.21.15.40:0/3394873286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda300a99d0 con 0x7fda301436b0 2024-03-24T00:39:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda2ea00640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda14004810 0x7fda3015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda2f400640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda301436b0 0x7fda30143b40 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fda300cf780 tx=0x7fda2800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda26a00640 1 -- 172.21.15.40:0/3394873286 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2800fd80 con 0x7fda301436b0 2024-03-24T00:39:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda26a00640 1 -- 172.21.15.40:0/3394873286 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2801d070 con 0x7fda301436b0 2024-03-24T00:39:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda35a00640 1 -- 172.21.15.40:0/3394873286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda300a9cb0 con 0x7fda301436b0 2024-03-24T00:39:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda26a00640 1 -- 172.21.15.40:0/3394873286 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28015670 con 0x7fda301436b0 2024-03-24T00:39:04.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda35a00640 1 -- 172.21.15.40:0/3394873286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda300aa270 con 0x7fda301436b0 2024-03-24T00:39:04.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda35a00640 1 -- 172.21.15.40:0/3394873286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda30068170 con 0x7fda301436b0 2024-03-24T00:39:04.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.300+0000 7fda26a00640 1 -- 172.21.15.40:0/3394873286 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fda28015810 con 0x7fda301436b0 2024-03-24T00:39:04.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.304+0000 7fda26a00640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fda0c067660 0x7fda0c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.304+0000 7fda26a00640 1 -- 172.21.15.40:0/3394873286 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fda28086130 con 0x7fda301436b0 2024-03-24T00:39:04.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.304+0000 7fda2ea00640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fda0c067660 0x7fda0c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.304+0000 7fda2ea00640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fda0c067660 0x7fda0c069b20 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fda20000c10 tx=0x7fda20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.308+0000 7fda26a00640 1 -- 172.21.15.40:0/3394873286 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fda28027020 con 0x7fda301436b0 2024-03-24T00:39:04.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 -- 172.21.15.40:0/3394873286 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fda30072f50 con 0x7fda301436b0 2024-03-24T00:39:04.532 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda26a00640 1 -- 172.21.15.40:0/3394873286 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fda28010040 con 0x7fda301436b0 2024-03-24T00:39:04.532 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:04.532 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:39:04.532 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_dump": [ 2024-03-24T00:39:04.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.532 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.a", 2024-03-24T00:39:04.532 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lCkU6EhAA0q21F90epsGPK5x17s9NLw==", 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.b", 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llkfOHhAAqKKFyEJzG0UxbLJMFPET8A==", 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:39:04.533 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.c", 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lKMmwGRAAAKaOSnhTRbHIpPKXXkAe+Q==", 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mds.d", 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l7quKJhAA8QsquiFFmj5H1G5yck0FpA==", 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.0", 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAVdP9lVk3uBxAA9TC1OpeB5GIFlfJl5+ILBg==", 2024-03-24T00:39:04.535 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.1", 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAYdP9lUc1IDhAAPwCV9eqL7bFibkfMFt9wPw==", 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.2", 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAbdP9lwjyuFBAAJ1JAIAnLXAn03+lQPwwY9Q==", 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.3", 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAedP9lZ1bgGRAAuLAay48N7jXjj8/yaYwSqg==", 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:39:04.537 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.4", 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAidP9lvyk4JhAA8w4/XBYHDLoqJ1zop9sUiw==", 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.5", 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAldP9lcb9qLBAAXts6VU/4kVqUWS/WORV7zQ==", 2024-03-24T00:39:04.538 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.6", 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAodP9l8ipTMhAAjpagojh+woHyGvlQlIkHDw==", 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "osd.7", 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQArdP9lmGfpORAAPvow5nF/O7RLmuXEqNWENg==", 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow profile osd", 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile osd", 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.0", 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw==", 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:39:04.540 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow r", 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.1", 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lpIZrNBAAzpGWZuQ6ndjNMAOmAsebvQ==", 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.541 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.2", 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9l6gXMMBAAd6Oe6S73hj/WfLXTsYiEZw==", 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.3", 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9llFHCOhAATtv8gcL6BHPnxB1dHrg6vw==", 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow", 2024-03-24T00:39:04.542 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow r", 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow rw", 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow rwx" 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "client.admin", 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQAQdP9lmmd7KxAA4uk6aUKP6egYj6O1NgdadQ==", 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr": "allow *", 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow *", 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.x", 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lebNrBxAAvbcCAJhDQn/5Uk1dJ01oHw==", 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: "entity": "mgr.y", 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "AQARdP9lEnZyAxAAW5Lrm7LZPTVfLCOY6As1PQ==", 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: "caps": { 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: "mds": "allow *", 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: "mon": "allow profile mgr", 2024-03-24T00:39:04.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd": "allow *" 2024-03-24T00:39:04.545 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.545 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:04.545 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-03-24T00:39:04.545 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:39:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 -- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fda0c067660 msgr2=0x7fda0c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fda0c067660 0x7fda0c069b20 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fda20000c10 tx=0x7fda20009040 comp rx=0 tx=0).stop 2024-03-24T00:39:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 -- 172.21.15.40:0/3394873286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda301436b0 msgr2=0x7fda30143b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda301436b0 0x7fda30143b40 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fda300cf780 tx=0x7fda2800f580 comp rx=0 tx=0).stop 2024-03-24T00:39:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 -- 172.21.15.40:0/3394873286 shutdown_connections 2024-03-24T00:39:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fda0c067660 0x7fda0c069b20 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fda301436b0 0x7fda30143b40 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fda300a94d0 0x7fda300aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 --2- 172.21.15.40:0/3394873286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fda14004810 0x7fda3015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 -- 172.21.15.40:0/3394873286 >> 172.21.15.40:0/3394873286 conn(0x7fda300b17b0 msgr2=0x7fda3015f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 -- 172.21.15.40:0/3394873286 shutdown_connections 2024-03-24T00:39:04.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.528+0000 7fda35a00640 1 -- 172.21.15.40:0/3394873286 wait complete. 2024-03-24T00:39:04.553 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:04.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.688+0000 7f09edc00640 1 Processor -- start 2024-03-24T00:39:04.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.688+0000 7f09edc00640 1 -- start start 2024-03-24T00:39:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09edc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e80b46c0 0x7f09e80b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09edc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f09e80b2ff0 0x7f09e80ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09edc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e8070f60 con 0x7f09e806e620 2024-03-24T00:39:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09edc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e80b3620 con 0x7f09e80b2ff0 2024-03-24T00:39:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09edc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e80b3900 con 0x7f09e80b46c0 2024-03-24T00:39:04.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f09e806e620 0x7f09e806ea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39802/0 (socket says 172.21.15.40:39802) 2024-03-24T00:39:04.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6a00640 1 -- 172.21.15.40:0/3806390869 learned_addr learned my addr 172.21.15.40:0/3806390869 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09dfe00640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e80b46c0 0x7f09e80b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e7400640 1 --2- 172.21.15.40:0/3806390869 >> v2:172.21.15.53:3300/0 conn(0x7f09e80b2ff0 0x7f09e80ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156016453 0 0) 0x7f09e8070f60 con 0x7f09e806e620 2024-03-24T00:39:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09cc003530 con 0x7f09e806e620 2024-03-24T00:39:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2898918477 0 0) 0x7f09cc003530 con 0x7f09e806e620 2024-03-24T00:39:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e80b46c0 msgr2=0x7f09e80b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e80b46c0 0x7f09e80b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 >> v2:172.21.15.53:3300/0 conn(0x7f09e80b2ff0 msgr2=0x7f09e80ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 --2- 172.21.15.40:0/3806390869 >> v2:172.21.15.53:3300/0 conn(0x7f09e80b2ff0 0x7f09e80ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e80b3be0 con 0x7f09e806e620 2024-03-24T00:39:04.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f09d8001680 con 0x7f09e806e620 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 >> v1:172.21.15.40:6789/0 conn(0x7f09e806e620 legacy=0x7f09e806ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc0039c0 0x7f09cc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09cc004810 0x7f09cc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e80b46c0 0x7f09cc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e80b3620 con 0x7f09cc0039c0 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e80b3900 con 0x7f09cc004810 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc00ba50 con 0x7f09e80b46c0 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6a00640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc0039c0 0x7f09cc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09dfe00640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09cc004810 0x7f09cc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e7400640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e80b46c0 0x7f09cc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6a00640 1 -- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e80b46c0 msgr2=0x7f09cc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6a00640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e80b46c0 0x7f09cc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.699 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6a00640 1 -- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09cc004810 msgr2=0x7f09cc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:04.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6a00640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09cc004810 0x7f09cc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e6a00640 1 -- 172.21.15.40:0/3806390869 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc003530 con 0x7f09cc0039c0 2024-03-24T00:39:04.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09e7400640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e80b46c0 0x7f09cc0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:04.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.692+0000 7f09dfe00640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09cc004810 0x7f09cc004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:04.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09e6a00640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc0039c0 0x7f09cc003de0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f09d800c190 tx=0x7f09d801cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dca41aedc1552dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8007c90 con 0x7f09cc0039c0 2024-03-24T00:39:04.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d80a9020 con 0x7f09cc0039c0 2024-03-24T00:39:04.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09e6000640 1 -- 172.21.15.40:0/3806390869 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e80cdcf0 con 0x7f09cc0039c0 2024-03-24T00:39:04.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09edc00640 1 -- 172.21.15.40:0/3806390869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc0039c0 msgr2=0x7f09cc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09edc00640 1 --2- 172.21.15.40:0/3806390869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc0039c0 0x7f09cc003de0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f09d800c190 tx=0x7f09d801cc00 comp rx=0 tx=0).stop 2024-03-24T00:39:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09e7400640 1 -- 172.21.15.40:0/3806390869 reap_dead start 2024-03-24T00:39:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09edc00640 1 -- 172.21.15.40:0/3806390869 shutdown_connections 2024-03-24T00:39:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09edc00640 1 -- 172.21.15.40:0/3806390869 >> 172.21.15.40:0/3806390869 conn(0x7f09e80b17b0 msgr2=0x7f09e80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09edc00640 1 -- 172.21.15.40:0/3806390869 shutdown_connections 2024-03-24T00:39:04.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09edc00640 1 -- 172.21.15.40:0/3806390869 wait complete. 2024-03-24T00:39:04.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09edc00640 1 Processor -- start 2024-03-24T00:39:04.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.696+0000 7f09edc00640 1 -- start start 2024-03-24T00:39:04.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09edc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc0039c0 0x7f09e81581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09edc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e81f97f0 0x7f09e8139460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09edc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e815a2d0 0x7f09e815a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09edc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e8144920 con 0x7f09cc0039c0 2024-03-24T00:39:04.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09edc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e8165c70 con 0x7f09e81f97f0 2024-03-24T00:39:04.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09edc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e81f6160 con 0x7f09e815a2d0 2024-03-24T00:39:04.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09e6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc0039c0 0x7f09e81581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09e6a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc0039c0 0x7f09e81581f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34534/0 (socket says 172.21.15.40:34534) 2024-03-24T00:39:04.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09e6a00640 1 -- 172.21.15.40:0/4184492638 learned_addr learned my addr 172.21.15.40:0/4184492638 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:04.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09e6a00640 1 -- 172.21.15.40:0/4184492638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e815a2d0 msgr2=0x7f09e815a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09e6a00640 1 --2- 172.21.15.40:0/4184492638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e815a2d0 0x7f09e815a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09e6a00640 1 -- 172.21.15.40:0/4184492638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e81f97f0 msgr2=0x7f09e8139460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09e6a00640 1 --2- 172.21.15.40:0/4184492638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e81f97f0 0x7f09e8139460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09e6a00640 1 -- 172.21.15.40:0/4184492638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e81f5830 con 0x7f09cc0039c0 2024-03-24T00:39:04.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09e6a00640 1 --2- 172.21.15.40:0/4184492638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc0039c0 0x7f09e81581f0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f09d801ca30 tx=0x7f09cc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09dea00640 1 -- 172.21.15.40:0/4184492638 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8004ab0 con 0x7f09cc0039c0 2024-03-24T00:39:04.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09dea00640 1 -- 172.21.15.40:0/4184492638 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d8004c50 con 0x7f09cc0039c0 2024-03-24T00:39:04.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09edc00640 1 -- 172.21.15.40:0/4184492638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e81f5b10 con 0x7f09cc0039c0 2024-03-24T00:39:04.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09dea00640 1 -- 172.21.15.40:0/4184492638 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d80070d0 con 0x7f09cc0039c0 2024-03-24T00:39:04.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09edc00640 1 -- 172.21.15.40:0/4184492638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e8158730 con 0x7f09cc0039c0 2024-03-24T00:39:04.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09edc00640 1 -- 172.21.15.40:0/4184492638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e815ac90 con 0x7f09cc0039c0 2024-03-24T00:39:04.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.700+0000 7f09dea00640 1 -- 172.21.15.40:0/4184492638 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f09d8007270 con 0x7f09cc0039c0 2024-03-24T00:39:04.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.704+0000 7f09dea00640 1 --2- 172.21.15.40:0/4184492638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f09c4067660 0x7f09c4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:04.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.704+0000 7f09dea00640 1 -- 172.21.15.40:0/4184492638 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f09d811ec50 con 0x7f09cc0039c0 2024-03-24T00:39:04.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.704+0000 7f09dfe00640 1 --2- 172.21.15.40:0/4184492638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f09c4067660 0x7f09c4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:04.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.704+0000 7f09dfe00640 1 --2- 172.21.15.40:0/4184492638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f09c4067660 0x7f09c4069b20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f09e8138f60 tx=0x7f09d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:04.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.708+0000 7f09dea00640 1 -- 172.21.15.40:0/4184492638 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f09d80be060 con 0x7f09cc0039c0 2024-03-24T00:39:04.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 -- 172.21.15.40:0/4184492638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09e8068170 con 0x7f09cc0039c0 2024-03-24T00:39:04.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09dea00640 1 -- 172.21.15.40:0/4184492638 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+12841 (secure 0 0 0) 0x7f09d8016910 con 0x7f09cc0039c0 2024-03-24T00:39:04.876 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:04.876 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":115,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:39:04.127100+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":104,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 -- 172.21.15.40:0/4184492638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f09c4067660 msgr2=0x7f09c4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 --2- 172.21.15.40:0/4184492638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f09c4067660 0x7f09c4069b20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f09e8138f60 tx=0x7f09d0008040 comp rx=0 tx=0).stop 2024-03-24T00:39:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 -- 172.21.15.40:0/4184492638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc0039c0 msgr2=0x7f09e81581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 --2- 172.21.15.40:0/4184492638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc0039c0 0x7f09e81581f0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f09d801ca30 tx=0x7f09cc005840 comp rx=0 tx=0).stop 2024-03-24T00:39:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 -- 172.21.15.40:0/4184492638 shutdown_connections 2024-03-24T00:39:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 --2- 172.21.15.40:0/4184492638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f09c4067660 0x7f09c4069b20 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 --2- 172.21.15.40:0/4184492638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e815a2d0 0x7f09e815a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 --2- 172.21.15.40:0/4184492638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e81f97f0 0x7f09e8139460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 --2- 172.21.15.40:0/4184492638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09cc0039c0 0x7f09e81581f0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 -- 172.21.15.40:0/4184492638 >> 172.21.15.40:0/4184492638 conn(0x7f09e80b17b0 msgr2=0x7f09e80cee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 -- 172.21.15.40:0/4184492638 shutdown_connections 2024-03-24T00:39:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:04.872+0000 7f09edc00640 1 -- 172.21.15.40:0/4184492638 wait complete. 2024-03-24T00:39:04.897 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:39:04.898 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:39:05.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.036+0000 7f0f0d200640 1 Processor -- start 2024-03-24T00:39:05.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.036+0000 7f0f0d200640 1 -- start start 2024-03-24T00:39:05.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.040+0000 7f0f0d200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f0f080af3c0 0x7f0f081658d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.040+0000 7f0f0d200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f080ad630 con 0x7f0f08074210 2024-03-24T00:39:05.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.040+0000 7f0f0d200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0806e540 con 0x7f0f0815ac10 2024-03-24T00:39:05.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.040+0000 7f0f0d200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f080ad8d0 con 0x7f0f080af3c0 2024-03-24T00:39:05.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.040+0000 7f0f05400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f0f08074210 0x7f0f080b3fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39818/0 (socket says 172.21.15.40:39818) 2024-03-24T00:39:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.040+0000 7f0f05400640 1 -- 172.21.15.40:0/3776242806 learned_addr learned my addr 172.21.15.40:0/3776242806 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.040+0000 7f0f06800640 1 --2- 172.21.15.40:0/3776242806 >> v2:172.21.15.40:3300/0 conn(0x7f0f080af3c0 0x7f0f081658d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.040+0000 7f0f06800640 1 -- 172.21.15.40:0/3776242806 >> v1:172.21.15.53:6789/0 conn(0x7f0f0815ac10 legacy=0x7f0f080b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.040+0000 7f0f06800640 1 -- 172.21.15.40:0/3776242806 >> v1:172.21.15.40:6789/0 conn(0x7f0f08074210 legacy=0x7f0f080b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.040+0000 7f0f06800640 1 -- 172.21.15.40:0/3776242806 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f080adbb0 con 0x7f0f080af3c0 2024-03-24T00:39:05.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.040+0000 7f0f06800640 1 --2- 172.21.15.40:0/3776242806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af3c0 0x7f0f081658d0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009d90 tx=0x7f0f0001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96623a7fbc8a5a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.040+0000 7f0f04a00640 1 -- 172.21.15.40:0/3776242806 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000b2070 con 0x7f0f080af3c0 2024-03-24T00:39:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f04a00640 1 -- 172.21.15.40:0/3776242806 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f000a5720 con 0x7f0f080af3c0 2024-03-24T00:39:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f04a00640 1 -- 172.21.15.40:0/3776242806 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f000a5af0 con 0x7f0f080af3c0 2024-03-24T00:39:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 -- 172.21.15.40:0/3776242806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af3c0 msgr2=0x7f0f081658d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 --2- 172.21.15.40:0/3776242806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af3c0 0x7f0f081658d0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f0f00009d90 tx=0x7f0f0001c970 comp rx=0 tx=0).stop 2024-03-24T00:39:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 -- 172.21.15.40:0/3776242806 shutdown_connections 2024-03-24T00:39:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 --2- 172.21.15.40:0/3776242806 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f080af3c0 0x7f0f081658d0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 -- 172.21.15.40:0/3776242806 >> 172.21.15.40:0/3776242806 conn(0x7f0f080b1e90 msgr2=0x7f0f080b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 -- 172.21.15.40:0/3776242806 shutdown_connections 2024-03-24T00:39:05.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 -- 172.21.15.40:0/3776242806 wait complete. 2024-03-24T00:39:05.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 Processor -- start 2024-03-24T00:39:05.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 -- start start 2024-03-24T00:39:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08074210 0x7f0f080992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f080af3c0 0x7f0f08153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0f0815ac10 0x7f0f08154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f08072960 con 0x7f0f08074210 2024-03-24T00:39:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080727e0 con 0x7f0f080af3c0 2024-03-24T00:39:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f0d200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f08072ae0 con 0x7f0f0815ac10 2024-03-24T00:39:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f05e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08074210 0x7f0f080992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f05e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08074210 0x7f0f080992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34548/0 (socket says 172.21.15.40:34548) 2024-03-24T00:39:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f05e00640 1 -- 172.21.15.40:0/1819971760 learned_addr learned my addr 172.21.15.40:0/1819971760 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f05400640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f080af3c0 0x7f0f08153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.044+0000 7f0f06800640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0f0815ac10 0x7f0f08154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.052 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0f05e00640 1 -- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0f0815ac10 msgr2=0x7f0f08154e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0f05e00640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0f0815ac10 0x7f0f08154e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0f05e00640 1 -- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f080af3c0 msgr2=0x7f0f08153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0f05e00640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f080af3c0 0x7f0f08153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0f05e00640 1 -- 172.21.15.40:0/1819971760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f080984f0 con 0x7f0f08074210 2024-03-24T00:39:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0f05e00640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08074210 0x7f0f080992b0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800c3f0 tx=0x7f0ef800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0efe000640 1 -- 172.21.15.40:0/1819971760 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8017070 con 0x7f0f08074210 2024-03-24T00:39:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0efe000640 1 -- 172.21.15.40:0/1819971760 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef80052d0 con 0x7f0f08074210 2024-03-24T00:39:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0efe000640 1 -- 172.21.15.40:0/1819971760 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8003840 con 0x7f0f08074210 2024-03-24T00:39:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0f0d200640 1 -- 172.21.15.40:0/1819971760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f080987d0 con 0x7f0f08074210 2024-03-24T00:39:05.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0f0d200640 1 -- 172.21.15.40:0/1819971760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f08098d90 con 0x7f0f08074210 2024-03-24T00:39:05.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0f0d200640 1 -- 172.21.15.40:0/1819971760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f08068880 con 0x7f0f08074210 2024-03-24T00:39:05.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0efe000640 1 -- 172.21.15.40:0/1819971760 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f0ef801a070 con 0x7f0f08074210 2024-03-24T00:39:05.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.048+0000 7f0efe000640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0ee8067690 0x7f0ee8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.052+0000 7f0efe000640 1 -- 172.21.15.40:0/1819971760 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0ef8084770 con 0x7f0f08074210 2024-03-24T00:39:05.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.052+0000 7f0f05400640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0ee8067690 0x7f0ee8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.052+0000 7f0f05400640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0ee8067690 0x7f0ee8069b50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4000f70 tx=0x7f0ef4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.056+0000 7f0efe000640 1 -- 172.21.15.40:0/1819971760 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0ef8005480 con 0x7f0f08074210 2024-03-24T00:39:05.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.220+0000 7f0f0d200640 1 -- 172.21.15.40:0/1819971760 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0f0814a790 con 0x7f0f08074210 2024-03-24T00:39:05.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.220+0000 7f0efe000640 1 -- 172.21.15.40:0/1819971760 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ef8055180 con 0x7f0f08074210 2024-03-24T00:39:05.226 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:39:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.224+0000 7f0f0d200640 1 -- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0ee8067690 msgr2=0x7f0ee8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.224+0000 7f0f0d200640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0ee8067690 0x7f0ee8069b50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f0ef4000f70 tx=0x7f0ef4009040 comp rx=0 tx=0).stop 2024-03-24T00:39:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.224+0000 7f0f0d200640 1 -- 172.21.15.40:0/1819971760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08074210 msgr2=0x7f0f080992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.224+0000 7f0f0d200640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08074210 0x7f0f080992b0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800c3f0 tx=0x7f0ef800c8b0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.224+0000 7f0f0d200640 1 -- 172.21.15.40:0/1819971760 shutdown_connections 2024-03-24T00:39:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.224+0000 7f0f0d200640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0ee8067690 0x7f0ee8069b50 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.224+0000 7f0f0d200640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0f0815ac10 0x7f0f08154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.224+0000 7f0f0d200640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f080af3c0 0x7f0f08153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.224+0000 7f0f0d200640 1 --2- 172.21.15.40:0/1819971760 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f0f08074210 0x7f0f080992b0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.224+0000 7f0f0d200640 1 -- 172.21.15.40:0/1819971760 >> 172.21.15.40:0/1819971760 conn(0x7f0f080b1e90 msgr2=0x7f0f080b3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.224+0000 7f0f0d200640 1 -- 172.21.15.40:0/1819971760 shutdown_connections 2024-03-24T00:39:05.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.224+0000 7f0f0d200640 1 -- 172.21.15.40:0/1819971760 wait complete. 2024-03-24T00:39:05.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:39:05.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.384+0000 7efc44c00640 1 Processor -- start 2024-03-24T00:39:05.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.384+0000 7efc44c00640 1 -- start start 2024-03-24T00:39:05.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.384+0000 7efc44c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efc400b46c0 0x7efc400b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.384+0000 7efc44c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc400b2ff0 0x7efc400ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.384+0000 7efc44c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc40070f60 con 0x7efc4006e620 2024-03-24T00:39:05.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.384+0000 7efc44c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc400b3400 con 0x7efc400b2ff0 2024-03-24T00:39:05.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.384+0000 7efc44c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc400b3620 con 0x7efc400b46c0 2024-03-24T00:39:05.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.384+0000 7efc3e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc400b2ff0 0x7efc400ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.384+0000 7efc3e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7efc400b2ff0 0x7efc400ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34562/0 (socket says 172.21.15.40:34562) 2024-03-24T00:39:05.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.384+0000 7efc3e800640 1 -- 172.21.15.40:0/3201889749 learned_addr learned my addr 172.21.15.40:0/3201889749 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:05.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc37e00640 1 --2- 172.21.15.40:0/3201889749 >> v2:172.21.15.53:3300/0 conn(0x7efc400b46c0 0x7efc400b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc3e800640 1 -- 172.21.15.40:0/3201889749 >> v2:172.21.15.53:3300/0 conn(0x7efc400b46c0 msgr2=0x7efc400b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc3e800640 1 --2- 172.21.15.40:0/3201889749 >> v2:172.21.15.53:3300/0 conn(0x7efc400b46c0 0x7efc400b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc3e800640 1 -- 172.21.15.40:0/3201889749 >> v1:172.21.15.53:6789/0 conn(0x7efc4006e620 legacy=0x7efc4006ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc3e800640 1 -- 172.21.15.40:0/3201889749 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc400b3900 con 0x7efc400b2ff0 2024-03-24T00:39:05.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc37e00640 1 --2- 172.21.15.40:0/3201889749 >> v2:172.21.15.53:3300/0 conn(0x7efc400b46c0 0x7efc400b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:05.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc3e800640 1 --2- 172.21.15.40:0/3201889749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc400b2ff0 0x7efc400ccfa0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7efc400cf680 tx=0x7efc3801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e707caedc77693 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc3d400640 1 -- 172.21.15.40:0/3201889749 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc400cdcf0 con 0x7efc400b2ff0 2024-03-24T00:39:05.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc3d400640 1 -- 172.21.15.40:0/3201889749 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc380a5790 con 0x7efc400b2ff0 2024-03-24T00:39:05.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc3d400640 1 -- 172.21.15.40:0/3201889749 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc380a5b60 con 0x7efc400b2ff0 2024-03-24T00:39:05.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc44c00640 1 -- 172.21.15.40:0/3201889749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc400b2ff0 msgr2=0x7efc400ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc44c00640 1 --2- 172.21.15.40:0/3201889749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc400b2ff0 0x7efc400ccfa0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7efc400cf680 tx=0x7efc3801c8d0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc44c00640 1 -- 172.21.15.40:0/3201889749 shutdown_connections 2024-03-24T00:39:05.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc44c00640 1 --2- 172.21.15.40:0/3201889749 >> v2:172.21.15.53:3300/0 conn(0x7efc400b46c0 0x7efc400b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.388+0000 7efc44c00640 1 --2- 172.21.15.40:0/3201889749 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc400b2ff0 0x7efc400ccfa0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc44c00640 1 -- 172.21.15.40:0/3201889749 >> 172.21.15.40:0/3201889749 conn(0x7efc400b17b0 msgr2=0x7efc400b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc44c00640 1 -- 172.21.15.40:0/3201889749 shutdown_connections 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc44c00640 1 -- 172.21.15.40:0/3201889749 wait complete. 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc44c00640 1 Processor -- start 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc44c00640 1 -- start start 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc44c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc4006e620 0x7efc40159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc44c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc400b2ff0 0x7efc40141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc44c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc400b46c0 0x7efc40141550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc44c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc40144920 con 0x7efc400b2ff0 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc44c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc400b4ad0 con 0x7efc400b46c0 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc44c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc401fb4b0 con 0x7efc4006e620 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc37e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc400b2ff0 0x7efc40141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc37e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc400b2ff0 0x7efc40141010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34574/0 (socket says 172.21.15.40:34574) 2024-03-24T00:39:05.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc37e00640 1 -- 172.21.15.40:0/2885580774 learned_addr learned my addr 172.21.15.40:0/2885580774 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc3e800640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc400b46c0 0x7efc40141550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc3de00640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc4006e620 0x7efc40159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc37e00640 1 -- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc4006e620 msgr2=0x7efc40159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc37e00640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc4006e620 0x7efc40159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc37e00640 1 -- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc400b46c0 msgr2=0x7efc40141550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc37e00640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc400b46c0 0x7efc40141550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc37e00640 1 -- 172.21.15.40:0/2885580774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc401f7180 con 0x7efc400b2ff0 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc37e00640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc400b2ff0 0x7efc40141010 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7efc2800c7f0 tx=0x7efc2800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc36000640 1 -- 172.21.15.40:0/2885580774 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc28015070 con 0x7efc400b2ff0 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc36000640 1 -- 172.21.15.40:0/2885580774 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc28007820 con 0x7efc400b2ff0 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.392+0000 7efc44c00640 1 -- 172.21.15.40:0/2885580774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc401f7460 con 0x7efc400b2ff0 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.396+0000 7efc36000640 1 -- 172.21.15.40:0/2885580774 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc28007c10 con 0x7efc400b2ff0 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.396+0000 7efc44c00640 1 -- 172.21.15.40:0/2885580774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc401f7a20 con 0x7efc400b2ff0 2024-03-24T00:39:05.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.396+0000 7efc44c00640 1 -- 172.21.15.40:0/2885580774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc40068170 con 0x7efc400b2ff0 2024-03-24T00:39:05.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.396+0000 7efc36000640 1 -- 172.21.15.40:0/2885580774 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7efc28018070 con 0x7efc400b2ff0 2024-03-24T00:39:05.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.396+0000 7efc36000640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efc1c067660 0x7efc1c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.396+0000 7efc36000640 1 -- 172.21.15.40:0/2885580774 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efc28085510 con 0x7efc400b2ff0 2024-03-24T00:39:05.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.396+0000 7efc3de00640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efc1c067660 0x7efc1c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.396+0000 7efc3de00640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efc1c067660 0x7efc1c069b20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7efc4015ad60 tx=0x7efc30007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.400+0000 7efc36000640 1 -- 172.21.15.40:0/2885580774 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7efc28055f20 con 0x7efc400b2ff0 2024-03-24T00:39:05.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.564+0000 7efc44c00640 1 -- 172.21.15.40:0/2885580774 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7efc40072f50 con 0x7efc400b2ff0 2024-03-24T00:39:05.570 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.564+0000 7efc36000640 1 -- 172.21.15.40:0/2885580774 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v115) v1 ==== 84+0+65 (secure 0 0 0) 0x7efc28085030 con 0x7efc400b2ff0 2024-03-24T00:39:05.570 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:05.570 INFO:teuthology.orchestra.run.smithi040.stdout:[ 2024-03-24T00:39:05.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-03-24T00:39:05.571 INFO:teuthology.orchestra.run.smithi040.stdout: "poolnum": 1, 2024-03-24T00:39:05.571 INFO:teuthology.orchestra.run.smithi040.stdout: "poolname": ".mgr" 2024-03-24T00:39:05.571 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:05.571 INFO:teuthology.orchestra.run.smithi040.stdout:] 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.568+0000 7efc44c00640 1 -- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efc1c067660 msgr2=0x7efc1c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.568+0000 7efc44c00640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efc1c067660 0x7efc1c069b20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7efc4015ad60 tx=0x7efc30007e40 comp rx=0 tx=0).stop 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.568+0000 7efc44c00640 1 -- 172.21.15.40:0/2885580774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc400b2ff0 msgr2=0x7efc40141010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.568+0000 7efc44c00640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc400b2ff0 0x7efc40141010 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7efc2800c7f0 tx=0x7efc2800ccb0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.568+0000 7efc44c00640 1 -- 172.21.15.40:0/2885580774 shutdown_connections 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.568+0000 7efc44c00640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7efc1c067660 0x7efc1c069b20 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.568+0000 7efc44c00640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc400b46c0 0x7efc40141550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.568+0000 7efc44c00640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7efc400b2ff0 0x7efc40141010 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.568+0000 7efc44c00640 1 --2- 172.21.15.40:0/2885580774 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc4006e620 0x7efc40159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.568+0000 7efc44c00640 1 -- 172.21.15.40:0/2885580774 >> 172.21.15.40:0/2885580774 conn(0x7efc400b17b0 msgr2=0x7efc400ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.568+0000 7efc44c00640 1 -- 172.21.15.40:0/2885580774 shutdown_connections 2024-03-24T00:39:05.573 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.568+0000 7efc44c00640 1 -- 172.21.15.40:0/2885580774 wait complete. 2024-03-24T00:39:05.592 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:39:05.592 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:39:05.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f0812a00640 1 Processor -- start 2024-03-24T00:39:05.737 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f0812a00640 1 -- start start 2024-03-24T00:39:05.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f0812a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c1628d0 0x7f080c0d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f0812a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c15e3e0 con 0x7f080c15e000 2024-03-24T00:39:05.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f0812a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c0b5f10 con 0x7f080c0b42d0 2024-03-24T00:39:05.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f0812a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c161ba0 con 0x7f080c1628d0 2024-03-24T00:39:05.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f080be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f080c15e000 0x7f080c0b3bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39824/0 (socket says 172.21.15.40:39824) 2024-03-24T00:39:05.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f080be00640 1 -- 172.21.15.40:0/3026902673 learned_addr learned my addr 172.21.15.40:0/3026902673 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:05.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003889472 0 0) 0x7f080c15e3e0 con 0x7f080c15e000 2024-03-24T00:39:05.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f0003530 con 0x7f080c15e000 2024-03-24T00:39:05.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f0803400640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c1628d0 0x7f080c0d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 982182575 0 0) 0x7f07f0003530 con 0x7f080c15e000 2024-03-24T00:39:05.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c1628d0 msgr2=0x7f080c0d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f080b400640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c1628d0 0x7f080c0d16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 >> v1:172.21.15.53:6789/0 conn(0x7f080c0b42d0 legacy=0x7f080c06dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 --> v1:172.21.15.40:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c161df0 con 0x7f080c15e000 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 <== mon.0 v1:172.21.15.40:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 4123871081 0 0) 0x7f07fc001680 con 0x7f080c15e000 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f0803400640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c1628d0 0x7f080c0d16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.732+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 >> v1:172.21.15.40:6789/0 conn(0x7f080c15e000 legacy=0x7f080c0b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f080b400640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07f00039f0 0x7f07f0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f080b400640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c1628d0 0x7f07f0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f080b400640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0004d50 0x7f07f00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c161ba0 con 0x7f07f0004d50 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07f000bb60 con 0x7f07f00039f0 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07f000bd60 con 0x7f080c1628d0 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f0810a00640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0004d50 0x7f07f00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f080be00640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07f00039f0 0x7f07f0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f0803400640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c1628d0 0x7f07f0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f0810a00640 1 -- 172.21.15.40:0/3026902673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c1628d0 msgr2=0x7f07f0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f0810a00640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f080c1628d0 0x7f07f0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f0810a00640 1 -- 172.21.15.40:0/3026902673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07f00039f0 msgr2=0x7f07f0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:05.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f0810a00640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07f00039f0 0x7f07f0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f0810a00640 1 -- 172.21.15.40:0/3026902673 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f0003530 con 0x7f07f0004d50 2024-03-24T00:39:05.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f0810a00640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0004d50 0x7f07f00051d0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f080c06f710 tx=0x7f080401c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b80cc5e1c06ce5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804001a90 con 0x7f07f0004d50 2024-03-24T00:39:05.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0804003b80 con 0x7f07f0004d50 2024-03-24T00:39:05.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.736+0000 7f080b400640 1 -- 172.21.15.40:0/3026902673 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804005080 con 0x7f07f0004d50 2024-03-24T00:39:05.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 -- 172.21.15.40:0/3026902673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0004d50 msgr2=0x7f07f00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 --2- 172.21.15.40:0/3026902673 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f07f0004d50 0x7f07f00051d0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f080c06f710 tx=0x7f080401c720 comp rx=0 tx=0).stop 2024-03-24T00:39:05.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0810a00640 1 -- 172.21.15.40:0/3026902673 reap_dead start 2024-03-24T00:39:05.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 -- 172.21.15.40:0/3026902673 shutdown_connections 2024-03-24T00:39:05.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 -- 172.21.15.40:0/3026902673 >> 172.21.15.40:0/3026902673 conn(0x7f080c0d0bb0 msgr2=0x7f080c0d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:05.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 -- 172.21.15.40:0/3026902673 shutdown_connections 2024-03-24T00:39:05.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 -- 172.21.15.40:0/3026902673 wait complete. 2024-03-24T00:39:05.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 Processor -- start 2024-03-24T00:39:05.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 -- start start 2024-03-24T00:39:05.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07f0004d50 0x7f080c15fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c14e650 0x7f080c15c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f080c0af570 0x7f080c0af9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c0b5b30 con 0x7f080c14e650 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c1623b0 con 0x7f080c0af570 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0812a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c0b5cb0 con 0x7f07f0004d50 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0803400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c14e650 0x7f080c15c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f080be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07f0004d50 0x7f080c15fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0803400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c14e650 0x7f080c15c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34590/0 (socket says 172.21.15.40:34590) 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0803400640 1 -- 172.21.15.40:0/1746155886 learned_addr learned my addr 172.21.15.40:0/1746155886 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0810a00640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f080c0af570 0x7f080c0af9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0803400640 1 -- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07f0004d50 msgr2=0x7f080c15fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0803400640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07f0004d50 0x7f080c15fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0803400640 1 -- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f080c0af570 msgr2=0x7f080c0af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0803400640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f080c0af570 0x7f080c0af9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0803400640 1 -- 172.21.15.40:0/1746155886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c072a40 con 0x7f080c14e650 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.740+0000 7f0810a00640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f080c0af570 0x7f080c0af9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:05.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.744+0000 7f0803400640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c14e650 0x7f080c15c9e0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f07f40116d0 tx=0x7f07f4011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.744+0000 7f0808c00640 1 -- 172.21.15.40:0/1746155886 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f4019020 con 0x7f080c14e650 2024-03-24T00:39:05.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.744+0000 7f0808c00640 1 -- 172.21.15.40:0/1746155886 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f4004230 con 0x7f080c14e650 2024-03-24T00:39:05.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.744+0000 7f0808c00640 1 -- 172.21.15.40:0/1746155886 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f400c070 con 0x7f080c14e650 2024-03-24T00:39:05.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.744+0000 7f0812a00640 1 -- 172.21.15.40:0/1746155886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080c0aea90 con 0x7f080c14e650 2024-03-24T00:39:05.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.744+0000 7f0812a00640 1 -- 172.21.15.40:0/1746155886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c0af050 con 0x7f080c14e650 2024-03-24T00:39:05.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.744+0000 7f0812a00640 1 -- 172.21.15.40:0/1746155886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080c068170 con 0x7f080c14e650 2024-03-24T00:39:05.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.744+0000 7f0808c00640 1 -- 172.21.15.40:0/1746155886 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f07f401c020 con 0x7f080c14e650 2024-03-24T00:39:05.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.744+0000 7f0808c00640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07e8067690 0x7f07e8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:05.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.744+0000 7f0808c00640 1 -- 172.21.15.40:0/1746155886 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f07f4085960 con 0x7f080c14e650 2024-03-24T00:39:05.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.748+0000 7f080be00640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07e8067690 0x7f07e8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:05.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.748+0000 7f080be00640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07e8067690 0x7f07e8069b50 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f07fc007cd0 tx=0x7f07fc001dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:05.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.752+0000 7f0808c00640 1 -- 172.21.15.40:0/1746155886 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f07f4023020 con 0x7f080c14e650 2024-03-24T00:39:05.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:05.912+0000 7f0812a00640 1 -- 172.21.15.40:0/1746155886 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f080c067b90 con 0x7f080c14e650 2024-03-24T00:39:06.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0808c00640 1 -- 172.21.15.40:0/1746155886 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v116) v1 ==== 138+0+0 (secure 0 0 0) 0x7f07f4056890 con 0x7f080c14e650 2024-03-24T00:39:06.124 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_metadata' created 2024-03-24T00:39:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0812a00640 1 -- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07e8067690 msgr2=0x7f07e8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0812a00640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07e8067690 0x7f07e8069b50 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f07fc007cd0 tx=0x7f07fc001dc0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0812a00640 1 -- 172.21.15.40:0/1746155886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c14e650 msgr2=0x7f080c15c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0812a00640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c14e650 0x7f080c15c9e0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f07f40116d0 tx=0x7f07f4011b90 comp rx=0 tx=0).stop 2024-03-24T00:39:06.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0812a00640 1 -- 172.21.15.40:0/1746155886 shutdown_connections 2024-03-24T00:39:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0812a00640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f07e8067690 0x7f07e8069b50 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0812a00640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f080c0af570 0x7f080c0af9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0812a00640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f080c14e650 0x7f080c15c9e0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0812a00640 1 --2- 172.21.15.40:0/1746155886 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f07f0004d50 0x7f080c15fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0812a00640 1 -- 172.21.15.40:0/1746155886 >> 172.21.15.40:0/1746155886 conn(0x7f080c0d0bb0 msgr2=0x7f080c06e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0812a00640 1 -- 172.21.15.40:0/1746155886 shutdown_connections 2024-03-24T00:39:06.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.120+0000 7f0812a00640 1 -- 172.21.15.40:0/1746155886 wait complete. 2024-03-24T00:39:06.150 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:39:06.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.296+0000 7fa1ca400640 1 Processor -- start 2024-03-24T00:39:06.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.296+0000 7fa1ca400640 1 -- start start 2024-03-24T00:39:06.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.296+0000 7fa1ca400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa1c40aec70 0x7fa1c40ae6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.296+0000 7fa1ca400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c4070ed0 con 0x7fa1c406d3a0 2024-03-24T00:39:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.296+0000 7fa1ca400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c40cf240 con 0x7fa1c415a560 2024-03-24T00:39:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.296+0000 7fa1ca400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1c40cf520 con 0x7fa1c40aec70 2024-03-24T00:39:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.296+0000 7fa1baa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa1c40aec70 0x7fa1c40ae6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.296+0000 7fa1baa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fa1c40aec70 0x7fa1c40ae6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34604/0 (socket says 172.21.15.40:34604) 2024-03-24T00:39:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.296+0000 7fa1baa00640 1 -- 172.21.15.40:0/2962967643 learned_addr learned my addr 172.21.15.40:0/2962967643 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.296+0000 7fa1baa00640 1 -- 172.21.15.40:0/2962967643 >> v1:172.21.15.53:6789/0 conn(0x7fa1c415a560 legacy=0x7fa1c40cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.296+0000 7fa1baa00640 1 -- 172.21.15.40:0/2962967643 >> v1:172.21.15.40:6789/0 conn(0x7fa1c406d3a0 legacy=0x7fa1c40b3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.296+0000 7fa1baa00640 1 -- 172.21.15.40:0/2962967643 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c40b3aa0 con 0x7fa1c40aec70 2024-03-24T00:39:06.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1baa00640 1 --2- 172.21.15.40:0/2962967643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c40aec70 0x7fa1c40ae6b0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009870 tx=0x7fa1ac01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799a96fd41f522d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:06.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1c2a00640 1 -- 172.21.15.40:0/2962967643 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac0b2070 con 0x7fa1c40aec70 2024-03-24T00:39:06.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1c2a00640 1 -- 172.21.15.40:0/2962967643 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ac0a9070 con 0x7fa1c40aec70 2024-03-24T00:39:06.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1c2a00640 1 -- 172.21.15.40:0/2962967643 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac0ad880 con 0x7fa1c40aec70 2024-03-24T00:39:06.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1ca400640 1 -- 172.21.15.40:0/2962967643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c40aec70 msgr2=0x7fa1c40ae6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1ca400640 1 --2- 172.21.15.40:0/2962967643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c40aec70 0x7fa1c40ae6b0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac009870 tx=0x7fa1ac01c440 comp rx=0 tx=0).stop 2024-03-24T00:39:06.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1ca400640 1 -- 172.21.15.40:0/2962967643 shutdown_connections 2024-03-24T00:39:06.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1ca400640 1 --2- 172.21.15.40:0/2962967643 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c40aec70 0x7fa1c40ae6b0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1ca400640 1 -- 172.21.15.40:0/2962967643 >> 172.21.15.40:0/2962967643 conn(0x7fa1c40b1660 msgr2=0x7fa1c40b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:06.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1ca400640 1 -- 172.21.15.40:0/2962967643 shutdown_connections 2024-03-24T00:39:06.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1ca400640 1 -- 172.21.15.40:0/2962967643 wait complete. 2024-03-24T00:39:06.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1ca400640 1 Processor -- start 2024-03-24T00:39:06.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.300+0000 7fa1ca400640 1 -- start start 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1ca400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c406d3a0 0x7fa1c41f7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1ca400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c40aec70 0x7fa1c40c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1ca400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c415a560 0x7fa1c4150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1ca400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c415aac0 con 0x7fa1c40aec70 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1ca400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c415a940 con 0x7fa1c406d3a0 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1ca400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c4168a70 con 0x7fa1c415a560 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1baa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c40aec70 0x7fa1c40c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1baa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c40aec70 0x7fa1c40c4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34608/0 (socket says 172.21.15.40:34608) 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1baa00640 1 -- 172.21.15.40:0/543820037 learned_addr learned my addr 172.21.15.40:0/543820037 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1c3400640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c406d3a0 0x7fa1c41f7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1c3e00640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c415a560 0x7fa1c4150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1baa00640 1 -- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c415a560 msgr2=0x7fa1c4150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.309 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1baa00640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c415a560 0x7fa1c4150770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1baa00640 1 -- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c406d3a0 msgr2=0x7fa1c41f7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:06.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1baa00640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c406d3a0 0x7fa1c41f7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:06.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1baa00640 1 -- 172.21.15.40:0/543820037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c40c4fd0 con 0x7fa1c40aec70 2024-03-24T00:39:06.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1baa00640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c40aec70 0x7fa1c40c4a90 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac019050 tx=0x7fa1ac0191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:06.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1bbe00640 1 -- 172.21.15.40:0/543820037 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac0b2040 con 0x7fa1c40aec70 2024-03-24T00:39:06.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1bbe00640 1 -- 172.21.15.40:0/543820037 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1ac0b6640 con 0x7fa1c40aec70 2024-03-24T00:39:06.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1bbe00640 1 -- 172.21.15.40:0/543820037 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac0b6a30 con 0x7fa1c40aec70 2024-03-24T00:39:06.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1ca400640 1 -- 172.21.15.40:0/543820037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c4150cb0 con 0x7fa1c40aec70 2024-03-24T00:39:06.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1ca400640 1 -- 172.21.15.40:0/543820037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c41f80e0 con 0x7fa1c40aec70 2024-03-24T00:39:06.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.304+0000 7fa1ca400640 1 -- 172.21.15.40:0/543820037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c4068170 con 0x7fa1c40aec70 2024-03-24T00:39:06.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.308+0000 7fa1bbe00640 1 -- 172.21.15.40:0/543820037 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa1ac0ab070 con 0x7fa1c40aec70 2024-03-24T00:39:06.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.308+0000 7fa1bbe00640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1a0067660 0x7fa1a0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:06.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.308+0000 7fa1bbe00640 1 -- 172.21.15.40:0/543820037 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fa1ac11fde0 con 0x7fa1c40aec70 2024-03-24T00:39:06.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.308+0000 7fa1c3400640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1a0067660 0x7fa1a0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:06.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.308+0000 7fa1c3400640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1a0067660 0x7fa1a0069b20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40012f0 tx=0x7fa1b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:06.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.312+0000 7fa1bbe00640 1 -- 172.21.15.40:0/543820037 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa1ac016150 con 0x7fa1c40aec70 2024-03-24T00:39:06.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:06.472+0000 7fa1ca400640 1 -- 172.21.15.40:0/543820037 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa1c40ae900 con 0x7fa1c40aec70 2024-03-24T00:39:07.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.120+0000 7fa1bbe00640 1 -- 172.21.15.40:0/543820037 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v117) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa1ac0f0cb0 con 0x7fa1c40aec70 2024-03-24T00:39:07.127 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'cephfs_data' created 2024-03-24T00:39:07.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.124+0000 7fa1ca400640 1 -- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1a0067660 msgr2=0x7fa1a0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:07.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.124+0000 7fa1ca400640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1a0067660 0x7fa1a0069b20 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fa1b40012f0 tx=0x7fa1b4009040 comp rx=0 tx=0).stop 2024-03-24T00:39:07.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.124+0000 7fa1ca400640 1 -- 172.21.15.40:0/543820037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c40aec70 msgr2=0x7fa1c40c4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:07.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.124+0000 7fa1ca400640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c40aec70 0x7fa1c40c4a90 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac019050 tx=0x7fa1ac0191e0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.124+0000 7fa1ca400640 1 -- 172.21.15.40:0/543820037 shutdown_connections 2024-03-24T00:39:07.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.124+0000 7fa1ca400640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa1a0067660 0x7fa1a0069b20 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.124+0000 7fa1ca400640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1c415a560 0x7fa1c4150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.124+0000 7fa1ca400640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa1c40aec70 0x7fa1c40c4a90 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.124+0000 7fa1ca400640 1 --2- 172.21.15.40:0/543820037 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1c406d3a0 0x7fa1c41f7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.124+0000 7fa1ca400640 1 -- 172.21.15.40:0/543820037 >> 172.21.15.40:0/543820037 conn(0x7fa1c40b1660 msgr2=0x7fa1c40cce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:07.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.124+0000 7fa1ca400640 1 -- 172.21.15.40:0/543820037 shutdown_connections 2024-03-24T00:39:07.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.124+0000 7fa1ca400640 1 -- 172.21.15.40:0/543820037 wait complete. 2024-03-24T00:39:07.167 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:39:07.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.308+0000 7ff517c00640 1 Processor -- start 2024-03-24T00:39:07.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff517c00640 1 -- start start 2024-03-24T00:39:07.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff517c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff51006cb80 0x7ff5100b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:07.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff517c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5100b4840 0x7ff510071b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:07.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff517c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5100b3260 con 0x7ff51015ac60 2024-03-24T00:39:07.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff517c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5100b3e70 con 0x7ff5100b4840 2024-03-24T00:39:07.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff517c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5100b3540 con 0x7ff51006cb80 2024-03-24T00:39:07.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff515c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5100b4840 0x7ff510071b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:07.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff515c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff5100b4840 0x7ff510071b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49314/0 (socket says 172.21.15.40:49314) 2024-03-24T00:39:07.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff515c00640 1 -- 172.21.15.40:0/8839152 learned_addr learned my addr 172.21.15.40:0/8839152 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:07.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff50fe00640 1 --2- 172.21.15.40:0/8839152 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff51006cb80 0x7ff5100b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:07.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff515c00640 1 -- 172.21.15.40:0/8839152 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff51006cb80 msgr2=0x7ff5100b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:07.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff515c00640 1 --2- 172.21.15.40:0/8839152 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff51006cb80 0x7ff5100b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff515c00640 1 -- 172.21.15.40:0/8839152 >> v1:172.21.15.53:6789/0 conn(0x7ff51015ac60 legacy=0x7ff5100b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:07.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.312+0000 7ff515c00640 1 -- 172.21.15.40:0/8839152 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5100b3820 con 0x7ff5100b4840 2024-03-24T00:39:07.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff515c00640 1 --2- 172.21.15.40:0/8839152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5100b4840 0x7ff510071b00 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7ff508009e50 tx=0x7ff50801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=790ecc5e90dd333c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:07.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff50f400640 1 -- 172.21.15.40:0/8839152 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5080a5660 con 0x7ff5100b4840 2024-03-24T00:39:07.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff50f400640 1 -- 172.21.15.40:0/8839152 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5080a5800 con 0x7ff5100b4840 2024-03-24T00:39:07.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff50f400640 1 -- 172.21.15.40:0/8839152 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5080a5bd0 con 0x7ff5100b4840 2024-03-24T00:39:07.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff517c00640 1 -- 172.21.15.40:0/8839152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5100b4840 msgr2=0x7ff510071b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:07.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff517c00640 1 --2- 172.21.15.40:0/8839152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5100b4840 0x7ff510071b00 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7ff508009e50 tx=0x7ff50801c9a0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff517c00640 1 -- 172.21.15.40:0/8839152 shutdown_connections 2024-03-24T00:39:07.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff517c00640 1 --2- 172.21.15.40:0/8839152 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff51006cb80 0x7ff5100b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff517c00640 1 --2- 172.21.15.40:0/8839152 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff5100b4840 0x7ff510071b00 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff517c00640 1 -- 172.21.15.40:0/8839152 >> 172.21.15.40:0/8839152 conn(0x7ff5100b1660 msgr2=0x7ff5100b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:07.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff517c00640 1 -- 172.21.15.40:0/8839152 shutdown_connections 2024-03-24T00:39:07.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff517c00640 1 -- 172.21.15.40:0/8839152 wait complete. 2024-03-24T00:39:07.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff517c00640 1 Processor -- start 2024-03-24T00:39:07.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.316+0000 7ff517c00640 1 -- start start 2024-03-24T00:39:07.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff517c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff51006cb80 0x7ff510158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:07.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff517c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5100b4840 0x7ff5101361e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:07.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff517c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff51015ac60 0x7ff510136720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff517c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff510167c50 con 0x7ff51006cb80 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff517c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff510167ad0 con 0x7ff51015ac60 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff517c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff510167dd0 con 0x7ff5100b4840 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff515200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff51006cb80 0x7ff510158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff515200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff51006cb80 0x7ff510158a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49330/0 (socket says 172.21.15.40:49330) 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff515200640 1 -- 172.21.15.40:0/900681016 learned_addr learned my addr 172.21.15.40:0/900681016 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff50fe00640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5100b4840 0x7ff5101361e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff515c00640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff51015ac60 0x7ff510136720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff515200640 1 -- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5100b4840 msgr2=0x7ff5101361e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff515200640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5100b4840 0x7ff5101361e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff515200640 1 -- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff51015ac60 msgr2=0x7ff510136720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff515200640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff51015ac60 0x7ff510136720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff515200640 1 -- 172.21.15.40:0/900681016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff510159070 con 0x7ff51006cb80 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff50fe00640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5100b4840 0x7ff5101361e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:07.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff515200640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff51006cb80 0x7ff510158a60 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ff50400c450 tx=0x7ff50400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:07.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff50cc00640 1 -- 172.21.15.40:0/900681016 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504016020 con 0x7ff51006cb80 2024-03-24T00:39:07.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff50cc00640 1 -- 172.21.15.40:0/900681016 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5040052d0 con 0x7ff51006cb80 2024-03-24T00:39:07.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff50cc00640 1 -- 172.21.15.40:0/900681016 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504007260 con 0x7ff51006cb80 2024-03-24T00:39:07.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff517c00640 1 -- 172.21.15.40:0/900681016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff510159350 con 0x7ff51006cb80 2024-03-24T00:39:07.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff517c00640 1 -- 172.21.15.40:0/900681016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff510159910 con 0x7ff51006cb80 2024-03-24T00:39:07.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.320+0000 7ff517c00640 1 -- 172.21.15.40:0/900681016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff510068170 con 0x7ff51006cb80 2024-03-24T00:39:07.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.324+0000 7ff50cc00640 1 -- 172.21.15.40:0/900681016 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff504007400 con 0x7ff51006cb80 2024-03-24T00:39:07.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.324+0000 7ff50cc00640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff4f0067660 0x7ff4f0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:07.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.324+0000 7ff50cc00640 1 -- 172.21.15.40:0/900681016 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ff504085f00 con 0x7ff51006cb80 2024-03-24T00:39:07.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.324+0000 7ff50fe00640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff4f0067660 0x7ff4f0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:07.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.324+0000 7ff50fe00640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff4f0067660 0x7ff4f0069b20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff510158fa0 tx=0x7ff4fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:07.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.328+0000 7ff50cc00640 1 -- 172.21.15.40:0/900681016 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff504020020 con 0x7ff51006cb80 2024-03-24T00:39:07.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:07.524+0000 7ff517c00640 1 -- 172.21.15.40:0/900681016 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff510135e00 con 0x7ff51006cb80 2024-03-24T00:39:07.998 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:39:08.002 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:39:08.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.136+0000 7ff50cc00640 1 -- 172.21.15.40:0/900681016 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v124) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff504005480 con 0x7ff51006cb80 2024-03-24T00:39:08.141 INFO:teuthology.orchestra.run.smithi040.stderr:new fs with metadata pool 18 and data pool 19 2024-03-24T00:39:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.140+0000 7ff517c00640 1 -- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff4f0067660 msgr2=0x7ff4f0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.140+0000 7ff517c00640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff4f0067660 0x7ff4f0069b20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff510158fa0 tx=0x7ff4fc008040 comp rx=0 tx=0).stop 2024-03-24T00:39:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.140+0000 7ff517c00640 1 -- 172.21.15.40:0/900681016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff51006cb80 msgr2=0x7ff510158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.140+0000 7ff517c00640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff51006cb80 0x7ff510158a60 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ff50400c450 tx=0x7ff50400c910 comp rx=0 tx=0).stop 2024-03-24T00:39:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.140+0000 7ff517c00640 1 -- 172.21.15.40:0/900681016 shutdown_connections 2024-03-24T00:39:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.140+0000 7ff517c00640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff4f0067660 0x7ff4f0069b20 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.140+0000 7ff517c00640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff51015ac60 0x7ff510136720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.140+0000 7ff517c00640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff5100b4840 0x7ff5101361e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.140+0000 7ff517c00640 1 --2- 172.21.15.40:0/900681016 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff51006cb80 0x7ff510158a60 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.140+0000 7ff517c00640 1 -- 172.21.15.40:0/900681016 >> 172.21.15.40:0/900681016 conn(0x7ff5100b1660 msgr2=0x7ff5100af7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.140+0000 7ff517c00640 1 -- 172.21.15.40:0/900681016 shutdown_connections 2024-03-24T00:39:08.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.140+0000 7ff517c00640 1 -- 172.21.15.40:0/900681016 wait complete. 2024-03-24T00:39:08.192 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:08.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42d2200640 1 Processor -- start 2024-03-24T00:39:08.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42d2200640 1 -- start start 2024-03-24T00:39:08.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42d2200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42cc13ec80 0x7f42cc1414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42d2200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42cc0ce800 0x7f42cc071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42d2200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42cc070e00 con 0x7f42cc1419f0 2024-03-24T00:39:08.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42d2200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42cc141e50 con 0x7f42cc0ce800 2024-03-24T00:39:08.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42d2200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc0cee20 con 0x7f42cc13ec80 2024-03-24T00:39:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42cb400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42cc13ec80 0x7f42cc1414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42cbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42cc0ce800 0x7f42cc071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42cb400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42cc13ec80 0x7f42cc1414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:58850/0 (socket says 172.21.15.40:58850) 2024-03-24T00:39:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42cb400640 1 -- 172.21.15.40:0/351982913 learned_addr learned my addr 172.21.15.40:0/351982913 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42cbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42cc0ce800 0x7f42cc071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:46938/0 (socket says 172.21.15.40:46938) 2024-03-24T00:39:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42cb400640 1 -- 172.21.15.40:0/351982913 >> v2:172.21.15.53:3300/0 conn(0x7f42cc0ce800 msgr2=0x7f42cc071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42cb400640 1 --2- 172.21.15.40:0/351982913 >> v2:172.21.15.53:3300/0 conn(0x7f42cc0ce800 0x7f42cc071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42cb400640 1 -- 172.21.15.40:0/351982913 >> v1:172.21.15.53:6789/0 conn(0x7f42cc1419f0 legacy=0x7f42cc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.332+0000 7f42cb400640 1 -- 172.21.15.40:0/351982913 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc0cf100 con 0x7f42cc13ec80 2024-03-24T00:39:08.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.336+0000 7f42cb400640 1 --2- 172.21.15.40:0/351982913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42cc13ec80 0x7f42cc1414b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009870 tx=0x7f42bc01c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd5f8b45c1e635bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:08.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.336+0000 7f42caa00640 1 -- 172.21.15.40:0/351982913 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0a9640 con 0x7f42cc13ec80 2024-03-24T00:39:08.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.336+0000 7f42caa00640 1 -- 172.21.15.40:0/351982913 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc0a97e0 con 0x7f42cc13ec80 2024-03-24T00:39:08.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.336+0000 7f42caa00640 1 -- 172.21.15.40:0/351982913 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0a9bb0 con 0x7f42cc13ec80 2024-03-24T00:39:08.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.336+0000 7f42d2200640 1 -- 172.21.15.40:0/351982913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42cc13ec80 msgr2=0x7f42cc1414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.336+0000 7f42d2200640 1 --2- 172.21.15.40:0/351982913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42cc13ec80 0x7f42cc1414b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f42bc009870 tx=0x7f42bc01c560 comp rx=0 tx=0).stop 2024-03-24T00:39:08.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.336+0000 7f42d2200640 1 -- 172.21.15.40:0/351982913 shutdown_connections 2024-03-24T00:39:08.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.336+0000 7f42d2200640 1 --2- 172.21.15.40:0/351982913 >> v2:172.21.15.53:3300/0 conn(0x7f42cc0ce800 0x7f42cc071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.336+0000 7f42d2200640 1 --2- 172.21.15.40:0/351982913 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42cc13ec80 0x7f42cc1414b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.336+0000 7f42d2200640 1 -- 172.21.15.40:0/351982913 >> 172.21.15.40:0/351982913 conn(0x7f42cc0b17b0 msgr2=0x7f42cc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:08.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.336+0000 7f42d2200640 1 -- 172.21.15.40:0/351982913 shutdown_connections 2024-03-24T00:39:08.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.336+0000 7f42d2200640 1 -- 172.21.15.40:0/351982913 wait complete. 2024-03-24T00:39:08.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42d2200640 1 Processor -- start 2024-03-24T00:39:08.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42d2200640 1 -- start start 2024-03-24T00:39:08.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42d2200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc0ce800 0x7f42cc139c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42d2200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42cc13ec80 0x7f42cc1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42d2200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42cc1419f0 0x7f42cc155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42d2200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc0723e0 con 0x7f42cc0ce800 2024-03-24T00:39:08.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42d2200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc072260 con 0x7f42cc13ec80 2024-03-24T00:39:08.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42d2200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42cc072560 con 0x7f42cc1419f0 2024-03-24T00:39:08.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42cb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc0ce800 0x7f42cc139c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42cb400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc0ce800 0x7f42cc139c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49344/0 (socket says 172.21.15.40:49344) 2024-03-24T00:39:08.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42cb400640 1 -- 172.21.15.40:0/394684623 learned_addr learned my addr 172.21.15.40:0/394684623 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:08.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42c2a00640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42cc13ec80 0x7f42cc1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42cbe00640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42cc1419f0 0x7f42cc155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42cb400640 1 -- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42cc1419f0 msgr2=0x7f42cc155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42cb400640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42cc1419f0 0x7f42cc155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42cb400640 1 -- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42cc13ec80 msgr2=0x7f42cc1550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42cb400640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42cc13ec80 0x7f42cc1550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42cb400640 1 -- 172.21.15.40:0/394684623 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc1594d0 con 0x7f42cc0ce800 2024-03-24T00:39:08.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42c2a00640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42cc13ec80 0x7f42cc1550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:08.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42cb400640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc0ce800 0x7f42cc139c20 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f42bc01ac10 tx=0x7f42bc019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:08.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42c3e00640 1 -- 172.21.15.40:0/394684623 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0a9640 con 0x7f42cc0ce800 2024-03-24T00:39:08.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42c3e00640 1 -- 172.21.15.40:0/394684623 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42bc0b1800 con 0x7f42cc0ce800 2024-03-24T00:39:08.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42d2200640 1 -- 172.21.15.40:0/394684623 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42cc1597b0 con 0x7f42cc0ce800 2024-03-24T00:39:08.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.340+0000 7f42c3e00640 1 -- 172.21.15.40:0/394684623 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42bc0b1bf0 con 0x7f42cc0ce800 2024-03-24T00:39:08.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.344+0000 7f42d2200640 1 -- 172.21.15.40:0/394684623 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42cc150dd0 con 0x7f42cc0ce800 2024-03-24T00:39:08.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.344+0000 7f42d2200640 1 -- 172.21.15.40:0/394684623 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42cc068170 con 0x7f42cc0ce800 2024-03-24T00:39:08.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.344+0000 7f42c3e00640 1 -- 172.21.15.40:0/394684623 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f42bc0c43f0 con 0x7f42cc0ce800 2024-03-24T00:39:08.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.344+0000 7f42c3e00640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f42a8067690 0x7f42a8069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.344+0000 7f42c3e00640 1 -- 172.21.15.40:0/394684623 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42bc11ff80 con 0x7f42cc0ce800 2024-03-24T00:39:08.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.344+0000 7f42c2a00640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f42a8067690 0x7f42a8069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.348+0000 7f42c2a00640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f42a8067690 0x7f42a8069b50 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f42cc1f7f90 tx=0x7f42b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:08.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.352+0000 7f42c3e00640 1 -- 172.21.15.40:0/394684623 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f42bc0ad020 con 0x7f42cc0ce800 2024-03-24T00:39:08.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.516+0000 7f42d2200640 1 -- 172.21.15.40:0/394684623 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42cc072b90 con 0x7f42cc0ce800 2024-03-24T00:39:08.521 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.516+0000 7f42c3e00640 1 -- 172.21.15.40:0/394684623 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16632 (secure 0 0 0) 0x7f42bc016150 con 0x7f42cc0ce800 2024-03-24T00:39:08.521 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:08.521 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":118,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:39:07.531564+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:05.964092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:06.525558+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:08.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.516+0000 7f42d2200640 1 -- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f42a8067690 msgr2=0x7f42a8069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.516+0000 7f42d2200640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f42a8067690 0x7f42a8069b50 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f42cc1f7f90 tx=0x7f42b4009040 comp rx=0 tx=0).stop 2024-03-24T00:39:08.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.516+0000 7f42d2200640 1 -- 172.21.15.40:0/394684623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc0ce800 msgr2=0x7f42cc139c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.516+0000 7f42d2200640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc0ce800 0x7f42cc139c20 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f42bc01ac10 tx=0x7f42bc019300 comp rx=0 tx=0).stop 2024-03-24T00:39:08.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.516+0000 7f42d2200640 1 -- 172.21.15.40:0/394684623 shutdown_connections 2024-03-24T00:39:08.524 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.516+0000 7f42d2200640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f42a8067690 0x7f42a8069b50 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.516+0000 7f42d2200640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42cc1419f0 0x7f42cc155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.516+0000 7f42d2200640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42cc13ec80 0x7f42cc1550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.516+0000 7f42d2200640 1 --2- 172.21.15.40:0/394684623 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f42cc0ce800 0x7f42cc139c20 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.516+0000 7f42d2200640 1 -- 172.21.15.40:0/394684623 >> 172.21.15.40:0/394684623 conn(0x7f42cc0b17b0 msgr2=0x7f42cc13f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:08.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.520+0000 7f42d2200640 1 -- 172.21.15.40:0/394684623 shutdown_connections 2024-03-24T00:39:08.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.520+0000 7f42d2200640 1 -- 172.21.15.40:0/394684623 wait complete. 2024-03-24T00:39:08.542 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:08.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.676+0000 7fa518000640 1 Processor -- start 2024-03-24T00:39:08.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.676+0000 7fa518000640 1 -- start start 2024-03-24T00:39:08.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa518000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa51006e620 0x7fa5100b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa518000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa5100ccfa0 0x7fa5100cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa518000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa510070f60 con 0x7fa51006ea00 2024-03-24T00:39:08.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa518000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5100b4c00 con 0x7fa5100ccfa0 2024-03-24T00:39:08.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa518000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100b3620 con 0x7fa51006e620 2024-03-24T00:39:08.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa514c00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fa51006ea00 0x7fa5100b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42626/0 (socket says 172.21.15.40:42626) 2024-03-24T00:39:08.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa514c00640 1 -- 172.21.15.40:0/2223170521 learned_addr learned my addr 172.21.15.40:0/2223170521 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:08.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa515600640 1 --2- 172.21.15.40:0/2223170521 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa51006e620 0x7fa5100b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa516000640 1 --2- 172.21.15.40:0/2223170521 >> v2:172.21.15.53:3300/0 conn(0x7fa5100ccfa0 0x7fa5100cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa50fe00640 1 -- 172.21.15.40:0/2223170521 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016447585 0 0) 0x7fa510070f60 con 0x7fa51006ea00 2024-03-24T00:39:08.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa50fe00640 1 -- 172.21.15.40:0/2223170521 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f8003530 con 0x7fa51006ea00 2024-03-24T00:39:08.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa515600640 1 -- 172.21.15.40:0/2223170521 >> v2:172.21.15.53:3300/0 conn(0x7fa5100ccfa0 msgr2=0x7fa5100cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:08.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa515600640 1 --2- 172.21.15.40:0/2223170521 >> v2:172.21.15.53:3300/0 conn(0x7fa5100ccfa0 0x7fa5100cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa515600640 1 -- 172.21.15.40:0/2223170521 >> v1:172.21.15.40:6789/0 conn(0x7fa51006ea00 legacy=0x7fa5100b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.680+0000 7fa515600640 1 -- 172.21.15.40:0/2223170521 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5100b3900 con 0x7fa51006e620 2024-03-24T00:39:08.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa515600640 1 --2- 172.21.15.40:0/2223170521 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa51006e620 0x7fa5100b46c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa5040097c0 tx=0x7fa50401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14addb8cf19af49f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:08.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa50fe00640 1 -- 172.21.15.40:0/2223170521 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5040b0070 con 0x7fa51006e620 2024-03-24T00:39:08.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa50fe00640 1 -- 172.21.15.40:0/2223170521 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5040ab600 con 0x7fa51006e620 2024-03-24T00:39:08.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa50fe00640 1 -- 172.21.15.40:0/2223170521 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5040ab9d0 con 0x7fa51006e620 2024-03-24T00:39:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa518000640 1 -- 172.21.15.40:0/2223170521 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa51006e620 msgr2=0x7fa5100b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa518000640 1 --2- 172.21.15.40:0/2223170521 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa51006e620 0x7fa5100b46c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa5040097c0 tx=0x7fa50401c3e0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa518000640 1 -- 172.21.15.40:0/2223170521 shutdown_connections 2024-03-24T00:39:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa518000640 1 --2- 172.21.15.40:0/2223170521 >> v2:172.21.15.53:3300/0 conn(0x7fa5100ccfa0 0x7fa5100cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa518000640 1 --2- 172.21.15.40:0/2223170521 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa51006e620 0x7fa5100b46c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa518000640 1 -- 172.21.15.40:0/2223170521 >> 172.21.15.40:0/2223170521 conn(0x7fa5100b17b0 msgr2=0x7fa5100b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa518000640 1 -- 172.21.15.40:0/2223170521 shutdown_connections 2024-03-24T00:39:08.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa518000640 1 -- 172.21.15.40:0/2223170521 wait complete. 2024-03-24T00:39:08.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa518000640 1 Processor -- start 2024-03-24T00:39:08.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.684+0000 7fa518000640 1 -- start start 2024-03-24T00:39:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa518000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51006e620 0x7fa51015a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa518000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa51006ea00 0x7fa510071300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa518000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5100ccfa0 0x7fa510071840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa518000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa510144920 con 0x7fa51006ea00 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa518000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa51006ede0 con 0x7fa51006e620 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa518000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5101fb150 con 0x7fa5100ccfa0 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa514c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa51006ea00 0x7fa510071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa514c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa51006ea00 0x7fa510071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49348/0 (socket says 172.21.15.40:49348) 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa514c00640 1 -- 172.21.15.40:0/1176582504 learned_addr learned my addr 172.21.15.40:0/1176582504 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa516000640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5100ccfa0 0x7fa510071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa515600640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51006e620 0x7fa51015a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa514c00640 1 -- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5100ccfa0 msgr2=0x7fa510071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa514c00640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5100ccfa0 0x7fa510071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa514c00640 1 -- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51006e620 msgr2=0x7fa51015a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa514c00640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51006e620 0x7fa51015a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa514c00640 1 -- 172.21.15.40:0/1176582504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5101fb3a0 con 0x7fa51006ea00 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa516000640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5100ccfa0 0x7fa510071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:08.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa514c00640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa51006ea00 0x7fa510071300 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00c480 tx=0x7fa4fc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa50d600640 1 -- 172.21.15.40:0/1176582504 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc017070 con 0x7fa51006ea00 2024-03-24T00:39:08.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa50d600640 1 -- 172.21.15.40:0/1176582504 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4fc005cf0 con 0x7fa51006ea00 2024-03-24T00:39:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa50d600640 1 -- 172.21.15.40:0/1176582504 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc007290 con 0x7fa51006ea00 2024-03-24T00:39:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa518000640 1 -- 172.21.15.40:0/1176582504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5101fb680 con 0x7fa51006ea00 2024-03-24T00:39:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa518000640 1 -- 172.21.15.40:0/1176582504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5101fbc40 con 0x7fa51006ea00 2024-03-24T00:39:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.688+0000 7fa518000640 1 -- 172.21.15.40:0/1176582504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa510068170 con 0x7fa51006ea00 2024-03-24T00:39:08.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.692+0000 7fa50d600640 1 -- 172.21.15.40:0/1176582504 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fa4fc01a070 con 0x7fa51006ea00 2024-03-24T00:39:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.692+0000 7fa50d600640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa4f0067660 0x7fa4f0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.692+0000 7fa50d600640 1 -- 172.21.15.40:0/1176582504 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4fc084d10 con 0x7fa51006ea00 2024-03-24T00:39:08.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.692+0000 7fa515600640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa4f0067660 0x7fa4f0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:08.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.692+0000 7fa515600640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa4f0067660 0x7fa4f0069b20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa5040ad000 tx=0x7fa5040a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:08.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.696+0000 7fa50d600640 1 -- 172.21.15.40:0/1176582504 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fa4fc08a020 con 0x7fa51006ea00 2024-03-24T00:39:08.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.856+0000 7fa518000640 1 -- 172.21.15.40:0/1176582504 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa510072f50 con 0x7fa51006ea00 2024-03-24T00:39:08.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa50d600640 1 -- 172.21.15.40:0/1176582504 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16632 (secure 0 0 0) 0x7fa4fc055980 con 0x7fa51006ea00 2024-03-24T00:39:08.864 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:08.864 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":118,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:39:07.531564+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:05.964092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:06.525558+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:08.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa518000640 1 -- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa4f0067660 msgr2=0x7fa4f0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa518000640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa4f0067660 0x7fa4f0069b20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa5040ad000 tx=0x7fa5040a6000 comp rx=0 tx=0).stop 2024-03-24T00:39:08.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa518000640 1 -- 172.21.15.40:0/1176582504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa51006ea00 msgr2=0x7fa510071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:08.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa518000640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa51006ea00 0x7fa510071300 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00c480 tx=0x7fa4fc00c940 comp rx=0 tx=0).stop 2024-03-24T00:39:08.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa518000640 1 -- 172.21.15.40:0/1176582504 shutdown_connections 2024-03-24T00:39:08.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa518000640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fa4f0067660 0x7fa4f0069b20 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa518000640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5100ccfa0 0x7fa510071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa518000640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fa51006ea00 0x7fa510071300 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa518000640 1 --2- 172.21.15.40:0/1176582504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa51006e620 0x7fa51015a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:08.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa518000640 1 -- 172.21.15.40:0/1176582504 >> 172.21.15.40:0/1176582504 conn(0x7fa5100b17b0 msgr2=0x7fa5100b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:08.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa518000640 1 -- 172.21.15.40:0/1176582504 shutdown_connections 2024-03-24T00:39:08.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:08.860+0000 7fa518000640 1 -- 172.21.15.40:0/1176582504 wait complete. 2024-03-24T00:39:08.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:39:09.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.024+0000 7fbd28a00640 1 Processor -- start 2024-03-24T00:39:09.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.024+0000 7fbd28a00640 1 -- start start 2024-03-24T00:39:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.024+0000 7fbd28a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbd2415bd00 0x7fbd240ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.024+0000 7fbd28a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbd2415ffd0 0x7fbd240cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.024+0000 7fbd28a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd24072d60 con 0x7fbd240728d0 2024-03-24T00:39:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.024+0000 7fbd28a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd240cf790 con 0x7fbd2415ffd0 2024-03-24T00:39:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.024+0000 7fbd28a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd24071de0 con 0x7fbd2415bd00 2024-03-24T00:39:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd22800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbd2415ffd0 0x7fbd240cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd22800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbd2415ffd0 0x7fbd240cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49354/0 (socket says 172.21.15.40:49354) 2024-03-24T00:39:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd22800640 1 -- 172.21.15.40:0/3854816537 learned_addr learned my addr 172.21.15.40:0/3854816537 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd21e00640 1 --2- 172.21.15.40:0/3854816537 >> v2:172.21.15.53:3300/0 conn(0x7fbd2415bd00 0x7fbd240ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd22800640 1 -- 172.21.15.40:0/3854816537 >> v2:172.21.15.53:3300/0 conn(0x7fbd2415bd00 msgr2=0x7fbd240ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd22800640 1 --2- 172.21.15.40:0/3854816537 >> v2:172.21.15.53:3300/0 conn(0x7fbd2415bd00 0x7fbd240ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd22800640 1 -- 172.21.15.40:0/3854816537 >> v1:172.21.15.40:6789/0 conn(0x7fbd240728d0 legacy=0x7fbd2415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd22800640 1 -- 172.21.15.40:0/3854816537 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd240cea50 con 0x7fbd2415ffd0 2024-03-24T00:39:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd21e00640 1 --2- 172.21.15.40:0/3854816537 >> v2:172.21.15.53:3300/0 conn(0x7fbd2415bd00 0x7fbd240ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd22800640 1 --2- 172.21.15.40:0/3854816537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2415ffd0 0x7fbd240cf250 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fbd240733c0 tx=0x7fbd1c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571dc744eb9548e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd21400640 1 -- 172.21.15.40:0/3854816537 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0aa650 con 0x7fbd2415ffd0 2024-03-24T00:39:09.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd21400640 1 -- 172.21.15.40:0/3854816537 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c0af070 con 0x7fbd2415ffd0 2024-03-24T00:39:09.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.028+0000 7fbd21400640 1 -- 172.21.15.40:0/3854816537 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0aaa10 con 0x7fbd2415ffd0 2024-03-24T00:39:09.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 -- 172.21.15.40:0/3854816537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2415ffd0 msgr2=0x7fbd240cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 --2- 172.21.15.40:0/3854816537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2415ffd0 0x7fbd240cf250 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fbd240733c0 tx=0x7fbd1c01c4f0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 -- 172.21.15.40:0/3854816537 shutdown_connections 2024-03-24T00:39:09.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 --2- 172.21.15.40:0/3854816537 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2415ffd0 0x7fbd240cf250 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 --2- 172.21.15.40:0/3854816537 >> v2:172.21.15.53:3300/0 conn(0x7fbd2415bd00 0x7fbd240ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 -- 172.21.15.40:0/3854816537 >> 172.21.15.40:0/3854816537 conn(0x7fbd240b1660 msgr2=0x7fbd240b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:09.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 -- 172.21.15.40:0/3854816537 shutdown_connections 2024-03-24T00:39:09.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 -- 172.21.15.40:0/3854816537 wait complete. 2024-03-24T00:39:09.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 Processor -- start 2024-03-24T00:39:09.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 -- start start 2024-03-24T00:39:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240728d0 0x7fbd2414d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd2415bd00 0x7fbd2415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2415ffd0 0x7fbd241557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2406d320 con 0x7fbd2415ffd0 2024-03-24T00:39:09.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2415c0e0 con 0x7fbd240728d0 2024-03-24T00:39:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd28a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2406d4a0 con 0x7fbd2415bd00 2024-03-24T00:39:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd22800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2415ffd0 0x7fbd241557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd22800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2415ffd0 0x7fbd241557b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49366/0 (socket says 172.21.15.40:49366) 2024-03-24T00:39:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd22800640 1 -- 172.21.15.40:0/1902879029 learned_addr learned my addr 172.21.15.40:0/1902879029 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd21e00640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240728d0 0x7fbd2414d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd19400640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd2415bd00 0x7fbd2415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd22800640 1 -- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd2415bd00 msgr2=0x7fbd2415ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd22800640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd2415bd00 0x7fbd2415ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd22800640 1 -- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240728d0 msgr2=0x7fbd2414d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd22800640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240728d0 0x7fbd2414d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd22800640 1 -- 172.21.15.40:0/1902879029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd240726a0 con 0x7fbd2415ffd0 2024-03-24T00:39:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd19400640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd2415bd00 0x7fbd2415ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd22800640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2415ffd0 0x7fbd241557b0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c0ac040 tx=0x7fbd1c01ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.032+0000 7fbd1aa00640 1 -- 172.21.15.40:0/1902879029 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0b3800 con 0x7fbd2415ffd0 2024-03-24T00:39:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.036+0000 7fbd1aa00640 1 -- 172.21.15.40:0/1902879029 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c0a6040 con 0x7fbd2415ffd0 2024-03-24T00:39:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.036+0000 7fbd1aa00640 1 -- 172.21.15.40:0/1902879029 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0b3c00 con 0x7fbd2415ffd0 2024-03-24T00:39:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.036+0000 7fbd28a00640 1 -- 172.21.15.40:0/1902879029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd2414dbf0 con 0x7fbd2415ffd0 2024-03-24T00:39:09.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.036+0000 7fbd28a00640 1 -- 172.21.15.40:0/1902879029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd24156d60 con 0x7fbd2415ffd0 2024-03-24T00:39:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.036+0000 7fbd28a00640 1 -- 172.21.15.40:0/1902879029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd24068170 con 0x7fbd2415ffd0 2024-03-24T00:39:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.036+0000 7fbd1aa00640 1 -- 172.21.15.40:0/1902879029 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbd1c0b2380 con 0x7fbd2415ffd0 2024-03-24T00:39:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.036+0000 7fbd1aa00640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbd00067690 0x7fbd00069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.036+0000 7fbd1aa00640 1 -- 172.21.15.40:0/1902879029 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd1c11f9d0 con 0x7fbd2415ffd0 2024-03-24T00:39:09.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.036+0000 7fbd21e00640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbd00067690 0x7fbd00069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:09.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.036+0000 7fbd21e00640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbd00067690 0x7fbd00069b50 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbd24156440 tx=0x7fbd14007810 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:09.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.040+0000 7fbd1aa00640 1 -- 172.21.15.40:0/1902879029 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbd1c016110 con 0x7fbd2415ffd0 2024-03-24T00:39:09.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:09.240+0000 7fbd28a00640 1 -- 172.21.15.40:0/1902879029 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbd24157540 con 0x7fbd2415ffd0 2024-03-24T00:39:10.122 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-03-24T00:39:10.116+0000 7fec30800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:39:10.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.168+0000 7fbd1aa00640 1 -- 172.21.15.40:0/1902879029 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v127) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbd1c0aae20 con 0x7fbd2415ffd0 2024-03-24T00:39:10.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.172+0000 7fbd28a00640 1 -- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbd00067690 msgr2=0x7fbd00069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.172+0000 7fbd28a00640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbd00067690 0x7fbd00069b50 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fbd24156440 tx=0x7fbd14007810 comp rx=0 tx=0).stop 2024-03-24T00:39:10.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.172+0000 7fbd28a00640 1 -- 172.21.15.40:0/1902879029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2415ffd0 msgr2=0x7fbd241557b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.172+0000 7fbd28a00640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2415ffd0 0x7fbd241557b0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c0ac040 tx=0x7fbd1c01ad30 comp rx=0 tx=0).stop 2024-03-24T00:39:10.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.172+0000 7fbd28a00640 1 -- 172.21.15.40:0/1902879029 shutdown_connections 2024-03-24T00:39:10.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.172+0000 7fbd28a00640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbd00067690 0x7fbd00069b50 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.172+0000 7fbd28a00640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2415ffd0 0x7fbd241557b0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.172+0000 7fbd28a00640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd2415bd00 0x7fbd2415ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.172+0000 7fbd28a00640 1 --2- 172.21.15.40:0/1902879029 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240728d0 0x7fbd2414d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.172+0000 7fbd28a00640 1 -- 172.21.15.40:0/1902879029 >> 172.21.15.40:0/1902879029 conn(0x7fbd240b1660 msgr2=0x7fbd24157d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:10.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.172+0000 7fbd28a00640 1 -- 172.21.15.40:0/1902879029 shutdown_connections 2024-03-24T00:39:10.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.172+0000 7fbd28a00640 1 -- 172.21.15.40:0/1902879029 wait complete. 2024-03-24T00:39:10.196 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-24T00:39:10.196 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-24T00:39:10.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.336+0000 7f94e0600640 1 Processor -- start 2024-03-24T00:39:10.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.336+0000 7f94e0600640 1 -- start start 2024-03-24T00:39:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.336+0000 7f94e0600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94d80d2270 0x7f94d8070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94e0600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d806d0f0 con 0x7f94d815df80 2024-03-24T00:39:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94e0600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d806b280 con 0x7f94d806c950 2024-03-24T00:39:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94e0600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d806b890 con 0x7f94d80d2270 2024-03-24T00:39:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94dd200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94d80d2270 0x7f94d8070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94dd200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f94d80d2270 0x7f94d8070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49380/0 (socket says 172.21.15.40:49380) 2024-03-24T00:39:10.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94dd200640 1 -- 172.21.15.40:0/2111196420 learned_addr learned my addr 172.21.15.40:0/2111196420 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:10.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94dd200640 1 -- 172.21.15.40:0/2111196420 >> v1:172.21.15.53:6789/0 conn(0x7f94d806c950 legacy=0x7f94d8161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:10.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94dd200640 1 -- 172.21.15.40:0/2111196420 >> v1:172.21.15.40:6789/0 conn(0x7f94d815df80 legacy=0x7f94d80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94dd200640 1 -- 172.21.15.40:0/2111196420 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d806ce10 con 0x7f94d80d2270 2024-03-24T00:39:10.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94dd200640 1 --2- 172.21.15.40:0/2111196420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d80d2270 0x7f94d8070ff0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f94d806ba10 tx=0x7f94c401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89fe882ea3458208 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:10.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94d7e00640 1 -- 172.21.15.40:0/2111196420 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c40aa5f0 con 0x7f94d80d2270 2024-03-24T00:39:10.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94d7e00640 1 -- 172.21.15.40:0/2111196420 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c40af070 con 0x7f94d80d2270 2024-03-24T00:39:10.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.340+0000 7f94d7e00640 1 -- 172.21.15.40:0/2111196420 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c40aa9b0 con 0x7f94d80d2270 2024-03-24T00:39:10.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 -- 172.21.15.40:0/2111196420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d80d2270 msgr2=0x7f94d8070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.347 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 --2- 172.21.15.40:0/2111196420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d80d2270 0x7f94d8070ff0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f94d806ba10 tx=0x7f94c401c4d0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 -- 172.21.15.40:0/2111196420 shutdown_connections 2024-03-24T00:39:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 --2- 172.21.15.40:0/2111196420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d80d2270 0x7f94d8070ff0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 -- 172.21.15.40:0/2111196420 >> 172.21.15.40:0/2111196420 conn(0x7f94d806ecf0 msgr2=0x7f94d806f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 -- 172.21.15.40:0/2111196420 shutdown_connections 2024-03-24T00:39:10.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 -- 172.21.15.40:0/2111196420 wait complete. 2024-03-24T00:39:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 Processor -- start 2024-03-24T00:39:10.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 -- start start 2024-03-24T00:39:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94d806c950 0x7f94d8155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d80d2270 0x7f94d815f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f94d815df80 0x7f94d80c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d816bfa0 con 0x7f94d80d2270 2024-03-24T00:39:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d816be20 con 0x7f94d806c950 2024-03-24T00:39:10.350 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94e0600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d816c120 con 0x7f94d815df80 2024-03-24T00:39:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94dd200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d80d2270 0x7f94d815f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94dd200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d80d2270 0x7f94d815f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49390/0 (socket says 172.21.15.40:49390) 2024-03-24T00:39:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94dd200640 1 -- 172.21.15.40:0/3902022590 learned_addr learned my addr 172.21.15.40:0/3902022590 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94de600640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f94d815df80 0x7f94d80c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94ddc00640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94d806c950 0x7f94d8155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94dd200640 1 -- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f94d815df80 msgr2=0x7f94d80c7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94dd200640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f94d815df80 0x7f94d80c7500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94dd200640 1 -- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94d806c950 msgr2=0x7f94d8155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94dd200640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94d806c950 0x7f94d8155a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.344+0000 7f94dd200640 1 -- 172.21.15.40:0/3902022590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d8157310 con 0x7f94d80d2270 2024-03-24T00:39:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.348+0000 7f94dd200640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d80d2270 0x7f94d815f500 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f94c40ac040 tx=0x7f94c401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.348+0000 7f94d5600640 1 -- 172.21.15.40:0/3902022590 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c40bd5f0 con 0x7f94d80d2270 2024-03-24T00:39:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.348+0000 7f94d5600640 1 -- 172.21.15.40:0/3902022590 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c40bd790 con 0x7f94d80d2270 2024-03-24T00:39:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.348+0000 7f94e0600640 1 -- 172.21.15.40:0/3902022590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d8156270 con 0x7f94d80d2270 2024-03-24T00:39:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.348+0000 7f94d5600640 1 -- 172.21.15.40:0/3902022590 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c40bdb60 con 0x7f94d80d2270 2024-03-24T00:39:10.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.348+0000 7f94e0600640 1 -- 172.21.15.40:0/3902022590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d8156830 con 0x7f94d80d2270 2024-03-24T00:39:10.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.348+0000 7f94e0600640 1 -- 172.21.15.40:0/3902022590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d8068170 con 0x7f94d80d2270 2024-03-24T00:39:10.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.348+0000 7f94d5600640 1 -- 172.21.15.40:0/3902022590 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f94c40b33f0 con 0x7f94d80d2270 2024-03-24T00:39:10.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.348+0000 7f94d5600640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f94b8067660 0x7f94b8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:10.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.348+0000 7f94d5600640 1 -- 172.21.15.40:0/3902022590 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94c411fed0 con 0x7f94d80d2270 2024-03-24T00:39:10.354 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.348+0000 7f94ddc00640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f94b8067660 0x7f94b8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:10.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.352+0000 7f94ddc00640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f94b8067660 0x7f94b8069b20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f94cc001770 tx=0x7f94cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:10.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.360+0000 7f94d5600640 1 -- 172.21.15.40:0/3902022590 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f94c40c3060 con 0x7f94d80d2270 2024-03-24T00:39:10.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:10.556+0000 7f94e0600640 1 -- 172.21.15.40:0/3902022590 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f94d80cf9d0 con 0x7f94d80d2270 2024-03-24T00:39:11.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.168+0000 7f94d5600640 1 -- 172.21.15.40:0/3902022590 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v128) v1 ==== 122+0+0 (secure 0 0 0) 0x7f94c40160f0 con 0x7f94d80d2270 2024-03-24T00:39:11.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.172+0000 7f94e0600640 1 -- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f94b8067660 msgr2=0x7f94b8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.172+0000 7f94e0600640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f94b8067660 0x7f94b8069b20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f94cc001770 tx=0x7f94cc009040 comp rx=0 tx=0).stop 2024-03-24T00:39:11.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.172+0000 7f94e0600640 1 -- 172.21.15.40:0/3902022590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d80d2270 msgr2=0x7f94d815f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.172+0000 7f94e0600640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d80d2270 0x7f94d815f500 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f94c40ac040 tx=0x7f94c401ad10 comp rx=0 tx=0).stop 2024-03-24T00:39:11.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.172+0000 7f94e0600640 1 -- 172.21.15.40:0/3902022590 shutdown_connections 2024-03-24T00:39:11.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.172+0000 7f94e0600640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f94b8067660 0x7f94b8069b20 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.172+0000 7f94e0600640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f94d815df80 0x7f94d80c7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.172+0000 7f94e0600640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f94d80d2270 0x7f94d815f500 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.172+0000 7f94e0600640 1 --2- 172.21.15.40:0/3902022590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94d806c950 0x7f94d8155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.172+0000 7f94e0600640 1 -- 172.21.15.40:0/3902022590 >> 172.21.15.40:0/3902022590 conn(0x7f94d806ecf0 msgr2=0x7f94d80b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.172+0000 7f94e0600640 1 -- 172.21.15.40:0/3902022590 shutdown_connections 2024-03-24T00:39:11.178 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.172+0000 7f94e0600640 1 -- 172.21.15.40:0/3902022590 wait complete. 2024-03-24T00:39:11.205 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-24T00:39:11.205 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:11.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a7dc00640 1 Processor -- start 2024-03-24T00:39:11.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a7dc00640 1 -- start start 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a7dc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a7813ec80 0x7f8a781414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a7dc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8a780ce800 0x7f8a78071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a7dc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a78070e00 con 0x7f8a781419f0 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a7dc00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a78141e50 con 0x7f8a7813ec80 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a7dc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a780cee20 con 0x7f8a780ce800 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a76a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a7813ec80 0x7f8a781414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a76a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8a7813ec80 0x7f8a781414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49398/0 (socket says 172.21.15.40:49398) 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a76a00640 1 -- 172.21.15.40:0/516371458 learned_addr learned my addr 172.21.15.40:0/516371458 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a77400640 1 --2- 172.21.15.40:0/516371458 >> v2:172.21.15.53:3300/0 conn(0x7f8a780ce800 0x7f8a78071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a76a00640 1 -- 172.21.15.40:0/516371458 >> v2:172.21.15.53:3300/0 conn(0x7f8a780ce800 msgr2=0x7f8a78071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:11.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a76a00640 1 --2- 172.21.15.40:0/516371458 >> v2:172.21.15.53:3300/0 conn(0x7f8a780ce800 0x7f8a78071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a76a00640 1 -- 172.21.15.40:0/516371458 >> v1:172.21.15.53:6789/0 conn(0x7f8a781419f0 legacy=0x7f8a780ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.384+0000 7f8a76a00640 1 -- 172.21.15.40:0/516371458 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a780cf100 con 0x7f8a7813ec80 2024-03-24T00:39:11.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.388+0000 7f8a76a00640 1 --2- 172.21.15.40:0/516371458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7813ec80 0x7f8a781414b0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f8a78067e60 tx=0x7f8a6801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27cc5e78156b62cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.388+0000 7f8a76000640 1 -- 172.21.15.40:0/516371458 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a680a9400 con 0x7f8a7813ec80 2024-03-24T00:39:11.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.388+0000 7f8a76000640 1 -- 172.21.15.40:0/516371458 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a680a95a0 con 0x7f8a7813ec80 2024-03-24T00:39:11.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.388+0000 7f8a76000640 1 -- 172.21.15.40:0/516371458 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a680a9970 con 0x7f8a7813ec80 2024-03-24T00:39:11.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.388+0000 7f8a7dc00640 1 -- 172.21.15.40:0/516371458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7813ec80 msgr2=0x7f8a781414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.388+0000 7f8a7dc00640 1 --2- 172.21.15.40:0/516371458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7813ec80 0x7f8a781414b0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f8a78067e60 tx=0x7f8a6801c520 comp rx=0 tx=0).stop 2024-03-24T00:39:11.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.388+0000 7f8a7dc00640 1 -- 172.21.15.40:0/516371458 shutdown_connections 2024-03-24T00:39:11.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.388+0000 7f8a7dc00640 1 --2- 172.21.15.40:0/516371458 >> v2:172.21.15.53:3300/0 conn(0x7f8a780ce800 0x7f8a78071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.388+0000 7f8a7dc00640 1 --2- 172.21.15.40:0/516371458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7813ec80 0x7f8a781414b0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.388+0000 7f8a7dc00640 1 -- 172.21.15.40:0/516371458 >> 172.21.15.40:0/516371458 conn(0x7f8a780b17b0 msgr2=0x7f8a780b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.388+0000 7f8a7dc00640 1 -- 172.21.15.40:0/516371458 shutdown_connections 2024-03-24T00:39:11.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.388+0000 7f8a7dc00640 1 -- 172.21.15.40:0/516371458 wait complete. 2024-03-24T00:39:11.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a7dc00640 1 Processor -- start 2024-03-24T00:39:11.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a7dc00640 1 -- start start 2024-03-24T00:39:11.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a7dc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a780ce800 0x7f8a7814e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a7dc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7813ec80 0x7f8a781500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a7dc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a781419f0 0x7f8a7813ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a7dc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780723e0 con 0x7f8a7813ec80 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a7dc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a78072260 con 0x7f8a780ce800 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a7dc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a78072560 con 0x7f8a781419f0 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a6fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7813ec80 0x7f8a781500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a76a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a780ce800 0x7f8a7814e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a76a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a780ce800 0x7f8a7814e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:47026/0 (socket says 172.21.15.40:47026) 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a76a00640 1 -- 172.21.15.40:0/2922246648 learned_addr learned my addr 172.21.15.40:0/2922246648 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:11.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a77400640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a781419f0 0x7f8a7813ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a77400640 1 -- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a780ce800 msgr2=0x7f8a7814e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a77400640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a780ce800 0x7f8a7814e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a77400640 1 -- 172.21.15.40:0/2922246648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7813ec80 msgr2=0x7f8a781500d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a77400640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7813ec80 0x7f8a781500d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a77400640 1 -- 172.21.15.40:0/2922246648 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a78157c50 con 0x7f8a781419f0 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a76a00640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a780ce800 0x7f8a7814e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a77400640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a781419f0 0x7f8a7813ccf0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000eba0 tx=0x7f8a7000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a6ea00640 1 -- 172.21.15.40:0/2922246648 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70018020 con 0x7f8a781419f0 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a6ea00640 1 -- 172.21.15.40:0/2922246648 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7000cd20 con 0x7f8a781419f0 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a6ea00640 1 -- 172.21.15.40:0/2922246648 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a700146c0 con 0x7f8a781419f0 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a7dc00640 1 -- 172.21.15.40:0/2922246648 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a78157f30 con 0x7f8a781419f0 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.392+0000 7f8a7dc00640 1 -- 172.21.15.40:0/2922246648 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a781584f0 con 0x7f8a781419f0 2024-03-24T00:39:11.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.396+0000 7f8a7dc00640 1 -- 172.21.15.40:0/2922246648 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a78068170 con 0x7f8a781419f0 2024-03-24T00:39:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.396+0000 7f8a6ea00640 1 -- 172.21.15.40:0/2922246648 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8a7001b020 con 0x7f8a781419f0 2024-03-24T00:39:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.396+0000 7f8a6ea00640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a54067660 0x7f8a54069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.396+0000 7f8a6ea00640 1 -- 172.21.15.40:0/2922246648 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a700864f0 con 0x7f8a781419f0 2024-03-24T00:39:11.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.396+0000 7f8a76a00640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a54067660 0x7f8a54069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.396+0000 7f8a76a00640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a54067660 0x7f8a54069b20 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8a68000c10 tx=0x7f8a680b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.406 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.400+0000 7f8a6ea00640 1 -- 172.21.15.40:0/2922246648 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8a70022030 con 0x7f8a781419f0 2024-03-24T00:39:11.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.632+0000 7f8a7dc00640 1 -- 172.21.15.40:0/2922246648 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a7806e690 con 0x7f8a781419f0 2024-03-24T00:39:11.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.632+0000 7f8a6ea00640 1 -- 172.21.15.40:0/2922246648 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+4982 (secure 0 0 0) 0x7f8a70010040 con 0x7f8a781419f0 2024-03-24T00:39:11.637 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:11.637 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":128,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1582567544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1582567544},{"type":"v1","addr":"172.21.15.53:6837","nonce":1582567544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8563,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2255755838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2255755838},{"type":"v1","addr":"172.21.15.53:6836","nonce":2255755838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8874,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4273998586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4273998586},{"type":"v1","addr":"172.21.15.40:6833","nonce":4273998586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:39:07.531884+0000","modified":"2024-03-24T00:39:10.561787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8877},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8877":{"gid":8877,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/773071485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":773071485},{"type":"v1","addr":"172.21.15.40:6835","nonce":773071485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8877,"qdb_cluster":[8877]},"id":9}]} 2024-03-24T00:39:11.637 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 128 2024-03-24T00:39:11.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.636+0000 7f8a7dc00640 1 -- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a54067660 msgr2=0x7f8a54069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.636+0000 7f8a7dc00640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a54067660 0x7f8a54069b20 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8a68000c10 tx=0x7f8a680b3040 comp rx=0 tx=0).stop 2024-03-24T00:39:11.640 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.636+0000 7f8a7dc00640 1 -- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a781419f0 msgr2=0x7f8a7813ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.636+0000 7f8a7dc00640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a781419f0 0x7f8a7813ccf0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000eba0 tx=0x7f8a7000c490 comp rx=0 tx=0).stop 2024-03-24T00:39:11.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.636+0000 7f8a7dc00640 1 -- 172.21.15.40:0/2922246648 shutdown_connections 2024-03-24T00:39:11.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.636+0000 7f8a7dc00640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8a54067660 0x7f8a54069b20 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.636+0000 7f8a7dc00640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a781419f0 0x7f8a7813ccf0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.636+0000 7f8a7dc00640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8a7813ec80 0x7f8a781500d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.636+0000 7f8a7dc00640 1 --2- 172.21.15.40:0/2922246648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a780ce800 0x7f8a7814e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.636+0000 7f8a7dc00640 1 -- 172.21.15.40:0/2922246648 >> 172.21.15.40:0/2922246648 conn(0x7f8a780b17b0 msgr2=0x7f8a7806e8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.636+0000 7f8a7dc00640 1 -- 172.21.15.40:0/2922246648 shutdown_connections 2024-03-24T00:39:11.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.636+0000 7f8a7dc00640 1 -- 172.21.15.40:0/2922246648 wait complete. 2024-03-24T00:39:11.668 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:39:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.804+0000 7facfaa00640 1 Processor -- start 2024-03-24T00:39:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.804+0000 7facfaa00640 1 -- start start 2024-03-24T00:39:11.809 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.804+0000 7facfaa00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7facf4141bc0 0x7facf40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.804+0000 7facfaa00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf4070e00 con 0x7facf40ce630 2024-03-24T00:39:11.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.804+0000 7facfaa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf40cee20 con 0x7facf413ec80 2024-03-24T00:39:11.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.804+0000 7facfaa00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf40cf100 con 0x7facf4141bc0 2024-03-24T00:39:11.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.804+0000 7facebe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7facf4141bc0 0x7facf40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.804+0000 7facebe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7facf4141bc0 0x7facf40ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49422/0 (socket says 172.21.15.40:49422) 2024-03-24T00:39:11.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.804+0000 7facebe00640 1 -- 172.21.15.40:0/339290951 learned_addr learned my addr 172.21.15.40:0/339290951 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:11.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.804+0000 7facebe00640 1 -- 172.21.15.40:0/339290951 >> v1:172.21.15.53:6789/0 conn(0x7facf413ec80 legacy=0x7facf41414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.804+0000 7facebe00640 1 -- 172.21.15.40:0/339290951 >> v1:172.21.15.40:6789/0 conn(0x7facf40ce630 legacy=0x7facf4071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.804+0000 7facebe00640 1 -- 172.21.15.40:0/339290951 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf40cf3e0 con 0x7facf4141bc0 2024-03-24T00:39:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.808+0000 7facebe00640 1 --2- 172.21.15.40:0/339290951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf4141bc0 0x7facf40ce0f0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7facdc009870 tx=0x7facdc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0eb7afdc4049c27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.808+0000 7facf3400640 1 -- 172.21.15.40:0/339290951 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc0a9640 con 0x7facf4141bc0 2024-03-24T00:39:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.808+0000 7facf3400640 1 -- 172.21.15.40:0/339290951 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc0a97e0 con 0x7facf4141bc0 2024-03-24T00:39:11.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.808+0000 7facf3400640 1 -- 172.21.15.40:0/339290951 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc0a9bb0 con 0x7facf4141bc0 2024-03-24T00:39:11.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.808+0000 7facfaa00640 1 -- 172.21.15.40:0/339290951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf4141bc0 msgr2=0x7facf40ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.808+0000 7facfaa00640 1 --2- 172.21.15.40:0/339290951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf4141bc0 0x7facf40ce0f0 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7facdc009870 tx=0x7facdc01c560 comp rx=0 tx=0).stop 2024-03-24T00:39:11.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.808+0000 7facfaa00640 1 -- 172.21.15.40:0/339290951 shutdown_connections 2024-03-24T00:39:11.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.808+0000 7facfaa00640 1 --2- 172.21.15.40:0/339290951 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf4141bc0 0x7facf40ce0f0 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.808+0000 7facfaa00640 1 -- 172.21.15.40:0/339290951 >> 172.21.15.40:0/339290951 conn(0x7facf40b17b0 msgr2=0x7facf40b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.808+0000 7facfaa00640 1 -- 172.21.15.40:0/339290951 shutdown_connections 2024-03-24T00:39:11.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.808+0000 7facfaa00640 1 -- 172.21.15.40:0/339290951 wait complete. 2024-03-24T00:39:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facfaa00640 1 Processor -- start 2024-03-24T00:39:11.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facfaa00640 1 -- start start 2024-03-24T00:39:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facfaa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facf40ce630 0x7facf413dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facfaa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facf413ec80 0x7facf4155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facfaa00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf4141bc0 0x7facf413c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facfaa00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf40725f0 con 0x7facf4141bc0 2024-03-24T00:39:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facfaa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf4072470 con 0x7facf40ce630 2024-03-24T00:39:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facfaa00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf4169d50 con 0x7facf413ec80 2024-03-24T00:39:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facf8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf4141bc0 0x7facf413c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facf3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facf40ce630 0x7facf413dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facf8a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf4141bc0 0x7facf413c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49436/0 (socket says 172.21.15.40:49436) 2024-03-24T00:39:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facf8a00640 1 -- 172.21.15.40:0/3275653538 learned_addr learned my addr 172.21.15.40:0/3275653538 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:11.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.812+0000 7facebe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facf413ec80 0x7facf4155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facf8a00640 1 -- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facf413ec80 msgr2=0x7facf4155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facf8a00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facf413ec80 0x7facf4155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facf8a00640 1 -- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facf40ce630 msgr2=0x7facf413dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facf8a00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facf40ce630 0x7facf413dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facf8a00640 1 -- 172.21.15.40:0/3275653538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf41561c0 con 0x7facf4141bc0 2024-03-24T00:39:11.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facf3e00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facf40ce630 0x7facf413dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:11.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facf8a00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf4141bc0 0x7facf413c040 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7facec00c730 tx=0x7facec00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facf0c00640 1 -- 172.21.15.40:0/3275653538 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec003b80 con 0x7facf4141bc0 2024-03-24T00:39:11.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facf0c00640 1 -- 172.21.15.40:0/3275653538 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facec003d20 con 0x7facf4141bc0 2024-03-24T00:39:11.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facf0c00640 1 -- 172.21.15.40:0/3275653538 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facec00aa30 con 0x7facf4141bc0 2024-03-24T00:39:11.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facfaa00640 1 -- 172.21.15.40:0/3275653538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf413ca20 con 0x7facf4141bc0 2024-03-24T00:39:11.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facfaa00640 1 -- 172.21.15.40:0/3275653538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf413cfe0 con 0x7facf4141bc0 2024-03-24T00:39:11.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facfaa00640 1 -- 172.21.15.40:0/3275653538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf4068170 con 0x7facf4141bc0 2024-03-24T00:39:11.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.816+0000 7facf0c00640 1 -- 172.21.15.40:0/3275653538 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7facec00abd0 con 0x7facf4141bc0 2024-03-24T00:39:11.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.820+0000 7facf0c00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7facd0067660 0x7facd0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:11.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.820+0000 7facf0c00640 1 -- 172.21.15.40:0/3275653538 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7facec085660 con 0x7facf4141bc0 2024-03-24T00:39:11.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.820+0000 7facf3e00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7facd0067660 0x7facd0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:11.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.820+0000 7facf3e00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7facd0067660 0x7facd0069b20 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7face4007fd0 tx=0x7face4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:11.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.824+0000 7facf0c00640 1 -- 172.21.15.40:0/3275653538 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7facec055bf0 con 0x7facf4141bc0 2024-03-24T00:39:11.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.984+0000 7facfaa00640 1 -- 172.21.15.40:0/3275653538 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facf4072b90 con 0x7facf4141bc0 2024-03-24T00:39:11.992 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facf0c00640 1 -- 172.21.15.40:0/3275653538 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v119) v1 ==== 74+0+16619 (secure 0 0 0) 0x7facec059c90 con 0x7facf4141bc0 2024-03-24T00:39:11.992 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:11.992 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":119,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:39:09.156343+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:05.964092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:06.525558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:39:11.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facfaa00640 1 -- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7facd0067660 msgr2=0x7facd0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facfaa00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7facd0067660 0x7facd0069b20 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7face4007fd0 tx=0x7face4009040 comp rx=0 tx=0).stop 2024-03-24T00:39:11.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facfaa00640 1 -- 172.21.15.40:0/3275653538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf4141bc0 msgr2=0x7facf413c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:11.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facfaa00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf4141bc0 0x7facf413c040 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7facec00c730 tx=0x7facec00cbf0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facfaa00640 1 -- 172.21.15.40:0/3275653538 shutdown_connections 2024-03-24T00:39:11.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facfaa00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7facd0067660 0x7facd0069b20 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facfaa00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7facf4141bc0 0x7facf413c040 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facfaa00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7facf413ec80 0x7facf4155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facfaa00640 1 --2- 172.21.15.40:0/3275653538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facf40ce630 0x7facf413dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:11.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facfaa00640 1 -- 172.21.15.40:0/3275653538 >> 172.21.15.40:0/3275653538 conn(0x7facf40b17b0 msgr2=0x7facf406cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:11.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facfaa00640 1 -- 172.21.15.40:0/3275653538 shutdown_connections 2024-03-24T00:39:11.995 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:11.988+0000 7facfaa00640 1 -- 172.21.15.40:0/3275653538 wait complete. 2024-03-24T00:39:12.014 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:39:12.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:39:12.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca3e00640 1 Processor -- start 2024-03-24T00:39:12.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca3e00640 1 -- start start 2024-03-24T00:39:12.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc9c145760 0x7fbc9c06f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c072750 0x7fbc9c06e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbc9c06ea50 0x7fbc9c0cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca3e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9c0cd630 con 0x7fbc9c145760 2024-03-24T00:39:12.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca3e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9c0cd910 con 0x7fbc9c06ea50 2024-03-24T00:39:12.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca3e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0cdbf0 con 0x7fbc9c072750 2024-03-24T00:39:12.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc9c145760 0x7fbc9c06f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca1400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbc9c145760 0x7fbc9c06f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49444/0 (socket says 172.21.15.40:49444) 2024-03-24T00:39:12.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca1400640 1 -- 172.21.15.40:0/3527845369 learned_addr learned my addr 172.21.15.40:0/3527845369 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca1400640 1 -- 172.21.15.40:0/3527845369 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c072750 msgr2=0x7fbc9c06e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca0a00640 1 --2- 172.21.15.40:0/3527845369 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c072750 0x7fbc9c06e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca1e00640 1 --2- 172.21.15.40:0/3527845369 >> v2:172.21.15.53:3300/0 conn(0x7fbc9c06ea50 0x7fbc9c0cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca1400640 1 --2- 172.21.15.40:0/3527845369 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c072750 0x7fbc9c06e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca1400640 1 -- 172.21.15.40:0/3527845369 >> v2:172.21.15.53:3300/0 conn(0x7fbc9c06ea50 msgr2=0x7fbc9c0cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca1400640 1 --2- 172.21.15.40:0/3527845369 >> v2:172.21.15.53:3300/0 conn(0x7fbc9c06ea50 0x7fbc9c0cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca1400640 1 -- 172.21.15.40:0/3527845369 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c06dd10 con 0x7fbc9c145760 2024-03-24T00:39:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca0a00640 1 --2- 172.21.15.40:0/3527845369 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c072750 0x7fbc9c06e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.152+0000 7fbca1e00640 1 --2- 172.21.15.40:0/3527845369 >> v2:172.21.15.53:3300/0 conn(0x7fbc9c06ea50 0x7fbc9c0cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:39:12.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbca1400640 1 --2- 172.21.15.40:0/3527845369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc9c145760 0x7fbc9c06f580 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0cea60 tx=0x7fbc9001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e43350443c21a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbc9be00640 1 -- 172.21.15.40:0/3527845369 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc900b0070 con 0x7fbc9c145760 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbc9be00640 1 -- 172.21.15.40:0/3527845369 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc900ab600 con 0x7fbc9c145760 2024-03-24T00:39:12.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbc9be00640 1 -- 172.21.15.40:0/3527845369 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc900ab9d0 con 0x7fbc9c145760 2024-03-24T00:39:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbca3e00640 1 -- 172.21.15.40:0/3527845369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc9c145760 msgr2=0x7fbc9c06f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbca3e00640 1 --2- 172.21.15.40:0/3527845369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc9c145760 0x7fbc9c06f580 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0cea60 tx=0x7fbc9001c520 comp rx=0 tx=0).stop 2024-03-24T00:39:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbca3e00640 1 -- 172.21.15.40:0/3527845369 shutdown_connections 2024-03-24T00:39:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbca3e00640 1 --2- 172.21.15.40:0/3527845369 >> v2:172.21.15.53:3300/0 conn(0x7fbc9c06ea50 0x7fbc9c0cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbca3e00640 1 --2- 172.21.15.40:0/3527845369 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c072750 0x7fbc9c06e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbca3e00640 1 --2- 172.21.15.40:0/3527845369 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc9c145760 0x7fbc9c06f580 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbca3e00640 1 -- 172.21.15.40:0/3527845369 >> 172.21.15.40:0/3527845369 conn(0x7fbc9c0d0c20 msgr2=0x7fbc9c0d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbca3e00640 1 -- 172.21.15.40:0/3527845369 shutdown_connections 2024-03-24T00:39:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.156+0000 7fbca3e00640 1 -- 172.21.15.40:0/3527845369 wait complete. 2024-03-24T00:39:12.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca3e00640 1 Processor -- start 2024-03-24T00:39:12.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca3e00640 1 -- start start 2024-03-24T00:39:12.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc9c06ea50 0x7fbc9c163e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc9c072750 0x7fbc9c163230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c145760 0x7fbc9c15af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca3e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c167a50 con 0x7fbc9c06ea50 2024-03-24T00:39:12.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c167bd0 con 0x7fbc9c072750 2024-03-24T00:39:12.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca3e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c167d50 con 0x7fbc9c145760 2024-03-24T00:39:12.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc9c06ea50 0x7fbc9c163e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca1400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc9c06ea50 0x7fbc9c163e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49454/0 (socket says 172.21.15.40:49454) 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca1400640 1 -- 172.21.15.40:0/828590125 learned_addr learned my addr 172.21.15.40:0/828590125 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca0a00640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc9c072750 0x7fbc9c163230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca1e00640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c145760 0x7fbc9c15af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca1400640 1 -- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c145760 msgr2=0x7fbc9c15af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca1400640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c145760 0x7fbc9c15af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca1400640 1 -- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc9c072750 msgr2=0x7fbc9c163230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca1400640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc9c072750 0x7fbc9c163230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca1400640 1 -- 172.21.15.40:0/828590125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c162190 con 0x7fbc9c06ea50 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca0a00640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc9c072750 0x7fbc9c163230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca1400640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc9c06ea50 0x7fbc9c163e90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fbc900ae040 tx=0x7fbc9001ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbc99600640 1 -- 172.21.15.40:0/828590125 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc900b0040 con 0x7fbc9c06ea50 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbc99600640 1 -- 172.21.15.40:0/828590125 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc900bc070 con 0x7fbc9c06ea50 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca3e00640 1 -- 172.21.15.40:0/828590125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9c162470 con 0x7fbc9c06ea50 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbc99600640 1 -- 172.21.15.40:0/828590125 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc900b6880 con 0x7fbc9c06ea50 2024-03-24T00:39:12.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.160+0000 7fbca3e00640 1 -- 172.21.15.40:0/828590125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9c162a30 con 0x7fbc9c06ea50 2024-03-24T00:39:12.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.164+0000 7fbca3e00640 1 -- 172.21.15.40:0/828590125 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9c068170 con 0x7fbc9c06ea50 2024-03-24T00:39:12.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.164+0000 7fbc99600640 1 -- 172.21.15.40:0/828590125 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbc900b6a20 con 0x7fbc9c06ea50 2024-03-24T00:39:12.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.164+0000 7fbc99600640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc7c067660 0x7fbc7c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.164+0000 7fbc99600640 1 -- 172.21.15.40:0/828590125 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc9011fd80 con 0x7fbc9c06ea50 2024-03-24T00:39:12.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.164+0000 7fbca0a00640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc7c067660 0x7fbc7c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.164+0000 7fbca0a00640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc7c067660 0x7fbc7c069b20 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0cd550 tx=0x7fbc88008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:12.173 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.168+0000 7fbc99600640 1 -- 172.21.15.40:0/828590125 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbc90016110 con 0x7fbc9c06ea50 2024-03-24T00:39:12.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.332+0000 7fbca3e00640 1 -- 172.21.15.40:0/828590125 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbc9c0b25f0 con 0x7fbc7c067660 2024-03-24T00:39:12.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.336+0000 7fbc99600640 1 -- 172.21.15.40:0/828590125 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252411 (secure 0 0 0) 0x7fbc9c0b25f0 con 0x7fbc7c067660 2024-03-24T00:39:12.345 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:12.346 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:39:12.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.340+0000 7fbca3e00640 1 -- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc7c067660 msgr2=0x7fbc7c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.340+0000 7fbca3e00640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc7c067660 0x7fbc7c069b20 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c0cd550 tx=0x7fbc88008040 comp rx=0 tx=0).stop 2024-03-24T00:39:12.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.340+0000 7fbca3e00640 1 -- 172.21.15.40:0/828590125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc9c06ea50 msgr2=0x7fbc9c163e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.340+0000 7fbca3e00640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc9c06ea50 0x7fbc9c163e90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fbc900ae040 tx=0x7fbc9001ad60 comp rx=0 tx=0).stop 2024-03-24T00:39:12.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.340+0000 7fbca3e00640 1 -- 172.21.15.40:0/828590125 shutdown_connections 2024-03-24T00:39:12.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.340+0000 7fbca3e00640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbc7c067660 0x7fbc7c069b20 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.340+0000 7fbca3e00640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c145760 0x7fbc9c15af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.340+0000 7fbca3e00640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc9c072750 0x7fbc9c163230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.340+0000 7fbca3e00640 1 --2- 172.21.15.40:0/828590125 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbc9c06ea50 0x7fbc9c163e90 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.348 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.340+0000 7fbca3e00640 1 -- 172.21.15.40:0/828590125 >> 172.21.15.40:0/828590125 conn(0x7fbc9c0d0c20 msgr2=0x7fbc9c143e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.340+0000 7fbca3e00640 1 -- 172.21.15.40:0/828590125 shutdown_connections 2024-03-24T00:39:12.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.340+0000 7fbca3e00640 1 -- 172.21.15.40:0/828590125 wait complete. 2024-03-24T00:39:12.366 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-03-24T00:39:11.294553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":371732,"kb_used_data":12624,"kb_used_omap":12,"kb_used_meta":358963,"kb_avail":754602988,"statfs":{"total":773094113280,"available":772713459712,"internally_reserved":0,"allocated":12926976,"data_stored":6304440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":367578674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.166237"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185562+0000","last_change":"2024-03-24T00:39:08.182796+0000","last_active":"2024-03-24T00:39:08.185562+0000","last_peered":"2024-03-24T00:39:08.185562+0000","last_clean":"2024-03-24T00:39:08.185562+0000","last_became_active":"2024-03-24T00:39:08.180585+0000","last_became_peered":"2024-03-24T00:39:08.180585+0000","last_unstale":"2024-03-24T00:39:08.185562+0000","last_undegraded":"2024-03-24T00:39:08.185562+0000","last_fullsized":"2024-03-24T00:39:08.185562+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:52:02.548249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.161569+0000","last_change":"2024-03-24T00:39:08.158422+0000","last_active":"2024-03-24T00:39:08.161569+0000","last_peered":"2024-03-24T00:39:08.161569+0000","last_clean":"2024-03-24T00:39:08.161569+0000","last_became_active":"2024-03-24T00:39:08.157737+0000","last_became_peered":"2024-03-24T00:39:08.157737+0000","last_unstale":"2024-03-24T00:39:08.161569+0000","last_undegraded":"2024-03-24T00:39:08.161569+0000","last_fullsized":"2024-03-24T00:39:08.161569+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:14:22.215663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185843+0000","last_change":"2024-03-24T00:39:08.185222+0000","last_active":"2024-03-24T00:39:08.185843+0000","last_peered":"2024-03-24T00:39:08.185843+0000","last_clean":"2024-03-24T00:39:08.185843+0000","last_became_active":"2024-03-24T00:39:08.184830+0000","last_became_peered":"2024-03-24T00:39:08.184830+0000","last_unstale":"2024-03-24T00:39:08.185843+0000","last_undegraded":"2024-03-24T00:39:08.185843+0000","last_fullsized":"2024-03-24T00:39:08.185843+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:24:26.554733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.178897+0000","last_change":"2024-03-24T00:39:08.172805+0000","last_active":"2024-03-24T00:39:08.178897+0000","last_peered":"2024-03-24T00:39:08.178897+0000","last_clean":"2024-03-24T00:39:08.178897+0000","last_became_active":"2024-03-24T00:39:08.172522+0000","last_became_peered":"2024-03-24T00:39:08.172522+0000","last_unstale":"2024-03-24T00:39:08.178897+0000","last_undegraded":"2024-03-24T00:39:08.178897+0000","last_fullsized":"2024-03-24T00:39:08.178897+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:40:39.588732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.167376+0000","last_change":"2024-03-24T00:39:08.162870+0000","last_active":"2024-03-24T00:39:08.167376+0000","last_peered":"2024-03-24T00:39:08.167376+0000","last_clean":"2024-03-24T00:39:08.167376+0000","last_became_active":"2024-03-24T00:39:08.162521+0000","last_became_peered":"2024-03-24T00:39:08.162521+0000","last_unstale":"2024-03-24T00:39:08.167376+0000","last_undegraded":"2024-03-24T00:39:08.167376+0000","last_fullsized":"2024-03-24T00:39:08.167376+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:44.048508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.171001+0000","last_change":"2024-03-24T00:39:08.165635+0000","last_active":"2024-03-24T00:39:08.171001+0000","last_peered":"2024-03-24T00:39:08.171001+0000","last_clean":"2024-03-24T00:39:08.171001+0000","last_became_active":"2024-03-24T00:39:08.165348+0000","last_became_peered":"2024-03-24T00:39:08.165348+0000","last_unstale":"2024-03-24T00:39:08.171001+0000","last_undegraded":"2024-03-24T00:39:08.171001+0000","last_fullsized":"2024-03-24T00:39:08.171001+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:24:26.554733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.173040+0000","last_change":"2024-03-24T00:39:08.171795+0000","last_active":"2024-03-24T00:39:08.173040+0000","last_peered":"2024-03-24T00:39:08.173040+0000","last_clean":"2024-03-24T00:39:08.173040+0000","last_became_active":"2024-03-24T00:39:08.171367+0000","last_became_peered":"2024-03-24T00:39:08.171367+0000","last_unstale":"2024-03-24T00:39:08.173040+0000","last_undegraded":"2024-03-24T00:39:08.173040+0000","last_fullsized":"2024-03-24T00:39:08.173040+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:47.552389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.183501+0000","last_change":"2024-03-24T00:39:08.182999+0000","last_active":"2024-03-24T00:39:08.183501+0000","last_peered":"2024-03-24T00:39:08.183501+0000","last_clean":"2024-03-24T00:39:08.183501+0000","last_became_active":"2024-03-24T00:39:08.182682+0000","last_became_peered":"2024-03-24T00:39:08.182682+0000","last_unstale":"2024-03-24T00:39:08.183501+0000","last_undegraded":"2024-03-24T00:39:08.183501+0000","last_fullsized":"2024-03-24T00:39:08.183501+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:32.441579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181131+0000","last_change":"2024-03-24T00:39:08.177992+0000","last_active":"2024-03-24T00:39:08.181131+0000","last_peered":"2024-03-24T00:39:08.181131+0000","last_clean":"2024-03-24T00:39:08.181131+0000","last_became_active":"2024-03-24T00:39:08.177669+0000","last_became_peered":"2024-03-24T00:39:08.177669+0000","last_unstale":"2024-03-24T00:39:08.181131+0000","last_undegraded":"2024-03-24T00:39:08.181131+0000","last_fullsized":"2024-03-24T00:39:08.181131+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:30.569403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.161584+0000","last_change":"2024-03-24T00:39:08.158652+0000","last_active":"2024-03-24T00:39:08.161584+0000","last_peered":"2024-03-24T00:39:08.161584+0000","last_clean":"2024-03-24T00:39:08.161584+0000","last_became_active":"2024-03-24T00:39:08.158143+0000","last_became_peered":"2024-03-24T00:39:08.158143+0000","last_unstale":"2024-03-24T00:39:08.161584+0000","last_undegraded":"2024-03-24T00:39:08.161584+0000","last_fullsized":"2024-03-24T00:39:08.161584+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:07:32.874502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.182289+0000","last_change":"2024-03-24T00:39:08.167873+0000","last_active":"2024-03-24T00:39:08.182289+0000","last_peered":"2024-03-24T00:39:08.182289+0000","last_clean":"2024-03-24T00:39:08.182289+0000","last_became_active":"2024-03-24T00:39:08.167395+0000","last_became_peered":"2024-03-24T00:39:08.167395+0000","last_unstale":"2024-03-24T00:39:08.182289+0000","last_undegraded":"2024-03-24T00:39:08.182289+0000","last_fullsized":"2024-03-24T00:39:08.182289+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:25.182467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180708+0000","last_change":"2024-03-24T00:39:08.171429+0000","last_active":"2024-03-24T00:39:08.180708+0000","last_peered":"2024-03-24T00:39:08.180708+0000","last_clean":"2024-03-24T00:39:08.180708+0000","last_became_active":"2024-03-24T00:39:08.171104+0000","last_became_peered":"2024-03-24T00:39:08.171104+0000","last_unstale":"2024-03-24T00:39:08.180708+0000","last_undegraded":"2024-03-24T00:39:08.180708+0000","last_fullsized":"2024-03-24T00:39:08.180708+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:29:12.671642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185064+0000","last_change":"2024-03-24T00:39:08.182082+0000","last_active":"2024-03-24T00:39:08.185064+0000","last_peered":"2024-03-24T00:39:08.185064+0000","last_clean":"2024-03-24T00:39:08.185064+0000","last_became_active":"2024-03-24T00:39:08.178314+0000","last_became_peered":"2024-03-24T00:39:08.178314+0000","last_unstale":"2024-03-24T00:39:08.185064+0000","last_undegraded":"2024-03-24T00:39:08.185064+0000","last_fullsized":"2024-03-24T00:39:08.185064+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:42:57.233295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.169020+0000","last_change":"2024-03-24T00:39:08.163534+0000","last_active":"2024-03-24T00:39:08.169020+0000","last_peered":"2024-03-24T00:39:08.169020+0000","last_clean":"2024-03-24T00:39:08.169020+0000","last_became_active":"2024-03-24T00:39:08.163245+0000","last_became_peered":"2024-03-24T00:39:08.163245+0000","last_unstale":"2024-03-24T00:39:08.169020+0000","last_undegraded":"2024-03-24T00:39:08.169020+0000","last_fullsized":"2024-03-24T00:39:08.169020+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:08:50.321160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.177306+0000","last_change":"2024-03-24T00:39:08.168038+0000","last_active":"2024-03-24T00:39:08.177306+0000","last_peered":"2024-03-24T00:39:08.177306+0000","last_clean":"2024-03-24T00:39:08.177306+0000","last_became_active":"2024-03-24T00:39:08.167698+0000","last_became_peered":"2024-03-24T00:39:08.167698+0000","last_unstale":"2024-03-24T00:39:08.177306+0000","last_undegraded":"2024-03-24T00:39:08.177306+0000","last_fullsized":"2024-03-24T00:39:08.177306+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:33:35.230906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.188236+0000","last_change":"2024-03-24T00:39:08.187521+0000","last_active":"2024-03-24T00:39:08.188236+0000","last_peered":"2024-03-24T00:39:08.188236+0000","last_clean":"2024-03-24T00:39:08.188236+0000","last_became_active":"2024-03-24T00:39:08.187105+0000","last_became_peered":"2024-03-24T00:39:08.187105+0000","last_unstale":"2024-03-24T00:39:08.188236+0000","last_undegraded":"2024-03-24T00:39:08.188236+0000","last_fullsized":"2024-03-24T00:39:08.188236+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:10.676965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180077+0000","last_change":"2024-03-24T00:39:08.179170+0000","last_active":"2024-03-24T00:39:08.180077+0000","last_peered":"2024-03-24T00:39:08.180077+0000","last_clean":"2024-03-24T00:39:08.180077+0000","last_became_active":"2024-03-24T00:39:08.178886+0000","last_became_peered":"2024-03-24T00:39:08.178886+0000","last_unstale":"2024-03-24T00:39:08.180077+0000","last_undegraded":"2024-03-24T00:39:08.180077+0000","last_fullsized":"2024-03-24T00:39:08.180077+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:18:26.080188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.183982+0000","last_change":"2024-03-24T00:39:08.174096+0000","last_active":"2024-03-24T00:39:08.183982+0000","last_peered":"2024-03-24T00:39:08.183982+0000","last_clean":"2024-03-24T00:39:08.183982+0000","last_became_active":"2024-03-24T00:39:08.173814+0000","last_became_peered":"2024-03-24T00:39:08.173814+0000","last_unstale":"2024-03-24T00:39:08.183982+0000","last_undegraded":"2024-03-24T00:39:08.183982+0000","last_fullsized":"2024-03-24T00:39:08.183982+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:27.318468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.182313+0000","last_change":"2024-03-24T00:39:08.169315+0000","last_active":"2024-03-24T00:39:08.182313+0000","last_peered":"2024-03-24T00:39:08.182313+0000","last_clean":"2024-03-24T00:39:08.182313+0000","last_became_active":"2024-03-24T00:39:08.167088+0000","last_became_peered":"2024-03-24T00:39:08.167088+0000","last_unstale":"2024-03-24T00:39:08.182313+0000","last_undegraded":"2024-03-24T00:39:08.182313+0000","last_fullsized":"2024-03-24T00:39:08.182313+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:23.563817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.184762+0000","last_change":"2024-03-24T00:39:08.179567+0000","last_active":"2024-03-24T00:39:08.184762+0000","last_peered":"2024-03-24T00:39:08.184762+0000","last_clean":"2024-03-24T00:39:08.184762+0000","last_became_active":"2024-03-24T00:39:08.177697+0000","last_became_peered":"2024-03-24T00:39:08.177697+0000","last_unstale":"2024-03-24T00:39:08.184762+0000","last_undegraded":"2024-03-24T00:39:08.184762+0000","last_fullsized":"2024-03-24T00:39:08.184762+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:44.048508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.167352+0000","last_change":"2024-03-24T00:39:08.161960+0000","last_active":"2024-03-24T00:39:08.167352+0000","last_peered":"2024-03-24T00:39:08.167352+0000","last_clean":"2024-03-24T00:39:08.167352+0000","last_became_active":"2024-03-24T00:39:08.161618+0000","last_became_peered":"2024-03-24T00:39:08.161618+0000","last_unstale":"2024-03-24T00:39:08.167352+0000","last_undegraded":"2024-03-24T00:39:08.167352+0000","last_fullsized":"2024-03-24T00:39:08.167352+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:27.318468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181278+0000","last_change":"2024-03-24T00:39:08.165672+0000","last_active":"2024-03-24T00:39:08.181278+0000","last_peered":"2024-03-24T00:39:08.181278+0000","last_clean":"2024-03-24T00:39:08.181278+0000","last_became_active":"2024-03-24T00:39:08.165398+0000","last_became_peered":"2024-03-24T00:39:08.165398+0000","last_unstale":"2024-03-24T00:39:08.181278+0000","last_undegraded":"2024-03-24T00:39:08.181278+0000","last_fullsized":"2024-03-24T00:39:08.181278+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:35:18.867104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.172652+0000","last_change":"2024-03-24T00:39:08.167415+0000","last_active":"2024-03-24T00:39:08.172652+0000","last_peered":"2024-03-24T00:39:08.172652+0000","last_clean":"2024-03-24T00:39:08.172652+0000","last_became_active":"2024-03-24T00:39:08.164889+0000","last_became_peered":"2024-03-24T00:39:08.164889+0000","last_unstale":"2024-03-24T00:39:08.172652+0000","last_undegraded":"2024-03-24T00:39:08.172652+0000","last_fullsized":"2024-03-24T00:39:08.172652+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:23.192198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181531+0000","last_change":"2024-03-24T00:39:08.166803+0000","last_active":"2024-03-24T00:39:08.181531+0000","last_peered":"2024-03-24T00:39:08.181531+0000","last_clean":"2024-03-24T00:39:08.181531+0000","last_became_active":"2024-03-24T00:39:08.166544+0000","last_became_peered":"2024-03-24T00:39:08.166544+0000","last_unstale":"2024-03-24T00:39:08.181531+0000","last_undegraded":"2024-03-24T00:39:08.181531+0000","last_fullsized":"2024-03-24T00:39:08.181531+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:26.950051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.169044+0000","last_change":"2024-03-24T00:39:08.165503+0000","last_active":"2024-03-24T00:39:08.169044+0000","last_peered":"2024-03-24T00:39:08.169044+0000","last_clean":"2024-03-24T00:39:08.169044+0000","last_became_active":"2024-03-24T00:39:08.163567+0000","last_became_peered":"2024-03-24T00:39:08.163567+0000","last_unstale":"2024-03-24T00:39:08.169044+0000","last_undegraded":"2024-03-24T00:39:08.169044+0000","last_fullsized":"2024-03-24T00:39:08.169044+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:22.784502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.186281+0000","last_change":"2024-03-24T00:39:08.180186+0000","last_active":"2024-03-24T00:39:08.186281+0000","last_peered":"2024-03-24T00:39:08.186281+0000","last_clean":"2024-03-24T00:39:08.186281+0000","last_became_active":"2024-03-24T00:39:08.179892+0000","last_became_peered":"2024-03-24T00:39:08.179892+0000","last_unstale":"2024-03-24T00:39:08.186281+0000","last_undegraded":"2024-03-24T00:39:08.186281+0000","last_fullsized":"2024-03-24T00:39:08.186281+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:02.900425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.184575+0000","last_change":"2024-03-24T00:39:08.181623+0000","last_active":"2024-03-24T00:39:08.184575+0000","last_peered":"2024-03-24T00:39:08.184575+0000","last_clean":"2024-03-24T00:39:08.184575+0000","last_became_active":"2024-03-24T00:39:08.178412+0000","last_became_peered":"2024-03-24T00:39:08.178412+0000","last_unstale":"2024-03-24T00:39:08.184575+0000","last_undegraded":"2024-03-24T00:39:08.184575+0000","last_fullsized":"2024-03-24T00:39:08.184575+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:21.681285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181063+0000","last_change":"2024-03-24T00:39:08.175175+0000","last_active":"2024-03-24T00:39:08.181063+0000","last_peered":"2024-03-24T00:39:08.181063+0000","last_clean":"2024-03-24T00:39:08.181063+0000","last_became_active":"2024-03-24T00:39:08.174872+0000","last_became_peered":"2024-03-24T00:39:08.174872+0000","last_unstale":"2024-03-24T00:39:08.181063+0000","last_undegraded":"2024-03-24T00:39:08.181063+0000","last_fullsized":"2024-03-24T00:39:08.181063+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:08:50.321160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.175837+0000","last_change":"2024-03-24T00:39:08.167248+0000","last_active":"2024-03-24T00:39:08.175837+0000","last_peered":"2024-03-24T00:39:08.175837+0000","last_clean":"2024-03-24T00:39:08.175837+0000","last_became_active":"2024-03-24T00:39:08.166946+0000","last_became_peered":"2024-03-24T00:39:08.166946+0000","last_unstale":"2024-03-24T00:39:08.175837+0000","last_undegraded":"2024-03-24T00:39:08.175837+0000","last_fullsized":"2024-03-24T00:39:08.175837+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:58.267823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.175804+0000","last_change":"2024-03-24T00:39:08.168189+0000","last_active":"2024-03-24T00:39:08.175804+0000","last_peered":"2024-03-24T00:39:08.175804+0000","last_clean":"2024-03-24T00:39:08.175804+0000","last_became_active":"2024-03-24T00:39:08.167869+0000","last_became_peered":"2024-03-24T00:39:08.167869+0000","last_unstale":"2024-03-24T00:39:08.175804+0000","last_undegraded":"2024-03-24T00:39:08.175804+0000","last_fullsized":"2024-03-24T00:39:08.175804+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:29.091630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.174275+0000","last_change":"2024-03-24T00:39:08.167080+0000","last_active":"2024-03-24T00:39:08.174275+0000","last_peered":"2024-03-24T00:39:08.174275+0000","last_clean":"2024-03-24T00:39:08.174275+0000","last_became_active":"2024-03-24T00:39:08.166373+0000","last_became_peered":"2024-03-24T00:39:08.166373+0000","last_unstale":"2024-03-24T00:39:08.174275+0000","last_undegraded":"2024-03-24T00:39:08.174275+0000","last_fullsized":"2024-03-24T00:39:08.174275+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:02.900425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181742+0000","last_change":"2024-03-24T00:39:08.180345+0000","last_active":"2024-03-24T00:39:08.181742+0000","last_peered":"2024-03-24T00:39:08.181742+0000","last_clean":"2024-03-24T00:39:08.181742+0000","last_became_active":"2024-03-24T00:39:08.180006+0000","last_became_peered":"2024-03-24T00:39:08.180006+0000","last_unstale":"2024-03-24T00:39:08.181742+0000","last_undegraded":"2024-03-24T00:39:08.181742+0000","last_fullsized":"2024-03-24T00:39:08.181742+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:10:31.283701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.187491+0000","last_change":"2024-03-24T00:39:08.186098+0000","last_active":"2024-03-24T00:39:08.187491+0000","last_peered":"2024-03-24T00:39:08.187491+0000","last_clean":"2024-03-24T00:39:08.187491+0000","last_became_active":"2024-03-24T00:39:08.185015+0000","last_became_peered":"2024-03-24T00:39:08.185015+0000","last_unstale":"2024-03-24T00:39:08.187491+0000","last_undegraded":"2024-03-24T00:39:08.187491+0000","last_fullsized":"2024-03-24T00:39:08.187491+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:59.444846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"118'4","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.188314+0000","last_change":"2024-03-24T00:39:07.156116+0000","last_active":"2024-03-24T00:39:08.188314+0000","last_peered":"2024-03-24T00:39:08.188314+0000","last_clean":"2024-03-24T00:39:08.188314+0000","last_became_active":"2024-03-24T00:39:07.155762+0000","last_became_peered":"2024-03-24T00:39:07.155762+0000","last_unstale":"2024-03-24T00:39:08.188314+0000","last_undegraded":"2024-03-24T00:39:08.188314+0000","last_fullsized":"2024-03-24T00:39:08.188314+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:41.871203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180699+0000","last_change":"2024-03-24T00:39:08.168733+0000","last_active":"2024-03-24T00:39:08.180699+0000","last_peered":"2024-03-24T00:39:08.180699+0000","last_clean":"2024-03-24T00:39:08.180699+0000","last_became_active":"2024-03-24T00:39:08.168372+0000","last_became_peered":"2024-03-24T00:39:08.168372+0000","last_unstale":"2024-03-24T00:39:08.180699+0000","last_undegraded":"2024-03-24T00:39:08.180699+0000","last_fullsized":"2024-03-24T00:39:08.180699+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:18.615058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"118'1","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185372+0000","last_change":"2024-03-24T00:39:07.156929+0000","last_active":"2024-03-24T00:39:08.185372+0000","last_peered":"2024-03-24T00:39:08.185372+0000","last_clean":"2024-03-24T00:39:08.185372+0000","last_became_active":"2024-03-24T00:39:07.156587+0000","last_became_peered":"2024-03-24T00:39:07.156587+0000","last_unstale":"2024-03-24T00:39:08.185372+0000","last_undegraded":"2024-03-24T00:39:08.185372+0000","last_fullsized":"2024-03-24T00:39:08.185372+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:14:21.207962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.163982+0000","last_change":"2024-03-24T00:39:08.160028+0000","last_active":"2024-03-24T00:39:08.163982+0000","last_peered":"2024-03-24T00:39:08.163982+0000","last_clean":"2024-03-24T00:39:08.163982+0000","last_became_active":"2024-03-24T00:39:08.159640+0000","last_became_peered":"2024-03-24T00:39:08.159640+0000","last_unstale":"2024-03-24T00:39:08.163982+0000","last_undegraded":"2024-03-24T00:39:08.163982+0000","last_fullsized":"2024-03-24T00:39:08.163982+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:17:18.299416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"118'2","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185964+0000","last_change":"2024-03-24T00:39:07.158050+0000","last_active":"2024-03-24T00:39:08.185964+0000","last_peered":"2024-03-24T00:39:08.185964+0000","last_clean":"2024-03-24T00:39:08.185964+0000","last_became_active":"2024-03-24T00:39:07.157789+0000","last_became_peered":"2024-03-24T00:39:07.157789+0000","last_unstale":"2024-03-24T00:39:08.185964+0000","last_undegraded":"2024-03-24T00:39:08.185964+0000","last_fullsized":"2024-03-24T00:39:08.185964+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:43:14.768323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.164970+0000","last_change":"2024-03-24T00:39:08.159013+0000","last_active":"2024-03-24T00:39:08.164970+0000","last_peered":"2024-03-24T00:39:08.164970+0000","last_clean":"2024-03-24T00:39:08.164970+0000","last_became_active":"2024-03-24T00:39:08.158609+0000","last_became_peered":"2024-03-24T00:39:08.158609+0000","last_unstale":"2024-03-24T00:39:08.164970+0000","last_undegraded":"2024-03-24T00:39:08.164970+0000","last_fullsized":"2024-03-24T00:39:08.164970+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:34:56.731869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"118'2","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.187532+0000","last_change":"2024-03-24T00:39:07.155423+0000","last_active":"2024-03-24T00:39:08.187532+0000","last_peered":"2024-03-24T00:39:08.187532+0000","last_clean":"2024-03-24T00:39:08.187532+0000","last_became_active":"2024-03-24T00:39:07.155065+0000","last_became_peered":"2024-03-24T00:39:07.155065+0000","last_unstale":"2024-03-24T00:39:08.187532+0000","last_undegraded":"2024-03-24T00:39:08.187532+0000","last_fullsized":"2024-03-24T00:39:08.187532+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:24.117504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181429+0000","last_change":"2024-03-24T00:39:08.174021+0000","last_active":"2024-03-24T00:39:08.181429+0000","last_peered":"2024-03-24T00:39:08.181429+0000","last_clean":"2024-03-24T00:39:08.181429+0000","last_became_active":"2024-03-24T00:39:08.173708+0000","last_became_peered":"2024-03-24T00:39:08.173708+0000","last_unstale":"2024-03-24T00:39:08.181429+0000","last_undegraded":"2024-03-24T00:39:08.181429+0000","last_fullsized":"2024-03-24T00:39:08.181429+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:17:18.299416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"118'1","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185477+0000","last_change":"2024-03-24T00:39:07.157052+0000","last_active":"2024-03-24T00:39:08.185477+0000","last_peered":"2024-03-24T00:39:08.185477+0000","last_clean":"2024-03-24T00:39:08.185477+0000","last_became_active":"2024-03-24T00:39:07.156750+0000","last_became_peered":"2024-03-24T00:39:07.156750+0000","last_unstale":"2024-03-24T00:39:08.185477+0000","last_undegraded":"2024-03-24T00:39:08.185477+0000","last_fullsized":"2024-03-24T00:39:08.185477+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:26.722631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.176816+0000","last_change":"2024-03-24T00:39:08.175384+0000","last_active":"2024-03-24T00:39:08.176816+0000","last_peered":"2024-03-24T00:39:08.176816+0000","last_clean":"2024-03-24T00:39:08.176816+0000","last_became_active":"2024-03-24T00:39:08.174162+0000","last_became_peered":"2024-03-24T00:39:08.174162+0000","last_unstale":"2024-03-24T00:39:08.176816+0000","last_undegraded":"2024-03-24T00:39:08.176816+0000","last_fullsized":"2024-03-24T00:39:08.176816+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:23.123726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"118'1","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.182300+0000","last_change":"2024-03-24T00:39:07.159939+0000","last_active":"2024-03-24T00:39:08.182300+0000","last_peered":"2024-03-24T00:39:08.182300+0000","last_clean":"2024-03-24T00:39:08.182300+0000","last_became_active":"2024-03-24T00:39:07.159408+0000","last_became_peered":"2024-03-24T00:39:07.159408+0000","last_unstale":"2024-03-24T00:39:08.182300+0000","last_undegraded":"2024-03-24T00:39:08.182300+0000","last_fullsized":"2024-03-24T00:39:08.182300+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:05:35.119226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181701+0000","last_change":"2024-03-24T00:39:08.179606+0000","last_active":"2024-03-24T00:39:08.181701+0000","last_peered":"2024-03-24T00:39:08.181701+0000","last_clean":"2024-03-24T00:39:08.181701+0000","last_became_active":"2024-03-24T00:39:08.179268+0000","last_became_peered":"2024-03-24T00:39:08.179268+0000","last_unstale":"2024-03-24T00:39:08.181701+0000","last_undegraded":"2024-03-24T00:39:08.181701+0000","last_fullsized":"2024-03-24T00:39:08.181701+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:24:08.067590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.148959+0000","last_change":"2024-03-24T00:39:07.152655+0000","last_active":"2024-03-24T00:39:08.148959+0000","last_peered":"2024-03-24T00:39:08.148959+0000","last_clean":"2024-03-24T00:39:08.148959+0000","last_became_active":"2024-03-24T00:39:07.152323+0000","last_became_peered":"2024-03-24T00:39:07.152323+0000","last_unstale":"2024-03-24T00:39:08.148959+0000","last_undegraded":"2024-03-24T00:39:08.148959+0000","last_fullsized":"2024-03-24T00:39:08.148959+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:29.561702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.186639+0000","last_change":"2024-03-24T00:39:08.183774+0000","last_active":"2024-03-24T00:39:08.186639+0000","last_peered":"2024-03-24T00:39:08.186639+0000","last_clean":"2024-03-24T00:39:08.186639+0000","last_became_active":"2024-03-24T00:39:08.183104+0000","last_became_peered":"2024-03-24T00:39:08.183104+0000","last_unstale":"2024-03-24T00:39:08.186639+0000","last_undegraded":"2024-03-24T00:39:08.186639+0000","last_fullsized":"2024-03-24T00:39:08.186639+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:10:37.340378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.148220+0000","last_change":"2024-03-24T00:39:07.150904+0000","last_active":"2024-03-24T00:39:08.148220+0000","last_peered":"2024-03-24T00:39:08.148220+0000","last_clean":"2024-03-24T00:39:08.148220+0000","last_became_active":"2024-03-24T00:39:07.150623+0000","last_became_peered":"2024-03-24T00:39:07.150623+0000","last_unstale":"2024-03-24T00:39:08.148220+0000","last_undegraded":"2024-03-24T00:39:08.148220+0000","last_fullsized":"2024-03-24T00:39:08.148220+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:10.960934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.182700+0000","last_change":"2024-03-24T00:39:08.177210+0000","last_active":"2024-03-24T00:39:08.182700+0000","last_peered":"2024-03-24T00:39:08.182700+0000","last_clean":"2024-03-24T00:39:08.182700+0000","last_became_active":"2024-03-24T00:39:08.176905+0000","last_became_peered":"2024-03-24T00:39:08.176905+0000","last_unstale":"2024-03-24T00:39:08.182700+0000","last_undegraded":"2024-03-24T00:39:08.182700+0000","last_fullsized":"2024-03-24T00:39:08.182700+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:44.874403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.155581+0000","last_change":"2024-03-24T00:39:07.146610+0000","last_active":"2024-03-24T00:39:08.155581+0000","last_peered":"2024-03-24T00:39:08.155581+0000","last_clean":"2024-03-24T00:39:08.155581+0000","last_became_active":"2024-03-24T00:39:07.146284+0000","last_became_peered":"2024-03-24T00:39:07.146284+0000","last_unstale":"2024-03-24T00:39:08.155581+0000","last_undegraded":"2024-03-24T00:39:08.155581+0000","last_fullsized":"2024-03-24T00:39:08.155581+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:24.117504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181323+0000","last_change":"2024-03-24T00:39:08.177972+0000","last_active":"2024-03-24T00:39:08.181323+0000","last_peered":"2024-03-24T00:39:08.181323+0000","last_clean":"2024-03-24T00:39:08.181323+0000","last_became_active":"2024-03-24T00:39:08.177097+0000","last_became_peered":"2024-03-24T00:39:08.177097+0000","last_unstale":"2024-03-24T00:39:08.181323+0000","last_undegraded":"2024-03-24T00:39:08.181323+0000","last_fullsized":"2024-03-24T00:39:08.181323+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:06:54.840723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.174852+0000","last_change":"2024-03-24T00:39:07.148239+0000","last_active":"2024-03-24T00:39:08.174852+0000","last_peered":"2024-03-24T00:39:08.174852+0000","last_clean":"2024-03-24T00:39:08.174852+0000","last_became_active":"2024-03-24T00:39:07.147917+0000","last_became_peered":"2024-03-24T00:39:07.147917+0000","last_unstale":"2024-03-24T00:39:08.174852+0000","last_undegraded":"2024-03-24T00:39:08.174852+0000","last_fullsized":"2024-03-24T00:39:08.174852+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:10.814422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.176942+0000","last_change":"2024-03-24T00:39:08.175413+0000","last_active":"2024-03-24T00:39:08.176942+0000","last_peered":"2024-03-24T00:39:08.176942+0000","last_clean":"2024-03-24T00:39:08.176942+0000","last_became_active":"2024-03-24T00:39:08.175078+0000","last_became_peered":"2024-03-24T00:39:08.175078+0000","last_unstale":"2024-03-24T00:39:08.176942+0000","last_undegraded":"2024-03-24T00:39:08.176942+0000","last_fullsized":"2024-03-24T00:39:08.176942+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:10.676965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"118'3","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185125+0000","last_change":"2024-03-24T00:39:07.153900+0000","last_active":"2024-03-24T00:39:08.185125+0000","last_peered":"2024-03-24T00:39:08.185125+0000","last_clean":"2024-03-24T00:39:08.185125+0000","last_became_active":"2024-03-24T00:39:07.153598+0000","last_became_peered":"2024-03-24T00:39:07.153598+0000","last_unstale":"2024-03-24T00:39:08.185125+0000","last_undegraded":"2024-03-24T00:39:08.185125+0000","last_fullsized":"2024-03-24T00:39:08.185125+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:57:39.605387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.174795+0000","last_change":"2024-03-24T00:39:08.168213+0000","last_active":"2024-03-24T00:39:08.174795+0000","last_peered":"2024-03-24T00:39:08.174795+0000","last_clean":"2024-03-24T00:39:08.174795+0000","last_became_active":"2024-03-24T00:39:08.167110+0000","last_became_peered":"2024-03-24T00:39:08.167110+0000","last_unstale":"2024-03-24T00:39:08.174795+0000","last_undegraded":"2024-03-24T00:39:08.174795+0000","last_fullsized":"2024-03-24T00:39:08.174795+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:31.763862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"118'6","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.189928+0000","last_change":"2024-03-24T00:39:07.148077+0000","last_active":"2024-03-24T00:39:08.189928+0000","last_peered":"2024-03-24T00:39:08.189928+0000","last_clean":"2024-03-24T00:39:08.189928+0000","last_became_active":"2024-03-24T00:39:07.146873+0000","last_became_peered":"2024-03-24T00:39:07.146873+0000","last_unstale":"2024-03-24T00:39:08.189928+0000","last_undegraded":"2024-03-24T00:39:08.189928+0000","last_fullsized":"2024-03-24T00:39:08.189928+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:21.776801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.186026+0000","last_change":"2024-03-24T00:39:08.183414+0000","last_active":"2024-03-24T00:39:08.186026+0000","last_peered":"2024-03-24T00:39:08.186026+0000","last_clean":"2024-03-24T00:39:08.186026+0000","last_became_active":"2024-03-24T00:39:08.183148+0000","last_became_peered":"2024-03-24T00:39:08.183148+0000","last_unstale":"2024-03-24T00:39:08.186026+0000","last_undegraded":"2024-03-24T00:39:08.186026+0000","last_fullsized":"2024-03-24T00:39:08.186026+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:42.456654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.159973+0000","last_change":"2024-03-24T00:39:07.151514+0000","last_active":"2024-03-24T00:39:08.159973+0000","last_peered":"2024-03-24T00:39:08.159973+0000","last_clean":"2024-03-24T00:39:08.159973+0000","last_became_active":"2024-03-24T00:39:07.151173+0000","last_became_peered":"2024-03-24T00:39:07.151173+0000","last_unstale":"2024-03-24T00:39:08.159973+0000","last_undegraded":"2024-03-24T00:39:08.159973+0000","last_fullsized":"2024-03-24T00:39:08.159973+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:48:58.379103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.178945+0000","last_change":"2024-03-24T00:39:08.164686+0000","last_active":"2024-03-24T00:39:08.178945+0000","last_peered":"2024-03-24T00:39:08.178945+0000","last_clean":"2024-03-24T00:39:08.178945+0000","last_became_active":"2024-03-24T00:39:08.164223+0000","last_became_peered":"2024-03-24T00:39:08.164223+0000","last_unstale":"2024-03-24T00:39:08.178945+0000","last_undegraded":"2024-03-24T00:39:08.178945+0000","last_fullsized":"2024-03-24T00:39:08.178945+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:06:54.840723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.148305+0000","last_change":"2024-03-24T00:39:07.145520+0000","last_active":"2024-03-24T00:39:08.148305+0000","last_peered":"2024-03-24T00:39:08.148305+0000","last_clean":"2024-03-24T00:39:08.148305+0000","last_became_active":"2024-03-24T00:39:07.145189+0000","last_became_peered":"2024-03-24T00:39:07.145189+0000","last_unstale":"2024-03-24T00:39:08.148305+0000","last_undegraded":"2024-03-24T00:39:08.148305+0000","last_fullsized":"2024-03-24T00:39:08.148305+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:38.387067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180002+0000","last_change":"2024-03-24T00:39:08.167748+0000","last_active":"2024-03-24T00:39:08.180002+0000","last_peered":"2024-03-24T00:39:08.180002+0000","last_clean":"2024-03-24T00:39:08.180002+0000","last_became_active":"2024-03-24T00:39:08.167375+0000","last_became_peered":"2024-03-24T00:39:08.167375+0000","last_unstale":"2024-03-24T00:39:08.180002+0000","last_undegraded":"2024-03-24T00:39:08.180002+0000","last_fullsized":"2024-03-24T00:39:08.180002+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:10:37.340378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"118'8","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.188562+0000","last_change":"2024-03-24T00:39:07.153365+0000","last_active":"2024-03-24T00:39:08.188562+0000","last_peered":"2024-03-24T00:39:08.188562+0000","last_clean":"2024-03-24T00:39:08.188562+0000","last_became_active":"2024-03-24T00:39:07.153043+0000","last_became_peered":"2024-03-24T00:39:07.153043+0000","last_unstale":"2024-03-24T00:39:08.188562+0000","last_undegraded":"2024-03-24T00:39:08.188562+0000","last_fullsized":"2024-03-24T00:39:08.188562+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:44:31.582035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180524+0000","last_change":"2024-03-24T00:39:08.168562+0000","last_active":"2024-03-24T00:39:08.180524+0000","last_peered":"2024-03-24T00:39:08.180524+0000","last_clean":"2024-03-24T00:39:08.180524+0000","last_became_active":"2024-03-24T00:39:08.168284+0000","last_became_peered":"2024-03-24T00:39:08.168284+0000","last_unstale":"2024-03-24T00:39:08.180524+0000","last_undegraded":"2024-03-24T00:39:08.180524+0000","last_fullsized":"2024-03-24T00:39:08.180524+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:48:22.697641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"118'1","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181236+0000","last_change":"2024-03-24T00:39:07.146935+0000","last_active":"2024-03-24T00:39:08.181236+0000","last_peered":"2024-03-24T00:39:08.181236+0000","last_clean":"2024-03-24T00:39:08.181236+0000","last_became_active":"2024-03-24T00:39:07.146613+0000","last_became_peered":"2024-03-24T00:39:07.146613+0000","last_unstale":"2024-03-24T00:39:08.181236+0000","last_undegraded":"2024-03-24T00:39:08.181236+0000","last_fullsized":"2024-03-24T00:39:08.181236+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:26.722631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185691+0000","last_change":"2024-03-24T00:39:08.184448+0000","last_active":"2024-03-24T00:39:08.185691+0000","last_peered":"2024-03-24T00:39:08.185691+0000","last_clean":"2024-03-24T00:39:08.185691+0000","last_became_active":"2024-03-24T00:39:08.184143+0000","last_became_peered":"2024-03-24T00:39:08.184143+0000","last_unstale":"2024-03-24T00:39:08.185691+0000","last_undegraded":"2024-03-24T00:39:08.185691+0000","last_fullsized":"2024-03-24T00:39:08.185691+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:03.669831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"118'2","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.182070+0000","last_change":"2024-03-24T00:39:07.156381+0000","last_active":"2024-03-24T00:39:08.182070+0000","last_peered":"2024-03-24T00:39:08.182070+0000","last_clean":"2024-03-24T00:39:08.182070+0000","last_became_active":"2024-03-24T00:39:07.155397+0000","last_became_peered":"2024-03-24T00:39:07.155397+0000","last_unstale":"2024-03-24T00:39:08.182070+0000","last_undegraded":"2024-03-24T00:39:08.182070+0000","last_fullsized":"2024-03-24T00:39:08.182070+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:03:03.981709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.187185+0000","last_change":"2024-03-24T00:39:08.184614+0000","last_active":"2024-03-24T00:39:08.187185+0000","last_peered":"2024-03-24T00:39:08.187185+0000","last_clean":"2024-03-24T00:39:08.187185+0000","last_became_active":"2024-03-24T00:39:08.182535+0000","last_became_peered":"2024-03-24T00:39:08.182535+0000","last_unstale":"2024-03-24T00:39:08.187185+0000","last_undegraded":"2024-03-24T00:39:08.187185+0000","last_fullsized":"2024-03-24T00:39:08.187185+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:31.763862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"118'1","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.184774+0000","last_change":"2024-03-24T00:39:07.159788+0000","last_active":"2024-03-24T00:39:08.184774+0000","last_peered":"2024-03-24T00:39:08.184774+0000","last_clean":"2024-03-24T00:39:08.184774+0000","last_became_active":"2024-03-24T00:39:07.159434+0000","last_became_peered":"2024-03-24T00:39:07.159434+0000","last_unstale":"2024-03-24T00:39:08.184774+0000","last_undegraded":"2024-03-24T00:39:08.184774+0000","last_fullsized":"2024-03-24T00:39:08.184774+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:24:07.059890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180699+0000","last_change":"2024-03-24T00:39:08.175017+0000","last_active":"2024-03-24T00:39:08.180699+0000","last_peered":"2024-03-24T00:39:08.180699+0000","last_clean":"2024-03-24T00:39:08.180699+0000","last_became_active":"2024-03-24T00:39:08.174730+0000","last_became_peered":"2024-03-24T00:39:08.174730+0000","last_unstale":"2024-03-24T00:39:08.180699+0000","last_undegraded":"2024-03-24T00:39:08.180699+0000","last_fullsized":"2024-03-24T00:39:08.180699+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:44:32.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.158259+0000","last_change":"2024-03-24T00:39:07.156738+0000","last_active":"2024-03-24T00:39:08.158259+0000","last_peered":"2024-03-24T00:39:08.158259+0000","last_clean":"2024-03-24T00:39:08.158259+0000","last_became_active":"2024-03-24T00:39:07.156411+0000","last_became_peered":"2024-03-24T00:39:07.156411+0000","last_unstale":"2024-03-24T00:39:08.158259+0000","last_undegraded":"2024-03-24T00:39:08.158259+0000","last_fullsized":"2024-03-24T00:39:08.158259+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:07:31.866801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.183352+0000","last_change":"2024-03-24T00:39:08.180333+0000","last_active":"2024-03-24T00:39:08.183352+0000","last_peered":"2024-03-24T00:39:08.183352+0000","last_clean":"2024-03-24T00:39:08.183352+0000","last_became_active":"2024-03-24T00:39:08.179894+0000","last_became_peered":"2024-03-24T00:39:08.179894+0000","last_unstale":"2024-03-24T00:39:08.183352+0000","last_undegraded":"2024-03-24T00:39:08.183352+0000","last_fullsized":"2024-03-24T00:39:08.183352+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:16:26.647757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"118'2","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.187073+0000","last_change":"2024-03-24T00:39:07.147319+0000","last_active":"2024-03-24T00:39:08.187073+0000","last_peered":"2024-03-24T00:39:08.187073+0000","last_clean":"2024-03-24T00:39:08.187073+0000","last_became_active":"2024-03-24T00:39:07.144597+0000","last_became_peered":"2024-03-24T00:39:07.144597+0000","last_unstale":"2024-03-24T00:39:08.187073+0000","last_undegraded":"2024-03-24T00:39:08.187073+0000","last_fullsized":"2024-03-24T00:39:08.187073+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:50.557232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.171774+0000","last_change":"2024-03-24T00:39:08.169263+0000","last_active":"2024-03-24T00:39:08.171774+0000","last_peered":"2024-03-24T00:39:08.171774+0000","last_clean":"2024-03-24T00:39:08.171774+0000","last_became_active":"2024-03-24T00:39:08.167859+0000","last_became_peered":"2024-03-24T00:39:08.167859+0000","last_unstale":"2024-03-24T00:39:08.171774+0000","last_undegraded":"2024-03-24T00:39:08.171774+0000","last_fullsized":"2024-03-24T00:39:08.171774+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:17:47.898414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.171595+0000","last_change":"2024-03-24T00:39:07.152044+0000","last_active":"2024-03-24T00:39:08.171595+0000","last_peered":"2024-03-24T00:39:08.171595+0000","last_clean":"2024-03-24T00:39:08.171595+0000","last_became_active":"2024-03-24T00:39:07.151431+0000","last_became_peered":"2024-03-24T00:39:07.151431+0000","last_unstale":"2024-03-24T00:39:08.171595+0000","last_undegraded":"2024-03-24T00:39:08.171595+0000","last_fullsized":"2024-03-24T00:39:08.171595+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:34:55.724168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180992+0000","last_change":"2024-03-24T00:39:08.178791+0000","last_active":"2024-03-24T00:39:08.180992+0000","last_peered":"2024-03-24T00:39:08.180992+0000","last_clean":"2024-03-24T00:39:08.180992+0000","last_became_active":"2024-03-24T00:39:08.171766+0000","last_became_peered":"2024-03-24T00:39:08.171766+0000","last_unstale":"2024-03-24T00:39:08.180992+0000","last_undegraded":"2024-03-24T00:39:08.180992+0000","last_fullsized":"2024-03-24T00:39:08.180992+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:23.192198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.171287+0000","last_change":"2024-03-24T00:39:07.154626+0000","last_active":"2024-03-24T00:39:08.171287+0000","last_peered":"2024-03-24T00:39:08.171287+0000","last_clean":"2024-03-24T00:39:08.171287+0000","last_became_active":"2024-03-24T00:39:07.154281+0000","last_became_peered":"2024-03-24T00:39:07.154281+0000","last_unstale":"2024-03-24T00:39:08.171287+0000","last_undegraded":"2024-03-24T00:39:08.171287+0000","last_fullsized":"2024-03-24T00:39:08.171287+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:43.804017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.174851+0000","last_change":"2024-03-24T00:39:08.167313+0000","last_active":"2024-03-24T00:39:08.174851+0000","last_peered":"2024-03-24T00:39:08.174851+0000","last_clean":"2024-03-24T00:39:08.174851+0000","last_became_active":"2024-03-24T00:39:08.166747+0000","last_became_peered":"2024-03-24T00:39:08.166747+0000","last_unstale":"2024-03-24T00:39:08.174851+0000","last_undegraded":"2024-03-24T00:39:08.174851+0000","last_fullsized":"2024-03-24T00:39:08.174851+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:39:55.017873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.153101+0000","last_change":"2024-03-24T00:39:07.156122+0000","last_active":"2024-03-24T00:39:08.153101+0000","last_peered":"2024-03-24T00:39:08.153101+0000","last_clean":"2024-03-24T00:39:08.153101+0000","last_became_active":"2024-03-24T00:39:07.155822+0000","last_became_peered":"2024-03-24T00:39:07.155822+0000","last_unstale":"2024-03-24T00:39:08.153101+0000","last_undegraded":"2024-03-24T00:39:08.153101+0000","last_fullsized":"2024-03-24T00:39:08.153101+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:23:37.082015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.173420+0000","last_change":"2024-03-24T00:39:08.165338+0000","last_active":"2024-03-24T00:39:08.173420+0000","last_peered":"2024-03-24T00:39:08.173420+0000","last_clean":"2024-03-24T00:39:08.173420+0000","last_became_active":"2024-03-24T00:39:08.165042+0000","last_became_peered":"2024-03-24T00:39:08.165042+0000","last_unstale":"2024-03-24T00:39:08.173420+0000","last_undegraded":"2024-03-24T00:39:08.173420+0000","last_fullsized":"2024-03-24T00:39:08.173420+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:32.441579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"118'2","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181855+0000","last_change":"2024-03-24T00:39:07.149981+0000","last_active":"2024-03-24T00:39:08.181855+0000","last_peered":"2024-03-24T00:39:08.181855+0000","last_clean":"2024-03-24T00:39:08.181855+0000","last_became_active":"2024-03-24T00:39:07.149712+0000","last_became_peered":"2024-03-24T00:39:07.149712+0000","last_unstale":"2024-03-24T00:39:08.181855+0000","last_undegraded":"2024-03-24T00:39:08.181855+0000","last_fullsized":"2024-03-24T00:39:08.181855+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:45:00.603021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.176030+0000","last_change":"2024-03-24T00:39:08.169399+0000","last_active":"2024-03-24T00:39:08.176030+0000","last_peered":"2024-03-24T00:39:08.176030+0000","last_clean":"2024-03-24T00:39:08.176030+0000","last_became_active":"2024-03-24T00:39:08.169119+0000","last_became_peered":"2024-03-24T00:39:08.169119+0000","last_unstale":"2024-03-24T00:39:08.176030+0000","last_undegraded":"2024-03-24T00:39:08.176030+0000","last_fullsized":"2024-03-24T00:39:08.176030+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:39:28.087874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.153115+0000","last_change":"2024-03-24T00:39:07.153597+0000","last_active":"2024-03-24T00:39:08.153115+0000","last_peered":"2024-03-24T00:39:08.153115+0000","last_clean":"2024-03-24T00:39:08.153115+0000","last_became_active":"2024-03-24T00:39:07.153075+0000","last_became_peered":"2024-03-24T00:39:07.153075+0000","last_unstale":"2024-03-24T00:39:08.153115+0000","last_undegraded":"2024-03-24T00:39:08.153115+0000","last_fullsized":"2024-03-24T00:39:08.153115+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:43.804017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.170143+0000","last_change":"2024-03-24T00:39:08.165180+0000","last_active":"2024-03-24T00:39:08.170143+0000","last_peered":"2024-03-24T00:39:08.170143+0000","last_clean":"2024-03-24T00:39:08.170143+0000","last_became_active":"2024-03-24T00:39:08.164881+0000","last_became_peered":"2024-03-24T00:39:08.164881+0000","last_unstale":"2024-03-24T00:39:08.170143+0000","last_undegraded":"2024-03-24T00:39:08.170143+0000","last_fullsized":"2024-03-24T00:39:08.170143+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:03.669831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.150587+0000","last_change":"2024-03-24T00:39:07.150130+0000","last_active":"2024-03-24T00:39:08.150587+0000","last_peered":"2024-03-24T00:39:08.150587+0000","last_clean":"2024-03-24T00:39:08.150587+0000","last_became_active":"2024-03-24T00:39:07.149798+0000","last_became_peered":"2024-03-24T00:39:07.149798+0000","last_unstale":"2024-03-24T00:39:08.150587+0000","last_undegraded":"2024-03-24T00:39:08.150587+0000","last_fullsized":"2024-03-24T00:39:08.150587+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:50.557232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.171353+0000","last_change":"2024-03-24T00:39:08.166623+0000","last_active":"2024-03-24T00:39:08.171353+0000","last_peered":"2024-03-24T00:39:08.171353+0000","last_clean":"2024-03-24T00:39:08.171353+0000","last_became_active":"2024-03-24T00:39:08.166356+0000","last_became_peered":"2024-03-24T00:39:08.166356+0000","last_unstale":"2024-03-24T00:39:08.171353+0000","last_undegraded":"2024-03-24T00:39:08.171353+0000","last_fullsized":"2024-03-24T00:39:08.171353+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:17:55.856314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.155012+0000","last_change":"2024-03-24T00:39:07.157488+0000","last_active":"2024-03-24T00:39:08.155012+0000","last_peered":"2024-03-24T00:39:08.155012+0000","last_clean":"2024-03-24T00:39:08.155012+0000","last_became_active":"2024-03-24T00:39:07.155634+0000","last_became_peered":"2024-03-24T00:39:07.155634+0000","last_unstale":"2024-03-24T00:39:08.155012+0000","last_undegraded":"2024-03-24T00:39:08.155012+0000","last_fullsized":"2024-03-24T00:39:08.155012+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:28.252944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.165589+0000","last_change":"2024-03-24T00:39:08.162717+0000","last_active":"2024-03-24T00:39:08.165589+0000","last_peered":"2024-03-24T00:39:08.165589+0000","last_clean":"2024-03-24T00:39:08.165589+0000","last_became_active":"2024-03-24T00:39:08.158902+0000","last_became_peered":"2024-03-24T00:39:08.158902+0000","last_unstale":"2024-03-24T00:39:08.165589+0000","last_undegraded":"2024-03-24T00:39:08.165589+0000","last_fullsized":"2024-03-24T00:39:08.165589+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:11.822123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.149818+0000","last_change":"2024-03-24T00:39:07.148740+0000","last_active":"2024-03-24T00:39:08.149818+0000","last_peered":"2024-03-24T00:39:08.149818+0000","last_clean":"2024-03-24T00:39:08.149818+0000","last_became_active":"2024-03-24T00:39:07.148295+0000","last_became_peered":"2024-03-24T00:39:07.148295+0000","last_unstale":"2024-03-24T00:39:08.149818+0000","last_undegraded":"2024-03-24T00:39:08.149818+0000","last_fullsized":"2024-03-24T00:39:08.149818+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:57:39.605387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.178618+0000","last_change":"2024-03-24T00:39:08.169371+0000","last_active":"2024-03-24T00:39:08.178618+0000","last_peered":"2024-03-24T00:39:08.178618+0000","last_clean":"2024-03-24T00:39:08.178618+0000","last_became_active":"2024-03-24T00:39:08.169023+0000","last_became_peered":"2024-03-24T00:39:08.169023+0000","last_unstale":"2024-03-24T00:39:08.178618+0000","last_undegraded":"2024-03-24T00:39:08.178618+0000","last_fullsized":"2024-03-24T00:39:08.178618+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:42.707367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.151244+0000","last_change":"2024-03-24T00:39:07.158892+0000","last_active":"2024-03-24T00:39:08.151244+0000","last_peered":"2024-03-24T00:39:08.151244+0000","last_clean":"2024-03-24T00:39:08.151244+0000","last_became_active":"2024-03-24T00:39:07.158516+0000","last_became_peered":"2024-03-24T00:39:07.158516+0000","last_unstale":"2024-03-24T00:39:08.151244+0000","last_undegraded":"2024-03-24T00:39:08.151244+0000","last_fullsized":"2024-03-24T00:39:08.151244+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:16:23.871407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.186421+0000","last_change":"2024-03-24T00:39:08.181520+0000","last_active":"2024-03-24T00:39:08.186421+0000","last_peered":"2024-03-24T00:39:08.186421+0000","last_clean":"2024-03-24T00:39:08.186421+0000","last_became_active":"2024-03-24T00:39:08.181253+0000","last_became_peered":"2024-03-24T00:39:08.181253+0000","last_unstale":"2024-03-24T00:39:08.186421+0000","last_undegraded":"2024-03-24T00:39:08.186421+0000","last_fullsized":"2024-03-24T00:39:08.186421+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:39:55.017873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.171450+0000","last_change":"2024-03-24T00:39:07.146864+0000","last_active":"2024-03-24T00:39:08.171450+0000","last_peered":"2024-03-24T00:39:08.171450+0000","last_clean":"2024-03-24T00:39:08.171450+0000","last_became_active":"2024-03-24T00:39:07.146537+0000","last_became_peered":"2024-03-24T00:39:07.146537+0000","last_unstale":"2024-03-24T00:39:08.171450+0000","last_undegraded":"2024-03-24T00:39:08.171450+0000","last_fullsized":"2024-03-24T00:39:08.171450+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:41.871203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.167673+0000","last_change":"2024-03-24T00:39:08.160739+0000","last_active":"2024-03-24T00:39:08.167673+0000","last_peered":"2024-03-24T00:39:08.167673+0000","last_clean":"2024-03-24T00:39:08.167673+0000","last_became_active":"2024-03-24T00:39:08.160228+0000","last_became_peered":"2024-03-24T00:39:08.160228+0000","last_unstale":"2024-03-24T00:39:08.167673+0000","last_undegraded":"2024-03-24T00:39:08.167673+0000","last_fullsized":"2024-03-24T00:39:08.167673+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:42.707367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.153267+0000","last_change":"2024-03-24T00:39:07.154983+0000","last_active":"2024-03-24T00:39:08.153267+0000","last_peered":"2024-03-24T00:39:08.153267+0000","last_clean":"2024-03-24T00:39:08.153267+0000","last_became_active":"2024-03-24T00:39:07.154648+0000","last_became_peered":"2024-03-24T00:39:07.154648+0000","last_unstale":"2024-03-24T00:39:08.153267+0000","last_undegraded":"2024-03-24T00:39:08.153267+0000","last_fullsized":"2024-03-24T00:39:08.153267+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:28.228194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.174923+0000","last_change":"2024-03-24T00:39:08.170158+0000","last_active":"2024-03-24T00:39:08.174923+0000","last_peered":"2024-03-24T00:39:08.174923+0000","last_clean":"2024-03-24T00:39:08.174923+0000","last_became_active":"2024-03-24T00:39:08.168582+0000","last_became_peered":"2024-03-24T00:39:08.168582+0000","last_unstale":"2024-03-24T00:39:08.174923+0000","last_undegraded":"2024-03-24T00:39:08.174923+0000","last_fullsized":"2024-03-24T00:39:08.174923+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:59.444846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"118'2","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.188168+0000","last_change":"2024-03-24T00:39:07.155465+0000","last_active":"2024-03-24T00:39:08.188168+0000","last_peered":"2024-03-24T00:39:08.188168+0000","last_clean":"2024-03-24T00:39:08.188168+0000","last_became_active":"2024-03-24T00:39:07.155196+0000","last_became_peered":"2024-03-24T00:39:07.155196+0000","last_unstale":"2024-03-24T00:39:08.188168+0000","last_undegraded":"2024-03-24T00:39:08.188168+0000","last_fullsized":"2024-03-24T00:39:08.188168+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:05:35.119226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":369,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.150210+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:39:08.150210+0000","last_peered":"2024-03-24T00:39:08.150210+0000","last_clean":"2024-03-24T00:39:08.150210+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:39:08.150210+0000","last_undegraded":"2024-03-24T00:39:08.150210+0000","last_fullsized":"2024-03-24T00:39:08.150210+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574949,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37352,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334488,"statfs":{"total":96636764160,"available":96598515712,"internally_reserved":0,"allocated":2056192,"data_stored":1228927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574950,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40960,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330880,"statfs":{"total":96636764160,"available":96594821120,"internally_reserved":0,"allocated":1486848,"data_stored":646946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1462272,"data_stored":638559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574950,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50672,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321168,"statfs":{"total":96636764160,"available":96584876032,"internally_reserved":0,"allocated":1478656,"data_stored":645404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72868,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94298972,"statfs":{"total":96636764160,"available":96562147328,"internally_reserved":0,"allocated":1462272,"data_stored":638559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574950,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40928,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330912,"statfs":{"total":96636764160,"available":96594853888,"internally_reserved":0,"allocated":1462272,"data_stored":638559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45100,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326740,"statfs":{"total":96636764160,"available":96590581760,"internally_reserved":0,"allocated":1462272,"data_stored":638559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574949,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47080,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324760,"statfs":{"total":96636764160,"available":96588554240,"internally_reserved":0,"allocated":2056192,"data_stored":1228927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:39:12.372 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:39:12.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.544+0000 7f522f800640 1 Processor -- start 2024-03-24T00:39:12.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f522f800640 1 -- start start 2024-03-24T00:39:12.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f522f800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52280b2490 0x7f52280cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f522f800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f52280d0a80 0x7f52280cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f522f800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52280b2870 con 0x7f52280b2aa0 2024-03-24T00:39:12.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f522f800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522806b6d0 con 0x7f52280d0a80 2024-03-24T00:39:12.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f522f800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52280b3190 con 0x7f52280b2490 2024-03-24T00:39:12.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f5227e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f52280b2aa0 0x7f52280d0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42656/0 (socket says 172.21.15.40:42656) 2024-03-24T00:39:12.552 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f5227e00640 1 -- 172.21.15.40:0/585617885 learned_addr learned my addr 172.21.15.40:0/585617885 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:12.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f522d800640 1 --2- 172.21.15.40:0/585617885 >> v2:172.21.15.53:3300/0 conn(0x7f52280d0a80 0x7f52280cc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f522ce00640 1 --2- 172.21.15.40:0/585617885 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52280b2490 0x7f52280cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f5227400640 1 -- 172.21.15.40:0/585617885 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761516958 0 0) 0x7f52280b2870 con 0x7f52280b2aa0 2024-03-24T00:39:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f5227400640 1 -- 172.21.15.40:0/585617885 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5210003530 con 0x7f52280b2aa0 2024-03-24T00:39:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f522d800640 1 -- 172.21.15.40:0/585617885 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52280b2490 msgr2=0x7f52280cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f522d800640 1 --2- 172.21.15.40:0/585617885 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52280b2490 0x7f52280cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f522d800640 1 -- 172.21.15.40:0/585617885 >> v1:172.21.15.40:6789/0 conn(0x7f52280b2aa0 legacy=0x7f52280d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.548+0000 7f522d800640 1 -- 172.21.15.40:0/585617885 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52280b3470 con 0x7f52280d0a80 2024-03-24T00:39:12.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f522d800640 1 --2- 172.21.15.40:0/585617885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52280d0a80 0x7f52280cc460 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5220009d90 tx=0x7f522001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f31a6182c5a574c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:12.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f5227400640 1 -- 172.21.15.40:0/585617885 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52200b2070 con 0x7f52280d0a80 2024-03-24T00:39:12.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f5227400640 1 -- 172.21.15.40:0/585617885 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52200a5940 con 0x7f52280d0a80 2024-03-24T00:39:12.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f5227400640 1 -- 172.21.15.40:0/585617885 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52200a5d10 con 0x7f52280d0a80 2024-03-24T00:39:12.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f522f800640 1 -- 172.21.15.40:0/585617885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52280d0a80 msgr2=0x7f52280cc460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f522f800640 1 --2- 172.21.15.40:0/585617885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52280d0a80 0x7f52280cc460 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f5220009d90 tx=0x7f522001c970 comp rx=0 tx=0).stop 2024-03-24T00:39:12.557 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f522f800640 1 -- 172.21.15.40:0/585617885 shutdown_connections 2024-03-24T00:39:12.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f522f800640 1 --2- 172.21.15.40:0/585617885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52280d0a80 0x7f52280cc460 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f522f800640 1 --2- 172.21.15.40:0/585617885 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52280b2490 0x7f52280cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f522f800640 1 -- 172.21.15.40:0/585617885 >> 172.21.15.40:0/585617885 conn(0x7f52280d1d10 msgr2=0x7f52280d2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f522f800640 1 -- 172.21.15.40:0/585617885 shutdown_connections 2024-03-24T00:39:12.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f522f800640 1 -- 172.21.15.40:0/585617885 wait complete. 2024-03-24T00:39:12.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f522f800640 1 Processor -- start 2024-03-24T00:39:12.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.552+0000 7f522f800640 1 -- start start 2024-03-24T00:39:12.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522f800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52280b2490 0x7f522815dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522f800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52280b2aa0 0x7f52280cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522f800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52280d0a80 0x7f522814edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522f800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52280cd580 con 0x7f52280d0a80 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522f800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52280cd400 con 0x7f52280b2aa0 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522f800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52280cd700 con 0x7f52280b2490 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52280d0a80 0x7f522814edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522d800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52280d0a80 0x7f522814edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49456/0 (socket says 172.21.15.40:49456) 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522d800640 1 -- 172.21.15.40:0/2362499181 learned_addr learned my addr 172.21.15.40:0/2362499181 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f5227e00640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52280b2aa0 0x7f52280cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522ce00640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52280b2490 0x7f522815dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522d800640 1 -- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52280b2490 msgr2=0x7f522815dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522d800640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52280b2490 0x7f522815dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522d800640 1 -- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52280b2aa0 msgr2=0x7f52280cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522d800640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52280b2aa0 0x7f52280cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522d800640 1 -- 172.21.15.40:0/2362499181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522815e5d0 con 0x7f52280d0a80 2024-03-24T00:39:12.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522d800640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52280d0a80 0x7f522814edc0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f522001b020 tx=0x7f5220019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:12.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f5224c00640 1 -- 172.21.15.40:0/2362499181 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52200b2050 con 0x7f52280d0a80 2024-03-24T00:39:12.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f5224c00640 1 -- 172.21.15.40:0/2362499181 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52200be070 con 0x7f52280d0a80 2024-03-24T00:39:12.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522f800640 1 -- 172.21.15.40:0/2362499181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522815e8b0 con 0x7f52280d0a80 2024-03-24T00:39:12.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f5224c00640 1 -- 172.21.15.40:0/2362499181 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220009450 con 0x7f52280d0a80 2024-03-24T00:39:12.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522f800640 1 -- 172.21.15.40:0/2362499181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52280c9070 con 0x7f52280d0a80 2024-03-24T00:39:12.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.556+0000 7f522f800640 1 -- 172.21.15.40:0/2362499181 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5228068170 con 0x7f52280d0a80 2024-03-24T00:39:12.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.560+0000 7f5224c00640 1 -- 172.21.15.40:0/2362499181 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f52200adc20 con 0x7f52280d0a80 2024-03-24T00:39:12.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.560+0000 7f5224c00640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5208067690 0x7f5208069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.560+0000 7f5224c00640 1 -- 172.21.15.40:0/2362499181 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5220120ad0 con 0x7f52280d0a80 2024-03-24T00:39:12.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.560+0000 7f522ce00640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5208067690 0x7f5208069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.560+0000 7f522ce00640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5208067690 0x7f5208069b50 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f521c009a50 tx=0x7f521c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:12.572 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.568+0000 7f5224c00640 1 -- 172.21.15.40:0/2362499181 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5220016560 con 0x7f52280d0a80 2024-03-24T00:39:12.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.728+0000 7f522f800640 1 -- 172.21.15.40:0/2362499181 --> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f52280b7b40 con 0x7f5208067690 2024-03-24T00:39:12.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.732+0000 7f5224c00640 1 -- 172.21.15.40:0/2362499181 <== mgr.4105 v2:172.21.15.53:6832/4217283018 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252411 (secure 0 0 0) 0x7f52280b7b40 con 0x7f5208067690 2024-03-24T00:39:12.741 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:12.743 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-03-24T00:39:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.736+0000 7f522f800640 1 -- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5208067690 msgr2=0x7f5208069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.736+0000 7f522f800640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5208067690 0x7f5208069b50 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f521c009a50 tx=0x7f521c008040 comp rx=0 tx=0).stop 2024-03-24T00:39:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.736+0000 7f522f800640 1 -- 172.21.15.40:0/2362499181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52280d0a80 msgr2=0x7f522814edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.736+0000 7f522f800640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52280d0a80 0x7f522814edc0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f522001b020 tx=0x7f5220019710 comp rx=0 tx=0).stop 2024-03-24T00:39:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.736+0000 7f522f800640 1 -- 172.21.15.40:0/2362499181 shutdown_connections 2024-03-24T00:39:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.736+0000 7f522f800640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5208067690 0x7f5208069b50 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.736+0000 7f522f800640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52280d0a80 0x7f522814edc0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.736+0000 7f522f800640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52280b2aa0 0x7f52280cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.736+0000 7f522f800640 1 --2- 172.21.15.40:0/2362499181 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52280b2490 0x7f522815dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.736+0000 7f522f800640 1 -- 172.21.15.40:0/2362499181 >> 172.21.15.40:0/2362499181 conn(0x7f52280d1d10 msgr2=0x7f52280aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.736+0000 7f522f800640 1 -- 172.21.15.40:0/2362499181 shutdown_connections 2024-03-24T00:39:12.748 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.736+0000 7f522f800640 1 -- 172.21.15.40:0/2362499181 wait complete. 2024-03-24T00:39:12.762 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2024-03-24T00:39:11.294553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":371732,"kb_used_data":12624,"kb_used_omap":12,"kb_used_meta":358963,"kb_avail":754602988,"statfs":{"total":773094113280,"available":772713459712,"internally_reserved":0,"allocated":12926976,"data_stored":6304440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":367578674},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.166237"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185562+0000","last_change":"2024-03-24T00:39:08.182796+0000","last_active":"2024-03-24T00:39:08.185562+0000","last_peered":"2024-03-24T00:39:08.185562+0000","last_clean":"2024-03-24T00:39:08.185562+0000","last_became_active":"2024-03-24T00:39:08.180585+0000","last_became_peered":"2024-03-24T00:39:08.180585+0000","last_unstale":"2024-03-24T00:39:08.185562+0000","last_undegraded":"2024-03-24T00:39:08.185562+0000","last_fullsized":"2024-03-24T00:39:08.185562+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:52:02.548249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.161569+0000","last_change":"2024-03-24T00:39:08.158422+0000","last_active":"2024-03-24T00:39:08.161569+0000","last_peered":"2024-03-24T00:39:08.161569+0000","last_clean":"2024-03-24T00:39:08.161569+0000","last_became_active":"2024-03-24T00:39:08.157737+0000","last_became_peered":"2024-03-24T00:39:08.157737+0000","last_unstale":"2024-03-24T00:39:08.161569+0000","last_undegraded":"2024-03-24T00:39:08.161569+0000","last_fullsized":"2024-03-24T00:39:08.161569+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:14:22.215663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185843+0000","last_change":"2024-03-24T00:39:08.185222+0000","last_active":"2024-03-24T00:39:08.185843+0000","last_peered":"2024-03-24T00:39:08.185843+0000","last_clean":"2024-03-24T00:39:08.185843+0000","last_became_active":"2024-03-24T00:39:08.184830+0000","last_became_peered":"2024-03-24T00:39:08.184830+0000","last_unstale":"2024-03-24T00:39:08.185843+0000","last_undegraded":"2024-03-24T00:39:08.185843+0000","last_fullsized":"2024-03-24T00:39:08.185843+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:24:26.554733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.178897+0000","last_change":"2024-03-24T00:39:08.172805+0000","last_active":"2024-03-24T00:39:08.178897+0000","last_peered":"2024-03-24T00:39:08.178897+0000","last_clean":"2024-03-24T00:39:08.178897+0000","last_became_active":"2024-03-24T00:39:08.172522+0000","last_became_peered":"2024-03-24T00:39:08.172522+0000","last_unstale":"2024-03-24T00:39:08.178897+0000","last_undegraded":"2024-03-24T00:39:08.178897+0000","last_fullsized":"2024-03-24T00:39:08.178897+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:40:39.588732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.167376+0000","last_change":"2024-03-24T00:39:08.162870+0000","last_active":"2024-03-24T00:39:08.167376+0000","last_peered":"2024-03-24T00:39:08.167376+0000","last_clean":"2024-03-24T00:39:08.167376+0000","last_became_active":"2024-03-24T00:39:08.162521+0000","last_became_peered":"2024-03-24T00:39:08.162521+0000","last_unstale":"2024-03-24T00:39:08.167376+0000","last_undegraded":"2024-03-24T00:39:08.167376+0000","last_fullsized":"2024-03-24T00:39:08.167376+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:44.048508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.171001+0000","last_change":"2024-03-24T00:39:08.165635+0000","last_active":"2024-03-24T00:39:08.171001+0000","last_peered":"2024-03-24T00:39:08.171001+0000","last_clean":"2024-03-24T00:39:08.171001+0000","last_became_active":"2024-03-24T00:39:08.165348+0000","last_became_peered":"2024-03-24T00:39:08.165348+0000","last_unstale":"2024-03-24T00:39:08.171001+0000","last_undegraded":"2024-03-24T00:39:08.171001+0000","last_fullsized":"2024-03-24T00:39:08.171001+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:24:26.554733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.173040+0000","last_change":"2024-03-24T00:39:08.171795+0000","last_active":"2024-03-24T00:39:08.173040+0000","last_peered":"2024-03-24T00:39:08.173040+0000","last_clean":"2024-03-24T00:39:08.173040+0000","last_became_active":"2024-03-24T00:39:08.171367+0000","last_became_peered":"2024-03-24T00:39:08.171367+0000","last_unstale":"2024-03-24T00:39:08.173040+0000","last_undegraded":"2024-03-24T00:39:08.173040+0000","last_fullsized":"2024-03-24T00:39:08.173040+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:19:47.552389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.183501+0000","last_change":"2024-03-24T00:39:08.182999+0000","last_active":"2024-03-24T00:39:08.183501+0000","last_peered":"2024-03-24T00:39:08.183501+0000","last_clean":"2024-03-24T00:39:08.183501+0000","last_became_active":"2024-03-24T00:39:08.182682+0000","last_became_peered":"2024-03-24T00:39:08.182682+0000","last_unstale":"2024-03-24T00:39:08.183501+0000","last_undegraded":"2024-03-24T00:39:08.183501+0000","last_fullsized":"2024-03-24T00:39:08.183501+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:32.441579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181131+0000","last_change":"2024-03-24T00:39:08.177992+0000","last_active":"2024-03-24T00:39:08.181131+0000","last_peered":"2024-03-24T00:39:08.181131+0000","last_clean":"2024-03-24T00:39:08.181131+0000","last_became_active":"2024-03-24T00:39:08.177669+0000","last_became_peered":"2024-03-24T00:39:08.177669+0000","last_unstale":"2024-03-24T00:39:08.181131+0000","last_undegraded":"2024-03-24T00:39:08.181131+0000","last_fullsized":"2024-03-24T00:39:08.181131+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:30.569403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.161584+0000","last_change":"2024-03-24T00:39:08.158652+0000","last_active":"2024-03-24T00:39:08.161584+0000","last_peered":"2024-03-24T00:39:08.161584+0000","last_clean":"2024-03-24T00:39:08.161584+0000","last_became_active":"2024-03-24T00:39:08.158143+0000","last_became_peered":"2024-03-24T00:39:08.158143+0000","last_unstale":"2024-03-24T00:39:08.161584+0000","last_undegraded":"2024-03-24T00:39:08.161584+0000","last_fullsized":"2024-03-24T00:39:08.161584+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:07:32.874502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.182289+0000","last_change":"2024-03-24T00:39:08.167873+0000","last_active":"2024-03-24T00:39:08.182289+0000","last_peered":"2024-03-24T00:39:08.182289+0000","last_clean":"2024-03-24T00:39:08.182289+0000","last_became_active":"2024-03-24T00:39:08.167395+0000","last_became_peered":"2024-03-24T00:39:08.167395+0000","last_unstale":"2024-03-24T00:39:08.182289+0000","last_undegraded":"2024-03-24T00:39:08.182289+0000","last_fullsized":"2024-03-24T00:39:08.182289+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:27:25.182467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180708+0000","last_change":"2024-03-24T00:39:08.171429+0000","last_active":"2024-03-24T00:39:08.180708+0000","last_peered":"2024-03-24T00:39:08.180708+0000","last_clean":"2024-03-24T00:39:08.180708+0000","last_became_active":"2024-03-24T00:39:08.171104+0000","last_became_peered":"2024-03-24T00:39:08.171104+0000","last_unstale":"2024-03-24T00:39:08.180708+0000","last_undegraded":"2024-03-24T00:39:08.180708+0000","last_fullsized":"2024-03-24T00:39:08.180708+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:29:12.671642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185064+0000","last_change":"2024-03-24T00:39:08.182082+0000","last_active":"2024-03-24T00:39:08.185064+0000","last_peered":"2024-03-24T00:39:08.185064+0000","last_clean":"2024-03-24T00:39:08.185064+0000","last_became_active":"2024-03-24T00:39:08.178314+0000","last_became_peered":"2024-03-24T00:39:08.178314+0000","last_unstale":"2024-03-24T00:39:08.185064+0000","last_undegraded":"2024-03-24T00:39:08.185064+0000","last_fullsized":"2024-03-24T00:39:08.185064+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:42:57.233295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.169020+0000","last_change":"2024-03-24T00:39:08.163534+0000","last_active":"2024-03-24T00:39:08.169020+0000","last_peered":"2024-03-24T00:39:08.169020+0000","last_clean":"2024-03-24T00:39:08.169020+0000","last_became_active":"2024-03-24T00:39:08.163245+0000","last_became_peered":"2024-03-24T00:39:08.163245+0000","last_unstale":"2024-03-24T00:39:08.169020+0000","last_undegraded":"2024-03-24T00:39:08.169020+0000","last_fullsized":"2024-03-24T00:39:08.169020+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:08:50.321160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.177306+0000","last_change":"2024-03-24T00:39:08.168038+0000","last_active":"2024-03-24T00:39:08.177306+0000","last_peered":"2024-03-24T00:39:08.177306+0000","last_clean":"2024-03-24T00:39:08.177306+0000","last_became_active":"2024-03-24T00:39:08.167698+0000","last_became_peered":"2024-03-24T00:39:08.167698+0000","last_unstale":"2024-03-24T00:39:08.177306+0000","last_undegraded":"2024-03-24T00:39:08.177306+0000","last_fullsized":"2024-03-24T00:39:08.177306+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:33:35.230906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.188236+0000","last_change":"2024-03-24T00:39:08.187521+0000","last_active":"2024-03-24T00:39:08.188236+0000","last_peered":"2024-03-24T00:39:08.188236+0000","last_clean":"2024-03-24T00:39:08.188236+0000","last_became_active":"2024-03-24T00:39:08.187105+0000","last_became_peered":"2024-03-24T00:39:08.187105+0000","last_unstale":"2024-03-24T00:39:08.188236+0000","last_undegraded":"2024-03-24T00:39:08.188236+0000","last_fullsized":"2024-03-24T00:39:08.188236+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:10.676965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180077+0000","last_change":"2024-03-24T00:39:08.179170+0000","last_active":"2024-03-24T00:39:08.180077+0000","last_peered":"2024-03-24T00:39:08.180077+0000","last_clean":"2024-03-24T00:39:08.180077+0000","last_became_active":"2024-03-24T00:39:08.178886+0000","last_became_peered":"2024-03-24T00:39:08.178886+0000","last_unstale":"2024-03-24T00:39:08.180077+0000","last_undegraded":"2024-03-24T00:39:08.180077+0000","last_fullsized":"2024-03-24T00:39:08.180077+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:18:26.080188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.183982+0000","last_change":"2024-03-24T00:39:08.174096+0000","last_active":"2024-03-24T00:39:08.183982+0000","last_peered":"2024-03-24T00:39:08.183982+0000","last_clean":"2024-03-24T00:39:08.183982+0000","last_became_active":"2024-03-24T00:39:08.173814+0000","last_became_peered":"2024-03-24T00:39:08.173814+0000","last_unstale":"2024-03-24T00:39:08.183982+0000","last_undegraded":"2024-03-24T00:39:08.183982+0000","last_fullsized":"2024-03-24T00:39:08.183982+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:27.318468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.182313+0000","last_change":"2024-03-24T00:39:08.169315+0000","last_active":"2024-03-24T00:39:08.182313+0000","last_peered":"2024-03-24T00:39:08.182313+0000","last_clean":"2024-03-24T00:39:08.182313+0000","last_became_active":"2024-03-24T00:39:08.167088+0000","last_became_peered":"2024-03-24T00:39:08.167088+0000","last_unstale":"2024-03-24T00:39:08.182313+0000","last_undegraded":"2024-03-24T00:39:08.182313+0000","last_fullsized":"2024-03-24T00:39:08.182313+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:06:23.563817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.184762+0000","last_change":"2024-03-24T00:39:08.179567+0000","last_active":"2024-03-24T00:39:08.184762+0000","last_peered":"2024-03-24T00:39:08.184762+0000","last_clean":"2024-03-24T00:39:08.184762+0000","last_became_active":"2024-03-24T00:39:08.177697+0000","last_became_peered":"2024-03-24T00:39:08.177697+0000","last_unstale":"2024-03-24T00:39:08.184762+0000","last_undegraded":"2024-03-24T00:39:08.184762+0000","last_fullsized":"2024-03-24T00:39:08.184762+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:44.048508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.167352+0000","last_change":"2024-03-24T00:39:08.161960+0000","last_active":"2024-03-24T00:39:08.167352+0000","last_peered":"2024-03-24T00:39:08.167352+0000","last_clean":"2024-03-24T00:39:08.167352+0000","last_became_active":"2024-03-24T00:39:08.161618+0000","last_became_peered":"2024-03-24T00:39:08.161618+0000","last_unstale":"2024-03-24T00:39:08.167352+0000","last_undegraded":"2024-03-24T00:39:08.167352+0000","last_fullsized":"2024-03-24T00:39:08.167352+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:03:27.318468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181278+0000","last_change":"2024-03-24T00:39:08.165672+0000","last_active":"2024-03-24T00:39:08.181278+0000","last_peered":"2024-03-24T00:39:08.181278+0000","last_clean":"2024-03-24T00:39:08.181278+0000","last_became_active":"2024-03-24T00:39:08.165398+0000","last_became_peered":"2024-03-24T00:39:08.165398+0000","last_unstale":"2024-03-24T00:39:08.181278+0000","last_undegraded":"2024-03-24T00:39:08.181278+0000","last_fullsized":"2024-03-24T00:39:08.181278+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:35:18.867104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.172652+0000","last_change":"2024-03-24T00:39:08.167415+0000","last_active":"2024-03-24T00:39:08.172652+0000","last_peered":"2024-03-24T00:39:08.172652+0000","last_clean":"2024-03-24T00:39:08.172652+0000","last_became_active":"2024-03-24T00:39:08.164889+0000","last_became_peered":"2024-03-24T00:39:08.164889+0000","last_unstale":"2024-03-24T00:39:08.172652+0000","last_undegraded":"2024-03-24T00:39:08.172652+0000","last_fullsized":"2024-03-24T00:39:08.172652+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:23.192198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181531+0000","last_change":"2024-03-24T00:39:08.166803+0000","last_active":"2024-03-24T00:39:08.181531+0000","last_peered":"2024-03-24T00:39:08.181531+0000","last_clean":"2024-03-24T00:39:08.181531+0000","last_became_active":"2024-03-24T00:39:08.166544+0000","last_became_peered":"2024-03-24T00:39:08.166544+0000","last_unstale":"2024-03-24T00:39:08.181531+0000","last_undegraded":"2024-03-24T00:39:08.181531+0000","last_fullsized":"2024-03-24T00:39:08.181531+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:21:26.950051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.169044+0000","last_change":"2024-03-24T00:39:08.165503+0000","last_active":"2024-03-24T00:39:08.169044+0000","last_peered":"2024-03-24T00:39:08.169044+0000","last_clean":"2024-03-24T00:39:08.169044+0000","last_became_active":"2024-03-24T00:39:08.163567+0000","last_became_peered":"2024-03-24T00:39:08.163567+0000","last_unstale":"2024-03-24T00:39:08.169044+0000","last_undegraded":"2024-03-24T00:39:08.169044+0000","last_fullsized":"2024-03-24T00:39:08.169044+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:22.784502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.186281+0000","last_change":"2024-03-24T00:39:08.180186+0000","last_active":"2024-03-24T00:39:08.186281+0000","last_peered":"2024-03-24T00:39:08.186281+0000","last_clean":"2024-03-24T00:39:08.186281+0000","last_became_active":"2024-03-24T00:39:08.179892+0000","last_became_peered":"2024-03-24T00:39:08.179892+0000","last_unstale":"2024-03-24T00:39:08.186281+0000","last_undegraded":"2024-03-24T00:39:08.186281+0000","last_fullsized":"2024-03-24T00:39:08.186281+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:02.900425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.184575+0000","last_change":"2024-03-24T00:39:08.181623+0000","last_active":"2024-03-24T00:39:08.184575+0000","last_peered":"2024-03-24T00:39:08.184575+0000","last_clean":"2024-03-24T00:39:08.184575+0000","last_became_active":"2024-03-24T00:39:08.178412+0000","last_became_peered":"2024-03-24T00:39:08.178412+0000","last_unstale":"2024-03-24T00:39:08.184575+0000","last_undegraded":"2024-03-24T00:39:08.184575+0000","last_fullsized":"2024-03-24T00:39:08.184575+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:21.681285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181063+0000","last_change":"2024-03-24T00:39:08.175175+0000","last_active":"2024-03-24T00:39:08.181063+0000","last_peered":"2024-03-24T00:39:08.181063+0000","last_clean":"2024-03-24T00:39:08.181063+0000","last_became_active":"2024-03-24T00:39:08.174872+0000","last_became_peered":"2024-03-24T00:39:08.174872+0000","last_unstale":"2024-03-24T00:39:08.181063+0000","last_undegraded":"2024-03-24T00:39:08.181063+0000","last_fullsized":"2024-03-24T00:39:08.181063+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:08:50.321160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.175837+0000","last_change":"2024-03-24T00:39:08.167248+0000","last_active":"2024-03-24T00:39:08.175837+0000","last_peered":"2024-03-24T00:39:08.175837+0000","last_clean":"2024-03-24T00:39:08.175837+0000","last_became_active":"2024-03-24T00:39:08.166946+0000","last_became_peered":"2024-03-24T00:39:08.166946+0000","last_unstale":"2024-03-24T00:39:08.175837+0000","last_undegraded":"2024-03-24T00:39:08.175837+0000","last_fullsized":"2024-03-24T00:39:08.175837+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:04:58.267823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.175804+0000","last_change":"2024-03-24T00:39:08.168189+0000","last_active":"2024-03-24T00:39:08.175804+0000","last_peered":"2024-03-24T00:39:08.175804+0000","last_clean":"2024-03-24T00:39:08.175804+0000","last_became_active":"2024-03-24T00:39:08.167869+0000","last_became_peered":"2024-03-24T00:39:08.167869+0000","last_unstale":"2024-03-24T00:39:08.175804+0000","last_undegraded":"2024-03-24T00:39:08.175804+0000","last_fullsized":"2024-03-24T00:39:08.175804+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:38:29.091630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.174275+0000","last_change":"2024-03-24T00:39:08.167080+0000","last_active":"2024-03-24T00:39:08.174275+0000","last_peered":"2024-03-24T00:39:08.174275+0000","last_clean":"2024-03-24T00:39:08.174275+0000","last_became_active":"2024-03-24T00:39:08.166373+0000","last_became_peered":"2024-03-24T00:39:08.166373+0000","last_unstale":"2024-03-24T00:39:08.174275+0000","last_undegraded":"2024-03-24T00:39:08.174275+0000","last_fullsized":"2024-03-24T00:39:08.174275+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:36:02.900425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181742+0000","last_change":"2024-03-24T00:39:08.180345+0000","last_active":"2024-03-24T00:39:08.181742+0000","last_peered":"2024-03-24T00:39:08.181742+0000","last_clean":"2024-03-24T00:39:08.181742+0000","last_became_active":"2024-03-24T00:39:08.180006+0000","last_became_peered":"2024-03-24T00:39:08.180006+0000","last_unstale":"2024-03-24T00:39:08.181742+0000","last_undegraded":"2024-03-24T00:39:08.181742+0000","last_fullsized":"2024-03-24T00:39:08.181742+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:10:31.283701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.187491+0000","last_change":"2024-03-24T00:39:08.186098+0000","last_active":"2024-03-24T00:39:08.187491+0000","last_peered":"2024-03-24T00:39:08.187491+0000","last_clean":"2024-03-24T00:39:08.187491+0000","last_became_active":"2024-03-24T00:39:08.185015+0000","last_became_peered":"2024-03-24T00:39:08.185015+0000","last_unstale":"2024-03-24T00:39:08.187491+0000","last_undegraded":"2024-03-24T00:39:08.187491+0000","last_fullsized":"2024-03-24T00:39:08.187491+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:59.444846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"118'4","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.188314+0000","last_change":"2024-03-24T00:39:07.156116+0000","last_active":"2024-03-24T00:39:08.188314+0000","last_peered":"2024-03-24T00:39:08.188314+0000","last_clean":"2024-03-24T00:39:08.188314+0000","last_became_active":"2024-03-24T00:39:07.155762+0000","last_became_peered":"2024-03-24T00:39:07.155762+0000","last_unstale":"2024-03-24T00:39:08.188314+0000","last_undegraded":"2024-03-24T00:39:08.188314+0000","last_fullsized":"2024-03-24T00:39:08.188314+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:41.871203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180699+0000","last_change":"2024-03-24T00:39:08.168733+0000","last_active":"2024-03-24T00:39:08.180699+0000","last_peered":"2024-03-24T00:39:08.180699+0000","last_clean":"2024-03-24T00:39:08.180699+0000","last_became_active":"2024-03-24T00:39:08.168372+0000","last_became_peered":"2024-03-24T00:39:08.168372+0000","last_unstale":"2024-03-24T00:39:08.180699+0000","last_undegraded":"2024-03-24T00:39:08.180699+0000","last_fullsized":"2024-03-24T00:39:08.180699+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:02:18.615058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"118'1","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185372+0000","last_change":"2024-03-24T00:39:07.156929+0000","last_active":"2024-03-24T00:39:08.185372+0000","last_peered":"2024-03-24T00:39:08.185372+0000","last_clean":"2024-03-24T00:39:08.185372+0000","last_became_active":"2024-03-24T00:39:07.156587+0000","last_became_peered":"2024-03-24T00:39:07.156587+0000","last_unstale":"2024-03-24T00:39:08.185372+0000","last_undegraded":"2024-03-24T00:39:08.185372+0000","last_fullsized":"2024-03-24T00:39:08.185372+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:14:21.207962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.163982+0000","last_change":"2024-03-24T00:39:08.160028+0000","last_active":"2024-03-24T00:39:08.163982+0000","last_peered":"2024-03-24T00:39:08.163982+0000","last_clean":"2024-03-24T00:39:08.163982+0000","last_became_active":"2024-03-24T00:39:08.159640+0000","last_became_peered":"2024-03-24T00:39:08.159640+0000","last_unstale":"2024-03-24T00:39:08.163982+0000","last_undegraded":"2024-03-24T00:39:08.163982+0000","last_fullsized":"2024-03-24T00:39:08.163982+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:17:18.299416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"118'2","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185964+0000","last_change":"2024-03-24T00:39:07.158050+0000","last_active":"2024-03-24T00:39:08.185964+0000","last_peered":"2024-03-24T00:39:08.185964+0000","last_clean":"2024-03-24T00:39:08.185964+0000","last_became_active":"2024-03-24T00:39:07.157789+0000","last_became_peered":"2024-03-24T00:39:07.157789+0000","last_unstale":"2024-03-24T00:39:08.185964+0000","last_undegraded":"2024-03-24T00:39:08.185964+0000","last_fullsized":"2024-03-24T00:39:08.185964+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:43:14.768323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.164970+0000","last_change":"2024-03-24T00:39:08.159013+0000","last_active":"2024-03-24T00:39:08.164970+0000","last_peered":"2024-03-24T00:39:08.164970+0000","last_clean":"2024-03-24T00:39:08.164970+0000","last_became_active":"2024-03-24T00:39:08.158609+0000","last_became_peered":"2024-03-24T00:39:08.158609+0000","last_unstale":"2024-03-24T00:39:08.164970+0000","last_undegraded":"2024-03-24T00:39:08.164970+0000","last_fullsized":"2024-03-24T00:39:08.164970+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:34:56.731869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"118'2","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.187532+0000","last_change":"2024-03-24T00:39:07.155423+0000","last_active":"2024-03-24T00:39:08.187532+0000","last_peered":"2024-03-24T00:39:08.187532+0000","last_clean":"2024-03-24T00:39:08.187532+0000","last_became_active":"2024-03-24T00:39:07.155065+0000","last_became_peered":"2024-03-24T00:39:07.155065+0000","last_unstale":"2024-03-24T00:39:08.187532+0000","last_undegraded":"2024-03-24T00:39:08.187532+0000","last_fullsized":"2024-03-24T00:39:08.187532+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:24.117504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181429+0000","last_change":"2024-03-24T00:39:08.174021+0000","last_active":"2024-03-24T00:39:08.181429+0000","last_peered":"2024-03-24T00:39:08.181429+0000","last_clean":"2024-03-24T00:39:08.181429+0000","last_became_active":"2024-03-24T00:39:08.173708+0000","last_became_peered":"2024-03-24T00:39:08.173708+0000","last_unstale":"2024-03-24T00:39:08.181429+0000","last_undegraded":"2024-03-24T00:39:08.181429+0000","last_fullsized":"2024-03-24T00:39:08.181429+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:17:18.299416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"118'1","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185477+0000","last_change":"2024-03-24T00:39:07.157052+0000","last_active":"2024-03-24T00:39:08.185477+0000","last_peered":"2024-03-24T00:39:08.185477+0000","last_clean":"2024-03-24T00:39:08.185477+0000","last_became_active":"2024-03-24T00:39:07.156750+0000","last_became_peered":"2024-03-24T00:39:07.156750+0000","last_unstale":"2024-03-24T00:39:08.185477+0000","last_undegraded":"2024-03-24T00:39:08.185477+0000","last_fullsized":"2024-03-24T00:39:08.185477+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:26.722631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.176816+0000","last_change":"2024-03-24T00:39:08.175384+0000","last_active":"2024-03-24T00:39:08.176816+0000","last_peered":"2024-03-24T00:39:08.176816+0000","last_clean":"2024-03-24T00:39:08.176816+0000","last_became_active":"2024-03-24T00:39:08.174162+0000","last_became_peered":"2024-03-24T00:39:08.174162+0000","last_unstale":"2024-03-24T00:39:08.176816+0000","last_undegraded":"2024-03-24T00:39:08.176816+0000","last_fullsized":"2024-03-24T00:39:08.176816+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:10:23.123726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"118'1","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.182300+0000","last_change":"2024-03-24T00:39:07.159939+0000","last_active":"2024-03-24T00:39:08.182300+0000","last_peered":"2024-03-24T00:39:08.182300+0000","last_clean":"2024-03-24T00:39:08.182300+0000","last_became_active":"2024-03-24T00:39:07.159408+0000","last_became_peered":"2024-03-24T00:39:07.159408+0000","last_unstale":"2024-03-24T00:39:08.182300+0000","last_undegraded":"2024-03-24T00:39:08.182300+0000","last_fullsized":"2024-03-24T00:39:08.182300+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:05:35.119226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181701+0000","last_change":"2024-03-24T00:39:08.179606+0000","last_active":"2024-03-24T00:39:08.181701+0000","last_peered":"2024-03-24T00:39:08.181701+0000","last_clean":"2024-03-24T00:39:08.181701+0000","last_became_active":"2024-03-24T00:39:08.179268+0000","last_became_peered":"2024-03-24T00:39:08.179268+0000","last_unstale":"2024-03-24T00:39:08.181701+0000","last_undegraded":"2024-03-24T00:39:08.181701+0000","last_fullsized":"2024-03-24T00:39:08.181701+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:24:08.067590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.148959+0000","last_change":"2024-03-24T00:39:07.152655+0000","last_active":"2024-03-24T00:39:08.148959+0000","last_peered":"2024-03-24T00:39:08.148959+0000","last_clean":"2024-03-24T00:39:08.148959+0000","last_became_active":"2024-03-24T00:39:07.152323+0000","last_became_peered":"2024-03-24T00:39:07.152323+0000","last_unstale":"2024-03-24T00:39:08.148959+0000","last_undegraded":"2024-03-24T00:39:08.148959+0000","last_fullsized":"2024-03-24T00:39:08.148959+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:26:29.561702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.186639+0000","last_change":"2024-03-24T00:39:08.183774+0000","last_active":"2024-03-24T00:39:08.186639+0000","last_peered":"2024-03-24T00:39:08.186639+0000","last_clean":"2024-03-24T00:39:08.186639+0000","last_became_active":"2024-03-24T00:39:08.183104+0000","last_became_peered":"2024-03-24T00:39:08.183104+0000","last_unstale":"2024-03-24T00:39:08.186639+0000","last_undegraded":"2024-03-24T00:39:08.186639+0000","last_fullsized":"2024-03-24T00:39:08.186639+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:10:37.340378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.148220+0000","last_change":"2024-03-24T00:39:07.150904+0000","last_active":"2024-03-24T00:39:08.148220+0000","last_peered":"2024-03-24T00:39:08.148220+0000","last_clean":"2024-03-24T00:39:08.148220+0000","last_became_active":"2024-03-24T00:39:07.150623+0000","last_became_peered":"2024-03-24T00:39:07.150623+0000","last_unstale":"2024-03-24T00:39:08.148220+0000","last_undegraded":"2024-03-24T00:39:08.148220+0000","last_fullsized":"2024-03-24T00:39:08.148220+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:34:10.960934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.182700+0000","last_change":"2024-03-24T00:39:08.177210+0000","last_active":"2024-03-24T00:39:08.182700+0000","last_peered":"2024-03-24T00:39:08.182700+0000","last_clean":"2024-03-24T00:39:08.182700+0000","last_became_active":"2024-03-24T00:39:08.176905+0000","last_became_peered":"2024-03-24T00:39:08.176905+0000","last_unstale":"2024-03-24T00:39:08.182700+0000","last_undegraded":"2024-03-24T00:39:08.182700+0000","last_fullsized":"2024-03-24T00:39:08.182700+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:16:44.874403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.155581+0000","last_change":"2024-03-24T00:39:07.146610+0000","last_active":"2024-03-24T00:39:08.155581+0000","last_peered":"2024-03-24T00:39:08.155581+0000","last_clean":"2024-03-24T00:39:08.155581+0000","last_became_active":"2024-03-24T00:39:07.146284+0000","last_became_peered":"2024-03-24T00:39:07.146284+0000","last_unstale":"2024-03-24T00:39:08.155581+0000","last_undegraded":"2024-03-24T00:39:08.155581+0000","last_fullsized":"2024-03-24T00:39:08.155581+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:24.117504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181323+0000","last_change":"2024-03-24T00:39:08.177972+0000","last_active":"2024-03-24T00:39:08.181323+0000","last_peered":"2024-03-24T00:39:08.181323+0000","last_clean":"2024-03-24T00:39:08.181323+0000","last_became_active":"2024-03-24T00:39:08.177097+0000","last_became_peered":"2024-03-24T00:39:08.177097+0000","last_unstale":"2024-03-24T00:39:08.181323+0000","last_undegraded":"2024-03-24T00:39:08.181323+0000","last_fullsized":"2024-03-24T00:39:08.181323+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:06:54.840723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.174852+0000","last_change":"2024-03-24T00:39:07.148239+0000","last_active":"2024-03-24T00:39:08.174852+0000","last_peered":"2024-03-24T00:39:08.174852+0000","last_clean":"2024-03-24T00:39:08.174852+0000","last_became_active":"2024-03-24T00:39:07.147917+0000","last_became_peered":"2024-03-24T00:39:07.147917+0000","last_unstale":"2024-03-24T00:39:08.174852+0000","last_undegraded":"2024-03-24T00:39:08.174852+0000","last_fullsized":"2024-03-24T00:39:08.174852+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:10.814422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.176942+0000","last_change":"2024-03-24T00:39:08.175413+0000","last_active":"2024-03-24T00:39:08.176942+0000","last_peered":"2024-03-24T00:39:08.176942+0000","last_clean":"2024-03-24T00:39:08.176942+0000","last_became_active":"2024-03-24T00:39:08.175078+0000","last_became_peered":"2024-03-24T00:39:08.175078+0000","last_unstale":"2024-03-24T00:39:08.176942+0000","last_undegraded":"2024-03-24T00:39:08.176942+0000","last_fullsized":"2024-03-24T00:39:08.176942+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:10.676965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"118'3","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185125+0000","last_change":"2024-03-24T00:39:07.153900+0000","last_active":"2024-03-24T00:39:08.185125+0000","last_peered":"2024-03-24T00:39:08.185125+0000","last_clean":"2024-03-24T00:39:08.185125+0000","last_became_active":"2024-03-24T00:39:07.153598+0000","last_became_peered":"2024-03-24T00:39:07.153598+0000","last_unstale":"2024-03-24T00:39:08.185125+0000","last_undegraded":"2024-03-24T00:39:08.185125+0000","last_fullsized":"2024-03-24T00:39:08.185125+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:57:39.605387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.174795+0000","last_change":"2024-03-24T00:39:08.168213+0000","last_active":"2024-03-24T00:39:08.174795+0000","last_peered":"2024-03-24T00:39:08.174795+0000","last_clean":"2024-03-24T00:39:08.174795+0000","last_became_active":"2024-03-24T00:39:08.167110+0000","last_became_peered":"2024-03-24T00:39:08.167110+0000","last_unstale":"2024-03-24T00:39:08.174795+0000","last_undegraded":"2024-03-24T00:39:08.174795+0000","last_fullsized":"2024-03-24T00:39:08.174795+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:31.763862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"118'6","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.189928+0000","last_change":"2024-03-24T00:39:07.148077+0000","last_active":"2024-03-24T00:39:08.189928+0000","last_peered":"2024-03-24T00:39:08.189928+0000","last_clean":"2024-03-24T00:39:08.189928+0000","last_became_active":"2024-03-24T00:39:07.146873+0000","last_became_peered":"2024-03-24T00:39:07.146873+0000","last_unstale":"2024-03-24T00:39:08.189928+0000","last_undegraded":"2024-03-24T00:39:08.189928+0000","last_fullsized":"2024-03-24T00:39:08.189928+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:10:21.776801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.186026+0000","last_change":"2024-03-24T00:39:08.183414+0000","last_active":"2024-03-24T00:39:08.186026+0000","last_peered":"2024-03-24T00:39:08.186026+0000","last_clean":"2024-03-24T00:39:08.186026+0000","last_became_active":"2024-03-24T00:39:08.183148+0000","last_became_peered":"2024-03-24T00:39:08.183148+0000","last_unstale":"2024-03-24T00:39:08.186026+0000","last_undegraded":"2024-03-24T00:39:08.186026+0000","last_fullsized":"2024-03-24T00:39:08.186026+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:57:42.456654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.159973+0000","last_change":"2024-03-24T00:39:07.151514+0000","last_active":"2024-03-24T00:39:08.159973+0000","last_peered":"2024-03-24T00:39:08.159973+0000","last_clean":"2024-03-24T00:39:08.159973+0000","last_became_active":"2024-03-24T00:39:07.151173+0000","last_became_peered":"2024-03-24T00:39:07.151173+0000","last_unstale":"2024-03-24T00:39:08.159973+0000","last_undegraded":"2024-03-24T00:39:08.159973+0000","last_fullsized":"2024-03-24T00:39:08.159973+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:48:58.379103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.178945+0000","last_change":"2024-03-24T00:39:08.164686+0000","last_active":"2024-03-24T00:39:08.178945+0000","last_peered":"2024-03-24T00:39:08.178945+0000","last_clean":"2024-03-24T00:39:08.178945+0000","last_became_active":"2024-03-24T00:39:08.164223+0000","last_became_peered":"2024-03-24T00:39:08.164223+0000","last_unstale":"2024-03-24T00:39:08.178945+0000","last_undegraded":"2024-03-24T00:39:08.178945+0000","last_fullsized":"2024-03-24T00:39:08.178945+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:06:54.840723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.148305+0000","last_change":"2024-03-24T00:39:07.145520+0000","last_active":"2024-03-24T00:39:08.148305+0000","last_peered":"2024-03-24T00:39:08.148305+0000","last_clean":"2024-03-24T00:39:08.148305+0000","last_became_active":"2024-03-24T00:39:07.145189+0000","last_became_peered":"2024-03-24T00:39:07.145189+0000","last_unstale":"2024-03-24T00:39:08.148305+0000","last_undegraded":"2024-03-24T00:39:08.148305+0000","last_fullsized":"2024-03-24T00:39:08.148305+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:12:38.387067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180002+0000","last_change":"2024-03-24T00:39:08.167748+0000","last_active":"2024-03-24T00:39:08.180002+0000","last_peered":"2024-03-24T00:39:08.180002+0000","last_clean":"2024-03-24T00:39:08.180002+0000","last_became_active":"2024-03-24T00:39:08.167375+0000","last_became_peered":"2024-03-24T00:39:08.167375+0000","last_unstale":"2024-03-24T00:39:08.180002+0000","last_undegraded":"2024-03-24T00:39:08.180002+0000","last_fullsized":"2024-03-24T00:39:08.180002+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:10:37.340378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"118'8","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.188562+0000","last_change":"2024-03-24T00:39:07.153365+0000","last_active":"2024-03-24T00:39:08.188562+0000","last_peered":"2024-03-24T00:39:08.188562+0000","last_clean":"2024-03-24T00:39:08.188562+0000","last_became_active":"2024-03-24T00:39:07.153043+0000","last_became_peered":"2024-03-24T00:39:07.153043+0000","last_unstale":"2024-03-24T00:39:08.188562+0000","last_undegraded":"2024-03-24T00:39:08.188562+0000","last_fullsized":"2024-03-24T00:39:08.188562+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:44:31.582035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180524+0000","last_change":"2024-03-24T00:39:08.168562+0000","last_active":"2024-03-24T00:39:08.180524+0000","last_peered":"2024-03-24T00:39:08.180524+0000","last_clean":"2024-03-24T00:39:08.180524+0000","last_became_active":"2024-03-24T00:39:08.168284+0000","last_became_peered":"2024-03-24T00:39:08.168284+0000","last_unstale":"2024-03-24T00:39:08.180524+0000","last_undegraded":"2024-03-24T00:39:08.180524+0000","last_fullsized":"2024-03-24T00:39:08.180524+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:48:22.697641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"118'1","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181236+0000","last_change":"2024-03-24T00:39:07.146935+0000","last_active":"2024-03-24T00:39:08.181236+0000","last_peered":"2024-03-24T00:39:08.181236+0000","last_clean":"2024-03-24T00:39:08.181236+0000","last_became_active":"2024-03-24T00:39:07.146613+0000","last_became_peered":"2024-03-24T00:39:07.146613+0000","last_unstale":"2024-03-24T00:39:08.181236+0000","last_undegraded":"2024-03-24T00:39:08.181236+0000","last_fullsized":"2024-03-24T00:39:08.181236+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:08:26.722631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.185691+0000","last_change":"2024-03-24T00:39:08.184448+0000","last_active":"2024-03-24T00:39:08.185691+0000","last_peered":"2024-03-24T00:39:08.185691+0000","last_clean":"2024-03-24T00:39:08.185691+0000","last_became_active":"2024-03-24T00:39:08.184143+0000","last_became_peered":"2024-03-24T00:39:08.184143+0000","last_unstale":"2024-03-24T00:39:08.185691+0000","last_undegraded":"2024-03-24T00:39:08.185691+0000","last_fullsized":"2024-03-24T00:39:08.185691+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:03.669831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"118'2","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.182070+0000","last_change":"2024-03-24T00:39:07.156381+0000","last_active":"2024-03-24T00:39:08.182070+0000","last_peered":"2024-03-24T00:39:08.182070+0000","last_clean":"2024-03-24T00:39:08.182070+0000","last_became_active":"2024-03-24T00:39:07.155397+0000","last_became_peered":"2024-03-24T00:39:07.155397+0000","last_unstale":"2024-03-24T00:39:08.182070+0000","last_undegraded":"2024-03-24T00:39:08.182070+0000","last_fullsized":"2024-03-24T00:39:08.182070+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:03:03.981709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.187185+0000","last_change":"2024-03-24T00:39:08.184614+0000","last_active":"2024-03-24T00:39:08.187185+0000","last_peered":"2024-03-24T00:39:08.187185+0000","last_clean":"2024-03-24T00:39:08.187185+0000","last_became_active":"2024-03-24T00:39:08.182535+0000","last_became_peered":"2024-03-24T00:39:08.182535+0000","last_unstale":"2024-03-24T00:39:08.187185+0000","last_undegraded":"2024-03-24T00:39:08.187185+0000","last_fullsized":"2024-03-24T00:39:08.187185+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:31.763862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"118'1","reported_seq":13,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.184774+0000","last_change":"2024-03-24T00:39:07.159788+0000","last_active":"2024-03-24T00:39:08.184774+0000","last_peered":"2024-03-24T00:39:08.184774+0000","last_clean":"2024-03-24T00:39:08.184774+0000","last_became_active":"2024-03-24T00:39:07.159434+0000","last_became_peered":"2024-03-24T00:39:07.159434+0000","last_unstale":"2024-03-24T00:39:08.184774+0000","last_undegraded":"2024-03-24T00:39:08.184774+0000","last_fullsized":"2024-03-24T00:39:08.184774+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:24:07.059890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180699+0000","last_change":"2024-03-24T00:39:08.175017+0000","last_active":"2024-03-24T00:39:08.180699+0000","last_peered":"2024-03-24T00:39:08.180699+0000","last_clean":"2024-03-24T00:39:08.180699+0000","last_became_active":"2024-03-24T00:39:08.174730+0000","last_became_peered":"2024-03-24T00:39:08.174730+0000","last_unstale":"2024-03-24T00:39:08.180699+0000","last_undegraded":"2024-03-24T00:39:08.180699+0000","last_fullsized":"2024-03-24T00:39:08.180699+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:44:32.589736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.158259+0000","last_change":"2024-03-24T00:39:07.156738+0000","last_active":"2024-03-24T00:39:08.158259+0000","last_peered":"2024-03-24T00:39:08.158259+0000","last_clean":"2024-03-24T00:39:08.158259+0000","last_became_active":"2024-03-24T00:39:07.156411+0000","last_became_peered":"2024-03-24T00:39:07.156411+0000","last_unstale":"2024-03-24T00:39:08.158259+0000","last_undegraded":"2024-03-24T00:39:08.158259+0000","last_fullsized":"2024-03-24T00:39:08.158259+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:07:31.866801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.183352+0000","last_change":"2024-03-24T00:39:08.180333+0000","last_active":"2024-03-24T00:39:08.183352+0000","last_peered":"2024-03-24T00:39:08.183352+0000","last_clean":"2024-03-24T00:39:08.183352+0000","last_became_active":"2024-03-24T00:39:08.179894+0000","last_became_peered":"2024-03-24T00:39:08.179894+0000","last_unstale":"2024-03-24T00:39:08.183352+0000","last_undegraded":"2024-03-24T00:39:08.183352+0000","last_fullsized":"2024-03-24T00:39:08.183352+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:16:26.647757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"118'2","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.187073+0000","last_change":"2024-03-24T00:39:07.147319+0000","last_active":"2024-03-24T00:39:08.187073+0000","last_peered":"2024-03-24T00:39:08.187073+0000","last_clean":"2024-03-24T00:39:08.187073+0000","last_became_active":"2024-03-24T00:39:07.144597+0000","last_became_peered":"2024-03-24T00:39:07.144597+0000","last_unstale":"2024-03-24T00:39:08.187073+0000","last_undegraded":"2024-03-24T00:39:08.187073+0000","last_fullsized":"2024-03-24T00:39:08.187073+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:50.557232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.171774+0000","last_change":"2024-03-24T00:39:08.169263+0000","last_active":"2024-03-24T00:39:08.171774+0000","last_peered":"2024-03-24T00:39:08.171774+0000","last_clean":"2024-03-24T00:39:08.171774+0000","last_became_active":"2024-03-24T00:39:08.167859+0000","last_became_peered":"2024-03-24T00:39:08.167859+0000","last_unstale":"2024-03-24T00:39:08.171774+0000","last_undegraded":"2024-03-24T00:39:08.171774+0000","last_fullsized":"2024-03-24T00:39:08.171774+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:17:47.898414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.171595+0000","last_change":"2024-03-24T00:39:07.152044+0000","last_active":"2024-03-24T00:39:08.171595+0000","last_peered":"2024-03-24T00:39:08.171595+0000","last_clean":"2024-03-24T00:39:08.171595+0000","last_became_active":"2024-03-24T00:39:07.151431+0000","last_became_peered":"2024-03-24T00:39:07.151431+0000","last_unstale":"2024-03-24T00:39:08.171595+0000","last_undegraded":"2024-03-24T00:39:08.171595+0000","last_fullsized":"2024-03-24T00:39:08.171595+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:34:55.724168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.180992+0000","last_change":"2024-03-24T00:39:08.178791+0000","last_active":"2024-03-24T00:39:08.180992+0000","last_peered":"2024-03-24T00:39:08.180992+0000","last_clean":"2024-03-24T00:39:08.180992+0000","last_became_active":"2024-03-24T00:39:08.171766+0000","last_became_peered":"2024-03-24T00:39:08.171766+0000","last_unstale":"2024-03-24T00:39:08.180992+0000","last_undegraded":"2024-03-24T00:39:08.180992+0000","last_fullsized":"2024-03-24T00:39:08.180992+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:49:23.192198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.171287+0000","last_change":"2024-03-24T00:39:07.154626+0000","last_active":"2024-03-24T00:39:08.171287+0000","last_peered":"2024-03-24T00:39:08.171287+0000","last_clean":"2024-03-24T00:39:08.171287+0000","last_became_active":"2024-03-24T00:39:07.154281+0000","last_became_peered":"2024-03-24T00:39:07.154281+0000","last_unstale":"2024-03-24T00:39:08.171287+0000","last_undegraded":"2024-03-24T00:39:08.171287+0000","last_fullsized":"2024-03-24T00:39:08.171287+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:43.804017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.174851+0000","last_change":"2024-03-24T00:39:08.167313+0000","last_active":"2024-03-24T00:39:08.174851+0000","last_peered":"2024-03-24T00:39:08.174851+0000","last_clean":"2024-03-24T00:39:08.174851+0000","last_became_active":"2024-03-24T00:39:08.166747+0000","last_became_peered":"2024-03-24T00:39:08.166747+0000","last_unstale":"2024-03-24T00:39:08.174851+0000","last_undegraded":"2024-03-24T00:39:08.174851+0000","last_fullsized":"2024-03-24T00:39:08.174851+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:39:55.017873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.153101+0000","last_change":"2024-03-24T00:39:07.156122+0000","last_active":"2024-03-24T00:39:08.153101+0000","last_peered":"2024-03-24T00:39:08.153101+0000","last_clean":"2024-03-24T00:39:08.153101+0000","last_became_active":"2024-03-24T00:39:07.155822+0000","last_became_peered":"2024-03-24T00:39:07.155822+0000","last_unstale":"2024-03-24T00:39:08.153101+0000","last_undegraded":"2024-03-24T00:39:08.153101+0000","last_fullsized":"2024-03-24T00:39:08.153101+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:23:37.082015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.173420+0000","last_change":"2024-03-24T00:39:08.165338+0000","last_active":"2024-03-24T00:39:08.173420+0000","last_peered":"2024-03-24T00:39:08.173420+0000","last_clean":"2024-03-24T00:39:08.173420+0000","last_became_active":"2024-03-24T00:39:08.165042+0000","last_became_peered":"2024-03-24T00:39:08.165042+0000","last_unstale":"2024-03-24T00:39:08.173420+0000","last_undegraded":"2024-03-24T00:39:08.173420+0000","last_fullsized":"2024-03-24T00:39:08.173420+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:56:32.441579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"118'2","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.181855+0000","last_change":"2024-03-24T00:39:07.149981+0000","last_active":"2024-03-24T00:39:08.181855+0000","last_peered":"2024-03-24T00:39:08.181855+0000","last_clean":"2024-03-24T00:39:08.181855+0000","last_became_active":"2024-03-24T00:39:07.149712+0000","last_became_peered":"2024-03-24T00:39:07.149712+0000","last_unstale":"2024-03-24T00:39:08.181855+0000","last_undegraded":"2024-03-24T00:39:08.181855+0000","last_fullsized":"2024-03-24T00:39:08.181855+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:45:00.603021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.176030+0000","last_change":"2024-03-24T00:39:08.169399+0000","last_active":"2024-03-24T00:39:08.176030+0000","last_peered":"2024-03-24T00:39:08.176030+0000","last_clean":"2024-03-24T00:39:08.176030+0000","last_became_active":"2024-03-24T00:39:08.169119+0000","last_became_peered":"2024-03-24T00:39:08.169119+0000","last_unstale":"2024-03-24T00:39:08.176030+0000","last_undegraded":"2024-03-24T00:39:08.176030+0000","last_fullsized":"2024-03-24T00:39:08.176030+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:39:28.087874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.153115+0000","last_change":"2024-03-24T00:39:07.153597+0000","last_active":"2024-03-24T00:39:08.153115+0000","last_peered":"2024-03-24T00:39:08.153115+0000","last_clean":"2024-03-24T00:39:08.153115+0000","last_became_active":"2024-03-24T00:39:07.153075+0000","last_became_peered":"2024-03-24T00:39:07.153075+0000","last_unstale":"2024-03-24T00:39:08.153115+0000","last_undegraded":"2024-03-24T00:39:08.153115+0000","last_fullsized":"2024-03-24T00:39:08.153115+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:43.804017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.170143+0000","last_change":"2024-03-24T00:39:08.165180+0000","last_active":"2024-03-24T00:39:08.170143+0000","last_peered":"2024-03-24T00:39:08.170143+0000","last_clean":"2024-03-24T00:39:08.170143+0000","last_became_active":"2024-03-24T00:39:08.164881+0000","last_became_peered":"2024-03-24T00:39:08.164881+0000","last_unstale":"2024-03-24T00:39:08.170143+0000","last_undegraded":"2024-03-24T00:39:08.170143+0000","last_fullsized":"2024-03-24T00:39:08.170143+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:12:03.669831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.150587+0000","last_change":"2024-03-24T00:39:07.150130+0000","last_active":"2024-03-24T00:39:08.150587+0000","last_peered":"2024-03-24T00:39:08.150587+0000","last_clean":"2024-03-24T00:39:08.150587+0000","last_became_active":"2024-03-24T00:39:07.149798+0000","last_became_peered":"2024-03-24T00:39:07.149798+0000","last_unstale":"2024-03-24T00:39:08.150587+0000","last_undegraded":"2024-03-24T00:39:08.150587+0000","last_fullsized":"2024-03-24T00:39:08.150587+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:45:50.557232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.171353+0000","last_change":"2024-03-24T00:39:08.166623+0000","last_active":"2024-03-24T00:39:08.171353+0000","last_peered":"2024-03-24T00:39:08.171353+0000","last_clean":"2024-03-24T00:39:08.171353+0000","last_became_active":"2024-03-24T00:39:08.166356+0000","last_became_peered":"2024-03-24T00:39:08.166356+0000","last_unstale":"2024-03-24T00:39:08.171353+0000","last_undegraded":"2024-03-24T00:39:08.171353+0000","last_fullsized":"2024-03-24T00:39:08.171353+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:17:55.856314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.155012+0000","last_change":"2024-03-24T00:39:07.157488+0000","last_active":"2024-03-24T00:39:08.155012+0000","last_peered":"2024-03-24T00:39:08.155012+0000","last_clean":"2024-03-24T00:39:08.155012+0000","last_became_active":"2024-03-24T00:39:07.155634+0000","last_became_peered":"2024-03-24T00:39:07.155634+0000","last_unstale":"2024-03-24T00:39:08.155012+0000","last_undegraded":"2024-03-24T00:39:08.155012+0000","last_fullsized":"2024-03-24T00:39:08.155012+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:39:28.252944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.165589+0000","last_change":"2024-03-24T00:39:08.162717+0000","last_active":"2024-03-24T00:39:08.165589+0000","last_peered":"2024-03-24T00:39:08.165589+0000","last_clean":"2024-03-24T00:39:08.165589+0000","last_became_active":"2024-03-24T00:39:08.158902+0000","last_became_peered":"2024-03-24T00:39:08.158902+0000","last_unstale":"2024-03-24T00:39:08.165589+0000","last_undegraded":"2024-03-24T00:39:08.165589+0000","last_fullsized":"2024-03-24T00:39:08.165589+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:10:11.822123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.149818+0000","last_change":"2024-03-24T00:39:07.148740+0000","last_active":"2024-03-24T00:39:08.149818+0000","last_peered":"2024-03-24T00:39:08.149818+0000","last_clean":"2024-03-24T00:39:08.149818+0000","last_became_active":"2024-03-24T00:39:07.148295+0000","last_became_peered":"2024-03-24T00:39:07.148295+0000","last_unstale":"2024-03-24T00:39:08.149818+0000","last_undegraded":"2024-03-24T00:39:08.149818+0000","last_fullsized":"2024-03-24T00:39:08.149818+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:57:39.605387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.178618+0000","last_change":"2024-03-24T00:39:08.169371+0000","last_active":"2024-03-24T00:39:08.178618+0000","last_peered":"2024-03-24T00:39:08.178618+0000","last_clean":"2024-03-24T00:39:08.178618+0000","last_became_active":"2024-03-24T00:39:08.169023+0000","last_became_peered":"2024-03-24T00:39:08.169023+0000","last_unstale":"2024-03-24T00:39:08.178618+0000","last_undegraded":"2024-03-24T00:39:08.178618+0000","last_fullsized":"2024-03-24T00:39:08.178618+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:42.707367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.151244+0000","last_change":"2024-03-24T00:39:07.158892+0000","last_active":"2024-03-24T00:39:08.151244+0000","last_peered":"2024-03-24T00:39:08.151244+0000","last_clean":"2024-03-24T00:39:08.151244+0000","last_became_active":"2024-03-24T00:39:07.158516+0000","last_became_peered":"2024-03-24T00:39:07.158516+0000","last_unstale":"2024-03-24T00:39:08.151244+0000","last_undegraded":"2024-03-24T00:39:08.151244+0000","last_fullsized":"2024-03-24T00:39:08.151244+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:16:23.871407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.186421+0000","last_change":"2024-03-24T00:39:08.181520+0000","last_active":"2024-03-24T00:39:08.186421+0000","last_peered":"2024-03-24T00:39:08.186421+0000","last_clean":"2024-03-24T00:39:08.186421+0000","last_became_active":"2024-03-24T00:39:08.181253+0000","last_became_peered":"2024-03-24T00:39:08.181253+0000","last_unstale":"2024-03-24T00:39:08.186421+0000","last_undegraded":"2024-03-24T00:39:08.186421+0000","last_fullsized":"2024-03-24T00:39:08.186421+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:39:55.017873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.171450+0000","last_change":"2024-03-24T00:39:07.146864+0000","last_active":"2024-03-24T00:39:08.171450+0000","last_peered":"2024-03-24T00:39:08.171450+0000","last_clean":"2024-03-24T00:39:08.171450+0000","last_became_active":"2024-03-24T00:39:07.146537+0000","last_became_peered":"2024-03-24T00:39:07.146537+0000","last_unstale":"2024-03-24T00:39:08.171450+0000","last_undegraded":"2024-03-24T00:39:08.171450+0000","last_fullsized":"2024-03-24T00:39:08.171450+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:56:41.871203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.167673+0000","last_change":"2024-03-24T00:39:08.160739+0000","last_active":"2024-03-24T00:39:08.167673+0000","last_peered":"2024-03-24T00:39:08.167673+0000","last_clean":"2024-03-24T00:39:08.167673+0000","last_became_active":"2024-03-24T00:39:08.160228+0000","last_became_peered":"2024-03-24T00:39:08.160228+0000","last_unstale":"2024-03-24T00:39:08.167673+0000","last_undegraded":"2024-03-24T00:39:08.167673+0000","last_fullsized":"2024-03-24T00:39:08.167673+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:11:42.707367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.153267+0000","last_change":"2024-03-24T00:39:07.154983+0000","last_active":"2024-03-24T00:39:08.153267+0000","last_peered":"2024-03-24T00:39:08.153267+0000","last_clean":"2024-03-24T00:39:08.153267+0000","last_became_active":"2024-03-24T00:39:07.154648+0000","last_became_peered":"2024-03-24T00:39:07.154648+0000","last_unstale":"2024-03-24T00:39:08.153267+0000","last_undegraded":"2024-03-24T00:39:08.153267+0000","last_fullsized":"2024-03-24T00:39:08.153267+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:28.228194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.174923+0000","last_change":"2024-03-24T00:39:08.170158+0000","last_active":"2024-03-24T00:39:08.174923+0000","last_peered":"2024-03-24T00:39:08.174923+0000","last_clean":"2024-03-24T00:39:08.174923+0000","last_became_active":"2024-03-24T00:39:08.168582+0000","last_became_peered":"2024-03-24T00:39:08.168582+0000","last_unstale":"2024-03-24T00:39:08.174923+0000","last_undegraded":"2024-03-24T00:39:08.174923+0000","last_fullsized":"2024-03-24T00:39:08.174923+0000","mapping_epoch":117,"log_start":"0'0","ondisk_log_start":"0'0","created":117,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:07.125390+0000","last_clean_scrub_stamp":"2024-03-24T00:39:07.125390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:59.444846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"118'2","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.188168+0000","last_change":"2024-03-24T00:39:07.155465+0000","last_active":"2024-03-24T00:39:08.188168+0000","last_peered":"2024-03-24T00:39:08.188168+0000","last_clean":"2024-03-24T00:39:08.188168+0000","last_became_active":"2024-03-24T00:39:07.155196+0000","last_became_peered":"2024-03-24T00:39:07.155196+0000","last_unstale":"2024-03-24T00:39:08.188168+0000","last_undegraded":"2024-03-24T00:39:08.188168+0000","last_fullsized":"2024-03-24T00:39:08.188168+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:39:06.117689+0000","last_clean_scrub_stamp":"2024-03-24T00:39:06.117689+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:05:35.119226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":369,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-24T00:39:08.150210+0000","last_change":"2024-03-24T00:30:55.181123+0000","last_active":"2024-03-24T00:39:08.150210+0000","last_peered":"2024-03-24T00:39:08.150210+0000","last_clean":"2024-03-24T00:39:08.150210+0000","last_became_active":"2024-03-24T00:30:55.179894+0000","last_became_peered":"2024-03-24T00:30:55.179894+0000","last_unstale":"2024-03-24T00:39:08.150210+0000","last_undegraded":"2024-03-24T00:39:08.150210+0000","last_fullsized":"2024-03-24T00:39:08.150210+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:30:53.992862+0000","last_clean_scrub_stamp":"2024-03-24T00:30:53.992862+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:12.376732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574949,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37352,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334488,"statfs":{"total":96636764160,"available":96598515712,"internally_reserved":0,"allocated":2056192,"data_stored":1228927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574950,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40960,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330880,"statfs":{"total":96636764160,"available":96594821120,"internally_reserved":0,"allocated":1486848,"data_stored":646946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574950,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36772,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335068,"statfs":{"total":96636764160,"available":96599109632,"internally_reserved":0,"allocated":1462272,"data_stored":638559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574950,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50672,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321168,"statfs":{"total":96636764160,"available":96584876032,"internally_reserved":0,"allocated":1478656,"data_stored":645404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72868,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94298972,"statfs":{"total":96636764160,"available":96562147328,"internally_reserved":0,"allocated":1462272,"data_stored":638559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574950,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40928,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330912,"statfs":{"total":96636764160,"available":96594853888,"internally_reserved":0,"allocated":1462272,"data_stored":638559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45100,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326740,"statfs":{"total":96636764160,"available":96590581760,"internally_reserved":0,"allocated":1462272,"data_stored":638559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574949,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47080,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324760,"statfs":{"total":96636764160,"available":96588554240,"internally_reserved":0,"allocated":2056192,"data_stored":1228927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:39:12.768 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:39:12.768 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-24T00:39:12.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.948+0000 7f597b400640 1 Processor -- start 2024-03-24T00:39:12.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.948+0000 7f597b400640 1 -- start start 2024-03-24T00:39:12.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f597b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f59740d2270 0x7f5974070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f597b400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f597406c950 0x7f5974161ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f597b400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f597406d0f0 con 0x7f597415df80 2024-03-24T00:39:12.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f597b400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f597406b890 con 0x7f597406c950 2024-03-24T00:39:12.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f597b400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f597406b280 con 0x7f59740d2270 2024-03-24T00:39:12.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f5979400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f597406c950 0x7f5974161ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f5979400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f597406c950 0x7f5974161ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49470/0 (socket says 172.21.15.40:49470) 2024-03-24T00:39:12.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f5979400640 1 -- 172.21.15.40:0/2716072762 learned_addr learned my addr 172.21.15.40:0/2716072762 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:12.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f5973e00640 1 --2- 172.21.15.40:0/2716072762 >> v2:172.21.15.53:3300/0 conn(0x7f59740d2270 0x7f5974070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f5979400640 1 -- 172.21.15.40:0/2716072762 >> v2:172.21.15.53:3300/0 conn(0x7f59740d2270 msgr2=0x7f5974070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:12.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f5979400640 1 --2- 172.21.15.40:0/2716072762 >> v2:172.21.15.53:3300/0 conn(0x7f59740d2270 0x7f5974070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f5979400640 1 -- 172.21.15.40:0/2716072762 >> v1:172.21.15.53:6789/0 conn(0x7f597415df80 legacy=0x7f59740d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.952+0000 7f5979400640 1 -- 172.21.15.40:0/2716072762 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597406ce10 con 0x7f597406c950 2024-03-24T00:39:12.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f5979400640 1 --2- 172.21.15.40:0/2716072762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f597406c950 0x7f5974161ad0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f5974074ee0 tx=0x7f596c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca7312cd67e5f80e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:12.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f5973400640 1 -- 172.21.15.40:0/2716072762 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c0a5840 con 0x7f597406c950 2024-03-24T00:39:12.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f5973400640 1 -- 172.21.15.40:0/2716072762 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c0a59e0 con 0x7f597406c950 2024-03-24T00:39:12.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f5973400640 1 -- 172.21.15.40:0/2716072762 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c0a5db0 con 0x7f597406c950 2024-03-24T00:39:12.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f597b400640 1 -- 172.21.15.40:0/2716072762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f597406c950 msgr2=0x7f5974161ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f597b400640 1 --2- 172.21.15.40:0/2716072762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f597406c950 0x7f5974161ad0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f5974074ee0 tx=0x7f596c01c9c0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.960 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f597b400640 1 -- 172.21.15.40:0/2716072762 shutdown_connections 2024-03-24T00:39:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f597b400640 1 --2- 172.21.15.40:0/2716072762 >> v2:172.21.15.53:3300/0 conn(0x7f59740d2270 0x7f5974070ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f597b400640 1 --2- 172.21.15.40:0/2716072762 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f597406c950 0x7f5974161ad0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f597b400640 1 -- 172.21.15.40:0/2716072762 >> 172.21.15.40:0/2716072762 conn(0x7f597406ecf0 msgr2=0x7f597406f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f597b400640 1 -- 172.21.15.40:0/2716072762 shutdown_connections 2024-03-24T00:39:12.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f597b400640 1 -- 172.21.15.40:0/2716072762 wait complete. 2024-03-24T00:39:12.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f597b400640 1 Processor -- start 2024-03-24T00:39:12.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.956+0000 7f597b400640 1 -- start start 2024-03-24T00:39:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f597b400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597406c950 0x7f59741554b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f597b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59740d2270 0x7f59740c7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f597b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f597415df80 0x7f5974156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f597b400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597416af90 con 0x7f59740d2270 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f597b400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597416ae10 con 0x7f597415df80 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f597b400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597416b110 con 0x7f597406c950 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5973e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59740d2270 0x7f59740c7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5973e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59740d2270 0x7f59740c7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49472/0 (socket says 172.21.15.40:49472) 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5973e00640 1 -- 172.21.15.40:0/2939690224 learned_addr learned my addr 172.21.15.40:0/2939690224 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5978a00640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597406c950 0x7f59741554b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5979400640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f597415df80 0x7f5974156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5973e00640 1 -- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597406c950 msgr2=0x7f59741554b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5973e00640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597406c950 0x7f59741554b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5973e00640 1 -- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f597415df80 msgr2=0x7f5974156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5973e00640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f597415df80 0x7f5974156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5973e00640 1 -- 172.21.15.40:0/2939690224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59740ade40 con 0x7f59740d2270 2024-03-24T00:39:12.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5978a00640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597406c950 0x7f59741554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5973e00640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59740d2270 0x7f59740c7a10 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f596000e930 tx=0x7f596000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5970c00640 1 -- 172.21.15.40:0/2939690224 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960017070 con 0x7f59740d2270 2024-03-24T00:39:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5970c00640 1 -- 172.21.15.40:0/2939690224 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596000ca00 con 0x7f59740d2270 2024-03-24T00:39:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5970c00640 1 -- 172.21.15.40:0/2939690224 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596000cdb0 con 0x7f59740d2270 2024-03-24T00:39:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f597b400640 1 -- 172.21.15.40:0/2939690224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59740ae120 con 0x7f59740d2270 2024-03-24T00:39:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f597b400640 1 -- 172.21.15.40:0/2939690224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59740ae6e0 con 0x7f59740d2270 2024-03-24T00:39:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f597b400640 1 -- 172.21.15.40:0/2939690224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5974068170 con 0x7f59740d2270 2024-03-24T00:39:12.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.960+0000 7f5970c00640 1 -- 172.21.15.40:0/2939690224 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f596001a070 con 0x7f59740d2270 2024-03-24T00:39:12.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.964+0000 7f5970c00640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5954067660 0x7f5954069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:12.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.964+0000 7f5970c00640 1 -- 172.21.15.40:0/2939690224 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5960085bc0 con 0x7f59740d2270 2024-03-24T00:39:12.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.964+0000 7f5978a00640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5954067660 0x7f5954069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:12.968 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.964+0000 7f5978a00640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5954067660 0x7f5954069b20 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5974156ca0 tx=0x7f5968009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:12.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:12.972+0000 7f5970c00640 1 -- 172.21.15.40:0/2939690224 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f59600562b0 con 0x7f59740d2270 2024-03-24T00:39:13.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.204+0000 7f597b400640 1 -- 172.21.15.40:0/2939690224 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f59740cf9d0 con 0x7f59740d2270 2024-03-24T00:39:13.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.204+0000 7f5970c00640 1 -- 172.21.15.40:0/2939690224 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+147 (secure 0 0 0) 0x7f596005a350 con 0x7f59740d2270 2024-03-24T00:39:13.210 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-03-24T00:39:13.210 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQARdP9lPEmpKhAAgTVLJeDfPEpJLzxBy2S6Hw== 2024-03-24T00:39:13.210 INFO:teuthology.orchestra.run.smithi040.stdout: caps mds = "allow" 2024-03-24T00:39:13.211 INFO:teuthology.orchestra.run.smithi040.stdout: caps mon = "allow r" 2024-03-24T00:39:13.211 INFO:teuthology.orchestra.run.smithi040.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-24T00:39:13.211 INFO:teuthology.orchestra.run.smithi040.stderr:updated caps for client.0 2024-03-24T00:39:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.208+0000 7f597b400640 1 -- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5954067660 msgr2=0x7f5954069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.208+0000 7f597b400640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5954067660 0x7f5954069b20 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5974156ca0 tx=0x7f5968009040 comp rx=0 tx=0).stop 2024-03-24T00:39:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.208+0000 7f597b400640 1 -- 172.21.15.40:0/2939690224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59740d2270 msgr2=0x7f59740c7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.208+0000 7f597b400640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59740d2270 0x7f59740c7a10 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f596000e930 tx=0x7f596000edf0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.208+0000 7f597b400640 1 -- 172.21.15.40:0/2939690224 shutdown_connections 2024-03-24T00:39:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.208+0000 7f597b400640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5954067660 0x7f5954069b20 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.208+0000 7f597b400640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f597415df80 0x7f5974156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.208+0000 7f597b400640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f59740d2270 0x7f59740c7a10 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.208+0000 7f597b400640 1 --2- 172.21.15.40:0/2939690224 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597406c950 0x7f59741554b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.208+0000 7f597b400640 1 -- 172.21.15.40:0/2939690224 >> 172.21.15.40:0/2939690224 conn(0x7f597406ecf0 msgr2=0x7f5974160c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.208+0000 7f597b400640 1 -- 172.21.15.40:0/2939690224 shutdown_connections 2024-03-24T00:39:13.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.208+0000 7f597b400640 1 -- 172.21.15.40:0/2939690224 wait complete. 2024-03-24T00:39:13.232 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.368+0000 7f6590000640 1 Processor -- start 2024-03-24T00:39:13.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.368+0000 7f6590000640 1 -- start start 2024-03-24T00:39:13.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.368+0000 7f6590000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6588141bc0 0x7f65880ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.368+0000 7f6590000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f65880ce630 0x7f6588071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.368+0000 7f6590000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6588070e00 con 0x7f658813ec80 2024-03-24T00:39:13.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.368+0000 7f6590000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65880cee20 con 0x7f65880ce630 2024-03-24T00:39:13.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.368+0000 7f6590000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65880cf100 con 0x7f6588141bc0 2024-03-24T00:39:13.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.368+0000 7f658e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f65880ce630 0x7f6588071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.368+0000 7f658e000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f65880ce630 0x7f6588071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49482/0 (socket says 172.21.15.40:49482) 2024-03-24T00:39:13.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.368+0000 7f658e000640 1 -- 172.21.15.40:0/2336801014 learned_addr learned my addr 172.21.15.40:0/2336801014 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.368+0000 7f658cc00640 1 --2- 172.21.15.40:0/2336801014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6588141bc0 0x7f65880ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f658e000640 1 -- 172.21.15.40:0/2336801014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6588141bc0 msgr2=0x7f65880ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f658e000640 1 --2- 172.21.15.40:0/2336801014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6588141bc0 0x7f65880ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f658e000640 1 -- 172.21.15.40:0/2336801014 >> v1:172.21.15.53:6789/0 conn(0x7f658813ec80 legacy=0x7f65881414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f658e000640 1 -- 172.21.15.40:0/2336801014 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65880cf3e0 con 0x7f65880ce630 2024-03-24T00:39:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f658e000640 1 --2- 172.21.15.40:0/2336801014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65880ce630 0x7f6588071300 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f6580009e50 tx=0x7f658001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a97f9049462a2b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f6587e00640 1 -- 172.21.15.40:0/2336801014 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65800a5920 con 0x7f65880ce630 2024-03-24T00:39:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f6587e00640 1 -- 172.21.15.40:0/2336801014 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65800a5ac0 con 0x7f65880ce630 2024-03-24T00:39:13.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f6587e00640 1 -- 172.21.15.40:0/2336801014 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65800ad3f0 con 0x7f65880ce630 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f6590000640 1 -- 172.21.15.40:0/2336801014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65880ce630 msgr2=0x7f6588071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f6590000640 1 --2- 172.21.15.40:0/2336801014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65880ce630 0x7f6588071300 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f6580009e50 tx=0x7f658001ca30 comp rx=0 tx=0).stop 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f6590000640 1 -- 172.21.15.40:0/2336801014 shutdown_connections 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f6590000640 1 --2- 172.21.15.40:0/2336801014 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6588141bc0 0x7f65880ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f6590000640 1 --2- 172.21.15.40:0/2336801014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f65880ce630 0x7f6588071300 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f6590000640 1 -- 172.21.15.40:0/2336801014 >> 172.21.15.40:0/2336801014 conn(0x7f65880b17b0 msgr2=0x7f65880b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.372+0000 7f6590000640 1 -- 172.21.15.40:0/2336801014 shutdown_connections 2024-03-24T00:39:13.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.376+0000 7f6590000640 1 -- 172.21.15.40:0/2336801014 wait complete. 2024-03-24T00:39:13.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.376+0000 7f6590000640 1 Processor -- start 2024-03-24T00:39:13.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.376+0000 7f6590000640 1 -- start start 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6590000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65880ce630 0x7f6588141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6590000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f658813ec80 0x7f6588137ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6590000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6588141bc0 0x7f65880af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6590000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65880723e0 con 0x7f658813ec80 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6590000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6588072260 con 0x7f6588141bc0 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6590000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6588072560 con 0x7f65880ce630 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f658cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f658813ec80 0x7f6588137ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f658cc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f658813ec80 0x7f6588137ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49486/0 (socket says 172.21.15.40:49486) 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f658cc00640 1 -- 172.21.15.40:0/3177560317 learned_addr learned my addr 172.21.15.40:0/3177560317 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f658d600640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65880ce630 0x7f6588141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f658e000640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6588141bc0 0x7f65880af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f658cc00640 1 -- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65880ce630 msgr2=0x7f6588141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f658cc00640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65880ce630 0x7f6588141030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f658cc00640 1 -- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6588141bc0 msgr2=0x7f65880af960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f658cc00640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6588141bc0 0x7f65880af960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f658cc00640 1 -- 172.21.15.40:0/3177560317 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6588139b90 con 0x7f658813ec80 2024-03-24T00:39:13.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f658e000640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6588141bc0 0x7f65880af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:13.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f658cc00640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f658813ec80 0x7f6588137ee0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f657400d800 tx=0x7f657400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6585600640 1 -- 172.21.15.40:0/3177560317 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657400b660 con 0x7f658813ec80 2024-03-24T00:39:13.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6585600640 1 -- 172.21.15.40:0/3177560317 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6574019070 con 0x7f658813ec80 2024-03-24T00:39:13.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6585600640 1 -- 172.21.15.40:0/3177560317 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657400ba00 con 0x7f658813ec80 2024-03-24T00:39:13.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6590000640 1 -- 172.21.15.40:0/3177560317 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6588139e30 con 0x7f658813ec80 2024-03-24T00:39:13.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6590000640 1 -- 172.21.15.40:0/3177560317 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658813a3f0 con 0x7f658813ec80 2024-03-24T00:39:13.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6590000640 1 -- 172.21.15.40:0/3177560317 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6588068170 con 0x7f658813ec80 2024-03-24T00:39:13.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.380+0000 7f6585600640 1 -- 172.21.15.40:0/3177560317 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f657400bba0 con 0x7f658813ec80 2024-03-24T00:39:13.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.384+0000 7f6585600640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6568067660 0x7f6568069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:13.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.384+0000 7f6585600640 1 -- 172.21.15.40:0/3177560317 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6574085c60 con 0x7f658813ec80 2024-03-24T00:39:13.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.384+0000 7f658d600640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6568067660 0x7f6568069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:13.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.384+0000 7f658d600640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6568067660 0x7f6568069b20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f657c001810 tx=0x7f657c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:13.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.388+0000 7f6585600640 1 -- 172.21.15.40:0/3177560317 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f65740568d0 con 0x7f658813ec80 2024-03-24T00:39:13.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.584+0000 7f6590000640 1 -- 172.21.15.40:0/3177560317 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6588072b90 con 0x7f658813ec80 2024-03-24T00:39:13.589 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.584+0000 7f6585600640 1 -- 172.21.15.40:0/3177560317 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+4982 (secure 0 0 0) 0x7f65740561d0 con 0x7f658813ec80 2024-03-24T00:39:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:13.589 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":128,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1582567544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1582567544},{"type":"v1","addr":"172.21.15.53:6837","nonce":1582567544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8563,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2255755838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2255755838},{"type":"v1","addr":"172.21.15.53:6836","nonce":2255755838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8874,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.40:6833/4273998586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4273998586},{"type":"v1","addr":"172.21.15.40:6833","nonce":4273998586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:39:07.531884+0000","modified":"2024-03-24T00:39:10.561787+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8877},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8877":{"gid":8877,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/773071485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":773071485},{"type":"v1","addr":"172.21.15.40:6835","nonce":773071485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8877,"qdb_cluster":[8877]},"id":9}]} 2024-03-24T00:39:13.589 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 128 2024-03-24T00:39:13.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.588+0000 7f6590000640 1 -- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6568067660 msgr2=0x7f6568069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.588+0000 7f6590000640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6568067660 0x7f6568069b20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f657c001810 tx=0x7f657c009040 comp rx=0 tx=0).stop 2024-03-24T00:39:13.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.588+0000 7f6590000640 1 -- 172.21.15.40:0/3177560317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f658813ec80 msgr2=0x7f6588137ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:13.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.588+0000 7f6590000640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f658813ec80 0x7f6588137ee0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f657400d800 tx=0x7f657400dcc0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.591 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.588+0000 7f6590000640 1 -- 172.21.15.40:0/3177560317 shutdown_connections 2024-03-24T00:39:13.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.588+0000 7f6590000640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6568067660 0x7f6568069b20 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.588+0000 7f6590000640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6588141bc0 0x7f65880af960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.588+0000 7f6590000640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f658813ec80 0x7f6588137ee0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.588+0000 7f6590000640 1 --2- 172.21.15.40:0/3177560317 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f65880ce630 0x7f6588141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:13.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.588+0000 7f6590000640 1 -- 172.21.15.40:0/3177560317 >> 172.21.15.40:0/3177560317 conn(0x7f65880b17b0 msgr2=0x7f658815ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:13.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.588+0000 7f6590000640 1 -- 172.21.15.40:0/3177560317 shutdown_connections 2024-03-24T00:39:13.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:13.588+0000 7f6590000640 1 -- 172.21.15.40:0/3177560317 wait complete. 2024-03-24T00:39:14.015 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 128, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:39:07.531884+0000', 'modified': '2024-03-24T00:39:10.561787+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8877}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8877': {'gid': 8877, 'name': 'a', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/773071485', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 773071485}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 773071485}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8877, 'qdb_cluster': [8877]} 2024-03-24T00:39:14.015 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-24T00:39:14.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:39:14.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.152+0000 7fcc6a000640 1 Processor -- start 2024-03-24T00:39:14.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.152+0000 7fcc6a000640 1 -- start start 2024-03-24T00:39:14.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.152+0000 7fcc6a000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc6415e0b0 0x7fcc640d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.152+0000 7fcc6a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcc640cdf20 0x7fcc6406b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc6a000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc640d0e10 con 0x7fcc640cd3d0 2024-03-24T00:39:14.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc6a000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc640d10f0 con 0x7fcc640cdf20 2024-03-24T00:39:14.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc6a000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6406efd0 con 0x7fcc6415e0b0 2024-03-24T00:39:14.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc5be00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fcc640cd3d0 0x7fcc640cd810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42666/0 (socket says 172.21.15.40:42666) 2024-03-24T00:39:14.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc5be00640 1 -- 172.21.15.40:0/1125830944 learned_addr learned my addr 172.21.15.40:0/1125830944 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:14.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc63400640 1 --2- 172.21.15.40:0/1125830944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc6415e0b0 0x7fcc640d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc63e00640 1 --2- 172.21.15.40:0/1125830944 >> v2:172.21.15.53:3300/0 conn(0x7fcc640cdf20 0x7fcc6406b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc62a00640 1 -- 172.21.15.40:0/1125830944 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277166375 0 0) 0x7fcc640d0e10 con 0x7fcc640cd3d0 2024-03-24T00:39:14.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc62a00640 1 -- 172.21.15.40:0/1125830944 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc48003530 con 0x7fcc640cd3d0 2024-03-24T00:39:14.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc63400640 1 -- 172.21.15.40:0/1125830944 >> v2:172.21.15.53:3300/0 conn(0x7fcc640cdf20 msgr2=0x7fcc6406b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:14.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc62a00640 1 -- 172.21.15.40:0/1125830944 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1727947858 0 0) 0x7fcc48003530 con 0x7fcc640cd3d0 2024-03-24T00:39:14.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc63400640 1 --2- 172.21.15.40:0/1125830944 >> v2:172.21.15.53:3300/0 conn(0x7fcc640cdf20 0x7fcc6406b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc63400640 1 -- 172.21.15.40:0/1125830944 >> v1:172.21.15.40:6789/0 conn(0x7fcc640cd3d0 legacy=0x7fcc640cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc63400640 1 -- 172.21.15.40:0/1125830944 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6406f2b0 con 0x7fcc6415e0b0 2024-03-24T00:39:14.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.156+0000 7fcc63e00640 1 --2- 172.21.15.40:0/1125830944 >> v2:172.21.15.53:3300/0 conn(0x7fcc640cdf20 0x7fcc6406b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:14.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc63400640 1 --2- 172.21.15.40:0/1125830944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc6415e0b0 0x7fcc640d0830 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcc540097c0 tx=0x7fcc5401c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0f81b977dda3931 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc62a00640 1 -- 172.21.15.40:0/1125830944 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc540b1070 con 0x7fcc6415e0b0 2024-03-24T00:39:14.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc62a00640 1 -- 172.21.15.40:0/1125830944 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc540a4d00 con 0x7fcc6415e0b0 2024-03-24T00:39:14.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc62a00640 1 -- 172.21.15.40:0/1125830944 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c001680 con 0x7fcc6415e0b0 2024-03-24T00:39:14.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 -- 172.21.15.40:0/1125830944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc6415e0b0 msgr2=0x7fcc640d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 --2- 172.21.15.40:0/1125830944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc6415e0b0 0x7fcc640d0830 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcc540097c0 tx=0x7fcc5401c100 comp rx=0 tx=0).stop 2024-03-24T00:39:14.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 -- 172.21.15.40:0/1125830944 shutdown_connections 2024-03-24T00:39:14.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 --2- 172.21.15.40:0/1125830944 >> v2:172.21.15.53:3300/0 conn(0x7fcc640cdf20 0x7fcc6406b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 --2- 172.21.15.40:0/1125830944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc6415e0b0 0x7fcc640d0830 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 -- 172.21.15.40:0/1125830944 >> 172.21.15.40:0/1125830944 conn(0x7fcc640d1c90 msgr2=0x7fcc640d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:14.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 -- 172.21.15.40:0/1125830944 shutdown_connections 2024-03-24T00:39:14.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 -- 172.21.15.40:0/1125830944 wait complete. 2024-03-24T00:39:14.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 Processor -- start 2024-03-24T00:39:14.166 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 -- start start 2024-03-24T00:39:14.167 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc640cd3d0 0x7fcc64154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc640cdf20 0x7fcc641524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc6415e0b0 0x7fcc640c57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6406fc30 con 0x7fcc6415e0b0 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6406fab0 con 0x7fcc640cd3d0 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.160+0000 7fcc6a000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6406fdb0 con 0x7fcc640cdf20 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc6415e0b0 0x7fcc640c57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc6415e0b0 0x7fcc640c57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49492/0 (socket says 172.21.15.40:49492) 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc63e00640 1 -- 172.21.15.40:0/1336175468 learned_addr learned my addr 172.21.15.40:0/1336175468 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc5be00640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc640cdf20 0x7fcc641524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc63400640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc640cd3d0 0x7fcc64154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc63e00640 1 -- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc640cdf20 msgr2=0x7fcc641524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc63e00640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc640cdf20 0x7fcc641524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc63e00640 1 -- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc640cd3d0 msgr2=0x7fcc64154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc63e00640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc640cd3d0 0x7fcc64154f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc63e00640 1 -- 172.21.15.40:0/1336175468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc640cac00 con 0x7fcc6415e0b0 2024-03-24T00:39:14.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc63e00640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc6415e0b0 0x7fcc640c57c0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c00efc0 tx=0x7fcc5c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc5b400640 1 -- 172.21.15.40:0/1336175468 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c018030 con 0x7fcc6415e0b0 2024-03-24T00:39:14.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc5b400640 1 -- 172.21.15.40:0/1336175468 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5c00cde0 con 0x7fcc6415e0b0 2024-03-24T00:39:14.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc6a000640 1 -- 172.21.15.40:0/1336175468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc640caee0 con 0x7fcc6415e0b0 2024-03-24T00:39:14.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc5b400640 1 -- 172.21.15.40:0/1336175468 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c003bc0 con 0x7fcc6415e0b0 2024-03-24T00:39:14.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc6a000640 1 -- 172.21.15.40:0/1336175468 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc640c9db0 con 0x7fcc6415e0b0 2024-03-24T00:39:14.170 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.164+0000 7fcc5b400640 1 -- 172.21.15.40:0/1336175468 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fcc5c01b070 con 0x7fcc6415e0b0 2024-03-24T00:39:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc5b400640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc40067660 0x7fcc40069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc5b400640 1 -- 172.21.15.40:0/1336175468 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc5c0861e0 con 0x7fcc6415e0b0 2024-03-24T00:39:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63400640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc40067660 0x7fcc40069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 Processor -- start 2024-03-24T00:39:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 -- start start 2024-03-24T00:39:14.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63400640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc40067660 0x7fcc40069b20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fcc540ae000 tx=0x7fcc54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221413d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 0x55e2213c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2213ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e221217680 con 0x55e2213c9ff0 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e22119eb00 con 0x55e221413920 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e221213720 con 0x55e221414240 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2213ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221413d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221413d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:47146/0 (socket says 172.21.15.40:47146) 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63400640 1 -- 172.21.15.40:0/4200398143 learned_addr learned my addr 172.21.15.40:0/4200398143 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc5be00640 1 --2- 172.21.15.40:0/4200398143 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 0x55e2213c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63400640 1 -- 172.21.15.40:0/4200398143 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 msgr2=0x55e2213c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63400640 1 --2- 172.21.15.40:0/4200398143 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 0x55e2213c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.180 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63400640 1 -- 172.21.15.40:0/4200398143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 msgr2=0x55e2213ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63400640 1 --2- 172.21.15.40:0/4200398143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2213ca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63400640 1 -- 172.21.15.40:0/4200398143 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2211ef940 con 0x55e221413920 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63e00640 1 --2- 172.21.15.40:0/4200398143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2213ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc63400640 1 --2- 172.21.15.40:0/4200398143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221413d00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fcc540af040 tx=0x7fcc54004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=267a4ea035974504 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc53e00640 1 -- 172.21.15.40:0/4200398143 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc540c8070 con 0x55e221413920 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc53e00640 1 -- 172.21.15.40:0/4200398143 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc540b7b80 con 0x55e221413920 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc53e00640 1 -- 172.21.15.40:0/4200398143 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc540b1040 con 0x55e221413920 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 -- 172.21.15.40:0/4200398143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 msgr2=0x55e221413d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 --2- 172.21.15.40:0/4200398143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221413d00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fcc540af040 tx=0x7fcc54004740 comp rx=0 tx=0).stop 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 -- 172.21.15.40:0/4200398143 shutdown_connections 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 --2- 172.21.15.40:0/4200398143 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2213ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 --2- 172.21.15.40:0/4200398143 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 0x55e2213c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 --2- 172.21.15.40:0/4200398143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221413d00 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 -- 172.21.15.40:0/4200398143 >> 172.21.15.40:0/4200398143 conn(0x55e2213c5fa0 msgr2=0x55e2213c8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 -- 172.21.15.40:0/4200398143 shutdown_connections 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.168+0000 7fcc6d509000 1 -- 172.21.15.40:0/4200398143 wait complete. 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 1 -- start start 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 1 Processor -- start 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2213f8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e2213f9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 0x55e2213f97c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e221216990 con 0x55e2213c9ff0 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2211d1ae0 con 0x55e221413920 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e2211af750 con 0x55e221414240 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 10 client.0 _ll_register_callbacks cb 0x55e2213b4ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2213f8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2213f8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49502/0 (socket says 172.21.15.40:49502) 2024-03-24T00:39:14.182 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc63400640 1 -- 172.21.15.40:0/1980233952 learned_addr learned my addr 172.21.15.40:0/1980233952 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc5be00640 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e2213f9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc63e00640 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 0x55e2213f97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc63400640 1 -- 172.21.15.40:0/1980233952 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 msgr2=0x55e2213f97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc63400640 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 0x55e2213f97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc63400640 1 -- 172.21.15.40:0/1980233952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 msgr2=0x55e2213f9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc63400640 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e2213f9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc63400640 1 -- 172.21.15.40:0/1980233952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e22125cd80 con 0x55e2213c9ff0 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc63400640 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2213f8d40 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fcc5401c5e0 tx=0x7fcc5401a940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8041b35fb1ca0735 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc50c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc50c00640 1 -- 172.21.15.40:0/1980233952 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc540d1060 con 0x55e2213c9ff0 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc50c00640 1 -- 172.21.15.40:0/1980233952 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc540ac440 con 0x55e2213c9ff0 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 1 -- 172.21.15.40:0/1980233952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc540b7b80 con 0x55e2213c9ff0 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc50c00640 1 -- 172.21.15.40:0/1980233952 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=128) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc540d7050 con 0x55e2213c9ff0 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 10 client.9009 fetch_fsmap learned FSMap version 128 2024-03-24T00:39:14.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 1 -- 172.21.15.40:0/1980233952 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=128,monmap=2+}) v3 -- 0x55e22119ed10 con 0x55e2213c9ff0 2024-03-24T00:39:14.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc50c00640 1 -- 172.21.15.40:0/1980233952 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc540b7b80 con 0x55e2213c9ff0 2024-03-24T00:39:14.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc50c00640 1 -- 172.21.15.40:0/1980233952 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 128) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fcc540da020 con 0x55e2213c9ff0 2024-03-24T00:39:14.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.172+0000 7fcc6d509000 10 client.9009 fetch_fsmap finished waiting for FSMap version 128 2024-03-24T00:39:14.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 10 client.9009 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:39:14.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 20 client.9009 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:14.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55e2213fa090 0x55e221425ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 4 client.9009 mds_command: new command op to 8877 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:14.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 -- 172.21.15.40:0/1980233952 --> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e2211b2b60 con 0x55e2213fa090 2024-03-24T00:39:14.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc5be00640 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55e2213fa090 0x55e221425ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc5be00640 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55e2213fa090 0x55e221425ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7177b1e50cdafd9 server_cookie=b368dc1784f26a83 in_seq=0 out_seq=0 2024-03-24T00:39:14.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc50c00640 10 client.9009 ms_handle_connect on v2:172.21.15.40:6834/773071485 2024-03-24T00:39:14.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc50c00640 1 -- 172.21.15.40:0/1980233952 <== mds.0 v2:172.21.15.40:6834/773071485 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e2211b2b60 con 0x55e2213fa090 2024-03-24T00:39:14.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc50c00640 10 client.9009 handle_command_reply: tid=0 2024-03-24T00:39:14.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 client.9009 shutdown 2024-03-24T00:39:14.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 -- 172.21.15.40:0/1980233952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 msgr2=0x55e2213f8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2213f8d40 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fcc5401c5e0 tx=0x7fcc5401a940 comp rx=0 tx=0).stop 2024-03-24T00:39:14.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 -- 172.21.15.40:0/1980233952 shutdown_connections 2024-03-24T00:39:14.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55e2213fa090 0x55e221425ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 0x55e2213f97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e2213f9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc50c00640 0 client.9009 ms_handle_reset on v2:172.21.15.40:6834/773071485 2024-03-24T00:39:14.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 --2- 172.21.15.40:0/1980233952 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2213f8d40 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 -- 172.21.15.40:0/1980233952 >> 172.21.15.40:0/1980233952 conn(0x55e2213c5fa0 msgr2=0x55e2213c8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:14.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 -- 172.21.15.40:0/1980233952 shutdown_connections 2024-03-24T00:39:14.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 1 -- 172.21.15.40:0/1980233952 wait complete. 2024-03-24T00:39:14.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.176+0000 7fcc6d509000 20 client.9009 trim_cache size 0 max 16384 2024-03-24T00:39:14.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 Processor -- start 2024-03-24T00:39:14.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 -- start start 2024-03-24T00:39:14.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e2213c9ff0 0x55e2213f8010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221412530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e221414240 0x55e221412a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc64203030 con 0x55e221414240 2024-03-24T00:39:14.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc642021e0 con 0x55e221413920 2024-03-24T00:39:14.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc640b53f0 con 0x55e2213c9ff0 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e221414240 0x55e221412a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc63e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e221414240 0x55e221412a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49508/0 (socket says 172.21.15.40:49508) 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc63e00640 1 -- 172.21.15.40:0/2122624665 learned_addr learned my addr 172.21.15.40:0/2122624665 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc63400640 1 --2- 172.21.15.40:0/2122624665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e2213c9ff0 0x55e2213f8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc5be00640 1 --2- 172.21.15.40:0/2122624665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221412530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc63e00640 1 -- 172.21.15.40:0/2122624665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e2213c9ff0 msgr2=0x55e2213f8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc63e00640 1 --2- 172.21.15.40:0/2122624665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e2213c9ff0 0x55e2213f8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc63e00640 1 -- 172.21.15.40:0/2122624665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 msgr2=0x55e221412530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc63e00640 1 --2- 172.21.15.40:0/2122624665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221412530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc63e00640 1 -- 172.21.15.40:0/2122624665 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2211e4540 con 0x55e221414240 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc63e00640 1 --2- 172.21.15.40:0/2122624665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e221414240 0x55e221412a70 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c052bd0 tx=0x7fcc5c052b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c5ad3588c9cc4b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc53400640 1 -- 172.21.15.40:0/2122624665 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c0843a0 con 0x55e221414240 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc53400640 1 -- 172.21.15.40:0/2122624665 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5c052de0 con 0x55e221414240 2024-03-24T00:39:14.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc53400640 1 -- 172.21.15.40:0/2122624665 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c055de0 con 0x55e221414240 2024-03-24T00:39:14.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 -- 172.21.15.40:0/2122624665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e221414240 msgr2=0x55e221412a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 --2- 172.21.15.40:0/2122624665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e221414240 0x55e221412a70 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c052bd0 tx=0x7fcc5c052b40 comp rx=0 tx=0).stop 2024-03-24T00:39:14.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 -- 172.21.15.40:0/2122624665 shutdown_connections 2024-03-24T00:39:14.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 --2- 172.21.15.40:0/2122624665 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e221414240 0x55e221412a70 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 --2- 172.21.15.40:0/2122624665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221412530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 --2- 172.21.15.40:0/2122624665 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e2213c9ff0 0x55e2213f8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 -- 172.21.15.40:0/2122624665 >> 172.21.15.40:0/2122624665 conn(0x55e2213c5fa0 msgr2=0x55e2213f7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:14.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 -- 172.21.15.40:0/2122624665 shutdown_connections 2024-03-24T00:39:14.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.196+0000 7fcc6d509000 1 -- 172.21.15.40:0/2122624665 wait complete. 2024-03-24T00:39:14.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 1 -- start start 2024-03-24T00:39:14.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 1 Processor -- start 2024-03-24T00:39:14.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2214789b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221478ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 0x55e221425ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2211bfa80 con 0x55e2213c9ff0 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc640b59f0 con 0x55e221413920 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e2212d4700 con 0x55e221414240 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 10 client.0 _ll_register_callbacks cb 0x7fcc540c6070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2214789b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2214789b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49510/0 (socket says 172.21.15.40:49510) 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc63400640 1 -- 172.21.15.40:0/2092812046 learned_addr learned my addr 172.21.15.40:0/2092812046 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc63e00640 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 0x55e221425ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc5be00640 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221478ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc63400640 1 -- 172.21.15.40:0/2092812046 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 msgr2=0x55e221425ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc63400640 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 0x55e221425ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc63400640 1 -- 172.21.15.40:0/2092812046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 msgr2=0x55e221478ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:14.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc63400640 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221478ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc63400640 1 -- 172.21.15.40:0/2092812046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e2211ab4b0 con 0x55e2213c9ff0 2024-03-24T00:39:14.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc63400640 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2214789b0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fcc5401c670 tx=0x7fcc540ba050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383715a3efffea5e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc50c00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:39:14.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc50c00640 1 -- 172.21.15.40:0/2092812046 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc540c8040 con 0x55e2213c9ff0 2024-03-24T00:39:14.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc50c00640 1 -- 172.21.15.40:0/2092812046 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc540ac440 con 0x55e2213c9ff0 2024-03-24T00:39:14.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 1 -- 172.21.15.40:0/2092812046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc5c052de0 con 0x55e2213c9ff0 2024-03-24T00:39:14.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc50c00640 1 -- 172.21.15.40:0/2092812046 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=128) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc540e8050 con 0x55e2213c9ff0 2024-03-24T00:39:14.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 10 client.9015 fetch_fsmap learned FSMap version 128 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 1 -- 172.21.15.40:0/2092812046 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=128,monmap=2+}) v3 -- 0x55e2211be2f0 con 0x55e2213c9ff0 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc50c00640 1 -- 172.21.15.40:0/2092812046 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5c052de0 con 0x55e2213c9ff0 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc50c00640 1 -- 172.21.15.40:0/2092812046 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 128) v1 ==== 3314+0+0 (secure 0 0 0) 0x7fcc540eb020 con 0x55e2213c9ff0 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 10 client.9015 fetch_fsmap finished waiting for FSMap version 128 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 10 client.9015 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 20 client.9015 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55e221426860 0x55e221426c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 4 client.9015 mds_command: new command op to 8877 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc6d509000 1 -- 172.21.15.40:0/2092812046 --> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e221224bc0 con 0x55e221426860 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc5be00640 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55e221426860 0x55e221426c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc5be00640 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55e221426860 0x55e221426c40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=776e116b58bcce53 server_cookie=1f7052bf90d08d67 in_seq=0 out_seq=0 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.200+0000 7fcc50c00640 10 client.9015 ms_handle_connect on v2:172.21.15.40:6834/773071485 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc50c00640 1 -- 172.21.15.40:0/2092812046 <== mds.0 v2:172.21.15.40:6834/773071485 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x55e221224bc0 con 0x55e221426860 2024-03-24T00:39:14.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc50c00640 10 client.9015 handle_command_reply: tid=0 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6d509000 1 client.9015 shutdown 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6d509000 1 -- 172.21.15.40:0/2092812046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 msgr2=0x55e2214789b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6d509000 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2214789b0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fcc5401c670 tx=0x7fcc540ba050 comp rx=0 tx=0).stop 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6d509000 1 -- 172.21.15.40:0/2092812046 shutdown_connections 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6d509000 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55e221426860 0x55e221426c40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6d509000 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e221414240 0x55e221425ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6d509000 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e221413920 0x55e221478ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6d509000 1 --2- 172.21.15.40:0/2092812046 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55e2213c9ff0 0x55e2214789b0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6d509000 1 -- 172.21.15.40:0/2092812046 >> 172.21.15.40:0/2092812046 conn(0x55e2213c5fa0 msgr2=0x55e2213f8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc50c00640 0 client.9015 ms_handle_reset on v2:172.21.15.40:6834/773071485 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6d509000 1 -- 172.21.15.40:0/2092812046 shutdown_connections 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6d509000 1 -- 172.21.15.40:0/2092812046 wait complete. 2024-03-24T00:39:14.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6d509000 20 client.9015 trim_cache size 0 max 16384 2024-03-24T00:39:14.212 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:39:14.212 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:39:14.212 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:39:14.212 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8877, 2024-03-24T00:39:14.212 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:39:14.212 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:39:14.212 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:39:14.212 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 6.065217573, 2024-03-24T00:39:14.212 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 128, 2024-03-24T00:39:14.212 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 119, 2024-03-24T00:39:14.212 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 119, 2024-03-24T00:39:14.212 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 10.927247542 2024-03-24T00:39:14.213 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:39:14.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6a000640 1 -- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc40067660 msgr2=0x7fcc40069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6a000640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc40067660 0x7fcc40069b20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fcc540ae000 tx=0x7fcc54008040 comp rx=0 tx=0).stop 2024-03-24T00:39:14.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6a000640 1 -- 172.21.15.40:0/1336175468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc6415e0b0 msgr2=0x7fcc640c57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6a000640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc6415e0b0 0x7fcc640c57c0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c00efc0 tx=0x7fcc5c009f90 comp rx=0 tx=0).stop 2024-03-24T00:39:14.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6a000640 1 -- 172.21.15.40:0/1336175468 shutdown_connections 2024-03-24T00:39:14.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6a000640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fcc40067660 0x7fcc40069b20 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6a000640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fcc6415e0b0 0x7fcc640c57c0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6a000640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc640cdf20 0x7fcc641524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6a000640 1 --2- 172.21.15.40:0/1336175468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc640cd3d0 0x7fcc64154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6a000640 1 -- 172.21.15.40:0/1336175468 >> 172.21.15.40:0/1336175468 conn(0x7fcc640d1c90 msgr2=0x7fcc640cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:14.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6a000640 1 -- 172.21.15.40:0/1336175468 shutdown_connections 2024-03-24T00:39:14.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.204+0000 7fcc6a000640 1 -- 172.21.15.40:0/1336175468 wait complete. 2024-03-24T00:39:14.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-24T00:39:14.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.364+0000 7f539c600640 1 Processor -- start 2024-03-24T00:39:14.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.364+0000 7f539c600640 1 -- start start 2024-03-24T00:39:14.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.364+0000 7f539c600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f539406fab0 0x7f539406fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.364+0000 7f539c600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53940d0e10 con 0x7f53940d0830 2024-03-24T00:39:14.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.364+0000 7f539c600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539406efd0 con 0x7f539415e0b0 2024-03-24T00:39:14.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.364+0000 7f539c600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53940d10f0 con 0x7f539406fab0 2024-03-24T00:39:14.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.364+0000 7f539a600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f539406fab0 0x7f539406fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.364+0000 7f539a600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f539406fab0 0x7f539406fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49520/0 (socket says 172.21.15.40:49520) 2024-03-24T00:39:14.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.364+0000 7f539a600640 1 -- 172.21.15.40:0/2654899873 learned_addr learned my addr 172.21.15.40:0/2654899873 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:14.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.364+0000 7f539a600640 1 -- 172.21.15.40:0/2654899873 >> v1:172.21.15.53:6789/0 conn(0x7f539415e0b0 legacy=0x7f53940cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.364+0000 7f539a600640 1 -- 172.21.15.40:0/2654899873 >> v1:172.21.15.40:6789/0 conn(0x7f53940d0830 legacy=0x7f53940cdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.364+0000 7f539a600640 1 -- 172.21.15.40:0/2654899873 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539406f2b0 con 0x7f539406fab0 2024-03-24T00:39:14.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.368+0000 7f539a600640 1 --2- 172.21.15.40:0/2654899873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f539406fab0 0x7f539406fec0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f538c009e50 tx=0x7f538c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49cbba7915512445 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.368+0000 7f5393e00640 1 -- 172.21.15.40:0/2654899873 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c0a5700 con 0x7f539406fab0 2024-03-24T00:39:14.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.368+0000 7f5393e00640 1 -- 172.21.15.40:0/2654899873 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c0a58a0 con 0x7f539406fab0 2024-03-24T00:39:14.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.368+0000 7f5393e00640 1 -- 172.21.15.40:0/2654899873 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c0a5c70 con 0x7f539406fab0 2024-03-24T00:39:14.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.368+0000 7f539c600640 1 -- 172.21.15.40:0/2654899873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f539406fab0 msgr2=0x7f539406fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.368+0000 7f539c600640 1 --2- 172.21.15.40:0/2654899873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f539406fab0 0x7f539406fec0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f538c009e50 tx=0x7f538c01ca30 comp rx=0 tx=0).stop 2024-03-24T00:39:14.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.368+0000 7f539c600640 1 -- 172.21.15.40:0/2654899873 shutdown_connections 2024-03-24T00:39:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.368+0000 7f539c600640 1 --2- 172.21.15.40:0/2654899873 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f539406fab0 0x7f539406fec0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.368+0000 7f539c600640 1 -- 172.21.15.40:0/2654899873 >> 172.21.15.40:0/2654899873 conn(0x7f53940d1c90 msgr2=0x7f53940d20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.368+0000 7f539c600640 1 -- 172.21.15.40:0/2654899873 shutdown_connections 2024-03-24T00:39:14.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.368+0000 7f539c600640 1 -- 172.21.15.40:0/2654899873 wait complete. 2024-03-24T00:39:14.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539c600640 1 Processor -- start 2024-03-24T00:39:14.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539c600640 1 -- start start 2024-03-24T00:39:14.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539c600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f539406fab0 0x7f539415d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539c600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53940d0830 0x7f539414e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539c600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f539415e0b0 0x7f53940b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539c600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539416bec0 con 0x7f539415e0b0 2024-03-24T00:39:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539c600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539416bd40 con 0x7f53940d0830 2024-03-24T00:39:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539c600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539416c040 con 0x7f539406fab0 2024-03-24T00:39:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f539415e0b0 0x7f53940b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539a600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f539415e0b0 0x7f53940b14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49528/0 (socket says 172.21.15.40:49528) 2024-03-24T00:39:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539a600640 1 -- 172.21.15.40:0/1956994357 learned_addr learned my addr 172.21.15.40:0/1956994357 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f5399c00640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f539406fab0 0x7f539415d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f5399200640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53940d0830 0x7f539414e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539a600640 1 -- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f539406fab0 msgr2=0x7f539415d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539a600640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f539406fab0 0x7f539415d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539a600640 1 -- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53940d0830 msgr2=0x7f539414e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539a600640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53940d0830 0x7f539414e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539a600640 1 -- 172.21.15.40:0/1956994357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53940b19f0 con 0x7f539415e0b0 2024-03-24T00:39:14.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f5399200640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53940d0830 0x7f539414e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:14.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539a600640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f539415e0b0 0x7f53940b14b0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f538c019640 tx=0x7f538c0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f5391600640 1 -- 172.21.15.40:0/1956994357 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c0b8670 con 0x7f539415e0b0 2024-03-24T00:39:14.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f5391600640 1 -- 172.21.15.40:0/1956994357 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c0be070 con 0x7f539415e0b0 2024-03-24T00:39:14.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539c600640 1 -- 172.21.15.40:0/1956994357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539414e7e0 con 0x7f539415e0b0 2024-03-24T00:39:14.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f5391600640 1 -- 172.21.15.40:0/1956994357 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c0b8a30 con 0x7f539415e0b0 2024-03-24T00:39:14.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.372+0000 7f539c600640 1 -- 172.21.15.40:0/1956994357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539414eda0 con 0x7f539415e0b0 2024-03-24T00:39:14.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.376+0000 7f539c600640 1 -- 172.21.15.40:0/1956994357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5394068170 con 0x7f539415e0b0 2024-03-24T00:39:14.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.376+0000 7f5391600640 1 -- 172.21.15.40:0/1956994357 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f538c0b8bd0 con 0x7f539415e0b0 2024-03-24T00:39:14.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.376+0000 7f5391600640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5374067660 0x7f5374069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:14.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.376+0000 7f5391600640 1 -- 172.21.15.40:0/1956994357 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f538c120ab0 con 0x7f539415e0b0 2024-03-24T00:39:14.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.376+0000 7f5399c00640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5374067660 0x7f5374069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:14.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.376+0000 7f5399c00640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5374067660 0x7f5374069b20 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5388001650 tx=0x7f5388009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:14.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.380+0000 7f5391600640 1 -- 172.21.15.40:0/1956994357 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f538c016620 con 0x7f539415e0b0 2024-03-24T00:39:14.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.576+0000 7f539c600640 1 -- 172.21.15.40:0/1956994357 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5394159bf0 con 0x7f539415e0b0 2024-03-24T00:39:14.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.576+0000 7f5391600640 1 -- 172.21.15.40:0/1956994357 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v128) v1 ==== 53+0+73 (secure 0 0 0) 0x7f538c0ac3a0 con 0x7f539415e0b0 2024-03-24T00:39:14.583 INFO:teuthology.orchestra.run.smithi040.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:39:14.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.580+0000 7f539c600640 1 -- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5374067660 msgr2=0x7f5374069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.580+0000 7f539c600640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5374067660 0x7f5374069b20 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5388001650 tx=0x7f5388009040 comp rx=0 tx=0).stop 2024-03-24T00:39:14.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.580+0000 7f539c600640 1 -- 172.21.15.40:0/1956994357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f539415e0b0 msgr2=0x7f53940b14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:14.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.580+0000 7f539c600640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f539415e0b0 0x7f53940b14b0 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f538c019640 tx=0x7f538c0197d0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.580+0000 7f539c600640 1 -- 172.21.15.40:0/1956994357 shutdown_connections 2024-03-24T00:39:14.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.580+0000 7f539c600640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5374067660 0x7f5374069b20 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.580+0000 7f539c600640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f539415e0b0 0x7f53940b14b0 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.580+0000 7f539c600640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53940d0830 0x7f539414e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.580+0000 7f539c600640 1 --2- 172.21.15.40:0/1956994357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f539406fab0 0x7f539415d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:14.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.580+0000 7f539c600640 1 -- 172.21.15.40:0/1956994357 >> 172.21.15.40:0/1956994357 conn(0x7f53940d1c90 msgr2=0x7f53940b07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:14.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.580+0000 7f539c600640 1 -- 172.21.15.40:0/1956994357 shutdown_connections 2024-03-24T00:39:14.586 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:14.580+0000 7f539c600640 1 -- 172.21.15.40:0/1956994357 wait complete. 2024-03-24T00:39:14.604 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:39:14.604 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:39:14.604 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi040.front.sepia.ceph.com 2024-03-24T00:39:14.604 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:39:14.604 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:39:14.605 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:39:14.605 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:39:14.605 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:39:14.605 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:39:14.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:14.605 DEBUG:teuthology.orchestra.run.smithi040:> ip addr 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 ::1/128 scope host 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:16 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f0 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:6b:fd:17 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp7s0f1 2024-03-24T00:39:14.615 INFO:teuthology.orchestra.run.smithi040.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f0 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.21.15.40/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft 41995sec preferred_lft 41995sec 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::ec4:7aff:febd:12cc/64 scope link 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0c:c4:7a:bd:12:cd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: altname enp1s0f1 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 02:42:9e:ab:82:f9 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:14.616 INFO:teuthology.orchestra.run.smithi040.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout:9: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-03-24T00:39:14.617 INFO:teuthology.orchestra.run.smithi040.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:39:14.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:14.618 DEBUG:teuthology.orchestra.run.smithi040:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:39:14.676 INFO:teuthology.orchestra.run.smithi040.stdout:1 2024-03-24T00:39:14.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:14.678 DEBUG:teuthology.orchestra.run.smithi040:> route 2024-03-24T00:39:14.727 INFO:teuthology.orchestra.run.smithi040.stdout:Kernel IP routing table 2024-03-24T00:39:14.727 INFO:teuthology.orchestra.run.smithi040.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:39:14.727 INFO:teuthology.orchestra.run.smithi040.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-03-24T00:39:14.727 INFO:teuthology.orchestra.run.smithi040.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-03-24T00:39:14.728 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-03-24T00:39:14.728 INFO:teuthology.orchestra.run.smithi040.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:39:14.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:14.728 DEBUG:teuthology.orchestra.run.smithi040:> (cd / && exec bash -c ' 2024-03-24T00:39:14.728 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:39:14.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-24T00:39:14.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-24T00:39:14.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-24T00:39:14.728 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:39:14.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:14.819 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-24T00:39:14.827 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:39:14.827 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:39:14.828 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:39:14.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:14.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link set brx.0 up 2024-03-24T00:39:14.890 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-24T00:39:14.891 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:39:14.891 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:39:14.942 INFO:teuthology.orchestra.run.smithi040.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:39:14.942 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:39:14.942 DEBUG:teuthology.orchestra.run.smithi040:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:39:14.994 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:39:15.054 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:39:15.107 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:39:15.108 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:39:15.109 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:39:15.110 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:39:15.110 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:39:15.110 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:39:15.110 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:39:15.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:15.111 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:39:15.162 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-24T00:39:15.163 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-24T00:39:15.209 DEBUG:teuthology.orchestra.run.smithi040:> sudo modprobe fuse 2024-03-24T00:39:15.230 DEBUG:teuthology.orchestra.run.smithi040:> cat /proc/self/mounts | awk '{print $2}' 2024-03-24T00:39:15.288 INFO:teuthology.orchestra.run.smithi040.stdout:/sys 2024-03-24T00:39:15.288 INFO:teuthology.orchestra.run.smithi040.stdout:/proc 2024-03-24T00:39:15.288 INFO:teuthology.orchestra.run.smithi040.stdout:/dev 2024-03-24T00:39:15.288 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/pts 2024-03-24T00:39:15.288 INFO:teuthology.orchestra.run.smithi040.stdout:/run 2024-03-24T00:39:15.288 INFO:teuthology.orchestra.run.smithi040.stdout:/ 2024-03-24T00:39:15.288 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/security 2024-03-24T00:39:15.288 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/shm 2024-03-24T00:39:15.288 INFO:teuthology.orchestra.run.smithi040.stdout:/run/lock 2024-03-24T00:39:15.288 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/cgroup 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/pstore 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/bpf 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/sys/fs/binfmt_misc 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/hugepages 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/dev/mqueue 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/debug 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/tracing 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/proc/fs/nfsd 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/fs/fuse/connections 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/sys/kernel/config 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/run/credentials/systemd-sysusers.service 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/run/rpc_pipefs 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/run/user/1000 2024-03-24T00:39:15.289 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph 2024-03-24T00:39:15.290 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-0 2024-03-24T00:39:15.290 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-1 2024-03-24T00:39:15.290 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-2 2024-03-24T00:39:15.290 INFO:teuthology.orchestra.run.smithi040.stdout:/var/lib/ceph/osd/ceph-3 2024-03-24T00:39:15.290 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns 2024-03-24T00:39:15.290 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:39:15.290 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:39:15.290 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:39:15.290 INFO:teuthology.orchestra.run.smithi040.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:39:15.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:15.291 DEBUG:teuthology.orchestra.run.smithi040:> ls /sys/fs/fuse/connections 2024-03-24T00:39:15.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:2024-03-24T00:39:15.308+0000 7fc9bb4d1580 -1 init, newargv = 0x560dafd20960 newargc=15 2024-03-24T00:39:15.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[53878]: starting ceph client 2024-03-24T00:39:15.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi040.stderr:ceph-fuse[53878]: starting fuse 2024-03-24T00:39:15.322 INFO:teuthology.orchestra.run.smithi040.stdout:43 2024-03-24T00:39:15.322 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-03-24T00:39:15.322 DEBUG:teuthology.orchestra.run.smithi040:> sudo stdin-killer -- python3 -c ' 2024-03-24T00:39:15.322 DEBUG:teuthology.orchestra.run.smithi040:> import glob 2024-03-24T00:39:15.322 DEBUG:teuthology.orchestra.run.smithi040:> import re 2024-03-24T00:39:15.322 DEBUG:teuthology.orchestra.run.smithi040:> import os 2024-03-24T00:39:15.322 DEBUG:teuthology.orchestra.run.smithi040:> import subprocess 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> def _find_admin_socket(client_name): 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> files = glob.glob(asok_path) 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> # Given a non-glob path, it better be there 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> if "*" not in asok_path: 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> assert(len(files) == 1) 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> return files[0] 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> for f in files: 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-24T00:39:15.323 DEBUG:teuthology.orchestra.run.smithi040:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-24T00:39:15.324 DEBUG:teuthology.orchestra.run.smithi040:> contents = proc_f.read() 2024-03-24T00:39:15.324 DEBUG:teuthology.orchestra.run.smithi040:> if mountpoint in contents: 2024-03-24T00:39:15.324 DEBUG:teuthology.orchestra.run.smithi040:> return f 2024-03-24T00:39:15.324 DEBUG:teuthology.orchestra.run.smithi040:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-24T00:39:15.324 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:39:15.324 DEBUG:teuthology.orchestra.run.smithi040:> print(_find_admin_socket("client.0")) 2024-03-24T00:39:15.324 DEBUG:teuthology.orchestra.run.smithi040:> ' 2024-03-24T00:39:15.449 INFO:teuthology.orchestra.run.smithi040.stdout:/var/run/ceph/ceph-client.0.53878.asok 2024-03-24T00:39:15.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-24T00:39:15.460 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53878.asok 2024-03-24T00:39:15.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:15.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53878.asok status 2024-03-24T00:39:15.634 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:39:15.634 INFO:teuthology.orchestra.run.smithi040.stdout: "metadata": { 2024-03-24T00:39:15.634 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-24T00:39:15.634 INFO:teuthology.orchestra.run.smithi040.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:39:15.634 INFO:teuthology.orchestra.run.smithi040.stdout: "entity_id": "0", 2024-03-24T00:39:15.634 INFO:teuthology.orchestra.run.smithi040.stdout: "hostname": "smithi040", 2024-03-24T00:39:15.634 INFO:teuthology.orchestra.run.smithi040.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "pid": "53878", 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "root": "/" 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_count": 0, 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "dentry_pinned_count": 0, 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 9027, 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "inst": { 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "name": { 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "client", 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "num": 9027 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:39:15.635 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2730233528 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": { 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: "type": "v1", 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: "addr": "192.168.0.1:0", 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: "nonce": 2730233528 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: "inst_str": "client.9027 192.168.0.1:0/2730233528", 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: "addr_str": "192.168.0.1:0/2730233528", 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: "inode_count": 1, 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_epoch": 128, 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch": 119, 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_epoch_barrier": 0, 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: "blocklisted": false, 2024-03-24T00:39:15.636 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs" 2024-03-24T00:39:15.637 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:39:15.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:15.649 DEBUG:teuthology.orchestra.run.smithi040:> stat --file-system '--printf=%T 2024-03-24T00:39:15.649 DEBUG:teuthology.orchestra.run.smithi040:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-24T00:39:15.659 INFO:teuthology.orchestra.run.smithi040.stdout:fuseblk 2024-03-24T00:39:15.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-24T00:39:15.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:15.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:39:15.722 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random True 2024-03-24T00:39:15.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.900+0000 7fd236800640 1 Processor -- start 2024-03-24T00:39:15.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.900+0000 7fd236800640 1 -- start start 2024-03-24T00:39:15.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd236800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd23015bd60 0x7fd2300b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd236800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd23009df60 0x7fd23015fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd236800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd23015c140 0x7fd23015ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd236800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2300719c0 con 0x7fd23015bd60 2024-03-24T00:39:15.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd236800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd230160880 con 0x7fd23009df60 2024-03-24T00:39:15.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd236800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd230072fb0 con 0x7fd23015c140 2024-03-24T00:39:15.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd235400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd23015bd60 0x7fd2300b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.908 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd235400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd23015bd60 0x7fd2300b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49540/0 (socket says 172.21.15.40:49540) 2024-03-24T00:39:15.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd235400640 1 -- 172.21.15.40:0/4187436363 learned_addr learned my addr 172.21.15.40:0/4187436363 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:15.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd234a00640 1 --2- 172.21.15.40:0/4187436363 >> v2:172.21.15.53:3300/0 conn(0x7fd23009df60 0x7fd23015fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd235e00640 1 --2- 172.21.15.40:0/4187436363 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd23015c140 0x7fd23015ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd235400640 1 -- 172.21.15.40:0/4187436363 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd23015c140 msgr2=0x7fd23015ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:15.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd235400640 1 --2- 172.21.15.40:0/4187436363 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd23015c140 0x7fd23015ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd235400640 1 -- 172.21.15.40:0/4187436363 >> v2:172.21.15.53:3300/0 conn(0x7fd23009df60 msgr2=0x7fd23015fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:15.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd235400640 1 --2- 172.21.15.40:0/4187436363 >> v2:172.21.15.53:3300/0 conn(0x7fd23009df60 0x7fd23015fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.909 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.904+0000 7fd235400640 1 -- 172.21.15.40:0/4187436363 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd230073290 con 0x7fd23015bd60 2024-03-24T00:39:15.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd235400640 1 --2- 172.21.15.40:0/4187436363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd23015bd60 0x7fd2300b0980 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fd230071690 tx=0x7fd22401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8abf69060e401e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.911 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd22fe00640 1 -- 172.21.15.40:0/4187436363 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2240aa430 con 0x7fd23015bd60 2024-03-24T00:39:15.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd22fe00640 1 -- 172.21.15.40:0/4187436363 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2240af070 con 0x7fd23015bd60 2024-03-24T00:39:15.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd22fe00640 1 -- 172.21.15.40:0/4187436363 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2240aa7f0 con 0x7fd23015bd60 2024-03-24T00:39:15.912 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd236800640 1 -- 172.21.15.40:0/4187436363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd23015bd60 msgr2=0x7fd2300b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd236800640 1 --2- 172.21.15.40:0/4187436363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd23015bd60 0x7fd2300b0980 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fd230071690 tx=0x7fd22401c4f0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd236800640 1 -- 172.21.15.40:0/4187436363 shutdown_connections 2024-03-24T00:39:15.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd236800640 1 --2- 172.21.15.40:0/4187436363 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd23015c140 0x7fd23015ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd236800640 1 --2- 172.21.15.40:0/4187436363 >> v2:172.21.15.53:3300/0 conn(0x7fd23009df60 0x7fd23015fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd236800640 1 --2- 172.21.15.40:0/4187436363 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd23015bd60 0x7fd2300b0980 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd236800640 1 -- 172.21.15.40:0/4187436363 >> 172.21.15.40:0/4187436363 conn(0x7fd2300b57a0 msgr2=0x7fd2300725e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:15.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd236800640 1 -- 172.21.15.40:0/4187436363 shutdown_connections 2024-03-24T00:39:15.913 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd236800640 1 -- 172.21.15.40:0/4187436363 wait complete. 2024-03-24T00:39:15.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd236800640 1 Processor -- start 2024-03-24T00:39:15.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.908+0000 7fd236800640 1 -- start start 2024-03-24T00:39:15.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd236800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd23009df60 0x7fd230156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd236800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd23015bd60 0x7fd2300cf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd236800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd23015c140 0x7fd2300cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd236800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2300cd6a0 con 0x7fd23015c140 2024-03-24T00:39:15.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd236800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2300cd520 con 0x7fd23015bd60 2024-03-24T00:39:15.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd236800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2300cd820 con 0x7fd23009df60 2024-03-24T00:39:15.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd235e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd23015c140 0x7fd2300cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd235e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd23015c140 0x7fd2300cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49550/0 (socket says 172.21.15.40:49550) 2024-03-24T00:39:15.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd235e00640 1 -- 172.21.15.40:0/458255457 learned_addr learned my addr 172.21.15.40:0/458255457 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:15.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd235400640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd23009df60 0x7fd230156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd234a00640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd23015bd60 0x7fd2300cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd235e00640 1 -- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd23009df60 msgr2=0x7fd230156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd235e00640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd23009df60 0x7fd230156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd235e00640 1 -- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd23015bd60 msgr2=0x7fd2300cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:15.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd235e00640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd23015bd60 0x7fd2300cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:15.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd235e00640 1 -- 172.21.15.40:0/458255457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2300ce4c0 con 0x7fd23015c140 2024-03-24T00:39:15.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd234a00640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd23015bd60 0x7fd2300cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:15.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd235e00640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd23015c140 0x7fd2300cfdb0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fd228007d80 tx=0x7fd22800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd22d600640 1 -- 172.21.15.40:0/458255457 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd228014420 con 0x7fd23015c140 2024-03-24T00:39:15.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd22d600640 1 -- 172.21.15.40:0/458255457 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd228019070 con 0x7fd23015c140 2024-03-24T00:39:15.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd22d600640 1 -- 172.21.15.40:0/458255457 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd228014800 con 0x7fd23015c140 2024-03-24T00:39:15.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd236800640 1 -- 172.21.15.40:0/458255457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2300ce7a0 con 0x7fd23015c140 2024-03-24T00:39:15.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd236800640 1 -- 172.21.15.40:0/458255457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2300cf040 con 0x7fd23015c140 2024-03-24T00:39:15.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd236800640 1 -- 172.21.15.40:0/458255457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd23004f410 con 0x7fd23015c140 2024-03-24T00:39:15.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.912+0000 7fd22d600640 1 -- 172.21.15.40:0/458255457 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd2280149a0 con 0x7fd23015c140 2024-03-24T00:39:15.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.916+0000 7fd22d600640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd210067660 0x7fd210069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:15.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.916+0000 7fd22d600640 1 -- 172.21.15.40:0/458255457 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd228085fe0 con 0x7fd23015c140 2024-03-24T00:39:15.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.916+0000 7fd235400640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd210067660 0x7fd210069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:15.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.916+0000 7fd235400640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd210067660 0x7fd210069b20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd2240ae000 tx=0x7fd2240a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:15.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:15.920+0000 7fd22d600640 1 -- 172.21.15.40:0/458255457 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd228056c60 con 0x7fd23015c140 2024-03-24T00:39:16.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.080+0000 7fd236800640 1 -- 172.21.15.40:0/458255457 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random}] v 0) v1 -- 0x7fd2300cf210 con 0x7fd23015c140 2024-03-24T00:39:16.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.084+0000 7fd22d600640 1 -- 172.21.15.40:0/458255457 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random}]=0 v68)=0 v68) v1 ==== 128+0+0 (secure 0 0 0) 0x7fd228056560 con 0x7fd23015c140 2024-03-24T00:39:16.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.088+0000 7fd236800640 1 -- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd210067660 msgr2=0x7fd210069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.088+0000 7fd236800640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd210067660 0x7fd210069b20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd2240ae000 tx=0x7fd2240a7000 comp rx=0 tx=0).stop 2024-03-24T00:39:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.088+0000 7fd236800640 1 -- 172.21.15.40:0/458255457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd23015c140 msgr2=0x7fd2300cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.088+0000 7fd236800640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd23015c140 0x7fd2300cfdb0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7fd228007d80 tx=0x7fd22800e4b0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.088+0000 7fd236800640 1 -- 172.21.15.40:0/458255457 shutdown_connections 2024-03-24T00:39:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.088+0000 7fd236800640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd210067660 0x7fd210069b20 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.088+0000 7fd236800640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd23015c140 0x7fd2300cfdb0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.088+0000 7fd236800640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd23015bd60 0x7fd2300cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.088+0000 7fd236800640 1 --2- 172.21.15.40:0/458255457 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd23009df60 0x7fd230156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.088+0000 7fd236800640 1 -- 172.21.15.40:0/458255457 >> 172.21.15.40:0/458255457 conn(0x7fd2300b57a0 msgr2=0x7fd2300adb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.088+0000 7fd236800640 1 -- 172.21.15.40:0/458255457 shutdown_connections 2024-03-24T00:39:16.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.088+0000 7fd236800640 1 -- 172.21.15.40:0/458255457 wait complete. 2024-03-24T00:39:16.116 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_distributed True 2024-03-24T00:39:16.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.308+0000 7f09b1a00640 1 Processor -- start 2024-03-24T00:39:16.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.308+0000 7f09b1a00640 1 -- start start 2024-03-24T00:39:16.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.308+0000 7f09b1a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09ac15ffd0 0x7f09ac0ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.308+0000 7f09b1a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f09ac0728d0 0x7f09ac0cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.308+0000 7f09b1a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ac071de0 con 0x7f09ac15bd00 2024-03-24T00:39:16.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.308+0000 7f09b1a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ac0cf790 con 0x7f09ac0728d0 2024-03-24T00:39:16.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.308+0000 7f09b1a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0cea50 con 0x7f09ac15ffd0 2024-03-24T00:39:16.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.308+0000 7f09ab400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f09ac0728d0 0x7f09ac0cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.308+0000 7f09ab400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f09ac0728d0 0x7f09ac0cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49558/0 (socket says 172.21.15.40:49558) 2024-03-24T00:39:16.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.308+0000 7f09a3e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09ac15ffd0 0x7f09ac0ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.308+0000 7f09ab400640 1 -- 172.21.15.40:0/1542754450 learned_addr learned my addr 172.21.15.40:0/1542754450 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:16.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09a3e00640 1 -- 172.21.15.40:0/1542754450 >> v2:172.21.15.40:3300/0 conn(0x7f09ac0728d0 msgr2=0x7f09ac0cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:16.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09a3e00640 1 --2- 172.21.15.40:0/1542754450 >> v2:172.21.15.40:3300/0 conn(0x7f09ac0728d0 0x7f09ac0cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09a3e00640 1 -- 172.21.15.40:0/1542754450 >> v1:172.21.15.53:6789/0 conn(0x7f09ac15bd00 legacy=0x7f09ac15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09aa000640 1 -- 172.21.15.40:0/1542754450 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220748612 0 0) 0x7f09ac071de0 con 0x7f09ac15bd00 2024-03-24T00:39:16.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09a3e00640 1 -- 172.21.15.40:0/1542754450 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac0ced30 con 0x7f09ac15ffd0 2024-03-24T00:39:16.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09ab400640 1 --2- 172.21.15.40:0/1542754450 >> v2:172.21.15.40:3300/0 conn(0x7f09ac0728d0 0x7f09ac0cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09a3e00640 1 --2- 172.21.15.40:0/1542754450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09ac15ffd0 0x7f09ac0ae8e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f09940097c0 tx=0x7f099401c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae87937853aa0a8c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09aa000640 1 -- 172.21.15.40:0/1542754450 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09940ab650 con 0x7f09ac15ffd0 2024-03-24T00:39:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09aa000640 1 -- 172.21.15.40:0/1542754450 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09940b0070 con 0x7f09ac15ffd0 2024-03-24T00:39:16.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09aa000640 1 -- 172.21.15.40:0/1542754450 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09940aba10 con 0x7f09ac15ffd0 2024-03-24T00:39:16.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09b1a00640 1 -- 172.21.15.40:0/1542754450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09ac15ffd0 msgr2=0x7f09ac0ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09b1a00640 1 --2- 172.21.15.40:0/1542754450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09ac15ffd0 0x7f09ac0ae8e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f09940097c0 tx=0x7f099401c470 comp rx=0 tx=0).stop 2024-03-24T00:39:16.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09b1a00640 1 -- 172.21.15.40:0/1542754450 shutdown_connections 2024-03-24T00:39:16.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09b1a00640 1 --2- 172.21.15.40:0/1542754450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09ac15ffd0 0x7f09ac0ae8e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09b1a00640 1 --2- 172.21.15.40:0/1542754450 >> v2:172.21.15.40:3300/0 conn(0x7f09ac0728d0 0x7f09ac0cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09b1a00640 1 -- 172.21.15.40:0/1542754450 >> 172.21.15.40:0/1542754450 conn(0x7f09ac0b1660 msgr2=0x7f09ac0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:16.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.312+0000 7f09b1a00640 1 -- 172.21.15.40:0/1542754450 shutdown_connections 2024-03-24T00:39:16.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09b1a00640 1 -- 172.21.15.40:0/1542754450 wait complete. 2024-03-24T00:39:16.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09b1a00640 1 Processor -- start 2024-03-24T00:39:16.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09b1a00640 1 -- start start 2024-03-24T00:39:16.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09b1a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09ac0728d0 0x7f09ac150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09b1a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac15bd00 0x7f09ac159f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09b1a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09ac15ffd0 0x7f09ac155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09b1a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac06d320 con 0x7f09ac15bd00 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09b1a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac15c0e0 con 0x7f09ac15ffd0 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09b1a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac06d4a0 con 0x7f09ac0728d0 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09a3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac15bd00 0x7f09ac159f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09a3e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac15bd00 0x7f09ac159f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49564/0 (socket says 172.21.15.40:49564) 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09a3e00640 1 -- 172.21.15.40:0/2132999556 learned_addr learned my addr 172.21.15.40:0/2132999556 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09a3e00640 1 -- 172.21.15.40:0/2132999556 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09ac0728d0 msgr2=0x7f09ac150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09a3e00640 1 --2- 172.21.15.40:0/2132999556 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09ac0728d0 0x7f09ac150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09a3e00640 1 -- 172.21.15.40:0/2132999556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09ac15ffd0 msgr2=0x7f09ac155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09a3e00640 1 --2- 172.21.15.40:0/2132999556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09ac15ffd0 0x7f09ac155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09a3e00640 1 -- 172.21.15.40:0/2132999556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ac15de60 con 0x7f09ac15bd00 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09a3e00640 1 --2- 172.21.15.40:0/2132999556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac15bd00 0x7f09ac159f60 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f09940ad040 tx=0x7f099401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09a2a00640 1 -- 172.21.15.40:0/2132999556 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09940abe60 con 0x7f09ac15bd00 2024-03-24T00:39:16.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09a2a00640 1 -- 172.21.15.40:0/2132999556 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09940bf630 con 0x7f09ac15bd00 2024-03-24T00:39:16.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09b1a00640 1 -- 172.21.15.40:0/2132999556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ac0ae2c0 con 0x7f09ac15bd00 2024-03-24T00:39:16.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09a2a00640 1 -- 172.21.15.40:0/2132999556 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09940bfa20 con 0x7f09ac15bd00 2024-03-24T00:39:16.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.316+0000 7f09b1a00640 1 -- 172.21.15.40:0/2132999556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ac15e3e0 con 0x7f09ac15bd00 2024-03-24T00:39:16.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.320+0000 7f09b1a00640 1 -- 172.21.15.40:0/2132999556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ac068170 con 0x7f09ac15bd00 2024-03-24T00:39:16.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.320+0000 7f09a2a00640 1 -- 172.21.15.40:0/2132999556 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f09940bd020 con 0x7f09ac15bd00 2024-03-24T00:39:16.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.320+0000 7f09a2a00640 1 --2- 172.21.15.40:0/2132999556 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0988067690 0x7f0988069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.320+0000 7f09a2a00640 1 -- 172.21.15.40:0/2132999556 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f099411fcf0 con 0x7f09ac15bd00 2024-03-24T00:39:16.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.320+0000 7f09aaa00640 1 --2- 172.21.15.40:0/2132999556 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0988067690 0x7f0988069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.320+0000 7f09aaa00640 1 --2- 172.21.15.40:0/2132999556 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0988067690 0x7f0988069b50 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f099c007fd0 tx=0x7f099c007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.328+0000 7f09a2a00640 1 -- 172.21.15.40:0/2132999556 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f09940c6030 con 0x7f09ac15bd00 2024-03-24T00:39:16.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.492+0000 7f09b1a00640 1 -- 172.21.15.40:0/2132999556 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_distributed}] v 0) v1 -- 0x7f09ac153ec0 con 0x7f09ac15bd00 2024-03-24T00:39:16.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.496+0000 7f09a2a00640 1 -- 172.21.15.40:0/2132999556 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_distributed}]=0 v69)=0 v69) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0994016090 con 0x7f09ac15bd00 2024-03-24T00:39:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.500+0000 7f09b1a00640 1 -- 172.21.15.40:0/2132999556 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0988067690 msgr2=0x7f0988069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.500+0000 7f09b1a00640 1 --2- 172.21.15.40:0/2132999556 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0988067690 0x7f0988069b50 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f099c007fd0 tx=0x7f099c007920 comp rx=0 tx=0).stop 2024-03-24T00:39:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.500+0000 7f09b1a00640 1 -- 172.21.15.40:0/2132999556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac15bd00 msgr2=0x7f09ac159f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.500+0000 7f09b1a00640 1 --2- 172.21.15.40:0/2132999556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac15bd00 0x7f09ac159f60 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f09940ad040 tx=0x7f099401acb0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.500+0000 7f09b1a00640 1 -- 172.21.15.40:0/2132999556 shutdown_connections 2024-03-24T00:39:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.500+0000 7f09b1a00640 1 --2- 172.21.15.40:0/2132999556 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0988067690 0x7f0988069b50 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.500+0000 7f09b1a00640 1 --2- 172.21.15.40:0/2132999556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09ac15ffd0 0x7f09ac155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.500+0000 7f09b1a00640 1 --2- 172.21.15.40:0/2132999556 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f09ac15bd00 0x7f09ac159f60 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.505 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.500+0000 7f09b1a00640 1 --2- 172.21.15.40:0/2132999556 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09ac0728d0 0x7f09ac150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.500+0000 7f09b1a00640 1 -- 172.21.15.40:0/2132999556 >> 172.21.15.40:0/2132999556 conn(0x7f09ac0b1660 msgr2=0x7f09ac158b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:16.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.500+0000 7f09b1a00640 1 -- 172.21.15.40:0/2132999556 shutdown_connections 2024-03-24T00:39:16.506 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.500+0000 7f09b1a00640 1 -- 172.21.15.40:0/2132999556 wait complete. 2024-03-24T00:39:16.524 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_random_max 1.0 2024-03-24T00:39:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f3839000640 1 Processor -- start 2024-03-24T00:39:16.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f3839000640 1 -- start start 2024-03-24T00:39:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f3839000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f383415bd00 0x7f38340ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f3839000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f38340728d0 0x7f38340cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f3839000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3834072d30 con 0x7f383415f8c0 2024-03-24T00:39:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f3839000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38340cf790 con 0x7f38340728d0 2024-03-24T00:39:16.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f3839000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3834071de0 con 0x7f383415bd00 2024-03-24T00:39:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f3832800640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f383415f8c0 0x7f383406cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:42682/0 (socket says 172.21.15.40:42682) 2024-03-24T00:39:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f3832800640 1 -- 172.21.15.40:0/641434357 learned_addr learned my addr 172.21.15.40:0/641434357 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f382be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f38340728d0 0x7f38340cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f3831e00640 1 --2- 172.21.15.40:0/641434357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f383415bd00 0x7f38340ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f3831400640 1 -- 172.21.15.40:0/641434357 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886760559 0 0) 0x7f3834072d30 con 0x7f383415f8c0 2024-03-24T00:39:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f382be00640 1 -- 172.21.15.40:0/641434357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f383415bd00 msgr2=0x7f38340ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f382be00640 1 --2- 172.21.15.40:0/641434357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f383415bd00 0x7f38340ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f382be00640 1 -- 172.21.15.40:0/641434357 >> v1:172.21.15.40:6789/0 conn(0x7f383415f8c0 legacy=0x7f383406cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.664+0000 7f382be00640 1 -- 172.21.15.40:0/641434357 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38340cea50 con 0x7f38340728d0 2024-03-24T00:39:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.668+0000 7f382be00640 1 --2- 172.21.15.40:0/641434357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38340728d0 0x7f38340cf250 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f381c0097c0 tx=0x7f381c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d29b2e684c70b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.668+0000 7f3831400640 1 -- 172.21.15.40:0/641434357 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c0a7070 con 0x7f38340728d0 2024-03-24T00:39:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.668+0000 7f3831400640 1 -- 172.21.15.40:0/641434357 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381c0b0070 con 0x7f38340728d0 2024-03-24T00:39:16.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.668+0000 7f3831400640 1 -- 172.21.15.40:0/641434357 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381c0ab890 con 0x7f38340728d0 2024-03-24T00:39:16.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.668+0000 7f3839000640 1 -- 172.21.15.40:0/641434357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38340728d0 msgr2=0x7f38340cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.668+0000 7f3839000640 1 --2- 172.21.15.40:0/641434357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38340728d0 0x7f38340cf250 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f381c0097c0 tx=0x7f381c01c470 comp rx=0 tx=0).stop 2024-03-24T00:39:16.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.668+0000 7f3839000640 1 -- 172.21.15.40:0/641434357 shutdown_connections 2024-03-24T00:39:16.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.668+0000 7f3839000640 1 --2- 172.21.15.40:0/641434357 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f38340728d0 0x7f38340cf250 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.668+0000 7f3839000640 1 --2- 172.21.15.40:0/641434357 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f383415bd00 0x7f38340ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.668+0000 7f3839000640 1 -- 172.21.15.40:0/641434357 >> 172.21.15.40:0/641434357 conn(0x7f38340b1660 msgr2=0x7f38340b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:16.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.668+0000 7f3839000640 1 -- 172.21.15.40:0/641434357 shutdown_connections 2024-03-24T00:39:16.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.668+0000 7f3839000640 1 -- 172.21.15.40:0/641434357 wait complete. 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3839000640 1 Processor -- start 2024-03-24T00:39:16.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3839000640 1 -- start start 2024-03-24T00:39:16.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3839000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38340728d0 0x7f3834157570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3839000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f383415bd00 0x7f38340af630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3839000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f383415f8c0 0x7f38340b0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3839000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383415c0e0 con 0x7f383415f8c0 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3839000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383406d630 con 0x7f38340728d0 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3839000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38341600e0 con 0x7f383415bd00 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3832800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f383415f8c0 0x7f38340b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3832800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f383415f8c0 0x7f38340b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:49588/0 (socket says 172.21.15.40:49588) 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3832800640 1 -- 172.21.15.40:0/1023787765 learned_addr learned my addr 172.21.15.40:0/1023787765 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f382be00640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f383415bd00 0x7f38340af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3831e00640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38340728d0 0x7f3834157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3832800640 1 -- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f383415bd00 msgr2=0x7f38340af630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3832800640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f383415bd00 0x7f38340af630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3832800640 1 -- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38340728d0 msgr2=0x7f3834157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3832800640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38340728d0 0x7f3834157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3832800640 1 -- 172.21.15.40:0/1023787765 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383414f3f0 con 0x7f383415f8c0 2024-03-24T00:39:16.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3831e00640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38340728d0 0x7f3834157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:16.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f382be00640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f383415bd00 0x7f38340af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:39:16.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3832800640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f383415f8c0 0x7f38340b0490 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f382c00c700 tx=0x7f382c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f382a000640 1 -- 172.21.15.40:0/1023787765 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c018070 con 0x7f383415f8c0 2024-03-24T00:39:16.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f382a000640 1 -- 172.21.15.40:0/1023787765 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f382c00ce10 con 0x7f383415f8c0 2024-03-24T00:39:16.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3839000640 1 -- 172.21.15.40:0/1023787765 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383414f6d0 con 0x7f383415f8c0 2024-03-24T00:39:16.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f382a000640 1 -- 172.21.15.40:0/1023787765 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c00a860 con 0x7f383415f8c0 2024-03-24T00:39:16.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3839000640 1 -- 172.21.15.40:0/1023787765 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38340af110 con 0x7f383415f8c0 2024-03-24T00:39:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.672+0000 7f3839000640 1 -- 172.21.15.40:0/1023787765 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3834068170 con 0x7f383415f8c0 2024-03-24T00:39:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.676+0000 7f382a000640 1 -- 172.21.15.40:0/1023787765 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f382c01b070 con 0x7f383415f8c0 2024-03-24T00:39:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.676+0000 7f382a000640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3810067660 0x7f3810069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:16.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.676+0000 7f382a000640 1 -- 172.21.15.40:0/1023787765 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f382c084ed0 con 0x7f383415f8c0 2024-03-24T00:39:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.676+0000 7f3831e00640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3810067660 0x7f3810069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:16.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.676+0000 7f3831e00640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3810067660 0x7f3810069b20 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3824007a70 tx=0x7f3824008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:16.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.680+0000 7f382a000640 1 -- 172.21.15.40:0/1023787765 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f382c055ab0 con 0x7f383415f8c0 2024-03-24T00:39:16.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.844+0000 7f3839000640 1 -- 172.21.15.40:0/1023787765 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_random_max}] v 0) v1 -- 0x7f383406e870 con 0x7f383415f8c0 2024-03-24T00:39:16.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f382a000640 1 -- 172.21.15.40:0/1023787765 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_random_max}]=0 v70)=0 v70) v1 ==== 131+0+0 (secure 0 0 0) 0x7f382c0553b0 con 0x7f383415f8c0 2024-03-24T00:39:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f3839000640 1 -- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3810067660 msgr2=0x7f3810069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f3839000640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3810067660 0x7f3810069b20 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f3824007a70 tx=0x7f3824008040 comp rx=0 tx=0).stop 2024-03-24T00:39:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f3839000640 1 -- 172.21.15.40:0/1023787765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f383415f8c0 msgr2=0x7f38340b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f3839000640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f383415f8c0 0x7f38340b0490 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f382c00c700 tx=0x7f382c00cbc0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f3839000640 1 -- 172.21.15.40:0/1023787765 shutdown_connections 2024-03-24T00:39:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f3839000640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3810067660 0x7f3810069b20 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f3839000640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f383415f8c0 0x7f38340b0490 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f3839000640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f383415bd00 0x7f38340af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f3839000640 1 --2- 172.21.15.40:0/1023787765 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38340728d0 0x7f3834157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f3839000640 1 -- 172.21.15.40:0/1023787765 >> 172.21.15.40:0/1023787765 conn(0x7f38340b1660 msgr2=0x7f3834159300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f3839000640 1 -- 172.21.15.40:0/1023787765 shutdown_connections 2024-03-24T00:39:16.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:16.852+0000 7f3839000640 1 -- 172.21.15.40:0/1023787765 wait complete. 2024-03-24T00:39:16.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:16.876 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:39:16.876 DEBUG:teuthology.orchestra.run.smithi040:> set -e 2024-03-24T00:39:16.876 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:39:16.876 DEBUG:teuthology.orchestra.run.smithi040:> # Use up a random number of inode numbers so the ephemeral pinning is not the same every test. 2024-03-24T00:39:16.876 DEBUG:teuthology.orchestra.run.smithi040:> mkdir .inode_number_thrash 2024-03-24T00:39:16.876 DEBUG:teuthology.orchestra.run.smithi040:> count=$((RANDOM % 1024)) 2024-03-24T00:39:16.876 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < count; i++)); do touch .inode_number_thrash/$i; done 2024-03-24T00:39:16.877 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf .inode_number_thrash 2024-03-24T00:39:16.877 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:39:21.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-24T00:39:21.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabcc800640 1 Processor -- start 2024-03-24T00:39:21.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabcc800640 1 -- start start 2024-03-24T00:39:21.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabcc800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fabc415e170 0x7fabc406f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabcc800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fabc4160080 0x7fabc415e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabcc800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc40cfcb0 con 0x7fabc40d0a10 2024-03-24T00:39:21.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabcc800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc4070100 con 0x7fabc4160080 2024-03-24T00:39:21.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabcc800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc406fe20 con 0x7fabc415e170 2024-03-24T00:39:21.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabc9400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fabc40d0a10 0x7fabc415f970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39012/0 (socket says 172.21.15.40:39012) 2024-03-24T00:39:21.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabc9400640 1 -- 172.21.15.40:0/1270677465 learned_addr learned my addr 172.21.15.40:0/1270677465 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:21.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabc9e00640 1 --2- 172.21.15.40:0/1270677465 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fabc415e170 0x7fabc406f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabca800640 1 --2- 172.21.15.40:0/1270677465 >> v2:172.21.15.53:3300/0 conn(0x7fabc4160080 0x7fabc415e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabc8a00640 1 -- 172.21.15.40:0/1270677465 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778034403 0 0) 0x7fabb4001120 con 0x7fabc40d0a10 2024-03-24T00:39:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabc8a00640 1 -- 172.21.15.40:0/1270677465 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabac003530 con 0x7fabc40d0a10 2024-03-24T00:39:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabc9e00640 1 -- 172.21.15.40:0/1270677465 >> v2:172.21.15.53:3300/0 conn(0x7fabc4160080 msgr2=0x7fabc415e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabc9e00640 1 --2- 172.21.15.40:0/1270677465 >> v2:172.21.15.53:3300/0 conn(0x7fabc4160080 0x7fabc415e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabc9e00640 1 -- 172.21.15.40:0/1270677465 >> v1:172.21.15.40:6789/0 conn(0x7fabc40d0a10 legacy=0x7fabc415f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.324+0000 7fabc9e00640 1 -- 172.21.15.40:0/1270677465 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc40cff30 con 0x7fabc415e170 2024-03-24T00:39:21.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabc9e00640 1 --2- 172.21.15.40:0/1270677465 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fabc415e170 0x7fabc406f7b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fabb80097c0 tx=0x7fabb801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58d87b0f758de2b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabc8a00640 1 -- 172.21.15.40:0/1270677465 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb80b2070 con 0x7fabc415e170 2024-03-24T00:39:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabc8a00640 1 -- 172.21.15.40:0/1270677465 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb80ad680 con 0x7fabc415e170 2024-03-24T00:39:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabc8a00640 1 -- 172.21.15.40:0/1270677465 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb80ada50 con 0x7fabc415e170 2024-03-24T00:39:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 -- 172.21.15.40:0/1270677465 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fabc415e170 msgr2=0x7fabc406f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 --2- 172.21.15.40:0/1270677465 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fabc415e170 0x7fabc406f7b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fabb80097c0 tx=0x7fabb801c470 comp rx=0 tx=0).stop 2024-03-24T00:39:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 -- 172.21.15.40:0/1270677465 shutdown_connections 2024-03-24T00:39:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 --2- 172.21.15.40:0/1270677465 >> v2:172.21.15.53:3300/0 conn(0x7fabc4160080 0x7fabc415e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 --2- 172.21.15.40:0/1270677465 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fabc415e170 0x7fabc406f7b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 -- 172.21.15.40:0/1270677465 >> 172.21.15.40:0/1270677465 conn(0x7fabc40b36a0 msgr2=0x7fabc40b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:21.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 -- 172.21.15.40:0/1270677465 shutdown_connections 2024-03-24T00:39:21.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 -- 172.21.15.40:0/1270677465 wait complete. 2024-03-24T00:39:21.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 Processor -- start 2024-03-24T00:39:21.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 -- start start 2024-03-24T00:39:21.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fabc40d0a10 0x7fabc41fb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc415e170 0x7fabc40b7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabc4160080 0x7fabc4155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc416b030 con 0x7fabc415e170 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc416aeb0 con 0x7fabc4160080 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.332+0000 7fabcc800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc416b1b0 con 0x7fabc40d0a10 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabc9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc415e170 0x7fabc40b7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabc9400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc415e170 0x7fabc40b7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37010/0 (socket says 172.21.15.40:37010) 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabc9400640 1 -- 172.21.15.40:0/1747323963 learned_addr learned my addr 172.21.15.40:0/1747323963 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabc9e00640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fabc40d0a10 0x7fabc41fb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabca800640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabc4160080 0x7fabc4155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabc9e00640 1 -- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabc4160080 msgr2=0x7fabc4155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabc9e00640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabc4160080 0x7fabc4155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabc9e00640 1 -- 172.21.15.40:0/1747323963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc415e170 msgr2=0x7fabc40b7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabc9e00640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc415e170 0x7fabc40b7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabc9e00640 1 -- 172.21.15.40:0/1747323963 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc41fb8c0 con 0x7fabc40d0a10 2024-03-24T00:39:21.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabc9400640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc415e170 0x7fabc40b7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:21.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabc9e00640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fabc40d0a10 0x7fabc41fb380 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fabb80ac000 tx=0x7fabb801acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:21.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabbe000640 1 -- 172.21.15.40:0/1747323963 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb80b2040 con 0x7fabc40d0a10 2024-03-24T00:39:21.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabbe000640 1 -- 172.21.15.40:0/1747323963 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb80ad970 con 0x7fabc40d0a10 2024-03-24T00:39:21.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabbe000640 1 -- 172.21.15.40:0/1747323963 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb80add60 con 0x7fabc40d0a10 2024-03-24T00:39:21.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabcc800640 1 -- 172.21.15.40:0/1747323963 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc40c5e90 con 0x7fabc40d0a10 2024-03-24T00:39:21.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabcc800640 1 -- 172.21.15.40:0/1747323963 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc4154b60 con 0x7fabc40d0a10 2024-03-24T00:39:21.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.336+0000 7fabcc800640 1 -- 172.21.15.40:0/1747323963 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc4068170 con 0x7fabc40d0a10 2024-03-24T00:39:21.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.340+0000 7fabbe000640 1 -- 172.21.15.40:0/1747323963 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fabb80bf030 con 0x7fabc40d0a10 2024-03-24T00:39:21.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.340+0000 7fabbe000640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faba8067660 0x7faba8069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:21.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.340+0000 7fabbe000640 1 -- 172.21.15.40:0/1747323963 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fabb811eed0 con 0x7fabc40d0a10 2024-03-24T00:39:21.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.340+0000 7fabc9400640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faba8067660 0x7faba8069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:21.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.340+0000 7fabc9400640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faba8067660 0x7faba8069b20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fabc40c73c0 tx=0x7fabb4007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:21.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.344+0000 7fabbe000640 1 -- 172.21.15.40:0/1747323963 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fabb80c6060 con 0x7fabc40d0a10 2024-03-24T00:39:21.560 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.556+0000 7fabcc800640 1 -- 172.21.15.40:0/1747323963 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7fabc4072f30 con 0x7fabc40d0a10 2024-03-24T00:39:21.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.864+0000 7fabbe000640 1 -- 172.21.15.40:0/1747323963 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v129) v1 ==== 105+0+0 (secure 0 0 0) 0x7fabc41fb8c0 con 0x7fabc40d0a10 2024-03-24T00:39:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.868+0000 7fabcc800640 1 -- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faba8067660 msgr2=0x7faba8069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.868+0000 7fabcc800640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faba8067660 0x7faba8069b20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fabc40c73c0 tx=0x7fabb4007be0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.868+0000 7fabcc800640 1 -- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fabc40d0a10 msgr2=0x7fabc41fb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.868+0000 7fabcc800640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fabc40d0a10 0x7fabc41fb380 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fabb80ac000 tx=0x7fabb801acb0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.868+0000 7fabcc800640 1 -- 172.21.15.40:0/1747323963 shutdown_connections 2024-03-24T00:39:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.868+0000 7fabcc800640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7faba8067660 0x7faba8069b20 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.868+0000 7fabcc800640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabc4160080 0x7fabc4155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.868+0000 7fabcc800640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fabc415e170 0x7fabc40b7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.868+0000 7fabcc800640 1 --2- 172.21.15.40:0/1747323963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fabc40d0a10 0x7fabc41fb380 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.868+0000 7fabcc800640 1 -- 172.21.15.40:0/1747323963 >> 172.21.15.40:0/1747323963 conn(0x7fabc40b36a0 msgr2=0x7fabc415cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.868+0000 7fabcc800640 1 -- 172.21.15.40:0/1747323963 shutdown_connections 2024-03-24T00:39:21.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:21.868+0000 7fabcc800640 1 -- 172.21.15.40:0/1747323963 wait complete. 2024-03-24T00:39:21.898 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:22.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.036+0000 7fd37d200640 1 Processor -- start 2024-03-24T00:39:22.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.036+0000 7fd37d200640 1 -- start start 2024-03-24T00:39:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.036+0000 7fd37d200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37806e620 0x7fd3780b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.036+0000 7fd37d200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd37806ea00 0x7fd3780b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.036+0000 7fd37d200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd378070f60 con 0x7fd3780b2ff0 2024-03-24T00:39:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.036+0000 7fd37d200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3780b4c00 con 0x7fd37806ea00 2024-03-24T00:39:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.036+0000 7fd37d200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3780b3620 con 0x7fd37806e620 2024-03-24T00:39:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.036+0000 7fd375e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37806e620 0x7fd3780b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.036+0000 7fd36fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd37806ea00 0x7fd3780b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.040+0000 7fd376800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd3780b2ff0 0x7fd3780ccfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58542/0 (socket says 172.21.15.40:58542) 2024-03-24T00:39:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.040+0000 7fd376800640 1 -- 172.21.15.40:0/3493923473 learned_addr learned my addr 172.21.15.40:0/3493923473 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:22.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.040+0000 7fd375e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37806e620 0x7fd3780b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:55152/0 (socket says 172.21.15.40:55152) 2024-03-24T00:39:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.040+0000 7fd375e00640 1 -- 172.21.15.40:0/3493923473 >> v2:172.21.15.53:3300/0 conn(0x7fd37806ea00 msgr2=0x7fd3780b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.040+0000 7fd375e00640 1 --2- 172.21.15.40:0/3493923473 >> v2:172.21.15.53:3300/0 conn(0x7fd37806ea00 0x7fd3780b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.040+0000 7fd375e00640 1 -- 172.21.15.40:0/3493923473 >> v1:172.21.15.53:6789/0 conn(0x7fd3780b2ff0 legacy=0x7fd3780ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.040+0000 7fd375400640 1 -- 172.21.15.40:0/3493923473 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490964056 0 0) 0x7fd378070f60 con 0x7fd3780b2ff0 2024-03-24T00:39:22.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.040+0000 7fd375e00640 1 -- 172.21.15.40:0/3493923473 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3780b3900 con 0x7fd37806e620 2024-03-24T00:39:22.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.040+0000 7fd375e00640 1 --2- 172.21.15.40:0/3493923473 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37806e620 0x7fd3780b46c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd3680097c0 tx=0x7fd36801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=934d82ca3e57d0ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.040+0000 7fd375400640 1 -- 172.21.15.40:0/3493923473 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680b0070 con 0x7fd37806e620 2024-03-24T00:39:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.040+0000 7fd375400640 1 -- 172.21.15.40:0/3493923473 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3680ab600 con 0x7fd37806e620 2024-03-24T00:39:22.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.040+0000 7fd375400640 1 -- 172.21.15.40:0/3493923473 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680ab9d0 con 0x7fd37806e620 2024-03-24T00:39:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.044+0000 7fd37d200640 1 -- 172.21.15.40:0/3493923473 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37806e620 msgr2=0x7fd3780b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.044+0000 7fd37d200640 1 --2- 172.21.15.40:0/3493923473 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37806e620 0x7fd3780b46c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd3680097c0 tx=0x7fd36801c410 comp rx=0 tx=0).stop 2024-03-24T00:39:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.044+0000 7fd37d200640 1 -- 172.21.15.40:0/3493923473 shutdown_connections 2024-03-24T00:39:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.044+0000 7fd37d200640 1 --2- 172.21.15.40:0/3493923473 >> v2:172.21.15.53:3300/0 conn(0x7fd37806ea00 0x7fd3780b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.044+0000 7fd37d200640 1 --2- 172.21.15.40:0/3493923473 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd37806e620 0x7fd3780b46c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.044+0000 7fd37d200640 1 -- 172.21.15.40:0/3493923473 >> 172.21.15.40:0/3493923473 conn(0x7fd3780b17b0 msgr2=0x7fd3780b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.044+0000 7fd37d200640 1 -- 172.21.15.40:0/3493923473 shutdown_connections 2024-03-24T00:39:22.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.044+0000 7fd37d200640 1 -- 172.21.15.40:0/3493923473 wait complete. 2024-03-24T00:39:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.044+0000 7fd37d200640 1 Processor -- start 2024-03-24T00:39:22.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.044+0000 7fd37d200640 1 -- start start 2024-03-24T00:39:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd37d200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37806e620 0x7fd37815a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd37d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd37806ea00 0x7fd37812fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd37d200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3780b2ff0 0x7fd37815e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd37d200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd378144920 con 0x7fd37806e620 2024-03-24T00:39:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd37d200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37806ede0 con 0x7fd37806ea00 2024-03-24T00:39:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd37d200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd378132450 con 0x7fd3780b2ff0 2024-03-24T00:39:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd375e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37806e620 0x7fd37815a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd36fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd37806ea00 0x7fd37812fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd375e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37806e620 0x7fd37815a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37026/0 (socket says 172.21.15.40:37026) 2024-03-24T00:39:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd375e00640 1 -- 172.21.15.40:0/588707968 learned_addr learned my addr 172.21.15.40:0/588707968 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd376800640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3780b2ff0 0x7fd37815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:22.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd375e00640 1 -- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3780b2ff0 msgr2=0x7fd37815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd375e00640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3780b2ff0 0x7fd37815e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd375e00640 1 -- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd37806ea00 msgr2=0x7fd37812fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd375e00640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd37806ea00 0x7fd37812fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd375e00640 1 -- 172.21.15.40:0/588707968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3781326d0 con 0x7fd37806e620 2024-03-24T00:39:22.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd375e00640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37806e620 0x7fd37815a3c0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fd36801a040 tx=0x7fd36801ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd36e000640 1 -- 172.21.15.40:0/588707968 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680b0050 con 0x7fd37806e620 2024-03-24T00:39:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd36e000640 1 -- 172.21.15.40:0/588707968 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3680bc070 con 0x7fd37806e620 2024-03-24T00:39:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd36e000640 1 -- 172.21.15.40:0/588707968 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680b5830 con 0x7fd37806e620 2024-03-24T00:39:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd37d200640 1 -- 172.21.15.40:0/588707968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3781329b0 con 0x7fd37806e620 2024-03-24T00:39:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.048+0000 7fd37d200640 1 -- 172.21.15.40:0/588707968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd378158540 con 0x7fd37806e620 2024-03-24T00:39:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.052+0000 7fd367400640 1 -- 172.21.15.40:0/588707968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd378068170 con 0x7fd37806e620 2024-03-24T00:39:22.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.052+0000 7fd36e000640 1 -- 172.21.15.40:0/588707968 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd3680b59d0 con 0x7fd37806e620 2024-03-24T00:39:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.052+0000 7fd36e000640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd354067660 0x7fd354069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:22.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.052+0000 7fd36e000640 1 -- 172.21.15.40:0/588707968 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd36811ff70 con 0x7fd37806e620 2024-03-24T00:39:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.052+0000 7fd36fe00640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd354067660 0x7fd354069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:22.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.052+0000 7fd36fe00640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd354067660 0x7fd354069b20 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd37815e3f0 tx=0x7fd360012040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:22.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.056+0000 7fd36e000640 1 -- 172.21.15.40:0/588707968 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd368016090 con 0x7fd37806e620 2024-03-24T00:39:22.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 -- 172.21.15.40:0/588707968 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd378072f50 con 0x7fd37806e620 2024-03-24T00:39:22.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd36e000640 1 -- 172.21.15.40:0/588707968 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4998 (secure 0 0 0) 0x7fd3680f0500 con 0x7fd37806e620 2024-03-24T00:39:22.300 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:22.300 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":130,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1582567544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1582567544},{"type":"v1","addr":"172.21.15.53:6837","nonce":1582567544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123},{"gid":8563,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6836/2255755838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2255755838},{"type":"v1","addr":"172.21.15.53:6836","nonce":2255755838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:39:07.531884+0000","modified":"2024-03-24T00:39:21.870145+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":8877,"mds_1":8874},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8874":{"gid":8874,"name":"c","rank":1,"incarnation":130,"state":"up:creating","state_seq":1,"addr":"172.21.15.40:6833/4273998586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4273998586},{"type":"v1","addr":"172.21.15.40:6833","nonce":4273998586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8877":{"gid":8877,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/773071485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":773071485},{"type":"v1","addr":"172.21.15.40:6835","nonce":773071485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8877,"qdb_cluster":[8877]},"id":9}]} 2024-03-24T00:39:22.300 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 130 2024-03-24T00:39:22.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 -- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd354067660 msgr2=0x7fd354069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:22.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd354067660 0x7fd354069b20 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd37815e3f0 tx=0x7fd360012040 comp rx=0 tx=0).stop 2024-03-24T00:39:22.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 -- 172.21.15.40:0/588707968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37806e620 msgr2=0x7fd37815a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:22.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37806e620 0x7fd37815a3c0 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fd36801a040 tx=0x7fd36801ac50 comp rx=0 tx=0).stop 2024-03-24T00:39:22.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 -- 172.21.15.40:0/588707968 shutdown_connections 2024-03-24T00:39:22.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd354067660 0x7fd354069b20 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd3780b2ff0 0x7fd37815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd37806ea00 0x7fd37812fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 --2- 172.21.15.40:0/588707968 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd37806e620 0x7fd37815a3c0 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:22.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 -- 172.21.15.40:0/588707968 >> 172.21.15.40:0/588707968 conn(0x7fd3780b17b0 msgr2=0x7fd3780cede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:22.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 -- 172.21.15.40:0/588707968 shutdown_connections 2024-03-24T00:39:22.302 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:22.296+0000 7fd367400640 1 -- 172.21.15.40:0/588707968 wait complete. 2024-03-24T00:39:22.723 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 130, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:39:07.531884+0000', 'modified': '2024-03-24T00:39:21.870145+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1], 'up': {'mds_0': 8877, 'mds_1': 8874}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8874': {'gid': 8874, 'name': 'c', 'rank': 1, 'incarnation': 130, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.40:6833/4273998586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 4273998586}, {'type': 'v1', 'addr': '172.21.15.40:6833', 'nonce': 4273998586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8877': {'gid': 8877, 'name': 'a', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/773071485', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 773071485}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 773071485}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8877, 'qdb_cluster': [8877]} 2024-03-24T00:39:22.723 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8874:up:creating 2024-03-24T00:39:23.723 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:23.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7569800640 1 Processor -- start 2024-03-24T00:39:23.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7569800640 1 -- start start 2024-03-24T00:39:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7569800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7564161880 0x7f75640b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7569800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75640ce610 0x7f75640cea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7569800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75640d0600 con 0x7f756406b8c0 2024-03-24T00:39:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7569800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7564161c60 con 0x7f7564161880 2024-03-24T00:39:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7569800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756406b150 con 0x7f75640ce610 2024-03-24T00:39:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7562a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7564161880 0x7f75640b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7562a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7564161880 0x7f75640b2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37042/0 (socket says 172.21.15.40:37042) 2024-03-24T00:39:23.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7562a00640 1 -- 172.21.15.40:0/1765899041 learned_addr learned my addr 172.21.15.40:0/1765899041 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f755be00640 1 --2- 172.21.15.40:0/1765899041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75640ce610 0x7f75640cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7562a00640 1 -- 172.21.15.40:0/1765899041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75640ce610 msgr2=0x7f75640cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7562a00640 1 --2- 172.21.15.40:0/1765899041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75640ce610 0x7f75640cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7562a00640 1 -- 172.21.15.40:0/1765899041 >> v1:172.21.15.53:6789/0 conn(0x7f756406b8c0 legacy=0x7f756406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:23.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.860+0000 7f7562a00640 1 -- 172.21.15.40:0/1765899041 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75640d08e0 con 0x7f7564161880 2024-03-24T00:39:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7562a00640 1 --2- 172.21.15.40:0/1765899041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7564161880 0x7f75640b2d50 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f7554009870 tx=0x7f755401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaaffd39657761fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:23.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7562000640 1 -- 172.21.15.40:0/1765899041 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75540b0070 con 0x7f7564161880 2024-03-24T00:39:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7562000640 1 -- 172.21.15.40:0/1765899041 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75540ab600 con 0x7f7564161880 2024-03-24T00:39:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7562000640 1 -- 172.21.15.40:0/1765899041 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75540ab9d0 con 0x7f7564161880 2024-03-24T00:39:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7569800640 1 -- 172.21.15.40:0/1765899041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7564161880 msgr2=0x7f75640b2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:23.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7569800640 1 --2- 172.21.15.40:0/1765899041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7564161880 0x7f75640b2d50 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f7554009870 tx=0x7f755401c490 comp rx=0 tx=0).stop 2024-03-24T00:39:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7569800640 1 -- 172.21.15.40:0/1765899041 shutdown_connections 2024-03-24T00:39:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7569800640 1 --2- 172.21.15.40:0/1765899041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75640ce610 0x7f75640cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7569800640 1 --2- 172.21.15.40:0/1765899041 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7564161880 0x7f75640b2d50 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7569800640 1 -- 172.21.15.40:0/1765899041 >> 172.21.15.40:0/1765899041 conn(0x7f75640b38f0 msgr2=0x7f75640b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7569800640 1 -- 172.21.15.40:0/1765899041 shutdown_connections 2024-03-24T00:39:23.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7569800640 1 -- 172.21.15.40:0/1765899041 wait complete. 2024-03-24T00:39:23.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7569800640 1 Processor -- start 2024-03-24T00:39:23.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.868+0000 7f7569800640 1 -- start start 2024-03-24T00:39:23.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7569800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f756406b8c0 0x7f75641603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:23.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7569800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75640ce610 0x7f7564153730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:23.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7569800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7564161880 0x7f75641547e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:23.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7569800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75640cdd70 con 0x7f756406b8c0 2024-03-24T00:39:23.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7569800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756406b430 con 0x7f7564161880 2024-03-24T00:39:23.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7569800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f756416b0b0 con 0x7f75640ce610 2024-03-24T00:39:23.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7562a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f756406b8c0 0x7f75641603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:23.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7562a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f756406b8c0 0x7f75641603c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37054/0 (socket says 172.21.15.40:37054) 2024-03-24T00:39:23.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7562a00640 1 -- 172.21.15.40:0/3375606420 learned_addr learned my addr 172.21.15.40:0/3375606420 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:23.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f755be00640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75640ce610 0x7f7564153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:23.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7563400640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7564161880 0x7f75641547e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:23.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7562a00640 1 -- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75640ce610 msgr2=0x7f7564153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:23.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7562a00640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75640ce610 0x7f7564153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7562a00640 1 -- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7564161880 msgr2=0x7f75641547e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:23.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7562a00640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7564161880 0x7f75641547e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:23.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7562a00640 1 -- 172.21.15.40:0/3375606420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7564158e50 con 0x7f756406b8c0 2024-03-24T00:39:23.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7562a00640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f756406b8c0 0x7f75641603c0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f755401ab40 tx=0x7f755401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:23.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f755aa00640 1 -- 172.21.15.40:0/3375606420 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75540ba020 con 0x7f756406b8c0 2024-03-24T00:39:23.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f755aa00640 1 -- 172.21.15.40:0/3375606420 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75540b6620 con 0x7f756406b8c0 2024-03-24T00:39:23.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f755aa00640 1 -- 172.21.15.40:0/3375606420 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75540b0040 con 0x7f756406b8c0 2024-03-24T00:39:23.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7569800640 1 -- 172.21.15.40:0/3375606420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75640b4990 con 0x7f756406b8c0 2024-03-24T00:39:23.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7569800640 1 -- 172.21.15.40:0/3375606420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7564154ff0 con 0x7f756406b8c0 2024-03-24T00:39:23.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.872+0000 7f7569800640 1 -- 172.21.15.40:0/3375606420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7564068170 con 0x7f756406b8c0 2024-03-24T00:39:23.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.876+0000 7f755aa00640 1 -- 172.21.15.40:0/3375606420 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f75540bd020 con 0x7f756406b8c0 2024-03-24T00:39:23.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.880+0000 7f755aa00640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7540067690 0x7f7540069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:23.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.880+0000 7f755aa00640 1 -- 172.21.15.40:0/3375606420 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f755411fd30 con 0x7f756406b8c0 2024-03-24T00:39:23.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.880+0000 7f755be00640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7540067690 0x7f7540069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:23.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.880+0000 7f755be00640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7540067690 0x7f7540069b50 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7564157b90 tx=0x7f754c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:23.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:23.880+0000 7f755aa00640 1 -- 172.21.15.40:0/3375606420 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f75540c4030 con 0x7f756406b8c0 2024-03-24T00:39:24.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 -- 172.21.15.40:0/3375606420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75640703d0 con 0x7f756406b8c0 2024-03-24T00:39:24.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f755aa00640 1 -- 172.21.15.40:0/3375606420 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+5020 (secure 0 0 0) 0x7f7554016110 con 0x7f756406b8c0 2024-03-24T00:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:24.088 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":133,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1582567544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1582567544},{"type":"v1","addr":"172.21.15.53:6837","nonce":1582567544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:39:07.531884+0000","modified":"2024-03-24T00:39:23.877796+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8877,"mds_1":8874,"mds_2":8563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8563":{"gid":8563,"name":"b","rank":2,"incarnation":132,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6836/2255755838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2255755838},{"type":"v1","addr":"172.21.15.53:6836","nonce":2255755838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8874":{"gid":8874,"name":"c","rank":1,"incarnation":130,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6833/4273998586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4273998586},{"type":"v1","addr":"172.21.15.40:6833","nonce":4273998586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8877":{"gid":8877,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/773071485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":773071485},{"type":"v1","addr":"172.21.15.40:6835","nonce":773071485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8877,"qdb_cluster":[8563,8874,8877]},"id":9}]} 2024-03-24T00:39:24.088 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 133 2024-03-24T00:39:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 -- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7540067690 msgr2=0x7f7540069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7540067690 0x7f7540069b50 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7564157b90 tx=0x7f754c008040 comp rx=0 tx=0).stop 2024-03-24T00:39:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 -- 172.21.15.40:0/3375606420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f756406b8c0 msgr2=0x7f75641603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f756406b8c0 0x7f75641603c0 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f755401ab40 tx=0x7f755401acd0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 -- 172.21.15.40:0/3375606420 shutdown_connections 2024-03-24T00:39:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7540067690 0x7f7540069b50 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7564161880 0x7f75641547e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75640ce610 0x7f7564153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 --2- 172.21.15.40:0/3375606420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f756406b8c0 0x7f75641603c0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 -- 172.21.15.40:0/3375606420 >> 172.21.15.40:0/3375606420 conn(0x7f75640b38f0 msgr2=0x7f756415bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 -- 172.21.15.40:0/3375606420 shutdown_connections 2024-03-24T00:39:24.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.084+0000 7f7569800640 1 -- 172.21.15.40:0/3375606420 wait complete. 2024-03-24T00:39:24.512 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 133, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:39:07.531884+0000', 'modified': '2024-03-24T00:39:23.877796+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 8877, 'mds_1': 8874, 'mds_2': 8563}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8563': {'gid': 8563, 'name': 'b', 'rank': 2, 'incarnation': 132, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.53:6836/2255755838', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 2255755838}, {'type': 'v1', 'addr': '172.21.15.53:6836', 'nonce': 2255755838}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8874': {'gid': 8874, 'name': 'c', 'rank': 1, 'incarnation': 130, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.40:6833/4273998586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 4273998586}, {'type': 'v1', 'addr': '172.21.15.40:6833', 'nonce': 4273998586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8877': {'gid': 8877, 'name': 'a', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/773071485', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 773071485}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 773071485}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8877, 'qdb_cluster': [8563, 8874, 8877]} 2024-03-24T00:39:24.512 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T00:39:24.512 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:39:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.648+0000 7f01a5000640 1 Processor -- start 2024-03-24T00:39:24.654 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.648+0000 7f01a5000640 1 -- start start 2024-03-24T00:39:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f01a5000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f01a00cfdf0 0x7f01a00d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f01a5000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01a00b43a0 0x7f01a00d0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f01a5000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a00af710 con 0x7f01a015f090 2024-03-24T00:39:24.656 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f01a5000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a00b2bb0 con 0x7f01a00b43a0 2024-03-24T00:39:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f01a5000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00b26a0 con 0x7f01a00cfdf0 2024-03-24T00:39:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f0197e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01a00b43a0 0x7f01a00d0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f0197e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f01a00b43a0 0x7f01a00d0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37070/0 (socket says 172.21.15.40:37070) 2024-03-24T00:39:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f0197e00640 1 -- 172.21.15.40:0/3340466831 learned_addr learned my addr 172.21.15.40:0/3340466831 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f019e800640 1 --1- 172.21.15.40:0/3340466831 >> v1:172.21.15.53:6789/0 conn(0x7f01a015f090 0x7f01a015f4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58570/0 (socket says 172.21.15.40:58570) 2024-03-24T00:39:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f019de00640 1 --2- 172.21.15.40:0/3340466831 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f01a00cfdf0 0x7f01a00d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f0197e00640 1 -- 172.21.15.40:0/3340466831 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f01a00cfdf0 msgr2=0x7f01a00d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f0197e00640 1 --2- 172.21.15.40:0/3340466831 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f01a00cfdf0 0x7f01a00d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.657 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f0197e00640 1 -- 172.21.15.40:0/3340466831 >> v1:172.21.15.53:6789/0 conn(0x7f01a015f090 legacy=0x7f01a015f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.658 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.652+0000 7f0197e00640 1 -- 172.21.15.40:0/3340466831 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a00b34f0 con 0x7f01a00b43a0 2024-03-24T00:39:24.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.656+0000 7f0197e00640 1 --2- 172.21.15.40:0/3340466831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a00b43a0 0x7f01a00d0710 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f0188009870 tx=0x7f018801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24325083b5bf38f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.656+0000 7f019d400640 1 -- 172.21.15.40:0/3340466831 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01880b2070 con 0x7f01a00b43a0 2024-03-24T00:39:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.656+0000 7f019d400640 1 -- 172.21.15.40:0/3340466831 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01880a9070 con 0x7f01a00b43a0 2024-03-24T00:39:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.656+0000 7f019d400640 1 -- 172.21.15.40:0/3340466831 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01880ad880 con 0x7f01a00b43a0 2024-03-24T00:39:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.656+0000 7f01a5000640 1 -- 172.21.15.40:0/3340466831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a00b43a0 msgr2=0x7f01a00d0710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.656+0000 7f01a5000640 1 --2- 172.21.15.40:0/3340466831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a00b43a0 0x7f01a00d0710 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f0188009870 tx=0x7f018801c560 comp rx=0 tx=0).stop 2024-03-24T00:39:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.656+0000 7f01a5000640 1 -- 172.21.15.40:0/3340466831 shutdown_connections 2024-03-24T00:39:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.656+0000 7f01a5000640 1 --2- 172.21.15.40:0/3340466831 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a00b43a0 0x7f01a00d0710 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.656+0000 7f01a5000640 1 --2- 172.21.15.40:0/3340466831 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f01a00cfdf0 0x7f01a00d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.656+0000 7f01a5000640 1 -- 172.21.15.40:0/3340466831 >> 172.21.15.40:0/3340466831 conn(0x7f01a00b3880 msgr2=0x7f01a00b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.656+0000 7f01a5000640 1 -- 172.21.15.40:0/3340466831 shutdown_connections 2024-03-24T00:39:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.656+0000 7f01a5000640 1 -- 172.21.15.40:0/3340466831 wait complete. 2024-03-24T00:39:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f01a5000640 1 Processor -- start 2024-03-24T00:39:24.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f01a5000640 1 -- start start 2024-03-24T00:39:24.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f01a5000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01a00b43a0 0x7f01a0157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f01a5000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f01a00cfdf0 0x7f01a01583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f01a5000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a015f090 0x7f01a0165f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f01a5000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00b07d0 con 0x7f01a015f090 2024-03-24T00:39:24.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f01a5000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00b0650 con 0x7f01a00b43a0 2024-03-24T00:39:24.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f01a5000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a00b0950 con 0x7f01a00cfdf0 2024-03-24T00:39:24.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f019de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01a00b43a0 0x7f01a0157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f019e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a015f090 0x7f01a0165f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f0197e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f01a00cfdf0 0x7f01a01583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f019de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01a00b43a0 0x7f01a0157300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:46832/0 (socket says 172.21.15.40:46832) 2024-03-24T00:39:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f019de00640 1 -- 172.21.15.40:0/1734990845 learned_addr learned my addr 172.21.15.40:0/1734990845 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f019de00640 1 -- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f01a00cfdf0 msgr2=0x7f01a01583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f019de00640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f01a00cfdf0 0x7f01a01583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f019de00640 1 -- 172.21.15.40:0/1734990845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a015f090 msgr2=0x7f01a0165f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f019de00640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a015f090 0x7f01a0165f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f019de00640 1 -- 172.21.15.40:0/1734990845 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a01666f0 con 0x7f01a00b43a0 2024-03-24T00:39:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f019e800640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a015f090 0x7f01a0165f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f019de00640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01a00b43a0 0x7f01a0157300 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f019000e930 tx=0x7f019000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f0196000640 1 -- 172.21.15.40:0/1734990845 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190017070 con 0x7f01a00b43a0 2024-03-24T00:39:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f0196000640 1 -- 172.21.15.40:0/1734990845 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019000ca00 con 0x7f01a00b43a0 2024-03-24T00:39:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f01a5000640 1 -- 172.21.15.40:0/1734990845 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a01669d0 con 0x7f01a00b43a0 2024-03-24T00:39:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.660+0000 7f0196000640 1 -- 172.21.15.40:0/1734990845 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019000cdb0 con 0x7f01a00b43a0 2024-03-24T00:39:24.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.664+0000 7f01a5000640 1 -- 172.21.15.40:0/1734990845 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a0159470 con 0x7f01a00b43a0 2024-03-24T00:39:24.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.664+0000 7f0196000640 1 -- 172.21.15.40:0/1734990845 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f019001a070 con 0x7f01a00b43a0 2024-03-24T00:39:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.664+0000 7f0196000640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f017c067690 0x7f017c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.664+0000 7f0196000640 1 -- 172.21.15.40:0/1734990845 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0190085840 con 0x7f01a00b43a0 2024-03-24T00:39:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 Processor -- start 2024-03-24T00:39:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f0197e00640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f017c067690 0x7f017c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 -- start start 2024-03-24T00:39:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f0197e00640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f017c067690 0x7f017c069b50 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f01880af000 tx=0x7f0188008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 0x55a5bd183d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd184240 0x55a5bd139ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd13a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5bcf87680 con 0x55a5bd183920 2024-03-24T00:39:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5bcf0eb00 con 0x55a5bd184240 2024-03-24T00:39:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a5bcf83720 con 0x55a5bd139ff0 2024-03-24T00:39:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f019de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 0x55a5bd183d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f019de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 0x55a5bd183d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37080/0 (socket says 172.21.15.40:37080) 2024-03-24T00:39:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f019de00640 1 -- 172.21.15.40:0/16131748 learned_addr learned my addr 172.21.15.40:0/16131748 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:24.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f019e800640 1 --2- 172.21.15.40:0/16131748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd13a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f0197e00640 1 --2- 172.21.15.40:0/16131748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd184240 0x55a5bd139ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f019de00640 1 -- 172.21.15.40:0/16131748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 msgr2=0x55a5bd13a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f019de00640 1 --2- 172.21.15.40:0/16131748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd13a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f019de00640 1 -- 172.21.15.40:0/16131748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd184240 msgr2=0x55a5bd139ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f019de00640 1 --2- 172.21.15.40:0/16131748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd184240 0x55a5bd139ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f019de00640 1 -- 172.21.15.40:0/16131748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5bcf5f940 con 0x55a5bd183920 2024-03-24T00:39:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f019de00640 1 --2- 172.21.15.40:0/16131748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 0x55a5bd183d00 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f0190052380 tx=0x7f0190056060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf451c0a4756dfa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f018ea00640 1 -- 172.21.15.40:0/16131748 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190021040 con 0x55a5bd183920 2024-03-24T00:39:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f018ea00640 1 -- 172.21.15.40:0/16131748 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0190054220 con 0x55a5bd183920 2024-03-24T00:39:24.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f018ea00640 1 -- 172.21.15.40:0/16131748 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019005c070 con 0x55a5bd183920 2024-03-24T00:39:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 -- 172.21.15.40:0/16131748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 msgr2=0x55a5bd183d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 --2- 172.21.15.40:0/16131748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 0x55a5bd183d00 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f0190052380 tx=0x7f0190056060 comp rx=0 tx=0).stop 2024-03-24T00:39:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 -- 172.21.15.40:0/16131748 shutdown_connections 2024-03-24T00:39:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 --2- 172.21.15.40:0/16131748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd13a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 --2- 172.21.15.40:0/16131748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd184240 0x55a5bd139ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 --2- 172.21.15.40:0/16131748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 0x55a5bd183d00 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 -- 172.21.15.40:0/16131748 >> 172.21.15.40:0/16131748 conn(0x55a5bd135fa0 msgr2=0x55a5bd138390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 -- 172.21.15.40:0/16131748 shutdown_connections 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 -- 172.21.15.40:0/16131748 wait complete. 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 -- start start 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.668+0000 7f01a84da000 1 Processor -- start 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd168d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 0x55a5bd169280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd184240 0x55a5bd1697c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5bcf86990 con 0x55a5bd183920 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5bcf41ae0 con 0x55a5bd184240 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a5bcf1f750 con 0x55a5bd139ff0 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 10 client.0 _ll_register_callbacks cb 0x55a5bd124ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f0197e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 0x55a5bd169280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f019de00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd168d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f019de00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd168d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:55180/0 (socket says 172.21.15.40:55180) 2024-03-24T00:39:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f0197e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 0x55a5bd169280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37094/0 (socket says 172.21.15.40:37094) 2024-03-24T00:39:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f019e800640 1 --2- 172.21.15.40:0/2386879009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd184240 0x55a5bd1697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f019de00640 1 -- 172.21.15.40:0/2386879009 learned_addr learned my addr 172.21.15.40:0/2386879009 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f019de00640 1 -- 172.21.15.40:0/2386879009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd184240 msgr2=0x55a5bd1697c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f019de00640 1 --2- 172.21.15.40:0/2386879009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd184240 0x55a5bd1697c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f019de00640 1 -- 172.21.15.40:0/2386879009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 msgr2=0x55a5bd169280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f019de00640 1 --2- 172.21.15.40:0/2386879009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 0x55a5bd169280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f019de00640 1 -- 172.21.15.40:0/2386879009 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a5bcfccd80 con 0x55a5bd139ff0 2024-03-24T00:39:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f019de00640 1 --2- 172.21.15.40:0/2386879009 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd168d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0190052330 tx=0x7f0190000bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fa256c01752abc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f0183400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:39:24.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f0183400640 1 -- 172.21.15.40:0/2386879009 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0190071070 con 0x55a5bd139ff0 2024-03-24T00:39:24.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f0183400640 1 -- 172.21.15.40:0/2386879009 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0190055570 con 0x55a5bd139ff0 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 1 -- 172.21.15.40:0/2386879009 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0190054220 con 0x55a5bd139ff0 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f0183400640 1 -- 172.21.15.40:0/2386879009 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=133) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0190021070 con 0x55a5bd139ff0 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 10 client.7583 fetch_fsmap learned FSMap version 133 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 1 -- 172.21.15.40:0/2386879009 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=133,monmap=2+}) v3 -- 0x55a5bcf0ed10 con 0x55a5bd139ff0 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f0183400640 1 -- 172.21.15.40:0/2386879009 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0190054220 con 0x55a5bd139ff0 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f0183400640 1 -- 172.21.15.40:0/2386879009 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 133) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f0190024060 con 0x55a5bd139ff0 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 10 client.7583 fetch_fsmap finished waiting for FSMap version 133 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 10 client.7583 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 20 client.7583 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 1 --2- 172.21.15.40:0/2386879009 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x55a5bd16a090 0x55a5bd195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 4 client.7583 mds_command: new command op to 8563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f01a84da000 1 -- 172.21.15.40:0/2386879009 --> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a5bcf22b60 con 0x55a5bd16a090 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f0197e00640 1 --2- 172.21.15.40:0/2386879009 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x55a5bd16a090 0x55a5bd195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f0197e00640 1 --2- 172.21.15.40:0/2386879009 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x55a5bd16a090 0x55a5bd195ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4a72c99cc25aafb3 server_cookie=53c6d6d277692ce1 in_seq=0 out_seq=0 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.672+0000 7f0183400640 10 client.7583 ms_handle_connect on v2:172.21.15.53:6834/2255755838 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f0183400640 1 -- 172.21.15.40:0/2386879009 <== mds.2 v2:172.21.15.53:6834/2255755838 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a5bcf22b60 con 0x55a5bd16a090 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f0183400640 10 client.7583 handle_command_reply: tid=0 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f01a84da000 1 client.7583 shutdown 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f01a84da000 1 -- 172.21.15.40:0/2386879009 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 msgr2=0x55a5bd168d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f01a84da000 1 --2- 172.21.15.40:0/2386879009 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd168d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0190052330 tx=0x7f0190000bd0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f01a84da000 1 -- 172.21.15.40:0/2386879009 shutdown_connections 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f01a84da000 1 --2- 172.21.15.40:0/2386879009 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x55a5bd16a090 0x55a5bd195ea0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f01a84da000 1 --2- 172.21.15.40:0/2386879009 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd184240 0x55a5bd1697c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f01a84da000 1 --2- 172.21.15.40:0/2386879009 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd183920 0x55a5bd169280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f0183400640 0 client.7583 ms_handle_reset on v2:172.21.15.53:6834/2255755838 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f01a84da000 1 --2- 172.21.15.40:0/2386879009 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd168d40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f01a84da000 1 -- 172.21.15.40:0/2386879009 >> 172.21.15.40:0/2386879009 conn(0x55a5bd135fa0 msgr2=0x55a5bd138c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f01a84da000 1 -- 172.21.15.40:0/2386879009 shutdown_connections 2024-03-24T00:39:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f01a84da000 1 -- 172.21.15.40:0/2386879009 wait complete. 2024-03-24T00:39:24.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.676+0000 7f01a84da000 20 client.7583 trim_cache size 0 max 16384 2024-03-24T00:39:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f01a84da000 1 Processor -- start 2024-03-24T00:39:24.696 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f01a84da000 1 -- start start 2024-03-24T00:39:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f01a84da000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd168ed0 0x55a5bd1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f01a84da000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd135fa0 0x55a5bd182530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f01a84da000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd139ff0 0x55a5bd182a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f01a84da000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a02028b0 con 0x55a5bd139ff0 2024-03-24T00:39:24.697 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f01a84da000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a006c5b0 con 0x55a5bd168ed0 2024-03-24T00:39:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f01a84da000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a5bcf2fa80 con 0x55a5bd135fa0 2024-03-24T00:39:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f0197e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd135fa0 0x55a5bd182530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f019de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd168ed0 0x55a5bd1397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f019e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd139ff0 0x55a5bd182a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f0197e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd135fa0 0x55a5bd182530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:55182/0 (socket says 172.21.15.40:55182) 2024-03-24T00:39:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f019e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd139ff0 0x55a5bd182a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37100/0 (socket says 172.21.15.40:37100) 2024-03-24T00:39:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f019de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd168ed0 0x55a5bd1397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:46878/0 (socket says 172.21.15.40:46878) 2024-03-24T00:39:24.698 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f0197e00640 1 -- 172.21.15.40:0/219847967 learned_addr learned my addr 172.21.15.40:0/219847967 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f0197e00640 1 -- 172.21.15.40:0/219847967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd168ed0 msgr2=0x55a5bd1397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f0197e00640 1 --2- 172.21.15.40:0/219847967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd168ed0 0x55a5bd1397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f0197e00640 1 -- 172.21.15.40:0/219847967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd139ff0 msgr2=0x55a5bd182a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f0197e00640 1 --2- 172.21.15.40:0/219847967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd139ff0 0x55a5bd182a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f0197e00640 1 -- 172.21.15.40:0/219847967 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5bd13e1c0 con 0x55a5bd135fa0 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f019de00640 1 --2- 172.21.15.40:0/219847967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd168ed0 0x55a5bd1397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f019e800640 1 --2- 172.21.15.40:0/219847967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd139ff0 0x55a5bd182a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f0197e00640 1 --2- 172.21.15.40:0/219847967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd135fa0 0x55a5bd182530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0188008010 tx=0x7f0188004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f34d745ef6f40c2a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f018e000640 1 -- 172.21.15.40:0/219847967 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01880cce00 con 0x55a5bd135fa0 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f018e000640 1 -- 172.21.15.40:0/219847967 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01880a9040 con 0x55a5bd135fa0 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.692+0000 7f018e000640 1 -- 172.21.15.40:0/219847967 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01880ad7c0 con 0x55a5bd135fa0 2024-03-24T00:39:24.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 -- 172.21.15.40:0/219847967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd135fa0 msgr2=0x55a5bd182530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 --2- 172.21.15.40:0/219847967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd135fa0 0x55a5bd182530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f0188008010 tx=0x7f0188004740 comp rx=0 tx=0).stop 2024-03-24T00:39:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 -- 172.21.15.40:0/219847967 shutdown_connections 2024-03-24T00:39:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 --2- 172.21.15.40:0/219847967 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd139ff0 0x55a5bd182a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 --2- 172.21.15.40:0/219847967 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd135fa0 0x55a5bd182530 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 --2- 172.21.15.40:0/219847967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd168ed0 0x55a5bd1397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 -- 172.21.15.40:0/219847967 >> 172.21.15.40:0/219847967 conn(0x55a5bd13c750 msgr2=0x55a5bd1665d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 -- 172.21.15.40:0/219847967 shutdown_connections 2024-03-24T00:39:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 -- 172.21.15.40:0/219847967 wait complete. 2024-03-24T00:39:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 -- start start 2024-03-24T00:39:24.701 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 Processor -- start 2024-03-24T00:39:24.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd135fa0 0x55a5bd1e95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd1e9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd168ed0 0x55a5bd196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5bcf41ae0 con 0x55a5bd168ed0 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5bd044700 con 0x55a5bd135fa0 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a5bcf1f750 con 0x55a5bd139ff0 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 10 client.0 _ll_register_callbacks cb 0x7f0190009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f019e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd168ed0 0x55a5bd196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f019e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd168ed0 0x55a5bd196ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37108/0 (socket says 172.21.15.40:37108) 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f019e800640 1 -- 172.21.15.40:0/2921616119 learned_addr learned my addr 172.21.15.40:0/2921616119 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f0197e00640 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd1e9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f019de00640 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd135fa0 0x55a5bd1e95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f0197e00640 1 -- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd135fa0 msgr2=0x55a5bd1e95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f0197e00640 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd135fa0 0x55a5bd1e95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f0197e00640 1 -- 172.21.15.40:0/2921616119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd168ed0 msgr2=0x55a5bd196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f0197e00640 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd168ed0 0x55a5bd196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f0197e00640 1 -- 172.21.15.40:0/2921616119 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a5bcf54540 con 0x55a5bd139ff0 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f019e800640 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd168ed0 0x55a5bd196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f0197e00640 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd1e9ae0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f018801a190 tx=0x7f018801ada0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7417c727b9f59c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f0183400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f0183400640 1 -- 172.21.15.40:0/2921616119 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01880dc060 con 0x55a5bd139ff0 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 -- 172.21.15.40:0/2921616119 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f01880a9040 con 0x55a5bd139ff0 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f0183400640 1 -- 172.21.15.40:0/2921616119 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01880cce00 con 0x55a5bd139ff0 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f0183400640 1 -- 172.21.15.40:0/2921616119 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=133) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01880cc2a0 con 0x55a5bd139ff0 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 10 client.7589 fetch_fsmap learned FSMap version 133 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.696+0000 7f01a84da000 1 -- 172.21.15.40:0/2921616119 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=133,monmap=2+}) v3 -- 0x55a5bcf1b4b0 con 0x55a5bd139ff0 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f0183400640 1 -- 172.21.15.40:0/2921616119 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01880a9040 con 0x55a5bd139ff0 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f0183400640 1 -- 172.21.15.40:0/2921616119 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 133) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f01880ad7c0 con 0x55a5bd139ff0 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 10 client.7589 fetch_fsmap finished waiting for FSMap version 133 2024-03-24T00:39:24.705 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 10 client.7589 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:39:24.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 20 client.7589 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:24.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x55a5bd183920 0x55a5bd1f2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 4 client.7589 mds_command: new command op to 8563 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:39:24.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 -- 172.21.15.40:0/2921616119 --> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a5bcf94bc0 con 0x55a5bd183920 2024-03-24T00:39:24.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f019de00640 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x55a5bd183920 0x55a5bd1f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f0183400640 10 client.7589 ms_handle_connect on v2:172.21.15.53:6834/2255755838 2024-03-24T00:39:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f019de00640 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x55a5bd183920 0x55a5bd1f2410 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=211ce73bdcaeeef6 server_cookie=5d17b8a58c59b813 in_seq=0 out_seq=0 2024-03-24T00:39:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f0183400640 1 -- 172.21.15.40:0/2921616119 <== mds.2 v2:172.21.15.53:6834/2255755838 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x55a5bcf94bc0 con 0x55a5bd183920 2024-03-24T00:39:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f0183400640 10 client.7589 handle_command_reply: tid=0 2024-03-24T00:39:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 client.7589 shutdown 2024-03-24T00:39:24.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 -- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 msgr2=0x55a5bd1e9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd1e9ae0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f018801a190 tx=0x7f018801ada0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 -- 172.21.15.40:0/2921616119 shutdown_connections 2024-03-24T00:39:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x55a5bd183920 0x55a5bd1f2410 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55a5bd168ed0 0x55a5bd196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55a5bd139ff0 0x55a5bd1e9ae0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 --2- 172.21.15.40:0/2921616119 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a5bd135fa0 0x55a5bd1e95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f0183400640 0 client.7589 ms_handle_reset on v2:172.21.15.53:6834/2255755838 2024-03-24T00:39:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 -- 172.21.15.40:0/2921616119 >> 172.21.15.40:0/2921616119 conn(0x55a5bd13c750 msgr2=0x55a5bd166e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 -- 172.21.15.40:0/2921616119 shutdown_connections 2024-03-24T00:39:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 1 -- 172.21.15.40:0/2921616119 wait complete. 2024-03-24T00:39:24.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a84da000 20 client.7589 trim_cache size 0 max 16384 2024-03-24T00:39:24.708 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:39:24.709 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:39:24.709 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-03-24T00:39:24.709 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8563, 2024-03-24T00:39:24.709 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:39:24.709 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:39:24.709 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:39:24.709 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 1.828114421, 2024-03-24T00:39:24.709 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 133, 2024-03-24T00:39:24.709 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 119, 2024-03-24T00:39:24.709 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 119, 2024-03-24T00:39:24.709 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 21.418838724 2024-03-24T00:39:24.709 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:39:24.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a5000640 1 -- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f017c067690 msgr2=0x7f017c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a5000640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f017c067690 0x7f017c069b50 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f01880af000 tx=0x7f0188008040 comp rx=0 tx=0).stop 2024-03-24T00:39:24.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a5000640 1 -- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01a00b43a0 msgr2=0x7f01a0157300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a5000640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01a00b43a0 0x7f01a0157300 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f019000e930 tx=0x7f019000edf0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a5000640 1 -- 172.21.15.40:0/1734990845 shutdown_connections 2024-03-24T00:39:24.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a5000640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f017c067690 0x7f017c069b50 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a5000640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f01a015f090 0x7f01a0165f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a5000640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f01a00cfdf0 0x7f01a01583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a5000640 1 --2- 172.21.15.40:0/1734990845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01a00b43a0 0x7f01a0157300 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a5000640 1 -- 172.21.15.40:0/1734990845 >> 172.21.15.40:0/1734990845 conn(0x7f01a00b3880 msgr2=0x7f01a015ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a5000640 1 -- 172.21.15.40:0/1734990845 shutdown_connections 2024-03-24T00:39:24.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.700+0000 7f01a5000640 1 -- 172.21.15.40:0/1734990845 wait complete. 2024-03-24T00:39:24.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:39:24.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.868+0000 7f5720800640 1 Processor -- start 2024-03-24T00:39:24.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.868+0000 7f5720800640 1 -- start start 2024-03-24T00:39:24.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f5720800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f571815e0b0 0x7f57180b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f5720800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f571806bd10 0x7f57180cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f5720800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57180cd3d0 con 0x7f57180b4b80 2024-03-24T00:39:24.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f5720800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f571815e490 con 0x7f571806bd10 2024-03-24T00:39:24.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f5720800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57180cdfe0 con 0x7f571815e0b0 2024-03-24T00:39:24.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f571de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f571815e0b0 0x7f57180b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f571e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f571806bd10 0x7f57180cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f571de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f571815e0b0 0x7f57180b43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:46894/0 (socket says 172.21.15.40:46894) 2024-03-24T00:39:24.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f571de00640 1 -- 172.21.15.40:0/3145398980 learned_addr learned my addr 172.21.15.40:0/3145398980 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:24.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f571d400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f57180b4b80 0x7f571806b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58584/0 (socket says 172.21.15.40:58584) 2024-03-24T00:39:24.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f571e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f571806bd10 0x7f57180cccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37120/0 (socket says 172.21.15.40:37120) 2024-03-24T00:39:24.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f571e800640 1 -- 172.21.15.40:0/3145398980 >> v2:172.21.15.53:3300/0 conn(0x7f571815e0b0 msgr2=0x7f57180b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f571e800640 1 --2- 172.21.15.40:0/3145398980 >> v2:172.21.15.53:3300/0 conn(0x7f571815e0b0 0x7f57180b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f571e800640 1 -- 172.21.15.40:0/3145398980 >> v1:172.21.15.53:6789/0 conn(0x7f57180b4b80 legacy=0x7f571806b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f571ca00640 1 -- 172.21.15.40:0/3145398980 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144033625 0 0) 0x7f57180cd3d0 con 0x7f57180b4b80 2024-03-24T00:39:24.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f571e800640 1 -- 172.21.15.40:0/3145398980 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57180cd6b0 con 0x7f571806bd10 2024-03-24T00:39:24.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.872+0000 7f571de00640 1 --2- 172.21.15.40:0/3145398980 >> v2:172.21.15.53:3300/0 conn(0x7f571815e0b0 0x7f57180b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:24.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f571e800640 1 --2- 172.21.15.40:0/3145398980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571806bd10 0x7f57180cccf0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f5718067e60 tx=0x7f571401f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a756fb67e1e741c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f571ca00640 1 -- 172.21.15.40:0/3145398980 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57140b4070 con 0x7f571806bd10 2024-03-24T00:39:24.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f571ca00640 1 -- 172.21.15.40:0/3145398980 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57140af600 con 0x7f571806bd10 2024-03-24T00:39:24.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f571ca00640 1 -- 172.21.15.40:0/3145398980 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57140af9d0 con 0x7f571806bd10 2024-03-24T00:39:24.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 -- 172.21.15.40:0/3145398980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571806bd10 msgr2=0x7f57180cccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 --2- 172.21.15.40:0/3145398980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571806bd10 0x7f57180cccf0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f5718067e60 tx=0x7f571401f110 comp rx=0 tx=0).stop 2024-03-24T00:39:24.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 -- 172.21.15.40:0/3145398980 shutdown_connections 2024-03-24T00:39:24.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 --2- 172.21.15.40:0/3145398980 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571806bd10 0x7f57180cccf0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 --2- 172.21.15.40:0/3145398980 >> v2:172.21.15.53:3300/0 conn(0x7f571815e0b0 0x7f57180b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 -- 172.21.15.40:0/3145398980 >> 172.21.15.40:0/3145398980 conn(0x7f57180b3880 msgr2=0x7f57180b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 -- 172.21.15.40:0/3145398980 shutdown_connections 2024-03-24T00:39:24.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 -- 172.21.15.40:0/3145398980 wait complete. 2024-03-24T00:39:24.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 Processor -- start 2024-03-24T00:39:24.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 -- start start 2024-03-24T00:39:24.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f571806bd10 0x7f571815fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180b4b80 0x7f57180cc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571815e0b0 0x7f57180b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571806c480 con 0x7f571815e0b0 2024-03-24T00:39:24.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57180cdc70 con 0x7f571806bd10 2024-03-24T00:39:24.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.876+0000 7f5720800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57180b6ff0 con 0x7f57180b4b80 2024-03-24T00:39:24.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571815e0b0 0x7f57180b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f571806bd10 0x7f571815fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571815e0b0 0x7f57180b4fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37136/0 (socket says 172.21.15.40:37136) 2024-03-24T00:39:24.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571d400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180b4b80 0x7f57180cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f571806bd10 0x7f571815fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:46898/0 (socket says 172.21.15.40:46898) 2024-03-24T00:39:24.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571e800640 1 -- 172.21.15.40:0/2488451770 learned_addr learned my addr 172.21.15.40:0/2488451770 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571de00640 1 -- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180b4b80 msgr2=0x7f57180cc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571de00640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180b4b80 0x7f57180cc570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571de00640 1 -- 172.21.15.40:0/2488451770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571815e0b0 msgr2=0x7f57180b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571de00640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571815e0b0 0x7f57180b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571de00640 1 -- 172.21.15.40:0/2488451770 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57181f82b0 con 0x7f571806bd10 2024-03-24T00:39:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571e800640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571815e0b0 0x7f57180b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f571de00640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f571806bd10 0x7f571815fb70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f570c00e8b0 tx=0x7f570c00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f5712000640 1 -- 172.21.15.40:0/2488451770 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c004030 con 0x7f571806bd10 2024-03-24T00:39:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f5712000640 1 -- 172.21.15.40:0/2488451770 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c0041d0 con 0x7f571806bd10 2024-03-24T00:39:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f5720800640 1 -- 172.21.15.40:0/2488451770 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571815f090 con 0x7f571806bd10 2024-03-24T00:39:24.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f5712000640 1 -- 172.21.15.40:0/2488451770 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c00c940 con 0x7f571806bd10 2024-03-24T00:39:24.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.880+0000 7f5720800640 1 -- 172.21.15.40:0/2488451770 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571815f650 con 0x7f571806bd10 2024-03-24T00:39:24.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f5712000640 1 -- 172.21.15.40:0/2488451770 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f570c00cae0 con 0x7f571806bd10 2024-03-24T00:39:24.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f5712000640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56fc067660 0x7f56fc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f5712000640 1 -- 172.21.15.40:0/2488451770 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f570c085700 con 0x7f571806bd10 2024-03-24T00:39:24.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f5723c39000 1 Processor -- start 2024-03-24T00:39:24.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f5723c39000 1 -- start start 2024-03-24T00:39:24.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f571d400640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56fc067660 0x7f56fc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f571d400640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56fc067660 0x7f56fc069b20 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f57180c6a50 tx=0x7f5708007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f5723c39000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64754920 0x561b64754d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f5723c39000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 0x561b6470aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f5723c39000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 0x561b6470b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f5723c39000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b64558680 con 0x561b64755240 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f5723c39000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b644dfb00 con 0x561b64754920 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f5723c39000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b64554720 con 0x561b6470aff0 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f571d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 0x561b6470aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f571d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 0x561b6470aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37138/0 (socket says 172.21.15.40:37138) 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f571d400640 1 -- 172.21.15.40:0/3813354767 learned_addr learned my addr 172.21.15.40:0/3813354767 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f571de00640 1 --2- 172.21.15.40:0/3813354767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64754920 0x561b64754d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f571e800640 1 --2- 172.21.15.40:0/3813354767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 0x561b6470b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f571d400640 1 -- 172.21.15.40:0/3813354767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 msgr2=0x561b6470b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f571d400640 1 --2- 172.21.15.40:0/3813354767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 0x561b6470b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f571d400640 1 -- 172.21.15.40:0/3813354767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64754920 msgr2=0x561b64754d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.884+0000 7f571d400640 1 --2- 172.21.15.40:0/3813354767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64754920 0x561b64754d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f571d400640 1 -- 172.21.15.40:0/3813354767 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b64530940 con 0x561b64755240 2024-03-24T00:39:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f571d400640 1 --2- 172.21.15.40:0/3813354767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 0x561b6470aab0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f57180b4fa0 tx=0x7f5708013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddceddb7f944b332 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5706a00640 1 -- 172.21.15.40:0/3813354767 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5708026070 con 0x561b64755240 2024-03-24T00:39:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5706a00640 1 -- 172.21.15.40:0/3813354767 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5708013b00 con 0x561b64755240 2024-03-24T00:39:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5706a00640 1 -- 172.21.15.40:0/3813354767 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5708012350 con 0x561b64755240 2024-03-24T00:39:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 -- 172.21.15.40:0/3813354767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 msgr2=0x561b6470aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 --2- 172.21.15.40:0/3813354767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 0x561b6470aab0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f57180b4fa0 tx=0x7f5708013000 comp rx=0 tx=0).stop 2024-03-24T00:39:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 -- 172.21.15.40:0/3813354767 shutdown_connections 2024-03-24T00:39:24.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 --2- 172.21.15.40:0/3813354767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 0x561b6470b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 --2- 172.21.15.40:0/3813354767 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 0x561b6470aab0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 --2- 172.21.15.40:0/3813354767 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64754920 0x561b64754d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 -- 172.21.15.40:0/3813354767 >> 172.21.15.40:0/3813354767 conn(0x561b64706fa0 msgr2=0x561b64709390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 -- 172.21.15.40:0/3813354767 shutdown_connections 2024-03-24T00:39:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 -- 172.21.15.40:0/3813354767 wait complete. 2024-03-24T00:39:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 -- start start 2024-03-24T00:39:24.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 Processor -- start 2024-03-24T00:39:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b6470aff0 0x561b64739d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b64754920 0x561b6473a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 0x561b6473a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b64557990 con 0x561b64755240 2024-03-24T00:39:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b64512ae0 con 0x561b6470aff0 2024-03-24T00:39:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b644f0750 con 0x561b64754920 2024-03-24T00:39:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f5723c39000 10 client.0 _ll_register_callbacks cb 0x561b646f5ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f571e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 0x561b6473a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f571de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b6470aff0 0x561b64739d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f571d400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b64754920 0x561b6473a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f571e800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 0x561b6473a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37148/0 (socket says 172.21.15.40:37148) 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.888+0000 7f571e800640 1 -- 172.21.15.40:0/206555498 learned_addr learned my addr 172.21.15.40:0/206555498 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f571e800640 1 -- 172.21.15.40:0/206555498 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b64754920 msgr2=0x561b6473a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f571e800640 1 --2- 172.21.15.40:0/206555498 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b64754920 0x561b6473a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f571e800640 1 -- 172.21.15.40:0/206555498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b6470aff0 msgr2=0x561b64739d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f571e800640 1 --2- 172.21.15.40:0/206555498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b6470aff0 0x561b64739d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f571e800640 1 -- 172.21.15.40:0/206555498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b6459dd80 con 0x561b64755240 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f571de00640 1 --2- 172.21.15.40:0/206555498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b6470aff0 0x561b64739d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f571e800640 1 --2- 172.21.15.40:0/206555498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 0x561b6473a7c0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f571409ba40 tx=0x7f571401d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be64dd7fa683577d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f56fb400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f56fb400640 1 -- 172.21.15.40:0/206555498 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57140b4040 con 0x561b64755240 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 -- 172.21.15.40:0/206555498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5708013b00 con 0x561b64755240 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f56fb400640 1 -- 172.21.15.40:0/206555498 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57140afd60 con 0x561b64755240 2024-03-24T00:39:24.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f56fb400640 1 -- 172.21.15.40:0/206555498 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57140ce050 con 0x561b64755240 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 10 client.9093 fetch_fsmap learned FSMap version 133 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 -- 172.21.15.40:0/206555498 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=2+}) v3 -- 0x561b644dfd10 con 0x561b64755240 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f56fb400640 1 -- 172.21.15.40:0/206555498 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5708013b00 con 0x561b64755240 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f56fb400640 1 -- 172.21.15.40:0/206555498 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 133) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f57140d1020 con 0x561b64755240 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 10 client.9093 fetch_fsmap finished waiting for FSMap version 133 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 10 client.9093 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 20 client.9093 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 --2- 172.21.15.40:0/206555498 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x561b6473b090 0x561b64766ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 4 client.9093 mds_command: new command op to 8874 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 -- 172.21.15.40:0/206555498 --> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b644f3b60 con 0x561b6473b090 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f571de00640 1 --2- 172.21.15.40:0/206555498 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x561b6473b090 0x561b64766ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f571de00640 1 --2- 172.21.15.40:0/206555498 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x561b6473b090 0x561b64766ea0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a57b9f781859379f server_cookie=4bd2a56bb58b27f2 in_seq=0 out_seq=0 2024-03-24T00:39:24.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f56fb400640 10 client.9093 ms_handle_connect on v2:172.21.15.40:6832/4273998586 2024-03-24T00:39:24.900 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f56fb400640 1 -- 172.21.15.40:0/206555498 <== mds.1 v2:172.21.15.40:6832/4273998586 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561b644f3b60 con 0x561b6473b090 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f56fb400640 10 client.9093 handle_command_reply: tid=0 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 client.9093 shutdown 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 -- 172.21.15.40:0/206555498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 msgr2=0x561b6473a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 --2- 172.21.15.40:0/206555498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 0x561b6473a7c0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f571409ba40 tx=0x7f571401d950 comp rx=0 tx=0).stop 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 -- 172.21.15.40:0/206555498 shutdown_connections 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 --2- 172.21.15.40:0/206555498 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x561b6473b090 0x561b64766ea0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 --2- 172.21.15.40:0/206555498 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64755240 0x561b6473a7c0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 --2- 172.21.15.40:0/206555498 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b64754920 0x561b6473a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 --2- 172.21.15.40:0/206555498 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b6470aff0 0x561b64739d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f5723c39000 1 -- 172.21.15.40:0/206555498 >> 172.21.15.40:0/206555498 conn(0x561b64706fa0 msgr2=0x561b64709c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.892+0000 7f56fb400640 0 client.9093 ms_handle_reset on v2:172.21.15.40:6832/4273998586 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.896+0000 7f5723c39000 1 -- 172.21.15.40:0/206555498 shutdown_connections 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.896+0000 7f5723c39000 1 -- 172.21.15.40:0/206555498 wait complete. 2024-03-24T00:39:24.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.896+0000 7f5723c39000 20 client.9093 trim_cache size 0 max 16384 2024-03-24T00:39:24.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 Processor -- start 2024-03-24T00:39:24.915 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 -- start start 2024-03-24T00:39:24.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64739ed0 0x561b6470a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 0x561b64753530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 0x561b64753a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5718200130 con 0x561b64706fa0 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57181ff470 con 0x561b64739ed0 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57181ffad0 con 0x561b6470aff0 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f571d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 0x561b64753530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f571d400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 0x561b64753530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37150/0 (socket says 172.21.15.40:37150) 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f571de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64739ed0 0x561b6470a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f571d400640 1 -- 172.21.15.40:0/2804156237 learned_addr learned my addr 172.21.15.40:0/2804156237 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f571e800640 1 --2- 172.21.15.40:0/2804156237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 0x561b64753a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f571de00640 1 -- 172.21.15.40:0/2804156237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 msgr2=0x561b64753a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f571de00640 1 --2- 172.21.15.40:0/2804156237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 0x561b64753a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f571de00640 1 -- 172.21.15.40:0/2804156237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 msgr2=0x561b64753530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f571de00640 1 --2- 172.21.15.40:0/2804156237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 0x561b64753530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f571de00640 1 -- 172.21.15.40:0/2804156237 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b6470f1d0 con 0x561b64739ed0 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f571d400640 1 --2- 172.21.15.40:0/2804156237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 0x561b64753530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:24.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f571de00640 1 --2- 172.21.15.40:0/2804156237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64739ed0 0x561b6470a7c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f570c083c80 tx=0x7f570c004620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=797160271efbc6bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5706000640 1 -- 172.21.15.40:0/2804156237 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c020040 con 0x561b64739ed0 2024-03-24T00:39:24.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5706000640 1 -- 172.21.15.40:0/2804156237 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c053dd0 con 0x561b64739ed0 2024-03-24T00:39:24.919 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5706000640 1 -- 172.21.15.40:0/2804156237 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c009350 con 0x561b64739ed0 2024-03-24T00:39:24.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 -- 172.21.15.40:0/2804156237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64739ed0 msgr2=0x561b6470a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 --2- 172.21.15.40:0/2804156237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64739ed0 0x561b6470a7c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f570c083c80 tx=0x7f570c004620 comp rx=0 tx=0).stop 2024-03-24T00:39:24.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 -- 172.21.15.40:0/2804156237 shutdown_connections 2024-03-24T00:39:24.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 --2- 172.21.15.40:0/2804156237 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 0x561b64753a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 --2- 172.21.15.40:0/2804156237 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 0x561b64753530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 --2- 172.21.15.40:0/2804156237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64739ed0 0x561b6470a7c0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 -- 172.21.15.40:0/2804156237 >> 172.21.15.40:0/2804156237 conn(0x561b6470d750 msgr2=0x561b647375d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 -- 172.21.15.40:0/2804156237 shutdown_connections 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 -- 172.21.15.40:0/2804156237 wait complete. 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.912+0000 7f5723c39000 1 -- start start 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 1 Processor -- start 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 0x561b64767ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 0x561b64768430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64739ed0 0x561b64768970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b64500a80 con 0x561b64706fa0 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57180c79e0 con 0x561b64739ed0 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x561b64615700 con 0x561b6470aff0 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 10 client.0 _ll_register_callbacks cb 0x7f5714009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 0x561b64767ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 0x561b64767ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37160/0 (socket says 172.21.15.40:37160) 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571de00640 1 -- 172.21.15.40:0/784094486 learned_addr learned my addr 172.21.15.40:0/784094486 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:24.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571e800640 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64739ed0 0x561b64768970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571d400640 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 0x561b64768430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571de00640 1 -- 172.21.15.40:0/784094486 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 msgr2=0x561b64768430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571de00640 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 0x561b64768430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571de00640 1 -- 172.21.15.40:0/784094486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64739ed0 msgr2=0x561b64768970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571de00640 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64739ed0 0x561b64768970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571de00640 1 -- 172.21.15.40:0/784094486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b64525540 con 0x561b64706fa0 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571de00640 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 0x561b64767ef0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f570c055a30 tx=0x7f5714004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=200a32bbfae940dc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f56fb400640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f56fb400640 1 -- 172.21.15.40:0/784094486 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570c020050 con 0x561b64706fa0 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f56fb400640 1 -- 172.21.15.40:0/784094486 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f570c0523a0 con 0x561b64706fa0 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 1 -- 172.21.15.40:0/784094486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f570c053dd0 con 0x561b64706fa0 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f56fb400640 1 -- 172.21.15.40:0/784094486 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) v2 ==== 24+0+0 (secure 0 0 0) 0x7f570c055180 con 0x561b64706fa0 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 10 client.9099 fetch_fsmap learned FSMap version 133 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 1 -- 172.21.15.40:0/784094486 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=2+}) v3 -- 0x561b644ec4b0 con 0x561b64706fa0 2024-03-24T00:39:24.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f56fb400640 1 -- 172.21.15.40:0/784094486 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570c053dd0 con 0x561b64706fa0 2024-03-24T00:39:24.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f56fb400640 1 -- 172.21.15.40:0/784094486 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 133) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f570c023060 con 0x561b64706fa0 2024-03-24T00:39:24.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 10 client.9099 fetch_fsmap finished waiting for FSMap version 133 2024-03-24T00:39:24.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 10 client.9099 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:39:24.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 20 client.9099 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:24.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x561b64754920 0x561b64769300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:24.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 4 client.9099 mds_command: new command op to 8874 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:39:24.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f5723c39000 1 -- 172.21.15.40:0/784094486 --> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561b64565bc0 con 0x561b64754920 2024-03-24T00:39:24.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571d400640 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x561b64754920 0x561b64769300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:24.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f571d400640 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x561b64754920 0x561b64769300 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8fb78a55f6da5dd2 server_cookie=1745156f93f09a40 in_seq=0 out_seq=0 2024-03-24T00:39:24.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.916+0000 7f56fb400640 10 client.9099 ms_handle_connect on v2:172.21.15.40:6832/4273998586 2024-03-24T00:39:24.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f56fb400640 1 -- 172.21.15.40:0/784094486 <== mds.1 v2:172.21.15.40:6832/4273998586 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x561b64565bc0 con 0x561b64754920 2024-03-24T00:39:24.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f56fb400640 10 client.9099 handle_command_reply: tid=0 2024-03-24T00:39:24.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5723c39000 1 client.9099 shutdown 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5723c39000 1 -- 172.21.15.40:0/784094486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 msgr2=0x561b64767ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5723c39000 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 0x561b64767ef0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f570c055a30 tx=0x7f5714004ac0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5723c39000 1 -- 172.21.15.40:0/784094486 shutdown_connections 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5723c39000 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x561b64754920 0x561b64769300 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5723c39000 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561b64739ed0 0x561b64768970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5723c39000 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x561b6470aff0 0x561b64768430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5723c39000 1 --2- 172.21.15.40:0/784094486 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x561b64706fa0 0x561b64767ef0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f56fb400640 0 client.9099 ms_handle_reset on v2:172.21.15.40:6832/4273998586 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5723c39000 1 -- 172.21.15.40:0/784094486 >> 172.21.15.40:0/784094486 conn(0x561b6470d750 msgr2=0x561b64737e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5723c39000 1 -- 172.21.15.40:0/784094486 shutdown_connections 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5723c39000 1 -- 172.21.15.40:0/784094486 wait complete. 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5723c39000 20 client.9099 trim_cache size 0 max 16384 2024-03-24T00:39:24.927 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:39:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:39:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-03-24T00:39:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8874, 2024-03-24T00:39:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:39:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:39:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:39:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 3.0520590759999999, 2024-03-24T00:39:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 133, 2024-03-24T00:39:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 119, 2024-03-24T00:39:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 119, 2024-03-24T00:39:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 21.646495082000001 2024-03-24T00:39:24.928 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:39:24.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5720800640 1 -- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56fc067660 msgr2=0x7f56fc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5720800640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56fc067660 0x7f56fc069b20 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f57180c6a50 tx=0x7f5708007d60 comp rx=0 tx=0).stop 2024-03-24T00:39:24.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5720800640 1 -- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f571806bd10 msgr2=0x7f571815fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:24.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5720800640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f571806bd10 0x7f571815fb70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f570c00e8b0 tx=0x7f570c00ed70 comp rx=0 tx=0).stop 2024-03-24T00:39:24.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5720800640 1 -- 172.21.15.40:0/2488451770 shutdown_connections 2024-03-24T00:39:24.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5720800640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f56fc067660 0x7f56fc069b20 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5720800640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f571815e0b0 0x7f57180b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5720800640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180b4b80 0x7f57180cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5720800640 1 --2- 172.21.15.40:0/2488451770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f571806bd10 0x7f571815fb70 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:24.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5720800640 1 -- 172.21.15.40:0/2488451770 >> 172.21.15.40:0/2488451770 conn(0x7f57180b3880 msgr2=0x7f5718159320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:24.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5720800640 1 -- 172.21.15.40:0/2488451770 shutdown_connections 2024-03-24T00:39:24.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:24.920+0000 7f5720800640 1 -- 172.21.15.40:0/2488451770 wait complete. 2024-03-24T00:39:24.944 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:39:25.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.080+0000 7ff45b600640 1 Processor -- start 2024-03-24T00:39:25.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.080+0000 7ff45b600640 1 -- start start 2024-03-24T00:39:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.080+0000 7ff45b600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45415e220 0x7ff4540b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.080+0000 7ff45b600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff4540cf120 0x7ff4540b70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.080+0000 7ff45b600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff4540b7620 0x7ff4540d0790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.080+0000 7ff45b600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4540d2320 con 0x7ff4540b7620 2024-03-24T00:39:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.080+0000 7ff45b600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4540d2590 con 0x7ff4540cf120 2024-03-24T00:39:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.080+0000 7ff45b600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4540d2870 con 0x7ff45415e220 2024-03-24T00:39:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff459600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff4540b7620 0x7ff4540d0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff453e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff4540cf120 0x7ff4540b70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.087 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff458c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45415e220 0x7ff4540b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff453e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff4540cf120 0x7ff4540b70e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:46944/0 (socket says 172.21.15.40:46944) 2024-03-24T00:39:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff459600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff4540b7620 0x7ff4540d0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37166/0 (socket says 172.21.15.40:37166) 2024-03-24T00:39:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff453e00640 1 -- 172.21.15.40:0/3216234638 learned_addr learned my addr 172.21.15.40:0/3216234638 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff453e00640 1 -- 172.21.15.40:0/3216234638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45415e220 msgr2=0x7ff4540b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff453e00640 1 --2- 172.21.15.40:0/3216234638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45415e220 0x7ff4540b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff453e00640 1 -- 172.21.15.40:0/3216234638 >> v2:172.21.15.40:3300/0 conn(0x7ff4540b7620 msgr2=0x7ff4540d0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff453e00640 1 --2- 172.21.15.40:0/3216234638 >> v2:172.21.15.40:3300/0 conn(0x7ff4540b7620 0x7ff4540d0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.088 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff453e00640 1 -- 172.21.15.40:0/3216234638 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4540d2b50 con 0x7ff4540cf120 2024-03-24T00:39:25.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff453e00640 1 --2- 172.21.15.40:0/3216234638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4540cf120 0x7ff4540b70e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff45408c6f0 tx=0x7ff44001c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dcaea4fd293570 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff453400640 1 -- 172.21.15.40:0/3216234638 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4400ab5f0 con 0x7ff4540cf120 2024-03-24T00:39:25.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff453400640 1 -- 172.21.15.40:0/3216234638 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4400b0070 con 0x7ff4540cf120 2024-03-24T00:39:25.091 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.084+0000 7ff453400640 1 -- 172.21.15.40:0/3216234638 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4400ab9b0 con 0x7ff4540cf120 2024-03-24T00:39:25.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 -- 172.21.15.40:0/3216234638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4540cf120 msgr2=0x7ff4540b70e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 --2- 172.21.15.40:0/3216234638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4540cf120 0x7ff4540b70e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7ff45408c6f0 tx=0x7ff44001c520 comp rx=0 tx=0).stop 2024-03-24T00:39:25.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 -- 172.21.15.40:0/3216234638 shutdown_connections 2024-03-24T00:39:25.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 --2- 172.21.15.40:0/3216234638 >> v2:172.21.15.40:3300/0 conn(0x7ff4540b7620 0x7ff4540d0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 --2- 172.21.15.40:0/3216234638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4540cf120 0x7ff4540b70e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 --2- 172.21.15.40:0/3216234638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45415e220 0x7ff4540b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 -- 172.21.15.40:0/3216234638 >> 172.21.15.40:0/3216234638 conn(0x7ff454066e80 msgr2=0x7ff454067290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:25.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 -- 172.21.15.40:0/3216234638 shutdown_connections 2024-03-24T00:39:25.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 -- 172.21.15.40:0/3216234638 wait complete. 2024-03-24T00:39:25.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 Processor -- start 2024-03-24T00:39:25.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 -- start start 2024-03-24T00:39:25.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff4540b7620 0x7ff45415d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4540cf120 0x7ff454152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45415e220 0x7ff454140040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4540b5d00 con 0x7ff45415e220 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4540b5b80 con 0x7ff4540cf120 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff45b600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4540b5e80 con 0x7ff4540b7620 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff458c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff4540b7620 0x7ff45415d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff459600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45415e220 0x7ff454140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff453e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4540cf120 0x7ff454152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff459600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45415e220 0x7ff454140040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37168/0 (socket says 172.21.15.40:37168) 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff458c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff4540b7620 0x7ff45415d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:55258/0 (socket says 172.21.15.40:55258) 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff459600640 1 -- 172.21.15.40:0/1313355480 learned_addr learned my addr 172.21.15.40:0/1313355480 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff459600640 1 -- 172.21.15.40:0/1313355480 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff4540b7620 msgr2=0x7ff45415d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff459600640 1 --2- 172.21.15.40:0/1313355480 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff4540b7620 0x7ff45415d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff459600640 1 -- 172.21.15.40:0/1313355480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4540cf120 msgr2=0x7ff454152930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff459600640 1 --2- 172.21.15.40:0/1313355480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4540cf120 0x7ff454152930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.088+0000 7ff459600640 1 -- 172.21.15.40:0/1313355480 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45415d660 con 0x7ff45415e220 2024-03-24T00:39:25.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.092+0000 7ff459600640 1 --2- 172.21.15.40:0/1313355480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45415e220 0x7ff454140040 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00ef00 tx=0x7ff44c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.092+0000 7ff450c00640 1 -- 172.21.15.40:0/1313355480 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c018020 con 0x7ff45415e220 2024-03-24T00:39:25.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.092+0000 7ff450c00640 1 -- 172.21.15.40:0/1313355480 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44c014400 con 0x7ff45415e220 2024-03-24T00:39:25.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.092+0000 7ff45b600640 1 -- 172.21.15.40:0/1313355480 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45415d940 con 0x7ff45415e220 2024-03-24T00:39:25.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.092+0000 7ff450c00640 1 -- 172.21.15.40:0/1313355480 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c0147f0 con 0x7ff45415e220 2024-03-24T00:39:25.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.092+0000 7ff45b600640 1 -- 172.21.15.40:0/1313355480 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff454150ea0 con 0x7ff45415e220 2024-03-24T00:39:25.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.092+0000 7ff450c00640 1 -- 172.21.15.40:0/1313355480 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff44c01b020 con 0x7ff45415e220 2024-03-24T00:39:25.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.092+0000 7ff450c00640 1 --2- 172.21.15.40:0/1313355480 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff434067690 0x7ff434069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.098 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.092+0000 7ff450c00640 1 -- 172.21.15.40:0/1313355480 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff44c085f60 con 0x7ff45415e220 2024-03-24T00:39:25.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.092+0000 7ff45eaa3000 1 Processor -- start 2024-03-24T00:39:25.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.092+0000 7ff458c00640 1 --2- 172.21.15.40:0/1313355480 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff434067690 0x7ff434069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.092+0000 7ff45eaa3000 1 -- start start 2024-03-24T00:39:25.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff458c00640 1 --2- 172.21.15.40:0/1313355480 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff434067690 0x7ff434069b50 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff448009ac0 tx=0x7ff448009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920e920 0x55b68920ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 0x55b6891c4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 0x55b6891c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b689012680 con 0x55b68920e920 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b688f99b00 con 0x55b6891c4ff0 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b68900e720 con 0x55b68920f240 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff458c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920e920 0x55b68920ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff458c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920e920 0x55b68920ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37174/0 (socket says 172.21.15.40:37174) 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff458c00640 1 -- 172.21.15.40:0/2066408562 learned_addr learned my addr 172.21.15.40:0/2066408562 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff459600640 1 --2- 172.21.15.40:0/2066408562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 0x55b6891c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff453e00640 1 --2- 172.21.15.40:0/2066408562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 0x55b6891c4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff458c00640 1 -- 172.21.15.40:0/2066408562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 msgr2=0x55b6891c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff458c00640 1 --2- 172.21.15.40:0/2066408562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 0x55b6891c4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff458c00640 1 -- 172.21.15.40:0/2066408562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 msgr2=0x55b6891c53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff458c00640 1 --2- 172.21.15.40:0/2066408562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 0x55b6891c53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff458c00640 1 -- 172.21.15.40:0/2066408562 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b688fea940 con 0x55b68920e920 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff459600640 1 --2- 172.21.15.40:0/2066408562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 0x55b6891c53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:25.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff458c00640 1 --2- 172.21.15.40:0/2066408562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920e920 0x55b68920ed00 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7ff448006aa0 tx=0x7ff448014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a68bda4fb8063b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff445600640 1 -- 172.21.15.40:0/2066408562 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44801b070 con 0x55b68920e920 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff445600640 1 -- 172.21.15.40:0/2066408562 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff448027070 con 0x55b68920e920 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff445600640 1 -- 172.21.15.40:0/2066408562 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff448003a30 con 0x55b68920e920 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2066408562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920e920 msgr2=0x55b68920ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2066408562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920e920 0x55b68920ed00 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7ff448006aa0 tx=0x7ff448014000 comp rx=0 tx=0).stop 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2066408562 shutdown_connections 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2066408562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 0x55b6891c53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2066408562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 0x55b6891c4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2066408562 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920e920 0x55b68920ed00 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2066408562 >> 172.21.15.40:0/2066408562 conn(0x55b6891c0fa0 msgr2=0x55b6891c3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2066408562 shutdown_connections 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2066408562 wait complete. 2024-03-24T00:39:25.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 -- start start 2024-03-24T00:39:25.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.096+0000 7ff45eaa3000 1 Processor -- start 2024-03-24T00:39:25.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 0x55b6891f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b68920e920 0x55b6891f4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 0x55b6891f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b689011990 con 0x55b6891c4ff0 2024-03-24T00:39:25.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b688fccae0 con 0x55b68920e920 2024-03-24T00:39:25.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b688faa750 con 0x55b68920f240 2024-03-24T00:39:25.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 10 client.0 _ll_register_callbacks cb 0x55b6891afff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:25.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff458c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 0x55b6891f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff453e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b68920e920 0x55b6891f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff459600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 0x55b6891f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff453e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b68920e920 0x55b6891f4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:46956/0 (socket says 172.21.15.40:46956) 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff453e00640 1 -- 172.21.15.40:0/2072164420 learned_addr learned my addr 172.21.15.40:0/2072164420 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff458c00640 1 -- 172.21.15.40:0/2072164420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 msgr2=0x55b6891f47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff458c00640 1 --2- 172.21.15.40:0/2072164420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 0x55b6891f47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff458c00640 1 -- 172.21.15.40:0/2072164420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b68920e920 msgr2=0x55b6891f4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff458c00640 1 --2- 172.21.15.40:0/2072164420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b68920e920 0x55b6891f4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff458c00640 1 -- 172.21.15.40:0/2072164420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b689057d80 con 0x55b6891c4ff0 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff453e00640 1 --2- 172.21.15.40:0/2072164420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b68920e920 0x55b6891f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff458c00640 1 --2- 172.21.15.40:0/2072164420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 0x55b6891f3d40 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7ff448009540 tx=0x7ff448004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a6031f579760fb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff43a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff43a000640 1 -- 172.21.15.40:0/2072164420 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44801b040 con 0x55b6891c4ff0 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff43a000640 1 -- 172.21.15.40:0/2072164420 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff448022880 con 0x55b6891c4ff0 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2072164420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff448027070 con 0x55b6891c4ff0 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff43a000640 1 -- 172.21.15.40:0/2072164420 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff44803c050 con 0x55b6891c4ff0 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 10 client.9111 fetch_fsmap learned FSMap version 133 2024-03-24T00:39:25.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2072164420 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=2+}) v3 -- 0x55b688f99d10 con 0x55b6891c4ff0 2024-03-24T00:39:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff43a000640 1 -- 172.21.15.40:0/2072164420 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff448027070 con 0x55b6891c4ff0 2024-03-24T00:39:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff43a000640 1 -- 172.21.15.40:0/2072164420 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 133) v1 ==== 3354+0+0 (secure 0 0 0) 0x7ff44803f020 con 0x55b6891c4ff0 2024-03-24T00:39:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 10 client.9111 fetch_fsmap finished waiting for FSMap version 133 2024-03-24T00:39:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 10 client.9111 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:39:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 20 client.9111 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2072164420 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55b6891f5090 0x55b689220ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 4 client.9111 mds_command: new command op to 8877 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:25.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2072164420 --> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b688fadb60 con 0x55b6891f5090 2024-03-24T00:39:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.100+0000 7ff453e00640 1 --2- 172.21.15.40:0/2072164420 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55b6891f5090 0x55b689220ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff43a000640 10 client.9111 ms_handle_connect on v2:172.21.15.40:6834/773071485 2024-03-24T00:39:25.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff453e00640 1 --2- 172.21.15.40:0/2072164420 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55b6891f5090 0x55b689220ea0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a05a3177816524b7 server_cookie=642de4b8f150777b in_seq=0 out_seq=0 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff43a000640 1 -- 172.21.15.40:0/2072164420 <== mds.0 v2:172.21.15.40:6834/773071485 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b688fadb60 con 0x55b6891f5090 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff43a000640 10 client.9111 handle_command_reply: tid=0 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff45eaa3000 1 client.9111 shutdown 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2072164420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 msgr2=0x55b6891f3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2072164420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 0x55b6891f3d40 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7ff448009540 tx=0x7ff448004740 comp rx=0 tx=0).stop 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2072164420 shutdown_connections 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2072164420 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55b6891f5090 0x55b689220ea0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2072164420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 0x55b6891f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2072164420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b68920e920 0x55b6891f4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2072164420 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 0x55b6891f3d40 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff43a000640 0 client.9111 ms_handle_reset on v2:172.21.15.40:6834/773071485 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2072164420 >> 172.21.15.40:0/2072164420 conn(0x55b6891c0fa0 msgr2=0x55b6891c3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2072164420 shutdown_connections 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2072164420 wait complete. 2024-03-24T00:39:25.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.104+0000 7ff45eaa3000 20 client.9111 trim_cache size 0 max 16384 2024-03-24T00:39:25.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.124+0000 7ff45eaa3000 1 Processor -- start 2024-03-24T00:39:25.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.124+0000 7ff45eaa3000 1 -- start start 2024-03-24T00:39:25.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 0x55b6891f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b68920e920 0x55b68920d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 0x55b68920da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff454203870 con 0x55b6891c4ff0 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff454202bf0 con 0x55b68920e920 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff454203210 con 0x55b68920f240 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff458c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 0x55b6891f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff458c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 0x55b6891f1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37190/0 (socket says 172.21.15.40:37190) 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff458c00640 1 -- 172.21.15.40:0/2036605188 learned_addr learned my addr 172.21.15.40:0/2036605188 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff453e00640 1 --2- 172.21.15.40:0/2036605188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b68920e920 0x55b68920d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff459600640 1 --2- 172.21.15.40:0/2036605188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 0x55b68920da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff458c00640 1 -- 172.21.15.40:0/2036605188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 msgr2=0x55b68920da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff458c00640 1 --2- 172.21.15.40:0/2036605188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 0x55b68920da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff458c00640 1 -- 172.21.15.40:0/2036605188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b68920e920 msgr2=0x55b68920d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff458c00640 1 --2- 172.21.15.40:0/2036605188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b68920e920 0x55b68920d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff458c00640 1 -- 172.21.15.40:0/2036605188 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b688fdf540 con 0x55b6891c4ff0 2024-03-24T00:39:25.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff458c00640 1 --2- 172.21.15.40:0/2036605188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 0x55b6891f1ef0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7ff448004170 tx=0x7ff44802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3eb6e3dfb954ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff444c00640 1 -- 172.21.15.40:0/2036605188 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44801b070 con 0x55b6891c4ff0 2024-03-24T00:39:25.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff444c00640 1 -- 172.21.15.40:0/2036605188 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff448027040 con 0x55b6891c4ff0 2024-03-24T00:39:25.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff444c00640 1 -- 172.21.15.40:0/2036605188 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff448014440 con 0x55b6891c4ff0 2024-03-24T00:39:25.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2036605188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 msgr2=0x55b6891f1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2036605188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 0x55b6891f1ef0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7ff448004170 tx=0x7ff44802e070 comp rx=0 tx=0).stop 2024-03-24T00:39:25.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2036605188 shutdown_connections 2024-03-24T00:39:25.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2036605188 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920f240 0x55b68920da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2036605188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b68920e920 0x55b68920d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/2036605188 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b6891c4ff0 0x55b6891f1ef0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2036605188 >> 172.21.15.40:0/2036605188 conn(0x55b6891c0fa0 msgr2=0x55b6891f15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:25.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2036605188 shutdown_connections 2024-03-24T00:39:25.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.128+0000 7ff45eaa3000 1 -- 172.21.15.40:0/2036605188 wait complete. 2024-03-24T00:39:25.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff45eaa3000 1 -- start start 2024-03-24T00:39:25.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff45eaa3000 1 Processor -- start 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff45eaa3000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 0x55b68920dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff45eaa3000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920e920 0x55b689221ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff45eaa3000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920f240 0x55b689222430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff45eaa3000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b688fbaa80 con 0x55b68920f240 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff45eaa3000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4540b88c0 con 0x55b6891c4ff0 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff45eaa3000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b6890cf700 con 0x55b68920e920 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff45eaa3000 10 client.0 _ll_register_callbacks cb 0x7ff44c009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff459600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920f240 0x55b689222430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff458c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 0x55b68920dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff458c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 0x55b68920dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:46982/0 (socket says 172.21.15.40:46982) 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff459600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920f240 0x55b689222430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37196/0 (socket says 172.21.15.40:37196) 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff458c00640 1 -- 172.21.15.40:0/3617096288 learned_addr learned my addr 172.21.15.40:0/3617096288 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:25.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff453e00640 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920e920 0x55b689221ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff459600640 1 -- 172.21.15.40:0/3617096288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920e920 msgr2=0x55b689221ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:25.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff459600640 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920e920 0x55b689221ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff459600640 1 -- 172.21.15.40:0/3617096288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 msgr2=0x55b68920dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:25.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff459600640 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 0x55b68920dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff459600640 1 -- 172.21.15.40:0/3617096288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b688fa64b0 con 0x55b68920f240 2024-03-24T00:39:25.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff458c00640 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 0x55b68920dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:25.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff453e00640 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920e920 0x55b689221ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:25.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff459600640 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920f240 0x55b689222430 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00b980 tx=0x7ff44c009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ed826cb865ce8d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff43a000640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:39:25.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff43a000640 1 -- 172.21.15.40:0/3617096288 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c0568d0 con 0x55b68920f240 2024-03-24T00:39:25.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff45eaa3000 1 -- 172.21.15.40:0/3617096288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff448027040 con 0x55b68920f240 2024-03-24T00:39:25.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff43a000640 1 -- 172.21.15.40:0/3617096288 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff44c06bdc0 con 0x55b68920f240 2024-03-24T00:39:25.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff43a000640 1 -- 172.21.15.40:0/3617096288 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=133) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff44c082050 con 0x55b68920f240 2024-03-24T00:39:25.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff45eaa3000 10 client.9117 fetch_fsmap learned FSMap version 133 2024-03-24T00:39:25.141 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff45eaa3000 1 -- 172.21.15.40:0/3617096288 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=133,monmap=2+}) v3 -- 0x55b688fb92f0 con 0x55b68920f240 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.132+0000 7ff43a000640 1 -- 172.21.15.40:0/3617096288 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff448027040 con 0x55b68920f240 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff43a000640 1 -- 172.21.15.40:0/3617096288 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 133) v1 ==== 3354+0+0 (secure 0 0 0) 0x7ff44c05e640 con 0x55b68920f240 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 10 client.9117 fetch_fsmap finished waiting for FSMap version 133 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 10 client.9117 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 20 client.9117 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55b689222970 0x55b6892737d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 4 client.9117 mds_command: new command op to 8877 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 -- 172.21.15.40:0/3617096288 --> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b68901fbc0 con 0x55b689222970 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff458c00640 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55b689222970 0x55b6892737d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff458c00640 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55b689222970 0x55b6892737d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec02b57d9bab7300 server_cookie=de4f895a764a782f in_seq=0 out_seq=0 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff43a000640 10 client.9117 ms_handle_connect on v2:172.21.15.40:6834/773071485 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff43a000640 1 -- 172.21.15.40:0/3617096288 <== mds.0 v2:172.21.15.40:6834/773071485 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55b68901fbc0 con 0x55b689222970 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff43a000640 10 client.9117 handle_command_reply: tid=0 2024-03-24T00:39:25.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 client.9117 shutdown 2024-03-24T00:39:25.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 -- 172.21.15.40:0/3617096288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920f240 msgr2=0x55b689222430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920f240 0x55b689222430 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00b980 tx=0x7ff44c009f50 comp rx=0 tx=0).stop 2024-03-24T00:39:25.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 -- 172.21.15.40:0/3617096288 shutdown_connections 2024-03-24T00:39:25.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x55b689222970 0x55b6892737d0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55b68920f240 0x55b689222430 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55b68920e920 0x55b689221ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 --2- 172.21.15.40:0/3617096288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55b6891c4ff0 0x55b68920dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 -- 172.21.15.40:0/3617096288 >> 172.21.15.40:0/3617096288 conn(0x55b6891c0fa0 msgr2=0x55b6891f1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:25.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff43a000640 0 client.9117 ms_handle_reset on v2:172.21.15.40:6834/773071485 2024-03-24T00:39:25.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 -- 172.21.15.40:0/3617096288 shutdown_connections 2024-03-24T00:39:25.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 1 -- 172.21.15.40:0/3617096288 wait complete. 2024-03-24T00:39:25.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45eaa3000 20 client.9117 trim_cache size 0 max 16384 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8877, 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 16.998246404, 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 133, 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 119, 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 119, 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 21.860274960999998 2024-03-24T00:39:25.144 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:39:25.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45b600640 1 -- 172.21.15.40:0/1313355480 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff434067690 msgr2=0x7ff434069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45b600640 1 --2- 172.21.15.40:0/1313355480 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff434067690 0x7ff434069b50 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff448009ac0 tx=0x7ff448009190 comp rx=0 tx=0).stop 2024-03-24T00:39:25.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45b600640 1 -- 172.21.15.40:0/1313355480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45415e220 msgr2=0x7ff454140040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45b600640 1 --2- 172.21.15.40:0/1313355480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45415e220 0x7ff454140040 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00ef00 tx=0x7ff44c00b480 comp rx=0 tx=0).stop 2024-03-24T00:39:25.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45b600640 1 -- 172.21.15.40:0/1313355480 shutdown_connections 2024-03-24T00:39:25.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45b600640 1 --2- 172.21.15.40:0/1313355480 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff434067690 0x7ff434069b50 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45b600640 1 --2- 172.21.15.40:0/1313355480 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff45415e220 0x7ff454140040 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45b600640 1 --2- 172.21.15.40:0/1313355480 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4540cf120 0x7ff454152930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45b600640 1 --2- 172.21.15.40:0/1313355480 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff4540b7620 0x7ff45415d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45b600640 1 -- 172.21.15.40:0/1313355480 >> 172.21.15.40:0/1313355480 conn(0x7ff454066e80 msgr2=0x7ff454157b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:25.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45b600640 1 -- 172.21.15.40:0/1313355480 shutdown_connections 2024-03-24T00:39:25.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.136+0000 7ff45b600640 1 -- 172.21.15.40:0/1313355480 wait complete. 2024-03-24T00:39:25.160 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-24T00:39:25.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.300+0000 7fc628400640 1 Processor -- start 2024-03-24T00:39:25.306 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.300+0000 7fc628400640 1 -- start start 2024-03-24T00:39:25.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.304+0000 7fc628400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc62015b960 0x7fc620070d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.304+0000 7fc628400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62015f970 con 0x7fc62015bd40 2024-03-24T00:39:25.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.304+0000 7fc628400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc62015fc50 con 0x7fc6200d16f0 2024-03-24T00:39:25.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.304+0000 7fc628400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc620160580 con 0x7fc62015b960 2024-03-24T00:39:25.307 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.304+0000 7fc625000640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc62015bd40 0x7fc6200d0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39020/0 (socket says 172.21.15.40:39020) 2024-03-24T00:39:25.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.304+0000 7fc625000640 1 -- 172.21.15.40:0/349102168 learned_addr learned my addr 172.21.15.40:0/349102168 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:25.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.304+0000 7fc625a00640 1 --2- 172.21.15.40:0/349102168 >> v2:172.21.15.40:3300/0 conn(0x7fc62015b960 0x7fc620070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.304+0000 7fc625a00640 1 -- 172.21.15.40:0/349102168 >> v1:172.21.15.53:6789/0 conn(0x7fc6200d16f0 legacy=0x7fc6200cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.304+0000 7fc625a00640 1 -- 172.21.15.40:0/349102168 >> v1:172.21.15.40:6789/0 conn(0x7fc62015bd40 legacy=0x7fc6200d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.304+0000 7fc61fe00640 1 -- 172.21.15.40:0/349102168 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4141422171 0 0) 0x7fc62015fc50 con 0x7fc6200d16f0 2024-03-24T00:39:25.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.304+0000 7fc625a00640 1 -- 172.21.15.40:0/349102168 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62015ff30 con 0x7fc62015b960 2024-03-24T00:39:25.310 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.304+0000 7fc625a00640 1 --2- 172.21.15.40:0/349102168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62015b960 0x7fc620070d60 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fc6140097c0 tx=0x7fc61401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3fb2b1acdd3ab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.308+0000 7fc61fe00640 1 -- 172.21.15.40:0/349102168 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6140b0070 con 0x7fc62015b960 2024-03-24T00:39:25.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.308+0000 7fc61fe00640 1 -- 172.21.15.40:0/349102168 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6140ab600 con 0x7fc62015b960 2024-03-24T00:39:25.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.308+0000 7fc61fe00640 1 -- 172.21.15.40:0/349102168 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6140ab9d0 con 0x7fc62015b960 2024-03-24T00:39:25.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.308+0000 7fc628400640 1 -- 172.21.15.40:0/349102168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62015b960 msgr2=0x7fc620070d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.308+0000 7fc628400640 1 --2- 172.21.15.40:0/349102168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62015b960 0x7fc620070d60 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fc6140097c0 tx=0x7fc61401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.308+0000 7fc628400640 1 -- 172.21.15.40:0/349102168 shutdown_connections 2024-03-24T00:39:25.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.308+0000 7fc628400640 1 --2- 172.21.15.40:0/349102168 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62015b960 0x7fc620070d60 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.308+0000 7fc628400640 1 -- 172.21.15.40:0/349102168 >> 172.21.15.40:0/349102168 conn(0x7fc62006ecf0 msgr2=0x7fc62006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:25.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.308+0000 7fc628400640 1 -- 172.21.15.40:0/349102168 shutdown_connections 2024-03-24T00:39:25.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.308+0000 7fc628400640 1 -- 172.21.15.40:0/349102168 wait complete. 2024-03-24T00:39:25.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.308+0000 7fc628400640 1 Processor -- start 2024-03-24T00:39:25.314 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.308+0000 7fc628400640 1 -- start start 2024-03-24T00:39:25.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc628400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6200d16f0 0x7fc620155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc628400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62015b960 0x7fc6200c52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc628400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc62015bd40 0x7fc6200d0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc628400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6200ced30 con 0x7fc62015b960 2024-03-24T00:39:25.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc628400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6200cebb0 con 0x7fc62015bd40 2024-03-24T00:39:25.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc628400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc620169b80 con 0x7fc6200d16f0 2024-03-24T00:39:25.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc625000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62015b960 0x7fc6200c52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc625a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6200d16f0 0x7fc620155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc625000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62015b960 0x7fc6200c52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37216/0 (socket says 172.21.15.40:37216) 2024-03-24T00:39:25.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc625a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6200d16f0 0x7fc620155c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:55300/0 (socket says 172.21.15.40:55300) 2024-03-24T00:39:25.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc625000640 1 -- 172.21.15.40:0/3919273309 learned_addr learned my addr 172.21.15.40:0/3919273309 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:25.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc626400640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc62015bd40 0x7fc6200d0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc625a00640 1 -- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc62015bd40 msgr2=0x7fc6200d0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc625a00640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc62015bd40 0x7fc6200d0190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc625a00640 1 -- 172.21.15.40:0/3919273309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62015b960 msgr2=0x7fc6200c52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc625a00640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62015b960 0x7fc6200c52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc625a00640 1 -- 172.21.15.40:0/3919273309 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62015eb50 con 0x7fc6200d16f0 2024-03-24T00:39:25.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc625000640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62015b960 0x7fc6200c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:25.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc625a00640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6200d16f0 0x7fc620155c00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc6140ab000 tx=0x7fc61401ace0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc61d600640 1 -- 172.21.15.40:0/3919273309 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6140b0040 con 0x7fc6200d16f0 2024-03-24T00:39:25.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc61d600640 1 -- 172.21.15.40:0/3919273309 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6140abe50 con 0x7fc6200d16f0 2024-03-24T00:39:25.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc61d600640 1 -- 172.21.15.40:0/3919273309 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc614004270 con 0x7fc6200d16f0 2024-03-24T00:39:25.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc628400640 1 -- 172.21.15.40:0/3919273309 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc62015ee30 con 0x7fc6200d16f0 2024-03-24T00:39:25.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.312+0000 7fc628400640 1 -- 172.21.15.40:0/3919273309 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6201661e0 con 0x7fc6200d16f0 2024-03-24T00:39:25.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.316+0000 7fc628400640 1 -- 172.21.15.40:0/3919273309 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc620068170 con 0x7fc6200d16f0 2024-03-24T00:39:25.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.316+0000 7fc61d600640 1 -- 172.21.15.40:0/3919273309 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc6140aa750 con 0x7fc6200d16f0 2024-03-24T00:39:25.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.316+0000 7fc61d600640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc600067660 0x7fc600069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:25.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.316+0000 7fc61d600640 1 -- 172.21.15.40:0/3919273309 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc614120220 con 0x7fc6200d16f0 2024-03-24T00:39:25.321 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.316+0000 7fc625000640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc600067660 0x7fc600069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:25.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.316+0000 7fc625000640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc600067660 0x7fc600069b20 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc62006bee0 tx=0x7fc60c007f20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:25.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.320+0000 7fc61d600640 1 -- 172.21.15.40:0/3919273309 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc614016090 con 0x7fc6200d16f0 2024-03-24T00:39:25.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.676+0000 7fc628400640 1 -- 172.21.15.40:0/3919273309 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7fc6200c57e0 con 0x7fc6200d16f0 2024-03-24T00:39:25.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc61d600640 1 -- 172.21.15.40:0/3919273309 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v134) v1 ==== 105+0+0 (secure 0 0 0) 0x7fc6140f0960 con 0x7fc6200d16f0 2024-03-24T00:39:25.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc628400640 1 -- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc600067660 msgr2=0x7fc600069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc628400640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc600067660 0x7fc600069b20 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fc62006bee0 tx=0x7fc60c007f20 comp rx=0 tx=0).stop 2024-03-24T00:39:25.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc628400640 1 -- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6200d16f0 msgr2=0x7fc620155c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:25.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc628400640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6200d16f0 0x7fc620155c00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc6140ab000 tx=0x7fc61401ace0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc628400640 1 -- 172.21.15.40:0/3919273309 shutdown_connections 2024-03-24T00:39:25.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc628400640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc600067660 0x7fc600069b20 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc628400640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc62015bd40 0x7fc6200d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc628400640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc62015b960 0x7fc6200c52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc628400640 1 --2- 172.21.15.40:0/3919273309 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6200d16f0 0x7fc620155c00 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:25.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc628400640 1 -- 172.21.15.40:0/3919273309 >> 172.21.15.40:0/3919273309 conn(0x7fc62006ecf0 msgr2=0x7fc62006cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:25.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc628400640 1 -- 172.21.15.40:0/3919273309 shutdown_connections 2024-03-24T00:39:25.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:25.892+0000 7fc628400640 1 -- 172.21.15.40:0/3919273309 wait complete. 2024-03-24T00:39:25.913 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:39:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.052+0000 7f5e4fc00640 1 Processor -- start 2024-03-24T00:39:26.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.052+0000 7f5e4fc00640 1 -- start start 2024-03-24T00:39:26.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.052+0000 7f5e4fc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e48141bc0 0x7f5e480ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.052+0000 7f5e4fc00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e48070e00 con 0x7f5e4813ec80 2024-03-24T00:39:26.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.052+0000 7f5e4fc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e480cee20 con 0x7f5e480ce630 2024-03-24T00:39:26.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.052+0000 7f5e4fc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e480cf100 con 0x7f5e48141bc0 2024-03-24T00:39:26.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.056+0000 7f5e4d200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f5e4813ec80 0x7f5e481414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:39026/0 (socket says 172.21.15.40:39026) 2024-03-24T00:39:26.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.056+0000 7f5e4d200640 1 -- 172.21.15.40:0/2026623156 learned_addr learned my addr 172.21.15.40:0/2026623156 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:26.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.056+0000 7f5e47e00640 1 --2- 172.21.15.40:0/2026623156 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e48141bc0 0x7f5e480ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.056+0000 7f5e47e00640 1 -- 172.21.15.40:0/2026623156 >> v1:172.21.15.53:6789/0 conn(0x7f5e480ce630 legacy=0x7f5e48071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.056+0000 7f5e47e00640 1 -- 172.21.15.40:0/2026623156 >> v1:172.21.15.40:6789/0 conn(0x7f5e4813ec80 legacy=0x7f5e481414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.056+0000 7f5e47400640 1 -- 172.21.15.40:0/2026623156 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561734036 0 0) 0x7f5e480cee20 con 0x7f5e480ce630 2024-03-24T00:39:26.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.056+0000 7f5e47e00640 1 -- 172.21.15.40:0/2026623156 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e480cf3e0 con 0x7f5e48141bc0 2024-03-24T00:39:26.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e47e00640 1 --2- 172.21.15.40:0/2026623156 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e48141bc0 0x7f5e480ce0f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5e340097f0 tx=0x7f5e3401c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b044bd1a20ace7b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:26.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e47400640 1 -- 172.21.15.40:0/2026623156 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e340b1070 con 0x7f5e48141bc0 2024-03-24T00:39:26.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e47400640 1 -- 172.21.15.40:0/2026623156 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e340ac640 con 0x7f5e48141bc0 2024-03-24T00:39:26.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e47400640 1 -- 172.21.15.40:0/2026623156 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e340aca10 con 0x7f5e48141bc0 2024-03-24T00:39:26.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e4fc00640 1 -- 172.21.15.40:0/2026623156 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e48141bc0 msgr2=0x7f5e480ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e4fc00640 1 --2- 172.21.15.40:0/2026623156 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e48141bc0 0x7f5e480ce0f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5e340097f0 tx=0x7f5e3401c370 comp rx=0 tx=0).stop 2024-03-24T00:39:26.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e4fc00640 1 -- 172.21.15.40:0/2026623156 shutdown_connections 2024-03-24T00:39:26.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e4fc00640 1 --2- 172.21.15.40:0/2026623156 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e48141bc0 0x7f5e480ce0f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e4fc00640 1 -- 172.21.15.40:0/2026623156 >> 172.21.15.40:0/2026623156 conn(0x7f5e480b17b0 msgr2=0x7f5e480b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:26.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e4fc00640 1 -- 172.21.15.40:0/2026623156 shutdown_connections 2024-03-24T00:39:26.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e4fc00640 1 -- 172.21.15.40:0/2026623156 wait complete. 2024-03-24T00:39:26.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e4fc00640 1 Processor -- start 2024-03-24T00:39:26.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.060+0000 7f5e4fc00640 1 -- start start 2024-03-24T00:39:26.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e4fc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e480ce630 0x7f5e480b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e4fc00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4813ec80 0x7f5e480b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e4fc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e48141bc0 0x7f5e48139610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e4fc00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e480725b0 con 0x7f5e4813ec80 2024-03-24T00:39:26.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e4fc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e48072430 con 0x7f5e480ce630 2024-03-24T00:39:26.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e4fc00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e48169d50 con 0x7f5e48141bc0 2024-03-24T00:39:26.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e47e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4813ec80 0x7f5e480b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e47e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4813ec80 0x7f5e480b2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37228/0 (socket says 172.21.15.40:37228) 2024-03-24T00:39:26.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e47e00640 1 -- 172.21.15.40:0/432163619 learned_addr learned my addr 172.21.15.40:0/432163619 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:26.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e4d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e480ce630 0x7f5e480b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e4dc00640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e48141bc0 0x7f5e48139610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e47e00640 1 -- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e48141bc0 msgr2=0x7f5e48139610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e47e00640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e48141bc0 0x7f5e48139610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e47e00640 1 -- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e480ce630 msgr2=0x7f5e480b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e47e00640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e480ce630 0x7f5e480b09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e47e00640 1 -- 172.21.15.40:0/432163619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e48151e80 con 0x7f5e4813ec80 2024-03-24T00:39:26.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e47e00640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4813ec80 0x7f5e480b2bb0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f5e34019fa0 tx=0x7f5e3401abb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:26.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e44c00640 1 -- 172.21.15.40:0/432163619 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e340b1050 con 0x7f5e4813ec80 2024-03-24T00:39:26.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e44c00640 1 -- 172.21.15.40:0/432163619 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e340bd070 con 0x7f5e4813ec80 2024-03-24T00:39:26.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e4fc00640 1 -- 172.21.15.40:0/432163619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e48155390 con 0x7f5e4813ec80 2024-03-24T00:39:26.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e44c00640 1 -- 172.21.15.40:0/432163619 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e340b6a30 con 0x7f5e4813ec80 2024-03-24T00:39:26.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e4fc00640 1 -- 172.21.15.40:0/432163619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e48152120 con 0x7f5e4813ec80 2024-03-24T00:39:26.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.064+0000 7f5e4fc00640 1 -- 172.21.15.40:0/432163619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e48068170 con 0x7f5e4813ec80 2024-03-24T00:39:26.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.068+0000 7f5e44c00640 1 -- 172.21.15.40:0/432163619 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f5e340b6bd0 con 0x7f5e4813ec80 2024-03-24T00:39:26.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.068+0000 7f5e44c00640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e28067660 0x7f5e28069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.068+0000 7f5e44c00640 1 -- 172.21.15.40:0/432163619 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e3411ff40 con 0x7f5e4813ec80 2024-03-24T00:39:26.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.068+0000 7f5e4d200640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e28067660 0x7f5e28069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.070 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.068+0000 7f5e4d200640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e28067660 0x7f5e28069b20 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c000f70 tx=0x7f5e3c007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:26.074 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.072+0000 7f5e44c00640 1 -- 172.21.15.40:0/432163619 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f5e34015ff0 con 0x7f5e4813ec80 2024-03-24T00:39:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.312+0000 7f5e4fc00640 1 -- 172.21.15.40:0/432163619 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e48072b90 con 0x7f5e4813ec80 2024-03-24T00:39:26.313 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.312+0000 7f5e44c00640 1 -- 172.21.15.40:0/432163619 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5020 (secure 0 0 0) 0x7f5e340f05f0 con 0x7f5e4813ec80 2024-03-24T00:39:26.314 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:39:26.314 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":134,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7439,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6837/1582567544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1582567544},{"type":"v1","addr":"172.21.15.53:6837","nonce":1582567544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:39:07.531884+0000","modified":"2024-03-24T00:39:25.684480+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8877,"mds_1":8874,"mds_2":8563},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8563":{"gid":8563,"name":"b","rank":2,"incarnation":132,"state":"up:active","state_seq":6,"addr":"172.21.15.53:6836/2255755838","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":2255755838},{"type":"v1","addr":"172.21.15.53:6836","nonce":2255755838}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8874":{"gid":8874,"name":"c","rank":1,"incarnation":130,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6833/4273998586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4273998586},{"type":"v1","addr":"172.21.15.40:6833","nonce":4273998586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8877":{"gid":8877,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":3,"addr":"172.21.15.40:6835/773071485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":773071485},{"type":"v1","addr":"172.21.15.40:6835","nonce":773071485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8877,"qdb_cluster":[8877,8874,8563]},"id":9}]} 2024-03-24T00:39:26.314 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 134 2024-03-24T00:39:26.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.316+0000 7f5e4fc00640 1 -- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e28067660 msgr2=0x7f5e28069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.316+0000 7f5e4fc00640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e28067660 0x7f5e28069b20 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c000f70 tx=0x7f5e3c007e30 comp rx=0 tx=0).stop 2024-03-24T00:39:26.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.316+0000 7f5e4fc00640 1 -- 172.21.15.40:0/432163619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4813ec80 msgr2=0x7f5e480b2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.316+0000 7f5e4fc00640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4813ec80 0x7f5e480b2bb0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f5e34019fa0 tx=0x7f5e3401abb0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.316+0000 7f5e4fc00640 1 -- 172.21.15.40:0/432163619 shutdown_connections 2024-03-24T00:39:26.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.316+0000 7f5e4fc00640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5e28067660 0x7f5e28069b20 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.316+0000 7f5e4fc00640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e48141bc0 0x7f5e48139610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.316+0000 7f5e4fc00640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f5e4813ec80 0x7f5e480b2bb0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.316+0000 7f5e4fc00640 1 --2- 172.21.15.40:0/432163619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e480ce630 0x7f5e480b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.316+0000 7f5e4fc00640 1 -- 172.21.15.40:0/432163619 >> 172.21.15.40:0/432163619 conn(0x7f5e480b17b0 msgr2=0x7f5e4815ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:26.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.316+0000 7f5e4fc00640 1 -- 172.21.15.40:0/432163619 shutdown_connections 2024-03-24T00:39:26.316 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.316+0000 7f5e4fc00640 1 -- 172.21.15.40:0/432163619 wait complete. 2024-03-24T00:39:26.737 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 134, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-24T00:39:07.531884+0000', 'modified': '2024-03-24T00:39:25.684480+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 8877, 'mds_1': 8874, 'mds_2': 8563}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8563': {'gid': 8563, 'name': 'b', 'rank': 2, 'incarnation': 132, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.53:6836/2255755838', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 2255755838}, {'type': 'v1', 'addr': '172.21.15.53:6836', 'nonce': 2255755838}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8874': {'gid': 8874, 'name': 'c', 'rank': 1, 'incarnation': 130, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.40:6833/4273998586', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6832', 'nonce': 4273998586}, {'type': 'v1', 'addr': '172.21.15.40:6833', 'nonce': 4273998586}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_8877': {'gid': 8877, 'name': 'a', 'rank': 0, 'incarnation': 124, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.40:6835/773071485', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.40:6834', 'nonce': 773071485}, {'type': 'v1', 'addr': '172.21.15.40:6835', 'nonce': 773071485}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 8877, 'qdb_cluster': [8877, 8874, 8563]} 2024-03-24T00:39:26.737 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-24T00:39:26.738 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:39:26.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.876+0000 7f7003800640 1 Processor -- start 2024-03-24T00:39:26.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.876+0000 7f7003800640 1 -- start start 2024-03-24T00:39:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.880+0000 7f7003800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ffc072310 0x7f6ffc0ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.880+0000 7f7003800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ffc070e00 con 0x7f6ffc13ec80 2024-03-24T00:39:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.880+0000 7f7003800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ffc0cee20 con 0x7f6ffc141bc0 2024-03-24T00:39:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.880+0000 7f7003800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc0cf100 con 0x7f6ffc072310 2024-03-24T00:39:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.880+0000 7f7001800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f6ffc072310 0x7f6ffc0ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.880+0000 7f6ffbe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6ffc141bc0 0x7f6ffc0ce0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58604/0 (socket says 172.21.15.40:58604) 2024-03-24T00:39:26.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.880+0000 7f6ffbe00640 1 -- 172.21.15.40:0/2102104714 learned_addr learned my addr 172.21.15.40:0/2102104714 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.880+0000 7f6ffb400640 1 -- 172.21.15.40:0/2102104714 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213505327 0 0) 0x7f6ffc0cee20 con 0x7f6ffc141bc0 2024-03-24T00:39:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.880+0000 7f7001800640 1 -- 172.21.15.40:0/2102104714 >> v1:172.21.15.53:6789/0 conn(0x7f6ffc141bc0 legacy=0x7f6ffc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.880+0000 7f7001800640 1 -- 172.21.15.40:0/2102104714 >> v1:172.21.15.40:6789/0 conn(0x7f6ffc13ec80 legacy=0x7f6ffc1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.880+0000 7f7001800640 1 -- 172.21.15.40:0/2102104714 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc0cf3e0 con 0x7f6ffc072310 2024-03-24T00:39:26.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f7001800640 1 --2- 172.21.15.40:0/2102104714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc072310 0x7f6ffc0ce800 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009d90 tx=0x7f6ff401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6cf69df5b246fc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f6ffb400640 1 -- 172.21.15.40:0/2102104714 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40b2070 con 0x7f6ffc072310 2024-03-24T00:39:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f6ffb400640 1 -- 172.21.15.40:0/2102104714 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff40a5a50 con 0x7f6ffc072310 2024-03-24T00:39:26.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f6ffb400640 1 -- 172.21.15.40:0/2102104714 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40a5e20 con 0x7f6ffc072310 2024-03-24T00:39:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f7003800640 1 -- 172.21.15.40:0/2102104714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc072310 msgr2=0x7f6ffc0ce800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f7003800640 1 --2- 172.21.15.40:0/2102104714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc072310 0x7f6ffc0ce800 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4009d90 tx=0x7f6ff401c970 comp rx=0 tx=0).stop 2024-03-24T00:39:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f7003800640 1 -- 172.21.15.40:0/2102104714 shutdown_connections 2024-03-24T00:39:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f7003800640 1 --2- 172.21.15.40:0/2102104714 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc072310 0x7f6ffc0ce800 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f7003800640 1 -- 172.21.15.40:0/2102104714 >> 172.21.15.40:0/2102104714 conn(0x7f6ffc0b17b0 msgr2=0x7f6ffc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f7003800640 1 -- 172.21.15.40:0/2102104714 shutdown_connections 2024-03-24T00:39:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f7003800640 1 -- 172.21.15.40:0/2102104714 wait complete. 2024-03-24T00:39:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f7003800640 1 Processor -- start 2024-03-24T00:39:26.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.884+0000 7f7003800640 1 -- start start 2024-03-24T00:39:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7003800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc072310 0x7f6ffc13a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7003800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc13ec80 0x7f6ffc140350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7003800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc141bc0 0x7f6ffc140890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7003800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc169f00 con 0x7f6ffc13ec80 2024-03-24T00:39:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7003800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc169d80 con 0x7f6ffc072310 2024-03-24T00:39:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7003800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc16a080 con 0x7f6ffc141bc0 2024-03-24T00:39:26.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f6ffbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc13ec80 0x7f6ffc140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7000e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc072310 0x7f6ffc13a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7000e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc072310 0x7f6ffc13a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:47020/0 (socket says 172.21.15.40:47020) 2024-03-24T00:39:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7000e00640 1 -- 172.21.15.40:0/2429219905 learned_addr learned my addr 172.21.15.40:0/2429219905 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f6ffbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc13ec80 0x7f6ffc140350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37248/0 (socket says 172.21.15.40:37248) 2024-03-24T00:39:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7001800640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc141bc0 0x7f6ffc140890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7000e00640 1 -- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc141bc0 msgr2=0x7f6ffc140890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7000e00640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc141bc0 0x7f6ffc140890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7000e00640 1 -- 172.21.15.40:0/2429219905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc13ec80 msgr2=0x7f6ffc140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7000e00640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc13ec80 0x7f6ffc140350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7000e00640 1 -- 172.21.15.40:0/2429219905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc13aaa0 con 0x7f6ffc072310 2024-03-24T00:39:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f6ffbe00640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc13ec80 0x7f6ffc140350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:26.889 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7000e00640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc072310 0x7f6ffc13a560 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000c500 tx=0x7f6ff000c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f6ff8c00640 1 -- 172.21.15.40:0/2429219905 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0016020 con 0x7f6ffc072310 2024-03-24T00:39:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f6ff8c00640 1 -- 172.21.15.40:0/2429219905 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff0005220 con 0x7f6ffc072310 2024-03-24T00:39:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7003800640 1 -- 172.21.15.40:0/2429219905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc0b4480 con 0x7f6ffc072310 2024-03-24T00:39:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f6ff8c00640 1 -- 172.21.15.40:0/2429219905 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0007030 con 0x7f6ffc072310 2024-03-24T00:39:26.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.888+0000 7f7003800640 1 -- 172.21.15.40:0/2429219905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc156df0 con 0x7f6ffc072310 2024-03-24T00:39:26.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f6ff8c00640 1 -- 172.21.15.40:0/2429219905 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f6ff00071d0 con 0x7f6ffc072310 2024-03-24T00:39:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f6ff8c00640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fdc067690 0x7f6fdc069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f6ff8c00640 1 -- 172.21.15.40:0/2429219905 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ff0085110 con 0x7f6ffc072310 2024-03-24T00:39:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f6ffbe00640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fdc067690 0x7f6fdc069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f7006bed000 1 Processor -- start 2024-03-24T00:39:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f7006bed000 1 -- start start 2024-03-24T00:39:26.893 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f6ffbe00640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fdc067690 0x7f6fdc069b50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc141260 tx=0x7f6fe8007b10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:26.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f7006bed000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a36d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f7006bed000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c689ecab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f7006bed000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c689ed3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f7006bed000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6883a680 con 0x564c68a37240 2024-03-24T00:39:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f7006bed000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c687c1b00 con 0x564c68a36920 2024-03-24T00:39:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f7006bed000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x564c68836720 con 0x564c689ecff0 2024-03-24T00:39:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f6ffbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c689ecab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f6ffbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c689ecab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37260/0 (socket says 172.21.15.40:37260) 2024-03-24T00:39:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f6ffbe00640 1 -- 172.21.15.40:0/4193107704 learned_addr learned my addr 172.21.15.40:0/4193107704 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f7000e00640 1 --2- 172.21.15.40:0/4193107704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a36d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.892+0000 7f7001800640 1 --2- 172.21.15.40:0/4193107704 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c689ed3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7001800640 1 -- 172.21.15.40:0/4193107704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 msgr2=0x564c68a36d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7001800640 1 --2- 172.21.15.40:0/4193107704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a36d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7001800640 1 -- 172.21.15.40:0/4193107704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 msgr2=0x564c689ecab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7001800640 1 --2- 172.21.15.40:0/4193107704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c689ecab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7001800640 1 -- 172.21.15.40:0/4193107704 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c68812940 con 0x564c689ecff0 2024-03-24T00:39:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7000e00640 1 --2- 172.21.15.40:0/4193107704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a36d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f6ffbe00640 1 --2- 172.21.15.40:0/4193107704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c689ecab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:26.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7001800640 1 --2- 172.21.15.40:0/4193107704 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c689ed3d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40ac000 tx=0x7f6ff4004ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be807fce528e6dca server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f6fed600640 1 -- 172.21.15.40:0/4193107704 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40b2040 con 0x564c689ecff0 2024-03-24T00:39:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f6fed600640 1 -- 172.21.15.40:0/4193107704 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff40b3040 con 0x564c689ecff0 2024-03-24T00:39:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f6fed600640 1 -- 172.21.15.40:0/4193107704 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40ad880 con 0x564c689ecff0 2024-03-24T00:39:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7006bed000 1 -- 172.21.15.40:0/4193107704 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 msgr2=0x564c689ed3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.897 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7006bed000 1 --2- 172.21.15.40:0/4193107704 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c689ed3d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6ff40ac000 tx=0x7f6ff4004ac0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7006bed000 1 -- 172.21.15.40:0/4193107704 shutdown_connections 2024-03-24T00:39:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7006bed000 1 --2- 172.21.15.40:0/4193107704 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c689ed3d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7006bed000 1 --2- 172.21.15.40:0/4193107704 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c689ecab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7006bed000 1 --2- 172.21.15.40:0/4193107704 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a36d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7006bed000 1 -- 172.21.15.40:0/4193107704 >> 172.21.15.40:0/4193107704 conn(0x564c689e8fa0 msgr2=0x564c689eb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7006bed000 1 -- 172.21.15.40:0/4193107704 shutdown_connections 2024-03-24T00:39:26.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.896+0000 7f7006bed000 1 -- 172.21.15.40:0/4193107704 wait complete. 2024-03-24T00:39:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 1 -- start start 2024-03-24T00:39:26.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 1 Processor -- start 2024-03-24T00:39:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c68a1bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a1c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c68a1c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c68839990 con 0x564c68a37240 2024-03-24T00:39:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c687f4ae0 con 0x564c68a36920 2024-03-24T00:39:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x564c687d2750 con 0x564c689ecff0 2024-03-24T00:39:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 10 client.0 _ll_register_callbacks cb 0x564c689d7ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7000e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c68a1bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7001800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c68a1c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7000e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c68a1bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:55354/0 (socket says 172.21.15.40:55354) 2024-03-24T00:39:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7000e00640 1 -- 172.21.15.40:0/1362693134 learned_addr learned my addr 172.21.15.40:0/1362693134 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:26.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f6ffbe00640 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a1c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7000e00640 1 -- 172.21.15.40:0/1362693134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 msgr2=0x564c68a1c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:26.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7000e00640 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a1c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7000e00640 1 -- 172.21.15.40:0/1362693134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 msgr2=0x564c68a1c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:26.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7000e00640 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c68a1c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7000e00640 1 -- 172.21.15.40:0/1362693134 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c6887fd80 con 0x564c689ecff0 2024-03-24T00:39:26.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7001800640 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c68a1c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:26.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7000e00640 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c68a1bd40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00556e0 tx=0x7f6ff0055890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=620159de056240f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:26.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f6fe2000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:39:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f6fe2000640 1 -- 172.21.15.40:0/1362693134 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff0072060 con 0x564c689ecff0 2024-03-24T00:39:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f6fe2000640 1 -- 172.21.15.40:0/1362693134 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ff0054da0 con 0x564c689ecff0 2024-03-24T00:39:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 1 -- 172.21.15.40:0/1362693134 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ff40b3040 con 0x564c689ecff0 2024-03-24T00:39:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f6fe2000640 1 -- 172.21.15.40:0/1362693134 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ff0053b10 con 0x564c689ecff0 2024-03-24T00:39:26.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 10 client.7640 fetch_fsmap learned FSMap version 134 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 1 -- 172.21.15.40:0/1362693134 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=134,monmap=2+}) v3 -- 0x564c687c1d10 con 0x564c689ecff0 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f6fe2000640 1 -- 172.21.15.40:0/1362693134 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff40b3040 con 0x564c689ecff0 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f6fe2000640 1 -- 172.21.15.40:0/1362693134 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 134) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f6ff0072200 con 0x564c689ecff0 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 10 client.7640 fetch_fsmap finished waiting for FSMap version 134 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 10 client.7640 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 20 client.7640 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x564c68a1d090 0x564c68a48ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 4 client.7640 mds_command: new command op to 8563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f7006bed000 1 -- 172.21.15.40:0/1362693134 --> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c687d5b60 con 0x564c68a1d090 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.900+0000 7f6ffbe00640 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x564c68a1d090 0x564c68a48ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f6ffbe00640 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x564c68a1d090 0x564c68a48ea0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e463522fd7c7cca2 server_cookie=df9bc67f599e1536 in_seq=0 out_seq=0 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f6fe2000640 10 client.7640 ms_handle_connect on v2:172.21.15.53:6834/2255755838 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f6fe2000640 1 -- 172.21.15.40:0/1362693134 <== mds.2 v2:172.21.15.53:6834/2255755838 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564c687d5b60 con 0x564c68a1d090 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f6fe2000640 10 client.7640 handle_command_reply: tid=0 2024-03-24T00:39:26.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f7006bed000 1 client.7640 shutdown 2024-03-24T00:39:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f7006bed000 1 -- 172.21.15.40:0/1362693134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 msgr2=0x564c68a1bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f7006bed000 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c68a1bd40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00556e0 tx=0x7f6ff0055890 comp rx=0 tx=0).stop 2024-03-24T00:39:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f7006bed000 1 -- 172.21.15.40:0/1362693134 shutdown_connections 2024-03-24T00:39:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f7006bed000 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x564c68a1d090 0x564c68a48ea0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f7006bed000 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c68a1c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f7006bed000 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a1c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f6fe2000640 0 client.7640 ms_handle_reset on v2:172.21.15.53:6834/2255755838 2024-03-24T00:39:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f7006bed000 1 --2- 172.21.15.40:0/1362693134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c68a1bd40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f7006bed000 1 -- 172.21.15.40:0/1362693134 >> 172.21.15.40:0/1362693134 conn(0x564c689e8fa0 msgr2=0x564c689ebc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f7006bed000 1 -- 172.21.15.40:0/1362693134 shutdown_connections 2024-03-24T00:39:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f7006bed000 1 -- 172.21.15.40:0/1362693134 wait complete. 2024-03-24T00:39:26.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.904+0000 7f7006bed000 20 client.7640 trim_cache size 0 max 16384 2024-03-24T00:39:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7006bed000 1 Processor -- start 2024-03-24T00:39:26.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7006bed000 1 -- start start 2024-03-24T00:39:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7006bed000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c68a35530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7006bed000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a35a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7006bed000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c68a98170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7006bed000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc203310 con 0x564c68a37240 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7006bed000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc202690 con 0x564c68a36920 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7006bed000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc202cb0 con 0x564c689ecff0 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7001800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c68a98170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f6ffbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a35a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7000e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c68a35530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f6ffbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a35a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:47040/0 (socket says 172.21.15.40:47040) 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7000e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c68a35530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:55366/0 (socket says 172.21.15.40:55366) 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f6ffbe00640 1 -- 172.21.15.40:0/3958691150 learned_addr learned my addr 172.21.15.40:0/3958691150 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7001800640 1 -- 172.21.15.40:0/3958691150 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 msgr2=0x564c68a35530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7001800640 1 --2- 172.21.15.40:0/3958691150 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c68a35530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7001800640 1 -- 172.21.15.40:0/3958691150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 msgr2=0x564c68a35a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7001800640 1 --2- 172.21.15.40:0/3958691150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a35a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7001800640 1 -- 172.21.15.40:0/3958691150 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c689f1200 con 0x564c68a37240 2024-03-24T00:39:26.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.920+0000 7f7000e00640 1 --2- 172.21.15.40:0/3958691150 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c68a35530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f7001800640 1 --2- 172.21.15.40:0/3958691150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c68a98170 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4019580 tx=0x7f6ff40a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4050ddc178db255b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:26.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f6fecc00640 1 -- 172.21.15.40:0/3958691150 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff40b2040 con 0x564c68a37240 2024-03-24T00:39:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f6fecc00640 1 -- 172.21.15.40:0/3958691150 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff40c6070 con 0x564c68a37240 2024-03-24T00:39:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f6fecc00640 1 -- 172.21.15.40:0/3958691150 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4004280 con 0x564c68a37240 2024-03-24T00:39:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f7006bed000 1 -- 172.21.15.40:0/3958691150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 msgr2=0x564c68a98170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f7006bed000 1 --2- 172.21.15.40:0/3958691150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c68a98170 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4019580 tx=0x7f6ff40a7000 comp rx=0 tx=0).stop 2024-03-24T00:39:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f7006bed000 1 -- 172.21.15.40:0/3958691150 shutdown_connections 2024-03-24T00:39:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f7006bed000 1 --2- 172.21.15.40:0/3958691150 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c68a37240 0x564c68a98170 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f7006bed000 1 --2- 172.21.15.40:0/3958691150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a36920 0x564c68a35a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f7006bed000 1 --2- 172.21.15.40:0/3958691150 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c689ecff0 0x564c68a35530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f7006bed000 1 -- 172.21.15.40:0/3958691150 >> 172.21.15.40:0/3958691150 conn(0x564c689e8fa0 msgr2=0x564c68a43f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f7006bed000 1 -- 172.21.15.40:0/3958691150 shutdown_connections 2024-03-24T00:39:26.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f7006bed000 1 -- 172.21.15.40:0/3958691150 wait complete. 2024-03-24T00:39:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f7006bed000 1 -- start start 2024-03-24T00:39:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.924+0000 7f7006bed000 1 Processor -- start 2024-03-24T00:39:26.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c689ecff0 0x564c68a49ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c68a36920 0x564c68a4a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a37240 0x564c68a4a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c687e2a80 con 0x564c689ecff0 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc1583b0 con 0x564c68a37240 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x564c688f7700 con 0x564c68a36920 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 10 client.0 _ll_register_callbacks cb 0x7f6ff00075b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7000e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c689ecff0 0x564c68a49ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6ffbe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c68a36920 0x564c68a4a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7001800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a37240 0x564c68a4a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6ffbe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c68a36920 0x564c68a4a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:55380/0 (socket says 172.21.15.40:55380) 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7000e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c689ecff0 0x564c68a49ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:37276/0 (socket says 172.21.15.40:37276) 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6ffbe00640 1 -- 172.21.15.40:0/3338731633 learned_addr learned my addr 172.21.15.40:0/3338731633 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:26.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6ffbe00640 1 -- 172.21.15.40:0/3338731633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a37240 msgr2=0x564c68a4a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:26.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6ffbe00640 1 --2- 172.21.15.40:0/3338731633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a37240 0x564c68a4a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6ffbe00640 1 -- 172.21.15.40:0/3338731633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c689ecff0 msgr2=0x564c68a49ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:26.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6ffbe00640 1 --2- 172.21.15.40:0/3338731633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c689ecff0 0x564c68a49ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6ffbe00640 1 -- 172.21.15.40:0/3338731633 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c68807540 con 0x564c68a36920 2024-03-24T00:39:26.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7000e00640 1 --2- 172.21.15.40:0/3338731633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c689ecff0 0x564c68a49ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:26.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6ffbe00640 1 --2- 172.21.15.40:0/3338731633 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c68a36920 0x564c68a4a430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8001ec0 tx=0x7f6fe8013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3389e2fc75038aa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:26.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6fe2000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:39:26.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6fe2000640 1 -- 172.21.15.40:0/3338731633 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8029730 con 0x564c68a36920 2024-03-24T00:39:26.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6fe2000640 1 -- 172.21.15.40:0/3338731633 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6fe80298d0 con 0x564c68a36920 2024-03-24T00:39:26.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 -- 172.21.15.40:0/3338731633 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ff40c6070 con 0x564c68a36920 2024-03-24T00:39:26.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6fe2000640 1 -- 172.21.15.40:0/3338731633 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fe8029ad0 con 0x564c68a36920 2024-03-24T00:39:26.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 10 client.7646 fetch_fsmap learned FSMap version 134 2024-03-24T00:39:26.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 -- 172.21.15.40:0/3338731633 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=134,monmap=2+}) v3 -- 0x564c687ce4b0 con 0x564c68a36920 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6fe2000640 1 -- 172.21.15.40:0/3338731633 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff40c6070 con 0x564c68a36920 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6fe2000640 1 -- 172.21.15.40:0/3338731633 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 134) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f6fe803d080 con 0x564c68a36920 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 10 client.7646 fetch_fsmap finished waiting for FSMap version 134 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 10 client.7646 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 20 client.7646 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 --2- 172.21.15.40:0/3338731633 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x564c68aa21e0 0x564c68aa45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 4 client.7646 mds_command: new command op to 8563 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 -- 172.21.15.40:0/3338731633 --> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564c68847bc0 con 0x564c68aa21e0 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7000e00640 1 --2- 172.21.15.40:0/3338731633 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x564c68aa21e0 0x564c68aa45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7000e00640 1 --2- 172.21.15.40:0/3338731633 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x564c68aa21e0 0x564c68aa45d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a3875860075b60ae server_cookie=b6197fd9afce4aff in_seq=0 out_seq=0 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6fe2000640 10 client.7646 ms_handle_connect on v2:172.21.15.53:6834/2255755838 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6fe2000640 1 -- 172.21.15.40:0/3338731633 <== mds.2 v2:172.21.15.53:6834/2255755838 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x564c68847bc0 con 0x564c68aa21e0 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6fe2000640 10 client.7646 handle_command_reply: tid=0 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 client.7646 shutdown 2024-03-24T00:39:26.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 -- 172.21.15.40:0/3338731633 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c68a36920 msgr2=0x564c68a4a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 --2- 172.21.15.40:0/3338731633 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c68a36920 0x564c68a4a430 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8001ec0 tx=0x7f6fe8013000 comp rx=0 tx=0).stop 2024-03-24T00:39:26.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 -- 172.21.15.40:0/3338731633 shutdown_connections 2024-03-24T00:39:26.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 --2- 172.21.15.40:0/3338731633 >> [v2:172.21.15.53:6834/2255755838,v1:172.21.15.53:6836/2255755838] conn(0x564c68aa21e0 0x564c68aa45d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 --2- 172.21.15.40:0/3338731633 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x564c68a37240 0x564c68a4a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 --2- 172.21.15.40:0/3338731633 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x564c68a36920 0x564c68a4a430 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f6fe2000640 0 client.7646 ms_handle_reset on v2:172.21.15.53:6834/2255755838 2024-03-24T00:39:26.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 --2- 172.21.15.40:0/3338731633 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x564c689ecff0 0x564c68a49ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.928+0000 7f7006bed000 1 -- 172.21.15.40:0/3338731633 >> 172.21.15.40:0/3338731633 conn(0x564c689e8fa0 msgr2=0x564c68a195d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:26.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7006bed000 1 -- 172.21.15.40:0/3338731633 shutdown_connections 2024-03-24T00:39:26.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7006bed000 1 -- 172.21.15.40:0/3338731633 wait complete. 2024-03-24T00:39:26.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7006bed000 20 client.7646 trim_cache size 0 max 16384 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 2, 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8563, 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 4.0542499220000003, 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 134, 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 119, 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 119, 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 23.644974177999998 2024-03-24T00:39:26.934 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:39:26.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7003800640 1 -- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fdc067690 msgr2=0x7f6fdc069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7003800640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fdc067690 0x7f6fdc069b50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc141260 tx=0x7f6fe8007b10 comp rx=0 tx=0).stop 2024-03-24T00:39:26.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7003800640 1 -- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc072310 msgr2=0x7f6ffc13a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:26.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7003800640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc072310 0x7f6ffc13a560 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6ff000c500 tx=0x7f6ff000c9c0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7003800640 1 -- 172.21.15.40:0/2429219905 shutdown_connections 2024-03-24T00:39:26.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7003800640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f6fdc067690 0x7f6fdc069b50 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7003800640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ffc141bc0 0x7f6ffc140890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7003800640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f6ffc13ec80 0x7f6ffc140350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7003800640 1 --2- 172.21.15.40:0/2429219905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc072310 0x7f6ffc13a560 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:26.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7003800640 1 -- 172.21.15.40:0/2429219905 >> 172.21.15.40:0/2429219905 conn(0x7f6ffc0b17b0 msgr2=0x7f6ffc071300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:26.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7003800640 1 -- 172.21.15.40:0/2429219905 shutdown_connections 2024-03-24T00:39:26.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:26.932+0000 7f7003800640 1 -- 172.21.15.40:0/2429219905 wait complete. 2024-03-24T00:39:26.958 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:39:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.100+0000 7fd50d600640 1 Processor -- start 2024-03-24T00:39:27.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.100+0000 7fd50d600640 1 -- start start 2024-03-24T00:39:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd50d600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd5080d08e0 0x7fd50815f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd50d600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5080b43a0 0x7fd50815f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd50d600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5080af710 con 0x7fd5080cfdf0 2024-03-24T00:39:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd50d600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5080b2bb0 con 0x7fd5080d08e0 2024-03-24T00:39:27.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd50d600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080b26a0 con 0x7fd5080b43a0 2024-03-24T00:39:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd4fe000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd5080d08e0 0x7fd50815f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd506a00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fd5080cfdf0 0x7fd5080d01d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34746/0 (socket says 172.21.15.40:34746) 2024-03-24T00:39:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd506a00640 1 -- 172.21.15.40:0/2653670277 learned_addr learned my addr 172.21.15.40:0/2653670277 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd507400640 1 --2- 172.21.15.40:0/2653670277 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5080b43a0 0x7fd50815f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd4fe000640 1 -- 172.21.15.40:0/2653670277 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5080b43a0 msgr2=0x7fd50815f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd4fe000640 1 --2- 172.21.15.40:0/2653670277 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5080b43a0 0x7fd50815f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd4fe000640 1 -- 172.21.15.40:0/2653670277 >> v1:172.21.15.40:6789/0 conn(0x7fd5080cfdf0 legacy=0x7fd5080d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.104+0000 7fd4fe000640 1 -- 172.21.15.40:0/2653670277 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5080b34f0 con 0x7fd5080d08e0 2024-03-24T00:39:27.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.108+0000 7fd4fe000640 1 --2- 172.21.15.40:0/2653670277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5080d08e0 0x7fd50815f090 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00097c0 tx=0x7fd4f001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10cccaf7a70fa4ed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.108+0000 7fd506000640 1 -- 172.21.15.40:0/2653670277 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f00b1070 con 0x7fd5080d08e0 2024-03-24T00:39:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.108+0000 7fd506000640 1 -- 172.21.15.40:0/2653670277 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f00ac410 con 0x7fd5080d08e0 2024-03-24T00:39:27.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.108+0000 7fd506000640 1 -- 172.21.15.40:0/2653670277 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f00ac7e0 con 0x7fd5080d08e0 2024-03-24T00:39:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.108+0000 7fd50d600640 1 -- 172.21.15.40:0/2653670277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5080d08e0 msgr2=0x7fd50815f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.108+0000 7fd50d600640 1 --2- 172.21.15.40:0/2653670277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5080d08e0 0x7fd50815f090 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00097c0 tx=0x7fd4f001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.108+0000 7fd50d600640 1 -- 172.21.15.40:0/2653670277 shutdown_connections 2024-03-24T00:39:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.108+0000 7fd50d600640 1 --2- 172.21.15.40:0/2653670277 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5080b43a0 0x7fd50815f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.108+0000 7fd50d600640 1 --2- 172.21.15.40:0/2653670277 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd5080d08e0 0x7fd50815f090 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.108+0000 7fd50d600640 1 -- 172.21.15.40:0/2653670277 >> 172.21.15.40:0/2653670277 conn(0x7fd5080b3880 msgr2=0x7fd5080b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.108+0000 7fd50d600640 1 -- 172.21.15.40:0/2653670277 shutdown_connections 2024-03-24T00:39:27.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.108+0000 7fd50d600640 1 -- 172.21.15.40:0/2653670277 wait complete. 2024-03-24T00:39:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd50d600640 1 Processor -- start 2024-03-24T00:39:27.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd50d600640 1 -- start start 2024-03-24T00:39:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd50d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5080b43a0 0x7fd508155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd50d600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5080cfdf0 0x7fd508155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd50d600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd508156dc0 0x7fd508155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd50d600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080b0810 con 0x7fd508156dc0 2024-03-24T00:39:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd50d600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080b0690 con 0x7fd5080b43a0 2024-03-24T00:39:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd50d600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080b0990 con 0x7fd5080cfdf0 2024-03-24T00:39:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd507400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd508156dc0 0x7fd508155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd4fe000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5080cfdf0 0x7fd508155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd506a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5080b43a0 0x7fd508155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.114 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd507400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd508156dc0 0x7fd508155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48670/0 (socket says 172.21.15.40:48670) 2024-03-24T00:39:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd4fe000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5080cfdf0 0x7fd508155860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:39334/0 (socket says 172.21.15.40:39334) 2024-03-24T00:39:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd507400640 1 -- 172.21.15.40:0/3196673931 learned_addr learned my addr 172.21.15.40:0/3196673931 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd507400640 1 -- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5080cfdf0 msgr2=0x7fd508155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd507400640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5080cfdf0 0x7fd508155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd507400640 1 -- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5080b43a0 msgr2=0x7fd508155000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd507400640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5080b43a0 0x7fd508155000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd507400640 1 -- 172.21.15.40:0/3196673931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5081585a0 con 0x7fd508156dc0 2024-03-24T00:39:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd4fe000640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5080cfdf0 0x7fd508155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd506a00640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5080b43a0 0x7fd508155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:27.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd507400640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd508156dc0 0x7fd508155da0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fd500009e30 tx=0x7fd50000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd4ff400640 1 -- 172.21.15.40:0/3196673931 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd500018020 con 0x7fd508156dc0 2024-03-24T00:39:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd4ff400640 1 -- 172.21.15.40:0/3196673931 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd500014480 con 0x7fd508156dc0 2024-03-24T00:39:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd50d600640 1 -- 172.21.15.40:0/3196673931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd508158810 con 0x7fd508156dc0 2024-03-24T00:39:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd4ff400640 1 -- 172.21.15.40:0/3196673931 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd500014870 con 0x7fd508156dc0 2024-03-24T00:39:27.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.112+0000 7fd50d600640 1 -- 172.21.15.40:0/3196673931 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5080b72a0 con 0x7fd508156dc0 2024-03-24T00:39:27.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.116+0000 7fd4ff400640 1 -- 172.21.15.40:0/3196673931 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd50001b020 con 0x7fd508156dc0 2024-03-24T00:39:27.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.116+0000 7fd4ff400640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd4e4067690 0x7fd4e4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.116+0000 7fd4ff400640 1 -- 172.21.15.40:0/3196673931 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd500086600 con 0x7fd508156dc0 2024-03-24T00:39:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd506a00640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd4e4067690 0x7fd4e4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd5109cd000 1 Processor -- start 2024-03-24T00:39:27.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd5109cd000 1 -- start start 2024-03-24T00:39:27.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd506a00640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd4e4067690 0x7fd4e4069b50 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8000f70 tx=0x7fd4f8007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd5109cd000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475362920 0x55c475362d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd5109cd000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475363240 0x55c475318ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd5109cd000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c4753193d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd5109cd000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c475166680 con 0x55c475318ff0 2024-03-24T00:39:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd5109cd000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4750edb00 con 0x55c475363240 2024-03-24T00:39:27.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd5109cd000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c475162720 con 0x55c475362920 2024-03-24T00:39:27.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd506a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475362920 0x55c475362d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd4fe000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475363240 0x55c475318ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd507400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c4753193d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd506a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475362920 0x55c475362d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:39340/0 (socket says 172.21.15.40:39340) 2024-03-24T00:39:27.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd4fe000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475363240 0x55c475318ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:36860/0 (socket says 172.21.15.40:36860) 2024-03-24T00:39:27.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd506a00640 1 -- 172.21.15.40:0/456273644 learned_addr learned my addr 172.21.15.40:0/456273644 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd506a00640 1 -- 172.21.15.40:0/456273644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475363240 msgr2=0x55c475318ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.123 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd506a00640 1 --2- 172.21.15.40:0/456273644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475363240 0x55c475318ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd506a00640 1 -- 172.21.15.40:0/456273644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 msgr2=0x55c4753193d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd506a00640 1 --2- 172.21.15.40:0/456273644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c4753193d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd506a00640 1 -- 172.21.15.40:0/456273644 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c47513e940 con 0x55c475362920 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd4fe000640 1 --2- 172.21.15.40:0/456273644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475363240 0x55c475318ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd506a00640 1 --2- 172.21.15.40:0/456273644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475362920 0x55c475362d00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8001c30 tx=0x7fd4f8012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1b419458701a8a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd4f7400640 1 -- 172.21.15.40:0/456273644 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8026070 con 0x55c475362920 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd4f7400640 1 -- 172.21.15.40:0/456273644 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f8006240 con 0x55c475362920 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.120+0000 7fd4f7400640 1 -- 172.21.15.40:0/456273644 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80065f0 con 0x55c475362920 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 -- 172.21.15.40:0/456273644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475362920 msgr2=0x55c475362d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 --2- 172.21.15.40:0/456273644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475362920 0x55c475362d00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8001c30 tx=0x7fd4f8012000 comp rx=0 tx=0).stop 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 -- 172.21.15.40:0/456273644 shutdown_connections 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 --2- 172.21.15.40:0/456273644 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c4753193d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 --2- 172.21.15.40:0/456273644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475363240 0x55c475318ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 --2- 172.21.15.40:0/456273644 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475362920 0x55c475362d00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 -- 172.21.15.40:0/456273644 >> 172.21.15.40:0/456273644 conn(0x55c475314fa0 msgr2=0x55c475317390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 -- 172.21.15.40:0/456273644 shutdown_connections 2024-03-24T00:39:27.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 -- 172.21.15.40:0/456273644 wait complete. 2024-03-24T00:39:27.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 -- start start 2024-03-24T00:39:27.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 Processor -- start 2024-03-24T00:39:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475318ff0 0x55c475347d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475362920 0x55c475348280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475363240 0x55c4753487c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c475165990 con 0x55c475363240 2024-03-24T00:39:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c475120ae0 con 0x55c475362920 2024-03-24T00:39:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c4750fe750 con 0x55c475318ff0 2024-03-24T00:39:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd5109cd000 10 client.0 _ll_register_callbacks cb 0x55c475303ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd507400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475363240 0x55c4753487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd4fe000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475362920 0x55c475348280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd506a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475318ff0 0x55c475347d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd507400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475363240 0x55c4753487c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48700/0 (socket says 172.21.15.40:48700) 2024-03-24T00:39:27.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.124+0000 7fd507400640 1 -- 172.21.15.40:0/2684547748 learned_addr learned my addr 172.21.15.40:0/2684547748 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:27.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd507400640 1 -- 172.21.15.40:0/2684547748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475318ff0 msgr2=0x55c475347d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd507400640 1 --2- 172.21.15.40:0/2684547748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475318ff0 0x55c475347d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd507400640 1 -- 172.21.15.40:0/2684547748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475362920 msgr2=0x55c475348280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd507400640 1 --2- 172.21.15.40:0/2684547748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475362920 0x55c475348280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd507400640 1 -- 172.21.15.40:0/2684547748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4751abd80 con 0x55c475363240 2024-03-24T00:39:27.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd4fe000640 1 --2- 172.21.15.40:0/2684547748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475362920 0x55c475348280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd507400640 1 --2- 172.21.15.40:0/2684547748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475363240 0x55c4753487c0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fd500009e00 tx=0x7fd500009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6b157802f40f45 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd4ebe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:39:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd4ebe00640 1 -- 172.21.15.40:0/2684547748 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd500083060 con 0x55c475363240 2024-03-24T00:39:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd4ebe00640 1 -- 172.21.15.40:0/2684547748 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd50005eff0 con 0x55c475363240 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 -- 172.21.15.40:0/2684547748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4f8006240 con 0x55c475363240 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd4ebe00640 1 -- 172.21.15.40:0/2684547748 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd500056070 con 0x55c475363240 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 10 client.9168 fetch_fsmap learned FSMap version 134 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 -- 172.21.15.40:0/2684547748 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=2+}) v3 -- 0x55c4750edd10 con 0x55c475363240 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd4ebe00640 1 -- 172.21.15.40:0/2684547748 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f8006240 con 0x55c475363240 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd4ebe00640 1 -- 172.21.15.40:0/2684547748 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 134) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fd500056210 con 0x55c475363240 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 10 client.9168 fetch_fsmap finished waiting for FSMap version 134 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 10 client.9168 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 20 client.9168 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 --2- 172.21.15.40:0/2684547748 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x55c475349030 0x55c475374ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 4 client.9168 mds_command: new command op to 8874 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 -- 172.21.15.40:0/2684547748 --> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c475101b60 con 0x55c475349030 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd506a00640 1 --2- 172.21.15.40:0/2684547748 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x55c475349030 0x55c475374ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd4ebe00640 10 client.9168 ms_handle_connect on v2:172.21.15.40:6832/4273998586 2024-03-24T00:39:27.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd506a00640 1 --2- 172.21.15.40:0/2684547748 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x55c475349030 0x55c475374ea0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9bc13b78674c45a4 server_cookie=9a3db00d26ee3629 in_seq=0 out_seq=0 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd4ebe00640 1 -- 172.21.15.40:0/2684547748 <== mds.1 v2:172.21.15.40:6832/4273998586 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c475101b60 con 0x55c475349030 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd4ebe00640 10 client.9168 handle_command_reply: tid=0 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 client.9168 shutdown 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 -- 172.21.15.40:0/2684547748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475363240 msgr2=0x55c4753487c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 --2- 172.21.15.40:0/2684547748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475363240 0x55c4753487c0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fd500009e00 tx=0x7fd500009d50 comp rx=0 tx=0).stop 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 -- 172.21.15.40:0/2684547748 shutdown_connections 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 --2- 172.21.15.40:0/2684547748 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x55c475349030 0x55c475374ea0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 --2- 172.21.15.40:0/2684547748 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475363240 0x55c4753487c0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 --2- 172.21.15.40:0/2684547748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475362920 0x55c475348280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 --2- 172.21.15.40:0/2684547748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475318ff0 0x55c475347d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 -- 172.21.15.40:0/2684547748 >> 172.21.15.40:0/2684547748 conn(0x55c475314fa0 msgr2=0x55c475317c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd4ebe00640 0 client.9168 ms_handle_reset on v2:172.21.15.40:6832/4273998586 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 -- 172.21.15.40:0/2684547748 shutdown_connections 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 1 -- 172.21.15.40:0/2684547748 wait complete. 2024-03-24T00:39:27.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.128+0000 7fd5109cd000 20 client.9168 trim_cache size 0 max 16384 2024-03-24T00:39:27.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 Processor -- start 2024-03-24T00:39:27.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 -- start start 2024-03-24T00:39:27.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475347ed0 0x55c475317390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475314fa0 0x55c4753178d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c475317e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5081fec60 con 0x55c475318ff0 2024-03-24T00:39:27.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50806c820 con 0x55c475314fa0 2024-03-24T00:39:27.149 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd50806ce20 con 0x55c475347ed0 2024-03-24T00:39:27.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd507400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c475317e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd506a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475347ed0 0x55c475317390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd507400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c475317e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48706/0 (socket says 172.21.15.40:48706) 2024-03-24T00:39:27.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd507400640 1 -- 172.21.15.40:0/1115483851 learned_addr learned my addr 172.21.15.40:0/1115483851 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd4fe000640 1 --2- 172.21.15.40:0/1115483851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475314fa0 0x55c4753178d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd506a00640 1 -- 172.21.15.40:0/1115483851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475314fa0 msgr2=0x55c4753178d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd506a00640 1 --2- 172.21.15.40:0/1115483851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475314fa0 0x55c4753178d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd506a00640 1 -- 172.21.15.40:0/1115483851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 msgr2=0x55c475317e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd506a00640 1 --2- 172.21.15.40:0/1115483851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c475317e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd506a00640 1 -- 172.21.15.40:0/1115483851 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c47531d1d0 con 0x55c475347ed0 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd507400640 1 --2- 172.21.15.40:0/1115483851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c475317e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd506a00640 1 --2- 172.21.15.40:0/1115483851 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475347ed0 0x55c475317390 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8023000 tx=0x7fd4f8013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc78e8d61a1c8425 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd4f6a00640 1 -- 172.21.15.40:0/1115483851 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8039070 con 0x55c475347ed0 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd4f6a00640 1 -- 172.21.15.40:0/1115483851 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f8034070 con 0x55c475347ed0 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd4f6a00640 1 -- 172.21.15.40:0/1115483851 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f8026040 con 0x55c475347ed0 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 -- 172.21.15.40:0/1115483851 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475347ed0 msgr2=0x55c475317390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 --2- 172.21.15.40:0/1115483851 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475347ed0 0x55c475317390 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8023000 tx=0x7fd4f8013000 comp rx=0 tx=0).stop 2024-03-24T00:39:27.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 -- 172.21.15.40:0/1115483851 shutdown_connections 2024-03-24T00:39:27.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 --2- 172.21.15.40:0/1115483851 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c475317e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 --2- 172.21.15.40:0/1115483851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475314fa0 0x55c4753178d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 --2- 172.21.15.40:0/1115483851 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475347ed0 0x55c475317390 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 -- 172.21.15.40:0/1115483851 >> 172.21.15.40:0/1115483851 conn(0x55c47531b750 msgr2=0x55c4753455d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 -- 172.21.15.40:0/1115483851 shutdown_connections 2024-03-24T00:39:27.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.148+0000 7fd5109cd000 1 -- 172.21.15.40:0/1115483851 wait complete. 2024-03-24T00:39:27.154 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 1 -- start start 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 1 Processor -- start 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475314fa0 0x55c475318730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c475375ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475347ed0 0x55c475376430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c47510ea80 con 0x55c475318ff0 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50815aba0 con 0x55c475314fa0 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c475223700 con 0x55c475347ed0 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 10 client.0 _ll_register_callbacks cb 0x7fd500056ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4fe000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c475375ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd506a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475314fa0 0x55c475318730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4fe000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c475375ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48708/0 (socket says 172.21.15.40:48708) 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4fe000640 1 -- 172.21.15.40:0/797565702 learned_addr learned my addr 172.21.15.40:0/797565702 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd507400640 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475347ed0 0x55c475376430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4fe000640 1 -- 172.21.15.40:0/797565702 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475347ed0 msgr2=0x55c475376430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4fe000640 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475347ed0 0x55c475376430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4fe000640 1 -- 172.21.15.40:0/797565702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475314fa0 msgr2=0x55c475318730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4fe000640 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475314fa0 0x55c475318730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4fe000640 1 -- 172.21.15.40:0/797565702 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c475133540 con 0x55c475318ff0 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd506a00640 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475314fa0 0x55c475318730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4fe000640 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c475375ef0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fd4f001ab50 tx=0x7fd4f001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9638e2a688a9952 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4ebe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4ebe00640 1 -- 172.21.15.40:0/797565702 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f00a7040 con 0x55c475318ff0 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4ebe00640 1 -- 172.21.15.40:0/797565702 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4f00a9040 con 0x55c475318ff0 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 1 -- 172.21.15.40:0/797565702 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4f8034070 con 0x55c475318ff0 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4ebe00640 1 -- 172.21.15.40:0/797565702 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4f00ca050 con 0x55c475318ff0 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 10 client.9174 fetch_fsmap learned FSMap version 134 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 1 -- 172.21.15.40:0/797565702 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=2+}) v3 -- 0x55c4750fa4b0 con 0x55c475318ff0 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4ebe00640 1 -- 172.21.15.40:0/797565702 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f8034070 con 0x55c475318ff0 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd4ebe00640 1 -- 172.21.15.40:0/797565702 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 134) v1 ==== 3354+0+0 (secure 0 0 0) 0x7fd4f00bd5c0 con 0x55c475318ff0 2024-03-24T00:39:27.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 10 client.9174 fetch_fsmap finished waiting for FSMap version 134 2024-03-24T00:39:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 10 client.9174 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:39:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 20 client.9174 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x55c475362920 0x55c475376970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 4 client.9174 mds_command: new command op to 8874 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:39:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd5109cd000 1 -- 172.21.15.40:0/797565702 --> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c475173bc0 con 0x55c475362920 2024-03-24T00:39:27.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.152+0000 7fd506a00640 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x55c475362920 0x55c475376970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd506a00640 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x55c475362920 0x55c475376970 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f2bff410657b5a8 server_cookie=214842ef99248d2b in_seq=0 out_seq=0 2024-03-24T00:39:27.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd4ebe00640 10 client.9174 ms_handle_connect on v2:172.21.15.40:6832/4273998586 2024-03-24T00:39:27.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd4ebe00640 1 -- 172.21.15.40:0/797565702 <== mds.1 v2:172.21.15.40:6832/4273998586 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x55c475173bc0 con 0x55c475362920 2024-03-24T00:39:27.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd4ebe00640 10 client.9174 handle_command_reply: tid=0 2024-03-24T00:39:27.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd5109cd000 1 client.9174 shutdown 2024-03-24T00:39:27.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd5109cd000 1 -- 172.21.15.40:0/797565702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 msgr2=0x55c475375ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd5109cd000 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c475375ef0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fd4f001ab50 tx=0x7fd4f001ace0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd5109cd000 1 -- 172.21.15.40:0/797565702 shutdown_connections 2024-03-24T00:39:27.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd5109cd000 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.40:6832/4273998586,v1:172.21.15.40:6833/4273998586] conn(0x55c475362920 0x55c475376970 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd5109cd000 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c475347ed0 0x55c475376430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd5109cd000 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x55c475318ff0 0x55c475375ef0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd5109cd000 1 --2- 172.21.15.40:0/797565702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c475314fa0 0x55c475318730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd4ebe00640 0 client.9174 ms_handle_reset on v2:172.21.15.40:6832/4273998586 2024-03-24T00:39:27.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd5109cd000 1 -- 172.21.15.40:0/797565702 >> 172.21.15.40:0/797565702 conn(0x55c47531b750 msgr2=0x55c475347190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd5109cd000 1 -- 172.21.15.40:0/797565702 shutdown_connections 2024-03-24T00:39:27.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd5109cd000 1 -- 172.21.15.40:0/797565702 wait complete. 2024-03-24T00:39:27.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd5109cd000 20 client.9174 trim_cache size 0 max 16384 2024-03-24T00:39:27.159 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:39:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:39:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 1, 2024-03-24T00:39:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8874, 2024-03-24T00:39:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:39:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:39:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:39:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 5.284622766, 2024-03-24T00:39:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 134, 2024-03-24T00:39:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 119, 2024-03-24T00:39:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 119, 2024-03-24T00:39:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 23.879058169 2024-03-24T00:39:27.160 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:39:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd50d600640 1 -- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd4e4067690 msgr2=0x7fd4e4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd50d600640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd4e4067690 0x7fd4e4069b50 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fd4f8000f70 tx=0x7fd4f8007e90 comp rx=0 tx=0).stop 2024-03-24T00:39:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd50d600640 1 -- 172.21.15.40:0/3196673931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd508156dc0 msgr2=0x7fd508155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd50d600640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd508156dc0 0x7fd508155da0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fd500009e30 tx=0x7fd50000c490 comp rx=0 tx=0).stop 2024-03-24T00:39:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd50d600640 1 -- 172.21.15.40:0/3196673931 shutdown_connections 2024-03-24T00:39:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd50d600640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd4e4067690 0x7fd4e4069b50 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd50d600640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd508156dc0 0x7fd508155da0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd50d600640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5080cfdf0 0x7fd508155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd50d600640 1 --2- 172.21.15.40:0/3196673931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5080b43a0 0x7fd508155000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd50d600640 1 -- 172.21.15.40:0/3196673931 >> 172.21.15.40:0/3196673931 conn(0x7fd5080b3880 msgr2=0x7fd5080cc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd50d600640 1 -- 172.21.15.40:0/3196673931 shutdown_connections 2024-03-24T00:39:27.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.156+0000 7fd50d600640 1 -- 172.21.15.40:0/3196673931 wait complete. 2024-03-24T00:39:27.176 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-24T00:39:27.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.316+0000 7f78d5600640 1 Processor -- start 2024-03-24T00:39:27.318 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.316+0000 7f78d5600640 1 -- start start 2024-03-24T00:39:27.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.316+0000 7f78d5600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f78d00d01d0 0x7f78d0070b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.316+0000 7f78d5600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d0160e10 con 0x7f78d00d05b0 2024-03-24T00:39:27.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.316+0000 7f78d5600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d00af420 con 0x7f78d0070250 2024-03-24T00:39:27.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.316+0000 7f78d5600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d00d27d0 con 0x7f78d00d01d0 2024-03-24T00:39:27.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cf400640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f78d00d05b0 0x7f78d0160700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:34758/0 (socket says 172.21.15.40:34758) 2024-03-24T00:39:27.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cf400640 1 -- 172.21.15.40:0/3619022418 learned_addr learned my addr 172.21.15.40:0/3619022418 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:39:27.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78ce000640 1 --1- 172.21.15.40:0/3619022418 >> v1:172.21.15.53:6789/0 conn(0x7f78d0070250 0x7f78d015fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:58210/0 (socket says 172.21.15.40:58210) 2024-03-24T00:39:27.319 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cea00640 1 --2- 172.21.15.40:0/3619022418 >> v2:172.21.15.40:3300/0 conn(0x7f78d00d01d0 0x7f78d0070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cd600640 1 -- 172.21.15.40:0/3619022418 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472526336 0 0) 0x7f78d0160e10 con 0x7f78d00d05b0 2024-03-24T00:39:27.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cd600640 1 -- 172.21.15.40:0/3619022418 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b4003530 con 0x7f78d00d05b0 2024-03-24T00:39:27.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cd600640 1 -- 172.21.15.40:0/3619022418 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421553005 0 0) 0x7f78d00af420 con 0x7f78d0070250 2024-03-24T00:39:27.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cea00640 1 -- 172.21.15.40:0/3619022418 >> v1:172.21.15.53:6789/0 conn(0x7f78d0070250 legacy=0x7f78d015fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cea00640 1 -- 172.21.15.40:0/3619022418 >> v1:172.21.15.40:6789/0 conn(0x7f78d00d05b0 legacy=0x7f78d0160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cea00640 1 -- 172.21.15.40:0/3619022418 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d00d2a20 con 0x7f78d00d01d0 2024-03-24T00:39:27.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cea00640 1 --2- 172.21.15.40:0/3619022418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78d00d01d0 0x7f78d0070b20 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f78c00097c0 tx=0x7f78c001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e7221c69447adf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cd600640 1 -- 172.21.15.40:0/3619022418 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c00b1070 con 0x7f78d00d01d0 2024-03-24T00:39:27.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cd600640 1 -- 172.21.15.40:0/3619022418 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d0070080 con 0x7f78d00d01d0 2024-03-24T00:39:27.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.320+0000 7f78cd600640 1 -- 172.21.15.40:0/3619022418 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c00ac630 con 0x7f78d00d01d0 2024-03-24T00:39:27.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 -- 172.21.15.40:0/3619022418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78d00d01d0 msgr2=0x7f78d0070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 --2- 172.21.15.40:0/3619022418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78d00d01d0 0x7f78d0070b20 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f78c00097c0 tx=0x7f78c001c4a0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 -- 172.21.15.40:0/3619022418 shutdown_connections 2024-03-24T00:39:27.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 --2- 172.21.15.40:0/3619022418 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78d00d01d0 0x7f78d0070b20 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 -- 172.21.15.40:0/3619022418 >> 172.21.15.40:0/3619022418 conn(0x7f78d00d1cb0 msgr2=0x7f78d00d20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 -- 172.21.15.40:0/3619022418 shutdown_connections 2024-03-24T00:39:27.324 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 -- 172.21.15.40:0/3619022418 wait complete. 2024-03-24T00:39:27.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 Processor -- start 2024-03-24T00:39:27.325 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 -- start start 2024-03-24T00:39:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78d00d01d0 0x7f78d0160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78d00d05b0 0x7f78d00cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78d015ec20 0x7f78d014cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0169e60 con 0x7f78d015ec20 2024-03-24T00:39:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0169ce0 con 0x7f78d00d01d0 2024-03-24T00:39:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78d5600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0169fe0 con 0x7f78d00d05b0 2024-03-24T00:39:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78cf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78d015ec20 0x7f78d014cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78cea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78d00d01d0 0x7f78d0160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78cf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78d015ec20 0x7f78d014cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48736/0 (socket says 172.21.15.40:48736) 2024-03-24T00:39:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78cf400640 1 -- 172.21.15.40:0/1396522118 learned_addr learned my addr 172.21.15.40:0/1396522118 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:27.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78ce000640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78d00d05b0 0x7f78d00cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78cf400640 1 -- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78d00d05b0 msgr2=0x7f78d00cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78cf400640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78d00d05b0 0x7f78d00cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78cf400640 1 -- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78d00d01d0 msgr2=0x7f78d0160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78cf400640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78d00d01d0 0x7f78d0160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.324+0000 7f78cf400640 1 -- 172.21.15.40:0/1396522118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d0160950 con 0x7f78d015ec20 2024-03-24T00:39:27.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.328+0000 7f78cf400640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78d015ec20 0x7f78d014cd60 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f78c800caa0 tx=0x7f78c800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.328+0000 7f78c6a00640 1 -- 172.21.15.40:0/1396522118 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8017070 con 0x7f78d015ec20 2024-03-24T00:39:27.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.328+0000 7f78c6a00640 1 -- 172.21.15.40:0/1396522118 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c800ac00 con 0x7f78d015ec20 2024-03-24T00:39:27.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.328+0000 7f78c6a00640 1 -- 172.21.15.40:0/1396522118 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8012400 con 0x7f78d015ec20 2024-03-24T00:39:27.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.328+0000 7f78d5600640 1 -- 172.21.15.40:0/1396522118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d015c4d0 con 0x7f78d015ec20 2024-03-24T00:39:27.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.328+0000 7f78d5600640 1 -- 172.21.15.40:0/1396522118 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d015e420 con 0x7f78d015ec20 2024-03-24T00:39:27.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.328+0000 7f78c6a00640 1 -- 172.21.15.40:0/1396522118 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f78c801a070 con 0x7f78d015ec20 2024-03-24T00:39:27.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.328+0000 7f78c6a00640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f78b0067660 0x7f78b0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.328+0000 7f78c6a00640 1 -- 172.21.15.40:0/1396522118 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78c8085580 con 0x7f78d015ec20 2024-03-24T00:39:27.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.328+0000 7f78cea00640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f78b0067660 0x7f78b0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 Processor -- start 2024-03-24T00:39:27.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 -- start start 2024-03-24T00:39:27.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78cea00640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f78b0067660 0x7f78b0069b20 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f78c00ae000 tx=0x7f78c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bdec920 0x56431bdecd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bded240 0x56431bda2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bda2ff0 0x56431bda33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431bbf0680 con 0x56431bdec920 2024-03-24T00:39:27.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431bb77b00 con 0x56431bda2ff0 2024-03-24T00:39:27.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x56431bbec720 con 0x56431bded240 2024-03-24T00:39:27.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78cea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bdec920 0x56431bdecd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78cea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bdec920 0x56431bdecd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48748/0 (socket says 172.21.15.40:48748) 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78cea00640 1 -- 172.21.15.40:0/4142443374 learned_addr learned my addr 172.21.15.40:0/4142443374 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78cf400640 1 --2- 172.21.15.40:0/4142443374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bda2ff0 0x56431bda33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78ce000640 1 --2- 172.21.15.40:0/4142443374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bded240 0x56431bda2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78cea00640 1 -- 172.21.15.40:0/4142443374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bded240 msgr2=0x56431bda2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78cea00640 1 --2- 172.21.15.40:0/4142443374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bded240 0x56431bda2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78cea00640 1 -- 172.21.15.40:0/4142443374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bda2ff0 msgr2=0x56431bda33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78cea00640 1 --2- 172.21.15.40:0/4142443374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bda2ff0 0x56431bda33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78cea00640 1 -- 172.21.15.40:0/4142443374 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56431bbc8940 con 0x56431bdec920 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78cf400640 1 --2- 172.21.15.40:0/4142443374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bda2ff0 0x56431bda33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78cea00640 1 --2- 172.21.15.40:0/4142443374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bdec920 0x56431bdecd00 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f78c00097c0 tx=0x7f78c0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9daddf4a8d8859f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78bb400640 1 -- 172.21.15.40:0/4142443374 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c00c6020 con 0x56431bdec920 2024-03-24T00:39:27.334 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78bb400640 1 -- 172.21.15.40:0/4142443374 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c0004020 con 0x56431bdec920 2024-03-24T00:39:27.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78bb400640 1 -- 172.21.15.40:0/4142443374 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c00b1040 con 0x56431bdec920 2024-03-24T00:39:27.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 -- 172.21.15.40:0/4142443374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bdec920 msgr2=0x56431bdecd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/4142443374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bdec920 0x56431bdecd00 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f78c00097c0 tx=0x7f78c0004740 comp rx=0 tx=0).stop 2024-03-24T00:39:27.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 -- 172.21.15.40:0/4142443374 shutdown_connections 2024-03-24T00:39:27.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/4142443374 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bda2ff0 0x56431bda33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/4142443374 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bded240 0x56431bda2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/4142443374 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bdec920 0x56431bdecd00 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 -- 172.21.15.40:0/4142443374 >> 172.21.15.40:0/4142443374 conn(0x56431bd9efa0 msgr2=0x56431bda1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 -- 172.21.15.40:0/4142443374 shutdown_connections 2024-03-24T00:39:27.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 -- 172.21.15.40:0/4142443374 wait complete. 2024-03-24T00:39:27.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 -- start start 2024-03-24T00:39:27.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.332+0000 7f78d8b6d000 1 Processor -- start 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bda2ff0 0x56431bdd1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431bdd2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 0x56431bdd27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431bbef990 con 0x56431bded240 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431bbaaae0 con 0x56431bdec920 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x56431bb88750 con 0x56431bda2ff0 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 10 client.0 _ll_register_callbacks cb 0x56431bd8dff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78cf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 0x56431bdd27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78cea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bda2ff0 0x56431bdd1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78cf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 0x56431bdd27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48764/0 (socket says 172.21.15.40:48764) 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78cea00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bda2ff0 0x56431bdd1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:39398/0 (socket says 172.21.15.40:39398) 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78cf400640 1 -- 172.21.15.40:0/2044126753 learned_addr learned my addr 172.21.15.40:0/2044126753 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78ce000640 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431bdd2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78cea00640 1 -- 172.21.15.40:0/2044126753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 msgr2=0x56431bdd2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78cea00640 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431bdd2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78cea00640 1 -- 172.21.15.40:0/2044126753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 msgr2=0x56431bdd27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78cea00640 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 0x56431bdd27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78cea00640 1 -- 172.21.15.40:0/2044126753 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56431bc35d80 con 0x56431bda2ff0 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78cf400640 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 0x56431bdd27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78cea00640 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bda2ff0 0x56431bdd1d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f78c001a0d0 tx=0x7f78c001ace0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53fe3cf560b09c38 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78afe00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3301/0 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78afe00640 1 -- 172.21.15.40:0/2044126753 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c00c6070 con 0x56431bda2ff0 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78afe00640 1 -- 172.21.15.40:0/2044126753 <== mon.2 v2:172.21.15.53:3301/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78c00036e0 con 0x56431bda2ff0 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2044126753 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78c0004020 con 0x56431bda2ff0 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78afe00640 1 -- 172.21.15.40:0/2044126753 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78c00d7050 con 0x56431bda2ff0 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 10 client.7673 fetch_fsmap learned FSMap version 134 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2044126753 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,fsmap=134,monmap=2+}) v3 -- 0x56431bb77d10 con 0x56431bda2ff0 2024-03-24T00:39:27.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78afe00640 1 -- 172.21.15.40:0/2044126753 <== mon.2 v2:172.21.15.53:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c0004020 con 0x56431bda2ff0 2024-03-24T00:39:27.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78afe00640 1 -- 172.21.15.40:0/2044126753 <== mon.2 v2:172.21.15.53:3301/0 5 ==== fsmap(e 134) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f78c00da020 con 0x56431bda2ff0 2024-03-24T00:39:27.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 10 client.7673 fetch_fsmap finished waiting for FSMap version 134 2024-03-24T00:39:27.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 10 client.7673 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:39:27.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 20 client.7673 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:27.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x56431bdd3090 0x56431bdfeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 4 client.7673 mds_command: new command op to 8877 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:39:27.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2044126753 --> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56431bb8bb60 con 0x56431bdd3090 2024-03-24T00:39:27.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78ce000640 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x56431bdd3090 0x56431bdfeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78ce000640 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x56431bdd3090 0x56431bdfeea0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=116e0bc259d44141 server_cookie=9f4f211f8ad56235 in_seq=0 out_seq=0 2024-03-24T00:39:27.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.336+0000 7f78afe00640 10 client.7673 ms_handle_connect on v2:172.21.15.40:6834/773071485 2024-03-24T00:39:27.343 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78afe00640 1 -- 172.21.15.40:0/2044126753 <== mds.0 v2:172.21.15.40:6834/773071485 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56431bb8bb60 con 0x56431bdd3090 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78afe00640 10 client.7673 handle_command_reply: tid=0 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78d8b6d000 1 client.7673 shutdown 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2044126753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bda2ff0 msgr2=0x56431bdd1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bda2ff0 0x56431bdd1d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f78c001a0d0 tx=0x7f78c001ace0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2044126753 shutdown_connections 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x56431bdd3090 0x56431bdfeea0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 0x56431bdd27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431bdd2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/2044126753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bda2ff0 0x56431bdd1d40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2044126753 >> 172.21.15.40:0/2044126753 conn(0x56431bd9efa0 msgr2=0x56431bda1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78afe00640 0 client.7673 ms_handle_reset on v2:172.21.15.40:6834/773071485 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2044126753 shutdown_connections 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2044126753 wait complete. 2024-03-24T00:39:27.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.340+0000 7f78d8b6d000 20 client.7673 trim_cache size 0 max 16384 2024-03-24T00:39:27.356 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78d8b6d000 1 Processor -- start 2024-03-24T00:39:27.357 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78d8b6d000 1 -- start start 2024-03-24T00:39:27.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78d8b6d000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bda2ff0 0x56431bdcfce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78d8b6d000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431bdeb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78d8b6d000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 0x56431bdeb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78d8b6d000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00cafe0 con 0x56431bded240 2024-03-24T00:39:27.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78d8b6d000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d0202a20 con 0x56431bdec920 2024-03-24T00:39:27.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78d8b6d000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00ca180 con 0x56431bda2ff0 2024-03-24T00:39:27.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78cf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 0x56431bdeb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78cf400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 0x56431bdeb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48778/0 (socket says 172.21.15.40:48778) 2024-03-24T00:39:27.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78cf400640 1 -- 172.21.15.40:0/637462184 learned_addr learned my addr 172.21.15.40:0/637462184 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:27.358 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78cea00640 1 --2- 172.21.15.40:0/637462184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bda2ff0 0x56431bdcfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78ce000640 1 --2- 172.21.15.40:0/637462184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431bdeb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78cf400640 1 -- 172.21.15.40:0/637462184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bda2ff0 msgr2=0x56431bdcfce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78cf400640 1 --2- 172.21.15.40:0/637462184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bda2ff0 0x56431bdcfce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78cf400640 1 -- 172.21.15.40:0/637462184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 msgr2=0x56431bdeb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78cf400640 1 --2- 172.21.15.40:0/637462184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431bdeb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.359 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78cf400640 1 -- 172.21.15.40:0/637462184 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56431bbbd540 con 0x56431bded240 2024-03-24T00:39:27.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78cf400640 1 --2- 172.21.15.40:0/637462184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 0x56431bdeb690 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f78c8002240 tx=0x7f78c8051f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf998f9df6e42f65 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78baa00640 1 -- 172.21.15.40:0/637462184 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c805dd70 con 0x56431bded240 2024-03-24T00:39:27.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78baa00640 1 -- 172.21.15.40:0/637462184 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c805df10 con 0x56431bded240 2024-03-24T00:39:27.360 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.356+0000 7f78baa00640 1 -- 172.21.15.40:0/637462184 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c8004d70 con 0x56431bded240 2024-03-24T00:39:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 -- 172.21.15.40:0/637462184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 msgr2=0x56431bdeb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/637462184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 0x56431bdeb690 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f78c8002240 tx=0x7f78c8051f40 comp rx=0 tx=0).stop 2024-03-24T00:39:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 -- 172.21.15.40:0/637462184 shutdown_connections 2024-03-24T00:39:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/637462184 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bded240 0x56431bdeb690 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/637462184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431bdeb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/637462184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bda2ff0 0x56431bdcfce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 -- 172.21.15.40:0/637462184 >> 172.21.15.40:0/637462184 conn(0x56431bd9efa0 msgr2=0x56431bdcf5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 -- 172.21.15.40:0/637462184 shutdown_connections 2024-03-24T00:39:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 -- 172.21.15.40:0/637462184 wait complete. 2024-03-24T00:39:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 -- start start 2024-03-24T00:39:27.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 Processor -- start 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bda2ff0 0x56431be519b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431be51ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bded240 0x56431bdfeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431bcad700 con 0x56431bda2ff0 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x56431bb98a80 con 0x56431bdec920 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x56431bbef990 con 0x56431bded240 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 10 client.0 _ll_register_callbacks cb 0x7f78bc007c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78cea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bda2ff0 0x56431be519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78cea00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bda2ff0 0x56431be519b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48792/0 (socket says 172.21.15.40:48792) 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78cea00640 1 -- 172.21.15.40:0/2470557424 learned_addr learned my addr 172.21.15.40:0/2470557424 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78ce000640 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431be51ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78cf400640 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bded240 0x56431bdfeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78cea00640 1 -- 172.21.15.40:0/2470557424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bded240 msgr2=0x56431bdfeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78cea00640 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bded240 0x56431bdfeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78cea00640 1 -- 172.21.15.40:0/2470557424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 msgr2=0x56431be51ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78cea00640 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431be51ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78cea00640 1 -- 172.21.15.40:0/2470557424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56431bb844b0 con 0x56431bda2ff0 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78ce000640 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431be51ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78cea00640 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bda2ff0 0x56431be519b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f78c0009580 tx=0x7f78c00a6040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90224e1ab7e4828c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78afe00640 10 client.0 ms_handle_connect on v2:172.21.15.40:3300/0 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78afe00640 1 -- 172.21.15.40:0/2470557424 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c00e0060 con 0x56431bda2ff0 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78afe00640 1 -- 172.21.15.40:0/2470557424 <== mon.0 v2:172.21.15.40:3300/0 2 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78c00b75c0 con 0x56431bda2ff0 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.360+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2470557424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78c805df10 con 0x56431bda2ff0 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78afe00640 1 -- 172.21.15.40:0/2470557424 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_get_version_reply(handle=1 version=134) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78c00e6050 con 0x56431bda2ff0 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 10 client.9195 fetch_fsmap learned FSMap version 134 2024-03-24T00:39:27.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2470557424 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,fsmap=134,monmap=2+}) v3 -- 0x56431bb972f0 con 0x56431bda2ff0 2024-03-24T00:39:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78afe00640 1 -- 172.21.15.40:0/2470557424 <== mon.0 v2:172.21.15.40:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c805df10 con 0x56431bda2ff0 2024-03-24T00:39:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78afe00640 1 -- 172.21.15.40:0/2470557424 <== mon.0 v2:172.21.15.40:3300/0 5 ==== fsmap(e 134) v1 ==== 3354+0+0 (secure 0 0 0) 0x7f78c00e9020 con 0x56431bda2ff0 2024-03-24T00:39:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 10 client.9195 fetch_fsmap finished waiting for FSMap version 134 2024-03-24T00:39:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 10 client.9195 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-24T00:39:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 20 client.9195 populate_metadata read hostname 'smithi040' 2024-03-24T00:39:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x56431bdff860 0x56431bdffc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 4 client.9195 mds_command: new command op to 8877 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:39:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2470557424 --> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56431bbfdbc0 con 0x56431bdff860 2024-03-24T00:39:27.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78ce000640 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x56431bdff860 0x56431bdffc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78afe00640 10 client.9195 ms_handle_connect on v2:172.21.15.40:6834/773071485 2024-03-24T00:39:27.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78ce000640 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x56431bdff860 0x56431bdffc40 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5754f2d9b6d52132 server_cookie=ed17642d810a02fc in_seq=0 out_seq=0 2024-03-24T00:39:27.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78afe00640 1 -- 172.21.15.40:0/2470557424 <== mds.0 v2:172.21.15.40:6834/773071485 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x56431bbfdbc0 con 0x56431bdff860 2024-03-24T00:39:27.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78afe00640 10 client.9195 handle_command_reply: tid=0 2024-03-24T00:39:27.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 client.9195 shutdown 2024-03-24T00:39:27.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2470557424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bda2ff0 msgr2=0x56431be519b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bda2ff0 0x56431be519b0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f78c0009580 tx=0x7f78c00a6040 comp rx=0 tx=0).stop 2024-03-24T00:39:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2470557424 shutdown_connections 2024-03-24T00:39:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.40:6834/773071485,v1:172.21.15.40:6835/773071485] conn(0x56431bdff860 0x56431bdffc40 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56431bded240 0x56431bdfeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56431bdec920 0x56431be51ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 --2- 172.21.15.40:0/2470557424 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x56431bda2ff0 0x56431be519b0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2470557424 >> 172.21.15.40:0/2470557424 conn(0x56431bd9efa0 msgr2=0x56431bdd1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78afe00640 0 client.9195 ms_handle_reset on v2:172.21.15.40:6834/773071485 2024-03-24T00:39:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2470557424 shutdown_connections 2024-03-24T00:39:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 1 -- 172.21.15.40:0/2470557424 wait complete. 2024-03-24T00:39:27.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d8b6d000 20 client.9195 trim_cache size 0 max 16384 2024-03-24T00:39:27.368 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-03-24T00:39:27.368 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_fsid": "185b869e-2a51-452f-88b9-d083720a50b9", 2024-03-24T00:39:27.369 INFO:teuthology.orchestra.run.smithi040.stdout: "whoami": 0, 2024-03-24T00:39:27.369 INFO:teuthology.orchestra.run.smithi040.stdout: "id": 8877, 2024-03-24T00:39:27.369 INFO:teuthology.orchestra.run.smithi040.stdout: "want_state": "up:active", 2024-03-24T00:39:27.369 INFO:teuthology.orchestra.run.smithi040.stdout: "state": "up:active", 2024-03-24T00:39:27.369 INFO:teuthology.orchestra.run.smithi040.stdout: "fs_name": "cephfs", 2024-03-24T00:39:27.369 INFO:teuthology.orchestra.run.smithi040.stdout: "rank_uptime": 19.222797491000001, 2024-03-24T00:39:27.369 INFO:teuthology.orchestra.run.smithi040.stdout: "mdsmap_epoch": 134, 2024-03-24T00:39:27.369 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch": 119, 2024-03-24T00:39:27.369 INFO:teuthology.orchestra.run.smithi040.stdout: "osdmap_epoch_barrier": 119, 2024-03-24T00:39:27.369 INFO:teuthology.orchestra.run.smithi040.stdout: "uptime": 24.084825868999999 2024-03-24T00:39:27.369 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-03-24T00:39:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d5600640 1 -- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f78b0067660 msgr2=0x7f78b0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d5600640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f78b0067660 0x7f78b0069b20 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f78c00ae000 tx=0x7f78c0008040 comp rx=0 tx=0).stop 2024-03-24T00:39:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d5600640 1 -- 172.21.15.40:0/1396522118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78d015ec20 msgr2=0x7f78d014cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d5600640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78d015ec20 0x7f78d014cd60 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f78c800caa0 tx=0x7f78c800cf60 comp rx=0 tx=0).stop 2024-03-24T00:39:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d5600640 1 -- 172.21.15.40:0/1396522118 shutdown_connections 2024-03-24T00:39:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d5600640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f78b0067660 0x7f78b0069b20 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d5600640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f78d015ec20 0x7f78d014cd60 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d5600640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78d00d05b0 0x7f78d00cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d5600640 1 --2- 172.21.15.40:0/1396522118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78d00d01d0 0x7f78d0160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d5600640 1 -- 172.21.15.40:0/1396522118 >> 172.21.15.40:0/1396522118 conn(0x7f78d00d1cb0 msgr2=0x7f78d0166540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d5600640 1 -- 172.21.15.40:0/1396522118 shutdown_connections 2024-03-24T00:39:27.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.364+0000 7f78d5600640 1 -- 172.21.15.40:0/1396522118 wait complete. 2024-03-24T00:39:27.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_export_ephemeral_distributed_factor 42.333333333333336 2024-03-24T00:39:27.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a29000640 1 Processor -- start 2024-03-24T00:39:27.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a29000640 1 -- start start 2024-03-24T00:39:27.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a29000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415bd00 0x7f2a240ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a29000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a240728d0 0x7f2a240cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a29000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2a2415f8c0 0x7f2a2415fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a29000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a240cf790 con 0x7f2a240728d0 2024-03-24T00:39:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a29000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a24160250 con 0x7f2a2415f8c0 2024-03-24T00:39:27.530 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a29000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a24072d30 con 0x7f2a2415bd00 2024-03-24T00:39:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a1be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2a240728d0 0x7f2a240cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a21e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415bd00 0x7f2a240ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a22800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2a2415f8c0 0x7f2a2415fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a21e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415bd00 0x7f2a240ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:39430/0 (socket says 172.21.15.40:39430) 2024-03-24T00:39:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a21e00640 1 -- 172.21.15.40:0/2713864535 learned_addr learned my addr 172.21.15.40:0/2713864535 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a1be00640 1 -- 172.21.15.40:0/2713864535 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415bd00 msgr2=0x7f2a240ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a1be00640 1 --2- 172.21.15.40:0/2713864535 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415bd00 0x7f2a240ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a1be00640 1 -- 172.21.15.40:0/2713864535 >> v2:172.21.15.53:3300/0 conn(0x7f2a2415f8c0 msgr2=0x7f2a2415fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:39:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a1be00640 1 --2- 172.21.15.40:0/2713864535 >> v2:172.21.15.53:3300/0 conn(0x7f2a2415f8c0 0x7f2a2415fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.531 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.528+0000 7f2a1be00640 1 -- 172.21.15.40:0/2713864535 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a24071de0 con 0x7f2a240728d0 2024-03-24T00:39:27.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a1be00640 1 --2- 172.21.15.40:0/2713864535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a240728d0 0x7f2a240cf250 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f2a24073320 tx=0x7f2a0c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1261b0ad4da735d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a21400640 1 -- 172.21.15.40:0/2713864535 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c0ab610 con 0x7f2a240728d0 2024-03-24T00:39:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a21400640 1 -- 172.21.15.40:0/2713864535 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0c0b0070 con 0x7f2a240728d0 2024-03-24T00:39:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a21400640 1 -- 172.21.15.40:0/2713864535 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c0ab9d0 con 0x7f2a240728d0 2024-03-24T00:39:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a29000640 1 -- 172.21.15.40:0/2713864535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a240728d0 msgr2=0x7f2a240cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a29000640 1 --2- 172.21.15.40:0/2713864535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a240728d0 0x7f2a240cf250 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f2a24073320 tx=0x7f2a0c01c450 comp rx=0 tx=0).stop 2024-03-24T00:39:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a29000640 1 -- 172.21.15.40:0/2713864535 shutdown_connections 2024-03-24T00:39:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a29000640 1 --2- 172.21.15.40:0/2713864535 >> v2:172.21.15.53:3300/0 conn(0x7f2a2415f8c0 0x7f2a2415fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a29000640 1 --2- 172.21.15.40:0/2713864535 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a240728d0 0x7f2a240cf250 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a29000640 1 --2- 172.21.15.40:0/2713864535 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415bd00 0x7f2a240ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a29000640 1 -- 172.21.15.40:0/2713864535 >> 172.21.15.40:0/2713864535 conn(0x7f2a240b1660 msgr2=0x7f2a240b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a29000640 1 -- 172.21.15.40:0/2713864535 shutdown_connections 2024-03-24T00:39:27.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.532+0000 7f2a29000640 1 -- 172.21.15.40:0/2713864535 wait complete. 2024-03-24T00:39:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a29000640 1 Processor -- start 2024-03-24T00:39:27.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a29000640 1 -- start start 2024-03-24T00:39:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a29000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a240728d0 0x7f2a240b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a29000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a2415bd00 0x7f2a2414f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a29000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415f8c0 0x7f2a240b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a29000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2406d320 con 0x7f2a240728d0 2024-03-24T00:39:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a29000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2415c0e0 con 0x7f2a2415bd00 2024-03-24T00:39:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a29000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2406d4a0 con 0x7f2a2415f8c0 2024-03-24T00:39:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a21e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a240728d0 0x7f2a240b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a21e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a240728d0 0x7f2a240b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:48822/0 (socket says 172.21.15.40:48822) 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a21e00640 1 -- 172.21.15.40:0/3640282969 learned_addr learned my addr 172.21.15.40:0/3640282969 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a1be00640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a2415bd00 0x7f2a2414f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a22800640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415f8c0 0x7f2a240b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a21e00640 1 -- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415f8c0 msgr2=0x7f2a240b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a21e00640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415f8c0 0x7f2a240b2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a21e00640 1 -- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a2415bd00 msgr2=0x7f2a2414f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a21e00640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a2415bd00 0x7f2a2414f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a21e00640 1 -- 172.21.15.40:0/3640282969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a240b4a20 con 0x7f2a240728d0 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a21e00640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a240728d0 0x7f2a240b44e0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400e890 tx=0x7f2a1400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a1a000640 1 -- 172.21.15.40:0/3640282969 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a14017070 con 0x7f2a240728d0 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a1a000640 1 -- 172.21.15.40:0/3640282969 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1400c940 con 0x7f2a240728d0 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a1a000640 1 -- 172.21.15.40:0/3640282969 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1400cd30 con 0x7f2a240728d0 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a29000640 1 -- 172.21.15.40:0/3640282969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a240b3440 con 0x7f2a240728d0 2024-03-24T00:39:27.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.536+0000 7f2a29000640 1 -- 172.21.15.40:0/3640282969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a240b3a00 con 0x7f2a240728d0 2024-03-24T00:39:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.540+0000 7f2a29000640 1 -- 172.21.15.40:0/3640282969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a24068170 con 0x7f2a240728d0 2024-03-24T00:39:27.540 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.540+0000 7f2a1a000640 1 -- 172.21.15.40:0/3640282969 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f2a1401a070 con 0x7f2a240728d0 2024-03-24T00:39:27.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.540+0000 7f2a1a000640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a00067660 0x7f2a00069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:39:27.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.540+0000 7f2a1a000640 1 -- 172.21.15.40:0/3640282969 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a14085f40 con 0x7f2a240728d0 2024-03-24T00:39:27.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.544+0000 7f2a1be00640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a00067660 0x7f2a00069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:39:27.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.544+0000 7f2a1be00640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a00067660 0x7f2a00069b20 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c0af000 tx=0x7f2a0c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:39:27.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.544+0000 7f2a1a000640 1 -- 172.21.15.40:0/3640282969 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f2a140565e0 con 0x7f2a240728d0 2024-03-24T00:39:27.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.708+0000 7f2a29000640 1 -- 172.21.15.40:0/3640282969 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command([{prefix=config set, name=mds_export_ephemeral_distributed_factor}] v 0) v1 -- 0x7f2a2406e870 con 0x7f2a240728d0 2024-03-24T00:39:27.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a1a000640 1 -- 172.21.15.40:0/3640282969 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_export_ephemeral_distributed_factor}]=0 v71)=0 v71) v1 ==== 154+0+0 (secure 0 0 0) 0x7f2a1405a680 con 0x7f2a240728d0 2024-03-24T00:39:27.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a29000640 1 -- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a00067660 msgr2=0x7f2a00069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a29000640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a00067660 0x7f2a00069b20 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c0af000 tx=0x7f2a0c0a7000 comp rx=0 tx=0).stop 2024-03-24T00:39:27.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a29000640 1 -- 172.21.15.40:0/3640282969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a240728d0 msgr2=0x7f2a240b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:39:27.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a29000640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a240728d0 0x7f2a240b44e0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f2a1400e890 tx=0x7f2a1400ed50 comp rx=0 tx=0).stop 2024-03-24T00:39:27.714 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a29000640 1 -- 172.21.15.40:0/3640282969 shutdown_connections 2024-03-24T00:39:27.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a29000640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f2a00067660 0x7f2a00069b20 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a29000640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2a2415f8c0 0x7f2a240b2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a29000640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a2415bd00 0x7f2a2414f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a29000640 1 --2- 172.21.15.40:0/3640282969 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f2a240728d0 0x7f2a240b44e0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:39:27.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a29000640 1 -- 172.21.15.40:0/3640282969 >> 172.21.15.40:0/3640282969 conn(0x7f2a240b1660 msgr2=0x7f2a2415a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:39:27.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a29000640 1 -- 172.21.15.40:0/3640282969 shutdown_connections 2024-03-24T00:39:27.715 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:39:27.712+0000 7f2a29000640 1 -- 172.21.15.40:0/3640282969 wait complete. 2024-03-24T00:39:27.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:39:27.733 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-03-24T00:39:27.733 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:39:27.733 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p tree 2024-03-24T00:39:27.733 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:39:27.733 DEBUG:teuthology.orchestra.run.smithi040:> setfattr -n ceph.dir.pin.distributed -v 1 tree 2024-03-24T00:39:27.733 DEBUG:teuthology.orchestra.run.smithi040:> 2024-03-24T00:39:27.733 DEBUG:teuthology.orchestra.run.smithi040:> for ((i = 0; i < 1000; i++)); do 2024-03-24T00:39:27.733 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p "tree/$i" 2024-03-24T00:39:27.733 DEBUG:teuthology.orchestra.run.smithi040:> echo file > "tree/$i/file" 2024-03-24T00:39:27.733 DEBUG:teuthology.orchestra.run.smithi040:> done 2024-03-24T00:39:27.733 DEBUG:teuthology.orchestra.run.smithi040:> ') 2024-03-24T00:39:27.745 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree 2024-03-24T00:39:27.750 INFO:teuthology.orchestra.run.smithi040.stderr:+ setfattr -n ceph.dir.pin.distributed -v 1 tree 2024-03-24T00:39:27.754 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i = 0 )) 2024-03-24T00:39:27.754 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.754 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/0 2024-03-24T00:39:27.760 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.767 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.767 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.767 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/1 2024-03-24T00:39:27.777 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.781 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.781 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.781 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/2 2024-03-24T00:39:27.788 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.791 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.791 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.791 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/3 2024-03-24T00:39:27.799 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.801 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.802 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/4 2024-03-24T00:39:27.809 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.812 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.812 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.812 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/5 2024-03-24T00:39:27.819 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.822 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.822 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.822 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/6 2024-03-24T00:39:27.829 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.832 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.832 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.832 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/7 2024-03-24T00:39:27.839 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.842 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.843 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.843 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/8 2024-03-24T00:39:27.850 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.853 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.853 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.853 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/9 2024-03-24T00:39:27.861 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.864 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.864 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.864 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/10 2024-03-24T00:39:27.871 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.874 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.874 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.874 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/11 2024-03-24T00:39:27.880 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.883 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.883 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.884 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/12 2024-03-24T00:39:27.891 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/13 2024-03-24T00:39:27.901 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.903 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.903 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.903 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/14 2024-03-24T00:39:27.910 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.912 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.913 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.913 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/15 2024-03-24T00:39:27.920 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.923 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.923 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.923 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/16 2024-03-24T00:39:27.931 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.934 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/17 2024-03-24T00:39:27.941 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.944 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/18 2024-03-24T00:39:27.951 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.954 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.954 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.954 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/19 2024-03-24T00:39:27.960 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.963 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.963 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.963 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/20 2024-03-24T00:39:27.969 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.971 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.971 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.971 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/21 2024-03-24T00:39:27.978 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.981 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.981 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.981 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/22 2024-03-24T00:39:27.987 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.989 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.990 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.990 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/23 2024-03-24T00:39:27.996 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:27.999 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/24 2024-03-24T00:39:28.006 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.008 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.009 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.009 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/25 2024-03-24T00:39:28.016 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.019 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.019 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.019 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/26 2024-03-24T00:39:28.026 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.028 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.029 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.029 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/27 2024-03-24T00:39:28.036 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.038 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.038 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.039 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/28 2024-03-24T00:39:28.046 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.048 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.049 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/29 2024-03-24T00:39:28.054 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.056 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.056 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.056 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/30 2024-03-24T00:39:28.064 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.066 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.067 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/31 2024-03-24T00:39:28.073 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.075 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.075 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.075 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/32 2024-03-24T00:39:28.082 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.084 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.085 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.085 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/33 2024-03-24T00:39:28.090 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.093 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.093 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.093 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/34 2024-03-24T00:39:28.099 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/35 2024-03-24T00:39:28.108 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.111 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.111 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.112 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/36 2024-03-24T00:39:28.119 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.121 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.121 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.121 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/37 2024-03-24T00:39:28.128 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.131 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.131 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.131 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/38 2024-03-24T00:39:28.137 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.140 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.140 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.140 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/39 2024-03-24T00:39:28.147 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.150 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/40 2024-03-24T00:39:28.157 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.159 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.159 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.160 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/41 2024-03-24T00:39:28.167 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.169 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.169 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.169 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/42 2024-03-24T00:39:28.175 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.178 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/43 2024-03-24T00:39:28.185 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.188 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.188 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.188 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/44 2024-03-24T00:39:28.195 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/45 2024-03-24T00:39:28.204 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.207 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.207 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.207 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/46 2024-03-24T00:39:28.214 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.217 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.217 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.217 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/47 2024-03-24T00:39:28.223 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/48 2024-03-24T00:39:28.234 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/49 2024-03-24T00:39:28.243 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.246 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.246 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.246 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/50 2024-03-24T00:39:28.254 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.257 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/51 2024-03-24T00:39:28.264 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.267 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/52 2024-03-24T00:39:28.273 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.275 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.276 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.276 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/53 2024-03-24T00:39:28.282 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.286 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.286 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.286 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/54 2024-03-24T00:39:28.297 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.300 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/55 2024-03-24T00:39:28.308 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.311 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.312 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.312 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/56 2024-03-24T00:39:28.317 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.320 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.320 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.320 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/57 2024-03-24T00:39:28.326 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.329 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.329 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.330 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/58 2024-03-24T00:39:28.336 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.338 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.338 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.338 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/59 2024-03-24T00:39:28.344 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.347 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.347 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.347 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/60 2024-03-24T00:39:28.354 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.356 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.357 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.357 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/61 2024-03-24T00:39:28.364 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.367 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.367 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.367 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/62 2024-03-24T00:39:28.373 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.376 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.377 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.377 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/63 2024-03-24T00:39:28.384 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.387 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.387 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.387 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/64 2024-03-24T00:39:28.393 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/65 2024-03-24T00:39:28.404 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.407 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.407 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.407 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/66 2024-03-24T00:39:28.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.416 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.416 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.416 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/67 2024-03-24T00:39:28.423 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.425 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.425 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.426 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/68 2024-03-24T00:39:28.432 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.434 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.434 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.434 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/69 2024-03-24T00:39:28.441 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.444 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.444 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.444 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/70 2024-03-24T00:39:28.451 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.453 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/71 2024-03-24T00:39:28.459 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.462 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.462 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.462 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/72 2024-03-24T00:39:28.469 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.470 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.470 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.470 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/73 2024-03-24T00:39:28.476 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.479 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/74 2024-03-24T00:39:28.486 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.488 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.488 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.488 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/75 2024-03-24T00:39:28.494 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.497 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.497 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.497 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/76 2024-03-24T00:39:28.504 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.506 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.506 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.506 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/77 2024-03-24T00:39:28.513 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.517 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.517 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.517 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/78 2024-03-24T00:39:28.525 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.527 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.528 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.528 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/79 2024-03-24T00:39:28.534 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.537 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.537 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.537 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/80 2024-03-24T00:39:28.544 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.547 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.547 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.547 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/81 2024-03-24T00:39:28.553 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.556 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.556 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.557 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/82 2024-03-24T00:39:28.563 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.565 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.565 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.565 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/83 2024-03-24T00:39:28.573 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.575 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.575 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.575 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/84 2024-03-24T00:39:28.581 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.583 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.584 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.584 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/85 2024-03-24T00:39:28.592 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.594 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.595 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.595 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/86 2024-03-24T00:39:28.601 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.605 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/87 2024-03-24T00:39:28.611 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.614 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.614 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.614 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/88 2024-03-24T00:39:28.621 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.624 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.624 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.624 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/89 2024-03-24T00:39:28.630 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.633 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.633 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.633 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/90 2024-03-24T00:39:28.640 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.642 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.642 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.642 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/91 2024-03-24T00:39:28.650 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.652 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.652 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.653 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/92 2024-03-24T00:39:28.659 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.662 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.662 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.662 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/93 2024-03-24T00:39:28.669 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.672 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/94 2024-03-24T00:39:28.679 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.681 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.681 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.681 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/95 2024-03-24T00:39:28.688 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.690 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.690 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.690 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/96 2024-03-24T00:39:28.696 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.698 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.698 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.698 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/97 2024-03-24T00:39:28.705 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.708 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.708 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.708 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/98 2024-03-24T00:39:28.715 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.718 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.718 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.718 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/99 2024-03-24T00:39:28.725 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.728 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/100 2024-03-24T00:39:28.734 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.736 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/101 2024-03-24T00:39:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.744 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.745 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/102 2024-03-24T00:39:28.751 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.753 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.753 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.753 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/103 2024-03-24T00:39:28.759 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.761 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.761 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.761 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/104 2024-03-24T00:39:28.767 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.769 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.770 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/105 2024-03-24T00:39:28.777 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.779 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.780 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.780 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/106 2024-03-24T00:39:28.787 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.790 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.790 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.790 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/107 2024-03-24T00:39:28.797 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.800 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.800 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.800 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/108 2024-03-24T00:39:28.807 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.810 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.810 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.810 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/109 2024-03-24T00:39:28.817 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.820 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.820 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.820 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/110 2024-03-24T00:39:28.826 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.829 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.829 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.829 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/111 2024-03-24T00:39:28.836 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.838 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.838 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.838 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/112 2024-03-24T00:39:28.845 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.848 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/113 2024-03-24T00:39:28.854 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/114 2024-03-24T00:39:28.864 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.866 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.866 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.866 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/115 2024-03-24T00:39:28.873 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.877 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/116 2024-03-24T00:39:28.883 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.886 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/117 2024-03-24T00:39:28.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.895 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.895 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.895 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/118 2024-03-24T00:39:28.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.905 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.905 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.906 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/119 2024-03-24T00:39:28.913 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.915 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.915 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.915 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/120 2024-03-24T00:39:28.922 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.924 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.924 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.924 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/121 2024-03-24T00:39:28.932 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.934 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.934 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.934 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/122 2024-03-24T00:39:28.942 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.945 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.945 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.945 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/123 2024-03-24T00:39:28.952 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.955 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.955 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.955 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/124 2024-03-24T00:39:28.962 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.965 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.965 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.965 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/125 2024-03-24T00:39:28.972 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.975 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.975 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.975 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/126 2024-03-24T00:39:28.982 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.985 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.985 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.985 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/127 2024-03-24T00:39:28.992 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:28.995 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:28.995 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:28.995 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/128 2024-03-24T00:39:29.002 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.005 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/129 2024-03-24T00:39:29.011 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.014 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.014 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.014 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/130 2024-03-24T00:39:29.021 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.024 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.024 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.024 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/131 2024-03-24T00:39:29.031 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.034 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.034 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.034 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/132 2024-03-24T00:39:29.040 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.044 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.044 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.044 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/133 2024-03-24T00:39:29.051 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.054 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.054 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.054 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/134 2024-03-24T00:39:29.060 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.101 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.101 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.101 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/135 2024-03-24T00:39:29.101 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.101 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/136 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/137 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/138 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/139 2024-03-24T00:39:29.106 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.109 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.109 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.109 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/140 2024-03-24T00:39:29.115 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.118 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/141 2024-03-24T00:39:29.124 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.126 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.126 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.126 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/142 2024-03-24T00:39:29.133 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.137 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/143 2024-03-24T00:39:29.144 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.147 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/144 2024-03-24T00:39:29.154 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.157 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.157 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.158 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/145 2024-03-24T00:39:29.164 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.168 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.168 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.168 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/146 2024-03-24T00:39:29.175 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.178 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/147 2024-03-24T00:39:29.185 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.188 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.188 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.188 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/148 2024-03-24T00:39:29.195 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/149 2024-03-24T00:39:29.206 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.209 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.209 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.209 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/150 2024-03-24T00:39:29.216 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.219 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.219 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.219 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/151 2024-03-24T00:39:29.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.229 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.229 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.229 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/152 2024-03-24T00:39:29.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.239 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.239 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.239 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/153 2024-03-24T00:39:29.246 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.249 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.249 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.249 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/154 2024-03-24T00:39:29.256 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.259 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.259 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.259 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/155 2024-03-24T00:39:29.266 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.269 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.269 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.269 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/156 2024-03-24T00:39:29.276 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.278 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.279 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.279 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/157 2024-03-24T00:39:29.286 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.288 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.288 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.288 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/158 2024-03-24T00:39:29.296 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.298 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.298 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.298 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/159 2024-03-24T00:39:29.305 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.307 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.307 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.308 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/160 2024-03-24T00:39:29.314 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.318 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.318 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.318 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/161 2024-03-24T00:39:29.324 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.327 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.327 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.327 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/162 2024-03-24T00:39:29.335 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.337 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.337 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.337 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/163 2024-03-24T00:39:29.344 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.346 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.346 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.346 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/164 2024-03-24T00:39:29.353 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.355 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.355 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.355 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/165 2024-03-24T00:39:29.362 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.365 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.365 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.365 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/166 2024-03-24T00:39:29.372 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.375 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/167 2024-03-24T00:39:29.382 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.384 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.384 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.384 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/168 2024-03-24T00:39:29.391 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.394 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.394 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.394 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/169 2024-03-24T00:39:29.401 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.404 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.404 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.404 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/170 2024-03-24T00:39:29.411 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.414 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/171 2024-03-24T00:39:29.422 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.425 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.425 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.425 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/172 2024-03-24T00:39:29.432 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.434 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.434 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.434 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/173 2024-03-24T00:39:29.441 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.444 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/174 2024-03-24T00:39:29.450 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.452 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.453 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.453 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/175 2024-03-24T00:39:29.459 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.461 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.462 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.462 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/176 2024-03-24T00:39:29.468 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.471 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.471 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.471 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/177 2024-03-24T00:39:29.478 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.481 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.481 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.481 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/178 2024-03-24T00:39:29.487 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.490 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.490 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.490 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/179 2024-03-24T00:39:29.496 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.499 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.499 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.499 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/180 2024-03-24T00:39:29.506 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.509 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.509 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.509 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/181 2024-03-24T00:39:29.516 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.518 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.518 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.519 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/182 2024-03-24T00:39:29.525 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.527 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.527 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.527 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/183 2024-03-24T00:39:29.534 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.536 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.536 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.536 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/184 2024-03-24T00:39:29.543 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.546 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.546 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.546 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/185 2024-03-24T00:39:29.553 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.555 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.555 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.555 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/186 2024-03-24T00:39:29.562 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.565 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.565 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.565 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/187 2024-03-24T00:39:29.572 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.575 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.575 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.575 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/188 2024-03-24T00:39:29.582 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.584 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/189 2024-03-24T00:39:29.591 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.594 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.594 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.594 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/190 2024-03-24T00:39:29.601 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.604 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/191 2024-03-24T00:39:29.612 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.615 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.615 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.615 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/192 2024-03-24T00:39:29.622 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.625 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.625 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.625 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/193 2024-03-24T00:39:29.631 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.634 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.634 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.634 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/194 2024-03-24T00:39:29.641 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.644 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.644 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.644 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/195 2024-03-24T00:39:29.652 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.654 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.654 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.654 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/196 2024-03-24T00:39:29.661 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.664 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.664 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.664 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/197 2024-03-24T00:39:29.670 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.673 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/198 2024-03-24T00:39:29.679 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.683 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.683 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.683 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/199 2024-03-24T00:39:29.689 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.692 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.692 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.692 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/200 2024-03-24T00:39:29.699 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.702 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.702 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.702 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/201 2024-03-24T00:39:29.709 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.711 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/202 2024-03-24T00:39:29.719 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.722 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.722 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.722 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/203 2024-03-24T00:39:29.728 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.731 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.731 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.731 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/204 2024-03-24T00:39:29.737 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.740 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.740 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.740 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/205 2024-03-24T00:39:29.747 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.749 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.749 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.749 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/206 2024-03-24T00:39:29.756 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.759 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.759 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.759 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/207 2024-03-24T00:39:29.765 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.768 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.768 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.768 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/208 2024-03-24T00:39:29.775 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.778 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.778 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.778 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/209 2024-03-24T00:39:29.784 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.787 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.787 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.787 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/210 2024-03-24T00:39:29.793 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.796 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.796 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.796 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/211 2024-03-24T00:39:29.803 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.806 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.806 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.806 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/212 2024-03-24T00:39:29.812 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.815 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.815 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.815 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/213 2024-03-24T00:39:29.821 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.825 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.825 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.825 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/214 2024-03-24T00:39:29.830 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.833 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.833 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.833 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/215 2024-03-24T00:39:29.840 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.843 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/216 2024-03-24T00:39:29.850 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.853 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.854 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.854 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/217 2024-03-24T00:39:29.860 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.863 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/218 2024-03-24T00:39:29.870 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.872 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/219 2024-03-24T00:39:29.879 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.882 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.882 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.882 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/220 2024-03-24T00:39:29.889 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.892 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.892 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.892 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/221 2024-03-24T00:39:29.899 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/222 2024-03-24T00:39:29.910 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.913 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.913 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.913 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/223 2024-03-24T00:39:29.921 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.924 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.924 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.924 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/224 2024-03-24T00:39:29.931 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.934 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/225 2024-03-24T00:39:29.941 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.944 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.944 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.944 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/226 2024-03-24T00:39:29.951 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.953 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.953 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.953 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/227 2024-03-24T00:39:29.961 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.964 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.964 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.964 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/228 2024-03-24T00:39:29.972 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.975 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.975 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.975 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/229 2024-03-24T00:39:29.982 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.985 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.985 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.985 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/230 2024-03-24T00:39:29.993 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:29.996 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:29.996 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:29.996 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/231 2024-03-24T00:39:30.003 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.006 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.006 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.006 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/232 2024-03-24T00:39:30.013 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.015 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.015 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.015 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/233 2024-03-24T00:39:30.022 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.024 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.024 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.024 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/234 2024-03-24T00:39:30.031 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.033 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/235 2024-03-24T00:39:30.040 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.042 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.042 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.042 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/236 2024-03-24T00:39:30.049 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.052 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/237 2024-03-24T00:39:30.058 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.060 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.060 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.061 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/238 2024-03-24T00:39:30.067 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.069 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.069 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.069 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/239 2024-03-24T00:39:30.076 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.079 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.080 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/240 2024-03-24T00:39:30.086 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.089 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.089 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.089 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/241 2024-03-24T00:39:30.096 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.099 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.099 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.100 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/242 2024-03-24T00:39:30.106 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.109 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.109 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.109 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/243 2024-03-24T00:39:30.115 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.118 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.118 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.118 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/244 2024-03-24T00:39:30.126 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.128 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.128 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.129 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/245 2024-03-24T00:39:30.136 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.138 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.138 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.138 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/246 2024-03-24T00:39:30.145 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.148 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.148 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.148 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/247 2024-03-24T00:39:30.154 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.158 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.158 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.158 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/248 2024-03-24T00:39:30.164 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.167 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.167 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.167 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/249 2024-03-24T00:39:30.174 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.177 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.177 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.177 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/250 2024-03-24T00:39:30.184 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.186 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.186 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.186 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/251 2024-03-24T00:39:30.192 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.195 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.195 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.195 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/252 2024-03-24T00:39:30.202 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.205 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.205 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.205 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/253 2024-03-24T00:39:30.212 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.215 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.215 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.215 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/254 2024-03-24T00:39:30.223 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/255 2024-03-24T00:39:30.233 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/256 2024-03-24T00:39:30.243 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.246 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.246 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.246 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/257 2024-03-24T00:39:30.253 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.256 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.256 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.256 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/258 2024-03-24T00:39:30.263 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.265 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.266 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.266 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/259 2024-03-24T00:39:30.273 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.276 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.276 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.276 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/260 2024-03-24T00:39:30.283 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.286 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.286 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.286 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/261 2024-03-24T00:39:30.292 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.295 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.295 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.295 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/262 2024-03-24T00:39:30.303 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.306 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.306 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.306 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/263 2024-03-24T00:39:30.312 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.315 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.315 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.315 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/264 2024-03-24T00:39:30.321 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.323 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.323 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.323 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/265 2024-03-24T00:39:30.329 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.332 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.332 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.332 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/266 2024-03-24T00:39:30.339 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.342 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.342 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.342 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/267 2024-03-24T00:39:30.349 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.352 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.352 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.352 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/268 2024-03-24T00:39:30.359 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.361 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.361 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.361 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/269 2024-03-24T00:39:30.368 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.370 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.371 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/270 2024-03-24T00:39:30.377 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.380 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/271 2024-03-24T00:39:30.388 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.390 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.390 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.390 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/272 2024-03-24T00:39:30.397 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.399 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.399 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.399 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/273 2024-03-24T00:39:30.407 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.410 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.410 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.410 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/274 2024-03-24T00:39:30.416 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.419 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.419 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.419 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/275 2024-03-24T00:39:30.427 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.429 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/276 2024-03-24T00:39:30.438 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.439 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.440 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/277 2024-03-24T00:39:30.446 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.448 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.449 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/278 2024-03-24T00:39:30.456 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.459 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.459 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.459 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/279 2024-03-24T00:39:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.469 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/280 2024-03-24T00:39:30.476 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.478 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.479 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.479 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/281 2024-03-24T00:39:30.486 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.488 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.488 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.488 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/282 2024-03-24T00:39:30.495 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.497 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.497 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.498 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/283 2024-03-24T00:39:30.504 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.507 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.507 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.507 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/284 2024-03-24T00:39:30.513 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.515 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.515 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.516 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/285 2024-03-24T00:39:30.522 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.525 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.525 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.525 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/286 2024-03-24T00:39:30.531 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.535 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.535 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.535 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/287 2024-03-24T00:39:30.541 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.544 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.544 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.544 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/288 2024-03-24T00:39:30.551 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.553 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.553 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.553 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/289 2024-03-24T00:39:30.559 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.562 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.562 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.562 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/290 2024-03-24T00:39:30.570 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.573 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.573 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.573 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/291 2024-03-24T00:39:30.579 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.582 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.582 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.582 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/292 2024-03-24T00:39:30.588 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.591 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/293 2024-03-24T00:39:30.597 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.601 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/294 2024-03-24T00:39:30.608 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.611 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.611 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.611 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/295 2024-03-24T00:39:30.618 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.621 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/296 2024-03-24T00:39:30.628 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.631 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/297 2024-03-24T00:39:30.638 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.641 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/298 2024-03-24T00:39:30.648 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.650 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.651 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/299 2024-03-24T00:39:30.657 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.660 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.660 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.660 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/300 2024-03-24T00:39:30.667 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.670 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.670 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.670 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/301 2024-03-24T00:39:30.677 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.680 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/302 2024-03-24T00:39:30.687 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.689 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.689 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.689 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/303 2024-03-24T00:39:30.696 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.699 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.699 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.699 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/304 2024-03-24T00:39:30.705 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.708 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.708 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.708 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/305 2024-03-24T00:39:30.715 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.718 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/306 2024-03-24T00:39:30.726 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.728 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/307 2024-03-24T00:39:30.736 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.738 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.738 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.738 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/308 2024-03-24T00:39:30.745 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.748 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.748 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.748 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/309 2024-03-24T00:39:30.755 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.757 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.757 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.757 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/310 2024-03-24T00:39:30.764 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.767 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.767 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.767 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/311 2024-03-24T00:39:30.773 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.776 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.776 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.776 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/312 2024-03-24T00:39:30.782 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.785 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.785 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.785 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/313 2024-03-24T00:39:30.792 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.794 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.794 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.794 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/314 2024-03-24T00:39:30.801 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.804 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.804 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.804 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/315 2024-03-24T00:39:30.812 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.815 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.815 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.815 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/316 2024-03-24T00:39:30.822 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.825 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.825 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.825 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/317 2024-03-24T00:39:30.830 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.833 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.833 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.833 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/318 2024-03-24T00:39:30.840 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.842 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.842 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.843 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/319 2024-03-24T00:39:30.850 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.853 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.853 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.853 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/320 2024-03-24T00:39:30.860 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.862 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.862 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.862 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/321 2024-03-24T00:39:30.869 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.871 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.872 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/322 2024-03-24T00:39:30.879 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.882 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.882 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.882 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/323 2024-03-24T00:39:30.889 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.893 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/324 2024-03-24T00:39:30.899 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.902 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/325 2024-03-24T00:39:30.909 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.912 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/326 2024-03-24T00:39:30.919 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.921 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.921 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.921 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/327 2024-03-24T00:39:30.928 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.931 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/328 2024-03-24T00:39:30.938 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.940 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.940 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.940 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/329 2024-03-24T00:39:30.946 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.949 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.949 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.949 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/330 2024-03-24T00:39:30.956 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.959 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.959 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.959 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/331 2024-03-24T00:39:30.965 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.968 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.968 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.968 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/332 2024-03-24T00:39:30.974 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.977 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.977 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.977 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/333 2024-03-24T00:39:30.984 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.986 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.986 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.986 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/334 2024-03-24T00:39:30.994 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:30.997 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:30.997 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:30.997 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/335 2024-03-24T00:39:31.004 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.008 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.008 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.008 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/336 2024-03-24T00:39:31.014 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.016 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.016 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.016 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/337 2024-03-24T00:39:31.024 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.027 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.027 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.027 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/338 2024-03-24T00:39:31.034 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.037 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/339 2024-03-24T00:39:31.044 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.046 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.046 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.046 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/340 2024-03-24T00:39:31.053 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.056 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.056 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.056 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/341 2024-03-24T00:39:31.062 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.065 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.065 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.065 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/342 2024-03-24T00:39:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.074 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.074 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.075 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/343 2024-03-24T00:39:31.082 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.085 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/344 2024-03-24T00:39:31.092 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.096 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.096 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.096 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/345 2024-03-24T00:39:31.103 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.106 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.106 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.106 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/346 2024-03-24T00:39:31.113 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.115 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.115 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.115 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/347 2024-03-24T00:39:31.122 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.124 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.125 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.125 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/348 2024-03-24T00:39:31.132 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.135 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.135 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.135 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/349 2024-03-24T00:39:31.141 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.144 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.144 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.144 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/350 2024-03-24T00:39:31.151 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.155 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.155 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.155 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/351 2024-03-24T00:39:31.162 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.165 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.165 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.165 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/352 2024-03-24T00:39:31.171 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.174 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.174 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.174 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/353 2024-03-24T00:39:31.180 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.183 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.183 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.183 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/354 2024-03-24T00:39:31.189 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.191 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.191 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.191 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/355 2024-03-24T00:39:31.197 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.200 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/356 2024-03-24T00:39:31.207 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.210 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/357 2024-03-24T00:39:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.219 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.219 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.219 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/358 2024-03-24T00:39:31.226 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.229 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/359 2024-03-24T00:39:31.236 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.238 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.239 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/360 2024-03-24T00:39:31.245 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.247 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/361 2024-03-24T00:39:31.254 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.256 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.257 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.257 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/362 2024-03-24T00:39:31.264 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.268 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.268 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.268 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/363 2024-03-24T00:39:31.274 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.277 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.277 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.277 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/364 2024-03-24T00:39:31.284 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.287 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.287 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.288 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/365 2024-03-24T00:39:31.294 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.297 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.297 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.297 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/366 2024-03-24T00:39:31.304 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.307 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.307 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.307 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/367 2024-03-24T00:39:31.313 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.317 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.317 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.318 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/368 2024-03-24T00:39:31.323 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.327 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.327 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.327 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/369 2024-03-24T00:39:31.334 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.337 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.337 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.337 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/370 2024-03-24T00:39:31.343 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.346 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.346 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.346 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/371 2024-03-24T00:39:31.353 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.356 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.356 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.356 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/372 2024-03-24T00:39:31.363 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.365 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.365 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.366 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/373 2024-03-24T00:39:31.372 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.375 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/374 2024-03-24T00:39:31.382 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.384 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.384 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.385 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/375 2024-03-24T00:39:31.391 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.394 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/376 2024-03-24T00:39:31.400 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.402 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.402 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/377 2024-03-24T00:39:31.410 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.413 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/378 2024-03-24T00:39:31.420 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.423 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.423 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.423 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/379 2024-03-24T00:39:31.430 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.432 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.432 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.432 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/380 2024-03-24T00:39:31.440 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.442 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.442 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.442 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/381 2024-03-24T00:39:31.449 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.452 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.452 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.452 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/382 2024-03-24T00:39:31.459 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.461 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.461 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.461 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/383 2024-03-24T00:39:31.469 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.471 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.471 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.471 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/384 2024-03-24T00:39:31.478 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.480 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.480 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.480 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/385 2024-03-24T00:39:31.486 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.489 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.489 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.489 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/386 2024-03-24T00:39:31.496 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.499 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.499 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.499 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/387 2024-03-24T00:39:31.506 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.509 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.509 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.510 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/388 2024-03-24T00:39:31.517 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.521 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/389 2024-03-24T00:39:31.527 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.530 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/390 2024-03-24T00:39:31.537 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.540 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.540 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.540 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/391 2024-03-24T00:39:31.546 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.549 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/392 2024-03-24T00:39:31.556 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.558 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/393 2024-03-24T00:39:31.565 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.567 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.567 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.567 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/394 2024-03-24T00:39:31.573 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.576 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.577 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/395 2024-03-24T00:39:31.583 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.586 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.586 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.586 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/396 2024-03-24T00:39:31.592 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.595 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.595 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.595 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/397 2024-03-24T00:39:31.602 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.605 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.605 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.605 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/398 2024-03-24T00:39:31.611 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.614 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.614 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.614 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/399 2024-03-24T00:39:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.623 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/400 2024-03-24T00:39:31.630 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.633 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.633 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.633 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/401 2024-03-24T00:39:31.640 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.643 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/402 2024-03-24T00:39:31.649 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.652 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/403 2024-03-24T00:39:31.659 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.663 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.663 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.663 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/404 2024-03-24T00:39:31.670 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.673 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.673 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.673 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/405 2024-03-24T00:39:31.679 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.682 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.682 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.682 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/406 2024-03-24T00:39:31.689 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.692 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/407 2024-03-24T00:39:31.699 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.702 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/408 2024-03-24T00:39:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/409 2024-03-24T00:39:31.718 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.720 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/410 2024-03-24T00:39:31.726 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.729 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.729 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.729 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/411 2024-03-24T00:39:31.735 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.738 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.738 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.738 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/412 2024-03-24T00:39:31.744 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.746 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.747 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.747 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/413 2024-03-24T00:39:31.753 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.756 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/414 2024-03-24T00:39:31.762 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.765 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.765 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.765 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/415 2024-03-24T00:39:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.774 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.774 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.774 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/416 2024-03-24T00:39:31.781 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.783 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.783 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.783 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/417 2024-03-24T00:39:31.790 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.793 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.793 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.793 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/418 2024-03-24T00:39:31.800 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.802 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.803 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.803 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/419 2024-03-24T00:39:31.810 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.813 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/420 2024-03-24T00:39:31.820 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.823 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.823 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.824 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/421 2024-03-24T00:39:31.829 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.832 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.832 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/422 2024-03-24T00:39:31.839 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.842 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.842 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.842 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/423 2024-03-24T00:39:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.851 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.851 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.851 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/424 2024-03-24T00:39:31.857 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/425 2024-03-24T00:39:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/426 2024-03-24T00:39:31.876 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.879 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.879 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.879 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/427 2024-03-24T00:39:31.885 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.888 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.888 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.888 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/428 2024-03-24T00:39:31.895 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.897 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.897 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.897 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/429 2024-03-24T00:39:31.904 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.906 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.906 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.907 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/430 2024-03-24T00:39:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.916 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.916 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.916 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/431 2024-03-24T00:39:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.926 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.926 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.926 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/432 2024-03-24T00:39:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.935 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.935 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.936 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/433 2024-03-24T00:39:31.942 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.944 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.944 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.944 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/434 2024-03-24T00:39:31.951 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.953 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.953 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.953 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/435 2024-03-24T00:39:31.960 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.963 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.963 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.963 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/436 2024-03-24T00:39:31.969 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.972 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.972 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.972 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/437 2024-03-24T00:39:31.979 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.982 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.982 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.982 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/438 2024-03-24T00:39:31.988 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:31.991 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/439 2024-03-24T00:39:31.997 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.000 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.000 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.000 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/440 2024-03-24T00:39:32.007 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.010 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/441 2024-03-24T00:39:32.018 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.020 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/442 2024-03-24T00:39:32.026 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.029 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.029 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.029 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/443 2024-03-24T00:39:32.035 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.038 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.038 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.038 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/444 2024-03-24T00:39:32.044 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.047 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.047 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.047 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/445 2024-03-24T00:39:32.053 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.057 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.057 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.057 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/446 2024-03-24T00:39:32.064 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.067 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.067 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.067 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/447 2024-03-24T00:39:32.074 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.077 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.077 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.077 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/448 2024-03-24T00:39:32.083 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.086 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.086 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.086 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/449 2024-03-24T00:39:32.092 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.095 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.096 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.096 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/450 2024-03-24T00:39:32.101 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.104 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.104 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.104 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/451 2024-03-24T00:39:32.110 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.113 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.113 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.113 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/452 2024-03-24T00:39:32.119 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.122 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.122 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.122 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/453 2024-03-24T00:39:32.128 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.131 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/454 2024-03-24T00:39:32.137 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.140 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.140 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.140 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/455 2024-03-24T00:39:32.147 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.150 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.150 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.150 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/456 2024-03-24T00:39:32.157 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.160 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.161 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.161 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/457 2024-03-24T00:39:32.168 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.171 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.171 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.172 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/458 2024-03-24T00:39:32.179 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.182 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.182 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.182 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/459 2024-03-24T00:39:32.189 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.192 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.192 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.192 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/460 2024-03-24T00:39:32.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.201 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.201 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.201 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/461 2024-03-24T00:39:32.207 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.209 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.209 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.209 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/462 2024-03-24T00:39:32.216 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.219 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/463 2024-03-24T00:39:32.225 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.228 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.228 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.228 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/464 2024-03-24T00:39:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.238 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/465 2024-03-24T00:39:32.245 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.248 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.248 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.248 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/466 2024-03-24T00:39:32.254 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.258 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/467 2024-03-24T00:39:32.265 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.267 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/468 2024-03-24T00:39:32.274 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.277 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.277 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.277 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/469 2024-03-24T00:39:32.283 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.285 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.286 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.286 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/470 2024-03-24T00:39:32.292 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.295 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.295 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.295 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/471 2024-03-24T00:39:32.301 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.304 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.304 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.304 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/472 2024-03-24T00:39:32.311 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.314 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.314 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.314 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/473 2024-03-24T00:39:32.321 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.323 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.324 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.324 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/474 2024-03-24T00:39:32.330 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.332 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.332 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.332 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/475 2024-03-24T00:39:32.339 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.342 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.342 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.342 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/476 2024-03-24T00:39:32.348 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.350 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/477 2024-03-24T00:39:32.356 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.358 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.358 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.358 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/478 2024-03-24T00:39:32.365 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.367 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.368 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.368 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/479 2024-03-24T00:39:32.374 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.377 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.378 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.378 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/480 2024-03-24T00:39:32.384 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.386 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.386 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.387 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/481 2024-03-24T00:39:32.394 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.396 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/482 2024-03-24T00:39:32.403 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.406 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.406 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.406 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/483 2024-03-24T00:39:32.413 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.415 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.415 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.415 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/484 2024-03-24T00:39:32.422 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.424 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/485 2024-03-24T00:39:32.431 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.433 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.433 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.433 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/486 2024-03-24T00:39:32.440 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.442 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.443 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.443 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/487 2024-03-24T00:39:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.451 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/488 2024-03-24T00:39:32.458 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.461 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.461 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.461 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/489 2024-03-24T00:39:32.467 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.469 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.470 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.470 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/490 2024-03-24T00:39:32.477 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.480 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.480 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.480 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/491 2024-03-24T00:39:32.487 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.490 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/492 2024-03-24T00:39:32.497 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.499 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.500 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.500 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/493 2024-03-24T00:39:32.506 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.508 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.509 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.509 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/494 2024-03-24T00:39:32.515 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.518 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.518 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.518 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/495 2024-03-24T00:39:32.524 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.527 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.527 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.527 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/496 2024-03-24T00:39:32.534 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.536 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.536 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.536 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/497 2024-03-24T00:39:32.542 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.545 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.545 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.545 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/498 2024-03-24T00:39:32.551 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.554 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.554 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.554 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/499 2024-03-24T00:39:32.561 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.563 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.563 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.564 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/500 2024-03-24T00:39:32.569 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.572 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.572 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.572 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/501 2024-03-24T00:39:32.579 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.581 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.581 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.582 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/502 2024-03-24T00:39:32.588 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.591 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.591 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.591 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/503 2024-03-24T00:39:32.597 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.599 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.599 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.599 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/504 2024-03-24T00:39:32.605 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.608 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.608 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.608 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/505 2024-03-24T00:39:32.614 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.617 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.617 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.617 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/506 2024-03-24T00:39:32.623 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.625 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.626 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.626 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/507 2024-03-24T00:39:32.632 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.634 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.634 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.634 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/508 2024-03-24T00:39:32.641 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.643 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.644 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.644 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/509 2024-03-24T00:39:32.651 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.653 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.653 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.654 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/510 2024-03-24T00:39:32.660 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.663 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.663 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.663 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/511 2024-03-24T00:39:32.669 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.672 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.672 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.672 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/512 2024-03-24T00:39:32.679 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.682 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.682 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.682 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/513 2024-03-24T00:39:32.689 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.691 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.691 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.691 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/514 2024-03-24T00:39:32.697 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.700 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.700 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.700 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/515 2024-03-24T00:39:32.706 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.709 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.709 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.709 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/516 2024-03-24T00:39:32.716 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.719 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.719 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.719 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/517 2024-03-24T00:39:32.727 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.729 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.730 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.730 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/518 2024-03-24T00:39:32.737 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/519 2024-03-24T00:39:32.747 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.749 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.749 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.749 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/520 2024-03-24T00:39:32.755 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.758 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.758 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.758 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/521 2024-03-24T00:39:32.764 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.767 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.767 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.767 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/522 2024-03-24T00:39:32.774 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.776 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.776 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.776 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/523 2024-03-24T00:39:32.784 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.786 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.787 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.787 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/524 2024-03-24T00:39:32.793 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.796 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/525 2024-03-24T00:39:32.803 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.805 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.805 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.805 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/526 2024-03-24T00:39:32.812 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.815 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.815 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.815 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/527 2024-03-24T00:39:32.822 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.825 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.825 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.826 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/528 2024-03-24T00:39:32.832 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.835 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.835 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.835 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/529 2024-03-24T00:39:32.842 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.845 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.845 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.845 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/530 2024-03-24T00:39:32.852 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.854 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.855 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.855 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/531 2024-03-24T00:39:32.860 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.863 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.863 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.863 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/532 2024-03-24T00:39:32.869 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.871 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.871 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.871 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/533 2024-03-24T00:39:32.878 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.881 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.881 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.881 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/534 2024-03-24T00:39:32.888 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.891 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.891 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.891 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/535 2024-03-24T00:39:32.897 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.899 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.900 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.900 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/536 2024-03-24T00:39:32.906 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.908 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.908 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.908 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/537 2024-03-24T00:39:32.914 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.917 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.917 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.917 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/538 2024-03-24T00:39:32.924 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.927 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.927 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.927 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/539 2024-03-24T00:39:32.934 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.937 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.937 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.937 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/540 2024-03-24T00:39:32.943 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.946 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.946 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.946 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/541 2024-03-24T00:39:32.953 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.956 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.956 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.956 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/542 2024-03-24T00:39:32.962 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.966 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.966 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.966 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/543 2024-03-24T00:39:32.972 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.974 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.975 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.975 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/544 2024-03-24T00:39:32.980 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.983 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.983 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.983 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/545 2024-03-24T00:39:32.991 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:32.993 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:32.993 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:32.994 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/546 2024-03-24T00:39:33.000 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.003 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.003 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.003 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/547 2024-03-24T00:39:33.010 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.013 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/548 2024-03-24T00:39:33.019 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.022 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/549 2024-03-24T00:39:33.029 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.031 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.031 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.032 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/550 2024-03-24T00:39:33.038 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.041 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.041 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.042 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/551 2024-03-24T00:39:33.049 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.052 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.052 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.052 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/552 2024-03-24T00:39:33.058 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.061 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.061 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.061 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/553 2024-03-24T00:39:33.068 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.071 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.071 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.071 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/554 2024-03-24T00:39:33.078 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.082 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.082 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.082 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/555 2024-03-24T00:39:33.090 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.093 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.093 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.093 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/556 2024-03-24T00:39:33.100 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.102 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/557 2024-03-24T00:39:33.110 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.112 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.112 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.112 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/558 2024-03-24T00:39:33.119 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.122 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.122 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.122 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/559 2024-03-24T00:39:33.129 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.131 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.131 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.131 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/560 2024-03-24T00:39:33.138 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.141 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.141 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.141 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/561 2024-03-24T00:39:33.148 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.151 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.151 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.151 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/562 2024-03-24T00:39:33.157 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.160 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.160 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.160 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/563 2024-03-24T00:39:33.167 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.170 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.170 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.170 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/564 2024-03-24T00:39:33.176 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.179 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.180 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.180 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/565 2024-03-24T00:39:33.187 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.189 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.189 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.189 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/566 2024-03-24T00:39:33.196 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.198 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/567 2024-03-24T00:39:33.205 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.207 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/568 2024-03-24T00:39:33.214 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.216 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.217 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/569 2024-03-24T00:39:33.224 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.227 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.227 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.227 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/570 2024-03-24T00:39:33.234 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.237 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.237 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.237 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/571 2024-03-24T00:39:33.245 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.248 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.248 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.248 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/572 2024-03-24T00:39:33.255 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.257 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.257 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.257 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/573 2024-03-24T00:39:33.264 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.267 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/574 2024-03-24T00:39:33.274 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.276 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.277 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/575 2024-03-24T00:39:33.288 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.291 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/576 2024-03-24T00:39:33.297 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.302 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.302 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.302 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/577 2024-03-24T00:39:33.311 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.315 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.316 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.316 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/578 2024-03-24T00:39:33.324 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.328 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/579 2024-03-24T00:39:33.340 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.359 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.359 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.359 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/580 2024-03-24T00:39:33.368 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.375 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/581 2024-03-24T00:39:33.386 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.390 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.390 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.391 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/582 2024-03-24T00:39:33.399 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.404 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.404 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.404 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/583 2024-03-24T00:39:33.412 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.417 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.417 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.417 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/584 2024-03-24T00:39:33.426 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.430 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.430 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.431 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/585 2024-03-24T00:39:33.439 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.443 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.443 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.443 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/586 2024-03-24T00:39:33.452 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.456 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.456 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.456 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/587 2024-03-24T00:39:33.465 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.469 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.469 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.469 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/588 2024-03-24T00:39:33.477 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.481 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.481 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.481 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/589 2024-03-24T00:39:33.489 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.493 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.493 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.493 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/590 2024-03-24T00:39:33.502 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.506 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.506 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.506 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/591 2024-03-24T00:39:33.514 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.519 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.519 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.519 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/592 2024-03-24T00:39:33.528 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.532 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.532 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.532 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/593 2024-03-24T00:39:33.542 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.546 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.547 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.547 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/594 2024-03-24T00:39:33.556 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.560 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.560 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.560 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/595 2024-03-24T00:39:33.568 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.573 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.573 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.573 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/596 2024-03-24T00:39:33.580 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.584 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.585 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.585 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/597 2024-03-24T00:39:33.592 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.596 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.596 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.596 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/598 2024-03-24T00:39:33.603 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.608 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/599 2024-03-24T00:39:33.617 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.621 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.621 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.621 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/600 2024-03-24T00:39:33.629 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.633 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.633 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.633 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/601 2024-03-24T00:39:33.640 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.645 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.645 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.645 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/602 2024-03-24T00:39:33.654 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.658 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.658 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.658 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/603 2024-03-24T00:39:33.666 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.669 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.669 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.669 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/604 2024-03-24T00:39:33.677 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.682 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.682 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.682 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/605 2024-03-24T00:39:33.691 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.695 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/606 2024-03-24T00:39:33.704 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.708 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.708 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.708 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/607 2024-03-24T00:39:33.717 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.722 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.722 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.722 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/608 2024-03-24T00:39:33.736 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:33.747 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:33.747 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:33.748 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/609 2024-03-24T00:39:38.060 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:39:38.065 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:39:38.492 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:43.335 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:43.335 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:43.336 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/610 2024-03-24T00:39:43.403 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:43.407 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/611 2024-03-24T00:39:48.720 INFO:teuthology.orchestra.run.smithi040.stderr:+ echo file 2024-03-24T00:39:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i++ )) 2024-03-24T00:39:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:+ (( i < 1000 )) 2024-03-24T00:39:48.723 INFO:teuthology.orchestra.run.smithi040.stderr:+ mkdir -p tree/612 2024-03-24T00:39:48.730 INFO:tasks.ceph.mds.b.smithi053.stderr:./src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2cd6200640 time 2024-03-24T00:39:48.734437+0000 2024-03-24T00:39:48.730 INFO:tasks.ceph.mds.b.smithi053.stderr:./src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:39:48.731 INFO:tasks.ceph.mds.b.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:39:48.731 INFO:tasks.ceph.mds.b.smithi053.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f2cdd581e7b] 2024-03-24T00:39:48.731 INFO:tasks.ceph.mds.b.smithi053.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f2cdd58202f] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5616747b64e8] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56167469e83b] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x56167469ee2c] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 6: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5616746c6793] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x5616746a6cc6] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561674692ff7] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x5616746194bb] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56167461996c] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5616745fee95] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 12: (DispatchQueue::entry()+0x5ea) [0x7f2cdd77bb0a] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 13: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f2cdd813911] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 14: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2cdcc94b43] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2cdcd26a00] 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr:2024-03-24T00:39:48.733+0000 7f2cd6200640 -1 ./src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2cd6200640 time 2024-03-24T00:39:48.734437+0000 2024-03-24T00:39:48.732 INFO:tasks.ceph.mds.b.smithi053.stderr:./src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:39:48.733 INFO:tasks.ceph.mds.b.smithi053.stderr: 2024-03-24T00:39:48.733 INFO:tasks.ceph.mds.b.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:39:48.733 INFO:tasks.ceph.mds.b.smithi053.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f2cdd581e7b] 2024-03-24T00:39:48.733 INFO:tasks.ceph.mds.b.smithi053.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f2cdd58202f] 2024-03-24T00:39:48.733 INFO:tasks.ceph.mds.b.smithi053.stderr: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5616747b64e8] 2024-03-24T00:39:48.733 INFO:tasks.ceph.mds.b.smithi053.stderr: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56167469e83b] 2024-03-24T00:39:48.733 INFO:tasks.ceph.mds.b.smithi053.stderr: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x56167469ee2c] 2024-03-24T00:39:48.733 INFO:tasks.ceph.mds.b.smithi053.stderr: 6: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5616746c6793] 2024-03-24T00:39:48.733 INFO:tasks.ceph.mds.b.smithi053.stderr: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x5616746a6cc6] 2024-03-24T00:39:48.734 INFO:tasks.ceph.mds.b.smithi053.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561674692ff7] 2024-03-24T00:39:48.734 INFO:tasks.ceph.mds.b.smithi053.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x5616746194bb] 2024-03-24T00:39:48.734 INFO:tasks.ceph.mds.b.smithi053.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56167461996c] 2024-03-24T00:39:48.734 INFO:tasks.ceph.mds.b.smithi053.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5616745fee95] 2024-03-24T00:39:48.734 INFO:tasks.ceph.mds.b.smithi053.stderr: 12: (DispatchQueue::entry()+0x5ea) [0x7f2cdd77bb0a] 2024-03-24T00:39:48.734 INFO:tasks.ceph.mds.b.smithi053.stderr: 13: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f2cdd813911] 2024-03-24T00:39:48.734 INFO:tasks.ceph.mds.b.smithi053.stderr: 14: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2cdcc94b43] 2024-03-24T00:39:48.734 INFO:tasks.ceph.mds.b.smithi053.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2cdcd26a00] 2024-03-24T00:39:48.734 INFO:tasks.ceph.mds.b.smithi053.stderr: 2024-03-24T00:39:48.734 INFO:tasks.ceph.mds.b.smithi053.stderr:*** Caught signal (Aborted) ** 2024-03-24T00:39:48.734 INFO:tasks.ceph.mds.b.smithi053.stderr: in thread 7f2cd6200640 thread_name:ms_dispatch 2024-03-24T00:39:48.735 INFO:tasks.ceph.mds.b.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:39:48.735 INFO:tasks.ceph.mds.b.smithi053.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2cdcc42520] 2024-03-24T00:39:48.735 INFO:tasks.ceph.mds.b.smithi053.stderr: 2: pthread_kill() 2024-03-24T00:39:48.735 INFO:tasks.ceph.mds.b.smithi053.stderr: 3: raise() 2024-03-24T00:39:48.735 INFO:tasks.ceph.mds.b.smithi053.stderr: 4: abort() 2024-03-24T00:39:48.735 INFO:tasks.ceph.mds.b.smithi053.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2cdd581ed5] 2024-03-24T00:39:48.735 INFO:tasks.ceph.mds.b.smithi053.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f2cdd58202f] 2024-03-24T00:39:48.735 INFO:tasks.ceph.mds.b.smithi053.stderr: 7: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5616747b64e8] 2024-03-24T00:39:48.735 INFO:tasks.ceph.mds.b.smithi053.stderr: 8: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56167469e83b] 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: 9: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x56167469ee2c] 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: 10: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5616746c6793] 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: 11: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x5616746a6cc6] 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561674692ff7] 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x5616746194bb] 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56167461996c] 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5616745fee95] 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: 16: (DispatchQueue::entry()+0x5ea) [0x7f2cdd77bb0a] 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: 17: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f2cdd813911] 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: 18: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2cdcc94b43] 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: 19: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2cdcd26a00] 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr:2024-03-24T00:39:48.733+0000 7f2cd6200640 -1 *** Caught signal (Aborted) ** 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: in thread 7f2cd6200640 thread_name:ms_dispatch 2024-03-24T00:39:48.736 INFO:tasks.ceph.mds.b.smithi053.stderr: 2024-03-24T00:39:48.737 INFO:tasks.ceph.mds.b.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:39:48.737 INFO:tasks.ceph.mds.b.smithi053.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2cdcc42520] 2024-03-24T00:39:48.737 INFO:tasks.ceph.mds.b.smithi053.stderr: 2: pthread_kill() 2024-03-24T00:39:48.737 INFO:tasks.ceph.mds.b.smithi053.stderr: 3: raise() 2024-03-24T00:39:48.737 INFO:tasks.ceph.mds.b.smithi053.stderr: 4: abort() 2024-03-24T00:39:48.737 INFO:tasks.ceph.mds.b.smithi053.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2cdd581ed5] 2024-03-24T00:39:48.737 INFO:tasks.ceph.mds.b.smithi053.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f2cdd58202f] 2024-03-24T00:39:48.737 INFO:tasks.ceph.mds.b.smithi053.stderr: 7: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5616747b64e8] 2024-03-24T00:39:48.737 INFO:tasks.ceph.mds.b.smithi053.stderr: 8: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56167469e83b] 2024-03-24T00:39:48.737 INFO:tasks.ceph.mds.b.smithi053.stderr: 9: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x56167469ee2c] 2024-03-24T00:39:48.737 INFO:tasks.ceph.mds.b.smithi053.stderr: 10: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5616746c6793] 2024-03-24T00:39:48.737 INFO:tasks.ceph.mds.b.smithi053.stderr: 11: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x5616746a6cc6] 2024-03-24T00:39:48.738 INFO:tasks.ceph.mds.b.smithi053.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561674692ff7] 2024-03-24T00:39:48.738 INFO:tasks.ceph.mds.b.smithi053.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x5616746194bb] 2024-03-24T00:39:48.738 INFO:tasks.ceph.mds.b.smithi053.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56167461996c] 2024-03-24T00:39:48.738 INFO:tasks.ceph.mds.b.smithi053.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5616745fee95] 2024-03-24T00:39:48.738 INFO:tasks.ceph.mds.b.smithi053.stderr: 16: (DispatchQueue::entry()+0x5ea) [0x7f2cdd77bb0a] 2024-03-24T00:39:48.738 INFO:tasks.ceph.mds.b.smithi053.stderr: 17: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f2cdd813911] 2024-03-24T00:39:48.738 INFO:tasks.ceph.mds.b.smithi053.stderr: 18: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2cdcc94b43] 2024-03-24T00:39:48.738 INFO:tasks.ceph.mds.b.smithi053.stderr: 19: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2cdcd26a00] 2024-03-24T00:39:48.738 INFO:tasks.ceph.mds.b.smithi053.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:39:48.738 INFO:tasks.ceph.mds.b.smithi053.stderr: 2024-03-24T00:39:48.774 INFO:tasks.ceph.mds.b.smithi053.stderr: -1> 2024-03-24T00:39:48.733+0000 7f2cd6200640 -1 ./src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2cd6200640 time 2024-03-24T00:39:48.734437+0000 2024-03-24T00:39:48.774 INFO:tasks.ceph.mds.b.smithi053.stderr:./src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f2cdd581e7b] 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f2cdd58202f] 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5616747b64e8] 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56167469e83b] 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x56167469ee2c] 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 6: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5616746c6793] 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x5616746a6cc6] 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561674692ff7] 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x5616746194bb] 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56167461996c] 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5616745fee95] 2024-03-24T00:39:48.775 INFO:tasks.ceph.mds.b.smithi053.stderr: 12: (DispatchQueue::entry()+0x5ea) [0x7f2cdd77bb0a] 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 13: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f2cdd813911] 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 14: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2cdcc94b43] 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2cdcd26a00] 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 0> 2024-03-24T00:39:48.733+0000 7f2cd6200640 -1 *** Caught signal (Aborted) ** 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: in thread 7f2cd6200640 thread_name:ms_dispatch 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2cdcc42520] 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 2: pthread_kill() 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 3: raise() 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 4: abort() 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2cdd581ed5] 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f2cdd58202f] 2024-03-24T00:39:48.776 INFO:tasks.ceph.mds.b.smithi053.stderr: 7: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5616747b64e8] 2024-03-24T00:39:48.777 INFO:tasks.ceph.mds.b.smithi053.stderr: 8: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56167469e83b] 2024-03-24T00:39:48.777 INFO:tasks.ceph.mds.b.smithi053.stderr: 9: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x56167469ee2c] 2024-03-24T00:39:48.777 INFO:tasks.ceph.mds.b.smithi053.stderr: 10: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5616746c6793] 2024-03-24T00:39:48.777 INFO:tasks.ceph.mds.b.smithi053.stderr: 11: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x5616746a6cc6] 2024-03-24T00:39:48.777 INFO:tasks.ceph.mds.b.smithi053.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561674692ff7] 2024-03-24T00:39:48.777 INFO:tasks.ceph.mds.b.smithi053.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x5616746194bb] 2024-03-24T00:39:48.777 INFO:tasks.ceph.mds.b.smithi053.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56167461996c] 2024-03-24T00:39:48.777 INFO:tasks.ceph.mds.b.smithi053.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5616745fee95] 2024-03-24T00:39:48.777 INFO:tasks.ceph.mds.b.smithi053.stderr: 16: (DispatchQueue::entry()+0x5ea) [0x7f2cdd77bb0a] 2024-03-24T00:39:48.777 INFO:tasks.ceph.mds.b.smithi053.stderr: 17: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f2cdd813911] 2024-03-24T00:39:48.777 INFO:tasks.ceph.mds.b.smithi053.stderr: 18: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2cdcc94b43] 2024-03-24T00:39:48.777 INFO:tasks.ceph.mds.b.smithi053.stderr: 19: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2cdcd26a00] 2024-03-24T00:39:48.778 INFO:tasks.ceph.mds.b.smithi053.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:39:48.778 INFO:tasks.ceph.mds.b.smithi053.stderr: 2024-03-24T00:39:48.785 INFO:tasks.ceph.mds.b.smithi053.stderr: -1> 2024-03-24T00:39:48.733+0000 7f2cd6200640 -1 ./src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2cd6200640 time 2024-03-24T00:39:48.734437+0000 2024-03-24T00:39:48.785 INFO:tasks.ceph.mds.b.smithi053.stderr:./src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:39:48.785 INFO:tasks.ceph.mds.b.smithi053.stderr: 2024-03-24T00:39:48.785 INFO:tasks.ceph.mds.b.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:39:48.785 INFO:tasks.ceph.mds.b.smithi053.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f2cdd581e7b] 2024-03-24T00:39:48.785 INFO:tasks.ceph.mds.b.smithi053.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f2cdd58202f] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5616747b64e8] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56167469e83b] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x56167469ee2c] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 6: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5616746c6793] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x5616746a6cc6] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561674692ff7] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x5616746194bb] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56167461996c] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5616745fee95] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 12: (DispatchQueue::entry()+0x5ea) [0x7f2cdd77bb0a] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 13: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f2cdd813911] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 14: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2cdcc94b43] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2cdcd26a00] 2024-03-24T00:39:48.786 INFO:tasks.ceph.mds.b.smithi053.stderr: 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: 0> 2024-03-24T00:39:48.733+0000 7f2cd6200640 -1 *** Caught signal (Aborted) ** 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: in thread 7f2cd6200640 thread_name:ms_dispatch 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f2cdcc42520] 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: 2: pthread_kill() 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: 3: raise() 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: 4: abort() 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f2cdd581ed5] 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f2cdd58202f] 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: 7: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5616747b64e8] 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: 8: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56167469e83b] 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: 9: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x56167469ee2c] 2024-03-24T00:39:48.787 INFO:tasks.ceph.mds.b.smithi053.stderr: 10: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5616746c6793] 2024-03-24T00:39:48.788 INFO:tasks.ceph.mds.b.smithi053.stderr: 11: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x5616746a6cc6] 2024-03-24T00:39:48.788 INFO:tasks.ceph.mds.b.smithi053.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561674692ff7] 2024-03-24T00:39:48.788 INFO:tasks.ceph.mds.b.smithi053.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x5616746194bb] 2024-03-24T00:39:48.788 INFO:tasks.ceph.mds.b.smithi053.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56167461996c] 2024-03-24T00:39:48.788 INFO:tasks.ceph.mds.b.smithi053.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5616745fee95] 2024-03-24T00:39:48.788 INFO:tasks.ceph.mds.b.smithi053.stderr: 16: (DispatchQueue::entry()+0x5ea) [0x7f2cdd77bb0a] 2024-03-24T00:39:48.788 INFO:tasks.ceph.mds.b.smithi053.stderr: 17: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f2cdd813911] 2024-03-24T00:39:48.788 INFO:tasks.ceph.mds.b.smithi053.stderr: 18: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f2cdcc94b43] 2024-03-24T00:39:48.788 INFO:tasks.ceph.mds.b.smithi053.stderr: 19: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f2cdcd26a00] 2024-03-24T00:39:48.788 INFO:tasks.ceph.mds.b.smithi053.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:39:48.788 INFO:tasks.ceph.mds.b.smithi053.stderr: 2024-03-24T00:39:49.308 INFO:tasks.ceph.mds.b.smithi053.stderr:daemon-helper: command crashed with signal 6 2024-03-24T00:39:54.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-24T00:40:00.843 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~7s 2024-03-24T00:40:06.947 INFO:tasks.ceph.mds.d.smithi053.stderr:./src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4858000640 time 2024-03-24T00:40:06.951164+0000 2024-03-24T00:40:06.947 INFO:tasks.ceph.mds.d.smithi053.stderr:./src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:40:06.948 INFO:tasks.ceph.mds.d.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:40:06.948 INFO:tasks.ceph.mds.d.smithi053.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f485f381e7b] 2024-03-24T00:40:06.948 INFO:tasks.ceph.mds.d.smithi053.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f485f38202f] 2024-03-24T00:40:06.948 INFO:tasks.ceph.mds.d.smithi053.stderr: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5618607ac4e8] 2024-03-24T00:40:06.948 INFO:tasks.ceph.mds.d.smithi053.stderr: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56186069483b] 2024-03-24T00:40:06.948 INFO:tasks.ceph.mds.d.smithi053.stderr: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x561860694e2c] 2024-03-24T00:40:06.948 INFO:tasks.ceph.mds.d.smithi053.stderr: 6: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5618606bc793] 2024-03-24T00:40:06.948 INFO:tasks.ceph.mds.d.smithi053.stderr: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x56186069ccc6] 2024-03-24T00:40:06.948 INFO:tasks.ceph.mds.d.smithi053.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561860688ff7] 2024-03-24T00:40:06.949 INFO:tasks.ceph.mds.d.smithi053.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x56186060f4bb] 2024-03-24T00:40:06.949 INFO:tasks.ceph.mds.d.smithi053.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56186060f96c] 2024-03-24T00:40:06.949 INFO:tasks.ceph.mds.d.smithi053.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5618605f4e95] 2024-03-24T00:40:06.949 INFO:tasks.ceph.mds.d.smithi053.stderr: 12: (DispatchQueue::entry()+0x5ea) [0x7f485f57bb0a] 2024-03-24T00:40:06.949 INFO:tasks.ceph.mds.d.smithi053.stderr: 13: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f485f613911] 2024-03-24T00:40:06.949 INFO:tasks.ceph.mds.d.smithi053.stderr: 14: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f485ea94b43] 2024-03-24T00:40:06.949 INFO:tasks.ceph.mds.d.smithi053.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f485eb26a00] 2024-03-24T00:40:06.949 INFO:tasks.ceph.mds.d.smithi053.stderr:2024-03-24T00:40:06.949+0000 7f4858000640 -1 ./src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4858000640 time 2024-03-24T00:40:06.951164+0000 2024-03-24T00:40:06.949 INFO:tasks.ceph.mds.d.smithi053.stderr:./src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:40:06.949 INFO:tasks.ceph.mds.d.smithi053.stderr: 2024-03-24T00:40:06.949 INFO:tasks.ceph.mds.d.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:40:06.949 INFO:tasks.ceph.mds.d.smithi053.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f485f381e7b] 2024-03-24T00:40:06.950 INFO:tasks.ceph.mds.d.smithi053.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f485f38202f] 2024-03-24T00:40:06.950 INFO:tasks.ceph.mds.d.smithi053.stderr: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5618607ac4e8] 2024-03-24T00:40:06.950 INFO:tasks.ceph.mds.d.smithi053.stderr: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56186069483b] 2024-03-24T00:40:06.950 INFO:tasks.ceph.mds.d.smithi053.stderr: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x561860694e2c] 2024-03-24T00:40:06.950 INFO:tasks.ceph.mds.d.smithi053.stderr: 6: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5618606bc793] 2024-03-24T00:40:06.950 INFO:tasks.ceph.mds.d.smithi053.stderr: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x56186069ccc6] 2024-03-24T00:40:06.950 INFO:tasks.ceph.mds.d.smithi053.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561860688ff7] 2024-03-24T00:40:06.950 INFO:tasks.ceph.mds.d.smithi053.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x56186060f4bb] 2024-03-24T00:40:06.950 INFO:tasks.ceph.mds.d.smithi053.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56186060f96c] 2024-03-24T00:40:06.950 INFO:tasks.ceph.mds.d.smithi053.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5618605f4e95] 2024-03-24T00:40:06.951 INFO:tasks.ceph.mds.d.smithi053.stderr: 12: (DispatchQueue::entry()+0x5ea) [0x7f485f57bb0a] 2024-03-24T00:40:06.951 INFO:tasks.ceph.mds.d.smithi053.stderr: 13: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f485f613911] 2024-03-24T00:40:06.951 INFO:tasks.ceph.mds.d.smithi053.stderr: 14: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f485ea94b43] 2024-03-24T00:40:06.951 INFO:tasks.ceph.mds.d.smithi053.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f485eb26a00] 2024-03-24T00:40:06.951 INFO:tasks.ceph.mds.d.smithi053.stderr: 2024-03-24T00:40:06.951 INFO:tasks.ceph.mds.d.smithi053.stderr:*** Caught signal (Aborted) ** 2024-03-24T00:40:06.951 INFO:tasks.ceph.mds.d.smithi053.stderr: in thread 7f4858000640 thread_name:ms_dispatch 2024-03-24T00:40:06.951 INFO:tasks.ceph.mds.d.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:40:06.952 INFO:tasks.ceph.mds.d.smithi053.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f485ea42520] 2024-03-24T00:40:06.952 INFO:tasks.ceph.mds.d.smithi053.stderr: 2: pthread_kill() 2024-03-24T00:40:06.952 INFO:tasks.ceph.mds.d.smithi053.stderr: 3: raise() 2024-03-24T00:40:06.952 INFO:tasks.ceph.mds.d.smithi053.stderr: 4: abort() 2024-03-24T00:40:06.952 INFO:tasks.ceph.mds.d.smithi053.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f485f381ed5] 2024-03-24T00:40:06.952 INFO:tasks.ceph.mds.d.smithi053.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f485f38202f] 2024-03-24T00:40:06.952 INFO:tasks.ceph.mds.d.smithi053.stderr: 7: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5618607ac4e8] 2024-03-24T00:40:06.952 INFO:tasks.ceph.mds.d.smithi053.stderr: 8: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56186069483b] 2024-03-24T00:40:06.952 INFO:tasks.ceph.mds.d.smithi053.stderr: 9: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x561860694e2c] 2024-03-24T00:40:06.952 INFO:tasks.ceph.mds.d.smithi053.stderr: 10: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5618606bc793] 2024-03-24T00:40:06.952 INFO:tasks.ceph.mds.d.smithi053.stderr: 11: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x56186069ccc6] 2024-03-24T00:40:06.952 INFO:tasks.ceph.mds.d.smithi053.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561860688ff7] 2024-03-24T00:40:06.953 INFO:tasks.ceph.mds.d.smithi053.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x56186060f4bb] 2024-03-24T00:40:06.953 INFO:tasks.ceph.mds.d.smithi053.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56186060f96c] 2024-03-24T00:40:06.953 INFO:tasks.ceph.mds.d.smithi053.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5618605f4e95] 2024-03-24T00:40:06.953 INFO:tasks.ceph.mds.d.smithi053.stderr: 16: (DispatchQueue::entry()+0x5ea) [0x7f485f57bb0a] 2024-03-24T00:40:06.953 INFO:tasks.ceph.mds.d.smithi053.stderr: 17: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f485f613911] 2024-03-24T00:40:06.953 INFO:tasks.ceph.mds.d.smithi053.stderr: 18: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f485ea94b43] 2024-03-24T00:40:06.953 INFO:tasks.ceph.mds.d.smithi053.stderr: 19: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f485eb26a00] 2024-03-24T00:40:06.953 INFO:tasks.ceph.mds.d.smithi053.stderr:2024-03-24T00:40:06.949+0000 7f4858000640 -1 *** Caught signal (Aborted) ** 2024-03-24T00:40:06.953 INFO:tasks.ceph.mds.d.smithi053.stderr: in thread 7f4858000640 thread_name:ms_dispatch 2024-03-24T00:40:06.953 INFO:tasks.ceph.mds.d.smithi053.stderr: 2024-03-24T00:40:06.953 INFO:tasks.ceph.mds.d.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:40:06.954 INFO:tasks.ceph.mds.d.smithi053.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f485ea42520] 2024-03-24T00:40:06.954 INFO:tasks.ceph.mds.d.smithi053.stderr: 2: pthread_kill() 2024-03-24T00:40:06.954 INFO:tasks.ceph.mds.d.smithi053.stderr: 3: raise() 2024-03-24T00:40:06.954 INFO:tasks.ceph.mds.d.smithi053.stderr: 4: abort() 2024-03-24T00:40:06.954 INFO:tasks.ceph.mds.d.smithi053.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f485f381ed5] 2024-03-24T00:40:06.954 INFO:tasks.ceph.mds.d.smithi053.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f485f38202f] 2024-03-24T00:40:06.954 INFO:tasks.ceph.mds.d.smithi053.stderr: 7: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5618607ac4e8] 2024-03-24T00:40:06.954 INFO:tasks.ceph.mds.d.smithi053.stderr: 8: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56186069483b] 2024-03-24T00:40:06.954 INFO:tasks.ceph.mds.d.smithi053.stderr: 9: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x561860694e2c] 2024-03-24T00:40:06.954 INFO:tasks.ceph.mds.d.smithi053.stderr: 10: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5618606bc793] 2024-03-24T00:40:06.955 INFO:tasks.ceph.mds.d.smithi053.stderr: 11: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x56186069ccc6] 2024-03-24T00:40:06.955 INFO:tasks.ceph.mds.d.smithi053.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561860688ff7] 2024-03-24T00:40:06.955 INFO:tasks.ceph.mds.d.smithi053.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x56186060f4bb] 2024-03-24T00:40:06.955 INFO:tasks.ceph.mds.d.smithi053.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56186060f96c] 2024-03-24T00:40:06.955 INFO:tasks.ceph.mds.d.smithi053.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5618605f4e95] 2024-03-24T00:40:06.955 INFO:tasks.ceph.mds.d.smithi053.stderr: 16: (DispatchQueue::entry()+0x5ea) [0x7f485f57bb0a] 2024-03-24T00:40:06.955 INFO:tasks.ceph.mds.d.smithi053.stderr: 17: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f485f613911] 2024-03-24T00:40:06.955 INFO:tasks.ceph.mds.d.smithi053.stderr: 18: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f485ea94b43] 2024-03-24T00:40:06.955 INFO:tasks.ceph.mds.d.smithi053.stderr: 19: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f485eb26a00] 2024-03-24T00:40:06.955 INFO:tasks.ceph.mds.d.smithi053.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:40:06.955 INFO:tasks.ceph.mds.d.smithi053.stderr: 2024-03-24T00:40:06.990 INFO:tasks.ceph.mds.d.smithi053.stderr: -1> 2024-03-24T00:40:06.949+0000 7f4858000640 -1 ./src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4858000640 time 2024-03-24T00:40:06.951164+0000 2024-03-24T00:40:06.990 INFO:tasks.ceph.mds.d.smithi053.stderr:./src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:40:06.990 INFO:tasks.ceph.mds.d.smithi053.stderr: 2024-03-24T00:40:06.990 INFO:tasks.ceph.mds.d.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:40:06.990 INFO:tasks.ceph.mds.d.smithi053.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f485f381e7b] 2024-03-24T00:40:06.990 INFO:tasks.ceph.mds.d.smithi053.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f485f38202f] 2024-03-24T00:40:06.990 INFO:tasks.ceph.mds.d.smithi053.stderr: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5618607ac4e8] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56186069483b] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x561860694e2c] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 6: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5618606bc793] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x56186069ccc6] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561860688ff7] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x56186060f4bb] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56186060f96c] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5618605f4e95] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 12: (DispatchQueue::entry()+0x5ea) [0x7f485f57bb0a] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 13: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f485f613911] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 14: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f485ea94b43] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f485eb26a00] 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 0> 2024-03-24T00:40:06.949+0000 7f4858000640 -1 *** Caught signal (Aborted) ** 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: in thread 7f4858000640 thread_name:ms_dispatch 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: 2024-03-24T00:40:06.991 INFO:tasks.ceph.mds.d.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f485ea42520] 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 2: pthread_kill() 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 3: raise() 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 4: abort() 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f485f381ed5] 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f485f38202f] 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 7: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5618607ac4e8] 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 8: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56186069483b] 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 9: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x561860694e2c] 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 10: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5618606bc793] 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 11: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x56186069ccc6] 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561860688ff7] 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x56186060f4bb] 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56186060f96c] 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5618605f4e95] 2024-03-24T00:40:06.992 INFO:tasks.ceph.mds.d.smithi053.stderr: 16: (DispatchQueue::entry()+0x5ea) [0x7f485f57bb0a] 2024-03-24T00:40:06.993 INFO:tasks.ceph.mds.d.smithi053.stderr: 17: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f485f613911] 2024-03-24T00:40:06.993 INFO:tasks.ceph.mds.d.smithi053.stderr: 18: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f485ea94b43] 2024-03-24T00:40:06.993 INFO:tasks.ceph.mds.d.smithi053.stderr: 19: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f485eb26a00] 2024-03-24T00:40:06.993 INFO:tasks.ceph.mds.d.smithi053.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:40:06.993 INFO:tasks.ceph.mds.d.smithi053.stderr: 2024-03-24T00:40:07.002 INFO:tasks.ceph.mds.d.smithi053.stderr: -1> 2024-03-24T00:40:06.949+0000 7f4858000640 -1 ./src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4858000640 time 2024-03-24T00:40:06.951164+0000 2024-03-24T00:40:07.002 INFO:tasks.ceph.mds.d.smithi053.stderr:./src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:40:07.002 INFO:tasks.ceph.mds.d.smithi053.stderr: 2024-03-24T00:40:07.002 INFO:tasks.ceph.mds.d.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:40:07.002 INFO:tasks.ceph.mds.d.smithi053.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7f485f381e7b] 2024-03-24T00:40:07.002 INFO:tasks.ceph.mds.d.smithi053.stderr: 2: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f485f38202f] 2024-03-24T00:40:07.002 INFO:tasks.ceph.mds.d.smithi053.stderr: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5618607ac4e8] 2024-03-24T00:40:07.002 INFO:tasks.ceph.mds.d.smithi053.stderr: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56186069483b] 2024-03-24T00:40:07.002 INFO:tasks.ceph.mds.d.smithi053.stderr: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x561860694e2c] 2024-03-24T00:40:07.002 INFO:tasks.ceph.mds.d.smithi053.stderr: 6: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5618606bc793] 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x56186069ccc6] 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561860688ff7] 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x56186060f4bb] 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56186060f96c] 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5618605f4e95] 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 12: (DispatchQueue::entry()+0x5ea) [0x7f485f57bb0a] 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 13: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f485f613911] 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 14: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f485ea94b43] 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f485eb26a00] 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 0> 2024-03-24T00:40:06.949+0000 7f4858000640 -1 *** Caught signal (Aborted) ** 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: in thread 7f4858000640 thread_name:ms_dispatch 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:40:07.003 INFO:tasks.ceph.mds.d.smithi053.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f485ea42520] 2024-03-24T00:40:07.004 INFO:tasks.ceph.mds.d.smithi053.stderr: 2: pthread_kill() 2024-03-24T00:40:07.004 INFO:tasks.ceph.mds.d.smithi053.stderr: 3: raise() 2024-03-24T00:40:07.004 INFO:tasks.ceph.mds.d.smithi053.stderr: 4: abort() 2024-03-24T00:40:07.004 INFO:tasks.ceph.mds.d.smithi053.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f485f381ed5] 2024-03-24T00:40:07.004 INFO:tasks.ceph.mds.d.smithi053.stderr: 6: /usr/lib/ceph/libceph-common.so.2(+0x18202f) [0x7f485f38202f] 2024-03-24T00:40:07.004 INFO:tasks.ceph.mds.d.smithi053.stderr: 7: (Locker::set_xlocks_done(MutationImpl*, bool)+0x3d8) [0x5618607ac4e8] 2024-03-24T00:40:07.004 INFO:tasks.ceph.mds.d.smithi053.stderr: 8: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3bb) [0x56186069483b] 2024-03-24T00:40:07.004 INFO:tasks.ceph.mds.d.smithi053.stderr: 9: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xbc) [0x561860694e2c] 2024-03-24T00:40:07.004 INFO:tasks.ceph.mds.d.smithi053.stderr: 10: (Server::handle_client_mkdir(boost::intrusive_ptr const&)+0x693) [0x5618606bc793] 2024-03-24T00:40:07.004 INFO:tasks.ceph.mds.d.smithi053.stderr: 11: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x6a6) [0x56186069ccc6] 2024-03-24T00:40:07.005 INFO:tasks.ceph.mds.d.smithi053.stderr: 12: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561860688ff7] 2024-03-24T00:40:07.005 INFO:tasks.ceph.mds.d.smithi053.stderr: 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x4db) [0x56186060f4bb] 2024-03-24T00:40:07.005 INFO:tasks.ceph.mds.d.smithi053.stderr: 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56186060f96c] 2024-03-24T00:40:07.005 INFO:tasks.ceph.mds.d.smithi053.stderr: 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x195) [0x5618605f4e95] 2024-03-24T00:40:07.005 INFO:tasks.ceph.mds.d.smithi053.stderr: 16: (DispatchQueue::entry()+0x5ea) [0x7f485f57bb0a] 2024-03-24T00:40:07.005 INFO:tasks.ceph.mds.d.smithi053.stderr: 17: /usr/lib/ceph/libceph-common.so.2(+0x413911) [0x7f485f613911] 2024-03-24T00:40:07.005 INFO:tasks.ceph.mds.d.smithi053.stderr: 18: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f485ea94b43] 2024-03-24T00:40:07.005 INFO:tasks.ceph.mds.d.smithi053.stderr: 19: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f485eb26a00] 2024-03-24T00:40:07.005 INFO:tasks.ceph.mds.d.smithi053.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:40:07.005 INFO:tasks.ceph.mds.d.smithi053.stderr: 2024-03-24T00:40:07.336 INFO:tasks.ceph.mds.d.smithi053.stderr:daemon-helper: command crashed with signal 6 2024-03-24T00:40:07.454 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~13s 2024-03-24T00:40:08.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:40:08.168 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:40:13.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~20s 2024-03-24T00:40:13.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-03-24T00:40:20.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~26s 2024-03-24T00:40:20.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~7s 2024-03-24T00:40:26.990 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~33s 2024-03-24T00:40:26.990 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~13s 2024-03-24T00:40:33.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~39s 2024-03-24T00:40:33.502 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~20s 2024-03-24T00:40:38.224 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:40:38.229 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:40:40.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~46s 2024-03-24T00:40:40.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~26s 2024-03-24T00:40:46.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~52s 2024-03-24T00:40:46.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~33s 2024-03-24T00:40:53.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~59s 2024-03-24T00:40:53.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~39s 2024-03-24T00:40:59.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~65s 2024-03-24T00:40:59.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~46s 2024-03-24T00:41:06.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~72s 2024-03-24T00:41:06.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~52s 2024-03-24T00:41:08.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:41:08.283 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:41:12.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~78s 2024-03-24T00:41:12.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~59s 2024-03-24T00:41:19.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~85s 2024-03-24T00:41:19.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~65s 2024-03-24T00:41:25.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~91s 2024-03-24T00:41:25.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~72s 2024-03-24T00:41:32.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~98s 2024-03-24T00:41:32.103 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~78s 2024-03-24T00:41:38.333 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:41:38.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:41:38.614 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~104s 2024-03-24T00:41:38.615 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~85s 2024-03-24T00:41:45.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~111s 2024-03-24T00:41:45.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~91s 2024-03-24T00:41:51.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~118s 2024-03-24T00:41:51.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~98s 2024-03-24T00:41:58.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~124s 2024-03-24T00:41:58.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~104s 2024-03-24T00:42:04.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~131s 2024-03-24T00:42:04.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~111s 2024-03-24T00:42:08.392 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:42:08.396 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:42:11.171 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~137s 2024-03-24T00:42:11.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~117s 2024-03-24T00:42:17.682 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~144s 2024-03-24T00:42:17.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~124s 2024-03-24T00:42:24.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~150s 2024-03-24T00:42:24.194 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~130s 2024-03-24T00:42:30.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~157s 2024-03-24T00:42:30.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~137s 2024-03-24T00:42:37.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~163s 2024-03-24T00:42:37.217 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~143s 2024-03-24T00:42:38.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:42:38.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:42:43.729 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~170s 2024-03-24T00:42:43.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~150s 2024-03-24T00:42:50.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~176s 2024-03-24T00:42:50.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~156s 2024-03-24T00:42:56.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~183s 2024-03-24T00:42:56.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~163s 2024-03-24T00:43:03.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~189s 2024-03-24T00:43:03.267 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~169s 2024-03-24T00:43:08.514 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:43:08.519 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:43:09.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~196s 2024-03-24T00:43:09.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~176s 2024-03-24T00:43:16.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~202s 2024-03-24T00:43:16.290 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~182s 2024-03-24T00:43:22.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~209s 2024-03-24T00:43:22.802 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~189s 2024-03-24T00:43:29.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~215s 2024-03-24T00:43:29.315 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~195s 2024-03-24T00:43:35.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~222s 2024-03-24T00:43:35.827 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~202s 2024-03-24T00:43:38.600 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:43:38.604 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:43:42.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~228s 2024-03-24T00:43:42.338 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~208s 2024-03-24T00:43:48.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~235s 2024-03-24T00:43:48.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~215s 2024-03-24T00:43:55.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~241s 2024-03-24T00:43:55.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~221s 2024-03-24T00:44:01.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~248s 2024-03-24T00:44:01.873 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~228s 2024-03-24T00:44:08.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~254s 2024-03-24T00:44:08.383 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~234s 2024-03-24T00:44:08.651 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:44:08.654 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:44:14.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~261s 2024-03-24T00:44:14.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~241s 2024-03-24T00:44:21.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~267s 2024-03-24T00:44:21.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~247s 2024-03-24T00:44:27.738 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-03-24T00:44:27.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~274s 2024-03-24T00:44:27.918 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~254s 2024-03-24T00:44:34.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~280s 2024-03-24T00:44:34.429 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~260s 2024-03-24T00:44:38.732 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:44:38.735 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:44:40.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~287s 2024-03-24T00:44:40.940 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~267s 2024-03-24T00:44:47.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~293s 2024-03-24T00:44:47.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~273s 2024-03-24T00:44:48.724 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-03-24T00:44:53.963 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~300s 2024-03-24T00:44:53.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~280s 2024-03-24T00:45:00.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~306s 2024-03-24T00:45:00.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~287s 2024-03-24T00:45:00.476 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-03-24T00:45:00.476 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:45:00.476 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:00.486 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:45:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:45:00.535 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-24T00:45:00.535 DEBUG:teuthology.orchestra.run.smithi040:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:45:00.641 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-24T00:45:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0039400640 1 Processor -- start 2024-03-24T00:45:00.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0039400640 1 -- start start 2024-03-24T00:45:00.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0039400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f003415e0b0 0x7f00340b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0039400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00340b4b80 0x7f00340cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0039400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00340cd3d0 con 0x7f003406b600 2024-03-24T00:45:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0039400640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003415e490 con 0x7f003415e0b0 2024-03-24T00:45:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0039400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00340cdfe0 con 0x7f00340b4b80 2024-03-24T00:45:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0032800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f003415e0b0 0x7f00340b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0032800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f003415e0b0 0x7f00340b43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34078/0 (socket says 172.21.15.40:34078) 2024-03-24T00:45:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0032800640 1 -- 172.21.15.40:0/451532014 learned_addr learned my addr 172.21.15.40:0/451532014 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:00.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f002be00640 1 --2- 172.21.15.40:0/451532014 >> v2:172.21.15.53:3300/0 conn(0x7f00340b4b80 0x7f00340cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0032800640 1 -- 172.21.15.40:0/451532014 >> v2:172.21.15.53:3300/0 conn(0x7f00340b4b80 msgr2=0x7f00340cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0032800640 1 --2- 172.21.15.40:0/451532014 >> v2:172.21.15.53:3300/0 conn(0x7f00340b4b80 0x7f00340cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0032800640 1 -- 172.21.15.40:0/451532014 >> v1:172.21.15.53:6789/0 conn(0x7f003406b600 legacy=0x7f00340b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.669+0000 7f0032800640 1 -- 172.21.15.40:0/451532014 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00340cd6b0 con 0x7f003415e0b0 2024-03-24T00:45:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.673+0000 7f0032800640 1 --2- 172.21.15.40:0/451532014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f003415e0b0 0x7f00340b43a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f0034071830 tx=0x7f002401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06e2a191f665c83 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.673+0000 7f0031e00640 1 -- 172.21.15.40:0/451532014 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00240b5070 con 0x7f003415e0b0 2024-03-24T00:45:00.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.673+0000 7f0031e00640 1 -- 172.21.15.40:0/451532014 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00240a8a10 con 0x7f003415e0b0 2024-03-24T00:45:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.673+0000 7f0031e00640 1 -- 172.21.15.40:0/451532014 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00240a8de0 con 0x7f003415e0b0 2024-03-24T00:45:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.673+0000 7f0039400640 1 -- 172.21.15.40:0/451532014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f003415e0b0 msgr2=0x7f00340b43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.673+0000 7f0039400640 1 --2- 172.21.15.40:0/451532014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f003415e0b0 0x7f00340b43a0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f0034071830 tx=0x7f002401ed10 comp rx=0 tx=0).stop 2024-03-24T00:45:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.673+0000 7f0039400640 1 -- 172.21.15.40:0/451532014 shutdown_connections 2024-03-24T00:45:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.673+0000 7f0039400640 1 --2- 172.21.15.40:0/451532014 >> v2:172.21.15.53:3300/0 conn(0x7f00340b4b80 0x7f00340cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.673+0000 7f0039400640 1 --2- 172.21.15.40:0/451532014 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f003415e0b0 0x7f00340b43a0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.673+0000 7f0039400640 1 -- 172.21.15.40:0/451532014 >> 172.21.15.40:0/451532014 conn(0x7f00340b3880 msgr2=0x7f00340b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.673+0000 7f0039400640 1 -- 172.21.15.40:0/451532014 shutdown_connections 2024-03-24T00:45:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.673+0000 7f0039400640 1 -- 172.21.15.40:0/451532014 wait complete. 2024-03-24T00:45:00.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0039400640 1 Processor -- start 2024-03-24T00:45:00.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0039400640 1 -- start start 2024-03-24T00:45:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0039400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f003406b600 0x7f00340b7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0039400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00340b4b80 0x7f00341515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0039400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f003415e0b0 0x7f00341f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0039400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003406c480 con 0x7f003406b600 2024-03-24T00:45:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0039400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340cdc70 con 0x7f003415e0b0 2024-03-24T00:45:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0039400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003406bab0 con 0x7f00340b4b80 2024-03-24T00:45:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0032800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f003406b600 0x7f00340b7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:00.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0032800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f003406b600 0x7f00340b7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34084/0 (socket says 172.21.15.40:34084) 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0032800640 1 -- 172.21.15.40:0/2403881085 learned_addr learned my addr 172.21.15.40:0/2403881085 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0033200640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f003415e0b0 0x7f00341f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f002be00640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00340b4b80 0x7f00341515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0032800640 1 -- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00340b4b80 msgr2=0x7f00341515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0032800640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00340b4b80 0x7f00341515f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0032800640 1 -- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f003415e0b0 msgr2=0x7f00341f7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0032800640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f003415e0b0 0x7f00341f7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0032800640 1 -- 172.21.15.40:0/2403881085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00341f80e0 con 0x7f003406b600 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0033200640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f003415e0b0 0x7f00341f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0032800640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f003406b600 0x7f00340b7140 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f002401b920 tx=0x7f002401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f002aa00640 1 -- 172.21.15.40:0/2403881085 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00240b5070 con 0x7f003406b600 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f002aa00640 1 -- 172.21.15.40:0/2403881085 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00240b0ab0 con 0x7f003406b600 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f002aa00640 1 -- 172.21.15.40:0/2403881085 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00240bb410 con 0x7f003406b600 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0039400640 1 -- 172.21.15.40:0/2403881085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00340b6950 con 0x7f003406b600 2024-03-24T00:45:00.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.677+0000 7f0039400640 1 -- 172.21.15.40:0/2403881085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00340c8ee0 con 0x7f003406b600 2024-03-24T00:45:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.681+0000 7f0039400640 1 -- 172.21.15.40:0/2403881085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0034068170 con 0x7f003406b600 2024-03-24T00:45:00.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.681+0000 7f002aa00640 1 -- 172.21.15.40:0/2403881085 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f00240c2060 con 0x7f003406b600 2024-03-24T00:45:00.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.681+0000 7f002aa00640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0010067660 0x7f0010069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:00.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.681+0000 7f002aa00640 1 -- 172.21.15.40:0/2403881085 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f0024122c10 con 0x7f003406b600 2024-03-24T00:45:00.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.681+0000 7f002be00640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0010067660 0x7f0010069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:00.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.685+0000 7f002be00640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0010067660 0x7f0010069b20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f00340cf150 tx=0x7f001c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:00.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.689+0000 7f002aa00640 1 -- 172.21.15.40:0/2403881085 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f0024018870 con 0x7f003406b600 2024-03-24T00:45:00.730 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-24T00:45:00.730 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:45:00.730 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:00.739 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:45:00.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.857+0000 7f0039400640 1 -- 172.21.15.40:0/2403881085 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00340d1a00 con 0x7f003406b600 2024-03-24T00:45:00.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f002aa00640 1 -- 172.21.15.40:0/2403881085 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+16789 (secure 0 0 0) 0x7f00240f3870 con 0x7f003406b600 2024-03-24T00:45:00.863 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:45:00.863 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:40:01.907736+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:05.964092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:06.525558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.c","mappings":[{"from":2,"to":0}]},{"pgid":"19.23","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6836/2255755838":"2024-03-25T00:40:01.907668+0000","172.21.15.53:6834/2255755838":"2024-03-25T00:40:01.907668+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:45:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f0039400640 1 -- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0010067660 msgr2=0x7f0010069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f0039400640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0010067660 0x7f0010069b20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f00340cf150 tx=0x7f001c008040 comp rx=0 tx=0).stop 2024-03-24T00:45:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f0039400640 1 -- 172.21.15.40:0/2403881085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f003406b600 msgr2=0x7f00340b7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f0039400640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f003406b600 0x7f00340b7140 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f002401b920 tx=0x7f002401bab0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f0039400640 1 -- 172.21.15.40:0/2403881085 shutdown_connections 2024-03-24T00:45:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f0039400640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f0010067660 0x7f0010069b20 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f0039400640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f003415e0b0 0x7f00341f7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f0039400640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00340b4b80 0x7f00341515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f0039400640 1 --2- 172.21.15.40:0/2403881085 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f003406b600 0x7f00340b7140 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f0039400640 1 -- 172.21.15.40:0/2403881085 >> 172.21.15.40:0/2403881085 conn(0x7f00340b3880 msgr2=0x7f003406ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f0039400640 1 -- 172.21.15.40:0/2403881085 shutdown_connections 2024-03-24T00:45:00.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.861+0000 7f0039400640 1 -- 172.21.15.40:0/2403881085 wait complete. 2024-03-24T00:45:00.886 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:39:05.964092+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:39:06.525558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:45:00.887 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:45:00.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.925+0000 7ffa69800640 1 Processor -- start 2024-03-24T00:45:00.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.925+0000 7ffa69800640 1 -- start start 2024-03-24T00:45:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.925+0000 7ffa69800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffa64145760 0x7ffa6406f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.925+0000 7ffa69800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa64072750 0x7ffa6406e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.925+0000 7ffa69800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa640cd630 con 0x7ffa6406ea50 2024-03-24T00:45:00.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.925+0000 7ffa69800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa640cd910 con 0x7ffa64145760 2024-03-24T00:45:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.925+0000 7ffa69800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa640cdbf0 con 0x7ffa64072750 2024-03-24T00:45:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.925+0000 7ffa62000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa64072750 0x7ffa6406e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.925+0000 7ffa62a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffa64145760 0x7ffa6406f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.925+0000 7ffa63400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ffa6406ea50 0x7ffa640cf200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:37040/0 (socket says 172.21.15.40:37040) 2024-03-24T00:45:00.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.925+0000 7ffa63400640 1 -- 172.21.15.40:0/3030544013 learned_addr learned my addr 172.21.15.40:0/3030544013 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:45:00.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.929+0000 7ffa62a00640 1 -- 172.21.15.40:0/3030544013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa64072750 msgr2=0x7ffa6406e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:00.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.929+0000 7ffa62a00640 1 --2- 172.21.15.40:0/3030544013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa64072750 0x7ffa6406e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.929+0000 7ffa62a00640 1 -- 172.21.15.40:0/3030544013 >> v1:172.21.15.53:6789/0 conn(0x7ffa6406ea50 legacy=0x7ffa640cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:00.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.929+0000 7ffa62a00640 1 -- 172.21.15.40:0/3030544013 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6406dd10 con 0x7ffa64145760 2024-03-24T00:45:00.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.929+0000 7ffa62000640 1 --2- 172.21.15.40:0/3030544013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa64072750 0x7ffa6406e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:00.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.933+0000 7ffa62a00640 1 --2- 172.21.15.40:0/3030544013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64145760 0x7ffa6406f580 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ffa540097c0 tx=0x7ffa5401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd5b99cb9c31aefd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:00.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.933+0000 7ffa61600640 1 -- 172.21.15.40:0/3030544013 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa540b0070 con 0x7ffa64145760 2024-03-24T00:45:00.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.933+0000 7ffa61600640 1 -- 172.21.15.40:0/3030544013 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa540ab600 con 0x7ffa64145760 2024-03-24T00:45:00.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.933+0000 7ffa61600640 1 -- 172.21.15.40:0/3030544013 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa540ab9d0 con 0x7ffa64145760 2024-03-24T00:45:00.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.933+0000 7ffa69800640 1 -- 172.21.15.40:0/3030544013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64145760 msgr2=0x7ffa6406f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:00.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.933+0000 7ffa69800640 1 --2- 172.21.15.40:0/3030544013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64145760 0x7ffa6406f580 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ffa540097c0 tx=0x7ffa5401c4a0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.933+0000 7ffa69800640 1 -- 172.21.15.40:0/3030544013 shutdown_connections 2024-03-24T00:45:00.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.933+0000 7ffa69800640 1 --2- 172.21.15.40:0/3030544013 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa64072750 0x7ffa6406e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.933+0000 7ffa69800640 1 --2- 172.21.15.40:0/3030544013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64145760 0x7ffa6406f580 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.933+0000 7ffa69800640 1 -- 172.21.15.40:0/3030544013 >> 172.21.15.40:0/3030544013 conn(0x7ffa640d0c20 msgr2=0x7ffa640d1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:00.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.933+0000 7ffa69800640 1 -- 172.21.15.40:0/3030544013 shutdown_connections 2024-03-24T00:45:00.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.933+0000 7ffa69800640 1 -- 172.21.15.40:0/3030544013 wait complete. 2024-03-24T00:45:00.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa69800640 1 Processor -- start 2024-03-24T00:45:00.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa69800640 1 -- start start 2024-03-24T00:45:00.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa69800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa6406ea50 0x7ffa64159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:00.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa69800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64072750 0x7ffa64161a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:00.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa69800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa64145760 0x7ffa64161f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:00.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa69800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa640cfa90 con 0x7ffa64145760 2024-03-24T00:45:00.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa69800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa640cf910 con 0x7ffa64072750 2024-03-24T00:45:00.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa69800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa64168a60 con 0x7ffa6406ea50 2024-03-24T00:45:00.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa64145760 0x7ffa64161f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:00.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa63400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa64145760 0x7ffa64161f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34088/0 (socket says 172.21.15.40:34088) 2024-03-24T00:45:00.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa63400640 1 -- 172.21.15.40:0/2222675938 learned_addr learned my addr 172.21.15.40:0/2222675938 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:00.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa62000640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64072750 0x7ffa64161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:00.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa62a00640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa6406ea50 0x7ffa64159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa63400640 1 -- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa6406ea50 msgr2=0x7ffa64159d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa63400640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa6406ea50 0x7ffa64159d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa63400640 1 -- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64072750 msgr2=0x7ffa64161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa63400640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64072750 0x7ffa64161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:00.943 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa63400640 1 -- 172.21.15.40:0/2222675938 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6415b240 con 0x7ffa64145760 2024-03-24T00:45:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.937+0000 7ffa62000640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64072750 0x7ffa64161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.941+0000 7ffa63400640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa64145760 0x7ffa64161f80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c010e80 tx=0x7ffa5c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.941+0000 7ffa5aa00640 1 -- 172.21.15.40:0/2222675938 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c019640 con 0x7ffa64145760 2024-03-24T00:45:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.941+0000 7ffa5aa00640 1 -- 172.21.15.40:0/2222675938 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5c0197e0 con 0x7ffa64145760 2024-03-24T00:45:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.941+0000 7ffa69800640 1 -- 172.21.15.40:0/2222675938 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6415b520 con 0x7ffa64145760 2024-03-24T00:45:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.941+0000 7ffa5aa00640 1 -- 172.21.15.40:0/2222675938 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c019b90 con 0x7ffa64145760 2024-03-24T00:45:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.941+0000 7ffa69800640 1 -- 172.21.15.40:0/2222675938 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6413beb0 con 0x7ffa64145760 2024-03-24T00:45:00.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.941+0000 7ffa69800640 1 -- 172.21.15.40:0/2222675938 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa64068170 con 0x7ffa64145760 2024-03-24T00:45:00.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.941+0000 7ffa5aa00640 1 -- 172.21.15.40:0/2222675938 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ffa5c007070 con 0x7ffa64145760 2024-03-24T00:45:00.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.945+0000 7ffa5aa00640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffa44067660 0x7ffa44069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:00.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.945+0000 7ffa5aa00640 1 -- 172.21.15.40:0/2222675938 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ffa5c08a290 con 0x7ffa64145760 2024-03-24T00:45:00.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.945+0000 7ffa62a00640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffa44067660 0x7ffa44069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:00.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.945+0000 7ffa62a00640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffa44067660 0x7ffa44069b20 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ffa540ad000 tx=0x7ffa540a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:00.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:00.949+0000 7ffa5aa00640 1 -- 172.21.15.40:0/2222675938 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ffa5c05a820 con 0x7ffa64145760 2024-03-24T00:45:01.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9b5400640 1 Processor -- start 2024-03-24T00:45:01.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9b5400640 1 -- start start 2024-03-24T00:45:01.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9b5400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff9b015df80 0x7ff9b0070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9b5400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b006c950 0x7ff9b00d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9b5400640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b006d0f0 con 0x7ff9b006bd20 2024-03-24T00:45:01.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9b5400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b006b890 con 0x7ff9b015df80 2024-03-24T00:45:01.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9b5400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b006b280 con 0x7ff9b006c950 2024-03-24T00:45:01.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9af200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7ff9b006bd20 0x7ff9b0161ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52930/0 (socket says 172.21.15.40:52930) 2024-03-24T00:45:01.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9af200640 1 -- 172.21.15.40:0/2224425120 learned_addr learned my addr 172.21.15.40:0/2224425120 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:45:01.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9ae800640 1 --2- 172.21.15.40:0/2224425120 >> v2:172.21.15.53:3300/0 conn(0x7ff9b015df80 0x7ff9b0070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9ade00640 1 --2- 172.21.15.40:0/2224425120 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b006c950 0x7ff9b00d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9ae800640 1 -- 172.21.15.40:0/2224425120 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b006c950 msgr2=0x7ff9b00d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:01.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9ae800640 1 --2- 172.21.15.40:0/2224425120 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b006c950 0x7ff9b00d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9ae800640 1 -- 172.21.15.40:0/2224425120 >> v1:172.21.15.40:6789/0 conn(0x7ff9b006bd20 legacy=0x7ff9b0161ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.037+0000 7ff9ae800640 1 -- 172.21.15.40:0/2224425120 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b006ce10 con 0x7ff9b015df80 2024-03-24T00:45:01.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.041+0000 7ff9ae800640 1 --2- 172.21.15.40:0/2224425120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b015df80 0x7ff9b0070ff0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff9a00097c0 tx=0x7ff9a001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90bef2000d0a5404 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.041+0000 7ff9ad400640 1 -- 172.21.15.40:0/2224425120 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a00b2070 con 0x7ff9b015df80 2024-03-24T00:45:01.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.041+0000 7ff9ad400640 1 -- 172.21.15.40:0/2224425120 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a00ad680 con 0x7ff9b015df80 2024-03-24T00:45:01.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.041+0000 7ff9ad400640 1 -- 172.21.15.40:0/2224425120 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a00ada50 con 0x7ff9b015df80 2024-03-24T00:45:01.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.041+0000 7ff9b5400640 1 -- 172.21.15.40:0/2224425120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b015df80 msgr2=0x7ff9b0070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.041+0000 7ff9b5400640 1 --2- 172.21.15.40:0/2224425120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b015df80 0x7ff9b0070ff0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff9a00097c0 tx=0x7ff9a001c470 comp rx=0 tx=0).stop 2024-03-24T00:45:01.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.041+0000 7ff9b5400640 1 -- 172.21.15.40:0/2224425120 shutdown_connections 2024-03-24T00:45:01.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.041+0000 7ff9b5400640 1 --2- 172.21.15.40:0/2224425120 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b006c950 0x7ff9b00d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.041+0000 7ff9b5400640 1 --2- 172.21.15.40:0/2224425120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b015df80 0x7ff9b0070ff0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.041+0000 7ff9b5400640 1 -- 172.21.15.40:0/2224425120 >> 172.21.15.40:0/2224425120 conn(0x7ff9b006ecf0 msgr2=0x7ff9b006f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:01.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.041+0000 7ff9b5400640 1 -- 172.21.15.40:0/2224425120 shutdown_connections 2024-03-24T00:45:01.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.041+0000 7ff9b5400640 1 -- 172.21.15.40:0/2224425120 wait complete. 2024-03-24T00:45:01.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.045+0000 7ff9b5400640 1 Processor -- start 2024-03-24T00:45:01.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.045+0000 7ff9b5400640 1 -- start start 2024-03-24T00:45:01.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.045+0000 7ff9b5400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b006bd20 0x7ff9b015a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.045+0000 7ff9b5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b006c950 0x7ff9b00b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.045+0000 7ff9b5400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b015df80 0x7ff9b00b7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.045+0000 7ff9b5400640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b016af90 con 0x7ff9b006bd20 2024-03-24T00:45:01.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.045+0000 7ff9b5400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b016ae10 con 0x7ff9b006c950 2024-03-24T00:45:01.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.045+0000 7ff9b5400640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b016b110 con 0x7ff9b015df80 2024-03-24T00:45:01.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9ae800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b006bd20 0x7ff9b015a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9ae800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b006bd20 0x7ff9b015a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34102/0 (socket says 172.21.15.40:34102) 2024-03-24T00:45:01.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9ae800640 1 -- 172.21.15.40:0/348419810 learned_addr learned my addr 172.21.15.40:0/348419810 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:01.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9ade00640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b006c950 0x7ff9b00b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.056 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9af200640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b015df80 0x7ff9b00b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9ae800640 1 -- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b015df80 msgr2=0x7ff9b00b7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9ae800640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b015df80 0x7ff9b00b7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9ae800640 1 -- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b006c950 msgr2=0x7ff9b00b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9ae800640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b006c950 0x7ff9b00b04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9ae800640 1 -- 172.21.15.40:0/348419810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b015af60 con 0x7ff9b006bd20 2024-03-24T00:45:01.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9ade00640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b006c950 0x7ff9b00b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:01.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9ae800640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b006bd20 0x7ff9b015a960 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7ff9a001ab20 tx=0x7ff9a001acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9a6a00640 1 -- 172.21.15.40:0/348419810 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a00b2040 con 0x7ff9b006bd20 2024-03-24T00:45:01.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9a6a00640 1 -- 172.21.15.40:0/348419810 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a00be070 con 0x7ff9b006bd20 2024-03-24T00:45:01.057 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9b5400640 1 -- 172.21.15.40:0/348419810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b00af6e0 con 0x7ff9b006bd20 2024-03-24T00:45:01.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9a6a00640 1 -- 172.21.15.40:0/348419810 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a00add50 con 0x7ff9b006bd20 2024-03-24T00:45:01.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9b5400640 1 -- 172.21.15.40:0/348419810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b00afca0 con 0x7ff9b006bd20 2024-03-24T00:45:01.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.049+0000 7ff9b5400640 1 -- 172.21.15.40:0/348419810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b0068170 con 0x7ff9b006bd20 2024-03-24T00:45:01.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.053+0000 7ff9a6a00640 1 -- 172.21.15.40:0/348419810 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff9a00c3460 con 0x7ff9b006bd20 2024-03-24T00:45:01.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.053+0000 7ff9a6a00640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff990067660 0x7ff990069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.053+0000 7ff9a6a00640 1 -- 172.21.15.40:0/348419810 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff9a011fd30 con 0x7ff9b006bd20 2024-03-24T00:45:01.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.053+0000 7ff9ade00640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff990067660 0x7ff990069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.053+0000 7ff9ade00640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff990067660 0x7ff990069b20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff9b015a750 tx=0x7ff99c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.061+0000 7ff9a6a00640 1 -- 172.21.15.40:0/348419810 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff9a0016090 con 0x7ff9b006bd20 2024-03-24T00:45:01.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 -- 172.21.15.40:0/348419810 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff9b0160440 con 0x7ff9b006bd20 2024-03-24T00:45:01.228 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9a6a00640 1 -- 172.21.15.40:0/348419810 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff9a00f0800 con 0x7ff9b006bd20 2024-03-24T00:45:01.228 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:45:01.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 -- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff990067660 msgr2=0x7ff990069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff990067660 0x7ff990069b20 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff9b015a750 tx=0x7ff99c008040 comp rx=0 tx=0).stop 2024-03-24T00:45:01.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 -- 172.21.15.40:0/348419810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b006bd20 msgr2=0x7ff9b015a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b006bd20 0x7ff9b015a960 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7ff9a001ab20 tx=0x7ff9a001acb0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 -- 172.21.15.40:0/348419810 shutdown_connections 2024-03-24T00:45:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff990067660 0x7ff990069b20 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9b015df80 0x7ff9b00b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9b006c950 0x7ff9b00b04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 --2- 172.21.15.40:0/348419810 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff9b006bd20 0x7ff9b015a960 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 -- 172.21.15.40:0/348419810 >> 172.21.15.40:0/348419810 conn(0x7ff9b006ecf0 msgr2=0x7ff9b01570e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 -- 172.21.15.40:0/348419810 shutdown_connections 2024-03-24T00:45:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.225+0000 7ff9b5400640 1 -- 172.21.15.40:0/348419810 wait complete. 2024-03-24T00:45:01.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.237+0000 7ffa69800640 1 -- 172.21.15.40:0/2222675938 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa6415dfa0 con 0x7ffa64145760 2024-03-24T00:45:01.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa5aa00640 1 -- 172.21.15.40:0/2222675938 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+16789 (secure 0 0 0) 0x7ffa5c05e8c0 con 0x7ffa64145760 2024-03-24T00:45:01.243 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:45:01.243 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:40:01.907736+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:05.964092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:06.525558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.c","mappings":[{"from":2,"to":0}]},{"pgid":"19.23","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6836/2255755838":"2024-03-25T00:40:01.907668+0000","172.21.15.53:6834/2255755838":"2024-03-25T00:40:01.907668+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:45:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa69800640 1 -- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffa44067660 msgr2=0x7ffa44069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa69800640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffa44067660 0x7ffa44069b20 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ffa540ad000 tx=0x7ffa540a6000 comp rx=0 tx=0).stop 2024-03-24T00:45:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa69800640 1 -- 172.21.15.40:0/2222675938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa64145760 msgr2=0x7ffa64161f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa69800640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa64145760 0x7ffa64161f80 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c010e80 tx=0x7ffa5c00a430 comp rx=0 tx=0).stop 2024-03-24T00:45:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa69800640 1 -- 172.21.15.40:0/2222675938 shutdown_connections 2024-03-24T00:45:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa69800640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ffa44067660 0x7ffa44069b20 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa69800640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ffa64145760 0x7ffa64161f80 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa69800640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffa64072750 0x7ffa64161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa69800640 1 --2- 172.21.15.40:0/2222675938 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffa6406ea50 0x7ffa64159d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa69800640 1 -- 172.21.15.40:0/2222675938 >> 172.21.15.40:0/2222675938 conn(0x7ffa640d0c20 msgr2=0x7ffa64144920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa69800640 1 -- 172.21.15.40:0/2222675938 shutdown_connections 2024-03-24T00:45:01.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.241+0000 7ffa69800640 1 -- 172.21.15.40:0/2222675938 wait complete. 2024-03-24T00:45:01.253 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:45:01.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:39:05.964092+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:39:06.525558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:45:01.267 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:45:01.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.449+0000 7f5236e00640 1 Processor -- start 2024-03-24T00:45:01.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.449+0000 7f5236e00640 1 -- start start 2024-03-24T00:45:01.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f5236e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f523015bd00 0x7f52300ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f5236e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52300728d0 0x7f52300cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f5236e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f523015f8c0 0x7f523015fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f5236e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52300cf790 con 0x7f52300728d0 2024-03-24T00:45:01.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f5236e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5230160250 con 0x7f523015bd00 2024-03-24T00:45:01.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f5236e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5230072d60 con 0x7f523015f8c0 2024-03-24T00:45:01.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f522f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52300728d0 0x7f52300cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f522f400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f52300728d0 0x7f52300cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34118/0 (socket says 172.21.15.40:34118) 2024-03-24T00:45:01.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f522f400640 1 -- 172.21.15.40:0/1418423263 learned_addr learned my addr 172.21.15.40:0/1418423263 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:01.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f522fe00640 1 --2- 172.21.15.40:0/1418423263 >> v2:172.21.15.53:3300/0 conn(0x7f523015bd00 0x7f52300ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f5234e00640 1 --2- 172.21.15.40:0/1418423263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f523015f8c0 0x7f523015fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f522f400640 1 -- 172.21.15.40:0/1418423263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f523015f8c0 msgr2=0x7f523015fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:01.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f522f400640 1 --2- 172.21.15.40:0/1418423263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f523015f8c0 0x7f523015fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f522f400640 1 -- 172.21.15.40:0/1418423263 >> v2:172.21.15.53:3300/0 conn(0x7f523015bd00 msgr2=0x7f52300ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:01.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f522f400640 1 --2- 172.21.15.40:0/1418423263 >> v2:172.21.15.53:3300/0 conn(0x7f523015bd00 0x7f52300ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7f522f400640 1 -- 172.21.15.40:0/1418423263 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5230071de0 con 0x7f52300728d0 2024-03-24T00:45:01.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7fc8bf000640 1 Processor -- start 2024-03-24T00:45:01.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.453+0000 7fc8bf000640 1 -- start start 2024-03-24T00:45:01.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8bf000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc8b815cc80 0x7fc8b806efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8bf000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b80b4e20 con 0x7fc8b815c190 2024-03-24T00:45:01.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8bf000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b80cc270 con 0x7fc8b815d2e0 2024-03-24T00:45:01.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8bf000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b80cc550 con 0x7fc8b815cc80 2024-03-24T00:45:01.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8bdc00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc8b815c190 0x7fc8b815c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52934/0 (socket says 172.21.15.40:52934) 2024-03-24T00:45:01.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8bdc00640 1 -- 172.21.15.40:0/1617884774 learned_addr learned my addr 172.21.15.40:0/1617884774 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:45:01.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8be600640 1 --2- 172.21.15.40:0/1617884774 >> v2:172.21.15.53:3300/0 conn(0x7fc8b815cc80 0x7fc8b806efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8b7e00640 1 -- 172.21.15.40:0/1617884774 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019357893 0 0) 0x7fc8b80b4e20 con 0x7fc8b815c190 2024-03-24T00:45:01.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8b7e00640 1 -- 172.21.15.40:0/1617884774 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a0003530 con 0x7fc8b815c190 2024-03-24T00:45:01.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8b7e00640 1 -- 172.21.15.40:0/1617884774 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732998978 0 0) 0x7fc8b80cc270 con 0x7fc8b815d2e0 2024-03-24T00:45:01.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8b7e00640 1 -- 172.21.15.40:0/1617884774 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b80b4e20 con 0x7fc8b815d2e0 2024-03-24T00:45:01.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8be600640 1 -- 172.21.15.40:0/1617884774 >> v1:172.21.15.53:6789/0 conn(0x7fc8b815d2e0 legacy=0x7fc8b8163730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8b7e00640 1 -- 172.21.15.40:0/1617884774 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 424870321 0 0) 0x7fc8a0003530 con 0x7fc8b815c190 2024-03-24T00:45:01.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8be600640 1 -- 172.21.15.40:0/1617884774 >> v1:172.21.15.40:6789/0 conn(0x7fc8b815c190 legacy=0x7fc8b815c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7fc8be600640 1 -- 172.21.15.40:0/1617884774 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b80cc830 con 0x7fc8b815cc80 2024-03-24T00:45:01.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7f522f400640 1 --2- 172.21.15.40:0/1418423263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52300728d0 0x7f52300cf250 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f52300733c0 tx=0x7f521c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a3e77c4cd96b8a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7f522ea00640 1 -- 172.21.15.40:0/1418423263 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c0aa610 con 0x7f52300728d0 2024-03-24T00:45:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7f522ea00640 1 -- 172.21.15.40:0/1418423263 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c0af070 con 0x7f52300728d0 2024-03-24T00:45:01.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.457+0000 7f522ea00640 1 -- 172.21.15.40:0/1418423263 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c0aa9d0 con 0x7f52300728d0 2024-03-24T00:45:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7f5236e00640 1 -- 172.21.15.40:0/1418423263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52300728d0 msgr2=0x7f52300cf250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7f5236e00640 1 --2- 172.21.15.40:0/1418423263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52300728d0 0x7f52300cf250 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f52300733c0 tx=0x7f521c01c4f0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7f5236e00640 1 -- 172.21.15.40:0/1418423263 shutdown_connections 2024-03-24T00:45:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7f5236e00640 1 --2- 172.21.15.40:0/1418423263 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f523015f8c0 0x7f523015fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7f5236e00640 1 --2- 172.21.15.40:0/1418423263 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f52300728d0 0x7f52300cf250 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7f5236e00640 1 --2- 172.21.15.40:0/1418423263 >> v2:172.21.15.53:3300/0 conn(0x7f523015bd00 0x7f52300ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7f5236e00640 1 -- 172.21.15.40:0/1418423263 >> 172.21.15.40:0/1418423263 conn(0x7f52300b1660 msgr2=0x7f52300b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:01.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7f5236e00640 1 -- 172.21.15.40:0/1418423263 shutdown_connections 2024-03-24T00:45:01.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7f5236e00640 1 -- 172.21.15.40:0/1418423263 wait complete. 2024-03-24T00:45:01.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7fc8be600640 1 --2- 172.21.15.40:0/1617884774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b815cc80 0x7fc8b806efa0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0009d90 tx=0x7fc8b001c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1004e9a624fbce11 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7fc8b7e00640 1 -- 172.21.15.40:0/1617884774 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac001680 con 0x7fc8b815cc80 2024-03-24T00:45:01.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7fc8b7e00640 1 -- 172.21.15.40:0/1617884774 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b00a64d0 con 0x7fc8b815cc80 2024-03-24T00:45:01.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7fc8b7e00640 1 -- 172.21.15.40:0/1617884774 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b00a68a0 con 0x7fc8b815cc80 2024-03-24T00:45:01.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7f5236e00640 1 Processor -- start 2024-03-24T00:45:01.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7fc8bf000640 1 -- 172.21.15.40:0/1617884774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b815cc80 msgr2=0x7fc8b806efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7fc8bf000640 1 --2- 172.21.15.40:0/1617884774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b815cc80 0x7fc8b806efa0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc8b0009d90 tx=0x7fc8b001c800 comp rx=0 tx=0).stop 2024-03-24T00:45:01.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7f5236e00640 1 -- start start 2024-03-24T00:45:01.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7fc8bf000640 1 -- 172.21.15.40:0/1617884774 shutdown_connections 2024-03-24T00:45:01.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7fc8bf000640 1 --2- 172.21.15.40:0/1617884774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b815cc80 0x7fc8b806efa0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7fc8bf000640 1 -- 172.21.15.40:0/1617884774 >> 172.21.15.40:0/1617884774 conn(0x7fc8b80cfc20 msgr2=0x7fc8b80d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:01.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7fc8bf000640 1 -- 172.21.15.40:0/1617884774 shutdown_connections 2024-03-24T00:45:01.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.461+0000 7fc8bf000640 1 -- 172.21.15.40:0/1617884774 wait complete. 2024-03-24T00:45:01.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5236e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52300728d0 0x7f5230159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5236e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f523015bd00 0x7f5230148340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5236e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f523015f8c0 0x7f523015c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5236e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523006d320 con 0x7f523015bd00 2024-03-24T00:45:01.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5236e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523015c0e0 con 0x7f523015f8c0 2024-03-24T00:45:01.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5236e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523006d4a0 con 0x7f52300728d0 2024-03-24T00:45:01.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f522f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f523015bd00 0x7f5230148340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f522f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f523015bd00 0x7f5230148340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34120/0 (socket says 172.21.15.40:34120) 2024-03-24T00:45:01.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f522f400640 1 -- 172.21.15.40:0/1916500386 learned_addr learned my addr 172.21.15.40:0/1916500386 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:01.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5234e00640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f523015f8c0 0x7f523015c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f522fe00640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52300728d0 0x7f5230159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f522f400640 1 -- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52300728d0 msgr2=0x7f5230159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f522f400640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52300728d0 0x7f5230159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f522f400640 1 -- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f523015f8c0 msgr2=0x7f523015c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f522f400640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f523015f8c0 0x7f523015c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f522f400640 1 -- 172.21.15.40:0/1916500386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52300b26b0 con 0x7f523015bd00 2024-03-24T00:45:01.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7fc8bf000640 1 Processor -- start 2024-03-24T00:45:01.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f522f400640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f523015bd00 0x7f5230148340 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f521c0ac040 tx=0x7f521c01ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7fc8bf000640 1 -- start start 2024-03-24T00:45:01.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5227e00640 1 -- 172.21.15.40:0/1916500386 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c0bd3f0 con 0x7f523015bd00 2024-03-24T00:45:01.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5227e00640 1 -- 172.21.15.40:0/1916500386 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c0bd590 con 0x7f523015bd00 2024-03-24T00:45:01.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5227e00640 1 -- 172.21.15.40:0/1916500386 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c0bd940 con 0x7f523015bd00 2024-03-24T00:45:01.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5236e00640 1 -- 172.21.15.40:0/1916500386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52301598e0 con 0x7f523015bd00 2024-03-24T00:45:01.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5236e00640 1 -- 172.21.15.40:0/1916500386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5230159ea0 con 0x7f523015bd00 2024-03-24T00:45:01.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7fc8bf000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b815c190 0x7fc8b815a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7fc8bf000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc8b815cc80 0x7fc8b814c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7fc8bf000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8b815d2e0 0x7fc8b806dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7fc8bf000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b8169a10 con 0x7fc8b815d2e0 2024-03-24T00:45:01.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7fc8bf000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b8169890 con 0x7fc8b815c190 2024-03-24T00:45:01.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7fc8bf000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b8169b90 con 0x7fc8b815cc80 2024-03-24T00:45:01.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7fc8bdc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b815c190 0x7fc8b815a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7fc8bd200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc8b815cc80 0x7fc8b814c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7fc8bdc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b815c190 0x7fc8b815a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:56912/0 (socket says 172.21.15.40:56912) 2024-03-24T00:45:01.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7fc8bdc00640 1 -- 172.21.15.40:0/760711737 learned_addr learned my addr 172.21.15.40:0/760711737 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:01.473 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.465+0000 7f5236e00640 1 -- 172.21.15.40:0/1916500386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5230068170 con 0x7f523015bd00 2024-03-24T00:45:01.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8be600640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8b815d2e0 0x7fc8b806dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8bdc00640 1 -- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc8b815cc80 msgr2=0x7fc8b814c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8bdc00640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc8b815cc80 0x7fc8b814c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8bdc00640 1 -- 172.21.15.40:0/760711737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8b815d2e0 msgr2=0x7fc8b806dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8bdc00640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8b815d2e0 0x7fc8b806dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8bdc00640 1 -- 172.21.15.40:0/760711737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b815a640 con 0x7fc8b815c190 2024-03-24T00:45:01.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8bdc00640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b815c190 0x7fc8b815a100 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00c520 tx=0x7fc8ac00c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8b5600640 1 -- 172.21.15.40:0/760711737 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac017070 con 0x7fc8b815c190 2024-03-24T00:45:01.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8b5600640 1 -- 172.21.15.40:0/760711737 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ac005d10 con 0x7fc8b815c190 2024-03-24T00:45:01.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8bf000640 1 -- 172.21.15.40:0/760711737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b8156170 con 0x7fc8b815c190 2024-03-24T00:45:01.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8b5600640 1 -- 172.21.15.40:0/760711737 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac0072b0 con 0x7fc8b815c190 2024-03-24T00:45:01.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8bf000640 1 -- 172.21.15.40:0/760711737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b8156a10 con 0x7fc8b815c190 2024-03-24T00:45:01.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7f5227e00640 1 -- 172.21.15.40:0/1916500386 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f521c0b33f0 con 0x7f523015bd00 2024-03-24T00:45:01.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7f5227e00640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5210067660 0x7f5210069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.475 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7f5227e00640 1 -- 172.21.15.40:0/1916500386 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f521c11f720 con 0x7f523015bd00 2024-03-24T00:45:01.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7fc8bf000640 1 -- 172.21.15.40:0/760711737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b8157130 con 0x7fc8b815c190 2024-03-24T00:45:01.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.469+0000 7f522fe00640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5210067660 0x7f5210069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.476 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.473+0000 7f522fe00640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5210067660 0x7f5210069b20 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5220009af0 tx=0x7f5220008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.473+0000 7fc8b5600640 1 -- 172.21.15.40:0/760711737 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc8ac01a070 con 0x7fc8b815c190 2024-03-24T00:45:01.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.473+0000 7fc8b5600640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc898067660 0x7fc898069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.473+0000 7fc8b5600640 1 -- 172.21.15.40:0/760711737 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc8ac084e60 con 0x7fc8b815c190 2024-03-24T00:45:01.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.473+0000 7fc8bd200640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc898067660 0x7fc898069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.473+0000 7fc8bd200640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc898067660 0x7fc898069b20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8166900 tx=0x7fc8a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.477+0000 7f5227e00640 1 -- 172.21.15.40:0/1916500386 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f521c0a7070 con 0x7f523015bd00 2024-03-24T00:45:01.481 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.477+0000 7fc8b5600640 1 -- 172.21.15.40:0/760711737 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc8ac055360 con 0x7fc8b815c190 2024-03-24T00:45:01.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.697+0000 7f5236e00640 1 -- 172.21.15.40:0/1916500386 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5230160030 con 0x7f523015bd00 2024-03-24T00:45:01.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.697+0000 7f5227e00640 1 -- 172.21.15.40:0/1916500386 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v122) v1 ==== 104+0+11 (secure 0 0 0) 0x7f521c016110 con 0x7f523015bd00 2024-03-24T00:45:01.701 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:45:01.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.697+0000 7f5236e00640 1 -- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5210067660 msgr2=0x7f5210069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.697+0000 7f5236e00640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5210067660 0x7f5210069b20 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5220009af0 tx=0x7f5220008040 comp rx=0 tx=0).stop 2024-03-24T00:45:01.702 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.697+0000 7f5236e00640 1 -- 172.21.15.40:0/1916500386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f523015bd00 msgr2=0x7f5230148340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.697+0000 7f5236e00640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f523015bd00 0x7f5230148340 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f521c0ac040 tx=0x7f521c01ad30 comp rx=0 tx=0).stop 2024-03-24T00:45:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.697+0000 7f5236e00640 1 -- 172.21.15.40:0/1916500386 shutdown_connections 2024-03-24T00:45:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.697+0000 7f5236e00640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f5210067660 0x7f5210069b20 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.697+0000 7f5236e00640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f523015f8c0 0x7f523015c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.697+0000 7f5236e00640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f523015bd00 0x7f5230148340 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.697+0000 7f5236e00640 1 --2- 172.21.15.40:0/1916500386 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52300728d0 0x7f5230159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.697+0000 7f5236e00640 1 -- 172.21.15.40:0/1916500386 >> 172.21.15.40:0/1916500386 conn(0x7f52300b1660 msgr2=0x7f52300cf910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.701+0000 7f5236e00640 1 -- 172.21.15.40:0/1916500386 shutdown_connections 2024-03-24T00:45:01.703 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.701+0000 7f5236e00640 1 -- 172.21.15.40:0/1916500386 wait complete. 2024-03-24T00:45:01.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.701+0000 7fc8bf000640 1 -- 172.21.15.40:0/760711737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8b804f410 con 0x7fc8b815c190 2024-03-24T00:45:01.706 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8b5600640 1 -- 172.21.15.40:0/760711737 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8ac059400 con 0x7fc8b815c190 2024-03-24T00:45:01.707 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:45:01.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8bf000640 1 -- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc898067660 msgr2=0x7fc898069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8bf000640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc898067660 0x7fc898069b20 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fc8b8166900 tx=0x7fc8a4009040 comp rx=0 tx=0).stop 2024-03-24T00:45:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8bf000640 1 -- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b815c190 msgr2=0x7fc8b815a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8bf000640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b815c190 0x7fc8b815a100 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00c520 tx=0x7fc8ac00c9e0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8bf000640 1 -- 172.21.15.40:0/760711737 shutdown_connections 2024-03-24T00:45:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8bf000640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc898067660 0x7fc898069b20 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8bf000640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc8b815d2e0 0x7fc8b806dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8bf000640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc8b815cc80 0x7fc8b814c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8bf000640 1 --2- 172.21.15.40:0/760711737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc8b815c190 0x7fc8b815a100 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8bf000640 1 -- 172.21.15.40:0/760711737 >> 172.21.15.40:0/760711737 conn(0x7fc8b80cfc20 msgr2=0x7fc8b80af900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8bf000640 1 -- 172.21.15.40:0/760711737 shutdown_connections 2024-03-24T00:45:01.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.705+0000 7fc8bf000640 1 -- 172.21.15.40:0/760711737 wait complete. 2024-03-24T00:45:01.724 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:45:01.738 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:45:01.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.865+0000 7fab4b200640 1 Processor -- start 2024-03-24T00:45:01.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.865+0000 7fab4b200640 1 -- start start 2024-03-24T00:45:01.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.865+0000 7fab4b200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab440d17f0 0x7fab44072580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.869+0000 7fab4b200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab44072ac0 0x7fab4415c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.869+0000 7fab4b200640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4415cdc0 con 0x7fab440cd270 2024-03-24T00:45:01.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.869+0000 7fab4b200640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab4415d380 con 0x7fab44072ac0 2024-03-24T00:45:01.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.869+0000 7fab4b200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab4415d0a0 con 0x7fab440d17f0 2024-03-24T00:45:01.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.869+0000 7fab43e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fab440cd270 0x7fab440cd650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52936/0 (socket says 172.21.15.40:52936) 2024-03-24T00:45:01.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.869+0000 7fab43e00640 1 -- 172.21.15.40:0/3153912966 learned_addr learned my addr 172.21.15.40:0/3153912966 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:45:01.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.869+0000 7fab49200640 1 --2- 172.21.15.40:0/3153912966 >> v2:172.21.15.40:3300/0 conn(0x7fab44072ac0 0x7fab4415c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.869+0000 7fab43400640 1 --2- 172.21.15.40:0/3153912966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab440d17f0 0x7fab44072580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.869+0000 7fab49200640 1 -- 172.21.15.40:0/3153912966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab440d17f0 msgr2=0x7fab44072580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:01.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.869+0000 7fab49200640 1 --2- 172.21.15.40:0/3153912966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab440d17f0 0x7fab44072580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.869+0000 7fab49200640 1 -- 172.21.15.40:0/3153912966 >> v1:172.21.15.40:6789/0 conn(0x7fab440cd270 legacy=0x7fab440cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.869+0000 7fab49200640 1 -- 172.21.15.40:0/3153912966 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab44071d80 con 0x7fab44072ac0 2024-03-24T00:45:01.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.873+0000 7fab49200640 1 --2- 172.21.15.40:0/3153912966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab44072ac0 0x7fab4415c3e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009d90 tx=0x7fab3c01c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545f903e4a710ab6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.873+0000 7fab42a00640 1 -- 172.21.15.40:0/3153912966 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c0b2070 con 0x7fab44072ac0 2024-03-24T00:45:01.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.873+0000 7fab42a00640 1 -- 172.21.15.40:0/3153912966 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3c0a5820 con 0x7fab44072ac0 2024-03-24T00:45:01.876 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.873+0000 7fab42a00640 1 -- 172.21.15.40:0/3153912966 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3c0a5bf0 con 0x7fab44072ac0 2024-03-24T00:45:01.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.873+0000 7fab4b200640 1 -- 172.21.15.40:0/3153912966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab44072ac0 msgr2=0x7fab4415c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.873+0000 7fab4b200640 1 --2- 172.21.15.40:0/3153912966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab44072ac0 0x7fab4415c3e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fab3c009d90 tx=0x7fab3c01c850 comp rx=0 tx=0).stop 2024-03-24T00:45:01.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.873+0000 7fab4b200640 1 -- 172.21.15.40:0/3153912966 shutdown_connections 2024-03-24T00:45:01.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.873+0000 7fab4b200640 1 --2- 172.21.15.40:0/3153912966 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab44072ac0 0x7fab4415c3e0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.873+0000 7fab4b200640 1 --2- 172.21.15.40:0/3153912966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab440d17f0 0x7fab44072580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.873+0000 7fab4b200640 1 -- 172.21.15.40:0/3153912966 >> 172.21.15.40:0/3153912966 conn(0x7fab440d0bb0 msgr2=0x7fab440d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:01.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.873+0000 7fab4b200640 1 -- 172.21.15.40:0/3153912966 shutdown_connections 2024-03-24T00:45:01.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.873+0000 7fab4b200640 1 -- 172.21.15.40:0/3153912966 wait complete. 2024-03-24T00:45:01.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab4b200640 1 Processor -- start 2024-03-24T00:45:01.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab4b200640 1 -- start start 2024-03-24T00:45:01.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab4b200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab44072ac0 0x7fab4414f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab4b200640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab440cd270 0x7fab440b0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab4b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab44156580 0x7fab440b12e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab4b200640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab44168d70 con 0x7fab44072ac0 2024-03-24T00:45:01.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab4b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab440cfaf0 con 0x7fab44156580 2024-03-24T00:45:01.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab4b200640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab44168ef0 con 0x7fab440cd270 2024-03-24T00:45:01.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab43e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab44072ac0 0x7fab4414f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab43e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab44072ac0 0x7fab4414f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34146/0 (socket says 172.21.15.40:34146) 2024-03-24T00:45:01.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab43e00640 1 -- 172.21.15.40:0/2874003638 learned_addr learned my addr 172.21.15.40:0/2874003638 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:01.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab43400640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab440cd270 0x7fab440b0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab49200640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab44156580 0x7fab440b12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab43e00640 1 -- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab440cd270 msgr2=0x7fab440b0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab43e00640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab440cd270 0x7fab440b0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab43e00640 1 -- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab44156580 msgr2=0x7fab440b12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab43e00640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab44156580 0x7fab440b12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab43e00640 1 -- 172.21.15.40:0/2874003638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab441554e0 con 0x7fab44072ac0 2024-03-24T00:45:01.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab49200640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab44156580 0x7fab440b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:01.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab43e00640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab44072ac0 0x7fab4414f3d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fab3400c450 tx=0x7fab3400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab3be00640 1 -- 172.21.15.40:0/2874003638 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34017070 con 0x7fab44072ac0 2024-03-24T00:45:01.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab3be00640 1 -- 172.21.15.40:0/2874003638 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab34005cf0 con 0x7fab44072ac0 2024-03-24T00:45:01.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab4b200640 1 -- 172.21.15.40:0/2874003638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab441557c0 con 0x7fab44072ac0 2024-03-24T00:45:01.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab3be00640 1 -- 172.21.15.40:0/2874003638 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34007290 con 0x7fab44072ac0 2024-03-24T00:45:01.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.877+0000 7fab4b200640 1 -- 172.21.15.40:0/2874003638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab44155d80 con 0x7fab44072ac0 2024-03-24T00:45:01.883 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.881+0000 7fab4b200640 1 -- 172.21.15.40:0/2874003638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab44068170 con 0x7fab44072ac0 2024-03-24T00:45:01.884 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.881+0000 7fab3be00640 1 -- 172.21.15.40:0/2874003638 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fab3401a070 con 0x7fab44072ac0 2024-03-24T00:45:01.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.881+0000 7fab3be00640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab24067660 0x7fab24069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.881+0000 7fab3be00640 1 -- 172.21.15.40:0/2874003638 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fab34084c00 con 0x7fab44072ac0 2024-03-24T00:45:01.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.881+0000 7fab43400640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab24067660 0x7fab24069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.886 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.885+0000 7fab43400640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab24067660 0x7fab24069b20 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fab440d0240 tx=0x7fab30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.889+0000 7fab3be00640 1 -- 172.21.15.40:0/2874003638 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fab3408b080 con 0x7fab44072ac0 2024-03-24T00:45:01.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4652a00640 1 Processor -- start 2024-03-24T00:45:01.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4652a00640 1 -- start start 2024-03-24T00:45:01.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4652a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f464c0cc4c0 0x7f464c0d0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4652a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f464c0d0750 0x7f464c165b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4652a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c0b33e0 con 0x7f464c15f5b0 2024-03-24T00:45:01.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4652a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464c0afa90 con 0x7f464c0cc4c0 2024-03-24T00:45:01.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4652a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464c0b22f0 con 0x7f464c0d0750 2024-03-24T00:45:01.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4650c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f464c0cc4c0 0x7f464c0d0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4651600640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f464c15f5b0 0x7f464c0cfb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52940/0 (socket says 172.21.15.40:52940) 2024-03-24T00:45:01.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4651600640 1 -- 172.21.15.40:0/3893419877 learned_addr learned my addr 172.21.15.40:0/3893419877 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:45:01.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4652000640 1 --2- 172.21.15.40:0/3893419877 >> v2:172.21.15.53:3300/0 conn(0x7f464c0d0750 0x7f464c165b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4650c00640 1 -- 172.21.15.40:0/3893419877 >> v2:172.21.15.53:3300/0 conn(0x7f464c0d0750 msgr2=0x7f464c165b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:01.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4650c00640 1 --2- 172.21.15.40:0/3893419877 >> v2:172.21.15.53:3300/0 conn(0x7f464c0d0750 0x7f464c165b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4650c00640 1 -- 172.21.15.40:0/3893419877 >> v1:172.21.15.40:6789/0 conn(0x7f464c15f5b0 legacy=0x7f464c0cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4650c00640 1 -- 172.21.15.40:0/3893419877 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c0af440 con 0x7f464c0cc4c0 2024-03-24T00:45:01.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f464be00640 1 -- 172.21.15.40:0/3893419877 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257071941 0 0) 0x7f464c0b33e0 con 0x7f464c15f5b0 2024-03-24T00:45:01.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.929+0000 7f4652000640 1 --2- 172.21.15.40:0/3893419877 >> v2:172.21.15.53:3300/0 conn(0x7f464c0d0750 0x7f464c165b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:01.936 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.933+0000 7f4650c00640 1 --2- 172.21.15.40:0/3893419877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f464c0cc4c0 0x7f464c0d0210 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f464c0724d0 tx=0x7f463801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db47e60fad86220b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.937 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.933+0000 7f464be00640 1 -- 172.21.15.40:0/3893419877 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46380b0070 con 0x7f464c0cc4c0 2024-03-24T00:45:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.933+0000 7f464be00640 1 -- 172.21.15.40:0/3893419877 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46380ab600 con 0x7f464c0cc4c0 2024-03-24T00:45:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.933+0000 7f464be00640 1 -- 172.21.15.40:0/3893419877 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46380ab9d0 con 0x7f464c0cc4c0 2024-03-24T00:45:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.933+0000 7f4652a00640 1 -- 172.21.15.40:0/3893419877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f464c0cc4c0 msgr2=0x7f464c0d0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.933+0000 7f4652a00640 1 --2- 172.21.15.40:0/3893419877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f464c0cc4c0 0x7f464c0d0210 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f464c0724d0 tx=0x7f463801c480 comp rx=0 tx=0).stop 2024-03-24T00:45:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.933+0000 7f4652a00640 1 -- 172.21.15.40:0/3893419877 shutdown_connections 2024-03-24T00:45:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.933+0000 7f4652a00640 1 --2- 172.21.15.40:0/3893419877 >> v2:172.21.15.53:3300/0 conn(0x7f464c0d0750 0x7f464c165b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.933+0000 7f4652a00640 1 --2- 172.21.15.40:0/3893419877 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f464c0cc4c0 0x7f464c0d0210 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.933+0000 7f4652a00640 1 -- 172.21.15.40:0/3893419877 >> 172.21.15.40:0/3893419877 conn(0x7f464c0b3620 msgr2=0x7f464c0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:01.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.933+0000 7f4652a00640 1 -- 172.21.15.40:0/3893419877 shutdown_connections 2024-03-24T00:45:01.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.933+0000 7f4652a00640 1 -- 172.21.15.40:0/3893419877 wait complete. 2024-03-24T00:45:01.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4652a00640 1 Processor -- start 2024-03-24T00:45:01.939 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4652a00640 1 -- start start 2024-03-24T00:45:01.941 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4652a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f464c0cc4c0 0x7f464c155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4652a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f464c0d0750 0x7f464c0b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4652a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f464c15f5b0 0x7f464c0caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4652a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c166c30 con 0x7f464c15f5b0 2024-03-24T00:45:01.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4652a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c166ab0 con 0x7f464c0cc4c0 2024-03-24T00:45:01.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4652a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c166db0 con 0x7f464c0d0750 2024-03-24T00:45:01.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4651600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f464c0cc4c0 0x7f464c155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4650c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f464c0d0750 0x7f464c0b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4651600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f464c0cc4c0 0x7f464c155410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:56930/0 (socket says 172.21.15.40:56930) 2024-03-24T00:45:01.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4651600640 1 -- 172.21.15.40:0/3342105310 learned_addr learned my addr 172.21.15.40:0/3342105310 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:01.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4652000640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f464c15f5b0 0x7f464c0caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4651600640 1 -- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f464c0d0750 msgr2=0x7f464c0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4651600640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f464c0d0750 0x7f464c0b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4651600640 1 -- 172.21.15.40:0/3342105310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f464c15f5b0 msgr2=0x7f464c0caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4651600640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f464c15f5b0 0x7f464c0caee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.937+0000 7f4651600640 1 -- 172.21.15.40:0/3342105310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c15a300 con 0x7f464c0cc4c0 2024-03-24T00:45:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.941+0000 7f4651600640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f464c0cc4c0 0x7f464c155410 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f464000c480 tx=0x7f464000c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.941+0000 7f4649600640 1 -- 172.21.15.40:0/3342105310 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640018070 con 0x7f464c0cc4c0 2024-03-24T00:45:01.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.941+0000 7f4649600640 1 -- 172.21.15.40:0/3342105310 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4640007500 con 0x7f464c0cc4c0 2024-03-24T00:45:01.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.941+0000 7f4652a00640 1 -- 172.21.15.40:0/3342105310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464c0b7040 con 0x7f464c0cc4c0 2024-03-24T00:45:01.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.941+0000 7f4649600640 1 -- 172.21.15.40:0/3342105310 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640004a30 con 0x7f464c0cc4c0 2024-03-24T00:45:01.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.941+0000 7f4652a00640 1 -- 172.21.15.40:0/3342105310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464c15a5a0 con 0x7f464c0cc4c0 2024-03-24T00:45:01.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.941+0000 7f4652a00640 1 -- 172.21.15.40:0/3342105310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464c04f410 con 0x7f464c0cc4c0 2024-03-24T00:45:01.945 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.941+0000 7f4649600640 1 -- 172.21.15.40:0/3342105310 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f464001b070 con 0x7f464c0cc4c0 2024-03-24T00:45:01.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.945+0000 7f4649600640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f462c067690 0x7f462c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:01.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.945+0000 7f4649600640 1 -- 172.21.15.40:0/3342105310 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f4640084c50 con 0x7f464c0cc4c0 2024-03-24T00:45:01.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.945+0000 7f4650c00640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f462c067690 0x7f462c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:01.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.945+0000 7f4650c00640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f462c067690 0x7f462c069b50 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f46380ad000 tx=0x7f4638008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:01.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:01.945+0000 7f4649600640 1 -- 172.21.15.40:0/3342105310 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f46400551d0 con 0x7f464c0cc4c0 2024-03-24T00:45:02.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.057+0000 7fab4b200640 1 -- 172.21.15.40:0/2874003638 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fab44067c60 con 0x7fab44072ac0 2024-03-24T00:45:02.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.057+0000 7fab3be00640 1 -- 172.21.15.40:0/2874003638 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v122) v1 ==== 100+0+11 (secure 0 0 0) 0x7fab340557f0 con 0x7fab44072ac0 2024-03-24T00:45:02.061 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:45:02.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.061+0000 7fab4b200640 1 -- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab24067660 msgr2=0x7fab24069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.061+0000 7fab4b200640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab24067660 0x7fab24069b20 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fab440d0240 tx=0x7fab30008040 comp rx=0 tx=0).stop 2024-03-24T00:45:02.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.061+0000 7fab4b200640 1 -- 172.21.15.40:0/2874003638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab44072ac0 msgr2=0x7fab4414f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.061+0000 7fab4b200640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab44072ac0 0x7fab4414f3d0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fab3400c450 tx=0x7fab3400c910 comp rx=0 tx=0).stop 2024-03-24T00:45:02.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.061+0000 7fab4b200640 1 -- 172.21.15.40:0/2874003638 shutdown_connections 2024-03-24T00:45:02.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.061+0000 7fab4b200640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab24067660 0x7fab24069b20 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.061+0000 7fab4b200640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab44156580 0x7fab440b12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.061+0000 7fab4b200640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab440cd270 0x7fab440b0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.061+0000 7fab4b200640 1 --2- 172.21.15.40:0/2874003638 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab44072ac0 0x7fab4414f3d0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.061+0000 7fab4b200640 1 -- 172.21.15.40:0/2874003638 >> 172.21.15.40:0/2874003638 conn(0x7fab440d0bb0 msgr2=0x7fab440cd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:02.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.061+0000 7fab4b200640 1 -- 172.21.15.40:0/2874003638 shutdown_connections 2024-03-24T00:45:02.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.061+0000 7fab4b200640 1 -- 172.21.15.40:0/2874003638 wait complete. 2024-03-24T00:45:02.083 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:45:02.116 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.113+0000 7f4652a00640 1 -- 172.21.15.40:0/3342105310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f464c0b7560 con 0x7f464c0cc4c0 2024-03-24T00:45:02.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.113+0000 7f4649600640 1 -- 172.21.15.40:0/3342105310 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v122) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4640059270 con 0x7f464c0cc4c0 2024-03-24T00:45:02.117 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:45:02.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.117+0000 7f4652a00640 1 -- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f462c067690 msgr2=0x7f462c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.117+0000 7f4652a00640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f462c067690 0x7f462c069b50 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f46380ad000 tx=0x7f4638008040 comp rx=0 tx=0).stop 2024-03-24T00:45:02.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.117+0000 7f4652a00640 1 -- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f464c0cc4c0 msgr2=0x7f464c155410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.117+0000 7f4652a00640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f464c0cc4c0 0x7f464c155410 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f464000c480 tx=0x7f464000c940 comp rx=0 tx=0).stop 2024-03-24T00:45:02.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.117+0000 7f4652a00640 1 -- 172.21.15.40:0/3342105310 shutdown_connections 2024-03-24T00:45:02.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.117+0000 7f4652a00640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f462c067690 0x7f462c069b50 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.117+0000 7f4652a00640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f464c15f5b0 0x7f464c0caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.117+0000 7f4652a00640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f464c0d0750 0x7f464c0b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.117+0000 7f4652a00640 1 --2- 172.21.15.40:0/3342105310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f464c0cc4c0 0x7f464c155410 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.117+0000 7f4652a00640 1 -- 172.21.15.40:0/3342105310 >> 172.21.15.40:0/3342105310 conn(0x7f464c0b3620 msgr2=0x7f464c15ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:02.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.117+0000 7f4652a00640 1 -- 172.21.15.40:0/3342105310 shutdown_connections 2024-03-24T00:45:02.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.117+0000 7f4652a00640 1 -- 172.21.15.40:0/3342105310 wait complete. 2024-03-24T00:45:02.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:45:02.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.225+0000 7fc102000640 1 Processor -- start 2024-03-24T00:45:02.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.225+0000 7fc102000640 1 -- start start 2024-03-24T00:45:02.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.225+0000 7fc102000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0fc15bca0 0x7fc0fc0d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc102000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0fc15c080 0x7fc0fc0d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc102000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc0fc0ce490 0x7fc0fc0ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc102000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0fc0d0780 con 0x7fc0fc15bca0 2024-03-24T00:45:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc102000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0fc0b4a90 con 0x7fc0fc0ce490 2024-03-24T00:45:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc102000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc0b4d70 con 0x7fc0fc15c080 2024-03-24T00:45:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc100c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0fc15bca0 0x7fc0fc0d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc100c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fc0fc15bca0 0x7fc0fc0d2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34182/0 (socket says 172.21.15.40:34182) 2024-03-24T00:45:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc100c00640 1 -- 172.21.15.40:0/1055262469 learned_addr learned my addr 172.21.15.40:0/1055262469 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:02.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc0fbe00640 1 --2- 172.21.15.40:0/1055262469 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0fc15c080 0x7fc0fc0d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc101600640 1 --2- 172.21.15.40:0/1055262469 >> v2:172.21.15.53:3300/0 conn(0x7fc0fc0ce490 0x7fc0fc0ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc100c00640 1 -- 172.21.15.40:0/1055262469 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0fc15c080 msgr2=0x7fc0fc0d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:02.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc100c00640 1 --2- 172.21.15.40:0/1055262469 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0fc15c080 0x7fc0fc0d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc100c00640 1 -- 172.21.15.40:0/1055262469 >> v2:172.21.15.53:3300/0 conn(0x7fc0fc0ce490 msgr2=0x7fc0fc0ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:02.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc100c00640 1 --2- 172.21.15.40:0/1055262469 >> v2:172.21.15.53:3300/0 conn(0x7fc0fc0ce490 0x7fc0fc0ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc100c00640 1 -- 172.21.15.40:0/1055262469 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0fc0b5050 con 0x7fc0fc15bca0 2024-03-24T00:45:02.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.229+0000 7fc100c00640 1 --2- 172.21.15.40:0/1055262469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0fc15bca0 0x7fc0fc0d2590 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc067e30 tx=0x7fc0f001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae4ebeb2ae98811 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:02.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc0fb400640 1 -- 172.21.15.40:0/1055262469 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f00b0070 con 0x7fc0fc15bca0 2024-03-24T00:45:02.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc0fb400640 1 -- 172.21.15.40:0/1055262469 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f00ab600 con 0x7fc0fc15bca0 2024-03-24T00:45:02.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc0fb400640 1 -- 172.21.15.40:0/1055262469 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f00ab9d0 con 0x7fc0fc15bca0 2024-03-24T00:45:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc102000640 1 -- 172.21.15.40:0/1055262469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0fc15bca0 msgr2=0x7fc0fc0d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc102000640 1 --2- 172.21.15.40:0/1055262469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0fc15bca0 0x7fc0fc0d2590 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc067e30 tx=0x7fc0f001c520 comp rx=0 tx=0).stop 2024-03-24T00:45:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc102000640 1 -- 172.21.15.40:0/1055262469 shutdown_connections 2024-03-24T00:45:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc102000640 1 --2- 172.21.15.40:0/1055262469 >> v2:172.21.15.53:3300/0 conn(0x7fc0fc0ce490 0x7fc0fc0ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc102000640 1 --2- 172.21.15.40:0/1055262469 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0fc15c080 0x7fc0fc0d0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc102000640 1 --2- 172.21.15.40:0/1055262469 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0fc15bca0 0x7fc0fc0d2590 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc102000640 1 -- 172.21.15.40:0/1055262469 >> 172.21.15.40:0/1055262469 conn(0x7fc0fc0d1aa0 msgr2=0x7fc0fc0d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc102000640 1 -- 172.21.15.40:0/1055262469 shutdown_connections 2024-03-24T00:45:02.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc102000640 1 -- 172.21.15.40:0/1055262469 wait complete. 2024-03-24T00:45:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.233+0000 7fc102000640 1 Processor -- start 2024-03-24T00:45:02.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc102000640 1 -- start start 2024-03-24T00:45:02.240 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc102000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0ce490 0x7fc0fc15e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc102000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0fc15bca0 0x7fc0fc156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc102000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0fc15c080 0x7fc0fc1502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc102000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc0cd150 con 0x7fc0fc15bca0 2024-03-24T00:45:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc102000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc0ccfd0 con 0x7fc0fc0ce490 2024-03-24T00:45:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc102000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc0cd2d0 con 0x7fc0fc15c080 2024-03-24T00:45:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc100c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0ce490 0x7fc0fc15e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc0fbe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0fc15bca0 0x7fc0fc156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc100c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0ce490 0x7fc0fc15e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:56946/0 (socket says 172.21.15.40:56946) 2024-03-24T00:45:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc100c00640 1 -- 172.21.15.40:0/667001448 learned_addr learned my addr 172.21.15.40:0/667001448 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc101600640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0fc15c080 0x7fc0fc1502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc100c00640 1 -- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0fc15c080 msgr2=0x7fc0fc1502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc100c00640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0fc15c080 0x7fc0fc1502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc100c00640 1 -- 172.21.15.40:0/667001448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0fc15bca0 msgr2=0x7fc0fc156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc100c00640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0fc15bca0 0x7fc0fc156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc100c00640 1 -- 172.21.15.40:0/667001448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0fc156860 con 0x7fc0fc0ce490 2024-03-24T00:45:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc100c00640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0ce490 0x7fc0fc15e860 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc0f001cdc0 tx=0x7fc0f001ad60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc0f8c00640 1 -- 172.21.15.40:0/667001448 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f00ba020 con 0x7fc0fc0ce490 2024-03-24T00:45:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc0f8c00640 1 -- 172.21.15.40:0/667001448 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f00aa7b0 con 0x7fc0fc0ce490 2024-03-24T00:45:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc0f8c00640 1 -- 172.21.15.40:0/667001448 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f00b0040 con 0x7fc0fc0ce490 2024-03-24T00:45:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc102000640 1 -- 172.21.15.40:0/667001448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0fc156b40 con 0x7fc0fc0ce490 2024-03-24T00:45:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.237+0000 7fc102000640 1 -- 172.21.15.40:0/667001448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0fc157100 con 0x7fc0fc0ce490 2024-03-24T00:45:02.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.241+0000 7fc0ee000640 1 -- 172.21.15.40:0/667001448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0fc09fb10 con 0x7fc0fc0ce490 2024-03-24T00:45:02.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.241+0000 7fc0f8c00640 1 -- 172.21.15.40:0/667001448 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc0f00bd020 con 0x7fc0fc0ce490 2024-03-24T00:45:02.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.241+0000 7fc0f8c00640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc0dc067660 0x7fc0dc069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.241+0000 7fc0f8c00640 1 -- 172.21.15.40:0/667001448 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc0f011fe10 con 0x7fc0fc0ce490 2024-03-24T00:45:02.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.241+0000 7fc0fbe00640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc0dc067660 0x7fc0dc069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.245+0000 7fc0fbe00640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc0dc067660 0x7fc0dc069b20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc1f5500 tx=0x7fc0e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:02.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.245+0000 7fc0f8c00640 1 -- 172.21.15.40:0/667001448 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc0f00c4030 con 0x7fc0fc0ce490 2024-03-24T00:45:02.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.325+0000 7f476b000640 1 Processor -- start 2024-03-24T00:45:02.330 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.325+0000 7f476b000640 1 -- start start 2024-03-24T00:45:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.329+0000 7f476b000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f47640b0bd0 0x7f47640cd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.329+0000 7f476b000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47640cf2e0 con 0x7f476415e0e0 2024-03-24T00:45:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.329+0000 7f476b000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47640cc7b0 con 0x7f47640d0070 2024-03-24T00:45:02.331 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.329+0000 7f476b000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47640cd0e0 con 0x7f47640b0bd0 2024-03-24T00:45:02.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.329+0000 7f4769200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f47640b0bd0 0x7f47640cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.329+0000 7f4769200640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f47640b0bd0 0x7f47640cd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34202/0 (socket says 172.21.15.40:34202) 2024-03-24T00:45:02.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.329+0000 7f4769200640 1 -- 172.21.15.40:0/3947760770 learned_addr learned my addr 172.21.15.40:0/3947760770 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:02.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.329+0000 7f4769200640 1 -- 172.21.15.40:0/3947760770 >> v1:172.21.15.53:6789/0 conn(0x7f47640d0070 legacy=0x7f476406b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.329+0000 7f4769200640 1 -- 172.21.15.40:0/3947760770 >> v1:172.21.15.40:6789/0 conn(0x7f476415e0e0 legacy=0x7f47640ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.333 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.329+0000 7f4769200640 1 -- 172.21.15.40:0/3947760770 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47640cca90 con 0x7f47640b0bd0 2024-03-24T00:45:02.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f4769200640 1 --2- 172.21.15.40:0/3947760770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640b0bd0 0x7f47640cd570 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f4750009870 tx=0x7f475001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=742706030b5d0541 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:02.335 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f4763e00640 1 -- 172.21.15.40:0/3947760770 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47500a7070 con 0x7f47640b0bd0 2024-03-24T00:45:02.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f4763e00640 1 -- 172.21.15.40:0/3947760770 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47500b0070 con 0x7f47640b0bd0 2024-03-24T00:45:02.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f4763e00640 1 -- 172.21.15.40:0/3947760770 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47500ab890 con 0x7f47640b0bd0 2024-03-24T00:45:02.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f476b000640 1 -- 172.21.15.40:0/3947760770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640b0bd0 msgr2=0x7f47640cd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.336 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f476b000640 1 --2- 172.21.15.40:0/3947760770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640b0bd0 0x7f47640cd570 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f4750009870 tx=0x7f475001c560 comp rx=0 tx=0).stop 2024-03-24T00:45:02.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f476b000640 1 -- 172.21.15.40:0/3947760770 shutdown_connections 2024-03-24T00:45:02.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f476b000640 1 --2- 172.21.15.40:0/3947760770 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640b0bd0 0x7f47640cd570 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f476b000640 1 -- 172.21.15.40:0/3947760770 >> 172.21.15.40:0/3947760770 conn(0x7f47640b3620 msgr2=0x7f47640b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:02.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f476b000640 1 -- 172.21.15.40:0/3947760770 shutdown_connections 2024-03-24T00:45:02.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f476b000640 1 -- 172.21.15.40:0/3947760770 wait complete. 2024-03-24T00:45:02.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f476b000640 1 Processor -- start 2024-03-24T00:45:02.338 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.333+0000 7f476b000640 1 -- start start 2024-03-24T00:45:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476b000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47640b0bd0 0x7f47640b7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476b000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640d0070 0x7f47640c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476b000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f476415e0e0 0x7f47640b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476b000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476416c7c0 con 0x7f47640d0070 2024-03-24T00:45:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476b000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476416c640 con 0x7f476415e0e0 2024-03-24T00:45:02.339 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476b000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f476416c940 con 0x7f47640b0bd0 2024-03-24T00:45:02.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f4769200640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640d0070 0x7f47640c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476a600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f476415e0e0 0x7f47640b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f4769c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47640b0bd0 0x7f47640b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476a600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f476415e0e0 0x7f47640b5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:56958/0 (socket says 172.21.15.40:56958) 2024-03-24T00:45:02.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476a600640 1 -- 172.21.15.40:0/1874242505 learned_addr learned my addr 172.21.15.40:0/1874242505 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f4769200640 1 -- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47640b0bd0 msgr2=0x7f47640b7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f4769200640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47640b0bd0 0x7f47640b7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f4769200640 1 -- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f476415e0e0 msgr2=0x7f47640b5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f4769200640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f476415e0e0 0x7f47640b5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f4769200640 1 -- 172.21.15.40:0/1874242505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47640b1ef0 con 0x7f47640d0070 2024-03-24T00:45:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476a600640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f476415e0e0 0x7f47640b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:02.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f4769200640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640d0070 0x7f47640c57d0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f47500ac000 tx=0x7f475001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:02.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f4761600640 1 -- 172.21.15.40:0/1874242505 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47500b4800 con 0x7f47640d0070 2024-03-24T00:45:02.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f4761600640 1 -- 172.21.15.40:0/1874242505 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47500b49c0 con 0x7f47640d0070 2024-03-24T00:45:02.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f4761600640 1 -- 172.21.15.40:0/1874242505 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47500b4db0 con 0x7f47640d0070 2024-03-24T00:45:02.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476b000640 1 -- 172.21.15.40:0/1874242505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47640b21d0 con 0x7f47640d0070 2024-03-24T00:45:02.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476b000640 1 -- 172.21.15.40:0/1874242505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47640b2790 con 0x7f47640d0070 2024-03-24T00:45:02.342 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.337+0000 7f476b000640 1 -- 172.21.15.40:0/1874242505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f476404f410 con 0x7f47640d0070 2024-03-24T00:45:02.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.341+0000 7f4761600640 1 -- 172.21.15.40:0/1874242505 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f47500c8410 con 0x7f47640d0070 2024-03-24T00:45:02.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.341+0000 7f4761600640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4744067660 0x7f4744069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.345 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.341+0000 7f4761600640 1 -- 172.21.15.40:0/1874242505 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f47501200f0 con 0x7f47640d0070 2024-03-24T00:45:02.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.341+0000 7f4769c00640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4744067660 0x7f4744069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.346 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.345+0000 7f4769c00640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4744067660 0x7f4744069b20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f476415db00 tx=0x7f4758009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:02.352 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.349+0000 7f4761600640 1 -- 172.21.15.40:0/1874242505 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f4750016150 con 0x7f47640d0070 2024-03-24T00:45:02.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.433+0000 7fc0ee000640 1 -- 172.21.15.40:0/667001448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc0fc0d1880 con 0x7fc0fc0ce490 2024-03-24T00:45:02.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.433+0000 7fc0f8c00640 1 -- 172.21.15.40:0/667001448 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v122) v1 ==== 80+0+122 (secure 0 0 0) 0x7fc0f0016110 con 0x7fc0fc0ce490 2024-03-24T00:45:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.53:6836/2255755838 2024-03-25T00:40:01.907668+0000 2024-03-24T00:45:02.437 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.53:6834/2255755838 2024-03-25T00:40:01.907668+0000 2024-03-24T00:45:02.437 INFO:teuthology.orchestra.run.smithi040.stderr:listed 2 entries 2024-03-24T00:45:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.437+0000 7fc0ee000640 1 -- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc0dc067660 msgr2=0x7fc0dc069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.437+0000 7fc0ee000640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc0dc067660 0x7fc0dc069b20 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc1f5500 tx=0x7fc0e8008040 comp rx=0 tx=0).stop 2024-03-24T00:45:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.437+0000 7fc0ee000640 1 -- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0ce490 msgr2=0x7fc0fc15e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.437+0000 7fc0ee000640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0ce490 0x7fc0fc15e860 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc0f001cdc0 tx=0x7fc0f001ad60 comp rx=0 tx=0).stop 2024-03-24T00:45:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.437+0000 7fc0ee000640 1 -- 172.21.15.40:0/667001448 shutdown_connections 2024-03-24T00:45:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.437+0000 7fc0ee000640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc0dc067660 0x7fc0dc069b20 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.437+0000 7fc0ee000640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc0fc15c080 0x7fc0fc1502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.437+0000 7fc0ee000640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc0fc15bca0 0x7fc0fc156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.437+0000 7fc0ee000640 1 --2- 172.21.15.40:0/667001448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc0fc0ce490 0x7fc0fc15e860 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.437+0000 7fc0ee000640 1 -- 172.21.15.40:0/667001448 >> 172.21.15.40:0/667001448 conn(0x7fc0fc0d1aa0 msgr2=0x7fc0fc0d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.437+0000 7fc0ee000640 1 -- 172.21.15.40:0/667001448 shutdown_connections 2024-03-24T00:45:02.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.437+0000 7fc0ee000640 1 -- 172.21.15.40:0/667001448 wait complete. 2024-03-24T00:45:02.467 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:45:02.467 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:02.476 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:45:02.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.513+0000 7f476b000640 1 -- 172.21.15.40:0/1874242505 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f47640cf6b0 con 0x7f47640d0070 2024-03-24T00:45:02.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.513+0000 7f4761600640 1 -- 172.21.15.40:0/1874242505 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v122) v1 ==== 100+0+11 (secure 0 0 0) 0x7f47500f0820 con 0x7f47640d0070 2024-03-24T00:45:02.516 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:45:02.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.517+0000 7f476b000640 1 -- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4744067660 msgr2=0x7f4744069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.517+0000 7f476b000640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4744067660 0x7f4744069b20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f476415db00 tx=0x7f4758009040 comp rx=0 tx=0).stop 2024-03-24T00:45:02.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.517+0000 7f476b000640 1 -- 172.21.15.40:0/1874242505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640d0070 msgr2=0x7f47640c57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.517+0000 7f476b000640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640d0070 0x7f47640c57d0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f47500ac000 tx=0x7f475001ada0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.517+0000 7f476b000640 1 -- 172.21.15.40:0/1874242505 shutdown_connections 2024-03-24T00:45:02.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.517+0000 7f476b000640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f4744067660 0x7f4744069b20 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.517+0000 7f476b000640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f476415e0e0 0x7f47640b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.517+0000 7f476b000640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f47640d0070 0x7f47640c57d0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.517+0000 7f476b000640 1 --2- 172.21.15.40:0/1874242505 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47640b0bd0 0x7f47640b7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.517+0000 7f476b000640 1 -- 172.21.15.40:0/1874242505 >> 172.21.15.40:0/1874242505 conn(0x7f47640b3620 msgr2=0x7f47640b3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:02.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.517+0000 7f476b000640 1 -- 172.21.15.40:0/1874242505 shutdown_connections 2024-03-24T00:45:02.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.517+0000 7f476b000640 1 -- 172.21.15.40:0/1874242505 wait complete. 2024-03-24T00:45:02.550 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:45:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a24a00640 1 Processor -- start 2024-03-24T00:45:02.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a24a00640 1 -- start start 2024-03-24T00:45:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a24a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a20162350 0x7f7a200ae790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a24a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a200c5180 0x7f7a200cf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a24a00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a200af250 con 0x7f7a200b02f0 2024-03-24T00:45:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a24a00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a200af530 con 0x7f7a20162350 2024-03-24T00:45:02.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a24a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a200af810 con 0x7f7a200c5180 2024-03-24T00:45:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a1e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a20162350 0x7f7a200ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a1e800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7a20162350 0x7f7a200ae790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34234/0 (socket says 172.21.15.40:34234) 2024-03-24T00:45:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a1e800640 1 -- 172.21.15.40:0/1779208028 learned_addr learned my addr 172.21.15.40:0/1779208028 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a1f200640 1 --2- 172.21.15.40:0/1779208028 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a200c5180 0x7f7a200cf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a1e800640 1 -- 172.21.15.40:0/1779208028 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a200c5180 msgr2=0x7f7a200cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a1e800640 1 --2- 172.21.15.40:0/1779208028 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a200c5180 0x7f7a200cf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a1e800640 1 -- 172.21.15.40:0/1779208028 >> v1:172.21.15.40:6789/0 conn(0x7f7a200b02f0 legacy=0x7f7a200c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.661 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.657+0000 7f7a1e800640 1 -- 172.21.15.40:0/1779208028 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a200afaf0 con 0x7f7a20162350 2024-03-24T00:45:02.665 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.661+0000 7f7a1e800640 1 --2- 172.21.15.40:0/1779208028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a20162350 0x7f7a200ae790 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f7a10009870 tx=0x7f7a1001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c85ae0dd6b59b688 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:02.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.661+0000 7f7a1d400640 1 -- 172.21.15.40:0/1779208028 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a100a9610 con 0x7f7a20162350 2024-03-24T00:45:02.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.661+0000 7f7a1d400640 1 -- 172.21.15.40:0/1779208028 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a100a97b0 con 0x7f7a20162350 2024-03-24T00:45:02.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.661+0000 7f7a1d400640 1 -- 172.21.15.40:0/1779208028 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a100a9b80 con 0x7f7a20162350 2024-03-24T00:45:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 -- 172.21.15.40:0/1779208028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a20162350 msgr2=0x7f7a200ae790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 --2- 172.21.15.40:0/1779208028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a20162350 0x7f7a200ae790 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f7a10009870 tx=0x7f7a1001c530 comp rx=0 tx=0).stop 2024-03-24T00:45:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 -- 172.21.15.40:0/1779208028 shutdown_connections 2024-03-24T00:45:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 --2- 172.21.15.40:0/1779208028 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a200c5180 0x7f7a200cf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 --2- 172.21.15.40:0/1779208028 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a20162350 0x7f7a200ae790 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 -- 172.21.15.40:0/1779208028 >> 172.21.15.40:0/1779208028 conn(0x7f7a200d1aa0 msgr2=0x7f7a200d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 -- 172.21.15.40:0/1779208028 shutdown_connections 2024-03-24T00:45:02.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 -- 172.21.15.40:0/1779208028 wait complete. 2024-03-24T00:45:02.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 Processor -- start 2024-03-24T00:45:02.669 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 -- start start 2024-03-24T00:45:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a200b02f0 0x7f7a200b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200c5180 0x7f7a200b6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a200b7250 0x7f7a20157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2016f0f0 con 0x7f7a200c5180 2024-03-24T00:45:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2016ef70 con 0x7f7a200b02f0 2024-03-24T00:45:02.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a24a00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2016f270 con 0x7f7a200b7250 2024-03-24T00:45:02.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a1de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200c5180 0x7f7a200b6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a1de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200c5180 0x7f7a200b6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34236/0 (socket says 172.21.15.40:34236) 2024-03-24T00:45:02.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a1de00640 1 -- 172.21.15.40:0/3110590993 learned_addr learned my addr 172.21.15.40:0/3110590993 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:02.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.665+0000 7f7a1e800640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a200b02f0 0x7f7a200b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a1f200640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a200b7250 0x7f7a20157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a1de00640 1 -- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a200b7250 msgr2=0x7f7a20157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a1de00640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a200b7250 0x7f7a20157be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a1de00640 1 -- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a200b02f0 msgr2=0x7f7a200b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a1de00640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a200b02f0 0x7f7a200b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a1de00640 1 -- 172.21.15.40:0/3110590993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a200b47f0 con 0x7f7a200c5180 2024-03-24T00:45:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a1de00640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200c5180 0x7f7a200b6d10 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00c420 tx=0x7f7a0c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a16a00640 1 -- 172.21.15.40:0/3110590993 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0052d0 con 0x7f7a200c5180 2024-03-24T00:45:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a16a00640 1 -- 172.21.15.40:0/3110590993 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0c005470 con 0x7f7a200c5180 2024-03-24T00:45:02.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a24a00640 1 -- 172.21.15.40:0/3110590993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a200b4ad0 con 0x7f7a200c5180 2024-03-24T00:45:02.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a16a00640 1 -- 172.21.15.40:0/3110590993 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c007260 con 0x7f7a200c5180 2024-03-24T00:45:02.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a24a00640 1 -- 172.21.15.40:0/3110590993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a200b5090 con 0x7f7a200c5180 2024-03-24T00:45:02.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.669+0000 7f7a24a00640 1 -- 172.21.15.40:0/3110590993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a200b5470 con 0x7f7a200c5180 2024-03-24T00:45:02.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.673+0000 7f7a16a00640 1 -- 172.21.15.40:0/3110590993 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f7a0c007400 con 0x7f7a200c5180 2024-03-24T00:45:02.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.673+0000 7f7a16a00640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a00067660 0x7f7a00069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.673+0000 7f7a16a00640 1 -- 172.21.15.40:0/3110590993 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f7a0c084d90 con 0x7f7a200c5180 2024-03-24T00:45:02.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.673+0000 7f7a1e800640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a00067660 0x7f7a00069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.676 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.673+0000 7f7a1e800640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a00067660 0x7f7a00069b20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7a100ae000 tx=0x7f7a10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:02.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.681+0000 7f7a16a00640 1 -- 172.21.15.40:0/3110590993 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f7a200b5470 con 0x7f7a200c5180 2024-03-24T00:45:02.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f5416600640 1 Processor -- start 2024-03-24T00:45:02.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f5416600640 1 -- start start 2024-03-24T00:45:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f5416600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f541015c080 0x7f54100d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f5416600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54100d0780 con 0x7f541015bca0 2024-03-24T00:45:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f5416600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54100b4a90 con 0x7f54100ceba0 2024-03-24T00:45:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f5416600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54100b4d70 con 0x7f541015c080 2024-03-24T00:45:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f5415200640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f541015bca0 0x7f54100ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52962/0 (socket says 172.21.15.40:52962) 2024-03-24T00:45:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f5415200640 1 -- 172.21.15.40:0/314151483 learned_addr learned my addr 172.21.15.40:0/314151483 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:45:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f5415c00640 1 --2- 172.21.15.40:0/314151483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f541015c080 0x7f54100d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f540f400640 1 -- 172.21.15.40:0/314151483 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178202807 0 0) 0x7f54100d0780 con 0x7f541015bca0 2024-03-24T00:45:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f540f400640 1 -- 172.21.15.40:0/314151483 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f8003530 con 0x7f541015bca0 2024-03-24T00:45:02.741 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f540f400640 1 -- 172.21.15.40:0/314151483 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826808136 0 0) 0x7f54100b4a90 con 0x7f54100ceba0 2024-03-24T00:45:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f540f400640 1 -- 172.21.15.40:0/314151483 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54100d0780 con 0x7f54100ceba0 2024-03-24T00:45:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f5415c00640 1 -- 172.21.15.40:0/314151483 >> v1:172.21.15.53:6789/0 conn(0x7f54100ceba0 legacy=0x7f54100b5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f5415c00640 1 -- 172.21.15.40:0/314151483 >> v1:172.21.15.40:6789/0 conn(0x7f541015bca0 legacy=0x7f54100ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.737+0000 7f5415c00640 1 -- 172.21.15.40:0/314151483 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54100b5050 con 0x7f541015c080 2024-03-24T00:45:02.745 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.741+0000 7f5415c00640 1 --2- 172.21.15.40:0/314151483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f541015c080 0x7f54100d2590 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5408009dc0 tx=0x7f540801c9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a1c43e1e2e997f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.741+0000 7f540f400640 1 -- 172.21.15.40:0/314151483 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54080b2070 con 0x7f541015c080 2024-03-24T00:45:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.741+0000 7f540f400640 1 -- 172.21.15.40:0/314151483 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54080a5960 con 0x7f541015c080 2024-03-24T00:45:02.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.741+0000 7f540f400640 1 -- 172.21.15.40:0/314151483 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54080a5d30 con 0x7f541015c080 2024-03-24T00:45:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.745+0000 7f5416600640 1 -- 172.21.15.40:0/314151483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f541015c080 msgr2=0x7f54100d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.745+0000 7f5416600640 1 --2- 172.21.15.40:0/314151483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f541015c080 0x7f54100d2590 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5408009dc0 tx=0x7f540801c9b0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.745+0000 7f5416600640 1 -- 172.21.15.40:0/314151483 shutdown_connections 2024-03-24T00:45:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.745+0000 7f5416600640 1 --2- 172.21.15.40:0/314151483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f541015c080 0x7f54100d2590 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.745+0000 7f5416600640 1 -- 172.21.15.40:0/314151483 >> 172.21.15.40:0/314151483 conn(0x7f54100d1aa0 msgr2=0x7f54100d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.745+0000 7f5416600640 1 -- 172.21.15.40:0/314151483 shutdown_connections 2024-03-24T00:45:02.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.745+0000 7f5416600640 1 -- 172.21.15.40:0/314151483 wait complete. 2024-03-24T00:45:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.745+0000 7f5416600640 1 Processor -- start 2024-03-24T00:45:02.749 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.745+0000 7f5416600640 1 -- start start 2024-03-24T00:45:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f5416600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54100ceba0 0x7f541014ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f5416600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f541015bca0 0x7f5410146e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f5416600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f541015c080 0x7f5410150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f5416600640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54100b6820 con 0x7f541015bca0 2024-03-24T00:45:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f5416600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54100b66a0 con 0x7f54100ceba0 2024-03-24T00:45:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f5416600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5410169a50 con 0x7f541015c080 2024-03-24T00:45:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f540fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f541015bca0 0x7f5410146e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f540fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f541015bca0 0x7f5410146e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34248/0 (socket says 172.21.15.40:34248) 2024-03-24T00:45:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f540fe00640 1 -- 172.21.15.40:0/1615414283 learned_addr learned my addr 172.21.15.40:0/1615414283 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f5415200640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54100ceba0 0x7f541014ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f5415c00640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f541015c080 0x7f5410150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.753 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f540fe00640 1 -- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f541015c080 msgr2=0x7f5410150a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f540fe00640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f541015c080 0x7f5410150a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f540fe00640 1 -- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54100ceba0 msgr2=0x7f541014ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f540fe00640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54100ceba0 0x7f541014ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f540fe00640 1 -- 172.21.15.40:0/1615414283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54101533d0 con 0x7f541015bca0 2024-03-24T00:45:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f540fe00640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f541015bca0 0x7f5410146e80 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00c490 tx=0x7f53fc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f540cc00640 1 -- 172.21.15.40:0/1615414283 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0052d0 con 0x7f541015bca0 2024-03-24T00:45:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f540cc00640 1 -- 172.21.15.40:0/1615414283 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53fc005470 con 0x7f541015bca0 2024-03-24T00:45:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f5416600640 1 -- 172.21.15.40:0/1615414283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541014e370 con 0x7f541015bca0 2024-03-24T00:45:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f540cc00640 1 -- 172.21.15.40:0/1615414283 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc012930 con 0x7f541015bca0 2024-03-24T00:45:02.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f5416600640 1 -- 172.21.15.40:0/1615414283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541014e930 con 0x7f541015bca0 2024-03-24T00:45:02.755 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.749+0000 7f5416600640 1 -- 172.21.15.40:0/1615414283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541009fb10 con 0x7f541015bca0 2024-03-24T00:45:02.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.753+0000 7f540cc00640 1 -- 172.21.15.40:0/1615414283 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f53fc012ad0 con 0x7f541015bca0 2024-03-24T00:45:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.753+0000 7f540cc00640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53f0067660 0x7f53f0069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.753+0000 7f540cc00640 1 -- 172.21.15.40:0/1615414283 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f53fc085550 con 0x7f541015bca0 2024-03-24T00:45:02.757 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.753+0000 7f5415200640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53f0067660 0x7f53f0069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:02.758 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.757+0000 7f5415200640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53f0067660 0x7f53f0069b20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f54040011f0 tx=0x7f5404007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:02.765 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.761+0000 7f540cc00640 1 -- 172.21.15.40:0/1615414283 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f53fc055a50 con 0x7f541015bca0 2024-03-24T00:45:02.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.849+0000 7f7a24a00640 1 -- 172.21.15.40:0/3110590993 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a2009fb10 con 0x7f7a200c5180 2024-03-24T00:45:02.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a16a00640 1 -- 172.21.15.40:0/3110590993 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+16789 (secure 0 0 0) 0x7f7a0c055320 con 0x7f7a200c5180 2024-03-24T00:45:02.855 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:45:02.855 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:40:01.907736+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:05.964092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:06.525558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.c","mappings":[{"from":2,"to":0}]},{"pgid":"19.23","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6836/2255755838":"2024-03-25T00:40:01.907668+0000","172.21.15.53:6834/2255755838":"2024-03-25T00:40:01.907668+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:45:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a24a00640 1 -- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a00067660 msgr2=0x7f7a00069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a24a00640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a00067660 0x7f7a00069b20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7a100ae000 tx=0x7f7a10009040 comp rx=0 tx=0).stop 2024-03-24T00:45:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a24a00640 1 -- 172.21.15.40:0/3110590993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200c5180 msgr2=0x7f7a200b6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a24a00640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200c5180 0x7f7a200b6d10 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00c420 tx=0x7f7a0c00c8e0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a24a00640 1 -- 172.21.15.40:0/3110590993 shutdown_connections 2024-03-24T00:45:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a24a00640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f7a00067660 0x7f7a00069b20 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a24a00640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a200b7250 0x7f7a20157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a24a00640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f7a200c5180 0x7f7a200b6d10 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a24a00640 1 --2- 172.21.15.40:0/3110590993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a200b02f0 0x7f7a200b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a24a00640 1 -- 172.21.15.40:0/3110590993 >> 172.21.15.40:0/3110590993 conn(0x7f7a200d1aa0 msgr2=0x7f7a2015c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a24a00640 1 -- 172.21.15.40:0/3110590993 shutdown_connections 2024-03-24T00:45:02.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.853+0000 7f7a24a00640 1 -- 172.21.15.40:0/3110590993 wait complete. 2024-03-24T00:45:02.876 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:39:05.964092+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:39:06.525558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:45:02.877 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:45:02.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.941+0000 7f5416600640 1 -- 172.21.15.40:0/1615414283 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f54100c8900 con 0x7f541015bca0 2024-03-24T00:45:02.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.941+0000 7f540cc00640 1 -- 172.21.15.40:0/1615414283 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v122) v1 ==== 80+0+122 (secure 0 0 0) 0x7f53fc059af0 con 0x7f541015bca0 2024-03-24T00:45:02.944 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.53:6836/2255755838 2024-03-25T00:40:01.907668+0000 2024-03-24T00:45:02.945 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.53:6834/2255755838 2024-03-25T00:40:01.907668+0000 2024-03-24T00:45:02.945 INFO:teuthology.orchestra.run.smithi040.stderr:listed 2 entries 2024-03-24T00:45:02.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.945+0000 7f5416600640 1 -- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53f0067660 msgr2=0x7f53f0069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.945+0000 7f5416600640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53f0067660 0x7f53f0069b20 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f54040011f0 tx=0x7f5404007d40 comp rx=0 tx=0).stop 2024-03-24T00:45:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.945+0000 7f5416600640 1 -- 172.21.15.40:0/1615414283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f541015bca0 msgr2=0x7f5410146e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.945+0000 7f5416600640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f541015bca0 0x7f5410146e80 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f53fc00c490 tx=0x7f53fc00c950 comp rx=0 tx=0).stop 2024-03-24T00:45:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.945+0000 7f5416600640 1 -- 172.21.15.40:0/1615414283 shutdown_connections 2024-03-24T00:45:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.945+0000 7f5416600640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f53f0067660 0x7f53f0069b20 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.945+0000 7f5416600640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f541015c080 0x7f5410150a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.945+0000 7f5416600640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f541015bca0 0x7f5410146e80 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.945+0000 7f5416600640 1 --2- 172.21.15.40:0/1615414283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f54100ceba0 0x7f541014ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.945+0000 7f5416600640 1 -- 172.21.15.40:0/1615414283 >> 172.21.15.40:0/1615414283 conn(0x7f54100d1aa0 msgr2=0x7f5410156bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.945+0000 7f5416600640 1 -- 172.21.15.40:0/1615414283 shutdown_connections 2024-03-24T00:45:02.947 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:02.945+0000 7f5416600640 1 -- 172.21.15.40:0/1615414283 wait complete. 2024-03-24T00:45:02.967 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi040.front.sepia.ceph.com... 2024-03-24T00:45:02.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:02.967 DEBUG:teuthology.orchestra.run.smithi040:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-24T00:45:02.989 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:45:02.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:02.990 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T00:45:03.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.021+0000 7fc25d000640 1 Processor -- start 2024-03-24T00:45:03.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.021+0000 7fc25d000640 1 -- start start 2024-03-24T00:45:03.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.021+0000 7fc25d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc258160080 0x7fc258168ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.021+0000 7fc25d000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2580cfcb0 con 0x7fc25815e170 2024-03-24T00:45:03.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.021+0000 7fc25d000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2580d1120 con 0x7fc25806f7b0 2024-03-24T00:45:03.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.021+0000 7fc25d000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc25806fc00 con 0x7fc258160080 2024-03-24T00:45:03.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.021+0000 7fc255e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fc25815e170 0x7fc2580d0a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52968/0 (socket says 172.21.15.40:52968) 2024-03-24T00:45:03.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.021+0000 7fc255e00640 1 -- 172.21.15.40:0/3441604248 learned_addr learned my addr 172.21.15.40:0/3441604248 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:45:03.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.021+0000 7fc256800640 1 --2- 172.21.15.40:0/3441604248 >> v2:172.21.15.53:3300/0 conn(0x7fc258160080 0x7fc258168ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.021+0000 7fc255400640 1 -- 172.21.15.40:0/3441604248 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920280067 0 0) 0x7fc248001120 con 0x7fc25815e170 2024-03-24T00:45:03.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.021+0000 7fc255400640 1 -- 172.21.15.40:0/3441604248 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc23c003530 con 0x7fc25815e170 2024-03-24T00:45:03.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.025+0000 7fc256800640 1 -- 172.21.15.40:0/3441604248 >> v1:172.21.15.53:6789/0 conn(0x7fc25806f7b0 legacy=0x7fc25815f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.025+0000 7fc256800640 1 -- 172.21.15.40:0/3441604248 >> v1:172.21.15.40:6789/0 conn(0x7fc25815e170 legacy=0x7fc2580d0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.025+0000 7fc256800640 1 -- 172.21.15.40:0/3441604248 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25806fe20 con 0x7fc258160080 2024-03-24T00:45:03.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc256800640 1 --2- 172.21.15.40:0/3441604248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc258160080 0x7fc258168ea0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc250009d90 tx=0x7fc25001f170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f944dc544aaabfc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc255400640 1 -- 172.21.15.40:0/3441604248 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2500b4070 con 0x7fc258160080 2024-03-24T00:45:03.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc255400640 1 -- 172.21.15.40:0/3441604248 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2500af600 con 0x7fc258160080 2024-03-24T00:45:03.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc255400640 1 -- 172.21.15.40:0/3441604248 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2500af9d0 con 0x7fc258160080 2024-03-24T00:45:03.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc25d000640 1 -- 172.21.15.40:0/3441604248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc258160080 msgr2=0x7fc258168ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc25d000640 1 --2- 172.21.15.40:0/3441604248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc258160080 0x7fc258168ea0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc250009d90 tx=0x7fc25001f170 comp rx=0 tx=0).stop 2024-03-24T00:45:03.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc25d000640 1 -- 172.21.15.40:0/3441604248 shutdown_connections 2024-03-24T00:45:03.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc25d000640 1 --2- 172.21.15.40:0/3441604248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc258160080 0x7fc258168ea0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc25d000640 1 -- 172.21.15.40:0/3441604248 >> 172.21.15.40:0/3441604248 conn(0x7fc2580b36a0 msgr2=0x7fc2580b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc25d000640 1 -- 172.21.15.40:0/3441604248 shutdown_connections 2024-03-24T00:45:03.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc25d000640 1 -- 172.21.15.40:0/3441604248 wait complete. 2024-03-24T00:45:03.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc25d000640 1 Processor -- start 2024-03-24T00:45:03.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.029+0000 7fc25d000640 1 -- start start 2024-03-24T00:45:03.035 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:45:03.035 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:03.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc25d000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc25806f7b0 0x7fc2580c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc25d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc25815e170 0x7fc2580b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc25d000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258160080 0x7fc2580b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc25d000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2580d0210 con 0x7fc258160080 2024-03-24T00:45:03.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc25d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc258070100 con 0x7fc25815e170 2024-03-24T00:45:03.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc25d000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2580d04f0 con 0x7fc25806f7b0 2024-03-24T00:45:03.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc256800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258160080 0x7fc2580b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc256800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258160080 0x7fc2580b5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34260/0 (socket says 172.21.15.40:34260) 2024-03-24T00:45:03.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc256800640 1 -- 172.21.15.40:0/2505306812 learned_addr learned my addr 172.21.15.40:0/2505306812 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc255e00640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc25806f7b0 0x7fc2580c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc24fe00640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc25815e170 0x7fc2580b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc256800640 1 -- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc25806f7b0 msgr2=0x7fc2580c6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc256800640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc25806f7b0 0x7fc2580c6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc256800640 1 -- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc25815e170 msgr2=0x7fc2580b4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc256800640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc25815e170 0x7fc2580b4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc256800640 1 -- 172.21.15.40:0/2505306812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc258159c80 con 0x7fc258160080 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc255e00640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc25806f7b0 0x7fc2580c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc24fe00640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc25815e170 0x7fc2580b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc256800640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258160080 0x7fc2580b5290 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fc250009d90 tx=0x7fc25001bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc24e000640 1 -- 172.21.15.40:0/2505306812 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2500b4040 con 0x7fc258160080 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc24e000640 1 -- 172.21.15.40:0/2505306812 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2500c1070 con 0x7fc258160080 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc25d000640 1 -- 172.21.15.40:0/2505306812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc258159f60 con 0x7fc258160080 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc24e000640 1 -- 172.21.15.40:0/2505306812 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250009450 con 0x7fc258160080 2024-03-24T00:45:03.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc25d000640 1 -- 172.21.15.40:0/2505306812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc258159910 con 0x7fc258160080 2024-03-24T00:45:03.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.033+0000 7fc25d000640 1 -- 172.21.15.40:0/2505306812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc258068170 con 0x7fc258160080 2024-03-24T00:45:03.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.037+0000 7fc24e000640 1 -- 172.21.15.40:0/2505306812 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fc2500ba6b0 con 0x7fc258160080 2024-03-24T00:45:03.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.037+0000 7fc24e000640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc234067660 0x7fc234069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.037+0000 7fc24e000640 1 -- 172.21.15.40:0/2505306812 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fc250124ce0 con 0x7fc258160080 2024-03-24T00:45:03.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.037+0000 7fc255e00640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc234067660 0x7fc234069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.037+0000 7fc255e00640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc234067660 0x7fc234069b20 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc2480012c0 tx=0x7fc248009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.041+0000 7fc24e000640 1 -- 172.21.15.40:0/2505306812 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fc250018cd0 con 0x7fc258160080 2024-03-24T00:45:03.047 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:45:03.048 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:45:03.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:03.048 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T00:45:03.099 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T00:45:03.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:45:03.100 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:45:03.100 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:03.109 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:45:03.109 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:45:03.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:03.109 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T00:45:03.159 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T00:45:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:45:03.160 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:45:03.160 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:03.169 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:45:03.170 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:45:03.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:03.170 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T00:45:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.201+0000 7fc25d000640 1 -- 172.21.15.40:0/2505306812 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc258075420 con 0x7fc258160080 2024-03-24T00:45:03.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc24e000640 1 -- 172.21.15.40:0/2505306812 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc2500f58b0 con 0x7fc258160080 2024-03-24T00:45:03.207 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:45:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc25d000640 1 -- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc234067660 msgr2=0x7fc234069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc25d000640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc234067660 0x7fc234069b20 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc2480012c0 tx=0x7fc248009040 comp rx=0 tx=0).stop 2024-03-24T00:45:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc25d000640 1 -- 172.21.15.40:0/2505306812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258160080 msgr2=0x7fc2580b5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc25d000640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258160080 0x7fc2580b5290 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fc250009d90 tx=0x7fc25001bf10 comp rx=0 tx=0).stop 2024-03-24T00:45:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc25d000640 1 -- 172.21.15.40:0/2505306812 shutdown_connections 2024-03-24T00:45:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc25d000640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fc234067660 0x7fc234069b20 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc25d000640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fc258160080 0x7fc2580b5290 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc25d000640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc25815e170 0x7fc2580b4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc25d000640 1 --2- 172.21.15.40:0/2505306812 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc25806f7b0 0x7fc2580c6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc25d000640 1 -- 172.21.15.40:0/2505306812 >> 172.21.15.40:0/2505306812 conn(0x7fc2580b36a0 msgr2=0x7fc258160490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc25d000640 1 -- 172.21.15.40:0/2505306812 shutdown_connections 2024-03-24T00:45:03.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.205+0000 7fc25d000640 1 -- 172.21.15.40:0/2505306812 wait complete. 2024-03-24T00:45:03.223 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T00:45:03.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:45:03.224 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:45:03.229 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:45:03.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.365+0000 7f870b800640 1 Processor -- start 2024-03-24T00:45:03.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.365+0000 7f870b800640 1 -- start start 2024-03-24T00:45:03.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.365+0000 7f870b800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f870406e620 0x7f87040b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.365+0000 7f870b800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87040ccfa0 0x7f87040cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f870b800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8704070f60 con 0x7f870406ea00 2024-03-24T00:45:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f870b800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87040b4c00 con 0x7f870406e620 2024-03-24T00:45:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f870b800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87040b3620 con 0x7f87040ccfa0 2024-03-24T00:45:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f8703e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f870406ea00 0x7f87040b2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52976/0 (socket says 172.21.15.40:52976) 2024-03-24T00:45:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f8703e00640 1 -- 172.21.15.40:0/3795521667 learned_addr learned my addr 172.21.15.40:0/3795521667 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:45:03.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f8708e00640 1 --2- 172.21.15.40:0/3795521667 >> v2:172.21.15.53:3300/0 conn(0x7f870406e620 0x7f87040b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f8709800640 1 --2- 172.21.15.40:0/3795521667 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87040ccfa0 0x7f87040cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f8703400640 1 -- 172.21.15.40:0/3795521667 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228003327 0 0) 0x7f8704070f60 con 0x7f870406ea00 2024-03-24T00:45:03.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f8703400640 1 -- 172.21.15.40:0/3795521667 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86ec003530 con 0x7f870406ea00 2024-03-24T00:45:03.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f8708e00640 1 -- 172.21.15.40:0/3795521667 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87040ccfa0 msgr2=0x7f87040cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:03.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f8708e00640 1 --2- 172.21.15.40:0/3795521667 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87040ccfa0 0x7f87040cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f8708e00640 1 -- 172.21.15.40:0/3795521667 >> v1:172.21.15.40:6789/0 conn(0x7f870406ea00 legacy=0x7f87040b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f8708e00640 1 -- 172.21.15.40:0/3795521667 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87040b3900 con 0x7f870406e620 2024-03-24T00:45:03.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.369+0000 7f8709800640 1 --2- 172.21.15.40:0/3795521667 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87040ccfa0 0x7f87040cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:03.374 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f8708e00640 1 --2- 172.21.15.40:0/3795521667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f870406e620 0x7f87040b46c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f86f80097c0 tx=0x7f86f801c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cd6d89d86f3f7bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f8703400640 1 -- 172.21.15.40:0/3795521667 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f80b0070 con 0x7f870406e620 2024-03-24T00:45:03.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f8703400640 1 -- 172.21.15.40:0/3795521667 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f80ab600 con 0x7f870406e620 2024-03-24T00:45:03.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f8703400640 1 -- 172.21.15.40:0/3795521667 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f80ab9b0 con 0x7f870406e620 2024-03-24T00:45:03.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f870b800640 1 -- 172.21.15.40:0/3795521667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f870406e620 msgr2=0x7f87040b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f870b800640 1 --2- 172.21.15.40:0/3795521667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f870406e620 0x7f87040b46c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f86f80097c0 tx=0x7f86f801c3e0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f870b800640 1 -- 172.21.15.40:0/3795521667 shutdown_connections 2024-03-24T00:45:03.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f870b800640 1 --2- 172.21.15.40:0/3795521667 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f87040ccfa0 0x7f87040cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f870b800640 1 --2- 172.21.15.40:0/3795521667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f870406e620 0x7f87040b46c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f870b800640 1 -- 172.21.15.40:0/3795521667 >> 172.21.15.40:0/3795521667 conn(0x7f87040b17b0 msgr2=0x7f87040b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f870b800640 1 -- 172.21.15.40:0/3795521667 shutdown_connections 2024-03-24T00:45:03.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f870b800640 1 -- 172.21.15.40:0/3795521667 wait complete. 2024-03-24T00:45:03.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f870b800640 1 Processor -- start 2024-03-24T00:45:03.377 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.373+0000 7f870b800640 1 -- start start 2024-03-24T00:45:03.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f870b800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f870406e620 0x7f8704159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f870b800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f870406ea00 0x7f8704159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f870b800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87040ccfa0 0x7f87040ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f870b800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8704144920 con 0x7f870406ea00 2024-03-24T00:45:03.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f870b800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870406ee10 con 0x7f87040ccfa0 2024-03-24T00:45:03.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f870b800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870415ccc0 con 0x7f870406e620 2024-03-24T00:45:03.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8703e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f870406ea00 0x7f8704159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8703e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f870406ea00 0x7f8704159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34268/0 (socket says 172.21.15.40:34268) 2024-03-24T00:45:03.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8703e00640 1 -- 172.21.15.40:0/3450748446 learned_addr learned my addr 172.21.15.40:0/3450748446 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:03.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8708e00640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f870406e620 0x7f8704159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8709800640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87040ccfa0 0x7f87040ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8703e00640 1 -- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f870406e620 msgr2=0x7f8704159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8703e00640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f870406e620 0x7f8704159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8703e00640 1 -- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87040ccfa0 msgr2=0x7f87040ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8703e00640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87040ccfa0 0x7f87040ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8703e00640 1 -- 172.21.15.40:0/3450748446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8704160d50 con 0x7f870406ea00 2024-03-24T00:45:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8703e00640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f870406ea00 0x7f8704159300 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f86f00010f0 tx=0x7f86f000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8700c00640 1 -- 172.21.15.40:0/3450748446 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f0017070 con 0x7f870406ea00 2024-03-24T00:45:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8700c00640 1 -- 172.21.15.40:0/3450748446 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f0005d10 con 0x7f870406ea00 2024-03-24T00:45:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f870b800640 1 -- 172.21.15.40:0/3450748446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8704161030 con 0x7f870406ea00 2024-03-24T00:45:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f8700c00640 1 -- 172.21.15.40:0/3450748446 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f0007310 con 0x7f870406ea00 2024-03-24T00:45:03.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f870b800640 1 -- 172.21.15.40:0/3450748446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8704161310 con 0x7f870406ea00 2024-03-24T00:45:03.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.377+0000 7f870b800640 1 -- 172.21.15.40:0/3450748446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8704068170 con 0x7f870406ea00 2024-03-24T00:45:03.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.381+0000 7f8700c00640 1 -- 172.21.15.40:0/3450748446 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f86f001a070 con 0x7f870406ea00 2024-03-24T00:45:03.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.381+0000 7f8700c00640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f86e4067690 0x7f86e4069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.381+0000 7f8700c00640 1 -- 172.21.15.40:0/3450748446 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f86f0084f30 con 0x7f870406ea00 2024-03-24T00:45:03.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.381+0000 7f8708e00640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f86e4067690 0x7f86e4069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.385+0000 7f8708e00640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f86e4067690 0x7f86e4069b50 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f86f80ad000 tx=0x7f86f80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.389+0000 7f8700c00640 1 -- 172.21.15.40:0/3450748446 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f86f0021020 con 0x7f870406ea00 2024-03-24T00:45:03.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.429+0000 7f3e9fe00640 1 Processor -- start 2024-03-24T00:45:03.434 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.429+0000 7f3e9fe00640 1 -- start start 2024-03-24T00:45:03.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e9fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3e9815bd00 0x7f3e980ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e9fe00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e98072d60 con 0x7f3e980728d0 2024-03-24T00:45:03.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e9fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e98071de0 con 0x7f3e9815ffd0 2024-03-24T00:45:03.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e9fe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e980cea50 con 0x7f3e9815bd00 2024-03-24T00:45:03.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e9ca00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f3e980728d0 0x7f3e9815f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52980/0 (socket says 172.21.15.40:52980) 2024-03-24T00:45:03.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e9ca00640 1 -- 172.21.15.40:0/883491544 learned_addr learned my addr 172.21.15.40:0/883491544 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:45:03.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e9d400640 1 --2- 172.21.15.40:0/883491544 >> v2:172.21.15.53:3300/0 conn(0x7f3e9815bd00 0x7f3e980ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e97e00640 1 -- 172.21.15.40:0/883491544 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028732772 0 0) 0x7f3e98072d60 con 0x7f3e980728d0 2024-03-24T00:45:03.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e97e00640 1 -- 172.21.15.40:0/883491544 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e80003530 con 0x7f3e980728d0 2024-03-24T00:45:03.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e97e00640 1 -- 172.21.15.40:0/883491544 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661046012 0 0) 0x7f3e98071de0 con 0x7f3e9815ffd0 2024-03-24T00:45:03.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e97e00640 1 -- 172.21.15.40:0/883491544 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e98072d60 con 0x7f3e9815ffd0 2024-03-24T00:45:03.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e9d400640 1 -- 172.21.15.40:0/883491544 >> v1:172.21.15.53:6789/0 conn(0x7f3e9815ffd0 legacy=0x7f3e9806cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e9d400640 1 -- 172.21.15.40:0/883491544 >> v1:172.21.15.40:6789/0 conn(0x7f3e980728d0 legacy=0x7f3e9815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.438 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.433+0000 7f3e9d400640 1 -- 172.21.15.40:0/883491544 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e980ced30 con 0x7f3e9815bd00 2024-03-24T00:45:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.437+0000 7f3e9d400640 1 --2- 172.21.15.40:0/883491544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e9815bd00 0x7f3e980ae8e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0097c0 tx=0x7f3e8c01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb3f8418a0f8ce15 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.437+0000 7f3e97e00640 1 -- 172.21.15.40:0/883491544 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0ab650 con 0x7f3e9815bd00 2024-03-24T00:45:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.437+0000 7f3e97e00640 1 -- 172.21.15.40:0/883491544 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c0b0070 con 0x7f3e9815bd00 2024-03-24T00:45:03.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.437+0000 7f3e97e00640 1 -- 172.21.15.40:0/883491544 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0aba10 con 0x7f3e9815bd00 2024-03-24T00:45:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.441+0000 7f3e9fe00640 1 -- 172.21.15.40:0/883491544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e9815bd00 msgr2=0x7f3e980ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.441+0000 7f3e9fe00640 1 --2- 172.21.15.40:0/883491544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e9815bd00 0x7f3e980ae8e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0097c0 tx=0x7f3e8c01c470 comp rx=0 tx=0).stop 2024-03-24T00:45:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.441+0000 7f3e9fe00640 1 -- 172.21.15.40:0/883491544 shutdown_connections 2024-03-24T00:45:03.443 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.441+0000 7f3e9fe00640 1 --2- 172.21.15.40:0/883491544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e9815bd00 0x7f3e980ae8e0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.441+0000 7f3e9fe00640 1 -- 172.21.15.40:0/883491544 >> 172.21.15.40:0/883491544 conn(0x7f3e980b1660 msgr2=0x7f3e980b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.441+0000 7f3e9fe00640 1 -- 172.21.15.40:0/883491544 shutdown_connections 2024-03-24T00:45:03.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.441+0000 7f3e9fe00640 1 -- 172.21.15.40:0/883491544 wait complete. 2024-03-24T00:45:03.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.441+0000 7f3e9fe00640 1 Processor -- start 2024-03-24T00:45:03.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.441+0000 7f3e9fe00640 1 -- start start 2024-03-24T00:45:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9fe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e980728d0 0x7f3e980b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e9815bd00 0x7f3e9814a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9fe00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9815ffd0 0x7f3e98146500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9fe00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9815c0e0 con 0x7f3e9815ffd0 2024-03-24T00:45:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e980cf660 con 0x7f3e9815bd00 2024-03-24T00:45:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9fe00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e980b2a60 con 0x7f3e980728d0 2024-03-24T00:45:03.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9815ffd0 0x7f3e98146500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9de00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9815ffd0 0x7f3e98146500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34270/0 (socket says 172.21.15.40:34270) 2024-03-24T00:45:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9de00640 1 -- 172.21.15.40:0/535790163 learned_addr learned my addr 172.21.15.40:0/535790163 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9ca00640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e9815bd00 0x7f3e9814a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9d400640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e980728d0 0x7f3e980b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9d400640 1 -- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e9815bd00 msgr2=0x7f3e9814a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9d400640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e9815bd00 0x7f3e9814a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9d400640 1 -- 172.21.15.40:0/535790163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9815ffd0 msgr2=0x7f3e98146500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9d400640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9815ffd0 0x7f3e98146500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9d400640 1 -- 172.21.15.40:0/535790163 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9814aad0 con 0x7f3e980728d0 2024-03-24T00:45:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9de00640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9815ffd0 0x7f3e98146500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9d400640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e980728d0 0x7f3e980b03a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0ac000 tx=0x7f3e8c01acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e95600640 1 -- 172.21.15.40:0/535790163 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0bf610 con 0x7f3e980728d0 2024-03-24T00:45:03.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e95600640 1 -- 172.21.15.40:0/535790163 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c0bf7f0 con 0x7f3e980728d0 2024-03-24T00:45:03.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e95600640 1 -- 172.21.15.40:0/535790163 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0bfbe0 con 0x7f3e980728d0 2024-03-24T00:45:03.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9fe00640 1 -- 172.21.15.40:0/535790163 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9814adb0 con 0x7f3e980728d0 2024-03-24T00:45:03.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.445+0000 7f3e9fe00640 1 -- 172.21.15.40:0/535790163 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9814b370 con 0x7f3e980728d0 2024-03-24T00:45:03.450 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.449+0000 7f3e9fe00640 1 -- 172.21.15.40:0/535790163 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e98068170 con 0x7f3e980728d0 2024-03-24T00:45:03.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.449+0000 7f3e95600640 1 -- 172.21.15.40:0/535790163 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f3e8c0bd020 con 0x7f3e980728d0 2024-03-24T00:45:03.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.449+0000 7f3e95600640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e78067660 0x7f3e78069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.449+0000 7f3e95600640 1 -- 172.21.15.40:0/535790163 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f3e8c11ffe0 con 0x7f3e980728d0 2024-03-24T00:45:03.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.449+0000 7f3e9ca00640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e78067660 0x7f3e78069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.449+0000 7f3e9ca00640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e78067660 0x7f3e78069b20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f3e980b15f0 tx=0x7f3e84007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.453+0000 7f3e95600640 1 -- 172.21.15.40:0/535790163 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f3e8c016090 con 0x7f3e980728d0 2024-03-24T00:45:03.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 -- 172.21.15.40:0/3450748446 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8704072f50 con 0x7f870406ea00 2024-03-24T00:45:03.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f8700c00640 1 -- 172.21.15.40:0/3450748446 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+4281 (secure 0 0 0) 0x7f86f0055640 con 0x7f870406ea00 2024-03-24T00:45:03.631 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:45:03.631 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":145,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:39:07.531884+0000","modified":"2024-03-24T00:43:12.194456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8877,"mds_1":8874,"mds_2":7439},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7439":{"gid":7439,"name":"d","rank":2,"incarnation":136,"state":"up:active","state_seq":20,"addr":"172.21.15.53:6837/1582567544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1582567544},{"type":"v1","addr":"172.21.15.53:6837","nonce":1582567544}]},"join_fscid":-1,"laggy_since":"2024-03-24T00:40:21.913457+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8874":{"gid":8874,"name":"c","rank":1,"incarnation":130,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6833/4273998586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4273998586},{"type":"v1","addr":"172.21.15.40:6833","nonce":4273998586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8877":{"gid":8877,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":64,"addr":"172.21.15.40:6835/773071485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":773071485},{"type":"v1","addr":"172.21.15.40:6835","nonce":773071485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8877,"qdb_cluster":[7439,8874,8877]},"id":9}]} 2024-03-24T00:45:03.632 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 145 2024-03-24T00:45:03.633 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 -- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f86e4067690 msgr2=0x7f86e4069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f86e4067690 0x7f86e4069b50 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f86f80ad000 tx=0x7f86f80a6000 comp rx=0 tx=0).stop 2024-03-24T00:45:03.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 -- 172.21.15.40:0/3450748446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f870406ea00 msgr2=0x7f8704159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f870406ea00 0x7f8704159300 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f86f00010f0 tx=0x7f86f000c830 comp rx=0 tx=0).stop 2024-03-24T00:45:03.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 -- 172.21.15.40:0/3450748446 shutdown_connections 2024-03-24T00:45:03.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f86e4067690 0x7f86e4069b50 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87040ccfa0 0x7f87040ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f870406ea00 0x7f8704159300 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 --2- 172.21.15.40:0/3450748446 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f870406e620 0x7f8704159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 -- 172.21.15.40:0/3450748446 >> 172.21.15.40:0/3450748446 conn(0x7f87040b17b0 msgr2=0x7f870406d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 -- 172.21.15.40:0/3450748446 shutdown_connections 2024-03-24T00:45:03.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.629+0000 7f870b800640 1 -- 172.21.15.40:0/3450748446 wait complete. 2024-03-24T00:45:03.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.637+0000 7f3e9fe00640 1 -- 172.21.15.40:0/535790163 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3e980b3a10 con 0x7f3e980728d0 2024-03-24T00:45:03.642 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e95600640 1 -- 172.21.15.40:0/535790163 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v122) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3e8c0f0590 con 0x7f3e980728d0 2024-03-24T00:45:03.643 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:45:03.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e9fe00640 1 -- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e78067660 msgr2=0x7f3e78069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e9fe00640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e78067660 0x7f3e78069b20 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f3e980b15f0 tx=0x7f3e84007d40 comp rx=0 tx=0).stop 2024-03-24T00:45:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e9fe00640 1 -- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e980728d0 msgr2=0x7f3e980b03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e9fe00640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e980728d0 0x7f3e980b03a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0ac000 tx=0x7f3e8c01acb0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e9fe00640 1 -- 172.21.15.40:0/535790163 shutdown_connections 2024-03-24T00:45:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e9fe00640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f3e78067660 0x7f3e78069b20 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e9fe00640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f3e9815ffd0 0x7f3e98146500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e9fe00640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e9815bd00 0x7f3e9814a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e9fe00640 1 --2- 172.21.15.40:0/535790163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3e980728d0 0x7f3e980b03a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e9fe00640 1 -- 172.21.15.40:0/535790163 >> 172.21.15.40:0/535790163 conn(0x7f3e980b1660 msgr2=0x7f3e980b4500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e9fe00640 1 -- 172.21.15.40:0/535790163 shutdown_connections 2024-03-24T00:45:03.646 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.641+0000 7f3e9fe00640 1 -- 172.21.15.40:0/535790163 wait complete. 2024-03-24T00:45:03.654 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:45:03.672 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:45:03.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.793+0000 7f8c1a800640 1 Processor -- start 2024-03-24T00:45:03.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.793+0000 7f8c1a800640 1 -- start start 2024-03-24T00:45:03.797 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.793+0000 7f8c1a800640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8c1413ec80 0x7f8c141414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.798 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.793+0000 7f8c1a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8c141419f0 0x7f8c140ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.793+0000 7f8c1a800640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c14070e00 con 0x7f8c140ce630 2024-03-24T00:45:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.793+0000 7f8c1a800640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c14141e50 con 0x7f8c1413ec80 2024-03-24T00:45:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.793+0000 7f8c1a800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c140cee20 con 0x7f8c141419f0 2024-03-24T00:45:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.793+0000 7f8c13400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8c1413ec80 0x7f8c141414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.793+0000 7f8c13400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8c1413ec80 0x7f8c141414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34280/0 (socket says 172.21.15.40:34280) 2024-03-24T00:45:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.793+0000 7f8c13400640 1 -- 172.21.15.40:0/3872175869 learned_addr learned my addr 172.21.15.40:0/3872175869 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.797+0000 7f8c12a00640 1 --2- 172.21.15.40:0/3872175869 >> v2:172.21.15.53:3300/0 conn(0x7f8c141419f0 0x7f8c140ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.797+0000 7f8c13400640 1 -- 172.21.15.40:0/3872175869 >> v2:172.21.15.53:3300/0 conn(0x7f8c141419f0 msgr2=0x7f8c140ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.797+0000 7f8c13400640 1 --2- 172.21.15.40:0/3872175869 >> v2:172.21.15.53:3300/0 conn(0x7f8c141419f0 0x7f8c140ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.797+0000 7f8c13400640 1 -- 172.21.15.40:0/3872175869 >> v1:172.21.15.40:6789/0 conn(0x7f8c140ce630 legacy=0x7f8c1415f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.797+0000 7f8c13400640 1 -- 172.21.15.40:0/3872175869 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c140cf100 con 0x7f8c1413ec80 2024-03-24T00:45:03.803 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.801+0000 7f8c13400640 1 --2- 172.21.15.40:0/3872175869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c1413ec80 0x7f8c141414b0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f8c1404fa60 tx=0x7f8c0401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25aba1db3770843e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.801+0000 7f8c12000640 1 -- 172.21.15.40:0/3872175869 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040a9400 con 0x7f8c1413ec80 2024-03-24T00:45:03.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.801+0000 7f8c12000640 1 -- 172.21.15.40:0/3872175869 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c040a95a0 con 0x7f8c1413ec80 2024-03-24T00:45:03.804 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.801+0000 7f8c12000640 1 -- 172.21.15.40:0/3872175869 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040a9970 con 0x7f8c1413ec80 2024-03-24T00:45:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.801+0000 7f8c1a800640 1 -- 172.21.15.40:0/3872175869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c1413ec80 msgr2=0x7f8c141414b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.801+0000 7f8c1a800640 1 --2- 172.21.15.40:0/3872175869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c1413ec80 0x7f8c141414b0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f8c1404fa60 tx=0x7f8c0401c520 comp rx=0 tx=0).stop 2024-03-24T00:45:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.801+0000 7f8c1a800640 1 -- 172.21.15.40:0/3872175869 shutdown_connections 2024-03-24T00:45:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.801+0000 7f8c1a800640 1 --2- 172.21.15.40:0/3872175869 >> v2:172.21.15.53:3300/0 conn(0x7f8c141419f0 0x7f8c140ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.801+0000 7f8c1a800640 1 --2- 172.21.15.40:0/3872175869 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c1413ec80 0x7f8c141414b0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.801+0000 7f8c1a800640 1 -- 172.21.15.40:0/3872175869 >> 172.21.15.40:0/3872175869 conn(0x7f8c140b17b0 msgr2=0x7f8c140b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.801+0000 7f8c1a800640 1 -- 172.21.15.40:0/3872175869 shutdown_connections 2024-03-24T00:45:03.805 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.801+0000 7f8c1a800640 1 -- 172.21.15.40:0/3872175869 wait complete. 2024-03-24T00:45:03.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c1a800640 1 Processor -- start 2024-03-24T00:45:03.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c1a800640 1 -- start start 2024-03-24T00:45:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c1a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c140ce630 0x7f8c14146b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c1a800640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c1413ec80 0x7f8c1413d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c1a800640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8c141419f0 0x7f8c14140350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c1a800640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c14168ec0 con 0x7f8c1413ec80 2024-03-24T00:45:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c1a800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c14168d40 con 0x7f8c140ce630 2024-03-24T00:45:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c1a800640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c14169040 con 0x7f8c141419f0 2024-03-24T00:45:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c12a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c1413ec80 0x7f8c1413d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c12a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c1413ec80 0x7f8c1413d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34284/0 (socket says 172.21.15.40:34284) 2024-03-24T00:45:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c12a00640 1 -- 172.21.15.40:0/1626714448 learned_addr learned my addr 172.21.15.40:0/1626714448 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c13400640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c140ce630 0x7f8c14146b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c13e00640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8c141419f0 0x7f8c14140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c12a00640 1 -- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8c141419f0 msgr2=0x7f8c14140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c12a00640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8c141419f0 0x7f8c14140350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c12a00640 1 -- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c140ce630 msgr2=0x7f8c14146b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c12a00640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c140ce630 0x7f8c14146b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c12a00640 1 -- 172.21.15.40:0/1626714448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c140b42f0 con 0x7f8c1413ec80 2024-03-24T00:45:03.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.805+0000 7f8c12a00640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c1413ec80 0x7f8c1413d6b0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000d5f0 tx=0x7f8c0000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.809+0000 7f8c0b400640 1 -- 172.21.15.40:0/1626714448 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000125f0 con 0x7f8c1413ec80 2024-03-24T00:45:03.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.809+0000 7f8c0b400640 1 -- 172.21.15.40:0/1626714448 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c00019070 con 0x7f8c1413ec80 2024-03-24T00:45:03.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.809+0000 7f8c0b400640 1 -- 172.21.15.40:0/1626714448 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000129b0 con 0x7f8c1413ec80 2024-03-24T00:45:03.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.809+0000 7f8c1a800640 1 -- 172.21.15.40:0/1626714448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c140b2e00 con 0x7f8c1413ec80 2024-03-24T00:45:03.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.809+0000 7f8c1a800640 1 -- 172.21.15.40:0/1626714448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c140b33c0 con 0x7f8c1413ec80 2024-03-24T00:45:03.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.809+0000 7f8c1a800640 1 -- 172.21.15.40:0/1626714448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c140b35c0 con 0x7f8c1413ec80 2024-03-24T00:45:03.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.809+0000 7f8c0b400640 1 -- 172.21.15.40:0/1626714448 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8c00012b50 con 0x7f8c1413ec80 2024-03-24T00:45:03.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.813+0000 7f8c0b400640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8bf4067660 0x7f8bf4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.813+0000 7f8c0b400640 1 -- 172.21.15.40:0/1626714448 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8c00085bf0 con 0x7f8c1413ec80 2024-03-24T00:45:03.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.813+0000 7f8c13400640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8bf4067660 0x7f8bf4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.813+0000 7f8c13400640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8bf4067660 0x7f8bf4069b20 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f8c040ae000 tx=0x7f8c040b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.821+0000 7f8c0b400640 1 -- 172.21.15.40:0/1626714448 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8c140b35c0 con 0x7f8c1413ec80 2024-03-24T00:45:03.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf33000640 1 Processor -- start 2024-03-24T00:45:03.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf33000640 1 -- start start 2024-03-24T00:45:03.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf33000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbf2c145450 0x7fbf2c0b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf33000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf2c0b62f0 0x7fbf2c06eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf33000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2c1605b0 con 0x7fbf2c144ec0 2024-03-24T00:45:03.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf33000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf2c1452a0 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf33000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf2c0d0600 con 0x7fbf2c145450 2024-03-24T00:45:03.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf32600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf2c0b62f0 0x7fbf2c06eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf32600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbf2c0b62f0 0x7fbf2c06eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34292/0 (socket says 172.21.15.40:34292) 2024-03-24T00:45:03.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf32600640 1 -- 172.21.15.40:0/3508325878 learned_addr learned my addr 172.21.15.40:0/3508325878 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:03.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf31200640 1 --2- 172.21.15.40:0/3508325878 >> v2:172.21.15.53:3300/0 conn(0x7fbf2c145450 0x7fbf2c0b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf32600640 1 -- 172.21.15.40:0/3508325878 >> v2:172.21.15.53:3300/0 conn(0x7fbf2c145450 msgr2=0x7fbf2c0b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:03.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf32600640 1 --2- 172.21.15.40:0/3508325878 >> v2:172.21.15.53:3300/0 conn(0x7fbf2c145450 0x7fbf2c0b5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf32600640 1 -- 172.21.15.40:0/3508325878 >> v1:172.21.15.53:6789/0 conn(0x7fbf2c144ec0 legacy=0x7fbf2c0cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.861+0000 7fbf32600640 1 -- 172.21.15.40:0/3508325878 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c15f7f0 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.865+0000 7fbf32600640 1 --2- 172.21.15.40:0/3508325878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c0b62f0 0x7fbf2c06eb00 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c071450 tx=0x7fbf2401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c828efd8cf0dc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.865+0000 7fbf2be00640 1 -- 172.21.15.40:0/3508325878 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf240a5620 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.865+0000 7fbf2be00640 1 -- 172.21.15.40:0/3508325878 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf240a57c0 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.869 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.865+0000 7fbf2be00640 1 -- 172.21.15.40:0/3508325878 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf240a5b90 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.865+0000 7fbf33000640 1 -- 172.21.15.40:0/3508325878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c0b62f0 msgr2=0x7fbf2c06eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.865+0000 7fbf33000640 1 --2- 172.21.15.40:0/3508325878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c0b62f0 0x7fbf2c06eb00 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c071450 tx=0x7fbf2401c960 comp rx=0 tx=0).stop 2024-03-24T00:45:03.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.865+0000 7fbf33000640 1 -- 172.21.15.40:0/3508325878 shutdown_connections 2024-03-24T00:45:03.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.865+0000 7fbf33000640 1 --2- 172.21.15.40:0/3508325878 >> v2:172.21.15.53:3300/0 conn(0x7fbf2c145450 0x7fbf2c0b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.865+0000 7fbf33000640 1 --2- 172.21.15.40:0/3508325878 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c0b62f0 0x7fbf2c06eb00 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.865+0000 7fbf33000640 1 -- 172.21.15.40:0/3508325878 >> 172.21.15.40:0/3508325878 conn(0x7fbf2c071a40 msgr2=0x7fbf2c071e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:03.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.865+0000 7fbf33000640 1 -- 172.21.15.40:0/3508325878 shutdown_connections 2024-03-24T00:45:03.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.865+0000 7fbf33000640 1 -- 172.21.15.40:0/3508325878 wait complete. 2024-03-24T00:45:03.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf33000640 1 Processor -- start 2024-03-24T00:45:03.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf33000640 1 -- start start 2024-03-24T00:45:03.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf33000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c0b62f0 0x7fbf2c0b4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf33000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf2c144ec0 0x7fbf2c15ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf33000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c145450 0x7fbf2c13cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf33000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c06f850 con 0x7fbf2c145450 2024-03-24T00:45:03.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf33000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c0ce9c0 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf33000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c06f9d0 con 0x7fbf2c144ec0 2024-03-24T00:45:03.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf31c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c0b62f0 0x7fbf2c0b4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf32600640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c145450 0x7fbf2c13cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf31c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c0b62f0 0x7fbf2c0b4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:57048/0 (socket says 172.21.15.40:57048) 2024-03-24T00:45:03.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf31c00640 1 -- 172.21.15.40:0/2313447642 learned_addr learned my addr 172.21.15.40:0/2313447642 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:03.873 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf31200640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf2c144ec0 0x7fbf2c15ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf31c00640 1 -- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf2c144ec0 msgr2=0x7fbf2c15ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf31c00640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf2c144ec0 0x7fbf2c15ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf31c00640 1 -- 172.21.15.40:0/2313447642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c145450 msgr2=0x7fbf2c13cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:03.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf31c00640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c145450 0x7fbf2c13cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:03.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf31c00640 1 -- 172.21.15.40:0/2313447642 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c0b3cb0 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf31c00640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c0b62f0 0x7fbf2c0b4d50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000c500 tx=0x7fbf2000c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf29600640 1 -- 172.21.15.40:0/2313447642 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20016020 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf29600640 1 -- 172.21.15.40:0/2313447642 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf20005220 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf29600640 1 -- 172.21.15.40:0/2313447642 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20007030 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.869+0000 7fbf33000640 1 -- 172.21.15.40:0/2313447642 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2c0b3f90 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.873+0000 7fbf33000640 1 -- 172.21.15.40:0/2313447642 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2c0b4550 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.873+0000 7fbf33000640 1 -- 172.21.15.40:0/2313447642 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2c04f410 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.873+0000 7fbf29600640 1 -- 172.21.15.40:0/2313447642 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbf200071f0 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.877+0000 7fbf29600640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf0c067660 0x7fbf0c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:03.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.877+0000 7fbf29600640 1 -- 172.21.15.40:0/2313447642 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fbf20084d60 con 0x7fbf2c0b62f0 2024-03-24T00:45:03.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.877+0000 7fbf31200640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf0c067660 0x7fbf0c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:03.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.877+0000 7fbf31200640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf0c067660 0x7fbf0c069b20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c13def0 tx=0x7fbf18009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:03.882 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:03.881+0000 7fbf29600640 1 -- 172.21.15.40:0/2313447642 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbf20055270 con 0x7fbf2c0b62f0 2024-03-24T00:45:04.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.001+0000 7f8c1a800640 1 -- 172.21.15.40:0/1626714448 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c14068170 con 0x7f8c1413ec80 2024-03-24T00:45:04.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.001+0000 7f8c0b400640 1 -- 172.21.15.40:0/1626714448 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+16789 (secure 0 0 0) 0x7f8c00056870 con 0x7f8c1413ec80 2024-03-24T00:45:04.005 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:45:04.005 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:40:01.907736+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:05.964092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:06.525558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.c","mappings":[{"from":2,"to":0}]},{"pgid":"19.23","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6836/2255755838":"2024-03-25T00:40:01.907668+0000","172.21.15.53:6834/2255755838":"2024-03-25T00:40:01.907668+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:45:04.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.005+0000 7f8c1a800640 1 -- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8bf4067660 msgr2=0x7f8bf4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.005+0000 7f8c1a800640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8bf4067660 0x7f8bf4069b20 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f8c040ae000 tx=0x7f8c040b3040 comp rx=0 tx=0).stop 2024-03-24T00:45:04.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.005+0000 7f8c1a800640 1 -- 172.21.15.40:0/1626714448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c1413ec80 msgr2=0x7f8c1413d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.005+0000 7f8c1a800640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c1413ec80 0x7f8c1413d6b0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f8c0000d5f0 tx=0x7f8c0000dab0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.005+0000 7f8c1a800640 1 -- 172.21.15.40:0/1626714448 shutdown_connections 2024-03-24T00:45:04.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.005+0000 7f8c1a800640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8bf4067660 0x7f8bf4069b20 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.005+0000 7f8c1a800640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8c141419f0 0x7f8c14140350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.005+0000 7f8c1a800640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8c1413ec80 0x7f8c1413d6b0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.005+0000 7f8c1a800640 1 --2- 172.21.15.40:0/1626714448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c140ce630 0x7f8c14146b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.005+0000 7f8c1a800640 1 -- 172.21.15.40:0/1626714448 >> 172.21.15.40:0/1626714448 conn(0x7f8c140b17b0 msgr2=0x7f8c140b1b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.005+0000 7f8c1a800640 1 -- 172.21.15.40:0/1626714448 shutdown_connections 2024-03-24T00:45:04.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.005+0000 7f8c1a800640 1 -- 172.21.15.40:0/1626714448 wait complete. 2024-03-24T00:45:04.036 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:30:53.260096+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:39:05.964092+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:39:06.525558+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '119', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:45:04.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:45:04.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.057+0000 7fbf33000640 1 -- 172.21.15.40:0/2313447642 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbf2c0cf3e0 con 0x7fbf2c0b62f0 2024-03-24T00:45:04.062 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.057+0000 7fbf29600640 1 -- 172.21.15.40:0/2313447642 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v122) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbf20059310 con 0x7fbf2c0b62f0 2024-03-24T00:45:04.062 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:45:04.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.061+0000 7fbf33000640 1 -- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf0c067660 msgr2=0x7fbf0c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.061+0000 7fbf33000640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf0c067660 0x7fbf0c069b20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c13def0 tx=0x7fbf18009190 comp rx=0 tx=0).stop 2024-03-24T00:45:04.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.061+0000 7fbf33000640 1 -- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c0b62f0 msgr2=0x7fbf2c0b4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.061+0000 7fbf33000640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c0b62f0 0x7fbf2c0b4d50 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000c500 tx=0x7fbf2000c9c0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.061+0000 7fbf33000640 1 -- 172.21.15.40:0/2313447642 shutdown_connections 2024-03-24T00:45:04.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.061+0000 7fbf33000640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbf0c067660 0x7fbf0c069b20 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.061+0000 7fbf33000640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbf2c145450 0x7fbf2c13cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.061+0000 7fbf33000640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbf2c144ec0 0x7fbf2c15ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.061+0000 7fbf33000640 1 --2- 172.21.15.40:0/2313447642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c0b62f0 0x7fbf2c0b4d50 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.061+0000 7fbf33000640 1 -- 172.21.15.40:0/2313447642 >> 172.21.15.40:0/2313447642 conn(0x7fbf2c071a40 msgr2=0x7fbf2c071e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.061+0000 7fbf33000640 1 -- 172.21.15.40:0/2313447642 shutdown_connections 2024-03-24T00:45:04.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.061+0000 7fbf33000640 1 -- 172.21.15.40:0/2313447642 wait complete. 2024-03-24T00:45:04.093 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T00:45:04.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.181+0000 7f459be00640 1 Processor -- start 2024-03-24T00:45:04.185 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.181+0000 7f459be00640 1 -- start start 2024-03-24T00:45:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.181+0000 7f459be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f459c15e0e0 0x7f459c0cd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.181+0000 7f459be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f459c0d0070 0x7f459c0d0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.181+0000 7f459be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c0cf2e0 con 0x7f459c0b0bd0 2024-03-24T00:45:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.181+0000 7f459be00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c0cd0e0 con 0x7f459c0d0070 2024-03-24T00:45:04.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.181+0000 7f459be00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c0cc7b0 con 0x7f459c15e0e0 2024-03-24T00:45:04.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.185+0000 7f459aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f459c15e0e0 0x7f459c0cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.185+0000 7f459aa00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f459c15e0e0 0x7f459c0cd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:37416/0 (socket says 172.21.15.40:37416) 2024-03-24T00:45:04.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.185+0000 7f459aa00640 1 -- 172.21.15.40:0/2455738583 learned_addr learned my addr 172.21.15.40:0/2455738583 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:04.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.185+0000 7f459a000640 1 --1- 172.21.15.40:0/2455738583 >> v1:172.21.15.53:6789/0 conn(0x7f459c0b0bd0 0x7f459c0ceab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.40:37116/0 (socket says 172.21.15.40:37116) 2024-03-24T00:45:04.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.185+0000 7f459b400640 1 --2- 172.21.15.40:0/2455738583 >> v2:172.21.15.53:3300/0 conn(0x7f459c0d0070 0x7f459c0d0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.185+0000 7f459aa00640 1 -- 172.21.15.40:0/2455738583 >> v2:172.21.15.53:3300/0 conn(0x7f459c0d0070 msgr2=0x7f459c0d0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.185+0000 7f459aa00640 1 --2- 172.21.15.40:0/2455738583 >> v2:172.21.15.53:3300/0 conn(0x7f459c0d0070 0x7f459c0d0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.185+0000 7f459aa00640 1 -- 172.21.15.40:0/2455738583 >> v1:172.21.15.53:6789/0 conn(0x7f459c0b0bd0 legacy=0x7f459c0ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.188 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.185+0000 7f459aa00640 1 -- 172.21.15.40:0/2455738583 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c0cca90 con 0x7f459c15e0e0 2024-03-24T00:45:04.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.189+0000 7f459aa00640 1 --2- 172.21.15.40:0/2455738583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f459c15e0e0 0x7f459c0cd570 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f458c009870 tx=0x7f458c01c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8337c48f77f086b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.189+0000 7f4599600640 1 -- 172.21.15.40:0/2455738583 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c0a7070 con 0x7f459c15e0e0 2024-03-24T00:45:04.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.189+0000 7f4599600640 1 -- 172.21.15.40:0/2455738583 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c0b0070 con 0x7f459c15e0e0 2024-03-24T00:45:04.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.189+0000 7f4599600640 1 -- 172.21.15.40:0/2455738583 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c0ab890 con 0x7f459c15e0e0 2024-03-24T00:45:04.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.189+0000 7f459be00640 1 -- 172.21.15.40:0/2455738583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f459c15e0e0 msgr2=0x7f459c0cd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.189+0000 7f459be00640 1 --2- 172.21.15.40:0/2455738583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f459c15e0e0 0x7f459c0cd570 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f458c009870 tx=0x7f458c01c560 comp rx=0 tx=0).stop 2024-03-24T00:45:04.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.189+0000 7f459be00640 1 -- 172.21.15.40:0/2455738583 shutdown_connections 2024-03-24T00:45:04.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.189+0000 7f459be00640 1 --2- 172.21.15.40:0/2455738583 >> v2:172.21.15.53:3300/0 conn(0x7f459c0d0070 0x7f459c0d0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.189+0000 7f459be00640 1 --2- 172.21.15.40:0/2455738583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f459c15e0e0 0x7f459c0cd570 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.189+0000 7f459be00640 1 -- 172.21.15.40:0/2455738583 >> 172.21.15.40:0/2455738583 conn(0x7f459c0b3620 msgr2=0x7f459c0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.189+0000 7f459be00640 1 -- 172.21.15.40:0/2455738583 shutdown_connections 2024-03-24T00:45:04.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.189+0000 7f459be00640 1 -- 172.21.15.40:0/2455738583 wait complete. 2024-03-24T00:45:04.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459be00640 1 Processor -- start 2024-03-24T00:45:04.196 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459be00640 1 -- start start 2024-03-24T00:45:04.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f459c0b0bd0 0x7f459c0b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f459c0d0070 0x7f459c14cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c15e0e0 0x7f459c15ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c06bcf0 con 0x7f459c15e0e0 2024-03-24T00:45:04.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c06bb70 con 0x7f459c0b0bd0 2024-03-24T00:45:04.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459be00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c0b3310 con 0x7f459c0d0070 2024-03-24T00:45:04.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c15e0e0 0x7f459c15ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459b400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c15e0e0 0x7f459c15ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34308/0 (socket says 172.21.15.40:34308) 2024-03-24T00:45:04.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459b400640 1 -- 172.21.15.40:0/3652563494 learned_addr learned my addr 172.21.15.40:0/3652563494 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:04.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459a000640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f459c0d0070 0x7f459c14cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.193+0000 7f459aa00640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f459c0b0bd0 0x7f459c0b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f459aa00640 1 -- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f459c0d0070 msgr2=0x7f459c14cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f459aa00640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f459c0d0070 0x7f459c14cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f459aa00640 1 -- 172.21.15.40:0/3652563494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c15e0e0 msgr2=0x7f459c15ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f459aa00640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c15e0e0 0x7f459c15ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f459aa00640 1 -- 172.21.15.40:0/3652563494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c14d4a0 con 0x7f459c0b0bd0 2024-03-24T00:45:04.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f459b400640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c15e0e0 0x7f459c15ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:04.201 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f459aa00640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f459c0b0bd0 0x7f459c0b1c90 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f458c0ac000 tx=0x7f458c01ada0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f4593400640 1 -- 172.21.15.40:0/3652563494 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c0b4800 con 0x7f459c0b0bd0 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f4593400640 1 -- 172.21.15.40:0/3652563494 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c0b49a0 con 0x7f459c0b0bd0 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f4593400640 1 -- 172.21.15.40:0/3652563494 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458c0b4d90 con 0x7f459c0b0bd0 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f459be00640 1 -- 172.21.15.40:0/3652563494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c0b2270 con 0x7f459c0b0bd0 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f459be00640 1 -- 172.21.15.40:0/3652563494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c0b2830 con 0x7f459c0b0bd0 2024-03-24T00:45:04.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.197+0000 7f459be00640 1 -- 172.21.15.40:0/3652563494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459c04f410 con 0x7f459c0b0bd0 2024-03-24T00:45:04.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.201+0000 7f4593400640 1 -- 172.21.15.40:0/3652563494 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f458c0c7440 con 0x7f459c0b0bd0 2024-03-24T00:45:04.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.201+0000 7f4593400640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f457c067690 0x7f457c069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.201+0000 7f4593400640 1 -- 172.21.15.40:0/3652563494 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f458c1200a0 con 0x7f459c0b0bd0 2024-03-24T00:45:04.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.201+0000 7f459a000640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f457c067690 0x7f457c069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.201+0000 7f459a000640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f457c067690 0x7f457c069b50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f459c15db00 tx=0x7f4588007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.205+0000 7f4593400640 1 -- 172.21.15.40:0/3652563494 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f458c016150 con 0x7f459c0b0bd0 2024-03-24T00:45:04.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.277+0000 7fbd2be00640 1 Processor -- start 2024-03-24T00:45:04.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.277+0000 7fbd2be00640 1 -- start start 2024-03-24T00:45:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbd2c0cea10 0x7fbd2c0d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fbd2c0cfdc0 0x7fbd2c06ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2be00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2c0cc190 con 0x7fbd2c0ce630 2024-03-24T00:45:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2be00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd2c0d1080 con 0x7fbd2c0cfdc0 2024-03-24T00:45:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2be00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd2c0d0da0 con 0x7fbd2c0cea10 2024-03-24T00:45:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2aa00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7fbd2c0ce630 0x7fbd2c06b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:52992/0 (socket says 172.21.15.40:52992) 2024-03-24T00:45:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2aa00640 1 -- 172.21.15.40:0/4163370634 learned_addr learned my addr 172.21.15.40:0/4163370634 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:45:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2b400640 1 --2- 172.21.15.40:0/4163370634 >> v2:172.21.15.40:3300/0 conn(0x7fbd2c0cfdc0 0x7fbd2c06ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2a000640 1 --2- 172.21.15.40:0/4163370634 >> v2:172.21.15.53:3300/0 conn(0x7fbd2c0cea10 0x7fbd2c0d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2b400640 1 -- 172.21.15.40:0/4163370634 >> v2:172.21.15.53:3300/0 conn(0x7fbd2c0cea10 msgr2=0x7fbd2c0d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2b400640 1 --2- 172.21.15.40:0/4163370634 >> v2:172.21.15.53:3300/0 conn(0x7fbd2c0cea10 0x7fbd2c0d2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2b400640 1 -- 172.21.15.40:0/4163370634 >> v1:172.21.15.40:6789/0 conn(0x7fbd2c0ce630 legacy=0x7fbd2c06b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.281+0000 7fbd2b400640 1 -- 172.21.15.40:0/4163370634 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c0cc430 con 0x7fbd2c0cfdc0 2024-03-24T00:45:04.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.285+0000 7fbd2b400640 1 --2- 172.21.15.40:0/4163370634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2c0cfdc0 0x7fbd2c06ba70 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c071540 tx=0x7fbd2401f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56abfe7c90f2a4fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.285+0000 7fbd29600640 1 -- 172.21.15.40:0/4163370634 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd240b4070 con 0x7fbd2c0cfdc0 2024-03-24T00:45:04.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.285+0000 7fbd29600640 1 -- 172.21.15.40:0/4163370634 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd240af600 con 0x7fbd2c0cfdc0 2024-03-24T00:45:04.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.285+0000 7fbd29600640 1 -- 172.21.15.40:0/4163370634 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd240af9d0 con 0x7fbd2c0cfdc0 2024-03-24T00:45:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.285+0000 7fbd2be00640 1 -- 172.21.15.40:0/4163370634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2c0cfdc0 msgr2=0x7fbd2c06ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.285+0000 7fbd2be00640 1 --2- 172.21.15.40:0/4163370634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2c0cfdc0 0x7fbd2c06ba70 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c071540 tx=0x7fbd2401f110 comp rx=0 tx=0).stop 2024-03-24T00:45:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.285+0000 7fbd2be00640 1 -- 172.21.15.40:0/4163370634 shutdown_connections 2024-03-24T00:45:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.285+0000 7fbd2be00640 1 --2- 172.21.15.40:0/4163370634 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2c0cfdc0 0x7fbd2c06ba70 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.285+0000 7fbd2be00640 1 --2- 172.21.15.40:0/4163370634 >> v2:172.21.15.53:3300/0 conn(0x7fbd2c0cea10 0x7fbd2c0d2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.285+0000 7fbd2be00640 1 -- 172.21.15.40:0/4163370634 >> 172.21.15.40:0/4163370634 conn(0x7fbd2c0d1aa0 msgr2=0x7fbd2c0d1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.285+0000 7fbd2be00640 1 -- 172.21.15.40:0/4163370634 shutdown_connections 2024-03-24T00:45:04.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.285+0000 7fbd2be00640 1 -- 172.21.15.40:0/4163370634 wait complete. 2024-03-24T00:45:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.289+0000 7fbd2be00640 1 Processor -- start 2024-03-24T00:45:04.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.289+0000 7fbd2be00640 1 -- start start 2024-03-24T00:45:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.289+0000 7fbd2be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd2c0ce630 0x7fbd2c0c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.289+0000 7fbd2be00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2c0cea10 0x7fbd2c09cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.289+0000 7fbd2be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2c0cfdc0 0x7fbd2c154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.289+0000 7fbd2be00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c06c640 con 0x7fbd2c0cea10 2024-03-24T00:45:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.289+0000 7fbd2be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c06c4c0 con 0x7fbd2c0cfdc0 2024-03-24T00:45:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.289+0000 7fbd2be00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2c06bfb0 con 0x7fbd2c0ce630 2024-03-24T00:45:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.289+0000 7fbd2a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2c0cea10 0x7fbd2c09cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.289+0000 7fbd2a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2c0cea10 0x7fbd2c09cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34320/0 (socket says 172.21.15.40:34320) 2024-03-24T00:45:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.289+0000 7fbd2a000640 1 -- 172.21.15.40:0/544700417 learned_addr learned my addr 172.21.15.40:0/544700417 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd2aa00640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd2c0ce630 0x7fbd2c0c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd2b400640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2c0cfdc0 0x7fbd2c154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd2a000640 1 -- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd2c0ce630 msgr2=0x7fbd2c0c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd2a000640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd2c0ce630 0x7fbd2c0c7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd2a000640 1 -- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2c0cfdc0 msgr2=0x7fbd2c154b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd2a000640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2c0cfdc0 0x7fbd2c154b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd2a000640 1 -- 172.21.15.40:0/544700417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c0c8d10 con 0x7fbd2c0cea10 2024-03-24T00:45:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd2a000640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2c0cea10 0x7fbd2c09cbe0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800e930 tx=0x7fbd1800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd22a00640 1 -- 172.21.15.40:0/544700417 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18017070 con 0x7fbd2c0cea10 2024-03-24T00:45:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd22a00640 1 -- 172.21.15.40:0/544700417 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1800ca00 con 0x7fbd2c0cea10 2024-03-24T00:45:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd22a00640 1 -- 172.21.15.40:0/544700417 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1800cdb0 con 0x7fbd2c0cea10 2024-03-24T00:45:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd2be00640 1 -- 172.21.15.40:0/544700417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd2c0c8ff0 con 0x7fbd2c0cea10 2024-03-24T00:45:04.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd2be00640 1 -- 172.21.15.40:0/544700417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd2c0c7ca0 con 0x7fbd2c0cea10 2024-03-24T00:45:04.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.293+0000 7fbd2be00640 1 -- 172.21.15.40:0/544700417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd2c09fb10 con 0x7fbd2c0cea10 2024-03-24T00:45:04.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.297+0000 7fbd22a00640 1 -- 172.21.15.40:0/544700417 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fbd1801a070 con 0x7fbd2c0cea10 2024-03-24T00:45:04.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.297+0000 7fbd22a00640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbd0c067660 0x7fbd0c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.297+0000 7fbd22a00640 1 -- 172.21.15.40:0/544700417 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fbd18085bf0 con 0x7fbd2c0cea10 2024-03-24T00:45:04.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.297+0000 7fbd2aa00640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbd0c067660 0x7fbd0c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.297+0000 7fbd2aa00640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbd0c067660 0x7fbd0c069b20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c000f70 tx=0x7fbd1c007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.301+0000 7fbd22a00640 1 -- 172.21.15.40:0/544700417 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fbd180562e0 con 0x7fbd2c0cea10 2024-03-24T00:45:04.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.373+0000 7f459be00640 1 -- 172.21.15.40:0/3652563494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f459c15a7c0 con 0x7f459c0b0bd0 2024-03-24T00:45:04.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f4593400640 1 -- 172.21.15.40:0/3652563494 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7f458c0f07f0 con 0x7f459c0b0bd0 2024-03-24T00:45:04.379 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 1 2024-03-24T00:45:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f459be00640 1 -- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f457c067690 msgr2=0x7f457c069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f459be00640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f457c067690 0x7f457c069b50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f459c15db00 tx=0x7f4588007e90 comp rx=0 tx=0).stop 2024-03-24T00:45:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f459be00640 1 -- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f459c0b0bd0 msgr2=0x7f459c0b1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f459be00640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f459c0b0bd0 0x7f459c0b1c90 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f458c0ac000 tx=0x7f458c01ada0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f459be00640 1 -- 172.21.15.40:0/3652563494 shutdown_connections 2024-03-24T00:45:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f459be00640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f457c067690 0x7f457c069b50 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f459be00640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f459c15e0e0 0x7f459c15ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f459be00640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f459c0d0070 0x7f459c14cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f459be00640 1 --2- 172.21.15.40:0/3652563494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f459c0b0bd0 0x7f459c0b1c90 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f459be00640 1 -- 172.21.15.40:0/3652563494 >> 172.21.15.40:0/3652563494 conn(0x7f459c0b3620 msgr2=0x7f459c06f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f459be00640 1 -- 172.21.15.40:0/3652563494 shutdown_connections 2024-03-24T00:45:04.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.377+0000 7f459be00640 1 -- 172.21.15.40:0/3652563494 wait complete. 2024-03-24T00:45:04.401 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:45:04.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd2be00640 1 -- 172.21.15.40:0/544700417 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbd2c04f4f0 con 0x7fbd2c0cea10 2024-03-24T00:45:04.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd22a00640 1 -- 172.21.15.40:0/544700417 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v122) v1 ==== 80+0+122 (secure 0 0 0) 0x7fbd1805a380 con 0x7fbd2c0cea10 2024-03-24T00:45:04.496 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.53:6836/2255755838 2024-03-25T00:40:01.907668+0000 2024-03-24T00:45:04.496 INFO:teuthology.orchestra.run.smithi040.stdout:172.21.15.53:6834/2255755838 2024-03-25T00:40:01.907668+0000 2024-03-24T00:45:04.496 INFO:teuthology.orchestra.run.smithi040.stderr:listed 2 entries 2024-03-24T00:45:04.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd2be00640 1 -- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbd0c067660 msgr2=0x7fbd0c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd2be00640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbd0c067660 0x7fbd0c069b20 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c000f70 tx=0x7fbd1c007e90 comp rx=0 tx=0).stop 2024-03-24T00:45:04.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd2be00640 1 -- 172.21.15.40:0/544700417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2c0cea10 msgr2=0x7fbd2c09cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd2be00640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2c0cea10 0x7fbd2c09cbe0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fbd1800e930 tx=0x7fbd1800edf0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd2be00640 1 -- 172.21.15.40:0/544700417 shutdown_connections 2024-03-24T00:45:04.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd2be00640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fbd0c067660 0x7fbd0c069b20 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd2be00640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2c0cfdc0 0x7fbd2c154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd2be00640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fbd2c0cea10 0x7fbd2c09cbe0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd2be00640 1 --2- 172.21.15.40:0/544700417 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbd2c0ce630 0x7fbd2c0c7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd2be00640 1 -- 172.21.15.40:0/544700417 >> 172.21.15.40:0/544700417 conn(0x7fbd2c0d1aa0 msgr2=0x7fbd2c161370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.493+0000 7fbd2be00640 1 -- 172.21.15.40:0/544700417 shutdown_connections 2024-03-24T00:45:04.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.497+0000 7fbd2be00640 1 -- 172.21.15.40:0/544700417 wait complete. 2024-03-24T00:45:04.519 ERROR:tasks.daemonwatchdog.daemon_watchdog:ignoring exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/daemonwatchdog.py", line 61, in bark mount.umount_wait(force=True) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/fuse_mount.py", line 399, in umount_wait self.umount(cleanup=False) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/fuse_mount.py", line 320, in umount if self.is_blocked(): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/mount.py", line 197, in is_blocked return self.addr in output TypeError: 'in ' requires string as left operand, not NoneType 2024-03-24T00:45:04.520 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:45:04.520 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:04.530 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:45:04.531 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.2 is not mounted at ubuntu@smithi040.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2 2024-03-24T00:45:04.531 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:45:04.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:04.531 DEBUG:teuthology.orchestra.run.smithi040:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-24T00:45:04.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.545+0000 7ff3b5c00640 1 Processor -- start 2024-03-24T00:45:04.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.545+0000 7ff3b5c00640 1 -- start start 2024-03-24T00:45:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3b5c00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff3b015bd00 0x7ff3b00ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3b5c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff3b015ffd0 0x7ff3b00cf250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3b5c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b0072d60 con 0x7ff3b00728d0 2024-03-24T00:45:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3b5c00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b00cf790 con 0x7ff3b015bd00 2024-03-24T00:45:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3b5c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b0071de0 con 0x7ff3b015ffd0 2024-03-24T00:45:04.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3aea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff3b015bd00 0x7ff3b00ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3aea00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7ff3b015bd00 0x7ff3b00ae8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34322/0 (socket says 172.21.15.40:34322) 2024-03-24T00:45:04.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3aea00640 1 -- 172.21.15.40:0/413624612 learned_addr learned my addr 172.21.15.40:0/413624612 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:04.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3af400640 1 --2- 172.21.15.40:0/413624612 >> v2:172.21.15.53:3300/0 conn(0x7ff3b015ffd0 0x7ff3b00cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3aea00640 1 -- 172.21.15.40:0/413624612 >> v2:172.21.15.53:3300/0 conn(0x7ff3b015ffd0 msgr2=0x7ff3b00cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3aea00640 1 --2- 172.21.15.40:0/413624612 >> v2:172.21.15.53:3300/0 conn(0x7ff3b015ffd0 0x7ff3b00cf250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3aea00640 1 -- 172.21.15.40:0/413624612 >> v1:172.21.15.53:6789/0 conn(0x7ff3b00728d0 legacy=0x7ff3b015f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.549+0000 7ff3aea00640 1 -- 172.21.15.40:0/413624612 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b00cea50 con 0x7ff3b015bd00 2024-03-24T00:45:04.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.553+0000 7ff3aea00640 1 --2- 172.21.15.40:0/413624612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3b015bd00 0x7ff3b00ae8e0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0070a80 tx=0x7ff3a001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887db6fb48fbad1a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.553+0000 7ff3ae000640 1 -- 172.21.15.40:0/413624612 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a00aa650 con 0x7ff3b015bd00 2024-03-24T00:45:04.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.553+0000 7ff3ae000640 1 -- 172.21.15.40:0/413624612 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a00af070 con 0x7ff3b015bd00 2024-03-24T00:45:04.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.553+0000 7ff3ae000640 1 -- 172.21.15.40:0/413624612 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a00aaa10 con 0x7ff3b015bd00 2024-03-24T00:45:04.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.553+0000 7ff3b5c00640 1 -- 172.21.15.40:0/413624612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3b015bd00 msgr2=0x7ff3b00ae8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.553+0000 7ff3b5c00640 1 --2- 172.21.15.40:0/413624612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3b015bd00 0x7ff3b00ae8e0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0070a80 tx=0x7ff3a001c4f0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.553+0000 7ff3b5c00640 1 -- 172.21.15.40:0/413624612 shutdown_connections 2024-03-24T00:45:04.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.553+0000 7ff3b5c00640 1 --2- 172.21.15.40:0/413624612 >> v2:172.21.15.53:3300/0 conn(0x7ff3b015ffd0 0x7ff3b00cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.553+0000 7ff3b5c00640 1 --2- 172.21.15.40:0/413624612 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3b015bd00 0x7ff3b00ae8e0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.553+0000 7ff3b5c00640 1 -- 172.21.15.40:0/413624612 >> 172.21.15.40:0/413624612 conn(0x7ff3b00b1660 msgr2=0x7ff3b00b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.553+0000 7ff3b5c00640 1 -- 172.21.15.40:0/413624612 shutdown_connections 2024-03-24T00:45:04.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.553+0000 7ff3b5c00640 1 -- 172.21.15.40:0/413624612 wait complete. 2024-03-24T00:45:04.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3b5c00640 1 Processor -- start 2024-03-24T00:45:04.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3b5c00640 1 -- start start 2024-03-24T00:45:04.559 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-24T00:45:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:45:04.560 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:45:04.560 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:04.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3b5c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3b00728d0 0x7ff3b013c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3b5c00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff3b015bd00 0x7ff3b013cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3b5c00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3b015ffd0 0x7ff3b0155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3b5c00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b006d320 con 0x7ff3b015ffd0 2024-03-24T00:45:04.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3b5c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b015c0e0 con 0x7ff3b00728d0 2024-03-24T00:45:04.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3b5c00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b006d4a0 con 0x7ff3b015bd00 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3aea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3b00728d0 0x7ff3b013c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3aea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3b00728d0 0x7ff3b013c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.40:57112/0 (socket says 172.21.15.40:57112) 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3aea00640 1 -- 172.21.15.40:0/1898323525 learned_addr learned my addr 172.21.15.40:0/1898323525 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3af400640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3b015ffd0 0x7ff3b0155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3a7e00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff3b015bd00 0x7ff3b013cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3aea00640 1 -- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff3b015bd00 msgr2=0x7ff3b013cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3aea00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff3b015bd00 0x7ff3b013cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3aea00640 1 -- 172.21.15.40:0/1898323525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3b015ffd0 msgr2=0x7ff3b0155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3aea00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3b015ffd0 0x7ff3b0155660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3aea00640 1 -- 172.21.15.40:0/1898323525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b0153920 con 0x7ff3b00728d0 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3aea00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3b00728d0 0x7ff3b013c4a0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff3a00ac040 tx=0x7ff3a001ad30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3a6a00640 1 -- 172.21.15.40:0/1898323525 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a00bd5f0 con 0x7ff3b00728d0 2024-03-24T00:45:04.563 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3a6a00640 1 -- 172.21.15.40:0/1898323525 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a00a6040 con 0x7ff3b00728d0 2024-03-24T00:45:04.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3a6a00640 1 -- 172.21.15.40:0/1898323525 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a00bd990 con 0x7ff3b00728d0 2024-03-24T00:45:04.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3b5c00640 1 -- 172.21.15.40:0/1898323525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b0153c00 con 0x7ff3b00728d0 2024-03-24T00:45:04.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.557+0000 7ff3b5c00640 1 -- 172.21.15.40:0/1898323525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b01541c0 con 0x7ff3b00728d0 2024-03-24T00:45:04.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.561+0000 7ff3b5c00640 1 -- 172.21.15.40:0/1898323525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b0068170 con 0x7ff3b00728d0 2024-03-24T00:45:04.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.561+0000 7ff3a6a00640 1 -- 172.21.15.40:0/1898323525 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7ff3a00b33f0 con 0x7ff3b00728d0 2024-03-24T00:45:04.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.561+0000 7ff3a6a00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff38c067660 0x7ff38c069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.561+0000 7ff3a6a00640 1 -- 172.21.15.40:0/1898323525 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff3a011f910 con 0x7ff3b00728d0 2024-03-24T00:45:04.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.561+0000 7ff3a7e00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff38c067660 0x7ff38c069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.561+0000 7ff3a7e00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff38c067660 0x7ff38c069b20 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff3b00b0180 tx=0x7ff398009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.569 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.565+0000 7ff3a6a00640 1 -- 172.21.15.40:0/1898323525 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7ff3a0016110 con 0x7ff3b00728d0 2024-03-24T00:45:04.573 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:45:04.573 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.1 is not mounted at ubuntu@smithi053.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1 2024-03-24T00:45:04.573 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:45:04.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:04.573 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-24T00:45:04.623 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-24T00:45:04.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:45:04.624 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:45:04.624 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:45:04.675 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:45:04.675 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.3 is not mounted at ubuntu@smithi053.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3 2024-03-24T00:45:04.675 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:45:04.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:45:04.676 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-24T00:45:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-24T00:45:04.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:45:04.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.729+0000 7ff3b5c00640 1 -- 172.21.15.40:0/1898323525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff3b006e870 con 0x7ff3b00728d0 2024-03-24T00:45:04.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.729+0000 7ff3a6a00640 1 -- 172.21.15.40:0/1898323525 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v122) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff3a00efea0 con 0x7ff3b00728d0 2024-03-24T00:45:04.733 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 32 2024-03-24T00:45:04.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.733+0000 7ff3b5c00640 1 -- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff38c067660 msgr2=0x7ff38c069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.733+0000 7ff3b5c00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff38c067660 0x7ff38c069b20 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7ff3b00b0180 tx=0x7ff398009040 comp rx=0 tx=0).stop 2024-03-24T00:45:04.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.733+0000 7ff3b5c00640 1 -- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3b00728d0 msgr2=0x7ff3b013c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.733+0000 7ff3b5c00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3b00728d0 0x7ff3b013c4a0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff3a00ac040 tx=0x7ff3a001ad30 comp rx=0 tx=0).stop 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.733+0000 7ff3b5c00640 1 -- 172.21.15.40:0/1898323525 shutdown_connections 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.733+0000 7ff3b5c00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7ff38c067660 0x7ff38c069b20 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.733+0000 7ff3b5c00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7ff3b015ffd0 0x7ff3b0155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.733+0000 7ff3b5c00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff3b015bd00 0x7ff3b013cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.733+0000 7ff3b5c00640 1 --2- 172.21.15.40:0/1898323525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3b00728d0 0x7ff3b013c4a0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.733+0000 7ff3b5c00640 1 -- 172.21.15.40:0/1898323525 >> 172.21.15.40:0/1898323525 conn(0x7ff3b00b1660 msgr2=0x7ff3b00cdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.733+0000 7ff3b5c00640 1 -- 172.21.15.40:0/1898323525 shutdown_connections 2024-03-24T00:45:04.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.733+0000 7ff3b5c00640 1 -- 172.21.15.40:0/1898323525 wait complete. 2024-03-24T00:45:04.761 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:45:04.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.941+0000 7f8ceba00640 1 Processor -- start 2024-03-24T00:45:04.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.941+0000 7f8ceba00640 1 -- start start 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ceba00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ce415e000 0x7f8ce40d16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ceba00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ce41628d0 0x7f8ce40b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ceba00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce415e3e0 con 0x7f8ce406f840 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ceba00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce4161ba0 con 0x7f8ce41628d0 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ceba00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce40b5f10 con 0x7f8ce415e000 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ce3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ce41628d0 0x7f8ce40b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ce3e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f8ce41628d0 0x7f8ce40b3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34328/0 (socket says 172.21.15.40:34328) 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ce3e00640 1 -- 172.21.15.40:0/1421237790 learned_addr learned my addr 172.21.15.40:0/1421237790 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ce9000640 1 --2- 172.21.15.40:0/1421237790 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ce415e000 0x7f8ce40d16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ce3e00640 1 -- 172.21.15.40:0/1421237790 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ce415e000 msgr2=0x7f8ce40d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ce3e00640 1 --2- 172.21.15.40:0/1421237790 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ce415e000 0x7f8ce40d16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ce3e00640 1 -- 172.21.15.40:0/1421237790 >> v1:172.21.15.53:6789/0 conn(0x7f8ce406f840 legacy=0x7f8ce4071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.945+0000 7f8ce3e00640 1 -- 172.21.15.40:0/1421237790 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce4161df0 con 0x7f8ce41628d0 2024-03-24T00:45:04.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.949+0000 7f8ce3e00640 1 --2- 172.21.15.40:0/1421237790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ce41628d0 0x7f8ce40b3bc0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0009870 tx=0x7f8cd001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47c98fa2e4fd25c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.949+0000 7f8ce3400640 1 -- 172.21.15.40:0/1421237790 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd00b2070 con 0x7f8ce41628d0 2024-03-24T00:45:04.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.949+0000 7f8ce3400640 1 -- 172.21.15.40:0/1421237790 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd00a9070 con 0x7f8ce41628d0 2024-03-24T00:45:04.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.949+0000 7f8ce3400640 1 -- 172.21.15.40:0/1421237790 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd00ad880 con 0x7f8ce41628d0 2024-03-24T00:45:04.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.949+0000 7f8ceba00640 1 -- 172.21.15.40:0/1421237790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ce41628d0 msgr2=0x7f8ce40b3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.949+0000 7f8ceba00640 1 --2- 172.21.15.40:0/1421237790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ce41628d0 0x7f8ce40b3bc0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f8cd0009870 tx=0x7f8cd001c560 comp rx=0 tx=0).stop 2024-03-24T00:45:04.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.949+0000 7f8ceba00640 1 -- 172.21.15.40:0/1421237790 shutdown_connections 2024-03-24T00:45:04.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.949+0000 7f8ceba00640 1 --2- 172.21.15.40:0/1421237790 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ce41628d0 0x7f8ce40b3bc0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.949+0000 7f8ceba00640 1 --2- 172.21.15.40:0/1421237790 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ce415e000 0x7f8ce40d16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.949+0000 7f8ceba00640 1 -- 172.21.15.40:0/1421237790 >> 172.21.15.40:0/1421237790 conn(0x7f8ce40d0bb0 msgr2=0x7f8ce40d0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:04.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.949+0000 7f8ceba00640 1 -- 172.21.15.40:0/1421237790 shutdown_connections 2024-03-24T00:45:04.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.949+0000 7f8ceba00640 1 -- 172.21.15.40:0/1421237790 wait complete. 2024-03-24T00:45:04.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ceba00640 1 Processor -- start 2024-03-24T00:45:04.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ceba00640 1 -- start start 2024-03-24T00:45:04.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ceba00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ce406f840 0x7f8ce414e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ceba00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ce415e000 0x7f8ce4159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ceba00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ce41628d0 0x7f8ce4155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ceba00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce416b0c0 con 0x7f8ce41628d0 2024-03-24T00:45:04.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ceba00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce416af40 con 0x7f8ce406f840 2024-03-24T00:45:04.956 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ceba00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce416b240 con 0x7f8ce415e000 2024-03-24T00:45:04.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ce41628d0 0x7f8ce4155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce9000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ce406f840 0x7f8ce414e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce9a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ce41628d0 0x7f8ce4155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34342/0 (socket says 172.21.15.40:34342) 2024-03-24T00:45:04.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce9a00640 1 -- 172.21.15.40:0/3640145942 learned_addr learned my addr 172.21.15.40:0/3640145942 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:04.957 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce3e00640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ce415e000 0x7f8ce4159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce3e00640 1 -- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ce406f840 msgr2=0x7f8ce414e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce3e00640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ce406f840 0x7f8ce414e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce3e00640 1 -- 172.21.15.40:0/3640145942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ce41628d0 msgr2=0x7f8ce4155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:04.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce3e00640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ce41628d0 0x7f8ce4155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:04.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce3e00640 1 -- 172.21.15.40:0/3640145942 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce414ec50 con 0x7f8ce415e000 2024-03-24T00:45:04.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce9a00640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ce41628d0 0x7f8ce4155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:04.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce3e00640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ce415e000 0x7f8ce4159af0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8ce406d670 tx=0x7f8cd0019300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce0c00640 1 -- 172.21.15.40:0/3640145942 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd00b2040 con 0x7f8ce415e000 2024-03-24T00:45:04.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce0c00640 1 -- 172.21.15.40:0/3640145942 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd00be070 con 0x7f8ce415e000 2024-03-24T00:45:04.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ce0c00640 1 -- 172.21.15.40:0/3640145942 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cd00b8660 con 0x7f8ce415e000 2024-03-24T00:45:04.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ceba00640 1 -- 172.21.15.40:0/3640145942 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce41564d0 con 0x7f8ce415e000 2024-03-24T00:45:04.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ceba00640 1 -- 172.21.15.40:0/3640145942 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce4159010 con 0x7f8ce415e000 2024-03-24T00:45:04.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.953+0000 7f8ceba00640 1 -- 172.21.15.40:0/3640145942 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce4068170 con 0x7f8ce415e000 2024-03-24T00:45:04.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.957+0000 7f8ce0c00640 1 -- 172.21.15.40:0/3640145942 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8cd00ab070 con 0x7f8ce415e000 2024-03-24T00:45:04.964 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.957+0000 7f8ce0c00640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8cc4067660 0x7f8cc4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:04.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.957+0000 7f8ce0c00640 1 -- 172.21.15.40:0/3640145942 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f8cd01200f0 con 0x7f8ce415e000 2024-03-24T00:45:04.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.957+0000 7f8ce9000640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8cc4067660 0x7f8cc4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:04.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.961+0000 7f8ce9000640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8cc4067660 0x7f8cc4069b20 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8009b60 tx=0x7f8cd8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:04.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:04.961+0000 7f8ce0c00640 1 -- 172.21.15.40:0/3640145942 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8cd0016150 con 0x7f8ce415e000 2024-03-24T00:45:05.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 -- 172.21.15.40:0/3640145942 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8ce40d19c0 con 0x7f8ce415e000 2024-03-24T00:45:05.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ce0c00640 1 -- 172.21.15.40:0/3640145942 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v122) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8cd00f0770 con 0x7f8ce415e000 2024-03-24T00:45:05.127 INFO:teuthology.orchestra.run.smithi040.stdout:pg_num: 64 2024-03-24T00:45:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 -- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8cc4067660 msgr2=0x7f8cc4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8cc4067660 0x7f8cc4069b20 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f8cd8009b60 tx=0x7f8cd8009190 comp rx=0 tx=0).stop 2024-03-24T00:45:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 -- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ce415e000 msgr2=0x7f8ce4159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ce415e000 0x7f8ce4159af0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8ce406d670 tx=0x7f8cd0019300 comp rx=0 tx=0).stop 2024-03-24T00:45:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 -- 172.21.15.40:0/3640145942 shutdown_connections 2024-03-24T00:45:05.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f8cc4067660 0x7f8cc4069b20 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f8ce41628d0 0x7f8ce4155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ce415e000 0x7f8ce4159af0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 --2- 172.21.15.40:0/3640145942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ce406f840 0x7f8ce414e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 -- 172.21.15.40:0/3640145942 >> 172.21.15.40:0/3640145942 conn(0x7f8ce40d0bb0 msgr2=0x7f8ce40b12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:05.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 -- 172.21.15.40:0/3640145942 shutdown_connections 2024-03-24T00:45:05.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.125+0000 7f8ceba00640 1 -- 172.21.15.40:0/3640145942 wait complete. 2024-03-24T00:45:05.148 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:45:05.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.285+0000 7f821ee00640 1 Processor -- start 2024-03-24T00:45:05.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.285+0000 7f821ee00640 1 -- start start 2024-03-24T00:45:05.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.285+0000 7f821ee00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8218141bc0 0x7f82180ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:05.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.285+0000 7f821ee00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82180ce630 0x7f8218071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:05.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.285+0000 7f821ee00640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8218070e00 con 0x7f821813ec80 2024-03-24T00:45:05.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.285+0000 7f821ee00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82180cee20 con 0x7f8218141bc0 2024-03-24T00:45:05.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.285+0000 7f821ee00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82180cf100 con 0x7f82180ce630 2024-03-24T00:45:05.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.289+0000 7f8217e00640 1 --1- >> v1:172.21.15.40:6789/0 conn(0x7f821813ec80 0x7f82181414b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.40:6789/0 says I am v1:172.21.15.40:53000/0 (socket says 172.21.15.40:53000) 2024-03-24T00:45:05.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.289+0000 7f8217e00640 1 -- 172.21.15.40:0/485576683 learned_addr learned my addr 172.21.15.40:0/485576683 (peer_addr_for_me v1:172.21.15.40:0/0) 2024-03-24T00:45:05.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.289+0000 7f820f400640 1 --2- 172.21.15.40:0/485576683 >> v2:172.21.15.53:3300/0 conn(0x7f8218141bc0 0x7f82180ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:05.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.289+0000 7f821ce00640 1 --2- 172.21.15.40:0/485576683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82180ce630 0x7f8218071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:05.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.289+0000 7f8217400640 1 -- 172.21.15.40:0/485576683 <== mon.0 v1:172.21.15.40:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076928996 0 0) 0x7f8218070e00 con 0x7f821813ec80 2024-03-24T00:45:05.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.289+0000 7f8217400640 1 -- 172.21.15.40:0/485576683 --> v1:172.21.15.40:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81fc003530 con 0x7f821813ec80 2024-03-24T00:45:05.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.289+0000 7f820f400640 1 -- 172.21.15.40:0/485576683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82180ce630 msgr2=0x7f8218071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:05.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.289+0000 7f820f400640 1 --2- 172.21.15.40:0/485576683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82180ce630 0x7f8218071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.289+0000 7f820f400640 1 -- 172.21.15.40:0/485576683 >> v1:172.21.15.40:6789/0 conn(0x7f821813ec80 legacy=0x7f82181414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.289+0000 7f8217400640 1 -- 172.21.15.40:0/485576683 <== mon.0 v1:172.21.15.40:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1930437556 0 0) 0x7f81fc003530 con 0x7f821813ec80 2024-03-24T00:45:05.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.289+0000 7f820f400640 1 -- 172.21.15.40:0/485576683 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82180cf3e0 con 0x7f8218141bc0 2024-03-24T00:45:05.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.289+0000 7f820f400640 1 --2- 172.21.15.40:0/485576683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8218141bc0 0x7f82180ce0f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f8200009d90 tx=0x7f820001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dad8bd662b95a41a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:05.294 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f8217400640 1 -- 172.21.15.40:0/485576683 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82000b2070 con 0x7f8218141bc0 2024-03-24T00:45:05.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f8217400640 1 -- 172.21.15.40:0/485576683 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82000a56b0 con 0x7f8218141bc0 2024-03-24T00:45:05.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f8217400640 1 -- 172.21.15.40:0/485576683 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8208001680 con 0x7f8218141bc0 2024-03-24T00:45:05.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 -- 172.21.15.40:0/485576683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8218141bc0 msgr2=0x7f82180ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 --2- 172.21.15.40:0/485576683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8218141bc0 0x7f82180ce0f0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f8200009d90 tx=0x7f820001c890 comp rx=0 tx=0).stop 2024-03-24T00:45:05.295 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 -- 172.21.15.40:0/485576683 shutdown_connections 2024-03-24T00:45:05.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 --2- 172.21.15.40:0/485576683 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82180ce630 0x7f8218071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 --2- 172.21.15.40:0/485576683 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8218141bc0 0x7f82180ce0f0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 -- 172.21.15.40:0/485576683 >> 172.21.15.40:0/485576683 conn(0x7f82180b17b0 msgr2=0x7f82180b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:05.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 -- 172.21.15.40:0/485576683 shutdown_connections 2024-03-24T00:45:05.296 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 -- 172.21.15.40:0/485576683 wait complete. 2024-03-24T00:45:05.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 Processor -- start 2024-03-24T00:45:05.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 -- start start 2024-03-24T00:45:05.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82180ce630 0x7f821813ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:05.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821813ec80 0x7f8218155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:05.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8218141bc0 0x7f821806df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:05.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8218072460 con 0x7f821813ec80 2024-03-24T00:45:05.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82180722e0 con 0x7f8218141bc0 2024-03-24T00:45:05.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f821ee00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82180725e0 con 0x7f82180ce630 2024-03-24T00:45:05.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f820f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821813ec80 0x7f8218155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:05.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.293+0000 7f820f400640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821813ec80 0x7f8218155b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34358/0 (socket says 172.21.15.40:34358) 2024-03-24T00:45:05.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f820f400640 1 -- 172.21.15.40:0/1068158095 learned_addr learned my addr 172.21.15.40:0/1068158095 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:05.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f8217e00640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82180ce630 0x7f821813ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:05.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f821ce00640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8218141bc0 0x7f821806df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:05.299 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f820f400640 1 -- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82180ce630 msgr2=0x7f821813ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f820f400640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82180ce630 0x7f821813ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f820f400640 1 -- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8218141bc0 msgr2=0x7f821806df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f820f400640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8218141bc0 0x7f821806df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f820f400640 1 -- 172.21.15.40:0/1068158095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82181560a0 con 0x7f821813ec80 2024-03-24T00:45:05.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f821ce00640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8218141bc0 0x7f821806df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:45:05.300 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f820f400640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821813ec80 0x7f8218155b60 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f820001af40 tx=0x7f8200019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:05.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f8214c00640 1 -- 172.21.15.40:0/1068158095 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82000b2040 con 0x7f821813ec80 2024-03-24T00:45:05.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f8214c00640 1 -- 172.21.15.40:0/1068158095 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82000bd070 con 0x7f821813ec80 2024-03-24T00:45:05.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f821ee00640 1 -- 172.21.15.40:0/1068158095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8218139b70 con 0x7f821813ec80 2024-03-24T00:45:05.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f8214c00640 1 -- 172.21.15.40:0/1068158095 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82000a5ca0 con 0x7f821813ec80 2024-03-24T00:45:05.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f821ee00640 1 -- 172.21.15.40:0/1068158095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821813a130 con 0x7f821813ec80 2024-03-24T00:45:05.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f821ee00640 1 -- 172.21.15.40:0/1068158095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8218068170 con 0x7f821813ec80 2024-03-24T00:45:05.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.297+0000 7f8214c00640 1 -- 172.21.15.40:0/1068158095 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7f8200004040 con 0x7f821813ec80 2024-03-24T00:45:05.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.301+0000 7f8214c00640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81f4067660 0x7f81f4069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:05.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.301+0000 7f8214c00640 1 -- 172.21.15.40:0/1068158095 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f820011ffe0 con 0x7f821813ec80 2024-03-24T00:45:05.303 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.301+0000 7f8217e00640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81f4067660 0x7f81f4069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:05.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.301+0000 7f8217e00640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81f4067660 0x7f81f4069b20 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8208007950 tx=0x7f8208009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:05.311 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.309+0000 7f8214c00640 1 -- 172.21.15.40:0/1068158095 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7f8200016480 con 0x7f821813ec80 2024-03-24T00:45:05.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.505+0000 7f821ee00640 1 -- 172.21.15.40:0/1068158095 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82180b2030 con 0x7f821813ec80 2024-03-24T00:45:05.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.505+0000 7f8214c00640 1 -- 172.21.15.40:0/1068158095 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+4281 (secure 0 0 0) 0x7f82000f0cb0 con 0x7f821813ec80 2024-03-24T00:45:05.510 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:45:05.510 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":145,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:39:07.531884+0000","modified":"2024-03-24T00:43:12.194456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8877,"mds_1":8874,"mds_2":7439},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7439":{"gid":7439,"name":"d","rank":2,"incarnation":136,"state":"up:active","state_seq":20,"addr":"172.21.15.53:6837/1582567544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1582567544},{"type":"v1","addr":"172.21.15.53:6837","nonce":1582567544}]},"join_fscid":-1,"laggy_since":"2024-03-24T00:40:21.913457+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8874":{"gid":8874,"name":"c","rank":1,"incarnation":130,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6833/4273998586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4273998586},{"type":"v1","addr":"172.21.15.40:6833","nonce":4273998586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8877":{"gid":8877,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":64,"addr":"172.21.15.40:6835/773071485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":773071485},{"type":"v1","addr":"172.21.15.40:6835","nonce":773071485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8877,"qdb_cluster":[7439,8874,8877]},"id":9}]} 2024-03-24T00:45:05.510 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 145 2024-03-24T00:45:05.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.509+0000 7f821ee00640 1 -- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81f4067660 msgr2=0x7f81f4069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.509+0000 7f821ee00640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81f4067660 0x7f81f4069b20 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f8208007950 tx=0x7f8208009040 comp rx=0 tx=0).stop 2024-03-24T00:45:05.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.509+0000 7f821ee00640 1 -- 172.21.15.40:0/1068158095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821813ec80 msgr2=0x7f8218155b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.509+0000 7f821ee00640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821813ec80 0x7f8218155b60 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f820001af40 tx=0x7f8200019630 comp rx=0 tx=0).stop 2024-03-24T00:45:05.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.509+0000 7f821ee00640 1 -- 172.21.15.40:0/1068158095 shutdown_connections 2024-03-24T00:45:05.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.509+0000 7f821ee00640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7f81f4067660 0x7f81f4069b20 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.509+0000 7f821ee00640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8218141bc0 0x7f821806df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.509+0000 7f821ee00640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7f821813ec80 0x7f8218155b60 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.509+0000 7f821ee00640 1 --2- 172.21.15.40:0/1068158095 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82180ce630 0x7f821813ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.509+0000 7f821ee00640 1 -- 172.21.15.40:0/1068158095 >> 172.21.15.40:0/1068158095 conn(0x7f82180b17b0 msgr2=0x7f821815e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:05.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.509+0000 7f821ee00640 1 -- 172.21.15.40:0/1068158095 shutdown_connections 2024-03-24T00:45:05.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.509+0000 7f821ee00640 1 -- 172.21.15.40:0/1068158095 wait complete. 2024-03-24T00:45:05.530 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:45:05.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.669+0000 7fab3a000640 1 Processor -- start 2024-03-24T00:45:05.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.669+0000 7fab3a000640 1 -- start start 2024-03-24T00:45:05.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.669+0000 7fab3a000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab340d08e0 0x7fab3415f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:05.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.669+0000 7fab3a000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab340b43a0 0x7fab3415f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:05.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.669+0000 7fab3a000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab340af710 con 0x7fab340cfdf0 2024-03-24T00:45:05.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.669+0000 7fab3a000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab340b2bb0 con 0x7fab340d08e0 2024-03-24T00:45:05.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.669+0000 7fab3a000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab340b26a0 con 0x7fab340b43a0 2024-03-24T00:45:05.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.669+0000 7fab32a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab340d08e0 0x7fab3415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:05.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.669+0000 7fab32a00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fab340d08e0 0x7fab3415f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34368/0 (socket says 172.21.15.40:34368) 2024-03-24T00:45:05.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.669+0000 7fab32a00640 1 -- 172.21.15.40:0/1305426390 learned_addr learned my addr 172.21.15.40:0/1305426390 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:05.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.669+0000 7fab33e00640 1 --2- 172.21.15.40:0/1305426390 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab340b43a0 0x7fab3415f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:05.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.673+0000 7fab32a00640 1 -- 172.21.15.40:0/1305426390 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab340b43a0 msgr2=0x7fab3415f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:05.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.673+0000 7fab32a00640 1 --2- 172.21.15.40:0/1305426390 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab340b43a0 0x7fab3415f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.673+0000 7fab32a00640 1 -- 172.21.15.40:0/1305426390 >> v1:172.21.15.40:6789/0 conn(0x7fab340cfdf0 legacy=0x7fab340d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.673+0000 7fab32a00640 1 -- 172.21.15.40:0/1305426390 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab340b34f0 con 0x7fab340d08e0 2024-03-24T00:45:05.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.673+0000 7fab32a00640 1 --2- 172.21.15.40:0/1305426390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab340d08e0 0x7fab3415f090 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fab20009870 tx=0x7fab2001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16c09fb1256656cd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:05.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.673+0000 7fab32000640 1 -- 172.21.15.40:0/1305426390 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab200a7070 con 0x7fab340d08e0 2024-03-24T00:45:05.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.673+0000 7fab32000640 1 -- 172.21.15.40:0/1305426390 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab200b0070 con 0x7fab340d08e0 2024-03-24T00:45:05.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.673+0000 7fab32000640 1 -- 172.21.15.40:0/1305426390 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab200ab890 con 0x7fab340d08e0 2024-03-24T00:45:05.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.673+0000 7fab3a000640 1 -- 172.21.15.40:0/1305426390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab340d08e0 msgr2=0x7fab3415f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.673+0000 7fab3a000640 1 --2- 172.21.15.40:0/1305426390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab340d08e0 0x7fab3415f090 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fab20009870 tx=0x7fab2001c560 comp rx=0 tx=0).stop 2024-03-24T00:45:05.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.677+0000 7fab3a000640 1 -- 172.21.15.40:0/1305426390 shutdown_connections 2024-03-24T00:45:05.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.677+0000 7fab3a000640 1 --2- 172.21.15.40:0/1305426390 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab340b43a0 0x7fab3415f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.677+0000 7fab3a000640 1 --2- 172.21.15.40:0/1305426390 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab340d08e0 0x7fab3415f090 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.677+0000 7fab3a000640 1 -- 172.21.15.40:0/1305426390 >> 172.21.15.40:0/1305426390 conn(0x7fab340b3880 msgr2=0x7fab340b3c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:05.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.677+0000 7fab3a000640 1 -- 172.21.15.40:0/1305426390 shutdown_connections 2024-03-24T00:45:05.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.677+0000 7fab3a000640 1 -- 172.21.15.40:0/1305426390 wait complete. 2024-03-24T00:45:05.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.677+0000 7fab3a000640 1 Processor -- start 2024-03-24T00:45:05.681 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.677+0000 7fab3a000640 1 -- start start 2024-03-24T00:45:05.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab3a000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab340b43a0 0x7fab3406fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:05.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab3a000640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab340cfdf0 0x7fab340b6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:05.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab3a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab341f83d0 0x7fab341f8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:05.683 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab3a000640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab340b0810 con 0x7fab340cfdf0 2024-03-24T00:45:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab3a000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab340b0690 con 0x7fab341f83d0 2024-03-24T00:45:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab3a000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab340b0990 con 0x7fab340b43a0 2024-03-24T00:45:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab32a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab340cfdf0 0x7fab340b6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab32a00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab340cfdf0 0x7fab340b6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34382/0 (socket says 172.21.15.40:34382) 2024-03-24T00:45:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab32a00640 1 -- 172.21.15.40:0/396210253 learned_addr learned my addr 172.21.15.40:0/396210253 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab33400640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab340b43a0 0x7fab3406fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab33e00640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab341f83d0 0x7fab341f8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab32a00640 1 -- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab340b43a0 msgr2=0x7fab3406fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab32a00640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab340b43a0 0x7fab3406fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab32a00640 1 -- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab341f83d0 msgr2=0x7fab341f8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab32a00640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab341f83d0 0x7fab341f8860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab32a00640 1 -- 172.21.15.40:0/396210253 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab340c9090 con 0x7fab340cfdf0 2024-03-24T00:45:05.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab33400640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab340b43a0 0x7fab3406fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:45:05.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab32a00640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab340cfdf0 0x7fab340b6c40 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fab200ad040 tx=0x7fab2001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:05.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab2b400640 1 -- 172.21.15.40:0/396210253 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab200b4800 con 0x7fab340cfdf0 2024-03-24T00:45:05.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab2b400640 1 -- 172.21.15.40:0/396210253 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab200b49a0 con 0x7fab340cfdf0 2024-03-24T00:45:05.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab2b400640 1 -- 172.21.15.40:0/396210253 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab200b4d90 con 0x7fab340cfdf0 2024-03-24T00:45:05.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab3a000640 1 -- 172.21.15.40:0/396210253 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab34070100 con 0x7fab340cfdf0 2024-03-24T00:45:05.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab3a000640 1 -- 172.21.15.40:0/396210253 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab340c7ae0 con 0x7fab340cfdf0 2024-03-24T00:45:05.686 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.681+0000 7fab3a000640 1 -- 172.21.15.40:0/396210253 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab34068170 con 0x7fab340cfdf0 2024-03-24T00:45:05.687 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.685+0000 7fab2b400640 1 -- 172.21.15.40:0/396210253 <== mon.0 v2:172.21.15.40:3300/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fab200c7390 con 0x7fab340cfdf0 2024-03-24T00:45:05.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.685+0000 7fab2b400640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab14067690 0x7fab14069b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:05.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.685+0000 7fab2b400640 1 -- 172.21.15.40:0/396210253 <== mon.0 v2:172.21.15.40:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fab2011ffe0 con 0x7fab340cfdf0 2024-03-24T00:45:05.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.685+0000 7fab33400640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab14067690 0x7fab14069b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:05.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.685+0000 7fab33400640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab14067690 0x7fab14069b50 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fab24000f70 tx=0x7fab24009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:05.692 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.689+0000 7fab2b400640 1 -- 172.21.15.40:0/396210253 <== mon.0 v2:172.21.15.40:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fab200c4060 con 0x7fab340cfdf0 2024-03-24T00:45:05.858 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.853+0000 7fab3a000640 1 -- 172.21.15.40:0/396210253 --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab340c85a0 con 0x7fab340cfdf0 2024-03-24T00:45:05.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab2b400640 1 -- 172.21.15.40:0/396210253 <== mon.0 v2:172.21.15.40:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+16789 (secure 0 0 0) 0x7fab340c9090 con 0x7fab340cfdf0 2024-03-24T00:45:05.860 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:45:05.860 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":122,"fsid":"185b869e-2a51-452f-88b9-d083720a50b9","created":"2024-03-24T00:30:41.988772+0000","modified":"2024-03-24T00:40:01.907736+0000","last_up_change":"2024-03-24T00:30:52.990515+0000","last_in_change":"2024-03-24T00:30:44.540889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:30:53.260096+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:39:05.964092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-03-24T00:39:06.525558+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"119","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"449203c7-93f4-418d-beef-133d2e26dc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6810","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6813","nonce":201988360}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6815","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6818","nonce":201988360}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6826","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6829","nonce":201988360}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6821","nonce":201988360},{"type":"v1","addr":"172.21.15.40:6824","nonce":201988360}]},"public_addr":"172.21.15.40:6813/201988360","cluster_addr":"172.21.15.40:6818/201988360","heartbeat_back_addr":"172.21.15.40:6829/201988360","heartbeat_front_addr":"172.21.15.40:6824/201988360","state":["exists","up"]},{"osd":1,"uuid":"8f8cbfbc-88e3-423a-a729-0586b4f90926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6808","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6812","nonce":4268319376}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6816","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6820","nonce":4268319376}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6830","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6831","nonce":4268319376}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6823","nonce":4268319376},{"type":"v1","addr":"172.21.15.40:6828","nonce":4268319376}]},"public_addr":"172.21.15.40:6812/4268319376","cluster_addr":"172.21.15.40:6820/4268319376","heartbeat_back_addr":"172.21.15.40:6831/4268319376","heartbeat_front_addr":"172.21.15.40:6828/4268319376","state":["exists","up"]},{"osd":2,"uuid":"66b79393-78ff-48c2-9fa2-e06f21854c29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6801","nonce":2411062682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6803","nonce":2411062682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6807","nonce":2411062682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":2411062682},{"type":"v1","addr":"172.21.15.40:6805","nonce":2411062682}]},"public_addr":"172.21.15.40:6801/2411062682","cluster_addr":"172.21.15.40:6803/2411062682","heartbeat_back_addr":"172.21.15.40:6807/2411062682","heartbeat_front_addr":"172.21.15.40:6805/2411062682","state":["exists","up"]},{"osd":3,"uuid":"29eacc84-6841-4519-ab42-3fed72243aa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6809","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6811","nonce":2475081617}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6814","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6817","nonce":2475081617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6825","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6827","nonce":2475081617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6819","nonce":2475081617},{"type":"v1","addr":"172.21.15.40:6822","nonce":2475081617}]},"public_addr":"172.21.15.40:6811/2475081617","cluster_addr":"172.21.15.40:6817/2475081617","heartbeat_back_addr":"172.21.15.40:6827/2475081617","heartbeat_front_addr":"172.21.15.40:6822/2475081617","state":["exists","up"]},{"osd":4,"uuid":"e9144226-cc4c-4d16-9c5f-8940de541a44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6803","nonce":3270909220}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6806","nonce":3270909220}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6812","nonce":3270909220}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":3270909220},{"type":"v1","addr":"172.21.15.53:6809","nonce":3270909220}]},"public_addr":"172.21.15.53:6803/3270909220","cluster_addr":"172.21.15.53:6806/3270909220","heartbeat_back_addr":"172.21.15.53:6812/3270909220","heartbeat_front_addr":"172.21.15.53:6809/3270909220","state":["exists","up"]},{"osd":5,"uuid":"c5a3943d-da15-4f7b-be59-3fa00b734007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6801","nonce":200349945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6807","nonce":200349945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6815","nonce":200349945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":200349945},{"type":"v1","addr":"172.21.15.53:6813","nonce":200349945}]},"public_addr":"172.21.15.53:6801/200349945","cluster_addr":"172.21.15.53:6807/200349945","heartbeat_back_addr":"172.21.15.53:6815/200349945","heartbeat_front_addr":"172.21.15.53:6813/200349945","state":["exists","up"]},{"osd":6,"uuid":"3daafa9e-c4d2-4fb1-ba4e-db255d3e20c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6817","nonce":3127578810}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6819","nonce":3127578810}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6823","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6825","nonce":3127578810}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3127578810},{"type":"v1","addr":"172.21.15.53:6822","nonce":3127578810}]},"public_addr":"172.21.15.53:6817/3127578810","cluster_addr":"172.21.15.53:6819/3127578810","heartbeat_back_addr":"172.21.15.53:6825/3127578810","heartbeat_front_addr":"172.21.15.53:6822/3127578810","state":["exists","up"]},{"osd":7,"uuid":"ada23732-17bc-4ce0-b7e0-8974631732b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":117,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6824","nonce":2592328924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6827","nonce":2592328924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6831","nonce":2592328924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2592328924},{"type":"v1","addr":"172.21.15.53:6829","nonce":2592328924}]},"public_addr":"172.21.15.53:6824/2592328924","cluster_addr":"172.21.15.53:6827/2592328924","heartbeat_back_addr":"172.21.15.53:6831/2592328924","heartbeat_front_addr":"172.21.15.53:6829/2592328924","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.095074+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.198627+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:49.967101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.204054+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.051456+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.108486+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.193501+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:30:50.123814+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.c","mappings":[{"from":2,"to":0}]},{"pgid":"19.23","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6836/2255755838":"2024-03-25T00:40:01.907668+0000","172.21.15.53:6834/2255755838":"2024-03-25T00:40:01.907668+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:45:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab3a000640 1 -- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab14067690 msgr2=0x7fab14069b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab3a000640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab14067690 0x7fab14069b50 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fab24000f70 tx=0x7fab24009040 comp rx=0 tx=0).stop 2024-03-24T00:45:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab3a000640 1 -- 172.21.15.40:0/396210253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab340cfdf0 msgr2=0x7fab340b6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab3a000640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab340cfdf0 0x7fab340b6c40 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fab200ad040 tx=0x7fab2001ada0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab3a000640 1 -- 172.21.15.40:0/396210253 shutdown_connections 2024-03-24T00:45:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab3a000640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fab14067690 0x7fab14069b50 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab3a000640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab341f83d0 0x7fab341f8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab3a000640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fab340cfdf0 0x7fab340b6c40 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab3a000640 1 --2- 172.21.15.40:0/396210253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab340b43a0 0x7fab3406fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:05.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab3a000640 1 -- 172.21.15.40:0/396210253 >> 172.21.15.40:0/396210253 conn(0x7fab340b3880 msgr2=0x7fab340d1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:05.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab3a000640 1 -- 172.21.15.40:0/396210253 shutdown_connections 2024-03-24T00:45:05.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:05.857+0000 7fab3a000640 1 -- 172.21.15.40:0/396210253 wait complete. 2024-03-24T00:45:05.882 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T00:45:05.883 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:45:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.021+0000 7fd452e00640 1 Processor -- start 2024-03-24T00:45:06.024 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.021+0000 7fd452e00640 1 -- start start 2024-03-24T00:45:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.021+0000 7fd452e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd44c06e620 0x7fd44c0b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.021+0000 7fd452e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd44c06ea00 0x7fd44c0b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.021+0000 7fd452e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd44c070f60 con 0x7fd44c0b2ff0 2024-03-24T00:45:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.021+0000 7fd452e00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd44c0b4c00 con 0x7fd44c06e620 2024-03-24T00:45:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.021+0000 7fd452e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd44c0b33d0 con 0x7fd44c06ea00 2024-03-24T00:45:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.021+0000 7fd44be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd44c06e620 0x7fd44c0b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.021+0000 7fd44be00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7fd44c06e620 0x7fd44c0b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.40:3300/0 says I am v2:172.21.15.40:34394/0 (socket says 172.21.15.40:34394) 2024-03-24T00:45:06.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.021+0000 7fd44be00640 1 -- 172.21.15.40:0/2648227181 learned_addr learned my addr 172.21.15.40:0/2648227181 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.025+0000 7fd443400640 1 --2- 172.21.15.40:0/2648227181 >> v2:172.21.15.53:3300/0 conn(0x7fd44c06ea00 0x7fd44c0b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.025+0000 7fd44be00640 1 -- 172.21.15.40:0/2648227181 >> v2:172.21.15.53:3300/0 conn(0x7fd44c06ea00 msgr2=0x7fd44c0b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:45:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.025+0000 7fd44be00640 1 --2- 172.21.15.40:0/2648227181 >> v2:172.21.15.53:3300/0 conn(0x7fd44c06ea00 0x7fd44c0b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.025+0000 7fd44be00640 1 -- 172.21.15.40:0/2648227181 >> v1:172.21.15.53:6789/0 conn(0x7fd44c0b2ff0 legacy=0x7fd44c0ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.025+0000 7fd44be00640 1 -- 172.21.15.40:0/2648227181 --> v2:172.21.15.40:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c0b3620 con 0x7fd44c06e620 2024-03-24T00:45:06.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.025+0000 7fd44be00640 1 --2- 172.21.15.40:0/2648227181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44c06e620 0x7fd44c0b46c0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fd44c06caa0 tx=0x7fd43c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c8318af33bddee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:06.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.025+0000 7fd44b400640 1 -- 172.21.15.40:0/2648227181 <== mon.0 v2:172.21.15.40:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0ab660 con 0x7fd44c06e620 2024-03-24T00:45:06.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.025+0000 7fd44b400640 1 -- 172.21.15.40:0/2648227181 <== mon.0 v2:172.21.15.40:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43c0b0070 con 0x7fd44c06e620 2024-03-24T00:45:06.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.025+0000 7fd44b400640 1 -- 172.21.15.40:0/2648227181 <== mon.0 v2:172.21.15.40:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0aba20 con 0x7fd44c06e620 2024-03-24T00:45:06.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 -- 172.21.15.40:0/2648227181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44c06e620 msgr2=0x7fd44c0b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 --2- 172.21.15.40:0/2648227181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44c06e620 0x7fd44c0b46c0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fd44c06caa0 tx=0x7fd43c01c460 comp rx=0 tx=0).stop 2024-03-24T00:45:06.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 -- 172.21.15.40:0/2648227181 shutdown_connections 2024-03-24T00:45:06.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 --2- 172.21.15.40:0/2648227181 >> v2:172.21.15.53:3300/0 conn(0x7fd44c06ea00 0x7fd44c0b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 --2- 172.21.15.40:0/2648227181 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44c06e620 0x7fd44c0b46c0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 -- 172.21.15.40:0/2648227181 >> 172.21.15.40:0/2648227181 conn(0x7fd44c0b17b0 msgr2=0x7fd44c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:06.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 -- 172.21.15.40:0/2648227181 shutdown_connections 2024-03-24T00:45:06.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 -- 172.21.15.40:0/2648227181 wait complete. 2024-03-24T00:45:06.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 Processor -- start 2024-03-24T00:45:06.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 -- start start 2024-03-24T00:45:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd44c06e620 0x7fd44c0b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd44c06ea00 0x7fd44c159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 --2- >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44c0b2ff0 0x7fd44c1419f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 -- --> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c144920 con 0x7fd44c0b2ff0 2024-03-24T00:45:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c0b41a0 con 0x7fd44c06ea00 2024-03-24T00:45:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd452e00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c13c8f0 con 0x7fd44c06e620 2024-03-24T00:45:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd44be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd44c06e620 0x7fd44c0b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd44be00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd44c06e620 0x7fd44c0b1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.40:37522/0 (socket says 172.21.15.40:37522) 2024-03-24T00:45:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd44be00640 1 -- 172.21.15.40:0/2567643458 learned_addr learned my addr 172.21.15.40:0/2567643458 (peer_addr_for_me v2:172.21.15.40:0/0) 2024-03-24T00:45:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd443400640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd44c06ea00 0x7fd44c159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.029+0000 7fd450e00640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44c0b2ff0 0x7fd44c1419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd44be00640 1 -- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd44c06ea00 msgr2=0x7fd44c159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd44be00640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd44c06ea00 0x7fd44c159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd44be00640 1 -- 172.21.15.40:0/2567643458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44c0b2ff0 msgr2=0x7fd44c1419f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd44be00640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44c0b2ff0 0x7fd44c1419f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd44be00640 1 -- 172.21.15.40:0/2567643458 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c15d4a0 con 0x7fd44c06e620 2024-03-24T00:45:06.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd443400640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd44c06ea00 0x7fd44c159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:45:06.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd44be00640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd44c06e620 0x7fd44c0b1190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0ad040 tx=0x7fd43c01aca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:06.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd448c00640 1 -- 172.21.15.40:0/2567643458 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0be5f0 con 0x7fd44c06e620 2024-03-24T00:45:06.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd448c00640 1 -- 172.21.15.40:0/2567643458 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43c0be790 con 0x7fd44c06e620 2024-03-24T00:45:06.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd452e00640 1 -- 172.21.15.40:0/2567643458 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd44c1429e0 con 0x7fd44c06e620 2024-03-24T00:45:06.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd448c00640 1 -- 172.21.15.40:0/2567643458 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c0beb40 con 0x7fd44c06e620 2024-03-24T00:45:06.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd452e00640 1 -- 172.21.15.40:0/2567643458 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd44c15d760 con 0x7fd44c06e620 2024-03-24T00:45:06.037 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd452e00640 1 -- 172.21.15.40:0/2567643458 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd44c068170 con 0x7fd44c06e620 2024-03-24T00:45:06.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.033+0000 7fd448c00640 1 -- 172.21.15.40:0/2567643458 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) v1 ==== 87083+0+0 (secure 0 0 0) 0x7fd43c0b43f0 con 0x7fd44c06e620 2024-03-24T00:45:06.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.037+0000 7fd448c00640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd428067660 0x7fd428069b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.037+0000 7fd448c00640 1 -- 172.21.15.40:0/2567643458 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fd43c120870 con 0x7fd44c06e620 2024-03-24T00:45:06.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.037+0000 7fd443400640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd428067660 0x7fd428069b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:45:06.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.037+0000 7fd443400640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd428067660 0x7fd428069b20 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd44c069520 tx=0x7fd43400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:45:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.037+0000 7fd448c00640 1 -- 172.21.15.40:0/2567643458 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177849 (secure 0 0 0) 0x7fd43c016110 con 0x7fd44c06e620 2024-03-24T00:45:06.237 INFO:tasks.ceph.osd.0:Sent signal 15 2024-03-24T00:45:06.238 INFO:tasks.ceph.osd.1:Sent signal 15 2024-03-24T00:45:06.238 INFO:tasks.ceph.osd.2:Sent signal 15 2024-03-24T00:45:06.238 INFO:tasks.ceph.osd.3:Sent signal 15 2024-03-24T00:45:06.238 INFO:tasks.ceph.osd.4:Sent signal 15 2024-03-24T00:45:06.239 INFO:tasks.ceph.osd.5:Sent signal 15 2024-03-24T00:45:06.239 INFO:tasks.ceph.osd.6:Sent signal 15 2024-03-24T00:45:06.239 INFO:tasks.ceph.osd.7:Sent signal 15 2024-03-24T00:45:06.239 INFO:tasks.ceph.mds.a:Sent signal 15 2024-03-24T00:45:06.239 INFO:tasks.ceph.mds.c:Sent signal 15 2024-03-24T00:45:06.240 INFO:tasks.ceph.mon.a:Sent signal 15 2024-03-24T00:45:06.240 INFO:tasks.ceph.mon.b:Sent signal 15 2024-03-24T00:45:06.240 INFO:tasks.ceph.mon.c:Sent signal 15 2024-03-24T00:45:06.240 INFO:tasks.ceph.mgr.y:Sent signal 15 2024-03-24T00:45:06.240 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-03-24T00:45:06.241 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-24T00:45:06.242+0000 7f9646a00640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 27611) UID: 0 2024-03-24T00:45:06.241 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-24T00:45:06.242+0000 7f9646a00640 -1 osd.4 122 *** Got signal Terminated *** 2024-03-24T00:45:06.241 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-03-24T00:45:06.242+0000 7f9646a00640 -1 osd.4 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:45:06.241 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-24T00:45:06.242+0000 7f7d1d000640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 27612) UID: 0 2024-03-24T00:45:06.241 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-24T00:45:06.242+0000 7f7d1d000640 -1 osd.5 122 *** Got signal Terminated *** 2024-03-24T00:45:06.242 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-03-24T00:45:06.242+0000 7f7d1d000640 -1 osd.5 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:45:06.242 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-24T00:45:06.242+0000 7fa2ad000640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 27615) UID: 0 2024-03-24T00:45:06.242 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-24T00:45:06.242+0000 7fa2ad000640 -1 osd.6 122 *** Got signal Terminated *** 2024-03-24T00:45:06.242 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-03-24T00:45:06.242+0000 7fa2ad000640 -1 osd.6 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:45:06.242 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-24T00:45:06.242+0000 7f3d89400640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 27616) UID: 0 2024-03-24T00:45:06.242 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-24T00:45:06.242+0000 7f3d89400640 -1 osd.7 122 *** Got signal Terminated *** 2024-03-24T00:45:06.242 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-03-24T00:45:06.242+0000 7f3d89400640 -1 osd.7 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:45:06.243 INFO:tasks.ceph.mon.b.smithi053.stderr:2024-03-24T00:45:06.242+0000 7fcd08400640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 27298) UID: 0 2024-03-24T00:45:06.243 INFO:tasks.ceph.mon.b.smithi053.stderr:2024-03-24T00:45:06.242+0000 7fcd08400640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-03-24T00:45:06.243 INFO:tasks.ceph.mon.c.smithi053.stderr:2024-03-24T00:45:06.242+0000 7fd9bf400640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 27300) UID: 0 2024-03-24T00:45:06.243 INFO:tasks.ceph.mon.c.smithi053.stderr:2024-03-24T00:45:06.242+0000 7fd9bf400640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-24T00:45:06.243 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:45:06.242+0000 7f4e69e00640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 27302) UID: 0 2024-03-24T00:45:06.243 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-03-24T00:45:06.242+0000 7f4e69e00640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-24T00:45:06.244 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-03-24T00:45:06.233+0000 7fa135600640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 27388) UID: 0 2024-03-24T00:45:06.244 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-03-24T00:45:06.233+0000 7fa135600640 -1 osd.0 122 *** Got signal Terminated *** 2024-03-24T00:45:06.244 INFO:tasks.ceph.osd.0.smithi040.stderr:2024-03-24T00:45:06.233+0000 7fa135600640 -1 osd.0 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:45:06.244 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-03-24T00:45:06.237+0000 7fbd5f600640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 27390) UID: 0 2024-03-24T00:45:06.244 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-03-24T00:45:06.237+0000 7fbd5f600640 -1 osd.1 122 *** Got signal Terminated *** 2024-03-24T00:45:06.244 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-03-24T00:45:06.237+0000 7fbd5f600640 -1 osd.1 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:45:06.245 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.237+0000 7fd452e00640 1 -- 172.21.15.40:0/2567643458 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd44c072f50 con 0x7fd44c06e620 2024-03-24T00:45:06.245 INFO:tasks.ceph.osd.3.smithi040.stderr:2024-03-24T00:45:06.237+0000 7f3793800640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 27394) UID: 0 2024-03-24T00:45:06.245 INFO:tasks.ceph.osd.3.smithi040.stderr:2024-03-24T00:45:06.237+0000 7f3793800640 -1 osd.3 122 *** Got signal Terminated *** 2024-03-24T00:45:06.245 INFO:tasks.ceph.osd.3.smithi040.stderr:2024-03-24T00:45:06.237+0000 7f3793800640 -1 osd.3 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:45:06.245 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-03-24T00:45:06.237+0000 7f3b67800640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 27392) UID: 0 2024-03-24T00:45:06.245 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-03-24T00:45:06.237+0000 7f3b67800640 -1 osd.2 122 *** Got signal Terminated *** 2024-03-24T00:45:06.245 INFO:tasks.ceph.osd.2.smithi040.stderr:2024-03-24T00:45:06.237+0000 7f3b67800640 -1 osd.2 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T00:45:06.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.237+0000 7fd448c00640 1 -- 172.21.15.40:0/2567643458 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+4281 (secure 0 0 0) 0x7fd43c0f0fc0 con 0x7fd44c06e620 2024-03-24T00:45:06.246 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-03-24T00:45:06.246 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":145,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":145,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-24T00:39:07.531884+0000","modified":"2024-03-24T00:43:12.194456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":122,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8877,"mds_1":8874,"mds_2":7439},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7439":{"gid":7439,"name":"d","rank":2,"incarnation":136,"state":"up:active","state_seq":20,"addr":"172.21.15.53:6837/1582567544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6835","nonce":1582567544},{"type":"v1","addr":"172.21.15.53:6837","nonce":1582567544}]},"join_fscid":-1,"laggy_since":"2024-03-24T00:40:21.913457+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8874":{"gid":8874,"name":"c","rank":1,"incarnation":130,"state":"up:active","state_seq":6,"addr":"172.21.15.40:6833/4273998586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6832","nonce":4273998586},{"type":"v1","addr":"172.21.15.40:6833","nonce":4273998586}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8877":{"gid":8877,"name":"a","rank":0,"incarnation":124,"state":"up:active","state_seq":64,"addr":"172.21.15.40:6835/773071485","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6834","nonce":773071485},{"type":"v1","addr":"172.21.15.40:6835","nonce":773071485}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8877,"qdb_cluster":[7439,8874,8877]},"id":9}]} 2024-03-24T00:45:06.246 INFO:teuthology.orchestra.run.smithi040.stderr:dumped fsmap epoch 145 2024-03-24T00:45:06.247 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-03-24T00:45:06.237+0000 7fec35800640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i a (PID: 53240) UID: 0 2024-03-24T00:45:06.247 INFO:tasks.ceph.mds.a.smithi040.stderr:2024-03-24T00:45:06.237+0000 7fec35800640 -1 mds.a *** got signal Terminated *** 2024-03-24T00:45:06.247 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-03-24T00:45:06.237+0000 7f87c2400640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mds -f --cluster ceph -i c (PID: 53241) UID: 0 2024-03-24T00:45:06.247 INFO:tasks.ceph.mds.c.smithi040.stderr:2024-03-24T00:45:06.237+0000 7f87c2400640 -1 mds.c *** got signal Terminated *** 2024-03-24T00:45:06.247 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-03-24T00:45:06.237+0000 7fa655400640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 27283) UID: 0 2024-03-24T00:45:06.247 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-03-24T00:45:06.237+0000 7fa655400640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T00:45:06.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.241+0000 7fd452e00640 1 -- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd428067660 msgr2=0x7fd428069b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.241+0000 7fd452e00640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd428067660 0x7fd428069b20 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fd44c069520 tx=0x7fd43400b040 comp rx=0 tx=0).stop 2024-03-24T00:45:06.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.241+0000 7fd452e00640 1 -- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd44c06e620 msgr2=0x7fd44c0b1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:45:06.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.241+0000 7fd452e00640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd44c06e620 0x7fd44c0b1190 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0ad040 tx=0x7fd43c01aca0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.241+0000 7fd452e00640 1 -- 172.21.15.40:0/2567643458 shutdown_connections 2024-03-24T00:45:06.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.241+0000 7fd452e00640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:6832/4217283018,v1:172.21.15.53:6833/4217283018] conn(0x7fd428067660 0x7fd428069b20 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.248 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.241+0000 7fd452e00640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] conn(0x7fd44c0b2ff0 0x7fd44c1419f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.241+0000 7fd452e00640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd44c06ea00 0x7fd44c159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.241+0000 7fd452e00640 1 --2- 172.21.15.40:0/2567643458 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd44c06e620 0x7fd44c0b1190 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:06.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.241+0000 7fd452e00640 1 -- 172.21.15.40:0/2567643458 >> 172.21.15.40:0/2567643458 conn(0x7fd44c0b17b0 msgr2=0x7fd44c0ae4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:45:06.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.241+0000 7fd452e00640 1 -- 172.21.15.40:0/2567643458 shutdown_connections 2024-03-24T00:45:06.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.241+0000 7fd452e00640 1 -- 172.21.15.40:0/2567643458 wait complete. 2024-03-24T00:45:06.268 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:45:06.442 INFO:tasks.ceph.mgr.y.smithi040.stderr:daemon-helper: command crashed with signal 15 2024-03-24T00:45:06.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.449+0000 7f2ddec00640 1 Processor -- start 2024-03-24T00:45:06.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.449+0000 7f2ddec00640 1 -- start start 2024-03-24T00:45:06.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.453+0000 7f2ddec00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dd80b2aa0 0x7f2dd80d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.453+0000 7f2ddec00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2dd80d0a80 0x7f2dd80cc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:06.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.453+0000 7f2ddec00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2dd80b2490 2024-03-24T00:45:06.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.453+0000 7f2ddec00640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2dd80b2aa0 2024-03-24T00:45:06.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.453+0000 7f2ddec00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2dd80d0a80 2024-03-24T00:45:06.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.453+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2dd80b2aa0 msgr2=0x7f2dd80d0540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:06.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.453+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dd80b2aa0 0x7f2dd80d0540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:06.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.453+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2dd80b2490 legacy=0x7f2dd80cfe30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:06.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.453+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2dd80d0a80 msgr2=0x7f2dd80cc460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:06.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.453+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2dd80d0a80 0x7f2dd80cc460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:06.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.657+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2dd80b2aa0 msgr2=0x7f2dd80d0540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:06.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.657+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dd80b2aa0 0x7f2dd80d0540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:06.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.657+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2dd80b2490 legacy=0x7f2dd80cfe30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:06.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.657+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2dd80d0a80 msgr2=0x7f2dd80cc460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:06.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:06.657+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2dd80d0a80 0x7f2dd80cc460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:07.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:07.061+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2dd80b2aa0 msgr2=0x7f2dd80d0540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:07.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:07.061+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dd80b2aa0 0x7f2dd80d0540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:07.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:07.061+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2dd80b2490 legacy=0x7f2dd80cfe30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:07.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:07.061+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2dd80d0a80 msgr2=0x7f2dd80cc460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:07.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:07.061+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2dd80d0a80 0x7f2dd80cc460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:07.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:07.865+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2dd80b2aa0 msgr2=0x7f2dd80d0540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:07.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:07.865+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dd80b2aa0 0x7f2dd80d0540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:07.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:07.865+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2dd80b2490 legacy=0x7f2dd80cfe30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:07.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:07.865+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2dd80d0a80 msgr2=0x7f2dd80cc460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:07.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:07.865+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2dd80d0a80 0x7f2dd80cc460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:08.794 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:45:08.798 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:45:09.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.453+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2dd80d0a80 msgr2=0x7f2dd80cc460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.453+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2dd80d0a80 0x7f2dd80cc460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.453+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2dd80b2aa0 msgr2=0x7f2dd80d0540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.453+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2dd80b2aa0 0x7f2dd80d0540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.453+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2dd80b2490 legacy=0x7f2dd80cfe30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.453+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40036a0 0x7f2db4005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.453+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db4006130 2024-03-24T00:45:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.453+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db40036a0 2024-03-24T00:45:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.453+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4006130 legacy=0x7f2db40085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.453+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40036a0 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:09.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.453+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40036a0 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:09.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.657+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4006130 legacy=0x7f2db40085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:09.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.657+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40036a0 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:09.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:09.657+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40036a0 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:10.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:10.061+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4006130 legacy=0x7f2db40085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:10.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:10.061+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40036a0 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:10.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:10.061+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40036a0 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:10.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:10.865+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4006130 legacy=0x7f2db40085f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:10.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:10.865+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40036a0 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:10.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:10.865+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40036a0 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:12.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40036a0 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:12.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40036a0 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:12.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4006130 legacy=0x7f2db40085f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:12.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:45:12.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:12.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d20 0x7f2db4008590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:12.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2dd80b2490 2024-03-24T00:45:12.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db400a590 2024-03-24T00:45:12.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4008d20 2024-03-24T00:45:12.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:12.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:12.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2dd80b2490 legacy=0x7f2db4009e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:12.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d20 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:12.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.453+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d20 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:12.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.657+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:12.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.657+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:12.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.657+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d20 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:12.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.657+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2dd80b2490 legacy=0x7f2db4009e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:12.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:12.657+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d20 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:13.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:13.061+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:13.059 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:13.061+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:13.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:13.061+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d20 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:13.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:13.061+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2dd80b2490 legacy=0x7f2db4009e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:13.060 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:13.061+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d20 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:13.865+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:13.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:13.865+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:13.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:13.865+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2dd80b2490 legacy=0x7f2db4009e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:13.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:13.865+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d20 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:13.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:13.865+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d20 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:15.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.457+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d20 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:15.456 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.457+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d20 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.457+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:15.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.457+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:15.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.457+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2dd80b2490 legacy=0x7f2db4009e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:15.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.457+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4011080 0x7f2db4013470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:15.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.457+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db40091d0 2024-03-24T00:45:15.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.457+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4011080 2024-03-24T00:45:15.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.457+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db40091d0 legacy=0x7f2db40095e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:15.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.457+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4011080 msgr2=0x7f2db4013470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:15.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.457+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4011080 0x7f2db4013470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.661+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db40091d0 legacy=0x7f2db40095e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.661+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4011080 msgr2=0x7f2db4013470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:15.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:15.661+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4011080 0x7f2db4013470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:16.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:16.065+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db40091d0 legacy=0x7f2db40095e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:16.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:16.065+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4011080 msgr2=0x7f2db4013470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:16.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:16.065+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4011080 0x7f2db4013470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:16.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:16.869+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db40091d0 legacy=0x7f2db40095e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:16.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:16.869+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4011080 msgr2=0x7f2db4013470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:16.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:16.869+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4011080 0x7f2db4013470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:18.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4011080 msgr2=0x7f2db4013470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:18.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4011080 0x7f2db4013470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:18.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db40091d0 legacy=0x7f2db40095e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:18.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:45:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d00 0x7f2db4008590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2db4009680 2024-03-24T00:45:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db400a590 2024-03-24T00:45:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4008d00 2024-03-24T00:45:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d00 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:18.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.457+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d00 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:18.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.661+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:18.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.661+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:18.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.661+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:18.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.661+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d00 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:18.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:18.661+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d00 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:19.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:19.065+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:19.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:19.065+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:19.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:19.065+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:19.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:19.065+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d00 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:19.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:19.065+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d00 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:19.869+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:19.869+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:19.869+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:19.869+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d00 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:19.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:19.869+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d00 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:21.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.457+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d00 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:21.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.457+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4008d00 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:21.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.457+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:21.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.457+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:21.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.457+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:21.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.457+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4013470 0x7f2db400e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:21.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.457+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db400e5a0 2024-03-24T00:45:21.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.457+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4013470 2024-03-24T00:45:21.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.457+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4013470 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:21.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.457+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:21.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.457+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4013470 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:21.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.661+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4013470 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:21.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.661+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:21.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:21.661+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4013470 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:22.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:22.065+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4013470 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:22.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:22.065+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:22.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:22.065+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4013470 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:22.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:22.869+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:22.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:22.869+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4013470 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:22.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:22.869+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4013470 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.457+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4013470 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.457+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4013470 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.457+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.457+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.457+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:45:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.457+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2db4009680 2024-03-24T00:45:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.457+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db400e5a0 2024-03-24T00:45:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.457+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db400a590 2024-03-24T00:45:24.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.457+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.457+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.457+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:24.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.457+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:24.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.661+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.661+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.661+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:24.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:24.661+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:25.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:25.065+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:25.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:25.065+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:25.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:25.065+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:25.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:25.065+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:25.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:25.868+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:25.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:25.868+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:25.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:25.868+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:25.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:25.868+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:27.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.456+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:27.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.456+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.456+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.456+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.456+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 0x7f2db4008590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.456+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.456+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4010b80 2024-03-24T00:45:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.456+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4008cd0 2024-03-24T00:45:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.456+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.456+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.456+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:27.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.456+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:27.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.660+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:27.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.660+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:27.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.660+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:27.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:27.660+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:28.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:28.064+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:28.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:28.064+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:28.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:28.064+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:28.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:28.064+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:28.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:28.868+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:28.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:28.868+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:28.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:28.868+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:28.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:28.868+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:30.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:30.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:30.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 0x7f2db4008590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:45:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db4009680 2024-03-24T00:45:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db400a590 2024-03-24T00:45:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4008cd0 2024-03-24T00:45:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:30.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.456+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:30.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.660+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:30.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.660+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:30.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.660+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:30.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.660+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:30.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:30.660+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:31.063 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:31.064+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:31.064+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:31.064+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:31.064+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:31.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:31.064+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:31.868+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:31.868+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:31.868+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:31.868+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:31.868+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:33.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.456+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 msgr2=0x7f2db4008590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:33.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.456+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cd0 0x7f2db4008590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.456+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:33.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.456+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:33.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.456+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:33.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.456+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:33.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.456+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2db400e5a0 2024-03-24T00:45:33.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.456+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4010b80 2024-03-24T00:45:33.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.456+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:33.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.456+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:33.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.456+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:33.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.660+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:33.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.660+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:33.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:33.660+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:34.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:34.064+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:34.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:34.064+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:34.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:34.064+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:34.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:34.868+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:34.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:34.868+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:34.868 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:34.868+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:36.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:36.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:36.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400e5a0 legacy=0x7f2db4013680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:36.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:36.459 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:36.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:45:36.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db4008cd0 2024-03-24T00:45:36.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db400a590 2024-03-24T00:45:36.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4010b80 2024-03-24T00:45:36.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:36.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:36.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2ddcc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:36.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:36.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.456+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:36.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.660+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:36.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.660+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:36.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.660+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:36.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.660+0000 7f2ddcc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:36.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:36.660+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:37.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:37.064+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:37.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:37.064+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:37.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:37.064+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:37.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:37.064+0000 7f2ddcc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:37.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:37.064+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:37.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:37.868+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:37.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:37.868+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:37.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:37.868+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:37.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:37.868+0000 7f2ddcc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:37.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:37.868+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:38.855 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:45:38.859 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:45:39.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.460+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:39.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:39.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.460+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:39.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:39.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.460+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:39.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.460+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4007ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:39.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.460+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2db4009680 2024-03-24T00:45:39.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.460+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db40105b0 2024-03-24T00:45:39.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.460+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:39.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.460+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:39.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.460+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:39.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.664+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:39.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.664+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:39.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:39.664+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:40.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:40.068+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:40.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:40.068+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:40.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:40.068+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:40.872+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:40.872+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:40.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:40.872+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:42.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:45:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db4009680 2024-03-24T00:45:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db400a590 2024-03-24T00:45:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4010b80 2024-03-24T00:45:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4008210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:42.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.460+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:42.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.664+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:42.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.664+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4008210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:42.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.664+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:42.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.664+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:42.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:42.664+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:43.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:43.068+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:43.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:43.068+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4008210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:43.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:43.068+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:43.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:43.068+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:43.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:43.068+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:43.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:43.872+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:43.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:43.872+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4008210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:43.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:43.872+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:43.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:43.872+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:43.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:43.872+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:45.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.460+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400e060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:45.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4010b80 0x7f2db400e060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:45.460 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.460+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:45.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:45.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.460+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4009680 legacy=0x7f2db4008210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:45.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.460+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4009e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:45.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.460+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2db4008cd0 2024-03-24T00:45:45.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.460+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db40105b0 2024-03-24T00:45:45.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.460+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:45.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.460+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:45.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.460+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:45.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.664+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:45.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.664+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:45.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:45.664+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:46.068+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:46.068+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:46.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:46.068+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:46.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:46.872+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:46.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:46.872+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:46.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:46.872+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:48.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:48.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:48.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:48.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:48.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4007ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:48.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:45:48.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db40081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:48.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db400a590 2024-03-24T00:45:48.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4009680 2024-03-24T00:45:48.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db4008720 con 0x7f2db40105b0 2024-03-24T00:45:48.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:48.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:48.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:48.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db40081e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:48.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db40081e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:48.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.460+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.664+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:48.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.664+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.664+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.664+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db40081e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.664+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:48.664 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:48.664+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db40081e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:49.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:49.068+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:49.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:49.068+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:49.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:49.068+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:49.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:49.068+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db40081e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:49.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:49.068+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:49.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:49.068+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db40081e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:49.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:49.872+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:49.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:49.872+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:49.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:49.872+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:49.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:49.872+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db40081e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:49.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:49.872+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db40081e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:49.872 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:49.872+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:51.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:51.460+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:51.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:51.460+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db40105b0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:51.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:51.460+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db40081e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:51.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:51.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db40081e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:51.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:51.460+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:51.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:51.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:51.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:51.460+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db4008cd0 2024-03-24T00:45:51.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:51.460+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2db4010b80 2024-03-24T00:45:51.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:51.460+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:51.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:51.460+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:51.664+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:51.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:51.664+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:52.066 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:52.068+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:52.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:52.068+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:52.872+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:52.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:52.872+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.460+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.460+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.460+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:45:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.460+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db4008cd0 2024-03-24T00:45:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.460+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db4010b80 2024-03-24T00:45:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.460+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db400a590 2024-03-24T00:45:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.460+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:54.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.460+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.460+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:54.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.460+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:54.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.664+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:54.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.664+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:54.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.664+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:54.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:54.664+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:55.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:55.068+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:55.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:55.068+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:55.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:55.068+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:55.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:55.068+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:55.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:55.872+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:45:55.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:55.872+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:45:55.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:55.872+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:55.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:55.872+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:45:57.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.460+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:45:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.460+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.460+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cd0 legacy=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:45:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.460+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4009680 0x7f2db4012470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db40129b0 0x7f2db4012d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:45:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.460+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db40129b0 2024-03-24T00:45:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.460+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4009680 2024-03-24T00:45:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.460+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4009680 msgr2=0x7f2db4012470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.460+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db40129b0 msgr2=0x7f2db4012d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.460+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4009680 0x7f2db4012470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:57.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.460+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db40129b0 0x7f2db4012d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:45:57.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.664+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db40129b0 msgr2=0x7f2db4012d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:57.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.664+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4009680 msgr2=0x7f2db4012470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:57.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.664+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4009680 0x7f2db4012470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:57.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:57.664+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db40129b0 0x7f2db4012d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:45:58.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:58.068+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db40129b0 msgr2=0x7f2db4012d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:58.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:58.068+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4009680 msgr2=0x7f2db4012470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:58.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:58.068+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4009680 0x7f2db4012470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:58.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:58.068+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db40129b0 0x7f2db4012d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:45:58.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:58.872+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4009680 msgr2=0x7f2db4012470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:45:58.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:58.872+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db40129b0 msgr2=0x7f2db4012d90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:45:58.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:58.872+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4009680 0x7f2db4012470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:45:58.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:45:58.872+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db40129b0 0x7f2db4012d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4009680 msgr2=0x7f2db4012470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4009680 0x7f2db4012470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db40129b0 msgr2=0x7f2db4012d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:00.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db40129b0 0x7f2db4012d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:46:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2db400a590 2024-03-24T00:46:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db4010b80 2024-03-24T00:46:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4009680 2024-03-24T00:46:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:00.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.460+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.664+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.664+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.664+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:00.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:00.664+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:01.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:01.068+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:01.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:01.068+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:01.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:01.068+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:01.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:01.068+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:01.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:01.872+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:01.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:01.872+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:01.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:01.872+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:01.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:01.872+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:03.462 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.460+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.460+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.460+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.460+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 0x7f2db4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 0x7f2db4007ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.460+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4010510 2024-03-24T00:46:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.460+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4008cb0 2024-03-24T00:46:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.460+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.460+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.460+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:03.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.460+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:03.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.664+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:03.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.664+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:03.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.664+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:03.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:03.664+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:04.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:04.068+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:04.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:04.068+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:04.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:04.068+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:04.067 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:04.068+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:04.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:04.872+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:04.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:04.872+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:04.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:04.872+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:04.871 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:04.872+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.460+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.460+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.460+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.460+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.460+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 0x7f2db4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.460+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:46:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.460+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db400a590 2024-03-24T00:46:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.460+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4009680 2024-03-24T00:46:06.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.460+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4008cb0 2024-03-24T00:46:06.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.464+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:06.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.464+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:06.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.464+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:06.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.464+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:06.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.464+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:06.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.668+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:06.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.668+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:06.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.668+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:06.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.668+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:06.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:06.668+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:07.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:07.072+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:07.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:07.072+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:07.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:07.072+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:07.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:07.072+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:07.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:07.072+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:07.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:07.876+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:07.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:07.876+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:07.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:07.876+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:07.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:07.876+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:07.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:07.876+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:08.926 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:46:08.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:46:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.464+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.464+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4008cb0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:09.463 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.464+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.464+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 0x7f2db4007590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.464+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2db4010b80 2024-03-24T00:46:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.464+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4010510 2024-03-24T00:46:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.464+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 msgr2=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.464+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 0x7f2db4007590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:09.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.464+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:09.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.668+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 msgr2=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:09.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.668+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 0x7f2db4007590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:09.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:09.668+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:10.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:10.072+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 msgr2=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:10.072+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 0x7f2db4007590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:10.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:10.072+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:10.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:10.876+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 msgr2=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:10.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:10.876+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 0x7f2db4007590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:10.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:10.876+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:12.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.464+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 msgr2=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:12.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010510 0x7f2db4007590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:12.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.464+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:12.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.464+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:46:12.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:12.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.464+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db400a590 2024-03-24T00:46:12.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.464+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2db4010b80 2024-03-24T00:46:12.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.464+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4009680 2024-03-24T00:46:12.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.464+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:12.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.464+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:12.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.464+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:12.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.464+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:12.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.668+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:12.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.668+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:12.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.668+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:12.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:12.668+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:13.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:13.072+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:13.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:13.072+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:13.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:13.072+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:13.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:13.072+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:13.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:13.876+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:13.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:13.876+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:13.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:13.876+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:13.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:13.876+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:15.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.464+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:15.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:15.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.464+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:15.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.464+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:15.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:15.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.464+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 0x7f2db4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:15.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.464+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4008cb0 2024-03-24T00:46:15.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.464+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4010510 2024-03-24T00:46:15.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.464+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:15.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.464+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:15.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.464+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:15.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.464+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:15.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.668+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:15.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.668+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:15.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.668+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:15.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:15.668+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:16.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:16.072+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:16.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:16.072+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:16.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:16.072+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:16.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:16.072+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:16.876+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:16.876+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:16.876+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:16.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:16.876+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:18.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:18.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:18.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:18.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 0x7f2db4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:18.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:46:18.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db400a590 2024-03-24T00:46:18.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4009680 2024-03-24T00:46:18.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4010510 2024-03-24T00:46:18.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:18.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:18.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:18.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:18.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.464+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:18.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.668+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:18.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.668+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:18.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.668+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:18.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.668+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:18.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:18.668+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:19.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:19.072+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:19.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:19.072+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:19.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:19.072+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:19.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:19.072+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:19.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:19.072+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:19.876+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:19.876+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:19.876+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:19.876+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:19.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:19.876+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:21.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.464+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:21.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.464+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010510 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.464+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:21.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:21.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.464+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db400a590 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:21.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:21.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.464+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db4010b80 2024-03-24T00:46:21.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.464+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4008cb0 2024-03-24T00:46:21.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.464+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:21.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.464+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:21.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.464+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:21.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.668+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:21.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.668+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:21.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:21.668+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:22.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:22.072+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:22.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:22.072+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:22.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:22.072+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:22.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:22.876+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:22.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:22.876+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:22.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:22.876+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:24.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:46:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db4010b80 2024-03-24T00:46:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4008cb0 2024-03-24T00:46:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4009680 2024-03-24T00:46:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:24.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.464+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.668+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.668+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.668+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.668+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:24.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:24.668+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:25.072+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:25.072+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:25.072+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:25.072+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:25.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:25.072+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:25.876+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:25.876+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:25.876+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:25.876+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:25.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:25.876+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.464+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db400ffd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db400ffd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.464+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:27.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.464+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4007590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.464+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.464+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db4010510 2024-03-24T00:46:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.464+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db400a590 2024-03-24T00:46:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.464+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010510 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.464+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:27.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.464+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:27.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.668+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010510 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:27.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.668+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:27.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:27.668+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:28.072+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010510 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:28.072+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:28.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:28.072+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:28.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:28.876+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010510 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:28.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:28.876+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:28.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:28.876+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:30.465 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010510 legacy=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db4010b80 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4009680 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db400a590 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:30.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.464+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:30.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.668+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:30.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.668+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:30.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.668+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:30.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.668+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:30.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:30.668+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:31.072+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:31.072+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:31.072+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:31.072+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:31.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:31.072+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:31.876+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:31.876+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:31.876+0000 7f2dcfe00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:31.876+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:31.876+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.464+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.464+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db400a590 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.464+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.464+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:33.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.464+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db40082b0 2024-03-24T00:46:33.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.464+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4008cb0 2024-03-24T00:46:33.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.464+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:33.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.464+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:33.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.464+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db40082b0 legacy=0x7f2db400efc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:33.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.668+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:33.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.668+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:33.666 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:33.668+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db40082b0 legacy=0x7f2db400efc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:34.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:34.072+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:34.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:34.072+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:34.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:34.072+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db40082b0 legacy=0x7f2db400efc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:34.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:34.876+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:34.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:34.876+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:34.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:34.876+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db40082b0 legacy=0x7f2db400efc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:36.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db4007ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:36.466 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db4007ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db40082b0 legacy=0x7f2db400efc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:46:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db400a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db40082b0 2024-03-24T00:46:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4008cb0 2024-03-24T00:46:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4009680 2024-03-24T00:46:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2ddcc00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db40082b0 legacy=0x7f2db400efc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:36.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.464+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:36.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.668+0000 7f2ddcc00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:36.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.668+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:36.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.668+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:36.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.668+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db40082b0 legacy=0x7f2db400efc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:36.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:36.668+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:37.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:37.072+0000 7f2ddcc00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:37.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:37.072+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:37.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:37.072+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db40082b0 legacy=0x7f2db400efc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:37.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:37.072+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:37.071 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:37.072+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:37.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:37.876+0000 7f2ddcc00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:37.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:37.876+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:37.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:37.876+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db40082b0 legacy=0x7f2db400efc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:37.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:37.876+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:37.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:37.876+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:38.981 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:46:38.985 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:46:39.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.468+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:39.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.468+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:39.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.468+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:39.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.468+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4008cb0 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:39.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.468+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db40082b0 legacy=0x7f2db400efc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:39.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.468+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4007ca0 0x7f2db4005a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:39.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.468+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db4010b80 2024-03-24T00:46:39.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.468+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4007ca0 2024-03-24T00:46:39.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.468+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:39.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.468+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4007ca0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:39.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.468+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4007ca0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:39.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.672+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:39.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.672+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4007ca0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:39.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:39.672+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4007ca0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:40.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:40.076+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:40.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:40.076+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4007ca0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:40.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:40.076+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4007ca0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:40.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:40.880+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:40.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:40.880+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4007ca0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:40.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:40.880+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4007ca0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:42.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4007ca0 msgr2=0x7f2db4005a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4007ca0 0x7f2db4005a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4010b80 legacy=0x7f2db4006580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:46:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b80 0x7f2db400a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db4008cb0 2024-03-24T00:46:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4010b80 2024-03-24T00:46:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4009680 2024-03-24T00:46:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400afa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b80 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:42.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:42.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.468+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.672+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.672+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400afa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.672+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b80 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.672+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:42.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:42.672+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:43.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:43.076+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:43.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:43.076+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400afa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:43.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:43.076+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b80 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:43.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:43.076+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:43.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:43.076+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:43.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:43.880+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:43.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:43.880+0000 7f2ddcc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400afa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:43.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:43.880+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b80 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:43.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:43.880+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:43.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:43.880+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:45.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.468+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:45.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.468+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:45.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.468+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b80 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:45.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.468+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b80 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:45.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.468+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400afa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:45.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.468+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 0x7f2db400e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:45.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.468+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db400ec10 2024-03-24T00:46:45.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.468+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4005a90 2024-03-24T00:46:45.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.468+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400ec10 legacy=0x7f2db400f0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:45.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.468+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 msgr2=0x7f2db400e6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:45.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.468+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 0x7f2db400e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:45.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.672+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 msgr2=0x7f2db400e6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:45.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.672+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400ec10 legacy=0x7f2db400f0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:45.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:45.672+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 0x7f2db400e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:46.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:46.076+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400ec10 legacy=0x7f2db400f0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:46.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:46.076+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 msgr2=0x7f2db400e6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:46.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:46.076+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 0x7f2db400e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:46.880+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400ec10 legacy=0x7f2db400f0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:46.880+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 msgr2=0x7f2db400e6d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:46.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:46.880+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 0x7f2db400e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:48.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.468+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 msgr2=0x7f2db400e6d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:48.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.468+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 0x7f2db400e6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:48.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.468+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db400ec10 legacy=0x7f2db400f0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:48.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.468+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:46:48.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.468+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:48.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.468+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db4008cb0 2024-03-24T00:46:48.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.468+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db4015650 2024-03-24T00:46:48.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.468+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4009680 2024-03-24T00:46:48.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.468+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:48.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.468+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4015650 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:48.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.468+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:48.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.468+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:48.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.672+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:48.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.672+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4015650 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:48.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.672+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:48.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:48.672+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:49.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:49.076+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:49.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:49.076+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4015650 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:49.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:49.076+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:49.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:49.076+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:49.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:49.880+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:49.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:49.880+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:49.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:49.880+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4015650 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:49.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:49.880+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:51.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.467+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:51.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.467+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:51.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.467+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4015650 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:51.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.467+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.467+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.467+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b50 0x7f2db400fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.467+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4010b50 2024-03-24T00:46:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.467+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4005a90 2024-03-24T00:46:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.467+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b50 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.467+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b50 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.467+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:51.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.467+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.671+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b50 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.671+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b50 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.671+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:51.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:51.671+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:52.075+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b50 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:52.075+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b50 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:52.075+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:52.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:52.075+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:52.879+0000 7f2dcfe00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b50 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:52.879+0000 7f2dcfe00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b50 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:52.879+0000 7f2dd7e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:52.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:52.879+0000 7f2dd7e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:54.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b50 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4010b50 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:46:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2db4008cb0 2024-03-24T00:46:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b3470 con 0x7f2db4015650 2024-03-24T00:46:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4009680 2024-03-24T00:46:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4015650 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:54.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.467+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:54.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.671+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:54.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.671+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4015650 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:54.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.671+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:54.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:54.671+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:55.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:55.075+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:55.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:55.075+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:55.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:55.075+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4015650 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:55.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:55.075+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:55.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:55.879+0000 7f2dcfe00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:46:55.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:55.879+0000 7f2ddcc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:46:55.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:55.879+0000 7f2dd7e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4015650 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:46:55.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:55.879+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:57.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.467+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:57.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.467+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:46:57.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.467+0000 7f2dd6000640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4015650 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:57.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.467+0000 7f2dd6000640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:46:57.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.467+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.467+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 0x7f2db400fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:46:57.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.467+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4005a90 2024-03-24T00:46:57.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.467+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4010b10 2024-03-24T00:46:57.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.467+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:57.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.467+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:57.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.467+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:57.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.467+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:46:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.671+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.671+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.671+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:57.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:57.671+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:46:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:58.075+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:58.075+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:58.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:58.075+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:58.076 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:58.075+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:46:58.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:58.879+0000 7f2dd7e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:46:58.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:58.879+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:46:58.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:58.879+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:46:58.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:46:58.879+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2ddcc00640 1 -- reap_dead start 2024-03-24T00:47:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 0x7f2db400fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dd6000640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd806b6d0 con 0x7f2db4005a90 2024-03-24T00:47:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dd6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dd80b3190 con 0x7f2db4009680 2024-03-24T00:47:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dd6000640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db4008720 con 0x7f2db4010b10 2024-03-24T00:47:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2ddcc00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:00.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.467+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.671+0000 7f2ddcc00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.671+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.671+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.671+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.671+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:00.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:00.671+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:01.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:01.075+0000 7f2ddcc00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:01.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:01.075+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:01.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:01.075+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:01.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:01.075+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:01.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:01.075+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:01.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:01.075+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:01.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:01.879+0000 7f2ddcc00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:01.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:01.879+0000 7f2ddcc00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:01.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:01.879+0000 7f2dcfe00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:01.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:01.879+0000 7f2dd7e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:01.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:01.879+0000 7f2dd7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:01.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:01.879+0000 7f2dcfe00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:03.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:03.467+0000 7f2dd6000640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 msgr2=0x7f2db400fec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:03.467+0000 7f2dd6000640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2db4010b10 0x7f2db400fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:03.467+0000 7f2dd6000640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 msgr2=0x7f2db4009e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:03.467+0000 7f2dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2db4009680 0x7f2db4009e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:03.467+0000 7f2dd6000640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 msgr2=0x7f2db400a390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:03.467+0000 7f2dd6000640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f2db4005a90 0x7f2db400a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:03.467+0000 7f2dd6000640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db4013470 con 0x7f2db4008cb0 2024-03-24T00:47:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:03.467+0000 7f2dd6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2870 con 0x7f2db4010400 2024-03-24T00:47:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:03.467+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:03.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:03.467+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010400 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:03.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:03.671+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:03.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:03.671+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010400 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:04.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:04.075+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:04.075 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:04.075+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010400 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:04.879+0000 7f2dd7e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f2db4008cb0 legacy=0x7f2db400f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:04.879 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:04.879+0000 7f2dcfe00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f2db4010400 legacy=0x7f2db400f020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:06.338 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T00:47:06.339 INFO:tasks.cephfs_test_runner:That ephemerally pinned subtrees are somewhat evenly distributed. ... ERROR 2024-03-24T00:47:06.341 INFO:tasks.cephfs_test_runner:ERROR 2024-03-24T00:47:06.342 INFO:tasks.cephfs_test_runner: 2024-03-24T00:47:06.342 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-24T00:47:06.342 INFO:tasks.cephfs_test_runner:ERROR: test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:47:06.342 INFO:tasks.cephfs_test_runner:That ephemerally pinned subtrees are somewhat evenly distributed. 2024-03-24T00:47:06.342 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T00:47:06.342 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-24T00:47:06.342 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr 2024-03-24T00:47:06.342 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2024-03-24T00:47:06.342 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read 2024-03-24T00:47:06.342 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/test_exports.py", line 436, in test_ephemeral_pin_distribution 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: self._setup_tree(count=1000, distributed=True) 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/test_exports.py", line 293, in _setup_tree 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: return self.mount_a.run_shell_payload(f""" 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/mount.py", line 794, in run_shell_payload 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: return self.run_shell(**kwargs) 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/mount.py", line 785, in run_shell 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: return self.client_remote.run(args=args, cwd=cwd, timeout=timeout, 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: r.wait() 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 148, in wait 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2024-03-24T00:47:06.343 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: for line in f: 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: line = self.readline() 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner:socket.timeout 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner: 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner:ERROR: test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:47:06.344 INFO:tasks.cephfs_test_runner:That ephemerally pinned subtrees are somewhat evenly distributed. 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/cephfs_test_case.py", line 208, in tearDown 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: self.mds_cluster.delete_all_filesystems() 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 500, in delete_all_filesystems 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: Filesystem(ctx=self._ctx, fscid=fs['id']).destroy() 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 859, in destroy 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: data_pools = self.get_data_pool_names(refresh=True) 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 989, in get_data_pool_names 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: self.get_pool_names(refresh = True) 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 958, in get_pool_names 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: status = self.status() 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 402, in status 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: return FSStatus(self.mon_manager, epoch) 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 88, in __init__ 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: self.map = json.loads(self.get_ceph_cmd_stdout(*cmd)) 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-03-24T00:47:06.345 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner: r.wait() 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner: 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner:Ran 8 tests in 933.278s 2024-03-24T00:47:06.346 INFO:tasks.cephfs_test_runner: 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner: 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner:ERROR: test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner:That ephemerally pinned subtrees are somewhat evenly distributed. 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner: 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner: 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/test_exports.py", line 436, in test_ephemeral_pin_distribution 2024-03-24T00:47:06.347 INFO:tasks.cephfs_test_runner: self._setup_tree(count=1000, distributed=True) 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/test_exports.py", line 293, in _setup_tree 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: return self.mount_a.run_shell_payload(f""" 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/mount.py", line 794, in run_shell_payload 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: return self.run_shell(**kwargs) 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/mount.py", line 785, in run_shell 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: return self.client_remote.run(args=args, cwd=cwd, timeout=timeout, 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: r.wait() 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 148, in wait 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: for line in f: 2024-03-24T00:47:06.348 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: line = self.readline() 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner:socket.timeout 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner:ERROR: test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner:That ephemerally pinned subtrees are somewhat evenly distributed. 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/cephfs_test_case.py", line 208, in tearDown 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: self.mds_cluster.delete_all_filesystems() 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 500, in delete_all_filesystems 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: Filesystem(ctx=self._ctx, fscid=fs['id']).destroy() 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 859, in destroy 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: data_pools = self.get_data_pool_names(refresh=True) 2024-03-24T00:47:06.349 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 989, in get_data_pool_names 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: self.get_pool_names(refresh = True) 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 958, in get_pool_names 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: status = self.status() 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 402, in status 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: return FSStatus(self.mon_manager, epoch) 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 88, in __init__ 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: self.map = json.loads(self.get_ceph_cmd_stdout(*cmd)) 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: return self.run_ceph_cmd(**kwargs).stdout.getvalue() 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: r.wait() 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-24T00:47:06.350 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-24T00:47:06.351 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-24T00:47:06.351 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json' 2024-03-24T00:47:06.351 INFO:tasks.cephfs_test_runner: 2024-03-24T00:47:06.351 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins), test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:47:06.571 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=809d1b44f79741a2b62cc123e8ffbdd3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins), test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins) 2024-03-24T00:47:06.572 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-24T00:47:06.581 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-24T00:47:06.629 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-24T00:47:06.629 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:47:06.629 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:06.642 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:47:06.643 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-03-24T00:47:06.643 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:06.690 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=716 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-24T00:47:06.691 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:47:06.691 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:06.700 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:47:06.701 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-03-24T00:47:06.701 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:47:06.752 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16357716k,nr_inodes=4089429,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=6209 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275992k,nr_inodes=818998,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-24T00:47:06.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:06.752 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-03-24T00:47:06.804 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-24T00:47:06.804 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-24T00:47:06.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:06.805 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T00:47:06.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:06.869 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-24T00:47:06.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:06.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link delete ceph-brx 2024-03-24T00:47:07.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:07.114 DEBUG:teuthology.orchestra.run.smithi040:> ip netns list 2024-03-24T00:47:07.123 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T00:47:07.124 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-24T00:47:07.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:07.125 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:47:07.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:07.188 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-24T00:47:07.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:47:07.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo ip link delete ceph-brx 2024-03-24T00:47:07.461 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-24T00:47:07.470 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-24T00:47:07.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:47:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.615+0000 7f705d600640 1 Processor -- start 2024-03-24T00:47:07.614 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.615+0000 7f705d600640 1 -- start start 2024-03-24T00:47:07.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.615+0000 7f705d600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 0x7f70580cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:07.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.615+0000 7f705d600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f705806d630 0x7f7058072c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:07.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.615+0000 7f705d600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f705806f030 con 0x7f705806d1f0 2024-03-24T00:47:07.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.615+0000 7f705d600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f705806d630 2024-03-24T00:47:07.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.615+0000 7f705d600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f705815e040 2024-03-24T00:47:07.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.615+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f705806d1f0 legacy=0x7f7058072520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:07.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.615+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 msgr2=0x7f70580cdd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:07.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.615+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f705806d630 msgr2=0x7f7058072c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:07.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.615+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 0x7f70580cdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:07.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.615+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f705806d630 0x7f7058072c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:07.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.819+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f705806d1f0 legacy=0x7f7058072520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:07.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.819+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f705806d630 msgr2=0x7f7058072c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:07.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.819+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 msgr2=0x7f70580cdd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:07.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.819+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 0x7f70580cdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:07.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:07.819+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f705806d630 0x7f7058072c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:08.223+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f705806d1f0 legacy=0x7f7058072520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:08.223+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 msgr2=0x7f70580cdd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:08.223+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f705806d630 msgr2=0x7f7058072c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:08.223+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 0x7f70580cdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:08.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:08.223+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f705806d630 0x7f7058072c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:09.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:09.027+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 msgr2=0x7f70580cdd20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:09.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:09.027+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f705806d630 msgr2=0x7f7058072c30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:09.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:09.027+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f705806d1f0 legacy=0x7f7058072520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:09.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:09.027+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 0x7f70580cdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:09.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:09.027+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f705806d630 0x7f7058072c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:09.037 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:47:09.041 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:47:10.616 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.615+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 msgr2=0x7f70580cdd20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.615+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 0x7f70580cdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.615+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f705806d630 msgr2=0x7f7058072c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.615+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f705806d630 0x7f7058072c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.615+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f705806d1f0 legacy=0x7f7058072520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.615+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380036d0 0x7f7038005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:10.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.615+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f7038006230 2024-03-24T00:47:10.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.615+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f70380036d0 2024-03-24T00:47:10.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.615+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038006230 legacy=0x7f70380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:10.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.615+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380036d0 msgr2=0x7f7038005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:10.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.615+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380036d0 0x7f7038005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:10.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.819+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038006230 legacy=0x7f70380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:10.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.819+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380036d0 msgr2=0x7f7038005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:10.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:10.819+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380036d0 0x7f7038005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:11.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:11.223+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038006230 legacy=0x7f70380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:11.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:11.223+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380036d0 msgr2=0x7f7038005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:11.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:11.223+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380036d0 0x7f7038005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:12.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:12.027+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038006230 legacy=0x7f70380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:12.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:12.027+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380036d0 msgr2=0x7f7038005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:12.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:12.027+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380036d0 0x7f7038005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:13.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380036d0 msgr2=0x7f7038005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:13.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380036d0 0x7f7038005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:13.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038006230 legacy=0x7f70380086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:13.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 0x7f7038009f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:47:13.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a4d0 0x7f7038005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:13.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703800af90 con 0x7f703800a8b0 2024-03-24T00:47:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f703800a4d0 2024-03-24T00:47:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f705815e040 2024-03-24T00:47:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a4d0 msgr2=0x7f7038005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a4d0 0x7f7038005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f703800a8b0 legacy=0x7f70380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 msgr2=0x7f7038009f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:13.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.615+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 0x7f7038009f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:13.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.819+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a4d0 msgr2=0x7f7038005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:13.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.819+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a4d0 0x7f7038005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:13.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.819+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f703800a8b0 legacy=0x7f70380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:13.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.819+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 msgr2=0x7f7038009f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:13.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:13.819+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 0x7f7038009f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:14.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:14.223+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a4d0 msgr2=0x7f7038005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:14.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:14.223+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a4d0 0x7f7038005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:14.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:14.223+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 msgr2=0x7f7038009f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:14.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:14.223+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f703800a8b0 legacy=0x7f70380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:14.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:14.223+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 0x7f7038009f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:15.027+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a4d0 msgr2=0x7f7038005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:15.027+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a4d0 0x7f7038005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:15.027+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f703800a8b0 legacy=0x7f70380086f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:15.027+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 msgr2=0x7f7038009f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:15.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:15.027+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 0x7f7038009f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:16.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.615+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 msgr2=0x7f7038009f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:16.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.615+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f705815e040 0x7f7038009f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:16.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.615+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a4d0 msgr2=0x7f7038005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:16.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.615+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a4d0 0x7f7038005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:16.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.615+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f703800a8b0 legacy=0x7f70380086f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:16.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.615+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380110d0 0x7f70380134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:16.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.615+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f703800d460 2024-03-24T00:47:16.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.615+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f70380110d0 2024-03-24T00:47:16.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.615+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800d460 legacy=0x7f703800f880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:16.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.615+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380110d0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:16.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.615+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380110d0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:16.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.819+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800d460 legacy=0x7f703800f880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:16.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.819+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380110d0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:16.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:16.819+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380110d0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:17.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:17.223+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800d460 legacy=0x7f703800f880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:17.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:17.223+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380110d0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:17.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:17.223+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380110d0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:18.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:18.027+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800d460 legacy=0x7f703800f880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:18.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:18.027+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380110d0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:18.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:18.027+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380110d0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:19.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380110d0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:19.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380110d0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:19.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800d460 legacy=0x7f703800f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:19.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f7038013fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:47:19.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380145a0 0x7f7038009f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:19.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f70380145a0 2024-03-24T00:47:19.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f7038013b80 2024-03-24T00:47:19.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7038010030 con 0x7f703800a490 2024-03-24T00:47:19.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380145a0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:19.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380145a0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:19.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:19.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f7038013fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:19.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:19.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.615+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f7038013fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:19.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.819+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380145a0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:19.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.819+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380145a0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:19.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.819+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:19.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.819+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f7038013fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:19.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.819+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:19.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:19.819+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f7038013fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:20.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:20.223+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380145a0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:20.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:20.223+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380145a0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:20.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:20.223+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f7038013fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:20.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:20.223+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:20.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:20.223+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:20.223 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:20.223+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f7038013fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:21.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:21.027+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380145a0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:21.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:21.027+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380145a0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:21.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:21.027+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f7038013fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:21.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:21.027+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:21.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:21.027+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f7038013fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:21.027 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:21.027+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:22.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:22.619+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:22.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:22.619+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:22.619+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f7038013fc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:22.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:22.619+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f7038013fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:22.619+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380145a0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:22.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:22.619+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380145a0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:22.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:22.619+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703800af90 con 0x7f7038010330 2024-03-24T00:47:22.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:22.619+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f7038005b10 2024-03-24T00:47:22.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:22.619+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f70380107a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:22.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:22.619+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038005b10 legacy=0x7f703800e850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:22.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:22.823+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f70380107a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:22.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:22.823+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038005b10 legacy=0x7f703800e850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:23.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:23.227+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f70380107a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:23.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:23.227+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038005b10 legacy=0x7f703800e850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:24.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:24.031+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f70380107a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:24.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:24.031+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038005b10 legacy=0x7f703800e850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:25.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.619+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038005b10 legacy=0x7f703800e850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:25.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.619+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f70380107a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:25.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.619+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:47:25.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.619+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f7038013fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:25.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.619+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f7038010330 2024-03-24T00:47:25.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.619+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703800af90 con 0x7f7038009960 2024-03-24T00:47:25.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.619+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013b80 2024-03-24T00:47:25.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.619+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:25.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.619+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f70380086d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:25.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.619+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f7038013fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:25.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.619+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f7038013fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:25.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.823+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:25.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.823+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f70380086d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:25.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.823+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f7038013fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:25.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:25.823+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f7038013fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:26.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:26.227+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:26.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:26.227+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f70380086d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:26.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:26.227+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f7038013fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:26.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:26.227+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f7038013fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:27.031+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:27.031+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f70380086d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:27.031+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f7038013fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:27.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:27.031+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f7038013fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:28.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.619+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f7038013fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:28.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.619+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f7038013fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:28.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.619+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f70380086d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:28.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.619+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:28.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.619+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.619+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038009f50 0x7f70380134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:28.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.619+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038009f50 2024-03-24T00:47:28.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.619+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f70380145a0 2024-03-24T00:47:28.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.619+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038009f50 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:28.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.619+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038009f50 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:28.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.619+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:28.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.619+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:28.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.823+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038009f50 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:28.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.823+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038009f50 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:28.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.823+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:28.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:28.823+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:29.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:29.227+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038009f50 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:29.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:29.227+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038009f50 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:29.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:29.227+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:29.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:29.227+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:30.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:30.031+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038009f50 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:30.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:30.031+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038009f50 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:30.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:30.031+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:30.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:30.031+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:31.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:31.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:31.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038009f50 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038009f50 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:47:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f7038010330 2024-03-24T00:47:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f7038009960 2024-03-24T00:47:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f70380145a0 2024-03-24T00:47:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:31.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:31.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.619+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:31.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.823+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:31.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.823+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:31.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.823+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:31.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:31.823+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:32.227+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:32.227+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:32.227+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:32.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:32.227+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:33.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:33.031+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:33.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:33.031+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:33.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:33.031+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:33.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:33.031+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:34.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.619+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.619+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.619+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.619+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.619+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.619+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.619+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013fa0 2024-03-24T00:47:34.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.619+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f7038013b80 2024-03-24T00:47:34.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.619+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:34.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.619+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:34.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.619+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:34.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.619+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:34.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.823+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:34.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.823+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:34.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.823+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:34.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:34.823+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:35.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:35.227+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:35.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:35.227+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:35.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:35.227+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:35.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:35.227+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:36.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:36.031+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:36.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:36.031+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:36.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:36.031+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:36.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:36.031+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:37.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:37.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:37.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:37.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:37.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:37.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:47:37.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703800af90 con 0x7f7038010330 2024-03-24T00:47:37.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f7038009960 2024-03-24T00:47:37.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013b80 2024-03-24T00:47:37.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:37.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:37.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:37.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.619+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:37.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.823+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:37.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.823+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:37.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.823+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:37.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:37.823+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:38.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:38.227+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:38.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:38.227+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:38.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:38.227+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:38.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:38.227+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:39.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:39.031+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:39.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:39.031+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:39.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:39.031+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:39.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:39.031+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:39.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:47:39.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:47:40.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.619+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:40.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.619+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:40.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.619+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:40.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.619+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038010330 legacy=0x7f7038010770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.619+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038009f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.619+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.619+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013fa0 2024-03-24T00:47:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.619+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f70380145a0 2024-03-24T00:47:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.619+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.619+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:40.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.619+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:40.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.619+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.823+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.823+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.823+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:40.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:40.823+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:41.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:41.227+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:41.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:41.227+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:41.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:41.227+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:41.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:41.227+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:42.031+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:42.031+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:42.031+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:42.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:42.031+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:43.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:43.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:43.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:43.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:43.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:43.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:47:43.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:43.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f7038009960 2024-03-24T00:47:43.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013b80 2024-03-24T00:47:43.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f70380145a0 2024-03-24T00:47:43.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:43.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:43.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:43.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:43.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.619+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.823+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.823+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.823+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.823+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:43.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:43.823+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:44.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:44.227+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:44.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:44.227+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:44.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:44.227+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:44.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:44.227+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:44.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:44.227+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:45.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:45.031+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:45.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:45.031+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:45.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:45.031+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:45.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:45.031+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:45.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:45.031+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:46.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.619+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.619+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.619+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.619+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.619+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.619+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.619+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703800af90 con 0x7f7038013fa0 2024-03-24T00:47:46.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.619+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038010330 2024-03-24T00:47:46.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.619+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:46.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.619+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:46.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.619+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:46.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.823+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:46.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.823+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:46.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:46.823+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:47.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:47.227+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:47.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:47.227+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:47.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:47.227+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:48.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:48.031+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:48.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:48.031+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:48.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:48.031+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:49.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.619+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.619+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.619+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.619+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.619+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:47:49.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.619+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f7038013fa0 2024-03-24T00:47:49.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.619+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703800af90 con 0x7f7038009960 2024-03-24T00:47:49.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.619+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f70380145a0 2024-03-24T00:47:49.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.619+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:49.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.619+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:49.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.619+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:49.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.619+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:49.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.823+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:49.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.823+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:49.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.823+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:49.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:49.823+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:50.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:50.227+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:50.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:50.227+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:50.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:50.227+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:50.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:50.227+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:51.030 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:51.031+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:51.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:51.031+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:51.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:51.031+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:51.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:51.031+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:52.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.619+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:52.621 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.619+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:52.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.619+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:52.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.619+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:52.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.619+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:52.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.619+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:52.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.619+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013b80 2024-03-24T00:47:52.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.619+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f7038010330 2024-03-24T00:47:52.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.619+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:52.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.619+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:52.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.619+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:52.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.619+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:52.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.823+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:52.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.823+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:52.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.823+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:52.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:52.823+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:53.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:53.227+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:53.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:53.227+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:53.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:53.227+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:53.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:53.227+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:54.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:54.031+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:54.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:54.031+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:54.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:54.031+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:54.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:54.031+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:55.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:55.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:55.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:55.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:55.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:55.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:55.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:47:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f7038009960 2024-03-24T00:47:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013b80 2024-03-24T00:47:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f70380145a0 2024-03-24T00:47:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:55.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.619+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.823+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:55.822 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.823+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:55.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.823+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:55.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.823+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:55.823 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:55.823+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:56.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:56.227+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:56.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:56.227+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:56.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:56.227+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:56.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:56.227+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:56.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:56.227+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:57.031+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:57.031+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:57.031+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:57.031+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:47:57.031 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:57.031+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:47:58.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.623+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:58.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.623+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:58.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.623+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:58.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.623+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:47:58.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.623+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:47:58.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.623+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:47:58.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.623+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f7038013fa0 2024-03-24T00:47:58.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.623+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038010330 2024-03-24T00:47:58.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.623+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:58.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.623+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:58.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.623+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:47:58.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.827+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.827+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:58.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:58.827+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:47:59.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:59.231+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:47:59.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:59.231+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:47:59.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:47:59.231+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:00.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:00.035+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:00.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:00.035+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:00.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:00.035+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:01.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:01.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:01.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:01.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:48:01.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:01.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013b80 2024-03-24T00:48:01.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f7038010330 2024-03-24T00:48:01.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7038010030 con 0x7f70380145a0 2024-03-24T00:48:01.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:01.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:01.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:01.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:01.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:01.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.623+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.827+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.827+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.827+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.827+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.827+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:01.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:01.827+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:02.231+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:02.231+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:02.231+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:02.231+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:02.231+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:02.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:02.231+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:03.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:03.035+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:03.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:03.035+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:03.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:03.035+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:03.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:03.035+0000 7f7057e00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:03.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:03.035+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:03.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:03.035+0000 7f7057e00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:04.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:04.623+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 msgr2=0x7f7038008c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:04.623+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f70380145a0 0x7f7038008c20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:04.623+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:04.623+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:04.623+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:04.623+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:04.623+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f7038013fa0 2024-03-24T00:48:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:04.623+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f7038009960 2024-03-24T00:48:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:04.623+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:04.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:04.623+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:04.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:04.827+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:04.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:04.827+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:05.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:05.231+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:05.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:05.231+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:06.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:06.035+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:06.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:06.035+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:07.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.623+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:07.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.623+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:07.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.623+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:07.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.623+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:48:07.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.623+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703800af90 con 0x7f7038013fa0 2024-03-24T00:48:07.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.623+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f7038009960 2024-03-24T00:48:07.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.623+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013b80 2024-03-24T00:48:07.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.623+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:07.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.623+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:07.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.623+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:07.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.623+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:07.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.827+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:07.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.827+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:07.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.827+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:07.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:07.827+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:08.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:08.231+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:08.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:08.231+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:08.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:08.231+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:08.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:08.231+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:09.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:09.035+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:09.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:09.035+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:09.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:09.035+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:09.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:09.035+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:09.173 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:48:09.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:48:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.623+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:10.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.623+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013b80 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.623+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.623+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f703800f850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.623+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.623+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:10.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.623+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038010330 2024-03-24T00:48:10.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.623+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f703800a490 2024-03-24T00:48:10.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.623+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:10.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.623+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:10.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.623+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:10.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.623+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:10.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.827+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:10.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.827+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:10.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.827+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:10.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:10.827+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:11.231+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:11.231+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:11.231+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:11.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:11.231+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:12.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:12.035+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:12.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:12.035+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:12.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:12.035+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:12.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:12.035+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:13.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:13.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:13.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:13.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:48:13.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:13.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f7038009960 2024-03-24T00:48:13.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038010330 2024-03-24T00:48:13.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f703800a490 2024-03-24T00:48:13.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:13.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:13.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:13.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:13.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.623+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:13.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.827+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:13.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.827+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:13.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.827+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:13.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.827+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:13.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:13.827+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:14.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:14.231+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:14.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:14.231+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:14.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:14.231+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:14.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:14.231+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:14.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:14.231+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:15.035+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:15.035+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:15.035+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:15.035+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:15.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:15.035+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.623+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f70380086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.623+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f70380086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.623+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:16.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.623+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:16.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.623+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038010710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:16.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.623+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:16.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.623+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703800af90 con 0x7f7038013b80 2024-03-24T00:48:16.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.623+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013fa0 2024-03-24T00:48:16.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.623+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013b80 legacy=0x7f7038006660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:16.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.623+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:16.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.623+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.827+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013b80 legacy=0x7f7038006660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:16.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.827+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:16.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:16.827+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:17.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:17.231+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:17.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:17.231+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013b80 legacy=0x7f7038006660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:17.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:17.231+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:18.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:18.035+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:18.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:18.035+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013b80 legacy=0x7f7038006660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:18.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:18.035+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:19.624 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013b80 legacy=0x7f7038006660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:48:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f7038006d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013fa0 2024-03-24T00:48:19.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f7038010330 2024-03-24T00:48:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7038010030 con 0x7f703800a490 2024-03-24T00:48:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f7038006d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:19.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.623+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f7038006d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:19.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.827+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:19.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.827+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:19.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.827+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:19.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.827+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f7038006d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:19.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.827+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:19.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:19.827+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f7038006d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:20.231+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:20.231+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:20.231+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:20.231+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f7038006d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:20.231+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:20.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:20.231+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f7038006d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:21.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:21.035+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:21.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:21.035+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:21.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:21.035+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:21.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:21.035+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f7038006d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:21.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:21.035+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:21.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:21.035+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f7038006d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:22.622+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 msgr2=0x7f7038006d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:22.622+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800a490 0x7f7038006d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:22.622+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:22.622+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010330 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:22.622+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 msgr2=0x7f70380134f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:22.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:22.622+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038013fa0 0x7f70380134f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:22.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:22.622+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f70380072b0 2024-03-24T00:48:22.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:22.622+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703800af90 con 0x7f7038009960 2024-03-24T00:48:22.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:22.622+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f70380072b0 legacy=0x7f7038010710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:22.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:22.622+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038006660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:22.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:22.826+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f70380072b0 legacy=0x7f7038010710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:22.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:22.826+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038006660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:23.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:23.230+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f70380072b0 legacy=0x7f7038010710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:23.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:23.230+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038006660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:24.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:24.034+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f70380072b0 legacy=0x7f7038010710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:24.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:24.034+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038006660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:25.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.622+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038009960 legacy=0x7f7038006660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:25.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.622+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f70380072b0 legacy=0x7f7038010710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:25.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.622+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:48:25.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.622+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f703800f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:25.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.622+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f7038008db0 2024-03-24T00:48:25.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.622+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f7038013b80 2024-03-24T00:48:25.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.622+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013fa0 2024-03-24T00:48:25.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.622+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038008db0 legacy=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:25.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.622+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:25.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.622+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:25.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.622+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:25.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.826+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038008db0 legacy=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:25.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.826+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:25.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.826+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:25.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:25.826+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:26.230+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038008db0 legacy=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:26.230+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:26.230+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:26.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:26.230+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:27.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:27.034+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038008db0 legacy=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:27.034+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:27.034+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:27.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:27.034+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:28.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.626+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.626+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.626+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.626+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038008db0 legacy=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.626+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010310 0x7f70380106f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.626+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380114e0 0x7f7038014b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.626+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f70380114e0 2024-03-24T00:48:28.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.626+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f7038010310 2024-03-24T00:48:28.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.626+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380114e0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:28.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.626+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380114e0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:28.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.626+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010310 msgr2=0x7f70380106f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:28.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.626+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010310 0x7f70380106f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:28.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.830+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380114e0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.830+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380114e0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.830+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010310 msgr2=0x7f70380106f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:28.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:28.830+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010310 0x7f70380106f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:29.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:29.234+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380114e0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:29.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:29.234+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380114e0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:29.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:29.234+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010310 msgr2=0x7f70380106f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:29.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:29.234+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010310 0x7f70380106f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:30.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:30.038+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380114e0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:30.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:30.038+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380114e0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:30.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:30.038+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010310 msgr2=0x7f70380106f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:30.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:30.038+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010310 0x7f70380106f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:31.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010310 msgr2=0x7f70380106f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:31.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010310 0x7f70380106f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:31.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f70380114e0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:31.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f70380114e0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038015080 0x7f7038017470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:48:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f7038014b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703800af90 con 0x7f7038013b80 2024-03-24T00:48:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038015080 2024-03-24T00:48:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f7038013fa0 2024-03-24T00:48:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038015080 msgr2=0x7f7038017470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:31.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038015080 0x7f7038017470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:31.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.626+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.830+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038015080 msgr2=0x7f7038017470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.830+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038015080 0x7f7038017470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.830+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.830+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:31.830+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:32.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:32.234+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038015080 msgr2=0x7f7038017470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:32.234+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038015080 0x7f7038017470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:32.234+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:32.234+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:32.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:32.234+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:33.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:33.038+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038015080 msgr2=0x7f7038017470 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:33.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:33.038+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038015080 0x7f7038017470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:33.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:33.038+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:33.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:33.038+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:33.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:33.038+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:34.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.622+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:34.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.622+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.622+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038015080 msgr2=0x7f7038017470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:34.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.622+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038015080 0x7f7038017470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:34.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.622+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:34.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.622+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038008db0 0x7f703800f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:34.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.622+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f7038009f50 2024-03-24T00:48:34.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.622+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038008db0 2024-03-24T00:48:34.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.622+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038009f50 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:34.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.622+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038008db0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:34.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.622+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038008db0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:34.826 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.826+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038009f50 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:34.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.826+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038008db0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:34.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:34.826+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038008db0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:35.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:35.230+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038009f50 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:35.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:35.230+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038008db0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:35.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:35.230+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038008db0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:36.034 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:36.034+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038009f50 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:36.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:36.034+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038008db0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:36.035 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:36.034+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038008db0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:37.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.626+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038008db0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:37.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.626+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038008db0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:37.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.626+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038009f50 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:37.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.626+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:48:37.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.626+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f7038014b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:37.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.626+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703800af90 con 0x7f7038009f50 2024-03-24T00:48:37.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.626+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f7038013b80 2024-03-24T00:48:37.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.626+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038013fa0 2024-03-24T00:48:37.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.626+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038009f50 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:37.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.626+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:37.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.626+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:37.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.626+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:37.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.830+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038009f50 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:37.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.830+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:37.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.830+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:37.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:37.830+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:38.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:38.234+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038009f50 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:38.234+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:38.234+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:38.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:38.234+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:39.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:39.038+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038009f50 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:39.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:39.038+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:39.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:39.038+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:39.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:39.038+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:39.262 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:48:39.265 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:48:40.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.626+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:40.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.626+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013fa0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:40.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.626+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:40.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.626+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038009f50 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:40.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.626+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 0x7f703800f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.626+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010a20 0x7f703800a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:40.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.626+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038008db0 2024-03-24T00:48:40.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.626+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f7038010a20 2024-03-24T00:48:40.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.626+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:40.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.626+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:40.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.626+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010a20 msgr2=0x7f703800a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:40.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.626+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010a20 0x7f703800a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:40.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.830+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.830+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.830+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010a20 msgr2=0x7f703800a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:40.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:40.830+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010a20 0x7f703800a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:41.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:41.234+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:41.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:41.234+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:41.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:41.234+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010a20 msgr2=0x7f703800a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:41.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:41.234+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010a20 0x7f703800a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:42.038+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:42.038+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:42.038+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010a20 msgr2=0x7f703800a330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:42.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:42.038+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010a20 0x7f703800a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:43.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f7038010a20 msgr2=0x7f703800a330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:43.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7038010a20 0x7f703800a330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 msgr2=0x7f703800f860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:43.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 0x7f703800f860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:43.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f70380190a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:43.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:48:43.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f7038013fa0 2024-03-24T00:48:43.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703800af90 con 0x7f7038013b80 2024-03-24T00:48:43.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f70380146b0 2024-03-24T00:48:43.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:43.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f70380190a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:43.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:43.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.626+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f70380190a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:43.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.830+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:43.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.830+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:43.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.830+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f70380190a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:43.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:43.830+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f70380190a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:44.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:44.234+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:44.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:44.234+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f70380190a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:44.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:44.234+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:44.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:44.234+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f70380190a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:45.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:45.038+0000 7f705cc00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:45.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:45.038+0000 7f7057400640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:45.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:45.038+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f70380190a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:45.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:45.038+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f70380190a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:46.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.626+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f70380190a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:46.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.626+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f70380190a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:46.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.626+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:46.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.626+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:46.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.626+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 0x7f7038014b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.626+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 0x7f7038009f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:46.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.626+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f7038008db0 2024-03-24T00:48:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.626+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f703800f860 2024-03-24T00:48:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.626+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.626+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.626+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:46.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.626+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.830+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.830+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.830+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:46.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:46.830+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:47.234+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:47.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:47.234+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:47.234+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:47.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:47.234+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:48.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:48.038+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:48.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:48.038+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:48.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:48.038+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:48.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:48.038+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:49.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f7038008db0 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 0x7f7038019680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 0x7f7038014b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:48:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f7038013b80 2024-03-24T00:48:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f703800a490 2024-03-24T00:48:49.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f703800f860 2024-03-24T00:48:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 msgr2=0x7f7038019680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 0x7f7038019680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:49.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.626+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:49.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.830+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 msgr2=0x7f7038019680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.830+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 0x7f7038019680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.830+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.830+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:49.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:49.830+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:50.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:50.234+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 msgr2=0x7f7038019680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:50.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:50.234+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 0x7f7038019680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:50.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:50.234+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:50.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:50.234+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:50.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:50.234+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:51.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:51.038+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 msgr2=0x7f7038019680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:51.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:51.038+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 0x7f7038019680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:51.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:51.038+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:51.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:51.038+0000 7f705cc00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:51.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:51.038+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:52.628 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.626+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 msgr2=0x7f7038014b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:52.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.626+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f703800f860 0x7f7038014b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.626+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 msgr2=0x7f7038019680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:52.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.626+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 0x7f7038019680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:52.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.626+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013b80 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:52.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.626+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:52.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.626+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f7038013fa0 2024-03-24T00:48:52.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.626+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f70380146b0 2024-03-24T00:48:52.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.626+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:52.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.626+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:52.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.626+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:52.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.830+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:52.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.830+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:52.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:52.830+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:53.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:53.234+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:53.234+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:53.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:53.234+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:54.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:54.038+0000 7f7057e00640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:54.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:54.038+0000 7f7057400640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:54.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:54.038+0000 7f7057400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:55.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:55.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:55.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f7038013fa0 legacy=0x7f7038010310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:55.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:48:55.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 0x7f7038014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:55.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f703800a490 2024-03-24T00:48:55.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f70380146b0 2024-03-24T00:48:55.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f7038013b80 2024-03-24T00:48:55.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800a490 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:55.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:55.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:55.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 msgr2=0x7f7038014ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:55.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.626+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 0x7f7038014ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.830+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800a490 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.830+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:55.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.830+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 msgr2=0x7f7038014ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:55.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.830+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:55.830 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:55.830+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 0x7f7038014ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:56.234 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:56.234+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800a490 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:56.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:56.234+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:56.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:56.234+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 msgr2=0x7f7038014ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:56.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:56.234+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:56.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:56.234+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 0x7f7038014ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:57.038 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:57.038+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800a490 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:48:57.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:57.038+0000 7f705cc00640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 msgr2=0x7f7038014ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:48:57.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:57.038+0000 7f7057e00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:48:57.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:57.038+0000 7f7057e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:57.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:57.038+0000 7f705cc00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 0x7f7038014ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:48:58.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.630+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 msgr2=0x7f7038014ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:58.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.630+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 0x7f7038014ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.630+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:58.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.630+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:48:58.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.630+0000 7f7055600640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800a490 legacy=0x7f70380086a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:48:58.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.630+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800c3d0 0x7f703800c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:48:58.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.630+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f7038013fa0 2024-03-24T00:48:58.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.630+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f703800c3d0 2024-03-24T00:48:58.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.630+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800c3d0 msgr2=0x7f703800c7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:58.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.630+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800c3d0 0x7f703800c7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:48:58.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.630+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013fa0 legacy=0x7f70380102d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:58.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.834+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800c3d0 msgr2=0x7f703800c7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:58.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.834+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800c3d0 0x7f703800c7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:48:58.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:58.834+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013fa0 legacy=0x7f70380102d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:48:59.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:59.238+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800c3d0 msgr2=0x7f703800c7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:48:59.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:59.238+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800c3d0 0x7f703800c7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:48:59.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:48:59.238+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013fa0 legacy=0x7f70380102d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:49:00.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:00.042+0000 7f7057400640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800c3d0 msgr2=0x7f703800c7e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:49:00.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:00.042+0000 7f7057400640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800c3d0 0x7f703800c7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:49:00.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:00.042+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013fa0 legacy=0x7f70380102d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:49:01.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f7055600640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800c3d0 msgr2=0x7f703800c7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:49:01.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800c3d0 0x7f703800c7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:01.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013fa0 legacy=0x7f70380102d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:49:01.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f705cc00640 1 -- reap_dead start 2024-03-24T00:49:01.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 0x7f7038014ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:01.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:01.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f7055600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058071760 con 0x7f7038013fa0 2024-03-24T00:49:01.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f7055600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f70380146b0 2024-03-24T00:49:01.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f7055600640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7058071480 con 0x7f7038013b80 2024-03-24T00:49:01.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013fa0 legacy=0x7f70380102d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:49:01.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 msgr2=0x7f7038014ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:49:01.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:49:01.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 0x7f7038014ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:49:01.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.630+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:49:01.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.834+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:49:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.834+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 msgr2=0x7f7038014ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:49:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.834+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013fa0 legacy=0x7f70380102d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:49:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.834+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 0x7f7038014ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:49:01.835 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:01.834+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:49:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:02.238+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013fa0 legacy=0x7f70380102d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:49:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:02.238+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 msgr2=0x7f7038014ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:49:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:02.238+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 0x7f7038014ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:49:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:02.238+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:49:02.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:02.238+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:49:03.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:03.042+0000 7f7057e00640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013fa0 legacy=0x7f70380102d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.53:6789/0 2024-03-24T00:49:03.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:03.042+0000 7f705cc00640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3300/0 2024-03-24T00:49:03.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:03.042+0000 7f7057400640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 msgr2=0x7f7038014ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.53:3301/0 2024-03-24T00:49:03.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:03.042+0000 7f705cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:49:03.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:03.042+0000 7f7057400640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 0x7f7038014ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:49:04.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.630+0000 7f7055600640 1 -- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 msgr2=0x7f7038014ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:49:04.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.630+0000 7f7055600640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7038013b80 0x7f7038014ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.630+0000 7f7055600640 1 -- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 msgr2=0x7f7038009f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:49:04.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.630+0000 7f7055600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70380146b0 0x7f7038009f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:49:04.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.630+0000 7f7055600640 1 -- >> v1:172.21.15.53:6789/0 conn(0x7f7038013fa0 legacy=0x7f70380102d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:49:04.630 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.630+0000 7f7055600640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 0x7f7038008660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:49:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.630+0000 7f7055600640 1 -- --> v1:172.21.15.40:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380096e0 con 0x7f703800ceb0 2024-03-24T00:49:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.630+0000 7f7055600640 1 -- --> v2:172.21.15.40:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7058071230 con 0x7f703800a490 2024-03-24T00:49:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.630+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800ceb0 legacy=0x7f703800e850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:49:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.630+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 msgr2=0x7f7038008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:49:04.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.630+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 0x7f7038008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:49:04.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.834+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800ceb0 legacy=0x7f703800e850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:49:04.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.834+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 msgr2=0x7f7038008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:49:04.834 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:04.834+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 0x7f7038008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:49:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:05.238+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800ceb0 legacy=0x7f703800e850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:49:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:05.238+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 msgr2=0x7f7038008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:49:05.238 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:05.238+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 0x7f7038008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:49:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:06.042+0000 7f7057400640 1 -- >> v1:172.21.15.40:6789/0 conn(0x7f703800ceb0 legacy=0x7f703800e850 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.40:6789/0 2024-03-24T00:49:06.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:06.042+0000 7f7057e00640 1 -- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 msgr2=0x7f7038008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.40:3300/0 2024-03-24T00:49:06.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-03-24T00:49:06.042+0000 7f7057e00640 1 --2- >> v2:172.21.15.40:3300/0 conn(0x7f703800a490 0x7f7038008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:49:07.499 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T00:49:07.500 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-24T00:49:07.514 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-24T00:49:07.515 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-24T00:49:07.515 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.517 INFO:tasks.ceph.mds.a:Stopped 2024-03-24T00:49:07.517 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-24T00:49:07.518 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.518 INFO:tasks.ceph.mds.c:Stopped 2024-03-24T00:49:07.518 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-24T00:49:07.518 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:49:07.519 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b' 2024-03-24T00:49:07.529 INFO:tasks.ceph.mds.b:Stopped 2024-03-24T00:49:07.529 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-24T00:49:07.529 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:49:07.529 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d' 2024-03-24T00:49:07.540 INFO:tasks.ceph.mds.d:Stopped 2024-03-24T00:49:07.540 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-24T00:49:07.540 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-24T00:49:07.540 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.541 INFO:tasks.ceph.osd.0:Stopped 2024-03-24T00:49:07.541 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-24T00:49:07.541 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.541 INFO:tasks.ceph.osd.1:Stopped 2024-03-24T00:49:07.541 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-24T00:49:07.541 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.542 INFO:tasks.ceph.osd.2:Stopped 2024-03-24T00:49:07.542 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-24T00:49:07.542 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.542 INFO:tasks.ceph.osd.3:Stopped 2024-03-24T00:49:07.542 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-24T00:49:07.542 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.543 INFO:tasks.ceph.osd.4:Stopped 2024-03-24T00:49:07.543 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-24T00:49:07.543 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.543 INFO:tasks.ceph.osd.5:Stopped 2024-03-24T00:49:07.543 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-24T00:49:07.543 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.544 INFO:tasks.ceph.osd.6:Stopped 2024-03-24T00:49:07.544 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-24T00:49:07.544 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.544 INFO:tasks.ceph.osd.7:Stopped 2024-03-24T00:49:07.544 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-24T00:49:07.544 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-24T00:49:07.544 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:49:07.545 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y' 2024-03-24T00:49:07.556 INFO:tasks.ceph.mgr.y:Stopped 2024-03-24T00:49:07.556 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-24T00:49:07.556 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.556 INFO:tasks.ceph.mgr.x:Stopped 2024-03-24T00:49:07.556 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-24T00:49:07.557 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-24T00:49:07.557 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.557 INFO:tasks.ceph.mon.a:Stopped 2024-03-24T00:49:07.557 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-24T00:49:07.557 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.557 INFO:tasks.ceph.mon.b:Stopped 2024-03-24T00:49:07.557 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-24T00:49:07.558 INFO:teuthology.orchestra.run:waiting for 300 2024-03-24T00:49:07.560 INFO:tasks.ceph.mon.c:Stopped 2024-03-24T00:49:07.560 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-24T00:49:07.560 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'Replacing daemon mds' | head -n 1 2024-03-24T00:49:07.605 INFO:teuthology.orchestra.run.smithi040.stdout:2024-03-24T00:43:08.287932+0000 mds.a (mds.0) 1 : cluster [WRN] 1 slow requests, 1 included below; oldest blocked for > 181.347407 secs 2024-03-24T00:49:07.606 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T00:49:07.606 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:49:07.606 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-24T00:49:07.839 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:49:07.840 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-24T00:49:08.092 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:49:08.092 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-24T00:49:08.260 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:49:08.260 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-24T00:49:08.448 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:49:08.449 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-24T00:49:08.638 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:49:08.639 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-24T00:49:08.966 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:49:08.967 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-24T00:49:09.120 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:49:09.121 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-24T00:49:09.199 INFO:tasks.ceph:Archiving mon data... 2024-03-24T00:49:09.200 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439/data/mon.a.tgz 2024-03-24T00:49:09.200 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-03-24T00:49:09.210 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.Dq8UAhVafO 2024-03-24T00:49:09.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Dq8UAhVafO 2024-03-24T00:49:09.324 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:49:09.327 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:49:09.354 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.Dq8UAhVafO 2024-03-24T00:49:09.503 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.Dq8UAhVafO is 391KB 2024-03-24T00:49:09.568 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.Dq8UAhVafO 2024-03-24T00:49:09.576 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439/data/mon.b.tgz 2024-03-24T00:49:09.576 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-03-24T00:49:09.586 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.kB33OMYoCa 2024-03-24T00:49:09.586 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.kB33OMYoCa 2024-03-24T00:49:09.732 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.kB33OMYoCa 2024-03-24T00:49:09.812 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.kB33OMYoCa is 391KB 2024-03-24T00:49:09.874 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.kB33OMYoCa 2024-03-24T00:49:09.882 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439/data/mon.c.tgz 2024-03-24T00:49:09.882 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-03-24T00:49:09.931 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.baTO3FgvSH 2024-03-24T00:49:09.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.baTO3FgvSH 2024-03-24T00:49:10.069 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.baTO3FgvSH 2024-03-24T00:49:10.144 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.baTO3FgvSH is 391KB 2024-03-24T00:49:10.206 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.baTO3FgvSH 2024-03-24T00:49:10.214 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-24T00:49:10.214 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-24T00:49:10.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-24T00:49:10.286 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-24T00:49:10.324 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-24T00:49:10.324 INFO:tasks.ceph:Archiving crash dumps... 2024-03-24T00:49:10.326 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439/remote/smithi040/crash 2024-03-24T00:49:10.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-24T00:49:10.355 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439/remote/smithi053/crash 2024-03-24T00:49:10.356 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-24T00:49:10.480 INFO:tasks.ceph:Shutting down logrotate 2024-03-24T00:49:10.480 DEBUG:teuthology.orchestra.run.smithi040:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:49:10.483 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-24T00:49:10.555 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-24T00:49:10.577 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-24T00:49:10.596 INFO:tasks.ceph:Compressing logs... 2024-03-24T00:49:10.596 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T00:49:10.621 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T00:49:10.652 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47737.log 2024-03-24T00:49:10.653 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35894.log 2024-03-24T00:49:10.654 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53416.log 2024-03-24T00:49:10.655 INFO:teuthology.orchestra.run.smithi040.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47737.log.gz 2024-03-24T00:49:10.655 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56065.log 2024-03-24T00:49:10.655 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35894.log.gz 2024-03-24T00:49:10.656 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35018.log 2024-03-24T00:49:10.656 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53416.log.gz 2024-03-24T00:49:10.657 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47659.log 2024-03-24T00:49:10.657 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56065.log.gz 2024-03-24T00:49:10.658 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46745.log 2024-03-24T00:49:10.658 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35018.log.gz 2024-03-24T00:49:10.659 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43087.log 2024-03-24T00:49:10.659 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47659.log.gz 2024-03-24T00:49:10.659 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44142.log 2024-03-24T00:49:10.659 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46745.log.gz 2024-03-24T00:49:10.660 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37102.log 2024-03-24T00:49:10.660 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43087.log.gz 2024-03-24T00:49:10.661 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54376.log 2024-03-24T00:49:10.661 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44142.log.gz 2024-03-24T00:49:10.662 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33843.log 2024-03-24T00:49:10.662 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37102.log.gz 2024-03-24T00:49:10.663 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53535.log 2024-03-24T00:49:10.664 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54376.log.gz 2024-03-24T00:49:10.664 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37206.log 2024-03-24T00:49:10.664 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33843.log.gz 2024-03-24T00:49:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32751.log 2024-03-24T00:49:10.665 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53535.log.gz 2024-03-24T00:49:10.665 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41291.log 2024-03-24T00:49:10.666 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37206.log.gz 2024-03-24T00:49:10.666 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34810.log 2024-03-24T00:49:10.666 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32751.log.gz 2024-03-24T00:49:10.667 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52589.log 2024-03-24T00:49:10.667 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41291.log.gz 2024-03-24T00:49:10.668 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43972.log 2024-03-24T00:49:10.668 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34810.log.gz 2024-03-24T00:49:10.669 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40810.log 2024-03-24T00:49:10.669 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52589.log.gz 2024-03-24T00:49:10.670 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41909.log 2024-03-24T00:49:10.670 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43972.log.gz 2024-03-24T00:49:10.671 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27401.log 2024-03-24T00:49:10.671 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27488.log 2024-03-24T00:49:10.672 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41909.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.40810.log.gz 2024-03-24T00:49:10.672 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54411.log 2024-03-24T00:49:10.672 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41909.log.gz 2024-03-24T00:49:10.672 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.27401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33061.log 2024-03-24T00:49:10.673 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27401.log.gz 2024-03-24T00:49:10.677 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27584.log 2024-03-24T00:49:10.678 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.27488.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-24T00:49:10.678 INFO:teuthology.orchestra.run.smithi053.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.27488.log.gz 2024-03-24T00:49:10.678 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.27584.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-24T00:49:10.678 INFO:teuthology.orchestra.run.smithi053.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.27584.log.gz 2024-03-24T00:49:10.678 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27512.log 2024-03-24T00:49:10.678 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27560.log 2024-03-24T00:49:10.678 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.27512.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-24T00:49:10.678 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.27512.log.gz 2024-03-24T00:49:10.679 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.27560.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-24T00:49:10.679 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.27560.log.gz 2024-03-24T00:49:10.679 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27440.log 2024-03-24T00:49:10.681 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-24T00:49:10.681 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.27440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27464.log 2024-03-24T00:49:10.682 INFO:teuthology.orchestra.run.smithi053.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.27440.log.gz 2024-03-24T00:49:10.682 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-24T00:49:10.683 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50291.log 2024-03-24T00:49:10.683 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54411.log.gz 2024-03-24T00:49:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44198.log 2024-03-24T00:49:10.684 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33061.log.gz 2024-03-24T00:49:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44081.log 2024-03-24T00:49:10.684 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50291.log.gz 2024-03-24T00:49:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35631.log 2024-03-24T00:49:10.684 INFO:teuthology.orchestra.run.smithi040.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44198.log.gz 2024-03-24T00:49:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33538.log 2024-03-24T00:49:10.684 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44081.log.gz 2024-03-24T00:49:10.684 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39440.log 2024-03-24T00:49:10.685 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35631.log.gz 2024-03-24T00:49:10.685 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48421.log 2024-03-24T00:49:10.685 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.33538.log.gz 2024-03-24T00:49:10.685 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45277.log 2024-03-24T00:49:10.685 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39440.log.gz 2024-03-24T00:49:10.685 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52120.log 2024-03-24T00:49:10.685 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45277.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.48421.log.gz 2024-03-24T00:49:10.685 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43199.log 2024-03-24T00:49:10.685 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45277.log.gz 2024-03-24T00:49:10.686 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35314.log 2024-03-24T00:49:10.686 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52120.log.gz 2024-03-24T00:49:10.686 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38799.log 2024-03-24T00:49:10.686 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43199.log.gz 2024-03-24T00:49:10.686 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44485.log 2024-03-24T00:49:10.686 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35314.log.gz 2024-03-24T00:49:10.690 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.27464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27416.log 2024-03-24T00:49:10.691 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.27464.log.gz 2024-03-24T00:49:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.32042.log 2024-03-24T00:49:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.27416.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.27416.log.gz 2024-03-24T00:49:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.3.32042.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-24T00:49:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-24T00:49:10.691 INFO:teuthology.orchestra.run.smithi053.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.3.32042.log.gz 2024-03-24T00:49:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-24T00:49:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-24T00:49:10.694 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-24T00:49:10.694 INFO:teuthology.orchestra.run.smithi053.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-24T00:49:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38799.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-24T00:49:10.695 INFO:teuthology.orchestra.run.smithi040.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38799.log.gz 2024-03-24T00:49:10.695 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.44485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51247.log 2024-03-24T00:49:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49818.log 2024-03-24T00:49:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40208.log 2024-03-24T00:49:10.696 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51247.log.gz 2024-03-24T00:49:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33009.log 2024-03-24T00:49:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40208.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.49818.log.gz 2024-03-24T00:49:10.696 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40208.log.gz 2024-03-24T00:49:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51273.log 2024-03-24T00:49:10.696 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41239.log 2024-03-24T00:49:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33009.log: /var/log/ceph/ceph-client.admin.51273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53978.log 2024-03-24T00:49:10.697 INFO:teuthology.orchestra.run.smithi040.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33009.log.gz 2024-03-24T00:49:10.697 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51273.log.gz 2024-03-24T00:49:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39312.log 2024-03-24T00:49:10.697 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41239.log.gz 2024-03-24T00:49:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53978.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.53978.log.gz 2024-03-24T00:49:10.697 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42143.log 2024-03-24T00:49:10.697 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39125.log 2024-03-24T00:49:10.697 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39312.log.gz 2024-03-24T00:49:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38981.log 2024-03-24T00:49:10.698 INFO:teuthology.orchestra.run.smithi040.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42143.log.gz 2024-03-24T00:49:10.698 INFO:teuthology.orchestra.run.smithi040.stderr: 94.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-24T00:49:10.698 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36081.log 2024-03-24T00:49:10.698 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39125.log.gz 2024-03-24T00:49:10.699 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.31936.log 2024-03-24T00:49:10.700 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.1.31936.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-24T00:49:10.700 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27536.log 2024-03-24T00:49:10.702 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.37363.log 2024-03-24T00:49:10.702 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38981.log.gz 2024-03-24T00:49:10.702 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35181.log 2024-03-24T00:49:10.702 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36081.log.gz 2024-03-24T00:49:10.702 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.37363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41161.log 2024-03-24T00:49:10.702 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36516.log 2024-03-24T00:49:10.703 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35181.log.gz 2024-03-24T00:49:10.703 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35340.log 2024-03-24T00:49:10.703 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41161.log.gz 2024-03-24T00:49:10.703 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42745.log 2024-03-24T00:49:10.703 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36516.log.gz 2024-03-24T00:49:10.705 INFO:teuthology.orchestra.run.smithi053.stderr: 89.8%/var/log/ceph/ceph-client.admin.27536.log: /var/log/ceph/ceph-osd.4.log: -- replaced with /var/log/ceph/ceph-client.1.31936.log.gz 2024-03-24T00:49:10.705 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32892.log 2024-03-24T00:49:10.706 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35340.log.gz 2024-03-24T00:49:10.706 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42745.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42745.log.gz 2024-03-24T00:49:10.706 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42716.log 2024-03-24T00:49:10.706 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51065.log 2024-03-24T00:49:10.706 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32892.log.gz 2024-03-24T00:49:10.706 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45627.log 2024-03-24T00:49:10.707 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42716.log.gz 2024-03-24T00:49:10.708 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47138.log 2024-03-24T00:49:10.708 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51065.log.gz 2024-03-24T00:49:10.708 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40423.log 2024-03-24T00:49:10.708 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45627.log.gz 2024-03-24T00:49:10.709 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.27536.log.gz 2024-03-24T00:49:10.710 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53246.log 2024-03-24T00:49:10.710 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47138.log.gz 2024-03-24T00:49:10.711 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53129.log 2024-03-24T00:49:10.711 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40423.log.gz 2024-03-24T00:49:10.711 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39991.log 2024-03-24T00:49:10.711 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53246.log.gz 2024-03-24T00:49:10.712 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33346.log 2024-03-24T00:49:10.712 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53129.log.gz 2024-03-24T00:49:10.713 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39991.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-24T00:49:10.713 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.39991.log.gz 2024-03-24T00:49:10.713 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38345.log 2024-03-24T00:49:10.714 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33346.log.gz 2024-03-24T00:49:10.714 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55534.log 2024-03-24T00:49:10.715 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48051.log 2024-03-24T00:49:10.716 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.38345.log.gz 2024-03-24T00:49:10.717 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45061.log 2024-03-24T00:49:10.717 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55534.log.gz 2024-03-24T00:49:10.717 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39008.log 2024-03-24T00:49:10.718 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45061.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48051.log.gz 2024-03-24T00:49:10.719 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45061.log.gz 2024-03-24T00:49:10.719 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52745.log 2024-03-24T00:49:10.720 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39008.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39008.log.gz 2024-03-24T00:49:10.720 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52451.log 2024-03-24T00:49:10.721 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.52745.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41407.log 2024-03-24T00:49:10.722 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52745.log.gz 2024-03-24T00:49:10.722 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33895.log 2024-03-24T00:49:10.723 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52451.log.gz 2024-03-24T00:49:10.724 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50074.log 2024-03-24T00:49:10.724 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41407.log.gz 2024-03-24T00:49:10.724 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54440.log 2024-03-24T00:49:10.725 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33895.log: /var/log/ceph/ceph-client.admin.50074.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33895.log.gz 2024-03-24T00:49:10.726 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50074.log.gz 2024-03-24T00:49:10.726 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37154.log 2024-03-24T00:49:10.726 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42678.log 2024-03-24T00:49:10.727 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54440.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54440.log.gz 2024-03-24T00:49:10.728 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37154.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37154.log.gz 2024-03-24T00:49:10.728 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38747.log 2024-03-24T00:49:10.728 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37050.log 2024-03-24T00:49:10.729 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42678.log: /var/log/ceph/ceph-client.admin.38747.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42678.log.gz 2024-03-24T00:49:10.730 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38747.log.gzgzip 2024-03-24T00:49:10.730 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41005.log 2024-03-24T00:49:10.730 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50343.log 2024-03-24T00:49:10.731 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37050.log: /var/log/ceph/ceph-client.admin.41005.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37050.log.gz 2024-03-24T00:49:10.732 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41213.log 87.8% 2024-03-24T00:49:10.732 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41005.log.gz 2024-03-24T00:49:10.732 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.50343.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48289.log 2024-03-24T00:49:10.733 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50343.log.gz 2024-03-24T00:49:10.734 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42400.log 2024-03-24T00:49:10.734 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41213.log.gz 2024-03-24T00:49:10.735 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41589.log 2024-03-24T00:49:10.735 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48289.log.gz 2024-03-24T00:49:10.736 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46182.log 2024-03-24T00:49:10.736 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42400.log.gz 2024-03-24T00:49:10.737 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48237.log 2024-03-24T00:49:10.738 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41589.log.gz 2024-03-24T00:49:10.738 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33560.log 2024-03-24T00:49:10.738 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46182.log.gz 2024-03-24T00:49:10.739 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45407.log 2024-03-24T00:49:10.740 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48237.log.gz 2024-03-24T00:49:10.740 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46774.log 2024-03-24T00:49:10.741 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33560.log.gz 2024-03-24T00:49:10.741 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45199.log 2024-03-24T00:49:10.742 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45407.log.gz 2024-03-24T00:49:10.743 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43411.log 2024-03-24T00:49:10.743 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46774.log.gz 2024-03-24T00:49:10.744 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45251.log 2024-03-24T00:49:10.744 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45199.log.gz 2024-03-24T00:49:10.745 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34784.log 2024-03-24T00:49:10.745 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43411.log.gz 2024-03-24T00:49:10.746 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46286.log 2024-03-24T00:49:10.746 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45251.log.gz 2024-03-24T00:49:10.747 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45783.log 2024-03-24T00:49:10.747 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34784.log: /var/log/ceph/ceph-client.admin.46286.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34784.log.gz 2024-03-24T00:49:10.748 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45601.log 2024-03-24T00:49:10.748 INFO:teuthology.orchestra.run.smithi040.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46286.log.gz 2024-03-24T00:49:10.749 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32809.log 2024-03-24T00:49:10.749 INFO:teuthology.orchestra.run.smithi040.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45783.log.gz 2024-03-24T00:49:10.750 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50538.log 2024-03-24T00:49:10.750 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9%/var/log/ceph/ceph-client.admin.32809.log: -- replaced with /var/log/ceph/ceph-client.admin.45601.log.gz 2024-03-24T00:49:10.751 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41721.log 2024-03-24T00:49:10.751 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32809.log.gz 2024-03-24T00:49:10.752 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45225.log 2024-03-24T00:49:10.752 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50538.log.gz 2024-03-24T00:49:10.753 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35842.log/var/log/ceph/ceph-client.admin.41721.log: 2024-03-24T00:49:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45225.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41721.log.gz 2024-03-24T00:49:10.754 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43356.log 2024-03-24T00:49:10.755 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45225.log.gz 2024-03-24T00:49:10.755 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55498.log 2024-03-24T00:49:10.756 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35842.log.gz 2024-03-24T00:49:10.756 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.43356.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36341.log 2024-03-24T00:49:10.757 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55498.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43356.log.gz 2024-03-24T00:49:10.758 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48159.log 2024-03-24T00:49:10.758 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55498.log.gz 2024-03-24T00:49:10.759 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32574.log 2024-03-24T00:49:10.759 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36341.log.gz 2024-03-24T00:49:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43764.log 2024-03-24T00:49:10.760 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48159.log: /var/log/ceph/ceph-client.admin.32574.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48159.log.gz 2024-03-24T00:49:10.761 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39177.log 2024-03-24T00:49:10.761 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32574.log.gz 2024-03-24T00:49:10.762 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46104.log 2024-03-24T00:49:10.762 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43764.log.gz 2024-03-24T00:49:10.763 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53878.log 2024-03-24T00:49:10.763 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39177.log.gz 2024-03-24T00:49:10.764 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47406.log 2024-03-24T00:49:10.764 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46104.log.gz 2024-03-24T00:49:10.765 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.53878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38864.log 2024-03-24T00:49:10.766 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37463.log 2024-03-24T00:49:10.767 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47406.log.gz 2024-03-24T00:49:10.768 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47607.log 2024-03-24T00:49:10.768 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38864.log.gz 2024-03-24T00:49:10.769 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35605.log 2024-03-24T00:49:10.769 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37463.log.gz 2024-03-24T00:49:10.770 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52972.log 2024-03-24T00:49:10.770 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47607.log.gz 2024-03-24T00:49:10.771 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35366.log 2024-03-24T00:49:10.771 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35605.log.gz 2024-03-24T00:49:10.772 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37258.log 2024-03-24T00:49:10.772 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52972.log.gz 2024-03-24T00:49:10.773 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45381.log 2024-03-24T00:49:10.773 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35366.log.gz 2024-03-24T00:49:10.774 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47112.log 2024-03-24T00:49:10.775 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.37258.log.gz 2024-03-24T00:49:10.775 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54516.log 2024-03-24T00:49:10.776 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45381.log.gz 2024-03-24T00:49:10.777 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35449.log 2024-03-24T00:49:10.777 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47112.log.gz 2024-03-24T00:49:10.777 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36574.log 2024-03-24T00:49:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35449.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54516.log.gz 2024-03-24T00:49:10.779 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42039.log 2024-03-24T00:49:10.779 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35449.log.gz 2024-03-24T00:49:10.780 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37024.log 2024-03-24T00:49:10.780 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36574.log.gz 2024-03-24T00:49:10.781 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45994.log 2024-03-24T00:49:10.781 INFO:teuthology.orchestra.run.smithi040.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42039.log.gz 2024-03-24T00:49:10.782 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51013.log 2024-03-24T00:49:10.782 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37024.log.gz 2024-03-24T00:49:10.783 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47190.log 2024-03-24T00:49:10.784 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45994.log.gz 2024-03-24T00:49:10.784 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41355.log 2024-03-24T00:49:10.785 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51013.log.gz 2024-03-24T00:49:10.785 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38189.log 2024-03-24T00:49:10.786 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47190.log.gz 2024-03-24T00:49:10.786 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38011.log 2024-03-24T00:49:10.787 INFO:teuthology.orchestra.run.smithi040.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41355.log.gz 2024-03-24T00:49:10.788 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43385.log 2024-03-24T00:49:10.788 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38189.log.gz 2024-03-24T00:49:10.789 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43605.log 2024-03-24T00:49:10.789 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38011.log.gz 2024-03-24T00:49:10.790 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50022.log 2024-03-24T00:49:10.791 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43385.log.gz 2024-03-24T00:49:10.791 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34550.log 2024-03-24T00:49:10.792 INFO:teuthology.orchestra.run.smithi040.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43605.log.gz 2024-03-24T00:49:10.792 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50022.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-24T00:49:10.793 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50022.log.gz 2024-03-24T00:49:10.793 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42169.log 2024-03-24T00:49:10.794 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34550.log.gz 2024-03-24T00:49:10.794 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40477.log 2024-03-24T00:49:10.796 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52275.log 2024-03-24T00:49:10.796 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42169.log.gz 2024-03-24T00:49:10.797 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45433.log 2024-03-24T00:49:10.797 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40477.log.gz 2024-03-24T00:49:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52275.log: 91.1%/var/log/ceph/ceph-client.admin.45433.log: -- replaced with /var/log/ceph/ceph-client.admin.52275.log.gz 2024-03-24T00:49:10.799 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35501.log 2024-03-24T00:49:10.800 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45433.log.gz 2024-03-24T00:49:10.800 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35711.log 2024-03-24T00:49:10.802 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44849.log 2024-03-24T00:49:10.803 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35501.log.gz 2024-03-24T00:49:10.803 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35711.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35711.log.gz 2024-03-24T00:49:10.805 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44849.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.44849.log.gz 2024-03-24T00:49:10.805 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42091.log 2024-03-24T00:49:10.806 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44116.log 2024-03-24T00:49:10.808 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34940.log 2024-03-24T00:49:10.808 INFO:teuthology.orchestra.run.smithi040.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42091.log.gz 2024-03-24T00:49:10.809 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47484.log 2024-03-24T00:49:10.810 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44116.log.gz 2024-03-24T00:49:10.810 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49996.log 2024-03-24T00:49:10.811 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34940.log.gz 2024-03-24T00:49:10.811 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47484.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47484.log.gz 2024-03-24T00:49:10.812 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55982.log 2024-03-24T00:49:10.812 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49996.log.gz 2024-03-24T00:49:10.813 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48315.log 2024-03-24T00:49:10.814 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36367.log 2024-03-24T00:49:10.814 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55982.log.gz 2024-03-24T00:49:10.815 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41031.log 2024-03-24T00:49:10.816 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48315.log.gz 2024-03-24T00:49:10.817 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33551.log 2024-03-24T00:49:10.817 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36367.log.gz 2024-03-24T00:49:10.818 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42195.log 2024-03-24T00:49:10.819 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41031.log.gz 2024-03-24T00:49:10.819 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33551.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33551.log.gz 2024-03-24T00:49:10.820 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52998.log 2024-03-24T00:49:10.821 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose 91.3% -- /var/log/ceph/ceph-client.admin.47711.log -- replaced with /var/log/ceph/ceph-client.admin.42195.log.gz 2024-03-24T00:49:10.821 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-03-24T00:49:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27509.log 2024-03-24T00:49:10.823 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52998.log.gz 2024-03-24T00:49:10.823 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47711.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47711.log.gz 2024-03-24T00:49:10.824 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37180.log 2024-03-24T00:49:10.825 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.27509.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.27509.log.gz 2024-03-24T00:49:10.825 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50695.log 2024-03-24T00:49:10.827 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43631.log 2024-03-24T00:49:10.827 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37180.log.gz 2024-03-24T00:49:10.828 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38103.log 2024-03-24T00:49:10.828 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50695.log.gz 2024-03-24T00:49:10.829 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44659.log 2024-03-24T00:49:10.829 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43631.log.gz 2024-03-24T00:49:10.830 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38103.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38103.log.gz 2024-03-24T00:49:10.832 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44659.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44659.log.gz 2024-03-24T00:49:10.832 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-24T00:49:10.833 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47841.log 2024-03-24T00:49:10.834 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56091.log 2024-03-24T00:49:10.835 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47841.log: 88.5% 93.2% -- replaced with /var/log/ceph/ceph-client.0.44485.log.gz 2024-03-24T00:49:10.836 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47841.log.gz 2024-03-24T00:49:10.836 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.34450.log 2024-03-24T00:49:10.837 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56091.log.gz 2024-03-24T00:49:10.837 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35044.log 2024-03-24T00:49:10.838 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.2.34450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36849.log 2024-03-24T00:49:10.839 INFO:teuthology.orchestra.run.smithi040.stderr: 91.1%/var/log/ceph/ceph-client.admin.35044.log: -- replaced with /var/log/ceph/ceph-client.2.34450.log.gz 2024-03-24T00:49:10.840 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45916.log 2024-03-24T00:49:10.840 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35044.log.gz 2024-03-24T00:49:10.841 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36849.log: gzip 88.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.36849.log.gz -- 2024-03-24T00:49:10.841 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.33332.log 2024-03-24T00:49:10.842 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43437.log 2024-03-24T00:49:10.842 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45916.log.gz 2024-03-24T00:49:10.843 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37808.log 2024-03-24T00:49:10.844 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33332.log.gz 2024-03-24T00:49:10.844 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42013.log 2024-03-24T00:49:10.845 INFO:teuthology.orchestra.run.smithi040.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43437.log.gz 2024-03-24T00:49:10.846 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37808.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.37808.log.gz 2024-03-24T00:49:10.846 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34357.log 2024-03-24T00:49:10.847 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41109.log 2024-03-24T00:49:10.847 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42013.log.gz 2024-03-24T00:49:10.848 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32979.log 2024-03-24T00:49:10.849 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34357.log.gz 2024-03-24T00:49:10.849 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54466.log 2024-03-24T00:49:10.850 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41109.log.gz 2024-03-24T00:49:10.851 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41649.log 2024-03-24T00:49:10.851 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32979.log.gz 2024-03-24T00:49:10.852 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46208.log 2024-03-24T00:49:10.853 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41649.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.54466.log.gz 2024-03-24T00:49:10.853 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51351.log 2024-03-24T00:49:10.854 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41649.log.gz 2024-03-24T00:49:10.855 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50616.log 2024-03-24T00:49:10.855 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46208.log.gz 2024-03-24T00:49:10.856 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50512.log 2024-03-24T00:49:10.856 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51351.log.gz 2024-03-24T00:49:10.857 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50616.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50616.log.gz 2024-03-24T00:49:10.857 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43738.log 2024-03-24T00:49:10.858 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43579.log 2024-03-24T00:49:10.858 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50512.log.gz 2024-03-24T00:49:10.859 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44380.log 2024-03-24T00:49:10.860 INFO:teuthology.orchestra.run.smithi040.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43738.log.gz 2024-03-24T00:49:10.861 INFO:teuthology.orchestra.run.smithi053.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-24T00:49:10.861 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43579.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43579.log.gz 2024-03-24T00:49:10.861 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35397.log 2024-03-24T00:49:10.862 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40532.log 2024-03-24T00:49:10.863 INFO:teuthology.orchestra.run.smithi040.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.44380.log.gz 2024-03-24T00:49:10.863 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48077.log 2024-03-24T00:49:10.864 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35397.log.gz 2024-03-24T00:49:10.864 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40532.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.40532.log.gz 2024-03-24T00:49:10.865 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50048.log 2024-03-24T00:49:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48077.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48077.log.gz 2024-03-24T00:49:10.866 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33191.log 2024-03-24T00:49:10.867 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50048.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50048.log.gz 2024-03-24T00:49:10.867 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36797.log 2024-03-24T00:49:10.868 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.33191.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37232.log 2024-03-24T00:49:10.869 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33191.log.gz 2024-03-24T00:49:10.869 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45087.log 2024-03-24T00:49:10.869 INFO:teuthology.orchestra.run.smithi040.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36797.log.gz 2024-03-24T00:49:10.870 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37232.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37232.log.gz 2024-03-24T00:49:10.871 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47815.log 2024-03-24T00:49:10.872 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37858.log 2024-03-24T00:49:10.872 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45087.log.gz 2024-03-24T00:49:10.873 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53311.log 2024-03-24T00:49:10.874 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47815.log.gz 2024-03-24T00:49:10.874 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37858.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.37858.log.gz 2024-03-24T00:49:10.875 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34077.log 2024-03-24T00:49:10.876 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53311.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53311.log.gz 2024-03-24T00:49:10.876 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47432.log 2024-03-24T00:49:10.877 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50877.log 2024-03-24T00:49:10.877 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34077.log.gz 2024-03-24T00:49:10.878 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39034.log 2024-03-24T00:49:10.878 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47432.log.gz 2024-03-24T00:49:10.879 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38565.log 2024-03-24T00:49:10.880 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50877.log.gz 2024-03-24T00:49:10.880 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39034.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39034.log.gz 2024-03-24T00:49:10.881 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41563.log 2024-03-24T00:49:10.882 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34310.log 2024-03-24T00:49:10.882 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38565.log.gz 2024-03-24T00:49:10.883 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39495.log 2024-03-24T00:49:10.883 INFO:teuthology.orchestra.run.smithi040.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41563.log.gz 2024-03-24T00:49:10.885 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.34310.log: 90.5% -- replaced with /var/log/ceph/ceph-client.0.34310.log.gz 2024-03-24T00:49:10.885 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.39495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50564.log 2024-03-24T00:49:10.886 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45575.log 2024-03-24T00:49:10.887 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50564.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50564.log.gz 2024-03-24T00:49:10.888 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38591.log 2024-03-24T00:49:10.889 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45575.log.gz 2024-03-24T00:49:10.889 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51506.log 2024-03-24T00:49:10.891 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38591.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38591.log.gz 2024-03-24T00:49:10.891 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35262.log 2024-03-24T00:49:10.892 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51506.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51506.log.gz 2024-03-24T00:49:10.893 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53773.log 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35262.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35262.log.gz 2024-03-24T00:49:10.895 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51143.log 2024-03-24T00:49:10.896 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.53773.log.gz 2024-03-24T00:49:10.897 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33297.log 2024-03-24T00:49:10.898 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51143.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51143.log.gz 2024-03-24T00:49:10.899 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33540.log 2024-03-24T00:49:10.900 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33297.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33297.log.gz 2024-03-24T00:49:10.900 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33765.log 2024-03-24T00:49:10.901 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33540.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33540.log.gz 2024-03-24T00:49:10.902 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45653.log 2024-03-24T00:49:10.903 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33765.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33765.log.gz 2024-03-24T00:49:10.904 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38669.log 2024-03-24T00:49:10.905 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45653.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45653.log.gz 2024-03-24T00:49:10.906 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38449.log 2024-03-24T00:49:10.906 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38669.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38669.log.gz 2024-03-24T00:49:10.907 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43252.log 2024-03-24T00:49:10.909 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38449.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38449.log.gz 2024-03-24T00:49:10.909 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33351.log 2024-03-24T00:49:10.910 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43252.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43252.log.gz 2024-03-24T00:49:10.911 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40918.log 2024-03-24T00:49:10.912 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33351.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33351.log.gz 2024-03-24T00:49:10.913 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34706.log 2024-03-24T00:49:10.914 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40918.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40918.log.gz 2024-03-24T00:49:10.915 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53617.log 2024-03-24T00:49:10.916 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34706.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34706.log.gz 2024-03-24T00:49:10.916 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43113.log 2024-03-24T00:49:10.918 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53617.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53617.log.gz 2024-03-24T00:49:10.918 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53103.log 2024-03-24T00:49:10.919 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43113.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43113.log.gz 2024-03-24T00:49:10.920 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34518.log 2024-03-24T00:49:10.921 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53103.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53103.log.gz 2024-03-24T00:49:10.922 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41135.log 2024-03-24T00:49:10.923 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34518.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34518.log.gz 2024-03-24T00:49:10.923 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50957.log 2024-03-24T00:49:10.925 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41135.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41135.log.gz 2024-03-24T00:49:10.925 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35763.log 2024-03-24T00:49:10.926 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50957.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50957.log.gz 2024-03-24T00:49:10.927 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33999.log 2024-03-24T00:49:10.928 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35763.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35763.log.gz 2024-03-24T00:49:10.929 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33219.log 2024-03-24T00:49:10.930 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33999.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33999.log.gz 2024-03-24T00:49:10.930 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55641.log 2024-03-24T00:49:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33219.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33219.log.gz 2024-03-24T00:49:10.932 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34862.log 2024-03-24T00:49:10.933 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55641.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55641.log.gz 2024-03-24T00:49:10.934 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47971.log 2024-03-24T00:49:10.935 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34862.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34862.log.gz 2024-03-24T00:49:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42795.log 2024-03-24T00:49:10.937 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47971.log.gz 2024-03-24T00:49:10.937 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47874.log 2024-03-24T00:49:10.939 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42795.log: 90.8%/var/log/ceph/ceph-client.admin.47874.log: -- replaced with /var/log/ceph/ceph-client.admin.42795.log.gz 2024-03-24T00:49:10.940 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36237.log 2024-03-24T00:49:10.940 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47874.log.gz 2024-03-24T00:49:10.941 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52946.log 2024-03-24T00:49:10.942 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36237.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36237.log.gz 2024-03-24T00:49:10.943 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52946.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52946.log.gz 2024-03-24T00:49:10.944 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37703.log 2024-03-24T00:49:10.945 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46469.log 2024-03-24T00:49:10.947 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37703.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37703.log.gz 2024-03-24T00:49:10.950 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.46469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41801.log 2024-03-24T00:49:10.953 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50669.log 2024-03-24T00:49:10.953 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35129.log 2024-03-24T00:49:10.953 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41801.log.gz 2024-03-24T00:49:10.954 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40384.log 2024-03-24T00:49:10.954 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50669.log.gz 2024-03-24T00:49:10.956 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35129.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35129.log.gz 2024-03-24T00:49:10.957 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52667.log 2024-03-24T00:49:10.959 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35475.log 2024-03-24T00:49:10.959 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52667.log: /var/log/ceph/ceph-client.admin.40384.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52667.log.gz 2024-03-24T00:49:10.960 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40384.log.gz 2024-03-24T00:49:10.963 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43035.log 2024-03-24T00:49:10.964 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35475.log.gz 2024-03-24T00:49:10.967 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43035.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43035.log.gz 2024-03-24T00:49:10.967 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41853.log 2024-03-24T00:49:10.970 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41853.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41853.log.gz 2024-03-24T00:49:10.970 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50160.log 2024-03-24T00:49:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50160.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50160.log.gz 2024-03-24T00:49:10.974 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47763.log 2024-03-24T00:49:10.981 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47763.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47763.log.gz 2024-03-24T00:49:10.981 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50786.log 2024-03-24T00:49:10.984 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52875.log 2024-03-24T00:49:10.984 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50786.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50786.log.gz 2024-03-24T00:49:10.985 INFO:teuthology.orchestra.run.smithi040.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.0.37363.log.gz 2024-03-24T00:49:10.985 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46364.log 2024-03-24T00:49:10.987 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52875.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52875.log.gz 2024-03-24T00:49:10.987 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.46364.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50754.log 2024-03-24T00:49:10.989 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.46364.log.gz 2024-03-24T00:49:10.989 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-24T00:49:10.990 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50754.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50754.log.gz 2024-03-24T00:49:10.991 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53561.log 2024-03-24T00:49:10.996 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.53561.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53561.log.gz 2024-03-24T00:49:10.996 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35977.log 2024-03-24T00:49:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35977.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35977.log.gz 2024-03-24T00:49:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32674.log 2024-03-24T00:49:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32674.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32674.log.gz 2024-03-24T00:49:11.001 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56039.log 2024-03-24T00:49:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56039.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56039.log.gz 2024-03-24T00:49:11.004 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47536.log 2024-03-24T00:49:11.007 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47536.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47536.log.gz 2024-03-24T00:49:11.007 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33340.log 2024-03-24T00:49:11.010 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33340.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33340.log.gz 2024-03-24T00:49:11.010 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50459.log 2024-03-24T00:49:11.013 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50459.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50459.log.gz 2024-03-24T00:49:11.014 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50108.log 2024-03-24T00:49:11.018 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39286.log 2024-03-24T00:49:11.018 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50108.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.50108.log.gz 2024-03-24T00:49:11.018 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48471.log 2024-03-24T00:49:11.019 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39286.log.gz 2024-03-24T00:49:11.021 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48471.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48471.log.gz 2024-03-24T00:49:11.022 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36107.log 2024-03-24T00:49:11.024 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37758.log 2024-03-24T00:49:11.025 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36107.log.gz 2024-03-24T00:49:11.027 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52693.log 2024-03-24T00:49:11.030 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37758.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.37758.log.gz 2024-03-24T00:49:11.032 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.52693.log: /var/log/ceph/ceph-client.admin.44694.log 2024-03-24T00:49:11.033 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52693.log.gz 2024-03-24T00:49:11.037 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.44694.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44694.log.gz 2024-03-24T00:49:11.037 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.55746.log 2024-03-24T00:49:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55746.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55746.log.gz 2024-03-24T00:49:11.040 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49970.log 2024-03-24T00:49:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49970.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49970.log.gz 2024-03-24T00:49:11.043 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52503.log 2024-03-24T00:49:11.045 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44799.log 2024-03-24T00:49:11.046 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52503.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52503.log.gz 2024-03-24T00:49:11.049 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44799.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.44799.log.gz 2024-03-24T00:49:11.049 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41935.log 2024-03-24T00:49:11.054 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53509.log 2024-03-24T00:49:11.054 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41935.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41935.log.gz 2024-03-24T00:49:11.054 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47250.log 2024-03-24T00:49:11.055 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53509.log.gz 2024-03-24T00:49:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47250.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47250.log.gz 2024-03-24T00:49:11.058 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36745.log 2024-03-24T00:49:11.060 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36745.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36745.log.gz 2024-03-24T00:49:11.061 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51299.log 2024-03-24T00:49:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51299.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.51299.log.gz 2024-03-24T00:49:11.064 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33271.log 2024-03-24T00:49:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33271.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33271.log.gz 2024-03-24T00:49:11.067 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38371.log 2024-03-24T00:49:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44559.log 2024-03-24T00:49:11.070 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38371.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38371.log.gz 2024-03-24T00:49:11.072 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44559.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44559.log.gz 2024-03-24T00:49:11.073 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55528.log 2024-03-24T00:49:11.076 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55528.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55528.log.gz 2024-03-24T00:49:11.076 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35685.log 2024-03-24T00:49:11.079 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35685.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35685.log.gz 2024-03-24T00:49:11.079 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43463.log 2024-03-24T00:49:11.082 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39062.log 2024-03-24T00:49:11.082 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43463.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43463.log.gz 2024-03-24T00:49:11.085 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39062.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39062.log.gz 2024-03-24T00:49:11.085 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53444.log 2024-03-24T00:49:11.088 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53444.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53444.log.gz 2024-03-24T00:49:11.088 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43330.log 2024-03-24T00:49:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43330.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43330.log.gz 2024-03-24T00:49:11.091 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33337.log 2024-03-24T00:49:11.092 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51406.log 2024-03-24T00:49:11.094 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33337.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33337.log.gz 2024-03-24T00:49:11.098 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.51406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42348.log 2024-03-24T00:49:11.099 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41187.log 2024-03-24T00:49:11.101 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42348.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42348.log.gz 2024-03-24T00:49:11.104 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41187.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41187.log.gz 2024-03-24T00:49:11.105 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45523.log 2024-03-24T00:49:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45523.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45523.log.gz 2024-03-24T00:49:11.112 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46260.log 2024-03-24T00:49:11.115 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44430.log 2024-03-24T00:49:11.116 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46260.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46260.log.gz 2024-03-24T00:49:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44430.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44430.log.gz 2024-03-24T00:49:11.120 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36875.log 2024-03-24T00:49:11.123 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55589.log 2024-03-24T00:49:11.126 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36875.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36875.log.gz 2024-03-24T00:49:11.126 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32835.log 2024-03-24T00:49:11.126 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45731.log 2024-03-24T00:49:11.127 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55589.log.gz 2024-03-24T00:49:11.128 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32835.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32835.log.gz 2024-03-24T00:49:11.132 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45731.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45731.log.gz 2024-03-24T00:49:11.132 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47906.log 2024-03-24T00:49:11.134 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34654.log 2024-03-24T00:49:11.136 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52225.log 2024-03-24T00:49:11.136 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47906.log.gz 2024-03-24T00:49:11.137 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34654.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34654.log.gz 2024-03-24T00:49:11.139 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39863.log 2024-03-24T00:49:11.141 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52225.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.52225.log.gz 2024-03-24T00:49:11.144 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39863.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39863.log.gz 2024-03-24T00:49:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42300.log 2024-03-24T00:49:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50239.log 2024-03-24T00:49:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.42300.log: /var/log/ceph/ceph-client.admin.50239.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50239.log.gz 2024-03-24T00:49:11.150 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46048.log 2024-03-24T00:49:11.159 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43225.log 2024-03-24T00:49:11.160 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46048.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46048.log.gz 2024-03-24T00:49:11.163 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53024.log 2024-03-24T00:49:11.164 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43225.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43225.log.gz 2024-03-24T00:49:11.165 INFO:teuthology.orchestra.run.smithi053.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-24T00:49:11.167 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38539.log 2024-03-24T00:49:11.168 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53024.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53024.log.gz 2024-03-24T00:49:11.171 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38155.log 2024-03-24T00:49:11.172 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38539.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38539.log.gz 2024-03-24T00:49:11.175 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40334.log 2024-03-24T00:49:11.176 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38155.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38155.log.gz 2024-03-24T00:49:11.179 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32863.log 2024-03-24T00:49:11.181 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40334.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.40334.log.gz 2024-03-24T00:49:11.183 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-24T00:49:11.184 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32863.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32863.log.gz 2024-03-24T00:49:11.187 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40684.log 2024-03-24T00:49:11.191 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53952.log 2024-03-24T00:49:11.192 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40684.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40684.log.gz 2024-03-24T00:49:11.196 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53952.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53952.log.gz 2024-03-24T00:49:11.198 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50213.log 2024-03-24T00:49:11.201 INFO:teuthology.orchestra.run.smithi053.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-24T00:49:11.203 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51454.log 2024-03-24T00:49:11.204 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50213.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50213.log.gz 2024-03-24T00:49:11.207 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48600.log 2024-03-24T00:49:11.208 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51454.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51454.log.gz 2024-03-24T00:49:11.211 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44250.log 2024-03-24T00:49:11.212 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48600.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48600.log.gz 2024-03-24T00:49:11.215 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44224.log 2024-03-24T00:49:11.216 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44250.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44250.log.gz 2024-03-24T00:49:11.219 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44723.log 2024-03-24T00:49:11.220 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44224.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44224.log.gz 2024-03-24T00:49:11.222 INFO:teuthology.orchestra.run.smithi040.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.0.46469.log.gz 2024-03-24T00:49:11.223 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38721.log 2024-03-24T00:49:11.225 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44723.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44723.log.gz 2024-03-24T00:49:11.226 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40234.log 2024-03-24T00:49:11.227 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38721.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38721.log.gz 2024-03-24T00:49:11.230 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.40234.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50590.log 2024-03-24T00:49:11.232 INFO:teuthology.orchestra.run.smithi040.stderr: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35790.log 2024-03-24T00:49:11.233 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50590.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50590.log.gz 2024-03-24T00:49:11.233 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40234.log.gz 2024-03-24T00:49:11.235 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47328.log 2024-03-24T00:49:11.241 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35790.log: /var/log/ceph/ceph-client.admin.47328.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47328.log.gz 2024-03-24T00:49:11.242 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36393.log 2024-03-24T00:49:11.242 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35790.log.gz 2024-03-24T00:49:11.242 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46850.log 2024-03-24T00:49:11.242 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36393.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36393.log.gz 2024-03-24T00:49:11.243 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47581.log 2024-03-24T00:49:11.247 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46850.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.46850.log.gz 2024-03-24T00:49:11.247 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53695.log 2024-03-24T00:49:11.247 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47581.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47581.log.gz 2024-03-24T00:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53695.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53695.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36029.log 2024-03-24T00:49:11.250 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-03-24T00:49:11.251 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50930.log 2024-03-24T00:49:11.253 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35659.log 2024-03-24T00:49:11.254 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50930.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50930.log.gz 2024-03-24T00:49:11.255 INFO:teuthology.orchestra.run.smithi040.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36029.log.gz 2024-03-24T00:49:11.258 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35659.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55687.log 2024-03-24T00:49:11.259 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35659.log.gz 2024-03-24T00:49:11.259 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55832.log 2024-03-24T00:49:11.261 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45173.log 2024-03-24T00:49:11.262 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55687.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55687.log.gz 2024-03-24T00:49:11.265 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45173.log: /var/log/ceph/ceph-client.admin.55832.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45173.log.gz 2024-03-24T00:49:11.265 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55832.log.gz 2024-03-24T00:49:11.265 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37128.log 2024-03-24T00:49:11.270 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37128.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37128.log.gz 2024-03-24T00:49:11.270 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48133.log 2024-03-24T00:49:11.275 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47789.log 2024-03-24T00:49:11.276 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48133.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48133.log.gz 2024-03-24T00:49:11.279 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33539.log 2024-03-24T00:49:11.280 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47789.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47789.log.gz 2024-03-24T00:49:11.282 INFO:teuthology.orchestra.run.smithi040.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.0.39495.log.gz 2024-03-24T00:49:11.283 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49701.log 2024-03-24T00:49:11.284 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33539.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33539.log.gz 2024-03-24T00:49:11.285 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49701.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49701.log.gz 2024-03-24T00:49:11.286 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50485.log 2024-03-24T00:49:11.288 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39338.log 2024-03-24T00:49:11.291 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39338.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39338.log.gz 2024-03-24T00:49:11.292 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36289.log 2024-03-24T00:49:11.292 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50485.log.gz 2024-03-24T00:49:11.294 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45329.log 2024-03-24T00:49:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45329.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45329.log.gz 2024-03-24T00:49:11.298 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36289.log.gz 2024-03-24T00:49:11.298 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53390.log 2024-03-24T00:49:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53390.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53390.log.gz 2024-03-24T00:49:11.301 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36471.log 2024-03-24T00:49:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36471.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36471.log.gz 2024-03-24T00:49:11.304 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48341.log 2024-03-24T00:49:11.306 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45139.log 2024-03-24T00:49:11.307 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48341.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48341.log.gz 2024-03-24T00:49:11.310 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35096.log 2024-03-24T00:49:11.311 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45139.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45139.log.gz 2024-03-24T00:49:11.314 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35096.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35096.log.gz 2024-03-24T00:49:11.314 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46020.log 2024-03-24T00:49:11.316 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46020.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46020.log.gz 2024-03-24T00:49:11.317 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55742.log 2024-03-24T00:49:11.318 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33087.log 2024-03-24T00:49:11.320 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55742.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55742.log.gz 2024-03-24T00:49:11.323 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33087.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33087.log.gz 2024-03-24T00:49:11.323 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46900.log 2024-03-24T00:49:11.326 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46900.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.46900.log.gz 2024-03-24T00:49:11.327 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43710.log 2024-03-24T00:49:11.332 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43710.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43710.log.gz 2024-03-24T00:49:11.333 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36211.log 2024-03-24T00:49:11.333 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36211.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36211.log.gz 2024-03-24T00:49:11.333 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36133.log 2024-03-24T00:49:11.335 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36133.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36133.log.gz 2024-03-24T00:49:11.335 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36159.log 2024-03-24T00:49:11.342 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53077.log 2024-03-24T00:49:11.342 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36159.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36159.log.gz 2024-03-24T00:49:11.344 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53077.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53077.log.gz 2024-03-24T00:49:11.346 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47685.log 2024-03-24T00:49:11.351 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46076.log 2024-03-24T00:49:11.352 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47685.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47685.log.gz 2024-03-24T00:49:11.355 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45889.log 2024-03-24T00:49:11.356 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46076.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46076.log.gz 2024-03-24T00:49:11.359 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38397.log 2024-03-24T00:49:11.360 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45889.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45889.log.gz 2024-03-24T00:49:11.363 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33113.log 2024-03-24T00:49:11.364 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38397.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38397.log.gz 2024-03-24T00:49:11.367 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33791.log 2024-03-24T00:49:11.368 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33113.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33113.log.gz 2024-03-24T00:49:11.371 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43790.log 2024-03-24T00:49:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33791.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33791.log.gz 2024-03-24T00:49:11.376 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43790.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43790.log.gz 2024-03-24T00:49:11.378 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33869.log 2024-03-24T00:49:11.383 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43061.log 2024-03-24T00:49:11.384 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33869.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33869.log.gz 2024-03-24T00:49:11.387 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45968.log 2024-03-24T00:49:11.388 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43061.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43061.log.gz 2024-03-24T00:49:11.389 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53926.log 2024-03-24T00:49:11.391 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45968.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45968.log.gz 2024-03-24T00:49:11.394 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44025.log 2024-03-24T00:49:11.396 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53926.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53926.log.gz 2024-03-24T00:49:11.398 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33548.log 2024-03-24T00:49:11.400 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44025.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44025.log.gz 2024-03-24T00:49:11.403 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33245.log 2024-03-24T00:49:11.404 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33548.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33548.log.gz 2024-03-24T00:49:11.406 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44533.log 2024-03-24T00:49:11.410 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33245.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33245.log.gz 2024-03-24T00:49:11.410 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44533.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44533.log.gz 2024-03-24T00:49:11.410 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53050.log 2024-03-24T00:49:11.415 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48574.log 2024-03-24T00:49:11.416 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53050.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53050.log.gz 2024-03-24T00:49:11.419 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54642.log 2024-03-24T00:49:11.420 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48574.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48574.log.gz 2024-03-24T00:49:11.421 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-24T00:49:11.422 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34732.log 2024-03-24T00:49:11.423 INFO:teuthology.orchestra.run.smithi040.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54642.log.gz 2024-03-24T00:49:11.427 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40449.log 2024-03-24T00:49:11.428 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34732.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34732.log.gz 2024-03-24T00:49:11.430 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33530.log 2024-03-24T00:49:11.435 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40449.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40449.log.gz 2024-03-24T00:49:11.435 INFO:teuthology.orchestra.run.smithi040.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-24T00:49:11.436 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33530.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33530.log.gz 2024-03-24T00:49:11.438 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36929.log 2024-03-24T00:49:11.443 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47458.log 2024-03-24T00:49:11.444 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36929.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36929.log.gz 2024-03-24T00:49:11.447 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36692.log 2024-03-24T00:49:11.448 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47458.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47458.log.gz 2024-03-24T00:49:11.451 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38695.log 2024-03-24T00:49:11.452 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36692.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36692.log.gz 2024-03-24T00:49:11.455 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41265.log 2024-03-24T00:49:11.456 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38695.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38695.log.gz 2024-03-24T00:49:11.459 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42117.log 2024-03-24T00:49:11.460 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41265.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41265.log.gz 2024-03-24T00:49:11.463 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35579.log 2024-03-24T00:49:11.464 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42117.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42117.log.gz 2024-03-24T00:49:11.467 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47276.log 2024-03-24T00:49:11.468 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35579.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35579.log.gz 2024-03-24T00:49:11.471 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50317.log 2024-03-24T00:49:11.472 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47276.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47276.log.gz 2024-03-24T00:49:11.476 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50317.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50317.log.gz 2024-03-24T00:49:11.478 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34602.log 2024-03-24T00:49:11.483 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43658.log 2024-03-24T00:49:11.484 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34602.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34602.log.gz 2024-03-24T00:49:11.487 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34758.log 2024-03-24T00:49:11.488 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43658.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43658.log.gz 2024-03-24T00:49:11.490 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35423.log 2024-03-24T00:49:11.492 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34758.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.34758.log.gz 2024-03-24T00:49:11.495 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39090.log 2024-03-24T00:49:11.498 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35423.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35423.log.gz 2024-03-24T00:49:11.498 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51221.log 2024-03-24T00:49:11.500 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39090.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39090.log.gz 2024-03-24T00:49:11.503 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38319.log 2024-03-24T00:49:11.504 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51221.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51221.log.gz 2024-03-24T00:49:11.507 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40506.log 2024-03-24T00:49:11.508 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38319.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38319.log.gz 2024-03-24T00:49:11.512 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40506.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40506.log.gz 2024-03-24T00:49:11.514 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46517.log 2024-03-24T00:49:11.518 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46414.log 2024-03-24T00:49:11.520 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46517.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46517.log.gz 2024-03-24T00:49:11.522 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39941.log 2024-03-24T00:49:11.524 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46414.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46414.log.gz 2024-03-24T00:49:11.526 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40760.log 2024-03-24T00:49:11.528 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39941.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.39941.log.gz 2024-03-24T00:49:11.530 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36771.log 2024-03-24T00:49:11.532 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40760.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40760.log.gz 2024-03-24T00:49:11.534 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41775.log 2024-03-24T00:49:11.536 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36771.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36771.log.gz 2024-03-24T00:49:11.538 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33035.log 2024-03-24T00:49:11.540 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41775.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41775.log.gz 2024-03-24T00:49:11.543 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51169.log 2024-03-24T00:49:11.544 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33035.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33035.log.gz 2024-03-24T00:49:11.548 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51169.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51169.log.gz 2024-03-24T00:49:11.550 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41749.log 2024-03-24T00:49:11.558 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43304.log 2024-03-24T00:49:11.558 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41749.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41749.log.gz 2024-03-24T00:49:11.559 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48626.log 2024-03-24T00:49:11.560 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43304.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43304.log.gz 2024-03-24T00:49:11.563 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34966.log 2024-03-24T00:49:11.564 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48626.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48626.log.gz 2024-03-24T00:49:11.567 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37308.log 2024-03-24T00:49:11.568 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34966.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34966.log.gz 2024-03-24T00:49:11.570 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51039.log 2024-03-24T00:49:11.572 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37308.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37308.log.gz 2024-03-24T00:49:11.575 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36666.log 2024-03-24T00:49:11.576 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51039.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51039.log.gz 2024-03-24T00:49:11.579 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54768.log 2024-03-24T00:49:11.580 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36666.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36666.log.gz 2024-03-24T00:49:11.583 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47164.log 2024-03-24T00:49:11.585 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54768.log: 93.2% -- replaced with /var/log/ceph/ceph-client.0.42300.log.gz 2024-03-24T00:49:11.585 INFO:teuthology.orchestra.run.smithi040.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54768.log.gz 2024-03-24T00:49:11.588 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47164.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47164.log.gz 2024-03-24T00:49:11.588 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54616.log 2024-03-24T00:49:11.590 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54566.log 2024-03-24T00:49:11.593 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54566.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.54566.log.gz 2024-03-24T00:49:11.595 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54616.log.gz 2024-03-24T00:49:11.595 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33344.log 2024-03-24T00:49:11.598 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33344.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33344.log.gz 2024-03-24T00:49:11.598 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43946.log 2024-03-24T00:49:11.604 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43946.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43946.log.gz 2024-03-24T00:49:11.604 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51091.log 2024-03-24T00:49:11.604 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51091.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51091.log.gz 2024-03-24T00:49:11.604 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35235.log 2024-03-24T00:49:11.607 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35235.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35235.log.gz 2024-03-24T00:49:11.607 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48527.log 2024-03-24T00:49:11.614 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.48527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42065.log 2024-03-24T00:49:11.616 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56166.log 2024-03-24T00:49:11.617 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42065.log.gz 2024-03-24T00:49:11.619 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39889.log 2024-03-24T00:49:11.624 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56166.log: /var/log/ceph/ceph-client.admin.39889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38077.log 2024-03-24T00:49:11.625 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39889.log.gz 2024-03-24T00:49:11.629 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.38077.log: 88.2% 95.8% -- replaced with /var/log/ceph/ceph-client.admin.56166.log.gz 2024-03-24T00:49:11.630 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52823.log 2024-03-24T00:49:11.631 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41616.log 2024-03-24T00:49:11.633 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52823.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52823.log.gz 2024-03-24T00:49:11.634 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38077.log.gz 2024-03-24T00:49:11.636 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41616.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41616.log.gz 2024-03-24T00:49:11.636 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45757.log 2024-03-24T00:49:11.639 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44749.log 2024-03-24T00:49:11.640 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45757.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45757.log.gz 2024-03-24T00:49:11.641 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50721.log 2024-03-24T00:49:11.644 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44749.log: /var/log/ceph/ceph-client.admin.50721.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50721.log.gz 2024-03-24T00:49:11.645 INFO:teuthology.orchestra.run.smithi040.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.44749.log.gz 2024-03-24T00:49:11.645 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45705.log 2024-03-24T00:49:11.652 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45705.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45705.log.gz 2024-03-24T00:49:11.653 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32953.log 2024-03-24T00:49:11.654 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43139.log 2024-03-24T00:49:11.655 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41987.log 2024-03-24T00:49:11.655 INFO:teuthology.orchestra.run.smithi040.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32953.log.gz 2024-03-24T00:49:11.659 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43139.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43139.log.gz 2024-03-24T00:49:11.660 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55792.log 2024-03-24T00:49:11.660 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41987.log.gz 2024-03-24T00:49:11.663 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55792.log: gzip -5 --verbose -- 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55792.log.gz 2024-03-24T00:49:11.663 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.45549.log 2024-03-24T00:49:11.668 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.45549.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45549.log.gz 2024-03-24T00:49:11.670 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.55880.log 2024-03-24T00:49:11.675 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36640.log 2024-03-24T00:49:11.676 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55880.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55880.log.gz 2024-03-24T00:49:11.680 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36640.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36640.log.gz 2024-03-24T00:49:11.682 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34836.log 2024-03-24T00:49:11.687 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53721.log 2024-03-24T00:49:11.688 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34836.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34836.log.gz 2024-03-24T00:49:11.691 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44053.log 2024-03-24T00:49:11.692 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53721.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53721.log.gz 2024-03-24T00:49:11.693 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55588.log 2024-03-24T00:49:11.695 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44053.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.44053.log.gz -5 2024-03-24T00:49:11.695 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46312.log 2024-03-24T00:49:11.696 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55588.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55588.log.gz 2024-03-24T00:49:11.699 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36992.log 2024-03-24T00:49:11.700 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46312.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46312.log.gz 2024-03-24T00:49:11.703 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46130.log 2024-03-24T00:49:11.704 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36992.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36992.log.gz 2024-03-24T00:49:11.708 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46130.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46130.log.gz 2024-03-24T00:49:11.710 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48263.log 2024-03-24T00:49:11.715 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38929.log 2024-03-24T00:49:11.716 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48263.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48263.log.gz 2024-03-24T00:49:11.719 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40971.log 2024-03-24T00:49:11.720 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38929.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38929.log.gz 2024-03-24T00:49:11.723 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35155.log 2024-03-24T00:49:11.724 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40971.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40971.log.gz 2024-03-24T00:49:11.725 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41459.log 2024-03-24T00:49:11.727 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35155.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35155.log.gz 2024-03-24T00:49:11.731 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33973.log 2024-03-24T00:49:11.732 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41459.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41459.log.gz 2024-03-24T00:49:11.735 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48393.log 2024-03-24T00:49:11.736 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33973.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.33973.log.gz 2024-03-24T00:49:11.738 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55828.log 2024-03-24T00:49:11.740 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48393.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48393.log.gz 2024-03-24T00:49:11.744 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55828.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55828.log.gz 2024-03-24T00:49:11.746 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52175.log 2024-03-24T00:49:11.751 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38051.log 2024-03-24T00:49:11.752 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52175.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.52175.log.gz 2024-03-24T00:49:11.755 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34992.log 2024-03-24T00:49:11.756 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38051.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.38051.log.gz 2024-03-24T00:49:11.758 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50985.log 2024-03-24T00:49:11.760 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34992.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.34992.log.gz 2024-03-24T00:49:11.763 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34129.log 2024-03-24T00:49:11.764 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50985.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50985.log.gz 2024-03-24T00:49:11.767 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47302.log 2024-03-24T00:49:11.769 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34129.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.34129.log.gz 2024-03-24T00:49:11.771 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52797.log 2024-03-24T00:49:11.772 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47302.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47302.log.gz 2024-03-24T00:49:11.775 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50395.log 2024-03-24T00:49:11.776 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52797.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52797.log.gz 2024-03-24T00:49:11.780 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50395.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50395.log.gz 2024-03-24T00:49:11.782 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40945.log 2024-03-24T00:49:11.786 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44328.log 2024-03-24T00:49:11.788 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40945.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40945.log.gz 2024-03-24T00:49:11.791 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40866.log 2024-03-24T00:49:11.792 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44328.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44328.log.gz 2024-03-24T00:49:11.795 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37076.log 2024-03-24T00:49:11.796 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40866.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40866.log.gz 2024-03-24T00:49:11.798 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35737.log 2024-03-24T00:49:11.800 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37076.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37076.log.gz 2024-03-24T00:49:11.802 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52529.log 2024-03-24T00:49:11.804 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35737.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35737.log.gz 2024-03-24T00:49:11.807 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46569.log 2024-03-24T00:49:11.808 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52529.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52529.log.gz 2024-03-24T00:49:11.811 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35925.log 2024-03-24T00:49:11.812 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46569.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46569.log.gz 2024-03-24T00:49:11.816 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35925.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35925.log.gz 2024-03-24T00:49:11.818 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52615.log 2024-03-24T00:49:11.823 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-24T00:49:11.824 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52615.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52615.log.gz 2024-03-24T00:49:11.827 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32927.log 2024-03-24T00:49:11.831 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33139.log 2024-03-24T00:49:11.835 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32927.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.32927.log.gz 2024-03-24T00:49:11.835 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53591.log 2024-03-24T00:49:11.835 INFO:teuthology.orchestra.run.smithi040.stderr: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-24T00:49:11.836 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33139.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.33139.log.gz 2024-03-24T00:49:11.840 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53591.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53591.log.gz 2024-03-24T00:49:11.842 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-03-24T00:49:11.847 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40582.log 2024-03-24T00:49:11.850 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33165.log 2024-03-24T00:49:11.853 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40582.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40582.log.gz 2024-03-24T00:49:11.855 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50642.log 2024-03-24T00:49:11.856 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33165.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33165.log.gz 2024-03-24T00:49:11.859 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41485.log 2024-03-24T00:49:11.860 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50642.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50642.log.gz 2024-03-24T00:49:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39208.log 2024-03-24T00:49:11.864 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41485.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41485.log.gz 2024-03-24T00:49:11.867 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52555.log 2024-03-24T00:49:11.868 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39208.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39208.log.gz 2024-03-24T00:49:11.870 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47997.log 2024-03-24T00:49:11.872 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52555.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52555.log.gz 2024-03-24T00:49:11.876 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47997.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47997.log.gz 2024-03-24T00:49:11.878 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-24T00:49:11.883 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35288.log 2024-03-24T00:49:11.887 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41881.log 2024-03-24T00:49:11.888 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35288.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35288.log.gz 2024-03-24T00:49:11.891 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34103.log 2024-03-24T00:49:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41881.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41881.log.gz 2024-03-24T00:49:11.895 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56117.log 2024-03-24T00:49:11.896 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34103.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34103.log.gz 2024-03-24T00:49:11.899 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44302.log 2024-03-24T00:49:11.903 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43278.log 2024-03-24T00:49:11.904 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44302.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44302.log.gz 2024-03-24T00:49:11.906 INFO:teuthology.orchestra.run.smithi040.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.admin.56117.log.gz 2024-03-24T00:49:11.907 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43553.log 2024-03-24T00:49:11.908 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43278.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43278.log.gz 2024-03-24T00:49:11.913 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43553.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43553.log.gz 2024-03-24T00:49:11.913 INFO:teuthology.orchestra.run.smithi040.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.51406.log.gz 2024-03-24T00:49:11.914 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33921.log 2024-03-24T00:49:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56013.log 2024-03-24T00:49:11.919 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45816.log 2024-03-24T00:49:11.919 INFO:teuthology.orchestra.run.smithi040.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.33921.log.gz 2024-03-24T00:49:11.920 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.56013.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56013.log.gz 2024-03-24T00:49:11.920 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53643.log 2024-03-24T00:49:11.922 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45816.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45816.log.gz 2024-03-24T00:49:11.922 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36315.log 2024-03-24T00:49:11.923 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53643.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53643.log.gz 2024-03-24T00:49:11.923 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55884.log 2024-03-24T00:49:11.925 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36315.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36315.log.gz 2024-03-24T00:49:11.926 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55936.log 2024-03-24T00:49:11.928 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55884.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55884.log.gz 2024-03-24T00:49:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46800.log 2024-03-24T00:49:11.932 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55936.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55936.log.gz 2024-03-24T00:49:11.933 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46800.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.46800.log.gz 2024-03-24T00:49:11.935 INFO:teuthology.orchestra.run.smithi053.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-24T00:49:11.938 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38773.log 2024-03-24T00:49:11.939 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37668.log 2024-03-24T00:49:11.940 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38773.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38773.log.gz 2024-03-24T00:49:11.943 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41057.log 2024-03-24T00:49:11.944 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37668.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37668.log.gz 2024-03-24T00:49:11.947 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39151.log 2024-03-24T00:49:11.948 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41057.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41057.log.gz 2024-03-24T00:49:11.951 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36823.log 2024-03-24T00:49:11.952 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39151.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39151.log.gz 2024-03-24T00:49:11.955 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34051.log 2024-03-24T00:49:11.956 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36823.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36823.log.gz 2024-03-24T00:49:11.959 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36957.log 2024-03-24T00:49:11.960 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52719.log 2024-03-24T00:49:11.960 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34051.log.gz 2024-03-24T00:49:11.961 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36957.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36957.log.gz 2024-03-24T00:49:11.964 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52719.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52719.log.gz 2024-03-24T00:49:11.966 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39569.log 2024-03-24T00:49:11.971 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48185.log 2024-03-24T00:49:11.972 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39569.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39569.log.gz 2024-03-24T00:49:11.975 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37437.log 2024-03-24T00:49:11.976 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48185.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48185.log.gz 2024-03-24T00:49:11.978 INFO:teuthology.orchestra.run.smithi053.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-24T00:49:11.979 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45459.log 2024-03-24T00:49:11.980 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37437.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37437.log.gz 2024-03-24T00:49:11.983 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46156.log 2024-03-24T00:49:11.984 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45459.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45459.log.gz 2024-03-24T00:49:11.987 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33343.log 2024-03-24T00:49:11.988 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46156.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46156.log.gz 2024-03-24T00:49:11.990 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53363.log 2024-03-24T00:49:11.992 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33343.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33343.log.gz 2024-03-24T00:49:11.994 INFO:teuthology.orchestra.run.smithi040.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-24T00:49:11.996 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53363.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53363.log.gz 2024-03-24T00:49:11.998 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34888.log 2024-03-24T00:49:12.003 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49768.log 2024-03-24T00:49:12.004 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34888.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34888.log.gz 2024-03-24T00:49:12.007 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46234.log 2024-03-24T00:49:12.009 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49768.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.49768.log.gz 2024-03-24T00:49:12.011 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50265.log 2024-03-24T00:49:12.012 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46234.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46234.log.gz 2024-03-24T00:49:12.015 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33817.log 2024-03-24T00:49:12.016 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50265.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50265.log.gz 2024-03-24T00:49:12.018 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50369.log 2024-03-24T00:49:12.020 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33817.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33817.log.gz 2024-03-24T00:49:12.023 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34914.log 2024-03-24T00:49:12.024 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50369.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50369.log.gz 2024-03-24T00:49:12.027 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45942.log 2024-03-24T00:49:12.028 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34914.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.34914.log.gz 2024-03-24T00:49:12.031 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41083.log 2024-03-24T00:49:12.032 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45942.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45942.log.gz 2024-03-24T00:49:12.036 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41083.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41083.log.gz 2024-03-24T00:49:12.038 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42845.log 2024-03-24T00:49:12.043 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39234.log 2024-03-24T00:49:12.044 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42845.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42845.log.gz 2024-03-24T00:49:12.047 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54818.log 2024-03-24T00:49:12.048 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39234.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.39234.log.gz 2024-03-24T00:49:12.051 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51117.log 2024-03-24T00:49:12.052 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54818.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54818.log.gz 2024-03-24T00:49:12.055 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52771.log 2024-03-24T00:49:12.056 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51117.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51117.log.gz 2024-03-24T00:49:12.059 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37732.log 2024-03-24T00:49:12.060 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52771.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52771.log.gz 2024-03-24T00:49:12.063 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36419.log 2024-03-24T00:49:12.063 INFO:teuthology.orchestra.run.smithi053.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-24T00:49:12.064 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37732.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37732.log.gz 2024-03-24T00:49:12.067 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32607.log 2024-03-24T00:49:12.068 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36419.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36419.log.gz 2024-03-24T00:49:12.073 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32607.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32607.log.gz 2024-03-24T00:49:12.074 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36185.log 2024-03-24T00:49:12.080 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38643.log 2024-03-24T00:49:12.081 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36185.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36185.log.gz 2024-03-24T00:49:12.084 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40710.log 2024-03-24T00:49:12.084 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38643.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38643.log.gz 2024-03-24T00:49:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39595.log 2024-03-24T00:49:12.088 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40710.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.40710.log.gz 2024-03-24T00:49:12.091 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52849.log 2024-03-24T00:49:12.092 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39595.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.39595.log.gz 2024-03-24T00:49:12.094 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53337.log 2024-03-24T00:49:12.096 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52849.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52849.log.gz 2024-03-24T00:49:12.099 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47633.log 2024-03-24T00:49:12.100 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53337.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53337.log.gz 2024-03-24T00:49:12.102 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39543.log 2024-03-24T00:49:12.104 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47633.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47633.log.gz 2024-03-24T00:49:12.108 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39543.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39543.log.gz 2024-03-24T00:49:12.110 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35070.log 2024-03-24T00:49:12.115 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53669.log 2024-03-24T00:49:12.116 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35070.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35070.log.gz 2024-03-24T00:49:12.119 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36003.log 2024-03-24T00:49:12.120 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53669.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53669.log.gz 2024-03-24T00:49:12.123 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35816.log 2024-03-24T00:49:12.124 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36003.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36003.log.gz 2024-03-24T00:49:12.125 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46710.log 2024-03-24T00:49:12.127 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35816.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35816.log.gz 2024-03-24T00:49:12.131 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43173.log 2024-03-24T00:49:12.132 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46710.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46710.log.gz 2024-03-24T00:49:12.135 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45303.log 2024-03-24T00:49:12.136 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43173.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43173.log.gz 2024-03-24T00:49:12.140 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45303.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45303.log.gz 2024-03-24T00:49:12.142 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49739.log 2024-03-24T00:49:12.147 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39364.log 2024-03-24T00:49:12.148 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49739.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49739.log.gz 2024-03-24T00:49:12.151 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2024-03-24T00:49:12.152 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39364.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.39364.log.gz 2024-03-24T00:49:12.155 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42245.log 2024-03-24T00:49:12.157 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40284.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gz 2024-03-24T00:49:12.159 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34025.log 2024-03-24T00:49:12.160 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42245.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42245.log.gz 2024-03-24T00:49:12.163 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34680.log 2024-03-24T00:49:12.164 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34025.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34025.log.gz 2024-03-24T00:49:12.165 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38483.log 2024-03-24T00:49:12.167 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34680.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34680.log.gz 2024-03-24T00:49:12.172 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38483.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38483.log.gz 2024-03-24T00:49:12.174 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48105.log 2024-03-24T00:49:12.179 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48367.log 2024-03-24T00:49:12.180 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48105.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48105.log.gz 2024-03-24T00:49:12.183 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41827.log 2024-03-24T00:49:12.186 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48367.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48367.log.gz 2024-03-24T00:49:12.186 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45679.log 2024-03-24T00:49:12.187 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43855.log 2024-03-24T00:49:12.188 INFO:teuthology.orchestra.run.smithi040.stderr: 87.7%/var/log/ceph/ceph-client.admin.45679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41433.log 2024-03-24T00:49:12.189 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43855.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43855.log.gz 2024-03-24T00:49:12.189 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41827.log.gz 2024-03-24T00:49:12.192 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41433.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41433.log.gz 2024-03-24T00:49:12.194 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38423.log 2024-03-24T00:49:12.195 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45679.log.gz 2024-03-24T00:49:12.196 INFO:teuthology.orchestra.run.smithi053.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-24T00:49:12.199 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54668.log 2024-03-24T00:49:12.200 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38423.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38423.log.gz 2024-03-24T00:49:12.203 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44354.log 2024-03-24T00:49:12.205 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.54668.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.54668.log.gz 2024-03-24T00:49:12.207 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50186.log 2024-03-24T00:49:12.209 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44354.log: 93.1% 88.7% -- replaced with /var/log/ceph/ceph-client.0.53878.log.gz 2024-03-24T00:49:12.209 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44354.log.gz 2024-03-24T00:49:12.209 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53823.log 2024-03-24T00:49:12.213 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50186.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50186.log.gz 2024-03-24T00:49:12.213 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51480.log 2024-03-24T00:49:12.213 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53823.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53823.log.gz 2024-03-24T00:49:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51480.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51480.log.gz 2024-03-24T00:49:12.216 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40892.log 2024-03-24T00:49:12.219 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40892.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40892.log.gz 2024-03-24T00:49:12.219 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32783.log 2024-03-24T00:49:12.225 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32783.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.32783.log.gz 2024-03-24T00:49:12.225 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35868.log 2024-03-24T00:49:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35868.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35868.log.gz 2024-03-24T00:49:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43527.log 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43527.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43527.log.gz 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52477.log 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52477.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52477.log.gz 2024-03-24T00:49:12.232 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34628.log 2024-03-24T00:49:12.235 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34628.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34628.log.gz 2024-03-24T00:49:12.235 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38955.log 2024-03-24T00:49:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38955.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.38955.log.gz 2024-03-24T00:49:12.238 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48023.log 2024-03-24T00:49:12.241 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48023.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48023.log.gz 2024-03-24T00:49:12.241 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42643.log 2024-03-24T00:49:12.246 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33739.log 2024-03-24T00:49:12.246 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42643.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42643.log.gz 2024-03-24T00:49:12.247 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33739.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47510.log 2024-03-24T00:49:12.247 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33739.log.gz 2024-03-24T00:49:12.249 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41511.log 2024-03-24T00:49:12.252 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41511.log: /var/log/ceph/ceph-client.admin.47510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36445.log 2024-03-24T00:49:12.254 INFO:teuthology.orchestra.run.smithi040.stderr: 87.9% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47510.log.gz 2024-03-24T00:49:12.254 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41511.log.gz 2024-03-24T00:49:12.255 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54718.log 2024-03-24T00:49:12.258 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36445.log: /var/log/ceph/ceph-client.admin.54718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27290.log 2024-03-24T00:49:12.260 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.27290.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.27290.log.gz 2024-03-24T00:49:12.262 INFO:teuthology.orchestra.run.smithi040.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.54718.log.gz 2024-03-24T00:49:12.263 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39260.log 2024-03-24T00:49:12.263 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36445.log.gz 2024-03-24T00:49:12.266 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39260.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39260.log.gz 2024-03-24T00:49:12.266 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38832.log 2024-03-24T00:49:12.269 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38832.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38832.log.gz 2024-03-24T00:49:12.269 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38129.log 2024-03-24T00:49:12.272 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38129.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38129.log.gz 2024-03-24T00:49:12.272 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53155.log 2024-03-24T00:49:12.275 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22482.log 2024-03-24T00:49:12.275 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53155.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53155.log.gz 2024-03-24T00:49:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.tmp-client.admin.22482.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22482.log.gz 2024-03-24T00:49:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46543.log 2024-03-24T00:49:12.281 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46543.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46543.log.gz 2024-03-24T00:49:12.281 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53214.log 2024-03-24T00:49:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53214.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53214.log.gz 2024-03-24T00:49:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40041.log 2024-03-24T00:49:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40041.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.40041.log.gz 2024-03-24T00:49:12.288 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38293.log 2024-03-24T00:49:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38293.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38293.log.gz 2024-03-24T00:49:12.291 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45035.log 2024-03-24T00:49:12.294 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53747.log 2024-03-24T00:49:12.294 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45035.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45035.log.gz 2024-03-24T00:49:12.296 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50851.log 2024-03-24T00:49:12.297 INFO:teuthology.orchestra.run.smithi040.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53747.log.gz 2024-03-24T00:49:12.300 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45113.log 2024-03-24T00:49:12.304 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45113.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45113.log.gz 2024-03-24T00:49:12.304 INFO:teuthology.orchestra.run.smithi040.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50851.log.gz 2024-03-24T00:49:12.305 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53181.log 2024-03-24T00:49:12.307 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53181.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53181.log.gz 2024-03-24T00:49:12.307 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47354.log 2024-03-24T00:49:12.310 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47354.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47354.log.gz 2024-03-24T00:49:12.311 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33947.log 2024-03-24T00:49:12.314 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33947.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33947.log.gz 2024-03-24T00:49:12.314 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48211.log 2024-03-24T00:49:12.315 INFO:teuthology.orchestra.run.smithi040.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-24T00:49:12.315 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41381.log 2024-03-24T00:49:12.316 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48211.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48211.log.gz 2024-03-24T00:49:12.317 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49868.log 2024-03-24T00:49:12.318 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41381.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41381.log.gz 2024-03-24T00:49:12.319 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51195.log 2024-03-24T00:49:12.320 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49868.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49868.log.gz 2024-03-24T00:49:12.321 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43999.log 2024-03-24T00:49:12.322 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51195.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51195.log.gz 2024-03-24T00:49:12.322 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41961.log 2024-03-24T00:49:12.324 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43999.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43999.log.gz 2024-03-24T00:49:12.324 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47216.log 2024-03-24T00:49:12.325 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41961.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41961.log.gz 2024-03-24T00:49:12.325 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38241.log 2024-03-24T00:49:12.327 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47216.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47216.log.gz 2024-03-24T00:49:12.327 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50903.log 2024-03-24T00:49:12.328 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38241.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.38241.log.gz 2024-03-24T00:49:12.329 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35951.log 2024-03-24T00:49:12.330 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50903.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50903.log.gz 2024-03-24T00:49:12.331 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50134.log 2024-03-24T00:49:12.332 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35951.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35951.log.gz 2024-03-24T00:49:12.332 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55637.log 2024-03-24T00:49:12.334 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50134.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50134.log.gz 2024-03-24T00:49:12.334 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39390.log 2024-03-24T00:49:12.335 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55637.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55637.log.gz 2024-03-24T00:49:12.335 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52149.log 2024-03-24T00:49:12.337 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39390.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.39390.log.gz 2024-03-24T00:49:12.338 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36901.log 2024-03-24T00:49:12.338 INFO:teuthology.orchestra.run.smithi040.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52149.log.gz 2024-03-24T00:49:12.339 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43684.log 2024-03-24T00:49:12.340 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36901.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36901.log.gz 2024-03-24T00:49:12.341 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39915.log 2024-03-24T00:49:12.342 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43684.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43684.log.gz 2024-03-24T00:49:12.342 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36055.log 2024-03-24T00:49:12.344 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39915.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39915.log.gz 2024-03-24T00:49:12.344 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49666.log 2024-03-24T00:49:12.345 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36055.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36055.log.gz 2024-03-24T00:49:12.346 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55930.log 2024-03-24T00:49:12.347 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49666.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49666.log.gz 2024-03-24T00:49:12.348 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35553.log 2024-03-24T00:49:12.348 INFO:teuthology.orchestra.run.smithi040.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55930.log.gz 2024-03-24T00:49:12.349 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36542.log 2024-03-24T00:49:12.350 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35553.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35553.log.gz 2024-03-24T00:49:12.351 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53472.log 2024-03-24T00:49:12.352 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36542.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36542.log.gz 2024-03-24T00:49:12.352 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38215.log 2024-03-24T00:49:12.353 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53472.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53472.log.gz 2024-03-24T00:49:12.353 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35209.log 2024-03-24T00:49:12.355 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38215.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.38215.log.gz 2024-03-24T00:49:12.356 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34576.log 2024-03-24T00:49:12.357 INFO:teuthology.orchestra.run.smithi040.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35209.log.gz 2024-03-24T00:49:12.357 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52641.log 2024-03-24T00:49:12.358 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34576.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34576.log.gz 2024-03-24T00:49:12.359 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52641.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52641.log.gz 2024-03-24T00:49:12.360 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34184.log 2024-03-24T00:49:12.361 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43823.log 2024-03-24T00:49:12.362 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34184.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34184.log.gz 2024-03-24T00:49:12.363 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43823.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43823.log.gz 2024-03-24T00:49:12.364 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55693.log 2024-03-24T00:49:12.365 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44585.log 2024-03-24T00:49:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.55693.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55693.log.gz 2024-03-24T00:49:12.367 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44585.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44585.log.gz 2024-03-24T00:49:12.368 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36719.log 2024-03-24T00:49:12.369 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32723.log 2024-03-24T00:49:12.371 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36719.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36719.log.gz 2024-03-24T00:49:12.371 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.32723.log: /var/log/ceph/ceph-client.admin.45355.log 2024-03-24T00:49:12.372 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32723.log.gz 2024-03-24T00:49:12.373 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42374.log 2024-03-24T00:49:12.374 INFO:teuthology.orchestra.run.smithi040.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45355.log.gz 2024-03-24T00:49:12.374 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52085.log 2024-03-24T00:49:12.376 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42374.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42374.log.gz 2024-03-24T00:49:12.376 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33559.log 2024-03-24T00:49:12.377 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52085.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52085.log.gz 2024-03-24T00:49:12.378 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34492.log 2024-03-24T00:49:12.380 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33559.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33559.log.gz 2024-03-24T00:49:12.381 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46338.log 2024-03-24T00:49:12.383 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34492.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.34492.log.gz 2024-03-24T00:49:12.383 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44276.log 2024-03-24T00:49:12.384 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46338.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46338.log.gz 2024-03-24T00:49:12.385 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38267.log 2024-03-24T00:49:12.386 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44276.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44276.log.gz 2024-03-24T00:49:12.386 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35527.log 2024-03-24T00:49:12.387 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38267.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38267.log.gz 2024-03-24T00:49:12.388 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41537.log 2024-03-24T00:49:12.389 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35527.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35527.log.gz 2024-03-24T00:49:12.390 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41537.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41537.log.gz 2024-03-24T00:49:12.391 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47380.log 2024-03-24T00:49:12.392 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44168.log 2024-03-24T00:49:12.393 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47380.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47380.log.gz 2024-03-24T00:49:12.394 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44168.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44168.log.gz 2024-03-24T00:49:12.395 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37411.log 2024-03-24T00:49:12.396 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33349.log 2024-03-24T00:49:12.397 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37411.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37411.log.gz 2024-03-24T00:49:12.398 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43920.log 2024-03-24T00:49:12.399 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33349.log.gz 2024-03-24T00:49:12.399 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40632.log 2024-03-24T00:49:12.401 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43920.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43920.log.gz 2024-03-24T00:49:12.402 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40632.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40632.log.gz 2024-03-24T00:49:12.403 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36263.log 2024-03-24T00:49:12.404 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38617.log 2024-03-24T00:49:12.406 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36263.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36263.log.gz 2024-03-24T00:49:12.406 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38617.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.38617.log.gz 2024-03-24T00:49:12.407 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52920.log 2024-03-24T00:49:12.409 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52920.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52920.log.gz 2024-03-24T00:49:12.664 INFO:teuthology.orchestra.run.smithi040.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-24T00:49:12.673 INFO:teuthology.orchestra.run.smithi040.stderr: 93.7% -- replaced with /var/log/ceph/ceph-client.0.48527.log.gz 2024-03-24T00:49:12.777 INFO:teuthology.orchestra.run.smithi040.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-24T00:49:13.045 INFO:teuthology.orchestra.run.smithi040.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-24T00:49:13.129 INFO:teuthology.orchestra.run.smithi053.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-24T00:49:13.217 INFO:teuthology.orchestra.run.smithi040.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-03-24T00:49:15.693 INFO:teuthology.orchestra.run.smithi053.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-24T00:49:15.695 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-03-24T00:49:15.695 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m5.050s 2024-03-24T00:49:15.696 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m12.973s 2024-03-24T00:49:15.696 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m1.000s 2024-03-24T00:49:16.297 INFO:teuthology.orchestra.run.smithi040.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-24T00:49:16.299 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-03-24T00:49:16.299 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m5.674s 2024-03-24T00:49:16.299 INFO:teuthology.orchestra.run.smithi040.stderr:user 0m17.374s 2024-03-24T00:49:16.299 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m2.272s 2024-03-24T00:49:16.300 INFO:tasks.ceph:Archiving logs... 2024-03-24T00:49:16.301 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439/remote/smithi040/log 2024-03-24T00:49:16.302 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T00:49:18.423 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439/remote/smithi053/log 2024-03-24T00:49:18.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T00:49:20.077 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-24T00:49:20.089 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T00:49:20.100 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-24T00:49:20.104 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T00:49:20.105 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T00:49:20.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T00:49:20.180 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T00:49:20.193 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T00:49:20.193 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:49:20.196 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:49:20.208 INFO:teuthology.orchestra.run.smithi040.stdout: remote refid st t when poll reach delay offset jitter 2024-03-24T00:49:20.208 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================== 2024-03-24T00:49:20.209 INFO:teuthology.orchestra.run.smithi040.stdout:*hv01.front.sepi 44.4.53.2 2 u 5 64 377 0.060 -1.733 0.102 2024-03-24T00:49:20.209 INFO:teuthology.orchestra.run.smithi040.stdout:+hv02.front.sepi 63.231.80.2 3 u 6 64 377 0.067 -0.367 0.125 2024-03-24T00:49:20.209 INFO:teuthology.orchestra.run.smithi040.stdout:+hv03.front.sepi 74.6.168.72 3 u 8 64 377 0.092 -2.540 0.117 2024-03-24T00:49:20.209 INFO:teuthology.orchestra.run.smithi040.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-03-24T00:49:20.212 INFO:teuthology.orchestra.run.smithi053.stdout: remote refid st t when poll reach delay offset jitter 2024-03-24T00:49:20.212 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================== 2024-03-24T00:49:20.212 INFO:teuthology.orchestra.run.smithi053.stdout:*hv01.front.sepi 44.4.53.2 2 u 4 64 377 0.084 -5.556 0.668 2024-03-24T00:49:20.212 INFO:teuthology.orchestra.run.smithi053.stdout:+hv02.front.sepi 63.231.80.2 3 u 5 64 377 0.118 -4.231 0.670 2024-03-24T00:49:20.213 INFO:teuthology.orchestra.run.smithi053.stdout:+hv03.front.sepi 74.6.168.72 3 u 23 64 377 0.106 -5.508 1.017 2024-03-24T00:49:20.213 INFO:teuthology.orchestra.run.smithi053.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-03-24T00:49:20.214 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T00:49:20.224 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T00:49:20.224 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T00:49:20.270 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T00:49:20.319 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T00:49:20.380 INFO:teuthology.task.internal:Duration was 1794.932156 seconds 2024-03-24T00:49:20.381 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T00:49:20.429 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T00:49:20.430 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T00:49:20.433 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T00:49:20.531 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T00:49:20.531 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-03-24T00:49:20.531 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T00:49:20.552 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-03-24T00:49:20.553 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T00:49:20.574 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T00:49:20.575 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T00:49:20.600 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T00:49:20.658 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T00:49:20.659 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T00:49:20.684 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T00:49:20.776 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T00:49:20.800 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T00:49:20.800 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T00:49:20.821 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T00:49:20.842 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T00:49:20.857 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:49:20.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:49:20.879 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-03-24T00:49:20.901 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-03-24T00:49:20.931 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:49:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:49:20.949 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:49:20.978 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi053.front.sepia.ceph.com, flagging run as failed 2024-03-24T00:49:20.979 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T00:49:20.995 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T00:49:20.996 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439/remote/smithi040 2024-03-24T00:49:20.997 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T00:49:21.026 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439/remote/smithi053 2024-03-24T00:49:21.027 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T00:49:37.644 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-03-24T00:49:37.658 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439/remote/smithi053/coredump/1711240788.33072.core.gz: gzip compressed data, was "1711240788.33072.core", last modified: Sun Mar 24 00:49:29 2024, max compression, original size modulo 2^32 272457728 2024-03-24T00:49:37.659 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-03-24T00:49:38.861 INFO:teuthology.task.internal: core looks like: /tmp/tmpfzr3199f: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph-mds -f --cluster ceph -i b', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/usr/bin/ceph-mds', platform: 'x86_64' 2024-03-24T00:49:38.914 INFO:teuthology.task.internal: dump_program: ceph-mds -f --cluster ceph -i b 2024-03-24T00:49:38.914 DEBUG:teuthology.orchestra.run.smithi053:> which 'ceph-mds -f --cluster ceph -i b' 2024-03-24T00:49:38.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:49:38.925 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 1: "which 'ceph-mds -f --cluster ceph -i b'" 2024-03-24T00:49:38.944 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T00:49:38.980 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T00:49:38.980 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T00:49:39.027 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T00:49:39.028 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T00:49:39.036 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T00:49:39.043 INFO:teuthology.orchestra.run.smithi040.stdout: 658116 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 24 00:49 /home/ubuntu/cephtest 2024-03-24T00:49:39.043 INFO:teuthology.orchestra.run.smithi040.stdout: 658117 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 24 00:49 /home/ubuntu/cephtest/archive 2024-03-24T00:49:39.043 INFO:teuthology.orchestra.run.smithi040.stdout: 658120 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 00:49 /home/ubuntu/cephtest/archive/syslog 2024-03-24T00:49:39.043 INFO:teuthology.orchestra.run.smithi040.stdout: 658122 4 -rw-rw-rw- 1 ubuntu ubuntu 2195 Mar 24 00:49 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-03-24T00:49:39.044 INFO:teuthology.orchestra.run.smithi040.stdout: 658121 204 -rw-rw-r-- 1 ubuntu ubuntu 205081 Mar 24 00:49 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T00:49:39.044 INFO:teuthology.orchestra.run.smithi040.stdout: 658123 200 -rw-rw-rw- 1 ubuntu ubuntu 201188 Mar 24 00:49 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-03-24T00:49:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:49:39.045 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-24T00:49:39.046 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2785, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2617, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 2383, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1638, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-03-24T00:49:39.063 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T00:49:39.073 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T00:49:39.103 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-03-24T00:49:39.103 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T00:49:39.112 INFO:teuthology.orchestra.run.smithi053.stdout: 655432 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 24 00:49 /home/ubuntu/cephtest 2024-03-24T00:49:39.112 INFO:teuthology.orchestra.run.smithi053.stdout: 658117 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 24 00:19 /home/ubuntu/cephtest/archive 2024-03-24T00:49:39.112 INFO:teuthology.orchestra.run.smithi053.stdout: 658120 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 00:49 /home/ubuntu/cephtest/archive/syslog 2024-03-24T00:49:39.112 INFO:teuthology.orchestra.run.smithi053.stdout: 658122 4 -rw-rw-rw- 1 ubuntu ubuntu 2172 Mar 24 00:49 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-03-24T00:49:39.112 INFO:teuthology.orchestra.run.smithi053.stdout: 658121 184 -rw-rw-r-- 1 ubuntu ubuntu 185770 Mar 24 00:49 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T00:49:39.113 INFO:teuthology.orchestra.run.smithi053.stdout: 658123 172 -rw-rw-rw- 1 ubuntu ubuntu 174336 Mar 24 00:49 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-03-24T00:49:39.113 INFO:teuthology.orchestra.run.smithi053.stdout: 658118 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 00:40 /home/ubuntu/cephtest/archive/coredump 2024-03-24T00:49:39.113 INFO:teuthology.orchestra.run.smithi053.stdout: 678723 37924 -rw------- 1 root root 288190464 Mar 24 00:40 /home/ubuntu/cephtest/archive/coredump/1711240807.33073.core 2024-03-24T00:49:39.113 INFO:teuthology.orchestra.run.smithi053.stdout: 677905 38700 -rw------- 1 root root 272457728 Mar 24 00:39 /home/ubuntu/cephtest/archive/coredump/1711240788.33072.core 2024-03-24T00:49:39.114 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-24T00:49:39.131 DEBUG:teuthology.nuke:shortname: smithi040 2024-03-24T00:49:39.131 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T00:49:39.151 DEBUG:teuthology.nuke:shortname: smithi053 2024-03-24T00:49:39.152 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T00:49:39.156 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:10:01.929763', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOIROjU8alZ2MN5QoW9J73ZR2EA4xzHHJqXzdUUZhxpZn2xukBbmH5MmEP4f2xX3XR+PCznb+rsWN6qVGjT02Zg='} 2024-03-24T00:49:39.172 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616439', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 00:10:01.927952', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKNKWqVs1XCeXnKRCFKfBlzYJzTo46EZn7pnrIMmZ42DE0CEaUJrjKTjobTmS/4GXn53Bvm8aYKZEGA5g+eICNI='} 2024-03-24T00:49:39.202 INFO:teuthology.orchestra.console.smithi040:Power off 2024-03-24T00:49:39.203 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T00:49:39.217 INFO:teuthology.orchestra.console.smithi053:Power off 2024-03-24T00:49:39.217 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T00:49:39.227 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-03-24T00:49:39.227 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:49:39.239 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-03-24T00:49:39.239 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:49:39.253 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-03-24T00:49:39.261 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-24T00:49:43.255 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:49:43.264 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:49:43.366 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-03-24T00:49:43.373 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-03-24T00:49:47.369 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:49:47.379 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T00:49:47.481 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-03-24T00:49:47.489 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-03-24T00:49:47.583 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-03-24T00:49:47.591 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-03-24T00:49:47.696 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/exports} duration: 1794.9321556091309 failure_reason: 'Test failure: test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins), test_ephemeral_pin_distribution (tasks.cephfs.test_exports.TestEphemeralPins)' flavor: default mon.a-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 mon.b-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 owner: mchangir sentry_event: https://sentry.ceph.com/organizations/ceph/?query=809d1b44f79741a2b62cc123e8ffbdd3 status: fail success: false 2024-03-24T00:49:47.696 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T00:49:47.782 INFO:teuthology.run:FAIL